The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
147,283 characters
Model-Estimation-Free, Dense, and High Dimensional Consistent Precision Matrix Estimators
\def\spacingset#1{\renewcommand{\baselinestretch}
{#1}\small\normalsize} \spacingset{1}
\title{Model-Estimation-Free, Dense, and High Dimensional Consistent Precision Matrix Estimators}
\author{\textsc{Mehmet Caner\thanks{
North Carolina State University, Nelson Hall, Department of Economics, NC 27695. Email: [email removed]. }}
\and \textsc {Agostino Capponi
\thanks{Department of Industrial Engineering and Operations Research, Columbia University. Email: [email removed].}}
\and \textsc{Mihailo Stojnic
\thanks{Department of Industrial Engineering and Operations Research, Columbia University. Email: [email removed]. }}}
\maketitle
\begin{abstract}
Precision matrix estimation is a cornerstone concept in statistics, economics,
and finance. Despite advances in recent years, estimation methods
that are simultaneously (i) dense, (ii) consistent, and (iii) model-free are lacking. While each of these targets can be met separately, achieving
them together is challenging.We address this gap by introducing a general class of estimators that unifies these features within a nonasymptotic framework, allowing for explicit characterization of the computational complexity–signal-to-noise ratio trade-off.
Our analysis identifies three fundamental random quantities—complexity, signal magnitude, and method bias—that jointly determine estimation error. A particularly striking result is that ridgeless regression, a tuning-free special case within our class, exhibits the double descent phenomenon. This establishes the first formal precision matrix analogue to the well-known double descent behavior in linear regression. Our theoretical analysis is supported by a thorough empirical study of the S\&P
500 index,
where we observe a doubly ascending Sharpe ratio pattern, which complements the double descent phenomenon.
\end{abstract}
\noindent {\bf{Keywords}}: Interpolation, Double Descent, Latent Space, Factor Models, Precision Matrix .
\newpage
\spacingset{1.9}
\section{Introduction}\label{intro}
Due to their usage in high dimensional portfolio formation in finance, network analysis in economics
and debiased policy coefficient estimation in cross section-causality analysis, precision matrix estimation has been among the few cornerstone topics in statistics and finance in recent years.
We propose a general class of estimators that are simultaneously {\it model estimation-free},
{\it dense}, and {\it consistent}. Existing literature can handle separately one of these properties, but no precision matrix estimator can handle all of them {\it jointly}. Such a joint handling is highly desired
as it is naturally expected to lead to better empirical performances.
Our design sharply differentiates our approach from classical shrinkage, factor-model, and nodewise-regression methods. For
example, unlike shrinkage type of estimators, e.g., \cite{lw2004b}, ours converge to precision
matrix, not to a modified version of it. Compared to dense-model estimation free nonlinear shrinkage estimator of \cite{lw2017}, our estimator is high dimension consistent.
Differently from the hidden factor model based estimator of , e.g., \cite{fan2013},
we do not have to impose identification, estimation, and rotation which allows for
model free estimation. Finally, in sharp contrast to the nodewise regression, e.g., \cite{caner2019},
\cite{mein2006}, we do not impose arbitrary sparsity assumption on the rows of precision matrix nor do we use penalized estimators that creates shrinkage bias. Our main contributions are summarized below.
(i). We
introduce a general class of consistent-non-sparse and unpenalized precision matrix
estimators. Linking a set of general dense-unpenalized estimators to precision matrix formula is new and novel. This is done without shrinkage, penalization or factor estimation.
Hidden factor analysis is used only for achieving consistency via dimension reduction. (ii). We derive non-asymptotic upper bounds for estimation errors that allow us to see the effects of random variable that captures model complexity, number of confounding factors, and signal-to-noise ratio.(iii). Our class encompasses a broad range of precision matrix estimators, including ridgeless regression. For this estimator, we show the double descent due to the increasing size of the signal, when the number of outcomes exceeds the number of observations.(iv). Theoretical findings are complemented with very good empirical performance in S\&P 500.
.
\subsection{Literature Review}\label{litsurv}
Several studies in the statistics literature have studied precision matrix estimation. An excellent survey is by \cite{crz2016} which shows neighborhood and penalization based approaches. We refer the readers to that survey for the literature and appendix that we have.
A distinct approach, commonly used in finance and statistics, derives the precision matrix by inverting the covariance matrix using the Sherman–Morrison–Woodbury (SMW) formula. In this framework, sparsity is assumed in the covariance matrix of the idiosyncratic errors, while asset returns are modeled through a factor structure (see \cite{fan2013}, \cite{caner2022}). Our approach is different from the
approaches discussed in the literature. First of all, we are not assuming exact sparsity in the precision matrix of outcomes. We are also not using SMW formula to
invert the covariance matrix.
Another approach is nodewise based.
Note that nodewise regression based estimator of \cite{caner2019}
uses sparsity of precision matrix of outcomes. Then residual nodewise regression estimator of \cite{caner2022} crucially depends on observed factors, and imposes sparse precision matrix of errors. Technique used in those papers cannot extend to hidden factors since observed factor structure in \cite{caner2022} uses OLS estimation which is not applicable to more complicated estimation problems such as hidden factor structure.
Recently, \cite{cd2025} developed a new deep learning based precision matrix estimator. However, the estimator is consistent only when number of outcomes is less than or equal to sample size, which limits applications. We also tie our results to signal-to-noise ratio aspect which is not available in the studies of \cite{caner2019}, \cite{caner2022}.
Our method does not impose sparsity like nodewise regression of \cite{mein2006}.
Nodewise regression uses an $l_1$ norm based penalization to form the rows of the precision matrix. Doing so, the underlying assumption is the imposition of sparsity in the precision matrix which may be interpreted as a strong assumption. An excellent paper by \cite{fan2013} provides
a statistical high-dimensional analysis of covariance matrix of outcomes via hidden factors based estimation.
Similar to us, it also relies on hidden factors but uses a thresholding based principal components estimator for the covariance matrix, and then inverts this through SMW formula. The key difference is that our estimator is model estimation-free and uses hidden factors for consistency proof only.
Our paper differs from \cite{bbsmw21} and \cite{bsmw22} in a fundamental way. These papers analyze general linear class of estimators in hidden factor models in linear regression model and is only interested in risk of the estimators. Their papers consider only one outcome variable and how it can be predicted with this new general class of estimators.
Our analysis on the other hand, considers the precision matrix estimation of outcome variable in linear regression via hidden factors.
In our paper, a novel idea is to use a linear linkage between outcome variables to obtain precision matrix estimator. One of the main difficulties, theoretically, is that the estimation of the main diagonal terms in the precision matrix estimation. The proof technique and ideas in \cite{bbsmw21}, \cite{bsmw22} do not simply extend to this case.
Another set of estimators are shrinkage based ones. For example \cite{lw2004b} proposes a linear shrinkage estimator for covariances in iid data setting. This is to solve the invertibility problem of sample covariance matrix. Their estimator for covariance is dense, model-free hence making a major contribution. However, unlike our estimators, the estimator converges to a modified version of the covariance consistently not to the covariance matrix itself. \cite{lw2017} propose a non-linear shrinkage estimator. This estimator specifically analyzes out-of-sample variance loss in iid data setting, and develop a modified covariance matrix estimator that increases the small eigenvalues, and decreases the large ones. It is a model-free dense estimator but the out-of-sample variance loss limit is different from the population one.
Unlike our estimator, the precision matrix estimator in their paper is not consistent. But as mentioned, their main concern is different than consistency in high dimensions. More discussion about literature and hidden factors are in Appendix C.
We consider notation now.
Define $C,c $ as generic positive constants.
Note that transpose of any matrix $Z: d \times l$ is represented by $Z'$, and transpose of a column vector, $z$ is represented by $z'$. Let $\| Z \|_2, \| Z \|_F$ represent the spectral norm and Frobenius norm of matrix $Z$ respectively, and $\|z\|_2$ the Euclidean norm of a vector $z$. Let $\| Z \|_{l_{\infty}}:= \max_{1 \le j \le d} \| Z_{j}\|_1$ be the maximum row sum norm, with $Z_{j}: l \times 1$ as the $j$ th row among $d$ rows in matrix Z, written in column format. The row format for $j$ th row of $Z$ is $Z_j': 1 \times l$. Let $\| Z \|_{\infty}:= \max_{i,j} |Z_{ij}|$, with $Z_{ij}$ representing $i,j$ cell of matrix Z.
Denote the $k$ th largest singular value of $Z$ matrix as $\sigma_k (Z)$ with $k=1$ being the largest singular value. Let $Z^{+}$ represent the Moore-Penrose inverse of matrix Z.
Let $\lambda_k (M)$ represent the $k$ th largest eigenvalue of $M: d \times d$, a square matrix, except $k=1, k=d$. However, for ease of understanding, the largest eigenvalue of $M$ is denoted as $Eigmax (M)$ and the minimum eigenvalue of $M$ is $Eigmin (M)$. $tr(M)$ represents the trace of matrix $M$.
Section \ref{model} proceeds with the idea of dimension reduction through hidden factors to achieve high dimensional consistency in the proofs. Section \ref{lgce} introduces the estimators. Section \ref{unicon} has consistency results.
Section \ref{empirics} provides empirical evidence. Conclusion shows future possible extensions of our work. Simulations, some of the empirics and all of the proofs, discussion about literature and hidden factors are relegated to Appendix.
\section{Dimension Reduction}\label{model}
In our setup we have $p$ outcome variables, and they are explained by $K$ common hidden factors to ensure dimension reduction, with $K <<p$. This will help in asymptotics and achieving consistency, however our estimators will be model estimation-free and will not use hidden factors estimators. Our estimators will only use outcome variables and this will be clear in the next section. Specifically, our estimators are defined in (\ref{3.1})(\ref{14a}).
Let $Y$ represent the $n \times p$ matrix of outcomes, each row representing the $t$ th time period, $t=1,\cdots,n$, and $j=1,\cdots,p$ representing the outcomes. $F$ represents $n \times K$ matrix of hidden factors, where each row, $t=1,\cdots, n$ represents a time period, and columns represent the hidden factors, $k=1,\cdots, K$. $A$ represents $p \times K$ matrix of factor loadings, and $U: n \times p$ as the error matrix.
We have the following compact form of the linear hidden factor structure for dimension reduction (which will be decomposed via rows of Y into two equations immediately below in (\ref{1})-(\ref{2})).
\begin{equation}
Y = F A' + U.\label{0}
\end{equation}
In accordance with our analysis, we subdivide all $p$ outcomes into $j$ th one and all the remaining ones. In that respect, start by defining each row of the outcome as $y_t: p \times 1$ (represents each row, in column form, whereas $y_t':1 \times p$ is the row vector form) of $Y$ above, and
$f_t:K \times 1$ represents each row of $F$, and $u_t: p \times 1$ is the $t$ th row of U. We decompose $A$ into $a_j: K \times 1$ factor loading vector for outcome j, and $A_{-j}: p-1 \times K$ as factor loading matrix for all other outcomes except $j$.
We model $y_{jt}$ as the $j$ th outcome variable at time period $t=1,\cdots,n$.
We define all the outcomes except $j$ th outcome as $Y_{-jt}: p-1 \times 1$ vector at time $t$. We decompose (\ref{0}) into $j$ th outcome and the rest.
In the $t$ th row of $Y$ in (\ref{0}),
$j$ th outcome can be written as
\begin{equation}
y_{jt}= f_t' a_j + u_{jt},\label{1}
\end{equation}
and we write all the outcomes except from the $j$ th one, in $t$ th row as
\begin{equation}
Y_{-jt}= A_{-j} f_t + U_{-jt}.\label{2}
\end{equation}
We use (\ref{1})(\ref{2}) in our proofs rather than (\ref{0}), and in this way our proofs are clear and tractable.
Common hidden factors affect the $j$ th outcome in (\ref{1}) as well as all remaining outcome variables in (\ref{2}). Let $u_{jt}, U_{-jt}$ represent the zero-mean errors for the $j$ th outcome and all other remaining outcomes, respectively. In vector form, the j-th column of $Y$ in (\ref{0}) can be written as
\begin{equation}
y_j= F a_j + u_j,\label{3}
\end{equation}
where $y_j: n \times 1$, $F: n \times K$ with each row as $f_t'$. Then the remainder, except for the $j$ th column, of Y is:
\begin{equation}
Y_{-j}= F A_{-j}' + U_{-j},\label{4}
\end{equation}
where $Y_{-j}: n \times p-1$ matrix, and $U_{-j}: n \times p-1$ matrix too.
As we will see later on, the vector form representation given in (\ref{3})(\ref{4}) will turn out to be beneficial in our proofs as well.
Define the variance of $j$ th error as $var(u_{jt}) = \sigma_{j}^2 \ge c >0$,where $c$ is a positive constant at time $t$, for $j=1\cdots,p$. Define the variance matrix of errors except the $j$ th one as
$var (U_{-jt}) = \Sigma_{U,-j}: p-1 \times p-1$, for $j=1,\cdots, p$ at time $t$.
Define the covariance matrix of factors as $\Sigma_f:= cov (f_t): K \times K$. Define $y_t=( y_{jt}, Y_{-jt}')'$ as the re-ordered $t$ th row of $Y$, and $\mu_j:= E y_{jt}: 1 \times 1$, $\mu_{-j}:= E Y_{-jt}: p-1 \times 1$, and $\mu:= E y_t: p \times 1$ at time $t$. Let the covariance matrix of outcomes be $\Sigma:= E (y_t - \mu) (y_t - \mu)'$. Assumption \ref{as1} below will show iid nature of our data across $t=1,\cdots,n$.
Define the precision matrix as
$\Theta:=\Sigma^{-1}$. We provide Assumptions next.
\begin{assum}\label{as1}
(i). $y_{jt}, Y_{-jt}$ are iid across $t=1,\cdots, n$ for each $j=1,\cdots,p$. (ii). $u_{jt}, U_{-jt}$ are zero mean random variable and vector respectively, which are iid across $t=1,\cdots,n$, for each $j=1,\cdots,p$. (iii). $f_t$ are iid across $t=1,\cdots, n$. (iv). $f_t, u_{jt}, U_{-jt}$ are mutually independent across $t=1,\cdots,n $ given $j=1,\cdots,p$.
\end{assum}
Assumption 1 is a standard convention used in hidden factor literature (see, e.g., \cite{bbsmw21} and \cite{bsmw22}.\footnote{Observe that (i)-(iii) implies that rows of F, Y, and U are iid.} Note also, that independence of errors implies $Eu_{jt}U_{-jt} = 0$ and $ u_t = (u_{jt},U_{-jt}')'$ which is a sparsity restriction
on the covariance matrix of errors. This is another assumption often present in the literature (see, e.g., \cite{fan2008}) and amounts to a diagonal error covariance matrix. Relaxing this
part of the assumption is certainly an interesting topic well worth of further exploring.
Relaxing the iid structure of data for financial applications is a good idea, however at this point our proofs, via Assumption 4 below as well, use subgausssian-iid based quadratic form concentration inequalities in high dimensional probability literature. Having a different structure in the proofs and solving time series nature is a key issue that we can follow up in future directions tied to financial applications.
We are not imposing sparsity on the links between outcome variables as done in nodewise regression context.
\begin{assum}\label{as2}
(i). $\min_{1 \le j \le p} Eigmin (\Sigma_{U,-j}) \ge c > 0$, where $Eigmin (\Sigma_{U,-j}) $ represents the minimum eigenvalue of the matrix $\Sigma_{U,-j}$. Also
$\max_{ 1 \le j \le p} \sigma_{j}^2 = \sigma^2 < \Gamma < \infty$, with $\Gamma$ as a large positive constant, and also $\min_{1 \le j \le p} \sigma_{j}^2 \ge c >0$.
(ii). Let $\beta:= \min_{1 \le j \le p} \| \Sigma_{U,-j} \|_2$, and $\beta>0$ is a large positive constant , and also
$ \max_{1 \le j \le p } \| \Sigma_{U,-j} \|_2 < \Gamma$, where $\Gamma>0$ is described in (i). Clearly $\beta < \Gamma$.
\end{assum}
Assumption 2(i) is also standard in high dimensional statistics literature. It
allows for non-singular noise matrices, large noise magnitudes. Since $\Sigma_{U,-j}$ is diagonal and diagonals have error variances, we use a common upper bound for a single variance in (i) and maximum variances over errors in (ii). This large positive $\Gamma$ is very much reflective of financial
markets.
\begin{assum}\label{as3}
(i). Define a positive constant $C_p>0$. Then assume the following relation between the number of outcomes, $p$ and the sample size $n$:
$ p \le n^{C _p/2}.$ (ii). For each $j=1,\cdots,p$, $A_{-j}: p-1 \times K$ has full rank $K$, and $p > K+1$. We impose $\max_{1 \le j \le p} \| A_{-j} \|_2 = O (\sqrt{p})$.
Also, $\Sigma_f$ has full rank $K$, and $Eigmax (\Sigma_f ) \le C < \infty$, for C a positive constant.
\end{assum}
With $C_p>2$, we can obtain $p>n$ and we cover high dimensional cases. In case of
$0<C_p \le 2$, we cover low dimensional cases, $p \le n$. This assumption also allows $p = \gamma n$, with $\gamma \in [0, \infty)$.
We allow the spectral norm of $A_{-j}: p-1 \times K$ matrix to be diverging in $n$, reflective of the high dimensional nature of $A_{-j}$, where both dimensions are allowed to grow. Specifically in (ii), we allow for pervasive factors with $\max_{j} \| A_{-j} \|_2 = C \sqrt{p}$, $C $ being a generic positive constant as well as weak factor structures if $\max_j \| A_{-j} \|_2 = O (d_{1n})$, with $d_{1n}/\sqrt{p}\to0$ with $p \to \infty, n \to \infty$. Hence our assumption is a weak one covering several possibilities.
\begin{assum}\label{as4}
(i). For each $j=1,\cdots,p$ let
$u_{jt}$ be zero mean $\sigma_{j} \gamma_e$ subgaussian random variable which means for each $j=1,\cdots,p$ $E exp (t_1 u_{jt}) \le exp(t_1^2 \sigma_{j}^2 \gamma_e^2/2)$ for all $ t_1 \in R$. Define the zero mean vector $U_{-j,t}:= \Sigma_{U,-j}^{1/2} \tilde{U}_{-j,t}$, where $\tilde{U}_{-j,t}$ ($p-1 \times 1$ vector) is $\gamma_{wj}$ sub-Gaussian vector which means
for any unit vector ($p-1$ dimension) $v$, $v' \tilde{U}_{-j,t} $ is $\gamma_{wj}$ sub-Gaussian.
Also we assume $E \tilde{U}_{-j,t} \tilde{U}_{-j,t}' = I_{p-1}$. Without losing any generality in the proofs we can use $\gamma_w$ constant instead of changing constants for each $j$, and $\gamma_{wj}$. (ii). Define $f_t:=\Sigma_f^{1/2} \tilde{f}_t$, where $\tilde{f}_t: K \times 1$,
is a subgaussian vector with $ E [\tilde{f}_t \tilde{f}_t'] = I_K$, and $\max_{1 \le k \le K } | E f_{tk} | \le C < \infty$, where $C>0$ is a positive constant.
(iii). We assume
$\| A \|_{\infty} \le C < \infty$.
\end{assum}
Assumption \ref{as4} is standard assumption used in \cite{bbsmw21} \cite{bsmw22}.
This assumption
is needed for quadratic form high dimensional probability inequalities.
It should be noted that we also relax the non-zero factor mean assumption used in earlier literature in \cite{bbsmw21}, and \cite{bsmw22}. Define the uniform version of the signal to noise ratio
\begin{equation}
\bar{\xi}= \min_{1 \le j \le p} \xi_j,\label{sn1}
\end{equation}
where as in \cite{bsmw22}, they have a pointwise version
\begin{equation}
\xi_j:=\frac{\lambda_K (A_{-j} \Sigma_f A_{-j}')}{\| \Sigma_{U,-j} \|_2},\label{sn2}
\end{equation}
with $\lambda_K (A_{-j} \Sigma_f A_{-j}' )$ representing the $K$ th largest eigenvalue of $A_{-j} \Sigma_f A_{-j}': p-1 \times p-1$ matrix, and that is the smallest positive eigenvalue of that matrix since $p-1 > K$, and $\Sigma_f: K \times K$ matrix. We have a discussion about the role of the signal-to-noise ratio in a specific scenario. Let us assume a weaker factor structure in Assumption \ref{as3}(ii), which is $\max_j \| A_{-j}\|_2 = O (d_{1n})$.
Since $A_{-j}: p-1 \times K$ for all matrices $j=1,\cdots,p$, via Assumption \ref{as3}, if the largest eigenvalue of $A_{-j}' A_{-j}$ grows at the same rate $K$-th largest eigenvalue, we can assume $\lambda_K (A_{-j}' A_{-j}) \asymp d_{1n}^2$ and since we impose $ max_{1 \le j \le p } \| \Sigma_{U,-j} \|_2 < \Gamma$ by Assumption \ref{as3}
then
\[ \lambda_K (A_{-j} \Sigma_f A_{-j}') \ge Eigmin (\Sigma_f) \lambda_K (A_{-j} A_{-j}') \ge c \lambda_K (A_{-j}' A_{-j}) \asymp d_{1n}^2 .\]
Utilizing (\ref{sn2}) and relying on Assumption \ref{as3} we obtain a useful signal-to-noise ratio lower bound formulation
\begin{equation}
\bar{\xi} \ge \frac{c d_{1n}^2}{\Gamma}.\label{ex1}
\end{equation}
Depending on how fast/slow $d_{1n}$ grows the signal-to-noise ratio lower bound in (\ref{ex1}) can fluctuate between fairly small and fairly large quantities. Assumption \ref{as5} below covers diagonal estimation of precision matrix. Note that $\bar{r}, \bar{\Psi}, \bar{\eta}$ are three random sequences that are explained in detail before Theorem 1, and they represent the complexity, bias, and size of the signal respectively. In special cases of RRE, and adaptive PCR, we will show non-random upper bounds for these three key quantities.
To state Assumption \ref{as5} below, we find it convenient to introduce rate of convergence, $r_{w1}$ which is used for off-diagonal term estimation.
\begin{equation}
r_{w1}:= \max \left( \sqrt{\frac{log n + \bar{r}}{n \bar{\eta}}}, \sqrt{\frac{K }{\bar{\xi} }}, \sqrt{ \frac{K}{\bar{\xi}} \frac{\bar{\Psi}}{\bar{\eta}} },
\sqrt{\frac{K}{\bar{\xi}} \frac{1}{\bar{\eta}}}
\right).\label{eqrw1}
\end{equation}
Define the relation between the factor loadings of $j$ th asset with all the others as
$d_{2n}:= \max_{1 \le j \le p} \| a_j' \Sigma_f A_{-j}'\|_2$.
\begin{assum}\label{as5}
(i). $ \frac{1}{\sqrt{K}} \frac{\sqrt{logn}}{\sqrt{logp}}\to 0$. (ii). $\sqrt{\frac{K^3}{n}} \to 0, \quad K \frac{\sqrt{ max (log p, log n)}}{n^{1/2}} \to 0.$ (iii). $K \sqrt{\frac{p(log p)}{n}} r_{w1} \stackrel{p}{\to} 0$, and $ d_{2n} r_{w1} \stackrel{p}\to 0$, and $d_{2n}$ is nondecreasing in $n$.
\end{assum}
While the meaning of Assumption \ref{as5} will become clearer when we get to our final results, a few introductory remarks regarding the origin of the assumption are in place. First, Assumption \ref{as5}(i)
is a technical assumption that facilitates the proof of Theorem \ref{thma} for the finite
sample results. Assumption \ref{as5}(ii) relates to the tradeoff between number of factors and time
span. In particular, it points out that the number of factors should be much smaller than the time span. Assumption \ref{as5}(iii) involves
stochastic terms and relates to consistency and rate of convergence of the off-diagonal
terms estimates. As we will see later on, the role of Assumption \ref{as5}(iii) will be discussed separately for adaptive PCR and
RRE estimators. But to provide some basic intuition in a simple subcase, let $K=1$, and
$\| \Sigma_{U,-j}\|_2 = I_{p-1}$, with $var(f_t)=1$, and set $r_{w1} := 1 / \sqrt{\xi}$
\[ d_{2n} r_{w1} = \frac{\max_{j} \sqrt{\sum_{i \neq j} (A_{i,-j} a_j)^2}}
{\min_j \sqrt{\sum_{i \neq j} A_{i,-j}^2
}} \to 0
,\]
where $A_{i,-j}$ is $i$ th element of vector $A_{-j}$ (see that this is a vector since $K=1$). So this is basically factor loadings correlations among themselves declining to zero.
\section{Linear General Class of Estimators}\label{lgce}
We propose estimating the rows of the precision matrix of asset returns with our estimator. To show consistency of our estimators in high dimensional context
we need to start with a precision matrix formula that can be tied to hidden factors.
One of the main obstacles is establishing relation between
the matrix inversion and linear regression within hidden factor context. The difficulty is that formula for main diagonal terms and off-diagonal terms of the precision matrix is different and this type of approach tying hidden factors and general linear class of estimators with precision matrix estimation has not been done before.
Define the following parameter vector which will form a key term in precision matrix formula:
\[ \alpha_j^*:= argmin_{\alpha_{j} \in R^{p-1}} E [ (y_{jt} - \mu_j) - (Y_{-jt} - \mu_{-j})' \alpha_{j}]^2.\]
This definition shows that we can relate to linear regression context. Note that, the definition of $\alpha_j^*$ is not coming from the hidden factor model directly in (\ref{1})(\ref{2}).
We also need the following definitions before the next Lemma. Let $\Sigma_f = \Sigma_f^{1/2} \Sigma_f^{1/2}$ since $\Sigma_f$ is a positive definite matrix.
Set $\bar{A}_{-j}:= A_{-j} \Sigma_f^{1/2}$, and $\bar{a}_j:= \Sigma_f^{1/2} a_j$, and
$ \bar{G}_j:= I_K + \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}.$
Next, we show how they relate to hidden factor structure. Let $\Theta_{j,j}$ represent the main $j$ th diagonal term, and $\Theta_{j,-j}$ represent the $j$ th row without $j$ th element. One key result in Lemma \ref{pm-form} is that we will not impose arbitrary zero-exact sparsity restrictions on $\alpha_j^*$ as done in
nodewise regression in \cite{mein2006}, \cite{caner2019}.
\begin{lemma}\label{pm-form}
Under Assumptions \ref{as1}-\ref{as3}, for $j=1,\cdots,p$
(i). \[ \Theta_{j,j}= \frac{1}{\tau_j^2},\quad
{\mbox with } \quad
\tau_j^2= (\bar{a}_j' \bar{a}_j + \sigma_j^2) - \bar{a}_j' \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{a}_j .\]
(ii). \[ \Theta_{j,-j}= \frac{-\alpha_j^{*'}}{\tau_j^2},\quad
\mbox{with} \quad \alpha_j^*= \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{a}_j.\]
(iii). Combining (i)(ii), we form $j$ th row of the precision matrix formula in terms of a hidden factor structure. Stacking each row will produce $\Theta$, the precision matrix.
\end{lemma}
Connecting further to the existing literature, one should note that \cite{bbsmw21} propose linear regression coefficient estimation and analyze the risk of an asset in a hidden factor model. However, neither the precision
matrix estimation nor the above formulae are discussed therein.
Lemma 1 and Theorem 1 below show how we manage to find a way to base the estimation on hidden factors but without actually imposing any identification
conditions on hidden factors. In that respect for each $j=1,\cdots,p$, we define
\begin{equation}
\tilde{\alpha}_j:= \hat{B} (Y_{-j} \hat{B})^+ y_j = \hat{B} ( \hat{B}' Y_{-j}' Y_{-j} \hat{B})^+ ( \hat{B}' Y_{-j}' y_j) ,\label{3.1}
\end{equation}
where $(.)^+$ is the Moore-Penrose inverse of a matrix, and $\hat{B}: p-1 \times q$ with $q \le p-1$ matrix, and the second equality is by (134) of \cite{bsmw22}. Dimension $q$ will be tied to specific estimators, and immediately below, we give examples about that. We let $\hat{B}$ be a deterministic matrix or a matrix that depends on $Y_{-j}$.
Our estimator is based on analysis of precision matrix in Lemma \ref{pm-form}.
Note that one of the main contributions of this paper is to link estimators (\ref{3.1}) with $\alpha_j^*$ which is related to the matrix inversion.
With different values of $\hat{B}$ we obtain different estimators. To see that, $\hat{B}= I_{p-1}$ is the ridgeless regression estimator (RRE) which is analyzed also after this section separately, so $q=p-1$. In case of adaptive PCR, $\hat{B}= V_{1,k_j}$ with $V_{1,k_j}$ being the $p-1 \times k_j $ matrix where the columns are the eigenvectors of $Y_{-j}' Y_{-j}/n$ matrix corresponding to the largest $k_j$ eigenvalues, and $q=k_j$, $j=1,\cdots,p$. We do not subscript $\hat{B}$ with $j$ to simplify the notation. \footnote{An optimal $\hat{B}$ may be analyzed but without knowing its structure it may be difficult and it is beyond current scope of the paper}. Let us define the reciprocal of the main diagonal terms in the precision matrix based on (B.11)\footnote{Note that only in case of RRE estimator as the subcase when $p-1>n$ the estimator simplifies and we benefit from that expression.}
\begin{equation}
\tilde{\tau}_j^2:= \frac{y_j' (y_j - Y_{-j} \tilde{\alpha}_j)}{n}.\label{14a}
\end{equation}
Then the $j$ the main diagonal term estimator in the precision matrix is:
\begin{equation}
\tilde{\Theta}_{j,j} = 1/\tilde{\tau}_j^2,\label{18}
\end{equation}
and
the $j$ th row of the precision matrix estimate without $j$ th element (off-diagonal terms in the row) is, via (B.13)
\begin{equation}
\tilde{\Theta}_{j,-j}= -\tilde{\alpha}_j'/\tilde{\tau}_j^2.\label{19}
\end{equation}
The $j$ th row of the estimate, to give an example, at $j=1$ will be $\tilde{\Theta}_1':= ( \frac{1}{\tilde{\tau}_1^2}, \frac{-\tilde{\alpha_{1}}'}{\tilde{\tau}_1^2}).$
\section{Uniform Consistency of General Estimator}\label{unicon}
In this section we provide first upper bounds on $l_2$ norm errors for each regression coefficient estimate, that is in (\ref{3.1}). Next, we consider (\ref{14a}), and obtain upper bounds for the estimation errors.
\subsection{Bounding Estimation Errors: Regression Coefficients}
In this section, our aim is to first obtain an upper bound for
$ \max_{1 \le j \le p } \| \tilde{\alpha}_j - \alpha_{j}^* \|_2,$
which is essential in getting consistency of estimates of rows of the precision matrix. We explain three random terms that will be essential understanding our estimation error bounds.
Let $P_{\hat{B}}$ be the projection matrix onto the range $\hat{B}$, i.e. $P_{\hat{B}}= \hat{B} [ \hat{B}' \hat{B}]^+ \hat{B}' = \hat{B} \hat{B}^+$, and the last equality is by Moore-Penrose matrix inverse rules such as (134) of \cite{bsmw22}. Define the rank of $Y_{-j} P_{\hat{B}}$ as
\begin{equation}\hat{r}_j:= rank (Y_{-j} P_{\hat{B}}).\label{eqn1}
\end{equation}
This term reflects the complexity, and increases the variance of the estimation of rows of the precision matrix, as we show below in Theorem \ref{l1-gl}. Define
\begin{equation}
\hat{\eta}_j:= \sigma_{\hat{r}_j}^2 (Y_{-j} P_{\hat{B}})/n,\label{eqn2}
\end{equation}
which is the $\hat{r}_j$- th largest singular value (squared, scaled by n) of $Y_{-j} P_{\hat{B}}$ ($n \times p-1$ matrix). This last term is obtained when we project $Y_{-j}$ onto range of $\hat{B}$, and will reduce both the bias and variance of estimating the rows of the precision matrix in Theorem \ref{l1-gl} below. It can be considered as size of the signal, and its key to consistency of our estimators.
Next, we define
\begin{equation}
\hat{\Psi}_j:= \sigma_1^2 (Y_{-j} M_{\hat{B}})/n,\label{eqn3}
\end{equation}
where $M_{\hat{B}}:= I_{p-1} - P_{\hat{B}}$, and $Y_{-j} M_{ \hat{B}}$ is $n \times p-1$ matrix. This term is the largest singular value of $Y_{-j} M_{\hat{B}}$ matrix scaled by $n$, and it represents the bias stemming from using a hidden factor structure to estimate $Y_{-j}$. This last term will increase the bias of estimating the rows of the precision matrix.
Define $\max_{1 \le j \le p } \hat{r}_j:= \bar{r}$, $\min_{1 \le j \le p} \hat{\eta}_j:= \bar{\eta}$, and $\max_{1 \le j \le p} \hat{\Psi}_j:= \bar{\Psi}$. We have the following uniform bounds for estimators for off-diagonal terms in each row of the precision matrix, with $\beta, \Gamma$ as positive large constants, and $\bar{r}, \bar{\eta},\bar{\Psi}, \bar{\xi}$ are random variables,
\begin{thm}\label{l1-gl}
(i).Under Assumptions \ref{as1}-\ref{as4}, $K log n \le c n $, with $c>0$ a positive constant, with probability at least $1 - 2/n^{C_p/2}$
\[ \max_{1 \le j \le p} \| \tilde{\alpha}_j - \alpha_{j}^* \|_2^2 \le 2 \gamma_e^2 \Gamma \left( \frac{2 log n + \bar{r}}{n \bar{\eta}}
\right) + \frac{C K }{\bar{\xi}} \left( 1 + \frac{1}{\beta} + \frac{\bar{\Psi}}{\bar{\eta} \beta } + \frac{\Gamma}{\bar{\eta} \beta}
\right).\]
(ii) Under Assumptions \ref{as1}-\ref{as4},
\[ \max_{1 \le j \le p} \| \tilde{\alpha}_j - \alpha_{j}^* \|_2^2 = O_p \left( \max( \frac{K}{\bar{\xi}}, \frac{K \bar{\Psi}}{\bar{\xi} \bar{\eta} }, \frac{K }{\bar{\xi} \bar{\eta}},
\frac{log n+\bar{r}}{n \bar{\eta}}) \right) = O_p (r_{w1}^2).\]
\end{thm}
\begin{remark}\label{rem2}
\begin{itemize}
\item The first term on the right side of Theorem \ref{l1-gl}(i) is the variance term and the second term is the bias term, and there is no sparsity parameter on the upper bounds and rates.
\item Note that $\bar{\xi}$ plays an essential role in getting consistency. Larger signal-to-noise ratio is needed, especially if the number of factors $K$ grow fast. With using example (\ref{ex1}), we see that we cannot have many very weak factors. As an example if the factor loadings are very weak in the following rate $d_{1n}^2 = O(K)$, there will be no consistency.
\item
$\bar{r}$ represents complexity, and affects the $l_2$ norm error adversely through the increase of the estimator's variance. On the other hand, $\bar{\eta}$ is a positive influence, with larger signal improving the performance. It should particularly be noted that this positive influence is reflected through the reduction of both the estimator's bias and variance. $\bar{\Psi}$
has an adverse effect
on $l_2$ error. This term increases the bias of the estimator.
\end{itemize}
\end{remark}
\subsubsection{Adaptive PCR}
In this subsection we consider principal components regression (adaptive PCR from now on) estimators. Let us designate $rank (Y_{-j}) \ge k_j \ge 1$, where $k_j$ is a positive integer and changes with $j=1,\cdots,p$. The adaptive PCR estimators use $\hat{B}= V_{1,k_j}$, where $V_{1,k_j}$ is $p-1 \times k_j$ matrix where $V_{1,k_j}$ represents a matrix which $k_j$ columns (eigenvectors) correspond to the largest $k_j$ eigenvalues of $Y_{-j}' Y_{-j}/n$ for $j=1,\cdots,p$.
The adaptive PCR estimator is, for $j=1,\cdots,p$
\[ \hat{\alpha}_{PCR,j} = V_{1,k_j} (Y_{-j} V_{1, k_j})^+ y_j,\]
where $(.)^+$ is Moore-Penrose inverse of a matrix. Before simplifying the terms in the upper bound in Theorem \ref{l1-gl} we define $\hat{\lambda}_{k_j}:= \frac{1}{n} \sigma_{k_j}^2 (Y_{-j})$, for example with $k_j=1$ we have the largest singular value of $Y_{-j}$, which is $\sigma_{1}^2 (Y_{-j})/n$ (squared-scaled by $n$). One big issue is the choice of $k_j$.
To choose $k_j$ we adapt the so-called elbow method, (for details, see (19) of \cite{bbsmw21})
\begin{equation}
\hat{s}_j:= \max \{ k_j \ge 0: \hat{\lambda}_{k_j} \ge C_0 \Delta_{-j} \},\label{17a}
\end{equation}
with $C_0 >1$ and, $c_{\Delta} >1$ are positive constants above one,
\begin{equation}
\Delta_{-j}:= c_{\Delta} [ \| \Sigma_{U,-j} \|_2 + tr (\Sigma_{U,-j})/n].\label{eq-delta}
\end{equation}
This type of choice is done to minimize the risk of adaptive PCR in \cite{bbsmw21}.
Here we need results uniformly over $j=1,\cdots,p$
and the choice given in (\ref{eq-delta}) and above
is a feasible estimator of $s_j$ after following Remark 3- fifth item to Corollary \ref{cl1}. Set $k_j= \hat{s}_j$.
By definition of $\hat{s}_j$ above
\begin{equation}
\hat{\lambda}_{\hat{s}_{j+1}} < C_0 \Delta_{-j} \le \hat{\lambda}_{\hat{s}_j}.\label{pcl1-4}
\end{equation}
We show in the proof of Corollary \ref{cl1} in Appendix
\begin{equation}
\max_{1 \le j \le p} P ( \hat{s}_j \le K ) \to 1.\label{21}
\end{equation}
which will simplify the upper bound for $l_2$ norm error for our estimators. We prove the following error bounds for adaptive PCR based estimator for the off-diagonal terms in a given row for the precision matrix. Note that three random terms $\bar{r}\le K$, $\bar{\eta}\ge \beta >0$, and $\bar{\Psi}/\bar{\eta}\le 1$ with probability approaching one to obtain Corollary \ref{cl1} from Theorem \ref{l1-gl}.
\begin{corollary}\label{cl1}
Under Assumptions \ref{as1}-\ref{as4}
with $0< c' <1$ a positive constant,
$K log n \le c n$, with $c$ a positive constant,
(i).
\[
\max_{1 \le j \le p}
\| \hat{\alpha}_{PCR,j} - \alpha_{j}^* \|_2^2 \le 2 \gamma_e^2 \Gamma \left[ \frac{2 log n +K}{n \beta}
\right]
+ \frac{C K}{\bar{\xi}} \left[ 1 + \frac{2}{\beta}+ \frac{\Gamma}{\beta^2} \right],
\]
with probability at least
$1 - 2/n^{C_p/2} - exp ((c'-1) n)$.
(ii).
\[ \max_{1 \le j \le p } \| \hat{\alpha}_{PCR,j} - \alpha_{j}^* \|_2^2 = O_p \left( max ( \frac{ log n + K}{n }, \frac{K}{\bar{\xi}})\right).\]
\end{corollary}
Note that noise constant $\Gamma$ play an important role in finite samples by increasing the variance and bias, but $\beta$ has the opposite effect. High dimensional consistency is possible when $p>n$, with $\bar{\xi}$ increasing with $p$
as can be seen in Example in (\ref{ex1}).
To have a feasible estimation with $K$ estimated, we follow (56) of \cite{bbsmw21} or (2.7) of \cite{bw19}. Assuming additionally in Corollary \ref{cl1} that $tr (\Sigma_{U,-j} ) \ge c'' \| \Sigma_{U,-j} \|_2 (n \wedge p)$, with $c''>0$ a positive constant, for each $j=1,\cdots,p$, Proposition 8 of \cite{bbsmw21} can be extended with a simple union bound and the feasible estimation of the unknown number of factors results do not change Corollary \ref{cl1}.
\subsubsection{Ridgeless Regression Estimation}
In this part, we consider RRE when $p-1<n$ first and then analyze $p>n$ case.
In that respect $\hat{B}= I_{p-1}$, which simplifies the upper bound in Theorem \ref{l1-gl}. We have
\[ \hat{\alpha}_{RRE,j} = (Y_{-j})^+ y_j,\]
for $j=1,\cdots,p$. Note that when $p-1<n$, RRE estimation is equivalent to OLS, and by using (132) of \cite{bsmw22},
$\hat{\alpha}_{RRE,j} = (Y_{-j})^+ y_j = (Y_{-j}' Y_{-j})^{-1} Y_{-j}' y_j.$
Three random terms simplify $\bar{r} < p$, $\bar{\Psi}=0$, and $\bar{\eta} \ge C >0$ with probability approaching one to obtain Corollary \ref{cl2} from Theorem \ref{l1-gl}.
\begin{corollary}\label{cl2} (RRE: $p-1<n$)
(i). Under Assumptions \ref{as1}-\ref{as4}, and $K log n < c n$, with $c>0$ a positive constant, with probability at least
$1 - 1/n^{C_p/2}$,
\[
\max_{1 \le j \le p}
\| \hat{\alpha}_{RRE,j} - \alpha_{j}^* \|_2^2 \le 2 \gamma_e^2 \Gamma \left[ \frac{2 log n +p}{C n}
\right]
+ \frac{ C K}{\bar{\xi}} \left[ 1 + \frac{1}{\beta}+ \frac{\Gamma}{C \beta }\right].
\]
(ii). Under the same Assumptions in (i)
\[ \max_{1 \le j \le p } \| \hat{\alpha}_{RRE,j} - \alpha_{j}^* \|_2^2 = O_p ( max ( \frac{ log n + p}{n }, \frac{K}{\bar{\xi}})).\]
\end{corollary}
Now we provide the Corollary for the RRE case when $p-1>n$. First we find it useful to introduce the effective rank as
$ re (\Sigma_{U,-j}) := \frac{tr (\Sigma_{U,-j})}{\| \Sigma_{U,-j} \|_2}.$ Next, we relax Assumption \ref{as3}(i) that relates $p$ to $n$. We show that in RRE with $p-1>n$ case, we can allow $p$ to grow exponentially in $n$. This implies that we can have much larger, in dimension, precision matrix estimation in RRE compared with adaptive PCR.
{\bf Assumption $3^*(i)$.}
{\it We allow
\[ n< p-1 \le \frac{exp(C_p n/2)}{4}.\]}
Now we provide our coefficient estimation result that will be helpful for precision matrix estimation based on RRE. Define $\delta_n:= \min_{1 \le j \le p}
tr (\Sigma_{U,-j})/n$. $\delta_n$ is nondecreasing in n, since $\Sigma_{U,-j}: p-1 \times p-1$ diagonal matrix, with positive elements in the diagonal and $p,n$ grow together under Assumption $3^*(i)$ with $p-1>n$. Note that three random variables in Theorem \ref{l1-gl} simplifies in Corollary \ref{cl3} as $\bar{r}=n$, the bias term is $\bar{\Psi}=0$, and the size of the signal is $\bar{\eta}\ge \delta_n$ wpa1.
\begin{corollary}\label{cl3} (RRE: $p-1>n$)
Under Assumptions \ref{as1}-\ref{as2}, and Assumptions $3^{*}(i)$-\ref{as3}(ii)-\ref{as4} and
$ C >0$, and for all $j=1,\cdots,p$,
$ re(\Sigma_{U,-j}) > C n,$
for $ C$ a positive constant $K log n < c n$, with $c$ a positive constant, and $\tilde{U}_{-j}$ has independent entries,
(i).
\[
\max_{1 \le j \le p}
\| \hat{\alpha}_{RRE,j} - \alpha_{j}^* \|_2^2 \le 2 \gamma_e^2 \Gamma \left[ \frac{2 log n +n}{C n \delta_n }
\right]
+ \frac{C K}{\bar{\xi}} \left[ 1 + \frac{1}{\beta}+ \frac{\Gamma}{\beta \delta_n}\right].
\]
with probability at least
$1 - exp(-C_p n/2)$, and
(ii).
\[ \max_{1 \le j \le p } \| \hat{\alpha}_{RRE,j} - \alpha_{j}^* \|_2^2 = O_p ( max ( \frac{ log n + n}{n \delta_n }, \frac{K}{\bar{\xi}})).\]
\end{corollary}
First, we want to see in the case of $n,p$ grow, whether RRE is consistent or not. With $p$ growing polynomially in $n$, with a power larger than one, RRE is consistent.
Clearly
$\frac{logn + n}{n \delta_n} \to 0,$
since $\delta_n$ may grow larger in $n$.Then $K/\bar{\xi} $ can be smaller if we use the example $(\ref{ex1})$, and $K / d_{1n}^2 \to 0$. So signal-to-noise ratio should grow and also dominate the number of factors as well to have consistency. Note that we may see double descent in RRE for off-diagonal term estimation of the parameter vector $\alpha_{j}^*$ in the precision matrix. We fix $n$ and $K$ and analyze the changes in $p$.
To see that in the previous Corollary we need $p/n \to 0$ to get estimation errors converging to zero. So very small $p$ in OLS (RRE $p-1<n$) case provides estimation error converging to zero. Then in minimum-norm interpolator case here (RRE with $p-1>n$), with $p/n \to \infty$ with $n$ fixed, and noise $\delta_n$ may grow with larger $p$, since it is a scaled version of $p-1 \times p-1$ diagonal matrix trace with all positive elements,
$\frac{logn + n}{n \delta_n} \to 0,$
and $K /\bar{\xi} \to 0$ as discussed above since $ d_{1n}^2$ depend on $p$ and if $d_{1n}^2$ dominates K, we may see estimation errors converging to zero, and hence double descent can be seen. To see double descent Assumption $\ref{as3}^{*}$(i),(ii) and (\ref{ex1}) is helpful.
We compare now RRE in this case with adaptive PCR. In RRE there is one less bias term compared with adaptive PCR ($\bar{\Psi}=0$ in RRE) in finite sample upper bound. Since $K < n$, adaptive PCR has a smaller variance term compared with RRE. Asymptotically bias terms have the same rate $K/\bar{\xi}$.
Adaptive PCR also may have one more theoretical advantage compared with RRE. We need the noise $\delta_n$ to be increasing in $n$ to get consistency in RRE, whereas in adaptive PCR this is not needed. But in financial markets we expect $\delta_n \to \infty$ so this may not be of an issue between two estimators in financial markets.
\subsection{Estimation of Diagonal Precision Matrix Entries}
We have the following Theorem for the reciprocal of the main $j$-th diagonal term in the precision matrix.
The technique in \cite{bbsmw21} considers only simple linear regression, while the precision matrix estimation and especially a diagonal term estimation are not handled. The proof of Theorem \ref{thma} extends the literature on diagonal term estimation of a large precision matrix. The results and the proof are novel and show the rate of convergence and the non-asymptotic upper bound for estimation errors, as well as the consistency of the estimator. Note that $r_{w1}$ rate is defined just before Assumption \ref{as5} in the main text.
\begin{thm}\label{thma}
(i). Under Assumptions \ref{as1}-\ref{as4} \ref{as5}(i),
\begin{eqnarray*}
\max_{1 \le j \le p} | \tilde{\tau}_j^2 - \tau_j^2| & = &
O_p \left(K (\sqrt{\frac{p (log p)}{n}}) r_{w1}\right) \nonumber \\ &+& O_p \left( d_{2n} r_{w1} \right)
+ O_p \left( max (\frac{K^{3/2}}{n^{1/2}}, K \frac{\max(\sqrt{logp}, \sqrt{logn})}{n^{1/2}})
\right).\label{69}
\end{eqnarray*}
(ii). Under Assumptions \ref{as1}-\ref{as5}, \[ \max_{1 \le j \le p} | \tilde{\tau}_j^2 - \tau_j^2| = o_p (1).\]
(iii). Under Assumptions \ref{as1}-\ref{as5}, where $c > 0$ is a positive constant
\[ \min_{1 \le j \le p} \tau_j^2 \ge c - o (1).\]
\end{thm}
\begin{remark}\label{rem-thma}
\begin{itemize}
\item When $p>n$ and $p/n \to (1,\infty)$ it is possible to get consistency since $p$ only enters the estimation errors directly via $\sqrt{log p}$ and with a larger n, the estimation errors will go to zero, if $K \sqrt{log p} \, r_{w1} = o_p (1)$, and $K \sqrt{log p/n} =o_p(1)$.
\item Theorem \ref{l1-gl}(ii) clearly shows that the rate of estimation error for off-diagonal-row elements is $r_{w1}$,
but our Theorem \ref{thma} shows for the reciprocal of the main diagonal term, this is slower than $r_{w1}$ since via Assumption \ref{as5}(iii), we need $ d_{2n} r_{w1}=o_p(1)$ for consistency in addition to other part of Assumption \ref{as5}(iii) as well as Assumption \ref{as5}(ii).
\item Note that Theorem \ref{thma} excludes the case of RRE with $p-1>n$ case, since formula in (\ref{14a}) is equal to zero due to interpolation. We provide an alternative estimator for RRE with $p-1>n$ case in Corollary A.3 in Online Appendix.
\end{itemize}
\end{remark}
\subsection{Consistent Estimation of Rows}
We provide our theorem for estimating rows of the precision matrix. This shows that each row of the precision matrix in general case can be estimated, uniformly over all rows, consistently through our estimator.
Specific results regarding adaptive PCR and RRE will be discussed in subsections \ref{apcr}-\ref{RRE}.
Note that the main diagonal terms rate determine the rate of convergence. The off-diagonal terms estimation is consistent but converges at a faster rate than the main diagonal terms.
This can be seen through comparing the rates in Theorem \ref{l1-gl} and via proof of Theorem \ref{thm1} and Remark \ref{rem-thma} in Theorem \ref{thma}. Denote each row of $\Theta$ as $\Theta_{j}'$, which is a $1 \times p$ vector. As an example, first row can be represented as $\Theta_1':= (\frac{1}{\tau_1^2}, \frac{-\alpha_1^{*'}}{\tau_1^2} )$. Let $\Theta_{j}$ be the column representation of the row $\Theta_{j}'$. Let the row estimator be $\tilde{\Theta}_j'$, and for the first row this is
$\tilde{\Theta}_1^{'} := (\frac{1}{\tilde{\tau}^2}, \frac{- \tilde{\alpha}_1^{'}}{\tilde{\tau}_1^2})$.
\begin{thm}\label{thm1}
Under Assumptions \ref{as1}-\ref{as5},
\begin{eqnarray*}
\max_{1 \le j \le p} \| \tilde{\Theta}_j' - \Theta_{j}' \|_2 &=& O_p ( K \frac{\sqrt{p log p}}{\sqrt{n}} r_{w1}) + O_p ( d_{2n} r_{w1})\\
&+& O_p \left( \max(\frac{K^{3/2}}{n^{1/2}}, K \frac{\sqrt{\max(logp, log n)}}{n^{1/2}})\right)= o_p (1),
\end{eqnarray*}
where $r_{w1}$ is defined in (\ref{eqrw1}).
\end{thm}
\begin{remark}
\begin{itemize}
\item First, we see that if $p/n \to [0,\infty)$, we obtain consistency and the price that we pay in terms of number of assets, $p$ is just $\sqrt{logp}$ at the first glance, as can be seen in Assumption \ref{as5}. However, we have terms such as $\bar{r}, \bar{\Psi}, \bar{\eta}$ which may depend on $p$, and there will be analysis of these terms in adaptive PCR, and RRE cases.
\item We see that through definition of $r_{w1}$ one of the keys to consistent estimation is the ratio $K/\bar{\xi}$, and this ratio needs to go to zero. So we cannot have a large hidden factor system and expect consistency.
\item Note that the rate for Theorem \ref{thm1} is obtained for all linear estimators in Section 3, except RRE when $p-1>n$, the interpolating estimator. This is explained in detail, below in Corollary
\ref{cl6} and Online Appendix Corollary A.3.
\end{itemize}
\end{remark}
\subsubsection{Adaptive PCR Estimator}\label{apcr}
In case of adaptive PCR, the random terms in the upper bound in Theorem \ref{thm1}, can be bounded as: $\bar{r} \le K$ wpa1, $\bar{\Psi}/\bar{\eta} \le1$ wpa1, $\bar{\eta} \ge \beta$ with wpa1. So Theorem \ref{thm1} rate can be simplified in adaptive PCR,
with $\hat{\Theta}_{PCR,j}'$ representing the estimator of $j$ th row of $\Theta$. The following Corollary follows from Theorem \ref{thm1} by replacing $r_{w1}$ with
$r_{w1,PCR}$ rate that is shown before Corollary A.1 in Online Appendix.
\begin{corollary}\label{cl4}
Let $p/n \to [0,\infty)$ and with Assumptions \ref{as1}-\ref{as5},
\begin{eqnarray*}
\max_{1 \le j \le p} \| \hat{\Theta}_{PCR,j}' - \Theta_{j}' \|_2 &=& O_p \left(K \sqrt{logp} \max ( \frac{\sqrt{log n + K}}{\sqrt{n }}, \frac{\sqrt{K}}{\sqrt{\bar{\xi} }})\right)\nonumber \\
&+& O_p \left( d_{2n} \max ( \frac{\sqrt{log n + K}}{\sqrt{n }} , \frac{\sqrt{K}}{\sqrt{\bar{\xi}}})\right)\\
& + &O_p \left( \max(\frac{K^{3/2}}{n^{1/2}}, K \frac{\sqrt{\max(logp, log n)}}{n^{1/2}})\right) = o_p (1) .
\end{eqnarray*}
\end{corollary}
\begin{remark}
\begin{itemize}
\item The price of consistency is $\sqrt{log p}$ by number of assets, even when $p>n$. So with $p/n \to (1,\infty)$, to get consistency we need the signal-to-noise ratio dominate the number of factors.
\item
We analyze how consistency can be achieved in this adaptive PCR. It can be shown that with $p>n$ and with both $p,n$ growing we need signal-to-noise ratio growing and $d_{2n}$ to grow slowly in $n$.
\end{itemize}
\end{remark}
\subsubsection{Ridgeless Estimators}\label{RRE}
When we consider the sub case of RRE with $p<n-1$, we set wpa1 $\bar{r} <p$, $\bar{\Psi}=0$, and $\bar{\eta} \ge c$, where $ c $ is a positive constant in Theorem \ref{thm1} result above. Let the $j$ th row of estimator of the precision matrix be $\hat{\Theta}_{RRE,j}'$. To obtain Corollary \ref{cl5}, we use Corollary A.2 and Corollary \ref{cl2} with $r_{w1}$ in Theorem \ref{thm1} replaced with $r_{RRE-l}$ rate after Corollary A.1 in Online Appendix.
\begin{corollary}\label{cl5}
Set $p/n = a_{2n} \to 0$, and under Assumptions \ref{as1}-\ref{as5},
\begin{eqnarray*}
\max_{1 \le j \le p} \| \hat{\Theta}_{RRE,j}' - \Theta_{j}' \|_2 &=& O_p \left(\max (K \sqrt{a_{2n} logp},1) \max ( \frac{\sqrt{log n + p}}{\sqrt{n }}, \frac{\sqrt{K}}{\sqrt{\bar{\xi}}})\right)\nonumber \\
&+& O_p \left( d_{2n} \max ( \frac{\sqrt{log n + p}}{\sqrt{n }} , \frac{\sqrt{K}}{\sqrt{\bar{\xi}}})\right)\\
& + & O_p \left( \max(\frac{K^{3/2}}{n^{1/2}}, K \frac{\sqrt{logn}}{n^{1/2}})\right) = o_p (1).
\end{eqnarray*}
\end{corollary}
\begin{remark} \label{rem8}
\begin{itemize}
\item Note that unless $p/n \to 0$, RRE will not be consistent.
\end{itemize}
\end{remark}
Next, we consider RRE estimator when $p-1>n$. Note that in this scenario main diagonal term estimation is done differently since RRE is an interpolating estimator. We prove this after Corollary A.2. In the proof of Theorem \ref{thm1}, it is clear that the rate of convergence should be the slower of the main diagonal or off-diagonal terms.
Let the j-th row of
the precision matrix be $\hat{\Theta}_{INT,j} $. Corollary 6 below proves $\hat{\Theta}_{INT,j}$'s consistency by combining Corollaries \ref{cl3} and A.3.
\begin{corollary}\label{cl6}
Under Assumptions \ref{as1}-\ref{as2}, Assumption $3^{*}(i)$ Assumptions \ref{as3}(ii)-\ref{as4}-\ref{as5}(ii), with signal-to-noise condition $ \sqrt{K} d_{2n}/\sqrt{\bar{\xi}} \to 0$,
and $\min_{ 1 \le j \le p} re (\Sigma_{U_{-j}}) > C n$ and $\tilde{U}_{-j}$ matrix with independent entries,
\begin{eqnarray*}
\max_{1 \le j \le p} \| \hat{\Theta}_{INT,j}' - \Theta_{INT,j}'\|_2 &= &O_p \left( \max (\frac{\sqrt{log n + n}}{\sqrt{n \delta_n}}, \frac{\sqrt{K} d_{2n}}{\sqrt{\bar{\xi}}})\right)\\
&+&O_p \left( \max(\frac{K^{3/2}}{n^{1/2}}, K \frac{\sqrt{logp}}{n^{1/2}})\right) = o_p (1).
\end{eqnarray*}
\end{corollary}
\begin{remark}
\begin{itemize}
\item
First of all we still get consistency with $p-1>n$,
as long as signal-to-noise ratio dominates number of factors multiplied by $d_{2n}$. This can be seen easily by $K \sqrt{logp/n} \to 0$,
and also we need signal-to-noise condition: $ \sqrt{K} d_{2n}/\sqrt{\bar{\xi}} \to 0$.
\item It is not clear from the rates of convergence that RRE or adaptive PCR may do better. But it is possible that with low number of factors, adaptive PCR can perform better. But there is no tuning parameter estimation in RRE.
\item Double descent is possible in RRE in precision matrix estimation. Large $p,n$ can result in increasing size of the signal, and signal-to-noise ratio $\delta_n, \bar{\xi}$ when $p>n$ respectively. Specifically when $\delta_n > log p$ we can observe double descent.
With $p<n$, the consistency is achieved with $p/n \to 0$.
This extends the concept of double descent from linear regression to precision matrix estimation. This may have important consequences in empirics. We show one of these findings in Sharpe Ratio in US Stock market.
\end{itemize}
\end{remark}
\section{Empirical Application}\label{empirics}
The objective of the empirical application is to conduct an out-of-sample comparison among estimations. We estimate Maximum Sharpe Ratio (MSR) portfolio using estimators, and use four commonly used financial metrics: the Sharpe Ratio (SR), average return, portfolio turnover, and portfolio variance. We use data from January 1995 to December 2023, with an out-of-sample period from January 2015 to December 2023. We selected all stocks included in the S$\&$P 500 for at least one month during the out-of-sample period, ensuring the data were available for the entire 1995-2023 period, resulting in 357 stocks. The out-of-sample period includes stock market of 2022 when S$\&$P 500 index was down by 18.1 \%.
\subsection{Metrics}
We use a rolling window set up for the out-of-sample estimation of the Sharpe Ratio following \cite{caner2019} and \cite{caner2022}. The sample size \textit{n} is divided into in-sample $(1:n_I)$ and out-of-sample $(n_I +1:n) $. Initially, we estimate the portfolio $\hat w_{n_I}$ in the in-sample period and the out-of-sample portfolio returns $\hat w'_{n_I} y_{n_I+1}$. Then, we roll the window forward by one element $(2:n_I +1 )$, forming a new in-sample portfolio $\hat w _{n_I +1 } $ and calculating out-of-sample portfolio returns $\hat w_{n_{I+1}}'y_{n_I +2}$. This process is repeated until the end of the sample.
The out-of-sample average return and variance without transaction costs are computed as follows:
$
\hat \mu_{os} = \frac{1}{n-n_I} \sum_{t=n_I}^{n-1} \hat w_t'y_{t+1}, \hat \Sigma^2_{y,os} = \frac{1}{n-n_I -1} \sum_{t=n_I}^{n-1} (\hat w_t' y_{t+1} - \hat \mu _{os})^2
$.
We estimate the Sharpe Ratios without transaction costs:
$
SR = \frac{\hat \mu_{os}}{\hat \Sigma_{y,os}}
$
When transaction costs are present, the definition of the metrics is modified. Let $c$ be the transaction cost, which we set 10 basis points. Let $y_{P,t+1} = \hat w_t ' y_{t+1}$ be the return of the portfolio $P$ in period $t+1$; in the presence of transaction costs, the return is defined as
$
y^{Net}_{P,t+1} = y_{P,t+1} - c(1+y_{P,t+1}) \sum_{j=1}^p | \hat w_{j,t+1} - \hat w^+_{j,t}|
$ where $\displaystyle \hat w^+_{j,t} = \frac{\hat w_{j,t} (1+y_{j,t+1})}{1 + y_{P,t+1}}$, with $y_{j,t+1}$ representing the excess return added to risk-free-rate for $j$th, and $y_{P,t+1}$ representing the portfolio $P$ excess return added to the risk-free rate. The adjustment made in $\hat w^+_{j,t}$ is because the portfolio at the end of the period has changed compared to the portfolio at the beginning of the period.
Then, we define the mean and variance of the portfolio returns with transaction costs as: $
\hat \mu_{os-c} = \frac{1}{n-n_I}\sum_{t=n_I}^{n-1} y^{Net}_{P,t} \mbox{ and } \hat \Sigma^2_{y,os-c} = \frac{1}{n-n_I-1} \sum_{t=n_I}^{n-1} (y^{Net}_{P,t} - \hat \mu_{os-c})^2
$. The portfolio returns are replaced by the returns with transaction costs when we calculated the Sharpe Ratio with transaction costs. Finally, portfolio turnover is defined as:
$
\text{turnover} = \sum_{j=1}^p | \hat w_{j,t+1} - \hat w_{j,t}^+|.
$
\subsection{Methods}
We conduct a comprehensive empirical analysis by fixing in-sample $n_I=240$ and considering both low-dimensional and high-dimensional cases across various portfolio sizes. Specifically, we selected 10, and 200 stocks from the pool to estimate the models with the same stocks for all windows in the low-dimensional experiments. For the high-dimensional experiment, we choose 357 (including all available stocks). We also analyzed portfolios with 25, 50, and 300 stocks, however, since these results are similar, they are not included here.
The model comparisons include several approaches from the literature. We consider the nonlinear shrinkage methods of \cite{lw2017}(NL-LW) and the single-factor nonlinear shrinkage (SF-NL-LW), POET method from \cite{fan2013}, and the nodewise (NW) method from \cite{caner2019}, along with the feasible residual-based nodewise approach from \cite{caner2022}. SF-NL-LW results are very similar to NL-LW in terms of standard deviation, so we have not included that in our Table. But these are available from authors on demand.
In particular, RNW-SF, RNW-3F, and RNW-5F refer to residual-based nodewise regressions using a single-factor model (market as the only factor), three-factor model, and five-factor Fama and French model, respectively.
Our new method introduces several variations: RRE, PCR-1F, PCR-3F, PCR-5F, PCR-7F and PCR-Adaptive. These denote the use of a RRE estimator, Principal component with fixed 1, 3, 5, 7 hidden factors, and an adaptive method to select the number of hidden factors, respectively. We use these number of factors since they are used in the empirics in the literature and seven factor is used to show large number of factors.
\subsection{Results}
We report the results for Maximum Sharpe Ratio portfolio, with transaction costs in Tables 1-3, and $NA$ represents "not available" for turnover in our tables with transaction costs. $SD$ represents Standard Deviation of the portfolio, and $Sharpe $ represents Sharpe Ratio of the portfolio respectively in Tables.
Tables without transaction costs are in Appendix and also have turnover results.
We have also done analysis for Global Minimum Variance portfolio, and the results are very similar. Results for the low-dimensional cases are presented in Tables \ref{tab:240-10} - \ref{tab:240-200}, while high-dimensional case is shown in Table \ref{tab:240-357}. In additionally, we use the same data but with different out-of-sample from January 2005 to December 2023 ($n_I = 120$, $n-n_I = 228$). This subsample yields similar results and is available upon request. In MSR portfolio, our PCR-based method is the best model across all 2 small-to-medium (low-dimensional) portfolios scenarios for Sharpe Ratio with transaction cost. Our RRE achieves the highest Sharpe Ratio for the largest size of portfolio with $ p=357$. Let us give an example by examining three portfolios of varying sizes for MSR: a small low-dimensional portfolio, a moderate-sized portfolio, and a large-sized portfolio. In Table \ref{tab:240-10}, for a portfolio of 10 stocks ($p=10,n_I=240$), PCR-1F is the best Sharpe Ratio with 0.2260. In Table \ref{tab:240-200} of 200 stocks ($p=200,n_I =240$), the highest Sharpe Ratio is with our PCR-7F at 0.2682 and the second best is our PCR-3F at 0.2263. Lastly, in Table \ref{tab:240-357}, for the largest portfolio with all 357 available stocks, the highest Sharpe ratio is RRE at 0.2203.
We use \cite{ledoit2008robust} test with circular bootstrap to test the significance of the Sharpe Ratio with transaction cost-winner. Specifically,
\begin{equation}
H_o: SR_{winner} \leq SR_0 \mbox{ vs } H_a : SR_{winner}>SR_0,
\end{equation}
Where $SR_{winner}$ is the method that has the highest Sharpe Ratio in each table, testing against all other methods and significance is set at threshold of $p=0.05$. A $"*"$ indicates statistical significance from the winner.
To see whether a double ascent in the Sharpe Ratio is possible in RRE, we analyze a figure. Figure \ref{fig:sub_2} shows the MSR Sharpe Ratio with transaction cost. In figure, we include the performance of RRE, Nodewise and PCR-3F, PCR-adaptive methods. We observe a double ascent pattern in the RRE estimator's performance in terms of the Sharpe Ratio. For example, with the in-sample window size set at $n_I = 240$, in low-dimensional case, the Sharpe Ratio starts at 0.2065. As the portfolio size increases to 200 stocks, the Sharpe Ratio declines to -0.0471. In the high-dimensional case, where the number of stocks exceeds 240, the SR increases to 0.2203 for a portfolio of 357 stocks. From a practitioner point of view, we observe that we can form a large portfolio with RRE, much larger than the sample size n, without any tuning parameters to fix as in other methods and get a good result.
\begin{figure}
\centering
\includegraphics[width=\textwidth]{figure_4.png}
\caption{MSR Sharpe Ratio vs. p}
\label{fig:sub_2}
\end{figure}
\begin{table}[h!]
\caption{Monthly Portfolio Performance of 10 Stocks, $ n_I=240$, $ n-n_I=108$\label{tab:240-10}}
\begin{center}
\begin{tabular}{rcllll}
\multicolumn{1}{c}{\bfseries Estimator}&\multicolumn{1}{c}{\bfseries }&\multicolumn{4}{c}{\bfseries Maximum Sharpe Ratio}\tabularnewline
\multicolumn{1}{r}{}&\multicolumn{1}{c}{}&\multicolumn{1}{c}{Return}&\multicolumn{1}{c}{SD}&\multicolumn{1}{c}{Sharpe}&\multicolumn{1}{c}{Turnover}\tabularnewline
{\bfseries with transaction cost}&&&&&\tabularnewline
RNW-GIC-SF&&0.0105&0.0526&0.1993& NA\tabularnewline
RNW-GIC-3F&&0.0106&0.0518&0.2040& NA\tabularnewline
RNW-GIC-5F&&0.0107&0.0509&0.2094& NA\tabularnewline
NW-GIC&&0.0110&0.0504&0.2184& NA\tabularnewline
POET&&0.0112&0.0502&0.2235& NA\tabularnewline
Ledoit-Wolf&&0.0105&0.0521&0.2025& NA\tabularnewline\hline
Inter-RRE&&0.0109&0.0528&0.2065& NA\tabularnewline
PCR-1F&&0.0111&0.0491&\textbf{0.2260}& NA\tabularnewline
PCR-3F&&0.0108&0.0493&0.2188& NA\tabularnewline
PCR-5F&&0.0111&0.0500&0.2209& NA\tabularnewline
PCR-7F&&0.0109&0.0507&0.2160& NA\tabularnewline
PCR-Adaptive&&0.0110&0.0491&0.2249$^{*}$& NA\tabularnewline
\end{tabular}\end{center}
\end{table}
\clearpage
\begin{table}[h!]
\caption{Monthly Portfolio Performance of 200 Stocks, $ n_I=240$, $ n-n_I=108$\label{tab:240-200}}
\begin{center}
\begin{tabular}{rllllcllll}
\multicolumn{1}{c}{\bfseries Estimator}&\multicolumn{1}{c}{\bfseries }&\multicolumn{4}{c}{\bfseries Maximum Sharpe Ratio}\tabularnewline
\multicolumn{1}{r}{}&\multicolumn{1}{c}{}&\multicolumn{1}{c}{Return}&\multicolumn{1}{c}{SD}&\multicolumn{1}{c}{Sharpe}&\multicolumn{1}{c}{Turnover}\tabularnewline
{\bfseries with transaction cost}&&&&&\tabularnewline
RNW-GIC-SF&& 0.0056&0.0441& 0.1269& NA\tabularnewline
RNW-GIC-3F&& 0.0069&0.0446& 0.1555& NA\tabularnewline
RNW-GIC-5F&& 0.0075&0.0480& 0.1564& NA\tabularnewline
NW-GIC&& 0.0088&0.0429& 0.2044& NA\tabularnewline
POET&& 0.0059&0.0431& 0.1359& NA\tabularnewline
Ledoit-Wolf&& 0.0047&0.0521& 0.0908& NA\tabularnewline\hline
Inter-RRE&&-0.0055&0.1166&-0.0471$^{*}$& NA\tabularnewline
PCR-1F&& 0.0074&0.0378& 0.1956& NA\tabularnewline
PCR-3F&& 0.0086&0.0379& 0.2263& NA\tabularnewline
PCR-5F&& 0.0069&0.0560& 0.1239& NA\tabularnewline
PCR-7F&& 0.0116&0.0433& \textbf{0.2682}& NA\tabularnewline
PCR-Adaptive&& 0.0074&0.0436& 0.1705& NA\tabularnewline
\end{tabular}\end{center}
\end{table}
\clearpage
\begin{table}[h!]
\caption{Monthly Portfolio Performance of 357 Stocks, $ n_I=240$, $ n-n_I=108$\label{tab:240-357}}
\begin{center}
\begin{tabular}{rllllcllll}
\multicolumn{1}{c}{\bfseries Estimator}&\multicolumn{1}{c}{\bfseries }&\multicolumn{4}{c}{\bfseries Maximum Sharpe Ratio}\tabularnewline
\multicolumn{1}{r}{}&\multicolumn{1}{c}{}&\multicolumn{1}{c}{Return}&\multicolumn{1}{c}{SD}&\multicolumn{1}{c}{Sharpe}&\multicolumn{1}{c}{Turnover}\tabularnewline
{\bfseries with transaction cost}&&&&&\tabularnewline
RNW-GIC-SF&&0.0049&0.0430&0.1136& NA\tabularnewline
RNW-GIC-3F&&0.0059&0.0439&0.1346& NA\tabularnewline
RNW-GIC-5F&&0.0058&0.0458&0.1267& NA\tabularnewline
NW-GIC&&0.0088&0.0423&0.2082& NA\tabularnewline
POET&&0.0059&0.0424&0.1399& NA\tabularnewline
Ledoit-Wolf&&0.0015&0.0530&0.0281& NA\tabularnewline\hline
Inter-RRE&&0.0100&0.0454&\textbf{0.2203}& NA\tabularnewline
PCR-1F&&0.0072&0.0366&0.1961& NA\tabularnewline
PCR-3F&&0.0057&0.0364&0.1567& NA\tabularnewline
PCR-5F&&0.0047&0.0528&0.0893& NA\tabularnewline
PCR-7F&&0.0073&0.0507&0.1436& NA\tabularnewline
PCR-Adaptive&&0.0079&0.0420&0.1872& NA\tabularnewline
\end{tabular}\end{center}
\end{table}
\section{Conclusion}
In this paper, we have introduced a general precision matrix estimator. Our estimator is model-free and does not rely on any sparsity assumptions. Under certain conditions, our estimator exhibits double descent in the estimation error.
Moreover, the Sharpe ratio produced by one of our estimators on the S\&P 500 data exhibits the double ascent, which to the best of our knowledge has never obtained in other empirical studies.\\
\begin{center}
{\bf \Large Appendix}
\end{center}
\setcounter{section}{0}\setcounter{equation}{0}\setcounter{lemma}{0}\setcounter{table}{0}
The Appendix is organized into three sections. Appendix A contains all the proofs from the text. Appendix B provides supporting results related to matrix algebra, including inverse matrix identities and two tail inequalities.
Appendix C provides simulations and extra empirics, and also literature discussion-comparison and hidden factor usage.
\begin{center}
{\bf \Large Appendix A}
\end{center}
.
\setcounter{equation}{0}
\setcounter{lemma}{0}
\setcounter{table}{0}
\setcounter{corollary}{0}
{\bf Proof of Theorem 1}:
Using (4) and (10) we get the following inequality
\begin{equation}
\| \tilde{\alpha}_j - \alpha_{j}^* \|_2^2 \le 3 \| \hat{B} (Y_{-j} \hat{B})^+ u_j \|_2^2 +
3 \| \hat{B} (Y_{-j} \hat{B})^+ F a_j \|_2^2
+ 3 \| \alpha_{j}^* \|_2^2.\label{pl1-gl1}
\end{equation}
The proof contains a detailed transformation of results that can be obtained in linear regression framework to precision matrix
estimation.
Since we ultimately aim to use our results for high dimensional outcomes, we need them to hold uniformly over $j = 1, \dots, p,$. It is important to note that this is in striking contrast with pointwise results (as are those from, e.g., proof of Lemma 17 \cite{bbsmw21} that would be applicable to only one outcome. Each of the right side terms in (\ref{pl1-gl1}) will be analyzed in each of Steps 1-3.
{\bf Step 1.} Start with the first right side term in (\ref{pl1-gl1})
\begin{equation}
\| \hat{B} (Y_{-j} \hat{B})^+ u_j \|_2^2 = u_j' M_j u_j,\label{pl1-gl2}
\end{equation}
with $M_j: n \times n$ matrix defined as
\begin{equation}
M_j:= (Y_{-j } \hat{B})^{+'} \hat{B}' \hat{B} (Y_{-j } \hat{B})^+.\label{pl1-gl2a}
\end{equation}
To simplify the proof, we consider the following trace term:
\[ tr (M_j)
= tr [ (Y_{-j} \hat{B})^{+'} \hat{B}' \hat{B} (Y_{-j} \hat{B})^+].\]
and can be upper bounded
\begin{eqnarray}
tr [ M_j] & \le & rank (M_j) \| M_j \|_2 = rank (Y_{-j} \hat{B})^+ \| M_j \|_2 \nonumber \\
& = & rank (Y_{-j } \hat{B}) \| M_j \|_2 = rank (Y_{-j} P_{\hat{B}}) \| M_j \|_2, = \hat{r}_j \| M_j \|_2 \label{pl1-gl5}
\end{eqnarray}
where the first inequality is due to $M_j$ being positive semidefinite, and $n \times n$ matrix and Exercise 7.49 of \cite{abamag2005}, and the first equality is by Exercise 8.27a and 4.24a of \cite{abamag2005}, and the second equality is by Exercise 10.28 of \cite{abamag2005}, and the third equality is again by Exercises 4.13d, 4.15b, 4.25b, and 10.28 of \cite{abamag2005} with $P_{\hat{B}} = \hat{B} \hat{B}^+$ in p.8 of \cite{bbsmw21}, and the last equality is just the definition of the complexity $\hat{r}_j$. Now we consider, as proved in (44) of \cite{bbsmw21}
\begin{equation}
\| M_j \|_2 \le (n \hat{\eta}_j)^{-1},\label{pl1-gl6}
\end{equation}
where $\hat{\eta}_j:= \hat{\sigma}_{\hat{r}_j}^2 (Y_{-j} P_{\hat{B}})/n$ which is the $\hat{r}_j$ th largest singular value, squared, of the matrix $Y_{-j} P_{\hat{B}}$, and divided by $n$. Define $C_p>0$ a positive constant.
Then by (\ref{pl1-gl5})(\ref{pl1-gl6}) and $\bar{\eta}$, $\bar{r}$ definitions
\begin{equation}
\gamma_e^2 \sigma_{j}^2 [ 2 C_p\| M_j \|_2 log n + tr (M_j)]
\le \gamma_e^2 \Gamma [ \frac{2 C_p log n }{n \bar{\eta}} + \frac{\bar{r}}{n \bar{\eta}}] .\label{pt1-s1}
\end{equation}
Next, define the following events
\begin{equation}
E_{1j}:=\{u_j' M_j u_j \le 2 \gamma_e^2 \sigma_{j}^2 [ 2 C_p \| M_j \|_2 log n + tr (M_j) ]\},\label{pl1-gl3}\end{equation}
and
\begin{equation}
\bar{E}_{1j}:=\{u_j' M_j u_j \le 2 \gamma_e^2 \Gamma [ \frac{2 C_p logn }{n \bar{\eta}}+ \frac{\bar{r}}{n \bar{\eta}}]\}.\label{pt1-s1a}
\end{equation} with
\begin{equation}
\bar{E}_1:= \{\max_{1\le j \le p} u_j' M_j u_j \le 2 \gamma_e^2 \Gamma [ \frac{2 C_p logn }{n \bar{\eta}}+ \frac{\bar{r}}{n \bar{\eta}}]\}.\label{pt1-s2}
\end{equation}
By (\ref{pt1-s1})
\begin{equation}
E_{1j} \subseteq \bar{E}_{1j}.\label{pt1-s1b}
\end{equation}
Clearly, since the upper bounds of the events $ \bar{E}_{1j}, \bar{E}_1$ are the same
\begin{equation}
\cap_{j=1}^p \bar{E}_{1j} = \bar{E}_1.\label{pt1-s1c}
\end{equation}
By (\ref{pt1-s1b})(\ref{pt1-s1c})
\begin{equation}
\cap_{j=1}^p E_{1j} \subseteq \bar{E_1}.\label{pt1-s1d}
\end{equation}
See that by $E_{1j}^c$ as the complement of $E_{1j}$
\begin{equation}
P (\cap_{j=1}^p E_{1j} ) + P (\cup_{j=1}^p E_{1j}^c) =1.\label{pt1-s2a}
\end{equation}
Note that we have, by (\ref{pt1-s1d})
\begin{equation}
P (\bar{E}_1) \ge P (\cap_{j=1}^p E_{1j} )\label{pt1-s1e}
\end{equation}
so by (\ref{pt1-s2a})(\ref{pt1-s1e}) and
\begin{eqnarray}
P (\bar{E}_1) &\ge& 1 - P (\cup_{j=1}^p E_{1j}^c)\nonumber \\
& \ge & 1 - \sum_{j=1}^p P (E_{1j}^c).\label{pt1-s1f}
\end{eqnarray}
Now we calculate the sum of the upper bound probability of $E_{1j}^c$, across $j$.
Note that $u_j$ is a subgaussian vector, and independent of $Y_{-j}$, and $\hat{B}$ by Assumption 1 and the definition of $\hat{B}$, with $M_j$ being a symmetric positive semi-definite matrix.
So $M_j$ is independent of $u_j$.
Then
Lemma \ref{lb1} (with $t=C_p log n $, $H=M_j, \zeta=u_j$, conditional on $M_j$) allows us to write
\begin{equation}
P (E_{1j}^c| M_j) =
P\left( u_j' M_j u_j > 2 \gamma_e^2 \sigma_{j}^2 [ 2 C_p \| M_j \|_2 log n + tr (M_j) ] | M_j\right) \le \frac{1}{n^{C_p}}.\label{pt1-s5}
\end{equation}
Next, use the union bound and by Assumption 3, using $log p \le (\frac{C_p}{2}) log n$
\begin{eqnarray}
\sum_{j=1}^p P (E_{1j}^c) & = & \sum_{j=1}^p P (E_{1j}^c | M_j) P (M_j)
\le \sum_{j=1}^p P (E_{1j}^c | M_j) \nonumber \\
&\le &
exp (logp - C_p logn) \le exp (-\frac{C_p}{2} log n)
= \frac{1}{n^{C_p/2}}
.\label{pt1-s6}
\end{eqnarray}
Use (\ref{pt1-s1f})(\ref{pt1-s6}) in (\ref{pl1-gl2})(\ref{pt1-s2})
\begin{equation}
P \left( \max_{1 \le j \le p} \| \hat{B} (Y_{-j} \hat{B})^+ u_j \|_2^2 \le 2
\gamma_e^2 \Gamma [ \frac{2 C_p logn }{n \bar{\eta}}+ \frac{\bar{r}}{n \bar{\eta}}] \right) \ge 1 - \frac{1}{n^{C_p/2}}.\label{pt1-s7}
\end{equation}
{\bf Step 2.} Now consider the second term on the right side of (\ref{pl1-gl1}). Keeping in mind that, $A_{-j}' (A_{-j}^{+})' = I_K$ and recalling on (5), while setting
$M_{\hat{B}}:=I_{p-1}- P_{\hat{B}}$, we further have
\begin{eqnarray*}
&&\hat{B} (Y_{-j} \hat{B})^+ F = \hat{B} (Y_{-j} \hat{B})^+ F A_{-j}' A_{-j}^{+'} = \hat{B} (Y_{-j} \hat{B})^+ (Y_{-j} - U_{-j}) A_{-j}^{+'} \\
& = & \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} P_{\hat{B}} A_{-j}^{+'} + \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} M_{\hat{B}} A_{-j}^{+'}
- \hat{B} (Y_{-j} \hat{B})^+ U_{-j} A_{j}^{+'}
\end{eqnarray*}
Use simple inequality $(a+b+c)^2 \le 3a^2 + 3b^2 + 3c^2$,
\begin{eqnarray}
& & \| \hat{B} (Y_{-j} \hat{B})^+ F a_j \|_2^2 \le 3 \| \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} P_{\hat{B}} A_{-j}^{+'} a_j \|_2^2 \nonumber \\
& + & 3 \| \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} (I_{p-1} - P_{\hat{B}}) A_{-j}^{+'} a_j \|_2^2
+ 3 \| \hat{B} (Y_{-j} \hat{B})^+ U_{-j} A_{j}^{+'} a_j \|_2^2.\label{pl1-gl7a}
\end{eqnarray}
In Steps 2a, 2b, and 2c below, we separately analyze each of the three terms on the right side of (\ref{pl1-gl7a}).
{\bf Step 2a.} So
\begin{equation}
\| \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} P_{\hat{B}} A_{-j}^{+'} a_j \|_2^2 \le \| \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} P_{\hat{B}}\|_2^2 \| A_{-j}^{+'} a_j \|_2^2.\label{pl1-gl8}
\end{equation}
Note that we can write the first term on the right side of (\ref{pl1-gl8}) as
\begin{equation}
\| \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} P_{\hat{B}}\|_2 = \| \hat{B} (\hat{B}' Y_{-j}' Y_{-j} \hat{B})^+ \hat{B}' Y_{-j}' Y_{-j} P_{\hat{B}} \|_2, \label{pl1-gl9}
\end{equation}
where we use Moore-Penrose inverse rule (134) in \cite{bsmw22} which is for a rectangular matrix $S$, we have $(S'S)^+ S' = S^+$, which we input
$S= Y_{-j} \hat{B}$ in our case above.
To simplify the expressions see that by the proof of Lemma 15 of \cite{bbsmw21} $P_{\hat{B}}= V_1 V_1',$ where $\hat{B}= V_1 D V_2'$ is a singular value decomposition.
$V_1,V_2$ are orthonormal matrices and with dimensions of $p-1 \times r_0, q \times r_0$ respectively, and $D$ is a $r_0$ dimensional diagonal matrix, and the rank of $(\hat{B}) = r_0$.
Rewrite right side of (\ref{pl1-gl9}) as
\begin{equation}
\| \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} P_{\hat{B}}\|_2 = \| V_1 D V_2' (V_2 D V_1' Y_{-j}' Y_{-j} V_1 D V_2')^+ V_2D V_1' Y_{-j}' Y_{-j} V_1 V_1' \|_2.\label{pl1-gl10} \end{equation}
Then in (\ref{pl1-gl10}) right side see that
\begin{equation}
(V_2 D V_1' Y_{-j}' Y_{-j} V_1 D V_2')^+ = V_2 D^{-1} (V_1' Y_{-j}' Y_{-j} V_1)^+ D^{-1} V_2',\label{pl1-gl11}
\end{equation}
which is shown in the proof of Lemma 15 of \cite{bbsmw21}. Then use (\ref{pl1-gl11}) in (\ref{pl1-gl10}), $V_2'V_2=I_{r_0}$
\begin{eqnarray*}
V_1 D V_2' (V_2 D V_1' Y_{-j}' Y_{-j} V_1 D V_2')^+ V_2 D V_1' & = & V_1 D V_2' V_2 D^{-1} [V_1' Y_{-j}' Y_{-j} V_1]^+ D^{-1} V_2' V_2 D V_1' \\
& = & V_1 [ V_1' Y_{-j}' Y_{-j} V_1]^+ V_1'
\end{eqnarray*}
Use the last expression in (\ref{pl1-gl10}) right side
\[
\| V_1 D V_2' (V_2 D V_1' Y_{-j}' Y_{-j} V_1 D V_2')^+ V_2DV_1' Y_{-j}' Y_{-j} V_1 V_1' \|_2 = \| V_1 [V_1' Y_{-j}' Y_{-j} V_1]^+ V_1' Y_{-j} Y_{-j}' V_1 V_1' \|_2.\]
Then clearly since $V_1: p-1 \times r_0$ matrix, and Exercise 10.26 of \cite{abamag2005} shows $[V_1' Y_{-j}' Y_{-j} V_1]^+ V_1' Y_{-j}' Y_{-j} V_1 $ is a symmetric idempotent matrix and all idempotent matrices eigenvalues are either 0 or 1, and since $V_1' V_1 =I_{r_0}$
\[ \| V_1 [V_1' Y_{-j}' Y_{-j} V_1]^+ V_1' Y_{-j}' Y_{-j} V_1 V_1' \|_2 \le \| V_1\|_2 \| [V_1' Y_{-j}' Y_{-j} V_1]^+ V_1' Y_{-j}' Y_{-j} V_1 \|_2 \| V_1' \|_2 \le 1.\]
So use this last inequality in (\ref{pl1-gl10})
\begin{equation}
\| \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} P_{\hat{B}}\|_2 \le 1,\label{pl1-gl12}
\end{equation}
Now consider the second term on the right side of (\ref{pl1-gl8}),
$ \| A_{-j}^{+'} a_j \|_2^2 = a_j' A_{-j}^+ A_{-j}^{+'} a_j = a_j' (A_{-j}' A_{-j})^{-1} a_j,$
since $A_{-j}^{+}= (A_{-j}' A_{-j})^{-1} A_{-j}^{'}$ with $p-1 > K$. Next
\begin{eqnarray}
a_j' A_{-j}^+ (A_{-j}^+)' a_j & = & a_j' \Sigma_f^{1/2} (\Sigma_f^{1/2} A_{-j}' A_{-j} \Sigma_f^{1/2})^{-1} \Sigma_f^{1/2} a_j \nonumber \\
& \le & \frac{a_j' \Sigma_f a_j}{\lambda_K (A_{-j} \Sigma_f A_{-j}')},\label{31a}
\end{eqnarray}
where we use first Rayleigh inequality on p.182 of \cite{abamag2005} and then Theorem 2.6.3b of \cite{hj2013}, where $\lambda_K(.)$ is the $K$ th largest eigenvalue for the matrix
$A_{-j} \Sigma_f A_{-j}': p-1 \times p-1$. By (6)(7), we try to rewrite (\ref{31a}) right-side.
\begin{eqnarray}
\lambda_K (A_{-j}' \Sigma_f A_{-j}) & = &\left[ \frac{\lambda_K (A_{-j}' \Sigma_f A_{-j})}{\| \Sigma_{U,-j} \|_2}\right] \| \Sigma_{U,-j} \|_2 \nonumber \\
& = & \xi_j \| \Sigma_{U,-j} \|_2 \ge \bar{\xi} \| \Sigma_{U,-j} \|_2 \nonumber \\
& \ge & \bar{\xi} \beta,\label{45a}
\end{eqnarray}
where we use $\beta:= \min_{1 \le j \le p} \| \Sigma_{U,-j} \|_2 >0$ by Assumption 2(ii).
Then by Assumption 4(iii)
\begin{equation}
a_j' \Sigma_f a_j \le \| a_j \|_2^2 Eigmax (\Sigma_f) \le C K,\label{46ab}
\end{equation}
by $Eigmax (\Sigma_f) \le C < \infty$ Assumption 3. So the first term on the right side of (\ref{31a}) can be bounded as as
\begin{equation}
\max_{1 \le j \le p} \| A_{-j}^{+'} a_j \|_2^2 =\max_{1 \le j \le p} \left[\frac{ (a_j' \Sigma_f a_j)}{\lambda_K (A_{-j} \Sigma_f A_{-j}')} \right]\le \frac{ C K}{\bar{\xi} \beta}.\label{pl1-gl13}
\end{equation}
Clearly by (\ref{pl1-gl12})(\ref{pl1-gl13}) in (\ref{pl1-gl8})
\begin{equation}
\max_{1 \le j \le p} \| \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} P_{\hat{B}} A_{-j}^{+'} a_j \|_2^2 \le \frac{ C K}{\bar{\xi} \beta}.\label{pl1-gl13a}
\end{equation}
{\bf Step 2b.} Take the second term on the right side of (\ref{pl1-gl7a})
\begin{equation}
\| \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} (I_{p-1} - P_{\hat{B}}) A_{-j}^{+'} a_j \|_2^2 \le \| \hat{B} (Y_{-j} \hat{B})^+ \|_2^2
\| Y_{-j} (I_{p-1} - P_{\hat{B}}) A_{-j}^{+'} a_j \|_2^2.\label{pl1-gl14}
\end{equation}
See that the first term on the right side in (\ref{pl1-gl14}) by $\| S_b \|_2^2 = \| S_b'S_b \|_2$ for a rectangular matrix $S_b:=\hat{B} (Y_{-j} \hat{B})^+$ and $M_j$ definition in (\ref{pl1-gl2a})
\begin{equation}
\| \hat{B} (Y_{-j} \hat{B})^+ \|_2^2 = \| M_j \|_2 \le \frac{1}{n \hat{\eta}_j},\label{pl1-gl15}
\end{equation}
where the last inequality is by (\ref{pl1-gl6}). Then by definition of $\hat{\Psi}_j$ in (16)
\begin{equation}
\| Y_{-j} (I_{p-1} - P_{\hat{B}}) \|_2^2 = \sigma_1^2 [(Y_{-j} (I_{p-1} - P_{\hat{B}})] = n \hat{\Psi}_j.\label{pl1-gl16}
\end{equation}
Now consider (\ref{pl1-gl13})(\ref{pl1-gl15})(\ref{pl1-gl16}) on the second term-right side of (\ref{pl1-gl7a}), with
$ \bar{\Psi}:= \max_{1 \le j \le p} \hat{\Psi}_j, \bar{\eta}:=\min_{1 \le j \le p} \hat{\eta}_j,$
with $\hat{\eta}_j$ defined in (15)
\begin{equation}
\max_{1 \le j \le p}
\| \hat{B} (Y_{-j} \hat{B})^+ Y_{-j} (I_{p-1} - P_{\hat{B}}) A_{-j}^{+'} a_j \|_2^2 \le C \left(\frac{\bar{\Psi}}{\bar{\eta}}\right) \left( \frac{K}{\bar{\xi} \beta}
\right).\label{pl1-gl17}
\end{equation}
{\bf Step 2c.} Consider the third term on the right side of (\ref{pl1-gl7a})
\begin{equation}
\| \hat{B} (Y_{-j} \hat{B})^+ U_{-j} A_{j}^{+'} a_j \|_2^2 \le \| \hat{B} (Y_{-j} \hat{B})^+ \|_2^2 \| U_{-j} A_{j}^{+'} a_j \|_2^2,\label{pl1-gl18}
\end{equation}
where we analyze the following on the right side of (\ref{pl1-gl18}): $
\| U_{-j} A_{j}^{+'} a_j \|_2^2.$ Define $\gamma_{e,j}^2 = \gamma_w^2 a_j' A_{j}^+ \Sigma_{U,-j} A_{j}^{+'} a_j$. Now we simplify $\gamma_{e,j}^2$ term. See that, uniformly over $j$, by Assumption 2 and (\ref{pl1-gl13})
\begin{eqnarray}
\max_{1 \le j \le p} \gamma_w^2 a_j' A_{j}^+ \Sigma_{U,-j} A_{j}^{+'} a_j & \le & [max_{1 \le j \le p} Eigmax (\Sigma_{U,-j})] [\max_{1 \le j \le p}
\gamma_w^2 a_j' A_{j}^{+} A_{j}^{+'} a_j] \nonumber \\
& \le & \frac{\gamma_w^2 C \Gamma K }{\bar{\xi} \beta}.\label{pl1-gl20}
\end{eqnarray}
Then define the events
\begin{equation}
E_{2j}:= \{ \|U_{-j} A_{-j}^{+'} a_j \|_2^2 \le (n + 2 C_p logn + 2 \sqrt{ C_p n log n} )\gamma_{e,j}^2 \}\label{pt1-s20}
\end{equation}
and
\begin{equation}
\bar{E}_{2j}:= \{ \|U_{-j} A_{-j}^{+'} a_j \|_2^2 \le (n+ 2 C_p log n +
2 \sqrt{ C_p n logn }) \frac{C \gamma_w^2 \Gamma K }{\bar{\xi} \beta} \}.\label{pt1-s21}
\end{equation}
By (\ref{pl1-gl20}) and the explanations above that equation, with (\ref{pt1-s20})(\ref{pt1-s21})
\begin{equation}
E_{2j} \subseteq \bar{E}_{2j}.\label{pt2c-1}
\end{equation}
Next, define
\begin{equation}
\bar{E}_2:= \{ \max_{1 \le j \le p} \|U_{-j} A_{-j}^{+'} a_j \|_2^2 \le (n+ 2 C_p log n +
2 \sqrt{ C_p n logn }) \frac{C \gamma_w^2 \Gamma K }{\bar{\xi} \beta} \}.\label{pt2c-2}
\end{equation}
See that by definitions in (\ref{pt1-s21})(\ref{pt2c-2})
\begin{equation}
\cap_{j=1}^p \bar{E}_{2j} = \bar{E}_2.\label{pt2c-3}
\end{equation}
Then by (\ref{pt2c-1})(\ref{pt2c-3})
\begin{equation}
\cap_{j=1}^p E_{2j} \subseteq \bar{E}_2.\label{pt2c-4}
\end{equation}
Then by (\ref{pt2c-4})
\begin{equation}
P (\bar{E}_2) \ge
P ( \cap_{j=1}^p E_{2j})
= 1 - P ( \cup E_{2j}^c)
\ge 1 - \sum_{j=1}^p P (E_{2j}^c).\label{pt1-s22}
\end{equation}
Since $U_{-j} A_{-j}^{+'} a_j$ is $\gamma_w \sqrt{a_j' A_{-j}^+ \Sigma_{U,-j} A_{-j}^{+'} a_j}$ subgaussian vector for each $j=1,\cdots,p$ via Assumption 4, apply Lemma \ref{lb1}
with $\epsilon_j:= U_{-j} A_{j}^{+'} a_j$ and $H:=I_n$ and $\gamma_{e,j}^2 = \gamma_w^2 a_j' A_{j}^+ \Sigma_{U,-j} A_{j}^{+'} a_j$ with $t=C_p logn $
\begin{equation}
P [ \| U_{-j} A_{-j}^{+'} a_j \|_2^2 > (n + 2 C_p logn + 2 \sqrt{C_p n log n} )\gamma_{e,j}^2] \le 1/n^{C_p}.\label{pl1-gl19}
\end{equation}
Then take the union bound and with Assumption 3, $log p \le \frac{C_p}{2} logn$
\begin{equation}
\sum_{j=1}^p P (E_{2j}^c) \le exp (log p - C_p log n)
\le exp ( \frac{-C_p}{2} log n) = \frac{1}{n^{C_p/2}}.\label{pt1-s23}
\end{equation}
Next use (\ref{pt1-s22})(\ref{pt1-s23})(\ref{pt2c-2})
\begin{equation}
P ( \max_{1 \le j \le p} \| U_{-j} A_{-j}^{+'} a_j \|_2^2 \le
(n+ 2 C_p log n +
2 \sqrt{ C_p n logn }) \frac{C \gamma_w^2 \Gamma K }{\bar{\xi} \beta} ) \ge 1 - \frac{1}{n^{C_p/2}}.\label{pt1-s24}
\end{equation}
Use (\ref{pl1-gl15}) and (\ref{pt1-s24}) in (\ref{pl1-gl18}) with probability at least $1-1/n^{C_p/2}$ (note that we ignore the smaller order terms than the right side term in (\ref{pl1-gl20}) since they play no role in the subsequent analysis), while incorporating $\gamma_w^2$ into C,
\begin{equation}
\max_{1 \le j \le p}
\| \hat{B} (Y_{-j} \hat{B})^+ U_{-j} A_{j}^{+'} a_j \|_2^2 \le \frac{C K \Gamma }{ \bar{\eta} \bar{\xi} \beta}.\label{pl1-gl21}
\end{equation}
Combine all Steps 2a-2c results (\ref{pl1-gl13a})(\ref{pl1-gl17})(\ref{pl1-gl21}) in (\ref{pl1-gl7a}) to have, with probability at least $1-1/n^{C_p/2}$, absorbing 3 into the constant $C$,
\begin{equation}
\max_{1 \le j \le p}
\| \hat{B} (Y_{-j} \hat{B})^+ F a_j \|_2^2 \le \frac{CK }{\bar{\xi} \beta} + \frac{ \bar{\Psi} K }{\bar{\eta} \bar{\xi} \beta} + \frac{C K \Gamma }{ \bar{\eta} \bar{\xi} \beta}.\label{pl1-gl22}
\end{equation}
{\bf Step 2d}. A combination of (\ref{pt1-s7}) (\ref{pl1-gl22}) provides the following upper bound
for the first two right side terms on (\ref{pl1-gl1}), with probability at least $1-2/n^{C_p/2}$
\begin{eqnarray}
\max_{1 \le j \le p}
\| \hat{B} (Y_{-j} \hat{B})^+ u_j \|_2^2 &+& \| \hat{B} (Y_{-j} \hat{B})^+ F a_j \|_2^2 \le 2 \gamma_e^2 \Gamma [ \frac{2 log n }{n \bar{\eta}} + \frac{\bar{r}}{n \bar{\eta}}] \nonumber \\
&
+&\frac{CK }{\bar{\xi} \beta} + \frac{ \bar{\Psi} K }{\bar{\eta} \bar{\xi} \beta} + \frac{ C K \Gamma }{ \bar{\eta}\bar{\xi} \beta}.\label{pl1-gl23}
\end{eqnarray}
{\bf Step 3.} After setting $\bar{a}_j:= \Sigma_f^{1/2} a_j, \bar{A}_{-j}:= A_{-j} \Sigma_f^{1/2}$, $\bar{G}_j:= I_K + \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}$ definitions, we upper bound the last term on the right side of (\ref{pl1-gl1}) via Lemma 1 in the following way
\begin{eqnarray*}
\| \alpha_j^*\|_2^2& = & \| \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{a}_j \|_2^2
= \bar{a}_j' \bar{G}_j^{-1} \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{a}_j \\
& \le & \frac{\bar{a}_j' \bar{G}_j^{-1} \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{a}_j}{Eigmin (\Sigma_{U,-j})}
= \frac{1}{Eigmin(\Sigma_{U,-j})} \bar{a}_j' \bar{G}_j^{-1} ( \bar{G}_j - I_K) \bar{G}_j^{-1} \bar{a}_j,
\end{eqnarray*}
Then
\begin{eqnarray}
\| \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{a}_j \|_2^2 & \le & \frac{1}{Eigmin(\Sigma_{U,-j})} [ \bar{a}_j' \bar{G}_j^{-1} \bar{a}_j - \bar{a}_j' \bar{G}_j^{-1} \bar{G}_j^{-1} \bar{a}_j] \nonumber \\
&\le & \frac{1}{Eigmin(\Sigma_{U,-j})} \bar{a}_j' \bar{G}_j^{-1} \bar{a}_j.\label{40a}
\end{eqnarray}
Next,
$ \bar{a}_j' \bar{G}_j^{-1} \bar{a}_j = \bar{a}_j' [ I_K + \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}]^{-1} \bar{a}_j.$
Define $H_j:= \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}$, and $H_j$ is invertible, with rank K
\begin{eqnarray}
\bar{a}_j' (I_K + H_j)^{-1} \bar{a}_j& = & \bar{a}_j' H_j^{-1/2} [ I_K + H_j^{-1} ]^{-1} H_j^{-1/2} \bar{a}_j \nonumber \\
& \le & \frac{\bar{a}_j' H_j^{-1} \bar{a}_j}{Eigmin(I_K + H_j^{-1})} \le \frac{\bar{a}_j' H_j^{-1} \bar{a}_j}{1+ Eigmin(H_j^{-1})} \nonumber \\
&\le& \bar{a}_j' H_j^{-1} \bar{a}_j = \bar{a}_j' [ \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}]^{-1} \bar{a}_j,\label{41a}
\end{eqnarray}
where we use Rayleigh inequality for the first inequality, and for the last equality we use $H_j$ definition. Next use $\bar{a}_j:= \Sigma_f^{1/2} a_j, \bar{A}_{-j}:= A_{-j} \Sigma_f^{1/2}$ definitions
$ \bar{a}_j' [ \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}]^{-1} \bar{a}_j = a_j' (A_{-j}' \Sigma_{U,-j}^{-1} A_{-j})^{-1} a_j.$
Then using the inequality after proof of (63) in \cite{bsmw22} at the end of Section A.2.3 of \cite{bsmw22}
\begin{equation}
\bar{a}_j' [ \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}]^{-1} \bar{a}_j \le \| \Sigma_{U,-j} \|_2 a_j' (A_{-j}' A_{-j})^{-1}a_j.\label{42a}
\end{equation}
Combine (\ref{41a})(\ref{42a}) in (\ref{40a}) to have, by (\ref{31a})
\begin{equation}
\| \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{a}_j \|_2^2 \le \left[\frac{\| \Sigma_{U,-j} \|_2}{Eigmin(\Sigma_{U,-j})}\right] \left[ \frac{ a_j' \Sigma_f a_j}{\lambda_K (A_{-j} \Sigma_f A_{-j}')}\right].\label{43}
\end{equation}
Next,
\begin{eqnarray}
\left[\frac{\| \Sigma_{U,-j} \|_2}{Eigmin(\Sigma_{U,-j})}\right]
\left[ \frac{ a_j' \Sigma_f a_j}{\lambda_K (A_{-j} \Sigma_f A_{-j}')}\right]
& \le & \left[\frac{\| \Sigma_{U,-j} \|_2}{Eigmin(\Sigma_{U,-j})}\right]
\left[ \frac{C K}{\xi_j \| \Sigma_{U,-j} \|_2}
\right] \nonumber \\
& = & \frac{C K}{Eigmin (\Sigma_{U,-j}) \xi_j} \le \frac{C K}{ \bar{\xi}},\label{46a}
\end{eqnarray}
where we use $\min_{ 1 \le j \le p} Eigmin (\Sigma_{U,-j}) \ge c > 0$ Assumption 2(i) for the last inequality as well as $\bar{\xi}$ definition, and second equality in (\ref{45a}), with (\ref{46ab}). Now we combine Steps 1-3 to have with probability at least $1-2/n^{C_p/2}$ to have the desired result.(ii). Asymptotics follows from (i).
{\bf Q.E.D.}
{\bf Proof of Corollary 1}. (i). We need to simplify three terms on the upper bound in Theorem 1 are $\bar{r}, \bar{\eta}, \bar{\Psi}$. Note that
$\bar{r}:= \max_{1 \le j \le p} \hat{r}_j$. In that respect,
\begin{equation}
\hat{r}_j = rank( Y_{-j} P_{V_{1, k_j}}) = rank (Y_{-j} V_{1, k_j}) = k_j,\label{pcl1-1}
\end{equation}
where the first equality is definition and the second equality is by (\ref{pl1-gl5}), and the third equality is
the proof of Corollary 5 in \cite{bbsmw21}. Then the term
\begin{equation}
\hat{\eta}_j= \frac{1}{n} \sigma_{k_j}^2 (Y_{-j} P_{V_{1,k_j}}) = \hat{\lambda}_{k_j},\label{pcl1-2}
\end{equation}
where the first equality is definition and the second equality is by the proof of Corollary 5 in \cite{bbsmw21}, and note that
$\hat{\lambda}_{k_j} = \frac{1}{n} \sigma_{k_j}^2 (Y_{-j})$ by definition. Then
\begin{equation}
\hat{\Psi}_j = \frac{\sigma_1^2 (Y_{-j} ( I_{p-1} - P_{V_{1,k_j}}) )}{n} = \hat{\lambda}_{k_j+1},\label{pcl1-3}
\end{equation}
where the first equality is definition, and the second one is by the proof of Corollary 5 in \cite{bbsmw21}, and with $\hat{\lambda}_{k_j+1}= \frac{1}{n} \sigma_{k_j+1}^2 (Y_{-j})$. We need to prove (20) to proceed further. First we define the event
\begin{equation}
E_{3j}:=\{ \sigma_{K+1}^2 (Y_{-j})/n < C_0 \Delta_{-j}\},\label{defev1}
\end{equation}
and its complement is $E_{3j}^c$. We start by proving
\begin{equation}
P [ \cap_{j=1}^p E_{3j}] \to 1.\label{pcl1-6}
\end{equation}
In that respect, by using Weyl's inequality
\begin{equation}
\sigma_{K+1} (Y_{-j}) \le \sigma_{K+1} (F A_{-j}') + \sigma_1 (U_{-j}) = \sigma_1 (U_{-j}),\label{pcl1-7}
\end{equation}
since $\sigma_{K+1} (F A_{-j}') =0$ by $F A_{-j}'$ having rank of $K$. See that
$\sigma_1^2 (U_{-j})/n= \| U_{-j}' U_{-j} \|_2/n= \| U_{-j} U_{-j}' \|_2/n$ by singular value definition. We provide a definition of $\tilde{U}_{-j}$, which is an $n \times p-1$ matrix, $U_{-j}$ also based on Assumption 4, with $\tilde{U}_{-jt}': 1 \times p-1$ $t$ th row of $\tilde{U}_{-j}$, and we stack these $n$ rows to get $\tilde{U}_{-j}$ and then form $U_{-j}$
$ U_{-j}:= \tilde{U}_{-j} \Sigma_{U,-j}^{1/2}.$
Then by Lemma 22 of \cite{bbsmw21} via Assumption 4, and definition (18)
\begin{equation}
P [ \frac{1}{n} \| \tilde{U}_{-j} \Sigma_{U,-j} \tilde{U}_{-j}'\|_2 \le C_0 \Delta_{-j} ] \ge 1 - exp (-n).\label{pcl1-8}
\end{equation}
Define
$ G_{1j}:= \{\frac{1}{n} \| \tilde{U}_{-j} \Sigma_{U,-j} \tilde{U}_{-j}'\|_2 \le C_0 \Delta_{-j}\}.$
Also see that by (\ref{pcl1-7})
\begin{equation}
G_{1j} \subseteq E_{3j},\label{pcl1-8a}
\end{equation}
for all $j=1,\cdots,p$.
By (\ref{pcl1-8})(\ref{pcl1-8a})
\begin{equation}
P (E_{3j})= P ( \frac{1}{n} \sigma_{K+1}^2 (Y_{-j}) < C_0 \Delta_{-j}) \ge P (G_{1j}) \ge 1 - exp (-n).\label{pcl1-9}
\end{equation}
So clearly
$ P( E_{3j}^c) \le exp(-n).$
To complete the proof of (\ref{pcl1-6})
\[P ( \cap_{j=1}^p E_{3j})
= 1 - P ( (\cap_{j=1}^p E_{3j})^c) = 1 - P ( \cup_{j=1}^p E_{3j}^c)
\ge 1 - \sum_{j=1}^p P (E_{3j}^c).
\]
Next, the union bound argument
\begin{equation}
P ( \cup_{j=1}^p E_{3j}^c) \le \sum_{j=1}^p P (E_{3j}^c) \le exp (log p -n) \le exp[ (c'-1) n],\label{pcl1-10}
\end{equation}
given Assumption 3, $log p \le c' n$ with $c'<1$ a positive constant upper bounded by one. So this clearly shows
\begin{equation}
P ( \cap_{j=1}^p E_{3j}) \ge 1 - exp ((c'-1)n).\label{a42a}
\end{equation}
Then by (17)(19)(\ref{a42a}), we have $P ( \max_{1 \le j \le p} \hat{s}_j < K+1) \ge 1 -exp((c'-1)n)$ hence
\begin{equation}
P ( \max_{1 \le j \le p} \hat{s}_j \le K) \ge 1 - (exp(c'-1)n),\label{cl1-s1}
\end{equation}
with $\hat{s}_j$ being a positive integer. By (\ref{pcl1-1}), $\hat{r}_j = k_j$, and since by $\hat{s}_j$ definition
in (17) we have $k_j = \hat{s}_j$, so
\begin{equation}
\hat{r}_j = \hat{s}_j.\label{cl1-s2}
\end{equation}
Then using (\ref{cl1-s1}) with (\ref{cl1-s2})
\begin{equation}
P ( \max_{1 \le j \le p} \hat{r}_j \le K)
\ge 1 - exp ((c'-1)n).\label{cl1-s3}
\end{equation}
Now we consider $\hat{\Psi}_j$ uniformly in $j$ (in probability).
The uniformity proof has two parts. First, define for each $j=1,\cdots,p$
\begin{equation}
G_{2j}:= \{ \hat{\lambda}_{\hat{s}_j+1} < C_0 \Delta_{-j}\}.\label{p.0}
\end{equation}
Then first by $\hat{\Psi}_j$ definition in (\ref{pcl1-3}), and since $\hat{s}_j= \hat{k}_j$ by $\hat{s}_j$ definition with (\ref{p.0})
\begin{equation}
P ( \cap_{j=1} \{ \hat{\Psi}_j < C_0 \Delta_{-j} \} ) =
P ( \cap_{j=1}^p \{ \hat{\lambda}_{k_j +1} < C_0 \Delta_{-j} \})
= P (\cap_{j=1}^p G_{2j}).\label{p.00}
\end{equation}
The first part of the uniformity proof is done. For the second part, note that by (19) and (\ref{cl1-s1}) $\max_{1 \le j \le p} s_j \le K $ with probability approaching one
\begin{equation}
\hat{\lambda}_{s_j + 1} \ge \hat{\lambda}_{K+1}.\label{p.1}
\end{equation}
By (\ref{defev1})(\ref{p.0})(\ref{p.1}), and $\hat{\lambda}_{K+1}:= \frac{1}{n} \sigma_{K+1}^2 (Y_{-j})$ with probability approaching one
\begin{equation}
G_{2j} \subseteq E_{3j}.\label{p.2}
\end{equation}
So if we prove $P (\cap_{j=1}^p E_{3j}) \to 1$, we will have $P (\cap_{j=1}^p G_{2j}) \to 1$. We have by (\ref{a42a})
$P (\cap_{j=1}^p E_{3j} ) \ge 1 - exp ((c'-1)n).$
That implies
\begin{equation}
P (\cap_{j=1}^p G_{2j} ) \ge 1 - exp ((c'-1)n).\label{p.3}
\end{equation}
So we proved uniformity across $j=1,\cdots, p$ for $\hat{\Psi}_j$ combining (\ref{p.00}) (\ref{p.3}). Now we consider $\hat{\eta}_j$ uniformly in $j$ (in probability).
\begin{equation}
P (\cap_{j=1}^p \{ \hat{\eta}_j \ge C_0 \Delta_{-j}\})
= P ( \cap_{j=1}^p \{ \hat{\lambda}_{\hat{s}_j} \ge C_0 \Delta_{-j}\}),\label{p.4}
\end{equation}
since $\hat{s}_j = k_j$ and using $\hat{\eta}_j$ definition in (\ref{pcl1-2}).
Note that since we proved $\max_{1 \le j \le p} \hat{s}_j \le K$ with probability approaching one, and by (17)(19), uniformly over $j$
$ \hat{\lambda}_{\hat{s}_j} \ge \hat{\lambda}_{K}.$
With (\ref{p.4})(19) and the above inequality
\begin{eqnarray}
P (\cap_{j=1}^p \{ \hat{\eta}_j \ge C_0 \Delta_{-j}\})& \ge &
P (\cap_{j=1}^p \{ \hat{\lambda}_{K} \ge C_0 \Delta_{-j}\}
= P (\cap_{j=1}^p \{ \hat{\lambda}_{K+1} < C_0 \Delta_{-j}\}) \nonumber \\
&\ge& 1- exp ((c'-1)n),\label{cl1-s5}
\end{eqnarray}
and the last inequality is by (\ref{a42a})(\ref{defev1}), and $\hat{\lambda}_{K+1}:= \sigma_{K+1}^2 (Y_{-j})/n$. Then $C_0 \Delta_{-j}$ in (18)
\begin{equation}
C_0 \Delta_{-j} = c_{\Delta}
[ \| \Sigma_{U_{-j}}\|_2+ tr (\Sigma_{U_{-j}})/n]
\ge c_{\Delta} \| \Sigma_{U_{-j}\|_2 } \ge \beta,\label{cl1-s6}
\end{equation}
since $c_{\Delta} >1$.
Define
$ G_{3j}:=\{ \hat{\eta}_j \ge \beta\}.$
Then by (\ref{defev1}) and above event definition, with equality in (\ref{cl1-s5})
$ \cap_{j=1}^p E_{3j} \subseteq \cap_{j=1}^p G_{3j},$
for each $j=1,\cdots,p$. So combine with (\ref{cl1-s5})(\ref{cl1-s6})
we have
\begin{equation}
P (\cap_{j=1}^p G_{3j}) =
P ( \cap_{j=1}^p \{ \hat{\eta}_j \ge \beta \} ) \ge P (\cap_{j=1}^p E_{3j})
\ge 1 - exp ((c'-1)n).\label{cl1-s7}
\end{equation}
Furthermore, using (\ref{p.00})(\ref{cl1-s5}),
uniformly in $j=1,\cdots,p,$ with probability approaching one
\begin{equation}
\frac{\hat{\Psi}_j}{\hat{\eta}_j} \le 1.\label{ratio1}
\end{equation}
Our proofs here are uniform and detailed extensions of pointwise results of proofs of Corollaries 5-6 in Sections B.2.1-B.2.2 of \cite{bbsmw21}. (ii). Note that asymptotic case is obtained from (i). {\bf Q.E.D.}
{\bf Proof of Corollary 2}.
(i). We consider first $p - 1 < n$ scenario (in which RRE effectively becomes OLS) and again proceed by simplifying three terms, $\bar{r}$, $\bar{\eta}$, $\bar{\Psi}$, that appear in the upper bound in Theorem 1.
In that respect $\hat{B}= I_{p-1}$ by definition of RRE. Then clearly $\hat{r}_j= rank (Y_{-j} )= p-1$ since $p-1<n$ in this Corollary. Hence
\begin{equation}
\bar{r}= \max_{1 \le j \le p} \hat{r}_j < p.\label{pcl2-1}
\end{equation}
Also since $I_{p-1} - P_{I_{p-1}} = 0$ we have
\begin{equation}
\bar{\Psi} = 0. \label{pcl2-2}
\end{equation}
This last equality shows that RRE has one less bias term in the upper bound compared to adaptive PCR estimator or compared to the general upper bound in Theorem 1. Use Corollary 11.6.7 in \cite{bern2018} (with $i=p-1, j=1$ there) or (11.6.18) of \cite{bern2018}
for the first inequality below, with $\Sigma_{-j,-j}$ as defined in Appendix B,
\begin{eqnarray}
\hat{\eta}_j & = & \frac{1}{n} \sigma_{p-1}^2 (Y_{-j}) = \frac{1}{n} \sigma_{p-1}^2 (Y_{-j} \Sigma_{-j,-j}^{-1/2} \Sigma_{-j,-j}^{1/2}) \nonumber \\
& \ge &\frac{1}{n} \sigma_{p-1}^2 (Y_{-j} \Sigma_{-j,-j}^{-1/2}) \sigma_{p-1}^2 (\Sigma_{-j,-j}^{1/2}) \nonumber \\
& = & [ \frac{1}{n} \sigma_{p-1}^2 (Y_{-j} \Sigma_{-j,-j}^{-1/2})
] [ Eigmin (\Sigma_{-j, -j})]
\ge c [ \frac{1}{n} \sigma_{p-1}^2 (Y_{-j} \Sigma_{-j,-j}^{-1/2})
],\label{pcl2-3}
\end{eqnarray}
where the last equality is by the definition of singular value. To get the last inequality in (\ref{pcl2-3}), clearly by $\Sigma_{-j,-j}, \Sigma_{U,-j}$ definitions, by using (5) and by Assumptions 2-3
$ min_{1 \le j \le p} Eigmin (\Sigma_{-j,-j}) \ge min_{1 \le j \le p} Eigmin (\Sigma_{U,-j}) \ge c.$
So we obtain
\begin{equation}
\min_{1 \le j \le p} \hat{\eta}_j \ge c \min_{1 \le j \le p}
[ \frac{1}{n}\sigma_{p-1}^2 (Y_{-j} \Sigma_{-j,-j}^{-1/2})].\label{a68a}
\end{equation}
To analyze (\ref{pcl2-3})(\ref{a68a}), define $D_j:=\{ \frac{1}{n} \sigma_{p-1}^2 (Y_{-j} \Sigma_{-j,-j}^{-1/2}) \ge C \}$.
\begin{eqnarray}
P (\min_{1 \le j \le p}\frac{1}{n}\sigma_{p-1}^2 (Y_{-j} \Sigma_{-j,-j}^{-1/2}) \ge C )
&= & P ( \cap_{j=1}^p D_j)
= 1 - P [ (\cap_{j=1}^p D_j)^c] = 1 - P ( \cup_{j=1}^p D_j^c) \nonumber \\
& \ge & 1 - \sum_{j=1}^p P (D_j^c)
\ge 1 - exp (log p - C_p log n) \nonumber \\
& \ge & 1 - exp (- \frac{C_p}{2} log n) = 1 - \frac{1}{n^{C_p/2}},\label{cl2-s1}
\end{eqnarray}
where to get the second inequality, with $Y_{j-1}: n \times p-1$, and $n>p-1$ here, by Theorem 5.39 of \cite{v2012} with Corollary 10 proof in \cite{bbsmw21} via Assumption 4, and applying a union bound (with $t_1 =\sqrt{logn}$ )and by assuming $log p \le (C_p/2) logn$ by Assumption 3. Next, we use (\ref{pcl2-3})(\ref{a68a}) and (\ref{cl2-s1}) with $D_j$ definition
\begin{equation}
P ( \min_{1 \le j \le p} \hat{\eta}_j \ge C) \ge 1 - \frac{1}{n^{C_p/2}}.\label{cl2-s2}
\end{equation}
Now substitute (\ref{pcl2-1})(\ref{pcl2-2})(\ref{cl2-s2}) in the upper bound in Theorem 1 to have the result. (ii). Asymptotics is derived by using (i).
{\bf Q.E.D}
{\bf Proof of Corollary 3}.
(i). There are again three terms that we consider, but we have $p-1>n$ in this RRE case. So
$
\bar{r}= \max_{1 \le j \le p} rank (Y_{-j}) =n.$
Then we have $\bar{\Psi}=0$ as in Corollary 2 due to $\hat{B}= I_{p-1}$. The last term is
$\hat{\eta}_j = \frac{1}{n} \sigma_n^2 (Y_{-j})$, which is the $n$ th largest singular value of $Y_{-j}: n \times p-1$ matrix.
Also we assume $ tr (\Sigma_{U,-j}) / \| \Sigma_{U,-j}\|_2 > C n$, and $\tilde{U}_{-j}$ has independent entries.
Use Lemma \ref{lb2}
\begin{equation}
P ( \hat{\eta}_j \ge tr (\Sigma_{U_{-j}}/n))
\ge 1 - 4 exp(-C_p n).\label{a72a}
\end{equation}
Also we have by definition $\delta_n:= min_{1 \le j \le p} tr (\Sigma_{U,-j})/n$. Define
$G_{4j}:= \{ \hat{\eta}_j \ge tr (\Sigma_{U,-j}/n)\}$, $ G_{5j}:= \{ \hat{\eta}_j \ge \delta_n\}.$
Note that for each $j=1,\cdots,p$
\begin{equation}
G_{4j} \subseteq G_{5j},
\mbox{hence}
\quad
\cap_{j=1}^p G_{4j} \subseteq \cap_{j=1}^p G_{5j}.\label{pcl3-s1}
\end{equation}
Note that by $G_{5j}$ definition,
\begin{equation}
\cap_{j=1}^p G_{5j} = \{ \min_{1 \le j \le p} \hat{\eta}_j \ge \delta_n\},\label{pcl3-s2}
\end{equation}
since the lower bounds are the same $ \delta_n$.
Next we have the following proof
\begin{eqnarray}
P (\min_{1 \le j \le p} \hat{\eta}_j \ge \delta_n ) & = & P (\cap_{j=1}^p G_{5j})
\ge P ( \cap_{j=1}^p G_{4j} ) = 1 - P ( \cup_{j=1}^p \{ \hat{\eta}_j < tr (\Sigma_{U,-j})/n\})\nonumber \\
& \ge & 1 - \sum_{j=1}^p P ( \{ \hat{\eta}_j < tr (\Sigma_{U,-j})/n\})
\ge 1 - exp (log 4 p - C_p n) \nonumber \\
& \ge & 1 - exp (-C_p n/2),\label{pcl3-1}
\end{eqnarray}
where the first equality is by (\ref{pcl3-s2}), and the second inequality is by (\ref{pcl3-s1}), and the third inequality is by (\ref{a72a}), and the last inequality is by Assumption $3^{*}$(i). The desired result follows by incorporating these three terms in the upper bound of Theorem 1. (ii). Asymptotics follows from (i).
{\bf Q.E.D.}
Before the following proof, we proceed with three inequalities that are used for rectangular matrices. Take a generic rectangular matrix $M$ and a vector $x$.
\begin{equation}
\| M x \|_2 \le \| M \|_2 \| x\|_2,\label{i1}
\end{equation}
by (11.4.4) of \cite{bern2018}. Then by (11.3.15) of \cite{bern2018} where for two generic rectangular matrices $M,L$
\begin{equation}
\| M L \|_2 \le \| M\|_2 \| L\|_2,\label{i2}
\end{equation}
We form the following matrix norm inequality for a generic rectangular matrix $M: m_1 \times m_2$
\begin{eqnarray}
\| M \|_2 \le \| M \|_F &= &[\sum_{i=1}^{m_1} \sum_{j=1}^{m_2} | M_{ij} |^2 ]^{1/2} \nonumber \\
& \le & \sqrt{m_1 m_2} \max_{1 \le i \le m_1} \max_{1 \le j \le m_2} | M_{ij} |= \sqrt{m_1 m_2} \| M \|_{\infty}.\label{61}
\end{eqnarray}
where $\|.\|_F $ is the Frobenius norm and we get the first inequality by (4.4) of \cite{v2019}.
{\bf Proof of Theorem 2}.
First note that $E y_{jt}= \mu_j = a_j' E f_t$, and $E Y_{-jt} = \mu_{-j} = A_{-j} E f_t$. Also see that
\begin{equation}
y_{jt} - \mu_j= a_j' (f_t - E f_t) + u_{jt},\label{eq1}
\end{equation}
and
\begin{equation}
Y_{-jt} - \mu_{-j}= A_{-j} (f_t - E f_t) + U_{-jt},\label{eq2}
\end{equation}
(i). Note that we need to estimate the $j$ th main diagonal term for the precision matrix of asset returns. In that sense the key element is the reciprocal of that term which is defined in (11), and the population version in (B.10). We use triangle inequality
to consider the estimation error by (11)(B.10)
\begin{equation}
| \tilde{\tau}_j^2 - \tau_j^2| \le | \frac{1}{n} \sum_{t=1}^n [y_{jt}^2 - E (y_{jt} - \mu_j)^2] | +
\left| [n^{-1} \sum_{t=1}^n y_{jt} Y_{-jt}'] \tilde{\alpha}_j - E (y_{jt}- \mu_j)( Y_{-jt} - \mu_{-j})'\alpha_{j}^*
\right|.\label{47}
\end{equation}
Next add and subtract to the second right side term $E [(y_{jt} - \mu_j)( Y_{-jt} - \mu_{-j})' ]\tilde{\alpha}_j$, by iid nature of data,
\begin{eqnarray}
| \tilde{\tau}_j^2 - \tau_j^2| &\le &| \frac{1}{n} \sum_{t=1}^n [y_{jt}^2 - E (y_{jt} -\mu_j)^2] |
+ \left| n^{-1} \sum_{t=1}^n [y_{jt} Y_{-jt}' - E (y_{jt} -\mu_j) ( Y_{-jt}- \mu_{-j})' \tilde{\alpha}_j
\right| \nonumber \\ &+& \left|E [(y_{jt} - \mu_j )( Y_{-jt} - \mu_{-j})'] (\tilde{\alpha}_j - \alpha_{j}^*)
\right|.\label{48}
\end{eqnarray}
Take uniform bounds over $j=1,\cdots,p$ and use Cauchy-Schwartz inequality for the second and third right side terms on (\ref{48})
\begin{eqnarray}
\max_{1 \le j \le p} | \tilde{\tau}_j^2 - \tau_j^2| &\le &
\max_{1 \le j \le p} | \frac{1}{n} \sum_{t=1}^n [y_{jt}^2 - E (y_{jt} - \mu_j) ^2] | \nonumber \\
& + & \max_{1 \le j \le p} \| n^{-1} \sum_{t=1}^n [ y_{jt} Y_{-jt}' - E (y_{jt} - \mu_j) ( Y_{-jt} - \mu_{-j})']\|_2 \max_{1 \le j \le p} \|\tilde{\alpha}_j\|_2 \nonumber \\
& + & \max_{1 \le j \le p} \| E [(y_{jt} - \mu_j )(Y_{-jt} - \mu_{-j})'] \|_2
\max_{1 \le j \le p} \| \tilde{\alpha}_j - \alpha_{j}^*\|_2.\label{49}
\end{eqnarray}
Each right side term in (\ref{49}) will be analyzed as a step in the proof.
{\bf Step 1.} Start with the first right side term on (\ref{49}), and by (2)
\[ y_{jt}^2 = a_j' f_t f_t' a_j + 2 a_j' f_t u_{jt} + u_{jt}^2,\]
and by Assumption 1(iv), and using (\ref{eq1})
\[ E (y_{jt} -\mu_j)^2 = a_j' \Sigma_f a_j + E (u_{jt}^2).\]
Use triangle inequality by the last two equalities, and noting $\Sigma_f:= E (f_t - E f_t )( f_t - E f_t)'$
\begin{eqnarray}
\max_{1 \le j \le p} | \frac{1}{n} \sum_{t=1}^n [y_{jt}^2 - E (y_{jt} -\mu_j)^2]|
& \le & \max_{1 \le j \le p} | a_j' [\frac{1}{n} \sum_{t=1}^n (f_t f_t'- \Sigma_f)]a_j | \nonumber \\
& +& \max_{1 \le j \le p} | \frac{1}{n} \sum_{t=1}^n [u_{jt}^2 - E (u_{jt}^2)]| \nonumber \\
&+& \max_{1 \le j \le p} 2 | a_j' [n^{-1}\sum_{t=1}^n f_t u_{jt} ]|.\label{50}
\end{eqnarray}
Take the first right side term in (\ref{50}) above, and use Cauchy-Schwarz inequality first and then (\ref{i1})
\begin{equation}
\max_{1 \le j \le p} | a_j' [\frac{1}{n} \sum_{t=1}^n (f_t f_t'- \Sigma_f)]a_j | \le
\left[ \max_{1 \le j \le p} \| a_j \|_2^2 \right] \| \frac{1}{n} \sum_{t=1}^n [f_t f_t' - \Sigma_f] \|_{2}.\label{51}
\end{equation}
We consider the second term on the right side of (\ref{51}), with $F: n \times K$ matrix,
and $ \| \Sigma_f \|_2 \le C < \infty$ by Assumption 3(ii)
\begin{equation}
\| \frac{F'F}{n} - \Sigma_f \|_2 \le C [ \sqrt{\frac{K}{n}} + \sqrt{\frac{log n}{n}}] \| \Sigma_f \|_2,\label{pla2-1}
\end{equation}
by Lemma 23 of \cite{bbsmw21} since rows of $F$ are independent subgaussian vectors, with probability at least $1-2/n^c$ via Assumption 1(iii), Assumption 4(ii), and note that $K < n$. By Assumption 4(iii)
\begin{equation}
\max_{1 \le j \le p} \| a_j \|_2^2 \le C K.\label{53}
\end{equation}
Use (\ref{pla2-1})-(\ref{53}) in (\ref{51}) and Assumption 3 to have
\begin{equation}
\max_{1 \le j \le p} | a_j' (\frac{1}{n} \sum_{t=1}^n [f_t f_t' - \Sigma_f ] a_j | = O_p \left(\max(\frac{K^{3/2}}{n^{1/2}}, K \sqrt{\frac{logn}{n}})\right).\label{54}
\end{equation}
Then in (\ref{50}) consider the second right side term
\begin{equation}
\max_{1 \le j \le p} | \frac{1}{n} \sum_{t=1}^n [u_{jt}^2 - E (u_{jt}^2)]|= O_p (\sqrt{\frac{log p}{n}}).\label{55}
\end{equation}
We now show how we obtain (\ref{55}).
Note that by Corollary 2.8.3 of \cite{v2019}, for $t_1>0$, and the union bound
\[ P \left[ \max_{1 \le j_1 \le p } \max_{1 \le j_2 \le p} | \frac{1}{n} \sum_{t=1}^n [u_{j_1,t} u_{j_2,t} - E (u_{j_1,t} u_{j_2,t})]|>t_1
\right]
\le p^2 exp \left(-c min(\frac{t_1^2}{C_{\psi}^2}, \frac{t_1}{C_{\psi}})n\right),\]
where we use $u_{j_1,t}$ and $u_{j_2,t}$ are both subgaussian random variables, so $u_{j_1,t} u_{j_2,t}$ are subexponential random variable (independent across $t=1,\cdots,n$) by Lemma 2.7.7 of \cite{v2019} and exercise 2.7.10 of \cite{v2019} shows centered subexponential are also subexponential random variables, and we have
\[ \max_{1 \le j_1 \le p} \max_{1 \le j_2 \le p} \| u_{j_1,t} u_{j_2,t} \|_{\psi} \le C_{\psi} < \infty\]
where
$C_{\psi}$ are the upper bound constant for the subexponential norm in Definition 2.7.5 of \cite{v2019}. Next
$
2 p^2 exp \left(-c min(\frac{t_1}{C_{\psi}^2}, \frac{t_1}{C_{\psi}}) n \right) = exp \left(log (2 p^2) - c \frac{t_1^2 n}{C_{\psi}^2}\right)$,
with $t_1:= C \frac{\sqrt{log p}}{\sqrt{n}}$ and with sufficiently large $n$ (i.e.$ \sqrt{n} \ge \frac{C}{C_{\psi}} (log p)^{1/2}$). Set
$ c_{\psi}:= c \frac{C^2}{C_{\psi}^2} - 2 > 0$
to have
$ exp [ log 2 + 2 log p - c \frac{C^2}{C_{\psi}^2} log p ] = \frac{2}{p^{c_{\psi}}}.$
Then
\[ P \left(\max_{1 \le j_1 \le p } \max_{ 1 \le j_2 \le p} | \frac{1}{n} \sum_{t=1}^n u_{j_1,t} u_{j_2,t} - E u_{j_1,t} u_{j_2,t}| > C \frac{\sqrt{log p}}{\sqrt{n}}
\right) \le \frac{2}{p^{c_{\psi}}}.\]
The asymptotics in (\ref{55}) are obtained after taking $n \to \infty, p \to \infty$. Consider the third right side term in (\ref{50}). Lemma 2.7.7 and Corollary 2.8.3 of \cite{v2019} together with Assumptions 1, 4 and Holder's inequality give
\begin{eqnarray}
\max_{1 \le j \le p} | a_j' (\frac{1}{n}\sum_{t=1}^n f_t u_{jt} )| & \le & \max_{1 \le j \le p} \| a_j \|_1 \max_{1 \le j \le p} \| \frac{1}{n} \sum_{t=1}^n f_t u_{jt} \|_{\infty} \nonumber \\
& = & O_p ( K \sqrt{\frac{log (Kp)}{n}}) = O_p ( \frac{K \sqrt{log p}}{\sqrt{n}}),\label{55a}
\end{eqnarray}
where the last equality uses $p-1>K$. By (\ref{54})-(\ref{55a}) we have the first right side term in (\ref{49})
\begin{equation}
\max_{1 \le j \le p} | \frac{1}{n} \sum_{t=1}^n [y_{jt}^2 - E (y_{jt} - \mu_j)^2] | = O_p \left( \max( \frac{K^{3/2}}{n^{1/2}}, K \sqrt{\frac{\max(logp, log n)}{n}}) \right).\label{55ab}
\end{equation}
{\bf Step 2.} We consider the second right side term in (\ref{49}). By (2)(3) and Assumption 1(iv)
\[ y_{jt} Y_{-jt}' = a_j' f_t f_t' A_{-j}' + a_j' f_t U_{-jt}' + u_{jt} f_t' A_{-j}' + u_{jt} U_{-jt}',\]
and by (\ref{eq1}) (\ref{eq2})
\[ E (y_{jt} - \mu_j)( Y_{-jt} - \mu_{-j})'= a_j' \Sigma_f A_{-j}'.\]
Use these last two equations to see that the following term
\[ \max_{1 \le j \le p} \| \frac{1}{n} \sum_{t=1}^n [y_{jt} Y_{-jt}' - E (y_{jt} - \mu_j)( Y_{-jt} - \mu_{-j})'] \|_2,\]
is upper bounded by
\begin{eqnarray}
&&\max_{1 \le j \le p} \| a_j'\left( \frac{1}{n} \sum_{t=1}^n [ f_t f_t' - \Sigma_f ] \right)A_{-j}' \|_2
+ \max_{1 \le j \le p} \|( \frac{1}{n} \sum_{t=1}^n u_{jt} f_t') A_{-j}' \|_2 \nonumber \\
&+& \max_{1 \le j \le p} \| a_j' ( \frac{1}{n} \sum_{t=1}^n f_t U_{-jt}' ) \|_2
+ \max_{ 1 \le j \le p} \| \frac{1}{n} \sum_{t=1}^n u_{jt} U_{-jt}' \|_2.\label{58}
\end{eqnarray}
Consider the first term on the right side of (\ref{58}) by (\ref{i1})(\ref{i2})
\begin{eqnarray}
& & \max_{1 \le j \le p} \| a_j' ( \frac{1}{n} \sum_{t=1}^n [f_t f_t' - \Sigma_f ] A_{-j}' \|_2
\le \max_{1 \le j \le p} \| a_j \|_2
\max_{1 \le j \le p}
\left\| \left(\frac{1}{n} \sum_{t=1}^n [ f_t f_t' - \Sigma_f ]\right)
A_{-j}' \right\|_2 \nonumber \\
& \le & ( \max_{1 \le j \le p} \| a_j \|_2 )\| \frac{1}{n} \sum_{t=1}^n [ f_t f_t' - \Sigma_f]\|_2
\times (\max_{1 \le j \le p} \| A_{-j} ' \|_2) \nonumber \\
& = & O (K^{1/2}) O_p ( \max (\frac{K^{1/2}}{n^{1/2}}, \frac{\sqrt{log n}}{n^{1/2}})) O (\sqrt{p})
= O_p ( max (\frac{K \sqrt{p}}{n^{1/2}}, \frac{ \sqrt{p} \sqrt{K log n}}{n^{1/2}})),\label{58a}
\end{eqnarray}
and the rates are by (\ref{pla2-1})(\ref{53}) and Assumption 3. Then consider the second term on the right side of (\ref{58})
\begin{eqnarray}
\max_{1 \le j \le p} \|( \frac{1}{n} \sum_{t=1}^n u_{jt} f_t') A_{-j}' \|_2 & \le & \max_{1 \le j \le p} \| ( \frac{1}{n} \sum_{t=1}^n u_{jt} f_t')\|_2 \max_{1 \le j \le p} \| A_{-j}' \|_2 \nonumber \\
& \le & \sqrt{K} \max_{1 \le j \le p} \| ( \frac{1}{n} \sum_{t=1}^n u_{jt} f_t')\|_{\infty} \max_{1 \le j \le p} \| A_{-j}' \|_2 \nonumber \\
& = & O_p (\sqrt{K} \frac{\sqrt{log p}}{\sqrt{n}}) O (\sqrt{p}),\label{60}
\end{eqnarray}
where we use (\ref{i1}) for the first inequality since $u_{jt} f_t': 1 \times K$ vector and $A_{-j}': K \times p-1$ matrix, and the second inequality is by simple vector norm inequality tying $l_2, l_{\infty}$ vector norms, and the rates are by Assumption 3(iii) $\max_{1 \le j \le p} \| A_{-j} ' \|_2 = O (\sqrt{p})$ and Lemma 2.7.7 and Corollary 2.8.3. of \cite{v2019} which provides $\| n^{-1} \sum_{t=1}^n u_{jt} f_t' \|_{\infty} = O_p (\sqrt{logp/n})$.
Now third term on the right side of (\ref{58})
\begin{eqnarray}
\max_{1 \le j \le p} \|a_j' ( \frac{1}{n} \sum_{t=1}^n f_t U_{-jt}') \|_2 & \le & \max_{1 \le j \le p} \| a_j \|_2 \max_{1 \le j \le p} \| \frac{1}{n} \sum_{t=1}^n f_{t} U_{-jt}' \|_2 \nonumber \\
& \le & \max_{1 \le j \le p} \| a_j \|_2 \sqrt{K p} \max_{1 \le j \le p} \| \frac{1}{n} \sum_{t=1}^n f_t U_{-jt}'\|_{\infty} \nonumber \\
& = & O (\sqrt{K}) O_p ( \frac{\sqrt{K p (logp)}}{\sqrt{n}}) = O_p ( K \sqrt{p} \frac{\sqrt{logp}}{\sqrt{n}}),\label{62}
\end{eqnarray}
where we use for the first inequality (\ref{i1})
and then for the second inequality we use (\ref{61}) and for the rates we use Assumption 5(iii), and Lemma 2.7.7 and Corollary 2.8.3 of \cite{v2019} with Assumption 4 since $f_t U_{-jt}'$ is $K \times p-1$ matrix. Then for the fourth term on the right side of (\ref{58}) via Corollary 2.8.3 of \cite{v2019} and Assumption 4
and simple vector norm inequality
\begin{equation}
\max_{1 \le j \le p} \| \frac{1}{n} \sum_{t=1}^n u_{jt} U_{-jt}' \|_2 \le \sqrt{p} \max_{1 \le j \le p} \| \frac{1}{n} \sum_{t=1}^n u_{jt} U_{-jt}' \|_{\infty} = O_p (\sqrt{p} \frac{\sqrt{log p}}{\sqrt{n}}).\label{63}
\end{equation}
Relying on (\ref{58a})(\ref{60})(\ref{62})(\ref{63}), and keeping in mind that (\ref{62}) has the largest rate by Assumptions 3 and 5(i), and we have the following for the second right side term in (\ref{49})
\begin{equation}
\max_{1 \le j \le p} \| \frac{1}{n} \sum_{t=1}^n [y_{jt} Y_{-jt}' - E (y_{jt} - \mu_j)( Y_{-jt} - \mu_{-j})']\|_2 = O_p ( K \sqrt{p} \frac{\sqrt{log p}}{\sqrt{n}}).\label{64}
\end{equation}
\noindent We find it useful to recall the rate definition
\begin{equation}
r_{w1}:= \max \left( \sqrt{\frac{log n + \bar{r}}{n \bar{\eta}}} , \sqrt{\frac{K }{\bar{\xi} }}, \sqrt{ \frac{K \bar{\Psi}}{\bar{\xi} \bar{\eta} }}, \sqrt{\frac{K}{\bar{\xi} \bar{\eta}}}\right),\label{64a}
\end{equation}
\noindent The proof of Theorem 1-Step 2d gives
\begin{equation}
\max_{1 \le j \le p} \| \tilde{\alpha}_j \|_2 = O_p (r_{w1}).\label{65}
\end{equation}
Combine (\ref{64})(\ref{65}) for the second right side term in (\ref{49})
\begin{eqnarray}
& & \max_{1 \le j \le p} \| \frac{1}{n} \sum_{t=1}^n [y_{jt} Y_{-jt}' - E (y_{jt} - \mu_j )( Y_{-jt} - \mu_{-j})']\|_2 \max_{1 \le j \le p} \| \tilde{\alpha}_j \|_2 \nonumber \\
&=&
O_p ( (K \sqrt{p logp/n} )r_{w1}) \label{66}
\end{eqnarray}
{\bf Step 3.} Now consider the third term on the right side of (\ref{49}). The key term is
$E [(y_{jt}-\mu_j)( Y_{-jt} -\mu_{-j}) '] = a_j \Sigma_f A_{-j}',$
where we use Assumption 1 and (2)(3). Since we have for rectangular matrices (\ref{i1})(\ref{i2})
\begin{equation}
\max_{1 \le j \le p} \| E (y_{jt} - \mu_j ) ( Y_{-jt} - \mu_{-j}) ' \|_2 = \max_{1 \le j \le p} \| a_j' \Sigma_f A_{-j} ' \|_2 = d_{2n},
\label{67}
\end{equation}
By (\ref{67}), the third term on the right side of (\ref{49}) is
via Theorem 1
\begin{equation}
\max_{1 \le j \le p} \| E (y_{jt} -\mu_j)( Y_{-jt} - \mu_{-j})' \|_2 \max_{1 \le j \le p} \| \hat{\alpha}_j - \alpha_{j}^*\|_2
= O_p ( d_{2n} r_{w1}).\label{68}
\end{equation}
{\bf Step 4.} We combine (\ref{55ab})(\ref{66})(\ref{68}) with Assumption 5(i) on the right side of (\ref{49}) to have the desired result.
{\bf Q.E.D.}
(ii).
By Assumption 5(ii)-(iii) we obtain the consistency. {\bf Q.E.D.}
(iii). By (B.11)
$ \tau_j^2:= E [(y_{jt} - \mu_j)^2 - E (y_{jt} -\mu_j)( Y_{-jt} - \mu_{-j})' ] \alpha_{j}^*.$
Then by Cauchy-Schwarz inequality
\begin{eqnarray}
\max_{1 \le j \le p} E [(y_{jt} - \mu_j)( Y_{-jt} - \mu_{-j})'] \alpha_{j}^*
& \le & \left[ \max_{1 \le j \le p} \| E [(y_{jt} -\mu_j )( Y_{-jt} - \mu_{-j})']\|_2
\right] \nonumber \\
& \times& \left[ \max_{1 \le j \le p} \| \alpha_{j}^* \|_2
\right] \nonumber \\
& \le& \frac{C K^{1/2} d_{2n}}{\sqrt{\bar{\xi}}} \to 0,\label{pla4iii-1}
\end{eqnarray}
where we use Step 3 of Theorem 2 together with Step 3 of Theorem 1 proof, and (\ref{67}), and Assumption 5(iii).
Next by using (2), Assumptions 1-3
\begin{eqnarray*}
\min_{1 \le j \le p} E (y_{jt} -\mu_j)^2 &\ge& min_{1 \le j \le p} a_j' \Sigma_f a_j + \min_{1 \le j \le p} E (u_{jt}^2) \\
& \ge & Eigmin (\Sigma_f ) \min_{1 \le j \le p} \| a_j \|_2^2 + \min_{1 \le j \le p} \sigma_{j}^2 \ge c >0.
\end{eqnarray*}
The last two results provide the desired result.
{\bf Q.E.D.}
{\bf Proof of Theorem 3}. First of all
\[ \max_{1 \le j \le p} \| \tilde{\Theta}_j' - \Theta_{j}' \|_2 \le
\max_{1 \le j \le p} \left| \frac{1}{\tilde{\tau}_j^2} - \frac{1}{\tau_j^2}
\right| + \max_{1 \le j \le p} \| \frac{\tilde{\alpha}_j}{\tilde{\tau}_j^2} - \frac{\alpha_{j}^*}{\tau_j^2}\|_2.\]
Then add and subtract $\alpha_{j}^*/\tilde{\tau}_j^2$ to the right hand side of the immediately above inequality, and via triangle inequality and arranging
\begin{eqnarray}
\max_{1 \le j \le p} \| \tilde{\Theta}_j' - \Theta_{j}' \|_2 &\le& \max_{1 \le j \le p} \left| \frac{1}{\tilde{\tau}_j^2} - \frac{1}{\tau_j^2}
\right|
+ \max_{1 \le j \le p} \| \tilde{\alpha}_j - \alpha_{j}^* \|_2 \max_{1 \le j \le p } [\frac{1}{\tilde{\tau}_j^2}] \nonumber \\
& + & \max_{1 \le j \le p} \| \alpha_{j}^* \|_2 \max_{1 \le j \le p} \left| \frac{1}{\tilde{\tau}_j^2} - \frac{1}{\tau_j^2}
\right|.\label{pthm1-1}
\end{eqnarray}
Note that since $\min_{1 \le j \le p} \tilde{\tau}_j^2 \ge \min_{1 \le j \le p} \tau_j^2 - \max_{1 \le j \le p} | \tilde{\tau}_j^2 - \tau_j^2|$
\[
\max_{1 \le j \le p} \left(\frac{1}{\tilde{\tau}_j^2} \right) \le \frac{1}{\min_{1 \le j \le p} \tau_j^2 - \max_{1 \le j \le p} | \tilde{\tau}_j^2 - \tau_j^2|}
\le \frac{1}{c - o_p (1)},
\]
by Theorem 2(ii)(iii) under Assumptions 1, 5. Then by (\ref{46a}) and Assumption 5(iii)
\begin{equation}
\max_{1 \le j \le p} \| \alpha_{j}^* \|_2 = O (\frac{\sqrt{K}}{\sqrt{\bar{\xi}}}) = o(1).\label{a87a}
\end{equation}
So the third term on right side of (\ref{pthm1-1}) converges to zero faster in probability than the first right side term and the first term converges to zero faster than the second term in (\ref{pthm1-1}) given Theorem 2.
{\bf Q.E.D}
We now provide three corollaries for specific subcases of our general estimator. Note that we provide the specific case of adaptive PCR estimator of $\tau_j^2$, which is defined as $\hat{\tau}_{j, PCR}^2$. In that respect the rate $r_{w1}$ simplifies considerably. In adaptive PCR case these rates are with wpa1
\begin{equation}
r_{w1,PCR}= \max \left( \sqrt{\frac{log n +K }{n }}, \sqrt{\frac{K}{\bar{\xi} }}
\right),\label{pcr-1}
\end{equation}
See that rate (\ref{pcr-1}) can be obtained by seeing $\bar{r} \le K, \bar{\Psi}/\bar{\eta} \le 1, \bar{\eta} \ge \beta$ wpa1 as shown in Corollary 1. Corollary \ref{coa1} below follows through replacing $r_{w1}$ in Theorem 2 with $r_{w1,PCR}$.
\begin{corollary}\label{coa1}
(i). Under Assumptions 1-5(i) and using $r_{w1,PCR}$ instead of $r_{w1}$,
\begin{eqnarray*}
\max_{1 \le j \le p} | \hat{\tau}_{PCR,j}^2 - \tau_j^2| &=&
O_p \left( K (\sqrt{p log p/n }) r_{w1,PCR}
\right)\\
& +& O_p \left( d_{2n} r_{w1,PCR}
\right)+ O_p \left( max (\frac{K^{3/2}}{n^{1/2}}, K \frac{\max(\sqrt{logp}, \sqrt{logn})}{n^{1/2}})
\right).
\end{eqnarray*}
(ii). Under Assumptions 1-5,
\[ \max_{1 \le j \le p} | \hat{\tau}_{PCR,j}^2 - \tau_j^2| = o_p (1).\]
(iii). Under Assumptions 1-5,
\[ \min_{1 \le j \le p} \tau_j^2 \ge c - o_p (1).\]
\end{corollary}
Here we consider the second subcase $p-1<n$ scenario of RRE estimator of $\tau_j^2$. We define it as $\hat{\tau}_{RRE,j}^2$.
We define the following rate which will be used instead of $r_{w1}$, wpa1, which can be deduced from Corollary 2 proof,
\begin{equation}
r_{RRE-l}:= \max( \sqrt{\frac{log n + p}{n}
}, \sqrt{\frac{K}{\bar{\xi}}}).\label{RREl}
\end{equation}
\begin{corollary}\label{coa2}
(i).Under Assumptions 1-5(i) and using $r_{RRE-l}$ instead of both $r_{w1}$, setting
$p/n = a_{2n} \to 0$,
\begin{eqnarray*}
\max_{1 \le j \le p} | \hat{\tau}_{RRE,j}^2 - \tau_j^2| &=&
O_p \left( K (\sqrt{ a_{2n} log p }) r_{RRE-l}
\right)\\
& + &O_p \left( d_{2n} r_{RRE-l}
\right) +
O_p \left( max (\frac{K^{3/2}}{n^{1/2}}, K \frac{\max(\sqrt{logp}, \sqrt{logn})}{n^{1/2}})
\right)
\end{eqnarray*}
(ii). Under Assumptions 1-5 with the conditions in (i)
\[ \max_{1 \le j \le p} | \hat{\tau}_{RRE,j}^2 - \tau_j^2| = o_p (1).\]
(iii). Under Assumptions 1-5 with the conditions in (i)
\[ \min_{1 \le j \le p} \tau_j^2 \ge c - o_p (1).\]
\end{corollary}
We now consider the third subcase of RRE estimator when $p-1>n$.
Note that this subcase will be different than the others, and we use an estimator of $\tau_j^2$ which is different due to interpolation property of RRE in this case. First let us show the interpolation and how it affects the estimator $\tilde{\tau}_j^2$ proposed, and the need to modify. In case of $p-1>n$, RRE estimator becomes the $l_2$ norm interpolator.
Define the minimum norm interpolator as in (1) of \cite{bsmw22}, for each $j=1,\cdots, p$
\[ \hat{\alpha}_j:= argmin \{ \| \alpha_{j} \|_2: \| Y_{-j} \alpha_{j} - y_j \|_2= \min_{\gamma_{j}} \| Y_{-j} \gamma_{j} - y_j \|_2.\},\]
Lemma 28 of \cite{bsmw22} shows
$ \hat{\alpha}_j = Y_{-j}^+ y_j = \hat{\alpha}_{RRE,j}.$
Then when $p-1>n$,
\[Y_{-j} \hat{\alpha}_j - y_j =
Y_{-j} Y_{-j}^+ y_j - y_j = y_j - y_j =
0,\]
with $Y_{-j} Y_{-j}^+= I_n $ by (136) of \cite{bsmw22}. This is interpolation, and we cannot use the formula for $\tilde{\tau}_j^2$ since with interpolation it can be shown, $\tilde{\tau}_j^2=0$ in RRE in (11), with $p-1>n$ case. So we propose the following estimator, which can be considered a subcase of general $\tilde{\tau}_j^2$. Define
\begin{equation}
\hat{\tau}_{INT,j}^2:= \frac{y_j'y_j}{n}.\label{ca3-1}
\end{equation}
\begin{corollary}\label{coa3}
Under Assumptions 1-2, Assumptions $3^{*}(i)$, 3(ii), 4
(i). \[
\max_{1 \le j \le p} | \hat{\tau}_{INT,j}^2 - \tau_j^2| = O_p \left( \frac{\sqrt{K} d_{2n}}{\sqrt{\bar{\xi}}}
\right) + O_p \left( max (\frac{K^{3/2}}{n^{1/2}}, K \frac{\sqrt{logp}}{n^{1/2}})
\right).
\]
(ii). With added to (i), Assumptions 5(ii) with $\sqrt{K} d_{2n}/\sqrt{\bar{\xi}} \to 0$
\[ \max_{1 \le j \le p} | \hat{\tau}_{INT,j}^2 - \tau_j^2| = o_p (1).\]
(iii). Using Assumptions in (ii)
\[ \min_{1 \le j \le p} \tau_j^2 \ge c - o_p (1).\]
\end{corollary}
{\bf Proof of Corollary \ref{coa3}}. (i). By (\ref{13})(\ref{ca3-1}) and triangle inequality
\begin{eqnarray*}
\max_{1 \le j \le p} | \hat{\tau}_{INT,j}^2 - \tau_j^2| &\le& \max_{1 \le j \le p} | \frac{1}{n} \sum_{t=1}^n [y_{jt}^2 - E (y_{jt} - \mu_j)^2 ]| \\
& +& \max_{1 \le j \le p} \| E (y_{jt}- \mu_j) ( Y_{-jt} - \mu_{-j})' \|_2 \max_{1 \le j \le p} \| \alpha_{j}^*\|_2.
\end{eqnarray*}
Next, Step 1 in the proof of Theorem 2 with (A.16), and (A.28) provides the result. (ii). Given Assumptions in part (i) with $\sqrt{K} d_{2n}/\sqrt{\bar{\xi}} \to 0$ provides the consistency. (iii). This is the same as in the proof of Theorem 2 (iii).
{\bf Q.E.D.}
\setcounter{equation}{0}
\setcounter{lemma}{0}
\setcounter{table}{0}
\begin{center}
\uppercase{Appendix B}
\end{center}
In this part B, we provide a precision matrix formula and then provide two lemmata on tail inequalities.
{\bf Precision Matrix Formula Derivation}
Precision matrix relation to a hidden factor model context is established here for the first time in the literature and in detail.
Define all assets in the portfolio as $y_t=(y_{jt}, Y_{-jt}')': p \times 1$. In case of $j=3$, this represents $y_{3t}$ in the third cell in the vector $y_t$ and $Y_{-3t}$ represents elements $1,2, 4,\cdots,p$ in $y_t$.
{\bf Proof of Lemma 1}.
We will provide formula for $j$ th row and then stack all rows to get an expression. Before that define $\Sigma_{j,j}:= E [(y_{jt}-\mu_j)^2]$ and $\Sigma_{j,-j}:=E [(y_{jt} -\mu_j) (Y_{-jt}- \mu_{-j})']: 1 \times p-1$ and represents $j$ th row of $\Sigma$ with $j$ th element missing, and
by symmetricity of $\Sigma$, $\Sigma_{j,-j}=(\Sigma_{-j,j})'$. Specifically, define $\Sigma_{-j,j}:=E [ (Y_{-jt} - \mu_{-j}) ( y_{jt} - \mu_j) ]: p-1 \times 1$, as the $j$ th column of $\Sigma$ where $j$ th element is missing.
In the same way we define $\Sigma_{-j,-j}:= E [ (Y_{-jt} - \mu_{-j}) (Y_{-jt} - \mu_{-j})']: p-1 \times p-1$ where this matrix is $j$ the row and column deleted from $\Sigma$. By Section 2.1 of \cite{yuan2010}, the main diagonal elements are given by the scalar
\begin{equation}
\Theta_{j,j}:= ( \Sigma_{j,j}- \Sigma_{j,-j} \Sigma_{-j, -j}^{-1} \Sigma_{-j,j})^{-1}.\label{5}
\end{equation}
and the $j$ th row without the $j$ th element is:
\begin{equation}
\Theta_{j,-j}:=- \Theta_{j,j} \Sigma_{j,-j} \Sigma_{-j,-j}^{-1}.\label{6}
\end{equation}
Now we relate these quantities to a linear regression context. Define the following minimizer
$ \alpha_{j}^*:= argmin_{\alpha_{j} \in R^{p-1}} E [ (y_{jt} - \mu_j) - (Y_{-jt} - \mu_{-j})' \alpha_{j}]^2,$
with given zero mean $(y_{jt}-\mu_j ), (Y_{-jt} - \mu_{-j})$ and finite second moments for $y_{jt}, Y_{-jt}$, Lemma 27 in \cite{bsmw22} shows that
\begin{equation}
\alpha_{j}^* = \Sigma_{-j,-j}^+ \Sigma_{-j,j},\label{7}
\end{equation}
where $\Sigma_{-j,-j}^+$ is the Moore-Penrose inverse of $\Sigma_{-j,-j}$.
First,
\begin{equation}
\Sigma_{-j,-j}= E [ (Y_{-jt} - \mu_{-j}) ( Y_{-jt} - \mu_{-j})']= A_{-j} \Sigma_f A_{-j}' + \Sigma_{U,-j},\label{b3a}
\end{equation}
by (3) and Assumption 1(iv). Then
\begin{eqnarray*}
\min_{ 1 \le j \le p} Eigmin( \Sigma_{-j,-j})& =& \min_{1 \le j \le p} Eigmin (A_{-j} \Sigma_f A_{-j}' + \Sigma_{U,-j}) \\
&\ge& min_{1 \le j \le p} Eigmin (\Sigma_{U,-j}) \ge c >0,
\end{eqnarray*}
by Assumption 2-3.
(\ref{7}) simplifies to
\begin{equation}
\alpha_{j}^*= \Sigma_{-j,-j}^{-1} \Sigma_{-j,j}.\label{8}
\end{equation}
Define $\eta_{jt}:= (y_{jt} - \mu_j) - (Y_{-jt} - \mu_{-j})' \alpha_{j}^*$ for $j=1,\cdots, p$. Now consider
\begin{eqnarray}
E [ (Y_{-jt} - \mu_{-j}) \eta_{jt}] & = & E [ (Y_{-jt} - \mu_{-j}) ( y_{jt} - \mu_j) ] - E [ (Y_{-jt} - \mu_{-j}) ( Y_{-jt} - \mu_{-j})'] \alpha_{j}^* \nonumber \\
& = & \Sigma_{-j,j} - \Sigma_{-j,-j} \alpha_{j}^*
= \Sigma_{-j,j} - \Sigma_{-j,-j} \Sigma_{-j,-j}^{-1} \Sigma_{-j,j} =0,\label{9}
\end{eqnarray}
where the first equality uses $\eta_{jt}$ definition, and the second equality uses definitions of moment matrices, and the third one uses (\ref{8}). Given (\ref{9}) we can write the following infeasible linear regression model,
\begin{equation}
y_{jt}-\mu_j = (Y_{-jt} - \mu_{-j}) ' \alpha_{j}^* + \eta_{jt}.\label{10}
\end{equation}
Now we relate (\ref{6}) using (\ref{8}). Then
\begin{equation}
\Theta_{j,-j}= - \Theta_{j,j} (\alpha_{j}^*)',\label{11}
\end{equation}
and then define $\tau_j^2:= E \eta_{jt}^2$. See that by (\ref{9})(\ref{10})
\[ E [y_{jt}- \mu_j] ^2 = (\alpha_{j}^*)' E [(Y_{-jt} - \mu_{-j}) (Y_{-jt} - \mu_{-j})'] \alpha_{j}^* + E [\eta_{jt}^2],\]
which implies by definitions $\Sigma_{j,j}:= E [ y_{jt}- \mu_j ]^2, \Sigma_{-j,-j}:= E [ (Y_{-jt} - \mu_{-j}) (Y_{-jt} - \mu_{-j})']$
\begin{eqnarray}
\tau_j^2 & = & \Sigma_{j,j} - (\alpha_{j}^*)' \Sigma_{-j,-j} \alpha_{j}^*
= \Sigma_{j,j}- \Sigma_{j,-j} \Sigma_{-j,-j}^{-1} \Sigma_{-j,j} \label{12a} \\
& = & 1/\Theta_{j,j},\label{12}
\end{eqnarray}
where we use (\ref{8}) for the second equality and (\ref{5}) for the third one. Clearly, also by (\ref{8})(\ref{12a})
\begin{equation}
\tau_j^2 = \Sigma_{j,j} - \Sigma_{j,-j} \alpha_{j}^*.\label{13}
\end{equation}
We can form the $j$ th row of precision matrix by using $j$ the main diagonal term, via (\ref{12})(\ref{13})
\begin{equation}
\Theta_{j,j} = \frac{1}{\Sigma_{j,j} - \Sigma_{j,-j} \alpha_{j}^*},\label{14}
\end{equation}
and the rest of that row, via (\ref{11})(\ref{12})
\begin{equation}
\Theta_{j,-j} = \frac{-(\alpha_{j}^*)'}{\tau_j^2}.\label{15}
\end{equation}
We start simplifying $\alpha_{j}^*$ formula in case of hidden factors as in (2)(3). The proof for (i) here is also a detailed version of the proof of Lemma 7 and (57) of \cite{bsmw22}. This is repeated to show that our scenario fits into their framework. Note that $\Sigma_f:= E (f_t - E f_t) (f_t - E f_t)'$. In that respect in (\ref{8}), by(2)-(3)
\begin{eqnarray}
\Sigma_{-j,j} = E [ (Y_{-jt} - \mu_{-j}) (y_{jt} - \mu_j) ]
&=& E \{[ A_{-j} (f_t - E f_t ) + U_{-jt}][(f_t - E f_t)' a_j + u_{jt}]\} \nonumber \\
& = & A_{-j} \Sigma_f a_j + A_{-j} E[ (f_t - E f_t) u_{jt}] \nonumber \\
&+& E [U_{-jt} (f_t- E f_t)'] a_j + E [U_{-jt} u_{jt}] \nonumber \\
& =& A_{-j} \Sigma_f a_j = \bar{A}_{-j} \bar{a}_j,\label{20}
\end{eqnarray}
by Assumption 1(iv). Then the same analysis in (\ref{20}) provides with $\bar{A}_{-j}:= A_{-j} \Sigma_f^{1/2}$,
\begin{equation}
\Sigma_{-j,-j}= E [ (Y_{-jt} - \mu_{-j}) (Y_{-jt} - \mu_{-j})'] = A_{-j} \Sigma_f A_{-j}' + \Sigma_{U,-j} = \bar{A}_{-j} \bar{A}_{-j}^{'} + \Sigma_{U,-j}.\label{21}
\end{equation}
Using Sherman-Morrison-Woodbury formula in p.19 of \cite{hj2013}
\begin{eqnarray}
\Sigma_{-j,-j}^{-1} & = & (A_{-j} \Sigma_f A_{-j}' + \Sigma_{U,-j})^{-1} \nonumber \\
& = & \Sigma_{U,-j}^{-1} - \Sigma_{U,-j}^{-1} A_{-j} (\Sigma_f^{-1} + A_{-j}' \Sigma_{U,-j}^{-1} A_{-j})^{-1} A_{-j}' \Sigma_{U,-j}^{-1} \nonumber \\
& = &
=\Sigma_{U,-j}^{-1} - \Sigma_{U,-j}^{-1} \bar{A}_{-j} ( I_K + \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j})^{-1} \bar{A}_{-j}' \Sigma_{U,-j}^{-1},\label{22}
\end{eqnarray}
where we use Assumption 3, $\Sigma_f$ is positive definite. Note also that
\begin{equation}
\bar{A}_{-j}^+ \bar{A}_{-j} = \bar{A}_{-j}' (\bar{A}_{-j}^+)' = I_K,\label{23}
\end{equation}
since $A_{-j}, \Sigma_f$ has full rank K by Assumption 3, so (135) of \cite{bsmw22} provides the last result (\ref{23}). Define $\bar{a}_j:= \Sigma_f^{1/2} a_j$ and (\ref{8})(\ref{20})
\begin{eqnarray}
\alpha_{j}^* &=& \Sigma_{-j,-j}^{-1} \bar{A}_{-j} \bar{a}_j
= \Sigma_{-j,-j}^{-1} [ \bar{A}_{-j} \{ \bar{A}_{-j}' (\bar{A}_{-j}^+)'\} \bar{a}_j] \nonumber \\
& = & \Sigma_{-j,-j}^{-1} [ \Sigma_{-j,-j} - \Sigma_{U,-j}] (\bar{A}_{-j}^+)' \bar{a}_j
= [ I_{p-1} - \Sigma_{-j,-j}^{-1} \Sigma_{U,-j}] (\bar{A}_{-j}^+)' \bar{a}_j,\label{24}
\end{eqnarray}
where we use (\ref{23}) for the second equality and for the third equality (\ref{21}) and $\bar{A}_{-j}$ definition. By using $\bar{G}_j$ definition before Lemma 1
and using (\ref{22}) on right side of (\ref{24}) for $\Sigma_{-j,-j}^{-1}$
\begin{eqnarray}
I_{p-1} - \Sigma_{-j,-j}^{-1} \Sigma_{U,-j} & = & I_{p-1} - [ \Sigma_{U,-j}^{-1} - \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{A}_{-j}' \Sigma_{U,-j}^{-1}] \Sigma_{U,-j} \nonumber \\
& = & \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{A}_{-j}'\label{25}
\end{eqnarray}
Use (\ref{25}) in (\ref{24})
\begin{equation}
\alpha_{j}^* = \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{A}_{-j}' (\bar{A}_{-j}^+)' \bar{a}_j
= \Sigma_{U,-j}^{-1} \bar{A}_{-j} \bar{G}_j^{-1} \bar{a}_j,\label{26}
\end{equation}
by (\ref{23}) for the second equality. One of the other elements in the precision matrix is the reciprocal of the main diagonal term, $\tau_j^2$ in (\ref{13}). We provide a formula for that in terms of hidden factor regression model, which will be new. In that respect, by (2)(3), Assumption 1
\begin{equation}
\Sigma_{j,j}:=E (y_{jt} - \mu_j)^2= a_j' \Sigma_f a_j + \sigma_j^2= \bar{a}_j' \bar{a}_j + \sigma_j^2.\label{27}
\end{equation}
Similarly, using (2)(3), Assumption 1
\begin{equation}
\Sigma_{j,-j}:= E (y_{jt} - \mu_j)(Y_{-jt} - \mu_{-j})' = \bar{a}_j' \bar{A}_{-j}'.\label{28}
\end{equation}
Then combine (\ref{26})(\ref{27})(\ref{28}) in (\ref{13})
\begin{equation}
\tau_j^2 = (\bar{a}_j' \bar{a}_j + \sigma_j^2) - \bar{a}_j' \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}
\bar{G}_{j}^{-1} \bar{a}_j.\label{28a}
\end{equation}
(i)-(ii). By (\ref{26}) and (\ref{28a}), and using them in (\ref{12})(\ref{15}) we have the desired result.
{\bf Q.E.D.}
{\bf Tail Inequalities}
Here we first provide a tail inequality for quadratic form of a mean-zero subgaussian random vector. This is Lemma 21 in \cite{bbsmw21}.
\begin{lemma}\label{lb1}
Let $\zeta$ be a zero mean $d$ dimensional $\gamma_e$ sub-Gaussian random vector. For all symmetric positive-semi definite matrices $H$, and all $t \ge 0$,
\[ P \left( \zeta' H \zeta > \gamma_e^2 (\sqrt{tr (H)} + \sqrt{2 \| H \|_2 t }
)^2 \right) \le exp(-t),\]
\end{lemma}
where $tr(H)$ represents trace of $H$, and $\gamma_e$ subgaussian random vector is defined in Assumption 4. Now, we provide a lemma for a lower bound inequality for singular value of a specific random matrix for RRE when $p-1>n$. This extends the zero factor mean in Proposition 14 of \cite{bsmw22} to a nonzero mean case below.
\begin{lemma}\label{lb2}
Under Assumptions $3^{*}(i)$, Assumption 3(ii), Assumption 4, with some $C>0$ large enough,
$re (\Sigma_{U,-j}) > C n $, for each $j=1,\cdots,p$, and $\tilde{U}_{-j}$ has independent entries, in the case of $p-1>n$
\[ P \left( \sigma_n^2 (Y_{-j}) \ge C tr (\Sigma_{U,-j})\right) \ge 1 - 4 exp (-C_p n) .\]
\end{lemma}
{\bf Proof of Lemma \ref{lb2}}.
So first we use (89) in \cite{bsmw22} and then equation below that on
p.42 of proof of Proposition 14 of \cite{bsmw22} but we replace Theorem 4.6.1 of \cite{v2019} in that proof with Theorem 5.3.9 of \cite{v2012}, (set $t=\sqrt{n}$) to take into account nonzero mean of factors.
The rest carries over from the proof of Proposition 14 of \cite{bsmw22}.{\bf Q.E.D.} \\
\setcounter{equation}{0}
\setcounter{lemma}{0}
\setcounter{table}{0}
\begin{center}
\uppercase{ Appendix C}
\end{center}
In this part C, we cover simulations and extra empirics without transaction costs and discussion of literature and hidden factors.
{\bf Simulations}
By (2) and (3) and Assumption 4, for each $t=1,...,n$, we have:
\[
y_{t}= A f_t + u_t,
\]
where $y_t = (y_{jt},Y_{-jt}')': p \times 1$ is the vector of excess return, $ A =(a_j,A'_{-j}): p\times K$ is the matrix of factor loadings, and $u_t=(u_{jt},U_{-jt})': p\times 1$ vector of errors. Let hidden-factors be normally distributed and $f_t \sim \mathcal{N}(\mu_f, \Sigma_f)$, where $\mu_f$ is the mean vector of the hidden factor, $K\times 1$ vector with each element equal to 0.5, and $\Sigma_f$ is the hidden-factor covariance matrix. Following \cite{bbsmw21}, we set the diagonal elements of $\Sigma_f$ as the $K-$ length sequence from 2.5 to 3 with equal increments. For example, if $K=3$, then $\Sigma_f = diag(2.5,2.75,3)$. The the off diagonal, $i,j$ th elements of the factor covariance matrix are defined as:
\[
[\Sigma_f]_{i,j} = (-1)^{i+j} (0.3)^{|i-j|} \min( [\Sigma_{f}]_{i,i}, [\Sigma_{f}]_{j,j} ).
\]
Each element of $A$ is drawn independently from the normal distribution $\mathcal{N} (0.5,1/\sqrt{K})$. The idiosyncratic error, $u_t$, is drawn from the normal distribution, $\mathcal{N}(0, \Sigma_u)$. In our design, we define $\Sigma_u:p\times p$ as a diagonal matrix with diagonal elements sampled from uniform distribution with $\text{Unif}(1,3) \times \log(p)$. With that we capture noisy setups in complex models and can understand the effect of noise on adaptive PCR and RRE with $p-1>n$ specifically.
The results for $K=3, 5$ (true number of hidden factors) are presented in Table \ref{tab:K=3}.
However note that we also use $K=20$ to see the effects of fitting larger number of factors when the true DGP has few factors.
We aim to address several questions through simulations. First, as described in the theorems, we investigate whether the estimation errors from our methods decrease when $\gamma = p/n$ is fixed and both $p$ and $n$ jointly increase \footnote{$\gamma=p/n=0.5$ in low-dimensional set up and $\gamma = p/n=1.5$ in high-dimensional set up }. Second, we compare the performance of our new methods with existing methods. Third, we examine the impact of increasing the number of hidden factors, and also see whether there is a symptom of double descent in RRE estimator. We consider estimation error for the precision matrix in Table. We consider the nonlinear shrinkage method of \cite{lw2017}(NL-LW), POET method from \cite{fan2013}, and the nodewise (NW) method from \cite{caner2019}. Our estimators are: RRE, PCR-3F, PCR-5F, PCR-20F and PCR-Adaptive, in which they are RRE estimator, Principal component with fixed 3, 5, or 20 hidden factors, and an adaptive method to select the number of hidden factors, respectively. We do not include \cite{caner2022} residual nodewise since it only uses observed factors. Table \ref{tab:K=3} shows the simulation results.
We ran 100 iterations in each simulation set up. First, the table shows that our method achieves consistency as stated in Corollary 4 for adaptive PCR. Analyzing Table \ref{tab:K=3} with $K=3$, we observe that for $p=n/2$, with $n=100$, our Adaptive-PCR has an estimation error at 0.0583. This error declines to 0.0251 when $p=n/2$ and $n=400$.
Next, we consider which method achieves the smallest estimation errors. Our Adaptive-PCR dominates all other techniques except fixed PCR-K.
As a side note to all of this, in RRE estimator, we see that estimation error for the low dimensional case is always higher than the double of high dimensional error scenario. This can be a symptom of double descent in RRE.
\begin{table}[ht!]
\caption{Estimation Errors\label{tab:K=3}}
\begin{center}
\begin{tabular}{r|llcllcll}
\multicolumn{9}{c}{K=3} \\ \hline \hline
\multicolumn{1}{c}{\bfseries Estimator}&\multicolumn{2}{c}{\bfseries n=100}&\multicolumn{1}{c}{\bfseries }&\multicolumn{2}{c}{\bfseries n=200}&\multicolumn{1}{c}{\bfseries }&\multicolumn{2}{c}{\bfseries n=400}\tabularnewline
\cline{2-3} \cline{5-6} \cline{8-9}
\multicolumn{1}{r}{}&\multicolumn{1}{c}{p=n/2}&\multicolumn{1}{c}{p=3n/2}&\multicolumn{1}{c}{}&\multicolumn{1}{c}{p=n/2}&\multicolumn{1}{c}{p=3n/2}&\multicolumn{1}{c}{}&\multicolumn{1}{c}{p=n/2}&\multicolumn{1}{c}{p=3n/2}\tabularnewline \hline
NW& 0.0752& 0.1016&& 0.0617& 0.0843&& 0.0521& 0.0669\tabularnewline
POET&47.0373&76.4717&&59.6223&101.4291&&84.7820&150.7152\tabularnewline
Ledoit-Wolf&40.4218&69.1745&&55.2778& 97.6674&&82.5374&150.9011\tabularnewline
RRE& 0.4583& 0.2041&& 0.3858& 0.1763&& 0.3369& 0.1562\tabularnewline
PCR-3F& 0.0583& 0.0738&& 0.0385& 0.0464&& 0.0251& 0.0302\tabularnewline
PCR-5F& 0.0641& 0.0803&& 0.0408& 0.0489&& 0.0263& 0.0313\tabularnewline
PCR-20F& 0.1276& 0.1463&& 0.0637& 0.0712&& 0.0361& 0.0401\tabularnewline
PCR-Adaptive& 0.0583& 0.0738&& 0.0385& 0.0464&& 0.0251& 0.0302\tabularnewline \hline
\multicolumn{9}{c}{K=5}\\ \hline \hline
NW& 0.0778& 0.1042&& 0.0683& 0.0888&& 0.0578& 0.0728\tabularnewline
POET&44.6537&72.7433&&58.4469&98.6137&&78.9982&135.5738\tabularnewline
Ledoit-Wolf&38.8199&66.0089&&54.5050&94.8428&&76.9331&135.0054\tabularnewline
RRE& 0.4307& 0.1752&& 0.3840& 0.1577&& 0.3360& 0.1417\tabularnewline
PCR-3F& 0.0564& 0.0686&& 0.0488& 0.0593&& 0.0452& 0.0524\tabularnewline
PCR-5F& 0.0580& 0.0732&& 0.0404& 0.0503&& 0.0259& 0.0302\tabularnewline
PCR-20F& 0.1193& 0.1397&& 0.0631& 0.0725&& 0.0355& 0.0395\tabularnewline
PCR-Adaptive& 0.0559& 0.0732&& 0.0403& 0.0503&& 0.0259& 0.0302\tabularnewline
\end{tabular}\end{center}
\end{table}
{\bf Extra Empirics}
Here we provide the results of the empirics without transaction costs in Tables C.2-C.4. $SD$ is the standard deviation of the portfolio, and $Sharpe$ is the Sharpe ratio. Our methods still achieve the best Sharpe Ratio,
turnover, standard deviation and return in Tables C.2-C.3 low dimension and moderate dimension cases. Table C.4, again our best method dominates the others in Return, Sharpe Ratio and Standard Deviation. In Turnover, nodewise has the best record.
The first 6 rows represent the models in the literature, and the rest is our models.
The bold faced numbers represent the winner in Sharpe Ratio.
\newpage
\begin{table}[h!]
\caption{Monthly Portfolio Performance of 10 Stocks, $ n_I=240$, $ n-n_I=108$\label{tab:240-10}}
\begin{center}
\begin{tabular}{rcllll}
\multicolumn{1}{c}{\bfseries Estimator}&\multicolumn{1}{c}{\bfseries }&\multicolumn{4}{c}{\bfseries Maximum Sharpe Ratio}\tabularnewline
\multicolumn{1}{r}{}&\multicolumn{1}{c}{}&\multicolumn{1}{c}{Return}&\multicolumn{1}{c}{SD}&\multicolumn{1}{c}{Sharpe}&\multicolumn{1}{c}{Turnover}\tabularnewline
{\bfseries without transaction cost}&&&&&\tabularnewline
RNW-GIC-SF&&0.0101&0.0526&0.1913&0.0953\tabularnewline
RNW-GIC-3F&&0.0101&0.0519&0.1945&0.0889\tabularnewline
RNW-GIC-5F&&0.0102&0.0510&0.2002&0.0876\tabularnewline
NW-GIC&&0.0106&0.0505&0.2093&0.0888\tabularnewline
POET&&0.0104&0.0507&0.2051&0.0700\tabularnewline
Ledoit-Wolf&&0.0100&0.0522&0.1914&0.0927\tabularnewline\hline \hline
Inter-RRE&&0.0104&0.0530&0.1957&0.0929\tabularnewline
PCR-1F&&0.0109&0.0490&\textbf{0.2221}&0.0518\tabularnewline
PCR-3F&&0.0105&0.0492&0.2142&0.0576\tabularnewline
PCR-5F&&0.0106&0.0501&0.2125&0.0948\tabularnewline
PCR-7F&&0.0105&0.0507&0.2075&0.0795\tabularnewline
PCR-Adaptive&&0.0108&0.0489&0.2211&0.0514\tabularnewline
\end{tabular}\end{center}
\end{table}
\begin{table}[h!]
\caption{Monthly Portfolio Performance of 200 Stocks, $ n_I=240$, $ n-n_I=108$\label{tab:240-200}}
\begin{center}
\begin{tabular}{rllllcllll}
\multicolumn{1}{c}{\bfseries Estimator}&\multicolumn{1}{c}{\bfseries }&\multicolumn{4}{c}{\bfseries Maximum Sharpe Ratio}\tabularnewline
\multicolumn{1}{r}{}&\multicolumn{1}{c}{}&\multicolumn{1}{c}{Return}&\multicolumn{1}{c}{SD}&\multicolumn{1}{c}{Sharpe}&\multicolumn{1}{c}{Turnover}\tabularnewline
{\bfseries without transaction cost}&&&&&\tabularnewline
RNW-GIC-SF&& 0.0053&0.0442& 0.1205&0.2761\tabularnewline
RNW-GIC-3F&& 0.0067&0.0447& 0.1496&0.3171\tabularnewline
RNW-GIC-5F&& 0.0072&0.0484& 0.1483&0.3996\tabularnewline
NW-GIC&& 0.0093&0.0429& 0.2169&0.1517\tabularnewline
POET&& 0.0057&0.0430& 0.1336&0.1989\tabularnewline
Ledoit-Wolf&& 0.0048&0.0521& 0.0928&0.7320\tabularnewline\hline\hline
Inter-RRE&&-0.0018&0.1168&-0.0158&6.1638\tabularnewline
PCR-1F&& 0.0075&0.0376& 0.2002&0.1203\tabularnewline
PCR-3F&& 0.0082&0.0382& 0.2151&0.2860\tabularnewline
PCR-5F&& 0.0081&0.0557& 0.1462&1.0298\tabularnewline
PCR-7F&& 0.0128&0.0433& \textbf{0.2964}&1.0439\tabularnewline
PCR-Adaptive&& 0.0081&0.0435& 0.1865&0.4231\tabularnewline
\end{tabular}\end{center}
\end{table}
\begin{table}[h!]
\caption{Monthly Portfolio Performance of 357 Stocks, $ n_I=240$, $ n-n_I=108$\label{tab:240-357}}
\begin{center}
\begin{tabular}{rllllcllll}
\multicolumn{1}{c}{\bfseries Estimator}&\multicolumn{1}{c}{\bfseries }&\multicolumn{4}{c}{\bfseries Maximum Sharpe Ratio}\tabularnewline
\multicolumn{1}{r}{}&\multicolumn{1}{c}{}&\multicolumn{1}{c}{Return}&\multicolumn{1}{c}{SD}&\multicolumn{1}{c}{Sharpe}&\multicolumn{1}{c}{Turnover}\tabularnewline
{\bfseries without transaction cost}&&&&&\tabularnewline
RNW-GIC-SF&&0.0046&0.0432&0.1068&0.2857\tabularnewline
RNW-GIC-3F&&0.0057&0.0440&0.1297&0.3580\tabularnewline
RNW-GIC-5F&&0.0056&0.0461&0.1209&0.4507\tabularnewline
NW-GIC&&0.0093&0.0422&0.2202&0.1333\tabularnewline
POET&&0.0058&0.0424&0.1364&0.2009\tabularnewline
Ledoit-Wolf&&0.0015&0.0534&0.0280&0.9073\tabularnewline\hline\hline
Inter-RRE&&0.0106&0.0453&\textbf{0.2338}&0.2725\tabularnewline
PCR-1F&&0.0073&0.0364&0.2004&0.1349\tabularnewline
PCR-3F&&0.0054&0.0366&0.1470&0.2798\tabularnewline
PCR-5F&&0.0056&0.0528&0.1056&1.6010\tabularnewline
PCR-7F&&0.0095&0.0506&0.1870&1.8931\tabularnewline
PCR-Adaptive&&0.0085&0.0420&0.2020&0.2608\tabularnewline
\end{tabular}\end{center}
\end{table}
\clearpage
{\bf Discussion of Literature}
Several studies in the statistics literature have studied precision matrix estimation. An excellent survey is by \cite{crz2016}.
In Section 3 of their paper, they provide
two major approaches to precision matrix estimation. The first approach is neighborhood-based and closely related to nodewise regression, where a lasso regression is performed for each variable against all others. Alternatives to lasso, such as the Dantzig selector, can also be used within this framework (see \cite{cl2011}, \cite{caner2019}). The second major approach is based on penalized likelihood estimation. As shown by \cite{blp2020}, lasso-type penalties in this context can induce shrinkage bias and lead to overly sparse precision matrix estimates, with many zero entries in each row. In a penalized likelihood based setting, they reformulate the problem as a mixed-integer optimization problem, which is solvable using convex optimization techniques. This approach produces better results than lasso based approaches. Sparsity of precision matrix is used in these two main approaches.
A distinct approach, commonly used in finance and statistics, derives the precision matrix by inverting the covariance matrix using the Sherman–Morrison–Woodbury (SMW) formula. In this framework, sparsity is assumed in the covariance matrix of the idiosyncratic errors, while asset returns are modeled through a factor structure (see \cite{fan2013}, \cite{caner2022}). Our approach is different from the
approaches discussed in the literature. First of all, we are not assuming exact sparsity in the precision matrix of outcomes. We are also not using SMW formula to
invert the covariance matrix.
This allows us to bypass imposing a structure
on the outcomes utilized in the covariance inversion. Instead we use a new
precision matrix formula without sparsity and rely on hidden factors to generate each
outcome. This effectively ensures that we do not benefit from the SMW matrix inversion formula. To be clear, we use model estimation-free dense estimation, but consistency in high dimensions is achieved through hidden factor based dimension reduction without a need to estimate factors or identify them.
There are also other technical and conceptual differences that are worth highlighting.
First, our proposed class of
estimators is general and in addition to the PCR based ones it also includes the ridgeless regression estimator (RRE). Second, our signal assumption is much
weaker as we do not need the minimum signal to grow
at the same rate as the number of assets in the portfolio. Moreover, as the root of our approach resembles a finite sample analysis, it also allows to explicitly tie all the estimation errors to signal-to-noise
ratio thereby making its importance in the rate of convergence estimates clearly visible.
The precision matrix estimation is different than the risk analysis of a single outcome-asset. We also provide an explicit connection in our estimation error upper bounds to signal-to-noise ratio. The results in subcases are different also in a risk analysis versus precision matrix estimation. \cite{bbsmw21}, \cite{bsmw22} show that in certain very high dimensional cases, ridgeless regression (RRE) can perform better than adaptive PCR in terms of risk asymptotically. However, in case of estimation errors of rows of precision matrix errors (in $l_2$ norm) we show adaptive PCR can perform better than RRE even in certain low-moderate dimension cases. These are due to different proofs for risk versus precision matrix estimation.
{\bf Hidden Factors}
The reason we use factor models is for dimension reduction. First of all, we show that precision matrix estimation is related to linear regression and that can be related to hidden factors. Each economic or financial variable can be related to latent factors and this can be thought as confounding variables as in causality related literature such as in \cite{kv2023}, \cite{ap2009}, \cite{ir2015}.
Also the true variables of interest may be measured with errors, and latent factors can represent the "true" variables rather than the reported ones, especially in emerging markets. Variable such as culture, sentiment, market can be extremely valuable in certain macro, financial data sets. These hidden factors in a linear model can be used to explain international stock returns such as in Singapore, Malaysia, USA as seen in \cite{bn2004}.
Another key aspect of using hidden factors is their effectiveness as a tool for dimension reduction in high-dimensional econometrics. This stems from the fact that the number of observed outcomes typically exceeds the number of underlying hidden factors. In our paper, we allow {\it both} the number of outcomes and the number of hidden factors to grow with the sample size.
\bibliographystyle{chicagoa}
\bibliography{interpolation}