Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
147,290 characters · 17 sections · 117 citation commands
Model-Estimation-Free, Dense, and High Dimensional Consistent Precision Matrix Estimators
\def\spacingset#1{ {#1}} \spacingset{1}
{\bf{Keywords}}: Interpolation, Double Descent, Latent Space, Factor Models, Precision Matrix .
\spacingset{1.9}
Due to their usage in high dimensional portfolio formation in finance, network analysis in economics and debiased policy coefficient estimation in cross section-causality analysis, precision matrix estimation has been among the few cornerstone topics in statistics and finance in recent years.
We propose a general class of estimators that are simultaneously {\it model estimation-free}, {\it dense}, and {\it consistent}. Existing literature can handle separately one of these properties, but no precision matrix estimator can handle all of them {\it jointly}. Such a joint handling is highly desired as it is naturally expected to lead to better empirical performances.
Our design sharply differentiates our approach from classical shrinkage, factor-model, and nodewise-regression methods. For example, unlike shrinkage type of estimators, e.g., lw2004b, ours converge to precision matrix, not to a modified version of it. Compared to dense-model estimation free nonlinear shrinkage estimator of lw2017, our estimator is high dimension consistent. Differently from the hidden factor model based estimator of , e.g., fan2013, we do not have to impose identification, estimation, and rotation which allows for model free estimation. Finally, in sharp contrast to the nodewise regression, e.g., caner2019, mein2006, we do not impose arbitrary sparsity assumption on the rows of precision matrix nor do we use penalized estimators that creates shrinkage bias. Our main contributions are summarized below.
(i). We introduce a general class of consistent-non-sparse and unpenalized precision matrix estimators. Linking a set of general dense-unpenalized estimators to precision matrix formula is new and novel. This is done without shrinkage, penalization or factor estimation. Hidden factor analysis is used only for achieving consistency via dimension reduction. (ii). We derive non-asymptotic upper bounds for estimation errors that allow us to see the effects of random variable that captures model complexity, number of confounding factors, and signal-to-noise ratio.(iii). Our class encompasses a broad range of precision matrix estimators, including ridgeless regression. For this estimator, we show the double descent due to the increasing size of the signal, when the number of outcomes exceeds the number of observations.(iv). Theoretical findings are complemented with very good empirical performance in S&P 500.
.
Several studies in the statistics literature have studied precision matrix estimation. An excellent survey is by crz2016 which shows neighborhood and penalization based approaches. We refer the readers to that survey for the literature and appendix that we have. A distinct approach, commonly used in finance and statistics, derives the precision matrix by inverting the covariance matrix using the Sherman–Morrison–Woodbury (SMW) formula. In this framework, sparsity is assumed in the covariance matrix of the idiosyncratic errors, while asset returns are modeled through a factor structure (see fan2013, caner2022). Our approach is different from the approaches discussed in the literature. First of all, we are not assuming exact sparsity in the precision matrix of outcomes. We are also not using SMW formula to invert the covariance matrix.
Another approach is nodewise based. Note that nodewise regression based estimator of caner2019 uses sparsity of precision matrix of outcomes. Then residual nodewise regression estimator of caner2022 crucially depends on observed factors, and imposes sparse precision matrix of errors. Technique used in those papers cannot extend to hidden factors since observed factor structure in caner2022 uses OLS estimation which is not applicable to more complicated estimation problems such as hidden factor structure. Recently, cd2025 developed a new deep learning based precision matrix estimator. However, the estimator is consistent only when number of outcomes is less than or equal to sample size, which limits applications. We also tie our results to signal-to-noise ratio aspect which is not available in the studies of caner2019, caner2022. Our method does not impose sparsity like nodewise regression of mein2006. Nodewise regression uses an $l_1$ norm based penalization to form the rows of the precision matrix. Doing so, the underlying assumption is the imposition of sparsity in the precision matrix which may be interpreted as a strong assumption. An excellent paper by fan2013 provides a statistical high-dimensional analysis of covariance matrix of outcomes via hidden factors based estimation. Similar to us, it also relies on hidden factors but uses a thresholding based principal components estimator for the covariance matrix, and then inverts this through SMW formula. The key difference is that our estimator is model estimation-free and uses hidden factors for consistency proof only.
Our paper differs from bbsmw21 and bsmw22 in a fundamental way. These papers analyze general linear class of estimators in hidden factor models in linear regression model and is only interested in risk of the estimators. Their papers consider only one outcome variable and how it can be predicted with this new general class of estimators. Our analysis on the other hand, considers the precision matrix estimation of outcome variable in linear regression via hidden factors. In our paper, a novel idea is to use a linear linkage between outcome variables to obtain precision matrix estimator. One of the main difficulties, theoretically, is that the estimation of the main diagonal terms in the precision matrix estimation. The proof technique and ideas in bbsmw21, bsmw22 do not simply extend to this case.
Another set of estimators are shrinkage based ones. For example lw2004b proposes a linear shrinkage estimator for covariances in iid data setting. This is to solve the invertibility problem of sample covariance matrix. Their estimator for covariance is dense, model-free hence making a major contribution. However, unlike our estimators, the estimator converges to a modified version of the covariance consistently not to the covariance matrix itself. lw2017 propose a non-linear shrinkage estimator. This estimator specifically analyzes out-of-sample variance loss in iid data setting, and develop a modified covariance matrix estimator that increases the small eigenvalues, and decreases the large ones. It is a model-free dense estimator but the out-of-sample variance loss limit is different from the population one. Unlike our estimator, the precision matrix estimator in their paper is not consistent. But as mentioned, their main concern is different than consistency in high dimensions. More discussion about literature and hidden factors are in Appendix C.
We consider notation now. Define $C,c $ as generic positive constants. Note that transpose of any matrix $Z: d \times l$ is represented by $Z'$, and transpose of a column vector, $z$ is represented by $z'$. Let $\| Z \|_2, \| Z \|_F$ represent the spectral norm and Frobenius norm of matrix $Z$ respectively, and $\|z\|_2$ the Euclidean norm of a vector $z$. Let $\| Z \|_{l_{\infty}}:= \max_{1 \le j \le d} \| Z_{j}\|_1$ be the maximum row sum norm, with $Z_{j}: l \times 1$ as the $j$ th row among $d$ rows in matrix Z, written in column format. The row format for $j$ th row of $Z$ is $Z_j': 1 \times l$. Let $\| Z \|_{\infty}:= \max_{i,j} |Z_{ij}|$, with $Z_{ij}$ representing $i,j$ cell of matrix Z. Denote the $k$ th largest singular value of $Z$ matrix as $\sigma_k (Z)$ with $k=1$ being the largest singular value. Let $Z^{+}$ represent the Moore-Penrose inverse of matrix Z. Let $\lambda_k (M)$ represent the $k$ th largest eigenvalue of $M: d \times d$, a square matrix, except $k=1, k=d$. However, for ease of understanding, the largest eigenvalue of $M$ is denoted as $Eigmax (M)$ and the minimum eigenvalue of $M$ is $Eigmin (M)$. $tr(M)$ represents the trace of matrix $M$. Section (ref) proceeds with the idea of dimension reduction through hidden factors to achieve high dimensional consistency in the proofs. Section (ref) introduces the estimators. Section (ref) has consistency results. Section (ref) provides empirical evidence. Conclusion shows future possible extensions of our work. Simulations, some of the empirics and all of the proofs, discussion about literature and hidden factors are relegated to Appendix.
In our setup we have $p$ outcome variables, and they are explained by $K$ common hidden factors to ensure dimension reduction, with $K <<p$. This will help in asymptotics and achieving consistency, however our estimators will be model estimation-free and will not use hidden factors estimators. Our estimators will only use outcome variables and this will be clear in the next section. Specifically, our estimators are defined in ((ref))((ref)). Let $Y$ represent the $n \times p$ matrix of outcomes, each row representing the $t$ th time period, $t=1,\cdots,n$, and $j=1,\cdots,p$ representing the outcomes. $F$ represents $n \times K$ matrix of hidden factors, where each row, $t=1,\cdots, n$ represents a time period, and columns represent the hidden factors, $k=1,\cdots, K$. $A$ represents $p \times K$ matrix of factor loadings, and $U: n \times p$ as the error matrix. We have the following compact form of the linear hidden factor structure for dimension reduction (which will be decomposed via rows of Y into two equations immediately below in ((ref))-((ref))).
In accordance with our analysis, we subdivide all $p$ outcomes into $j$ th one and all the remaining ones. In that respect, start by defining each row of the outcome as $y_t: p \times 1$ (represents each row, in column form, whereas $y_t':1 \times p$ is the row vector form) of $Y$ above, and $f_t:K \times 1$ represents each row of $F$, and $u_t: p \times 1$ is the $t$ th row of U. We decompose $A$ into $a_j: K \times 1$ factor loading vector for outcome j, and $A_{-j}: p-1 \times K$ as factor loading matrix for all other outcomes except $j$. We model $y_{jt}$ as the $j$ th outcome variable at time period $t=1,\cdots,n$. We define all the outcomes except $j$ th outcome as $Y_{-jt}: p-1 \times 1$ vector at time $t$. We decompose ((ref)) into $j$ th outcome and the rest. In the $t$ th row of $Y$ in ((ref)), $j$ th outcome can be written as
and we write all the outcomes except from the $j$ th one, in $t$ th row as
We use ((ref))((ref)) in our proofs rather than ((ref)), and in this way our proofs are clear and tractable. Common hidden factors affect the $j$ th outcome in ((ref)) as well as all remaining outcome variables in ((ref)). Let $u_{jt}, U_{-jt}$ represent the zero-mean errors for the $j$ th outcome and all other remaining outcomes, respectively. In vector form, the j-th column of $Y$ in ((ref)) can be written as
where $y_j: n \times 1$, $F: n \times K$ with each row as $f_t'$. Then the remainder, except for the $j$ th column, of Y is:
where $Y_{-j}: n \times p-1$ matrix, and $U_{-j}: n \times p-1$ matrix too. As we will see later on, the vector form representation given in ((ref))((ref)) will turn out to be beneficial in our proofs as well.
Define the variance of $j$ th error as $var(u_{jt}) = \sigma_{j}^2 \ge c >0$,where $c$ is a positive constant at time $t$, for $j=1\cdots,p$. Define the variance matrix of errors except the $j$ th one as $var (U_{-jt}) = \Sigma_{U,-j}: p-1 \times p-1$, for $j=1,\cdots, p$ at time $t$. Define the covariance matrix of factors as $\Sigma_f:= cov (f_t): K \times K$. Define $y_t=( y_{jt}, Y_{-jt}')'$ as the re-ordered $t$ th row of $Y$, and $\mu_j:= E y_{jt}: 1 \times 1$, $\mu_{-j}:= E Y_{-jt}: p-1 \times 1$, and $\mu:= E y_t: p \times 1$ at time $t$. Let the covariance matrix of outcomes be $\Sigma:= E (y_t - \mu) (y_t - \mu)'$. Assumption (ref) below will show iid nature of our data across $t=1,\cdots,n$. Define the precision matrix as $\Theta:=\Sigma^{-1}$. We provide Assumptions next.
Assumption 1 is a standard convention used in hidden factor literature (see, e.g., bbsmw21 and bsmw22.\footnote{Observe that (i)-(iii) implies that rows of F, Y, and U are iid.} Note also, that independence of errors implies $Eu_{jt}U_{-jt} = 0$ and $ u_t = (u_{jt},U_{-jt}')'$ which is a sparsity restriction on the covariance matrix of errors. This is another assumption often present in the literature (see, e.g., fan2008) and amounts to a diagonal error covariance matrix. Relaxing this part of the assumption is certainly an interesting topic well worth of further exploring. Relaxing the iid structure of data for financial applications is a good idea, however at this point our proofs, via Assumption 4 below as well, use subgausssian-iid based quadratic form concentration inequalities in high dimensional probability literature. Having a different structure in the proofs and solving time series nature is a key issue that we can follow up in future directions tied to financial applications.
We are not imposing sparsity on the links between outcome variables as done in nodewise regression context.
Assumption 2(i) is also standard in high dimensional statistics literature. It allows for non-singular noise matrices, large noise magnitudes. Since $\Sigma_{U,-j}$ is diagonal and diagonals have error variances, we use a common upper bound for a single variance in (i) and maximum variances over errors in (ii). This large positive $\Gamma$ is very much reflective of financial markets.
With $C_p>2$, we can obtain $p>n$ and we cover high dimensional cases. In case of $0<C_p \le 2$, we cover low dimensional cases, $p \le n$. This assumption also allows $p = \gamma n$, with $\gamma \in [0, \infty)$. We allow the spectral norm of $A_{-j}: p-1 \times K$ matrix to be diverging in $n$, reflective of the high dimensional nature of $A_{-j}$, where both dimensions are allowed to grow. Specifically in (ii), we allow for pervasive factors with $\max_{j} \| A_{-j} \|_2 = C \sqrt{p}$, $C $ being a generic positive constant as well as weak factor structures if $\max_j \| A_{-j} \|_2 = O (d_{1n})$, with $d_{1n}/\sqrt{p}\to0$ with $p \to \infty, n \to \infty$. Hence our assumption is a weak one covering several possibilities.
Assumption (ref) is standard assumption used in bbsmw21 bsmw22. This assumption is needed for quadratic form high dimensional probability inequalities. It should be noted that we also relax the non-zero factor mean assumption used in earlier literature in bbsmw21, and bsmw22. Define the uniform version of the signal to noise ratio
where as in bsmw22, they have a pointwise version
with $\lambda_K (A_{-j} \Sigma_f A_{-j}' )$ representing the $K$ th largest eigenvalue of $A_{-j} \Sigma_f A_{-j}': p-1 \times p-1$ matrix, and that is the smallest positive eigenvalue of that matrix since $p-1 > K$, and $\Sigma_f: K \times K$ matrix. We have a discussion about the role of the signal-to-noise ratio in a specific scenario. Let us assume a weaker factor structure in Assumption (ref)(ii), which is $\max_j \| A_{-j}\|_2 = O (d_{1n})$. Since $A_{-j}: p-1 \times K$ for all matrices $j=1,\cdots,p$, via Assumption (ref), if the largest eigenvalue of $A_{-j}' A_{-j}$ grows at the same rate $K$-th largest eigenvalue, we can assume $\lambda_K (A_{-j}' A_{-j}) \asymp d_{1n}^2$ and since we impose $ max_{1 \le j \le p } \| \Sigma_{U,-j} \|_2 < \Gamma$ by Assumption (ref) then \[ \lambda_K (A_{-j} \Sigma_f A_{-j}') \ge Eigmin (\Sigma_f) \lambda_K (A_{-j} A_{-j}') \ge c \lambda_K (A_{-j}' A_{-j}) \asymp d_{1n}^2 .\] Utilizing ((ref)) and relying on Assumption (ref) we obtain a useful signal-to-noise ratio lower bound formulation
Depending on how fast/slow $d_{1n}$ grows the signal-to-noise ratio lower bound in ((ref)) can fluctuate between fairly small and fairly large quantities. Assumption (ref) below covers diagonal estimation of precision matrix. Note that $\bar{r}, \bar{\Psi}, \bar{\eta}$ are three random sequences that are explained in detail before Theorem 1, and they represent the complexity, bias, and size of the signal respectively. In special cases of RRE, and adaptive PCR, we will show non-random upper bounds for these three key quantities. To state Assumption (ref) below, we find it convenient to introduce rate of convergence, $r_{w1}$ which is used for off-diagonal term estimation.
Define the relation between the factor loadings of $j$ th asset with all the others as $d_{2n}:= \max_{1 \le j \le p} \| a_j' \Sigma_f A_{-j}'\|_2$.
While the meaning of Assumption (ref) will become clearer when we get to our final results, a few introductory remarks regarding the origin of the assumption are in place. First, Assumption (ref)(i) is a technical assumption that facilitates the proof of Theorem (ref) for the finite sample results. Assumption (ref)(ii) relates to the tradeoff between number of factors and time span. In particular, it points out that the number of factors should be much smaller than the time span. Assumption (ref)(iii) involves stochastic terms and relates to consistency and rate of convergence of the off-diagonal terms estimates. As we will see later on, the role of Assumption (ref)(iii) will be discussed separately for adaptive PCR and RRE estimators. But to provide some basic intuition in a simple subcase, let $K=1$, and $\| \Sigma_{U,-j}\|_2 = I_{p-1}$, with $var(f_t)=1$, and set $r_{w1} := 1 / \sqrt{\xi}$ \[ d_{2n} r_{w1} = \frac{\max_{j} \sqrt{\sum_{i \neq j} (A_{i,-j} a_j)^2}} {\min_j \sqrt{\sum_{i \neq j} A_{i,-j}^2 }} \to 0 ,\] where $A_{i,-j}$ is $i$ th element of vector $A_{-j}$ (see that this is a vector since $K=1$). So this is basically factor loadings correlations among themselves declining to zero.
We propose estimating the rows of the precision matrix of asset returns with our estimator. To show consistency of our estimators in high dimensional context we need to start with a precision matrix formula that can be tied to hidden factors. One of the main obstacles is establishing relation between the matrix inversion and linear regression within hidden factor context. The difficulty is that formula for main diagonal terms and off-diagonal terms of the precision matrix is different and this type of approach tying hidden factors and general linear class of estimators with precision matrix estimation has not been done before. Define the following parameter vector which will form a key term in precision matrix formula: \[ \alpha_j^*:= argmin_{\alpha_{j} \in R^{p-1}} E [ (y_{jt} - \mu_j) - (Y_{-jt} - \mu_{-j})' \alpha_{j}]^2.\] This definition shows that we can relate to linear regression context. Note that, the definition of $\alpha_j^*$ is not coming from the hidden factor model directly in ((ref))((ref)). We also need the following definitions before the next Lemma. Let $\Sigma_f = \Sigma_f^{1/2} \Sigma_f^{1/2}$ since $\Sigma_f$ is a positive definite matrix. Set $\bar{A}_{-j}:= A_{-j} \Sigma_f^{1/2}$, and $\bar{a}_j:= \Sigma_f^{1/2} a_j$, and $ \bar{G}_j:= I_K + \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}.$ Next, we show how they relate to hidden factor structure. Let $\Theta_{j,j}$ represent the main $j$ th diagonal term, and $\Theta_{j,-j}$ represent the $j$ th row without $j$ th element. One key result in Lemma (ref) is that we will not impose arbitrary zero-exact sparsity restrictions on $\alpha_j^*$ as done in nodewise regression in mein2006, caner2019.
Connecting further to the existing literature, one should note that bbsmw21 propose linear regression coefficient estimation and analyze the risk of an asset in a hidden factor model. However, neither the precision matrix estimation nor the above formulae are discussed therein. Lemma 1 and Theorem 1 below show how we manage to find a way to base the estimation on hidden factors but without actually imposing any identification conditions on hidden factors. In that respect for each $j=1,\cdots,p$, we define
where $(.)^+$ is the Moore-Penrose inverse of a matrix, and $\hat{B}: p-1 \times q$ with $q \le p-1$ matrix, and the second equality is by (134) of bsmw22. Dimension $q$ will be tied to specific estimators, and immediately below, we give examples about that. We let $\hat{B}$ be a deterministic matrix or a matrix that depends on $Y_{-j}$. Our estimator is based on analysis of precision matrix in Lemma (ref).
Note that one of the main contributions of this paper is to link estimators ((ref)) with $\alpha_j^*$ which is related to the matrix inversion. With different values of $\hat{B}$ we obtain different estimators. To see that, $\hat{B}= I_{p-1}$ is the ridgeless regression estimator (RRE) which is analyzed also after this section separately, so $q=p-1$. In case of adaptive PCR, $\hat{B}= V_{1,k_j}$ with $V_{1,k_j}$ being the $p-1 \times k_j $ matrix where the columns are the eigenvectors of $Y_{-j}' Y_{-j}/n$ matrix corresponding to the largest $k_j$ eigenvalues, and $q=k_j$, $j=1,\cdots,p$. We do not subscript $\hat{B}$ with $j$ to simplify the notation. \footnote{An optimal $\hat{B}$ may be analyzed but without knowing its structure it may be difficult and it is beyond current scope of the paper}. Let us define the reciprocal of the main diagonal terms in the precision matrix based on (B.11)\footnote{Note that only in case of RRE estimator as the subcase when $p-1>n$ the estimator simplifies and we benefit from that expression.}
Then the $j$ the main diagonal term estimator in the precision matrix is:
and the $j$ th row of the precision matrix estimate without $j$ th element (off-diagonal terms in the row) is, via (B.13)
The $j$ th row of the estimate, to give an example, at $j=1$ will be $\tilde{\Theta}_1':= ( \frac{1}{\tilde{\tau}_1^2}, \frac{-\tilde{\alpha_{1}}'}{\tilde{\tau}_1^2}).$
In this section we provide first upper bounds on $l_2$ norm errors for each regression coefficient estimate, that is in ((ref)). Next, we consider ((ref)), and obtain upper bounds for the estimation errors.
In this section, our aim is to first obtain an upper bound for $ \max_{1 \le j \le p } \| \tilde{\alpha}_j - \alpha_{j}^* \|_2,$ which is essential in getting consistency of estimates of rows of the precision matrix. We explain three random terms that will be essential understanding our estimation error bounds. Let $P_{\hat{B}}$ be the projection matrix onto the range $\hat{B}$, i.e. $P_{\hat{B}}= \hat{B} [ \hat{B}' \hat{B}]^+ \hat{B}' = \hat{B} \hat{B}^+$, and the last equality is by Moore-Penrose matrix inverse rules such as (134) of bsmw22. Define the rank of $Y_{-j} P_{\hat{B}}$ as
This term reflects the complexity, and increases the variance of the estimation of rows of the precision matrix, as we show below in Theorem (ref). Define
which is the $\hat{r}_j$- th largest singular value (squared, scaled by n) of $Y_{-j} P_{\hat{B}}$ ($n \times p-1$ matrix). This last term is obtained when we project $Y_{-j}$ onto range of $\hat{B}$, and will reduce both the bias and variance of estimating the rows of the precision matrix in Theorem (ref) below. It can be considered as size of the signal, and its key to consistency of our estimators. Next, we define
where $M_{\hat{B}}:= I_{p-1} - P_{\hat{B}}$, and $Y_{-j} M_{ \hat{B}}$ is $n \times p-1$ matrix. This term is the largest singular value of $Y_{-j} M_{\hat{B}}$ matrix scaled by $n$, and it represents the bias stemming from using a hidden factor structure to estimate $Y_{-j}$. This last term will increase the bias of estimating the rows of the precision matrix.
Define $\max_{1 \le j \le p } \hat{r}_j:= \bar{r}$, $\min_{1 \le j \le p} \hat{\eta}_j:= \bar{\eta}$, and $\max_{1 \le j \le p} \hat{\Psi}_j:= \bar{\Psi}$. We have the following uniform bounds for estimators for off-diagonal terms in each row of the precision matrix, with $\beta, \Gamma$ as positive large constants, and $\bar{r}, \bar{\eta},\bar{\Psi}, \bar{\xi}$ are random variables,
In this subsection we consider principal components regression (adaptive PCR from now on) estimators. Let us designate $rank (Y_{-j}) \ge k_j \ge 1$, where $k_j$ is a positive integer and changes with $j=1,\cdots,p$. The adaptive PCR estimators use $\hat{B}= V_{1,k_j}$, where $V_{1,k_j}$ is $p-1 \times k_j$ matrix where $V_{1,k_j}$ represents a matrix which $k_j$ columns (eigenvectors) correspond to the largest $k_j$ eigenvalues of $Y_{-j}' Y_{-j}/n$ for $j=1,\cdots,p$. The adaptive PCR estimator is, for $j=1,\cdots,p$ \[ \hat{\alpha}_{PCR,j} = V_{1,k_j} (Y_{-j} V_{1, k_j})^+ y_j,\] where $(.)^+$ is Moore-Penrose inverse of a matrix. Before simplifying the terms in the upper bound in Theorem (ref) we define $\hat{\lambda}_{k_j}:= \frac{1}{n} \sigma_{k_j}^2 (Y_{-j})$, for example with $k_j=1$ we have the largest singular value of $Y_{-j}$, which is $\sigma_{1}^2 (Y_{-j})/n$ (squared-scaled by $n$). One big issue is the choice of $k_j$. To choose $k_j$ we adapt the so-called elbow method, (for details, see (19) of bbsmw21)
with $C_0 >1$ and, $c_{\Delta} >1$ are positive constants above one,
This type of choice is done to minimize the risk of adaptive PCR in bbsmw21. Here we need results uniformly over $j=1,\cdots,p$ and the choice given in ((ref)) and above is a feasible estimator of $s_j$ after following Remark 3- fifth item to Corollary (ref). Set $k_j= \hat{s}_j$. By definition of $\hat{s}_j$ above
We show in the proof of Corollary (ref) in Appendix
which will simplify the upper bound for $l_2$ norm error for our estimators. We prove the following error bounds for adaptive PCR based estimator for the off-diagonal terms in a given row for the precision matrix. Note that three random terms $\bar{r}\le K$, $\bar{\eta}\ge \beta >0$, and $\bar{\Psi}/\bar{\eta}\le 1$ with probability approaching one to obtain Corollary (ref) from Theorem (ref).
Note that noise constant $\Gamma$ play an important role in finite samples by increasing the variance and bias, but $\beta$ has the opposite effect. High dimensional consistency is possible when $p>n$, with $\bar{\xi}$ increasing with $p$ as can be seen in Example in ((ref)). To have a feasible estimation with $K$ estimated, we follow (56) of bbsmw21 or (2.7) of bw19. Assuming additionally in Corollary (ref) that $tr (\Sigma_{U,-j} ) \ge c'' \| \Sigma_{U,-j} \|_2 (n \wedge p)$, with $c''>0$ a positive constant, for each $j=1,\cdots,p$, Proposition 8 of bbsmw21 can be extended with a simple union bound and the feasible estimation of the unknown number of factors results do not change Corollary (ref).
In this part, we consider RRE when $p-1<n$ first and then analyze $p>n$ case. In that respect $\hat{B}= I_{p-1}$, which simplifies the upper bound in Theorem (ref). We have \[ \hat{\alpha}_{RRE,j} = (Y_{-j})^+ y_j,\] for $j=1,\cdots,p$. Note that when $p-1<n$, RRE estimation is equivalent to OLS, and by using (132) of bsmw22, $\hat{\alpha}_{RRE,j} = (Y_{-j})^+ y_j = (Y_{-j}' Y_{-j})^{-1} Y_{-j}' y_j.$ Three random terms simplify $\bar{r} < p$, $\bar{\Psi}=0$, and $\bar{\eta} \ge C >0$ with probability approaching one to obtain Corollary (ref) from Theorem (ref).
Now we provide the Corollary for the RRE case when $p-1>n$. First we find it useful to introduce the effective rank as $ re (\Sigma_{U,-j}) := \frac{tr (\Sigma_{U,-j})}{\| \Sigma_{U,-j} \|_2}.$ Next, we relax Assumption (ref)(i) that relates $p$ to $n$. We show that in RRE with $p-1>n$ case, we can allow $p$ to grow exponentially in $n$. This implies that we can have much larger, in dimension, precision matrix estimation in RRE compared with adaptive PCR.
{\bf Assumption $3^*(i)$.} {\it We allow \[ n< p-1 \le \frac{exp(C_p n/2)}{4}.\]}
Now we provide our coefficient estimation result that will be helpful for precision matrix estimation based on RRE. Define $\delta_n:= \min_{1 \le j \le p} tr (\Sigma_{U,-j})/n$. $\delta_n$ is nondecreasing in n, since $\Sigma_{U,-j}: p-1 \times p-1$ diagonal matrix, with positive elements in the diagonal and $p,n$ grow together under Assumption $3^*(i)$ with $p-1>n$. Note that three random variables in Theorem (ref) simplifies in Corollary (ref) as $\bar{r}=n$, the bias term is $\bar{\Psi}=0$, and the size of the signal is $\bar{\eta}\ge \delta_n$ wpa1.
First, we want to see in the case of $n,p$ grow, whether RRE is consistent or not. With $p$ growing polynomially in $n$, with a power larger than one, RRE is consistent. Clearly $\frac{logn + n}{n \delta_n} \to 0,$ since $\delta_n$ may grow larger in $n$.Then $K/\bar{\xi} $ can be smaller if we use the example $(\ref{ex1})$, and $K / d_{1n}^2 \to 0$. So signal-to-noise ratio should grow and also dominate the number of factors as well to have consistency. Note that we may see double descent in RRE for off-diagonal term estimation of the parameter vector $\alpha_{j}^*$ in the precision matrix. We fix $n$ and $K$ and analyze the changes in $p$. To see that in the previous Corollary we need $p/n \to 0$ to get estimation errors converging to zero. So very small $p$ in OLS (RRE $p-1<n$) case provides estimation error converging to zero. Then in minimum-norm interpolator case here (RRE with $p-1>n$), with $p/n \to \infty$ with $n$ fixed, and noise $\delta_n$ may grow with larger $p$, since it is a scaled version of $p-1 \times p-1$ diagonal matrix trace with all positive elements, $\frac{logn + n}{n \delta_n} \to 0,$ and $K /\bar{\xi} \to 0$ as discussed above since $ d_{1n}^2$ depend on $p$ and if $d_{1n}^2$ dominates K, we may see estimation errors converging to zero, and hence double descent can be seen. To see double descent Assumption $\ref{as3}^{*}$(i),(ii) and ((ref)) is helpful.
We compare now RRE in this case with adaptive PCR. In RRE there is one less bias term compared with adaptive PCR ($\bar{\Psi}=0$ in RRE) in finite sample upper bound. Since $K < n$, adaptive PCR has a smaller variance term compared with RRE. Asymptotically bias terms have the same rate $K/\bar{\xi}$.
Adaptive PCR also may have one more theoretical advantage compared with RRE. We need the noise $\delta_n$ to be increasing in $n$ to get consistency in RRE, whereas in adaptive PCR this is not needed. But in financial markets we expect $\delta_n \to \infty$ so this may not be of an issue between two estimators in financial markets.
We have the following Theorem for the reciprocal of the main $j$-th diagonal term in the precision matrix. The technique in bbsmw21 considers only simple linear regression, while the precision matrix estimation and especially a diagonal term estimation are not handled. The proof of Theorem (ref) extends the literature on diagonal term estimation of a large precision matrix. The results and the proof are novel and show the rate of convergence and the non-asymptotic upper bound for estimation errors, as well as the consistency of the estimator. Note that $r_{w1}$ rate is defined just before Assumption (ref) in the main text.
We provide our theorem for estimating rows of the precision matrix. This shows that each row of the precision matrix in general case can be estimated, uniformly over all rows, consistently through our estimator. Specific results regarding adaptive PCR and RRE will be discussed in subsections (ref)-(ref). Note that the main diagonal terms rate determine the rate of convergence. The off-diagonal terms estimation is consistent but converges at a faster rate than the main diagonal terms. This can be seen through comparing the rates in Theorem (ref) and via proof of Theorem (ref) and Remark (ref) in Theorem (ref). Denote each row of $\Theta$ as $\Theta_{j}'$, which is a $1 \times p$ vector. As an example, first row can be represented as $\Theta_1':= (\frac{1}{\tau_1^2}, \frac{-\alpha_1^{*'}}{\tau_1^2} )$. Let $\Theta_{j}$ be the column representation of the row $\Theta_{j}'$. Let the row estimator be $\tilde{\Theta}_j'$, and for the first row this is $\tilde{\Theta}_1^{'} := (\frac{1}{\tilde{\tau}^2}, \frac{- \tilde{\alpha}_1^{'}}{\tilde{\tau}_1^2})$.
In case of adaptive PCR, the random terms in the upper bound in Theorem (ref), can be bounded as: $\bar{r} \le K$ wpa1, $\bar{\Psi}/\bar{\eta} \le1$ wpa1, $\bar{\eta} \ge \beta$ with wpa1. So Theorem (ref) rate can be simplified in adaptive PCR, with $\hat{\Theta}_{PCR,j}'$ representing the estimator of $j$ th row of $\Theta$. The following Corollary follows from Theorem (ref) by replacing $r_{w1}$ with $r_{w1,PCR}$ rate that is shown before Corollary A.1 in Online Appendix.
When we consider the sub case of RRE with $p<n-1$, we set wpa1 $\bar{r} <p$, $\bar{\Psi}=0$, and $\bar{\eta} \ge c$, where $ c $ is a positive constant in Theorem (ref) result above. Let the $j$ th row of estimator of the precision matrix be $\hat{\Theta}_{RRE,j}'$. To obtain Corollary (ref), we use Corollary A.2 and Corollary (ref) with $r_{w1}$ in Theorem (ref) replaced with $r_{RRE-l}$ rate after Corollary A.1 in Online Appendix.
Next, we consider RRE estimator when $p-1>n$. Note that in this scenario main diagonal term estimation is done differently since RRE is an interpolating estimator. We prove this after Corollary A.2. In the proof of Theorem (ref), it is clear that the rate of convergence should be the slower of the main diagonal or off-diagonal terms. Let the j-th row of the precision matrix be $\hat{\Theta}_{INT,j} $. Corollary 6 below proves $\hat{\Theta}_{INT,j}$'s consistency by combining Corollaries (ref) and A.3.
The objective of the empirical application is to conduct an out-of-sample comparison among estimations. We estimate Maximum Sharpe Ratio (MSR) portfolio using estimators, and use four commonly used financial metrics: the Sharpe Ratio (SR), average return, portfolio turnover, and portfolio variance. We use data from January 1995 to December 2023, with an out-of-sample period from January 2015 to December 2023. We selected all stocks included in the S$\&$P 500 for at least one month during the out-of-sample period, ensuring the data were available for the entire 1995-2023 period, resulting in 357 stocks. The out-of-sample period includes stock market of 2022 when S$\&$P 500 index was down by 18.1 %.
We use a rolling window set up for the out-of-sample estimation of the Sharpe Ratio following caner2019 and caner2022. The sample size n is divided into in-sample $(1:n_I)$ and out-of-sample $(n_I +1:n) $. Initially, we estimate the portfolio $\hat w_{n_I}$ in the in-sample period and the out-of-sample portfolio returns $\hat w'_{n_I} y_{n_I+1}$. Then, we roll the window forward by one element $(2:n_I +1 )$, forming a new in-sample portfolio $\hat w _{n_I +1 } $ and calculating out-of-sample portfolio returns $\hat w_{n_{I+1}}'y_{n_I +2}$. This process is repeated until the end of the sample.
The out-of-sample average return and variance without transaction costs are computed as follows: $ \hat \mu_{os} = \frac{1}{n-n_I} \sum_{t=n_I}^{n-1} \hat w_t'y_{t+1}, \hat \Sigma^2_{y,os} = \frac{1}{n-n_I -1} \sum_{t=n_I}^{n-1} (\hat w_t' y_{t+1} - \hat \mu _{os})^2 $. We estimate the Sharpe Ratios without transaction costs: $ SR = \frac{\hat \mu_{os}}{\hat \Sigma_{y,os}} $
When transaction costs are present, the definition of the metrics is modified. Let $c$ be the transaction cost, which we set 10 basis points. Let $y_{P,t+1} = \hat w_t ' y_{t+1}$ be the return of the portfolio $P$ in period $t+1$; in the presence of transaction costs, the return is defined as $ y^{Net}_{P,t+1} = y_{P,t+1} - c(1+y_{P,t+1}) \sum_{j=1}^p | \hat w_{j,t+1} - \hat w^+_{j,t}| $ where $\displaystyle \hat w^+_{j,t} = \frac{\hat w_{j,t} (1+y_{j,t+1})}{1 + y_{P,t+1}}$, with $y_{j,t+1}$ representing the excess return added to risk-free-rate for $j$th, and $y_{P,t+1}$ representing the portfolio $P$ excess return added to the risk-free rate. The adjustment made in $\hat w^+_{j,t}$ is because the portfolio at the end of the period has changed compared to the portfolio at the beginning of the period. Then, we define the mean and variance of the portfolio returns with transaction costs as: $ \hat \mu_{os-c} = \frac{1}{n-n_I}\sum_{t=n_I}^{n-1} y^{Net}_{P,t} \mbox{ and } \hat \Sigma^2_{y,os-c} = \frac{1}{n-n_I-1} \sum_{t=n_I}^{n-1} (y^{Net}_{P,t} - \hat \mu_{os-c})^2 $. The portfolio returns are replaced by the returns with transaction costs when we calculated the Sharpe Ratio with transaction costs. Finally, portfolio turnover is defined as: $ \text{turnover} = \sum_{j=1}^p | \hat w_{j,t+1} - \hat w_{j,t}^+|. $
We conduct a comprehensive empirical analysis by fixing in-sample $n_I=240$ and considering both low-dimensional and high-dimensional cases across various portfolio sizes. Specifically, we selected 10, and 200 stocks from the pool to estimate the models with the same stocks for all windows in the low-dimensional experiments. For the high-dimensional experiment, we choose 357 (including all available stocks). We also analyzed portfolios with 25, 50, and 300 stocks, however, since these results are similar, they are not included here.
The model comparisons include several approaches from the literature. We consider the nonlinear shrinkage methods of lw2017(NL-LW) and the single-factor nonlinear shrinkage (SF-NL-LW), POET method from fan2013, and the nodewise (NW) method from caner2019, along with the feasible residual-based nodewise approach from caner2022. SF-NL-LW results are very similar to NL-LW in terms of standard deviation, so we have not included that in our Table. But these are available from authors on demand. In particular, RNW-SF, RNW-3F, and RNW-5F refer to residual-based nodewise regressions using a single-factor model (market as the only factor), three-factor model, and five-factor Fama and French model, respectively. Our new method introduces several variations: RRE, PCR-1F, PCR-3F, PCR-5F, PCR-7F and PCR-Adaptive. These denote the use of a RRE estimator, Principal component with fixed 1, 3, 5, 7 hidden factors, and an adaptive method to select the number of hidden factors, respectively. We use these number of factors since they are used in the empirics in the literature and seven factor is used to show large number of factors.
We report the results for Maximum Sharpe Ratio portfolio, with transaction costs in Tables 1-3, and $NA$ represents "not available" for turnover in our tables with transaction costs. $SD$ represents Standard Deviation of the portfolio, and $Sharpe $ represents Sharpe Ratio of the portfolio respectively in Tables.
Tables without transaction costs are in Appendix and also have turnover results. We have also done analysis for Global Minimum Variance portfolio, and the results are very similar. Results for the low-dimensional cases are presented in Tables (ref) - (ref), while high-dimensional case is shown in Table (ref). In additionally, we use the same data but with different out-of-sample from January 2005 to December 2023 ($n_I = 120$, $n-n_I = 228$). This subsample yields similar results and is available upon request. In MSR portfolio, our PCR-based method is the best model across all 2 small-to-medium (low-dimensional) portfolios scenarios for Sharpe Ratio with transaction cost. Our RRE achieves the highest Sharpe Ratio for the largest size of portfolio with $ p=357$. Let us give an example by examining three portfolios of varying sizes for MSR: a small low-dimensional portfolio, a moderate-sized portfolio, and a large-sized portfolio. In Table (ref), for a portfolio of 10 stocks ($p=10,n_I=240$), PCR-1F is the best Sharpe Ratio with 0.2260. In Table (ref) of 200 stocks ($p=200,n_I =240$), the highest Sharpe Ratio is with our PCR-7F at 0.2682 and the second best is our PCR-3F at 0.2263. Lastly, in Table (ref), for the largest portfolio with all 357 available stocks, the highest Sharpe ratio is RRE at 0.2203.
We use ledoit2008robust test with circular bootstrap to test the significance of the Sharpe Ratio with transaction cost-winner. Specifically,
Where $SR_{winner}$ is the method that has the highest Sharpe Ratio in each table, testing against all other methods and significance is set at threshold of $p=0.05$. A $"*"$ indicates statistical significance from the winner.
To see whether a double ascent in the Sharpe Ratio is possible in RRE, we analyze a figure. Figure (ref) shows the MSR Sharpe Ratio with transaction cost. In figure, we include the performance of RRE, Nodewise and PCR-3F, PCR-adaptive methods. We observe a double ascent pattern in the RRE estimator's performance in terms of the Sharpe Ratio. For example, with the in-sample window size set at $n_I = 240$, in low-dimensional case, the Sharpe Ratio starts at 0.2065. As the portfolio size increases to 200 stocks, the Sharpe Ratio declines to -0.0471. In the high-dimensional case, where the number of stocks exceeds 240, the SR increases to 0.2203 for a portfolio of 357 stocks. From a practitioner point of view, we observe that we can form a large portfolio with RRE, much larger than the sample size n, without any tuning parameters to fix as in other methods and get a good result.
In this paper, we have introduced a general precision matrix estimator. Our estimator is model-free and does not rely on any sparsity assumptions. Under certain conditions, our estimator exhibits double descent in the estimation error. Moreover, the Sharpe ratio produced by one of our estimators on the S&P 500 data exhibits the double ascent, which to the best of our knowledge has never obtained in other empirical studies.\\
\setcounter{section}{0}\setcounter{equation}{0}\setcounter{lemma}{0}\setcounter{table}{0}
The Appendix is organized into three sections. Appendix A contains all the proofs from the text. Appendix B provides supporting results related to matrix algebra, including inverse matrix identities and two tail inequalities. Appendix C provides simulations and extra empirics, and also literature discussion-comparison and hidden factor usage.
.
\setcounter{equation}{0} \setcounter{lemma}{0} \setcounter{table}{0} \setcounter{corollary}{0}
{\bf Proof of Theorem 1}:
Using (4) and (10) we get the following inequality
The proof contains a detailed transformation of results that can be obtained in linear regression framework to precision matrix estimation. Since we ultimately aim to use our results for high dimensional outcomes, we need them to hold uniformly over $j = 1, \dots, p,$. It is important to note that this is in striking contrast with pointwise results (as are those from, e.g., proof of Lemma 17 bbsmw21 that would be applicable to only one outcome. Each of the right side terms in ((ref)) will be analyzed in each of Steps 1-3.
{\bf Step 1.} Start with the first right side term in ((ref))
with $M_j: n \times n$ matrix defined as
To simplify the proof, we consider the following trace term: \[ tr (M_j) = tr [ (Y_{-j} \hat{B})^{+'} \hat{B}' \hat{B} (Y_{-j} \hat{B})^+].\] and can be upper bounded
where the first inequality is due to $M_j$ being positive semidefinite, and $n \times n$ matrix and Exercise 7.49 of abamag2005, and the first equality is by Exercise 8.27a and 4.24a of abamag2005, and the second equality is by Exercise 10.28 of abamag2005, and the third equality is again by Exercises 4.13d, 4.15b, 4.25b, and 10.28 of abamag2005 with $P_{\hat{B}} = \hat{B} \hat{B}^+$ in p.8 of bbsmw21, and the last equality is just the definition of the complexity $\hat{r}_j$. Now we consider, as proved in (44) of bbsmw21
where $\hat{\eta}_j:= \hat{\sigma}_{\hat{r}_j}^2 (Y_{-j} P_{\hat{B}})/n$ which is the $\hat{r}_j$ th largest singular value, squared, of the matrix $Y_{-j} P_{\hat{B}}$, and divided by $n$. Define $C_p>0$ a positive constant. Then by ((ref))((ref)) and $\bar{\eta}$, $\bar{r}$ definitions
Next, define the following events
and
with
By ((ref))
Clearly, since the upper bounds of the events $ \bar{E}_{1j}, \bar{E}_1$ are the same
By ((ref))((ref))
See that by $E_{1j}^c$ as the complement of $E_{1j}$
Note that we have, by ((ref))
so by ((ref))((ref)) and
Now we calculate the sum of the upper bound probability of $E_{1j}^c$, across $j$. Note that $u_j$ is a subgaussian vector, and independent of $Y_{-j}$, and $\hat{B}$ by Assumption 1 and the definition of $\hat{B}$, with $M_j$ being a symmetric positive semi-definite matrix. So $M_j$ is independent of $u_j$. Then Lemma (ref) (with $t=C_p log n $, $H=M_j, \zeta=u_j$, conditional on $M_j$) allows us to write
Next, use the union bound and by Assumption 3, using $log p \le (\frac{C_p}{2}) log n$
Use ((ref))((ref)) in ((ref))((ref))
{\bf Step 2.} Now consider the second term on the right side of ((ref)). Keeping in mind that, $A_{-j}' (A_{-j}^{+})' = I_K$ and recalling on (5), while setting $M_{\hat{B}}:=I_{p-1}- P_{\hat{B}}$, we further have
Use simple inequality $(a+b+c)^2 \le 3a^2 + 3b^2 + 3c^2$,
In Steps 2a, 2b, and 2c below, we separately analyze each of the three terms on the right side of ((ref)).
{\bf Step 2a.} So
Note that we can write the first term on the right side of ((ref)) as
where we use Moore-Penrose inverse rule (134) in bsmw22 which is for a rectangular matrix $S$, we have $(S'S)^+ S' = S^+$, which we input $S= Y_{-j} \hat{B}$ in our case above. To simplify the expressions see that by the proof of Lemma 15 of bbsmw21 $P_{\hat{B}}= V_1 V_1',$ where $\hat{B}= V_1 D V_2'$ is a singular value decomposition. $V_1,V_2$ are orthonormal matrices and with dimensions of $p-1 \times r_0, q \times r_0$ respectively, and $D$ is a $r_0$ dimensional diagonal matrix, and the rank of $(\hat{B}) = r_0$. Rewrite right side of ((ref)) as
Then in ((ref)) right side see that
which is shown in the proof of Lemma 15 of bbsmw21. Then use ((ref)) in ((ref)), $V_2'V_2=I_{r_0}$
Use the last expression in ((ref)) right side \[ \| V_1 D V_2' (V_2 D V_1' Y_{-j}' Y_{-j} V_1 D V_2')^+ V_2DV_1' Y_{-j}' Y_{-j} V_1 V_1' \|_2 = \| V_1 [V_1' Y_{-j}' Y_{-j} V_1]^+ V_1' Y_{-j} Y_{-j}' V_1 V_1' \|_2.\] Then clearly since $V_1: p-1 \times r_0$ matrix, and Exercise 10.26 of abamag2005 shows $[V_1' Y_{-j}' Y_{-j} V_1]^+ V_1' Y_{-j}' Y_{-j} V_1 $ is a symmetric idempotent matrix and all idempotent matrices eigenvalues are either 0 or 1, and since $V_1' V_1 =I_{r_0}$ \[ \| V_1 [V_1' Y_{-j}' Y_{-j} V_1]^+ V_1' Y_{-j}' Y_{-j} V_1 V_1' \|_2 \le \| V_1\|_2 \| [V_1' Y_{-j}' Y_{-j} V_1]^+ V_1' Y_{-j}' Y_{-j} V_1 \|_2 \| V_1' \|_2 \le 1.\] So use this last inequality in ((ref))
Now consider the second term on the right side of ((ref)), $ \| A_{-j}^{+'} a_j \|_2^2 = a_j' A_{-j}^+ A_{-j}^{+'} a_j = a_j' (A_{-j}' A_{-j})^{-1} a_j,$ since $A_{-j}^{+}= (A_{-j}' A_{-j})^{-1} A_{-j}^{'}$ with $p-1 > K$. Next
where we use first Rayleigh inequality on p.182 of abamag2005 and then Theorem 2.6.3b of hj2013, where $\lambda_K(.)$ is the $K$ th largest eigenvalue for the matrix $A_{-j} \Sigma_f A_{-j}': p-1 \times p-1$. By (6)(7), we try to rewrite ((ref)) right-side.
where we use $\beta:= \min_{1 \le j \le p} \| \Sigma_{U,-j} \|_2 >0$ by Assumption 2(ii). Then by Assumption 4(iii)
by $Eigmax (\Sigma_f) \le C < \infty$ Assumption 3. So the first term on the right side of ((ref)) can be bounded as as
Clearly by ((ref))((ref)) in ((ref))
{\bf Step 2b.} Take the second term on the right side of ((ref))
See that the first term on the right side in ((ref)) by $\| S_b \|_2^2 = \| S_b'S_b \|_2$ for a rectangular matrix $S_b:=\hat{B} (Y_{-j} \hat{B})^+$ and $M_j$ definition in ((ref))
where the last inequality is by ((ref)). Then by definition of $\hat{\Psi}_j$ in (16)
Now consider ((ref))((ref))((ref)) on the second term-right side of ((ref)), with $ \bar{\Psi}:= \max_{1 \le j \le p} \hat{\Psi}_j, \bar{\eta}:=\min_{1 \le j \le p} \hat{\eta}_j,$ with $\hat{\eta}_j$ defined in (15)
{\bf Step 2c.} Consider the third term on the right side of ((ref))
where we analyze the following on the right side of ((ref)): $ \| U_{-j} A_{j}^{+'} a_j \|_2^2.$ Define $\gamma_{e,j}^2 = \gamma_w^2 a_j' A_{j}^+ \Sigma_{U,-j} A_{j}^{+'} a_j$. Now we simplify $\gamma_{e,j}^2$ term. See that, uniformly over $j$, by Assumption 2 and ((ref))
Then define the events
and
By ((ref)) and the explanations above that equation, with ((ref))((ref))
Next, define
See that by definitions in ((ref))((ref))
Then by ((ref))((ref))
Then by ((ref))
Since $U_{-j} A_{-j}^{+'} a_j$ is $\gamma_w \sqrt{a_j' A_{-j}^+ \Sigma_{U,-j} A_{-j}^{+'} a_j}$ subgaussian vector for each $j=1,\cdots,p$ via Assumption 4, apply Lemma (ref) with $\epsilon_j:= U_{-j} A_{j}^{+'} a_j$ and $H:=I_n$ and $\gamma_{e,j}^2 = \gamma_w^2 a_j' A_{j}^+ \Sigma_{U,-j} A_{j}^{+'} a_j$ with $t=C_p logn $
Then take the union bound and with Assumption 3, $log p \le \frac{C_p}{2} logn$
Next use ((ref))((ref))((ref))
Use ((ref)) and ((ref)) in ((ref)) with probability at least $1-1/n^{C_p/2}$ (note that we ignore the smaller order terms than the right side term in ((ref)) since they play no role in the subsequent analysis), while incorporating $\gamma_w^2$ into C,
Combine all Steps 2a-2c results ((ref))((ref))((ref)) in ((ref)) to have, with probability at least $1-1/n^{C_p/2}$, absorbing 3 into the constant $C$,
{\bf Step 2d}. A combination of ((ref)) ((ref)) provides the following upper bound for the first two right side terms on ((ref)), with probability at least $1-2/n^{C_p/2}$
{\bf Step 3.} After setting $\bar{a}_j:= \Sigma_f^{1/2} a_j, \bar{A}_{-j}:= A_{-j} \Sigma_f^{1/2}$, $\bar{G}_j:= I_K + \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}$ definitions, we upper bound the last term on the right side of ((ref)) via Lemma 1 in the following way
Then
Next, $ \bar{a}_j' \bar{G}_j^{-1} \bar{a}_j = \bar{a}_j' [ I_K + \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}]^{-1} \bar{a}_j.$ Define $H_j:= \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}$, and $H_j$ is invertible, with rank K
where we use Rayleigh inequality for the first inequality, and for the last equality we use $H_j$ definition. Next use $\bar{a}_j:= \Sigma_f^{1/2} a_j, \bar{A}_{-j}:= A_{-j} \Sigma_f^{1/2}$ definitions $ \bar{a}_j' [ \bar{A}_{-j}' \Sigma_{U,-j}^{-1} \bar{A}_{-j}]^{-1} \bar{a}_j = a_j' (A_{-j}' \Sigma_{U,-j}^{-1} A_{-j})^{-1} a_j.$ Then using the inequality after proof of (63) in bsmw22 at the end of Section A.2.3 of bsmw22
Combine ((ref))((ref)) in ((ref)) to have, by ((ref))
Next,
where we use $\min_{ 1 \le j \le p} Eigmin (\Sigma_{U,-j}) \ge c > 0$ Assumption 2(i) for the last inequality as well as $\bar{\xi}$ definition, and second equality in ((ref)), with ((ref)). Now we combine Steps 1-3 to have with probability at least $1-2/n^{C_p/2}$ to have the desired result.(ii). Asymptotics follows from (i).
{\bf Q.E.D.}
{\bf Proof of Corollary 1}. (i). We need to simplify three terms on the upper bound in Theorem 1 are $\bar{r}, \bar{\eta}, \bar{\Psi}$. Note that $\bar{r}:= \max_{1 \le j \le p} \hat{r}_j$. In that respect,
where the first equality is definition and the second equality is by ((ref)), and the third equality is the proof of Corollary 5 in bbsmw21. Then the term
where the first equality is definition and the second equality is by the proof of Corollary 5 in bbsmw21, and note that $\hat{\lambda}_{k_j} = \frac{1}{n} \sigma_{k_j}^2 (Y_{-j})$ by definition. Then
where the first equality is definition, and the second one is by the proof of Corollary 5 in bbsmw21, and with $\hat{\lambda}_{k_j+1}= \frac{1}{n} \sigma_{k_j+1}^2 (Y_{-j})$. We need to prove (20) to proceed further. First we define the event
and its complement is $E_{3j}^c$. We start by proving
In that respect, by using Weyl's inequality
since $\sigma_{K+1} (F A_{-j}') =0$ by $F A_{-j}'$ having rank of $K$. See that $\sigma_1^2 (U_{-j})/n= \| U_{-j}' U_{-j} \|_2/n= \| U_{-j} U_{-j}' \|_2/n$ by singular value definition. We provide a definition of $\tilde{U}_{-j}$, which is an $n \times p-1$ matrix, $U_{-j}$ also based on Assumption 4, with $\tilde{U}_{-jt}': 1 \times p-1$ $t$ th row of $\tilde{U}_{-j}$, and we stack these $n$ rows to get $\tilde{U}_{-j}$ and then form $U_{-j}$ $ U_{-j}:= \tilde{U}_{-j} \Sigma_{U,-j}^{1/2}.$ Then by Lemma 22 of bbsmw21 via Assumption 4, and definition (18)
Define $ G_{1j}:= \{\frac{1}{n} \| \tilde{U}_{-j} \Sigma_{U,-j} \tilde{U}_{-j}'\|_2 \le C_0 \Delta_{-j}\}.$ Also see that by ((ref))
for all $j=1,\cdots,p$. By ((ref))((ref))
So clearly $ P( E_{3j}^c) \le exp(-n).$ To complete the proof of ((ref)) \[P ( \cap_{j=1}^p E_{3j}) = 1 - P ( (\cap_{j=1}^p E_{3j})^c) = 1 - P ( \cup_{j=1}^p E_{3j}^c) \ge 1 - \sum_{j=1}^p P (E_{3j}^c). \] Next, the union bound argument
given Assumption 3, $log p \le c' n$ with $c'<1$ a positive constant upper bounded by one. So this clearly shows
Then by (17)(19)((ref)), we have $P ( \max_{1 \le j \le p} \hat{s}_j < K+1) \ge 1 -exp((c'-1)n)$ hence
with $\hat{s}_j$ being a positive integer. By ((ref)), $\hat{r}_j = k_j$, and since by $\hat{s}_j$ definition in (17) we have $k_j = \hat{s}_j$, so
Then using ((ref)) with ((ref))
Now we consider $\hat{\Psi}_j$ uniformly in $j$ (in probability). The uniformity proof has two parts. First, define for each $j=1,\cdots,p$
Then first by $\hat{\Psi}_j$ definition in ((ref)), and since $\hat{s}_j= \hat{k}_j$ by $\hat{s}_j$ definition with ((ref))
The first part of the uniformity proof is done. For the second part, note that by (19) and ((ref)) $\max_{1 \le j \le p} s_j \le K $ with probability approaching one
By ((ref))((ref))((ref)), and $\hat{\lambda}_{K+1}:= \frac{1}{n} \sigma_{K+1}^2 (Y_{-j})$ with probability approaching one
So if we prove $P (\cap_{j=1}^p E_{3j}) \to 1$, we will have $P (\cap_{j=1}^p G_{2j}) \to 1$. We have by ((ref)) $P (\cap_{j=1}^p E_{3j} ) \ge 1 - exp ((c'-1)n).$ That implies
So we proved uniformity across $j=1,\cdots, p$ for $\hat{\Psi}_j$ combining ((ref)) ((ref)). Now we consider $\hat{\eta}_j$ uniformly in $j$ (in probability).
since $\hat{s}_j = k_j$ and using $\hat{\eta}_j$ definition in ((ref)). Note that since we proved $\max_{1 \le j \le p} \hat{s}_j \le K$ with probability approaching one, and by (17)(19), uniformly over $j$ $ \hat{\lambda}_{\hat{s}_j} \ge \hat{\lambda}_{K}.$ With ((ref))(19) and the above inequality
and the last inequality is by ((ref))((ref)), and $\hat{\lambda}_{K+1}:= \sigma_{K+1}^2 (Y_{-j})/n$. Then $C_0 \Delta_{-j}$ in (18)
since $c_{\Delta} >1$. Define $ G_{3j}:=\{ \hat{\eta}_j \ge \beta\}.$ Then by ((ref)) and above event definition, with equality in ((ref)) $ \cap_{j=1}^p E_{3j} \subseteq \cap_{j=1}^p G_{3j},$ for each $j=1,\cdots,p$. So combine with ((ref))((ref)) we have
Furthermore, using ((ref))((ref)), uniformly in $j=1,\cdots,p,$ with probability approaching one
Our proofs here are uniform and detailed extensions of pointwise results of proofs of Corollaries 5-6 in Sections B.2.1-B.2.2 of bbsmw21. (ii). Note that asymptotic case is obtained from (i). {\bf Q.E.D.}
{\bf Proof of Corollary 2}. (i). We consider first $p - 1 < n$ scenario (in which RRE effectively becomes OLS) and again proceed by simplifying three terms, $\bar{r}$, $\bar{\eta}$, $\bar{\Psi}$, that appear in the upper bound in Theorem 1. In that respect $\hat{B}= I_{p-1}$ by definition of RRE. Then clearly $\hat{r}_j= rank (Y_{-j} )= p-1$ since $p-1<n$ in this Corollary. Hence
Also since $I_{p-1} - P_{I_{p-1}} = 0$ we have
This last equality shows that RRE has one less bias term in the upper bound compared to adaptive PCR estimator or compared to the general upper bound in Theorem 1. Use Corollary 11.6.7 in bern2018 (with $i=p-1, j=1$ there) or (11.6.18) of bern2018 for the first inequality below, with $\Sigma_{-j,-j}$ as defined in Appendix B,
where the last equality is by the definition of singular value. To get the last inequality in ((ref)), clearly by $\Sigma_{-j,-j}, \Sigma_{U,-j}$ definitions, by using (5) and by Assumptions 2-3 $ min_{1 \le j \le p} Eigmin (\Sigma_{-j,-j}) \ge min_{1 \le j \le p} Eigmin (\Sigma_{U,-j}) \ge c.$ So we obtain
To analyze ((ref))((ref)), define $D_j:=\{ \frac{1}{n} \sigma_{p-1}^2 (Y_{-j} \Sigma_{-j,-j}^{-1/2}) \ge C \}$.
where to get the second inequality, with $Y_{j-1}: n \times p-1$, and $n>p-1$ here, by Theorem 5.39 of v2012 with Corollary 10 proof in bbsmw21 via Assumption 4, and applying a union bound (with $t_1 =\sqrt{logn}$ )and by assuming $log p \le (C_p/2) logn$ by Assumption 3. Next, we use ((ref))((ref)) and ((ref)) with $D_j$ definition
Now substitute ((ref))((ref))((ref)) in the upper bound in Theorem 1 to have the result. (ii). Asymptotics is derived by using (i).
{\bf Q.E.D}
{\bf Proof of Corollary 3}. (i). There are again three terms that we consider, but we have $p-1>n$ in this RRE case. So $ \bar{r}= \max_{1 \le j \le p} rank (Y_{-j}) =n.$ Then we have $\bar{\Psi}=0$ as in Corollary 2 due to $\hat{B}= I_{p-1}$. The last term is $\hat{\eta}_j = \frac{1}{n} \sigma_n^2 (Y_{-j})$, which is the $n$ th largest singular value of $Y_{-j}: n \times p-1$ matrix. Also we assume $ tr (\Sigma_{U,-j}) / \| \Sigma_{U,-j}\|_2 > C n$, and $\tilde{U}_{-j}$ has independent entries. Use Lemma (ref)
Also we have by definition $\delta_n:= min_{1 \le j \le p} tr (\Sigma_{U,-j})/n$. Define $G_{4j}:= \{ \hat{\eta}_j \ge tr (\Sigma_{U,-j}/n)\}$, $ G_{5j}:= \{ \hat{\eta}_j \ge \delta_n\}.$ Note that for each $j=1,\cdots,p$
Note that by $G_{5j}$ definition,
since the lower bounds are the same $ \delta_n$. Next we have the following proof
where the first equality is by ((ref)), and the second inequality is by ((ref)), and the third inequality is by ((ref)), and the last inequality is by Assumption $3^{*}$(i). The desired result follows by incorporating these three terms in the upper bound of Theorem 1. (ii). Asymptotics follows from (i).
{\bf Q.E.D.}
Before the following proof, we proceed with three inequalities that are used for rectangular matrices. Take a generic rectangular matrix $M$ and a vector $x$.
by (11.4.4) of bern2018. Then by (11.3.15) of bern2018 where for two generic rectangular matrices $M,L$
We form the following matrix norm inequality for a generic rectangular matrix $M: m_1 \times m_2$
where $\|.\|_F $ is the Frobenius norm and we get the first inequality by (4.4) of v2019.
{\bf Proof of Theorem 2}.
First note that $E y_{jt}= \mu_j = a_j' E f_t$, and $E Y_{-jt} = \mu_{-j} = A_{-j} E f_t$. Also see that
and
(i). Note that we need to estimate the $j$ th main diagonal term for the precision matrix of asset returns. In that sense the key element is the reciprocal of that term which is defined in (11), and the population version in (B.10). We use triangle inequality to consider the estimation error by (11)(B.10)
Next add and subtract to the second right side term $E [(y_{jt} - \mu_j)( Y_{-jt} - \mu_{-j})' ]\tilde{\alpha}_j$, by iid nature of data,
Take uniform bounds over $j=1,\cdots,p$ and use Cauchy-Schwartz inequality for the second and third right side terms on ((ref))
Each right side term in ((ref)) will be analyzed as a step in the proof.
{\bf Step 1.} Start with the first right side term on ((ref)), and by (2) \[ y_{jt}^2 = a_j' f_t f_t' a_j + 2 a_j' f_t u_{jt} + u_{jt}^2,\] and by Assumption 1(iv), and using ((ref)) \[ E (y_{jt} -\mu_j)^2 = a_j' \Sigma_f a_j + E (u_{jt}^2).\] Use triangle inequality by the last two equalities, and noting $\Sigma_f:= E (f_t - E f_t )( f_t - E f_t)'$
Take the first right side term in ((ref)) above, and use Cauchy-Schwarz inequality first and then ((ref))
We consider the second term on the right side of ((ref)), with $F: n \times K$ matrix, and $ \| \Sigma_f \|_2 \le C < \infty$ by Assumption 3(ii)
by Lemma 23 of bbsmw21 since rows of $F$ are independent subgaussian vectors, with probability at least $1-2/n^c$ via Assumption 1(iii), Assumption 4(ii), and note that $K < n$. By Assumption 4(iii)
Use ((ref))-((ref)) in ((ref)) and Assumption 3 to have
Then in ((ref)) consider the second right side term
We now show how we obtain ((ref)). Note that by Corollary 2.8.3 of v2019, for $t_1>0$, and the union bound \[ P \left[ \max_{1 \le j_1 \le p } \max_{1 \le j_2 \le p} | \frac{1}{n} \sum_{t=1}^n [u_{j_1,t} u_{j_2,t} - E (u_{j_1,t} u_{j_2,t})]|>t_1 \right] \le p^2 exp \left(-c min(\frac{t_1^2}{C_{\psi}^2}, \frac{t_1}{C_{\psi}})n\right),\] where we use $u_{j_1,t}$ and $u_{j_2,t}$ are both subgaussian random variables, so $u_{j_1,t} u_{j_2,t}$ are subexponential random variable (independent across $t=1,\cdots,n$) by Lemma 2.7.7 of v2019 and exercise 2.7.10 of v2019 shows centered subexponential are also subexponential random variables, and we have \[ \max_{1 \le j_1 \le p} \max_{1 \le j_2 \le p} \| u_{j_1,t} u_{j_2,t} \|_{\psi} \le C_{\psi} < \infty\] where $C_{\psi}$ are the upper bound constant for the subexponential norm in Definition 2.7.5 of v2019. Next $ 2 p^2 exp \left(-c min(\frac{t_1}{C_{\psi}^2}, \frac{t_1}{C_{\psi}}) n \right) = exp \left(log (2 p^2) - c \frac{t_1^2 n}{C_{\psi}^2}\right)$, with $t_1:= C \frac{\sqrt{log p}}{\sqrt{n}}$ and with sufficiently large $n$ (i.e.$ \sqrt{n} \ge \frac{C}{C_{\psi}} (log p)^{1/2}$). Set $ c_{\psi}:= c \frac{C^2}{C_{\psi}^2} - 2 > 0$ to have $ exp [ log 2 + 2 log p - c \frac{C^2}{C_{\psi}^2} log p ] = \frac{2}{p^{c_{\psi}}}.$ Then \[ P \left(\max_{1 \le j_1 \le p } \max_{ 1 \le j_2 \le p} | \frac{1}{n} \sum_{t=1}^n u_{j_1,t} u_{j_2,t} - E u_{j_1,t} u_{j_2,t}| > C \frac{\sqrt{log p}}{\sqrt{n}} \right) \le \frac{2}{p^{c_{\psi}}}.\] The asymptotics in ((ref)) are obtained after taking $n \to \infty, p \to \infty$. Consider the third right side term in ((ref)). Lemma 2.7.7 and Corollary 2.8.3 of v2019 together with Assumptions 1, 4 and Holder's inequality give
where the last equality uses $p-1>K$. By ((ref))-((ref)) we have the first right side term in ((ref))
{\bf Step 2.} We consider the second right side term in ((ref)). By (2)(3) and Assumption 1(iv) \[ y_{jt} Y_{-jt}' = a_j' f_t f_t' A_{-j}' + a_j' f_t U_{-jt}' + u_{jt} f_t' A_{-j}' + u_{jt} U_{-jt}',\] and by ((ref)) ((ref)) \[ E (y_{jt} - \mu_j)( Y_{-jt} - \mu_{-j})'= a_j' \Sigma_f A_{-j}'.\] Use these last two equations to see that the following term
\[ \max_{1 \le j \le p} \| \frac{1}{n} \sum_{t=1}^n [y_{jt} Y_{-jt}' - E (y_{jt} - \mu_j)( Y_{-jt} - \mu_{-j})'] \|_2,\] is upper bounded by
Consider the first term on the right side of ((ref)) by ((ref))((ref))
and the rates are by ((ref))((ref)) and Assumption 3. Then consider the second term on the right side of ((ref))
where we use ((ref)) for the first inequality since $u_{jt} f_t': 1 \times K$ vector and $A_{-j}': K \times p-1$ matrix, and the second inequality is by simple vector norm inequality tying $l_2, l_{\infty}$ vector norms, and the rates are by Assumption 3(iii) $\max_{1 \le j \le p} \| A_{-j} ' \|_2 = O (\sqrt{p})$ and Lemma 2.7.7 and Corollary 2.8.3. of v2019 which provides $\| n^{-1} \sum_{t=1}^n u_{jt} f_t' \|_{\infty} = O_p (\sqrt{logp/n})$. Now third term on the right side of ((ref))
where we use for the first inequality ((ref)) and then for the second inequality we use ((ref)) and for the rates we use Assumption 5(iii), and Lemma 2.7.7 and Corollary 2.8.3 of v2019 with Assumption 4 since $f_t U_{-jt}'$ is $K \times p-1$ matrix. Then for the fourth term on the right side of ((ref)) via Corollary 2.8.3 of v2019 and Assumption 4 and simple vector norm inequality
Relying on ((ref))((ref))((ref))((ref)), and keeping in mind that ((ref)) has the largest rate by Assumptions 3 and 5(i), and we have the following for the second right side term in ((ref))
We find it useful to recall the rate definition
The proof of Theorem 1-Step 2d gives
Combine ((ref))((ref)) for the second right side term in ((ref))
{\bf Step 3.} Now consider the third term on the right side of ((ref)). The key term is $E [(y_{jt}-\mu_j)( Y_{-jt} -\mu_{-j}) '] = a_j \Sigma_f A_{-j}',$ where we use Assumption 1 and (2)(3). Since we have for rectangular matrices ((ref))((ref))
By ((ref)), the third term on the right side of ((ref)) is via Theorem 1
{\bf Step 4.} We combine ((ref))((ref))((ref)) with Assumption 5(i) on the right side of ((ref)) to have the desired result.
{\bf Q.E.D.}
(ii). By Assumption 5(ii)-(iii) we obtain the consistency. {\bf Q.E.D.}
(iii). By (B.11) $ \tau_j^2:= E [(y_{jt} - \mu_j)^2 - E (y_{jt} -\mu_j)( Y_{-jt} - \mu_{-j})' ] \alpha_{j}^*.$ Then by Cauchy-Schwarz inequality
where we use Step 3 of Theorem 2 together with Step 3 of Theorem 1 proof, and ((ref)), and Assumption 5(iii). Next by using (2), Assumptions 1-3
The last two results provide the desired result. {\bf Q.E.D.}
{\bf Proof of Theorem 3}. First of all \[ \max_{1 \le j \le p} \| \tilde{\Theta}_j' - \Theta_{j}' \|_2 \le \max_{1 \le j \le p} \left| \frac{1}{\tilde{\tau}_j^2} - \frac{1}{\tau_j^2} \right| + \max_{1 \le j \le p} \| \frac{\tilde{\alpha}_j}{\tilde{\tau}_j^2} - \frac{\alpha_{j}^*}{\tau_j^2}\|_2.\] Then add and subtract $\alpha_{j}^*/\tilde{\tau}_j^2$ to the right hand side of the immediately above inequality, and via triangle inequality and arranging
Note that since $\min_{1 \le j \le p} \tilde{\tau}_j^2 \ge \min_{1 \le j \le p} \tau_j^2 - \max_{1 \le j \le p} | \tilde{\tau}_j^2 - \tau_j^2|$ \[ \max_{1 \le j \le p} \left(\frac{1}{\tilde{\tau}_j^2} \right) \le \frac{1}{\min_{1 \le j \le p} \tau_j^2 - \max_{1 \le j \le p} | \tilde{\tau}_j^2 - \tau_j^2|} \le \frac{1}{c - o_p (1)}, \] by Theorem 2(ii)(iii) under Assumptions 1, 5. Then by ((ref)) and Assumption 5(iii)
So the third term on right side of ((ref)) converges to zero faster in probability than the first right side term and the first term converges to zero faster than the second term in ((ref)) given Theorem 2. {\bf Q.E.D}
We now provide three corollaries for specific subcases of our general estimator. Note that we provide the specific case of adaptive PCR estimator of $\tau_j^2$, which is defined as $\hat{\tau}_{j, PCR}^2$. In that respect the rate $r_{w1}$ simplifies considerably. In adaptive PCR case these rates are with wpa1
See that rate ((ref)) can be obtained by seeing $\bar{r} \le K, \bar{\Psi}/\bar{\eta} \le 1, \bar{\eta} \ge \beta$ wpa1 as shown in Corollary 1. Corollary (ref) below follows through replacing $r_{w1}$ in Theorem 2 with $r_{w1,PCR}$.
Here we consider the second subcase $p-1<n$ scenario of RRE estimator of $\tau_j^2$. We define it as $\hat{\tau}_{RRE,j}^2$. We define the following rate which will be used instead of $r_{w1}$, wpa1, which can be deduced from Corollary 2 proof,
We now consider the third subcase of RRE estimator when $p-1>n$. Note that this subcase will be different than the others, and we use an estimator of $\tau_j^2$ which is different due to interpolation property of RRE in this case. First let us show the interpolation and how it affects the estimator $\tilde{\tau}_j^2$ proposed, and the need to modify. In case of $p-1>n$, RRE estimator becomes the $l_2$ norm interpolator. Define the minimum norm interpolator as in (1) of bsmw22, for each $j=1,\cdots, p$ \[ \hat{\alpha}_j:= argmin \{ \| \alpha_{j} \|_2: \| Y_{-j} \alpha_{j} - y_j \|_2= \min_{\gamma_{j}} \| Y_{-j} \gamma_{j} - y_j \|_2.\},\] Lemma 28 of bsmw22 shows $ \hat{\alpha}_j = Y_{-j}^+ y_j = \hat{\alpha}_{RRE,j}.$ Then when $p-1>n$, \[Y_{-j} \hat{\alpha}_j - y_j = Y_{-j} Y_{-j}^+ y_j - y_j = y_j - y_j = 0,\] with $Y_{-j} Y_{-j}^+= I_n $ by (136) of bsmw22. This is interpolation, and we cannot use the formula for $\tilde{\tau}_j^2$ since with interpolation it can be shown, $\tilde{\tau}_j^2=0$ in RRE in (11), with $p-1>n$ case. So we propose the following estimator, which can be considered a subcase of general $\tilde{\tau}_j^2$. Define
{\bf Proof of Corollary (ref)}. (i). By ((ref))((ref)) and triangle inequality
Next, Step 1 in the proof of Theorem 2 with (A.16), and (A.28) provides the result. (ii). Given Assumptions in part (i) with $\sqrt{K} d_{2n}/\sqrt{\bar{\xi}} \to 0$ provides the consistency. (iii). This is the same as in the proof of Theorem 2 (iii). {\bf Q.E.D.}
\setcounter{equation}{0} \setcounter{lemma}{0} \setcounter{table}{0}
In this part B, we provide a precision matrix formula and then provide two lemmata on tail inequalities.
{\bf Precision Matrix Formula Derivation}
Precision matrix relation to a hidden factor model context is established here for the first time in the literature and in detail. Define all assets in the portfolio as $y_t=(y_{jt}, Y_{-jt}')': p \times 1$. In case of $j=3$, this represents $y_{3t}$ in the third cell in the vector $y_t$ and $Y_{-3t}$ represents elements $1,2, 4,\cdots,p$ in $y_t$.
{\bf Proof of Lemma 1}. We will provide formula for $j$ th row and then stack all rows to get an expression. Before that define $\Sigma_{j,j}:= E [(y_{jt}-\mu_j)^2]$ and $\Sigma_{j,-j}:=E [(y_{jt} -\mu_j) (Y_{-jt}- \mu_{-j})']: 1 \times p-1$ and represents $j$ th row of $\Sigma$ with $j$ th element missing, and by symmetricity of $\Sigma$, $\Sigma_{j,-j}=(\Sigma_{-j,j})'$. Specifically, define $\Sigma_{-j,j}:=E [ (Y_{-jt} - \mu_{-j}) ( y_{jt} - \mu_j) ]: p-1 \times 1$, as the $j$ th column of $\Sigma$ where $j$ th element is missing. In the same way we define $\Sigma_{-j,-j}:= E [ (Y_{-jt} - \mu_{-j}) (Y_{-jt} - \mu_{-j})']: p-1 \times p-1$ where this matrix is $j$ the row and column deleted from $\Sigma$. By Section 2.1 of yuan2010, the main diagonal elements are given by the scalar
and the $j$ th row without the $j$ th element is:
Now we relate these quantities to a linear regression context. Define the following minimizer $ \alpha_{j}^*:= argmin_{\alpha_{j} \in R^{p-1}} E [ (y_{jt} - \mu_j) - (Y_{-jt} - \mu_{-j})' \alpha_{j}]^2,$ with given zero mean $(y_{jt}-\mu_j ), (Y_{-jt} - \mu_{-j})$ and finite second moments for $y_{jt}, Y_{-jt}$, Lemma 27 in bsmw22 shows that
where $\Sigma_{-j,-j}^+$ is the Moore-Penrose inverse of $\Sigma_{-j,-j}$. First,
by (3) and Assumption 1(iv). Then
by Assumption 2-3. ((ref)) simplifies to
Define $\eta_{jt}:= (y_{jt} - \mu_j) - (Y_{-jt} - \mu_{-j})' \alpha_{j}^*$ for $j=1,\cdots, p$. Now consider
where the first equality uses $\eta_{jt}$ definition, and the second equality uses definitions of moment matrices, and the third one uses ((ref)). Given ((ref)) we can write the following infeasible linear regression model,
Now we relate ((ref)) using ((ref)). Then
and then define $\tau_j^2:= E \eta_{jt}^2$. See that by ((ref))((ref)) \[ E [y_{jt}- \mu_j] ^2 = (\alpha_{j}^*)' E [(Y_{-jt} - \mu_{-j}) (Y_{-jt} - \mu_{-j})'] \alpha_{j}^* + E [\eta_{jt}^2],\] which implies by definitions $\Sigma_{j,j}:= E [ y_{jt}- \mu_j ]^2, \Sigma_{-j,-j}:= E [ (Y_{-jt} - \mu_{-j}) (Y_{-jt} - \mu_{-j})']$
where we use ((ref)) for the second equality and ((ref)) for the third one. Clearly, also by ((ref))((ref))
We can form the $j$ th row of precision matrix by using $j$ the main diagonal term, via ((ref))((ref))
and the rest of that row, via ((ref))((ref))
We start simplifying $\alpha_{j}^*$ formula in case of hidden factors as in (2)(3). The proof for (i) here is also a detailed version of the proof of Lemma 7 and (57) of bsmw22. This is repeated to show that our scenario fits into their framework. Note that $\Sigma_f:= E (f_t - E f_t) (f_t - E f_t)'$. In that respect in ((ref)), by(2)-(3)
by Assumption 1(iv). Then the same analysis in ((ref)) provides with $\bar{A}_{-j}:= A_{-j} \Sigma_f^{1/2}$,
Using Sherman-Morrison-Woodbury formula in p.19 of hj2013
where we use Assumption 3, $\Sigma_f$ is positive definite. Note also that
since $A_{-j}, \Sigma_f$ has full rank K by Assumption 3, so (135) of bsmw22 provides the last result ((ref)). Define $\bar{a}_j:= \Sigma_f^{1/2} a_j$ and ((ref))((ref))
where we use ((ref)) for the second equality and for the third equality ((ref)) and $\bar{A}_{-j}$ definition. By using $\bar{G}_j$ definition before Lemma 1 and using ((ref)) on right side of ((ref)) for $\Sigma_{-j,-j}^{-1}$
Use ((ref)) in ((ref))
by ((ref)) for the second equality. One of the other elements in the precision matrix is the reciprocal of the main diagonal term, $\tau_j^2$ in ((ref)). We provide a formula for that in terms of hidden factor regression model, which will be new. In that respect, by (2)(3), Assumption 1
Similarly, using (2)(3), Assumption 1
Then combine ((ref))((ref))((ref)) in ((ref))
(i)-(ii). By ((ref)) and ((ref)), and using them in ((ref))((ref)) we have the desired result. {\bf Q.E.D.}
{\bf Tail Inequalities}
Here we first provide a tail inequality for quadratic form of a mean-zero subgaussian random vector. This is Lemma 21 in bbsmw21.
where $tr(H)$ represents trace of $H$, and $\gamma_e$ subgaussian random vector is defined in Assumption 4. Now, we provide a lemma for a lower bound inequality for singular value of a specific random matrix for RRE when $p-1>n$. This extends the zero factor mean in Proposition 14 of bsmw22 to a nonzero mean case below.
{\bf Proof of Lemma (ref)}. So first we use (89) in bsmw22 and then equation below that on p.42 of proof of Proposition 14 of bsmw22 but we replace Theorem 4.6.1 of v2019 in that proof with Theorem 5.3.9 of v2012, (set $t=\sqrt{n}$) to take into account nonzero mean of factors. The rest carries over from the proof of Proposition 14 of bsmw22.{\bf Q.E.D.} \\
\setcounter{equation}{0} \setcounter{lemma}{0} \setcounter{table}{0}
In this part C, we cover simulations and extra empirics without transaction costs and discussion of literature and hidden factors.
{\bf Simulations}
By (2) and (3) and Assumption 4, for each $t=1,...,n$, we have: \[ y_{t}= A f_t + u_t, \] where $y_t = (y_{jt},Y_{-jt}')': p \times 1$ is the vector of excess return, $ A =(a_j,A'_{-j}): p\times K$ is the matrix of factor loadings, and $u_t=(u_{jt},U_{-jt})': p\times 1$ vector of errors. Let hidden-factors be normally distributed and $f_t \sim \mathcal{N}(\mu_f, \Sigma_f)$, where $\mu_f$ is the mean vector of the hidden factor, $K\times 1$ vector with each element equal to 0.5, and $\Sigma_f$ is the hidden-factor covariance matrix. Following bbsmw21, we set the diagonal elements of $\Sigma_f$ as the $K-$ length sequence from 2.5 to 3 with equal increments. For example, if $K=3$, then $\Sigma_f = diag(2.5,2.75,3)$. The the off diagonal, $i,j$ th elements of the factor covariance matrix are defined as: \[ [\Sigma_f]_{i,j} = (-1)^{i+j} (0.3)^{|i-j|} \min( [\Sigma_{f}]_{i,i}, [\Sigma_{f}]_{j,j} ). \] Each element of $A$ is drawn independently from the normal distribution $\mathcal{N} (0.5,1/\sqrt{K})$. The idiosyncratic error, $u_t$, is drawn from the normal distribution, $\mathcal{N}(0, \Sigma_u)$. In our design, we define $\Sigma_u:p\times p$ as a diagonal matrix with diagonal elements sampled from uniform distribution with $\text{Unif}(1,3) \times \log(p)$. With that we capture noisy setups in complex models and can understand the effect of noise on adaptive PCR and RRE with $p-1>n$ specifically. The results for $K=3, 5$ (true number of hidden factors) are presented in Table (ref). However note that we also use $K=20$ to see the effects of fitting larger number of factors when the true DGP has few factors.
We aim to address several questions through simulations. First, as described in the theorems, we investigate whether the estimation errors from our methods decrease when $\gamma = p/n$ is fixed and both $p$ and $n$ jointly increase \footnote{$\gamma=p/n=0.5$ in low-dimensional set up and $\gamma = p/n=1.5$ in high-dimensional set up }. Second, we compare the performance of our new methods with existing methods. Third, we examine the impact of increasing the number of hidden factors, and also see whether there is a symptom of double descent in RRE estimator. We consider estimation error for the precision matrix in Table. We consider the nonlinear shrinkage method of lw2017(NL-LW), POET method from fan2013, and the nodewise (NW) method from caner2019. Our estimators are: RRE, PCR-3F, PCR-5F, PCR-20F and PCR-Adaptive, in which they are RRE estimator, Principal component with fixed 3, 5, or 20 hidden factors, and an adaptive method to select the number of hidden factors, respectively. We do not include caner2022 residual nodewise since it only uses observed factors. Table (ref) shows the simulation results. We ran 100 iterations in each simulation set up. First, the table shows that our method achieves consistency as stated in Corollary 4 for adaptive PCR. Analyzing Table (ref) with $K=3$, we observe that for $p=n/2$, with $n=100$, our Adaptive-PCR has an estimation error at 0.0583. This error declines to 0.0251 when $p=n/2$ and $n=400$. Next, we consider which method achieves the smallest estimation errors. Our Adaptive-PCR dominates all other techniques except fixed PCR-K. As a side note to all of this, in RRE estimator, we see that estimation error for the low dimensional case is always higher than the double of high dimensional error scenario. This can be a symptom of double descent in RRE.
{\bf Extra Empirics}
Here we provide the results of the empirics without transaction costs in Tables C.2-C.4. $SD$ is the standard deviation of the portfolio, and $Sharpe$ is the Sharpe ratio. Our methods still achieve the best Sharpe Ratio, turnover, standard deviation and return in Tables C.2-C.3 low dimension and moderate dimension cases. Table C.4, again our best method dominates the others in Return, Sharpe Ratio and Standard Deviation. In Turnover, nodewise has the best record. The first 6 rows represent the models in the literature, and the rest is our models. The bold faced numbers represent the winner in Sharpe Ratio.
{\bf Discussion of Literature}
Several studies in the statistics literature have studied precision matrix estimation. An excellent survey is by crz2016.
In Section 3 of their paper, they provide two major approaches to precision matrix estimation. The first approach is neighborhood-based and closely related to nodewise regression, where a lasso regression is performed for each variable against all others. Alternatives to lasso, such as the Dantzig selector, can also be used within this framework (see cl2011, caner2019). The second major approach is based on penalized likelihood estimation. As shown by blp2020, lasso-type penalties in this context can induce shrinkage bias and lead to overly sparse precision matrix estimates, with many zero entries in each row. In a penalized likelihood based setting, they reformulate the problem as a mixed-integer optimization problem, which is solvable using convex optimization techniques. This approach produces better results than lasso based approaches. Sparsity of precision matrix is used in these two main approaches. A distinct approach, commonly used in finance and statistics, derives the precision matrix by inverting the covariance matrix using the Sherman–Morrison–Woodbury (SMW) formula. In this framework, sparsity is assumed in the covariance matrix of the idiosyncratic errors, while asset returns are modeled through a factor structure (see fan2013, caner2022). Our approach is different from the approaches discussed in the literature. First of all, we are not assuming exact sparsity in the precision matrix of outcomes. We are also not using SMW formula to invert the covariance matrix.
This allows us to bypass imposing a structure on the outcomes utilized in the covariance inversion. Instead we use a new precision matrix formula without sparsity and rely on hidden factors to generate each outcome. This effectively ensures that we do not benefit from the SMW matrix inversion formula. To be clear, we use model estimation-free dense estimation, but consistency in high dimensions is achieved through hidden factor based dimension reduction without a need to estimate factors or identify them.
There are also other technical and conceptual differences that are worth highlighting. First, our proposed class of estimators is general and in addition to the PCR based ones it also includes the ridgeless regression estimator (RRE). Second, our signal assumption is much weaker as we do not need the minimum signal to grow at the same rate as the number of assets in the portfolio. Moreover, as the root of our approach resembles a finite sample analysis, it also allows to explicitly tie all the estimation errors to signal-to-noise ratio thereby making its importance in the rate of convergence estimates clearly visible.
The precision matrix estimation is different than the risk analysis of a single outcome-asset. We also provide an explicit connection in our estimation error upper bounds to signal-to-noise ratio. The results in subcases are different also in a risk analysis versus precision matrix estimation. bbsmw21, bsmw22 show that in certain very high dimensional cases, ridgeless regression (RRE) can perform better than adaptive PCR in terms of risk asymptotically. However, in case of estimation errors of rows of precision matrix errors (in $l_2$ norm) we show adaptive PCR can perform better than RRE even in certain low-moderate dimension cases. These are due to different proofs for risk versus precision matrix estimation.
{\bf Hidden Factors}
The reason we use factor models is for dimension reduction. First of all, we show that precision matrix estimation is related to linear regression and that can be related to hidden factors. Each economic or financial variable can be related to latent factors and this can be thought as confounding variables as in causality related literature such as in kv2023, ap2009, ir2015. Also the true variables of interest may be measured with errors, and latent factors can represent the "true" variables rather than the reported ones, especially in emerging markets. Variable such as culture, sentiment, market can be extremely valuable in certain macro, financial data sets. These hidden factors in a linear model can be used to explain international stock returns such as in Singapore, Malaysia, USA as seen in bn2004.
Another key aspect of using hidden factors is their effectiveness as a tool for dimension reduction in high-dimensional econometrics. This stems from the fact that the number of observed outcomes typically exceeds the number of underlying hidden factors. In our paper, we allow {\it both} the number of outcomes and the number of hidden factors to grow with the sample size.