Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
98,514 characters · 17 sections · 78 citation commands
High-dimensional canonical correlation analysis
\address[Anna Bykhovskaya]{Duke University} \email{[email removed]}
\address[Vadim Gorin]{University of California at Berkeley} \email{[email removed]}
\@startsection{section}{1} \z@{1.0\linespacing\@plus\linespacing}{0.5\linespacing} {\normalfont}{Introduction}
Canonical correlation analysis (CCA) is a classical statistical analysis method used to find a common structure between two data sets. It was first introduced in harold1936relations and remains in active use today. CCA can be viewed as a generalization of principal component analysis (PCA) from one set of variables to two: in PCA the goal is to find a signal (or factor) in a large matrix with a large amount of noise, while CCA searches for a common signal among two large matrices with a large amount of noise. We refer the reader to the textbooks thompson1984canonical,gittins1985canonical,anderson1958introduction, and muirhead2009aspects for introductions to CCA.
There are numerous applications of CCA in the social and natural sciences. In genomics CCA is used to find commonalities between multiple assays coming from the same set of individuals (see, e.g., witten2009extensions and references therein). In neuroscience it is used to match brain measurements with behavioral and medical scores (e.g., wang2020finding,zhuang2020technical). Similarly, in ecology CCA is used to correlate characteristics of living species with those of their habitats (e.g., gittins1985canonical and simon1998assessing). In econometrics CCA is used to select the number of factors driving the behaviour of various processes (see, e.g., breitung2013canonical,andreou2019inference, choi2021canonical, and franchi2023estimating). Further, in time series econometrics CCA appears in cointegration analysis: a strong correlation between first differences (which are stationary) and lags (which are nonstationary) indicates the presence of cointegration (cf. johansen1988).
Beyond the cases outlined above, there are numerous settings where CCA can provide a meaningful alternative approach and complement previous methods. For example, in sociology and applied economics CCA can provide a framework for analysis of the relationships between various measures of socioeconomic characteristics and investments, on one side, and labour market outcomes and associated skills, on the other side (cf. the production function in cunha2010estimating and the results of papageorge2020genes). In finance and financial econometrics an important question, which can be addressed via CCA, is whether different types of stocks are highly correlated and, thus, can be used to create portfolios for pairs trading (e.g., one could exploit a correlation between stocks from different industries or between traditional stocks and cryptocurrencies). Further, this pertains to finding a maximally predictable portfolio (see lo1997maximizing for seminal results and goulet2023maximally and references therein for most recent developments); recently firoozye2023canonical applied CCA in a closely related setting.
One may notice that most of the above examples ideally require working with very large (across all dimensions) data sets. However, the high-dimensional machinery for CCA (in contrast to that for PCA) is still very much underdeveloped. In particular, while all the above scenarios crucially require estimates of canonical vectors (or variables), the literature has not provided any theoretic guarantees for the consistency of such estimates in general settings when we do not know anything about their structure and do not impose further restrictions such as sparsity, smoothness, etc. Thus, the development of new tools and theory suitable for high-dimensional settings remains vital. In this paper we aim to provide such techniques. Our key result quantifies the inconsistency in CCA by providing exact formulas to measure how far the sample canonical vectors are from their population counterparts. These formulas are easy to evaluate and are readily applicable for practitioners seeking to assess the quality of their estimates.
Formally, if we have two random vectors $\mathbf u\in\mathbb{R}^K$ and $\mathbf v\in\mathbb{R}^M$, then the first goal of CCA is to find deterministic vectors $\boldsymbol\alpha\in\mathbb{R}^K,\,\boldsymbol\beta\in\mathbb{R}^M$ that maximize the correlation between $\mathbf u^\mathsf T\boldsymbol\alpha$ and $\mathbf v^\mathsf T\boldsymbol\beta$, where $^\mathsf T$ is the matrix transposition. That is, we are trying to find highly correlated combinations of coordinates of $\mathbf u$ and $\mathbf v$. The maximal correlation value is called the largest canonical correlation, and the corresponding $\mathbf u^\mathsf T\boldsymbol\alpha$ and $\mathbf v^\mathsf T\boldsymbol\beta$ form the first pair of canonical variables. Other canonical correlations (there are $\min(K,M)$ of them) can be found iteratively. Equivalently, squares of all canonical correlations can be found as the eigenvalues of $K\times K$ matrix $(\mathbb E \mathbf u \mathbf u^\mathsf T)^{-1} (\mathbb E\mathbf u \mathbf v^\mathsf T) (\mathbb E\mathbf v \mathbf v^\mathsf T)^{-1} (\mathbb E\mathbf v \mathbf u^\mathsf T)$ or as the eigenvalues of the ${M\times M}$ matrix $(\mathbb E\mathbf v \mathbf v^\mathsf T)^{-1} (\mathbb E\mathbf v \mathbf u^\mathsf T) (\mathbb E\mathbf u \mathbf u^\mathsf T)^{-1} (\mathbb E\mathbf u \mathbf v^\mathsf T)$. Although proving it is not straightforward, the equivalence of the two procedures is a known fact in linear algebra; we also explain it in Appendix (ref). By multiplying the corresponding eigenvectors of the former and the latter matrices by $\mathbf u^\mathsf T$ and $\mathbf v^\mathsf T$, respectively, we obtain all the pairs of canonical variables. We call the setup with two random vectors a population formulation.
In contrast, in the sample formulation, two vectors are replaced by two matrices (e.g., we observe $S$ samples of $\mathbf u$ and $S$ samples of $\mathbf v$). Let those samples be $\mathbf U$ ($K\times S$ matrix) and $\mathbf V$ ($M\times S$ matrix). The finite sample analogues of the canonical correlations, the sample canonical correlations, are computed by maximizing the sample correlations between the $S$-dimensional vectors $\mathbf U^\mathsf T\widehat \boldsymbol\alpha$ and $\mathbf V^\mathsf T\widehat \boldsymbol\beta$. Equivalently, their squares are eigenvalues of $(\mathbf U\mathbf U^\mathsf T)^{-1}\mathbf U\mathbf V^\mathsf T(\mathbf V\mathbf V^\mathsf T)^{-1}\mathbf V\mathbf U^\mathsf T$. The canonical correlation vectors are eigenvectors of the preceding product of matrices and another similar expression. The corresponding $\mathbf U^\mathsf T\widehat \boldsymbol\alpha$ and $\mathbf V^\mathsf T\widehat \boldsymbol\beta$ are called sample canonical variables.
Assume that the columns of $\mathbf U$ and $\mathbf V$ are independent samples from the joint law of $(\mathbf u,\mathbf v)$. Then, when $S$ is large while $K$ and $M$ are fixed, the sample covariances of $\mathbf U$ and $\mathbf V$ as well as their cross-covariances are consistent estimators of their population counterparts. Therefore, $(\mathbf U\mathbf U^\mathsf T)^{-1}\mathbf U\mathbf V^\mathsf T(\mathbf V\mathbf V^\mathsf T)^{-1}\mathbf V\mathbf U^\mathsf T$ is a consistent estimate of its population analogue $(\mathbb E \mathbf u \mathbf u^\mathsf T)^{-1} (\mathbb E\mathbf u \mathbf v^\mathsf T) (\mathbb E\mathbf v \mathbf v^\mathsf T)^{-1} (\mathbb E\mathbf v \mathbf u^\mathsf T)$, and we obtain consistent estimates of the squared correlation coefficients and corresponding vectors. The exact distribution of the sample canonical correlations and variables and their asymptotic behavior in the fixed $K$ and $M$ regime have attracted the attention of many researchers, with the seminal results going back to hsu1939distribution,hsu1941limiting and constantine1963some and the latest corrections to the results on canonical variables being much more recent (see anderson1999asymptotic).
The properties of CCA when all three dimensions $S,K,M$ are large and comparable have not yet been properly analyzed. The remarkable seminal result in this direction is that of wachter1980limiting, who studies the case when $\mathbf u$ and $\mathbf v$ are uncorrelated and columns $\mathbf U$ and $\mathbf V$ are independent samples of $\mathbf u$ and $\mathbf v$, respectively. wachter1980limiting shows that despite $\mathbf U$ and $\mathbf V$ being independent, the empirical distribution of the sample canonical correlations has a nontrivial limit as $S,K,M\to\infty$ with $S/K\to \tau_K$, $S/M\to\tau_M$. An independent rediscovery of this result is presented in bouchaud2007large. In particular, the majority of the correlations are bounded away from $0$. The conceptual conclusion is that in the large $S,K,M$ setting the sample canonical correlations do not deliver consistent estimates of the population canonical correlations, which in this case all equal zero due to $\mathbf u$ and $\mathbf v$ being independent. These results have been extended very recently in bao2019canonical, yang2022limiting, and ma2023sample to the case when some (but finitely many as the dimensions grow) population correlations are allowed to be nonzero. Conceptually similar results are obtained: the sample squared canonical correlations are always larger and bounded away from their population counterparts. However, there is an explicit dependence between the sample and population canonical correlations, and therefore, knowing the former allows one to reconstruct the latter. Hence, it is possible to identify nontrivial population canonical correlations.
No progress has been achieved so far to tackle the challenging properties of the corresponding vectors of canonical variables in the regime in which $S,K,M$ are large and proportional to each other. In practice squared correlation coefficients can be used to test whether there is a common signal between two data sets (i.e., the largest coefficient is statistically larger than zero). However, vectors are required to be able to say something about the nature of this commonality: where it comes from, what it represents, etc. The asymptotic analysis of these vectors is the central topic of the present paper.
To address the problem of estimating CCA vectors, we first assume that $\mathbf U$ and $\mathbf V$ are Gaussian and independent across $S$ samples of $\mathbf u$ and $\mathbf v$ but allow $\mathbf u$ to be correlated with $\mathbf v$. We start with $\mathbf u$ and $\mathbf v$ having exactly one nonzero canonical correlation and refer to it as the signal, while the remaining uncorrelated parts of $\mathbf u$ and $\mathbf v$ are viewed as the noise. We characterize (in terms of the parameters $S,K,M$ and the covariances between coordinates of $\mathbf u$ and $\mathbf v$) when we can detect a nonzero canonical correlation, i.e., when there is enough signal in the data compared to the pure noise case. Next, we develop formulas for sample $\widehat \boldsymbol\alpha$ and $\widehat \boldsymbol\beta$, which allow us to show that, for finite ratios $S/K,\,S/M$, one cannot consistently estimate true values $\boldsymbol\alpha$ and $\boldsymbol\beta$. The estimated canonical variables $\mathbf U^\mathsf T\widehat \boldsymbol\alpha$ and $\mathbf V^\mathsf T\widehat \boldsymbol\beta$ will always lie on cones around the true population canonical variables $\mathbf U^\mathsf T\boldsymbol\alpha$ and $\mathbf V^\mathsf T\boldsymbol\beta$. We provide explicit formulas for the width of these cones. They show that the cones shrink as we increase the ratios $S/K,\,S/M$ and that ultimately consistency is restored in the limit.
Next, we provide various generalizations from the i.i.d. Gaussian setting with one signal. First, motivated by the fact that often data are far from normal (especially in financial applications), we relax our assumptions to accommodate any distribution as long as its first four moments match the Gaussian moments. Second, we allow for correlated realizations, treating the signal part (i.e., the signal is not independent across $S$) and the noise part (i.e., the noise is not independent across $S$) separately.
We do not cover the case of simultaneously correlated structures for the signal and noise in this paper. We remark that such setting could be of interest for the cointegration testing, where the no-signal (or no-cointegration) case has been studied in recent works such as onatski_ecta, onatski2019extreme and BG1, BG2.
Finally, we extend the machinery to allow for multiple vectors $\boldsymbol\alpha$ and $\boldsymbol\beta$, i.e.,\ multiple nonzero canonical correlations in the population setting. In other words, we allow the signal to be of any finite rank, and the vectors $\boldsymbol\alpha$ and $\boldsymbol\beta$ are replaced by matrices.
While the precise measure of CCA inconsistency was not available before, researchers were aware that the quality of the estimates delivered through sample canonical correlations and variables deteriorates when the ratios between the number of samples and dimension of the space, $S/K$ and $S/M$, are not large. Drawing intuition from many other high-dimensional problems, it is common to impose additional restrictions on canonical variables (or other model parameters) to address this issue. For example, gao2015minimax,gao2017sparse analyze CCA under sparsity assumptions, while tuzhilina2023canonical (see also references therein) explore other regularization methods. These approches help reduce dimensionality and often produce easily interpretable results. However, this advantage often comes at the cost of complex numerical computations and the loss of some desirable properties. In contrast, without restrictions, sample canonical variables are maximum likelihood estimators of the population variables, offering certain optimality guarantees.
From a practical perspective, it is crucial to ensure that the data satisfies the assumptions being imposed. For example, one must assess whether sparsity is a reasonable assumption for a given data set --- a topic that remains debated. giannone2021economic reports that many large data sets in macroeconomics, microeconomics, and finance exhibit dense rather than sparse structures. Similarly, in our financial data set example in Section (ref), most coefficients of the estimated leading canonical variables are positive and of comparable magnitude, making the sparsity assumption unrealistic. Therefore, it is essential to develop methods, such as those in this paper, that analyze CCA in settings with only minimal assumptions.
CCA is closely connected to the literature on factors. First, as mentioned in Section (ref), some methods for factor estimation rely on CCA. The idea is that the correlation between different sets of variables, which is captured by CCA, can be fundamentally related to the presence and properties of factors. For instance, one can distinguish between common and group-specific factors (see, e.g., andreou2019inference and choi2021canonical). franchi2023estimating relies on the CCA between levels $X_t$ and sums of $X_t$ to infer the dimension of non-stationary subspace in $X_t$.
Another point of view on the interconnection between factors and CCA comes from the interpretation of canonical variables as common explanatory factors between two data sets. The main difference is that factors operate with a single matrix/data set while canonical correlations require two matrices/data sets. The main approach for finding factors is via PCA and its modifications (cf.\ bai2008large,stock2011dynamic,fan2016overview and references therein). There are many similarities in the spirit of the modeling and results between CCA and PCA. However, CCA as considered in our present paper is much more challenging and involved than PCA. This is perhaps an inherent feature of the CCA procedure, which involves matrix inversion, a much more complicated operation to carry out than simply calculating a product.\footnote{The reader is invited to compare the difficulty of the answers in our Theorem (ref) with that of the parallel Theorems 2.9 and 2.10 in benaych2012singular for the PCA setting.}
In statistics and probability factors (signals in the data) are often modeled as a spiked random matrix, a term going back to johnstone2001distribution. A spiked random matrix is a sum of a full-rank noise matrix and a small-rank signal matrix. Often this is embedded in the PCA setting: one observes a matrix $C=A+B$, where $A$ is treated as a low-rank signal and $B$ is treated as noise, and tries to reconstruct $A$ from observing $C$ through its singular values and vectors. $A$, $B$, and $C$ here are rectangular matrices with both dimensions assumed to be large. In econometrics $A$ represents a product of factors and their loadings, while $B$ is composed of idiosyncratic errors.
The spiked covariance model, which begins with a positive-definite covariance matrix $\Omega=\Sigma+\sigma^2 I_N$, can be viewed as a special case of the spiked random matrix framework. In this model $\Sigma$ is a low-rank signal matrix, $I_N$ is the $N\times N$ identity matrix and $\sigma^2$ is a real parameter that quantifies the noise strength. One observes $N\times S$ matrix $X$, whose columns are i.i.d.\ samples of a random vector with covariance $\Omega$. The goal is to reconstruct $\Omega$ (equivalently, $\Sigma$) from the sample covariance matrix $\frac{1}{S}X X^\mathsf T$. For such a setting, baik2005phase and baik2006eigenvalues discovered the phenomenon now known as the BBP phase transition: when the signal is small while the strength of the noise is large, one cannot detect the presence of the signal matrix $\Sigma$ from eigenvalues of $\frac{1}{S}X X^\mathsf T$ , while for a large signal, the largest singular values of $\frac{1}{S}X X^\mathsf T$ connect to those of $\Sigma$ through explicit formulas. The corresponding singular vectors represent an inconsistent estimate of $\Sigma$, in parallel to what we observe in the CCA setting (see johnstone2009consistency,paul2007asymptotics,nadler2008finite and references therein for earlier work in the learning theory literature in physics). If the rank of the signal matrix $\Sigma$ is not restricted to be small, a popular approach for estimating it from $\frac{1}{S}X X^\mathsf T$ is a method known as “shrinkage”. While this approach does not yield consistent estimators, it is optimal in a certain sense, see ledoit2022power for a review.
The phase transition analogous to the BBP has also been observed in the econometrics literature on factor models. When factors are weakly influential, their identification becomes possible only if the signal's strength is above a threshold (see onatski2012asymptotics and references therein). Moreover, when factors become stronger, estimation consistency is restored.
Closely related to the CCA framework is a spiked $F$-matrix model, explored recently in, e.g., gavish2023matrix,hou2023spiked. An $F$-matrix, arising from the $F$-test, involves the ratio of two large matrices. In the benchmark case of no spikes (or no signal), there exists a direct correspondence between an $F$-matrix and the random matrices used in CCA. When signals are present in the data, a more delicate connection between non-central $F$-matrices and CCA has been established in recent work (see bai2022limiting and zhang2023limiting). However, this connection applies only to eigenvalues, not eigenvectors, leaving it unclear how to extend these results to analyze the vectors of canonical variables, which are the focus of our study.
The remainder of the paper is organised as follows. Section (ref) discusses the basic setting of i.i.d. normal vectors. Various generalizations (to the cases with nonnormal errors, correlated observations, and multiple signals) are presented in Section (ref). Section (ref) provides two empirical illustrations of our results. Finally, Section (ref) concludes. Proofs, additional computer simulations, and data details are given in appendices.
\@startsection{section}{1} \z@{1.0\linespacing\@plus\linespacing}{0.5\linespacing} {\normalfont}{Basic framework}
Let $\mathbf u=(u_1,\dots,u_K)^\mathsf T$ and $\mathbf v=(v_1,\dots,v_M)^\mathsf T$ be $K$- and $M$- dimensional random vectors with zero means and nondegenerate covariance matrices, where $\,^\mathsf T$ here and below denotes matrix transposition. Without loss of generality, we assume $K\le M$.
The pair $(\mathbf u^\mathsf T\boldsymbol\alpha, \mathbf v^\mathsf T\boldsymbol\beta)$ represents the signal in the data, while the remaining part is treated as the noise, which is uncorrelated with the signal. Ultimately, we are interested in the signal and would like to filter out the influence of the noise.
The number $r^2$ and the vectors $\boldsymbol\alpha$ and $\boldsymbol\beta$ of Assumption (ref) can be read from the covariance structure of $\mathbf u$ and $\mathbf v$, as the following lemma explains.
In the setting of Example (ref), the statement of Lemma (ref) is straightforward. The general situation is reduced to this example by a change of basis.
Let $\mathbf W=
$ be a $(K+M)\times S$ matrix composed of $S$ independent samples of $
$. The matrix $\mathbf W$ represents observed data, which comes from the population setting \ref{Section_basic_population_setting}. We are interested in finding the signal $(\mathbf u^\mathsf T\boldsymbol\alpha, \mathbf v^\mathsf T\boldsymbol\beta)$ or vectors $(\boldsymbol\alpha,\boldsymbol\beta)$. In the sample setting they come from the squared sample canonical correlations and their corresponding vectors. The vectors for the largest correlation represent the sample analogues of $\boldsymbol\alpha$ and $\boldsymbol\beta$. The following definition is motivated as a sample version of Lemma (ref).
In the sample setting of Section (ref), we are going to assume that $S/K\to \tau_K$, $S/M\to\tau_M$, and $r^2\to \rho^2$. We define six constants which depend on these parameters:
The constants play the following role. First, by wachter1980limiting and Johnstone_Jacobi, most of the squared sample canonical correlations should belong to the $[\lambda_-,\lambda_+]$ interval. wachter1980limiting shows that as $S\to\infty$, the empirical distribution of the squared sample canonical correlations between independent data sets converges to the Wachter distribution, which has the density $\omega_{\tau_K,\tau_M}(x)=\frac{\tau_K}{2\pi} \frac{\sqrt{(x-\lambda_-)(\lambda_+-x)}}{x (1-x)} \mathbf 1_{[\lambda_-,\lambda_+]}(x)$, see Appendix (ref) for more details and Figure (ref) for an illustration. This fact remains true for our setting, as follows from Lemma (ref) in Appendix.
Second, bao2019canonical have shown that the largest squared canonical correlation undergoes a phase transition depending on the value of $\rho^2$: if $\rho^2>\rho_c^2$, then the largest squared canonical correlation is close to $z_\rho>\lambda_+$; otherwise, it is close to $\lambda_+$. In the former case, this squared canonical correlation can be clearly seen on the histogram as an outlier from the Wachter distribution, see Figure (ref). We focus on the case $\rho^2>\rho_c^2$ and examine how close the sample canonical variables corresponding to the largest correlation are to their population counterparts by analyzing the angles between them. The constants $\mathfrak s_x$ and $\mathfrak s_y$ provide the answers, as stated in the following main theorem.
Figure (ref) illustrates the dependence of $\mathfrak s_x,\mathfrak s_y,z_{\rho}$ on the squared correlation coefficient $\rho^2$. Notice that $z_\rho$ as a function of $\rho^2$ has a minimum at the cutoff $\rho^2=\rho_c^2$ and is monotone increasing above the cutoff. The values of $\mathfrak s_x$ and $\mathfrak s_y$ are between $0$ and $1$ for $\rho^2$ above the same cutoff. These properties continue to hold for general values of $\tau_K,\tau_M>1$.
Figure (ref) shows results from a single simulation (for each fixed value $\rho^2$, we run one simulation) vs. theoretical predictions. As we can see, the simulated path is very close to the theoretical one, with largest discrepancy around the cutoff $\rho_c^2$. We also report in Appendix (ref) analogous results obtained from many instead of a single Monte Carlo simulations, as well as the finite sample effects on the convergence of angles.
The conditions $\tau_K>1$, $\tau_M>1$, and $\tau_M^{-1}+\tau_K^{-1}<1$ are not artifacts of our proofs, but rather conceptual restrictions of CCA. Suppose that $\tau_M\leq1$, so that $M\geq S$. Then, almost surely, the rows of $M\times S$ matrix $\mathbf V$ span the entire $S$--dimensional space. Therefore, the sample canonical correlations are all equal to $1$ and do not convey information about the population setting (the matrices in Definition (ref) are not invertible in this case, and one should use the equivalent definition of Lemma (ref)).
Similarly, if $\tau_K>1$, $\tau_M>1$, but $\tau_M^{-1}+\tau_K^{-1}>1$, which means $K<S$, $M<S$, but $K+M>S$, then any two subspaces of dimensions $K$ and $M$ must intersect in an $S$--dimensional vector space. The dimension of the intersection is at least $K+M-S$. Hence, in this situation $K+M-S$ largest canonical correlations are equal to $1$ (again, this can be seen from Lemma (ref) and subsequent discussion), no matter what the parameters in the population setting are. This makes the largest canonical correlation useless. Nevertheless, in this situation other sample canonical correlations remain bounded away from $1$ and potentially might be used to extract information about $\rho^2$. Yet, this is still an open question. The condition $\tau_M^{-1}+\tau_K^{-1}<1$ is very common in CCA literature, e.g., it is also imposed in bao2019canonical and yang2022limiting.
Finally, in the boundary case $\tau_K^{-1}+\tau_M^{-1}=1$, we have $\lambda_+=1$ and it becomes impossible to have sample canonical correlations larger than $\lambda_+$.
There are several aspects of Theorem (ref) worth emphasizing. First, in practice, when one is working with real data, the true value of $\rho$ is unknown, and the results of Theorem (ref) should be applied in the following way.
Given that the model matches the data, by wachter1980limiting and Johnstone_Jacobi, the histogram of the squared canonical correlations should resemble the Wachter distribution supported on the $[\lambda_-,\lambda_+]$ interval. If there is a gap between the largest canonical correlation $\lambda_1$ and $\lambda_+$, as in Figure (ref), then in line with Eq. (ref), one can take $\lambda_1$ as an approximation of $z_\rho$. Treating $z_\rho$ as known and approximating $\tau_K$, $\tau_M$ with $S/K$, $S/M$, Eq. (ref) becomes a quadratic equation in $\rho^2$. Solving it (using $ \rho_c^2\le \rho^2\le 1$ to choose the correct root out of the two, cf.\ Figure (ref)), we obtain an estimate for $\rho^2$, and further plugging into (ref) and (ref) and using (ref) and (ref), we obtain an estimate for the angle between $\mathbf x$ and $\widehat \mathbf x$ or between $\mathbf y$ and $\widehat \mathbf y$. If several canonical correlations larger than $\lambda_+$ are observed, then one needs to use an extension of Theorem (ref) provided in Theorem (ref): for each of these canonical correlations one can use exactly the same procedure as just outlined.
Next, the angles $\theta_x$, $\theta_y$ and the limiting value $z_\rho$ for the largest squared canonical correlation depend on $\rho$, $\tau_K$, and $\tau_M$ in a nontrivial fashion, leading to important features of the formulas (ref), (ref), and (ref):
The final essential aspect of Theorem (ref) is the choice of the angles $\theta_x$ (between $\widehat \mathbf x$ and $\mathbf x$) and $\theta_y$ (between $\widehat \mathbf y$ and $\mathbf y$) as the measure of the quality of approximations of $\widehat\boldsymbol\alpha$ and $\widehat\boldsymbol\beta$ for $\boldsymbol\alpha$ and $\boldsymbol\beta$, respectively. In principle, one could have concentrated on the angle between $\widehat \boldsymbol\alpha$ and $\boldsymbol\alpha$ (or $\widehat \boldsymbol\beta$ and $\boldsymbol\beta$) instead. However, our choice offers significant advantages that are essential for our developments.
The angle between $\widehat \boldsymbol\alpha$ and $\boldsymbol\alpha$ (or $\widehat \boldsymbol\beta$ and $\boldsymbol\beta$) depends on the choice of units of measurement: the angle changes if we multiply one of the coordinates of $\boldsymbol\alpha$ by a constant (equivalently, divide a component of $\mathbf u$ or a row in $\mathbf U$ by the same constant). Hence, asymptotic theory for these angles would require some normalization conditions on the coordinates of $\mathbf u$ (and, in fact, correlations between different components of $\mathbf u$ would also become important), but such natural normalizations are rare in real data, because different coordinates of $\mathbf u$ might be coming from very different sources or types of observations. By concentrating on $\theta_x$ and $\theta_y$, we avoid this problem entirely, and in particular, the result in Theorem (ref) does not depend on the covariance matrix of the vector $\mathbf u$ (and similarly on the covariance matrix of the vector $\mathbf v$), which can be arbitrary, as long as it satisfies Assumption (ref).
From the technical perspective, computing an angle between $\widehat \boldsymbol\alpha$ and $\boldsymbol\alpha$ is equivalent to decomposing $\widehat \boldsymbol\alpha = p \boldsymbol\alpha + p' \boldsymbol\alpha^\perp$, where $p,p'\in\mathbb R$ and $\boldsymbol\alpha^\perp$ is orthogonal to $\boldsymbol\alpha$ (for simplicity, assume that $\widehat \boldsymbol\alpha$, $\boldsymbol\alpha$, and $\boldsymbol\alpha^\perp$ are unit vectors). To obtain the angle between $\widehat \mathbf x$ and $\mathbf x$ of Definition (ref), one multiplies the decomposition by $\mathbf U^\mathsf T$, leading to $\widehat \mathbf x=p\mathbf x + q \mathbf U^\mathsf T \boldsymbol\alpha^\perp$. The problem is that $\mathbf U^\mathsf T$ is not an orthogonal transformation, and there is no reason for $\mathbf U^\mathsf T \boldsymbol\alpha^\perp$ to be orthogonal to $\mathbf x$. A special case is when the covariance matrix of $\mathbf u$ is proportional to the identity matrix, so that all $K\times S$ matrix elements of $\mathbf U$ are i.i.d. Then by the law of large numbers, for any deterministic unit vector $\mathbf d$, the length of $\mathbf U^\mathsf T \mathbf d$ is $S+o(S)$ as $S\to\infty$. Therefore, by the polarization identity, multiplication by $\mathbf U^\mathsf T$ preserves the angles between the vectors, up to a small error. In contrast, when the covariance matrix of $\mathbf u$ is generic and unknown to us, the knowledge of the angle between $\widehat \mathbf x$ and $\mathbf x$ does not provide information about the angle between $\widehat \boldsymbol\alpha$ and $\boldsymbol\alpha$. Indeed, if we multiply $\mathbf u$ by an arbitrary $K\times K$ matrix $\Upsilon$ (therefore transforming the covariance of $\mathbf u$), Assumption (ref) still holds, hence, the conclusion (ref) continues to hold with no changes. But the angle between $\widehat \boldsymbol\alpha$ and $\boldsymbol\alpha$ becomes the angle between $\Upsilon \widehat \boldsymbol\alpha$ and $\Upsilon \boldsymbol\alpha$, which can be made arbitrary by an appropriate choice of $\Upsilon$, depending only on $\boldsymbol\alpha$. Similarly, since the sample canonical correlations are invariant with respect to multiplication of the data by $\Upsilon$, consistent estimation of the angle between $\widehat \boldsymbol\alpha$ and $\boldsymbol\alpha$ is impossible without additional information beyond these correlations.
We supplement this discussion with two numeric simulations shown in Figure (ref) where $\boldsymbol\alpha=(1,0,0,\dots)^\mathsf T$, $\boldsymbol\beta=(1,0,0,\dots)^\mathsf T$ and vector $\mathbf v$ has i.i.d.\ $\mathcal N(0,1)$ components. In the first one the vector $\mathbf u$ has i.i.d.\ $\mathcal N(0,1)$ components. In the second one the components are independent, the first component is $\mathcal N(0,4)$ and all other components are $\mathcal N(0,1)$. In line with Theorem (ref), the angles between $\widehat \mathbf x$ and $\mathbf x$ are very close to each other in two simulations. However, the angles between $\widehat \boldsymbol\alpha$ and $\boldsymbol\alpha$ are different, and only in the first case they are close to the ones between $\widehat \mathbf x$ and $\mathbf x$.
\@startsection{section}{1} \z@{1.0\linespacing\@plus\linespacing}{0.5\linespacing} {\normalfont}{General framework}
In the previous section we presented our main theorem in the basic setting of i.i.d.\ Gaussian data with a single nonzero canonical correlation in the population. The results are the most transparent in this case, yet many important generalizations can be derived. In this section we state four such generalizations by
Our theorems accommodate combinations of these generalizations. The first, third, and fourth extensions can be applied together. The second extension---by allowing for any distribution of the signal---partially overlaps with the first extension and also works in combination with the fourth. While it is plausible that additional combinations of these extensions might be feasible, we currently do not see a way to allow all four generalizations simultaneously. This limitation is inherent to our approach, as the asymptotic concentration in (ref), (ref), and (ref) relies on a form of the law of large numbers. To achieve this we require at least some degree of independence --- or, at a minimum, a decay of correlations along the $S$ dimension.
Definition (ref) is equivalent to requiring the fourth joint moments of $(X_1,\dots,X_n)$ to satisfy the Wick rule: for any $1\le i,j,k,l\le n$, we should have
For $n=1$, (ref) reduces to a single condition $\mathbb E X_1^4= 3 [\mathbb E X_1^2]^2$. For $n=2$, there are five conditions: $\mathbb E X_1^4=3 [\mathbb E X_1^2]^2$, $\mathbb E X_1^3 X_2=3 \mathbb E X_1^2 \mathbb E X_1 X_2$, $\mathbb E X_1^2 X_2^2 = \mathbb E X_1^2 \mathbb E X_2^2 + 2 [\mathbb E X_1 X_2]^2$, $\mathbb E X_1 X_2^3= 3 \mathbb E X_1 X_2 \mathbb E X_2^2$, and $\mathbb E X_2^4=3 [\mathbb E X_2^2]^2$.
We now present a relaxation of Assumption (ref) on the random vectors $\mathbf u\in\mathbb R^K$ and $\mathbf v\in\mathbb R^M$.
Assumption (ref) splits the vectors $\mathbf u$ and $\mathbf v$ into two components: the correlated signal part $(\mathbf u^\mathsf T\boldsymbol\alpha, \mathbf v^\mathsf T\boldsymbol\beta)$ and the remaining noise part $(A\mathbf u, B\mathbf v)$. The coordinates of the latter are independent but not necessarily identically distributed. Assumptions (ref) and (ref) coincide in the case of Gaussian $\mathbf u$ and $\mathbf v$.
The implications of Theorem (ref) are the same as those discussed in Section (ref). Thus, practical implementations remain the same.
The moment conditions in Assumption (ref) are used in the proof of Lemma (ref) in the appendix: they simplify the derivations by insuring that some terms are uncorrelated and, therefore, satisfy the most basic form of the Law of Large Numbers. One can possibly relax this condition and use LLN for correlated data or come up with alternatives to our moment conditions (see yang2022limiting for one possible approach). As shown in Figure (ref), uniform errors $U[-1,1]$, whose fourth moment does not coincide with the fourth moment of a Gaussian distribution, still lead to the same results as in Theorem (ref). Moreover, Figure (ref) illustrates the same alignment between the theoretical results and a single simulation using $t$ distribution with $3$ degrees of freedom for errors, where the third moment does not exist.
We anticipate that the limits (ref), (ref), (ref) remain the same under conditions weaker than Assumption (ref). However, we expect the size of the fluctuations around the limit to be sensitive to the fourth moments. For related results, see yang2022limiting and muirhead1980asymptotic, with the latter addressing the regime where $S\to\infty$ while $K$ and $M$ remain fixed.
For the next generalization we need to adjust the procedure of Section (ref) and no longer assume that the data are obtained from i.i.d.\ samples of $\mathbf u$ and $\mathbf v$ vectors.
We still use Definition (ref): we are given a $K\times S$ matrix $\mathbf U$ and $M\times S$ matrix $\mathbf V$, and we compute their sample canonical correlations and corresponding variables. What changes is the probabilistic mechanism creating these matrices: while in Section (ref) we started from vectors $\mathbf u$ and $\mathbf v$ and then considered i.i.d.\ samples of them, now we weaken the i.i.d.\ assumption and therefore have to introduce assumptions directly on the distributions of $\mathbf U$ and $\mathbf V$ rather than on $\mathbf u$ and $\mathbf v$. Despite the more complicated probabilistic setting, the final interpretation remains the same: the signal part of the data comes from two deterministic vectors $\boldsymbol\alpha\in\mathbb R^K$ and $\boldsymbol\beta\in\mathbb R^M$, and we are trying to reconstruct them using $\widehat \boldsymbol\alpha$ and $\widehat \boldsymbol\beta$ of Definition (ref).
Hence, our assumptions are now phrased in terms of $K\times S$ and $M\times S$ matrices $\mathbf U$ and $\mathbf V$, respectively.
Note that there are no restrictions on $\mathbf x=\mathbf U^\mathsf T \boldsymbol\alpha$ and $\mathbf y =\mathbf V^\mathsf T \boldsymbol\beta$ in Assumption (ref). In particular, they are allowed to have correlated or even deterministic coordinates. In the setting of Assumption (ref), we replace the squared correlation coefficient $r^2$ of Definition (ref) with its sample version:
As before, we treat vectors $\mathbf x$ and $\mathbf y$ as the signal part of the data and the rest as the noise. Note that if $\mathbf x$ and $\mathbf y$ have i.i.d.\ Gaussian components, then Assumption (ref) coincides with Assumption (ref) (up to a linear transformation of the noise part). In this situation $\hat r^2$ differs from $r^2$ from Definition (ref), but their difference tends to $0$ as $S\to\infty$ by the law of large numbers.
The practical implementation of the procedure discussed at the beginning of Section (ref) continues to hold in the setting of Theorem (ref). Theorem (ref) uses the Gaussianity of the noise part of the data, $A\mathbf U$ and $B\mathbf V$. While it is plausible that this restriction can be relaxed, we do not address such a question in this paper.
In the next extension we relax the assumption that the noise has independent coordinates. This complements the correlated signal of Section (ref). The results for the correlated noise depend on the knowledge of the canonical correlations of the noise itself, and the formulas become much more complicated than in Theorems (ref), (ref), and (ref). Nevertheless, one can still efficiently use them when working with the data. Our assumptions are again phrased in terms of $K\times S$ and $M\times S$ matrices $\mathbf U$ and $\mathbf V$, respectively.
In the setting of Assumption (ref), we set $\displaystyle r^2:=\frac{C_{uv}^2}{C_{uu} C_{vv}}$. The matrices $A \mathbf U$ and $B\mathbf V$ in the assumption might depend on each other and might be deterministic. These are $(K-1)\times S$ and $(M-1)\times S$ matrices, respectively, and we denote through $1\ge c_1^2 \ge c_2^2\ge \dots \ge c_{K-1}^2\ge 0$ their sample squared canonical correlations, which are eigenvalues of the $(K-1)\times(K-1)$ matrix $(A \mathbf U \mathbf U^\mathsf T A^\mathsf T)^{-1} A \mathbf U \mathbf V^\mathsf T B^\mathsf T (B \mathbf V \mathbf V^\mathsf T B^\mathsf T)^{-1} B \mathbf V \mathbf U^\mathsf T A^\mathsf T$. We start by stating two theoretical results and then present a practical statistical implication as a corollary.
The formulas in Theorem (ref) depend on the unknown function $G(z)$. There are two ways to avoid this difficulty. First, by imposing additional assumptions on the noise, one can deduce exact asymptotic formulas for $G(z)$---this is how we deduce Theorems (ref) and (ref) from Theorem (ref). In those theorems $G(z)$ approximates the Stieltjes transform of the Wachter distribution, with the density shown by orange curves in Figures (ref), (ref), and (ref) as well as in Figures (ref) and (ref) appearing later.
Alternatively, one can reuse the observed squared canonical correlations $\lambda_1\ge \lambda_2\ge \dots\ge \lambda_K$ through the following approximation statement, which is a direct corollary of Lemma (ref) from the appendix.
The primary application of Theorem (ref) arises when $i=1$, focusing on the largest canonical correlation $\lambda_1$. In this case Theorem (ref), combined with Lemma (ref) and the interlacement inequalities from Lemma (ref), imply the following statement.
In practice, one only knows the values of $S$, $\lambda_1$ and $\lambda_2$, but not $\varepsilon$. Thus, some guidance is needed to decide if Corollary (ref) is applicable. As a safe choice, we suggest to approximate $r^2$ via (ref) when $\lambda_1-\lambda_2$ is much larger than $S^{-1/2}$ (say, $\lambda_1-\lambda_2\ge 5 S^{-1/2}$). The intuition behind this suggestion is as follows. We expect the $o(1)$ error to be of order $S^{-1/2}$, cf. bao2019canonical, yet, as $\lambda_1-\lambda_2$ decreases it can grow. At the same time the eigenvalue $\lambda_2$ contributes to $\hat G(\lambda_1)$ the term of order $\frac{1}{S(\lambda_1-\lambda_2)}$, thus, having $\lambda_1-\lambda_2$ of order $S^{-1/2}$ is a critical scale which balances these two contributions. A more precise analysis of the dependence of $o(1)$ errors in (ref) on $\lambda_1-\lambda_2$ is left for future research.
The four theorems---(ref), (ref), (ref), and (ref)--- assumed that there is a unique signal in the $\mathbf U$ part of the data and a unique signal in the $\mathbf V$ part of the data. All the theorems have extensions to the situation of several signals. In the extended statements exactly the same procedures are used for each signal.
We start from the basic setup of Section (ref). The population setting of Assumption (ref) says that there is exactly one nonzero canonical correlation\footnote{In the population setting, squared canonical correlations can be computed as eigenvalues of the $K\times K$ matrix $(\mathbb E \mathbf u \mathbf u^\mathsf T)^{-1} (\mathbb E \mathbf u \mathbf v^\mathsf T) (\mathbb E \mathbf v \mathbf v^\mathsf T)^{-1} (\mathbb E \mathbf v \mathbf u^\mathsf T)$.} between $\mathbf u$ and $\mathbf v$, and it corresponds to the canonical variables $\mathbf u^\mathsf T\boldsymbol\alpha$ and $\mathbf v^\mathsf T\boldsymbol\beta$. Instead, assume that there are $\mathbbm q$ nonzero canonical correlations:
As in Section (ref), we let $\mathbf W=
$ be a $(K+M)\times S$ matrix composed of $S$ independent samples of $
$. The squared sample canonical correlations $\lambda_1\ge \lambda_2\ge \dots\ge \lambda_K$ are the eigenvalues of the $K\times K$ matrix $(\mathbf U \mathbf U^\mathsf T)^{-1} \mathbf U \mathbf V^\mathsf T (\mathbf V \mathbf V^\mathsf T)^{-1} \mathbf V \mathbf U^\mathsf T$ or $K$ largest eigenvalues of the $M\times M$ matrix $(\mathbf V \mathbf V^\mathsf T)^{-1} \mathbf V \mathbf U^\mathsf T (\mathbf U \mathbf U^\mathsf T)^{-1} \mathbf U \mathbf V^\mathsf T$. Choosing one eigenvalue $\lambda_i$, we set $\widehat \boldsymbol\alpha^i$ to be the corresponding eigenvector of the former matrix, and we set $\widehat \boldsymbol\beta^i$ to be the eigenvector of the latter matrix, corresponding to the same eigenvalue. The sample canonical variables are defined as $\widehat \mathbf x^i=\mathbf U^\mathsf T \widehat \boldsymbol\alpha^i$ and $\widehat \mathbf y^i =\mathbf V^\mathsf T \widehat \boldsymbol\beta^i$. We also set $\mathbf x^q=\mathbf U^\mathsf T \boldsymbol\alpha^q$ and $\mathbf y^q =\mathbf V^\mathsf T \boldsymbol\beta^q$, where $q$ is chosen so that $\rho^2[q]$ is the $q$-th largest elements of $\rho^2[1],\dots,\rho^2[\mathbbm q]$.
Figure (ref) illustrates Theorem (ref): there are three signals of different strength indicated by the three separate right-most eigenvalues. The corresponding angles increase when the strength of the signal goes down (smaller $\rho^2$), as can be seen from Table (ref). Note also that the theoretical formulas are close to what one obtains in the simulations.
Theorem (ref) helps one find the number of signals when it is unknown. The eigenvalues to the right of $\lambda_+$ represent signals, and thus, one can visually deduce\footnote{One should be able to construct a formal test of the number of signals, e.g., by using CLT-type results of bao2019canonical, yang2022limiting. In the setting of factors a parallel question generated considerable interest, see, e.g., bai2002determining, bai2007determining,hallin2007determining,onatski2009testing,ahn2013eigenvalue. } how many there are by looking at a histogram (e.g, one clearly sees three signals in Figure (ref)). In practice we do not know the values of $\rho^2[q]$ a priori. Instead, we should look at the squared sample canonical correlations $\lambda_1\ge \lambda_2\ge\dots$: if several of the largest ones are well separated from $\lambda_+$ and from each other, then we can use Theorem (ref), reconstruct the corresponding $\rho^2[q]$ and deduce the values for the angles $\theta^q_x$ and $\theta^q_y$.
The requirement that all $\rho^2[q]$ are distinct is not simply a technical artifact. Indeed, if two canonical correlations coincide, then the corresponding canonical variables are no longer well defined because any linear combination of two eigenvectors with the same eigenvalue is again an eigenvector with the same eigenvalue.
Theorems (ref), (ref), and (ref) have very similar extensions to the case of $\mathbbm q$ nontrivial canonical correlations. The extension of Theorem (ref) is exactly the same: the data are allowed to be fourth-moment Gaussian rather than Gaussian. In the extension of Theorem (ref) we have $2\mathbbm q$ vectors $\mathbf x^q$, $\mathbf y^q$, $1\le q\le \mathbbm q$, which represent $\mathbbm q$ “true” canonical variables (and therefore $\mathbf U^\mathsf T \mathbf x^q$, $\mathbf V^\mathsf T \mathbf y^{q'}$ are pairwise orthogonal except for the allowed correlation when $q=q'$, which should result in distinct correlation coefficients as we vary $q$). In the extension of Theorem (ref), we have $2\mathbbm q$ random vectors $\mathbf x^q$, $\mathbf y^q$, $1\le q\le \mathbbm q$, with i.i.d.\ components, which represent the canonical variables in the population, and the corresponding canonical correlations should be all distinct. In each extension, the final statement is as in Theorem (ref): the relations for each $q=1,2,\dots,\mathbbm q$ are exactly the same as for the $\mathbbm q=1$ situation, and we omit further details.
\@startsection{section}{1} \z@{1.0\linespacing\@plus\linespacing}{0.5\linespacing} {\normalfont}{Empirical illustrations}
In this section we provide an empirical implementation of our approach in two data sets. The first example provides insights into the behavior of the stock market and analyzes the relationship between cyclical and non-cyclical (defensive) stocks. The second example complements bao2019canonical in analyzing a limestone grassland data set.
How to model and explain stock returns has always been an important question in finance and financial econometrics. There are many different approaches and techniques, including asset pricing models, volatility models, and dimension-reduction factor models. This section adds CCA to the list and shows how this method can be used to analyze stock returns. Our illustration is in parallel with the literature on maximally predictable portfolio, which originated with lo1997maximizing and aims at predicting some combination of stocks and bonds from a combination of observable economic variables. Canonical variables in CCA can serve as such portfolio. Along these lines, recently firoozye2023canonical applied CCA to current and past returns to construct a predictable portfolio.
Compared to the preceding, in our application we do not distinguish between outcome variables and predictors and treat them symmetrically. We use two corpuses of data and concentrate on consumer cyclical and non-cyclical (consumer defensive) stocks. The former are known to follow the state of the economy, while the latter are not related to business cycles and are useful during economic slowdowns. Knowing their correlations can be helpful for portfolio allocation. Hence, we apply CCA and search for maximally correlated combinations of cyclical and non-cyclical stocks.
We use weekly returns (which are widely believed to be uncorrelated across time) for the $80$ largest cyclical and $80$ largest non-cyclical stocks over ten years ($01.01.2010-01.01.2020$), which gives us $521$ observations across time. The time window is chosen to exclude the $2007-2008$ financial crisis and COVID-19 periods. The exact list of stocks is reported in Appendix (ref). Figure (ref) shows the squared sample canonical correlations between the returns of the two groups of stocks. We can see that the overall shape of the histogram of the correlations matches the Wachter distribution (as discussed in Section (ref), this indicates that the data matches our modelling assumptions and we can use the formulas of Theorem (ref)) and, at the same time, that the three largest correlations are clearly separated from the rest. The largest squared sample canonical correlation, which is close to $0.9$, represents a very strong signal, which can be interpreted as an overall market movement. In favor of this interpretation is also the fact that the majority of the coordinates in the corresponding pair of eigenvectors are positive, though the eigenvectors are not reported here for the sake of space. For the cyclicals, the sum of squares of the positive coordinates is $0.85$ out of $1$, and for the noncyclicals, it is $0.78$ out of $1$. Two other correlations represent medium-strength signals. The presence of exactly three signals is interesting for several reasons. First, it is resonant of the three factors of fama1992cross. Second, it is in noticeable contrast to the results of a principal component analysis (PCA) applied to the same stock returns. PCA often shows few or even only one clearly distinct factor for a large group of stocks; see, e.g., potters2020first. For our two groups of stocks, PCA indicates the presence of at most two factors---fewer than what we detect with CCA. Figure (ref) in Appendix (ref) shows the PCA eigenvalues, i.e., eigenvalues of $(\mathbf V \mathbf V^\mathsf T)/S$ and $(\mathbf U \mathbf U^\mathsf T)/S$, where $\mathbf V$ and $\mathbf U$ are the de-meaned cyclical and non-cyclical returns, respectively. We can see that the cyclicals have only one strong factor while the non-cyclicals have two. As with CCA, the largest eigenvalue in PCA is believed to be market related. In our stock returns data set, the largest factors are highly correlated across the two groups (the correlation is $0.83$) and with their CCA counterparts (the first pair of canonical variables). The two other CCA pairs of canonical variables are capturing additional information (cf. the size and value factors in fama1992cross) not reflected by PCA. Based on the above, we observe that CCA picks up more information than PCA by extracting additional relevant factors.
The exact values of the CCA signals are reported in Table (ref). The correlations and angles in Table (ref) are calculated based on the results of Theorem (ref). Since we have $K=M=80$, the angles are the same for the cyclicals and non-cyclicals, and we report only one angle per signal. In line with the strength of the signals, the angle between the first estimated signal and the unobserved truth is the smallest, showing precise estimation. The two other signals are weaker, leading to larger angles, i.e., large estimation error. This observation is reinforced by Figure (ref), which shows a scatter plot of the cyclical vs. non-cyclical canonical variables. The points in Figure (ref) show the tightest fit to the $45$-degree line, demonstrating a very strong correlation, while those in Figures (ref) and (ref) are more spread out. The tightness in Figures (ref) and (ref) is approximately the same, in line with the second- and third-largest eigenvalues being very close.
One of the classic data sets examined by CCA is the limestone grassland community data from Anglesey, North Wales (gittins1985canonical); for instance, these data were discussed recently in bao2019canonical. The goal of the CCA application in this case is to identify the relationship between several soil properties and the representation of plant species. There are $M=8$ species considered---Helictotrichon pubescens, Trifolium pratense, Poterium sanguisorba, Phleum bertolonii, Rhytidiadelphus squarrosus, Hieracium pilosella, \textit{Briza media}, and \textit{Thymus drucei}---and $K=6$ soil characteristics---depth ($d$), extractable phosphate ($P$), exchangeable potassium ($K$), and cross-product terms between all three soil variables ($d\times P,\, d\times K,\, P\times K$). The number of random samples is $S=45$. Although the dimensions seem small, Monte Carlo simulations (see Appendix (ref)) indicate that Theorem (ref) gives a reasonable approximation for these values: the theoretical and simulated angles are close to each other.
To analyze the data, we first de-mean all the variables (the means are calculated across $S$-space). After that we find that the six nonzero squared canonical correlations are $0.83,0.52,0.36,0.11,0.09,0.04$, and we have $\lambda_{+}=0.53$, which indicates the presence of a rank $1$ signal. This signal has estimated strength $|\rho|=0.86$ or $\rho^2=0.75$. As can be seen from Table (ref), the estimation precision is quite high; i.e., angles $\theta_x$ and $\theta_y$ are small (the angles are obtained via Theorem (ref)). The estimated vectors $\widehat \boldsymbol\alpha$ and $\widehat \boldsymbol\beta$ suggest that most of the correlation comes from the first coordinate of $\widehat \boldsymbol\beta$ (Helictotrichon pubescens) and the third coordinate of $\widehat \boldsymbol\alpha$ (potassium). That is, more potassium in the soil is strongly correlated with the presence of Helictotrichon pubescens. Figure (ref) shows a scatter plot of $(\mathbf U^\mathsf T \widehat \boldsymbol\alpha,\,\mathbf V^\mathsf T \widehat \boldsymbol\beta)$. As we can see, the data points lie close to the $-45$-degree line, which indicates a correlation close to $-1$.
\@startsection{section}{1} \z@{1.0\linespacing\@plus\linespacing}{0.5\linespacing} {\normalfont}{Conclusion}
High-dimensional econometrics and statistics have been playing an increasingly prominent role in the analysis of data in social sciences (see, e.g., fan2011sparse). One popular methods for dealing with two large data sets is canonical correlation analysis (CCA). However, little is known about the performance of CCA in the high-dimensional setting where all dimensions of data are large and comparable. This paper provides one of the first results on the precision of CCA in high dimensions. To be more precise, we show that estimation of canonical vectors is inconsistent and compute the exact degree of inconsistency in terms of angles between the true and estimated canonical variables. Consistency can be restored as one of the dimensions becomes much larger than the others.
There are various directions for future research on high-dimensional CCA. The first important generalization is to allow for full correlation across $S$ in the data, i.e., for a setting where both the signal and the noise are dependent across the $S$ dimension. This extension will make the results useful for a typical time series setting, where the coordinate $S$ represents time. One would hope that for stationary time series results in line with Theorem (ref) continue to hold. An interesting application would be to then use the procedure on the FRED-MD macro time series data set (mccracken2016fred) and look for canonical correlations between real and nominal variables. Generally, the traditional approach to searching for factors in the FRED-MD data is principal component analysis (PCA). However, this approach does not use any additional information on the type of each variable; i.e., real and nominal variables are mixed together. CCA applied to a subset of real variables vs. a subset of nominal variables can potentially provide another “common” factor.
Another noteworthy area of interest concerns deriving the asymptotic distribution of the angles $\theta_x$ and $\theta_y$---this would require developing a version of a central limit theorem. An asymptotic distribution of the angles can be required for various procedures that use angles as an input (such as confidence interval construction, the delta method, and simulation of bootstrap standard errors).
Finally, a more refined analysis of what is happening near the identification boundary $\rho^2> \rho_c^2$ in the spirit of local to unit root limit theory is of theoretical interest. We expect that novel equations involving random functions should appear in the process of answering this question.