Revealing Unobservables by Deep Learning: Generative Element Extraction Networks (GEEN)

Revealing Unobservables by Deep Learning: Generative Element Extraction Networks (GEEN)

Introduction

Unobservables play a crucial role in scientific research because empirical researchers often encounter a discrepancy between what is described in a model and what is observed in the data. A typical example is the so-called hidden Markov models, where a series of latent variables are observed with errors in multiple periods under conditional independence assumptions. While there is a huge literature on the estimation of the model with latent variables (e.g.,aigner1984latent, bishop1998latent), this paper focuses on the estimation of realizations of the latent variable, which are not observed anywhere in the data. Suppose that the ideal data for the estimation of a model is an i.i.d. sample of $(X^1, X^2,...,X^k,X^*)$ \footnote{We use capital letters to stand for a random variable and lower case letters to stand for the realization of a random variable. For example, $f_{V}(v)$ stands for the probability density function of random variable $V$ with realization argument $v$, and $f_{V|U}(v|u)$ denote the conditional density of $V$ on $U$.} and that the researcher only observed $(X^1, X^2,...,X^k)$ in the sample. Generally, we consider $X^j, j=1...k,$ as multiple measurements of $X^*$. Under conditional independence assumptions, this paper provides a deep learning method to extract the common element $X^*$ from multiple observables $(X^1, X^2,...,X^k)$. We build a Generative Element Extraction Networks (GEEN) to reveal realizations or draws of $X^*$ to achieve a complete sample of $(X^1, X^2,...,X^k,X^*)$ in the sense that the generated draws are observationally equivalent to the true values in the sample.

This paper is different from the imputation method because the latent variable is not observed anywhere in the sample and needs to be identified. For example, true earnings of households are not observed anywhere in household survey data, but are of great interest to know. By contrast, imputation requires at least some observations of the underlying variable.

Researchers have already applied deep generative models for data imputation. yoon2018gain creatively use the Generative Adversarial Imputation Nets (GAIN) to provide an imputation method, in which missing values are estimated so that they are observationally equivalent to the observed values from the GAIN's perspective. yoon2018gain creatively use the Generative Adversarial Imputation Nets (GAIN) to provide an imputation method, in which missing values are estimated so that they are observationally equivalent to the observed values from the GAIN's perspective.li2019misgan also propose a GAN-based (goodfellow2014generative) framework for learning from complex, high-dimensional incomplete data to impute missing data. mattei2019miwae introduce the missing data importance-weighted autoencoder for training a deep latent variable model to handle missing-at-random data. nazabal2020handling present a general framework for Variational Autoencoders (VAEs) (kingma2013auto) that effectively incorporates incomplete data and heterogenous observations. muzellec2020missing leverage optimal transport to define a loss function for missing value imputation. yoon2020gamin propose a novel Generative Adversarial Multiple Imputation Network (GAMIN) for highly missing Data. In this literature, latent spaces are used to represent high-dimensional observations, but are not identifiable because their latent spaces may vary with parameter initialization. In addition, all the missing data models require true values to be partially observed.

However, relatively little research has focused on estimating realizations of latent variables, which are unobserved, or completely missing. In the economics literature, Kalman filter and structural vector autoregressions have often been used to estimate the realizations of latent variables, such as potential output kuttner1994estimating, natural rate of interest laubach2003measuring,holston2017measuring, and natural rate of unemployment king2007search, but the literature makes parametric assumptions about the dynamics of latent variables and thus belongs to the estimation of models with latent variables.

In our setting, we argue that the conditional independence restrictions imply the local identification of the true values. That allows us to provide an estimator in the continuous case. Our method is nonparametric in the sense that we do not assume the distribution of the variables belong to a parametric family as in the widely-used VAEs kingma2013auto, which use the so-called Evidence Lower Bound (ELBO) to provide a tractable unbiased Monte Carlo estimator. The VAEs focus on the estimation of a parametric model. In this paper, we focus on the estimation of the true values in each observation in the sample without imposing a parametric structure on the distributions.

Our loss function is a distance between two nonparametric density functions with and without the conditional independence. Such a distance is based on a powerful nonparametric identification result in the measurement error literature hu2008instrumental. (See hu2017econometrics and schennach2020mismeasured for a review.) It shows that the joint distribution of a latent variable and its measurements is uniquely determined by the joint distribution of the observed measurements under a key conditional independence assumption, together with other technical restrictions. To measure the distance between two density functions, we choose the Kullback–Leibler divergence kullback1951information, which plays a leading role in machine learning and neuroscience perez2008kullback. A large literature has studied the estimation of the Kullback–Leibler divergence darbellay1999estimation,moreno2003kullback, wang2005divergence,lee2006estimation, wang2006nearest, nguyen2010estimating,nowozin2016f,belghazi2018mutual. We use a combination of a deep neural network and kernel density estimators to generate density functions with and without the conditional independence and then compute their divergence.

In this paper, we make a further argument that the nonparametric identification of the latent variable distribution implies that the true values in the sample are locally separable in the continuous case. To the best of our knowledge, this paper is the first to provide such identification in observation. We expect such identification will change how researchers deal with latent variables and make our GEEN broadly applicable.

This paper is organized as follows. Section 2 provides the identification arguments. Section 3 describe the neural network and the algorithm. The Monte Carlo simulations are provided in Section 4. Section 5 summarizes the paper. Given the page limit, we put in the Online Appendix a high-level application in the estimation of fixed effects in panel data models.

From identification in distribution to identification in observation

We assume that a researcher observe the distribution of $\{X^1, X^2,...,X^k\}$ from a random sample. Putting the estimation of the population distribution $f_{X^1, X^2,...,X^k}$ from the random sample aside, we face a key identification challenge: How to determine the distribution $f_{X^1, X^2,...,X^k,X^*}$ from the observed distribution $f_{X^1, X^2,...,X^k}$? Here we use a general nonparametric identification result in the measurement error literature. We assume

We may consider the observables $(X^1, X^2,...,X^k)$ as measurements of $X^*$. Here we use hu2008instrumental to show the uniqueness of $f(X^1, X^2,...,X^k,X^*)$. We assume three of the $k$ measurements are informative enough for the results in hu2008instrumental. We assume

Before introducing more assumptions, we define an integral operator corresponding to $f_{X^1\vert X^{ \ast }}$, which maps $f_{X^{ \ast }}$ over support $\mathcal{X}^{ \ast }$ to $f_{X^1}$ over support $\mathcal{X}^{ 1 }$. Suppose that we know both $f_{X^{ \ast }}$ and $f_{X^1}$ are bounded and integrable. We define $\mathcal{L}_{b n d}^{1} \left (\mathcal{X}^{ \ast }\right )$ as the set of bounded and integrable functions defined on $\mathcal{X}^{ \ast }$, i.e.,

The linear operator can be defined as

In order to identify the unknown distributions, we need the observables to be informative so that the following assumptions hold.

The functional $M$ may be the mean, mode, medium, or another quantile of the distribution $f_{X^1\vert X^{ \ast }} \left ( \cdot \vert x^{ \ast }\right ) $. The identification result may be summarized as follows:

In the remaining discussion, we still use the conditional independence in Equation ((ref)) because we are interested in the common element $X^*$ across all the observables. This identification result implies that if we have qualified measurements $X^1$, $X^2$ and $X^3$, we are able to provide a consistent estimator of $f_{X^1, X^2,...,X^k,X^*}$ from a sample of $(X^1, X^2,...,X^k)$.

Identification in observation Next, we argue that draws of $X^*$ are locally identified in the sense that there is no observationally equivalent uncorrelated deviation from these draws.

Let $X^*_i$ be a random draw of $X^*$ in observation $i$ and we define an uncorrelated deviation from that draw as

where $(X^*_i,\delta_i)$ is a i.i.d. random draw from the joint distribution of $(X^*,\delta)$. Notice that if we replace $X^*_i$ with $X^*_i+\delta_i$ as the new common element, the variance of the common element becomes $var(X^*) + var(\delta)$. That means the variance of the uncorrelated deviation must be different from that of the original $X^*$, i.e., $var(X^*)$. The distribution of $X^*+\delta$ must be different from that of $X^*$. These two different distributions can not lead to the same observed distribution, $f_{X^1, X^2,...,X^k}$, because Theorem (ref) implies that $f_{X^1, X^2,...,X^k}$ uniquely determines $f_{X^*}$, including its variance $var(X^*)$. In other words, $X^*+\delta$ and $X^*$ can not be observationally equivalent given the sample of $(X^1, X^2,...,X^k)$. Therefore, the draws of $X^*$ are locally identified in the following sense:

Notice that we only use the identified variance of the latent $X^*$ to make this argument. The results in Theorem (ref) implies that all the moments of the latent $X^*$ are identified. Therefore, such a local identification result as in Theorem (ref) should hold for more general deviations than the uncorrelated deviations defined in Equation ((ref)).

Furthermore, we may look at this problem from a different angle. Suppose we insert generated draws $\hat{X}^*_i$ in the sample $\{X^1_i, X^2_i,...,X^k_i \}$ to obtain $\{X^1_i, X^2_i,...,X^k_i, \hat{X}^*_i \}$. And we also suppose that the conditional independence in Equation ((ref)) holds with the generated draws, i.e.,

In this case, even if $\hat{X}^*_i$ is not equal to the true $X^*_i$ in the infeasible full sample $\{X^1_i, X^2_i,...,X^k_i, X^*_i \}$, our inserted $\hat{X}^*_i$ will be observationally equivalent to the true $X^*_i$ because Theorem (ref) guarantees that the distributions $f_{X^1|X^*}$, $f_{X^2|X^*}$, and $ f_{X^3,...,X^k,X^*}$ are uniquely determined by the observed $f_{X^1, X^2,...,X^k}$. Even if $\hat{X}^*_i \neq X^*_i$, we can still correctly estimate $f_{X^1|X^*}$, $f_{X^2|X^*}$, and $ f_{X^3,...,X^k,X^*}$ using sample $(X_i^1, X_i^2,...,X_i^k,\hat{X}^*_i)_{i=1,2,...,N}$ with inserted $\hat{X}^*_i$, instead of the true values $X^*_i$.

In addition, Theorem (ref) implies that if we add a noise $\delta_i$ to the inserted $\hat{X}^*_i$, where $\delta_i$ is an uncorrelated deviation from $\hat{X}^*_i$, the conditional independence fails when $\hat{X}^*_i$ is replaced with $\hat{X}^*_i+\delta_i$. That means the inserted draws $\hat{X}^*_i$ are locally unique among uncorrelated deviations.

The identification result in Theorem (ref) can be extended to the case where $\delta_i$ is uncorrelated with $X^*_i$ conditional on the observables $(X^1, X^2,...,X^k)$ because the conditional distribution $f_{X^*|X^1, X^2,...,X^k}$ is identified by Theorem (ref). We define a conditionally uncorrelated deviation from $X^*_i$ as $X^*_i+\delta_i$ with

where $(X^*_i ,\delta_i , X_i^1, X_i^2,...,X_i^k)$ is a i.i.d. random draw from their corresponding joint distribution. The variance, and therefore distribution, of $X^*_i+\delta_i $ conditional on $(X^1, X^2,...,X^k)$ is different from those of $f_{X^*|X^1, X^2,...,X^k}$. Theorem (ref) implies that they must correspond to different $f_{X^1, X^2,...,X^k}$. Therefore, there is no observationally equivalent conditionally uncorrelated deviation from latent draws $X^*_i$. We summarize this extension as follows:

The identification results in Theorems (ref) and (ref) are based on the identification of second moments of $f_{X^*}$ and $f_{X^*|X^1, X^2,...,X^k}$. The fact that the two distributions are identified nonparametrically, instead of just for second moments, implies that identification in observation may hold for more general deviations than uncorrelated deviations and conditionally uncorrelated deviations. Furthermore, the continuity of $(X_i^1, X_i^2,...,X_i^k)$ also plays a role in the identification in observation. In the case where $(X_i^1, X_i^2,...,X_i^k)$ is discrete, a realization $(X_i^1, X_i^2,...,X_i^k)=(x^1, x^2,...,x^k)$ may have a positive probability. For all the observations with $(X_i^1, X_i^2,...,X_i^k)=(x^1, x^2,...,x^k)$, one may draw $X^*$ from $f_{X^*|X^1, X^2,...,X^k}$. Such a generated sample will have a sampling distribution converging to $f_{X^*, X^1, X^2,...,X^k}$ as the sample size goes to infinity. Therefore, the identification of $X^*_i$ is up to the permutation of the draws of $X^*$ in each observation with $(X_i^1, X_i^2,...,X_i^k)=(x^1, x^2,...,x^k)$. In the case where $(X_i^1, X_i^2,...,X_i^k)$ is continuous, however, the event $(X_i^1, X_i^2,...,X_i^k)=(x^1, x^2,...,x^k)$ has a zero probability. Such a realization only appears once in the sample even as the sample size goes to infinity and this generated sample may not have a sampling distribution converging to $f_{X^*, X^1, X^2,...,X^k}$ as the sample size goes to infinity.

Convergence argument and loss function For an estimator of a finite-dimensional parameter, its consistency requires that the estimator should converge to its true value as the sample size goes to infinity. In this paper, we have a large number of unknown parameters, i.e., the realized value of $X^*$ in each observation. Here, we present sufficient conditions for the consistency of a large number of $\hat{X}^*_i $, rather than all $\hat{X}^*_i $.

Suppose our identification results suggest that our estimates $\hat{X}^*_i $ should have the same distribution (and variance) as $X^*_i$. Then the sample moments of $\hat{X}^*_i $ should converge to the true moments. In other words, we have

Such a condition implies the consistency of our estimator under following assumptions.

Proof: With $\hat{X}^*_i = X^*_i + \delta_i $, we have

Therefore, $$ \frac{1}{N} \sum_{i=1}^{N} ( \delta_i )^2 = o_p(1) $$ Furthermore, we have

Therefore, for any $\epsilon>0$, we have

$$\frac{1}{N} \sum_{i=1}^{N} I( |\delta_i| > \epsilon)= o_p(1)$$ $\qed$

This result implies that the estimator for each observation is consistent w.r.t the sampling distribution. Theorem (ref) does not guarantee the consistency of $\hat{X}^*_i $ in a given observation, but the probability of a randomly-drew estimator $\hat{X}^*_i$ being consistent should converge to one.

This result can be extended to more general deviations. For example, condition (ref) may be replaced with

where $c$ is a constant satisfying $c \neq -\frac{1}{2}$. That means the identification result remains in some cases where the deviations are correlated with the true values in the limit.

Finally, the discussion above implies that we can use a loss function measuring the distance between a general joint distribution $p = f_{X^1, X^2,...,X^k,X^*}$ and a distribution satisfying conditional independence $p_{ci}= f_{X^1|X^*} f_{X^2|X^*}...f_{X^k|X^*}f_{X^*} $ in order to search for latent draws $X^*_i$. A natural choice is the Kullback–Leibler divergence

$$D_{KL} \left (p(x) || p_{ci}(x) \right ) = \int p(x) \ln \left ( \frac{p(x)}{p_{ci}(x)} \right) dx.$$

Generative Element Extraction Networks (GEEN)

We build a Generative Element Extraction Network (GEEN), $G$, to generate the latent realizations of $X^*_i$ satisfying the conditional independence. Let $\vec{V}$ stand for the vector of draws of variable $V$ in the sample, i.e., $\vec{X}^*=(X^*_1,X^*_2,...,X^*_N)^T$ and $\vec{X}^j=(X^j_{1},X^j_{2},...,X^j_{N})^T$. We generate $\vec{\hat{X^*}}$ as follows:

with $\vec{\hat{X^*}}=(\hat{X^*_{1}},\hat{X^*_{2}},...,\hat{X^*_{N}})^T$. The deep neural network $G$ is trained to minimize the Kullback–Leibler divergence

with $ \hat{p} = \hat{f}_{X^1, X^2,...,X^k,\hat{X}^*} $ and $ \hat{p}_{ci}= \hat{f}_{X^1|\hat{X}^*} \hat{f}_{X^2|\hat{X}^*}...\hat{f}_{X^k|\hat{X}^*}\hat{f}_{\hat{X}^*} $ where $\hat{f}$ are empirical distribution functions based on sample $(\vec{X}^1,\vec{X}^2,...,\vec{X}^k, \vec{\hat{X^*}})$. Furthermore, we have

Notice that $G$ enters the loss function through $\vec{\hat{X^*}}=(\hat{X^*_{1}},\hat{X^*_{2}},...,\hat{X^*_{N}})^T$ in density estimators. To be specific, we can have a kernel density estimator \footnote{One may choose other types of density estimators.}

where $h$ stands for bandwidths, $N$ the total sampled observations, $m$ the number of points in each observation and $k$ is the number of features. In the loss function defined in equation ((ref)), it requires more than one data point to estimate the kernel density function. As a result, unlike other use cases that one training point is enough to calculate its corresponding loss, we need to sample $m$ ($>1$) points as one observation to calculate its loss. For example, to build the training sample we sample with replacement $m$ points from the entire training data points and repeat $N$ times, and we end up with $N$ observations in our training sample. The same practice is followed to construct our validation and test samples. The kernel function $K(\cdot)$ can simply be the standard normal density function. For the bandwidth, we adopt the so-called Silverman's rule, i.e., $h^j = w \sigma^j N^{-1/5}$ where $ \sigma^j$ is the standard error of $X^j$, and $w$ is the window size that is determined by hyper parameters tuning. Similarly, we may take $h^* = w \sigma^* N^{-1/5}$, where $ \sigma^*$ is the standard error of $X^*$.

In this paper, we experiment GEEN with multilayer perceptrons (MLPs), but this framework can be readily applied to other deep neural network architectures. In our simulations, we impose a convolution structure on $X^1$ so that the normalization condition can be simplified. The parameters of our deep neural network are estimated by minimizing the loss function:

Early stopping is applied when the loss does not improve for certain epochs in the validation sample. We do not use any information from true $X_i^*$ during training, validation or hyper-parameter tuning. Instead we use the loss defined in the above equation for validation and true $X_i^*$ are only used for final testing.

Simulations

This section presents the performance of our neural network through simulations. We generate the sample as follows:

for $j=1,2,...,k$ and $i=1,2,...,N$. Without loss of generality, we normalize $ m^1(x)= x $ and $ E[\epsilon^1 | X^*]=0. $ We pick distributions for $(\epsilon^1,...,\epsilon^k, X^*)$ and functions $(m^2,...,m^k)$ to generate a sample $(X^1,...,X^k, X^* )$. We then train $G$ using the observed sample $(\vec{X}^1,\vec{X}^2,...,\vec{X}^k)$ to generate $(\vec{X}^1,\vec{X}^2,...,\vec{X}^k, \hat{X^*})$. That is $\vec{\hat{X^*}} = G(\vec{X}^1,\vec{X}^2,...,\vec{X}^k).$ We check the performance of $G$ by calculating the correlation coefficient between $\vec{X^*}$ and $\vec{\hat{X^*}}$.

We use a 6-layer with 10 hidden nodes fully connected neural network. The window size $w$ and normalization term $\lambda$ are tuned as hyper-parameters. We use kernel functions to approximate their density functions. Theoretically if a distribution is normal, the best choice for $w$ used in the kernel function is 1, so to tune $w$ we choose the range from 0.5 to 2. To tune $\lambda$, we arbitrarily choose the range from 0.1 to 0.5. For every experiment, we run 25 times to evaluate the robustness of model performance on its initialization. For the baseline case, we use

We sample 8000 points as training points from the above distributions for $X^*$, $\epsilon^1$, $\epsilon^2$, $\epsilon^3$ and $\epsilon^4$. Then we sample another 1000 points for validation points and 1000 points for test points. We draw 500 points from the training points with replacement 8000 times to build our training set and 1000 times from the validation/test points to build our validation/test set. Figure (ref) shows the relationship between $X^1$, $X^2$, $X^3$, $X^4$ and $X^*$.

In the second experiment, we let the error terms correlate with $X^*$ while keeping the rest setup the same as the baseline. Figure (ref) shows the relationship between $X^1$, $X^2$, $X^3$, $X^4$ and $X^*$ in this one. Specifically, we use:

In the third experiment, we double the variance of the error terms while keeping the rest setup the same as the baseline. Figure (ref) shows the relationship between $X^1$, $X^2$, $X^3$, $X^4$ and $X^*$ in the third experiment.

Table (ref) demonstrates the min, median and max correlations of $\vec{X^*}$ and $\vec{\hat{X^*}}$ in the test sample for the three experiments after running each one 25 times. Figure (ref) shows their best runs respectively. It is clear that GEEN is robust with initialization with very tight distributions of the correlations of $\vec{X^*}$ and $\vec{\hat{X^*}}$ and improves significantly if we simply use $X^1$ to directly measure $X^*$.

In the forth experiment, we loose the normalization condition while keeping the rest setup the same as the baseline. Figure (ref) shows the relationship between $X^1$, $X^2$, $X^3$, $X^4$ and $X^*$ in this experiment.

With this setup, generated $\hat{X^*}$ is not anchored, and as shown in the left hand side of Figure (ref) its values deviate significantly from $X^*$. However, the KL loss helps keep the similarity of the two distributions of generated $\hat{X^*}$ and $X^*$. As shown in the right hand side of Figure (ref), with 25 runs of this experiment most of the absolute values of the correlation between $\vec{X^*}$ and $\vec{\hat{X^*}}$ are around 0.9. This suggests that even without normalization our framework can still help provide an estimation of the direction.

Summary

This paper proposes a Generative Element Extraction Networks (GEEN) to reveal unobserved variables in a random sample, which contains multiple measurements of a latent variable of interest. We present the sufficient conditions, under which the joint distribution of a latent variable and its measurements can be uniquely determined. We then argue that the true values of the latent variable in the sample are locally unique in a class of deviations, which allows us to estimate the true values. To the best of our knowledge, this paper is the first to provide such identification in observation. This approach is based on the key assumption that the measurements are independent conditional on the latent variable. We then propose an algorithm to minimize the probability densities with and without the conditional independence. We use the Kullback–Leibler distance between the two probability densities as the loss function to train the deep neutral network, which maps from the observed measurements to the true values of the latent variable in the sample. The simulation results imply that this proposed estimator works quite well and the estimated values are highly correlated with the true values with a correlation coefficient usually higher than 90%. We then apply this estimator to a panel data model to reveal the fixed effects (see our online appendix), which we believe is highly applicable to studies using longitudinal data. We expect this algorithm will change how researchers deal with latent variables in empirical research.