Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
54,704 characters · 11 sections · 56 citation commands
Adaptive Bayesian Estimation of Mixed Discrete-Continuous Distributions under Smoothness and Sparsity
Mixture models have proven to be very useful for Bayesian nonparametric modeling of univariate and multivariate distributions of continuous variables. These models possess outstanding asymptotic frequentist properties: in Bayesian nonparametric estimation of smooth densities the posterior in these models contracts at optimal adaptive rates up to a log factor (Rousseau:10, KruijerRousseauVaart:09, ShenTokdarGhosal2013). Tractable Markov chain Monte Carlo (MCMC) algorithms for exploring posterior distributions of these models are available (EscobarWest:95, MacEachernMuller98, Neal:00, MillerHarrison17, Norets2017mcmc) and they are widely used in empirical work (see PractNonSemiparamBayes:98, ChamberlainHirano99, BurdaHardingHausman:08, ChibGreenberg10, and JensenMaheu14 among many others).
In most applications, data contain both continuous and discrete variables. From the computational perspective, discrete variables can be easily accommodated through the use of continuous latent variables in Bayesian MCMC estimation (AlbertChib_binpoly:93, McCullochRossi:94). In nonparametric modelling of discrete-continuous data by mixtures, latent variables were used by CanaleDunson:11 and NoretsPelenis2012 among others. Some results on frequentist asymptotic properties of the posterior distribution in such models have also been established. NoretsPelenis2012 obtained approximation results in Kullback-Leibler distance and weak posterior consistency for mixture models with a prior on the number of mixture components. DeYoreoKottas17 establish weak posterior consistency for Dirichlet process mixtures. In similar settings, canale2015bayesian derived posterior contraction rates that are not optimal. The question we address in the present paper is whether a mixture of normal model that uses latent variables for modeling the discrete part of the distribution can deliver (near) optimal and adaptive posterior contraction rates for nonparametric estimation of discrete-continuous distributions.
Our contribution has two main parts. First, we derive lower bounds on the estimation rate for mixed multivariate discrete-continuous distributions under anisotropic smoothness conditions and potentially growing support of the discrete part of the distribution. Second, we study the posterior contraction rate for a mixture of normals model with a variable number of components that uses continuous latent variables for the discrete part of the observations. We show that the posterior in this model contracts at rates that are equal to the derived lower bounds up to a log factor. Thus, Bayesian mixture models can be used for (up to a log factor) optimal adaptive estimation of mixed discrete-continuous distributions. These results are obtained in a rich asymptotic framework where the multivariate discrete part of the data generating distribution can have either a large or a small number of support points and it can be either very smooth or not, and these characteristics can differ from one discrete coordinate to another. In these settings, smoothing is beneficial only for a subset of discrete variables with a quickly growing number of support points and/or high level of smoothness. In a sense, this subset is automatically and correctly selected by the mixture model. The obtained optimal posterior contraction rates are adaptive since the priors we consider do not depend on the number of support points and the smoothness of the data generating process.
Our results on lower bounds have independent value outside of the literature on Bayesian mixture models and their frequentist properties. Let us briefly review most relevant results on lower bounds and place our results in that context. The minimax estimation rates for mixed discrete continuous distributions appear to be studied first by Efromovich2011. He considers discrete variables with a fixed support and shows that the optimal rates for discrete continuous distributions are equal to the optimal nonparametric rates for the continuous part of the distribution. Relaxing the assumption of the fixed support for the discrete part of the distribution is very desirable in nonparametric settings. It has been commonly observed at least since AitchisonAitken76 that smoothing discrete data in nonparametric estimation improves results in practice. HallTitterington87 introduced an asymptotic framework that provided a precise theoretical justification for improvements resulting from smoothing in the context of estimating a univariate discrete distribution with a support that can grow with the sample size. In their setup, the support is an ordered set and the probability mass function is $\beta$-smooth (in a sense that analogs of $\beta$-order Taylor expansions hold). They show that in their setup the minimax rate is the smaller one of the following two: (i) the optimal estimation rate for a continuous density with the smoothness level $\beta$, $n^{-\beta/(2\beta+1)}$, and (ii) the rate of convergence of the standard frequency estimator, $(N/n)^{1/2}$, where $N$ is the cardinality of the support and $n$ is the sample size. HallTitterington87 refer to their setup as “Sparse Multinomial Data” since $N$ can be larger than $n$ and this is the reason we refer to sparsity in the tile of the present paper. Burman87 established similar results for $\beta=2$. Subsequent literature in multivariate settings (e.g., DongSimonoff1995, AertsAugustynsJanssen97statistics) did not consider lower bounds but demonstrated that when the support of the discrete distribution grows sufficiently fast then estimators that employ smoothing can achieve the standard nonparametric rates for $\beta$-smooth densities on $\mathbb{R}^d$, $n^{-\beta/(2\beta+d)}$.
We generalize the results of HallTitterington87 on lower bounds for univariate discrete distributions to multivariate mixed discrete-continuous case and anisotropic smoothness. Alternatively, our results can be viewed as a generalization of results in Efromovich2011 to settings with potentially growing supports for discrete variables.
Some details of our settings and assumptions differ from those in HallTitterington87 and Efromovich2011 because our original motivation was in understanding the behavior of the posterior in mixture models with latent variables. Specifically, we consider lower and upper estimation bounds in the total variation distance since posterior concentration in nonparametric settings is much better understood when the total variation distance is considered (GhosalGhoshVaart:2000). We also introduce a new definition of anisotropic smoothness that, on the one hand, accommodates an extension of techniques for deriving lower bounds from IbragimovHasminskii1984 and, on the other hand, lets us exploit approximation results for mixtures of multivariate normal distributions developed by ShenTokdarGhosal2013.
The rest of the paper is organized as follows. In Section (ref), we describe our framework and define notation. Section (ref) presents our results on lower bounds for estimation rates. The results on the posterior contraction rates are given in Section (ref). Appendix contains auxiliary results and some proofs.
Let us denote the continuous part of observations by $x \in \mathcal{X} \subset \mathbb{R}^{d_x}$ and the discrete part by $y=(y_1,\ldots,y_{d_y}) \in \mathcal{Y}$, where
is a grid on $[0,1]^{d_y}$ (a product symbol $\Pi$ applied to sets hereafter denotes a Cartesian product). The number of values that the discrete coordinates $y_j$ can take, $N_j$, can potentially grow with the sample size or stay constant.
For $y=(y_1,\ldots,y_{d_y}) \in \mathcal{Y}$, let $A_{y} = \prod_{j=1}^{d_y} A_{y_j}$, where
and let us represent the data generating density-probability mass function as
where $f_0$ belongs to $\mathcal{D}$, the set of probability density functions on $\mathbb{R}^{d_x+d_y}$ with respect to the Lebesgue measure. The representation of a mixed discrete-continuous distribution in (ref) is so far without a loss of generality since for any given $p_0$ one could always define $f_0$ using a mixture of densities with non-overlapping supports included in $A_y$, $y \in \mathcal{Y}$.
In this paper, we consider independently identically distributed observations from $p_0$: $(Y^n,X^n)=(Y_1,X_1,\ldots,Y_n,X_n)$. Let $P_0$, $E_0$, $P_0^n$, and $E_0^n$ denote the probability measures and expectations corresponding to $p_0$ and its product $p_0^n$.
When $N_j$'s grow with the sample size the generality of the representation in (ref) can be lost when assumptions such as smoothness are imposed on $f_0$. Nevertheless, in what follows we do impose a smoothness assumption on $f_0$. The interpretation of this assumption is that the values of discrete variables can be ordered and that borrowing of information from nearby discrete points can be useful in estimation.
To get more refined results, we allow $N_j$'s to grow at different rates for different $j$'s. For the same reason, we work with anisotropic smoothness. Let $\mathbb{Z}_+$ denote the set of non-negative integers. For smoothness coefficients $\beta_i>0$, $i=1,\ldots,d$, $d=d_x+d_y$, and an envelope function $L:\mathbb{R}^{2d}\rightarrow \mathbb{R}$, an anisotropic $(\beta_1,\ldots,\beta_d)$-Holder class, $\mathcal{C}^{\beta_1,\ldots,\beta_d,L}$, is defined as follows.
In this definition, a Holder condition is imposed on $D^k f$ for a coordinate $j$ when $D^k f$ cannot be differentiated with respect to $z_j$ anymore ($\sum_{i=1}^d k_i/\beta_i<1$ but $\sum_{i=1}^d k_i/\beta_i + 1/\beta_j \geq 1 $). This definition slightly differs from definitions available in the literature on anisotropic smoothness that we found. Section 13.2 in Schumaker2007splines presents some general anisotropic smoothness definitions but restricts attention to integer smoothness coefficients. IbragimovHasminskii1984, and most of the literature on minimax rates under anisotropic smoothness that followed including BarronBirgeMassart99 and BhattacharyaPatiDunson2014, do not restrict mixed derivatives. ShenTokdarGhosal2013 use $|\Delta z_j|^{\min(\beta_j - k_j,1)}$ instead of $|\Delta z_j|^{\beta_j(1-\sum k_i/\beta_i)}$ in (ref). Their requirement is stronger than ours for functions with bounded support, and it appears too strong for our derivation of lower bounds on the estimation rate. However, our definition is sufficiently strong to obtain a Taylor expansion with remainder terms that have the same order as those in ShenTokdarGhosal2013 (while the definitions that do not restrict mixed derivatives do not deliver such an expansion).
When $\beta_j=\beta$, $\forall j$ and $\sum_{i=1}^d k_i/\beta + 1/\beta \geq 1 $, $\beta_j(1-\sum k_i/\beta_i) = \beta - \lfloor \beta \rfloor$, where $\lfloor \beta \rfloor$ is the largest integer that is strictly smaller than $\beta$, and we get the standard definition of $\beta$-Holder smoothness for the isotropic case.
The envelope $L$ can be assumed to be a function of $(z,\Delta z)$ to accommodate densities with unbounded support. We derive lower bounds on estimation rates for a constant envelope function. Upper bounds on posterior contraction rates are derived under more general assumptions on $L$ as in ShenTokdarGhosal2013.
Some extra notation: for a multi-index $k=(k_1,\ldots,k_d) \in \mathbb{Z}_+^d$, $k!=\prod_{i=1}^d k_i!$, and for $z \in \mathbb{R}^d$, $z^k=\prod_{i=1}^d z_i^{k_i}$. The $m$-dimensional simplex is denoted by $\Delta^{m-1}$. $I_d$ stands for the $d \times d$ identity matrix. Let $\phi_{\mu,\sigma}(\cdot)$ and $\phi(\cdot; \mu,\sigma)$ denote a multivariate normal density with mean $\mu \in \mathbb{R}^d$ and covariance matrix $\sigma^2 I_d$ (or a diagonal matrix with squared elements of $\sigma$ on the diagonal when $\sigma$ is a $d$-vector). For $z \in \mathbb{R}^d$ and $J \subset \{1,2,\ldots,d\}$, $z_J$ denotes sub-vector $\{z_i,\, i \in J\}$. Operator “$\lesssim$” denotes less or equal up to a multiplicative positive constant relation.
Let $\mathcal{A}$ denote a collection of all subsets of indices for discrete coordinates $\{1, \ldots,d_y\}$. For $J \in \mathcal{A}$, define $J^c=\{1,\ldots,d\}\setminus J$, \[ N_{J}=\prod_{i\in J} N_i, \;\;\;\;\;\;\; \beta_{J^c} = \left [ \sum_{i \in J^c} \beta_i^{-1}\right ]^{-1}, \] $N_{\emptyset}=1$, $\beta_{\emptyset} = \infty$, and $\beta_{\emptyset}/(2\beta_{\emptyset} + 1 )=1/2$.
For a class of probability distributions $\mathcal{P}$, $\zeta$ is said to be a lower bound on the estimation error in metric $\rho$ if \[ \inf_{\hat{p}} \sup_{p \in \mathcal{P}} P\left(\rho(\hat{p},p)\geq \zeta \right) \geq const > 0. \]
We consider the following class of probability distributions: for a positive constant $L$, let
One could recognize expression $\left[ N_J/n \right ]^{\frac{\beta_{J^c}}{2\beta_{J^c}+1}}$ in (ref) as the standard estimation rate for a $card(J^c)$-dimensional density with anisotropic smoothness coefficients $\{\beta_j, \, j \in J^c\}$ and the sample size $n/N_J$ (IbragimovHasminskii1984). One way to interpret this is that the density of $\{x,\tilde{y}_j,\, j \in J^c\}$ conditional on $y_J$ is $\{\beta_j, \, j \in J^c\}$-smooth and the number of observations available for its estimation (observations with the same value of $y_J$) should be of the order $n/N_J$; also, the estimation rate for the marginal probability mass function for $y_J$ is $[N_J/n]^{1/2}$, which is at least as fast as $\left[ N_J/n \right ]^{\frac{\beta_{J^c}}{2\beta_{J^c}+1}}$. In this interpretation, smoothing is not performed over the discrete coordinates with indices in set $J$, and the lower bound is obtained when $J$ minimizes $\left[ N_J/n \right ]^{\frac{\beta_{J^c}}{2\beta_{J^c}+1}}$. Thus, an estimator that delivers the rate in (ref) should, in a sense, optimally choose the subset of discrete variables over which to perform smoothing.
We set up the notation and an outline of the proof of Theorem (ref) below and delegate detailed calculations to lemmas in Appendix (ref). The proof of the theorem is based on a general theorem from the literature on lower bounds, which we present next in a slightly simplified form.
The following standard result on bounding the number of unequal elements in binary sequences is used in our construction of $q_j$, $j=1,\ldots,M$.
To define $q_j$'s for our problem, we need some additional notation. Let \[ K_0(u) = \exp\{-1/(1-u^2)\} \cdot 1\{|u|\leq 1\}. \] This function has bounded derivatives of all orders and it smoothly decreases to zero at the boundary of its support. This type of kernel functions is usually used for constructing hypotheses for lower bounds, see Section 2.5 in Tsybakov:08. Since we need to construct a smooth density that integrates to 1, we define (as illustrated in Figure (ref)) \[ g(u)=c_0 [K_0(4(u+1/4))-K_0(4(u-1/4))], \] where $c_0>0$ is a sufficiently small constant that will be specified below. \\[\intextsep]
\\[\intextsep] Function $g$ will be used as a kernel in construction of $q_k$'s. Let us define the bandwidth for these kernels first.
For the continuous coordinates, we define the bandwidth as in IbragimovHasminskii1984, \[ h_i=\Gamma_n^{1/\beta_i},\, i \in \{d_y+1,\ldots,d\}. \] For the discrete ones, over which smoothing is beneficial, we define the bandwidth as \[ h_i=\varrho_i\cdot\Gamma_n^{1/\beta_i}=\frac{2}{N_i}\cdot R_i,\, i \in J_\ast^c \cap \{1,\ldots,d_y\}, \] where $R_i=\lfloor \Gamma_n^{1/\beta_i} N_i / 2\rfloor +1$ is a positive integer and $\varrho_i\in(1,2]$ as shown in Lemma (ref).
For the rest of the discrete coordinates, our innovation is to first define artificial anisotropic smoothness coefficients $\beta_i^\ast=-\log(\Gamma_n)/\log{N_i}$, $i \in J_\ast$, at which the rate in (ref) would have the same value whether we smooth over $y_i$ ($i \in J_\ast^c$) or not ($i \in J_\ast$). Then, we define the bandwidth as \[h_i=2\cdot \Gamma_n^{1/\beta_i^\ast}=2/N_i, \ i \in J_\ast.\] To streamline the notation, we also define $\beta_i^\ast=\beta_i$ for $i \in J_\ast^c$.
Let $m_i$ be the integer part of $h_i^{-1}$, $i=1,\ldots,d$. Let us consider $\bar{m}=\prod_{i=1}^d m_i$ adjacent rectangles in $[0,1]^d$, $B_r$, $r=1,\ldots,\bar{m}$, with the side lengths $(h_1,\ldots,h_d)$ and centers $c^r=(c^r_1,\ldots,c^r_d)$, $c_i^r=h_i(k_{ir} - 1/2)$, $k_{ir} \in\{1,\ldots,m_i\}$. For $z \in \mathbb{R}^d$ and $r=1,\ldots,\bar{m}$, define \[ g_r (z)= \Gamma_n \prod_{i=1}^d g((z_i-c^r_i)/h_i), \] which can be non-zero only on $B_r$. A set of hypotheses is defined by sequences of binary weights on $g_r$'s as follows
where $w^j_r \in \{0,1\}$, $j=0,\ldots,M$, and $M$ are defined in Lemma (ref).
The rest of the proof is delegated to lemmas in Appendix (ref), which show that $q_k$ in (ref) satisfy the sufficient conditions from Lemma (ref). Specifically, Lemma (ref) derives the lower bound on the total variation distance. Lemma (ref) verifies condition (ref) when $\bar{m}\geq 8$. Lemma (ref), part (i) of Lemma (ref), and the fact that $q_k$'s are defined on $[0,1]^d$ imply $q_j \in \mathcal{C}^{\beta_1,\ldots,\beta_d,L}$, $j=0,\ldots,M$.
This argument (Lemma (ref) specifically) requires $\bar{m}\geq 8$ as it relies on Lemma (ref). Observe that as $n\rightarrow \infty$, $\bar{m}\geq 8$ if there are continuous variables or there are discrete variables over which smoothing is beneficial ($J_\ast^c \neq \emptyset$). Thus, $\bar{m}<8$ can happen only if there are no continuous variables and $N_{J_\ast}=N_1 \cdots N_d$ is bounded. This is just a problem of estimating a multinomial distribution with finite support and the standard results for parametric problems deliver the usual $n^{-1/2}$ rate.
Finally, note that we prove the lower bound results for a class of densities that includes densities that are in $\mathcal{C}^{\beta_1,\ldots,\beta_d,L}$ on $[0,1]^d$. It is straightforward to modify the proof so that it works for a class of smooth densities on $\mathbb{R}^d$. To accomplish this we can replace $ 1_{[0,1]^d}(\cdot) $ in (ref) with a smooth function on $\mathbb{R}^d$ that has a bounded support and is bounded away from zero on $[0,1]^d$, for example, \[ \prod_{i=1}^d \left[ 1_{[0,1]}(z_i) + IK_0(z_i+1)*1(z_i<0) + IK_0(2-z_i)*1(z_i>1) \right], \] multiplied by a normalization constant, where $IK_0(z_i)=\int_{-1}^{z_i} K_0(u)du / \int_{-1}^1 K_0(u)du$. Then, proofs of Lemmas (ref)-(ref) go through with minor modifications.
In this section, we consider a Bayesian model for the data generating process in (ref). We use a mixture of normal distributions with a variable number of components for modelling the joint distribution of $(\tilde{y},x)$,
where $\theta=(\mu_j^y, \mu_j^x, \alpha_j, j=1,2,\ldots,m; \sigma)$.
We assume the following conditions on the prior $\Pi$ for $(\theta,m)$. For positive constants $a_1, a_2, \ldots, a_{9}$, for each $i\in \{1,\ldots,d\}$ the prior for $\sigma_i$ satisfies
An example of a prior that satisfies (ref)-(ref) is the inverse Gamma prior for $\sigma_i$.
Prior for $(\alpha_1,\ldots,\alpha_m)$ conditional on $m$ is Dirichlet$(a/m,\ldots,a/m)$, $a > 0$. Prior for the number of mixture components $m$ is
More generally, a prior that can be bounded above and below by functions in the form of the right hand side of (ref), possibly with different constants, would also work.
A priori, the components of $\mu_{j}$, $\mu_{j,i}$, $i=1,\dots,d$ are independent from each other, other parameters, and across $j$. Prior density for $\mu_{j,i}$ is bounded below for some $a_{12}, \tau_2 > 0$ by
and for some $a_{13}, \tau_3>0$ and all sufficiently large $\mu > 0$,
In what follows, we consider a fixed subset of discrete indices $J \in \mathcal{A}$ and show that under regularity conditions, the posterior contraction rate is bounded above by $\left[ \frac{N_J}{n} \right ]^{\frac{\beta_{J^c}}{2\beta_{J^c}+1}}$ times a log factor. If the regularity conditions we describe below for a fixed $J$ hold for every subset of $\mathcal{A}$, then the posterior contraction rate matches the lower bound in (ref) up to a log factor.
Without a loss of generality, let $J=\{1,\ldots,d_J\}$, $I=\{d_{J}+1,\ldots,d_y\}$, $J^c=\{1,\ldots,d\}\setminus J$, and $d_{J^c}=card(J^c)$. Similarly to $\mathcal{Y}$ and $A_{y}$ defined in Section (ref), we define $\mathcal{Y}_J=\prod_{j\in J} \mathcal{Y}_j$ and $A_{y_J}= \prod_{i \in J} A_{y_i}$. Also, let $y_{J}=\{y_i\}_{i \in J}$, $\tilde{y}_{I}=\{\tilde{y}_i\}_{i \in I}$, $\tilde{x}=(\tilde{y}_I,x) \in \tilde{\mathcal{X}}=\mathbb{R}^{d_{J^c}}$.
To formulate the assumptions on the data generating process, we need additional notation,
Also, let $F_{0|J}$ and $E_{0|J}$ denote the conditional probability and expectation corresponding to $f_{0|J}$. If $\pi_{0J}(y_J)=0$ for a particular $y_J$, then we can define the conditional density $f_{0|J}(\tilde{x}|y_J)$ arbitrarily. We make the following assumptions on the data generating process.
It appears that all the papers on (near) optimal posterior contraction rates for mixtures of normal densities impose similar tail conditions on data generating densities.
This assumption always holds for $A_{y_I} \subset [0,1]^{d_{J^c}-d_x}$. When $A_{y_I}$ is a rectangle with at least one infinite side, an interpretation of this assumption is that the tail probabilities for $\tilde{y}_I$ conditional on $(x,y_J)$ decline uniformly in $(x,y_J)$. Bounded support for $\tilde{y}_I$ is a sufficient condition for this assumption.
The smoothness assumption (ref) on the conditional density $f_{0|J}$ is implied by the smoothness of the joint density $f_0$ at least under boundedness away from zero assumption, see Lemma (ref).
The envelope function and restrictions on its behaviour are mostly relevant for the case of unbounded support. Condition (ref) suggests that the envelope function $\tilde{L}$ should be comparable to $f_{0|J}$.
We impose this assumption to exclude from consideration the cases with very slow (non-polynomial) rates as some parts of the proof require $\log (1/\epsilon_n)$ to be of order $\log n$.
Let
where $(\tau,\tau_1, \tau_2)$ are defined in Sections (ref)-(ref).
As in Section (ref), when $J^c = \emptyset$, $\beta_{J^c}$ can be defined to be infinity and $\beta_{J^c}/(2\beta_{J^c} + 1 )=1/2$ in (ref).
Under the assumptions of the corollary, Theorem (ref) delivers a valid upper bound on the posterior contraction rate for every $J \in \mathcal{A}$ including the one for which the minimum in (ref) is attained. Hence, the corollary is an immediate implication of Theorem (ref) whose proof is presented in the following section.
The results on lower bounds in Section (ref) hold for any class of data generating densities that includes $f_0$ satisfying the following conditions: $f_0 \in \mathcal{C}^{\beta_1,\ldots,\beta_d,L}$, $f_0=0$ outside $[0,1]^d$, and $\overline{f} \geq f_0\geq \underline{f}>0$, where $L$, $\overline{f}$, and $\underline{f}$ are finite positive constants. It is worth pointing out that these conditions imply Assumptions (ref)-(ref), which combined with Assumption (ref) for $N_{J_\ast}$ and a prior specified in Section (ref) would deliver the sufficient conditions of Corollary (ref).
To prove Theorem (ref), we use the following sufficient conditions for posterior contraction from Theorem 2.1 in GhosalVaart:01. Let $\epsilon_n$ and $\tilde{\epsilon}_n$ be positive sequences with $\tilde{\epsilon}_n \leq \epsilon_n$, $\epsilon_n \to 0$, and $n \tilde{\epsilon}_n^2 \to \infty$, and $c_1$, $c_2$, $c_3$, and $c_4$ be some positive constants. Let $\rho$ be Hellinger or total variation distance. Suppose $\mathcal{F}_n \subset \mathcal{F}$ is a sieve with the following bound on the metric entropy $M_e( \epsilon_n, \mathcal{F}_n, \rho)$
Suppose also that the prior thickness condition holds
where the generalized Kullback-Leibler neighborhood $\mathcal{K}(p_0,\tilde{\epsilon}_n)$ is defined by
Then, there exists $\bar{M}>0$ such that \[ \Pi\left( p: \rho(p,p_0) > \bar{M} \epsilon_n | Y^n,X^n\right) \stackrel{P_0^n}{\rightarrow} 0. \]
The definition of the sieve and verification of conditions (ref) and (ref) closely follow analogous results in the literature on contraction rates for mixture models in the context of density estimation. The details are given in Lemma (ref) in Appendix (ref). Verification of the prior thickness condition is more involved and we formulate it as a separate result in the following theorem.
Approximation results are key for showing the prior thickness condition (ref). Appropriate approximation results for $f_{0J}(y_J,\tilde{x})= f_{0|J}(\tilde{x}|y_J) \pi_{0J}(y_J)$ are obtained as follows. Based on approximation results for continuous densities by normal mixtures from ShenTokdarGhosal2013, we obtain approximations for $f_{0|J}(\cdot|y_J)$ for every $y_J$ in the form
where the parameters of the mixture will be defined precisely below. For the discrete variables over which smoothing is not performed, $y_J$, we show that $\pi_{0J}(y_J)$ can be appropriately approximated by \[ \int_{A_{y_J}} \sum_{y_J^\prime} \pi_{0J}(y_J^\prime) \phi(\tilde{y}_J; y_J^\prime, \sigma_{J}^\star) d\tilde{y}_J, \] where $\int_{A_{y_J}} \phi(\tilde{y}_J, y_J^\prime, \sigma_{J}^\star) d\tilde{y_J}$ behaves like an indicator $1\{{y_J}=y_J^\prime\}$ for sufficiently small $\sigma_{J}^\star$. The following subsection presents proof details.
Define $\beta= d_{J^c} \left [ \sum_{k \in J^c} \beta_k^{-1}\right ]^{-1}$, $\beta_{\min}=\min_{j\in J^c} \beta_j$, and $\sigma_n = [\tilde{\epsilon}_n / \log (1/ \tilde{\epsilon}_n) ]^{1/\beta}$. For $\varepsilon$ defined in (ref)-(ref), $b$ and $\tau$ defined in (ref), and a sufficiently small $\delta>0$, let $a_0 = \{ (8\beta + 4\varepsilon +8 + 8 \beta/\beta_{\min})/(b \delta)\}^{1/\tau}$, $a_{\sigma_n} = a_0 \{\log (1/\sigma_n) \}^{1/\tau}$, and $b_1 > \max \{1, 1/ 2\beta \}$ satisfying $\tilde{\epsilon}_n^{b_1} \{ \log (1/ \tilde{\epsilon}_n) \}^{5/4} \leq \tilde{\epsilon}_n$. Then, the proofs of Theorems 4 and 6 in ShenTokdarGhosal2013 imply the following two claims for each $y_J=k\in\mathcal{Y}_J$ under the assumptions of Section (ref).
First, there exists a partition $\{U_{j|k}, j=1,\ldots,K\}$ of $\{\tilde{x} \in \tilde{\mathcal{X}}: ||\tilde{x}|| \leq 2a_{\sigma_n}\}$, such that for $j=1,\ldots,N$, $U_{j|k}$ is contained within an ellipsoid with center $\mu^\star_{j|k}$ and radii $\{ \sigma_n^{\beta/\beta_i} \tilde{\epsilon}_n^{2 b_1}, \, i \in J^c\}$ \[ U_{j|k} \subset \left\{ \tilde{x}: \sum_{i=1}^{d_{J^c}} \left[(\tilde{x}_i-\mu^\star_{j|k,i})/(\sigma_n^{\beta/\beta_{d_J+i}} \tilde{\epsilon}_n^{2b_1})\right]^2\leq 1 \right\};\] for $j=N+1,\ldots,K$, $U_{j|k}$ is contained within an ellipsoid with radii $\{ \sigma_n^{\beta/\beta_i}, \, i \in J^c\}$, and $1 \leq N < K \leq C_1 \sigma_n^{-d_{J^c}} \{\log (1/ \tilde{\epsilon}_n) \}^{d_{J^c}+d_{J^c}/\tau}$, where $C_1>0$ does not depend on $n$ and $y_J$.
Second, for each $k\in\mathcal{Y}_J$ there exist $\alpha_{j|k}^\star$, $j = 1,\ldots,K$, with $\alpha_{j|k}^\star=0$ for $j > N$, and $\mu_{jk}^{x\star} \in U_{j|k}$ for $j=N+1,\ldots,K$ such that for a positive constant $C_{2}$ and $\sigma^\star_{J^c}=\{\sigma_n^{\beta/\beta_i}$ for $i\in J^c\}$,
where $f_{|J}^\star$ is defined in (ref). Constant $C_{2}$ is the same for all $k \in \mathcal{Y}_J$ since all the bounds on $f_{0|J}$ assumed in Section (ref) are uniform over $k$.
Note also that our smoothness definition is different from the one used by ShenTokdarGhosal2013. In Lemmas (ref) and (ref) we show that our smoothness definition ($ f_{0|J} \in \mathcal{C}^{L,\beta_{d_J+1},\ldots,\beta_{d}})$ delivers an anisotropic Taylor expansion with bounds on remainder terms such that the argument on p. 637 of ShenTokdarGhosal2013 goes through.
Third, by Lemma (ref), which is an extension of a part of Proposition 1 in ShenTokdarGhosal2013, there exists a constant $B_0>0$ such that for all $y_J\in\mathcal{Y}_{J}$
where \[ \underline{\sigma}_{n} = \min_{i\in J^c}\sigma_{n}^{\beta/\beta_i}. \]
For $m=N_JK$ we define $\theta^\star$ and $S_{\theta^\star}$ as:
The rest of the proof of the Kullback-Leibler thickness condition follows the general argument developed for mixture models in GhosalVandervaart:07 and ShenTokdarGhosal2013 among others. First, we will show that for $m=N_JK$ and $\theta\in S_{\theta^\star}$, the Hellinger distance $d_H^2(p_0(\cdot,\cdot), p(\cdot,\cdot | \theta, m ))$ can be bounded by $\sigma_n^{2\beta}$ up to a multiplicative constant. Second, we construct bounds on the ratios $p(\cdot,\cdot | \theta, m )/p_0(\cdot,\cdot)$ and combine them with the bound on the Hellinger distance using Lemma (ref). Finally, we will show that the prior puts sufficient probability on $m=N_JK$ and $S_{\theta^\star}$.
For $f_{|J}^\star$ defined in (ref), let us define \[ p_{|J}^\star(y_I,x |y_J)=\int_{A_{y_I}} f_{|J}^\star(\tilde{y}_I,x |y_J) d\tilde{y}_I. \] For $m=N_JK$ and $\theta\in S_{\theta^\star}$, we can bound the Hellinger distance between the DGP and the model as follows,
It follows from (ref) and Lemma (ref) linking distances between probability mass functions and corresponding latent variable densities that the first term on the right hand side of this inequality is bounded by $(C_2)^2\sigma_n^{2\beta}$. Combining this result with the bound on $d_H^2(p_{|J}^\star(\cdot|\cdot)\pi_{0J}(\cdot), p(\cdot,\cdot | \theta, m ))$ from Lemma (ref) we obtain
Next, for $\theta \in S_{\theta^\star}$ and $m=N_JK$, let us consider lower bounds on the ratio $p(y_J,y_I,{x} | \theta, m)/p_{0}(y_J,y_I,x)$. In Lemma (ref) in the Appendix we show that lower bounds on the ratio $f_J(y_J,\tilde{x} | \theta, m)/f_{0|J}(\tilde{x} | y_J)\pi_{0}(y_J)$ imply the following bounds for all sufficiently large $n$: for any ${x} \in {\mathcal{{X}}}$ with $\left\Vert{x}\right\Vert \leq a_{\sigma_n}$,
for some constant $C_3>0$; and for any ${x} \in {\mathcal{{X}}}$ with $\left\Vert{x}\right\Vert > a_{\sigma_n}$,
for some constant $C_4>0$. Consider all sufficiently large $n$ such that $\lambda_n < e^{-1}$ and (ref) and (ref) hold. Then, for any $\theta \in S_{\theta^\star}$,
for some constant $C_{5}>0$ and all sufficiently large $n$, where the last inequality holds by the tail condition in (ref), (ref), and $(\log n)^2\sigma_n^{2\beta+\varepsilon}\underline{\sigma}_{n}^8 \rightarrow 0$.
Furthermore, as $\lambda_n< e^{-1}$,
and, therefore,
Inequalities (ref), (ref), and (ref) combined with Lemma (ref) imply \[ E_0 \left( \log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)} \right)\leq A\tilde{\epsilon}_n^2, \; E_0 \left( \left[ \log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)} \right]^2 \right) \leq A\tilde{\epsilon}_n^2 \] for any $\theta \in S_{\theta^\star}$, $m=N_JK$, and some positive constant $A$ (details are provided in Lemma (ref) in the Appendix).
By Lemma (ref) in the Appendix for all sufficiently large $n$, $s = 1 + 1/\beta + 1/\tau$, and some $C_6>0$,
The last expression of the above display is bounded below by $\exp\{-C n \tilde{\epsilon}_n^2 \}$ for any $C>0$, $\tilde{\epsilon}_n = \left[ \frac{N_J}{n}\right]^{\beta/(2\beta + d_{J^c})} (\log n)^{t_J}$, any $t_J > (d_{J^c}s + \max\{\tau_1,1,\tau_2/\tau\}) / (2 + d_{J^c}/\beta)$, and all sufficiently large $n$. Since the inequality in the definition of $t_J$ is strict, the claim of the theorem follows.
When $J = \emptyset$ and $N_J=1$, the preceding argument delivers the claim of the theorem if we add an artificial discrete coordinate with only one possible value to the vector of observables.
In this case, the proof from the previous subsection can be simplified as follows. For $m=N_J$ and for any $\beta>0$ we define $\theta^\star$ and $S_{\theta^\star}$ as
For $m=N_J$ and $\theta\in S_{\theta^\star}$, a simplification of the proof of Lemma (ref) delivers
A simplification of derivations in Lemma (ref) show that for all $y_J\in\mathcal{Y}_J$
Then, for any $\theta \in S_{\theta^\star}$
as $\frac{p(y_J| \theta, m)}{p_0(y_J)} \geq \lambda_n$ for all $y_J\in\mathcal{Y}_J$. As $\lambda_n\rightarrow 0$, by Lemma (ref) for $\lambda_n<\lambda_0$, both $E_0 ( \log \frac{p_0(y_J)}{p(y_J| \theta, m)} )$ and $E_0 ( [ \log \frac{p_0(y_J)}{p(y_J| \theta, m)} ]^2)$ are bounded by $C_{7} \log (1/\lambda_n)^2\sigma_n^{2\beta} \leq A\tilde{\epsilon}_n^2$ for some constant $A$. By the simplification of Lemma (ref) for this particular case for all sufficiently large $n$ and some $C_{8}>0$,
The last expression of the above display is bounded below by $\exp\{-C n \tilde{\epsilon}_n^2 \}$ for any $C>0$, $\tilde{\epsilon}_n = \left[ \frac{N_J}{n}\right]^{1/2} (\log n)^{t_J}$, any $t_J > \max\{\tau_1,1\} /2 $, and all sufficiently large $n$. Since the inequality in the definition of $t_J$ is strict, the claim of the theorem follows.
It seems feasible to extend the results of this paper to conditional density estimation by covariate dependent mixtures along the lines of norets_pati_2017. We leave this to future work.