The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
54,700 characters
Adaptive Bayesian Estimation of Mixed Discrete-Continuous Distributions under Smoothness and Sparsity
\begin{frontmatter}
\title{Adaptive Bayesian Estimation of Mixed Discrete-Continuous Distributions under Smoothness and Sparsity
}
\runtitle{
Adaptive Bayesian Estimation of Mixed Distributions}
\thankstext{T1}{First version: December 2017, current version: \today.}
\thankstext{T2}{We thank participants of Harvard-MIT econometrics workshop, OBayes 2017, CIREQ 2018, and SBIES 2018 for helpful comments.}
\begin{aug}
\author{\fnms{Andriy} \snm{Norets}\thanksref{t1}
\ead[label=e1]{andriy\[email removed]}
}
\and
\author{\fnms{Justinas} \snm{Pelenis}\thanksref{t2}
\ead[label=e2]{[email removed]}
}
\thankstext{t1}{Associate Professor, Department of Economics, Brown University
}
\thankstext{t2}{Assistant Professor, Vienna Institute for Advanced Studies
}
\runauthor{A. Norets and J. Pelenis}
\affiliation{Brown University and Vienna Institute for Advanced Studies}
\address{Economics Department, \\
Brown University,
Providence, RI 02912
\\
\printead{e1}
}
\address{ Institute for Advanced Studies Vienna,
\\ Josefstaedter Strasse 39,
\\ Vienna 1080, Austria
\printead{e2}
}
\end{aug}
\begin{abstract}
We consider nonparametric estimation of a mixed discrete-continuous distribution
under anisotropic smoothness conditions and
possibly increasing number of support points for the discrete part of the distribution.
For these settings, we derive lower bounds on the estimation rates in the total variation distance. Next, we consider a nonparametric mixture of normals model that uses continuous latent variables for the discrete part of the observations. We show that the posterior in this model contracts at rates that are equal to the derived lower bounds up to a log factor.
Thus, Bayesian mixture of normals models can be used for optimal adaptive estimation of mixed discrete-continuous distributions.
\end{abstract}
\begin{keyword}
\kwd{Bayesian nonparametrics, adaptive rates, minimax rates, posterior contraction, discrete-continuous distribution, mixed scale, mixtures of normal distributions, latent variables.}
\end{keyword}
\end{frontmatter}
\section{Introduction}
Mixture models have proven to be very useful for Bayesian nonparametric modeling of univariate and multivariate distributions of continuous variables.
These models possess outstanding asymptotic frequentist properties:
in Bayesian nonparametric estimation of smooth densities the posterior in these models contracts at optimal adaptive rates up to a log factor
(\cite{Rousseau:10}, \cite{KruijerRousseauVaart:09}, \cite*{ShenTokdarGhosal2013}).
Tractable Markov chain Monte Carlo (MCMC) algorithms for exploring posterior distributions of these models are available (\cite{EscobarWest:95}, \cite{MacEachernMuller98}, \cite{Neal:00}, \cite{MillerHarrison17}, \cite{Norets2017mcmc}) and they are widely used in empirical work (see \cite*{PractNonSemiparamBayes:98},
\cite{ChamberlainHirano99}, \cite*{BurdaHardingHausman:08}, \cite{ChibGreenberg10}, and \cite{JensenMaheu14} among many others).
In most applications, data contain both continuous and discrete variables.
From the computational perspective, discrete variables can be easily accommodated through
the use of continuous latent variables in Bayesian MCMC estimation (\cite{AlbertChib_binpoly:93}, \cite{McCullochRossi:94}). In nonparametric modelling of discrete-continuous data by mixtures,
latent variables were used by \cite{CanaleDunson:11} and \cite{NoretsPelenis2012} among others.
Some results on frequentist asymptotic properties of the posterior distribution in such models
have also been established. \cite{NoretsPelenis2012} obtained approximation results in Kullback-Leibler distance and weak posterior consistency for mixture models with a prior on the number of mixture components. \cite{DeYoreoKottas17} establish weak posterior consistency for Dirichlet process mixtures.
In similar settings, \cite{canale2015bayesian} derived posterior contraction rates that are not optimal.
The question we address in the present paper is whether
a mixture of normal model that uses latent variables for modeling the discrete part of the distribution
can deliver (near) optimal and adaptive posterior contraction rates for nonparametric estimation of discrete-continuous distributions.
Our contribution has two main parts. First, we derive lower bounds on the estimation rate for mixed multivariate discrete-continuous distributions under anisotropic smoothness conditions and potentially growing support of the discrete part of the distribution. Second, we study the posterior contraction rate
for a mixture of normals model with a variable number of components that uses continuous latent variables for the discrete part of the observations. We show that the posterior in this model contracts at rates that are equal to the derived lower bounds up to a log factor.
Thus, Bayesian mixture models can be used for (up to a log factor) optimal adaptive estimation of mixed discrete-continuous distributions. These results are obtained in a rich asymptotic framework where the multivariate discrete part of the data generating distribution can have either a large or a small number of support points and it can be either very smooth or not, and these characteristics can differ from one discrete coordinate to another. In these settings,
smoothing is beneficial only for a subset of discrete variables with
a quickly growing number of support points and/or high level of smoothness.
In a sense, this subset is automatically and correctly selected by the mixture model.
The obtained optimal posterior contraction rates are
adaptive since the priors we consider do not depend on the number of support points and the smoothness of the data generating process.
Our results on lower bounds have independent value outside of the literature on Bayesian mixture models and their frequentist properties. Let us briefly review most relevant results on lower bounds and place our results in that context.
The minimax estimation rates for mixed discrete continuous distributions appear to be studied first by
\cite{Efromovich2011}. He considers discrete variables with a fixed support and shows that the optimal rates for discrete continuous distributions are equal to the optimal nonparametric rates for
the continuous part of the distribution. Relaxing the assumption of the fixed support
for the discrete part of the distribution is very desirable in nonparametric settings.
It has been commonly observed at least since \cite{AitchisonAitken76} that smoothing discrete
data in nonparametric estimation improves results in practice.
\cite{HallTitterington87} introduced an asymptotic framework that provided a precise theoretical
justification for improvements resulting from smoothing in the context of estimating a univariate discrete distribution with a support that can grow with the sample size.
In their setup, the support is an ordered set and the probability mass function is $\beta$-smooth (in a sense that analogs of $\beta$-order Taylor expansions hold). They show that in their setup the minimax rate
is the smaller one of the following two: (i) the optimal estimation rate for a continuous density with the smoothness level $\beta$, $n^{-\beta/(2\beta+1)}$, and (ii) the rate of convergence of the standard frequency estimator, $(N/n)^{1/2}$, where $N$ is the cardinality of the support and $n$ is the sample size. \cite{HallTitterington87} refer to their setup as ``Sparse Multinomial Data'' since $N$ can be larger than $n$ and this is the reason we refer to sparsity in the tile of the present paper.
\cite{Burman87} established similar results for $\beta=2$.
Subsequent literature in multivariate settings (e.g., \cite{DongSimonoff1995},
\cite{AertsAugustynsJanssen97statistics}) did not consider lower bounds but demonstrated that
when the support of the discrete distribution grows sufficiently fast then estimators that employ smoothing can achieve the standard nonparametric rates for $\beta$-smooth densities on $\mathbb{R}^d$,
$n^{-\beta/(2\beta+d)}$.
We generalize the results of \cite{HallTitterington87} on lower bounds for univariate discrete distributions to multivariate mixed discrete-continuous case and anisotropic smoothness.
Alternatively, our results can be viewed as a generalization of results in \cite{Efromovich2011}
to settings with potentially growing supports for discrete variables.
Some details of our settings and assumptions differ from those in \cite{HallTitterington87}
and \cite{Efromovich2011} because our original motivation was in understanding the behavior of the posterior in mixture models with latent variables.
Specifically, we consider lower and upper estimation bounds in the total variation distance
since posterior concentration in nonparametric settings is much better understood when
the total variation distance is considered (\cite{GhosalGhoshVaart:2000}).
We also introduce a new definition of anisotropic smoothness that, on the one hand,
accommodates an extension of techniques for deriving lower bounds from
\cite{IbragimovHasminskii1984} and, on the other hand, lets us exploit approximation results
for mixtures of multivariate normal distributions developed by \cite{ShenTokdarGhosal2013}.
The rest of the paper is organized as follows. In Section \ref{sec:notation}, we describe our framework and define notation. Section \ref{sec:main_results} presents our results on lower bounds for estimation rates.
The results on the posterior contraction rates are given in Section \ref{sec:post_rates}.
Appendix contains auxiliary results and some proofs.
\section{Preliminaries and Notation}
\label{sec:notation}
Let us denote the continuous part of observations by $x \in \mathcal{X} \subset \mathbb{R}^{d_x}$ and the discrete part by
$y=(y_1,\ldots,y_{d_y}) \in \mathcal{Y}$, where
\begin{align*}
\mathcal{Y}=\prod_{j=1}^{d_y} \mathcal{Y}_j, \text{ with }
\mathcal{Y}_j=\left \{ \frac{1-1/2}{N_j},\frac{2-1/2}{N_j}, \ldots, \frac{N_j-1/2}{N_j} \right\} ,
\end{align*}
is a grid on $[0,1]^{d_y}$ (a product symbol $\Pi$ applied to sets hereafter denotes a Cartesian product).
The number of values that the discrete coordinates $y_j$ can take, $N_j$, can potentially grow with the sample size or stay constant.
For $y=(y_1,\ldots,y_{d_y}) \in \mathcal{Y}$, let $A_{y} = \prod_{j=1}^{d_y} A_{y_j}$, where
\begin{equation*}
A_{y_j} = \begin{cases}
(-\infty, y_j+0.5/N_j] & \text{if } y_j=0.5/N_j\\
(y_j-0.5/N_j, \infty) &\text{if } y_j=1-0.5/N_j\\
(y_j-0.5/N_j, y_j+0.5/N_j] &\text{otherwise}
\end{cases}
\end{equation*}
and let us represent the data generating density-probability mass function as
\begin{equation}
\label{eq:p0f0repr}
p_0(y,x)= \int_{A_y} f_0(\tilde{y}, x) d\tilde{y},
\end{equation}
where $f_0$ belongs to $\mathcal{D}$, the set of probability density functions
on $\mathbb{R}^{d_x+d_y}$ with respect to the Lebesgue measure. The representation of a mixed discrete-continuous distribution in \eqref{eq:p0f0repr} is so far without a loss of generality
since for any given $p_0$ one could always define $f_0$ using a mixture of densities with non-overlapping supports included in $A_y$, $y \in \mathcal{Y}$.
In this paper, we consider independently identically distributed observations from $p_0$:
$(Y^n,X^n)=(Y_1,X_1,\ldots,Y_n,X_n)$. Let $P_0$, $E_0$, $P_0^n$, and $E_0^n$ denote the probability measures and expectations corresponding to $p_0$ and its product $p_0^n$.
When $N_j$'s grow with the sample size the generality of the representation in \eqref{eq:p0f0repr} can be lost when assumptions such as smoothness are imposed on $f_0$.
Nevertheless, in what follows we do impose a smoothness assumption on $f_0$. The interpretation of this assumption is that
the values of discrete variables can be ordered and that borrowing of information from nearby discrete points can be useful in estimation.
To get more refined results, we
allow $N_j$'s to grow at different rates for different $j$'s.
For the same reason, we work with anisotropic smoothness.
Let $\mathbb{Z}_+$ denote the set of non-negative integers. For smoothness coefficients $\beta_i>0$, $i=1,\ldots,d$, $d=d_x+d_y$,
and an envelope function $L:\mathbb{R}^{2d}\rightarrow \mathbb{R}$,
an anisotropic $(\beta_1,\ldots,\beta_d)$-Holder class, $\mathcal{C}^{\beta_1,\ldots,\beta_d,L}$, is defined as follows.
\begin{definition}
\label{def:anis_smooth}
$f \in \mathcal{C}^{\beta_1,\ldots,\beta_d,L}$ if for any $k=(k_1,\ldots,k_d) \in \mathbb{Z}_+^d$, $\sum_{i=1}^d k_i/\beta_i<1$,
mixed partial derivative of order $k$, $D^k f$, is finite and
\begin{equation}
\label{eq:smooth_def}
|D^k f (z+\Delta z)-D^k f (z) | \leq L(z, \Delta z) \sum_{j=1}^d |\Delta z_j|^{\beta_j(1-\sum_{i=1}^d k_i/\beta_i)},
\end{equation}
where $\Delta z_j=0$ when $\sum_{i=1}^d k_i/\beta_i + 1/\beta_j < 1 $.
\end{definition}
In this definition, a Holder condition is imposed on $D^k f$ for a coordinate $j$ when $D^k f$ cannot be differentiated with respect to $z_j$ anymore
($\sum_{i=1}^d k_i/\beta_i<1$ but $\sum_{i=1}^d k_i/\beta_i + 1/\beta_j \geq 1 $).
This definition slightly differs from definitions available in the literature on anisotropic smoothness that we found.
Section 13.2 in \cite{Schumaker2007splines} presents some general anisotropic smoothness definitions but restricts attention to integer smoothness coefficients.
\cite{IbragimovHasminskii1984}, and most of the literature on minimax rates under anisotropic smoothness that followed including
\cite{BarronBirgeMassart99} and \cite{BhattacharyaPatiDunson2014}, do not restrict mixed derivatives.
\cite{ShenTokdarGhosal2013} use $|\Delta z_j|^{\min(\beta_j - k_j,1)}$ instead of $|\Delta z_j|^{\beta_j(1-\sum k_i/\beta_i)}$ in \eqref{eq:smooth_def}. Their requirement is stronger than ours for functions with bounded support, and
it appears too strong for our derivation of lower bounds on the estimation rate.
However, our definition is sufficiently strong to obtain a Taylor expansion with remainder terms that have the same order as those in \cite{ShenTokdarGhosal2013}
(while the definitions that do not restrict mixed derivatives do not deliver such an expansion).
When $\beta_j=\beta$, $\forall j$ and $\sum_{i=1}^d k_i/\beta + 1/\beta \geq 1 $,
$\beta_j(1-\sum k_i/\beta_i) = \beta - \lfloor \beta \rfloor$,
where $\lfloor \beta \rfloor$ is the largest integer that is strictly smaller than $\beta$, and we
get the standard definition of $\beta$-Holder smoothness for the isotropic case.
The envelope $L$ can be assumed to be a function of $(z,\Delta z)$ to accommodate densities with unbounded support.
We derive lower bounds on estimation rates for a constant envelope function.
Upper bounds on posterior contraction rates are derived under more general assumptions on $L$ as in \cite{ShenTokdarGhosal2013}.
Some extra notation: for a multi-index $k=(k_1,\ldots,k_d) \in \mathbb{Z}_+^d$, $k!=\prod_{i=1}^d k_i!$, and
for $z \in \mathbb{R}^d$, $z^k=\prod_{i=1}^d z_i^{k_i}$.
The $m$-dimensional simplex is denoted by $\Delta^{m-1}$. $I_d$ stands for the $d \times d$ identity matrix.
Let $\phi_{\mu,\sigma}(\cdot)$ and $\phi(\cdot; \mu,\sigma)$ denote a multivariate normal density with mean $\mu \in \mathbb{R}^d$ and covariance
matrix $\sigma^2 I_d$ (or a diagonal matrix with squared elements of $\sigma$ on the diagonal
when $\sigma$ is a $d$-vector). For $z \in \mathbb{R}^d$ and $J \subset \{1,2,\ldots,d\}$, $z_J$ denotes sub-vector $\{z_i,\, i \in J\}$.
Operator ``$\lesssim$'' denotes less or equal up to a multiplicative positive constant relation.
\section{Lower Bounds on Estimation Rates}
\label{sec:main_results}
Let $\mathcal{A}$ denote a collection of all subsets of indices for discrete coordinates $\{1, \ldots,d_y\}$.
For $J \in \mathcal{A}$, define $J^c=\{1,\ldots,d\}\setminus J$,
\[
N_{J}=\prod_{i\in J} N_i, \;\;\;\;\;\;\;
\beta_{J^c} = \left [ \sum_{i \in J^c} \beta_i^{-1}\right ]^{-1},
\]
$N_{\emptyset}=1$, $\beta_{\emptyset} = \infty$,
and $\beta_{\emptyset}/(2\beta_{\emptyset} + 1 )=1/2$.
For a class of probability distributions $\mathcal{P}$, $\zeta$ is said to be a lower bound on the estimation error in metric $\rho$ if
\[
\inf_{\hat{p}} \sup_{p \in \mathcal{P}} P\left(\rho(\hat{p},p)\geq \zeta \right) \geq const > 0.
\]
We consider the following class of probability distributions: for a positive constant $L$, let
\begin{equation}
\label{eq:classP}
\mathcal{P}=\left \{p: \: p(y,x)=\int_{A_y} f(\tilde{y}, x) d\tilde{y}, \, f \in \mathcal{C}^{\beta_1,\ldots,\beta_d,L} \cap \mathcal{D} \right\}.
\end{equation}
\begin{theorem}
\label{th:LowerBounds}
For $\mathcal{P}$ defined in \eqref{eq:classP},
\begin{equation}
\label{eq:l_b}
\Gamma_n = \min_{J \in \mathcal{A}} \left[ \frac{N_J}{n} \right ]^{\frac{\beta_{J^c}}{2\beta_{J^c}+1}}
=\left[ \frac{N_{J_\ast}}{n} \right ]^{\frac{\beta_{J^c_\ast}}{2\beta_{J^c_\ast}+1}}
\end{equation}
multiplied by a positive constant is a lower bound on estimation error in the total variation distance.
\end{theorem}
One could recognize expression $\left[ N_J/n \right ]^{\frac{\beta_{J^c}}{2\beta_{J^c}+1}}$ in \eqref{eq:l_b}
as the standard estimation rate for a $card(J^c)$-dimensional density with anisotropic smoothness coefficients $\{\beta_j, \, j \in J^c\}$ and the sample size $n/N_J$ (\cite{IbragimovHasminskii1984}).
One way to interpret this is
that the density of $\{x,\tilde{y}_j,\, j \in J^c\}$ conditional on $y_J$ is $\{\beta_j, \, j \in J^c\}$-smooth and the number of observations available for its estimation (observations with the same value of $y_J$)
should be of the order $n/N_J$; also, the estimation rate for the marginal probability mass function for $y_J$ is $[N_J/n]^{1/2}$, which is at least as fast as $\left[ N_J/n \right ]^{\frac{\beta_{J^c}}{2\beta_{J^c}+1}}$.
In this interpretation, smoothing is not performed over the discrete coordinates with indices in set $J$, and the lower bound is obtained when $J$ minimizes $\left[ N_J/n \right ]^{\frac{\beta_{J^c}}{2\beta_{J^c}+1}}$.
Thus, an estimator that delivers the rate in \eqref{eq:l_b} should, in a sense, optimally choose the subset of discrete variables over which to perform smoothing.
We set up the notation and an outline of the proof of Theorem \ref{th:LowerBounds} below and delegate detailed calculations to lemmas in Appendix \ref{sec:proofs_extras}.
The proof of the theorem is based on a general theorem
from the literature on lower bounds, which we present next in a slightly simplified form.
\begin{lemma}
\label{lm:AbstrLBTsyb}
(Theorem 2.5 in \cite{Tsybakov:08}, see also \cite{IbragimovHasminskii1977})
$\zeta$ is a lower bound on the estimation error in metric $\rho$ for a class $\mathcal{Q}$ if
there exist a positive integer $M \geq 2$ and $q_j,q_i \in \mathcal{Q}$, $0\leq j < i \leq M$ such that
$\rho(q_j,q_i) \geq 2 \zeta$, $q_j << q_0$, $j=1,\ldots,M$ and
\begin{equation}
\label{eq:avgKLcond}
\sum_{j=1}^M KL(Q_{j}^n,Q_{0}^n)/M < \log(M)/8,
\end{equation}
where $KL$ is the Kullback-Leibler divergence and $Q_{j}^n$ is the distribution of a random sample from $q_j$.
\end{lemma}
The following standard result on bounding the number of unequal elements in binary sequences is used in our construction of $q_j$, $j=1,\ldots,M$.
\begin{lemma}
\label{lm:VarshamovGilbert}
(Varshamov-Gilbert bound, Lemma 2.9 in \cite{Tsybakov:08})
Consider the set of all binary sequences of length $\bar{m}$,
$\Omega =\left\{w=(w_1, \ldots, w_{\bar{m}}):\; w_r \in \{0, 1\} \right \} = \{0, 1\}^{\bar{m}}.$
Suppose $\bar{m} \geq 8$. Then there exists
a subset $\{w^1, \ldots, w^M\}$ of $\Omega$ such that $w^0 = (0,\ldots, 0)$,
\[
\sum_{r=1}^{\bar{m}} 1\{w^j_r\neq w^i_r\} \geq \bar{m}/8, \; \forall 0 \leq j < i \leq M,
\]
and
\[M \geq 2^{\bar{m}/8}.
\]
\end{lemma}
To define $q_j$'s for our problem, we need some additional notation.
Let
\[
K_0(u) = \exp\{-1/(1-u^2)\} \cdot 1\{|u|\leq 1\}.
\]
This function has bounded derivatives of all orders and it
smoothly decreases to zero at the boundary of its support.
This type of kernel functions is usually used for constructing hypotheses for lower bounds, see Section 2.5 in \cite{Tsybakov:08}.
Since we need to construct a smooth density
that integrates to 1, we define
(as illustrated in Figure \ref{fig:fun_g})
\[
g(u)=c_0 [K_0(4(u+1/4))-K_0(4(u-1/4))],
\]
where $c_0>0$ is a sufficiently small constant that will be specified below.
\\[\intextsep]
\begin{minipage}{\linewidth}
\centering
\includegraphics[width=100mm]{fun_g.png}
\figcaption{Function $g$ for $c_0=1$.}
\label{fig:fun_g}
\end{minipage}
\\[\intextsep]
Function $g$ will be used as a kernel in construction of $q_k$'s. Let us define the bandwidth for these kernels first.
For the continuous coordinates, we define the bandwidth as in \cite{IbragimovHasminskii1984},
\[
h_i=\Gamma_n^{1/\beta_i},\, i \in \{d_y+1,\ldots,d\}.
\]
For the discrete ones, over which smoothing is beneficial,
we define the bandwidth as
\[
h_i=\varrho_i\cdot\Gamma_n^{1/\beta_i}=\frac{2}{N_i}\cdot R_i,\, i \in J_\ast^c \cap \{1,\ldots,d_y\},
\]
where
$R_i=\lfloor \Gamma_n^{1/\beta_i} N_i / 2\rfloor +1$ is a positive integer and
$\varrho_i\in(1,2]$ as shown in
Lemma \ref{lm:b_bast_h_N}.
For the rest of the discrete coordinates, our innovation is to first define artificial anisotropic smoothness coefficients $\beta_i^\ast=-\log(\Gamma_n)/\log{N_i}$, $i \in J_\ast$, at which the rate in
\eqref{eq:l_b} would have the same value whether we smooth over $y_i$ ($i \in J_\ast^c$) or not ($i \in J_\ast$).
Then, we define the bandwidth as
\[h_i=2\cdot \Gamma_n^{1/\beta_i^\ast}=2/N_i, \ i \in J_\ast.\]
To streamline the notation, we also define $\beta_i^\ast=\beta_i$ for $i \in J_\ast^c$.
Let $m_i$ be the integer part of $h_i^{-1}$, $i=1,\ldots,d$.
Let us consider $\bar{m}=\prod_{i=1}^d m_i$ adjacent rectangles in $[0,1]^d$, $B_r$, $r=1,\ldots,\bar{m}$, with the side lengths $(h_1,\ldots,h_d)$ and
centers $c^r=(c^r_1,\ldots,c^r_d)$,
$c_i^r=h_i(k_{ir} - 1/2)$, $k_{ir} \in\{1,\ldots,m_i\}$.
For $z \in \mathbb{R}^d$ and $r=1,\ldots,\bar{m}$, define
\[
g_r (z)= \Gamma_n \prod_{i=1}^d g((z_i-c^r_i)/h_i),
\]
which can be non-zero only on $B_r$.
A set of hypotheses is defined by sequences of binary weights on $g_r$'s as follows
\begin{equation}
\label{eq:q_j}
q_j(y,x) = \int_{A_y} \left [1_{[0,1]^d}(\tilde{y},x) + \sum_{r=1}^{\bar{m}} w^j_r g_r(\tilde{y},x) \right ] d\tilde{y},
\end{equation}
where $w^j_r \in \{0,1\}$, $j=0,\ldots,M$, and $M$ are defined in Lemma \ref{lm:VarshamovGilbert}.
The rest of the proof is delegated to lemmas in Appendix \ref{sec:proofs_extras},
which show that $q_k$ in \eqref{eq:q_j} satisfy the sufficient conditions
from Lemma \ref{lm:AbstrLBTsyb}.
Specifically, Lemma \ref{lm:q_tvd} derives the lower bound on the total variation distance.
Lemma \ref{lm:q_kl} verifies condition \eqref{eq:avgKLcond}
when $\bar{m}\geq 8$.
Lemma \ref{lm:q_smooth}, part (i) of Lemma \ref{lm:b_bast_h_N}, and the fact that $q_k$'s are defined on $[0,1]^d$ imply $q_j \in \mathcal{C}^{\beta_1,\ldots,\beta_d,L}$, $j=0,\ldots,M$.
This argument (Lemma \ref{lm:q_kl} specifically) requires $\bar{m}\geq 8$ as it relies on Lemma \ref{lm:VarshamovGilbert}.
Observe that as $n\rightarrow \infty$, $\bar{m}\geq 8$ if there are continuous variables or there are discrete variables over which smoothing is beneficial ($J_\ast^c \neq \emptyset$).
Thus, $\bar{m}<8$ can happen only if there are no continuous variables and $N_{J_\ast}=N_1 \cdots N_d$ is bounded. This is just a problem of estimating a multinomial distribution with finite support
and the standard results for parametric problems deliver the usual $n^{-1/2}$ rate.
Finally, note that we prove the lower bound results for a class of densities that includes densities that are in $\mathcal{C}^{\beta_1,\ldots,\beta_d,L}$ on $[0,1]^d$.
It is straightforward to modify the proof so that it works for a class of
smooth densities on $\mathbb{R}^d$. To accomplish this we can replace $ 1_{[0,1]^d}(\cdot) $
in \eqref{eq:q_j} with a smooth function on $\mathbb{R}^d$ that has a bounded support and is bounded away from zero on $[0,1]^d$, for example,
\[
\prod_{i=1}^d \left[ 1_{[0,1]}(z_i) + IK_0(z_i+1)*1(z_i<0) + IK_0(2-z_i)*1(z_i>1) \right],
\]
multiplied by a normalization constant,
where $IK_0(z_i)=\int_{-1}^{z_i} K_0(u)du / \int_{-1}^1 K_0(u)du$.
Then, proofs of Lemmas \ref{lm:q_tvd}-\ref{lm:q_smooth} go through with minor modifications.
\pagebreak
\section{Posterior Contraction Rates for a Mixture of Normals Model}
\label{sec:post_rates}
\subsection{Model and Prior}
\label{sec:model_prior}
In this section, we consider a Bayesian model for the data generating process in \eqref{eq:p0f0repr}. We use a mixture of normal distributions with a variable number of components for modelling the joint distribution of $(\tilde{y},x)$,
\begin{align}
\label{eq:joint_mix_def}
f(\tilde{y},x| \theta, m) &= \sum_{j=1}^m \alpha_j \phi(\tilde{y},x;\mu_j,\sigma) \notag \\
p(y,x| \theta, m) &= \int_{A_y} f(\tilde{y},x| \theta, m) d\tilde{y},
\end{align}
where $\theta=(\mu_j^y, \mu_j^x, \alpha_j, j=1,2,\ldots,m; \sigma)$.
We assume the following conditions on the prior $\Pi$ for $(\theta,m)$. For positive constants $a_1, a_2, \ldots, a_{9}$, for each $i\in \{1,\ldots,d\}$ the prior for $\sigma_i$ satisfies
\begin{eqnarray}
\Pi( \sigma^{-2}_i \geq s) &\leq& a_1 \exp \{-a_2 s^{a_3} \} \quad \text{for all sufficiently large} \, \, s > 0
\label{eq:asnPrior_sigma1}\\
\Pi( \sigma^{-2}_i < s) &\leq& a_4 s ^{a_5} \quad \text{for all sufficiently small} \, \, s > 0 \label{eq:asnPrior_sigma2}\\
\Pi\{ s < \sigma^{-2}_i < s(1+t) \} &\geq& a_6 s^{a_7} t^{a_8} \exp \{-a_9 s^{1/2}\}, \quad s > 0, \quad t \in (0,1).
\label{eq:asnPrior_sigma3}
\end{eqnarray}
An example of a prior that satisfies \eqref{eq:asnPrior_sigma1}-\eqref{eq:asnPrior_sigma3} is the inverse Gamma prior
for $\sigma_i$.
Prior for
$(\alpha_1,\ldots,\alpha_m)$ conditional on $m$ is Dirichlet$(a/m,\ldots,a/m)$, $a > 0$.
Prior for the number of mixture components $m$ is
\begin{equation}
\label{eq:asnPrior_m}
\Pi(m=i) \propto \exp(-a_{10} i (\log i)^{\tau_1}), i =2, 3, \ldots, \quad a_{10}>0, \tau_1 \geq 0.
\end{equation}
More generally, a prior that can be bounded above and below by functions in the form of the right hand side of \eqref{eq:asnPrior_m}, possibly with different constants, would also work.
A priori, the components of $\mu_{j}$, $\mu_{j,i}$, $i=1,\dots,d$ are independent from each other, other parameters, and across $j$.
Prior density for $\mu_{j,i}$ is bounded below for some $a_{12}, \tau_2 > 0$ by
\begin{equation}
\label{eq:asnPrior_mu_lb}
a_{11}\exp(-a_{12} |\mu_{j,i}|^{\tau_2} ),
\end{equation}
and for some $a_{13}, \tau_3>0$ and all sufficiently large $\mu > 0$,
\begin{equation}
\label{eq:asnPrior_mu_tail_ub}
\Pi(\mu_{j,i} \notin [-\mu,\mu]) \leq \exp(-a_{13} \mu^{\tau_3}).
\end{equation}
\subsection{Assumptions on the Data Generating Process}
\label{sec:assumpt_dgp}
In what follows, we consider a fixed subset of discrete indices $J \in \mathcal{A}$ and show that under regularity conditions,
the posterior contraction rate is bounded above by $\left[ \frac{N_J}{n} \right ]^{\frac{\beta_{J^c}}{2\beta_{J^c}+1}}$ times a log factor.
If the regularity conditions we describe below for a fixed $J$ hold for every subset of $\mathcal{A}$, then the posterior contraction rate matches the lower bound in
\eqref{eq:l_b} up to a log factor.
Without a loss of generality, let $J=\{1,\ldots,d_J\}$, $I=\{d_{J}+1,\ldots,d_y\}$, $J^c=\{1,\ldots,d\}\setminus J$, and $d_{J^c}=card(J^c)$.
Similarly to $\mathcal{Y}$ and $A_{y}$ defined in Section \ref{sec:notation}, we define
$\mathcal{Y}_J=\prod_{j\in J} \mathcal{Y}_j$ and $A_{y_J}= \prod_{i \in J} A_{y_i}$.
Also, let $y_{J}=\{y_i\}_{i \in J}$, $\tilde{y}_{I}=\{\tilde{y}_i\}_{i \in I}$, $\tilde{x}=(\tilde{y}_I,x) \in \tilde{\mathcal{X}}=\mathbb{R}^{d_{J^c}}$.
To formulate the assumptions on the data generating process, we need additional notation,
\begin{align*}
f_{0J}(y_J,\tilde{x})&= \int_{A_{y_J}} f_0(\tilde{y}_J, \tilde{x}) d\tilde{y}_J,\\
\pi_{0J}(y_J)&=\int_{\tilde{\mathcal{X}}} f_{0J}(y_J,\tilde{x}) d\tilde{x}, \\
f_{0|J}(\tilde{x}|y_J)&=\frac{ f_{0J}(y_J,\tilde{x})}{\pi_{0J}(y_J)},\\
p_{0|J}(y_I,x|y_J)&= \int_{A_{y_I}} f_{0|J}(\tilde{y}_I, x|y_J) d\tilde{y}_I.
\end{align*}
Also, let $F_{0|J}$ and $E_{0|J}$ denote the conditional probability and expectation corresponding to $f_{0|J}$.
If $\pi_{0J}(y_J)=0$ for a particular $y_J$, then we can define the conditional density $f_{0|J}(\tilde{x}|y_J)$ arbitrarily.
We make the following assumptions on the data generating process.
\begin{assumption}
\label{as:f0_subexp_tail}
There are positive finite constants $b,\bar{f}_0,\tau$ such that for any $y_J \in\mathcal{Y}_J$ and $\tilde{x} \in \tilde{\mathcal{X}}$
\begin{align}
\label{eq:tail_cond2}
f_{0|J}(\tilde{x}|y_J ) \leq \bar{f}_0 \exp \left( -b ||\tilde{x}||^{\tau}\right).
\end{align}
\end{assumption}
It appears that all the papers on (near) optimal posterior contraction rates for mixtures of normal densities impose similar tail conditions on data generating densities.
\begin{assumption}
\label{as:f0_Ay_tail_unif_x}
There exists a positive and finite $\bar{y}$ such that for any
$(y_I,y_J) \in \mathcal{Y}$ and $x \in \mathcal{X}$
\begin{equation}
\label{eq:ass_y_I_tail_indep_x}
\int_{A_{y_I} \cap \{||\tilde{y}_I|| \leq \bar{y} \}}
f_{0|J}(\tilde{y}_I, x |y_J) d\tilde{y}_I
\geq
\int_{A_{y_I} \cap \{||\tilde{y}_I|| > \bar{y} \}}
f_{0|J}(\tilde{y}_I, x |y_J) d\tilde{y}_I.
\end{equation}
\end{assumption}
This assumption always holds for $A_{y_I} \subset [0,1]^{d_{J^c}-d_x}$. When $A_{y_I}$ is a rectangle with at least one infinite side,
an interpretation of this assumption is that the tail probabilities for $\tilde{y}_I$ conditional on $(x,y_J)$ decline uniformly in $(x,y_J)$.
Bounded support for $\tilde{y}_I$ is a sufficient condition for this assumption.
\begin{assumption}
\label{as:f0_in_CbL}
We assume that
\begin{equation}
\label{eq:foInC}
f_{0|J} \in \mathcal{C}^{\beta_{d_J+1},\ldots,\beta_{d},L},
\end{equation}
where for some $\tau_0\geq 0$ and any $(\tilde{x}, \Delta \tilde{x}) \in \mathbb{R}^{2d_{J^c}}$
\begin{equation}
\label{eq:L_defb}
L(\tilde{x},\Delta \tilde{x}) = \tilde{L}(\tilde{x}) \exp\left\{ \tau_0 ||\Delta \tilde{x}||^2 \right\},
\end{equation}
\begin{equation}
\label{eq:tildeL_ineq}
\tilde{L}(\tilde{x}+\Delta \tilde{x}) \leq \tilde{L}(\tilde{x}) \exp\left\{ \tau_0 ||\Delta \tilde{x}||^2 \right\}.
\end{equation}
\end{assumption}
The smoothness assumption \eqref{eq:foInC} on the conditional density $f_{0|J}$ is implied by the smoothness of the joint density $f_0$ at least under boundedness away from zero assumption,
see Lemma \ref{lm:finCb_fcondJinC}.
\begin{assumption}
\label{as:Ef0_tildeL}
There are positive finite constants $\varepsilon$ and $\bar{F}$, such that
for any $y_J \in\mathcal{Y}_J$ and $ k =\{k_i\}_{i \in J^c} \in \mathbb{N}_0^{d_{J^c}}$, $\sum_{i \in J^c} k_i/\beta_i < 1$,
\begin{align}
\label{eq:asnE0Dff0}
& \int \left[\frac{|D^k f_{0|J}(\tilde{x}|y_J)|}{f_{0|J}(\tilde{x}|y_J)} \right]^{\frac{(2+\varepsilon\beta_{J^c}^{-1} d_{J^c}^{-1})}{\sum_{i \in J^c} k_i/\beta_i}} f_{0|J}(\tilde{x}|y_J) d \tilde{x} <\bar{F}, \\
\label{eq:asnE0Lf0}
& \int \left[\frac{\tilde{L}(\tilde{x})}{f_{0|J}(\tilde{x}|y_J)}\right]^{2+\varepsilon\beta_{J^c}^{-1} d_{J^c}^{-1}} f_{0|J}(\tilde{x}|y_J) d \tilde{x} < \bar{F}.
\end{align}
\end{assumption}
The envelope function and restrictions on its behaviour are mostly relevant for the case of unbounded support. Condition \eqref{eq:asnE0Lf0} suggests that the envelope function $\tilde{L}$ should be comparable to $f_{0|J}$.
\begin{assumption}
\label{as:Nj_o_n_1nu}
For some small $\nu>0$,
\begin{equation}
\label{eq:Npoly_n}
N_J=o(n^{1-\nu}).
\end{equation}
\end{assumption}
We impose this assumption
to exclude from consideration
the cases with
very slow (non-polynomial) rates
as some parts of the proof require $\log (1/\epsilon_n)$ to be of order $\log n$.
\subsection{Posterior Contraction Rates}
\label{sec:contr_rate}
Let
\begin{equation}
\label{eq:t0def}
t_{J0} = \begin{cases}
\frac{d_{J^c}[1 + 1/(\beta_{J^c}d_{J^c}) + 1/\tau] + \max\{\tau_1,1,\tau_2/\tau\}} {2 + 1/\beta_{J^c}} & \text{if } J^c\neq\emptyset\\
\max\{\tau_1,1\} / 2 &\text{if }J^c = \emptyset
\end{cases}
\end{equation}
where $(\tau,\tau_1, \tau_2)$ are defined in Sections \ref{sec:model_prior}-\ref{sec:assumpt_dgp}.
\begin{theorem} \label{th:post_contr}
Suppose the assumptions from Sections \ref{sec:model_prior}-\ref{sec:assumpt_dgp} hold for a given $J \in \mathcal{A}$.
Let
\begin{equation}
\label{eq:def_eps}
\epsilon_n =
\left[\frac{N_J}{n}\right]^{\beta_{J^c}/(2\beta_{J^c} + 1 )} (\log n)^{t_J},
\end{equation}
where $t_J > t_{J0} + \max \{0, (1- \tau_1)/2\}$.
Suppose also $n \epsilon_n^2 \rightarrow \infty$.
Then, there exists $\bar{M}>0$ such that
\[
\Pi\left( p: d_{TV}(p,p_0) > \bar{M} \epsilon_n | Y^n,X^n\right) \stackrel{P_0^n}{\rightarrow} 0.
\]
\end{theorem}
As in Section \ref{sec:main_results}, when $J^c = \emptyset$, $\beta_{J^c}$ can be defined to be infinity and $\beta_{J^c}/(2\beta_{J^c} + 1 )=1/2$ in \eqref{eq:def_eps}.
\begin{corollary} \label{cl:post_contr_min}
Suppose the assumptions from Sections \ref{sec:model_prior}-\ref{sec:assumpt_dgp} hold for every $J \in \mathcal{A}$.
Let
\begin{equation}
\label{eq:eps_w_min}
\epsilon_n = \min_{J \in \mathcal{A}}
\left[\frac{N_J}{n}\right]^{\beta_{J^c}/(2\beta_{J^c} + 1 )} (\log n)^{t_{J}},
\end{equation}
where $t_J > t_{J0} + \max \{0, (1- \tau_1)/2\}$.
Suppose also $n \epsilon_n^2 \rightarrow \infty$.
Then, there exists $\bar{M}>0$ such that
\[
\Pi\left( p: d_{TV}(p,p_0) > \bar{M} \epsilon_n | Y^n,X^n\right) \stackrel{P_0^n}{\rightarrow} 0.
\]
\end{corollary}
Under the assumptions of the corollary, Theorem \ref{th:post_contr} delivers a valid upper bound on the posterior contraction rate for every $J \in \mathcal{A}$ including the one for which
the minimum in \eqref{eq:eps_w_min} is attained. Hence, the corollary is an immediate implication of Theorem \ref{th:post_contr} whose proof is presented in the following section.
The results on lower bounds in Section \ref{sec:main_results} hold for
any class of data generating densities that includes $f_0$ satisfying the following conditions:
$f_0 \in \mathcal{C}^{\beta_1,\ldots,\beta_d,L}$, $f_0=0$ outside $[0,1]^d$, and
$\overline{f} \geq f_0\geq \underline{f}>0$, where
$L$, $\overline{f}$, and $\underline{f}$ are finite positive constants.
It is worth pointing out that these conditions imply
Assumptions \ref{as:f0_subexp_tail}-\ref{as:Ef0_tildeL},
which combined with Assumption \ref{as:Nj_o_n_1nu} for $N_{J_\ast}$ and a prior specified in Section \ref{sec:model_prior} would deliver
the sufficient conditions of Corollary \ref{cl:post_contr_min}.
\subsection{Proof of Posterior Contraction Results}
\label{sec:post_c_proof_discuss}
To prove Theorem \ref{th:post_contr}, we use
the following sufficient conditions for posterior contraction from Theorem 2.1 in \cite{GhosalVaart:01}.
Let $\epsilon_n$ and $\tilde{\epsilon}_n$ be positive sequences with
$\tilde{\epsilon}_n \leq \epsilon_n$, $\epsilon_n \to 0$, and $n \tilde{\epsilon}_n^2 \to \infty$, and
$c_1$, $c_2$, $c_3$, and $c_4$ be some positive constants.
Let $\rho$ be Hellinger or total variation distance.
Suppose $\mathcal{F}_n \subset \mathcal{F}$ is a sieve with the following bound on the metric entropy $M_e( \epsilon_n, \mathcal{F}_n, \rho)$
\begin{eqnarray} \label{eq:entropy}
\log M_e( \epsilon_n, \mathcal{F}_n, \rho) \leq c_1 n \epsilon_n^2,
\end{eqnarray}
\begin{eqnarray}\label{eq:sievecomplement}
\Pi(\mathcal{F}_n^c) \leq c_3 \exp\{ -(c_2+4)n \tilde{\epsilon}_n^2\}.
\end{eqnarray}
Suppose also that the prior thickness condition holds
\begin{equation}
\label{eq:prior_thick}
\Pi(\mathcal{K}(p_0,\tilde{\epsilon}_n)) \geq c_4 \exp\{ -c_2 n \tilde{\epsilon}_n^2\},
\end{equation}
where the generalized Kullback-Leibler neighborhood $\mathcal{K}(p_0,\tilde{\epsilon}_n)$ is defined by
\begin{align*}
\mathcal{K}(p_0,\epsilon)=\left \{p: \; \int_{\mathcal{X}} \sum_{y\in \mathcal{Y}} p_0(y,x) \log \frac{ p_0(y,x)}{p(y,x)}dx< \epsilon^2, \; \int_{\mathcal{X}} \sum_{y\in \mathcal{Y}} p_0(y,x) \left[\log\frac{ p_0(y,x)}{p(y,x)}\right]^2 dx< \epsilon^2
\right \}.
\end{align*}
Then, there exists $\bar{M}>0$ such that
\[
\Pi\left( p: \rho(p,p_0) > \bar{M} \epsilon_n | Y^n,X^n\right) \stackrel{P_0^n}{\rightarrow} 0.
\]
The definition of the sieve and verification of conditions \eqref{eq:entropy} and \eqref{eq:sievecomplement} closely follow analogous results in the literature on contraction rates for mixture models
in the context of density estimation. The details are given in Lemma \ref{lm:sieve_n} in Appendix \ref{sec:app_sieve_entropy}.
Verification of the prior thickness condition is more involved and we formulate it as a separate result in the following theorem.
\begin{theorem} \label{th:prior_thickness}
Suppose the assumptions from Sections \ref{sec:model_prior}-\ref{sec:assumpt_dgp} hold for a given $J \in \mathcal{A}$.
Let $t_J > t_{J0}$, where $t_{J0}$ is defined in \eqref{eq:t0def}, and
\begin{equation}
\label{eq:def_tilde_eps}
\tilde{\epsilon}_n =
\left[\frac{N_J}{n}\right]^{\beta_{J^c}/(2\beta_{J^c} + 1 )} (\log n)^{t_J}.
\end{equation}
For any $C > 0$ and all sufficiently large $n$,
\begin{eqnarray}\label{eq:KL}
\Pi( \mathcal{K}(p_0, \tilde{\epsilon}_n)) \geq \exp \{-C n \tilde{\epsilon}_n^2 \}.
\end{eqnarray}
\end{theorem}
Approximation results are key for showing the prior thickness condition \eqref{eq:KL}.
Appropriate approximation results for $f_{0J}(y_J,\tilde{x})= f_{0|J}(\tilde{x}|y_J) \pi_{0J}(y_J)$ are obtained as follows.
Based on approximation results for continuous densities by normal mixtures from \cite{ShenTokdarGhosal2013}, we obtain approximations
for $f_{0|J}(\cdot|y_J)$ for every $y_J$ in the form
\begin{equation}
\label{eq:f_J_def}
f_{|J}^\star(\tilde{x}|y_J)=\sum_{j=1}^K \alpha^\star_{j|y_J}\phi(\tilde{x};\mu^\star_{j|y_J},
\sigma_{J^c}^\star),
\end{equation}
where the parameters of the mixture will be defined precisely below.
For the discrete variables over which smoothing is not performed, $y_J$, we show that
$\pi_{0J}(y_J)$ can be appropriately approximated by
\[
\int_{A_{y_J}} \sum_{y_J^\prime} \pi_{0J}(y_J^\prime) \phi(\tilde{y}_J; y_J^\prime, \sigma_{J}^\star) d\tilde{y}_J,
\]
where $\int_{A_{y_J}} \phi(\tilde{y}_J, y_J^\prime, \sigma_{J}^\star) d\tilde{y_J}$ behaves like an indicator $1\{{y_J}=y_J^\prime\}$ for sufficiently small $\sigma_{J}^\star$.
The following subsection presents proof details.
\subsubsection{Proof of Theorem \ref{th:prior_thickness} for $J^c\neq \emptyset$}
Define $\beta= d_{J^c} \left [ \sum_{k \in J^c} \beta_k^{-1}\right ]^{-1}$, $\beta_{\min}=\min_{j\in J^c} \beta_j$, and $\sigma_n = [\tilde{\epsilon}_n / \log (1/ \tilde{\epsilon}_n) ]^{1/\beta}$.
For $\varepsilon$ defined in \eqref{eq:asnE0Dff0}-\eqref{eq:asnE0Lf0},
$b$ and $\tau$ defined in \eqref{eq:tail_cond2},
and a sufficiently small $\delta>0$,
let $a_0 = \{ (8\beta + 4\varepsilon +8 + 8 \beta/\beta_{\min})/(b \delta)\}^{1/\tau}$,
$a_{\sigma_n} = a_0 \{\log (1/\sigma_n) \}^{1/\tau}$,
and $b_1 > \max \{1, 1/ 2\beta \}$ satisfying $\tilde{\epsilon}_n^{b_1} \{ \log (1/ \tilde{\epsilon}_n) \}^{5/4} \leq \tilde{\epsilon}_n$.
Then, the proofs of Theorems 4 and 6 in \cite{ShenTokdarGhosal2013}
imply the following two claims for each $y_J=k\in\mathcal{Y}_J$ under the assumptions of Section \ref{sec:assumpt_dgp}.
First,
there exists a partition $\{U_{j|k}, j=1,\ldots,K\}$ of $\{\tilde{x} \in \tilde{\mathcal{X}}: ||\tilde{x}||
\leq 2a_{\sigma_n}\}$,
such that for $j=1,\ldots,N$,
$U_{j|k}$ is contained within an ellipsoid with
center $\mu^\star_{j|k}$ and radii $\{ \sigma_n^{\beta/\beta_i} \tilde{\epsilon}_n^{2 b_1},
\, i \in J^c\}$
\[
U_{j|k} \subset \left\{ \tilde{x}: \sum_{i=1}^{d_{J^c}} \left[(\tilde{x}_i-\mu^\star_{j|k,i})/(\sigma_n^{\beta/\beta_{d_J+i}} \tilde{\epsilon}_n^{2b_1})\right]^2\leq 1 \right\};\]
for $j=N+1,\ldots,K$,
$U_{j|k}$ is contained within an ellipsoid
with
radii
$\{ \sigma_n^{\beta/\beta_i}, \, i \in J^c\}$,
and $1 \leq N < K \leq C_1 \sigma_n^{-d_{J^c}} \{\log (1/ \tilde{\epsilon}_n) \}^{d_{J^c}+d_{J^c}/\tau}$,
where $C_1>0$ does not depend on $n$ and $y_J$.
Second, for each $k\in\mathcal{Y}_J$ there exist
$\alpha_{j|k}^\star$, $j = 1,\ldots,K$, with $\alpha_{j|k}^\star=0$ for $j > N$,
and $\mu_{jk}^{x\star} \in U_{j|k}$ for $j=N+1,\ldots,K$
such that for a positive constant $C_{2}$ and
$\sigma^\star_{J^c}=\{\sigma_n^{\beta/\beta_i}$ for $i\in J^c\}$,
\begin{equation}
\label{eq:f0upsbeta}
d_H\left(f_{0|J}(\cdot|k),f_{|J}^\star(\cdot|k)\right) \leq C_{2} \sigma_n^\beta,
\end{equation}
where $f_{|J}^\star$ is defined in \eqref{eq:f_J_def}.
Constant $C_{2}$ is the same for all $k \in \mathcal{Y}_J$ since all the bounds on $f_{0|J}$ assumed in Section \ref{sec:assumpt_dgp} are uniform over $k$.
Note also that our smoothness definition is different from the one used by \cite{ShenTokdarGhosal2013}.
In Lemmas \ref{lm:AnisoTaylor} and \ref{lm:AnisoTaylorRemainderBd} we show that our smoothness definition ($ f_{0|J} \in \mathcal{C}^{L,\beta_{d_J+1},\ldots,\beta_{d}})$ delivers an anisotropic Taylor expansion with bounds on remainder terms such that the argument on p. 637 of \cite{ShenTokdarGhosal2013} goes through.
Third, by Lemma \ref{lm:ProbBoundOutside}, which is an extension of a part of Proposition 1 in \cite{ShenTokdarGhosal2013},
there exists a constant $B_0>0$ such that for all $y_J\in\mathcal{Y}_{J}$
\begin{align}
\label{eq:tail_prob_bound}
F_{0|J}\left(||\tilde{X}|| > a_{\sigma_n}| y_J \right)\leq B_0 \sigma_n^{4\beta+2\varepsilon}\underline{\sigma}_{n} ^8,
\end{align}
where
\[
\underline{\sigma}_{n} = \min_{i\in J^c}\sigma_{n}^{\beta/\beta_i}.
\]
For $m=N_JK$ we define $\theta^\star$ and $S_{\theta^\star}$ as:
\begin{align*}
\theta^\star = \bigg\{
& \{ \mu_1^\star,\ldots,\mu_m^\star \} =
\left\{
(k,\mu_{j|k}^{\star}),\:
j=1,\ldots,K, \; k\in\mathcal{Y}_J \right\},
\\
& \{\alpha_1^\star,\ldots,\alpha_m^\star\}
= \left\{ \alpha_{jk}^\star=\alpha_{j|k}^{\star}\pi_{0J}(k), \:
j=1,\ldots,K, \; k\in\mathcal{Y}_J \right\},
\\
&\sigma_{J}^{\star2}=\{\sigma_i^{\star2}=1/[64 N_i^2 \beta \log (1/\sigma_n)], \: i \in J \}
\\
&\sigma_{J^c}^\star=\{\sigma_i^\star=\sigma_{n}^{\beta/\beta_i}, \: i \in J^c \},
\bigg \}
\end{align*}
\begin{align*}
S_{\theta^\star} =
\bigg\{
& \{\mu_1,\ldots,\mu_m\} =
\left\{
(\mu_{jk,J},\mu_{jk,J^c}), \:
j=1,\ldots,K, \, k \in \mathcal{Y}_J \right\},
\\
&
\mu_{jk,J^c} \in U_{j|k}, \;\; \mu_{jk,i} \in \left[ k_i - \frac{1}{4N_i}, k_i + \frac{1}{4N_i}\right] , \, i \in J,
\\
& \sigma_i^2 \in \left( 0, \sigma_{i}^{\star2} \right), \, i \in J,
\\
&\sigma_i^2 \in \left(\sigma_{i}^{\star2}(1+\sigma_n^{2\beta})^{-1}, \sigma_{i}^{\star2}\right), \, i \in J^c,
\\
& (\alpha_1,\ldots,\alpha_m) =
\{ \alpha_{jk}, \, j=1,\ldots,K, \, k\in\mathcal{Y}_J \} \in \Delta^{m - 1}, \\
&
\sum_{r=1}^{m}
|\alpha_{r}-\alpha_{r}^\star| \leq 2 \sigma_n^{2\beta},\:\:\:\:\:
\min_{j\leq K, k\in\mathcal{Y}_J} \alpha_{jk} \geq \frac{\sigma_n^{2\beta + d_{J^c}}}{2m^2}
\bigg\}.
\end{align*}
The rest of the proof of the Kullback-Leibler thickness condition follows the general argument developed for mixture models in \cite{GhosalVandervaart:07} and \cite{ShenTokdarGhosal2013} among others.
First, we will show that for $m=N_JK$ and $\theta\in S_{\theta^\star}$, the Hellinger distance $d_H^2(p_0(\cdot,\cdot), p(\cdot,\cdot | \theta, m ))$ can be bounded by
$\sigma_n^{2\beta}$ up to a multiplicative constant. Second, we construct bounds on the ratios $p(\cdot,\cdot | \theta, m )/p_0(\cdot,\cdot)$ and combine them with the bound on the Hellinger distance using
Lemma \ref{lm:dH_KL}. Finally, we will show that the prior puts sufficient probability on $m=N_JK$ and $S_{\theta^\star}$.
For $f_{|J}^\star$ defined in \eqref{eq:f_J_def}, let us define
\[
p_{|J}^\star(y_I,x |y_J)=\int_{A_{y_I}} f_{|J}^\star(\tilde{y}_I,x |y_J) d\tilde{y}_I.
\]
For $m=N_JK$ and $\theta\in S_{\theta^\star}$, we can bound the Hellinger distance between the DGP and the model as follows,
\begin{align*}
&d_H^2(p_0(\cdot,\cdot), p(\cdot,\cdot | \theta, m )) =d_H^2(p_{0|J}(\cdot|\cdot)\pi_0(\cdot), p(\cdot,\cdot | \theta, m )) \notag \\
&\leq
d_H^2(p_{0|J}(\cdot|\cdot)\pi_{0J}(\cdot), p_{|J}^\star(\cdot|\cdot)\pi_{0J}(\cdot)) +
d_H^2(p_{|J}^\star(\cdot|\cdot)\pi_{0J}(\cdot), p(\cdot,\cdot | \theta, m )).
\end{align*}
It follows from \eqref{eq:f0upsbeta} and
Lemma \ref{lm:p_f_distance_bds} linking distances between probability mass functions and corresponding latent variable densities that the first term on the right hand side of this inequality is bounded by
$(C_2)^2\sigma_n^{2\beta}$.
Combining this result with the bound on $d_H^2(p_{|J}^\star(\cdot|\cdot)\pi_{0J}(\cdot), p(\cdot,\cdot | \theta, m ))$ from Lemma \ref{lm:H_dist_model_pi0_model} we obtain
\begin{align}
\label{eq:H2_bound_dgp_model}
d_H^2(p_0(\cdot,\cdot), p(\cdot,\cdot | \theta, m )) \lesssim \sigma_n^{2\beta}.
\end{align}
Next, for $\theta \in S_{\theta^\star}$ and $m=N_JK$, let us consider lower bounds on the ratio $p(y_J,y_I,{x} | \theta, m)/p_{0}(y_J,y_I,x)$.
In Lemma \ref{lm:Bound_Model_truef_ratio} in the Appendix we show that
lower bounds on the ratio
$f_J(y_J,\tilde{x} | \theta, m)/f_{0|J}(\tilde{x} | y_J)\pi_{0}(y_J)$
imply the following bounds for all sufficiently large $n$:
for any ${x} \in {\mathcal{{X}}}$ with $\left\Vert{x}\right\Vert \leq a_{\sigma_n}$,
\begin{align}
\label{eq:define_lambda_n_MAIN}
\frac{p(y_J,y_I,{x} | \theta, m)}{p_{0}( y_J,y_I,x)}
\geq
C_3 \frac{\sigma_n^{2\beta }}{2m^2}
\equiv \lambda_n,
\end{align}
for some constant $C_3>0$; and for any ${x} \in {\mathcal{{X}}}$ with $\left\Vert{x}\right\Vert > a_{\sigma_n}$,
\begin{align}
\label{eq:boundratio_forlarge_x_MAIN}
\frac{p(y_J,y_I,{x} | \theta, m)}{p_{0}( y_J,y_I,x)}
\geq
\exp\left\{ - \frac{8||{x}||^2}{\underline{\sigma}_{n}^2} - C_4 \log n \right\},
\end{align}
for some constant $C_4>0$.
Consider all sufficiently large $n$ such that
$\lambda_n < e^{-1}$
and \eqref{eq:define_lambda_n_MAIN}
and \eqref{eq:boundratio_forlarge_x_MAIN} hold.
Then, for any $\theta \in S_{\theta^\star}$,
\begin{align}
\label{eq:exp_log_ratio2}
&
\sum_{y\in\mathcal{Y}}
\int_{{\mathcal{X}}}
\bigg( \log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)}\bigg)^2
\textbf{1} \left \{ \frac{p(y_J,y_I,{x}| \theta, m)}{p_0(y_J,y_I,x)} < \lambda_n \right\}
p_0(y_J,y_I,x) d{x} \notag \\
&=
\sum_{y\in\mathcal{Y}}
\int_{\tilde{\mathcal{X}}}
\bigg( \log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)}\bigg)^2
\textbf{1} \left \{ \frac{p(y_J,y_I,{x}| \theta, m)}{p_0(y_J,y_I,x)} < \lambda_n
\right\}
\textbf{1} \left \{\tilde{y}_I\in A_{y_I} \right\}
f_{0J}(y_J,\tilde{x}) d\tilde{x} \notag \\
& =
\sum_{y\in\mathcal{Y}}
\int_{\tilde{\mathcal{X}}}
\bigg( \log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)}\bigg)^2
\textbf{1} \left \{ \frac{p(y_J,y_I,{x}| \theta, m)}{p_0(y_J,y_I,x)} < \lambda_n , ||{x}||> a_{\sigma_n}
,\tilde{y}_I\in A_{y_I}
\right\}
f_{0J}(y_J,\tilde{x}) d\tilde{x} \notag\\
& \leq
\sum_{y\in\mathcal{Y}}
\int_{\left\{\tilde{x}:||{x}||>a_{\sigma_n}\right\}}
\bigg( \log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)}\bigg)^2
\textbf{1} \left \{\tilde{y}_I\in A_{y_I} \right\}
f_{0J}(y_J,\tilde{x}) d\tilde{x}
\notag
\\
& \leq
\sum_{y\in\mathcal{Y}}
\int_{\left\{\tilde{x}:||{x}||>a_{\sigma_n}\right\}} \left [
\frac{128}{\underline{\sigma}_n^4} ||{x}||^4 + 2(C_4 \log n)^2\right]f_{0|J}(\tilde{x}|y_J)
\textbf{1} \left \{\tilde{y}_I\in A_{y_I} \right\}
d\tilde{x}
\pi_{0J}(y_J)\notag \\
&
\leq
\sum_{y_J\in\mathcal{Y}_J}
\int_{\left\{\tilde{x}:||\tilde{x}||>a_{\sigma_n}\right\}}
\left [
\frac{128}{\underline{\sigma}_n^4} ||\tilde{x}||^4 + 2(C_4 \log n)^2\right]
f_{0|J}(\tilde{x}|y_J) d\tilde{x}
\pi_{0J}(y_J)
\notag \\
&\leq
\frac{128}{\underline{\sigma}_n^4}
\sum_{y_J\in\mathcal{Y}_J} E_{0|y_J}\left(\left\Vert\tilde{X}\right\Vert^8\right)^{1/2} \left(F_{0|y_J}\left(\left\Vert\tilde{X}\right\Vert > a_{\sigma_n}\right)\right)^{1/2} \pi_{0J}(y_J)
+ 2(C_4 \log n)^2 B_0 \sigma_n^{4\beta+2\varepsilon}\underline{\sigma}_{n} ^8
\notag
\\
& \leq C_5 \sigma_n^{2\beta + \varepsilon}
\end{align}
for some constant $C_{5}>0$ and
all sufficiently large $n$,
where the last inequality holds by the tail condition in \eqref{eq:tail_cond2}, \eqref{eq:tail_prob_bound}, and $(\log n)^2\sigma_n^{2\beta+\varepsilon}\underline{\sigma}_{n}^8 \rightarrow 0$.
Furthermore, as $\lambda_n< e^{-1}$,
\begin{align*}
& \log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)}
\textbf{1} \left \{ \frac{p(y_J,y_I,{x}| \theta, m)}{p_0(y_J,y_I,x)} < \lambda_n \right\}
\\
&\leq
\bigg( \log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)}\bigg)^2
\textbf{1} \left \{ \frac{p(y_J,y_I,{x}| \theta, m)}{p_0(y_J,y_I,x)} < \lambda_n \right\}
\end{align*}
and, therefore,
\begin{align}
\label{eq:exp_log_ratio}
\sum_{y\in\mathcal{Y}}
\int_{{\mathcal{X}}}
\log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)}
\textbf{1} \left \{ \frac{p(y_J,y_I,{x}| \theta, m)}{p_0(y_J,y_I,x)} < \lambda_n \right\}
p_0(y_J,y_I,x) d{x}
\leq C_{5} \sigma_n^{2\beta + \varepsilon}.
\end{align}
Inequalities
\eqref{eq:H2_bound_dgp_model}, \eqref{eq:exp_log_ratio2}, and \eqref{eq:exp_log_ratio}
combined with Lemma \ref{lm:dH_KL} imply
\[
E_0 \left( \log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)} \right)\leq A\tilde{\epsilon}_n^2, \;
E_0 \left( \left[ \log \frac{p_0(y_J,y_I,x)}{p(y_J,y_I,{x}| \theta, m)} \right]^2 \right) \leq A\tilde{\epsilon}_n^2
\]
for any $\theta \in S_{\theta^\star}$, $m=N_JK$, and some positive constant $A$ (details are provided in Lemma \ref{lm:gKL_inequality} in the Appendix).
By Lemma \ref{lm:prior_bound} in the Appendix
for all sufficiently large $n$, $s = 1 + 1/\beta + 1/\tau$, and some $C_6>0$,
\begin{eqnarray*}
\Pi( \mathcal{K}(p_0, \tilde{\epsilon}_n)) \geq \Pi(m=N_JK, \theta \in S_{\theta^\star} ) \geq
\exp \left[-C_6 N_J\tilde{\epsilon}_n^{-d_{J^c}/\beta} \{\log (n)\}^{d_{J^c}s + \max\{\tau_1,1,\tau_2/\tau\} }\right] .
\end{eqnarray*}
The last expression of the above display is bounded below by $\exp\{-C n \tilde{\epsilon}_n^2 \}$ for any $C>0$,
$\tilde{\epsilon}_n = \left[ \frac{N_J}{n}\right]^{\beta/(2\beta + d_{J^c})} (\log n)^{t_J}$, any $t_J > (d_{J^c}s + \max\{\tau_1,1,\tau_2/\tau\}) / (2 + d_{J^c}/\beta)$, and all sufficiently large $n$. Since the inequality in the definition of $t_J$ is strict, the claim of the theorem follows.
When $J = \emptyset$ and $N_J=1$, the preceding argument delivers the claim of the theorem if
we add an artificial discrete coordinate with only one possible value to the vector of observables.
\subsubsection{Proof of Theorem \ref{th:prior_thickness} for $J^c= \emptyset$}
In this case, the proof from the previous subsection can be simplified as follows.
For $m=N_J$ and for any $\beta>0$ we define $\theta^\star$ and $S_{\theta^\star}$ as
\begin{align*}
\theta^\star= \bigg\{
&\{\mu_1^\star,\ldots,\mu_m^\star\} = \left\{
k, \; k\in\mathcal{Y}_J \right\},
\\
&\{\alpha_1^\star,\ldots,\alpha_m^\star\} = \left\{
\alpha_{k}^\star, \; k\in\mathcal{Y}_J \right\}
= \left\{
\pi_{0}(k)
\right\}_{k\in\mathcal{Y}_J},
\\
&\sigma^{\star2}=\{\sigma_i^{\star2} = \frac{1}{64 N_i^2 \beta \log (1/\sigma_n)},\; i \in J\}
\bigg\},
\end{align*}
\begin{align*}
S_{\theta^\star} =
&\bigg\{
\{\mu_1,\ldots,\mu_m\} = \left\{
\mu_{k}, \; k\in\mathcal{Y}_J
\right\}, \;
\mu_{k,i} \in \left[ k_i - \frac{1}{4N_i}, k_i + \frac{1}{4N_i}\right], \; i=1,\ldots,d_J,\\
&\sigma= \{\sigma_i \in (0, \sigma_i^\star), \; i \in J\},\\
& \{\alpha_{j}, \; j=1,\ldots,m\} = \{\alpha_{k}, \; k \in \mathcal{Y}_J\} \in \Delta^{m - 1},\\
& \sum_{k\in\mathcal{Y}_J}
|\alpha_{k}-\alpha_{k}^\star| \leq 2 \sigma_n^{2\beta}, \;\;\;\;\;
\min_{k\in\mathcal{Y}_J} \alpha_{k} \geq \frac{\sigma_n^{2\beta }}{2m^2}
\bigg\}.
\end{align*}
For $m=N_J$ and $\theta\in S_{\theta^\star}$,
a simplification of the proof of Lemma \ref{lm:H_dist_model_pi0_model} delivers
\begin{align*}
d_H^2(p_0(\cdot), p(\cdot | \theta, m ))
\leq
2 \max_{k\in\mathcal{Y}_J} \int_{A^c_{k}}\phi(\tilde{y}_J;\mu_{k},\sigma) d\tilde{y}_J
+
\sum_{k\in\mathcal{Y}_J}
\left|
\alpha_{k}^\star
-
\alpha_{k}
\right| \notag \lesssim \sigma_n^{2\beta}. \notag
\end{align*}
A simplification of derivations in Lemma \ref{lm:Bound_Model_truef_ratio} show that for all $y_J\in\mathcal{Y}_J$
\begin{align*}
\frac{p(y_J | \theta, m)}{p_{0}( y_J)}
\geq
\frac{1}{2} \frac{\sigma_n^{2\beta }}{2m^2}
\equiv \lambda_n. \notag
\end{align*}
Then,
for any $\theta \in S_{\theta^\star}$
\begin{align}
\sum_{y_J\in\mathcal{Y}_J}
\bigg( \log \frac{p_0(y_J)}{p(y_J| \theta, m)}\bigg)^2
\textbf{1} \left \{ \frac{p(y_J| \theta, m)}{p_0(y_J)} < \lambda_n \right\}
p_0(y_J) &=0 \notag \\
\sum_{y_J\in\mathcal{Y}_J}
\bigg( \log \frac{p_0(y_J)}{p(y_J| \theta, m)}\bigg)
\textbf{1} \left \{ \frac{p(y_J| \theta, m)}{p_0(y_J)} < \lambda_n \right\}
p_0(y_J) &=0 \notag \\
\end{align}
as $\frac{p(y_J| \theta, m)}{p_0(y_J)} \geq \lambda_n$ for all $y_J\in\mathcal{Y}_J$.
As $\lambda_n\rightarrow 0$, by Lemma \ref{lm:dH_KL} for $\lambda_n<\lambda_0$,
both
$E_0 ( \log \frac{p_0(y_J)}{p(y_J| \theta, m)} )$ and $E_0 ( [ \log \frac{p_0(y_J)}{p(y_J| \theta, m)} ]^2)$
are bounded by $C_{7} \log (1/\lambda_n)^2\sigma_n^{2\beta} \leq A\tilde{\epsilon}_n^2$ for some constant $A$.
By the simplification of Lemma \ref{lm:prior_bound} for this particular case
for all sufficiently large $n$ and some $C_{8}>0$,
\begin{eqnarray*}
\Pi( \mathcal{K}(p_0, \tilde{\epsilon}_n)) \geq \Pi(m=N_J, \theta \in S_{\theta^\star} ) \geq
\exp \left[-C_{8}N_J \{\log (n)\}^{ \max\{\tau_1,1\} }\right] .
\end{eqnarray*}
The last expression of the above display is bounded below by $\exp\{-C n \tilde{\epsilon}_n^2 \}$ for any $C>0$,
$\tilde{\epsilon}_n = \left[ \frac{N_J}{n}\right]^{1/2} (\log n)^{t_J}$, any $t_J > \max\{\tau_1,1\} /2 $, and all sufficiently large $n$. Since the inequality in the definition of $t_J$ is strict, the claim of the theorem follows.
\section{Future Work}
\label{sec:conclusion}
It seems feasible to extend the results of this paper to conditional density estimation by covariate dependent mixtures
along the lines of \cite{norets_pati_2017}. We leave this to future work.
\bibliographystyle{../../../latex_common/ecta}
\bibliography{../../../latex_common/allreferences}