Semiparametrics via parametrics and contiguity

Semiparametrics via parametrics and contiguity

\thispagestyle{empty}

\begingroup \footnote{Adam Lee would like to thank participants at various conferences for their helpful comments. Emil A. Stoltenberg would like to the Department of Data Science at BI Norwegian Business School where the greater part of this work was carried out. Per A. Mykland would like to thank the (U.S.) National Science Foundation for financial support under grant DMS-2413952. } \addtocounter{footnote}{-1} \endgroup

Introduction

Drawing valid inference about the parametric component of a semiparametric model is often a challenging task. To deal with some of these challenges, we explore a simplifying strategy (inspired by research in high-frequency econometrics, see mykland2009inference) in which the semiparametric problem is made (almost) fully parametric. The idea is to pretend that the data stem from a parametric distribution, allowing one to derive estimators and study their properties using standard parametric techniques. We let the these parametric distributions grow in such a way that they are mutually contiguous with respect to the full semiparametric distribution; and, finally, we use contiguity and Le Cam{'}s third lemma to switch the analysis back to the semiparametric model in the limit.

The idea of approximating semiparametric models with growing parametric models is not new, as there is a well developed literature on sieves (grenander1981abstract; see shen1997methods and chenshen98 for seminal contributions to this literature, and chen07survey for an excellent review). Our framework parallels the conventional sieve literature in its use of growing parametric models, but differs by carrying out the analysis {\it under} these parametric models themselves, i.e., assuming that the data stem from a member of such a parametric model. As we shall see, combining this analysis under parametric models with the use of contiguity to return to the semiparametric setting makes our approach -- which we call contiguous sieves -- markedly different from that of conventional sieves (i.e., growing parametric models, analysed under the same semiparametric distribution at all steps). In the examples we have considered this approach allows us to establish the asymptotic normality and efficiency of semiparametric maximum likelihood estimators under conditions which are relatively user-friendly and straightforward to verify, essentially because they require just a little more than standard parametric likelihood theory. In particular, we are able to show asymptotic efficiency of semiparametric maximum likelihood estimators in a broad class of models without imposing any empirical process type conditions. Working with contiguous parametric submodels has several advantages. There is no ambiguity in defining a likelihood function (assuming the model is dominated), and based on this likelihood function, estimators can be derived or defined without difficulty. Carrying out the analysis {\it as if} these submodels were the models generating the data, enables us to work as though we have a correctly specified parametric model at each step; consequently, the required regularity conditions for these estimators to be asymptotically normal and efficient can be verified similarly as in classical parametric maximum likelihood theory. Finally, since the parametric distributions are constructed such that they are mutually contiguous to the true semiparametric distribution, we can transfer the analysis back to the semiparametric distribution in the limit, thus obtaining asymptotic results under the true distribution.

The article proceeds as follows. In Section (ref) we outline the general setup we work in and motivate our approach. Section (ref) introduces the assumptions we work under. Sections (ref) & (ref) contain our theoretical results. In Section (ref) we provide a detailed analysis of two examples to demonstrate the application of our approach: the partially linear (regression) model and the Cox model. As we exemplify with the Cox model, parametric approximations might also be used as a purely theoretical tool (and not for estimation); this allows for efficiency proofs that we believe to be simpler than those currently available. Section (ref) concludes.

General setup and parametric theory

Let $X_1, \ldots, X_n$ be i.i.d. replicates of $X$, where $X$ has distribution $P_0 \coloneqq P_{\theta_0, \eta_0}$ belonging to a semiparametric family of distributions $\mathcal{P}= \{P_{\theta, \eta}: \theta\in \Theta, \eta\in \mathcal{H}\}$, where $\Theta\subset \mathbb{R}^p$ for some $p \geq 1$, and $\mathcal{H}$ is a space of infinite dimension. We suppose that $\mathcal{P}$ is dominated by some $\sigma$-finite measure $\mu$, and write $p_{\theta,\eta}$ for the densities. The problem is to do inference on $\theta$ in the presence of the infinite dimensional nuisance parameter $\eta$. The simplifying strategy discussed in the introduction involves the construction of certain parametric submodels: For each $m \geq 1$, let $\mathcal{H}_m$ be a family of parametric functions $\eta_{\gamma}$ indexed by a parameter $\gamma \in \Gamma_m \subset \mathbb{R}^{k_m}$. Typically these shall be such that $\mathcal{H}_m \subset \mathcal{H}_{m + 1}$ for all $m$ with $\cup_{m \geq 1} \mathcal{H}_m$ dense in $\mathcal{H}$ in an appropriate topology. Let $T_m \colon \mathcal{H}_m \to \Gamma_m$ be an isomorphism between the function space $\mathcal{H}_m$ and $\Gamma^{m}$ so that $T_m \eta_{\gamma} = \gamma \in \Gamma_m$ and $T_{m}^{-1}\gamma = \eta_\gamma \in \mathcal{H}_m$. Then for each $m \geq 1$,

is a $p + k_m$ dimensional parametric model, and $P_{\theta,T_m^{-1}\gamma}$ has density $p_{\theta, T_m^{-1}\gamma}$ with respect to $\mu$. As discussed above, the idea is to carry out the analysis {\it as if} $X_1,\ldots,X_n$ was an i.i.d. sample from a member of $\mathcal{P}_m$, let $m$ increase with the sample size $n$, and use contiguity to switch the analysis back to $P_{\theta_0,\eta_0}$ in the limit. The natural choice for a parametric approximation is to suppose that the sample stems from $P_{m} \coloneqq P_{\theta_0, T_m^{-1}\gamma_{0}}$, where $\theta_0$ equals the true value in the big model $P_0$, and $(\gamma_{0})_{m \geq 1} = (\gamma_{0,m})_{m \geq 1}$ is a sequence of growing vectors so that $\eta_{\gamma_{0}}$ approaches the true $\eta_0$ as $m$ tends to infinity. To declutter the notation, we avoid indexing ($\gamma$ and) $\gamma_0$ by $m$, as the size of these parameter vectors should be clear from the context. The densities of $P_0$ and $P_m$ with respect to $\mu$ are denoted $p_0$ and $p_m$, respectively, with similar subscripts for the expectations, $\mathbb{E}_0\, g(X) = \int g(x)\,{\rm d} P_0(x)$ and $\mathbb{E}_m\, g(X) = \int g(x)\,{\rm d} P_m(x)$. The product measure arising from an i.i.d. sample of size $n$ is indicated by a superscript $n$, e.g.: $P_{\theta,\eta}^n = P_{\theta,\eta} \times \cdots \times P_{\theta,\eta}$. The score functions with respect to $\theta$ and $\gamma$ under $\mathcal{P}_m$ are

When evaluated in $(\theta_0,\gamma_0)$, i.e., the `true values' under $\mathcal{P}_m$, we write $\dot{\ell}_m = \dot{\ell}_{\theta_0,T_m^{-1}\gamma_0}$ and $\dot{v}_m = \dot{v}_{\theta_0,T_m^{-1}\gamma_0}$. The log-likelihood function under $\mathcal{P}_m$ is $(\theta,\gamma) \mapsto \sum_{i=1}^n \log p_{\theta,T_m^{-1}\gamma}(X_i)$ for $(\theta,\gamma) \in \Theta \times \Gamma_m$, and the corresponding maximum likelihood estimator (MLE) for $\theta$ is $\widehat{\theta}_{m,n}$. Let $i_m$ be the Fisher information matrix under $P_m$ and block partition it as follows

The standard maximum likelihood theory for inference on $\theta$ in the presence of a finite dimensional (fixed $m$) nuisance parameter $\gamma \in \Gamma_m$ goes as follows: Under regularity conditions (see, e.g., vandervaart1998asymptotic), maximum likelihood estimators are asymptotically linear in the parametric efficient influence function, that is, for fixed $m$,

as $n$ tends to infinity, where $\tilde{\ell}_m$ is the efficient score function and $J_m$ its variance:

with $\Pi_m$ the orthogonal projection onto the linear span of $\{\dot{v}_{m, j}: j=1, \ldots, k_m\}$ in $L_2(P_m)$.

Provided the model $\mathcal{P}_m$ is differentiable in quadratic mean at $(\theta_0,\gamma_0)$ and the efficient information matrix $J_m$ is nonsingular, then (ref) is equivalent to $\widehat\theta_{m, n}$ being the best regular estimator vandervaart1998asymptotic. This means that as $n$ tends to infinity, but $m$ remains fixed, $\sqrt{n}(\widehat\theta_{n, m} - \theta_0)$ converges in distribution under $P_{m}^n$ to a mean zero normal distribution with variance matrix $J_m^{-1}$, this being the smallest possible asymptotic variance matrix of any regular estimator (see Section (ref) for a formal definition).

To illustrate our general strategy as well as the definitions and the notation introduced above, consider the partially linear regression model where $X = (W,Y,Z)$ for a real valued outcome $Y$ that given covariates $(W,Z) = (w,z)$ follows the regression model

Here $\varepsilon$ is a noise term independent of $(W,Z)$, $\eta$ is an infinite dimensional nuisance parameter, and $\theta \in \mathbb{R}$ is the parameter on which we seek to make inference. If $\varepsilon$ is assumed to be mean zero normal with variance $\sigma^2$ and the covariates have a density, then the observation $(W,Y,Z)$ has a density. This density, however, cannot be used to define a maximum likelihood estimator for $(\theta,\eta)$, because the maximiser for $\eta$ will just interpolate the data (see andersen1993statistical for a discussion of these difficulties). To overcome these issues we instead pretend that $Y$ given $(W,Z) = (w,z)$ stems from the parametric model

where $\beta_m = (\beta_{m,1},\ldots,\beta_{m,k_m})^{{\rm t}}$ is a collection of orthonormal (or other basis) functions, $\gamma = (\gamma_1,\ldots,\gamma_{k_m})^{{\rm t}}$ is a Euclidean parameter vector, and $\varepsilon$ and $(W,Z)$ have the same distribution as the similarly denoted random variables above. Now, maximum likelihood estimation is a least squares problem, and we readily obtain a maximum likelihood estimator for $(\theta,\gamma)$, say $(\widehat{\theta}_{m,n},\widehat{\gamma}_{m,n})$. Assuming that the data in fact stem from the parametric model in (ref), we get from standard parametric likelihood theory that $\sqrt{n}(\widehat{\theta}_{m,n} - \theta)$ converges in distribution to a mean zero normal with variance $J_m^{-1}$, where $J_m = \sigma^{-2}\big(\mathbb{E}\, W^2 - \sum_{j=1}^{k_m} (\mathbb{E}\, \{\beta_{m,j}(Z)W\})^2 \big)$, and that this is the efficient information under the model in (ref).

For parametric inference ($m$ fixed) the conclusion above is in many ways the end of the maximum likelihood story: $\sqrt{n}(\widehat{\theta}_{m,n} - \theta_0) \rightsquigarrow {\rm N}(0,J_m^{-1})$, and $J_m^{-1}$ is the smallest possible variance (of a regular estimator). Part of the motivation for the present paper, however, is the observation that if we let $m$ tend to infinity, it is often the case that $J_{m} \to J$ for some nonsingular matrix $J$. It is then tempting to conjecture both that (i) $\sqrt{n}(\widehat\theta_{m_n, n} - \theta_0) \rightsquigarrow {\rm N}(0,J^{-1})$ under $P_{m_n}^n$ for some subsequence $(m_n)_{n \geq 1}$ tending to infinity with $n$; and (ii) that $J$, being the limit of a sequence of efficient information matrices, must be semiparametrically efficient under $\mathcal{P}$. Of course, we desire $\sqrt{n}(\widehat\theta_{m_n, n} - \theta_0) \rightsquigarrow {\rm N}(0,J^{-1})$ under $P_0^n$ rather than $P_{m_n}^n$; we shall subsequently restrict the class of approximating models such that we can make this change of measure.

A case in point is the partially linear model where it follows from Parseval{'}s identity that with $J_m$ the efficient information under (ref)

where the limit is positive (provided $W$ is not a.s. equal to $\mathbb{E}\,(W \,|\, Z)$). Here we also recognise the limit as the efficient information under the semiparametric model in (ref) (see, e.g., BKRW98), demonstrating one case in which the limit of (parametric) efficient information matrices is efficient.

Assumptions for contiguity and efficiency

For our subsequent efficiency results to make sense, we need to impose some structure on the semiparametric model. In Section (ref) we outline this structure, introduce some notation, and define what we mean by asymptotic efficiency. In Section (ref) the conditions we impose on the parametric approximations are presented, along with a few lemmata easing the verification of these.

The true semiparametric model and efficiency

We concentrate on smooth models for i.i.d. data, as in the classical parametric theory. This is made precise by imposing a differentiability in quadratic mean (DQM) condition on the semiparametric model $\mathcal{P}$ lecam1986, BKRW98. Let $B$ be a linear space. We will consider measures $P_{\theta_0 + \tau/\sqrt{n}, \eta_n(b)}$ for $h = (\tau, b)\in \mathbb{R}^p \times B$ where $\eta_n(b)\to \eta_0$ as $n$ tends to infinity, and $\eta_n(0) = \eta_0$.

It follows from the convergence in Assumption (ref) that $Ah\in L_2(P_0)$ and $\int Ah \,{\rm d} P_0 = 0$ vandervaart1998asymptotic. Since $A$ is assumed to be linear we can split out the contributions of the parametric parameter of interest from the infinite dimensional nuisance as $Ah = \tau^{\rm t} \dot{\ell} + Db$, where $\dot{\ell}$ is the ordinary score function for $\theta$ in a model where the nuisance $\eta$ is fixed; while $D \colon B \to \mathbb{R}$ is a linear operator, and $Db$ has the interpretation of a score function for $\eta$ with $\theta$ fixed.

An estimator $\widehat{\theta}_n$ of $\theta_0$ is said to be regular if $\sqrt{n}(\widehat{\theta}_n - \theta_0 - \tau/\sqrt{n}) \rightsquigarrow L$ under $P_{\theta_0+\tau/\sqrt{n}, \eta_n(b)}$, for some law $L$ and each $(\tau, b)\in \mathbb{R}^p\times B$. Requiring regularity excludes superefficient estimators such as the Hodges-Le Cam estimator (see e.g. vandervaart1998asymptotic). The efficiency bound for regular estimators for estimation of $\theta_0$ is determined by the efficient score,

where $\Pi$ denotes the orthogonal projection onto the closure of $\{Db: b\in B\}$ in $L_2(P_0)$. The efficiency bound is the inverse (provided it exists) of the variance of $\tilde{\ell}$,

More precisely, provided $J$ is nonsingular, by the H{\'a}jek--Le Cam convolution theorem, the limiting distribution of any regular sequence of estimators can be represented by the convolution of a ${\rm N}(0, J^{-1})$ with some probability distribution (see e.g. BKRW98; vandervaart1998asymptotic). As such, any regular estimator whose limiting distribution is $L = {\rm N}(0, J^{-1})$ is called best regular; this is what we refer to as asymptotic efficiency.

The discussion immediately above required $J$ to be nonsingular. In fact, this is necessary for the existence of regular estimators chamberlain86asymptotic and therefore we assume this throughout the rest of the paper.

The parametric approximations

We now introduce our assumption on the parametric approximations $P_m = P_{\theta_0,T_m^{-1}\gamma_0}$ to $P_0 = P_{\theta_0,\eta_0}$. In particular, we assume that $P_0$ can be approximated, in an appropriate sense, by a sequence of contiguous alternatives, similar to those in Assumption (ref). Since $P_0$ is unknown, checking Assumptions (ref) and (ref) below entails in practice checking it for any member of $\mathcal{P}$. Formally, however, these assumptions are required to hold only for the specific member $P_0$ of $\mathcal{P}$ that generated the data.

In our applications, we will seek that $g=0$, but the orthogonality $\mathbb{E}_0\,g(X) \tilde{\ell}(X) = 0$ is all that is actually needed for our results. The log-likelihood ratio expansion in (ref) is equivalent to the densities $p_{m_n} = {\rm d} P_{m_n}/{\rm d} \mu$ satisfying the DQM type condition

for $g$ such that $\mathbb{E}_0\,g(X) \tilde{\ell}(X) = 0$, that is, for the function $g$ appearing in (ref). See for example strasser1985mathematical or lecam1986 for the equivalence of (ref) and (ref). Thus, to check Assumption (ref) it suffices to show either (ref) or (ref).

The log-likelihood ratio expansion in (ref) is key to our results. In particular, since the data are assumed i.i.d., Assumption (ref) and the central limit theorem yield

Since $\exp(Z)$ is positive and $\mathbb{E}\exp(Z) = 1$, we get from Le Cam{'}s first lemma that $P_0^n$ and $P_{m_n}^n$ are mutually contiguous (see vandervaart1988large). Provided $\sqrt{n}(\widehat{\theta}_{m_n,n} - \theta_0)$ converges jointly with $\log ({\rm d} P_0^n /{\rm d} P_{m_n}^n)$ to a Gaussian limit under $P_{m_n}^n$, the asymptotic distribution of $\sqrt{n}(\widehat{\theta}_{m_n,n} - \theta_0)$ under $P_{0}^n$ can be recovered by Le Cam{'}s third lemma lecam1986. As such, Assumption (ref) restricts the possible change in the limiting distribution of $\sqrt{n}(\widehat{\theta}_{m_n,n} - \theta_0)$ resulting from a change of measure from the parametric $P_{m_n}$ back to the semiparametric $P_0$. In particular, provided the limiting distribution of $\sqrt{n}(\widehat{\theta}_{m_n,n} - \theta_0)$ under $P_{m_n}^n$ is orthogonal to $g$ (i.e. $\lim_{n\to\infty} {\rm Cov}(\sqrt{n}(\widehat{\theta}_{m_n,n} - \theta_0),n^{-1/2}\sum_{i=1}^n g(X_i) ) = 0$), changing the measure from $P_{m_n}$ to $P_0$ does not affect the limiting distribution.

In some examples, approximating models satisfying Assumption (ref) can be derived directly from the submodels used to establish Assumption (ref), given the similarity between the DQM required in Assumption (ref) and equation (ref). We view this as a virtue of our contiguous sieve framework which closely links the submodels used to define semiparametric efficiency with those used to estimate $\theta$.

In addition to the contiguity condition in Assumption (ref), we require that the efficient scores in the parametric submodels approximate the efficient score of the full semiparametric model in a statistically relevant sense. As we will take our limits along the subsequence $(m_n)_{n\geq 1}$ of Assumption (ref), it is sufficient that this approximation holds along this subsequence. This is convenient as Assumption (ref) implies that $P_{m_n} \to P_0$ in total variation (via (ref)), which can help to simplify the demonstration of (ref) (see Lemma (ref) below).

Explicitly performing the orthogonal projection to compute $\tilde{\ell}$ can, in many models, be quite challenging. Fortunately, one may verify Assumption (ref) without explicitly performing this projection, as the following lemma demonstrates. Here the space $B$ and the linear operator $D$ are as defined in connection with Assumption (ref).

We now present two lemmas which can help with the verification of (ref), or of (ref) and (ref). Their straightforward proofs are deferred to Appendix (ref).

That $f_{m_n}^2$ is uniformly $P_{m_n}$-integrable means that $\lim_{K\to\infty}\sup_n \int_{|f_{m_n}| \geq K} f_{m_n}^2\,{\rm d} P_{m_n} = 0$. An alternative approach which circumvents the requirement to establish uniform square integrability directly can be based on a lemma originally due to riesz1928convergence (cf. vandervaart1998asymptotic). This lemma also connects Assumption (ref) with the observation that in certain models the sequence of parametric efficient information matrices has the semiparametric efficient information matrix as a limit, as discussed in Section (ref).

If we use either of these lemmas to verify (ref) directly, we are required to first find the efficient score under the semiparametric model $\mathcal{P}$, a task which involves performing sometimes complicated projections. Lemma (ref), on the other hand, does not involve any projections. The following corollary permits us to find the efficient semiparametric score $\tilde{\ell}$ as a limit.

Asymptotic efficiency

We now present the main result of the paper. In order to approximate the semiparametric model, we let $m$ increase with $n$. This means that, for our purposes, the property corresponding to the asymptotic linearity in the parametric efficient score function exhibited in (ref), is

Note that property (ref) is not implied by (ref), as (ref) only requires that the remainder $\sqrt{n}(\widehat\theta_{m,n} - \theta_0) -n^{-1/2}\sum_{i=1}^n J_{m}^{-1}\tilde{\ell}_{m}(X_i)$ is $o_{P_{m}^n}(1)$ as $n \to \infty$, for fixed $m$. Verification of (ref) depends on the definition of the estimator $\widehat\theta_{m_n,n}$. In Section (ref) we use a profile likelihood technique to provide sufficient conditions for (ref) to hold for a sequence of maximum likelihood estimators $\widehat{\theta}_{m_n,n}$ in growing contiguous parametric submodels.

Combined with Assumptions (ref)--(ref), the linearity of $\sqrt{n}(\widehat\theta_{m_n,n} - \theta_0)$ in the influence function displayed in (ref) implies the asymptotic efficiency of the estimator $\widehat\theta_{m_n, n}$. The essential idea for proving this is outlined in the following heuristic argument: If Assumption (ref) holds, then $J_{m_n}^{-1}\tilde{\ell}_{m_n}$ in (ref) may be replaced by $J^{-1}\tilde{\ell}$, thus (ref) becomes $\sqrt{n}(\widehat\theta_{m_n,n} - \theta_0) = n^{-1/2}\sum_{i=1}^n J^{-1}\tilde{\ell}(X_i)+o_{P_{m_n}^n}(1)$. Combined with Assumption (ref), this asymptotic linearity ensures that $\sqrt{n}(\widehat\theta_{m_n,n} - \theta_0)$ converges jointly with $ \log ({\rm d} P_0^n /{\rm d} P_{m_n}^n)$ under $P_{m_n}^n$, and hence, by Le Cam's third lemma, one can change the measure from $P_{m_n}$ back to $P_0$ at the cost of adding a bias term of $J^{-1}\mathbb{E}_0\,\tilde{\ell}(X)g(X)$ to the limiting distribution of $\sqrt{n}(\widehat\theta_{m_n,n} - \theta_0)$ under $P_0$. But by the condition on $g$ in Assumption (ref), this bias term is zero.

Contiguous sieve MLE

In addition to Assumptions (ref)--(ref), Theorem (ref) requires that the linear expansion (ref) holds. In this section we provide two sets of conditions under which this linear expansion is satisfied for MLEs in growing contiguous parametric models. Both sets of conditions are growing parametric versions of a profile likelihood theorem due to murphy2000profile. These authors provide conditions under which the semiparametric profile likelihood admits a quadratic expansion which, in turn, implies a condition like (ref). In this section we argue similarly, but replace the {\it semiparametric} profile likelihood with a {\it contiguous sieve} profile likelihood which permits us to conclude that (ref) holds. There is a key technical advantage of working with contiguous sieve profile likelihoods in place of the semiparametric profile likelihood. The latter requires the careful construction of {`}approximately least favourable submodels{'}, which can be quite complicated to construct, as can be seen from the examples in the cited article. In contrast, as the likelihoods we work with are parametric, exact least favourable submodels can be constructed following a clear recipe as they always take the same form.

To introduce the profile likelihood, let $(\theta,\gamma) \mapsto L_m(\theta,\gamma)(x) = p_{\theta,T_m^{-1}\gamma}(x)$ be the likelihood function under $\mathcal{P}_m$, and $L_{m,n}(\theta,\gamma) = \prod_{i=1}^n L_m(\theta,\gamma)(X_i)$ be the likelihood based on an i.i.d. sample $X_1,\ldots,X_n$. Denote by ${\rm pl}_{m,n}(\theta)$ the profile likelihood based on $\mathcal{P}_m$

For each $\theta$, let $\widehat{\gamma}(\theta)$ be the value achieving this supremum, that is ${\rm pl}_{m,n}(\theta) = L_{m,n}(\theta,\widehat{\gamma}(\theta))$. We assume throughout that for large enough $n$, such a value exists.

Quadratic expansion & log concavity

A straightforward set of sufficient conditions for (ref) (or (ref)) can be obtained using the results of HjortPollard93. Specifically, let $A_{m,n}(h) = \log {\rm pl}_{m,n}(\theta_0 + h/\sqrt{n}) - \log{\rm pl}_{m,n}(\theta_0)$, then, if the functions $h \mapsto A_{m,n}(h)$ are concave and one manages to find a subsequence $(m_n)$ such that

for each $h$, then the {`}Basic Corollary{'} in HjortPollard93 immediately delivers (ref). This setting covers a large class of semiparametric models of practical interest, including the examples we study in detail in Section (ref) below. We emphasise that the concavity requirement is local, being imposed only on $A_{m, n}$ (actually only the $A_{m_n, n}$ being concave suffices).

For cases in which $A_{m, n}$ is concave and (ref) can be shown to hold, this provides a complete proof of asymptotic normality and efficiency of the contiguous sieve MLE without requiring any empirical process type arguments. We summarise this in a proposition.

Compared to Theorem (ref), the above proposition allows us to replace (ref) with (ref), provided $h \mapsto A_{m,n}(h)$ is concave. One key advantage of this is that (ref) and concavity give us {\it both} consistency and asymptotic normality of $\widehat{\theta}_{m_n,n}$, while to establish (ref) without concavity, one typically first needs to establish consistency.

We now present a theorem giving conditions under which (ref) holds. This theorem is a stripped down and sieved version of Theorem 1 in murphy2000profile, and requires that we introduce some quantities inspired by that paper. It is in the construction of these quantities we gain a lot in simplicity by working with growing parametric models, compared to attacking the semiparametric model directly. This is because under $P_m = P_{\theta_0,T_m^{-1}\gamma_0}$, the least favourable submodel for estimating $\theta_0$ always takes the form $\theta \mapsto P_{\theta,T_{m}^{-1}\gamma(\theta)}$, where $\gamma(\theta) = \gamma_0 + i_{m,11}^{-1}i_{m,10}(\theta_0 - \theta)$. In view of this, define $\gamma_t^{{\rm sub}}(\theta,\gamma) = \gamma + i_{m,11}^{-1}i_{m,10}(\theta - t)$, where we suppress the dependence of $\gamma_t^{{\rm sub}}(\theta,\gamma)$ upon $m$ from the notation. For each $m$ and each $(\theta,\gamma) \in \Theta \times \Gamma_m$ define the mappings $t \mapsto l_m(t,\theta,\gamma) \coloneqq \log L_m(t,\eta_{\gamma_t^{{\rm sub}}(\theta,\gamma)})$. These functions bridge the gap between the log-profile likelihood and the efficient score. To see this, notice that $\eta_{\gamma_{\theta}^{{\rm sub}}}(\theta,\gamma) = \gamma$, and that the derivative of $l_m(t,\theta,\gamma)$ with respect to $t$ is

in particular, $\dot{l}_m(\theta_0,\theta_0,\gamma_0) = \tilde{\ell}_{m}$. At the same time, mimicking the sandwiching technique of murphy2000profile, we have, writing $\mathbb{P}_n f = n^{-1}\sum_{i=1}^n f(X_i)$ for the integral with respect to the empirical measure, that for arbitrary $\tilde{\theta}_n$

and

In particular, the process $h \mapsto A_{m,n}(h) = \log {\rm pl}_{m,n}(\theta_0 + h/\sqrt{n}) - \log {\rm pl}_{m,n}(\theta_0)$ can be squeezed between two quantities approximating the {`}efficient LAN expansion{'} of (ref).

Random quadratic expansions

The concavity assumption that we worked under in the previous section can be substituted with a consistency assumption. Suppose that for any random sequence $\tilde{\theta}_n$ such that $\tilde{\theta}_n = \theta_0 + o_{P_{m_n}^n}(1)$ for some subsequence $(m_n)$, with $\tilde{h}_n = \tilde{\theta}_n - \theta_0$, we have

where $r_{n}(\tilde{\theta}_n) = o_{P_{m_n}^n}(\sqrt{n}\|\tilde{h}_n\| + n\|\tilde{h}_n\|^2 + 1)$. This gives a proposition that is similar to Proposition (ref), replacing concavity with consistency.

The following theorem, which is analogous to Theorem (ref), provides conditions under which the expansion given in (ref) holds.

Proposition (ref) and Theorem (ref) are growing parametric versions of Corollary 1 and Theorem 1 (respectively) in murphy2000profile. As they are proven by making what are essentially notational adjustments to the proofs of the cited results, we defer their proofs to Appendix (ref). In Appendix (ref) we also provide sufficient conditions for the assumptions made in Theorem (ref).

Applications

In this section we continue the example of the partially linear model, and also apply our theory to the Cox regression model. Both models satisfy the background assumptions on the semiparametric model (Assumption (ref) and (ref)), consequently we concentrate on the assumptions made on the parametric approximations, that is Assumptions (ref) and (ref), along with (ref). For both models we take, for simplicity, $\theta \in \Theta \subset \mathbb{R}$, and $\eta$ a real valued $s$-times continuously differentiable function on the unit interval. The parametric approximations employed take the form $\beta_m(z)^{{\rm t}}\gamma = (T_m^{-1}\gamma)(z)$, where for each $m$, $\beta_m = (\beta_{m,1},\ldots,\beta_{m,k_m})^{{\rm t}}$ is a collection of orthonormal functions, and $\gamma = (\gamma_1,\ldots,\gamma_{k_m})^{{\rm t}} \in \Gamma_m \subset \mathbb{R}^{k_m}$ are coefficients such that $\beta_m^{{\rm t}}\gamma \to \eta$ in $L_2([0,1],\nu)$, where $\nu$ is an appropriate finite measure (which is always possible, see e.g. Theorem 14.3.1 in szego1975orthogonal). To not overburden the notation, the vectors $\gamma$ are not indexed by $m$. Some of the details of the two following examples are left to appendices (ref) and (ref).

We note that the partially linear model considered immediately below is a special case of the class of {`}partially linear GLMs{'} studied by mammen1997penalized, amongst others. It should be uncomplicated to extend our results in Section (ref) below to a large subclass of such partially linear GLMs. In particular, it is straightforward to establish the concavity of $A_{m, n}$ for any such model with a canonical link function.

The partially linear model

We have $n$ i.i.d. replicates of $X = (W,Y,Z)$, for an outcome $Y$ and covariates $(W,Z)$ with values in $\mathbb{R} \times [0,1]$. The observation $X$ stems from the model $P_0 = P_{\theta_0,\eta_0}$, where each member $P_{\theta,\eta} \in \mathcal{P}$ has density

with respect to Lebesgue measure, and $f_{W,Z}(w,z)$ is the joint density of $(W,Z)$. We assume that $\mathbb{E}\,|W|^{2+\delta}<\infty$ for some $\delta>0$, and denote $P_Z$ the (marginal) law of $Z$. The parametric densities in $\mathcal{P}_m$ are of the same form as (ref), with $\eta$ replaced by $\beta_m^{{\rm t}}\gamma$. The score functions under $\mathcal{P}_m$ are then

Due to the orthonormality of $\beta_m = (\beta_{m,1},\ldots,\beta_{m,k_m})^{{\rm t}}$, the Fisher information matrix under $\mathcal{P}_m$ takes an appealing form, in particular $i_{m,11} = \sigma^{-2}I_{k_m}$ where $I_{k_m}$ is the $k_m$-dimensional identity matrix, and $i_{m,01} = \sigma^{-2}\, \mathbb{E}\, W \beta_m(Z)^{{\rm t}}$. The efficient score under $\mathcal{P}_m$ is therefore

Define $b_0(z) = \mathbb{E}\,(W \,|\, Z = z)$ and $b_m(z) = \beta_m(z)^{{\rm t}}\mathbb{E}\,[W \beta_m(Z)] = \sum_{j=1}^{k_m}\beta_{m,j}(z) \langle \beta_{m,j},b_0\rangle$. As the $\beta_m$ form an orthonormal basis for $L_2([0, 1], P_Z)$, we have $b_m\to b_0$ in $L_2([0, 1], P_Z)$. We first turn to Assumption (ref). The log-likelihood ratio of $P_{m}^n$ with respect to $P_0^n$ can be written

in terms of $h_{m,n}(z) = (\sqrt{n}/\sigma)(\beta_m(z)^{{\rm t}}\gamma_0 - \eta_0(z))$ and $\varepsilon_i = Y_i - \eta_0(Z_i) - \theta_0 W_i$. Assume that there is a subsequence $(m_n)_{n \geq 1}$ and a function $h$ such that $h_{m_n,n}\to h$ in $L_2(P_0)$. (Conditions under which this holds are given as (ref) & (ref) in Appendix section (ref).) Under this assumption it follows from the fact that the data are i.i.d. with $\varepsilon$ independent of $Z$ and the law of large numbers that, with $g(x) = h(z)(y - \eta_0(z) - \theta_0 w)/\sigma$, we have

as $\mathbb{E}_0\, h(Z)^2 = \mathbb{E}_0\, h(Z)^2 (\varepsilon / \sigma)^2$. To conclude that Assumption (ref) holds, we need also show that $\mathbb{E}_0 \, g(X) \tilde{\ell}(X) = 0$, which requires that we determine the form of the efficient score. Working formally, we expect that $\tilde{\ell}_{\theta_0,T_m^{-1}\gamma_0}$ will converge to

We verify that this is true in $L_1(P_0)$ in Appendix (ref). As $ \tilde{\ell}_{\theta_0,T_{m}^{-1}\gamma_0}(X) \sim \sigma^{-2}(W - b_m(Z))\varepsilon$ under $P_{m}$, $\mathbb{E}\,(|W|^{2+\delta})\,\mathbb{E}\,(|\varepsilon|^{2+\delta})<\infty$ by assumption, and

it follows that $\tilde{\ell}_{\theta_0,T_{m_n}^{-1}\gamma_0}$ is square uniformly $P_{m_n}$-integrable. Since also $P_{m_n} \to P_0$ in total variation by (ref) (via (ref)), Lemma (ref) in the appendix yields

Since (ref) and (ref) also hold (as shown in Appendix (ref)), it follows from Corollary (ref) that $u = \tilde{\ell}$ $P_0$-almost surely. We now have an expression for the efficient score, and can verify that Assumption (ref) holds as

In Appendix (ref) we show that equations (ref) and (ref) hold, entailing that Assumption (ref) also holds by Lemma (ref). From the developments so far, we see that the semiparametric efficient information is $J = \sigma^{-2}(\mathbb{E}\,W^2 - \|b_0\|^2) = \sigma^{-2}\mathbb{E}\,\big(W^2 - (\mathbb{E}\,[W \,|\, Z])^2\big)$, and also note that

which may also be verified directly using Parseval{'}s identity.

Let $(\widehat{\theta}_{m,n},\widehat{\gamma}_{m,n})$ be the maximum likelihood estimator under $\mathcal{P}_m$. To establish that $\widehat{\theta}_{m_n,n}$ is best regular, we show that $h \mapsto \log {\rm pl}_{m,n}(\theta_0 + h/\sqrt{n}) - \log {\rm pl}_{m,n}(\theta_0)$ is concave and admits a quadratic expansion as in (ref), which by Proposition (ref) will allow us to conclude that $\widehat{\theta}_{m_n,n}$ is best regular. The value achieving the supremum in $\sup_{\gamma \in \Gamma_m} L_{m,n}(\theta,\gamma)$ is the least squares solution $\widehat{\gamma}(\theta) = B_{m,n}^{-1}n^{-1}\sum_{i=1}^n \beta_m(Z_i)(Y_i - \theta W_i)$, where $B_{m,n} = n^{-1}\sum_{i=1}^n \beta_m(Z_i)\beta_m(Z_i)^{{\rm t}}$. The log-profile likelihood can then be expressed as $\log {\rm pl}_{m,n}(\theta) = - 1/(2\sigma^2)\sum_{i=1}^n\big(Y_i - \breve{Y}_{m,n,i} - \theta(W_i - \breve{W}_{m,n,i})\big)^2$, with $\breve{W}_{m,n,i} = \beta_m(Z_i)^{{\rm t}}B_{m,n}^{-1}n^{-1}\sum_{j=1}^n \beta_m(Z_j) W_j$ and $\breve{Y}_{m,n,i} = \beta_m(Z_i)^{{\rm t}}B_{m,n}^{-1}n^{-1}\sum_{j=1}^n \beta_m(Z_j) Y_j$. We see that $\log {\rm pl}_{m,n}(\theta)$ is concave in $\theta$. The expansion (ref) is verified Appendix (ref) under conditions (ref) and (ref). In summary, under the basic conditions given here along with (ref)--(ref), the conditions of Proposition (ref) are satisfied and consequently the estimator $\widehat{\theta}_{m_n,n}$ is best regular in $\mathcal{P}$.

Efficiency of the Cox partial likelihood estimator

The Cox regression model differs from the partially linear model in an important way. Whilst in the partially linear model a maximum likelihood estimator for $\theta$ cannot be defined without recourse to sieves, penalisation, or the like (as discussed in the Section (ref)), no such techniques are called for in the Cox regression model, as the Cox partial likelihood permits us to work directly with the semiparametric model. This entails that an estimator for the parametric part of the Cox regression model can be derived straightforwardly, without (for example) the theory developed in this paper. This theory does, however, lead to a simple proof of the efficiency of the Cox partial likelihood estimator, as we will now show. In other words, in this application the theory developed in this paper is used solely as a theoretical tool. Consequently, the choice of basis functions is of no practical importance, and one may (and, indeed, one ought to) use basis functions that make the analysis particularly tractable, which is what we do here.

Suppose we have $n$ i.i.d. replicates of $X = (T,\Delta,W)$ observed over $[0,1]$, with $T = \min(T^{\prime},C)$ and $\Delta = I(T^{\prime} \leq C)$, where the lifetime $T^{\prime}$ and the censoring time $C$ are independent given $W$, and $T^{\prime}$ given $W = w$ follows a Cox model, i.e., its hazard rate is of the form $\eta(t) \exp(\theta w)$ where $\theta \in \Theta \subset \mathbb{R}$ for simplicity, and $\eta$ belongs to the space $\mathcal{H}$ of one time continuously differentiable functions $\eta \colon [0,1] \to (0,\infty)$. As above, $P_{0} = P_{\theta_0,\eta_0}$ is the true data generating mechanism, while $P_m = P_{\theta_0,T_m^{-1}\gamma_0}$ are the parametric approximations. Here we take the basis function

where $V_{m,1},\ldots,V_{m,k_m}$ are disjoint intervals whose union make up $[0,1]$, and each interval is of length $1/k_m$. With this basis, a natural choice is to work under the coefficient $\gamma_0 = (\gamma_{0,1},\ldots,\gamma_{0,k_m})^{{\rm t}}$ with $\gamma_{0,j} = \eta_0((j-1)/k_m)/k_m$ for $j = 1,\ldots,k_m$. Introduce the counting process $N(t) = \Delta I(T \leq t)$, the at-risk process $Y(t) = I(T \geq t)$, and let $(\mathcal{F}_{t})_{t \in [0,1]}$ be the filtration generated by these. Then, with respect to this filtration, $M(t) = N(t) - \int_0^t Y(s)\eta_0(s)\exp(\theta_0 W)\,{\rm d} s$ and $M^m(t) = N(t) - \int_0^t Y(s)\beta_m(s)^{{\rm t}} \gamma_0\exp(\theta_0 W)\,{\rm d} s$ are square integrable martingales with respect to $P_0$ and $P_m$, respectively.

Using the Taylor-expansion $\log(1 + a) = a - \tfrac{1}{2} a^2 + a^2R(a)$ where $R(a) \to 0$ as $a \to 0$, the log-likelihood ratio based on the full sample can be written

where $h_{m,n} = \sqrt{n}(\beta_m^{{\rm t}}\gamma_0 - \eta_0)$, and $r_{m,n} = n^{-1}\sum_{i=1}^n\int_0^1 h_{m,n}^2/\eta_0^2R(n^{-1/2}h_{m,n}/\eta_0)\,{\rm d} N_i$ Let $(m_n)$ be so that $\sqrt{n}/k_{m_n} \to 0$. Then we can apply Lenglart{'}s inequality (e.g., andersen1993statistical or jacod2003limit) to show that $\log {\rm d} P_{m_n}^n/{\rm d} P_{0}^n$ tends to zero in probability, and Assumption (ref) is satisfied with $g =0$. In fact, with the above choice of basis functions, this is the only possible limit.

Next, we verify Assumption (ref). Under $\mathcal{P}_m$, the score functions are

when evaluated in $(\theta_0,T_m^{-1}\gamma_0)$. With $s_{m}^{(k)}(t) = \mathbb{E}_m\, Y(t)W^k \exp(W \theta_0)$ for $k = 0,1,2$, this gives

Inserting the locally constant basis functions in (ref), we find the efficient score under $\mathcal{P}_m$,

At this point it is tempting to conjecture that with $s^{(k)}$ the pointwise limits of $s_m^{(k)}$, the efficient score under $\mathcal{P}$ is $\tilde{\ell}(X) = \int_0^1 (W - s^{(1)}(t)/s^{(0)}(t) ) \,{\rm d} M(t)$. This is indeed the case, as can be established using Lemma (ref), the conditions of which we verify in Appendix (ref) by a simple, if tedious, application of Lemma (ref).

The main message is that these lemmata, combined with Corollary (ref), allow us to conclude that the efficient score under $\mathcal{P}$ is as conjectured above, that is, $\tilde{\ell}(X) = \int_0^1 (W - s^{(1)}/s^{(0)} ) \,{\rm d} M$; and consequently, the efficient information is

This method of finding the efficient score and the efficient information, leads to a new, relatively simple, and rather parametric proof of the efficiency of the Cox partial likelihood estimator (see for example BKRW98 or kosorok2008introduction for proofs that differ from ours). Let $L_{n}^{{\rm cox}}(\theta)$ be Cox{'}s partial likelihood function (see gill1984understanding for an excellent introduction). We may then define the process

and note that $h \mapsto A_{n}^{\rm cox}(h)$ is concave. Under standard regularity conditions in the Cox regression setting -- given as (ref)--(ref) in Appendix (ref) -- $A_{n}^{\rm cox}(h)$ admits the expansion $A_{n}^{\rm cox}(h) = hn^{-1/2}\sum_{i=1}^n \tilde{\ell}(X_i) - \tfrac{1}{2} h^2 J + o_{P_0}(1)$, where $\tilde{\ell}$ and $J$ are as defined above (this follows from a Taylor expansion combined with the second part of Theorem 3.2 in andersen1982cox). Since $h \mapsto A_{n}^{\rm cox}(h)$ is concave, the Basic Corollary in HjortPollard93 entails that the maximiser of $L_{n}^{{\rm cox}}(\theta)$, say $\widehat{\theta}_n^{\rm cox}$, satisfies (ref), that is $\sqrt{n}(\widehat{\theta}_n^{\rm cox} - \theta_0) = n^{-1/2} \sum_{i=1}^n J^{-1}\tilde{\ell}(X_i) + o_{P_0^n}(1)$. That the Cox partial likelihood estimator $\widehat{\theta}_n^{\rm cox}$ is best regular under $\mathcal{P}$ now follows from Lemma 25.23 and Lemma 25.25 in vandervaart1998asymptotic.

Conclusion

This paper develops an alternative approach to establishing the asymptotic normality and efficiency of certain approximate maximum likelihood estimators in semiparametric models. These estimators are contiguous sieve estimators, being maximum likelihood estimators in contiguously growing parametric models. The approach detailed in this paper, however, departs substantially from the conventional sieve literature as we work both with and under the approximating contiguous parametric models, switching back to the semiparametric model in the limit via Le Cam's third lemma. Working with (growing) parametric models allows for a straightforward definition of a maximum likelihood estimator; working under these same parametric models ensures that we may work as if our approximating models were correctly specified at each step. In the examples we have considered in detail, this approach leads to substantial simplifications and relatively straightforward proofs of asymptotic efficiency.