Online Generalized Method of Moments for Time Series

Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.

89,409 characters · 22 sections · 130 citation commands

Rendered from LaTeX for readability, not typeset faithfully. Citation keys are highlighted; maths is left as source; figures, tables and equation environments are summarised rather than reproduced; unrecognised commands are greyed out so nothing is silently dropped. Email addresses are removed.

} \affil[1]{Department of Statistics, University of Illinois at Urbana-Champaign} \affil[2]{Department of Statistics, The Chinese University of Hong Kong} \affil[3]{Department of Statistics and Data Science, and Department of Economics, Washington University in St. Louis}

abstractOnline learning has gained popularity in recent years due to the urgent need to analyse large-scale streaming data, which can be collected in perpetuity and serially dependent. This motivates us to develop the online generalized method of moments ($\mathsf{OGMM}$), an explicitly updated estimation and inference framework in the time series setting. The $\mathsf{OGMM}$ inherits many properties of offline $\mathsf{GMM}$, such as its broad applicability to many problems in econometrics and statistics, natural accommodation for over-identification, and achievement of semiparametric efficiency under temporal dependence. As an online method, the key gain relative to offline $\mathsf{GMM}$ is the vast improvement in time complexity and memory requirement. Building on the $\mathsf{OGMM}$ framework, we propose improved versions of online Sargan--Hansen and structural stability tests following recent work in econometrics and statistics. Through Monte Carlo simulations, we observe encouraging finite-sample performance in online instrumental variables regression, online over-identifying restrictions test, online quantile regression, and online anomaly detection. Interesting applications of $\mathsf{OGMM}$ to stochastic volatility modelling and inertial sensor calibration are presented to demonstrate the effectiveness of $\mathsf{OGMM}$.

{\it Keywords:} Instrumental variable; Online learning; Quantile regression; Recursive estimation; Streaming data.

\setstretch{1.25}

Introduction

The generalized method of moments ($\mathsf{GMM}$, hansen1982gmm) is a fundamental estimation and inference framework in econometrics and statistics. It is widely used to analyse economic and financial data, and subsumes many popular statistical methods such as least squares, maximum likelihood, and instrumental variables regression hall2005gmm. However, despite being partly motivated by situations in which asymptotically efficient estimation (e.g., maximum likelihood) is computationally burdensome hansen1982gmm,hall2005gmm, $\mathsf{GMM}$ does not scale to modern data sets with millions of data points. For large-scale streaming data, which can be collected in perpetuity and serially dependent, the classical (offline) $\mathsf{GMM}$ is too computationally expensive and may fail due to violation of memory constraints chen2023sgmm.

With the emergence of streaming data collection techniques, it has become important to develop online estimation and inference procedures that can be computationally (and memory) efficient while preserving statistical properties of their offline counterparts. In the context of $\mathsf{GMM}$, chen2023sgmm adapted the stochastic approximation robbins1951sgd to linear instrumental variables regression for independent data and provided plug-in and self-normalized inference procedures. They also discussed learning rate selection, multi-epoch estimation, and nonlinear $\mathsf{GMM}$. luo2020renew developed renewable estimation and incremental inference for generalized linear models with independent and identically distributed (iid) data. Their proposed estimator is based on estimating equations, which correspond to $\mathsf{GMM}$ with exact identification. Subsequently, luo2022qif extended renewable estimation to longitudinal data analysis using the quadratic inference function qu2000qif, which is also a special case of $\mathsf{GMM}$. They further discussed anomaly detection based on a modified Sargan--Hansen test. In a related work, luo2023qif focused on streaming longitudinal data with a fixed set of participants. Under a first-order autoregressive working correlation structure, they proposed a new decomposition for the quadratic inference function that allows online updates. They also considered time-varying parameter estimation by introducing an exponential smoothing factor that dynamically adjusts the weights applied to historical data batches.

The overarching goal of this article is to develop the online generalized method of moments ($\mathsf{OGMM}$), a general and efficient estimation and inference framework for streaming time series, which seems lacking in the literature. Using the first-order Taylor approximation and a lagged linearization (i.e., a linearization step with the $\mathsf{OGMM}$ estimator obtained at the previous step plugged in), we derive an explicitly updated estimator in contrast to implicitly updated renewable estimators in the literature schifano2016renew,luo2020renew,luo2022qif,luo2023qif; see toulis2017implicit for the difference between explicit and implicit updates in the context of stochastic approximation. To achieve semiparametric efficiency, we employ the optimal weighting matrix by inverting the online long-run variance estimator in rlrv. This allows us to perform efficient estimation and inference for the parameter in a fully online fashion.

Our major contributions include:

enumerate• We develop an online $\mathsf{GMM}$ estimator in the time series setting and show that the $\mathsf{OGMM}$ estimator is consistent, asymptotically normal, and achieves the same asymptotic efficiency as offline $\mathsf{GMM}$ and implicitly updated renewable estimators; • Our framework is autocorrelation-robust by handling the serial dependence in a nonparametric fashion. The semiparametric efficiency is thus achieved through online estimation of the optimal weighting matrix. Other than the user-chosen parameters involved in the online updates of the weighting matrix, our algorithm does not involve any tuning parameter such as the learning rate in stochastic approximation; • Following the work of chen2023sgmm and luo2022qif, we propose improved versions of online Sargan--Hansen and structural stability tests, which do not suffer from considerable size distortion chen2023sgmm nor require historical raw data luo2022qif; • Owing to the broad framework of $\mathsf{GMM}$, we can include online one-step estimation, online least squares, online quantile regression and online instrumental variables regression as special cases. Numerical comparison with several existing methods shows that the finite-sample performance of our $\mathsf{OGMM}$ is very competitive.

The rest of this paper is organized as follows. (ref) reviews relevant literature and introduces some notation. (ref) formulates $\mathsf{OGMM}$ by solving three computational challenges faced by $\mathsf{GMM}$, and shows that $\mathsf{OGMM}$ covers many existing statistical methods as special cases. (ref) develops the asymptotic theory of $\mathsf{OGMM}$. (ref) compares $\mathsf{OGMM}$ with some existing statistical methods in Monte Carlo experiments. (ref) presents two applications in stochastic volatility modelling and inertial sensor calibration. (ref) discusses our findings and some future directions. All experiments are performed on Red Hat Enterprise Linux 9.4 with an Intel Xeon Gold 6148 CPU and R version 4.2.3. An R-package ogmm that implements our framework is available online. All proofs and some additional results are also deferred to the online Supplement supp.

Literature review

Consider stationary and ergodic time series data $\{ D_i = (x_{i,1}, x_{i,2}, \ldots, x_{i,n_i}) \}_{i=1}^b$ arriving in batches, where $x_{i,j} \in \mathbb{R}^d$ is the $j$-th vector of observations in the $i$-th batch of data with sample size $n_i$. Let $\theta^* \in \Theta \subset \mathbb{R}^p$ be a vector of unknown parameters which are to be estimated, and $g(\theta, x)$ be a $q \times 1$ vector of functions that satisfies the population moment condition $\mathbb{E}\{ g(\theta^*, x_{i,j}) \} = 0$ for all $i,j$ with $q \ge p$. To simplify the notation, write $G(\theta; D_i) = \sum_{j=1}^{n_i} g(\theta, x_{i,j})$ and $N_b = \sum_{i=1}^b n_i$. The celebrated $\mathsf{GMM}$ estimator of $\theta^*$ is

equation[equation omitted — 167 chars of source]

where $\bar{g}_b(\theta) = N_b^{-1} \sum_{i=1}^b G(\theta; D_i)$ is the sample moment and $\tilde{W}$ is a weighting matrix.

Under suitable conditions (hansen1982gmm), it is well known that (ref) is asymptotically optimal when $\tilde{W} = \tilde\Sigma^{-1}$, where $\tilde\Sigma$ is a consistent estimator of the long-run variance matrix

equation[equation omitted — 105 chars of source]

In this case, $\sqrt{N_b}(\tilde\theta_{b, \mathsf{GMM}} - \theta^*) \stackrel{\mathrm{d}}{\to} \mathrm{N}(0, (V^\intercal \Sigma^{-1} V)^{-1})$, where $V = \mathbb{E}\{ \nabla g(\theta^*, x_{i,j}) \}$ is the expectation of the gradient of $g(\theta^*, x_{i,j})$ and $\stackrel{\mathrm{d}}{\to}$ denotes convergence in distribution. However, as $\Sigma$ depends on the unknown $\theta^*$, substitutes of $\theta^*$ are necessary for efficient estimation. Classical variants of $\mathsf{GMM}$ such as two-step, iterated and continuously updating differ in the way to substitute $\theta^*$; see hall2005gmm and the references therein.

There is a rich literature on the estimation of (ref) because the long-run variance is an important quantity in time series analysis. However, classical nonparametric estimators utilizing the overlapping batch means meketon1984obm or kernels/lag windows parzen1957kernel,andrews1991kernel cannot be updated online. wu2009recursive and rtavc,rtacm proposed different online alternatives based on subsample selection rules, which replace the constant batch size of a batch means estimator with a sequence of batch sizes. rlrv developed a different approach by decomposing kernels, which led to estimators with higher statistical and computational efficiency.

To update $\mathsf{GMM}$ and obtain online estimates, the stochastic approximation robbins1951sgd is a natural candidate. By averaging the iterates polyak1992average, one can further conduct inference using an online long-run variance estimator chen2020sgd,zhu2023sgd,rlrv or a self-normalizer/random scaling lee2022sgd. chen2023sgmm appears to be the first to extend the stochastic approximation to the $\mathsf{GMM}$ setting, and they developed the stochastic $\mathsf{GMM}$ ($\mathsf{SGMM}$) for independent data. They also proposed an online Sargan--Hansen test statistic and state its asymptotic distribution under the null hypothesis. Nevertheless, their premier algorithm assumes a fixed sample size and may not be efficient after the first epoch, which is why they recommend running over multiple epochs in practice chen2023sgmm. The computational cost or number of epochs to obtain an efficient estimate may vary, which makes $\mathsf{SGMM}$ unsuitable for streaming data collected in perpetuity. On the other hand, their online Sargan--Hansen test may suffer from considerable size distortion in finite sample. Since $\mathsf{SGMM}$ is based on explicit stochastic approximation, its performance can also be sensitive to the choice of initial learning rate toulis2017implicit. We will revisit each of these issues and show numerical evidence in (ref).

Another candidate for updating $\mathsf{GMM}$ is renewable estimation, a term coined in luo2020renew. Essentially, one stores a finite number of statistics after first-order Taylor approximations so that the estimating equation and its solution (i.e., the renewable estimator) only depends on the stored statistics and the latest batch of data schifano2016renew,luo2020renew. luo2022qif,luo2023qif investigated extension to longitudinal data analysis using quadratic inference function, which allows for over-identification due to the inverse working correlation matrix being approximated by a linear combination of some basis matrices. Renewable estimation is also widely applied in variants of online quantile regression jiang2022rdensity,sun2024rindicator,jiang2024runconditional. Nevertheless, renewable estimators involve solving fixed-point equations so their updates depend on the stopping criteria and are implicit in the sense of toulis2017implicit. Explicitly updated estimators with a lower computational cost remain largely unexplored in the literature. Additionally, the anomaly detection statistic in luo2022qif requires storing a reference data batch and solving an optimization problem for every new data batch. A fully online detector remains to be developed. We will discuss renewable estimation again in (ref) as it is closely related to our proposal. Table (ref) highlights some key differences between our proposal and some existing ones.

table[table omitted — 701 chars of source]

Before ending this section, we introduce some notations. Denote the gradient operation with respect to $\theta$ by $\nabla_\theta$. For any vector $a$ and matrix $A$, denote the $l^2$-vector norm and Frobenius norm by $\left\lVert a \right\rVert_2 = \surd a^\intercal a$ and $\left\lVert A \right\rVert_F = \surd \mathrm{tr}(A^\intercal A)$, respectively. Denote the identity matrix of size $q$ by $I_q$. We reserve the tilde, circumflex and check for offline, $\mathsf{OGMM}$ and other online methods (e.g., $\tilde\theta_{b, \mathsf{GMM}}$, $\hat\theta_{b, \mathsf{OGMM}}$ and $\check\theta_{k, \mathsf{SGMM}}$), respectively. We may also suppress the subscript if the context is clear.

Methodology

In \crefrange{sec:online-moment}{sec:online-weighting-matrix}, we formulate $\mathsf{OGMM}$ by solving three computational challenges involved in adapting $\mathsf{GMM}$ to an online setting. Then, we discuss implementation and some special cases in (ref). Some online inference examples are given in (ref).

Challenges in moment computation

In general, the sample moment $\bar{g}_b(\theta)$ cannot be updated online when $g(\theta, x)$ is nonlinear in $\theta$. To see this, suppose we have $\bar{g}_{b-1}(\theta_1)$ from the previous step and would like to compute $\bar{g}_b(\theta_2)$. We cannot apply the recursive formula for sample mean because it requires the evaluation of $\bar{g}_{b-1}(\theta_2)$. However, we can derive an online moment using the first-order Taylor approximation. Let $\hat\theta_1$ be an initial estimate based on $D_1$ and $\hat{W}$ be a weighting matrix. For $b = 1, 2, \ldots$, define

align*[align* omitted — 213 chars of source]

which are estimators of $V = \mathbb{E}\{ \nabla g(\theta^*, x_{i,j}) \}$ and $U = \mathbb{E}\{ g(\theta^*, x_{i,j}) \} -V \theta^*$, respectively. Since $V_b$ and $U_b$ are the latest online estimates after $\hat\theta_b$ is obtained, $U_b +V_b \theta$ is an online approximation of the sample moment $\bar{g}_b(\theta)$. When $b = 2$, we have

align*[align* omitted — 322 chars of source]

By storing $U_1$, $V_1$ and $N_1$ in advance, we can compute $N_1 (U_1 +V_1 \theta) +G(\theta; D_2)$ in $O(n_2 q)$ time without using $D_1$ explicitly. If the error term is asymptotically negligible, a reasonable online estimator can be obtained from

equation[equation omitted — 235 chars of source]

The derivation of (ref) shares the same idea as renewable estimation, except that the literature (e.g., luo2020renew) consider solving the first-order optimality condition

equation[equation omitted — 246 chars of source]

Rewriting (ref) gives $\hat\theta_2 = -(V_2 \hat{W} V_2)^{-1} V_2^\intercal \hat{W} U_2$, which is a fixed-point equation because $U_2$ and $V_2$ depend on $\hat\theta_2$. Therefore, the Newton--Raphson method is usually used to solve (ref) iteratively in the literature.

Challenges in stable computation

For time-sensitive applications, solving an optimization problem like (ref) or a fixed-point equation like (ref) for every new data batch may be undesirable because the time to meet the stopping criteria may vary. In light of it, we propose to estimate $V$ and $U$ by

align*[align* omitted — 493 chars of source]

respectively. Essentially, we replace $\hat\theta_b$ by $\hat\theta_{b-1}$ in $V_b$ and $U_b$ to obtain $\hat{V}_b$ and $\hat{U}_b$, respectively. Then, we can stabilize the computational time by using the lagged $\hat\theta_1$ to linearize $G(\hat\theta_2; D_2)$ and replacing $\nabla G(\hat\theta_2; D_2)$ with $\nabla G(\hat\theta_1; D_2)$ in (ref), which leads to \[ \hat{V}_2^\intercal \hat{W} (\hat{U}_2 +\hat{V}_2 \hat\theta_2) = 0. \tag{\ref*{eq:ogmm-foc}$'$} \] The solution $\hat\theta_2 = -(\hat{V}_2 \hat{W} \hat{V}_2)^{-1} \hat{V}_2^\intercal \hat{W} \hat{U}_2$ is explicit and does not involve any tuning parameter. These are also true for $b > 2$ with

equation[equation omitted — 156 chars of source]

see Algorithm (ref) for the recursive steps involved in the computation of $\hat\theta_{b, \mathsf{OGMM}}$.

If $\hat{W} = I_q$ and $g(\theta, x)$ is the score function, (ref) is same as the implicit estimator in luo2020renew obtained via the Newton--Raphson method with one iteration. In other words, (ref) can be interpreted as an explicit estimator, which seems largely unexplored in the renewable estimation literature. For stochastic gradient descent, toulis2017implicit showed that implicit and explicit procedures have identical asymptotic performance, but explicit estimators are more sensitive to the learning rate. Interestingly, (ref) also achieves the same asymptotic efficiency as its offline and implicit online counterparts under an additional condition that $n_i = o(N_{i-1}^2)$ for $i = 2, \ldots, b$; see Theorem (ref) and Remark (ref) for a discussion. However, (ref) does not require learning rate selection.

Challenges in weighting matrix computation

The final challenge is the choice of $\hat{W}$, which is important as it affects the statistical efficiency. Computationally, there are at least three estimation approaches as discussed below. (1) We compute $\hat\Sigma_1$ based on $D_1$ and fix $\hat{W} = \hat\Sigma_1^{-1}$. Many classical estimators are available and the computational cost is minimized. Nevertheless, this approach may be inefficient since it does not utilize the sequentially arrived data. (2) We assume $g(\theta^*, x_{i,j})$ follows some parametric model (e.g., a vector autoregressive moving average model; see Section 3.5.2 of hall2005gmm) and update such model recursively. Then, we can plug in the online parameter estimates to obtain $\hat\Sigma_b$ via the long-run variance estimator based on the parametric model and set $\hat{W} = \hat\Sigma_b^{-1}$. This is similar to using a working correlation matrix in luo2022qif,luo2023qif, which leads to a loss of statistical efficiency in case of model misspecification. (3) We update $\hat{W}$ using an online nonparametric estimator $\hat\Sigma_b(K)$, where $K$ is a kernel function that satisfies some conditions. This approach yields both statistical efficiency and autocorrelation robustness. While it involves some smoothing parameter, there are automatic optimal selectors in the literature rtacm,rlrv so that users only need to provide problem-specific information, e.g., the strength of serial dependence and the memory constraint.

In general, we recommend the third approach. If users have a strict time budget or are certain that the data are independent, they can use a fixed weighting matrix or invert welford1962recursive's welford1962recursive sample variance estimator, which are implemented in our R-package ogmm. However, the default choice is to invert the long-run variance estimator in rlrv, whose implementation allows batch updates and includes a positive definiteness adjustment. To be specific, their long-run variance estimator takes the form

equation[equation omitted — 172 chars of source]

where $X_k$'s are the evaluated moment functions indexed by a single subscript in our setting, i.e., $X_k = g(\hat\theta_b, x_{b,j})$ if $k = \sum_{i=1}^{b-1} n_i +j$, and $\bar{X}_{N_b} = N_b^{-1} \sum_{i=1}^{N_b} X_i \equiv N_b^{-1} \sum_{j=1}^b G(\hat\theta_j; D_j)$. Their kernel takes the form \[ K_n(i, j) = \left( 1 -\frac{|i-j|^\lambda}{t_n^\lambda} \right) \mathbb{1}_{|i-j| \le s'_{i \lor j}}, \] where $s_n = \min( \lfloor \Psi n^\psi \rfloor, n-1)$; $t_n = \min( \lceil \Xi n^\xi \rceil, n)$; \[ s'_n = \left\{

array[array omitted — 155 chars of source]

\right. \] $\Psi,\Xi \in \mathbb{R}^+$; $\psi,\xi \in (0,1)$; $\lambda \in \mathbb{Z}^+$; and $\phi \in [1,\infty)$. Their optimal parameters selector automatically handles $\Psi$, $\Xi$, $\psi$ and $\xi$ so users only need to choose $\lambda$ and $\phi$. Following rlrv, we recommend $\lambda=1$ ($\lambda=3$) if the serial dependence is strong (weak), and $\phi=1$ ($\phi=2$) if the memory is abundant (scarce).

Implementation and special cases

algorithm[algorithm omitted — 1,222 chars of source]

Algorithm (ref) summarizes the key steps in $\mathsf{OGMM}$ estimation. Alternatively, let $\hat\theta'_b = \hat\theta'_{b-1} -\{ (\hat{V}'_b)^\intercal \hat{W} \hat{V}'_b \}^{-1} (\hat{V}'_b)^\intercal \hat{W} \hat{U}'_b$, where $V'_1 = \nabla \bar{g}_1(\hat\theta'_1)$, $V'_b = N_b^{-1} \{ N_{b-1} V'_{b-1} +\nabla G(\hat\theta'_b; D_b) \}$, $\hat{V}'_b = N_b^{-1} \{ N_{b-1} V'_{b-1} +\nabla G(\hat\theta'_{b-1}; D_b) \}$, $U'_1 = \bar{g}_1(\hat\theta'_1)$, $U'_b = N_b^{-1} \{ N_{b-1} U'_{b-1} +N_{b-1} V'_{b-1} (\hat\theta'_b -\hat\theta'_{b-1}) +G(\hat\theta'_b; D_b) \}$, and $\hat{U}'_b = N_b^{-1} \{ N_{b-1} U'_{b-1} +G(\hat\theta'_{b-1}; D_b) \}$. Here, $U'_b$ and $\hat{U}'_b$ are estimators of $U +V \theta^* = \mathbb{E}\{ g(\theta^*, x_{i,j}) \}$. The following proposition verifies that $\hat\theta'_b \equiv \hat\theta_b$ and $U'_b \equiv U_b +V_b \hat\theta_b$. Since fewer arithmetic operations are involved in this alternative definition, we actually implement $\hat\theta'_b$ in our R-package ogmm.

proposition[Validity of the telescoping-based formula] If $\hat\theta'_1 = \hat\theta_1$, then $\hat\theta'_b \equiv \hat\theta_b$ and $U'_b \equiv U_b +V_b \hat\theta_b$ for $b = 1, 2, \ldots$.

Besides, it is not necessary to initialize $\hat\theta_1$ using $\mathsf{GMM}$. Any $\sqrt{N_1}$-consistent $\hat\theta_1$ will be sufficient for $\hat\theta_2$ to achieve the same asymptotic efficiency as $\mathsf{GMM}$. For example, one can compute $\hat\theta_1$ using the two-stage least squares ($\mathsf{2SLS}$) for linear instrumental variables estimation. The consistency of $\hat\theta_1$ is important so users may want to combine several data batches to form a large $D_1$ in practice, which is also recommended in luo2022qif.

We further analyse the computational complexities of Algorithm (ref) after initialization. For lines 7--11 and 13--14, the total time and space complexities are $O(n_b p q +p q^2)$ and $O(n_b q +q^2)$, respectively. For line 12 with the long-run variance estimator in rlrv, the time complexity is $O(n_b q^2 +q^3)$, and the space complexity is $O(N_b^{1/(1+2\lambda)} q^2 \mathbb{1}_{\phi<2} +n_b q +q^2)$. For comparison, the time complexity of offline $\mathsf{GMM}$ is at least $O(N_b q)$, which can be greater depending on the optimization method and how the weighting matrix is chosen, and the space complexity is $O(N_b q +q^2)$. As for general $\mathsf{SGMM}$, the time complexity to update for $n_b$ iterations is $O(n_b p q^2)$, and the space complexity is $O(n_b q +q^2)$ (or $O(N_b q +q^2)$ if multiple epochs are needed). We will compare the computational cost of $\mathsf{GMM}$, $\mathsf{SGMM}$ and $\mathsf{OGMM}$ in simulation studies later.

Same as $\mathsf{GMM}$, $\mathsf{OGMM}$ covers many statistical methods as special cases. We give a few examples below to demonstrate the generality of Algorithm (ref).

example[One-step estimation] Suppose $\hat{W} = I_q$ and $g(\theta, x)$ is the score function. Let $\hat\theta_1$ be a $\sqrt{N_1}$-consistent estimator and $D_2$ be empty. Then, $ \hat\theta_2 = -(\hat{V}_2 \hat{W} \hat{V}_2)^{-1} \hat{V}_2^\intercal \hat{W} \hat{U}_2 = \hat\theta_1 -(V_1^\intercal I_q V_1)^{-1} V_1^\intercal I_q \bar{g}_1(\hat\theta_1) = \hat\theta_1 -V_1^{-1} \bar{g}_1(\hat\theta_1) $ performs lecam1956asymptotic's lecam1956asymptotic one-step estimation provided that $V_1$ is a consistent estimator for the Fisher information matrix. In this regard, $\mathsf{OGMM}$ can be viewed as a one-step estimator for $\mathsf{GMM}$ with online data.

example[Online least squares] Let $Y_i = (y_{i,1}, \ldots, y_{i,n_i})^\intercal$ and $X_i = (x_{i,1}, \ldots, x_{i,n_i})^\intercal$ be the $i$-th batch of responses and predictors, respectively. The ordinary least squares is equivalent to $\mathsf{GMM}$ with $\hat{W} = I_q$ and $g(\theta, \{x, y\}) = x (y -x^\intercal \theta)$. We can verify that $\nabla g(\theta, \{x, y\}) = -x x^\intercal$, $\hat{V}_b = -N_b^{-1} \sum_{i=1}^b X_i^\intercal X_i$ and $\hat{U}_b = N_b^{-1} \sum_{i=1}^b X_i^\intercal Y_i$. Therefore, $\hat\theta_b = (\sum_{i=1}^b X_i^\intercal X_i)^{-1} (\sum_{i=1}^b X_i^\intercal Y_i)$ exactly recovers the ordinary least squares.

example[Online quantile regression] To simplify the discussion here, suppose we only use an initial estimate $\hat\theta_{\tau, 1}$ to update the summary statistics of $\mathsf{OGMM}$ and the online linear estimator for quantile regression ($\mathsf{LEQR}$) in chen2019rindicator, where $\tau$ is a quantile level of interest. We will compare the actual estimators in (ref). Let $\check{U}_{b, \mathsf{LEQR}} = N_b^{-1} \sum_{i=1}^b X_i^\intercal [H \{ (Y_i -X_i \hat\theta_{\tau, 1})/h_i \} +\tau -1 +(Y_i/h_i) H \{ (Y_i -X_i \hat\theta_{\tau, 1})/h_i \}]$, where $H(\cdot)$ is a smooth approximation of the indicator function $\mathbb{1}_{\cdot > 0}$, $\nabla H(\cdot)$ is the gradient of $H(\cdot)$, and $\{ h_i \}$ is a sequence of bandwidths. The smoothed quantile regression decastro2019iv is equivalent to $\mathsf{GMM}$ with $\hat{W} = I_q$ and $g(\theta_\tau, \{x, y, h\}) = x [H\{ (y -x^\intercal \theta_\tau)/h \} +\tau -1]$. We can verify that $\nabla g(\theta_\tau, \{x, y, h\}) = -(x x^\intercal/h) \nabla H \{ (y -x^\intercal \theta_\tau)/h \}$, $\hat{V}_b = -N_b^{-1} \sum_{i=1}^b (X_i^\intercal X_i/h_i) \nabla H \{ (Y_i -X_i \hat\theta_{\tau, 1})/h_i \}$ and $\hat{U}_b = N_b^{-1} \sum_{i=1}^b X_i^\intercal [H \{ (Y_i -X_i \hat\theta_{\tau, 1})/h_i \} +\tau -1 +(X_i \hat\theta_{\tau, 1}/h_i) H \{ (Y_i -X_i \hat\theta_{\tau, 1})/h_i \}]$. The resulting estimator $\hat\theta_b = \hat{V}_b^{-1} \hat{U}_b$ is almost same as $\check\theta_{b, \mathsf{LEQR}} = \hat{V}_b^{-1} \check{U}_{b, \mathsf{LEQR}}$, except that $X_i \hat\theta_{\tau, 1}/h_i$ in $\hat{U}_b$ is replaced with $Y_i/h_i$ in $\check{U}_{b, \mathsf{LEQR}}$.

Online inference and applications

To conduct inference under the $\mathsf{OGMM}$ framework, we can simply replace offline statistics in the $\mathsf{GMM}$ literature with online counterparts from Algorithm (ref). Consequently, we can improve the computational efficiency while preserving statistical properties of many classical inferential procedures.

example[Over-identifying restrictions testing and confidence region estimation] Let $\chi_{p, \alpha}^2$ be the $\alpha$-quantile of a chi-squared distribution with $p$ degrees of freedom. The offline Sargan--Hansen test statistic is $N_b \bar{g}_b(\tilde\theta_{b, \mathsf{GMM}})^\intercal \tilde\Sigma^{-1} \bar{g}_b(\tilde\theta_{b, \mathsf{GMM}})$, where $\tilde\Sigma$ is an offline estimator of $\Sigma$. Since $U_b +V_b \theta$ is an online approximation of $\bar{g}_b(\theta)$, a natural online test statistic is $T = N_b (U_b +V_b \hat\theta_{b, \mathsf{OGMM}})^\intercal \hat\Sigma_b^{-1} (U_b +V_b \hat\theta_{b, \mathsf{OGMM}})$. We reject the null hypothesis of $\mathbb{E}\{g(\theta^*, x_{i,j})\} = 0$ at $100\alpha\%$ nominal level if $T > \chi_{q-p, 1-\alpha}^2$. Similarly, an online $100(1-\alpha)\%$ confidence region for $\theta^*$ is $\{\theta: N_b (\hat\theta_{b, \mathsf{OGMM}}-\theta)^\intercal (V_b^\intercal \hat\Sigma_b^{-1} V_b)^{-1} (\hat\theta_{b, \mathsf{OGMM}}-\theta) \le \chi_{p, 1-\alpha}^2 \}$.

example[Anomaly detection] Consider the abnormal data batch detection setting in luo2022qif, where $D_1$ is the normal reference and $D_b$ is possibly abnormal. We are interested in testing $H_0: \mathbb{E}\{ g(\theta^*, x_{1,j}) \} = \mathbb{E}\{ g(\theta^*, x_{b,h}) \} = 0$ against $H_1: \mathbb{E}\{ g(\theta^*, x_{1,j}) \} = 0$, but $\mathbb{E}\{ g(\theta^*, x_{b,h}) \} \ne 0$ for $j \in \{1, \ldots, n_1\}$ and $h \in \{1, \ldots, n_b\}$. luo2022qif proposed \begin{align} T_\mathrm{R} &= n_1^{-1} G(\tilde{\theta}_{b, \mathrm{R}}; D_1)^\intercal \tilde{C}_1^{-1} G(\tilde{\theta}_{b, \mathrm{R}}; D_1) +n_b^{-1} G(\tilde{\theta}_{b, \mathrm{R}}; D_b)^\intercal \tilde{C}_b^{-1} G(\tilde{\theta}_{b, \mathrm{R}}; D_b), \quad where \\ \tilde{\theta}_{b, \mathrm{R}} &= \mathop{\mathrm{argmin}}_{\theta \in \Theta} \left\{ n_1^{-1} G(\theta; D_1)^\intercal \tilde{C}_1(\theta)^{-1} G(\theta; D_1) +n_b^{-1} G(\theta; D_b)^\intercal \tilde{C}_b(\theta)^{-1} G(\theta; D_b) \right\} \end{align} is a restricted estimator, and $\tilde{C}_i(\theta)$ is the sample variance that depends on $\theta$ and $D_i$ for $i \in \{1,b\}$. Same as (ref), one can obtain (ref) using two-step, iterated or continuously updating $\mathsf{GMM}$. The test statistic in (ref) is not autocorrelation-robust though and its computational complexities depend on $D_1$. If the normal reference contains all previous data batches, computing (ref) will be very costly. Therefore, we propose \begin{equation} T_\mathrm{F} = n_1 (U_1 +V_1 \hat\theta_{b, \mathrm{F}})^\intercal \hat\Sigma_1^{-1} (U_1 +V_1 \hat\theta_{b, \mathrm{F}}) +n_b^{-1} G(\hat\theta_{b, \mathrm{F}}; D_b)^\intercal \hat\Sigma_1^{-1} G(\hat\theta_{b, \mathrm{F}}; D_b), \end{equation} where $\hat\theta_{b, \mathrm{F}}$ is the full-sample estimator that can be obtained by an $\mathsf{OGMM}$ update with $D_b$, and $\hat\Sigma_1$ is the estimated long-run variance before receiving $D_b$ (assuming the serial dependence structure does not change over time). Under $H_0$, both (ref) and (ref) converge in distribution to $\chi_{2q-p}^2$. However, (ref) does not require solving another optimization problem nor storing previous data batches.

example[Structural stability testing] Abnormality in Example (ref) corresponds to a more general behaviour termed structural instability with a known break point in the context of $\mathsf{GMM}$. Indeed, replacing the restricted estimator in (ref) with the full-sample estimator in (ref) is inspired by a common practice in structural stability testing; see p.173--174 of hall2005gmm for additional references and discussion. Now, for $i \in \{1, b\}$, let $\Sigma_i(\theta)$ and $P_i(\theta)$ be the subsample analogues of $\Sigma(\theta) = \lim_{N_b \to \infty} N_b \mathrm{Var}\{ \bar{g}_b(\theta) \}$ and $P(\theta) = \Sigma(\theta)^{-1/2} V(\theta) \{V(\theta)^\intercal \Sigma(\theta)^{-1} V(\theta)\}^{-1} V(\theta)^\intercal \Sigma(\theta)^{-1/2}$ for $D_i$, respectively, where $V(\theta) = \mathbb{E}\{ \nabla g(\theta, x_{i,j}) \}$. hall1999stability decomposed $H_0$ into \begin{align*} H_{0, I}: & P_1(\theta^*) \Sigma_1(\theta^*)^{-1/2} \mathbb{E}\{ g(\theta^*, x_{1,j}) \} = P_b(\theta^*) \Sigma_b(\theta^*)^{-1/2} \mathbb{E}\{ g(\theta^*, x_{b,j}) \} = 0 \quad and \\ H_{0, O}: & \{I_q -P_1(\theta^*)\} \Sigma_1(\theta^*)^{-1/2} \mathbb{E}\{ g(\theta^*, x_{1,j}) \} = \{I_q -P_b(\theta^*)\} \Sigma_b(\theta^*)^{-1/2} \mathbb{E}\{ g(\theta^*, x_{b,j}) \} = 0. \end{align*} If $H_0$ is rejected and $\mathbb{E}\{ g(\theta^*, x_{b,h}) \} \ne 0$ for $h \in \{1, \ldots, n_b\}$, there are two possibilities. (1) There is some unique $\theta' \in \Theta \setminus \{\theta^*\}$ that satisfies $\mathbb{E}\{ g(\theta', x_{b,h}) \} = 0$, which means that the instability is caused by a change point in parameter values. In this case, only $H_{0, \text{I}}$ is violated. (2) There is no $\theta \in \Theta$ that satisfies $\mathbb{E}\{ g(\theta, x_{b,h}) \} = 0$, which means that the instability is caused by a more fundamental misspecification instead of a change point in parameter value. In this case, $H_{0, \text{O}}$ and likely $H_{0, \text{I}}$ are both violated. The test statistics in Example (ref) are not able to distinguish between these two cases. Therefore, based on hall1999stability, we propose \begin{equation} T_\mathrm{U} = n_1 (U_1 +V_1 \hat\theta_1)^\intercal \hat\Sigma_1^{-1} (U_1 +V_1 \hat\theta_1) +n_b^{-1} G(\tilde\theta_{b, \mathrm{U}}; D_b)^\intercal \tilde\Sigma_{b, \mathrm{U}}^{-1} G(\tilde\theta_{b, \mathrm{U}}; D_b), \end{equation} where $\tilde\theta_{b, \mathrm{U}}$ and $\tilde\Sigma_{b, \mathrm{U}}$ are the (unrestricted) $\mathsf{GMM}$ and long-run variance estimator computed with $D_b$ only. Since $U_1$, $V_1$, $\hat\theta_1$ and $\hat\Sigma_1$ are available before receiving $D_b$, (ref) is computationally efficient. Under the null hypothesis, (ref) converges in distribution to $\chi_{2(q-p)}^2$ and is asymptotically independent of (ref) and (ref). Therefore, given that (ref) or (ref) is rejected, we may believe that there is a change point in parameter value if (ref) is not rejected, and the model is misspecified if (ref) is rejected. The unknown break point case seems more challenging and is left for future research.

Theory

To compare with $\mathsf{GMM}$ on the same theoretical basis, we develop the asymptotic theory of $\mathsf{OGMM}$ based on the assumptions in hall2005gmm. Two different cases that correspond to the scenario (S2) in luo2022qif are considered: (a) $\min(n_1, \ldots, n_b) \to \infty$; (b) $n_1 \to \infty$ and no restrictions on $n_2, n_3, \ldots$. We allow $b$ to be any finite positive integer but require $n_1 \to \infty$ to ensure the initial estimate is consistent, which is implicitly assumed in luo2020renew and many subsequent works.

Consistency

assumption[Strict stationarity] The time series data $\{x_{1,1}, \ldots, x_{1,n_1}, x_{2,1}, \ldots, x_{2,n_2}, \ldots\}$ form a strictly stationary process with sample space $\mathcal{X} \subset \mathbb{R}^d$.

assumption[Regularity conditions for $g(\theta, x)$] The function $g(\theta, x)$ satisfies: \begin{enumerate} • it is continuous on $\Theta$ for each $x \in \mathcal{X}$; • $\mathbb{E}\{g(\theta, x_{i,j})\}$ exists and is finite for every $\theta \in \Theta$; • $\mathbb{E}\{g(\theta, x_{i,j})\}$ is continuous on $\Theta$. \end{enumerate}

assumption[Population moment condition] The data $\{x_{i,j}\}$ and the parameter $\theta^*$ satisfy the population moment condition $\mathbb{E}\{g(\theta^*, x_{i,j})\} = 0$.

assumption[Global identification] $\mathbb{E}\{g(\theta', x_{i,j})\} \ne 0$ for all $\theta' \in \Theta \setminus \{\theta^*\}$.

assumption[Regularity conditions for $\nabla g(\theta, x)$] \begin{enumerate} • $\nabla g(\theta, x)$ exists and is continuous on $\Theta$ for each $x \in \mathcal{X}$; • $\theta^*$ is an interior point of $\Theta$; • $\mathbb{E}\{\nabla g(\theta^*, x_{i,j})\}$ exists and is finite. \end{enumerate}

assumption[Weighting matrix] $\hat{W}$ is a positive semi-definite matrix which converges in probability to a positive definite constant matrix $W$.

assumption[Ergodicity] The random process $\{x_{i,j}\}$ is ergodic.

assumption[Compactness] $\Theta$ is a compact set.

assumption[Domination of $g(\theta, x)$] $\mathbb{E}\{\sup_{\theta \in \Theta} \left\lVert g(\theta, x_{i,j}) \right\rVert_2 \} < \infty$.

assumption[Long-run variance] \begin{enumerate} • $\mathbb{E}\{ g(\theta^*, x_{i,j}) g(\theta^*, x_{i,j})^\intercal \}$ exists and is finite; • $\Sigma$ defined in (ref) exists and is a finite positive definite matrix. \end{enumerate}

assumption[Local Lipschitz continuity of $\nabla g(\theta, x)$] Let $B_\epsilon = \{\theta \in \Theta: \left\lVert \theta-\theta^* \right\rVert_2 \le \epsilon\}$. \begin{enumerate} • For some $\epsilon > 0$, there exists $L > 0$ such that $\left\lVert \mathbb{E}\{\nabla g(\theta, x_{i,j})\} -\mathbb{E}\{\nabla g(\theta^*, x_{i,j})\} \right\rVert_F \le L \left\lVert \theta-\theta^* \right\rVert_2$ for all $\theta \in B_\epsilon$. • For some $\epsilon > 0$, there exists $L > 0$ such that $\left\lVert \nabla g(\theta, x) -\nabla g(\theta^*, x) \right\rVert_F \le L \left\lVert \theta-\theta^* \right\rVert_2$ for all $\theta \in B_\epsilon$ and $x \in \mathcal{X}$. \end{enumerate}

assumption[Local uniform convergence to $\mathbb{E}\{\nabla g(\theta, x_{i,j})\}$] \begin{enumerate} • For some $\epsilon > 0$ and $i=1, \ldots, b$, as $n_i \to \infty$, \begin{enumerate}[label=(\roman*)] • $\sup_{\theta \in B_\epsilon} \left\lVert n_i^{-1} \nabla G(\theta; D_i) -\mathbb{E}\{\nabla g(\theta, x_{i,j})\} \right\rVert_F = o_p(1)$, which is implied by • $\sup_{\theta \in B_\epsilon} \left\lVert n_i^{-1} \nabla G(\theta; D_i) -\mathbb{E}\{\nabla g(\theta, x_{i,j})\} \right\rVert_F = O_p(n_i^{-1/2})$. \end{enumerate} • For some $\epsilon > 0$, as $N_b \to \infty$, \begin{enumerate}[label=(\roman*)] • $\left\lVert N_b^{-1} \sum_{k=1}^b \nabla G(\theta^*; D_k) -\mathbb{E}\{\nabla g(\theta^*, x_{i,j})\} \right\rVert_F = o_p(1)$, which is implied by • $\sup_{\theta \in B_\epsilon} \left\lVert N_b^{-1} \sum_{k=1}^b \nabla G(\theta; D_k) -\mathbb{E}\{\nabla g(\theta, x_{i,j})\} \right\rVert_F = o_p(1)$. \end{enumerate} \end{enumerate}

Assumptions (ref)--(ref) are identical to Assumptions 3.1--3.5 and 3.7--3.11 in hall2005gmm. Assumptions (ref) and (ref) are required to hold for some neighbourhood $B_\epsilon$ of $\theta^*$ only so both are local assumptions. Although Assumptions (ref)(a) and (b) (local Lipschitz continuity) are stronger than Assumption 3.12 (continuity) in hall2005gmm, they are standard in the renewable estimation literature to bound the Taylor approximation error. Compared with Condition 3 in luo2020renew, Assumption (ref)(b) is slightly weaker because only local smoothness is required, and Assumption (ref)(a) is implied by Assumption (ref)(b). Note that Assumption (ref)(a)(ii) is stronger than Assumption (ref)(b)(ii), where the latter is identical to Assumption 3.13 in hall2005gmm.

theorem[Consistency of $\hat{\theta}_b$] Suppose Assumptions (ref)--(ref) hold. \begin{enumerate} • Under Assumptions (ref)(a) and (ref)(a)(i), $\hat\theta_b \stackrel{\mathrm{p}}{\to} \theta^*$ as $\min(n_1, \ldots, n_b) \to \infty$. • Under Assumptions (ref)(b) and (ref)(b)(i), $\hat\theta_b \stackrel{\mathrm{p}}{\to} \theta^*$ as $n_1 \to \infty$. \end{enumerate}

When $b = 1$, Theorem (ref) is exactly Theorem 3.1 in hall2005gmm, which states the consistency of $\hat\theta_1$ if it is computed using $\mathsf{GMM}$. Then, for $b \ge 2$, Assumptions (ref) and (ref)(i) ensure the Taylor approximation error is negligible so that $\hat\theta_b$ remains consistent.

Asymptotic normality

theorem[Asymptotic normality of $\hat\theta_b$] Recall that $V = \mathbb{E}\{\nabla g(\theta^*, x_{i,j})\}$. Let $H = (V^\intercal W V)^{-1} V^\intercal W$. Suppose Assumptions (ref)--(ref) hold and $n_i = o(N_{i-1}^2)$ for $i = 2, \ldots, b$. \begin{enumerate} • Under Assumptions (ref)(a) and (ref)(a)(ii), $\sqrt{N_b} (\hat\theta_b -\theta^*) \stackrel{\mathrm{d}}{\to} \mathrm{N}(0, H \Sigma H^\intercal)$ as $\min(n_1, \ldots, n_b) \to \infty$. • Under Assumptions (ref)(b) and (ref)(b)(ii), $\sqrt{N_b} (\hat\theta_b -\theta^*) \stackrel{\mathrm{d}}{\to} \mathrm{N}(0, H \Sigma H^\intercal)$ as $n_1 \to \infty$. \end{enumerate}

When $b = 1$, Theorem (ref) is exactly Theorem 3.2 in hall2005gmm, which states the asymptotic normality of $\hat\theta_1$ if it is computed using $\mathsf{GMM}$. The asymptotic variance is minimized at $(V^\intercal \Sigma^{-1} V)^{-1}$ when $W = \Sigma^{-1}$. For $b \ge 2$, $\hat\theta_b$ achieves the same asymptotic efficiency regardless of whether the initial estimator is $\mathsf{GMM}$. As briefly discussed in Example (ref), any $\sqrt{N_1}$-consistent $\hat\theta_1$ will be sufficient for $\hat\theta_2$ to achieve the same asymptotic efficiency as $\mathsf{GMM}$. Therefore, users have some flexibility in choosing $\hat\theta_1$ in Algorithm (ref).

Compared with renewable estimators in the literature, $\mathsf{OGMM}$ also achieves the same asymptotic efficiency as its offline counterpart in Theorem (ref). The condition $n_i = o(N_{i-1}^2)$ for $i = 2, \ldots, b$ appears to be new and mild, and is needed to control the Taylor approximation error of explicit updates in $\mathsf{OGMM}$. It is easily seen to be satisfied when the batch size is constant, and can be fulfilled in practice by splitting a large batch of new data into smaller batches. Other assumptions are similar to the standard conditions in the $\mathsf{GMM}$ or renewable estimation literature.

remark[Explicit renewable estimators] The renewable estimators (4.5) and (4.12) for unconditional quantile regression in jiang2024runconditional are also explicit. Their additional condition is $n_i = O(n_1)$ for $i = 2, \ldots, b$, which implies our condition.

remark[Verification of Assumptions (ref)(b) and (ref)(b)(ii)] In practice, case (b) is usually of broader interest because there are no restrictions on $n_2,n_3,\ldots$. Therefore, we give some examples where we can verify Assumptions (ref)(b) and (ref)(b)(ii). \begin{enumerate} • For the ordinary least squares in Example (ref), Assumption (ref)(b) holds because $\nabla g(\theta, \{x,y\}) = -x x^\intercal$ does not depend on $\theta$. By the law of large numbers, Assumption (ref)(b)(ii) also holds if $\mathbb{E}\{ \left\lVert x x^\intercal \right\rVert_F \} < \infty$. • The instrumental variables estimation is equivalent to $\mathsf{GMM}$ with $g(\theta, \{x, y, z\}) = z (y -x^\intercal \theta)$, where $z$ is a vector of instrumental variables. Similar to the ordinary least squares, Assumption (ref)(b) holds because $\nabla g(\theta, \{x, y, z\}) = -z x^\intercal$ does not depend on $\theta$, and Assumption (ref)(b)(ii) holds if $\mathbb{E}\{ \left\lVert z x^\intercal \right\rVert_F \} < \infty$. • The maximum likelihood estimation is equivalent to $\mathsf{GMM}$ with $g(\theta, x) = \nabla \ell(\theta, x)$, where $\ell(\theta, x)$ is the log-likelihood function. If $\ell(\theta, x)$ is twice continuously differentiable for all $\theta \in \Theta$ (Condition 3 in luo2020renew), Assumption (ref)(b) holds by the mean value theorem. If $\Theta$ is compact and $\mathbb{E}\{\sup_{\theta \in \Theta} \left\lVert \nabla^2 \ell(\theta, x) \right\rVert_F \} < \infty$, then Assumption (ref)(b)(ii) also holds by Lemma 2.4 in newey1994asymptotic. \end{enumerate}

Over-identifying restrictions test

theorem[Asymptotic distributions of the online sample moment and online Sargan--Hansen test statistic] If the conditions in Theorem (ref)(a) or (b) hold, then $\hat{W}^{1/2} \sqrt{N_b} (U_b +V_b \hat\theta_b) \stackrel{\mathrm{d}}{\to} \mathrm{N}(0, R W^{1/2} \Sigma (W^{1/2})^\intercal R^\intercal)$ as (a) $\min(n_1, \ldots, n_b) \to \infty$; or (b) $n_1 \to \infty$, where $R = I_q -W^{1/2} V (V^\intercal W V)^{-1} V^\intercal (W^{1/2})^\intercal$. Suppose further that $\hat\Sigma$ is positive semi-definite and converges in probability to $\Sigma$ in Assumption (ref). Then $N_b (U_b +V_b \hat\theta_b)^\intercal \hat\Sigma^{-1} (U_b +V_b \hat\theta_b) \stackrel{\mathrm{d}}{\to} \chi_{q-p}^2$ as (a) $\min(n_1, \ldots, n_b) \to \infty$; or (b) $n_1 \to \infty$.

When $b = 1$, Theorem (ref) is equivalent to Theorems 3.3 and 5.1 in hall2005gmm if $\hat\theta_1$ is computed using $\mathsf{GMM}$ because $U_1 +V_1 \hat\theta_1 \equiv \bar{g}_1(\hat\theta_1)$. For $b \ge 2$, the online sample moment and online Sargan--Hansen test statistic differ from their offline version as more than one $\hat\theta_i$'s are plugged in. Therefore, we have an additional first-order correction term $N_b^{-1} \sum_{i=1}^b \nabla G(\hat\theta_i; D_i) (\hat\theta_b -\hat\theta_i)$ compared with chen2023sgmm. We remark that the condition on $\hat\Sigma$ is satisfied by the online long-run variance estimator in rlrv under standard conditions.

Simulation results

In this section, we examine the finite-sample performance of $\mathsf{OGMM}$, as compared to several competitors in the literature. Specifically, we report the results for online instrumental variables regression in (ref), for online Sargan-Hansen test in (ref), for online quantile regression in (ref) and for online anomaly detection in (ref).

Throughout (ref), we may lighten the notation by using a single subscript to index variables, e.g., $x_k = x_{b, j}$ with $k = \sum_{i=1}^{b-1} n_i +j$. Unless otherwise stated, we use the R-package momentfit 0.5 for offline two-step $\mathsf{GMM}$ and $\mathsf{2SLS}$, sandwich 3.1.1 for the Bartlett kernel estimator with a bandwidth selected by fitting a first-order autoregressive model andrews1991kernel and prewhite=FALSE, rlaser 0.1.1 for rlrv's rlrv recursive long-run variance estimator with $\lambda=1$, $\phi=1$, pilot equals to $n_1$ and warm=FALSE, and \texttt{ogmm} 0.0.1 for $\mathsf{OGMM}$, $\mathsf{SGMM}$ and $\mathsf{LEQR}$. Non-default arguments will be discussed whenever they are modified. All simulations are repeated for $1000$ times.

Online instrumental variables regression

Before we investigate the finite-sample performance of $\mathsf{OGMM}$, we describe $\mathsf{SGMM}$ chen2023sgmm, another framework for computing $\mathsf{GMM}$ with online updates. Let $\eta_k = \eta_0 k^{-a}$ be the learning rate at the $k$-th iteration for some $\eta_0 > 0$ and $a \in (1/2, 1)$. The general $\mathsf{SGMM}$ algorithm considers $ \check\theta_{k+1} = \check\theta_k -\eta_{k+1} (\check{V}_k^\intercal \check{W}_k \check{V}_k)^{-1} \check{V}_k^\intercal \check{W}_k g(\check\theta_k, x_{k+1}), $ where $\check{V}_k$ is an online average that estimate $V = \mathbb{E}\{ \nabla g(\theta^*, x_{i,j}) \}$, and $\check{W}_k$ is the inverse of the second sample moment updated using the Sherman--Morrison--Woodbury formula. By updating the average $\bar\theta_N = N^{-1} \sum_{k=1}^N \check\theta_k$, inference on $\theta^*$ can be performed using a plug-in approach. To conduct the Sargan--Hansen test, chen2023sgmm proposed the test statistic

equation[equation omitted — 179 chars of source]

For the initial learning rate, consider using an initialization data batch $D_0$ to compute $\check\theta_0$, $\check{V}_0$ and $\check{W}_0$. Then, chen2023sgmm proposed to use $\eta_0 = 1/\Psi_0(\kappa)$, where

equation[equation omitted — 256 chars of source]

$\kappa$ is a predetermined quantile level and $\left\lVert \cdot \right\rVert_S$ is the spectral norm in (ref).

Now, consider the following linear instrumental variables regression models:

enumerate[series=enum-model] • Independent: let $ y_k = x_k^\intercal \theta^* +5 \exp(z_{k,q}) (\nu_k +\varepsilon_k), $ where $\nu_k, \varepsilon_k \stackrel{\mathrm{iid}}{\sim} \mathrm{N}(0,1)$, $z_k \stackrel{\mathrm{iid}}{\sim} \mathrm{N}_q(0, \mathcal{V})$ with $\mathcal{V}_{i,j} = \rho^{|i-j|}$, $x_{k,1} = 0.1 \sum_{j=2}^p x_{k,j} +0.5 \sum_{j=p}^q z_{k,j} +\nu_k$, and $x_{k,j} = z_{k,j-1}$ for $j = 2, \ldots, p$. We exactly replicate a simulation model in chen2023sgmm by setting $\theta^* = (1, \ldots, 1)^\intercal$, $p = 5$, $q = 20$, and $\rho = 0.5$. • Dependent: let $ y_k = \theta_1^* y_{k-1} +\theta_2^* y_{k-2} +\phi_1^* \varepsilon_{k-1} +\phi_2^* \varepsilon_{k-2} +\varepsilon_k, $ where $\varepsilon_k \stackrel{\mathrm{iid}}{\sim} \mathrm{N}(0,1)$. We exactly replicate a simulation model in pierre2010gmm by setting $\theta^* = (1.4, -0.6)^\intercal$ and $\phi^* = (0.6, -0.3)^\intercal$. The selected instruments are $z_k = (y_{k-3}, \ldots, y_{k-6})^\intercal$.

We are interested in estimating $\theta^*$ at $N_b = 2000b$ for $b = 1, 2, \ldots, 100$ using

enumerate• $\mathsf{2SLS}$: the two-stage least squares as the offline benchmark and initial estimator for model (ref). The Bartlett kernel estimator is used for inference. • $\mathsf{GMM}$: the two-step $\mathsf{GMM}$ as the offline benchmark and initial estimator for model (ref). The Bartlett kernel estimator is used for inference and the weighting matrix. We also use $\tilde\theta_{b-1}$ as the initial value when we optimize for $\tilde\theta_b$, and $\tilde\theta_0 = (0, 0)^\intercal$. • $\mathsf{SGMM}$: for model (ref), we consider different initial learning rates selected at $\kappa \in \{0.5, 0.7, 0.9\}$. For model (ref), we only consider $\kappa=0.5$ but further try rlrv's rlrv recursive long-run variance estimator for both inference and the weighting matrix. For both models, we set $a = 0.501$ in the learning rate $\eta_k$. However, the stochastic $\mathsf{2SLS}$ algorithm is not used for warm up as in chen2023sgmm because it requires a randomly partitioned subsample with size that scales with $N_b$, which is not feasible in our online setting where $N_b$ is increasing. Similarly, we do not consider the multi-epoch approach in chen2023sgmm. • $\mathsf{OGMM}$: both welford1962recursive's welford1962recursive and rlrv's rlrv estimators are considered for inference and the weighting matrix. We further compare explicit and implicit updates with the stopping criteria in luo2022qif. Specifically, we stop when the number of Newton--Raphson iterations reaches $50$ or $\{ (\hat{V}_b^{(r)})^\intercal \hat{W} \hat{U}_b^{(r)} \}^\intercal \{ (\hat{V}_b^{(r)})^\intercal \hat{W} \hat{V}_b^{(r)} \}^{-1} (\hat{V}_b^{(r)})^\intercal \hat{W} \hat{U}_b^{(r)} < 10^{-6}$, where $\hat{V}_b^{(r)} = N_b^{-1} \{ N_{b-1} V'_{b-1} +\nabla G(\hat\theta_b^{(r-1)}; D_b) \}$, $\hat{U}_b^{(r)} = N_b^{-1} \{ N_{b-1} U'_{b-1} +N_{b-1} V'_{b-1} (\hat\theta_b^{(r-1)} -\hat\theta'_{b-1}) +G(\hat\theta_b^{(r-1)}; D_b) \}$, and $\hat\theta_b^{(r)} = \hat\theta_b^{(r-1)} -\{ (\hat{V}_b^{(r)})^\intercal \hat{W} \hat{V}_b^{(r)} \}^{-1} (\hat{V}_b^{(r)})^\intercal \hat{W} \hat{U}_b^{(r)}$.

figure[figure omitted — 337 chars of source]

figure[figure omitted — 274 chars of source]

In Figures (ref) & (ref), we compare mean squared errors of the point estimates, coverage rates of the $95\%$ confidence intervals for $\theta_1^*$, and computational time in seconds. Figure (ref) confirms that suboptimal $\sqrt{N_1}$-consistent $\hat\theta_1$ (such as $\mathsf{2SLS}$) is still sufficient for $\mathsf{OGMM}$ to be asymptotically optimal. Despite the fact that data are independent under model (ref), using a recursive long-run variance estimator for autocorrelation robustness does not affect the performance of $\mathsf{OGMM}$ much. On the other hand, $\mathsf{SGMM}$ is not sensitive to the choice of $\kappa$ here. Its mean squared errors relative to $\mathsf{2SLS}$ and coverage rates are similar to those reported in chen2023sgmm. Nevertheless, $\mathsf{SGMM}$ is not as efficient as $\mathsf{OGMM}$ in a moderately large sample ($N_b \le 2 \times 10^5$).

Figure (ref) verifies the asymptotic efficiency of $\mathsf{OGMM}$ relative to $\mathsf{GMM}$. It also shows that $\mathsf{OGMM}$ and $\mathsf{SGMM}$ greatly reduce the computational cost of $\mathsf{GMM}$. Since data are dependent under model (ref), recursive long-run variance estimation improves the coverage rates of $\mathsf{SGMM}$, which was originally designed for independent data. However, $\mathsf{OGMM}$ still outperforms $\mathsf{SGMM}$ when the same long-run variance estimator is used. It does not gain efficiency through implicit updates though because the moment is linear in $\theta$; see Remark (ref). Additional results with the same findings are deferred to the Supplement.

Online Sargan--Hansen test

In (ref), we observe that there is a first-order correction term in our online Sargan--Hansen test statistic because we are plugging in a sequence of estimates rather than a single estimate in the offline setting. On the other hand, there is no first-order correction term in (ref). To investigate the influence of this term, consider the following models:

enumerate[resume*=enum-model] • Independent: let $y_k = \theta_1^* x_k +\theta_2^* z_{k,1} +\varepsilon_k$ and $x_k = z_{k,1} +z_{k,2} +\nu_k$, where $(z_{k,1}, z_{k,2}, \nu_k, \varepsilon_k)^\intercal \stackrel{\mathrm{iid}}{\sim} \mathrm{N}_4(0, \mathcal{V})$ with $\mathcal{V}_{i,j} = \mathbb{1}_{i=j} +0.5 \mathbb{1}_{(i,j) \in \{(1,2),(2,1),(3,4),(4,3)\}}$. This is the simulation model in hall2000overident, and we consider $\theta_1^* = 1$ and $\theta_2^* \in \{0, 0.05, \ldots, 0.5\}$. • Dependent: same as model (ref) but $(z_{k,1}, z_{k,2}, \nu_k, \varepsilon_k)^\intercal$ follows a vector autoregressive model of order $1$ with mean $(0, \ldots, 0)^\intercal$, coefficient matrix $\mathrm{diag}(0.5, \ldots, 0.5)$ and Gaussian noise covariance matrix $\mathcal{V}$ in model (ref).

We estimate $\theta_1^*$ at $N_b = (100) 2^b$ for $b = 1, 2, \ldots, 5$ under the assumption that $\mathbb{E}\{ z_k(y_k -\theta_1^* x_k)\} = 0$. Clearly, misspecification happens when $\theta_2^* \ne 0$. The method settings are same as those for model (ref). We continue to use two-step $\mathsf{GMM}$ as the offline benchmark and initial estimator, and $\mathsf{SGMM}$ with $\kappa=0.5$ and $a=0.501$.

figure[figure omitted — 555 chars of source]

Figure (ref) reports the empirical size and size-adjusted power of different Sargan--Hansen tests for $\theta_2^* \in \{0, 0.2, 0.4\}$. Overall, the performance of $\mathsf{OGMM}$ is very encouraging. Although the batch size $n_b = N_{b-1}$ is increasing, the statistical efficiency of $\mathsf{OGMM}$ remains comparable to that of $\mathsf{GMM}$, which corroborates with our theory in Theorems (ref) and (ref). Recursive long-run variance estimation does not cause size distortion under model (ref) and allows autocorrelation-robust online testing under model (ref). On the other hand, $\mathsf{SGMM}$ suffers from considerable size distortion and exhibits non-monotonic size-adjusted power (with respect to the sample size) that seems counter-intuitive and cannot be explained by the variance estimator. This confirms the validity and necessity of the first-order correction term in Theorem (ref) when a sequence of estimates is plugged in. The results for other values of $\theta_2^*$ are similar and deferred to the Supplement.

Online quantile regression

As illustrated in Example (ref), online methods for quantile regression are often based on smoothing the indicator function in the quantile loss chen2019rindicator,sun2024rindicator,jiang2024runconditional. There is another class of online methods based on a different kind of smoothing jiang2022rdensity. However, we continue with Example (ref) to compare with chen2019rindicator because other estimators are based on implicit updates. Suppose observations are divided into intervals by their indices. Define $b_0 = 0$, $c_0 = -\infty$, and $c_{2k-1} = 2^{k-1} +1/2$ and $c_{2k} = 2^{k-1} +3/4$ for $k \ge 1$. Let $m$ be the memory constraint and $b_l = \lfloor m^{c_{l-1}} \rfloor +1$ be the first index in the $l$-th interval. chen2019rindicator elaborated that $\{b_l\}$ was chosen in this way because there would be no improvement of online $\mathsf{LEQR}$ after $n^2$ fresh observations if the previous estimate was based on $n$ observations. This is similar to our condition $n_i = o(N_{i-1}^2)$ because $n_i$ is the number of fresh observations and $N_{i-1}$ is previous sample size. However, apart from the minor difference in Example (ref), $\mathsf{OGMM}$ is methodologically different from $\mathsf{LEQR}$ in two ways. First, $\mathsf{OGMM}$ includes all data in the summary statistics, whereas $\mathsf{LEQR}$ mainly includes data from the previous and current intervals; see Algorithm 2 in chen2019rindicator. Second, $\mathsf{OGMM}$ may utilize the memory constraint of size $m$ so that $\hat\theta_{b-1}$ is updated frequently, whereas $\mathsf{LEQR}$ only updates the estimate that is used to compute summary statistics at the start of a new interval. Now, consider the following models:

enumerate[resume*=enum-model] • Independent: let $ y_k = x_k^\intercal \theta^* +\varepsilon_k, $ where $x_{k,1} = 1$, $x_{k,2}, \ldots, x_{k,p}$ follow a uniform distribution on $[0,1]$ with $\mathrm{Corr}(x_{k,i}, x_{k,j}) = 0.5^{|i-j|}$, and $\varepsilon_k \stackrel{\mathrm{iid}}{\sim} \mathrm{N}(0,1)$. We set $\theta^* = (1, \ldots, 1)^\intercal$ and $p=10$, which replicates a simulation model in chen2019rindicator. • Dependent: same as model (ref) but $(\varepsilon_k, x_{k,2}, \ldots, x_{k,p})^\intercal$ follows a vector autoregressive model of order $1$ with mean $(0, \ldots, 0)^\intercal$, coefficient matrix $\mathrm{diag}(0.5, \ldots, 0.5)$ and Gaussian noise covariance matrix $\mathcal{V}_{i,j} = 0.5^{|i-j|} \mathbb{1}_{i \ne 1} \mathbb{1}_{j \ne 1} +\mathbb{1}_{i = 1} \mathbb{1}_{j = 1}$ for $1 \le i,j \le p$.

We are interested in estimating $\theta_\tau^*$ at $N_b = 2000b$ for $b = 1, \ldots, 250$. Recall that $\theta_\tau^*$ can be computed by shifting $\varepsilon_k$ such that $\mathbb{P}(\varepsilon_k \le 0 \mid x_k) = \tau$ chen2019rindicator. For $\tau = 0.1$, the intercept $\theta_{\tau 1}^*$ in models (ref) and (ref) are approximately $-0.28$ and $-0.48$, respectively. The method settings are close to those for model (ref). The differences are that we use the default modified Barrodale and Roberts algorithm in the R-package quantreg koenker2005quant for initial estimation, $\mathsf{LEQR}$ as the online benchmark, and $\mathsf{SGMM}$ with $\kappa \in \{0.97, 0.98, 0.99\}$ and $a=0.501$. For the smoothing in Example (ref), we follow chen2019rindicator to use $H(x) = \{1/2 +(15/16)(x -2x^3/3 +x^5/5)\} \mathbb{1}_{|x| < 1} +\mathbb{1}_{x \ge 1}$ and $h_b = \sqrt{p/N_{b-1}}$, where $b$ is the current number of intervals ($\mathsf{LEQR}$) or batches ($\mathsf{SGMM}$/$\mathsf{OGMM}$).

figure[figure omitted — 391 chars of source]

In Figure (ref), we compare the median absolute errors of point estimates, the computational time in seconds, and the coverage rates of $95\%$ confidence intervals for the summed coefficient $J^\intercal \theta_\tau^*$, where $J = (1, \ldots, 1)^\intercal$ and $\tau=0.1$. The median absolute error is considered here because $\mathsf{SGMM}$ with $\kappa \in \{0.97, 0.98\}$ failed to initialize or gave unreasonable estimates in some replications. Figure (ref) illustrates that $\mathsf{OGMM}$'s methodological differences from $\mathsf{LEQR}$ are beneficial. The computational cost of $\mathsf{OGMM}$ is slightly higher due to recursive long-run variance estimation, but it enables autocorrelation-robust inference in model (ref). On the other hand, $\mathsf{SGMM}$ is very sensitive to the choice of $\kappa$ here because a lot of $\nabla g(\check\theta_0, x)$ in (ref) are zero or very small. It is also computationally inefficient because we can only apply the general algorithm instead of the one specialized for instrumental variables in chen2023sgmm. The latter one includes a Sherman--Morrison--Woodbury formula for $(\check{V}_k^\intercal \check{W}_k \check{V}_k)^{-1}$, which is not available in the general version.

Online anomaly detection

Recall from Examples (ref) and (ref) that there are two types of abnormality or structural instability caused by a change point in parameter value and a misspecified model, respectively. Now, we extend the experiments in (ref) by modifying the models there:

enumerate[resume*=enum-model] • Independent: let $y_k = (\theta_1^*+\theta_2^* \mathbb{1}_{k \in \Lambda_\text{I}}) x_k +\theta_2^* \mathbb{1}_{k \in \Lambda_\text{O}} z_{k,1} +\varepsilon_k$ and $x_k = z_{k,1} +z_{k,2} +\nu_k$, where $(z_{k,1}, z_{k,2}, \nu_k, \varepsilon_k)^\intercal \stackrel{\mathrm{iid}}{\sim} \mathrm{N}_4(0, \mathcal{V})$ with $\mathcal{V}_{i,j} = \mathbb{1}_{i=j} +0.5 \mathbb{1}_{(i,j) \in \{(1,2),(2,1),(3,4),(4,3)\}}$, $\Lambda_\text{I} = \{2001, \ldots, 2500, 6001, \ldots, 6500 \}$ and $\Lambda_\text{O} = \{4001, \ldots, 4500, 8001, \ldots, 8500 \}$. Same as model (ref), we consider $\theta_1^* = 1$ and $\theta_2^* \in \{0, 0.05, \ldots, 0.5\}$. • Dependent: same as model (ref) but $(z_{k,1}, z_{k,2}, \nu_k, \varepsilon_k)^\intercal$ follows the vector autoregressive model in model (ref).

We estimate $\theta_1^*$ at $N_b = 500 b$ for $b = 1, 2, \ldots, 20$ under the assumption that $\mathbb{E}\{ z_k(y_k -\theta_1^* x_k)\} = 0$. There are abnormalities caused by a change point in parameter value and a misspecified model for $b \in \{5,13\}$ and $b \in \{9,17\}$, respectively. To detect them, consider

enumerate• luo2022qif: the offline detection statistic in (ref) with $\mathsf{2SLS}$ and two-step $\mathsf{GMM}$ as the initial estimator and the restricted estimator in (ref), respectively. The Bartlett kernel estimator is used for inference and the weighting matrix. • $\mathsf{OGMM}$ I: the online detection statistic in (ref) with $\mathsf{2SLS}$ as the initial estimator. The $\mathsf{OGMM}$ summary statistics are based on $D_1$ only (initial) or updated with data batches where abnormality is not detected (cumulative). rlrv's rlrv estimator is used for inference and the weighting matrix. • $\mathsf{OGMM}$ O: same as $\mathsf{OGMM}$ I but (ref) is used. We expect that $\mathsf{OGMM}$ O should be sensitive to model misspecification only.

figure[figure omitted — 700 chars of source]

We compare the empirical rejection rates and size-adjusted rejection rates of different proposals. Figure (ref) reports the results for $\theta_2^* = 0.2$ and for $b \in \{5,9\}$. It confirms that $\mathsf{OGMM}$ is able to preserve the statistical properties when we replace offline statistics with online counterparts in our framework. For example, $\mathsf{OGMM}$ O only rejects when $b=9$ but not when $b=5$, which helps identify the two possible causes of structural instability. We can also use more normal data to improve the sensitivity of $\mathsf{OGMM}$ I without worrying about the computational cost. Under model (ref), there are $38\%$ (initial) and $22.6\%$ (cumulative) reductions in averaged time compared with luo2022qif. Under model (ref), the reductions change to $26.7\%$ (initial) and $10\%$ (cumulative) due to long-run variance estimation, but it allows autocorrelation-robust online testing. The results for other values of $\theta_2^*$ or $b$ are similar and deferred to the Supplement.

Applications

In (ref), we consider the rolling approach of stochastic volatility modelling pascalau2023sv and find that our $\mathsf{OGMM}$ can be a handy addition to a forecaster's toolbox. In (ref), we show that $\mathsf{OGMM}$ can be extended to the frequency domain and offers some alternative insights into the inertial sensor calibration problem.

Stochastic volatility modelling

Forecasting is a central topic in econometrics and statistics. With wider availability of high frequency data, practitioners are interested in updating models quickly to generate real-time prediction. However, traditional stochastic volatility models are usually trained with daily aggregated data. In light of it, pascalau2023sv proposed a rolling approach to incorporate intraday information. We consider different online estimation methods via $\mathsf{OGMM}$ here. Our data set consists of $7976478$ one-minute bar euro dollar rate (EUR/USD) from 2000 to 2023 (UTC-5), which are freely available on \href{https://www.histdata.com/}{HistData.com}. We exclude $1681$ observations in the weekend (UTC+2) and compute the close-to-close log 5-minute return $r_{t,h,m}$ for $m = 1,\ldots, 12$ in the $h$-th hour on day $t$. Define the rolling daily realized variance and realized quarticity \[ \mathrm{RV}_{t,h} = \sum_{i=h-23}^{h} \sum_{j=1}^{12} r_{t,i,j}^2 \quad \text{and} \quad \mathrm{RQ}_{t,h} = \frac{24 \times 12}{3} \sum_{i=h-23}^{h} \sum_{j=1}^{12} r_{t,i,j}^4, \] where $r_{t,i-\ell,j} = r_{t-1,i+24-\ell,j}$ if $i-\ell \in (-24, 0]$, and $r_{t,i+\ell,j} = r_{t+1,i-24+\ell,j}$ if $i+\ell \in (24, 48]$. We forecast the one-day-ahead daily realized variance $\mathrm{RV}_{t+1,24}$ using the following approaches with all available observations to date.

enumerate• Traditional: the quarticity-adjusted heterogeneous autoregressive model \begin{equation} \mathrm{RV}_{t,24} = \theta_0^* +(\theta_1^* +\theta_2^* \sqrt{\mathrm{RQ}_{t-1,24}}) \mathrm{RV}_{t-1,24} +\theta_3^* \sum_{\ell=1}^5 \frac{\mathrm{RV}_{t-\ell,24}}{5} +\theta_4^* \sum_{\ell=1}^{21} \frac{\mathrm{RV}_{t-\ell,24}}{21} +\varepsilon_{t,24} \end{equation} in bollerslev2016sv is estimated using ordinary least squares and updated daily. Following pascalau2023sv, the initial sample size is $1000$. • Rolling: the model in (ref) is modified to \begin{equation} \mathrm{RV}_{t,h} = \theta_0^* +(\theta_1^* +\theta_2^* \sqrt{\mathrm{RQ}_{t-1,h}}) \mathrm{RV}_{t-1,h} +\theta_3^* \sum_{\ell=1}^5 \frac{\mathrm{RV}_{t-\ell,h}}{5} +\theta_4^* \sum_{\ell=1}^{21} \frac{\mathrm{RV}_{t-\ell,h}}{21} +\varepsilon_{t,h} \end{equation} using the rolling approach pascalau2023sv. Therefore, the initial sample size becomes $1000 \times 24$. Other settings are same as Traditional and we focus on forecasting $\mathrm{RV}_{t+1,24}$ at day $t$'s end. • $\mathsf{OGMM}$: consider (ref) with different moment conditions. (LS) least squares; see Example (ref). (QR) quantile regression with $\tau = 0.5$; see (ref). (IV) instrumental variables regression with $1$, $\sum_{\ell=1}^5 \mathrm{RV}_{t-\ell,h}/5$, $\sum_{\ell=1}^{21} \mathrm{RV}_{t-\ell,h}/21$ and $\{ \sqrt{\mathrm{RQ}_{t-i,h}} \}_{i=k}^{21}$ as instruments, and $\mathsf{2SLS}$ as the initial estimator; see (ref). As bollerslev2016sv suggested that the measurement error in $\mathrm{RV}_{t-1,h}$ is most serious, we try to use $\{ \sqrt{\mathrm{RQ}_{t-i,h}} \}_{i=k}^{21}$ as instruments for terms that involve $\mathrm{RV}_{t-1,h}$. The weighting matrix is inverted from welford1962recursive's welford1962recursive estimator for (LS) and (QR), and rlrv's rlrv estimator for (IV). Other settings are same as Rolling. • $\mathsf{SGMM}$: same as $\mathsf{OGMM}$. The learning rate is chosen by (ref) with $\kappa=0.5$ and $a=0.501$.

table[table omitted — 1,020 chars of source]

Table (ref) reports the mean squared error and computation time relative to the traditional approach to update models and generate predictions. It confirms that intraday information can be used to deliver more accurate forecasts, which is consistent with the findings in pascalau2023sv. Furthermore, $\mathsf{OGMM}$ LS vastly improves the computational efficiency of the rolling approach by reducing its computing time by approximately 160 times. Meanwhile, $\mathsf{OGMM}$ QR and IV open the door to statistical refinements by further reducing the relative mean squared error from 0.997 to 0.994 and then to 0.96 (with $k=5$). $\mathsf{SGMM}$ is unable to achieve them as its mean squared error is higher than that of the traditional method and its computational cost is approximately 2.5 to 3 times that of $\mathsf{OGMM}$. Moreover, $\mathsf{SGMM}$ is affected by the choice of learning rate. In practice, $\mathsf{OGMM}$ allows us to conduct online estimation and inference easily given any moment condition. Users can maintain several sets of $\mathsf{OGMM}$ summary statistics and ensemble forecasts in real time. The efficiency and flexibility of $\mathsf{OGMM}$ are thus valuable in the analysis of large-scale streaming data.

Inertial sensor calibration

Inertial sensors are very common in modern vehicles and mobile robots. However, they may be corrupted by stochastic errors of complex structure, e.g.,

equation[equation omitted — 75 chars of source]

where $\{ y_t \}$ are the observed error signals, $z_t^{(i)} = \rho_i z_{t-1}^{(i)} +u_t^{(i)}$ follow latent first-order autoregressive processes, and $u_1^{(i)},u_2^{(i)},\ldots \stackrel{\mathrm{iid}}{\sim} \mathrm{N}(0, \sigma_i^2)$ and are independent across $i$ stebler2014gmwm. To estimate $\theta^* = (\rho_1, \sigma_1^2, \rho_2, \sigma_2^2, \sigma_3^2)^\intercal$, we can apply the generalized method of wavelet moments ($\mathsf{GMWM}$, guerrier2013gmwm). Specifically, let $w_{t,j}$ be the maximal overlap discrete wavelet transform of $y_t$ at scale $\tau_j = 2^{j-1}$ using the Haar filter. Then, the $j$-th element of the moment function is \[ w_{t,j}^2 -\sum_{i=1}^2 \frac{(0.5\tau_j -3\rho_i -0.5\tau_j \rho_i^2 +4 \rho_i^{1 +0.5\tau_j} -\rho_i^{1+\tau_j}) \sigma_i^2}{0.5 \tau_j^2 (1-\rho_i)^2 (1-\rho_i^2)} -\frac{\sigma_3^2}{\tau_j}, \quad \text{for} \quad j = 1, \ldots, q; \] see the R-package \href{https://github.com/SMAC-Group/gmwm}{gmwm} and Table 3 in guerrier2013gmwm. When $q > 5$, the Sargan--Hansen test can be used to assess the goodness of fit of (ref).

Now, consider the y-axis gyroscope data set available through imar.gyro in the R-package \href{https://github.com/SMAC-Group/imudata}{imudata}. 5760000 error signals were collected at 400 Hz in 4 hours. We divide them into $b=48$ equal batches (each of size 120000) and fit (ref) using $q = 10$ and

enumerate• $\mathsf{GMWM}$: the two-step $\mathsf{GMWM}$ implemented in the R-package \href{https://github.com/SMAC-Group/gmwm}{gmwm} with full data. The default first-step weighting matrix is diagonal, which may not be optimal. Therefore, parametric bootstraps of size $100$ are used for the second-step weighting matrix in estimation and the asymptotic variance in inference separately. • $\mathsf{OGMM}$: $\mathsf{OGMM}$ with normal data batches. Using the two-step $\mathsf{GMWM}$ as the initial estimator, we update $\mathsf{OGMM}$ summary statistics if (ref) and (ref) are not rejected; see the cumulative strategy in (ref).

table[table omitted — 864 chars of source]

Table (ref) reports the estimation results. For $\mathsf{GMWM}$, the Sargan–Hansen test suggests that there are abnormal data batches, or (ref) is a poorly fitted model. To investigate the latter possibility, we also estimate (\ref*{eq:sensor}$-$) $y_t = z_t^{(1)} +u_t^{(2)}$ and (\ref*{eq:sensor}$+$) $y_t = \sum_{i=1}^3 z_t^{(i)} +u_t^{(4)}$, which are considered in guerrier2013gmwm and stebler2014gmwm. The Sargan--Hansen test $p$-values are (\ref*{eq:sensor}$-$) $1.021 \times 10^{-14}$ and (\ref*{eq:sensor}$+$) $8.822 \times 10^{-10}$, which indicate worse fit. In contrast, applying $\mathsf{GMWM}$ and $\mathsf{OGMM}$ to the first data batch leads to $p$-values of $0.494$ and $0.173$, respectively. This evidence supports structural stability testing in $\mathsf{OGMM}$. Using (ref) and (ref), we find that $17$ batches of data (number 10, 11, 13, 14, 15, 16, 17, 19, 21, 24, 25, 29, 36, 39, 41, 45, 46) are normal, and most abnormalities are due to model misspecification because (ref) is rejected for $28$ batches of data. The resulting $\mathsf{OGMM}$ estimate of $\sigma_1^2$ is much smaller than $\mathsf{GMWM}$, while estimates of other parameters are closer. The $\mathsf{OGMM}$ $p$-value $0.107$ suggests better fit than $\mathsf{GMWM}$, although it is not adjusted for possible post-screening issues. In terms of total time (estimation and testing), (ref) requires solving an optimization problem for each data batch. Nevertheless, $\mathsf{OGMM}$ offers online estimates and remains faster than $\mathsf{GMWM}$ because it does not perform bootstrap for optimal estimation and inference. Overall, this application illustrates that the $\mathsf{OGMM}$ framework is very general and offers an alternative approach to estimation and inference for time series models formulated in wavelet domains.

Discussion

Time series data are inherently serially dependent and sequential. Motivated by the need to analyse large streaming time series data, we develop an online version of the classical $\mathsf{GMM}$, which is robust to weak serial dependence, preserves the asymptotic efficiency of offline $\mathsf{GMM}$, and vastly reduces the time complexity and memory requirements. Our methodological development covers $\mathsf{OGMM}$ point estimation, confidence region construction, over-identifying restrictions testing and anomaly detection with rigorous theoretical justification. Our numerical simulation and data illustration convincingly demonstrate the favourable performance of $\mathsf{OGMM}$-based estimation and inference as compared to several existing procedures in terms of computational time and statistical errors.

To conclude, we mention a few future research topics. Throughout this paper the dimension of the parameter is assumed fixed. It would be interesting to develop an online version of $\mathsf{GMM}$ that can accommodate high dimensionality of the parameter; see belloni2018hd for regularized $\mathsf{GMM}$ in the offline setting. Additionally, $\mathsf{GMM}$ has been extended to handle the estimation of a finite dimensional parameter when a nuisance function is present; see ackerberg2014semi and chen2015semi. Developing an online version of two-step $\mathsf{GMM}$ to deal with this semiparametric setting would be interesting. Finally, it is of interest to allow for time-varying parameter luo2023qif and/or local stationarity of streaming time series, as stationarity and time-invariant parameter might be too strong to be satisfied for a very long streaming time series.

Acknowledgments

Chan was supported in part by grants GRF-14306421 and GRF-14307922 provided by the Research Grants Council of HKSAR. Shao would like to acknowledge partial financial support from National Science Foundation DMS-2210002 and DMS-2412833.

\addcontentsline{toc}{section}{\refname}