Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
56,701 characters · 20 sections · 40 citation commands
ForeComp: An R Package for Comparing Predictive Accuracy Using Fixed-Smoothing Asymptotics
\onehalfspacing
Comparing the predictive accuracy of competing forecasts is a fundamental task in empirical economics and finance. The seminal test of DieboldMariano1995 (hereafter DM) provides a simple and widely used framework: given a sequence of loss differentials between two forecasts, the DM test assesses whether the expected loss differential is zero. The test statistic is straightforward to compute, requires minimal distributional assumptions, and is applicable to a broad range of loss functions.
Despite its simplicity, the DM test is well known to suffer from size distortions in finite samples, particularly when the evaluation sample size is small. The primary source of distortion is the estimation of the long-run variance of the loss differential. The original DM recommendation uses a rectangular kernel truncated at $h-1$ lags, where $h$ is the forecast horizon, exploiting the theoretical MA$(h-1)$ structure of optimal forecast errors. In practice, however, the loss differential may exhibit serial correlation beyond $h-1$ lags, for example when forecasts are suboptimal, when in-sample parameter estimation introduces additional dependence, or when models are misspecified. In short evaluation samples, empirically relevant bandwidth choices can change both the size of the test and the resulting rejection decision.
A growing literature has proposed alternative inference procedures that deliver more reliable size control, especially in small samples. These include the modified DM test of HarveyLeybourneNewbold1997, the use of positive-semidefinite kernels with data-dependent bandwidth selection NeweyWest1994, and fixed-smoothing approaches. Fixed-smoothing asymptotics, developed by KieferVogelsang2005 for the fixed-$b$ case and Sun2013 for the fixed-$m$ case, derive the limiting distribution of the test statistic under the assumption that the smoothing parameter does not vanish with sample size. The resulting reference distributions account for the estimation error in the long-run variance, improving size properties in short samples. LLSW2018 provide practical recommendations for fixed-smoothing inference and emphasize the size--power tradeoff inherent in bandwidth choice.
The ForeComp package for R Rlang consolidates these approaches into a single toolkit under a common interface. Table (ref) summarizes the available procedures, spanning classical DM inference, fixed-$b$ and fixed-$m$ methods, orthonormal-series long-run variance estimation, and clustering-based approaches. The main testing functions take a vector of loss differentials as input and return a test statistic and rejection decision (and $p$-values where available), making it easy to compare conclusions across standard and fixed-smoothing procedures. Beyond offering multiple tests, ForeComp provides a visual diagnostic, Plot_Tradeoff, that assesses bandwidth robustness by plotting the implied size distortion--power loss tradeoff across a grid of bandwidth values and indicating where the null is rejected. This diagnostic helps users distinguish robust findings from bandwidth-driven rejections.
We illustrate the package with three empirical applications using the Survey of Professional Forecasters, including replications of Stark2010 and CoroneoIacone2020 and a bandwidth sensitivity analysis. We also report a Monte Carlo study based on McCracken2019 that summarizes finite-sample size and size-corrected power across the main procedures implemented in ForeComp. The simulations show that traditional normal-approximation procedures can substantially over-reject in short samples, while fixed-smoothing methods deliver improved size control with competitive size-corrected power.
The remainder of this paper is organized as follows. Section (ref) establishes the testing framework. Section (ref) describes the testing procedures available in ForeComp. Section (ref) illustrates the common interface and the Plot_Tradeoff diagnostic. Sections (ref)--(ref) present three empirical applications, Section (ref) reports a Monte Carlo simulation study of size and power, and Section (ref) concludes. Complete simulation tables are collected in Appendix (ref).
Consider a scalar target variable $y_{t+h}$ for which two competing $h$-step-ahead forecasts, $\hat{y}^{(1)}_{t+h|t}$ and $\hat{y}^{(2)}_{t+h|t}$, are available. Let $L(\cdot)$ denote a loss function (e.g., squared error), and define the loss differential
where $P$ denotes the number of out-of-sample evaluation observations. A positive $d_{t+h}$ indicates that forecast 1 incurs a larger loss than forecast 2 at time $t+h$.
The null hypothesis of equal predictive ability (EPA) is
Under mild regularity conditions---stationarity and ergodicity of $\{d_{t+h}\}$, and the existence of a positive long-run variance $\sigma^2 = \sum_{j=-\infty}^{\infty} \gamma_j > 0$ where $\gamma_j = \mathrm{Cov}(d_{t+h},\, d_{t+h+j})$---the DM test statistic takes the form
where $\bar{d} = P^{-1}\sum_{t=1}^{P} d_{t+h}$ is the sample mean of the loss differentials and $\hat{\sigma}$ is a consistent estimator of the long-run standard deviation $\sigma$. The critical challenge lies in the construction of $\hat{\sigma}$.
The generic long-run variance estimator based on weighted autocovariances is
where $\hat{\gamma}_j = P^{-1}\sum_{t=1}^{P-j}(d_t - \bar{d})(d_{t+j} - \bar{d})$ is the sample autocovariance at lag $j$, $k(\cdot)$ is a kernel function, and $M$ is a bandwidth parameter. The choice of kernel, bandwidth, and asymptotic approximation jointly determine the finite-sample properties of the resulting test.
Table (ref) provides a summary of all testing procedures implemented in ForeComp. We organize the discussion into two groups: standard procedures that use normal or $t$-distribution critical values with small bandwidths (Section (ref)), and alternatives based on fixed-smoothing asymptotics or nonparametric methods (Section (ref)).
\paragraph{DM-R: Original DM test (dm.test.r).} The original recommendation of DieboldMariano1995 exploits the theoretical result that, under the null hypothesis with an optimal $h$-step-ahead forecast, the loss differential follows an MA$(h-1)$ process. This motivates using the rectangular kernel with truncation at $M = h-1$:
The test statistic $\mathrm{DM}_R = \sqrt{P}\,\bar{d}\,/\,\hat{\sigma}_{\mathrm{R}}$ is compared to standard normal critical values. When $h=1$ (one-step-ahead forecasts), the long-run variance reduces to the sample variance. This estimator is not guaranteed to be nonnegative.
\paragraph{DM-M: Modified DM test (dm.test.r.m).} HarveyLeybourneNewbold1997 proposed two finite-sample corrections. First, a bias correction factor:
Second, the test statistic is compared to $t_{P-1}$ critical values rather than standard normal, providing heavier tails that partially account for estimation uncertainty.
\paragraph{DM-NW: Bartlett kernel with normal approximation (dm.test.bt).} To avoid the risk of negative variance estimates inherent in the rectangular kernel, one may use the Bartlett (triangular) kernel:
which guarantees $\hat{\sigma}^2 \geq 0$. The default bandwidth follows NeweyWest1994:
Standard normal critical values are used. While the positive-semidefiniteness of the Bartlett kernel is advantageous, the relatively small NW bandwidth may fail to capture serial dependence beyond $h-1$ lags, leading to size distortions similar to DM-R.
\paragraph{DM-FB: Bartlett kernel with fixed-$b$ asymptotics (dm.test.bt.fb).} The fixed-$b$ approach of KieferVogelsang2005 uses the same Bartlett kernel but with a substantially larger bandwidth,
following the recommendation of LLSW2018. Their bandwidth rule is motivated as a practical default that approximately minimizes a loss function trading off size distortions against power loss for fixed-$b$ tests, rather than as an MSE-optimal choice for long-run variance estimation. In ForeComp, this LLSW rule is the default for dm.test.bt.fb when $M$ is not supplied. The key difference from DM-NW is the asymptotic theory: letting $b = M/P$ remain fixed as $P \to \infty$, the test statistic converges to a nonstandard distribution $\Phi_{\mathrm{BART}}(b)$ whose quantiles depend on $b$. For the Bartlett kernel, KieferVogelsang2005 provide the approximation:
for the 97.5% quantile (used in a two-sided 5% test). The larger bandwidth captures more serial dependence, while the adjusted critical values account for the inconsistency of the variance estimator, jointly producing much better size control.
\paragraph{DM-EWC: Equal-weighted cosine estimator (dm.test.ewc.fb).} The equal-weighted cosine (EWC) estimator of LLSW2018 is an orthonormal series estimator of the long-run variance:
with bandwidth $B = \lfloor 0.4\,P^{2/3}\rfloor$. In LLSW2018, this default is motivated by the same size--power tradeoff criterion used to select the Bartlett fixed-$b$ bandwidth. Under fixed-$b$ asymptotics, the test statistic follows a $t_B$ distribution Muller2004, making critical values readily available from standard tables.
\paragraph{DM-WPE: Weighted periodogram estimator with fixed-$m$ asymptotics (dm.test.wpe.fb).} CoroneoIacone2020 apply the weighted periodogram estimator (WPE) with the Daniell kernel to forecast comparison:
where $I(\lambda_j) = |{(2\pi P)^{-1/2}\sum_{t=1}^{P} d_t e^{-i\lambda_j t}}|^2$ is the periodogram and $m = \lfloor P^{1/3}\rfloor$ is the bandwidth. Under fixed-$m$ asymptotics Sun2013, the test statistic $\sqrt{P}\,\bar{d}\,/\,\hat{\sigma}_{\mathrm{WPE}}$ follows a $t_{2m}$ distribution.
\paragraph{DM-IM: Ibragimov--M\"uller test (dm.test.im).} IbragimovMuller2010 propose a clustering approach that avoids explicit long-run variance estimation. The evaluation sample is divided into $q$ nonoverlapping blocks. The block means $\bar{d}_1, \ldots, \bar{d}_q$ are treated as $q$ independent observations, and a standard $t$-test is applied:
where $\bar{m} = q^{-1}\sum_{j=1}^{q}\bar{d}_j$. The critical values come from the $t_{q-1}$ distribution. The approach is simple and robust but may lack power due to the small effective sample size $q$.
The ForeComp package can be installed from the package source:
All testing functions in ForeComp accept a numeric vector d of loss differentials as their first argument and return a list with elements \$rej (logical rejection decision), \$stat (test statistic), and \$pval ($p$-value, where available). The significance level is controlled by the argument cl (default 0.05).
A typical workflow proceeds in two steps. As a running example, we compare the SPF consensus nowcast ($h = 0$) of real output growth against the no-change benchmark over 2007:Q1--2016:Q4 ($T = 40$). First, compute the loss differential from the forecast errors:
Second, apply the desired test:
Table (ref) summarizes the key arguments for each function.
The common interface makes it straightforward to compare results across methods. Continuing the running example:
The standard tests (DM-R, DM-M, DM-NW) all reject the null at 5%, while every fixed-smoothing alternative (DM-FB, DM-EWC, DM-WPE, DM-IM) does not. This divergence suggests that the rejections by the standard tests may be driven by size distortion rather than by strong evidence against equal predictive ability---a pattern consistent with the Monte Carlo findings in Section (ref), which document substantial over-rejection by DM-R and DM-NW in small samples. When such a divergence arises, we recommend favoring the fixed-smoothing results, which are designed to maintain reliable size control in finite samples.
The WCE-B test (fixed-$b$ Bartlett) requires the practitioner to choose a bandwidth parameter $M$. In CoroneoIacone2020, a common baseline is $M = \lfloor T^{1/2} \rfloor$ (“CI baseline”); the authors justify this choice by noting that it performed well in a preliminary Monte Carlo study (see their Supporting Information Appendix). In ForeComp, if the user does not supply $M$, dm.test.bt.fb defaults to the LLSW rule $M = \lceil 1.3\sqrt{T}\rceil$ (“package default”). This default is motivated by LLSW2018's loss-minimization criterion that trades off size distortions against power loss under fixed-$b$ inference. In practice, the rejection decision can be sensitive to the bandwidth choice: a test that rejects at one bandwidth may fail to reject at a slightly different $M$, or vice versa.
The Plot_Tradeoff function provides a visual diagnostic for this sensitivity. Given two sets of forecasts and a realized outcome series, it computes the WCE-B test statistic at each bandwidth $M$ in a user-specified grid (or a default grid centered around the package default), and for each $M$ it estimates the size distortion and maximum power loss via simulation. The size distortion at bandwidth $M$ is defined as the difference between the empirical rejection rate under the null and the nominal level (5%), where the null distribution is approximated by fitting an ARIMA model to the demeaned loss differential and simulating from it. The maximum power loss is the largest gap between the oracle power envelope (the power of a test with known long-run variance) and the size-corrected power of the WCE-B test, maximized over a grid of local alternatives. The green marker highlights the package default bandwidth. The function returns a list containing a ggplot2 plot object and the underlying data.
As an illustration, consider testing whether the SPF nowcast of real output growth outperforms the no-change benchmark in the subsample 2007:Q1--2016:Q4 ($T = 40$):
Figure (ref) shows the resulting tradeoff plot. Each marker on the curve corresponds to a specific bandwidth value $M$, annotated next to the point. Red crosses ($\times$) indicate bandwidths at which the WCE-B test rejects $H_0$ at the 5% level; red circles ($\bullet$) indicate bandwidths at which it does not reject. The green marker highlights the package default $M = \lceil 1.3\sqrt{T} \rceil = 9$. As $M$ increases, the size distortion generally decreases (the test becomes more conservative) while the maximum power loss increases (the test loses ability to detect alternatives)---producing the characteristic tradeoff curve. In this example, only the smallest bandwidths ($M = 1, 2, 3$) reject the null; at the package default $M = 9$ and all larger bandwidths, the test does not reject. The non-rejection is robust across most of the frontier, and the figure makes it transparent how the practitioner could deviate from the default, moving to smaller $M$ to favor power or to larger $M$ to favor size control, while seeing exactly where the rejection decision changes. Application 3 presents additional examples using SPF data.
Our first application replicates selected results from Stark2010, who evaluates the predictive accuracy of the Survey of Professional Forecasters (SPF) for U.S.\ real output growth. The SPF, maintained by the Federal Reserve Bank of Philadelphia, provides median forecasts at horizons from the current quarter ($h=0$) through four quarters ahead ($h=4$).
Stark2010 compares SPF median forecasts against three benchmark models---no-change (NC), indirect autoregression (IAR), and direct autoregression (DAR)---using RMSE and Diebold--Mariano tests with a Bartlett kernel. The evaluation period is 1985:Q1--2007:Q4. A distinctive feature of this analysis is the use of multiple realization vintages (initial release through latest available) to assess the sensitivity of forecast evaluation to data revisions.
Using the ForeComp package data and the dm.test.bt function, we replicate Table 1 of Stark2010. The data are loaded from the package:
For each realization vintage (Realiz1 through Realiz5) and each forecast horizon, we compute the squared forecast errors and apply the DM test with a Bartlett kernel and truncation $M = h + 4$, following the paper's specification:
Table (ref) reproduces Panels 1 and 2 of Stark2010 Table 1. Panel 1 uses the initial-release realization; Panel 2 uses the realization available one quarter after initial release. We also replicate the remaining panels (not shown): Panels 1--4 match the original to the third decimal place, while Panel 5 (latest vintage) differs slightly because our data incorporate revisions beyond the 2010:Q2 vintage used in the original paper.
The results show that the SPF consistently outperforms the no-change benchmark (ratios well below 1 with small $p$-values), but performs similarly to the autoregressive benchmarks at longer horizons (ratios near 1 with large $p$-values). The RMSE levels increase across panels as the realization vintage moves further from initial release, reflecting the accumulation of data revisions. This application demonstrates both the direct usability of the package functions and the importance of data vintage choices in real-time forecast evaluation.
Our second application replicates CoroneoIacone2020, who demonstrate that standard DM tests can yield spuriously significant results in small evaluation samples, and that fixed-smoothing asymptotics provides a corrective.
CoroneoIacone2020 evaluate the predictive accuracy of the SPF and the ECB Survey of Professional Forecasters against a random walk benchmark. They apply three variants of the DM test:
The evaluation covers four U.S.\ macroeconomic variables (real output growth, inflation, unemployment, and the 3-month Treasury bill rate) over 1987:Q1--2016:Q4 ($T = 120$) and three 10-year subsamples ($T = 40$ each), plus ECB SPF forecasts of GDP growth and HICP inflation over 2006:Q1--2016:Q4 ($T = 44$).
A key methodological choice is the handling of missing values. Observations where the forecast error is undefined (due to \#N/A entries in the SPF data) are set to zero rather than dropped, preserving the sample sizes reported in the paper. Additionally, only the first-release realization (Realiz1) is used for all horizons---the Realiz1--Realiz5 columns represent different data vintages, not different forecast horizons.
Table (ref) reproduces Table 1 of CoroneoIacone2020 for real output growth. The remaining four tables---GNP/GDP inflation, unemployment, the 3-month Treasury bill, and the ECB SPF---are reported in Appendix (ref) (Tables (ref)--(ref)). Our replication matches all 258 cells across all five tables exactly at two decimal places, confirming that the ForeComp implementations are consistent with the original analysis.
The results illustrate the practical importance of fixed-smoothing asymptotics. For real output growth, the standard normal-approximation DM statistic (WCE-DM) and the fixed-smoothing procedures (WCE-B and WPE-D) typically deliver the same qualitative inference, especially at conventional significance levels. There are, however, a few instances where WCE-DM rejects but the fixed-smoothing procedures do not. For example, in the 1997:Q1--2006:Q4 subsample at horizon $h = 4$, WCE-DM equals 1.84 and would reject at the 10% level under the normal approximation, while both WCE-B and WPE-D fail to reject at 10%. Such disagreements are consistent with the tendency of normal-approximation DM tests to over-reject in short samples, as documented in our Monte Carlo results in Section (ref).
In Application 2, we follow CoroneoIacone2020 and report WCE-B results under the CI baseline bandwidth $M = \lfloor T^{1/2} \rfloor$. In ForeComp, however, the default bandwidth for dm.test.bt.fb is the LLSW rule $M = \lceil 1.3\sqrt{T}\rceil$. Since the rejection decision may depend on this choice in small samples, the Plot_Tradeoff function allows practitioners to visualize how the test conclusion varies across a range of bandwidths. We apply this diagnostic to three cases drawn from the SPF evaluation in Application 2, comparing the SPF consensus forecast against a na\"ive (no-change) benchmark under squared-error loss.
\paragraph{Real output growth, $h = 0$, 2007:Q1--2016:Q4 ($T = 40$).} Figure (ref) displays the size--power tradeoff for the WCE-B test applied to the SPF nowcast of real output growth in the final subsample. The test rejects the null only at the smallest bandwidths ($M = 1, 2, 3$, marked with crosses); for all remaining bandwidths, including the package default $M = 9$ (green marker), the test does not reject (circles). The non-rejection at the default bandwidth is robust: the decision does not change as $M$ increases beyond 9.
The default $M = 9$ sits on the tradeoff frontier. Moving to a slightly smaller $M$ would reduce the maximum power loss but at the cost of increased size distortion; importantly, this shift does not change the rejection decision. Conversely, moving to a slightly larger $M$ would reduce size distortion further, at the cost of additional power loss, again without changing the conclusion.
This figure illustrates a key benefit of the Plot_Tradeoff diagnostic: it allows the practitioner to navigate the tradeoff and deviate from the default bandwidth in a principled way. Depending on the practitioner's preference---whether to prioritize size control (by moving to larger $M$) or to preserve power (by moving to smaller $M$)---the figure makes the tradeoff explicit and enables an informed choice. In this example, the non-rejection is stable across most of the frontier, giving the practitioner confidence in the conclusion regardless of bandwidth preference.
\paragraph{Unemployment, $h = 4$: robust non-rejection and robust rejection ($T = 40$).} Figure (ref) displays the tradeoff plots for two subsamples of the unemployment forecast at the 4-quarter horizon. Panel (a) illustrates robust non-rejection (except at the smallest bandwidth), while panel (b) illustrates robust rejection across the bandwidth grid.
Panel (a) shows the middle subsample (1997:Q1--2006:Q4). The WCE-B test fails to reject the null at virtually every bandwidth; only at $M = 1$ does the test reject. At that bandwidth, the Bartlett kernel collapses to a simple variance estimator and the resulting test has severe size distortions. At the default $M = 9$ and all larger bandwidths, every marker is a circle (H0 not rejected).
Panel (b) shows the earlier subsample (1987:Q1--1996:Q4). Here the WCE-B test rejects at every bandwidth in the grid---all markers are crosses---indicating that the SPF forecast significantly outperforms the na\"ive benchmark in this subsample. Since the rejection is robust across bandwidths, the practitioner can be confident that the conclusion is not an artifact of bandwidth choice.
One difference worth highlighting is the local shape of the tradeoff frontier around the default bandwidth (green marker). Relative to panel (a), panel (b) features a steeper “cliff”: moving to slightly smaller $M$ reduces the maximum power loss (i.e., improves power) at negligible cost in terms of size distortion. In this example such a change would not overturn the test decision, but the visualization makes the practitioner's size--power tradeoff transparent and underscores that the most attractive bandwidth adjustment can be case-by-case.
These three examples illustrate the range of outcomes that Plot_Tradeoff can reveal. When the rejection decision changes across the bandwidth grid (Figure (ref)), the test conclusion is fragile, and the practitioner should interpret the result with caution or consider the tradeoff between size control and power in choosing $M$. When the decision is stable across the entire frontier (Figure (ref)), either rejecting or not rejecting, the bandwidth choice is immaterial and the test provides a clear signal.
To systematically evaluate the finite-sample properties of the testing procedures in ForeComp, we conduct a Monte Carlo study based on the simulation design of McCracken2019. We consider two data-generating processes (DGPs) that differ in the serial correlation structure of the loss differential, and we examine both empirical size and size-corrected power across a wide range of sample sizes and forecast horizons.
Both DGPs generate data under the null hypothesis of equal predictive ability, where a zero-forecast (Model 1: $\hat{y}^{(1)}_{t+h|t} = 0$) is compared against a rolling-mean forecast (Model 2: $\hat{y}^{(2)}_{t+h|t} = \widetilde{R}^{-1}\sum_{s=t-\widetilde{R}+1}^{t} y_s$) using squared-error loss. The parameters $R$ and $\widetilde{R}$ govern, respectively, the serial correlation structure and the rolling estimation window.
\paragraph{Unconditional-Rolling (UCR) DGP.} The target variable is generated as an MA$(h-1)$ process:
where $\theta_k = 0.5^k$ and $\mu$ is chosen so that the two models have equal expected squared-error loss. The resulting loss differential $d_{t+h} = e_{1,t+h}^2 - e_{2,t+h}^2$ is serially correlated up to displacement $R - 1$, which can far exceed $h - 1$. This DGP is challenging for tests that assume MA$(h-1)$ structure (DM-R, DM-M), because the loss differential's actual serial dependence extends well beyond the truncation lag.
\paragraph{Conditional-Rolling (CR) DGP.} The target variable is generated as an ARMA process:
with 10{,}000 burn-in observations to reach stationarity. Under this DGP, the loss differential is serially correlated only up to displacement $h - 1$. The CR design is therefore favorable to the rectangular-kernel methods (DM-R, DM-M), whose truncation at $h - 1$ lags correctly captures the dependence structure.
For each DGP, we vary four parameters:
This yields $3 \times 4 \times 4 \times 5 = 240$ parameter combinations per DGP. Each combination is replicated 5{,}000 times. At each replication, we apply nine tests at the 5% nominal level: DM-R, DM-M, DM-NW, DM-NW-L, DM-FB, DM-EWC, and DM-IM with $q \in \{2, 5, 10\}$. DM-NW uses the NeweyWest1994 plug-in bandwidth $M = \lceil 4(P/100)^{2/9}\rceil$ with normal critical values. DM-NW-L uses the larger LLSW bandwidth $M = \lceil 1.3\sqrt{P}\rceil$ but retains normal critical values, isolating the effect of the bandwidth increase from the change in critical values. For the fixed-smoothing procedures DM-FB and DM-EWC, we use the package default bandwidth choices: DM-FB sets $M = \lceil 1.3\sqrt{P}\rceil$ and DM-EWC sets $B = \lfloor 0.4 P^{2/3}\rfloor$, both following LLSW2018.
The parameters $R$ and $\widetilde{R}$ play distinct roles. The serial correlation range $R$ is a feature of the DGP that determines the extent of dependence in the loss differential: under the UCR design, the loss differential is autocorrelated up to displacement $R - 1$, which can far exceed the forecast horizon $h$. The rolling window $\widetilde{R}$ is the in-sample estimation window used by the rolling-mean forecaster. When $R = \widetilde{R}$, the DGP is calibrated so that the two forecasters have equal expected loss---i.e., the null hypothesis holds exactly. When $R \neq \widetilde{R}$, the null is violated and one forecaster dominates, providing a setting for measuring power.
The ratio $P / R$ is also informative. When $P$ is small relative to $R$---as often arises in macroeconomic forecast evaluation, where evaluation samples are short---the serial correlation in the loss differential is large relative to the sample, and standard tests are most vulnerable to size distortion. Conversely, when $P$ is large (e.g., $P = 1{,}000$), all methods tend to be well-sized because the law of large numbers dominates.
The empirical size of a test is the fraction of the 5{,}000 replications in which the null hypothesis is rejected when $R = \widetilde{R}$ (i.e., under $H_0$). A well-calibrated test at the 5% nominal level should yield empirical rejection frequencies close to 0.05.
Table (ref) reports empirical size by forecast horizon $h$ for two evaluation sample sizes, $P \in \{75, 1{,}000\}$, in both DGPs. Each entry averages over the four on-diagonal values $R = \widetilde{R} \in \{25, 75, 125, 175\}$.
Under the UCR DGP (Panel A), size distortion is concentrated in the smaller evaluation sample. The standard procedures DM-R, DM-M, and DM-NW over-reject, with the problem most pronounced at longer horizons. For example, at $P = 75$ and $h = 12$, DM-R rejects 0.160 of the time and DM-NW rejects 0.127, far above the nominal 0.05. By contrast, the fixed-smoothing methods DM-FB and DM-EWC remain close to nominal size across horizons and sample sizes. The IM test is well sized for $q = 2$ and becomes mildly oversized as the number of blocks increases.
The DM-NW-L row isolates the effect of simply increasing the bandwidth without adjusting the critical values. At $P = 75$, DM-NW-L over-rejects at rates of 0.099--0.118, worse than DM-NW at short horizons ($h = 1, 3$) despite using the same LLSW bandwidth as DM-FB. The large bandwidth reduces bias in the long-run variance estimate but introduces substantial estimation noise; normal critical values, which do not account for this noise, lead to even larger size distortions. By contrast, DM-FB pairs the same bandwidth with fixed-$b$ critical values that widen the rejection region to account for estimation uncertainty, yielding size close to 0.05. At $P = 1{,}000$, DM-NW-L approaches the nominal level (0.050--0.052) as the estimation noise diminishes, whereas DM-FB is mildly conservative (0.038--0.040). This comparison demonstrates that the improved size control of the LLSW2018 recommendation comes from the combination of a large bandwidth and fixed-$b$ critical values, not from the bandwidth increase alone.
Under the CR DGP (Panel B), where the dependence structure is favorable to the rectangular-kernel truncation at $h-1$, size distortions are smaller but the same message holds in short samples. At $P = 75$ and $h = 12$, DM-R and DM-NW still reject at rates of 0.155 and 0.116, while fixed-smoothing procedures are much closer to 0.05. DM-NW-L again over-rejects (0.110--0.119), confirming that the large bandwidth alone does not deliver reliable size control. For DM-IM, larger block counts lead to over-rejection in the large sample, for example DM-IM($q{=}5$) rejects 0.093 at $P = 1{,}000$ and $h = 3$.
Raw rejection frequencies under $H_1$ (i.e., when $\widetilde{R} \neq R$) confound the test's genuine ability to detect deviations from the null with its tendency to over-reject under $H_0$. To isolate the former, we compute size-corrected power. For each on-diagonal configuration $(R = \widetilde{R}, P, h)$, we extract the 95th percentile of $|\text{DM}|$ across the 5{,}000 replications as the size-corrected critical value:
We then recompute the rejection frequency for all off-diagonal configurations ($\widetilde{R} \neq R$) using this critical value instead of the asymptotic one:
Because the critical value $c^*_{0.95}$ is calibrated to deliver exactly 5% rejection under $H_0$, the size-corrected power strips away size distortion and measures only the test's ability to discriminate between the null and alternative.
Table (ref) reports the size-corrected power under the UCR DGP for two representative scenarios that hold $P$ and $\widetilde{R}$ fixed. In each panel, the rolling window is fixed at $\widetilde{R} = 25$ and the power is averaged over the three off-diagonal values $R \in \{75, 125, 175\}$. Panel A uses $P = 75$ (a small evaluation sample typical of macroeconomic forecast comparisons), while Panel B uses $P = 1{,}000$ (a large sample in which all tests have substantial power). We report DM-IM with $q \in \{2, 5, 10\}$ to examine how the number of blocks affects the size--power trade-off.
Table (ref) shows that, after correcting for size, fixed-smoothing methods do not sacrifice power relative to traditional normal-approximation procedures. In the small-sample case (Panel A, $P = 75$), the fixed-$b$ Bartlett test (DM-FB) and the EWC-based test (DM-EWC) deliver size-corrected power that is comparable to, and often higher than, DM-R and DM-NW (for example, at $h = 1$, 0.127 and 0.121 versus 0.086 and 0.104). In the large-sample case (Panel B, $P = 1{,}000$), all kernel-based methods have high power, with DM-FB and DM-EWC matching or slightly exceeding DM-R. Notably, DM-NW-L and DM-FB have identical size-corrected power in every cell, which is expected: both use the same bandwidth and hence the same test statistic, so size correction equalizes them. The difference between the two lies entirely in the choice of critical values, which affects only uncorrected size. Thus, in our designs, the improved size control of fixed-smoothing inference does not come at a meaningful power cost. As expected, the IM test gains power as the number of blocks increases, and power declines at longer horizons.
The ForeComp package provides an R toolkit for comparing predictive accuracy using Diebold--Mariano type tests under a common interface. It includes both standard and fixed-smoothing procedures and the Plot_Tradeoff diagnostic for bandwidth sensitivity. Across three Survey of Professional Forecasters applications and Monte Carlo experiments, we find that standard normal-approximation tests can substantially over-reject in short samples, while fixed-smoothing methods offer improved size control with competitive size-corrected power. We recommend reporting fixed-smoothing results and using Plot_Tradeoff to assess bandwidth robustness.