Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
39,545 characters · 12 sections · 81 citation commands
W-Transformers : A Wavelet-based Transformer Framework for Univariate Time Series Forecasting
Forecasting the future movement and value of time series is a key component of formulating effective strategies in most business, industrial fields, and medical domains, among many others hyndman2018forecasting, hyndman2008forecasting. Specific applications include forecasting epidemiological cases (including Covid-19) chakraborty2020real, panja2022epicasting, stock price prediction Mehtab_2020, weather forecasts REN2021100178, web traffic forecasting Casado-Vara2021WebTraffic and others applied fields vautard2001earth,qin2019wind. Over the last decades, many techniques have evolved in time series forecasting literature. Statistics models (such as autoregressive integrated moving average (ARIMA) box2015time, state-space exponential smoothing (ETS) hyndman2008forecasting, and self-excited threshold autoregressive (SETAR) tong2009threshold models) have been intensively used in several applied domains due to their easy interpretability. However, they tend to fail when highly complex situations arise (for example, nonlinear and nonstationary time series) and require expertise in manually selecting trends, seasonality, and other components. On the other hand, machine learning methods such as LightGBM ke2017GBM and autoregressive neural networks (ARNN) faraway1998time can leverage the ground truth data to learn the trends and patterns in an automated manner while sometimes failing when there is long-term dependence on the observed time series data.
Current progress in deep learning has brought us deep autoregressive (DeepAR) salinas2020deepar models, temporal convolutional networks (TCN) bai2018TCN and many others petropoulos2022forecasting, lim2021time which can deal with nonlinear and non stationary time series. Time series data have also been modeled using Recurrent Neural Network (RNN) Hewamalage2019RNN in an autoregressive way that can tackle the irregularity of the time series. However, due to the gradient vanishing and exploding problem, RNNs are challenging to train. The problems continue to exist despite the introduction of numerous versions, such as long short-term memory (LSTM) Hochreiter1997LSTM and gated recurrent unit (GRU) cho2018GRconv that often outperform standard deep models. However, practical forecasting applications frequently contain both long and short-term recurrent patterns, and RNN-based methods sometimes struggle to capture long-term dependencies.
The introduction of Transformers in deep learning vaswani2017attention has brought great interests for researchers in natural language processing wolf2020transformers, computer vision dosovitskiy2020image, and other disciplines wen2022transformers. Transformers have shown great success in modeling and extrapolating the long-range dependencies and interactions in temporal data; hence they are pertinent in time series forecasting godahewa2021monash. Among various applications of Transformers for time series forecasting, some seminal papers are: probabilistic Transformer for multivariate time series representation learning zerveas2021transformer, tang2021probabilistic, deep transformer models for epidemic forecasting wu2020deep and many others wen2022transformers. The multi-headed attention mechanism of Transformers makes them particularly suitable for time series data analysis: (a) they concurrently represent each input sequence element by considering its context, (b) while multiple attention heads can consider different representation subspaces, i.e., multiple aspects of relevance between input elements -- for time series, this, for example, may correspond to multiple periodicities in the signal zerveas2021transformer. Other applications of Transformers include spatio-temporal forecasting in which, introducing a temporal Transformer block to capture temporal dependencies, it is also essential to design a spatial Transformer block wen2022transformers. Transformers are also proven effective in various time series classification tasks due to their superior capability in capturing long-term dependency feng2022multi. Compared to RNN and other neural network structures, the advantage of Transformers is that they easily enable parallel processing and reduce computational time since they do not involve recurrent behavior. A remaining challenge within Transformers is how to effectively model long-range and short-range temporal dependency and capture nonstationarity and nonlinearity simultaneously for forecasting time series data.
So far, many sophisticated decomposition techniques have shown strong competitiveness in time series pre-processing tasks, such as Fourier transformation sneddon1995fourier, fast Fourier transformation cochran1967fast, and Wavelet decomposition percival2000wavelet to cite a few. These methods of decomposing the complex time series into simpler parts have often led to satisfactory results. Among them, Wavelet decomposition, a mathematical tool that can reveal information within the signals in both the time and scale (frequency) domains, overcomes the basic drawback of Fourier analysis chakraborty2020real. Most of the previous applications of wavelets have used discrete wavelet transform (DWT) and during the task of forecasting, it has been combined with statistical or machine learning algorithm liu2013forecasting, boto2010wavelet. The limitation of DWT on the signal length made us use the Maximal Overlap discrete transform (MODWT) algorithm which counters this limitation but has the same abilities percival1997analysis. In fact, MODWT preserves the number of coefficients of the original time series for all the levels in the decomposition. Wavelet-ARIMA (WARIMA) aminghafari2007forecasting and Wavelet neural network panja2022epicasting are some of the hybrid algorithms that combine MODWT algorithm and statistical or machine learning models. Those hybrid approaches have been used in several practical applications, for example, epidemics panja2022epicasting, hydrology sang2013review, traffic volumes boto2010wavelet, and wind speed liu2013forecasting forecasting problems.
Motivated by the above observations, we propose a novel framework combining the MODWT algorithm with Transformers to effectively model the long-range and short-range temporal dependency and capture nonstationarity and nonlinearity simultaneously of the time series. Incorporating wavelets within Transformers will enable the model to capture all the irregularities of the time series by having a more detailed decomposition of low and high-frequency coefficients to separate the signal from the noise. Furthermore, scaling the wavelet used on the signal will reveal more about the frequency or the time. In fact, stretching the wavelet will give more information about the frequency while losing some time localization. Similarly, squeezing the wavelet will result in losing a part of the frequency domain while better preserving time location. Given those observations, our proposed W-Transformers is expected to generate more accurate and stable out-of-sample forecasts for long-term horizons for nonstationary time series data. Our main contributions to this paper can be described as follows :
This section discusses the mathematical foundation of the wavelet decomposition to be used in our proposed W-Transformers model. For more details, one may refer to percival2000wavelet.
Wavelet is a wave-like oscillation localized in time. It can capture the time and the frequency of a signal, contrary to Fourier Transform, which only captures the frequency of the signal. There are two different kinds of wavelets: (a) Continuous wavelet transforms (CWT), which applies an infinite number of wavelet (i.e., every possible wavelet) and (b) Discrete wavelet transform (DWT), where a finite number of wavelets in specific time and location are applied. In our work, we focus on DWT that represents a signal using an orthonormal basis representation, widely used in hydrology sang2013review, epidemics panja2022epicasting, and geophysics grinsted2004application to name a few. This decomposition will give us a set of time series where each time series has a coefficient describing the evolution in time of the signal in a specific frequency band. The DWT uses a discrete dyadic (octave) grid for scale parameter $i$ and shift parameter $k$, and the equation for forward DWT is as follows percival2000wavelet: $$ c(i, k)=\sum_{t} f(t) \Psi_{i, k}^{*}(t), \text { where } \Psi_{i, k}^{*}(t)=2^{\frac{i}{2}} \Psi\left(2^{i} t-k\right). $$ The mathematical equation for inverse DWT is as follows: $$ f(t)=\sum_{k} \sum_{i} c(i, k) \Psi_{i, k}(t), $$ where $f(\cdot)$ is a function and $\Psi$ the mother wavelet percival2000wavelet.
Since DWT restricts the sample size to be exactly a power of $2$, a modified version of the DWT, namely maximal overlapping discrete wavelet transformations (MODWT), is used for decomposing arbitrary time series percival1997analysis. Both MODWT and DWT can handle multi-resolution analysis, which is a scaled-based additive decomposition, but in the case of the MODWT, it also has the capacity to catch the circular shift of the signal. The shift-invariance property of MODWT is an added advantage over DWT in its time series applications. Below, we define the detail coefficients $(\Tilde{s}_{j,l})$ and scaling coefficients $(\Tilde{s}_{j,l})$ that are produced by an MODWT algorithm with Haar filter (most commonly used wavelet filter in time series literature) zhu2014modwt. $$ \Tilde{d}_{j,l} = {d_{j,l}}/2^{\frac{j}{2}} \; \; \text{and} \; \; {\Tilde{s}_{j,l}} = {s_{j,l}}/{2^{\frac{j}{2}}},$$ where $d_{j,l}$ and $s_{j,l}$ are wavelet and scaling filters of DWT algorithm. The coefficient of level $j$ $(l = 0,1,\ldots,L-1)$ then convolve the original time series $\{Y_t: t=0,1,\ldots,N-1\}$ and the MODWT pyramid algorithm generates the MODWT detail coefficients $(\Tilde{D}_{j,t})$ and scaling coefficients $(\Tilde{S}_{j,t})$ as follows percival2000wavelet: $$\Tilde{D}_{j,t}=\sum_{l=0}^{L_{j}-1}\Tilde{d}_{j,l}Y_{(t-l) mod N} \; \; \text{and} \; \; \Tilde{S}_{j,t}=\sum_{l=0}^{L_{j}-1}\Tilde{s}_{j,l}Y_{(t-l) mod N},$$ where $L_{j} = (2^{j}-1)(L-1)+1$. All the levels and the smooth series will have the same length as the original one.
In our proposed approach, we use the MODWT algorithm (discussed in Section (ref)) as a data pre-processing method to capture more explicit information about the observed time series. The MODWT algorithm will provide us with $J$ detail coefficients (higher frequency bands) and a smooth (trend component) of the available time series $Y_t \; (t=1,2,\ldots,N)$, where $N$ is the number of past observations. We show a pictorial view of the MODWT decomposition using a Haar filter for the publicly available website traffic data in Fig. (ref). Wavelet levels are chosen as an integer value that specifies the number of time series decompositions into smooth details. We set it as $J+1 = \lfloor \log_{e}{N} \rfloor$ based on the recommendation in percival2000wavelet. These high and low-frequency decomposed series are termed wavelet (details), and scaling (smooth) coefficients that can track the original series as: $$Y_t = \sum_{j=1}^J D_{j,t} + S_{j,t},$$ where $D_{j,t}$ is the $j^{th}$ level details and $S_{j,t}$ is the smooth of the decomposed time series. The MRA-based MODWT decomposed series can handle nonstationarity and seasonality in the time series datasets and capture the overall trend of the series. Each level of the time series and the smooth component are then passed into several independent Transformers for the prediction task. Time series datasets are passed through two parallel networks starting both an embedding layer. The encoder embedding takes the input of the batch train data and the decoder takes the output of the batch train data. Further, the output of both the encoder and decoder will enter into a second decoder. The encoder is composed of multi-head attention, a normalization layer, and a feedforward layer. The first decoder has a masked multi-head attention layer and normalization, and the second decoder has the same structure as the encoder. We apply a linear and a softmax layer to the output of the second decoder, which gives us the weight parameters of our model. Once all the epochs are done, we have our final model ready for making predictions and one-step ahead forecasts. In the end, we predict a time series for each level of decomposition and the smooth size of the forecasting horizon with their respective transformer model and reverse the process of MODWT with inverse MODWT as discussed in Sec. (ref). The result of the inverse MODWT will be our final prediction that could be compared with the test time series. Multi-step ahead forecasts are generated using an iterative method applied to the fitted W-Transformers model. To generate $h$-step ahead forecasts $(\hat{Y}_{N+h})$, we use the following equation:
where
and $f$ is the Transformer model. We describe the mathematical formulation of the Transformer model that we use in our ensemble W-Transformers framework to keep the current work self-contained, following vaswani2017attention. Once the nonstationarity and seasonality are handled by the MODWT algorithm, then Transformers model the long-range dependencies and interactions in sequential decomposed datasets. Mathematically, the self-attention mechanism of Transformers is defined as follows: $$Attn(Q,K,V) = \text{softmax}\left(\frac{Q K^{T}}{\sqrt{d}}\right) V,$$ where $Q$ is the query matrix, $K$ is the key matrix, $V$ is the value matrix, and $d$ is the dimension of $Q$, $K$, and $V$. Next, the softmax function is used to obtain the weights on the values to be put into a matrix $Q$. Now, the keys, queries, and values are then linearly projected, and these projections perform attention function in parallel yielding the output values. Theoretically, we define these heads as follows: $$head_{p} = Attn(QW^{Q}_{p},KW^{K}_{p},VW^{V}_{p}),$$ where the projections are parameter matrices, denoted by $W^{Q}_{p}, W^{K}_{p}, W^{V}_{p}$. Finally, the multi-head attention is simply a concatenation of the $head$ for $m$-times, where $m$ is an hyper-parameter that is usually set to a pre-specified integer value $8$ (as in vaswani2017attention) and $d$ the dimension of $Q$ and $V$ defined by $d = \frac{d_{model}}{m}$, where $d_{model}$ is the dimension of the output of the embedding layer. To jointly attend the information from various subspaces, we use multi-head attention defined as follows: $$\text{MultiHead}(Q,K,V) = \text{Concat}(head_{1},...,head_{m})W^{O},$$ where $W^{O}$ is the projection matrix of the output layer.
A detailed explanation of the W-Transformers is given in Algorithm (ref) and a schematic diagram is also presented in Fig. (ref).
In this section, we present the used datasets, the performance measures, the detailed description of the implementation scheme of the proposed W-Transformers, and the baseline methods for comparisons.
We considered seven publicly available datasets (see Table (ref)) from various applied fields for our experiments. For example, one dataset is from the stock market, three are from epidemiological fields, two are related to internet traffic, and the last one is about the sunspot data.
On these seven univariate datasets, we applied KPSS Test for stationarity, Terasvirta test for linearity, and Hurst exponent check for long-term dependencies chakraborty2022nowcasting, hyndman2018forecasting. Results of these tests on different characteristics are shown in Table (ref). Other than the Japan Flu dataset, all other datasets exhibit strong nonstationarity and long-term dependency. Only NFLX and Sunspot datasets show linear structure in the time series.
To comprehensive analysis and evaluation of different methods, we considered four commonly used performance metrics including Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Symmetric Mean Absolute Percentage Error (sMAPE), and Mean Absolute Scaled Error (MASE) hyndman2018forecasting. The lower values of these measures indicate better forecasting results.
To perform MODWT and inverse MODWT, we used the python package 'Pywavelet' Lee2019PyWavelet. To use MODWT for time series analysis, we carefully chose the number of levels for our decomposition to get the optimal performance in the wavelet domain as discussed in Sec. (ref). Once our proposed model is implemented, we computed the different metrics for all the baseline models for different forecasting horizons on each dataset. The different forecasting horizons are defined as short-term forecasting and long-term forecasting, as shown in the test size column in Table (ref). The implementation of Transformers on the decomposed datasets is done using `darts' Herzen2021Darts Python implementation tools by setting the number of epochs to be 200 for all the series. We tested several configurations for the input/output length, and regarding the result, we normalized the value to 12/1 for every dataset. The number of heads $m$ is set to $8$ and the dimension of the model, $d$, is 16, and the batch size is set to 32. We have used two encoder layers as well as two decoder layers. The dropout is set to 0.1, and the activation used in the W-Transformers model is `relu' zerveas2021transformer. Our proposed model was trained on GPU (graphics processing unit) but can also be computed using a CPU (central processing unit).
For a comprehensive comparison, we computed the results of our proposed W-Transformers and ten baseline models on seven datasets with two different forecast horizons. The experimental results for short-term and long-term forecasting are shown in Table (ref) and Table (ref), respectively. The experiments show that the performance of the models mainly depends on the length of the forecast horizons. W-Transformers is, in fact, more efficient for long-term forecasting. This comes from the MODWT decomposition; wavelets can handle nonstationarity and Transformers work mostly well for long-term forecasting. To demonstrate the statistical robustness of the obtained results, we applied multiple comparisons with the best (MCB) test by combining all the results based on four performance metrics and reporting the results in Figures (ref) and (ref). The overall performance of our W-Transformers is superior compared to all deep learning models and statistical forecasters considered in this study. For small sample-sized datasets, the performance of statistical methods is comparatively better than that of W-Transformers. However, the proposal performed better than all the baseline models for large temporal datasets (for example, the Network Analytics dataset, etc.). All the data and codes are available at \url{https://github.com/CapWidow/W-Transformer} for public use to support the principle of reproducible research.
We proposed a novel approach called W-Transformers for time series forecasting. Our model uses wavelet decomposition within the Transformers framework. The proposed model achieves much better results on average than several existing models. Moreover, our proposed approach consistently outperforms all conventional Transformers-like methods on the considered datasets. It is worth noting, nevertheless, that our proposed approach works better with large-size datasets. Statistical models can easily outperform deep learning-based approaches with too small data sizes. Another important finding from W-Transformer compared with the benchmarks is that the proposal is superior for long-term forecasting. In future work, extending the proposed method to work better with small data sizes will be of interest. Improving the results for short-term forecasting is also a key challenge and can be considered a future scope of the study. Applying the proposed W-Transformers approach to multivariate time series is under investigation.
The support of TotalEnergies is fully acknowledged. Lena Sasal (PhD Student) and Abdenour Hadid (Professor, Industry Chair at SCAI Center of Abu Dhabi) are funded by TotalEnergies collaboration agreement with Sorbonne University Abu Dhabi.