Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
40,943 characters · 22 sections · 65 citation commands
My Publication Title --- Multiple Authors
Electricity price forecasting is closely linked to residential welfare and deserves thorough investigation. Modern society relies heavily on a stable supply of electrical energy, but the cost can be a significant financial burden. Ensuring reliable power supply while reducing living costs is vital, especially for low-income populations. In some countries, residents buy electricity directly from the market, while in others, state-owned companies purchase and retail it. Regardless of the form, the electricity price in the power market directly affects residents’ cost. Prices fluctuate throughout the day, leading to varying consumption costs bielen2017future. Advanced energy storage systems allow residents to buy electricity during low-price periods and use stored energy during high-price periods, reducing expenses. This strategy relies on accurate price forecasts, which directly impact financial savings zheng2020impact,gil2012forecasting. Therefore, effective electricity price forecasting is essential for optimizing residents' revenue and enhancing societal welfare.
The day-ahead market is a pivotal part of the electricity market, and this paper focuses on forecasting day-ahead electricity prices. Forecasting electricity prices involves various tasks depending on the specific market: Day-ahead, Intra-day, or Balancing markets maciejowska2022forecasting. Among these, the day-ahead market is the most significant, attracting the majority of traders Castelli2020Forecasting.
Current methods for forecasting day-ahead electricity prices often underperform. These approaches primarily involve the straightforward adaptation of general Time Series Forecast (TSF) models to the day-ahead electricity market lago2021survey. However, the electricity market is characterized by large price fluctuations, a lack of clear periodicity and temporal non-stationarity. Consequently, existing methods may not be effective patel2021energy.
Current forecasting methods are limited by their inability to capture the strong, variable correlations between prices and supply-demand. The price formation mechanism in the electricity market is clear and well-understood; economics indicates that prices are determined by the balance between supply and demand Martin_2013. Therefore, the correlation between electricity prices and market supply-demand is crucial for accurate forecasting ZIEL2016435X-Model. However, TSF methods struggle with this. Some approaches just ignore the impact of supply-demand variables Zeng2022dlinear, nie2022patchtst, while others trying to use data-driven methods to model these correlations liu2023itransformer, zhang2023crossformer. Nonetheless, we find that in trained models, these correlations do not significantly contribute to forecasting results.
We propose a simple model to forecast day-ahead electricity prices by modeling the correlation between prices and supply-demand. Instead of directly using TSF models to forecast prices, we utilize supply-related variables forecasted by TSF models and focus on establishing the correlation between these variables and prices based on the price formation mechanism. This approach avoids the weaknesses of current TSF models in capturing variable correlations and provides better interpretability. Specifically, we introduce the assumption of the short-term stability of the supply curve, under which we can forecast the correlation between supply and price using recent historical data. We fit the supply curve using a simple Correlation-based Piecewise Linear Model (CoPiLinear), then use this and the forecasted demand to forecast the price, as Figure (ref) shows.
In summary, our main contributions include:
Recent studies on day-ahead electricity price forecasting have employed models directly migrated from existing TSF methods, can be categorized into two types based on the selection of known variables. The first type focuses on the temporal dependency of the target variable, capturing patterns like periodicity and trends. This includes univariate and channel-independent multivariate TSF models. The second type, in addition to considering the temporal dependency of the target variable, also takes into account the correlation between the target variable and auxiliary variables, represented by various channel-dependent multivariate TSF models.
TSF models emphasize end-to-end forecasting. Advanced second type methods generally capture inter-variable correlations through modules like MLP and attention in parts of the model’s substructure. This approach introduces additional parameters and makes it challenging to intuitively explain the correlations between variables. Moreover, advanced second type methods do not achieve more accurate forecasts with more information as intended. This has been demonstrated by the superior performance of first-type methods like TimesNet wu2022timesnet and PatchTST nie2022patchtst on numerous multivariate datasets.
Modeling the price formation mechanism aligns with economic theory but faces practical challenges due to data unavailability. Accurate projection of supply and demand curves requires data on production and purchase propensities at various price points. However, obtaining comprehensive data is difficult in extensive markets due to the private nature of individual buying and selling decisions, complicating precise measurements of willingness to buy or sell at each price.
The summary of related work is listed in Table (ref). The Model column indicates the primary model used for forecasting in the literature. The Domain-Specific column shows whether the method is tailored for day-ahead electricity price forecasting. A $\bullet$ in the Domain-Specific column denotes methods developed for economic research, not applicable in many markets. The Variable Correlation column reflects if the literature leverages variable correlations for forecasting.
While regulatory frameworks for day-ahead electricity markets vary by country, their structure generally follows a standard model EU2024, Haidar2021. Prices for each time slot is independently determined through an auction-based system Shi2023, driven by supply and demand dynamics. Take the day-ahead electricity market in Shanxi Province, China, as an example. There are 96 trading time slots in a day, each 15 minutes long. As shown in Figure (ref), an authoritative third party announces the day-ahead electricity prices for day $D + 1$ and other forecast variables on day D, then market participants submit their bids before the day $D + 2$ market closes. The third party considers anticipated production and costs reported by power producers and accounts for constraints such as grid dispatch limitations, operational characteristics of power plants, and emergency reserves. Through optimization calculations, generation schedules and dispatch plans are coordinated to meet societal electricity demand. Power generation companies then formulate their production plans based on the finalized schedule shanxi_power_market. Through this pricing process of balancing supply and demand, the day-ahead electricity prices for day $D + 2$ in the entire Shanxi market is determined.
Day-ahead electricity prices show significant dispersion in the high-price range. In the ISO New England market, while 75% of prices are below \$40/MWh, peaks can reach \$300/MWh. This irregularity and lack of clear time series patterns result in poor performance of baseline models, especially time series forecasting models, in forecasting high prices rahaman2019spectral,Xu_2020freqpricep.
High prices are mainly driven by high demand, which can help forecast these irregularities. However, TSF models struggle to capture the correlation between prices and supply-demand dynamics. According to the price formation mechanism, forecasted demand should significantly impacts forecasted price. Yet, even with substantial changes in demands, existing models show minimal price variation. This inconsistency with real market behavior indicates that these models do not adequately incorporate supply and demand variables, leading to poor forecasting performance. See Appendix A for detailed experiment results.
In Figure (ref), the ISO New England Dataset from July 17, 2023, shows forecasted peak demand surpassing the previous week’s peak, indicating high prices. Models based solely on temporal dependence fail to account for forecasted demand, leading to inaccurate high price forecasts. While existing TSF models that capture variable correlations perform better in predicting price trends, they underestimate price increases during significant demand surges, overlooking the price formation mechanism.
Our goal is to forecast the day-ahead electricity price for each time slot on day $D+x$ before market closure. The variables at our disposal include day-ahead electricity prices, capacity, weather data and market demand. The capacity indicates the maximum power the generator can produce. The load rate, calculated by dividing supply quantity by capacity, shows the operational status of power generation equipment.
Suppose that $P_{h}^{D}$ is the day-ahead electricity price at the $h$ moment on $D$ day, $C^{D}$ is the capacity on $D$ day, $Qs_{h}^{D}$ is the supply quantity at the $h$ moment on $D$ day, $Qd_{h}^{D}$ is the market demand quantity at the $h$ moment on $D$ day and $LR_{h}^{D}$ is the load rate at the $h$ moment on $D$ day. The forecast weather data for day $D+x$ include temperature and wind speed, denoted as $\hat{W}_{h}^{D+x} $, and the forecast market demand quantities published by third parties, denoted as $\hat{Qd}_{h}^{D+x} $.
This study evaluates the economic revenue of day-ahead price forecasts for residents. Based on related work zheng2020impact, we formulate an optimization problem for the purchasing strategy, using given forecast prices and the energy storage systems. The cost savings from this strategy represent the economic revenue of price forecasts, and improved forecast accuracy enhances the revenue. The purchasing cost $C$ reaches its theoretical minimum $C_a$ if forecasts match actual prices perfectly. Without the strategy, the cost is $C_0$. The economic revenue $R$ equals $C_0 - C$. To assess the impact of forecast accuracy, we calculate $\alpha = \frac{R}{C_0 - C_a}$, reflecting the percentage of the maximum theoretical revenue achieved by the current forecast.
We notice that the day-ahead electricity price is determined by the supply and demand, so we directly use this correlation to forecast the price. Forecasting precise supply and demand curves is impractical due to the need to forecast individual trading decisions at various price levels. Obtaining such detailed data is challenging because of the private nature of these decisions.
To address this issue, we introduce the assumption of the short-term stability of the supply curve. Using recent historical data, we reconstruct the supply-price correlation and obtain a forecast for the future supply curve. We fit the supply-price relationship using a simple Correlation-based Piecewise Linear Model (CoPiLinear). With this curve and supply volume forecasted by TSF models, we can forecast the price, as Figure (ref) shows.
It is reasonable to assume the supply curve remains constant over a specific period. Commodities from different electricity power generators are homogenized, making the supply curve equivalent to the marginal cost roberts1987large. Notably, within a supplier, the marginal cost is primarily dictated by the load rate rather than the supply quantity. The load rate reflects the generator’s operating conditions, which directly impact costs Roozbehani2022: costs rise rapidly when transitioning from standby to startup, slow down upon reaching a stable segment, and surge again if a backup unit is activated. Due to managerial inertia, power plants typically do not alter operational units over several consecutive days, leading to a stable correlation between marginal cost and load rate during this period. More details about the short-term stability are listed in the Appendix B.
Using the load rate instead of supply quantity as the horizontal axis of the supply curve, the correlation between price and load factor exhibits short-term stability over recent days. This conversion is straightforward by dividing the supply quantity by the capacity on that day.
As shown in Figure (ref), we selected a period from March 23, 2023, to April 3, 2023, within the Shanxi day-ahead electricity market to illustrate the short-term stability. The supply curve shape remains relatively stable across several consecutive days, requiring only minor shifts for approximate overlap. By using the load rate instead of supply quantity, we can align the supply curves of adjacent dates. This alignment reveals the short-term stability of the supply curve over consecutive days. Additionally, the similarity of supply curves decreases with longer intervals between dates.
Assuming short-term stability, we can forecast the supply curve by fitting it to recent days’ data. Our challenge is to determine the recent supply curve without data on willingness to buy or sell at each price. While the demand curve position varies significantly at different time slots, the supply curve remains relatively stable. We can get historical equilibrium day-ahead electricity prices and power supply volume at various trading time slots, corresponding to the intersections of the supply and demand curve. These intersections, all on the same supply curve, can be used to fit the supply curve in reverse, as shown in Figure (ref).
We use the most relevant historical data to ensure accuracy. Supply curves are more similar between closer dates, as shown in Figure (ref). Therefore, our initial step is to identify and select the most suitable historical data. We model the supply curve for each day and examine these curves for deviations in shape. If a deviation exceeds a threshold, we retain only the data after the deviation.
The necessity for the most recent dates limits the amount of usable data, so it is imperative to employ a model that is relatively simple and easy to fit the supply curve. We represent the supply curve with a piecewise linear function, reflecting its phases with different slopes. Due to power plant generator characteristics Roozbehani2022, the supply curve consistently exhibited piecewise phases: rapid growth, steady growth, and then rapid growth again. Therefore, we employ the simplest form of a n-segment piecewise linear function to represent the supply curve, named CoPiLinear. The slopes and intercepts of the n lines are represented by \(w_1\), \(w_2\), \(\ldots\) , \(w_n\) and \(b_1\), \(b_2\), $\ldots$, \(b_n\) respectively. And $LR_1^*$, $\ldots$, $LR_{n-1}^*$ is the breakpoints of the piecewise linear function.
We optimize the parameters using historical equilibrium points data. Throughout the day, we can observe prices and load rates at various transaction time slots. These correspond to scatter points on the supply and demand curve plane, note as \((LR_{1}, P_{1})\), \((LR_{2}, P_{2})\), \(\ldots\), \((LR_{N}, P_{N})\). The set of scatter points includes the closest $d$ days' data, each day has $k$ time slots. The goal is to fit these scatter points with CoPiLinear, minimizing the sum of distances from all points to the n-segment line. The final constraint equation ensures the continuity of the supply curve.
Fitting piecewise linear lines is a well-established problem with mature solutions. We use the Python package pwlf jekel2019pwlf to quickly approximate the supply curve. However, direct application that package on certain datasets may result in inaccuracies. To enhance fitting accuracy, we employ specific techniques detailed in Appendix C.
Electricity demand fluctuates significantly over time, making the demand curve challenging to forecast. By forecasting the supply curve, we can determine the price by identifying the demand quantity at the intersection of the supply and demand curves, without needing to forecast the entire demand curve.
Next, we need to forecast key supply-demand variables. The forecasted capacity data is used to adjust the horizontal axis of the CoPiLinear model from load rate to supply volume, creating the actual supply curve. The forecasted demand volume is then input into the CoPiLinear model to obtain the forecasted price.
These variables exhibit strong periodicity, making TSF models particularly effective, as detailed in Appendix D. Given their importance for the market, numerous mature TSF methods exist. Many service providers and market operators offer forecast data to aid market participants’ decision-making, so we directly utilize publicly available forecasts from third parties. In markets lacking forecasts for certain variable, such as capacity, we employ a simple model, Random Forest, to generate the necessary forecasts.
In our experiments, we utilized datasets from two distinct regions’ day-ahead electricity markets: Shanxi and ISO New England, some information is shown in Table (ref). The details are listed in the Appendix E.
We selected two types of baselines: models relying solely on temporal dependence and those capturing variable correlations. For temporal dependence, we chose domain-specific methods such as SARIMA zhao2017improvingsarima and VMD-LSTM xiong2023vmdlstm, and generic TSF methods like DLinear Zeng2022dlinear and PatchTST nie2022patchtst. For variable correlations, we selected domain-specific methods including Linear uniejewski2016automatedlinear, SVM prahara2022improvedsvm, XGBoost manfre2023hybridxgboost, DNN lago2018forecastingdnn, and Lasso-RF ludwig2015lassorf, along with generic TSF methods such as Koopa liu2023koopa, TSMixer chen2023tsmixer, Informer zhou2021informer, Autoformer wu2021autoformer, FEDformer zhou2022fedformer, Crossformer zhang2023crossformer, and iTransformer liu2023itransformer. The variables used in these generic TSF models are exactly the same as those in the CoPiLinear, including historical price, capacity, demand data and forecast demand, weather data.
To simulate real market applications, our model uses a daily rolling training method, adding new data each day for retraining. All baselines also adopt this method, updating the training and validation sets daily. Deep learning methods with short training sets tend to perform poorly. To improve accuracy, we use the longest possible training set and adjust hyperparameters during testing. More baseline details are in Appendix F.
Our experiment platform is a server with 12 CPU cores (AMD Ryzen 9 7900X), and 32 GB RAM. Our GPU is NVIDIA GeForce RTX 4060 Ti 16 GB.
Based on related work lago2021survey, we select MAE and sMAPE to evaluate forecast accuracy. Reference zheng2020impact guides our choice of $R$ and $\alpha$ to assess the revenue of the forecast. Detailed descriptions of the evaluation metrics are provided in Appendix G.
In Table (ref), we present the main results. For consistency, prices in the Shanxi dataset are converted to US dollars using the exchange rate of 7.16 RMB to 1 USD as of the writing date. For the Shanxi dataset, CoPiLinear achieves the lowest MAE and sMAPE, outperforming the second-best by 22.9% in MAE and 38.0% in sMAPE. This superior forecast accuracy also boosts electricity purchase revenue, with CoPiLinear generating the highest revenue and alpha. To ensure reproducibility, we also test on the ISO New England dataset, where CoPiLinear again shows the lowest forecast error and achieving the highest revenue. Note that the electricity volume consumed by residents in Shanxi is approximately 3 times that of ISO New England, resulting in higher revenue.
To ensure forecast reliability, we verify that the method maintains high accuracy across different times. We divide the ISO dataset’s test set (1 year) by month, select the top five methods with the best overall performance, and calculate each method’s monthly forecast error (MAE). A robust method should exhibit consistent accuracy each month. We plot the monthly forecast errors as box plots in Figure (ref), revealing that CoPiLinear is the most stable. Additionally, higher MAEs are observed in summer months, coinciding with increased market energy consumption and price.
For fitting the supply curve, we use a piecewise linear function, accurately capturing the different phases characterized by varying slopes. Alternative fitting methods include cubic functions, exponential functions, or models like MLP and XGBoost to implicitly learn the price-load correlation. To validate our approach, we tested several CoPiLinear variants: 1) Co-PLOY: cubic function; 2) Co-EXP: exponential function; 3) Co-XGB: XGBoost model; 4) Co-MLP: MLP model. These models differ only in the supply curve fitting function. Our ablation study results Table (ref) show that the original CoPiLinear model outperforms all variants on both datasets, confirming the effectiveness of using a piecewise linear function.
When shifting the supply curve’s horizontal axis from quantity to load rate, we use a Random Forest model to forecast capacity. Given the limited capacity data (one entry per day), we select models with smaller training set requirements to compare forecasting accuracy: SARIMA, TimesNet and XGBoost. What's more, the ISO New England dataset includes the organization’s official capacity forecasts, allowing direct comparison. As shown in Table (ref), our model achieves exceptional accuracy with only a 3% sMAPE error, significantly outperforming the official forecast. Our Random Forest model’s higher accuracy compared to other models further validates the design of this module. This demonstrates the validity and effectiveness of our capacity forecasting method.
Our method outperforms existing approaches in capturing the correlation between supply-demand and price. This allows for more sensitive adjustments to price forecasts based on forecasted demand or other variables, especially during significant supply and demand fluctuations. Figure (ref) analyzes the challenging high prices in the ISO New England market on July 17, 2023, due to a sharp increase in forecasted demand. Figure (ref) shows the CoPiLinear forecast results align more closely with actual prices compared to other methods. The CoPiLinear forecasted price peaks a few phases earlier than the actual prices, it can be explained that it synchronizes completely with the demand surge, whereas the energy storage system in reality delays the impact of the demand surge on prices.
We validate this in two datasets. If the forecasted demand during the day exceeds the previous week’s maximum or falls below the minimum, we anticipate significant price fluctuations. We selected these dates and compared the MAE of various forecasting methods, as shown in the $MAE_{CS}$ column of Table (ref). All methods showed increased MAE on these dates, with CoPiLinear achieving the highest accuracy in both datasets.
This paper has highlighted the limitations of current day-ahead electricity price forecasting methods and introduced a novel approach to fit supply curve. By assuming short-term stability, CoPiLinear has significantly improved accuracy, benefiting residents. Rigorous testing on two electricity datasets shows that CoPiLinear outperforms top-tier methods, underscoring its potential to reduce residents' costs and enhance forecast reliability in day-ahead market operations. Future work will refine the model and explore its applicability to other markets. This achievement offers new perspectives for electricity price forecasting research and showcases the potential of artificial intelligence in the socio-economic field.
This paper:
Does this paper make theoretical contributions? (yes)
If yes, please complete the list below.
Does this paper rely on one or more datasets? (yes)
If yes, please complete the list below.
Does this paper include computational experiments? (yes)
If yes, please complete the list below.