Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
84,835 characters · 32 sections · 70 citation commands
A data-driven merit order: Learning a fundamental electricity price model
\def\spacingset#1{ {#1}} \spacingset{1}
\if00 \fi
\if10 {
} \fi
{\it Keywords:} Electricity price, power, forecasting, fundamental, data-driven, merit order, nested system, estimation
\spacingset{1.45}
Accurate forecasting of electricity prices is essential for efficient operation and planning across the power system. Electricity producers rely on forecasts for power plant scheduling, buyers use them to optimize procurement strategies, and grid operators depend on them to maintain system stability and reduce the need for costly backup generation.
As electricity markets evolve, the price time series has become increasingly volatile and complex, particularly with the growing share of renewables and the shifts following the European energy crisis. In Europe, the most important electricity market is the day-ahead market, a blind daily auction that determines hourly prices for the following day. Each day at 12:00 CET, electricity prices are settled for every hour of the next day based on buy (bid) and sell (ask) volume-price offers submitted by wholesale market participants. Day-ahead prices are established through the aggregation of these bids, forming a market equilibrium where supply meets demand. This market serves not only as the primary platform for physical electricity trading but also as the underlying for the pricing of futures contracts and other derivatives. Its behavior reflects a broad range of physical and economic factors, including availability of conventional plants, renewable generation, demand fluctuations, fuel and $\text{CO}_2$ prices (see Figure (ref)).
The integration of large-scale intermittent renewable energy, such as wind and solar, has contributed to more frequent occurrences of very low or even negative prices. This trend is particularly evident in countries like Germany, where over 50% of total installed capacity are renewable energy sources (RES). This is mainly because many RES operate under subsidy schemes that allow them to bid at negative prices. Even without subsidies, their marginal costs are close to zero, enabling them to bid at very low or zero prices. As a result, RES dominate electricity generation during low-price periods, while conventional power plants tend to ramp up when prices are higher (see Figure (ref)).
These conditions have contributed not only to a rise in negative prices during periods of high RES generation but also to an increase in the number of consecutive hours with negative prices (see Figure (ref)). Another important development is the increasing concentration of prices around zero, along with the tendency for such low prices to occur in consecutive price clusters. Figure (ref).b shows that, following the European energy crisis, the electricity price distribution remained bimodal, but the gap between the two modes widened. This was driven by both an upward shift in the average price and a higher frequency of low or negative price occurrences. The two modes reflect different pricing regimes: one associated with high prices, typically linked to fossil fuel-based generation, and another with low or negative prices, typically observed during periods of high renewable output.
These dynamics call for price models that can address two key requirements: first, the ability to capture regime shifts in market behavior; and second, the flexibility to adapt to changing system conditions, such as the increasing clustering of prices. This implies a need for models that go beyond statistical pattern recognition by incorporating fundamental drivers of electricity prices.
Electricity price forecasting methods can be broadly classified into two categories: fundamental and data-driven models petropoulos2022forecasting, weron2014electricity, ziel2016electricity. Each have their strengths and weaknesses which are summarized in Figure (ref).
Data-driven models include classical econometric, statistical, as well as advanced machine learning approaches. These models use historical data to forecast power prices by identifying past patterns and projecting them into the future hastie2009elements, lutkepohl2005new, brockwell2002introduction. These are primarily used for short-term forecasting cuaresma2004forecasting, narajewski2020econometric, uniejewski2019understanding, petropoulos2022forecasting because they effectively capture the high volatility of power prices and run efficiently, especially with linear models and online learning algorithms wintenberger2017optimal, cesa2021online, adjakossa2024kalman, ziel2022smoothed.
Fundamental models on the other hand simulate the electricity market mechanism and power plant fleet within a geographic area ringkjob2018review, brown2017pypsa, petropoulos2022forecasting, beran2021multi. These are also referred to as or contain structural models, market-based equilibrium models, cost optimization models and agent-based models. These models are mostly optimized by minimizing overall power production costs or maximizing overall welfare, often for complex networks spanning multiple countries. Their outputs range from optimal power plant schedules to capacity buildouts and power plant mixes, with the price of power typically being only one of the outputs, calculated for example as the shadow price or dual variable of the demand constraint marcos2019electricity, brown2017pypsa. Traditional fundamental models require detailed technical information and complex operational constraints, making them computationally expensive and unsuitable for short-term forecasts that need frequent updates. Moreover, for short-term forecasting, they are often described as too "flat" meaning that they fail to capture the high volatility of power prices pape2016fundamentals, marcos2019electricity, beran2021multi (see Figure (ref)). However, unlike data-driven models they, are able to capture price clusters, which often form around zero in markets with a high share of RES (see Figure (ref)). Examples of open-source fundamental and agent-based models include PyPSA brown2017pypsa, Calliope pfenninger2018calliope, OSeMOSYS howells2011osemosys, AMIRIS schimeczek2023amiris and ASSUME harder2025assume.
While data-driven models excel in short-term forecasting accuracy, they are highly sensitive to the historical data used for training and can fail under regime changes, even if the market mechanism remains unchanged. This issue is particularly problematic for long-term forecasts ghelasi2025day.
For instance, consider a market consisting only of gas and coal power plants, with a simple linear econometric model predicting prices based on fuel costs, as illustrated in Figure (ref). The figure shows the model's merit-order (MO) representation, created by rearranging and plotting the estimated coefficients, demonstrating that any econometric price model can be visualized in this form.
When a new technology such as wind power is introduced, the data-driven model cannot capture its effect, and therefore cannot predict prices accurately if the impact is substantial, due to the lack of historical data needed to estimate a wind-related coefficient. Although pure data-driven models can very well detect the pattern of lower prices during periods of high renewable energy (RES) generation, they lack a fundamental explanation for this effect, namely, that RES has near-zero marginal costs. As a result, they can fail to capture the clustering of prices around zero and tend to overshoot actual prices, as seen in Figure (ref). This is expected, as such models learn only from historical patterns and lack a fundamental understanding of the underlying market mechanisms.
In contrast, a fundamental model does not rely on historical data. It simulates the market mechanism and can generate prices based on current or expected inputs such as forecasted RES expansion or fuel prices, provided the underlying mechanism remains unchanged. If a new technology like wind is introduced, its effect can be easily modeled by specifying its current cost parameters and available capacity (see the merit order model in the sections below). However, fundamental models are often too slow or inflexible for short-term forecasting. This highlights the need for a hybrid approach that combines the strengths of both methods.
There have been some attempts in literature to combine the strength of both modelling approaches. There are two main categories, first the hybrid models and inverse optimizations. Table (ref) summarizes some of the most relevant papers in these categories by their methodology, data, period and performance. The list was compiled and manually filtered to only include papers that explicitly also model electricity prices. bello2016medium, beran2021multi, gonzalez2011forecasting, marcos2019electricity, gabrielli2022data.
Regime-switching models. In the past, Markov regime-switching models and variations thereon have been proposed to capture the price-switching behavior of electricity markets, and the literature on them is extensive. These models initially focused on modeling price spikes rather than low or negative prices, reflecting the conditions of the early 2000s when renewable energy penetration was limited and peak prices were the primary concern weron2004modeling, haldrup2006regime, mount2006predicting, karakatsani2008intra, but more timely applications have been tried as well kapoor2023analyzing. However, they are strictly data-driven and lack fundamental interpretability, offering limited insight into the underlying market mechanisms. Fundamental models, while not regime-switching models, can still capture this type of behavior intrinsically as they simulate the underlying mechanism.
Hybrid models. In the electricity price forecasting literature, the term hybrid is often used to describe approaches that combine different model types. A particularly relevant subset involves the combination of fundamental models with data-driven methods. In such cases, the hybridization is nevertheless implemented as a post-processing step, meaning that the output of the fundamental model is used as an additional input to the data-driven model. This form of integration is extrinsically hybrid, as the two components remain functionally and structurally separate. Table (ref) summarizes the papers in this category.
For example, beran2021multi use forecasts from a parsimonious fundamental model as inputs to a set of expert data-driven models, specifically ARX models, following the framework of ziel2018day. This improves short-term price forecasting in the German market. Similarly, gonzalez2011forecasting apply a merit-order-based fundamental model to generate price forecasts for the UK market, which are then fed into ARMA and LSTR (logistic smooth transition regression) models. Comparable procedures are used by marcos2019electricity and gabrielli2022data for the Spanish and UK markets, respectively. Although these methods have shown improvements in forecast accuracy, the coupling between the fundamental and data-driven models is limited to a feed-forward mechanism, and the integration is not realized at a structural or algorithmic level. Other numerous approaches also labeled as hybrid zhang2020adaptive, alkawaz2022day, yang2019hybrid, shafie2011price, wan2013hybrid, huang2021novel involve combinations of only data-driven methods, such as neural networks, support vector machines, or ensemble learning techniques, without including any market-based or physical modeling. While these models can effectively capture statistical regularities in the data, they may lack interpretability and may struggle to generalize in the presence of structural market shifts or regime changes.
The major limitation of this approach is that the data-driven model functions purely as a post-processing step, lacking any integration of fundamental constraints or causal interpretability. As a result, it offers no guarantees of generalization to new market conditions or of capturing structural effects such as price clustering driven by RES.
Inverse optimization. A more intrinsic approach to combining the two modeling types is offered by inverse optimization methods. The term inverse can refer to multiple approaches, as summarized by esser2025multi: inverse problem theory refers to the estimation of a cause given an effect, instead of the other way around tarantola2005inverse, kirsch2011introduction. This includes parameter estimation or inverse modelling da2023computational, groetsch1993inverse, richter2015inverse, where the unknown cause is inferred given a known effect and model, as well as \textit{function estimation}, which refers to the estimation of the model itself given a known cause and effect gonzalez2022building. \textit{Inverse simulation} refers to using simulations to replicate observed behavior kurahashi2018model, murray2000inverse. Newly \textit{inverse optimization} is also used refer to problems where variables traditionally treated as inputs are treated as outputs instead and optimized for esser2025multi.
In the context of energy systems modeling, inverse optimization typically refers to recovering market data parameters from observed market outcomes birge2017inverse. This involves finding optimal model parameters that best reproduce observed outcomes—essentially solving the inverse of the classical forward problem, where fixed inputs produce modelled outputs. In inverse optimization, inputs (i.e., parameters) are adjusted to minimize the error between the model's output and actual market data. This aligns with the essence of data-driven models, where parameters are estimated by minimizing output error, rather than being fixed or based on expert assumptions.
In the context of energy systems modeling it is sufficient to note that inverse optimization refers to recovering market data parameters from an actual market outcome birge2017inverse. This means finding the optimal fundamental model parameters given the observed data. This is in contrast to the classical forward problem, as discussed for the hybrid models, where the fundamental model has fixed inputs and produces modelled outputs. Inverse optimization methods reverse this process by finding those input values, in our case parameters, that minimize the error between the output of the model and the true value. In inverse optimization, inputs, i.e., parameters, are adjusted to minimize the error between the model's output and actual market data. This aligns with the essence of data-driven models, where parameters are estimated by minimizing output error, rather than being fixed or based on expert assumptions.
Literature on inverse optimization in electricity markets remains limited but spans various applications, such as recovering consumption decision models kovacs2021inverse, estimating unobserved demand components esteban2024estimating, and determining agents' objective functions nguyen2020cooperative. A significant portion focuses on reverse-engineering optimal bidding strategies in electricity markets saez2016data, saez2017short, gallego2017inverse, bertsimas2015data, fernandez2021inverse, chen2017strategic, chen2019learning, particularly on the demand side. Most of these works aim to estimate the bidding behavior or optimal operation of price-responsive loads based on market outcomes, though many rely on simulations rather than real-world data.
Some studies have applied inverse optimization to price modeling or forecasting, as summarized in Table (ref). liang2023data applied inverse optimization to a cost-minimization model of the NYISO market to recover marginal offer prices at the generator level, minimizing the error between modelled and observed prices. chen2017strategic used a simulation-based approach to estimate optimal bidding behavior in the day-ahead market. birge2017inverse inferred technical parameters, such as utilizations and transmission line capacities, by minimizing errors between modeled and actual prices. Similarly, ruiz2013revealing recovered unobserved offer prices of rival producers using a cost-minimization model.
However, none of these studies apply inverse optimization to a fundamental model with the sole purpose of real-world electricity price forecasting, in a data-driven forecasting framework. Most rely on simulations and involve complex optimization frameworks with numerous constraints, limiting their suitability for short-term forecasting due to computational demands.
The literature review above show that while regime switching models, hybrid models have been proposed in the context of electricity price forecasting, they do not fully leverage the strengths of both fundamental and data-driven approaches. Inverse optimization methods, while promising, have not been applied for real-world electricity price forecasting in a data-driven framework. We fill this gap by proposing a novel, highly-efficient data-driven fundamental merit order model tailored to electricity price forecasting, which is in essence an inverse optimization approach applied to a fundamental model, however, we essentially use it exactly as a data-driven econometric or machine learning model is used, i.e., estimating the parameters by minimizing the forecasting error on the training set and then using the model to forecast prices on a test set. This approach allows us to retain the causal interpretability of fundamental models and their implicit capability of capturing market effects such as regime switches and RES expansion, while achieving the accuracy and computational efficiency of data-driven models.
This approach intrinsically combines the strengths of both fundamental and data-driven models, as summarized in Figure (ref). The key novelties of this approach are:
We apply this model to day-ahead electricity price forecasting in Germany and demonstrate that it outperforms both classical fundamental and data-driven models of similar complexity. To our knowledge, this is the first implementation of such an approach.
This paper is structured as follows. Section (ref) introduces the data and insights into power price formation, motivating the use of a fundamental model as the starting point. Section (ref) introduces the fundamental merit order (MO) or supply stack model and the formulation of the data-driven approach. Section (ref) presents parameter estimation approaches for the MO model and compares the results to state-of-the-art benchmarks which are discussed in Section (ref). Finally, section (ref) concludes and outlines future research directions.
In this study, we use data from Germany, Europe's largest electricity market with a balanced energy mix across major sources. Unlike countries like France, dominated by nuclear power, Germany's diverse generation allows for meaningful analysis of both renewables and conventional technologies, making it ideal for applying the merit order model. A key component of the merit order model is the available capacities and generation of power plants. This data is sourced from the ENTSOE Transparency platform, which reports unavailabilities, installed capacities, and hourly generation at the power plant unit level. We aggregated this data by power plant type to produce the values shown in Figure (ref). Net imports are calculated as the difference between Germany's total imports and exports, which have increased significantly since the nuclear phase-out on April 16, 2023. Available capacities show some volatility due to planned maintenance and temporary unplanned outages. Yearly patterns are evident in conventional plants like coal and gas, where maintenance occurs in summer when prices are typically lower due to high photovoltaic (PV) infeed. Day-ahead forecasts for RES production and load are also provided on the platform. However, since day-ahead forecasts for net import and hydro are unavailable, we generate our own forecasts for these categories.
For convenience, we group all power plant types in a set $\mathcal{T}$, with subsets for RES plants $\mathcal{R}$ and conventional plants $\mathcal{C}$:
Another essential input is fuel prices, including gas, coal, and $\textrm{CO}_2$ European Emission Allowances (EUA). These prices are sourced as front-month futures from the Refinitiv EIKON information platform (see Figure (ref)).
Power plant parameters such as the efficiency, $\textrm{CO}_2$ intensity factors and marginal costs are further critical inputs for the merit order model. These parameters are traditionally estimated by experts and treated as fixed in the model. In the next section, we introduce these parameters along with the model, as they will serve as starting values for our estimation procedures (see Table (ref)).
Determining which power plant type sets the price at any given time is challenging as it requires analyzing the generation stack relative to the day-ahead price. Figure (ref) illustrates the average production percentages for different price intervals in 2023–2024. Conventional plants like gas, coal, and lignite ramp up production during higher price periods, while RES producers bid even at negative prices due to subsidy schemes. Some conventional plants also offer power at negative prices due to high ramp-down costs, contractual obligations, or combined heat and power (CHP) mechanisms. Imports increase during high-price periods, while excess RES production is exported during low-price periods. Certain plant types, such as gas, lignite, coal, oil, nuclear, and hydro, exhibit a clear must-run stack, as their total generation never drops to zero (see Figures (ref) and (ref)).
The merit order model is a basic fundamental approach to power price modeling, based on standard economic theory applied to the energy market. Figure (ref) shows this relationship.
Also referred to as the supply stack model, it simulates the electricity market by constructing the supply curve as the marginal costs of power plants sorted from lowest to highest. This is the equivalent of the supply curve in economics. The corresponding demand (load) is modelled as inelastic, under the assumption that retail customers such as households and firms use electricity regardless of the wholesale price. The curves are then intersected to determine the market price. The two curves can change for each time step, depending on the available capacities, production, fuel prices and seasonality of load.
The model relies on three main components: the marginal costs of power plants, their available capacities and the load. The steps in constructing the merit order as summarized in Figure (ref) are the following:
The individual merit order components, $\text{MOComp}_{\text{pl},t}(q)$, represented by the colored areas on the right side of Figure (ref), are the marginal generation proportion for each plant type applied to the merit order curve. For each plant type, this is calculated by multiplying the price for each segment with the ratio of its generated power to total generated power. In other words, for each plant type the ratio of the power quantity produced is multiplied with the price for each segment where the overlaps change.
where the fraction is the ratio of the differential of the whole inverse merit order to the differential of the individual merit order for a certain price. The fraction is not the derivative. The differentials $\text{d} \text{MO}_t^{-1}(p)$ and $\text{d} \text{MO}_{\text{pl},t}^{-1}(p)$ are infinitesimal differences in quantity when the price changes by a very small amount $\text{d} p$.
In the linear case, it is sufficient to compute the differences between the bounds of each linear segment of the merit order, with interpolation between them (see right hand of Figure (ref)):
where $p_i \in (\mathcal{P}, \le)$ represents the ordered set of all lower and upper variable costs for all technologies, defined as $\mathcal{P} = \{ (\text{VarCost}_{\text{pl},t}^{\text{L}})_{\text{pl} \in \mathcal{T}}, (\text{VarCost}_{\text{pl},t}^{\text{U}})_{\text{pl} \in \mathcal{T}}\}$, and $q_i = \text{MO}_t^{-1}(p_i)$ is the quantity corresponding to $p_i$.
We introduce the data-driven merit order as a generalization of ((ref)). For this we first define the volume space $\mathbb{Q} = [0, \infty )$ as the set of all possible quantities or volumes $q$ produced in MWh, and the price space $\mathbb{P}= [-500,3000]$\footnote{The market operator can adjust the upper and lower price bounds, as stated in Art. 41(1) of Commission Regulation (EU) 2015/1222 of 24th July 2015 (CACM Regulation). In Germany, the maximum price day-ahead price was raised from 3000 EUR/MWh to 4000 EUR/MWh on 10 May 2022, as announced by EpexSpot on 11 April 2022. Link: https://www.epexspot.com/en/news/harmonised-maximum-clearing-price-sdac-be-set-4000-eurmwh-10th-may-2022} as the set of all possible prices $p$ in EUR/MWh. Then, the data-driven merit order is a function which maps the domain (input-space) $\mathbb{Q}$ to the range (output-space) $\mathbb{P}$:
where $\Theta$ is the set of optimizable input parameters:
In contrast to the classical merit order ((ref)), these parameters are not fixed but rather given as an additional. All other inputs, such as the capacities, fuel prices, and $\text{CO}_2$ intensities, are assumed to be exogenous to the model and are held fixed. Thus, the classical merit order model ((ref)) is a special case of $\text{MO}_t(q;\Theta)$:
where $\Theta_{\text{init}}$ represents the expert estimates from Table (ref).
The optimal parameter set $\hat{\Theta}$ is determined by solving:
which minimizes the mean absolute error (MAE) between the modelled price $\text{MO}_t(q_t; \Theta) $ and the observed price $\text{Price}_t$, where $q_t = \text{Load}_t$. The parameter search space in constrained by the lower and upper bounds $\Theta^{\text{L}}$ and $ \Theta^{\text{U}}$, as summarized in Table (ref). The initial starting point for the optimization $\Theta_{\text{init}}$ lies within the search space and corresponds to the parameters of the classical merit order model. Consequently, the classical merit order model is an embedded within this framework. This means that, if the expert assumptions of the classical merit order model are indeed optimal, then the optimization will return these same parameters.
Fundamental power plant parameters in $\Theta$ are assumed to be constant over time, hence they do not depend on the time index $t$. They are solved once for the training period and held constant for the test period. The optimization problem is non-convex and involves multivariate nonlinear parameter estimation, making it a black-box optimization problem. To solve this problem, we use the R package mlrMBO mlrMBO, which does not require a gradient. Each optimization run was configured to execute for 60 minutes.
The function itself $\text{MO}_t(q;\Theta)$ and the parameter set $\Theta$ can be extended to include other modeling aspects. We introduce these extensions below and collect them in $\Theta_{\text{ext}}$:
where the additional components are detailed below. These parameters are estimated using the same procedure as in ((ref)).
Reported plant capacities (Figure (ref)) are systematically underreported, as only plants above 100MW must disclose data on capacity, outages, and generation\footnote{As required by Commission Regulation (EU) No 543/2013 of 14 June 2013.}. To account for this, we apply a correction factor $1 \le \text{cf}_{\text{pl}} \le 2$ by multiplying it with the reported available capacity $\text{Cap}_{\text{pl},t}$ of non-intermittent power plants where capacities less than 100MW are common. This assumes similar maintenance behavior across reported and unreported plants\footnote{Alternatively, $\text{cf}_{\text{pl}}$ could be added to $\text{Cap}_{\text{pl},t}$, thus the type of operator could be treated as a parameter to be optimized.}.
Figures (ref) and (ref) show that many plant types sustain a minimum generation level, never dropping to zero. This must-run behavior reflects plants operating continuously, regardless of market signals. Contributing factors include long-term contracts, high startup and ramping costs which can exceed losses incurred from negative price periods, or combined heat and power (CHP) obligations. We capture this with a must-run share $0 \le \text{mr}_{\text{pl}} \le 1$ for each plausible plant type. The must-run capacities are subtracted from the individual available capacities and aggregated system-wide:
It is represented as a virtual plant with a marginal cost of $-500$ EUR/MWh, ensuring dispatch guarantee.
Gas power plants are highly flexible and traditionally among the most expensive conventional sources after oil. They are typically divided into combined cycle gas turbines (CCGT), which are more efficient, and open cycle gas turbines (OCGT), used mainly for peak loads. To reflect this in the merit order model, we introduce a gas stack split factor $0 \le \text{gs} \le 1$, defining the share of CCGT and OCGT capacity:
This introduces two gas plant types, each with distinct parameters, increasing model flexibility. The optimal split is optimized. The same approach can be extended to other technologies, for example splitting RES by subsidy regime.
Hydropower in Germany includes run-of-river, reservoir, and pumped storage plants. As pumped storage involves both consumption and generation for temporal price arbitrage purposes, it is excluded from the merit order model to avoid such complexities. Thus, our model considers only run-of-river and reservoir output (Figure (ref)). Generation is forecasted using a LASSO-estimated model incorporating autoregressive terms, cross-hour dependencies, weekday effects, and seasonal patterns (see Appendix (ref)). Hydropower is then added to the merit order model as an additional technology type similar to RES.
Although classical merit order models exclude cross-border flows, net imports significantly affect Germany's power balance (Figure (ref)). We define net imports as:
where $t$ denotes the hour, and $\mathcal{B}_{\text{DE}}$ is the set of all bidding zones with physical power flows to and from Germany. Net imports are forecasted using a LASSO-estimated model with autoregressive terms, cross-hour effects, and weekly and annual seasonality (see Appendix (ref)). In the merit order model, net imports are treated as a virtual plant with a marginal cost of $-500$ EUR/MWh, implicitly assuming that Germany imports when foreign prices are lower. This shifts the supply curve to the right for net imports or to the left for net exports.
To evaluate the proposed model, we follow a standard data-driven forecasting setup. The data is split into a training set of 1 year (01/10/2022–01/10/2023) for parameter estimation as in ((ref)), and a test set of 1 year (01/10/2023–01/10/2024) for out-of-sample evaluation. The parameters $\Theta$ are estimated using the training set and then fixed for the test set. During training, actual RES generation is used. In the test phase, RES values are replaced with day-ahead forecasts.
The merit order model was implemented in C++ and interfaced with R via Rcpp R package rcpp. It is highly efficient, requiring approximately 1 second per run to simulate a full year of hourly prices. Table (ref) reports computation time statistics using the microbenchmark R package microbenchmark on a Linux machine (64 GB RAM, 14-core 12th Gen Intel® Core™ i7-12700H).
Training used Bayesian stochastic optimization via the mlrMBO R package with a 60-minute time limit. Around 3600 parameter configurations were evaluated per run, with the best (lowest training error) retained.
To assess the data-driven merit order model, we compare it against leading data-driven forecasting methods: Lasso, XGBoost, and a Neural Network, all trained on the same data as the merit order model, including autoregressive and cross-hours effects (see Appendix (ref)), and evaluated using a rolling window of 1 year over the test period. A Naive persistence model (see Appendix (ref)) serves as a baseline. Results are reported in Table (ref). The optimal parameter values, bounds and initial values for the merit order model are summarized in Appendix Table (ref).
The proposed data-driven merit order achieves the lowest MAE, improving by $1.2$EUR/MWh ($\approx 10\%$) over the best machine learning benchmark, Lasso. Notably, this result is obtained without using autoregressive price terms, relying solely on an optimized fundamental framework, unlike the machine-learning benchmarks, which include autoregressive components (see Appendix ((ref))).
Figure (ref).a shows that the optimized merit order has a more complex shape than the classical version (Figure (ref)), featuring exponential tails and a flat middle section. This structure mitigates the issues of the classical model, improves the representation of price volatility at both extremes, and captures clustering around zero due to the flat segment (Figure (ref).b).
To evaluate the contribution of each model extension to forecast accuracy, we perform optimizations for all $2^6 = 64$ combinations of the six parameter types. Using a greedy forward selection procedure, we begin with single-parameter optimizations and iteratively add parameters, selecting the combination with the lowest training error at each step. This process is illustrated in Figure (ref), where the forward selection path is highlighted in grey and the optimal test-based data-driven merit order configuration is highlighted in green.
After efficiencies, the most impactful extensions are joint optimization of capacity factors, net imports, gas split, and hydro. The must-run extension appears to add little additional value. While training error generally decreases with each added parameter, though not guaranteed due to the stochastic optimization, the test error does not always follow the same trend. This highlights the model's sensitivity to the calibration window and potential overfitting.
To further evaluate model performance, Table (ref) reports the MAE across quantiles of residual load, defined as day-ahead load minus day-ahead renewable generation: $\text{ResLoad}_t = \text{Load}_t - (\text{PV}_t + \text{WindOnshore}_t + \text{WindOffshore}_t)$. This represents the portion of demand that must be met by conventional power plants. Since high levels of residual load often coincides with price spikes and low levels with low or negative prices (see Figure (ref)), upper quantiles reflect high-price periods, while lower quantiles reflect low-price conditions. This residual load-based binning is preferred over price-based binning due to structural price shifts following the 2022 European energy crisis.
The data-driven merit order and its variants outperform nearly all benchmarks across all quantiles, except in the upper range, where Lasso performs better on two occasions. This is likely to be due to its autoregressive component capturing price peaks. Nonetheless, the merit order model is competitive in these cases and even surpasses Lasso at the $65\%$ quantile. It performs best across most quantiles, clearly outperforming all other models.
The superior performance of the data-driven merit order model over established machine-learning benchmarks, despite not using autoregressive inputs, is noteworthy. Figure (ref) compares the models in terms of linear and non-linear similarities as measured by the Pearson correlation and distance correlation coefficients respectively. The data-driven merit order model is most similar to the classical merit order, confirming its fundamental nature, yet it also shows considerable similarity to Lasso and XGBoost. Nonetheless, results in Figure (ref).b and Table (ref) indicate that each model captures distinct aspects, suggesting potential for model combination.
A key strength of the proposed model lies in its ability to estimate market and plant-level parameters (see Appendix Table (ref)). Most estimates align with expert values, though coal and oil efficiencies tend to fall at their lower bounds. Capacity factors are often estimated above 1, implying underreported capacities at the unit level. In contrast, must-run shares are less conclusive as coal and gas plants show unexpectedly low values. This highlights the trade-off in using an unconstrained optimization for price prediction. Future work could impose tighter constraints or adopt a joint objective function for both price and production, exploiting the customizable loss function, thus making estimated parameters more useful for further analysis.
Beyond forecasting, the merit order model provides deeper structural insights. Unlike econometric models, which model prices directly and implicitly weigh the influence of all plant types, the merit order framework focuses on the market-clearing point where supply meets demand and characterizes the price-setting dynamics more explicitly.
The clearing price corresponds to the marginal cost of the last dispatched (at-the-money, ATM) plant. Generators with lower costs are in-the-money (ITM) and are dispatched; those with higher costs are out-of-the-money (OTM) and idle. This setup enables the identification of price-setting technologies and their marginal influence (see Figure (ref).a).
The model also reveals fuel-switching dynamics (see Figure (ref).b), which occur when relative cost changes alter the dispatch order. For example, during the European energy crisis in mid-2021, surging gas prices triggered a switch from oil to gas. This trend reversed at the beginning of 2023 as gas prices normalized. By late 2024, fuel prices had stabilized. According to the data-driven merit order model, oil remained the most expensive fuel, followed by gas, coal, and lignite. Nuclear and lignite continued to serve as base-load technologies due to their low variable costs and were largely unaffected by fuel switches.
We propose a novel data-driven merit order model that combines the interpretability of fundamental models with the flexibility of data-driven approaches. It directly embeds the classical merit order, meaning that if the expert-defined fundamental model were optimal, the algorithm would select it. Thus, the classical model becomes a nested special case within our framework. By optimizing key parameters using historical data, the proposed model significantly improves price forecasting accuracy—outperforming state-of-the-art methods like Lasso, XGBoost, and neural networks, even without autoregressive features. We introduce several impactful extensions to the merit order model, including capacity corrections for underreportings, hydro power modeling, imports/exports modeling, technology splits (e.g., gas split into CCGT vs. OCGT), and must-run shares.These embedded extensions enable the model to better capture physical system behavior and price-setting mechanisms. Though now calibrated to data, the model remains a fundamental model in essence, retaining the ability to explain causal drivers such as marginal plant types, dispached plants, production and fuel switches, which are inaccessible to traditional machine learning models. Its structure allows for clear interpretation and inspection of economic effects.
To our knowledge, this is the first full-fledged, intrinsic integration of fundamental and data-driven modeling in electricity price forecasting. The results are highly promising and open up a new frontier for research and application in the field. The model provides a solid foundation for future extensions, including a more accurate reflection of policy constraints (e.g., RES subsidies), more refined modeling of cross-border flows, probabilistic forecasting, the inclusion of storage and market coupling effects, extension of the forecasting horizon and smoothening of the merit order curve.
During the preparation of this work the authors used GitHub Copilot and OpenAI ChatGPT-3.5 in order to generate rephrasing suggestions to improve the language and readability of this paper. After using these services, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.
Funding: This work was supported by the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft) project numbers 505565850 and 520388526 to P.G. and F.Z.
The raw data required to reproduce the findings in this work include the freely available resources: day-ahead power prices, power plant capacities, generation and unavailabilities, power imports and exports, actual values and day-ahead forecasts for load, solar, onshore wind, and offshore wind infeed. This data can be downloaded from the ENTSOE Transparency Platform (https://transparency.entsoe.eu/). Data not freely available include daily closing prices for futures of gas, coal, oil, and European Emission Allowances (EUA), accessible via the information platform Refinitiv Eikon at a cost (https://eikon.refinitiv.com/).