Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
82,014 characters · 26 sections · 95 citation commands
Adaptive Pricing in Insurance: Generalized Linear Models and Gaussian Process Regression Approaches
We study the application of dynamic pricing from the perspective of an insurance company. Here the insurance company looks to set prices to optimize the long-run revenue from selling insurance products which also experience a distribution of claims. This can be cast as a revenue management problem, see Philips PhillipsRobert2005 and Talluri and Ryzin TalluriRyzin2005 for an overview of this field. If the distribution of demand and claims was known to the insurance company, then this could be formulated as a relatively straight-forward optimization problem. However, in the real world, demand for a new product at each price is not deterministic and known. Thus, we assume that the insurance company only observes the realised demand and does not know the underlying distribution of demand and claims for the insurance product. This is particular relevant for the release of new insurance products.
Given demand and claims are not known, the retailer faces a learning & pricing problem. This is sometimes known as a exploration-exploitation trade-off. At the beginning of each selling period, we set a price close to the estimated best price and then study the changes of demand and claims when varying prices. This is the exploration process, which enables us to find the relationship between price and demand & claims distributions. Further by setting the price close to the estimated best price, we are able to exploit what we have learned. This is the exploitation process. Choosing prices that are far from the best estimated price encourages exploration but can be inefficient in exploiting available information. On the other hand, choosing pricing close to the best estimate price may not discover enough about the underlying distribution of claims to converge to the optimal price. Therefore, the insurance company must create a policy for pricing the insurance product that reveals sufficient information about the underlying demand and claims distributions so as to optimize the long-run revenue of the insurance company. The policy that we consider provides a mechanism for efficiently exploring different prices offering, and then exploiting that knowledge to achieve the revenue maximization objective.
We consider this pricing problem as a multi-armed bandit problem, which has been widely used to address the trade-off between exploration and exploitation in sequential decision making. We investigate two regression models for the learning & pricing problem: the Generalized Linear Models (GLM) and the Gaussian Process (GP). GLMs are a classical statistical technique, introduced by Nelder and Wedderburn NelderWedderburn1972, and was first applied in insurance rating by McCullagh and Nedler McCullaghNelder1989. However, for sequential decision making, prices derived from maximum likelihood estimates may not be consistent due to insufficient exploration LaiRobbins1979. To solve this problem, the strong consistency of least squares estimates is required, which is established by Lai and Robbins LaiRobbins1981, LaiRobbins1982 and further generalized by Lai and Wei LaiWei1982. This analysis is a crucial step in the field of online estimation and optimization. Lai Lai2003 gives a comprehensive survey of several related developments and discussions. Its application to revenue management is given by den Boer and Zwart denBoerZwart2013, in which bounds on cumulative regrets are analyzed as a measure of performance. Here, regret is defined to be the difference between the expected revenue and the optimal revenue. We develop these models for the setting of insurance where there are both demand and claims. We, then, consider a second approach to this problem based on a Bayesian optimization. It was proposed by Mockus Mockus1989, Mockus1994 for optimizing an unknown function using a Gaussian Process. A Gaussian Process is a generalization of the Gaussian probability distribution, where random variables are modelled by stochastic processes. Over the last two decades, GPs have been widely used in machine learning. We investigate the upper-confidence bound (UCB) approach taken by Srinivas et al.\@ Srinivas2010, Srinivas2012. By maximizing the UCB aquisition function, we can determine the price at each time period. For more details on Gaussian Process regression and Bayesian optimization in general, we refer to Rasmussen and Williams RasmussenWilliams2006 and Brochu Brochu2010.
In a summary, the contributions of this work are as follows:
Dynamic pricing and online learning have been successfully applied in a variety of industries such as airline ticketing, hotel bookings, car rentals, and fashion. However, to the best of our knowledge, online learning has not been applied to insurance pricing in any literature. Thus, motivated by the powerful machine learning techniques and the growing applicability insurance industry, this paper is among the first to investigate these methods to address problems in insurance pricing. As insurance increasingly sold online and with insurance products continually changing, we believe that these methods will be important for actuaries now and in the future swissre2015.
In this section, we provide a brief review of insurance pricing and dynamic pricing. We also highlight related work on applying online learning to two statistical models: Generalized Linear Models and Gaussian Processes. Finally, we discuss the previous work on revenue management with uncertainty. \paragraph{Insurance Pricing and Dynamic Pricing} Many researchers such as B{\"{u}}hlmann Buhlmann1970, McClenahan McClenahan2001, Jong and Heller JongHeller2008 point out that mathematical and statistical methods are needed to support actuaries to make pricing decisions. The linear models have been applied extensively in actuarial work. For example, early literature uses linear models in motor insurance, see Baxter Baxter1980 and Coutts Coutts1984. In 1960, Bailey and Simon BaileySimon1960 introduce the minimum bias technique in classification ratemaking, which is an important milestone in non-life insurance pricing development David2015. In the 1980s, British actuaries introduced GLMs to insurance pricing and this has now become a standard approach in many countries OhlssonJohansen2010. A good overview of the use of GLMs in different situations in actuarial work is available in Haberman and Renshaw HabermanRenshaw1996; for further research applying GLMs in insurance pricing see JongHeller2008,OhlssonJohansen2010,KaasGDD2009,Frees2010. In the last few years, the non-life insurance market has changed due to the increase of online services. Machine learning techniques have become more popular in applications in the insurance sector. These enhance and supplement the standard GLMs analysis. We refer to W{\"{u}}thrich and Buser WuthrichBuser2018 for an overview and insight into GLMs and machine learning methods in non-life insurance pricing.
Dynamic pricing is the study of how demand responds to prices in a changing environment. In recent decades, interest in dynamic pricing has grown rapidly. Early profit optimization problems assume sellers have complete knowledge of the market, which means demand functions are known or can be found from previous selling experience. Evans Evans1924, Evans1930 is one of the first to propose a dynamic pricing model by adding time derivatives of prices to a static model. Greenleaf Greenleaf1995 numerically shows the significant effects of reference prices and develops an optimal dynamic pricing strategy in a monopoly setting. Kopalle et al.\@ KopallePraveen1996 analytically generalize these results to a duopoly and an oligopoly settings. Following this, Fibich et al.\@ FibichGadi2003 then calculate explicitly the optimal pricing strategy in various nonsmooth optimization problems. All of these works assume the demand function of consumers is deterministic and known. Surveys by Aviv and Vulcano AvivVulcano2012 and den Boer DenBoer2015 provide an excellent overview of this area.
\paragraph{Adaptive Generalized Linear Models} Nelder and Wedderburn NelderWedderburn1972 first introduce Generalized Linear Models (GLM), which is an extension to classical linear regression. As discussed above, it has become a well-established and standardised statistical technique to price the insurance products OhlssonJohansen2010, Wuthrich2017. In the GLM framework, maximum likelihood estimation is a commonly used technique to find the parameters of a given Generalized Linear Models. Wedderburn Wedderburn1974 proposes a method named quasi-likelihood estimation, an extension of likelihood estimations but only the first two moments of the observations are needed. McCullagh and Nedler McCullaghNelder1989 then apply GLMs with quasi-likelihood estimation to insurance ratemaking. They fit a GLM to different types of data, including average claim costs for a motor insurance portfolio and claims frequency for marine insurance.
To price products, often a certainty equivalence rule is used. Here the optimal price is chosen for the estimated parameters. Thus when optimizing we treat estimates as if they were the true (unknown) parameters of the model. Anderson and Taylor AndersonTaylor1976 apply a certainty equivalence rule to solve a multiperiod control problem. However, strong consistency may not hold when applying a certainty equivalence rule to maximum quasi-likelihood estimates LaiRobbins1982, Lai2003. To deal with this problem, conditions are proposed to ensure the strong consistency for parameters estimators LaiRobbins1979, LaiRobbinsWei1979. Lai and Robbins LaiRobbins1981 introduce further conditions for an adaptive design, and Lai and Wei LaiWei1982 generalize these conditions to multiple regression models with errors given by a martingale difference sequence. Chen et al.\@ ChenHY1999 extend the results of LaiWei1982, LaiRobbinsWei1979 to GLMs under both fixed and adaptive designs.
\paragraph{Multi-armed Bandits and Bayesian Optimization}
A multi-arm bandit problem refers to a broad class of sequential decision making problems. At each time step, one must choose an arm amoungst a set of arms, each of which has unknown rewards. There is a trade-off between exploration, i.e. estimating the distribution of rewards for all arms in the past, and exploitation, i.e. choosing the arm with higher expected reward. Bubeck and Cesa-Bianchi BubeckBianchi2012 present a comprehensive review of work on multi-armed bandit problems. In multi-armed bandits problem, the upper confidence bound (UCB) rule is commonly used to select arms at each time period. The UCB algorithm constructs a confidence interval for the mean of each arm, and then chooses the arm that maximizes revenue under this estimation. The UCB strategy is introduced by Auer et al.\@ Auer2002 to address a specific bandit model, and is used for asymptotic analysis of regret as first discussed in Lai and Robbins LaiRobbins1985. Regret bounds for multi-armed bandits problems have attracted a great deal of interest in different cases, such as linear models Dani2008, RusmevichientongTsitsiklis2010, Generalized Linear Models Filippi2010, Lipschitz functions Bubeck2011, Kleinberg2008, Gaussian Process Srinivas2010 and Thompson Sampling Agrawal2012, Agrawal2012b, Russo2013. In the insurance context, each price is an “arm” and its revenue is the “reward”.
Bayesian optimization Mockus1978 provides an efficient approach to address global optimization of an unknown potentially random or noisy function. It is applicable and efficient when objective functions are unknown or are expensive to evaluate. There are two significant stages in Bayesian optimization. The first stage is to learn the objective function from available samples. Bayesian optimization typically works by assuming the unknown function is sampled from a Gaussian Process (GP) Snoek2012. The second stage is to optimize a acquisition function to determine the next sampling points for the evaluation of the objective function. High acquisition function values occur either because there is large uncertainty in the objective function (exploration) or a high prediction given by the model (exploitation). Srinivas et al.\@ Srinivas2010 consider a GP-based Bayesian optimization. In this work, the authors propose a Gaussian Process upper confidence bound (GP) algorithm, where they sample the reward function from a GP and apply a UCB algorithm to bound the regret. They achieve sublinear regret in terms of the maximum information gain, the maximum amount of informarion the algorithm could learn about the reward function. For a comprehensive review of the Bayesian optimization and its applications, we refer to Brochu et al.\@ Brochu2010. To the best of our knowledge this is the first paper to consider Bayesian optimization in the context of insurance.
\paragraph{Revenue Management with Unknown Demand} Finally we discuss developments on revenue management with uncertainties. Gallego and van Ryzin GallegoRyzin1994 introduce a single-product dynamical pricing to revenue management. Subsequent works have adapted this model to allow for unknown demand. One popular case is to consider a parametric setting, where demand can be modeled with fixed but not known parameters. Aviv and Pazgal AvivPazgal2005 are among the first to consider model uncertainty. They derive a closed form model with a single unknown parameter and assume that the arrival of consumers follows a Poisson distribution. Harrison et al.\@ HarrisonJ2012 improve the learning and profit performance by a new method named the myopic Bayesian policy. Broder and Rusmevichientong BroderRusmevichientong2012 present a maximum-likelihood based model for the analysis of regret in dynamic pricing problems with a general parametric model. They show that in a general case, upper bound of the $T$-period regret is $O(\sqrt{T})$. den Boer and Zwart denBoerZwart2013 propose a controlled variance pricing policy, in which they create taboo intervals around the average of previously chosen prices to ensure sufficient price dispersion. This policy is the first to consider the parametric model with unknown demand by maximizing the quasi-likelihood estimation. They obtain an asymptotic upper bound on $T$-period regret as $O(T^{1/2+\delta})$, where $\delta >0$ is extremely small. This work forms the base of our insurance pricing model.
The pricing problem can also be addressed in a nonparametric way. Kleinberg and Leighton KleinbergLeighton2003 provide an analysis of an online auction and introduce regret to measure of the performance of a pricing strategy. Cope CopeEric2007 applies a nonparametric Bayesian approach using Dirichlet distributions as priors to achieve a revenue-maximizing goal in an e-commerce market. Rusmevichientong et al.\@ RusmevichientongVanRoy2006 develop a nonparametric approach to a multiproduct pricing problem based on a real automobile data set. Besbes and Zeevi BesbesZeevi2009, BesbesZeevi2012 use blind pricing policies to balance exploration-exploitation trade-offs and achieves asymptotically optimal.
\paragraph{Bandit Online Learning Problem} Traditionally, delays can be considered as a fixed constant. Under this setting, Dudik et al.\@ Dudik2011 provide an efficient algorithm for stochastic contextual bandits and show that regret is additive. Chapelle and Li ChapelleLi2012 present the influence of delayed feedback for contextual bandits in news article recommendation. Cesa--Bianchi et al.\@ CesaBianchi2016 study networks of nonstochastic bandits. Pike-Burke et al.\@ PikeBurke2018 discuss the case with delayed, aggregated anonymous feedback and the expected delay is known. They assume only the sum of regret is available while individual regret is unknown. In general, delays may be a stochastic process. Agarwal and Duchi AgarwalDuchi2011 analyze stochastic gradient-based optimization algorithms when delays are i.i.d randomly distributed. Desautels et al.\@ Desautels2012 study parallel experiments with a bounded delay between an experiment and observation in a Gaussian Process bandit problems. Vernade et al.\@ Vernade2017 consider infinite stochastic delays where some feedback can not be observed after a threshold. For a systematic study of online learning with delayed feedback and the effects of delay on regret, we refer to Joulani et al.\@ Joulani2013. In their work, they show that delays additively increases regret in stochastic problems without requiring knowledge on distributions of delays.
The sections of the paper are structured as follows. In Section (ref), we describe the optimization pricing problem in the insurance setting and define our pricing models. We study GLM and GP models, as well as assumptions and estimation methods associated with each of these models. In Section (ref), we propose GLM and GP pricing algorithms and explain how they work, respectively. The main result of this paper is presented in Section (ref). We consider bounds on cumulative regret, which help to measure the performance of each pricing policy. In Section (ref), we extend both models with unknown delayed claims. Section (ref) illustrates an experimental set-up and numerical results . Finally, a conclusion and discussion of future work are provided in Section (ref). Auxiliary results and proofs are gathered in the \nameref{Appendix}.
In this section, we introduce two regression models and important assumptions. We give a brief overview of the insurance pricing problem in Section (ref). In Section (ref), we discuss an adaptive Generalized Linear Model (GLM), which is an parametric model, and explain how to estimate unknown parameters by quasi-likelihood estimation. This model is built on ideas of den Boer and Zwart denBoerZwart2013, and Lai and Wei LaiWei1982. In Section (ref), we introduce an adaptive Gaussian Process (GP) model with an UCB rule.
We consider an insurance company which sells a single product over a selling horizon $T > 0$. The selling price $p_{t}$ is determined at the beginning of each time period $t \in \{0,\dots,T\}$. We define the set of acceptable prices by $\mathcal{P} = [p_l, p_h]$, where $0 < p_l < p_h$ are the minimum and maximum selling prices.
We assume that dynamic pricing is only associated with past prices. Given a determined selling price $p_{t}$ at time period $t$, the insurance company observes demand $d_{t} = D_{t} (p_{t})$, which is independent realisations of the random demand function $D(\cdot)$ for the selling price $p_{t}$. Similarly, we denote the total claims as $C_{t} (p_{t})$ during the time period $t$ and observe the total claims $c_{t} = C_{t} (p_{t})$. (Often we will suppress the sub-script $t$ from i.i.d. random variables $D(\cdot)$ and $C(\cdot)$.)
In insurance, the premium is the expected income that the insurance company earns and claims are the amount that the insurance company loses. If the selling price is known, the revenue collected in a single time period $t$ is $p_{t} d_{t} - c_{t}$. The expected revenue at time $t$ is given by
Both demand and total claims respond to changes in prices at each time period simultaneously. Once the price is specified, we assume the demand and total claims are independent of each other. The insurance company aims to find an optimal pricing policy that generates maximum revenue, based on previous selling prices $\{p_i: \, i = 1, \dots, t-1\}$ and observations $\{d_i \,, c_i: \, i = 1, \dots, t-1\}$. We use cumulative regret to measure the performance of pricing policies. The regret is the expected revenue loss caused by not using the optimal price. More formally, we define the cumulative regret over time horizon $T$ as
Here, $r({p}^{*})$ is the revenue generated by the optimal price ${p}^{*}$:
The objective of the seller is to maximize the sum of revenue, that is, to minimize the cumulative regret.
We first consider the dynamic optimization pricing problem in a GLM setting. Here the expected revenue can not be calculated directly because it depends on unknown parameters that must be inferred. We apply the maximum quasi-likelihood estimation (MQLE) to estimate the unknown parameters in the model. We are concerned about the strong consistency for MQLE of regression parameters in the Generalized Linear Models (GLMs).
Our model is based on the work of den Boer and Zwart denBoerZwart2013. However, we consider a single insurance product with demand and heavy-tailed claims claims. Here, we use the log of claims to describe large insurance claims.
We assume that the insurance company knows the functional forms of the first two moments of demand and claims. The model for demand distribution at time $t$ is given by
Similarly, we assume the log of total claims is with expectation and variance, given by
Here, parameters $a_0, a_1$ and $b_0, b_1$ are all unknown. Notice here that we take the logarithm of total claims which is slightly non-standard when compared with results in revenue management. In the context of insurance this can be used to model heavy-tailed claims distributions, such as the log-normal distribution.
We consider functions $h_{1}(\cdot), h_{2}(\cdot)$ are known link functions of price $p$ and unknown parameters. The variance functions $v_{1}(\cdot), v_{2}(\cdot)$ are the variance of the expected demand & claims. The variances of the randomly distributed demand & claims are functions of the variance functions with constants $\sigma_{1}, \sigma_{2} > 0$. Functions $h(\cdot)$ and $v(\cdot)$ are twice continuously differentiable with first and second derivatives denoted by $\dot{h}(\cdot), \ddot{h}(\cdot)$ and $\dot{v}(\cdot), \ddot{v}(\cdot)$, respectively. The link function is called a canonical link function when $\dot{h}(x) = v (h)$, otherwise it is called a general link function.
Denote $\bm{a} = (a_{0}, a_{1})^{\top}$ and $\bm{b} = (b_{0}, b_{1})^{\top}$, the expected revenue in (ref) can be written as
Moreover, the cumulative regret in (ref) after $T$ time periods becomes
Here, the optimal price ${p}^{*}$ is defined in (ref).
Finally, we define the design matrix $P_{t}$ to be the sum of the transpose matrices achieved from price vectors $\bm{p}(i) = (1, p_i)^{\top}$ for $i = 1, \dots, t$. For $\bm{p} \in \mathcal{P} \times \{1\}$, the design matrix is given by
We denote the largest eigenvalue of the design matrix $P_{t}$ as $\lambda_{\max} (t) = \lambda_{\max} (P_{t})$ and denote the smallest eigenvalue as $\lambda_{\min} (t) = \lambda_{\min} (P_{t})$.
The optimal policy cannot be calculated directly because regret depends on unknown parameters $\bm{a}, \bm{b}$. To simplify the notations, we define parameter matrix as $\bm{\beta} = \left(\bm{a}, \bm{b}\right)$. And we use $\bm{\beta}_{0}$ to denote the true values of regression parameter $\bm{\beta}$. The maximum quasi-likelihood estimators, denoted by $\widehat{\bm{\beta}}_{t}$, are solutions to
Let the filtration $(\mathcal{F}_{t})_{t\in \mathbb{N}}$ be generated by $\{p_i \,,d_i \,, c_i: \, i = 1, \dots, t-1\}$ for each $t$. Write $\eta_{i} = y_{i} - {h (\bm{p}^\top(i)\widehat{\bm{\beta}}_{t})}$. The error terms $\eta_{i}$ form a martingale difference sequence w.r.t.\@ $\mathcal{F}_{t}$, that is, $\eta_{i}$ is $\mathcal{F}_{t}$-measurable and $\mathbb{E}[\eta_{i} \,| \,\mathcal{F}_{i-1}] = 0 $. We also assume that for some $\gamma > 2$ almost surely,
Now, we construct a Bayesian model by sampling the expected demand and expected total claims from Gaussian Processes (GP). Our pricing model is an extension of Srinivas et al.'s\@ Srinivas2010 with an alternative UCB rule to the field of insurance where demands and claims are considered.
First, we offer a brief introduction of Gaussian Process regression and more complete details can be found in Rasmussen and Williams RasmussenWilliams2006. A Gaussian Process is a collection of random variables, any finite number of which have a joint Gaussian distribution. It is completely specified by its mean function $\mu({p})$ and covariance function (or kernel) $k({p}, {p}')$ given by
Then, we can generate a Gaussian Process as
Without loss of generality, we assume that mean is a constant and covariance function is strictly bounded.
For a noisy sample $\bm{y}_{T} = [y_{1}, \dots, y_{T}]^{\top}$, given a collection of input points $\{{p}_{1}, \dots, {p}_{T}\}$. We define ${p}_{t}$ as the $t$-th sample and ${y}_{t} = f({p}_{t}) + \varepsilon_{t}$, here $\varepsilon_{t} \sim \mathcal{N}(0,\,\sigma^{2})$ is independent and identically distributed Gaussian noise with variance $\sigma^{2}$. As Gaussian Process can describe a distribution over functions, we use $\mathcal{GP}(\mu({p}), k({p}, {p}'))$ as the prior distribution over $f$. The posterior over $f$ is also a GP distribution with mean $\mu_{T}({p})$ and covariance function $k_{T}({p},{p}')$ given by
where $\bm{k}_{T}({p}) = [k({p}_{1},{p}), \dots, k({p}_{T},{p})]^{\top}$ and covariance matrix $\bm{K}_{T}$ is the positive definite matrix whose entries are $\bm{K}_{i,j} = k({p_{i}},{p}_{j})$ for $i, j = 1, \dots, T$.
The kernel determines how observations influence the prediction of nearby or similarity inputs. There are two commonly used kernels: the squared exponential kernel $k_{\sigma, l}$ and the Mat\'{e}rn kernel $k_{\nu, l}$, given by
Here, $l$ is the length-scale and $\nu$ is the smoothness parameter. Moreover, $\Gamma(\cdot)$ is the Gamma function and $B_{\nu}(\cdot)$ is the modified Bessel function of the second kind of order $\nu$. Note that the Mat\'ern kernel reduces to the exponential kernel $k_{\nu, l}(r)= \exp \left(-{r }{l}\right)$ when the smoothness parameter $\nu = 1/2$ and reduces to the squared exponential kernel when $\nu \rightarrow \infty$.
We define the function of expected demand at price $p_{t}$ as $f_{d}(p_{t})$, and similarly define the expected claims as $f_{c}(p_{t})$. Functions $f_{d}(\cdot), f_{c}(\cdot)$ are independently sampled from GPs with known means $\mu_{d}, \mu_{c}$ and kernels $k_{d}(p,p'), k_{c}(p,p')$. That is, $f_{d} \sim \mathcal{GP} (\mu_{d}, k_{d})$ and $f_{c} \sim \mathcal{GP} (\mu_{c}, k_{c})$. The posteriors over $f_{d}$ and $f_{c}$ are GPs and also follow GP posterior update in (ref).
The expected revenue function given a determined price $p_{t}$ at time $t$ is
Here, the noise term $\varepsilon_{r} \sim \mathcal{N}(0, \sigma_{r}^2)$ is a combination of demand noise and claims noise.
We can see that $r(\cdot)$ is sampled from a GP as well, with an additive kernel. This is because the sum of GPs is also a GP, and the kernel has the form of a direct sum. Then, we have $r \sim \mathcal{GP}(\mu_{r}, k_{r})$ with known $\mu_{r} = p \cdot \mu_{d} - \mu_{c}$ and $k_{r} = p^2 \cdot k_{d} + k_{c}$. The cumulative regret over time horizon $T$ becomes
Our model is an extension of Srinivas et al.'s\@ Srinivas2010 work to the field of insurance. Srinivas et al.\@ Srinivas2010 study a only one function $f$. In our case, we consider an additive form, that is the revenue function $r$ contains two components: $f_{d}(\cdot)$ and $f_{c}(\cdot)$ and is sampled with an additive kernel. Here, the samplings of $f_{d}(\cdot)$ and $f_{c}(\cdot)$ are independent of each other.
In this section, we propose two algorithms called “GLM Pricing Algorithm" and “GP Pricing Algorithm" in Section (ref) and Section (ref), respectively.
A popular pricing policy is certainty equivalent pricing, proposed by Anderson and Taylor in a simple linear regression model AndersonTaylor1976 and further developed by the authors in a more general multiple regression model AndersonTaylor1979. We denote the certainty equivalent price by $\bm{p}_{cep}$. It is efficient provided current parameter estimates are correct. Given MQLE $\widehat{\bm{\beta}}_{t}$, $\bm{p}_{cep}$ is the price that maximizes the expected revenue, given by
Certainty equivalent pricing is popular, essentially because it separates the statistical problem from the problem of optimizing revenue and reward. However, it is possible that parameter estimates converge to incorrect values due to convergence being too quick. Specifically, Lai and Robbin LaiRobbins1982 prove that inconsistency may occur for a linear demand function with constant variance under an iterative least squared policy. den Boer and Zwart denBoerZwart2013 further relax the assumptions on the values of prices and propose a variant certainty equivalent pricing: controlled variance pricing policy. By creating taboo intervals around mean prices that have been chosen, they choose the next price outside these intervals to obtain more information. Keskin and Zeevi KeskinZeevi2014 also point out that the certainty equivalent pricing has poor performance of due to the existence of “uninformative” price and therefore introduce a constrained iterative least squared policy.
Based on the work of den Boer and Zwart denBoerZwart2013, we propose a dynamic pricing policy with an additional constraint on the smallest eigenvalue of the design matrix with the aim to control the convergence of the pricing policy. If we ensure a lower bound on $\lambda_{\min}(t)$, we can say that there is sufficient price dispersion guaranteeing the strong convergence of the MQLE. However, there is no simple explicit relation between $\lambda_{\min}(t)$ and $\lambda_{\min}(t+1)$. We introduce the inverse of the trace of the inverse design matrix $\operatorname{tr}(P_{t}^{-1})^{-1}$. For any $n \times n$ positive definite matrix $A$, we have $\operatorname{tr}(A^{-1})^{-1} \leq \lambda_{\min}(A) \leq n \operatorname{tr}(A^{-1})^{-1}$.
Let $\bm{v}_{1}, \bm{v}_{2}$ be associated unit eigenvectors, which are an orthonormal basis of $\mathbb{R}^{2}$. For $t > 2$, the optimal price can be written as a linear combination of these unit eigenvectors, that is $\bm{p}_{cep} = \sum_{i=1}^{2}\alpha_{i}\bm{v}_{i}$.
Let $\mathcal{L}$ be a class of positive differentiable monotone increasing functions $L$ such that $L(t)\to \infty$ and $t \to \frac{1}{L(t)}$ is convex. Choose a function $L_1(t) \in \mathcal{L}$, and let $\operatorname{tr}(P_{t}^{-1})^{-1}\geq L_{1}(t)$ for all $t \geq 2$, then we have
A proper choice of function $L_1(t)$ guarantees sufficient price dispersion, and therefore, guarantees the convergence of parameter estimates to the true parameters, as well as the asymptotical convergence of our price sequence to the optimal price. Specifically, we present the optimal regret bounds when $L_{1}(t) = c \sqrt{t\log t}$ for some $c >0$.
Now, we show how the pricing algorithm works. First, we choose three linearly independent initial price vectors to estimate the unknown parameters ${\widehat{\bm{\beta}}_{t}}$ by (ref). If the MQLE ${\widehat{\bm{\beta}}_{t}}$ does not exist or the constraint on the price dispersion $L_{1}(t)$ is not met, we repeat previous prices until we can find solutions of ${\widehat{\bm{\beta}}_{t}}$ to the MQLE and satisfy the sufficient price dispersion requirement as shown in (ref) in Algorithm (ref). Condition $\operatorname{tr} ({P_{t+j}}^{-1})^{-1} \geq L_{1}(t+j)$ will eventually be met because $j$ is always finite. If the MQLE ${\widehat{\bm{\beta}}_{t}}$ exist and the constraint on the price dispersion $L_{1}(t)$ is also satisfied, we set the next price to be the certainty equivalent price $\bm{p}(t+1) = \bm{p}_{cep}$, and check whether the condition in (ref) in Algorithm (ref) holds or not. If it does not hold, we then choose $\bm{p}(t+1) = \bm{p}_{cep} + \bm{\phi}_{t}$. Here,
and $v_{2,1}$ is the first component of $\bm{v}_{2}$. Constant $K>0$ must satisfy $\left|K \sqrt{\dot{L}_{1}(t)}\right| \leq 1 $. These pricing steps are given in Algorithm (ref).
The following proposition guarantees that when prices are chosen, the price dispersion condition (ref) is satisfied.
In the GP setting, we determine our pricing policy by the upper confidence bound (UCB) rule. We start by reviewing Srinivas et al.'s work Srinivas2010, where the posterior GP is used to construct a UCB function. At each time step $t-1$, they set the next sampling point to be the one that maximizes the UCB function, given by
Here, the posterior mean ${\mu}_{t-1}(\cdot)$ is obtained after $t-1$ observations and is the current estimate of $f$. The posterior standard deviation ${\sigma}_{t-1}(\cdot)$ is the uncertainty associated with this estimate. Term ${\mu}_{t-1}(\cdot)$ is the explicit exploitation on what we have known and ${\sigma}_{t-1}(\cdot)$ is the exploration on what we haven't known. Parameter $\varphi_{t}$ is used to balance the trade-off between exploitation ${\mu}_{t-1}(\cdot)$ and exploration ${\sigma}_{t-1}(\cdot)$.
In our work, we update the UCB function with additive mean and kernels, defined by
Here,
Now, we present the implementation of GP algorithm for pricing. At time $t-1$, the algorithm generates a price that maximizes the UCB function, which is the optimal price ${p}^{*}$. Each price is determined by the history $\mathcal{F}_{t-1}$ and the policy followed by the UCB rule. In the continuous price set, the optimal price ${p}^{*}$ exists and is unique. Then, we set the optimal price to be the next sampling price ${p}_{t}$. Given the price ${p}_{t}$, we sample $f_{d}({p}_{t})$ and $f_{c}({p}_{t})$ individually. Since the revenue is determined by two parts: premium ${p}_{t} \cdot f_{d}({p}_{t})$ and total claims $f_{c}({p}_{t})$ and chosen price can be considered as a constant, the sampled revenue is a linear combination of sampled functions $f_{d}({p}_{t})$ and $f_{c}({p}_{t})$. Therefore, we can obtain the sampled revenue function. Next, we perform the GP posterior update (ref) to obtain posterior distributions for demand and total claims and update the posterior distribution of $f_{d}$ and $f_{c}$ given $\mathcal{F}_{t} = \mathcal{F}_{t-1}\cup \{p_{t}, d_{t}, c_{t}\}$, respectively. Finally, we obtain the mean ${\mu}_{t}^{r}$ and standard variance ${\sigma}_{t}^{r}$ defined in (ref) and then updated the UCB function to determine the next selling price. The pseudocode for pricing a new released insurance product via GP pricing algorithm is shown in Algorithm (ref).
In this section, we show theoretical analysis obtained by proposed approaches. Our main results on cumulative regret are stated in Theorem (ref) in Section (ref) and Theorem (ref) in Section (ref), as well as proofs .
We show the strong consistency and convergence of the maximum quasi-likelihood estimators in Generalized Linear Models (GLMs). Lai and Wei LaiWei1982 study the least-squares estimate in linear stochastic regression models. Chen et al.\@ ChenHY1999 extend the results of Lai and Wei LaiWei1982 to GLMs with canonical link functions. Regret bounds depend on the lower bound on the smallest eigenvalue $\lambda_{\min}(t)$ of the design matrix $P_{t}$ and the expected value of difference between the chosen prices and optimal prices.
We will first show that studying the bound on the regret is equivalent to studying the bound on $\left\|{\bm{\beta}} -{\bm{\beta}}_{0}\right\|^2$.
We focus on a simple case, where link functions are canonical, that is, $\dot{h}(\cdot) = v (h(\cdot))$. This gives
Under the Assumptions A(ref) and A(ref), we show the MQLE $\widehat{\bm{\beta}}_{t}$ eventually exists and this estimator is also strongly consistent.
{Theorem (ref) provides an upper bound on the regret in terms of the function $L_{1}(t)$.}
When $L_{1}(t) = c \sqrt{t \log(t)}$, we find an optimal pricing strategy that achieves $\operatorname{Regret} (T) = O \left(\sqrt{T\log{T}}\right)$. The regret bound that we obtained corresponds to the results in Kleinberg and Leighton KleinbergLeighton2003 but with different algorithms.
We now establish cumulative regret for the Gaussian Process (GP) Bayesian optimization, which is bounded by the maximum information gain $\gamma_{T}$. Suppose the subset $\mathcal{A} = \{p_{1}, \dots, p_{T}\} \subset \mathcal{P}$ is finite. Let $\bm{y}_{\mathcal{A}} = f(\bm{p}_{\mathcal{A}}) + \bm{\varepsilon}_{\mathcal{A}}$ denote the observations and let $f_{\mathcal{A}}$ denote the function values at these points, that is $f_{\mathcal{A}} = f(\bm{p}_{\mathcal{A}})$. We introduce the Shannon Mutual Information and denote it as $I(\cdot)$. For a Gaussian Process, $I\left(\bm{y}_{\mathcal{A}}; f_{\mathcal{A}}\right) = H(\bm{y}_{\mathcal{A}}) - H(\bm{y}_{\mathcal{A}}|f)$, where $H$ is the entropy. In the multivariate Gaussian case, $H(\mathcal{N}(\mu, \bm{\Sigma})) = \frac{1}{2} \log |2\pi e \bm{\Sigma}|$, so that $I\left(\bm{y}_{\mathcal{A}}; f_{\mathcal{A}}\right) = \frac{1}{2} \log |\bm{I} + \sigma^{-2} \bm{K}_{\mathcal{A}}|$, where $\bm{K}_{\mathcal{A}} = [k({p}, {p}')]_{\bm{p},\bm{p}'\in \mathcal{A}}$. After $T$ rounds, the maximum information gain between $\bm{y}_{\mathcal{A}}$ and $f_{\mathcal{A}}$ is
The bounds on information gains $\gamma_{T}$ depend on the kernels used. For example, $\gamma_{T} = O(d \log T)$ under a linear regression, $\gamma_{T} = O((\log T)^{d+1})$ under a squared exponential kernel, and $\gamma_{T} = O\left(T^{d(d+1)/(2\nu + d(d+1))}(\log T)\right)$ under a Mat\'{e}rn kernel with $\nu > 1$. We refer to Srinivas at al.\@ Srinivas2010 for more details. In our model, we denote the maximum information gain as $\gamma_{T}(k_{d}+k_{c})$, which describes the maximum amount of information that the algorithm could learn about the demand and total claims function.
Assume that both demand and total claims functions $f_{d}, f_{c}$ satisfy the following Assumption (ref). It allows us to derive the bound on the cumulative regret w.r.t.\@ the maximum information gain $\gamma_{T}(k_{d}+k_{c})$ in Theorem (ref).
The proof follows Srinivas et al.\@ Srinivas2010. We extend their model to the case of demands and claims. We present related lemmas and results from Srinivas et al.\@ Srinivas2010 in \nameref{Appendix} (ref).
Note that for the combination of additive kernels, we have $\gamma_{T}(k_{d}+k_{c}) \leq \gamma_{T}(k_{d}) + \gamma_{T}(k_{c})$ (see Lemma (ref) in \nameref{Appendix} (ref) for more details).
In previous sections, we assume that the insurance premium and total claims are paid at the beginning of each insurance period. This gives the basic outline of how these methods can be applied in an insurance setting. However, in the real world, claims are triggered when the insured events happen, thus are not paid out when an insurance product is purchase. An immediate pricing decision during the delay may give a wrong optimal price and increase the regret. As a consequence, we involve delayed claims here and consider our pricing problem as a delayed bandit problem, based on the idea of Joulani Joulani2016.
We assume that demands are generated and observed immediately when a price is chosen, while claims might be delayed. It might lead to delayed revenue. Suppose a price is chosen at time $t$ and denote the corresponding delayed time is $\tau_{t}$. We assume that $(\tau_{t})_{t \geq 1}$ is an i.i.d. sequence and is independent of prices and claims. It is possible that $\tau_{t} = 0$ when there is no delay. Assume there is a maximum waiting time $m$, that the delayed claims can only be observed by time $t + m$ or if $\tau_{t} \leq m$. Without loss of generality, we assume that all information (demands and claims) can be received by the end of time horizon $T$.
When delay occurs, the insurance company observes revenue $r_{t}$ at time $t + \tau_{t}$. Therefore, the next selling price $p_{t}$ is chosen based on history $\mathcal{H}_{t}$, which is the past information of prices and demands $\{p_i\,, d_i : \, i = 1, \dots, t-1\}$ and delayed claims $\{c_{i}: \, i + \tau_{i} \leq t-1 \}$. If $\tau_{t} = 0$ for all $t$, we have $\mathcal{H}_{t} = \mathcal{F}_{t}$.
If a price $p_{t'}$ is chosen at time $t'$, we denote the claims observed at time $t$ $(= t' + \tau_{t'} )$ by ${c}_{t' + \tau_{t'}}$. The insurance company collects the set of start times for delayed claims: $\mathcal{C}_{t} = \{t':\, t' = t - \tau_{t'}\}$ by the end of time $t$. Here, the delayed time set can be $\mathcal{C}_{t} = \emptyset$ for instance when $\tau_{t'} = 0$ or $\tau_{t'} > m$ for all $t'$. The corresponding observed revenue at time $t$ is $r_t = p_{t'} d_{t'} - {c}_{t' + \tau_{t'}}$. It is worth noticing that there is less information to decide $\bm{p}(t+1)$ due to the delay of claims.
We denote the time that the insurance company observes the $s$-th claim as $\rho(s)$ and the corresponding revenue as $r_{\rho(s)}$, here $s = 1, \dots, T$. Let $\widetilde{r}_{s} = r_{\rho(s)}$. The insurance company determines the next selling price $\widetilde{p}_{s + 1}$ after observing $\widetilde{r}_{s}$. For $t = 1, \dots, T$, we denote the number of claims observed by the end of time $t-1$ as $N(t) = \sum_{i=1}^{t-1} \mathbbm{1}\{{i + \tau_{i} < t}\}$. Since the company determines the next selling price by the latest observed claim, we have $p_{t} = \widetilde{p}_{N(t) + 1}$.
Let $\widetilde{\tau}_{s} = s - 1 - N(\rho(s))$. Here, $s - 1$ is the number of claims that can be observed if there is no delay. $N(\rho(s))$ is the actual number of claims that observed by time $\rho(s) -1$. Therefore, $\widetilde{\tau}_{s}$ is the number of delays that have not been updated when the algorithm chooses $p_{\rho(s)}$ at time $\rho(s)$ until the corresponding revenue $r_{\rho(s)}$ can be observed. Then we have $p_{\rho(s)} = \widetilde{p}_{s-\widetilde{\tau}_{s}}$ and $\sum_{s=1}^{T}\widetilde{\tau}_{s} = \sum_{t=1}^{T}{\tau}_{t}$ (for a proof we refer to Lemma (ref) in \nameref{Appendix} (ref)). Hence, the regret is
We can see that the regret with delayed claims has two parts: the non-delayed regret and an additional regret caused by delayed claims. We need to bound each of these two terms to get the overall regret bound.
In GLMs regression model without delays, we find the unknown parameters by maximizing the quasi-likelihood estimation. In the delayed case, we use a similar method. We denote the new least squares estimator of delayed claims by $\widehat{\bm{b}}_{t}'$. Specifically $\widehat{\bm{b}}_{t}'$ is the modified MQLE given by
The following result extends Theorem (ref) to delayed claims.
Currently, there do not exist theoretical bound for GP with delays. In this section, we present the implementation of an alternative GP pricing algorithm for insurance in a delayed-claim setting. For example, at each time $t$, we set a price by GP Algorithm (ref) and collect the set of start times for delayed claims: $\mathcal{C}_{t} = \{t':\, t' = t - \tau_{t'}\}$. For each time $t' \in \mathcal{C}_{t}$, the premium can be observed at time $t'$ given as ${p}_{t'} \cdot f_{t'}^{d}({p}_{t'})$, while claims are received at time $t'+\tau_{t'}$. We denote the delayed claims by $f_{t'+\tau_{t'}}^{c}({p}_{t'})$. After receiving the delayed claims, we update the GP for each claim and its premium. This then determines the next selling price. The pseudocode for pricing a new released insurance product with delays via GP algorithm is shown in Algorithm (ref).
In this section, we present the simulation result of the GLMs and GP pricing algorithms and both with delayed claims in numerical examples.
Assume demand follows logistics distribution and total claims follow lognormal distribution under canonical link functions. For example, let the demand and claims model be
Set three initial price vectors to be $\bm{p}(1) = (1, 3)^{\top}$, $\bm{p}(2) = (1, 3.3)^{\top}$ and $\bm{p}(3) = (1, 4.7)^{\top}$, with the lowest and highest price ${p}_{l} = 0.5$ and ${p}_{h} = 10$. We now go about estimating the parameters of the above optimization.
We plot price dispersion and convergence of parameter estimates. According to Figure (ref), the price dispersion $\operatorname{tr}(P_{t}^{-1})^{-1}/\sqrt{t \log(t)}$ converges to 0.01. Thus, we set $L_{1}(t) = 0.01 \sqrt{t \log(t)}$, which can guarantee sufficient price dispersion. Figure (ref) presents $\left\| \widehat{\bm{\beta}}_{t} - {\bm{\beta}}_{0}\right \|^2$, the squared norm of the difference between the parameter estimates and the true parameters. As $t \to \infty$, $\left\| \widehat{\bm{\beta}}_{t} - {\bm{\beta}}_{0}\right \|^2 \to 0$, that is, the parameter estimates converge to the true parameters. It implies the strong consistency of parameter estimates.
The most interesting kernels for machine learning are Mat\'{e}rn kernels with $\nu = 3/2$ and $\nu = 5/2$, given by
We sample random functions $f_{d}, f_{c}$ from GPs with Mat\'{e}rn kernels with $\nu = 1.5, 2.5$ and length scale $l=1$, respectively. Set sample noise to be $\sigma = 0.05$. The GP algorithm runs for $T = 100$ iterations with $\delta = 0.1$. This indicates that our choice of the parameter $\bm{\beta}$ depends on $t$.
In order to find the effects of delays, we consider same settings as those in non-delayed cases. Delays are non-negative integers and unknown, and we generate delays randomly.
We present the numerical results in Figure (ref) and Figure (ref). The cumulative regret incurred by the GLM pricing algorithm is plotted in (ref). The regret in delayed setting is always larger than that in non-delayed setting, which suggests that the insurance company suffers from an extra loss. As we mentioned above, in the delayed case, there is less information to decide selling prices due to the delay of claims. Figure (ref) illustrates the regret converges at the rate of $1/\sqrt{T \log(T)}$. The convergence rate of regret with delayed claims is asymptotic of the rate without delayed claims.
The results obtained by GP pricing algorithm in Figure (ref) is similar to GLM pricing algorithm. (ref) shows that regret in delayed setting is larger than the regret in non-delayed setting and (ref) shows that convergence rate of regret with delayed and without delayed claims are asymptotic.
By comparing Figure (ref) and Figure (ref), we can see that the cumulative regret obtained by GP pricing algorithm converges faster and gives smaller regret. Figure (ref) and Figure (ref) illustrate the same results that both algorithms achieve same convergence rate, that is, $1/\sqrt{T \log(T)}$. Overall, the GP pricing algorithm outperforms GLM pricing algorithm.
We considered dynamic learning and pricing problems in an insurance context. Our work is among the first to apply online learning to insurance. We found both GLM and GP pricing models have good performance. Applying prior work to an insurance setting with both demand and claims, we demonstrate this theoretically with precise bounds on regret. In numerical results, we found GP pricing has better convergence; however, GLMs have a long history of suitable implementation in insurance. Thus it is important to also consider this setting. More broadly, our findings suggest that the new Gaussian Process regression is potentially applicable in insurance. However it is currently under studied.
Since an insurance company cannot immediately observe claims, in Section (ref), we extended the GLM pricing model algorithm to have delayed claims. We prove that the asymptotic regret bound with delayed claims is the same as that achieved in the non-delayed case. The same result for the GP algorithm remains open, but numerical results suggest the same regret bound is achieved.
There are several ideas for future investigation. Thus far we focus on the long-run revenue. In the insurance market, claims may cause loss and even bankruptcy to the insurance company. One idea is to incorporate ruin probability, which is a measure for the risk used for decision taking.
Another possible idea is to implement reinforcement learning (RL) with these online revenue management problems. Bandit problems, as considered in this paper, are simple reinforcement learning routines. However, more complex future market interactions might be considered. This problem may be modeled as a Markov Decision Process (MDP). RL iteratively interacts with a simulation of the insurance model and then use the feedback from the environment to select actions that maximize the insurer's objective. Although MDPs have been applied for many years in insurance, extending the online learning framework considered here to incorporate forward planning is an area that is yet to be studied.
This work was supported by China Scholarship Council (CSC) Grant.