Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
63,451 characters · 22 sections · 46 citation commands
Testing for Asymmetric Information in Insurance with Deep Learning
\onehalfspacing
\thispagestyle{empty}
\setcounter{page}{1}
In insurance markets, the positive correlation property (PCP) states that insurees' choices of coverage should be positively related with ex-post measures of their risk, such as the occurrence and severity of claims. This can be either because of adverse selection (i.e., riskier insurees self-select into contracts with higher coverage, as in rothschildstiglitz76), or because of moral hazard (i.e., a higher coverage discourages prevention effort). cjss:2006 proved that the PCP obtains quite generally in models of competitive equilibrium in insurance markets.
chiappori_testing_2000 used the PCP to propose a test for asymmetric information in insurance; and they applied it to French car insurance data. In its simplest form, the PCP states that the conditional correlation of coverage and ex-post risk should be positive, for all values of the vector of covariates that are observed both by the insurer and the insuree\footnote{ As explained in cjss:2006 and dionne_asymmetric_2013, the precise statement of the positive correlation property needs to be adapted in more general models.}. Suppose that coverage is treated as a binary choice (i.e.\ minimal versus comprehensive); and that ex-post risk is also binary (e.g.\ whether or not the insuree filed a claim). chiappori_testing_2000 argued that under the PCP, the residuals of two binary choice models for coverage and ex-post risk should be positively correlated, if all public covariates are controlled for. Hence, they tested the hypothesis that the correlation of the generalized residuals of two univariate probits was zero; they also estimated and tested for the correlation of the generalized residuals of a bivariate probit.
To their surprise, chiappori_testing_2000 found no statistical evidence for the PCP with any of these tests in the French car insurance data they were using: the correlation of coverage and risk was close to zero. \footnote{ This is not a universal finding; while the PCP was not documented with car insurance data, there is evidence for the PCP in some other markets. Our purpose is not to discuss the PCP evidence, but to explore how the outcome of the PCP test depends on design and implementation of the testing procedure.} This remarkable fact implies that essentially all relevant information is contained in publicly observed covariates.
However, the analysis of chiappori_testing_2000 relied on restrictive specifications that may have limited power to detect the PCP. They only included about fifty regressors in their probit regressions, which was a very small subset of the covariates they could have constructed by interacting the covariates. Moreover, their bivariate probit model was built on a simplifying assumption of constant correlation. \footnote{The test procedures in chiappori_testing_2000 have been extended in several directions. KimEtAlJRI09 showed how the probit for coverage can be replaced by an ordered multinomial choice model when more than two types of contracts are available. chiappori_testing_2000 had also proposed a fairly basic nonparametric test. Following ideas in DionneGV01, DionneGV06, SuSpindlerJBES13 and SpindlerGRIR14 used a more powerful nonparametric test of conditional independence. However, this is only practical when there are no more than three continuous covariates and a small number of discrete covariates.}.
The extraordinary development in machine learning methods in the past decade suggests revisiting some seemingly well-established empirical findings. Our goal here is to show how these methods can be implemented in the context of the PCP, and to check whether more powerful testing methods can alter the conclusions of chiappori_testing_2000 about its quantitative unimportance.
Ideally, one would want to do two things: estimate flexibly the conditional correlation of risk and coverage for any given values of the covariates, and test that this conditional correlation is positive for all values in a given subset (e.g. for all male, 40- to 45-year old drivers who use a 5-year old car). We show that both goal can be achieved by combining the flexibility of machine learning methods and standard econometric tests. From a growing catalog of machine learning methods on the market, we choose deep learning as our main estimation method because of its popularity and its remarkable success in many applications.
For the sake of comparison, we use the same car insurance data as in chiappori_testing_2000. The dataset contains only 6,333 observations, a typical size for many microeconometric applications. This allows us to explore the effectiveness of deep learning techniques in such settings. Each observation includes information on the car (brand, model, age, power, \ldots ) and the client's demographics (age, profession, residence,\ldots ); we use these variables and their interactions to construct the covariates $x$. We summarize coverage $c$ and ex-post risk $r $ by two binary variables. Let $X$ denote all publicly observed covariates. We denote $p_{jk}(x)$ the probability that $c=j$ and $r=k$ given $X=x$, for $ j,k=0,1$. Finally, we let $p(x)=p_{10}(x)+p_{11}(x)$ (resp.\ $ q(x)=p_{01}(x)+p_{11}(x)$) denote the probability that $c=1$ (resp.\ that $ r=1$) conditional on $X=x$. The probability $p(x)$ is the conditional choice probability of the higher coverage, and $q(x)$ is the conditional probability of an at-fault claim.
The most basic form of the positive correlation property states that for all values of $x$, the covariance of $c$ and $r$ conditional on $X=x$ is non-negative:
A shortcoming of the covariance is that its value is not easily interpretable. We therefore also state the positive correlation property in terms of the correlation coefficient:
Estimating the covariance and correlation functions for given values of the covariates $X=x$ requires estimating the $p_{jk}$ probabilities flexibly. This is not a simple task, as many interactions between the covariates can have explanatory power. It is notoriously hard, for instance, to model the risk $q(x)$ parsimoniously. It is even more difficult to test that $\rho $ is non-negative over a subset of covariates: interesting subsets typically are very large, leading to a multiple testing problem where the distributions of the estimated covariances or correlations for different $x$ are not independent.
We implement three approaches to apply machine learning to testing for the PCP. Our first test relies on a feedforward neural network to predict the conditional probabilities $p_{jk}(x)$: in the terminology of this field, this is a 4-way classification problem. A potential complication we face when testing the positive correlation property is that the neural network estimates $\hat{p} _{jk}(x)$ have a relatively slow rate of convergence and act as nuisance parameters. As it turns out, the covariance function has a nice double robustness property; the presence of these nuisance parameters is not an issue. On the other hand, the presence of such nuisance parameters does complicate inferences about the correlation function.
To remedy this, we use the double-debiasing method of chernozhukov_doubledebiased_2018, which extends the idea of Neyman orthogonalization to a broad range of models and estimation methods. We combine this double-debiasing method with results from SemenovaCherno:EJ2021 to obtain consistent and asymptotically normal estimators of the average values of the covariance and correlation function within groups of observations. We then use the intersection tests developed in (chernozhukov_intersection_2013) to test the positive correlation property for a variety of groups of observations. We test, for instance, that the correlation is positive on average for all modalities of the \textquotedblleft age of the car\textquotedblright \ variable.
We find that the neural network predicts the purchase of coverage considerably better than it does the accident occurrence. This is not that surprising: at-fault claims are relatively low-probability events and insurers know that they are hard to predict. The range of the estimated covariance function lies mostly in the interval $[-0.01,0.01]$. While the correlation function has a larger range, it narrows considerably when we double-debias it and we average within groups. Our intersection tests show that for any of our eight covariates, we can reject the hypothesis that the correlations are positive on average for all values of its modalities. On the other hand, we cannot reject the hypothesis that these average correlations are larger (i.e., less negative) than a small negative number like $-0.05$. In the end, we obtain a 95% confidence interval for the range of these average correlations that is confined to a narrow interval around zero.
Our second method relies on the \textquotedblleft sorted groups\textquotedblright \ approach of chernodemirerduflofv23. We first cross-fit the neural network model that we selected in our first approach. We then allocate observations into groups sorted by the value of the predicted covariance or correlation. To maximize the power of the test of the PCP, we focus on the observations for which the estimated correlation is smallest in algebraic terms. We find here again that only very small values of the covariance and correlation are consistent with the data, and we find no significant evidence for the PCP.
Finally, we run two variants of our first approach in which we replace the neural network with two other popular machine learning methods---random forests and gradient-boosted trees. While these two methods put different weights on the various covariates, the results of the intersection tests are similar to those obtained by using our baseline deep learning method.
To summarize: machine learning methods do not alter the qualitative conclusion of chiappori_testing_2000 that the correlation of coverage and risk is essentially zero. Even more remarkably, we could not find evidence for a positive correlation in any reasonably-sized subpopulation. While adverse selection and moral hazard are clearly important phenomena in many markets, they do not seem to play much of a role in this one.
The rest of the paper is as follows. Section (ref) describes the data used for estimation. Our analysis has three steps: In Section (ref), we fit a neural network to classify insurees into the four alternatives $c,r\in \left \{ 0,1\right \} \times \left \{ 0,1\right \} $. In Section (ref), we use the double-debiasing methods of chernozhukov_doubledebiased_2018 to correct the correlation function. In Section (ref) we test the positive correlation property by running the intersection test of chernozhukov_intersection_2013. Finally, Section (ref) compares our results with those obtained using methods based on decision trees.
chiappori_testing_2000 obtained their data from the French federation of insurance companies FFSA, which ran a survey of automobile insurance in 1989. They selected a subsample that only includes \textquotedblleft young\textquotedblright \ drivers---insurees who obtained their driver's licence within the past three years. We focus on an even narrower subsample of insurees whose driving license is one year old at most. Since these individuals have no previous driving history, there is no concern about the impact of experience rating on driving behavior; it also reduces the unobserved heterogeneity in the sample.
Our selection leaves us with a sample of 6,333 observations. The data on each insuree and car are quite rich. Each observation includes information on the car (brand, model, age, power, \ldots ), the client's demographics (age, profession, residence,\ldots ), the type of contract, and the claim record. As in chiappori_testing_2000, we code the type of contract and the claim record as binary.
The data also record if the insurance contracts covered all or part of the year. Only about 40% of insurees were insured throughout the year, and roughly 15% were covered for less than two months. Like in chiappori_testing_2000, we use sampling weights $w$ that represent the number of days that the insuree was insured during the year. We indicate sampling weights with $w$ subscripts when needed.
In automobile insurance, the main distinction between contracts is whether they only include third-party (liability) coverage---which is compulsory in France---or whether they also cover damages that the insuree caused to his/her own car. We call the latter \textquotedblleft comprehensive coverage\textquotedblright , and we will neglect variations within this class, such as the amount of the deductible. Since the difference between third-party and comprehensive coverage only matters when the insuree is at fault, we define our claim variable accordingly. This results in the two binary variables $c,r\in \left \{ 0,1\right \}$:
We also define four indicator variables as $y_{jk}=1$ if $(c=j\mbox{ and } r=k)$ for $j,k=0,1$. Table (ref) classifies the 6,333 observations by the four indicators.
We use the same set of covariates $X$ as chiappori_testing_2000; they are created from the eight variables that insurers identified as being the most important. We have up to 28,800 categories of insurees: nine age categories, eight professions, four types of use, ten regions, five rural-to-urban codes, and gender; and 72 car categories: six categories for the performance of the car, and twelve for its age. Combining them would yield more than 2 million dummy variables, a number that dwarfs the sample size. Even if we had a much larger dataset, there are many more variables in the data. This is a clear-cut \textquotedblleft $p\gg n$\textquotedblright \ case, which calls for model selection.
We fit the 4-way probabilities $p_{jk}(x)=\Pr (c=j,r=k|X=x)$ for $j,k=0,1$ with a neural network. The values of the eight covariates $x$ enter the input layer. The neural network has $D$ hidden layers, each with $W$ neurons; the last hidden layer feeds into a 4-node output layer which uses a \textquotedblleft softmax\textquotedblright \ function to generate the probabilities $p_{jk}(x)$. We train the neural network using the Adam optimizer KingmaAdam:2017 to minimize the cross-entropy loss function adjusted by the sampling weights. For a sample of observations $(x_{i},c_{i},r_{i},w_{i})_{i\in I}$, the loss is
Our code relies on the Python package Keras CholletKeras:2021 with the TensorFlow backend AbadiTensorflow:2015.
The class of neural networks we consider has $P\equiv n_{X}W+(D-1)W^{2}+3W$ parameters for $D>0$, where $n_{X}$ is the number of covariates in the input layer\footnote{ For $D=0$, the number of parameters is $3n_{X}$.}. Adding up the number of categories (minus one category per variable) of our eight covariates, plus the constant, gives us $n_{X}=49$ and results in $P=W(2+(D-1)W)$ parameters. To illustrate, a very modest neural network with $D=2$ hidden layers of $ W=16 $ neurons each has $P=1,088$ parameters.
With such a large number of parameters, overfitting is an obvious concern. To guard against it, we resort to cross-validation: we use a validation sample to decide when to stop learning. In addition, we randomly drop out some of the neurons in the hidden layers during training. The idea of dropout regularization, pioneered by SrivastavaDropout:2014, is that the network will learn to compensate for the missing neurons, and that the resulting model will be more robust to overfitting. The fraction $d$ of neurons that are dropped out is called the dropout rate. Dropout allows us to use a more powerful network with a larger number of neurons. The optimal dropout rate typically increases with the number of neurons in each hidden layer, as a higher dropout rate is needed to deal with higher overfitting.
The neural network we consider has many hyperparameters: its depth $D$ and width $W$, the size of the validation sample, the size of the mini-batches, the choice of optimizer, etc.\ We decided to optimize over the depth and width, and also over the dropout rate $d$.
Since our sample size is relatively small, we only fit smallish neural networks that vary in:
To optimize in hyperparameter space, we split the data into training, validation and testing sets, representing 70%, 15%, and 15% of the data, respectively. For each 3-uple of values of $D$, $W$ and $d$, we fit the neural network on the training sample and we use the validation sample to stop training soon after the loss on the validation sample stops decreasing. To discard local minima, we keep training the network for a small number of epochs to see if the validation loss keeps increasing\footnote{ The parameter that controls the number of periods that we wait is called \textquotedblleft patience\textquotedblright ; we set it at 10 epochs.}. We then store the parameters of the neural network that correspond to the epoch with the smallest validation loss. Then we measure the loss on the test sample.
After fitting all of these models, we select the combination of the three hyperparameters that leads to the smallest loss on the test sample\footnote{ Fitting a neural network involves a choice of initial values for the weights. TensorFlow uses a well-tested procedure that injects some randomness into the process. As a result, different runs can select slightly different neural networks. We find that this had almost no impact on the results of our final tests of the positive correlation property.}. The hyperoptimized model has $D=2$ hidden layers; $W=16$ neurons in each layer; and a dropout rate $d=0.1$. The hyperoptimizing procedure took 150 seconds on a Mac Studio. The resulting neural network has 1,524 parameters.
We also fit a 2-way classification model for $c$ and another for $r$; we hyperoptimized them in a similar way. The neural network for $c$ (the choice of coverage) has no hidden layer and dropout = $0.7$. The model for $r$ (the occurrence of a claim) has 2 hidden layers; 8 neurons in each layer; and dropout = $0.4$.
These loss values can be compared with those of probit models that use the same set of regressors as in chiappori_testing_2000, as well as to “constant” models that use no regressors. As Table (ref) shows, our deep learners massively outperform the probits of chiappori_testing_2000. Since the loss is just minus the average log-likelihood per observation, these improvements are quite large. For instance, the $0.188$ gain in average log-likelihood from bivariate probit to the bivariate deep learner yields a likelihood ratio statistic of $2\times 6,333 \times 0.188=2381.2$, for an additional $1,524 - (2\times 48 +1)=1,425$ parameters. The corresponding $p$-value is minuscule\footnote{This is only meant to be illustrative; it is not clear that one can use asymptotic approximations with such a large number of parameters.}.
We also applied our procedure to the sampling weights. We use an activation function that takes into account the truncation of weights in $[0,1]$ and a loss function that allows for a mass point at $w=1$. Not surprisingly, the selected neural network is rather sparse: it has no hidden layers ($D=0$) and a dropout rate $d=0.5$. The whole hyperoptimizing procedure took 130 seconds. The hyperoptimized model has a loss of $0.098$ on the test sample, while a constant model has a test loss of $0.114$, which is only 15% larger than the model.
Figure (ref) plots the estimated probabilities $\hat{p}_{jk}$, both when the corresponding $y_{jk}$ is 0 and when it is 1. The dashed red line represents the mean of $y_{jk}$ in the sample. A perfect fit would have $\hat{p}_{jk}=y_{jk}$ within each panel. It is clear that the left column performs better than the right column in this respect. It is not that surprising as there are more than 15 times as many observations with $r=0$ as with $r=1$: the neural network puts a large weight on fitting these observations. Figure (ref) shows that the model can predict the purchase of coverage considerably better than the accident occurrence. This is a common finding with insurance data. It is more surprising that the neural network overestimates the probability of a claim to the extent shown in the right panel. This seems to be a side effect of its efforts to fit the choice of contract (the variable $c$). For comparison, Figure (ref) shows the results obtained with the 2-way classification neural networks for $c$ and for $r$.
Figure (ref) plots the fitted weights. As already mentioned, the model for weights has low explanatory power. Many contracts cover the entire year, as can be seen from the vertical cluster on the right. The covariates do not help much in predicting how long the car is insured within a given year, as it depends on decisions to buy, sell or exchange a vehicle that result in starting or terminating coverage within a given calendar year.
In a linear model, we could use partial $R^{2}$'s to evaluate the contribution of various covariates to explaining the left-hand side variable. In a neural network, a natural alternative is to train the model again while omitting one covariate and to measure the additional loss. This is a very partial indication, as the neural network interacts different groups of variables in potentially complex ways. Still, it is a reasonable starting point.
Our application has eight groups of variables; accordingly, we run eight neural networks, each of which omits one of these groups. Figure (ref) plots the results. The dashed vertical line represents the test loss of the complete neural network, which is denoted \textquotedblleft None\textquotedblright . A larger positive number denotes that this group of variables contributes more to the quality of the fit (that is, reduces the test loss more).
{Figure (ref) shows clearly that the age of the car dominates the fit. Five other groups of variables contribute (taken by themselves) to the fit. In decreasing order of importance, they are: the (work/leisure) usage of the car, the profession, the rural/urban zone, the age of the insuree, and her gender.} Looking more closely at the models for $ c$ and $r$ shows that drivers of older cars are less likely to buy comprehensive coverage---again, a common finding.
We use the hyperoptimized models of Section (ref) to get “raw' and “cross-fitted” estimates for the covariances and correlation functions. The raw estimates are simply the values predicted over the whole sample. To obtain the cross-fitted estimates, we split the sample randomly into five subsets; we predict the covariances and correlations over a subset using the hyperoptimized neural network trained over the other four subsets only. We will need the cross-fitted estimates when we move to testing in Section (ref).
Table (ref) gives the results. Only 40.4% of the predicted raw covariances (and correlations) are positive. The cross-fitted estimates are very similar to the raw estimates, if somewhat more dispersed.
Figure (ref) plots the density of our estimated covariance function $\hat{C}(x)$ and correlation function $\hat{\rho}\left( x\right) $ over the sample.
Essentially all the mass of the distribution of the covariance is situated in the interval $[-0.01,0.01]$. These seem like small values, but they are not easily interpretable. We next consider the correlation function. It is negatively skewed; more than 60% of the estimates are negative. The correlations are small, however: 99% of the mass is in the $[-0.2,0.2]$ interval.
It would be tempting to interpret the bimodal shape of the estimated density of $\hat{\rho}$ as a mixture of the densities for the two genders. We already know from Figure (ref) that gender has low explanatory power, however. Figure (ref) has boxplots of the distributions of $\hat{\rho}$ when a variable takes one particular value. The top-left panel, for instance, shows that the estimated correlation tends to decrease with the age of the car. Men seem to have a lower correlation of risk and coverage than women, rural drivers a lower correlation than urban drivers.
Our ultimate goal is to test the sign of the covariance function $C(x)$ and the correlation function $\rho(x)$. These are easily estimated by plugging in the probabilities $\hat{p}_{jk}(x)$ predicted by the neural network. Still, these probability estimates have a relatively slow rate of convergence, which is likely to contaminate inference. This is a common issue with machine learning methods: they yield predictors that must be treated as nuisance parameters in later stages of statistical procedures. To remedy this problem, chernozhukov_doubledebiased_2018 developed a double-debiasing method that extends the idea of Neyman orthogonalization to a broad range of models and estimation methods.
To give the intuition behind double-debiasing, consider the vector of probabilities $\eta \equiv (p_{00},p_{01}, p_{10}, p_{11})$. We denote $ \eta_0 $ its true value. Suppose that we want to estimate some parameter vector $\beta_0$ that satisfies a set of conditions $M(\beta_0,\eta_0)=0$. In our application, this $\beta_0$ will be the average covariance or correlation for a given subset of observations---for instance, for all young men.
The neural network (or any other estimation procedure) gives us an estimate $ \hat{\eta}$. A natural way to proceed would be to estimate $\beta_0$ by the value $\hat{\beta}$ that minimizes some norm of $\hat{M}(\beta,\hat{\eta})$, where the function $\hat{M}$ is the sample analog of $M$. Given an appropriate set of assumptions, the standard Taylor expansion around the true values $(\beta _{0},\eta _{0})$ gives
The presence of the second term on the right-hand side is what makes the $ \hat{\eta}$ estimates nuisance parameters: the estimation error $\hat{\eta}$ contaminates the asymptotic distribution of $\hat{\beta}$ if the gradient $ \nabla _{\eta }M(\beta _{0},\eta _{0})$ is nonzero.
If the $\hat{\eta}$ estimates converge at the usual parametric rate, the additional term only changes the usual sandwich formula. This clearly does not apply here, as the neural network estimates converge more slowly, and this invalidates standard inference over $\beta _{0}$. To get rid of this nuisance effect, we need to change the estimating equation to an equation for which the gradient of $M$ with respect to $\eta $ is zero at the true values $(\beta _{0},\eta _{0})$. This can be done by projecting the estimating equation on the orthogonal subspace to the gradient $\nabla _{\eta }\hat{M}$. This is the idea that underlies Neyman orthogonalization and its modern successor, double-debiasing.
We describe how double-debiasing applies to the covariance and the correlation functions in Sections (ref) and (ref), respectively.
Given our estimates $\hat{p}_{jk}$ of the probabilities of the four alternatives, we can write the covariance as
where $\hat{E}$ (resp.\ $\hat{E}_{w}$) denotes an unweighted (resp.\ weighted) sample mean; $\hat{p}(x)\equiv \hat{p}_{10}(x)+\hat{p}_{11}(x)$ estimates $\Pr (c=1|X=x)$ and $\hat{q}(x)\equiv \hat{p}_{01}(x)+\hat{p} _{11}(x)$ estimates $\Pr (r=1|X=x)$. In these equations, the neural network estimates $\hat{p}_{jk}(x)$ act as nuisance parameters $\hat{\eta}$. However, it turned out that for the covariance coefficients, the presence of such parameters does not interfere in the estimation procedure, so that standard inference methods apply.
To see this, consider estimating the sample-weighted average covariance $ \beta_g$ for a group of observations $i\in g$. We will use weighted non-linear least squares. This corresponds to the estimating equation
The gradient of $M$ with respect to $\eta$ is a weighted sum of gradients of $C(x)$ with respect to $\eta $. Now consider, for some $x$ and some $j,k=0,1$ ,
By definition, at the true values $\eta _{0}$ we have
Since the partial derivatives of $p$ and $q$ with respect to $p_{jk}$ are either 0 or 1, the gradient of $M$ with respect to $p_{jk}$ is zero at the true values and the estimates of $\hat{\eta}$ do not affect the asymptotic distribution of $\hat{\beta}$. To put it differently: the covariance is doubly robust and does not need to double-debiased.
We now turn to the correlation function $\rho (x)$. A naive estimate of the correlation function would be
The gradient of $\hat{\rho}$ with respect to $\hat{\eta}$ now involves terms like
which are clearly not zero at the true values. Therefore the presence of nuisance parameters does interfere in the estimation procedure and thus, can invalidate inference unless we use double-debiasing.
Proceeding as with the covariance function, let our estimation equation be
Again, the gradient of $M$ with respect to $\eta $ is a weighted sum of gradients of $\rho (x)$ with respect to $\eta (x)$. We already know that we the derivatives of $C(x)$ with respect to $\eta (x)$ are zero. This leaves us with
To apply double-debiasing, we need to project our estimating equation on the orthogonal space to this gradient. This can be done quite simply by running a weighted regression of $\hat{\rho}(x_{i})$ on the variables that correspond to $(j,k)=(0,1),(1,0)$, and $(1,1)$. In fact, the last one is the sum of the previous two, so that we only need to use the following two regressors\footnote{ Using $p=p_{10}+p_{11}$ and $q=p_{01}+p_{11}$.}:
To obtain a double-debiased estimator of $\beta _{g}=E_{w}(\rho (x_{i})|i\in g)$, we can simply regress $\hat{\rho}_{i}$ on $\hat{\nabla}_{1i}$, $\hat{ \nabla}_{2i}$, and a constant over the sample $i\in g$ with weights $w_{i}$. The double-debiased estimator is the coefficient of the constant; we will denote it $\tilde{\rho}_{g}$ from now on on.
We implement two ways of testing the positive correlation property: an intersection test and a sorted group approach, in Sections (ref) and (ref) respectively. Both methods require that we use cross-fitted predictors of the covariance and correlation; we use the estimates described in Section (ref) for this purpose.
It is clearly not feasible to test that the covariance (or correlation) is positive for all possible values of all covariates. However, results in SemenovaCherno:EJ2021 show that we can use standard inference for their mean values over subgroups of observations.
More precisely, let $T$ denote either covariance or correlation, and consider testing the following hypothesis:
where $h$ is a function whose values lie in a low-dimensional space, and $B$ is a subset of its range. We could for instance test that the positive correlation property holds on average over all women in rural areas who drive cars that are more than 5 year old.
SemenovaCherno:EJ2021 derive assumptions under which one can run a sieve regression
and use its fitted values $p(h(x_{i}))\hat{\beta}$, instead of the neural network estimate of $T(x_{i})$, to test the multiple hypothesis (ref). Note that $p$ applies to the values of $h$; in the example of the previous paragraph, $p$ could only be a function of gender, whether the insuree lives in a rural area, and whether the car is more than 5 years old.
The statistic $T$ must be double-debiased if needed (that is, we use $\hat{C} $ and $\tilde{\rho}$); it must converge faster than $n^{-1/4}$, which holds under reasonable assumptions on the neural network; and the sieve basis $p$ must expand at the appropriate rate, which implicitly limits the dimensionality of the function $h$.
The testing procedure simplifies further if we focus on groups of observations. To be more precise, suppose that we define disjoint groups $ g_{1},\ldots ,g_{L}$ by the values of some of our covariates. The union of the groups could be the whole set of observations, but that is not necessary. For instance, $g_{1}$ could have all young men and $g_{2}$ all women who have an old car. Our goal is to test the null hypothesis that
where $T$ is either the covariance or the correlation. In the notation of the previous paragraphs, this amounts to using indicators of the groups as the basis functions $p\circ h$ in the sieve regression; the model is saturated and the sieve basis trivially satisfies the conditions in SemenovaCherno:EJ2021.
Then for each of our groups $g_{l}$, we compute the sample-weighted average predicted covariance $\hat{C}_{l}$. We apply the double-debiasing procedure described in the previous subsection to obtain $\tilde{\rho}_{l}$, the double-debiased, sample-weighted average correlation within this group. We compute the standard error $\hat{\sigma}_{l}$ of the estimators $\hat{T}_{l}= \tilde{\rho}_{l}$ or $\hat{C}_{l}$ by the usual formula.
If $L=1$ (for instance, we only want to test the positive correlation property for young men), then we are done: we reject the null hypothesis at the 5% level if $\hat{T}_{l}+1.64\hat{\sigma}_{l}<0$. If $L>1$, we want to test that $\min_{l=1,\ldots ,L}E(T(X)|X\in g_{l})>0$. This is an intersection test; we use the procedure described in chernozhukov_intersection_2013:
In Sections (ref) and (ref), we will implement the intersection test for the covariance and the correlation coefficient, respectively.{\ To define the groups $g_{l}$, we use the age of the car, which is by far the variable with the most predictive content.} We split it into quartiles, and we also run tests for all of its modalities.
Table (ref) gives the results of the intersection tests for the variable \textquotedblleft car age\textquotedblright \ and its split by quartiles (the results of the intersection tests for other variables are provided in Table (ref) in the Appendix)\footnote{ The bounds of the confidence intervals are computed as $a=\min_{l}(\hat{C} _{l}+k\hat{\sigma}_{l})$ and $b=\max_{l}(\hat{C}_{l}-k\hat{\sigma}_{l})$: these are the values such that we are at the margin of rejecting the hypothesis that $\min_{l}C_{l}>a$ and the hypothesis that $\max_{l}C_{l}<b$.} .
Note that $k$ (the number of standard errors used in the last step of the intersection test) varies noticeably. For the 5% test, it varies from $1.62$ (that is, the standard value) to $2.54$, which is larger than what a naive normal approximation would suggest. The test statistics show that for any of the partitions into groups, we can reject the hypothesis that the covariances are all positive at any reasonable level. On the other hand, we can also reject that any of them is smaller (i.e., less negative) than, say, $-0.01$; this can be seen in the \textquotedblleft confidence interval\textquotedblright \ column.
Let us turn to the correlation function. The results in Table (ref) are very similar to those for the covariance: once again, the positive correlation property is rejected for all partitions of the sample (the test results for other variables are shown in Table (ref) in the Appendix).
Moreover, the confidence intervals confine the (doubly-debiased) correlation to a narrow interval just below zero. This is not at all what Figure (ref) (which plots the density of $\hat{\rho}$, not that of the double-debiased version $\tilde{\rho}$) would have suggested; it is a good illustration of the fact that raw predictions from neural networks are not free from bias and noise.
A recent contribution by chernodemirerduflofv23 adopts a different approach to recover estimators of group averages that have standard asymptotics. While they define it in the more complex framework of conditional average treatment effects, their method a fortiori applies in our setting. The underlying idea is to split the sample into a main sample and an auxiliary subsample. A machine learning model is trained and optimized on the auxiliary subsample to predict the statistics of interest (here, the probabilities of the four alternatives). The predictors are applied to the observations in the main sample, which are then allocated into groups sorted by the value of, in our case, the predicted covariance or correlation. chernodemirerduflofv23 show that regressing the outcomes $y_{jk}$ observed in the main sample on group indicators gives estimators of the average probabilities that have standard asymptotics and can therefore be used to construct standard tests.
As recommended by chernodemirerduflofv23, we use several random splits into main and auxiliary samples, and we report the median test statistics and $p$-values. We train and select a neural network on the auxiliary subsample exactly as explained in Section (ref). Once we have predicted probabilities $\hat{p}_{jk}$ on the main sample we use them to compute the covariance for each observation using the formul\ae \ in Section (ref). We then sort the observations according to the predicted covariance, and we define four groups $q=1,2,3,4$, splitting at the quartiles. In each group, we regress $y_{jk}$ on the group indicators to obtain new predicted group-average probabilities $\bar{p}_{jk}(q)$. The results in chernodemirerduflofv23 imply that these $\bar{p}$ statistics have standard asymptotics. This allows us to test the PCP on each group $q$, and to define confidence intervals for the $q$-group covariances.
The results are reported in Table (ref), where the medians are computed over 100 random splits. While the PCP is not rejected at the 5% level, the confidence interval for the average covariance on the lowest-covariance quartile is very narrow. This is consistent with the results in Tables (ref) and (ref).
We proceed in exactly the same way for the correlation. Remarkably, the main/auxiliary method used in chernodemirerduflofv23 allows us to circumvent double-debiasing altogether. As Table (ref) shows, we come close to rejecting the PCP for the lowest-correlation quartile.
The confidence interval shows once more that only very small values of the correlation are consistent with the data.
For the sake of comparison, in this section we use another popular machine approach: ensemble learning applied to decision trees. To train the ensemble, we consider two alternative learners: bagging and boosting. Bagging applied to feature selection using decision trees is referred to as \textquotedblleft random forests\textquotedblright , while the boosting method yields gradient-boosted trees. Using these two methods, we estimate the covariance and the correlation functions, we double-debias the correlation, and we test the positive correlation property using the intersection test.
In the machine learning setting, predictions from all types of models can be averaged. This approach has become especially popular with models based on decision trees --- ensemble learners that average over a large numbers of shallow decision trees that are trained over random subsamples and features.
We use $N=500$ ensemble members and 5-fold cross validation with the weighted entropy criterion. We do a grid search over three hyperparameter values: the maximum depth of each decision tree; the minimum number of observations in each leaf; and the maximum number of features that are randomly selected before each split (for the random forest) or the learning rate (for the gradient-boosted tree). To optimize in hyperparameter space, we split the data into training and testing sets, representing 80% and 20% of the data respectively; we select the model that gives the best fit on a test sample. Both procedures report “feature importance” scores; we plot them in Figures (ref) and (ref).
The best random forest has a maximum depth of 5; a minimum leaf size of 10; and randomization over a maximum of 5 features. Figure (ref) shows that the age of the car again is the most important explanatory variable by far. The best gradient-boosted tree has a maximum depth of 4; a minimum leaf size of 20; and a learning rate of $0.1$. Figure (ref) shows that the importance of the various features for the gradient boosting is very similar to that for our selected random forest.
When computed over the whole sample, the weighted entropy losses of the three methods are $0.386$ for the neural network, $0.392$ for the gradient-boosted tree, and $0.402$ for the random forest. We were surprised that the tree-based learners do not work better on this tabular data. These differences are not very large, however, and they may be quite specific to our dataset.
Table (ref) reports our estimates of the covariances and correlation coefficients produced by random forest and gradient boosting methods.
Figure (ref) plots the density of the predicted correlations $\hat{\rho}_{i}$ over the sample under all three of our machine learning methods: deep learning, random forest, and gradient boosting. The range of values of the correlation coefficient is similar for all three methods: essentially all mass belongs to the $[-0.2,0.2]$ interval. While the density of $\rho (x)$ obtained from the neural network is bimodal, with two asymmetric peaks in the negative and positive ranges, the densities produced by the ensembles are unimodal. Another important difference is that the gradient-boosted tree produces a much broader range of variation than the other two methods on this data.
Figure (ref) (resp.\ Figure (ref)) shows the correlation for different groups produced by the random forest method (resp.\ the gradient boosting method). These plots show some marked differences with the equivalent plot for the neural network (Figure (ref)), most notably for the important \textquotedblleft Age of car\textquotedblright \ variable.
We repeated the intersection tests of the positive correlation property with our random forest and our gradient-boosted tree. We focus on the correlation to save space. Table (ref) reports the results for the random forest method, and Table (ref) gives them for the gradient boosting method (the results for other variables are provided in the Appendix).
Since the lower bounds are negative for all methods, we can reject the hypothesis that the covariances are all positive. The lower bounds given by the random forest and especially the gradient-boosted tree are lower than with the neural network; still, even with the gradient-boosted tree the lower bound is not economically significant from zero. We can safely conclude that the three machine learning methods produce qualitatively similar implications.
Finally, Tables (ref) and (ref) report the group-averaged correlations produced by the three machine learners at two levels of grouping on the \textquotedblleft Age of the car\textquotedblright \ variable: with respectively four and twelve modalities\footnote{ For completeness, the Appendix compares the group-averaged correlations produced by the three methods for the other variables.}. The great majority of the correlations are negative, and they tend to shrink after double-debiasing (indicated by “DD”).
Even with the very flexible methods used in our paper, this dataset shows no evidence for the positive correlation property. In addition to this empirical finding, our paper contains a methodological contribution. When we started this project, it was not clear to us that deep learning methods could be applied fruitfully to a non-trivial testing problem on such a (relatively) small dataset. With our sample of just $6,333$ observations, deep learning cannot go very deep: neural networks with at most two hidden layers and a small dropout rate work best. Still, they deploy many more parameters than any econometric model. Our neural network in fact predict the choice of coverage much better than parametric procedures, and they also outperform them on claim occurrence. Double-debiasing, or sorted groups, give us consistent and asymptotically normal estimates that can be used in standard test procedures. Our overall conclusion is that deep learning can provide both robust and useful conclusions even for relatively small-scale applications. Methods based on decision trees---random forests and gradient boosting---also perform well, and give very similar results. In further work, we plan to apply our three-step method to larger samples of insurees, and to explore the value of reducing the number of covariates to those that seem to have the largest effect.