Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
15,556 characters · 8 sections · 15 citation commands
Distribution Shift in Airline Customer Behavior during COVID-19
The novel coronavirus pandemic has had a seismic impact on many industries, including travel. During these unprecedented times, along with industrial operations, customer behavior also changed drastically sheth2020impact. Due to this, machine learning systems built on sequential decision making were affected the most. Machine learning applications often implicitly or explicitly assume that data sets are drawn from stationary distributions, and the sudden shift in underlying data makes the model prone to break. Pricing based on context is one such application that is prominently driven by customers' behaviour ye2018customized, shukla2019pricing. The motivation of this work stems from examining the performance of machine learning models deployed to price add-on (ancillary) products for an international airline. To illustrate, the performance of one of the deployed models that predicts the probability of ancillary purchase dropped from 75% during training to 50% during testing despite exhaustive offline analysis with K-fold cross validation as shown in Figure (ref). Since the onset of COVID-19, one of the features used in the model started experiencing distribution change and needed further investigation.
Demand forecasting is frequently used for price determination given customers' context talluri2004theory, ferreira2016analytics. Pricing applications that rely on forecasting need a distribution shift detection as the price that a customer is willing to pay can be affected by unanticipated events, such as COVID-19, that alter the system dynamics and change the data generating process.
In this work, we (a) investigate the problem of covariate shift and concept drift for a forecasting model used for dynamic pricing, where the marginal distribution of the covariates, $P(X)$, and the conditional distibution, $P(Y|X)$, between the training and test sets differ joaquin2009datasetshift, (b) explore two potential techniques for detecting distribution shift for online use-case, specifically, we use Fast Generalized Subset Scan daniel2013fgss to detect covariate shift in terms of $P(X)$ and Causal Forests athey2018generalized to identify concept drift, that is, change in $P(Y|X)$, and (c) discuss the proposal of a robust framework for contextual pricing applications in a real-world setting.
In this section, we define how Fast Generalized Subset Scan daniel2013fgss and Causal Forests athey2018generalized can be used for detecting distribution shift due to shocks, such as COVID-19, experienced by a system.
Fast Generalized Subset Scan (FGSS) is an unsupervised anomalous pattern detection technique, proposed by daniel2013fgss. We use FGSS for covariate shift detection (change in $P(X)$) with the objective of finding shifted patterns in the test set. Given a set of observations $R_1...R_N$ and features $A_1...A_M$ in a test set, under the null hypothesis that there is no anomalous pattern in the test set, we use FGSS to find a subset $S^*=R^*\times{A^*}$ of self-similar groups that are anomalous, where $R^*\subseteq\{R_1...R_N\}$ and $A^*\subseteq\{A_1...A_M\}$, using a scoring function $F(S)$ defining the anomalousness of the subset $S$. If the null hypothesis is true, then the test set is generated from the same distribution as the training set. Otherwise, the training and test distributions are different with a significantly higher score $F(S)$ for those subset of observations in the test set.
where $N(S)$ represents the size of the subset $S$ and $N_{\alpha}(S)$ represents the total number of p-values (obtained by passing the observed values in the test set through the inverse eCDF of the training set) that are significant at level $\alpha$ in $S$. To efficiently find the subset $S^* = arg\,max\ F(S)$, we need the function $\phi(\alpha, N_{\alpha}(S), N(S))$ to be monotonically increasing w.r.t. $N_{\alpha}$, monotonically decreasing w.r.t. $N$ and $\alpha$, and be convex daniel2013fgss. We use the Berk-Jones statistic bj1979statistic as $\phi$ in our experiments which satisfies these properties.
Causal forests in a supervised method from Generalized Random Forests athey2018generalized that estimates heterogeneity in treatment effects. A treatment effect refers to a causal effect of a treatment or intervention on an outcome variable. Causal forests can be used to estimate the Conditional Average Treatment Effect (CATE). This is useful in identifying the observations for which the treatment is positive and that benefit the most from a treatment; essentially an estimation of optimal policy assignment athey2019economists. CATE cannot be directly observed for a unit because of the “fundamental problem of causal inference” holland1986causal, making it impossible to observe unit-level causal effects and the reason why we can never directly observe the counterfactual condition of a unit of observation. For each observation $X \in \mathbb{R}^{m}$ where $m$ is the number of covariates, there are two potential outcomes $Y_1$ and $Y_0$ corresponding to the binary treatment variable $D\in \{0,1\}$, but only one of them is observed. The conditional expectation of an outcome for the treatment or control, $\mu_{d}(x)$ is defined as:
and CATE ($\tau(x)$) is the difference in expectation of the potential outcomes given x,
We use causal forests to estimate the causal effect of COVID-19 intervention and identify the concept drift $P(Y|X)$ in the system. This requires some data observed post-intervention to be used for training so as to learn the unit-level interventional change in treatment.
Ancillaries are optional products or services sold by businesses to complement their primary product bockelie2017incorporating. In this work, we utilized the following datasets from the airline industry for an ancillary market: (1) simulated interaction of ancillary pricing, and (2) real-world ancillary booking requests data from a large airline containing price variability.
To test our approach, we generated a simulated dataset of customer and flight seat (ancillary) interactions using open-sourced flight simulator \footnote{https://github.com/deepair-io/flai}. In this dataset, we artificially varied the arrival rates of the customers in train and test data. The details of the data-generation process can be found in Appendix (ref). Figure (ref)(a) shows the generated data distribution. The training set represent the control period ($d=0$) and test set represents the treatment period ($d=1$). Figures (ref)(b) and (c) show the results from Causal Forests and FGSS, respectively. FGSS is able to identify 26.01% points in the test set as shifted. The shifted points have AdvancedPurchase feature values greater than 5000 despite the test distribution having a peak around 4000. This is because the train distribution has few points in the values between 0 and 4000, which the FGSS model identifies as normal behavior. On the other hand, Causal Forests is able to identify 23.26% points with positive shift and only 0.56% points with negative shift, indicating a superior performance of Causal Forests model to identify shift on intervention. The positive shift occurs over the range of test set values, which aligns with the distribution of test set simulated significantly different from the train set.
The real-world dataset consists of customer booking requests from March 2019 to September 2020. We use this dataset to evaluate the performance of FGSS and Causal Forests in detecting distribution shift due to COVID-19 in real-world setting. The airline identifies the time period starting March 2020 as the “COVID” era because a large portion of scheduled flights started getting canceled/rescheduled and observed decreased ticket sales. We use five features in our experiments - AdvancedPurchase, LengthOfStay, GroupSize, TotalDuration and TripType. The description of the features can be found in Appendix: Table (ref). Figure (ref) and (ref) show the results obtained on the hold-out test set. Both Causal Forests and FGSS results indicate that there is significant shift in all of the features. LengthOfStay feature has shift towards value $0$ (indefinite stay at destination), indicating most customers are not opting for vacation/business travel but essential movement only. The same conclusion can be drawn from the shift observed in \textit{TripType} feature with customers preferring to travel one-way. \textit{GroupSize} feature has a shift towards value $1$, indicating customers are traveling alone and not in large groups during the pandemic. \textit{AdvancedPurchase} has a shift towards smaller values indicating customers are not booking tickets way ahead of the travel date due to the uncertain nature of the pandemic. \textit{TotalDuration} has a shift towards smaller values as well, indicating the operational changes made by the airline to operate on shorter routes during the pandemic.
We explored two techniques, Fast Generalized Subset Scan and Causal Forests, for covariate shift detection when a system experiences a shock. We applied these techniques to an airline ancillary purchase use case and saw that some of the features used for ancillary pricing experienced a covariate shift while the probability of purchase experienced a concept drift during COVID-19. We identified the observations in the test set that follow a significantly different distribution compared to the training set. Causal forests, while able to detect a concept drift in real-time, need a significant amount of data post-intervention for training. On the other hand, FGSS can only detect covariate shift on a batch of data but does not need data post-intervention for training. Hence, there is a possibility to combine the two approaches for a unified and robust distribution shift detection. At the same time, the construction of a good testing pipeline, along with an agreement by multiple models as well as a visual inspection, can jointly help identify shift patterns. These patterns can then be cross-checked with designers' hypothesis and domain experts' knowledge for validation and potential changes to the system before being corrected for enhanced model performance. In the future, we aim to extend this work to other applications, test multiple shift correction approaches and provide recommendations for adapting to a shift induced by sudden shocks to the system (see Appendix (ref) for details).