Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
44,615 characters · 16 sections · 39 citation commands
Solving the Data Sparsity Problem in Predicting the Success of the Startups with Machine Learning Methods
Predicting the future success of startup companies is of great importance for both startup companies and venture capital (VC) firms. For startup companies, predicting the future development of themselves and their competitors can help them adjust their development strategies and capture opportunities effectively. For VC firms, predicting the future success of startup companies helps them balance their profit and risk.
For late-stage companies, the evaluation of future success is mostly based on financial and operating information. However, for early-stage companies, there is usually not enough data publicly available for prediction. Traditionally, the evaluation of startup companies relies heavily on investor's personal experience. In recent years, machine learning is developing rapidly and achieve great success in many areas. There exists some research applying machine learning algorithm to predict the future success of startup companies yankov2014models,mckenzie2017man,arroyo2019assessment,kaiser2020value, but their methods are not well suited for dealing with sparse data, which is common in datasets of startup companies. With the development of machine learning methods, recent algorithms like XGBoost chen2016xgboost and LightGBM ke2017lightgbm have the potential to solve this data sparsity problem.
In this paper, we aim to make the following three contributions. First, we try to leverage the recent progress of machine learning to handle this data sparsity problem. We validate that the recently developed algorithms, such as XGBoost chen2016xgboost, LightGBM ke2017lightgbm, and soft Decision Tree frosst2017distilling, outperform many traditional algorithms including Logistic Regression, K Nearest Neighbor, Decision Tree, Multilayer Perceptron, and Random Forests. Second, we construct 19 factors using the data from Crunchbase \footnote{http://www.crunchbase.com}, a data aggregation platform built to track startups on a global scale. We define multiple time windows to enrich the number of data samples and take factors like macroeconomy into consideration. This time window definition is more practical in VC practices. Third, we introduce interpretability into machine learning models. We interpret the predictions from the perspective of the contribution of each factor, finding that company age and past funding experience are the most important factors.
The rest of the paper is structured as follows: The previous research and theoretical background are reviewed in Sec (ref). Our approach is introduced in Sec (ref). Experimental results are discussed in Sec (ref). The construction of the portfolio is described in Sec (ref). Sec (ref) summarizes our main conclusion and discusses future research directions.
The problem of predicting the future success of startup companies has existed for a long time schendel1979new,chandler1993measuring and is still exploring by scholars arroyo2019assessment, kaiser2020value.
Common solutions are classification models based on decisive factors. Most earlier studies for the prediction of the success of startup companies are based on regression analysis such as logistic regression lussier1995nonfinancial,kaiser2020value, ordered probit model lussier2001crossnational halabi2014model lussier2010three, and log-logistic hazard models holmes2010analysis. Researchers also develop expert systems ragothaman2003predicting and rule-based methods yankov2014models.
In recent years, with the emergence of platforms aggregating business information about companies and the development of machine learning approaches, it is possible to use machine learning methods to solve the problem of startup prediction. yankov2014models test several tree-based, rule-based, and Bayes-based machine learning methods based on questionnaires gathered from 142 startup companies in Bulgarian. The authors suggest that the best-derived model is the tree-based C4.5 quinlan1993c4. mckenzie2017man compare the performance of human experts and several machine learning methods, including Least Absolute Shrinkage and Selection Operator, Support Vector Machines, and Boosted Regression. They analyze 2,506 firms in a business plan competition in Nigeria. The author suggests that investors using the combination of man and machine rather than relying on human judges or machine learning-chosen portfolios. arroyo2019assessment analyze the performance of several machine learning methods in a dataset of over 120,000 startup companies retrieved from Crunchbase. They consider five machine learning algorithms: Support Vector Machines, Decision Tree, Random Forests, Extremely Randomized Trees, and Gradient Tree Boosting. The results suggest that the Gradient Tree Boosting performs better in predicting the next funding round, while Random Forests and Extremely Randomized Trees perform better in predicting acquisition. One common problem of the datasets with startup companies is their sparsity nature. For most early-stage companies, there is usually not very much data available to the public. Current approaches are not well suited for this problem.
Machine learning is developing rapidly in recent years, and many new models have emerged. Gradient Boosting Decision Tree (GBDT) friedman2001greedy is a highly effective and widely used machine learning method, due to its efficiency, accuracy, and interpretability. It has several effective implementations recently, including XGBoost chen2016xgboost and LightGBM ke2017lightgbm. XGBoost chen2016xgboost is a scalable end-to-end tree boosting system, which is widely used in data science and achieves excellent results. XGBoost proposes a novel sparsity-aware algorithm for sparse data and a weighted quantile sketch algorithm for approximate tree learning. It uses the second-order approximation of the convex loss function that can optimize the objective quickly. LightGBM ke2017lightgbm proposes novel Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB) methods. With GOSS, LightGBM can obtain a quite accurate estimation of the information gain with much smaller data size. The EFB method bundles mutually exclusive features using a greedy algorithm, solving the data sparsity problem. Tree-based classifiers are usually more preferred in investment-related research, mainly because they can be interpreted. Soft Decision Tree frosst2017distilling takes the knowledge acquired by neural nets and expresses the knowledge in a model that relies on hierarchical decisions, creating a more explicable model. To our knowledge, there is no previous approach applying these methods in predicting the future success of startup companies. We leverage this recent progress and compare the performance of these methods in predicting the future success of startup companies
We define the concept of 'success' to include raising new funding, being acquired, or going for an IPO. To make the prediction closer to the reality of VC investment, we further restrict the concept of future success to be in a defined time window (18 months).
We formulate the problem of evaluating startup companies as a binary classification problem. For each company $i$, we synthesis a set of variables ${\bm{x_i}} \in \mathbb{R}^m$ to evaluate its future success, $m$ is the size of features. The selection of the features is discussed in detail in Sec (ref). The label $y_i$ is assigned according to whether the company will succeed in the defined time window:
Our data set can be denoted as: $\mathcal{D} = \{({\bm{x_i}}, y_i)\}$, $i = 1,2, ... , n$, $n$ is the size of the dataset. Given the labeled sample, the machine learning methods learn the conditional probability of $y$ given $\bm{x}$, i.e. $p(y|{\bm{x}})$. Given a new data sample with no label, this model then outputs a prediction $\hat{y}$ that places the sample into the class of success or failure.
where $th$ is a threshold manually selected. In our experiment, th is 0.5 for fair comparison.
We then try to construct portfolios according to the suggestion of these algorithms. The aim of constructing a portfolio is to select a subset $\mathcal{S} = \{c_1, c_2, ... , c_k\}$ of companies with size $k$, each $c_i$ corresponds to a company, such that we maximize the expected number of success companies in the subset $\mathcal{S}$ .
If we assume that the success of each company is independent, $\mathcal{S}$ is consisted of the companies with the $k$ largest $p(y_i=1|{\bm{x_i}})$.
Our data sample is extracted using the daily CSV export of Crunchbase on October 20, 2020. The full dataset contains 1,166,402 organizations and 351,236 venture deals. We filter out the companies that the date of establishment is missing. The companies founded before 1990 are removed. Unique ids are assigned to each company to distinguish from the duplication of company names. Total 776,273 organizations remain after this preprocessing.
To make the prediction closer to the reality of VC investment, we restrict the concept of future success to be in a defined time window. We expect that the company will raise new funding, be acquired, or go for an IPO within a time threshold after the prediction. These evaluation time windows can be interpreted as the time intervals for investors to evaluate the return of investments. We use multiple time windows to enrich the number of data samples and take more factors like macroeconomy into consideration.
In practice, most VC firms believe the time segment between two funding rounds is usually around 18 months. Table (ref) summarizes the time startup companies need to raise next round funding, validating that more than half of the companies achieve their next round within 18 months in most of the rounds. So we define the evaluation time window of 18 months. The time intervals and the according label distribution are shown in Table (ref) \footnote{There might be survival bias since the Crunchbase was founded in 2007. The companies failed before the creation of Crunchbase may not register in the dataset.}. The time $t_s$ denotes the start of the evaluation window, which is the time we make the prediction and can be considered as the moment VC investors invest in a company. The time $t_f$ denotes the end of the evaluation time window. The companies that were acquired, went for an IPO, closed, or had no funding events before $t_s$ are removed. Accumulated in all the time windows, the final data sample consists of 398,489 sample events.
Crunchbase provides information about companies, news, founders, funding rounds, and acquisitions. We compile a set of 19 factors grouped in three categories related to the growth of companies based on the information available in the dataset, as summarized in Table (ref). Since we use multiple time windows, all the factors are associated with the beginning of the evaluation time window ($t_s$). Some variables that may have changed with time are omitted in our analysis.
The first group of factors is about the general information of companies. The year the company founded and the economic situation of that year are important for startup companies holmes2010analysis. Since we only consider companies founded after 1990 in our dataset, we use the years elapsed since 1990 to denote the founding year for convenience. We take the number of newly established companies in the founding year as an indicator of the macroeconomy. For different time windows, the company age at $t_s$ is an important indicator of the status of a company. It is counted in months in our analysis. News-related factors are useful measures of company performance xiang2012supervised. We count the total number of news and its total number of news associated with each company. The information of geographic environment porter2001innovation,hoenen2012patents and business sectors clarysse2011impact are also very important for the development of startup companies. We quantify the prosperity of an area by the number of companies headquartered in the area registered in Crunchbase. Each company is associated with several industries in Crunchbase. The local prosperity of an industry is quantified by the number of companies associated with the industry in the area. We calculate these factors with the geographical granularity of province and city. When associating the prosperity of the industries to the company, we calculate the average and max local prosperity of all industries associated with the company.
The second group of factors is related to funding rounds and investors. A startup company receives funding in a sequence of rounds. The past funding experience is very important for both startup companies and venture capital firms nahata2008venture, nanda2020persistent. We calculated the number of funding rounds the company achieved and the total amount raised in USD before $t_{s}$. Research nahata2008venture shows that reputable VC firms are more likely to lead their companies to successful exits. The reputation of past investors is evaluated based on their historical investment data. For each investor, we define the IPO fraction as the number of venture deals the investor invested in and exited with an IPO before $t_{s}$ divide by the total number of venture deals of the investor. The definition of the acquisition fraction is similar to the IPO fraction, except we use the fraction of venture deals exited with an acquisition. A company often has more than one investors, we utilize the average and max IPO and acquisition fraction of all its investors.
The third group of factors is about the founders. The experience of the founders jenkins2014individual, littunen2010rapid and the founding team composition nann2010power,eesley2014contingent are also indicators that may influence the potential success of a company. We define the fail fraction as the fraction that a founder had previously founded companies and failed before $t_{s}$. As a company may have more than one founder, we consider the average and max fail fraction of all founders.
We test eight different machine learning classifier algorithms: Logistic Regression, K Nearest Neighbor, Decision Tree, Multilayer Perceptron, Random Forests, XGBoost chen2016xgboost, LightGBM ke2017lightgbm, and soft Decision Tree frosst2017distilling.
We use 90% of the sample in the training phase, 10% for the testing phase. For the training dataset, Synthetic Minority Over-sampling TEchnique (SMOTE) chawla2002smote is used to handle the class imbalance problem, as shown in Table (ref). SMOTE over-samples the minority class by taking each minority class sample and introducing synthetic examples along the line segments joining the k (k=5 in this paper) minority class nearest neighbors. Except for over-sampling methods, some algorithms can handle the class imbalance problem by adjusting weights inversely proportional to class frequencies to the samples in the minority class. For tree-based models (Decision Tree, Random Forests, XGBoost, and LightGBM ) the weights are adjusted in the calculation of split gain. For Logistic Regression, Multilayer Perceptron, and soft Decision Tree, the weights are adjusted in the calculation of the loss function. To optimize the hyperparameters with high efficiency, we use Bayesian optimization to tune the hyperparameters. We choose the best-performed hyperparameters for the experiments, some of the principal parameters are summarized in Table (ref).
The following parameters are used in computing the performance metrics:
ROC curve (receiver operating characteristic curve) swets1988measuring, a graph summarizing classifier performance of a classification model over a range of tradeoffs between TPR and FPR, is used to compare the discriminative power of different models.
The results of different machine learning classifier algorithms are shown in Table (ref). The results without SMOTE or weight adjustment are shown in Table (ref) (a). Due to the class imbalance problem, the results tend to have relatively higher precision and lower recall. The results with SMOTE over-sampling are shown in Table (ref) (b). SMOTE can alleviate the class imbalance problem and improve the recall metric to some extent. Adjusting weight balance in the model leads to better results in most of the models, as shown in Table (ref) (c). The results show that LightGBM and XGBoost with weight adjustment performs best among eight machine learning methods, achieving 53.03% and 52.96% F1, respectively. The data of startup companies are usually incomplete for startup companies, especially for early-stage companies. LightGBM and XGBoost are sparsity-aware algorithms and efficiently solve this data sparsity problem. The ROC curves of different models are shown in Fig (ref).
We define multiple time windows to enrich the number of data samples and to conduct time-aware analysis. There may be a potential risk that the events that happen later in the training set can influence the prediction of earlier events, resulting in lower prediction power for future events. To eliminate this concern, we conduct several experiments comparing models using single and multiple time windows to validate the robustness in predicting current and future events. We use LightGBM for the experiments.
First, we compare the performance of models using single and multiple time windows in predicting in-sample current events. To do this, we use the test set in the last time window to evaluate the performance of the models. Take the time window from January 2009 to June 2010 as an example. For the single time window scenario, we take the 15517 sample events in this time window. Then we split 90% of them as the training set and 10% as the test set. For the multiple time window scenario, we take 37517 sample events from all the time windows before January 2009 and combine them with the training set of the current time window to form the training set. For a fair comparison, the test set of the multiple time window scenario is the same as the single time window scenario. For each time window, we train two models using LightGBM with single and multiple time windows scenarios. We use Bayesian optimization to tune the hyperparameters of each model. The result shows in Fig (ref) (a). Since the time windows start from January 2000, the results of the two scenarios in the first time window are the same. The performance with multiple time windows scenario is slightly better than the single time window scenario, showing that adding historical data helps to improve the performance. The difference is larger in earlier time windows when the amount of sample events is small. After July 2013, there are more than 40,000 startup companies in a single time window, which is large enough to provide enough information and diversity for the prediction. Thus the performance of the two scenarios is similar in recent time windows.
We also compare the performance of models using single and multiple time windows in predicting out-of-sample future events. For this purpose, we use the models trained with historical time windows to predict the future success of the next time window. For example, to predict the future success of the 15517 sample events in the time window from January 2009 to June 2010, we use the models trained on the datasets of time windows before January 2009. For the single time window scenario, we take the 11490 sample events in the time window from July 2007 to December 2008 to form the training set. For the multiple time window scenario, we take the 37517 sample events merging all the time windows before January 2009 to form the training set. The result shows in Fig (ref) (b). Since the time windows start from January 2000 the results of these two scenarios in the first time window are the same. The performance with multiple time windows scenario is slightly better than the single time window scenario. This result agrees with the experiment of in-sample events, showing that adding historical data has a positive impact on the extrapolation to predict future events.
Based on the experimental results, we use LightGBM to explore the importance of different factors. The factor importance is calculated as the total gains of splits which use the feature. The higher the value the more important and predictive the factor. As shown in Fig (ref), the most important factors are company age and past funding experience. The reputation of past investors, local prosperity, macroeconomy, and news also have some predictive power, while the experience of the founders has little influence on the future success of the companies.
When an investor decides whether to take actions based on a prediction, understanding why a model makes a certain prediction is also important. Explaining the reasons behind the prediction provides more insights into the model and makes the prediction more convincing. SHAP (SHapley Additive exPlanations) NIPS2017_7062, a game-theoretic approach to explain the output of any machine learning model, is used to assign each feature an importance value for each prediction.
We take the company Market Logic Software as an example. The output of LightGBM based on the data on 2008-12-31 is 0.71, indicating that the company had a high success probability in the following 18 months (2009-01-01 to 2010-06-30)\footnote{Actually, Market Logic Software raised its next round on May 5 2010.}. Figure (ref) shows how each factor contribute to this output. The base value is the average model output sampled from the training dataset. Features pushing the prediction higher are shown in red, while those pushing the prediction lower are in blue. As shown in the figure, company age, the number of funding rounds and reputable investors have positive effects on the future success of the company, local prosperity of its industrial sector is the main drawback.
To validate that the models trained on current values can predict the future success of companies, we validate the models with out-of-sample periods. We calculate the conditional probability of the success of each company in the time window from Jan 1st, 2019 to Jun 30, 2020, using their features on Dec 31, 2018. This out-of-sample set is composed of 121462 companies, with 20184 of them are successful. This time window is later than the last time window in the training. We construct portfolios of different sizes with LightGBM, XGBoost, and Logistic Regression according to Eq. (ref). We use the number of successful companies in the portfolios to evaluate the predictive power of these machine learning models. The number of companies that succeed in the portfolio versus the portfolio sizes is shown in Figure (ref). The LightGBM and XGBoost model performs better comparing to Logistic Regression, which agrees with the results of the in-sample period test dataset, showing that these two algorithms have a great generalization and extrapolation power in predicting future events. In the portfolio constructed by LightGBM of size 10, 8 succeed in the following 18 months, as listed in Table (ref).
The performance of several top venture capital firms are also shown in Figure (ref). The portfolio sizes of these points are the number of companies that the firm invests in 2018. This result shows that the success rates of machine learning models are better comparing to human experts.
Most of the companies listed in Table (ref) are in their late stages. To further validate the robustness of our models in the early stages, we test the predictive power of the models in early-stage companies. We select companies in investment stages before Series A, Series A, and Series B according to their last funding round before 2019. The top-ranking companies are listed in Tables (ref), (ref), and (ref). The comparison with several top venture capital firms is shown in Figure (ref). The performance of machine learning models is still comparable to human experts.
Our models could help investors to decrease the failure rates in their portfolios. The success rates of machine learning methods are high since we do not consider many practical factors, such as funding size, investment preference, whether the company is within reach, and so on.
In this work, we try to solve the data sparsity problem with recent machine learning methods. We analyze several machine learning methods using a large dataset derived from CrunchBase. We conduct a time-aware analysis based on multiple time windows, which is more practical in real-world scenarios. We expand the scope of success that includes raising new funding, being acquired, or going for an IPO. The results show that the two sparsity-aware algorithms, LightGBM and XGBoost, perform best among eight machine learning methods and achieve 53.03% and 52.96% F1 scores in the prediction, respectively. Through feature mining, we find that company age and past funding experience are among the most important factors. We also interpret the predictions from the perspective of feature contribution. We construct portfolio suggestions according to these methods with out-of-sample periods, which achieve better results compared to human experts. The results show that our methods have great generalization and extrapolation power in predicting future events. These findings have substantial implications on how machine learning methods can help investors identify potential business opportunities.
Future studies will include integrating more data sources and discovering more features, such as features related to founders, public opinions, and so on. Instead of building a model to predict all companies, we will try to build multiple models according to sectors so that we can create customized features to help improve the performance of the prediction in different sectors. We will also focus on the generalization and interpretation of the models. such as introducing causal inference methods to extract the potential reasons for a successful prediction.
We would like to thank Jiren Zhu and Haomin Wang for their supports and great insights on success prediction. We would also like to thank Yaoqiang Xing for his supports on data management. We would like to thank Dr. Kaifu Lee for reviewing the paper and giving very illuminative suggestion. Our work would not have been possible without their support.
\biboptions{authoryear}