Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
33,536 characters · 15 sections · 18 citation commands
Identifying Different Definitions of Future in the Assessment of Future Economic Conditions: Application of PU Learning and Text Mining
The Economy Watcher Survey is a market survey published by the Japanese government. The data consists of two types of assessments of economic conditions, current and future economic conditions, with five ranks. Although this survey provides policymakers with deep insights, it is difficult to interpret the assessments of future economic conditions because the meaning of future is not clearly defined and the definition thereof relies on the respondent's interpretation. Therefore, to obtain a clear understanding of survey participants’ expectations, our approach was to classify assessments of future economic conditions into those pertaining to the near and distant future, respectively. This led us to propose a novel method that uses text data and a machine-learning algorithm in an attempt to grasp these expectations with respect to future economic conditions using data from the Economy Watcher Survey. For the classification task, we apply an algorithm that learns from positive and unlabeled data (\emph{PU learning}), which is a machine-learning algorithm that enables us to train a classifier only from positive and unlabeled data.
Among studies of economic trends, methods using information contained in text data have become popular. Pioneering methods in this field are doi:10.1111/j.1540-6261.2007.01232.x,RePEc:bla:jfinan:v:63:y:2008:i:3:p:1437-1467, which involved the construction of sentimental indexes from articles of a column in the Wall Street Journal and an analysis of the predictability of the stock market. article0 predicted the residential price by using the number of searches on Google. article1 also constructed real-time inflation expectations from search queries on Google.
PU learning is an algorithm of weakly supervised learning elkan2008learning,ward2009presence,blanchard2010semi,nguyen2011positive. In the section describing the problem setting, we consider a situation in which only positive and unlabeled data exist, and use only these data to train a binary classifier. PU learning has two scenarios known as censoring scenario and case-control scenario elkan2008learning. In the study presented in this paper, we only focus on the case-control scenario, in which positive data are obtained separately from unlabeled data, and unlabeled data are sampled from the entire population. In this study, we construct our algorithm on the basis of subsequent research known as unbiased PU learning ICML:duPlessis+etal:2015, which minimizes the unbiased estimator of the classification risk.
After classifying the assessments of future economic conditions into those relating to the near and distant future, we calculated the averaged ranks for both the near and distant future. As a result, we found that a significant difference exists between economic conditions relating to these two future periods. This result infers the possibility that people's definition of the future differs. This fact is important from the viewpoint of economics. In macroeconomics, a researcher may be interested in the possibility of controlling people's expectations of the market. Our empirical analysis reports the fact that assessments of the economic conditions of the distant future were mainly based on economic fundamentals such as the population and diplomatic relationships.
In the following sections, we describe our problem setting and propose an algorithm that solves the problem. Subsequently, we present the results and interpretations of our empirical analysis.
We consider the binary classification of text data. In the following parts, we describe the dataset and classification problem in detail.
In our analysis, we used the Economy Watchers Survey, a dataset that contains text data and is published by the Japanese government \footnote{Particulars of the dataset are provided on the homepage of the Japanese government, \url{https://www5.cao.go.jp/keizai3/watcher-e/index-e.html}. The survey enlists the cooperation of people holding employment positions that enable them to observe activity closely related to the regional economy. We downloaded the dataset from the page.}. The purpose of this survey is to enable the region-by-region economic trends to be grasped accurately. This survey consists of two assessments, an assessment of current and future economic conditions with the possibility of entering sentences to motivate the answers by providing reasons. Respondents evaluated the current and future economic conditions by five ranks, $0,1,2,3,4$. The evaluation $0$ means “worse” or “will get worse” compared with a previous period. The evaluation $4$ means “better' or “will get better” compared with a previous period. The evaluation $2$ represents a neutral position on the assessment of economic conditions.
\paragraph{Interpretation of Assessment of Future Economic Conditions:} Assessments of current and future economic conditions provide us with deep insights into economic reality. However, in the questionnaire, there is no clear definition of the concept of the “future” with respect to future economic conditions. Hence, different people interpret the duration of “future” in their own way. Whereas one person may imagine the future as just one week, the “future” might be a few months for another person. Therefore, to analyze the assessments more accurately, we need to classify assessments of future economic conditions as being either near or distant economic conditions.
To classify future economic conditions into those expected to occur in either the near or distant future, we propose assuming that current economic conditions share similar sentences with those expected in the near future. Our classification strategy is to regard current economic conditions as positive data and future economic conditions as unlabeled data, which potentially consists of positive and negative data. In this paper, positive data are assessments of the current economic conditions and those expected in the near future, whereas negative data are assessments of economic conditions foreseen to prevail in the distant future. We illustrate the relationship between assessments of current and future economic conditions of our assumption on Figure (ref). We train our classifier only from positive and unlabeled data by using an algorithm that employs PU learning. Therefore, the goal of this problem is to classify $\bm{x} \in \mathcal{X} \subset \mathbb{R}^d$ into one of the two classes $\{-1, +1\}$, where $+1$ denotes assessments of current economic conditions and those expected in the near future (positive data) and $-1$ denotes economic conditions relating to the distant future (negative data).
Let us describe the data generating process of our problem. Let us assume that we have $n$ data points at $t$-th period and denote the $i$-th text data as $\bm{x}_i \in \mathcal{X} \subset \mathbb{R}^d$. If the target of text data $\bm{x}_i$ describes current or near future economic conditions, we attach a positive label, i.e., $y_i=+1$. If the target of text data $\bm{x}_i$ describes distant future economic conditions, we attach a negative label, i.e., $y_i=-1$. However, in the dataset, we can only observe positive data, and unlabeled data, which includes both positive and negative data. In addition, if the text data $\bm{x}_i$ belongs to a period $t\in\{1,...,T\}$, we denote the fact as $z_i=t$. Using these notations, we define our data generating process as follows:
where ${\{\bm{x}_i\}_{i=1}^{n}}$ and ${\{\bm{x}'_i\}_{i=1}^{n'}}$ denote the positive and negative data at $t$-th period, and $p(\bm{x}|z=t)$ can be decomposed as
To classify data consisting only of positive and unlabeled data, we propose using multi-task PU learning (MTPU). In this section, we provide details of the proposed algorithm.
Before explaining our model, let us explain the standard setting of PU learning. In PU learning, we consider a binary classification problem to classify $\bm{x} \in \mathcal{X} \subset \mathbb{R}^d$ into one of the two classes $\{-1, +1\}$. We assume that there exists a joint distribution $p(\bm{x}, y)$, where $y \in \{-1, +1\}$ is the class label of $\bm{x}$. PU learning relies on two distinct sampling schemes, namely the censoring scenario and case-control scenario elkan2008learning. The PU learning framework we use in this study is the case-control scenario, in which we suppose access to a positive dataset ${\{\bm{x}_i\}_{i=1}^{n}}\overset{\mathrm{i.i.d.}}{\sim} p(\bm{x}|y=+1)$ and an unlabeled dataset ${\{\bm{x}'_i\}_{i=1}^{n'}}\overset{\mathrm{i.i.d.}}{\sim} p(\bm{x})$. Let $\ell: \mathbb{R}\times \{\pm1\}\to {\mathbb{R}^{+}}$ be a loss function, where ${\mathbb{R}^{+}}$ is the set of non-negative real values, and $\mathcal{F}$ be the set of measurable functions from $\mathcal{X}$ to $[\epsilon, 1-\epsilon]$, where $\epsilon \in (0, 1/2)$ is a small positive value. This constant $\epsilon$ is introduced to ensure the following optimization problem is well-defined based on the result of kato2018learning. Here, ICML:duPlessis+etal:2015 showed that the classification risk of $f \in \mathcal{F}$ can be expressed as
where $\mathbb{E}_\mathrm{p}$ and $\mathbb{E}_\mathrm{u}$ are the expectations over $p(\bm{x}|y=+1)$ and $p(\bm{x})$, respectively. The above formulation of PU learning provides the unbiased risk of the classification problem.
In addition to the standard setting of PU learning, we could take the time structure into account. The Economy Watcher Survey comprises monthly data, with approximately $2,600$ records for each month. Here, we would need to use different classifiers for the data included in each month for the following two reasons. First, the model can vary across periods. Second, we would not be able to include data of the $(t+1)$-th period to train a model of data of the $t$-th period because the data of the $(t+1)$-th period might have information of the data of the $t$-th period. This made it necessary to use different models across different periods. For $z=t$, we denote the model as $f_{z=t}$ and the risk as follows:
where $\hat{\mathbb{E}}_{\mathrm{p, t}}$ denotes the averaging operator over positive data, $\hat{\mathbb{E}}_{\mathrm{u, t}}$ denotes averaging over the unlabeled data at the $t$-th period. We additionally introduce a model for multi-task learning to PU learning. Multi-task learning is proposed to train neural networks efficiently by using the common features across different tasks Caruana1997. If a common feature exists across periods, we can train our models more efficiently by sharing the common feature among models $f_{z=t}$ for $t=1,...,T$ through the layers named shared layers, the structure of which is shown in Figure (ref).
We named this model MTPU. Details of its structure are provided in the section for empirical experiments.
When we train a classifier, we can naively replace the expectations with the corresponding sample averages. However, NIPS2017_6765 pointed out that the basic form of the unbiased PU learning is ineffective with a deep neural network because of over-fitting caused by the fact that the risk is not lower bounded. To implement PU learning with deep neural networks, we applied the non-negative risk proposed by NIPS2017_6765 to the empirical risk defined in ((ref)). For a hypothesis set $\mathcal{H}$, let us define the following risk minimization problem,
where $\widehat{R}_{\mathrm{nnPU}}(f_{z=t}, z=t)$ is a sample approximation of $R_{\mathrm{PU}}(f_{z=t}, z=t)$ with non-negative transformation proposed by NIPS2017_6765 and $\mathcal{R}$ is a regularization term.
The remaining problem is to make a decision regarding the class prior $p(y=+1|z=t)$. The class prior $p(y=+1|z=t)$ would be different across periods $t$. Although several algorithms have been proposed to estimate the class prior IEICE:duPlessis+Sugiyama:2014,pmlr-v48-ramaswamy16,jain2016nonparametric, the estimation is still known to be a difficult task. However, we can avoid the problematic estimation in the case of the particular goal we hope to reach. In our experiments, we assume that the class prior is $p(y=+1|z=t)=0.2$ for all periods, $t=1,2...,T$. This assumption is not realistic because the probability would have different values across the periods. However, 1809.05710,kato2018learning showed that the function $f_{z=t}$ is simply linear-proportional to the class prior, i.e., the following relationship holds even if we miss-specify the class prior:
Therefore, even when we cannot obtain the exact value of $p(y=+1|\bm{x}, z=t)$, we can still identify the order of $p(y=+1|\bm{x}, z=t)$ with regard to $\bm{x}$. Our empirical analysis separates the assessment of future economic conditions into near and distant future economic conditions based on this property. We classify $1/5$ of data from the highest value of $f_{z=t}$ into assessments of near future economic conditions, and $1/5$ of data from the lowest value of $f_{z=t}$ into assessments of distant future economic conditions. In addition to the robustness to the miss-specified class prior, the function $f_{z=t}$ also holds the relationship (ref) under the selection bias of positive data kato2018learning if our assumption is mild. Thus, our results can reduce the influence of the miss-specified class prior and selection bias.
In this section, we report the results of the empirical analysis of data from the Economy Watcher Survey. The survey was conducted every month starting in 2000. Our analysis only used data from January 2016 to June 2019, i.e., $42$ months' data. Each month includes approximately $2,600$ samples. The reason for the heterogeneity among the data is the lack of text in respondents’ answers. In total, we had 111,501 samples.
We used Bag-of-Words to represent the documents as $16,914$-dimensional vectors. After vectorizing the text data, we applied PU learning with the aforementioned MTPU. In addition to the model, we also used the standard model of PU learning to compare the performance. We used this model of PU learning in two ways. First, we used all samples to train one model. Second, we prepared one model for each month. Details of the neural networks are provided in the following section. After training our classifier, we classified the assessment of future economic conditions using unlabeled data that we used for training.
\paragraph{Neural network model:} First, we describe the model we used for MTPU. The model for the shared network was a $3$-layer multilayer perceptron (MLP) with ReLU Nair:2010:RLU:3104322.3104425 (more specifically, $16914-500-500-500$). The neural network model following the shared network was a $2$-layer MLP (more specifically, $500-500-1$) with ReLU. Next, we describe the model we used for non-negative PU learning. The model for the neural network was a $5$-layer MLP (more specifically, $16914-500-500-500-500-1$) with ReLU. We set $p(y=+1|z=t)=0.2$ for all $t\in\{1,2,...,42\}$. For both methods, we use logistic loss for the loss function $\ell$.
In this section, we report the extent to which assessments differ across current, future, near future, and distant future economic conditions.
\paragraph{Averaged Assessments and t-test:} We report the averaged assessments of economic conditions in the near and distant future in comparison with those of the current and future. Assessments of the near and distant future economic conditions are estimated by MTPU and non-negative PU learning with neural networks. For non-negative PU learning, we used two models. The first (named PU1) entailed training one model for all samples. The second (named PU2) involved using different models for the data of different months. The results are presented in Table (ref). For each period, we show the results of the two-sample t-test with unequal variances between the assessments of the economic conditions of the near and distant future. Values for which the difference between the mean of the assessments is significant are indicated by superscript $*$ in the table. One $*$ and two $*$s mean that the null hypothesis of the two-sample t-test is rejected at the $5\%$ and $1\%$ significance levels, respectively.
\paragraph{Visualization as a Time Series:} To facilitate a more intuitive understanding of the reported results, we plotted the averaged assessments in the time series in Figure (ref), where the $x$-axis corresponds to the time series, and the $y$-axis corresponds to the value of the assessment. The blue, orange, green, and red lines correspond to assessments of the economic conditions in the near future, distant future, at the present time, and in the future. The horizontal black dashed line at $y=2.0$ represents the neutral condition in the $5$-step evaluations for the economic conditions from $0$ (bad) to $4$ (good). The vertical red lines perpendicular to the line $y=2.0$ indicate that the difference between the average assessments of the economic conditions in the near and distant future is significant in the two-sample t-test. The bold vertical lines represent that the null hypothesis of the two-sample t-test is rejected at the $1\%$ significance level and the other red lines represent that two $*$ means that the null hypothesis of the two-sample t-test is rejected at the $5\%$ significance level. For example, the assessments of the economic conditions in the near future in 2017 are significantly higher than those of the distant economic conditions.
This section presents our analysis of the text based on assessments of the text data. For text mining, we use tf-idf and the Jaccard coefficient, which are standard techniques of natural language processing. First, we separate the assessments of the economic conditions in the near and distant future for the month in which the assessments were published, i.e., we form groups of monthly assessments. Then, we denote a set of the group of assessments as $\mathcal{M}$, and we apply tf-idf to identify the word that characterizes the document. Then, for the $50$ words with the highest tf-idf, we measure the Jaccard coefficient Manning:1999:FSN:311445, which measures the similarity between two sets. Let $\mathcal{M}_w\in\mathcal{M}$ be a set of sentences including the word $w$. The Jaccard coefficient $J(\mathcal{M}_a, \mathcal{M}_b)$ for a word $a$ and a word $b$ can be expressed as follows:
Based on these results, we plotted the co-occurrence networks in Figure (ref)\footnote{We translated from Japanese to English using an API provided by Google (\url{https://pypi.org/project/googletrans/}). }. Because of the limitation placed on the length of the paper, we only show the network of assessments in June 2016 and February 2017. June 2016 is one of the periods in which the value of assessments greatly changed. Throughout 2017, the economic conditions of the near future are less than those in the distant future, and February 2016 is one of these periods. Because of the small size of our graphs, we placed enlarged versions of these graphs in the appendix in both English and Japanese.
Figure (ref) displays words related to economic fundamentals, such as the structure of the labor supply and international politics. In other words, these results can be interpreted as meaning that assessments of the economic conditions of the near future represent the economic cycle, whereas assessments of the economic conditions of the distant future represent the economic trend. For example, the words “U.K.” and “withdrawal” appear, both of which are related to Brexit among the economic conditions of the distant future, “Business cycle,” and “Trend” in June 2016. The words “US” and “President” appear in Feb. 2017. On the other hand, the economic conditions of the near future in June 2016 and Feb, 2017 are represented by words that have less relationship with economic fundamentals such as “rainy season” and “Valentin's day.” For policymakers, this is an insightful finding because the result infers that they cannot easily change people's expectations based on economic fundamentals.
In this paper, we proposed a new application of PU learning and text mining to data consisting of financial text. We developed a new model named MTPU to train neural networks efficiently using data with a time structure. Our empirical analysis showed the classification result and interpretations based on text mining and economics. The result is insightful to policymakers because the result infers that people might have a different interpretation of the definition of the future and may assess the future economic outlook differently based on their interpretations of the future. Besides, we also found that there are different main reasons between near and distant future economic assessments.
\onecolumn