A model of discrete choice based on reinforcement learning under short-term memory

frontmatter\address{ [email removed]\\ Department of Mathematics\\ University of Houston\\ 4800 Calhoun Rd. \\ Houston, TX.} \begin{abstract} A family of models of individual discrete choice are constructed by means of statistical averaging of choices made by a subject in a reinforcement learning process, where the subject has short, k-term memory span. The choice probabilities in these models combine in a non-trivial, non-linear way the initial learning bias and the experience gained through learning. The properties of such models are discussed and, in particular, it is shown that probabilities deviate from Luce's Choice Axiom, even if the initial bias adheres to it. Moreover, we shown that the latter property is recovered as the memory span becomes large. Two applications in utility theory are considered. In the first, we use the discrete choice model to generate binary preference relation on simple lotteries. We show that the preferences violate transitivity and independence axioms of expected utility theory. Furthermore, we establish the dependence of the preferences on frames, with risk aversion for gains, and risk seeking for losses. Based on these findings we propose next a parametric model of choice based on the probability maximization principle, as a model for deviations from expected utility principle. To illustrate the approach we apply it to the classical problem of demand for insurance. \end{abstract} \begin{keyword} Discrete choice models \sep Luce's choice axiom \sep reinforcement learning \sep expected utility principle \end{keyword}

section{Introduction} The problem of choice is one of the fundamental problems in psychology, economics and behavioral biology. The second half of the last century saw a rapid growth of theoretical works on this subject as well as increasing amount of experimental data. In economics, the field was dominated by expected utility theory (EUT), its critique based on the experimental evidence and its ramifications. EUT was put forward by Von Neumann and Morgenstern (1947) as a mathematical formalization of what one can call rational preferences between contingent prospects. EUT is an axiomatic theory that starts out with postulates about preferences among prospects. The postulates are completeness, transitivity, continuity and independence (substitution) axioms. The theory derives a utility function $u$ which assigns values to payoffs, and a random prospect $X$ is ranked according to its expectation $\mathbb{E}[u(X)].$ Psychology differs from economics in the approach to the choice behavior by assuming a more general description of choices as being probabilistic and dependent on the set of alternatives offered to a subject. The axiomatic treatment for discrete choice was undertaken in a seminal work by Luce (1959), who introduced a choice axiom (Luce's choice axiom) that postulates how the probability to select an alternative from one set is related to the probability to select this alternative from a larger set. Luce's theory establishes existence of a value function $v$ on a finite set of alternatives $T$ such that the probability to select $i$ from set $S\subset T$ equals \begin{equation} P_S(i) = \frac{v(i)}{\sum_{j\in S}v(j)},\quad i\in S. \end{equation} Psychologically interpreted, value function (scale ratio) $v(i)$ is a subject's response strength for alternative $i.$ Under Luce's choice axiom, choice probabilities verify the principle of independence of irrelevant alternatives of Arrow (1951), and thus, when the latter is normative, becomes a reasonable assumption. Utility function and ratio scale function provide convenient tools for analysis of decision making. Empirically, the values of these functions can obtained by comparing pairs of alternatives. While the first three axioms of EUT are generally accepted, the last one, independence axiom, drew a significant amount of critique from the experimentalists, starting with Allais (1953). Over the years several alternative utility theories were proposed that provide some variants of the expected utility without the independence axiom or with its weaker version. Among them, the generalized expected utility of Machina (1982), weighted utility theory of Chew and MacCrimmon (1979), the regret theory developed independently by Bell (1982), Fishburn (1982), Loomis and Sugden (1982), rank dependent utility theory of Quiggin (1982, 1993), and the dual utility theory by Yaari (1987). Kahneman and Tversky (1979, 1984, 1992) based on their experimental findings, introduced framing effects, value functions, and probability weights into the analysis and incorporated them into the prospect theory that was later developed, using the approach of the rank dependent utility theory, into the cumulative prospect theory. In the theory of discrete choice Luce's choice axiom (LCA) is not a universal imperative either and there are situations where it does not apply, as in the example provided by Debreu (1960). This example was further developed by Tversky (1972) who attributed it to the similarity effect and proposed aspect theory as a refinement of the probabilistic decision making. More detailed discussion of validity of LCA can be found in Luce (1977), or in a more recent review of Pleskac (2013). Let us now return to the work of Luce (1959) and mention its another major contribution, this time to the field of reinforcement learning theories. Learning theories concern with subjects building their choice probabilities through experience, by adapting their responses accordingly to received stimuli. Following Bush and Mosteller (1951, 1955) the learning models were typically formulated in terms of the choice probabilities to select option $i$ at time $n:\, P^n(i),$ which is determined as function of the probabilities from the last period and an outcome of some random event conditioned on the last choice. Luce (1957), citing works of Thurstone (1930) and Gulliksen (1953), argued that learning must be formulated in terms of the strength of response to each alternative, with the choice probabilities being dependent variables of responses, for example, through the relation (ref). Several learning models of this types were proposed typically with a linear law between response for alternative $i$ at times $n$ and $n-1:$ \begin{equation} U^n(i)=\mu U^{n-1}(i) + (1-\mu)u^{n}(i), \end{equation} where $u^{n}(i)$ is the response to the stimuli from the prior selection. This approach has been widely accepted by scientific community, and used in such fields as mathematical biology, game theory, and engineering, see for example Harley (1981), Roth and Erev (1995, 1998), Fudenberg and Levine (1998), Sutton and Barto (1998). There seems to be an unanimous agreement about general principles of reinforcement learning and, naturally, one can turn to it as the tool in constructing models of individual choice. In contrast with axiomatic approaches, one starts with a model of learning. Its parameters should be found experimentally, but once fixed, the model serves as a “decision automaton” that generates a series of choices, or in mathematical language, a stochastic process on a suitable state space. The process is analyzed for convergence to some stationary process, as the number of learning periods increases. In the stationary process the choice probabilities might settle at certain values that change little as more and more choices are being made, but we will not force this assumption. In particular, the stationary process can be consistent with the case when a repeated series of positive experiences with a particular alternative increases the probability to select this alternative in a non-negligible way. What is needed for the theory is existence of well-defined statistical averages for the choice probabilities. The latter are computed and recorded as the probabilities for a discrete choice model. The model can be effectively described as an “expected probability” choice model. However, except for some special cases that we will mention later, it is not an LCA-type model (ref), nor any expected utility-type model. The properties of such models is the focus of this work. To that end we proceed, first, with the mathematical framework. In section (ref) we derive exact formulas for the choice probabilities for a finite set of alternatives, showing by this that our approach is computationally feasible. We introduce a class of $k$--term learning models, where in the process of learning a subject accounts only for responses to the last $k$ stimuli obtained for his/her actions. For example, with $k=1,$ the response strength to alternative $i$ at period $n$ is given by \begin{equation} U^n(i) =U^0(i) + u^n(i),\quad i\in S, \end{equation} where $U^0(i)$ is a learning prior (bias) for alternative $i,$ and $u^n(i)$ is either zero, if alternative $i$ was not selected at period $n-1,$ or, if it was, is the response to the received (random) stimulus for that alternative. Notice that the contribution of the learning prior $U^0(i)$ is not diminishing in the course of learning (as $n$ increases) and it will enter into the formulas of asymptotic probabilities. Another important assumption is implicitly included in (ref). The initial priors $U^0(i)$ do not depend on the subset $S$ of alternatives offered to the subject. That is, at the beginning of the experiment the subject has choice probabilities vitrifying Luce's choice axiom. Modified by learning, they enter into an asymptotic, expected probability model, which in general, loses that property. In section (ref) we turn to applications and our choice here will be on deterministic binary preferences, since they take a prominent place in economics. We consider an individual presented with a set $T$ of $q$ alternatives. For every pair of alternatives $S=\{x,y\}\subseteq T,$ the individual constructs choice probabilities for $x$ and $y$ from set $S,$ according to a learning process described above, where for simplicity we assume that the individual has very short, one period, memory span ($k=1$). The binary preference can be derived from a probabilistic choice model in many different ways, most obvious being the trace relation which defines $x\succ y$ iff the probability to chose $x$ from set $\{x,y\}$ is greater than 1/2. Depending on the parameters of the model, we observe a wide range of behaviors. There are some extreme cases when the binary preferences are according to EUT, while, generically, the preferences are not transitive and violate independence axiom. Intransitivity, in particular, implies that the choice probabilities, from which the binary preferences were derived, violate Luce's choice axiom. There is more to it however, as we show that the binary preferences are characterized by a “framing effect”, with risk averse preferences for gains and risk seeking for losses, similar to the preferences in prospect theory of Kahneman and Tversky (1979, 1984). The binary preferences also shown to detect persistently better alternatives, by adhering to first order stochastic dominance. Intransitivity of the preferences and violations of independence axiom are two phenomena that typically enter any set of empirical data. The fact that they are revealed in our choice model, combined with the fact that the model is founded on the behavioral principles, warrants the interest in the experimental verification of the model. This work however is limited only to presentation of the model and its properties, not to establishing its empirical validity. In section (ref) we introduce a parametric variant of the expected probability model with a partial motivation to facilitate the task of performing statistical tests. The material of sections (ref) and (ref) serves as a motivation for a model for “deviations from the EU principle", which we describe in section (ref). The model is best described as a mediator between two expected utility principles. One is based on maximization of response $\mathbb{E}[u(X)].$ The other is based on the minimization of \[ \mathbb{E}\left[\frac{1}{1+ e^{u(X)}}\right], \] the quantity related to the expected probability. One can notice the dependence of this type of preferences on a “frame" through the shift of scales from $u$ to $u+u_0,$ which can change the ordering of preferences. Interestingly enough it also establishes higher risk aversion for gains, and risk seeking for losses, even if $u$ itself is risk averse (concave), the phenomenon that we have mentioned earlier. A generic case, described by formulas (ref)--(ref) on page (ref), combines two types of EU principles into one. This case, however, is not EU-type principle any longer. At the end of section (ref) we apply the model of “deviations from EUT" to a classical problem of determining demand for insurance.

section{K-period reinforcement learning choice model} \subsection{Response} All reinforcement learning models have three ingredients in common. The reinforcement schedule, the response, as a function of the stimulus, and the choice probabilities depending on the response strength. We will follow the approach of Luce (1959), according to which a subject has a mental record of responses to each choice alternative and updates them according to the realized reinforcement (stimulus) for a corresponding alternative. Then, the subject implements choice through a subject-specific function that selects an alternative with a probability proportional to the response strength for this choice. Consider a succession of experiments in which a subject is offered a stimulus for an alternative he chooses. Let $r_i^n$ be the reinforcement given to the subject at $n^{th}$ trail, when alternative $i$ was chosen last. We assume that each $r_i^n$ is sampled from a random variable $R_i,$ independently from other alternatives and independently from one period to another. Reinforcement is measured in the experiment-dependent units such as dollars, carrots, intensities of light signals, etc. Let $U_i^n$ be the total response strength for alternative $i$ that subject has for this alternative at the end of trial $n.$ It expresses the cumulative effect of the past reinforcements for alternative $i$ on the subject's attitude toward this alternative on some internal scales. {\it Reinforcement-to-Response law:} we will assume that the total response strength is additive over the incremental response strengths from each reinforcement. That is, there is a subject specific function $u=u(r)$ that maps the reinforcement value $r$ into the subject's response scales such that \begin{equation} U_i^n=U^0_i +\sum_{j\in[0..k-1]} u(r_i^{n-j})/N(n,i), \end{equation} where we agreed that if alternative $i$ is not selected during period $j,$ then $u(r_i^j)=0.$ $N(n,i)$ is the count of the number of times during the last $k$ trials when alternative $i$ was selected. A partial justification for the short-memory model are the findings of Kaheman and Trversky (1979) in the context of utility theory. They report that choices are governed by the increments in the subject's wealth rather than the total accumulated wealth. The law (ref) expresses the balance of contributions of the default response $U_i^0$ and new experience to the total response strength, when the former has non-diminishing contribution. (ref) can be thought as a balance between the instinct and the experience, or the balance between things learned in childhood, or bias, and the presents reinforcement. {\it Response-to-Probability law:} choice probability $P_i^n$ for alternative $i$ is determined through the relation \begin{equation} P_i^n=\frac{\Phi(U_i^n)}{\sum_{j=1}^n \Phi(U_j^n)}, \end{equation} where $\Phi=\Phi(U)$ is a re-scaling of the subject's response values $U$ into the range of strictly positive numbers. The function is selected to be non-decreasing to preserve the ordering of responses. \subsection{Statistical count} Consider now an experiment in which a subject is responding according to the rule just described. An outside observer with a capacity for statistical computations will notice that the proportion of times each alternative $i$ is selected becomes fixated with time and settles at certain positive number $\bar{P}_T(i).$ Appendix section gives the formal mathematical account of this type of asymptotic behavior together with the formulas for probabilities $\left\{\bar{P}_T(i)\right\}.$ When the experiment is repeated with $T$ replaced by any subset $S\subset T,$ it leads to probabilities $\left\{\bar{P}_S(i)\right\}$ for alternatives to be selected from $S.$ In this way we obtain a family of probability measures \begin{equation} \bar{P}_S(R)=\frac{\sum_{j\in R}\bar{P}_S(j)}{\sum_{j\in S} \bar{P}_S(j)},\quad R\subsetS, \end{equation} on subsets of $T.$ We will refer to this set of probabilities as RL(k) choice model. The properties of (ref) are the main interest of the paper. Their significance, hypothetically, arises from the fact that the subject may learn the statistics of self behavior and use it next time the choice is to be made.. Or, he/she might run a quick mental “simulation" of the learning process and come up with the same choice probabilities even when presented with choice once. Better yet, with the memory span $k=1,$ when the response is based only on the last reinforcement, the subject can “generalize" other people' one-time experiences about the alternatives as his/her own, in arriving at the probabilities $\bar{P}_T.$ These considerations warrant the interest in theoretical analysis of the family of probabilities (ref). We start with few simple observations. Consider a special case when an experiment provides no reinforcement whatsoever. Then, the choice probabilities \[ \bar{P}_S(R) = \frac{\sum_{j\in R}\Phi(U_j^0)}{\sum_{j\in S}\Phi(U_j^0)}, \] which verify Luce's Choice Axiom. Another extreme case occurs when the memory span becomes increasingly large. \subsection{Long-term-memory limit} Consider model RL(k) with increasingly large values of $k.$ We will proceed informally by noticing that the averages of responses to reinforcement in the last $k$ trails in (ref) converge by the law of large numbers to the mean: \[ \sum_{j\in[0..k-1]} u(r_i^{n-j})/N(n,i) \rightarrow \mathbb{E}[u(R_i)],\quad i=1..q. \] Thus for large values of $k$ and $n,$ the total response strength for alternative $i,$ $U_i^n\approx \bar{U}_i{}={}U_i^0{}+{}\mathbb{E}[u(R_i)],$ i.e. they change little from trial to trial. In the limit we obtain constant response levels and the corresponding choice probabilities become LCA probabilities with the value scales $\Phi(\bar{U}_i),\,i=1..q.$ This is almost the expected utility principle. It is in fact agrees with it if the priors $U_i^0$ are all equal, or if they are proportional to the corresponding expected response strength $\mathbb{E}[u(R_i)].$

section{Binary preference for lotteries} The probabilistic description of choice is more general than algebraic (deterministic), and so, there are many different ways in which the latter can be derived from the former. Given a set of choice probabilities one can, for example, introduce a preference relation for alternatives by postulating some relation between the corresponding probabilities $P_{\{i,j\}}(i)$ and $P_{\{i,j\}}(j).$ One of such preferences is called trace relation which defines $i\succeq j$ if and only if $P_{\{i,j\}}(i){}\geq{}P_{\{i,j\}}(j).$ It was shown by Luce (1959), that if the family of probabilities $P_T$ verifies LCA, then trace relation is a weak order relation. In this section we consider the trace relation for alternatives that are monetary payoffs contingent on random events, with objectively known probabilities. We will assume that the positive scale function $\Phi(u){}={}exp(u/\beta),$ for some positive parameter $\beta,$ the response strength function $u(s)$ equals to the payoff $s$, and denote by $R_i$ random payoff to alternative $i,$ and the initial bias $U^0_i{}={}\mathbb{E}[u(R_i)].$ The rationale for making such selections is the following. Function $\Phi$ is the function from the logit probability model, originated from random utility theory, see Marschak (1960), and it is customary used as a scale function in the models of learning, see for example Fudenberg and Levine (1998). The response $u$ is a linear function. We could have selected $u=ar+b,\,a>0,$ as a generic approximation of arbitrary non-decreasing function, but in order to simplify the exposition we restrict analysis to the case $a=1,\,b=0.$ It reasonable to assume that the initial bias $U^0_i$ is some stochastic characteristic of the random payoff $R_i,$ known a priori, and having the units of “utility” $u(s).$ Thus, the choice of $U^0_i$ as the expected “utility”. In section (ref) we consider slightly more general model. Lastly, we restrict the analysis only to the learning models with the shortest memory span $k=1.$ \subsection{Intransitivity of binary preferences} Let $X,Y$ be two lotteries. Let $\bar{P}_1,\,\bar{P}_2$ be the equilibrium probabilities constructed from model RL(1) with set $T$ of two alternatives. We say that $X\succ Y$ if and only if $\bar{P}_1>\bar{P}_2.$ Equivalently (see appendix), the preference is defined by the inequality \begin{equation} \mathbb{E}\left[\frac{e^{\mathbb{E}[Y]/\beta}}{e^{\mathbb{E}[Y]/\beta}+e^{((\mathbb{E}[X]+X)/\beta)}} \right]< \mathbb{E}\left[ \frac{e^{\mathbb{E}[X]/\beta}}{e^{\mathbb{E}[X]/\beta}+e^{((\mathbb{E}[Y]+Y)/\beta)}} \right]. \end{equation} We define relation $X\succeq Y$ when the inequality in (ref) is not strict, and $X\sim Y$ when the expectations are equal. Notice that if the alternatives are restricted so that they have the same expected payoffs $\mathbb{E}[X]{}={}const.,$ then binary preference $\succ$ corresponds to EU principle with “utility" $ \tilde{u}(s)=(1+ e^{s/\beta})^{-1}.$ We will come back to this property in section (ref). Our first finding is that relation $\succ$ is not transitive. The proof of this is postponed to the appendix as it based on some technical manipulations with the integrals. The intransitivity of the trace relation in its turn implies that choice probabilities $\bar{P}_S,$ violate Luce's choice axiom, see Luce (1959). Intransitivity of preferences is a severe obstacle in constructing any choice theory which limits their usefulness. However, RL(1) model provides much more information than just the binary preferences. In fact, any finite number of lotteries can be evaluated and and unambiguous rankings can be constructed. We will exploit this property in full in section (ref). \subsection{Independence axiom and Allais experiment} The independence axiom for preferences states that if lottery $L_1\succ L_2$ and $L$ is any other lottery, then a complex lottery in which $L_1$ is mixed with $L$ in some proportion dominates $L_2$ mixed with $L$ in the same proportion. This is the axiom for expected utility that drew the earliest critique to Neumann-Morgenstein theory, in particularly by Allais (1953). The axiom requires preference to be linear in the probability distribution. Quick look at the formula (ref) reveals that the binary preference relation depends in a non-linear way on the distribution of the random variables. It is not surprising that such preferences violate the independence axiom of the expected utility theory. To illustrate this property we consider an Allais-type experiment. In what follows we say that lottery $L(a,b:p)$ pays $\$a$ with probability $p$ and $\$b$ with probability $1-p.$ We shorten notation to $L(c:1)$ to describe lottery with 100% of $\$c$ payoff. We compare lottery $L(c:1)$ and $L(1,0:x)$ according to (ref). The result is represented graphically in figure (ref) on page (ref). The red line divides the unit square into two parts: below the line, the certain bet $L(c:1)\succ L(1,0:x),$ above, $L(1,0:x)\succ L(c:1),$ and the line itself is the indifference curve (certainty equivalent curve). Similar to Allais experiment, we mix each lottery with 80% chance of lottery $L(0:1)$ which pays \$0 for sure. The new lotteries to compare are $L(c,0:0.2)$ and $L(1,0:0.2x),$ and the indifference curve is marked blue in figure (ref). The independence axiom requires the same preference between the new lotteries as between the original. For the binary preferences in question, however, for all pairs $(c,x)$ between two curves in figure (ref) preferences are reversed. \subsection{Losses and Gains} Consider now the situation when the wealth increments are framed as losses (negative) or gains (positive). In this setting we are going to look at the problem of determining the certainty equivalent for probabilistic lotteries involving only positive or only negative payoffs. First, we consider lottery $L(1,0:x)$ that pays \$0 with probability $1-x$ and \$1 with probability $x.$ For such lottery we find its certainty equivalent \$c according to (ref). Figure (ref) on page (ref) shows (blue line) the corresponding indifference curve, in the first quadrant. Then, we consider lotteries $L(-1,0:x)$ that pay \$0 with probability $1-x$ and $-1$ with probability $x.$ The red line in the third quadrant is the indifference curve for losses. For comparison, we draw on the same figure the expected payoff curve $\mathbb{E}[L(1,0:x)],$ which is the same the lottery weight $x.$ As seen from the figure, the indifference curve for the binary preferences lies above expected utility in the region of gains, and below it, for losses. Thus, an agent is more risk averse, compared to the expected utility preferences, in gains, and more risk tolerant in losses. This example is a generic fact as proved in the next \begin{lemma} Let $X$ be a random payoff of a lottery. Then, if $X\geq 0$ ($X\leq 0$), the certainty equivalent of $X$ is less (greater) than the expected utility $\mathbb{E}[X].$ \end{lemma} \begin{proof} The certainty equivalent $c$ for lottery $X$ in the binary preferences is defined by the equation \[ \mathbb{E}\left[ \frac{e^{c/\beta}}{e^{c/\beta}+e^{(\mathbb{E}[X] +X)/\beta}}\right]{}={}\frac{e^{\mathbb{E}[X]/\beta}}{e^{\mathbb{E}[X]/\beta} + e^{2c/\beta}}. \] Consider first the gains: $X\geq0.$ Let $p(c)$ denote the left-hand side of the equation, and $q(c)$ the right-hand side. We have: $p'(c)>0,$ and $q'(c)<0.$ Moreover, \[ p(\mathbb{E}[X]){}={}\mathbb{E}\left[ \frac{1}{1+e^{X/\beta}}\right]\geq \frac{1}{1+e^{\mathbb{E}[X]/\beta}}=q(\mathbb{E}[X]), \] because function $(1+e^{x/\beta})^{-1}$ is convex for $x\geq0.$ Thus, the point of intersection, $c,$ of graphs of $p$ and $q$ is not greater than $\mathbb{E}[X].$ The other case is proved analogously, using the fact that $(1+e^{x/\beta})^{-1}$ is concave for $x\leq 0.$ \end{proof} It should be noted that the situation in figure (ref) applies only to two-valued lotteries described there. Since (ref) is not an expected utility no conclusions about other types of lotteries can be drawn from the graph of the certainty equivalents of such lotteries. In particular, the type of convexity of the certainty equivalent curve can not be used to characterize (ref) preferences in the relation to the attitude toward risk. \subsection{First order stochastic dominance} We will show in this section that an individual using binary preferences (ref) can detect the prospect with persistently better payoffs than the other. This concept is formalized as first order stochastic dominance. Consider two lotteries $X$ and $Y.$ It is said that $X$ stochastically dominates $Y,$ if for any non-decreasing function $u$ and any non-increasing function $v,$ \[ \mathbb{E}[u(Y)] \leq \mathbb{E}[u(X)],\quad \mathbb{E}[v(X)] \leq \mathbb{E}[v(Y)]. \] \begin{lemma} If $X$ stochastically dominates $Y,$ then $X \succeq Y$ in the binary preferences (ref). \end{lemma} \begin{proof} Recall that $X$ is better than $Y$ if \[ \mathbb{E}\left[ \frac{e^{\mathbb{E}[Y]/\beta} }{e^{\mathbb{E}[Y]/\beta} + e^{(\mathbb{E}[X]+X)/\beta}}\right] \leq \mathbb{E}\left[ \frac{e^{\mathbb{E}[X]/\beta}}{e^{\mathbb{E}[X]/\beta} + e^{(\mathbb{E}[Y]+Y)/\beta}}\right]. \] This inequality can be proved by using the definition of the stochastic dominance: \begin{multline*} \mathbb{E}\left[ \frac{e^{\mathbb{E}[Y]/\beta} }{e^{\mathbb{E}[Y]/\beta} + e^{(\mathbb{E}[X]+X)/\beta}}\right]= \mathbb{E}\left[ \frac{1}{1 + e^{(\mathbb{E}[X]-\mathbb{E}[Y])/\beta)}e^{X/\beta}}\right]\\ \leq \mathbb{E}\left[ \frac{1}{1 + e^{(-\mathbb{E}[X]+\mathbb{E}[Y])/\beta}e^{X/\beta}}\right] \leq \mathbb{E}\left[ \frac{1}{1 + e^{(-\mathbb{E}[X]+\mathbb{E}[Y])/\beta}e^{Y/\beta}}\right]\\ =\mathbb{E}\left[ \frac{e^{\mathbb{E}[X]/\beta}}{e^{\mathbb{E}[X]/\beta} + e^{(\mathbb{E}[Y]+Y)/\beta}}\right]. \end{multline*} \end{proof}

section{Appendix} \subsection{Model RL(k)} In this section we will give a proof that in the course of the learning according to the process described section (ref), the choice probabilities settle at some equilibrium values and provide the formulas for them. To simplify the presentation but not the generality we will assume that choices are made from the set of all $n$ alternatives. The learning process can be described as a Markov chain on the finite state space $S^k$ of $k$ most recent alternatives that a subject has selected, i.e., \[ S^k{}=\left\{ (i_1,..,i_k)\,:\, i_j{}\in{}1..n \right\}. \] We denote the stochastic process as $X^n{}={}(i^n_1,..,i^n_k).$ From the formula (ref) we see that the state $X^{n+1}$ is completely determined from the current state $X^n,$ i.e. $\{X^n\}_{n=0}^{\infty}$ is a Markov chain. We will proceed with the computation of the transition probabilities from the state $\bar{i}=(i_1,..,i_k)$ to the state $\bar{m}=(m_1,..,m_k),$ that we denote by $p(\bar{m}:\bar{i}).$ This probability is zero unless $m_2=i_1,$ $m_3=i_2,..,m_k=i_{k-1}.$ In the remaining cases, according to (ref) and (ref) for $j=1..n,$ \begin{equation} p(j,i_1,i_2,..,i_{k-1}:i_1,..,i_k)= \mathbb{E}\left[ \frac{\Phi(U_j)}{\sum_{l=1}^n \Phi(U_l)} \right]>0, \end{equation} where \[ U_l{}={}U^0_l {}+{}\sum_{l\in[1..k]} u(R_{i_l})/N(l,i_1,..,i_k,R_{i_1},..,R_{i_k}), \] and $N(l,i_1,..,i_k,r_{i_1},..,r_{i_k})$ is a random variable that counts the number of times alternative $l$ has been selected, given the last $k$ selections $i_1,..,i_k$ and the last $k$ reinforcements $r_{i_1},..,r_{i_k}.$ The expectation in (ref) is with respect to the joint distribution of independent random variables $(R_{i_1},..,R_{i_k}).$ The Markov chain $\{X^n\}_{i=0}^{\infty}$ is irreducible and all states are ergodic, see the monograph of Feller (1957) for the theory Markov chains. This implies that the distribution of $X^n$ converges to an invariant measure on $S^k$ that we denote my $\mu.$ The probability that alternative $i$ has been selected last is computed from $\mu$ by the formula \[ \bar{P}_T(i){}={}\sum_{i_2,..,i_k\in[1..n]} \mu(i,i_2,..,i_k). \] For the learning model with $k=1$ computation of $\{\bar{P}_T\}$ is somewhat simplified. The details are presented below as this is the case of the principle interest of the paper. For $k=1,$ $S^1{}={}\{1,..,n\}$ is simply the set of alternatives, and the transition probabilities equal \[ p(j:i) {}={} \mathbb{E}\left[ \frac{\Phi(U_j)}{\sum_{l=1}^n \Phi(U_l)} \right]{}>{}0, \] where \[ U_l{}={}\left\{ \begin{array}{ll} U^0_i {}+{} u(R_{i}), & l=i,\\ U^0_l, & l\not=i. \end{array}\right. \] The vector of equilibrium measure $\mu=(\mu(1),..,\mu(n))^t$ is determined from the linear system \begin{equation} \mu =M\mu, \end{equation} where the elements of the transition matrix $M$ equal $M_{j,i}{}={}p(j:i).$ The $i^{th}$ equation in this system reads: \begin{multline*} \mu(i)=\sum_{k\not=i}\mu(k)\mathbb{E}\left[ \frac{\Phi(U^0_i)}{\sum_{l\not =k} \Phi(U^0_l)+\Phi(U^0_k+u(R_k))}\right]\\+ \mu(i)\mathbb{E}\left[ \frac{\Phi(U^0_i+u(R_i))}{\sum_{l\not =i} \Phi(U^0_l)+\Phi(U^0_i+u(R_i))} \right]. \end{multline*} It can be rearranged as \begin{multline*} \mu(i)=\sum_{k}\mu(k)\mathbb{E}\left[ \frac{\Phi(U^0_i)}{\sum_{l\not =k} \Phi(U^0_l)+\Phi(U^0_k+u(R_k))}\right]+ \\+ \mu(i)\mathbb{E}\left[ \frac{\Phi(U^0_i+u(R_i))-\Phi(U^0_i)}{\sum_{l\not =i} \Phi(U^0_l)+\Phi(U^0_i+u(R_i))} \right], \end{multline*} or as \begin{multline*} \mu(i)(\Phi(U^0_i))^{-1}\mathbb{E}\left[ \frac{\sum_l \Phi(U^0_l) }{\sum_{l\not =i} \Phi(U^0_l)+\Phi(U^0_i+u(R_i))}\right]\\ = \sum_{k}\mathbb{E}\left[ \frac{\mu(k)}{\sum_{l\not =k} \Phi(U^0_l)+\Phi(U^0_k+u(R_k))}\right]. \end{multline*} The right-hand side of this equation is independent of $i;$ we denote it by $K.$ Also we assign $K_0{}={}\sum_{l} \Phi(U^0_l).$ Then, the last equation provides a formula \begin{equation} \mu(i)=K\Phi(U^0_i)\left(\mathbb{E}\left[ \frac{K_0}{K_0+\Phi(U^0_i+u(R_i))-\Phi(U^0_i)}\right]\right)^{-1},\quad i=1..q. \end{equation} The set of equilibrium choice probabilities $\{\bar{P}_T\}$ is the same as measure $\mu.$ If only two alternatives are present, $T=\{1,2\},$ (ref) can be reduced to a single equation, since $\mu(1)+\mu(2)=1:$ \begin{equation} \mu(1)\mathbb{E}\left[ \frac{\Phi(U^0_2)}{\Phi(U^0_2)+\Phi(U^0_1+u(R_1))} \right]= \mu(2)\mathbb{E}\left[ \frac{\Phi(U^0_1)}{\Phi(U^0_1)+\Phi(U^0_2+u(R_2))} \right], \end{equation} which we use in applications to compute ratio $\mu(1)/\mu(2).$ \subsection{Intransitivity} \begin{lemma} For the binary preferences defined in (ref), there are lotteries $X,Y,Z$ such that $X\succ Y,$ $Z\succ X$ but $Y\succ Z$. \end{lemma} \begin{proof} We will show that there are $X,\,Y,\,Z$ such that $X\succeq Y,$ $Z\succeq X$ but $Y\succ Z,$ i.e. that $\succeq$ is not transitive. The lotteries $X,\, Y,\,Z$ can be suitably perturbed to show that $\succ$ is not transitive as well. Let $X,Y$ be two lotteries such that $\mathbb{E}[X]=\mathbb{E}[Y]=0,$ \begin{equation} \mathbb{E}\left[ \frac{1}{1+e^{X}}\right]=\mathbb{E}\left[ \frac{1}{1+e^{Y}}\right], \end{equation} and \begin{equation} \mathbb{E}\left[ \frac{e^X}{(1+e^{X})^2}\right]>\mathbb{E}\left[ \frac{e^Y}{(1+e^{Y})^2}\right] \end{equation} Let $\bar{Z}$ be a lottery with $\mathbb{E}[\bar{Z}]=0$ and \begin{equation} \mathbb{E}\left[ \frac{1}{1+e^{\bar{Z}}}\right]>\mathbb{E}\left[ \frac{1}{1+e^{X}}\right]. \end{equation} First we establish the following \begin{claim} For any number $z_0>0$ there is $z\in(0,z_0)$ and a lottery $Z$ such that $\mathbb{E}[Z]=z,$ and \begin{equation} \mathbb{E}\left[ \frac{1}{1+e^{z+Z}}\right]=\mathbb{E}\left[ \frac{1}{1+e^{-z+X}}\right]. \end{equation} \end{claim} \begin{proof} Consider functions $f(z) = \mathbb{E}\left[ \frac{1}{1+e^{2z+\bar{Z}}}\right]$ and $g(z) = \mathbb{E}\left[ \frac{1}{1+e^{-z+X}}\right].$ From (ref) we have $f(0)>g(0).$ Moreover $f(z)$ is monotone function with $f'(0)<0,$ and $g(z)$ is monotone with $g'(0)>0$ as can be verified by taking derivatives. If lottery $\bar{Z}$ is chosen sufficiently close to $X,$ graphs of $f(z)$ and $g(z)$ will intersect at some point in the interval $(0,z_0).$ For such point, call it $z,$ and lottery $Z = z+\bar{Z},$ equation (ref) holds. \end{proof} Condition (ref) means that $X\succeq Y.$ Consider function $f(z) = \mathbb{E}\left[ \frac{1}{1+e^{-z+X}}\right]$ and $g(z) = \mathbb{E}\left[ \frac{1}{1+e^{-z+Y}}\right].$ Conditions (ref),(ref) imply that there is $z_0>0$ such that for all $s\in(0,z_0),$ \[ \mathbb{E}\left[ \frac{1}{1+e^{-s+X}}\right] > \mathbb{E}\left[ \frac{1}{1+e^{-s+Y}}\right]. \] Given this $z_0$ and using the claim we find $z\in(0,z_0)$ and lottery $Z,$ such (ref) holds, which implies that $Z\succeq X.$ On the other hand \[ \mathbb{E}\left[ \frac{1}{1+e^{-z+X}}\right] > \mathbb{E}\left[ \frac{1}{1+e^{-z+Y}}\right]. \] Thus, \[ \mathbb{E}\left[ \frac{1}{1+e^{z+Z}}\right]{}>{}\mathbb{E}\left[ \frac{1}{1+e^{-z+X}}\right]. \] By formula (ref) it implies that $Y\succ Z$ (recall that $\mathbb{E}[Z]=z,\, \mathbb{E}[Y]=0$), while (ref) establishes that $X\succeq Y.$ \end{proof}

thebibliography{XXX} \bibitem{Allais} Allais, M. (1953) Les comportement de l'homme rationnal devant le risque: critique des postulates and axioms de l'ecole americaine. Econometrica, 21. \bibitem{Arrow} Arrow, K. (1951). Social Choice and Individual Values. Wiley & Sons, New York, NY. \bibitem{BM1} Bush, R.R., and Mosteller, F. (1951). A mathematical model for simple learning. Psychological Review 58, 313--323. \bibitem{BM2} Bush, R.R. and Mosteller, F. (1955). Stochastic models for learning. Wiley & Sons, New York, NY. \bibitem{Erev_Roth} Erev, I. and Roth, A. E. (1996). On the need of low rationality cognitive game theory: reinforcement learning in experimental games with unique mixed equilibria. Mimeo. University of Pittsburgh. \bibitem{Erev_Roth_I} Erev, I. and Roth, A. E. (1998). Predicting how people play games: reinforcement learning in experimental games with unique, mixed strategy equilibrium. American Econ. Review 88, 848--881. \bibitem{Feller} Feller, W. (1957). An Introduction to Probability Theory and Its Applications. Vol II. John Wiley & Sons, New York, NY. \bibitem{FudenbergLevine} Fudenberg, D., and Levine, D. (1998). The theory of learning in games. MIT Press, Cambridge, MA., London, England. \bibitem{Harley} Harley, C.B. (1981). Learning the Evolutionary Stable Strategy. J. theor. Biol. 89, 611--633. \bibitem{KT1} Kahneman, D., and Tversky, A. (1979). Prospect Theory: An Analysis of Decision under Risk, Econometrica, XVLII 263--291. \bibitem{KT2} Kahneman, D., and Tversky, A. (1984). Choices, Values and Frames. American Psychologist 39, 341--350. \bibitem{Marschak} Marschak, J. (1960). Binary choice constraints on random utility indicators. in K. Arrow (ed.), Stanford Symposium on Mathematical Models in the Social Sciences, Stanford University Press, Stanford, CA. \bibitem{Luce} Luce, R. D. (1959). Individual Choice Behavior. Wiley &Sons, New York, NY. \bibitem{Luce1} Luce, R. D. (1977). The choice axiom after twenty years. Journal of Math. Psych. 15, 215--233. \bibitem{Mach} Machina, M.J. (1982). Expected utility analysis without the independence axiom". Econometrica. 50 (2), 277--323. \bibitem{Q1} Quiggin, J. (1982). A theory of anticipated utility. Journal of Economic Behavior and Organization 3(4), 323--343. \bibitem{Q2} Quiggin, J. (1993). Generalized Expected Utility Theory. The Rank-Dependent Model. Kluwer Academic Publishers, Boston, MA. \bibitem{Pleskac} Pleskac, T. (2015). Decision and Choice: Luce's Choice Axiom. in P. Bona, (ed.) International Encyclopedia of the Social & Behavioral Sciences. \bibitem{RE} Roth, A.E., and Erev, I. (1995). Learning in extensive-form games: experimental data and simple dynamics models in the intermediate term. Games and Economic Behavior, 8 164--212. \bibitem{Sutton} Sutton, R.S. and Barto, A.G. (1998). Reinforcement learning: an introduction. MIT Press, Cambridge MA. \bibitem{KT3} Tversky, A., Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5:297--323. \bibitem{VNM} Von Neumann, J. and Morgenstern, O. (1947). Theory of Games and Economic Behavior. Princeton, NJ: Princeton University Press. \bibitem{Y} Yaari, M. (1987). The dual theory of choice under risk. Econometrica, 55, 95--115.