The value of initiating a pursuit in temporal decision-making
- Department of Neuroscience, Johns Hopkins University School of Medicine, United States
- Kavli Neuroscience Discovery Institute, Johns Hopkins University, United States
- Microsoft, United States
- The Department of Neuroscience, Johns Hopkins University School of Medicine, United States
Figures
Figure 1
Fundamental classes of temporal decision-making regarding initiating a pursuit: ‘Forgo’ and ‘Choice’.
(A, B) (Top row) Topologies. The temporal structure of worlds exemplifying Forgo (A) and Choice (B) decisions mapped as their topologies. Forgo: A Forgo decision to accept or reject the purple …
Figure 2
Global reward rate with respect to parceling the world into ‘in’ and ‘outside’ the considered pursuit.
Figure 3 with 2 supplements
Forgo decision-making.
(A) When the reward rate of the considered pursuit (slope of the purple line) exceeds that of its outside rate (slope of gold), the global reward rate (slope of magenta) will be greater than the …
Figure 3—figure supplement 1
Forgo world with n-pursuits occurring at the same frequency.
An agent experiencing a Forgo decision making world (topology as in (A), left) where n-pursuits of varying times and rewards (purple, aqua, orange) occurring at the same frequency from the default …
Figure 3—figure supplement 2
Forgo world with n-pursuits occurring at different frequencies.
An agent experiencing a Forgo decision making world (topology as in (A), left) where n-pursuits of varying times and rewards (purple, aqua, orange) occurring at different frequency from the default …
Figure 4
The subjective value (sv) of a pursuit is the global reward-rate-equivalent immediate reward magnitude.
The subjective value (green bar) of a pursuit is that amount of reward requiring no investment of time that the agent would take as equivalent to a policy of accepting and acquiring the considered …
Figure 5
Equivalent expressions for subjective value reveal time’s cost comprises an apportionment as well as opportunity cost.
(A) The subjective value of a pursuit can be expressed in terms of the global reward rate obtained under a policy of accepting the pursuit. rin = 4, tin = 4, rout = .7, tout = 3. (B) The cost of …
Figure 6
The impact of outside reward on the subjective value of a pursuit.
(A) The subjective value (green dot) of a considered pursuit type (purple) in the context of its ‘outside’ (gold) is the resulting global reward rate vector’s (magenta) intersection of the y-axis in …
Figure 7
The impact of the apportionment cost of time on the subjective value of a pursuit.
(A) The apportionment cost of time can best be illustrated dissociated from the contribution of the opportunity cost of time by considering the special instance in which the outside has no net …
Figure 8
The effect of changing the outside time and thus the outside reward rate, on the subjective value of a pursuit.
(A) The subjective value (green dot) of the considered pursuit, when (B) changing the outside time, and thus, outside reward rate (green dots). (C) As outside time increases under these conditions …
Figure 9
Policy options considered during the initiation of pursuits in worlds with a ‘Choice’ topology.
(A-C) Choice topology, and policies of choosing the smaller-sooner or larger-later pursuit, as in Figure 1 ‘Choice’. (D) The world divided into ‘Inside’ and ‘Outside’ the selected pursuit type, as …
Figure 10
The effect of increasing outside reward on subjective value in choice decision-making.
The effect of increasing the outside reward while holding the outside time constant is to linearly increase the cost of time, thus decreasing the subjective value of pursuits considered in choice …
Figure 11
Effect of apportionment cost on subjective value in Choice decision-making.
The effect of apportionment cost can be isolated from the effect of opportunity cost by increasing the outside time while holding outside rate constant. Doing so results in decreasing the …
Figure 12
Effect of varying outside time and outside reward rate.
The effect of increasing the outside time while maintaining outside reward is to decrease the apportionment as well as the opportunity cost of time, thus increasing pursuit’s subjective value. …
Figure 13
The temporal discounting function of a global reward-rate-optimal agent is a hyperbolic function relating the apportionment and opportunity cost of time.
(A-C) The effect, as exemplified in three different worlds, of varying the outside time and reward on the subjective value of a pursuit as its reward is displaced into the future. The subjective …
Figure 14
Reward-rate-maximizing agents would exhibit the ‘Delay Effect’.
A ‘switch’ in preference from a SS―when the delay to the pursuits is relatively short (upper left)―to a LL pursuit, when the delay to the pursuits is relatively long (upper right), would occur as a …
Figure 15
Reward-rate-maximizing agents would exhibit the ‘Magnitude effect’.
(A, B) The global reward rate (the slope of magenta vectors) that would be obtained when acquiring a considered pursuit’s reward of a given size (either relatively large as in A or small as in B) …
Figure 16
Reward-rate-maximizing agents would exhibit the ‘Sign effect’.
(A, B) The global reward rate (the slope of magenta lines) that would be obtained when acquiring a considered pursuit’s outcome of a given magnitude but differing in sign (either rewarding as in A, …
Figure 17
Relationship between outside time and reward with optimal temporal decision-making behavioral transitions.
An agent may be presented with three decisions: the decision to take or forgo a smaller, sooner reward of 2.5 units after 2.5 s (SS pursuit), the decision to take or forgo a larger, later reward of …
Figure 18
Patterns of suboptimal temporal decision-making behavior resulting from time and/or reward misestimation.
Patterns of temporal decision-making in Choice and Forgo situations deviate from optimal (top row) under various parameter misestimations (subsequent rows). Characterization of the nature of …
Figure 19
The cost of time of a pursuit comprises both an opportunity as well as an apportionment cost.
The global reward rate under a policy of accepting the considered pursuit type (slope of magenta time), times the time that pursuit takes (tin), is the pursuit’s time’s cost (height of maroon bar). …
Figure 20
Comparison of typical hyperbolic discounting versus apparent discounting of a reward-rate-optimal agent.
Whereas (A) the curvature of hyperbolic discounting models is typically controlled by the free fit parameter k, (B) the curvature and steepness of the apparent discounting function of a …
Figure 21
The Malapportionment Hypothesis.
(A-E) Solid lines indicate true reward-rate maximizing values. Dashed lines indicate those of an agent described by the Malapportionment Hypothesis that underweights the apportionment of time …
Tables
Table 1
Definitions for misestimating global reward rate-enabling parameters.
Each misestimated variable (column 1) is multiplied by an error term, , to give , the misestimated global reward rate (column 2). When the variable is underestimated, when the variable is …
| Misestimated Variable | Misestimated Global Reward Rate |
|---|---|
| True (No Misestimation) | |
| Outside Time | |
| Outside Reward | |
| Outside Time and Reward (maintaining ) | |
| Inside Time | |
| Inside Reward | |
| Inside Reward and Time (maintaining ) |
Table 2
Opportunity cost, apportionment cost, time cost, and subjective value functions by change in outside and inside reward and time.
Functions assume positive inside and outside rewards and times.
| Reward | Time | |||
|---|---|---|---|---|
| Outside | Inside | Outside | Inside | |
| Opportunity Cost* | Linear Positive slope | No Effect | Hyperbolic Negative slope | Linear Positive slope |
| Apportionment Cost | Linear Negative slope | Linear Positive slope | Hyperbolic - Hyperbolic† Negative slope | Hyperbolic - Linear† Negative slope |
| Time’s Cost | Linear Positive slope | Linear Positive slope | Hyperbolic Negative slope | Hyperbolic Positive slope |
| Subjective Value | Linear Negative slope | Linear Positive slope | Hyperbolic Positive Slope | Hyperbolic Negative slope |
-
*
If outside reward rate is zero, opportunity cost becomes a constant at zero.
-
†
If outside reward rate is zero, as outside or inside time is varied, apportionment cost becomes purely hyperbolic.
