Reward-based training of recurrent neural networks for cognitive and value-based tasks
- New York University, United States
- NYU Shanghai, China
Figures
Figure 1 with 4 supplements
Recurrent neural networks for reinforcement learning.
(A) Task structure for a simple perceptual decision-making task with variable stimulus duration. The agent must maintain fixation () until the go cue, which indicates the start of a decision …
Figure 1—figure supplement 1
Learning curves for the simple perceptual decision-making task.
(A) Average reward per trial. Black indicates the network realization shown in the main text, gray additional realizations, i.e., trained with different random number generator seeds. (B) Percent …
Figure 1—figure supplement 2
Reaction-time version of the perceptual decision-making task, in which the go cue coincides with the onset of stimulus, allowing the agent to choose when to respond.
(A) Task structure for the reaction-time version of the simple perceptual decision-making task, in which the agent can choose to respond any time after the onset of stimulus. (B) Reaction time as a …
Figure 1—figure supplement 3
Learning curves for the reaction-time version of the simple perceptual decision-making task.
(A) Average reward per trial. Black indicates the network realization shown in the main text, gray additional realizations, i.e., trained with different random number generator seeds. (B) Percent …
Figure 1—figure supplement 4
Learning curves for the simple perceptual decision-making task with a linear readout of the decision network as the baseline.
(A) Average reward per trial. Black indicates the network realization shown in the main text, gray additional realizations, i.e., trained with different random number generator seeds. (B) Percent …
Figure 2 with 3 supplements
Performance and neural activity of RNNs trained for 'simple' cognitive tasks in which the correct response depends only on the task condition.
Left column shows behavioral performance, right column shows mixed selectivity for task parameters of example units in the decision network. (A) Context-dependent integration task (Mante et al., 2013…
Figure 2—figure supplement 1
Learning curves for the context-dependent integration task.
(A) Average reward per trial. Black is for the network realization in the main text, gray for additional realizations, i.e., trained with different random number generator seeds. (B) Percent …
Figure 2—figure supplement 2
Learning curves for the multisensory integration task.
(A) Average reward per trial. Black indicates the network realization shown in the main text, gray additional realizations, i.e., trained with different random number generator seeds. (B) Percent …
Figure 2—figure supplement 3
Learning curves for the parametric working memory task.
(A) Average reward per trial. Black indicates the network realization shown in the main text, gray additional realizations, i.e., trained with different random number generator seeds. (B) Percent …
Figure 3 with 1 supplement
Perceptual decision-making task with postdecision wagering, based on Kiani and Shadlen (2009).
(A) Task structure. On a random half of the trials, a sure option is presented during the delay period, and on these trials the network has the option of receiving a smaller (compared to correctly …
Figure 3—figure supplement 1
Learning curves for the postdecision wager task.
(A) Average reward per trial. Black indicates the network realization shown in the main text, gray additional realizations, i.e., trained with different random number generator seeds. (B) Percent …
Figure 4 with 3 supplements
Value-based economic choice task (Padoa-Schioppa and Assad, 2006).
(A) Choice pattern when the reward contingencies are indifferent for roughly 1 'juice' of A and 2 'juices' of B (upper) or 1 juice of A and 4 juices of B (lower). (B) Mean activity of example value …
Figure 4—figure supplement 1
Fit of cumulative Gaussian with parameters , to the choice pattern in Figure 4 (upper), and the deduced indifference point .
https://doi.org/10.7554/eLife.21492.017
Figure 4—figure supplement 2
Fit of cumulative Gaussian with parameters , to the choice pattern in Figure 4A (lower), and the deduced indifference point .
https://doi.org/10.7554/eLife.21492.018
Figure 4—figure supplement 3
Learning curves for the value-based economic choice task.
(A) Average reward per trial. Black indicates the network realization shown in the main text, gray additional realizations, i.e., trained with different random number generator seeds. (B) Percentage …
Tables
Table 1
Parameters for reward-based recurrent neural network training. Unless noted otherwise in the text, networks were trained and run with the parameters listed here.
| Parameter | Symbol | Default value |
|---|---|---|
| Learning rate | 0.004 | |
| Maximum gradient norm | 1 | |
| Size of decision/value network | 100 | |
| Connection probability (decision network) | 0.1 | |
| Connection probability (value network) | 1 | |
| Time step | 10 ms | |
| Unit time constant | 100 ms | |
| Recurrent noise | 0.01 | |
| Initial spectral radius for recurrent weights | 2 | |
| Number of trials per gradient update | # of task conditions |
Table 2
Psychophysical thresholds , , and obtained from fits of cumulative Gaussian functions to the psychometric curves in visual only, auditory only, and multisensory trials in the multisensory …
| 2.124 | 2.099 | 1.451 | 0.449 | 0.475 |
| 2.107 | 2.086 | 1.448 | 0.455 | 0.477 |
| 2.276 | 2.128 | 1.552 | 0.414 | 0.415 |
| 2.118 | 2.155 | 1.508 | 0.438 | 0.440 |
| 2.077 | 2.171 | 1.582 | 0.444 | 0.400 |
| 2.088 | 2.149 | 1.480 | 0.446 | 0.457 |
