Behavioral task.

(A) The flexible visual-motion discrimination (FVMD) task. Monkeys needed to report their decision about the direction of the visual motion stimuli by choosing either the green or red saccade target. The appearance of the two color targets preceded that of the visual-motion stimulus, and the target positions were randomly chosen on each trial to avoid fixed mapping between motion stimulus and saccade direction. Monkeys could initiate their saccade as soon as they had made their decision. (B-C) Psychometric curves for the two monkeys. The performance accuracy for each monkey is plotted as the proportion of trials in which 315° was chosen as a function of the directions and coherence levels of the motion stimuli. (D-E) Chronometric curves are shown separately for the two monkeys. (F-G) Two trial conditions, contralateral (F) and ipsilateral (G), defined according to the spatial configurations of task stimuli during neural recording.

Nonlinear feedback modulation of saccade choice on sensory evaluation in LIP.

(A) The activity of one example neuron in the CT condition of the FVMD task is shown for each motion-coherence level. The zero-coherence trials were grouped based on the monkey’s choices. The two vertical dashed lines denote the time of target and motion stimulus onset, respectively. (B) The same example neuron’s activity in the IT condition of the FVMD task. (C-F) The activities of two more example neurons. (G-H) The averaged population activities of all direction-selective neurons (n = 83) that were collected during the recording sessions in which the saccade targets were arranged in either horizontal or oblique directions. The activity to each motion direction and coherence level is shown separately for the CT condition (G) and IT condition (H). (I) The activity differences between CT and IT conditions (CT minus IT) of single LIP neurons were plotted for both preferred and nonpreferred motion directions. Each dot represents the activity of a single neuron. The histograms in horizontal and vertical axis represent the distribution of activity difference between CT and IT conditions for preferred and nonpreferred motion directions, respectively. (J) An ROC analysis was used for quantifying the motion DS for both CT (solid) and IT (dashed) conditions. The colored dots denote the time points for which there was a significant difference between the CT and IT conditions (paired t-test: P < 0.01). (K) The average DS for low- and zero-coherence trials is shown as in (J). (L) Variance in LIP population activity as explained by the individual demixed principal components. Each bar shows the proportion of total explained variance that was contributed by the four task variables. The pie chart shows the total variance explained by each task variable. H, high; M, medium; L, low; 0, zero; P, preferred; NP nonpreferred.

Feedback modulation specifically impacted the decision-correlated activity.

(A-B) Averaged population activities on low-coherence trials in the CT condition (A) and IT condition (B) are shown separately for correct (corr) and error (err) trials. (C-D) An ROC analysis quantified the stimulus-related and decision-related LIP DS on low-coherence trials. The correlations between LIP neural activity and the monkeys’ decisions about motion direction (red) or the physical direction of the motion stimulus (blue) in both the CT (C) and IT (D) conditions are shown over time. (E-F) Partial correlation analysis revealed the decision-related and stimulus-related components of LIP activity. The values for r-decision (E) (the partial correlation between neuronal activity and monkeys’ choice, given the stimulus direction) and r-stimulus (F) (partial correlation between neuronal activity and stimulus direction, given the monkeys’ choices) were compared between the IT and CT conditions. (G-H) Correlation between LIP DS and the monkeys’ RTs on zero-coherence trials. The choice selectivity on zero-coherence trials is shown for the faster RT and slower RT trials. Shaded areas denote ±SEM. The black stars indicate time periods in which there was a significant difference (paired t-test: P < 0.01).

Multi-module RNNs trained with the FVMD task.

(A) Model schematic of the RNNs. Each RNN consists of nine motion direction tuned input units, eight color tuned target input units, 200 hidden units, and two response units. The hidden layer of each RNN consists of two main modules. Each main module consists of two nominal modules, each of which receives either the visual motion input (motion module) or the target color input (target module). Only the target modules project to the two response units, and each main module projects primarily to one response unit. All four nominal modules were assigned with equal number of units (25%) in the network, which consisted of 80% excitatory and 20% inhibitory units. (B-C) The performance accuracies (B) and RTs (C) of all 50 trained RNNs are shown separately for each motion coherence level. (D-E) Two example units from an example RNN. (D) The neural activity of an example unit from the motion module is shown for each motion direction and coherence level. (E) The neural activity of an example unit from the target module is shown for each saccade direction and coherence level. (F-G) The motion DS for the motion module (F) and saccade DS for the target module (G) for the example RNN. (H-I) The averaged population activities of all direction-selective units in the motion module of the example RNN are shown for the CT condition (H) and IT condition (I). (J-K) The averaged motion DS in the motion module of the example RNN for both the CT (solid) and IT (dashed) conditions was quantified by ROC analysis. (L-M) Partial correlation analysis. The values for r-decision (L) and r-stimulus (M) are compared between the IT and CT conditions for the example RNN. (N) A comparison of the motion DS in the motion module of all trained RNNs between the CT and IT conditions is shown for each coherence level. (O) A comparison of the r-decision and r-stimulus between the IT and CT conditions is shown for all trained RNNs. (Paired t-test: ***, P < 0.001).

The circuit mechanisms underlying the nonlinear feedback modulation in RNNs.

(A) The averaged cross-module connection weights are shown for both feedforward and feedback connections. Units in the motion module (M) and target module (T) were grouped based on their preferences for motion DS (315° vs. 135°) and target color selectivity (red [r] vs. green [g]), respectively. (B) The correlation between the match extent of the neural encoding between units in the motion module and those in target modules and the connection weights between them. Each dot denotes data from one RNN. (C-D) A comparison of the performance accuracy (C) and RT (D) of the full-model RNNs, RNNs without feedback connections, and RNNs with shuffled feedback connections. (E-F) The averaged activity of units in the motion module of the example RNN after the feedback connections were ablated is shown for CT (E) and IT (F) conditions. (G) The comparison of the averaged motion DS between CT and IT conditions for all trained RNNs after feedback connections were ablated. An ROC analysis was used to quantify the motion DS. (H) The differences in motion DS between CT and IT conditions were compared between full-model RNNs and RNNs with shuffled feedback connectivity.

The computational mechanisms underlying the nonlinear feedback modulation in decision-making.

(A) The averaged activity of units in the target module of the example RNN, after the feedback connections were ablated, is shown separately for different saccade directions and coherence levels. (B) The averaged saccade DS in the target module of the example RNN after full feedback ablation. (C) The averaged saccade DS in the target module of the full-model RNNs, RNNs without feedback connections, and RNNs with shuffled feedback connections. (Paired t-test: *, P < 0.05; **, P < 0.01; ***, P < 0.001; n.s., not significant). (D-G) The evolution of the averaged energy landscapes was shown over time. A numerical approximation of the energy landscape in the 1-D decision (saccade choice) subspace is constructed for both full-model RNNs and RNNs with various types of projection-specific inactivation. Each plot represents the averaged results of 50 RNNs, where Position 0 signifies the SVM decision boundary, and the vertical dashed line marks the time of motion stimulus onset. Unvisited portions of the state space are left blank as there is no gradient or potential estimate. (H-k) Averaged numerical estimate of energy landscapes for trials with different task difficulty levels (motion coherence). Results from two different saccade choices were average together. Only the potential values at positions continuously visited by four or more models were retained. Shaded areas denote ±SEM.