We re-examined data from a previous study in which 30 participants completed a food choice task (Bakkour et al., 2019). Prior to the main experiment, participants were asked to indicate their willingness to pay for each of 60 snack items on a scale from 0 to US$3 (Figure 1A). We refer to these explicitly reported values as s-values, or ${\displaystyle v_s}$ (where ${\displaystyle s}$ stands for ‘static’ as opposed to the ‘dynamic’ values we define below). In the main experiment (conducted in an MRI scanner), participants were shown pairs of images of previously rated snack items and had to choose which snack they would prefer to consume at the end of the study (Figure 1B).

Figure 1

Download asset Open asset

The data from Bakkour et al., 2019 replicate the behavior typically observed in such tasks. Both choice and response time were systematically related to the difference in s-value, ${\displaystyle \mathrm{\Delta }{v}_s}$, between the right and left items. Participants were more likely to choose the item to which they assigned a higher value during the rating phase (p<0.0001; ${\displaystyle {\mathcal{H}}_0:{\beta }_1=0}$; Equation 2). They were also more likely to respond faster when the absolute value of the difference between the items, ${\displaystyle {\displaystyle |\mathrm{\Delta }{v}_s|}}$, was greater (p<0.0001; ${\displaystyle {\mathcal{H}}_0:{\beta }_1=0}$; Equation 3).

The relationship between ${\displaystyle \mathrm{\Delta }{v}_s}$, choice, and response time is well described by a bounded evidence accumulation model (Krajbich et al., 2010; Bakkour et al., 2019). The solid lines in Figure 1C–D illustrate the fit of such a model in which the drift rate depends on ${\displaystyle \mathrm{\Delta }{v}_s}$. Overall, the behavior of our participants in the task is similar to that observed in other studies using the same task (e.g. Krajbich et al., 2010; Folke et al., 2016; Sepulveda et al., 2020).

An intriguing aspect of the decision process in the food choice task is its highly stochastic nature. This is evident from the shallowness of the choice function (Figure 1C): participants chose the item with a higher s-value in only 64% of the trials. This variability is typically attributed to unspecified noise when recalling item values from memory (e.g. Krajbich et al., 2010). An alternative explanation is rooted in constructive value theories, which suggest that the value of each item is constructed, not retrieved, during the decision process (Lichtenstein and Slovic, 2006; Shadlen and Shohamy, 2016; Johnson et al., 2007). This construction process is sensitive to the context in which it is elicited (e.g. the identity of items being compared), so the values reported during the valuation process may differ from those used in the choice task. According to this idea, the apparently stochastic choice is a veridical reflection of the constructed values.

If this were true, then the choice on any one cynosure trial—that is, the trial we are scrutinizing—would be better explained by values inferred from the choices on the other trials than by the s-values. We therefore compared two regression models that produce the log odds of the choice on each cynosure trial. The first regression model uses the s-values plus a potential bias for the left or right item. The second regression model includes one regression coefficient per item plus a left/right bias. It uses all the other trials (except repetitions of the identical pair of items) to establish the weights. While this model has more free parameters, the comparison is valid because we are using the models to predict the choices made on trials that were not used for model fitting. The better model is the one that produces larger log odds of the choice on the cynosure trial. As shown in Figure 2, the second regression model is superior.

Figure 2

Download asset Open asset

To ensure that this result is not produced artifactually from the algorithm, we performed the same analysis on simulated data. We fit the experimentally observed choices using a logistic regression model with ${\displaystyle \mathrm{\Delta }{v}_s}$ and an offset as independent variables, and simulated the choices by sampling from Bernoulli distributions with parameter, ${\displaystyle p}$, specified by the logistic function that best fit each participant’s choices (i.e., weighted-coin flips). We repeated the model comparison using the simulated choices and found that, contrary to what we observed in the experimental data, the model using explicit value reports is the better predictor (Figure 2, red).

Taken together, these analyses show that explicit value reports have limited power to predict choices, which partially explains their apparent stochasticity (Konovalov and Krajbich, 2019; Verhoef and Franses, 2003; Wardman, 1988). In the following sections, we elaborate on this observation. Not only do the values used to make the binary choices differ from the s-values, they drift apart during the experiment. We show that these changes arise through the deliberative process leading to the preference decisions themselves.

In the experiment, a subset of snack pairs were presented twice, in a random order within the sequence of trials. These trials allow us to assess whether preferences change over the course of a session. For these duplicated item pairs, we calculate the average number of times that the same item was chosen on both presentations—which we refer to as the match probability. Participants were more likely to select the same option when presentations of the same pair were closer in time (Figure 3). To assess the significance of this effect, we fit a logistic regression model using all pairs of trials with identical stimuli to predict the probability that the same item would be chosen on both occasions. The regression coefficient associated with the number of trials between repetitions was negative and highly significant (p<0.0001; t-test, Equation 8). It therefore follows that preferences are not fixed, not even over the course of a single experimental session.

Figure 3

Download asset Open asset

We propose a simple algorithm to characterize how preferences changed over the course of the session. It assumes that on each decision, the value of the chosen item increases by an amount equal to ${\displaystyle \delta }$, and the value of the unchosen item decreases by the same amount (Figure 4A). We refer to the updated values as d-values, or ${\displaystyle v_d}$, where ${\displaystyle d}$ stands for ‘dynamic’.

Figure 4

Download asset Open asset

Figure 4B illustrates how the value of the items changes over the course of the session, for a given value of ${\displaystyle \delta }$, for three snack items. For example, while the item shown with the green curve is initially very valuable, as indicated by its high initial rating, its value decreases over the course of the session each time it was not selected.

We determined the degree of revaluation that best explained the participants’ choices. For each participant, we find the value of ${\displaystyle \delta }$ that minimizes the deviance of a logistic regression model that uses the d-values to fit the choices made on each trial,

where ${\displaystyle p_{\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{i}\mathrm{c}\mathrm{e}}}$ is the probability of choosing the item that was presented on the right. The d-values are initialized to the explicitly reported values for all items, and they are updated by plus or minus ${\displaystyle \delta }$ when an item is chosen or rejected, respectively. Importantly, the updated values only affect future decisions involving the items.

Figure 4C shows the deviance of the logistic regression model for a representative participant, as a function of ${\displaystyle \delta }$. For this participant, the best explanation of the choices is obtained with a value of ${\displaystyle \delta \approx \mathrm{\$}0.15}$. We fit the value of ${\displaystyle \delta }$ independently for each participant to minimize the deviance of the logistic regression model fit to the choices. On average, each choice changed the value of the chosen and unchosen items by ${\displaystyle \mathrm{\$}0.18\pm 0.016}$ (mean ± s.e.m., (Figure 4C), inset).

The values derived from the Reval algorithm explain the choices better than the explicit value reports. The choices are more sensitive to variation in ${\displaystyle \mathrm{\Delta }{v}_d}$, evidenced by the steeper slope (Figure 5A). When ${\displaystyle \mathrm{\Delta }{v}_d}$ and ${\displaystyle \mathrm{\Delta }{v}_s}$ are allowed to compete for the same binomial variance, the former explains away the latter. This assertion is supported by a logistic regression model that incorporates both ${\displaystyle \mathrm{\Delta }{v}_s}$ and ${\displaystyle \mathrm{\Delta }{v}_d}$ as explanatory variables (Equation 7). The coefficient associated with ${\displaystyle \mathrm{\Delta }{v}_s}$ is not significantly different from zero while the one associated with ${\displaystyle \mathrm{\Delta }{v}_d}$ remains positive and highly significant (Figure 5—figure supplement 1).

Figure 5 with 3 supplements see all

Download asset Open asset

More surprisingly, Reval allows us to explain the response times better than the explicit value reports, even though RTs were not used to establish the d-values. We used the d-values to fit a drift-diffusion model to the single-trial choice and response time data, and compared this model with the one that was fit using the s-values (Figure 5A). To calculate the fraction of RT variance explained by each model, we subtracted from each trial’s RT the models’ expectation, conditional on ${\displaystyle \mathrm{\Delta }{v}_x}$ (with ${\displaystyle {\displaystyle x\in \{s,d\}}}$) and choice. The model that relies on the d-values explains a larger fraction of variance in RT than the model that relies on the s-values (Figure 5B). This indicates that the re-assignment of values following Reval improved the capacity of a DDM to explain the response times.

The DDM that uses the dynamic values also explains the combined choice-RT data better than the one that uses the static values. We compared their goodness of fit using the Bayesian Information Criteria (BIC), penalizing the DDM that uses the dynamic values for the revaluation update parameter, ${\displaystyle \delta }$. For all participants, the DDM that uses the dynamic values provided a better fit than the DDM that uses the static values (Figure 5—figure supplement 2A). To control for the possibility that the model comparison is biased by the extra parameter in the dynamic model (${\displaystyle \delta }$), we simulated choice and RT data for each participant from the DDM model fit to the static values, and fit these simulated data to the DDMs using static and dynamic values (in the latter case applying the Reval algorithm prior to fitting). For the simulated data, the model comparison favored the DDM using static values for most participants (Figure 5—figure supplement 2B), indicating that the additional parameter in the dynamic model does not strongly bias the model comparison.

The time it takes to make a decision, and the difference in value between the items under comparison, can be considered complementary measures of decision difficulty. On average, the more similar in value the two items are, the longer it would take to commit to a choice. Under this assumption, we can compare how well the static and the dynamic values predict the difficulty of the choices as judged by their response times. The application of Reval revealed that some decisions that were initially considered difficult, because ${\displaystyle \mathrm{\Delta }{v}_s}$ was small, were actually easy, because ${\displaystyle \mathrm{\Delta }{v}_d}$ was large, and vice versa. Grouping trials by the ${\displaystyle \mathrm{\Delta }{v}_d}$ led to a wider range of mean RTs compared to when we grouped them by ${\displaystyle \mathrm{\Delta }{v}_s}$ (Figure 5A). The effect can also be observed for individual participants (Figure 5C). For each participant, we grouped trials into two categories depending on whether the difference in value was less than or greater than the median difference. We then calculated the mean RT for each of the two groups of trials. The difference in RT between the two groups was greater when we grouped the trials using the d-values than when we used the s-values. This implies the d-values were better than the s-values at assessing the difficulty of a decision as reflected in the response time.

We verified that the improvement in fit was not just due to the additional free parameter (${\displaystyle \delta }$). To do this, we again used simulated choices sampled from logistic regression models fit to the participants’ choices, as we did for Figure 2. Because the choices are sampled from logistic functions fit to the choice data, they lead to a psychometric function that is similar to that obtained with the experimental data. We reasoned that if revaluation were an artifact of the analysis method, then applying the revaluation algorithm to these simulated data should lead to values of ${\displaystyle \delta }$ and goodness of fit similar to those of the real data. To the contrary, (i) the optimal values of ${\displaystyle \delta }$ for the simulated data were close to zero (Figure 6A); (ii) the reduction in deviance after applying Reval was negligible compared to the reduction in the actual data (Figure 6B); and (iii) we found no significant difference in the RT median splits between s-values and d-values (Figure 6C). This shows that the improvements in fit quality due to Reval are neither guaranteed nor an artifact of the procedure.

Figure 6

Download asset Open asset

The idea that a choice can induce a change in preference is certainly not new (Festinger, 1957). Choice-induced preference change (CIPC) has been documented using a free-choice paradigm (Brehm, 1956), whereby participants first rate several items, and then choose between pairs of items to which they have assigned the same rating, and finally rate the items again. A robust finding is that items that were chosen are given higher ratings and items that were not chosen are given lower ratings relative to pre-choice ratings, leading to the interpretation that the act of choosing changes the preferences for the items under comparison. However, it has been suggested that the CIPC demonstrated with the free-choice paradigm can be explained as an artifact (Chen and Risen, 2010). Put simply, the initial report of value may be a noisy rendering of the true latent value of the item. If two items, A and B, received the same rating but A was chosen over B, then it is likely that the true value for item A is greater than for item B, not because the act of choosing changes preferences, but because the choices are informative about the true values of the items, which are unchanging.

We examined whether Reval could be explained by the same artifact. We considered the possibility that the items’ valuation in the choice phase are static but potentially different from those reported in the ratings phase. If the values are static, but different from those explicitly reported, then Reval could still improve choice and RT predictions by revealing the true subjective value of the items.

We reasoned that if values were static, the improvements we observed in the logistic fits when we applied Reval should be the same regardless of how we ordered the trials before applying it. To test this, we applied Reval in the order in which the trials were presented in the experiment, and also in the reverse order (i.e. from the last trial to the first). If the values were static, then the quality of the fits should be statistically identical in both cases. In contrast, we observed that the variance explained by Reval was greater (i.e. the deviance was lower) when it was applied in the correct order than when it was applied in the opposite order (Figure 7A; p<0.0001, paired t-test). This rules out the possibility that the values were static. Moreover, the values produced by applying Reval in the reverse direction explained the choices better than the static values (Figure 7B). This might seem counterintuitive, given that the initial values for the Reval algorithm are the s-values, which are explicitly reported before the main experiment. In a later section, we show that this effect stems from the same process that gives rise to revaluation (Is revaluation a byproduct of deliberation?).

Figure 7

Download asset Open asset

So far we have assumed that a choice increases the value of the chosen option by ${\displaystyle \delta }$ and decreases the value of the unchosen option by the same amount. Here, we evaluate the possibility that the degree of revaluation is different for the chosen and unchosen options. We fit a variant of the Reval algorithm with two values of ${\displaystyle \delta }$, one for the chosen option (${\displaystyle {\delta }_{\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{n}}}$) and one for the unchosen option (${\displaystyle {\delta }_{\mathrm{u}\mathrm{n}\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{n}}}$). Figure 8 shows the values that best fit the data for each participant. For each participant, ${\displaystyle {\delta }_{\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{n}}>0}$ and ${\displaystyle {\delta }_{\mathrm{u}\mathrm{n}\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{n}}<0}$; in other words, the value of the chosen item typically increases, while the value of the unchosen item tends to decrease following a choice. Further, for most participants, the degree of revaluation is greater for the chosen option than for the unchosen option. As we speculate in the discussion, this result may be related to the unequal distribution of attention between the chosen and unchosen items (Krajbich et al., 2010).

Figure 8

Download asset Open asset

Several brain areas, in particular the ventromedial prefrontal cortex (vmPFC), have been shown to represent the value of decision alternatives during value-based decisions (Kennerley et al., 2009; Plassmann et al., 2007; Bartra et al., 2013). Based on our finding that the d-values provide a better explanation of the behavioral data than the s-values, we reasoned that the d-values might explain the BOLD activity in these areas beyond that explained by the s-values. We included both the s-value and the d-value of the chosen item in a whole-brain regression analysis of BOLD activity. This parameterization reveals significant correlation of the BOLD signal in the vmPFC with d-value, controlling for s-value (Figure 9 and Supplementary file 1). In fact, in a separate model that only included s-value, the effect of s-value on BOLD in the vmPFC did not survive correction for familywise error rate at a whole-brain level (Figure 9—figure supplement 1b and Supplementary file 2 top). In contrast, another model that only included d-value revealed a robust effect of d-value on BOLD in vmPFC that survived whole-brain correction (Figure 9—figure supplement 1 and Supplementary file 2 middle). Finally, to evaluate whether the effect shown in Figure 9 is not simply captured by the difference in d-value and s-value, we ran a fourth model that included only (d-value${\displaystyle -}$s-value). The effect of this difference between d-value and s-value on BOLD in vmPFC did not survive whole-brain correction (Figure 9—figure supplement 1 and Supplementary file 2 bottom). Collectively, these findings provide additional evidence for revaluation, as capturing a meaningful aspect of the data, in the sense that it accounts for the activity of brain areas known to reflect the value of the choice alternatives.

Figure 9 with 1 supplement see all

Download asset Open asset

To assess the generality of our behavioral results, we applied Reval to other publicly available datasets. All involve binary choices between food snacks, similar to Bakkour et al., 2019. We analyze data from experiments reported in Folke et al., 2016 and from the two value-based decision tasks reported in Sepulveda et al., 2020.

Reval yields results that are largely similar to those observed in the data from Bakkour et al., 2019. The values derived from Reval led to a better classification of choice difficulty than the explicit value reports (Figure 10A). In all three datasets, the ${\displaystyle \delta }$ values were significantly larger than those obtained from simulated data under the assumption that the values were static and equal to the explicitly reported values (Figure 10B). Furthermore, the reduction in the deviance resulting from the application of Reval (Equation 7) was significantly greater than the reduction observed in simulated data (Figure 10C). [All p-values, derived from two-tailed paired t-tests, are shown in the figure].

Figure 10

Download asset Open asset

In the dataset from Folke et al., 2016, the deviance was significantly smaller when Reval was applied in the forward than in the backward direction, replicating the result in our main experiment. However, in the dataset of Sepulveda et al., 2020, no significant difference in deviance was observed (Figure 10D). We do not know what explains this discrepancy, although we believe that the differences in experimental design may play a role. In the experiment of Sepulveda et al., 2020, unlike the other two datasets that we analyzed, participants performed the experiment in two framing conditions: one in which they chose the item the liked the most, and another one in which the chose the item they disliked the most. These two conditions alternated in short blocks of 40 trials. This alternation may affect valuation in a way that is not captured by the Reval algorithm. We expand on this in Discussion.

We hypothesize that the sequential dependencies we identified with Reval may be a corollary of the process by which values are constructed during deliberation. The subjective value of an item depends on the decision-maker’s mindset, which may change more slowly than the rate of trial presentations. Therefore, the subjective value of an item on a given trial may be informative about the value of the item the next time it is presented. Subjective values are not directly observable, but choices are informative about the items’ value.

We assessed the plausibility of this hypothesis with a bounded evidence accumulation model that includes a parameter that controls the correlation between successive evidence samples for a given item. We call this the correlated-evidence drift-diffusion model (ceDDM). We assume that the decision is resolved by accumulating evidence for and against the different alternatives until a decision threshold is crossed.

The model differs from standard drift-diffusion, where the momentary evidence is a sample drawn from a Normal distribution with expectation equal to ${\displaystyle \mathrm{\Delta }{v}_s}$ plus unbiased noise, ${\displaystyle \mathcal{N}(0,\sqrt{dt})}$. Instead, the value of each of the items evolves separately such that the expectations of its value updates are constructed as a Markov chain Monte Carlo (MCMC) process thereby introducing autocorrelation between successive samples of the unbiased noise (see Methods). Crucially, the correlation is not limited to the duration of a trial but extends across trials containing the same item. When an item is repeated in another trial, the process continues to evolve from its value at the time a decision was last made for or against the item.

We fit the model to the data from Bakkour et al., 2019. The model was able to capture the relationship between choice, response time and ${\displaystyle \mathrm{\Delta }{v}_s}$ (Figure 11A). Figure 11B shows the degree of correlation in the evidence stream as a function of time, for the model that best fit each participant’s data. After 1 s of evidence sampling, the correlation was 0.1062 ± 0.0113 (mean ± s.e.m. across participants). This is neither negligible (which would make the model equivalent to the DDM) nor very high (which would render sequential sampling useless, since it can only average out the noise that is not shared across time).

Figure 11

Download asset Open asset

The assumptions embodied by the ceDDM are consistent with the results of the Reval analysis. We applied the Reval algorithm to simulated data obtained from the best-fitting ceDDM. The results were in good agreement with the experimental data. The best-fitting ${\displaystyle \delta }$ values were positive for all participants and in a range similar to what we observed in the data (Figure 11C). Reval increased the range of RTs when trials were divided by difficulty, implying that Reval led to a better classification of easy and difficult decisions (Figure 11D). Reval applied to the trials in the true order explained the simulated choices better than when applied in the opposite direction (Figure 11E). This is because the model assumes that when an item first appears, the last sample obtained for that item was the value reported in the ratings phase for that item. As more samples are obtained for a given item, the correlation with the static values gradually decreases. Additionally, the values obtained from applying Reval in the backward direction provided a better explanation of the simulated choices than the static values (Figure 11F), mirroring the pattern observed observed in the behavioral data (Figure 7B). Taken together, the success of ceDDM implies that the sequential dependencies we identify with Reval may be the result of a value construction process necessary to make a preferential choice.

We identified sequential dependencies between choices in a value-based decision task. Participants performed a task in which they had to make a sequence of choices among a limited set of items. The best explanation for future choices was obtained by assuming that the subjective value of the chosen item increases and the value of the unchosen item decreases after each decision. Evidence for revaluation was obtained by analyzing the probability that participants make the same decision in pairs of trials with identical options. We also identified revaluation using an algorithm we call Reval. The same algorithm allowed us to identify revaluation in other datasets obtained with the food-choice task (Folke et al., 2016; Sepulveda et al., 2020).

The sequential effects we identified can be interpreted as a manifestation of choice-induced preference change. The usual paradigms for detecting the presence of CIPCs are based on the comparison of value ratings reported before and after a choice (for a review see Izuma and Murayama, 2013; Enisman et al., 2021). After a difficult decision, the rating of the chosen alternative often increases and that of the rejected alternative often decreases—an effect termed the ‘spreading of alternatives’. Many variants of the free choice paradigm have been developed to control for or eliminate the statistical artifact reported by Chen and Risen, 2010. One common approach is to compare the ‘spreading of alternatives” observed in the free-choice paradigm (rate-choose-rate, or RCR) with a control task in which a different set of participants rate the items twice before the choice phase (RRC). Any spread observed in the RRC condition cannot be explained by the CIPC, since in the RRC condition there is no choice between the two rating phases. The CIPC is measured indirectly, as the difference in the spread of the alternatives between the RCR and the RRC. Other approaches involve asking participants to rate an item that they are led to believe they have chosen, when in fact they have not (Sharot et al., 2010; Johansson et al., 2014). Any change in ratings cannot be due to the information provided by a choice, since no real choice was made. In addition to the complications introduced by deceiving the participants (e.g. participants may suspect the deception but not mention it to the experimenter), the elimination of a real choice prevents these paradigms from being used to study the process through which subjective values undergo revision during decision formation.

In contrast, our approach to identify changes in value does not require pre- and post-choice ratings. Instead, it requires a sequence of trials in which the same items are presented multiple times (as in Luettgau et al., 2020). The revaluation effect we find cannot be explained by the artifact identified by Chen and Risen, 2010. Using trials with identical items, we show that the nearer in time the trials with identical items are to each other, the more likely people are to choose the same option. Further, the revaluation algorithm explains choices better when applied in the order in which the trials were presented than when applied in the reverse order. These observations are inconsistent with the notion that item values are fixed (i.e. do not change) during the experiment, regardless of whether values are the same or different from those reported during the rating phase.

We cannot determine with certainty whether the revaluation occurs after the decision or during the deliberation process leading up to the decision. At face value, it might seem that Reval implements change after each decision (Festinger, 1957). Yet, Reval simply identifies a change in value, which may well occur during the deliberation leading to the decision, perhaps owing to a comparison of other items (on other trials) that happen to suggest a dimension of comparison that increases in importance on the current trial (Lee and Daunizeau, 2020; Lichtenstein and Slovic, 2006). More broadly, the subjective value of an option depends on the mindset of the decision maker. This internal state, which in the food-choice task includes aspects such as degree of satiety or sugar craving, can vary over time, causing the value of the items to vary as well. If changes in mindset are slow—that is, lasting longer than the duration of a decision—then the value of items will be correlated over time.

We proposed a decision model (ceDDM) in which evidence samples are correlated over time. Fitting the model to account for each participant’s choices and response times produces a revaluation of magnitude similar to what we observed experimentally. It also predicts that applying Reval in the direction in which the trials were presented explains the choices better than applying it in the opposite direction, as we observed in the data. This modeling exercise suggests that the CIPC-like effects we identified may be due to processes that occur during the deliberation leading up to a choice, rather than post-decision processes that attempt to reduce cognitive dissonance. To be clear, we interpret the ceDDM only as a proxy for a variety of more nuanced processes. If the mindset endures many individual decisions, the subjective value of an item will be correlated over time. While the ceDDM captures only a small aspect of this complex process, it has allowed us to explain the sequential dependencies we identified with Reval.

The ceDDM belongs to a class of sequential sampling models in which the drift rate varies over time. Such models have already been studied in the context of value-based decisions. For example, in the attentional drift-diffusion model (Krajbich et al., 2010), the drift rate varies depending on which item is attended, as if the value of the unattended items are discounted by a multiplicative factor. In Dynamic Field Theory (Busemeyer and Townsend, 1993), the drift rate varies depending on which attribute is attended. Recently, Lee and Pezzulo, 2022 showed that a sequential sampling model in which the drift rate varies over time can explain the ‘spreading of alternatives’ (SoA) characteristic of choice-induced preference change. Lee and Pezzulo, 2022 propose that the initial rating of the items may be constructed using only the most salient attributes of each item, while in a difficult decision more attributes may be considered, leading to a revaluation that informs the rating reported after the decision phase (see also Voigt et al., 2019). Consistent with our proposal, Lee and Pezzulo, 2022 argue that thinking about non-prominent features during decision-making increases the likelihood that these features will be recalled when evaluating options in subsequent instances.

We observed that the degree of revaluation was higher for the chosen item than for the unchosen item. This was revealed by a variant of the Reval algorithm in which we allowed both items to have different updates. We speculate that this difference can be explained by the asymmetric distribution of attention between the chosen and unchosen items. It is known that the chosen item is looked at longer than the unchosen item (Krajbich et al., 2010). Further, CIPC is more likely for items that are remembered to have been chosen or unchosen (Salti et al., 2014). So one possibility is that the revaluation is larger for the chosen than for the unchosen item because participants spent more time looking at the chosen item and thus are more likely to remember it, leading to a larger change in value (Voigt et al., 2019).

Another possibility derives from the constructive view of preferences and the potential role of attention in decision-making. It is often assumed that value-based decisions involve gathering evidence from different alternatives, and that more evidence is gathered from alternatives that are attended to for longer (Callaway et al., 2021; Li and Ma, 2021; Krajbich et al., 2010). In the ceDDM, the correlation in value for a given item decreases with the number of evidence samples collected from the item (Figure 11B). Therefore, the more that attention is focused on a given item, the greater the difference between the item’s value before and after the decision. Because chosen items are attended to for longer than unchosen items (e.g. Krajbich et al., 2010), the chosen item should exhibit larger revaluation than the unchosen one, which is what we observed in the data (Figure 8).

One limitation of our study is that we only examined tasks in which static values were elicited from explicit reports of the value of food items. It remains to be determined if other ways of eliciting subjective values (e.g. Jensen and Miller, 2010) would lead to similar results. We think so, as the analysis of trials with identical item pairs (Figure 3) and the difference between forward and backward Reval (Figure 7A) are inconsistent with the notion that values are static, regardless of their precise value. It also remains to be determined if our results will generalize to non-food items whose value is less sensitive to satiety and other dynamic bodily states. Perceptual decisions also exhibit sequential dependencies, and it remains to be explored whether these can be explained as a process of value construction, similar to what we propose here for the food-choice task (Gupta et al., 2024; Cho et al., 2002; Zylberberg et al., 2018; Abrahamyan et al., 2016).

Another limitation of our study is that, in one of the datasets we analyzed (Sepulveda et al., 2020), applying Reval in the forward direction was no better than applying it in the backward direction (Figure 10). We speculate that this failure is related to idiosyncrasies of the experimental design, in particular, the use of alternating blocks of trials with different instructions (select preferred vs. select non-preferred). More importantly, Reval applied in the backward direction led to a significant reduction in deviance relative to that obtained using the static values (Figure 7B). This reduction was also observed in the ceDDM, suggesting that the effect may be explained by changes in valuation during deliberation. However, we cannot discard a contribution from other, non-dynamic changes in valuation between the rating and choice phase including contextual effects (Lichtenstein and Slovic, 2006), stochastic variability in explicit value reporting (Polanía et al., 2019), and the limited range of numerical scales used to report value.

Finally, we emphasize that the ceDDM should be interpreted as a proof-of-principle model used to illustrate how stochastic fluctuations in item desirability can explain many of our results. We chose to model value changes following an MCMC process. However, other stochastic processes or other ways of introducing sequential dependencies (e.g. variability in the starting point of evidence accumulation) may also explain the behavioral observations. Furthermore, there likely are other ways to induce changes in the value of items other than through past decisions. For example, attentional manipulations or other experiences (e.g. actual food consumption) may change one’s preference for an item. The current version of the ceDDM does not allow for these influences on value, but we see no fundamental limitation to incorporating them in future instantiations of the model.

Our research contributes to a growing body of work exploring the impact of memory on decision-making and preference formation (Biderman et al., 2020), and in particular to the CIPC. It has been suggested that the retrieval of an item’s value during decision-making renders it susceptible to modification, leading to a revaluation that influences subsequent valuations through a process that has a neural correlate in the hippocampus (Luettgau et al., 2020). The link between memorability and preference is also supported by experiments in which the presentation of an item coincides with an unrelated rapid motor response that increases subsequent preference for the item (Botvinik-Nezer et al., 2021) and by experiments demonstrating that people prefer items to which they have previously been exposed (Zajonc, 1968). As in these studies, ours also highlights the role of memory in revaluation. Due to the associative nature of memory, successive evidence samples are likely to be dependent (Rhodes and Turvey, 2007). A compelling illustration of this effect was provided by Elias Costa and colleagues (Costa et al., 2009). Participants were asked to report the first word that came to mind when presented with a word generated by another participant, which was then shown to yet another participant. The resulting chain resembled Lévy flights in semantic space, characterized by mostly short transitions to nearby words and occasional large jumps. Similar dynamic processes have been used to describe eye movements during visual search (Bella-Fernández et al., 2022) and the movement of animals during reward foraging (Brown et al., 2007; Hills et al., 2015). It is intriguing to consider that a similar process may describe how decision-makers search their memory for evidence that bears on a decision.

A total of 30 participants completed the snack task, which consisted of a rating and a choice phase. The experimental procedures were approved by the Institutional Review Board (IRB) at Columbia University, and participants provided signed informed consent before participating in the study. The data were previously published in Bakkour et al., 2019.

Association between the s-values, choice and RT. We used the following logistic regression model to evaluate the association between the s-values and the probability of choosing the item on the right:

where ${\displaystyle I_{\mathrm{i}}}$ is an indicator variable that takes the value 1 if the trial was completed by subject ${\displaystyle i}$ and 0 otherwise. We used a t-test to evaluate the hypothesis that the corresponding regression coefficient is zero, using the standard error of the estimated regression coefficient.

Similarly, we used a linear regression model to test the influence of ${\displaystyle \mathrm{\Delta }{v}_s}$ on response times:

where ${\displaystyle |\cdot |}$ denotes absolute value and ${\displaystyle \mathrm{\Sigma }{v}_s}$ is the sum of the value of the two items presented on each trial. The last term was included to account for the potential influence of value sum on response time (Smith and Krajbich, 2019).

We fit the choice and RT data with a drift-diffusion model. It assumes that the decision variable, ${\displaystyle x}$, is given by the accumulation of signal and noise, where the signal is a function of the difference in value between the items, ${\displaystyle \mathrm{\Delta }v}$, and the noise is equal to ${\displaystyle \sqrt{dt}}$, where ${\displaystyle dt}$ is the time step, such that the accumulated noise after 1 s of unbounded accumulation, the variance of the accumulated noise is equal to 1. The decision variable follows the difference equation,

where ${\displaystyle {\eta }_t}$ is sampled from a normal distribution with a mean 0 and variance 1, ${\displaystyle \kappa }$ is a signal-noise parameter, μ is the drift rate and ${\displaystyle {\mu }_0}$ is a bias coefficient that is included to account for potential asymmetries between right and left choices.

We assume that the drift rate is a (potentially nonlinear) function of ${\displaystyle {\displaystyle \mathrm{\Delta }v}}$. We parameterize this relationship as a power law, so that

where ${\displaystyle \mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}}$ is the sign operation, ${\displaystyle ||}$ indicates absolute value, and ${\displaystyle \gamma }$ is a fit parameter.

The decision terminates when the accumulated evidence reaches an upper bound, signaling a rightward choice, or a lower bound, signaling a leftward choice. The bound is assumed to collapse over time. It is constant until time ${\displaystyle d}$, and then it collapses at rate ${\displaystyle a}$:

Collapsing bounds are needed to explain why choices that are consistent with the value ratings are usually faster than inconsistent choices for the same ${\displaystyle {\displaystyle \mathrm{\Delta }v}}$.

The response time is the sum of the the decision time, given by the time taken by the diffusing particle to reach of the bounds, and a non-decision time which is assumed to be normally distributed with mean ${\displaystyle {\mu }_{nd}}$ and standard deviation ${\displaystyle {\sigma }_{nd}}$.

The model has 8 parameters: ${\displaystyle \{\kappa ,B_0,a,d,\gamma ,{\mu }_0,{\mu }_{nd},{\sigma }_{nd}\}}$. The standard deviation of the non-decision times (${\displaystyle {\sigma }_{nd}}$) was fixed to ${\displaystyle 0.05}$ s. For the fits shown in Figures 1C–D –5A, we fit the model to grouped data from all participants. For the analysis of variance explained (Figure 5) and model comparison (Figure 5—figure supplement 2), we fit the model separately for each participant. The model was fit to maximize the log of the likelihood of the parameters given the single-trial choice and RT:

We evaluate the likelihood by numerically solving the Fokker-Planck (FP) equation that described the dynamics of the drift-diffusion process, using the Chang-Cooper fully-implicit method (Chang and Cooper, 1970; Kiani and Shadlen, 2009; Zylberberg et al., 2016). For computational considerations, we bin the values of ${\displaystyle {\displaystyle \mathrm{\Delta }v}}$ to multiples of $0.1. From the numerical solution of the FP equation, we obtain the distribution of decision times, which is convolved with the truncated Gaussian distribution of non-decision latencies. The truncation ensures that the non-decision times are non-negative, which could otherwise occur during the optimization process for large values of ${\displaystyle {\sigma }_{nd}}$. The parameter search was performed using the Bayesian Adaptive Direct Search (BADS) algorithm (Acerbi and Ma, 2017).

The Reval algorithm was applied to each participant independently. The values are initialized to those reported during the ratings phase. They are then revised, based on the outcome of each trial, in the order of the experiment.

The value of the chosen item is increased by ${\displaystyle \delta }$ and the value of the unchosen item is decreased by the same amount. The revaluation affects future decisions in which the same item is presented.

We searched for the value of ${\displaystyle {\delta }^{\ast }}$ that minimizes the deviance of the logistic regression model specified by Equation 1. The model’s deviance is given by:

where the sum is over trials and ${\displaystyle \widehat{c_i}}$ is the probability assigned to the choice on trial ${\displaystyle i}$ obtained from the best-fitting logistic regression model.

We complemented this iterative algorithm with a second approach that estimates ${\displaystyle {\delta }^{\ast }}$ using the history of choices preceding each trial. Nearly identical ${\displaystyle \delta }$ values are derived using a single logistic regression model in which the binary choice made on each trial depends on the number of times each of the two items was selected and rejected on previous trials. The model is:

where, as before, ${\displaystyle I_i}$ is an indicator variable that takes a value of 1 if the trial was completed by subject ${\displaystyle i}$ and 0 otherwise. The key variable is ${\displaystyle {\mathrm{\Delta }}_{ch}}$. It depends on the number of past trials in which the item presented on the right in the current trial was chosen (${\displaystyle n_{\mathrm{c}\mathrm{h}}^{\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}}}$) and not chosen (${\displaystyle n_{\mathrm{\neg }\mathrm{c}\mathrm{h}}^{\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}}}$), and similarly, the number of past trials in which the item presented on the left in the current trial was chosen (${\displaystyle n_{\mathrm{c}\mathrm{h}}^{\mathrm{l}\mathrm{e}\mathrm{f}\mathrm{t}}}$) and not chosen (${\displaystyle n_{\mathrm{\neg }\mathrm{c}\mathrm{h}}^{\mathrm{l}\mathrm{e}\mathrm{f}\mathrm{t}}}$):

The variable ${\displaystyle {\mathrm{\Delta }}_{ch}}$ represents the influence of past choices. The signs in Equation 15 are such that a positive (negative) value of ${\displaystyle {\mathrm{\Delta }}_{ch}}$ indicates a bias toward the right (left) item. To obtain the ${\displaystyle {\delta }^{\ast }}$ in units equivalent to those derived with Reval, we need to divide the regression coefficient ${\displaystyle {\beta }_{2,i}}$ by the sensitivity coefficient ${\displaystyle {\beta }_{1,i}}$, separately for each subject ${\displaystyle i}$. As can be seen in Figure 5—figure supplement 3, the values obtained with this method are almost identical to those obtained with the Reval algorithm.

The model assumes that at each moment during the decision-making process, the decision-maker can only access a noisy sample of the value of each item. These samples are normally distributed, with parameters such that their unbounded accumulation over one second is also normally distributed with a mean equal to ${\displaystyle \kappa v_s}$, where ${\displaystyle v_s}$ is the explicit value reported during the Ratings phase and ${\displaystyle \kappa }$ is a measure of signal-to-noise, and a standard deviation equal to 1.

Crucially, for each item, the noise in successive samples is correlated. To generate the correlated samples, we sample from a Markov chain using the Metropolis-Hastings algorithm (Chib and Greenberg, 1995). The target distribution is the normally distributed value function described in the previous paragraph. The proposal density is also normally distributed. Its width determines the degree of correlation between consecutive samples. Typically, the correlation between successive samples is considered a limitation of the Metropolis-Hastings algorithm. Here, however, it allows us to generate correlated samples from a target distribution. The standard deviation of the proposal density is ${\displaystyle \sqrt{dt}/\tau }$. Higher values of ${\displaystyle \tau }$ result in a narrower proposal density, hence more strongly correlated samples. We sample from the same Markov chain across different trials in which the same item is presented, so that the last sample obtained about an item in a given trial is the initial state of the Markov chain the next time the item is presented.

At each moment (${\displaystyle dt=40ms}$), we sample one value for the left item and another for the right item, compute their difference (right minus left), and accumulate this difference until it crosses a threshold at ${\displaystyle +B_0}$, signaling a rightward choice, or at ${\displaystyle -B_0}$, signaling a leftward choice. The decision time is added to the non-decision time, ${\displaystyle {\mu }_{nd}}$, to obtain the response time.

We fit the model to the data as follows. For each item, we simulate many Markov chains. In each trial, ${\displaystyle i}$, we take samples from each chain until the accumulation of these samples reaches one of the two decision thresholds. Then we calculate the likelihood (${\displaystyle L}$) of obtaining the choice and the RT displayed by the participant on that trial as:

where ${\displaystyle N=1,000}$ is the number of Markov chains, ${\displaystyle {\displaystyle {\mathbb{1}}_{i,j}}}$ is an indicator function that takes the value 1 if the choice made on chain ${\displaystyle j}$ is the same as the choice made by the participant on trial ${\displaystyle i}$ and 0 otherwise, ${\displaystyle \mathcal{N}(x|y,z)}$ is the normal probability density function with mean ${\displaystyle y}$ and standard deviation ${\displaystyle z}$ evaluated at ${\displaystyle x}$, and ${\displaystyle {\sigma }_{nd}}$ is a parameter fit to the data.

When an item is presented again in a future trial, the initial state of each Markov chain depends on the state it was in the last time the item was presented. The initial state of each chain is obtained by sampling 1000 values (one per chain) from the distribution given by the final state of each chain. The sampling is weighted by the value of ${\displaystyle L_j}$ of each chain (Equation 16), so that chains that better explained the choice and RT in the last trial are more likely to be sampled from in future trials.

The model has 5 parameters per participant: ${\displaystyle \{\kappa ,B_0,\tau ,{\mu }_{nd},{\sigma }_{nd}\}}$, which were fit to maximize the sum, across trials, of the log of ${\displaystyle L}$ using BADS (Acerbi and Ma, 2017).

The correlations in Figure 11B were generated using the best-fitting parameters for each participant to simulate 100,000 Markov chains. We generate Markov chain samples independently for the left and right items over a 1 s period. To illustrate noise correlations, the simulations assume that the static value of both the left and right items is zero. We then calculate the difference in dynamic value (${\displaystyle x}$) between the left and right items at each time (${\displaystyle t}$) and for each of the Markov chains (${\displaystyle i}$). Pearson’s correlation is computed between these differences at time zero, ${\displaystyle x_i(t=0)}$, and at time ${\displaystyle x_i(t=\tau )}$, for different time lags ${\displaystyle \tau }$. Correlations were calculated independently for each participant. Each trace in Figure 11B represents a different participant.

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

We thank Ari Pakman for helpful discussions.

This work was supported by the National Institutes of Health (R01NS113113 to MNS and MH121093 to DS), the Air Force Office of Scientific Research under award (FA9550-22-1-0337 to MNS), the Howard Hughes Medical Institute (MNS), The McKnight Foundation Memory and Cognitive Disorders Award (DS), and the National Science Foundation (1822619 to DS and 1606916 to AB).

1. Preprint posted: January 15, 2024
2. Sent for peer review: April 2, 2024
3. Reviewed Preprint version 1: May 23, 2024
4. Reviewed Preprint version 2: November 18, 2024
5. Version of Record published: December 10, 2024

You can cite all versions using the DOI https://doi.org/10.7554/eLife.96997. This DOI represents all versions, and will always resolve to the latest one.

© 2024, Zylberberg et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.