The ability to detect a change in stimulus statistics improved in trials that provided more time before the change (‘change time’ in Figure 1A) for subjects to listen to the baseline statistics of the texture. Performance also increased monotonically to different asymptotic levels for the four tested change sizes (50, 80, 110, 140%, Figure 2A). Asymptotic performance depended on change size, with bigger changes in marginal probability leading to greater asymptotic performance especially between levels, from 50% to 95% (Figure 2A, p_size < 10⁻⁵, Friedman; p_time < 10⁻⁵, Friedman). Change size also influenced the dependence on change time, such that greater change sizes led to improved performance at shorter change times than for smaller change sizes (Figure 2A). This translates to a combined steepening and leftward shift of the performance curves with change size. The significance of this effect was assessed by fitting the performance curves for individual subjects with a parametric function of sigmoidal shape (an Erlang CDF, see Materials and methods) in order to extract the change size-dependent time constant. The characteristic time constant τ decreased significantly as a function of change size (Figure 2B; p<10⁻⁶, Kruskal-Wallis).

Figure 1

Download asset Open asset

Figure 2 with 3 supplements see all

Download asset Open asset

The observed performance could alternatively be explained by a timing strategy or a pattern recognition strategy. Both of these explanations can be rejected based on the data and the paradigm: if subjects had used a timing strategy, their instantaneous false alarm rate (as a function of change time, divided by the window length) should never reach a constant value. Instead, the false alarm rate exhibits an initial linear increase, followed by a constant false alarm rate per unit time (Figure 2D), a feature that was embodied in the behavior of the models (see Figures 7E/8F). Furthermore, the initial rising portion of the false alarm rate is a consequence of the dual estimation task design. The uniform regime of false alarm rate is consistent with the use of an exponential distribution of change times, which keeps the change occurrence probability constant per unit of time (see Figure 1B and Materials and methods).

Some subjects could have attempted to use a pattern recognition strategy, i.e. effectively ignoring the statistics of the first stimulus. However, based on the stimulus design, a pattern recognition strategy would have failed, since the first stimulus was drawn randomly for each trial, and the second was a stochastic modification of the first. Further, in this case, detection performance should not have depended on change time. All together these results are inconsistent with both a pattern recognition and a timing strategy.

Using the time-dependent false alarm rate, the sensitivity of the subjects to detect a change can be analyzed with a time-dependent d' (see Materials and methods for details of computing this d'). This analysis exhibited similar monotonically increasing shapes as a function of both change time and size (Figure 2C). Further, probability in a frequency bin was positively correlated with change detection (Figure 2—figure supplement 1), consistent with the idea that a high rate of samples provided a better estimate of the probability value in a frequency bin. We can rule out that only large probability bins were attended to, since the performance for equal size chances in large probability bins is dominated by the change in other, lower probability bins (Figure 2—figure supplement 2). Finally, longer stimulus duration in the current trial predicted a reduced performance in the following trial (Figure 2—figure supplement 3), suggesting that the converged estimate in the previous trial could ‘contaminate’ the estimation process in the subsequent trial. This is another indication that subjects were not using a pattern recognition strategy, as such a strategy completely ignores the statistics presented in the previous trial.

In summary, those findings indicate that change detection (i) improves with time allowed to sample the stimulus, (ii) improves with the size of the change and (iii) saturates with longer observation intervals. These properties are consistent with statistical decision-making, where a decision can only be made if the observed change in a stimulus property is substantial compared to the current uncertainty about the same property. Subjects using statistical decision making can (i) reduce their uncertainty by collecting more stimulus information over time, (ii) use larger differences in the stimulus property to overcome the uncertainty more rapidly, and (iii) will not be able to improve their performance once the estimation of the stimulus statistics has saturated.

The dependence of performance on change time suggests a dynamical mechanism performing an on-going estimation of the initial statistics. To gain insights into these dynamics, we examined the dependence of reaction times on the parameters of the change, especially its size, which intuitively correlates inversely with task difficulty according to Piéron’s law (Pins and Bonnet, 1996) and time of occurrence (or ‘change time’).

Reaction time distributions changed both in duration and shape as a function of change size (Figure 3A). Median reaction time decreased with larger change sizes (p<10⁻³; Kruskal-Wallis, Figure 3B), in accordance with the increase in performance with larger change sizes. Receiver operating curve (ROC)-based analysis indicated that the distributions of reaction times were different across change sizes and chance level (Figure 3—figure supplement 1; p<10⁻⁷; Friedman). More specifically we found a significant difference between the most difficult condition and chance level (p<10⁻⁵; Kruskal-Wallis), confirming that subjects were performing at all change sizes. This suggests that the time necessary to detect the deviation between the pre- and post-change stimulus statistics was reduced for larger change sizes.

Figure 3 with 1 supplement see all

Download asset Open asset

For shorter change times, reaction time distribution changed in a qualitatively similar manner as was observed for smaller change sizes, although the effect was less pronounced (Figure 3C). Median reaction times decreased with change times, mirroring dependence of performance on change times (p<10⁻⁵; Kruskal-Wallis, Figure 3D). This dependence can already be seen in the raw data (Figure 1C), where hit trials (black) for longer change times exhibited shorter reaction times. Again the timing of the first correct responses decreased correspondingly with longer change time, suggesting more accurate estimation of the initial statistics.

Changes in stimulus statistics are effectively a probabilistic redistribution of the stimulus energy in the spectrotemporal domain, here restricted to the spectral axis. Therefore, we hypothesized that a detection process acting in a spectrally localized manner should perform better when the total energy of the change is concentrated in a restricted frequency region. Indeed, we found that performance decreased for non-localized changes when compared with localized ones (Figure 4A). This effect was significant only when distances below and above eight semitones were grouped (Figure 4A; p<5.10⁻³). Finally, we found that performance did not vary as a function of relative position along the frequency axis (p=0.28; Figure 4B), contrary to the predictions of a recent study (Catz and Noreña, 2013) showing that the cortical representation at the extreme edges of the stimulus spectrum could be enhanced for sharp contrast, resulting in lower change detection thresholds.

Figure 4

Download asset Open asset

We collected neural responses using electroencephalography (EEG) in human subjects performing the above psychoacoustic task to study the relationship between behavioral performance and neural responses, and to narrow down the scalp regions whose neural response reflects the change in statistics. The analysis was focused on a subset of the recording electrodes, namely an auditory (central location, El.1; corresponding to the center in Nie et al., 2014) and a centro-parietal (14,27,28; corresponding to Twomey et al., 2015) set. Depicted potentials show averages across each set of electrodes. Subjects exhibited similar performance and reaction time dependencies on change time as in the psychophysical experiments (Figure 5—figure supplement 1). Change times were binned into four bins based on their distribution and Hit rate to equalize trials per bin.

At stimulus onset, the average auditory potential exhibited a classical, large and rapid event-related potential (ERP) (Figure 5A,C, composed of N1 and P2), followed by a negative sustained potential (indicated as NS in the figure) previously described for prolonged stimulus duration (Hari et al., 1980; Lammertmann and Lütkenhöner, 2001; Lütkenhöner et al., 2011). However, there was no systematic evidence for a response to the change in statistics (Figure 5B1, EEG of Hit trials aligned to change times). EEG aligned to subjects’ response time also did not show a significant response (Figure 5B2, EEG of Hit trials aligned to button-press, different colors indicate different change sizes, averaged over all change times, see below for differences in change time). This suggests that the detection of the change in statistics was not accompanied by an overall response in the auditory cortex comparable to other stimulus changes such as stimulus onset or offset (compare also to the model responses in Figure 8B, see also Discussion). While this does not preclude the information about the change to be available in early auditory cortex, there is no specific, overall reaction to the change, compared to the continuous representation of the stimulus.

Figure 5 with 2 supplements see all

Download asset Open asset

The centro-parietal electrodes exhibited a centro-parietal positivity (CPP) reported previously (O’Connell et al., 2012; Kelly and O'Connell, 2013; Twomey et al., 2015) in a similar location (see Figure 5F for its topography at response time). In contrast to the central electrodes, the CPP did not display any clear response to sound onset (Figure 5D) but exhibited a long-lasting response following change events (Figure 5E1). This increase in the EEG signal was building-up and preceded subjects’ responses across change sizes (Figure 5E2), outlasting the timing of the button press. The difference between change sizes was colocalized with the CPP (Figure 5F, inset), indicating that the difference in amplitude is not due to a global shift in potential). In previous studies, the CPP potential was clearly linked to evidence integration in decision making tasks, e.g. in simple visual and auditory detection tasks (O’Connell et al., 2012) and a complex visual discrimination task (Kelly and O'Connell, 2013). We therefore hypothesized the CPP to also be indicative of evidence integration in complex auditory detection tasks. In order to assess this, we examined how the CPP potential depended on the amount of evidence, and whether it exhibited accumulation-to-threshold dynamics.

Both the slope (Figure 5G) and the height (Figure 5K) of the response-aligned CPP potential depended on the stimulus parameters. The slope increased significantly with change size (Figure 5H, p<<0.001, 2-way ANOVA across change size and change time), but was not significantly dependent on change time (Figure 5I, p=0.074, same ANOVA). The effect of change size on slope is consistent with a representation of task-related evidence in the CPP signal, as reported previously in other change detection tasks (O’Connell et al., 2012).

The height of the potential also increased significantly with change size (Figure 5L, p<<0.001, 2-way ANOVA across change size and change time), and decreased significantly as a function of time (Figure 5M, p<<0.001, same ANOVA). Such a change size dependence has been reported before (see Figure 2 in O’Connell et al., 2012), and at first appears inconsistent with a fixed threshold. However, since the execution of the button press requires some time, the application of the threshold has to precede the button press by some delay. The observed difference in heights could thus reflect a continued accumulation of evidence at different slopes, during the time interval between decision commitment and response completion, until the execution of the decision is communicated to the CPP source. Consistent with this interpretation, CPP height did not exhibit a dependence on change size, if measured in a window of 200–100 ms preceding the response time (p=0.16, same ANOVA, close to the crossing in Figure 5E2). In addition, we verified that the CPP height did not depend on the reaction time (Figure 6), as expected from an evidence accumulator signal (Kelly and O'Connell, 2013).

Figure 6

Download asset Open asset

The height decrease as a function of change time is indicative of a reduction in threshold as a function of time (Figure 5M). However, we did not observe an increase in FA rate later in the trial (Figure 2D), suggesting no increase in unfounded decisions. Although the time-dependence of CPP height could result in a decrease of CPP height for late versus early reaction times, we did not find any significant decrease in CPP height for late reaction times, which may be due to a rather small effect-size (Figure 6B).

Finally, CPP responses aligned to false alarms exhibited similar slope and amplitude as the lower signal conditions (50%, 80%), however, were overall significantly lower than the overall signal conditions (p<<0.001, 1-way ANOVA, across change size). Neither slope nor height displayed a dependence as a function of time into the trial (Figure 5J/N, p=0.76, p=0.43, respectively, 1-way ANOVA, across different time-into-trial bins). Together these results suggest that the decision threshold on the CPP is close to the lowest change size / false alarm height.

Neither of these results depended on the detrending method, as verified by the alternative use of a classical high-pass filter (see Materials and methods and Figure 5—figure supplement 2).

In summary, we found central and centro-parietal electrodes to respond in a diametrically opposed manner to stimulus onset and (detection of) change in statistics. The CPP potential remained practically silent to stimulus onset, but reflected properties of the stimulus/task when aligned to button press. These results reinforce the notion that the CPP potential reflects sensory evidence accumulation and exhibits accumulation-to-threshold dynamics, with the possibility of continued integration until actual response execution. As a function of change time, only the CPP potential's height reduced, suggesting a time-dependent threshold.

The psychoacoustic results demonstrate that a listener’s ability to detect a change in a statistical property of the environment depends on the time available to estimate this parameter, both for the pre- and post-change stimulus. However, how does the listener know, when to start estimating the new statistics? Since - as in real life - the change occurs at an unexpectable time, one solution would be to compare the recent statistics to a longer term estimate of the same statistics, acting as a baseline - or 'null' - distribution. A minimal implementation of this solution consists of two processes estimating the same statistical property on different timescales (Figure 7).

Figure 7

Download asset Open asset

For this purpose, we turned to models of statistical estimation of the drift diffusion type, used previously to account for visual and auditory decision making in paradigms where subjects were asked to choose between two alternative choices (Britten et al., 1996; Brunton et al., 2013). In these models a dynamic variable compares the stimulus information in favor of the two alternatives, and when reaching a predefined bound, a decision is made. We extended this model to a pair of variables, estimating the statistical property on different timescales (Figure 7A–B and Materials and methods). A deviation is detected if the long-term estimate (Figure 7B, $P_{slow}$) and the short-term estimate ($P_{fast}$) differ by more than the difference between the thresholds (Figure 7B, $T$). As introduced above, this was intended to capture the dual task the participants faced in our paradigm, namely to estimate the base (initial) statistics while simultaneously scanning for deviations from these statistics. The modified model is governed by four parameters, which control the timescales of the dynamics variables and the threshold. To make the model applicable to our auditory textures, we assume that multiple copies of it operate in parallel in different frequency channels (see Materials and methods).

We presented an analogous stimulus to the model, exhibiting a change in the probability of tone occurrence at a random time (Figure 7A left and 7B, gray, only one frequency bin shown) and in a random frequency location, and quantified the model’s response in performance and response time. The model exhibited a comparable behavior on individual trials as humans (Figure 7C, compare to Figure 1C), with an initial hesitation to respond, and a mixture of false alarms (gray), correct response (black) and misses (not shown). We quantified the performance (performance, false alarms, misses) and the reaction times as a function of the change times and the change size (Figure 7D–G). The match between the human data and the model was close, with an average residual (mean squared error) of 0.049 (in units of probability). The correlation coefficients between the real data and the fit were [0.97,0.99,0.98], for performance (Figure 7D), false alarms (Figure 7E) and misses (Figure 7F), respectively.

The reaction times could be accounted for both in mean and distribution for different change sizes (r = 0.95, MSE = 0.009 (norm. prob.), Figure 7G). For the condition with the biggest step, a certain fraction of the responses occurred very early, which may be subject-dependent and we were unable to replicate in the present model.

The parameters that best fit the average human data were τ_f = 0.2 s, τ_s = 1.1 s, τ_a = 0.65 s, and $T$=0.40 (in units of probability). Hence, the time constants of the fast and the slow processes differed by more than fivefold, and the threshold for detecting a step was surprisingly high. The time for eliciting a motor signal was consistent with the asymptotic times we found in the human data (see Figures 3B and 140 %). The time constant of the transitional period represents (as the other parameters) an average over the subjects. Inspecting individual subjects revealed some variability in their propensity to react early (min median: 0.77 s; max median: 1.03 s).

The residual differences in the fit could be a consequence of the fact that the data from multiple listeners was pooled, rather than fitted individually. With the current limitation of ~1000 trials / listener, a single listener fit would be dominated by within-subject variability across trials, requiring more trials before stabilizing.

In summary, the dual timescale estimation model captures the human performance and reaction times well, suggesting that its basic principle may be implemented by the brain. The fitted timescales of estimation suggest that a rapid estimate of the present statistics can be formed within 200 ms. While this time appears sufficient to reliably distinguish the larger steps in statistics, it is insufficient to detect small changes in occurrence probability, which are often perceived as unchanged statistics.

In relation to the CPP's response properties, it is noteworthy that the decision variable in the dual-time scale model exhibits a similar, positive dependence between slope and change size (evident from Equations 2/3/3). If the evidence accumulation continued during the time interval between decision commitment and the actual motor execution, this would translate into a dependence of the height on the change-size as well. In agreement with the CPP dynamics, the slope should not depend on change time. Instead, the estimate of $P_{slow}$(t) should exhibit better convergence as a function of change time, leading to improved discrimination against $P_{fast}$(t). This non-decision period separating the crossing of the threshold and the actual response execution is implicitly incorporated in the model as the motor-related increment in reaction times.

The dual timescale model successfully captures human performance via an estimation of stimulus statistics. While this suggests a consistency with the principle of statistical estimation, it does not provide any insights into putative neural implementations. For this purpose, we turn to an established model of auditory cortical processing ('cortical model', Chi et al., 2005; Elhilali et al., 2009; Krishnan et al., 2014; Patil et al., 2012; Yang et al., 1992), which we augment here with a decision stage specific to the present task. In particular, this alternative model investigates whether the cortical model (and hence the primary auditory cortex) represents the acoustic stimulus in a way that supports an account of our psychoacoustic data, i.e. supports decision making in certain statistical contexts.

The cortical model emulates the spectrotemporal response properties of neurons in primary auditory cortex, which have been extensively studied by various groups (Ahrens et al., 2008; Eggermont, 2002; Kowalski et al., 1996). Its responses are based on a filterbank-based, joint spectrotemporal modulation analysis following the output of the early stages of the auditory system (auditory spectrogram, Figure 8A). Parameters and properties were set to approximate the responses of neurons in auditory cortex (see Materials and methods for details) (Chi et al., 2005; Yang et al., 1992). The spectrotemporal filters cover the experimentally observed range of 1–30 Hz and 0.5–8 cycle/oct, whose outputs are weighted in correspondence with the experimentally observed abundance of these properties in A1 (Kowalski et al., 1996), Figure 8B).

Figure 8

Download asset Open asset

We simulated two types of readouts from the model to account for two of the main experimental constraints. For the first, we summed all cortical outputs to simulate an effective EEG recording with limited spatial separation of sources, leading to a global response. As expected, in this case, trial onsets and offsets produced strong responses (Figure 8B), with a plateau of sustained response for the whole duration of the stimulus. The responses due to the statistical change in the stimulus were largely diluted in the summed response and thus could not been discerned, consistent with the present EEG recordings of the auditory electrodes (Figure 5B).

The ranges of spectral bandwidths and timescales related to the change were kept constant over the whole duration of the task. Consequently, a more optimal strategy would be to focus on the temporal modulation filters in cortex that are most activated by the statistical change. Hence, we postulated that high-order areas could monitor the outputs of the task-relevant temporal filters. For example, subjects could make their decisions based on the largest output produced by the change. These would be sampled from the filter with the temporal dynamics and spectral modulation that roughly matched those of the stimuli. The response thus selected is shown in Figure 8A. Aside from the strong responses at stimulus onset and offset, the responses now exhibited in addition a prominent intermediate peak due to the change in statistics (Figure 8A). This change-induced response peak vanished in trials when the pre-change interval was very short because it became fused with the large onset peak (Figure 8C). Change size was encoded in the slope of this cortical response (Figure 8D), consistent with the neural CPP response (Figure 5H). The variability in responses of the cortical outputs was solely due to the random tone-clouds preceding and following the change in the stimulus.

To quantitatively simulate the perceptual decisions of the listeners, we analyzed the cortical filter outputs for individual trials. In order to take into account the time-dependence of the CPP amplitude in the EEG recordings, we used a time-varying threshold that remained identical across all conditions (see Figure 8A and Materials and methods). The first peak exceeding this threshold (if any) was considered to be the decision point (purple arrow in Figure 8A). This readout mechanism was fitted to the performance and false alarm rate across change sizes and change times by allowing five free parameters (Figure 8E–F): the (bandwidth) scale Ω, the (temporal) rate ω, and the decision parameters (λ, a, b; see Materials and methods). The parameters that best fitted the human dataset were Ω = 0.54 cycle/oct, ω = 0.72 Hz (a rate corresponding approximately to dynamics or an integration time-constant of the order of 1–2 s), and a = 6.2, b = 10.8, and λ = 1.14 s (ρ = 0.95; p<5.10⁻¹⁶; MSE = 0.7%). The scale value corresponds to a full width at half-maximum for the scale filter of approximately 0.56 octave, very close to the frequency region spanned by localized changes (0.55 octave). This may indicate that subjects preferentially used a single scale value for monitoring the frequency modulation and that they estimated the most common frequency modulation across trials since half of the trials contained localized changes. Importantly, reaction times predicted by the model matched subject reaction times remarkably well both in distributional shape, mean and spread (Figure 8G), although the fitting procedure did not make use of this information (ρ = 0.90; p<5.10⁻¹³; MSE = 9.1%).

Scale filters integrate frequency modulations over a limited spectral bandwidth set by the scale factor Ω. Such scale filters are more prone to detect changes localized in the spectrotemporal modulation domain. Therefore spectrally distributed changes could be missed by the decision stage as they elicit less activity in the filter outputs. This is reminiscent of the observation that listeners detected changes more efficiently if their energy was concentrated in the frequency domain (Figure 4A). Consistent with this, we found a decrease in the model performance for non-localized changes, without fitting the model parameters to this aspect of the data (Figure 8H).

Thus, the model describes a physiological mechanism that accounts for the behavioral data, as well as suggesting an implementation for the basis of statistical estimation in neural terms. In relation to the neural responses, it provides an interpretation for the lack of change-related signal in the auditory EEG electrodes, and the decision signal's slope also scales with the change size (Figure 8D), i.e. the amount of evidence. Similar to the dual timescale model, the peak size of the decision signal increases with change size (Figure 8D), unless interrupted by the threshold. The reduction of the threshold over time (to compensate for the task designs) is consistent with the experimentally observed reduction in CPP size as a function of change time (Figure 5M).

The present experimental paradigm mimics the unexpected transformation of a sound source within a natural auditory environment. There are some relations to previous research on spectral representations of sound, e.g. profile analysis (Green, 1992; Green and Berg, 1991; Hartmann, 1986; Lentz and Richards, 1997; Neff and Green, 1987). Our work, however, differs in several relevant ways. In profile analysis, subjects detected spectral shape changes on static spectra that were presented in isolation for short times (fraction of a second each). By comparison, our stimuli were dynamic and sustained (multiple seconds), and changes were detected in the midst of a continuous background with an explicit measure of reaction times. This enabled us to explore the dynamic acquisition of the statistical information.

Further, a series of recent studies investigated detection of change occurring in first- and second-order sound statistics (Sohoglu and Chait, 2016; Barascud et al., 2016). In particular, these authors probed the detection of appearing or disappearing regular sound sources in an acoustic scene (Sohoglu and Chait, 2016). This type of changes featured modifications of first- and second-order sound statistics, which also included an increase in the overall sound level. In comparison, our stimulus design allowed us to limit the change to the first-order statistics while keeping the overall sound level constant.

Our experimental task offers a compromise between complexity of spectrotemporal structure versus tractability and interpretability of the changes. Furthermore, the task design and acoustic stimulus are well-suited for electrophysiological studies with behaving animals, where one can easily estimate neuronal receptive fields from the responses to tone clouds at the same time as the animal detects the changes (Ahrens et al., 2008; Wang et al., 2012).

Another important aspect of the experiments was their interleaved (as opposed to block-based) design for change sizes and other parameters, which had several consequences. For instance, it is likely that the observed performance underestimated optimal performance, since the time, location and size of changes were unexpected. This also prevented subjects from using a template-match strategy on the largest change size, and provided access to reaction times, which consistently mirrored performance, and perhaps the certainty of the subjects in their decisions (Kiani et al., 2014).

Following the modeling steps proposed by Marr (1982), we provided an algorithmic and an (neural) implementational model of our subjects’ behavior. The algorithmic approach implemented the principle of statistical estimation, while the neural model leveraged principles of auditory cortex processing. Although both models analyzed recent inputs, and effectively detected deviations from them, they differed fundamentally in their levels of description and abstraction.

The statistical estimation model implements the principle of statistical integration in a close-to minimal form, and provides a link to classical drift-diffusion models. It is a mechanistic, non-neural description of the process that performed statistical estimation in the classical sense, by representing and comparing the probability of stimuli in frequency bins, based on a lossy memory. Previous work has suggested a possible neural implementation of such a decision making process, in the form of competing neuronal populations, each corresponding to one alternative choice (reviewed in Insabato et al., 2014). While this approach can in principle be extended to the estimation of other properties of a stimulus distribution, i.e. moments or correlations, it has to be adapted more specifically to each particular task. In the present case we chose a static set of parameters, since the change time distribution remained unchanged in a session. More generally, (temporal) integration properties can adapt to the recent statistics, as recently shown in related contexts (Raviv et al., 2012; Ossmy et al., 2013).

The cortical model differs fundamentally in that it seeks to capture basic sensory neural responses and is inspired by physiological mechanisms. In this sense, it is agnostic to the type of stimulus, and can be readily extended to handle more complex scenarios such as changes in natural stimuli, speech and music. To create behavioral performance from its representation, we merely added a filter selection and a decision criterion. The spectrotemporal filters implemented in the cortical model exhibit alternating excitatory (positive) and inhibitory (negative) fields (Figure 8A) that compare the spectral stimulus properties over a given time window set by a filter's temporal rate. As such, it effectively integrates the recent input with opposing signs to detect a change, which can be compared to the difference between the fast and slow estimators in the statistical estimation model.

Therefore, we may view this model as approximating a neural implementation of the statistical model, and thus as a bridge to a neural interpretation of the behavioral performance and the EEG recordings. Several properties of human performance and of the neural data can be considered within each model’s framework. The most relevant are (i) reduced performance in detecting early changes, (ii) longer reaction times for early changes, and (iii) reduced height of the centro-parietal EEG responses for late changes.

In the cortical model, the reduced performance results primarily from the large onset response masking the responses to the smaller subsequent change, rendering the peak response less detectable (i). In order to simulate the instructions to the subjects not to report the stimulus onset as a change, the detection threshold was set to decrease from a larger initial value, which will delay responses for early changes (ii). Interestingly, this choice for the threshold is in line with the reducing CPP potential heights as a function of change times (iii). Overall, the integration time-constants in the cortical model on the order of 1 s (due to bandpass filters tuned at rates near ~1 Hz) appear sufficiently long to explain the decision dynamics exhibited by the subjects (Figure 2). These time-constants, while on the slow-end of the range, are still found in the primary and secondary auditory cortical regions (Kowalski et al., 1996; Liang et al., 2002).

In the statistical model, the reduced performance (i) is a consequence of the model’s design having two estimators: one with a fast and the other with an adaptive time-constant (τ_f and τ_s). At stimulus (trial) onset, the absence of prior evidence is reflected by the equality of the two time-constants. As the trial progresses, τ_s becomes longer, and the difference between the two estimator outputs increases to reflect the buildup of evidence for a change in stimulus statistics (see Materials and methods). the dynamics are a consequence of the time-constant dynamics (as above) as well as the not-yet converged estimate of the initial occurrence probability (ii). There is no correspondence for the observed decrease of the CPP potential as a function of change time (iii).

It has previously been proposed that subjects may be trading response speed for accuracy (Teichert et al., 2014). We think this may apply to the first period up to 1 s, where subjects responded very little. After this point accuracy quickly rose (Figure 2C), as did the false alarm rate (before reaching its plateau). The time controlling the divergence between the two estimators in the statistical model (τ_a = 0.65 s) roughly matches this time scale and may be accounting for an initial postponing of decision by the subjects. An alternative modelling strategy would include a dead-time, corresponding to the minimal time subjects take before responding. While this appeared unnecessary for the present data, such a model may become relevant if the paradigm includes blocks of different response window length, where subjects are forced to respond more quickly to perform successfully.

In summary, what is typically termed accumulation of evidence (and its associated performance and dynamics) could be explained by the dynamics of the onset response in the cortical model intertwined with its integration time-constants. Future experiments need to further test the validity of this neural interpretation, given the ubiquity of such ‘sudden’ events in natural stimuli due to saccades (in vision), attentional switches, or trial onsets, which could also influence the detectability of changes (as e.g. in change blindness, Levin and Simons, 1997; Rensink et al., 2000).

As discussed above, recognizing a change in the statistics of a complex spectrotemporal sound requires the extraction and accumulation of evidence from the stimulus to estimate decision-relevant properties. This transition from a stimulus-related to a task-related representation needs to occur along several stations of the auditory system. Our EEG recordings provide partial evidence regarding their putative location. Specifically, we found a clear difference in the representation of the stimulus at the central electrodes (estimated to originate from auditory cortex activity) and at the centro-parietal electrodes (estimated to reflect parietal activity): while the central electrodes exhibited a sharp onset response at stimulus onset (Figure 5A) and offset, they showed little evidence of the change response or of the presumed accumulation of evidence for a change (Figure 5B).

In sharp contrast, the centro-parietal electrodes displayed no response to the onset (Figure 5D), and a clear evidence of the sensory evidence accumulation after the change aligned to response (Figure 5E2). Previous studies using a linear increase of sensory evidence found a quadratic time progression of the centro-parietal potential. In the present task, the constant amount of evidence as function of time resulted in more linear dynamics of the centro-parietal potential, supporting the integration hypothesis. The task-irrelevant abrupt sensory event (i.e. the onset) was thus filtered out in the parietal EEG response while the task-relevant event (the change in statistics), although more subtle in nature, was selectively integrated and converted into a decision signal.

A set of related EEG studies termed the corresponding potential the centro-parietal positivity (CPP, Kelly and O'Connell, 2013; O’Connell et al., 2012). Consistently, we found the CPP slope and amplitude at the response time to be correlated with evidence level (change size). However, the amplitude was independent of reaction time, and did not depend on change size before the response time (200–100 ms). Additionally, the amplitude of the CPP was found to be inversely related with change times, as one would expect if performance would be influenced by the estimated, maximal trial duration. Such a reduction would be expected both as a consequence of the general task design, i.e. on the one hand, the requirement to not respond to the onset of the first texture, and on the other hand, the possibility to approximate the maximal trial duration. For practical reasons, arbitrary trial lengths are not realizable. Hence, subjects could form an expectation of the maximal sound duration, which means participants could be subject to an urgency signal that would lower their criterion in this time-range.

On the other hand, the texture onset may also play an important role, since the requirement to not respond to an otherwise salient change, may be regulated by a change in threshold (as modeled in the decision stage of the cortical model).

A decrease in decision threshold has been used in models of decision-making for dealing with speed-accuracy trade-offs (Bogacz et al., 2006) and observed in electrophysiological studies (Heitz and Schall, 2012). Although it could in theory explain the time-dependence we observed for CPP heights, subjects did not exhibit any urgency to respond, even after a few seconds, as exemplified by the constant FA rate per unit of time (Figure 2D). Instead, we propose that this decrease in threshold reflects a more sensitive criterion for change detection, via a more settled estimate of the initial statistics. The decision threshold would thus be dynamically adjusted during the course of a trial. Importantly, the lack of increase in FA rate suggests that the improved estimate of the initial statistics would also reduce the neural response to expectable deviations, such that the sensitivity (type I errors) stays at the same level.

This predominance of decision-related signals in the centro-parietal electrodes is consistent with decades of research in the accumulation of task-related visual information in the parietal cortex, more specifically in decision-making with saccades in the lateral intraparietal (LIP) cortex (Huk and Shadlen, 2005; Roitman and Shadlen, 2002; Shadlen and Newsome, 2001). Neurons in LIP have been shown to exhibit activity correlated with the accumulation of visual evidence coming from MT (Huk and Shadlen, 2005; Mazurek et al., 2003). Their firing rate usually exhibits a linear increase until the animal makes a decision (Shadlen and Newsome, 2001). In these studies, typically a fixed threshold on neural firing rate is used to relate neural activity to decision making.

It has recently been suggested that individual neurons change their firing rate instantaneously at the single trial level (Latimer et al., 2015). We presently observed gradual, rather than step-wise changes in our across-trial averages. However, we predict that even single trial EEG signals would be gradual as these step-changes occur randomly between neurons, and hence are unlikely to be synchronized at the population-level. Due to the large ensemble of neural responses contributing to a single scalp location’s potential, this instead results in the commonly seen ramping activity on the EEG level, as observed in our data.

The lack of evidence for a change-related signal in the auditory EEG potentials can, however, not fully rule out the presence of a change-related signal in auditory cortex in the present stimulus context. The representation of the change could be diverse and distributed, which may average out in the non-selective, coarse averaging on the EEG level (see Figure 8B). This is also consistent with recent work, demonstrating choice-related signal in auditory cortex (e.g. Bizley et al., 2013; Tsunada et al., 2016). Our cortical modelling suggests that the representation in auditory cortex provides a good substrate for initial accumulation of sensory information about changes in stimulus statistics, which is then selected and amplified in parietal cortex, leading up to the sustained parietal activity and a full representation of accumulated evidence and choice (Shadlen and Newsome, 2001). This interpretation is supported by the match in performance, reaction times (Figure 8E–G), and in the progression of activity between specific filters of the cortical model (Figure 8C) and neural data (Figure 5E).

In conclusion, as with many other cognitive functions, it is likely that higher-order areas such as the LIP and PFC select and potentially amplify task-relevant outputs of the auditory cortex. To test this hypothesis and the value of the proposed models, it will be necessary to extend change detection tasks to more natural and complex stimuli. As shown previously (Lewicki, 2002; Smith and Lewicki, 2006), natural statistics shape neural processing, and in a similar way should be informative about which changes to focus on in research. Furthermore, the models should be extended to include the effects of cognitive functions in modulating this process, such as attention or expectations.

In the main psychophysical study, 15 normal hearing subjects (mean age: 24.8y, 6 females) participated, 10 of which could be included for final analysis (see below for criteria). A different set of 18 subjects participated in the combined psychophysics and EEG experiment (mean age: 30 ± 10 years, 7 females), all of which could be included for final analysis (see below for criteria). All experiments were performed in accordance with the guidelines of the Helsinki Declaration. The Ethics Committees for Health Sciences at Université Paris Descartes approved the experimental procedures.

Acoustic Stimulation Subjects were seated in front of a screen with access to a response box in an acoustically-sealed booth (Industrial Acoustics Company GmbH). Acoustic stimulus presentation and behavioral control were performed using custom-written software in MATLAB (BAPHY, from the Neural Systems Laboratory, University of Maryland, College Park; available upon request). The acoustic stimulus was sampled at 100 kHz, and converted to an analog signal using an IO board (National Instruments, PCIe-6353) before being sent to diotic presentation using high-fidelity headphones (Sennheiser i380, calibrated flat, i.e. ±5 dB, within 100–20000 Hz). Reaction times were measured via a custom-built response box and collected by the same IO card sampled at 1 kHz.

Electroencephalogram (EEG) acquisition EEG recordings were collected in a separate set of 18 normal-hearing subjects while listening and responding to the texture change stimuli. Continuous EEG data were recorded using a 64-channel system (ActiCap, BrainProducts, Gilching, Germany) at a sampling rate of 500 Hz with one reference and one ground electrode. In order to standardize electrode placement on the skull, we used a default fabric head-cap that holds the electrodes (EasyCap, Equidistant layout, 60 scalp, four ocular channels). The analysis of EEG responses was carried out offline (see section Data analysis).

We investigated the conditions under which listeners could detect a change in the statistics of complex acoustic stimuli. More specifically, we wondered how subjects capture the percept of a spectrotemporally complex stimulus, and then use it as a background to detect changes relative to it. Concretely, in an experimental trial, a sound texture was presented, allowing the subjects a randomly varying period of time to form a percept of the stimulus (i.e. ‘estimate the baseline statistics’), and then a change in the frequency distribution of the tones was introduced (while maintaining the overall sound level). After the change, subjects had up to 2 s to indicate that they had detected it. The stimulus captures the central textural properties of complex spectrotemporal structure and statistical predictability. Both the stimulus design and the procedure are described in detail below.

The ability of the subjects to detect the change in stimulus statistics was quantified using two measures, performance and d-prime, denoted d'. These analysis (Figures 1–4) were performed on the data obtained during the psychophysics experiments, and restricted to the trials embedding localized changes unless stated in the text. In addition, reaction times dependences over stimulus parameters were analyzed.

We assume that subjects continuously estimate a wide range of statistical properties of the acoustic environment, and are able to detect unexpected deviations in these properties for the purpose of detecting changes in the ongoing sound. Among these properties are the probabilities of having a tone in the different frequency channels. Since these are the only determining properties in our stimulus design, we developed a phenomenological model, which estimates and detects changes in the marginal tone probabilities across multiple frequency channels (see next section for a more biologically motivated model based on a cortical filter-bank).

The model consists of change-detector modules, which operate independently on a limited spectral range and whose output is combined to enable change-detection on a full spectrum. For simplicity, the spectral division of the modules was matched to the presently relevant division of the psychophysical stimulus $S(t,f)$ (see above), i.e. we here consider four modules, one for each pair of frequency bins, whose marginal probability could change. Since the modules operate independently, frequency separation is not relevant in the present model (but see below in the cortical model). For the present model, these frequency bins are referred to as $S_i\left(t\right)$ (with i∈[1,4]), which contain a random set of tones, adhering to the same marginal probabilities as the psychophysical stimulus.

For each frequency bin $S_i\left(t\right)$, a pair of dynamical processes {$P_{slow}\left(t\right)$,$P_{fast}\left(t\right)$}_i, acts as a change detector. $P_{slow,i}$ estimates the long-term probability of the presence of a tone at a given time in $S_i\left(t\right)$, and $P_{fast,i}$estimates the more recent probability of the presence of a tone in $S_i\left(t\right)$. The dynamics of the processes are given by:

where ${\tau }_s$ > ${\tau }_f$, which separates the speed of the processes. Normally, $P_{fast,i}$ and $P_{slow,i}$ are going to have similar values, since $P_{fast,i}$ is simply tracking faster than $P_{slow,i}$. However, if a change in the probability of occurrence occurs in the stimulus, the difference between the two processes will grow, since $P_{fast,i}$ will react faster to this change. A change in the environmental statistics is hence detected, if $| P_{fast,i} -P_{slow,i} | > T$, where $T$ is a threshold and a free variable of the model. Identical models exist for different frequency channels $S_i\left(t\right)$. If $T$ is exceeded in a particular $S_i\left(t\right)$, this is considered as a detected change in the environment at the corresponding time $T_i$. Hence, only the first detected change in any $S_i$ is recorded as the response. The time of actual response is then given by $T = T_i+T_m$, where $T_m$ is a constant time equals to 250 ms to account for the non-integration related process, such as stimulus representation and motor execution, up to the button press (akin to the non-decision time, by Ratcliff and McKoon [2008]). The model is termed a dual timescale model.

If we use the model as described so far, it would - correctly - detect a change in statistics at the onset of the stimulus (transition from silence to stimulus). In the present task design, the subjects were instructed to ignore the change associated with the start of the stimulus, but only detect the change in statistics within the stimulus. As laid out in the introduction, two estimations needed to be performed simultaneously: (1) estimate the probability from stimulus onset, (2) compare this estimate to the changed probability in the latter part of the stimulus (which occurs at an unknown time). To account for this initial period of estimation, we change the dynamics of $P_{slow}$ (the slower tracking process) as a function of stimulus time. Intuitively this means that $P_{slow}$ and $P_{fast}$initially operate on the same timescales, and thus θ is never exceeded. The modified equations therefore become

The speed at which the tracking dynamics diverge is regulated by ${\tau }_a$. Overall, the model has four free parameters ($T$, ${\tau }_f$, ${\tau }_s$, ${\tau }_a$), which were matched to account for the experimentally collected data. The phenomenological model accounted for the dependence of performance on change time and change size. Given the numerators in (2) and (3), the slope of the both $P_{slow}$ and $P_{fast}$ and their difference will depend on change size (compare to the EEG data in Figure 5). Simulations were run at a sampling rate of 100 Hz. Fitting was performed by exhaustive search in the parameter space to avoid local minima and biasing by initial values.

The model structure is inspired by earlier accounts for decision-making in random-dot motion stimuli, i.e. so-called drift-diffusion models (Bogacz et al., 2006; Britten et al., 1996), which have also recently been used to account for acoustic click-rate comparison tasks (Brunton et al., 2013). In contrast to these models, the dynamical process $P_{slow}$ in our case becomes an estimate of the medium-term occurrence probability, and $P_{fast}$ an estimate of the recent occurrence probability, and a decision is made across the set of estimators (similar to Churchland et al., 2008) Note, that the processes can transiently exceed 1, however, on average the right hand side of the dynamical equations is zero, when the dynamical process equals the probability that Si is drawn from.

The cortical model is an approximation to the analysis performed up to primary auditory cortex, which has been used successfully in a range of different auditory projects. A full description of the model can be found in Chi et al. (2005) and Yang et al. (1992), but an outline of its basic principles is provided below.

If not specified otherwise, nonparametric tests were used. When data were normally distributed (for performance), we checked that statistical conclusions were the same. One-way analysis of variance was computed with the Kruskal-Wallis' test; two-way using Friedman's test. Error bars are ±2 SEM (standard error of the mean), unless specified otherwise. All statistical analysis was performed using Matlab (The Mathworks, Natick).

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

We would like to thank the Equipe Audition, the Donders Center for Cognitive Neuroscience, and the NSF-funded Neuromorphic Cognition Engineering Workshop in Telluride, CO, USA, for allowing us to use sound booths and EEG equipment, with helpful discussions. Funding was provided through the ERC ADAM project. ANR-10-LABX-0087 IEC and ANR-10-IDEX-0001–02 PSL* supported the research unit. SS was also supported by an ARO grant 63113-LS. BE was supported by a European Commission's Marie Curie Grant (660328).

Human subjects: All experiments were performed in accordance with the guidelines of the Helsinki Declaration. The Ethics Committees for Health Sciences at Université Paris Descartes approved the experimental procedures.

© 2017, Boubenec et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.