1 Introduction

1.1 Hierarchical Bayesian models of perception and learning

Bayesian perspectives on perception have proposed that our brain inverts a hierarchical generative model to infer the causes of its sensory inputs and predict future events (Dayan et al., 1995; Rao & Ballard, 1999; Friston, 2010; Doya et al., 2011). Under this “Bayesian brain” view, perception corresponds to integrating expectations (prior beliefs about hidden states of the world) with incoming sensory information to yield a posterior belief, while learning refers to the updating of beliefs about the model’s parameters, which takes place more slowly, as experience accumulates. Formally, beliefs are modelled as probability distributions, such that the width of the distribution reflects the uncertainty (inverse precision) associated with that belief. Humans have been shown to take uncertainty into account when combining different sources of information in a manner that conforms to the statistical optimum as prescribed by Bayes’ rule (Ernst & Banks, 2002; Angelaki et al., 2009).

Because sensory signals are generated by interacting causes in the external environment that span multiple spatial and temporal scales, the brain is assumed to reflect this hierarchy of causes in a correspondingly hierarchical generative model. The Bayesian inversion of this generative model results in a hierarchy of beliefs, where higher levels encode beliefs about increasingly abstract, general, and stable features of the environment. These higher-level beliefs serve as priors for the inference on lower levels. Specifically, at each level of the hierarchy, belief updates serve to reconcile predictions (priors) from higher levels with the actual input (likelihood) from lower levels. Furthermore, under fairly general assumptions (i.e., for all probability distributions from the exponential family, Mathys, 2016; Mathys & Weber, 2020), these belief updates rest on (precision-weighted) prediction errors (PEs), i.e., the (weighted) mismatch between the model’s predictions and the actual input (Friston, 2010).

Popular hierarchical Bayesian models of perception and learning that are built on these ideas are predictive coding (Rao & Ballard, 1999; Friston, 2005) and hierarchical Gaussian filtering (Mathys et al., 2011, 2014a). In these models, estimates of uncertainty are central: the impact of prediction errors on belief updates depends on a precision ratio, which relates the precision of the prior to that of the observation, thus scaling the relative impact that new information has on belief updates. Put simply, mismatches (PEs) elicit stronger belief updates if the prediction about the input (likelihood) is precise, relative to the belief in the current estimate (prior). This form of adaptive scaling, a key element of healthy inference and learning, has also been proposed to lie at the heart of perceptual disturbances observed in mental disorders. For example, an imbalance between the influence of expectations and sensory inputs has featured prominently in attempts to explain the emergence of positive symptoms in schizophrenia, such as hallucinations and delusions (Stephan et al., 2006; Fletcher & Frith, 2009; Corlett et al., 2009, 2011; Adams et al., 2013; Friston et al., 2016; Sterzer et al., 2018; Corlett et al., 2019).

The HGF (we use “the HGF” as a shorthand for “hierarchical Gaussian filtering models of various configurations”) has been particularly useful in this context, as it can be fit to participants’ empirically observed behaviour or physiology, and thereby used to infer individual trajectories of precision-weighted PEs and predictions from data. By formulating a response model that links trial-wise perceptual quantities (such as predictions and PEs) to measured quantities (such as choices, reaction times, eye movements, evoked response amplitude in EEG, etc.), the HGF can quantify individual differences in inference and learning in terms of model parameters that encode prior beliefs about higher-order structure in the environment (de Berker et al., 2016; Lawson et al., 2017; Powers et al., 2017; Siegel et al., 2020; Siegel et al., 2020; Sevgi et al., 2020; Henco et al., 2020; Rossi-Goldthorpe et al., 2021; Suthaharan et al., 2021; Kafadar et al., 2022; Sapey-Triomphe et al., 2022; Fromm et al., 2023; Drusko et al., 2023). Such a mechanistic characterization of inter-subject variability is of particular interest for fields like Computational Psychiatry, because such differences may explain the heterogeneous nature of psychiatric diseases, and form a basis for mapping them out in a continuous conceptual space or dividing them into more homogeneous subgroups (Stephan & Mathys, 2014; Mathys, 2016).

1.2 Volatility-coupling, value-coupling, and noise-coupling

The type of belief hierarchy modelled by any particular approach depends on the nature of the generative model. The HGF assumes a particular form of generative model, where hidden states of the world evolve as coupled random walks in time. In current HGF models, the mean (value) of the higher level determines the variance (step size) of the lower level’s random walk. In other words, higher levels encode the volatility (or inverse stability) of lower levels (we will call this volatility coupling). This is motivated by the observation that learners must take into account different sources of uncertainty in their belief updates, one of which is the current level of stability in the world: if the world is currently changing (volatile), the agent needs to learn faster. Accordingly, in the HGF, subjective estimates of increased environmental volatility directly influence the uncertainty associated with lower level beliefs, leading to faster belief updates on the lower level (note that the word subjective does not imply conscious accessibility here - it only differentiates the inferred quantities from the “true“ values as produced by the environment or the generative model). Previous work has shown that human learners indeed adjust their learning rate according to experimentally manipulated levels of volatility (Behrens et al., 2007).

By contrast, predictive coding models typically focus on hierarchies in which higher levels predict the value of lower levels. This is implemented as the mean of the probability distribution that represents the lower-level belief, or in other words, the expectation of the value. We will refer to these hierarchies as implementing value coupling. This type of hierarchy is useful for understanding how beliefs about lower-level features depend on higher-level beliefs - for example, how the perceived brightness of a patch in an image depends on the context (objects, shadows) in which that patch is presented (Rao & Ballard, 1999; Adelson, 2005). Modern predictive coding neural networks are powerful machine learning tools (Millidge et al., 2022a; Song et al., 2020; Whittington & Bogacz, 2017). While value coupling - the type of hierarchical coupling in predictive coding - is also often used in theoretical treatments of hierarchical Bayesian modelling, the HGF, in contrast, offers a flexibly applicable implementation of hierarchical beliefs connected by volatility coupling that is being widely used for empirical data analysis.

In a noteworthy exception to the typical value hierarchies in predictive coding, Kanai et al. (2015) presented a model where higher levels encode the (spatial) precision of lower levels in a static environment. This is different from volatility coupling, where higher levels are concerned with the rate of change on lower levels, but captures a second source of uncertainty in beliefs about hidden states: the level of noise or reliability of the sensory input. Relatedly, in the learning and decision-making literature, two classes of models separately deal with the estimation of process noise (volatility, Behrens et al., 2007; Mathys et al., 2011; Piray & Daw, 2020) and observation noise (stochasticity, Lee et al., 2020; Nassar et al., 2010), with recent attempts to capture both sources of uncertainty in a joint model (Piray & Daw, 2021). Importantly, both volatility and stochasticity estimation can be captured as higher-level belief nodes coupled to the precision of lower-level beliefs.

Here, we show that the HGF can be extended to encompass all of these architectures: precision coupling between hidden states (volatility coupling) and at the observation stage (noise coupling) and, importantly, linear and nonlinear forms of value coupling. Our extension thus links HGF with predictive coding and also includes a principled way of modelling inference on observation noise.

The generalised HGF represents a new class of artificial neural networks that unifies computational principles of predictive coding and HGF. Practically, it allows for modular construction of ANNs with complex hierarchical structure, and it provides a flexible tool for empirical data analysis in computational psychiatry via implementation in open software. These theoretical and practical advances will be unpacked in the following sections.

2 Theory: Introducing value coupling

The generative model for value coupling

The HGF assumes that an agent is trying to infer on (and learn about) a continuous uncertain quantity x in their environment, which moves (changes) over time. Without any information about the specific form of movement, a generic way of describing movement of a continuous quantity is a Gaussian random walk:

In the original formulation (Mathys et al., 2011, 2014a), coupling between environmental states at different hierarchical levels was implemented in the form of volatility coupling, where the step size ϑ (or rate of change/volatility) of a state x_a varies a function of a higher-level state X_ǎ:

with parameters κ_ǎ,a (scaling the impact of volatility parent x_ǎ on x_a) and ω_a (capturing the “tonic” step size or volatility, which does not vary with time). By simultaneously inferring on the state x_a and its rate of change x_ǎ, the agent can learn faster (slower) in times when x_a is changing more (less). We call x_ǎ a volatility parent of x_a.

In contrast, here, we consider the case where a higher-level state x_b influences the value (mean) of the lower-level state x_a:

where

is the coupling function with parameter α_b,a (scaling the impact of value parent x_b on x_a), the constant drift parameter ρ_a, and the function g. The state’s mean now evolves as a function of its previous value , and a function of state . Crucially, as we will show below, the HGF can deal with both linear and nonlinear transformation functions g, as long as the function g is twice continuously differentiable almost everywhere. States x_b and x_a interact via value coupling.

The effect of a higher-level state on a lower-level state with this kind of coupling can be understood as a phasic drift signal (see Figure 1 for illustration). The drift parameter ρ determines the “tonic” drift, equivalent to the tonic volatility parameter w. We can use the same configuration also to capture situations where the higher-level state represents a mean value to which the lower-level state reverts back to over time. This is achieved simply by inserting parameter λ_a ∈ [0,1] that encodes the state’s auto-connection strength:

Figure 1.

When λ_a < 1, the state x_a will change as an autoregressive process, reverting back to the total mean M_a, which is given by:

and the value of λ will determine the speed with which it does so. In other words, the meaning of ρ and x_b for the evolution of x_a depends on λ. The difference between the cases λ = 1 and λ < 1 is illustrated in Figure 1. Finally, if λ = 0, the state does not perform a Gaussian random walk around its own previous mean anymore, instead, its values are normally distributed around a constant or the value of a parent state (reminiscent of an observation of the parent with Gaussian noise).

Importantly, different forms of coupling can be present at the same time: A state x_a can have both a volatility parent x_ǎ (generating changes in its rate of change) and a value parent x_b (generating changes in its mean value). It can also have a drift parameter ρ_a which is a constant influencing its mean - equivalent to its tonic volatility parameter ω_a which determines its step size in the absence of a phasic volatility influence. Finally, we allow for inputs to arrive at irregular intervals; therefore, we multiply the total variance of the random walk and the total mean drift by the time that has passed since the arrival of the previous sensory input at index k − 1 (Mathys et al., 2014a). Together with suitably chosen priors on parameters and initial states (see Mathys et al., 2014a), the following equation forms the generative model for a state x_a with both volatility and value parent:

In the even more general case, a state could have multiple value parents and multiple volatility parents, each affecting the mean value and rate of change of state x_a in proportion to their respective coupling strengths (we are here only considering the additive effect of multiple parents on a given state, but more sophisticated interactions are conceivable):

Because a given state can also be parent to multiple child states at the same time, these extensions allow us to model fairly complex networks of interacting states of the world. In Figure 2 we have drawn an example setup with 11 different environmental states and two outcomes. For this example, and together with priors on parameters and initial states (see Mathys et al., 2014a), the following equations describe the generative model (for simplicity, the example uses linear value coupling, no tonic drifts (ρ = 0, λ = 1), and inputs at regular intervals, i.e., t^(k) = 1 ∀ k ):

Figure 2.

Note that this example also includes binary states (x_b) and observable outcomes (u_a and u_b). Our main discussion will focus on continuous states performing Gaussian random walks, but we will briefly touch on other types of states (binary, categorical, input) in section 6.3.

Using this example network, we introduce two general motifs. First, all states that are value parents of other states (or outcomes) by default have their own volatility parent (and volatility parents therefore share the index with their child node, for example, states x_a and x_ǎ). Even if in practice, many environmental states might have constant volatility, from the perspective of the agent, it makes sense to a-priori allow for phasic changes in volatility. From a modelling perspective, these volatility parents could be removed in scenarios with constant volatility.

Second, states that are volatility parents to other states can either have a value parent (as states x_ǎ and ), or no parents (as states x_č, x_ě and ). This is because in practice, volatility parents of volatility parents are rarely required. Instead, we suggest that value parents of volatility states are more useful. In particular, these can be used to model “global” or shared volatility states that affect multiple lower-level volatility beliefs (such as state x_f in this example, which influences volatility beliefs about states x_a and x_d), separately from “local” volatility states that only affect speed of change in a single lower-level state (such as the volatility states x_č, x_ě and ).

In this section, we have introduced value coupling to the generative model of the HGF, for the first time considering the case where the mean of x_a is a function not only of its own previous value but also (some transformation of) the current value of some higher-level state x_b, scaled by a coupling parameter α_b,a We will now show how an agent can infer on the values of such hidden states.

The belief update equations for value coupling

An agent employing a generative model of the kind described above to do perceptual inference holds a belief about the current value of each of the states (i.e., every x) of this model at every time point k. We describe this belief about state x at time k as a Gaussian distribution, fully characterized by its mean μ(^k) and its inverse variance, or precision, π(^k) at time k.

In the approximate inversion of the generative model for volatility coupling, Mathys et al. (2011) derived a set of single-step update equations that represent the approximately Bayes-optimal changes in these beliefs in response to incoming stimuli. Repeating this derivation for the case of value coupling similarly leads us to simple one-step equations for updating beliefs about states that serve as value parents (for the full derivation of these equations, cf. Appendix 6.1). Assuming state x_b is a value parent to state x_a with a coupling strength α_b,a, then the new posterior belief about state x_b after observing a new input at time step k is given by:

where and refer to the prediction about state x_b before seeing the new input, and is the prediction error about the child state x_a.

Note that these equations also hold when introducing the parameter λ from above. This parameter will only feature in the computation of the prediction of state x_a and thus affect the prediction error .

In the case of linear value coupling (g_b,a(x) = Ax + B), the update further simplifies, as (a factor that we can absorb into α_b,a) and :

As in the case of volatility coupling, the belief updates are thus driven by precision-weighted prediction errors about the lower belief state in the hierarchy.

To get an intuition for these update equations in the case of nonlinear value coupling, let us consider an example where state x_a acts as a rectified linear unit (ReLU), that is, we choose the function g as the rectifier function, a popular activation function for deep neural networks (Hahnloser et al., 2000; Lecun et al., 2015):

The first and second derivative of g are then

and

respectively. For our purposes, we can treat the second derivative as

Plugging this into equation 23, we get

and

In other words, the impact of lower-level prediction errors on the posterior belief at the higher level and depends on the previous state of the higher-level node, such that beliefs only change in response to inputs if they were above zero (or “active”) in the first place.

Similarly, we can construct the reverse coupling function to model saturation:

Here, state x_b only exerts an influence on the lower-level state x_a, if its own value is below a threshold ν. In the inference on state x_b, this means that the belief about the parent node, mu_b, will stop being updated after crossing this threshold ν (i.e., after becoming saturated):

and

This cessation of updates after crossing a threshold does not mean that a node will remain stuck in place for all future time points. The thresholding only affects the HGF’s update step while the prediction step keeps affecting and , eventually possibly moving back to a region where updates take place again.

These examples demonstrate that our extension of the HGF allows us to build deep networks with non-linear coupling functions useful for both artificial and physiological neural networks.

3 Belief updates in the HGF: A network of nodes

Just like we can model complex networks of interacting states in the environment using the generative model of the HGF (Figure 2), we can also think of the inference process of an agent in that environment as a network of interdependent beliefs. The agent entertains a belief about each of the relevant environmental states, and updates these beliefs based on new sensory inputs. Because the agent models its world as a set of hierarchically interacting states, its beliefs about these states will form a hierarchy as well (Figure 3). Before new inputs arrive, higher-level beliefs inform predictions about lower-level beliefs (top-down information flow), whereas after the arrival of a new piece of information, changes in lower-level beliefs trigger updates of higher-level beliefs (bottom-up information flow). Previous work using the HGF has tended to depict the generative model (Figure 3A), which then defines the inference model or set of belief update equations. However, the part of the HGF that represents a model of cognition - the evolution of hierarchically interacting beliefs and the relevant flow of information through this hierarchy - is contained in the inference model (or belief update equations, Figure 3B).

Figure 3.

We conceptualize each belief modelled by the HGF as a node in a network, where belief updates involve computations within nodes as well as message passing between nodes. The specific within-node computations and messages passed between nodes will depend on the nature of the coupling. Putting the equations for value and volatility coupling side by side discloses a modular network architecture. This has important consequences for the implementation of the inference model (both in a computer and in a brain), which we summarize in Figure 3B. We will mainly focus on continuous HGF nodes here - nodes which represent beliefs about continuous quantities that evolve in time as Gaussian random walks. Subsequently, we will consider other types of nodes in HGF models, including categorical nodes and input nodes.

We start by noting that the computations of any node within a time step can be subdivided into three steps, an update step, where a new posterior belief is computed based on a prediction and an incoming input or prediction error (PE), a PE step, where the difference between expectation (prediction) and new posterior is computed for further message passing upwards, and a prediction step, where the new posterior is used to predict the value at the next time step. These can be ordered in time as shown in the box:

Node a at time step k

• (re) compute

← receive from child node

UPDATE step

• compute

• given: and

→ send to child node

PE step

• compute

• given: and

→ send to parent node

← receive from parent node

PREDICTION step

• compute

• given: and .

Two things are worth noting here. First of all, the PE step is a computation that the node performs in service of its parents. From the perspective of the parent node b, the represents an expectation of the child node’s state, and the corresponds to the actual state of this child at time step k. The difference between the two amounts to the prediction error which will serve to update the parent node - in other words, the parent’s PE.

Second, we have placed the prediction step at the end of a time step. This is because usually, we think about the beginning of a time step as starting with receiving a new input, and of a prediction as being present before that input is received. However, in some cases the prediction also depends on the time that has passed in between time steps (e.g., when considering drifts), which is something that can only be evaluated once the new input arrives - hence the additional computation of the (current) prediction at the beginning of the time step. Conceptually, it makes the most sense to think of the prediction as happening continuously between time steps. It is however implementationally more convenient only to compute the prediction once the new input (and with it its arrival time) enters. This ensures both that the posterior means of parent nodes have had enough time to be sent back to their children for preparation for the new input, and that the arrival time of the new input can be taken into account appropriately.

A node of the above kind is the first computational sub-unit in our perceptual model, and it can be connected to other nodes via volatility or value coupling depending on the underlying generative model. For node a, another node b can function as a parent node if the two are connected and node b represents a belief about a higher-level quantity which affects the belief about node a according to the generative model. On the other hand, if node b refers to a lower-level quantity and is connected to node a, it serves as a child node for node a.

Computations of nodes in the HGF

In the following, we examine the exact computations at time step k within a node for each of the three steps introduced above. We will compare the relevant computations between volatility and value coupling and identify the messages that have to be sent and received in each step. Since the update step relies on quantities computed in the (previous) prediction step, we start with the computation of the predictions for the current time step (which, as explained above, we think of as being computed prior to the arrival of a new input). We will first only consider the case of linear value coupling (alongside volatility coupling), and then separately examine any differences in these computations for the case of nonlinear value coupling. The results of this analysis are summarized in Figures 3–5.

Figure 4.

Figure 5.

The prediction step

In the prediction step, node a prepares for receiving the new input. This entails computing a new prediction, based on the previously updated posterior beliefs. In general, the prediction of a new mean (or the new mean of the predictive distribution) will depend on whether node a has any value parents, whereas the precision of the new prediction will be influenced by the presence and posterior mean of the node’s volatility parents.

In the general case, the mean and precision of the new prediction are computed as follows:

where is the total predicted drift of the mean and is the total predicted volatility (or step size). In the case of value coupling, the relative contribution of the node’s own previous value and the predicted drift is determined by the parameter λ (see Section 2). Both the total predicted drift as well as the volatility are computed as a sum of a constant (or tonic) term, given by a model parameter, and a time-varying (or phasic) term which is driven by their parents:

The total predicted mean drift equals the sum of constant term ρ_a (the tonic drift parameter of node a) and the sum of posterior means of all value parents (where parents are indexed by i) at k − 1, weighted by their connection strengths :

where t^(k) denotes the time that has passed between k — 1 and k, and N_vapa is the number of value parents. Similarly, the total predicted volatility is a function of a constant term ω_a (the tonic volatility parameter) and the posterior means of all volatility parents at the previous time point k — 1, weighted by their connection strengths :

If node a does not have any parents, both the predicted drift P_a and the predicted volatility Ω_a are fully determined by constant parameters (p_a and ω_a) and the time between subsequent observations. In HGFs without drift (ρ_a = 0), the predicted mean for the next time step is equal to the posterior mean of the current time step. Equations 34 to 37 nicely reflect the roles that value parents and volatility parents play in the generative model, where value parents model a phasic influence on a child node’s mean, and volatility parents model a phasic influence on a child node’s step size or volatility.

In sum, the prediction step for node a only depends on knowing its own posterior belief from the previous time step and having received its parents’ posteriors in time before the new input arrives. The implied message passing for this computational step is visualized in the left panel of Figure 4 for value coupling, and Figure 5 for volatility coupling.

The update step

The update step consists of computing a new posterior belief, i.e., a new mean μ^(k) and a new precision π^k given a new input from the level (node) below (usually, a prediction error δ), and the node’s own prediction ( and ). In this case, the exact computations within a node depend on the nature of its children: If node b is the value parent of node a, then the following update equations apply to node b:

Thus, at the time of the update, node i needs to have access to the following quantities:

• Its own prediction: ,

• Coupling strength: α_b,a

• From level below: ,

All of these are available at the time of the update. Node b therefore only needs to receive the PE and the precision of the prediction from the child nodes to perform its update. The middle panel of Figure 4 illustrates these computations.

Note that from the equations above, we can define another quantity, the precision-weighted prediction error (pwPE):

This quantity summarizes the size of the belief update in node b due to changes in node a (before accounting for connection strength) and is often of interest in experimental investigations of neural correlates of prediction errors in belief updating (Iglesias et al., 2013a; Diaconescu et al., 2020; Weber et al., 2020, 2022).

For a node ǎ which is the volatility parent of node a, the update equations for computing a new posterior mean and a new posterior precision have been described by Mathys et al. (2011). Here, we will introduce two changes to the notation to simplify the equations themselves and their implementation:

First, we will express the volatility PE, or VOPE, as a function of the previously defined value PE, or VAPE. That means from now on, we will use the symbol δ only for VAPEs:

We use the symbol Δ for VOPEs, which we define as

For a derivation of this definition based on the equations given in Mathys et al. (2011), cf.Appendix 6.2.

Second, we will introduce another quantity, which reflects the volatility-weighted orecision of the urediction:

which will be computed as part of the prediction step and will be termed effective precision of the prediction owing to its role in the update equations. This definition serves to simplify the equations and the corresponding message passing.

With these two definitions, namely those of the VOPE Δ^(k) and of the effective precision of the prediction γ^(k), the update equations for the precision and the mean of volatility parent ǎ simplify to:

This means that at the time of the update, volatility parent node ǎ needs to have access to the following quantities:

• Its own prediction: ,

• Coupling strength: κ_{ǎ, a}

• From level below: ,

These equations are illustrated in the middle panel of Figure 5. We note the structural similarities between nodes that serve as value parents and nodes that serve as volatility parents: updates of the mean are always driven by precision-weighted prediction errors, and updates of the precision require some estimate of the prediction of the precision of the child node ( or γ_a). These similarities allow us to make statements about the message passing architecture within and across nodes that generalize across coupling types (see Figure 3B).

An interesting difference to the implied message passing in predictive coding proposals (Bastos et al., 2012; Shipp, 2016) arises from the update step: The HGF architecture requires that not only (precision-weighted) prediction errors are being sent bottom-up between nodes, but also estimates of prediction precision ( or γ_a) which serve to update belief precision in the higher-level node.

The prediction error step

Finally, in the PE step, a node computes the deviation of its recently updated posterior from its time step-specific prediction. This can result in two different types of PEs: VAPEs and VOPEs. These will, in turn, be used to communicate with the node’s parent nodes, if it has any. Therefore, this step again depends on the nature of a node’s parent nodes and can also be considered as the process of gathering all the information required by any existing parents. In addition to the PE, parent nodes will require some estimate of the precision of the prediction (see the previous section on the update step).

If node a is the value child of node b, the following quantities have to be sent up to node b:

Precision of the prediction:

Prediction error:

Node a has already performed the prediction step (see above), so it has already computed the precision of the prediction for the current time step, . Hence, in the PE step, it needs to perform only the following calculation (illustrated in the right panel of Figure 4):

Note that represents a prediction error from the perspective of the parent node - the difference between the expected state of the child and the actual state at time step k. From the perspective of the child node a, the difference between its prior and its posterior instead represents a belief update (Bayesian surprise).

If node a is the volatility child of node ǎ, the following quantities have to be sent up to node ǎ (see also necessary information from level below in a volatility parent’s update step):

Effective precision of the prediction:

Prediction error:

Node a has already performed the prediction step at the previous time step, so it has already computed the precision of the prediction, , and the total predicted volatility, , and out of these the effective precision of the prediction, , for the current time step. Hence, in the PE step, it needs to perform only the following calculations (illustrated in the right panel of Figure 5):

In other words, if node a has any parents, the VAPE will always be computed(as it features in both scenarios), whereas the computation of a VOPE is only necessary if node a has a volatility parent.

Our framework allows for multiple parent nodes of either type (e.g., more than one volatility parent). However, it is important to note that all parent nodes of the same type will be sent the same bottom-up prediction errors by their child node. The only difference between the parent nodes is in their coupling strength with the child node. Thus, their relative contribution to the belief about the child node is determined by their starting value and coupling strength. This reflects the fact that the learning agent does not have direct access to these latent states and only learns about them through the information from the sensory inputs (or lower-level beliefs).

As soon as the agent has access to some other source of information about these states, they can be decoupled. For example, if two parent nodes share a child node, but one of them is additionally linked to another child node, the agent can form independent beliefs about the two parent nodes given the different bottom-up signals derived from the child nodes.

Differences for nonlinear value coupling

So far, we have assumed linear value coupling in presenting the computations of value parent and children nodes. In the case of nonlinear value coupling, the update equations only change slightly. Specifically, in the prediction step, we now have the function g during the computation of the new predicted mean. Assuming that node a is the (nonlinear) value child of nodes , the total predicted mean drift for time step k (previously equation 36) will be

In other words, the influence of the higher-level belief on the prediction of the lower-level belief is mediated by the function gb_{i, a}, just as we would expect it to be based on the generative model (equation 52).

In the update step for the value parent (previously equation 38), we now have:

Consequently, for the update step, the node b now also needs access to its own previous posterior mean . Apart from these changes, all update equations from the previous section apply. This extension allows us to account for the fact that most states in the world interact non-linearly.

Other types of nodes

We have thus far focused our discussion on nodes that represent beliefs about continuous states which evolve in time as random walks (whether Gaussian or auto-regressive). However, the generalized HGF can also accommodate beliefs about any type of state governed by an exponential family distribution by filtering this distribution’s sufficient statistics Mathys & Weber (2020). This makes it possible to track binary and categorical states, where value parents track the probabilities with which one of several possible states is occupied, or with which transitions happen. This can be applied in a range of contexts, from experimental tasks where categorical probabilities must be learned Marshall et al. (2016) to inversions of other classes of discrete state space generative models like the Partially Observable Markov Decision Process models often used in Active Inference (Parr et al., 2022).

Furthermore, at the lowest level of the state hierarchy, we find states that are observable. These typically do not perform random walks, but are instead generated by their parent states independently on every time step. For continuous states in the case of value coupling, this corresponds to setting λ to zero so that the state only depends on its value parent and not on its own past anymore. Similarly, a volatility parent of an outcome state becomes a noise parent - because the variance in the Gaussian distribution no longer corresponds to a step size with respect to a previous mean but instead to the deviation from the current mean which is fully determined by the state’s value parent (s). In other words, an agent using this type of coupling in the generative model forms an explicit and dynamic belief about the level of observation noise (stochasticity) in a particular outcome (noise coupling, see also Example Simulation 2, Figure 7).

Figure 6.

Figure 7.

Table 1.

Input nodes are directly fed with observations (sensory inputs) (which can be continuous or binary) instead of receiving prediction errors from other nodes. In many applications, we would assume that the lowest level modelled (e.g., primary sensory cortices) is already somewhat distant to the actual sensors (e.g., the retina), which means we can cast its inputs as prediction errors generated during downstream processing input (e.g., in subcortical structures). However, in the case of noise coupling (see Simulation Example 2, Figure 7), and for the algorithmic implementation of our perceptual model, the equations goverining these nodes matter. Their treatment is presented in Appendix 6.3.

Summary: A network of nodes

In this section, we have used the update equations of the HGF to propose a conceptualization of the inference machinery as a network of nodes which compute beliefs (i.e., probability distributions) and exchange messages with other nodes. Every node in this network represents an agent’s current belief about a hidden state in its environment, on which it infers given its sensory inputs. Within every node, belief updating in response to a new input proceeds in three steps (an update step, a PE step, and a prediction step).

We have presented the computations for these steps for the two different kinds of coupling that the HGF comprises: value coupling and volatility coupling. While the update equations for volatility coupling have been derived and discussed previously (Mathys et al., 2011, 2014a), approximately Bayes-optimal inference equations for (linear and nonlinear) value coupling under the HGF have not been considered prior to our treatment here. Furthermore, our analysis identifies not only the computations entailed by each computational step, but also the message passing between nodes that is required by each step. This is interesting from a theoretical point of view, where we can compare our architecture to other proposals of belief propagation.

From a practical point of view, the division of the belief updating machinery into subunits (nodes) allows for a modular implementation, where networks can easily be extended and modified by adding or removing nodes, or by changing the type of coupling between nodes, without having to derive the relevant equations for the whole network anew. In two open-source projects, we provide such an implementation (in Python and Julia, Legrand et al., 2024; Thestrup Waade et al., in prep), which allows users to flexibly design their own HGF structures that can be used for simulation and empirical parameter estimation. These tools are freely available as part of the TAPAS software collection (Frässle et al., 2021, https://github.com/ComputationalPsychiatry).

Which conclusions can be drawn with respect to the message passing implied by the hgf? First, while the exact computations performed during the three computational steps depend on the position of the node within the network (e.g., number of children and parent nodes) and the nature of the coupling to other nodes (value vs. volatility coupling), we have identified generic structures in these equations (see Figure 3B), which are of interest from a theoretical point of view, but also facilitate implementation.

For example, belief updates in a node always require messages from lower-level nodes that contain prediction errors (δ_a for value coupling, Δ_a for volatility coupling) and estimates of precision ( in value coupling and in volatility coupling). Similarly, forming a new prediction always entails modifying the mean of the belief by an expectation of drift, and modifying the precision by an expectation of volatility during the next time step. Expectations of drift will be driven by value parents, expectations of volatility by volatility parents, but the structure of the equations is the same for both types of coupling (see equations 34 to 37).

In Figures 4 and 5, we additionally provide a more detailed overview of what happens within and between nodes for specific coupling types. For this purpose, we additionally consider separate subunits for calculations concerning means versus precisions versus prediction errors. Exploring in how far these architectures might map onto structures and networks in biological brains will be an interesting future task.

4 Practical applications

In this section, we provide a few example simulations which show the range of generative models our generalization of the HGF encompasses. We focus on relatively low-dimensional situations that have been of interest in empirical research in cognitive neuroscience and computational psychiatry, but have so far been challenging to model.

In the first example, we exploit the fact that nodes in the generalised HGF can share parent nodes. This allows us to model generalisation. For example, an agent who experiences high volatility in one context or domain might change their higher-level beliefs about the stability of the world more generally, and thus start to treat the dynamics in other domains or contexts as similarly volatile. In our minimal example for this, two hidden states separately generate two streams of observations (Figure 6). These could for example represent the observed behaviours of two different acquaintances the agent interacts with, or the availability of food in two different patches. Each state also has its own phasic volatility parent. However, these volatility states are both influenced by a shared value parent that encodes a higher-level state of global environmental volatility. For example, while the availability of food varies independently over time in the two patches, the overall season might produce upticks in volatility across different food sources simultaneously. We simulate two hidden states (x_a and x_b), with one of them more volatile than the other (volatility parent x_ǎ > ) The shared value (drift) parent of the two volatility nodes, x_c, produces a marked up-tick (fast drift starting at trial 50) and one down-tick in the individual volatilities. The simulation shows that under the chosen parameter settings the inference network is able to infer the individual as well as the shared (“global”) volatility states. The level of noise in the outcomes (u_a and u_b) was chosen relatively low. This model architecture could be used to explore how beliefs about environmental volatility generalize across different environmental states, and under which conditions this breaks down (e.g., overgeneralisation).

In the second example, we revisit a widely studied perceptual process: multi-sensory cue combination. When humans observe two cues (e.g., an auditory and a visual one) that each provide information about a hidden state of interest x_c (such as the location of an object, Figure 7), they can integrate both sources of information, weighing each source according to its reliability (Ernst & Banks, 2002). Under natural circumstances, these reliabilities might dynamically change, for example due to physiological fluctuations or changes in the external environment. Here, we model a situation where two observation sequences are each subject to dynamically changing noise (hidden states x_ǎ and ) using nodes connected via noise coupling. Our model features a shared value parent node (hidden state x_c) that generates both outcomes. The simulations show that the agent can infer these changing noise levels and adjust its inference on the shared hidden state x_c according to its current estimate of relative cue reliability. Critically, this setup allows us to model and infer changes in the subjective beliefs about the relative reliabilities and their impact on sequential belief updating. The states x_ǎ and could additionally be children of higher-level states, for example, to model higher-level beliefs about one’s confidence in different sensory modalities. The model architecture is relevant for cognitive tasks that assess perceptual integration between different modalities (such as integrating interoceptive signals with auditory feedback when inferring one’s own heart rate) in trial-by-trial fashion. Maladaptive over- or under-weighting of interoceptive versus exteroceptive signals is believed to drive symptoms of mental health (Paulus & Stein, 2010).

Finally, in our last example, we exploit the fact that the same node in the generalised HGF can have parent or child nodes with different coupling types, and that any node within a network can be connected with an outcome node, if the state becomes somehow observable. In Figure 8, we consider an extension of an experiment that has found widespread application in decision neuroscience: a reversal learning task, where participants track the probability of a binary outcome over a sequence of observations (e.g., whether a certain choice or stimulus is rewarded or not). The example sequence chosen here includes probability reversals and stable periods of p = 0.5 (i.e., both outcomes are equally likely). The typical model for this is that the participant infers on a hidden state x_a (and potentially its volatility x_ǎ) which represents the probability or tendency for one outcome over the other. In our example, we add an additional (continuous) observation to this hidden state x_a, in other words, in addition to the binary outcomes on every trial, the agent sometimes also has access to a (noisy) sample of the continuous hidden probability state. The simulation shows that inference on small jumps in probability as well as stable periods of 50/50 is difficult based on binary observations alone (trials 50 to 150), but that adding a continuous observation of the probability itself, even if it is only a very noisy readout of the actual probability, stabilizes inference. The model architecture thus allows us to make predictions about the relative usefulness of different types of observations when inferring on a hidden state.

Figure 8.

5 Conclusions

The work presented here makes several contributions. First, our extension to value coupling includes principles of predictive coding in the HGF framework. This offers a general and versatile modelling framework, offering an approximation to optimal Bayesian inference for different types of interactions between states in the world, allowing for inter-individual differences in the dynamics of belief updating, and providing a principled treatment of the multiple forms of uncertainties agents are confronted with.

Second, we present a modular architecture for HGF networks, where beliefs represent nodes that perform three basic computational steps: an Update step, a PE step, and a prediction step, in response to new input. While the equations for these steps differ depending on the coupling of a node to other nodes, we identify a generic structure that allows for a modular implementation, in which nodes can easily be added to or removed from a network, without having to derive the corresponding update equations for the model anew. This feature takes significant load off researchers wanting to apply this modelling framework and to create custom models that suit their experiments. We provide such implementations in two open-source projects (Legrand et al., 2024; Thestrup Waade et al., in prep) which are available as part of the TAPAS software collection (https://github.com/ComputationalPsychiatry Frässle et al., 2021).

Finally, by considering the case of nonlinear value coupling and deriving the message passing scheme implied by this, we enable a formal comparison to other proposed architectures for hierarchical Bayesian inference, most prominently Bayesian (or generalized) predictive coding (Bastos et al., 2012; Shipp, 2016) and belief propagation in active inference (e.g., Fig. 5.1 in Parr et al., 2022).

5.1 Modelling different sources of uncertainty

Whenever agents are faced with observations that violate their expectations, they need to arbitrate between different explanations – has the world changed, requiring an update of beliefs about hidden states, or was the deviation merely due to noise in their observations? As has been shown previously (Mathys et al., 2011, 2014a), the HGF models belief updating in an agent who takes into account several forms of uncertainty for determining the optimal learning rate in the face of new observations: sensory uncertainty (how noisy are the sensory inputs I receive), informational uncertainty (how much do I already know about the hidden state that generates the inputs), and environmental uncertainty (what is the rate of change I expect in the hidden state). All of these together will determine whether (and how much) the agent updates its beliefs about a hidden state in response to unexpected observations.

Here, we show that under the HGF, the agent cannot only learn about environmental volatility – where higher estimates of volatility lead to faster learning, but in an undirected manner –, but also about higher-level hidden states that cause changes in lower-level hidden states in a directed fashion. For example, the weather might be more volatile in some seasons compared to others, making the agent less certain in its predictions (and faster to learn) about the likelihood of rainfall (volatility coupling). On the other hand, it might expect more or less rainfall in certain seasons (value coupling).

This flexibility in building models of hierarchically interacting states in the world allows for some particularly interesting use cases. In Figure 6, we have provided an example where two hidden states evolve with their own respective evolution rates (both determined by a tonic component and a phasic component), but share a higher-level value parent (“global volatility”) that drives changes in the mean of their respective phasic volatility states. This setup allows for separate estimation of “local” volatility (specific to each hidden state) and more global influences on volatility. For example, the rate of change in the availability of two different foods in the environment might vary over time and seasons in a way that is specific to each type of food, but when switching to a different environment (or after a global change to the overall climate), the availability of both foods might become more or less stable. Investigating how the brain represents each form of volatility, and thus adjusts learning rates in a modality-specific as opposed to a general manner is an important part of understanding how the brain achieves and maintains the delicate balance of precision across different hierarchical levels (Kanai et al., 2015; Clark, 2013) which appears crucial for mental health (Petzschner et al., 2017; Sterzer et al., 2018).

Finally, agents are confronted not only with phasic variations of volatility (driving changes in the agent’s environmental uncertainty), but also with dynamically changing sensory (or observation) noise. While existing modelling approaches have focused on either accounting for changes in volatility (Behrens et al., 2007; Mathys et al., 2011; Piray & Daw, 2020) or block-wise changes in stochasticity or noise (Lee et al., 2020; Nassar et al., 2010, 2012, 2016), it has recently been pointed out that real-world agents need to able to detect (and distinguish) dynamic changes in both at the same time (Piray & Daw, 2021), and recent empirical work has demonstrated that human participants are indeed able to do so (Piray & Daw, 2024; Foucault et al., 2025). In the HGF, dynamic changes in observation noise can easily be accommodated by hidden states that serve as noise parents to observable outcome states (see Appendix 6.3 for the equations, Figure 7 for an example simulation, and Mikus et al., 2025 , for an application to a dataset in computational psychiatry).

5.2 Implementing the HGF’s message passing scheme

Hierarchical filtering and predictive coding are two prominent classes of hierarchical Bayesian models that cast perception as inference and model belief updates in proportion to precision-weighted prediction errors. Models from both classes are widely used, both in basic (computational) neuroscience, and for understanding mental disorders in computational psychiatry (Petzschner et al., 2017).

While the message passing architecture implied by different predictive coding models has been examined in detail and partly matched with neuroanatomy and -physiology (for overviews, see Spratling, 2019; Keller & Mrsic-Flogel, 2018; Bastos et al., 2012; Shipp, 2016), and hierarchical filtering models share many similarities with predictive coding models, it is currently not clear whether the respective inference networks would place distinct requirements on implementation (in computers or brains), or make distinct predictions about neural readouts of perceptual inference and learning. This is partly due to their non-overlapping applications: predictive coding models consider hierarchies in which higher levels affect the mean of lower levels, and they are typically used to model inference about static sensory inputs in continuous time (for a recent extension to dynamic inputs, see Millidge et al., 2024).

We reduce this gap by introducing the HGF scheme for value coupling alongside volatility coupling. Our results show (1) that HGF inference networks for value coupling are largely compatible with recently proposed predictive coding architectures in that messages passed between nodes of the network entail a bottom-up signalling of precision-weighted prediction errors, and a top-down influence on predictions; and (2) that there are slight but interesting differences in the updating of belief uncertainty.

One noteworthy difference between the architectures is that the update equations in the HGF require a bottom-up transmission of lower-level precision estimates (Figures 3 and 4). (It is not surprising that the differences between the models concern the signalling of precision: The HGF derivation explicitly includes update equations for the precision associated with beliefs - as do other hierarchical Bayesian architectures based on Markovian processes [Friston et al., 2013]. In contrast, most predictive coding schemes only focus on the optimization of the first moment [mode or expectation] of the posterior distribution for perceptual inference [Friston, 2005; Bogacz, 2017], although the variational approach does allow approximation of the full posterior distribution including its variance.) This is interesting, given that recent neuroanatomical studies point to additional pathways besides the classical forward (from lower-level supragranular to higher-level granular layers) and backward (from higher-level infragranular to lower-level extragranular layers) connections (Markov et al., 2013, 2014). Our architecture is compatible with an ascending connection within supragranular layers (for bottom-up communication of lower-level precision) that runs in parallel to a descending connection within these layers (for top-down modulation of PEs by higher-level precision), reminiscent of the “cortical counter streams” identified by these studies. We hope to capitalize on methodological advancements in high-resolution laminar fMRI (Stephan et al., 2019; Haarsma et al., 2022) in future studies to test these predictions.

5.3 Dynamics within versus across time steps

Although potential neurobiological implementations of approximate Bayesian inference in the brain are still hotly debated (Knill & Pouget, 2004; Aitchison & Lengyel, 2017), a growing body of literature suggests that predictive coding-like architectures can account for a large range of neurophysiological findings in perception research (for a recent overview, see Walsh et al., 2020), making these architectures particularly relevant for understanding human perception and inference. It would thus be interesting to examine in detail the commonalities and potential differences between message-passing schemes implied by different forms of coupling in HGF models and classical predictive coding schemes, as we have started to do here.

Importantly, however, direct comparisons of the two models are further complicated by differences in each model’s concept of time: while the HGF captures belief updates across time steps in discrete time (i.e., across sequentially arriving sensory inputs), predictive coding describes the evolution of beliefs in continuous time and, typically, in response to static sensory input (Rao & Ballard, 1999; Friston, 2005) (in fact, the standard form of the generative model in predictive coding does not contain temporal dynamics of the hidden states, but see Friston 2008, 2010] for extensions). Differential equations capture the evolution of beliefs and predictions errors and are used to simulate perceptual inference, starting with new input to the lowest level of the belief hierarchy, and ending when the ensuing PEs have been reconciled, i.e., a stable new posterior belief has emerged (Bogacz, 2017). This can be used to make predictions about neural activity that can be compared against measurements. While, to our knowledge, existing predictive coding models have not been fit to data, the simulated neural dynamics display many features that are observed in real data, such as oscillatory tendencies, even in very small networks (Bogacz, 2017). We refer to these simulations as within-step dynamics of belief updating. Furthermore, predictive coding networks have recently been rediscovered as potent candidates for solving machine learning tasks (e.g., the classification or generation of static images, Sun & Orchard, 2020; Song et al., 2020) and even a potential alternative to the backpropagation algorithm (Song et al., 2024; Millidge et al., 2022b; Whittington & Bogacz, 2017).

On the other hand, the generative model of the HGF represents a Markovian process in discrete time; in the inference model, one-step update equations, derived based on a mean field approximation to the full Bayesian solution, quantify the change from prior to posterior on all levels of the belief hierarchy. The model is thus examined in sequential input settings to capture step-by-step learning - in other words, across-step dynamics of belief updating. The model provides an approximately Bayes-optimal solution to stepwise belief updating, useful for ideal observer analyses (Stefanics et al., 2018; Weber et al., 2020, 2022; Hauke et al., 2022), but the parameters of the HGF can also be fit to behavioural responses of individual participants. Model-derived agent-specific trajectories of predictions and PEs have proven particularly useful for identifying potential neural and physiological correlates of computational quantities in empirical data (Iglesias et al., 2013b; Vossel et al., 2015; de Berker et al., 2016; Diaconescu et al., 2017; Weilnhammer et al., 2018; Katthagen et al., 2018; Palmer et al., 2019; Deserno et al., 2020; Henco et al., 2020; Cole et al., 2020; Lawson et al., 2021; Hein et al., 2021; Hein & Herro jo Ruiz, 2022; Harris et al., 2022; Fromm et al., 2023). In the future, we propose to explicitly consider potential within-step dynamics of belief updates compatible with the update equations of the HGF (see Weber, 2020, for a first attempt). Such equations could for example be derived by treating the HGF posterior values of all nodes (quantities) as the equilibrium point towards which all dynamics must converge (inspired by Bogacz, 2017). Establishing equations for within-step belief updating dynamics under the HGF might allow for empirical tests of the proposed architecture in a two-step procedure: first, individual trajectories of predictions and prediction errors are inferred from observed behaviour by fitting the HGF to participants’ responses, second, these stepwise point estimates are subsequently used to simulate expected continuous-time neuronal responses according to the differential equations.

A modular implementation of the generalized HGF makes it easy to build large networks with considerable hierarchical depth, opening up exciting possibilities of applying this model architecture in machine learning applications and comparing its performance to alternative neural network architectures (Whittington & Bogacz, 2017; Millidge et al., 2022a; Song et al., 2024). When applied as a model of human cognition, however, an increase in model complexity must be matched by sufficiently rich data to successfully fit the model. In the generalised HGF, a branching out into the higher levels of the belief hierarchy (multiple parent nodes) will typically require some amount of branching out in the other direction (multiple child nodes) to disambiguate beliefs about hidden states further up.

In summary, we have presented a generalization of the HGF that extends its scope of hierarchical inference mechanisms to include cross-level couplings as proposed by predictive coding. The result is a new class of artificial neural networks that incorporate computational principles of two popular hierarchical Bayesian inference schemes: HGF and predictive coding. Furthermore, we have demonstrated how this extension can be cast as a modular architecture that allows for flexible changes to a model without having to re-derive update equations, thus allowing for a modular construction of complex hierarchically structured ANNs. Finally, the availability of these developments as open source software expands the toolkit of computational psychiatry and we hope that it will facilitate future investigations of perceptual inference in health and disease.

6 Appendix

6.1 Approximate inversion for value coupling

In Mathys et al. (2011), we presented a variational approximation to the exact Bayesian inversion of our generative model which employed a mean-field approximation, and derived analytic one-step update equations using a new quadratic approximation to the variational energies. Following this procedure for the case of value coupling, we specify the variational energy for a value parent to derive the update equations in the main text.

The generative model for a state x_a with a (non) linear value parent x_b (and a volatility parent x_ǎ is given by (note that for brevity, we are omitting all priors here - strictly speaking, these equations only form a generative model if combined with appropriate priors on the model parameters and the initial states.)

where the value coupling between x_a and x_b is mediated by function g, which can be nonlinear.

Using the mean-field approximation as in Mathys et al. (2011), the variational energy and its first two derivatives for the value parent x_b, are given by

We calculate the mean and precision of the Gaussian posterior for the value parent using the rules as stated in Mathys et al. (2011) (equations 38 and 40 there), which follow a quadratic approximation to the variational energy with expansion point at the posterior belief from the previous time step. For this, we need the derivatives of the variational energies at this point:

Here, we identify the prediction of the mean about the value child state x_a as

and thus the prediction error about x_a as

Therefore, the first derivative of the variational energy becomes

Similarly, the second derivative then reads:

With these, we can specify the update equations for the precision π_b, and the mean μ_b of the value parent (see Mathys et al. (2011) and Appendix B of Mathys et al. (2014a)):

If g(x) = x (linear value coupling), then g′(x) = 1 and g″(x) = 0, and we obtain the update equations specified in section 3.

6.2 Definition of a VOPE

In the main text, we introduced a new definition Δ of the volatility prediction error or VOPE, which we express as a function of the previously defined value prediction error δ, or VAPE. Here, we show how our new definition derives from the definition contained in earlier work (Mathys et al., 2011):

From the first to the second line, we have used the following definition:

This ensures that a given node does not need to have access to the posterior precision from the level below: , which facilitates implementation.

In sum, we are introducing a second prediction error unit Δ which is concerned with deviations from predicted uncertainty and is informed by value prediction errors and estimates of uncertainty. It is this prediction error - a function of the squared value prediction error - which communicates between a node and its volatility parent.

6.3 Computations of other types of nodes

In the main text, we focus on continuous state nodes (i.e., states performing a Gaussian or auto-regressive random walk). We now also specify the nodalized implementation of inference for binary (and, by extension, categorical) belief nodes as well as input nodes (corresponding to observable/outcome states).

Input nodes differ from regular nodes in that the inputs do not perform a random walk (λ = 0), but are noisily generated by peripheral states at each time step (see section 2 of the main text). We refer to the observations or sensory stimuli that enter the network as “inputs” and therefore call the receiving nodes input nodes. However, from the perspective of the generative model, these are the observable outputs of the network.

The input nodes are important elements in the HGF belief network. Processes which need to take place in an input node at a given time step are:

• Receive a new input and store it

• Either receive as a second input the exact time interval since the previous input, or infer the time as ‘previous plus 1’ (e.g., next time step)

• Compute all quantities which need to be signalled to the parent node (e.g., prediction error)

• Send these quantities to the parent node

• Receive top-down messages from the parent node (e.g., μ

• Compute surprise (i.e., the negative logarithm of the input’s probability given the prediction)

The quantities being signalled bottom-up, and the computation of surprise, depend on the nature of the input node (continuous or binary) and on the nature of the coupling with the parent.

Because input nodes are different from HGF state nodes, but rather serve as a relay station for the input and for computing surprise, and because they capture any observation noise that might be inherent in the input, the message passing and the within-node computations differ from the generic scheme presented in the main text.

Continuous input nodes

A continuous input node receives inputs u which can be any real number. In terms of the generative model, we think of these inputs as being sampled from a Gaussian distribution with a mean determined by the state node it is coupled to and a variance which is either constant or determined by another hgf node. This variance is the observation noise.

As with the coupling types introduced in the main text, we call the state node that determines the input’s mean its value parent. However, the state node which represents the phasic component of the input’s variance (or observation noise), is not a volatility parent, since the input has no volatility because it is not a state. Instead, we call such a parent node a noise parent. Every continuous input node has one value parent, but having a noise parent is optional. When noise is not determined by a noise parent, it is a constant parameter of the input node.

Value parents of continuous input nodes

The predicted mean of an input node is simply the prediction from the value parent p:

Since the mean parent might have a drift parameter, the current prediction can only be computed once the new input has arrived. Then it needs to be signalled top-down immediately.

In the absence of a noise parent, the precision of the prediction for the input node is fully determined the input node’s noise parameter ε:

However, in the presence of a noise parent q, this will additionally depend on the posterior of that parent at the previous time step, and the coupling parameter κ_q,u of the input node with its noise parent q:

In the update step, the posterior mean of the input node is the input itself (the posterior precision is not required, but would be infinite, as the input is known):

Finally, in the PE step, the value PE (or VAPE) will be computed as the difference the prediction and the posterior:

This means that prior to the update of the input node, it needs to receive the current prediction of its parent node.

The update of the value parent node will look like the regular value coupling updates from previous chapters:

This means that the input node needs to signal bottom-up to its mean parent:

Precision of the prediction:

Prediction error: .

The implicit assumption here is that the connection between a continuous input node and its value parent has a connection weight of α = 1. Should a use case arise where this is inconvenient, it can easily be changed to be a variable parameter.

It may seem slightly artificial to construct the computational steps for the input node in this way since the actual belief about the input is represented in the parent node. However, this allows for a more modular implementation where the value parent can remain agnostic as to whether its child is another state node or a continuous input node.

Finally, to compute the surprise associated with the current input, the node needs to compute the negative logarithm of the probability of input u^(κ) under a Gaussian prediction with as mean and as the precision:

Noise parents of continuous input nodes

Having a noise parent for a continuous input node means that a noise PE (or NOPE) will be computed and signalled bottom-up during the PE step. We denote this PE with the symbol ε_u. Importantly, the NOPE (as opposed to the VOPE) is not a direct function of the VAPE. Instead, both the posterior precision as well as the posterior mean are taken from the value parent p:

This in turn requires that the update of the value parent happens before the computation of the NOPE, and the posterior of the value parent is already available to the input node.

The update step for the noise parent is similar to the update in volatility parents (equations 45) with a modified prediction error and an effective precision term γ_u fixed to 1:

This similarity again means that the parent node can remain agnostic as to whether it serves as a volatility or a noise parent - as long as the input node also signals a value of 1 as the effective precision term γ at every time step. Importantly, this also works for the update of the precision of the noise parent. Setting γ_u to 1 (and replacing the VOPE Δ with the NOPE ε and vopa with q) in the previously established precision update for volatility parents (equation 45) leads to:

Peculiarities of continuous input nodes and consequences for their parents

The possibility for a given state node to be the value parent or the noise parent of a continuous input node has a number of consequences for the implementation of state nodes:

First, owing to the dependence of the NOPE on the posterior beliefs of the mean parent, the continuous input node needs to communicate with its value parent first and wait for the posteriors to be computed there and sent top-down in order to trigger an update in its noise parent.

Second, the value parent needs to send top-down not only the posterior mean, but also the posterior precision, for the same reason.

Third, the connection weight for value connections will always be α = 1.

Fourth, for issuing a new prediction , the node needs to receive the predicted mean of its value parent at the beginning of a new time step. This means it must be possible to elicit a new prediction in regular hgf nodes without actually sending a prediction error, instead by only sending a new time point. The hgf node needs to react to this by sending top-down the new predicted mean, such that the input node can compute the PE and signal it back bottom-up for an update.

Thus, the steps for a continuous input node are:

• receive input u

• determine time of input

• send bottom-up to value parent: time of input (to elicit a prediction)

• receive top-down: predicted mean

• compute prediction and retrieve

• compute surprise using u, and

• compute VAPE using u and

• send bottom-up to value parent: VAPE, , and time

• receive top-down: posteriors μ_p and π_p

• if relevant, compute NOPE using u, , μ_p and π_p

• send bottom-up to noise parent: NOPE, γ_u = 1, and time

• receive top-down: posterior μ_q

• compute new precision of its prediction using its tonic observation noise ε and, if present, the posterior mean of its noise parent .

The value parent of a continuous input node needs to

1. be able to elicit new predictions based on time input

2. send new predictions top-down immediately in the case of a continuous input node child

3. send down not only its posterior mean, but also the precision after each update.

Binary input nodes

Binary input nodes serve to receive inputs that can only take on one of two values. These input nodes can only have one value parent because their stochastic properties are fully described by a Bernoulli distribution which only has one parameter. The value parents binary input nodes are binary state nodes, which are special cases of state nodes which themselves only have value parents. This implementation results in the value parents of binary HGF nodes being regular state nodes which can be agnostic as to whether their child node is a regular state node, a binary hgf node, or a continuous input node.

For binary input nodes, the observation noise is given by their noise parameter ε Therefore, the precision of the input prediction is constant (i.e., we can treat it as a parameter). We here only present the case without observaton noise (i.e., ε_u = 0 or ), and leave the case of finite precision for a future treatment.

In general, the steps for a binary input node are:

• receive input u

• determine time of input

• compute prediction errors, if necessary

• send bottom-up: u, input precision, and time

• receive top-down: prediction of parent

• compute surprise based on message from parent.

If is infinite, then the bottom-up messages are simply this precision itself, and u. The surprise computation is also very simple:

The special cases that follow for the update of the parent node are restricted to binary HGF nodes, which therefore represent their own special case of HGF nodes.

Binary state nodes

Binary nodes are parents of binary input nodes. Their cycle starts with receiving a bottom-up message from their child node, which, first of all, needs to trigger the prediction step. Similarly to continuous input nodes, the predictions of a binary HGF node depend on its parent’s predictions (Adams et al., 2018):

The precision of the prediction is a direct function of the mean owing to the binary nature of the state.

Again, we need to introduce an additional top-down signalling step at the beginning of each time step, where the parent node sends down its current prediction of the mean, given the time interval since the last input.

The bottom-up message that binary state nodes receive consists of three quantities (,u^(κ) and the time of the input), and the time since the last input). The updates read:

Finally, in the PE step, the binary node computes a VAPE for its value parent:

The parent node will also perform its update step according to the following equations (Mathys et al., 2014b ; Adams et al., 2018):

For the implementation, this means that we can either give the hgf node knowledge about who its child is and let the exact update depend on that, or we can let this be solved by the value connection, in which case this connection would need to signal the precision weight that is used for the mean update separately from the term that is used for the precision update.

In any case, the information which needs to be sent bottom-up from a binary HGF node to its value parent is:

Prediction error:

Predicted precision:

In case the parent does not have knowledge about its child, the information would have to be sent in the following form:

Prediction error:

Precision weight for mean update: 1

Precision term for precision update:

Summary: Implementational consequences

Due to the special cases of continuous input nodes and binary state nodes, which both can be potential children of regular state nodes, we need to introduce a few changes to the update and connection logic of the regular state nodes:

• State nodes need to emit new predictions if prompted by receiving information about the time of the new input, and send this prediction top-down. This is needed both for the computation of surprise in the continuous input nodes, but also for the computation of prediction error in continuous input nodes, and for the computation of predictions in binary state nodes.

• In the case of value parents of continuous input nodes, state nodes need to signal top-down not only their posterior mean, but also their posterior precision.

• Value-coupling connections need separately to signal bottom-up the precision weight of the upcoming prediction error, and the precision term needed to update the parent’s precision.

• Implementing noise and volatility connections in the same way allows for an implementation where regular state nodes are completely unaware about which kind of node their child is. The computation necessary for the precision update, which is more elaborate in volatility and noise coupling is then part of the connection logic.

Everything else that is unusual about the computations within binary input nodes, continuous input nodes, and binary state nodes can then be implemented within these nodes withoutaffecting the regular continuous node implementation.

6.4 Notation overview

Table 2 provides an overview of the notation used throughout this paper.

Table 2.

Data availability

The current manuscript is a computational study, so no data have been generated for this manuscript. Modelling code is available as part of an open source toolbox.

Acknowledgements

This work was supported by the René and Susanne Braginsky Foundation (KES), the Aarhus Universitets Forskningsfond (grant AUFF-E-2019-7-10) (CM), and the Carlsberg Foundation (grant CF21-0439) (CM).

Additional information

Funding

René and Susanne Braginsky Foundation

• Klaas Enno Stephan

Aarhus Universitets Forskningsfond (Aarhus University Research Foundation) (AUFF-E-2019-7-10)

• Christoph Mathys

Carlsbergfondet (Carlsberg Foundation) (CF21-0439)

• Christoph Mathys