The synapse is the major site of inter-cellular communication in the brain. The amplitude of synaptic postsynaptic potentials (PSPs) are usually highly variable or stochastic. This variability arises primarily presynaptically: the release of neurotransmitter from presynaptically housed vesicles into the synaptic cleft has variable release probabilities and variable quantal sizes (Lisman and Harris, 1993; Branco and Staras, 2009; Brock et al., 2020). Unreliable synaptic transmission seems puzzling, especially in light of evidence for low-noise, almost failure-free transmission at some synapses (Paulsen and Heggelund, 1994; Paulsen and Heggelund, 1996; Bellingham et al., 1998). Moreover, the degree to which a synapse is unreliable does not just vary from one synapse type to another, there is also an heterogeneity of precision amongst synapses of the same type (Murthy et al., 1997; Dobrunz and Stevens, 1997). Given that there is capacity for more precise transmission, why is this capacity not used in more synapses?

Unreliable transmission degrades accuracy but Laughlin et al., 1998, showed that the synaptic connection from a photoreceptor to a retinal large monopolar cell could increase its precision by increasing the number of synapses, averaging the noise away, but this comes at the cost of extra energy per bit of information transmitted. Moreover, Levy and Baxter, 2002, demonstrated that there is a value for the precision which optimises the energy cost of information transmission. In this paper, we explore this notion of a performance-energy tradeoff.

However, it is important to consider precision and energy cost in the context of neuronal computation; the brain does not simply transfer information from neuron to neuron, it performs computation through the interaction between neurons. However, models outlining a synaptic energy-performance tradeoff, (Laughlin et al., 1998; Levy and Baxter, 2002; Goldman, 2004; Harris et al., 2012; Harris et al., 2019; Karbowski, 2019), predominantly consider information transmission between just two neurons and the corresponding information-theoretic view treats the synapse as an isolated conduit of information (Shannon, 1948). In contrast, in reality, a single synapse is just one unit of the computational machinery of the brain. As such, the performance of an individual synapse needs to be considered in the context of circuit performance. To perform computation in an energy-efficient way the circuit as a whole needs to allocate resources across different synapses to optimise the overall energy cost of computation (Yu et al., 2016; Schug et al., 2021).

Here, we consider the consequences of a tradeoff between network performance and energetic reliability costs that depend explicitly upon synapse precision. We estimate the energy costs associated with precision by considering the biological mechanisms underpinning synaptic transmission. By including these costs in a neural network designed to perform a classification task, we observe a heterogeneity in synaptic precision and find that this ‘allocation’ of precision is related to signatures of synapse ‘importance’, which can be understood formally on the grounds of Bayesian inference.

We proposed energetic costs for reliable synaptic transmission and then measured their consequences in an artificial neural network (ANN).

Biophysical costs

Here, we seek to understand the biophysical energetic costs of synaptic transmission, and how those costs relate to the reliability of transmission (Figure 1a). We start by considering the underlying mechanisms of synaptic transmission. In particular, synaptic transmission begins with the arrival of a spike at the axon terminal. This triggers a large influx of calcium ions into the axon terminal. The increase in calcium concentration causes the release of neurotransmitter-filled vesicles docked at axonal release sites. The neurotransmitter diffuses across the synaptic cleft to the postsynaptic dendritic membrane. There, the neurotransmitter binds with ligand-gated ion channels causing a change in voltage, i.e., a PSP. This process is often quantified using the Katz and Miledi, 1965, quantal model of neurotransmitter release. Under this model, for each connection between two cells, there are ${\displaystyle n}$ docked, readily releasable vesicles (see Figure 1a for an illustration of a single synaptic connection with multi-vesicular release).

Figure 1

Download asset Open asset

An alternative interpretation of this model might consider ${\displaystyle n}$ the number of uni-vesicular connections between two neurons. When the presynaptic cell spikes, each docked vesicle releases with probability ${\displaystyle {\displaystyle p}}$ and each released vesicle causes a PSP of size ${\displaystyle q}$. Thus, the mean, μ, and variance, ${\displaystyle {\sigma }^2}$, of the PSP can be written (see Figure 1b),

(1) $\begin{array}{l}\mu =npq\\ {\sigma }^2=np(1-p)q^2\end{array}$

where ${\displaystyle q}$ is considered a scaling variable. An assertion in our model is that variability in PSP strength is the result of variable numbers of vesicle release, not variability in ${\displaystyle q}$; here, during any PSP, ${\displaystyle q}$ is assumed constant across vesicles. While there is some suggestion that intra- and inter-site variability in ${\displaystyle q}$ is a significant component of PSP variability (see Silver, 2003), we ultimately expect quantal variability to be small relative to the variability attributed to vesicular release. This is supported by the classic observation that PSP amplitude histograms have a multi-peak structure (Boyd and Martin, 1956; Holler et al., 2021), and by more direct measurement and modelling of vesicle release (Forti et al., 1997; Raghavachari and Lisman, 2004).

We considered four biophysical costs associated with improving the reliability of synaptic transmission, while keeping the mean fixed, and derived the associated scaling of the energetic cost with PSP variance.

Calcium efflux. Reliability is higher when the probability of vesicle release, ${\displaystyle p}$, is higher. As vesicle release is triggered by an increase in intracellular calcium, greater calcium concentration implies higher release probability. However, increased calcium concentration implies higher energetic costs. In particular, calcium that enters the synaptic bouton will subsequently need to be pumped out. We take the cost of pumping out calcium ions to be proportional to the calcium concentration, and take the relationship between release probability and calcium concentration to be governed by a Hill Equation, following Sakaba and Neher, 2001. The resulting relationship between energetic costs and reliability is ${\displaystyle \text{cost}\propto {\sigma }^{-1/2}}$ (Figure 1cI; see Appendix 1, ‘Reliability costs’ for further details).

Vesicle membrane surface area. There may also be energetic costs associated with producing and maintaining a large amount of vesicle membrane. Purdon et al., 2002, argues that phospholipid metabolism may take a considerable proportion of the brain’s energy budget. Additionally, costs associated with membrane surface area may arise because of leakage of hydrogen ions across vesicles (Pulido and Ryan, 2021). Importantly, a cost for vesicle surface area is implicitly a cost on reliability. In particular, we could obtain highly reliable synaptic release by releasing many small vesicles, such that stochasticity in individual vesicle release events averages out. However, the resulting many small vesicles have a far larger surface area than a single large vesicle, with the same mean PSP. Thus, a cost on surface area implies a relationship between energetic costs and reliability; in particular ${\displaystyle \text{cost}\propto {\sigma }^{-2/3}}$ (Figure 1cII; see Appendix 1, ‘Reliability costs’ for further details).

Actin. Another cost for small but numerous vesicles arises from a demand for structural organisation of the vesicles pool by filaments such as actin (Cingolani and Goda, 2008; Gentile et al., 2022). Critically, there are physical limits to the number of vesicles that can be attached to an actin filament of a given length. In particular, if vesicles are smaller we can attach more vesicles to a given length of actin, but at the same time, the total vesicle volume (and hence the total quantity of neurotransmitter) will be smaller (Figure 1cIII). A fixed cost per unit length of actin thus implies a relationship between energetic costs and reliability of, ${\displaystyle \text{cost}\propto {\sigma }^{-4/3}}$ (see Appendix 1, ‘Reliability costs’).

Trafficking. A final class of costs is proportional to the number of vesicles (Laughlin et al., 1998). One potential biophysical mechanism by which such a cost might emerge is from active transport of vesicles along actin filaments or microtubles to release sites (Chenouard et al., 2020). In particular, vesicles are transported by ATP-dependent myosin-V motors (Bridgman, 1999), so more vesicles require a greater energetic cost for trafficking. Any such cost proportional to the number of vesicles gives rise to a relationship between energetic cost and PSP variance of the form, ${\displaystyle \text{cost}\propto {\sigma }^{-2}}$ (Figure 1cIV; see Appendix 1, ‘Reliability costs’).

Costs related to PSP mean/magnitude. While costs on precision are the central focus of this paper, it is certainly the case that other costs relating to the mean PSP magnitude constitute a major cost of synaptic transmission. For example, high amplitude PSPs require a large quantity of neurotransmitter, high probability of vesicle release, and a large number of postsynaptic receptors (Attwell and Laughlin, 2001). These can be formalised as costs on the PSP mean, μ , and can additionally be related to L1 weight decay in a machine learning context (Rosset and Zhu, 2006; Sacramento et al., 2015).

Reliability costs in ANNs

Next, we sought to understand how these biophysical energetic costs of reliability might give rise to patterns of variability in a trained neural network. Specifically, we trained ANNs using an objective that embodied a tradeoff between performance and reliability costs,

(2) $\mathrm{O}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{t}=\text{performance}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{t}+\text{magnitude}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{t}+\text{reliability}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{t}.$

The ‘performance cost’ term measures the network’s performance on the task, for instance in our classification tasks we used the usual cross-entropy cost. The ‘magnitude cost’ term captures costs that depend on the PSP mean, while the ‘reliability cost’ term captures costs that depend on the PSP precision. In particular,

(3) ${\displaystyle \mathrm{m}\mathrm{a}\mathrm{g}\mathrm{n}\mathrm{i}\mathrm{t}\mathrm{u}\mathrm{d}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{t}=\lambda \sum _i|{\mu }_i|,}$

(4) ${\displaystyle \mathrm{r}\mathrm{e}\mathrm{l}\mathrm{i}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{t}=c\sum _i{\sigma }_i^{-\rho }.}$

Here, ${\displaystyle i}$ indexes synapses, and recall that σ_i is the standard deviation of the ith synapse. The multiplier ${\displaystyle c}$ in the reliability cost determines the strength of the reliability cost relative to the performance cost. Small values for ${\displaystyle c}$ imply that the reliability cost term is less important, permitting precise transmission and higher performance. Large values for ${\displaystyle c}$ give greater importance to the reliability cost encouraging energy efficiency by allowing higher levels of synaptic noise, causing detriment to performance (see Figure 2).

Figure 2

Download asset Open asset

We trained fully connected, rate-based neural network to classify MNIST digits. Stochastic synaptic PSPs were sampled from a Normal distribution,

(5) ${\displaystyle w_i\sim \text{Normal}({\mu }_i,{\sigma }_i).}$

where, recall, μ_i is the PSP mean and σ_i² is the PSP variance for the ith synapse. The output firing rate was given by

(6) ${\displaystyle \mathrm{f}\mathrm{i}\mathrm{r}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}=f\left(\sum _i{w}_i{x}_i-w_0\right).}$

Here, ${\displaystyle \sum _i{w}_i{x}_i-w_0}$ can be understood as the somatic membrane potential, and ${\displaystyle f}$ represents the relationship between somatic membrane potential and firing rate; we used ReLU (Fukushima, 1975). We optimised network parameters ${\displaystyle {\mu }_i}$ and ${\displaystyle {\sigma }_i}$ using Adam (Kingma and Ba, 2014) (see Materials and methods for details on architecture and hyperparameters).

The tradeoff between accuracy and reliability costs in trained networks

Next we sought to understand how the tradeoff between accuracy and reliability cost manifests in trained networks. Perhaps the critical parameter in the objective (Equation 2 and Equation 4) was ${\displaystyle c}$, which controlled the importance of the reliability cost relative to the performance cost. We trained networks with a variety of different values of ${\displaystyle c}$, and with four values for ${\displaystyle \rho }$ motivated by the biophysical costs (the different columns).

In practice all the reliability costs and others we may have overlooked should together constitute an overall energetic reliability cost. However, it is difficult to estimate the specific contributions of different costs, i.e., the individual values of ${\displaystyle c}$. While Attwell and Laughlin, 2001; Engl and Attwell, 2015, estimate the ATP demands for various synaptic processes, it is difficult to relate these to the relative scale of each cost at a synapse level. Therefore, for simplicity, we kept each cost separate, training neural networks with just one choice of reliability cost; emphasising results shared across all costs. It is possible that one cost dominates all the others, but if that is not the case it will be necessary to use a more complicated reliability cost. However, since we have considered four costs with very different power-law behaviours, it is likely the behaviour will not be significantly different to what we have observed.

As expected, we found that as ${\displaystyle c}$ increased, performance fell (Figure 2a) and the average synaptic standard deviation increased (Figure 2b). Importantly, we considered two different settings. First, we considered an homogeneous noise setting, where ${\displaystyle {\sigma }_i}$ is optimised but kept the same across all synapses (grey lines). Second, we considered an heterogeneous noise setting, where ${\displaystyle {\sigma }_i}$ is allowed to vary across synapses, and is optimised on a per-synapse basis. We found that heterogeneous noise (i.e. allowing the noise to vary on a per-synapse basis) improved accuracy considerably for a fixed value of ${\displaystyle c}$, but only reduced the average noise slightly.

The findings in Figure 2 imply a tradeoff between accuracy and average noise level, ${\displaystyle \sigma }$, as we change ${\displaystyle c}$. If we explicitly plot the accuracy against the noise level using the data from Figure 2, we see that as the synaptic noise level increases, the accuracy decreases (Figure 3a). Further, the synaptic noise level is associated with a reliability cost (Figure 3b), and this relationship changes in the different columns as they use different values of ${\displaystyle \rho }$ associated with different biological mechanisms that might give rise to the dominant biophysical reliability cost. Thus, there is also a relationship between accuracy and reliability costs (Figure 3c), with accuracy increasing as we allow the system to invest more energy in becoming more reliable, which implies a higher reliability cost. Again, we plotted both the homogeneous (grey lines) and heterogeneous noise cases (green lines). We found that heterogeneous noise allowed for considerably improved accuracy at a given average noise standard deviation or a given reliability cost.

Figure 3

Download asset Open asset

Energy-efficient patterns of synapse variability

We found that the heterogeneous noise setting, where we individually optimise synaptic noise on a per-synapse basis, performed considerably better than the homogeneous noise setting (Figure 3). This raised an important question: how does the network achieve such large improvements by optimising the noise levels on a per-synapse basis? We hypothesised that the system invests a lot of energy in improving the reliability for ‘important’ synapses, i.e., synapses whose weights have a large impact on predictions and accuracy (Figure 4a). Conversely, the system allows unimportant synapses to have high variability, which reduces reliability costs (Figure 4b). To get further intuition, we compared both ${\displaystyle w_1}$ and ${\displaystyle w_2}$ on the same plot (Figure 4c). Specifically, we put the important synapse, ${\displaystyle w_1}$ from Figure 4a, on the horizontal axis, and the unimportant synapse, ${\displaystyle w_2}$ from Figure 4b, on the vertical axis. In Figure 4c, the relative importance of the synapse is now depicted by how the cost increases as we move away from the optimal value of the weight. Specifically, the cost increases rapidly as we move away from the optimal value of ${\displaystyle w_1}$, but increases much more slowly as we move away from the optimal value of ${\displaystyle w_2}$. Now, consider deviations in the synaptic weight driven by homogeneous synaptic variability (Figure 4c left, grey points). Many of these points have poor performance (i.e. a high performance cost), due to relatively high noise on the important synapse (i.e. ${\displaystyle w_1}$). Next, consider deviations in the synaptic weight driven by heterogeneous, optimised variability (Figure 4c left, green points). Critically, optimising synaptic noise reduces variability for the important synapse, and that reduces the average performance cost by eliminating large deviations on the important synapse. Thus, for the same overall reliability cost, heterogeneous, optimised variability can achieve much lower performance costs, and hence much lower overall costs than homogeneous variability (Figure 4d).

Figure 4

Download asset Open asset

To investigate experimental predictions arising from optimised, heterogeneous variability, we needed a way to formally assess the ‘importance’ of synapses. We used the ‘curvature’ of the performance cost: namely the degree to which small deviations in the weights from their optimal values will degrade performance. If the curvature is large (Figure 4a), then small deviations in the weights, e.g., those caused by noise, can drastically reduce performance. In contrast, if the curvature is smaller (Figure 4b), then small deviations in the weights cause a much smaller reduction in performance. As a formal measure of the curvature of the objective, we used the Hessian matrix, ${\displaystyle \mathit{H}}$. This describes the shape of the objective as a function of the synaptic weights, the ${\displaystyle {\displaystyle w_i\mathrm{s}}}$: specifically, it is the matrix of second derivatives of the objective, with respect to the weights, and measures the local curvature of objective. We were interested in the diagonal elements, ${\displaystyle H_{ii}}$; the second derivatives of the objective with respect to ${\displaystyle w_i}$.

We began by looking at how the optimised synaptic noise varied with synapse importance, as measured by the curvature or, more formally, the Hessian (Figure 5a). The Hessian values were estimated using the average-squared gradient, see Appendix 3, ‘Synapse importance and gradient magnitudes’. We found that as the importance of the synapse increased, the optimised noise level decreased. These patterns of synapse variability make sense because noise is more detrimental at important synapses and so it is worth investing energy to reduce the noise in those synapses.

Figure 5

Download asset Open asset

However, this relationship (Figure 5a) between the importance of a synapse and the synaptic variability is not experimentally testable, as we are not able to directly measure synapse importance. That said, we are able to obtain two testable predictions. First, the input rate in our simulations was negatively correlated with optimised synaptic variability (Figure 5b). Second, the optimised synaptic variability was larger for synapses with larger learning rates (Figure 5c). Critically, similar patterns have been observed in experimental data. In Figure 6a we present the negative correlation between learning rate and synaptic reliability presented by Schug et al., 2021, from in vitro measurements of V1 (layer 5) pyramidal synapses before and after STDP-induced long-term plasticity (LTP) conducted by Sjöström et al., 2001. Furthermore, a relationship between input firing rate and synaptic variability was observed by Aitchison et al., 2021, using in vivo functional recordings from V1 (layer 2/3) (Ko et al., 2013; Figure 6b).

Figure 6

Download asset Open asset

To understand why these patterns of variability emerge in our simulations and in data, we need to understand the connection between synapse importance, synaptic inputs (Figure 5b, Figure 6b), and synaptic learning (Figure 5c, Figure 6a). Perhaps the easiest connection is between the synapse importance and the input firing rate. If the input cell never fires, then the synaptic weight cannot affect the network output, and the synapse has zero importance (and also zero Hessian; see Appendix 2, ‘High input rates and high precision at important synapses’). This would suggest a tendency for synapses with higher input firing rates to be more important, and hence to have lower variability. This pattern is indeed borne out in our simulations (Figure 5b; also see Appendix 6—figure 1), though of course there is a considerable amount of noise: there are a few important synapses with low input rates, and vice versa.

Next, we consider the connection between learning rate and synapse importance. To understand this connection, we need to choose a specific scheme for modulating the learning rate as a function of the inputs. While the specific scheme for modulating the learning rate is ultimately an assumption, we believe modern deep learning offers strong guidance as to the optimal family of schemes for modulating the learning rate. In particular, modern, state-of-the-art, update rules for ANNs almost always use an adaptive learning rate. These adaptive learning rates, ${\displaystyle {\eta }_i}$ (including the most common such as Adam and variants), almost always use a normalising learning rate which decreases in response to high incoming gradients,

(7) ${\eta }_i=\frac{{\eta }_{\text{base}}}{\sqrt{⟨g_i^2⟩}}.$

Specifically, the local learning rate for the ith synapse, η_i, is usually a base learning rate, η_base, divided by the root-mean-squared gradient at this synapse ${\displaystyle \sqrt{⟨g_i^2⟩}}$. Critically, the root-mean-squared gradient turns out to be strongly related to synapse importance. Intuitively, important synapses with greater impact on network predictions will have larger gradients (see Appendix 3, ‘Synapse importance and gradient magnitudes’).

In vivo performance requires selective formation, stabilisation, and elimination of LTP (Yang et al., 2009), raising the questions as to which biological mechanisms are able to provide this selectivity. Reducing updates at historically important synapses is one potential approach to determining which synapses should have their strengths adjusted and which should be stabilised. Adjusting learning rates based on synapse importance enables fast, stable learning (LeCun et al., 2002; Kingma and Ba, 2014; Khan et al., 2018; Aitchison, 2020; Martens, 2020; Jegminat et al., 2022).

For our purposes, the crucial point is that when training using an adaptive learning rate such as Equation 7, important synapses have higher root-mean-squared gradients, and hence lower learning rates. Here, we use a specific set of update rules which uses this adaptive learning rate (i.e. Adam: Kingma and Ba, 2014; Yang and Li, 2022). Thus, we can use learning rate as a proxy for importance, allowing us to obtain the predictions tested in Figure 5b which match Figure 5a/c.

The connection to Bayesian inference

Surprisingly, our experimental predictions obtained for optimised, heterogeneous synaptic variability (Figure 5) match those arising from Bayesian synapses presented in Figure 6 (i.e. synapses that use Bayes to infer their weights; Aitchison et al., 2021). Our first prediction was that lower variability implies a lower learning rate. The same prediction also arises if we consider Bayesian synapses. In particular, if variability and hence uncertainty is low, then a Bayesian synapse is very certain that it is close to the optimal value. In that case, new information should have less impact on the synaptic weight, and the learning rate should be lower. Our second prediction was that higher presynaptic firing rates imply less variability. Again, this arises in Bayesian synapses: Bayesian synapses should become more certain and less variable if the presynaptic cell fires more frequently. Every time the presynaptic cell fires, the synapse gets a feedback signal which gives a small amount of information about the right value for that synaptic weight. So the more times the presynaptic cell fires, the more information the synapse receives, and the more certain it becomes.

This match between observations for our energy-efficient synapses and previous work on Bayesian synapses led us to investigate potential connections between energy efficiency and Bayesian inference. Intuitively, there turns out to be a strong connection between synapse importance and uncertainty. Specifically, if a synapse is very important, then the performance cost changes dramatically when there are errors in that synaptic weight. That synapse therefore receives large gradients, and hence strong information about the correct value, rapidly reducing uncertainty.

To assess the connection between Bayesian posteriors and energy-efficient variability in more depth, we estimated and plotted the posterior variance against the optimised synaptic variability (Figure 7a) (see Materials and methods). We considered our four different biophysical mechanisms (values for ${\displaystyle \rho }$; Figure 7a, columns), and values for ${\displaystyle c}$ (Figure 7a, rows). In all cases, there was a clear correlation between the posterior and the optimised variability: synapses with larger posterior variance also had large optimised variance. To further assess this connection, we used the relation between the Hessian and posterior variance given by Equation 54c and the analytic result given in Appendix 5, ‘Analytic predictions for ${\displaystyle {\sigma }_i}$’ to plot the relationships between ${\displaystyle {\sigma }_i}$ and the posterior variability, ${\displaystyle {\sigma }_{\text{post}}}$, as a function of ${\displaystyle \rho }$ (Figure 7b) and as a function of ${\displaystyle c}$ (Figure 7c). Again, these plots show a clear correlation between synapse variance and posterior variance, though the relationship is far from perfect. For a perfect relationship, we would expect the lines in Figure 7b and c to all lie along the diagonal with slope equal to one. In contrast, these lines actually have a slope smaller than one, indicating that optimised variability is less heterogeneous than posterior variance (Figure 7b and c). Interestingly, the slope increases towards one as the associated ${\displaystyle \rho }$ is decreased, this suggests that synapse variability best approximates the posterior when ${\displaystyle \rho }$ is small.

Figure 7

Download asset Open asset

This strong, but not perfect, connection between the patterns of variability in Bayesian inference and energy-efficient networks motivated us to seek a formal connection between Bayesian and efficient synapses. As such, in the Appendix, we derive a theoretical connection between our overall performance cost and Bayesian inference (see Appendix 4, ‘Energy-efficient noise and variational Bayes for neural network weights’). Moreover, this connection is subsequently used to provide an explanation for why synapse variability aligns closer to posterior variance for small ${\displaystyle \rho }$ (see Equation 51), specifically, variational inference, a well-known procedure for performing (approximate) Bayesian inference in NNs (Hinton and van Camp, 1993; Graves, 2011; Blundell et al., 2015). Variational inference optimises the ‘evidence lower bound objective’ (ELBO) (Barber and Bishop, 1998; Jordan et al., 1999; Blei et al., 2017), which surprisingly turns out to resemble our performance cost. Specifically, the ELBO includes a term which encourages the entropy of the approximating posterior distribution (which could be interpreted as our noise distribution) to be larger. This resembles a reliability cost, as our reliability costs also encourage the noise distribution to be larger. Critically, the biological power-law reliability cost has a different form from the ideal, entropic reliability cost. However, we are able to derive a formal relationship: the biological power-law reliability costs bound the ideal entropic reliability cost. Remarkably, this implies that our overall cost (Equation 2) bounds the ELBO, so reducing our cost (Equation 2) tightens the ELBO bound and gives an improved guarantee on the quality of Bayesian inference.

Comparing the brain’s computational roles with associated energetic costs provides a useful means for deducing properties of efficient neurophysiology. Here, we applied this approach to PSP variability. We began by looking at the biophysical mechanisms of synaptic transmission, and how the energy costs for transmission might vary with synaptic reliability. We modified a standard ANN to incorporate unreliable synapses and trained this on a classification task using an objective that combined classification accuracy and an energetic cost on reliability. This led to a performance-reliability cost tradeoff and heterogeneous patterns of synapse variability that correlated with input rate and learning rate. We noted that these patterns of variability have been previously observed in data (see Figure 6). Remarkably, these are also the patterns of variability predicted by Bayesian synapses (Aitchison et al., 2021) (i.e. when distributions over synaptic weights correspond with the Bayesian posterior). Finally, we showed empirical and formal connections between the synaptic variability implied by Bayesian synapses and our performance-reliability cost tradeoff.

The reliability cost in terms of the synaptic variability (Equation 4) is a critical component of the numerical experiments we present here. While the precise form of the cost is inevitably uncertain, we attempted to mitigate the uncertainty by considering a wide range of functional forms for the reliability cost. In particular, we considered four biophysical mechanisms, corresponding to four power-law exponents, (${\displaystyle \rho =\frac{1}{2},\frac{2}{3},\frac{4}{3},2}$). Moreover, these different power-law costs already cover a reasonably wide-range of potential penalties and we would expect the results to hold for many other forms of reliability cost as the intuition behind the results ultimately relies merely on there being some penalty for increasing reliability.

The biophysical cost also includes a multiplicative factor, ${\displaystyle c}$, which sets the magnitude of the reliability cost. In fact, the patterns of variability exhibited in Figure 5 are preserved as ${\displaystyle c}$ is changed: this was demonstrated for values of ${\displaystyle c}$ which are 10 times larger and 10 times smaller, Appendix 6—figure 2. This multiplicative factor should be understood as being determined by the properties of the physics and chemistry underpinning synaptic dynamics, e.g., it could represent the quantity of ATP required by the metabolic costs of synaptic transmission (although this factor could vary, e.g. in different cell types).

Our ANNs used backpropagation to optimise the mean and variance of synaptic weights. While there are a number of schemes by which biological circuits might implement backpropagation (Whittington and Bogacz, 2017; Sacramento et al., 2018; Richards and Lillicrap, 2019), it is not yet clear whether backpropagation is implemented by the brain (see Lillicrap et al., 2020, for a review on the plausibility of propagation in the brain). Regardless, backpropagation is merely the route we used in our ANN setting to reach an energy-efficient configuration. The patterns we have observed are characteristic of an energy-efficient network and therefore should not depend on the learning rule that the brain uses to achieve energy efficiency.

Our results in ANNs used MNIST classification as an example of a task; this may appear somewhat artificial, but all brain areas ultimately do have a task: to maximise fitness (or reward as a proxy for fitness). Moreover, our results all ultimately arise from trading off biophysical reliability costs against the fact that if a synapse is important to performing a task, then variability in that synapse substantially impairs performance. Of course performance, in different brain areas, might mean reward, fitness, or some other measures. In contrast, if a synapse is unimportant, variability in that synapse impairs performance less. In all tasks there will be some synapses that are more, and some synapses that are less important, and our task, while relatively straightforward, captures this important property.

Our results have important implications for the understanding of Bayesian inference in synapses. In particular, we show that energy efficiency considerations give rise to two phenomena that are consistent with predictions outlined in previous work on Bayesian synapses (Aitchison et al., 2021). First, that normalised variability decreases for synapses with higher presynaptic firing rates. Second, that synaptic plasticity is higher for synapses with higher variability.

Specifically, these findings suggest that synapses connect their uncertainty in the value of the optimal synaptic weight (see Aitchison et al., 2021, for details) to variability. This is in essence a synaptic variant of the ‘sampling hypothesis’. Under the sampling hypothesis, neural activity is believed to represent a potential state of the world, and variability is believed to represent uncertainty (Hoyer and Hyvärinen, 2002; Knill and Pouget, 2004; Ma et al., 2006; Fiser et al., 2010; Berkes et al., 2011; Orbán et al., 2016; Aitchison and Lengyel, 2016; Haefner et al., 2016; Lange and Haefner, 2017; Shivkumar et al., 2018; Bondy et al., 2018; Echeveste et al., 2020; Festa et al., 2021; Lange et al., 2021; Lange and Haefner, 2022). This variability in neural activity, representing uncertainty in the state of the world, can then be read out by downstream circuits to inform behaviour. Here, we showed that a connection between synaptic uncertainty and variability can emerge simply as a consequence of maximising energy efficiency. This suggest that Bayesian synapses may emerge without any necessity for specific synaptic biophysical implementations of Bayesian inference.

Importantly though, while the brain might use synaptic noise for Bayesian computation, these results are also consistent with an alternative interpretation: that the brain is not Bayesian, it just looks Bayesian because it is energy efficient. To distinguish between these two interpretations, we ultimately need to know whether downstream brain areas exploit or ignore information about uncertainty that arises from synaptic variability.

The ANN simulations were run in PyTorch with feedforward, fully connected neural networks with two hidden layers of width 100. The input dimension of 784 corresponded to the number of pixels in the greyscale MNIST images of handwritten digits, while the output dimension of 10 corresponded to the number of classes. We used the reparameterisation trick to backpropagate with respect to the mean and variance of the weights, in particular, we set ${\displaystyle w_i={\mu }_i+{\sigma }_i\xi }$, where ${\displaystyle \xi \sim \text{Normal}(0,1)}$ (Kingma et al., 2015). MNIST classification was learned through optimisation of Gaussian parameters with respect to a cross-entropy loss in addition to reliability costs using minibatch gradient descent under Adam optimisation with a minibatch size of 20. To prevent negative values for the σs, they were reparameterised using a softplus function with argument ${\displaystyle {\varphi }_i}$, with ${\displaystyle {\sigma }_i=\text{softplus}({\varphi }_i)}$. The base learning rate in Equation 7 is ${\displaystyle {\eta }_{\text{base}}=5\times {10}^{-4}}$. The μ_is were initialised homogeneously across the network from ${\displaystyle \text{Uniform}(-0.1,0.1)}$ and the σ_is were initialised homogeneously across the network at ${\displaystyle {10}^{-4}}$. Hyperparameters were chosen via grid search on the validation dataset to enable smooth learning, high performance, and rapid convergence. In the objective ${\displaystyle {\mathcal{L}}_{\text{BI}}}$ used to train our simulations, we also add an L1 regularisation term over synaptic weights, ${\displaystyle \lambda |\mu {|}_1}$, where ${\displaystyle \lambda ={10}^{-4}}$.

Plots in Figure 2 present mappings from hyperparameter, ${\displaystyle c}$, to accuracy and ${\displaystyle \sigma }$. A different neural network was trained for each ${\displaystyle c}$, after 50 training epochs the average ${\displaystyle \sigma }$ across synapses was computed, and accuracy was evaluated on the test dataset. Plots in Figure 3 present mappings of this ${\displaystyle \sigma }$ against accuracy and reliability cost. The reliability cost was computed using fixed ${\displaystyle s=1}$ (see Equation _48).

To compute the Hessian in Figure 5 and elsewhere, we used the empirical Fisher information approximation (Fisher, 1922), ${\displaystyle H\approx g^2}$. This was evaluated by taking the average ${\displaystyle g^2}$ at ${\displaystyle w^{\ast }=\mu }$ over 10 epochs after full training for 50 epochs. The average learning rate ${\displaystyle \gamma |g{|}^{-1}}$ and the average input rate ${\displaystyle |x|}$ were also evaluated over 10 epochs following training. The data presented illustrate these variables with regard to the weights of the second hidden layer. We set hyperparameter ${\displaystyle s=0.001}$ (see Equation 47) in these simulations.

For geometric comparisons between the distribution over synapses and the Bayesian posterior presented in Figure 7 we used the analytic results in Appendix 5, ‘Analytic predictions for ${\displaystyle {\sigma }_i}$’. To estimate the posterior used in Figure 7, we optimised a factorised Gaussian approximation to the posterior over weights using variational inference and Bayes by backpropagation (Blundell et al., 2015). We then took ${\displaystyle {\sigma }_{post}}$ from two optimised weights. For the variance and slope comparisons between Bayesian and efficient synapses in Figure 7, we used the analytic results in Appendix 5, ‘Analytic predictions for ${\displaystyle {\sigma }_i}$’.

Source code used in simulations is available at: https://github.com/JamesMalkin/EfficientBayes copy archived at Malkin, 2024.

The current manuscript is a computational study, so any data generated is simulated data. The previously published dataset listed below and data from Ko et al., 2013 were used. All newly generated data is available at: https://github.com/JamesMalkin/EfficientBayes, copy archived at Malkin, 2024.

1. 1. Costa R
   2. Froemke R
   3. Sjöström P
   4. van Rossum M

   (2015) Dryad Digital Repository

   Data from: Unified pre- and postsynaptic long-term plasticity enables reliable and flexible learning.

   https://doi.org/10.5061/dryad.p286g

1. 1. Aitchison L
   2. Lengyel M

   (2016) The hamiltonian brain: efficient probabilistic inference with excitatory-inhibitory neural circuit dynamics

   PLOS Computational Biology 12:e1005186.

   https://doi.org/10.1371/journal.pcbi.1005186

   • PubMed
   • Google Scholar
2. Conference

   1. Aitchison L

   (2020)

   Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods

   Advances in Neural Information Processing Systems. pp. 18173–18182.

   • Google Scholar
3. 1. Aitchison L
   2. Jegminat J
   3. Menendez JA
   4. Pfister J-P
   5. Pouget A
   6. Latham PE

   (2021) Synaptic plasticity as Bayesian inference

   Nature Neuroscience 24:565–571.

   https://doi.org/10.1038/s41593-021-00809-5

   • PubMed
   • Google Scholar
4. 1. Attwell D
   2. Laughlin SB

   (2001) An energy budget for signaling in the grey matter of the brain

   Journal of Cerebral Blood Flow & Metabolism 21:1133–1145.

   https://doi.org/10.1097/00004647-200110000-00001

   • Google Scholar
5. 1. Barber D
   2. Bishop CM

   (1998)

   Ensemble learning in Bayesian neural networks

   Nato ASI Series F Computer and Systems Sciences 168:215–238.

   • Google Scholar
6. 1. Bellingham MC
   2. Lim R
   3. Walmsley B

   (1998) Developmental changes in EPSC quantal size and quantal content at a central glutamatergic synapse in rat

   The Journal of Physiology 511 (Pt 3):861–869.

   https://doi.org/10.1111/j.1469-7793.1998.861bg.x

   • PubMed
   • Google Scholar
7. 1. Berkes P
   2. Orbán G
   3. Lengyel M
   4. Fiser J

   (2011) Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment

   Science 331:83–87.

   https://doi.org/10.1126/science.1195870

   • PubMed
   • Google Scholar
8. 1. Blei DM
   2. Kucukelbir A
   3. McAuliffe JD

   (2017) Variational inference: a review for statisticians

   Journal of the American Statistical Association 112:859–877.

   https://doi.org/10.1080/01621459.2017.1285773

   • Google Scholar
9. Conference

   1. Blundell D
   2. Cornebise J
   3. Kavukcuoglu K
   4. Wierstra D

   (2015)

   Weight uncertainty in neural network

   International conference on machine learning PMLR. pp. 1613–1622.

   • Google Scholar
10. 1. Bondy AG
    2. Haefner RM
    3. Cumming BG

    (2018) Feedback determines the structure of correlated variability in primary visual cortex

    Nature Neuroscience 21:598–606.

    https://doi.org/10.1038/s41593-018-0089-1

    • PubMed
    • Google Scholar
11. 1. Boyd IA
    2. Martin AR

    (1956) The end-plate potential in mammalian muscle

    The Journal of Physiology 132:74–91.

    https://doi.org/10.1113/jphysiol.1956.sp005503

    • PubMed
    • Google Scholar
12. 1. Branco T
    2. Staras K

    (2009) The probability of neurotransmitter release: variability and feedback control at single synapses

    Nature Reviews Neuroscience 10:373–383.

    https://doi.org/10.1038/nrn2634

    • Google Scholar
13. 1. Bridgman PC

    (1999) Myosin Va movements in normal and dilute-lethal axons provide support for a dual filament motor complex

    The Journal of Cell Biology 146:1045–1060.

    https://doi.org/10.1083/jcb.146.5.1045

    • PubMed
    • Google Scholar
14. 1. Brock JA
    2. Thomazeau A
    3. Watanabe A
    4. Li SSY
    5. Sjöström PJ

    (2020) A practical guide to using CV analysis for determining the locus of synaptic plasticity

    Frontiers in Synaptic Neuroscience 12:11.

    https://doi.org/10.3389/fnsyn.2020.00011

    • PubMed
    • Google Scholar
15. 1. Chenouard N
    2. Xuan F
    3. Tsien RW

    (2020) Synaptic vesicle traffic is supported by transient actin filaments and regulated by PKA and NO

    Nature Communications 11:5318.

    https://doi.org/10.1038/s41467-020-19120-1

    • PubMed
    • Google Scholar
16. 1. Cingolani LA
    2. Goda Y

    (2008) Actin in action: the interplay between the actin cytoskeleton and synaptic efficacy

    Nature Reviews. Neuroscience 9:344–356.

    https://doi.org/10.1038/nrn2373

    • PubMed
    • Google Scholar
17. 1. Davis GW
    2. Müller M

    (2015) Homeostatic control of presynaptic neurotransmitter release

    Annual Review of Physiology 77:251–270.

    https://doi.org/10.1146/annurev-physiol-021014-071740

    • PubMed
    • Google Scholar
18. 1. Dobrunz LE
    2. Stevens CF

    (1997) Heterogeneity of release probability, facilitation, and depletion at central synapses

    Neuron 18:995–1008.

    https://doi.org/10.1016/s0896-6273(00)80338-4

    • PubMed
    • Google Scholar
19. 1. Echeveste R
    2. Aitchison L
    3. Hennequin G
    4. Lengyel M

    (2020) Cortical-like dynamics in recurrent circuits optimized for sampling-based probabilistic inference

    Nature Neuroscience 23:1138–1149.

    https://doi.org/10.1038/s41593-020-0671-1

    • PubMed
    • Google Scholar
20. 1. Engl E
    2. Attwell D

    (2015) Non-signalling energy use in the brain

    The Journal of Physiology 593:3417–3429.

    https://doi.org/10.1113/jphysiol.2014.282517

    • PubMed
    • Google Scholar
21. 1. Festa D
    2. Aschner A
    3. Davila A
    4. Kohn A
    5. Coen-Cagli R

    (2021) Neuronal variability reflects probabilistic inference tuned to natural image statistics

    Nature Communications 12:3635.

    https://doi.org/10.1038/s41467-021-23838-x

    • PubMed
    • Google Scholar
22. 1. Fiser J
    2. Berkes P
    3. Orbán G
    4. Lengyel M

    (2010) Statistically optimal perception and learning: from behavior to neural representations

    Trends in Cognitive Sciences 14:119–130.

    https://doi.org/10.1016/j.tics.2010.01.003

    • PubMed
    • Google Scholar
23. 1. Fisher RA

    (1922) On the mathematical foundations of theoretical statistics

    Philosophical Transactions of the Royal Society of London 222:309–368.

    https://doi.org/10.1098/rsta.1922.0009

    • Google Scholar
24. 1. Forti L
    2. Bossi M
    3. Bergamaschi A
    4. Villa A
    5. Malgaroli A

    (1997) Loose-patch recordings of single quanta at individual hippocampal synapses

    Nature 388:874–878.

    https://doi.org/10.1038/42251

    • PubMed
    • Google Scholar
25. 1. Fukushima K

    (1975) Cognitron: A self-organizing multilayered neural network

    Biological Cybernetics 20:121–136.

    https://doi.org/10.1007/BF00342633

    • Google Scholar
26. 1. Gentile JE
    2. Carrizales MG
    3. Koleske AJ

    (2022) Control of synapse structure and function by actin and its regulators

    Cells 11:603.

    https://doi.org/10.3390/cells11040603

    • PubMed
    • Google Scholar
27. 1. Goldman MS

    (2004) Enhancement of information transmission efficiency by synaptic failures

    Neural Computation 16:1137–1162.

    https://doi.org/10.1162/089976604773717568

    • Google Scholar
28. 1. Gramlich MW
    2. Klyachko VA

    (2017) Actin/Myosin-V- and activity-dependent inter-synaptic vesicle exchange in central neurons

    Cell Reports 18:2096–2104.

    https://doi.org/10.1016/j.celrep.2017.02.010

    • Google Scholar
29. Conference

    1. Graves A

    (2011)

    Practical variational inference for neural networks

    Advances in Neural Information Processing Systems 24 (NIPS 2011).

    • Google Scholar
30. 1. Haefner RM
    2. Berkes P
    3. Fiser J

    (2016) Perceptual decision-making as probabilistic inference by neural sampling

    Neuron 90:649–660.

    https://doi.org/10.1016/j.neuron.2016.03.020

    • Google Scholar
31. 1. Harris JJ
    2. Jolivet R
    3. Attwell D

    (2012) Synaptic energy use and supply

    Neuron 75:762–777.

    https://doi.org/10.1016/j.neuron.2012.08.019

    • Google Scholar
32. 1. Harris JJ
    2. Engl E
    3. Attwell D
    4. Jolivet RB

    (2019) Energy-efficient information transfer at thalamocortical synapses

    PLOS Computational Biology 15:e1007226.

    https://doi.org/10.1371/journal.pcbi.1007226

    • PubMed
    • Google Scholar
33. 1. Heidelberger R
    2. Heinemann C
    3. Neher E
    4. Matthews G

    (1994) Calcium dependence of the rate of exocytosis in a synaptic terminal

    Nature 371:513–515.

    https://doi.org/10.1038/371513a0

    • Google Scholar
34. Conference

    1. Hinton GE
    2. van Camp D

    (1993) Keeping the neural networks simple by minimizing the description length of the weights

    Proceedings of COLT-93. pp. 5–13.

    https://doi.org/10.1145/168304.168306

    • Google Scholar
35. 1. Holler S
    2. Köstinger G
    3. Martin KAC
    4. Schuhknecht GFP
    5. Stratford KJ

    (2021) Structure and function of a neocortical synapse

    Nature 591:111–116.

    https://doi.org/10.1038/s41586-020-03134-2

    • Google Scholar
36. Conference

    1. Hoyer P
    2. Hyvärinen A

    (2002)

    Interpreting neural response variability as Monte Carlo sampling of the posterior

    Advances in Neural Information Processing Systems.

    • Google Scholar
37. 1. Jegminat J
    2. Surace SC
    3. Pfister J-P

    (2022) Learning as filtering: Implications for spike-based plasticity

    PLOS Computational Biology 18:e1009721.

    https://doi.org/10.1371/journal.pcbi.1009721

    • PubMed
    • Google Scholar
38. 1. Jordan MI
    2. Ghahramani Z
    3. Jaakkola TS
    4. Saul LK

    (1999) An introduction to variational methods for graphical models

    Machine Learning 37:183–233.

    https://doi.org/10.1023/A:1007665907178

    • Google Scholar
39. 1. Karbowski J

    (2019) Metabolic constraints on synaptic learning and memory

    Journal of Neurophysiology 122:1473–1490.

    https://doi.org/10.1152/jn.00092.2019

    • PubMed
    • Google Scholar
40. 1. Karunanithi S
    2. Marin L
    3. Wong K
    4. Atwood HL

    (2002) Quantal size and variation determined by vesicle size in normal and mutant Drosophila glutamatergic synapses

    The Journal of Neuroscience 22:10267–10276.

    https://doi.org/10.1523/JNEUROSCI.22-23-10267.2002

    • PubMed
    • Google Scholar
41. 1. Katz B
    2. Miledi R

    (1965) The measurement of synaptic delay, and the time course of acetylcholine release at the neuromuscular junction

    Proceedings of the Royal Society of London. Series B, Biological Sciences 161:483–495.

    https://doi.org/10.1098/rspb.1965.0016

    • PubMed
    • Google Scholar
42. Conference

    1. Khan M
    2. Nielsen D
    3. Tangkaratt V
    4. Lin W
    5. Gal Y
    6. Srivastava A

    (2018)

    Fast and scalable bayesian deep learning by weight-perturbation in adam

    International conference on machine learning PMLR. pp. 2611–2620.

    • Google Scholar
43. Preprint

    1. Kingma DP
    2. Ba J

    (2014) Adam: a method for stochastic optimization

    arXiv.

    https://arxiv.org/abs/1412.6980

    • Google Scholar
44. Conference

    1. Kingma DP
    2. Salimans T
    3. Welling M

    (2015)

    Variational dropout and the local reparameterization trick

    Advances in Neural Information Processing Systems.

    • Google Scholar
45. 1. Knill DC
    2. Pouget A

    (2004) The Bayesian brain: the role of uncertainty in neural coding and computation

    TRENDS in Neurosciences 27:712–719.

    https://doi.org/10.1016/j.tins.2004.10.007

    • PubMed
    • Google Scholar
46. 1. Ko H
    2. Cossell L
    3. Baragli C
    4. Antolik J
    5. Clopath C
    6. Hofer SB
    7. Mrsic-Flogel TD

    (2013) The emergence of functional microcircuits in visual cortex

    Nature 496:96–100.

    https://doi.org/10.1038/nature12015

    • PubMed
    • Google Scholar
47. 1. Lange RD
    2. Haefner RM

    (2017) Characterizing and interpreting the influence of internal variables on sensory activity

    Current Opinion in Neurobiology 46:84–89.

    https://doi.org/10.1016/j.conb.2017.07.006

    • PubMed
    • Google Scholar
48. 1. Lange RD
    2. Chattoraj A
    3. Beck JM
    4. Yates JL
    5. Haefner RM

    (2021) A confirmation bias in perceptual decision-making due to hierarchical approximate inference

    PLOS Computational Biology 17:e1009517.

    https://doi.org/10.1371/journal.pcbi.1009517

    • PubMed
    • Google Scholar
49. 1. Lange RD
    2. Haefner RM

    (2022) Task-induced neural covariability as a signature of approximate Bayesian learning and inference

    PLOS Computational Biology 18:e1009557.

    https://doi.org/10.1371/journal.pcbi.1009557

    • PubMed
    • Google Scholar
50. 1. Laughlin SB
    2. de Ruyter van Steveninck RR
    3. Anderson JC

    (1998) The metabolic cost of neural information

    Nature Neuroscience 1:36–41.

    https://doi.org/10.1038/236

    • PubMed
    • Google Scholar
51. Book

    1. LeCun Y
    2. Bottou L
    3. Orr GB

    (2002) Efficient backprop

    In: LeCun Y, editors. Neural Networks: Tricks of the Trade. Springer. pp. 9–50.

    https://doi.org/10.1007/3-540-49430-8_2

    • Google Scholar
52. 1. Levy WB
    2. Baxter RA

    (2002) Energy-efficient neuronal computation via quantal synaptic failures

    The Journal of Neuroscience 22:4746–4755.

    https://doi.org/10.1523/JNEUROSCI.22-11-04746.2002

    • PubMed
    • Google Scholar
53. 1. Lillicrap TP
    2. Santoro A
    3. Marris L
    4. Akerman CJ
    5. Hinton G

    (2020) Backpropagation and the brain

    Nature Reviews. Neuroscience 21:335–346.

    https://doi.org/10.1038/s41583-020-0277-3

    • PubMed
    • Google Scholar
54. 1. Lisman JE
    2. Harris KM

    (1993) Quantal analysis and synaptic anatomy--integrating two views of hippocampal plasticity

    Trends in Neurosciences 16:141–147.

    https://doi.org/10.1016/0166-2236(93)90122-3

    • PubMed
    • Google Scholar
55. 1. Ma WJ
    2. Beck JM
    3. Latham PE
    4. Pouget A

    (2006) Bayesian inference with probabilistic population codes

    Nature Neuroscience 9:1432–1438.

    https://doi.org/10.1038/nn1790

    • PubMed
    • Google Scholar
56. 1. MacKay DJC

    (1992a) A practical bayesian framework for backpropagation networks

    Neural Computation 4:448–472.

    https://doi.org/10.1162/neco.1992.4.3.448

    • Google Scholar
57. 1. MacKay DJC

    (1992b) The evidence framework applied to classification networks

    Neural Computation 4:720–736.

    https://doi.org/10.1162/neco.1992.4.5.720

    • Google Scholar
58. Software

    1. Malkin J

    (2024) Efficient bayes, version swh:1:rev:a341bff3e9241f958f351cb460b30047ec60d3a3

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:627379d202b9a70e656b8b3a6e7f75c5a4d2b55a;origin=https://github.com/JamesMalkin/EfficientBayes;visit=swh:1:snp:76bcd130a9aef445414073e0a5e8fc9e3f15e557;anchor=swh:1:rev:a341bff3e9241f958f351cb460b30047ec60d3a3
59. 1. Martens J

    (2020)

    New insights and perspectives on the natural gradient method

    The Journal of Machine Learning Research 21:5776–5851.

    • Google Scholar
60. Book

    1. Murphy KP

    (2012)

    Machine Learning: A Probabilistic Perspective

    MIT press.

    • Google Scholar
61. 1. Murthy VN
    2. Sejnowski TJ
    3. Stevens CF

    (1997) Heterogeneous release properties of visualized individual hippocampal synapses

    Neuron 18:599–612.

    https://doi.org/10.1016/s0896-6273(00)80301-3

    • PubMed
    • Google Scholar
62. 1. Orbán G
    2. Berkes P
    3. Fiser J
    4. Lengyel M

    (2016) Neural variability and sampling-based probabilistic representations in the visual cortex

    Neuron 92:530–543.

    https://doi.org/10.1016/j.neuron.2016.09.038

    • PubMed
    • Google Scholar
63. 1. Paulsen O
    2. Heggelund P

    (1994) The quantal size at retinogeniculate synapses determined from spontaneous and evoked EPSCs in guinea‐pig thalamic slices

    The Journal of Physiology 480:505–511.

    https://doi.org/10.1113/jphysiol.1994.sp020379

    • PubMed
    • Google Scholar
64. 1. Paulsen O
    2. Heggelund P

    (1996) Quantal properties of spontaneous EPSCs in neurones of the guinea‐pig dorsal lateral geniculate nucleus

    The Journal of Physiology 496:759–772.

    https://doi.org/10.1113/jphysiol.1996.sp021725

    • PubMed
    • Google Scholar
65. 1. Pulido C
    2. Ryan TA

    (2021) Synaptic vesicle pools are a major hidden resting metabolic burden of nerve terminals

    Science Advances 7:eabi9027.

    https://doi.org/10.1126/sciadv.abi9027

    • PubMed
    • Google Scholar
66. 1. Purdon AD
    2. Rosenberger TA
    3. Shetty HU
    4. Rapoport SI

    (2002) Energy consumption by phospholipid metabolism in mammalian brain

    Neurochemical Research 27:1641–1647.

    https://doi.org/10.1023/a:1021635027211

    • PubMed
    • Google Scholar
67. 1. Raghavachari S
    2. Lisman JE

    (2004) Properties of quantal transmission at CA1 synapses

    Journal of Neurophysiology 92:2456–2467.

    https://doi.org/10.1152/jn.00258.2004

    • PubMed
    • Google Scholar
68. 1. Richards BA
    2. Lillicrap TP

    (2019) Dendritic solutions to the credit assignment problem

    Current Opinion in Neurobiology 54:28–36.

    https://doi.org/10.1016/j.conb.2018.08.003

    • PubMed
    • Google Scholar
69. 1. Rosset S
    2. Zhu J

    (2006) Sparse, flexible and efficient modeling using L 1 regularization

    Feature Extraction: Foundations and Applications 01:375–394.

    https://doi.org/10.1007/978-3-540-35488-8_17

    • Google Scholar
70. 1. Sacramento J
    2. Wichert A
    3. van Rossum MCW

    (2015) Energy efficient sparse connectivity from imbalanced synaptic plasticity rules

    PLOS Computational Biology 11:e1004265.

    https://doi.org/10.1371/journal.pcbi.1004265

    • PubMed
    • Google Scholar
71. Conference

    1. Sacramento J
    2. Ponte Costa R
    3. Bengio Y

    (2018)

    Dendritic cortical microcircuits approximate the backpropagation algorithm

    Advances in Neural Information Processing Systems.

    • Google Scholar
72. 1. Sakaba T
    2. Neher E

    (2001) Quantitative relationship between transmitter release and calcium current at the calyx of held synapse

    The Journal of Neuroscience 21:462–476.

    https://doi.org/10.1523/JNEUROSCI.21-02-00462.2001

    • PubMed
    • Google Scholar
73. 1. Schug S
    2. Benzing F
    3. Steger A

    (2021) Presynaptic stochasticity improves energy efficiency and helps alleviate the stability-plasticity dilemma

    eLife 10:e69884.

    https://doi.org/10.7554/eLife.69884

    • PubMed
    • Google Scholar
74. 1. Shannon CE

    (1948) A mathematical theory of communication

    Bell System Technical Journal 27:379–423.

    https://doi.org/10.1002/j.1538-7305.1948.tb01338.x

    • Google Scholar
75. Conference

    1. Shivkumar S
    2. Lange R
    3. Chattoraj A

    (2018)

    A probabilistic population code based on neural samples

    Advances in Neural Information Processing Systems.

    • Google Scholar
76. 1. Silver RA

    (2003) Estimation of nonuniform quantal parameters with multiple-probability fluctuation analysis: theory, application and limitations

    Journal of Neuroscience Methods 130:127–141.

    https://doi.org/10.1016/j.jneumeth.2003.09.030

    • PubMed
    • Google Scholar
77. 1. Sjöström PJ
    2. Turrigiano GG
    3. Nelson SB

    (2001) Rate, timing, and cooperativity jointly determine cortical synaptic plasticity

    Neuron 32:1149–1164.

    https://doi.org/10.1016/s0896-6273(01)00542-6

    • PubMed
    • Google Scholar
78. 1. Turrigiano GG
    2. Leslie KR
    3. Desai NS
    4. Rutherford LC
    5. Nelson SB

    (1998) Activity-dependent scaling of quantal amplitude in neocortical neurons

    Nature 391:892–896.

    https://doi.org/10.1038/36103

    • PubMed
    • Google Scholar
79. 1. Turrigiano GG
    2. Nelson SB

    (2004) Homeostatic plasticity in the developing nervous system

    Nature Reviews. Neuroscience 5:97–107.

    https://doi.org/10.1038/nrn1327

    • PubMed
    • Google Scholar
80. 1. Whittington JCR
    2. Bogacz R

    (2017) An approximation of the error backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity

    Neural Computation 29:1229–1262.

    https://doi.org/10.1162/NECO_a_00949

    • PubMed
    • Google Scholar
81. 1. Yang G
    2. Pan F
    3. Gan W-B

    (2009) Stably maintained dendritic spines are associated with lifelong memories

    Nature 462:920–924.

    https://doi.org/10.1038/nature08577

    • PubMed
    • Google Scholar
82. Preprint

    1. Yang Y
    2. Li P

    (2022) Synaptic Dynamics Realize First-Order Adaptive Learning and Weight Symmetry

    arXiv.

    https://arxiv.org/abs/2212.09440

    • Google Scholar
83. 1. Yu L
    2. Zhang C
    3. Liu L
    4. Yu Y

    (2016) Energy-efficient population coding constrains network size of a neuronal array system

    Scientific Reports 6:19369.

    https://doi.org/10.1038/srep19369

    • PubMed
    • Google Scholar

Author details

1. James Malkin

   Faculty of Engineering, University of Bristol, Bristol, United Kingdom

   Contribution

   Conceptualization, Formal analysis, Investigation, Visualization, Methodology, Writing - original draft, Writing – review and editing

   For correspondence

   james.malkin@bristol.ac.uk

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0009-0001-7473-1442

2. Cian O'Donnell

   1. Faculty of Engineering, University of Bristol, Bristol, United Kingdom
   2. Intelligent Systems Research Centre, School of Computing, Engineering, and Intelligent Systems, Ulster University, Derry/Londonderry, United Kingdom

   Contribution

   Supervision, Funding acquisition, Visualization, Writing – review and editing

   Contributed equally with

   Conor J Houghton

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2031-9177

3. Conor J Houghton

   Faculty of Engineering, University of Bristol, Bristol, United Kingdom

   Contribution

   Formal analysis, Supervision, Visualization, Writing – review and editing

   Contributed equally with

   Cian O'Donnell

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-5017-9473

4. Laurence Aitchison

   Faculty of Engineering, University of Bristol, Bristol, United Kingdom

   Contribution

   Conceptualization, Formal analysis, Supervision, Visualization, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

• 874

  views

• 72

  downloads

• 3

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.