Introduction

The synapse is the major site of inter-cellular communication in the brain. The amplitude of synaptic post-synaptic potentials (PSPs) are usually highly variable or stochastic. This variability arises primarily presynaptically: the release of neurotransmitter from presynaptically-housed vesicles into the synaptic cleft has variable release probabilities and variable quantal sizes (Lisman and Harris, 1993; Branco and Staras, 2009; Brock et al., 2020). Unreliable synaptic transmission seems puzzling, especially in light of evidence for low-noise, almost failure-free transmission at some synapses (Paulsen and Heggelund, 1994, 1996; Bellingham et al., 1998). Moreover, the degree to which a synapse is unreliable does not just vary from one synapse type to another, there is also an heterogeneity of precision amongst synapses of the same type (Murthy et al., 1997; Dobrunz and Stevens, 1997). Given that there is capacity for more precise transmission, why is this capacity not used in more synapses?

Unreliable transmission degrades accuracy but Laughlin et al. (1998) showed that the synaptic connection from a photoreceptor to a retinal large monopolar cell could increase its precision by increasing the number of synapses, averaging the noise away, but this comes at the cost of extra energy per bit of information transmitted. Moreover, Levy and Baxter (2002) demonstrated that there is a value for the precision which optimises the energy cost of information transmission. In this paper, we explore this notion of a performance-energy tradeoff.

However, it is important to consider precision and energy cost in the context of neuronal computation; the brain does not simply transfer information from neuron to neuron, it performs computation through the interaction between neurons. However, models outlining a synaptic energy-performance tradeoff, (Laughlin et al., 1998; Levy and Baxter, 2002; Goldman, 2004; Harris et al., 2012, 2019; Karbowski, 2019), predominantly consider information transmission between just two neurons and the corresponding information-theoretic view treats the synapse as an isolated conduit of information (Shannon, 1948). In contrast, in reality, a single synapse is just one unit of the computational machinery of the brain. As such, the performance of an individual synapse needs to be considered in the context of circuit performance. To perform computation in an energy-efficient way the circuit as a whole needs to allocate resources across different synapses to optimise the overall energy cost of computation (Yu et al., 2016; Schug et al., 2021).

Here, we consider the consequences of a tradeoff between network performance and energetic reliability costs that depend explicitly upon synapse precision. We estimate the energy costs associated with precision by considering the biological mechanisms underpinning synaptic transmission. By including these costs in a neural network designed to perform a classification task, we observe a heterogeneity in synaptic precision and find that this “allocation” of precision is related to signatures of synapse “importance”, which can be understood formally on the grounds of Bayesian inference.

Results

We proposed energetic costs for reliable synaptic transmission and then measured their consequences in an artificial neural network.

Biophysical costs

Here, we seek to understand the biophysical energetic costs of synaptic transmission, and how those costs relate to the reliability of transmission (Fig. 1a). We start by considering the underlying mechanisms of synaptic transmission. In particular, synaptic transmission begins with the arrival of a spike at the axon terminal. This triggers a large influx of calcium ions into the axon terminal. The increase in calcium concentration causes the release of neurotransmitter-filled vesicles docked at axonal release sites. The neurotransmitter diffuses across the synaptic cleft to the postsynaptic dendritic membrane. There, the neurotransmitter binds with ligand-gated ion channels causing a change in voltage, i.e. a postsynaptic potential. This process is often quantified using the Katz and Miledi (1965) quantal model of neurotransmitter release. Under this model, for each connection between two cells, there are n docked, readily releasable vesicles. When the presynaptic cell spikes, each docked vesicle releases with probability p and each released vesicle causes a postsynaptic potential of size q. Thus, the mean, μ, and variance, σ², of the PSP can be written (see Fig. 1b),

Figure 1.

We considered four biophysical costs associated with improving the reliability of synaptic transmission, while keeping the mean fixed, and derived the associated scaling of the energetic cost with PSP variance.

Calcium efflux.

Reliability is higher when the probability of vesicle release, p, is higher. As vesicle release is triggered by an increase in intracellular calcium, greater calcium concentration implies higher release probability. However, increased calcium concentration implies higher energetic costs. In particular, calcium that enters the synaptic bouton will subsequently need to be pumped out. We take the cost of pumping out calcium ions to be proportional to the calcium concentration, and take the relationship between release probability and calcium concentration to be governed by a Hill Equation, following Sakaba and Neher (2001). The resulting relationship between energetic costs and reliability is cost ∝ σ^−1/2 (Fig. 1c (I); see Appendix - Reliability costs for further details).

Vesicle membrane surface area.

There may also be energetic costs associated with producing and maintaining a large amount of vesicle membrane. Purdon et al. (2002) argues that phospholipid metabolism may take a considerable proportion of the brain’s energy budget. Additionally, costs associated with membrane surface area may arise because of leakage of hydrogen ions across vesicles (Pulido and Ryan, 2021). Importantly, a cost for vesicle surface area is implicitly a cost on reliability. In particular, we could obtain highly reliable synaptic release by releasing many small vesicles, such that stochasticity in individual vesicle release events averages out. However, the resulting many small vesicles have a far larger surface area than a single large vesicle, with the same mean PSP. Thus, a cost on surface area implies a relationship between energetic costs and reliability; in particular cost ∝ σ^−2/3 (Fig. 1c (II); see Appendix - Reliability costs for further details).

Actin.

Another cost for small but numerous vesicles arises from a demand for structural organisation of the vesicles pool by filaments such as actin (Cingolani and Goda, 2008; Gentile et al., 2022). Critically, there are physical limits to the number of vesicles that can be attached to an actin filament of a given length. In particular, if vesicles are smaller we can attach more vesicles to a given length of actin, but at the same time, the total vesicle volume (and hence the total quantity of neurotransmitter) will be smaller (Fig. 1c (III)). A fixed cost per unit length of actin thus implies a relationship between energetic costs and reliability of, cost ∝ σ^−4/3 (see Appendix - Reliability costs).

Trafficking.

A final class of costs is proportional to the number of vesicles (Laughlin et al., 1998). One potential biophysical mechanism by which such a cost might emerge is from active transport of vesicles along actin filaments or microtubles to release sites (Chenouard et al., 2020). In particular, vesicles are transported by ATP-dependent myosin-V motors (Bridgman, 1999), so more vesicles require a greater energetic cost for trafficking. Any such cost proportional to the number of vesicles gives rise to a relationship between energetic cost and PSP variance of the form, cost ∞ σ⁻² (Fig. 1c (IV); see Appendix - Reliability costs).

Costs related to PSP mean/magnitude

While costs on precision are the central focus of this paper, it is certainly the case that other costs relating to the mean PSP magnitude constitute a major cost of synaptic transmission. For example, high amplitude PSPs require a large quantity of neurotransmitter, high probability of vesicle release, and a large number of post-synaptic receptors (Attwell and Laughlin, 2001). These can be formalised as costs on the PSP mean, μ, and can additionally be related to L1 weight decay in a machine learning context (Rosset and Zhu, 2006; Sacramento et al., 2015).

Reliability costs in artificial neural networks

Next, we sought to understand how these biophysical energetic costs of reliability might give rise to patterns of variability in a trained neural network. Specifically, we trained artificial neural networks (ANNs) using an objective that embodied a tradeoff between performance and reliability costs,

The “performance cost" term measures the network’s performance on the task, for instance in our classification tasks we used the usual cross-entropy cost. The “magnitude cost” term captures costs that depend on the PSP mean, while the “reliability cost” term captures costs that depend on the PSP precision. In particular,

Here, i indexes synapses, and recall that σ_i is the standard deviation of the ith synapse. The multiplier c in the reliability cost determines the strength of the reliability cost relative to the performance cost. Small values for c imply that the reliability cost term is less important, permitting precise transmission and higher performance. Large values for c give greater importance to the reliability cost encouraging energy efficiency by allowing higher levels of synaptic noise, causing detriment to performance (see Fig. 2).

Figure 2.

We trained fully-connected, rate-based neural network to classify MNIST digits. Stochastic synaptic PSPs were sampled from a Normal distribution,

where, recall, μ_i is the PSP mean and is the PSP variance for the ith synapse. The output firing rate was given by,

Here, can be understood as the somatic membrane potential, and f represents the relationship between somatic membrane potential and firing rate; we used ReLU (Fukushima, 1975). We optimised network parameters μ_i and σ_i using Adam (Kingma and Ba, 2014) (see Methods for details on architecture and hyperparameters).

The tradeoff between accuracy and reliability costs in trained networks

Next we sought to understand how the tradeoff between accuracy and reliability cost manifests in trained networks. Perhaps the critical parameter in the objective, (Eq. 2 and Eq. 4) was c, which controlled the importance of the reliability cost relative to the performance cost. We trained networks with a variety of different values of c, and with four values for ρ motivated by the biophysical costs (the different columns). As expected, we found that as c increased, performance fell (Fig. 2a) and the average synaptic standard deviation increased (Fig. 2b). Importantly, we considered two different settings. First, we considered an homogeneous noise setting, where σ_i is optimised but kept the same across all synapses (grey lines). Second, we considered an heterogeneous noise setting, where σ_i is allowed to vary across synapses, and is optimised on a per-synapse basis. We found that heterogeneous noise (i.e. allowing the noise to vary on a per-synapse basis) improved accuracy considerably for a fixed value of c, but only reduced the average noise slightly.

The findings in Fig. 2 imply a tradeoff between accuracy and average noise level, σ, as we change c. If we explicitly plot the accuracy against the noise level using the data from Fig. 2, we see that as the synaptic noise level increases, the accuracy decreases (Fig. 3a). Further, the synaptic noise level is associated with a reliability cost (Fig. 3b), and this relationship changes in the different columns as they use different values of ρ associated with different biological mechanisms that might give rise to the dominant biophysical reliability cost. Thus, there is also a relationship between accuracy and reliability costs (Fig. 3c), with accuracy increasing as we allow the system to invest more energy in becoming more reliable, which implies a higher reliability cost. Again, we plotted both the homogeneous (grey lines) and heterogeneous noise cases (green lines). We found that heterogeneous noise allowed for considerably improved accuracy at a given average noise standard deviation or a given reliability cost.

Figure 3.

Energy-efficient patterns of synapse variability

We found that the heterogeneous noise setting, where we individually optimise synaptic noise on a per-synapse basis, performed considerably better than the homogeneous noise setting (Fig. 3). This raised an important question: how does the network achieve such large improvements by optimising the noise levels on a per-synapse basis? We hypothesised that the system invests a lot of energy in improving the reliability for “important” synapses, i.e. synapses whose weights have a large impact on predictions and accuracy (Fig. 4a). Conversely, the system allows unimportant synapses to have high variability, which reduces reliability costs (Fig. 4b). To get further intuition, we compared both ω₁ and ω₂ on the same plot (Fig. 4c). Specifically, we put the important synapse, ω₁ from Fig. 4a, on the horizontal axis, and the unimportant synapse, ω₂ from Fig. 4b, on the vertical axis. In Fig. 4c, the relative importance of the synapse is now depicted by how the cost increases as we move away from the optimal value of the weight. Specifically, the cost increases rapidly as we move away from the optimal value of ω₁, but increases much more slowly as we move away from the optimal value of ω₂. Now, consider deviations in the synaptic weight driven by homogeneous synaptic variability (Fig. 4c left, grey points). Many of these points have poor performance (i.e. a high performance cost), due to relatively high noise on the important synapse (i.e. ω₁). Next, consider deviations in the synaptic weight driven by heterogeneous, optimised variability (Fig. 4c left, green points). Critically, optimising synaptic noise reduces variability for the important synapse, and that reduces the average performance cost by eliminating large deviations on the important synapse. Thus, for the same overall reliability cost, heterogeneous, optimised variability can achieve much lower performance costs, and hence much lower overall costs than homogeneous variability (Fig. 4d).

Figure 4.

To investigate experimental predictions arising from optimised, heterogeneous variability, we needed a way to formally assess the “importance” of synapses. We used the “curvature” of the performance cost: namely the degree to which small deviations in the weights from their optimal values will degrade performance. If the curvature is large (Fig. 4a), then small deviations in the weights, e.g. those caused by noise, can drastically reduce performance. In contrast, if the curvature is smaller (Fig. 4b), then small deviations in the weights cause a much smaller reduction in performance. As a formal measure of the curvature of the objective, we used the Hessian matrix, H. This describes the shape of the objective as a function of the synaptic weights, the ω_is: specifically, it is the matrix of second derivatives of the objective, with respect to the weights, and measures the local curvature of objective. We were interested in the diagonal elements, H_ii; the second derivatives of the objective with respect to ω_i.

We began by looking at how the optimised synaptic noise varied with synapse importance, as measured by the curvature or, more formally, the Hessian (Fig. 5a). We found that as the importance of the synapse increased, the optimised noise level decreased. These patterns of synapse variability make sense because noise is more detrimental at important synapses and so it is worth investing energy to reduce the noise in those synapses.

Figure 5.

However, this relationship (Fig. 5a) between the importance of a synapse and the synaptic variability is not experimentally testable, as we are not able to directly measure synapse importance. That said, we are able to obtain two testable predictions. First, the input rate in our simulations was negatively correlated with optimised synaptic variability (Fig. 5b). Second, the optimised synaptic variability was larger for synapses with larger learning rates (Fig. 5c). Critically, both of these patterns have been observed in experimental data. The relationship between input firing rate and synaptic variability was first observed by Aitchison et al. (2021) using data from Ko et al. (2013) (Fig. 6a). The relationship between learning rate and synaptic variability was first observed by Schug et al. (2021), using data from Sjöström et al. (2003) as processed by Costa et al. (2017) (Fig. 6b).

Figure 6.

To understand why these patterns of variability emerge in our simulations and in data, we need to understand the connection between synapse importance, synaptic inputs (Fig. 5b, Fig. 6b) and synaptic learning (Fig. 5c, Fig. 6a). Perhaps the easiest connection is between the synapse importance and the input firing rate. If the input cell never fires, then the synaptic weight cannot affect the network output, and the synapse has zero importance (and also zero Hessian (see Appendix - High input rates and high precision at important synapses)). This would suggest a tendency for synapses with higher input firing rates to be more important, and hence to have lower variability. This pattern is indeed borne out in our simulations (Fig. 5b; also see Supplementary - Appendix 6—Fig. 1), though of course there is a considerable amount of noise: there are a few important synapses with low input rates, and vice-versa.

Next, we consider the connection between learning rate and synapse importance. To understand this connection, we need to choose a specific scheme for modulating the learning rate as a function of the inputs. Modern, state-of-the-art, update rules for artificial neural networks often use an adaptive learning rate. These adaptive learning rates, η_i, (including the most common such as Adam and variants) almost always use a normalising learning rate which decreases in response to high incoming gradients,

Specifically, the local learning rate for the ith synapse, η_i, is usually a base learning rate, η_base, divided by the root-meansquare gradient at this synapse . Critically, the root-mean-square gradient turns out to be strongly related to synapse importance. Intuitively, important synapses with greater impact on network predictions will have larger gradients Appendix - Synapse importance and gradient magnitudes.

In-vivo performance requires selective formation, stabilisation and elimination of long term plasticity (LTP) (Yang et al., 2009), raising the questions as to which biological mechanisms are able to provide this selectivity. Reducing updates at historically important synapses is one potential approach to determining which synapses should have their strengths adjusted and which should be stabilised. Adjusting learning rates based on synapse importance enables fast, stable learning (LeCun et al., 2002; Kingma and Ba, 2014; Khan et al., 2018; Aitchison, 2020; Martens, 2020).

For our purposes, the crucial point is that when training using an adaptive learning rate such as Eq. 7, important synapses have higher root-mean-squared gradients, and hence lower learning rates. Here we use a specific set of update rules which uses this adaptive learning rate (i.e. Adam (Kingma and Ba, 2014; Yang and Li, 2022)). Thus, we can use learning rate as a proxy for importance, allowing us to obtain the predictions tested in Fig. 5b which match Fig. 5a/c.

The connection to Bayesian inference

Surprisingly, our experimental predictions obtained for optimised, heterogeneous synaptic variability (Fig. 5,6) match those arising from Bayesian synapses (i.e. synapses that use Bayes to infer their weights (Aitchison et al., 2021)). Our first prediction was that lower variability implies a lower learning rate. The same prediction also arises if we consider Bayesian synapses. In particular, if variability and hence uncertainty is low, then a Bayesian synapse is very certain that it is close to the optimal value. In that case, new information should have less impact on the synaptic weight, and the learning rate should be lower. Our second prediction was that higher presynaptic firing rates imply less variability. Again, this arises in Bayesian synapses: Bayesian synapses should become more certain and less variable if the presynaptic cell fires more frequently. Every time the presynaptic cell fires, the synapse gets a feedback signal which gives a small amount of information about the right value for that synaptic weight. So the more times the presynaptic cell fires, the more information the synapse receives, and the more certain it becomes.

This match between observations for our energy-efficient synapses and previous work on Bayesian synapses led us to investigate potential connections between energy efficiency and Bayesian inference. Intuitively, there turns out to be a strong connection between synapse importance and uncertainty. Specifically, if a synapse is very important, then the performance cost changes dramatically when there are errors in that synaptic weight. That synapse therefore receives large gradients, and hence strong information about the correct value, rapidly reducing uncertainty.

To assess the connection between Bayesian posteriors and energy efficient variability in more depth, we plotted the posterior variance against the optimised synaptic variability (Fig. 7a). We considered our four different biophysical mechanisms (values for ρ; Fig. 7a, columns), and values for c (Fig. 7a, rows). In all cases, there was a clear correlation between the posterior and the optimised variability: directions with larger posterior variance also had large optimised variability. To more formally assess this connection, we derived the relationships between the optimised noise, σ_i and the posterior variable, σ_post as a function of ρ (Fig. 7b;) and as a function of c (Fig. 7c). Again, these plots show a clear correlation between optimised variability and posterior variance; though the relationship is far from perfect. For a perfect relationship, we would expect the lines in Fig. 7bc to all lie along the diagonal. In contrast, these lines actually have a slope smaller than one, indicating that optimised variability is less heterogeneous than posterior variance (Fig. 7bc).

Figure 7.

Finally, in the Appendix, we derive a formal connection between our overall performance cost and Bayesian inference. Specifically, variational inference, a well-known procedure for performing (approximate) Bayesian inference in NNs (Hinton and van Camp, 1993; Graves, 2011; Blundell et al., 2015). Variational inference optimises the “evidence lower bound objective” (ELBO) (Barber and Bishop, 1998; Jordan et al., 1999; Blei et al., 2017), which surprisingly turns out to resemble our performance cost. Specifically, the ELBO includes a term which encourages the entropy of the approximating posterior distribution (which could be interpreted as our noise distribution) to be larger. This resembles a reliability cost, as our reliability costs also encourage the noise distribution to be larger. Critically, the biological power-law reliability cost has a different form from the ideal, entropic reliability cost. However, we are able to derive a formal relationship: the biological power-law reliability costs bounds the ideal entropic reliability cost. Remarkably, this implies that our overall cost (Eq. 2) bounds the ELBO, so reducing our cost (Eq. 2) tightens the ELBO bound and gives an improved guarantee on the quality of Bayesian inference.

Discussion

Comparing the brain’s computational roles with associated energetic costs provides a useful means for deducing properties of efficient neurophysiology. Here, we applied this approach to PSP variability. We began by looking at the biophysical mechanisms of synaptic transmission, and how the energy costs for transmission might vary with synaptic reliability. We modified a standard ANN to incorporate unreliable synapses and trained this on a classification task using an objective that combined classification accuracy and an energetic cost on reliability. This led to a performance-reliability cost tradeoff and heterogeneous patterns of synapse variability that correlated with input rate and learning rate. We noted that these patterns of variability have been previously observed in data (see Fig. 6). Remarkably, these are also the patterns of variability predicted by Bayesian synapses (Aitchison et al., 2021) (i.e. when distributions over synaptic weights correspond with the Bayesian posterior). Finally, we showed empirical and formal connections between the synaptic variability implied by Bayesian synapses and our performance-reliability cost tradeoff.

The reliability cost in terms of the synaptic variability (Eq. 4) is a critical component of the numerical experiments we present here. While the precise form of the cost is inevitably uncertain, we attempted to mitigate the uncertainty by considering a wide range of functional forms for the reliability cost. In particular, we considered four biophysical mechanisms, corresponding to four power-law exponents, . Moreover, these different power-law costs already cover a reasonably wide-range of potential penalties and we would expect the results to hold for many other forms of reliability cost as the intuition behind the results ultimately relies merely on there being some penalty for increasing reliability.

The biophysical cost also includes a multiplicative factor, c, which sets the magnitude of the reliability cost. In fact, the patterns of variability exhibited in Fig. 5 are preserved as c is changed: this was demonstrated for values of c which are ten times larger and ten times smaller, Supplementary - Appendix 6–Fig. 2. This multiplicative factor should be understood as being determined by the properties of the physics and chemistry underpinning synaptic dynamics, for example it could represent the quantity of ATP required by the metabolic costs of synaptic transmission (although this factor could vary e.g. in different cell types).

Our artificial neural networks used backpropagation to optimise the mean and variance of synaptic weights. While there are a number of schemes by which biological circuits might implement backpropagation (Whittington and Bogacz, 2017; Sacramento et al., 2018; Richards and Lillicrap, 2019), it is not yet clear whether backpropagation is implemented by the brain (see Lillicrap et al. (2020) for a review on the plausibility of propagation in the brain). Regardless, backpropagation is merely the route we used in our ANN setting to reach an energy efficient configuration. The patterns we have observed are characteristic of an energy-efficient network and therefore should not depend on the learning rule that the brain uses to achieve energy-efficiency.

Our results in ANNs used MNIST classification as an example of a task; this may appear somewhat artificial, but all brain areas ultimately do have a task: to maximise fitness (or reward as a proxy for fitness). Moreover, our results all ultimately arise from trading off biophysical reliability costs against the fact that if a synapse is important to performing a task, then variability in that synapse substantially impairs performance. Of course performance, in different brain areas, might mean reward, fitness or some other measures. In contrast, if a synapse is unimportant, variability in that synapse impairs performance less. In all tasks there will be some synapses that are more, and some synapses that are less important, and our task, while relatively straightforward, captures this important property.

Our results have important implications for the understanding of Bayesian inference in synapses. In particular, we show that energy efficiency considerations give rise to two phenomena that are consistent with predictions outlined in previous work on Bayesian synapses (Aitchison et al., 2021). First, that normalised variability decreases for synapses with higher presynaptic firing rates. Second, that synaptic plasticity is higher for synapses with higher variability.

Specifically, these findings suggest that synapses connect their uncertainty in the value of the optimal synaptic weight (see Aitchison et al., 2021, for details) to variability. This is in essence a synaptic variant of the “sampling hypothesis”.

Under the sampling hypothesis, neural activity is believed to represent a potential state of the world, and variability is believed to represent uncertainty (Hoyer and Hyvärinen, 2002; Knill and Pouget, 2004; Ma et al., 2006; Fiser et al., 2010; Berkes et al., 2011; Orbán et al., 2016; Aitchison and Lengyel, 2016; Haefner et al., 2016; Lange and Haefner, 2017; Shivkumar et al., 2018; Bondy et al., 2018; Echeveste et al., 2020; Festa et al., 2021; Lange et al., 2021; Lange and Haefner, 2022). This variability in neural activity, representing uncertainty in the state of the world, can then be read out by downstream circuits to inform behaviour. Here, we showed that a connection between synaptic uncertainty and variability can emerge simply as a consequence of maximising energy efficiency. This suggest that Bayesian synapses may emerge without any necessity for specific synaptic biophysical implementations of Bayesian inference.

Importantly though, while the brain might use synaptic noise for Bayesian computation, these results are also consistent with an alternative interpretation: that the brain is not Bayesian, it just looks Bayesian because it is energy efficient. To distinguish between these two interpretations, we ultimately need to know whether downstream brain areas exploit or ignore information about uncertainty that arises from synaptic variability.

Materials and methods

The ANN simulations were run in PyTorch with feedforward, fully-connected neural networks with two hidden layers of width 100. The input dimension of 784 corresponded to the number of pixels in the greyscale MNIST images of handwritten digits, while the output dimension of ten corresponded to the number of classes. We used the reparameterisation trick to backpropagate with respect to the mean and variance of the weights, in particular, we set ω_i = μ_i + σ_iξ where ξ ~ Normal(0, 1) (Kingma et al., 2015). MNIST classification was learned through optimisation of Gaussian parameters with respect to a cross-entropy loss in addition to reliability costs using minibatch gradient descent under Adam optimisation with a minibatch size of 20. To prevent negative values for the σs, they were re-parameterised using a softplus function with argument ϕ_i, with σ_i = softplus(ϕ_i). The base learning rate in Eq.7 is η_base = 5 × 10⁻⁴. The μ_is were initialised homogeneously across the network from Uniform(−0.1,0.1) and the σ_is were initialised homogeneously across the network at 10⁻⁴. Hyperparameters were chosen via grid search on the validation dataset to enable smooth learning, high performance and rapid convergence. In the objective L_BI used to train our simulations, we also add an L1 regularisation term over synaptic weights, λ|μ|₁, where λ = 10⁻⁴.

Plots in Fig. 2 present mappings from hyperparameter c, to accuracy and σ. A different neural network was trained for each c, after 50 training epochs the average σ across synapses was computed, and accuracy was evaluated on the test dataset. Plots in Fig. 3 present mappings of this σ against accuracy and reliability cost. The reliability cost was computed using fixed s = 1 (see Appendix 51). To compute the Hessian in Fig. 5 and elsewhere, we used the empirical Fisher information approximation (Fisher, 1922), H ≈ g². This was evaluated by taking the average g² at ω* = μ over ten epochs after full training for 50 epochs. The average learning rate γ|g|⁻¹ and the average input rate |x| were also evaluated over ten epochs following training. The data presented illustrate these variables with regard to the weights of the second hidden layer. We set hyperparameter s = 0.001 (see Eq.51) in these simulations.

The experimental data in Fig. 6a were originally from data collected from plasticity experiments conducted by Sjöström et al. (2003). Costa et al. (2017) use this data to derive quantal parameters of a binomial model. Using the data processed by Costa et al. (2017) we calculated the normalised PSP variance before and after the experimental protocol, following the approach taken by Schug et al. (2021).

For geometric comparisons between the distribution over synapses and the Bayesian posterior presented in Fig. 7 we used the analytic results in Appendix - Analytic predictions for σ_i.

Source code used for simulations available at github.com/JamesMalkin/EfficientBayes

Acknowledgements

We are grateful to Dr Stewart whose philanthropy supported GPU compute used in this project. JM was funded by the Engineering and Physical Sciences Research Council (2482786). COD was funded by the Leverhulme Trust (RPG-2019-229) and Biotechnology and Biological Sciences Research Council (BB/W001845/1). CH is supported by the Leverhulme Trust (RF-2021-533).

Appendix 1

Reliability costs

The difficulty in determining reliability costs is that σ depends on three variables: n, the number of vesicles, p the probability of release and q, the quantal size, which measures the amount of neurotransmitter in each vesicle:

However, these variables also determine the mean

so a straighforward optimisation of σ under reliability costs will also change μ and one pitfall is to accidentally consider only those changes in reliability that derive from changes in mean. A solution to this problem is to eliminate one of the variables so that σ is a function of μ, and the remaining variables. Eliminating q gives

The idea is to consider energetic costs associated with p and n and relate these to σ while holding μ fixed. To simplify the biological motivation, we assume that during changes to synapse aimed at manipulating the energetic cost, q will also change to compensate for any collateral changes in μ making μ constant and q a “compensatory variable”. Moreover, it is possible, biologically, that q is the mechanism used in real synapses to fix μ (Turrigiano et al., 1998; Karunanithi et al., 2002) and so using this is justifiable.

In what follows four different energy costs are considered, the first depends on p, the next two on n, the situation for the final one is less clear, but in each case we derive a reliability cost in the form σ^-ρ for some value of ρ. Of course, since we are considering costs for fixed μ the coefficient of variation k = σ/μ is proportional to σ; it may be helpful to think of these calculations as finding the relationship between the cost and k.

Calcium efflux - ρ = 2

Calcium influx into the synapse is an essential part of the mechanism for vesicle release. Presynaptic calcium pumps act to restore calcium concentrations in the synapse; this pumping is a significant portion of synaptic transmission costs (Attwell and Laughlin, 2001). By rearranging the Hill equation defined by Sakaba and Neher 2001, it can be shown that vesicle release has an interaction coefficient of four, this means the odds of release per vesicle are related to intracellular calcium amplitude via the fourth power (Heidelberger et al., 1994; Sakaba and Neher, 2001):

To recover basal synaptic calcium concentration, the calcium influx is reversed by ATP-driven calcium pumps, where [Ca] ∝ ATP:

Since this physiological cost does not depend on n we assume it is fixed and so the odds of release p/(1 — p) is proportional to σ². Thus

or ρ = 1/2.

Vesicle membrane -

There is a cost associated with the total area of vesicle membrane. Evidence in Pulido and Ryan 2021 suggest stored vesicles emit charged H⁺ ions, with the number emitted proportional to the number of v-glut, glutamate transporters, on the surface of vesicles. v-ATPase pumps reverse this process maintaining the pH of the cell. It is suggested that this cost is 44% of the resting synaptic energy consumption. In addition, metabolism of the phospholipid bilayer that form the membrane of neurotransmitter filled vesicles has been identified as a major energetic cost (Purdon et al., 2002). Provided the total volume is the same, release of the same amount of neurotransmitter into the synaptic cleft can involve many smaller vesicles or fewer larger ones. However, while having many small vesicles will be more reliable, it requires a greater surface area of costly membrane. With fixed μ and p,

Since r² ∝ q^2/3 and using μ = npq this give

Since this reliability cost depends on n, so p is regarded as constant, so σ² ∝ n and hence ρ = 2/3.

Actin -

Actin polymers are an energy costly structural filament that support the structural organisation of vesicle reserve pools (Cingolani and Goda, 2008), with the vesicles strung out along the filamants. We assume each vesicle to require a length of actin roughly proportional its diameter, this means that the total length of actin is proportional to nr. Hence

The calculation then proceeds much as for the membrane cost, but with r instead of r² giving ρ = 4/3.

Trafficking - ρ = 2

ATP fueled myosin motors drive trains of actin filament along with associated cargo such as vesicles and actin- myson trafficking moves vesicles from vesicle reserve pools to release sites sustaining the readily releasable pool (RRP) following vesicle release (Bridgman, 1999; Gramlich and Klyachko, 2017). This gives a cost for vesicle recruitment proportional to np, the number of vesicles released:

so if n is regarded as the principal way this cost is changed, with p fixed then n ∝ σ^-2 and so ρ = 2. This is the view point we are taking to motivate examining reliability costs with ρ = 2. This is certainly useful in considering the range of model behaviours over a broad a range of ρ values.

Nonetheless, it is sensible to ask whether the likely biological mechanism behind a varying trafficking cost is one which changes np itself. In this case, since a constant μ for varying nr means q ∝ 1/np we have

which, is, again, of the form cost∝ σ^-ρ provided p is small. For larger p, however, it depends on exactly how p changes as np changes.

Generally, throughout these calculation we have supposed that q is a compensatory variable and that either p or n changes in the process that changes the cost at a synapse. The benefit of this is that we are able to model costs directly in terms of reliability; in the future, though, it might be interesting to consider models which use n, p and q instead of μ and σ; this would certainly be convenient for the sort of comparisons we are making here and interesting, although two formulations seem equivalent, this does not mean that the learning dynamics will be the same.

Appendix 2

High input rates and high precision at important synapses

Using a generalised linear model, we demonstrate that synapse variability optimised under the performancereliability cost tradeoff depends on the input rate and the Hessian. To do this, consider a set of predictor-response pairs (y_a, x_a) where a labels the data points, and each data point x_a is a vector over features we label with index i. Logistic regression is an attempt to approximate y_a as a function of x_a:

where w are the regression coefficients which correspond with synaptic weights, f is the link function, in the example of logistic regression this would be the logit function. One approach to regression is to minimise the normalised square error in this approximation:

Network weights are drawn independently from a normal distribution, Q(w),

To solve for optimal σ in addition to μ we optimise L_BI = performance cost + reliability cost.

where v_a = f(y_a). Also note that ε² is the expected square error, the variance of P(y|w, x). For brevity we have disregarded the magnitude cost since it is a cost on the synapse mean while our focus in this section is on how synapse variance relates to input rate and synapse importance.

Expanding the brackets and using the commutativity of the dot-product, , this gives

Now E[ww^T] = μμ^T + ∑ where ∑ is the multivariate covariance matrix for the σ_i; since these are independent ∑_ii = σ_i and the other entries are zero. With this

To obtain the Hessian we first take the first derivative with respect to μ,

We can set the derivative to zero and solve to get the least squares formula:

where for convenience we write . Note that A is the auto-correlation matrix of the neural network input evaluated over data-points.

The second derivative gives the Hessian for synaptic weights:

We can use this to calculate the trace of the Hessian,

To find σ_i that optimises the tradeoff we take the derivative with respect to σ_i; writing in index form we have

so

which means

or

Hence, for generalised linear models, the σ_i that solves or optimises the tradeoff is related to the size of synapse input rate, through a power law relationship depending on ρ, with larger input giving smaller variance. This relationship is particularly clear for ρ = 2 since this simplifies the expression for :

Appendix 3

Synapse importance and gradient magnitudes

In our ANN simulations we train synaptic weights using the most established adaptive optimisation scheme, Adam (Kingma and Ba, 2014); which has recently been realised using biologically plausible mechanisms (Yang and Li, 2022). Adam uses a synapse-specific learning rate, η_i, which decreases in response to high gradients at that synapse,

Specifically, the local learning rate for the ith synapse, η_i, is usually a base learning rate, η_base, divided by , the root-mean-square of the gradient for each datapoint / minibatch.

The key intuition is that if the gradients for each datapoint/minibatch are large, that means that this synapse is important, as it has a big impact on the predictions for every datapoint. In fact, these mean-square gradients can be related to our formal measure of synapse importance, the Hessian. Specifically, for data generated from the model, the Hessian (or Fisher information Fisher, 1922) is equivalent to the mean squared gradient, where the gradient is taken over each datapoint separately,

Of course, in practice, the data is not drawn from the model, so the relationship between the squared gradients and the Hessian computed for real data is only approximate. But it is close enough to induce a relationship between synapse importance (measured as the diagonal of the Hessian) and learning rates (which are inversely proportional to the root-mean-square gradients) in our simulations.

Appendix 4

Energy efficient noise and variational-Bayes for neural network weights

Introduction to Variational Bayes for neural network weights

One approach to performing Bayesian inference for the weights of a neural network is to use variational Bayes. In variational Bayes, we introduce a parametric approximate posterior, Q (w), and fit the parameters of that approximate posterior using gradient descent on an objective, the ELBO (Blundell et al., 2015). In particular,

where H[Q(w)] is the entropy of the approximate posterior. Maximising the ELBO is particularly useful for selecting models as it forms a lower-bound on the marginal-likelihood, or evidence, log P(y|x) (MacKay, 1992a; Barber and Bishop, 1998). When using variational Bayes for neural network weights, we usually use Gaussian approximate posteriors (Blundell et al., 2015),

Note that optimising the ELBO wrt the parameters of Q is difficult, because Q is the distribution over which the expectation is taken in Eq. (36). To circumvent this issue, we use the reparameterisation trick (Kingma et al., 2015), which involves writing the weights in terms of IID standard Gaussian variables, ε,

Thus, we can write the ELBO as an expectation over ε, which has a fixed IID standard Gaussian distribution,

Note that we use w, μ etc. without indicies in this expression to indicate all the weights / mean weights.

Identifying the log-likelihood and log-prior.

Following the usual practice in deep learning, we assume the likelihood is formed by a categorical distribution, with probabilities obtained by applying the softmax function to the output of the neural network. The log-likelihood for a categorical distribution with softmax probabilities is the negative cross-entropy (Murphy, 2012),

where f (x, w) is the output of the network with weights, w and inputs, x. We can additionally identify a Laplace prior with the magnitude cost. Specifically, if we take the prior to be Laplace, with scale 1/λ,

where we identify the magnitude cost using Eq. (3).

Connecting the entropy and the biological reliability cost.

Now, we have identified the log-likelihood and log-prior terms in the ELBO (Eq. 36) with terms in our biological cost (Eq. 2). We thus have two terms left: the entropy in the ELBO (Eq. 36) and the reliability cost in the biological cost (Eq. 2). Critically, the entropy term also acts as a reliability cost, in that it encourages more variability in the weights (as the entropy is positive in the ELBO (Eq. 36) and we are trying to maximise the ELBO, the entropy term gives a bonus for more variability). Intuitively, we can think of the entropy term in the ELBO (Eq. 36) as being an “entropic reliabity cost”, as compared to the “biological reliability cost” in (Eq. 2).

This intuitive connection suggests that we might be able to find a more formal link. Specifically, we can define an entropic reliability cost as simply the negative entropy,

Our goal is to write the biological reliability cost, C_BI;i, as a bound on C_VI;i. By rearranging and introducing s and ρ,

and noting that log a ≤ a — 1,

we can demonstrate that any reliability cost expressed as a generic power-law forms an upper bound on the entropic cost,

where,

The parameter c sets the importance of the reliability cost within the performance-reliability cost tradeoff (see Fig. 2)

Note that while both reliability costi and C_BI;i represent the biological reliability costs, they are slightly different in that C_BI;i includes an additive constant. Importantly, this additive constant is independent of σ_i and μ_i, it does not affect learning and can be ignored.

Given that the biological reliability cost forms a bound on the ideal entropic reliability cost we can consider using the biological reliability cost in place of the entropic reliability cost,

we find that our overall biological cost (Eq. 2) forms a bound on the ELBO, which itself forms a bound on the evidence, log P(y|x). Thus, pushing down the overall biological cost (Eq. 2) pushes up a bound on the model evidence.

Predictive probabilities arising from biological reliability costs

Given the connections between our overall cost and the ELBO, we expect that optimising our overall cost will give a similar result to variational Bayes. To check this connection, we plotted the distribution of predictions induced by noisy weights arising from variational Bayes (Appendix 4—Fig. 1a) and our overall costs (Appendix 4—Fig. 1b). Variational Bayes maximises the ELBO, therefore its predictive distribution is optimised to reflect the data distribution from which data is drawn (MacKay, 1992b). We found comparable patterns for the predictive distributions learned through variational Bayes and our overall costs, albeit with some breakdown in predictive performance with higher values for ρ.

Appendix 4—figure 1.

Interpreting c, ρ and s

As discussed in the main text, c and ρ are the fundamental parameters, and they are set by properties of the underlying biological system. It may nonetheless be interesting to consider the effects of s and ρ on the tightness of the bound of the biological reliability cost on the variational reliability cost. In particular, we consider settings of s and ρ for which the bound is looser or tighter (though again, there is no free choice in these parameters: they are set by properties of the biological system).

First, the biological reliability cost becomes equal to the ideal entropic reliability cost in the limit as ρ → 0.

Thus, taking ρ = x, and z = s/σ_i,

Thus,

This explains the apparent improvement in predictive performance (Appendix 4—Fig. 1) and in matching the posteriors (Fig. 7) with lower values of ρ.

Second, the biological reliability cost becomes equal to the ideal entropic reliability cost when s = σ_i,

as the first term in Eq. 47 cancels. However, s cannot be set individually across synapses, but is instead roughly constant, with a value set by underlying biological constraints. In particular, s can be written as a function of ρ and c (Eq. 51), and ρ and c are quantities that are roughly constant across synapses, with their values set by biological constraints. Thus, biological implications of a tightening bound as s tends to σ_i are not clear.

Appendix 5

Analytic predictions for σ_i

At various points in the main text, we note a connection between the Hessian, synapse importance and optimal variability. To develop a formal understanding of these connections, we perform a second-order Taylor expansion of the performance cost and magnitude cost, considering only the quadratic term, as that is the only term that depends on the variances (the other terms are incorporated in const),

Writing this as a Trace,

Cyclically permuting the Trace,

Noting that the weights are the only stochastic quantities,

And noting that the biological synaptic noise (e.g. from vesicle release) is independent,

Now, combining the reliability cost, with the performance and magnitude cost gives the overall cost,

Note that under this quadratic approximation, the dependence of the overall cost on σ_i has become decoupled across synapses. We can therefore compute the optimal synaptic noise level, σ_i as a function only of the Hessian for that synapse, H_ii. In particular, we solve for where the gradient of the cost is zero,

Solving for σ_i this give

or

Thus,

which is borne out in Appendix 5—Fig. 1a. Now that we know that optimal synapse variance scales with the Hessian, we can combine this with other quantities that scale with the Hessian; such as the learning rate, input rate or the Bayesian posterior variance. Allowing us to derive how these quantities scale with optimised synapse variance.

The first row uses the analytic relationship between the mean squared gradient and the Hessian (Eq. 35) to obtain a relationship between the optimal synaptic noise level and the learning rate providing the slope in Appendix 5—Fig. 1b. The second row expresses the analytic relationship between the Hessian and the mean square of the input rate (see Appendix - High input rates and high precision at important synapses), and gives the analytic relationship in Appendix 5—Fig. 1c). The third row expresses the analytic relationship between the Hessian and the Bayesian posterior variance, used in Fig. 7.

To test these predictions, we performed a simpler simulation classifying MNIST in a network with no hidden layers. We found that the analytic results closely matched the simulations, and that the biological slopes tend to match the ρ = 2 better than the other values for ρ. However, while the direction of the slope was consistent in deeper networks, the exact value of the slope was not consistent (Fig. 5 and Supplementary - Appendix 6—Fig. 1), so it is unclear whether we can draw any strong conclusions here.

Appendix 5—figure 1.

Appendix 6

Appendix 6—figure 1.

Appendix 6—figure 2.