Investigation of in vitro patterned networks using a real-time closed-loop stimulation framework with RL control.

a, In vitro neuronal networks in PDMS microstructures on MEAs. Bidirectional circular microstructures confine cell bodies in seeding wells (open at the top) and guide axons through microchannels. Microstructures are aligned such that each microchannel covers one electrode. b, 15 networks fit on one 60-electrode MEA. Confocal laser-scanning microscopy image of GFP-expressing Ngn2-induced neurons on DIV 14 that were seeded as spheroids. c, Electrophysiology board of inkube. MEA electrodes are recorded via 4 ASICs. d, The inkudock system maintains physiological culture conditions. e, RL environment implementation on biological networks. A stimulation action An is encoded on each of the 4 electrodes at each time step n. Action vector values in [−1, 0) convey no stimulation, whilst values between 0 and 1 are translated linearly into stimulation delays from 5 to 0 ms. A blanking delay tD is added to all stimulations. Spikes are recorded for 20 ms post-stimulation and constitute the state Sn. For the real-time spike detection, only the cut-out waveforms are stored. The waveforms are plotted in the subfigure. The reward Rn is the length of the longest valid clockwise sequence found across all possible starting electrodes, where consecutive spikes on successive electrodes must fall within a 0.5-5 ms inter-spike interval corresponding to the expected synaptic transmission window. f, Timing of inkube, interface, and RL process communication. The RL process consists of the environment for inter-process communication and the RL agent. The agent sends an action vector for its respective network for the period n through the step command. It is forwarded via the environment to the interface, which translates and stacks the actions from all agents into a m × 3 matrix, with m being the number of stimuli and the columns corresponding to timing, target network, and electrode. Once the interface has received an action from all RL processes or approaches the block time, the stimulus vector is sent out. The inkube main process applies the stimulus, given that it is received before a block phase for internal communication. After recording, the spikes are split by network and distributed to the environments, which locally compute the reward and state based on their settings. The agent can incorporate the feedback information into its action choice by updating its value functions. This cycle repeats at a frequency 1/TStim.

Responses are stable and separable across the action space.

a, 500 random action vectors were generated, with at least one electrode stimulated. Actions were drawn uniformly at random and applied at 4 Hz for a total of 50,000 stimulations per network (each action was applied around 100 times). b, The reward associated with the same action was extracted across repetitions. The four shown actions represent the mean reward and reward standard deviation maxima and minima of a single network on DIV 24. c, Post-stimulus raster plots are shown for selected actions and the extrema actions from b across repetitions on the same network as in b. A spike on one of the network electrodes after stimulation is marked by a dot at its latency, colour-coded by electrode as in Fig. 1a. d, Action performance was compared for all actions on all networks, averaged across repetitions. The distribution of mean rewards peaks near 2.8, meaning the mode of the distribution corresponds to more than half a cycle inside the microstructure. e, The reward standard deviation was calculated for all actions and all networks across repetitions, showing a right-skewed distribution centred around 0.9. f, The inverse coefficient of variation (mean reward divided by standard deviation) was computed for all actions and networks. The median of 2.67 indicates that for most actions, the mean reward exceeds the trial-to-trial variability by more than twofold. g, Reward standard deviation was summarised per network across four MEAs (MEA A spheroid seeding, B-D suspension seeding). In total, we included 21 networks in our analysis, which showed an average firing rate on all 4 electrodes of at least 4 Hz while recording spontaneous activity and good signal quality. Independent networks were sorted by median standard deviation, showing significant differences in variability between networks. h, The mean number of evoked spikes per action per network was analysed. Networks with higher spike counts tend to show higher reward variability. i, To assess the range of achievable performance, the maximum mean reward across all 500 actions is shown for each of the networks. Networks are now sorted by MEA as indicated by the colour code. This encoding is kept consistent with Fig. 4a,c. j, To verify that reward signals remain stable over the full duration of the experiment, the Augmented Dickey-Fuller test for stationarity was applied to the reward sequence of each action. Across networks, approximately 90 % of actions show stationary responses (p < 0.01), confirming that the evoked responses do not systematically drift over the course of the experiment.

State representation of responses is separable by the stimulation history.

a, A pair of random actions X and Y was generated and applied in a random order to test whether the previous action can be determined from the network response after the current action. Action pairs that stimulate at least one electrode were generated and applied 1920 times. The post-stimulus response Sn as defined in Fig. 1e was recorded and compressed to Ŝn using a state compression algorithm. With the state we stored the current action An and previous action An−1 . The elements were summarised into vectors Ŝn, An, and An−1 for each action pair. Ŝn was then split into 4 groups by the different current and previous actions: XX, XY, YX, YY. b, The distributions for every state dimension of each subset of Ŝn were calculated. Subsequently, the distributions of the states with the same current action An were compared in the nstate-dimensional space using the Sliced Wasserstein distance metric (see Methods for details). c, The distances of the subsets split by previous action An−1 were compared with a null distribution, obtained by arbitrarily splitting with a scrambled vector An−1 . The z-score of each action with respect to the scrambled distribution was extracted. Based on this, we analysed whether the previous action led to a larger distance of the distributions of a group when split by the previous action instead of a scrambled vector. A larger distance between the groups when split by previous action means discriminability, and therefore a dependence of the state on the previous action. d, The z-score of each pair of actions in comparison with its respective null distribution was tested for significance (p < 0.01; z > 2.326). The significant proportion is shown on a logarithmic scale. The PCA approach has 35 % of significant z-scores, the DCNN approach 26.2 %, meaning those proportions of the tested action pairs enabled separation of the state distributions based on the previous action. e, The proportion of significant action pairs was compared across PCA and DCNN state compression methods with varying dimensionality and binning parameters. The data was aggregated by network and stimulation frequency (6 networks on MEA D, random seeding; 7 networks each on MEA E and F, spheroid seeding). The different number of XY pairs tested were 50 (4 Hz; MEA D), 32 (4 Hz; MEA E), 50 (16 Hz; MEA E), and 86 (4Hz; MEA F), for the 4 datasets. f, Results obtained at stimulation frequencies of 4 and 16 Hz were compared using DCNN-5. The significant proportion is shown on a logarithmic scale. At 16 Hz, a significantly larger proportion of action pairs yielded separable network states (29.6 % vs. 19.3 %, two-proportion z-test, p < 0.001), suggesting that higher stimulation frequency increases state dependence.

All agents learn to improve reward over random stimulation.

a, Mean test reward across agents, networks, and DIVs. MAB and LCB are tested on both the continuous (c) and discretised (d) action space. DynamicLCB operates on the discrete space with an additional state-dependent action value matrix. Colour encodes network identity, shape encodes DIV. b, Pairwise statistical comparison of mean test reward. Colour encodes the p-value of a two-sided Wilcoxon signed-rank test on min-max normalised, per-network averaged rewards (for details, see agent comparison in Methods). Asterisks denote significance at α = 0.05. All agents perform significantly better than random stimulation. Differences between agents are small. The continuous LCB performs worse than the discretised version and the discrete MAB. c, Top 20 % test reward, defined as the mean of the highest-scoring 20 % of test samples per run. The corresponding pairwise comparison matrix is shown in Fig. S19. d, Training curves for each agent, excluding the random baseline. Thin lines show individual network runs smoothed with a 1,000-step moving average, left-padded with NaN values. Bold lines show the mean across networks. Agents differ in the total number of training steps required for plateauing. For DynamicLCB, the onset of state-dependent action switching is visible at approximately one-third of the training steps. For the adaptive MAB the fluctuations can be seen when the action space is adjusted every 2,500 steps. The continuous LCB shows only a minor increase in average reward over the majority of the training steps. Training with a shorter moving average filtering of 50 steps is provided in Fig. S23 for the first 1,000 steps, the full training, and the transition to the test phase.

Best-performing actions are non-trivial and state-based agents exploit action switching.

a, Example post-stimulus raster plots during the test phase on the same network for an MAB (left) and discrete LCB (right) agent, respectively. Stimulation timing is marked with diamonds, spikes are represented by dots, both colour-coded according to Fig. 1a. The reward per period is shown on the right. Responses remain stable even after hours of continuous stimulation. b, Distribution of stimulation delays chosen by the respective best-performing action averaged over different networks for four types of agents. A single bin at the left represents no stimulation on that electrode. Actions span the entire action space. c, Reward matrix for an agent on one network. Actions were grouped by computing the L2 distance between consecutive four-dimensional action vectors and partitioning these distances into equal-frequency quantile bins (target n = 5). Adjacent bins whose width fell below 0.25 were merged. The effective number of action groups varied between agents and networks. Matrix entries were aggregated by current action group (rows) and previous action group (columns). Each cell shows the mean reward and number of trials. The diagonal corresponds to repeated actions, non-diagonal entries to action switches. The agent applied certain action pairs more frequently, indicating learned switching preferences. Bold text and symbols * mark the best-performing actions (average reward significantly higher; Mann-Whitney U, p < 0.05). d, The matrices show the corresponding reward difference compared to repeating the same action. Asterisks denote statistically significant differences (Mann-Whitney U, p < 0.05). Several action switches yielded a measurable reward modulation over repetition. e, Proportion of non-diagonal action pairs showing a statistically significant reward difference, summarised across all networks for each state-based agent type. At least some beneficial switching pairs exist for all agents, though the proportion varies across networks. f, Difference in mean reward between diagonal (repeated) and non-diagonal (switched) entries. No agent showed a significant bias (two-sided Wilcoxon signed-rank, all pbonf > 1.0). The effect was centred near zero, indicating that on average the reward benefit from switching was modest, with positive and negative exceptions. g, Difference in maximum reward between diagonal and non-diagonal entries. No agent showed a significant difference (two-sided Wilcoxon signed-rank, all pbonf > 0.05).