Introduction

In neuroscience, brain activity and behavior are often studied as separate entities that do not interact in real time. Assessments are typically performed post hoc, and experimental designs rarely incorporate dynamic feedback based on ongoing neural activity. In contrast, closed-loop systems provide a framework where real-time interaction occurs between the brain, subject, and goal-directed outcomes, enabling precise modulation of neural and behavioral dynamics (Clancy et al. 2014). First introduced in the pioneering work of (Fetz 1969), closed-loop feedback has since demonstrated significant potential in understanding and modulating brain-behavior relationships. There has been recent interest in its application to rodent models (Clancy and Mrsic-Flogel 2021; Clancy et al. 2014; Prsa, Galiñanes, and Huber 2017; Srinivasan et al. 2018; Knudsen and Wallis 2020; Paz et al. 2013; Ching et al. 2013; Rosin et al. 2011; Widge and Moritz 2014; Sun et al. 2022). Some related studies have demonstrated the feasibility of closed-loop feedback in rodents, including hippocampal electrical feedback to disrupt memory consolidation (Girardeau et al. 2009), optogenetic perturbations of somatosensory circuits during behavior (O’Connor et al. 2013), and more recent advances employing targeted optogenetic interventions to guide behavior (Abbasi et al. 2023). Opportunities also exist in extending real time pose classification (Forys et al. 2020; Kane et al. 2020) and movement perturbation (M. W. Mathis, Mathis, and Uchida 2017) to shape aspects of an animal’s motor repertoire. This gap is due, in part, to technical challenges such as the need for low-latency data acquisition and processing, compact and portable hardware, and seamless integration with animal behavior. Moreover, the inherent complexity of rodent neurophysiology and behavior (e.g., undesired movements, variability in neural signals, and behavioral metrics) further complicates the implementation of closed-loop paradigms.

Closed-loop systems are particularly promising for addressing critical questions in systems neuroscience, such as how brain activity dynamically adapts to feedback or how neural circuits reorganize during motor learning and recovery. Recent advancements, such as genetically encoded calcium indicators and mesoscopic imaging, have opened new avenues for studying these phenomena (Clancy and Mrsic-Flogel 2021; Clancy et al. 2014; Prsa, Galiñanes, and Huber 2017). However, many of the available closed-loop systems remain technically complex, expensive, or inaccessible to labs with limited resources. Additionally, most existing platforms focus on narrowly defined use cases, limiting their adaptability to diverse experimental paradigms. Nevertheless, recent work has demonstrated behavioral control through feedback, such as in (M. W. Mathis, Mathis, and Uchida 2017), where mice learned to pull a joystick to a target and adapt this motion under force perturbations.

To address these gaps, we present CLoPy - a flexible, cost-effective, and open-source Python-based platform for closed-loop feedback experiments in neuroscience. CLoPy integrates real-time data acquisition, processing, and feedback delivery, providing researchers with a robust tool to study dynamic brain-behavior interactions in head-fixed mice. The platform supports two modes of graded feedback: (1) Closed-Loop Neurofeedback (CLNF), which derives feedback from neuronal activity, and (2) Closed-Loop Movement Feedback (CLMF), which bases feedback on observed body movements. By combining genetically encoded calcium imaging with behavioral tracking, CLoPy enables researchers to explore questions such as how cortical dynamics contribute to motor learning or how feedback modulates neural plasticity in neurological models.

Crucially, CLoPy is designed with accessibility and versatility in mind. The platform utilizes widely available hardware, including Raspberry Pi and Nvidia Jetson, and leverages Python-based open-source software libraries. These features make CLoPy an ideal resource for the broader open-source neuroscience community, particularly for labs seeking to implement closed-loop systems without substantial financial or technical barriers. Furthermore, CLoPy’s modular design allows researchers to adapt it for a wide range of experimental paradigms, from studying cortical reorganization during motor learning to investigating sensory feedback mechanisms. By framing CLoPy within the context of accessible, open-source neuroscience tools, we aim to broaden its impact and utility across the systems neuroscience community.

In this study, we demonstrate the implementation and efficacy of CLoPy in head-fixed mice (Figure 1), showcasing its ability to deliver precise, low-latency feedback. We provide all software, hardware schematics, and protocols necessary for its adaptation to various experimental scenarios, positioning CLoPy as a valuable tool for advancing closed-loop neuroscience research.

Figure 1

Results

Consistent with prior studies demonstrating that mice can exert volitional control over their brain activity through feedback-based paradigms (Clancy et al. 2014; Luo et al. 2024; Neely, Koralek, et al. 2018; Clancy and Mrsic-Flogel 2021), our experiments show that mice learned to robustly modulate cortical signals when provided with closed-loop neurofeedback. Across training sessions, animals not only acquired the task (Figure 4A, initial rule RM ANOVA p=2.83e-5; Figure 4B, No-rule-change group RM-ANOVA p=9.6e-7) but also exhibited adaptability to altered feedback rules (Figure 4A, performance drop after rule change ANOVA, p=8.7e-9, new rule RM ANOVA p=8.3e-10), highlighting the flexibility of their learning. While feedback-driven brain control is well established, our work extends these findings by including multiple cortical regions, and by demonstrating that feedback based on movement-related signals can also drive learning despite the additional challenge of real-time behavioral tracking.

In our closed-loop experiment, mice were trained to associate a specific body movement with a reward. Over the course of 10 training sessions, the mice progressively learned the task, as evidenced by increased task performance across sessions (Figure 4A). The acquisition of this learned behavior was consistent with previous findings on motor learning, where reinforcement and task repetition lead to the refinement of motor skills (Wolpert, Diedrichsen, and Flanagan 2011; Krakauer et al. 2019). Importantly, the reward was provided 1 s after a successful trial to study the timing and nature of cortical activation in response to both the task and the reward. Concurrent with behavioral training, widefield cortical activity was recorded observing distinct patterns of neural activation that evolved as the mice learned the task.

We implemented and tested a compact, cross-platform system designed for closed-loop feedback experiments. Using this system, head-fixed mice successfully learned to associate either cortical activity or movement-derived signals with auditory feedback and water rewards. Performance increased steadily over sessions: in both one-ROI and two-ROI paradigms, animals showed a significant rise in the number of rewards obtained compared to their baseline (day 0) and control groups without feedback (Animations 1A–F). These results indicate that the system effectively links cortical dynamics or movement signals to behaviorally meaningful outcomes. Notably, both neurofeedback (CLNF) and movement-feedback (CLMF) modes produced reliable and graded closed-loop feedback.

Figure 2:

In CLNF experiments, auditory tones mapped to ΔF/F₀ values from cortical regions of interest accurately reflected ongoing neural activity (Figure 1). In two-ROI paradigms, differences between ROI activity were similarly mapped to sound frequency, and the commanded audio outputs were verified to be accurate across the full frequency range (Supplementary Figure 2). Improvements to the audio generation pipeline eliminated previous nonlinearities above 10 kHz, confirming that the feedback signal matched the intended output. Importantly, feedback latency was tested and found to be well within the range necessary for effective closed-loop learning. Using LED-triggering experiments (Supplementary Figure 1), we confirmed that the Raspberry Pi–based CLNF system delivered graded auditory feedback with an average latency of ∼63 ± 15 ms from event detection to output, using GPIO pins. Please note this latency calculation used GPIO pins for delay measurements, and there could be additional minimal overheads to generate the actual audio feedback signals. These findings demonstrate that our system provides rapid and accurate feedback sufficient to drive volitional control of brain or movement signals in head-fixed mice.

For the CLMF experiments, an Omron Sentech STC-MCCM401U3V USB3 Vision camera, connected to the Nvidia Jetson via its Python software developer kit (SDK), was used to calculate the feedback delay. The incoming stream of frames was processed in real-time using a custom deep-neural-network model that was trained using DeepLabCut (A. Mathis et al. 2018) and DeepLabCut-Live (Kane et al. 2020), designed to track previously defined points on the mouse. The model was incrementally improved by fine-tuning and re-training 26 times, using 2080 manually labeled frames spanning 52 videos of 10 mice. The pre-trained model, along with the data, is available at the link shared in the data and code availability statement in methods for anyone to use and fine-tune to adapt for similar platforms. The model was integrated into our CLMF program and deployed on an Nvidia Jetson device for real-time inference of tracked points. A Python-based virtual environment using Conda was created to install all software dependencies. The coordinates of the tracked points for each frame were appended to a list, forming a temporal sequence referred to as “tracks.” Upon offline analysis, these real-time tracks were found to be both accurate and stable throughout the duration of the videos.

To calculate the feedback delay for movements, a red LED was placed within the behavior camera’s field of view. Whenever a threshold-crossing movement was detected in the real-time tracking, the system triggered the LED to turn on. Temporal traces of the tracked left forelimb and the pixel brightness of the red LED were then extracted from the recorded video. By comparing these traces, the average delay between the detected movements and the LED illumination was calculated to be 67±20 ms, which represents the delay in the closed-loop feedback for the CLMF experiments.

For CLMF, mice were headfixed in a specially designed head-fixing chamber (Figure 1Biii, design guide and 3D model in methods) to achieve multiview behavioral recording from a single camera using mirrors (see methods section). In brief, mice are headfixed in a transparent rectangular tunnel (top view) with a mirror at the front (front view) and at the bottom (bottom view) that allows multiple views of the mouse body. The body parts that we track are: snout-top (snout in the top view), tail-top (base of the tail in the top view), snout-front (snout in the front view), FLL-front (left forelimb in the front view), FLR-front (right forelimb in the front view), snout-bottom (snout in the bottom view), FLL-bottom (left forelimb in the bottom view), FLR-bottom (right forelimb in the bottom view), HLL-bottom (left hindlimb in the bottom view), HLR-bottom (right hindlimb in the bottom view), tail-bottom (base of the tail in the bottom view) (Figure 1B). The rationale for selecting these body parts in a particular view was that they needed to be visible at all times to avoid misclassification in real-time tracking. By combining the tracks of a body part in different views, we can form a 3D track of the body part. In a 3D coordinate space having X, Y, and Z axes, tracked points (xb, yb) in the bottom view were treated as being in the 3D X-Y plane, and tracked points (xf, yf) in the front view were treated as being in the 3D Y-Z plane. Thus, X = xb, Y = yb, Z = yf formed tracked points in 3D for a given body part that was tracked in multiple views.

For example, FLL-front and FLL-bottom were tracking the left forelimb in front and bottom views, and by combining the tracks of these two points, we obtained a 3D track of the left forelimb (Figure 3E). Although these 3D tracks are available in real-time, for our CLMF experiments, we used 2D tracks (xb, yb) for behavioral feedback to keep the task simpler and comparable across mice. Audio output channels and GPIO pins on the Nvidia Jetson were used for audio feedback and reward delivery, respectively. Tracks of each body part were used to calculate the speed of those points in real-time, and a selected body part (also referred to as a control point) was mapped to a function generating proportional audio frequency (same as in CLNF, details in the method section). In the software we have developed, one can also choose to calculate acceleration, radial distance, angular velocity, etc., from these tracks and map it to the function generating varying audio frequency feedback. For our work, a range of spontaneous speeds was calculated from a baseline recording before the start of the training. The threshold speed to receive a reward was also calculated from the baseline recording and set at a value that would have yielded a reward rate of one reward per two-minute period (translates to a basal performance of 25% in our trial task structure). This thresholding step was done to allow the mice to discover task rules and keep them motivated.

Figure 3

Both CLNF and CLMF experiments shared a similar experimental protocol (surgery, habituation, then several days of training, Figure 3A). A daily session starts with a 30 sec rest period (no rewards or task feedback) followed by 60 trials (maximum 30 sec each) with a 10 sec rest between trials in CLNF, and a minimum of 10 sec rest or until tracked points are stable for 5 sec in CLMF (Figure 3B, 3D). After the habituation period, a spontaneous session of 30 minutes was recorded where mice were not given any feedback or rewards.

Figure 4:

The spontaneous session was used to establish baseline levels of GCaMP activity (in target ROI(s) for CLNF experiments) or speed of a target body part for CLMF experiments. This was done to calculate the animal-specific threshold for future training sessions. A success in the trial resulted in a water drop reward that was delivered 1 sec after the end of the trial, and a failed trial ended with a buzzer vibrator 1 sec after the end of the trial (Figure 3B, 3D).

Mice can explore and learn an arbitrary task, rule, and target conditions

CLNF training (Figure 1A) required real-time image processing, feedback, and reward generation. Feedback was a graded auditory tone mapped to the relative changes in selected cortical ROI(s) or movement speed of a tracked body part. Training was conducted using multiple sets of cortical ROI(s) (list of ROI rules and their description can be found in Table 3 and visualized in Supplementary Figure 7) on both male and female mice (see Table 5 for list of mice and their task rules), wherein the task was to increase the activity in the selected ROI(s), according to the rule (also referred to as ‘the rule’ in the future), to exceed a predetermined threshold (Figure 6A, 6B). Fluorescence activity changes (ΔF/F₀) were calculated using a running baseline of 5 seconds, with single or dual ROIs on the dorsal cortical map selected as targets. The distribution of target ROI(s) activity on day 9 of training was higher than on day 1 (Figure 5A). A linear regression analysis of whole-session ΔF/F₀ activity in the 2-ROI experiment revealed a shift in the regression line slope, favoring the ROI that was required to increase according to the task rule (Figure 6C). In general, all ROIs assessed (Table 3, Supplementary Figure 7) that encompassed sensory, pre-motor, and motor areas were capable of supporting increased reward rates over time based on analysis of group and pooled ROI data (Figure 4A, Animation 1). A ΔF/F₀ threshold value was calculated from a baseline session on day 0 that would have allowed 25% performance. Starting from this basal performance of around 25% on day 1, mice (CLNF No-rule-change, N=23, n=60 and CLNF Rule-change, N=17, n=60) were able to discover the task rule and perform above 80% over ten days of training (Figure 4A, RM ANOVA p=2.83e-5), and Rule-change mice even learned a change in ROIs or rule reversal (Figure 4A, RM ANOVA p=8.3e-10, Table 5 for different rule changes). There were no significant differences between male and female mice (Supplementary Figure 3A). To investigate whether certain ROI(s) were better than others in terms of performance, we performed linear regression of the success rate over the days and, based on the slope of the fitted line, discovered ROI rules that yielded statistically different progression (fast and relatively slower) of success rate from the mean slope of all ROIs (Supplementary Figure 3C). We visualized these significantly different ROI-rule-based success rates and segregated them into fast and slow based on mean slope (>=0.095 slope was designated fast, else slow) of the progression (Supplementary Figure 3D). Our analysis revealed that certain ROI rules (see description in methods) lead to a greater increase in success rate over time than others (Supplementary Figure 3D). As mice learned the CLNF task and their performance improved, task latency progressively decreased (Figure 4C). Here, task latency refers to the time required by the mice to complete the task within the 30-s trial window. For this measure, we included all trials, both successful and unsuccessful. In the 2-ROI experiment where the task rule required, for example, “ROI1 - ROI2” activity to cross a threshold for reward delivery, mice displayed divergent strategies. Some animals predominantly increased ROI1 activity, whereas others reduced ROI2 activity, both approaches could lead to successful threshold crossing (Figure 11; see average reward centered responses).

Figure 5:

Figure 6:

In CLMF training, the real-time speed of a selected (also referred to as control point (CP) in terms of tracked points) point was mapped to the graded audio tone generator function. We trained the mice with the FLL bottom as the CP for auditory feedback and reward generation (example trial in Animation 2). Mice reached 80% performance (Figure 4B, CLMF Rule-change, No-rule-change) in the task within four days of training (RM ANOVA, p = 8.03e-7). They were eliciting the target behavior (i.e., moving CP at high speed) more frequently on later days compared to the first day of training (Figure 5B), and the correlations of CP speed profiles became more pronounced during the trial periods as compared to the rest period (Figure 8A).

Mice can rapidly adapt to changes in the task rule

We have assessed how mice respond to changes in closed-loop feedback rules. CLNF Rule-change mice went through a change in regulated cortical ROI(s) (Figure 6A, 6B) after being trained on an initial set of ROI(s) (Table 5, Rule-change). The new ROI(s) were chosen from a list of candidate ROI(s) for which the cortical activations were still low (see “Determining ROI(s) for change in CLNF task rule” under Methods). Reward threshold values were then recalculated for the ROI(s). A full list of ROI(s) we tried in separate mice is listed in Table 5. Mouse performance degraded on the first day of the rule switch to the new ROIs (ANOVA, p=8.7e-9), compared to the previous day (Figure 4A), and quickly recovered within five days (RM ANOVA p=8.3e-10) of further training. All new cortical ROI(s) appeared to support recovery to the pre-perturbation success rate (Table 5), and data were pooled across mice. This can also be observed in the bivariate distribution of a whole session ΔF/F₀ values in ROI1 vs ROI2 on day 9 (before rule change) and day 19 (last day after rule change) in Figure 5C. Additionally, the linear regression analysis of whole-session ΔF/F₀ activity after rule change in the 2-ROI experiment revealed a shift in the regression line slope, favoring the ROI that was required to increase according to the new task rule (Figure 6C). Following the rule change, CLNF task latency increased substantially and persisted at a high level across subsequent sessions, eventually declining as task performance adapted to the new rule (Figure 4C).

Similarly, in the CLMF experiments, the target body part was changed from FLL to FLR for Rule-change mice on day 5 and was associated with a significant drop in their success rate from 75-80% to ∼40% (ANOVA p=0.008, Figure 4B). Surprisingly, mice quickly adapted to the rule change and started performing above 70% on day 6 (example trial in Animation 3). Looking closely at their reward rate on day 5 (day of rule change), they had a higher reward rate in the second half of the session as compared to the first half (Supplementary Figure 4A), indicating they were adapting to the rule change within one session. This can also be observed in the bivariate distribution of left and right paw speeds on day 4 (before rule change) and day 10 (last day after rule change) in Figure 5D. As the mice were learning the CLMF task (increasing performance), task latency decreased (Figure 4D). Task latency in this context is the time taken by mice to perform the task within a trial period of 30 s. We included all trials, both successful and unsuccessful, in our calculation. Given that the maximum trial duration is 30 s, the task latency was capped at 30 s. Following the trend in task performance, the task latency started decreasing during the first four days but increased on day 5 (rule change) and started to drop again afterward.

We also examined the average paw speeds and distributions during trial periods. It is worthwhile noting that for the rule-change group, left forelimb speeds were higher than right forelimb speeds from day 1 to day 4 (when task rule was to move left forelimb). When the rule was switched from left forelimb to right forelimb on day 5, left forelimb speeds dropped below the right forelimb speeds (Supplementary Figure 4E).

Graded feedback helps in task exploration during learning, but not after learning

To investigate the role of audio feedback in our task, we also had a group with no task-related graded audio feedback (CLMF No-feedback) and instead received audio with a constant frequency (1 kHz) throughout the trials. CLMF No-rule-change mice who received continuous graded auditory feedback significantly improved their task performance (CLMF No-rule-change RM-ANOVA p = 9.6e-7) and outperformed the CLMF No-feedback mice (Figure 4B) very early (No-feedback RM-ANOVA p = 0.49), indicating the positive role of graded feedback for task exploration and learning the association. When graded audio feedback was removed for CLMF Rule-change mice on day 10, it did not affect their task performance, indicating the feedback was not essential to keep performing the task that they had already learned.

Cortical responses became focal and more correlated as mice learned the rewarded behavior in CLMF

There are reports of cortical plasticity during motor learning tasks, both at cellular and mesoscopic scales (Makino et al. 2017; Huber et al. 2012; W. E. Allen et al. 2017). As mice become proficient in a task, their brain activity becomes more focused and less widespread (Figure 9). We noticed the peak activity in different regions of the cortex around the rewarding movement decreased ΔF/F₀ in amplitude over days (Figure 9A, 9B, 9C, 9E). To quantify this, we measured the peak ΔF/F₀ (ΔF/F₀peak) value in the time window from -1s to +1s relative to the body part speed threshold crossing event in each cortical region. Consistent with our observations, ΔF/F₀peak in several cortical regions gradually decreased over sessions, including the olfactory bulb, sensory forelimb, and primary visual cortex (Figure 9A, 9B, 9C). Notably, when the task rule was changed from FLL to FLR in the CLMF Rule-change mice, we observed a significant increase in ΔF/F₀peak in regions including the olfactory bulb (OB), forelimb cortex (FL), hindlimb cortex (HL), and primary visual cortex (V1), as shown in Figure 9. We believe the decrease in ΔF/F₀peak is unlikely to be driven by changes in movement, as movement amplitudes did not decrease significantly during these periods (Figure 9D CLMF Rule-change). However, the decrease in ΔF/F₀peak followed the same trend as task latency (Figure 4D), suggesting that the decrease in ΔF/F₀peak is especially prominent in trials where mice were prepared to make the fast movement.

Figure 7:

Figure 8:

Figure 9:

These results suggest that motor learning led to less cortical activation across multiple regions, which may reflect more efficient processing of movement-related activity. In addition to ΔF/F₀peak reflecting signs of potentially more efficient cortical signaling, intracortical GCAMP transients measured over days became more correlated (to each other) during the trials as the task performance increased over the sessions (Figure 9E). Analysis of pairwise correlations between cortical regions (referred to as seed pixel correlation maps) revealed distinct network activations during rest and trial periods (Figure 10). While the general structure of the correlation maps remained consistent between trial and rest periods, certain correlations, such as between V1 and RS, were heightened during task trials, whereas others, such as between M1, M2, FL, and HL, consistently increased over the training sessions (Figure 10).

Figure 10:

To statistically examine the differences in seed pixel correlations during trials and rest periods, as well as how these correlations changed over training sessions (Day 1-10), we conducted a two-way repeated measures ANOVA (RM-ANOVA) on the seed pixel correlation maps for each day. The two variables for the RM-ANOVA were experimental condition (trial vs. rest) and session number (Day 1-10). This analysis generated two distinct matrices of Bonferroni-corrected p-values (Figure 10), one corresponding to each variable, which segregated the seed pixel correlations that were different between trial and rest periods and those that changed over the training sessions.

Distinct task- and reward-related cortical dynamics

During the early sessions (days 1 to 3), cortical activity was observed to be spatially widespread and engaged multiple cortical regions. Temporally, the activity spanned both task-related and reward-related events, with no clear distinction between the two phases (Figure 9B, left). This broad activation pattern is consistent with the initial stages of learning, where the brain recruits extensive cortical networks to process novel tasks and integrate sensory feedback (Peters, Chen, and Komiyama 2014).

As the mouse performance improved in the later sessions (Days 8 to 10), the cortical activity became more segregated both spatially and temporally (Figure 9B, middle). This segregation was particularly notable in mice that received closed-loop feedback (Rule-change), where the spatiotemporal patterns of cortical activation were more closely aligned with the specific task and reward events. This transition from widespread to segregated activation is indicative of the brain’s optimization of neural resources as the task becomes more familiar, a phenomenon that has been reported in studies of skill acquisition and motor learning (Makino et al. 2017; Huber et al. 2012). Previous studies have shown that feedback, especially when provided in a temporally precise manner, can accelerate the cortical plasticity associated with learning (Ganguly and Carmena 2009). Overall, these findings highlight the importance of closed-loop feedback in motor learning and suggest that real-time neurofeedback can enhance the specificity of cortical representations, potentially leading to more efficient and robust learning outcomes.

Figure 11:

Discussion

Flexible, cost-effective, and open-source system for a range of closed-loop experiments

We developed a versatile, cost-efficient, and open-source platform designed to support a wide array of closed-loop neuroscience experiments, including closed-loop neural feedback (CLNF) and closed-loop movement feedback (CLMF). This system enhances accessibility for the broader research community, enabling the execution of complex experimental paradigms with minimal resource constraints. Our system is built on readily available hardware components, such as Raspberry Pi and Nvidia Jetson platforms, and leverages Python-based software for real-time data processing, analysis, and feedback implementation. The modular approach ensures that the system can be easily customized to meet the specific requirements of different experimental paradigms. Our study demonstrates the effectiveness of a versatile and cost-effective closed-loop feedback system for modulating brain activity and behavior in head-fixed mice. By integrating real-time feedback based on cortical GCaMP imaging and behavior tracking, we provide strong evidence that such closed-loop systems can be instrumental in exploring the dynamic interplay between brain activity and behavior. The system’s ability to provide graded auditory feedback and rewards in response to specific neural and behavioral events showcases its potential for advancing research in neurofeedback and behavior modulation.

The hardware backbone of our system is designed around the Raspberry Pi 4B+ and Nvidia Jetson devices, chosen for their compactness, low cost, high computational power, and wide community support. These platforms have been demonstrated to be effective in neuroscience research, for several applications (Michelson et al. 2023; Murphy et al. 2020; Silasi et al. 2018; Dhillon et al. 2021). By utilizing open-source software frameworks such as Python, the system can be modified and extended to incorporate additional functionalities or adapt to new experimental protocols.

Our Python-based software stack includes libraries for real-time data acquisition, signal processing, and feedback control. For instance, we utilize OpenCV for video processing and DeepLabCut-Live (A. Mathis et al. 2018; Kane et al. 2020) for pose estimation, which are essential components in experiments requiring precise monitoring and feedback based on animal behavior. Additionally, we have integrated libraries for handling neural data streams, such as NumPy and SciPy, which facilitate the implementation of complex experimental designs (Akam et al. 2022) involving multiple data sources and feedback modalities. By integrating the system with LED drivers and opsin-expressing transgenic mouse lines, it is straightforward to achieve precise temporal control over neural activation, enabling the study of causal relationships between neural circuit activity and behavior (Lee et al. 2020).

The cost-effectiveness of our system is a significant advantage, making it accessible to a broader range of research labs, including those with limited funding. The use of off-the-shelf components and open-source software drastically reduces the overall cost compared to commercially available systems, which often require expensive proprietary hardware and software licenses (C. Allen and Mehler 2019; White et al. 2019). The open-source nature of our closed-loop platform fosters collaboration and innovation within the research community. By providing unrestricted access, other labs can freely modify, enhance, and share the system, promoting a culture of transparency and shared progress. Importantly, our goal is to increase the number of studies utilizing this platform, enabling a broader exploration of closed-loop methodologies across diverse research contexts. This approach not only accelerates the pace of discovery but also enhances reproducibility and adaptability, ensuring the platform remains a robust tool for advancing neuroscience research.

The system’s flexibility is demonstrated by its successful application across two closed-loop experimental paradigms (CLNF, CLMF). For example, in our study, we utilized the system to implement real-time feedback based on intracellular calcium-induced fluorescence imaging in awake, behaving mice. The system provided auditory feedback in response to changes in cortical activation, allowing us to explore the role of real-time feedback in modulating both neural activity and behavior.

Importance of closed-loop feedback systems

Closed-loop feedback systems have gained recognition for their ability to modulate neural circuits and behavior in real time, an approach that aligns with the principles of motor learning and neuroplasticity. The ability of mice to learn and adapt to tasks based on cortical or behavioral feedback underscores the parallels between closed-loop systems and natural proprioceptive mechanisms, where continuous sensory feedback is crucial for motor coordination and spatial awareness. This study builds upon the foundational work of (Fetz 1969), who first explored the potential of closed-loop neurofeedback, and expands its application to modern neuroscience by incorporating optical brain-computer interfaces.

Our findings are consistent with previous research showing that rodents can achieve volitional control over externally represented variables linked to their behavior (Clancy et al. 2014; Neely, Piech, et al. 2018; Prsa, Galiñanes, and Huber 2017; Clancy and Mrsic-Flogel 2021). Moreover, the rapid adaptation observed in mice when task rules were altered demonstrates the system’s capacity to facilitate learning and neuroplasticity, even when the conditions for achieving rewards are modified. The quick recovery of task performance after a rule change, as evidenced by the improved performance within days of training, highlights the robustness of the closed-loop feedback mechanism. We have focused on large regional cortical GCAMP signals that are relatively slow in kinetics.

Neuroplasticity and cortical dynamics

The observation that cortical responses became more localized as mice learned the rewarded behavior aligns with established theories of motor learning, where neural efficiency improves with practice (Makino et al. 2016; Huber et al. 2012; C. B. Allen, Celikel, and Feldman 2003). As mice became more proficient in the task, the widespread cortical activity observed during the initial training sessions became more regionally localized, indicating more efficient neural processing. The reduction in ΔF/F₀peak values across sessions suggests that the brain becomes more efficient at processing task-relevant information, a phenomenon consistent with the optimization of neural circuits observed in skilled motor learning (Wolpert, Diedrichsen, and Flanagan 2011; Krakauer et al. 2019).

The distinct spatiotemporal patterns of cortical activation observed in mice receiving closed-loop feedback further support the role of real-time feedback in enhancing cortical plasticity. The pronounced segregation of task-related and reward-related cortical dynamics in the later training sessions indicates that closed-loop feedback facilitates the refinement of neural circuits involved in motor learning. These findings are in line with previous studies that have demonstrated the importance of temporally precise feedback in accelerating cortical reorganization and enhancing learning outcomes (Ganguly and Carmena 2009).

Value of the CLoPy platform in systems neuroscience

The CLoPy platform addresses a critical gap in systems neuroscience by providing a flexible and cost-effective closed-loop feedback system tailored to modulate neural activity and behavior in real time. Many current neuroscience studies assess brain activity and behavior separately, with analyses performed post hoc. CLoPy advances this paradigm by enabling researchers to directly investigate the dynamic interplay between these domains through real-time feedback, a fundamental principle of neuroplasticity and motor learning (Fetz 1969; Wolpert, Diedrichsen, and Flanagan 2011). This integration has broad implications for experimental neuroscience, allowing researchers to probe causal relationships between brain activity, behavior, and environmental contingencies in a controlled and scalable manner.

By enabling closed-loop experiments based on neural imaging (e.g., GCaMP fluorescence) and real-time behavioral tracking (e.g., DeepLabCut-Live), CLoPy allows researchers to address a range of experimental questions. For example, the system can be used to study how cortical plasticity facilitates motor learning, how the brain adapts to rule changes in a task, and how feedback mechanisms influence recovery from neurological injuries. These capabilities align with the needs of systems neuroscience researchers who seek to elucidate the mechanisms underlying adaptive brain-behavior interactions across different experimental conditions. Compared to other platforms, such as pyControl (Akam et al. 2022), CLoPy uniquely integrates mesoscopic calcium imaging, real-time tracking, and feedback delivery, making it particularly suitable for exploring neural dynamics at a finer resolution.

Positioning within the open-source neuroscience community

As part of the broader open-source neuroscience movement, CLoPy exemplifies the principles of accessibility, modularity, and reproducibility. The platform’s reliance on widely available hardware, such as Raspberry Pi and Nvidia Jetson, and its use of Python-based libraries ensure that researchers with varying levels of technical expertise can adopt and modify it. This is particularly important in the current landscape, where open-source tools like pyControl (Akam et al. 2022), OpenEphys (Siegle et al. 2017), and Bonsai (Lopes and Monteiro 2021) have transformed experimental neuroscience by reducing costs and fostering collaboration.

CLoPy complements these efforts by extending open-source frameworks to the realm of closed-loop neuroscience. It provides detailed documentation, hardware schematics, and adaptable software, empowering researchers to customize the system for diverse experimental needs. For example, CLoPy enables experiments involving optogenetic stimulation, multimodal feedback, and integration with advanced imaging techniques. By lowering barriers to entry, it democratizes access to closed-loop experimental paradigms, particularly for labs with limited resources.

Experimental applications and implications

CLoPy’s flexibility and modularity make it suitable for a wide range of experimental applications. In this study, we demonstrated its utility in closed-loop neural feedback (CLNF) and closed-loop movement feedback (CLMF) paradigms. These experiments showcase how CLoPy can be used to study fundamental questions about the dynamic interplay between neural activity, feedback, and behavior. For instance, the system’s ability to deliver graded auditory feedback in response to cortical activation allowed us to explore how real-time neurofeedback modulates neural dynamics. Similarly, its real-time tracking capabilities enabled precise feedback delivery based on specific behavioral events, such as limb movements, facilitating the investigation of motor learning and adaptation. In our system, we measured latencies of ∼63 ms for CLNF and ∼67 ms for CLMF. While such latencies may limit applications requiring millisecond precision, such as fast whisker movements, saccades, or fine-reaching kinematics, we emphasize that many relevant behaviors, including postural adjustments, limb movements, locomotion, and sustained cortical state changes, occur on timescales that are well within the capture range of our system. It is also important to note that these latencies are not solely dictated by hardware constraints. A significant component arises from the inherent biological dynamics of the calcium indicator (GCaMP6s) and calcium signaling itself, which introduce slower temporal kinetics independent of processing delays. Newer variants, such as GCaMP8f, offer faster response times and could further reduce effective biological latency in future implementations. We acknowledge that Raspberry Pi provides a low-cost solution but contributes to modest computational delays, while Nvidia Jetson offers faster inference at higher cost. Our choice reflects a balance between accessibility, cost-effectiveness, and performance, making the system deployable in many laboratories. Importantly, the modular and open-source design means the pipeline can readily be adapted to higher-performance GPUs or integrated with electrophysiological recordings, which provide higher temporal resolution. In the case of CLNF we have focused on large regional cortical GCAMP signals that are relatively slow in kinetics. While such changes are well suited for transcranial mesoscale imaging assessment, it is possible that cellular 2-photon imaging (Yu et al. 2021) or preparations that employ cleared crystal skulls (Kim et al. 2016) could resolve more localized and higher frequency kinetic signatures. In this case it would be important to potentially tune and enhance closed-loop methodology to more closely follow these events.

The platform’s ability to modulate neural and behavioral outputs in real time has far-reaching implications for experimental neuroscience. It enables researchers to study neural mechanisms underlying learning, adaptation, and plasticity with a level of precision and scalability that was previously difficult to achieve. For example, CLoPy can be used to investigate how the brain reorganizes itself after injury or how neural circuits adapt to changing task demands.

Future directions and broad impact

To further enhance its utility, future iterations of CLoPy could integrate advanced machine learning algorithms for real-time data analysis and decision-making, enabling more sophisticated feedback paradigms. Additionally, expanding the platform’s compatibility with other experimental modalities, such as multi-photon imaging and electrophysiology, could broaden its applications in neuroscience research. CLoPy’s scalability also opens avenues for its use in translational research, such as testing neurorehabilitation interventions or brain-computer interfaces in larger animal models or humans. By providing an open-source, flexible, and cost-effective solution, CLoPy has the potential to significantly advance the field of systems neuroscience. It not only enables researchers to address complex experimental questions but also fosters collaboration and innovation within the neuroscience community. As more labs adopt and refine this platform, we anticipate that it will accelerate the pace of discovery, enhance reproducibility, and contribute to the development of novel therapeutic strategies for neurological disorders. While we aimed to include multiple animals for each experimental condition, certain task rules are represented by only one or a few animals. This limitation primarily arises from logistical challenges, including the extensive time required for training under specific task contingencies and the variability in maintaining high-quality optical windows over prolonged experimental periods. Consequently, while these data provide valuable insights into the feasibility and mechanisms of the task, we acknowledge that broader generalizations should be made cautiously and that further replication will be important in future studies.

In conclusion, our study highlights the significant potential of closed-loop feedback systems for advancing neuroscience research. By providing a flexible, cost-effective, and open-source platform, we offer a valuable tool for exploring the complex interactions between brain activity and behavior, with implications for both basic research and clinical applications.

Materials and methods

Animals

Mouse protocols were approved by the University of British Columbia Animal Care Committee (ACC) and followed the Canadian Council on Animal Care and Use guidelines (protocol A22-0054). A total of 56 mice (postnatal 104-140) were used in this study: 26 female and 30 male transgenic C57BL/6 mice expressing GCaMP6s were used. CLNF experiments (n=40, 17 females, 23 males) were done with tetO-GCaMP6s x CAMK tTA (Wekselblatt et al., 2016), and CLMF experiments (n=16, 9 females, 7 males) were done with Ai94 from the Allen Institute for Brain Science, crossed to Emx1–cre and CaMK2-tTA line (Jackson Labs) (Madisen et al., 2015). Mice were housed in a conventional facility in plastic cages with micro-isolator tops and kept on a normal 12 hr. light cycle with lights on at 7 AM. Most experiments were performed toward the end of the mouse light cycle. Mice that were unable to achieve a success rate of 70% after 7 days of training in CLNF experiments were excluded from the study (total 6 mice).

Animal surgery, chronic transcranial window preparation

Animals were anesthetized with isoflurane, and a transcranial window was installed as previously described (Silasi et al. 2016; Vanni and Murphy 2014) and in an amended and more extensive protocol described here. A sterile field was created by placing a surgical drape over the previously cleaned surgical table, and surgical instruments were sterilized with a hot bead sterilizer for 20 s (Fine Science Tools; Model 18000–45). Mice were anesthetized with isoflurane (2% induction, 1.5% maintenance in air) and then mounted in a stereotactic frame with the skull level between lambda and bregma. The eyes were treated with eye lubricant (Lacrilube; www.well.ca) to keep the cornea moist, and body temperature was maintained at 37°C using a feedback-regulated heating pad monitored by a rectal probe. Lidocaine (0.1 ml, 0.2%) was injected under the scalp, and mice also received a 0.5 ml subcutaneous injection of a saline solution containing buprenorphine (2 mg/ml), atropine (3 μg/ml), and glucose (20 mM). The fur on the head of the mouse (from the cerebellar plate to near the eyes) was removed using a fine battery-powered beard trimmer, and the skin was prepared with a triple scrub of 0.1% Betadine in water followed by 70% ethanol. Respiration rate and response to toe pinch were checked every 10–15 min to maintain the surgical anesthetic plane.

Before starting the surgery, a cover glass was cut with a diamond pen (Thorlabs, Newton, NJ, USA; Cat#: S90W) to the size of the final cranial window (∼9 mm diameter). A skin flap extending over both hemispheres approximately 3 mm anterior to bregma and to the posterior end of the skull and down lateral was cut and removed. A #10 scalpel (curved) and sterile cotton tips were used to gently wipe off any fascia or connective tissue on the skull surface, making sure it was completely clear of debris and dry before proceeding. The clear version of C and B-Metabond (Parkell, Edgewood, NY, USA; Product: C and B Metabond) dental cement was prepared by mixing 1 scoop of C and B Metabond powder (Product: S399), 7 drops of C and B Metabond Quick Base (Product: S398), and one drop of C and B Universal catalyst (Product: S371) in a ceramic or glass dish (do not use plastic). Once the mixture reaches a consistency that makes it stick to the end of a wooden stir stick, a titanium fixation bar (22.2 × 2.7 × 3.2 mm) was placed so that there was a 4 mm posterior space between the bar edge and bregma, by applying a small amount of dental cement and holding it pressed against the skull until the cement partially dried (1–2 min). With the bar in place, a layer of dental adhesive was applied directly on the intact skull. The precut cover glass was gently placed on top of the mixture before it solidified (within 1 min), taking care to avoid bubble formation. If necessary, extra dental cement was applied around the edge of the cover slip to ensure that all the exposed bone was covered, and that the incision site was sealed at the edges. The skin naturally tightens itself around the craniotomy, and sutures are not necessary. The mixture remains transparent after it solidifies, and one should be able to clearly see large surface veins and arteries at the end of the procedure. Once the dental cement around the coverslip is completely solidified (up to 20 min), the animal received a second subcutaneous injection of saline (0.5 ml) with 20 mM of glucose and was allowed to recover in the home cage with an overhead heat lamp and intermittent monitoring (hourly for the first 4 hr. and every 4–8 hr. thereafter for activity level). Then, the mouse was allowed to recover for 7 days before task training.

Water deprivation and habituation to experiments

Around 10–21 days after the surgery, the animals were placed on a schedule of water deprivation. Given the variation in weight due to initial ad libitum water consumption, the mouse weight was defined 24 hr. after the start of water restriction. If mice did not progress well through training, they were still given up to 1 ml of water daily, there was also a 15% maximal weight loss criterion used for supplementation (see detailed protocol). All mice were habituated for 5 days before data collection. Awake mice were head-fixed and placed in a dark imaging chamber for training and data collection for each session. We tried to not keep animals headfixed for more than 45 minutes in each session as they become less engaged with long duration headfixed sessions. After headfixing them, it takes about 15 minutes to get the experiment going and therefore 30 - 40 minutes long recorded sessions seemed appropriate before they stop being engaged or before they get satiated in the task. High performing animals were able to maintain body weight and gain weight towards pre-surgery and pre-water restriction values.

CLNF, CLMF setup and behavior experiments

Our goal has been to deliver a robust, cross-platform, and cost-effective solution for closed-loop feedback experiments. We have designed and tested a behavioral paradigm where head-fixed mice learn an association between their cortical or behavioral activity, external feedback, and rewards. We tested our CLNF system on Raspberry Pi, and CLMF system on an Nvidia Jetson GPU device, for their compactness, general-purpose input/output (GPIO) programmability, and wide community support. While our investigations center around mesoscale cortical imaging of genetically encoded calcium sensors and behavior imaging, the approach could be adapted to any video or microscopy-dependent signal where relative changes in brightness or keypoint behavior are observed and feedback is given based on a specific rule. This system benefits from advances in pose estimation (Forys et al. 2020; Kane et al. 2020)and employs strategies to improve real-time processing of pre-defined keypoints on compact computers such as the Nvidia Jetson. We have constructed a software and hardware-based platform built around open-source components.

To examine the performance of the closed-loop system, we used water-restricted adult transgenic mice that expressed GCaMP6s widely in the cortex (details in the methods section and in Table 5 and Table 6). Using the transcranial window imaging technique, we assessed the ability of these animals to control brain regions of interest and obtain water rewards. After an initial period of habituation to manual head fixation, mice were switched to closed-loop task training. In training, we had two primary groups: 1) a single cortical ROI linked to water reward and the subject of auditory feedback (Animation 1A, 1B, 1C), or 2) a pairing between two cortical ROIs where auditory feedback and water rewards were given based on the difference in activity between sites (Animation 1D, 1E, 1F).

For neurofeedback-CLNF, we calculate an online ΔF/F₀ value (GCAMP signal) which represents a relative change of intensity in the region(s) of interest. In brief, a running baseline (F₀) is computed at each sample point, subtracted and divided (ΔF/F₀) from the raw fluorescence values; more details about online ΔF/F₀ calculation are in the methods section. These calculations are made in near real-time on the Raspberry Pi and are used to control the GPIO output pins that provide digital signals for closed-loop feedback and water rewards to water-restricted mice. In 1-ROI experiments, we mapped the range of average ΔF/F₀ values in the ROI (Figure 1) to a range of audio frequencies (1 kHz - 22 kHz), which acted as feedback to the animal. For 2-ROI experiments, the magnitude of the ΔF/F₀ activity difference between the ROIs, based on the specified rule (e.g., ROI1-ROI2), was mapped to the range of audio frequencies. We confirmed that these frequencies were accurately generated and mapped by audio recordings (Supplementary Figure 2 and see Methods) obtained at 200 kHz using an ultrasonic microphone (Dodotronic, Ultramic UM200K) positioned within the recording chamber ∼10 cm from the audio speaker. To confirm the timing of feedback latency, LED lights were triggered instead of water rewards (Supplementary Figure 1), and the delay was calculated between the detected event (green LED ON for CLNF and paw movement for CLMF) and the red LED flash. In the case of CLNF, the camera recording brain activity was used to record both the flashing green and red LEDs. Temporal traces of green and red LED pixels were extracted from the recorded video, and the average delay between the green and red LEDs becoming bright was calculated as the delay in closed-loop feedback for CLNF experiments. Performing this analysis indicated that the Raspberry Pi system could provide reliable graded feedback using GPIO within ∼63 ± 15 ms for CLNF experiments.

An imaging rig was developed using two Raspberry Pi 4B+ single-board computers, designated as “brain-pi” (master device for initiating all recordings) for widefield GCaMP imaging and “behavior-pi” (slave device waiting for a trigger to start recording) for simultaneous behavior recording. Both devices were connected to the internet via Ethernet cables and communicated with each other through 3.3 V transistor-transistor logic (TTL) via general-purpose input outputs (GPIOs). To synchronize session initiation, GPIO pin #17 on the brain-pi (configured as an output) was connected to GPIO pin #17 on the behavior-pi (configured as an input), allowing the brain-pi to send a TTL signal to the behavior-pi at the start of each session. Additionally, frames were aligned based on a LED ON event for each session.

Additional hardware components were integrated into the brain-pi setup. GPIO pin#27 (output) was connected to a solenoid (Gems Sensor, 45M6131) circuit to deliver water rewards, while GPIO pin#12 (output) was linked to a buzzer (Adafruit product #1739) positioned under the head-fixing apparatus to signal trial failures. GPIO pin#21 (output) was used to trigger a LED driver controlling both short blue (447.5 nm) and long blue (470 nm) LEDs, which were essential for cortical imaging. A speaker was connected to the brain-pi’s 3.5 mm audio jack to provide auditory output (PulseAudio driver) during the experiments. Both the brain-pi and behavior-pi devices were equipped with compatible external hard drives, connected via USB 3.0 ports, to store imaging and behavioral data, ensuring reliable data capture throughout the experimental sessions.

Mice with implanted transcranial windows on the dorsal cortex were head-fixed in a transparent acrylic tube (1.5-inch outer diameter, 1 ⅛ inch inner diameter) and placed such that the imaging camera (RGB Raspberry Pi Camera, OmniVision OV5647 CMOS sensor) was above the transcranial window, optimally focused on the cortical surface for GCaMP imaging. The GCaMP imaging cameras had lenses with a focal length of 3.6 mm and a field of view of ∼10.2 × 10.2 mm, leading to a pixel size of ∼40 μm, and were equipped with triple-bandpass filters (Chroma 69013m), which allowed for the separation of GCaMP epifluorescence signals and 447 nm reflectance signals into the green and blue channels, respectively. The depth of field (∼3 mm) was similar to previous reports (Lim et al. 2012) which provided both a large focal volume over which to collect fluorescence and made the system less sensitive to potential changes in the z-axis position. To reduce image file size, we binned data at 256 × 256 pixels on the camera for brain imaging data. Brain images were saved as 8-bit RGB stacks in HDF5 file format. We manually fixed the camera frame rate to 15 Hz, turned off automatic exposure and auto white balance, and set white balance gains to unity. For both green epifluorescence and blue reflection channels, we adjusted the intensity of illumination so that all values were below 180 out of 256 grey levels (higher levels increase the chance of cross-talk and saturation).

Mesoscale GCaMP imaging can often be performed with single-wavelength illumination (Gilad and Helmchen 2020; Nakai et al. 2023). However, in this experiment, we utilized dual-LED illumination of the cortex (Michelson et al. 2023). One LED (short-wavelength blue, 447 nm Royal Blue Luxeon Rebel LED SP-01-V4 paired with a Thorlabs FB 440-10 nm bandpass filter) monitored light reflectance to account for hemodynamic changes (Xiao et al. 2017), while the second LED (long-wavelength blue, 470 nm Luxeon Rebel LED SP-01-B6 combined with a Chroma 480/30 nm filter) was used to excite GCaMP for green epifluorescence. Both signals were captured simultaneously using an RGB camera. For each mouse, light from both the excitation and reflectance LEDs was channeled into a single liquid light guide, positioned to illuminate the cortex (Figure 1v) (Michelson et al. 2023). Further specifics are outlined in the accompanying Parts List and assembly instructions document. A custom-built LED driver, controlled by a Raspberry Pi, activated each LED at the beginning of the session and deactivated them at the session’s end. This on-off illumination shift was later used in post hoc analysis to synchronize frames from the brain and behavior cameras.

Before initiating the experimental session, key session parameters were configured in the config.ini file, as detailed in the “Key Configuration Parameters” section below. Additionally, we ensured that the waterspout was positioned appropriately for the mouse to access and consume the reward. To launch the experiment, open a terminal on the brain-pi (master) device, ensuring that the current directory is set to clopy/brain/. The experiment can be started by entering the command python3 <SCRIPT_NAME<.py, where <SCRIPT_NAME> corresponds to the appropriate Python script for the session (Supplementary Figure 7). We provide two pre-defined scripts in the codebase: one for 1ROI and another for 2ROI experiments. Both scripts are functionally similar. Upon execution, the script prompts for the “mouse_id,” which can be entered as an array of characters, followed by pressing ‘Enter.’ Upon initialization, two preview windows appear. The first window displays live captured images with intensity values overlaid in the green and blue channels, which allow for adjustments to LED brightness levels. The second window displays real-time ΔF/F₀ values overlaid on cortical locations, enabling brain alignment checks. These preview windows are used to assess imaging quality and ensure appropriate settings before starting the experiment (Supplementary Figure 6). Once all settings are confirmed, pressing the ‘Esc’ key starts the experiment. At this point, only one window displaying incoming images is shown. The experiment begins with a rest period of duration specified in the config.ini file, followed by alternating trial and rest periods. Data acquired during the session are saved in real-time on the device. For more details and the latest instructions on system setup, please refer to the project’s GitHub page.

An additional imaging rig for CLMF was developed utilizing an Nvidia-Jetson Orin device (8-core ARM Cortex CPU, 2048 CUDA cores, 64 GB memory), which served as the master device responsible for triggering all recordings. The Nvidia-Jetson was connected to an Omron Sentech STC-MCCM401U3V USB3 Vision camera for behavior imaging and real-time pose tracking. A personal computer (serving as the slave device) running EPIX software and frame grabber (PIXCI® E4 Camera Link Frame Grabber) was connected to a Pantera TF 1M60 CCD camera (Dalsa) for widefield GCaMP imaging. Both the Nvidia-Jetson and the EPIX PC were linked via transistor-transistor logic (TTL) for communication. For session synchronization, the GPIO pin #17 on the Nvidia-Jetson, configured as an output, was connected to the trigger input pin of the EPIX framegrabber E4DB. This configuration enabled the Nvidia-Jetson to send a TTL signal to initiate frame capture on the EPIX system at the start of each session. Additionally, this same pin was connected to an LED driver to trigger the activation of a long blue LED (470 nm) for GCaMP imaging simultaneously.

Several hardware components were integrated into the Nvidia-Jetson setup to support experimental protocols. GPIO pin #13 (output) was linked to a solenoid circuit to deliver water rewards, while GPIO pin #7 (output) was connected to a buzzer placed under the head-fixation apparatus to signal trial failures. Auditory feedback during the experiments was provided through a speaker connected to the Nvidia-Jetson’s audio output. Both the Nvidia-Jetson and EPIX PC were equipped with sufficient storage space to ensure the reliable capture and storage of behavioral video recordings and widefield image stacks, respectively, throughout the experimental sessions.

Mice, implanted with a transcranial window over the dorsal cortex, were head-fixed in a custom-made transparent, rectangular chamber (dimensions provided). The chamber was designed using CAD software and fabricated from a 3 mm-thick acrylic sheet via laser cutting. Complete model files, acrylic sheet specifications, and assembly instructions are available on the GitHub repository. A mirror was positioned at the bottom and front of the chamber (Figure 1B) to allow multiple views of the mouse for improved tracking accuracy. The Omron Sentech camera was equipped with an infrared (IR)-only filter to exclusively capture IR illumination, effectively blocking other light sources and ensuring consistent behavioral imaging for accurate pose tracking. For widefield GCaMP imaging, the Dalsa camera was equipped with two front-to-front lenses (50 mm, f/1.435 mm and f/2; Nikon Nikkor) and a bandpass emission filter (525/36 nm, Chroma). The 12-bit images were captured at a frame rate of 30 Hz (exposure time of 33.3 ms) with 8×8 on-chip spatial binning (resulting in 128x128 pixels) using EPIX XCAP v3.8 imaging software. The cortex was illuminated using a blue LED (473 nm, Thorlabs) with a bandpass filter (467–499 nm) to excite calcium indicators, and the blue LED was synchronized with frame acquisition via TTL signaling. GCaMP fluorescence image stacks were automatically saved to disk upon session completion.

Prior to starting a session, key experimental parameters were configured within the config.ini file (see “Key Configuration Parameters” below for details). The waterspout was positioned close to the mouse to allow easy access for licking and consuming rewards. To initiate the experiment on the Nvidia-Jetson (master device), the terminal’s current directory was set to clopy/behavior/. The experiment was launched by executing the command python3 <SCRIPT_NAME>.py, where <SCRIPT_NAME> corresponds to the Python script responsible for running the experiment (Supplementary Figure 6). The provided script maps the speed of a specific control-point (FLL_bottom) to the audio feedback and executes 60 trials. Once launched, the program prompts for the “mouse_id,” which could be entered as an array of characters, followed by pressing Enter. A preview window displaying a live behavioral view was then presented, allowing for adjustments of IR brightness levels and the waterspout (Supplementary Figure 6). Once imaging quality and settings were confirmed to be optimal, pressing the “esc” key started the experiment. During the experiment, the tracked points were overlaid on the real-time video, and the session alternated between rest and trial periods. All acquired data were saved locally on the Nvidia-Jetson during the experiment and automatically transferred to the EPIX PC after completion. Additional setup instructions and system details can be found on the associated GitHub page.

Key configuration parameters

Prior to running the experiment script, session parameters were set in the config.ini file under a designated configuration section. This file contains multiple config sections, each tailored to a different experiment type, such as brain-pi or behavior-pi. The appropriate config section is specified in the experiment script. Key parameters in this section are listed in the table below.

Table 1:

Further details on these parameters can be accessed via the GitHub repository.

CLoPy platform

Closed-Loop Feedback Training System (CLoPy) is an open-source software and hardware system implemented in the Python (>=3.8) programming language. This work is accompanied by a package to replicate the system, reproduce figures in this publication, and an extensive supplemental guide with full construction illustrations and parts lists to build the platform used. See https://github.com/pankajkgupta/clopy for details and accompanying acquisition and analysis code. We also provide model files for machined and 3D-printed parts in the repository; links to neural and behavioral data can also be found at the federated research data repository- https://doi.org/10.20383/102.0400.

While the presented CLNF experiments were conducted on a Raspberry Pi 4B+ device, the system can be used on any other platform where the Python runtime environment is supported. Similarly, CLMF experiments were conducted on an Nvidia Jetson Orin device, but it can be deployed on any other device with a GPU for real-time inference. For any programmable camera to be used with the system, one can implement a wrapper Python class that implements the CameraFactory interface functions for integration with the system.

Graded feedback (online ΔF/F₀-audio mapping)

The auditory feedback was proportional to the magnitude of the neural activity. We translated fluorescence levels into the appropriate feedback frequency and played the frequency on speakers mounted on two sides of the imaging platform. Frequencies used for auditory feedback ranged from 1 to 22 kHz in quarter-octave increments (Han et al. 2007; Clancy et al. 2014). When a target was hit, a circuit driven solenoid delivered a reward to mice.

To map the fluorescence activity 𝐹 to the quarter-octave index 𝑛, we used linear scaling. Let 𝐹_𝑚𝑖𝑛 and 𝐹_max be the minimum and maximum fluorescence values.𝑛_𝑚𝑎𝑥 The maximum number of quarter-octave steps between 1 kHz and 22 kHz.

The number of quarter-octave steps between 1 kHz and 22 kHz can be calculated as:

The quarter-octave index 𝑛 can be mapped linearly from fluorescence activity 𝐹 as:

Thus, the final equation mapping fluorescence activity to audio frequency is:

where 𝑓_𝑚𝑖𝑛 = 1𝑘𝐻𝑧, 𝑓_𝑚𝑎𝑥 = 22𝑘𝐻𝑧, and 𝑛_𝑚𝑎𝑥 = 4 · 𝑙𝑜𝑔₂(22) ≈ 13. 45

This equation allows us to map any fluorescence activity 𝐹 within the range [𝐹_min, 𝐹_max] to a corresponding frequency within the 1–22 kHz range, in quarter-octave increments.

Checking the dynamic range of graded auditory feedback

To assess the dynamic range of audio signals from the speakers, we used an Ultra microphone (Dodotronic) to record feedback signals generated during a session, detect the frequencies, and compare the detected frequencies through our speakers to what was commanded through the program. We verified a linear relationship between 1 and 22 kilohertz (Supplementary figure 2). Analysis of GCAMP imaging experiments indicated that baseline activity values were associated with mapped sounds in the 6±4 kHz range. At reward points, which are threshold-dependent (Delta F over F values), the commanded auditory feedback values were significantly higher (18±2 kHz range). A previous version of the CLNF system was found to have non-linear audio generation above 10 kHz, partly due to problems in the audio generation library and partly due to the consumer-grade speaker hardware we were employing. This was fixed by switching to the Audiostream (https://github.com/kivy/audiostream) library for audio generation and testing the speakers to make sure they could output the commanded frequencies (supplementary figure 2).

Determining reward threshold based on a baseline session

Before starting the experiments, and after the habituation period, a baseline session (day 0) of the same duration as the experiments is recorded. This session is similar to the experimental sessions in the coming days, except that the mice do not receive any rewards. Offline analysis of this session is used to establish a threshold value for the target activity (see the target activity section to read more). In brief, for 1 ROI experiments, target activity is the average ΔF/F₀ activity in that ROI. For 2 ROI experiments, target activity is based on the specified rule in the config.ini file. For example, target activity for the rule “ROI1-ROI2” would be “average ΔF/F₀ activity in ROI1 - average ΔF/F₀ activity in ROI2.”

Determining ROI(s) for changes in CLNF task rules

Dorsal cortical widefield activity is dynamic, and ongoing spatiotemporal motifs involve multiple regions changing activity. Choosing ROI(s) for CLNF has some caveats and requires some considerations before choosing. It was relatively straightforward for initial training experiments where baseline (day 0) sessions were used to establish a threshold for the selected ROI on day 1. Mice learn to modulate the selected ROI over the training sessions. Interestingly, other cortical ROIs were also changing along with the target ROI as mice were learning the task. We needed to be careful when changing the ROI on day 11 because if we changed to an ROI that also changes along with the previous ROI, mice could keep getting rewards without realizing any change. To address this issue, we analyzed the neural data from day 1 to day 10 and found potential ROIs for which the threshold crossings did not increase significantly or were not on par with the previous ROI, and one of these ROIs was chosen for the rule change.

Online ΔF/F₀ calculation

In calcium imaging, ΔF/F₀ is often used to represent the change in fluorescence relative to a baseline fluorescence (F₀), which helps in normalizing the data. For CLNF real-time feedback, we computed ΔF/F₀ online, frame by frame.

Let 𝐹(𝑡) represent the fluorescence signal at time.𝑡. A running baseline 𝐹₀ (𝑡)is estimated by applying a sliding window to the fluorescence signal to calculate a moving average over a window of size 𝑁.

Once the running baseline.𝐹₀ (𝑡)is computed, the ΔF/F₀ at time 𝑡 is calculated as:

Each incoming frame contained both a green channel (capturing GCaMP6s fluorescence) and a short blue channel (representing blood volume reflectance) (Ma et al. 2016; Wekselblatt et al. 2016; Valley et al. 2020). These frames were appended to a doubly-ended queue (deque), a Python data structure, for running baseline 𝐹₀ (𝑡)The length of the deque.𝑁 was defined by the product of two parameters from the configuration file: “dff_history” (in seconds) and “framerate” (in frames per second), which determined the number of frames used for the running baseline. For the CLNF experiments, we used a dff_history of 6 s (a dynamic baseline) and a framerate of 15 fps, resulting in a deque length of 90 frames. When the deque reached full capacity, appending a new frame automatically removed the oldest frame, ensuring an updated running ΔF/F₀ throughout the experiment, regardless of session duration. This method effectively avoided memory limitations over time. We would like to emphasize that we are not directly averaging activity over 6 s to compare against the reward threshold. Instead, the preceding 6 s of activity is used solely to compute a dynamic baseline for ΔF/F₀ (ΔF/F₀ = (F – F₀)/F₀). Here, F₀ is calculated as the mean fluorescence intensity over the prior 6 s window and is updated continuously throughout the session. This baseline is then subtracted from the instantaneous fluorescence signal to detect relative changes in activity. The reward threshold is therefore evaluated against these baseline-corrected ΔF/F₀ values at the current time point, not against an average over 6 s. This moving-window baseline correction is a standard approach in calcium imaging analyses, as it helps control for slow drifts in signal intensity, bleaching effects, or ongoing fluctuations unrelated to the behavior of interest. Thus, the 6 s window is not introducing a temporal lag in reward assignment but is instead providing a reference to detect rapid increases in cortical activity. For each channel (green and blue), the running F₀ was subtracted from each incoming frame to obtain ΔF, followed by ΔF/F₀ calculations, applied per pixel as a vectorized operation in Python. To correct for hemodynamic artifacts (Ma et al. 2016), we subtracted the blue excitation and epifluorescence channel ΔF/F₀ from the green channel ΔF/F₀.

It is important to note that while blood volume reflectance is typically captured using green light (Ma et al. 2016), we used short blue light due to technical constraints associated with the Raspberry Pi camera’s rolling shutter which made strobing infeasible. The short blue light (447 nm) with a 440 ± 5 nm filter is close to the hemoglobin isosbestic point and has been shown to correlate well with the 530 nm green light signal as a proxy for hemodynamic activity (Michelson et al. 2023; Murphy et al. 2020). Additionally, the 447 nm LED would be expected to produce minimal green epifluorescence at the low power settings used in our experiments (Dana et al. 2014). Previous studies have evaluated and compared the performance of corrected versus uncorrected signals using this method (Xiao et al. 2017; Murphy et al. 2020).

Offline ΔF/F₀ calculation

For offline analysis of GCaMP6s fluorescence in CLNF experiments, the green and blue channels (Ma et al. 2016; Wekselblatt et al. 2016; Valley et al. 2020) were converted to ΔF/F₀ values. For each channel, a baseline image (F₀) was computed by averaging across all frames of the recording session. The F₀ was then subtracted from each individual frame producing a difference image (ΔF). This difference was divided by F₀, resulting in the fractional change in intensity (ΔF/F₀) for each pixel as a function of time. To further correct for hemodynamic artifacts (Ma et al., 2016), the blue channel reflected light ΔF/F₀ signal, reflecting blood volume changes, was subtracted from the green channel ΔF/F₀ signal, isolating the corrected GCaMP6s fluorescence response from any potential confounding vascular contributions.

Offline analysis of GCaMP6s fluorescence in CLMF experiments involved processing the green epifluorescence channel. We did not collect the hemodynamic signal in this experiment because we intended to employ the second channel for optogenetic stimulation (not included in this article). A baseline image (F₀) was computed by averaging across all frames of the recording session. The F₀ was then subtracted from each individual frame, producing a difference image (ΔF). This difference was divided by F₀, resulting in the fractional change in intensity (ΔF/F₀) for each pixel as a function of time.

Seed-pixel correlation matrices

Widefield cortical image stacks were registered to the Allen Mouse Brain Atlas (Wang et al. 2020; Chon et al. 2019) and segmented into distinct regions, including the olfactory bulb (OB), anterolateral motor cortex (ALM), primary motor cortex (M1), secondary motor cortex (M2), sensory forelimb (FL), sensory hind limb (HL), barrel cortex (BC), primary visual cortex (V1), and retrosplenial cortex (RS) in both the left and right cortical hemispheres. The average activity in a 0.4 x 0.4 mm² area (equivalent to a 10 x 10 pixel region) centered on these regions (also referred to as seed pixels) was calculated, representing the activity within each region. These signals were then used to generate temporal plots. The temporal plots were epoched into trial and rest conditions, and correlations between the regions were computed, resulting in correlation matrices for each condition across each day. Each element of the matrix represents the pairwise correlation between two cortical regions. For each pair of cortical regions, we obtained correlation values during both trial and rest periods for every session, creating a time series (over sessions) with two conditions (trial and rest).

Statistics

Various statistical tests were performed to support the analysis presented in accompanying figures. For p-value matrices in Figure 10B, 10C, a two-way repeated-measures ANOVA with Bonferroni post-hoc correction was used to test the significance of correlation changes across two factors: sessions (days 1–10) and condition (trial vs. rest). Significant changes in the correlation matrices along these two variables showed complementary patterns (Figure 10). Changes across sessions involved the bilateral M1, M2, FL, HL, and BC regions, while changes between the trial and rest conditions involved bilateral OB, ALM, V1, and RS regions.

A list of statistical tests used in a figure, its purpose and data used are summarized in the table below.

Table 2:

CLNF rules

The task rules for CLNF experiments involved cortical activity at selected locations. These locations were standard Allen Mouse Brain Atlas (Wang et al. 2020; Chon et al. 2019) CCF coordinates. The table below lists all the task rules we tested and what they meant in the context of CLNF experiments.

Table 3:

CLMF rules

The task rules for CLMF experiments involved points on body parts that were tracked (listed in results section). The table below lists the task rules (i.e. tracked points) we tested and what they meant in the context of CLMF experiments.

Table 4:

Table 5:

Table 6:

Data availability

The source data used in this paper is available here- https://doi.org/10.20383/103.01152. Code to replicate the system, recreate the figures, and associated pre-processed data are publicly available and hosted on GitHub - https://github.com/pankajkgupta/clopy. Any additional information required to reanalyze the data reported in this work is available from the lead contact upon request.

Acknowledgements

This work was supported by Canadian Institutes of Health Research (CIHR) foundation grant FDN-143209 and project grant PJT-180631 (to T.H.M.). T.H.M. was also supported by the Brain Canada Neurophotonics Platform, a Heart and Stroke Foundation of Canada grant in aid, the National Science and Engineering Council of Canada (NSERC; GPIN-2022-03723), and a Leducq Foundation grant. This work was supported by resources made available through the Dynamic Brain Circuits cluster and the NeuroImaging and NeuroComputation Centre at the UBC Djavad Mowafaghian Centre for Brain Health (RRID SCR_019086) and made use of the DataBinge forum, and computational resources and services provided by Advanced Research Computing (ARC) at the University of British Columbia. We thank Pumin Wang and Cindy Jiang for surgical assistance, Jamie Boyd and Jeffrey M LeDue for technical assistance.

Additional files

Animations

Supplementary Figures

Additional information

Funding

Canadian Institutes of Health Research (CIHR) (FDN-143209)

Canadian Institutes of Health Research (CIHR) (PJT-180631)