Real-time closed-loop feedback system for mouse mesoscale cortical signal and movement control
- Department of Psychiatry, University of British Columbia, Detwiller Pavilion, Canada
- Djavad Mowafaghian Centre for Brain Health (DMCBH), University of British Columbia, Canada
Figures
Figure 1 with 3 supplements
Setup of real-time feedback for GCaMP6 cortical activity and movements.
(A) GCaMP-based closed-loop feedback and reward system: Mice expressing GCaMP6, with surgically implanted transcranial windows and a head-bar, were positioned beneath the imaging camera, with GCaMP6 excitation light at 470 nm. A secondary wavelength of 440 nm was used for continuous reflectance signals to measure hemodynamic changes, which were then applied to correct the fluorescence signal. The RGB camera was equipped with bandpass filters that allowed only 520 nm epifluorescence and 440 nm reflectance to be simultaneously collected. (i) Mice with transcranial windows were head-fixed beneath an imaging camera, with the cortical window illuminated using 440 nm (for reflectance signal used for hemodynamic correction) and 470 nm (for GCaMP excitation) light. Epifluorescence at 520 nm and reflectance at 440 nm were captured at 15 fps using a bandpass filter, integrated within the cortical imaging system. (ii) The captured images were simultaneously saved and processed to compute ΔF/F0 in real time using a Raspberry Pi 4B model. Pre-selected regions of interest (ROIs) were continuously monitored, and rule-specific activation was calculated based on the ΔF/F0 signal. The left panel displayed widefield cortical GCaMP6 fluorescence (green) and reflectance (blue), while the right panel showed the real-time calculated and corrected ΔF/F0 map, generated using a moving average of the captured images. The target ROIs were marked as green (R1) and red (R2), although a single ROI could also be selected for monitoring. (iii) For example, as shown on the ΔF/F0 map, ROIs R1 and R2 were continuously monitored, and the average activity across these ROIs was calculated. (iv) When the task rule was defined as ‘R1-R2 >threshold’, the difference between R1 and R2 activities was mapped to a nonlinear function that generated graded audio tone frequencies (ranging from 1 kHz to 22 kHz), as illustrated in the figure (also refer to Figure 6—figure supplement 1). Task rules could be modified within the setup on any given day, and the corresponding activation levels were automatically mapped to the audio frequencies. The ‘threshold’, expressed in ΔF/F0 units as a percent change, was adjustable based on the experimental design. (v) Upon reaching the rule-specific threshold for activity, in addition to the increase in audio tone frequency, a water reward was delivered to the head-fixed mouse. (B) Closed-loop behavior feedback and reward setup: A specialized transparent head-fixation chamber was custom-designed using 3-mm-thick plexiglass material (3D model available on GitHub link, https://github.com/pankajkgupta/clopy; copy archived at Gupta, 2026) to enable multi-view behavioral imaging and real-time tracking of body parts. The rectangular chamber was equipped with two strategically positioned mirrors – one at the bottom, angled at 45 degrees, and one at the front, angled at 30 degrees – facilitating multi-view imaging of the head-fixed mouse with a single camera. (i) A Dalsa CCD camera was connected to a PC for widefield cortical imaging during the session. An additional camera connected to a PC with a GPU was used to capture the behavior video stream of the head-fixed mouse. (ii) Video frames of the behavior were processed in real time on a GPU (Nvidia Jetson Orin), which tracked the body parts using a custom trained DeepLabCut-Live (Kane et al., 2020) model. (iii) Auditory feedback was provided using a nonlinear function mapping paw speeds to corresponding audio tone frequencies. (iv) The head-fixed mouse, positioned in the transparent chamber, was able to freely move its body parts, while its behavior was continuously recorded. This setup allowed for the capture of three distinct views of the mouse – side, front, and bottom profiles – and enabled the real-time tracking of multiple body parts, including the snout, left and right forelimbs, left and right hindlimbs, and the base of the tail.
Figure 1—figure supplement 1
Method for calculating feedback and reward delay in closed-loop neurofeedback (CLNF) and closed-loop movement feedback (CLMF) setups.
Two devices (device running closed-loop system, an observer device that was just recording the scene), each connected to a camera with overlapping field of view, were employed. (A) Setup for calculating feedback delay in CLNF experiments deployed on Raspberry Pi. The difference between the onset of the green LED and red LED ON events is the feedback delay. (1) The Raspberry Pi generates a HIGH signal on the general-purpose input/output (GPIO) pin connected to the green LED. (2) The HIGH signal is received at the anode of the green LED, causing it to glow. (3) Light emitted by the green LED is received by the brain camera sensor. (4) An image capturing the green emitted light is captured and sent to the Raspberry Pi. (5) The CLNF program processes the image and detects changes in brightness. (6) The CLNF program maps the brightness change to an audio frequency and generates an audio tone output at that frequency. (7) The CLNF program sends a HIGH signal to the GPIO pin connected to the red LED. (8) The HIGH signal is received at the anode of the red LED, causing it to glow. (9) Light emitted by the red LED is received by the sensor of the brain camera. (10) An image frame capturing the red LED light is captured and sent to the Raspberry Pi. (11) During the previous steps, an observer camera records the green and red LEDs. In offline analysis, detect the brightness change in the green LED. (12) Detect the brightness change in the red LED. (13) Calculate the response delay as the time difference between the green and red LED brightness change events. (B) Setup for calculating feedback delay in CLMF experiments deployed on Nvidia Jetson Orin. The difference between the onset of movement and the red LED ON events is the feedback delay. (1) The head-fixed mouse in the CLMF apparatus moved the control point crossing the speed threshold. (2) Behavior is captured by the behavior camera and observer camera sensors. (3) The captured image is sent to Nvidia Jetson running the CLMF program. (4) The image frame is analyzed, and tracked points are detected. Calculate the speed of the control point. (5) Map the control point speed to an audio frequency and generate audio tone output at the frequency. (6) Send a HIGH signal to the GPIO pin connected to the red LED. (7) The HIGH signal is received at the anode of the red LED. (8) Bright light from the red LED reaches the behavior camera and observer camera sensors. (9) The captured image is sent to Nvidia Jetson and the observer device. (10) In offline analysis, detect the control point movement event. (11) Detect the brightness change event in the red LED. (12) Calculate the response delay as the time difference between the control point movement event and the red LED brightness change event. (C) Offline analysis to calculate response time using data collected in an experiment. Left: Brightness change events of green LED and red LED in CLNF setup, data represents 100 trials. Right: Control point movement event and brightness change event of red LED in CLMF setup, data represents 30 trials.
Figure 1—figure supplement 2
Verification of generated audio tone frequencies.
(A) Left: Spectrogram of an audio recorded in a session where graded audio tone frequencies were generated sequentially from 1 kHz to 22.627 kHz using the closed-loop neurofeedback (CLNF) program. Desired frequency values are shown on the x-axis and expected frequencies are marked in red on the y-axis. The frequency values increase exponentially, and therefore the power bands shrink as frequency increases. Right: Peak powers detected compared to the intended frequency, using the spectrogram on the left. (B) The power spectral density of the audio recording used is A. The width of the peaks becomes narrower as the frequency increases since the x-axis is on a log scale.
Figure 1—figure supplement 3
Closed-loop neurofeedback (CLNF), closed-loop movement feedback (CLMF) command line operation and preview steps.
(A) Top: Terminal commands to launch a CLNF experiment. Bottom left: Showing widefield calcium activity (green) and reflectance (blue) signals as a composite RGB image. Overlaid text indicates maximum green and blue intensities in the middle row of the image; bottom curves show intensity in the green channel (faded green) and blue channel (bright green) along the middle row of pixels. Two red squares indicate the regions of interest (ROIs) in the 2ROI experiment. White dots are the registered centers of the cortical locations specified in the config.ini file. Bottom right: Showing the corrected ΔF/F0 map of the image on its left in real time. The overlaid artifacts in the form of text and lines are there since we draw those text and lines on the captured image for preview. These artifacts are not present when a session starts. (B) Top: Terminal commands to launch a CLMF experiment. Bottom: The preview window shows a live view with tracked points overlaid on the live view, indicated in red. To start the experiment, press ‘Esc’ on the keyboard, type the mouse ID in the prompt, and then press the Enter button to start the experiment.
Figure 2
Schematic of the closed-loop feedback training system (CLoPy) for neurofeedback and specified movement feedback.
(A) The components of CLoPy closed-loop neurofeedback (CLNF) system are presented in a block diagram. Modular components such as the configuration file, camera factory, audio tone generator, and reward delivery system are displayed and are utilized by both the CLNF and closed-loop movement feedback (CLMF, in B) systems. The configuration file (config.ini) stored all configurable parameters of the system, including camera settings, feedback parameters, reward thresholds, number of trials, and the duration of trial and rest periods, under an experiment-specific section. The camera factory was an abstract class that provided a unified interface for a programmable camera to the CLNF and CLMF (in B) systems. This abstraction allowed the core system to remain independent of the specific camera libraries required for image streaming. Camera-specific routines were implemented in separate ‘brain_camera_stream’ and ‘behavior_camera_stream’ classes, which inherited functions from the ‘camera_factory’ superclass and ran in independent thread processes. The region of interest (ROI) manager was used by the CLNF core to maintain a list of ROIs, as well as routines to perform rule-specific operations on them, as specified in config.ini. An ROI could be defined as a rectangle (with the upper-left corner coordinates, height, and width) or as a circle (with center coordinates and radius). The audio tone generator mapped the target activity (fluorescence signal in CLNF) to graded audio tone frequencies. It generated audio signals at 44.1 kHz sampling based on the specified frequency and sent the signal to the audio output. Reward delivery was controlled by opening a solenoid valve for a specified duration, which was managed in a separate process thread. The CLNF core was the main program responsible for running the CLNF system. It utilized config.ini, the camera factory, the ROI manager, and integrated the audio tone generator and reward delivery functions. The system also saved the recorded data and configuration parameters with unique identifiers. (B) The components of CLoPy CLMF system are presented in a block diagram. The common components of the setup are described in A. The audio tone generator mapped the target activity (control point speed in CLMF) to graded audio tone frequencies. It generated audio signals at 44.1 kHz sampling based on the specified frequency and sent the signal to the audio output. The CLMF core was the primary program responsible for operating the CLMF system. It utilized config.ini, the camera factory, and DeepLabCut-Live (Kane et al., 2020), integrating the audio tone generator and reward delivery functions. This module also saved the data and configuration parameters with unique identifiers.
Figure 3 with 1 supplement
Experimental protocol and trial structure.
(A) Experimental protocol (detailed in Materials and methods): In brief, 90-day-old transgenic male and female mice were implanted with a transcranial window and allowed to recover for a minimum of 7 days. One day before the start of the experiment, the mice were placed on a water-restriction protocol (as detailed in Materials and methods). Closed-loop experiment training commenced on day 1, during which mice were required to modulate either their target brain activity (GCaMP signals in regions of interest) or target behavior (tracked paw-speed) during daily sessions of approximately 45 min over the course of 10 days. Throughout this period, both cortical and behavioral activities were recorded. After the final experimental session, the mice were removed from the water-restriction protocol. (B) Trial structure of cortical GCaMP-based feedback sessions: Each trial was preceded by a minimum of 10 s of rest, which was extended if the tracked body parts of the mouse were not stable (sum of changes across body parts >1.5 mm, i.e. 5 pixel values). Once the mouse remained stable and refrained from moving its limbs, the trial began with a basal audio tone of 1 kHz. The mice then had 30 s to increase rule-based activations (in the selected region of interest [ROI]) up to a threshold value to receive a water reward. A trial ended as soon as the activation reached the threshold, triggering a reward delivery for success, or timed out after 30 s with an overhead buzzer serving as a negative signal of failure. Both the reward and the negative signal were delivered within 1 s after the audio ceased at the end of each trial. (C) Example dorsal cortical ΔF/F0 activity was recorded and overlaid with a subset of Allen CCF coordinates, which could be selected as the center of candidate ROIs shown as green and pink color squares. (D) Trial structure of behavior feedback sessions: The behavioral feedback trials followed a similar structure to the cortical feedback trials, with each trial preceded by at least 10 s of rest. The trial began with a basal tone of 1 kHz, which increased in frequency as the mouse’s paw speed increased. (E) Example forelimb tracking during feedback sessions: Forelimb tracking was performed in both the left (blue) and right (green) forelimbs using a camera coordinate system on day 4 of training with mouse FM2, which received feedback based on left forelimb speed. The forelimbs were tracked in 3D, leveraging multiple camera views captured using mirrors positioned at the bottom and front of the setup (Figure 1B).
Figure 3—figure supplement 1
Visual representation of 1 region of interest (ROI) and 2ROI rules in closed-loop neurofeedback (CLNF) experiment.
(A) ROIs overlaid on mouse dorsal cortical map during 1ROI experiments. (B) ROIs overlaid on mouse dorsal cortical map during 2ROI experiments.
Figure 4 with 2 supplements
Closed-loop feedback helped mice learn the task and achieve superior performance in closed-loop neurofeedback (CLNF) and closed-loop movement feedback (CLMF) in both experiments.
(A) No-rule-change (N=23, n=60, in Table 5) and Rule-change (N=17, n=60) mice were able to learn the CLNF task over several sessions, with performance above 70% by the 10th session (repeated-measures ANOVA [RM-ANOVA], p=2.83e-5). The rule change (in pink, day 11) led to a sharp decline in performance (ANOVA, p=8.7e-9), but the mice were able to adapt and learn the task rule change (RM-ANOVA, p=8.3e-10; see Table 5 for different rule changes). The method to determine the regions of interest (ROIs) used in the changed task rule is described in the Materials and methods section. (B) Three groups were employed for CLMF experiments. The ‘Rule-change’ group (N=8, n=60 received feedback, in pink) was trained with task rule mapping auditory feedback to the speed of the left forelimb and was able to perform above a 70% success rate in 4 days. The task rule mapping was changed from the left to the right forelimb on day 5, so the rewards as well as audio frequencies would now be controlled by the right forelimb. The ‘No-rule-change’ group (N=4, n=60 received audio feedback, no rule change, in green) and the ‘No-feedback’ group (N=4, n=60 no graded audio feedback, no rule change, in blue) were control groups to investigate the role of audio feedback. The performance of the ‘No-feedback’ mice, who did not receive the graded feedback, was never on par (RM-ANOVA, p=0.49) with the ‘No-rule-change’ group that received the feedback (RM-ANOVA, p=9.6e-7). Additionally, to test the effect of auditory feedback on mice that already had learned the task, we turned off the auditory feedback on day 10 for all mice (indicated by the speaker with a cross). There was no significant change in performance due to feedback removal, indicating that feedback was not necessary once the task was learned. (C) Task latencies in each group follow the trend of their performance. Rule change (N=8) and no rule change (N=4) task latencies gradually came down, with an exception on day 5 for rule change when the task rule was changed. No feedback (N=4) task latencies are never on par with the groups that received feedback. (D) CLMF Rule-change (N=8) behavior, we looked at the maximum speeds of the left and right forelimbs. The paw with the maximum speed follows the task rule and switches with the change in the task rule. It is worth noting that the task was not restricted to other body parts, i.e., the mice were free to move other body parts along with the control point.
Figure 4—figure supplement 1
Learning progression was similar in male and female mice, and even similar between 1 region of interest (ROI) and 2ROI experiments.
There were some ROI rules that were faster to learn than others. (A) The success rates over days in male and female mice are not significantly different. Data from all mice in the No-rule-change and Rule-change groups were separated based on their sex (male, female). For days 1–10, the total number of males was 14 and the total number of females was 9; for days 11–19, the total number of males was 9 and the total number of females was 9. (B) Success rate over days based on 1ROI and 2ROI experiments. Data from all mice in the No-rule-change and Rule-change groups were segregated based on the ROI type (1ROI or 2ROI). From day 1 to day 10, the total number of mice with 1ROI experiments was 5, and with 2ROI it was 18. From day 11 to day 19, the total number of mice with 1ROI was 10, and with 2 ROI it was 7. (C) Slope values of linear regression line on the success rate over days for different ROI rules. (D) ROI rules with slope values greater than or equal to the mean (0.095) were combined in the red group, and those with slope values lower than the mean were combined in the blue group. At a broad level, we can observe that ROI rules in the red group were faster than those in the blue group.
Figure 4—figure supplement 2
Closed-loop movement feedback (CLMF) behavior traces.
(A) Cumulative rewards show performance during the whole session from day 1 to day 10, including a perturbation on day 5 where cumulative rewards drop, in a CLMF rule-change mouse. (B) Kullback-Leibler divergence scores of speed distributions of different tracked points (snout, left forelimb, right forelimb, left hind limb, right hind limb, and tail base) on the mouse body. Kernel density estimates (KDEs) were derived for these tracked points on each day, and divergence scores compared to day 1 were calculated. (C) Reward centered left (in blue) and right (in green) forelimb speeds, top row (day 1) and third row (day 4). Trial end centered left (in blue) and right (in green) forelimb speeds, second (day 1) and fourth row (day 4). The target control point during training was left forelimb. (D) Left (in blue) and right (in green) forelimb tracks on day 1 (top) and day 4 (bottom) with the control point being the left forelimb.
Figure 5
Closed-loop feedback helped mice learn the task and achieve superior performance in closed-loop neurofeedback (CLNF) and closed-loop movement feedback (CLMF) in both experiments.
(A) Reward-aligned average (N=4) ΔF/F0 signals associated with the target rule on day 1 and day 9 (top plot). Kernel density estimate (KDE) of target ΔF/F0 values during the whole session on day 1 and day 9 of 1ROI experiments (bottom plot). (B) Reward-aligned average (N=4) target paw speed on day 1 and day 10 (top plot). KDE of target paw speeds on day 1 and day 10 (bottom plot). (C) In the context of CLNF 2ROI experiments, bivariate distribution of ROI1 and ROI2 ΔF/F0 values during whole sessions on day 9 and day 19, with densities projected on the marginal axes. The task rule on day 9 was ‘ROI1-ROI2>thresh’ as opposed to ‘ROI2-ROI1>thresh’ on day 19. The bivariate distribution is significantly different (multivariate two-sample permutation test, p=2.3e-12) on these days, indicating a robust change in activity within these brain regions. (D) Joint (bivariate) distribution of left and right paw speeds during the whole session on day 4 and day 10 of CLMF. Left and right forelimbs were control point (CP) on day 4 and day 10, respectively. There is a visible bias in the bivariate distribution toward the CP on respective days.
Figure 6 with 1 supplement
Closed-loop neurofeedback (CLNF): cortical activity during the closed-loop neurofeedback training.
(A) Target-rule-based ΔF/F0 traces in green on day 1 with rule-1 (top row), on day 10 with rule-1 (second row), on day 11 with new rule-2 (third row), and on day 19 with rule-2 (fourth row). Shared regions are trial periods and regions between gray areas are rest periods. The gray horizontal line depicts the threshold above which mice would receive a reward. Golden stars show the rewards received, and short vertical lines in black show the spout licks. (B) Representative reward-centered average cortical responses of the two regions of interest (ROIs) experiment on labeled days. ROI1 (green) and ROI2 (pink) are overlaid on the brain maps. The task rule on day 1 and day 4 was ‘ROI1-ROI2>thresh’, as opposed to ‘ROI2-ROI1>thresh’ on day 11 and day 19. (C) Linear regression on ROI1 and ROI2 ΔF/F0 during whole sessions for an example mouse. The regression fit inclines toward ROI1 in sessions where the rule was ‘ROI1-ROI2>thresh’ (day 1 slope = 0.44, day 6 slope = 0.32) while it leans toward ROI2 after the task rule switches to ‘ROI2-ROI1>thresh’ (day 11 slope = 0.76, day 17 slope = 0.80).
Figure 6—figure supplement 1
Example trials along with audio feedback during closed-loop neurofeedback (CLNF) and closed-loop movement feedback (CLMF) training.
(A) Target cortical activity in green, along with horizontal threshold line, yellow stars as rewards, black ticks showing licks, and orange trace showing the audio frequency. (B) Target paw speed in red, along with horizontal threshold line, yellow stars as rewards, and orange trace showing the audio frequency.
Figure 7
Closed-loop movement feedback (CLMF): speed of the tracked target body part and cortical activity.
(A) Left forelimb speed (black), target threshold (gray line), and rewards (golden stars) during a sample period in a session on day 1 of the closed-loop training (top row). Shaded regions are trial periods with interspersed rest periods in white. Left forelimb speed and rewards on day 4 (second row). The target body part was changed from the left forelimb to the right forelimb on day 5 (third row). Thus, day 5 is the first training day with the new rule. Right forelimb speed, and rewards on day 10 of the training (fourth row). (B) Reward-centered average cortical responses on days corresponding to rows in A. The target threshold was crossed at -1 s, and the reward was delivered at 0 s. Notice the task rule change on day 5.
Figure 8
Closed-loop movement feedback (CLMF): correlation matrix showing pairwise correlations of left and right forelimb speed profiles.
(A) Top row: During rewarded trials, over the training sessions of CLMF Rule-change (left), No-rule-change (middle), and No-feedback (right). High correlations (dark cells) between speed profiles of control point (CP) indicate a unilateral bias in the movement. It is worth noting the drastic changes in correlations as the CP was changed from left forelimb to right forelimb in Rule-change mice on day 4. Bottom row: During rest periods, over the training sessions of CLMF Rule-change (left), No-rule-change (middle), and No-feedback (right).
Figure 9
Cortical dynamics and network changes during longitudinal closed-loop movement feedback (CLMF) training.
(A) Reward-centered average responses in the olfactory bulb (OB) decrease over the days as performance increases. Data shown is a representative example from sessions of mouse. (B) Cortical responses become focal and closely aligned to the paw movement (green line) and reward (cyan line) events on day 10 for group-1 (received feedback) as compared to group-3 (no-feedback). (C) ΔF/F0 peak values during successful trials (pink) and during rest (cyan) over the 10-day training period in the OB (left, day 1-day 4 p-value = 0.025, day 4-day 5 p-value = 0.008), forelimb area (FL) (center, day 1-day 4 p-value = 0.008, day 4-day 5 p-value = 0.04), and primary visual cortex (right, day 1-day 4 p-value = 0.04, day 4-day 5 p-value = 0.002). Statistical significance was assessed using the Mann-Whitney test and corrected for multiple comparisons with the Benjamini-Hochberg procedure. (D) Average movement (mm) of different tracked body parts during trials in CLMF Rule-change (left), No-rule-change (center), No-feedback (right). (E) Correlations between cortical activation on each training session in barrel cortex (BC, top left), anterolateral motor cortex (ALM, top right), secondary motor cortex (M2, bottom left), and retrosplenial cortex (RS, bottom right).
Figure 10 with 1 supplement
Closed-loop movement feedback (CLMF) cortex-wide seed pixel correlation matrices.
Pairwise correlations between activity at cortical locations (also referred to as seed pixel locations). (A) Top row: No-rule-change average seed pixel correlation matrix during trial periods (left), during rest periods (middle), and difference of average correlation matrix during trial and rest (right). Bottom row: No-feedback average seed pixel correlation matrix during trial periods (left), during rest periods (middle), and difference of average correlation matrix during trial and rest (right). (B) Significant increase in pairwise seed pixel correlations as repeated-measures ANOVA (RM-ANOVA) p-value (Bonferroni corrected) matrix between training sessions over the days (left) and between trial vs rest periods (right) for CLMF No-rule-change mice. (C) Significant increase in pairwise seed pixel correlations as RM-ANOVA p-value (Bonferroni corrected) matrix between training sessions over the days (left) and between trial vs rest periods (right) for CLMF No-feedback mice.
Figure 10—figure supplement 1
Closed-loop movement feedback (CLMF) seed pixel correlation maps during rewarded trial, rest periods, and their difference.
(A) Seed pixel correlation maps of labeled seed pixels on day 1, day 4, day 5 (rule change), and day 10 during rewarded trials (excluding the reward consumption period). Seed pixel locations are indicated by black points on the brain map. (B) Seed pixel correlation maps of labeled seed pixels on day 1, day 4, day 5 (rule change), and day 10 during rest periods (excluding the reward consumption period). Seed pixel locations are indicated by black points on the brain map. (C) Difference of seed pixel correlation maps of labeled seed pixels during rewarded trials and rest periods (A–B) on day 1, day 4, day 5 (rule change), and day 10. Seed pixel locations are indicated by black points on the brain map. There is a distinct increase in correlation during rewarded trials in cortical regions such as primary visual cortex (V1) and retrosplenial cortex (RS) across all seed pixel maps.
Figure 11
Average reward-centered activity in regions of interest (ROIs) for 2ROI experiment.
Examples of reward-centered activity in both the ROIs in a 2ROI experiment reveal different strategies adopted by mice. For a task rule specifying ‘ROI1 - ROI2>threshold’, mouse BZ1 and BZ2 decreased ROI2 activity to meet the reward threshold while mouse BU2 and BU3 increased ROI1 activity.
Videos
Animation 1
Closed-loop neurofeedback (CLNF) success centered trial averages under various task rules.
(A) Success centered ΔF/F0 cortical dynamics in 1 region of interest (ROI) experiment with target task rule as M1_L. (B) Success centered ΔF/F0 cortical dynamics in 1ROI experiment with target task rule as V1_L. (C) Success centered ΔF/F0 cortical dynamics in 1ROI experiment with target task rule as M1_L. (D) Success centered ΔF/F0 cortical dynamics in 2ROI experiment with target task rule as HL_L – HL_R. (E) Success centered ΔF/F0 cortical dynamics in 2ROI experiment with target task rule as BC_L – HL_L. (F) Success centered ΔF/F0 cortical dynamics in 2ROI experiment with target task rule as BC_L – BC_R.
Animation 2
Closed-loop movement feedback (CLMF) single trial with FLL_bottom as control point.
Behavior video on the left panel, two of the tracked body parts (FLL_bottom, FLR_bottom) at the center, and corresponding cortical widefield ΔF/F0 activity of a trial with target control point as FLL_bottom (left forelimb in camera bottom view) on the right panel.
Animation 3
Closed-loop movement feedback (CLMF) single trial with FLR_bottom as control point.
Behavior video on the left panel, two of the tracked body parts (FLL_bottom, FLR_bottom) at the center, and corresponding cortical widefield ΔF/F0 activity of a trial with target control point as FLR_bottom (right forelimb in camera bottom view) in the same mouse shown in Animation 2 on the right panel.
Tables
Table 1
Key configuration parameters in CLoPy.
| Parameter | Description |
|---|---|
| vid_source | Specifies the class responsible for the image stream, which may come from a programmable camera or another video source. |
| data_root | Directory path for saving the recorded sessions and current configuration. |
| raw_image_file | File name for saving the image stream. |
| resolution | Image stream resolution in (x, y) pixels. |
| frame rate | Number of frames per second from the image stream. |
| awb_mode | Auto-white-balance mode (only used in CLNF, True for Behavior-Pi, False for Brain-Pi). |
| shutter_speed | Sets the camera sensor exposure. |
| dff_history | The duration (in s) used to calculate ΔF/F0, only used in CLNF. |
| ppmm | Pixels-per-mm value at the focal plane of the camera. |
| bregma | Y, X pixel coordinates of the bregma on the dorsal cortex in the image frame, only used in CLNF. |
| seeds_mm | List of cortical locations for inclusion in the closed-loop training rule, each defined by a name and coordinates relative to bregma (in mm), only used in CLNF. |
| roi_operation | Specifies the ROI(s) and operation (+ or -) for closed-loop training. |
| roi_size | Size of the ROI(s) in mm. |
| n_tones | Number of distinct audio frequencies for graded auditory feedback. |
| reward_delay | Delay (in s) after crossing the threshold. |
| reward_threshold | Threshold value in terms of ΔF/F0, determined from baseline sessions. |
| adaptive_threshold | A setting for adjusting the reward threshold. If set to 0, the threshold remains constant throughout the session; if set to 1, the threshold increases or decreases by 0.02 steps based on the reward rate. |
| total_trials | Total number of trials in the session. |
| max_trial_dur | Maximum trial duration (in s). |
| success_rest_dur | Rest duration after a successful trial (in s). |
| fail_rest_dur | Rest duration after a failed trial (in s). |
| initial_rest_dur | Rest period at the beginning of the session before the first trial (in s). |
| summary_file | File name to save the experiment summary as comma-separated values (CSV). |
| summary_header | List of variable names to be saved in the summary file. |
| dlc_model_path | Path of the DeepLabCut-Live model for real-time pose tracking (used only in CLMF). |
| control_point | Name of the tracked point for closed-loop feedback (used only in CLMF). |
| speed_threshold | Speed threshold for a success in a trial and receive reward (used only in CLMF). |
Table 2
List of statistical tests performed.
| Figure | Test name | Purpose | Arguments |
|---|---|---|---|
| Figure 4A | Mann-Kendall test (pymannkendall package in Python) | To determine a monotonic upward trend in the success rate of all CLNF mice before the rule change (from day 1 to day 10). | Normalized success rate per mouse before the rule change (N=40). |
| To determine a monotonic upward trend in the success rate of CLNF rule-change mice after the rule change (day 11 to day 19). | Normalized success rate per mouse after the rule change (N=17). | ||
| Figure 4A | RM-ANOVA (Pingouin package in Python) | Test the significance of the success rate increase before the rule change in the CLNF No-rule-change group (day 1 to day 10). | Normalized success rate per mouse in No-rule-change mice before the rule change (N=23). |
| Test the significance of the success rate increase after the rule change in the CLNF Rule-change group (day 11 to day 19). | Normalized success rate per mouse in Rule-change mice after the rule change (N=17). | ||
| Figure 4A | Two-way RM-ANOVA (Pingouin package in Python) | Test significance of day and group on the CLNF success rate before rule change (day 1 to day 10). | Normalized success rate per mouse in Rule-change and No-rule-change mice before rule change, along with group labels. |
| Figure 4A | Mann-Whitney U-test with Benjamini-Hochberg correction (pymannkendall package in Python) | Test the difference in the distribution of CLNF success rates between: day 1 no-rule-change and day 3 no-rule-change day 10 No-rule change and day 10 Rule change day 10 rule change and day 11 rule change day 10 rule change and day 19 rule change. | Normalized success rate per mouse in Rule-change and No-rule-change mice before rule change, along with group labels. |
| Figure 4B | RM-ANOVA (Pingouin package in Python) | Test the significance of the success rate increase before the rule change in the CLMF Rule-change group (day 1 to day 4). Test the significance of the success rate increase after the rule change in the CLMF Rule-change group (from day 5 to day 10). Test the significance of the success rate increase in the CLMF no-rule-change group (day 1 to day 10). Test significance of success rate increase in CLMF No-feedback group (day 1 to day 10). | Normalized success rate per mouse in Rule-change and No-rule-change mice before rule change, along with group labels. |
| Figure 5C | Permutation test (MMD function in hyppo package) | Multivariate two-sample test to compare ROI1 vs ROI2 bivariate distribution on day 9 and day 19. | Bivariate distribution of ROI1 and ROI2 on day 9 and day 19. |
| Figure 9C | Mann-Whitney, corrected for multiple comparisons with Benjamini-Hochberg (pymannkendall package in Python) | Test the significance of the difference in ΔF/F0 peak distribution between. Day 1 and day 4 in cortical OBL of CLMF rule-change mice. Day 4 and day 5 in cortical OBL of CLMF rule-change mice. Day 1 and day 4 in cortical FLL of CLMF rule-change mice. Day 4 and day 5 in cortical FLL of CLMF rule-change mice Day 1 and day 4 in cortical V1L of CLMF rule-change mice. Day 4 and day 5 in cortical V1L of CLMF rule-change mice. | ΔF/F0 peak values in named cortical regions on each day for all mice along with their group labels. |
| Figure 10B, Figure 10C | Two-way RM-ANOVA, Bonferroni corrected for multiple comparisons (Pingouin package in Python) | Test significant changes in seed pixel correlations over days and between trial and rest conditions for CLMF No-rule-change mice (Figure 10B) and for CLMF No-feedback mice (Figure 10C). | Pairwise seed pixel correlation values for all combinations of cortical seed locations over days for each mouse, along with group labels. |
| Figure 4—figure supplement 1A | Mann-Whitney U-test with Benjamini-Hochberg correction (pymannkendall package in Python) | Test the difference in the distribution of CLNF success rates between. Day 1 male and day 3 male. Day 4 male and day 4 female. Day 10 female and day 11 female. | Normalized success rate per mouse along with their sex labels. |
| Figure 4—figure supplement 1B | Mann-Whitney U-test with Benjamini-Hochberg correction (pymannkendall package in Python) | Test the difference in the distribution of CLNF success rates between. Day 1 single ROI and day 3 single ROI Day 10 single ROI and day 10 dual ROI. | Normalized success rate per mouse along with their ROI-type of the experiment. |
Table 3
List of closed-loop neurofeedback (CLNF) task rules with their description.
| CLNF rule name | Description |
|---|---|
| M1_L | ΔF/F0 activity in left hemisphere primary motor area. |
| M1_R | ΔF/F0 activity in right hemisphere primary motor area. |
| M2_L | ΔF/F0 activity in left hemisphere secondary motor area. |
| HL_L | ΔF/F0 activity in left hemisphere sensory hindlimb area. |
| RL_L | ΔF/F0 activity in left hemisphere rostrolateral area of visual cortex. |
| RL_R | ΔF/F0 activity in right hemisphere rostrolateral area of visual cortex. |
| V1_L | ΔF/F0 activity in left hemisphere primary visual area. |
| BC_L – BC_R | ΔF/F0 activity in left hemisphere barrel cortex subtracted by ΔF/F0 activity in right hemisphere barrel cortex. |
| HL_L – HL_R | ΔF/F0 activity in left sensory hindlimb area subtracted by ΔF/F0 activity in right hemisphere sensory hindlimb area. |
| BC_L – HL_L | ΔF/F0 activity in left hemisphere barrel cortex subtracted by ΔF/F0 activity in left hemisphere sensory hindlimb area. |
| M1_L – HL_L | ΔF/F0 activity in left hemisphere primary motor area subtracted by ΔF/F0 activity in left hemisphere sensory hindlimb area. |
| TR_L – M2_L | ΔF/F0 activity in the left hemisphere rostral part of the temporal association area is subtracted by ΔF/F0 activity in the left hemisphere secondary motor area. |
| V1_R – V1_L | ΔF/F0 activity in the right hemisphere primary visual area subtracted by ΔF/F0 activity in the left hemisphere primary visual area. |
| M2_L – TR_L | ΔF/F0 activity in left hemisphere secondary motor area subtracted by ΔF/F0 activity in left hemisphere rostral part of temporal association area. |
| HL_L – M2_L | ΔF/F0 activity in left hemisphere sensory hindlimb area subtracted by ΔF/F0 activity in left hemisphere secondary motor area. |
Table 4
List of closed-loop movement feedback (CLMF) task rules with their description.
| CLMF rule name | Description |
|---|---|
| FLL | Speed of left forelimb in the bottom view of the camera |
| FLR | Speed of right forelimb in the bottom view of the camera |
Table 5
List of mice – closed-loop neurofeedback (CLNF).
| MouseID | Group | ROIs | Initial rule | change_rule | DoB | Expt. start date | Age (days) | Sex | Reward rate | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Day 1 | Day 5 | Day 11 | Day 16 | Genotype | ||||||||||
| 1 | BM2 | G1 | 2ROI | BC_L – BC_R | – | 20220122 | 20220713 | 172 | m | 0.01 | 0.8 | – | – | Па-gcamp68 |
| 2 | BM3 | G1 | 2ROI | BC_L – BC_R | – | 20220122 | 20220712 | 171 | m | 0.28 | 1 | – | – | tta-gcamp6s |
| 3 | BM4 | G1 | 2ROI | BC_L – BC_R | – | 20220122 | 20220712 | 171 | m | 0.01 | 0.83 | * | – | tta-gcampós |
| 4 | BG2 | G1 | 2ROI | HL_L – HL_R | – | 20210601 | 20211023 | 144 | m | 0.00 | 0.63 | – | – | #3-gcamp68 |
| 5 | BG3 | G1 | 2ROI | HL_L – HL_R | – | 20210601 | 20211023 | 144 | m | 0.00 | 0.64 | – | – | tta-gcamp6s |
| 6 | AC4 | G1 | 2ROI | HIL_L – HL_R | – | 20210610 | 20211009 | 121 | f | 0.00 | 0.71 | – | – | tta-gcamp6s |
| 7 | TW1 | G1 | IROI | RL_L | – | 20200729 | 20210113 | 168 | m | 0.00 | 0.35 | – | – | tta-gcamp68 |
| 8 | FM1 | G1 | IROI | MI_L | – | 20190915 | 20200129 | 136 | f | 0.13 | 0.66 | – | – | tta-gcamp6s |
| 9 | FM2 | G1 | IROI | MI_L | – | 20190915 | 20200129 | 136 | f | 0.03 | 1 | * | – | tta-gcampós |
| 10 | FM3 | G1 | IROI | MI_L | – | 20190915 | 20200129 | 136 | f | 0.02 | 1 | – | – | Па-gcamp68 |
| 11 | FM4 | G1 | IROI | MI_L | – | 20190915 | 20200129 | 136 | f | 0.25 | 1 | – | – | tta-gcamp6s |
| 12 | CV2 | G1 | 2ROI | BC_L – H_ L | – | 20190815 | 20191205 | 112 | m | 0.01 | 0.72 | * | * | tta-gcamp6s |
| 13 | CV3 | G1 | 2ROI | BC_L – HL_L | – | 20190815 | 20191205 | 112 | m | 0.01 | 1 | – | – | tta-gcamp68 |
| 14 | DU1 | G1 | 2ROI | MI_L – HL_L | – | 20190719 | 20191205 | 139 | f | 0.01 | 1 | – | – | tta-gcamp6s |
| 15 | DU3 | G1 | 2ROI | M1_L – HL_L | – | 20190719 | 20191205 | 139 | f | 0.02 | 0.39 | – | – | tta-gcampós |
| 16 | BZ1 | G1 | 2ROI | TR_L – M2_L | – | 20190406 | 20190806 | 122 | m | 0.24 | 0.29 | – | – | tta-gcamp68 |
| 17 | BZ2 | G1 | 2ROI | TR_L – M2_L | – | 20190406 | 20190806 | 122 | m | 0.02 | 0.32 | – | – | tta-gcamp68 |
| 18 | DUI | G1 | 2ROI | VI_R – VI_L | – | 20190217 | 20190807 | 171 | f | 0.00 | 1 | – | – | tta-gcamp6s |
| 19 | DU2 | G1 | 2ROI | VI_R – VI_L | – | 20190217 | 20190807 | 171 | f | 0.00 | 0.46 | – | – | tta-gcamp68 |
| 20 | BZ1 | G1 | 2ROI | M2_L – TR_L | – | 20181230 | 20190516 | 137 | m | 0.19 | 0.19 | – | – | tta-gcamp6s |
| 21 | BZ2 | G1 | 2ROI | M2_L – TR_L | – | 20181230 | 20190516 | 137 | m | 0.02 | 0.71 | * | – | tta-gcamp6s |
| 22 | BU2 | G1 | 2ROI | M2_L – TR_L | – | 20181219 | 20190517 | 149 | m | 0.20 | 1 | – | – | tta-gcamp6s |
| 23 | BU3 | G1 | 2ROI | M2_L – TR_L | – | 20181219 | 20190517 | 149 | m | 0.01 | 0.61 | – | – | tta-gcamp66 |
| 24 | BMI | G2 | 2ROI | BC_L – BC R | RFLL – CFLL (day 11) | 20220122 | 20220712 | 171 | m | 0.00 | 0.63 | 0.03 | 0.52 | |
| 25 | ACI | G2 | 2ROI | BC_L – BC_R | HL_L – HL_R (day 11) | 20210610 | 20211009 | 121 | f | 0.27 | 0.41 | 0.12 | 1 | |
| 26 | AC2 | G2 | 2ROI | BC_L – BC_R | HL_L – HL_R (day 11) | 20210610 | 20211009 | 121 | f | 0.00 | 1 | 0.29 | 0.96 | |
| 27 | TWI | G2 | IROI | RL_L | HL_L (day 11) | 20200802 | 20210113 | 129 | m | 0.01 | 0.58 | 0.00 | 0.71 | |
| 28 | TW2 | G2 | IROI | RL_L | PM_L (day 11) | 20200802 | 20210113 | 129 | m | 0.02 | 0.28 | 0.32 | 0.90 | |
| 29 | TW22 | G2 | IROI | RL_R | RL_L (day 11) | 20200730 | 20201119 | 112 | m | 0.10 | 0.48 | 0.35 | 0.60 | |
| 30 | TW11 | G2 | IROI | M2_L | RL_L (day 11) | 20200730 | 20201119 | 112 | m | 0.05 | 0.32 | 0.10 | 0.64 | |
| 31 | TW33 | G2 | IROI | MI_R | RL_R (day 11) | 20200730 | 20201119 | 112 | m | 0.02 | 0.41 | 0.40 | 0.60 | |
| 32 | FM3 | G2 | IROI | VI_L | MI_R (day 11) | 20190917 | 20200129 | 134 | f | 0.08 | 0.62 | 0.03 | 0.37 | |
| 33 | FM1 | G2 | IROI | MI_L | VI_L (day 11) | 20190917 | 20200129 | 134 | f | 0.00 | 0.65 | 0.02 | 0.61 | |
| 34 | FM2 | G2 | IROI | MI_L | MI_R (day 11) | 20190917 | 20200129 | 134 | f | 0.10 | 1 | 0.14 | 1 | |
| 35 | FM3 | G2 | IROI | MI_L | VI_L (day 11) | 20190917 | 20200129 | 134 | f | 0.63 | 0.20 | 0.46 | ||
| 36 | FM4 | G2 | IROI | MI_L | VI_L (day 11) | 20190917 | 20200129 | 134 | f | 0.01 | 0.31 | 0.00 | 0.62 | |
| 37 | DR2 | G2 | 2ROI | M2_L – TR_L | HL_L – M2_L (day 11) | 20190401 | 20190806 | 127 | m | 0.09 | 1 | 0.01 | 1 | |
| 38 | DR3 | G2 | 2ROI | M2_L – TR_L | HL_L – M2_L (day 11) | 20190401 | 20190806 | 127 | m | 0.03 | 1 | 0.06 | 0.74 | tta-pcamp6s |
| 39 | DR4 | G2 | 2ROI | M2_L – TR_L | HL_L – M2_L (day 11) | 20190401 | 20190806 | 127 | m | 0.00 | 0.64 | 0.02 | 1 | tta-gcamp6s |
| 40 | DU3 | G2 | 2ROI | M2_L – TR_L | VI_R – VI_L (day 11) | 20190408 | 20190807 | 121 | f | 0.13 | 0.36 | 0.10 | 1 | tta-gcamp6s |
Table 6
List of mice – closed-loop movement feedback (CLMF).
| MouseID | Group | initial_rule | change_rule | DoB | Exp. start | Age (days) | Sex | Reward rate | Genotype | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Day 1 | Day 4 | Day 5 | Day 10 | ||||||||||
| 1 | FW2 | Rule change | FLL | FLR (day 5) | 20230209 | 20230713 | 154 | f | 0.30 | 0.60 | 0.21 | 0.95 | Ai94 |
| 2 | FW3 | Rule change | FLL | FLR (day 5) | 20230209 | 20230718 | 159 | f | 0.09 | 0.65 | 0.00 | 0.92 | Ai94 |
| 3 | FW22 | Rule change | FLL | FLR (day 5) | 20230209 | 20230719 | 160 | f | 0.25 | 0.70 | 0.44 | 1 | Ai94 |
| 4 | GT3 | Rule change | FLL | FLR (day 5) | 20230219 | 20230715 | 146 | f | 0.11 | 0.93 | 0.63 | 0.85 | Ai94 |
| 5 | GT33 | Rule change | FLL | FLR (day 5) | 20230219 | 20230719 | 150 | f | 0.12 | 0.78 | 0.00 | 0.87 | Ai94 |
| 6 | HA2 | Rule change | FLL | FLR (day 5) | 20230225 | 20230719 | 144 | m | 0.12 | 0.72 | 0.28 | 0.9 | Ai94 |
| 7 | GER2 | Rule change | FLL | FLR (day 5) | 20230823 | 20231228 | 127 | m | 0.34 | 0.95 | 0.50 | 0.83 | Ai94 |
| 8 | HYL3 | Rule change | FLL | FLR (day 5) | 20230827 | 20240103 | 129 | m | 0.33 | 0.93 | 0.60 | 0.91 | Ai94 |
| 9 | BR1 | No rule change | FLL | – | 20230729 | 20231201 | 125 | m | 0.11 | 0.58 | 0.97 | 1 | Ai94 |
| 10 | BR2 | No rule change | FLL | – | 20230729 | 20231201 | 125 | m | 0.38 | 1 | 0.84 | 0.82 | Ai94 |
| 11 | GER1 | No rule change | FLL | – | 20230811 | 20231228 | 139 | f | 0.35 | 0.88 | 0.84 | 0.82 | Ai94 |
| 12 | HYL2 | No rule change | FLL | – | 20240103 | 20240103 | 121 | f | 0.33 | 0.93 | 0.88 | 0.93 | Ai94 |
| 13 | GIL2 | No feedback | FLL | – | 20231219 | 20231219 | 112 | m | 0.09 | 0.52 | 0.28 | 0.95 | Ai94 |
| 14 | GIL3 | No feedback | FLL | – | 20231219 | 20231219 | 135 | f | 0.23 | 0.53 | 0.57 | 0.51 | Ai94 |
| 15 | GIL4 | No feedback | FLL | – | 20231219 | 20231219 | 135 | f | 0.19 | 0.63 | 0.60 | 0.69 | Ai94 |
| 16 | GER3 | No feedback | FLL | – | 20230811 | 20231219 | 130 | m | 0.08 | 0.48 | 0.43 | 0.75 | Ai94 |
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Real-time closed-loop feedback system for mouse mesoscale cortical signal and movement control
eLife 14:RP105070.
https://doi.org/10.7554/eLife.105070.4
