Back to Projects

Action Segmentation in Mice

An evaluation pipeline for a neural behavior classifier used in computational neuroscience research.

PythonPyTorchUMAPOpenCVTime-Series Analysis

The Problem

Understanding animal behavior from continuous pose-tracking and sensor data requires segmenting it into discrete, meaningful actions — and once you have a model that does that, you need real tooling to check whether its predictions actually hold up across many different recording sessions and labs, not just one dataset.

The Approach

Built the evaluation and behavioral-fingerprinting pipeline around DAART, a pretrained Temporal Convolutional Network classifier from the International Brain Laboratory (IBL), applying it across 57 real research sessions spanning 8 different labs — pulling data via the IBL's own ONE API and validating predictions both quantitatively and qualitatively.

DAART semi-supervised temporal convolutional network architecture diagram
The DAART model: an encoder maps behavioral features to a latent embedding, classified against sparse hand labels and dense heuristic labels, with a predictor forecasting future features for self-supervision.
Composite figure: video-overlay frames per behavioral state, paw-speed transition plots, and aligned discrete-state heatmaps
Top: video frames overlaid with paw position per predicted state (Still, Move, Wheel Turn, Groom). Middle: paw speed aligned to state transitions across trials. Bottom: predicted-state heatmaps for one session, split by correct vs. incorrect trials.
Model inference compared across Dropbox, IBL, and IBL-smoothed datasets
Validating the custom smoothing filter: state predictions and time-spent-per-state stay consistent whether run on the original data or the Savitzky-Golay-smoothed version.
Per-trial predicted state timeline aligned with wheel velocity and paw position traces
Two individual trials: predicted state sequence lined up against raw wheel velocity and paw position, showing the classifier tracking real behavioral transitions.
Histograms of time spent in each behavioral state across two trial periods
The raw material for the 88-dim behavioral fingerprint: state-duration histograms across 11 time bins, split by trial period (first movement to feedback vs. feedback to trial end).
UMAP projection of behavioral fingerprints across sessions and trial splits
UMAP projection of the 88-dim behavioral fingerprint — left: across sessions from two data sources; right: even vs. odd trials within a session, checking that a mouse's behavioral signature stays consistent.
  • ▹Loaded a pretrained daart.models.Segmenter (TCN) via PyTorch state dict, ran inference across sessions pulled with one.api.ONE and brainbox.io.one.SessionLoader
  • ▹Custom non-uniform Savitzky-Golay smoothing: fits a least-squares polynomial per window to handle irregular timestamps from dropped tracking frames, rather than assuming uniform sampling

Highlights

  • ▹Wrote a custom non-uniform Savitzky-Golay filter to smooth pose-tracking data that had irregular dropped/missing frames — the standard scipy implementation only handles uniformly-sampled data
  • ▹Built an 88-dimensional behavioral "fingerprint" per trial (state-duration histograms across time bins and task periods) and visualized cross-session structure with UMAP
  • ▹Validated the model two ways: quantitatively via macro-averaged F1 agreement scores and event-aligned heatmaps, and qualitatively via frame-by-frame video overlays of predicted states against real footage
  • ▹Pulled real multi-lab data (57 sessions, 8 IBL labs) via the IBL's ONE API and brainbox tooling, not a single canned dataset
View on GitHub