State Space Model POS Tagger
A part-of-speech tagger with hand-derived forward and backward passes through a state-space model — no autodiff.
The Problem
Part-of-speech tagging requires modeling sequential dependencies between words. Most modern taggers reach for an RNN or Transformer and an autodiff framework — I wanted to actually implement the underlying state-space model math by hand, forward and backward, to understand it properly.
The Approach
Built a structured state-space sequence model (the same family of ideas behind S4/Mamba-style models) entirely in MATLAB, including a bilinear (Tustin) discretization of the state transition matrix and a fully hand-derived backward pass — no deep learning framework, no automatic differentiation.
Try It Live
A real reimplementation of the model's forward pass (HiPPO-LegS state matrix, bilinear discretization, 4-tap convolution kernel) and its hand-derived backward pass — no autodiff, same as the original — training live in your browser right now, at a reduced 16-dim hidden state (the real model uses 64) on 320 template-generated toy sentences instead of the full CoNLL 2003 corpus.
Training… epoch 0/200
- ▹State transition matrix discretized via the bilinear (Tustin) transform: disc_state = (I − dt/2·A)⁻¹(I + dt/2·A) — the same discretization method used in S4-family state-space models
- ▹Forward pass: hidden state = Σⱼ kernel(j)·u(j) + residual·u, convolving the precomputed kernel over a 4-token context window (hidden dim 64) rather than recurrently stepping through the sequence
Highlights
- ▹Hand-derived both the forward AND backward pass through a bilinear-discretized state-space model — gradients for the output layer, classification weights, and convolution kernel are all computed manually via matrix calculus, not autodiff
- ▹Forward pass uses the SSM "convolutional view": convolves a precomputed kernel over a 4-token context window rather than a step-by-step recurrent scan
- ▹Trained on CoNLL 2003, collapsing the dataset's 46 fine-grained POS tags into 4 coarse categories via an explicit mapping
- ▹Early stopping after 5 epochs without validation improvement, on top of plain SGD — real training hygiene, not just a fixed epoch count