Back to Projects

TDNN-Conformer ASR

A hybrid speech recognition architecture combining TDNN with the Conformer model.

PyTorchWeNettorchaudioLibriSpeech

The Problem

Speech recognition models typically lean on either Time Delay Neural Networks (TDNNs) for efficient convolutional temporal modeling, or Conformer blocks for capturing local and global context via self-attention. Combining their strengths in a single architecture is less explored — I wanted to know what a TDNN gets you when it replaces the Conformer's own convolution module rather than sitting alongside it.

The Approach

Built a hybrid encoder that keeps the Conformer's macaron block structure (feed-forward → self-attention → conv module → feed-forward) but replaces the conv module itself with a custom TDNN block, then trained a 12.5M-parameter model end-to-end inside the WeNet ASR toolkit on LibriSpeech.

TDNN-Conformer encoder architecture diagram
The full encoder pipeline and macaron-style block internals — the TDNN module replaces the Conformer's own convolution module.
TDNN module internal architecture diagram
Inside the TDNN module: pointwise conv → GLU gating → three parallel dilated convolutions → concat → merge.
Combined training loss curve over 40,000 steps
Combined CTC/attention training loss (0.3/0.7 blend) over 40k steps on a single RTX 2070.
Token-level training accuracy curve over 40,000 steps
Token-level accuracy climbing to ~87% over training.
  • ▹TDNN module internals: pointwise conv projects to the block width → GLU gate controls information flow → three depthwise convolutions run in parallel at increasing dilation (context sizes d, 2d, 3d) to capture short- and longer-range temporal patterns simultaneously → outputs concatenate and pass through a 1×1 merge conv, LayerNorm, ReLU, and a final pointwise conv
  • ▹Training recipe: AdamW (β₁=0.9, β₂=0.98), Transformer learning-rate schedule with a 25,000-step warmup to a 0.001 peak, gradient clipping at 10.0, batch size 12

Highlights

  • ▹Custom TDNN module: pointwise conv → GLU gating → three parallel dilated depthwise convolutions (dilations d, 2d, 3d) → concat → merge → LayerNorm → ReLU
  • ▹6-block encoder (dim 256, 4 attention heads, relative positional encoding), 6-layer Transformer decoder, 12.5M parameters total
  • ▹Trained with a hybrid CTC (0.3) / attention (0.7) loss, SpecAugment, speed perturbation, and label smoothing — the standard modern WeNet recipe
  • ▹Grounded the architecture in the original TDNN literature (Waibel et al., 1989) and factorized-TDNN work (Povey et al., 2018), rather than an ad hoc design

Results

On LibriSpeech test-clean, the model reached 14.63% WER via CTC prefix beam search (its best decoding mode) after 35 hours of training on a single consumer GPU. The accompanying write-up is explicit that this trails baseline Conformer variants (2.1–2.7% WER, trained on 960 hours with far more compute) — the goal was an honest, controlled comparison of the architectural change, not a leaderboard result, and the paper says so directly rather than overselling it.

View on GitHub