Granite Speech in FMS
Ported IBM's 8B-parameter Granite Speech model into IBM's Foundation Model Stack from scratch and validated it under torch.compile.
The Problem
Granite Speech only existed as a Hugging Face Transformers implementation — research-friendly, but built with dynamic control flow and framework glue that resists graph tracing. Getting it running end-to-end under torch.compile inside FMS meant rebuilding the model's architecture natively rather than wrapping the existing one, while staying numerically identical to IBM's reference implementation.
The Approach
Working with three Columbia HPML classmates (Aneesh Durai, Geonsik Moon, In Keun Kim), advised by IBM Research's Dr. Kaoutar El Maghraoui and Dr. Rashed Bhatti, reimplemented the Conformer encoder, Q-Former projector, and multimodal integration layer natively in FMS with zero Hugging Face dependencies in the new code path, built a weight-conversion pipeline to load IBM's original released checkpoints, and validated every component against both the reference HF implementation and its own torch.compile-traced form.
- ▹Component test breakdown: 53 Conformer encoder tests (28 unit + 25 HF-equivalence), 20 Q-Former projector tests, 62 full Granite Speech model tests, 11 generation tests
- ▹Dedicated torch.compile parity suite (test_granite_speech_torch_compile.py) confirming compiled and eager execution stay numerically equivalent at each stage
Highlights
- ▹Reimplemented Granite Speech's Conformer encoder, Q-Former projector, and multimodal integration entirely natively in FMS — no Hugging Face dependency in the feature-extraction path
- ▹Built a full HF → FMS checkpoint weight-conversion pipeline so the ported model loads IBM's original released weights directly
- ▹157 tests across 9 files (~2,535 lines of production code, ~1,500+ lines of tests) covering unit correctness, numerical equivalence against HF, and torch.compile activation/output parity
- ▹Benchmarked end-to-end on an H200 GPU and found the model is decoder-bound — 96–99% of latency lives in the LLM decoder, meaning encoder-side compilation work has little effect on real end-to-end speed
Results
On an NVIDIA H200 GPU (bf16), end-to-end latency was 489.4ms for 3s of audio, 532.3ms for 10s, and 2662.2ms for 30s — with 96.4–99.3% of that time spent in the LLM decoder rather than the encoder (encoder latency stayed roughly flat at 17.7–18.8ms regardless of clip length). Throughput ranged 120.7–249.0 tokens/sec (real-time factor 0.053–0.163) depending on clip length.