Hierarchical Image Classifier
A two-headed fine-tuned ResNet-50 with confidence-based detection of unseen categories.
The Problem
Standard flat image classifiers treat every category as equally distinct and have no real way to say "I don't know" when they see something outside their training categories — a hierarchical classifier that also knows what it doesn't know is more useful in practice.
The Approach
Working with two Columbia classmates, fully fine-tuned a ResNet-50 backbone with two independent classification heads — one predicting a coarse super-class, one predicting a fine-grained sub-class — trained jointly, plus a confidence-threshold rule to flag inputs the model has never seen a category for. Benchmarked against a CLIP/ViT-B-32 baseline to understand the real tradeoffs of each approach.
- ▹Backbone: torchvision ResNet-50 (pretrained, fully fine-tuned), final layer replaced with identity → 2048-dim features → two linear heads (coarse super-class + fine-grained sub-class, no additional hidden layers)
- ▹Loss = cross-entropy(super-class) + cross-entropy(sub-class), summed and backpropagated jointly; AdamW at lr=1e-4 with cosine annealing (T_max=30)
Highlights
- ▹Two-headed architecture on a fully fine-tuned (not frozen) ResNet-50 backbone, trained with two independent cross-entropy losses summed together
- ▹Novelty detection via softmax confidence thresholding, flagging inputs whose top prediction confidence falls below a tuned threshold
- ▹Directly benchmarked against a CLIP/ViT-B-32 baseline and found a genuine, documented tradeoff rather than a one-sided win
- ▹Mixed-precision training with a cosine-annealed learning rate on a Colab T4 GPU
Results
On the held-out evaluation leaderboard, the model reached 75.89% overall super-class accuracy and 53.39% sub-class accuracy. Benchmarked directly against a CLIP/ViT-B-32 baseline, the tradeoff was clear and worth documenting honestly: CLIP generalized far better to genuinely novel, unseen categories (80.4% vs. 16.6% accuracy), while this fine-tuned model was substantially stronger at fine-grained sub-classification on categories it had seen (53.4% vs. 13.1%) — a real architectural tradeoff between zero-shot generalization and fine-grained specialization, not a simple win or loss.