Wikipedia Article Clustering
An unsupervised pipeline that clusters Wikipedia articles by real semantic similarity.
PythonSentence-BERTUMAPScikit-LearnNLP
The Problem
Large text corpora like Wikipedia contain natural topic structure that isn't labeled — surfacing it requires methods that capture real semantic similarity, not just keyword overlap.
The Approach
Built an end-to-end pipeline: live-scrape ~155 real Wikipedia articles across 5 categories via the Wikipedia API, clean and lemmatize the text, embed it with a sentence-transformer model, reduce to 2D with UMAP, and cluster with K-Means — choosing the number of clusters systematically rather than guessing.
Try It Live
- ▹Embedding: SentenceTransformer('paraphrase-MiniLM-L6-v2') → StandardScaler normalization → UMAP(n_components=2)
- ▹Cluster selection: loops k=2..19, fits KMeans(init='k-means++') for each, and keeps the k with the maximum silhouette score
Highlights
- ▹Scrapes real articles live from the Wikipedia API (155 articles, 5 categories) rather than using a prepackaged dataset
- ▹Uses Sentence-BERT embeddings (paraphrase-MiniLM-L6-v2) to capture real semantic meaning, not bag-of-words or TF-IDF
- ▹Chooses K for K-Means systematically — fits every k from 2 to 19 and selects the one with the highest silhouette score, instead of picking an arbitrary cluster count
- ▹Full text cleanup pipeline: citation-marker stripping, tokenization, stopword removal, and lemmatization before embedding