Wikipedia Analysis
URL → scrape → tokenize → top frequent words with visuals.
- ROLE
- Developer
- DOMAIN
- Student work
OVERVIEW
A small Flask app that takes a Wikipedia URL, scrapes the article text with BeautifulSoup, filters out stopwords, and charts the most frequent words. It is stateless: each request is scraped and counted fresh, with no database behind it. The point is a quick read of what an article emphasizes before reading all of it.
ARRIVED AS
“Quickly summarize long articles.”
I wanted a quick read on the key themes of long Wikipedia articles. Instead of reading thousands of words, I wanted the most frequent terms to show what the article emphasizes. That meant handling Wikipedia's messy HTML, getting through long text quickly, and showing the result as a chart you can read at a glance.
WHAT I BUILT
- 01BeautifulSoup to scrape; tokenization + stopword removal.
- 02Frequency counts → charts; simple Flask UI.
WHAT CHANGED
- Minutes to insights; handy for research & notes.
ARCHITECTURE
A plain pipeline: a Flask web server receives Wikipedia URLs from users, a scraping layer extracts article content using BeautifulSoup, a text processing engine tokenizes and filters the content, and a visualization layer generates frequency charts. It keeps no state and has no database. The frontend is a little JavaScript and Chart.js; the backend does the counting.
Data flow
click a stage
User submits Wikipedia article URL through the web interface. URL validation ensures it's a proper Wikipedia domain to prevent scraping arbitrary websites.
COMPONENT
Flask APILightweight REST endpoint that accepts Wikipedia URLs, runs the scraping and analysis pipeline, and returns JSON data containing top terms and their frequencies for frontend rendering.
Decisions, with the cost of each.
A decision without its trade-off is marketing. Each row says what was chosen, why, and what it gave up.
Stateless processing
Results aren't saved to a database. Each request is processed fresh, so there's nothing to maintain and the tool stays light and deployable anywhere.
AT THE COST OF
No history: every analysis starts from scratch, so past results cannot be revisited or compared.
BeautifulSoup over the API
Wikipedia offers an official API, but BeautifulSoup on raw HTML provided more control over content extraction, specifically the ability to exclude specific sections like references and external links that would skew frequency analysis with non-article terms.
AT THE COST OF
Tied to Wikipedia's HTML layout: a markup change can break the selectors, where the API would stay stable.
An extended stopword list
Standard NLTK stopwords weren't sufficient for Wikipedia content. Added domain-specific stopwords like 'wikipedia', 'cite', 'reference', 'retrieved', 'archive' to filter out meta-content that appears frequently but doesn't relate to article topics.
AT THE COST OF
A hand-kept list: new meta terms have to be noticed and added by hand.
Show the top 25 only
The chart shows the top 25 terms, not every unique word. That keeps attention on the main themes and the chart readable. Long-tail words (appearing 1-2 times) add noise without insight.
AT THE COST OF
A rare but telling word never reaches the chart.
The part that mattered.
The code that did the work.
def scrape_wikipedia(url):
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
# Extract main content, exclude references and nav
content_div = soup.find('div', {'id': 'mw-content-text'})
if content_div is None:
return ""
# Remove unwanted sections
for unwanted in content_div.find_all(['table', 'sup', 'div'], class_=['infobox', 'reference', 'navbox']):
unwanted.decompose()
return content_div.get_text()
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
def process_text(text):
# Tokenize and normalize
tokens = word_tokenize(text.lower())
from nltk.probability import FreqDist
# Extended stopword list
stop_words = set(stopwords.words('english'))
wiki_stops = {'wikipedia', 'cite', 'reference', 'retrieved', 'archive', 'edit', 'page'}
all_stops = stop_words.union(wiki_stops)
# Filter and count
filtered = [t for t in tokens if t.isalpha() and t not in all_stops]
freq_dist = FreqDist(filtered)
return freq_dist.most_common(25)
@app.route('/analyze', methods=['POST'])
def analyze():
url = request.json.get('url')
if not is_valid_wikipedia_url(url):
return jsonify({'error': 'Invalid Wikipedia URL'}), 400
try:
text = scrape_wikipedia(url)
top_words = process_text(text)
return jsonify({
'success': True,
'data': {
'words': [w[0] for w in top_words],
'frequencies': [w[1] for w in top_words]
}
})
except Exception as e:
return jsonify({'error': str(e)}), 500
LEARNED
Domain text needs domain filtering
Generic NLP needs tuning for each domain. Wikipedia articles contain structural elements (citations, infoboxes, navigation) that must be filtered out. Without a Wikipedia-specific stopword list, metadata terms dominate the counts.
Scraping is brittle
Wikipedia's HTML structure can change, breaking CSS selectors. The scraper needed defensive code: check elements exist before touching them, handle the different page types (disambiguation, redirect, standard), and degrade gracefully when the structure doesn't match.
Quick beats exact here
Perfect linguistic analysis wasn't the point; quick answers were. I chose simple tokenization over advanced NLP (stemming, lemmatization, named entity recognition) because speed and simplicity outweighed small accuracy gains. For a quick read of an article, a rougher answer now beat a finer one later.
The chart is the product
Raw frequency lists are hard to interpret. With a bar chart, users could see an article's themes at a glance. The chart mattered as much as the counting.
It does not scale
Processing individual Wikipedia articles works well, but the architecture doesn't scale to bulk analysis of hundreds of articles. If I came back to it, I'd add caching and batch processing, and maybe move to Wikipedia dumps for large-scale corpus analysis rather than real-time scraping.