Build a Crawler-and-Topic Pipeline for Public-Sector Web Analytics
Overview
What this challenge is about.
Build a polite crawler and topic pipeline for public forums, then surface trends in a Streamlit dashboard. Earn a verifiable certificate.
The scenario
The agency (around 200 staff, public-funded) currently relies on manual scans of these forums; senior advisors miss emerging concerns by 2-3 weeks, which has cost two recent service-redesign cycles.
The Brief
What you'll do, and what you'll demonstrate.
Build a crawler + topic pipeline + dashboard that surfaces civic concerns on municipal forums in near-real-time.
Earning criteria — what you'll demonstrate
- Build an ethical, compliant web crawler at modest scale
- Apply modern topic modeling to bilingual short-text data
- Design an analyst dashboard with delta-over-time views
- Document data-handling and privacy choices defensibly
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Aligned coursework coming soon.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Web Crawling
Apply web crawling to solve real industry problems and demonstrate production-level capability.
- Topic Modeling
Apply topic modeling to solve real industry problems and demonstrate production-level capability.
- Nlp Pipeline
Apply nlp pipeline to solve real industry problems and demonstrate production-level capability.
- Dashboard Design
Apply dashboard design to solve real industry problems and demonstrate production-level capability.
- Python
Write clean, efficient Python for data processing, automation, and backend services.
- Data Storytelling
Apply data storytelling to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Data Engineer
Owning a crawler-to-dashboard pipeline with privacy discipline is the entry-point work for a data engineer on a civic-tech or analytics team.
This challenge sharpens
- web-crawling
- nlp-pipeline
- dashboard-design
NLP Engineer
Applying modern topic-modeling to bilingual short text is the daily reality of NLP engineers in govtech and analytics.
This challenge sharpens
- nlp-pipeline
- topic-modeling
- python
AI Product Designer
Designing a usable analyst surface for topic-delta exploration is exactly an AI product designer's craft in mission-driven orgs.
This challenge sharpens
- dashboard-design
- data-storytelling
- topic-modeling