Build a Crawler-and-Topic Pipeline for Public-Sector Web Analytics
Overview
What this challenge is about.
Build a Crawler-and-Topic Pipeline for Public-Sector Web Analytics. Intermediate challenge in code. Writing production code that solves real engineering prob...
The Brief
What you'll do, and what you'll demonstrate.
Build a crawler + topic pipeline + dashboard that surfaces civic concerns on municipal forums in near-real-time.
This is not a coding exercise. It is the work a software engineer does between a Jira ticket and a merged PR. That distinction matters to every hiring manager who has seen candidates solve LeetCode problems and none who have shipped production code under real constraints.
When you finish, you will have something most graduates do not: a real-world deliverable, verified by Ewance, that you can show to a hiring manager and say "I did this. Here is the proof."
Earning criteria — what you'll demonstrate
- Build an ethical, compliant web crawler at modest scale
- Apply modern topic modeling to bilingual short-text data
- Design an analyst dashboard with delta-over-time views
- Document data-handling and privacy choices defensibly
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Aligned coursework coming soon.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Web Crawling
Apply web crawling to solve real industry problems and demonstrate production-level capability.
- Topic Modeling
Apply topic modeling to solve real industry problems and demonstrate production-level capability.
- Nlp Pipeline
Apply nlp pipeline to solve real industry problems and demonstrate production-level capability.
- Dashboard Design
Apply dashboard design to solve real industry problems and demonstrate production-level capability.
- Python
Write clean, efficient Python for data processing, automation, and backend services.
- Data Storytelling
Apply data storytelling to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Data Engineer
Owning a crawler-to-dashboard pipeline with privacy discipline is the entry-point work for a data engineer on a civic-tech or analytics team.
This challenge sharpens
- web-crawling
- nlp-pipeline
- dashboard-design
NLP Engineer
Applying modern topic-modeling to bilingual short text is the daily reality of NLP engineers in govtech and analytics.
This challenge sharpens
- nlp-pipeline
- topic-modeling
- python
AI Product Designer
Designing a usable analyst surface for topic-delta exploration is exactly an AI product designer's craft in mission-driven orgs.
This challenge sharpens
- dashboard-design
- data-storytelling
- topic-modeling