Build a Serverless ETL Pipeline for a Climate-Tech Sensor Fleet
Overview
What this challenge is about.
Build the pipeline using managed services only (e.g., S3 + Lambda + EventBridge + Glue, or GCS + Cloud Functions + Cloud Scheduler + BigQuery external tables). Source the data from a public sample (or a generator that simulates 50M rows/day). Pipeline must: (1) ingest in 5-minute micro-batches, (2) validate against a schema and quarantine bad rows, (3) land Parquet partitioned by date+sensor_region, (4) be safely re-runnable for any 24-hour window. Deliver Terraform or Pulumi infrastructure-as-code, a load test report, and a 4-page ADR.
The Brief
What you'll do, and what you'll demonstrate.
Replace a fragile cron pipeline with a serverless, idempotent ETL that survives single-region cloud incidents and ingests 50M rows/day reliably.
Earning criteria — what you'll demonstrate
- Compose a real serverless ETL pipeline using managed cloud primitives
- Apply idempotency and reprocessing patterns
- Validate and quarantine bad data at ingest
- Write an Architecture Decision Record that survives executive review
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
Data Engineer
Serverless ETL on managed cloud primitives with infrastructure-as-code is the day-one work of a junior data engineer at any IoT or climate-tech company.
This challenge sharpens
- etl-pipelines
- serverless-architecture
- data-validation
MLOps Engineer
Idempotent pipelines + infrastructure-as-code + monitoring habits transfer directly into MLOps work on production model pipelines.
This challenge sharpens
- infrastructure-as-code
- monitoring-design
- cloud-services
AI Solutions Architect
Writing an Architecture Decision Record that survives executive review is the core craft of an AI solutions architect.
This challenge sharpens
- serverless-architecture
- cloud-services
- infrastructure-as-code