Design Parallel I/O for a Climate-Simulation Data Pipeline
Overview
What this challenge is about.
Replace serial HDF5 with parallel MPI-IO for a climate simulation, benchmark on Lustre, and deliver a report. You get a verifiable certificate.
The scenario
The center (publicly funded, runs roughly 600 simulations/year) cannot expand to higher-resolution ocean grids until I/O stops being the binding constraint.
The Brief
What you'll do, and what you'll demonstrate.
Replace per-rank serial HDF5 output with parallel HDF5 + MPI-IO collective writes, cutting I/O time to under 15 percent of wall-clock at 512 ranks.
Earning criteria — what you'll demonstrate
- Replace per-rank file output with collective parallel I/O
- Align HDF5 chunk size to Lustre stripe size for max bandwidth
- Use MPI-IO collective writes to amortize metadata cost
- Benchmark parallel I/O at scale and identify the binding constraint
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
High-Performance and Scientific Computing
Master · Systems
Strong alignment
This challenge maps to High-Performance and Scientific Computing at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Mpi Io
Apply mpi io to solve real industry problems and demonstrate production-level capability.
- Parallel Hdf5
Apply parallel hdf5 to solve real industry problems and demonstrate production-level capability.
- Lustre Filesystem
Apply lustre filesystem to solve real industry problems and demonstrate production-level capability.
- Fortran Programming
Apply fortran programming to solve real industry problems and demonstrate production-level capability.
- Chunking Strategy
Apply chunking strategy to solve real industry problems and demonstrate production-level capability.
- Performance Analysis
Apply performance analysis to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles: