DPO Preference-Tune a Code Assistant for Style Compliance
Overview
What this challenge is about.
You receive a 7B coding base model, a client's published code-style guide (Python, around 80 pages), and a generated preference dataset (4,000 pairs of code snippets where one matches the style guide more than the other). Run DPO using TRL's DPOTrainer and compare against an SFT baseline on the chosen (style-conformant) outputs. Evaluate on a held-out 500-prompt suite using a style-linter (Ruff/Black + a custom checker) and a human review of 50 outputs. Write a 2-page methodology memo on when DPO beats SFT for style preferences.
The Brief
What you'll do, and what you'll demonstrate.
Use DPO to align a coding model with a client style guide and quantify when DPO beats SFT on style conformance and code correctness.
Earning criteria — what you'll demonstrate
- Implement DPO using TRL's DPOTrainer on a real coding model
- Compare DPO against SFT fairly on style and correctness
- Build automated style-conformance evaluation
- Reason about when preference optimization beats supervised fine-tuning
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
ML Researcher
Comparing DPO vs. SFT with proper beta sweeps and failure-mode galleries is the daily reality of applied LLM research at any consulting or model-as-a-service firm.
This challenge sharpens
- dpo
- preference-optimization
- llm-evaluation
AI Engineer
Owning the per-client preference-tuning pipeline plus a reusable decision tree is core AI-engineer work in consulting and platform-AI teams.
This challenge sharpens
- dpo
- trl
- fine-tuning
Applied AI Scientist
Translating preference-optimization results into a reusable client playbook is exactly what applied AI scientists ship at AI consulting firms.
This challenge sharpens
- preference-optimization
- code-generation
- llm-evaluation