Audit BLEU vs. COMET on a Multilingual Customer-Support Corpus
Overview
What this challenge is about.
You receive 600 source-translation-reference triples covering 6 languages (EN as source; ES/FR/DE/JA/PT-BR/HI as targets), each scored on adequacy and fluency (1-6) by 3 professional translators. Compute BLEU, chrF++, COMET-22, and COMET-Kiwi (reference-less) per segment. Report Pearson and Kendall-tau correlations with the human scores per language. Discuss systematic failure cases (e.g., morphologically rich targets, code-switching). Recommend the dashboard's metric setup in a 4-page memo.
The Brief
What you'll do, and what you'll demonstrate.
Decide which automatic MT metric (or mix) the internal quality dashboard should standardize on, backed by per-language correlation with human judgement.
Earning criteria — what you'll demonstrate
- Apply lexical and learned MT metrics across multiple language pairs
- Quantify metric-to-human-judgement correlation
- Diagnose where individual metrics systematically fail
- Recommend a multi-metric dashboard stack with explicit reasoning
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
Applied AI Scientist
Metric-stack audits across multiple languages and human judgements are the applied-AI-scientist's contribution to any multilingual ML org.
This challenge sharpens
- mt-evaluation
- statistical-analysis
- multilingual-evaluation
NLP Engineer
Knowing where each MT metric breaks per language is what makes NLP engineers credible on internationalization-heavy teams.
This challenge sharpens
- mt-evaluation
- neural-mt
- multilingual-evaluation
ML Researcher
Designing a fair metric-vs-human study with per-language correlations mirrors the rigor expected in MT-research projects.
This challenge sharpens
- statistical-analysis
- benchmarking
- mt-evaluation