MinHash Similarity Sketch for a Job-Board Deduplication Pipeline
Overview
What this challenge is about.
Build a MinHash LSH deduplication pipeline for job postings, measure precision/recall, and earn a verifiable certificate.
The scenario
The startup (around USD 4M ARR, 80 percent of revenue from recruiter subscriptions) loses recruiters when search results are spammed with duplicates — deduplication quality is a direct retention lever.
The Brief
What you'll do, and what you'll demonstrate.
Build a MinHash + LSH sketch that finds near-duplicate job postings at a Jaccard threshold of 0.7 with at least 95 percent recall and at least 85 percent precision.
Earning criteria — what you'll demonstrate
- Derive the relationship between LSH bands (b), rows per band (r), and the S-curve probability of collision
- Implement MinHash with permutation-based hashing without leaking bias
- Measure approximate-vs-exact deduplication quality with labeled data
- Reason about the recall/precision trade controlled by b and r
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Randomized Algorithms
Master · Algorithms
Strong alignment
This challenge maps to Randomized Algorithms at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Minhash
Apply minhash to solve real industry problems and demonstrate production-level capability.
- Locality Sensitive Hashing
Apply locality sensitive hashing to solve real industry problems and demonstrate production-level capability.
- Probabilistic Data Structures
Apply probabilistic data structures to solve real industry problems and demonstrate production-level capability.
- Python Programming
Apply python programming to solve real industry problems and demonstrate production-level capability.
- Evaluation Metrics
Apply evaluation metrics to solve real industry problems and demonstrate production-level capability.
- Text Processing
Apply text processing to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles: