Ewance
← All challenges Applied AI Scientist

Fine-Tune a Vision-Language Model for Image Captioning

Duration
3 weeks
Level
Advanced
Type
Research
Team
1 to 5 people

The situation

The Tel Aviv consumer AI startup (around 18 staff, post-seed) ships an iOS-first accessibility app with around 40,000 active users and treats caption usefulness as its core differentiation.

Your task

Fine-tune a vision-language model so its captions are actually useful for low-vision users, validated by a 30-user study.

The brief

What you do

  1. Take BLIP-2 or LLaVA-1.6 as the base.
  2. Fine-tune (LoRA is fine) on a 4,000-image accessibility-curated dataset where each image has a useful caption written by a low-vision-experienced annotator.
  3. Use an instruction-following caption style.
  4. Evaluate with both automated metrics (CIDEr, SPICE) and a 30-user study comparing fine-tuned vs base captions on perceived usefulness.
  5. Report a Likert-scale comparison.
  6. Write a 4-page memo with a go/no-go recommendation on shipping the fine-tune.

What you will practice

  • Fine-tune a vision-language model with parameter-efficient methods
  • Design a user study that measures real downstream usefulness
  • Balance automated metrics with human judgment
  • Make a ship/no-ship call on a model fine-tune

A suggested way in

  1. LoRA on the language head only is often enough for caption-style adaptation
  2. Automated caption metrics correlate poorly with usefulness — the user study is the source of truth
  3. Always include base-model captions in the user study unblinded to control for expectation bias

Skills you will practice

Vision language models, Lora fine tuning, Pytorch, User study design, Image captioning, Evaluation

Where it leads

Applied AI Scientist, ML Researcher, AI Product Designer

How it is assessed

  • Fine-tune quality25%

    Fine-tuned captions are visibly better than base on the sample gallery.

  • Automated metrics20%

    CIDEr/SPICE reported honestly, with caveats about their limits.

  • User-study design30%

    Study has a clear hypothesis, paired comparisons, honest sample-size caveats.

  • Memo quality25%

    Ship/no-ship recommendation is decisive and traceable to the data.

Resources

  • PyTorchTool

    Underlying framework for vision-language model fine-tuning.

  • Hugging Face PEFTTool

    LoRA implementation that keeps the fine-tune budget tractable.

  • Weights & BiasesTool

    Track training loss + automated metrics across fine-tuning runs.

  • Google FormsTool

    Run the 30-user paired-comparison study cheaply.

  • Lambda LabsTool

    Affordable per-hour GPU rental for the fine-tuning budget.

  • NotionTool

    Compose the 4-page memo with embedded sample gallery.

Sample Ewance certificate of completion, verified by LearnCoin
Sample certificate
Your proof

Finish it. Earn this.

A verified certificate with your name. Anyone can check it.

Time to practice.

Solve this challenge
Related challenges

More like this.