A one-page report that says which of two small models is better at maths word problems, with error bars, sorted failures and a test you can rerun.
Before running anything, write down the success measure and the rule you'll decide by, then take 200 questions from GSM8K's test split on Hugging Face.
Write a strict checker that pulls the final number from each reply, and run two small models in Ollama (say llama3.2:3b and qwen2.5:3b) at temperature 0 with the same prompt.
Read 30 failures from each model, sort them into piles such as arithmetic slip, misread question and bad format, and count the piles.
Have a third, larger model grade whether each reply's reasoning holds up, and check it against your own marks on 30 replies with Cohen's kappa.
Put 95% bootstrap intervals on both scores, and run a paired sign test on the questions where the two models disagree.
Turn the checker and 20 fixed questions into a regression test that runs with one command.
How you'll know it worksA one-page report with both scores and their intervals, the sign test's p-value, the failure counts, the grader's kappa and a decision made by your rule, plus a regression test that passes from a clean checkout.
Python, Ollama with two small models and a larger grader such as qwen2.5:7b (no API key; an hour or two of laptop time), SciPy, scikit-learn and pytest. Uses lessons 01 to 16.
Project: An evaluation report on two local models
From insideout, Evaluation: A one-page report that says which of two small models is better at maths word problems, with error bars, sorted failures and a test you can rerun.
Steps:
1. Before running anything, write down the success measure and the rule you'll decide by, then take 200 questions from GSM8K's test split on Hugging Face.
2. Write a strict checker that pulls the final number from each reply, and run two small models in Ollama (say llama3.2:3b and qwen2.5:3b) at temperature 0 with the same prompt.
3. Read 30 failures from each model, sort them into piles such as arithmetic slip, misread question and bad format, and count the piles.
4. Have a third, larger model grade whether each reply's reasoning holds up, and check it against your own marks on 30 replies with Cohen's kappa.
5. Put 95% bootstrap intervals on both scores, and run a paired sign test on the questions where the two models disagree.
6. Turn the checker and 20 fixed questions into a regression test that runs with one command.
How you'll know it works: A one-page report with both scores and their intervals, the sign test's p-value, the failure counts, the grader's kappa and a decision made by your rule, plus a regression test that passes from a clean checkout.
Runs on: Python, Ollama with two small models and a larger grader such as qwen2.5:7b (no API key; an hour or two of laptop time), SciPy, scikit-learn and pytest.
Help me build this on my machine, one step at a time. Ask what I've tried first, give hints before full answers, and check my work against "How you'll know it works".