Compare
LLM-as-judge vs human graders
People decide what good means, and a checked judge marks the rest.
The short answer
Use people to decide what good means: they write the rubric, mark a sample, and make the call on expert work and big launches. Use a model judge to apply that rubric to hundreds of answers on every change, once you've checked it against people on 50 to 100 of them. Until it's checked, a judge's scores tell you little.
Model judge
Use it when
- You re-grade on every change.
- Hundreds of open-ended answers need marking each run.
- You have a written rubric and a human-marked sample to check it against.
- What it costs
- Dollars, not hours: at $0.002 a call, 500 answers marked on 4 rubric parts is about $4 a run. Plus a human-marked sample to check it against.
- Watch out
- Judges favour the first slot, longer answers and their own model family. Judge pairs in both orders, and pin the judge's model and prompt so scores stay comparable.
Human graders
Use it when
- The task is new, with no rubric yet.
- The call needs expertise or taste, such as medical accuracy or a brand's voice.
- A big launch, where a wrong call is costly.
- What it costs
- Hours, not cents. 300 answers, each marked by two people at three minutes an answer, is thirty hours of work: $900 at an example $30 an hour.
- Watch out
- People disagree too. Measure agreement beyond chance with Cohen's kappa, and when it's middling, fix the guidelines before marking more.
At a glance
| Question | Model judge | Human graders |
|---|---|---|
| Speed | Hundreds of answers in minutes | Days for a few hundred answers |
| Cost | Cents to dollars a run | Hundreds of dollars for a small study |
| Consistency | Steady while its model and prompt stay pinned | Drifts with fatigue; needs overlap and known answers |
| Known biases | Position, length and its own model family | Unclear guidelines and personal taste |
| Facts it doesn't know | Guesses, unless you give it the source | An expert can check them |
| Best use | Grading every change against a checked rubric | Setting the standard and checking the judge |
How to choose
- Write the rubric first, with a description of every level, so a person and a model can mark the same way twice.
- Have two people mark 50 to 100 answers, measure their agreement, and fix the guidelines until it holds.
- Run the judge on the same answers and count how many of the bad ones it catches, as well as how often it agrees.
- If it catches enough, let it grade every change, and redo the human check whenever the judge's model or prompt changes.
- Keep people on the calls a judge can't make: expert facts, taste, and the final word on a big launch.
When you need both
Learn it properly
Questions
Can an LLM judge replace human evaluation?
For routine runs, mostly, once it has been checked. It can't replace the people who decide what good means, and it stays weak on facts it doesn't know and maths it can't do, so experts still grade those.
How do I know my LLM judge is any good?
Have people mark a sample, often 50 to 100 answers, with the same rubric, run the judge on the same answers, and compare. Look past raw agreement: when most answers pass, a judge that passes everything already agrees most of the time. Count how many failures it catches.
What biases do LLM judges have?
Three come up again and again: favouring an answer for its position, giving points for length, and preferring output from their own model family. Judge both orders, say in the rubric that length earns nothing, and use a judge from a third family when comparing two providers.
Should a judge score one answer or compare two?
Score answers one at a time to track a system over time or hold a fixed bar. Compare two side by side to choose between versions, since small differences show up more clearly when the judge only has to pick the better one.