LLM judge calibration against human ratings: a protocol that ends in a decision per criterion

Who this guide is written for.

• CTOs and VPs of Engineering whose release gate is an LLM judge score and who have been asked, by a customer or an auditor, why that score should be trusted
• Teams that have tried several evaluation tools, wired a judge into CI, and still review outputs by hand because nobody trusts the number
• Platform and ML engineers who changed the judge model or its prompt and saw the pass rate move without any change to the product
• Product owners of agents in production who need evaluation after launch and only have a pre-launch benchmark

Why a judge needs calibration at all

An LLM judge is a language model asked to score another model's output against a rubric. It inherits the properties that make language models hard to evaluate in the first place, and the research on this is specific. Zheng and co-authors (2023), who introduced MT-Bench and Chatbot Arena, name position bias, verbosity bias, self-enhancement bias and limited reasoning ability as the judge's failure modes. Wang and co-authors (2023) flipped the ranking of two candidate answers by swapping their order in the prompt: Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the evaluator, through reordering alone. Saito and co-authors (2023) measured that GPT-4 prefers longer answers more than humans do. Panickssery, Bowman and Feng (2024) found that evaluators recognise their own generations and favour them in proportion to how well they recognise them. Ye and co-authors (2024) catalogued twelve biases in LLM judges, among them authority, bandwagon, sentiment, fallacy oversight and refinement awareness, where being told an answer was refined changes the score.

None of this makes the judge useless. Zheng and co-authors also reported that a strong judge reached over 80% agreement with human preferences, the level humans reached with each other. The point is that the level is measured, per task, and a judge dropped into a CI pipeline without that measurement is a second system whose errors nobody has looked at. Practitioners on developer forums describe exactly this state: outputs eyeballed and called done, a judge from the same family scoring its own answers, and no evaluation at all once the agent is in production. Calibration turns the judge's score from an opinion into a number with a known error.

Humans first: the reference set and how well people agree with each other

A judge can only be as good as the labels it is compared against, so the protocol starts with people and with the rubric. Each criterion gets a written definition, a scale and worked examples per scale point. A sample of real outputs, drawn from production traffic or the held-out set, is labelled independently by at least two raters who cannot see each other's scores. Disagreements go to adjudication, and the adjudicated label becomes the reference. Every rubric edit made during adjudication is recorded, because a rubric people could not apply consistently is the first artifact that needs fixing, before the judge is run at all.

Agreement between the humans is measured per criterion, with a chance-corrected statistic next to raw percent agreement: Cohen's kappa for two raters, Krippendorff's alpha for more raters or missing labels, Spearman correlation for ordinal scores. McHugh (2012) recommends reporting both percent agreement and kappa and reads any kappa below 0.60 as inadequate, while the older Landis and Koch scale calls 0.41 to 0.60 moderate; the protocol sets no universal threshold, because the right one depends on what a wrong pass costs in your product. What the human-human figure does fix is the ceiling: a judge cannot be expected to agree with people more often than people agree with each other on that criterion. A criterion where humans disagree is not ready to be automated, whatever the judge reports.

Measuring the judge: the calibration protocol

With the reference set in place the judge scores the same sample, and every one of its known biases gets its own test. The table is the protocol as run; the source links at the end of the page carry the reasoning behind each row.

The decision per criterion

The output of calibration is a decision, written per criterion, on what the judge is allowed to do. Three outcomes cover practice. The judge gates alone where its agreement with the adjudicated labels matches the level the humans reached with each other and the confusion matrix shows its errors falling on the cheap side. The judge screens where its agreement is lower but false passes are rare: it scores everything, and every fail plus a random sample of passes go to a person. The criterion stays with people where the humans themselves disagreed, and the work goes back to the rubric.

This decision is what gets wired into CI. A single pass rate across criteria is replaced by per-criterion gates with different owners, and the calibration set is stored beside the test suite, versioned with the rubric. A release that changes the judge prompt reruns calibration before it changes the gate.

• Gate alone: agreement at the human ceiling, errors on the cheap side; the judge blocks the merge on this criterion.
• Screen and route: agreement below the ceiling, false passes rare; the judge sorts, a person decides on the flagged share.
• Stay human: humans disagree with each other; rubric is rewritten and the criterion is relabelled before the judge is tried again.
• Every decision names the criterion, the statistic, the value, the human ceiling and the person who owns the gate.

Recalibration and evaluation after launch

A calibrated judge is calibrated for one model version, one prompt and one rubric. Chen, Zaharia and Zou (2023) measured the same GPT-4 endpoint in March and June 2023 and found its accuracy at identifying prime numbers went from 84% to 51%, with instruction following degrading alongside. A judge served through an API can change under you the same way, so the judge model is pinned to a version identifier and recalibration runs on four triggers: judge model or version change, judge prompt change, rubric change, and a shift in what the product sends, such as a new language or task type.

Calibration before launch is half the work. After launch the judge scores a sample of production traces, and at a fixed cadence a person relabels a random slice of what the judge scored. Agreement becomes a time series per criterion, and a drop is the signal to look at the judge or the product before customers do. OpenAI's grader documentation describes the same check from the training side: a model that has learned to satisfy the grader scores well with the grader and poorly with expert humans, and the only way to see that is to keep asking the humans.

Where this goes wrong, and what we do about it.

1. 1
   The judge comes from the family that produced the answers

   The same model, or a sibling from the same provider, generates the output and scores it. Panickssery, Bowman and Feng (2024) showed that evaluators recognise their own text and favour it in proportion to how well they recognise it, so the pass rate carries a bonus for style that no human gave. The team reads a high score as quality because nothing in the pipeline reports which family scored what.

   What we do about it

   We rerun the calibration sample with a judge from a different family and with the candidate's identity hidden, and we compare agreement with the adjudicated labels for both judges. Where the same-family judge scores its own outputs higher than humans do, the judge is replaced or the criterion moves to screening. We record judge model and version next to every score so the family is visible in every report.

   What stays with you: agreement comparison for two judge families on the same sample, and a judge specification with model, version and prompt hash
2. 2
   Order and length decide the verdict

   Pairwise comparisons are sent in a fixed order and the judge favours the first or the second slot; single-answer scores rise with output length regardless of correctness. Wang and co-authors (2023) flipped a benchmark result by reordering alone, and Saito and co-authors (2023) measured that GPT-4 prefers longer answers more than humans do. A prompt change that makes the product more verbose then reads as a quality improvement.

   What we do about it

   We run every pairwise judgement in both orders and count the share of pairs where the verdict changes, and we compute the correlation between score and length for the judge and for the human labels on the same sample. Pairs that flip are excluded from the gate until the prompt is fixed, and a length correlation that the humans do not share moves the criterion to screening.

   What stays with you: position consistency rate and score-length correlation per criterion, judge next to humans
3. 3
   Judge agreement measured against one person's labels

   One engineer labels the reference set alone, the judge agrees with those labels most of the time, and the number is presented as calibration. Nobody knows whether a second person would have labelled the same way, so the figure has no ceiling to be read against. When the judge later disagrees with a reviewer in production, there is no way to say which of them is off.

   What we do about it

   We have every item labelled independently by at least two raters, adjudicate disagreements, and report the human-human statistic per criterion before any judge figure. Criteria where the humans do not agree go back to the rubric with the disagreements attached, because those are the definitions people could not apply, and a judge cannot be calibrated against them.

   What stays with you: reference set with two independent labels and an adjudicated label per item, and the human-human agreement per criterion
4. 4
   One aggregate score hides the criterion that fails

   The judge returns a single number per output, or per-criterion scores are averaged into a pass rate. Agreement on the aggregate looks acceptable while one criterion, typically the one that costs the most when wrong, has the judge passing what humans fail. The average hides it because the other criteria are easy and numerous.

   What we do about it

   We compute agreement and a confusion matrix per criterion and name the cost of each cell: a false pass on a safety or correctness criterion is weighted differently from a false fail on tone. Each criterion gets its own decision and its own gate owner, and the aggregate score is dropped from the pipeline.

   What stays with you: confusion matrix per criterion with the cost of each cell, and a decision register with one row per criterion
5. 5
   The judge model changed and the calibration did not

   The judge is called through an alias that points at the provider's current version, or its prompt is edited during an unrelated change. Chen, Zaharia and Zou (2023) documented behaviour of the same endpoint moving substantially within three months. The gate keeps passing or starts failing, and the team debugs the product for a change that happened in the judge.

   What we do about it

   We pin the judge to a version identifier, hash the judge prompt, and add a test that fails when either changes without a recalibration run recorded against it. Recalibration reuses the same reference set, so the before and after agreement are directly comparable, and the decision register is updated with the date and the new values.

   What stays with you: recalibration log with agreement per criterion before and after each judge or rubric change, and the pinned judge version

What stays with you.

• Calibration set: sampled outputs with two independent human labels and an adjudicated label per item and criterion, versioned with the rubric
• Rubric with definitions, scales and worked examples per scale point, including the edits made during adjudication and why
• Agreement report per criterion: human-human and judge-human statistics, confusion matrices with costed cells, position swap and length control results
• Decision register: for every criterion, gate alone, screen and route, or stay human, with the values behind the decision and the gate owner
• Judge specification: model family, pinned version, prompt hash, order randomisation and temperature, with the CI check that fails when any of them changes
• Recalibration procedure with its four triggers, and a production sampling plan with the relabelling cadence and the agreement time series it feeds

How we work.

1. 1
   1. Rubric and reference set

   We take the criteria you already gate on, write or tighten the definition, scale and examples for each, and draw a sample from production traces or your held-out set. The sample is stratified so that rare but expensive cases are present in numbers that can be measured.

   Week 1-2
2. 2
   2. Human labelling and adjudication

   Two raters from your team, or ours where you prefer an outside pair, label every item independently. We adjudicate disagreements with you, record every rubric change, and report human-human agreement per criterion. Criteria that fail here go back to step 1 before the judge is touched.

   Week 2-3
3. 3
   3. Judge measurement

   We run your current judge and at least one judge from a different family on the reference set, in both orders for pairwise items, with and without a reasoning step. We compute agreement, confusion matrices, position consistency and length correlation per criterion.

   Week 3-4
4. 4
   4. Decision and CI wiring

   We write the decision register with you, replace the aggregate gate with per-criterion gates and owners, pin the judge version and prompt, and add the check that blocks a judge change without a recorded recalibration.

   Week 4-5
5. 5
   5. Handover and first production cycle

   We hand over the calibration set, the report and the recalibration procedure, run the first production relabelling with your team, and leave the agreement time series in your observability stack so the next drop is visible without us.

   Week 5-6

The rest of this cluster.

Where this turns into a service.

Where the dates and numbers come from.

• Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: position, verbosity and self-enhancement bias; over 80% agreement with humans, matching human-human agreement
• Wang et al. (2023), Large Language Models are not Fair Evaluators: position bias, Vicuna-13B beating ChatGPT on 66 of 80 queries by reordering, balanced position calibration
• Panickssery, Bowman and Feng (2024), LLM Evaluators Recognize and Favor Their Own Generations: self-recognition correlates with self-preference
• Saito et al. (2023), Verbosity Bias in Preference Labeling by Large Language Models: GPT-4 prefers longer answers more than humans
• Ye et al. (2024), Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge: twelve biases including authority, bandwagon, sentiment, fallacy oversight and refinement awareness
• Chen, Zaharia and Zou (2023), How is ChatGPT's behavior changing over time?: same endpoint, March versus June 2023, prime identification from 84% to 51%
• McHugh (2012), Interrater reliability: the kappa statistic, Biochemia Medica: report percent agreement and kappa together; interpretation scales
• Anthropic, Create strong empirical evaluations: LLM-based grading and the guidance to grade with a different model than the one under test
• OpenAI, Graders: model graders and detecting grader hacking by comparing with expert human evaluation