LLM as a judge, carefully

Using language models to grade and evaluate other language model outputs sounds elegant in theory but breaks quietly in practice. The automated judge develops implicit biases toward output it prefers: answers that are verbose, deferential in tone, and express high confidence regardless of actual accuracy.

Without proper calibration you end up tuning your entire system against those judge biases rather than actual ground truth quality. The result is longer answers that sound safer and more polished but are often subtly incorrect in ways that matter downstream in production.

The only reliable solution is the traditional labour-intensive approach: label many examples by hand with ground truth labels from human experts. We labeled one hundred fifty representative cases, measured how often the judge agreed with human reviewers, and tuned the rubric until agreement reached eighty-five percent.

At eighty-five percent agreement with human judgment you finally have a tool you can confidently run at scale. Below that threshold you are fundamentally optimizing against noise. The framework and wording matter far less than achieving that calibration baseline.

After achieving calibration, the automated judge sustains the pace of iteration and enables experiments on production data without needing human review on every single output, which accelerates the development cycle significantly.

Before achieving that calibration, you possess only false rigour and you make product decisions based on noise rather than signal.