Evals That Actually Work
Johannes Hayer
·1 min read·en
An eval is just a function: turn the model's output into a number you can average and trend.
Keep goldens small and mean#
ts
type Case = { input: string; expected: string; rubric?: (out: string) => Score };Ten sharp cases beat a thousand blurry ones. If a case never flips a run red or green, it's dead weight.
Run them on every commit#
Fast, deterministic, and checked into the repo. Flaky evals train you to ignore failures.
Weiterführend in der AI Engineering for Builders-Serie: dieser Post ist der tl;dr — die tiefe Version mit Eval-Pipeline, AI-as-Judge und Modell-Auswahl steht in Part 3 — Evaluation First. Starte am Anfang mit Part 1 — The AI Engineering Mental Model.
AI engineering, weekly.
Join developers getting practical AI engineering in their inbox.