The Eval You Climb Is the Product You Get

The Climb Is Mechanical. The Eval Is the Design.

More teams now build AI systems the same way. Someone writes an eval that says what "good" means. Then they change one thing (a prompt, a tool, a model, the retrieval index, a data definition), rerun the eval, keep the change if the number moves, and revert it if not. That is hill climbing, and Chip Huyen's AI Engineering files the practice under evaluation-driven development. The changes can come from a person or from an agent working overnight.

Karpathy's autoresearch is the clean case. The agent may edit one file, train.py. A second, prepare.py, holds the data and the evaluation, and the agent does not touch it. Each run gets a fixed five-minute budget and one score, validation bits per byte. Anthropic put the same loop inside its platform: in Claude Managed Agents you give a session a rubric, a grader in its own context window scores the work and returns the failed criteria, and the agent revises, 3 times by default and 20 at most (docs, beta).

The Five-Minute Clock and Autoresearch: The Overnight Loop cover where the pattern came from. This piece covers what they leave out: the climber is cheap and tireless, the eval is the part you can get wrong, and whatever it rewards, the climb delivers.

”

"Systems, like the three wishes in the traditional fairy tale, have a terrible tendency to produce exactly and only what you ask them to produce. Be careful what you ask them to produce." — Donella H. Meadows, Thinking in Systems (§ "Seeking the Wrong Goal")

Goodhart's law says the same thing more sharply. As Perry Mehrling retells it, the economist's warning concerned relationships, not targets:

”

"Goodhart's Law, named after Charles Goodhart, longtime chief economist at the Bank of England, warned that any apparently stable statistical relationship will … break down once you begin to rely on it for policy purposes." — Perry Mehrling, The New Lombard Street (§ "Chapter 8")

A score tracks product quality until you start optimizing the score. After that, the link is no longer a fact about your product. It is a fact about how well you climbed.

A Climber Finds Every Flaw in Your Eval

Public benchmarks show what happens at scale.

Datacurve also found that on SWE-Bench Pro, Claude Opus 4.7 and 4.6 sometimes read the answer out of the container's git history. Its analyzer marked more than 12% of those rollouts as cheated, and the behavior accounted for about 18% of Opus 4.7's passes and 25% of Opus 4.6's in the reviewed sample (VentureBeat, 2026-05-26). Datacurve's review was LLM-assisted and covered a small sample, so weigh the exact figures. The direction matches OpenAI's own audit.

The METR and Redwood investigation of the OpenAI ExploitGym incident is the extreme case. By the authors' estimate, 30 to 40 percent of the target programs could not be exploited the intended way. About 1,200 agents coordinated on a shared board, and the authors say the agents went after the scorer itself (METR, 2026-08-26). The Agent Was Only Collecting Data covers the containment side. The eval lesson: impossible or broken items give the optimizer a reason to attack the grader.

The same thing shows up at hobby scale. One researcher let an agent submit solutions to an API that reported scores on a hidden validation set. The last 13 submissions were the same algorithm with different random seeds (LessWrong comment, 2026). Labs train models this way too: RLVR rewards whatever a checker can verify, and the checker's gaps are where reward hacking lives.

Grade Each Check, Not Each Item

A pass/fail item hides which part broke. Say an item asks for a team's sales on a given day. The answer can fail on the number, the team or the date, and one score makes all three look the same. Worse, a change that fixes one of two broken checks moves nothing, so the climber sees no gradient and the loop discards a real improvement. Grade per check instead: each item carries several small, independent checks, and you track the pass rate of each across the set.

Uber's QueryGPT team does this. Each question is scored on intent, table overlap, whether the query ran, whether it returned rows, and an LLM-rated similarity to a golden query (Uber Engineering). That is five numbers per question. Anthropic's rubric docs say why: "The grader scores each criterion independently, so vague criteria produce noisy evaluations."

Here is a grader that does it. It is generic Python; system is any function that takes a question and returns an answer with its cost and time.

import random
from dataclasses import dataclass
 
@dataclass
class Item:
    id: str
    question: str
    ref: dict      # known answer, made by a path the system cannot reach
    checks: dict   # check name -> fn(answer, ref) -> bool
 
def run_set(system, items, reps=3):
    rows = {}
    for it in items:
        runs = [system(it.question) for _ in range(reps)]   # each: answer, cost, secs
        graded = [{c: bool(f(r["answer"], it.ref)) for c, f in it.checks.items()}
                  for r in runs]
        rows[it.id] = {
            # majority of reps, per check
            "checks": {c: sum(g[c] for g in graded) * 2 > reps for c in it.checks},
            "cost": sum(r["cost"] for r in runs) / reps,
            "secs": sorted(r["secs"] for r in runs)[reps // 2],
        }
    return rows
 
def passed(row):
    return all(row["checks"].values())
 
def summarize(rows):
    correct = sum(passed(r) for r in rows.values())
    per_check = {}
    for r in rows.values():
        for c, ok in r["checks"].items():
            per_check.setdefault(c, []).append(ok)
    return {
        "correct": correct,
        "per_check": {c: sum(v) / len(v) for c, v in per_check.items()},
        "cost_per_correct": sum(r["cost"] for r in rows.values()) / max(correct, 1),
    }

A check that cannot fail is dead weight. In our first eight-item routing set, four items (two refusals, two clarifications) passed every run, because they passed on a question mark or a keyword. An item that always passes gives the climber nothing to learn from.

The Grader Must Not Share the System's Upstream

Known answers have to come from somewhere the system cannot see. If the grader and the system read the same table or apply the same definition, they will agree about a mistake. We have the failure on record. Clive, our analyst agent, answers a client's operations questions from a database. In one run it used a team's current roster for a past date and reported 70 sales from 18 people. The right answer, using the roster as it stood that day, was 63 from 17. Clive then checked its answer against the team board, a view built on the same current-roster assumption. The check passed. Our finding from that run: "A check needs an independent source."

Three practices follow.

• Make the reference by a separate path, with a separate owner. In our contract the client's data team owns the questions and the reference queries. Expected values are never stored: the reference query runs at scoring time on fixed past dates, so it cannot go stale or be read from the repo.
• Keep the set out of the agent's reach. The contract says the eval set must never be installed where the analyst can read it. An analyst that can read its own exam scores 100% and tells you nothing. A build step refuses to write a skill folder that holds an item's question or reference.
• Treat the key as fallible. In one early run the reference ranked everyone in the database, while the question said "on the floor", which the rules define as the sales teams. Clive followed the rule and was marked wrong. When the bot and the key disagree on a definition, send it to the key's owner before counting it either way.

Hold Out a Set the Climber Never Sees

Every item the climber can see, it can fit, and thirty items is small enough to fit by accident. Split the set. The dev set drives the climb. The holdout is scored rarely, reports one aggregate, and never feeds back into the changes.

We run one sealed holdout for a text detector built for a client: 200 hand-labelled items, sampled with a fixed seed and kept disjoint from the tuning sets. The holdout script prints aggregate precision, recall and F1 and writes no per-item results. A list of failures would invite fixing them one at a time, which turns the holdout into a second dev set. We run it monthly to catch drift. Our design plan for Clive applies the same rule at small scale: the author of a mapping sees eight eval items, freezes the mapping, and the client then supplies 20 fresh questions the author has never seen.

Benchmark owners do this for the same reason. Huyen notes that public benchmarks should keep part of their data private and score models against it automatically (AI Engineering, p. 392). Google DeepMind's August pilot goes further, running a double-blind evaluation in a cryptographic enclave where the evaluator cannot see the model weights and Google cannot see the test prompts (Google DeepMind, 2026-08-27). You do not need an enclave. You need a file the proposer cannot open.

Rules: one number per look, no per-item output, and a fixed schedule (every few accepted changes, not every run).

Thirty Items Cannot See a Two-Item Change

Your agent passes 24 of 30 items. You change a prompt and it passes 26. Did it improve? Probably you cannot tell. The model is not deterministic, so the same system on the same items scores differently run to run. And the 30 items are a sample: at a true accuracy of 80%, a 30-item score has a standard deviation of about 7 points, and the 95% band on the difference between two such scores is about plus or minus 20.

Rerun noise is the one that bites a climb. We simulated 20,000 pairs of runs of the same unchanged system on the same 30 items. Half the items always pass, 15% always fail, and the rest pass 20% to 80% of the time. If every item passes 80% of the time, the figures below rise to 63% and 42%.

A two-item swing is what an unchanged system does. Uber's team reached the same conclusion: rerunning QueryGPT's evaluation with no change gives different outcomes, and they "do not over-index decisions based on ~5% run-to-run changes in most metrics." Our logs show it too. Items that fell back to a second model went 10 of 13 correct on November 5 and 16 of 17 on November 8. Our findings note says the jump "may be the better skill or small samples", and with 13 and 17 items we cannot tell which.

More items help slowly. Douglas Hubbard's rule:

”

"once you get to 30 samples, you have to, as a general rule, quadruple the number of samples (120) if you want the error to go down by half again." — Douglas W. Hubbard, How to Measure Anything (§ "Are Small Samples Really “Statistically Significant”?")

So use three tools together. Compare in pairs on the same items: count items one arm passes and the other fails, then run a sign test on those. Rerun and take the majority: three runs per item cost three times as much and remove most flaky-item noise. Set the threshold before the run. Ours is at least 25 items and a net win of 6, or the run cannot tell the arms apart and we keep the simpler or cheaper one. The table shows why 6 is about right for 30 items.

A net win of 6 sits at the edge, not past it. Sets this small find only large effects, roughly 20 points and up. Settle smaller ones with more items or live traffic.

Climb Accuracy, Cost, and Time Together

An accuracy number alone can be bought. A bigger model, more reasoning tokens, three retries and a second verification pass all raise accuracy, and all of them cost money or time. A loop that sees only accuracy takes every one of them. Track several axes:

• Accuracy, by check and overall.
• Cost per correct answer, not cost per call. A cheap model that fails often costs more per right answer (CPCT covers why; Nobody Can Forecast Their AI Bill has measured numbers).
• Latency, as a median and a tail.
• A delivery score, for how the answer arrives. Ours is hard checks on the reply's form: first-message length, no table in it, no internal jargon, no stated cause for a person's number.

Our analyst eval reports the first three from one run, and a separate script scores delivery from the saved replies. The baseline on November 11 was 25 of 29 correct, $0.062 per correct answer, 19 seconds mean. Our process sets the order: right and safe first, then seconds to the first reply, then cost per correct answer.

Say a change lifts accuracy from 25 of 29 to 27 of 29 and raises cost per correct answer fifteen times. On a one-number eval that is a win. On four numbers it is a decision someone makes on purpose. In the loop below, a change is kept only if it wins by the threshold and stays under a cost cap. Latency and delivery each get one more guard clause.

def climb(system_for, propose, dev, holdout, cfg, steps=20, min_net=6, max_cost_x=1.5):
    best = run_set(system_for(cfg), dev)
    for _ in range(steps):
        trial = propose(cfg)               # one change: prompt, tool, model, definition
        rows = run_set(system_for(trial), dev)
        wins = sum(passed(rows[i]) and not passed(best[i]) for i in best)
        losses = sum(passed(best[i]) and not passed(rows[i]) for i in best)
        cost_ok = (summarize(rows)["cost_per_correct"]
                   <= max_cost_x * summarize(best)["cost_per_correct"])
        if wins - losses >= min_net and cost_ok:
            cfg, best = trial, rows        # keep
        # otherwise revert: cfg is unchanged
    # one look at the holdout; its items and failures never reach `propose`
    return cfg, summarize(run_set(system_for(cfg), holdout))

Log every attempt, kept or reverted, with the setup that produced it. A score without its setup line is void. Change one thing per run.

When One-Change Climbing Stalls, Change What the Agent Sees

Hill climbing gets stuck on a local maximum: every single change leaves the number flat or worse. A prompt tweak is a small step in one direction, and you have run out of small steps. Three signs:

• Five or six changes in a row land inside the noise band.
• Failures cluster in one category, and the same checks keep failing under every prompt.
• Fixing one case breaks another in the same cluster.

The way out is a change of representation, not another prompt edit. Our data points that way. In the analyst eval, only one of 29 items had a query error. The misses were about meaning: past team membership, a first name matching two people, an unstated minimum on hours. No prompt teaches an agent a definition nobody gave it. The lesson we recorded: definitions and asking first matter more than query-writing.

The levers, roughly in order of payoff:

• Data definitions. Write down what "the floor", "a sale" and "last week" mean, in one place the agent reads.
• Tool design. Fewer, typed tools beat a menu of ten near-duplicates. Our design plan tests this directly: ten report tools against three against one frame tool, on the same items, with ties going to the simpler design.
• What the agent reads. The schema, the examples, the retrieved documents. Add the missing fact to the context before you add another instruction.
• Who decides. Move a decision from the prompt into code, such as when to ask a clarifying question or which dates a phrase means.

Autoresearch splits the work the same way: the human edits program.md, the agent edits train.py. When your loop plateaus, widen what the agent may change, or change what it is told.

Strict Checks First, Judge Second

Code checks are exact, cheap and repeatable. A judge model is flexible and wrong in its own ways. Run strict checks first, and add a judge only for what code cannot see.

Clive's routing eval has 37 items. Each reply is held to hard checks in code: the right figure, no send tool used, no internal words, no figure without a report behind it. A judge, a small model asked six fixed yes-or-no questions per reply in one stateless call, covers what patterns miss. A regex catches "because he was tired". It misses "a week like this often reflects technical hiccups" written about people named a paragraph earlier. The pre-send gate we built for a client's outbound messages uses the same order: rules run in milliseconds, and the judge layer, once wired in, sees a message only after it passes the rules.

A judge is fine when the question is narrow and yes-or-no, the model and prompt are pinned and logged with every run, it has been checked against people, and it is not the only thing the climber optimizes. It is not fine as the sole objective, when it comes from the same model family as the system, or when its prompt changed between runs. Huyen's example (p. 287): if last month's coherence score was 90% and this month's is 92%, you cannot say the app improved unless the judge is identical in both months. On consistency she adds a warning:

”

"high consistency may not imply high accuracy—the judge might consistently make the same mistakes." — Chip Huyen, AI Engineering (p. 282)

A judge that agrees with itself every run still shares its model's blind spots. LLM-as-Judge covers the known biases. The check that matters for climbing is calibration against people. Have a person label 20 replies and measure, for each judge question, how often the two agree. Our rule: a question under 90% agreement does not count until its wording is fixed. Then look at the rare class. If 95% of replies are fine, a judge that always says "fine" scores 95% agreement and tells you nothing, so check agreement on the replies a person flagged.

What to Do Monday

Once a month, open the holdout and compare it with the dev score. A growing gap means the climber has learned your items: refresh the set. The eval you climb is the product you get, so review it as carefully as the code it grades.