Your LLM judge is wrong on the median session, not the tail

Everyone hardens their eval harness for the scary tail. The jailbreak, the prompt injection, the one-in-a-hundred edge case. Meanwhile the flattened LLM-as-a-judge is quietly wrong on the median session: the boring support conversation that ran a hundred tool calls, looked fine, and wasn't.

Not that judges fail on hard inputs (everyone knows that), but that the way most teams run a judge makes it wrong on the typical input. The fix is not a better prompt. It is a different shape of evaluation.

What production agent traffic actually looks like

Open your traces. If you run agents in production, the median session is not a two or three turn chat. It is a long tool-using trajectory: the agent calls a search, reads a record, calls another tool, retries after a token expires, loops back, and does it again forty times before it answers.

This is not the tail. This is the median. One of our design partners is an insurance voice-agent team running production agentic systems, and their real pain is not the exotic jailbreak. It is context rot: sessions that spiral into a hundred-plus tool calls, token-expiry retries, and loops that no single response ever reveals. The scary edge case gets attention because it is scary. The long boring session gets a green check because nobody looks past the final answer.

That gap is the whole problem. The place your judge is most confident is the place it has the least evidence.

Why flattening fails on the median

Here is how most "LLM-as-a-judge" setups work under the hood. Take the session, serialize it into one big block of text, paste it into a prompt, ask a model "was this good?" The session gets truncated to fit the context window. The tool calls become flat strings the model cannot execute or verify. Cross-turn state (what the agent knew at turn 5 versus turn 40) collapses into a wall of text with no way to navigate it.

A judge in that position cannot verify a stateful tool action. It cannot check whether the agent looped. It cannot tell whether turn 30 contradicted turn 8. It can only pattern-match on how the final response reads. On a two-turn chat that is often enough. On a hundred-turn trajectory it is guessing.

External work points the same direction. Be precise about what it does and does not say.

A published case study of a production multi-turn ordering agent found that the LLM-as-a-judge surfaced roughly 18 to 22 percent of human-confirmed defects, missing almost all of the cross-turn state failures (Zhang et al., arXiv:2606.10315). That is one agent in one study, not a general law, so do not read it as "judges catch one in five defects everywhere." Read it as a concrete existence proof: on a real multi-turn agent, the flattened judge missed exactly the class of failure that only shows up when you can see across turns.

There is also a proposed taxonomy of agent-judge biases (BabelJudge, arXiv:2606.22329, a single-author preprint) that names three failure modes worth having language for: trajectory-length bias, hallucination blindness, and argument blindness. Treat it as a proposed taxonomy rather than settled fact, but the three names map cleanly onto what you see when a judge has to swallow a long trajectory whole.

The objection you are already forming: "not another dashboard, not another eval tool. Process beats tools." Fair. This is not a metrics dashboard, and it is not a new chart to stare at. It is automated error analysis at scale: it surfaces the first upstream failure across every trajectory, the way a careful engineer would if they had time to read every session. It does not remove human judgment. It points it at the right place instead of at a haystack. The problem we are describing is not "you lack a tool." It is "the judge you already run is blind to the median session." That is a capability gap, not a UI gap.

The tell: our own eval miner outran our judge

We did not arrive at this from theory. We arrived at it because a system we already run started producing evals a response-only judge cannot score, even in principle.

We run an eval-mining engine that reads production traces and proposes new eval rubrics: failure modes it sees recurring that no one wrote a rule for. This is also how we fight eval staleness, the slow decay where a hand-written eval set stops matching production the week behavior shifts. Some of what it proposes are trajectory-level rubrics. One rubric type it emits is "repeated identical tool call loop detection." Think about what it would take to score that rubric. You have to look across the whole sequence of tool calls and decide whether the agent got stuck. A judge that only sees the final response has no way to answer it. There is nothing in the response to look at. The evidence lives in the trajectory the judge never gets to see.

A clean, live example from the mining engine: on a claims-assistant agent, it surfaced a failure-mode cluster we internally call "renewal, no actionable steps," clustered from 39 real production sessions around one failure the rule-based strategies had missed entirely. It is a genuine, recurring quality problem, and scoring it well means reasoning about what the agent did across the session, not grading the last message.

This was the aha. The evaluator we already run implies a judge we did not yet have. Our miner had outrun our judge. So we stopped flattening.

(For the "so you built eval mining, cool" crowd: mining traces into evals and auto-proposing fixes are increasingly table stakes, and the incumbents ship versions of both. Useful, and we run mining too. But it is support here, not the headline. The headline is the judge.)

Evaluation as an investigation, not a verdict

The shift is this. Stop treating evaluation as a single verdict emitted from a single flattened prompt. Treat it as an investigation of the trajectory: search for evidence, retrieve precedent, verify claims, then rule.

We credit Judgment Labs here explicitly. Their "Agent Judge" framing (queryable trajectories, adaptive rubrics, environment verification) is prior art we build on, not something we claim. The core idea that an agent-as-a-judge should query the trajectory rather than swallow it whole is theirs, and it is right. Our divergence comes later.

Here is the shape of what we built.

First, a trajectory digest. This is pure code, zero LLM tokens. It computes the shape of the session up front: tool statistics, loop signals, a segmented timeline, a turn index. It is the map. It costs nothing and it exists before any model is called.

Second, a coordinator on the reasoning tier. It reads the digest and plans the investigation: what to look at, what to pull, what to check. It does not grade blindly. It decides where the evidence is, then goes and gets it, and it emits all the rubric verdicts at the end.

Third, read-only tools the coordinator investigates with: get_events, search_in_session, similar_labeled_cases (retrieval against human-labeled precedent), get_tool_stats. A verify_action tool is reserved for the verification leg. These are read-only by design. The judge investigates; it does not act.

Fourth, readers on a cheap tier (Haiku-class or a small model). They compress evidence for the coordinator. They never judge. Their only job is to turn raw events into a summary the reasoning tier can act on, cheaply.

The economics matter, because "run a whole investigation per session" sounds expensive. One investigation amortizes across all rubrics. The cost of scoring N rubrics is roughly one investigation plus N cheap emissions, not N full runs. And cost control lives inside the judge (routing, amortization, prompt caching, budgets, distillation later), not as a cheap flattened gate bolted in front of it.

Verdicts land in eval_results tagged judge_model='agent-judge'. Everything downstream (KPIs, mining, the UI) is unchanged, which means the new judge can run in shadow next to the old one and be compared on the same label set before it takes over anything.

What's built, and where the honest gap is

This is shipped, not a slide. Agent judge v1 landed and runs in shadow mode right now, writing agent-judge verdicts alongside the existing eval-runner. When it wins on the label set, it flips to primary. Trajectory rubrics (the ones a response-only judge cannot score at all) route to it immediately, because there is no honest alternative.

The full system has three legs, and I want to be straight about which are done:

• Search is live. A ClickHouse event graph plus pgvector for retrieval. The judge can query the trajectory and pull similar labeled cases.
• Adaptation is live. The mining engine (disagreement, anomaly, and clustering strategies) plus a trust layer where mined candidates never become live config until a human approves them. Nothing auto-promotes.
• Verification is the phase-behind gap. This is the leg that would let the judge re-execute or confirm a tool action against the customer's own environment via connectors. It is designed, the verify_action tool slot is reserved, and it is not built yet. If you need the judge to independently verify that a stateful action really happened, we are not there. I would rather tell you that now than have you find it later.

Even with two of three legs, the shipped judge already scores the trajectory rubrics a flattened judge cannot touch at all. That is the part that is real today.

Why investigation beats per-step scoring

Almost everyone now says "trajectory evaluation." Look under the hood and much of it is per-step rubric scoring: chop the trajectory into steps, score each step, aggregate. That is better than flattening the whole thing into one prompt. It is still scoring, not investigating. It cannot decide, mid-evaluation, that the interesting thing is a contradiction between turn 8 and turn 30 and go pull both.

The difference is a judge that plans, queries, retrieves precedent, and then rules. That is the durable difference, and it is the one thing that does not commoditize the week after you ship it.

And to name our divergence from the prior art plainly: Judgment Labs' loop refines the rubric. Ours refines the agent and specializes the judge through retrieval against the customer's own labeled cases, so the judge gets sharper on your domain the more you label. Same starting insight, different thing we do with it.

Where this goes

The agent judge is the product. Not a dashboard around a judge, not a faster gate in front of one. A judge that investigates the trajectory the way you would if you had time to read every session, and that gets more specialized to your domain the longer it runs.

This is post one of our launch series. The next posts go deeper on the pieces named here: the specialization flywheel, the self-healing loop, and the cost ladder inside the judge.

If you run agents in production and you have ever shipped a green check on a session nobody actually read, we should talk. Follow the series, or reach out if you want to point this at your own traces.