Decision Memo

Would You Change Anything If It Failed?

The instinct to measure everything feels like rigor. It produces noise. An eval suite that flags failures on dimensions that wouldn't change anything about the system isn't a safety net. It's maintenance overhead that competes with the signal you actually need. The goal is maximum reliability on the dimensions that determine whether the interaction succeeded, not maximum coverage of what the eval framework can expose.

The selection problem

Most eval frameworks offer a menu of metrics: tool fidelity (the right tool called with the right arguments), routing fidelity (the request handled by the expected agent), semantic similarity, faithfulness, context relevance, and more. The frameworks don’t force you to run all of them. Teams run all of them because more measurement feels safer. It isn’t.

When you measure a dimension that doesn’t affect the outcome, a failure on that dimension creates work without a corresponding system improvement. An FAQ that routes to a domain agent instead of staying in the root agent, but fires the right tool, has succeeded. If routing fidelity fails in that case, nothing about the system should change, because the routing is an implementation detail rather than a success condition. Measuring it produces a signal you’ll learn to ignore, and signals you learn to ignore erode confidence in the ones that matter.

One caveat: routing is only an implementation detail if it doesn’t change anything you care about. If the domain agent is slower, costs more, or has different permissions, then the route does affect the outcome, and it deserves a dimension of its own under the name of what it affects.

The same logic applies in the other direction. Take a guardrail. Its contract has two parts: it fires on the inputs it should, and when it fires it says the right thing. The response is static, so once the guardrail has fired, similarity to the expected text is close to automatic. The failure that matters is the guardrail not firing at all. Running tool fidelity or faithfulness against a fixed response doesn’t add coverage. It adds checks that trivially pass, which gives the suite false confidence.

The practical filter

One question clears the selection: if this dimension failed, would I change anything about the system?

If the answer is no, because a failure here reflects an implementation detail rather than a behavior the system is responsible for, the dimension doesn’t belong in the gate. This isn’t about lowering standards. It’s about making the standards reflect the actual contract. An interaction type has specific success conditions, and the gate should measure those conditions, not the full set of things that could theoretically be measured.

Not everything that fails the filter has to disappear. A dimension can be a diagnostic: you compute it when an outcome fails, to find out why, but it never blocks a release or raises an alert. Routing fidelity is a good example. It’s no use as a gate and it’s useful when you’re debugging. The mistake is letting a diagnostic act as a gate.

Which dimensions matter depends on the interaction type, and some only mean anything given the outcome of others. Routing, for example, matters only in light of whether the right tool fired. Try it: pick an interaction type, then switch between running everything and a suite targeted to its contract. Each dimension can be a gate, a diagnostic, or off, and you can override any of them. Then simulate a routing refactor to see what each suite does with it.

Would you change anything if it failed?

Contract: Fires the right tool. Which agent handled it is an implementation detail, unless it changes latency, cost or permissions.

Tool fidelityRight tool, right arguments. Gate: a failure changes the system.
Routing fidelityHandled by the expected agent. Useful for debugging, but a failure here wouldn’t change the system.
Semantic similarityResponse means the right thing. Noise: a failure changes nothing.
FaithfulnessAnswer is supported by sources. Noise: a failure changes nothing.
Fired when it shouldTriggered on the right inputs only. Noise: a failure changes nothing.
1 gating the contract · 4 noise

Routing fidelity, Semantic similarity, Faithfulness, Fired when it should wouldn’t change anything if it failed. Cut it, or demote it to a diagnostic.

FAQ and guardrail are the memo’s cases. The other three are examples added to show different contracts.

What targeted evals buy you

Three things follow from a targeted suite that most teams undervalue until they’ve run a comprehensive one in production.

The signal is more reliable. When a dimension fails, it’s a real failure on something that matters. The cost of that failure is the same as any other mistake in a shipped system, but you’re more likely to catch it and act on it when it isn’t buried in failures on dimensions you’ve learned to discount. Noise doesn’t just create maintenance burden. It trains the team to treat eval failures as expected background rather than actionable signal.

The evals survive architecture changes. A suite that gates on routing fidelity breaks when you refactor the routing. A suite that gates on tool fidelity doesn’t care how the routing changed, because the contract it’s verifying stayed the same. Systems under active development change frequently, and evals coupled to implementation details require nearly as much maintenance as the agents themselves. That maintenance cost compounds over every refactor.

The suite costs less to run. Many dimensions are scored by a model, so every extra dimension is an extra model call on every test case, on every run. A noise dimension is a recurring bill as well as a recurring distraction.

Takeaway

Start from what success looks like for each interaction type, then select the dimensions that measure that. Don’t start from the framework’s menu and filter down later. By then the suite is already shaped by what was easy to measure rather than what matters. The eval reflects your understanding of the system’s contract. If it’s measuring things that wouldn’t change anything when they fail, the contract is underspecified.