The instinct to measure everything feels like rigor. It produces noise. An eval suite that flags failures on dimensions that wouldn't change anything about the system isn't a safety net — it's maintenance overhead that competes with the signal you actually need. The goal is maximum reliability on the dimensions that determine whether the interaction succeeded, not maximum coverage of what the framework can expose.
The selection problem
Most eval frameworks offer a menu of metrics — tool fidelity, routing fidelity, semantic similarity, faithfulness, context relevance, and more. The frameworks don’t force you to run all of them. Teams run all of them because more measurement feels safer. It isn’t.
When you measure a dimension that doesn’t affect the outcome, a failure on that dimension creates work without a corresponding system improvement. An FAQ that routes to a domain agent instead of staying in the root agent, but fires the right tool, has succeeded. If routing fidelity fails in that case, nothing about the system should change — the routing is an implementation detail, not a success condition. Measuring it produces a signal you’ll learn to ignore, and signals you learn to ignore erode confidence in the ones that matter.
The same logic applies in the other direction. A guardrail response is static — the content is determined in advance and doesn’t vary with the model’s output. Semantic similarity is the entire signal. Running additional dimensions against a fixed response doesn’t add coverage; it adds noise with a passing grade, which is a different kind of problem.
The practical filter
One question clears the selection: if this dimension failed, would I change anything about the system?
If the answer is no — if a failure here reflects an implementation detail rather than a behavior the system is responsible for — the dimension doesn’t belong in the eval suite. This isn’t about lowering standards. It’s about making the standards reflect the actual contract. An interaction type has specific success conditions, and the eval should measure those conditions, not the full set of things that could theoretically be measured.
The conditional structure matters too. Some dimensions are only meaningful given the outcome of others. Tool fidelity is the relevant measure for an FAQ. Whether that tool was called from the root agent or a domain agent is an architecture question, not an outcome question. Measuring routing fidelity as though it were independent of tool fidelity conflates the two — and produces eval failures that reflect architecture choices you might deliberately change without affecting the outcome at all.
What targeted evals buy you
Two things follow from a targeted suite that most teams undervalue until they’ve run a comprehensive one in production.
The signal is more reliable. When a dimension fails, it’s a real failure on something that matters. The cost of that failure is the same as any other mistake in a shipped system — but you’re more likely to catch it and act on it when it isn’t buried in failures on dimensions you’ve learned to discount. Noise doesn’t just create maintenance burden; it trains the team to treat eval failures as expected background rather than actionable signal.
The evals survive architecture changes. A suite that measures routing fidelity breaks when you refactor the routing. A suite that measures tool fidelity doesn’t care how the routing changed — the contract it’s verifying stayed the same. Systems under active development change frequently, and evals coupled to implementation details require nearly as much maintenance as the agents themselves. That maintenance cost isn’t recoverable; it compounds over every refactor.
Takeaway
Start from what success looks like for each interaction type, then select the dimensions that measure that. Don’t start from the framework’s menu and filter down later — by then the suite is already shaped by what was easy to measure rather than what matters. The eval reflects your understanding of the system’s contract. If it’s measuring things that wouldn’t change anything when they fail, that contract is underspecified.