Evals and the Trust Threshold: Measuring an Agent You Can Defend

Every serious agent project reaches the same moment. The thing works in the demo, someone on the business side asks to let it act on its own, and the room goes quiet, because nobody can say how they would know it is safe to. The instinct is to answer with a story: it did well in testing, the team is confident, the demo was clean. In a regulated context, none of that survives contact with a risk committee.

However, there is a mechanism that answers the question properly, and it is the same mechanism that separates a governed agent from a hopeful one. Evals are how an agent earns autonomy, and the trust threshold is the line an eval has to clear before the agent is allowed to act. Not a story about confidence. A measurement, repeated, against a standard set in advance. An agent does not start trusted. It becomes trusted by passing, and it stays trusted only as long as it keeps passing.

The reframe worth making early is that an eval is not a test you run to feel good before launch. It is a standing instrument that gates how much the agent is allowed to do, and it is wired directly to the autonomy the agent is granted.

"We tested it once" is not an eval

Start by killing the most common substitute, because most failed agent governance dies here.

A single pass over a fixed set of examples before go-live is a test. It tells you the agent worked on those inputs, on that day, against that version of the model. An eval is something stronger on three counts, and the difference is not pedantic.

An eval is repeated, not one-time, because the system it measures changes underneath you. The model can be updated overnight, the data distribution shifts, and an agent that passed last month can degrade this month without anyone shipping a line of code. A one-time test cannot see that; a standing eval that reruns on a schedule and on every model change can. An eval is held to a threshold set in advance, not judged after the fact, because a bar you draw after seeing the results is not a bar. And an eval is representative of production, not of the happy path, because an agent graded only on the cases it handles well is being flattered, not measured.

"We tested it once" fails all three. It is a snapshot of a moving system, judged by a standard that did not exist before the results came in. It is the single most common reason an agent that demoed beautifully cannot be defended six weeks later.

What to actually measure

A defensible eval is a small battery of measurements, not one accuracy number, because "accuracy" hides exactly the failures that matter in a regulated process. Five things carry most of the weight.

Concordance with human review. On a representative sample, how often does the agent reach the same decision a qualified human reviewer would? This is the headline number, and it is meaningful only against a human baseline, so you need decisions a human already made to compare against. Concordance is the metric a regulator understands intuitively, because it answers "is it at least as reliable as the person it replaces?"

Performance on hard-case slices, not just the average. An average accuracy of 95% can hide a 60% accuracy on the 10% of cases that carry most of the risk. Slice the eval by difficulty, by edge case, by the segments where a mistake is expensive, and hold each slice to its own threshold. The average is where dangerous agents hide.

Calibration. When the agent says it is confident, is it actually right that often? A well-calibrated agent that reports low confidence on a case it is likely to get wrong is far safer than an overconfident one with the same raw accuracy, because calibration is what makes automated escalation trustworthy. An agent that does not know what it does not know cannot be allowed to decide when to ask for help.

Escalation quality. When the agent hands a case to a human, was it the right case to hand over? Measure both directions: cases it escalated that it could have handled, and worse, cases it resolved on its own that it should have escalated. The second kind is where quiet harm accumulates.

Drift. Run the eval over time and watch the trend, not just the point. A stable-looking agent that is losing a point of concordance each week is failing slowly, and only a repeated eval catches it before a customer does.

Thresholds gate the autonomy ladder

Here is where evals stop being measurement and become governance. Autonomy is not a switch. It is a ladder, and the eval thresholds are the rungs.

An agent starts in shadow: it observes and proposes, but a human makes and ships every decision. Its outputs are scored against what the human actually did, and that comparison is the baseline eval. Only when its concordance and its hard-slice performance clear a defined threshold does it earn the next rung, supervised, where it acts but a human approves each step. Clear a higher bar, with calibration and escalation quality now in scope, and it moves to guided, acting autonomously while humans monitor and handle the exceptions it flags. Full autonomy, within a bounded and documented blast radius, is the top of the ladder and the hardest bar, and even there the consequential decisions still resolve to a human by policy.

The discipline is that the threshold is set before the agent is measured against it, and movement up the ladder is earned, logged, and reversible. If drift pushes the eval back below the bar, the agent moves back down a rung, automatically, and that demotion is recorded. Autonomy that can only be granted and never revoked is not governance. The kill-switch and the autonomy matrix are documented and tested, so the answer to "how did this agent get permission to act?" is always a measurement and a logged decision, never a vibe.

This is what lets a committee sign off. They are not being asked to trust the agent. They are being asked to approve a rule: this agent may act at this level only while it clears this threshold on these metrics, and it drops back the moment it does not. That is a control, not a promise.

Adversarial evals: measure what an attacker can make it do

One more class of eval belongs in any honest program, and it measures a different thing than accuracy: what a hostile input can make the agent do.

An agent reads content it did not author, and some of that content is trying to manipulate it. Standard accuracy evals will not catch this, because they measure whether the agent gets benign cases right, not whether a poisoned document can hijack it. Adversarial evaluation, of the kind the research community has been building with benchmarks like AgentDojo, probes exactly that: does a crafted input cause the agent to take an action it should not, exfiltrate data, or ignore its constraints? Public benchmarks like AgentDojo are a reference point for how to construct these probes, not a score to cite as your own result. Your adversarial eval has to run against your agent, your tools, and your data, and it earns its keep by finding the attack that works before an attacker does.

So the question that should gate every autonomy decision you make is not "did it pass the demo?" It is what does this agent have to keep proving, on which metrics, against which threshold, for it to hold the autonomy we have given it, and does it drop back the moment it stops proving it? If you cannot answer that, the agent does not have earned autonomy. It has borrowed trust, and borrowed trust is what fills the pilot graveyard.

If you want to define the trust threshold for a real process and the eval battery that gates it, book a meeting and bring us the decision you would least want an agent to get wrong.