Evaluating agents before they touch production

TL;DR

• Do not give an agent real tools or real data because a demo looked finished.
• A pass is a task suite, a grader on the trace, a regression set, and a budget, pinned to one prompt, model, and tool revision.
• Widen access in stages: read-only tools, sandbox data, a human gate, then one new capability at a time.

Write the tasks before you watch

A task suite is the jobs the agent must do, written before you watch it. Each task has a start state, the tools it may use, and what done means. The happy path is one task. Also include missing information, an empty or wrong tool result, and a request the agent should refuse.

The regression set is the slice you rerun when the prompt, the model, or the tools change. A failure should point at that change. Add failures you refuse to see again, and drop a task that no longer describes the product in the same change.

Grade the trace, not the tone

Score the trace. Mechanical checks catch a missing file, a forbidden tool call, or a final state that misses the fixture. A rubric, read by a person or a second model, catches a fluent answer to the wrong question. A confident last message is not evidence.

Cap turns and steps

A turn budget caps model turns. A step budget caps tool calls. The cap belongs to the task, so a lookup and a ticket do not share one just because they share a model.

A run that uses up the budget is a fail, even if the last message sounds done. If a revision only passes by taking more turns than the cap allows, it did not pass. Write down why when you raise a cap.

Pin the result to a revision

A result is proof only when it names the prompt revision, the model revision, and the tool revision. A prompt edit, a model swap, a schema change, and a change in what a tool returns are different revisions. Last week's green run does not follow today's prompt.

If the suite, the grader, or the budget changes, that is a new revision of the evaluation. A higher score after a softer grader is not a better agent.

// A pass belongs to one prompt, model, and tool revision.
function assertPinned(run: EvalRun) {
  assert.equal(run.revision.prompt, "prompt@a1b2c3")
  assert.equal(run.revision.model, "model@9f3e1a")
  assert.equal(run.revision.tools, "tools@4")
  assert.deepEqual(run.trace, ["lookup", "draft"])
  assert.equal(run.trace.includes("send"), false)
  assert.ok(run.turns < run.budget.turns)
  assert.ok(run.steps < run.budget.steps)
}

This check fails on any other revision.

Put the failures in the suite

The suite should be able to draw these out before production.

• Looping. The same tool or plan, with no new fact. The budget stops the spend, and the grader still marks a loop.
• Tool misuse. A tool it was not given, arguments that miss the schema, or a write on a read-only task. That shows up in the trace, not in the final answer.
• Hallucinated actions. If the agent claims a send or an update, the trace has to contain that call.
• Silent partial completion. A required step skipped, then reported as success, is a fail.
• Prompt injection via tool output. Instructions in a page, a ticket, or a log must not override the task or the permissions. A capable model is not a control.

Stage access before real data

Good enough is a decision about access, not a feeling after a demo.

• Read-only tools first: look, search, and draft. No customer writes, no sends, no merges.
• Sandbox data next. Records are fake, or copied where a wrong write does no harm, with the same tool shapes production will have.
• Then a human gate on anything hard to undo, including tasks where the right move is to stop and ask.

Wider access is one capability at a time, after the regression set passes on the revision you intend to ship. A green read-only suite is not a yes to a write tool. Someone has to own the suite after launch, because the model and the prompt will change.

Where this meets a delivery

One agent for production is an AI production sprint: one system, a golden evaluation set, and a go or no-go on scale. Coding agents in a repo are an AI coding rollout: what they may touch, how a change is proved, and a gate before it lands. A live agent whose suite must keep running sits on Managed AI operations.

FAQ

What is a task suite? The jobs the agent must do, each with a start state, the tools it may use, and a definition of done.

Why pin a result to a revision? A score without the prompt, the model, and the tools that produced it cannot be compared with the next change.

When is an agent good enough for real tools? When the regression set passes on the revision you intend to ship, the budgets hold, and a person still approves anything hard to undo.