Skip to content
Blentera

Resources, AI strategy

Measuring AI agent performance: metrics, evals and reporting

An agent that runs is not the same as an agent that works. This guide sets out the metrics that show whether AI agents deliver, how to build an evaluation set, how to compare models per task and how to report the results to leadership.

Updated 20 November 2026. 9 minute read.

In short

  • Measure outcomes, not activity. The core metrics are task success rate, quality review scores, time saved, cost per completed task, escalation and approval rates, and error and rework rates.
  • Divide costs by tasks completed successfully, not by tasks attempted, and include the employee time spent on review and rework.
  • Build an evaluation set from real tasks, including past failures. Anthropic suggests 20 to 50 tasks is a good start. Grade the outcome, run each task more than once and read the transcripts.
  • Compare models per task on the same evaluation set, weighing quality, cost and speed together. Re-run the set whenever the model, instructions or tools change.
  • Report to leadership monthly on one page: value delivered, cost against budget, quality and risk, and the decisions needed.

What to measure, and why it is different

Agents are harder to measure than ordinary software. OpenAI's evaluation guidance notes that traditional software tests are not enough, because a model can give different outputs for the same input. Agents add more variation: they choose their own steps, call tools and work over many turns, so one early error can change the whole result.

The NIST AI Risk Management Framework gives a useful structure. Its Measure function covers analysing, assessing, benchmarking and monitoring AI systems with quantitative and qualitative methods, before deployment and while in use. For agents, that means two activities: testing on a fixed evaluation set before each change, and monitoring real work in production.

Start from the business outcome. Write down what done looks like for each task before you measure anything. A metric without a definition of done measures activity, not value.

Six core metrics

1. Task success rate

The share of tasks the agent completes to your definition of done without an employee having to fix the result. Count tasks the agent abandoned or handed back as not successful. For tasks that must work every time, also test consistency: Anthropic distinguishes pass@k, the chance of at least one success in k attempts, from pass^k, the chance that all k attempts succeed. Pass^k is the better guide for work customers rely on.

2. Quality review score

A score from a reviewer against a short rubric, such as accuracy, completeness, tone and policy compliance. Review a fixed random sample each week, not only the cases that went wrong, so the score reflects normal work.

3. Time saved

Baseline minutes per task before the agent, minus the employee minutes per task with the agent, multiplied by tasks completed. Employee minutes must include review, correction and handling escalations. Measure the baseline before launch by timing a sample of tasks, so you compare against a real figure.

4. Cost per completed task

Total cost divided by tasks completed successfully. Total cost includes AI usage, platform and tool fees, and employee time for review and rework. Dividing by successful tasks, not attempts, stops a cheap but unreliable agent from looking efficient.

5. Escalation and approval rates

The escalation rate is the share of tasks the agent hands to an employee. Track approvals in three parts: approved unchanged, changed before approval and rejected. A high change rate shows where the agent needs better instructions or tools. A change rate near zero over many weeks can mean the agent is ready for more autonomy, or that approvers have stopped checking. Look at a sample to tell which.

6. Error and rework rates

The error rate is the share of completed tasks later found to be wrong, by a reviewer, a customer or a downstream process. The rework rate is the share of outputs an employee had to correct. Classify errors by severity, so one serious error is not hidden by many trivial ones.

Hypothetical example, with invented figures: a support agent completes 400 of 500 tickets in a month. AI usage costs €180, and review and rework take 20 hours of employee time costed at €40 an hour, so €800. Total cost is €980, and cost per completed ticket is €2.45. If the 100 unfinished tickets were counted too, the figure would look like €1.96 and flatter the agent.

Building an evaluation set

An evaluation set is a fixed collection of tasks, each with an input and success criteria, that you run the agent against before every change. Anthropic's guidance on evaluating agents sets out the vocabulary: a task is one test case, each attempt is a trial, a grader scores the result, and the transcript records everything the agent did.

  1. Collect real tasks. Anthropic notes that teams often think they need hundreds of tasks, but 20 to 50 simple tasks drawn from real failures is a good start. Its research team began with about 20 queries that reflected real use.
  2. Balance the set. Include cases where the agent should act and cases where it should not, such as a request it must refuse or escalate.
  3. Write clear success criteria for each task, and grade the outcome, not the exact path. Anthropic warns that checking each step rigidly breaks when the agent finds another valid route.
  4. Choose graders. Code-based checks are fast and objective but miss valid variations. Model-based graders handle nuance but need calibration against employee judgement. Review by employees is the most reliable and the slowest.
  5. Run each task several times. Agents vary between runs, so one pass proves little.
  6. Read the transcripts. Anthropic's guidance is that you will not know whether your graders work unless you read transcripts and grades from many trials.
  7. Keep two kinds of set. Capability evaluations test what the agent cannot yet do well and start with a low pass rate. Regression evaluations check that it still handles what it used to, and should pass at close to 100%.

When a model grades another model's work, check it for bias. A NeurIPS 2023 study of model-based grading found that a strong model could agree with human preferences more than 80% of the time, but also identified position, verbosity and self-enhancement biases. Compare a sample of model grades with employee grades before you rely on them.

Comparing models per task

No model is best at everything, and the right choice differs per task. Anthropic suggests routing simple requests to smaller, cheaper models and hard ones to more capable models. Decide this from your own evaluation set, not from public benchmarks alone, because public benchmarks rarely match your tasks and data.

  • Keep everything else fixed. Use the same tasks, instructions, tools and memory, and change only the model.
  • Run several trials per task and compare the success rate, pass^k where consistency matters, and the quality score.
  • Record cost per completed task and time to completion for each model, not only the price per token. A cheaper model that needs more steps or more rework can cost more per completed task.
  • Decide per task: the cheapest model that meets your quality bar for that task wins.
  • Re-run the comparison when a provider releases or updates a model, and when you change instructions or tools.

Hypothetical example, with invented figures: on a 40-task evaluation set for summarising contracts, model A passes 36 tasks at €0.30 per completed task, and model B passes 30 at €0.12. If 90% is the quality bar, model A is the right choice for this task, even though model B may be the better choice for routing emails.

Monitoring in production

An evaluation set shows how an agent performs on known tasks. Production shows how it performs on everything else. OpenAI recommends running evaluations on every change and monitoring live use to find new cases, which then grow the evaluation set.

  • Log every task with its outcome, tools used, approvals, escalations and cost. Anthropic found that full production tracing was what let it diagnose agent failures systematically.
  • Review a fixed random sample of completed tasks every week, using the same rubric as the evaluation set.
  • Turn every confirmed failure into a new evaluation task, so it cannot return unnoticed.
  • Watch for drift. A falling success rate or a rising change rate after an update is a signal to investigate, or to roll back.
  • Combine automated checks with human review. Anthropic's human testers found problems its automated evaluation missed, such as a bias towards low-quality sources.

Reporting to leadership

Leadership needs to know whether agents create value, what they cost, whether they are under control and what to decide next. A one-page monthly report per team covers this. Keep the same structure every month, so trends are visible.

  1. Value: tasks completed, task success rate and time saved, against the baseline.
  2. Cost: total spend and cost per completed task, against budget and against the cost of the previous process.
  3. Quality: review score and error rate, with the trend over three months.
  4. Control: escalation and approval rates, serious errors and incidents, and what changed as a result.
  5. Decisions: proposals to expand, raise autonomy, change model or stop an agent, each with the evidence.

Report figures from your own logs and evaluations, and say how they were measured. Do not mix in vendor claims or industry averages, which describe other companies' tasks. Treat time saved as capacity freed for other work, and say what that capacity was used for.

Hypothetical example, with invented figures: the support team's report shows 1,600 tickets completed, an 82% success rate, 210 hours saved against the timed baseline and a cost per completed ticket of €2.10, within budget. It proposes letting the agent send order-status replies without approval, because the change rate on those replies has been under 2% for three months.

How Blentera helps

Blentera is a platform where companies run AI agents with employees in charge. The activity log records what each agent does, which helps when sampling work and tracing failures. Approval rules can be set per agent, project, app or tool, so approval and change rates can be followed per action. Budgets warn at 80% and pause at the limit. Because one company memory stays when you switch AI models, you can compare models on the same task with the same context. Blentera is hosted in the EU.

Questions

Sources

  1. Demystifying evals for AI agents, Anthropic Engineering, 9 January 2026
  2. How we built our multi-agent research system, Anthropic Engineering
  3. Building effective agents, Anthropic Engineering
  4. Evaluation best practices, OpenAI API documentation
  5. Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1
  6. AI RMF Playbook: Measure, NIST
  7. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena, Zheng et al., NeurIPS 2023, arXiv:2306.05685

This guide is general information, not legal advice. Check how the rules apply to your company with your own advisers.

Get started

Put your first agent to work in minutes.

Three steps, and your first agent is working for your company.

  1. Step 1

    Create an account

    Sign up with your work email and invite your team. No sales call needed.

  2. Step 2

    Add your first agent

    Create an agent, give it a role, and connect your own AI key and the apps it may use.

  3. Step 3

    Give it a task

    Ask in Blentera or your chat app, and follow the work until it is done.