How do I test a cheaper AI agent without losing quality?
Build a repeatable comparison of task quality, complete-run cost, latency and review effort before a rollout.
By OlypsisUpdated 3 minute read
Overview
Define a useful result first. Run both versions on the same saved tasks. Include hard cases. Compare the full cost of the results that pass. Inspect new failures. Check the outcome on a small live rollout before you expand it.
Write down what a pass means
State what each task must get right. Name the mistakes that make a result useless. Include needed fields, sources, sums and allowed actions. Decide when a person must review it. A fluent answer can still fail the job.
Use code to check facts and rules where you can. Ask a skilled person to review the parts that need context. If you use an AI judge, check its scores against human reviews first.
Use the human-review planning guide to decide which actions still need a person.
Keep a test set you do not tune against
Save routine tasks, costly tasks and known failures. Remove private data where you can. Keep the details that make a case hard. Protect the test set with the same access rules as the source records.
Use one set while making changes. Keep another set apart for the final test. Save the version of each input, tool, prompt and model. Date any external data. If a tool reads live data, replay a saved result or state that the input changed.
For more on datasets, evaluators and offline versus online tests, see LangSmith’s evaluation concepts. The same distinctions are useful with your own test harness.
Score the whole result
| Measure | How to compare it | What it prevents |
|---|---|---|
| Task quality | Pass/fail rules and blind review of the paired outputs | Choosing fluent but incorrect answers |
| Automatic completion | Tasks meeting the standard without human fixes | Hiding repair work inside a success rate |
| Final correct completion | Unique tasks meeting the standard, including repaired tasks | Counting unfinished work as done |
| Cost per correctly completed task | All model, tool and human costs divided by final correct completions | Hiding spend on failed attempts and repairs |
| Latency | Whole-task time and slow cases | Trading a lower bill for unusable delays |
| Human review | Review time and corrections per task | Moving model cost into staff work |
| Unsafe or invalid actions | Action logs and explicit permission checks | A good average hiding a critical error |
Use the agent cost calculator to include that work in the estimate.
A 70% automatic completion rate can become 100% final completion if people fix the remaining 30%. Include their time and count each completed task once. If some tasks still fail, keep them out of the completion count but keep their costs in the total.
Reliable checks and routing are part of that result. Undetected wrong answers are failures, not automatic successes. Log review minutes and extra correction minutes separately so the same time is not billed twice.
Inspect differences before calling it a win
Give both versions the same test conditions. Repeat cases where outputs vary. Hide which version was cheaper from the reviewers. Inspect each new serious failure, even if the mean score improves.
Report the sample and its limits. A small replay can catch faults and show where to look next. It cannot prove the error rate of the whole service. Live users may get a different result.
The Thesis Finance replay case separates a computed cost result from the live comparison that was still needed.
Roll out with a clear stop rule
Before release, choose the live traffic and how long to watch it. Set rules for when to roll back. Track failures, cost, time and review effort by task type. A shadow run can compare outputs without acting. It still costs money and must follow data access rules.
Expand only after the cheaper version meets your rules. Keep the old version ready until you can explain the live result. You should have evidence for the change and a way to undo it.
Have a project in mind? Tell us what you’d like to improve, and we’ll help you work out where to start.
Book a call