The right model for every agent is rarely the biggest one
A practical way to match model cost, context, speed, and reasoning depth to the work an agent actually does.
By Ada Nwosu
The short version
- Use the least expensive model that clears the quality bar for the job.
- Judge models on the work your agent performs, not a public leaderboard.
- Track corrections beside token cost so cheap mistakes don't look efficient.
- Keep the agent role stable when you test or replace its model.
Bigger models hide bad job design
A model upgrade can rescue a vague instruction, but it also makes the job harder to understand. If a support agent receives every ticket, every document, and an open-ended request to help, a larger model may produce a better answer. It will also charge you for reading context that should never have reached the session.
Start by narrowing the work. Give the agent one role, a clear finish condition, and access to the few sources it needs. Once the request is bounded, model choice becomes a practical decision instead of a guess about which name sits highest on a benchmark chart.
Test the job you actually have
Public evaluations measure useful things, but they don't measure your approval rules, your customer vocabulary, or the odd shape of your internal data. Build a small evaluation set from real work. Remove secrets and personal data, then keep the requests that exposed meaningful differences between models.
Run each candidate with the same agent instructions and tools. Record the answer, token cost, latency, tool failures, and whether a person had to repair the result. Ten representative requests will tell you more than a broad benchmark that never touches your workflow.
- Routine classification and extraction usually reward speed and low cost.
- Long account reviews need enough context without paying for unused reasoning.
- Code changes need models that can inspect files and follow tool feedback.
- Sensitive decisions need a stricter quality bar and human review.
Give each agent a budget
Cost limits work best at the agent level because jobs consume tokens differently. A ticket triage agent may run hundreds of short sessions. A research agent might run twice a day and read far more material each time. One workspace-wide model policy treats those jobs as though they have the same economics.
Set a target cost for a successful run, then watch the distribution rather than one average. A sudden tail of expensive sessions often points to oversized context, repeated tool calls, or retries that the agent cannot resolve. Changing the model can help, but first check what the session was asked to carry.
Change the model, not the role
The agent should remain the stable unit. Its instructions, connected tools, knowledge, and team access describe the job; the model is one replaceable part. That separation lets you test a faster model on support work without rebuilding the support agent or moving its history somewhere else.
When a provider lowers prices or a smaller model improves, rerun the same evaluation set. If quality holds, switch the assignment. Your teammates still talk to the same agent, completed sessions stay together, and the savings appear on the next run.
Continue reading
From a Slack mention to finished work, without losing the thread