The most common way AI agent projects die isn’t technical failure — it’s the second month’s API bill. The build works, the demo impresses, and then real volume meets frontier-model pricing on every single step. The fix isn’t a cheaper model; it’s routing: matching each step in your system to the least expensive model that does that step well.
This is how we run our own production systems, where routing discipline is the difference between pipelines that cost cents per output and pipelines that would be unaffordable. The principles below are model-agnostic and survive every pricing change.
Why routing works: the difficulty distribution
Look at any real AI workflow step by step and a pattern appears: most steps are mechanical, few are judgment. Extracting fields from text, classifying items, reformatting, tagging, simple summarization — these are tasks that small, cheap models handle at near-parity with frontier models. The steps that genuinely benefit from a frontier model — nuanced writing, complex reasoning, ambiguous decisions — are usually a small minority of total calls.
Paying frontier prices on mechanical steps is buying a surgeon to take temperatures. The cost difference between model tiers is routinely 10–50× per token (check current provider pricing — the ratio persists even as prices move), so routing the mechanical majority to the cheap tier collapses the bill while the quality-critical minority keeps its quality.
The three-tier pattern
We organize models into working tiers rather than picking favorites — the tiers outlive any specific model release:
- Worker tier (cheap, fast): classification, extraction, tagging, routing decisions, first-pass filtering, format conversion. The overwhelming majority of calls by count.
- Engineer tier (strong): drafting that ships, multi-step reasoning, code generation, decisions with consequences. Fewer calls, each one earning its price.
- Review tier (strongest, metered): periodic quality audits, evaluating the workers’ output samples, hard escalations. Rare by design — think scheduled inspections, not a step in the pipeline.
Assign every step in your workflow to a tier explicitly, in writing or config. Ad-hoc model choices drift expensive.
The routing decision, step by step
- Default every new step to the worker tier. Promotion requires evidence, not vibes.
- Define what “good enough” means for the step — a checkable output standard, not a feeling.
- Sample the cheap model’s output against that standard. Twenty examples tell you more than intuition. Passes → it stays cheap. Fails specifically → try a better prompt before a better model; prompt fixes are free.
- Promote only the failing step, not the whole pipeline. Tier upgrades are surgical.
- Re-audit quarterly. Cheap models improve fast; steps promoted a year ago often demote today.
The escalation pattern: cheap first, strong on failure
For steps where the cheap model succeeds most of the time, don’t promote — escalate: run the cheap model, validate its output with a guard (does it parse, does it meet the standard), and only on failure re-run the step on the strong model. You pay frontier prices only for the hard cases, which is exactly where they’re worth paying. This one pattern often preserves 90%+ of the savings while erasing the quality gap.
The costs people forget to route
- Context length is spend. Sending the same bloated context to every call multiplies cost invisibly. Trim inputs per step — the classification step doesn’t need the full document history.
- Retries multiply everything. A flaky step that retries three times costs 4×. Fix reliability before scaling volume.
- Batching beats streaming calls. One call summarizing eight items costs far less than eight calls summarizing one — and providers increasingly offer discounted batch endpoints for non-urgent work. Check what your provider currently offers.
- Logging is your meter. Log model, tokens, and step name for every call from day one. You cannot route what you cannot see — a simple cost-per-output number per pipeline is the metric that keeps you honest.
A worked shape (ratios, not quotes)
Take the daily research agent: thirty items classified by a worker-tier model, survivors summarized in one engineer-tier call. The classification calls, despite being 30× more numerous, typically cost a fraction of the single strong call — and the whole day’s run lands in the cents. Invert the routing (strong model on every classification) and the same agent costs a large multiple more for output nobody can tell apart. That asymmetry is the entire argument.
Frequently asked questions
Doesn’t managing multiple models add complexity?
A little, once — a config file mapping steps to models, which good workflow tools and API layers make trivial. The complexity of routing is a rounding error next to the complexity of explaining an unaffordable bill.
Can I just use one mid-tier model for everything?
It’s a defensible simplification for low volume, and strictly worse at pipeline volume: you overpay on mechanical steps and underdeliver on judgment steps simultaneously. The two-tier minimum (worker + engineer) captures most of the benefit.
How do I know quality isn’t silently degrading on the cheap tier?
Guards catch structural failures instantly; for judgment quality, sample-audit — the review tier’s job. A weekly random sample of ten outputs, evaluated against your written standard, detects drift long before customers do.
Apply it: build the research agent · see where agents fit your business in five small-business use cases · or browse the AI Agents hub.