On this page 10 sections
Cost reductions fall into three tiers: changes that carry no quality risk at all (caching, retry hygiene, batch routing), changes that need measurement but usually hold (reranking, model tiering, prompt trimming), and changes that are genuine trades (smaller models on hard tasks, shorter context). Do them in that order, with an evaluation run between each.
The reason cost work gets deferred is a belief that spend and quality are tightly coupled. For a large part of most bills they are not.
Key takeaways
- Do the zero-risk tier first. It is usually a meaningful share of the bill and costs you nothing.
- Never change cost and prompt logic in the same deploy. You will not know which moved the score.
- Reranking often lets you send less context and get better answers.
- Model tiering by difficulty is the highest-leverage change with manageable risk.
- Every change past tier one needs a before-and-after evaluation score, per category.
Who this applies to
You have attributed your spend per workflow, and you want to reduce it without discovering the cost in customer complaints six weeks later.
Prerequisite: an evaluation set with a measured baseline. Without one, everything below is guesswork with a bill attached. Build one first; it takes a few days.
Tier one: no quality risk
These change what you pay, not what the system does. Ship them without ceremony.
1. Prompt caching
If your requests share a stable prefix - system instructions, few-shot examples, fixed policy text - providers that offer prompt caching will charge substantially less for the cached portion. The output is byte-identical, so there is nothing to evaluate.
Requires that the stable part comes first and genuinely does not vary. A timestamp injected at the top of an otherwise fixed preamble defeats it entirely, which is a common and expensive accident.
2. Exact-match response caching
Identical question, stable answer, same context: serve from cache. In support workloads the repeat rate is often high, and it spikes exactly when you least want the cost - during an incident, when hundreds of people ask the same thing within an hour.
Set a sensible time to live and invalidate on content changes so you do not cache an answer past its policy update.
3. Fix retry hygiene
Retries are billed. Failed schema validation, timeouts and tool errors that silently retry three times are pure waste.
Parse permissively so cosmetic drift does not trigger a retry, validate strictly so genuine problems fail loudly rather than looping, and cap retries explicitly. This one usually pays for itself in a morning.
4. Move batch work to batch pricing
Backfills, re-indexing, nightly summarisation and bulk classification do not need a synchronous answer. Where a provider offers a discounted asynchronous tier, work that can wait belongs there. Where they do not, it still belongs on a cheaper model with a rate limit.
5. Cap agent loops
A step ceiling and a per-request cost cap are not quality reductions, they are a fix for a failure mode. An agent that would have looped forty times was not producing a good answer at step thirty-nine.
Tier two: measure, but it usually holds
These normally maintain or improve quality. Verify anyway.
6. Rerank, then send less
The most underrated change on this list.
The usual pattern retrieves the top ten or twenty chunks by vector similarity and sends them all. Vector similarity is a coarse signal, so much of that context is loosely related and contributes nothing.
Instead: retrieve twenty candidates, run a cheap reranking model over them, send the top three. Input tokens fall sharply. Quality frequently improves, because you have removed distracting near-misses that were competing with the correct passage for the model’s attention.
Evaluate per category. The exception is genuinely multi-source questions - “compare these three plans” - where fewer passages hurt. Those may need a different retrieval path.
7. Tier models by difficulty
Route each request to the cheapest model that handles it. A small classifier decides; simple lookups, classification and extraction from clean text go to a cheap model, and only genuinely hard reasoning reaches the expensive one.
This commonly moves the majority of traffic down a tier. The risk is misrouting, so:
- Evaluate each tier separately against the cases it will receive
- Make the router conservative: when uncertain, route up
- Track the routing distribution and alert if it shifts
The failure mode to avoid is tuning the router on aggregate accuracy, which hides a small category being sent to a model that cannot handle it.
8. Trim the prompt
Prompts accumulate. Instructions get added to fix specific failures and are rarely removed once the underlying cause is addressed elsewhere. Few-shot examples that were needed on an older model may be redundant now.
Remove them one group at a time, with an evaluation run between each. Some will turn out to be load-bearing in ways nobody remembers, which is exactly why you do this with a test set rather than by reading.
Tier three: genuine trades
Real quality decisions. Make them explicitly, with numbers, and with someone accountable for the trade.
A smaller model on hard tasks. May be fine. Measure per category and expect the loss to concentrate in ambiguous and multi-step cases rather than spreading evenly.
Shorter context on complex questions. Below some threshold, answers degrade. Find your threshold empirically rather than adopting someone else’s.
Reducing output length. Cheaper, and can make answers less useful. Worth doing where you are paying for padding, worth measuring where you are not.
Removing a verification step. A second model checking the first costs roughly double on that path. Removing it saves real money and removes a safety net. Only reasonable where the cost of an error is low.
The discipline that makes this safe
Four rules, all learned the same way.
- One change per evaluation run. Batching four optimisations and finding the score dropped three points tells you nothing about which one to revert.
- Never combine a cost change with a prompt change in the same deploy.
- Score per category, not on average. Nearly every cost reduction that looks fine on average is concentrating its damage somewhere specific.
- Record cost per case alongside the score. The output of this work is a curve, not a number: what you pay and what you get. Decisions get much easier when both are on the table.
A useful artefact is a short table of options with the measured spend reduction and the measured quality delta for each, so the business chooses rather than engineering choosing on its behalf.
What we do first, and what we refuse to do
Our standard sequence on a cost engagement is attribution, then the whole of tier one, then a re-measure before anything in tier two is touched. The reason is that tier one frequently changes the picture enough that some of the tier-two work is no longer worth its risk.
The thing we decline to do is reduce cost on a system with no evaluation set. It sounds like an upsell and it is a genuine constraint: without a baseline there is no way to distinguish a saving from a degradation, and the degradation arrives weeks later as complaint volume that nobody connects to the change. When a client wants the cost work and not the measurement, the honest answer is that we can do the tier-one items safely and nothing else, because tier one is the part where the output is provably identical.
The other thing worth saying: we quote the cost audit as read-only and fixed-fee, ending in a ranked list with quality risk attached to each item. Implementation is a separate decision. Bundling them creates an incentive to recommend the changes that bill the most hours rather than the ones that should be made.
When to stop
Cost work has diminishing returns and the stopping point is a business judgement.
When spend is small against the value delivered. A system saving $9,000 of labour for $400 a month does not need a second optimisation pass.
When the next item is a real trade you would not take. If the remaining options all cost measurable quality, you are done. Set a budget alert and move on.
When the effort exceeds the saving. Two engineer-weeks to save $200 a month is a poor trade unless the volume is about to grow substantially.
Frequently asked questions
How much should we expect to save?
It depends entirely on how the system was built, and any single percentage quoted across all systems is invented. What is reliable: tier one is free and worth doing on every system, and the size of the opportunity correlates with how quickly the system was built.
Will a cheaper model always be worse?
No. For classification, extraction from clean text and short rewriting, small models often match large ones at a fraction of the cost. The gap appears on multi-step reasoning and ambiguous inputs.
Should we fine-tune a small model to replace a large one?
Sometimes viable at high steady volume on a narrow task. You take on training cost, a data pipeline and a maintenance obligation. Do the arithmetic over three years, not three months, and include the engineering time.
Is reducing context the same as reducing quality?
Not below the point where you were sending redundant context. Above it, yes. Reranking is how you find where that line is without guessing.
How do we present this to a stakeholder who fears quality loss?
With the curve. Show the measured evaluation score alongside the measured spend for each option, including “change nothing”. A trade is easy to discuss when both axes are numbers rather than one number and one worry.
Next step
The AI cost audit produces exactly that curve: spend attributed per workflow, and a ranked reduction list with measured quality risk against each item, as a fixed-fee read-only engagement.
Related: How to measure whether an AI system works · Building an eval set from real tickets · Your provider changed the model, what broke · AI cost audit