The Bill Nobody Modelled: Where LLM Spend Actually Goes
The first surprise on almost every production LLM bill is that it does not look like the estimate. The estimate was built from an average request: so many input tokens, so many output tokens, multiplied by expected volume. The bill is built from the tail. Output tokens are priced at a multiple of input tokens at essentially every provider. The ratio varies and it changes, but it is always a multiple and never parity, which means a workload that generates long prose is structurally more expensive than one that generates a short structured object carrying the same information. Teams that never separate input and output in their forecasting are working from a model that cannot predict their own spend.
The second line item nobody budgets is retries. A three percent transport failure rate with automatic retry adds three percent, which is fine. A validation-failure loop is not fine: when a model returns malformed structured output and your code retries with a corrective message, you pay for the failed attempt, the corrective prompt and the second generation. On workloads we have inherited, retry overhead of 20 to 40 percent of total spend is common, and it is almost always invisible because the retries are logged as successes at the application level. Nobody is looking at attempts per successful response as a metric until somebody goes looking for money.
The third and largest is the agent loop. A single-call workflow costs one call. An agent working a task in 8 to 15 steps costs 8 to 15 calls, and each step re-sends the accumulated history, so total tokens grow roughly with the square of the step count rather than linearly. Add a long system prompt to that and the arithmetic gets ugly fast: a 4,000-token system prompt on a 20-turn conversation is 80,000 input tokens of pure repetition before anyone has said anything useful. Gartner projects that 40 percent of enterprise applications will feature task-specific AI agents by the end of 2026, up from less than 5 percent in 2025, so this shape of bill is about to become the normal shape.
The Routing Idea in One Sentence
Route each request to the cheapest model that can actually handle it. That is the whole idea, and its power comes from a fact about real traffic that averages hide: workloads are not uniform. In the production systems we see, something like 60 to 80 percent of traffic is routine work — classifying an incoming message, extracting fields from a form, summarising a short passage, reformatting text, answering a question the retrieval layer has already answered. Perhaps 10 to 25 percent genuinely needs the strongest model available: multi-step reasoning, ambiguous instructions, synthesis across contradictory sources, anything where being wrong is expensive.
If you send all of it to the frontier model, you are paying frontier prices for classification. If you send all of it to a small model, you fail the hard quarter of your traffic and lose users. Choosing one model for the whole application forces you to price the entire workload at the cost of its hardest request, which is the same mistake as sizing every server in a fleet for peak load. The RouteLLM paper published at ICLR 2025 reported up to 85 percent cost reduction while retaining roughly 95 percent of GPT-4 quality on MT-Bench, which is the academic version of the point: most requests do not need the expensive model, and a router can tell which ones.
It is worth being clear about what routing is not. It is not a machine learning project. The overwhelming majority of the available saving comes from boring, deterministic rules written by someone who understands the workload, and the sophisticated approaches add a modest increment on top. Teams that start by training a router classifier usually spend two months to reach a result they could have had in two days. Start with rules, measure, and only add intelligence where the rules demonstrably misroute. This is the same order of operations we recommend for small language models: prove the cheap path works on a slice before industrialising it.
Where the Routing Decision Is Made
Routing needs a place to live, and that place is a gateway: a single layer every model call passes through. If the model name is hard-coded at forty call sites across your application, you do not have a routing problem, you have a refactoring problem, and no amount of clever router design will help until it is fixed. The gateway is where model selection, fallback on provider error, retry policy, caching, budget enforcement, rate limiting and structured logging all belong. LiteLLM is a widely used open-source implementation of exactly this layer, and building your own thin version is a few days of work.
The gateway needs metadata to route on, and this is the part teams under-design. Every call should arrive carrying: the task type, which is the single most predictive signal; the tenant or customer, because enterprise tiers may have contractual model requirements; the user tier, because a free plan and a paid plan can legitimately get different routes; the latency budget, because an interactive chat and a nightly batch job have completely different constraints; and a data sensitivity flag, because some content must never leave a particular jurisdiction or must run on a self-hosted model regardless of cost.
That last dimension is where cost engineering meets compliance, and it is why routing and governance are the same conversation in regulated Turkish deployments. A request touching special-category personal data under Law No. 6698 may be routed to a self-hosted model not because it is cheaper but because it is the only lawful option, and the gateway is the only sensible place to enforce that. Once the metadata is flowing, the routing rules themselves are usually thirty lines of code. Getting the metadata to the gateway is the actual engineering work, and it is worth doing properly because everything else in this article depends on it.
Router Types and What They Cost in Latency
Rule-based routing is a lookup and a few comparisons: endpoint implies task type, document length implies context requirement, tenant implies tier. It executes on the order of a millisecond. It is unglamorous, fully explainable, trivially testable, and in our experience it captures the large majority of the available saving on a first pass. Every routing programme should start here and stay here until measurement proves the rules are misrouting a meaningful share of traffic. A rule you can read in a code review is also a rule you can debug at two in the morning when a customer says the answers got worse.
Embedding-similarity routing embeds the incoming request and compares it against labelled centroids for each route. It costs one embedding call, typically 10 to 40 milliseconds, and it handles the case where the same endpoint receives genuinely different kinds of question. The semantic-router project is the common open-source reference. A step further, a small fine-tuned encoder classifier trained on your own labelled traffic runs in the tens of milliseconds and can encode subtleties a centroid cannot, at the cost of a labelling exercise and a model you now have to maintain and re-evaluate whenever the traffic distribution shifts.
LLM-as-router is the expensive option: you call a model to decide which model to call. It costs a full inference, typically hundreds of milliseconds to a couple of seconds, plus tokens, and it is only justified when the routing decision genuinely requires understanding that nothing cheaper can supply. In most systems it is a sign the task taxonomy has not been thought through. Note the orders of magnitude here as an engineer would: a rule at roughly one millisecond and a classifier at tens of milliseconds sit one to three orders of magnitude below model inference measured in hundreds to thousands of milliseconds. Latency is almost never a defensible reason not to route.
The Cascade Pattern: Cheap First, Escalate on Doubt
The cascade inverts the routing question. Instead of predicting in advance which model a request needs, you run the cheap model, check whether the answer is good enough, and escalate only when it is not. This is strictly more accurate than prediction, because you are judging an actual answer rather than guessing from the question, and it degrades gracefully: a misjudged escalation costs money, not quality. The cost of the pattern is one extra decision step and, on escalated requests, a wasted cheap call. The design question is entirely about the confidence check.
Your options for that check, roughly in order of reliability: a deterministic validator, which is the best one whenever it applies — does the output parse against the schema, are the required fields present, does the extracted total match the sum of the line items, is the cited document ID one that was actually retrieved. Then logprob-derived uncertainty where the provider exposes it. Then agreement between two independent cheap runs, which doubles the cheap cost but is very effective on extraction tasks. Then a small judge model. Last and least reliable, the model's own stated confidence, which is close to worthless on its own and should never be your only gate.
The arithmetic is simple enough to do on a whiteboard. If the cheap model handles a fraction of traffic successfully and the rest escalates, your total cost is one cheap call for every request plus one expensive call for the escalated fraction. The cascade pays whenever the cheap call is a small fraction of the expensive one and the escalation rate stays low; it stops paying when escalation climbs, because you are then paying for both models on most requests. Instrument the escalation rate as a first-class metric and alert on it, because a rising escalation rate is usually the first visible sign that your traffic mix has shifted underneath you.
Prompt Caching and the Stable Prefix
Prompt caching is the cheapest win available and the one most often left on the table through prompt structure alone. Providers cache a prefix of your prompt and charge substantially less for cached input tokens on subsequent requests that share it. The requirement is strict: the prefix must be identical, token for token, from one request to the next. A single character difference at position ten invalidates everything after it. This turns prompt layout from a stylistic matter into a cost decision, and it is the reason a prompt that reads nicely can cost several times more than one that reads slightly worse.
The rule follows directly: static content first, dynamic content last. System instructions, tool and function definitions, few-shot examples, policy text, glossaries and long reference documents go at the top, in a fixed order, and never move. The user's current turn, retrieved chunks, timestamps, session identifiers and anything else that changes per request go at the bottom. The classic mistake is a current-date line or a request ID at the top of the system prompt, which is convenient for debugging and destroys the cache on every single call. The second classic mistake is retrieved chunks whose order is not stable across otherwise identical queries.
Two caveats keep this honest. Cache lifetimes are short, typically measured in minutes, so caching pays enormously on bursty multi-turn conversations and barely at all on sparse one-shot traffic arriving hours apart. And on workloads with very long system prompts, caching frequently saves more than model choice does, which means the correct order of work is: fix prompt structure first, then route, then consider cascades. This is where routing overlaps with context engineering, and where deciding between long context and retrieval, covered in long context or RAG, becomes a cost decision rather than an architectural preference.
Output-Token Discipline
Output is the expensive half of the bill and also the slow half, because generation is sequential in a way that reading input is not. Every token the model writes costs money and adds latency, so an answer that is twice as long is roughly twice as expensive and twice as slow for no additional information. The single most effective intervention is to stop asking for prose when the consumer of the answer is code. If a downstream service is going to parse the result, ask for a structured object against a schema and nothing else. In our experience converting a chat-style prompt to a schema-constrained one cuts output tokens by 30 to 60 percent for the same information content.
Set a maximum token limit per task, deliberately, as a budget rather than a safety valve. Writing be concise in a prompt is a request; a hard limit is a control, and the difference matters when a model decides to be thorough on the one request that happens to be from your largest customer. Use stop sequences where the output has a natural terminator. Where the model is selecting rather than generating, ask for identifiers instead of content: returning a document ID and a character span is a handful of tokens, while re-typing the paragraph it points at is hundreds, and the retyped version can also be subtly wrong.
Then look at the prompt from the other direction. Few-shot examples are input tokens you pay for on every uncached call, and teams routinely carry eight examples where three would do because nobody ever removed the ones added during debugging. Long chain-of-thought instructions on tasks that do not need reasoning generate reasoning tokens you pay for and then discard. For genuinely hard problems the reasoning is worth it, which is why reasoning models belong on a specific route rather than as your default. Routing them by task type is exactly the sort of decision the gateway exists to make.
Batching and Asynchronous Processing
Most providers offer a batch pathway that trades latency for a substantial discount: you submit a set of requests, you get results within a window measured in hours, and you pay meaningfully less per token. The interesting question is not whether the discount exists but how much of your workload is genuinely synchronous. In practice, far less than teams assume. Nightly classification of the previous day's tickets, backfilling metadata across an archive, regenerating embeddings after a model upgrade, enriching a catalogue, running an evaluation suite: none of these need an answer in two seconds, and all of them are routinely run through the interactive endpoint because that is what the code already did.
Restructuring for asynchrony changes product design as well as cost. A user who submits a long document for analysis and receives a notification when it is ready has a better experience than one who watches a spinner for ninety seconds, and the asynchronous version can be batched, retried, prioritised and rate-limited without the user noticing. The design rule is to identify the requests where a human is actually waiting on the answer, keep those on the interactive path with a strict latency budget, and push everything else onto a queue where cost optimisation is free.
Deduplication belongs in the same section because it is the same insight applied to repetition. Exact-match caching on identical requests within a time window is trivial to implement and catches more than people expect, particularly on support and internal-tooling workloads where many users ask the same question the same way. Semantic caching, where a near-identical query returns a previously computed answer above a similarity threshold, typically achieves 10 to 30 percent hit rates on those workloads in our experience. Set the threshold conservatively, log every hit, and make sure a cached answer can never cross a tenant boundary.
The Real Risk: Silent Quality Regression
Here is the failure mode that makes routing dangerous, and the reason many teams that try it quietly roll it back. When routing goes wrong it does not throw an error. The cheap model returns a fluent, confident, plausible answer that is slightly worse: a nuance dropped, a condition missed, a number rounded, a caveat omitted. Nothing fails. No alert fires. The cost dashboard goes down and everybody congratulates themselves, and six weeks later the support team mentions that answers have felt off lately, and nobody can point to when it started because nothing was measured before the change.
The defence is an evaluation gate that runs before the change merges, not after it ships. Build a fixed evaluation set per task type — 100 to 300 cases is a realistic starting size for a first production workload — with graded expected outputs, and run it against any change to routing rules, model versions, prompts or thresholds. Set a pass threshold and refuse merges below it, exactly as you would for a failing unit test. This is the same discipline described in AI observability and eval, applied specifically to the question of whether the cheap route is good enough.
Two details make the gate actually work. First, measure quality per route, never in aggregate: an aggregate score stays comfortable while the cheap route degrades badly on the fifteen percent of traffic it handles worst, because the expensive route's good scores mask it. Second, use shadow mode before switching traffic. Run the candidate route in parallel with the incumbent, log both outputs, compare offline, and only then move a small percentage of live traffic. Routing changes deserve the same release discipline as schema migrations, because they are just as capable of degrading a system in ways that take weeks to surface.
Instrumenting Cost per Workflow, and How We Run This
The metric that matters is cost per workflow execution, not cost per token and not the monthly invoice. Total spend rising alongside volume is a healthy business; cost per execution rising is a regression, and only the second one tells you anything actionable. To measure it, attach a trace identifier at the entry point of every workflow and accumulate the token counts, model identifiers and latencies of every call made under that trace, including retries, escalations, guardrail calls and embedding calls. Divide by completed executions and you can finally say what a resolved support ticket, a screened contract or an enriched record actually costs.
Slice that number by tenant, task type, route and prompt version, and the optimisation targets identify themselves. One tenant with unusual document sizes, one task type where the escalation rate has drifted upward, one prompt version that quietly doubled its few-shot block. Alert on cost per execution rather than on total spend, and put it on the same dashboard as your quality scores so that nobody can improve one while silently damaging the other. This is also the number that makes an honest AI project cost conversation possible with a finance team, because it converts token accounting into unit economics they already understand.
In our own work the sequence rarely varies. We put a gateway in front of everything before touching model selection, because routing without a single choke point is not implementable. We fix prompt structure for cacheability next, since it is the cheapest change with no quality risk. Then rule-based routing by task type, then an evaluation gate, and only then cascades or a learned router if measurement says the rules are misrouting. Where a client's volume on a narrow task is high enough, the routing conversation naturally becomes an economics of fine-tuning conversation, and our LLM training and fine-tuning work usually begins exactly there: with a route that is called often enough to justify a purpose-built model behind it.