LLM costs do not scale the way your intuition says

The demo costs almost nothing. You run the feature twenty times while building it, glance at the usage dashboard, and conclude the economics are fine. Then it ships, and the monthly bill is two orders of magnitude above the estimate.

The gap is almost never the price per token. It is that a demo and a production system consume tokens in structurally different ways.

Where the money actually goes

Retries and loops. An agent that occasionally takes eight steps instead of two does not cost four times more on average — it costs whatever the tail of that distribution costs, and the tail is where agents live. A single failure mode that sends the loop to its step limit can dominate the entire bill.

Context that grows with the conversation. If you resend the full history on every turn, a twenty-turn conversation is not twenty requests. It is a quadratic amount of input tokens. This is the single most common surprise.

Inputs you did not size. A document summarizer tested on two-page PDFs meets a 300-page one on day three. Whatever the user can upload, they will.

Output length nobody constrained. Models are verbose by default. A response that could be forty tokens comes back as four hundred, and on output-heavy workloads output is the expensive side.

The fixes, in order of payoff

Cache the stable prefix. Long system prompts, tool definitions, and retrieved documents that repeat across requests should not be re-billed at full rate every call. Prompt caching is usually the largest single win and requires no change to product behavior — only to how the prompt is ordered, since the cached part has to come first and stay byte-identical.

Put a hard ceiling on agent loops. Step limits, wall-clock limits, and a cost limit per request. An agent without a budget is a program without a base case.

Stop resending what the model does not need. Summarize old turns, drop tool outputs that have already been acted on, and keep a working set rather than a transcript.

Route by difficulty. Most requests in a typical product are easy. Sending all of them to the largest model is the equivalent of running every query through the slowest code path because some queries are hard.

Constrain output explicitly. Ask for the format you want and cap the length. "Answer in one sentence" is a cost optimization.

The part that is not about tokens

A meaningful share of LLM spend in production is not the model at all — it is that the feature was designed to call the model where it did not need to. Classification that a regex handles. Extraction from a document that already has structured metadata. Summarization of text the user is about to read anyway.

The cheapest token is the one you never send. That is a product decision, made during scoping, and it is worth more than every prompt-level optimization combined.