# What tokens actually cost, and where the money goes

> Source: https://codelearningdojo.com/token-economics/
> Part of Code Learning Dojo, free to read.

Almost every surprising API bill comes from the same short list of causes, and almost none of them are "we sent too many prompts". They are structural: a cache that never hits, a retry loop that quadruples cost on the worst day, a stream nobody is listening to any more.

This page is the model. The implementation is language-specific and lives on each site — [Python](https://learn-python.com/ai/tokenomics/), [JavaScript](https://learn-javascript.org/ai/tokenomics/), [TypeScript](https://learn-typescript.org/ai/tokenomics/), [Go](https://learn-go.org/ai/tokenomics/), [NestJS](https://learn-nestjs.com/ai/tokenomics/), [Bash](https://learn-bash.net/ai/tokenomics/), and [C](https://learn-c.net/ai/tokenomics/) for the local-inference case.

## Five prices, not one

Every major provider bills at least five different rates against the same request. Getting this wrong is the root of most cost surprises.

| What | Roughly | Why it matters |
|---|---|---|
| **Input** (uncached) | the baseline | grows with every document you stuff in |
| **Output** | typically several times input | the expensive one, and the one you control least |
| **Cache write** | a modest premium on input | you pay extra *once* to store a prefix |
| **Cache read** | a large discount on input | you pay a fraction on every subsequent hit |
| **Batch / async** | around half | for anything that does not need an answer now |

Exact multipliers differ by provider and change often — check current pricing rather than trusting a number in any article, including this one. What is stable is the **shape**: output costs several times input, cached input costs a small fraction of uncached, and batch is roughly half price.

:::verdict The three consequences
1. **A verbose answer costs more than a verbose question.** Constrain output length before you spend a day trimming your prompt.
2. **A stable prefix is nearly free to reuse.** Cache hits are the single biggest lever available, and they are a *structural* property of how you build prompts.
3. **Anything that can wait should not be a live call.** Batch endpoints halve the bill for evals, backfills and nightly jobs.
:::

## Prompt caching is the whole game

If you take one thing from this page: **most systems can cut their bill by more than half without changing a model, a prompt's content, or a single feature — purely by making the prefix stable.**

Caching works on a prefix match. Everything up to the first byte that differs can be reused; everything after it is charged in full. So the order of your prompt determines your bill.

```text
BAD  — cache never hits after the first token
  [timestamp] [user id] [system rules 4KB] [docs 40KB] [question]
   ^ changes every call, so nothing after it can be reused

GOOD — 44KB reused on every call
  [system rules 4KB] [docs 40KB] [timestamp] [user id] [question]
                                  ^ first difference is here
```

Same tokens, same output, and in the second version 44KB of prefix is charged at the cache-read rate instead of the full one.

The rules that follow from this:

- **Static content first, in a fixed order.** System prompt, tool definitions, few-shot examples, retrieved documents that do not change per request.
- **Volatile content last.** Timestamps, user identifiers, request ids, the actual question.
- **Never put a timestamp near the top.** This is the single most common cache killer and it is usually there by accident, in a "current date" line.
- **Do not reorder tool definitions between calls.** If they come from a dictionary or a set, sort them — iteration order that varies breaks the prefix.
- **Watch the TTL.** Caches expire on the order of minutes unless extended. A conversation with long human pauses will miss; a batch loop will hit constantly.

:::warn Cache writes cost more than plain input
Caching a prefix you use once is a small loss. It pays from roughly the second hit onward. So cache the things that repeat — system prompts, tool schemas, a document you will ask ten questions about — and do not cache a one-shot request.
:::

## The multipliers nobody budgets for

These are the ones that turn a projected bill into an actual one.

**Retries.** A naive retry-on-error wrapper with three attempts costs 3x on the day your provider has an incident — which is exactly the day you are least watching. Cap total attempts, use exponential backoff with jitter, and only retry on 429 and 5xx, never on a 400.

**Agent loops.** A tool-calling loop resends the entire conversation on every turn. A ten-turn loop over a 20k-token context is not 20k tokens, it is closer to 200k, because turn *n* includes everything from turns 1 to *n−1*. This is the largest and most consistently underestimated line item in agentic systems. Cap the turns, and prune tool results that are no longer needed.

**Abandoned streams.** A user closes the tab. Your server keeps consuming from the provider. You are billed for every token of an answer nobody will read. Propagating cancellation from the client through to the upstream request is a small change that shows up directly on the bill.

**Failed parses.** Output that does not validate gets regenerated, so your effective cost per successful result is higher than your cost per call. If 10% of responses fail to parse, you are paying 11% more than you think — and a bad prompt change can make that 40% overnight without any error appearing in your logs.

**Eval runs in CI.** Running a scored dataset on every push is a bill that scales with your commit rate. Nightly, plus on changes to prompt files, is almost always right.

## Routing: most requests do not need your best model

The largest structural saving after caching. Real workloads are mostly easy requests with a long tail of hard ones, and using one model for both means paying frontier prices for classification.

```text
classify / extract / route / format   -> small model
summarise / draft / answer            -> mid model
reason / plan / write code            -> large model
```

Two ways to implement it, in increasing order of effort:

1. **Static routing by task.** Each call site picks a tier. Crude, effective, and takes an afternoon.
2. **Escalation.** Try the cheap model, validate the result, escalate on failure. Works well when validation is cheap and objective — a schema parse, an enum check, a confidence threshold. Note that a failed cheap call plus a large call costs *more* than going large directly, so this only pays when the cheap model succeeds most of the time. Measure the success rate before assuming it does.

**Do not route by guessing difficulty with another model call.** You have added a call to save a call.

## Cutting output, not just input

Output is the expensive side and it is the one people leave alone.

- **Set `max_tokens` deliberately** on every call. Not as a safety net — as a budget. A classification endpoint does not need 4,000 tokens available.
- **Ask for the smallest useful shape.** `{"category":"billing"}` costs a fraction of a paragraph explaining the reasoning. If you do not read the reasoning, do not ask for it.
- **Beware "think step by step" as a default.** Reasoning tokens are output tokens. They are worth it for genuinely hard problems and pure cost on classification.
- **Structured output is cheaper than prose** and easier to validate — you save on both the tokens and the retries.

## Measure per feature, not per month

A monthly total tells you that costs went up. It does not tell you which feature, which customer or which change did it. Every call should be tagged at the point it is made:

```text
feature       which product surface
model         so you can see routing working
user / tenant so you can find the one customer costing 40% of the bill
env           dev / ci / prod, separated
version       prompt or code version, so a regression is attributable
```

With those five dimensions you can answer the questions that actually matter: *which feature is most expensive per active user*, *did last Tuesday's prompt change move cost*, *what is our gross margin per customer*. Without them you have a number that goes up.

Three metrics worth putting on a dashboard from day one:

- **Cache hit rate.** Should be high and stable. A sudden drop means someone put something volatile near the top of a prompt.
- **Cost per successful result**, not per call. Catches the failed-parse multiplier.
- **Tokens per request, p50 and p99.** The p99 is where the runaway loops live.

## What agent-assisted development itself costs

A separate budget, often larger than the application's, and usually invisible because it sits on a different bill.

The same principles apply, with one addition specific to coding agents: **context is the cost driver, and most of it is avoidable.** A session that has read forty files to find one function is paying for all forty on every subsequent turn. That is why the [repo structure advice](https://learn-python.com/ai/context/) is also cost advice, and why an MCP server you installed and never use is a standing charge on every request.

Practical levers, roughly in order of effect:

- **Start a new session when the task changes.** Carrying a finished task's context into the next one is pure waste, and it makes the model worse as well as more expensive.
- **Prune your MCP servers.** Tool definitions sit in every request. Check what they cost — `/context` in Claude Code shows the breakdown — and remove what you have not used this month.
- **Keep the instructions file short.** [Under 100 lines](https://learn-python.com/ai/agents-md/). It is prepended to everything.
- **Point at files, do not paste them.** An agent that can read a path on demand beats one handed 30KB it may not need.
- **Let it fail fast.** A [five-second test suite](https://learn-python.com/ai/feedback-loops/) is a cost optimisation as well as a quality one — the model converges in fewer turns, and each turn is cheaper because the context is smaller.

## A budget that holds

Three layers, and you want all three:

1. **A hard provider-side spend cap.** Not an alert — a cap. Alerts arrive after the money is gone.
2. **An application-level budget per request or per tenant**, enforced in code before the call is made. This is the one that stops a single runaway loop.
3. **A circuit breaker on anomaly**, not just on total. "This tenant used 50x their weekly average in an hour" catches an abuse case that a monthly cap does not.

Separate keys for dev, CI and production, with separate caps. A test loop that runs away on a shared key takes production down with it.

## Where to go next

The implementation — counting tokens, tagging calls, enforcing budgets, building the report — differs enough by language to be worth its own page:

- [Python](https://learn-python.com/ai/tokenomics/) — decorators for accounting, batch APIs, analysing the log
- [JavaScript](https://learn-javascript.org/ai/tokenomics/) — streaming, abandonment, middleware
- [TypeScript](https://learn-typescript.org/ai/tokenomics/) — making budgets a type error
- [Go](https://learn-go.org/ai/tokenomics/) — context-carried budgets, atomic counters, Prometheus
- [NestJS](https://learn-nestjs.com/ai/tokenomics/) — interceptors, per-tenant quotas, guards
- [Bash](https://learn-bash.net/ai/tokenomics/) — reading your own agent spend from the CLI
- [C](https://learn-c.net/ai/tokenomics/) — local inference, where the currency is memory bandwidth rather than tokens

## Common questions

### What is the single biggest saving available?

Prompt caching, in almost every system, and it is usually a reordering rather than a rewrite. Move everything stable to the front and everything volatile to the back. Most teams that do this see the majority of their input tokens move to the cached rate.

### Should I fine-tune to save money?

Rarely, and not primarily for cost. Fine-tuning trades a smaller prompt for training cost, a slower iteration loop and a model you now own the maintenance of. Prompt caching gets you most of the same input-token saving with none of that. Fine-tune for capability or latency, not for the bill.

### Is a cheaper model always worth trying?

Try it, but measure the *total* cost including retries and escalations, not the sticker price. A model that is a fifth of the price and fails a third of the time can easily be more expensive once you count the re-runs and the human time spent on the bad outputs.

### How much detail is worth building before it matters?

Tag every call with feature, model and tenant from day one — it is a few lines and it is nearly impossible to backfill. Everything else (dashboards, per-tenant budgets, circuit breakers) can wait until the bill is large enough to be worth an afternoon.