How to Reduce LLM API Costs Without Making Your Product Worse

The first invoice for an AI feature is rarely the scary one. The scary one arrives three months later, when usage has grown, someone has added two more tools to the agent, the system prompt has doubled, and the product is still running every request through the most expensive model because that is what the prototype used. It is a common way for an AI product to end up working well and costing more to run than it earns.

The good news is that most of that spend is waste, not a law of physics. This guide walks through the changes that reliably reduce LLM API costs, in the order we would apply them, with a worked example built on prices we checked on the providers' own pages in November 2026. It is written for the people who sign off on the bill: founders, CTOs and product leads who need to know what to ask their team for, and what each change risks.

Step 0: Measure cost per task, not cost per token

Token prices are the wrong unit for a business decision. What you care about is the cost of one completed job: one resolved support conversation, one processed invoice, one generated report. That number includes retries, failed tool calls, the follow-up the user had to send because the first answer was wrong, and the human who had to step in.

Before you optimise anything, log four numbers per request: input tokens, output tokens, which model handled it, and whether the task succeeded. Tag each request with the feature it belongs to. Within a week you will usually find that one or two features drive most of the spend, and that the expensive part is not where anyone guessed.

At the same time, build a small evaluation set: 100 to 300 real, anonymised requests with known good outcomes. Every step below changes model behaviour in some way, and this set is how you prove a saving did not quietly make the product worse.

The worked example

To keep this concrete, here is a typical customer-support assistant. The numbers are illustrative assumptions, not data from a client, but the shape is common:

• 300,000 requests a month.
• Each request carries a 6,000-token stable prefix (system prompt, tool definitions, policy excerpts) plus 1,500 tokens of conversation and retrieved context.
• Each response is about 400 tokens.
• Everything runs on Claude Sonnet 5.5, which Anthropic's pricing page lists at $2 per million input tokens and $10 per million output tokens.

That is 2,250 million input tokens ($4,500) and 120 million output tokens ($1,200): about $5,700 a month before any optimisation. Here is how it comes down.

Treat the hit rate, routing share and output length as assumptions to replace with your own logs. The method is what transfers, not the exact percentages.

Step 1: Cache the part of the prompt that never changes

Most production prompts are mostly boilerplate. The instructions, the tool schemas, the brand voice rules and the policy excerpts are identical on every request; only the last few thousand tokens differ. Prompt caching lets the provider store the processed version of that shared prefix so you do not pay full price to process it again.

The pricing is what makes this the first move. According to Anthropic's current price list, writing a prompt to the five-minute cache on Sonnet 5.5 costs $2.50 per million tokens, a one-hour cache write costs $4, and a cache hit costs $0.10. The prompt caching documentation states that hits on Opus 5.5 and Sonnet 5.5 are billed at 0.05x the base input price, with most older models at 0.1x.

OpenAI works differently. Its prompt caching guide says caching is enabled by default on supported models, with a minimum cacheable prompt of 1,024 tokens on GPT-5.6 and later. For those models the default 30-minute lifetime has a cache-write rate of 1.25x the uncached input rate, and a 12-hour option costs 2.5x. So on both platforms, caching only pays when the same prefix is reused enough times to cover the write premium.

In the example, 1,800 million of the 2,250 million monthly input tokens are prefix. At a 95% hit rate, the cached reads cost about $171, the cache writes about $225, the variable input $900 and output stays at $1,200. Total: roughly $2,500, down from $5,700.

What breaks caching

• Anything dynamic at the top of the prompt. A timestamp, user name or session ID in the first line changes the prefix and turns every request into a cache miss. Put stable content first and variable content last.
• Reordered tools or documents. If your code builds the tool list from an unordered map, the prefix can differ between requests even when the content is the same.
• Low traffic. With a five-minute lifetime, a feature that gets a request every twenty minutes will keep paying the write premium. Either use the longer lifetime or accept that caching will not help that feature much.
• Parallel bursts. Anthropic's documentation notes that if you need cache hits for parallel requests, you should wait for the first response before sending the rest. Fire fifty requests at once on a cold cache and you pay fifty writes.
• Prefixes below the minimum. Each model has a minimum cacheable length, which Anthropic lists at between 512 and 4,096 tokens depending on the model. Short prompts are not cached at all.

Step 2: Route easy requests to a smaller model

Prototypes use one model for everything because it is simpler. Production traffic is not uniform. In a support assistant, a large share of messages are order-status lookups, password resets and questions the help centre already answers. Those do not need the model you chose for its reasoning.

Routing means a cheap first step decides which model handles each request. That step can be a rules layer (intent keywords, request type, user tier), a small classifier model, or a combination. The research supports the idea: the RouteLLM paper (Ong et al., 2024) reported that learned routers between a strong and a weak model cut costs "by over 2 times in certain cases" without compromising response quality on the benchmarks they tested. That is a benchmark result, not a guarantee for your traffic.

In the example, we send 60% of requests to Claude Haiku 5.5, which Anthropic lists at $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens. The Haiku share costs about $80 a month, the remaining Sonnet traffic about $1,000, and a short routing call on every request about $20. Total: roughly $1,100.

Three rules keep routing from backfiring:

1. Route on evidence. Run the evaluation set through both models and only route request types where the small model's success rate is close enough to the large one.
2. Escalate on doubt. If the small model's answer fails validation, calls a tool incorrectly or the user rephrases the question, retry on the larger model. Count those escalations in your cost per task.
3. Watch the pricing tiers. Some models change price with prompt length. Anthropic lists Haiku 5.5 at five times its base input rate for prompts over 100,000 tokens. A router that sends long-context jobs to the "cheap" model can make them more expensive.

Routing is also where agent architecture matters. If you are deciding whether one agent or several should handle a workflow, our breakdown of single agent vs multi-agent systems covers how much more token use a multi-agent design adds and when it is worth it.

Step 3: Stop paying for words nobody reads

On almost every current model, output tokens cost five times as much as input. Sonnet 5.5 is $2 in and $10 out; Opus 5.5 is $4 in and $20 out. Yet output length is often the least managed part of a prompt. Models are verbose by default, and a support answer that restates the question, apologises twice and ends with a list of further options costs more and helps less.

Practical fixes:

• Set an explicit length and format in the instructions ("answer in under 120 words; use a list only for steps").
• Set a hard maximum output limit per feature as a safety net, so a runaway response cannot cost ten times the average.
• Use structured output (JSON with defined fields) for anything a machine reads next. It is shorter and easier to validate.
• For reasoning models, check whether you need the highest reasoning setting on every request. Extended reasoning produces extra output tokens, and you pay for them.

In the example, trimming the average answer from 400 to 250 tokens saves about $190 a month, taking the total to roughly $900. More importantly, shorter answers tend to be read.

Step 4: Move work that can wait to a batch API

Not every model call needs to be answered in two seconds. Nightly ticket tagging, backfilling summaries for old records, document classification, evaluation runs and weekly reports are all jobs where nobody is staring at a spinner.

Both major providers discount these heavily. Anthropic's Message Batches API cuts costs by 50% and says most batches finish in under an hour. OpenAI's Batch API also offers a 50% discount, with a 24-hour completion window. If the support team in our example also ran a nightly job to tag and summarise all 300,000 conversations, putting that job through a batch API would halve its cost with no product change at all.

The work here is mostly plumbing: a queue, a job that submits batches, a job that collects results, and error handling for the requests that fail. It is a few days of engineering for a permanent 50% cut on that workload.

Step 5: Shrink what goes into the context

The variable part of the prompt grows quietly. Retrieval pipelines that fetch ten chunks when three would do, full conversation histories replayed on every turn, and tool results pasted in raw (an entire API response when the model needs two fields) all add input tokens on every single request.

• Tune retrieval. Measure answer quality with fewer, better-ranked chunks. More context is not always more accuracy.
• Summarise long conversations. After a set number of turns, replace older messages with a short running summary.
• Trim tool outputs. Return only the fields the model needs. This is often the single cheapest fix in agent systems, where tool results can dwarf everything else.
• Drop unused tools. Every tool definition is sent on every request. If a feature uses three of twelve tools, load three.

What not to do

A few cost-cutting moves cause more trouble than they save:

• Downgrading the whole product to the cheapest model without an evaluation set. Savings show up on the invoice immediately; quality loss shows up later in churn and support tickets.
• Caching responses by exact match for open-ended questions. Response caching works for repeated, deterministic queries; for anything personalised it risks serving one user's answer to another.
• Self-hosting to save money before you have applied the steps above. GPUs, inference servers, monitoring and on-call time are real costs. Self-hosting is a strong option for data residency or a fine-tuned small model, and a weak one as a first response to a big API bill.
• Optimising without per-feature attribution. If you cannot say which feature costs what, you will spend a sprint optimising the wrong one.

A checklist to hand your team

1. Log tokens, model and task success per request, tagged by feature.
2. Build a 100 to 300 request evaluation set from real traffic.
3. Restructure prompts so stable content comes first, then enable and monitor caching.
4. Test a smaller model per request type; route only where the evaluation set supports it, with escalation on failure.
5. Set output length guidance and hard limits per feature.
6. Move every non-interactive job to a batch API.
7. Audit retrieval depth, conversation history and tool outputs for context bloat.
8. Review cost per successful task monthly, alongside quality metrics, not instead of them.

If you are earlier in the process and still working out what an AI feature will cost before you build it, our cost breakdown for an AI claims intake agent shows how to estimate build and run costs together, and a scoped AI proof of concept is a cheap way to get real token numbers before you commit to an architecture.

The bottom line

A high LLM bill is usually a design problem, not a pricing problem. Caching, routing, shorter outputs and batching are well-documented, provider-supported features, and together they routinely remove more than half of the spend in a product that was built prototype-first. The discipline that makes them safe is measurement: cost per task, an evaluation set, and a rule that no optimisation ships without passing it.

If your AI feature works but the unit economics do not, Northell's AI development team can audit where the tokens go and rebuild the pipeline around the numbers. Tell us what you are running, and we will tell you plainly which of these steps are worth doing first.