How to Cut LLM API Costs Without Degrading Output Quality

Enterprise LLM spend jumped from $3.5 billion to $8.4 billion in the first half of 2025 alone, according to Menlo Ventures. Token prices for a GPT-3.5-level system fell more than 280-fold between late 2022 and late 2024, per Stanford HAI's 2025 AI Index. So the bills should be shrinking, not growing. They're not, because agentic workflows loop, chain, and re-send the same context over and over, and that's a consumption problem. Consumption problems get solved by engineering teams, not by waiting on a provider to cut rates, and that distinction is the whole point of this piece.

Flexera's 2026 AI Pulse Report found 80% of leaders increased AI investment, and 36% admit they've overspent. Cost unpredictability sits in the top three challenges organizations face when scaling AI. That's not a rare misstep. Most teams treat it as one anyway, as if overspend were a surprise instead of the default outcome of shipping agentic workflows without watching what they cost per step.

How to think about the four levers before touching any of them

Four decisions drive almost all of what a production LLM system spends: which model handles a given request, whether a past answer can just be reused, whether the request can wait a few hours instead of running right now, and how many tokens the request actually needs to carry.

Most teams reach for the wrong one first. They start trimming prompts, because it feels like the safe, low-risk move, when routing and caching are sitting right there with the bigger payoff. Prompt trimming and batching are real, but they're refinements you layer on after the two big levers are already banked, not a starting point.

The levers don't apply evenly across every workload, either. A real-time chat feature answers to different constraints than a nightly batch job, and a RAG pipeline behaves nothing like a single-turn support bot. So pick the lever that fits whichever workload is burning the most money right now, not the one that sounds most sophisticated on a slide.

Quality risk isn't symmetric across these levers. Done right, routing and caching preserve output quality for the end user. Done carelessly, they create quality regressions that don't surface as error codes and can be difficult to trace back to the optimization that caused them.

A gateway or proxy layer sitting between the application and the model providers is where most of this actually gets enforced. Set the routing rules, caching policy, and batching logic once at that layer, and every service behind it inherits the savings without a rewrite. Start with whichever lever addresses the highest-cost workload, get a baseline on cost and quality, then add levers one at a time. Not all four at once, no matter how tempting that feels once the first one works.

Model routing: sending each request to the cheapest model that can handle it

A large share of production requests don't need a frontier model at all. Classification, basic Q&A, formatting, simple lookups: these get routed to the flagship model out of habit, not necessity, and that habit is where most of the wasted spend actually lives.

The price gap makes the case on its own. Claude Haiku 4.5 runs $1 per million input tokens. Claude Opus 4 runs $15 per million input tokens for the same classification task, a 15x difference before a single other optimization gets applied. Route 70% of traffic to the cheaper model and keep 30% on the frontier model, and the input-token bill drops by roughly two-thirds. Push an 80/20 split toward an even cheaper tier, and savings approach 79%.

Three approaches, in order of how much engineering they demand. Rule-based tiers come first: define three or four complexity buckets by hand, ship it, measure what happens. Classifier-based routing comes next, using a small model, often BERT-sized, to predict how complex a query is before the expensive model ever sees it. Cascade routing is the most adaptive of the three: send everything to the cheap model first, and only escalate when confidence is low or the cheap model fails outright.

The research backs up how far this goes. RouteLLM, from UC Berkeley and Anyscale (ICLR 2025), cut costs by 85% on MT Bench while holding onto 95% of GPT-4's performance, sending only 14% of queries to the strong model. Stanford's FrugalGPT work found up to 98% cost reduction versus always defaulting to the best API, at equivalent output quality.

Start with rule-based tiers. Skip straight to a classifier only once the rule-based version has run long enough to show where it's actually failing, because building a classifier before that is solving a problem nobody's measured yet. And evaluation coverage on routed traffic isn't optional: it's the only thing standing between routing and guessing.

Prompt caching: paying once for context you send on every request

The key-value cache stores the vectors for tokens the model has already processed, so a request sharing the same prefix as a prior one doesn't recompute from scratch. The model pulls those tokens from cache instead, at a fraction of the normal input price.

The economics are steep. Anthropic's prefix caching charges $0.30 per million tokens for cache reads against $3.00 per million for fresh input, a 90% discount. OpenAI's automatic caching, which runs by default, delivers 50% savings on cached input. There's a minimum prefix length to qualify: 4,096 tokens on Anthropic's Opus 4.6 and Haiku 4.5, 1,024 tokens on Sonnet 4.6, Opus 4.8, and Sonnet 5.

The agentic tax is where this stops being theoretical. ProjectDiscovery's agent swarm system prompt runs 2,547 lines. Without caching, every step of a multi-step task re-sends that entire prefix at full price, so a 40-step task means paying for a 20,000-token system prompt forty separate times over. Turn caching on, and that math changes fast: ProjectDiscovery's own numbers show a 59% cut in LLM costs against full input rates, climbing to 66% after further tuning, with the trailing ten-day average hitting 70%.

Flipping on caching isn't as simple as adding a cache_control header and walking away, either. Without deliberate prompt design and some care around how requests get scheduled, parallel or high-throughput workloads end up creating redundant caches and missing hits that should have landed clean. The fix is structural: stable, shared content, system instructions, document context, tool definitions, needs to sit at the front of the prompt, forming a consistent prefix the cache can actually recognize. Dynamic, user-specific content goes at the end, after the cacheable block. For teams running open-weight models on their own infrastructure, tools like LMCache extend this further, offloading and reusing KV caches across CPU, disk, and S3 for vLLM and SGLang workloads.

Batching: the 50% discount that requires only a scheduling decision

OpenAI and Anthropic both run asynchronous batch APIs that process requests inside a 24-hour window at roughly half of standard token pricing, on input and output alike. Stack that against prompt caching, and on the right workload the combined discount can be substantial.

Plenty of workloads qualify with zero redesign: nightly document classification, bulk summarization, embedding backfills, evaluation runs, dataset labeling, report generation. Anywhere no human is sitting there waiting on the response in real time.

This is an audit, not an engineering project. Which production calls are actually latency-sensitive, and which ones run synchronously simply because that's how someone first wired them up years ago and nobody's revisited it since? Batching also spreads fixed request overhead across many queries at once instead of paying setup costs per call, which adds up at volume even before the headline discount kicks in.

On the implementation side, batch APIs return results through polling or a webhook rather than an immediate response. That means wiring up result retrieval, but the change to the actual API call itself stays small.

Prompt engineering and output compression as a secondary multiplier

Every token in a system prompt gets billed at full price on every call that isn't cached. Cut a system prompt from 2,000 tokens down to 800, and that reduction pays off on every single request from then on, quietly, forever.

Output shape matters just as much as prompt length. Asking for structured output, JSON or a plain list, instead of prose can meaningfully cut output token count. Stack prompt trimming with context compaction on top of that, and it adds further savings on top of whatever routing and caching already captured.

Worth an actual audit here: system prompts accumulate redundant instructions, too many examples, and boilerplate left over from early prototyping that nobody ever cleaned out. Every token removed without hurting output quality was never worth paying for in the first place. "Respond in JSON with fields X, Y, Z" costs systematically less than "please provide a detailed explanation of," and that gap compounds across every call the prompt template touches. Cost and quality actually pull in the same direction here, which is rare enough to be worth pointing out: tighter prompts shrink the surface area where a model can hallucinate, so trimming isn't a tradeoff. It's a two-for-one.

RAG versus long-context stuffing as a token-volume decision

Deciding whether to feed a model an entire document corpus or retrieve the relevant chunks first isn't just about which produces a better answer. It's a token-volume decision, and most teams get it backwards by defaulting to long context because it's easier to build.

RAG is substantially cheaper than long-context approaches for typical workloads, with better latency to go with it. That range reflects how much workloads vary, but the direction holds consistently across them. RAG wins clearly when the underlying data is dynamic or spans a wide variety of sources: retrieval keeps only the relevant chunks in front of the model, so input token counts stay bounded no matter how large the corpus grows behind it.

Long-context models earn their place in exactly one scenario: long, static documents where the whole document genuinely matters, and chunking it up through retrieval would break the reasoning apart. That's a narrower case than most teams assume when they reach for it by default. Any team that started stuffing full documents into context the moment long-context windows became available should go back and check whether the workload actually needs that, or whether retrieval gets comparable quality for a fraction of the tokens. Most of the time, it does.

RAG and caching also stack well together. A stable system prompt combined with retrieved chunks that repeat often enough can be structured to clear the caching threshold, which puts retrieval savings directly on top of caching savings instead of choosing between them.

Measuring cost against quality so optimization doesn't become silent degradation

Routing a request to a cheaper model saves money, and it logs as a successful API call either way. Neither fact tells anyone whether the output still holds up. Churn and user complaints are lagging indicators. They show up only after something already broke, weeks after the routing change that caused it.

The number that actually matters is cost per outcome, not cost per token. A 50% cut in token spend that comes with a 20% drop in task completion isn't a win, no matter what the dashboard says. Before touching any lever, get three things in place first: a labeled evaluation set with expected outputs across complexity tiers, per-request cost attribution broken out by feature and model, and a quality metric, whether that's human review, LLM-as-judge, or a task-specific accuracy score, run against a sample of routed traffic.

Establish that baseline before optimizing anything, then move one lever at a time. Stack all four at once and there's no way to trace a quality shift back to whichever change actually caused it, which defeats the point of measuring in the first place. A gateway-level observability layer is the most durable place to build this, since one proxy sitting in front of every model call captures cost, latency, and cache-hit rate across the whole system without instrumenting each application separately.

That 36% of leaders who told Flexera they'd overspent on AI tend to share one thing: they didn't tie spend to outcome tracking early enough to catch it before it compounded. Once routing and caching go live, run the evaluation set on a weekly cadence. Model behavior shifts on the provider side too, quietly, and it can move quality independently of anything the team itself changed.

Putting the levers together: a sequenced rollout for production teams

Weeks one through four: roll out rule-based model routing on the highest-volume, lowest-complexity request type, and instrument a cost-and-quality baseline before doing anything else. That step alone typically targets 40% to 70% savings on the traffic it touches.

Month two: add prompt caching to any workload with a stable system prompt above the minimum threshold, 4,096 tokens for Opus and Haiku, 1,024 for the Sonnet variants. Restructure the prompt so stable content sits at the front, forming a consistent prefix. Alongside that, audit which calls run synchronously out of habit rather than genuine need, and move the eligible ones onto batch APIs for the flat 50% discount.

Month three: apply prompt trimming and output compression across the whole system prompt library, aiming for meaningful output-token reduction available through structured formats. Pair it with an architectural review: are any document Q&A or knowledge-base features stuffing full documents into long context where retrieval would deliver comparable quality at significantly lower token cost?

Stacked in the right order, on the right workload, routing, caching, and batching together can approach 95% off input token costs, the flat 50% batch discount compounding with caching discounts running as high as 90%, though Anthropic's own documentation only formally states the flat 50% figure on its own. The sequence matters because each layer shrinks the token volume the next layer has to work with.

None of this is a project with an end date. Per-token prices keep falling, and usage keeps growing faster than the price drops can offset. Measuring cost against quality, continually, is the only habit standing between a falling price curve and a bill that climbs anyway.