Cost Optimization for LLM Applications in Production

An LLM application costs money every time it runs, in proportion to the tokens it reads and writes. That makes cost unusually controllable: every token you avoid is money saved, and many applications send the same instructions thousands of times, generate answers twice as long as needed, and use their largest model for their easiest requests.

This guide breaks the cost into its parts, measures two of the biggest levers on a local model, then prices five configurations for a month of traffic with published list rates.

Where the money goes

A request has three parts, priced differently:

On OpenAI's price list on 3 October 2026, its flagship gpt-6-astra cost $10 per million input tokens and $50 per million output tokens, five times as much (OpenAI: pricing). Before optimising, log the token counts of real requests and see which part dominates.

Two levers, measured

The script first runs two experiments on a local model, then turns the findings into a monthly bill.

import json
import urllib.request

# ---- Part 1: two levers measured on a local model -------------------------------------------
HANDBOOK = open("handbook.txt", encoding="utf-8").read()     # a long, fixed system prompt


def chat(system, question, **options):
    body = {"model": "llama3.2", "stream": False,
            "options": {"temperature": 0, "seed": 42, "num_ctx": 4096, **options},
            "messages": [{"role": "system", "content": system}, {"role": "user", "content": question}]}
    req = urllib.request.Request("http://localhost:11434/api/chat", json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=900) as r:
        return json.load(r)


print("same long system prompt, three different questions in a row")
for question in ["How long do I have to return an item?", "How much is delivery?", "When is support open?"]:
    r = chat(HANDBOOK, question, num_predict=40)
    print(f"   prompt tokens processed {r['prompt_eval_count']:>5}   "
          f"prompt processing took {r['prompt_eval_duration'] / 1e9:.1f} seconds")

print("\nthe same question, with and without a length instruction")
for label, system in [("no instruction", "You are a helpful assistant."),
                      ("'one sentence'", "You are a helpful assistant. Answer in one sentence.")]:
    r = chat(system, "Why does prompt caching reduce the cost of an LLM application?")
    print(f"   {label:<15} {r['eval_count']:>4} output tokens")

# ---- Part 2: a monthly cost model with published list prices -------------------------------
# US dollars per million tokens, from OpenAI's pricing page on 3 October 2026
PRICES = {"gpt-6-astra": {"input": 10.00, "cached": 1.00, "write": 12.50, "output": 50.00},
          "gpt-6-luna": {"input": 0.10, "cached": 0.01, "write": 0.125, "output": 0.50}}
REQUESTS = 100_000                      # a month of a support assistant
PREFIX, VARIABLE, OUTPUT = 3_000, 1_500, 300   # tokens: fixed instructions, question plus context, answer
HIT_RATE = 0.9                          # share of requests that find the prefix already cached


def monthly(model, cache=False, output=OUTPUT, batch=False):
    p = PRICES[model]
    if cache:
        prefix_cost = PREFIX * (HIT_RATE * p["cached"] + (1 - HIT_RATE) * p["write"])
    else:
        prefix_cost = PREFIX * p["input"]
    per_request = (prefix_cost + VARIABLE * p["input"] + output * p["output"]) / 1e6
    return REQUESTS * per_request * (0.5 if batch else 1)


scenarios = [("large model, no caching", monthly("gpt-6-astra")),
             ("large model, cached prefix", monthly("gpt-6-astra", cache=True)),
             ("large model, cached, answers halved", monthly("gpt-6-astra", cache=True, output=150)),
             ("small model, cached prefix", monthly("gpt-6-luna", cache=True)),
             ("large model, cached, batch (offline work)", monthly("gpt-6-astra", cache=True, batch=True))]
base = scenarios[0][1]
print(f"\n{REQUESTS:,} requests a month: {PREFIX:,} fixed + {VARIABLE:,} variable input tokens, {OUTPUT} output")
for name, cost in scenarios:
    print(f"   {name:<42} ${cost:>9,.0f}   {cost / base:>6.1%} of the first")

same long system prompt, three different questions in a row
   prompt tokens processed   551   prompt processing took 22.3 seconds
   prompt tokens processed   546   prompt processing took 0.4 seconds
   prompt tokens processed   546   prompt processing took 0.4 seconds

the same question, with and without a length instruction
   no instruction   277 output tokens
   'one sentence'    43 output tokens

100,000 requests a month: 3,000 fixed + 1,500 variable input tokens, 300 output
   large model, no caching                    $    6,000   100.0% of the first
   large model, cached prefix                 $    3,645    60.8% of the first
   large model, cached, answers halved        $    2,895    48.2% of the first
   small model, cached prefix                 $       36     0.6% of the first
   large model, cached, batch (offline work)  $    1,823    30.4% of the first

What the output shows

• Caching removes repeated work. The first request processed the long system prompt in 22.3 seconds. The next two, with the same prompt and different questions, took 0.4 seconds each, because the model reused the work already done on the shared prefix. Hosted providers turn that saving into a discount.
• Asking for brevity cut output from 277 tokens to 43. Since output is the expensive part, a one-line instruction is often the cheapest optimisation there is.
• Caching the prefix cut the modelled bill to 60.8%. With 3,000 fixed tokens per request and a 90% hit rate, $6,000 a month became $3,645.
• Halving answer length took it to 48.2%.
• A small model with caching cost $36, 0.6% of the original. That is only a saving if the small model is good enough for the task, which your evaluation set has to show.
• Batch processing halved the cached cost again, for work that does not need an immediate answer.

The levers

Prompt caching. OpenAI caches automatically on supported models: reused input is discounted by up to 95% and starts responding sooner. On recent models, prefixes need at least 1,024 tokens, a cache write costs 1.25 times the input rate, a read 0.1 times, and entries stay usable for 30 minutes after their last use (OpenAI: prompt caching). Anthropic's rates are similar: writes at 1.25 times and reads at 0.1 times the base price, with a 5-minute cache refreshed on each use and a 1-hour option at extra cost (Anthropic: prompt caching). To benefit, put stable content first and variable content last, and keep the stable part identical between requests.

Output control. Ask for the length you need, set a maximum output limit as a backstop, and use structured output when code reads the answer.

Model routing. Send simple, high-volume requests to a small model and reserve the large one for hard cases. A cheap classifier or rules can decide. Measure quality per route.

Context trimming. Retrieve fewer, smaller passages, and summarise long histories. In a comparison of chunking settings, the same answers were found with a fifth of the text. See LlamaIndex vs LangChain and context windows explained.

Batching. Work that can wait, such as classifying a backlog, summarising documents overnight or running evaluations, can go through batch APIs. OpenAI's costs 50% less and completes within 24 hours (OpenAI: Batch API). Anthropic's Batch API also gives 50% off input and output tokens (Anthropic: pricing).

Response caching. Identical questions, such as the same FAQ asked by thousands of users, can be answered from your own cache without calling the model at all.

Self-hosting. At high, steady volume, running an open model on your own GPUs can cost less per token. See vLLM and model serving and open-weight vs closed LLMs.

Protect quality, then protect the budget

Every saving changes what the model sees or which model answers, so run the evaluation set before and after each change. A cheaper configuration that answers worse is not a saving. See evaluating LLM outputs.

Then add limits: per-user and per-day caps, rate limits, alerts on spending, and step limits for agents, whose loops multiply calls. See AI agents explained.

Common mistakes

• Optimising without measuring which part of the request costs most.
• Putting variable content first, which breaks prefix caching.
• Unlimited output length.
• One large model for every request.
• Ignoring batch pricing for offline work.
• Cutting cost without re-running the evaluation set.

Cost and latency tuning is part of the Evaluation and Optimization module of the Generative AI course.

Sources

• Pricing (OpenAI API documentation) — OpenAI
• Prompt caching (OpenAI API documentation) — OpenAI
• Batch API (OpenAI API documentation) — OpenAI
• Prompt caching (Claude Platform Docs) — Anthropic
• Pricing (Claude Platform Docs) — Anthropic