Prompt caching

Prompt caching lets the model provider reuse the processed form of a large, stable prompt prefix (system prompts, tool definitions, long documents) across requests. Repeat requests that hit the cache start faster and bill the cached portion at a discounted cache-read rate instead of the full input rate.

Inferel’s router is additionally KV-cache-aware: requests that share a prompt prefix are routed to the same upstream so the provider-side cache stays warm across your conversation turns.

Tip

Send a stable prompt_cache_key (or user) on requests that share a prefix — that value is what pins them to the same upstream. Without it each request is routed independently and a warm cache is a coincidence.

Using caching on Anthropic models

On anthropic/* models, mark the prompt blocks you want cached with cache_control via the Anthropic-compatible messages endpoint:

curl https://inference.inferel.ai/v1/messages \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $INFEREL_API_KEY" \
  -d '{
    "model": "claude-opus-4.8",
    "max_tokens": 512,
    "system": [
      {
        "type": "text",
        "text": "<large stable system prompt or document>",
        "cache_control": {"type": "ephemeral"}
      }
    ],
    "messages": [{"role": "user", "content": "First question..."}]
  }'

The first request writes the cache; subsequent requests with the same prefix read it. The response’s usage object breaks the input down:

{
  "usage": {
    "input_tokens": 8,
    "cache_creation_input_tokens": 6764,
    "cache_read_input_tokens": 0,
    "output_tokens": 4
  }
}

On Anthropic models, input_tokens counts only the non-cached input; cache writes and reads are reported separately.

OpenAI-style models

On OpenAI-compatible models (GPT, Gemini, and most others), caching is automatic where the upstream supports it — no request changes needed. Cached tokens are reported inside the normal prompt count:

{
  "usage": {
    "prompt_tokens": 6772,
    "completion_tokens": 4,
    "prompt_tokens_details": { "cached_tokens": 6764 }
  }
}

Here prompt_tokens includes the cached tokens; cached_tokens says how many of them were served from cache.

How cached tokens are billed

Each model has up to four per-1M-token rates: input, output, cache read, and cache write. Cache rates are listed with the model’s other prices in the console and returned as input_cache_read / input_cache_write by GET /v1/models. Billing follows the usage semantics above:

Token kind	Anthropic models	OpenAI-style models
Non-cached input	input_tokens × input rate	(prompt_tokens − cached_tokens) × input rate
Cache read	cache_read_input_tokens × cache-read rate	cached_tokens × cache-read rate
Cache write	cache_creation_input_tokens × cache-write rate	—
Output	output_tokens × output rate	completion_tokens × output rate

Typical cache rates follow the upstream convention — cache reads at 0.1× the input rate and cache writes at 1.25× the input rate. For example, claude-opus-4.8 at $15 input / $75 output per 1M bills cache reads at $1.50 and cache writes at $18.75 per 1M.

Tip

A prompt prefix must be reused to pay off: one cache write costs slightly more than plain input, and every subsequent read of it costs ~10× less. Caching large stable prefixes that you reuse across turns (agent system prompts, tool schemas, reference documents) routinely cuts input spend by 5–10×.

Models without published cache rates bill cached tokens at the regular input rate on OpenAI-style models; per-organization discounts apply to cache rates the same way they apply to input/output rates.