Chat completions

POST https://inference.inferel.ai/v1/chat/completions

Creates a model response for a chat conversation. The request and response formats follow the OpenAI Chat Completions API, so official OpenAI SDKs work unmodified.

Request body

Parameter	Type	Required	Description
model	string	yes	Model ID, e.g. claude-sonnet-4.6. See List models. Optionally suffix a resource group, e.g. model:pool.
messages	array	yes	Conversation messages: {"role": "system" | "user" | "assistant", "content": "..."}.
stream	boolean	no	Stream the response as server-sent events. Default false.
temperature	number	no	Sampling temperature (0–2).
max_tokens	integer	no	Maximum tokens to generate.
top_p	number	no	Nucleus sampling.
stop	string / array	no	Stop sequences.
stream_options	object	no	{"include_usage": true} adds a final usage chunk to a streamed response.
prompt_cache_key	string	no	Groups requests that share a prompt prefix so they route to the same upstream and reuse its cache. Falls back to user. See Prompt caching.
user	string	no	End-user identifier; also used as the cache-affinity key when prompt_cache_key is absent.

Other OpenAI parameters (e.g. tools, response_format, frequency_penalty) are passed through to the underlying model; support depends on the model you select.

Example

curl https://inference.inferel.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $INFEREL_API_KEY" \
  -d '{
    "model": "claude-sonnet-4.6",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is an inference platform?"}
    ],
    "temperature": 0.7,
    "max_tokens": 300
  }'

Response:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1751000000,
  "model": "claude-sonnet-4.6",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "An inference platform runs trained AI models and serves their predictions over an API..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 28,
    "completion_tokens": 74,
    "total_tokens": 102
  }
}

Streaming

With stream: true the response is delivered as server-sent events (Content-Type: text/event-stream), one data: line per chunk, terminated by data: [DONE].

Set stream_options: {"include_usage": true} to receive a final usage chunk so a streamed call reports its exact token counts. Billing is metered from the same counts whether or not you ask for the chunk:

data: {"choices":[{"delta":{"content":"An"}}], ...}
data: {"choices":[{"delta":{"content":" inference"}}], ...}
...
data: {"choices":[],"usage":{"prompt_tokens":28,"completion_tokens":74,"total_tokens":102}}
data: [DONE]

Reliability

Inferel automatically retries transient failures (rate limits and 5xx responses) on alternative capacity before returning an error. Requests with a shared conversation prefix are routed KV-cache-aware to the same upstream where possible, improving latency and cache hit rates on long conversations.