Chat completions

POST https://inference.inferel.ai/v1/chat/completions

Creates a model response for a chat conversation. The request and response formats follow the OpenAI Chat Completions API, so official OpenAI SDKs work unmodified.

Request body

Parameter	Type	Required	Description
model	string	yes	Model ID, e.g. anthropic/claude-sonnet-4.6. See List models. Optionally suffix a resource group, e.g. model:pool.
messages	array	yes	Conversation messages: {"role": "system" | "user" | "assistant", "content": "..."}.
stream	boolean	no	Stream the response as server-sent events. Default false.
temperature	number	no	Sampling temperature (0–2).
max_tokens	integer	no	Maximum tokens to generate.
top_p	number	no	Nucleus sampling.
stop	string / array	no	Stop sequences.

Other OpenAI parameters (e.g. tools, response_format, frequency_penalty) are passed through to the underlying model; support depends on the model you select.

Example

curl https://inference.inferel.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $INFEREL_API_KEY" \
  -d '{
    "model": "anthropic/claude-sonnet-4.6",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is an inference platform?"}
    ],
    "temperature": 0.7,
    "max_tokens": 300
  }'

Response:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1751000000,
  "model": "anthropic/claude-sonnet-4.6",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "An inference platform runs trained AI models and serves their predictions over an API..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 28,
    "completion_tokens": 74,
    "total_tokens": 102
  }
}

Streaming

With stream: true the response is delivered as server-sent events (Content-Type: text/event-stream), one data: line per chunk, terminated by data: [DONE].

Inferel always includes a final usage chunk on streamed requests (equivalent to setting stream_options: {"include_usage": true}), so streamed calls report exact token usage for billing:

data: {"choices":[{"delta":{"content":"An"}}], ...}
data: {"choices":[{"delta":{"content":" inference"}}], ...}
...
data: {"choices":[],"usage":{"prompt_tokens":28,"completion_tokens":74,"total_tokens":102}}
data: [DONE]

Reliability

Inferel routes each request across multiple upstream providers and automatically retries transient upstream failures (rate limits and 5xx responses) on alternative endpoints before returning an error. Requests with a shared conversation prefix are routed KV-cache-aware to the same upstream where possible, improving latency and cache hit rates on long conversations.