Skip to main content

Streaming and SSE

Verdictan supports text response streams through the public HTTP request families. Clients enable a stream when they send "stream": true in the request body.

Before you test, start a gateway with an agent binding, a provider credential, and a model that supports streaming. Use a different application request token for the call.

Public route families​

Use streaming with the public text-family routes documented in Runtime Request Families:

  • POST /v1/chat/completions
  • POST /v1/responses
  • POST /v1/messages

Minimal provider config​

pack:
  name: streaming-sse
  version: 0.1.0
  enabled: true
providers:
  targets:
  - id: openai
    provider: openai
    model: your-streaming-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Example request​

Use an application API token, not the connected gateway's runtime token:

export VERDICTAN_REQUEST_TOKEN="vdt_..."

curl -N http://127.0.0.1:41002/v1/chat/completions \
  -H "Authorization: Bearer ${VERDICTAN_REQUEST_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-streaming-model",
    "stream": true,
    "messages": [{"role": "user", "content": "Hello"}]
  }'

-N disables cURL output buffering so cURL shows each SSE frame when it receives it. For Chat Completions, a completed stream ends with the provider's terminal event, which can be data: [DONE]. Capture the HTTP status and the body when you investigate a stream:

curl -N --fail-with-body \
  -D stream-headers.txt \
  http://127.0.0.1:41002/v1/chat/completions \
  -H "Authorization: Bearer ${VERDICTAN_REQUEST_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"model":"your-streaming-model","stream":true,"messages":[{"role":"user","content":"Reply with one short sentence."}]}'

Practical guidance​

  • A usual provider target and a streaming upstream model are sufficient gateway config.
  • Chat Completions, Responses, and Messages use the documented access, input, tool, output, and audit stages. Input policies run before upstream dispatch. Output policies can make the gateway buffer content. They can also stop a stream with a terminal policy error.
  • Provider fallback can select a different target before a stream starts. It does not start the stream again on a different target after it sends chunks to the client.

Verification and troubleshooting​

Tail decision events in a second terminal while sending one identifiable request:

verdictan events tail --since 10m --follow
  • 401 or 403 before the first frame: verify the application token, not the gateway runtime or provider credential.
  • A non-streaming JSON response: make sure that the request body contains "stream": true and the selected model supports streaming.
  • Headers followed by a terminal policy error before content: examine the Event policy results. An output control can use content received up to that time or stop delivery.
  • A stream can send some content before a network error. Do not infer that the gateway retried a different provider.
  • No matching Event: verify event delivery, organization, region, and time filters before concluding that the request bypassed the gateway.

Next steps​