Provider configuration

The providers: section specifies upstream LLM targets, routes, retry operation, data handling, circuit breakers, and provider groups.

Before you start the gateway, replace each your-...-model placeholder with an active model ID for that provider account.

Reference

pack:
  name: config-providers-providers-1
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Provider targets

Each target is one upstream endpoint. When it uses the ordered route strategy, the gateway evaluates targets in their specified sequence.

Minimal target

pack:
  name: config-providers-providers-2
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

The schema specifies only id and provider as necessary fields. A provider alias can supply default values for base_url, the API key header, and the request path.

It does not make a credential or secret_key_ref. An ordinary target that must authenticate is not active without a resolvable credential.

Full target reference

pack:
  name: config-providers-providers-3
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    model: your-openai-model
    base_url: https://api.openai.com
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Credential resolution

An ordinary target must have a resolvable credential before authenticated dispatch.

Without a credential, the target is not active. Providers with optional or self-contained credentials use their documented authentication method.

For each ordinary target that you want active, set a specified secret_key_ref value. Remove targets until their configuration is complete.

pack:
  name: config-providers-credentials
  version: 1.0.0
  enabled: true
providers:
  routing:
    strategy: ordered
  targets:
  - id: openai
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
  - id: azure
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: VERDICTAN_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

The gateway examines subsequent eligible targets without a YAML fallback switch.

When the request must not move, set providers.routing.allow_fallbacks: false. You can also use a provider pin or one eligible target.

Data policies

Record the data terms from your provider contract. The data-routing-policy uses this customer-declared metadata to filter targets. Keep evidence for the provider contract in its owning workflow.

pack:
  name: config-providers-providers-4
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-zdr
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
    data_policy:
      zero_data_retention: true
      training_opt_out: true
      retention_days: 0
  - id: azure-standard
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: VERDICTAN_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
    data_policy:
      zero_data_retention: false
      training_opt_out: true
      retention_days: 30
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Token pricing

Provider targets can contain a pricing block for spend tracking and routing scores. Set input_price_per_million, cached_input_price_per_million, and output_price_per_million from your active provider contract. Do not copy prices from an example.

Nested models

One target can be for multiple models. Each model can have its own aliases, prices, and escalation routes.

pack:
  name: config-providers-providers-6
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    base_url: https://api.openai.com
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
    models:
      - model_id: your-openai-model
        aliases: [fast]
      - model_id: your-second-openai-model
        aliases: [reasoning]
        escalation_routing:
          team_id: 11111111-1111-4111-8111-111111111111
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Escalation route override

Send escalations to specified teams or users for each provider or model.

pack:
  name: config-providers-providers-7
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: healthcare-openai
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
    escalation_routing:
      team_id: 11111111-1111-4111-8111-111111111111
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Health probes

Active health probes run in the background. They record each probe result as the active provider health status.

pack:
  name: config-providers-providers-8
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
    health_probe:
      endpoint: https://health.example.com/ready
      interval_seconds: 30
      timeout_ms: 2000
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Health probes send a bare GET. They do not add the target credential or custom provider headers.

Set endpoint to an operator-controlled 2xx readiness route that operates without authentication.

Use the default {base_url}/health only when that upstream gives a public health endpoint.

OAuth2 authentication

Use this structure for providers that must use OAuth2 bearer tokens, such as Databricks, Snowflake Cortex, and custom endpoints.

pack:
  name: config-providers-providers-9
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: databricks-prod
    provider: databricks
    model: your-databricks-model
    base_url: https://my-workspace.databricks.net
    oauth2:
      grant_type: client_credentials
      token_endpoint: https://accounts.example.com/oauth2/token
      client_id: verdictan-gateway
      client_secret_env: VERDICTAN_DATABRICKS_CLIENT_SECRET
      scopes: [all-apis]
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Cloud provider fields

Azure OpenAI
AWS Bedrock
Google Vertex AI
Anthropic

pack:
  name: config-providers-providers-10
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: azure-primary
    provider: azure-openai
    model: your-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: VERDICTAN_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

pack:
  name: config-providers-providers-11
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: bedrock-claude
    provider: aws-bedrock
    provider_type: aws-bedrock
    format: anthropic
    model: anthropic.replace-with-bedrock-model-id
    base_url: https://bedrock-runtime.us-east-1.amazonaws.com
    aws_region: us-east-1
    bedrock_model_family: anthropic_messages
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Use an AWS credential chain that can call the selected model. The endpoint, region, and model family must agree.

pack:
  name: config-providers-providers-12
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: vertex-gemini
    provider: google-vertex
    model: your-gemini-model
    gcp_project: my-gcp-project
    gcp_region: us-central1
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

pack:
  name: config-providers-providers-13
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: anthropic-claude
    provider: anthropic
    model: your-anthropic-model
    base_url: https://api.anthropic.com
    secret_key_ref:
      env: VERDICTAN_ANTHROPIC_API_KEY
    provider_type: anthropic
    format: anthropic
    anthropic_version: your-supported-anthropic-version
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Before production use, validate the provider request type, authentication method, and failure operation.

Routing strategies

The routing.strategy field controls how the gateway selects a target for each request.

providers:
  routing:
    strategy: ordered
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY

Strategy	Description
`ordered`	The gateway tries targets in their specified sequence. The first healthy target wins.
`round_robin`	The gateway selects each target in turn.
`weighted_round_robin`	The gateway uses each target `weight` value during its cycle.
`random`	The gateway selects a target at random.
`simple_shuffle`	The gateway shuffles targets and then reads the list.
`lowest_latency`	The gateway ranks sampled targets by the observed P50 time to first token.
`highest_throughput`	The gateway ranks sampled targets by the observed P50 throughput.
`least_connections`	The gateway selects the target with the fewest active requests.
`least_busy`	The gateway selects the target with the lowest load at selection time.
`usage_based`	The gateway selects the target with the lowest total use.
`semantic`	The gateway compares the request embedding with target descriptions.

Semantic routes change the specified sequence only when the runtime can resolve an embedding target.

The runtime uses routing.semantic_embedding_provider or the configured cache embedding provider. A minimum of one candidate target must have a nonempty description.

The runtime applies semantic_similarity_threshold. If all targets are below the limit, the initial candidate sequence stays unchanged.

A missing embedding provider or description also keeps the initial sequence.

Region and quantization filtering

providers:
  routing:
    strategy: ordered
    require_region: eu
    require_quantizations:
    - fp16
    - int8
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    region: eu
    quantizations: [fp16, int8]
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY

require_region is fail-closed. If no target matches the configured region, the gateway blocks the request.

There is no fallback option across regions.

Target filtering

providers:
  routing:
    strategy: round_robin
    only:
    - openai-primary
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY

Provider attempts and fallback

The gateway puts eligible targets in sequence and retries a target according to providers.retry_policy. Then, it examines subsequent targets.

A transport error moves the request to the next eligible target. An HTTP response causes movement only when its classification is in the runtime trigger list.

A response that succeeds stops the loop.

Active runtime classifications are:

rate_limit for HTTP 429
server_error for HTTP 408, 500, 502, 503, and 504
server_error for transport timeouts

The gateway returns other upstream responses directly. These responses include content filter failures and context window failures.

To use only the first eligible target, set providers.routing.allow_fallbacks: false. You can also pin the request to a provider. Test each request family because a streamed response cannot move after client output starts.

Context window compression

Set max_context_tokens on a provider target to enable the active middle-out compression path for requests that exceed that limit.

Zero-completion handling

For buffered POST /v1/chat/completions responses, enable the built-in zero-completion check with zero_completion_insurance.enabled.

When its built-in detector finds a zero completion, the gateway always continues to the next eligible provider.

providers:
  zero_completion_insurance:
    enabled: true

Do not enable this control when a replay to the next eligible provider is not approved.

Model groups

Put targets into model groups with aliases and fallback chains.

providers:
  model_groups:
    - name: "fast-models"
      aliases: ["fast", "quick"]
      description: "Low-latency models for real-time use"
      targets: ["fast-groq", "fast-openai"]
      fallback_group: "standard-models"
    - name: "standard-models"
      aliases: ["standard", "default"]
      targets: ["standard-openai", "standard-azure"]

fallback_group is an active runtime field. It adds that group target IDs after the active group targets in the ordered candidate chain.

The standard provider attempt rules control movement through the chain.

Provider pipelines

Put multiple provider targets behind one virtual model name. Use mode: sequence when one model output must be the input for the next step.

Use mode: fan_out to get answers from multiple models and merge outputs from models that succeed.

pack:
  name: config-providers-providers-25
  version: 1.0.0
  enabled: true
providers:
  pipelines:
  - name: draft-and-review
    aliases: [reviewed-answer]
    mode: sequence
    steps:
    - name: draft
      target: writer
    - name: review
      target: reviewer
      instruction: Review and improve the draft.
      input_mode: replace
      inject_as: user
  targets:
  - id: writer
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
  - id: reviewer
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: VERDICTAN_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Pipeline rules:

The usual request model field starts a pipeline, as it does for model_groups.
Each step must reference a target with a specified providers.targets[].model.
For different models from one provider, declare a different target for each model.
After the first sequence step, a step can add the previous output to the initial request with append.
A sequence step can use replace to replace the initial request with the previous output.
fan_out runs each step against the initial request. It makes one output from step outputs in sequence or returns the first output from a model that succeeds.
At this time, pipelines support POST /v1/chat/completions and POST /v1/responses. They do not support embeddings or moderations.
The gateway rejects direct pipeline model pins with X-Verdictan-Provider or X-Verdictan-Model.

Circuit breaker

The circuit breaker automatically removes unhealthy targets from the pool.

providers:
  circuit_breaker:
    enabled: true
    consecutive_failure_threshold: 5
    cooldown_seconds: 30
    half_open_successes: 1

Retry policy

Configure the retry policy.

providers:
  retry_policy:
    max_retries: 3
    per_trigger:
      rate_limit: 5
    backoff:
      strategy: "exponential"        # exponential | fixed
      base_ms: 500

These retries use the same provider before the outer provider loop can continue.

Active dispatch reads trigger limits for rate_limit and server_error. Transport failures use max_retries.

Failures that occur when dispatch reads a response body also use max_retries.

Logging controls

providers:
  logging:
    redact_message_bodies: true

The gateway applies redact_message_bodies to payload logs.

Provider aliases and authentication

The gateway includes more than 100 provider aliases. An alias can supply base_url, api_key_header, and path_template.

It does not supply credentials. The schema makes secret_key_ref optional, but an ordinary target must have a resolvable reference for activation.

Targets with optional or self-contained credentials use their provider guidance.

An alias, provider profile, or wire format translator proves only that the gateway can load that configuration path.

It does not prove a complete production request contract. Before a rollout, check the provider guide in the Integrations section.

The guide gives the active runtime status and the necessary verification steps.

Provider	Aliases	Auth
OpenAI	`openai`	`VERDICTAN_OPENAI_API_KEY`
Anthropic	`anthropic`, `claude`	`VERDICTAN_ANTHROPIC_API_KEY`
Google AI Studio	`google`, `gemini`	`VERDICTAN_GEMINI_API_KEY`
Google Vertex AI	`google-vertex`, `vertex`	Service account or OAuth2
Azure OpenAI	`azure-openai`, `azure`	`VERDICTAN_AZURE_OPENAI_API_KEY`
AWS Bedrock	`aws-bedrock`, `bedrock`	SigV4 with `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` and optional `AWS_SESSION_TOKEN`
Groq	`groq`	`VERDICTAN_GROQ_API_KEY`
Mistral	`mistral`	`VERDICTAN_MISTRAL_API_KEY`
DeepSeek	`deepseek`	`VERDICTAN_DEEPSEEK_API_KEY`
Together AI	`togetherai`, `together`	`VERDICTAN_TOGETHER_API_KEY`
Fireworks AI	`fireworks`, `fireworks-ai`	`VERDICTAN_FIREWORKS_API_KEY`
Cerebras	`cerebras`	`VERDICTAN_CEREBRAS_API_KEY`
Perplexity	`perplexity`	`VERDICTAN_PERPLEXITY_API_KEY`
OpenRouter	`openrouter`	`VERDICTAN_OPENROUTER_API_KEY`
GitHub Models	`github`, `github-models`	`VERDICTAN_GITHUB_TOKEN`
Cohere	`cohere`	`VERDICTAN_COHERE_API_KEY`
HuggingFace	`huggingface`, `hf`	`VERDICTAN_HF_API_TOKEN`
Replicate	`replicate`	`VERDICTAN_REPLICATE_API_TOKEN`
Databricks	`databricks`	OAuth2 or `VERDICTAN_DATABRICKS_TOKEN`
Snowflake Cortex	`snowflake-cortex`	OAuth2
Cloudflare AI	`cloudflare-ai`	`VERDICTAN_CF_API_TOKEN`
Cloudflare AI Gateway	`cloudflare-gateway`	The gateway uses the underlying provider credential. The OpenAI BYOK example uses `VERDICTAN_OPENAI_API_KEY`.
Alibaba or Qwen	`alibaba`, `qwen`, `dashscope`	`VERDICTAN_DASHSCOPE_API_KEY`
SambaNova	`sambanova`	`VERDICTAN_SAMBANOVA_API_KEY`
xAI (Grok)	`xai`, `grok`	`VERDICTAN_XAI_API_KEY`
Vercel AI Gateway	`vercel`, `vercel-ai`	`VERDICTAN_VERCEL_AI_GATEWAY_API_KEY`

Wire format translation

The gateway has translators for the listed wire formats. It can get a format from a recognized provider profile.

Set format explicitly when the provider guide specifies it. A translator does not prove that all provider fields or request structures can be used in production.

Format	Providers
`openai`	OpenAI, Azure, Groq, Mistral, Together, Fireworks, most OpenAI-compatible
`anthropic`	Anthropic, Claude
`cohere`	Cohere
`huggingface`	HuggingFace Inference Endpoints
`replicate`	Replicate
`watsonx`	IBM watsonx
`google-gemini`	Google Gemini (AI Studio and Vertex)

Complete provider example

pack:
  name: openai-and-azure-routing
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
  - id: azure-openai-fallback
    provider: azure-openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: VERDICTAN_AZURE_OPENAI_API_KEY
    azure_deployment: your-azure-deployment
    azure_api_version: your-supported-api-version
  routing:
    strategy: ordered
  circuit_breaker:
    enabled: true
    consecutive_failure_threshold: 5
    cooldown_seconds: 30
  retry_policy:
    max_retries: 3
    backoff:
      strategy: exponential
      base_ms: 500
policies:
  chain:
  - prompt-injection
  - pii-detector
  - quality-scorer
policy:
  prompt-injection:
    response:
      action: block
  pii-detector:
    action: redact
  quality-scorer:
    thresholds:
      min_aggregate: 0.8

In this example, retry_policy controls retries for the same target. After the retry budget is empty, the usual provider loop can examine the next eligible target.

The loop uses its default trigger list. It operates without a specified fallback block.

Reference​

Provider targets​

Minimal target​

Full target reference​

Credential resolution​

Data policies​

Token pricing​

Nested models​

Escalation route override​

Health probes​

OAuth2 authentication​

Cloud provider fields​

Routing strategies​

Region and quantization filtering​

Target filtering​

Provider attempts and fallback​

Context window compression​

Zero-completion handling​

Model groups​

Provider pipelines​

Circuit breaker​

Retry policy​

Logging controls​

Provider aliases and authentication​

Wire format translation​

Complete provider example​

Next steps​