Rate limit configuration

Verdictan supports global, per-IP, per-user, and token use limits. It also supports byte limits for selected LLM proxy paths.

The active distributed backend shares only the global request counter. The IP, user, and token counters stay in each gateway process.

Reference

global_rate_limit:
  max_requests: 1000
  window_seconds: 60

ip_rate_limit:
  max_requests: 100
  window_seconds: 60

user_rate_limit:
  max_requests: 30
  window_seconds: 60
  header_names: ["x-user-id"]

token_rate_limit:
  max_tokens: 500000
  window_seconds: 3600
  scope: "global"

size_limits:
  max_body_bytes: 1048576
  max_response_bytes: 10485760

Global rate limit

Public proxy handlers that use the gateway request checks share one counter. Health, configuration, and administrative endpoints do not use this counter.

global_rate_limit:
  max_requests: 1000        # required
  window_seconds: 60         # required

Field	Type	Necessary	Default	Description
max_requests	integer	yes	—	This is the maximum request count for each window.
window_seconds	integer	yes	—	This is the constant window time in seconds.

Runtime operation: An atomic counter uses an epoch-based constant window. The counter starts at zero for each new window. A limit failure returns HTTP 429 Too Many Requests with a Retry-After header.

Per-IP rate limit

Each client IP address has its own counter.

ip_rate_limit:
  max_requests: 100          # required
  window_seconds: 60          # required
  trusted_proxy_cidrs:         # optional, default: []
    - 10.0.0.0/8

Field	Type	Necessary	Default	Description
max_requests	integer	yes	—	This is the maximum request count for each IP and window.
window_seconds	integer	yes	—	This is the window time in seconds.
trusted_proxy_cidrs	string[]	no	[]	These IPv4 or IPv6 CIDRs can append X-Forwarded-For.

Behind a reverse proxy

When the gateway is behind nginx or a load balancer, list each proxy network that can append X-Forwarded-For.

The gateway starts with the direct socket peer. It reads the chain from right to left while each hop is in a listed network:

# Gateway behind one nginx reverse proxy
ip_rate_limit:
  max_requests: 50
  window_seconds: 60
  trusted_proxy_cidrs:
    - 10.0.0.0/8

Per-user rate limit

Each user identity from the request headers has its own counter.

user_rate_limit:
  max_requests: 30           # required
  window_seconds: 60          # required
  header_names:               # optional
    - "x-user-id"
    - "x-consumer-id"

Field	Type	Necessary	Default	Description
max_requests	integer	yes	—	This is the maximum request count for each user and window.
window_seconds	integer	yes	—	This is the window time in seconds.
header_names	string[]	no	["x-user-id"]	The gateway gets the user identity from these headers. It uses the first nonempty value.

A request without a nonempty matching header does not use the per-user counter. The gateway does not put it in an unknown group.

If the limit must apply to each request, configure a trusted identity header at the gateway boundary. A trusted proxy must remove and replace the caller-supplied value.

Token rate limit

This moving-window limit uses the token count in buffered Chat Completions responses.

token_rate_limit:
  max_tokens: 500000         # required
  window_seconds: 3600        # required
  scope: "global"             # optional: global | per_key | per_ip

Field	Type	Necessary	Default	Description
max_tokens	integer	yes	—	This is the maximum token count for each window.
window_seconds	integer	yes	—	This is the moving window time in seconds.
scope	string	no	"global"	This group scope is global, per_key, or per_ip.

Runtime operation: The limiter uses six moving subwindows for buffered POST /v1/chat/completions traffic.

After an upstream response succeeds, the limiter records tokens when the JSON body contains usage.total_tokens.

The limiter checks the next Chat Completions request against the available budget. It returns HTTP 429 when the budget is zero.

The limiter does not check or record streaming responses or other request types.

Scope examples

# Global: one bucket for all accounted buffered Chat Completions traffic
token_rate_limit:
  max_tokens: 1000000
  window_seconds: 3600
  scope: "global"

# Per API key: each key gets its own budget
token_rate_limit:
  max_tokens: 100000
  window_seconds: 3600
  scope: "per_key"

# Per IP: each direct socket peer gets its own budget
token_rate_limit:
  max_tokens: 50000
  window_seconds: 3600
  scope: "per_ip"

token_rate_limit.scope: per_ip uses the direct socket peer address. It does not use X-Forwarded-For or ip_rate_limit.trusted_proxy_cidrs.

Thus, callers behind a reverse proxy can share the proxy token group.

Size limits

These fields set byte limits for the active LLM proxy.

size_limits:
  max_body_bytes: 1048576        # 1 MB request body
  max_header_bytes: 8192         # 8 KB headers
  max_url_bytes: 4096            # 4 KB URL
  max_response_bytes: 10485760   # 10 MB response

Field	Type	Necessary	Default	Description
max_body_bytes	integer	no	unlimited	This is the maximum request body size.
max_header_bytes	integer	no	unlimited	This is the maximum total header size.
max_url_bytes	integer	no	unlimited	This is the maximum URL length.
max_response_bytes	integer	no	unlimited	This is the maximum response body size.

The active request check runs on POST /v1/chat/completions. It checks the body, headers, and then the URL.

A request limit failure returns HTTP 413 Payload Too Large. Do not use these three fields as protection for other gateway routes.

The gateway applies max_response_bytes to each buffered Chat Completions and Responses path.

An oversized buffered upstream response returns HTTP 502 Bad Gateway with the response_size_exceeded error code.

For a streaming response, the gateway has sent the HTTP headers. It emits a terminal SSE error and stops the stream.

Configure request limits for consumer groups in consumer_groups.groups[].rate_limit.max_requests. Use the top-level token rate limit for token accounting.

Distributed rate limiting

By default, each process keeps its rate limit counters in memory. A Redis or Valkey backend can share global_rate_limit across gateway instances.

It does not share the per-IP, per-user, or token counters.

Inline configuration

distributed_rate_limit:
  backend: "redis"
  url_env: "VERDICTAN_LLM_CACHE_REDIS_URL"

Field	Type	Necessary	Default	Description
backend	string	yes	—	The value is redis or valkey. The two values use the Redis wire protocol.
url_env	string	no	"VERDICTAN_LLM_CACHE_REDIS_URL"	This environment variable contains the connection URL.

global_rate_limit must also be configured. Its max_requests and window_seconds fields define the single shared counter.

Keep the backend in the shown top-level distributed_rate_limit block.

Build and connection behavior

The installed CLI must include distributed-state support. Gateway startup fails when a configured backend is unavailable in that build.

Connected and multi-node profiles that must use shared state fail startup when the URL is missing or initialization fails. A backend loss makes dependent requests and /readyz return HTTP 503 with dependency.distributed_state_unavailable.

Process-local fallback is limited to a specified one-node, self-hosted development profile. It is not a recovery path for a necessary distributed deployment.

Environment variable wiring

Before you start the gateway, set the environment variable in url_env to the Redis or Valkey connection string.

Complete rate limit example

pack:
  name: "rate-limited-gateway"
  version: "1.0.0"
  enabled: true

# Global ceiling
global_rate_limit:
  max_requests: 5000
  window_seconds: 60

# Per-IP protection
ip_rate_limit:
  max_requests: 100
  window_seconds: 60
  trusted_proxy_cidrs:
    - 10.0.0.0/8

# Per-user fairness
user_rate_limit:
  max_requests: 30
  window_seconds: 60
  header_names: ["x-user-id", "x-consumer-id"]

# Token budget
token_rate_limit:
  max_tokens: 1000000
  window_seconds: 3600
  scope: "global"

# Request size protection
size_limits:
  max_body_bytes: 2097152      # 2 MB
  max_response_bytes: 20971520 # 20 MB

# Shared global request counter
distributed_rate_limit:
  backend: "valkey"
  url_env: "VERDICTAN_LLM_CACHE_REDIS_URL"

providers:
  targets:
    - id: "openai-prod"
      provider: "openai"
      model: "your-openai-model"
      secret_key_ref:
        env: "VERDICTAN_OPENAI_API_KEY"

policies:
  chain:
    - "audit-logger"

Replace your-openai-model with a model identifier that is available to your OpenAI account.

Next steps

• Routes and consumer groups
• Security and network configuration
• Provider configuration
• Declarative configuration reference