Skip to main content

Hugging Face Inference

Connect Verdictan to a Hugging Face inference endpoint with native model-path routing. Applications call the local Verdictan endpoint. The gateway owns the upstream credential and applies the configured policy chain.

Use this page when​

  • You have access to Hugging Face Inference and must route it through Verdictan.
  • You want one specified provider target. You can lint and review it before rollout.
  • You want a stable integration contract. You do not want mutable prices, context limits, or retirement dates in the config.

Prerequisites​

  • Install the verdictan CLI.
  • Get the upstream credential and an enabled model or endpoint from Hugging Face Inference.
  • Get a Verdictan runtime API token for the gateway.
  • Get a different Verdictan client API token for client requests.
  • The --agent flow uses the runtime API token for agent-bound configuration.

Configure the provider​

Before you start the gateway, replace the replace-with-... values. The example uses the active huggingface runtime contract.

pack:
  name: huggingface-integration
  version: 1.0.0
  enabled: true
policies:
  chain:
    - prompt-injection
    - pii-detector
    - audit-logger
providers:
  targets:
    - id: huggingface-primary
      provider: huggingface
      provider_type: huggingface
      format: huggingface
      model: "replace-with-organization/model"
      base_url: https://router.huggingface.co/hf-inference
      secret_key_ref:
        env: VERDICTAN_HF_API_TOKEN

The gateway process resolves the provider credential. Client applications do not send this credential to Verdictan.

Start and verify​

export VERDICTAN_API_TOKEN="replace-with-verdictan-api-token"
export VERDICTAN_HF_API_TOKEN="replace-with-upstream-credential"

verdictan policy lint --file policy-config.yaml
verdictan gateway run \
  --agent huggingface-integration \
  --listen 127.0.0.1:41002 \
  --policy-config policy-config.yaml

VERDICTAN_API_TOKEN authenticates the gateway runtime and control-plane synchronization. Do not reuse it as the client credential.

In a different terminal, use these commands:

export VERDICTAN_CLIENT_TOKEN="replace-with-different-client-token"

curl -fsS http://127.0.0.1:41002/healthz
curl -fsS http://127.0.0.1:41002/v1/chat/completions \
  -H "Authorization: Bearer ${VERDICTAN_CLIENT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"model":"replace-with-organization/model","messages":[{"role":"user","content":"Reply with one short sentence."}]}'

/healthz verifies process liveness only. The authenticated Chat request verifies the client token, provider credential, model access, and request path.

If you did not configure a specified multi-model route, use the same model identifier in the request and target.

Active Verdictan contract​

SettingBehavior
providerhuggingface
Upstream requestThe gateway sends requests to the HF Inference router endpoint at /hf-inference/models/{model}.
Upstream authenticationThe gateway uses a Bearer token from VERDICTAN_HF_API_TOKEN.
Client endpointClient applications send requests to /v1/chat/completions on the Verdictan gateway.

Use the specified repository model identifier that HF Inference accepts. The active adapter maps chat messages to an inputs string.

The adapter maps token and sampling options to parameters. It reads generated_text from the response.

Select a text-generation model that accepts and returns that contract. This target does not support other Hugging Face tasks or arbitrary endpoint schemas.

Transport and evidence boundary​

Verdictan configuration and adapter tests verify parsing and request translation. They do not verify live credentials or active model availability.

The command above exercises one buffered Chat Completions request. It does not establish streaming, tools, structured output, or a different request family. Verify each necessary feature with the selected model before production use.

Model and production checks​

  • Before rollout, verify the model ID, region, endpoint availability, and account permissions in the official provider surface.
  • Add pricing, context, retention, or certification metadata only from the active provider contract.
  • Validate optional request features for the selected request family and model.
  • These features include streaming, tools, and structured output.
  • Do not make provider-wide assumptions.
  • Keep the upstream credential on the server.
  • Bind production listeners only to the specified interface.
  • Protect non-loopback listeners with authentication, network access controls, and TLS.

Next steps​