Skip to main content

Quality assertion configuration

Use quality-scorer to compare model output with specified checks before the gateway accepts the response.

Use stream: false for this control. The active SSE path does not run the quality scorer against the buffered output.

Schema requirements​

Each assertion object has these fields:

  • type: the assertion type
  • Optional gate fields such as threshold, weight, mode, and severity
  • config: the options for the specified type

A configuration is invalid if a type-specific field is at the top assertion level. Put fields such as value or rubric in config.

Minimal full example​

pack:
  name: config-quality-assertions-example-1
  version: 1.0.0
  enabled: true
policies:
  chain:
    - quality-scorer
policy:
  quality-scorer:
    assertions:
      - type: contains
        config:
          value: disclaimer
      - type: llm-rubric
        threshold: 0.8
        config:
          rubric: Response must be factually accurate
    thresholds:
      min_aggregate: 0.8
    failure_action:
      action: block

Assertion object fields​

FieldTypeNotes
typestringThis field specifies the necessary assertion type.
namestringThis optional field gives a readable label.
enabledbooleanThis field controls the assertion without its deletion.
thresholdnumberThis is the minimum passing score for the assertion.
weightnumberThis is the relative weight in the total score.
configobjectThis field contains options for the specified type.
modestringenforce, audit, or shadow.
severitystringcritical, warning, or info.

Standard assertion types​

Contains​

- type: contains
  name: has-disclaimer
  threshold: 1.0
  weight: 0.5
  mode: enforce
  severity: critical
  config:
    value: disclaimer

LLM rubric​

- type: llm-rubric
  threshold: 0.8
  config:
    rubric: Response must be accurate and directly answer the question

Word count​

- type: word-count
  config:
    min: 40

Moderation​

- type: moderation
  config:
    categories:
      - violence
      - hate

Multiple assertion gates​

Use multiple specified assertions when one quality gate must examine safety, source support, and response structure.

pack:
  name: config-quality-assertions-example-52
  version: 1.0.0
  enabled: true
policies:
  chain:
    - quality-scorer
policy:
  quality-scorer:
    assertions:
      - type: moderation
        config:
          categories:
            - violence
            - hate
            - self-harm
      - type: is-refusal
        config:
          expected: false
      - type: context-faithfulness
        threshold: 0.8
      - type: contains
        config:
          value: source
      - type: llm-rubric
        threshold: 0.8
        config:
          rubric: Response must be helpful

Threshold and result fields​

policy:
  quality-scorer:
    thresholds:
      min_aggregate: 0.8
      min_accuracy: 0.8
    pass_policy:
      strategy: weighted_average
      threshold: 0.8
    failure_action:
      action: block
      fallback_message: I cannot provide a sufficiently reliable answer.

A threshold key starts its built-in scorer when the benchmarks block does not set the related benchmark.

For example, min_relevancy calculates relevancy. It adds the result to the total with a default weight of 1.0.

The gateway also saves configured min_coherence and min_completeness values in event quality scores.

Use failure_action.action: block to reject a failed response. Use fallback to replace it with fallback_message and continue with that replacement.

Judge-backed scoring​

Use judge when a second model must add a rubric result to the quality details. The result can be pass, warn, or fail.

The judge result is audit data. By itself, it does not change the primary quality gate verdict.

policy:
  quality-scorer:
    judge:
      enabled: true
      endpoint: https://api.openai.com/v1/chat/completions
      model: your-openai-review-model
      secret_key_ref:
        env: VERDICTAN_OPENAI_API_KEY
      threshold: 0.8
      warn_threshold: 0.6
      timeout_ms: 5000

Use judge.sampling_rate to set the judge call frequency.

Complete quality gate example​

pack:
  name: quality-enforced
  version: 1.0.0
  enabled: true
policies:
  chain:
    - prompt-injection
    - quality-scorer
    - audit-logger
policy:
  prompt-injection:
    response:
      action: block
  quality-scorer:
    assertions:
      - type: contains
        config:
          value: source
      - type: word-count
        config:
          min: 40
      - type: llm-rubric
        threshold: 0.8
        config:
          rubric: Response must be accurate, helpful, and well structured
    thresholds:
      min_aggregate: 0.8
      min_accuracy: 0.8
    pass_policy:
      strategy: weighted_average
      threshold: 0.8
    failure_action:
      action: block
      fallback_message: Quality check failed.
    judge:
      enabled: true
      endpoint: https://api.openai.com/v1/chat/completions
      model: your-openai-review-model
      secret_key_ref:
        env: VERDICTAN_OPENAI_API_KEY
      threshold: 0.8
      warn_threshold: 0.6
    benchmarks:
      ragas_faithfulness: true
      ragas_relevancy: true
  audit-logger: {}
  • Keep deterministic assertions simple.
  • Add judge scoring only when rubric analysis has more value than its increased latency and cost.
  • When you add new assertions, use audit or shadow mode.
  • Use a small set of specified assertions. A very large quality gate is hard to examine.

Next steps​