Quality Scorer

The quality-scorer policy checks provider responses after generation. It can pass, replace, or block output according to the configured quality result.

Use stream: false for this control. The active SSE path does not run the quality scorer against buffered output.

Phase and verdicts

• Phase: output
• Verdicts: A pass returns allow. A failure returns block by default. fallback returns allow with replacement content.

Configuration

pack:
  name: quality-scorer-example-1
  version: "1.0.0"
  enabled: true

policies:
  chain:
    - quality-scorer

policy:
  quality-scorer:
    min_output_chars: 80
    min_sentences: 2
    benchmarks:
      ragas_faithfulness: true
      ragas_relevancy: true
      bleu_score: false
      nli_entailment: false
    assertions:
      - type: is-json
        name: response-is-json
        threshold: 1.0
        mode: enforce
        severity: critical
        config: {}
      - type: llm-rubric
        name: concise-and-grounded
        threshold: 0.7
        weight: 0.5
        mode: audit
        severity: warning
        config:
          rubric: Explain the answer clearly and stay grounded in the supplied context.
    thresholds:
      min_aggregate: 0.7
      min_faithfulness: 0.8
      min_relevancy: 0.75
    weights:
      faithfulness: 0.5
      relevancy: 0.5
    failure_action:
      action: fallback
      fallback_message: I cannot provide a sufficiently accurate response right now.
    pass_policy:
      strategy: weighted_average
      threshold: 0.7

Supported top-level fields

Field	Type	Default	Notes
min_output_chars	integer	0	Fails when output text is shorter than this value.
min_sentences	integer	0	Fails when the sentence count is below this value.
benchmarks	object	{}	Supports the listed benchmark fields. A threshold field enables its benchmark when the benchmark is not configured.
bleu_reference	string	—	Used for BLEU when no request context is available.
assertions	array	[]	Assertion list evaluated against the output.
thresholds	object	{}	Supports min_aggregate, min_faithfulness, min_relevancy, min_bleu, min_accuracy, min_coherence, and min_completeness.
weights	object	{}	Supports the listed weight fields. An enabled benchmark uses 1.0 when it has no configured weight.
failure_action	object	{ action: "block" }	fallback is the only value that replaces the provider response inline.
pass_policy	object	{ strategy: "all", quorum: 0.5, threshold: 0.5 }	Controls how failing assertions are aggregated.
industry_profiles	object	{}	Applied only when the request contains verdictan.industry.
judge	object	—	Optional judge call. Its result is attached to details and does not change the policy verdict.

Assertion fields

Each assertion entry can contain:

Field	Type	Default	Notes
type	string	—	Necessary. Unsupported types are reported as assertion errors.
name	string	assertion type	Optional display name.
enabled	boolean	true	Disabled assertions are skipped.
threshold	number	type-specific	Compared with the assertion score when configured.
weight	number	1.0	Used by weighted aggregation.
mode	string	enforce	audit and shadow do not block.
severity	string	critical	Only enforce + critical failures contribute to blocking.
config	object	{}	Assertion-specific parameters.

Active behavior

1. Gets text from the upstream response body.
2. Applies min_output_chars and min_sentences.
3. Calculates enabled benchmark metrics.
4. Evaluates assertions.
5. Applies pass_policy only to assertion failures with mode: enforce and severity: critical.
6. Applies threshold checks for aggregate and benchmark scores.
7. Sets the failure action.

Important behavior notes

• A threshold-only config calculates quality scores. For example, thresholds.min_relevancy enables relevance scoring unless its benchmark is configured.
• industry_profiles are selected by request.verdictan.industry.
• Active industry-profile overrides apply to min_aggregate, min_faithfulness, min_relevancy, min_bleu, min_accuracy, min_coherence, and min_completeness.
• failure_action.action: fallback replaces assistant text. block suppresses failed output.
• The quality engine scores llm-rubric assertions. The optional judge block starts a different judge call and adds its result to the details.

Minimal correct example

policy:
  quality-scorer:
    min_output_chars: 20
    assertions:
      - type: contains
        name: mentions-policy
        threshold: 1.0
        mode: enforce
        severity: critical
        config:
          value: policy

Next steps

• Config Quality Assertions
• Policy Controls Catalog
• Human Oversight