Skip to main content

Content Extractor

The content-extractor policy gets content from URLs in request messages. It records the URL text in policy details. A host allowlist and size limits control each request.

Configuration​

policy:
  content-extractor:
    allow_hosts:
    - docs.example.com
    - support.example.com
    timeout_ms: 2000
    max_bytes: 65536
    fetch_urls: true
    action_on_error: warn
pack:
  name: content-extractor-example-1
  version: 1.0.0
  enabled: true
policies:
  chain:
  - content-extractor

Fields​

FieldTypeDescriptionDefault
allow_hostsstring[]Permitted hostnames. An empty list denies all URLs. Add each specified hostname. Wildcards are not supported.[]
timeout_msinteger (min: 1)Fetch timeout for each URL in milliseconds. Requests that exceed this timeout are errors.2000
max_bytesinteger (min: 1)Maximum response body size in bytes. Responses exceeding this limit are truncated.65536 (64 KB)
fetch_urlsboolDetects and gets URLs from request content. A value of false stops this work.true
action_on_errorstringAction when extraction returns a blocked_reason. "warn" allows the request. "block" rejects it. Errors include invalid URLs, unsupported schemes, denied hosts, DNS failures, fetch failures, and body-read failures. Oversize bodies are truncated and do not trigger this setting."warn"

Use cases​

Audit allowlisted URL fetches​

Get referenced documentation from approved hosts. Record the URL text in the policy result details.

pack:
  name: rag-content-fetch
  version: 0.1.0
  enabled: true
policies:
  chain:
  - content-extractor
  - audit-logger
policy:
  content-extractor:
    allow_hosts:
    - docs.company.com
    - confluence.company.com
    timeout_ms: 5000
    max_bytes: 131072
    fetch_urls: true
    action_on_error: warn
  audit-logger: {}

Fail-closed allowlisted fetches​

Get linked documents from approved hosts with strict size limits. Block the request when safe extraction cannot finish.

pack:
  name: "doc-summarizer"
  version: "0.1.0"
  enabled: true

policies:
  chain:
    - content-extractor

policy:
  content-extractor:
    allow_hosts:
      - "files.example.com"
    timeout_ms: 10000
    max_bytes: 524288
    fetch_urls: true
    action_on_error: "block"

Verify that request URLs are available from an approved host. Block requests with broken or denied links.

pack:
  name: "link-verifier"
  version: "0.1.0"
  enabled: true

policies:
  chain:
    - content-extractor

policy:
  content-extractor:
    allow_hosts:
      - "api.example.com"
    timeout_ms: 3000
    max_bytes: 1024
    fetch_urls: true
    action_on_error: "block"

How it works​

  1. URL detection: When fetch_urls is enabled, the gateway checks request messages for URLs.
  2. Safety checks: Each URL must use http or https and match allow_hosts.
  3. The URL must resolve only to public IP addresses. The gateway rejects private and internal addresses.
  4. HTTP request: The gateway gets permitted URLs with HTTP GET and the configured timeout_ms.
  5. It truncates a response that is larger than max_bytes.
  6. Policy details: The policy stores content and detected URLs in details.extracted_text.
  7. Errors: The extractor stops at the first invalid or failed URL and sets blocked_reason.
  8. action_on_error: "warn" allows the request. action_on_error: "block" blocks the request.

The evaluator does not add the fetched text to request messages. Subsequent prompt, DLP, and safety policies examine their documented inputs, not details.extracted_text.

Best practices​

  • Always specify allow_hosts. An empty list blocks all requests. Do not use broad host lists.
  • Set small max_bytes limits. Large documents increase fetch and event-detail size. Start with 64 KB.
  • Use "block" for critical pipelines. Set action_on_error to "block" when the fetch must succeed. Then, a fetch error blocks the request.
  • Protect policy details. They can contain the fetched text. Limit access and retention through the owning evidence workflow.
  • Do not assume downstream inspection. The extractor does not rewrite the forwarded request body.
  • Monitor timeout settings. The default is 2 seconds. Increase timeout_ms only for a slower host or document.

Next steps​