Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Overview

WebFang is a production-ready web scraper built in Rust (1.91), with Clean Architecture, a TLS-fingerprinting HTTP client, optional AI semantic cleaning (remote embeddings — see the migration guide, ONNX removal), and sitemap-based crawling.

This book is the narrative documentation. The complete API reference for the workspace crates is generated from source via cargo doc and published alongside this book under /api/<crate>/ (e.g. /api/webfang_core/).

Crates

CrateRole
webfang_coreDomain, application, and infrastructure layers — the scraping engine (semantic cleaning via remote EmbeddingPort adapters included)
webfang_mcpMCP server (36 tools)
webfang_cliCLI binary (webfang)
webfang_test_utilsShared test helpers

Installing

WebFang ships binaries only — nothing is published to crates.io, so cargo install webfang cannot work. Every release publishes three archives per platform — the CLI, the MCP HTTP server and the MCP stdio server — across Linux x86_64/ARM64, macOS Apple Silicon and Windows x86_64, plus a SHA256SUMS.txt. Intel macOS has no artifact by design. Start here: Installation & Uninstall.

Chapters

  • Installation & Uninstall — verified download and checksum commands per platform, platform floors, and every on-disk root a release leaves behind.
  • Debugging & Observability — built-in tracing, correlation IDs, and the jq query cookbook for debug.jsonl.
  • Testing — E2E integration tests, snapshot strategy, and coverage exclusions.
  • Troubleshooting — diagnosing slow crawls, silent failures, WAF blocks, and async deadlocks.
  • CLI Reference — the complete webfang flag reference, auto-generated from the binary itself.

Regenerating this documentation

All heavy compute (mdBook build + rustdoc + link checks) runs in CI. Locally you only pull the generated result:

just docs        # downloads the combined NotebookLM/LLM markdown from the latest CI run
just docs-local  # local preview: builds and serves this mdBook only

Installation & Uninstall

WebFang is a binaries-only distribution. There is no cargo install webfang and nothing is published to crates.io — the release automation sets git_only = true and publish = false in release-plz.toml, so a cargo install webfang cannot work. Get the binary from GitHub Releases.

Releases · Troubleshooting · CLI Reference · Release Feature Matrix


Quick path

  1. Pick your platform in the asset table and copy its download command.
  2. Verify the checksum with the command for your shell.
  3. Put the binary on your PATH using the per-platform steps (Linux, macOS, Windows).
  4. Confirm it runs: webfang --version.

If step 4 fails, go straight to install-time failures.


1. Pick your asset

Every release publishes twelve archives — three per platform — plus one checksum file. The asset name carries the full Rust target triple — there is no friendly OS alias and no version in the name (the version is in the tag/URL, not the file). Each archive contains a bare binary: no install script, no LICENSE, no README inside.

PlatformCLI asset (inside the release)Format
Linux x86_64webfang-x86_64-unknown-linux-gnu.tar.gz.tar.gz
Linux ARM64webfang-aarch64-unknown-linux-gnu.tar.gz.tar.gz
macOS Apple Siliconwebfang-aarch64-apple-darwin.tar.gz.tar.gz
Windows x86_64webfang-x86_64-pc-windows-msvc.zip.zip
Any platformSHA256SUMS.txtchecksum manifest

The other two archives per platform hold the MCP servers — same targets, same format as the CLI archive for that platform:

PlatformHTTP serverstdio server
Linux x86_64webfang-mcp-x86_64-unknown-linux-gnu.tar.gzwebfang-mcp-stdio-x86_64-unknown-linux-gnu.tar.gz
Linux ARM64webfang-mcp-aarch64-unknown-linux-gnu.tar.gzwebfang-mcp-stdio-aarch64-unknown-linux-gnu.tar.gz
macOS Apple Siliconwebfang-mcp-aarch64-apple-darwin.tar.gzwebfang-mcp-stdio-aarch64-apple-darwin.tar.gz
Windows x86_64webfang-mcp-x86_64-pc-windows-msvc.zipwebfang-mcp-stdio-x86_64-pc-windows-msvc.zip

The CLI archive never contains an MCP binary: webfang-mcp and webfang-mcp-stdio each have their own archive, because every archive holds exactly one binary.

There is no Intel macOS artifact. x86_64-apple-darwin is deliberately not built. The original reason was ONNX-local (no x64 macOS prebuilt for ONNX Runtime); the ONNX removal (ADR-0004 paso 6) makes that reason moot, but the target list was never extended — each release re-trains on the artifact list as-is, so the entry stays. If you are on an Intel Mac, there is nothing to mis-download — see Intel macOS.

Set the version once and reuse it. Copy the tag from the latest release:

VERSION=v2.12.0   # ← the tag you just looked up
Verify these names yourself (any release, any version)
gh release view v2.12.0 --json assets --jq '.assets[].name'

Observed output for v2.11.0:

SHA256SUMS.txt
SHA256SUMS.txt.sigstore.json
THIRD_PARTY_NOTICES.txt
THIRD_PARTY_NOTICES.txt.sigstore.json
webfang-aarch64-apple-darwin.tar.gz
webfang-aarch64-apple-darwin.tar.gz.sigstore.json
webfang-aarch64-unknown-linux-gnu.tar.gz
webfang-aarch64-unknown-linux-gnu.tar.gz.sigstore.json
webfang.spdx.json
webfang.spdx.json.sigstore.json
webfang-x86_64-pc-windows-msvc.zip
webfang-x86_64-pc-windows-msvc.zip.sigstore.json
webfang-x86_64-unknown-linux-gnu.tar.gz
webfang-x86_64-unknown-linux-gnu.tar.gz.sigstore.json

2. Verify the checksum

Each release ships a SHA256SUMS.txt listing the SHA-256 of every archive. Verify the archive you downloaded, before extracting it. The command differs per platform: sha256sum is GNU coreutils and does not exist on stock macOS, and PowerShell has neither it nor shasum.

Linux (GNU coreutils) — checks the archive against the manifest:

sha256sum -c SHA256SUMS.txt --ignore-missing

macOS (shasum, ships with the system) — BSD coreutils has no -c/--ignore-missing; check the one file you downloaded:

shasum -a 256 webfang-aarch64-apple-darwin.tar.gz
grep 'aarch64-apple-darwin' SHA256SUMS.txt

The two hashes must be identical.

Windows (PowerShell) — Get-FileHash, compared against the manifest line:

(Get-FileHash webfang-x86_64-pc-windows-msvc.zip -Algorithm SHA256).Hash.ToLower()
Select-String -Path SHA256SUMS.txt -Pattern 'pc-windows-msvc'

If you have GitHub CLI installed, it fetches both files for you (it does not verify them — do step 2 anyway):

gh release download "${VERSION}" --pattern 'SHA256SUMS.txt'
gh release download "${VERSION}" --pattern 'webfang-x86_64-unknown-linux-gnu.tar.gz'

Every published asset carries a real SLSA provenance attestation, minted keyless by the release workflow, plus a .sigstore.json bundle next to it in the release. Verify a downloaded file against the digest you actually have:

gh attestation verify webfang-x86_64-unknown-linux-gnu.tar.gz \
  --repo XaviCodeDev1000/webfang

SHA256SUMS.txt is the offline check and stays authoritative for integrity. The provenance table in the release body is WebFang's own L1 gate — a summary written by the workflow, not the attestation itself; verify against the API or the .sigstore.json bundle, never against that table.


3. Install

Every recipe below is copy-pasteable as a unit. The three unix steps — download, verify, extract — are identical apart from the asset name, so only the differing part is repeated per platform.

Linux

# 1. Download (x86_64; use aarch64-unknown-linux-gnu on ARM64)
VERSION=v2.12.0
curl -fLO "https://github.com/XaviCodeDev1000/webfang/releases/download/${VERSION}/webfang-x86_64-unknown-linux-gnu.tar.gz"
curl -fLO "https://github.com/XaviCodeDev1000/webfang/releases/download/${VERSION}/SHA256SUMS.txt"

# 2. Verify BEFORE extracting
sha256sum -c SHA256SUMS.txt --ignore-missing

# 3. Extract; the archive holds a single file named `webfang`
tar xzf webfang-x86_64-unknown-linux-gnu.tar.gz

# 4. Put it on PATH (system-wide)
sudo install -m 0755 webfang /usr/local/bin/webfang

#    …or per-user, no sudo:
# mkdir -p ~/.local/bin && install -m 0755 webfang ~/.local/bin/webfang
#   then add to PATH if needed: export PATH="$HOME/.local/bin:$PATH"

# 5. Confirm
webfang --version

Notes:

  • The archive holds a bare binary and preserves its executable bit, so tar xzf alone is usually enough. Use install -m 0755 (as above) to guarantee the bit regardless of how the archive was extracted — a file-manager double-click can drop it.
  • The binary is stripped, but it is not fully static: it links libc/libm (glibc), plus libstdc++ and libgcc_s (BoringSSL is C++). All are standard on glibc distros, but a minimal or musl-based image will fail to load it until they are present. There is no .deb/.rpm and no bundled runtime to install.

macOS (Apple Silicon)

# 1. Download
VERSION=v2.12.0
curl -fLO "https://github.com/XaviCodeDev1000/webfang/releases/download/${VERSION}/webfang-aarch64-apple-darwin.tar.gz"
curl -fLO "https://github.com/XaviCodeDev1000/webfang/releases/download/${VERSION}/SHA256SUMS.txt"

# 2. Verify BEFORE extracting (no sha256sum on stock macOS)
shasum -a 256 webfang-aarch64-apple-darwin.tar.gz
grep 'aarch64-apple-darwin' SHA256SUMS.txt

# 3. Extract; the archive holds a single file named `webfang`
tar xzf webfang-aarch64-apple-darwin.tar.gz

# 4. Put it on PATH (system-wide)
sudo install -m 0755 webfang /usr/local/bin/webfang

#    …or per-user:
# mkdir -p ~/.local/bin && install -m 0755 webfang ~/.local/bin/webfang

# 5. Clear the quarantine flag (see note), then confirm
xattr -d com.apple.quarantine webfang 2>/dev/null || true
webfang --version

Gatekeeper. A binary downloaded with curl/Safari and then extracted carries the com.apple.quarantine extended attribute, and macOS kills it on first run with "cannot be opened because the developer cannot be verified". It is not corrupt and the checksum will match. Remove the attribute with xattr -d com.apple.quarantine webfang (or xattr -dr com.apple.quarantine <dir>), then run it. Right-click → Open works too, once.

Intel macOS (unsupported)

No artifact exists for Intel macOS, and that is deliberate — it is a decision, not a release gap, so do not go looking for an Intel macOS archive among the assets.

Building from source works on the normal developer path (Rust 1.91 toolchain, cmake for BoringSSL — see from source). The old ONNX excuse no longer applies (ADR-0004 paso 6 removed the ai feature and the local engine), so a source build has no semantic holes: --clean-ai resolves its endpoint through --embedding-provider, Obsidian vault search included. What you lose on one machine is only runtime network policy — except under --offline, where NO embeddings provider is usable: the guard rejects every one of them, a remote endpoint or a loopback one, and the run exits with 78. Since 3.0.0 there is no local engine to fall back to, so --clean-ai and --offline are mutually exclusive. And --offline gates the SEMANTIC-CLEANING providers, not the crawler's network: it means "no semantic cleaning", not "no network" — it only works offline when no embeddings provider is selected at all (no --embedding-provider, and no provider with the embedding capability in the config), otherwise the run fails with exit 78 (ADR-0004 §46, corrected 2026-10-21).

Behind a corporate proxy?

WebFang ignores the proxy environment (HTTP_PROXY/HTTPS_PROXY/ALL_PROXY) by design — for the AI endpoints and for crawling alike (#2500, see data flow). If your network only egresses through a proxy, run a local forwarding sidecar, point the provider's base_url at it, and opt that provider into loopback endpoints; the sidecar talks to the corporate proxy, WebFang does not. webfang doctor states this posture on every run.

Windows

# 1. Download (PowerShell 5.1+)
$Version = "v8.0.0"
$Asset   = "webfang-x86_64-pc-windows-msvc.zip"
Invoke-WebRequest "https://github.com/XaviCodeDev1000/webfang/releases/download/$Version/$Asset" -OutFile $Asset
Invoke-WebRequest "https://github.com/XaviCodeDev1000/webfang/releases/download/$Version/SHA256SUMS.txt" -OutFile SHA256SUMS.txt

# 2. Verify BEFORE extracting
(Get-FileHash $Asset -Algorithm SHA256).Hash.ToLower()
Select-String -Path SHA256SUMS.txt -Pattern 'pc-windows-msvc'

# 3. Extract; the archive holds a single file named `webfang.exe`
Expand-Archive -Path $Asset -DestinationPath .

# 4. Put it on PATH (per-user, no admin)
$Bin = "$env:USERPROFILE\bin"
New-Item -ItemType Directory -Force -Path $Bin | Out-Null
Move-Item -Force webfang.exe "$Bin\webfang.exe"
[Environment]::SetEnvironmentVariable(
    "Path",
    [Environment]::GetEnvironmentVariable("Path", "User") + ";$Bin",
    "User")

# 5. Open a NEW terminal, then confirm
webfang --version

Notes:

  • The binary is webfang.exe on Windows, webfang everywhere else. The command is webfang either way.
  • %USERPROFILE%\bin avoids needing Administrator rights. The PATH edit only reaches new terminals — re-open yours before step 5.
  • If the system-wide %ProgramFiles%\webfang route is preferred, copy webfang.exe there instead.

Visual C++ runtime. The Windows artifact is built with MSVC, which links the dynamic CRT by default. A machine without the Visual C++ Redistributable fails at startup with "The code execution cannot proceed because VCRUNTIME140.dll was not found". That is the redistributable being absent, not a broken download — install the Microsoft Visual C++ Redistributable 2015-2022 (x64) from Microsoft and re-run. Verify your checksum first either way, so the two causes stay distinguishable.

MCP servers (from the release artifact)

Two of the three archives per platform hold an MCP server. Both expose the same 36 tools; what differs is how a client reaches them.

BinaryTransportUse it when
webfang-mcpStreamable HTTP, 127.0.0.1:8080/mcp by defaulta local service that several clients share over HTTP
webfang-mcp-stdiostdin/stdout (JSON-RPC)a desktop client — Claude Desktop, OpenCode, Cline — spawns the server as a subprocess

Installing them is the CLI's recipe unchanged: same SHA256SUMS.txt verification, same extraction, same PATH. Only the asset name changes.

stdio. There is nothing to start by hand — the client owns the process. Point its command at the installed binary; an absolute path is safest, because a desktop client usually does not inherit your shell's PATH:

webfang-mcp-stdio

HTTP. Run it as a local service:

webfang-mcp --allow-anonymous                       # loopback only, no token
webfang-mcp --auth-token "$WEBFANG_MCP_AUTH_TOKEN"  # clients send Authorization: Bearer <token>
webfang-mcp --allow-anonymous --bind 127.0.0.1:9090  # move it off the default port

webfang-mcp refuses to start without either --allow-anonymous or --auth-token. An unset credential used to mean "anyone who can reach the socket", so the token-less default is a hard error, not a warning. --allow-anonymous is a loopback affordance only: a routable bind still requires --auth-token, and the transport is plain HTTP, so terminate TLS in front of it before exposing it beyond localhost.

Both binaries accept --export-roots <dir> (repeatable or comma-separated). Absolute output_dir / checkpoint_dir paths in a tool call are rejected unless they fall under a declared root — relative paths always work, so you only need this when a client passes absolute ones.


Platform requirements

PlatformRequirementWhy
Linux x86_64 / ARM64glibc 2.39 or newer; no musl build existsThe artifacts are built on GitHub's ubuntu-latest runner (Ubuntu 24.04, glibc 2.39) and link glibc directly. The floor is inherited from the builder, not a chosen target — so a distro older than Ubuntu 24.04 has no compatible artifact.
macOSApple Silicon onlyIntel macOS has no artifact (decision pre-dating the ONNX removal; the target list was never extended).
WindowsVisual C++ Redistributable 2015-2022 (x64)MSVC's default dynamic CRT.
Any~17 MB download, no installer, no bundled runtimestripped binary; needs only standard system libraries (glibc + libstdc++/libgcc_s).

If a Linux binary dies immediately with /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.xx' not found, the distro is older than the floor above. There is no musl/static artifact to fall back to; build from source instead.


What is not in the release binary

The release build is one invocation that compiles the CLI and both MCP servers with no capability features enabled:

cargo build --release --locked --features "webfang_cli/mcp" \
  --target <target-triple> -p webfang_cli -p webfang_mcp --bins

The mcp feature of that command is inert (an explicit-empty marker): the webfang CLI carries zero MCP code — the servers are the separate webfang-mcp / webfang-mcp-stdio binaries in their own archives. (The former ai feature no longer exists at all, ADR-0004 paso 6: remote-embedding semantic cleaning ships in the default build — migration guide.)

The full capability-by-capability matrix — what each of the three binaries can do, and the flags that bring each capability back when building from source — lives in the Release Feature Matrix. The short version, before you file a bug:

Missing from the release binariesConsequence
chromium feature--js-strategy full is rejected at preflight (it needs a Chromium downloader). The default static strategy and the 3-layer hybrid strategy are unaffected.
persistence featureThe SQLite checkpoint store is absent. File-based --resume state is unaffected.
console featuretokio-console is not available. Tracing via --trace-file is.
adaptive-selectors featureAdaptive selector learning is off — its flags do not exist in this build.

From source (developers)

Not an installation route for end users — this is the contributor path. It downloads and compiles every dependency, BoringSSL included.

Prerequisites: Rust 1.91 (rust-toolchain.toml pins it), a C/C++ compiler, git, and cmake. cmake is mandatory: the HTTP client wreq → btls → btls-sys (formerly boring2/boring-sys2) compiles BoringSSL from C++ source on first build, and without cmake the failure surfaces deep inside a build script with an error that says nothing about the real cause.

git clone https://github.com/XaviCodeDev1000/webfang.git
cd webfang
cmake --version        # must print a version; install cmake if empty
cargo build --release --locked --features mcp -p webfang_cli
./target/release/webfang --version
  • -p webfang_cli builds only the CLI. Omitting it builds every workspace member.
  • The first build compiles BoringSSL from source and takes minutes, not seconds. This is normal.
  • Build-time egress: since the ONNX removal (ADR-0004 paso 6) the only third-party endpoint a build touches is crates.io. (Pre-paso-6 builds also pulled ONNX Runtime from cdn.pyke.io.)

Uninstall

WebFang never removes anything for you. Each binary is one file; the cache and output trees it creates are entirely your responsibility.

1. Remove the binary

Install locationCommand
Linux, /usr/local/binsudo rm /usr/local/bin/webfang
Linux, ~/.local/binrm ~/.local/bin/webfang
macOS, /usr/local/binsudo rm /usr/local/bin/webfang
Windows, %USERPROFILE%\binRemove-Item "$env:USERPROFILE\bin\webfang.exe"

If you also installed the MCP servers, they are removed the same way: the same directory, the same commands, with webfang replaced by webfang-mcp and webfang-mcp-stdio (.exe on Windows). They share no state of their own — the caches below are the only other thing a run leaves behind.

2. Remove the data

Check where things actually live first. The biggest artifacts live under the webfang cache base; the expected output tree lives next to your runs. (The historical ~372 MB ONNX model cache is a pre-ADR-0004-paso-6 item: the current binary never creates it — see the legacy note below.) Start at the webfang cache base:

# Where the webfang cache base resolves on THIS machine — run the line
# that matches your platform, and note the printed path.
echo "${XDG_CACHE_HOME:-$HOME/.cache}"           # Linux
# echo "${XDG_CACHE_HOME:-$HOME/Library/Caches}"   # macOS
# echo "$env:LOCALAPPDATA"                         # Windows (PowerShell)

Every webfang/… row in the table below is relative to that base — the table does not repeat the webfang/ prefix on Windows, so %LOCALAPPDATA%\webfang\state is the state directory, not %LOCALAPPDATA%\webfang\webfang\state.

The config base is the sibling you will also want when cleaning up: ~/.config (Linux), ~/Library/Application Support (macOS), %APPDATA% (Windows).

⚠️ The model cache was NOT under any of those bases — legacy, pre-paso-6 (ADR-0004). It was the only entry in this table that did not live under the webfang cache base, and § below keeps its exact location on each platform for cleanup after an upgrade from an old release. XDG_CACHE_HOME moves the webfang roots; it never moved the model.

Also note XDG_CACHE_HOME / XDG_CONFIG_HOME are honored only when they hold an absolute path — a relative or empty value is ignored, and the platform default is used instead. The echo above prints the variable even when WebFang ignores it; a path that looks wrong is probably this.

Path (relative to the base above)Created bySizeAuto-cleaned?
webfang/state/<domain>.json--resumeKBs, grows with processed-URL countNo — relocatable via --state-dir / WEBFANG_STATE_DIR
webfang/state/<domain>.json.lockevery RecordStore write0 BNo — permanent by design. See the note below.
webfang/state/<domain>.json.bakmigrating a stale state versionsame as stateNo — an existing backup is kept as-is
~/.webfang/crawl.db — absolute, under $HOME, not the cache basecrawl/scrape runs that persist to SQLite (only with the persistence feature — absent from the release binary): extraction fingerprints, resources, chunks, notesgrows with fingerprinted pagesRows older than N days are pruned (plus VACUUM) when a run passes --retention-days N; otherwise delete the file while no run is active. Relocatable via --db-path / WEBFANG_DB_PATH
huggingface/hub/ — legacy only, see belowpre-paso-6 releases only (ADR-0004) — current builds never write it0 for current builds; up to 372 MB / ~1.2 GB for old onesNo — relocatable via HF_HOME, a HuggingFace variable, not WEBFANG_*
<output_dir>/ — Markdown, export.jsonl, rag_dataset/, _inbox/every rununbounded — your dataOpt-in: files older than N days are pruned when a run passes --retention-days N. Relocatable via -o / WEBFANG_OUTPUT; defaults to output/ in the current directory
webfang/config.toml (under the config base: ~/.config, ~/Library/Application Support, %APPDATA%)you, by editing itKBsNo — relocatable via WEBFANG_CONFIG or XDG_CONFIG_HOME

The .lock sentinel is supposed to survive. <domain>.json.lock is created on demand and intentionally never unlinked; the real mutual exclusion is an advisory flock the kernel releases when the process exits, including after a kill -9. Its presence after an uninstall is correct behaviour, not a failed removal. See troubleshooting.md.

To remove the webfang-owned cache roots and leave everything else alone:

# Linux / macOS: cache + state, but NOT the model cache
rm -rf "${XDG_CACHE_HOME:-$HOME/.cache}/webfang"     # Linux
rm -rf "${XDG_CACHE_HOME:-$HOME/Library/Caches}/webfang"  # macOS
# Windows: cache + state, but NOT the model cache
Remove-Item -Recurse -Force "$env:LOCALAPPDATA\webfang"

The model cache is the exception (legacy)

Since the ONNX removal (ADR-0004 paso 6) the current binary never creates this cache: there is no local model to fetch, and no hf-hub dependency remains. If you are upgrading from a pre-paso-6 release, the directory a previous binary filled still exists, is inert, and is entirely yours to delete; the details:

hf_hub (used by pre-paso-6 releases) does not use WebFang's platform helper. Its default is dirs::home_dir()/.cache/huggingface/hub — literally ~/.cache/… on every platform, including macOS and Windows, where the webfang cache base is ~/Library/Caches and %LOCALAPPDATA% respectively. It is only moved by HF_HOME, and no WEBFANG_* or XDG_* variable relocates it.

PlatformDefault model cache (no HF_HOME)With HF_HOME=/some/dir
Linux~/.cache/huggingface/hub/some/dir/hub
macOS~/.cache/huggingface/hub — not ~/Library/Caches/…/some/dir/hub
Windows%USERPROFILE%\.cache\huggingface\hub — not %LOCALAPPDATA%\…/some/dir/hub

To confirm before deleting, and to remove it:

# Unix
echo "${HF_HOME:-$HOME/.cache/huggingface}"
rm -rf "${HF_HOME:-$HOME/.cache/huggingface}"
# Windows
$HF = if ($env:HF_HOME) { $env:HF_HOME } else { "$env:USERPROFILE\.cache\huggingface" }
$HF
Remove-Item -Recurse -Force $HF

If HF_HOME is set, the default location is empty and deleting it recovers nothing — which is why the confirmation line comes first, and why the two are never collapsed into one rm.

Never delete a webfang directory without reading the path you built above — a relative or empty base would resolve to somewhere unintended, and <output_dir> is user data with no backup.


Checklist

A fresh install is done when all of these are true:

  • I picked the asset whose target triple matches my machine (Apple Silicon, not "Intel").
  • The SHA-256 of my archive matches its line in SHA256SUMS.txt.
  • The binary is executable (chmod +x / install -m 0755) and on PATH.
  • webfang --version prints a version.
  • On macOS, com.apple.quarantine is cleared if the first run was killed.
  • I know where my output/ directory is, and that uninstalling will not remove it.

Next step

Data Flow & Privacy

This page is the data-flow disclosure: what leaves your machine when you run webfang, to which destinations, under what conditions, what is stored locally, and how to turn every flow off. Everything stated here is enforced by scripts/check_data_flow_doc.sh against the source tree, so it cannot silently drift from the code.


Quick path: run with zero AI egress

  1. Do not pass --clean-ai or --extract-with-llm, and do not use the MCP AI tools — no AI request is ever made without an explicit flag or tool call.
  2. Do not configure an AI provider in the providers config — without one, an AI request fails loudly at startup with exit 78, it never falls back to a remote default.
  3. Run with --offline on cleaning paths — every embeddings provider (remote or loopback) is rejected, exit 78, before any endpoint is inspected.
  4. Scraped-site traffic is inherent to crawling: fetching the pages you asked for is the product. Restrict it with the crawl scope flags; every destination is checked by the SSRF guard before a socket exists.

What leaves the machine

DestinationWhenWhat is sentHow to turn it off
The site being scrapedEvery crawl/scrapeHTTP GET requests for the pages (and, with asset download, for images/documents) — browser-like fetches via the TLS-fingerprinted client, or by the real Chrome browser itself under --js-strategy fullOmit the URL(s); scope with crawl flags
The operator-configured LLM endpoint (POST {base_url}/chat/completions)--extract-with-llmExtracted text chunks plus the extraction schemaOmit the flag; or configure no completion provider (exit 78 at startup)
The operator-configured embeddings endpoint (POST {base_url}/embeddings)--clean-ai (and the MCP semantic_cleaner tool)Extracted text chunks (bounded by --max-chars)--offline; or configure no embedding provider

There is no other network destination: no telemetry, no metrics, no update check, no crash reporting. Every HTTP fetch in the product goes through the SSRF-guarded downloader or through the two OpenAI-compatible AI adapters above. (The historical Hugging Face / local-ONNX paths were removed — see the ONNX migration page.)

Proxy posture: the proxy environment is ignored by design

WebFang dials directly. The proxy environment variables (HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXY) are ignored for every fetch path — crawling, assets, and both AI adapters — on every platform.

This is deliberate, not an omission. The HTTP client honours those variables by default, and under a proxy the guard's validating DNS resolver resolves the proxy's host instead of the destination. Two consequences made silent proxy support unacceptable:

  • A corporate proxy on a private range (RFC1918) — the common deployment — is rejected by the SSRF guard before a socket exists, with the operator seeing an SSRF diagnosis instead of a working proxy.
  • When the proxy itself is allowed, the destination's IP checks are delegated to the proxy: the client never resolves the destination, so the "forbidden ranges are rejected before the socket exists" guarantee stops covering it.

Where this is pinned: DefaultSsrfGuard::secure_client* applies no_proxy() at the single client-construction choke point, and tests/proxy_posture_e2e_test.rs proves that a request with HTTP_PROXY set reaches the target with zero requests observable at the proxy (both through the downloader and the webfang binary). webfang doctor states the posture on every run.

If you cannot dial out directly (mandatory corporate proxy): run a local forwarding sidecar, point the provider's base_url at it, and opt into loopback endpoints for that provider — the sidecar, not WebFang, talks to the corporate proxy. WebFang never reads the proxy environment itself.


The payload contract: extracted text, never raw HTML

The remote model never receives your raw page source. Before any text is sent to an AI endpoint it passes through the text-extraction stage:

  • The semantic cleaner extracts readable text from the page HTML — first a legibility parse, falling back to a plain tag-strip — and then chunks that text under the --max-chars budget. This is the invariant stated in the orchestrator: clean (raw HTML never reaches the LLM). Implementation: application/llm_extraction.rs, and infrastructure/semantic_cleaner.rs::extract_text_content.
  • The structured-extraction path (--extract-with-llm) composes the same cleaner with the chunker, so the completion payload is chunked extracted text plus your schema — never the page's HTML.
  • Authentication: requests carry an Authorization: Bearer header only if the provider config supplies a credential; a provider with auth: { source: "none" } sends an anonymous request. Credentials resolve locally at startup; they are never logged.

Endpoint reachability is enforced by the same fetch-guard chain used for crawling (entry validation → timeouts → redirect policy → SSRF at the socket dial → retry classification → capped body read). Loopback embedding endpoints are rejected unless explicitly opted in, and --offline rejects every provider before the endpoint is even inspected.


Log and trace content policy

The observability stack (structured logs on stderr, the --trace-file JSONL) carries operational metadata only:

  • What you will find: timestamps, levels, span and target names, URLs, depths, HTTP statuses, durations, pacing waits, error messages with their stage, and correlation identities. The debugging guide documents every span and field.
  • What never appears: page bodies, HTML, extracted Markdown, chunk text, or provider credentials — no tracing field in the product crates carries page content. The JSONL is identifiers and outcomes, not payloads.
  • MCP responses: every tool result that carries third-party text is neutralized (ANSI/control characters stripped), byte-capped with a visible truncation marker, and wrapped in a nonce-delimited UNTRUSTED provenance envelope. See docs/security/prompt-injection-policy.md for the consuming rules.
  • Trace files are local, ordinary files at the path you choose with --trace-file. They are never uploaded anywhere; delete them whenever you like.

Local retention

Path (Linux / XDG)What lives thereCreated byRemoval
~/.webfang/crawl.dbSQLite database (crawl results / RAG export)Any run that persists results; override with WEBFANG_DB_PATH / --db-pathDelete the file
~/.cache/webfang/state/Export state JSON plus the JSON-per-domain record store (one versioned envelope per site)Resume/checkpoint runs; override with --state-dirDelete the directory
<run persistence root>/.webfang-batch-capture.jsonlThe batch spool: every page body the batch fetched, verbatimBatch runs, until the batch finishes and it is discardedDiscarded at the end of a healthy batch; a crashed run leaves it, and the next batch run sweeps stale spools — or delete it by hand
~/.config/webfang/config.toml and the encrypted credential store credentials.ageConfiguration and webfang connectDelete the directory (also part of uninstall)
output/ (under your working directory)Exported Markdown/JSON and downloaded assetsEvery run that exports or downloadsDelete at will
Your --trace-file pathThe JSONL trace (metadata only, see above)Runs with tracing enabledDelete at will

Windows/macOS use the platform equivalents documented in installation.md; the paths above follow the standard XDG conventions the platform-paths helper enforces.

SQLite state and the WAL sidecar files (-wal/-shm) have their own recovery procedure: see the State Recovery Runbook before deleting a database you still need.


Review checklist

  • I know every destination my run can talk to (the table above) and none of them is hard-coded by webfang.
  • I can run a full crawl with zero AI egress (Quick path above).
  • I understand the AI payload is extracted text chunks, never raw HTML.
  • I know where every retained file lives and how to remove each one.

Release Feature Matrix

Every release publishes three binaries — the webfang CLI and the two MCP servers (webfang-mcp, webfang-mcp-stdio) — and every binary is built from the same feature set, always. This page answers "what can the binary I just downloaded do?" capability by capability, without you having to read the release workflow.

The capabilities below are compile-time Cargo features, not runtime configuration: if a release binary was built without a feature, the capability is not in the file, and no flag combination brings it back. The bottom of this page carries a machine-readable manifest of the feature set, and CI drift-checks it against the [features] table of crates/webfang_cli/Cargo.toml (scripts/check_release_feature_matrix.sh, a required step of the repo-guards job) — so this page cannot silently rot when a feature is added or removed.

What is NOT covered here: which platforms get artifacts (see Installation) and how to verify a download. This page is only about compiled-in capability.


The three artifacts

BinaryCrateWhat it is
webfangwebfang_clithe scraper CLI
webfang-mcpwebfang_mcpMCP server, Streamable HTTP on 127.0.0.1:8080/mcp
webfang-mcp-stdiowebfang_mcpMCP server over stdin/stdout (JSON-RPC), for desktop clients

The exact release build is one invocation (.github/workflows/release.yml, step "Build release binary"):

cargo build --release --locked \
  --features "webfang_cli/mcp" \
  --target <target-triple> \
  -p webfang_cli -p webfang_mcp --bins

No capability feature is enabled: webfang_cli's default = [], and chromium, persistence, console and adaptive-selectors are capability features that no release artifact carries. webfang_cli/mcp is an explicit-empty marker (mcp = []): the key must exist because the CLI test harness and several scripts build with --features mcp, but it selects nothing — it puts zero MCP code into the webfang binary. The MCP surface lives in the webfang_mcp crate, which is built as its own two binaries and shipped in its own archives.


Capability matrix

Capabilitywebfang CLI (release)MCP servers (release)Build from source
MCP (36 tools)Not embedded. The CLI exposes no MCP flag. The servers are the two separate binaries — see MCP servers.Included — this is what those binaries are.cargo build -p webfang_mcp --bins
--js-strategy full (Chromium rendering)Rejected at preflight with an honest error — the Chromium downloader is not compiled in. The default static strategy and the 3-layer hybrid strategy are unaffected.The get_accessibility_snapshot tool is registered but answers with a feature-gated error instead of driving a browser.--features chromium (needs a Chromium/Playwright runtime at run time)
SQLite persistenceAbsent. The crawl.db checkpoint store does not exist; file-based --resume state is unaffected.Absent.--features persistence (enables the SQLite store; --elastic / --db-path ingestion paths use it)
Adaptive selector learningAbsent — the flags do not exist in this build.Absent.--features adaptive-selectors
tokio-consoleAbsent. Tracing via --trace-file is unaffected.Absent.--features console
dev-tracingNever in a release — a development aid, not a capability.Never.--features dev-tracing

Historical note: the former ai feature (local ONNX semantic cleaning) was removed in ADR-0004 paso 6, not feature-gated — semantic cleaning is runtime-configured through --embedding-provider in every build, and --offline rejects every provider. See the migration guide.


The machine-readable contract

This manifest is the guard's input, not reader-facing prose: it is parsed by scripts/check_release_feature_matrix.sh and must mirror the [features] table of crates/webfang_cli/Cargo.toml exactly — every feature key present once, default absent, and each kind matching how the feature is declared (capability = forwards a webfang_core feature that gates code or dependencies; marker = an explicit-empty feature that selects nothing). Adding or renaming a feature in the manifest without touching Cargo.toml — or the other way around — fails the repo-guards job.

chromium           | capability | --js-strategy full rejected at preflight (#685)
mcp                | marker     | explicit-empty marker (#1815 T4); MCP ships as the webfang_mcp binaries
persistence        | capability | SQLite store absent from release artifacts
console            | capability | no tokio-console in release artifacts
adaptive-selectors | capability | flags absent from release artifacts
dev-tracing        | capability | development tracing, never in a release

Release Process

Two tools cut a release, and each owns exactly one half:

toolowns
release-plzthe version bump, CHANGELOG.md, the Release PR, the v* tag
release.ymlthe binaries, the archive normalization and the GitHub Release page, triggered by the tag

The configuration lives in release-plz.toml and the workflow in .github/workflows/release-plz.yml. This page explains the three decisions that are not written in either file: which commits may cut a version, how to check the filter without touching the remote, and what the support metadata means.

Which commits may cut a version

release-plz bumps on every commit by default, so a test:, ci:, docs: or plain refactor: commit publishes a tag, a build matrix and a GitHub Release for a change no user can observe. The workspace sets release_commits to fix that:

the user-facing types always release; any type releases when it is breaking.

The exact pattern is in release-plz.toml — it is the source of truth and it is pinned by scripts/test_release_commits_regex.sh, which reads it from the file and runs a table against it (a duplicated regex keeps passing while the configuration is broken).

commitreleases?why
feat:, fix(scope):, perf:, revert:yesuser-visible
chore(deps): bump …yesthe binary changes and the user cannot rebuild it
any type with ! — refactor(api)!:, docs!:yesa breaking change is user-visible by definition
refactor:, refactor(scope):nobehaviour-preserving; it ships with the next real release
test:, ci:, docs:, style:, chore:nonothing the user can observe

Two consequences worth knowing:

  • A breaking change is never delayed by a non-breaking type. The ! marker is the canonical one (with or without a scope), so refactor(exporter)!: releases even though refactor: does not. This repo has shipped breaking refactors that way.
  • Filtered commits are not lost. They still appear in the CHANGELOG.md of the next release that does fire, in their own section. The filter decides when a version is cut, not what the notes say.

Checking the filter without touching the remote

There is no --dry-run for the Release-PR path, but there is a safe pre-check: run the PR command with a token that cannot possibly work.

release-plz release-pr --git-token invalid

It resolves the versions, logs no commit matches the 'release_commits' regex per package when the window has nothing releasable, and exits without opening a PR or modifying the working tree. A window that does match fails later, at the first API call — which is exactly the signal you want: it means a PR would have been proposed.

Do not use release-plz update as the pre-check. It mutates the checkout (it rewrites Cargo.toml and the changelog), and on a workspace where every package is filtered it writes the new version anyway: it puts the bumped workspace version into Cargo.toml without updating Cargo.lock, leaving a tree that cannot build. That is a behaviour of the update command, not of the PR path — but it makes it the wrong tool for a quick look, and a dangerous one to run in a working checkout. Measured on release-plz 0.3.168 and 0.3.169 (the version the GitHub Action installs).

A probe commit has to touch a file of the crate. release-plz filters by the files of each package before it evaluates the commit message, so an empty commit (git commit --allow-empty) reports no commit matches no matter what the subject says, and a commit that only touches docs/ or scripts/ does the same for all three packages. To probe the positive path, append a line to a file inside the crate you expect to release (for example crates/webfang_core/src/lib.rs).

Also note that the GitHub Action installs its own release-plz version; a local CLI may be a different one. The pre-check is a smoke test, not the verdict — the verdict is the next real run of the workflow.

A commit that only moves Cargo.lock does not cut a version

With release_commits active, a commit that changes only Cargo.lock reports no commit matches the 'release_commits' regex and does not release — even when its subject matches (chore(deps):, or fix:). Measured on release-plz 0.3.168 and 0.3.169, with a clean window (a single lock-only commit on top of v8.0.0): without release_commits the same commit produces next version is 7.7.2; with the filter, nothing. release-plz decides by the files of each package, and Cargo.lock lives at the workspace root, outside every package.

The hole only bites when no other commit in the window releases: a window with a feat: or fix: releases anyway and carries the lock change with it (that is how #2433 and #2361 shipped). It is not a hypothetical shape, though: with the filter, test:, ci: and docs: commits accumulate without cutting a version, so a lock-only bump can easily be the only thing in the window.

The convention that closes it: the commit has to touch a packaged file of the affected crate, so the change is attributed to that package. In this workspace that rules out the root Cargo.toml: the dependency requirements live in [workspace.dependencies], and the root file belongs to no package — raising libc there still reports no commit matches (measured on 0.3.168 and 0.3.169). What works is a file inside the crate:

  • if the crate declares the dependency with its own version requirement, raise it there (a real change: it records the new minimum);
  • when the dependency is transitive and the crate only says libc = { workspace = true }, add a comment in that Cargo.toml naming the advisory, e.g. # <crate>: <RUSTSEC-...> patched in the lockfile. — measured on release-plz 0.3.168 and 0.3.169: all three packages get the patch version.

The comment is a marker, not a fix: prefer the requirement change whenever the crate actually owns one.

Why a Release PR cannot be generated against an untagged main

release-plz derives the next version from the commits between the newest v* tag and main, and in git_only mode that tag list is the local one. A main that is ahead of its own tag — the version bump merged, the tag seconds away — is a state it cannot read, and it proposes the version that is already there.

The PR job therefore syncs its checkout to origin/main and waits, bounded, for the tag of the declared version before running release-plz (scripts/await_release_tag.sh). When the tag arrives, there is nothing to release; when it does not, the step fails loudly instead of opening a phantom Release PR. A repository with no v* tag at all is still the initial release and is not awaited.

Reverts must be conventional

The filter reads the commit subject, and git revert writes Revert "…" — capital R, not a conventional type — which matches nothing. A revert that must ship has to be recorded as revert: … (lowercase), either by amending the subject or by writing the revert commit by hand. Otherwise the change it undoes stays published until the next release that fires for another reason.

What released_at means in .github/support.json

.github/support.json describes the support lines, and its two date fields answer different questions:

  • latest — a fact about what this repository published. It is written by scripts/refresh_support_latest.sh, the single writer, and it moves on every patch.
  • released_at — the date the minor line went stable. It is written once, during the rotation of a minor (scripts/rotate-stable.sh --pending-publication), and a patch does not move it. scripts/rotate-stable.sh reads it, so it is a policy field, not a label: it is the clock the support window is measured from.

So seeing latest at a patch while released_at still holds the minor's date is the correct state, not drift.

Approving the release PR's linked issue

Validate PR metadata checks that every linked issue carries status:approved, and it reads the label from the issue through the API. The pull_request workflow that runs the check listens to the PR's own events (opened, synchronize, labeled, …), so approving the issue is not a PR event: the check does not re-run on its own and the run keeps showing the old verdict.

If the metadata check is red after the issue was approved, re-run the failed job — gh run rerun <run-id> --failed, or "Re-run jobs" in the UI. Both attempts observed on #2788 and #2791 were manual re-runs (same triggering_actor, event pull_request), not label events.

set-version is not a release flow

release-plz set-version exists to correct a version that was not set as expected (for example a commit that was not prefixed correctly). It is a correction tool, not a way to cut a release: it does not open a PR, it does not write the changelog entry, and it has not been exercised in this repository. If a release has to be forced for a window that the filter does not match, treat that as an incident to investigate — not as a routine path.

Debugging & Trace Analysis

WebFang ships built-in, always-available observability. No external collector, no feature flags, no infrastructure: run with --trace-file and post-process the JSONL with jq.

Mandate: every new feature or hot path must be observable. See the "Observability (MANDATORY)" section of ../../AGENTS.md.


The stack

LayerWhat it doesAlways on?
FileTraceLayerWrites every tracing span/event to a JSONL file (--trace-file)✅ Yes
Correlation IDsNative CorrelationId (UUID v7 trace_id + span_id); one trace_id per operation, unique span_id per unit of work✅ Yes
Structured loggingtracing-subscriber to stderr (-v/-vv/-vvv, --log-format json)✅ Yes
Tokio ConsoleAsync task/resource inspection for concurrency bugs--features console

There is no OpenTelemetry (removed in #356). If you need a metric, emit a structured tracing event and query it from the JSONL.


Generating a trace

# Full trace + verbose logging
webfang --url https://example.com --trace-file debug.jsonl -vvv

# Batch / crawl
webfang --url https://example.com --max-pages 100 --trace-file crawl.jsonl -v

Each line of debug.jsonl is a JSON object:

{
  "timestamp": "2026-01-29T10:00:00.123Z",
  "level": "INFO",
  "target": "webfang_core::application::crawler::engine",
  "span": "crawl_page",
  "span_id": "0000000000000042",
  "parent_id": "0000000000000001",
  "trace_id": "0000000000000001",
  "fields": {
    "url": "https://example.com/page1",
    "depth": 1,
    "correlation_id": "00-01949e0e8b8e70008000000000000001-0000000000000042-01"
  }
}

Top-level trace_id is the root span Id (16-hex), one per run, EPHEMERAL to the process/run (identity-within-run, not a durable global identity). Do not persist or join on it across runs. Durable run correlation is the CorrelationId UUID in span_fields.trace_id / span_fields.correlation_id (W3C traceparent).

When a span closes, a second record type is emitted carrying a top-level span_duration_ms (wall-clock milliseconds) — this is what the "Slowest spans" query below reads:

{
  "timestamp": "2026-01-29T10:00:00.456Z",
  "record": "span_close",
  "level": "INFO",
  "target": "webfang_core::application::crawler::engine",
  "span": "crawl_page",
  "span_id": "0000000000000042",
  "parent_id": "0000000000000001",
  "trace_id": "0000000000000001",
  "span_duration_ms": 333,
  "span_fields": {
    "url": "https://example.com/page1"
  }
}

Query cookbook

A ready-made script lives at scripts/analyze-trace.sh. The most useful queries:

Reconstruct one run by top-level trace_id

ROOT=0000000000000001
jq -c "select(.trace_id == \"$ROOT\")" debug.jsonl

$ROOT is the 16-hex root span Id (top-level trace_id, EPHEMERAL to the run). It returns every page plus errors for that run. The same query is scripts/analyze-trace.sh debug.jsonl trace $ROOT.

All errors, with full context

jq -c 'select(.level == "ERROR") | {target, url: .fields.url, stage: .fields.stage, error: .fields.error, msg: .fields.message}' debug.jsonl

The cause field (connection failures)

The error events carry a cause field: the machine-readable class of a connection failure, as a stable wire value. It is not a message fragment — the values are a contract:

causemeaningobserved end to end
dns_resolutionthe name never came back: the resolution itself failed or hungyes
connect_refusedthe peer refused the connection (RST)yes
connect_timeoutthe name resolved and the TCP connect did not complete in timeyes
read_timeoutthe connection was established and the response never arrivednot yet
connection_closedthe peer closed the connection before answering (EOF/reset)not yet
tls_handshakethe TLS handshake failednot yet
unknownthe classifier ran and could not decide — see belowyes

The three values marked not yet are DECLARED, not measured: the transport does not expose the evidence they need (an EOF and a stalled response reach the classifier with no readable kind), so they arrive as unknown today. Their names and their meaning are already fixed — a value never changes meaning — but they are not expected in a trace until the transport surfaces the evidence behind them.

The field is the contract; the message is prose. Every classified failure also names its cause in the message, in Spanish, before the transport's own words. A consumer must still read the cause field: the message is written for a human and its wording may change, while the wire value will not.

The field is omitted when the failure carries no classified cause. That absence means "this failure was not classified" — it does NOT mean "this was not a network failure": a transport failure whose evidence the wiring cannot read is reported without the field. unknown is the classified version of that same fact, and it is the value to look for when a network failure carries no better answer.

unknown is the honest answer, and it is not a shrug. It covers two cases: a chain the classifier could not read by type, and a name resolution that was still in flight when the request failed but could not be attributed to that request — typically because another request for the same host was running at the same time. The rule is deliberately conservative: a failure is only called dns_resolution when the evidence points at THAT request's own resolution. With one request in flight per host the attribution is reliable; with several, it is not, and the value is unknown rather than a wrong cause.

# the failures by cause
jq -r 'select(.fields.cause) | .fields.cause' trace.jsonl | sort | uniq -c | sort -rn

# every failure with its cause, in order
jq -c 'select(.fields.cause) | {url: .fields.url, cause: .fields.cause, error: .fields.error}' trace.jsonl

Slowest spans (where the time goes)

jq -r 'select(.span_duration_ms != null) | [.span_duration_ms, .span] | @tsv' debug.jsonl | sort -rn | head -20

Time distribution per pipeline stage

jq -r 'select(.span == "pipeline_stage") | .fields.stage' debug.jsonl | sort | uniq -c | sort -rn

Crawl progress over time

jq -c 'select(.fields.message? == "crawl progress") | {pages: .fields.pages_crawled, pct: .fields.progress_pct, eta_s: .fields.eta_secs}' debug.jsonl

Final crawl summary

jq -c 'select(.fields.message? == "crawl completed")' debug.jsonl

Count operations by span type

jq -r 'select(.record != "span_close") | .span // "event"' debug.jsonl | sort | uniq -c | sort -rn

span_close records share the same .span name, so they must be excluded when counting events (otherwise every span is double-counted).

URLs that failed

jq -r 'select(.level == "ERROR") | .fields.url // empty' debug.jsonl | sort -u

Was this run slow, or was it PACED?

Pacing and admission waits emit rate limit wait / semaphore acquire events at DEBUG level. -vvv is not required for these: the --trace-file layer always runs at TRACE, so they land in the JSONL whenever a trace file was requested. They exist because a wait used to be invisible — a throttled crawl looked exactly like a slow network.

# Total time spent waiting, per pacing scope.
scripts/analyze-trace.sh debug.jsonl pacing
ScopeMeaning
crawl_discoveryPer-page pacing inside the crawl engine
cli_scrapePer-URL pacing in the CLI scrape phase
batch_scrapePer-URL pacing in the batch scrape path
http_clientPer-session quota in the HTTP client

A scope with total_waited_ms near zero means pacing was free and the run's time went to the network. A large cancelled count means shutdown was shedding queued work, not that the fetches were slow.

Which targets absorbed the wait (each event names the URL whose permit it was waiting for, and the page's correlation_id):

scripts/analyze-trace.sh debug.jsonl pacing-hot 10

Reconstruct one page's pacing from its own identity:

CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.fields.correlation_id? == \"$CID\" and .message? == \"rate limit wait\")" debug.jsonl

The same wait is also folded into the page's own span, so a page record answers "how much of this page was pacing" without a join:

jq -r 'select(.span == "crawl_page" and .record == "span_close")
       | [.span_fields.outcome, .span_fields.rate_limit_wait_ms, .span_duration_ms] | @tsv' \
  debug.jsonl | sort -k2 -rn | head

Admission and dispatch queues

scripts/analyze-trace.sh debug.jsonl admission

One row per operation, covering the downloader resource governor (browser.resource_governor.acquire), the AI inference pool permit (ai.inference_pool.acquire) and the AI pool dispatch queue (ai.inference_pool.dispatch). max_in_flight is the pool occupancy the waiter observed. If max_waited_ms is high and max_in_flight equals the worker count, the pool was saturated — raise the worker budget or lower the concurrency feeding it, rather than raising timeouts.


Spans you will see

SpanEmitted byKey fields
crawl_site / crawl_site_with_optionscrawler::enginecorrelation_id, trace_id, seed_url, max_depth, max_pages
crawl_pagecrawler::engine::run_crawl_taskcorrelation_id, trace_id, url, depth
executepipeline::PipelineExecutorurl, stages
pipeline_stagepipeline::PipelineExecutorstage, url
export_batchJsonlExporter / VectorExporter / FileExporterexporter, documents
scrape_single_url → scrape_singlecrawler::discovery::scrape_single_urlurl (outer), correlation_id, trace_id, url (inner, #501)
scrape_with_configscraper_serviceurl, correlation_id, trace_id, has_downloads
scrape_multiple_with_limitscraper_serviceurls, concurrency

Identity follows the root-child contract: the OPERATION owns one root CorrelationId, and every unit of work derives .child() from it — same trace_id, fresh span_id. In a CLI run the orchestrator mints the root and announces it with a run identity event (correlation_id, trace_id in .fields); scrape_multiple_with_limit does the same with a scrape_multiple identity event. So in a multi-page scrape:

  • Top-level trace_id is the single logical run id: the root span Id (16-hex), EPHEMERAL to the run. Reconstruct the whole run offline with select(.trace_id == $ROOT) — pages plus errors, no orphans.
  • span_fields.trace_id is the shared run-root UUID (CorrelationId, durable across systems) across all page spans.
  • span_fields.correlation_id (full W3C traceparent) is unique per page; its trace part is the run-root UUID without dashes.

Identity is declared at span creation time: FileTraceLayer snapshots the span's declared fields in on_new_span, and #[instrument] can only see function arguments. Fields that are only known at the END of an operation (an outcome, an HTTP status) are therefore declared tracing::field::Empty and filled in with tracing::Span::record, which on_record merges into the same snapshot — they reach the JSONL both in the span_fields of later events and in the span_close record. ScrapedContent and the RAG exports carry the same identity, so an exported document's correlation_id matches its page's span_fields.correlation_id:

# Reconstruct an entire run by the single top-level trace_id (root span Id)
ROOT=0000000000000001
jq -c "select(.trace_id == \"$ROOT\")" debug.jsonl

# Same run by the durable run-root UUID (CorrelationId in span_fields)
CUUID=01949e0e-8b8e-7000-8000-000000000001
jq -c "select(.span_fields.trace_id == \"$CUUID\")" debug.jsonl

# The run-root identity (the `run identity` event carries it in .fields)
jq -c 'select(.message? == "run identity") | .fields' debug.jsonl

# Every page identity present in the trace
jq -r '.span_fields.correlation_id // empty' debug.jsonl | sort -u

# Reconstruct one page's scrape by its correlation_id
CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.span_fields.correlation_id? == \"$CID\")" debug.jsonl

Events (not spans): run identity, scrape_multiple identity, crawl progress, crawl completed, and any log_scrape_error(...) error carrying error, url, stage, trace_id.


Concurrency debugging (Tokio Console)

For deadlocks, starved tasks, or async resource leaks, use the Tokio Console:

RUSTFLAGS="--cfg tokio_unstable" cargo run --features console -- --url https://example.com

This opens an interactive TUI showing live tasks, their states, and poll times.


Troubleshooting

See troubleshooting.md for common problems (slow crawls, silent page failures, WAF blocks, async deadlocks, poor content) and how to diagnose each with the trace queries above.


For contributors

When you add a hot path or operation, follow the observability mandate in AGENTS.md:

  • #[instrument(skip(...), fields(url = %url, ...))] on the function.
  • Propagate the operation's CorrelationId; derive .child() per unit of work.
  • Use log_scrape_error(...) on error paths (never a bare warn! for an operational error).
  • Use .instrument(span) on async futures — never hold span.enter() across .await.
  • Verify with: webfang ... --trace-file debug.jsonl -vvv and the queries above.

Correlating a wait with the page that paid it

A pacing wait happens BEFORE the page span is created (the permit is taken before any socket opens), so it is emitted as its own event rather than inside the page span. It is still attributable, twice over:

# 1. By identity — the wait carries the same correlation_id the page span does.
CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.fields.correlation_id? == \"$CID\")" debug.jsonl

# 2. By span field — crawl_page records the wait it absorbed.
jq -c 'select(.span == "crawl_page" and .record == "span_close")
       | {url: .span_fields.url, outcome: .span_fields.outcome,
          paced_ms: .span_fields.rate_limit_wait_ms,
          total_ms: .span_duration_ms}' debug.jsonl

A page where paced_ms is a large share of total_ms was queued, not slow. That distinction is the whole reason the wait emits at all.

Testing Guide

End-to-end (E2E) tests live as integration test crates under tests/ and invoke the real webfang binary via assert_cmd. Mock HTTP servers (wiremock) stand in for target sites and tempfile::TempDir captures scrape output.

Test crates

CrateFileGateWhat it covers
behavioraltests/behavioral/main.rsnone (always built)Single-page scrape, CLI help, unreachable host, slow server, obsidian frontmatter, CSS-selector and full-page extraction (tests/behavioral/cli/)
cli_binarytests/cli_binary_test.rsnone (always built)--version, --help, network-error exit codes

No E2E crate is feature-gated. An earlier revision of this table listed a cli_behavioral crate gated on images and documents; that crate does not exist and that #![cfg(all(feature = "images", feature = "documents"))] gate never existed anywhere in the source tree — its coverage (obsidian tag/metadata/wiki-link conversion) lives ungated inside behavioral, at tests/behavioral/cli/obsidian_test.rs.

The images and documents marker features themselves were deleted in #1813: they selected no capability, and turning them off silently shrank a MIME table instead of shrinking anything else. Do not reintroduce them as empty markers — add a feature when it gates something real.

Running tests

# all E2E crates
cargo nextest run --test behavioral --test cli_binary

# a single crate
cargo nextest run --test behavioral

# a single test (libtest, prints the full snapshot diff on mismatch)
cargo test --test behavioral selector_h3_extracts_only_h3

Ignored tests (e.g. optional live-site checks) are excluded by default; run them with cargo nextest run --test behavioral --run-ignored ignored-only.

Mocking a remote embeddings provider

--clean-ai needs a configured remote embedding provider: ADR-0004 retired the local ONNX stack, and --export-format vector needs --clean-ai (preflight #796). So a test that exercises the vector export has to provide one.

crates/webfang_core/tests/common/embeddings.rs does it hermetically — the provider IS the test's own wiremock server, the credential is anonymous, and nothing leaves the machine:

let t = BehavioralTest::new().await;
crate::common::embeddings::mount_embeddings(&t.server).await;
let provider_config = crate::common::embeddings::provider_config(&t.server);

t.scraper_cmd()
    .env("WEBFANG_CONFIG", &provider_config) // absolute path: the CLI refuses a relative one
    .arg("--embedding-provider")
    .arg(crate::common::embeddings::PROVIDER_ID)
    .arg("--export-format")
    .arg("vector")
    .arg("--clean-ai")
    .assert()
    .success();

Two details the fake has to honor: the endpoint answers one vector per input (the adapter rejects a body whose vector count does not match the text count), and the provider is declared in the config file with auth = { source = "none" } and allow_loopback = true — the loopback opt-in is config-file-only by design (#1462); there is deliberately no CLI flag for it.

Snapshot testing with insta

Content assertions use insta snapshots instead of brittle content.contains(...) checks, so a full output change is reviewed as a diff rather than a silent boolean flip.

Review gate (RED → GREEN)

cargo insta is not installed in this environment. Use the env-var workflow instead:

  1. RED — first run fails because the .snap is missing or differs, and a *.snap.new pending file is written next to it:

    cargo nextest run --test behavioral
    
  2. GREEN — regenerate and accept the pending snapshots, then re-run with no flag to confirm they are now stable (no new *.snap.new should appear):

    INSTA_UPDATE=always cargo nextest run --test behavioral
    cargo nextest run --test behavioral        # must stay green
    
  3. Inspect the generated *.snap files, then stage them with the code change.

*.snap.new is git-ignored (see .gitignore). Never commit a *.snap.new; commit the accepted *.snap.

Where snapshots live

insta resolves the snapshot directory from the module where assert_snapshot! expands. The thin assert_snapshot_* wrappers therefore live at each test crate's root module so snapshots land where the suite expects:

  • tests/behavioral/snapshots/ — root behavioral snapshots
  • tests/behavioral/cli/snapshots/ — obsidian snapshots (local helper inside cli/obsidian_test.rs)
  • tests/snapshots/ — cli_binary__*.snap

Redaction conventions

Scrape output embeds per-run, machine-specific, and non-deterministic values. A shared helper, tests/common/cli_harness.rs::redact_nondeterministic, collapses them before snapshotting so approved snapshots stay stable across machines and runs:

LeakRedacted to
TempDir absolute path<OUT_DIR>
ANSI color escape sequences(stripped)
ISO-8601 timestamps (timestamp_utc, scrapeDate, scrape_date, …) with or without fractional seconds and any offset/Z<TIMESTAMP>
Wiremock 127.0.0.1:<port>127.0.0.1:<PORT>

cli_behavioral additionally emits a bare date: frontmatter field (date only, no time component) that the helper cannot catch, so assert_content_snapshot applies an insta add_filter for date: \d{4}-\d{2}-\d{2} → date: [DATE] (see tests/cli_behavioral_test.rs).

Adding a new snapshot test

  1. Build the scrape output through the shared harness (BehavioralTest / cmd).
  2. Call the crate's assert_snapshot_* wrapper (root module) or, for free-text content, assert_content_snapshot in cli_behavioral.
  3. If a new non-deterministic field appears, extend redact_nondeterministic (centralized) rather than adding a per-test hack.
  4. Generate + accept via INSTA_UPDATE=always, then verify with a plain run.

Lint

cargo clippy -p webfang_core --test behavioral --test cli_binary -- -D warnings

Gate clippy on the specific test crates (not --tests): webfang_core's own lib tests have a pre-existing tokio::time::pause failure that requires the test-util feature and is out of scope for E2E changes.

Coverage exclusions (LCOV)

Defensive error paths — invariants by design — must not drag down the codecov/patch target (80% on new lines). Annotate them with LCOV exclusion markers (issue #527).

Policy

Only annotate arms that "should not happen in normal operation":

  • internal / mutex-poisoning / integer-overflow invariants
  • compile-time-constant failures (CSS selectors, regexes, hardcoded URLs)
  • panic/expect paths guarded by proven invariants (e.g. NonZeroU32 after a zero-check, chunks_exact slice conversion)

NEVER annotate business paths: reachable errors like HTTP connection failures, parse errors, or config validation. Reachable error handling is exercised by tests and counted like any other code.

Syntax

  • Single statement: // LCOV_EXCL_LINE on its OWN comment line immediately ABOVE the code line — never inline on the code line.
  • Multi-line arm/block: // LCOV_EXCL_START above the block and // LCOV_EXCL_STOP below it, each on its own line.
  • Every marker site carries a justification comment starting with // defensive: <variant> <rationale> — merged into the marker line or as a preceding line.

Safety net

Excluded paths are still mutation-tested: a surviving mutant in the weekly cargo-mutants baseline, or in a PR touching gated hot paths (cargo-mutants PR diff), is reported. The markers only affect coverage accounting — they do not affect mutant survival.

Hot-path rule

In files under .cargo/mutants.toml globs, markers MUST be own-line comments (never inline) so the diff adds no mutable code lines.

Never lower codecov.yml thresholds; use markers per path instead.

Troubleshooting

Common problems and how to diagnose them with WebFang's built-in tracing.

Install and uninstall paths live in installation.md. If the binary never started, that page is the first place to look.

Generate a trace first: webfang --url <URL> --trace-file debug.jsonl -vvv, then query it with scripts/analyze-trace.sh or jq. See debugging.md for the full query cookbook.


The downloaded binary won't run

These are install-time failures — they happen before any tracing exists, so none of the trace workflow below applies yet. Each symptom names its cause and its one-line fix; the full verified commands are in installation.md.

SymptomCauseFix
GLIBC_2.xx not found (Linux)Distro older than the ubuntu-latest builder (glibc 2.39); there is no musl artifactBuild from source, or use a newer distro
command not found: webfangExtracted but never made executable, or not on PATHchmod +x webfang, then sudo install -m 0755 webfang /usr/local/bin/webfang
cannot be opened because the developer cannot be verified (macOS)com.apple.quarantine xattr on the downloaded archive's contentsxattr -d com.apple.quarantine webfang, then re-run
VCRUNTIME140.dll was not found (Windows)MSVC dynamic CRT / Visual C++ Redistributable absentInstall the VC++ Redistributable 2015-2022 (x64)
sha256sum: command not found (macOS)sha256sum is GNU coreutils; stock macOS ships shasumshasum -a 256 <archive> and compare with the SHA256SUMS.txt line
A hash does not match SHA256SUMS.txtTruncated/partial download, or the file is not from this releaseRe-download the archive; do not run the binary
No binary for your platform (Intel Mac)x86_64-apple-darwin is not built (the original reason was the ONNX prebuilt; the target list was not extended after the ONNX removal)Nothing to download; see Intel macOS
--js-strategy full or SQLite resume absentchromium / persistence are not in the release buildExpected; see what is not in the release binary
webfang-mcp or webfang-mcp-stdio missing after extracting webfang-<target>Each binary has its own archive; the CLI archive holds only webfangDownload webfang-mcp-<target> / webfang-mcp-stdio-<target>; see MCP servers

If a binary fails with something not in this table, verify the checksum first — it separates "my download is damaged" from "this is a real bug" before you file anything.


The crawl is slow

Diagnose:

scripts/analyze-trace.sh debug.jsonl slow 20      # slowest spans
scripts/analyze-trace.sh debug.jsonl stages       # time per pipeline stage

Common causes:

  • A single stage dominates (e.g. clean with the AI feature) — check the stages distribution.
  • Network latency / rate limiting — look for large gaps between crawl_page spans; consider --delay and concurrency tuning.
  • Export bottleneck — check export_batch span durations.

Pages are failing silently

Every operational error is logged as a structured ERROR event with url, stage, and (when available) trace_id.

scripts/analyze-trace.sh debug.jsonl errors       # all errors with context
scripts/analyze-trace.sh debug.jsonl urls-failed  # unique failed URLs

Common causes by stage:

stageMeaningFix
fetchHTTP/network failure or WAF challengeCheck connectivity; the site may be blocking — see WAF section below
extractContent extraction produced too little textThe page may be JS-rendered or non-article; try a CSS --selector or JS rendering

A local or internal target discovers nothing

Symptom. You point WebFang at a localhost, 127.0.0.1, 10.x, 192.168.x or 169.254.169.254 target and get nothing back. How loud that is depends on how the target is spelled:

$ webfang http://127.0.0.1:8080/ --dry-run     # an IP literal
Warning: SSRF detectado: la IP 127.0.0.1 del host '127.0.0.1' está prohibida (acceso a
red interna/cloud metadata bloqueado). La vista previa no descubrió URLs porque el guard
cortó la semilla antes de abrir conexión. Para un destino interno de confianza: …
$ echo $?
2
$ webfang http://localhost:8080/ --dry-run     # a name that resolves into the same range
Dry-run: 0 URL(s) would be scraped:
$ echo $?
0

The literal case names its own cause (since #1391): exit 2, the code a null discovery result already uses, with no Dry-run: line to misread. The hostname case still reads as an empty site, because the layer that refuses it — the connect-time resolver — answers with what looks like a DNS failure, and the preview cannot tell that from a broken name. Either way, the same target in a real crawl fails loudly.

What happened. The SSRF guard refused the target before any socket opened: zero packets leave the process, so nothing is retried and nothing can time out. Discovery itself still answers Ok with an empty URL list — inside a crawl, one refused URL is a legitimate skip, not a failed run — and the engine's own page-error count (crawl completed … errors: 1) is not part of that list. What changed in #1391 is that the preview asks the guard its own question before calling the result a success. The guard policy is the long-standing one: the scrape path has always paid it, the MCP crawl tools since #1355, and plain DOM discovery since #1369 retired the last entry point that fetched without the full guard chain.

What you see, by surface. The same refusal, spelled two ways, produces five different-looking outcomes:

You ranWhat you seeExit
webfang <literal-target> --dry-runWarning: SSRF detectado: la IP … está prohibida …, naming both hatches — and no Dry-run: line (#1391)2
webfang <hostname-target> --dry-runDry-run: 0 URL(s) would be scraped:0
webfang <target> (crawl)Failed to scrape <target>: URL inválida: SSRF detectado: la IP 127.0.0.1 del host '127.0.0.1' está prohibida (acceso a red interna/cloud metadata bloqueado)69
webfang <target> --single-pagethe same SSRF detectado line — the scrape path carries the same guard69
discover_urls_unified / discover_urls_recursive (library)Ok with an empty Vec—

Two exit codes for one refusal, and both are right. The crawl is loud because discovery returns nothing, plan_urls re-injects the seed, and the scrape path refuses it a second time — a scrape that produced no page is a network-class failure, so it leaves at 69. The preview is the other case: nothing failed to run, there was simply no URL to schedule, which is the null result the sitemap arms already report as 2 — now with its cause named, so the code stops reading as "the site is empty". --single-page never runs discovery, so it only ever shows the loud half. Do not read either code as "retry me and it might work": both are one policy refusal. Whether a path is seed-guarded at all depends on which of the four SSRF layers it wires; the layer map lives in docs/ssrf-layers.md.

How to tell a refusal from a genuinely empty site. The refusal is logged, never hidden — and for an IP-literal seed the exit code now says it too (2, with the cause named). A hostname that resolves into a forbidden range is the case that still looks silent, so the WARN line is the discriminator: at default verbosity it is already on stderr.

WARN webfang_core::domain::ssrf_guard: SSRF literal-IP target rejected at entry (no socket opened), host: 127.0.0.1, ip: 127.0.0.1

To query it instead of squinting at stderr, add --trace-file (with -v the engine also emits its run summary, where the refused seed shows up as errors: 1 on total_pages: 0):

webfang http://127.0.0.1:8080/ --dry-run -v --trace-file debug.jsonl
jq -c 'select((.message // "") | test("SSRF|crawl completed"))' debug.jsonl
scripts/analyze-trace.sh debug.jsonl errors

One trap inside the trap: the hostname row above is not a bug waiting to be fixed, and its download error: DNS error: name resolution failed is not a typo. The validating resolver refused to hand back the address, and at this boundary that is indistinguishable from a name that genuinely does not exist — which is why the preview keeps exiting 0 for localhost and 2 for 127.0.0.1. Literals and hostnames are checked by different layers; see docs/ssrf-layers.md.

Opting out, for a target you trust. There is no per-target allowlist flag; the disarmers are environment variables, and only the exact value 1 disarms anything:

# Literal address (127.0.0.1, 10.0.0.5, …): lifts the literal entry guard.
WEBFANG_DISABLE_SSRF_ENTRY_GUARD=1 webfang http://127.0.0.1:8080/ --dry-run

# Hostname that resolves into a private range: lifts the connect-time resolver.
WEBFANG_DISABLE_SSRF_RESOLVER=1 webfang http://localhost:8080/ --dry-run

# A local target that answers to both spellings, or that you simply want to work:
WEBFANG_DISABLE_SSRF_ENTRY_GUARD=1 WEBFANG_DISABLE_SSRF_RESOLVER=1 \
  webfang http://127.0.0.1:8080/ --dry-run

Rules of thumb:

  • Set them per command, never in a profile. These variables disarm a control that exists because a crawler follows links written by strangers: an allow-everything environment is how a scraping run becomes an internal-network scan. A WEBFANG_* line in your shell rc applies to every target you will ever crawl, including the ones you did not mean to fetch.
  • The resolver value is read once, at client construction. Export it before starting a long-lived process (the MCP server, a batch runner); setting it mid-run changes nothing for the clients already built.
  • MCP needs its own variable too. WEBFANG_MCP_DISABLE_SSRF=1 lifts MCP's entry pre-check (layer 1) only; the core layers above still apply. A local target driven through MCP therefore needs the MCP switch and the core pair.
  • Redirects have a fourth, separate disarmer (WEBFANG_DISABLE_SSRF_REDIRECT_GUARD=1). Leave it armed: a public site that redirects into your internal range is exactly the case the guard was written for.

The full layer map — which variable lifts which layer, and what stays protected after you set it — is in docs/ssrf-layers.md.


WAF / bot detection blocks

scripts/analyze-trace.sh debug.jsonl waf          # WAF challenges + banned domains

If you see WAF challenge detected errors:

  • The site is presenting a CAPTCHA / challenge page. WebFang bans the domain for the rest of the crawl to avoid hammering it.
  • Try a different TLS fingerprint profile (--tls-emulation): a WAF challenge short-circuits the fetch at the layer that saw it, so WebFang never re-renders the page with JavaScript to get past it.
  • Slow down (--delay, lower concurrency) to avoid rate-limit triggers.

I can't tell which logs belong to one page / one crawl

  • One crawl shares a single trace_id. Filter by it:
    scripts/analyze-trace.sh debug.jsonl trace <trace_id>
    
  • Each page is a crawl_page span with its own span_id under that trace_id.

Non-deterministic snapshot failures in tests

correlation_id / trace_id are internal and #[serde(skip)] on scraped output, so they never appear in scraped JSON/JSONL snapshots. If a new field you added is non-deterministic (timestamps, ports, temp paths, random IDs), redact it via redact_nondeterministic() in tests/common/cli_harness.rs.


Async deadlocks / starved tasks

For concurrency bugs (a crawl hangs, tasks never complete), use the Tokio Console:

RUSTFLAGS="--cfg tokio_unstable" cargo run --features console -- --url <URL>

This shows live task states and poll times, making stuck tasks visible.


Empty or poor content

  • content extraction failed (stage: extract) — the fallback extractor got less than the minimum content. The page is likely JS-rendered, an interactive app, or not an article.
  • Try --selector '.main-content' (or the right CSS selector for the site), or enable JS rendering for SPA content.

Leftover .lock files in the state directory — is my state corrupted?

Symptom. After a --resume run (or a killed one) the --state-dir contains <domain>.json.lock files next to the <domain>.json state files, even though no writer is running.

Short answer. No — nothing is corrupted, and nothing is stuck. The .lock file is a permanent sentinel that is created on demand and intentionally never removed; the real mutual exclusion is the advisory flock on that file, which the kernel releases when the process exits (including after SIGKILL).


Windows: output is mojibake

Symptom. Accented or ñ characters render as garbage in the terminal.

webfang switches the Windows console code page to UTF-8 as its first action (#1808, XP-K-03), so this normally cannot happen. If it does, the code page was most likely reset afterwards — some terminals and CI shells re-apply the profile default when they attach. Fix it in that shell, not in webfang:

chcp 65001

Windows: a deeply nested output path fails to write

Symptom. A crawl into a deeply nested --output directory fails with a path error while the same crawl succeeds elsewhere.

Known limitation (XP-P-07). Windows caps a path at MAX_PATH — 260 characters for the whole path, which is a separate limit from the 255-character-per-component rule webfang enforces and validates. webfang validates every component, but it does not rewrite paths into the extended \\?\ form: that form disables ./.. normalization and changes UNC handling, so applying it at every call site would trade a documented limit for a set of subtle ones.

Workaround. Either shorten the output root, or enable long paths for the machine (administrator PowerShell, then sign out and back in):

New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" `
  -Name LongPathsEnabled -Value 1 -PropertyType DWORD -Force

Windows: closing the console window kills the run without draining

Should not happen. A console close, logoff, or shutdown event now drives the same graceful drain as Ctrl+C (#1808, XP-S-02) — in-flight pages are persisted and a bounded checkpoint is written before exit.

One residual, stated plainly: the console host allows a close-event handler about five seconds before it kills the process. That window belongs to Windows, not to webfang, so console-close draining is best-effort. Logoff and shutdown events are generous. If you interrupt a long crawl repeatedly, prefer Ctrl+C, which has no such deadline.

State Recovery Runbook

What to do when webfang's persisted state looks wrong.

This page covers the SQLite/WAL vault database (~/.webfang/crawl.db) — the artifact that had no recovery procedure anywhere — and routes the other persisted artifacts (the crawl checkpoint and the JSON resume state) to their own contract.

Every behaviour described here is pinned by a test that already exists in the repository, and the test is named next to the claim. A step marked not covered by a test is either SQLite's own behaviour or an operator instruction: it is never a promise webfang makes.

The durability contract — what is guaranteed and what is explicitly not — lives in docs/persistence-resume.md. This page is the procedure: what to check, what to run, what to delete.

Where the files live and how to remove them: installation.md. Symptoms that are not state at all: troubleshooting.md.


Triage — which artifact is broken?

SymptomArtifactSection
A run fails with base de datos con versión de esquema …vault DBThe vault database
A run fails with vector BLOB corrupto: longitud N no es múltiplo de 4vault DBA corrupt embedding
A run with --elastic fails before it crawlsvault DBThe schema-version gate
--resume re-scrapes pages it had already doneJSON stateThe JSON state
A <domain>.json.bak appeared next to the state fileJSON stateThe JSON state
A <domain>.json.lock is still there after the run—Nothing is broken: the sentinel is permanent by design — see troubleshooting.md

The vault database

Where the database lives

~/.webfang/crawl.db by default — absolute, under $HOME, not under the webfang cache base. --db-path (or WEBFANG_DB_PATH) moves it.

BehaviourPinned by
The default path is <home>/.webfang/crawl.dbtest_default_db_path_under_home — crates/webfang_core/src/infrastructure/autotuning.rs
The CLI flag wins over the env var, which wins over the defaulttest_resolve_db_path_priority — crates/webfang_core/src/infrastructure/autotuning.rs
The file is created lazily: the pool is built first, the file appears when the schema is initialised — that is the fail-fast point if the path is not writabletest_db_file_created_when_absent — crates/webfang_core/src/infrastructure/persistence/sqlite.rs
The database only exists in builds compiled with the persistence feature. The release binary is built with --features "mcp" only, so a release install never creates itnot covered by a test — the feature gate is crates/webfang_core/Cargo.toml (persistence = ["dep:rusqlite", "dep:deadpool-sqlite", "dep:sys-info"]) and the release command is .github/workflows/release.yml; the same fact is stated for operators in installation.md

How it is opened

BehaviourPinned by
Every pooled connection runs journal_mode=WAL, synchronous=NORMAL, cache_size=-4000 (the first is persistent in the DB header, the other two are per-connection and are re-applied by a post_create hook)test_wal_pragmas_applied_to_connection — crates/webfang_core/src/infrastructure/persistence/sqlite.rs
The schema DDL is idempotent (CREATE … IF NOT EXISTS), so re-initialising an existing database is a no-optest_setup_schema_is_idempotent — same file

The schema-version gate: refuse, never migrate

The database carries its schema version in SQLite's own on-disk header slot (PRAGMA user_version). webfang writes 1 and opens only 1.

BehaviourPinned by
A fresh database is stamped with the current version, and the stamp survives a reopen; a second initialisation neither fails nor re-stampssetup_schema_stamps_the_schema_version_marker — crates/webfang_core/src/infrastructure/persistence/sqlite.rs
A database stamped with another version is refused before any DDL runs — the refused file is never opened for writing, and it keeps its own markersetup_schema_refuses_a_foreign_schema_version_without_writing — same file
A database stamped 0 (written before the marker existed, or brand new) is adopted, not refusedsetup_schema_adopts_an_unversioned_database — same file
The refusal message names both versions, in Spanish: base de datos con versión de esquema {found}, incompatible con la versión {SCHEMA_VERSION} de webfang; actualizá webfang o usá una base de datos distinta (nunca se modifica una base desconocida)setup_schema_refuses_a_foreign_schema_version_without_writing — same file (it asserts both numbers appear in the message)

There is no migration mechanism: a bump of the marker without a migration would turn every existing database into a refusal, so the gate refuses by design. The behaviour that pins this is the refusal itself (setup_schema_refuses_a_foreign_schema_version_without_writing): a foreign version is refused, never converted.

Recovery procedure

  1. Stop every webfang process. The database is only safe to touch when no run holds it open.

  2. Keep the evidence before you touch anything:

    cp ~/.webfang/crawl.db ~/.webfang/crawl.db.broken.$(date +%s)
    
  3. Read the marker (SQLite's own tool — not covered by a test):

    sqlite3 ~/.webfang/crawl.db 'PRAGMA user_version;'
    

    1 (today's SCHEMA_VERSION) is a database this build owns. 0 is an unversioned one, which webfang adopts. Anything else is a database webfang refuses — it was written by a build with a different column layout.

  4. Check the file's integrity (SQLite's own tool — not covered by a test; webfang never runs it):

    sqlite3 ~/.webfang/crawl.db 'PRAGMA integrity_check;'
    
  5. Choose the recovery:

    SituationWhat to do
    The marker is foreign (webfang refuses the file)Either update webfang so the version matches, or point --db-path / WEBFANG_DB_PATH at a different file. This is what the refusal message itself asks for. Note that --db-path only has an effect together with --elastic — on its own it is rejected with exit 78 (not covered by a test; the guard is crates/webfang_core/src/cli/preflight.rs::check_db_path_requires_elastic).
    The marker is foreign and the ingested vectors are not neededDelete crawl.db, crawl.db-wal and crawl.db-shm together, with no run active, and let the next run create a fresh database. The DDL is idempotent, so the new file is created and stamped on the spot.
    The marker is 1 and the file passes integrity_checkNothing to recover — the refusal came from somewhere else. Re-read the error text; the two webfang-raised failures are the two rows below.
    The file is corrupt and the run fails on a vector readSee A corrupt embedding.

What is lost when the database is deleted. The database holds the ingestion tables — resources, chunks, notes, note_chunks and extraction_fingerprints. Deleting it discards the ingested vectors, the stored chunks and the failure fingerprints. It does not touch the Markdown/JSONL the run wrote to the output directory: those are separate files, written by the exporters.

Not covered by a test: the sidecars -wal and -shm are SQLite's own artifacts. webfang never creates, truncates or deletes them by name, and it never runs PRAGMA wal_checkpoint. Deleting the three files together with no process attached is SQLite's own recovery practice, not a webfang guarantee.

A corrupt embedding blob

BehaviourPinned by
A stored embedding whose length is not a multiple of 4 is refused with a Spanish error — vector BLOB corrupto: longitud N no es múltiplo de 4 — instead of being decoded into garbagetest_bytes_to_f32_vec_rejects_invalid_blob_length — crates/webfang_core/src/infrastructure/persistence/sqlite.rs
The same failure surfaces through the public read path (get_vector) as an error, never as a panictest_corrupt_blob_returns_error — same file

A corrupt blob surfaces as an error on the read of that row. Whether the rest of the database keeps working is SQLite's own behaviour — not covered by a test. The recovery is the same as above: the file itself is not damaged.

Retention

BehaviourPinned by
--retention-days N prunes rows older than the cutoff from all four tables, children before parents, and runs VACUUM afterwardsvault_db_rows_older_than_cutoff_are_pruned_children_first — crates/webfang_core/src/cli/retention.rs (the test asserts the row counts; the VACUUM is in the code path, not asserted)
Rows inside the cutoff survive the prunesame test (it asserts the fresh rows are still there)

Without --retention-days nothing is ever pruned and the run prints no retention summary — default_retention_is_disabled_and_never_prunes — crates/webfang_core/tests/retention_test.rs. Rows whose created_at is NULL or unparseable are never pruned — the SQL condition is in prune_older_than (crates/webfang_core/src/infrastructure/persistence/sqlite.rs); the test seeds only dated rows, so the NULL case is not covered by a test.


The JSON state

The crawl checkpoint and the JSON resume state are covered by the durability contract in docs/persistence-resume.md. This section only lists what the recovery procedure needs.

BehaviourPinned by
The record store writes a temp file and renames it over the live file; rename(2) is the commit point — a crash before it leaves the previous complete state, never a torn filecrash_between_saves_leaves_exported_state_reloadable_and_completable and crash_after_committed_rename_sees_committed_on_reload — crates/webfang_core/src/infrastructure/export/record_store.rs
A crash at any of the pinned crash points, followed by a --resume rerun, recovers with zero lost URLs and no duplicated committed lineevery row of crates/webfang_core/tests/crash_matrix_test.rs (for example crash_mid_state_file_write_truncated_tmp_is_gced, crash_tmp_written_pre_rename_discards_tmp_and_recovers, crash_while_holding_lock_releases_and_recovers)
A stale .tmp-* sibling left by a crash is garbage-collected on the next load and is never read as datastale_tmp_files_are_garbage_collected_on_load_and_never_read — crates/webfang_core/src/infrastructure/export/record_store.rs
A discarded state file leaves a <file>.bak sibling carrying the exact pre-migration bytes; an existing backup is never overwritten, so the first bytes wintest_load_or_default_stale_version_preserves_bak — crates/webfang_core/src/infrastructure/export/state_store.rs; preserve_abandoned_bytes_writes_rejected_bytes_once — crates/webfang_core/src/application/resume.rs
The .bak extends the artifact's own extension (state.json → state.json.bak)backup_sibling_extends_the_existing_extension — crates/webfang_core/src/application/resume.rs
The crawl checkpoint is [4-byte CRC32][JSON payload]; a checksum mismatch refuses the file and leaves the exact rejected bytes in a .bakchecksum_is_first_four_bytes and crc32_mismatch_preserves_bak_before_the_rewrite — crates/webfang_core/src/application/crawler/checkpoint.rs
A checkpoint whose schema version is superseded is discarded (never resumed half-understood) and its pre-migration file is preservedstale_checkpoint_version_is_discarded_not_resumed — crates/webfang_core/tests/checkpoint_store_integration.rs
The <state>.lock sentinel is permanent: the lock is released when the process exits, the file is never unlinkeddropping_the_store_lock_releases_but_never_unlinks_the_sentinel — crates/webfang_core/src/infrastructure/export/record_store.rs

Recovery in one line: if a state file looks wrong, the .bak sibling next to it holds the bytes webfang refused. Copy it back (with no run active) if you need the old state; otherwise delete it and let the next --resume start from scratch.


What this runbook does not promise

  • Power loss. Every atomic write in the persistence layer is temp-file + rename(2) with no fsync: it is atomic against process death, not against power loss. The contract is docs/persistence-resume.md.
  • An integrity check on the database. webfang never runs PRAGMA integrity_check and never checkpoints the WAL. Both commands in the recovery procedure above are SQLite's own tools, invoked by the operator.
  • A migration for a foreign database. The gate refuses; it does not convert.
  • Automatic repair. Nothing in this repository repairs a damaged file on its own. The repairs this page lists are operator actions.

Where the code lives

ComponentPath
SQLite pool, pragmas, schema, version gatecrates/webfang_core/src/infrastructure/persistence/sqlite.rs
Fingerprint table (same pool)crates/webfang_core/src/infrastructure/persistence/fingerprint.rs
Retention prunecrates/webfang_core/src/cli/retention.rs
DB path resolutioncrates/webfang_core/src/infrastructure/autotuning.rs
JSON state store and record storecrates/webfang_core/src/infrastructure/export/{state_store,record_store}.rs
Crawl checkpointcrates/webfang_core/src/application/crawler/checkpoint.rs
.bak policy (one policy for every destructive rewrite)crates/webfang_core/src/application/resume.rs
Crash-injection harnesscrates/webfang_core/src/cli/crash_points.rs

CLI Reference

Complete flag reference for the webfang binary. The block in the "Complete flag reference" section below is auto-generated from the binary itself by scripts/gen-cli-reference.sh (built with the exact release feature set, ai mcp), so flags, defaults, and WEBFANG_* environment variables always match reality. CI drift-checks the chapter against the binary on every doc run; regenerate locally with just cli-doc-regen.

Every flag in this reference is compiled into the binary you are running. The release artifacts are built with no capability features, so feature-gated behavior (--js-strategy full, SQLite persistence, adaptive selectors, tokio-console) is absent or rejected there — the per-flag consequences are in the Release Feature Matrix.

Quick start

Scrape a single page by passing the URL positionally:

webfang https://example.com

Crawl a whole site following its sitemap.xml:

webfang https://example.com --use-sitemap --max-pages 50

Extract only the elements matching a CSS selector and export for a RAG pipeline:

webfang https://example.com -s "article.main-content" --export-format jsonl

Resume an interrupted crawl from its last checkpoint:

webfang --batch-file urls.txt --resume

Zero values

Numeric flags follow the Zero Silent Loss policy: a zero is never silently turned into another value.

  • Zero disables the feature where an "off" state exists: --delay-ms 0 disables request pacing (no token bucket is allocated), --max-depth 0 scrapes only the seed URL.
  • Zero is rejected with a usage error where zero is meaningless or destructive: --concurrency, --download-concurrency, --download-timeout, --max-pages, --rate-limit-burst, --timeout-secs.
  • Legacy exception: a non-numeric --rate-limit-burst value warns and falls back to the hardware-derived default (numeric zero and out-of-range values are still rejected).

Configuration file

Flags can also come from a TOML file, which is how you set values once instead of on every invocation. webfang looks for webfang/config.toml under the platform's user config directory:

PlatformResolved from
Linux$XDG_CONFIG_HOME, else ~/.config
macOS~/Library/Application Support
Windows%APPDATA%

An absolute XDG_CONFIG_HOME is honored on every platform, not just Linux — that is what makes hermetic test fixtures possible.

To point at a specific file instead, set WEBFANG_CONFIG to the path of the config.toml itself. It takes precedence over the table above, and it differs from XDG_CONFIG_HOME in two ways worth knowing:

  • It names the file, while XDG_CONFIG_HOME names a directory that webfang appends webfang/ to. Use it when your file does not live in that subdirectory.
  • It is honored on Windows, where no environment variable reaches the platform config lookup: that lookup calls SHGetKnownFolderPath(FOLDERID_RoamingAppData), a Windows Shell API that reads no env var at all. Until WEBFANG_CONFIG existed, a Windows install could not be redirected to a config file by env at all.

A missing file at the platform path is not an error: webfang falls back to its defaults, because a fresh install has no config yet.

That tolerance stops at an explicit WEBFANG_CONFIG. Once you name a file, webfang holds you to it — the run stops with exit code 78 (configuration error) and a message naming the path, instead of quietly running with defaults. A typo in a config path used to be indistinguishable from a successful run.

WEBFANG_CONFIGResult
unset, or emptyPlatform lookup; a missing file means defaults
absolute path to a readable, valid TOML fileUsed
relative pathRejected, exit 78
path does not existRejected, exit 78
unreadable — wrong permissions, or a directoryRejected, exit 78
not valid TOMLRejected, exit 78

Two edge cases matter because this variable is usually set from a script or a service unit rather than typed:

  • An empty value counts as unset. WEBFANG_CONFIG= webfang ... behaves exactly like not setting the variable at all, so a script that exports it before it has a value falls back to the platform lookup instead of failing.
  • A relative path is rejected rather than resolved. It used to be resolved against the current working directory of the process, not against your home directory — predictable in an interactive shell, and arbitrary under a daemon or service manager, whose working directory is whatever the supervisor chose. The same variable could therefore name a different file depending on who started the process, which is why an absolute path is now required.
WEBFANG_CONFIG=/etc/webfang/config.toml webfang https://example.com

Complete flag reference

Everything below is the verbatim output of webfang --help and webfang completions --help — including every default value and the matching WEBFANG_* environment variable for each flag.

$ webfang --help

CLI Arguments for the webfang binary.

Parsed using `clap` with derive macros.

# Examples

```no_run use webfang_core::Args; use clap::Parser;

let args = Args::parse_from([ "webfang", "--url", "https://example.com", "--output", "./output", "--export-format", "jsonl", "--resume", ]);

assert_eq!( args.crawler.url.as_ref().map(webfang_core::domain::ValidUrl::as_str), Some("https://example.com") ); ```

Usage: webfang [OPTIONS] [URL]
       webfang [OPTIONS] [URL] <COMMAND>

Commands:
  completions  Generate shell completion scripts
  connect      Detect and validate a remote embedding endpoint, then write its config
  doctor       Diagnose the embedding path — config, credential, network, budget
  show         Dump the effective embeddings provider, the configured budget, and the historical model-profile metadata
  recover      Recover pages from a stranded batch capture spool into markdown files
  help         Print this message or the help of the given subcommand(s)

Arguments:
  [URL]
          URL to scrape (positional shorthand — equivalent to --url). Hardened through the SAME argv-boundary parser as `--url` (#1239): the shorthand cannot bypass the credential strip

Options:
  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Target:
  -u, --url <URL>
          URL to scrape (required unless using a subcommand)
          
          [env: WEBFANG_URL=]

  -s, --selector <SELECTOR>
          Matched elements become the content directly (no main-content cleanup); title and date come from the whole document, excerpt stays empty
          
          [env: WEBFANG_SELECTOR=]
          [default: body]

Discovery:
      --delay-ms <DELAY_MS>
          Delay between requests in milliseconds
          
          [env: WEBFANG_DELAY_MS=]
          [default: 1000]

      --jitter-ms <JITTER_MS>
          Random addition to each delay between requests, in milliseconds (0 = fixed delay; the mean interval becomes delay + jitter/2, so the request rate drops; with 0, a run that fits in the burst is not paced at all)
          
          [env: WEBFANG_JITTER_MS=]
          [default: 0]

      --max-pages <MAX_PAGES>
          Maximum pages to scrape
          
          [env: WEBFANG_MAX_PAGES=]
          [default: 10]

      --concurrency <CONCURRENCY>
          Concurrency level (auto or number, minimum 1)
          
          [env: WEBFANG_CONCURRENCY=]
          [default: auto]

      --rate-limit-burst <RATE_LIMIT_BURST>
          Explicit rate-limiter burst permits (token-bucket capacity).
          
          Overrides the hardware-derived budget-model default (Q1: burst is decoupled from crawl concurrency). Accepts a positive integer, or the keyword `auto` for the derived default (the same spelling `--concurrency` understands). Raw string here ON PURPOSE: validation/conversion happens once in preflight staging via `parse_rate_limit_burst` so CLI, env, and programmatic input all share one semantic. That semantic FAILS CLOSED: `0`, an out-of-range value, and any other typo are rejected with exit 78 rather than degrading to the default — only `auto` and an empty value mean "not set".
          
          [env: WEBFANG_RATE_LIMIT_BURST=]

      --use-sitemap
          Use sitemap for URL discovery NOTE: HTTP redirects (301/302) are resolved at scrape-time, not parse-time. This avoids redundant HEAD requests during sitemap parsing for better performance
          
          [env: WEBFANG_USE_SITEMAP=]

      --sitemap-url <SITEMAP_URL>
          Explicit sitemap URL
          
          [env: WEBFANG_SITEMAP_URL=]

Behavior:
      --single-page
          Scrape only the seed URL without discovery or crawling (batch mode always scrapes one page per URL)
          
          [env: WEBFANG_SINGLE_PAGE=]

      --resume
          Resume mode - skip URLs already processed
          
          [env: WEBFANG_RESUME=]

      --state-dir <STATE_DIR>
          Custom state directory for resume mode
          
          [env: WEBFANG_STATE_DIR=]

      --download-images
          Download images from the page
          
          [env: WEBFANG_DOWNLOAD_IMAGES=]

      --download-documents
          Download documents from the page
          
          [env: WEBFANG_DOWNLOAD_DOCUMENTS=]

      --download-assets
          Download all assets (images + documents) from the page
          
          [env: WEBFANG_DOWNLOAD_ASSETS=]

      --extraction-fingerprint
          Record extraction failure fingerprints in SQLite and attach them to low-quality extraction hints (#792). Repeated low-score extractions on the same site/selector pair accumulate a failure count surfaced in the hint, instead of degrading silently
          
          [env: WEBFANG_EXTRACTION_FINGERPRINT=]

      --clean-ai
          Use AI-powered semantic cleaning for better RAG output
          
          [env: WEBFANG_CLEAN_AI=]
          [alias: --ai]

Display:
  -v, --verbose...
          Verbosity level: -v (INFO), -vv (DEBUG), -vvv (TRACE)
          
          [env: WEBFANG_VERBOSE=]

  -q, --quiet
          Quiet mode — suppress info/debug output
          
          [env: WEBFANG_QUIET=]

  -n, --dry-run
          Dry-run mode — discover URLs and print without scraping
          
          [env: WEBFANG_DRY_RUN=]

      --trace-file <TRACE_FILE>
          Path to write OTel spans as JSONL for offline debugging
          
          [env: WEBFANG_TRACE_FILE=]

Crawler Settings:
      --max-depth <MAX_DEPTH>
          Maximum depth to crawl (0 = only seed URL)
          
          [env: WEBFANG_MAX_DEPTH=]
          [default: 2]

      --timeout-secs <TIMEOUT_SECS>
          Request timeout in seconds
          
          [env: WEBFANG_TIMEOUT_SECS=]
          [default: 30]

      --include-pattern <INCLUDE_PATTERNS>
          URL patterns to include (glob-style). Three modes:
          
          * Path: starts with `/` → matched against URL path, e.g. `/pricing`, `/admin/*` * Path glob: starts with `*/` → matched against URL path, e.g. `*/api/*` * Host (default): matched against hostname, e.g. `example.com`, `*.example.com`
          
          Example: to exclude a path, use `--exclude-pattern "/admin/*"`, not `*admin*`
          
          [env: WEBFANG_INCLUDE=]

      --exclude-pattern <EXCLUDE_PATTERNS>
          URL patterns to exclude (glob-style, same three modes as --include-pattern). Deny takes precedence over allow
          
          [env: WEBFANG_EXCLUDE=]

Download Settings:
      --asset-naming <ASSET_NAMING>
          Estrategia de nombre de archivo para assets descargados: hash (default), slug, content-disposition
          
          [default: hash]
          [possible values: hash, slug, content-disposition]

      --download-concurrency <DOWNLOAD_CONCURRENCY>
          Máximo de descargas de assets concurrentes por página (mínimo 1)
          
          [env: WEBFANG_DOWNLOAD_CONCURRENCY=]

      --max-file-size <MAX_FILE_SIZE>
          Maximum file size to download in bytes (default: 50MB)
          
          [env: WEBFANG_MAX_FILE_SIZE=]
          [default: 52428800]

      --download-timeout <DOWNLOAD_TIMEOUT>
          Timeout for individual asset downloads in seconds (minimum 1)
          
          [env: WEBFANG_DOWNLOAD_TIMEOUT=]
          [default: 30]

HTTP Client Settings:
      --max-retries <MAX_RETRIES>
          Maximum number of retry attempts
          
          [env: WEBFANG_MAX_RETRIES=]
          [default: 3]

      --backoff-base-ms <BACKOFF_BASE_MS>
          Base delay for exponential backoff (ms)
          
          [env: WEBFANG_BACKOFF_BASE_MS=]
          [default: 1000]

      --backoff-max-ms <BACKOFF_MAX_MS>
          Maximum delay for exponential backoff (ms)
          
          [env: WEBFANG_BACKOFF_MAX_MS=]
          [default: 10000]

      --accept-language <ACCEPT_LANGUAGE>
          Accept-Language header value
          
          [env: WEBFANG_ACCEPT_LANGUAGE=]
          [default: en-US,en;q=0.9]

      --user-agent <USER_AGENT>
          Custom User-Agent header value (overrides Chrome 145 default)
          
          [env: WEBFANG_USER_AGENT=]

  -H, --header <NAME: VALUE>
          Inject a custom HTTP header as `Name: Value` (repeatable).
          
          Overrides any default header with the same (case-insensitive) name. Example: `-H "Authorization: Bearer TOKEN"`.
          
          [env: WEBFANG_HEADER=]

      --cookie <NAME=VALUE>
          Inject a custom cookie as `name=value` (repeatable).
          
          Seeded into the cookie jar before the first request so authenticated crawls work without a prior login round-trip. Example: `--cookie "session=abc123"`.
          
          [env: WEBFANG_COOKIE=]

Sitemap Settings:
      --sitemap-depth <SITEMAP_DEPTH>
          Maximum recursion depth for sitemap indexes
          
          [env: WEBFANG_SITEMAP_DEPTH=]
          [default: 3]

Competitive Features:
      --checkpoint-interval <CHECKPOINT_INTERVAL>
          Pages between automatic checkpoint saves (0 = disabled) — unified via PersistenceMode with --resume (Checkpoint/Full when enabled)
          
          [env: WEBFANG_CHECKPOINT_INTERVAL=]
          [default: 100]

      --no-checkpoint
          Disable checkpoint persistence entirely — PersistenceMode disables checkpoint (Resume only when combined with --resume)
          
          [env: WEBFANG_NO_CHECKPOINT=]

      --ignore-robots
          Skip robots.txt enforcement
          
          [env: WEBFANG_IGNORE_ROBOTS=]

      --ignore-waf
          Skip WAF/CAPTCHA classification (REQ-WAF-07): challenged responses are reported as plain HTTP errors instead of WAF blocks. It does not rescue the fetch — a challenge page is never scraped as content (F-11).
          
          [env: WEBFANG_IGNORE_WAF=]

      --autoscale
          Enable autoscaled concurrency — dynamically adjusts task concurrency based on RAM usage
          
          [env: WEBFANG_AUTOSCALE=]

      --no-session-health
          Disable session pool health checks
          
          [env: WEBFANG_NO_SESSION_HEALTH=]

      --h2-profile <H2_PROFILE>
          TLS/HTTP2 profile name (default: Chrome145)
          
          [env: WEBFANG_H2_PROFILE=]
          [default: Chrome145]

JS Rendering:
      --js-strategy <JS_STRATEGY>
          JavaScript rendering strategy: static (wreq only), hybrid (3-layer), full (Chromiumoxide only)

          Possible values:
          - static: Static HTTP only (wreq). Fastest, no JS rendering
          - hybrid: Hybrid 3-layer: wreq → Obscura → Chromiumoxide
          - full:   Full JS rendering only (Chromiumoxide). Slowest, handles all SPAs
          
          [env: WEBFANG_JS_STRATEGY=]
          [default: static]

      --js-wait <JS_WAIT>
          Post-load settlement wait for the chromium render path: idle (network-idle, default), <ms> fixed wait (1..=30000), none (capture immediately)
          
          [env: WEBFANG_JS_WAIT=]
          [default: idle]

      --obscura-binary <OBSCURA_BINARY>
          Path to the obscura binary (default: "obscura")
          
          [env: WEBFANG_OBSCURA_BINARY=]
          [default: obscura]

Cleanup:
      --dom-preprune [<DOM_PREPRUNE>]
          Enable DOM pre-pruning before Readability (removes invisible/empty wrappers). Default: enabled (true). Set to false via --dom-preprune=false or WEBFANG_DOM_PREPRUNE=false
          
          [env: WEBFANG_DOM_PREPRUNE=]
          [default: true]
          [possible values: true, false]

Output:
  -o, --output <OUTPUT>
          Output directory for scraped content
          
          [env: WEBFANG_OUTPUT=]
          [default: output]

  -f, --content-format <FORMAT>
          Output format for individual files (markdown, text, json) NOTE: For RAG pipeline export, use --pipeline-format instead

          Possible values:
          - markdown: Markdown format with YAML frontmatter (recommended for RAG)
          - json:     Structured JSON with metadata
          - text:     Plain text without formatting
          
          [env: WEBFANG_FORMAT=]
          [default: markdown]

      --pipeline-format <EXPORT_FORMAT>
          Export format for RAG pipeline (jsonl, vector, auto) NOTE: Use --content-format for output file format (markdown, text, json)

          Possible values:
          - jsonl:  JSONL format (JSON Lines - one JSON object per line) Optimal for RAG pipelines and vector database ingestion
          - vector: Vector format (JSON with metadata header) Supports embeddings and cosine similarity
          - auto:   Auto-detect format from existing export files
          
          [env: WEBFANG_EXPORT_FORMAT=]
          [default: jsonl]

      --retention-days <RETENTION_DAYS>
          Prune exports and vault-DB rows older than N days at the end of a successful run (0 = disabled)
          
          [env: WEBFANG_RETENTION_DAYS=]
          [default: 0]

Elastic Ingestion:
      --cpu-cores <CPU_CORES>
          CPU core override for the elastic ingestion Rayon pool (else auto-detect)
          
          [env: WEBFANG_CPU_CORES=]

      --ram-budget <RAM_BUDGET>
          RAM budget override for the byte-weighted semaphore (`8GB`, `2048MB`, or bytes)
          
          [env: WEBFANG_RAM_BUDGET=]

      --db-path <DB_PATH>
          SQLite database path override for persisted resources/chunks
          
          [env: WEBFANG_DB_PATH=]

      --elastic
          Enable elastic ingestion pipeline (streaming, SQLite dedup, Rayon CPU bridge)
          
          [env: WEBFANG_ELASTIC=]

      --output-vectors <OUTPUT_VECTORS>
          Write extracted vectors to a JSONL file for RAG pipelines. Use `-` for stdout. No SQLite dependency — available in every build (core binary too)
          
          [env: WEBFANG_OUTPUT_VECTORS=]

Batch Processing:
      --batch
          Enable batch mode — scrape each URL from stdin (one per line), one page per URL, no crawling
          
          [env: WEBFANG_BATCH=]

      --batch-file <BATCH_FILE>
          Path to a file containing URLs to scrape (one per line), one page per URL, no crawling
          
          [env: WEBFANG_BATCH_FILE=]

      --batch-concurrency <BATCH_CONCURRENCY>
          Maximum concurrent URLs in batch mode (omit = auto from budget model)
          
          [env: WEBFANG_BATCH_CONCURRENCY=]

      --export-checkpoint <EXPORT_CHECKPOINT>
          Flush captured pages to markdown every N pages during a batch (0 = only at the clean close)
          
          [env: WEBFANG_EXPORT_CHECKPOINT=]
          [default: 0]

Item Pipeline:
      --pipeline
          Enable item pipeline processing (validate → clean → output)
          
          [env: WEBFANG_PIPELINE=]

      --pipeline-output <PIPELINE_OUTPUT>
          Pipeline output format: jsonl (default), none

          Possible values:
          - jsonl: Write items as JSON Lines to a file (default)
          - none:  No pipeline output — items are processed but not written
          
          [env: WEBFANG_PIPELINE_OUTPUT=]
          [default: jsonl]

Obsidian:
      --obsidian-wiki-links
          Convert same-domain links to Obsidian [[wiki-link]] syntax
          
          [env: WEBFANG_OBSIDIAN_WIKI_LINKS=]

      --obsidian-tags <OBSIDIAN_TAGS>
          Tags to include in YAML frontmatter (comma-separated)
          
          [env: WEBFANG_OBSIDIAN_TAGS=]

      --obsidian-relative-assets
          Rewrite downloaded asset paths as relative to the .md file
          
          [env: WEBFANG_OBSIDIAN_RELATIVE_ASSETS=]

      --vault <VAULT>
          Path to Obsidian vault (auto-detects if not provided).
          
          When provided explicitly, the vault becomes the output base: Markdown,
          downloaded assets and the RAG export are written inside it — no need
          to duplicate the path in `-o` (which then must stay at its default).
          Auto-detected or config-file vaults do NOT redirect output (#762).
          
          [env: WEBFANG_OBSIDIAN_VAULT=]

      --quick-save
          Quick-save mode: save directly to vault _inbox folder
          
          [env: WEBFANG_OBSIDIAN_QUICK_SAVE=]

      --obsidian-rich-metadata
          Add rich metadata to frontmatter
          
          [env: WEBFANG_OBSIDIAN_RICH_METADATA=]

AI Settings:
      --threshold <THRESHOLD>
          Relevance threshold for AI semantic filtering (0.0-1.0)
          
          [env: WEBFANG_THRESHOLD=]
          [default: 0.3]

      --max-chars <MAX_CHARS>
          Maximum characters per chunk before rejection, >= 1 (a chunk-size guard, not a context-window setting; the effective ceiling is the embedding backend's own limit, which a remote endpoint enforces server-side as HTTP 400/413)
          
          [env: WEBFANG_MAX_CHARS=]
          [default: 98304]

      --offline
          Reject every embeddings provider (remote or loopback) with exit 78: no local model since 3.0.0
          
          [env: WEBFANG_OFFLINE=]

LLM Extraction:
      --extract-with-llm
          Run structured extraction through the configured LLM provider (requires a provider with the `completion` capability; fails at startup otherwise)
          
          [env: WEBFANG_EXTRACT_WITH_LLM=]

      --llm-provider <LLM_PROVIDER>
          Provider id to use for LLM extraction (default: first provider declaring the `completion` capability)
          
          [env: WEBFANG_LLM_PROVIDER=]

      --embedding-provider <EMBEDDING_PROVIDER>
          Provider id to use for embeddings (default: first provider declaring the `embedding` capability)
          
          [env: WEBFANG_EMBEDDING_PROVIDER=]

Tier 2 Embeddings:
      --tier2-embedding-provider <TIER2_EMBEDDING_PROVIDER>
          Provider id to use for Tier 2 semantic inspector embeddings (default: none; the vault-search provider is never adopted — a remote id requires --tier2-consent)
          
          [env: WEBFANG_TIER2_EMBEDDING_PROVIDER=]

      --tier2-consent
          Explicit consent for Tier 2 remote embeddings egress (required when --tier2-embedding-provider is remote; hash of provider|endpoint logged for audit)
          
          [env: WEBFANG_TIER2_CONSENT=]

EXIT CODES:
  0    Success
  2    No URLs discovered
  3    All scrapers failed
  64   Bad CLI arguments (usage error)
  65   Data format error (extraction without usable content)
  69   Network error, partial success, or SSRF refusal
  74   I/O error
  76   Protocol error
  77   All URLs blocked by robots.txt
  78   Configuration error

EXAMPLES:
  webfang https://example.com
  webfang -u https://example.com
  webfang -u https://example.com --ai
  webfang -u https://example.com -f jsonl
  webfang -u https://example.com -v
  webfang -u https://example.com -vv  # DEBUG
  webfang --batch-file urls.txt --resume
$ webfang completions --help

Generate shell completion scripts

Usage: webfang completions <SHELL>

Arguments:
  <SHELL>
          Shell to generate completions for

          Possible values:
          - bash:        Bash shell completions
          - elvish:      Elvish shell completions
          - fish:        Fish shell completions
          - power-shell: PowerShell completions
          - zsh:         Zsh shell completions

Options:
  -h, --help
          Print help (see a summary with '-h')

Layer 2 (Obscura): a cheap render, not an anti-WAF guarantee

--js-strategy hybrid fetches in three layers: L1 is wreq (static HTTP), L2 is the obscura subprocess, L3 is Chromiumoxide. (--js-strategy full goes straight to L3 and is rejected at preflight by the release binaries, which carry no chromium feature — Release Feature Matrix.)

L2 is accepted as a cheap render — webfang does not require obscura's -stealth build variant, so L2 carries no anti-WAF promise. That is the decision recorded in #1710; this section is where it lives.

What L2 does

  • Runs obscura fetch --dump html once per page — never markdown — with no cookies and no storage directory (the sessionless contract), and marks the result as text/html.
  • Costs ~30 MB per subprocess instead of the ~200 MB + CDP that Layer 3 pays: that is the whole reason it stays in the stack.
  • Escalates to Layer 3 when the page comes back empty or the subprocess fails.

What L2 does not do

  • It is not an anti-WAF guarantee. webfang never passes -stealth, and a WAF challenge detected at any layer aborts the fetch: the router never escalates against a challenge.

Version floor. The hybrid preflight requires obscura 0.2.5 or newer (MINIMUM_OBSCURA_VERSION in crates/webfang_core/src/cli/preflight.rs). An older binary fails the preflight with exit 78 (configuration error), naming the version it found and the version it requires. A binary whose --version probe fails or prints something unparseable is not blocked: the preflight degrades to a warning and the run continues best-effort.

Reading the L2 hit-rate. There are no counters — the hit-rate is derived from the per-layer log events the hybrid router already emits. With a trace file, count the layer events:

webfang https://example.com --js-strategy hybrid --trace-file run.jsonl

# L2 attempts
jq -r 'select(.message | test("Layer 2 \\(Obscura\\): attempting fetch")) | .message' run.jsonl | wc -l
# L2 hits (the render came back with content)
jq -r 'select(.message | test("^Layer 2 returned [0-9]+ bytes$")) | .message' run.jsonl | wc -l
# escalations to L3
jq -r 'select(.message | test("Layer 3 \\(Chromiumoxide\\): attempting fetch")) | .message' run.jsonl | wc -l

The trace file records every level independently of -v (the file layer runs at TRACE level for the webfang target), so the layer events are in it even without -vv. The rest of the query cookbook is in debugging.md.

The four consequences of the decision, recorded so the promise is not re-invented later:

  1. The L2 hit-rate is derived from the per-layer logs — no new counters, and no strict mode.
  2. The preflight's Unknown verdict (an unreadable --version) degrades to a warning instead of failing the run: an unreadable version never blocks the crawl.
  3. The floor in force is 0.2.5. The 0.2.3 ask in #1710 is stale: the upstream release moved on, and the floor follows the current stable.
  4. #1710 stays open. It is commented with this status, not closed.

State directory contents

Companion note to --state-dir (documented in the generated block above). Each --resume run keeps one JSON state file per domain plus a <...>.json.lock sentinel file next to it. The sentinel is created on demand and intentionally never removed: its presence does NOT mean a writer is alive. The real mutual exclusion is the advisory flock held on the sentinel, which the kernel releases when the process exits — including after SIGKILL. There is exactly one sentinel per state file, so the leftover is bounded. Do not delete the .lock file while a run holds it: removing the path breaks the lock and lets two writers enter the critical section at once, losing state updates.

Migración desde ONNX local

ONNX local ya no existe en el producto (ADR-0004, paso 6). Esta página dice qué cambió, a quién le afectó y qué hacer ahora, en el orden de los pasos. La decisión y su secuencia están en ADR 0004.

Qué cambió y cuándo

MomentoQué pasó
Aviso (2 minors antes del corte)Se anunció. No se borró nada: los shims seguían funcionando y --clean-ai seguía usando ONNX local
Ventana de migraciónEl --embedding-provider resolvió el slot de vault search mientras el ONNX seguía vivo
Corte (paso 6 del ADR-0004)ONNX local, el crate webfang_ai y la feature ai se eliminaron. Este es el estado actual
Retiro de los shimsLos shims de migración se eliminaron: ya no parsean. Un flag retirado responde con el error estándar de clap (unexpected argument, exit 64)

cargo build --features ai falla ahora en la resolución de features: la feature desapareció, y no deja un shim detrás. Un shim detrás de la feature que se borra no existe, en ningún lenguaje.

A quién le afecta

Tu setupQué te pasa ahora
--clean-ai con el modelo local (Granite 97M / 311M)Te afecta. El cleaner resuelve su endpoint por --embedding-provider. Perdiste el offline total
--clean-ai + --embedding-provider ya configuradoSirve hoy: el flag mueve el slot de vault search y alimenta el cleaner vía el EmbeddingPort des-gateado
--ai-model, AI_MODEL_ID, WEBFANG_AI_MODEL_ID, WEBFANG_AI_ENGINERetirados. Ya no son flags: --ai-model responde con el error estándar de clap (unexpected argument, exit 64) y las env se ignoran en silencio. Usá --embedding-provider / WEBFANG_EMBEDDING_PROVIDER
--max-tokens, WEBFANG_MAX_TOKENSRetirados. --max-tokens ya no parsea (error estándar de clap, exit 64) y WEBFANG_MAX_TOKENS se ignora. Usá --max-chars / WEBFANG_MAX_CHARS
Vaults con vectores de GraniteHay que re-indexar: el endpoint nuevo no produce vectores comparables. El coste (y el BYO-key) lo pagás vos
No usás la superficie AINada. Los avisos no aparecen en corridas que no la tocan

Cómo migrar

  1. Configurá el endpoint remoto — detecta y valida, y escribe la config:

    webfang connect
    

    Comprobá que quedó bien, con el detalle de cada valor:

    webfang doctor
    webfang show
    
  2. Pasá --clean-ai al endpoint remoto — disponible desde el corte.

    El cleaner resuelve su EmbeddingPort por --embedding-provider (o su env WEBFANG_EMBEDDING_PROVIDER; el flag gana al env):

    webfang --url https://example.com --clean-ai --embedding-provider ollama
    

    En --offline no hay provider de embeddings que sirva: el guard rechaza cualquier endpoint —uno remoto público o uno en 127.0.0.1— y sale con exit 78. El "loopback permitido" que decía el aviso del paso 5 se refería al modelo local ONNX, que este paso eliminó; el aviso ya no lo dice y ahora nombra la regla real (rechaza cualquier provider de embeddings, remoto o loopback). Sin ese camino, --offline solo sirve para corridas sin limpieza semántica. Un endpoint LAN sigue necesitando la allowlist por CIDR (allow_cidrs), que no incluye 100.64.0.0/10 salvo opt-in explícito.

  3. Re-indexá los vaults que tengan vectores de Granite. No hay forma de detectar la mezcla: el fail-closed por dimensión no puede dispararse sobre los vaults que nunca tuvieron model_id, que son justamente los afectados. Vas a ver un aviso de "legado"; re-indexar es lo que lo cierra.

  4. Renombrá --max-tokens → --max-chars. El shim se retiró: --max-tokens ya no parsea y WEBFANG_MAX_TOKENS se ignora en silencio.

Qué sigue funcionando

--clean-ai y --offline empezaron como flags activos siempre en el corte: se des-gatearon, quedaron en el build por defecto, sin feature. --clean-ai cambió de semántica (ONNX → endpoint remoto); no fue un shim, es el mismo flag haciendo el trabajo con otro backend.

El umbral 0.3

El 0.3 por defecto estuvo calibrado para Granite. Retirado el único modelo calibrado, un endpoint remoto retiene distinto con el mismo número, así que la corrida avisa una vez mientras el umbral sea el default. El aviso no es decoración: es la señal de que el número ya no significa lo que decía.

Elegí un umbral según tu modelo:

webfang --url https://example.com --clean-ai --embedding-provider ollama --threshold 0.45

El valor efectivo de cada corrida aparece como threshold_used en el summary.

Si recién necesitás ONNX local

El borrado quedó reversible desde una etiqueta git de recuperación creada antes del borrado: tag onnx-tree/pre-step5 (commit 38ab6a31). Recuperar ese árbol es el único camino oficial hacia el ONNX de origen; desde ahí, un resurgimiento requiere un ADR nuevo — no es un rollback ni queda plementado en la línea actual.

Referencias: CLI reference · ADR 0004

MCP error contract — which channel carries which failure

Status: published decision table for issue #1613. Describes the server as it stands on top of commit ccdef614 plus the uncommitted working-tree changes that converted the last 16 reason-less rejection sites. Commit ccdef614 is the state after code slices 2–5 landed (EC-02 … EC-08) and closed EC-08's machine-readable half everywhere it was originally scoped; the working tree closes it everywhere else. This page states what the server does, not what it should do; a future change that invalidates a row must change this page in the same commit.

Evidence class: mixed, and the split is deliberate. Rows A1, A2, A3, A6, A7, A8, A9 and C1 are observed over a real transport — they are asserted on the wire in tests/mcp_error_channel_mapping_test.rs (A1, A2, A3), tests/mcp_ssrf_error_class_test.rs (A6–A9) and tests/mcp_validation_reason_code_test.rs (C1 and the data shape). Every other row is still statically derived: read out of the source, not exercised end-to-end. Rows A4, A5, A10–A19, B1–B8, C2–C4 and T1–T9 are in that second group. (A5 is the only SSRF row not covered end-to-end: no_host is unreachable from outside, which is exactly what its row says.)

Slug evidence is narrower than row evidence, and the two must not be conflated. The wire-level slug assertions cover exactly three validation slugs (unsupported_scheme, too_long, out_of_range in tests/mcp_validation_reason_code_test.rs) plus three SSRF slugs (forbidden_ip_literal, forbidden_ip_resolved, dns_resolution_failed in tests/mcp_ssrf_error_class_test.rs). The A4.x and A10–A14 slugs are not transport-observed: A4.x is pinned by the in-crate unit test path_gate::every_rejection_carries_field_and_reason_slug, which samples every branch of confine, and the A10–A14 slugs by construction — the reason-less constructor those sites used to call has been deleted, and no bare-string data producer remains anywhere in the crate (see the appendix). Nothing here is inferred from a hypothetical transport any more; see Unsettled rows for the one question that changed evidence class in the other direction and the one deliberate non-action.

Quick path for an agent consumer

HTTP status != 200                      → Channel 0. Transport refused the request. Fix the request shape.
JSON-RPC "error" member present         → Channel A. The call never ran. Fix the arguments.
result.isError == true                  → Channel B. The call ran and failed. The text is the reason.
result without isError                  → Channel C. The call ran. Read the body: it may be a
                                          diagnostic, a partial result, or a success.

Two things that table does not say, and that bite:

  • A malformed argument does not always reach Channel A. A deserialization rejection (rows A1, A2) arrives as isError: true — Channel B. Only a rejection raised by the handler body (row A3) is a JSON-RPC -32602. The type is the same on both routes; the wire shape is what differs. See A1–A3.
  • Channels A and B overlap textually. A isError: true body can contain the literal string -32602. Never discriminate on the code appearing in the text — see Channel overlap.

"Retriable" in this table means: repeating the identical call can succeed without the caller changing anything. Anything else is a caller change, not a retry.

The three channels (plus the one before them)

ChannelWire shapeHTTPCarriesCode
0 — transportbare status, usually no JSON-RPC body (T9 is the exception)401/406/408/413/415/422/429/500a request that never reached a toolnone (T9 adds a -32603 body)
A — protocol error{"error": {"code": …, "message": …, "data": …}}200the call was unroutable, or its arguments were refused by the handler body-32601 / -32602 / -32603
B — tool errorresult.isError = true200the tool ran and the job failed — or its arguments never deserializednone
C — success-shaped diagnosticresult, no isError200the tool ran; the body describes a non-success outcomenone

Channel A is the only channel with a machine-readable code and a machine-readable data.reason slug (see the reason taxonomy). Channel B carries no code: the only machine-readable signal is the boolean, and the reason is free text. Channel C has the same gap, which is the whole reason it is listed separately — see row C1.

Channel overlap — the text is not the discriminator

A Channel B body can literally contain -32602. Two independent facts combine:

  1. McpError::invalid_params(msg, data) builds an rmcp::ErrorData, and ErrorData's Display renders "{code}: {message}({data})" — the numeric code is part of the string form (rmcp-1.8.0/src/error.rs:8-16).
  2. On the deserialization route, rmcp wraps any serde error as "failed to deserialize parameters: {error}", so that rendered ErrorData — code and data included — is what lands in the isError: true body.

An A1/A2 rejection therefore reads, in the tool text, roughly failed to deserialize parameters: -32602: …({"field":"url","reason":"malformed"}) while carrying no error member at all. The only reliable discriminator is the wire shape: is there a top-level error member, or is result.isError true? Branch on that, never on a code appearing in the text.

This is also why the reason slug is the machine-readable half on this route: it is readable in the text (rmcp's Display appends the data JSON verbatim) even though it is not delivered as structured data. Parsing it out of prose is a last resort, not the contract — rows A1/A2 give no data a client can index.


Channel A — JSON-RPC protocol errors

Emitted when a tool function returns Err(McpError); McpHandler::call_tool (mcp_server/mod.rs:359-382) forwards that Err unchanged to rmcp.

Two things about this channel are not obvious from the table, and both were settled by issue #1613 rather than assumed:

  • It is not the channel every argument rejection takes. Only a rejection the handler body raises is Channel A — settled by observation in slice 3.
  • error.data is not a single shape. There are two live shapes, listed in the reason taxonomy: an object carrying both field and reason, and an object carrying reason alone. Every argument refusal carries a slug — the pre-EC-08 bare-string shape and the reason-less shape are both gone. The only Channel A rows still left with no data at all are the -32603 server faults (A15, A16), which never had a caller mistake to describe.

A1–A3 · argument deserialization (before the handler body)

#ConditionCodeRetriableAgent shouldSource
A1URL argument is not http(s), is oversize, or carries embedded credentialsnone — result.isError = true, no error member (observed)NoReshape the URL. Never retry unchanged.mcp_server/params.rs:81-99 (McpUrl::try_from); observed in tests/mcp_error_channel_mapping_test.rs:99-127
A2Unknown JSON key in the argument object (deny_unknown_fields)none — result.isError = true, no error member (observed)NoDrop the unknown key.mcp_server/params.rs:7 (per-struct attribute, first at :316); observed in tests/mcp_error_channel_mapping_test.rs:139-166
A3In-handler params.validate()? semantic rejection-32602 (observed)NoFix the named field; data.field names it.every validate() in mcp_server/params.rs (e.g. 461-475); observed in tests/mcp_error_channel_mapping_test.rs:196-217

A1 and A3 produce the same Rust error type (McpError::invalid_params is rmcp::ErrorData) but travel different routes and land on different channels: A1 is raised inside rmcp's Parameters<P> deserialization and arrives as result.isError = true; A3 is returned from the tool function and arrives as error.code = -32602 with no result member. That was the open question under Unsettled rows; it is now settled by test, and the answer is that they do not reach the client as the same code.

The mechanism is worth knowing, because it is why McpUrl::try_from does not have to fake anything. rmcp 1.8.0's Parameters<P> extractor wraps any serde error with its own "failed to deserialize parameters: " prefix (rmcp-1.8.0/src/handler/server/tool.rs:181-196), and into_tool_argument_error downgrades to isError: true only when the code is INVALID_PARAMS and that prefix is present (rmcp-1.8.0/src/handler/server/router/tool.rs:144-156). A handler-raised rejection never carries the prefix, so it is never downgraded. One shared Rust type, two routes, two channels.

A3 covers every branch below. They share one constructor (validation::invalid_params_with_reason), so they share one channel, one code, and one data shape — the split is by condition, not by envelope.

#A3 sub-conditionFielddata.reasonRetriableAgent shouldSource
A3.1URL empty, > 8192 B, unparseable, or non-http(s) schemeurl / variesempty / too_long / malformed / unsupported_schemeNoResend a valid absolute http(s) URL.mcp_server/validation.rs:137-174
A3.2Path empty, > 1024 B, absolute, drive-lettered, ..-bearing, or a rejected path componentvariesempty / too_long / path_not_allowedNoSend a relative, traversal-free path.mcp_server/validation.rs:176-237, 333-374
A3.3HTML / markdown / content blob > 1 MiBvariestoo_longNoChunk the input.mcp_server/validation.rs:41, 376-393
A3.4max_pages outside 1..=100 000, max_depth outside 0..=10max_pages / max_depthout_of_rangeNoClamp to the advertised cap.mcp_server/params.rs:152-183
A3.5urls empty or over cap, concurrency out of rangeurls / concurrencyempty / out_of_rangeNoFix the batch.mcp_server/params.rs:461-485, mcp_server/validation.rs:555-618
A3.6js_strategy not a known strategyjs_strategynot_in_allowed_setNoUse static / hybrid / full.mcp_server/params.rs:526-532
A3.7checkpoint_dir present but blankcheckpoint_diremptyNoOmit it or name a real directory.mcp_server/params.rs:533-540
A3.8Enum-ish string not in the allowed setvariesnot_in_allowed_setNoUse a listed value; the message enumerates them.mcp_server/validation.rs:751-762
A3.9Filename not a single safe component (traversal, separator, reserved name, Windows-invalid char)filenameempty / too_long / path_not_allowedNoSend one flat component.mcp_server/validation.rs:620-733, 287-331

Since slice 5, error.data on every row above is the structured object {"field": "<field>", "reason": "<slug>"}. Before that it was a bare JSON string holding only the field tag, so a client could tell which field was wrong but never why, and had to parse prose. The field survived the restructure — it is now data.field — and data.reason is the new machine-readable half. Both are contract: see the reason taxonomy.

A4 · path confinement (root-of-trust)

#ConditionCodedata.reasonRetriableAgent shouldSource
A4.1Empty path-32602emptyNoName a directory.mcp_server/path_gate.rs:281-286 via mcp_server/state.rs:332, 352
A4.2Path over MAX_PATH_LEN (1024 B)-32602too_longNoShorten the path.mcp_server/path_gate.rs:288-293
A4.3Rooted-but-not-absolute form (C:foo, \foo)-32602path_not_allowedNoUse a genuinely absolute or genuinely relative path.mcp_server/path_gate.rs:296-303
A4.4.. traversal component-32602path_not_allowedNoSend a traversal-free path.mcp_server/path_gate.rs:308-313
A4.5Absolute path with no export roots configured-32602path_not_allowedNoUse a relative path, or tell the operator to set --export-roots.mcp_server/path_gate.rs:321-334 via mcp_server/state.rs:332, 352
A4.6Absolute path outside every configured export root-32602path_not_allowedNoRe-target under a configured root.mcp_server/path_gate.rs:340-348

The gate is fail-closed: with zero roots configured, every absolute path is refused (mcp_server/path_gate.rs:321). Relative paths are allowed and resolve against the server's CWD (mcp_server/path_gate.rs:315-319).

All six branches now emit the full {"field": "output_dir" | "checkpoint_dir", "reason": <slug>} envelope — there is no reason-less shape left. Only A4.1 and A4.2 differ from the shared path_not_allowed: their remedy is fill it in and shorten it, not use a different path, and a slug that sends the agent down the wrong remedy is worse than no slug at all. The four path_not_allowed branches are deliberately not split further — "use a safe path inside an allowed root" is one instruction for all of them, and the message carries which rule fired. Only the data payload moved: the gate's control flow, messages, -32602 code and allow/deny decisions are byte-for-byte unchanged.

A5–A9 · SSRF entry pre-check

All five branches come from one function, validate_url_no_ssrf (mcp_server/ssrf.rs:96-187). Since slice 2 they are separated into three classes, and the split is expressed twice: in the JSON-RPC code, and in a data.reason slug.

#ConditionClassCodedata.reasonRetriableAgent shouldSource
A5URL has no hostcaller input-32602no_hostNoUnreachable in practice — McpUrl always carries a host.mcp_server/ssrf.rs:118-123
A6Literal IP is in a forbidden range (loopback / private / link-local / CGNAT / ULA)policy-32602forbidden_ip_literalNoStop. Retargeting is the only fix.mcp_server/ssrf.rs:130-142
A7DNS resolver returned an errorinfrastructure-32603dns_resolution_failedYesRetry — the argument is fine, the server's resolver is not.mcp_server/ssrf.rs:148-159
A8DNS returned an empty answer setinfrastructure-32603dns_no_addressesYesRetry — same class as A7.mcp_server/ssrf.rs:161-170
A9A resolved address is in a forbidden rangepolicy-32602forbidden_ip_resolvedNoStop. This is a policy refusal, not a transient.mcp_server/ssrf.rs:172-185

A7 and A8 changed class in slice 2. They used to be -32602 with data: None, identical to a caller aiming at internal infrastructure — the only way to tell a transient DNS outage from an SSRF attempt was to parse Spanish message text. They are now -32603 internal_error: the argument is correct and the server's resolver is not, which is the same fault class this crate already reports that way (rows A15, A16). Retriability follows the code: the two DNS rows are the only retriable rows in Channel A.

Policy and caller input still share -32602 on purpose — the knob-matrix suite pins the loopback refusal at exactly that code, and the SSRF knob documentation advertises the shape, so changing it would break pinned probes for no gain. That is why data.reason is load-bearing here and not decoration: the slug, not the code, is what separates the classes. The five SSRF slugs are a separate, closed set from the seven validation slugs, and they carry no field (an SSRF refusal is not a bad field); see the reason taxonomy.

This pre-check is not the enforcement point. Every scrape client is independently guarded at connect time by webfang_core::domain::ssrf_guard (literal-IP redirect policy + validating resolver), so A6/A9 are fast-fail UX and defense in depth. See docs/ssrf-layers.md for the full layer matrix.

A10–A17 · handler-local protocol errors

#ConditionCodedata.reasonRetriableAgent shouldSource
A10content_format / pipeline_format not a known export format-32602not_in_allowed_setNoUse jsonl / vector / auto.mcp_server/handlers/export.rs:196-202, 409-415
A11filename rejected by the sanitizer, or unparseable inside the synthetic export URL-32602path_not_allowedNoSend one flat component.mcp_server/handlers/export.rs:209-215, 234-240, 324-331, 362-369, 469-475
A12Synthetic export URL for the filename fails the hardened URL gate-32602malformedNoReshape the filename.mcp_server/handlers/export.rs:244-257
A13js_strategy failed to parse in the handler (post-validate path)-32602not_in_allowed_setNoUse a known strategy.mcp_server/handlers/scraping.rs:457-463
A14Explicit sitemap_url fails the hardened URL re-wrap-32602unsupported_schemeNoReshape the sitemap URL.mcp_server/handlers/scraping.rs:599-610
A15Per-category semaphore closed or poisoned-32603none — data is nullNoReport it; the server is unhealthy.mcp_server/macros.rs:16
A16Record conversion failed: DocumentChunk::validate() or metadata serialization-32603none — data is nullNoReport it; the payload was not produced.mcp_server/handlers/scraping.rs:106, 109, 212, 215, 1121, 1127
A17SSRF pre-check failure on the axtree tool (re-raised unchanged)as A5–A9as A5–A9as A5–A9As A5–A9 — read data.reason, the code is the SSRF channel's.mcp_server/handlers/axtree.rs:66-74

A15, A16 and the two SSRF DNS branches (A7, A8) are the only -32603 values this crate constructs for a tool call. They are server faults, not caller mistakes, and must not be retried — which is also why they are the only Channel A rows with no data at all: there is nothing for the caller to branch on. A17 is a forwarding site, not a construction: it re-raises whatever validate_url_no_ssrf produced, so its code and slug are inherited.

Two slug choices in this table are not mechanical, and both are deliberate. A12 reports malformed, not unsupported_scheme: the scheme there is server-built (https://webfang.local/<filename>), so unsupported_scheme would describe a URL the caller never sent and hand it unusable advice; try_from_url also collapses "unparseable" and "not http(s)" into one error, exactly as the McpUrl boundary does. A14 is the mirror image and reports unsupported_scheme: there the value was already parsed by McpUrl, so the scheme allow-list is the only reachable cause and the precise slug is the true one.

A18–A19 · produced by rmcp, not by this crate

#ConditionCodeRetriableAgent shouldSource
A18Unknown JSON-RPC method (e.g. a tool name that is not registered)-32601NoCall a name from tools/list.rmcp; not constructed here — mcp_server/server.rs:166-177, pinned by tests/mcp_transport_contract_test.rs:138
A19A panic escapes the HTTP request path-32603 body, HTTP 500NoReport it. The body echoes the JSON-RPC id the client sent whenever one can be recovered (see the rules below). The tool-level panic path (B1) is the one that keeps the session alive.mcp_server/panic_containment.rs (jsonrpc_internal_error_response), outermost backstop in mcp_server/server.rs (jsonrpc_panic_response)

Which id a contained panic echoes

The -32603 body always carries an id, and what it carries is a stated rule, not a best effort. Both containment layers share one response builder, so the shape never depends on which layer caught the panic.

id echoedwhenrule id
the parsed id memberPOST body is an object with an id memberenvelope
nullthe object closed with no id member (a JSON-RPC notification)notification
nullthe object sent "id": null explicitlynull_id
nullthe body is a top-level array (a JSON-RPC batch)batch
nullnon-JSON, empty, a bare scalar, a non-POST request, or an unreadable bodynot_a_request
nullthe envelope did not close inside the 64 KiB scan windowscan_window

A null id is recorded on the jsonrpc.id_source span field (mcp_jsonrpc_panic_containment), so "the client sent a batch" is distinguishable from "the body was garbage" in the trace. null never means the mapping failed: the code is -32603 either way. Source of truth: crates/webfang_mcp/src/mcp_server/panic_containment.rs.


The reason-slug taxonomy — the machine-readable contract

data.reason is the half of this contract an agent branches on. Before slice 5, Channel A told a caller which field was wrong and nothing about why, so every consumer parsed prose. Since EC-08 the data member is an object and reason is one of a closed set of stable slugs, declared as a contract in mcp_server/validation.rs:64-99. There is deliberately no reason-less constructor left in the crate: the invalid_params(field, msg) escape hatch that path_gate.rs and the handler-inline sites used to call was deleted once nothing called it any more, and a caller that cannot name a slug must now pick the closest one (mcp_server/validation.rs:101-104). That is what makes the taxonomy exhaustive — a consumer branching on data.reason can rely on it being there.

The seven validation slugs

SlugMeansWhat the caller should do
emptya required value is emptyfill it in
too_longthe value exceeds a byte capshorten it
malformedpresent but unparseablefix its shape
unsupported_schemethe URL scheme is not http/httpsuse http or https
path_not_allowedany structural path rejection — the filename rules (absolute, UNC prefix, drive letter, .., non-flat filename, Windows reserved name, Windows-invalid character, trailing ./space) and the export-root confinement rules in path_gate.rs (rooted-not-absolute, .., absolute with no roots, absolute outside every root)use a safe relative path and a plain filename
out_of_rangea numeric value is outside its inclusive boundsclamp it
not_in_allowed_setan enum-ish value is not in the allowed listpick a listed value

Coarse is deliberate, and the set is load-bearing. A slug's job is to tell the agent what to do, not to restate the message; seven stable values beat forty brittle ones. Downstream tooling branches on these strings, so do not rename one, add an eighth, or remap a condition to a different slug without changing this page in the same change. path_not_allowed deliberately collapses those twelve rules into one slug, because their remedy is one instruction — use a safe path — and the message carries the specifics.

The SSRF slugs

A separate, disjoint set — the SSRF pre-check classifies a target, not a request field, so it carries no field at all:

SlugClassCode
no_hostcaller input-32602
forbidden_ip_literalpolicy-32602
forbidden_ip_resolvedpolicy-32602
dns_resolution_failedinfrastructure-32603
dns_no_addressesinfrastructure-32603

The data shapes a client must tolerate

ShapeEmitted byReader
{"field": "<field>", "reason": "<slug>"}every field-bearing rejection: the 44-branch validation.rs funnel, the six inline bounds in params.rs (A1, A3.x), the six path_gate.rs branches (A4.x), and the ten handler-inline sites in handlers/export.rs / handlers/scraping.rs (A10–A14)index field and reason
{"reason": "<slug>"} — no fieldssrf.rs, 3 branches (A5, A6, A9)index reason; field is legitimately absent

That is the whole of what the server produces. A client may still meet a bare string or a data with no reason in the wild, because error.data is sender-defined and validation::reason_of deliberately tolerates all three cases (data absent, a non-object, or an object with no reason key) rather than panicking (mcp_server/validation.rs:122-124, pinned by reason_of_tolerates_a_missing_or_foreign_data_payload). Keep that tolerance in the reader; do not document either as a shape this server can emit.

validate_url's parallel channel

The one success-shaped diagnostic that reports a rejection reuses the same slug under a different key, because it is not a JSON-RPC error and has no data: the body is {"valid": false, "reason": <message only>, "reason_code": <slug>} (row C1). reason is now the message alone; it used to be ErrorData::to_string(), which leaked the numeric code and a raw data blob into a human-facing field. reason_code carries the slug, so one taxonomy covers both channels. The valid: true field set is unchanged.


Message language — the ratified (inconsistent) contract

The repo-wide rule is "user-facing errors in Spanish, internal logs in English". The MCP error channels do not follow it uniformly, and that is now a documented state rather than an accident to be papered over.

ChannelLanguage as it actually is
A — structural / validationEnglish in validation.rs (every require_* message) and in path_gate.rs; Spanish in one deliberate island, validation::filename_component_error (mcp_server/validation.rs:291-323 — all six of its messages), and also in ssrf.rs, handlers/export.rs, and the js_strategy / checkpoint_dir / sitemap_url messages in params.rs and handlers/scraping.rs
B — operationalSpanish by intent, with a few English literals surviving: "no valid URLs provided" (mcp_server/handlers/scraping.rs:264) and the three "HTTP error: …" messages (mcp_server/handlers/scraping.rs:780-783, 852, 1019). Messages that forward an upstream Error::to_string() are language-dependent on their source.
C — success-shapedmixed: "no vault detected" and "not an SPA - sufficient content found" are English, and validate_url's failure reason is whatever require_http_url produced — also English.

Unifying this was deliberately not done in issue #1613. The reason is that it would buy nothing for a programmatic consumer: data.reason makes the branch language-free, and the message text is explicitly not part of the contract. Rewriting the message strings behind these rows would churn the diff of a contract change, put the language split in the path of existing message-matching probes, and change nothing a client can rely on. (How many strings that is was deliberately not counted: an exact figure would be a snapshot, and this page is about a contract, not a census.)

This is a deviation from the repo-wide convention, recorded here so a maintainer can revisit it. The honest framing: the convention says these strings are user-facing, and they are read by an autonomous agent rather than a person, so "user-facing" is arguably the wrong lens — but that is an argument, not a decision, and the decision belongs to the maintainer. If it is revisited, the taxonomy section is the anchor: only the slugs are frozen, so translating a message is a compatible change while renaming a slug is not.


Channel B — isError: true tool results

Every row here is built by provenance::neutralized_error (mcp_server/provenance.rs:275-294) or by the honest_error wrappers (mcp_server/handlers/ai.rs:38-42, mcp_server/handlers/axtree.rs:25-28). Text is neutralized (ANSI + C0/DEL stripped, fence sentinels escaped) and capped, but no provenance envelope and no code is added — the reason is the text and nothing else.

Slice 4 moved four conditions into this channel from Channel C: three serialization-failure classes that used to return a placeholder body as a success (now B8.1–B8.3) and the inverted Obsidian severity (now B7.3). The rule they restore is the one render_metrics already followed: if the tool could not produce the answer it promised, the answer is isError: true — never a success whose body happens to read like a failure.

B1 · contained panic

#ConditionCodeRetriableAgent shouldSource
B1A tool body panicked; the panic was caught at the dispatch boundarynoneYes, with a caveatRetry, but know that containment restores the transport, not the side effects — a tool that panicked after writing an export or charging a rate-limit token will duplicate that work.mcp_server/mod.rs:359-380 (containment + rationale), client text at mcp_server/mod.rs:268-270

B2 · site policy denial (robots.txt / WAF)

#ConditionCodeRetriableAgent shouldSource
B2.1robots.txt rules forbid the URLnoneNoStop, or retry with the tool's own opt-out (ignore_robots) if the caller has permission.mcp_server/handlers/scraping.rs:1154-1171 (robots_denied_response), used by mcp_server/handlers/scraping.rs:759-771 (discover_urls), mcp_server/handlers/scraping.rs:957-970 (detect_spa), and the inline gate at mcp_server/handlers/scraping.rs:74-84; mcp_server/handlers/ai.rs:122-124
B2.2The policy guard refused (guard condition, not a rules denial)noneNoTreat as a policy refusal.mcp_server/state.rs:375-390
B2.3Live scrape inside the export pipeline deniednoneNoAs B2.1; the message is prefixed with the target URL.mcp_server/handlers/export.rs:428-435

B3 · scrape / crawl operational failure

#ConditionCodeRetriableAgent shouldSource
B3.1scrape_url — fetch, Readability extraction, or content sink failednoneDepends on the class in the textRead the text; a transient network fault and a hard WAF block look the same here.mcp_server/handlers/scraping.rs:121-130
B3.2scrape_with_options — samenoneSameSame.mcp_server/handlers/scraping.rs:227-236
B3.3scrape_batch — whole-batch failurenoneYes if transientRetry the batch; per-URL failures inside a successful batch are rows, not errors (C3).mcp_server/handlers/scraping.rs:380-398
B3.4crawl_site — engine run failednoneYes if transientRetry the crawl.mcp_server/handlers/scraping.rs:534-544
B3.5crawl_with_sitemap — discovery failednoneYes if transientRetry.mcp_server/handlers/scraping.rs:713-731
B3.6crawl_with_sitemap — sitemap session run failednoneYes if transientRetry.mcp_server/handlers/scraping.rs:694-711
B3.7discover_urls / discover_sitemap — link extraction or sitemap read failednoneYes if transientRetry.mcp_server/handlers/scraping.rs:830-840, mcp_server/handlers/scraping.rs:915-925
B3.8download_assets — asset fetch or write failednoneYes if transientRetry; already-downloaded assets are content-addressed.mcp_server/handlers/assets.rs:88
B3.9extract_links — base-URL resolution failednoneNoFix the input HTML or base URL.mcp_server/handlers/content.rs:87
B3.10url_to_file_path — OutputPath::from_url failednoneNoFix the URL.mcp_server/handlers/url_utils.rs:210
B3.11crawl_site / crawl_with_sitemap — the run's extracted records exceed the session result budget (64 MiB default), so nothing was retained for export (#1611, F7)noneYes, with a smaller runRe-run with fewer max_pages, or crawl per section. The text names the budget, the measured size, and how many records were kept vs dropped.mcp_server/handlers/scraping.rs (within_session_budget, session_budget_message, store_session_results)

The per-class retry policy these texts inherit is docs/error-classification-matrix.md. The MCP channel carries no exit code, so the class is not machine-readable here — the agent must parse the text.

B4 · direct-fetch HTTP failures

#ConditionCodeRetriableAgent shouldSource
B4.1discover_urls — non-2xx status (a real failure, deliberately not an empty link set)none4xx no / 5xx yesRead the status in the text.mcp_server/handlers/scraping.rs:776-791
B4.2discover_urls — transport failurenoneYesRetry.mcp_server/handlers/scraping.rs:844-853
B4.3detect_spa — transport failurenoneYesRetry.mcp_server/handlers/scraping.rs:1010-1020
B4.4semantic_cleaner — page fetch failednoneYes if transientRetry.mcp_server/handlers/ai.rs:134-143
B4.5get_accessibility_snapshot — snapshot fetch failed (both formats)noneYes if transientRetry.mcp_server/handlers/axtree.rs:136-147, 168-180

B5 · export failures

#ConditionCodeRetriableAgent shouldSource
B5.1No session results to export (no crawl ran, or its extraction produced nothing)noneNoRun a crawl first. This is a precondition, not a fault.mcp_server/handlers/export.rs:115-121
B5.2Blocking-pool join for the session snapshot failednoneYesRetry.mcp_server/handlers/export.rs:136-142
B5.3export_file with empty / whitespace-only contentnoneNoSend real content.mcp_server/handlers/export.rs:186-190
B5.4export_file — document chunk validation failednoneNoFix the content.mcp_server/handlers/export.rs:269-275
B5.5Exporter construction failednoneDependsRead the text.mcp_server/handlers/export.rs:278-284
B5.6Export write failed (all four export tools)noneDependsRead the text.mcp_server/handlers/export.rs:57-61, 294-298
B5.7process_export_pipeline — live scrape failednoneDependsRead the text; it names the URL.mcp_server/handlers/export.rs:444-449

B6 · feature gating and missing ports

#ConditionCodeRetriableAgent shouldSource
B6.1AI tool called with no embeddings provider configurednoneNoConfigure a remote embeddings provider (--embedding-provider <id> or WEBFANG_EMBEDDING_PROVIDER); there is no local model and no flag that turns one on.mcp_server/handlers/ai.rs:58-64 (ai_unavailable), used at mcp_server/handlers/ai.rs:137, mcp_server/handlers/ai.rs:206-208, mcp_server/handlers/ai.rs:216
B6.2AI not compiled into the binary retired (ADR-0004 paso 6: the ai feature is gone; the AI tools are always compiled in and resolve their EmbeddingPort remotely via --embedding-provider)noneNonone — rebuild advice no longer applies; see ai-onnx-migration.md.same helper, mcp_server/handlers/ai.rs (line numbers may move)
B6.3Vault note repository not configurednoneNoOperator must configure persistence.mcp_server/handlers/ai.rs:201-204
B6.4chromium not compiled (axtree tool)noneNoRebuild with --features chromium.mcp_server/handlers/axtree.rs:185-197
B6.5Semantic cleaner / inference / embedding failednoneNoReport it; this is a server-side AI fault.mcp_server/handlers/ai.rs:144-147, 270
B6.6Response serialization failed (AI + axtree)noneNoReport it; the payload could not be built.mcp_server/handlers/ai.rs:161-168, 263-268; mcp_server/handlers/axtree.rs:121-133

B7 · local integration failures

#ConditionCodeRetriableAgent shouldSource
B7.1build_obsidian_uri / open_in_obsidian — vault name or file path rejectednoneNoFix the vault name or note path.mcp_server/handlers/obsidian.rs:72-77, 99-104
B7.2open_in_obsidian — the dispatch command could not be started at all (Err)noneYesRetry; Obsidian may not be installed.mcp_server/handlers/obsidian.rs:137
B7.3open_in_obsidian — the URI handler reported DispatchStatus::HandlerFailed (the OS handler exited non-zero)noneYesRetry after installing/registering the handler. Moved here from Channel C in slice 4: the requested action did not happen, so it is a tool error. The ⚠️ marker that used to fake the severity inside a success body is gone — the envelope carries it now.mcp_server/handlers/obsidian.rs:134-136 (in dispatch_outcome_to_result, mcp_server/handlers/obsidian.rs:126-139)
B7.4get_scrape_metrics — no scrape has been recorded in this processnoneNoRun a scrape first. Precondition, not a fault.mcp_server/handlers/security.rs:204-208
B7.5get_scrape_metrics — snapshot serialization failednoneNoReport it. This is the render_metrics precedent the four B8 sites now follow.mcp_server/handlers/security.rs:209-213
B7.6scrape_batch — every input URL was rejected before the runnoneNoFix the URLs.mcp_server/handlers/scraping.rs:263-265

B8 · reply serialization failed (server-side)

Added in slice 4. Each of these four sites used to do to_string_pretty(..).unwrap_or_else(|_| "failed to serialize") and return that literal as a successful result — a lost report that was indistinguishable from a real one, which is the single shape an agent cannot detect. They are ordinary Channel B rows now.

#ConditionCodeRetriableAgent shouldSource
B8.1download_assets — the asset list could not be serializednoneNoReport it; the download report was lost. Do not treat any body as authoritative.mcp_server/handlers/assets.rs:113-125 (assets_tool_result)
B8.2extract_links — the link list could not be serializednoneNoReport it; the link list was lost.mcp_server/handlers/content.rs:234-246 (links_tool_result)
B8.3discover_urls / discover_sitemap — the URL list could not be serializednoneNoReport it; the link list was lost.mcp_server/handlers/scraping.rs:1239-1256 (discovered_urls_tool_result)

The old placeholder body cannot be produced any more — a client no longer needs to special-case the literal string "failed to serialize". These rows are not -32603: the enclosing match already returns a CallToolResult directly, and ?-propagating would have split one function's failures across two channels. Retrying will not help; the payload types' Serialize impls are total, so these branches are defensive.


Channel C — success-shaped diagnostics

A result with no isError and no error member. The call succeeded; the body describes something that is not a plain success. This is the channel an agent cannot distinguish from a real success by shape alone — that is why it is enumerated rather than folded into B.

Slice 4 shrank this channel from six conditions to the four below, and the IDs were renumbered: the old C2 (the Obsidian severity inversion) became B7.3, and the old C3 (the "failed to serialize" placeholder) was eliminated outright — its four sites are now B8.1–B8.3. Old C4/C5/C6 are therefore new C2/C3/C4. What remains here is a deliberate diagnostic, not a leak: every row below is a tool that succeeded at the only thing it actually does, and reports a negative finding in the body.

| # | Condition | Code | Retriable | Agent should | Source | | :-: | :--- | :-: | :-: | :-: | :-: | :-: | | C1 | validate_url reports an unusable URL as {"valid": false, "reason": …, "reason_code": …} | none | No | Parse the JSON and branch on valid — never on isError. This is the only tool whose failure is a success by design (issue #590, bug #7). Since slice 5 its reason is the message alone and the slug moved to its own reason_code field; the tool's description now states the success-shaped contract outright. | mcp_server/handlers/url_utils.rs:73-82; success twin at mcp_server/handlers/url_utils.rs:58-71 | | C2 | A run completed with per-item failures (failed_url records; errors / error_breakdown fields) | none | Per item | Read the per-item records; the run itself is a success. | mcp_server/handlers/scraping.rs:1129-1142 (failure records), returned through mcp_server/handlers/scraping.rs:372-379; crawl summary at mcp_server/handlers/scraping.rs:519-534 | | C3 | detect_obsidian_vault found no vault ("no vault detected") | none | No | A negative result, not a failure. | mcp_server/handlers/obsidian.rs:53-55 | | C4 | detect_spa found sufficient content ("not an SPA - sufficient content found") | none | No | A negative result, not a failure. | mcp_server/handlers/scraping.rs:1005-1007 |

The old C3 row is gone for good. It promised that a body equal to the literal "failed to serialize" meant a lost report. Slice 4 removed the four sites that could produce it, so no tool can emit that body any more. A client still special-casing the string is guarding against something the server no longer does — the equivalent failure now arrives as an ordinary isError: true result (B8.1–B8.3).


Channel 0 — transport refusals (no JSON-RPC exists yet)

These never reach a tool. The client sees an HTTP status and, in most cases, no JSON-RPC body at all. Every row here is statically derived — these statuses are produced by middleware, not by a tool, and no test in this crate observes them over a live listener.

#ConditionStatusRetriableAgent shouldSource
T1Missing / malformed / wrong bearer token, and the server's own fail-closed default: no token configured and anonymous operation not explicitly enabled (#1611, G-18)401NoFix the Authorization header — or, if the server was started without a token, know that the operator must set one (--auth-token / WEBFANG_MCP_AUTH_TOKEN) or opt in to --allow-anonymous on loopback. A 401 with no token configured is a SERVER configuration state, not a wrong guess: the binary refuses to start in that state unless the opt-in is present.mcp_server/auth.rs, require_auth_or_explicit_anonymous in mcp_server/server.rs
T2Rate-limit quota exceeded429 + Retry-AfterYes, after backoffSlow down for the advertised number of seconds. The quota is per credential (keyed), so another caller's traffic does not consume yours.mcp_server/server.rs (rate_limit_middleware, rate_key)
T3Session-admission cap exceeded429Yes, after the windowSlow down. Bounded by --max-sessions / --session-cap-window-secs.mcp_server/server.rs:629-677
T4Request took longer than --request-timeout-secs408YesRetry. Note what it bounds: the request FUTURE, not a streaming body — once an SSE stream is answering, events keep flowing past this deadline (the accepted residual half of #1611 G-20; the "SSE channels uncapped" half no longer holds on rmcp 1.8.0, whose session/event channels are built at SessionConfig::channel_capacity = 16).mcp_server/server.rs (TimeoutLayer)
T5Body over --body-limit-bytes413NoShrink the payload.mcp_server/server.rs:225
T6Accept missing both application/json and text/event-stream406NoFix the request headers.rmcp StreamableHttpService; matrix at mcp_server/server.rs:166-173
T7Content-Type not application/json, or the body is not JSON-RPC 2.0415NoFix the request.rmcp; same matrix
T8Non-initialize request with no mcp-session-id422NoComplete the handshake first.rmcp; same matrix
T9Panic on the HTTP request path500 + JSON-RPC -32603 bodyNoReport it. The body echoes the JSON-RPC id when one is recoverable; see the id rules above.mcp_server/panic_containment.rs

Layer order matters for reading this, and #1611 F5 changed it. Outermost to innermost: panic backstop → tracing → body limit → timeout → auth → rate limit → session cap → id-echoing panic containment → the rmcp service. Two properties follow, and each is pinned by a test rather than by a comment:

  • A request refused 401 (no or wrong token) never reaches the rate limiter, so an unauthenticated flood cannot spend the authenticated caller's quota and lock the operator out of its own server.
  • A request refused 401 or shed 429 never consumes a session slot, so the session cap measures sessions rather than rejections.

Unsettled rows

EC-02 — SETTLED by test, not by reasoning. The question was whether A1/A2 (raised inside rmcp's argument deserialization) and A3 (raised by the handler body) reach the client as the same JSON-RPC code, given that they share one Rust error type (McpError::invalid_params is rmcp::ErrorData). They do not, and crates/webfang_mcp/tests/mcp_error_channel_mapping_test.rs now records both error.code and result.isError for all three routes against a live server:

RouteObservedTest
McpUrl deserialization (scrape_url with file:///etc/passwd)result.isError = true, no error membermcp_error_channel_mapping_test.rs:99-127
unknown field (scrape_with_options with typo_field)result.isError = true, no error membermcp_error_channel_mapping_test.rs:139-166
handler params.validate()? (crawl_site with max_depth: 11)error.code = -32602, no result membermcp_error_channel_mapping_test.rs:196-217

The mechanism is rmcp's own message prefix, described in A1–A3. No production code changed to settle it. The older hedged helper tests/params_rejection_test.rs::assert_url_argument_rejected, which "accepts EITHER rejection shape", is now strictly weaker than the new suite and is kept only because it also asserts the reason text; it is no longer the evidence for these rows.

Rows the code slices changed — all landed:

FindingRow affectedSliceStatus
EC-02 — the rmcp argument-rejection mappingA1, A2, A33Landed — settled by test, rows rewritten above
EC-03 — SSRF policy and DNS failures shared invalid_paramsA7, A82Landed — -32603 + data.reason; classes separated
EC-04 — the third channel was undocumentedC15Landed — reason_code slug + an explicit success-shaped description
EC-06 — a serialization failure returned success with a placeholderwas C34Landed — C3 deleted; the four sites are now B8.1–B8.3
EC-07 — DispatchStatus::HandlerFailed was returned as tool successwas C24Landed — moved to B7.3, ⚠️ marker dropped
EC-08 — validation text was English while operational errors were Spanish; no machine-readable reasonA3.x, A4.x, A10–A14, B*, C15Landed on the machine-readable half — data.reason is now present on every argument refusal the server raises (the -32603 server faults, A15/A16, carry no data and never did). The 16 sites slice 5 left behind (6 in path_gate.rs, 10 handler-inline in handlers/export.rs / handlers/scraping.rs) were converted, and the reason-less constructor they called was deleted, so both shapes that survived slice 5 are gone; see A4, A10–A17 and the shape table. The language half was deliberately not done, and remains open: see Message language.

EC-05 — a deliberate, justified non-action, not an oversight. The seven panic-capable serialization expect sites are unchanged by design: mcp_server/handlers/content.rs:209, mcp_server/handlers/scraping.rs:532, 691, 1002, and mcp_server/handlers/url_utils.rs:70, 80, 207 (all expect("serializing JSON to a string cannot fail")). Each carries an explicit #[allow(clippy::expect_used)] under a crate-wide deny; each operand is a serde_json::Value, a total Serialize; and a panic there is contained into isError: true at the dispatch boundary (row B1) without killing the session. The correct remedy — a proven-total serialization boundary or an explicit internal_error mapping — would be a behavioural change, not a refactor, and the issue explicitly warns that blind replacement is churn. Recorded here so the next reader does not re-open it as a bug.

Overlapping, not duplicated here: EC-08's message-language half overlaps #1604, which normalizes operational error event text to structured fields. #1604 does not own the message-language or machine-readable-reason halves; those are decided on this page.


Appendix — construction-site inventory

Counted in crates/webfang_mcp/src/mcp_server/ in the working tree — that is, commit ccdef614 plus the uncommitted reason-slug changes described under EC-08 — production code only (test modules and doc comments excluded). ccdef614 is the base, not a clean checkout: validation.rs, path_gate.rs, handlers/export.rs and handlers/scraping.rs are all dirty in this tree.

How the counts were derived — each row is a grep for one construction form, restricted to the mcp_server/ tree, with comment-only lines dropped by a /// / //! filter. Two counts deserve a note rather than a bare number:

  • provenance::neutralized_error is counted as production call sites, so the fn neutralized_error definition and every #[cfg(test)] call are excluded. That leaves 37, of which 2 are the honest_error wrapper bodies and 4 are the mapper helpers slice 4 introduced; the 31 remaining are the same direct handler-body calls the previous revision of this table counted (that figure reproduces exactly at 763714b6, which is the cross-check that the method is stable).
  • Channel C has no neutralized_error/CallToolResult::error sites at all, and its local_text sites are only 5. The old "10 construction sites across 6 conditions" is not reproducible by any single grep; the honest number is 5 local_text sites across 3 standalone conditions, plus C2, which is a field inside a success payload rather than a construction of one.
ChannelConstruction formSitesAccounted for
AMcpError::invalid_params called directly4 — the one surviving factory (mcp_server/validation.rs:110) and the SSRF channel (mcp_server/ssrf.rs:122, 135, 178)A5, A6, A9
Avalidation::invalid_params_with_reason branches (the validation funnel)44 — all in mcp_server/validation.rs (lines 139–756)A3.1–A3.9
Avalidation::invalid_params_with_reason call sites in params.rs6 — mcp_server/params.rs:97, 155, 176, 463, 527, 536A1, A3.4, A3.5, A3.6, A3.7
Avalidation::invalid_params_with_reason call sites in path_gate.rs6 — mcp_server/path_gate.rs:282, 289, 298, 309, 327, 345A4.1–A4.6
Avalidation::invalid_params_with_reason call sites in handlers/export.rs8 — mcp_server/handlers/export.rs:197, 210, 235, 252, 326, 364, 410, 470A10–A12
Avalidation::invalid_params_with_reason call sites in handlers/scraping.rs2 — mcp_server/handlers/scraping.rs:458, 605A13, A14
AMcpError::internal_error9 — mcp_server/handlers/scraping.rs:106, 109, 212, 215, 1121, 1127; mcp_server/macros.rs:16; mcp_server/ssrf.rs:154, 166A7, A8, A15, A16
Are-raised Err(e) — forwarding, not a construction1 — mcp_server/handlers/axtree.rs:74A17
Bprovenance::neutralized_error called directly37 — 31 in handler bodies + 2 honest_error wrapper bodies (handlers/ai.rs:41, handlers/axtree.rs:27) + 4 in the slice-4 mapper helpers (handlers/assets.rs:120, handlers/content.rs:241, handlers/scraping.rs:1251, handlers/obsidian.rs:134)B1–B8
Bhonest_error wrapper definitions2 — mcp_server/handlers/ai.rs:40, mcp_server/handlers/axtree.rs:26(definitions only)
Bcalls through those wrappers14 — mcp_server/handlers/ai.rs:124, 129, 142, 146, 164, 197, 202, 207, 265, 270; mcp_server/handlers/axtree.rs:130, 144, 176, 194B2.1, B4.4, B4.5, B6.1–B6.6
Bdirect CallToolResult::error1 — mcp_server/mod.rs:377B1
Cprovenance::local_text on a non-success outcome5 — mcp_server/handlers/url_utils.rs:68, 78; mcp_server/handlers/obsidian.rs:54, 55; mcp_server/handlers/scraping.rs:1005C1, C3, C4
Cper-item failure records inside a success payload (not a standalone construction)2 payloads — mcp_server/handlers/scraping.rs:372, 519C2
0HTTP status producers in this crate6 — mcp_server/auth.rs:59; mcp_server/server.rs:223, 225, 279, 311, 672T1–T5, T9
0HTTP status producers owned by rmcp's StreamableHttpService3 — not constructed here; enumerated in the matrix at mcp_server/server.rs:166-173T6–T8

Totals: 136 error-producing construction sites on the JSON-RPC channels (79 on A, 52 on B, 5 on C), plus 1 forwarding site and 6 in-crate HTTP status producers. Every one is accounted for by a row above; 0 unclassified. A is 4 + 44 + 6 + 6 + 8 + 2 + 9; B is 37 + 14 + 1.

Two scope statements, so the totals are not read as more than they are:

  • These are sites, not conditions. The eight invalid_params_with_reason calls in handlers/export.rs collapse into three rows (A10–A12), because several sites report the same condition for different tools; the reverse also holds, with 44 funnel branches collapsing into nine A3.x rows.
  • A1's construction site is the McpUrl::try_from branch at params.rs:97, but it is not a Channel A construction — it travels on Channel B. It appears in this inventory only because that is where its Rust error is defined; its row (A1) is a Channel B row, and no error member is emitted for it at all.

MCP Public Surface Policy

What may change in webfang_mcp without breaking somebody, and what requires a release. This is the written decision for issue #1614; the code attributes and the machine check below exist only because this document states the rules first.

The source of the findings is AUDIT-MCP-API-CONTRACT-20260925-035505/ → BREAKING_CHANGE_RISKS.md (BC-01 … BC-10).


The one-sentence version

A client written against the advertised inputSchema of release N must keep working against release N+1 for any change classified additive below, and must be re-validated by the human for anything classified breaking — and every classification below is checked by a test, not by good intentions.


1. Three boundaries, not one

The audit speaks of "the public surface" as if it were a single thing. It is three, with three different sets of consumers, and conflating them is what produces both false alarms and false comfort.

#BoundaryWho consumes itStability promise
AThe MCP wire contract: the tool inventory and each tool's advertised inputSchemaMCP clients in other processes. The only boundary with consumers outside this workspace.Strong. Classified below.
BThe Rust surface of webfang_mcp: pub items in crates/webfang_mcp/src/mcp_server/This workspace only. See §2.Explicitly none. No semver promise.
CThe webfang_core port traits (domain::*_port, domain::*, application::crawler)Implementors. Today: this workspace only (see §2).Stated per trait in §6.

BC-10 is answered by this table, and the answer is not "we promise backwards compatibility". The reviewed source establishes no semver promise, and none is being invented here for a crate that is never published. What is being promised is a narrower and checkable thing: the wire contract is versioned by contract_version, and the workspace-internal Rust surfaces are declared unstable.

Why B carries no promise

release-plz.toml sets publish = false for every crate in the workspace and AGENTS.md states distribution is binaries-only. No webfang_mcp type has an external Rust consumer today, and webfang_cli does not depend on webfang_mcp at all (its mcp feature forwards to webfang_core/mcp).

The consequence is the opposite of the comforting reading: because nothing outside the workspace compiles against webfang_mcp, a Rust-level change to it cannot break a user. Changes there are reviewed on their own merits, not under a compatibility rubric. #[non_exhaustive] therefore buys a webfang_mcp type nothing against users — it only constrains this repository. That is why §4 stamps it in exactly three places and records a deliberate "no" everywhere else.


2. Consumers, stated once

Verified in this tree at 2b35f6ee:

  • webfang_cli → webfang_mcp: no dependency edge.
  • webfang_mcp's pub items referenced outside the crate: only its own integration tests under crates/webfang_mcp/tests/.
  • crates/webfang_mcp/tests/* are separate crates: they link webfang_mcp as an external dependency, so #[non_exhaustive] binds them exactly as it would bind a downstream user. This is not a technicality — it is the reason §4's list is short.

3. Compatibility classes (boundary A)

ClassMeaningRelease
S0 — stableNo client's observable behaviour changes. Docs, descriptions, internal logging.patch
S1 — additiveEvery previously-valid request stays valid and behaves identically. New clients gain capability.minor
S2 — breakingSome previously-valid request, or some previously-served response, changes meaning or stops being accepted.major

3.1 The compatibility matrix

One row per dimension of a property line in the checked-in manifest (crates/webfang_mcp/tests/fixtures/mcp_public_surface.tsv). "Tightens" always means fewer requests are accepted or more requests mean something different.

Change to a tool's advertised schemaClassNote
A new tool is registeredS1The inventory is append-only within a major.
A tool is removed or renamedS2Includes moving it behind cfg(feature).
A tool's description changesS0Prose, except where it makes a behavioural claim — see §7.
A Rust doc comment on a params type or enum changesS0 in intent, wire in factSee §3.3. schemars emits the doc comment into the advertised $defs.<Type>.description, so it is a wire string, not documentation.
A property is added and is not in requiredS1
A property is added and is in requiredS2Existing requests omit it.
A property is removedS2Even if the handler still tolerates it: deny_unknown_fields is on every params struct, so it becomes a rejection.
A property's type changesS2
Nullability widens (accepts null where it did not)S1Strictly more requests accepted; the new requests previously errored.
Nullability tightens (null no longer accepted)S2
default is added, and the runtime already applied exactly that valueS1Pure advertisement of existing behaviour.
default changes, and the runtime changed with itS2A silent behaviour change for a client that omitted the field.
default changes in the schema onlyS0 defect, not S1This is drift, not a change. It is a bug in whichever side moved alone; see §5.
A bound widens (minimum lowers, maximum raises)S1
A bound tightensS2
An enum gains a variantS1Announce it. A client that matches exhaustively over JSON strings must add a wildcard.
An enum loses a variantS2
additionalProperties: false is removedS2Widens acceptance; deny_unknown_fields is a deliberate control.

3.2 Response payloads

Response payloads are out of scope for versioning in this issue (audit slice 4, BC-08). Three channels already exist and must be used instead of a fourth shape: see mcp-error-contract.md. For success payloads the promise this policy makes is narrow and honest: the envelope is stable, the body is not versioned. Handlers that emit Content::text with ad-hoc JSON are not covered by the manifest and must not be described as covered.

3.3 A doc comment is not a doc change

Discovered while implementing this issue, and worth stating before somebody learns it the expensive way: schemars copies a type's Rust doc comment into the advertised schema as $defs.<Type>.description. Writing a thorough doc comment on a type named by a params struct therefore changes what an MCP client reads.

The first draft of #[non_exhaustive] on SnapshotFormatParams carried its rationale in the doc comment and failed mcp_advertised_schema_snapshot_test with a multi-line description diff. The rationale now lives in a // comment directly above the doc comment, and the doc comment is frozen.

Rule: a type that appears in an advertised schema keeps its doc comment byte-stable. Put the reasoning next to it, not in it. Types that never reach a schema (Outcome, DefaultOverride) are unconstrained.


4. #[non_exhaustive] — where, and why not everywhere

#[non_exhaustive] is not free. Adding it to a type that external code matches exhaustively, or constructs with a struct literal, is itself a breaking change for those consumers — the standard library's own std::collections::hash_map::Entry addition was a breaking change for exactly this reason.

4.1 Applied

TypeFileWhy it is safe here
SnapshotFormatParamsparams.rs (BC-02)It is deserialized, never constructed: McpUrl sets the precedent (private field, TryFrom<String> only). No test crate constructs it. The two in-crate exhaustive matches in handlers/axtree.rs stay legal — non_exhaustive only binds outside the defining crate.
DefaultOverrideschema_bridge.rsA bridge vocabulary already extended once in place (SetBounds, #1294), i.e. it demonstrably grows. Only apply_default_overrides matches it, in-crate.
Outcomemetrics.rsA metrics bucket whose set has already grown with each new outcome concept (admission control, #1611). Only in-crate.

4.2 Deliberately NOT applied

The 16 public params structs (BC-03). Not an oversight, and not a half-measure — a decision with three reasons:

  1. It would break the only consumers that exist. crates/webfang_mcp/tests/ build these structs with struct literals in 16 places (mcp_params_validation_test.rs). Adding the attribute turns every one of them into a compile error, and the fix is 16 constructor builders that buy nothing: no user compiles against this crate (§2).
  2. The compatibility they threaten is already protected by the right mechanism. The wire surface of these structs is the advertised schema, and that is pinned byte-for-byte by mcp_advertised_schema_snapshot_test (#1612) and semantically by options_spec_parity_test. The gap #[non_exhaustive] would close — "someone constructs this struct from outside" — is empty.
  3. The attribute encodes the wrong axis. These structs grow by field, and their required-ness is a wire property that already has a machine check. An attribute that constrains struct literals would not notice a new required field, which is the actual S2 risk.

If this crate is ever published, revisit 4.2 with constructors — not before.

Origin (provenance.rs). Its three variants are a complete semantic partition ("where does this text come from"), not a growing vocabulary. Adding a variant would change provenance labelling semantics, which is a correctness matter under §7, not a compatibility lever.


5. Defaults, nullability and bounds (BC-09)

These are compatibility-bearing, and the audit's "no-op schema fields may activate on a later build" is the sharp form of the risk: a field that is inert today and accidentally becomes live is a behaviour change with no code review signal.

Three rules, each machine-checked:

  1. An advertised default must equal the applied default. Enforced for OptionsSpec-backed properties by options_spec_parity_test (overridden_defaults_advertise_runtime_effective_values) and for MCP-only properties by mcp_advertised_defaults_test (#1294 NS-04). The bridge's DefaultOverride::{Set, Unset, SetBounds} is the single sanctioned way to make them differ deliberately, and each override must name a real property — an override naming an unknown property warns and is skipped, so a renamed field cannot leave a stale override behind silently.
  2. A nullability union in the schema must match serde's acceptance. promote_spec_to_nullable reconciles an Option<T> field's derived ["<inner>", "null"] with the spec rendering; a mismatch is an S2 defect.
  3. A bound must equal the bound the validator enforces, taken from the same constant. concurrency advertised minimum: 0 while the validator rejected 0 — the class of drift this rule exists for.

The manifest in §8 pins the advertised side of all three for every registered tool, not only the bridged seven.


6. Port traits (BC-04, BC-05, BC-06)

Adding a required method to a public trait is source-breaking for every existing implementor. What determines the class is whether the trait is sealed:

TraitSealed?Adding a required method
CheckpointStore (application/crawler/checkpoint.rs)Yes — mod private { pub trait Sealed {} }, private moduleS1. No external implementor exists. This is the reference pattern.
SemanticCleaner (domain/semantic_cleaner.rs)Yes — same pattern. pub mod private rather than mod private, which is a visibility difference with no sealing effect: the bound private::Sealed is what seals it, and it is unsatisfiable outside the crate.S1.
NoteRepository, VaultNoteReaderNoS2 for any downstream implementor.
TextChunker, EmbeddingPortNoS2 for any downstream implementor.

BC-05 is settled: SemanticCleaner is sealed. The audit's "Needs verification" is closed by reading the bound, not by a compile-fail fixture — which would have required a trybuild dev-dependency this issue may not add. The residual risk is a future contributor changing pub mod private to pub(crate) mod private; that still seals, and a blanket private::Sealed bound is the thing to protect. Adding the sealing compile-fail fixture remains open and is recorded here as future work rather than silently skipped.

The migration rule for BC-04: an unsealed port trait that acquires a new capability should first gain a defaulted method (S1), and only take a required one (S2) behind a major. Sealing first makes every later method S1.


7. Always-listed tools (BC-07)

Verified in this tree: handlers::ai::build_router and handlers::axtree::build_router are called unconditionally from build_tool_router, with no #[cfg] on the function or the call. So the tool inventory is identical across feature builds, and that is a deliberate strength the audit correctly told us not to regress.

The client-facing consequence, which is the part BC-07 left open:

  • A tool is never absent. Availability depends on features and runtime, not on registration.
  • An unavailable tool answers with a result, not an absence. Handlers route through honest_error, producing isError: true with Spanish content text and a reason slug — never a protocol error, never a false success.
  • Therefore: gating a registration behind cfg(feature), or dropping a tool from the always-listed inventory, is S2. The audit's "verify the client policy before changing registration" is answered here, and the answer is "do not".

The one exception is test_probe, registered only under an env gate. It is a zero-argument probe for #1646 panic containment, not a product tool; it is excluded from the manifest for that reason, and the exclusion is stated here rather than left implicit.


8. What is checked, and by what

Policy nobody can check drifts; this section exists so the coverage claim is auditable rather than asserted.

RuleEnforced byRuns in CI
Every registered tool's inventory (names + count)public_surface_policy_test::tool_inventory_is_pinnedyes
Per-property type / nullability / default / minimum / maximum / requiredpublic_surface_policy_test::advertised_property_matrix_is_pinnedyes
Advertised default == applied default (bridged + MCP-only)options_spec_parity_test, mcp_advertised_defaults_test (#1612, #1294)yes
Bridged schema byte-for-bytemcp_advertised_schema_snapshot_test (#1612)yes
A shape change moved contract_versionscripts/check_mcp_public_surface.shyes — step MCP public-surface version marker gate in repo-guards
The whole surface is feature-independent (36 tools, 81 properties, identical under default and --all-features)public_surface_policy_test run under both feature setsyes

8.1 contract_version

A single integer in crates/webfang_mcp/tests/fixtures/mcp_public_surface.tsv. It starts at 1, and:

  • Any S2 change bumps it, and requires a major release.
  • Any S1 change that alters a compatibility-bearing row bumps it, and requires a minor release.
  • S0 changes do not bump it.

8.2 The honest gap

The manifest test compares the live surface against the committed fixture, so it fails in CI the moment a shape moves and the fixture is not updated. It cannot know whether the author bumped contract_version, because both live in the same file.

scripts/check_mcp_public_surface.sh closes that half. It compares the fixture against a base commit, classifies every changed row per §3.1, and exits non-zero when the surface moved and contract_version did not — and also in the reverse case, a version bump with no surface change, which would silence the guard without a reason.

The base is not the merge-base with main, and that is deliberate. On a feature branch that introduces the fixture mid-stack, a merge-base base makes every row read as "added" and the guard goes blind right after the introducing commit. The local fast-gate call uses the last commit that touched either file; the CI step uses the event's own base (the PR's base SHA, before on a push, base_sha on a merge queue entry). Both ask the question that actually matters: did this change move the surface without moving the marker?

It runs in two places. Locally it is part of scripts/ci_fast_gate.sh (the pre-push gate). In CI it is the MCP public-surface version marker gate step of the repo-guards job in .github/workflows/ci.yml. repo-guards has no paths: filter and CI Gate already requires it in both scopes, so a marker that does not move reds the required check even on a docs-only PR; the path-filtered mcp lane cannot host it, because a job behind paths: never reports on every PR and therefore cannot be a required context.

The default depth-1 checkout does not contain the event's base commit, so the step fetches that one commit (--depth=1) and fails closed if it is still missing. That check is load-bearing: the guard reads a fixture that is absent at base as the bootstrap commit and exits 0, so a silently failed fetch would otherwise turn the gate green. The guard draws that line itself too: a base that is not a commit, or a tracked file that exists at base and cannot be read, exits 2 instead of being flattened into the same empty file as a genuine bootstrap. Schedule and workflow_dispatch runs have no change under review and skip the step with an explicit note.

The edges of this wiring are pinned by scripts/test_mcp_surface_wiring.sh, which runs in repo-guards beside the gate and in the fast gate's CI-only lane: the step passes --base from the resolve step's output (never a literal), repo-guards stays unfiltered and unconditional, gate waits on it in both scopes, and a base that is not a commit is refused rather than read as bootstrap.


9. Per-finding disposition

idfindingdisposition
BC-01Export names not aligned (export_format ↔ wire format)Root cause owned by #1612. Here: recorded as the canonical case for §3.1 — renaming either side is S2. Not re-fixed.
BC-02SnapshotFormatParams exhaustiveFixed — #[non_exhaustive], §4.1.
BC-0316 params structs unannotatedDecided: no attribute, with reasons, §4.2.
BC-04Unsealed public port traitsPolicy declared, §6. Migration rule stated. Sealing not performed — see §10.
BC-05SemanticCleaner sealing unverifiedSettled: it is sealed. §6. Compile-fail fixture remains open.
BC-06CheckpointStore sealed, no method-addition policyPolicy declared, §6. Confirmed as the reference pattern.
BC-07AI/chromium always registeredClient policy declared, §7. Verified ungated — and the pinned surface is byte-identical under both feature sets, which proves the claim instead of asserting it.
BC-08Output payloads untyped/unversionedDeferred — the manifest covers input schemas only, and §3.2 says so. Follow-up recorded.
BC-09Defaults / nullability / boundsRules + machine check, §5.
BC-10No semver promise statedDeclared, §1 — narrowed to three boundaries, with B and C explicitly promised nothing.

10. Explicitly not done

  • No re-fix of the schema drift (BC-01). #1612 owns it and it is delivered. Nothing here changes a wire name, a field name, or an advertised schema.
  • No sealing of the BC-04 traits. It is a webfang_core change with a wide blast radius across ai, mcp, cli, benchmark and test_utils, and three sibling issues branch off this branch. The policy above states what sealing would buy; the migration is its own change.
  • No output-DTO versioning (BC-08). Recorded as slice 4.
  • No compile-fail fixture for BC-05 — it needs a dev-dependency this issue may not add. The bound was read instead, which settles the same question.

11. When you change something on this surface

  1. Classify the change with §3.1 before writing it.
  2. Run cargo nextest run -p webfang_mcp. A red manifest test is not a chore to re-record — it is the answer to "did I classify this right?".
  3. If a row changed, run scripts/check_mcp_public_surface.sh --record, then read its diff. It prints the class of every row that moved.
  4. If any row moved, bump contract_version and put the release requirement in the commit body.
  5. If a row moved and you classified it S0, you are wrong about one of them — go back to §3.1.