Overview
WebFang is a production-ready web scraper built in Rust (1.91), with Clean Architecture, a TLS-fingerprinting HTTP client, optional AI semantic cleaning (remote embeddings — see the migration guide, ONNX removal), and sitemap-based crawling.
This book is the narrative documentation. The complete API reference for
the workspace crates is generated from source via cargo doc and published
alongside this book under /api/<crate>/ (e.g. /api/webfang_core/).
Crates
| Crate | Role |
|---|---|
webfang_core | Domain, application, and infrastructure layers — the scraping engine (semantic cleaning via remote EmbeddingPort adapters included) |
webfang_mcp | MCP server (36 tools) |
webfang_cli | CLI binary (webfang) |
webfang_test_utils | Shared test helpers |
Installing
WebFang ships binaries only — nothing is published to crates.io, so
cargo install webfang cannot work. Every release publishes three archives
per platform — the CLI, the MCP HTTP server and the MCP stdio server — across
Linux x86_64/ARM64, macOS Apple Silicon and Windows x86_64, plus a
SHA256SUMS.txt. Intel macOS has no artifact by design. Start here:
Installation & Uninstall.
Chapters
- Installation & Uninstall — verified download and checksum commands per platform, platform floors, and every on-disk root a release leaves behind.
- Debugging & Observability — built-in tracing, correlation
IDs, and the
jqquery cookbook fordebug.jsonl. - Testing — E2E integration tests, snapshot strategy, and coverage exclusions.
- Troubleshooting — diagnosing slow crawls, silent failures, WAF blocks, and async deadlocks.
- CLI Reference — the complete
webfangflag reference, auto-generated from the binary itself.
Regenerating this documentation
All heavy compute (mdBook build + rustdoc + link checks) runs in CI. Locally you only pull the generated result:
just docs # downloads the combined NotebookLM/LLM markdown from the latest CI run
just docs-local # local preview: builds and serves this mdBook only
Installation & Uninstall
WebFang is a binaries-only distribution. There is no cargo install webfang and nothing is published to crates.io — the release automation sets
git_only = true and publish = false in release-plz.toml, so a
cargo install webfang cannot work. Get the binary from
GitHub Releases.
Releases · Troubleshooting · CLI Reference · Release Feature Matrix
Quick path
- Pick your platform in the asset table and copy its download command.
- Verify the checksum with the command for your shell.
- Put the binary on your
PATHusing the per-platform steps (Linux, macOS, Windows). - Confirm it runs:
webfang --version.
If step 4 fails, go straight to install-time failures.
1. Pick your asset
Every release publishes twelve archives — three per platform — plus one checksum file. The asset name carries the full Rust target triple — there is no friendly OS alias and no version in the name (the version is in the tag/URL, not the file). Each archive contains a bare binary: no install script, no LICENSE, no README inside.
| Platform | CLI asset (inside the release) | Format |
|---|---|---|
| Linux x86_64 | webfang-x86_64-unknown-linux-gnu.tar.gz | .tar.gz |
| Linux ARM64 | webfang-aarch64-unknown-linux-gnu.tar.gz | .tar.gz |
| macOS Apple Silicon | webfang-aarch64-apple-darwin.tar.gz | .tar.gz |
| Windows x86_64 | webfang-x86_64-pc-windows-msvc.zip | .zip |
| Any platform | SHA256SUMS.txt | checksum manifest |
The other two archives per platform hold the MCP servers — same targets, same format as the CLI archive for that platform:
| Platform | HTTP server | stdio server |
|---|---|---|
| Linux x86_64 | webfang-mcp-x86_64-unknown-linux-gnu.tar.gz | webfang-mcp-stdio-x86_64-unknown-linux-gnu.tar.gz |
| Linux ARM64 | webfang-mcp-aarch64-unknown-linux-gnu.tar.gz | webfang-mcp-stdio-aarch64-unknown-linux-gnu.tar.gz |
| macOS Apple Silicon | webfang-mcp-aarch64-apple-darwin.tar.gz | webfang-mcp-stdio-aarch64-apple-darwin.tar.gz |
| Windows x86_64 | webfang-mcp-x86_64-pc-windows-msvc.zip | webfang-mcp-stdio-x86_64-pc-windows-msvc.zip |
The CLI archive never contains an MCP binary: webfang-mcp and
webfang-mcp-stdio each have their own archive, because every archive holds
exactly one binary.
There is no Intel macOS artifact. x86_64-apple-darwin is deliberately
not built. The original reason was ONNX-local (no x64 macOS prebuilt for
ONNX Runtime); the ONNX removal (ADR-0004 paso 6) makes that reason moot, but
the target list was never extended — each release re-trains on the artifact
list as-is, so the entry stays. If you are on an Intel Mac, there is nothing
to mis-download — see Intel macOS.
Set the version once and reuse it. Copy the tag from the latest release:
VERSION=v2.12.0 # ← the tag you just looked up
Verify these names yourself (any release, any version)
gh release view v2.12.0 --json assets --jq '.assets[].name'
Observed output for v2.11.0:
SHA256SUMS.txt
SHA256SUMS.txt.sigstore.json
THIRD_PARTY_NOTICES.txt
THIRD_PARTY_NOTICES.txt.sigstore.json
webfang-aarch64-apple-darwin.tar.gz
webfang-aarch64-apple-darwin.tar.gz.sigstore.json
webfang-aarch64-unknown-linux-gnu.tar.gz
webfang-aarch64-unknown-linux-gnu.tar.gz.sigstore.json
webfang.spdx.json
webfang.spdx.json.sigstore.json
webfang-x86_64-pc-windows-msvc.zip
webfang-x86_64-pc-windows-msvc.zip.sigstore.json
webfang-x86_64-unknown-linux-gnu.tar.gz
webfang-x86_64-unknown-linux-gnu.tar.gz.sigstore.json
2. Verify the checksum
Each release ships a SHA256SUMS.txt listing the SHA-256 of every archive.
Verify the archive you downloaded, before extracting it. The command
differs per platform: sha256sum is GNU coreutils and does not exist on
stock macOS, and PowerShell has neither it nor shasum.
Linux (GNU coreutils) — checks the archive against the manifest:
sha256sum -c SHA256SUMS.txt --ignore-missing
macOS (shasum, ships with the system) — BSD coreutils has no
-c/--ignore-missing; check the one file you downloaded:
shasum -a 256 webfang-aarch64-apple-darwin.tar.gz
grep 'aarch64-apple-darwin' SHA256SUMS.txt
The two hashes must be identical.
Windows (PowerShell) — Get-FileHash, compared against the manifest line:
(Get-FileHash webfang-x86_64-pc-windows-msvc.zip -Algorithm SHA256).Hash.ToLower()
Select-String -Path SHA256SUMS.txt -Pattern 'pc-windows-msvc'
If you have GitHub CLI installed, it fetches both files for you (it does not verify them — do step 2 anyway):
gh release download "${VERSION}" --pattern 'SHA256SUMS.txt'
gh release download "${VERSION}" --pattern 'webfang-x86_64-unknown-linux-gnu.tar.gz'
Every published asset carries a real SLSA provenance attestation, minted keyless by the release workflow, plus a
.sigstore.jsonbundle next to it in the release. Verify a downloaded file against the digest you actually have:gh attestation verify webfang-x86_64-unknown-linux-gnu.tar.gz \ --repo XaviCodeDev1000/webfang
SHA256SUMS.txtis the offline check and stays authoritative for integrity. The provenance table in the release body is WebFang's own L1 gate — a summary written by the workflow, not the attestation itself; verify against the API or the.sigstore.jsonbundle, never against that table.
3. Install
Every recipe below is copy-pasteable as a unit. The three unix steps — download, verify, extract — are identical apart from the asset name, so only the differing part is repeated per platform.
Linux
# 1. Download (x86_64; use aarch64-unknown-linux-gnu on ARM64)
VERSION=v2.12.0
curl -fLO "https://github.com/XaviCodeDev1000/webfang/releases/download/${VERSION}/webfang-x86_64-unknown-linux-gnu.tar.gz"
curl -fLO "https://github.com/XaviCodeDev1000/webfang/releases/download/${VERSION}/SHA256SUMS.txt"
# 2. Verify BEFORE extracting
sha256sum -c SHA256SUMS.txt --ignore-missing
# 3. Extract; the archive holds a single file named `webfang`
tar xzf webfang-x86_64-unknown-linux-gnu.tar.gz
# 4. Put it on PATH (system-wide)
sudo install -m 0755 webfang /usr/local/bin/webfang
# …or per-user, no sudo:
# mkdir -p ~/.local/bin && install -m 0755 webfang ~/.local/bin/webfang
# then add to PATH if needed: export PATH="$HOME/.local/bin:$PATH"
# 5. Confirm
webfang --version
Notes:
- The archive holds a bare binary and preserves its executable bit, so
tar xzfalone is usually enough. Useinstall -m 0755(as above) to guarantee the bit regardless of how the archive was extracted — a file-manager double-click can drop it. - The binary is
stripped, but it is not fully static: it linkslibc/libm(glibc), pluslibstdc++andlibgcc_s(BoringSSL is C++). All are standard on glibc distros, but a minimal or musl-based image will fail to load it until they are present. There is no.deb/.rpmand no bundled runtime to install.
macOS (Apple Silicon)
# 1. Download
VERSION=v2.12.0
curl -fLO "https://github.com/XaviCodeDev1000/webfang/releases/download/${VERSION}/webfang-aarch64-apple-darwin.tar.gz"
curl -fLO "https://github.com/XaviCodeDev1000/webfang/releases/download/${VERSION}/SHA256SUMS.txt"
# 2. Verify BEFORE extracting (no sha256sum on stock macOS)
shasum -a 256 webfang-aarch64-apple-darwin.tar.gz
grep 'aarch64-apple-darwin' SHA256SUMS.txt
# 3. Extract; the archive holds a single file named `webfang`
tar xzf webfang-aarch64-apple-darwin.tar.gz
# 4. Put it on PATH (system-wide)
sudo install -m 0755 webfang /usr/local/bin/webfang
# …or per-user:
# mkdir -p ~/.local/bin && install -m 0755 webfang ~/.local/bin/webfang
# 5. Clear the quarantine flag (see note), then confirm
xattr -d com.apple.quarantine webfang 2>/dev/null || true
webfang --version
Gatekeeper. A binary downloaded with
curl/Safariand then extracted carries thecom.apple.quarantineextended attribute, and macOS kills it on first run with "cannot be opened because the developer cannot be verified". It is not corrupt and the checksum will match. Remove the attribute withxattr -d com.apple.quarantine webfang(orxattr -dr com.apple.quarantine <dir>), then run it. Right-click → Open works too, once.
Intel macOS (unsupported)
No artifact exists for Intel macOS, and that is deliberate — it is a decision, not a release gap, so do not go looking for an Intel macOS archive among the assets.
Building from source works on the normal developer path (Rust 1.91 toolchain,
cmake for BoringSSL — see from source). The old
ONNX excuse no longer applies (ADR-0004 paso 6 removed the ai feature and
the local engine), so a source build has no semantic holes: --clean-ai
resolves its endpoint through --embedding-provider, Obsidian vault search
included. What you lose on one machine is only runtime network policy —
except under --offline, where NO embeddings provider is usable: the
guard rejects every one of them, a remote endpoint or a loopback one, and
the run exits with 78. Since 3.0.0 there is no local engine to fall back
to, so --clean-ai and --offline are mutually exclusive. And --offline
gates the SEMANTIC-CLEANING providers, not the crawler's network: it means
"no semantic cleaning", not "no network" — it only works offline when no
embeddings provider is selected at all (no --embedding-provider, and no
provider with the embedding capability in the config), otherwise the run
fails with exit 78 (ADR-0004 §46, corrected 2026-10-21).
Behind a corporate proxy?
WebFang ignores the proxy environment (HTTP_PROXY/HTTPS_PROXY/ALL_PROXY)
by design — for the AI endpoints and for crawling alike (#2500, see
data flow).
If your network only egresses through a proxy, run a local forwarding sidecar,
point the provider's base_url at it, and opt that provider into loopback
endpoints; the sidecar talks to the corporate proxy, WebFang does not.
webfang doctor states this posture on every run.
Windows
# 1. Download (PowerShell 5.1+)
$Version = "v8.0.0"
$Asset = "webfang-x86_64-pc-windows-msvc.zip"
Invoke-WebRequest "https://github.com/XaviCodeDev1000/webfang/releases/download/$Version/$Asset" -OutFile $Asset
Invoke-WebRequest "https://github.com/XaviCodeDev1000/webfang/releases/download/$Version/SHA256SUMS.txt" -OutFile SHA256SUMS.txt
# 2. Verify BEFORE extracting
(Get-FileHash $Asset -Algorithm SHA256).Hash.ToLower()
Select-String -Path SHA256SUMS.txt -Pattern 'pc-windows-msvc'
# 3. Extract; the archive holds a single file named `webfang.exe`
Expand-Archive -Path $Asset -DestinationPath .
# 4. Put it on PATH (per-user, no admin)
$Bin = "$env:USERPROFILE\bin"
New-Item -ItemType Directory -Force -Path $Bin | Out-Null
Move-Item -Force webfang.exe "$Bin\webfang.exe"
[Environment]::SetEnvironmentVariable(
"Path",
[Environment]::GetEnvironmentVariable("Path", "User") + ";$Bin",
"User")
# 5. Open a NEW terminal, then confirm
webfang --version
Notes:
- The binary is
webfang.exeon Windows,webfangeverywhere else. The command iswebfangeither way. %USERPROFILE%\binavoids needing Administrator rights. The PATH edit only reaches new terminals — re-open yours before step 5.- If the system-wide
%ProgramFiles%\webfangroute is preferred, copywebfang.exethere instead.
Visual C++ runtime. The Windows artifact is built with MSVC, which links the dynamic CRT by default. A machine without the Visual C++ Redistributable fails at startup with "The code execution cannot proceed because
VCRUNTIME140.dllwas not found". That is the redistributable being absent, not a broken download — install the Microsoft Visual C++ Redistributable 2015-2022 (x64) from Microsoft and re-run. Verify your checksum first either way, so the two causes stay distinguishable.
MCP servers (from the release artifact)
Two of the three archives per platform hold an MCP server. Both expose the same 36 tools; what differs is how a client reaches them.
| Binary | Transport | Use it when |
|---|---|---|
webfang-mcp | Streamable HTTP, 127.0.0.1:8080/mcp by default | a local service that several clients share over HTTP |
webfang-mcp-stdio | stdin/stdout (JSON-RPC) | a desktop client — Claude Desktop, OpenCode, Cline — spawns the server as a subprocess |
Installing them is the CLI's recipe unchanged: same SHA256SUMS.txt
verification, same extraction, same PATH. Only the asset name changes.
stdio. There is nothing to start by hand — the client owns the process.
Point its command at the installed binary; an absolute path is safest,
because a desktop client usually does not inherit your shell's PATH:
webfang-mcp-stdio
HTTP. Run it as a local service:
webfang-mcp --allow-anonymous # loopback only, no token
webfang-mcp --auth-token "$WEBFANG_MCP_AUTH_TOKEN" # clients send Authorization: Bearer <token>
webfang-mcp --allow-anonymous --bind 127.0.0.1:9090 # move it off the default port
webfang-mcp refuses to start without either --allow-anonymous or
--auth-token. An unset credential used to mean "anyone who can reach the
socket", so the token-less default is a hard error, not a warning.
--allow-anonymous is a loopback affordance only: a routable bind still
requires --auth-token, and the transport is plain HTTP, so terminate TLS in
front of it before exposing it beyond localhost.
Both binaries accept --export-roots <dir> (repeatable or comma-separated).
Absolute output_dir / checkpoint_dir paths in a tool call are rejected
unless they fall under a declared root — relative paths always work, so you
only need this when a client passes absolute ones.
Platform requirements
| Platform | Requirement | Why |
|---|---|---|
| Linux x86_64 / ARM64 | glibc 2.39 or newer; no musl build exists | The artifacts are built on GitHub's ubuntu-latest runner (Ubuntu 24.04, glibc 2.39) and link glibc directly. The floor is inherited from the builder, not a chosen target — so a distro older than Ubuntu 24.04 has no compatible artifact. |
| macOS | Apple Silicon only | Intel macOS has no artifact (decision pre-dating the ONNX removal; the target list was never extended). |
| Windows | Visual C++ Redistributable 2015-2022 (x64) | MSVC's default dynamic CRT. |
| Any | ~17 MB download, no installer, no bundled runtime | stripped binary; needs only standard system libraries (glibc + libstdc++/libgcc_s). |
If a Linux binary dies immediately with
/lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.xx' not found, the distro
is older than the floor above. There is no musl/static artifact to fall back
to; build from source instead.
What is not in the release binary
The release build is one invocation that compiles the CLI and both MCP servers with no capability features enabled:
cargo build --release --locked --features "webfang_cli/mcp" \
--target <target-triple> -p webfang_cli -p webfang_mcp --bins
The mcp feature of that command is inert (an explicit-empty marker): the
webfang CLI carries zero MCP code — the servers are the separate
webfang-mcp / webfang-mcp-stdio binaries in their own archives. (The
former ai feature no longer exists at all, ADR-0004 paso 6: remote-embedding
semantic cleaning ships in the default build —
migration guide.)
The full capability-by-capability matrix — what each of the three binaries can do, and the flags that bring each capability back when building from source — lives in the Release Feature Matrix. The short version, before you file a bug:
| Missing from the release binaries | Consequence |
|---|---|
chromium feature | --js-strategy full is rejected at preflight (it needs a Chromium downloader). The default static strategy and the 3-layer hybrid strategy are unaffected. |
persistence feature | The SQLite checkpoint store is absent. File-based --resume state is unaffected. |
console feature | tokio-console is not available. Tracing via --trace-file is. |
adaptive-selectors feature | Adaptive selector learning is off — its flags do not exist in this build. |
From source (developers)
Not an installation route for end users — this is the contributor path. It downloads and compiles every dependency, BoringSSL included.
Prerequisites: Rust 1.91 (rust-toolchain.toml pins it), a C/C++ compiler,
git, and cmake. cmake is mandatory: the HTTP client wreq → btls
→ btls-sys (formerly boring2/boring-sys2) compiles BoringSSL from C++ source on first build, and
without cmake the failure surfaces deep inside a build script with an error
that says nothing about the real cause.
git clone https://github.com/XaviCodeDev1000/webfang.git
cd webfang
cmake --version # must print a version; install cmake if empty
cargo build --release --locked --features mcp -p webfang_cli
./target/release/webfang --version
-p webfang_clibuilds only the CLI. Omitting it builds every workspace member.- The first build compiles BoringSSL from source and takes minutes, not seconds. This is normal.
- Build-time egress: since the ONNX removal (ADR-0004 paso 6) the only
third-party endpoint a build touches is crates.io. (Pre-paso-6 builds also
pulled ONNX Runtime from
cdn.pyke.io.)
Uninstall
WebFang never removes anything for you. Each binary is one file; the cache and output trees it creates are entirely your responsibility.
1. Remove the binary
| Install location | Command |
|---|---|
Linux, /usr/local/bin | sudo rm /usr/local/bin/webfang |
Linux, ~/.local/bin | rm ~/.local/bin/webfang |
macOS, /usr/local/bin | sudo rm /usr/local/bin/webfang |
Windows, %USERPROFILE%\bin | Remove-Item "$env:USERPROFILE\bin\webfang.exe" |
If you also installed the MCP servers, they are removed the same way: the same
directory, the same commands, with webfang replaced by webfang-mcp and
webfang-mcp-stdio (.exe on Windows). They share no state of their own —
the caches below are the only other thing a run leaves behind.
2. Remove the data
Check where things actually live first. The biggest artifacts live under the webfang cache base; the expected output tree lives next to your runs. (The historical ~372 MB ONNX model cache is a pre-ADR-0004-paso-6 item: the current binary never creates it — see the legacy note below.) Start at the webfang cache base:
# Where the webfang cache base resolves on THIS machine — run the line
# that matches your platform, and note the printed path.
echo "${XDG_CACHE_HOME:-$HOME/.cache}" # Linux
# echo "${XDG_CACHE_HOME:-$HOME/Library/Caches}" # macOS
# echo "$env:LOCALAPPDATA" # Windows (PowerShell)
Every webfang/… row in the table below is relative to that base — the
table does not repeat the webfang/ prefix on Windows, so
%LOCALAPPDATA%\webfang\state is the state directory, not
%LOCALAPPDATA%\webfang\webfang\state.
The config base is the sibling you will also want when cleaning up:
~/.config (Linux), ~/Library/Application Support (macOS), %APPDATA%
(Windows).
⚠️ The model cache was NOT under any of those bases — legacy, pre-paso-6 (ADR-0004). It was the only entry in this table that did not live under the webfang cache base, and § below keeps its exact location on each platform for cleanup after an upgrade from an old release.
XDG_CACHE_HOMEmoves the webfang roots; it never moved the model.Also note
XDG_CACHE_HOME/XDG_CONFIG_HOMEare honored only when they hold an absolute path — a relative or empty value is ignored, and the platform default is used instead. Theechoabove prints the variable even when WebFang ignores it; a path that looks wrong is probably this.
| Path (relative to the base above) | Created by | Size | Auto-cleaned? |
|---|---|---|---|
webfang/state/<domain>.json | --resume | KBs, grows with processed-URL count | No — relocatable via --state-dir / WEBFANG_STATE_DIR |
webfang/state/<domain>.json.lock | every RecordStore write | 0 B | No — permanent by design. See the note below. |
webfang/state/<domain>.json.bak | migrating a stale state version | same as state | No — an existing backup is kept as-is |
~/.webfang/crawl.db — absolute, under $HOME, not the cache base | crawl/scrape runs that persist to SQLite (only with the persistence feature — absent from the release binary): extraction fingerprints, resources, chunks, notes | grows with fingerprinted pages | Rows older than N days are pruned (plus VACUUM) when a run passes --retention-days N; otherwise delete the file while no run is active. Relocatable via --db-path / WEBFANG_DB_PATH |
huggingface/hub/ — legacy only, see below | pre-paso-6 releases only (ADR-0004) — current builds never write it | 0 for current builds; up to 372 MB / ~1.2 GB for old ones | No — relocatable via HF_HOME, a HuggingFace variable, not WEBFANG_* |
<output_dir>/ — Markdown, export.jsonl, rag_dataset/, _inbox/ | every run | unbounded — your data | Opt-in: files older than N days are pruned when a run passes --retention-days N. Relocatable via -o / WEBFANG_OUTPUT; defaults to output/ in the current directory |
webfang/config.toml (under the config base: ~/.config, ~/Library/Application Support, %APPDATA%) | you, by editing it | KBs | No — relocatable via WEBFANG_CONFIG or XDG_CONFIG_HOME |
The
.locksentinel is supposed to survive.<domain>.json.lockis created on demand and intentionally never unlinked; the real mutual exclusion is an advisoryflockthe kernel releases when the process exits, including after akill -9. Its presence after an uninstall is correct behaviour, not a failed removal. See troubleshooting.md.
To remove the webfang-owned cache roots and leave everything else alone:
# Linux / macOS: cache + state, but NOT the model cache
rm -rf "${XDG_CACHE_HOME:-$HOME/.cache}/webfang" # Linux
rm -rf "${XDG_CACHE_HOME:-$HOME/Library/Caches}/webfang" # macOS
# Windows: cache + state, but NOT the model cache
Remove-Item -Recurse -Force "$env:LOCALAPPDATA\webfang"
The model cache is the exception (legacy)
Since the ONNX removal (ADR-0004 paso 6) the current binary never creates this cache: there is no local model to fetch, and no hf-hub dependency remains. If you are upgrading from a pre-paso-6 release, the directory a previous binary filled still exists, is inert, and is entirely yours to delete; the details:
hf_hub (used by pre-paso-6 releases) does not use WebFang's platform
helper. Its default is
dirs::home_dir()/.cache/huggingface/hub — literally ~/.cache/… on
every platform, including macOS and Windows, where the webfang cache base
is ~/Library/Caches and %LOCALAPPDATA% respectively. It is only moved by
HF_HOME, and no WEBFANG_* or XDG_* variable relocates it.
| Platform | Default model cache (no HF_HOME) | With HF_HOME=/some/dir |
|---|---|---|
| Linux | ~/.cache/huggingface/hub | /some/dir/hub |
| macOS | ~/.cache/huggingface/hub — not ~/Library/Caches/… | /some/dir/hub |
| Windows | %USERPROFILE%\.cache\huggingface\hub — not %LOCALAPPDATA%\… | /some/dir/hub |
To confirm before deleting, and to remove it:
# Unix
echo "${HF_HOME:-$HOME/.cache/huggingface}"
rm -rf "${HF_HOME:-$HOME/.cache/huggingface}"
# Windows
$HF = if ($env:HF_HOME) { $env:HF_HOME } else { "$env:USERPROFILE\.cache\huggingface" }
$HF
Remove-Item -Recurse -Force $HF
If HF_HOME is set, the default location is empty and deleting it recovers
nothing — which is why the confirmation line comes first, and why the two are
never collapsed into one rm.
Never delete a webfang directory without reading the path you built
above — a relative or empty base would resolve to somewhere unintended, and
<output_dir> is user data with no backup.
Checklist
A fresh install is done when all of these are true:
- I picked the asset whose target triple matches my machine (Apple Silicon, not "Intel").
-
The SHA-256 of my archive matches its line in
SHA256SUMS.txt. -
The binary is executable (
chmod +x/install -m 0755) and onPATH. -
webfang --versionprints a version. -
On macOS,
com.apple.quarantineis cleared if the first run was killed. -
I know where my
output/directory is, and that uninstalling will not remove it.
Next step
- Troubleshooting — install-time failures, slow crawls, WAF blocks, SSRF refusals.
- CLI Reference — the complete flag reference.
- Debugging & Observability — tracing, correlation IDs, and
the
jqquery cookbook.
Data Flow & Privacy
This page is the data-flow disclosure: what leaves your machine when you
run webfang, to which destinations, under what conditions, what is stored
locally, and how to turn every flow off. Everything stated here is enforced by
scripts/check_data_flow_doc.sh against the source tree, so it cannot silently
drift from the code.
Quick path: run with zero AI egress
- Do not pass
--clean-aior--extract-with-llm, and do not use the MCP AI tools — no AI request is ever made without an explicit flag or tool call. - Do not configure an AI provider in the providers config — without one, an AI request fails loudly at startup with exit 78, it never falls back to a remote default.
- Run with
--offlineon cleaning paths — every embeddings provider (remote or loopback) is rejected, exit 78, before any endpoint is inspected. - Scraped-site traffic is inherent to crawling: fetching the pages you asked for is the product. Restrict it with the crawl scope flags; every destination is checked by the SSRF guard before a socket exists.
What leaves the machine
| Destination | When | What is sent | How to turn it off |
|---|---|---|---|
| The site being scraped | Every crawl/scrape | HTTP GET requests for the pages (and, with asset download, for images/documents) — browser-like fetches via the TLS-fingerprinted client, or by the real Chrome browser itself under --js-strategy full | Omit the URL(s); scope with crawl flags |
The operator-configured LLM endpoint (POST {base_url}/chat/completions) | --extract-with-llm | Extracted text chunks plus the extraction schema | Omit the flag; or configure no completion provider (exit 78 at startup) |
The operator-configured embeddings endpoint (POST {base_url}/embeddings) | --clean-ai (and the MCP semantic_cleaner tool) | Extracted text chunks (bounded by --max-chars) | --offline; or configure no embedding provider |
There is no other network destination: no telemetry, no metrics, no update check, no crash reporting. Every HTTP fetch in the product goes through the SSRF-guarded downloader or through the two OpenAI-compatible AI adapters above. (The historical Hugging Face / local-ONNX paths were removed — see the ONNX migration page.)
Proxy posture: the proxy environment is ignored by design
WebFang dials directly. The proxy environment variables
(HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXY) are ignored for every
fetch path — crawling, assets, and both AI adapters — on every platform.
This is deliberate, not an omission. The HTTP client honours those variables by default, and under a proxy the guard's validating DNS resolver resolves the proxy's host instead of the destination. Two consequences made silent proxy support unacceptable:
- A corporate proxy on a private range (RFC1918) — the common deployment — is rejected by the SSRF guard before a socket exists, with the operator seeing an SSRF diagnosis instead of a working proxy.
- When the proxy itself is allowed, the destination's IP checks are delegated to the proxy: the client never resolves the destination, so the "forbidden ranges are rejected before the socket exists" guarantee stops covering it.
Where this is pinned: DefaultSsrfGuard::secure_client* applies no_proxy()
at the single client-construction choke point, and
tests/proxy_posture_e2e_test.rs proves that a request with HTTP_PROXY set
reaches the target with zero requests observable at the proxy (both through
the downloader and the webfang binary). webfang doctor states the posture
on every run.
If you cannot dial out directly (mandatory corporate proxy): run a local
forwarding sidecar, point the provider's base_url at it, and opt into
loopback endpoints for that provider — the sidecar, not WebFang, talks to the
corporate proxy. WebFang never reads the proxy environment itself.
The payload contract: extracted text, never raw HTML
The remote model never receives your raw page source. Before any text is sent to an AI endpoint it passes through the text-extraction stage:
- The semantic cleaner extracts readable text from the page HTML — first a
legibility parse, falling back to a plain tag-strip — and then chunks that
text under the
--max-charsbudget. This is the invariant stated in the orchestrator: clean (raw HTML never reaches the LLM). Implementation:application/llm_extraction.rs, andinfrastructure/semantic_cleaner.rs::extract_text_content. - The structured-extraction path (
--extract-with-llm) composes the same cleaner with the chunker, so the completion payload is chunked extracted text plus your schema — never the page's HTML. - Authentication: requests carry an
Authorization: Bearerheader only if the provider config supplies a credential; a provider withauth: { source: "none" }sends an anonymous request. Credentials resolve locally at startup; they are never logged.
Endpoint reachability is enforced by the same fetch-guard chain used for
crawling (entry validation → timeouts → redirect policy → SSRF at the socket
dial → retry classification → capped body read). Loopback embedding endpoints
are rejected unless explicitly opted in, and --offline rejects every
provider before the endpoint is even inspected.
Log and trace content policy
The observability stack (structured logs on stderr, the --trace-file JSONL)
carries operational metadata only:
- What you will find: timestamps, levels, span and target names, URLs, depths, HTTP statuses, durations, pacing waits, error messages with their stage, and correlation identities. The debugging guide documents every span and field.
- What never appears: page bodies, HTML, extracted Markdown, chunk text,
or provider credentials — no
tracingfield in the product crates carries page content. The JSONL is identifiers and outcomes, not payloads. - MCP responses: every tool result that carries third-party text is
neutralized (ANSI/control characters stripped), byte-capped with a visible
truncation marker, and wrapped in a nonce-delimited UNTRUSTED provenance
envelope. See
docs/security/prompt-injection-policy.mdfor the consuming rules. - Trace files are local, ordinary files at the path you choose with
--trace-file. They are never uploaded anywhere; delete them whenever you like.
Local retention
| Path (Linux / XDG) | What lives there | Created by | Removal |
|---|---|---|---|
~/.webfang/crawl.db | SQLite database (crawl results / RAG export) | Any run that persists results; override with WEBFANG_DB_PATH / --db-path | Delete the file |
~/.cache/webfang/state/ | Export state JSON plus the JSON-per-domain record store (one versioned envelope per site) | Resume/checkpoint runs; override with --state-dir | Delete the directory |
<run persistence root>/.webfang-batch-capture.jsonl | The batch spool: every page body the batch fetched, verbatim | Batch runs, until the batch finishes and it is discarded | Discarded at the end of a healthy batch; a crashed run leaves it, and the next batch run sweeps stale spools — or delete it by hand |
~/.config/webfang/ | config.toml and the encrypted credential store credentials.age | Configuration and webfang connect | Delete the directory (also part of uninstall) |
output/ (under your working directory) | Exported Markdown/JSON and downloaded assets | Every run that exports or downloads | Delete at will |
Your --trace-file path | The JSONL trace (metadata only, see above) | Runs with tracing enabled | Delete at will |
Windows/macOS use the platform equivalents documented in installation.md; the paths above follow the standard XDG conventions the platform-paths helper enforces.
SQLite state and the WAL sidecar files (-wal/-shm) have their own
recovery procedure: see the State Recovery Runbook
before deleting a database you still need.
Review checklist
- I know every destination my run can talk to (the table above) and none of them is hard-coded by webfang.
- I can run a full crawl with zero AI egress (Quick path above).
- I understand the AI payload is extracted text chunks, never raw HTML.
- I know where every retained file lives and how to remove each one.
Release Feature Matrix
Every release publishes three binaries — the webfang CLI and the two MCP
servers (webfang-mcp, webfang-mcp-stdio) — and every binary is built from
the same feature set, always. This page answers "what can the binary I
just downloaded do?" capability by capability, without you having to read the
release workflow.
The capabilities below are compile-time Cargo features, not runtime
configuration: if a release binary was built without a feature, the capability
is not in the file, and no flag combination brings it back. The bottom of this
page carries a machine-readable manifest of the feature set, and CI
drift-checks it against the [features] table of
crates/webfang_cli/Cargo.toml
(scripts/check_release_feature_matrix.sh, a required step of the
repo-guards job) — so this page cannot silently rot when a feature is added
or removed.
What is NOT covered here: which platforms get artifacts (see Installation) and how to verify a download. This page is only about compiled-in capability.
The three artifacts
| Binary | Crate | What it is |
|---|---|---|
webfang | webfang_cli | the scraper CLI |
webfang-mcp | webfang_mcp | MCP server, Streamable HTTP on 127.0.0.1:8080/mcp |
webfang-mcp-stdio | webfang_mcp | MCP server over stdin/stdout (JSON-RPC), for desktop clients |
The exact release build is one invocation
(.github/workflows/release.yml, step "Build release binary"):
cargo build --release --locked \
--features "webfang_cli/mcp" \
--target <target-triple> \
-p webfang_cli -p webfang_mcp --bins
No capability feature is enabled: webfang_cli's default = [], and
chromium, persistence, console and adaptive-selectors are capability
features that no release artifact carries. webfang_cli/mcp is an
explicit-empty marker (mcp = []): the key must exist because the CLI
test harness and several scripts build with --features mcp, but it selects
nothing — it puts zero MCP code into the webfang binary. The MCP
surface lives in the webfang_mcp crate, which is built as its own two
binaries and shipped in its own archives.
Capability matrix
| Capability | webfang CLI (release) | MCP servers (release) | Build from source |
|---|---|---|---|
| MCP (36 tools) | Not embedded. The CLI exposes no MCP flag. The servers are the two separate binaries — see MCP servers. | Included — this is what those binaries are. | cargo build -p webfang_mcp --bins |
--js-strategy full (Chromium rendering) | Rejected at preflight with an honest error — the Chromium downloader is not compiled in. The default static strategy and the 3-layer hybrid strategy are unaffected. | The get_accessibility_snapshot tool is registered but answers with a feature-gated error instead of driving a browser. | --features chromium (needs a Chromium/Playwright runtime at run time) |
| SQLite persistence | Absent. The crawl.db checkpoint store does not exist; file-based --resume state is unaffected. | Absent. | --features persistence (enables the SQLite store; --elastic / --db-path ingestion paths use it) |
| Adaptive selector learning | Absent — the flags do not exist in this build. | Absent. | --features adaptive-selectors |
| tokio-console | Absent. Tracing via --trace-file is unaffected. | Absent. | --features console |
dev-tracing | Never in a release — a development aid, not a capability. | Never. | --features dev-tracing |
Historical note: the former ai feature (local ONNX semantic cleaning) was
removed in ADR-0004 paso 6, not feature-gated — semantic cleaning is
runtime-configured through --embedding-provider in every build, and
--offline rejects every provider. See the
migration guide.
The machine-readable contract
This manifest is the guard's input, not reader-facing prose: it is parsed by
scripts/check_release_feature_matrix.sh and must mirror the [features]
table of crates/webfang_cli/Cargo.toml exactly — every feature key
present once, default absent, and each kind matching how the feature is
declared (capability = forwards a webfang_core feature that gates code or
dependencies; marker = an explicit-empty feature that selects nothing).
Adding or renaming a feature in the manifest without touching Cargo.toml
— or the other way around — fails the repo-guards job.
chromium | capability | --js-strategy full rejected at preflight (#685)
mcp | marker | explicit-empty marker (#1815 T4); MCP ships as the webfang_mcp binaries
persistence | capability | SQLite store absent from release artifacts
console | capability | no tokio-console in release artifacts
adaptive-selectors | capability | flags absent from release artifacts
dev-tracing | capability | development tracing, never in a release
Release Process
Two tools cut a release, and each owns exactly one half:
| tool | owns |
|---|---|
release-plz | the version bump, CHANGELOG.md, the Release PR, the v* tag |
release.yml | the binaries, the archive normalization and the GitHub Release page, triggered by the tag |
The configuration lives in release-plz.toml
and the workflow in
.github/workflows/release-plz.yml.
This page explains the three decisions that are not written in either file: which commits
may cut a version, how to check the filter without touching the remote, and what the support
metadata means.
Which commits may cut a version
release-plz bumps on every commit by default, so a test:, ci:, docs: or plain
refactor: commit publishes a tag, a build matrix and a GitHub Release for a change no user
can observe. The workspace sets release_commits to fix that:
the user-facing types always release; any type releases when it is breaking.
The exact pattern is in release-plz.toml — it is the source of truth and it is pinned by
scripts/test_release_commits_regex.sh, which reads it from the file and runs a table
against it (a duplicated regex keeps passing while the configuration is broken).
| commit | releases? | why |
|---|---|---|
feat:, fix(scope):, perf:, revert: | yes | user-visible |
chore(deps): bump … | yes | the binary changes and the user cannot rebuild it |
any type with ! — refactor(api)!:, docs!: | yes | a breaking change is user-visible by definition |
refactor:, refactor(scope): | no | behaviour-preserving; it ships with the next real release |
test:, ci:, docs:, style:, chore: | no | nothing the user can observe |
Two consequences worth knowing:
- A breaking change is never delayed by a non-breaking type. The
!marker is the canonical one (with or without a scope), sorefactor(exporter)!:releases even thoughrefactor:does not. This repo has shipped breaking refactors that way. - Filtered commits are not lost. They still appear in the
CHANGELOG.mdof the next release that does fire, in their own section. The filter decides when a version is cut, not what the notes say.
Checking the filter without touching the remote
There is no --dry-run for the Release-PR path, but there is a safe pre-check: run the PR
command with a token that cannot possibly work.
release-plz release-pr --git-token invalid
It resolves the versions, logs no commit matches the 'release_commits' regex per package
when the window has nothing releasable, and exits without opening a PR or modifying the
working tree. A window that does match fails later, at the first API call — which is
exactly the signal you want: it means a PR would have been proposed.
Do not use release-plz update as the pre-check. It mutates the checkout (it rewrites
Cargo.toml and the changelog), and on a workspace where every package is filtered it
writes the new version anyway: it puts the bumped workspace version into Cargo.toml
without updating Cargo.lock, leaving a tree that cannot build. That is a behaviour of the
update command, not of the PR path — but it makes it the wrong tool for a quick look, and a
dangerous one to run in a working checkout. Measured on release-plz 0.3.168 and 0.3.169 (the
version the GitHub Action installs).
A probe commit has to touch a file of the crate. release-plz filters by the files of each
package before it evaluates the commit message, so an empty commit (git commit --allow-empty)
reports no commit matches no matter what the subject says, and a commit that only touches
docs/ or scripts/ does the same for all three packages. To probe the positive path,
append a line to a file inside the crate you expect to release (for example
crates/webfang_core/src/lib.rs).
Also note that the GitHub Action installs its own release-plz version; a local CLI may be a different one. The pre-check is a smoke test, not the verdict — the verdict is the next real run of the workflow.
A commit that only moves Cargo.lock does not cut a version
With release_commits active, a commit that changes only Cargo.lock reports
no commit matches the 'release_commits' regex and does not release — even when its subject
matches (chore(deps):, or fix:). Measured on release-plz 0.3.168 and 0.3.169, with a clean
window (a single lock-only commit on top of v8.0.0): without release_commits the same
commit produces next version is 7.7.2; with the filter, nothing. release-plz decides by the
files of each package, and Cargo.lock lives at the workspace root, outside every package.
The hole only bites when no other commit in the window releases: a window with a feat:
or fix: releases anyway and carries the lock change with it (that is how #2433 and #2361
shipped). It is not a hypothetical shape, though: with the filter, test:, ci: and docs:
commits accumulate without cutting a version, so a lock-only bump can easily be the only thing
in the window.
The convention that closes it: the commit has to touch a packaged file of the affected
crate, so the change is attributed to that package. In this workspace that rules out the
root Cargo.toml: the dependency requirements live in [workspace.dependencies], and the
root file belongs to no package — raising libc there still reports no commit matches
(measured on 0.3.168 and 0.3.169). What works is a file inside the crate:
- if the crate declares the dependency with its own version requirement, raise it there (a real change: it records the new minimum);
- when the dependency is transitive and the crate only says
libc = { workspace = true }, add a comment in thatCargo.tomlnaming the advisory, e.g.# <crate>: <RUSTSEC-...> patched in the lockfile.— measured on release-plz 0.3.168 and 0.3.169: all three packages get the patch version.
The comment is a marker, not a fix: prefer the requirement change whenever the crate actually owns one.
Why a Release PR cannot be generated against an untagged main
release-plz derives the next version from the commits between the newest v* tag and
main, and in git_only mode that tag list is the local one. A main that is ahead of
its own tag — the version bump merged, the tag seconds away — is a state it cannot read, and
it proposes the version that is already there.
The PR job therefore syncs its checkout to origin/main and waits, bounded, for the tag of
the declared version before running release-plz (scripts/await_release_tag.sh). When the
tag arrives, there is nothing to release; when it does not, the step fails loudly instead of
opening a phantom Release PR. A repository with no v* tag at all is still the initial
release and is not awaited.
Reverts must be conventional
The filter reads the commit subject, and git revert writes Revert "…" — capital R, not
a conventional type — which matches nothing. A revert that must ship has to be recorded as
revert: … (lowercase), either by amending the subject or by writing the revert commit by
hand. Otherwise the change it undoes stays published until the next release that fires for
another reason.
What released_at means in .github/support.json
.github/support.json describes the support lines, and its two date fields answer different
questions:
latest— a fact about what this repository published. It is written byscripts/refresh_support_latest.sh, the single writer, and it moves on every patch.released_at— the date the minor line went stable. It is written once, during the rotation of a minor (scripts/rotate-stable.sh --pending-publication), and a patch does not move it.scripts/rotate-stable.shreads it, so it is a policy field, not a label: it is the clock the support window is measured from.
So seeing latest at a patch while released_at still holds the minor's date is the correct
state, not drift.
Approving the release PR's linked issue
Validate PR metadata checks that every linked issue carries status:approved, and it reads
the label from the issue through the API. The pull_request workflow that runs the check
listens to the PR's own events (opened, synchronize, labeled, …), so approving the issue
is not a PR event: the check does not re-run on its own and the run keeps showing the old
verdict.
If the metadata check is red after the issue was approved, re-run the failed job —
gh run rerun <run-id> --failed, or "Re-run jobs" in the UI. Both attempts observed on
#2788 and #2791 were manual re-runs (same triggering_actor, event pull_request), not label
events.
set-version is not a release flow
release-plz set-version exists to correct a version that was not set as expected (for
example a commit that was not prefixed correctly). It is a correction tool, not a way to cut a
release: it does not open a PR, it does not write the changelog entry, and it has not been
exercised in this repository. If a release has to be forced for a window that the filter does
not match, treat that as an incident to investigate — not as a routine path.
Debugging & Trace Analysis
WebFang ships built-in, always-available observability. No external
collector, no feature flags, no infrastructure: run with --trace-file and
post-process the JSONL with jq.
Mandate: every new feature or hot path must be observable. See the "Observability (MANDATORY)" section of
../../AGENTS.md.
The stack
| Layer | What it does | Always on? |
|---|---|---|
| FileTraceLayer | Writes every tracing span/event to a JSONL file (--trace-file) | ✅ Yes |
| Correlation IDs | Native CorrelationId (UUID v7 trace_id + span_id); one trace_id per operation, unique span_id per unit of work | ✅ Yes |
| Structured logging | tracing-subscriber to stderr (-v/-vv/-vvv, --log-format json) | ✅ Yes |
| Tokio Console | Async task/resource inspection for concurrency bugs | --features console |
There is no OpenTelemetry (removed in #356). If you need a metric, emit a structured tracing event and query it from the JSONL.
Generating a trace
# Full trace + verbose logging
webfang --url https://example.com --trace-file debug.jsonl -vvv
# Batch / crawl
webfang --url https://example.com --max-pages 100 --trace-file crawl.jsonl -v
Each line of debug.jsonl is a JSON object:
{
"timestamp": "2026-01-29T10:00:00.123Z",
"level": "INFO",
"target": "webfang_core::application::crawler::engine",
"span": "crawl_page",
"span_id": "0000000000000042",
"parent_id": "0000000000000001",
"trace_id": "0000000000000001",
"fields": {
"url": "https://example.com/page1",
"depth": 1,
"correlation_id": "00-01949e0e8b8e70008000000000000001-0000000000000042-01"
}
}
Top-level trace_id is the root span Id (16-hex), one per run, EPHEMERAL
to the process/run (identity-within-run, not a durable global identity). Do
not persist or join on it across runs. Durable run correlation is the
CorrelationId UUID in span_fields.trace_id /
span_fields.correlation_id (W3C traceparent).
When a span closes, a second record type is emitted carrying a top-level
span_duration_ms (wall-clock milliseconds) — this is what the "Slowest spans"
query below reads:
{
"timestamp": "2026-01-29T10:00:00.456Z",
"record": "span_close",
"level": "INFO",
"target": "webfang_core::application::crawler::engine",
"span": "crawl_page",
"span_id": "0000000000000042",
"parent_id": "0000000000000001",
"trace_id": "0000000000000001",
"span_duration_ms": 333,
"span_fields": {
"url": "https://example.com/page1"
}
}
Query cookbook
A ready-made script lives at scripts/analyze-trace.sh. The most useful
queries:
Reconstruct one run by top-level trace_id
ROOT=0000000000000001
jq -c "select(.trace_id == \"$ROOT\")" debug.jsonl
$ROOT is the 16-hex root span Id (top-level trace_id, EPHEMERAL to the
run). It returns every page plus errors for that run. The same query is
scripts/analyze-trace.sh debug.jsonl trace $ROOT.
All errors, with full context
jq -c 'select(.level == "ERROR") | {target, url: .fields.url, stage: .fields.stage, error: .fields.error, msg: .fields.message}' debug.jsonl
The cause field (connection failures)
The error events carry a cause field: the machine-readable class of a connection
failure, as a stable wire value. It is not a message fragment — the values are a contract:
cause | meaning | observed end to end |
|---|---|---|
dns_resolution | the name never came back: the resolution itself failed or hung | yes |
connect_refused | the peer refused the connection (RST) | yes |
connect_timeout | the name resolved and the TCP connect did not complete in time | yes |
read_timeout | the connection was established and the response never arrived | not yet |
connection_closed | the peer closed the connection before answering (EOF/reset) | not yet |
tls_handshake | the TLS handshake failed | not yet |
unknown | the classifier ran and could not decide — see below | yes |
The three values marked not yet are DECLARED, not measured: the transport does not expose
the evidence they need (an EOF and a stalled response reach the classifier with no readable
kind), so they arrive as unknown today. Their names and their meaning are already fixed —
a value never changes meaning — but they are not expected in a trace until the transport
surfaces the evidence behind them.
The field is the contract; the message is prose. Every classified failure also names its
cause in the message, in Spanish, before the transport's own words. A consumer must still read
the cause field: the message is written for a human and its wording may change, while the
wire value will not.
The field is omitted when the failure carries no classified cause. That absence means "this
failure was not classified" — it does NOT mean "this was not a network failure": a transport
failure whose evidence the wiring cannot read is reported without the field. unknown is the
classified version of that same fact, and it is the value to look for when a network failure
carries no better answer.
unknown is the honest answer, and it is not a shrug. It covers two cases: a chain the
classifier could not read by type, and a name resolution that was still in flight when the
request failed but could not be attributed to that request — typically because another
request for the same host was running at the same time. The rule is deliberately conservative:
a failure is only called dns_resolution when the evidence points at THAT request's own
resolution. With one request in flight per host the attribution is reliable; with several, it
is not, and the value is unknown rather than a wrong cause.
# the failures by cause
jq -r 'select(.fields.cause) | .fields.cause' trace.jsonl | sort | uniq -c | sort -rn
# every failure with its cause, in order
jq -c 'select(.fields.cause) | {url: .fields.url, cause: .fields.cause, error: .fields.error}' trace.jsonl
Slowest spans (where the time goes)
jq -r 'select(.span_duration_ms != null) | [.span_duration_ms, .span] | @tsv' debug.jsonl | sort -rn | head -20
Time distribution per pipeline stage
jq -r 'select(.span == "pipeline_stage") | .fields.stage' debug.jsonl | sort | uniq -c | sort -rn
Crawl progress over time
jq -c 'select(.fields.message? == "crawl progress") | {pages: .fields.pages_crawled, pct: .fields.progress_pct, eta_s: .fields.eta_secs}' debug.jsonl
Final crawl summary
jq -c 'select(.fields.message? == "crawl completed")' debug.jsonl
Count operations by span type
jq -r 'select(.record != "span_close") | .span // "event"' debug.jsonl | sort | uniq -c | sort -rn
span_close records share the same .span name, so they must be excluded
when counting events (otherwise every span is double-counted).
URLs that failed
jq -r 'select(.level == "ERROR") | .fields.url // empty' debug.jsonl | sort -u
Was this run slow, or was it PACED?
Pacing and admission waits emit rate limit wait / semaphore acquire
events at DEBUG level. -vvv is not required for these: the
--trace-file layer always runs at TRACE, so they land in the JSONL
whenever a trace file was requested. They exist because a wait used to be
invisible — a throttled crawl looked exactly like a slow network.
# Total time spent waiting, per pacing scope.
scripts/analyze-trace.sh debug.jsonl pacing
| Scope | Meaning |
|---|---|
crawl_discovery | Per-page pacing inside the crawl engine |
cli_scrape | Per-URL pacing in the CLI scrape phase |
batch_scrape | Per-URL pacing in the batch scrape path |
http_client | Per-session quota in the HTTP client |
A scope with total_waited_ms near zero means pacing was free and the
run's time went to the network. A large cancelled count means shutdown
was shedding queued work, not that the fetches were slow.
Which targets absorbed the wait (each event names the URL whose permit it
was waiting for, and the page's correlation_id):
scripts/analyze-trace.sh debug.jsonl pacing-hot 10
Reconstruct one page's pacing from its own identity:
CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.fields.correlation_id? == \"$CID\" and .message? == \"rate limit wait\")" debug.jsonl
The same wait is also folded into the page's own span, so a page record answers "how much of this page was pacing" without a join:
jq -r 'select(.span == "crawl_page" and .record == "span_close")
| [.span_fields.outcome, .span_fields.rate_limit_wait_ms, .span_duration_ms] | @tsv' \
debug.jsonl | sort -k2 -rn | head
Admission and dispatch queues
scripts/analyze-trace.sh debug.jsonl admission
One row per operation, covering the downloader resource governor
(browser.resource_governor.acquire), the AI inference pool permit
(ai.inference_pool.acquire) and the AI pool dispatch queue
(ai.inference_pool.dispatch). max_in_flight is the pool occupancy the
waiter observed. If max_waited_ms is high and max_in_flight equals the
worker count, the pool was saturated — raise the worker budget or lower the
concurrency feeding it, rather than raising timeouts.
Spans you will see
| Span | Emitted by | Key fields |
|---|---|---|
crawl_site / crawl_site_with_options | crawler::engine | correlation_id, trace_id, seed_url, max_depth, max_pages |
crawl_page | crawler::engine::run_crawl_task | correlation_id, trace_id, url, depth |
execute | pipeline::PipelineExecutor | url, stages |
pipeline_stage | pipeline::PipelineExecutor | stage, url |
export_batch | JsonlExporter / VectorExporter / FileExporter | exporter, documents |
scrape_single_url → scrape_single | crawler::discovery::scrape_single_url | url (outer), correlation_id, trace_id, url (inner, #501) |
scrape_with_config | scraper_service | url, correlation_id, trace_id, has_downloads |
scrape_multiple_with_limit | scraper_service | urls, concurrency |
Identity follows the root-child contract: the OPERATION owns one root
CorrelationId, and every unit of work derives .child() from it — same
trace_id, fresh span_id. In a CLI run the orchestrator mints the root
and announces it with a run identity event (correlation_id, trace_id
in .fields); scrape_multiple_with_limit does the same with a
scrape_multiple identity event. So in a multi-page scrape:
- Top-level
trace_idis the single logical run id: the root spanId(16-hex), EPHEMERAL to the run. Reconstruct the whole run offline withselect(.trace_id == $ROOT)— pages plus errors, no orphans. span_fields.trace_idis the shared run-root UUID (CorrelationId, durable across systems) across all page spans.span_fields.correlation_id(full W3C traceparent) is unique per page; its trace part is the run-root UUID without dashes.
Identity is declared at span creation time: FileTraceLayer snapshots
the span's declared fields in on_new_span, and #[instrument] can only
see function arguments. Fields that are only known at the END of an
operation (an outcome, an HTTP status) are therefore declared
tracing::field::Empty and filled in with tracing::Span::record, which
on_record merges into the same snapshot — they reach the JSONL both in
the span_fields of later events and in the span_close record. ScrapedContent and the RAG exports carry the same identity, so
an exported document's correlation_id matches its page's
span_fields.correlation_id:
# Reconstruct an entire run by the single top-level trace_id (root span Id)
ROOT=0000000000000001
jq -c "select(.trace_id == \"$ROOT\")" debug.jsonl
# Same run by the durable run-root UUID (CorrelationId in span_fields)
CUUID=01949e0e-8b8e-7000-8000-000000000001
jq -c "select(.span_fields.trace_id == \"$CUUID\")" debug.jsonl
# The run-root identity (the `run identity` event carries it in .fields)
jq -c 'select(.message? == "run identity") | .fields' debug.jsonl
# Every page identity present in the trace
jq -r '.span_fields.correlation_id // empty' debug.jsonl | sort -u
# Reconstruct one page's scrape by its correlation_id
CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.span_fields.correlation_id? == \"$CID\")" debug.jsonl
Events (not spans): run identity, scrape_multiple identity,
crawl progress, crawl completed, and any log_scrape_error(...) error
carrying error, url, stage, trace_id.
Concurrency debugging (Tokio Console)
For deadlocks, starved tasks, or async resource leaks, use the Tokio Console:
RUSTFLAGS="--cfg tokio_unstable" cargo run --features console -- --url https://example.com
This opens an interactive TUI showing live tasks, their states, and poll times.
Troubleshooting
See troubleshooting.md for common problems (slow crawls, silent page failures, WAF blocks, async deadlocks, poor content) and how to diagnose each with the trace queries above.
For contributors
When you add a hot path or operation, follow the observability mandate in
AGENTS.md:
#[instrument(skip(...), fields(url = %url, ...))]on the function.- Propagate the operation's
CorrelationId; derive.child()per unit of work. - Use
log_scrape_error(...)on error paths (never a barewarn!for an operational error). - Use
.instrument(span)on async futures — never holdspan.enter()across.await. - Verify with:
webfang ... --trace-file debug.jsonl -vvvand the queries above.
Correlating a wait with the page that paid it
A pacing wait happens BEFORE the page span is created (the permit is taken before any socket opens), so it is emitted as its own event rather than inside the page span. It is still attributable, twice over:
# 1. By identity — the wait carries the same correlation_id the page span does.
CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.fields.correlation_id? == \"$CID\")" debug.jsonl
# 2. By span field — crawl_page records the wait it absorbed.
jq -c 'select(.span == "crawl_page" and .record == "span_close")
| {url: .span_fields.url, outcome: .span_fields.outcome,
paced_ms: .span_fields.rate_limit_wait_ms,
total_ms: .span_duration_ms}' debug.jsonl
A page where paced_ms is a large share of total_ms was queued, not
slow. That distinction is the whole reason the wait emits at all.
Testing Guide
End-to-end (E2E) tests live as integration test crates under tests/ and invoke the
real webfang binary via assert_cmd. Mock HTTP servers
(wiremock) stand in for target sites and tempfile::TempDir
captures scrape output.
Test crates
| Crate | File | Gate | What it covers |
|---|---|---|---|
behavioral | tests/behavioral/main.rs | none (always built) | Single-page scrape, CLI help, unreachable host, slow server, obsidian frontmatter, CSS-selector and full-page extraction (tests/behavioral/cli/) |
cli_binary | tests/cli_binary_test.rs | none (always built) | --version, --help, network-error exit codes |
No E2E crate is feature-gated. An earlier revision of this table listed a
cli_behavioral crate gated on images and documents; that crate does not
exist and that #![cfg(all(feature = "images", feature = "documents"))] gate
never existed anywhere in the source tree — its coverage (obsidian
tag/metadata/wiki-link conversion) lives ungated inside behavioral, at
tests/behavioral/cli/obsidian_test.rs.
The images and documents marker features themselves were deleted in
#1813: they selected no capability, and turning them off silently shrank a MIME
table instead of shrinking anything else. Do not reintroduce them as empty
markers — add a feature when it gates something real.
Running tests
# all E2E crates
cargo nextest run --test behavioral --test cli_binary
# a single crate
cargo nextest run --test behavioral
# a single test (libtest, prints the full snapshot diff on mismatch)
cargo test --test behavioral selector_h3_extracts_only_h3
Ignored tests (e.g. optional live-site checks) are excluded by default; run them with
cargo nextest run --test behavioral --run-ignored ignored-only.
Mocking a remote embeddings provider
--clean-ai needs a configured remote embedding provider: ADR-0004 retired the
local ONNX stack, and --export-format vector needs --clean-ai (preflight #796).
So a test that exercises the vector export has to provide one.
crates/webfang_core/tests/common/embeddings.rs does it hermetically — the provider
IS the test's own wiremock server, the credential is anonymous, and nothing
leaves the machine:
let t = BehavioralTest::new().await;
crate::common::embeddings::mount_embeddings(&t.server).await;
let provider_config = crate::common::embeddings::provider_config(&t.server);
t.scraper_cmd()
.env("WEBFANG_CONFIG", &provider_config) // absolute path: the CLI refuses a relative one
.arg("--embedding-provider")
.arg(crate::common::embeddings::PROVIDER_ID)
.arg("--export-format")
.arg("vector")
.arg("--clean-ai")
.assert()
.success();
Two details the fake has to honor: the endpoint answers one vector per input
(the adapter rejects a body whose vector count does not match the text count), and
the provider is declared in the config file with auth = { source = "none" } and
allow_loopback = true — the loopback opt-in is config-file-only by design
(#1462); there is deliberately no CLI flag for it.
Snapshot testing with insta
Content assertions use insta snapshots instead of brittle
content.contains(...) checks, so a full output change is reviewed as a diff rather than
a silent boolean flip.
Review gate (RED → GREEN)
cargo insta is not installed in this environment. Use the env-var workflow instead:
-
RED — first run fails because the
.snapis missing or differs, and a*.snap.newpending file is written next to it:cargo nextest run --test behavioral -
GREEN — regenerate and accept the pending snapshots, then re-run with no flag to confirm they are now stable (no new
*.snap.newshould appear):INSTA_UPDATE=always cargo nextest run --test behavioral cargo nextest run --test behavioral # must stay green -
Inspect the generated
*.snapfiles, then stage them with the code change.
*.snap.newis git-ignored (see.gitignore). Never commit a*.snap.new; commit the accepted*.snap.
Where snapshots live
insta resolves the snapshot directory from the module where assert_snapshot! expands.
The thin assert_snapshot_* wrappers therefore live at each test crate's root module so
snapshots land where the suite expects:
tests/behavioral/snapshots/— rootbehavioralsnapshotstests/behavioral/cli/snapshots/— obsidian snapshots (local helper insidecli/obsidian_test.rs)tests/snapshots/—cli_binary__*.snap
Redaction conventions
Scrape output embeds per-run, machine-specific, and non-deterministic values. A shared
helper, tests/common/cli_harness.rs::redact_nondeterministic, collapses them before
snapshotting so approved snapshots stay stable across machines and runs:
| Leak | Redacted to |
|---|---|
TempDir absolute path | <OUT_DIR> |
| ANSI color escape sequences | (stripped) |
ISO-8601 timestamps (timestamp_utc, scrapeDate, scrape_date, …) with or without fractional seconds and any offset/Z | <TIMESTAMP> |
Wiremock 127.0.0.1:<port> | 127.0.0.1:<PORT> |
cli_behavioral additionally emits a bare date: frontmatter field (date only, no time
component) that the helper cannot catch, so assert_content_snapshot applies an insta
add_filter for date: \d{4}-\d{2}-\d{2} → date: [DATE] (see
tests/cli_behavioral_test.rs).
Adding a new snapshot test
- Build the scrape output through the shared harness (
BehavioralTest/cmd). - Call the crate's
assert_snapshot_*wrapper (root module) or, for free-text content,assert_content_snapshotincli_behavioral. - If a new non-deterministic field appears, extend
redact_nondeterministic(centralized) rather than adding a per-test hack. - Generate + accept via
INSTA_UPDATE=always, then verify with a plain run.
Lint
cargo clippy -p webfang_core --test behavioral --test cli_binary -- -D warnings
Gate clippy on the specific test crates (not --tests): webfang_core's own lib
tests have a pre-existing tokio::time::pause failure that requires the test-util
feature and is out of scope for E2E changes.
Coverage exclusions (LCOV)
Defensive error paths — invariants by design — must not drag down the codecov/patch target (80% on new lines). Annotate them with LCOV exclusion markers (issue #527).
Policy
Only annotate arms that "should not happen in normal operation":
- internal / mutex-poisoning / integer-overflow invariants
- compile-time-constant failures (CSS selectors, regexes, hardcoded URLs)
- panic/expect paths guarded by proven invariants (e.g.
NonZeroU32after a zero-check,chunks_exactslice conversion)
NEVER annotate business paths: reachable errors like HTTP connection failures, parse errors, or config validation. Reachable error handling is exercised by tests and counted like any other code.
Syntax
- Single statement:
// LCOV_EXCL_LINEon its OWN comment line immediately ABOVE the code line — never inline on the code line. - Multi-line arm/block:
// LCOV_EXCL_STARTabove the block and// LCOV_EXCL_STOPbelow it, each on its own line. - Every marker site carries a justification comment starting with
// defensive: <variant> <rationale>— merged into the marker line or as a preceding line.
Safety net
Excluded paths are still mutation-tested: a surviving mutant in the weekly
cargo-mutants baseline, or in a PR touching gated hot paths (cargo-mutants
PR diff), is reported. The markers only affect coverage accounting — they do
not affect mutant survival.
Hot-path rule
In files under .cargo/mutants.toml globs, markers MUST be own-line comments
(never inline) so the diff adds no mutable code lines.
Never lower codecov.yml thresholds; use markers per path instead.
Troubleshooting
Common problems and how to diagnose them with WebFang's built-in tracing.
Install and uninstall paths live in installation.md. If the binary never started, that page is the first place to look.
Generate a trace first:
webfang --url <URL> --trace-file debug.jsonl -vvv, then query it withscripts/analyze-trace.shorjq. See debugging.md for the full query cookbook.
The downloaded binary won't run
These are install-time failures — they happen before any tracing exists, so none of the trace workflow below applies yet. Each symptom names its cause and its one-line fix; the full verified commands are in installation.md.
| Symptom | Cause | Fix |
|---|---|---|
GLIBC_2.xx not found (Linux) | Distro older than the ubuntu-latest builder (glibc 2.39); there is no musl artifact | Build from source, or use a newer distro |
command not found: webfang | Extracted but never made executable, or not on PATH | chmod +x webfang, then sudo install -m 0755 webfang /usr/local/bin/webfang |
cannot be opened because the developer cannot be verified (macOS) | com.apple.quarantine xattr on the downloaded archive's contents | xattr -d com.apple.quarantine webfang, then re-run |
VCRUNTIME140.dll was not found (Windows) | MSVC dynamic CRT / Visual C++ Redistributable absent | Install the VC++ Redistributable 2015-2022 (x64) |
sha256sum: command not found (macOS) | sha256sum is GNU coreutils; stock macOS ships shasum | shasum -a 256 <archive> and compare with the SHA256SUMS.txt line |
A hash does not match SHA256SUMS.txt | Truncated/partial download, or the file is not from this release | Re-download the archive; do not run the binary |
| No binary for your platform (Intel Mac) | x86_64-apple-darwin is not built (the original reason was the ONNX prebuilt; the target list was not extended after the ONNX removal) | Nothing to download; see Intel macOS |
--js-strategy full or SQLite resume absent | chromium / persistence are not in the release build | Expected; see what is not in the release binary |
webfang-mcp or webfang-mcp-stdio missing after extracting webfang-<target> | Each binary has its own archive; the CLI archive holds only webfang | Download webfang-mcp-<target> / webfang-mcp-stdio-<target>; see MCP servers |
If a binary fails with something not in this table, verify the checksum first — it separates "my download is damaged" from "this is a real bug" before you file anything.
The crawl is slow
Diagnose:
scripts/analyze-trace.sh debug.jsonl slow 20 # slowest spans
scripts/analyze-trace.sh debug.jsonl stages # time per pipeline stage
Common causes:
- A single stage dominates (e.g.
cleanwith the AI feature) — check thestagesdistribution. - Network latency / rate limiting — look for large gaps between
crawl_pagespans; consider--delayand concurrency tuning. - Export bottleneck — check
export_batchspan durations.
Pages are failing silently
Every operational error is logged as a structured ERROR event with url,
stage, and (when available) trace_id.
scripts/analyze-trace.sh debug.jsonl errors # all errors with context
scripts/analyze-trace.sh debug.jsonl urls-failed # unique failed URLs
Common causes by stage:
stage | Meaning | Fix |
|---|---|---|
fetch | HTTP/network failure or WAF challenge | Check connectivity; the site may be blocking — see WAF section below |
extract | Content extraction produced too little text | The page may be JS-rendered or non-article; try a CSS --selector or JS rendering |
A local or internal target discovers nothing
Symptom. You point WebFang at a localhost, 127.0.0.1, 10.x, 192.168.x or
169.254.169.254 target and get nothing back. How loud that is depends on how the target
is spelled:
$ webfang http://127.0.0.1:8080/ --dry-run # an IP literal
Warning: SSRF detectado: la IP 127.0.0.1 del host '127.0.0.1' está prohibida (acceso a
red interna/cloud metadata bloqueado). La vista previa no descubrió URLs porque el guard
cortó la semilla antes de abrir conexión. Para un destino interno de confianza: …
$ echo $?
2
$ webfang http://localhost:8080/ --dry-run # a name that resolves into the same range
Dry-run: 0 URL(s) would be scraped:
$ echo $?
0
The literal case names its own cause (since #1391): exit 2, the code a null discovery result
already uses, with no Dry-run: line to misread. The hostname case still reads as an empty
site, because the layer that refuses it — the connect-time resolver — answers with what
looks like a DNS failure, and the preview cannot tell that from a broken name. Either way,
the same target in a real crawl fails loudly.
What happened. The SSRF guard refused the target before any socket opened: zero
packets leave the process, so nothing is retried and nothing can time out. Discovery itself
still answers Ok with an empty URL list — inside a crawl, one refused URL is a legitimate
skip, not a failed run — and the engine's own page-error count (crawl completed … errors: 1)
is not part of that list. What changed in #1391 is that the preview asks the guard its own
question before calling the result a success. The guard policy is the long-standing one: the
scrape path has always paid it, the MCP crawl tools since #1355, and plain DOM discovery
since #1369 retired the last entry point that fetched without the full guard chain.
What you see, by surface. The same refusal, spelled two ways, produces five different-looking outcomes:
| You ran | What you see | Exit |
|---|---|---|
webfang <literal-target> --dry-run | Warning: SSRF detectado: la IP … está prohibida …, naming both hatches — and no Dry-run: line (#1391) | 2 |
webfang <hostname-target> --dry-run | Dry-run: 0 URL(s) would be scraped: | 0 |
webfang <target> (crawl) | Failed to scrape <target>: URL inválida: SSRF detectado: la IP 127.0.0.1 del host '127.0.0.1' está prohibida (acceso a red interna/cloud metadata bloqueado) | 69 |
webfang <target> --single-page | the same SSRF detectado line — the scrape path carries the same guard | 69 |
discover_urls_unified / discover_urls_recursive (library) | Ok with an empty Vec | — |
Two exit codes for one refusal, and both are right. The crawl is loud because discovery
returns nothing, plan_urls re-injects the seed, and the scrape path refuses it a second
time — a scrape that produced no page is a network-class failure, so it leaves at 69. The
preview is the other case: nothing failed to run, there was simply no URL to schedule,
which is the null result the sitemap arms already report as 2 — now with its cause named, so
the code stops reading as "the site is empty". --single-page never runs discovery, so it
only ever shows the loud half. Do not read either code as "retry me and it might work": both
are one policy refusal. Whether a path is seed-guarded at all depends on which of the four
SSRF layers it wires; the layer map lives in docs/ssrf-layers.md.
How to tell a refusal from a genuinely empty site. The refusal is logged, never
hidden — and for an IP-literal seed the exit code now says it too (2, with the cause
named). A hostname that resolves into a forbidden range is the case that still looks
silent, so the WARN line is the discriminator: at default verbosity it is already on
stderr.
WARN webfang_core::domain::ssrf_guard: SSRF literal-IP target rejected at entry (no socket opened), host: 127.0.0.1, ip: 127.0.0.1
To query it instead of squinting at stderr, add --trace-file (with -v the engine also
emits its run summary, where the refused seed shows up as errors: 1 on total_pages: 0):
webfang http://127.0.0.1:8080/ --dry-run -v --trace-file debug.jsonl
jq -c 'select((.message // "") | test("SSRF|crawl completed"))' debug.jsonl
scripts/analyze-trace.sh debug.jsonl errors
One trap inside the trap: the hostname row above is not a bug waiting to be fixed, and
its download error: DNS error: name resolution failed is not a typo. The validating
resolver refused to hand back the address, and at this boundary that is indistinguishable
from a name that genuinely does not exist — which is why the preview keeps exiting 0 for
localhost and 2 for 127.0.0.1. Literals and hostnames are checked by different layers;
see docs/ssrf-layers.md.
Opting out, for a target you trust. There is no per-target allowlist flag; the
disarmers are environment variables, and only the exact value 1 disarms anything:
# Literal address (127.0.0.1, 10.0.0.5, …): lifts the literal entry guard.
WEBFANG_DISABLE_SSRF_ENTRY_GUARD=1 webfang http://127.0.0.1:8080/ --dry-run
# Hostname that resolves into a private range: lifts the connect-time resolver.
WEBFANG_DISABLE_SSRF_RESOLVER=1 webfang http://localhost:8080/ --dry-run
# A local target that answers to both spellings, or that you simply want to work:
WEBFANG_DISABLE_SSRF_ENTRY_GUARD=1 WEBFANG_DISABLE_SSRF_RESOLVER=1 \
webfang http://127.0.0.1:8080/ --dry-run
Rules of thumb:
- Set them per command, never in a profile. These variables disarm a control that
exists because a crawler follows links written by strangers: an allow-everything
environment is how a scraping run becomes an internal-network scan. A
WEBFANG_*line in your shell rc applies to every target you will ever crawl, including the ones you did not mean to fetch. - The resolver value is read once, at client construction. Export it before starting a long-lived process (the MCP server, a batch runner); setting it mid-run changes nothing for the clients already built.
- MCP needs its own variable too.
WEBFANG_MCP_DISABLE_SSRF=1lifts MCP's entry pre-check (layer 1) only; the core layers above still apply. A local target driven through MCP therefore needs the MCP switch and the core pair. - Redirects have a fourth, separate disarmer (
WEBFANG_DISABLE_SSRF_REDIRECT_GUARD=1). Leave it armed: a public site that redirects into your internal range is exactly the case the guard was written for.
The full layer map — which variable lifts which layer, and what stays protected after you
set it — is in docs/ssrf-layers.md.
WAF / bot detection blocks
scripts/analyze-trace.sh debug.jsonl waf # WAF challenges + banned domains
If you see WAF challenge detected errors:
- The site is presenting a CAPTCHA / challenge page. WebFang bans the domain for the rest of the crawl to avoid hammering it.
- Try a different TLS fingerprint profile (
--tls-emulation): a WAF challenge short-circuits the fetch at the layer that saw it, so WebFang never re-renders the page with JavaScript to get past it. - Slow down (
--delay, lower concurrency) to avoid rate-limit triggers.
I can't tell which logs belong to one page / one crawl
- One crawl shares a single
trace_id. Filter by it:scripts/analyze-trace.sh debug.jsonl trace <trace_id> - Each page is a
crawl_pagespan with its ownspan_idunder thattrace_id.
Non-deterministic snapshot failures in tests
correlation_id / trace_id are internal and #[serde(skip)] on scraped
output, so they never appear in scraped JSON/JSONL snapshots. If a new
field you added is non-deterministic (timestamps, ports, temp paths, random
IDs), redact it via redact_nondeterministic() in tests/common/cli_harness.rs.
Async deadlocks / starved tasks
For concurrency bugs (a crawl hangs, tasks never complete), use the Tokio Console:
RUSTFLAGS="--cfg tokio_unstable" cargo run --features console -- --url <URL>
This shows live task states and poll times, making stuck tasks visible.
Empty or poor content
content extraction failed(stage: extract) — the fallback extractor got less than the minimum content. The page is likely JS-rendered, an interactive app, or not an article.- Try
--selector '.main-content'(or the right CSS selector for the site), or enable JS rendering for SPA content.
Leftover .lock files in the state directory — is my state corrupted?
Symptom. After a --resume run (or a killed one) the --state-dir
contains <domain>.json.lock files next to the <domain>.json state files,
even though no writer is running.
Short answer. No — nothing is corrupted, and nothing is stuck. The .lock
file is a permanent sentinel that is created on demand and intentionally never
removed; the real mutual exclusion is the advisory flock on that file, which
the kernel releases when the process exits (including after SIGKILL).
Windows: output is mojibake
Symptom. Accented or ñ characters render as garbage in the terminal.
webfang switches the Windows console code page to UTF-8 as its first action (#1808, XP-K-03), so this normally cannot happen. If it does, the code page was most likely reset afterwards — some terminals and CI shells re-apply the profile default when they attach. Fix it in that shell, not in webfang:
chcp 65001
Windows: a deeply nested output path fails to write
Symptom. A crawl into a deeply nested --output directory fails with a
path error while the same crawl succeeds elsewhere.
Known limitation (XP-P-07). Windows caps a path at MAX_PATH — 260
characters for the whole path, which is a separate limit from the
255-character-per-component rule webfang enforces and validates. webfang
validates every component, but it does not rewrite paths into the extended
\\?\ form: that form disables ./.. normalization and changes UNC handling,
so applying it at every call site would trade a documented limit for a set of
subtle ones.
Workaround. Either shorten the output root, or enable long paths for the machine (administrator PowerShell, then sign out and back in):
New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" `
-Name LongPathsEnabled -Value 1 -PropertyType DWORD -Force
Windows: closing the console window kills the run without draining
Should not happen. A console close, logoff, or shutdown event now drives the same graceful drain as Ctrl+C (#1808, XP-S-02) — in-flight pages are persisted and a bounded checkpoint is written before exit.
One residual, stated plainly: the console host allows a close-event handler about five seconds before it kills the process. That window belongs to Windows, not to webfang, so console-close draining is best-effort. Logoff and shutdown events are generous. If you interrupt a long crawl repeatedly, prefer Ctrl+C, which has no such deadline.
State Recovery Runbook
What to do when webfang's persisted state looks wrong.
This page covers the SQLite/WAL vault database (~/.webfang/crawl.db) —
the artifact that had no recovery procedure anywhere — and routes the other
persisted artifacts (the crawl checkpoint and the JSON resume state) to their
own contract.
Every behaviour described here is pinned by a test that already exists in the repository, and the test is named next to the claim. A step marked not covered by a test is either SQLite's own behaviour or an operator instruction: it is never a promise webfang makes.
The durability contract — what is guaranteed and what is explicitly not — lives in
docs/persistence-resume.md. This page is the procedure: what to check, what to run, what to delete.Where the files live and how to remove them: installation.md. Symptoms that are not state at all: troubleshooting.md.
Triage — which artifact is broken?
| Symptom | Artifact | Section |
|---|---|---|
A run fails with base de datos con versión de esquema … | vault DB | The vault database |
A run fails with vector BLOB corrupto: longitud N no es múltiplo de 4 | vault DB | A corrupt embedding |
A run with --elastic fails before it crawls | vault DB | The schema-version gate |
--resume re-scrapes pages it had already done | JSON state | The JSON state |
A <domain>.json.bak appeared next to the state file | JSON state | The JSON state |
A <domain>.json.lock is still there after the run | — | Nothing is broken: the sentinel is permanent by design — see troubleshooting.md |
The vault database
Where the database lives
~/.webfang/crawl.db by default — absolute, under $HOME, not under the
webfang cache base. --db-path (or WEBFANG_DB_PATH) moves it.
| Behaviour | Pinned by |
|---|---|
The default path is <home>/.webfang/crawl.db | test_default_db_path_under_home — crates/webfang_core/src/infrastructure/autotuning.rs |
| The CLI flag wins over the env var, which wins over the default | test_resolve_db_path_priority — crates/webfang_core/src/infrastructure/autotuning.rs |
| The file is created lazily: the pool is built first, the file appears when the schema is initialised — that is the fail-fast point if the path is not writable | test_db_file_created_when_absent — crates/webfang_core/src/infrastructure/persistence/sqlite.rs |
The database only exists in builds compiled with the persistence feature. The release binary is built with --features "mcp" only, so a release install never creates it | not covered by a test — the feature gate is crates/webfang_core/Cargo.toml (persistence = ["dep:rusqlite", "dep:deadpool-sqlite", "dep:sys-info"]) and the release command is .github/workflows/release.yml; the same fact is stated for operators in installation.md |
How it is opened
| Behaviour | Pinned by |
|---|---|
Every pooled connection runs journal_mode=WAL, synchronous=NORMAL, cache_size=-4000 (the first is persistent in the DB header, the other two are per-connection and are re-applied by a post_create hook) | test_wal_pragmas_applied_to_connection — crates/webfang_core/src/infrastructure/persistence/sqlite.rs |
The schema DDL is idempotent (CREATE … IF NOT EXISTS), so re-initialising an existing database is a no-op | test_setup_schema_is_idempotent — same file |
The schema-version gate: refuse, never migrate
The database carries its schema version in SQLite's own on-disk header slot
(PRAGMA user_version). webfang writes 1 and opens only 1.
| Behaviour | Pinned by |
|---|---|
| A fresh database is stamped with the current version, and the stamp survives a reopen; a second initialisation neither fails nor re-stamps | setup_schema_stamps_the_schema_version_marker — crates/webfang_core/src/infrastructure/persistence/sqlite.rs |
| A database stamped with another version is refused before any DDL runs — the refused file is never opened for writing, and it keeps its own marker | setup_schema_refuses_a_foreign_schema_version_without_writing — same file |
A database stamped 0 (written before the marker existed, or brand new) is adopted, not refused | setup_schema_adopts_an_unversioned_database — same file |
The refusal message names both versions, in Spanish: base de datos con versión de esquema {found}, incompatible con la versión {SCHEMA_VERSION} de webfang; actualizá webfang o usá una base de datos distinta (nunca se modifica una base desconocida) | setup_schema_refuses_a_foreign_schema_version_without_writing — same file (it asserts both numbers appear in the message) |
There is no migration mechanism: a bump of the marker without a migration
would turn every existing database into a refusal, so the gate refuses by
design. The behaviour that pins this is the refusal itself
(setup_schema_refuses_a_foreign_schema_version_without_writing): a foreign
version is refused, never converted.
Recovery procedure
-
Stop every webfang process. The database is only safe to touch when no run holds it open.
-
Keep the evidence before you touch anything:
cp ~/.webfang/crawl.db ~/.webfang/crawl.db.broken.$(date +%s) -
Read the marker (SQLite's own tool — not covered by a test):
sqlite3 ~/.webfang/crawl.db 'PRAGMA user_version;'1(today'sSCHEMA_VERSION) is a database this build owns.0is an unversioned one, which webfang adopts. Anything else is a database webfang refuses — it was written by a build with a different column layout. -
Check the file's integrity (SQLite's own tool — not covered by a test; webfang never runs it):
sqlite3 ~/.webfang/crawl.db 'PRAGMA integrity_check;' -
Choose the recovery:
Situation What to do The marker is foreign ( webfangrefuses the file)Either update webfang so the version matches, or point --db-path/WEBFANG_DB_PATHat a different file. This is what the refusal message itself asks for. Note that--db-pathonly has an effect together with--elastic— on its own it is rejected with exit 78 (not covered by a test; the guard iscrates/webfang_core/src/cli/preflight.rs::check_db_path_requires_elastic).The marker is foreign and the ingested vectors are not needed Delete crawl.db,crawl.db-walandcrawl.db-shmtogether, with no run active, and let the next run create a fresh database. The DDL is idempotent, so the new file is created and stamped on the spot.The marker is 1and the file passesintegrity_checkNothing to recover — the refusal came from somewhere else. Re-read the error text; the two webfang-raised failures are the two rows below. The file is corrupt and the run fails on a vector read See A corrupt embedding.
What is lost when the database is deleted. The database holds the ingestion tables —
resources,chunks,notes,note_chunksandextraction_fingerprints. Deleting it discards the ingested vectors, the stored chunks and the failure fingerprints. It does not touch the Markdown/JSONL the run wrote to the output directory: those are separate files, written by the exporters.Not covered by a test: the sidecars
-waland-shmare SQLite's own artifacts. webfang never creates, truncates or deletes them by name, and it never runsPRAGMA wal_checkpoint. Deleting the three files together with no process attached is SQLite's own recovery practice, not a webfang guarantee.
A corrupt embedding blob
| Behaviour | Pinned by |
|---|---|
A stored embedding whose length is not a multiple of 4 is refused with a Spanish error — vector BLOB corrupto: longitud N no es múltiplo de 4 — instead of being decoded into garbage | test_bytes_to_f32_vec_rejects_invalid_blob_length — crates/webfang_core/src/infrastructure/persistence/sqlite.rs |
The same failure surfaces through the public read path (get_vector) as an error, never as a panic | test_corrupt_blob_returns_error — same file |
A corrupt blob surfaces as an error on the read of that row. Whether the rest of the database keeps working is SQLite's own behaviour — not covered by a test. The recovery is the same as above: the file itself is not damaged.
Retention
| Behaviour | Pinned by |
|---|---|
--retention-days N prunes rows older than the cutoff from all four tables, children before parents, and runs VACUUM afterwards | vault_db_rows_older_than_cutoff_are_pruned_children_first — crates/webfang_core/src/cli/retention.rs (the test asserts the row counts; the VACUUM is in the code path, not asserted) |
| Rows inside the cutoff survive the prune | same test (it asserts the fresh rows are still there) |
Without --retention-days nothing is ever pruned and the run prints no
retention summary — default_retention_is_disabled_and_never_prunes —
crates/webfang_core/tests/retention_test.rs. Rows whose
created_at is NULL or unparseable are never pruned — the SQL condition
is in prune_older_than
(crates/webfang_core/src/infrastructure/persistence/sqlite.rs); the test seeds
only dated rows, so the NULL case is not covered by a test.
The JSON state
The crawl checkpoint and the JSON resume state are covered by the durability
contract in docs/persistence-resume.md. This section only lists what the
recovery procedure needs.
| Behaviour | Pinned by |
|---|---|
The record store writes a temp file and renames it over the live file; rename(2) is the commit point — a crash before it leaves the previous complete state, never a torn file | crash_between_saves_leaves_exported_state_reloadable_and_completable and crash_after_committed_rename_sees_committed_on_reload — crates/webfang_core/src/infrastructure/export/record_store.rs |
A crash at any of the pinned crash points, followed by a --resume rerun, recovers with zero lost URLs and no duplicated committed line | every row of crates/webfang_core/tests/crash_matrix_test.rs (for example crash_mid_state_file_write_truncated_tmp_is_gced, crash_tmp_written_pre_rename_discards_tmp_and_recovers, crash_while_holding_lock_releases_and_recovers) |
A stale .tmp-* sibling left by a crash is garbage-collected on the next load and is never read as data | stale_tmp_files_are_garbage_collected_on_load_and_never_read — crates/webfang_core/src/infrastructure/export/record_store.rs |
A discarded state file leaves a <file>.bak sibling carrying the exact pre-migration bytes; an existing backup is never overwritten, so the first bytes win | test_load_or_default_stale_version_preserves_bak — crates/webfang_core/src/infrastructure/export/state_store.rs; preserve_abandoned_bytes_writes_rejected_bytes_once — crates/webfang_core/src/application/resume.rs |
The .bak extends the artifact's own extension (state.json → state.json.bak) | backup_sibling_extends_the_existing_extension — crates/webfang_core/src/application/resume.rs |
The crawl checkpoint is [4-byte CRC32][JSON payload]; a checksum mismatch refuses the file and leaves the exact rejected bytes in a .bak | checksum_is_first_four_bytes and crc32_mismatch_preserves_bak_before_the_rewrite — crates/webfang_core/src/application/crawler/checkpoint.rs |
| A checkpoint whose schema version is superseded is discarded (never resumed half-understood) and its pre-migration file is preserved | stale_checkpoint_version_is_discarded_not_resumed — crates/webfang_core/tests/checkpoint_store_integration.rs |
The <state>.lock sentinel is permanent: the lock is released when the process exits, the file is never unlinked | dropping_the_store_lock_releases_but_never_unlinks_the_sentinel — crates/webfang_core/src/infrastructure/export/record_store.rs |
Recovery in one line: if a state file looks wrong, the .bak sibling next
to it holds the bytes webfang refused. Copy it back (with no run active) if
you need the old state; otherwise delete it and let the next --resume start
from scratch.
What this runbook does not promise
- Power loss. Every atomic write in the persistence layer is
temp-file +
rename(2)with nofsync: it is atomic against process death, not against power loss. The contract isdocs/persistence-resume.md. - An integrity check on the database. webfang never runs
PRAGMA integrity_checkand never checkpoints the WAL. Both commands in the recovery procedure above are SQLite's own tools, invoked by the operator. - A migration for a foreign database. The gate refuses; it does not convert.
- Automatic repair. Nothing in this repository repairs a damaged file on its own. The repairs this page lists are operator actions.
Where the code lives
| Component | Path |
|---|---|
| SQLite pool, pragmas, schema, version gate | crates/webfang_core/src/infrastructure/persistence/sqlite.rs |
| Fingerprint table (same pool) | crates/webfang_core/src/infrastructure/persistence/fingerprint.rs |
| Retention prune | crates/webfang_core/src/cli/retention.rs |
| DB path resolution | crates/webfang_core/src/infrastructure/autotuning.rs |
| JSON state store and record store | crates/webfang_core/src/infrastructure/export/{state_store,record_store}.rs |
| Crawl checkpoint | crates/webfang_core/src/application/crawler/checkpoint.rs |
.bak policy (one policy for every destructive rewrite) | crates/webfang_core/src/application/resume.rs |
| Crash-injection harness | crates/webfang_core/src/cli/crash_points.rs |
CLI Reference
Complete flag reference for the webfang binary. The block in the
"Complete flag reference" section below is auto-generated from the
binary itself by scripts/gen-cli-reference.sh (built with the exact release
feature set, ai mcp), so flags, defaults, and WEBFANG_* environment
variables always match reality. CI drift-checks the chapter against the binary
on every doc run; regenerate locally with just cli-doc-regen.
Every flag in this reference is compiled into the binary you are running.
The release artifacts are built with no capability features, so
feature-gated behavior (--js-strategy full, SQLite persistence, adaptive
selectors, tokio-console) is absent or rejected there — the per-flag
consequences are in the
Release Feature Matrix.
Quick start
Scrape a single page by passing the URL positionally:
webfang https://example.com
Crawl a whole site following its sitemap.xml:
webfang https://example.com --use-sitemap --max-pages 50
Extract only the elements matching a CSS selector and export for a RAG pipeline:
webfang https://example.com -s "article.main-content" --export-format jsonl
Resume an interrupted crawl from its last checkpoint:
webfang --batch-file urls.txt --resume
Zero values
Numeric flags follow the Zero Silent Loss policy: a zero is never silently turned into another value.
- Zero disables the feature where an "off" state exists:
--delay-ms 0disables request pacing (no token bucket is allocated),--max-depth 0scrapes only the seed URL. - Zero is rejected with a usage error where zero is meaningless or
destructive:
--concurrency,--download-concurrency,--download-timeout,--max-pages,--rate-limit-burst,--timeout-secs. - Legacy exception: a non-numeric
--rate-limit-burstvalue warns and falls back to the hardware-derived default (numeric zero and out-of-range values are still rejected).
Configuration file
Flags can also come from a TOML file, which is how you set values once
instead of on every invocation. webfang looks for webfang/config.toml
under the platform's user config directory:
| Platform | Resolved from |
|---|---|
| Linux | $XDG_CONFIG_HOME, else ~/.config |
| macOS | ~/Library/Application Support |
| Windows | %APPDATA% |
An absolute XDG_CONFIG_HOME is honored on every platform, not just
Linux — that is what makes hermetic test fixtures possible.
To point at a specific file instead, set WEBFANG_CONFIG to the path of
the config.toml itself. It takes precedence over the table above, and it
differs from XDG_CONFIG_HOME in two ways worth knowing:
- It names the file, while
XDG_CONFIG_HOMEnames a directory that webfang appendswebfang/to. Use it when your file does not live in that subdirectory. - It is honored on Windows, where no environment variable reaches the
platform config lookup: that lookup calls
SHGetKnownFolderPath(FOLDERID_RoamingAppData), a Windows Shell API that reads no env var at all. UntilWEBFANG_CONFIGexisted, a Windows install could not be redirected to a config file by env at all.
A missing file at the platform path is not an error: webfang falls back to its defaults, because a fresh install has no config yet.
That tolerance stops at an explicit WEBFANG_CONFIG. Once you name a file,
webfang holds you to it — the run stops with exit code 78 (configuration
error) and a message naming the path, instead of quietly running with
defaults. A typo in a config path used to be indistinguishable from a
successful run.
WEBFANG_CONFIG | Result |
|---|---|
| unset, or empty | Platform lookup; a missing file means defaults |
| absolute path to a readable, valid TOML file | Used |
| relative path | Rejected, exit 78 |
| path does not exist | Rejected, exit 78 |
| unreadable — wrong permissions, or a directory | Rejected, exit 78 |
| not valid TOML | Rejected, exit 78 |
Two edge cases matter because this variable is usually set from a script or a service unit rather than typed:
- An empty value counts as unset.
WEBFANG_CONFIG= webfang ...behaves exactly like not setting the variable at all, so a script that exports it before it has a value falls back to the platform lookup instead of failing. - A relative path is rejected rather than resolved. It used to be resolved against the current working directory of the process, not against your home directory — predictable in an interactive shell, and arbitrary under a daemon or service manager, whose working directory is whatever the supervisor chose. The same variable could therefore name a different file depending on who started the process, which is why an absolute path is now required.
WEBFANG_CONFIG=/etc/webfang/config.toml webfang https://example.com
Complete flag reference
Everything below is the verbatim output of webfang --help and
webfang completions --help — including every default value and the
matching WEBFANG_* environment variable for each flag.
$ webfang --help
CLI Arguments for the webfang binary.
Parsed using `clap` with derive macros.
# Examples
```no_run use webfang_core::Args; use clap::Parser;
let args = Args::parse_from([ "webfang", "--url", "https://example.com", "--output", "./output", "--export-format", "jsonl", "--resume", ]);
assert_eq!( args.crawler.url.as_ref().map(webfang_core::domain::ValidUrl::as_str), Some("https://example.com") ); ```
Usage: webfang [OPTIONS] [URL]
webfang [OPTIONS] [URL] <COMMAND>
Commands:
completions Generate shell completion scripts
connect Detect and validate a remote embedding endpoint, then write its config
doctor Diagnose the embedding path — config, credential, network, budget
show Dump the effective embeddings provider, the configured budget, and the historical model-profile metadata
recover Recover pages from a stranded batch capture spool into markdown files
help Print this message or the help of the given subcommand(s)
Arguments:
[URL]
URL to scrape (positional shorthand — equivalent to --url). Hardened through the SAME argv-boundary parser as `--url` (#1239): the shorthand cannot bypass the credential strip
Options:
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
Target:
-u, --url <URL>
URL to scrape (required unless using a subcommand)
[env: WEBFANG_URL=]
-s, --selector <SELECTOR>
Matched elements become the content directly (no main-content cleanup); title and date come from the whole document, excerpt stays empty
[env: WEBFANG_SELECTOR=]
[default: body]
Discovery:
--delay-ms <DELAY_MS>
Delay between requests in milliseconds
[env: WEBFANG_DELAY_MS=]
[default: 1000]
--jitter-ms <JITTER_MS>
Random addition to each delay between requests, in milliseconds (0 = fixed delay; the mean interval becomes delay + jitter/2, so the request rate drops; with 0, a run that fits in the burst is not paced at all)
[env: WEBFANG_JITTER_MS=]
[default: 0]
--max-pages <MAX_PAGES>
Maximum pages to scrape
[env: WEBFANG_MAX_PAGES=]
[default: 10]
--concurrency <CONCURRENCY>
Concurrency level (auto or number, minimum 1)
[env: WEBFANG_CONCURRENCY=]
[default: auto]
--rate-limit-burst <RATE_LIMIT_BURST>
Explicit rate-limiter burst permits (token-bucket capacity).
Overrides the hardware-derived budget-model default (Q1: burst is decoupled from crawl concurrency). Accepts a positive integer, or the keyword `auto` for the derived default (the same spelling `--concurrency` understands). Raw string here ON PURPOSE: validation/conversion happens once in preflight staging via `parse_rate_limit_burst` so CLI, env, and programmatic input all share one semantic. That semantic FAILS CLOSED: `0`, an out-of-range value, and any other typo are rejected with exit 78 rather than degrading to the default — only `auto` and an empty value mean "not set".
[env: WEBFANG_RATE_LIMIT_BURST=]
--use-sitemap
Use sitemap for URL discovery NOTE: HTTP redirects (301/302) are resolved at scrape-time, not parse-time. This avoids redundant HEAD requests during sitemap parsing for better performance
[env: WEBFANG_USE_SITEMAP=]
--sitemap-url <SITEMAP_URL>
Explicit sitemap URL
[env: WEBFANG_SITEMAP_URL=]
Behavior:
--single-page
Scrape only the seed URL without discovery or crawling (batch mode always scrapes one page per URL)
[env: WEBFANG_SINGLE_PAGE=]
--resume
Resume mode - skip URLs already processed
[env: WEBFANG_RESUME=]
--state-dir <STATE_DIR>
Custom state directory for resume mode
[env: WEBFANG_STATE_DIR=]
--download-images
Download images from the page
[env: WEBFANG_DOWNLOAD_IMAGES=]
--download-documents
Download documents from the page
[env: WEBFANG_DOWNLOAD_DOCUMENTS=]
--download-assets
Download all assets (images + documents) from the page
[env: WEBFANG_DOWNLOAD_ASSETS=]
--extraction-fingerprint
Record extraction failure fingerprints in SQLite and attach them to low-quality extraction hints (#792). Repeated low-score extractions on the same site/selector pair accumulate a failure count surfaced in the hint, instead of degrading silently
[env: WEBFANG_EXTRACTION_FINGERPRINT=]
--clean-ai
Use AI-powered semantic cleaning for better RAG output
[env: WEBFANG_CLEAN_AI=]
[alias: --ai]
Display:
-v, --verbose...
Verbosity level: -v (INFO), -vv (DEBUG), -vvv (TRACE)
[env: WEBFANG_VERBOSE=]
-q, --quiet
Quiet mode — suppress info/debug output
[env: WEBFANG_QUIET=]
-n, --dry-run
Dry-run mode — discover URLs and print without scraping
[env: WEBFANG_DRY_RUN=]
--trace-file <TRACE_FILE>
Path to write OTel spans as JSONL for offline debugging
[env: WEBFANG_TRACE_FILE=]
Crawler Settings:
--max-depth <MAX_DEPTH>
Maximum depth to crawl (0 = only seed URL)
[env: WEBFANG_MAX_DEPTH=]
[default: 2]
--timeout-secs <TIMEOUT_SECS>
Request timeout in seconds
[env: WEBFANG_TIMEOUT_SECS=]
[default: 30]
--include-pattern <INCLUDE_PATTERNS>
URL patterns to include (glob-style). Three modes:
* Path: starts with `/` → matched against URL path, e.g. `/pricing`, `/admin/*` * Path glob: starts with `*/` → matched against URL path, e.g. `*/api/*` * Host (default): matched against hostname, e.g. `example.com`, `*.example.com`
Example: to exclude a path, use `--exclude-pattern "/admin/*"`, not `*admin*`
[env: WEBFANG_INCLUDE=]
--exclude-pattern <EXCLUDE_PATTERNS>
URL patterns to exclude (glob-style, same three modes as --include-pattern). Deny takes precedence over allow
[env: WEBFANG_EXCLUDE=]
Download Settings:
--asset-naming <ASSET_NAMING>
Estrategia de nombre de archivo para assets descargados: hash (default), slug, content-disposition
[default: hash]
[possible values: hash, slug, content-disposition]
--download-concurrency <DOWNLOAD_CONCURRENCY>
Máximo de descargas de assets concurrentes por página (mínimo 1)
[env: WEBFANG_DOWNLOAD_CONCURRENCY=]
--max-file-size <MAX_FILE_SIZE>
Maximum file size to download in bytes (default: 50MB)
[env: WEBFANG_MAX_FILE_SIZE=]
[default: 52428800]
--download-timeout <DOWNLOAD_TIMEOUT>
Timeout for individual asset downloads in seconds (minimum 1)
[env: WEBFANG_DOWNLOAD_TIMEOUT=]
[default: 30]
HTTP Client Settings:
--max-retries <MAX_RETRIES>
Maximum number of retry attempts
[env: WEBFANG_MAX_RETRIES=]
[default: 3]
--backoff-base-ms <BACKOFF_BASE_MS>
Base delay for exponential backoff (ms)
[env: WEBFANG_BACKOFF_BASE_MS=]
[default: 1000]
--backoff-max-ms <BACKOFF_MAX_MS>
Maximum delay for exponential backoff (ms)
[env: WEBFANG_BACKOFF_MAX_MS=]
[default: 10000]
--accept-language <ACCEPT_LANGUAGE>
Accept-Language header value
[env: WEBFANG_ACCEPT_LANGUAGE=]
[default: en-US,en;q=0.9]
--user-agent <USER_AGENT>
Custom User-Agent header value (overrides Chrome 145 default)
[env: WEBFANG_USER_AGENT=]
-H, --header <NAME: VALUE>
Inject a custom HTTP header as `Name: Value` (repeatable).
Overrides any default header with the same (case-insensitive) name. Example: `-H "Authorization: Bearer TOKEN"`.
[env: WEBFANG_HEADER=]
--cookie <NAME=VALUE>
Inject a custom cookie as `name=value` (repeatable).
Seeded into the cookie jar before the first request so authenticated crawls work without a prior login round-trip. Example: `--cookie "session=abc123"`.
[env: WEBFANG_COOKIE=]
Sitemap Settings:
--sitemap-depth <SITEMAP_DEPTH>
Maximum recursion depth for sitemap indexes
[env: WEBFANG_SITEMAP_DEPTH=]
[default: 3]
Competitive Features:
--checkpoint-interval <CHECKPOINT_INTERVAL>
Pages between automatic checkpoint saves (0 = disabled) — unified via PersistenceMode with --resume (Checkpoint/Full when enabled)
[env: WEBFANG_CHECKPOINT_INTERVAL=]
[default: 100]
--no-checkpoint
Disable checkpoint persistence entirely — PersistenceMode disables checkpoint (Resume only when combined with --resume)
[env: WEBFANG_NO_CHECKPOINT=]
--ignore-robots
Skip robots.txt enforcement
[env: WEBFANG_IGNORE_ROBOTS=]
--ignore-waf
Skip WAF/CAPTCHA classification (REQ-WAF-07): challenged responses are reported as plain HTTP errors instead of WAF blocks. It does not rescue the fetch — a challenge page is never scraped as content (F-11).
[env: WEBFANG_IGNORE_WAF=]
--autoscale
Enable autoscaled concurrency — dynamically adjusts task concurrency based on RAM usage
[env: WEBFANG_AUTOSCALE=]
--no-session-health
Disable session pool health checks
[env: WEBFANG_NO_SESSION_HEALTH=]
--h2-profile <H2_PROFILE>
TLS/HTTP2 profile name (default: Chrome145)
[env: WEBFANG_H2_PROFILE=]
[default: Chrome145]
JS Rendering:
--js-strategy <JS_STRATEGY>
JavaScript rendering strategy: static (wreq only), hybrid (3-layer), full (Chromiumoxide only)
Possible values:
- static: Static HTTP only (wreq). Fastest, no JS rendering
- hybrid: Hybrid 3-layer: wreq → Obscura → Chromiumoxide
- full: Full JS rendering only (Chromiumoxide). Slowest, handles all SPAs
[env: WEBFANG_JS_STRATEGY=]
[default: static]
--js-wait <JS_WAIT>
Post-load settlement wait for the chromium render path: idle (network-idle, default), <ms> fixed wait (1..=30000), none (capture immediately)
[env: WEBFANG_JS_WAIT=]
[default: idle]
--obscura-binary <OBSCURA_BINARY>
Path to the obscura binary (default: "obscura")
[env: WEBFANG_OBSCURA_BINARY=]
[default: obscura]
Cleanup:
--dom-preprune [<DOM_PREPRUNE>]
Enable DOM pre-pruning before Readability (removes invisible/empty wrappers). Default: enabled (true). Set to false via --dom-preprune=false or WEBFANG_DOM_PREPRUNE=false
[env: WEBFANG_DOM_PREPRUNE=]
[default: true]
[possible values: true, false]
Output:
-o, --output <OUTPUT>
Output directory for scraped content
[env: WEBFANG_OUTPUT=]
[default: output]
-f, --content-format <FORMAT>
Output format for individual files (markdown, text, json) NOTE: For RAG pipeline export, use --pipeline-format instead
Possible values:
- markdown: Markdown format with YAML frontmatter (recommended for RAG)
- json: Structured JSON with metadata
- text: Plain text without formatting
[env: WEBFANG_FORMAT=]
[default: markdown]
--pipeline-format <EXPORT_FORMAT>
Export format for RAG pipeline (jsonl, vector, auto) NOTE: Use --content-format for output file format (markdown, text, json)
Possible values:
- jsonl: JSONL format (JSON Lines - one JSON object per line) Optimal for RAG pipelines and vector database ingestion
- vector: Vector format (JSON with metadata header) Supports embeddings and cosine similarity
- auto: Auto-detect format from existing export files
[env: WEBFANG_EXPORT_FORMAT=]
[default: jsonl]
--retention-days <RETENTION_DAYS>
Prune exports and vault-DB rows older than N days at the end of a successful run (0 = disabled)
[env: WEBFANG_RETENTION_DAYS=]
[default: 0]
Elastic Ingestion:
--cpu-cores <CPU_CORES>
CPU core override for the elastic ingestion Rayon pool (else auto-detect)
[env: WEBFANG_CPU_CORES=]
--ram-budget <RAM_BUDGET>
RAM budget override for the byte-weighted semaphore (`8GB`, `2048MB`, or bytes)
[env: WEBFANG_RAM_BUDGET=]
--db-path <DB_PATH>
SQLite database path override for persisted resources/chunks
[env: WEBFANG_DB_PATH=]
--elastic
Enable elastic ingestion pipeline (streaming, SQLite dedup, Rayon CPU bridge)
[env: WEBFANG_ELASTIC=]
--output-vectors <OUTPUT_VECTORS>
Write extracted vectors to a JSONL file for RAG pipelines. Use `-` for stdout. No SQLite dependency — available in every build (core binary too)
[env: WEBFANG_OUTPUT_VECTORS=]
Batch Processing:
--batch
Enable batch mode — scrape each URL from stdin (one per line), one page per URL, no crawling
[env: WEBFANG_BATCH=]
--batch-file <BATCH_FILE>
Path to a file containing URLs to scrape (one per line), one page per URL, no crawling
[env: WEBFANG_BATCH_FILE=]
--batch-concurrency <BATCH_CONCURRENCY>
Maximum concurrent URLs in batch mode (omit = auto from budget model)
[env: WEBFANG_BATCH_CONCURRENCY=]
--export-checkpoint <EXPORT_CHECKPOINT>
Flush captured pages to markdown every N pages during a batch (0 = only at the clean close)
[env: WEBFANG_EXPORT_CHECKPOINT=]
[default: 0]
Item Pipeline:
--pipeline
Enable item pipeline processing (validate → clean → output)
[env: WEBFANG_PIPELINE=]
--pipeline-output <PIPELINE_OUTPUT>
Pipeline output format: jsonl (default), none
Possible values:
- jsonl: Write items as JSON Lines to a file (default)
- none: No pipeline output — items are processed but not written
[env: WEBFANG_PIPELINE_OUTPUT=]
[default: jsonl]
Obsidian:
--obsidian-wiki-links
Convert same-domain links to Obsidian [[wiki-link]] syntax
[env: WEBFANG_OBSIDIAN_WIKI_LINKS=]
--obsidian-tags <OBSIDIAN_TAGS>
Tags to include in YAML frontmatter (comma-separated)
[env: WEBFANG_OBSIDIAN_TAGS=]
--obsidian-relative-assets
Rewrite downloaded asset paths as relative to the .md file
[env: WEBFANG_OBSIDIAN_RELATIVE_ASSETS=]
--vault <VAULT>
Path to Obsidian vault (auto-detects if not provided).
When provided explicitly, the vault becomes the output base: Markdown,
downloaded assets and the RAG export are written inside it — no need
to duplicate the path in `-o` (which then must stay at its default).
Auto-detected or config-file vaults do NOT redirect output (#762).
[env: WEBFANG_OBSIDIAN_VAULT=]
--quick-save
Quick-save mode: save directly to vault _inbox folder
[env: WEBFANG_OBSIDIAN_QUICK_SAVE=]
--obsidian-rich-metadata
Add rich metadata to frontmatter
[env: WEBFANG_OBSIDIAN_RICH_METADATA=]
AI Settings:
--threshold <THRESHOLD>
Relevance threshold for AI semantic filtering (0.0-1.0)
[env: WEBFANG_THRESHOLD=]
[default: 0.3]
--max-chars <MAX_CHARS>
Maximum characters per chunk before rejection, >= 1 (a chunk-size guard, not a context-window setting; the effective ceiling is the embedding backend's own limit, which a remote endpoint enforces server-side as HTTP 400/413)
[env: WEBFANG_MAX_CHARS=]
[default: 98304]
--offline
Reject every embeddings provider (remote or loopback) with exit 78: no local model since 3.0.0
[env: WEBFANG_OFFLINE=]
LLM Extraction:
--extract-with-llm
Run structured extraction through the configured LLM provider (requires a provider with the `completion` capability; fails at startup otherwise)
[env: WEBFANG_EXTRACT_WITH_LLM=]
--llm-provider <LLM_PROVIDER>
Provider id to use for LLM extraction (default: first provider declaring the `completion` capability)
[env: WEBFANG_LLM_PROVIDER=]
--embedding-provider <EMBEDDING_PROVIDER>
Provider id to use for embeddings (default: first provider declaring the `embedding` capability)
[env: WEBFANG_EMBEDDING_PROVIDER=]
Tier 2 Embeddings:
--tier2-embedding-provider <TIER2_EMBEDDING_PROVIDER>
Provider id to use for Tier 2 semantic inspector embeddings (default: none; the vault-search provider is never adopted — a remote id requires --tier2-consent)
[env: WEBFANG_TIER2_EMBEDDING_PROVIDER=]
--tier2-consent
Explicit consent for Tier 2 remote embeddings egress (required when --tier2-embedding-provider is remote; hash of provider|endpoint logged for audit)
[env: WEBFANG_TIER2_CONSENT=]
EXIT CODES:
0 Success
2 No URLs discovered
3 All scrapers failed
64 Bad CLI arguments (usage error)
65 Data format error (extraction without usable content)
69 Network error, partial success, or SSRF refusal
74 I/O error
76 Protocol error
77 All URLs blocked by robots.txt
78 Configuration error
EXAMPLES:
webfang https://example.com
webfang -u https://example.com
webfang -u https://example.com --ai
webfang -u https://example.com -f jsonl
webfang -u https://example.com -v
webfang -u https://example.com -vv # DEBUG
webfang --batch-file urls.txt --resume
$ webfang completions --help
Generate shell completion scripts
Usage: webfang completions <SHELL>
Arguments:
<SHELL>
Shell to generate completions for
Possible values:
- bash: Bash shell completions
- elvish: Elvish shell completions
- fish: Fish shell completions
- power-shell: PowerShell completions
- zsh: Zsh shell completions
Options:
-h, --help
Print help (see a summary with '-h')
Layer 2 (Obscura): a cheap render, not an anti-WAF guarantee
--js-strategy hybrid fetches in three layers: L1 is wreq (static
HTTP), L2 is the obscura subprocess, L3 is Chromiumoxide.
(--js-strategy full goes straight to L3 and is rejected at preflight by
the release binaries, which carry no chromium feature —
Release Feature Matrix.)
L2 is accepted as a cheap render — webfang does not require
obscura's -stealth build variant, so L2 carries no anti-WAF promise. That
is the decision recorded in #1710; this section is where it lives.
What L2 does
- Runs
obscura fetch --dump htmlonce per page — never markdown — with no cookies and no storage directory (the sessionless contract), and marks the result astext/html. - Costs ~30 MB per subprocess instead of the ~200 MB + CDP that Layer 3 pays: that is the whole reason it stays in the stack.
- Escalates to Layer 3 when the page comes back empty or the subprocess fails.
What L2 does not do
- It is not an anti-WAF guarantee. webfang never passes
-stealth, and a WAF challenge detected at any layer aborts the fetch: the router never escalates against a challenge.
Version floor. The hybrid preflight requires obscura 0.2.5 or newer
(MINIMUM_OBSCURA_VERSION in crates/webfang_core/src/cli/preflight.rs).
An older binary fails the preflight with exit 78 (configuration error),
naming the version it found and the version it requires. A binary whose
--version probe fails or prints something unparseable is not blocked:
the preflight degrades to a warning and the run continues best-effort.
Reading the L2 hit-rate. There are no counters — the hit-rate is derived from the per-layer log events the hybrid router already emits. With a trace file, count the layer events:
webfang https://example.com --js-strategy hybrid --trace-file run.jsonl
# L2 attempts
jq -r 'select(.message | test("Layer 2 \\(Obscura\\): attempting fetch")) | .message' run.jsonl | wc -l
# L2 hits (the render came back with content)
jq -r 'select(.message | test("^Layer 2 returned [0-9]+ bytes$")) | .message' run.jsonl | wc -l
# escalations to L3
jq -r 'select(.message | test("Layer 3 \\(Chromiumoxide\\): attempting fetch")) | .message' run.jsonl | wc -l
The trace file records every level independently of -v (the file layer runs
at TRACE level for the webfang target), so the layer events are in it even
without -vv. The rest of the query cookbook is in
debugging.md.
The four consequences of the decision, recorded so the promise is not re-invented later:
- The L2 hit-rate is derived from the per-layer logs — no new counters, and no strict mode.
- The preflight's
Unknownverdict (an unreadable--version) degrades to a warning instead of failing the run: an unreadable version never blocks the crawl. - The floor in force is 0.2.5. The
0.2.3ask in #1710 is stale: the upstream release moved on, and the floor follows the current stable. - #1710 stays open. It is commented with this status, not closed.
State directory contents
Companion note to --state-dir (documented in the generated block above).
Each --resume run keeps one JSON state file per domain plus a
<...>.json.lock sentinel file next to it. The sentinel is created on
demand and intentionally never removed: its presence does NOT mean a
writer is alive. The real mutual exclusion is the advisory flock held on
the sentinel, which the kernel releases when the process exits — including
after SIGKILL. There is exactly one sentinel per state file, so the leftover
is bounded. Do not delete the .lock file while a run holds it: removing
the path breaks the lock and lets two writers enter the critical section at
once, losing state updates.
Migración desde ONNX local
ONNX local ya no existe en el producto (ADR-0004, paso 6). Esta página dice qué cambió, a quién le afectó y qué hacer ahora, en el orden de los pasos. La decisión y su secuencia están en ADR 0004.
Qué cambió y cuándo
| Momento | Qué pasó |
|---|---|
| Aviso (2 minors antes del corte) | Se anunció. No se borró nada: los shims seguían funcionando y --clean-ai seguía usando ONNX local |
| Ventana de migración | El --embedding-provider resolvió el slot de vault search mientras el ONNX seguía vivo |
| Corte (paso 6 del ADR-0004) | ONNX local, el crate webfang_ai y la feature ai se eliminaron. Este es el estado actual |
| Retiro de los shims | Los shims de migración se eliminaron: ya no parsean. Un flag retirado responde con el error estándar de clap (unexpected argument, exit 64) |
cargo build --features ai falla ahora en la resolución de features: la feature desapareció, y no
deja un shim detrás. Un shim detrás de la feature que se borra no existe, en ningún lenguaje.
A quién le afecta
| Tu setup | Qué te pasa ahora |
|---|---|
--clean-ai con el modelo local (Granite 97M / 311M) | Te afecta. El cleaner resuelve su endpoint por --embedding-provider. Perdiste el offline total |
--clean-ai + --embedding-provider ya configurado | Sirve hoy: el flag mueve el slot de vault search y alimenta el cleaner vía el EmbeddingPort des-gateado |
--ai-model, AI_MODEL_ID, WEBFANG_AI_MODEL_ID, WEBFANG_AI_ENGINE | Retirados. Ya no son flags: --ai-model responde con el error estándar de clap (unexpected argument, exit 64) y las env se ignoran en silencio. Usá --embedding-provider / WEBFANG_EMBEDDING_PROVIDER |
--max-tokens, WEBFANG_MAX_TOKENS | Retirados. --max-tokens ya no parsea (error estándar de clap, exit 64) y WEBFANG_MAX_TOKENS se ignora. Usá --max-chars / WEBFANG_MAX_CHARS |
| Vaults con vectores de Granite | Hay que re-indexar: el endpoint nuevo no produce vectores comparables. El coste (y el BYO-key) lo pagás vos |
| No usás la superficie AI | Nada. Los avisos no aparecen en corridas que no la tocan |
Cómo migrar
-
Configurá el endpoint remoto — detecta y valida, y escribe la config:
webfang connectComprobá que quedó bien, con el detalle de cada valor:
webfang doctor webfang show -
Pasá
--clean-aial endpoint remoto — disponible desde el corte.El cleaner resuelve su
EmbeddingPortpor--embedding-provider(o su envWEBFANG_EMBEDDING_PROVIDER; el flag gana al env):webfang --url https://example.com --clean-ai --embedding-provider ollamaEn
--offlineno hay provider de embeddings que sirva: el guard rechaza cualquier endpoint —uno remoto público o uno en127.0.0.1— y sale con exit 78. El "loopback permitido" que decía el aviso del paso 5 se refería al modelo local ONNX, que este paso eliminó; el aviso ya no lo dice y ahora nombra la regla real (rechaza cualquier provider de embeddings, remoto o loopback). Sin ese camino,--offlinesolo sirve para corridas sin limpieza semántica. Un endpoint LAN sigue necesitando la allowlist por CIDR (allow_cidrs), que no incluye100.64.0.0/10salvo opt-in explícito. -
Re-indexá los vaults que tengan vectores de Granite. No hay forma de detectar la mezcla: el fail-closed por dimensión no puede dispararse sobre los vaults que nunca tuvieron
model_id, que son justamente los afectados. Vas a ver un aviso de "legado"; re-indexar es lo que lo cierra. -
Renombrá
--max-tokens→--max-chars. El shim se retiró:--max-tokensya no parsea yWEBFANG_MAX_TOKENSse ignora en silencio.
Qué sigue funcionando
--clean-ai y --offline empezaron como flags activos siempre en el corte: se des-gatearon,
quedaron en el build por defecto, sin feature. --clean-ai cambió de semántica (ONNX → endpoint
remoto); no fue un shim, es el mismo flag haciendo el trabajo con otro backend.
El umbral 0.3
El 0.3 por defecto estuvo calibrado para Granite. Retirado el único modelo calibrado, un endpoint
remoto retiene distinto con el mismo número, así que la corrida avisa una vez mientras el umbral sea
el default. El aviso no es decoración: es la señal de que el número ya no significa lo que decía.
Elegí un umbral según tu modelo:
webfang --url https://example.com --clean-ai --embedding-provider ollama --threshold 0.45
El valor efectivo de cada corrida aparece como threshold_used en el summary.
Si recién necesitás ONNX local
El borrado quedó reversible desde una etiqueta git de recuperación creada antes del borrado:
tag onnx-tree/pre-step5 (commit 38ab6a31). Recuperar ese árbol es el único camino oficial hacia
el ONNX de origen; desde ahí, un resurgimiento requiere un ADR nuevo — no es un rollback ni queda
plementado en la línea actual.
Referencias: CLI reference · ADR 0004
MCP error contract — which channel carries which failure
Status: published decision table for issue #1613. Describes the server as it stands on top
of commit ccdef614 plus the uncommitted working-tree changes that converted the last 16
reason-less rejection sites. Commit ccdef614 is the state after code slices 2–5 landed
(EC-02 … EC-08) and closed EC-08's machine-readable half everywhere it was originally scoped;
the working tree closes it everywhere else. This page states what the server does, not what
it should do; a future change that invalidates a row must change this page in the same commit.
Evidence class: mixed, and the split is deliberate. Rows A1, A2, A3, A6, A7, A8, A9
and C1 are observed over a real transport — they are asserted on the wire in
tests/mcp_error_channel_mapping_test.rs (A1, A2, A3),
tests/mcp_ssrf_error_class_test.rs (A6–A9) and
tests/mcp_validation_reason_code_test.rs (C1 and the data shape). Every other row is
still statically derived: read out of the source, not exercised end-to-end. Rows A4, A5,
A10–A19, B1–B8, C2–C4 and T1–T9 are in that second group. (A5 is the only SSRF row not
covered end-to-end: no_host is unreachable from outside, which is exactly what its row
says.)
Slug evidence is narrower than row evidence, and the two must not be conflated. The
wire-level slug assertions cover exactly three validation slugs (unsupported_scheme,
too_long, out_of_range in tests/mcp_validation_reason_code_test.rs) plus three SSRF
slugs (forbidden_ip_literal, forbidden_ip_resolved, dns_resolution_failed in
tests/mcp_ssrf_error_class_test.rs). The A4.x and A10–A14 slugs are not transport-observed:
A4.x is pinned by the in-crate unit test
path_gate::every_rejection_carries_field_and_reason_slug, which samples every branch of
confine, and the A10–A14 slugs by construction — the reason-less constructor those sites
used to call has been deleted, and no bare-string data producer remains anywhere in the
crate (see the appendix). Nothing here is inferred
from a hypothetical transport any more; see
Unsettled rows for the one question that changed evidence class in the
other direction and the one deliberate non-action.
Quick path for an agent consumer
HTTP status != 200 → Channel 0. Transport refused the request. Fix the request shape.
JSON-RPC "error" member present → Channel A. The call never ran. Fix the arguments.
result.isError == true → Channel B. The call ran and failed. The text is the reason.
result without isError → Channel C. The call ran. Read the body: it may be a
diagnostic, a partial result, or a success.
Two things that table does not say, and that bite:
- A malformed argument does not always reach Channel A. A deserialization rejection
(rows A1, A2) arrives as
isError: true— Channel B. Only a rejection raised by the handler body (row A3) is a JSON-RPC-32602. The type is the same on both routes; the wire shape is what differs. See A1–A3. - Channels A and B overlap textually. A
isError: truebody can contain the literal string-32602. Never discriminate on the code appearing in the text — see Channel overlap.
"Retriable" in this table means: repeating the identical call can succeed without the caller changing anything. Anything else is a caller change, not a retry.
The three channels (plus the one before them)
| Channel | Wire shape | HTTP | Carries | Code |
|---|---|---|---|---|
| 0 — transport | bare status, usually no JSON-RPC body (T9 is the exception) | 401/406/408/413/415/422/429/500 | a request that never reached a tool | none (T9 adds a -32603 body) |
| A — protocol error | {"error": {"code": …, "message": …, "data": …}} | 200 | the call was unroutable, or its arguments were refused by the handler body | -32601 / -32602 / -32603 |
| B — tool error | result.isError = true | 200 | the tool ran and the job failed — or its arguments never deserialized | none |
| C — success-shaped diagnostic | result, no isError | 200 | the tool ran; the body describes a non-success outcome | none |
Channel A is the only channel with a machine-readable code and a machine-readable
data.reason slug (see the reason taxonomy).
Channel B carries no code: the only machine-readable signal is the boolean, and the
reason is free text. Channel C has the same gap, which is the whole reason it is listed
separately — see row C1.
Channel overlap — the text is not the discriminator
A Channel B body can literally contain
-32602. Two independent facts combine:
McpError::invalid_params(msg, data)builds anrmcp::ErrorData, andErrorData'sDisplayrenders"{code}: {message}({data})"— the numeric code is part of the string form (rmcp-1.8.0/src/error.rs:8-16).- On the deserialization route, rmcp wraps any serde error as
"failed to deserialize parameters: {error}", so that renderedErrorData— code anddataincluded — is what lands in theisError: truebody.An A1/A2 rejection therefore reads, in the tool text, roughly
failed to deserialize parameters: -32602: …({"field":"url","reason":"malformed"})while carrying noerrormember at all. The only reliable discriminator is the wire shape: is there a top-levelerrormember, or isresult.isErrortrue? Branch on that, never on a code appearing in the text.This is also why the reason slug is the machine-readable half on this route: it is readable in the text (rmcp's
Displayappends thedataJSON verbatim) even though it is not delivered as structured data. Parsing it out of prose is a last resort, not the contract — rows A1/A2 give nodataa client can index.
Channel A — JSON-RPC protocol errors
Emitted when a tool function returns Err(McpError); McpHandler::call_tool
(mcp_server/mod.rs:359-382) forwards that Err unchanged to rmcp.
Two things about this channel are not obvious from the table, and both were settled by issue #1613 rather than assumed:
- It is not the channel every argument rejection takes. Only a rejection the handler body raises is Channel A — settled by observation in slice 3.
error.datais not a single shape. There are two live shapes, listed in the reason taxonomy: an object carrying bothfieldandreason, and an object carryingreasonalone. Every argument refusal carries a slug — the pre-EC-08 bare-string shape and the reason-less shape are both gone. The only Channel A rows still left with nodataat all are the-32603server faults (A15, A16), which never had a caller mistake to describe.
A1–A3 · argument deserialization (before the handler body)
| # | Condition | Code | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| A1 | URL argument is not http(s), is oversize, or carries embedded credentials | none — result.isError = true, no error member (observed) | No | Reshape the URL. Never retry unchanged. | mcp_server/params.rs:81-99 (McpUrl::try_from); observed in tests/mcp_error_channel_mapping_test.rs:99-127 |
| A2 | Unknown JSON key in the argument object (deny_unknown_fields) | none — result.isError = true, no error member (observed) | No | Drop the unknown key. | mcp_server/params.rs:7 (per-struct attribute, first at :316); observed in tests/mcp_error_channel_mapping_test.rs:139-166 |
| A3 | In-handler params.validate()? semantic rejection | -32602 (observed) | No | Fix the named field; data.field names it. | every validate() in mcp_server/params.rs (e.g. 461-475); observed in tests/mcp_error_channel_mapping_test.rs:196-217 |
A1 and A3 produce the same Rust error type (McpError::invalid_params is
rmcp::ErrorData) but travel different routes and land on different channels: A1
is raised inside rmcp's Parameters<P> deserialization and arrives as
result.isError = true; A3 is returned from the tool function and arrives as
error.code = -32602 with no result member. That was the open question under
Unsettled rows; it is now settled by test, and the answer is that
they do not reach the client as the same code.
The mechanism is worth knowing, because it is why McpUrl::try_from does not have to fake
anything. rmcp 1.8.0's Parameters<P> extractor wraps any serde error with its own
"failed to deserialize parameters: " prefix
(rmcp-1.8.0/src/handler/server/tool.rs:181-196), and into_tool_argument_error
downgrades to isError: true only when the code is INVALID_PARAMS and that prefix is
present (rmcp-1.8.0/src/handler/server/router/tool.rs:144-156). A handler-raised
rejection never carries the prefix, so it is never downgraded. One shared Rust type, two
routes, two channels.
A3 covers every branch below. They share one constructor
(validation::invalid_params_with_reason), so they share one channel, one code, and one
data shape — the split is by condition, not by envelope.
| # | A3 sub-condition | Field | data.reason | Retriable | Agent should | Source |
|---|---|---|---|---|---|---|
| A3.1 | URL empty, > 8192 B, unparseable, or non-http(s) scheme | url / varies | empty / too_long / malformed / unsupported_scheme | No | Resend a valid absolute http(s) URL. | mcp_server/validation.rs:137-174 |
| A3.2 | Path empty, > 1024 B, absolute, drive-lettered, ..-bearing, or a rejected path component | varies | empty / too_long / path_not_allowed | No | Send a relative, traversal-free path. | mcp_server/validation.rs:176-237, 333-374 |
| A3.3 | HTML / markdown / content blob > 1 MiB | varies | too_long | No | Chunk the input. | mcp_server/validation.rs:41, 376-393 |
| A3.4 | max_pages outside 1..=100 000, max_depth outside 0..=10 | max_pages / max_depth | out_of_range | No | Clamp to the advertised cap. | mcp_server/params.rs:152-183 |
| A3.5 | urls empty or over cap, concurrency out of range | urls / concurrency | empty / out_of_range | No | Fix the batch. | mcp_server/params.rs:461-485, mcp_server/validation.rs:555-618 |
| A3.6 | js_strategy not a known strategy | js_strategy | not_in_allowed_set | No | Use static / hybrid / full. | mcp_server/params.rs:526-532 |
| A3.7 | checkpoint_dir present but blank | checkpoint_dir | empty | No | Omit it or name a real directory. | mcp_server/params.rs:533-540 |
| A3.8 | Enum-ish string not in the allowed set | varies | not_in_allowed_set | No | Use a listed value; the message enumerates them. | mcp_server/validation.rs:751-762 |
| A3.9 | Filename not a single safe component (traversal, separator, reserved name, Windows-invalid char) | filename | empty / too_long / path_not_allowed | No | Send one flat component. | mcp_server/validation.rs:620-733, 287-331 |
Since slice 5, error.data on every row above is the structured object
{"field": "<field>", "reason": "<slug>"}. Before that it was a bare JSON string
holding only the field tag, so a client could tell which field was wrong but never why,
and had to parse prose. The field survived the restructure — it is now data.field — and
data.reason is the new machine-readable half. Both are contract: see
the reason taxonomy.
A4 · path confinement (root-of-trust)
| # | Condition | Code | data.reason | Retriable | Agent should | Source |
|---|---|---|---|---|---|---|
| A4.1 | Empty path | -32602 | empty | No | Name a directory. | mcp_server/path_gate.rs:281-286 via mcp_server/state.rs:332, 352 |
| A4.2 | Path over MAX_PATH_LEN (1024 B) | -32602 | too_long | No | Shorten the path. | mcp_server/path_gate.rs:288-293 |
| A4.3 | Rooted-but-not-absolute form (C:foo, \foo) | -32602 | path_not_allowed | No | Use a genuinely absolute or genuinely relative path. | mcp_server/path_gate.rs:296-303 |
| A4.4 | .. traversal component | -32602 | path_not_allowed | No | Send a traversal-free path. | mcp_server/path_gate.rs:308-313 |
| A4.5 | Absolute path with no export roots configured | -32602 | path_not_allowed | No | Use a relative path, or tell the operator to set --export-roots. | mcp_server/path_gate.rs:321-334 via mcp_server/state.rs:332, 352 |
| A4.6 | Absolute path outside every configured export root | -32602 | path_not_allowed | No | Re-target under a configured root. | mcp_server/path_gate.rs:340-348 |
The gate is fail-closed: with zero roots configured, every absolute path is refused
(mcp_server/path_gate.rs:321). Relative paths are allowed and resolve against the
server's CWD (mcp_server/path_gate.rs:315-319).
All six branches now emit the full {"field": "output_dir" | "checkpoint_dir", "reason": <slug>} envelope — there is no reason-less shape left. Only A4.1 and A4.2 differ from the
shared path_not_allowed: their remedy is fill it in and shorten it, not use a
different path, and a slug that sends the agent down the wrong remedy is worse than no
slug at all. The four path_not_allowed branches are deliberately not split further —
"use a safe path inside an allowed root" is one instruction for all of them, and the
message carries which rule fired. Only the data payload moved: the gate's control flow,
messages, -32602 code and allow/deny decisions are byte-for-byte unchanged.
A5–A9 · SSRF entry pre-check
All five branches come from one function, validate_url_no_ssrf
(mcp_server/ssrf.rs:96-187). Since slice 2 they are separated into three classes,
and the split is expressed twice: in the JSON-RPC code, and in a data.reason slug.
| # | Condition | Class | Code | data.reason | Retriable | Agent should | Source |
|---|---|---|---|---|---|---|---|
| A5 | URL has no host | caller input | -32602 | no_host | No | Unreachable in practice — McpUrl always carries a host. | mcp_server/ssrf.rs:118-123 |
| A6 | Literal IP is in a forbidden range (loopback / private / link-local / CGNAT / ULA) | policy | -32602 | forbidden_ip_literal | No | Stop. Retargeting is the only fix. | mcp_server/ssrf.rs:130-142 |
| A7 | DNS resolver returned an error | infrastructure | -32603 | dns_resolution_failed | Yes | Retry — the argument is fine, the server's resolver is not. | mcp_server/ssrf.rs:148-159 |
| A8 | DNS returned an empty answer set | infrastructure | -32603 | dns_no_addresses | Yes | Retry — same class as A7. | mcp_server/ssrf.rs:161-170 |
| A9 | A resolved address is in a forbidden range | policy | -32602 | forbidden_ip_resolved | No | Stop. This is a policy refusal, not a transient. | mcp_server/ssrf.rs:172-185 |
A7 and A8 changed class in slice 2. They used to be -32602 with data: None,
identical to a caller aiming at internal infrastructure — the only way to tell a
transient DNS outage from an SSRF attempt was to parse Spanish message text. They are now
-32603 internal_error: the argument is correct and the server's resolver is not, which
is the same fault class this crate already reports that way (rows A15, A16). Retriability
follows the code: the two DNS rows are the only retriable rows in Channel A.
Policy and caller input still share -32602 on purpose — the knob-matrix suite pins the
loopback refusal at exactly that code, and the SSRF knob documentation advertises the
shape, so changing it would break pinned probes for no gain. That is why data.reason is
load-bearing here and not decoration: the slug, not the code, is what separates the
classes. The five SSRF slugs are a separate, closed set from the seven validation slugs,
and they carry no field (an SSRF refusal is not a bad field); see
the reason taxonomy.
This pre-check is not the enforcement point. Every scrape client is independently guarded
at connect time by webfang_core::domain::ssrf_guard (literal-IP redirect policy +
validating resolver), so A6/A9 are fast-fail UX and defense in depth. See
docs/ssrf-layers.md for the full layer matrix.
A10–A17 · handler-local protocol errors
| # | Condition | Code | data.reason | Retriable | Agent should | Source |
|---|---|---|---|---|---|---|
| A10 | content_format / pipeline_format not a known export format | -32602 | not_in_allowed_set | No | Use jsonl / vector / auto. | mcp_server/handlers/export.rs:196-202, 409-415 |
| A11 | filename rejected by the sanitizer, or unparseable inside the synthetic export URL | -32602 | path_not_allowed | No | Send one flat component. | mcp_server/handlers/export.rs:209-215, 234-240, 324-331, 362-369, 469-475 |
| A12 | Synthetic export URL for the filename fails the hardened URL gate | -32602 | malformed | No | Reshape the filename. | mcp_server/handlers/export.rs:244-257 |
| A13 | js_strategy failed to parse in the handler (post-validate path) | -32602 | not_in_allowed_set | No | Use a known strategy. | mcp_server/handlers/scraping.rs:457-463 |
| A14 | Explicit sitemap_url fails the hardened URL re-wrap | -32602 | unsupported_scheme | No | Reshape the sitemap URL. | mcp_server/handlers/scraping.rs:599-610 |
| A15 | Per-category semaphore closed or poisoned | -32603 | none — data is null | No | Report it; the server is unhealthy. | mcp_server/macros.rs:16 |
| A16 | Record conversion failed: DocumentChunk::validate() or metadata serialization | -32603 | none — data is null | No | Report it; the payload was not produced. | mcp_server/handlers/scraping.rs:106, 109, 212, 215, 1121, 1127 |
| A17 | SSRF pre-check failure on the axtree tool (re-raised unchanged) | as A5–A9 | as A5–A9 | as A5–A9 | As A5–A9 — read data.reason, the code is the SSRF channel's. | mcp_server/handlers/axtree.rs:66-74 |
A15, A16 and the two SSRF DNS branches (A7, A8) are the only -32603 values this crate
constructs for a tool call. They are server faults, not caller mistakes, and must not be
retried — which is also why they are the only Channel A rows with no data at all:
there is nothing for the caller to branch on. A17 is a forwarding site, not a construction:
it re-raises whatever validate_url_no_ssrf produced, so its code and slug are inherited.
Two slug choices in this table are not mechanical, and both are deliberate. A12 reports
malformed, not unsupported_scheme: the scheme there is server-built
(https://webfang.local/<filename>), so unsupported_scheme would describe a URL the
caller never sent and hand it unusable advice; try_from_url also collapses "unparseable"
and "not http(s)" into one error, exactly as the McpUrl boundary does. A14 is the mirror
image and reports unsupported_scheme: there the value was already parsed by McpUrl, so
the scheme allow-list is the only reachable cause and the precise slug is the true one.
A18–A19 · produced by rmcp, not by this crate
| # | Condition | Code | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| A18 | Unknown JSON-RPC method (e.g. a tool name that is not registered) | -32601 | No | Call a name from tools/list. | rmcp; not constructed here — mcp_server/server.rs:166-177, pinned by tests/mcp_transport_contract_test.rs:138 |
| A19 | A panic escapes the HTTP request path | -32603 body, HTTP 500 | No | Report it. The body echoes the JSON-RPC id the client sent whenever one can be recovered (see the rules below). The tool-level panic path (B1) is the one that keeps the session alive. | mcp_server/panic_containment.rs (jsonrpc_internal_error_response), outermost backstop in mcp_server/server.rs (jsonrpc_panic_response) |
Which id a contained panic echoes
The -32603 body always carries an id, and what it carries is a stated
rule, not a best effort. Both containment layers share one response builder, so
the shape never depends on which layer caught the panic.
id echoed | when | rule id |
|---|---|---|
the parsed id member | POST body is an object with an id member | envelope |
null | the object closed with no id member (a JSON-RPC notification) | notification |
null | the object sent "id": null explicitly | null_id |
null | the body is a top-level array (a JSON-RPC batch) | batch |
null | non-JSON, empty, a bare scalar, a non-POST request, or an unreadable body | not_a_request |
null | the envelope did not close inside the 64 KiB scan window | scan_window |
A null id is recorded on the jsonrpc.id_source span field
(mcp_jsonrpc_panic_containment), so "the client sent a batch" is
distinguishable from "the body was garbage" in the trace. null never means
the mapping failed: the code is -32603 either way. Source of truth:
crates/webfang_mcp/src/mcp_server/panic_containment.rs.
The reason-slug taxonomy — the machine-readable contract
data.reason is the half of this contract an agent branches on. Before slice 5, Channel A
told a caller which field was wrong and nothing about why, so every consumer parsed
prose. Since EC-08 the data member is an object and reason is one of a closed set of
stable slugs, declared as a contract in mcp_server/validation.rs:64-99. There is
deliberately no reason-less constructor left in the crate: the
invalid_params(field, msg) escape hatch that path_gate.rs and the handler-inline sites
used to call was deleted once nothing called it any more, and a caller that cannot name a
slug must now pick the closest one (mcp_server/validation.rs:101-104). That is what makes
the taxonomy exhaustive — a consumer branching on data.reason can rely on it being there.
The seven validation slugs
| Slug | Means | What the caller should do |
|---|---|---|
empty | a required value is empty | fill it in |
too_long | the value exceeds a byte cap | shorten it |
malformed | present but unparseable | fix its shape |
unsupported_scheme | the URL scheme is not http/https | use http or https |
path_not_allowed | any structural path rejection — the filename rules (absolute, UNC prefix, drive letter, .., non-flat filename, Windows reserved name, Windows-invalid character, trailing ./space) and the export-root confinement rules in path_gate.rs (rooted-not-absolute, .., absolute with no roots, absolute outside every root) | use a safe relative path and a plain filename |
out_of_range | a numeric value is outside its inclusive bounds | clamp it |
not_in_allowed_set | an enum-ish value is not in the allowed list | pick a listed value |
Coarse is deliberate, and the set is load-bearing. A slug's job is to tell the agent
what to do, not to restate the message; seven stable values beat forty brittle ones.
Downstream tooling branches on these strings, so do not rename one, add an eighth, or
remap a condition to a different slug without changing this page in the same change.
path_not_allowed deliberately collapses those twelve rules into one slug, because
their remedy is one instruction — use a safe path — and the message carries the specifics.
The SSRF slugs
A separate, disjoint set — the SSRF pre-check classifies a target, not a request field,
so it carries no field at all:
| Slug | Class | Code |
|---|---|---|
no_host | caller input | -32602 |
forbidden_ip_literal | policy | -32602 |
forbidden_ip_resolved | policy | -32602 |
dns_resolution_failed | infrastructure | -32603 |
dns_no_addresses | infrastructure | -32603 |
The data shapes a client must tolerate
| Shape | Emitted by | Reader |
|---|---|---|
{"field": "<field>", "reason": "<slug>"} | every field-bearing rejection: the 44-branch validation.rs funnel, the six inline bounds in params.rs (A1, A3.x), the six path_gate.rs branches (A4.x), and the ten handler-inline sites in handlers/export.rs / handlers/scraping.rs (A10–A14) | index field and reason |
{"reason": "<slug>"} — no field | ssrf.rs, 3 branches (A5, A6, A9) | index reason; field is legitimately absent |
That is the whole of what the server produces. A client may still meet a bare string or a
data with no reason in the wild, because error.data is sender-defined and
validation::reason_of deliberately tolerates all three cases (data absent, a non-object,
or an object with no reason key) rather than panicking
(mcp_server/validation.rs:122-124, pinned by
reason_of_tolerates_a_missing_or_foreign_data_payload). Keep that tolerance in the reader;
do not document either as a shape this server can emit.
validate_url's parallel channel
The one success-shaped diagnostic that reports a rejection reuses the same slug under a
different key, because it is not a JSON-RPC error and has no data: the body is
{"valid": false, "reason": <message only>, "reason_code": <slug>} (row C1). reason is
now the message alone; it used to be ErrorData::to_string(), which leaked the numeric
code and a raw data blob into a human-facing field. reason_code carries the slug, so
one taxonomy covers both channels. The valid: true field set is unchanged.
Message language — the ratified (inconsistent) contract
The repo-wide rule is "user-facing errors in Spanish, internal logs in English". The MCP error channels do not follow it uniformly, and that is now a documented state rather than an accident to be papered over.
| Channel | Language as it actually is |
|---|---|
| A — structural / validation | English in validation.rs (every require_* message) and in path_gate.rs; Spanish in one deliberate island, validation::filename_component_error (mcp_server/validation.rs:291-323 — all six of its messages), and also in ssrf.rs, handlers/export.rs, and the js_strategy / checkpoint_dir / sitemap_url messages in params.rs and handlers/scraping.rs |
| B — operational | Spanish by intent, with a few English literals surviving: "no valid URLs provided" (mcp_server/handlers/scraping.rs:264) and the three "HTTP error: …" messages (mcp_server/handlers/scraping.rs:780-783, 852, 1019). Messages that forward an upstream Error::to_string() are language-dependent on their source. |
| C — success-shaped | mixed: "no vault detected" and "not an SPA - sufficient content found" are English, and validate_url's failure reason is whatever require_http_url produced — also English. |
Unifying this was deliberately not done in issue #1613. The reason is that it would
buy nothing for a programmatic consumer: data.reason makes the branch language-free,
and the message text is explicitly not part of the contract. Rewriting the message
strings behind these rows would churn the diff of a contract change, put the language split
in the path of existing message-matching probes, and change nothing a client can rely on.
(How many strings that is was deliberately not counted: an exact figure would be a
snapshot, and this page is about a contract, not a census.)
This is a deviation from the repo-wide convention, recorded here so a maintainer can revisit it. The honest framing: the convention says these strings are user-facing, and they are read by an autonomous agent rather than a person, so "user-facing" is arguably the wrong lens — but that is an argument, not a decision, and the decision belongs to the maintainer. If it is revisited, the taxonomy section is the anchor: only the slugs are frozen, so translating a message is a compatible change while renaming a slug is not.
Channel B — isError: true tool results
Every row here is built by provenance::neutralized_error
(mcp_server/provenance.rs:275-294) or by the honest_error wrappers
(mcp_server/handlers/ai.rs:38-42, mcp_server/handlers/axtree.rs:25-28). Text is
neutralized (ANSI + C0/DEL stripped, fence sentinels escaped) and capped, but no
provenance envelope and no code is added — the reason is the text and nothing else.
Slice 4 moved four conditions into this channel from Channel C: three
serialization-failure classes that used to return a placeholder body as a success (now
B8.1–B8.3) and the inverted Obsidian severity (now B7.3). The rule they restore is the one
render_metrics already followed: if the tool could not produce the answer it promised,
the answer is isError: true — never a success whose body happens to read like a
failure.
B1 · contained panic
| # | Condition | Code | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| B1 | A tool body panicked; the panic was caught at the dispatch boundary | none | Yes, with a caveat | Retry, but know that containment restores the transport, not the side effects — a tool that panicked after writing an export or charging a rate-limit token will duplicate that work. | mcp_server/mod.rs:359-380 (containment + rationale), client text at mcp_server/mod.rs:268-270 |
B2 · site policy denial (robots.txt / WAF)
| # | Condition | Code | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| B2.1 | robots.txt rules forbid the URL | none | No | Stop, or retry with the tool's own opt-out (ignore_robots) if the caller has permission. | mcp_server/handlers/scraping.rs:1154-1171 (robots_denied_response), used by mcp_server/handlers/scraping.rs:759-771 (discover_urls), mcp_server/handlers/scraping.rs:957-970 (detect_spa), and the inline gate at mcp_server/handlers/scraping.rs:74-84; mcp_server/handlers/ai.rs:122-124 |
| B2.2 | The policy guard refused (guard condition, not a rules denial) | none | No | Treat as a policy refusal. | mcp_server/state.rs:375-390 |
| B2.3 | Live scrape inside the export pipeline denied | none | No | As B2.1; the message is prefixed with the target URL. | mcp_server/handlers/export.rs:428-435 |
B3 · scrape / crawl operational failure
| # | Condition | Code | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| B3.1 | scrape_url — fetch, Readability extraction, or content sink failed | none | Depends on the class in the text | Read the text; a transient network fault and a hard WAF block look the same here. | mcp_server/handlers/scraping.rs:121-130 |
| B3.2 | scrape_with_options — same | none | Same | Same. | mcp_server/handlers/scraping.rs:227-236 |
| B3.3 | scrape_batch — whole-batch failure | none | Yes if transient | Retry the batch; per-URL failures inside a successful batch are rows, not errors (C3). | mcp_server/handlers/scraping.rs:380-398 |
| B3.4 | crawl_site — engine run failed | none | Yes if transient | Retry the crawl. | mcp_server/handlers/scraping.rs:534-544 |
| B3.5 | crawl_with_sitemap — discovery failed | none | Yes if transient | Retry. | mcp_server/handlers/scraping.rs:713-731 |
| B3.6 | crawl_with_sitemap — sitemap session run failed | none | Yes if transient | Retry. | mcp_server/handlers/scraping.rs:694-711 |
| B3.7 | discover_urls / discover_sitemap — link extraction or sitemap read failed | none | Yes if transient | Retry. | mcp_server/handlers/scraping.rs:830-840, mcp_server/handlers/scraping.rs:915-925 |
| B3.8 | download_assets — asset fetch or write failed | none | Yes if transient | Retry; already-downloaded assets are content-addressed. | mcp_server/handlers/assets.rs:88 |
| B3.9 | extract_links — base-URL resolution failed | none | No | Fix the input HTML or base URL. | mcp_server/handlers/content.rs:87 |
| B3.10 | url_to_file_path — OutputPath::from_url failed | none | No | Fix the URL. | mcp_server/handlers/url_utils.rs:210 |
| B3.11 | crawl_site / crawl_with_sitemap — the run's extracted records exceed the session result budget (64 MiB default), so nothing was retained for export (#1611, F7) | none | Yes, with a smaller run | Re-run with fewer max_pages, or crawl per section. The text names the budget, the measured size, and how many records were kept vs dropped. | mcp_server/handlers/scraping.rs (within_session_budget, session_budget_message, store_session_results) |
The per-class retry policy these texts inherit is
docs/error-classification-matrix.md. The MCP channel
carries no exit code, so the class is not machine-readable here — the agent must parse
the text.
B4 · direct-fetch HTTP failures
| # | Condition | Code | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| B4.1 | discover_urls — non-2xx status (a real failure, deliberately not an empty link set) | none | 4xx no / 5xx yes | Read the status in the text. | mcp_server/handlers/scraping.rs:776-791 |
| B4.2 | discover_urls — transport failure | none | Yes | Retry. | mcp_server/handlers/scraping.rs:844-853 |
| B4.3 | detect_spa — transport failure | none | Yes | Retry. | mcp_server/handlers/scraping.rs:1010-1020 |
| B4.4 | semantic_cleaner — page fetch failed | none | Yes if transient | Retry. | mcp_server/handlers/ai.rs:134-143 |
| B4.5 | get_accessibility_snapshot — snapshot fetch failed (both formats) | none | Yes if transient | Retry. | mcp_server/handlers/axtree.rs:136-147, 168-180 |
B5 · export failures
| # | Condition | Code | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| B5.1 | No session results to export (no crawl ran, or its extraction produced nothing) | none | No | Run a crawl first. This is a precondition, not a fault. | mcp_server/handlers/export.rs:115-121 |
| B5.2 | Blocking-pool join for the session snapshot failed | none | Yes | Retry. | mcp_server/handlers/export.rs:136-142 |
| B5.3 | export_file with empty / whitespace-only content | none | No | Send real content. | mcp_server/handlers/export.rs:186-190 |
| B5.4 | export_file — document chunk validation failed | none | No | Fix the content. | mcp_server/handlers/export.rs:269-275 |
| B5.5 | Exporter construction failed | none | Depends | Read the text. | mcp_server/handlers/export.rs:278-284 |
| B5.6 | Export write failed (all four export tools) | none | Depends | Read the text. | mcp_server/handlers/export.rs:57-61, 294-298 |
| B5.7 | process_export_pipeline — live scrape failed | none | Depends | Read the text; it names the URL. | mcp_server/handlers/export.rs:444-449 |
B6 · feature gating and missing ports
| # | Condition | Code | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| B6.1 | AI tool called with no embeddings provider configured | none | No | Configure a remote embeddings provider (--embedding-provider <id> or WEBFANG_EMBEDDING_PROVIDER); there is no local model and no flag that turns one on. | mcp_server/handlers/ai.rs:58-64 (ai_unavailable), used at mcp_server/handlers/ai.rs:137, mcp_server/handlers/ai.rs:206-208, mcp_server/handlers/ai.rs:216 |
| B6.2 | ai feature is gone; the AI tools are always compiled in and resolve their EmbeddingPort remotely via --embedding-provider) | none | No | none — rebuild advice no longer applies; see ai-onnx-migration.md. | same helper, mcp_server/handlers/ai.rs (line numbers may move) |
| B6.3 | Vault note repository not configured | none | No | Operator must configure persistence. | mcp_server/handlers/ai.rs:201-204 |
| B6.4 | chromium not compiled (axtree tool) | none | No | Rebuild with --features chromium. | mcp_server/handlers/axtree.rs:185-197 |
| B6.5 | Semantic cleaner / inference / embedding failed | none | No | Report it; this is a server-side AI fault. | mcp_server/handlers/ai.rs:144-147, 270 |
| B6.6 | Response serialization failed (AI + axtree) | none | No | Report it; the payload could not be built. | mcp_server/handlers/ai.rs:161-168, 263-268; mcp_server/handlers/axtree.rs:121-133 |
B7 · local integration failures
| # | Condition | Code | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| B7.1 | build_obsidian_uri / open_in_obsidian — vault name or file path rejected | none | No | Fix the vault name or note path. | mcp_server/handlers/obsidian.rs:72-77, 99-104 |
| B7.2 | open_in_obsidian — the dispatch command could not be started at all (Err) | none | Yes | Retry; Obsidian may not be installed. | mcp_server/handlers/obsidian.rs:137 |
| B7.3 | open_in_obsidian — the URI handler reported DispatchStatus::HandlerFailed (the OS handler exited non-zero) | none | Yes | Retry after installing/registering the handler. Moved here from Channel C in slice 4: the requested action did not happen, so it is a tool error. The ⚠️ marker that used to fake the severity inside a success body is gone — the envelope carries it now. | mcp_server/handlers/obsidian.rs:134-136 (in dispatch_outcome_to_result, mcp_server/handlers/obsidian.rs:126-139) |
| B7.4 | get_scrape_metrics — no scrape has been recorded in this process | none | No | Run a scrape first. Precondition, not a fault. | mcp_server/handlers/security.rs:204-208 |
| B7.5 | get_scrape_metrics — snapshot serialization failed | none | No | Report it. This is the render_metrics precedent the four B8 sites now follow. | mcp_server/handlers/security.rs:209-213 |
| B7.6 | scrape_batch — every input URL was rejected before the run | none | No | Fix the URLs. | mcp_server/handlers/scraping.rs:263-265 |
B8 · reply serialization failed (server-side)
Added in slice 4. Each of these four sites used to do
to_string_pretty(..).unwrap_or_else(|_| "failed to serialize") and return that literal as
a successful result — a lost report that was indistinguishable from a real one, which
is the single shape an agent cannot detect. They are ordinary Channel B rows now.
| # | Condition | Code | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| B8.1 | download_assets — the asset list could not be serialized | none | No | Report it; the download report was lost. Do not treat any body as authoritative. | mcp_server/handlers/assets.rs:113-125 (assets_tool_result) |
| B8.2 | extract_links — the link list could not be serialized | none | No | Report it; the link list was lost. | mcp_server/handlers/content.rs:234-246 (links_tool_result) |
| B8.3 | discover_urls / discover_sitemap — the URL list could not be serialized | none | No | Report it; the link list was lost. | mcp_server/handlers/scraping.rs:1239-1256 (discovered_urls_tool_result) |
The old placeholder body cannot be produced any more — a client no longer needs to
special-case the literal string "failed to serialize". These rows are not -32603:
the enclosing match already returns a CallToolResult directly, and ?-propagating
would have split one function's failures across two channels. Retrying will not help; the
payload types' Serialize impls are total, so these branches are defensive.
Channel C — success-shaped diagnostics
A result with no isError and no error member. The call succeeded; the body
describes something that is not a plain success. This is the channel an agent cannot
distinguish from a real success by shape alone — that is why it is enumerated rather than
folded into B.
Slice 4 shrank this channel from six conditions to the four below, and the IDs were
renumbered: the old C2 (the Obsidian severity inversion) became B7.3, and the old C3 (the
"failed to serialize" placeholder) was eliminated outright — its four sites are now
B8.1–B8.3. Old C4/C5/C6 are therefore new C2/C3/C4. What remains here is a deliberate
diagnostic, not a leak: every row below is a tool that succeeded at the only thing it
actually does, and reports a negative finding in the body.
| # | Condition | Code | Retriable | Agent should | Source |
| :-: | :--- | :-: | :-: | :-: | :-: | :-: |
| C1 | validate_url reports an unusable URL as {"valid": false, "reason": …, "reason_code": …} | none | No | Parse the JSON and branch on valid — never on isError. This is the only tool whose failure is a success by design (issue #590, bug #7). Since slice 5 its reason is the message alone and the slug moved to its own reason_code field; the tool's description now states the success-shaped contract outright. | mcp_server/handlers/url_utils.rs:73-82; success twin at mcp_server/handlers/url_utils.rs:58-71 |
| C2 | A run completed with per-item failures (failed_url records; errors / error_breakdown fields) | none | Per item | Read the per-item records; the run itself is a success. | mcp_server/handlers/scraping.rs:1129-1142 (failure records), returned through mcp_server/handlers/scraping.rs:372-379; crawl summary at mcp_server/handlers/scraping.rs:519-534 |
| C3 | detect_obsidian_vault found no vault ("no vault detected") | none | No | A negative result, not a failure. | mcp_server/handlers/obsidian.rs:53-55 |
| C4 | detect_spa found sufficient content ("not an SPA - sufficient content found") | none | No | A negative result, not a failure. | mcp_server/handlers/scraping.rs:1005-1007 |
The old C3 row is gone for good. It promised that a body equal to the literal
"failed to serialize"meant a lost report. Slice 4 removed the four sites that could produce it, so no tool can emit that body any more. A client still special-casing the string is guarding against something the server no longer does — the equivalent failure now arrives as an ordinaryisError: trueresult (B8.1–B8.3).
Channel 0 — transport refusals (no JSON-RPC exists yet)
These never reach a tool. The client sees an HTTP status and, in most cases, no JSON-RPC body at all. Every row here is statically derived — these statuses are produced by middleware, not by a tool, and no test in this crate observes them over a live listener.
| # | Condition | Status | Retriable | Agent should | Source |
|---|---|---|---|---|---|
| T1 | Missing / malformed / wrong bearer token, and the server's own fail-closed default: no token configured and anonymous operation not explicitly enabled (#1611, G-18) | 401 | No | Fix the Authorization header — or, if the server was started without a token, know that the operator must set one (--auth-token / WEBFANG_MCP_AUTH_TOKEN) or opt in to --allow-anonymous on loopback. A 401 with no token configured is a SERVER configuration state, not a wrong guess: the binary refuses to start in that state unless the opt-in is present. | mcp_server/auth.rs, require_auth_or_explicit_anonymous in mcp_server/server.rs |
| T2 | Rate-limit quota exceeded | 429 + Retry-After | Yes, after backoff | Slow down for the advertised number of seconds. The quota is per credential (keyed), so another caller's traffic does not consume yours. | mcp_server/server.rs (rate_limit_middleware, rate_key) |
| T3 | Session-admission cap exceeded | 429 | Yes, after the window | Slow down. Bounded by --max-sessions / --session-cap-window-secs. | mcp_server/server.rs:629-677 |
| T4 | Request took longer than --request-timeout-secs | 408 | Yes | Retry. Note what it bounds: the request FUTURE, not a streaming body — once an SSE stream is answering, events keep flowing past this deadline (the accepted residual half of #1611 G-20; the "SSE channels uncapped" half no longer holds on rmcp 1.8.0, whose session/event channels are built at SessionConfig::channel_capacity = 16). | mcp_server/server.rs (TimeoutLayer) |
| T5 | Body over --body-limit-bytes | 413 | No | Shrink the payload. | mcp_server/server.rs:225 |
| T6 | Accept missing both application/json and text/event-stream | 406 | No | Fix the request headers. | rmcp StreamableHttpService; matrix at mcp_server/server.rs:166-173 |
| T7 | Content-Type not application/json, or the body is not JSON-RPC 2.0 | 415 | No | Fix the request. | rmcp; same matrix |
| T8 | Non-initialize request with no mcp-session-id | 422 | No | Complete the handshake first. | rmcp; same matrix |
| T9 | Panic on the HTTP request path | 500 + JSON-RPC -32603 body | No | Report it. The body echoes the JSON-RPC id when one is recoverable; see the id rules above. | mcp_server/panic_containment.rs |
Layer order matters for reading this, and #1611 F5 changed it. Outermost to innermost: panic backstop → tracing → body limit → timeout → auth → rate limit → session cap → id-echoing panic containment → the rmcp service. Two properties follow, and each is pinned by a test rather than by a comment:
- A request refused
401(no or wrong token) never reaches the rate limiter, so an unauthenticated flood cannot spend the authenticated caller's quota and lock the operator out of its own server. - A request refused
401or shed429never consumes a session slot, so the session cap measures sessions rather than rejections.
Unsettled rows
EC-02 — SETTLED by test, not by reasoning. The question was whether A1/A2 (raised
inside rmcp's argument deserialization) and A3 (raised by the handler body) reach the
client as the same JSON-RPC code, given that they share one Rust error type
(McpError::invalid_params is rmcp::ErrorData). They do not, and
crates/webfang_mcp/tests/mcp_error_channel_mapping_test.rs now records both
error.code and result.isError for all three routes against a live server:
| Route | Observed | Test |
|---|---|---|
McpUrl deserialization (scrape_url with file:///etc/passwd) | result.isError = true, no error member | mcp_error_channel_mapping_test.rs:99-127 |
unknown field (scrape_with_options with typo_field) | result.isError = true, no error member | mcp_error_channel_mapping_test.rs:139-166 |
handler params.validate()? (crawl_site with max_depth: 11) | error.code = -32602, no result member | mcp_error_channel_mapping_test.rs:196-217 |
The mechanism is rmcp's own message prefix, described in
A1–A3. No production code
changed to settle it. The older hedged helper
tests/params_rejection_test.rs::assert_url_argument_rejected, which "accepts EITHER
rejection shape", is now strictly weaker than the new suite and is kept only because it
also asserts the reason text; it is no longer the evidence for these rows.
Rows the code slices changed — all landed:
| Finding | Row affected | Slice | Status |
|---|---|---|---|
| EC-02 — the rmcp argument-rejection mapping | A1, A2, A3 | 3 | Landed — settled by test, rows rewritten above |
EC-03 — SSRF policy and DNS failures shared invalid_params | A7, A8 | 2 | Landed — -32603 + data.reason; classes separated |
| EC-04 — the third channel was undocumented | C1 | 5 | Landed — reason_code slug + an explicit success-shaped description |
| EC-06 — a serialization failure returned success with a placeholder | was C3 | 4 | Landed — C3 deleted; the four sites are now B8.1–B8.3 |
EC-07 — DispatchStatus::HandlerFailed was returned as tool success | was C2 | 4 | Landed — moved to B7.3, ⚠️ marker dropped |
| EC-08 — validation text was English while operational errors were Spanish; no machine-readable reason | A3.x, A4.x, A10–A14, B*, C1 | 5 | Landed on the machine-readable half — data.reason is now present on every argument refusal the server raises (the -32603 server faults, A15/A16, carry no data and never did). The 16 sites slice 5 left behind (6 in path_gate.rs, 10 handler-inline in handlers/export.rs / handlers/scraping.rs) were converted, and the reason-less constructor they called was deleted, so both shapes that survived slice 5 are gone; see A4, A10–A17 and the shape table. The language half was deliberately not done, and remains open: see Message language. |
EC-05 — a deliberate, justified non-action, not an oversight. The seven panic-capable
serialization expect sites are unchanged by design: mcp_server/handlers/content.rs:209,
mcp_server/handlers/scraping.rs:532, 691, 1002, and
mcp_server/handlers/url_utils.rs:70, 80, 207 (all
expect("serializing JSON to a string cannot fail")). Each carries an explicit
#[allow(clippy::expect_used)] under a crate-wide deny; each operand is a
serde_json::Value, a total Serialize; and a panic there is contained into
isError: true at the dispatch boundary (row B1) without killing the session. The correct
remedy — a proven-total serialization boundary or an explicit internal_error mapping —
would be a behavioural change, not a refactor, and the issue explicitly warns that blind
replacement is churn. Recorded here so the next reader does not re-open it as a bug.
Overlapping, not duplicated here: EC-08's message-language half overlaps #1604, which normalizes operational error event text to structured fields. #1604 does not own the message-language or machine-readable-reason halves; those are decided on this page.
Appendix — construction-site inventory
Counted in crates/webfang_mcp/src/mcp_server/ in the working tree — that is, commit
ccdef614 plus the uncommitted reason-slug changes described under
EC-08 — production code only (test modules and doc comments excluded).
ccdef614 is the base, not a clean checkout: validation.rs, path_gate.rs,
handlers/export.rs and handlers/scraping.rs are all dirty in this tree.
How the counts were derived — each row is a grep for one construction form,
restricted to the mcp_server/ tree, with comment-only lines dropped by a /// / //!
filter. Two counts deserve a note rather than a bare number:
provenance::neutralized_erroris counted as production call sites, so thefn neutralized_errordefinition and every#[cfg(test)]call are excluded. That leaves 37, of which 2 are thehonest_errorwrapper bodies and 4 are the mapper helpers slice 4 introduced; the 31 remaining are the same direct handler-body calls the previous revision of this table counted (that figure reproduces exactly at763714b6, which is the cross-check that the method is stable).- Channel C has no
neutralized_error/CallToolResult::errorsites at all, and itslocal_textsites are only 5. The old "10 construction sites across 6 conditions" is not reproducible by any single grep; the honest number is 5local_textsites across 3 standalone conditions, plus C2, which is a field inside a success payload rather than a construction of one.
| Channel | Construction form | Sites | Accounted for |
|---|---|---|---|
| A | McpError::invalid_params called directly | 4 — the one surviving factory (mcp_server/validation.rs:110) and the SSRF channel (mcp_server/ssrf.rs:122, 135, 178) | A5, A6, A9 |
| A | validation::invalid_params_with_reason branches (the validation funnel) | 44 — all in mcp_server/validation.rs (lines 139–756) | A3.1–A3.9 |
| A | validation::invalid_params_with_reason call sites in params.rs | 6 — mcp_server/params.rs:97, 155, 176, 463, 527, 536 | A1, A3.4, A3.5, A3.6, A3.7 |
| A | validation::invalid_params_with_reason call sites in path_gate.rs | 6 — mcp_server/path_gate.rs:282, 289, 298, 309, 327, 345 | A4.1–A4.6 |
| A | validation::invalid_params_with_reason call sites in handlers/export.rs | 8 — mcp_server/handlers/export.rs:197, 210, 235, 252, 326, 364, 410, 470 | A10–A12 |
| A | validation::invalid_params_with_reason call sites in handlers/scraping.rs | 2 — mcp_server/handlers/scraping.rs:458, 605 | A13, A14 |
| A | McpError::internal_error | 9 — mcp_server/handlers/scraping.rs:106, 109, 212, 215, 1121, 1127; mcp_server/macros.rs:16; mcp_server/ssrf.rs:154, 166 | A7, A8, A15, A16 |
| A | re-raised Err(e) — forwarding, not a construction | 1 — mcp_server/handlers/axtree.rs:74 | A17 |
| B | provenance::neutralized_error called directly | 37 — 31 in handler bodies + 2 honest_error wrapper bodies (handlers/ai.rs:41, handlers/axtree.rs:27) + 4 in the slice-4 mapper helpers (handlers/assets.rs:120, handlers/content.rs:241, handlers/scraping.rs:1251, handlers/obsidian.rs:134) | B1–B8 |
| B | honest_error wrapper definitions | 2 — mcp_server/handlers/ai.rs:40, mcp_server/handlers/axtree.rs:26 | (definitions only) |
| B | calls through those wrappers | 14 — mcp_server/handlers/ai.rs:124, 129, 142, 146, 164, 197, 202, 207, 265, 270; mcp_server/handlers/axtree.rs:130, 144, 176, 194 | B2.1, B4.4, B4.5, B6.1–B6.6 |
| B | direct CallToolResult::error | 1 — mcp_server/mod.rs:377 | B1 |
| C | provenance::local_text on a non-success outcome | 5 — mcp_server/handlers/url_utils.rs:68, 78; mcp_server/handlers/obsidian.rs:54, 55; mcp_server/handlers/scraping.rs:1005 | C1, C3, C4 |
| C | per-item failure records inside a success payload (not a standalone construction) | 2 payloads — mcp_server/handlers/scraping.rs:372, 519 | C2 |
| 0 | HTTP status producers in this crate | 6 — mcp_server/auth.rs:59; mcp_server/server.rs:223, 225, 279, 311, 672 | T1–T5, T9 |
| 0 | HTTP status producers owned by rmcp's StreamableHttpService | 3 — not constructed here; enumerated in the matrix at mcp_server/server.rs:166-173 | T6–T8 |
Totals: 136 error-producing construction sites on the JSON-RPC channels (79 on A, 52 on B, 5 on C), plus 1 forwarding site and 6 in-crate HTTP status producers. Every one is accounted for by a row above; 0 unclassified. A is 4 + 44 + 6 + 6 + 8 + 2 + 9; B is 37 + 14 + 1.
Two scope statements, so the totals are not read as more than they are:
- These are sites, not conditions. The eight
invalid_params_with_reasoncalls inhandlers/export.rscollapse into three rows (A10–A12), because several sites report the same condition for different tools; the reverse also holds, with 44 funnel branches collapsing into nine A3.x rows. - A1's construction site is the
McpUrl::try_frombranch atparams.rs:97, but it is not a Channel A construction — it travels on Channel B. It appears in this inventory only because that is where its Rust error is defined; its row (A1) is a Channel B row, and noerrormember is emitted for it at all.
MCP Public Surface Policy
What may change in webfang_mcp without breaking somebody, and what requires a
release. This is the written decision for issue #1614; the code attributes and
the machine check below exist only because this document states the rules first.
The source of the findings is AUDIT-MCP-API-CONTRACT-20260925-035505/ →
BREAKING_CHANGE_RISKS.md (BC-01 … BC-10).
The one-sentence version
A client written against the advertised
inputSchemaof release N must keep working against release N+1 for any change classified additive below, and must be re-validated by the human for anything classified breaking — and every classification below is checked by a test, not by good intentions.
1. Three boundaries, not one
The audit speaks of "the public surface" as if it were a single thing. It is three, with three different sets of consumers, and conflating them is what produces both false alarms and false comfort.
| # | Boundary | Who consumes it | Stability promise |
|---|---|---|---|
| A | The MCP wire contract: the tool inventory and each tool's advertised inputSchema | MCP clients in other processes. The only boundary with consumers outside this workspace. | Strong. Classified below. |
| B | The Rust surface of webfang_mcp: pub items in crates/webfang_mcp/src/mcp_server/ | This workspace only. See §2. | Explicitly none. No semver promise. |
| C | The webfang_core port traits (domain::*_port, domain::*, application::crawler) | Implementors. Today: this workspace only (see §2). | Stated per trait in §6. |
BC-10 is answered by this table, and the answer is not "we promise
backwards compatibility". The reviewed source establishes no semver promise,
and none is being invented here for a crate that is never published. What is
being promised is a narrower and checkable thing: the wire contract is
versioned by contract_version, and the workspace-internal Rust surfaces are
declared unstable.
Why B carries no promise
release-plz.toml sets publish = false for every crate in the workspace and
AGENTS.md states distribution is binaries-only. No webfang_mcp type has an
external Rust consumer today, and webfang_cli does not depend on
webfang_mcp at all (its mcp feature forwards to webfang_core/mcp).
The consequence is the opposite of the comforting reading: because nothing
outside the workspace compiles against webfang_mcp, a Rust-level change to
it cannot break a user. Changes there are reviewed on their own merits, not
under a compatibility rubric. #[non_exhaustive] therefore buys a webfang_mcp
type nothing against users — it only constrains this repository. That is why
§4 stamps it in exactly three places and records a deliberate "no" everywhere
else.
2. Consumers, stated once
Verified in this tree at 2b35f6ee:
webfang_cli→webfang_mcp: no dependency edge.webfang_mcp'spubitems referenced outside the crate: only its own integration tests undercrates/webfang_mcp/tests/.crates/webfang_mcp/tests/*are separate crates: they linkwebfang_mcpas an external dependency, so#[non_exhaustive]binds them exactly as it would bind a downstream user. This is not a technicality — it is the reason §4's list is short.
3. Compatibility classes (boundary A)
| Class | Meaning | Release |
|---|---|---|
| S0 — stable | No client's observable behaviour changes. Docs, descriptions, internal logging. | patch |
| S1 — additive | Every previously-valid request stays valid and behaves identically. New clients gain capability. | minor |
| S2 — breaking | Some previously-valid request, or some previously-served response, changes meaning or stops being accepted. | major |
3.1 The compatibility matrix
One row per dimension of a property line in the checked-in manifest
(crates/webfang_mcp/tests/fixtures/mcp_public_surface.tsv). "Tightens" always
means fewer requests are accepted or more requests mean something different.
| Change to a tool's advertised schema | Class | Note |
|---|---|---|
| A new tool is registered | S1 | The inventory is append-only within a major. |
| A tool is removed or renamed | S2 | Includes moving it behind cfg(feature). |
A tool's description changes | S0 | Prose, except where it makes a behavioural claim — see §7. |
| A Rust doc comment on a params type or enum changes | S0 in intent, wire in fact | See §3.3. schemars emits the doc comment into the advertised $defs.<Type>.description, so it is a wire string, not documentation. |
A property is added and is not in required | S1 | |
A property is added and is in required | S2 | Existing requests omit it. |
| A property is removed | S2 | Even if the handler still tolerates it: deny_unknown_fields is on every params struct, so it becomes a rejection. |
A property's type changes | S2 | |
Nullability widens (accepts null where it did not) | S1 | Strictly more requests accepted; the new requests previously errored. |
Nullability tightens (null no longer accepted) | S2 | |
default is added, and the runtime already applied exactly that value | S1 | Pure advertisement of existing behaviour. |
default changes, and the runtime changed with it | S2 | A silent behaviour change for a client that omitted the field. |
default changes in the schema only | S0 defect, not S1 | This is drift, not a change. It is a bug in whichever side moved alone; see §5. |
A bound widens (minimum lowers, maximum raises) | S1 | |
| A bound tightens | S2 | |
| An enum gains a variant | S1 | Announce it. A client that matches exhaustively over JSON strings must add a wildcard. |
| An enum loses a variant | S2 | |
additionalProperties: false is removed | S2 | Widens acceptance; deny_unknown_fields is a deliberate control. |
3.2 Response payloads
Response payloads are out of scope for versioning in this issue (audit
slice 4, BC-08). Three channels already exist and must be used instead of a
fourth shape: see mcp-error-contract.md. For success
payloads the promise this policy makes is narrow and honest: the envelope is
stable, the body is not versioned. Handlers that emit Content::text with
ad-hoc JSON are not covered by the manifest and must not be described as
covered.
3.3 A doc comment is not a doc change
Discovered while implementing this issue, and worth stating before somebody
learns it the expensive way: schemars copies a type's Rust doc comment into the
advertised schema as $defs.<Type>.description. Writing a thorough doc
comment on a type named by a params struct therefore changes what an MCP client
reads.
The first draft of #[non_exhaustive] on SnapshotFormatParams carried its
rationale in the doc comment and failed mcp_advertised_schema_snapshot_test
with a multi-line description diff. The rationale now lives in a // comment
directly above the doc comment, and the doc comment is frozen.
Rule: a type that appears in an advertised schema keeps its doc comment
byte-stable. Put the reasoning next to it, not in it. Types that never reach a
schema (Outcome, DefaultOverride) are unconstrained.
4. #[non_exhaustive] — where, and why not everywhere
#[non_exhaustive] is not free. Adding it to a type that external code
matches exhaustively, or constructs with a struct literal, is itself a breaking
change for those consumers — the standard library's own
std::collections::hash_map::Entry addition was a breaking change for exactly
this reason.
4.1 Applied
| Type | File | Why it is safe here |
|---|---|---|
SnapshotFormatParams | params.rs (BC-02) | It is deserialized, never constructed: McpUrl sets the precedent (private field, TryFrom<String> only). No test crate constructs it. The two in-crate exhaustive matches in handlers/axtree.rs stay legal — non_exhaustive only binds outside the defining crate. |
DefaultOverride | schema_bridge.rs | A bridge vocabulary already extended once in place (SetBounds, #1294), i.e. it demonstrably grows. Only apply_default_overrides matches it, in-crate. |
Outcome | metrics.rs | A metrics bucket whose set has already grown with each new outcome concept (admission control, #1611). Only in-crate. |
4.2 Deliberately NOT applied
The 16 public params structs (BC-03). Not an oversight, and not a half-measure — a decision with three reasons:
- It would break the only consumers that exist.
crates/webfang_mcp/tests/build these structs with struct literals in 16 places (mcp_params_validation_test.rs). Adding the attribute turns every one of them into a compile error, and the fix is 16 constructor builders that buy nothing: no user compiles against this crate (§2). - The compatibility they threaten is already protected by the right
mechanism. The wire surface of these structs is the advertised schema, and
that is pinned byte-for-byte by
mcp_advertised_schema_snapshot_test(#1612) and semantically byoptions_spec_parity_test. The gap#[non_exhaustive]would close — "someone constructs this struct from outside" — is empty. - The attribute encodes the wrong axis. These structs grow by field, and
their
required-ness is a wire property that already has a machine check. An attribute that constrains struct literals would not notice a new required field, which is the actual S2 risk.
If this crate is ever published, revisit 4.2 with constructors — not before.
Origin (provenance.rs). Its three variants are a complete semantic
partition ("where does this text come from"), not a growing vocabulary. Adding a
variant would change provenance labelling semantics, which is a correctness
matter under §7, not a compatibility lever.
5. Defaults, nullability and bounds (BC-09)
These are compatibility-bearing, and the audit's "no-op schema fields may activate on a later build" is the sharp form of the risk: a field that is inert today and accidentally becomes live is a behaviour change with no code review signal.
Three rules, each machine-checked:
- An advertised default must equal the applied default. Enforced for
OptionsSpec-backed properties by
options_spec_parity_test(overridden_defaults_advertise_runtime_effective_values) and for MCP-only properties bymcp_advertised_defaults_test(#1294 NS-04). The bridge'sDefaultOverride::{Set, Unset, SetBounds}is the single sanctioned way to make them differ deliberately, and each override must name a real property — an override naming an unknown property warns and is skipped, so a renamed field cannot leave a stale override behind silently. - A nullability union in the schema must match serde's acceptance.
promote_spec_to_nullablereconciles anOption<T>field's derived["<inner>", "null"]with the spec rendering; a mismatch is an S2 defect. - A bound must equal the bound the validator enforces, taken from the same
constant.
concurrencyadvertisedminimum: 0while the validator rejected 0 — the class of drift this rule exists for.
The manifest in §8 pins the advertised side of all three for every registered tool, not only the bridged seven.
6. Port traits (BC-04, BC-05, BC-06)
Adding a required method to a public trait is source-breaking for every existing implementor. What determines the class is whether the trait is sealed:
| Trait | Sealed? | Adding a required method |
|---|---|---|
CheckpointStore (application/crawler/checkpoint.rs) | Yes — mod private { pub trait Sealed {} }, private module | S1. No external implementor exists. This is the reference pattern. |
SemanticCleaner (domain/semantic_cleaner.rs) | Yes — same pattern. pub mod private rather than mod private, which is a visibility difference with no sealing effect: the bound private::Sealed is what seals it, and it is unsatisfiable outside the crate. | S1. |
NoteRepository, VaultNoteReader | No | S2 for any downstream implementor. |
TextChunker, EmbeddingPort | No | S2 for any downstream implementor. |
BC-05 is settled: SemanticCleaner is sealed. The audit's "Needs
verification" is closed by reading the bound, not by a compile-fail fixture —
which would have required a trybuild dev-dependency this issue may not add.
The residual risk is a future contributor changing pub mod private to
pub(crate) mod private; that still seals, and a blanket private::Sealed
bound is the thing to protect. Adding the sealing compile-fail fixture remains
open and is recorded here as future work rather than silently skipped.
The migration rule for BC-04: an unsealed port trait that acquires a new capability should first gain a defaulted method (S1), and only take a required one (S2) behind a major. Sealing first makes every later method S1.
7. Always-listed tools (BC-07)
Verified in this tree: handlers::ai::build_router and
handlers::axtree::build_router are called unconditionally from
build_tool_router, with no #[cfg] on the function or the call. So the tool
inventory is identical across feature builds, and that is a deliberate
strength the audit correctly told us not to regress.
The client-facing consequence, which is the part BC-07 left open:
- A tool is never absent. Availability depends on features and runtime, not on registration.
- An unavailable tool answers with a result, not an absence. Handlers route
through
honest_error, producingisError: truewith Spanish content text and a reason slug — never a protocol error, never a false success. - Therefore: gating a registration behind
cfg(feature), or dropping a tool from the always-listed inventory, is S2. The audit's "verify the client policy before changing registration" is answered here, and the answer is "do not".
The one exception is test_probe, registered only under an env gate. It is a
zero-argument probe for #1646 panic containment, not a product tool; it is
excluded from the manifest for that reason, and the exclusion is stated here
rather than left implicit.
8. What is checked, and by what
Policy nobody can check drifts; this section exists so the coverage claim is auditable rather than asserted.
| Rule | Enforced by | Runs in CI |
|---|---|---|
| Every registered tool's inventory (names + count) | public_surface_policy_test::tool_inventory_is_pinned | yes |
Per-property type / nullability / default / minimum / maximum / required | public_surface_policy_test::advertised_property_matrix_is_pinned | yes |
| Advertised default == applied default (bridged + MCP-only) | options_spec_parity_test, mcp_advertised_defaults_test (#1612, #1294) | yes |
| Bridged schema byte-for-byte | mcp_advertised_schema_snapshot_test (#1612) | yes |
A shape change moved contract_version | scripts/check_mcp_public_surface.sh | yes — step MCP public-surface version marker gate in repo-guards |
The whole surface is feature-independent (36 tools, 81 properties, identical under default and --all-features) | public_surface_policy_test run under both feature sets | yes |
8.1 contract_version
A single integer in crates/webfang_mcp/tests/fixtures/mcp_public_surface.tsv.
It starts at 1, and:
- Any S2 change bumps it, and requires a major release.
- Any S1 change that alters a compatibility-bearing row bumps it, and requires a minor release.
- S0 changes do not bump it.
8.2 The honest gap
The manifest test compares the live surface against the committed
fixture, so it fails in CI the moment a shape moves and the fixture is not
updated. It cannot know whether the author bumped contract_version, because
both live in the same file.
scripts/check_mcp_public_surface.sh closes that half. It compares the fixture
against a base commit, classifies every changed row per §3.1, and exits
non-zero when the surface moved and contract_version did not — and also in
the reverse case, a version bump with no surface change, which would silence the
guard without a reason.
The base is not the merge-base with main, and that is deliberate. On a
feature branch that introduces the fixture mid-stack, a merge-base base makes
every row read as "added" and the guard goes blind right after the introducing
commit. The local fast-gate call uses the last commit that touched either file;
the CI step uses the event's own base (the PR's base SHA, before on a push,
base_sha on a merge queue entry). Both ask the question that actually matters:
did this change move the surface without moving the marker?
It runs in two places. Locally it is part of scripts/ci_fast_gate.sh (the
pre-push gate). In CI it is the MCP public-surface version marker gate step of
the repo-guards job in .github/workflows/ci.yml. repo-guards has no
paths: filter and CI Gate already requires it in both scopes, so a marker
that does not move reds the required check even on a docs-only PR; the
path-filtered mcp lane cannot host it, because a job behind paths: never
reports on every PR and therefore cannot be a required context.
The default depth-1 checkout does not contain the event's base commit, so the
step fetches that one commit (--depth=1) and fails closed if it is still
missing. That check is load-bearing: the guard reads a fixture that is absent
at base as the bootstrap commit and exits 0, so a silently failed fetch would
otherwise turn the gate green. The guard draws that line itself too: a base that
is not a commit, or a tracked file that exists at base and cannot be read, exits
2 instead of being flattened into the same empty file as a genuine bootstrap.
Schedule and workflow_dispatch runs have no change under review and skip the
step with an explicit note.
The edges of this wiring are pinned by scripts/test_mcp_surface_wiring.sh,
which runs in repo-guards beside the gate and in the fast gate's CI-only lane:
the step passes --base from the resolve step's output (never a literal),
repo-guards stays unfiltered and unconditional, gate waits on it in both
scopes, and a base that is not a commit is refused rather than read as
bootstrap.
9. Per-finding disposition
| id | finding | disposition |
|---|---|---|
| BC-01 | Export names not aligned (export_format ↔ wire format) | Root cause owned by #1612. Here: recorded as the canonical case for §3.1 — renaming either side is S2. Not re-fixed. |
| BC-02 | SnapshotFormatParams exhaustive | Fixed — #[non_exhaustive], §4.1. |
| BC-03 | 16 params structs unannotated | Decided: no attribute, with reasons, §4.2. |
| BC-04 | Unsealed public port traits | Policy declared, §6. Migration rule stated. Sealing not performed — see §10. |
| BC-05 | SemanticCleaner sealing unverified | Settled: it is sealed. §6. Compile-fail fixture remains open. |
| BC-06 | CheckpointStore sealed, no method-addition policy | Policy declared, §6. Confirmed as the reference pattern. |
| BC-07 | AI/chromium always registered | Client policy declared, §7. Verified ungated — and the pinned surface is byte-identical under both feature sets, which proves the claim instead of asserting it. |
| BC-08 | Output payloads untyped/unversioned | Deferred — the manifest covers input schemas only, and §3.2 says so. Follow-up recorded. |
| BC-09 | Defaults / nullability / bounds | Rules + machine check, §5. |
| BC-10 | No semver promise stated | Declared, §1 — narrowed to three boundaries, with B and C explicitly promised nothing. |
10. Explicitly not done
- No re-fix of the schema drift (BC-01).
#1612owns it and it is delivered. Nothing here changes a wire name, a field name, or an advertised schema. - No sealing of the BC-04 traits. It is a
webfang_corechange with a wide blast radius acrossai,mcp,cli,benchmarkandtest_utils, and three sibling issues branch off this branch. The policy above states what sealing would buy; the migration is its own change. - No output-DTO versioning (BC-08). Recorded as slice 4.
- No compile-fail fixture for BC-05 — it needs a dev-dependency this issue may not add. The bound was read instead, which settles the same question.
11. When you change something on this surface
- Classify the change with §3.1 before writing it.
- Run
cargo nextest run -p webfang_mcp. A red manifest test is not a chore to re-record — it is the answer to "did I classify this right?". - If a row changed, run
scripts/check_mcp_public_surface.sh --record, then read its diff. It prints the class of every row that moved. - If any row moved, bump
contract_versionand put the release requirement in the commit body. - If a row moved and you classified it S0, you are wrong about one of them — go back to §3.1.