Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

CLI Reference

Complete flag reference for the webfang binary. The block in the "Complete flag reference" section below is auto-generated from the binary itself by scripts/gen-cli-reference.sh (built with the exact release feature set, ai mcp), so flags, defaults, and WEBFANG_* environment variables always match reality. CI drift-checks the chapter against the binary on every doc run; regenerate locally with just cli-doc-regen.

Every flag in this reference is compiled into the binary you are running. The release artifacts are built with no capability features, so feature-gated behavior (--js-strategy full, SQLite persistence, adaptive selectors, tokio-console) is absent or rejected there — the per-flag consequences are in the Release Feature Matrix.

Quick start

Scrape a single page by passing the URL positionally:

webfang https://example.com

Crawl a whole site following its sitemap.xml:

webfang https://example.com --use-sitemap --max-pages 50

Extract only the elements matching a CSS selector and export for a RAG pipeline:

webfang https://example.com -s "article.main-content" --export-format jsonl

Resume an interrupted crawl from its last checkpoint:

webfang --batch-file urls.txt --resume

Zero values

Numeric flags follow the Zero Silent Loss policy: a zero is never silently turned into another value.

  • Zero disables the feature where an "off" state exists: --delay-ms 0 disables request pacing (no token bucket is allocated), --max-depth 0 scrapes only the seed URL.
  • Zero is rejected with a usage error where zero is meaningless or destructive: --concurrency, --download-concurrency, --download-timeout, --max-pages, --rate-limit-burst, --timeout-secs.
  • Legacy exception: a non-numeric --rate-limit-burst value warns and falls back to the hardware-derived default (numeric zero and out-of-range values are still rejected).

Configuration file

Flags can also come from a TOML file, which is how you set values once instead of on every invocation. webfang looks for webfang/config.toml under the platform's user config directory:

PlatformResolved from
Linux$XDG_CONFIG_HOME, else ~/.config
macOS~/Library/Application Support
Windows%APPDATA%

An absolute XDG_CONFIG_HOME is honored on every platform, not just Linux — that is what makes hermetic test fixtures possible.

To point at a specific file instead, set WEBFANG_CONFIG to the path of the config.toml itself. It takes precedence over the table above, and it differs from XDG_CONFIG_HOME in two ways worth knowing:

  • It names the file, while XDG_CONFIG_HOME names a directory that webfang appends webfang/ to. Use it when your file does not live in that subdirectory.
  • It is honored on Windows, where no environment variable reaches the platform config lookup: that lookup calls SHGetKnownFolderPath(FOLDERID_RoamingAppData), a Windows Shell API that reads no env var at all. Until WEBFANG_CONFIG existed, a Windows install could not be redirected to a config file by env at all.

A missing file at the platform path is not an error: webfang falls back to its defaults, because a fresh install has no config yet.

That tolerance stops at an explicit WEBFANG_CONFIG. Once you name a file, webfang holds you to it — the run stops with exit code 78 (configuration error) and a message naming the path, instead of quietly running with defaults. A typo in a config path used to be indistinguishable from a successful run.

WEBFANG_CONFIGResult
unset, or emptyPlatform lookup; a missing file means defaults
absolute path to a readable, valid TOML fileUsed
relative pathRejected, exit 78
path does not existRejected, exit 78
unreadable — wrong permissions, or a directoryRejected, exit 78
not valid TOMLRejected, exit 78

Two edge cases matter because this variable is usually set from a script or a service unit rather than typed:

  • An empty value counts as unset. WEBFANG_CONFIG= webfang ... behaves exactly like not setting the variable at all, so a script that exports it before it has a value falls back to the platform lookup instead of failing.
  • A relative path is rejected rather than resolved. It used to be resolved against the current working directory of the process, not against your home directory — predictable in an interactive shell, and arbitrary under a daemon or service manager, whose working directory is whatever the supervisor chose. The same variable could therefore name a different file depending on who started the process, which is why an absolute path is now required.
WEBFANG_CONFIG=/etc/webfang/config.toml webfang https://example.com

Complete flag reference

Everything below is the verbatim output of webfang --help and webfang completions --help — including every default value and the matching WEBFANG_* environment variable for each flag.

$ webfang --help

CLI Arguments for the webfang binary.

Parsed using `clap` with derive macros.

# Examples

```no_run use webfang_core::Args; use clap::Parser;

let args = Args::parse_from([ "webfang", "--url", "https://example.com", "--output", "./output", "--export-format", "jsonl", "--resume", ]);

assert_eq!( args.crawler.url.as_ref().map(webfang_core::domain::ValidUrl::as_str), Some("https://example.com") ); ```

Usage: webfang [OPTIONS] [URL]
       webfang [OPTIONS] [URL] <COMMAND>

Commands:
  completions  Generate shell completion scripts
  connect      Detect and validate a remote embedding endpoint, then write its config
  doctor       Diagnose the embedding path — config, credential, network, budget
  show         Dump the effective embeddings provider, the configured budget, and the historical model-profile metadata
  recover      Recover pages from a stranded batch capture spool into markdown files
  help         Print this message or the help of the given subcommand(s)

Arguments:
  [URL]
          URL to scrape (positional shorthand — equivalent to --url). Hardened through the SAME argv-boundary parser as `--url` (#1239): the shorthand cannot bypass the credential strip

Options:
  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Target:
  -u, --url <URL>
          URL to scrape (required unless using a subcommand)
          
          [env: WEBFANG_URL=]

  -s, --selector <SELECTOR>
          Matched elements become the content directly (no main-content cleanup); title and date come from the whole document, excerpt stays empty
          
          [env: WEBFANG_SELECTOR=]
          [default: body]

Discovery:
      --delay-ms <DELAY_MS>
          Delay between requests in milliseconds
          
          [env: WEBFANG_DELAY_MS=]
          [default: 1000]

      --jitter-ms <JITTER_MS>
          Random addition to each delay between requests, in milliseconds (0 = fixed delay; the mean interval becomes delay + jitter/2, so the request rate drops; with 0, a run that fits in the burst is not paced at all)
          
          [env: WEBFANG_JITTER_MS=]
          [default: 0]

      --max-pages <MAX_PAGES>
          Maximum pages to scrape
          
          [env: WEBFANG_MAX_PAGES=]
          [default: 10]

      --concurrency <CONCURRENCY>
          Concurrency level (auto or number, minimum 1)
          
          [env: WEBFANG_CONCURRENCY=]
          [default: auto]

      --rate-limit-burst <RATE_LIMIT_BURST>
          Explicit rate-limiter burst permits (token-bucket capacity).
          
          Overrides the hardware-derived budget-model default (Q1: burst is decoupled from crawl concurrency). Accepts a positive integer, or the keyword `auto` for the derived default (the same spelling `--concurrency` understands). Raw string here ON PURPOSE: validation/conversion happens once in preflight staging via `parse_rate_limit_burst` so CLI, env, and programmatic input all share one semantic. That semantic FAILS CLOSED: `0`, an out-of-range value, and any other typo are rejected with exit 78 rather than degrading to the default — only `auto` and an empty value mean "not set".
          
          [env: WEBFANG_RATE_LIMIT_BURST=]

      --use-sitemap
          Use sitemap for URL discovery NOTE: HTTP redirects (301/302) are resolved at scrape-time, not parse-time. This avoids redundant HEAD requests during sitemap parsing for better performance
          
          [env: WEBFANG_USE_SITEMAP=]

      --sitemap-url <SITEMAP_URL>
          Explicit sitemap URL
          
          [env: WEBFANG_SITEMAP_URL=]

Behavior:
      --single-page
          Scrape only the seed URL without discovery or crawling (batch mode always scrapes one page per URL)
          
          [env: WEBFANG_SINGLE_PAGE=]

      --resume
          Resume mode - skip URLs already processed
          
          [env: WEBFANG_RESUME=]

      --state-dir <STATE_DIR>
          Custom state directory for resume mode
          
          [env: WEBFANG_STATE_DIR=]

      --download-images
          Download images from the page
          
          [env: WEBFANG_DOWNLOAD_IMAGES=]

      --download-documents
          Download documents from the page
          
          [env: WEBFANG_DOWNLOAD_DOCUMENTS=]

      --download-assets
          Download all assets (images + documents) from the page
          
          [env: WEBFANG_DOWNLOAD_ASSETS=]

      --extraction-fingerprint
          Record extraction failure fingerprints in SQLite and attach them to low-quality extraction hints (#792). Repeated low-score extractions on the same site/selector pair accumulate a failure count surfaced in the hint, instead of degrading silently
          
          [env: WEBFANG_EXTRACTION_FINGERPRINT=]

      --clean-ai
          Use AI-powered semantic cleaning for better RAG output
          
          [env: WEBFANG_CLEAN_AI=]
          [alias: --ai]

Display:
  -v, --verbose...
          Verbosity level: -v (INFO), -vv (DEBUG), -vvv (TRACE)
          
          [env: WEBFANG_VERBOSE=]

  -q, --quiet
          Quiet mode — suppress info/debug output
          
          [env: WEBFANG_QUIET=]

  -n, --dry-run
          Dry-run mode — discover URLs and print without scraping
          
          [env: WEBFANG_DRY_RUN=]

      --trace-file <TRACE_FILE>
          Path to write OTel spans as JSONL for offline debugging
          
          [env: WEBFANG_TRACE_FILE=]

Crawler Settings:
      --max-depth <MAX_DEPTH>
          Maximum depth to crawl (0 = only seed URL)
          
          [env: WEBFANG_MAX_DEPTH=]
          [default: 2]

      --timeout-secs <TIMEOUT_SECS>
          Request timeout in seconds
          
          [env: WEBFANG_TIMEOUT_SECS=]
          [default: 30]

      --include-pattern <INCLUDE_PATTERNS>
          URL patterns to include (glob-style). Three modes:
          
          * Path: starts with `/` → matched against URL path, e.g. `/pricing`, `/admin/*` * Path glob: starts with `*/` → matched against URL path, e.g. `*/api/*` * Host (default): matched against hostname, e.g. `example.com`, `*.example.com`
          
          Example: to exclude a path, use `--exclude-pattern "/admin/*"`, not `*admin*`
          
          [env: WEBFANG_INCLUDE=]

      --exclude-pattern <EXCLUDE_PATTERNS>
          URL patterns to exclude (glob-style, same three modes as --include-pattern). Deny takes precedence over allow
          
          [env: WEBFANG_EXCLUDE=]

Download Settings:
      --asset-naming <ASSET_NAMING>
          Estrategia de nombre de archivo para assets descargados: hash (default), slug, content-disposition
          
          [default: hash]
          [possible values: hash, slug, content-disposition]

      --download-concurrency <DOWNLOAD_CONCURRENCY>
          Máximo de descargas de assets concurrentes por página (mínimo 1)
          
          [env: WEBFANG_DOWNLOAD_CONCURRENCY=]

      --max-file-size <MAX_FILE_SIZE>
          Maximum file size to download in bytes (default: 50MB)
          
          [env: WEBFANG_MAX_FILE_SIZE=]
          [default: 52428800]

      --download-timeout <DOWNLOAD_TIMEOUT>
          Timeout for individual asset downloads in seconds (minimum 1)
          
          [env: WEBFANG_DOWNLOAD_TIMEOUT=]
          [default: 30]

HTTP Client Settings:
      --max-retries <MAX_RETRIES>
          Maximum number of retry attempts
          
          [env: WEBFANG_MAX_RETRIES=]
          [default: 3]

      --backoff-base-ms <BACKOFF_BASE_MS>
          Base delay for exponential backoff (ms)
          
          [env: WEBFANG_BACKOFF_BASE_MS=]
          [default: 1000]

      --backoff-max-ms <BACKOFF_MAX_MS>
          Maximum delay for exponential backoff (ms)
          
          [env: WEBFANG_BACKOFF_MAX_MS=]
          [default: 10000]

      --accept-language <ACCEPT_LANGUAGE>
          Accept-Language header value
          
          [env: WEBFANG_ACCEPT_LANGUAGE=]
          [default: en-US,en;q=0.9]

      --user-agent <USER_AGENT>
          Custom User-Agent header value (overrides Chrome 145 default)
          
          [env: WEBFANG_USER_AGENT=]

  -H, --header <NAME: VALUE>
          Inject a custom HTTP header as `Name: Value` (repeatable).
          
          Overrides any default header with the same (case-insensitive) name. Example: `-H "Authorization: Bearer TOKEN"`.
          
          [env: WEBFANG_HEADER=]

      --cookie <NAME=VALUE>
          Inject a custom cookie as `name=value` (repeatable).
          
          Seeded into the cookie jar before the first request so authenticated crawls work without a prior login round-trip. Example: `--cookie "session=abc123"`.
          
          [env: WEBFANG_COOKIE=]

Sitemap Settings:
      --sitemap-depth <SITEMAP_DEPTH>
          Maximum recursion depth for sitemap indexes
          
          [env: WEBFANG_SITEMAP_DEPTH=]
          [default: 3]

Competitive Features:
      --checkpoint-interval <CHECKPOINT_INTERVAL>
          Pages between automatic checkpoint saves (0 = disabled) — unified via PersistenceMode with --resume (Checkpoint/Full when enabled)
          
          [env: WEBFANG_CHECKPOINT_INTERVAL=]
          [default: 100]

      --no-checkpoint
          Disable checkpoint persistence entirely — PersistenceMode disables checkpoint (Resume only when combined with --resume)
          
          [env: WEBFANG_NO_CHECKPOINT=]

      --ignore-robots
          Skip robots.txt enforcement
          
          [env: WEBFANG_IGNORE_ROBOTS=]

      --ignore-waf
          Skip WAF/CAPTCHA classification (REQ-WAF-07): challenged responses are reported as plain HTTP errors instead of WAF blocks. It does not rescue the fetch — a challenge page is never scraped as content (F-11).
          
          [env: WEBFANG_IGNORE_WAF=]

      --autoscale
          Enable autoscaled concurrency — dynamically adjusts task concurrency based on RAM usage
          
          [env: WEBFANG_AUTOSCALE=]

      --no-session-health
          Disable session pool health checks
          
          [env: WEBFANG_NO_SESSION_HEALTH=]

      --h2-profile <H2_PROFILE>
          TLS/HTTP2 profile name (default: Chrome145)
          
          [env: WEBFANG_H2_PROFILE=]
          [default: Chrome145]

JS Rendering:
      --js-strategy <JS_STRATEGY>
          JavaScript rendering strategy: static (wreq only), hybrid (3-layer), full (Chromiumoxide only)

          Possible values:
          - static: Static HTTP only (wreq). Fastest, no JS rendering
          - hybrid: Hybrid 3-layer: wreq → Obscura → Chromiumoxide
          - full:   Full JS rendering only (Chromiumoxide). Slowest, handles all SPAs
          
          [env: WEBFANG_JS_STRATEGY=]
          [default: static]

      --js-wait <JS_WAIT>
          Post-load settlement wait for the chromium render path: idle (network-idle, default), <ms> fixed wait (1..=30000), none (capture immediately)
          
          [env: WEBFANG_JS_WAIT=]
          [default: idle]

      --obscura-binary <OBSCURA_BINARY>
          Path to the obscura binary (default: "obscura")
          
          [env: WEBFANG_OBSCURA_BINARY=]
          [default: obscura]

Cleanup:
      --dom-preprune [<DOM_PREPRUNE>]
          Enable DOM pre-pruning before Readability (removes invisible/empty wrappers). Default: enabled (true). Set to false via --dom-preprune=false or WEBFANG_DOM_PREPRUNE=false
          
          [env: WEBFANG_DOM_PREPRUNE=]
          [default: true]
          [possible values: true, false]

Output:
  -o, --output <OUTPUT>
          Output directory for scraped content
          
          [env: WEBFANG_OUTPUT=]
          [default: output]

  -f, --content-format <FORMAT>
          Output format for individual files (markdown, text, json) NOTE: For RAG pipeline export, use --pipeline-format instead

          Possible values:
          - markdown: Markdown format with YAML frontmatter (recommended for RAG)
          - json:     Structured JSON with metadata
          - text:     Plain text without formatting
          
          [env: WEBFANG_FORMAT=]
          [default: markdown]

      --pipeline-format <EXPORT_FORMAT>
          Export format for RAG pipeline (jsonl, vector, auto) NOTE: Use --content-format for output file format (markdown, text, json)

          Possible values:
          - jsonl:  JSONL format (JSON Lines - one JSON object per line) Optimal for RAG pipelines and vector database ingestion
          - vector: Vector format (JSON with metadata header) Supports embeddings and cosine similarity
          - auto:   Auto-detect format from existing export files
          
          [env: WEBFANG_EXPORT_FORMAT=]
          [default: jsonl]

      --retention-days <RETENTION_DAYS>
          Prune exports and vault-DB rows older than N days at the end of a successful run (0 = disabled)
          
          [env: WEBFANG_RETENTION_DAYS=]
          [default: 0]

Elastic Ingestion:
      --cpu-cores <CPU_CORES>
          CPU core override for the elastic ingestion Rayon pool (else auto-detect)
          
          [env: WEBFANG_CPU_CORES=]

      --ram-budget <RAM_BUDGET>
          RAM budget override for the byte-weighted semaphore (`8GB`, `2048MB`, or bytes)
          
          [env: WEBFANG_RAM_BUDGET=]

      --db-path <DB_PATH>
          SQLite database path override for persisted resources/chunks
          
          [env: WEBFANG_DB_PATH=]

      --elastic
          Enable elastic ingestion pipeline (streaming, SQLite dedup, Rayon CPU bridge)
          
          [env: WEBFANG_ELASTIC=]

      --output-vectors <OUTPUT_VECTORS>
          Write extracted vectors to a JSONL file for RAG pipelines. Use `-` for stdout. No SQLite dependency — available in every build (core binary too)
          
          [env: WEBFANG_OUTPUT_VECTORS=]

Batch Processing:
      --batch
          Enable batch mode — scrape each URL from stdin (one per line), one page per URL, no crawling
          
          [env: WEBFANG_BATCH=]

      --batch-file <BATCH_FILE>
          Path to a file containing URLs to scrape (one per line), one page per URL, no crawling
          
          [env: WEBFANG_BATCH_FILE=]

      --batch-concurrency <BATCH_CONCURRENCY>
          Maximum concurrent URLs in batch mode (omit = auto from budget model)
          
          [env: WEBFANG_BATCH_CONCURRENCY=]

      --export-checkpoint <EXPORT_CHECKPOINT>
          Flush captured pages to markdown every N pages during a batch (0 = only at the clean close)
          
          [env: WEBFANG_EXPORT_CHECKPOINT=]
          [default: 0]

Item Pipeline:
      --pipeline
          Enable item pipeline processing (validate → clean → output)
          
          [env: WEBFANG_PIPELINE=]

      --pipeline-output <PIPELINE_OUTPUT>
          Pipeline output format: jsonl (default), none

          Possible values:
          - jsonl: Write items as JSON Lines to a file (default)
          - none:  No pipeline output — items are processed but not written
          
          [env: WEBFANG_PIPELINE_OUTPUT=]
          [default: jsonl]

Obsidian:
      --obsidian-wiki-links
          Convert same-domain links to Obsidian [[wiki-link]] syntax
          
          [env: WEBFANG_OBSIDIAN_WIKI_LINKS=]

      --obsidian-tags <OBSIDIAN_TAGS>
          Tags to include in YAML frontmatter (comma-separated)
          
          [env: WEBFANG_OBSIDIAN_TAGS=]

      --obsidian-relative-assets
          Rewrite downloaded asset paths as relative to the .md file
          
          [env: WEBFANG_OBSIDIAN_RELATIVE_ASSETS=]

      --vault <VAULT>
          Path to Obsidian vault (auto-detects if not provided).
          
          When provided explicitly, the vault becomes the output base: Markdown,
          downloaded assets and the RAG export are written inside it — no need
          to duplicate the path in `-o` (which then must stay at its default).
          Auto-detected or config-file vaults do NOT redirect output (#762).
          
          [env: WEBFANG_OBSIDIAN_VAULT=]

      --quick-save
          Quick-save mode: save directly to vault _inbox folder
          
          [env: WEBFANG_OBSIDIAN_QUICK_SAVE=]

      --obsidian-rich-metadata
          Add rich metadata to frontmatter
          
          [env: WEBFANG_OBSIDIAN_RICH_METADATA=]

AI Settings:
      --threshold <THRESHOLD>
          Relevance threshold for AI semantic filtering (0.0-1.0)
          
          [env: WEBFANG_THRESHOLD=]
          [default: 0.3]

      --max-chars <MAX_CHARS>
          Maximum characters per chunk before rejection, >= 1 (a chunk-size guard, not a context-window setting; the effective ceiling is the embedding backend's own limit, which a remote endpoint enforces server-side as HTTP 400/413)
          
          [env: WEBFANG_MAX_CHARS=]
          [default: 98304]

      --offline
          Reject every embeddings provider (remote or loopback) with exit 78: no local model since 3.0.0
          
          [env: WEBFANG_OFFLINE=]

LLM Extraction:
      --extract-with-llm
          Run structured extraction through the configured LLM provider (requires a provider with the `completion` capability; fails at startup otherwise)
          
          [env: WEBFANG_EXTRACT_WITH_LLM=]

      --llm-provider <LLM_PROVIDER>
          Provider id to use for LLM extraction (default: first provider declaring the `completion` capability)
          
          [env: WEBFANG_LLM_PROVIDER=]

      --embedding-provider <EMBEDDING_PROVIDER>
          Provider id to use for embeddings (default: first provider declaring the `embedding` capability)
          
          [env: WEBFANG_EMBEDDING_PROVIDER=]

Tier 2 Embeddings:
      --tier2-embedding-provider <TIER2_EMBEDDING_PROVIDER>
          Provider id to use for Tier 2 semantic inspector embeddings (default: none; the vault-search provider is never adopted — a remote id requires --tier2-consent)
          
          [env: WEBFANG_TIER2_EMBEDDING_PROVIDER=]

      --tier2-consent
          Explicit consent for Tier 2 remote embeddings egress (required when --tier2-embedding-provider is remote; hash of provider|endpoint logged for audit)
          
          [env: WEBFANG_TIER2_CONSENT=]

EXIT CODES:
  0    Success
  2    No URLs discovered
  3    All scrapers failed
  64   Bad CLI arguments (usage error)
  65   Data format error (extraction without usable content)
  69   Network error, partial success, or SSRF refusal
  74   I/O error
  76   Protocol error
  77   All URLs blocked by robots.txt
  78   Configuration error

EXAMPLES:
  webfang https://example.com
  webfang -u https://example.com
  webfang -u https://example.com --ai
  webfang -u https://example.com -f jsonl
  webfang -u https://example.com -v
  webfang -u https://example.com -vv  # DEBUG
  webfang --batch-file urls.txt --resume
$ webfang completions --help

Generate shell completion scripts

Usage: webfang completions <SHELL>

Arguments:
  <SHELL>
          Shell to generate completions for

          Possible values:
          - bash:        Bash shell completions
          - elvish:      Elvish shell completions
          - fish:        Fish shell completions
          - power-shell: PowerShell completions
          - zsh:         Zsh shell completions

Options:
  -h, --help
          Print help (see a summary with '-h')

Layer 2 (Obscura): a cheap render, not an anti-WAF guarantee

--js-strategy hybrid fetches in three layers: L1 is wreq (static HTTP), L2 is the obscura subprocess, L3 is Chromiumoxide. (--js-strategy full goes straight to L3 and is rejected at preflight by the release binaries, which carry no chromium feature — Release Feature Matrix.)

L2 is accepted as a cheap render — webfang does not require obscura's -stealth build variant, so L2 carries no anti-WAF promise. That is the decision recorded in #1710; this section is where it lives.

What L2 does

  • Runs obscura fetch --dump html once per page — never markdown — with no cookies and no storage directory (the sessionless contract), and marks the result as text/html.
  • Costs ~30 MB per subprocess instead of the ~200 MB + CDP that Layer 3 pays: that is the whole reason it stays in the stack.
  • Escalates to Layer 3 when the page comes back empty or the subprocess fails.

What L2 does not do

  • It is not an anti-WAF guarantee. webfang never passes -stealth, and a WAF challenge detected at any layer aborts the fetch: the router never escalates against a challenge.

Version floor. The hybrid preflight requires obscura 0.2.5 or newer (MINIMUM_OBSCURA_VERSION in crates/webfang_core/src/cli/preflight.rs). An older binary fails the preflight with exit 78 (configuration error), naming the version it found and the version it requires. A binary whose --version probe fails or prints something unparseable is not blocked: the preflight degrades to a warning and the run continues best-effort.

Reading the L2 hit-rate. There are no counters — the hit-rate is derived from the per-layer log events the hybrid router already emits. With a trace file, count the layer events:

webfang https://example.com --js-strategy hybrid --trace-file run.jsonl

# L2 attempts
jq -r 'select(.message | test("Layer 2 \\(Obscura\\): attempting fetch")) | .message' run.jsonl | wc -l
# L2 hits (the render came back with content)
jq -r 'select(.message | test("^Layer 2 returned [0-9]+ bytes$")) | .message' run.jsonl | wc -l
# escalations to L3
jq -r 'select(.message | test("Layer 3 \\(Chromiumoxide\\): attempting fetch")) | .message' run.jsonl | wc -l

The trace file records every level independently of -v (the file layer runs at TRACE level for the webfang target), so the layer events are in it even without -vv. The rest of the query cookbook is in debugging.md.

The four consequences of the decision, recorded so the promise is not re-invented later:

  1. The L2 hit-rate is derived from the per-layer logs — no new counters, and no strict mode.
  2. The preflight's Unknown verdict (an unreadable --version) degrades to a warning instead of failing the run: an unreadable version never blocks the crawl.
  3. The floor in force is 0.2.5. The 0.2.3 ask in #1710 is stale: the upstream release moved on, and the floor follows the current stable.
  4. #1710 stays open. It is commented with this status, not closed.

State directory contents

Companion note to --state-dir (documented in the generated block above). Each --resume run keeps one JSON state file per domain plus a <...>.json.lock sentinel file next to it. The sentinel is created on demand and intentionally never removed: its presence does NOT mean a writer is alive. The real mutual exclusion is the advisory flock held on the sentinel, which the kernel releases when the process exits — including after SIGKILL. There is exactly one sentinel per state file, so the leftover is bounded. Do not delete the .lock file while a run holds it: removing the path breaks the lock and lets two writers enter the critical section at once, losing state updates.