Overview

WebFang is a production-ready web scraper built in Rust (1.88), with Clean Architecture, a TLS-fingerprinting HTTP client, an interactive TUI selector, optional AI semantic cleaning, and sitemap-based crawling.

This book is the narrative documentation. The complete API reference for all six crates is generated from source via cargo doc and published alongside this book under /api/<crate>/ (e.g. /api/webfang_core/).

Crates

Chapters

• Debugging & Observability — built-in tracing, correlation IDs, and the jq query cookbook for debug.jsonl.
• Testing — E2E integration tests, snapshot strategy, and coverage exclusions.
• Troubleshooting — diagnosing slow crawls, silent failures, WAF blocks, and async deadlocks.
• TUI Unified Design — the collapsible config + URL selector design.

Regenerating this documentation

All heavy compute (mdBook build + rustdoc + link checks) runs in CI. Locally you only pull the generated result:

just docs        # downloads the combined NotebookLM/LLM markdown from the latest CI run
just docs-local  # local preview: builds and serves this mdBook only

Debugging & Trace Analysis

WebFang ships built-in, always-available observability. No external collector, no feature flags, no infrastructure: run with --trace-file and post-process the JSONL with jq.

Mandate: every new feature or hot path must be observable. See the "Observability (MANDATORY)" section of AGENTS.md.

The stack

There is no OpenTelemetry (removed in #356). If you need a metric, emit a structured tracing event and query it from the JSONL.

Generating a trace

# Full trace + verbose logging
webfang --url https://example.com --trace-file debug.jsonl -vvv

# Batch / crawl
webfang --url https://example.com --max-pages 100 --trace-file crawl.jsonl -v

Each line of debug.jsonl is a JSON object:

{
  "timestamp": "2026-01-29T10:00:00.123Z",
  "level": "INFO",
  "target": "webfang_core::application::crawler::engine",
  "span": "crawl_page",
  "span_id": "0000000000000042",
  "trace_id": "01949e0e8b8e70008000000000000001",
  "fields": {
    "url": "https://example.com/page1",
    "depth": 1,
    "correlation_id": "00-01949e0e8b8e70008000000000000001-0000000000000042-01"
  }
}

When a span closes, a second record type is emitted carrying a top-level span_duration_ms (wall-clock milliseconds) — this is what the "Slowest spans" query below reads:

{
  "timestamp": "2026-01-29T10:00:00.456Z",
  "record": "span_close",
  "level": "INFO",
  "target": "webfang_core::application::crawler::engine",
  "span": "crawl_page",
  "span_id": "0000000000000042",
  "parent_id": "0000000000000001",
  "trace_id": "0000000000000001",
  "span_duration_ms": 333,
  "span_fields": {
    "url": "https://example.com/page1"
  }
}

Query cookbook

A ready-made script lives at scripts/analyze-trace.sh. The most useful queries:

Reconstruct one operation (crawl / scrape) by trace_id

TRACE=01949e0e8b8e70008000000000000001
jq -c "select(.trace_id == \"$TRACE\" or (.fields.trace_id? // \"\" | contains(\"$TRACE\")))" debug.jsonl

All errors, with full context

jq -c 'select(.level == "ERROR") | {target, url: .fields.url, stage: .fields.stage, error: .fields.error, msg: .fields.message}' debug.jsonl

Slowest spans (where the time goes)

jq -r 'select(.span_duration_ms != null) | [.span_duration_ms, .span] | @tsv' debug.jsonl | sort -rn | head -20

Time distribution per pipeline stage

jq -r 'select(.span == "pipeline_stage") | .fields.stage' debug.jsonl | sort | uniq -c | sort -rn

Crawl progress over time

jq -c 'select(.fields.message? == "crawl progress") | {pages: .fields.pages_crawled, pct: .fields.progress_pct, eta_s: .fields.eta_secs}' debug.jsonl

Final crawl summary

jq -c 'select(.fields.message? == "crawl completed")' debug.jsonl

Count operations by span type

jq -r '.span // "event"' debug.jsonl | sort | uniq -c | sort -rn

URLs that failed

jq -r 'select(.level == "ERROR") | .fields.url // empty' debug.jsonl | sort -u

Spans you will see

Identity follows the root-child contract: the OPERATION owns one root CorrelationId, and every unit of work derives .child() from it — same trace_id, fresh span_id. In a CLI run the orchestrator mints the root and announces it with a run identity event (correlation_id, trace_id in .fields); scrape_multiple_with_limit does the same with a scrape_multiple identity event. So in a multi-page scrape:

• span_fields.trace_id is the shared run-root UUID across all page spans — the whole run is reconstructable by it.
• span_fields.correlation_id (full W3C traceparent) is unique per page; its trace part is the run-root UUID without dashes.

Identity is declared at span creation time because FileTraceLayer snapshots span fields in on_new_span — fields recorded later never reach the JSONL. ScrapedContent and the RAG exports carry the same identity, so an exported document's correlation_id matches its page's span_fields.correlation_id:

# Reconstruct an entire run by the shared run-root trace_id
ROOT=01949e0e-8b8e-7000-8000-000000000001
jq -c "select(.span_fields.trace_id == \"$ROOT\")" debug.jsonl

# The run-root identity (the `run identity` event carries it in .fields)
jq -c 'select(.message? == "run identity") | .fields' debug.jsonl

# Every page identity present in the trace
jq -r '.span_fields.correlation_id // empty' debug.jsonl | sort -u

# Reconstruct one page's scrape by its correlation_id
CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.span_fields.correlation_id? == \"$CID\")" debug.jsonl

Events (not spans): run identity, scrape_multiple identity, crawl progress, crawl completed, and any log_scrape_error(...) error carrying error, url, stage, trace_id.

Concurrency debugging (Tokio Console)

For deadlocks, starved tasks, or async resource leaks, use the Tokio Console:

RUSTFLAGS="--cfg tokio_unstable" cargo run --features console -- --url https://example.com

This opens an interactive TUI showing live tasks, their states, and poll times.

Troubleshooting

See troubleshooting.md for common problems (slow crawls, silent page failures, WAF blocks, async deadlocks, poor content) and how to diagnose each with the trace queries above.

For contributors

When you add a hot path or operation, follow the observability mandate in AGENTS.md:

• #[instrument(skip(...), fields(url = %url, ...))] on the function.
• Propagate the operation's CorrelationId; derive .child() per unit of work.
• Use log_scrape_error(...) on error paths (never a bare warn! for an operational error).
• Use .instrument(span) on async futures — never hold span.enter() across .await.
• Verify with: webfang ... --trace-file debug.jsonl -vvv and the queries above.

Testing Guide

End-to-end (E2E) tests live as integration test crates under tests/ and invoke the real webfang binary via assert_cmd. Mock HTTP servers (wiremock) stand in for target sites and tempfile::TempDir captures scrape output.

Test crates

cli_behavioral is #![cfg(all(feature = "images", feature = "documents"))]. It is built and run by default; with --no-default-features it is skipped entirely (no compile_error!).

Running tests

# all E2E crates
cargo nextest run --test behavioral --test cli_binary --test cli_behavioral

# a single crate
cargo nextest run --test cli_behavioral

# a single test (libtest, prints the full snapshot diff on mismatch)
cargo test --test cli_behavioral test_selector_h3_extracts_only_h3

Ignored tests (e.g. optional live-site checks) are excluded by default; run them with cargo nextest run --test behavioral --run-ignored ignored-only.

Snapshot testing with insta

Content assertions use insta snapshots instead of brittle content.contains(...) checks, so a full output change is reviewed as a diff rather than a silent boolean flip.

Review gate (RED → GREEN)

cargo insta is not installed in this environment. Use the env-var workflow instead:

1. RED — first run fails because the .snap is missing or differs, and a *.snap.new pending file is written next to it:

   cargo nextest run --test cli_behavioral
   

2. GREEN — regenerate and accept the pending snapshots, then re-run with no flag to confirm they are now stable (no new *.snap.new should appear):

   INSTA_UPDATE=always cargo nextest run --test cli_behavioral
   cargo nextest run --test cli_behavioral        # must stay green
   

3. Inspect the generated *.snap files, then stage them with the code change.

*.snap.new is git-ignored (see .gitignore). Never commit a *.snap.new; commit the accepted *.snap.

Where snapshots live

insta resolves the snapshot directory from the module where assert_snapshot! expands. The thin assert_snapshot_* wrappers therefore live at each test crate's root module so snapshots land where the suite expects:

• tests/behavioral/snapshots/ — root behavioral snapshots
• tests/behavioral/cli/snapshots/ — obsidian snapshots (local helper inside cli/obsidian_test.rs)
• tests/snapshots/ — cli_binary__*.snap and cli_behavioral__*.snap

Redaction conventions

Scrape output embeds per-run, machine-specific, and non-deterministic values. A shared helper, tests/common/cli_harness.rs::redact_nondeterministic, collapses them before snapshotting so approved snapshots stay stable across machines and runs:

cli_behavioral additionally emits a bare date: frontmatter field (date only, no time component) that the helper cannot catch, so assert_content_snapshot applies an insta add_filter for date: \d{4}-\d{2}-\d{2} → date: [DATE] (see tests/cli_behavioral_test.rs).

Adding a new snapshot test

1. Build the scrape output through the shared harness (BehavioralTest / cmd).
2. Call the crate's assert_snapshot_* wrapper (root module) or, for free-text content, assert_content_snapshot in cli_behavioral.
3. If a new non-deterministic field appears, extend redact_nondeterministic (centralized) rather than adding a per-test hack.
4. Generate + accept via INSTA_UPDATE=always, then verify with a plain run.

Lint

cargo clippy -p webfang_core --test behavioral --test cli_binary --test cli_behavioral -- -D warnings

Gate clippy on the specific test crates (not --tests): webfang_core's own lib tests have a pre-existing tokio::time::pause failure that requires the test-util feature and is out of scope for E2E changes.

Coverage exclusions (LCOV)

Defensive error paths — invariants by design — must not drag down the codecov/patch target (80% on new lines). Annotate them with LCOV exclusion markers (issue #527).

Policy

Only annotate arms that "should not happen in normal operation":

• internal / mutex-poisoning / integer-overflow invariants
• compile-time-constant failures (CSS selectors, regexes, hardcoded URLs)
• panic/expect paths guarded by proven invariants (e.g. NonZeroU32 after a zero-check, chunks_exact slice conversion)

NEVER annotate business paths: reachable errors like HTTP connection failures, parse errors, or config validation. Reachable error handling is exercised by tests and counted like any other code.

Syntax

• Single statement: // LCOV_EXCL_LINE on its OWN comment line immediately ABOVE the code line — never inline on the code line.
• Multi-line arm/block: // LCOV_EXCL_START above the block and // LCOV_EXCL_STOP below it, each on its own line.
• Every marker site carries a justification comment starting with // defensive: <variant> <rationale> — merged into the marker line or as a preceding line.

Safety net

Excluded paths are still mutation-tested: a surviving mutant in the weekly cargo-mutants baseline, or in a PR touching gated hot paths (cargo-mutants PR diff), is reported. The markers only affect coverage accounting — they do not affect mutant survival.

Hot-path rule

In files under .cargo/mutants.toml globs, markers MUST be own-line comments (never inline) so the diff adds no mutable code lines.

Never lower codecov.yml thresholds; use markers per path instead.

Known Issues

Sitemap Discovery Regression (Pre-existing)

Seven behavioral tests are marked #[ignore] due to a pre-existing crawler regression where auto-discovered sitemaps exit with code 2 on mock-server scenarios. This is NOT related to the insta snapshot migration and was exposed when the root test suite was wired in PR-0 (these tests were previously unwired and never ran).

Affected tests: crawl_test.rs (4 tests), robots_test.rs (1 test), and 2 tests in cli_behavioral_test.rs — all tagged with #[ignore = "Pre-existing stale test, out of scope for insta migration"].

Troubleshooting

Common problems and how to diagnose them with WebFang's built-in tracing.

Generate a trace first: webfang --url <URL> --trace-file debug.jsonl -vvv, then query it with scripts/analyze-trace.sh or jq. See debugging.md for the full query cookbook.

The crawl is slow

Diagnose:

scripts/analyze-trace.sh debug.jsonl slow 20      # slowest spans
scripts/analyze-trace.sh debug.jsonl stages       # time per pipeline stage

Common causes:

• A single stage dominates (e.g. clean with the AI feature) — check the stages distribution.
• Network latency / rate limiting — look for large gaps between crawl_page spans; consider --delay and concurrency tuning.
• Export bottleneck — check export_batch span durations.

Pages are failing silently

Every operational error is logged as a structured ERROR event with url, stage, and (when available) trace_id.

scripts/analyze-trace.sh debug.jsonl errors       # all errors with context
scripts/analyze-trace.sh debug.jsonl urls-failed  # unique failed URLs

Common causes by stage:

WAF / bot detection blocks

scripts/analyze-trace.sh debug.jsonl waf          # WAF challenges + banned domains

If you see WAF challenge detected errors:

• The site is presenting a CAPTCHA / challenge page. WebFang bans the domain for the rest of the crawl to avoid hammering it.
• Try a different TLS fingerprint profile (--tls-emulation) or JS rendering.
• Slow down (--delay, lower concurrency) to avoid rate-limit triggers.

I can't tell which logs belong to one page / one crawl

• One crawl shares a single trace_id. Filter by it:

  scripts/analyze-trace.sh debug.jsonl trace <trace_id>
  

• Each page is a crawl_page span with its own span_id under that trace_id.

Non-deterministic snapshot failures in tests

correlation_id / trace_id are internal and #[serde(skip)] on scraped output, so they never appear in scraped JSON/JSONL snapshots. If a new field you added is non-deterministic (timestamps, ports, temp paths, random IDs), redact it via redact_nondeterministic() in tests/common/cli_harness.rs.

Async deadlocks / starved tasks

For concurrency bugs (a crawl hangs, tasks never complete), use the Tokio Console:

RUSTFLAGS="--cfg tokio_unstable" cargo run --features console -- --url <URL>

This shows live task states and poll times, making stuck tasks visible.

Empty or poor content

• content extraction failed (stage: extract) — the fallback extractor got less than the minimum content. The page is likely JS-rendered, an interactive app, or not an article.
• Try --selector '.main-content' (or the right CSS selector for the site), or enable JS rendering for SPA content.

TUI Unified Design — Collapsible Config + URL Selector

Problem

Two separate flags (--interactive and --config-tui) that should be one flow. Config form covers only 12/45 CLI flags (27%).

Solution: Single --tui flag with two-phase flow

Phase 1: Config Form (collapsible sections)
  ├─ ▶ Target (collapsed by default)
  │    └─ url, selector
  ├─ ▶ Output (collapsed by default)
  │    └─ output, format, export_format
  ├─ ▼ Discovery (expanded — most used)
  │    └─ use_sitemap, sitemap_url, max_pages, max_depth
  ├─ ▶ Crawler (collapsed)
  │    └─ timeout_secs, max_retries, delay_ms, concurrency
  ├─ ▶ Network (collapsed)
  │    └─ user_agent, accept_language, h2_profile, js_strategy
  ├─ ▶ Download (collapsed)
  │    └─ download_images, download_documents, max_file_size
  ├─ ▶ Obsidian (collapsed)
  │    └─ obsidian_wiki_links, obsidian_tags, vault, quick_save, etc.
  ├─ ▶ Advanced (collapsed)
  │    └─ elastic, pipeline, batch, checkpoint, autoscale
  └─ [Start Scraping] button

Phase 2: URL Selector (after config submitted)
  └─ Select which URLs to scrape from discovered list

Collapsible Section Implementation

Since ratatui-accordion is reserved, implement with ratatui's List widget:

struct ConfigSection {
    title: String,
    expanded: bool,
    fields: Vec<FormField>,
}

struct CollapsibleConfig {
    sections: Vec<ConfigSection>,
    cursor: usize,  // which section is focused
}

impl CollapsibleConfig {
    fn handle_key(&mut self, key: KeyEvent) {
        match key.code {
            KeyCode::Up => self.cursor = self.cursor.saturating_sub(1),
            KeyCode::Down => self.cursor = (self.cursor + 1).min(self.sections.len() - 1),
            KeyCode::Enter | KeyCode::Right => self.sections[self.cursor].expanded = true,
            KeyCode::Left => self.sections[self.cursor].expanded = false,
            KeyCode::Char(' ') => self.sections[self.cursor].expanded ^= true,
            _ => {}
        }
    }
    
    fn render(&self, frame: &mut Frame, area: Rect) {
        // Each section renders as:
        // ▶ Section Title          (collapsed)
        // ▼ Section Title          (expanded)
        //   ├─ field1: value
        //   ├─ field2: value
        //   └─ field3: value
    }
}

Field Mapping (45 fields → 8 sections)

Keyboard Navigation

Migration Plan

1. Create src/adapters/tui/collapsible_config.rs — new collapsible form
2. Update src/adapters/tui/config_form.rs — use collapsible sections
3. Update src/main.rs — unify --interactive + --config-tui → --tui
4. Update src/cli/args.rs — replace two flags with one
5. Update src/cli/preflight.rs — handle all 45 fields in merge
6. Add tests for collapsible navigation and field mapping
7. Deprecate old flags (keep for backward compatibility)