Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Data Flow & Privacy

This page is the data-flow disclosure: what leaves your machine when you run webfang, to which destinations, under what conditions, what is stored locally, and how to turn every flow off. Everything stated here is enforced by scripts/check_data_flow_doc.sh against the source tree, so it cannot silently drift from the code.


Quick path: run with zero AI egress

  1. Do not pass --clean-ai or --extract-with-llm, and do not use the MCP AI tools — no AI request is ever made without an explicit flag or tool call.
  2. Do not configure an AI provider in the providers config — without one, an AI request fails loudly at startup with exit 78, it never falls back to a remote default.
  3. Run with --offline on cleaning paths — every embeddings provider (remote or loopback) is rejected, exit 78, before any endpoint is inspected.
  4. Scraped-site traffic is inherent to crawling: fetching the pages you asked for is the product. Restrict it with the crawl scope flags; every destination is checked by the SSRF guard before a socket exists.

What leaves the machine

DestinationWhenWhat is sentHow to turn it off
The site being scrapedEvery crawl/scrapeHTTP GET requests for the pages (and, with asset download, for images/documents) — browser-like fetches via the TLS-fingerprinted client, or by the real Chrome browser itself under --js-strategy fullOmit the URL(s); scope with crawl flags
The operator-configured LLM endpoint (POST {base_url}/chat/completions)--extract-with-llmExtracted text chunks plus the extraction schemaOmit the flag; or configure no completion provider (exit 78 at startup)
The operator-configured embeddings endpoint (POST {base_url}/embeddings)--clean-ai (and the MCP semantic_cleaner tool)Extracted text chunks (bounded by --max-chars)--offline; or configure no embedding provider

There is no other network destination: no telemetry, no metrics, no update check, no crash reporting. Every HTTP fetch in the product goes through the SSRF-guarded downloader or through the two OpenAI-compatible AI adapters above. (The historical Hugging Face / local-ONNX paths were removed — see the ONNX migration page.)

Proxy posture: the proxy environment is ignored by design

WebFang dials directly. The proxy environment variables (HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXY) are ignored for every fetch path — crawling, assets, and both AI adapters — on every platform.

This is deliberate, not an omission. The HTTP client honours those variables by default, and under a proxy the guard's validating DNS resolver resolves the proxy's host instead of the destination. Two consequences made silent proxy support unacceptable:

  • A corporate proxy on a private range (RFC1918) — the common deployment — is rejected by the SSRF guard before a socket exists, with the operator seeing an SSRF diagnosis instead of a working proxy.
  • When the proxy itself is allowed, the destination's IP checks are delegated to the proxy: the client never resolves the destination, so the "forbidden ranges are rejected before the socket exists" guarantee stops covering it.

Where this is pinned: DefaultSsrfGuard::secure_client* applies no_proxy() at the single client-construction choke point, and tests/proxy_posture_e2e_test.rs proves that a request with HTTP_PROXY set reaches the target with zero requests observable at the proxy (both through the downloader and the webfang binary). webfang doctor states the posture on every run.

If you cannot dial out directly (mandatory corporate proxy): run a local forwarding sidecar, point the provider's base_url at it, and opt into loopback endpoints for that provider — the sidecar, not WebFang, talks to the corporate proxy. WebFang never reads the proxy environment itself.


The payload contract: extracted text, never raw HTML

The remote model never receives your raw page source. Before any text is sent to an AI endpoint it passes through the text-extraction stage:

  • The semantic cleaner extracts readable text from the page HTML — first a legibility parse, falling back to a plain tag-strip — and then chunks that text under the --max-chars budget. This is the invariant stated in the orchestrator: clean (raw HTML never reaches the LLM). Implementation: application/llm_extraction.rs, and infrastructure/semantic_cleaner.rs::extract_text_content.
  • The structured-extraction path (--extract-with-llm) composes the same cleaner with the chunker, so the completion payload is chunked extracted text plus your schema — never the page's HTML.
  • Authentication: requests carry an Authorization: Bearer header only if the provider config supplies a credential; a provider with auth: { source: "none" } sends an anonymous request. Credentials resolve locally at startup; they are never logged.

Endpoint reachability is enforced by the same fetch-guard chain used for crawling (entry validation → timeouts → redirect policy → SSRF at the socket dial → retry classification → capped body read). Loopback embedding endpoints are rejected unless explicitly opted in, and --offline rejects every provider before the endpoint is even inspected.


Log and trace content policy

The observability stack (structured logs on stderr, the --trace-file JSONL) carries operational metadata only:

  • What you will find: timestamps, levels, span and target names, URLs, depths, HTTP statuses, durations, pacing waits, error messages with their stage, and correlation identities. The debugging guide documents every span and field.
  • What never appears: page bodies, HTML, extracted Markdown, chunk text, or provider credentials — no tracing field in the product crates carries page content. The JSONL is identifiers and outcomes, not payloads.
  • MCP responses: every tool result that carries third-party text is neutralized (ANSI/control characters stripped), byte-capped with a visible truncation marker, and wrapped in a nonce-delimited UNTRUSTED provenance envelope. See docs/security/prompt-injection-policy.md for the consuming rules.
  • Trace files are local, ordinary files at the path you choose with --trace-file. They are never uploaded anywhere; delete them whenever you like.

Local retention

Path (Linux / XDG)What lives thereCreated byRemoval
~/.webfang/crawl.dbSQLite database (crawl results / RAG export)Any run that persists results; override with WEBFANG_DB_PATH / --db-pathDelete the file
~/.cache/webfang/state/Export state JSON plus the JSON-per-domain record store (one versioned envelope per site)Resume/checkpoint runs; override with --state-dirDelete the directory
<run persistence root>/.webfang-batch-capture.jsonlThe batch spool: every page body the batch fetched, verbatimBatch runs, until the batch finishes and it is discardedDiscarded at the end of a healthy batch; a crashed run leaves it, and the next batch run sweeps stale spools — or delete it by hand
~/.config/webfang/config.toml and the encrypted credential store credentials.ageConfiguration and webfang connectDelete the directory (also part of uninstall)
output/ (under your working directory)Exported Markdown/JSON and downloaded assetsEvery run that exports or downloadsDelete at will
Your --trace-file pathThe JSONL trace (metadata only, see above)Runs with tracing enabledDelete at will

Windows/macOS use the platform equivalents documented in installation.md; the paths above follow the standard XDG conventions the platform-paths helper enforces.

SQLite state and the WAL sidecar files (-wal/-shm) have their own recovery procedure: see the State Recovery Runbook before deleting a database you still need.


Review checklist

  • I know every destination my run can talk to (the table above) and none of them is hard-coded by webfang.
  • I can run a full crawl with zero AI egress (Quick path above).
  • I understand the AI payload is extracted text chunks, never raw HTML.
  • I know where every retained file lives and how to remove each one.