Data Flow & Privacy
This page is the data-flow disclosure: what leaves your machine when you
run webfang, to which destinations, under what conditions, what is stored
locally, and how to turn every flow off. Everything stated here is enforced by
scripts/check_data_flow_doc.sh against the source tree, so it cannot silently
drift from the code.
Quick path: run with zero AI egress
- Do not pass
--clean-aior--extract-with-llm, and do not use the MCP AI tools — no AI request is ever made without an explicit flag or tool call. - Do not configure an AI provider in the providers config — without one, an AI request fails loudly at startup with exit 78, it never falls back to a remote default.
- Run with
--offlineon cleaning paths — every embeddings provider (remote or loopback) is rejected, exit 78, before any endpoint is inspected. - Scraped-site traffic is inherent to crawling: fetching the pages you asked for is the product. Restrict it with the crawl scope flags; every destination is checked by the SSRF guard before a socket exists.
What leaves the machine
| Destination | When | What is sent | How to turn it off |
|---|---|---|---|
| The site being scraped | Every crawl/scrape | HTTP GET requests for the pages (and, with asset download, for images/documents) — browser-like fetches via the TLS-fingerprinted client, or by the real Chrome browser itself under --js-strategy full | Omit the URL(s); scope with crawl flags |
The operator-configured LLM endpoint (POST {base_url}/chat/completions) | --extract-with-llm | Extracted text chunks plus the extraction schema | Omit the flag; or configure no completion provider (exit 78 at startup) |
The operator-configured embeddings endpoint (POST {base_url}/embeddings) | --clean-ai (and the MCP semantic_cleaner tool) | Extracted text chunks (bounded by --max-chars) | --offline; or configure no embedding provider |
There is no other network destination: no telemetry, no metrics, no update check, no crash reporting. Every HTTP fetch in the product goes through the SSRF-guarded downloader or through the two OpenAI-compatible AI adapters above. (The historical Hugging Face / local-ONNX paths were removed — see the ONNX migration page.)
Proxy posture: the proxy environment is ignored by design
WebFang dials directly. The proxy environment variables
(HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXY) are ignored for every
fetch path — crawling, assets, and both AI adapters — on every platform.
This is deliberate, not an omission. The HTTP client honours those variables by default, and under a proxy the guard's validating DNS resolver resolves the proxy's host instead of the destination. Two consequences made silent proxy support unacceptable:
- A corporate proxy on a private range (RFC1918) — the common deployment — is rejected by the SSRF guard before a socket exists, with the operator seeing an SSRF diagnosis instead of a working proxy.
- When the proxy itself is allowed, the destination's IP checks are delegated to the proxy: the client never resolves the destination, so the "forbidden ranges are rejected before the socket exists" guarantee stops covering it.
Where this is pinned: DefaultSsrfGuard::secure_client* applies no_proxy()
at the single client-construction choke point, and
tests/proxy_posture_e2e_test.rs proves that a request with HTTP_PROXY set
reaches the target with zero requests observable at the proxy (both through
the downloader and the webfang binary). webfang doctor states the posture
on every run.
If you cannot dial out directly (mandatory corporate proxy): run a local
forwarding sidecar, point the provider's base_url at it, and opt into
loopback endpoints for that provider — the sidecar, not WebFang, talks to the
corporate proxy. WebFang never reads the proxy environment itself.
The payload contract: extracted text, never raw HTML
The remote model never receives your raw page source. Before any text is sent to an AI endpoint it passes through the text-extraction stage:
- The semantic cleaner extracts readable text from the page HTML — first a
legibility parse, falling back to a plain tag-strip — and then chunks that
text under the
--max-charsbudget. This is the invariant stated in the orchestrator: clean (raw HTML never reaches the LLM). Implementation:application/llm_extraction.rs, andinfrastructure/semantic_cleaner.rs::extract_text_content. - The structured-extraction path (
--extract-with-llm) composes the same cleaner with the chunker, so the completion payload is chunked extracted text plus your schema — never the page's HTML. - Authentication: requests carry an
Authorization: Bearerheader only if the provider config supplies a credential; a provider withauth: { source: "none" }sends an anonymous request. Credentials resolve locally at startup; they are never logged.
Endpoint reachability is enforced by the same fetch-guard chain used for
crawling (entry validation → timeouts → redirect policy → SSRF at the socket
dial → retry classification → capped body read). Loopback embedding endpoints
are rejected unless explicitly opted in, and --offline rejects every
provider before the endpoint is even inspected.
Log and trace content policy
The observability stack (structured logs on stderr, the --trace-file JSONL)
carries operational metadata only:
- What you will find: timestamps, levels, span and target names, URLs, depths, HTTP statuses, durations, pacing waits, error messages with their stage, and correlation identities. The debugging guide documents every span and field.
- What never appears: page bodies, HTML, extracted Markdown, chunk text,
or provider credentials — no
tracingfield in the product crates carries page content. The JSONL is identifiers and outcomes, not payloads. - MCP responses: every tool result that carries third-party text is
neutralized (ANSI/control characters stripped), byte-capped with a visible
truncation marker, and wrapped in a nonce-delimited UNTRUSTED provenance
envelope. See
docs/security/prompt-injection-policy.mdfor the consuming rules. - Trace files are local, ordinary files at the path you choose with
--trace-file. They are never uploaded anywhere; delete them whenever you like.
Local retention
| Path (Linux / XDG) | What lives there | Created by | Removal |
|---|---|---|---|
~/.webfang/crawl.db | SQLite database (crawl results / RAG export) | Any run that persists results; override with WEBFANG_DB_PATH / --db-path | Delete the file |
~/.cache/webfang/state/ | Export state JSON plus the JSON-per-domain record store (one versioned envelope per site) | Resume/checkpoint runs; override with --state-dir | Delete the directory |
<run persistence root>/.webfang-batch-capture.jsonl | The batch spool: every page body the batch fetched, verbatim | Batch runs, until the batch finishes and it is discarded | Discarded at the end of a healthy batch; a crashed run leaves it, and the next batch run sweeps stale spools — or delete it by hand |
~/.config/webfang/ | config.toml and the encrypted credential store credentials.age | Configuration and webfang connect | Delete the directory (also part of uninstall) |
output/ (under your working directory) | Exported Markdown/JSON and downloaded assets | Every run that exports or downloads | Delete at will |
Your --trace-file path | The JSONL trace (metadata only, see above) | Runs with tracing enabled | Delete at will |
Windows/macOS use the platform equivalents documented in installation.md; the paths above follow the standard XDG conventions the platform-paths helper enforces.
SQLite state and the WAL sidecar files (-wal/-shm) have their own
recovery procedure: see the State Recovery Runbook
before deleting a database you still need.
Review checklist
- I know every destination my run can talk to (the table above) and none of them is hard-coded by webfang.
- I can run a full crawl with zero AI egress (Quick path above).
- I understand the AI payload is extracted text chunks, never raw HTML.
- I know where every retained file lives and how to remove each one.