Overview
WebFang is a production-ready web scraper built in Rust (1.91), with Clean Architecture, a TLS-fingerprinting HTTP client, optional AI semantic cleaning (remote embeddings — see the migration guide, ONNX removal), and sitemap-based crawling.
This book is the narrative documentation. The complete API reference for
the workspace crates is generated from source via cargo doc and published
alongside this book under /api/<crate>/ (e.g. /api/webfang_core/).
Crates
| Crate | Role |
|---|---|
webfang_core | Domain, application, and infrastructure layers — the scraping engine (semantic cleaning via remote EmbeddingPort adapters included) |
webfang_mcp | MCP server (36 tools) |
webfang_cli | CLI binary (webfang) |
webfang_test_utils | Shared test helpers |
Installing
WebFang ships binaries only — nothing is published to crates.io, so
cargo install webfang cannot work. Every release publishes three archives
per platform — the CLI, the MCP HTTP server and the MCP stdio server — across
Linux x86_64/ARM64, macOS Apple Silicon and Windows x86_64, plus a
SHA256SUMS.txt. Intel macOS has no artifact by design. Start here:
Installation & Uninstall.
Chapters
- Installation & Uninstall — verified download and checksum commands per platform, platform floors, and every on-disk root a release leaves behind.
- Debugging & Observability — built-in tracing, correlation
IDs, and the
jqquery cookbook fordebug.jsonl. - Testing — E2E integration tests, snapshot strategy, and coverage exclusions.
- Troubleshooting — diagnosing slow crawls, silent failures, WAF blocks, and async deadlocks.
- CLI Reference — the complete
webfangflag reference, auto-generated from the binary itself.
Regenerating this documentation
All heavy compute (mdBook build + rustdoc + link checks) runs in CI. Locally you only pull the generated result:
just docs # downloads the combined NotebookLM/LLM markdown from the latest CI run
just docs-local # local preview: builds and serves this mdBook only