Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Overview

WebFang is a production-ready web scraper built in Rust (1.91), with Clean Architecture, a TLS-fingerprinting HTTP client, optional AI semantic cleaning (remote embeddings — see the migration guide, ONNX removal), and sitemap-based crawling.

This book is the narrative documentation. The complete API reference for the workspace crates is generated from source via cargo doc and published alongside this book under /api/<crate>/ (e.g. /api/webfang_core/).

Crates

CrateRole
webfang_coreDomain, application, and infrastructure layers — the scraping engine (semantic cleaning via remote EmbeddingPort adapters included)
webfang_mcpMCP server (36 tools)
webfang_cliCLI binary (webfang)
webfang_test_utilsShared test helpers

Installing

WebFang ships binaries only — nothing is published to crates.io, so cargo install webfang cannot work. Every release publishes three archives per platform — the CLI, the MCP HTTP server and the MCP stdio server — across Linux x86_64/ARM64, macOS Apple Silicon and Windows x86_64, plus a SHA256SUMS.txt. Intel macOS has no artifact by design. Start here: Installation & Uninstall.

Chapters

  • Installation & Uninstall — verified download and checksum commands per platform, platform floors, and every on-disk root a release leaves behind.
  • Debugging & Observability — built-in tracing, correlation IDs, and the jq query cookbook for debug.jsonl.
  • Testing — E2E integration tests, snapshot strategy, and coverage exclusions.
  • Troubleshooting — diagnosing slow crawls, silent failures, WAF blocks, and async deadlocks.
  • CLI Reference — the complete webfang flag reference, auto-generated from the binary itself.

Regenerating this documentation

All heavy compute (mdBook build + rustdoc + link checks) runs in CI. Locally you only pull the generated result:

just docs        # downloads the combined NotebookLM/LLM markdown from the latest CI run
just docs-local  # local preview: builds and serves this mdBook only