Testing Guide
End-to-end (E2E) tests live as integration test crates under tests/ and invoke the
real webfang binary via assert_cmd. Mock HTTP servers
(wiremock) stand in for target sites and tempfile::TempDir
captures scrape output.
Test crates
| Crate | File | Gate | What it covers |
|---|---|---|---|
behavioral | tests/behavioral/main.rs | none (always built) | Single-page scrape, CLI help, unreachable host, slow server, obsidian frontmatter, CSS-selector and full-page extraction (tests/behavioral/cli/) |
cli_binary | tests/cli_binary_test.rs | none (always built) | --version, --help, network-error exit codes |
No E2E crate is feature-gated. An earlier revision of this table listed a
cli_behavioral crate gated on images and documents; that crate does not
exist and that #![cfg(all(feature = "images", feature = "documents"))] gate
never existed anywhere in the source tree — its coverage (obsidian
tag/metadata/wiki-link conversion) lives ungated inside behavioral, at
tests/behavioral/cli/obsidian_test.rs.
The images and documents marker features themselves were deleted in
#1813: they selected no capability, and turning them off silently shrank a MIME
table instead of shrinking anything else. Do not reintroduce them as empty
markers — add a feature when it gates something real.
Running tests
# all E2E crates
cargo nextest run --test behavioral --test cli_binary
# a single crate
cargo nextest run --test behavioral
# a single test (libtest, prints the full snapshot diff on mismatch)
cargo test --test behavioral selector_h3_extracts_only_h3
Ignored tests (e.g. optional live-site checks) are excluded by default; run them with
cargo nextest run --test behavioral --run-ignored ignored-only.
Mocking a remote embeddings provider
--clean-ai needs a configured remote embedding provider: ADR-0004 retired the
local ONNX stack, and --export-format vector needs --clean-ai (preflight #796).
So a test that exercises the vector export has to provide one.
crates/webfang_core/tests/common/embeddings.rs does it hermetically — the provider
IS the test's own wiremock server, the credential is anonymous, and nothing
leaves the machine:
let t = BehavioralTest::new().await;
crate::common::embeddings::mount_embeddings(&t.server).await;
let provider_config = crate::common::embeddings::provider_config(&t.server);
t.scraper_cmd()
.env("WEBFANG_CONFIG", &provider_config) // absolute path: the CLI refuses a relative one
.arg("--embedding-provider")
.arg(crate::common::embeddings::PROVIDER_ID)
.arg("--export-format")
.arg("vector")
.arg("--clean-ai")
.assert()
.success();
Two details the fake has to honor: the endpoint answers one vector per input
(the adapter rejects a body whose vector count does not match the text count), and
the provider is declared in the config file with auth = { source = "none" } and
allow_loopback = true — the loopback opt-in is config-file-only by design
(#1462); there is deliberately no CLI flag for it.
Snapshot testing with insta
Content assertions use insta snapshots instead of brittle
content.contains(...) checks, so a full output change is reviewed as a diff rather than
a silent boolean flip.
Review gate (RED → GREEN)
cargo insta is not installed in this environment. Use the env-var workflow instead:
-
RED — first run fails because the
.snapis missing or differs, and a*.snap.newpending file is written next to it:cargo nextest run --test behavioral -
GREEN — regenerate and accept the pending snapshots, then re-run with no flag to confirm they are now stable (no new
*.snap.newshould appear):INSTA_UPDATE=always cargo nextest run --test behavioral cargo nextest run --test behavioral # must stay green -
Inspect the generated
*.snapfiles, then stage them with the code change.
*.snap.newis git-ignored (see.gitignore). Never commit a*.snap.new; commit the accepted*.snap.
Where snapshots live
insta resolves the snapshot directory from the module where assert_snapshot! expands.
The thin assert_snapshot_* wrappers therefore live at each test crate's root module so
snapshots land where the suite expects:
tests/behavioral/snapshots/— rootbehavioralsnapshotstests/behavioral/cli/snapshots/— obsidian snapshots (local helper insidecli/obsidian_test.rs)tests/snapshots/—cli_binary__*.snap
Redaction conventions
Scrape output embeds per-run, machine-specific, and non-deterministic values. A shared
helper, tests/common/cli_harness.rs::redact_nondeterministic, collapses them before
snapshotting so approved snapshots stay stable across machines and runs:
| Leak | Redacted to |
|---|---|
TempDir absolute path | <OUT_DIR> |
| ANSI color escape sequences | (stripped) |
ISO-8601 timestamps (timestamp_utc, scrapeDate, scrape_date, …) with or without fractional seconds and any offset/Z | <TIMESTAMP> |
Wiremock 127.0.0.1:<port> | 127.0.0.1:<PORT> |
cli_behavioral additionally emits a bare date: frontmatter field (date only, no time
component) that the helper cannot catch, so assert_content_snapshot applies an insta
add_filter for date: \d{4}-\d{2}-\d{2} → date: [DATE] (see
tests/cli_behavioral_test.rs).
Adding a new snapshot test
- Build the scrape output through the shared harness (
BehavioralTest/cmd). - Call the crate's
assert_snapshot_*wrapper (root module) or, for free-text content,assert_content_snapshotincli_behavioral. - If a new non-deterministic field appears, extend
redact_nondeterministic(centralized) rather than adding a per-test hack. - Generate + accept via
INSTA_UPDATE=always, then verify with a plain run.
Lint
cargo clippy -p webfang_core --test behavioral --test cli_binary -- -D warnings
Gate clippy on the specific test crates (not --tests): webfang_core's own lib
tests have a pre-existing tokio::time::pause failure that requires the test-util
feature and is out of scope for E2E changes.
Coverage exclusions (LCOV)
Defensive error paths — invariants by design — must not drag down the codecov/patch target (80% on new lines). Annotate them with LCOV exclusion markers (issue #527).
Policy
Only annotate arms that "should not happen in normal operation":
- internal / mutex-poisoning / integer-overflow invariants
- compile-time-constant failures (CSS selectors, regexes, hardcoded URLs)
- panic/expect paths guarded by proven invariants (e.g.
NonZeroU32after a zero-check,chunks_exactslice conversion)
NEVER annotate business paths: reachable errors like HTTP connection failures, parse errors, or config validation. Reachable error handling is exercised by tests and counted like any other code.
Syntax
- Single statement:
// LCOV_EXCL_LINEon its OWN comment line immediately ABOVE the code line — never inline on the code line. - Multi-line arm/block:
// LCOV_EXCL_STARTabove the block and// LCOV_EXCL_STOPbelow it, each on its own line. - Every marker site carries a justification comment starting with
// defensive: <variant> <rationale>— merged into the marker line or as a preceding line.
Safety net
Excluded paths are still mutation-tested: a surviving mutant in the weekly
cargo-mutants baseline, or in a PR touching gated hot paths (cargo-mutants
PR diff), is reported. The markers only affect coverage accounting — they do
not affect mutant survival.
Hot-path rule
In files under .cargo/mutants.toml globs, markers MUST be own-line comments
(never inline) so the diff adds no mutable code lines.
Never lower codecov.yml thresholds; use markers per path instead.