Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Testing Guide

End-to-end (E2E) tests live as integration test crates under tests/ and invoke the real webfang binary via assert_cmd. Mock HTTP servers (wiremock) stand in for target sites and tempfile::TempDir captures scrape output.

Test crates

CrateFileGateWhat it covers
behavioraltests/behavioral/main.rsnone (always built)Single-page scrape, CLI help, unreachable host, slow server, obsidian frontmatter, CSS-selector and full-page extraction (tests/behavioral/cli/)
cli_binarytests/cli_binary_test.rsnone (always built)--version, --help, network-error exit codes

No E2E crate is feature-gated. An earlier revision of this table listed a cli_behavioral crate gated on images and documents; that crate does not exist and that #![cfg(all(feature = "images", feature = "documents"))] gate never existed anywhere in the source tree — its coverage (obsidian tag/metadata/wiki-link conversion) lives ungated inside behavioral, at tests/behavioral/cli/obsidian_test.rs.

The images and documents marker features themselves were deleted in #1813: they selected no capability, and turning them off silently shrank a MIME table instead of shrinking anything else. Do not reintroduce them as empty markers — add a feature when it gates something real.

Running tests

# all E2E crates
cargo nextest run --test behavioral --test cli_binary

# a single crate
cargo nextest run --test behavioral

# a single test (libtest, prints the full snapshot diff on mismatch)
cargo test --test behavioral selector_h3_extracts_only_h3

Ignored tests (e.g. optional live-site checks) are excluded by default; run them with cargo nextest run --test behavioral --run-ignored ignored-only.

Mocking a remote embeddings provider

--clean-ai needs a configured remote embedding provider: ADR-0004 retired the local ONNX stack, and --export-format vector needs --clean-ai (preflight #796). So a test that exercises the vector export has to provide one.

crates/webfang_core/tests/common/embeddings.rs does it hermetically — the provider IS the test's own wiremock server, the credential is anonymous, and nothing leaves the machine:

let t = BehavioralTest::new().await;
crate::common::embeddings::mount_embeddings(&t.server).await;
let provider_config = crate::common::embeddings::provider_config(&t.server);

t.scraper_cmd()
    .env("WEBFANG_CONFIG", &provider_config) // absolute path: the CLI refuses a relative one
    .arg("--embedding-provider")
    .arg(crate::common::embeddings::PROVIDER_ID)
    .arg("--export-format")
    .arg("vector")
    .arg("--clean-ai")
    .assert()
    .success();

Two details the fake has to honor: the endpoint answers one vector per input (the adapter rejects a body whose vector count does not match the text count), and the provider is declared in the config file with auth = { source = "none" } and allow_loopback = true — the loopback opt-in is config-file-only by design (#1462); there is deliberately no CLI flag for it.

Snapshot testing with insta

Content assertions use insta snapshots instead of brittle content.contains(...) checks, so a full output change is reviewed as a diff rather than a silent boolean flip.

Review gate (RED → GREEN)

cargo insta is not installed in this environment. Use the env-var workflow instead:

  1. RED — first run fails because the .snap is missing or differs, and a *.snap.new pending file is written next to it:

    cargo nextest run --test behavioral
    
  2. GREEN — regenerate and accept the pending snapshots, then re-run with no flag to confirm they are now stable (no new *.snap.new should appear):

    INSTA_UPDATE=always cargo nextest run --test behavioral
    cargo nextest run --test behavioral        # must stay green
    
  3. Inspect the generated *.snap files, then stage them with the code change.

*.snap.new is git-ignored (see .gitignore). Never commit a *.snap.new; commit the accepted *.snap.

Where snapshots live

insta resolves the snapshot directory from the module where assert_snapshot! expands. The thin assert_snapshot_* wrappers therefore live at each test crate's root module so snapshots land where the suite expects:

  • tests/behavioral/snapshots/ — root behavioral snapshots
  • tests/behavioral/cli/snapshots/ — obsidian snapshots (local helper inside cli/obsidian_test.rs)
  • tests/snapshots/ — cli_binary__*.snap

Redaction conventions

Scrape output embeds per-run, machine-specific, and non-deterministic values. A shared helper, tests/common/cli_harness.rs::redact_nondeterministic, collapses them before snapshotting so approved snapshots stay stable across machines and runs:

LeakRedacted to
TempDir absolute path<OUT_DIR>
ANSI color escape sequences(stripped)
ISO-8601 timestamps (timestamp_utc, scrapeDate, scrape_date, …) with or without fractional seconds and any offset/Z<TIMESTAMP>
Wiremock 127.0.0.1:<port>127.0.0.1:<PORT>

cli_behavioral additionally emits a bare date: frontmatter field (date only, no time component) that the helper cannot catch, so assert_content_snapshot applies an insta add_filter for date: \d{4}-\d{2}-\d{2} → date: [DATE] (see tests/cli_behavioral_test.rs).

Adding a new snapshot test

  1. Build the scrape output through the shared harness (BehavioralTest / cmd).
  2. Call the crate's assert_snapshot_* wrapper (root module) or, for free-text content, assert_content_snapshot in cli_behavioral.
  3. If a new non-deterministic field appears, extend redact_nondeterministic (centralized) rather than adding a per-test hack.
  4. Generate + accept via INSTA_UPDATE=always, then verify with a plain run.

Lint

cargo clippy -p webfang_core --test behavioral --test cli_binary -- -D warnings

Gate clippy on the specific test crates (not --tests): webfang_core's own lib tests have a pre-existing tokio::time::pause failure that requires the test-util feature and is out of scope for E2E changes.

Coverage exclusions (LCOV)

Defensive error paths — invariants by design — must not drag down the codecov/patch target (80% on new lines). Annotate them with LCOV exclusion markers (issue #527).

Policy

Only annotate arms that "should not happen in normal operation":

  • internal / mutex-poisoning / integer-overflow invariants
  • compile-time-constant failures (CSS selectors, regexes, hardcoded URLs)
  • panic/expect paths guarded by proven invariants (e.g. NonZeroU32 after a zero-check, chunks_exact slice conversion)

NEVER annotate business paths: reachable errors like HTTP connection failures, parse errors, or config validation. Reachable error handling is exercised by tests and counted like any other code.

Syntax

  • Single statement: // LCOV_EXCL_LINE on its OWN comment line immediately ABOVE the code line — never inline on the code line.
  • Multi-line arm/block: // LCOV_EXCL_START above the block and // LCOV_EXCL_STOP below it, each on its own line.
  • Every marker site carries a justification comment starting with // defensive: <variant> <rationale> — merged into the marker line or as a preceding line.

Safety net

Excluded paths are still mutation-tested: a surviving mutant in the weekly cargo-mutants baseline, or in a PR touching gated hot paths (cargo-mutants PR diff), is reported. The markers only affect coverage accounting — they do not affect mutant survival.

Hot-path rule

In files under .cargo/mutants.toml globs, markers MUST be own-line comments (never inline) so the diff adds no mutable code lines.

Never lower codecov.yml thresholds; use markers per path instead.