Debugging & Trace Analysis
WebFang ships built-in, always-available observability. No external
collector, no feature flags, no infrastructure: run with --trace-file and
post-process the JSONL with jq.
Mandate: every new feature or hot path must be observable. See the "Observability (MANDATORY)" section of
../../AGENTS.md.
The stack
| Layer | What it does | Always on? |
|---|---|---|
| FileTraceLayer | Writes every tracing span/event to a JSONL file (--trace-file) | ✅ Yes |
| Correlation IDs | Native CorrelationId (UUID v7 trace_id + span_id); one trace_id per operation, unique span_id per unit of work | ✅ Yes |
| Structured logging | tracing-subscriber to stderr (-v/-vv/-vvv, --log-format json) | ✅ Yes |
| Tokio Console | Async task/resource inspection for concurrency bugs | --features console |
There is no OpenTelemetry (removed in #356). If you need a metric, emit a structured tracing event and query it from the JSONL.
Generating a trace
# Full trace + verbose logging
webfang --url https://example.com --trace-file debug.jsonl -vvv
# Batch / crawl
webfang --url https://example.com --max-pages 100 --trace-file crawl.jsonl -v
Each line of debug.jsonl is a JSON object:
{
"timestamp": "2026-01-29T10:00:00.123Z",
"level": "INFO",
"target": "webfang_core::application::crawler::engine",
"span": "crawl_page",
"span_id": "0000000000000042",
"parent_id": "0000000000000001",
"trace_id": "0000000000000001",
"fields": {
"url": "https://example.com/page1",
"depth": 1,
"correlation_id": "00-01949e0e8b8e70008000000000000001-0000000000000042-01"
}
}
Top-level trace_id is the root span Id (16-hex), one per run, EPHEMERAL
to the process/run (identity-within-run, not a durable global identity). Do
not persist or join on it across runs. Durable run correlation is the
CorrelationId UUID in span_fields.trace_id /
span_fields.correlation_id (W3C traceparent).
When a span closes, a second record type is emitted carrying a top-level
span_duration_ms (wall-clock milliseconds) — this is what the "Slowest spans"
query below reads:
{
"timestamp": "2026-01-29T10:00:00.456Z",
"record": "span_close",
"level": "INFO",
"target": "webfang_core::application::crawler::engine",
"span": "crawl_page",
"span_id": "0000000000000042",
"parent_id": "0000000000000001",
"trace_id": "0000000000000001",
"span_duration_ms": 333,
"span_fields": {
"url": "https://example.com/page1"
}
}
Query cookbook
A ready-made script lives at scripts/analyze-trace.sh. The most useful
queries:
Reconstruct one run by top-level trace_id
ROOT=0000000000000001
jq -c "select(.trace_id == \"$ROOT\")" debug.jsonl
$ROOT is the 16-hex root span Id (top-level trace_id, EPHEMERAL to the
run). It returns every page plus errors for that run. The same query is
scripts/analyze-trace.sh debug.jsonl trace $ROOT.
All errors, with full context
jq -c 'select(.level == "ERROR") | {target, url: .fields.url, stage: .fields.stage, error: .fields.error, msg: .fields.message}' debug.jsonl
The cause field (connection failures)
The error events carry a cause field: the machine-readable class of a connection
failure, as a stable wire value. It is not a message fragment — the values are a contract:
cause | meaning | observed end to end |
|---|---|---|
dns_resolution | the name never came back: the resolution itself failed or hung | yes |
connect_refused | the peer refused the connection (RST) | yes |
connect_timeout | the name resolved and the TCP connect did not complete in time | yes |
read_timeout | the connection was established and the response never arrived | not yet |
connection_closed | the peer closed the connection before answering (EOF/reset) | not yet |
tls_handshake | the TLS handshake failed | not yet |
unknown | the classifier ran and could not decide — see below | yes |
The three values marked not yet are DECLARED, not measured: the transport does not expose
the evidence they need (an EOF and a stalled response reach the classifier with no readable
kind), so they arrive as unknown today. Their names and their meaning are already fixed —
a value never changes meaning — but they are not expected in a trace until the transport
surfaces the evidence behind them.
The field is the contract; the message is prose. Every classified failure also names its
cause in the message, in Spanish, before the transport's own words. A consumer must still read
the cause field: the message is written for a human and its wording may change, while the
wire value will not.
The field is omitted when the failure carries no classified cause. That absence means "this
failure was not classified" — it does NOT mean "this was not a network failure": a transport
failure whose evidence the wiring cannot read is reported without the field. unknown is the
classified version of that same fact, and it is the value to look for when a network failure
carries no better answer.
unknown is the honest answer, and it is not a shrug. It covers two cases: a chain the
classifier could not read by type, and a name resolution that was still in flight when the
request failed but could not be attributed to that request — typically because another
request for the same host was running at the same time. The rule is deliberately conservative:
a failure is only called dns_resolution when the evidence points at THAT request's own
resolution. With one request in flight per host the attribution is reliable; with several, it
is not, and the value is unknown rather than a wrong cause.
# the failures by cause
jq -r 'select(.fields.cause) | .fields.cause' trace.jsonl | sort | uniq -c | sort -rn
# every failure with its cause, in order
jq -c 'select(.fields.cause) | {url: .fields.url, cause: .fields.cause, error: .fields.error}' trace.jsonl
Slowest spans (where the time goes)
jq -r 'select(.span_duration_ms != null) | [.span_duration_ms, .span] | @tsv' debug.jsonl | sort -rn | head -20
Time distribution per pipeline stage
jq -r 'select(.span == "pipeline_stage") | .fields.stage' debug.jsonl | sort | uniq -c | sort -rn
Crawl progress over time
jq -c 'select(.fields.message? == "crawl progress") | {pages: .fields.pages_crawled, pct: .fields.progress_pct, eta_s: .fields.eta_secs}' debug.jsonl
Final crawl summary
jq -c 'select(.fields.message? == "crawl completed")' debug.jsonl
Count operations by span type
jq -r 'select(.record != "span_close") | .span // "event"' debug.jsonl | sort | uniq -c | sort -rn
span_close records share the same .span name, so they must be excluded
when counting events (otherwise every span is double-counted).
URLs that failed
jq -r 'select(.level == "ERROR") | .fields.url // empty' debug.jsonl | sort -u
Was this run slow, or was it PACED?
Pacing and admission waits emit rate limit wait / semaphore acquire
events at DEBUG level. -vvv is not required for these: the
--trace-file layer always runs at TRACE, so they land in the JSONL
whenever a trace file was requested. They exist because a wait used to be
invisible — a throttled crawl looked exactly like a slow network.
# Total time spent waiting, per pacing scope.
scripts/analyze-trace.sh debug.jsonl pacing
| Scope | Meaning |
|---|---|
crawl_discovery | Per-page pacing inside the crawl engine |
cli_scrape | Per-URL pacing in the CLI scrape phase |
batch_scrape | Per-URL pacing in the batch scrape path |
http_client | Per-session quota in the HTTP client |
A scope with total_waited_ms near zero means pacing was free and the
run's time went to the network. A large cancelled count means shutdown
was shedding queued work, not that the fetches were slow.
Which targets absorbed the wait (each event names the URL whose permit it
was waiting for, and the page's correlation_id):
scripts/analyze-trace.sh debug.jsonl pacing-hot 10
Reconstruct one page's pacing from its own identity:
CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.fields.correlation_id? == \"$CID\" and .message? == \"rate limit wait\")" debug.jsonl
The same wait is also folded into the page's own span, so a page record answers "how much of this page was pacing" without a join:
jq -r 'select(.span == "crawl_page" and .record == "span_close")
| [.span_fields.outcome, .span_fields.rate_limit_wait_ms, .span_duration_ms] | @tsv' \
debug.jsonl | sort -k2 -rn | head
Admission and dispatch queues
scripts/analyze-trace.sh debug.jsonl admission
One row per operation, covering the downloader resource governor
(browser.resource_governor.acquire), the AI inference pool permit
(ai.inference_pool.acquire) and the AI pool dispatch queue
(ai.inference_pool.dispatch). max_in_flight is the pool occupancy the
waiter observed. If max_waited_ms is high and max_in_flight equals the
worker count, the pool was saturated — raise the worker budget or lower the
concurrency feeding it, rather than raising timeouts.
Spans you will see
| Span | Emitted by | Key fields |
|---|---|---|
crawl_site / crawl_site_with_options | crawler::engine | correlation_id, trace_id, seed_url, max_depth, max_pages |
crawl_page | crawler::engine::run_crawl_task | correlation_id, trace_id, url, depth |
execute | pipeline::PipelineExecutor | url, stages |
pipeline_stage | pipeline::PipelineExecutor | stage, url |
export_batch | JsonlExporter / VectorExporter / FileExporter | exporter, documents |
scrape_single_url → scrape_single | crawler::discovery::scrape_single_url | url (outer), correlation_id, trace_id, url (inner, #501) |
scrape_with_config | scraper_service | url, correlation_id, trace_id, has_downloads |
scrape_multiple_with_limit | scraper_service | urls, concurrency |
Identity follows the root-child contract: the OPERATION owns one root
CorrelationId, and every unit of work derives .child() from it — same
trace_id, fresh span_id. In a CLI run the orchestrator mints the root
and announces it with a run identity event (correlation_id, trace_id
in .fields); scrape_multiple_with_limit does the same with a
scrape_multiple identity event. So in a multi-page scrape:
- Top-level
trace_idis the single logical run id: the root spanId(16-hex), EPHEMERAL to the run. Reconstruct the whole run offline withselect(.trace_id == $ROOT)— pages plus errors, no orphans. span_fields.trace_idis the shared run-root UUID (CorrelationId, durable across systems) across all page spans.span_fields.correlation_id(full W3C traceparent) is unique per page; its trace part is the run-root UUID without dashes.
Identity is declared at span creation time: FileTraceLayer snapshots
the span's declared fields in on_new_span, and #[instrument] can only
see function arguments. Fields that are only known at the END of an
operation (an outcome, an HTTP status) are therefore declared
tracing::field::Empty and filled in with tracing::Span::record, which
on_record merges into the same snapshot — they reach the JSONL both in
the span_fields of later events and in the span_close record. ScrapedContent and the RAG exports carry the same identity, so
an exported document's correlation_id matches its page's
span_fields.correlation_id:
# Reconstruct an entire run by the single top-level trace_id (root span Id)
ROOT=0000000000000001
jq -c "select(.trace_id == \"$ROOT\")" debug.jsonl
# Same run by the durable run-root UUID (CorrelationId in span_fields)
CUUID=01949e0e-8b8e-7000-8000-000000000001
jq -c "select(.span_fields.trace_id == \"$CUUID\")" debug.jsonl
# The run-root identity (the `run identity` event carries it in .fields)
jq -c 'select(.message? == "run identity") | .fields' debug.jsonl
# Every page identity present in the trace
jq -r '.span_fields.correlation_id // empty' debug.jsonl | sort -u
# Reconstruct one page's scrape by its correlation_id
CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.span_fields.correlation_id? == \"$CID\")" debug.jsonl
Events (not spans): run identity, scrape_multiple identity,
crawl progress, crawl completed, and any log_scrape_error(...) error
carrying error, url, stage, trace_id.
Concurrency debugging (Tokio Console)
For deadlocks, starved tasks, or async resource leaks, use the Tokio Console:
RUSTFLAGS="--cfg tokio_unstable" cargo run --features console -- --url https://example.com
This opens an interactive TUI showing live tasks, their states, and poll times.
Troubleshooting
See troubleshooting.md for common problems (slow crawls, silent page failures, WAF blocks, async deadlocks, poor content) and how to diagnose each with the trace queries above.
For contributors
When you add a hot path or operation, follow the observability mandate in
AGENTS.md:
#[instrument(skip(...), fields(url = %url, ...))]on the function.- Propagate the operation's
CorrelationId; derive.child()per unit of work. - Use
log_scrape_error(...)on error paths (never a barewarn!for an operational error). - Use
.instrument(span)on async futures — never holdspan.enter()across.await. - Verify with:
webfang ... --trace-file debug.jsonl -vvvand the queries above.
Correlating a wait with the page that paid it
A pacing wait happens BEFORE the page span is created (the permit is taken before any socket opens), so it is emitted as its own event rather than inside the page span. It is still attributable, twice over:
# 1. By identity — the wait carries the same correlation_id the page span does.
CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.fields.correlation_id? == \"$CID\")" debug.jsonl
# 2. By span field — crawl_page records the wait it absorbed.
jq -c 'select(.span == "crawl_page" and .record == "span_close")
| {url: .span_fields.url, outcome: .span_fields.outcome,
paced_ms: .span_fields.rate_limit_wait_ms,
total_ms: .span_duration_ms}' debug.jsonl
A page where paced_ms is a large share of total_ms was queued, not
slow. That distinction is the whole reason the wait emits at all.