Playwright/passthrough 개선 + telemetry + reranker title-injection (v0.2.0 → next)

.env.example

@@ -12,6 +12,10 @@ TRAWL_EMBED_MODEL=bge-m3
 # bge-reranker-v2-m3 served by llama-server with --reranking --pooling rank
 TRAWL_RERANK_URL=http://localhost:8083/v1
 TRAWL_RERANK_MODEL=bge-reranker-v2-m3
+# Include page title + section heading as labelled fields in the
+# reranker input (DeepQSE-style). Default on. Set to 0 to restore
+# the legacy heading-only format.
+# TRAWL_RERANK_INCLUDE_TITLE=1
 
 # ---- HyDE query expansion (optional; off by default) ----
 # Small utility LLM (e.g. Gemma 4B)
@@ -42,3 +46,9 @@ TRAWL_VLM_MAX_TOKENS=2048
 # no chunking, no embedding) up to this many bytes. Default: 262144
 # (256 KB ≈ 64K tokens — fits most local LLM context windows).
 # TRAWL_PASSTHROUGH_MAX_BYTES=262144
+
+# ---- Telemetry (opt-in; used to inform C4 decision) ----
+# Set TRAWL_TELEMETRY=1 to append one JSON line per fetch_relevant() call.
+# TRAWL_TELEMETRY=1
+# TRAWL_TELEMETRY_PATH=~/.cache/trawl/telemetry.jsonl
+# TRAWL_TELEMETRY_MAX_BYTES=67108864

.gitignore

@@ -24,3 +24,6 @@ benchmarks/wcxb/data/
 
 # git worktrees
 .worktrees/
+
+# local notes / research scratchpads (not for upstream)
+notes/

ARCHITECTURE.md

@@ -396,6 +396,22 @@ Full results: `benchmarks/results/`.
 Anything on this list that's not justified by real-usage data is
 speculation. Don't implement speculatively.
 
+## Telemetry (optional)
+
+Opt-in JSONL collector for `fetch_relevant()` calls. Off by default.
+Activated with `TRAWL_TELEMETRY=1`; writes to `~/.cache/trawl/telemetry.jsonl`
+(override with `TRAWL_TELEMETRY_PATH`). Single-generation size rotation
+at `TRAWL_TELEMETRY_MAX_BYTES` (default 64 MB) — older data moves to
+`telemetry.jsonl.1`.
+
+Each line captures host, URL (plaintext), query SHA-1 prefix (query
+plaintext is never stored), fetcher path, profile hit/miss, rerank and
+HyDE flags, and latency/size breakdown. Full schema: see
+`docs/superpowers/specs/2026-04-15-c4-telemetry-design.md`.
+
+Purpose: feed the C4 (`notes/RESEARCH.md`) decision on whether
+index-based extraction as a profile fallback has a problem to solve.
+
 ## Provenance
 
 trawl is the packaged form of work that lived across three spikes:

CLAUDE.md

@@ -81,6 +81,11 @@ trawl directory. Humans should read `README.md` first, then
     take an httpx fast path; suffix-less API endpoints are detected by
     response `Content-Type`. Byte cap via `TRAWL_PASSTHROUGH_MAX_BYTES`
     (default 256 KB).
+  - **Telemetry** (opt-in) — `TRAWL_TELEMETRY=1` appends one JSON line
+    per `fetch_relevant()` call to `~/.cache/trawl/telemetry.jsonl`
+    (override via `TRAWL_TELEMETRY_PATH`). Single-generation rotation
+    at 64 MB. Purpose: feed the C4 decision in `notes/RESEARCH.md`.
+    Schema: `src/trawl/telemetry.py` + the C4 spec doc.
 
 ## Quick Reference
 

Dockerfile

@@ -1,6 +1,9 @@
 # trawl MCP server — HTTP transport, for HTTP-only MCP clients.
 # Base image ships chromium + runtime libs pre-installed for Playwright.
-FROM mcr.microsoft.com/playwright/python:v1.47.0-jammy
+# The tag version MUST match the `playwright==` pin in pyproject.toml —
+# the base image's /ms-playwright/ browsers only work with the matching
+# Python package revision. Bump both together.
+FROM mcr.microsoft.com/playwright/python:v1.58.0-jammy
 
 WORKDIR /app