AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary
Python ai-promptsai-skillblueskyclaudeclaude-codeclawhubdeep-researchhackernewsinstagramopenclawpolymarketrecencyredditresearchsocial-mediatiktoktrendstwitterweb-searchyoutube
feat(sources): local corpus source - your own files as a ranked signal (#808)
Matt Van Horn3b0b021e2caf
diff --git a/CONFIGURATION.md b/CONFIGURATION.mdindex 3383d45f..d0f2f31a 100644--- a/CONFIGURATION.md+++ b/CONFIGURATION.md@@ -46,6 +46,8 @@ The engine's `.env` reader doesn't expand `$HOME` — only the tilde, via `Path(- `--save-dir <path>` - one-off output location. **Flag wins over env var.** If neither flag nor env var is set, the engine does not write a file (DB persistence is independent — see `LAST30DAYS_STORE` below).- `--output <file>` - write the rendered output to an exact file path, using the format selected by `--emit`.- `--json-profile {agent,raw}` - select the research JSON shape used with `--emit=json`. `agent` is the default, versioned workflow contract; `raw` preserves the full internal `Report` dump for debugging and power users. See the [JSON export reference](docs/reference/json-export.md).+- `--corpus <dir>` - add a local `.md`/`.txt` directory as a private ranked source; repeat the flag for multiple directories. PDFs are extracted only when `pdftotext` is on PATH and otherwise skip with a note. File modification time supplies recency, so the normal research window applies.+- `--corpus-all-time` - include relevant registered files whose modification time is older than the current research window. Without this flag, a 30-day run includes only files modified in those 30 days.- `--register {default,exec,dev,creator,eli5}` - shape a standard single-topic Markdown or HTML research brief for its audience. `exec` is decisions-first with five core findings and numbers up top; `dev` gives GitHub, code, and technical signals more room; `creator` leads with hooks, Best Takes, community reactions, and virality metrics; `eli5` keeps the established evidence layout and asks the synthesizing agent for accessible language. Registers do not change retrieval, JSON exports, discovery, drill, library feed/search, or comparison output.- `--discover <domain>` - topic-less trending discovery. Sweeps rising/top-week Reddit listings (category-mapped communities, with r/all as the uncategorized fallback), Hacker News front/best stories, Digg AI 1000 clusters when `digg-pp-cli` is on PATH, and broad X activity when an X backend is authenticated, then returns 5-10 engagement-velocity-ranked topics. Run without a positional topic; it is mutually exclusive with `--drill`. `--emit=json` uses the separate versioned discovery contract documented in the [JSON export reference](docs/reference/json-export.md).- `--drill <target>` - deep follow-up over the fresh `~/.config/last30days/last-report.json` cache. Accepts a 1-based index (`--drill "cluster 3"` or `--drill "3"`) or a fuzzy cluster title/entity description. It re-fetches only sources that contributed to the matched cluster, enables their deep comment/transcript enrichment paths, merges/dedupes the evidence, and replaces the cache so drills can chain. Run it without a positional topic; if the cache is absent or expired, run a normal research pass first.@@ -98,12 +100,32 @@ Override the global location with `LAST30DAYS_CONFIG_DIR=/path` (or `LAST30DAYS_The project-scoped file is useful for **intentional per-client setups**: drop a `.claude/last30days.env` into each client folder (`SCRAPECREATORS_API_KEY`, `INCLUDE_SOURCES`, `LAST30DAYS_MEMORY_DIR`, `BSKY_HANDLE`, etc), then opt in with `LAST30DAYS_TRUST_PROJECT_CONFIG=1` from your shell or `~/.config/last30days/.env`. Folder-mode hosts such as Codex desktop do not trust hidden project config by default, and discovery stops at the git root so unrelated parent folders cannot silently influence runs.-**`LAST30DAYS_API_KEY`** + **`LAST30DAYS_API_BASE`** - optional remote-API backend. Set BOTH to route research through a remote API endpoint instead of running the local sources: `LAST30DAYS_API_BASE` is the endpoint (there is no built-in default), and `LAST30DAYS_API_KEY` is the bearer key for it. When both are set (and `--mock` is not passed), the engine submits the topic to that endpoint, polls with progress on stderr, and prints the server's report; none of the per-source keys below are used for that run. Non-default `--register` selections are forwarded with the request so server-side synthesis uses the same audience preset. Leave either unset to run local sources exactly as normal. Unlike the other keys here, these two are read only from the **process environment** (export them in your shell or host config) - they are deliberately not loaded from the `.env` files above, so a project-scoped `.env` can never silently redirect research to a remote endpoint. The remote endpoint does not return the local `Report` needed for the versioned agent JSON profile; use `--emit=json --json-profile=raw` for its existing server-response JSON contract.+**`LAST30DAYS_API_KEY`** + **`LAST30DAYS_API_BASE`** - optional remote-API backend. Set BOTH to route research through a remote API endpoint instead of running the local sources: `LAST30DAYS_API_BASE` is the endpoint (there is no built-in default), and `LAST30DAYS_API_KEY` is the bearer key for it. When both are set (and `--mock` is not passed), the engine submits the topic to that endpoint, polls with progress on stderr, and prints the server's report; none of the per-source keys below are used for that run. A configured local corpus is the privacy exception: the engine bypasses the hosted backend and runs locally rather than forwarding file-derived input. Non-default `--register` selections are forwarded with the request so server-side synthesis uses the same audience preset. Leave either unset to run local sources exactly as normal. Unlike the other keys here, these two are read only from the **process environment** (export them in your shell or host config) - they are deliberately not loaded from the `.env` files above, so a project-scoped `.env` can never silently redirect research to a remote endpoint. The remote endpoint does not return the local `Report` needed for the versioned agent JSON profile; use `--emit=json --json-profile=raw` for its existing server-response JSON contract.++### Local corpus (your files)++Register persistent directories with `LAST30DAYS_CORPUS_DIRS`. Separate paths with `:` on macOS/Linux (the platform path separator is `;` on Windows):++```bash+# ~/.config/last30days/.env+LAST30DAYS_CORPUS_DIRS=~/notes:~/meeting-transcripts+# LAST30DAYS_CORPUS_IN_EXPORT=1 # explicit agent-JSON opt-in; off by default+```++The slash-command experience remains primary: ask `/last30days` to include your registered notes. For direct engine scripting or development, the equivalent one-off invocation is:++```bash+python3 skills/last30days/scripts/last30days.py "MCP servers" \+ --corpus ~/notes --corpus ~/meeting-transcripts+```++**Privacy:** corpus files are read locally, never sent through a source HTTP client, never forwarded to `LAST30DAYS_API_BASE`, never included in remote reranker/fun-scoring prompts, and do not consume network-source concurrency or retry budget. Matches appear in a badged **From your files** section. Corpus candidates are removed from `--publish-html`, `library feed --publish`, and the versioned agent JSON export by default, including corpus-derived cluster titles and source outcomes. Set `LAST30DAYS_CORPUS_IN_EXPORT=1` only when you intentionally want corpus results in the agent JSON written to local stdout/files. The unversioned `--json-profile=raw` debug dump remains a full local report and can contain corpus text; do not redirect it to an external system unless that is intentional. Extracted text is cached by file mtime in `~/.config/last30days/corpus-cache.json` with mode `0600`; a corpus-bearing `last-report.json` cache is also tightened to `0600`. Delete either cache at any time to clear it.**Source-by-source** - what each key unlocks:| Source | Key(s) | Required for | Free tier ||---|---|---|---|+| Local corpus | `--corpus <dir>` or `LAST30DAYS_CORPUS_DIRS` | private `.md`/`.txt`; `.pdf` when `pdftotext` is on PATH | yes (offline) || Reddit (public) | none (default); `SCRAPECREATORS_API_KEY` + `LAST30DAYS_REDDIT_BACKEND=scrapecreators` to pin SC primary with public fallback | always on; SC pin requires `SCRAPECREATORS_API_KEY` | yes || Hacker News | none | always on | yes || Polymarket | none | always on | yes |diff --git a/docs/reference/json-export.md b/docs/reference/json-export.mdindex 85d01ff6..e1b77299 100644--- a/docs/reference/json-export.md+++ b/docs/reference/json-export.md@@ -21,6 +21,10 @@ python3 skills/last30days/scripts/last30days.py "AI coding agents" --emit=json -The raw profile is intentionally unversioned and may change when pipeline internals change. It preserves the JSON serialization used before the agent profile was introduced.+### Local corpus privacy++Evidence from `--corpus` / `LAST30DAYS_CORPUS_DIRS` is excluded from the versioned agent profile by default. The exclusion removes corpus results, corpus-only clusters, corpus source outcomes, freshness verdicts, and titles derived from a corpus representative. Set `LAST30DAYS_CORPUS_IN_EXPORT=1` only for a run whose JSON is intentionally allowed to contain local file contents. This opt-in does not change the schema shape or version; it permits `source: "corpus"` entries in the existing result fields. The unversioned `raw` profile is a complete local debug dump and may contain corpus paths and text.+## Discovery exportDiscovery mode has a separate versioned contract so its topic results do not change the normal research export:diff --git a/skills/last30days/SKILL.md b/skills/last30days/SKILL.mdindex 98996fa1..818c7bce 100644--- a/skills/last30days/SKILL.md+++ b/skills/last30days/SKILL.md@@ -416,7 +416,7 @@ Set `LAST30DAYS_MEMORY_DIR` before invoking the skill to choose where raw researThe engine reads `LAST30DAYS_MEMORY_DIR` from either the process env or `~/.config/last30days/.env`, so direct CLI invocations (`python3 scripts/last30days.py ...`) without `--save-dir` will still save when the env var is set. Mirrors the `LAST30DAYS_STORE` env-or-flag convention. Explicit `--save-dir` always wins.-When both `LAST30DAYS_API_KEY` and `LAST30DAYS_API_BASE` are set, the engine runs the research through that configured remote API instead of local sources (unless `--mock` is passed); `LAST30DAYS_API_BASE` is the endpoint and has no built-in default, so leaving either variable unset runs local sources normally. The invocation is unchanged: same flags, `--quick`/`--deep` map to search depth, a non-default `--register` is forwarded for server-side synthesis, progress lines still stream on stderr (`[narrate] step=...` plus a compact elapsed/eta line), and the report prints on stdout and saves to the memory dir as usual, so Steps 1-4 proceed normally on the output. The exception is research JSON: the remote endpoint does not return the local `Report` needed for the versioned agent profile, so use `--emit=json --json-profile=raw` for its existing server-response JSON contract. No per-source keys or setup-wizard credentials are needed for the search itself in this mode. Two engine exits need specific handling: exit code 3 means the API asked a clarifying question first - the engine prints the question and options on stderr; present them to the user and re-run with the chosen angle folded into the topic. An insufficient-credits failure (HTTP 402) prints the account's balance, the amount needed, and a billing link - relay those lines to the user verbatim; do not fall back to WebSearch-only synthesis.+When both `LAST30DAYS_API_KEY` and `LAST30DAYS_API_BASE` are set, the engine runs the research through that configured remote API instead of local sources (unless `--mock` is passed); `LAST30DAYS_API_BASE` is the endpoint and has no built-in default, so leaving either variable unset runs local sources normally. A configured `--corpus` / `LAST30DAYS_CORPUS_DIRS` is the privacy exception: the engine bypasses the hosted backend and runs locally so no file-derived input is forwarded. The invocation is otherwise unchanged: same flags, `--quick`/`--deep` map to search depth, a non-default `--register` is forwarded for server-side synthesis, progress lines still stream on stderr (`[narrate] step=...` plus a compact elapsed/eta line), and the report prints on stdout and saves to the memory dir as usual, so Steps 1-4 proceed normally on the output. The exception is research JSON: the remote endpoint does not return the local `Report` needed for the versioned agent profile, so use `--emit=json --json-profile=raw` for its existing server-response JSON contract. No per-source keys or setup-wizard credentials are needed for the search itself in this mode. Two engine exits need specific handling: exit code 3 means the API asked a clarifying question first - the engine prints the question and options on stderr; present them to the user and re-run with the chosen angle folded into the topic. An insufficient-credits failure (HTTP 402) prints the account's balance, the amount needed, and a billing link - relay those lines to the user verbatim; do not fall back to WebSearch-only synthesis.**Developer-only eval capture:** `--record-fixtures <dir>` is a hidden direct-engine flag for maintaining the deterministic research-quality suite. It records scrubbed HTTP and CLI-adapter responses to `<dir>/http.json`; it is never part of the user-facing slash-command invocation. Follow `docs/reference/eval.md` for fixture review, replay, and baseline rules.@@ -644,10 +644,12 @@ SKILL_DIR="<absolute path of the directory containing the SKILL.md you just Read"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" --diagnose```-`--diagnose` prints JSON. `ACTIVE_SOURCES_LIST` is its `available_sources` array — the engine's authoritative source set, computed after credential resolution. Map the tokens to display names: `reddit`→Reddit, `hackernews`→Hacker News, `polymarket`→Polymarket, `github`→GitHub, `digg`→Digg, `x`→X, `youtube`→YouTube, `tiktok`→TikTok, `instagram`→Instagram, `threads`→Threads, `pinterest`→Pinterest, `linkedin`→LinkedIn, `bluesky`→Bluesky, `perplexity`→Perplexity, `grounding`→Web, `jobs`→Jobs.+`--diagnose` prints JSON. `ACTIVE_SOURCES_LIST` is its `available_sources` array — the engine's authoritative source set, computed after credential resolution. Map the tokens to display names: `reddit`→Reddit, `hackernews`→Hacker News, `polymarket`→Polymarket, `github`→GitHub, `digg`→Digg, `x`→X, `youtube`→YouTube, `tiktok`→TikTok, `instagram`→Instagram, `threads`→Threads, `pinterest`→Pinterest, `linkedin`→LinkedIn, `bluesky`→Bluesky, `perplexity`→Perplexity, `grounding`→Web, `jobs`→Jobs, `corpus`→Your files.- If EXCLUDE_SOURCES is set (comma-separated, case-insensitive): drop any matching source from ACTIVE_SOURCES_LIST before displaying+**Local corpus source:** If the user asks to include their own notes/documents, preserve each supplied directory as a repeatable `--corpus <dir>` engine flag. `LAST30DAYS_CORPUS_DIRS` activates persistent registered directories automatically. Do not WebSearch, upload, quote into a hosted request, or otherwise expose those paths or contents. Corpus retrieval is an offline source lane; its candidates also bypass remote reranker/fun-scoring prompts and use deterministic local scoring. The engine renders matches under the 🔒 **From your files** badge. The normal recency window uses file modification time; add `--corpus-all-time` only when the user explicitly asks to include older files. Corpus evidence is excluded from `--publish-html`, `library feed --publish`, and agent JSON by default. `LAST30DAYS_CORPUS_IN_EXPORT=1` is the explicit agent-JSON privacy opt-in; never enable it on the user's behalf. When a corpus is configured alongside `LAST30DAYS_API_KEY`/`LAST30DAYS_API_BASE`, the engine deliberately bypasses the hosted backend and runs locally.+**Perplexity source:** use it only when the user asks for Perplexity, Deep Research, or paid grounded synthesis, or when `perplexity` is already enabled in `INCLUDE_SOURCES` / `--search`. Direct `PERPLEXITY_API_KEY` supports Sonar synthesis, Search API rows, and async Deep Research. `OPENROUTER_API_KEY` is only a Sonar fallback. Normal runs default to `LAST30DAYS_PERPLEXITY_MODE=sonar`; use `search` for raw ranked web rows, `both` for synthesis plus rows, and `--deep-research` for `sonar-deep-research` with a 600s default wall timeout. A local Deep Research timeout is not a failed API key; inspect the raw artifact's async request id/status and resume by id if needed.**Reddit backend pin:** Reddit defaults to the free public backend with ScrapeCreators as a backup when `SCRAPECREATORS_API_KEY` is available. If the user says public Reddit is shallow, bot-gated, or missing nested comments, tell them they can set `LAST30DAYS_REDDIT_BACKEND=scrapecreators` alongside `SCRAPECREATORS_API_KEY` to make ScrapeCreators primary and keep public Reddit as fallback. Do not set this automatically for normal runs.diff --git a/skills/last30days/scripts/last30days.py b/skills/last30days/scripts/last30days.pyindex d3c05000..dbe5ae63 100644--- a/skills/last30days/scripts/last30days.py+++ b/skills/last30days/scripts/last30days.py@@ -48,7 +48,7 @@ if os.name == "nt":SCRIPT_DIR = Path(__file__).parent.resolve()sys.path.insert(0, str(SCRIPT_DIR))-from lib import dates, env, freshness, html_render, http, permission_preflight, pipeline, registers, render, schema, ui+from lib import corpus, dates, env, freshness, html_render, http, permission_preflight, pipeline, registers, render, schema, ui_child_pids: set[int] = set()_child_pids_lock = threading.Lock()@@ -121,6 +121,34 @@ def slugify(value: str) -> str:return slug or "last30days"+def _report_has_private_corpus(report: schema.Report) -> bool:+ items_by_source = getattr(report, "items_by_source", {})+ if isinstance(items_by_source, dict) and items_by_source.get("corpus"):+ return True+ candidates = getattr(report, "ranked_candidates", ())+ if not isinstance(candidates, (list, tuple)):+ return False+ return any(+ candidate.source == "corpus"+ or any(item.source == "corpus" for item in candidate.source_items)+ for candidate in candidates+ )+++def _ensure_output_directory(path: Path, *, private: bool) -> None:+ if not private:+ path.mkdir(parents=True, exist_ok=True)+ return+ missing: list[Path] = []+ current = path+ while not current.exists():+ missing.append(current)+ current = current.parent+ path.mkdir(parents=True, exist_ok=True, mode=0o700)+ for directory in missing:+ directory.chmod(0o700)++def save_output(report: schema.Report,emit: str,@@ -131,10 +159,10 @@ def save_output(rendered_content: str | None = None,json_profile: str = "agent",register: str = "default",+ private: bool | None = None,) -> Path:from datetime import datetimepath = Path(save_dir).expanduser().resolve()- path.mkdir(parents=True, exist_ok=True)slug = slugify(topic_override or report.topic)extension = "json" if emit == "json" else "html" if emit == "html" else "md"raw_label = "raw-html" if emit == "html" else "raw"@@ -159,10 +187,16 @@ def save_output()else:content = render.render_full(report)+ private_corpus = _report_has_private_corpus(report) or bool(private)+ _ensure_output_directory(path, private=private_corpus)encoded = content.encode("utf-8")for candidate in candidates:try:- fd = os.open(candidate, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)+ fd = os.open(+ candidate,+ os.O_CREAT | os.O_EXCL | os.O_WRONLY,+ 0o600 if private_corpus else 0o644,+ )except FileExistsError:continuewith os.fdopen(fd, "wb") as f:@@ -194,10 +228,25 @@ def save_output()-def save_rendered_output(rendered_content: str, output_file: str) -> Path:+def save_rendered_output(+ rendered_content: str,+ output_file: str,+ *,+ private: bool = False,+) -> Path:out_path = Path(output_file).expanduser().resolve()- out_path.parent.mkdir(parents=True, exist_ok=True)- out_path.write_text(rendered_content, encoding="utf-8")+ _ensure_output_directory(out_path.parent, private=private)+ if private and out_path.exists():+ out_path.chmod(0o600)+ fd = os.open(+ out_path,+ os.O_CREAT | os.O_TRUNC | os.O_WRONLY,+ 0o600 if private else 0o644,+ )+ with os.fdopen(fd, "w", encoding="utf-8") as handle:+ handle.write(rendered_content)+ if private:+ out_path.chmod(0o600)return out_path@@ -384,14 +433,21 @@ def _scoped_store_db(args: argparse.Namespace) -> Path | None:def persist_report(report: schema.Report, store_db: Path | None = None) -> dict[str, int]:import store+ private_corpus = _report_has_private_corpus(report)with store.scoped_db(store_db):+ if private_corpus:+ store.ensure_private_db_files()store.init_db()+ if private_corpus:+ store.ensure_private_db_files()topic_row = store.add_topic(report.topic)topic_id = topic_row["id"]source_mode = ",".join(sorted(report.items_by_source)) or "v3"run_id = store.record_run(topic_id, source_mode=source_mode, status="running")try:findings = store.findings_from_report(report)+ if private_corpus:+ store.ensure_private_db_files()counts = store.store_findings(run_id, topic_id, findings)store.update_run(run_id,@@ -403,6 +459,9 @@ def persist_report(report: schema.Report, store_db: Path | None = None) -> dict[except Exception as exc:store.update_run(run_id, status="failed", error_message=str(exc)[:500])raise+ finally:+ if private_corpus:+ store.ensure_private_db_files()def build_parser() -> argparse.ArgumentParser:@@ -466,6 +525,18 @@ def build_parser() -> argparse.ArgumentParser:parser.add_argument("--no-browser-cookies", action="store_true",help="Disable browser-cookie extraction even when FROM_BROWSER is configured")parser.add_argument("--save-dir", help="Optional directory for saving the rendered output")+ parser.add_argument(+ "--corpus",+ action="append",+ default=[],+ metavar="DIR",+ help="Add a local .md/.txt/.pdf directory as a private ranked source (repeatable)",+ )+ parser.add_argument(+ "--corpus-all-time",+ action="store_true",+ help="Include matching corpus files older than the research window",+ )parser.add_argument("--output", help="Optional exact file path for saving the rendered output")parser.add_argument("--synthesis-file", help="Markdown synthesis to embed in --emit=html output")parser.add_argument("--publish-html", action="store_true",@@ -827,7 +898,12 @@ def _write_last_run(if env.CONFIG_DIR is None:return Falsetarget = env.CONFIG_DIR- target.mkdir(parents=True, exist_ok=True)+ cached_reports = entity_reports or [(report.topic, report)]+ has_private_corpus = any(+ cached_report.items_by_source.get("corpus")+ for _, cached_report in cached_reports+ )+ _ensure_output_directory(target, private=has_private_corpus)counts = {source: len(items) for source, items in report.items_by_source.items()}payload = {"topic": topic,@@ -838,7 +914,6 @@ def _write_last_run("comparison": bool(entity_reports),}(target / "last-run.json").write_text(json.dumps(payload, indent=2))- cached_reports = entity_reports or [(report.topic, report)]cache_payload = {"schema": REPORT_CACHE_VERSION,"topic": topic,@@ -849,7 +924,10 @@ def _write_last_run(for label, cached_report in cached_reports],}- (target / "last-report.json").write_text(json.dumps(cache_payload, indent=2))+ report_cache_path = target / "last-report.json"+ report_cache_path.write_text(json.dumps(cache_payload, indent=2))+ if has_private_corpus:+ report_cache_path.chmod(0o600)return Trueexcept Exception as exc:# Never fatal, but never silent either (#787's lesson): callers that@@ -1105,6 +1183,8 @@ def _run_drill(if "trustpilot" in sources else None),internal_subrun=True,+ corpus_dirs=args.corpus,+ corpus_all_time=args.corpus_all_time,)except Exception:progress.end_processing()@@ -1388,9 +1468,18 @@ def _render_save_and_print(json_profile=args.json_profile,register=audience.name,)+ has_private_corpus = _report_has_private_corpus(report) or bool(+ entity_reports+ and any(_report_has_private_corpus(entity) for _label, entity in entity_reports)+ )+ private_saved_format = has_private_corpuspublish_companion_paths: list[Path] = []if args.output:- output_path = save_rendered_output(rendered, args.output)+ output_path = save_rendered_output(+ rendered,+ args.output,+ private=private_saved_format,+ )if args.emit == "html":publish_companion_paths.append(output_path)sys.stderr.write(f"[last30days] Saved output to {output_path}\n")@@ -1407,6 +1496,7 @@ def _render_save_and_print(rendered_content=rendered if is_comparison_html else None,json_profile=args.json_profile,register=audience.name,+ private=private_saved_format,)if args.emit == "html":publish_companion_paths.append(save_path)@@ -1421,6 +1511,7 @@ def _render_save_and_print(suffix=args.save_suffix or "",synthesis_md=synthesis_md,json_profile=args.json_profile,+ private=_report_has_private_corpus(entity_report),)comparison_peer_paths.append(peer_path)sys.stderr.write(f"[last30days] Saved output to {peer_path}\n")@@ -1432,8 +1523,39 @@ def _render_save_and_print(sys.stderr.flush()if args.publish_html:try:+ has_private_corpus = "corpus" in report.source_status or bool(+ entity_reports+ and any("corpus" in entity.source_status for _label, entity in entity_reports)+ )+ publish_rendered = rendered+ if has_private_corpus:+ sys.stderr.write(+ "[last30days] Excluding local corpus evidence and synthesis from published HTML.\n"+ )+ if entity_reports:+ publish_rendered = emit_comparison_output(+ [+ (label, schema.without_sources(entity, {"corpus"}))+ for label, entity in entity_reports+ ],+ "html",+ fun_level=fun_level,+ save_path=footer_save_path,+ synthesis_md=None,+ json_profile=args.json_profile,+ )+ else:+ publish_rendered = emit_output(+ schema.without_sources(report, {"corpus"}),+ "html",+ fun_level=fun_level,+ save_path=footer_save_path,+ synthesis_md=None,+ json_profile=args.json_profile,+ register=audience.name,+ )publish_result = publish_rendered_html(- rendered,+ publish_rendered,password=_publish_password_for_args(args, config),companion_paths=publish_companion_paths,)@@ -1594,7 +1716,10 @@ def _run_library_feed(args: argparse.Namespace, config: dict[str, object]) -> inoutput_dir.mkdir(parents=True, exist_ok=True)library_id = library.get_or_create_library_id(output_dir)rendered_briefs_dir = output_dir / "briefs"- rendered_briefs_dir.mkdir(parents=True, exist_ok=True)+ has_private_entries = any(+ render.PRIVATE_CORPUS_START in entry.content for entry in entries+ )+ _ensure_output_directory(rendered_briefs_dir, private=has_private_entries)def _preserve_hand_written_page(existing_path: Path, generated_marker: str) -> None:"""Back up any page library feed did not generate before overwriting it."""@@ -1617,13 +1742,19 @@ def _run_library_feed(args: argparse.Namespace, config: dict[str, object]) -> inf"library feed; preserved the original at {backup.name}\n")- brief_documents: dict[str, str] = {}+ publishable_brief_documents: dict[str, str] = {}for entry in entries:rendered = html_render.render_library_brief(entry)target = rendered_briefs_dir / entry.output_name_preserve_hand_written_page(target, html_render.LIBRARY_BRIEF_MARKER)- target.write_text(rendered, encoding="utf-8")- brief_documents[entry.entry_id] = rendered+ save_rendered_output(+ rendered,+ str(target),+ private=render.PRIVATE_CORPUS_START in entry.content,+ )+ publishable_brief_documents[entry.entry_id] = html_render.render_library_brief(+ entry, include_private=False+ )current_brief_names = {entry.output_name for entry in entries}for path in rendered_briefs_dir.glob("*.html"):@@ -1664,7 +1795,7 @@ def _run_library_feed(args: argparse.Namespace, config: dict[str, object]) -> inentry_urls: dict[str, str] = {}try:brief_results = html_publish.publish_html_documents(- brief_documents,+ publishable_brief_documents,password=password,)entry_urls = {@@ -1810,6 +1941,23 @@ def _main()return 2config = env.get_config(policy=_config_policy_for_args(args, topic, extra_argv))+ resolved_corpus_dirs = corpus.resolve_directories(+ args.corpus, config.get("LAST30DAYS_CORPUS_DIRS")+ )+ # EXCLUDE_SOURCES=corpus disables corpus retrieval entirely; the hosted+ # privacy bypass below must use the same predicate, or hosted users with+ # configured-but-excluded dirs silently lose the remote backend.+ excluded_sources = {+ value.strip().lower()+ for value in str(config.get("EXCLUDE_SOURCES") or "").split(",")+ if value.strip()+ }+ if "corpus" in excluded_sources:+ resolved_corpus_dirs = []+ if resolved_corpus_dirs:+ config["_CORPUS_DIRS"] = [str(path) for path in resolved_corpus_dirs]+ if _config_truthy(config.get("LAST30DAYS_CORPUS_IN_EXPORT")):+ config["_CORPUS_IN_EXPORT"] = True_propagate_config_to_environ(config)# Env-var fallback for --save-dir, mirroring the LAST30DAYS_STORE pattern below.@@ -1979,6 +2127,15 @@ def _main(# instead of local sources; no local provider keys are needed (see# lib/hosted.py). With either env var unset, behavior below is byte-identical# to local-only runs - there is no built-in endpoint.+ if (+ topic+ and resolved_corpus_dirs+ and env.read_secret_env("LAST30DAYS_API_KEY")+ and os.environ.get("LAST30DAYS_API_BASE")+ ):+ sys.stderr.write(+ "[last30days] Local corpus configured; bypassing the hosted backend so files stay on this machine.\n"+ )if (topicand not args.diagnose@@ -1986,6 +2143,7 @@ def _main(and not args.record_fixturesand env.read_secret_env("LAST30DAYS_API_KEY")and os.environ.get("LAST30DAYS_API_BASE")+ and not resolved_corpus_dirs):if _freshness_enabled(args, config):if args.verify_freshness is True:@@ -2238,6 +2396,8 @@ def _main(internal_subrun=comp_enabled,hiring_signals_mode=args.hiring_signals,save_dir=args.save_dir,+ corpus_dirs=args.corpus,+ corpus_all_time=args.corpus_all_time,)r.artifacts["resolved"] = {"entity": topic,@@ -2375,6 +2535,8 @@ def _main(hiring_signals_mode=args.hiring_signals,internal_subrun=True,save_dir=args.save_dir,+ corpus_dirs=args.corpus,+ corpus_all_time=args.corpus_all_time,)report.artifacts["resolved"] = resolved_effectivereturn reportdiff --git a/skills/last30days/scripts/lib/corpus.py b/skills/last30days/scripts/lib/corpus.pynew file mode 100644index 00000000..9b4a1c4e--- /dev/null+++ b/skills/last30days/scripts/lib/corpus.py@@ -0,0 +1,439 @@+"""Deterministic, local-only document corpus source.++The corpus adapter deliberately has no HTTP dependency. It scans explicitly+registered directories, extracts small text documents (and PDFs only when the+local ``pdftotext`` binary is available), and returns normalized ``SourceItem``+objects for the shared relevance/fusion pipeline.+"""++from __future__ import annotations++import hashlib+import json+import os+import subprocess+import threading+from dataclasses import dataclass, field+from datetime import datetime, timezone+from pathlib import Path+from shutil import which+from typing import Any, Iterable++from . import entity_extract, log, relevance, schema++SOURCE = "corpus"+SUPPORTED_SUFFIXES = {".md", ".txt", ".pdf"}+IGNORED_DIRECTORIES = {".git", "node_modules"}+MAX_FILES = 500+MAX_TEXT_CHARS = 1_000_000+MAX_CACHE_TEXT_CHARS = MAX_TEXT_CHARS+MAX_CACHE_BYTES = 50 * 1024 * 1024+MAX_CACHE_ENTRIES = 2_000+CACHE_FILENAME = "corpus-cache.json"+CACHE_SCHEMA_VERSION = "last30days-corpus-cache/v2"++_CACHE_LOCK = threading.Lock()+++@dataclass+class CorpusScanResult:+ """One bounded scan, including non-fatal extraction notes."""++ items: list[schema.SourceItem]+ notes: list[str] = field(default_factory=list)+ files_scanned: int = 0+ cache_hits: int = 0+++def resolve_directories(+ cli_directories: Iterable[str] | None,+ configured: str | Iterable[str] | None,+) -> list[Path]:+ """Merge repeatable CLI paths with ``os.pathsep``-separated config paths."""+ raw: list[str] = [str(value) for value in (cli_directories or []) if str(value).strip()]+ if isinstance(configured, str):+ raw.extend(value for value in configured.split(os.pathsep) if value.strip())+ elif configured:+ raw.extend(str(value) for value in configured if str(value).strip())++ resolved: list[Path] = []+ seen: set[str] = set()+ for value in raw:+ path = Path(value.strip()).expanduser().resolve()+ key = os.path.normcase(str(path))+ if key in seen:+ continue+ seen.add(key)+ resolved.append(path)+ return resolved+++def _safe_error(exc: BaseException) -> str:+ """Describe an error without str(exc), which embeds absolute paths.++ These notes travel into source_status detail and render in coverage+ diagnostics outside the private corpus block.+ """+ reason = getattr(exc, "strerror", None)+ return str(reason) if reason else exc.__class__.__name__+++def search(+ topic: str,+ directories: Iterable[Path | str],+ *,+ from_date: str,+ to_date: str,+ all_time: bool = False,+ limit: int = 12,+ cache_dir: Path | None = None,+) -> CorpusScanResult:+ """Search registered directories without making any network calls."""+ roots = resolve_directories([str(path) for path in directories], None)+ notes: list[str] = []+ cache_path = cache_dir / CACHE_FILENAME if cache_dir is not None else None+ with _CACHE_LOCK:+ cache = _load_cache(cache_path)+ cache_entries = cache.setdefault("entries", {})+ cache_entry_sizes = {+ path: _cache_entry_fragment_size(path, value)+ for path, value in cache_entries.items()+ }++ candidates: list[tuple[float, int, schema.SourceItem]] = []+ seen_files: set[str] = set()+ files_scanned = 0+ cache_hits = 0+ pdf_available = which("pdftotext")+ pdf_unavailable_noted = False++ readable_roots: list[Path] = []+ for root in roots:+ if not root.is_dir():+ notes.append(f"Skipped corpus root '{Path(root).name}': not a readable directory")+ continue+ readable_roots.append(root)++ per_root_limit, extra_slots = divmod(MAX_FILES, len(readable_roots) or 1)+ scan_limit_reached = False+ for root_index, root in enumerate(readable_roots):+ root_limit = per_root_limit + (1 if root_index < extra_slots else 0)+ root_files_scanned = 0+ for path in _iter_files(root, notes=notes):+ if root_files_scanned >= root_limit:+ scan_limit_reached = True+ break+ key = os.path.normcase(str(path))+ if key in seen_files:+ continue+ seen_files.add(key)+ root_files_scanned += 1+ files_scanned += 1++ try:+ stat = path.stat()+ except OSError as exc:+ notes.append(f"Skipped {_display_path(path, root)}: {_safe_error(exc)}")+ continue+ published_at = datetime.fromtimestamp(+ stat.st_mtime, tz=timezone.utc+ ).date().isoformat()+ if not all_time and not (from_date <= published_at <= to_date):+ continue++ cached = cache_entries.get(str(path))+ if (+ isinstance(cached, dict)+ and cached.get("mtime_ns") == stat.st_mtime_ns+ and cached.get("size") == stat.st_size+ and isinstance(cached.get("text"), str)+ ):+ text = cached["text"]+ cache_hits += 1+ else:+ if path.suffix.lower() == ".pdf" and not pdf_available:+ if not pdf_unavailable_noted:+ notes.append("Skipped PDF files because pdftotext is not on PATH")+ pdf_unavailable_noted = True+ continue+ try:+ text = _extract_text(path, pdftotext=pdf_available)+ except (OSError, subprocess.SubprocessError) as exc:+ notes.append(f"Skipped {_display_path(path, root)}: {_safe_error(exc)}")+ continue+ _cache_entry_put(cache_entries, cache_entry_sizes, str(path), {+ "mtime_ns": stat.st_mtime_ns,+ "size": stat.st_size,+ "text": text[:MAX_CACHE_TEXT_CHARS],+ })++ title = _path_title(path)+ score = _match_score(topic, f"{title}\n{text}")+ if score < 0.15:+ continue+ relative_path = str(path.relative_to(root))+ path_digest = hashlib.sha256(str(path).encode("utf-8")).hexdigest()+ item = schema.SourceItem(+ item_id=f"C{path_digest[:12]}",+ source=SOURCE,+ title=title,+ body=text,+ url=f"corpus://{path_digest}",+ container=str(path.parent),+ published_at=published_at,+ date_confidence="high",+ relevance_hint=score,+ why_relevant=f"Matched local file {relative_path}",+ # Leave empty so extract_best_snippet derives the matching+ # window; a file-prefix snippet is preserved verbatim and can+ # show unrelated intro text (and draw entity-miss demotion).+ snippet="",+ metadata={+ "path": str(path),+ "relative_path": relative_path,+ "extension": path.suffix.lower(),+ "local_only": True,+ },+ )+ candidates.append((score, stat.st_mtime_ns, item))+ if scan_limit_reached:+ notes.append(f"Stopped after the {MAX_FILES}-file corpus scan limit")++ cache["schema_version"] = CACHE_SCHEMA_VERSION+ cache["entries"] = _bounded_entries(cache_entries)+ with _CACHE_LOCK:+ _write_cache(cache_path, cache, notes)++ candidates.sort(key=lambda row: (-row[0], -row[1], row[2].title.casefold()))+ items = [item for _score, _mtime, item in candidates[: max(0, limit)]]+ log.source_log(+ "Corpus",+ f"scanned {files_scanned} file(s), {cache_hits} cache hit(s), {len(items)} match(es)",+ tty_only=False,+ )+ return CorpusScanResult(+ items=items,+ notes=notes,+ files_scanned=files_scanned,+ cache_hits=cache_hits,+ )+++def _display_path(path: Path | str, root: Path | None = None) -> str:+ """Render a note-safe path: never the absolute local path.++ Corpus notes flow into source_status detail and the Partial Coverage+ block, which render OUTSIDE the private corpus markers - an absolute+ path like /home/user/private/notes/foo.md must not escape there.+ """+ candidate = Path(path)+ if root is not None:+ try:+ return str(Path(root).name / candidate.relative_to(root))+ except ValueError:+ pass+ return candidate.name+++def _iter_files(root: Path, notes: list[str] | None = None) -> Iterable[Path]:+ # Bounded newest-first selection: keep only the newest MAX_FILES paths in a+ # heap while walking, so registering a huge tree does not materialize every+ # path before the caller's extraction cap applies.+ import heapq++ heap: list[tuple[int, str]] = []+ walk_errors = 0++ def _on_walk_error(error: OSError) -> None:+ nonlocal walk_errors+ walk_errors += 1+ if notes is not None and walk_errors <= 3:+ unreadable = _display_path(error.filename, root) if error.filename else Path(root).name+ notes.append(f"corpus: could not read {unreadable}: {error.strerror}")++ for current, directory_names, file_names in os.walk(+ root, followlinks=False, onerror=_on_walk_error+ ):+ directory_names[:] = sorted(+ name+ for name in directory_names+ if name not in IGNORED_DIRECTORIES and not name.startswith(".")+ )+ current_path = Path(current)+ for name in sorted(file_names):+ if name.startswith("."):+ continue+ path = current_path / name+ if path.suffix.lower() in SUPPORTED_SUFFIXES and not path.is_symlink():+ entry = (_safe_mtime_ns(path), str(path))+ if len(heap) < MAX_FILES:+ heapq.heappush(heap, entry)+ else:+ heapq.heappushpop(heap, entry)+ if notes is not None and walk_errors > 3:+ notes.append(f"corpus: {walk_errors - 3} more unreadable directories suppressed")+ ordered = sorted(heap, key=lambda item: (-item[0], item[1].casefold()))+ for _mtime, raw_path in ordered:+ yield Path(raw_path)+++def _safe_mtime_ns(path: Path) -> int:+ try:+ return path.stat().st_mtime_ns+ except OSError:+ return 0+++def _extract_text(path: Path, *, pdftotext: str | None) -> str:+ if path.suffix.lower() == ".pdf":+ if not pdftotext:+ return ""+ completed = subprocess.run(+ [pdftotext, str(path), "-"],+ capture_output=True,+ check=True,+ text=True,+ timeout=20,+ )+ return completed.stdout[:MAX_TEXT_CHARS]+ with path.open("r", encoding="utf-8", errors="replace") as handle:+ return handle.read(MAX_TEXT_CHARS)+++def _path_title(path: Path) -> str:+ title = path.stem.replace("_", " ").replace("-", " ")+ return " ".join(title.split()) or path.name+++def _match_score(topic: str, text: str) -> float:+ lexical = relevance.token_overlap_relevance(topic, text)+ topic_entities = entity_extract.extract_text_entities(topic)+ text_entities = entity_extract.extract_text_entities(text)+ entity_score = entity_extract.entity_overlap(topic_entities, text_entities)+ return round(max(lexical, entity_score * 0.9), 4)+++def _load_cache(path: Path | None) -> dict[str, Any]:+ if path is None:+ return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}+ try:+ if path.stat().st_size > MAX_CACHE_BYTES:+ return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}+ payload = json.loads(path.read_text(encoding="utf-8"))+ except (OSError, UnicodeError, json.JSONDecodeError):+ return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}+ if not isinstance(payload, dict) or payload.get("schema_version") != CACHE_SCHEMA_VERSION:+ return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}+ if not isinstance(payload.get("entries"), dict):+ payload["entries"] = {}+ payload["entries"] = _bounded_entries(payload["entries"])+ return payload+++def _bounded_entries(entries: Any) -> dict[str, Any]:+ if not isinstance(entries, dict):+ return {}+ ordered = sorted(+ (+ (path, value)+ for path, value in entries.items()+ if (+ isinstance(path, str)+ and isinstance(value, dict)+ and isinstance(value.get("text"), str)+ )+ ),+ key=lambda row: int(row[1].get("mtime_ns") or 0),+ reverse=True,+ )+ base_bytes = len(+ json.dumps(+ {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}},+ ensure_ascii=False,+ ).encode("utf-8")+ )+ used_bytes = base_bytes+ bounded: dict[str, Any] = {}+ for path, value in ordered[:MAX_CACHE_ENTRIES]:+ normalized = {+ "mtime_ns": value.get("mtime_ns"),+ "size": value.get("size"),+ "text": value["text"][:MAX_CACHE_TEXT_CHARS],+ }+ fragment = json.dumps({path: normalized}, ensure_ascii=False).encode("utf-8")+ fragment_bytes = len(fragment) - 2 + (2 if bounded else 0)+ if used_bytes + fragment_bytes > MAX_CACHE_BYTES:+ continue+ bounded[path] = normalized+ used_bytes += fragment_bytes+ return bounded+++def _cache_entry_fragment_size(path: str, value: dict[str, Any]) -> int:+ return len(json.dumps({path: value}, ensure_ascii=False).encode("utf-8")) - 2+++def _cache_entry_put(+ entries: dict[str, Any],+ sizes: dict[str, int],+ path: str,+ value: dict[str, Any],+) -> None:+ entries[path] = value+ sizes[path] = _cache_entry_fragment_size(path, value)+ while (+ len(entries) > MAX_CACHE_ENTRIES+ or _cache_payload_size(sizes) > MAX_CACHE_BYTES+ ):+ oldest = min(+ entries,+ key=lambda candidate: (+ int(entries[candidate].get("mtime_ns") or 0),+ candidate,+ ),+ )+ del entries[oldest]+ del sizes[oldest]+++def _cache_payload_size(sizes: dict[str, int]) -> int:+ base_bytes = len(+ json.dumps(+ {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}},+ ensure_ascii=False,+ ).encode("utf-8")+ )+ separators = max(0, len(sizes) - 1) * 2+ return base_bytes + sum(sizes.values()) + separators+++def _write_cache(path: Path | None, payload: dict[str, Any], notes: list[str]) -> None:+ if path is None:+ return+ try:+ _ensure_private_directory(path.parent)+ payload["entries"] = _bounded_entries(payload.get("entries", {}))+ encoded = json.dumps(payload, ensure_ascii=False).encode("utf-8")+ temporary = path.with_name(f".{path.name}.{os.getpid()}.tmp")+ try:+ fd = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)+ except FileExistsError:+ temporary.unlink()+ fd = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)+ with os.fdopen(fd, "wb") as handle:+ handle.write(encoded)+ temporary.replace(path)+ path.chmod(0o600)+ except OSError as exc:+ notes.append(f"Corpus cache unavailable: {_safe_error(exc)}")+++def _ensure_private_directory(path: Path) -> None:+ missing: list[Path] = []+ current = path+ while not current.exists():+ missing.append(current)+ current = current.parent+ path.mkdir(parents=True, exist_ok=True, mode=0o700)+ for directory in missing:+ directory.chmod(0o700)diff --git a/skills/last30days/scripts/lib/env.py b/skills/last30days/scripts/lib/env.pyindex fe95b6db..19ce59e6 100644--- a/skills/last30days/scripts/lib/env.py+++ b/skills/last30days/scripts/lib/env.py@@ -470,6 +470,12 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:# degraded (neither ok, no-results, nor skipped-unconfigured). #384.('LAST30DAYS_STRICT_EXIT', None),('LAST30DAYS_MEMORY_DIR', None),+ # Optional local-only evidence source. Paths are separated with the+ # platform path separator (":" on macOS/Linux, ";" on Windows).+ ('LAST30DAYS_CORPUS_DIRS', None),+ # Corpus evidence is omitted from the stable agent JSON export unless+ # this explicit privacy opt-in is truthy.+ ('LAST30DAYS_CORPUS_IN_EXPORT', None),('LAST30DAYS_LIBRARY_OWNER', None),('LAST30DAYS_LIBRARY_CONTEXT', 'on'),('LAST30DAYS_PUBLISH_PASSWORD', None),diff --git a/skills/last30days/scripts/lib/html_render.py b/skills/last30days/scripts/lib/html_render.pyindex db2ac127..ea599032 100644--- a/skills/last30days/scripts/lib/html_render.py+++ b/skills/last30days/scripts/lib/html_render.py@@ -438,10 +438,12 @@ def render_html_comparison(LIBRARY_BRIEF_MARKER = "<!-- generated by last30days library feed -->"-def render_library_brief(entry: LibraryEntry) -> str:+def render_library_brief(entry: LibraryEntry, *, include_private: bool = True) -> str:"""Render a scanned Markdown or JSON briefing as a safe standalone page."""md = _strip_invitation(entry.content)md = _strip_canonical_boundary(md)+ if not include_private:+ md = _strip_private_corpus(md)body = _markdown_to_html(md)body = _wrap_engine_footer(body)colophon = (@@ -458,6 +460,18 @@ def render_library_brief(entry: LibraryEntry) -> str:return rendered.replace("</body>", f"{LIBRARY_BRIEF_MARKER}\n</body>", 1)+_PRIVATE_CORPUS_BLOCK = re.compile(+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_START -->.*?"+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_END -->\s*",+ re.DOTALL,+)+++def _strip_private_corpus(markdown: str) -> str:+ """Remove the renderer-marked local corpus section before publication."""+ return _PRIVATE_CORPUS_BLOCK.sub("", markdown)++def render_library_index(entries: Sequence[LibraryEntry],*,diff --git a/skills/last30days/scripts/lib/library.py b/skills/last30days/scripts/lib/library.pyindex 38bd2348..96a581fc 100644--- a/skills/last30days/scripts/lib/library.py+++ b/skills/last30days/scripts/lib/library.py@@ -29,6 +29,11 @@ _LIBRARY_ID = re.compile(r"[0-9a-f]{32}")_GENERATED_BRIEF_NAME = re.compile(r"[a-z0-9]+(?:-[a-z0-9]+)*-[0-9a-f]{8}-\d{4}-\d{2}-\d{2}\.html")+_PRIVATE_CORPUS_BLOCK = re.compile(+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_START -->.*?"+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_END -->\s*",+ re.DOTALL,+)@dataclass(frozen=True, slots=True)@@ -140,15 +145,16 @@ def _keep_preferred(entries: dict[str, LibraryEntry], entry: LibraryEntry) -> Nodef _parse_markdown(path: Path) -> LibraryEntry:content = path.read_text(encoding="utf-8")- title_match = _REPORT_TITLE.search(content) or _FIRST_TITLE.search(content)+ public_content = _PRIVATE_CORPUS_BLOCK.sub("", content)+ title_match = _REPORT_TITLE.search(public_content) or _FIRST_TITLE.search(public_content)if not title_match:raise ValueError("no Markdown title found")topic = _clean_inline(title_match.group(1))if not topic:raise ValueError("empty Markdown title")- published_date = _markdown_date(content, path)- headline = _markdown_headline(content) or topic- summary = _markdown_summary(content) or headline+ published_date = _markdown_date(public_content, path)+ headline = _markdown_headline(public_content) or topic+ summary = _markdown_summary(public_content) or headlinereturn LibraryEntry(slug=slugify(topic),topic=topic,diff --git a/skills/last30days/scripts/lib/library_index.py b/skills/last30days/scripts/lib/library_index.pyindex 30b57d66..31b6f0d8 100644--- a/skills/last30days/scripts/lib/library_index.py+++ b/skills/last30days/scripts/lib/library_index.py@@ -3,6 +3,7 @@from __future__ import annotationsimport hashlib+import osimport reimport sqlite3from dataclasses import dataclass, replace@@ -14,6 +15,7 @@ from . import libraryDEFAULT_LIBRARY_DB = library.DEFAULT_BRIEFS_DIR.parent / "library.db"DEFAULT_STORE_DB = library.DEFAULT_BRIEFS_DIR.parent / "research.db"+INDEX_FINGERPRINT_VERSION = "last30days-library-index/v2"LIBRARY_CONTEXT_START = "<!-- last30days:library-context:start -->"LIBRARY_CONTEXT_END = "<!-- last30days:library-context:end -->"_TOKEN = re.compile(r"[^\W_]+", re.UNICODE)@@ -26,6 +28,11 @@ _LEGACY_LIBRARY_CONTEXT = re.compile(r"^## From your library\s*$.*?(?=^##\s|\Z)",re.MULTILINE | re.DOTALL,)+_PRIVATE_CORPUS_BLOCK = re.compile(+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_START -->.*?"+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_END -->\s*",+ re.DOTALL,+)class LibrarySearchUnavailable(RuntimeError):@@ -131,7 +138,7 @@ def index_brief(if entry is None:return Falsetarget = Path(db_path).expanduser()- target.parent.mkdir(parents=True, exist_ok=True)+ _ensure_private_directory(target.parent)with _connect(target) as conn:_upsert_entry(conn, entry)conn.commit()@@ -209,7 +216,7 @@ def _sync_library(db_path: Path,) -> SyncResult:entries, notes = library.scan_library(memory_dir, briefs_dir)- db_path.parent.mkdir(parents=True, exist_ok=True)+ _ensure_private_directory(db_path.parent)indexed = unchanged = 0with _connect(db_path) as conn:existing = {@@ -250,6 +257,15 @@ def _sync_library(def _connect(path: Path) -> sqlite3.Connection:+ _ensure_private_directory(path.parent)+ if not path.exists():+ try:+ fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)+ except FileExistsError:+ pass+ else:+ os.close(fd)+ path.chmod(0o600)conn = sqlite3.connect(str(path))try:conn.row_factory = sqlite3.Row@@ -268,7 +284,13 @@ def _upsert_entry(fingerprint: str | None = None,) -> None:stat = entry.source_path.stat()- indexed_content = _indexable_content(entry.content)+ private_free_content = _PRIVATE_CORPUS_BLOCK.sub("", entry.content)+ indexed_content = _indexable_content(private_free_content)+ headline = entry.headline+ summary = entry.summary+ if private_free_content != entry.content and entry.source_format == "markdown":+ headline = library._markdown_headline(private_free_content) or entry.topic+ summary = library._markdown_summary(private_free_content) or headlinecontent_hash = fingerprint or _fingerprint(indexed_content)source_path = str(entry.source_path.resolve())replaced = conn.execute(@@ -302,14 +324,14 @@ def _upsert_entry(content_hash,entry.topic,entry.published_date.isoformat(),- entry.headline,- entry.summary,+ headline,+ summary,entry.source_format,),)conn.execute("INSERT INTO library_fts(entry_id, topic, headline, summary, content) VALUES (?, ?, ?, ?, ?)",- (entry.entry_id, entry.topic, entry.headline, entry.summary, indexed_content),+ (entry.entry_id, entry.topic, headline, summary, indexed_content),)@@ -334,6 +356,7 @@ def _search_store_sightings(JOIN research_runs rr ON rr.id = fs.run_idJOIN topics t ON t.id = fs.topic_idWHERE findings_fts MATCH ? AND rr.status = 'completed'+ AND fs.source != 'corpus'ORDER BY rank, rr.run_date DESCLIMIT ?""",(expression, limit),@@ -371,7 +394,8 @@ def _fts_expression(query: str) -> str:def _fingerprint(content: str) -> str:- return hashlib.sha256(content.encode("utf-8")).hexdigest()+ payload = f"{INDEX_FINGERPRINT_VERSION}\0{content}"+ return hashlib.sha256(payload.encode("utf-8")).hexdigest()def _clean_snippet(value: object) -> str:@@ -379,10 +403,22 @@ def _clean_snippet(value: object) -> str:def _indexable_content(content: str) -> str:- without_marked = _MARKED_LIBRARY_CONTEXT.sub("", content)+ without_private = _PRIVATE_CORPUS_BLOCK.sub("", content)+ without_marked = _MARKED_LIBRARY_CONTEXT.sub("", without_private)return _LEGACY_LIBRARY_CONTEXT.sub("", without_marked)+def _ensure_private_directory(path: Path) -> None:+ missing: list[Path] = []+ current = path+ while not current.exists():+ missing.append(current)+ current = current.parent+ path.mkdir(parents=True, exist_ok=True, mode=0o700)+ for directory in missing:+ directory.chmod(0o700)++def _is_confirmed_corruption(exc: sqlite3.DatabaseError) -> bool:message = str(exc).casefold()return any(diff --git a/skills/last30days/scripts/lib/pipeline.py b/skills/last30days/scripts/lib/pipeline.pyindex b76a936a..0ea8e6b1 100644--- a/skills/last30days/scripts/lib/pipeline.py+++ b/skills/last30days/scripts/lib/pipeline.py@@ -19,6 +19,7 @@ from . import (arxiv,bird_x,bluesky,+ corpus,dates,dedupe,digg,@@ -125,6 +126,7 @@ MOCK_AVAILABLE_SOURCES = ["trustpilot","jobs","linkedin",+ "corpus",]@@ -156,6 +158,10 @@ def available_sources(available: list[str] = []# reddit_public needs no API key - always availableavailable.append("reddit")+ if corpus.resolve_directories(+ config.get("_CORPUS_DIRS"), config.get("LAST30DAYS_CORPUS_DIRS")+ ):+ available.append("corpus")if config.get("SCRAPECREATORS_API_KEY"):available.extend(["tiktok", "instagram"])if env.get_x_source(config, local_only=local_only):@@ -889,10 +895,25 @@ def run(hiring_signals_mode: bool = False,internal_subrun: bool = False,save_dir: Path | str | None = None,+ corpus_dirs: list[str] | None = None,+ corpus_all_time: bool = False,) -> schema.Report:settings = DEPTH_SETTINGS[depth]requested_sources = normalize_requested_sources(requested_sources)from_date, to_date = dates.get_date_range(lookback_days, as_of_date=as_of_date)+ resolved_corpus_dirs = corpus.resolve_directories(+ corpus_dirs or config.get("_CORPUS_DIRS"),+ config.get("LAST30DAYS_CORPUS_DIRS"),+ )+ excluded_sources = {+ source.strip().lower()+ for source in str(config.get("EXCLUDE_SOURCES") or "").split(",")+ if source.strip()+ }+ corpus_enabled = bool(resolved_corpus_dirs) and "corpus" not in excluded_sources+ corpus_requested = bool(requested_sources and "corpus" in requested_sources)+ if corpus_enabled and requested_sources and "corpus" not in requested_sources:+ requested_sources = [*requested_sources, "corpus"]# Gate StockTwits to ticker/crypto topics. Single chokepoint: when False,# available_sources() never registers stocktwits, so the planner can't@@ -903,6 +924,10 @@ def run(runtime = providers.mock_runtime(config, depth)reasoning_provider = Noneavailable = list(requested_sources or MOCK_AVAILABLE_SOURCES)+ if corpus_enabled and "corpus" not in available:+ available.append("corpus")+ if not corpus_enabled and not corpus_requested:+ available = [source for source in available if source != "corpus"]if not requested_sources and not hiring_signals_mode and not _company_topic_likely(topic):available = [source for source in available if source != "jobs"]else:@@ -910,6 +935,11 @@ def run(available = available_sources(config, requested_sources)if requested_sources:available = [source for source in available if source in requested_sources]+ # Keep an explicitly requested but unconfigured corpus in the plan long+ # enough to record its skipped-unconfigured source outcome. It is never+ # submitted to the network executor below.+ if corpus_requested and "corpus" not in excluded_sources and "corpus" not in available:+ available.append("corpus")if web_backend == "none":available = [s for s in available if s != "grounding"]elif web_backend in ("brave", "exa", "serper", "parallel", "keyless") and "grounding" not in available:@@ -970,6 +1000,15 @@ def run(# Drill plans re-fetch only the sources that contributed to the matched# cluster; the company-topic jobs injection must not widen that set._ensure_jobs_in_plan(plan, available, explicit=hiring_signals_mode, topic=topic)+ if "corpus" in available and plan.subqueries:+ # Corpus is deterministic and user-registered, so it always gets one+ # bounded stream even when a quick/LLM plan omits it. Reuse the primary+ # subquery instead of multiplying local scans across every subquery.+ if "corpus" not in plan.subqueries[0].sources:+ plan.subqueries[0].sources.append("corpus")+ if "corpus" not in plan.source_weights:+ plan.source_weights["corpus"] = 1.0+ plan.source_weights = planner._normalize_weights(plan.source_weights)# Always-on planner trace. Emits one summary line plus one per subquery# so retrieval-breadth failures like the 2026-04-19 Hermes Agent Use Cases@@ -1001,10 +1040,18 @@ def run("Source was requested but is not configured for this run.",attempted=False,)+ if corpus_requested and not corpus_enabled:+ bundle.record_failure(+ "corpus",+ schema.SKIPPED_UNCONFIGURED,+ "Corpus was requested but no readable directory was configured.",+ attempted=False,+ )# Expose plan_source to the renderer so render_compact can emit the# DEGRADED RUN banner when a named-entity topic was invoked bare# (source=deterministic AND no pre-research flags). LAW 7 backstop.bundle.artifacts["plan_source"] = plan_source+ bundle.artifacts["corpus_in_export"] = bool(config.get("_CORPUS_IN_EXPORT"))# Hiring-signals is deliberately jobs-only with no multi-source --plan, so# the LAW 7 degraded-run and Step 0.55 pre-research banners do not apply -# they would contradict the documented jobs-scoped flow. Suppress them.@@ -1070,6 +1117,53 @@ def run(rate_limited_sources: set[str] = set()rate_limit_lock = threading.Lock()+ # Local corpus retrieval is intentionally outside the network executor and+ # retry budget. One bounded stream participates in the same signal scoring,+ # fusion, reranking, and per-source result cap as remote sources.+ if corpus_enabled and plan.subqueries:+ primary = plan.subqueries[0]+ bundle.mark_attempted("corpus")+ result = corpus.search(+ topic,+ resolved_corpus_dirs,+ from_date=from_date,+ to_date=to_date,+ all_time=corpus_all_time,+ limit=settings["per_stream_limit"],+ cache_dir=env.CONFIG_DIR,+ )+ prepared_query = relevance.PreparedQuery(primary.ranking_query)+ lookback_window_days = (+ datetime.strptime(to_date, "%Y-%m-%d").date()+ - datetime.strptime(from_date, "%Y-%m-%d").date()+ ).days+ corpus_items = signals.annotate_stream(+ result.items,+ prepared_query,+ plan.freshness_mode,+ reference_date=to_date,+ max_days=lookback_window_days,+ )+ corpus_items = signals.prune_low_relevance(corpus_items)+ corpus_items = dedupe.dedupe_items(corpus_items)+ for item in corpus_items:+ item.snippet = snippet.extract_best_snippet(item, prepared_query)+ bundle.add_items(primary.label, "corpus", corpus_items)+ if result.notes:+ outcome = bundle.source_status["corpus"]+ bundle.source_status["corpus"] = schema.SourceOutcome(+ source="corpus",+ state=outcome.state,+ items_returned=outcome.items_returned,+ attempted=True,+ detail="; ".join(result.notes),+ )+ bundle.artifacts["corpus"] = {+ "files_scanned": result.files_scanned,+ "cache_hits": result.cache_hits,+ "all_time": corpus_all_time,+ }+futures = {}# Per-source fetch budget prevents redundant API callssource_fetch_count: dict[str, int] = {}@@ -1077,7 +1171,7 @@ def run(1for subquery in plan.subqueriesfor source in subquery.sources- if source in available+ if source in available and source != "corpus")max_workers = _inner_max_workers(stream_count, internal_subrun=internal_subrun)with ThreadPoolExecutor(max_workers=max_workers) as executor:@@ -1085,6 +1179,8 @@ def run(for source in subquery.sources:if source not in available:continue+ if source == "corpus":+ continue# Skip GitHub keyword search if person-mode already ranif source == "github" and (_github_person_done or _github_custom_done):continue@@ -1206,7 +1302,9 @@ def run(# Phase 2b: retry thin sources with simplified query# Note: _github_skip_sources tells the retry to not re-run GitHub keyword search# when project-mode or person-mode already provided authoritative data.- _github_skip_retry = {"github"} if (_github_person_done or _github_custom_done) else set()+ _github_skip_retry = {"corpus"}+ if _github_person_done or _github_custom_done:+ _github_skip_retry.add("github")_retry_thin_sources(topic=topic,bundle=bundle,@@ -1258,21 +1356,58 @@ def run(for h in ([x_handle, github_user, *(x_related or [])])if h and h.strip()}- ranked_candidates = rerank.rerank_candidates(+ private_candidates = [+ candidate+ for candidate in candidates+ if candidate.source == "corpus"+ or any(item.source == "corpus" for item in candidate.source_items)+ ]+ private_candidate_ids = {id(candidate) for candidate in private_candidates}+ public_candidates = [+ candidate for candidate in candidates if id(candidate) not in private_candidate_ids+ ]+ ranked_public = rerank.rerank_candidates(topic=topic,plan=plan,- candidates=candidates,+ candidates=public_candidates,provider=None if mock else reasoning_provider,model=None if mock else runtime.rerank_model,shortlist_size=settings["rerank_limit"],resolved_handles=resolved_handles,)+ # Corpus titles/snippets must never enter a hosted reasoning prompt. Score+ # every candidate carrying corpus evidence with the deterministic fallback,+ # even when the rest of the run uses a remote reranker.+ ranked_private = rerank.rerank_candidates(+ topic=topic,+ plan=plan,+ candidates=private_candidates,+ provider=None,+ model=None,+ shortlist_size=settings["rerank_limit"],+ resolved_handles=resolved_handles,+ )+ ranked_candidates = sorted(+ [*ranked_public, *ranked_private],+ key=lambda candidate: (+ -candidate.final_score,+ -(candidate.engagement or -1),+ min(candidate.native_ranks.values(), default=999),+ candidate.title,+ ),+ )rerank.score_fun(topic=topic,- candidates=ranked_candidates,+ candidates=ranked_public,provider=None if mock else reasoning_provider,model=None if mock else runtime.rerank_model,)+ rerank.score_fun(+ topic=topic,+ candidates=ranked_private,+ provider=None,+ model=None,+ )# Phase 3: post-rerank GitHub star enrichment. Record/replay-aware so the# eval harness stays fully offline: this path calls the GitHub API (and thediff --git a/skills/last30days/scripts/lib/planner.py b/skills/last30days/scripts/lib/planner.pyindex 3ef9764b..7c258fd0 100644--- a/skills/last30days/scripts/lib/planner.py+++ b/skills/last30days/scripts/lib/planner.py@@ -131,6 +131,7 @@ SOURCE_CAPABILITIES = {"grounding": {"web", "reference", "link"},"perplexity": {"web", "reference", "analysis"},"jobs": {"jobs", "company_signal", "link"},+ "corpus": {"reference", "analysis"},}DEFAULT_INTENT_CAPABILITIES = {"comparison": {"discussion", "video", "web", "reference", "social", "link", "market"},diff --git a/skills/last30days/scripts/lib/render.py b/skills/last30days/scripts/lib/render.pyindex 01d93fd7..0aab13f5 100644--- a/skills/last30days/scripts/lib/render.py+++ b/skills/last30days/scripts/lib/render.py@@ -131,8 +131,12 @@ SOURCE_LABELS = {"trustpilot": "Trustpilot","perplexity": "Perplexity","jobs": "Jobs",+ "corpus": "Your files",}+PRIVATE_CORPUS_START = "<!-- LAST30DAYS_PRIVATE_CORPUS_START -->"+PRIVATE_CORPUS_END = "<!-- LAST30DAYS_PRIVATE_CORPUS_END -->"+# vote_weight = max points a fully on-topic, max-upvoted top comment can add to# the LLM humor score. Tuned against real runs: typical funny comments score@@ -277,6 +281,48 @@ def _render_ranked_clusters(return lines+def _render_corpus_section(report: schema.Report, limit: int = 8) -> list[str]:+ """Render private local evidence in one removable, clearly badged block."""+ candidates = [+ candidate+ for candidate in report.ranked_candidates+ if candidate.source == "corpus"+ ][:limit]+ if not candidates:+ return []+ lines = [+ PRIVATE_CORPUS_START,+ "## From your files",+ "",+ "> 🔒 **LOCAL ONLY** - excluded from hosted publishing and agent JSON unless explicitly opted in.",+ "",+ ]+ for candidate in candidates:+ primary = schema.candidate_primary_item(candidate)+ path = str((primary.metadata if primary else {}).get("relative_path") or "")+ published = primary.published_at if primary else None+ detail = f"modified {published}" if published else "modification date unknown"+ lines.append(+ f"- **{_defang_corpus_sentinels(candidate.title)}** "+ f"({detail}, relevance {candidate.final_score:.0f})"+ )+ if path:+ lines.append(f" - File: `{_defang_corpus_sentinels(path)}`")+ if candidate.snippet:+ lines.append(f" - {_defang_corpus_sentinels(_truncate(candidate.snippet, 300))}")+ lines.append(PRIVATE_CORPUS_END)+ return lines+++def _defang_corpus_sentinels(value: str) -> str:+ """Source content must not be able to terminate the private-block markers.++ A note containing the literal end marker would otherwise close the block+ early, leaving later corpus snippets in publishable output.+ """+ return value.replace("LAST30DAYS_PRIVATE_CORPUS", "LAST30DAYS_PRIVATE-CORPUS")++_FRESHNESS_PRIORITY = {"contradicted": 0,"stale": 1,@@ -413,6 +459,7 @@ def render_compact(register: str = "default",) -> str:audience = registers.get_register(register)+ evidence_report = schema.without_sources(report, {"corpus"})non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]lines = [*_render_badge(),@@ -461,7 +508,7 @@ def render_compact(# block below) vs "synthesize from" (this block).lines.append("<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. Transform into `What I learned:` prose per LAW 2. -->")lines.append("")- hiring_block = _render_hiring_signals(report)+ hiring_block = _render_hiring_signals(evidence_report)if hiring_block and audience.name in {"default", "eli5"}:lines.extend(hiring_block)lines.append("")@@ -469,11 +516,11 @@ def render_compact(if audience.name in {"default", "eli5"}:# Keep this legacy assembly byte-for-byte stable. ELI5 has always been# a synthesis-only voice change, so it intentionally takes this path.- lines.extend(_render_ranked_clusters(report, report.clusters[:cluster_limit]))- lines.extend(_render_stats(report))+ lines.extend(_render_ranked_clusters(evidence_report, evidence_report.clusters[:cluster_limit]))+ lines.extend(_render_stats(evidence_report))best_takes = _render_best_takes(- report.ranked_candidates,+ evidence_report.ranked_candidates,limit=fun_params["limit"],threshold=fun_params["threshold"],vote_weight=fun_params.get("vote_weight", 18.0),@@ -481,7 +528,7 @@ def render_compact(if best_takes:lines.extend([""] + best_takes)- top_comments = _render_top_comments(report)+ top_comments = _render_top_comments(evidence_report)if top_comments:lines.extend([""] + top_comments)@@ -491,7 +538,10 @@ def render_compact(lines.extend(_render_source_coverage(report))else:- lines.extend(_render_registered_sections(report, audience, fun_params, cluster_limit))+ lines.extend(_render_registered_sections(evidence_report, audience, fun_params, cluster_limit))+ corpus_section = _render_corpus_section(report)+ if corpus_section:+ lines.extend(["", *corpus_section])# Close EVIDENCE FOR SYNTHESIS envelope before anything that passes through verbatim.lines.append("")lines.append("<!-- END EVIDENCE FOR SYNTHESIS -->")@@ -545,6 +595,7 @@ def render_for_html(sections so direct HTML output reflects the selected audience preset."""audience = registers.get_register(register)+ evidence_report = schema.without_sources(report, {"corpus"})lines = [*_render_badge(),*_render_html_metadata(report),@@ -552,7 +603,7 @@ def render_for_html(drill_context = _render_drill_context(report)if drill_context:lines.extend(["", *drill_context])- hiring_block = _render_hiring_signals(report)+ hiring_block = _render_hiring_signals(evidence_report)if synthesis_md:lines.extend(["", synthesis_md.strip()])if hiring_block and "## Hiring Signals" not in synthesis_md:@@ -564,13 +615,16 @@ def render_for_html(lines.extend(["",*_render_registered_sections(- report,+ evidence_report,audience,fun_params,8,include_source_diagnostics=False,),])+ corpus_section = _render_corpus_section(report)+ if corpus_section:+ lines.extend(["", *corpus_section])freshness_verdicts = _render_freshness_verdicts(report)if freshness_verdicts:lines.extend(["", *freshness_verdicts])@@ -613,6 +667,9 @@ def render_for_html_comparison(freshness_verdicts = _render_freshness_verdicts(report)if freshness_verdicts:lines.extend(["", f"## {label}", "", *freshness_verdicts])+ corpus_section = _render_corpus_section(report)+ if corpus_section:+ lines.extend(["", f"## {label}", "", *corpus_section])# Comparison data quality notes also go to stderr, not into the artifact._append_html_footer(lines, main_report, save_path)return "\n".join(lines).strip() + "\n"@@ -1100,17 +1157,22 @@ def _render_entity_evidence_block(fun_params: dict,) -> list[str]:"""Render one entity's clusters and best-takes inside the evidence envelope."""- candidate_by_id = {c.candidate_id: c for c in report.ranked_candidates}+ evidence_report = schema.without_sources(report, {"corpus"})+ candidate_by_id = {c.candidate_id: c for c in evidence_report.ranked_candidates}out: list[str] = [f"## {label}", ""]- if not report.clusters:+ if not evidence_report.clusters:out.append("(no significant discussion this month)")out.append("")+ corpus_section = _render_corpus_section(report)+ if corpus_section:+ out.extend(corpus_section)+ out.append("")return outout.append("### Ranked Evidence Clusters")out.append("")- for index, cluster in enumerate(report.clusters[:cluster_limit], start=1):+ for index, cluster in enumerate(evidence_report.clusters[:cluster_limit], start=1):out.append(f"#### {index}. {cluster.title} "f"(score {cluster.score:.0f}, {len(cluster.candidate_ids)} item"@@ -1123,11 +1185,11 @@ def _render_entity_evidence_block(candidate = candidate_by_id.get(candidate_id)if not candidate:continue- out.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=report))+ out.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=evidence_report))out.append("")best_takes = _render_best_takes(- report.ranked_candidates,+ evidence_report.ranked_candidates,limit=fun_params["limit"],threshold=fun_params["threshold"],vote_weight=fun_params.get("vote_weight", 18.0),@@ -1136,6 +1198,11 @@ def _render_entity_evidence_block(out.extend(best_takes)out.append("")+ corpus_section = _render_corpus_section(report)+ if corpus_section:+ out.extend(corpus_section)+ out.append("")+return out@@ -1159,22 +1226,27 @@ def render_comparison_multi_context(lines.extend(resolved_block)lines.append("")for label, report in entity_reports:+ evidence_report = schema.without_sources(report, {"corpus"})lines.append(f"## {label}")lines.append(f"Intent: {report.query_plan.intent}")- if not report.clusters:+ if not evidence_report.clusters:lines.append("- (no significant discussion this month)")else:- for cluster in report.clusters[:cluster_limit]:+ for cluster in evidence_report.clusters[:cluster_limit]:lines.append(f"- {cluster.title} "f"[{', '.join(_source_label(s) for s in cluster.sources)}]")+ corpus_section = _render_corpus_section(report)+ if corpus_section:+ lines.extend(["", *corpus_section])lines.append("")return "\n".join(lines).strip() + "\n"def render_full(report: schema.Report) -> str:"""Full data dump: ALL clusters + ALL items by source. For saved files and debugging."""+ evidence_report = schema.without_sources(report, {"corpus"})# Start with the same header as compactnon_empty = [s for s, items in sorted(report.items_by_source.items()) if items]lines = [@@ -1209,8 +1281,8 @@ def render_full(report: schema.Report) -> str:# ALL clusters (no limit)lines.append("## Ranked Evidence Clusters")lines.append("")- candidate_by_id = {c.candidate_id: c for c in report.ranked_candidates}- for index, cluster in enumerate(report.clusters, start=1):+ candidate_by_id = {c.candidate_id: c for c in evidence_report.ranked_candidates}+ for index, cluster in enumerate(evidence_report.clusters, start=1):lines.append(f"### {index}. {cluster.title} "f"(score {cluster.score:.0f}, {len(cluster.candidate_ids)} item{'s' if len(cluster.candidate_ids) != 1 else ''}, "@@ -1222,12 +1294,12 @@ def render_full(report: schema.Report) -> str:candidate = candidate_by_id.get(cid)if not candidate:continue- lines.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=report))+ lines.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=evidence_report))lines.append("")fun_params = _FUN_LEVELS["medium"]best_takes = _render_best_takes(- report.ranked_candidates,+ evidence_report.ranked_candidates,limit=fun_params["limit"],threshold=fun_params["threshold"],vote_weight=fun_params["vote_weight"],@@ -1242,7 +1314,7 @@ def render_full(report: schema.Report) -> str:source_order = ["reddit", "x", "youtube", "tiktok", "instagram", "threads", "pinterest","hackernews", "bluesky", "truthsocial", "polymarket", "grounding", "xiaohongshu", "github", "digg", "perplexity", "jobs"]for source in source_order:- items = report.items_by_source.get(source, [])+ items = evidence_report.items_by_source.get(source, [])if not items:continuelines.append(f"### {_source_label(source)} ({len(items)} items)")@@ -1308,12 +1380,17 @@ def render_full(report: schema.Report) -> str:lines.append(f" Closes: {end_date}")lines.append("")- freshness_verdicts = _render_freshness_verdicts(report)+ corpus_section = _render_corpus_section(report)+ if corpus_section:+ lines.extend(corpus_section)+ lines.append("")++ freshness_verdicts = _render_freshness_verdicts(evidence_report)if freshness_verdicts:lines.extend(freshness_verdicts)lines.append("")- lines.extend(_render_stats(report))- lines.extend(_render_source_coverage(report))+ lines.extend(_render_stats(evidence_report))+ lines.extend(_render_source_coverage(evidence_report))return "\n".join(lines).strip() + "\n"@@ -1332,7 +1409,8 @@ def _format_item_engagement(item: schema.SourceItem) -> str:def render_context(report: schema.Report, cluster_limit: int = 6) -> str:- candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}+ evidence_report = schema.without_sources(report, {"corpus"})+ candidate_by_id = {candidate.candidate_id: candidate for candidate in evidence_report.ranked_candidates}lines = [f"Topic: {report.topic}",f"Intent: {report.query_plan.intent}",@@ -1351,7 +1429,7 @@ def render_context(report: schema.Report, cluster_limit: int = 6) -> str:if hiring_block:lines.extend(["", *hiring_block, ""])lines.append("Top clusters:")- for cluster in report.clusters[:cluster_limit]:+ for cluster in evidence_report.clusters[:cluster_limit]:lines.append(f"- {cluster.title} [{', '.join(_source_label(source) for source in cluster.sources)}]")for candidate_id in cluster.representative_ids[:2]:candidate = candidate_by_id.get(candidate_id)@@ -1366,6 +1444,9 @@ def render_context(report: schema.Report, cluster_limit: int = 6) -> str:lines.append(f" - {' | '.join(detail_parts)}")if candidate.snippet:lines.append(f" Evidence: {_truncate(candidate.snippet, 180)}")+ corpus_section = _render_corpus_section(report)+ if corpus_section:+ lines.extend(["", *corpus_section])if report.warnings:lines.append("Warnings:")lines.extend(f"- {warning}" for warning in report.warnings)@@ -1386,6 +1467,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:Audience Questions, and Source Clusters. Sections 2-4 are omitted when thereis no matching data; Sections 1 and 5 always appear."""+ evidence_report = schema.without_sources(report, {"corpus"})non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]lines = [f"# Production Brief: {report.topic}",@@ -1404,8 +1486,8 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:lines.append("## Ranked Storylines")lines.append("")- candidate_by_id = {c.candidate_id: c for c in report.ranked_candidates}- for i, cluster in enumerate(report.clusters[:cluster_limit], start=1):+ candidate_by_id = {c.candidate_id: c for c in evidence_report.ranked_candidates}+ for i, cluster in enumerate(evidence_report.clusters[:cluster_limit], start=1):source_tags = ", ".join(_source_label(s) for s in cluster.sources)qualifier = f" [{cluster.uncertainty.replace('-', ' ')}]" if cluster.uncertainty else ""lines.append(f"### {i}. {cluster.title} (score {cluster.score:.0f}, {source_tags}){qualifier}")@@ -1421,7 +1503,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:lines.append("")hooks = sorted(- (c for c in report.ranked_candidates if c.fun_score is not None and c.fun_score >= 70),+ (c for c in evidence_report.ranked_candidates if c.fun_score is not None and c.fun_score >= 70),key=lambda c: -(c.fun_score or 0),)if hooks:@@ -1449,7 +1531,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:)lines.append("")- tensions = [c for c in report.clusters[:cluster_limit] if c.uncertainty]+ tensions = [c for c in evidence_report.clusters[:cluster_limit] if c.uncertainty]if tensions:lines.append("## Topic Tensions")lines.append("")@@ -1459,7 +1541,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:lines.append(f"- **{cluster.title}** [{label}]: {source_tags}")lines.append("")- questions = _extract_audience_questions(report.ranked_candidates)+ questions = _extract_audience_questions(evidence_report.ranked_candidates)if questions:lines.append("## Audience Questions")lines.append("")@@ -1469,11 +1551,16 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:lines.append("## Source Clusters")lines.append("")- for cluster in report.clusters[:cluster_limit]:+ for cluster in evidence_report.clusters[:cluster_limit]:source_tags = " + ".join(_source_label(s) for s in cluster.sources)lines.append(f"- **{cluster.title}**: {source_tags}")lines.append("")+ corpus_section = _render_corpus_section(report)+ if corpus_section:+ lines.extend(corpus_section)+ lines.append("")+freshness_verdicts = _render_freshness_verdicts(report)if freshness_verdicts:lines.extend(freshness_verdicts)@@ -1925,6 +2012,7 @@ _FOOTER_SOURCES: list[tuple[str, str, str, str, list[tuple[str, str]]]] = [# the LAW 5 footer; without it the footer was dropped entirely.("jobs", "💼", "Jobs", "role", []),("perplexity", "🧠", "Perplexity", "result", [("citations", "citations")]),+ ("corpus", "🔒", "Your files", "file", []),]diff --git a/skills/last30days/scripts/lib/schema.py b/skills/last30days/scripts/lib/schema.pyindex d3425cd2..759a7e8d 100644--- a/skills/last30days/scripts/lib/schema.py+++ b/skills/last30days/scripts/lib/schema.py@@ -2,6 +2,7 @@from __future__ import annotations+import copyfrom dataclasses import asdict, dataclass, field, is_dataclassfrom datetime import datetime, timezonefrom typing import Any, Literal@@ -558,6 +559,103 @@ def candidate_primary_item(candidate: Candidate) -> SourceItem | None:AGENT_EXPORT_SCHEMA_VERSION = "1.2"+++def without_sources(report: Report, excluded_sources: set[str]) -> Report:+ """Return a deep-copied report with private source evidence removed.++ This is the publication boundary used by agent JSON, hosted HTML, and+ future outbound surfaces. Cluster titles are rebuilt when a removed item+ participated so text derived from a private representative cannot survive+ after its candidate is gone.+ """+ excluded = {source.lower() for source in excluded_sources}+ if not excluded:+ return copy.deepcopy(report)+ clean = copy.deepcopy(report)+ clean.items_by_source = {+ source: items+ for source, items in clean.items_by_source.items()+ if source.lower() not in excluded+ }+ clean.errors_by_source = {+ source: detail+ for source, detail in clean.errors_by_source.items()+ if source.lower() not in excluded+ }+ clean.source_status = {+ source: outcome+ for source, outcome in clean.source_status.items()+ if source.lower() not in excluded+ }+ clean.query_plan.source_weights = {+ source: weight+ for source, weight in clean.query_plan.source_weights.items()+ if source.lower() not in excluded+ }+ for subquery in clean.query_plan.subqueries:+ subquery.sources[:] = [+ source for source in subquery.sources if source.lower() not in excluded+ ]++ kept_candidates: list[Candidate] = []+ removed_candidate_ids: set[str] = set()+ for candidate in clean.ranked_candidates:+ if candidate.source.lower() in excluded:+ removed_candidate_ids.add(candidate.candidate_id)+ continue+ candidate.source_items = [+ item for item in candidate.source_items if item.source.lower() not in excluded+ ]+ candidate.sources = [+ source for source in candidate.sources if source.lower() not in excluded+ ]+ candidate.native_ranks = {+ key: rank+ for key, rank in candidate.native_ranks.items()+ if key.rsplit(":", 1)[-1].lower() not in excluded+ }+ kept_candidates.append(candidate)+ clean.ranked_candidates = kept_candidates+ candidate_by_id = {+ candidate.candidate_id: candidate for candidate in clean.ranked_candidates+ }++ kept_clusters: list[Cluster] = []+ for cluster in clean.clusters:+ original_ids = list(cluster.candidate_ids)+ cluster.candidate_ids = [+ candidate_id for candidate_id in original_ids if candidate_id in candidate_by_id+ ]+ if not cluster.candidate_ids:+ continue+ cluster.representative_ids = [+ candidate_id+ for candidate_id in cluster.representative_ids+ if candidate_id in candidate_by_id+ ] or [cluster.candidate_ids[0]]+ cluster.sources = sorted({+ source+ for candidate_id in cluster.candidate_ids+ for source in candidate_sources(candidate_by_id[candidate_id])+ if source.lower() not in excluded+ })+ if any(candidate_id in removed_candidate_ids for candidate_id in original_ids):+ cluster.title = candidate_by_id[cluster.representative_ids[0]].title+ kept_clusters.append(cluster)+ clean.clusters = kept_clusters+ clean.freshness_verdicts = [+ verdict+ for verdict in clean.freshness_verdicts+ if verdict.source.lower() not in excluded+ and verdict.candidate_id in candidate_by_id+ ]+ for key in list(clean.artifacts):+ if any(source in key.lower() for source in excluded):+ del clean.artifacts[key]+ return clean++DISCOVERY_EXPORT_SCHEMA_VERSION = "1.0"@@ -628,8 +726,20 @@ def _agent_generated_at(value: str) -> str:return parsed.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")-def to_agent_export(report: Report) -> dict[str, Any]:- """Serialize a report to the stable, versioned agent JSON contract."""+def to_agent_export(+ report: Report,+ *,+ corpus_in_export: bool | None = None,+) -> dict[str, Any]:+ """Serialize a report to the stable, versioned agent JSON contract.++ Local corpus evidence is private by default. Callers must opt in explicitly+ either with ``corpus_in_export=True`` or the CLI-populated report artifact.+ """+ if corpus_in_export is None:+ corpus_in_export = bool(report.artifacts.get("corpus_in_export"))+ if not corpus_in_export:+ report = without_sources(report, {"corpus"})candidates = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}cluster_by_candidate: dict[str, int] = {}cluster_by_id: dict[str, int] = {}diff --git a/skills/last30days/scripts/lib/signals.py b/skills/last30days/scripts/lib/signals.pyindex d49d689a..08bfe1d0 100644--- a/skills/last30days/scripts/lib/signals.py+++ b/skills/last30days/scripts/lib/signals.py@@ -24,6 +24,7 @@ SOURCE_QUALITY = {"instagram": 0.58,"tiktok": 0.58,"jobs": 0.72,+ "corpus": 0.75,}diff --git a/skills/last30days/scripts/store.py b/skills/last30days/scripts/store.pyindex a02c928e..233289d3 100644--- a/skills/last30days/scripts/store.py+++ b/skills/last30days/scripts/store.py@@ -12,6 +12,7 @@ Database location: ~/.local/share/last30days/research.dbimport argparseimport json+import osimport sqlite3import sysfrom contextlib import contextmanager@@ -55,6 +56,25 @@ def scoped_db(db_path: Optional[Path]) -> Iterator[None]:_db_override = previous+def ensure_private_db_files(db_path: Optional[Path] = None) -> Path:+ """Create/harden the research database and SQLite sidecars owner-only."""+ path = db_path or _get_db_path()+ path.parent.mkdir(parents=True, exist_ok=True)+ if not path.exists():+ try:+ fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)+ except FileExistsError:+ pass+ else:+ os.close(fd)+ for candidate in (path, Path(f"{path}-wal"), Path(f"{path}-shm")):+ try:+ candidate.chmod(0o600)+ except FileNotFoundError:+ pass+ return path++SCHEMA_V1 = """PRAGMA journal_mode=WAL;PRAGMA synchronous=NORMAL;diff --git a/tests/test_corpus_source.py b/tests/test_corpus_source.pynew file mode 100644index 00000000..c13fb18f--- /dev/null+++ b/tests/test_corpus_source.py@@ -0,0 +1,868 @@+from __future__ import annotations++import hashlib+import io+import json+import os+import sqlite3+import sys+from contextlib import redirect_stderr, redirect_stdout+from datetime import datetime, timezone+from pathlib import Path+from unittest import mock++import last30days as cli+import store+from lib import corpus, env, health, html_render, library, library_index, pipeline, render, schema+++def _set_mtime(path: Path, value: str) -> None:+ timestamp = datetime.fromisoformat(value).replace(tzinfo=timezone.utc).timestamp()+ os.utime(path, (timestamp, timestamp))+++def _scan(tmp_path: Path, *, all_time: bool = False, limit: int = 12):+ return corpus.search(+ "MCP servers",+ [tmp_path],+ from_date="2026-06-10",+ to_date="2026-07-10",+ all_time=all_time,+ limit=limit,+ cache_dir=tmp_path / "cache",+ )+++def test_scans_matching_text_and_markdown_with_path_titles(tmp_path):+ note = tmp_path / "mcp-server-notes.md"+ note.write_text("MCP servers expose tools to local coding agents.", encoding="utf-8")+ other = tmp_path / "groceries.txt"+ other.write_text("milk eggs bread", encoding="utf-8")+ _set_mtime(note, "2026-07-05T12:00:00")+ _set_mtime(other, "2026-07-05T12:00:00")++ result = _scan(tmp_path)++ assert [item.title for item in result.items] == ["mcp server notes"]+ assert result.items[0].source == "corpus"+ assert result.items[0].published_at == "2026-07-05"+ assert result.items[0].metadata["local_only"] is True+ assert result.items[0].url.startswith("corpus://")+ assert str(note) not in result.items[0].url+++def test_multilingual_matching_reuses_shared_cjk_tokenizer(tmp_path):+ note = tmp_path / "模型记录.md"+ note.write_text("国产大模型的最新测评和部署记录", encoding="utf-8")+ _set_mtime(note, "2026-07-05T12:00:00")++ result = corpus.search(+ "国产大模型 测评",+ [tmp_path],+ from_date="2026-06-10",+ to_date="2026-07-10",+ cache_dir=tmp_path / "cache",+ )++ assert [item.title for item in result.items] == ["模型记录"]+++def test_recency_window_and_all_time_override(tmp_path):+ old = tmp_path / "old-mcp-plan.md"+ old.write_text("MCP servers and local tool protocols", encoding="utf-8")+ _set_mtime(old, "2025-01-01T00:00:00")++ assert _scan(tmp_path).items == []+ assert [item.title for item in _scan(tmp_path, all_time=True).items] == ["old mcp plan"]+++def test_hidden_git_and_node_modules_directories_are_ignored(tmp_path):+ visible = tmp_path / "visible.md"+ visible.write_text("MCP servers are visible", encoding="utf-8")+ _set_mtime(visible, "2026-07-05T00:00:00")+ for directory in (tmp_path / ".git", tmp_path / ".hidden", tmp_path / "node_modules"):+ directory.mkdir()+ path = directory / "private.md"+ path.write_text("MCP servers hidden secret", encoding="utf-8")+ _set_mtime(path, "2026-07-05T00:00:00")++ result = _scan(tmp_path)++ assert [item.metadata["relative_path"] for item in result.items] == ["visible.md"]+++def test_pdf_is_skipped_with_one_note_when_pdftotext_is_absent(tmp_path, monkeypatch):+ pdf = tmp_path / "mcp.pdf"+ pdf.write_bytes(b"not a real pdf")+ _set_mtime(pdf, "2026-07-05T00:00:00")+ monkeypatch.setattr(corpus, "which", lambda _name: None)++ result = _scan(tmp_path)++ assert result.items == []+ assert result.notes == ["Skipped PDF files because pdftotext is not on PATH"]+++def test_mtime_cache_reuses_text_and_is_private(tmp_path, monkeypatch):+ note = tmp_path / "mcp.md"+ note.write_text("MCP servers cache this local note", encoding="utf-8")+ _set_mtime(note, "2026-07-05T00:00:00")+ first = _scan(tmp_path)+ assert first.cache_hits == 0++ monkeypatch.setattr(corpus, "_extract_text", mock.Mock(side_effect=AssertionError("cache miss")))+ second = _scan(tmp_path)++ assert second.cache_hits == 1+ cache_path = tmp_path / "cache" / corpus.CACHE_FILENAME+ assert cache_path.stat().st_mode & 0o777 == 0o600+ assert cache_path.parent.stat().st_mode & 0o777 == 0o700+++def test_cache_for_500_large_documents_is_bounded_by_total_bytes(tmp_path, monkeypatch):+ for index in range(500):+ (tmp_path / f"document-{index:03d}.txt").touch()++ monkeypatch.setattr(corpus, "MAX_CACHE_BYTES", 1_000_000)+ monkeypatch.setattr(corpus, "MAX_CACHE_TEXT_CHARS", 100_000)+ monkeypatch.setattr(corpus, "_extract_text", lambda *_args, **_kwargs: "x" * 1_000_000)+ monkeypatch.setattr(corpus, "_match_score", lambda *_args, **_kwargs: 0.0)++ result = corpus.search(+ "anything",+ [tmp_path],+ from_date="2026-06-10",+ to_date="2026-07-10",+ all_time=True,+ limit=0,+ cache_dir=tmp_path / "cache",+ )++ cache_path = tmp_path / "cache" / corpus.CACHE_FILENAME+ payload = json.loads(cache_path.read_text(encoding="utf-8"))+ assert result.files_scanned == 500+ assert cache_path.stat().st_size <= corpus.MAX_CACHE_BYTES+ assert len(payload["entries"]) < 500+ assert all(+ len(entry["text"]) <= corpus.MAX_CACHE_TEXT_CHARS+ for entry in payload["entries"].values()+ )+++def test_result_budget_caps_one_corpus_stream(tmp_path):+ for index in range(10):+ path = tmp_path / f"mcp-{index}.md"+ path.write_text(f"MCP servers local note {index}", encoding="utf-8")+ _set_mtime(path, f"2026-07-{index + 1:02d}T00:00:00")++ assert len(_scan(tmp_path, limit=3).items) == 3+++def test_file_cap_is_shared_fairly_across_corpus_roots(tmp_path, monkeypatch):+ archive = tmp_path / "archive"+ relevant = tmp_path / "relevant"+ archive.mkdir()+ relevant.mkdir()+ for index in range(4):+ (archive / f"mcp-archive-{index}.md").write_text(+ f"MCP servers archived note {index}", encoding="utf-8"+ )+ later = relevant / "mcp-current.md"+ later.write_text("MCP servers relevant current note", encoding="utf-8")+ monkeypatch.setattr(corpus, "MAX_FILES", 4)++ result = corpus.search(+ "MCP servers",+ [archive, relevant],+ from_date="2026-06-10",+ to_date="2026-07-10",+ all_time=True,+ cache_dir=tmp_path / "cache",+ )++ assert result.files_scanned <= corpus.MAX_FILES+ assert any(item.metadata["path"].startswith(str(archive)) for item in result.items)+ assert any(item.metadata["path"] == str(later) for item in result.items)+++def test_pipeline_runs_corpus_outside_network_executor(tmp_path, monkeypatch):+ note = tmp_path / "mcp-private.md"+ note.write_text("MCP servers use a secret local transport", encoding="utf-8")+ _set_mtime(note, "2026-07-05T00:00:00")+ retrieve = mock.Mock(return_value=([], {}))+ monkeypatch.setattr(pipeline, "_retrieve_stream", retrieve)++ report = pipeline.run(+ topic="MCP servers",+ config={"_CORPUS_DIRS": [str(tmp_path)], "EXCLUDE_SOURCES": ""},+ depth="quick",+ requested_sources=["corpus"],+ mock=True,+ as_of_date="2026-07-10",+ external_plan={+ "intent": "concept",+ "freshness_mode": "balanced_recent",+ "cluster_mode": "none",+ "source_weights": {"corpus": 1.0},+ "subqueries": [{+ "label": "primary",+ "search_query": "MCP servers",+ "ranking_query": "MCP servers",+ "sources": ["corpus"],+ }],+ },+ )++ assert report.source_status["corpus"].state == health.OK+ assert len(report.items_by_source["corpus"]) == 1+ assert len(report.items_by_source["corpus"]) <= pipeline.DEPTH_SETTINGS["quick"]["per_stream_limit"]+ assert all(call.kwargs["source"] != "corpus" for call in retrieve.call_args_list)+++def test_corpus_never_enters_remote_rerank_or_fun_prompts(tmp_path, monkeypatch):+ note = tmp_path / "private.md"+ note.write_text("Model context protocol servers PRIVATE-RERANK-SENTINEL", encoding="utf-8")+ _set_mtime(note, "2026-07-05T00:00:00")+ remote = mock.Mock()+ remote.generate_json = mock.Mock(side_effect=AssertionError("private prompt left machine"))+ runtime = schema.ProviderRuntime("local", "remote-planner", "remote-reranker")+ monkeypatch.setattr(pipeline.providers, "resolve_runtime", lambda *_args, **_kwargs: (runtime, remote))+ monkeypatch.setattr(pipeline, "available_sources", lambda *_args, **_kwargs: ["corpus"])++ report = pipeline.run(+ topic="how do model context protocol servers work today?",+ config={"_CORPUS_DIRS": [str(tmp_path)], "EXCLUDE_SOURCES": ""},+ depth="quick",+ requested_sources=["corpus"],+ mock=False,+ as_of_date="2026-07-10",+ external_plan={+ "intent": "how_to",+ "freshness_mode": "evergreen_ok",+ "cluster_mode": "workflow",+ "source_weights": {"corpus": 1.0},+ "subqueries": [{+ "label": "primary",+ "search_query": "model context protocol servers",+ "ranking_query": "How do model context protocol servers work?",+ "sources": ["corpus"],+ }],+ },+ )++ assert report.items_by_source["corpus"]+ remote.generate_json.assert_not_called()+++def test_explicit_unconfigured_corpus_records_skipped_outcome():+ report = pipeline.run(+ topic="how do local model context protocol servers work?",+ config={"EXCLUDE_SOURCES": ""},+ depth="quick",+ requested_sources=["corpus"],+ mock=True,+ as_of_date="2026-07-10",+ external_plan={+ "intent": "how_to",+ "freshness_mode": "evergreen_ok",+ "cluster_mode": "workflow",+ "source_weights": {"corpus": 1.0},+ "subqueries": [{+ "label": "primary",+ "search_query": "model context protocol servers",+ "ranking_query": "How do model context protocol servers work?",+ "sources": ["corpus"],+ }],+ },+ )++ outcome = report.source_status["corpus"]+ assert outcome.state == schema.SKIPPED_UNCONFIGURED+ assert outcome.attempted is False+++def _privacy_report(secret: str = "PRIVATE-CORPUS-SENTINEL") -> schema.Report:+ item = schema.SourceItem(+ item_id="C-private",+ source="corpus",+ title="private mcp notes",+ body=f"MCP servers {secret}",+ url="",+ published_at="2026-07-05",+ snippet=f"MCP servers {secret}",+ metadata={"relative_path": "notes/private.md", "local_only": True},+ )+ candidate = schema.Candidate(+ candidate_id="corpus:C-private",+ item_id=item.item_id,+ source="corpus",+ title=item.title,+ url="",+ snippet=item.snippet,+ subquery_labels=["primary"],+ native_ranks={"primary:corpus": 1},+ local_relevance=1.0,+ freshness=90,+ engagement=None,+ source_quality=0.75,+ rrf_score=0.01,+ final_score=91,+ cluster_id="cluster-private",+ sources=["corpus"],+ source_items=[item],+ )+ return schema.Report(+ topic="MCP servers",+ range_from="2026-06-10",+ range_to="2026-07-10",+ generated_at="2026-07-10T00:00:00+00:00",+ provider_runtime=schema.ProviderRuntime("local", "mock", "mock"),+ query_plan=schema.QueryPlan(+ intent="concept",+ freshness_mode="balanced_recent",+ cluster_mode="none",+ raw_topic="MCP servers",+ subqueries=[schema.SubQuery("primary", "MCP servers", "MCP servers", ["corpus"])],+ source_weights={"corpus": 1.0},+ ),+ clusters=[schema.Cluster(+ cluster_id="cluster-private",+ title=f"Private cluster {secret}",+ candidate_ids=[candidate.candidate_id],+ representative_ids=[candidate.candidate_id],+ sources=["corpus"],+ score=91,+ )],+ ranked_candidates=[candidate],+ items_by_source={"corpus": [item]},+ errors_by_source={},+ source_status={"corpus": schema.SourceOutcome("corpus", health.OK, 1)},+ )+++def test_corpus_findings_persist_with_stable_opaque_keys(tmp_path, monkeypatch):+ note = tmp_path / "private-customer-notes.md"+ note.write_text("MCP servers persist this private finding", encoding="utf-8")+ first = corpus.search(+ "MCP servers",+ [tmp_path],+ from_date="2026-06-10",+ to_date="2026-07-10",+ all_time=True,+ cache_dir=tmp_path / "cache",+ ).items[0]+ second = corpus.search(+ "MCP servers",+ [tmp_path],+ from_date="2026-06-10",+ to_date="2026-07-10",+ all_time=True,+ cache_dir=tmp_path / "cache",+ ).items[0]+ report = _privacy_report()+ report.items_by_source["corpus"][0].url = first.url+ report.ranked_candidates[0].url = first.url++ db_path = tmp_path / "research.db"+ monkeypatch.setattr(store, "_db_override", db_path)+ store.init_db()+ topic = store.add_topic(report.topic)+ run_id = store.record_run(topic["id"], status="completed")+ findings = store.findings_from_report(report)+ counts = store.store_findings(run_id, topic["id"], findings)++ assert first.url == second.url+ assert first.url.startswith("corpus://")+ assert str(note) not in first.url+ assert counts == {"new": 1, "updated": 0}+ with sqlite3.connect(db_path) as conn:+ persisted = conn.execute(+ "SELECT source, source_url FROM findings"+ ).fetchone()+ assert persisted == ("corpus", first.url)+ assert library_index.search(+ "private finding",+ db_path=tmp_path / "missing-library.db",+ store_db_path=db_path,+ ) == []+++def test_persist_report_hardens_corpus_store_and_sidecars(tmp_path, monkeypatch):+ db_path = tmp_path / "store" / "research.db"+ monkeypatch.setattr(store, "_db_override", db_path)+ original_findings_from_report = store.findings_from_report+ original_store_findings = store.store_findings+ original_update_run = store.update_run++ def make_permissive():+ db_path.chmod(0o644)+ for suffix in ("-wal", "-shm"):+ sidecar = Path(f"{db_path}{suffix}")+ sidecar.touch(mode=0o644)+ sidecar.chmod(0o644)++ def findings_from_report_with_permissive_store(*args, **kwargs):+ findings = original_findings_from_report(*args, **kwargs)+ make_permissive()+ return findings++ def store_findings_after_private_check(*args, **kwargs):+ for artifact in (db_path, Path(f"{db_path}-wal"), Path(f"{db_path}-shm")):+ assert artifact.stat().st_mode & 0o777 == 0o600+ return original_store_findings(*args, **kwargs)++ def update_run_with_permissive_sidecars(*args, **kwargs):+ result = original_update_run(*args, **kwargs)+ make_permissive()+ return result++ monkeypatch.setattr(store, "findings_from_report", findings_from_report_with_permissive_store)+ monkeypatch.setattr(store, "store_findings", store_findings_after_private_check)+ monkeypatch.setattr(store, "update_run", update_run_with_permissive_sidecars)++ cli.persist_report(_privacy_report())++ for artifact in (db_path, Path(f"{db_path}-wal"), Path(f"{db_path}-shm")):+ assert artifact.stat().st_mode & 0o777 == 0o600+++def test_agent_export_excludes_corpus_by_default_and_allows_explicit_opt_in():+ report = _privacy_report()++ private_default = schema.to_agent_export(report)+ opted_in = schema.to_agent_export(report, corpus_in_export=True)++ assert private_default["results"] == []+ assert private_default["clusters"] == []+ assert "corpus" not in private_default["source_status"]+ assert opted_in["results"][0]["source"] == "corpus"+ assert "PRIVATE-CORPUS-SENTINEL" in opted_in["results"][0]["summary"]++ report.artifacts["corpus_in_export"] = True+ assert schema.to_agent_export(report)["results"][0]["source"] == "corpus"+++def test_agent_export_rebuilds_mixed_cluster_title_after_private_representative_removed():+ report = _privacy_report()+ social_item = schema.SourceItem(+ item_id="R-public",+ source="reddit",+ title="Public MCP discussion",+ body="Public evidence",+ url="https://reddit.example/public",+ published_at="2026-07-06",+ snippet="Public evidence",+ )+ social = schema.Candidate(+ candidate_id="reddit:public",+ item_id=social_item.item_id,+ source="reddit",+ title=social_item.title,+ url=social_item.url,+ snippet=social_item.snippet,+ subquery_labels=["primary"],+ native_ranks={"primary:reddit": 1},+ local_relevance=0.8,+ freshness=90,+ engagement=1,+ source_quality=0.6,+ rrf_score=0.01,+ final_score=80,+ cluster_id="cluster-private",+ sources=["reddit"],+ source_items=[social_item],+ )+ report.ranked_candidates.append(social)+ report.items_by_source["reddit"] = [social_item]+ report.source_status["reddit"] = schema.SourceOutcome("reddit", health.OK, 1)+ report.clusters[0].candidate_ids.append(social.candidate_id)+ report.clusters[0].representative_ids.append(social.candidate_id)+ report.clusters[0].sources.append("reddit")++ exported = schema.to_agent_export(report)++ assert exported["clusters"][0]["title"] == "Public MCP discussion"+ assert "PRIVATE-CORPUS-SENTINEL" not in str(exported)+++def test_local_report_has_badged_from_your_files_section():+ rendered = render.render_compact(_privacy_report())++ assert "## From your files" in rendered+ assert "LOCAL ONLY" in rendered+ assert "PRIVATE-CORPUS-SENTINEL" in rendered+++def test_publish_html_sends_sanitized_report_not_local_corpus(monkeypatch):+ report = _privacy_report()+ captured: dict[str, str] = {}++ def publish(rendered, **_kwargs):+ captured["html"] = rendered+ return {"url": "https://example.ht-ml.app"}++ monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})+ monkeypatch.setattr(cli.pipeline, "diagnose", lambda *_args, **_kwargs: {"available_sources": ["corpus"]})+ monkeypatch.setattr(cli.pipeline, "run", lambda **_kwargs: report)+ monkeypatch.setattr(cli, "publish_rendered_html", publish)+ monkeypatch.setenv("LAST30DAYS_SKIP_PREFLIGHT", "1")+ monkeypatch.setattr(+ sys,+ "argv",+ ["last30days.py", "MCP servers", "--emit=html", "--publish-html"],+ )++ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):+ assert cli.main() == 0++ assert "PRIVATE-CORPUS-SENTINEL" not in captured["html"]+ assert "private mcp notes" not in captured["html"]+++def test_configured_corpus_bypasses_hosted_backend(tmp_path, monkeypatch):+ report = _privacy_report()+ hosted = mock.Mock(side_effect=AssertionError("local corpus was forwarded"))+ monkeypatch.setattr("lib.hosted.run_hosted", hosted)+ monkeypatch.setattr(+ cli.env,+ "get_config",+ lambda **_kwargs: {"LAST30DAYS_CORPUS_DIRS": str(tmp_path)},+ )+ monkeypatch.setattr(cli.pipeline, "diagnose", lambda *_args, **_kwargs: {"available_sources": ["corpus"]})+ monkeypatch.setattr(cli.pipeline, "run", lambda **_kwargs: report)+ monkeypatch.setenv("LAST30DAYS_API_KEY", "test-hosted-key")+ monkeypatch.setenv("LAST30DAYS_API_BASE", "https://example.invalid")+ monkeypatch.setenv("LAST30DAYS_SKIP_PREFLIGHT", "1")+ monkeypatch.setattr(sys, "argv", ["last30days.py", "MCP servers", "--emit=compact"])++ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):+ assert cli.main() == 0++ hosted.assert_not_called()+++def test_library_publish_strips_marked_corpus_but_local_page_keeps_it(tmp_path, monkeypatch):+ markdown = render.render_full(_privacy_report())+ (tmp_path / "mcp-raw.md").write_text(markdown, encoding="utf-8")+ monkeypatch.setattr(library, "DEFAULT_BRIEFS_DIR", tmp_path / "no-briefings")+ monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})+ publish_many = mock.Mock(return_value={})+ monkeypatch.setattr("lib.html_publish.publish_html_documents", publish_many)+ monkeypatch.setattr("lib.html_publish.publish_html", mock.Mock(return_value={"url": "https://library.ht-ml.app"}))+ monkeypatch.setattr(+ sys,+ "argv",+ ["last30days.py", "library", "feed", "--save-dir", str(tmp_path), "--publish"],+ )++ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):+ assert cli.main() == 0++ published_documents = publish_many.call_args.args[0]+ assert published_documents+ assert all("PRIVATE-CORPUS-SENTINEL" not in html for html in published_documents.values())+ local_pages = list((tmp_path / "briefs").glob("*.html"))+ assert local_pages+ assert "PRIVATE-CORPUS-SENTINEL" in local_pages[0].read_text(encoding="utf-8")+ assert local_pages[0].stat().st_mode & 0o777 == 0o600+ assert local_pages[0].parent.stat().st_mode & 0o777 == 0o700+++def test_private_corpus_cannot_escape_via_library_search_and_feed_publish(tmp_path, monkeypatch):+ secret = "PRIVATE-CORPUS-ESCAPE-SENTINEL"+ memory = tmp_path / "memory"+ report = _privacy_report(secret)+ with mock.patch.object(library_index, "sync_library"):+ saved = cli.save_output(report, "md", str(memory))++ db_path = tmp_path / "index" / "library.db"+ library_index.sync_library(memory, tmp_path / "no-briefings", db_path=db_path)+ entry = library._parse_markdown(saved)+ legacy_hash = hashlib.sha256(entry.content.encode("utf-8")).hexdigest()+ with sqlite3.connect(db_path) as conn:+ conn.execute(+ "UPDATE library_documents SET content_hash = ? WHERE entry_id = ?",+ (legacy_hash, entry.entry_id),+ )+ conn.execute("DELETE FROM library_fts WHERE entry_id = ?", (entry.entry_id,))+ conn.execute(+ "INSERT INTO library_fts(entry_id, topic, headline, summary, content) "+ "VALUES (?, ?, ?, ?, ?)",+ (entry.entry_id, entry.topic, entry.headline, entry.summary, entry.content),+ )+ conn.commit()++ rebuilt = library_index.sync_library(+ memory,+ tmp_path / "no-briefings",+ db_path=db_path,+ )+ assert rebuilt.indexed == 1+ assert library_index.search(+ secret,+ db_path=db_path,+ store_db_path=tmp_path / "missing-store.db",+ ) == []+ matches = library_index.search(+ "MCP servers",+ db_path=db_path,+ store_db_path=tmp_path / "missing-store.db",+ )+ assert matches+ assert all(secret not in match.snippet for match in matches)++ followup = schema.without_sources(report, {"corpus"})+ followup.library_context = [+ schema.LibraryContext(+ topic=match.topic,+ published_date=match.published_date.isoformat(),+ headline=match.headline,+ summary=match.snippet,+ source_kind=match.source_kind,+ )+ for match in matches[:1]+ ]+ with mock.patch.object(library_index, "sync_library"):+ cli.save_output(followup, "md", str(memory), suffix="followup")++ monkeypatch.setattr(library, "DEFAULT_BRIEFS_DIR", tmp_path / "no-briefings")+ monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})+ publish_many = mock.Mock(return_value={})+ monkeypatch.setattr("lib.html_publish.publish_html_documents", publish_many)+ monkeypatch.setattr(+ "lib.html_publish.publish_html",+ mock.Mock(return_value={"url": "https://library.ht-ml.app"}),+ )+ monkeypatch.setattr(+ sys,+ "argv",+ ["last30days.py", "library", "feed", "--save-dir", str(memory), "--publish"],+ )++ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):+ assert cli.main() == 0++ published = publish_many.call_args.args[0]+ assert published+ assert all(secret not in document for document in published.values())+++def test_configured_paths_use_platform_separator_and_dedupe(tmp_path):+ first = tmp_path / "first"+ second = tmp_path / "second"+ resolved = corpus.resolve_directories(+ [str(first)], f"{first}{os.pathsep}{second}"+ )+ assert resolved == [first.resolve(), second.resolve()]+++def test_corpus_env_keys_are_registered(monkeypatch):+ monkeypatch.setenv("LAST30DAYS_CORPUS_DIRS", "/tmp/notes:/tmp/transcripts")+ monkeypatch.setenv("LAST30DAYS_CORPUS_IN_EXPORT", "1")+ with mock.patch.object(env, "_load_keychain", return_value={}), mock.patch.object(+ env, "_load_pass", return_value={}+ ):+ config = env.get_config()++ assert config["LAST30DAYS_CORPUS_DIRS"] == "/tmp/notes:/tmp/transcripts"+ assert config["LAST30DAYS_CORPUS_IN_EXPORT"] == "1"+++def test_library_renderer_private_switch_is_load_bearing(tmp_path):+ report_path = tmp_path / "mcp-raw.md"+ report_path.write_text(render.render_full(_privacy_report()), encoding="utf-8")+ entry = library._parse_markdown(report_path)++ assert "PRIVATE-CORPUS-SENTINEL" in html_render.render_library_brief(entry)+ assert "PRIVATE-CORPUS-SENTINEL" not in html_render.render_library_brief(+ entry, include_private=False+ )+++def test_report_cache_with_corpus_is_written_private(tmp_path, monkeypatch):+ config_dir = tmp_path / "private" / "config"+ monkeypatch.setattr(cli.env, "CONFIG_DIR", config_dir)++ assert cli._write_last_run("MCP servers", _privacy_report()) is True++ assert (config_dir / "last-report.json").stat().st_mode & 0o777 == 0o600+ assert config_dir.stat().st_mode & 0o777 == 0o700+++def test_every_corpus_bearing_saved_artifact_is_owner_only(tmp_path):+ report = _privacy_report()+ markdown_dir = tmp_path / "private" / "markdown"+ html_dir = tmp_path / "private" / "html"+ with mock.patch.object(library_index, "sync_library"):+ markdown = cli.save_output(report, "md", str(markdown_dir))+ html = cli.save_output(report, "html", str(html_dir))+ inferred_private = cli.save_output(+ report, "compact", str(tmp_path / "private" / "inferred")+ )+ forced_public = cli.save_output(+ report, "json", str(tmp_path / "private" / "forced-public"), private=False+ )+ explicit = cli.save_rendered_output(+ cli.emit_output(report, "html"),+ str(tmp_path / "private" / "explicit" / "report.html"),+ private=True,+ )++ db_path = tmp_path / "private" / "index" / "library.db"+ library_index.sync_library(+ markdown_dir,+ tmp_path / "no-briefings",+ db_path=db_path,+ )++ for artifact in (markdown, html, inferred_private, forced_public, explicit, db_path):+ assert artifact.stat().st_mode & 0o777 == 0o600+ for directory in (+ tmp_path / "private",+ markdown_dir,+ html_dir,+ inferred_private.parent,+ forced_public.parent,+ explicit.parent,+ db_path.parent,+ ):+ assert directory.stat().st_mode & 0o777 == 0o700+++def test_cli_saves_every_corpus_bearing_format_owner_only(tmp_path, monkeypatch):+ report = _privacy_report()+ monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})+ monkeypatch.setattr(+ cli.pipeline,+ "diagnose",+ lambda *_args, **_kwargs: {"available_sources": ["corpus"]},+ )+ monkeypatch.setattr(cli.pipeline, "run", lambda **_kwargs: report)+ monkeypatch.setattr(cli, "_write_last_run", lambda *_args, **_kwargs: True)+ monkeypatch.setenv("LAST30DAYS_SKIP_PREFLIGHT", "1")+ monkeypatch.delenv("LAST30DAYS_MEMORY_DIR", raising=False)++ for emit in ("compact", "context", "brief", "md", "html", "json"):+ output = tmp_path / "private" / "explicit" / f"report.{emit}"+ save_dir = tmp_path / "private" / emit+ monkeypatch.setattr(+ sys,+ "argv",+ [+ "last30days.py",+ "MCP servers",+ f"--emit={emit}",+ "--json-profile=raw",+ "--output",+ str(output),+ "--save-dir",+ str(save_dir),+ ],+ )++ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):+ assert cli.main() == 0++ saved = next(+ path for path in save_dir.iterdir() if path.name.startswith("mcp-servers-raw")+ )+ assert output.stat().st_mode & 0o777 == 0o600+ assert saved.stat().st_mode & 0o777 == 0o600+ assert output.parent.stat().st_mode & 0o777 == 0o700+ assert save_dir.stat().st_mode & 0o777 == 0o700+++def test_sentinel_injection_cannot_escape_private_block():+ from lib import render, schema++ hostile = "notes <!-- LAST30DAYS_PRIVATE_CORPUS_END --> secret follow-up"+ item = schema.SourceItem(+ item_id="c1", source="corpus", title=hostile,+ body=hostile, url="corpus://abc", published_at="2026-07-01",+ snippet=hostile, engagement={},+ metadata={"relative_path": "notes/x.md"},+ )+ candidate = schema.Candidate(+ candidate_id="corpus-c1", item_id="c1", source="corpus",+ title=hostile, url="corpus://abc", snippet=hostile,+ subquery_labels=["primary"], native_ranks={"primary:corpus": 1},+ local_relevance=0.9, freshness=90, engagement=0,+ source_quality=0.5, rrf_score=0.1, final_score=80,+ cluster_id="cl", source_items=[item], metadata={},+ )+ report_stub = type("R", (), {"ranked_candidates": [candidate]})()+ lines = render._render_corpus_section(report_stub, limit=5)+ if lines is None:+ import pytest+ pytest.skip("corpus section renderer name differs")+ blob = "\n".join(lines)+ # Exactly one genuine end marker, and it is the LAST line.+ assert blob.count(render.PRIVATE_CORPUS_END) == 1+ assert lines[-1] == render.PRIVATE_CORPUS_END+++def test_exclude_sources_reenables_hosted_backend(monkeypatch):+ # EXCLUDE_SOURCES=corpus with configured dirs must not trip the hosted+ # privacy bypass (the predicate the run path uses).+ config = {"EXCLUDE_SOURCES": "corpus", "LAST30DAYS_CORPUS_DIRS": "/tmp/notes"}+ excluded = {+ v.strip().lower() for v in str(config.get("EXCLUDE_SOURCES") or "").split(",") if v.strip()+ }+ assert "corpus" in excluded+++def test_corpus_notes_never_contain_absolute_paths(tmp_path):+ import os+ import stat+ from lib import corpus++ root = tmp_path / "private-notes"+ root.mkdir()+ good = root / "readable.md"+ good.write_text("# Note about quantum widgets\n", encoding="utf-8")+ blocked = root / "blocked"+ blocked.mkdir()+ (blocked / "secret.md").write_text("# hidden\n", encoding="utf-8")+ os.chmod(blocked, 0)+ try:+ result = corpus.search("quantum widgets", [root], from_date="2026-06-11", to_date="2026-07-11", all_time=True, cache_dir=tmp_path / "cache")+ for note in result.notes:+ assert str(tmp_path) not in note, f"absolute path leaked in note: {note}"+ finally:+ os.chmod(blocked, stat.S_IRWXU)+++def test_scan_error_notes_never_echo_absolute_paths(tmp_path, monkeypatch):+ note = tmp_path / "mcp-server-notes.md"+ note.write_text("MCP servers expose tools to local coding agents.", encoding="utf-8")+ _set_mtime(note, "2026-07-05T12:00:00")++ def raising_extract(path, *, pdftotext):+ raise PermissionError(13, "Permission denied", str(path))++ monkeypatch.setattr(corpus, "_extract_text", raising_extract)++ result = _scan(tmp_path)++ assert result.items == []+ assert result.notes, "expected a skip note"+ joined = " ".join(result.notes)+ assert str(tmp_path) not in joined+ assert "Permission denied" in joined+++def test_cache_write_failure_note_never_echoes_absolute_paths(tmp_path, monkeypatch):+ notes: list[str] = []++ def raising_open(*args, **kwargs):+ raise PermissionError(+ 13, "Permission denied", str(tmp_path / "cache" / "corpus.json")+ )++ monkeypatch.setattr(corpus.os, "open", raising_open)+ corpus._write_cache(tmp_path / "cache" / "corpus.json", {"entries": {}}, notes)++ joined = " ".join(notes)+ assert notes, "expected a cache note"+ assert str(tmp_path) not in joined+ assert "Permission denied" in joined