Skip to content

AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary

feat(sources): local corpus source - your own files as a ranked signal (#808)

Matt Van Horn3b0b021e2caf
diff --git a/CONFIGURATION.md b/CONFIGURATION.md
index 3383d45f..d0f2f31a 100644
--- a/CONFIGURATION.md
+++ b/CONFIGURATION.md
@@ -46,6 +46,8 @@ The engine's `.env` reader doesn't expand `$HOME` — only the tilde, via `Path(
- `--save-dir <path>` - one-off output location. **Flag wins over env var.** If neither flag nor env var is set, the engine does not write a file (DB persistence is independent — see `LAST30DAYS_STORE` below).
- `--output <file>` - write the rendered output to an exact file path, using the format selected by `--emit`.
- `--json-profile {agent,raw}` - select the research JSON shape used with `--emit=json`. `agent` is the default, versioned workflow contract; `raw` preserves the full internal `Report` dump for debugging and power users. See the [JSON export reference](docs/reference/json-export.md).
+- `--corpus <dir>` - add a local `.md`/`.txt` directory as a private ranked source; repeat the flag for multiple directories. PDFs are extracted only when `pdftotext` is on PATH and otherwise skip with a note. File modification time supplies recency, so the normal research window applies.
+- `--corpus-all-time` - include relevant registered files whose modification time is older than the current research window. Without this flag, a 30-day run includes only files modified in those 30 days.
- `--register {default,exec,dev,creator,eli5}` - shape a standard single-topic Markdown or HTML research brief for its audience. `exec` is decisions-first with five core findings and numbers up top; `dev` gives GitHub, code, and technical signals more room; `creator` leads with hooks, Best Takes, community reactions, and virality metrics; `eli5` keeps the established evidence layout and asks the synthesizing agent for accessible language. Registers do not change retrieval, JSON exports, discovery, drill, library feed/search, or comparison output.
- `--discover <domain>` - topic-less trending discovery. Sweeps rising/top-week Reddit listings (category-mapped communities, with r/all as the uncategorized fallback), Hacker News front/best stories, Digg AI 1000 clusters when `digg-pp-cli` is on PATH, and broad X activity when an X backend is authenticated, then returns 5-10 engagement-velocity-ranked topics. Run without a positional topic; it is mutually exclusive with `--drill`. `--emit=json` uses the separate versioned discovery contract documented in the [JSON export reference](docs/reference/json-export.md).
- `--drill <target>` - deep follow-up over the fresh `~/.config/last30days/last-report.json` cache. Accepts a 1-based index (`--drill "cluster 3"` or `--drill "3"`) or a fuzzy cluster title/entity description. It re-fetches only sources that contributed to the matched cluster, enables their deep comment/transcript enrichment paths, merges/dedupes the evidence, and replaces the cache so drills can chain. Run it without a positional topic; if the cache is absent or expired, run a normal research pass first.
@@ -98,12 +100,32 @@ Override the global location with `LAST30DAYS_CONFIG_DIR=/path` (or `LAST30DAYS_
The project-scoped file is useful for **intentional per-client setups**: drop a `.claude/last30days.env` into each client folder (`SCRAPECREATORS_API_KEY`, `INCLUDE_SOURCES`, `LAST30DAYS_MEMORY_DIR`, `BSKY_HANDLE`, etc), then opt in with `LAST30DAYS_TRUST_PROJECT_CONFIG=1` from your shell or `~/.config/last30days/.env`. Folder-mode hosts such as Codex desktop do not trust hidden project config by default, and discovery stops at the git root so unrelated parent folders cannot silently influence runs.
-**`LAST30DAYS_API_KEY`** + **`LAST30DAYS_API_BASE`** - optional remote-API backend. Set BOTH to route research through a remote API endpoint instead of running the local sources: `LAST30DAYS_API_BASE` is the endpoint (there is no built-in default), and `LAST30DAYS_API_KEY` is the bearer key for it. When both are set (and `--mock` is not passed), the engine submits the topic to that endpoint, polls with progress on stderr, and prints the server's report; none of the per-source keys below are used for that run. Non-default `--register` selections are forwarded with the request so server-side synthesis uses the same audience preset. Leave either unset to run local sources exactly as normal. Unlike the other keys here, these two are read only from the **process environment** (export them in your shell or host config) - they are deliberately not loaded from the `.env` files above, so a project-scoped `.env` can never silently redirect research to a remote endpoint. The remote endpoint does not return the local `Report` needed for the versioned agent JSON profile; use `--emit=json --json-profile=raw` for its existing server-response JSON contract.
+**`LAST30DAYS_API_KEY`** + **`LAST30DAYS_API_BASE`** - optional remote-API backend. Set BOTH to route research through a remote API endpoint instead of running the local sources: `LAST30DAYS_API_BASE` is the endpoint (there is no built-in default), and `LAST30DAYS_API_KEY` is the bearer key for it. When both are set (and `--mock` is not passed), the engine submits the topic to that endpoint, polls with progress on stderr, and prints the server's report; none of the per-source keys below are used for that run. A configured local corpus is the privacy exception: the engine bypasses the hosted backend and runs locally rather than forwarding file-derived input. Non-default `--register` selections are forwarded with the request so server-side synthesis uses the same audience preset. Leave either unset to run local sources exactly as normal. Unlike the other keys here, these two are read only from the **process environment** (export them in your shell or host config) - they are deliberately not loaded from the `.env` files above, so a project-scoped `.env` can never silently redirect research to a remote endpoint. The remote endpoint does not return the local `Report` needed for the versioned agent JSON profile; use `--emit=json --json-profile=raw` for its existing server-response JSON contract.
+
+### Local corpus (your files)
+
+Register persistent directories with `LAST30DAYS_CORPUS_DIRS`. Separate paths with `:` on macOS/Linux (the platform path separator is `;` on Windows):
+
+```bash
+# ~/.config/last30days/.env
+LAST30DAYS_CORPUS_DIRS=~/notes:~/meeting-transcripts
+# LAST30DAYS_CORPUS_IN_EXPORT=1 # explicit agent-JSON opt-in; off by default
+```
+
+The slash-command experience remains primary: ask `/last30days` to include your registered notes. For direct engine scripting or development, the equivalent one-off invocation is:
+
+```bash
+python3 skills/last30days/scripts/last30days.py "MCP servers" \
+ --corpus ~/notes --corpus ~/meeting-transcripts
+```
+
+**Privacy:** corpus files are read locally, never sent through a source HTTP client, never forwarded to `LAST30DAYS_API_BASE`, never included in remote reranker/fun-scoring prompts, and do not consume network-source concurrency or retry budget. Matches appear in a badged **From your files** section. Corpus candidates are removed from `--publish-html`, `library feed --publish`, and the versioned agent JSON export by default, including corpus-derived cluster titles and source outcomes. Set `LAST30DAYS_CORPUS_IN_EXPORT=1` only when you intentionally want corpus results in the agent JSON written to local stdout/files. The unversioned `--json-profile=raw` debug dump remains a full local report and can contain corpus text; do not redirect it to an external system unless that is intentional. Extracted text is cached by file mtime in `~/.config/last30days/corpus-cache.json` with mode `0600`; a corpus-bearing `last-report.json` cache is also tightened to `0600`. Delete either cache at any time to clear it.
**Source-by-source** - what each key unlocks:
| Source | Key(s) | Required for | Free tier |
|---|---|---|---|
+| Local corpus | `--corpus <dir>` or `LAST30DAYS_CORPUS_DIRS` | private `.md`/`.txt`; `.pdf` when `pdftotext` is on PATH | yes (offline) |
| Reddit (public) | none (default); `SCRAPECREATORS_API_KEY` + `LAST30DAYS_REDDIT_BACKEND=scrapecreators` to pin SC primary with public fallback | always on; SC pin requires `SCRAPECREATORS_API_KEY` | yes |
| Hacker News | none | always on | yes |
| Polymarket | none | always on | yes |
diff --git a/docs/reference/json-export.md b/docs/reference/json-export.md
index 85d01ff6..e1b77299 100644
--- a/docs/reference/json-export.md
+++ b/docs/reference/json-export.md
@@ -21,6 +21,10 @@ python3 skills/last30days/scripts/last30days.py "AI coding agents" --emit=json -
The raw profile is intentionally unversioned and may change when pipeline internals change. It preserves the JSON serialization used before the agent profile was introduced.
+### Local corpus privacy
+
+Evidence from `--corpus` / `LAST30DAYS_CORPUS_DIRS` is excluded from the versioned agent profile by default. The exclusion removes corpus results, corpus-only clusters, corpus source outcomes, freshness verdicts, and titles derived from a corpus representative. Set `LAST30DAYS_CORPUS_IN_EXPORT=1` only for a run whose JSON is intentionally allowed to contain local file contents. This opt-in does not change the schema shape or version; it permits `source: "corpus"` entries in the existing result fields. The unversioned `raw` profile is a complete local debug dump and may contain corpus paths and text.
+
## Discovery export
Discovery mode has a separate versioned contract so its topic results do not change the normal research export:
diff --git a/skills/last30days/SKILL.md b/skills/last30days/SKILL.md
index 98996fa1..818c7bce 100644
--- a/skills/last30days/SKILL.md
+++ b/skills/last30days/SKILL.md
@@ -416,7 +416,7 @@ Set `LAST30DAYS_MEMORY_DIR` before invoking the skill to choose where raw resear
The engine reads `LAST30DAYS_MEMORY_DIR` from either the process env or `~/.config/last30days/.env`, so direct CLI invocations (`python3 scripts/last30days.py ...`) without `--save-dir` will still save when the env var is set. Mirrors the `LAST30DAYS_STORE` env-or-flag convention. Explicit `--save-dir` always wins.
-When both `LAST30DAYS_API_KEY` and `LAST30DAYS_API_BASE` are set, the engine runs the research through that configured remote API instead of local sources (unless `--mock` is passed); `LAST30DAYS_API_BASE` is the endpoint and has no built-in default, so leaving either variable unset runs local sources normally. The invocation is unchanged: same flags, `--quick`/`--deep` map to search depth, a non-default `--register` is forwarded for server-side synthesis, progress lines still stream on stderr (`[narrate] step=...` plus a compact elapsed/eta line), and the report prints on stdout and saves to the memory dir as usual, so Steps 1-4 proceed normally on the output. The exception is research JSON: the remote endpoint does not return the local `Report` needed for the versioned agent profile, so use `--emit=json --json-profile=raw` for its existing server-response JSON contract. No per-source keys or setup-wizard credentials are needed for the search itself in this mode. Two engine exits need specific handling: exit code 3 means the API asked a clarifying question first - the engine prints the question and options on stderr; present them to the user and re-run with the chosen angle folded into the topic. An insufficient-credits failure (HTTP 402) prints the account's balance, the amount needed, and a billing link - relay those lines to the user verbatim; do not fall back to WebSearch-only synthesis.
+When both `LAST30DAYS_API_KEY` and `LAST30DAYS_API_BASE` are set, the engine runs the research through that configured remote API instead of local sources (unless `--mock` is passed); `LAST30DAYS_API_BASE` is the endpoint and has no built-in default, so leaving either variable unset runs local sources normally. A configured `--corpus` / `LAST30DAYS_CORPUS_DIRS` is the privacy exception: the engine bypasses the hosted backend and runs locally so no file-derived input is forwarded. The invocation is otherwise unchanged: same flags, `--quick`/`--deep` map to search depth, a non-default `--register` is forwarded for server-side synthesis, progress lines still stream on stderr (`[narrate] step=...` plus a compact elapsed/eta line), and the report prints on stdout and saves to the memory dir as usual, so Steps 1-4 proceed normally on the output. The exception is research JSON: the remote endpoint does not return the local `Report` needed for the versioned agent profile, so use `--emit=json --json-profile=raw` for its existing server-response JSON contract. No per-source keys or setup-wizard credentials are needed for the search itself in this mode. Two engine exits need specific handling: exit code 3 means the API asked a clarifying question first - the engine prints the question and options on stderr; present them to the user and re-run with the chosen angle folded into the topic. An insufficient-credits failure (HTTP 402) prints the account's balance, the amount needed, and a billing link - relay those lines to the user verbatim; do not fall back to WebSearch-only synthesis.
**Developer-only eval capture:** `--record-fixtures <dir>` is a hidden direct-engine flag for maintaining the deterministic research-quality suite. It records scrubbed HTTP and CLI-adapter responses to `<dir>/http.json`; it is never part of the user-facing slash-command invocation. Follow `docs/reference/eval.md` for fixture review, replay, and baseline rules.
@@ -644,10 +644,12 @@ SKILL_DIR="<absolute path of the directory containing the SKILL.md you just Read
"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" --diagnose
```
-`--diagnose` prints JSON. `ACTIVE_SOURCES_LIST` is its `available_sources` array — the engine's authoritative source set, computed after credential resolution. Map the tokens to display names: `reddit`→Reddit, `hackernews`→Hacker News, `polymarket`→Polymarket, `github`→GitHub, `digg`→Digg, `x`→X, `youtube`→YouTube, `tiktok`→TikTok, `instagram`→Instagram, `threads`→Threads, `pinterest`→Pinterest, `linkedin`→LinkedIn, `bluesky`→Bluesky, `perplexity`→Perplexity, `grounding`→Web, `jobs`→Jobs.
+`--diagnose` prints JSON. `ACTIVE_SOURCES_LIST` is its `available_sources` array — the engine's authoritative source set, computed after credential resolution. Map the tokens to display names: `reddit`→Reddit, `hackernews`→Hacker News, `polymarket`→Polymarket, `github`→GitHub, `digg`→Digg, `x`→X, `youtube`→YouTube, `tiktok`→TikTok, `instagram`→Instagram, `threads`→Threads, `pinterest`→Pinterest, `linkedin`→LinkedIn, `bluesky`→Bluesky, `perplexity`→Perplexity, `grounding`→Web, `jobs`→Jobs, `corpus`→Your files.
- If EXCLUDE_SOURCES is set (comma-separated, case-insensitive): drop any matching source from ACTIVE_SOURCES_LIST before displaying
+**Local corpus source:** If the user asks to include their own notes/documents, preserve each supplied directory as a repeatable `--corpus <dir>` engine flag. `LAST30DAYS_CORPUS_DIRS` activates persistent registered directories automatically. Do not WebSearch, upload, quote into a hosted request, or otherwise expose those paths or contents. Corpus retrieval is an offline source lane; its candidates also bypass remote reranker/fun-scoring prompts and use deterministic local scoring. The engine renders matches under the 🔒 **From your files** badge. The normal recency window uses file modification time; add `--corpus-all-time` only when the user explicitly asks to include older files. Corpus evidence is excluded from `--publish-html`, `library feed --publish`, and agent JSON by default. `LAST30DAYS_CORPUS_IN_EXPORT=1` is the explicit agent-JSON privacy opt-in; never enable it on the user's behalf. When a corpus is configured alongside `LAST30DAYS_API_KEY`/`LAST30DAYS_API_BASE`, the engine deliberately bypasses the hosted backend and runs locally.
+
**Perplexity source:** use it only when the user asks for Perplexity, Deep Research, or paid grounded synthesis, or when `perplexity` is already enabled in `INCLUDE_SOURCES` / `--search`. Direct `PERPLEXITY_API_KEY` supports Sonar synthesis, Search API rows, and async Deep Research. `OPENROUTER_API_KEY` is only a Sonar fallback. Normal runs default to `LAST30DAYS_PERPLEXITY_MODE=sonar`; use `search` for raw ranked web rows, `both` for synthesis plus rows, and `--deep-research` for `sonar-deep-research` with a 600s default wall timeout. A local Deep Research timeout is not a failed API key; inspect the raw artifact's async request id/status and resume by id if needed.
**Reddit backend pin:** Reddit defaults to the free public backend with ScrapeCreators as a backup when `SCRAPECREATORS_API_KEY` is available. If the user says public Reddit is shallow, bot-gated, or missing nested comments, tell them they can set `LAST30DAYS_REDDIT_BACKEND=scrapecreators` alongside `SCRAPECREATORS_API_KEY` to make ScrapeCreators primary and keep public Reddit as fallback. Do not set this automatically for normal runs.
diff --git a/skills/last30days/scripts/last30days.py b/skills/last30days/scripts/last30days.py
index d3c05000..dbe5ae63 100644
--- a/skills/last30days/scripts/last30days.py
+++ b/skills/last30days/scripts/last30days.py
@@ -48,7 +48,7 @@ if os.name == "nt":
SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))
-from lib import dates, env, freshness, html_render, http, permission_preflight, pipeline, registers, render, schema, ui
+from lib import corpus, dates, env, freshness, html_render, http, permission_preflight, pipeline, registers, render, schema, ui
_child_pids: set[int] = set()
_child_pids_lock = threading.Lock()
@@ -121,6 +121,34 @@ def slugify(value: str) -> str:
return slug or "last30days"
+def _report_has_private_corpus(report: schema.Report) -> bool:
+ items_by_source = getattr(report, "items_by_source", {})
+ if isinstance(items_by_source, dict) and items_by_source.get("corpus"):
+ return True
+ candidates = getattr(report, "ranked_candidates", ())
+ if not isinstance(candidates, (list, tuple)):
+ return False
+ return any(
+ candidate.source == "corpus"
+ or any(item.source == "corpus" for item in candidate.source_items)
+ for candidate in candidates
+ )
+
+
+def _ensure_output_directory(path: Path, *, private: bool) -> None:
+ if not private:
+ path.mkdir(parents=True, exist_ok=True)
+ return
+ missing: list[Path] = []
+ current = path
+ while not current.exists():
+ missing.append(current)
+ current = current.parent
+ path.mkdir(parents=True, exist_ok=True, mode=0o700)
+ for directory in missing:
+ directory.chmod(0o700)
+
+
def save_output(
report: schema.Report,
emit: str,
@@ -131,10 +159,10 @@ def save_output(
rendered_content: str | None = None,
json_profile: str = "agent",
register: str = "default",
+ private: bool | None = None,
) -> Path:
from datetime import datetime
path = Path(save_dir).expanduser().resolve()
- path.mkdir(parents=True, exist_ok=True)
slug = slugify(topic_override or report.topic)
extension = "json" if emit == "json" else "html" if emit == "html" else "md"
raw_label = "raw-html" if emit == "html" else "raw"
@@ -159,10 +187,16 @@ def save_output(
)
else:
content = render.render_full(report)
+ private_corpus = _report_has_private_corpus(report) or bool(private)
+ _ensure_output_directory(path, private=private_corpus)
encoded = content.encode("utf-8")
for candidate in candidates:
try:
- fd = os.open(candidate, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
+ fd = os.open(
+ candidate,
+ os.O_CREAT | os.O_EXCL | os.O_WRONLY,
+ 0o600 if private_corpus else 0o644,
+ )
except FileExistsError:
continue
with os.fdopen(fd, "wb") as f:
@@ -194,10 +228,25 @@ def save_output(
)
-def save_rendered_output(rendered_content: str, output_file: str) -> Path:
+def save_rendered_output(
+ rendered_content: str,
+ output_file: str,
+ *,
+ private: bool = False,
+) -> Path:
out_path = Path(output_file).expanduser().resolve()
- out_path.parent.mkdir(parents=True, exist_ok=True)
- out_path.write_text(rendered_content, encoding="utf-8")
+ _ensure_output_directory(out_path.parent, private=private)
+ if private and out_path.exists():
+ out_path.chmod(0o600)
+ fd = os.open(
+ out_path,
+ os.O_CREAT | os.O_TRUNC | os.O_WRONLY,
+ 0o600 if private else 0o644,
+ )
+ with os.fdopen(fd, "w", encoding="utf-8") as handle:
+ handle.write(rendered_content)
+ if private:
+ out_path.chmod(0o600)
return out_path
@@ -384,14 +433,21 @@ def _scoped_store_db(args: argparse.Namespace) -> Path | None:
def persist_report(report: schema.Report, store_db: Path | None = None) -> dict[str, int]:
import store
+ private_corpus = _report_has_private_corpus(report)
with store.scoped_db(store_db):
+ if private_corpus:
+ store.ensure_private_db_files()
store.init_db()
+ if private_corpus:
+ store.ensure_private_db_files()
topic_row = store.add_topic(report.topic)
topic_id = topic_row["id"]
source_mode = ",".join(sorted(report.items_by_source)) or "v3"
run_id = store.record_run(topic_id, source_mode=source_mode, status="running")
try:
findings = store.findings_from_report(report)
+ if private_corpus:
+ store.ensure_private_db_files()
counts = store.store_findings(run_id, topic_id, findings)
store.update_run(
run_id,
@@ -403,6 +459,9 @@ def persist_report(report: schema.Report, store_db: Path | None = None) -> dict[
except Exception as exc:
store.update_run(run_id, status="failed", error_message=str(exc)[:500])
raise
+ finally:
+ if private_corpus:
+ store.ensure_private_db_files()
def build_parser() -> argparse.ArgumentParser:
@@ -466,6 +525,18 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument("--no-browser-cookies", action="store_true",
help="Disable browser-cookie extraction even when FROM_BROWSER is configured")
parser.add_argument("--save-dir", help="Optional directory for saving the rendered output")
+ parser.add_argument(
+ "--corpus",
+ action="append",
+ default=[],
+ metavar="DIR",
+ help="Add a local .md/.txt/.pdf directory as a private ranked source (repeatable)",
+ )
+ parser.add_argument(
+ "--corpus-all-time",
+ action="store_true",
+ help="Include matching corpus files older than the research window",
+ )
parser.add_argument("--output", help="Optional exact file path for saving the rendered output")
parser.add_argument("--synthesis-file", help="Markdown synthesis to embed in --emit=html output")
parser.add_argument("--publish-html", action="store_true",
@@ -827,7 +898,12 @@ def _write_last_run(
if env.CONFIG_DIR is None:
return False
target = env.CONFIG_DIR
- target.mkdir(parents=True, exist_ok=True)
+ cached_reports = entity_reports or [(report.topic, report)]
+ has_private_corpus = any(
+ cached_report.items_by_source.get("corpus")
+ for _, cached_report in cached_reports
+ )
+ _ensure_output_directory(target, private=has_private_corpus)
counts = {source: len(items) for source, items in report.items_by_source.items()}
payload = {
"topic": topic,
@@ -838,7 +914,6 @@ def _write_last_run(
"comparison": bool(entity_reports),
}
(target / "last-run.json").write_text(json.dumps(payload, indent=2))
- cached_reports = entity_reports or [(report.topic, report)]
cache_payload = {
"schema": REPORT_CACHE_VERSION,
"topic": topic,
@@ -849,7 +924,10 @@ def _write_last_run(
for label, cached_report in cached_reports
],
}
- (target / "last-report.json").write_text(json.dumps(cache_payload, indent=2))
+ report_cache_path = target / "last-report.json"
+ report_cache_path.write_text(json.dumps(cache_payload, indent=2))
+ if has_private_corpus:
+ report_cache_path.chmod(0o600)
return True
except Exception as exc:
# Never fatal, but never silent either (#787's lesson): callers that
@@ -1105,6 +1183,8 @@ def _run_drill(
if "trustpilot" in sources else None
),
internal_subrun=True,
+ corpus_dirs=args.corpus,
+ corpus_all_time=args.corpus_all_time,
)
except Exception:
progress.end_processing()
@@ -1388,9 +1468,18 @@ def _render_save_and_print(
json_profile=args.json_profile,
register=audience.name,
)
+ has_private_corpus = _report_has_private_corpus(report) or bool(
+ entity_reports
+ and any(_report_has_private_corpus(entity) for _label, entity in entity_reports)
+ )
+ private_saved_format = has_private_corpus
publish_companion_paths: list[Path] = []
if args.output:
- output_path = save_rendered_output(rendered, args.output)
+ output_path = save_rendered_output(
+ rendered,
+ args.output,
+ private=private_saved_format,
+ )
if args.emit == "html":
publish_companion_paths.append(output_path)
sys.stderr.write(f"[last30days] Saved output to {output_path}\n")
@@ -1407,6 +1496,7 @@ def _render_save_and_print(
rendered_content=rendered if is_comparison_html else None,
json_profile=args.json_profile,
register=audience.name,
+ private=private_saved_format,
)
if args.emit == "html":
publish_companion_paths.append(save_path)
@@ -1421,6 +1511,7 @@ def _render_save_and_print(
suffix=args.save_suffix or "",
synthesis_md=synthesis_md,
json_profile=args.json_profile,
+ private=_report_has_private_corpus(entity_report),
)
comparison_peer_paths.append(peer_path)
sys.stderr.write(f"[last30days] Saved output to {peer_path}\n")
@@ -1432,8 +1523,39 @@ def _render_save_and_print(
sys.stderr.flush()
if args.publish_html:
try:
+ has_private_corpus = "corpus" in report.source_status or bool(
+ entity_reports
+ and any("corpus" in entity.source_status for _label, entity in entity_reports)
+ )
+ publish_rendered = rendered
+ if has_private_corpus:
+ sys.stderr.write(
+ "[last30days] Excluding local corpus evidence and synthesis from published HTML.\n"
+ )
+ if entity_reports:
+ publish_rendered = emit_comparison_output(
+ [
+ (label, schema.without_sources(entity, {"corpus"}))
+ for label, entity in entity_reports
+ ],
+ "html",
+ fun_level=fun_level,
+ save_path=footer_save_path,
+ synthesis_md=None,
+ json_profile=args.json_profile,
+ )
+ else:
+ publish_rendered = emit_output(
+ schema.without_sources(report, {"corpus"}),
+ "html",
+ fun_level=fun_level,
+ save_path=footer_save_path,
+ synthesis_md=None,
+ json_profile=args.json_profile,
+ register=audience.name,
+ )
publish_result = publish_rendered_html(
- rendered,
+ publish_rendered,
password=_publish_password_for_args(args, config),
companion_paths=publish_companion_paths,
)
@@ -1594,7 +1716,10 @@ def _run_library_feed(args: argparse.Namespace, config: dict[str, object]) -> in
output_dir.mkdir(parents=True, exist_ok=True)
library_id = library.get_or_create_library_id(output_dir)
rendered_briefs_dir = output_dir / "briefs"
- rendered_briefs_dir.mkdir(parents=True, exist_ok=True)
+ has_private_entries = any(
+ render.PRIVATE_CORPUS_START in entry.content for entry in entries
+ )
+ _ensure_output_directory(rendered_briefs_dir, private=has_private_entries)
def _preserve_hand_written_page(existing_path: Path, generated_marker: str) -> None:
"""Back up any page library feed did not generate before overwriting it."""
@@ -1617,13 +1742,19 @@ def _run_library_feed(args: argparse.Namespace, config: dict[str, object]) -> in
f"library feed; preserved the original at {backup.name}\n"
)
- brief_documents: dict[str, str] = {}
+ publishable_brief_documents: dict[str, str] = {}
for entry in entries:
rendered = html_render.render_library_brief(entry)
target = rendered_briefs_dir / entry.output_name
_preserve_hand_written_page(target, html_render.LIBRARY_BRIEF_MARKER)
- target.write_text(rendered, encoding="utf-8")
- brief_documents[entry.entry_id] = rendered
+ save_rendered_output(
+ rendered,
+ str(target),
+ private=render.PRIVATE_CORPUS_START in entry.content,
+ )
+ publishable_brief_documents[entry.entry_id] = html_render.render_library_brief(
+ entry, include_private=False
+ )
current_brief_names = {entry.output_name for entry in entries}
for path in rendered_briefs_dir.glob("*.html"):
@@ -1664,7 +1795,7 @@ def _run_library_feed(args: argparse.Namespace, config: dict[str, object]) -> in
entry_urls: dict[str, str] = {}
try:
brief_results = html_publish.publish_html_documents(
- brief_documents,
+ publishable_brief_documents,
password=password,
)
entry_urls = {
@@ -1810,6 +1941,23 @@ def _main(
)
return 2
config = env.get_config(policy=_config_policy_for_args(args, topic, extra_argv))
+ resolved_corpus_dirs = corpus.resolve_directories(
+ args.corpus, config.get("LAST30DAYS_CORPUS_DIRS")
+ )
+ # EXCLUDE_SOURCES=corpus disables corpus retrieval entirely; the hosted
+ # privacy bypass below must use the same predicate, or hosted users with
+ # configured-but-excluded dirs silently lose the remote backend.
+ excluded_sources = {
+ value.strip().lower()
+ for value in str(config.get("EXCLUDE_SOURCES") or "").split(",")
+ if value.strip()
+ }
+ if "corpus" in excluded_sources:
+ resolved_corpus_dirs = []
+ if resolved_corpus_dirs:
+ config["_CORPUS_DIRS"] = [str(path) for path in resolved_corpus_dirs]
+ if _config_truthy(config.get("LAST30DAYS_CORPUS_IN_EXPORT")):
+ config["_CORPUS_IN_EXPORT"] = True
_propagate_config_to_environ(config)
# Env-var fallback for --save-dir, mirroring the LAST30DAYS_STORE pattern below.
@@ -1979,6 +2127,15 @@ def _main(
# instead of local sources; no local provider keys are needed (see
# lib/hosted.py). With either env var unset, behavior below is byte-identical
# to local-only runs - there is no built-in endpoint.
+ if (
+ topic
+ and resolved_corpus_dirs
+ and env.read_secret_env("LAST30DAYS_API_KEY")
+ and os.environ.get("LAST30DAYS_API_BASE")
+ ):
+ sys.stderr.write(
+ "[last30days] Local corpus configured; bypassing the hosted backend so files stay on this machine.\n"
+ )
if (
topic
and not args.diagnose
@@ -1986,6 +2143,7 @@ def _main(
and not args.record_fixtures
and env.read_secret_env("LAST30DAYS_API_KEY")
and os.environ.get("LAST30DAYS_API_BASE")
+ and not resolved_corpus_dirs
):
if _freshness_enabled(args, config):
if args.verify_freshness is True:
@@ -2238,6 +2396,8 @@ def _main(
internal_subrun=comp_enabled,
hiring_signals_mode=args.hiring_signals,
save_dir=args.save_dir,
+ corpus_dirs=args.corpus,
+ corpus_all_time=args.corpus_all_time,
)
r.artifacts["resolved"] = {
"entity": topic,
@@ -2375,6 +2535,8 @@ def _main(
hiring_signals_mode=args.hiring_signals,
internal_subrun=True,
save_dir=args.save_dir,
+ corpus_dirs=args.corpus,
+ corpus_all_time=args.corpus_all_time,
)
report.artifacts["resolved"] = resolved_effective
return report
diff --git a/skills/last30days/scripts/lib/corpus.py b/skills/last30days/scripts/lib/corpus.py
new file mode 100644
index 00000000..9b4a1c4e
--- /dev/null
+++ b/skills/last30days/scripts/lib/corpus.py
@@ -0,0 +1,439 @@
+"""Deterministic, local-only document corpus source.
+
+The corpus adapter deliberately has no HTTP dependency. It scans explicitly
+registered directories, extracts small text documents (and PDFs only when the
+local ``pdftotext`` binary is available), and returns normalized ``SourceItem``
+objects for the shared relevance/fusion pipeline.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import subprocess
+import threading
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from pathlib import Path
+from shutil import which
+from typing import Any, Iterable
+
+from . import entity_extract, log, relevance, schema
+
+SOURCE = "corpus"
+SUPPORTED_SUFFIXES = {".md", ".txt", ".pdf"}
+IGNORED_DIRECTORIES = {".git", "node_modules"}
+MAX_FILES = 500
+MAX_TEXT_CHARS = 1_000_000
+MAX_CACHE_TEXT_CHARS = MAX_TEXT_CHARS
+MAX_CACHE_BYTES = 50 * 1024 * 1024
+MAX_CACHE_ENTRIES = 2_000
+CACHE_FILENAME = "corpus-cache.json"
+CACHE_SCHEMA_VERSION = "last30days-corpus-cache/v2"
+
+_CACHE_LOCK = threading.Lock()
+
+
+@dataclass
+class CorpusScanResult:
+ """One bounded scan, including non-fatal extraction notes."""
+
+ items: list[schema.SourceItem]
+ notes: list[str] = field(default_factory=list)
+ files_scanned: int = 0
+ cache_hits: int = 0
+
+
+def resolve_directories(
+ cli_directories: Iterable[str] | None,
+ configured: str | Iterable[str] | None,
+) -> list[Path]:
+ """Merge repeatable CLI paths with ``os.pathsep``-separated config paths."""
+ raw: list[str] = [str(value) for value in (cli_directories or []) if str(value).strip()]
+ if isinstance(configured, str):
+ raw.extend(value for value in configured.split(os.pathsep) if value.strip())
+ elif configured:
+ raw.extend(str(value) for value in configured if str(value).strip())
+
+ resolved: list[Path] = []
+ seen: set[str] = set()
+ for value in raw:
+ path = Path(value.strip()).expanduser().resolve()
+ key = os.path.normcase(str(path))
+ if key in seen:
+ continue
+ seen.add(key)
+ resolved.append(path)
+ return resolved
+
+
+def _safe_error(exc: BaseException) -> str:
+ """Describe an error without str(exc), which embeds absolute paths.
+
+ These notes travel into source_status detail and render in coverage
+ diagnostics outside the private corpus block.
+ """
+ reason = getattr(exc, "strerror", None)
+ return str(reason) if reason else exc.__class__.__name__
+
+
+def search(
+ topic: str,
+ directories: Iterable[Path | str],
+ *,
+ from_date: str,
+ to_date: str,
+ all_time: bool = False,
+ limit: int = 12,
+ cache_dir: Path | None = None,
+) -> CorpusScanResult:
+ """Search registered directories without making any network calls."""
+ roots = resolve_directories([str(path) for path in directories], None)
+ notes: list[str] = []
+ cache_path = cache_dir / CACHE_FILENAME if cache_dir is not None else None
+ with _CACHE_LOCK:
+ cache = _load_cache(cache_path)
+ cache_entries = cache.setdefault("entries", {})
+ cache_entry_sizes = {
+ path: _cache_entry_fragment_size(path, value)
+ for path, value in cache_entries.items()
+ }
+
+ candidates: list[tuple[float, int, schema.SourceItem]] = []
+ seen_files: set[str] = set()
+ files_scanned = 0
+ cache_hits = 0
+ pdf_available = which("pdftotext")
+ pdf_unavailable_noted = False
+
+ readable_roots: list[Path] = []
+ for root in roots:
+ if not root.is_dir():
+ notes.append(f"Skipped corpus root '{Path(root).name}': not a readable directory")
+ continue
+ readable_roots.append(root)
+
+ per_root_limit, extra_slots = divmod(MAX_FILES, len(readable_roots) or 1)
+ scan_limit_reached = False
+ for root_index, root in enumerate(readable_roots):
+ root_limit = per_root_limit + (1 if root_index < extra_slots else 0)
+ root_files_scanned = 0
+ for path in _iter_files(root, notes=notes):
+ if root_files_scanned >= root_limit:
+ scan_limit_reached = True
+ break
+ key = os.path.normcase(str(path))
+ if key in seen_files:
+ continue
+ seen_files.add(key)
+ root_files_scanned += 1
+ files_scanned += 1
+
+ try:
+ stat = path.stat()
+ except OSError as exc:
+ notes.append(f"Skipped {_display_path(path, root)}: {_safe_error(exc)}")
+ continue
+ published_at = datetime.fromtimestamp(
+ stat.st_mtime, tz=timezone.utc
+ ).date().isoformat()
+ if not all_time and not (from_date <= published_at <= to_date):
+ continue
+
+ cached = cache_entries.get(str(path))
+ if (
+ isinstance(cached, dict)
+ and cached.get("mtime_ns") == stat.st_mtime_ns
+ and cached.get("size") == stat.st_size
+ and isinstance(cached.get("text"), str)
+ ):
+ text = cached["text"]
+ cache_hits += 1
+ else:
+ if path.suffix.lower() == ".pdf" and not pdf_available:
+ if not pdf_unavailable_noted:
+ notes.append("Skipped PDF files because pdftotext is not on PATH")
+ pdf_unavailable_noted = True
+ continue
+ try:
+ text = _extract_text(path, pdftotext=pdf_available)
+ except (OSError, subprocess.SubprocessError) as exc:
+ notes.append(f"Skipped {_display_path(path, root)}: {_safe_error(exc)}")
+ continue
+ _cache_entry_put(cache_entries, cache_entry_sizes, str(path), {
+ "mtime_ns": stat.st_mtime_ns,
+ "size": stat.st_size,
+ "text": text[:MAX_CACHE_TEXT_CHARS],
+ })
+
+ title = _path_title(path)
+ score = _match_score(topic, f"{title}\n{text}")
+ if score < 0.15:
+ continue
+ relative_path = str(path.relative_to(root))
+ path_digest = hashlib.sha256(str(path).encode("utf-8")).hexdigest()
+ item = schema.SourceItem(
+ item_id=f"C{path_digest[:12]}",
+ source=SOURCE,
+ title=title,
+ body=text,
+ url=f"corpus://{path_digest}",
+ container=str(path.parent),
+ published_at=published_at,
+ date_confidence="high",
+ relevance_hint=score,
+ why_relevant=f"Matched local file {relative_path}",
+ # Leave empty so extract_best_snippet derives the matching
+ # window; a file-prefix snippet is preserved verbatim and can
+ # show unrelated intro text (and draw entity-miss demotion).
+ snippet="",
+ metadata={
+ "path": str(path),
+ "relative_path": relative_path,
+ "extension": path.suffix.lower(),
+ "local_only": True,
+ },
+ )
+ candidates.append((score, stat.st_mtime_ns, item))
+ if scan_limit_reached:
+ notes.append(f"Stopped after the {MAX_FILES}-file corpus scan limit")
+
+ cache["schema_version"] = CACHE_SCHEMA_VERSION
+ cache["entries"] = _bounded_entries(cache_entries)
+ with _CACHE_LOCK:
+ _write_cache(cache_path, cache, notes)
+
+ candidates.sort(key=lambda row: (-row[0], -row[1], row[2].title.casefold()))
+ items = [item for _score, _mtime, item in candidates[: max(0, limit)]]
+ log.source_log(
+ "Corpus",
+ f"scanned {files_scanned} file(s), {cache_hits} cache hit(s), {len(items)} match(es)",
+ tty_only=False,
+ )
+ return CorpusScanResult(
+ items=items,
+ notes=notes,
+ files_scanned=files_scanned,
+ cache_hits=cache_hits,
+ )
+
+
+def _display_path(path: Path | str, root: Path | None = None) -> str:
+ """Render a note-safe path: never the absolute local path.
+
+ Corpus notes flow into source_status detail and the Partial Coverage
+ block, which render OUTSIDE the private corpus markers - an absolute
+ path like /home/user/private/notes/foo.md must not escape there.
+ """
+ candidate = Path(path)
+ if root is not None:
+ try:
+ return str(Path(root).name / candidate.relative_to(root))
+ except ValueError:
+ pass
+ return candidate.name
+
+
+def _iter_files(root: Path, notes: list[str] | None = None) -> Iterable[Path]:
+ # Bounded newest-first selection: keep only the newest MAX_FILES paths in a
+ # heap while walking, so registering a huge tree does not materialize every
+ # path before the caller's extraction cap applies.
+ import heapq
+
+ heap: list[tuple[int, str]] = []
+ walk_errors = 0
+
+ def _on_walk_error(error: OSError) -> None:
+ nonlocal walk_errors
+ walk_errors += 1
+ if notes is not None and walk_errors <= 3:
+ unreadable = _display_path(error.filename, root) if error.filename else Path(root).name
+ notes.append(f"corpus: could not read {unreadable}: {error.strerror}")
+
+ for current, directory_names, file_names in os.walk(
+ root, followlinks=False, onerror=_on_walk_error
+ ):
+ directory_names[:] = sorted(
+ name
+ for name in directory_names
+ if name not in IGNORED_DIRECTORIES and not name.startswith(".")
+ )
+ current_path = Path(current)
+ for name in sorted(file_names):
+ if name.startswith("."):
+ continue
+ path = current_path / name
+ if path.suffix.lower() in SUPPORTED_SUFFIXES and not path.is_symlink():
+ entry = (_safe_mtime_ns(path), str(path))
+ if len(heap) < MAX_FILES:
+ heapq.heappush(heap, entry)
+ else:
+ heapq.heappushpop(heap, entry)
+ if notes is not None and walk_errors > 3:
+ notes.append(f"corpus: {walk_errors - 3} more unreadable directories suppressed")
+ ordered = sorted(heap, key=lambda item: (-item[0], item[1].casefold()))
+ for _mtime, raw_path in ordered:
+ yield Path(raw_path)
+
+
+def _safe_mtime_ns(path: Path) -> int:
+ try:
+ return path.stat().st_mtime_ns
+ except OSError:
+ return 0
+
+
+def _extract_text(path: Path, *, pdftotext: str | None) -> str:
+ if path.suffix.lower() == ".pdf":
+ if not pdftotext:
+ return ""
+ completed = subprocess.run(
+ [pdftotext, str(path), "-"],
+ capture_output=True,
+ check=True,
+ text=True,
+ timeout=20,
+ )
+ return completed.stdout[:MAX_TEXT_CHARS]
+ with path.open("r", encoding="utf-8", errors="replace") as handle:
+ return handle.read(MAX_TEXT_CHARS)
+
+
+def _path_title(path: Path) -> str:
+ title = path.stem.replace("_", " ").replace("-", " ")
+ return " ".join(title.split()) or path.name
+
+
+def _match_score(topic: str, text: str) -> float:
+ lexical = relevance.token_overlap_relevance(topic, text)
+ topic_entities = entity_extract.extract_text_entities(topic)
+ text_entities = entity_extract.extract_text_entities(text)
+ entity_score = entity_extract.entity_overlap(topic_entities, text_entities)
+ return round(max(lexical, entity_score * 0.9), 4)
+
+
+def _load_cache(path: Path | None) -> dict[str, Any]:
+ if path is None:
+ return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}
+ try:
+ if path.stat().st_size > MAX_CACHE_BYTES:
+ return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, UnicodeError, json.JSONDecodeError):
+ return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}
+ if not isinstance(payload, dict) or payload.get("schema_version") != CACHE_SCHEMA_VERSION:
+ return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}
+ if not isinstance(payload.get("entries"), dict):
+ payload["entries"] = {}
+ payload["entries"] = _bounded_entries(payload["entries"])
+ return payload
+
+
+def _bounded_entries(entries: Any) -> dict[str, Any]:
+ if not isinstance(entries, dict):
+ return {}
+ ordered = sorted(
+ (
+ (path, value)
+ for path, value in entries.items()
+ if (
+ isinstance(path, str)
+ and isinstance(value, dict)
+ and isinstance(value.get("text"), str)
+ )
+ ),
+ key=lambda row: int(row[1].get("mtime_ns") or 0),
+ reverse=True,
+ )
+ base_bytes = len(
+ json.dumps(
+ {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}},
+ ensure_ascii=False,
+ ).encode("utf-8")
+ )
+ used_bytes = base_bytes
+ bounded: dict[str, Any] = {}
+ for path, value in ordered[:MAX_CACHE_ENTRIES]:
+ normalized = {
+ "mtime_ns": value.get("mtime_ns"),
+ "size": value.get("size"),
+ "text": value["text"][:MAX_CACHE_TEXT_CHARS],
+ }
+ fragment = json.dumps({path: normalized}, ensure_ascii=False).encode("utf-8")
+ fragment_bytes = len(fragment) - 2 + (2 if bounded else 0)
+ if used_bytes + fragment_bytes > MAX_CACHE_BYTES:
+ continue
+ bounded[path] = normalized
+ used_bytes += fragment_bytes
+ return bounded
+
+
+def _cache_entry_fragment_size(path: str, value: dict[str, Any]) -> int:
+ return len(json.dumps({path: value}, ensure_ascii=False).encode("utf-8")) - 2
+
+
+def _cache_entry_put(
+ entries: dict[str, Any],
+ sizes: dict[str, int],
+ path: str,
+ value: dict[str, Any],
+) -> None:
+ entries[path] = value
+ sizes[path] = _cache_entry_fragment_size(path, value)
+ while (
+ len(entries) > MAX_CACHE_ENTRIES
+ or _cache_payload_size(sizes) > MAX_CACHE_BYTES
+ ):
+ oldest = min(
+ entries,
+ key=lambda candidate: (
+ int(entries[candidate].get("mtime_ns") or 0),
+ candidate,
+ ),
+ )
+ del entries[oldest]
+ del sizes[oldest]
+
+
+def _cache_payload_size(sizes: dict[str, int]) -> int:
+ base_bytes = len(
+ json.dumps(
+ {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}},
+ ensure_ascii=False,
+ ).encode("utf-8")
+ )
+ separators = max(0, len(sizes) - 1) * 2
+ return base_bytes + sum(sizes.values()) + separators
+
+
+def _write_cache(path: Path | None, payload: dict[str, Any], notes: list[str]) -> None:
+ if path is None:
+ return
+ try:
+ _ensure_private_directory(path.parent)
+ payload["entries"] = _bounded_entries(payload.get("entries", {}))
+ encoded = json.dumps(payload, ensure_ascii=False).encode("utf-8")
+ temporary = path.with_name(f".{path.name}.{os.getpid()}.tmp")
+ try:
+ fd = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+ except FileExistsError:
+ temporary.unlink()
+ fd = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+ with os.fdopen(fd, "wb") as handle:
+ handle.write(encoded)
+ temporary.replace(path)
+ path.chmod(0o600)
+ except OSError as exc:
+ notes.append(f"Corpus cache unavailable: {_safe_error(exc)}")
+
+
+def _ensure_private_directory(path: Path) -> None:
+ missing: list[Path] = []
+ current = path
+ while not current.exists():
+ missing.append(current)
+ current = current.parent
+ path.mkdir(parents=True, exist_ok=True, mode=0o700)
+ for directory in missing:
+ directory.chmod(0o700)
diff --git a/skills/last30days/scripts/lib/env.py b/skills/last30days/scripts/lib/env.py
index fe95b6db..19ce59e6 100644
--- a/skills/last30days/scripts/lib/env.py
+++ b/skills/last30days/scripts/lib/env.py
@@ -470,6 +470,12 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:
# degraded (neither ok, no-results, nor skipped-unconfigured). #384.
('LAST30DAYS_STRICT_EXIT', None),
('LAST30DAYS_MEMORY_DIR', None),
+ # Optional local-only evidence source. Paths are separated with the
+ # platform path separator (":" on macOS/Linux, ";" on Windows).
+ ('LAST30DAYS_CORPUS_DIRS', None),
+ # Corpus evidence is omitted from the stable agent JSON export unless
+ # this explicit privacy opt-in is truthy.
+ ('LAST30DAYS_CORPUS_IN_EXPORT', None),
('LAST30DAYS_LIBRARY_OWNER', None),
('LAST30DAYS_LIBRARY_CONTEXT', 'on'),
('LAST30DAYS_PUBLISH_PASSWORD', None),
diff --git a/skills/last30days/scripts/lib/html_render.py b/skills/last30days/scripts/lib/html_render.py
index db2ac127..ea599032 100644
--- a/skills/last30days/scripts/lib/html_render.py
+++ b/skills/last30days/scripts/lib/html_render.py
@@ -438,10 +438,12 @@ def render_html_comparison(
LIBRARY_BRIEF_MARKER = "<!-- generated by last30days library feed -->"
-def render_library_brief(entry: LibraryEntry) -> str:
+def render_library_brief(entry: LibraryEntry, *, include_private: bool = True) -> str:
"""Render a scanned Markdown or JSON briefing as a safe standalone page."""
md = _strip_invitation(entry.content)
md = _strip_canonical_boundary(md)
+ if not include_private:
+ md = _strip_private_corpus(md)
body = _markdown_to_html(md)
body = _wrap_engine_footer(body)
colophon = (
@@ -458,6 +460,18 @@ def render_library_brief(entry: LibraryEntry) -> str:
return rendered.replace("</body>", f"{LIBRARY_BRIEF_MARKER}\n</body>", 1)
+_PRIVATE_CORPUS_BLOCK = re.compile(
+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_START -->.*?"
+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_END -->\s*",
+ re.DOTALL,
+)
+
+
+def _strip_private_corpus(markdown: str) -> str:
+ """Remove the renderer-marked local corpus section before publication."""
+ return _PRIVATE_CORPUS_BLOCK.sub("", markdown)
+
+
def render_library_index(
entries: Sequence[LibraryEntry],
*,
diff --git a/skills/last30days/scripts/lib/library.py b/skills/last30days/scripts/lib/library.py
index 38bd2348..96a581fc 100644
--- a/skills/last30days/scripts/lib/library.py
+++ b/skills/last30days/scripts/lib/library.py
@@ -29,6 +29,11 @@ _LIBRARY_ID = re.compile(r"[0-9a-f]{32}")
_GENERATED_BRIEF_NAME = re.compile(
r"[a-z0-9]+(?:-[a-z0-9]+)*-[0-9a-f]{8}-\d{4}-\d{2}-\d{2}\.html"
)
+_PRIVATE_CORPUS_BLOCK = re.compile(
+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_START -->.*?"
+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_END -->\s*",
+ re.DOTALL,
+)
@dataclass(frozen=True, slots=True)
@@ -140,15 +145,16 @@ def _keep_preferred(entries: dict[str, LibraryEntry], entry: LibraryEntry) -> No
def _parse_markdown(path: Path) -> LibraryEntry:
content = path.read_text(encoding="utf-8")
- title_match = _REPORT_TITLE.search(content) or _FIRST_TITLE.search(content)
+ public_content = _PRIVATE_CORPUS_BLOCK.sub("", content)
+ title_match = _REPORT_TITLE.search(public_content) or _FIRST_TITLE.search(public_content)
if not title_match:
raise ValueError("no Markdown title found")
topic = _clean_inline(title_match.group(1))
if not topic:
raise ValueError("empty Markdown title")
- published_date = _markdown_date(content, path)
- headline = _markdown_headline(content) or topic
- summary = _markdown_summary(content) or headline
+ published_date = _markdown_date(public_content, path)
+ headline = _markdown_headline(public_content) or topic
+ summary = _markdown_summary(public_content) or headline
return LibraryEntry(
slug=slugify(topic),
topic=topic,
diff --git a/skills/last30days/scripts/lib/library_index.py b/skills/last30days/scripts/lib/library_index.py
index 30b57d66..31b6f0d8 100644
--- a/skills/last30days/scripts/lib/library_index.py
+++ b/skills/last30days/scripts/lib/library_index.py
@@ -3,6 +3,7 @@
from __future__ import annotations
import hashlib
+import os
import re
import sqlite3
from dataclasses import dataclass, replace
@@ -14,6 +15,7 @@ from . import library
DEFAULT_LIBRARY_DB = library.DEFAULT_BRIEFS_DIR.parent / "library.db"
DEFAULT_STORE_DB = library.DEFAULT_BRIEFS_DIR.parent / "research.db"
+INDEX_FINGERPRINT_VERSION = "last30days-library-index/v2"
LIBRARY_CONTEXT_START = "<!-- last30days:library-context:start -->"
LIBRARY_CONTEXT_END = "<!-- last30days:library-context:end -->"
_TOKEN = re.compile(r"[^\W_]+", re.UNICODE)
@@ -26,6 +28,11 @@ _LEGACY_LIBRARY_CONTEXT = re.compile(
r"^## From your library\s*$.*?(?=^##\s|\Z)",
re.MULTILINE | re.DOTALL,
)
+_PRIVATE_CORPUS_BLOCK = re.compile(
+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_START -->.*?"
+ r"<!-- LAST30DAYS_PRIVATE_CORPUS_END -->\s*",
+ re.DOTALL,
+)
class LibrarySearchUnavailable(RuntimeError):
@@ -131,7 +138,7 @@ def index_brief(
if entry is None:
return False
target = Path(db_path).expanduser()
- target.parent.mkdir(parents=True, exist_ok=True)
+ _ensure_private_directory(target.parent)
with _connect(target) as conn:
_upsert_entry(conn, entry)
conn.commit()
@@ -209,7 +216,7 @@ def _sync_library(
db_path: Path,
) -> SyncResult:
entries, notes = library.scan_library(memory_dir, briefs_dir)
- db_path.parent.mkdir(parents=True, exist_ok=True)
+ _ensure_private_directory(db_path.parent)
indexed = unchanged = 0
with _connect(db_path) as conn:
existing = {
@@ -250,6 +257,15 @@ def _sync_library(
def _connect(path: Path) -> sqlite3.Connection:
+ _ensure_private_directory(path.parent)
+ if not path.exists():
+ try:
+ fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+ except FileExistsError:
+ pass
+ else:
+ os.close(fd)
+ path.chmod(0o600)
conn = sqlite3.connect(str(path))
try:
conn.row_factory = sqlite3.Row
@@ -268,7 +284,13 @@ def _upsert_entry(
fingerprint: str | None = None,
) -> None:
stat = entry.source_path.stat()
- indexed_content = _indexable_content(entry.content)
+ private_free_content = _PRIVATE_CORPUS_BLOCK.sub("", entry.content)
+ indexed_content = _indexable_content(private_free_content)
+ headline = entry.headline
+ summary = entry.summary
+ if private_free_content != entry.content and entry.source_format == "markdown":
+ headline = library._markdown_headline(private_free_content) or entry.topic
+ summary = library._markdown_summary(private_free_content) or headline
content_hash = fingerprint or _fingerprint(indexed_content)
source_path = str(entry.source_path.resolve())
replaced = conn.execute(
@@ -302,14 +324,14 @@ def _upsert_entry(
content_hash,
entry.topic,
entry.published_date.isoformat(),
- entry.headline,
- entry.summary,
+ headline,
+ summary,
entry.source_format,
),
)
conn.execute(
"INSERT INTO library_fts(entry_id, topic, headline, summary, content) VALUES (?, ?, ?, ?, ?)",
- (entry.entry_id, entry.topic, entry.headline, entry.summary, indexed_content),
+ (entry.entry_id, entry.topic, headline, summary, indexed_content),
)
@@ -334,6 +356,7 @@ def _search_store_sightings(
JOIN research_runs rr ON rr.id = fs.run_id
JOIN topics t ON t.id = fs.topic_id
WHERE findings_fts MATCH ? AND rr.status = 'completed'
+ AND fs.source != 'corpus'
ORDER BY rank, rr.run_date DESC
LIMIT ?""",
(expression, limit),
@@ -371,7 +394,8 @@ def _fts_expression(query: str) -> str:
def _fingerprint(content: str) -> str:
- return hashlib.sha256(content.encode("utf-8")).hexdigest()
+ payload = f"{INDEX_FINGERPRINT_VERSION}\0{content}"
+ return hashlib.sha256(payload.encode("utf-8")).hexdigest()
def _clean_snippet(value: object) -> str:
@@ -379,10 +403,22 @@ def _clean_snippet(value: object) -> str:
def _indexable_content(content: str) -> str:
- without_marked = _MARKED_LIBRARY_CONTEXT.sub("", content)
+ without_private = _PRIVATE_CORPUS_BLOCK.sub("", content)
+ without_marked = _MARKED_LIBRARY_CONTEXT.sub("", without_private)
return _LEGACY_LIBRARY_CONTEXT.sub("", without_marked)
+def _ensure_private_directory(path: Path) -> None:
+ missing: list[Path] = []
+ current = path
+ while not current.exists():
+ missing.append(current)
+ current = current.parent
+ path.mkdir(parents=True, exist_ok=True, mode=0o700)
+ for directory in missing:
+ directory.chmod(0o700)
+
+
def _is_confirmed_corruption(exc: sqlite3.DatabaseError) -> bool:
message = str(exc).casefold()
return any(
diff --git a/skills/last30days/scripts/lib/pipeline.py b/skills/last30days/scripts/lib/pipeline.py
index b76a936a..0ea8e6b1 100644
--- a/skills/last30days/scripts/lib/pipeline.py
+++ b/skills/last30days/scripts/lib/pipeline.py
@@ -19,6 +19,7 @@ from . import (
arxiv,
bird_x,
bluesky,
+ corpus,
dates,
dedupe,
digg,
@@ -125,6 +126,7 @@ MOCK_AVAILABLE_SOURCES = [
"trustpilot",
"jobs",
"linkedin",
+ "corpus",
]
@@ -156,6 +158,10 @@ def available_sources(
available: list[str] = []
# reddit_public needs no API key - always available
available.append("reddit")
+ if corpus.resolve_directories(
+ config.get("_CORPUS_DIRS"), config.get("LAST30DAYS_CORPUS_DIRS")
+ ):
+ available.append("corpus")
if config.get("SCRAPECREATORS_API_KEY"):
available.extend(["tiktok", "instagram"])
if env.get_x_source(config, local_only=local_only):
@@ -889,10 +895,25 @@ def run(
hiring_signals_mode: bool = False,
internal_subrun: bool = False,
save_dir: Path | str | None = None,
+ corpus_dirs: list[str] | None = None,
+ corpus_all_time: bool = False,
) -> schema.Report:
settings = DEPTH_SETTINGS[depth]
requested_sources = normalize_requested_sources(requested_sources)
from_date, to_date = dates.get_date_range(lookback_days, as_of_date=as_of_date)
+ resolved_corpus_dirs = corpus.resolve_directories(
+ corpus_dirs or config.get("_CORPUS_DIRS"),
+ config.get("LAST30DAYS_CORPUS_DIRS"),
+ )
+ excluded_sources = {
+ source.strip().lower()
+ for source in str(config.get("EXCLUDE_SOURCES") or "").split(",")
+ if source.strip()
+ }
+ corpus_enabled = bool(resolved_corpus_dirs) and "corpus" not in excluded_sources
+ corpus_requested = bool(requested_sources and "corpus" in requested_sources)
+ if corpus_enabled and requested_sources and "corpus" not in requested_sources:
+ requested_sources = [*requested_sources, "corpus"]
# Gate StockTwits to ticker/crypto topics. Single chokepoint: when False,
# available_sources() never registers stocktwits, so the planner can't
@@ -903,6 +924,10 @@ def run(
runtime = providers.mock_runtime(config, depth)
reasoning_provider = None
available = list(requested_sources or MOCK_AVAILABLE_SOURCES)
+ if corpus_enabled and "corpus" not in available:
+ available.append("corpus")
+ if not corpus_enabled and not corpus_requested:
+ available = [source for source in available if source != "corpus"]
if not requested_sources and not hiring_signals_mode and not _company_topic_likely(topic):
available = [source for source in available if source != "jobs"]
else:
@@ -910,6 +935,11 @@ def run(
available = available_sources(config, requested_sources)
if requested_sources:
available = [source for source in available if source in requested_sources]
+ # Keep an explicitly requested but unconfigured corpus in the plan long
+ # enough to record its skipped-unconfigured source outcome. It is never
+ # submitted to the network executor below.
+ if corpus_requested and "corpus" not in excluded_sources and "corpus" not in available:
+ available.append("corpus")
if web_backend == "none":
available = [s for s in available if s != "grounding"]
elif web_backend in ("brave", "exa", "serper", "parallel", "keyless") and "grounding" not in available:
@@ -970,6 +1000,15 @@ def run(
# Drill plans re-fetch only the sources that contributed to the matched
# cluster; the company-topic jobs injection must not widen that set.
_ensure_jobs_in_plan(plan, available, explicit=hiring_signals_mode, topic=topic)
+ if "corpus" in available and plan.subqueries:
+ # Corpus is deterministic and user-registered, so it always gets one
+ # bounded stream even when a quick/LLM plan omits it. Reuse the primary
+ # subquery instead of multiplying local scans across every subquery.
+ if "corpus" not in plan.subqueries[0].sources:
+ plan.subqueries[0].sources.append("corpus")
+ if "corpus" not in plan.source_weights:
+ plan.source_weights["corpus"] = 1.0
+ plan.source_weights = planner._normalize_weights(plan.source_weights)
# Always-on planner trace. Emits one summary line plus one per subquery
# so retrieval-breadth failures like the 2026-04-19 Hermes Agent Use Cases
@@ -1001,10 +1040,18 @@ def run(
"Source was requested but is not configured for this run.",
attempted=False,
)
+ if corpus_requested and not corpus_enabled:
+ bundle.record_failure(
+ "corpus",
+ schema.SKIPPED_UNCONFIGURED,
+ "Corpus was requested but no readable directory was configured.",
+ attempted=False,
+ )
# Expose plan_source to the renderer so render_compact can emit the
# DEGRADED RUN banner when a named-entity topic was invoked bare
# (source=deterministic AND no pre-research flags). LAW 7 backstop.
bundle.artifacts["plan_source"] = plan_source
+ bundle.artifacts["corpus_in_export"] = bool(config.get("_CORPUS_IN_EXPORT"))
# Hiring-signals is deliberately jobs-only with no multi-source --plan, so
# the LAW 7 degraded-run and Step 0.55 pre-research banners do not apply -
# they would contradict the documented jobs-scoped flow. Suppress them.
@@ -1070,6 +1117,53 @@ def run(
rate_limited_sources: set[str] = set()
rate_limit_lock = threading.Lock()
+ # Local corpus retrieval is intentionally outside the network executor and
+ # retry budget. One bounded stream participates in the same signal scoring,
+ # fusion, reranking, and per-source result cap as remote sources.
+ if corpus_enabled and plan.subqueries:
+ primary = plan.subqueries[0]
+ bundle.mark_attempted("corpus")
+ result = corpus.search(
+ topic,
+ resolved_corpus_dirs,
+ from_date=from_date,
+ to_date=to_date,
+ all_time=corpus_all_time,
+ limit=settings["per_stream_limit"],
+ cache_dir=env.CONFIG_DIR,
+ )
+ prepared_query = relevance.PreparedQuery(primary.ranking_query)
+ lookback_window_days = (
+ datetime.strptime(to_date, "%Y-%m-%d").date()
+ - datetime.strptime(from_date, "%Y-%m-%d").date()
+ ).days
+ corpus_items = signals.annotate_stream(
+ result.items,
+ prepared_query,
+ plan.freshness_mode,
+ reference_date=to_date,
+ max_days=lookback_window_days,
+ )
+ corpus_items = signals.prune_low_relevance(corpus_items)
+ corpus_items = dedupe.dedupe_items(corpus_items)
+ for item in corpus_items:
+ item.snippet = snippet.extract_best_snippet(item, prepared_query)
+ bundle.add_items(primary.label, "corpus", corpus_items)
+ if result.notes:
+ outcome = bundle.source_status["corpus"]
+ bundle.source_status["corpus"] = schema.SourceOutcome(
+ source="corpus",
+ state=outcome.state,
+ items_returned=outcome.items_returned,
+ attempted=True,
+ detail="; ".join(result.notes),
+ )
+ bundle.artifacts["corpus"] = {
+ "files_scanned": result.files_scanned,
+ "cache_hits": result.cache_hits,
+ "all_time": corpus_all_time,
+ }
+
futures = {}
# Per-source fetch budget prevents redundant API calls
source_fetch_count: dict[str, int] = {}
@@ -1077,7 +1171,7 @@ def run(
1
for subquery in plan.subqueries
for source in subquery.sources
- if source in available
+ if source in available and source != "corpus"
)
max_workers = _inner_max_workers(stream_count, internal_subrun=internal_subrun)
with ThreadPoolExecutor(max_workers=max_workers) as executor:
@@ -1085,6 +1179,8 @@ def run(
for source in subquery.sources:
if source not in available:
continue
+ if source == "corpus":
+ continue
# Skip GitHub keyword search if person-mode already ran
if source == "github" and (_github_person_done or _github_custom_done):
continue
@@ -1206,7 +1302,9 @@ def run(
# Phase 2b: retry thin sources with simplified query
# Note: _github_skip_sources tells the retry to not re-run GitHub keyword search
# when project-mode or person-mode already provided authoritative data.
- _github_skip_retry = {"github"} if (_github_person_done or _github_custom_done) else set()
+ _github_skip_retry = {"corpus"}
+ if _github_person_done or _github_custom_done:
+ _github_skip_retry.add("github")
_retry_thin_sources(
topic=topic,
bundle=bundle,
@@ -1258,21 +1356,58 @@ def run(
for h in ([x_handle, github_user, *(x_related or [])])
if h and h.strip()
}
- ranked_candidates = rerank.rerank_candidates(
+ private_candidates = [
+ candidate
+ for candidate in candidates
+ if candidate.source == "corpus"
+ or any(item.source == "corpus" for item in candidate.source_items)
+ ]
+ private_candidate_ids = {id(candidate) for candidate in private_candidates}
+ public_candidates = [
+ candidate for candidate in candidates if id(candidate) not in private_candidate_ids
+ ]
+ ranked_public = rerank.rerank_candidates(
topic=topic,
plan=plan,
- candidates=candidates,
+ candidates=public_candidates,
provider=None if mock else reasoning_provider,
model=None if mock else runtime.rerank_model,
shortlist_size=settings["rerank_limit"],
resolved_handles=resolved_handles,
)
+ # Corpus titles/snippets must never enter a hosted reasoning prompt. Score
+ # every candidate carrying corpus evidence with the deterministic fallback,
+ # even when the rest of the run uses a remote reranker.
+ ranked_private = rerank.rerank_candidates(
+ topic=topic,
+ plan=plan,
+ candidates=private_candidates,
+ provider=None,
+ model=None,
+ shortlist_size=settings["rerank_limit"],
+ resolved_handles=resolved_handles,
+ )
+ ranked_candidates = sorted(
+ [*ranked_public, *ranked_private],
+ key=lambda candidate: (
+ -candidate.final_score,
+ -(candidate.engagement or -1),
+ min(candidate.native_ranks.values(), default=999),
+ candidate.title,
+ ),
+ )
rerank.score_fun(
topic=topic,
- candidates=ranked_candidates,
+ candidates=ranked_public,
provider=None if mock else reasoning_provider,
model=None if mock else runtime.rerank_model,
)
+ rerank.score_fun(
+ topic=topic,
+ candidates=ranked_private,
+ provider=None,
+ model=None,
+ )
# Phase 3: post-rerank GitHub star enrichment. Record/replay-aware so the
# eval harness stays fully offline: this path calls the GitHub API (and the
diff --git a/skills/last30days/scripts/lib/planner.py b/skills/last30days/scripts/lib/planner.py
index 3ef9764b..7c258fd0 100644
--- a/skills/last30days/scripts/lib/planner.py
+++ b/skills/last30days/scripts/lib/planner.py
@@ -131,6 +131,7 @@ SOURCE_CAPABILITIES = {
"grounding": {"web", "reference", "link"},
"perplexity": {"web", "reference", "analysis"},
"jobs": {"jobs", "company_signal", "link"},
+ "corpus": {"reference", "analysis"},
}
DEFAULT_INTENT_CAPABILITIES = {
"comparison": {"discussion", "video", "web", "reference", "social", "link", "market"},
diff --git a/skills/last30days/scripts/lib/render.py b/skills/last30days/scripts/lib/render.py
index 01d93fd7..0aab13f5 100644
--- a/skills/last30days/scripts/lib/render.py
+++ b/skills/last30days/scripts/lib/render.py
@@ -131,8 +131,12 @@ SOURCE_LABELS = {
"trustpilot": "Trustpilot",
"perplexity": "Perplexity",
"jobs": "Jobs",
+ "corpus": "Your files",
}
+PRIVATE_CORPUS_START = "<!-- LAST30DAYS_PRIVATE_CORPUS_START -->"
+PRIVATE_CORPUS_END = "<!-- LAST30DAYS_PRIVATE_CORPUS_END -->"
+
# vote_weight = max points a fully on-topic, max-upvoted top comment can add to
# the LLM humor score. Tuned against real runs: typical funny comments score
@@ -277,6 +281,48 @@ def _render_ranked_clusters(
return lines
+def _render_corpus_section(report: schema.Report, limit: int = 8) -> list[str]:
+ """Render private local evidence in one removable, clearly badged block."""
+ candidates = [
+ candidate
+ for candidate in report.ranked_candidates
+ if candidate.source == "corpus"
+ ][:limit]
+ if not candidates:
+ return []
+ lines = [
+ PRIVATE_CORPUS_START,
+ "## From your files",
+ "",
+ "> 🔒 **LOCAL ONLY** - excluded from hosted publishing and agent JSON unless explicitly opted in.",
+ "",
+ ]
+ for candidate in candidates:
+ primary = schema.candidate_primary_item(candidate)
+ path = str((primary.metadata if primary else {}).get("relative_path") or "")
+ published = primary.published_at if primary else None
+ detail = f"modified {published}" if published else "modification date unknown"
+ lines.append(
+ f"- **{_defang_corpus_sentinels(candidate.title)}** "
+ f"({detail}, relevance {candidate.final_score:.0f})"
+ )
+ if path:
+ lines.append(f" - File: `{_defang_corpus_sentinels(path)}`")
+ if candidate.snippet:
+ lines.append(f" - {_defang_corpus_sentinels(_truncate(candidate.snippet, 300))}")
+ lines.append(PRIVATE_CORPUS_END)
+ return lines
+
+
+def _defang_corpus_sentinels(value: str) -> str:
+ """Source content must not be able to terminate the private-block markers.
+
+ A note containing the literal end marker would otherwise close the block
+ early, leaving later corpus snippets in publishable output.
+ """
+ return value.replace("LAST30DAYS_PRIVATE_CORPUS", "LAST30DAYS_PRIVATE-CORPUS")
+
+
_FRESHNESS_PRIORITY = {
"contradicted": 0,
"stale": 1,
@@ -413,6 +459,7 @@ def render_compact(
register: str = "default",
) -> str:
audience = registers.get_register(register)
+ evidence_report = schema.without_sources(report, {"corpus"})
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
lines = [
*_render_badge(),
@@ -461,7 +508,7 @@ def render_compact(
# block below) vs "synthesize from" (this block).
lines.append("<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. Transform into `What I learned:` prose per LAW 2. -->")
lines.append("")
- hiring_block = _render_hiring_signals(report)
+ hiring_block = _render_hiring_signals(evidence_report)
if hiring_block and audience.name in {"default", "eli5"}:
lines.extend(hiring_block)
lines.append("")
@@ -469,11 +516,11 @@ def render_compact(
if audience.name in {"default", "eli5"}:
# Keep this legacy assembly byte-for-byte stable. ELI5 has always been
# a synthesis-only voice change, so it intentionally takes this path.
- lines.extend(_render_ranked_clusters(report, report.clusters[:cluster_limit]))
- lines.extend(_render_stats(report))
+ lines.extend(_render_ranked_clusters(evidence_report, evidence_report.clusters[:cluster_limit]))
+ lines.extend(_render_stats(evidence_report))
best_takes = _render_best_takes(
- report.ranked_candidates,
+ evidence_report.ranked_candidates,
limit=fun_params["limit"],
threshold=fun_params["threshold"],
vote_weight=fun_params.get("vote_weight", 18.0),
@@ -481,7 +528,7 @@ def render_compact(
if best_takes:
lines.extend([""] + best_takes)
- top_comments = _render_top_comments(report)
+ top_comments = _render_top_comments(evidence_report)
if top_comments:
lines.extend([""] + top_comments)
@@ -491,7 +538,10 @@ def render_compact(
lines.extend(_render_source_coverage(report))
else:
- lines.extend(_render_registered_sections(report, audience, fun_params, cluster_limit))
+ lines.extend(_render_registered_sections(evidence_report, audience, fun_params, cluster_limit))
+ corpus_section = _render_corpus_section(report)
+ if corpus_section:
+ lines.extend(["", *corpus_section])
# Close EVIDENCE FOR SYNTHESIS envelope before anything that passes through verbatim.
lines.append("")
lines.append("<!-- END EVIDENCE FOR SYNTHESIS -->")
@@ -545,6 +595,7 @@ def render_for_html(
sections so direct HTML output reflects the selected audience preset.
"""
audience = registers.get_register(register)
+ evidence_report = schema.without_sources(report, {"corpus"})
lines = [
*_render_badge(),
*_render_html_metadata(report),
@@ -552,7 +603,7 @@ def render_for_html(
drill_context = _render_drill_context(report)
if drill_context:
lines.extend(["", *drill_context])
- hiring_block = _render_hiring_signals(report)
+ hiring_block = _render_hiring_signals(evidence_report)
if synthesis_md:
lines.extend(["", synthesis_md.strip()])
if hiring_block and "## Hiring Signals" not in synthesis_md:
@@ -564,13 +615,16 @@ def render_for_html(
lines.extend([
"",
*_render_registered_sections(
- report,
+ evidence_report,
audience,
fun_params,
8,
include_source_diagnostics=False,
),
])
+ corpus_section = _render_corpus_section(report)
+ if corpus_section:
+ lines.extend(["", *corpus_section])
freshness_verdicts = _render_freshness_verdicts(report)
if freshness_verdicts:
lines.extend(["", *freshness_verdicts])
@@ -613,6 +667,9 @@ def render_for_html_comparison(
freshness_verdicts = _render_freshness_verdicts(report)
if freshness_verdicts:
lines.extend(["", f"## {label}", "", *freshness_verdicts])
+ corpus_section = _render_corpus_section(report)
+ if corpus_section:
+ lines.extend(["", f"## {label}", "", *corpus_section])
# Comparison data quality notes also go to stderr, not into the artifact.
_append_html_footer(lines, main_report, save_path)
return "\n".join(lines).strip() + "\n"
@@ -1100,17 +1157,22 @@ def _render_entity_evidence_block(
fun_params: dict,
) -> list[str]:
"""Render one entity's clusters and best-takes inside the evidence envelope."""
- candidate_by_id = {c.candidate_id: c for c in report.ranked_candidates}
+ evidence_report = schema.without_sources(report, {"corpus"})
+ candidate_by_id = {c.candidate_id: c for c in evidence_report.ranked_candidates}
out: list[str] = [f"## {label}", ""]
- if not report.clusters:
+ if not evidence_report.clusters:
out.append("(no significant discussion this month)")
out.append("")
+ corpus_section = _render_corpus_section(report)
+ if corpus_section:
+ out.extend(corpus_section)
+ out.append("")
return out
out.append("### Ranked Evidence Clusters")
out.append("")
- for index, cluster in enumerate(report.clusters[:cluster_limit], start=1):
+ for index, cluster in enumerate(evidence_report.clusters[:cluster_limit], start=1):
out.append(
f"#### {index}. {cluster.title} "
f"(score {cluster.score:.0f}, {len(cluster.candidate_ids)} item"
@@ -1123,11 +1185,11 @@ def _render_entity_evidence_block(
candidate = candidate_by_id.get(candidate_id)
if not candidate:
continue
- out.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=report))
+ out.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=evidence_report))
out.append("")
best_takes = _render_best_takes(
- report.ranked_candidates,
+ evidence_report.ranked_candidates,
limit=fun_params["limit"],
threshold=fun_params["threshold"],
vote_weight=fun_params.get("vote_weight", 18.0),
@@ -1136,6 +1198,11 @@ def _render_entity_evidence_block(
out.extend(best_takes)
out.append("")
+ corpus_section = _render_corpus_section(report)
+ if corpus_section:
+ out.extend(corpus_section)
+ out.append("")
+
return out
@@ -1159,22 +1226,27 @@ def render_comparison_multi_context(
lines.extend(resolved_block)
lines.append("")
for label, report in entity_reports:
+ evidence_report = schema.without_sources(report, {"corpus"})
lines.append(f"## {label}")
lines.append(f"Intent: {report.query_plan.intent}")
- if not report.clusters:
+ if not evidence_report.clusters:
lines.append("- (no significant discussion this month)")
else:
- for cluster in report.clusters[:cluster_limit]:
+ for cluster in evidence_report.clusters[:cluster_limit]:
lines.append(
f"- {cluster.title} "
f"[{', '.join(_source_label(s) for s in cluster.sources)}]"
)
+ corpus_section = _render_corpus_section(report)
+ if corpus_section:
+ lines.extend(["", *corpus_section])
lines.append("")
return "\n".join(lines).strip() + "\n"
def render_full(report: schema.Report) -> str:
"""Full data dump: ALL clusters + ALL items by source. For saved files and debugging."""
+ evidence_report = schema.without_sources(report, {"corpus"})
# Start with the same header as compact
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
lines = [
@@ -1209,8 +1281,8 @@ def render_full(report: schema.Report) -> str:
# ALL clusters (no limit)
lines.append("## Ranked Evidence Clusters")
lines.append("")
- candidate_by_id = {c.candidate_id: c for c in report.ranked_candidates}
- for index, cluster in enumerate(report.clusters, start=1):
+ candidate_by_id = {c.candidate_id: c for c in evidence_report.ranked_candidates}
+ for index, cluster in enumerate(evidence_report.clusters, start=1):
lines.append(
f"### {index}. {cluster.title} "
f"(score {cluster.score:.0f}, {len(cluster.candidate_ids)} item{'s' if len(cluster.candidate_ids) != 1 else ''}, "
@@ -1222,12 +1294,12 @@ def render_full(report: schema.Report) -> str:
candidate = candidate_by_id.get(cid)
if not candidate:
continue
- lines.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=report))
+ lines.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=evidence_report))
lines.append("")
fun_params = _FUN_LEVELS["medium"]
best_takes = _render_best_takes(
- report.ranked_candidates,
+ evidence_report.ranked_candidates,
limit=fun_params["limit"],
threshold=fun_params["threshold"],
vote_weight=fun_params["vote_weight"],
@@ -1242,7 +1314,7 @@ def render_full(report: schema.Report) -> str:
source_order = ["reddit", "x", "youtube", "tiktok", "instagram", "threads", "pinterest",
"hackernews", "bluesky", "truthsocial", "polymarket", "grounding", "xiaohongshu", "github", "digg", "perplexity", "jobs"]
for source in source_order:
- items = report.items_by_source.get(source, [])
+ items = evidence_report.items_by_source.get(source, [])
if not items:
continue
lines.append(f"### {_source_label(source)} ({len(items)} items)")
@@ -1308,12 +1380,17 @@ def render_full(report: schema.Report) -> str:
lines.append(f" Closes: {end_date}")
lines.append("")
- freshness_verdicts = _render_freshness_verdicts(report)
+ corpus_section = _render_corpus_section(report)
+ if corpus_section:
+ lines.extend(corpus_section)
+ lines.append("")
+
+ freshness_verdicts = _render_freshness_verdicts(evidence_report)
if freshness_verdicts:
lines.extend(freshness_verdicts)
lines.append("")
- lines.extend(_render_stats(report))
- lines.extend(_render_source_coverage(report))
+ lines.extend(_render_stats(evidence_report))
+ lines.extend(_render_source_coverage(evidence_report))
return "\n".join(lines).strip() + "\n"
@@ -1332,7 +1409,8 @@ def _format_item_engagement(item: schema.SourceItem) -> str:
def render_context(report: schema.Report, cluster_limit: int = 6) -> str:
- candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
+ evidence_report = schema.without_sources(report, {"corpus"})
+ candidate_by_id = {candidate.candidate_id: candidate for candidate in evidence_report.ranked_candidates}
lines = [
f"Topic: {report.topic}",
f"Intent: {report.query_plan.intent}",
@@ -1351,7 +1429,7 @@ def render_context(report: schema.Report, cluster_limit: int = 6) -> str:
if hiring_block:
lines.extend(["", *hiring_block, ""])
lines.append("Top clusters:")
- for cluster in report.clusters[:cluster_limit]:
+ for cluster in evidence_report.clusters[:cluster_limit]:
lines.append(f"- {cluster.title} [{', '.join(_source_label(source) for source in cluster.sources)}]")
for candidate_id in cluster.representative_ids[:2]:
candidate = candidate_by_id.get(candidate_id)
@@ -1366,6 +1444,9 @@ def render_context(report: schema.Report, cluster_limit: int = 6) -> str:
lines.append(f" - {' | '.join(detail_parts)}")
if candidate.snippet:
lines.append(f" Evidence: {_truncate(candidate.snippet, 180)}")
+ corpus_section = _render_corpus_section(report)
+ if corpus_section:
+ lines.extend(["", *corpus_section])
if report.warnings:
lines.append("Warnings:")
lines.extend(f"- {warning}" for warning in report.warnings)
@@ -1386,6 +1467,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
Audience Questions, and Source Clusters. Sections 2-4 are omitted when there
is no matching data; Sections 1 and 5 always appear.
"""
+ evidence_report = schema.without_sources(report, {"corpus"})
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
lines = [
f"# Production Brief: {report.topic}",
@@ -1404,8 +1486,8 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
lines.append("## Ranked Storylines")
lines.append("")
- candidate_by_id = {c.candidate_id: c for c in report.ranked_candidates}
- for i, cluster in enumerate(report.clusters[:cluster_limit], start=1):
+ candidate_by_id = {c.candidate_id: c for c in evidence_report.ranked_candidates}
+ for i, cluster in enumerate(evidence_report.clusters[:cluster_limit], start=1):
source_tags = ", ".join(_source_label(s) for s in cluster.sources)
qualifier = f" [{cluster.uncertainty.replace('-', ' ')}]" if cluster.uncertainty else ""
lines.append(f"### {i}. {cluster.title} (score {cluster.score:.0f}, {source_tags}){qualifier}")
@@ -1421,7 +1503,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
lines.append("")
hooks = sorted(
- (c for c in report.ranked_candidates if c.fun_score is not None and c.fun_score >= 70),
+ (c for c in evidence_report.ranked_candidates if c.fun_score is not None and c.fun_score >= 70),
key=lambda c: -(c.fun_score or 0),
)
if hooks:
@@ -1449,7 +1531,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
)
lines.append("")
- tensions = [c for c in report.clusters[:cluster_limit] if c.uncertainty]
+ tensions = [c for c in evidence_report.clusters[:cluster_limit] if c.uncertainty]
if tensions:
lines.append("## Topic Tensions")
lines.append("")
@@ -1459,7 +1541,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
lines.append(f"- **{cluster.title}** [{label}]: {source_tags}")
lines.append("")
- questions = _extract_audience_questions(report.ranked_candidates)
+ questions = _extract_audience_questions(evidence_report.ranked_candidates)
if questions:
lines.append("## Audience Questions")
lines.append("")
@@ -1469,11 +1551,16 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
lines.append("## Source Clusters")
lines.append("")
- for cluster in report.clusters[:cluster_limit]:
+ for cluster in evidence_report.clusters[:cluster_limit]:
source_tags = " + ".join(_source_label(s) for s in cluster.sources)
lines.append(f"- **{cluster.title}**: {source_tags}")
lines.append("")
+ corpus_section = _render_corpus_section(report)
+ if corpus_section:
+ lines.extend(corpus_section)
+ lines.append("")
+
freshness_verdicts = _render_freshness_verdicts(report)
if freshness_verdicts:
lines.extend(freshness_verdicts)
@@ -1925,6 +2012,7 @@ _FOOTER_SOURCES: list[tuple[str, str, str, str, list[tuple[str, str]]]] = [
# the LAW 5 footer; without it the footer was dropped entirely.
("jobs", "💼", "Jobs", "role", []),
("perplexity", "🧠", "Perplexity", "result", [("citations", "citations")]),
+ ("corpus", "🔒", "Your files", "file", []),
]
diff --git a/skills/last30days/scripts/lib/schema.py b/skills/last30days/scripts/lib/schema.py
index d3425cd2..759a7e8d 100644
--- a/skills/last30days/scripts/lib/schema.py
+++ b/skills/last30days/scripts/lib/schema.py
@@ -2,6 +2,7 @@
from __future__ import annotations
+import copy
from dataclasses import asdict, dataclass, field, is_dataclass
from datetime import datetime, timezone
from typing import Any, Literal
@@ -558,6 +559,103 @@ def candidate_primary_item(candidate: Candidate) -> SourceItem | None:
AGENT_EXPORT_SCHEMA_VERSION = "1.2"
+
+
+def without_sources(report: Report, excluded_sources: set[str]) -> Report:
+ """Return a deep-copied report with private source evidence removed.
+
+ This is the publication boundary used by agent JSON, hosted HTML, and
+ future outbound surfaces. Cluster titles are rebuilt when a removed item
+ participated so text derived from a private representative cannot survive
+ after its candidate is gone.
+ """
+ excluded = {source.lower() for source in excluded_sources}
+ if not excluded:
+ return copy.deepcopy(report)
+ clean = copy.deepcopy(report)
+ clean.items_by_source = {
+ source: items
+ for source, items in clean.items_by_source.items()
+ if source.lower() not in excluded
+ }
+ clean.errors_by_source = {
+ source: detail
+ for source, detail in clean.errors_by_source.items()
+ if source.lower() not in excluded
+ }
+ clean.source_status = {
+ source: outcome
+ for source, outcome in clean.source_status.items()
+ if source.lower() not in excluded
+ }
+ clean.query_plan.source_weights = {
+ source: weight
+ for source, weight in clean.query_plan.source_weights.items()
+ if source.lower() not in excluded
+ }
+ for subquery in clean.query_plan.subqueries:
+ subquery.sources[:] = [
+ source for source in subquery.sources if source.lower() not in excluded
+ ]
+
+ kept_candidates: list[Candidate] = []
+ removed_candidate_ids: set[str] = set()
+ for candidate in clean.ranked_candidates:
+ if candidate.source.lower() in excluded:
+ removed_candidate_ids.add(candidate.candidate_id)
+ continue
+ candidate.source_items = [
+ item for item in candidate.source_items if item.source.lower() not in excluded
+ ]
+ candidate.sources = [
+ source for source in candidate.sources if source.lower() not in excluded
+ ]
+ candidate.native_ranks = {
+ key: rank
+ for key, rank in candidate.native_ranks.items()
+ if key.rsplit(":", 1)[-1].lower() not in excluded
+ }
+ kept_candidates.append(candidate)
+ clean.ranked_candidates = kept_candidates
+ candidate_by_id = {
+ candidate.candidate_id: candidate for candidate in clean.ranked_candidates
+ }
+
+ kept_clusters: list[Cluster] = []
+ for cluster in clean.clusters:
+ original_ids = list(cluster.candidate_ids)
+ cluster.candidate_ids = [
+ candidate_id for candidate_id in original_ids if candidate_id in candidate_by_id
+ ]
+ if not cluster.candidate_ids:
+ continue
+ cluster.representative_ids = [
+ candidate_id
+ for candidate_id in cluster.representative_ids
+ if candidate_id in candidate_by_id
+ ] or [cluster.candidate_ids[0]]
+ cluster.sources = sorted({
+ source
+ for candidate_id in cluster.candidate_ids
+ for source in candidate_sources(candidate_by_id[candidate_id])
+ if source.lower() not in excluded
+ })
+ if any(candidate_id in removed_candidate_ids for candidate_id in original_ids):
+ cluster.title = candidate_by_id[cluster.representative_ids[0]].title
+ kept_clusters.append(cluster)
+ clean.clusters = kept_clusters
+ clean.freshness_verdicts = [
+ verdict
+ for verdict in clean.freshness_verdicts
+ if verdict.source.lower() not in excluded
+ and verdict.candidate_id in candidate_by_id
+ ]
+ for key in list(clean.artifacts):
+ if any(source in key.lower() for source in excluded):
+ del clean.artifacts[key]
+ return clean
+
+
DISCOVERY_EXPORT_SCHEMA_VERSION = "1.0"
@@ -628,8 +726,20 @@ def _agent_generated_at(value: str) -> str:
return parsed.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")
-def to_agent_export(report: Report) -> dict[str, Any]:
- """Serialize a report to the stable, versioned agent JSON contract."""
+def to_agent_export(
+ report: Report,
+ *,
+ corpus_in_export: bool | None = None,
+) -> dict[str, Any]:
+ """Serialize a report to the stable, versioned agent JSON contract.
+
+ Local corpus evidence is private by default. Callers must opt in explicitly
+ either with ``corpus_in_export=True`` or the CLI-populated report artifact.
+ """
+ if corpus_in_export is None:
+ corpus_in_export = bool(report.artifacts.get("corpus_in_export"))
+ if not corpus_in_export:
+ report = without_sources(report, {"corpus"})
candidates = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
cluster_by_candidate: dict[str, int] = {}
cluster_by_id: dict[str, int] = {}
diff --git a/skills/last30days/scripts/lib/signals.py b/skills/last30days/scripts/lib/signals.py
index d49d689a..08bfe1d0 100644
--- a/skills/last30days/scripts/lib/signals.py
+++ b/skills/last30days/scripts/lib/signals.py
@@ -24,6 +24,7 @@ SOURCE_QUALITY = {
"instagram": 0.58,
"tiktok": 0.58,
"jobs": 0.72,
+ "corpus": 0.75,
}
diff --git a/skills/last30days/scripts/store.py b/skills/last30days/scripts/store.py
index a02c928e..233289d3 100644
--- a/skills/last30days/scripts/store.py
+++ b/skills/last30days/scripts/store.py
@@ -12,6 +12,7 @@ Database location: ~/.local/share/last30days/research.db
import argparse
import json
+import os
import sqlite3
import sys
from contextlib import contextmanager
@@ -55,6 +56,25 @@ def scoped_db(db_path: Optional[Path]) -> Iterator[None]:
_db_override = previous
+def ensure_private_db_files(db_path: Optional[Path] = None) -> Path:
+ """Create/harden the research database and SQLite sidecars owner-only."""
+ path = db_path or _get_db_path()
+ path.parent.mkdir(parents=True, exist_ok=True)
+ if not path.exists():
+ try:
+ fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+ except FileExistsError:
+ pass
+ else:
+ os.close(fd)
+ for candidate in (path, Path(f"{path}-wal"), Path(f"{path}-shm")):
+ try:
+ candidate.chmod(0o600)
+ except FileNotFoundError:
+ pass
+ return path
+
+
SCHEMA_V1 = """
PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
diff --git a/tests/test_corpus_source.py b/tests/test_corpus_source.py
new file mode 100644
index 00000000..c13fb18f
--- /dev/null
+++ b/tests/test_corpus_source.py
@@ -0,0 +1,868 @@
+from __future__ import annotations
+
+import hashlib
+import io
+import json
+import os
+import sqlite3
+import sys
+from contextlib import redirect_stderr, redirect_stdout
+from datetime import datetime, timezone
+from pathlib import Path
+from unittest import mock
+
+import last30days as cli
+import store
+from lib import corpus, env, health, html_render, library, library_index, pipeline, render, schema
+
+
+def _set_mtime(path: Path, value: str) -> None:
+ timestamp = datetime.fromisoformat(value).replace(tzinfo=timezone.utc).timestamp()
+ os.utime(path, (timestamp, timestamp))
+
+
+def _scan(tmp_path: Path, *, all_time: bool = False, limit: int = 12):
+ return corpus.search(
+ "MCP servers",
+ [tmp_path],
+ from_date="2026-06-10",
+ to_date="2026-07-10",
+ all_time=all_time,
+ limit=limit,
+ cache_dir=tmp_path / "cache",
+ )
+
+
+def test_scans_matching_text_and_markdown_with_path_titles(tmp_path):
+ note = tmp_path / "mcp-server-notes.md"
+ note.write_text("MCP servers expose tools to local coding agents.", encoding="utf-8")
+ other = tmp_path / "groceries.txt"
+ other.write_text("milk eggs bread", encoding="utf-8")
+ _set_mtime(note, "2026-07-05T12:00:00")
+ _set_mtime(other, "2026-07-05T12:00:00")
+
+ result = _scan(tmp_path)
+
+ assert [item.title for item in result.items] == ["mcp server notes"]
+ assert result.items[0].source == "corpus"
+ assert result.items[0].published_at == "2026-07-05"
+ assert result.items[0].metadata["local_only"] is True
+ assert result.items[0].url.startswith("corpus://")
+ assert str(note) not in result.items[0].url
+
+
+def test_multilingual_matching_reuses_shared_cjk_tokenizer(tmp_path):
+ note = tmp_path / "模型记录.md"
+ note.write_text("国产大模型的最新测评和部署记录", encoding="utf-8")
+ _set_mtime(note, "2026-07-05T12:00:00")
+
+ result = corpus.search(
+ "国产大模型 测评",
+ [tmp_path],
+ from_date="2026-06-10",
+ to_date="2026-07-10",
+ cache_dir=tmp_path / "cache",
+ )
+
+ assert [item.title for item in result.items] == ["模型记录"]
+
+
+def test_recency_window_and_all_time_override(tmp_path):
+ old = tmp_path / "old-mcp-plan.md"
+ old.write_text("MCP servers and local tool protocols", encoding="utf-8")
+ _set_mtime(old, "2025-01-01T00:00:00")
+
+ assert _scan(tmp_path).items == []
+ assert [item.title for item in _scan(tmp_path, all_time=True).items] == ["old mcp plan"]
+
+
+def test_hidden_git_and_node_modules_directories_are_ignored(tmp_path):
+ visible = tmp_path / "visible.md"
+ visible.write_text("MCP servers are visible", encoding="utf-8")
+ _set_mtime(visible, "2026-07-05T00:00:00")
+ for directory in (tmp_path / ".git", tmp_path / ".hidden", tmp_path / "node_modules"):
+ directory.mkdir()
+ path = directory / "private.md"
+ path.write_text("MCP servers hidden secret", encoding="utf-8")
+ _set_mtime(path, "2026-07-05T00:00:00")
+
+ result = _scan(tmp_path)
+
+ assert [item.metadata["relative_path"] for item in result.items] == ["visible.md"]
+
+
+def test_pdf_is_skipped_with_one_note_when_pdftotext_is_absent(tmp_path, monkeypatch):
+ pdf = tmp_path / "mcp.pdf"
+ pdf.write_bytes(b"not a real pdf")
+ _set_mtime(pdf, "2026-07-05T00:00:00")
+ monkeypatch.setattr(corpus, "which", lambda _name: None)
+
+ result = _scan(tmp_path)
+
+ assert result.items == []
+ assert result.notes == ["Skipped PDF files because pdftotext is not on PATH"]
+
+
+def test_mtime_cache_reuses_text_and_is_private(tmp_path, monkeypatch):
+ note = tmp_path / "mcp.md"
+ note.write_text("MCP servers cache this local note", encoding="utf-8")
+ _set_mtime(note, "2026-07-05T00:00:00")
+ first = _scan(tmp_path)
+ assert first.cache_hits == 0
+
+ monkeypatch.setattr(corpus, "_extract_text", mock.Mock(side_effect=AssertionError("cache miss")))
+ second = _scan(tmp_path)
+
+ assert second.cache_hits == 1
+ cache_path = tmp_path / "cache" / corpus.CACHE_FILENAME
+ assert cache_path.stat().st_mode & 0o777 == 0o600
+ assert cache_path.parent.stat().st_mode & 0o777 == 0o700
+
+
+def test_cache_for_500_large_documents_is_bounded_by_total_bytes(tmp_path, monkeypatch):
+ for index in range(500):
+ (tmp_path / f"document-{index:03d}.txt").touch()
+
+ monkeypatch.setattr(corpus, "MAX_CACHE_BYTES", 1_000_000)
+ monkeypatch.setattr(corpus, "MAX_CACHE_TEXT_CHARS", 100_000)
+ monkeypatch.setattr(corpus, "_extract_text", lambda *_args, **_kwargs: "x" * 1_000_000)
+ monkeypatch.setattr(corpus, "_match_score", lambda *_args, **_kwargs: 0.0)
+
+ result = corpus.search(
+ "anything",
+ [tmp_path],
+ from_date="2026-06-10",
+ to_date="2026-07-10",
+ all_time=True,
+ limit=0,
+ cache_dir=tmp_path / "cache",
+ )
+
+ cache_path = tmp_path / "cache" / corpus.CACHE_FILENAME
+ payload = json.loads(cache_path.read_text(encoding="utf-8"))
+ assert result.files_scanned == 500
+ assert cache_path.stat().st_size <= corpus.MAX_CACHE_BYTES
+ assert len(payload["entries"]) < 500
+ assert all(
+ len(entry["text"]) <= corpus.MAX_CACHE_TEXT_CHARS
+ for entry in payload["entries"].values()
+ )
+
+
+def test_result_budget_caps_one_corpus_stream(tmp_path):
+ for index in range(10):
+ path = tmp_path / f"mcp-{index}.md"
+ path.write_text(f"MCP servers local note {index}", encoding="utf-8")
+ _set_mtime(path, f"2026-07-{index + 1:02d}T00:00:00")
+
+ assert len(_scan(tmp_path, limit=3).items) == 3
+
+
+def test_file_cap_is_shared_fairly_across_corpus_roots(tmp_path, monkeypatch):
+ archive = tmp_path / "archive"
+ relevant = tmp_path / "relevant"
+ archive.mkdir()
+ relevant.mkdir()
+ for index in range(4):
+ (archive / f"mcp-archive-{index}.md").write_text(
+ f"MCP servers archived note {index}", encoding="utf-8"
+ )
+ later = relevant / "mcp-current.md"
+ later.write_text("MCP servers relevant current note", encoding="utf-8")
+ monkeypatch.setattr(corpus, "MAX_FILES", 4)
+
+ result = corpus.search(
+ "MCP servers",
+ [archive, relevant],
+ from_date="2026-06-10",
+ to_date="2026-07-10",
+ all_time=True,
+ cache_dir=tmp_path / "cache",
+ )
+
+ assert result.files_scanned <= corpus.MAX_FILES
+ assert any(item.metadata["path"].startswith(str(archive)) for item in result.items)
+ assert any(item.metadata["path"] == str(later) for item in result.items)
+
+
+def test_pipeline_runs_corpus_outside_network_executor(tmp_path, monkeypatch):
+ note = tmp_path / "mcp-private.md"
+ note.write_text("MCP servers use a secret local transport", encoding="utf-8")
+ _set_mtime(note, "2026-07-05T00:00:00")
+ retrieve = mock.Mock(return_value=([], {}))
+ monkeypatch.setattr(pipeline, "_retrieve_stream", retrieve)
+
+ report = pipeline.run(
+ topic="MCP servers",
+ config={"_CORPUS_DIRS": [str(tmp_path)], "EXCLUDE_SOURCES": ""},
+ depth="quick",
+ requested_sources=["corpus"],
+ mock=True,
+ as_of_date="2026-07-10",
+ external_plan={
+ "intent": "concept",
+ "freshness_mode": "balanced_recent",
+ "cluster_mode": "none",
+ "source_weights": {"corpus": 1.0},
+ "subqueries": [{
+ "label": "primary",
+ "search_query": "MCP servers",
+ "ranking_query": "MCP servers",
+ "sources": ["corpus"],
+ }],
+ },
+ )
+
+ assert report.source_status["corpus"].state == health.OK
+ assert len(report.items_by_source["corpus"]) == 1
+ assert len(report.items_by_source["corpus"]) <= pipeline.DEPTH_SETTINGS["quick"]["per_stream_limit"]
+ assert all(call.kwargs["source"] != "corpus" for call in retrieve.call_args_list)
+
+
+def test_corpus_never_enters_remote_rerank_or_fun_prompts(tmp_path, monkeypatch):
+ note = tmp_path / "private.md"
+ note.write_text("Model context protocol servers PRIVATE-RERANK-SENTINEL", encoding="utf-8")
+ _set_mtime(note, "2026-07-05T00:00:00")
+ remote = mock.Mock()
+ remote.generate_json = mock.Mock(side_effect=AssertionError("private prompt left machine"))
+ runtime = schema.ProviderRuntime("local", "remote-planner", "remote-reranker")
+ monkeypatch.setattr(pipeline.providers, "resolve_runtime", lambda *_args, **_kwargs: (runtime, remote))
+ monkeypatch.setattr(pipeline, "available_sources", lambda *_args, **_kwargs: ["corpus"])
+
+ report = pipeline.run(
+ topic="how do model context protocol servers work today?",
+ config={"_CORPUS_DIRS": [str(tmp_path)], "EXCLUDE_SOURCES": ""},
+ depth="quick",
+ requested_sources=["corpus"],
+ mock=False,
+ as_of_date="2026-07-10",
+ external_plan={
+ "intent": "how_to",
+ "freshness_mode": "evergreen_ok",
+ "cluster_mode": "workflow",
+ "source_weights": {"corpus": 1.0},
+ "subqueries": [{
+ "label": "primary",
+ "search_query": "model context protocol servers",
+ "ranking_query": "How do model context protocol servers work?",
+ "sources": ["corpus"],
+ }],
+ },
+ )
+
+ assert report.items_by_source["corpus"]
+ remote.generate_json.assert_not_called()
+
+
+def test_explicit_unconfigured_corpus_records_skipped_outcome():
+ report = pipeline.run(
+ topic="how do local model context protocol servers work?",
+ config={"EXCLUDE_SOURCES": ""},
+ depth="quick",
+ requested_sources=["corpus"],
+ mock=True,
+ as_of_date="2026-07-10",
+ external_plan={
+ "intent": "how_to",
+ "freshness_mode": "evergreen_ok",
+ "cluster_mode": "workflow",
+ "source_weights": {"corpus": 1.0},
+ "subqueries": [{
+ "label": "primary",
+ "search_query": "model context protocol servers",
+ "ranking_query": "How do model context protocol servers work?",
+ "sources": ["corpus"],
+ }],
+ },
+ )
+
+ outcome = report.source_status["corpus"]
+ assert outcome.state == schema.SKIPPED_UNCONFIGURED
+ assert outcome.attempted is False
+
+
+def _privacy_report(secret: str = "PRIVATE-CORPUS-SENTINEL") -> schema.Report:
+ item = schema.SourceItem(
+ item_id="C-private",
+ source="corpus",
+ title="private mcp notes",
+ body=f"MCP servers {secret}",
+ url="",
+ published_at="2026-07-05",
+ snippet=f"MCP servers {secret}",
+ metadata={"relative_path": "notes/private.md", "local_only": True},
+ )
+ candidate = schema.Candidate(
+ candidate_id="corpus:C-private",
+ item_id=item.item_id,
+ source="corpus",
+ title=item.title,
+ url="",
+ snippet=item.snippet,
+ subquery_labels=["primary"],
+ native_ranks={"primary:corpus": 1},
+ local_relevance=1.0,
+ freshness=90,
+ engagement=None,
+ source_quality=0.75,
+ rrf_score=0.01,
+ final_score=91,
+ cluster_id="cluster-private",
+ sources=["corpus"],
+ source_items=[item],
+ )
+ return schema.Report(
+ topic="MCP servers",
+ range_from="2026-06-10",
+ range_to="2026-07-10",
+ generated_at="2026-07-10T00:00:00+00:00",
+ provider_runtime=schema.ProviderRuntime("local", "mock", "mock"),
+ query_plan=schema.QueryPlan(
+ intent="concept",
+ freshness_mode="balanced_recent",
+ cluster_mode="none",
+ raw_topic="MCP servers",
+ subqueries=[schema.SubQuery("primary", "MCP servers", "MCP servers", ["corpus"])],
+ source_weights={"corpus": 1.0},
+ ),
+ clusters=[schema.Cluster(
+ cluster_id="cluster-private",
+ title=f"Private cluster {secret}",
+ candidate_ids=[candidate.candidate_id],
+ representative_ids=[candidate.candidate_id],
+ sources=["corpus"],
+ score=91,
+ )],
+ ranked_candidates=[candidate],
+ items_by_source={"corpus": [item]},
+ errors_by_source={},
+ source_status={"corpus": schema.SourceOutcome("corpus", health.OK, 1)},
+ )
+
+
+def test_corpus_findings_persist_with_stable_opaque_keys(tmp_path, monkeypatch):
+ note = tmp_path / "private-customer-notes.md"
+ note.write_text("MCP servers persist this private finding", encoding="utf-8")
+ first = corpus.search(
+ "MCP servers",
+ [tmp_path],
+ from_date="2026-06-10",
+ to_date="2026-07-10",
+ all_time=True,
+ cache_dir=tmp_path / "cache",
+ ).items[0]
+ second = corpus.search(
+ "MCP servers",
+ [tmp_path],
+ from_date="2026-06-10",
+ to_date="2026-07-10",
+ all_time=True,
+ cache_dir=tmp_path / "cache",
+ ).items[0]
+ report = _privacy_report()
+ report.items_by_source["corpus"][0].url = first.url
+ report.ranked_candidates[0].url = first.url
+
+ db_path = tmp_path / "research.db"
+ monkeypatch.setattr(store, "_db_override", db_path)
+ store.init_db()
+ topic = store.add_topic(report.topic)
+ run_id = store.record_run(topic["id"], status="completed")
+ findings = store.findings_from_report(report)
+ counts = store.store_findings(run_id, topic["id"], findings)
+
+ assert first.url == second.url
+ assert first.url.startswith("corpus://")
+ assert str(note) not in first.url
+ assert counts == {"new": 1, "updated": 0}
+ with sqlite3.connect(db_path) as conn:
+ persisted = conn.execute(
+ "SELECT source, source_url FROM findings"
+ ).fetchone()
+ assert persisted == ("corpus", first.url)
+ assert library_index.search(
+ "private finding",
+ db_path=tmp_path / "missing-library.db",
+ store_db_path=db_path,
+ ) == []
+
+
+def test_persist_report_hardens_corpus_store_and_sidecars(tmp_path, monkeypatch):
+ db_path = tmp_path / "store" / "research.db"
+ monkeypatch.setattr(store, "_db_override", db_path)
+ original_findings_from_report = store.findings_from_report
+ original_store_findings = store.store_findings
+ original_update_run = store.update_run
+
+ def make_permissive():
+ db_path.chmod(0o644)
+ for suffix in ("-wal", "-shm"):
+ sidecar = Path(f"{db_path}{suffix}")
+ sidecar.touch(mode=0o644)
+ sidecar.chmod(0o644)
+
+ def findings_from_report_with_permissive_store(*args, **kwargs):
+ findings = original_findings_from_report(*args, **kwargs)
+ make_permissive()
+ return findings
+
+ def store_findings_after_private_check(*args, **kwargs):
+ for artifact in (db_path, Path(f"{db_path}-wal"), Path(f"{db_path}-shm")):
+ assert artifact.stat().st_mode & 0o777 == 0o600
+ return original_store_findings(*args, **kwargs)
+
+ def update_run_with_permissive_sidecars(*args, **kwargs):
+ result = original_update_run(*args, **kwargs)
+ make_permissive()
+ return result
+
+ monkeypatch.setattr(store, "findings_from_report", findings_from_report_with_permissive_store)
+ monkeypatch.setattr(store, "store_findings", store_findings_after_private_check)
+ monkeypatch.setattr(store, "update_run", update_run_with_permissive_sidecars)
+
+ cli.persist_report(_privacy_report())
+
+ for artifact in (db_path, Path(f"{db_path}-wal"), Path(f"{db_path}-shm")):
+ assert artifact.stat().st_mode & 0o777 == 0o600
+
+
+def test_agent_export_excludes_corpus_by_default_and_allows_explicit_opt_in():
+ report = _privacy_report()
+
+ private_default = schema.to_agent_export(report)
+ opted_in = schema.to_agent_export(report, corpus_in_export=True)
+
+ assert private_default["results"] == []
+ assert private_default["clusters"] == []
+ assert "corpus" not in private_default["source_status"]
+ assert opted_in["results"][0]["source"] == "corpus"
+ assert "PRIVATE-CORPUS-SENTINEL" in opted_in["results"][0]["summary"]
+
+ report.artifacts["corpus_in_export"] = True
+ assert schema.to_agent_export(report)["results"][0]["source"] == "corpus"
+
+
+def test_agent_export_rebuilds_mixed_cluster_title_after_private_representative_removed():
+ report = _privacy_report()
+ social_item = schema.SourceItem(
+ item_id="R-public",
+ source="reddit",
+ title="Public MCP discussion",
+ body="Public evidence",
+ url="https://reddit.example/public",
+ published_at="2026-07-06",
+ snippet="Public evidence",
+ )
+ social = schema.Candidate(
+ candidate_id="reddit:public",
+ item_id=social_item.item_id,
+ source="reddit",
+ title=social_item.title,
+ url=social_item.url,
+ snippet=social_item.snippet,
+ subquery_labels=["primary"],
+ native_ranks={"primary:reddit": 1},
+ local_relevance=0.8,
+ freshness=90,
+ engagement=1,
+ source_quality=0.6,
+ rrf_score=0.01,
+ final_score=80,
+ cluster_id="cluster-private",
+ sources=["reddit"],
+ source_items=[social_item],
+ )
+ report.ranked_candidates.append(social)
+ report.items_by_source["reddit"] = [social_item]
+ report.source_status["reddit"] = schema.SourceOutcome("reddit", health.OK, 1)
+ report.clusters[0].candidate_ids.append(social.candidate_id)
+ report.clusters[0].representative_ids.append(social.candidate_id)
+ report.clusters[0].sources.append("reddit")
+
+ exported = schema.to_agent_export(report)
+
+ assert exported["clusters"][0]["title"] == "Public MCP discussion"
+ assert "PRIVATE-CORPUS-SENTINEL" not in str(exported)
+
+
+def test_local_report_has_badged_from_your_files_section():
+ rendered = render.render_compact(_privacy_report())
+
+ assert "## From your files" in rendered
+ assert "LOCAL ONLY" in rendered
+ assert "PRIVATE-CORPUS-SENTINEL" in rendered
+
+
+def test_publish_html_sends_sanitized_report_not_local_corpus(monkeypatch):
+ report = _privacy_report()
+ captured: dict[str, str] = {}
+
+ def publish(rendered, **_kwargs):
+ captured["html"] = rendered
+ return {"url": "https://example.ht-ml.app"}
+
+ monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})
+ monkeypatch.setattr(cli.pipeline, "diagnose", lambda *_args, **_kwargs: {"available_sources": ["corpus"]})
+ monkeypatch.setattr(cli.pipeline, "run", lambda **_kwargs: report)
+ monkeypatch.setattr(cli, "publish_rendered_html", publish)
+ monkeypatch.setenv("LAST30DAYS_SKIP_PREFLIGHT", "1")
+ monkeypatch.setattr(
+ sys,
+ "argv",
+ ["last30days.py", "MCP servers", "--emit=html", "--publish-html"],
+ )
+
+ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
+ assert cli.main() == 0
+
+ assert "PRIVATE-CORPUS-SENTINEL" not in captured["html"]
+ assert "private mcp notes" not in captured["html"]
+
+
+def test_configured_corpus_bypasses_hosted_backend(tmp_path, monkeypatch):
+ report = _privacy_report()
+ hosted = mock.Mock(side_effect=AssertionError("local corpus was forwarded"))
+ monkeypatch.setattr("lib.hosted.run_hosted", hosted)
+ monkeypatch.setattr(
+ cli.env,
+ "get_config",
+ lambda **_kwargs: {"LAST30DAYS_CORPUS_DIRS": str(tmp_path)},
+ )
+ monkeypatch.setattr(cli.pipeline, "diagnose", lambda *_args, **_kwargs: {"available_sources": ["corpus"]})
+ monkeypatch.setattr(cli.pipeline, "run", lambda **_kwargs: report)
+ monkeypatch.setenv("LAST30DAYS_API_KEY", "test-hosted-key")
+ monkeypatch.setenv("LAST30DAYS_API_BASE", "https://example.invalid")
+ monkeypatch.setenv("LAST30DAYS_SKIP_PREFLIGHT", "1")
+ monkeypatch.setattr(sys, "argv", ["last30days.py", "MCP servers", "--emit=compact"])
+
+ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
+ assert cli.main() == 0
+
+ hosted.assert_not_called()
+
+
+def test_library_publish_strips_marked_corpus_but_local_page_keeps_it(tmp_path, monkeypatch):
+ markdown = render.render_full(_privacy_report())
+ (tmp_path / "mcp-raw.md").write_text(markdown, encoding="utf-8")
+ monkeypatch.setattr(library, "DEFAULT_BRIEFS_DIR", tmp_path / "no-briefings")
+ monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})
+ publish_many = mock.Mock(return_value={})
+ monkeypatch.setattr("lib.html_publish.publish_html_documents", publish_many)
+ monkeypatch.setattr("lib.html_publish.publish_html", mock.Mock(return_value={"url": "https://library.ht-ml.app"}))
+ monkeypatch.setattr(
+ sys,
+ "argv",
+ ["last30days.py", "library", "feed", "--save-dir", str(tmp_path), "--publish"],
+ )
+
+ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
+ assert cli.main() == 0
+
+ published_documents = publish_many.call_args.args[0]
+ assert published_documents
+ assert all("PRIVATE-CORPUS-SENTINEL" not in html for html in published_documents.values())
+ local_pages = list((tmp_path / "briefs").glob("*.html"))
+ assert local_pages
+ assert "PRIVATE-CORPUS-SENTINEL" in local_pages[0].read_text(encoding="utf-8")
+ assert local_pages[0].stat().st_mode & 0o777 == 0o600
+ assert local_pages[0].parent.stat().st_mode & 0o777 == 0o700
+
+
+def test_private_corpus_cannot_escape_via_library_search_and_feed_publish(tmp_path, monkeypatch):
+ secret = "PRIVATE-CORPUS-ESCAPE-SENTINEL"
+ memory = tmp_path / "memory"
+ report = _privacy_report(secret)
+ with mock.patch.object(library_index, "sync_library"):
+ saved = cli.save_output(report, "md", str(memory))
+
+ db_path = tmp_path / "index" / "library.db"
+ library_index.sync_library(memory, tmp_path / "no-briefings", db_path=db_path)
+ entry = library._parse_markdown(saved)
+ legacy_hash = hashlib.sha256(entry.content.encode("utf-8")).hexdigest()
+ with sqlite3.connect(db_path) as conn:
+ conn.execute(
+ "UPDATE library_documents SET content_hash = ? WHERE entry_id = ?",
+ (legacy_hash, entry.entry_id),
+ )
+ conn.execute("DELETE FROM library_fts WHERE entry_id = ?", (entry.entry_id,))
+ conn.execute(
+ "INSERT INTO library_fts(entry_id, topic, headline, summary, content) "
+ "VALUES (?, ?, ?, ?, ?)",
+ (entry.entry_id, entry.topic, entry.headline, entry.summary, entry.content),
+ )
+ conn.commit()
+
+ rebuilt = library_index.sync_library(
+ memory,
+ tmp_path / "no-briefings",
+ db_path=db_path,
+ )
+ assert rebuilt.indexed == 1
+ assert library_index.search(
+ secret,
+ db_path=db_path,
+ store_db_path=tmp_path / "missing-store.db",
+ ) == []
+ matches = library_index.search(
+ "MCP servers",
+ db_path=db_path,
+ store_db_path=tmp_path / "missing-store.db",
+ )
+ assert matches
+ assert all(secret not in match.snippet for match in matches)
+
+ followup = schema.without_sources(report, {"corpus"})
+ followup.library_context = [
+ schema.LibraryContext(
+ topic=match.topic,
+ published_date=match.published_date.isoformat(),
+ headline=match.headline,
+ summary=match.snippet,
+ source_kind=match.source_kind,
+ )
+ for match in matches[:1]
+ ]
+ with mock.patch.object(library_index, "sync_library"):
+ cli.save_output(followup, "md", str(memory), suffix="followup")
+
+ monkeypatch.setattr(library, "DEFAULT_BRIEFS_DIR", tmp_path / "no-briefings")
+ monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})
+ publish_many = mock.Mock(return_value={})
+ monkeypatch.setattr("lib.html_publish.publish_html_documents", publish_many)
+ monkeypatch.setattr(
+ "lib.html_publish.publish_html",
+ mock.Mock(return_value={"url": "https://library.ht-ml.app"}),
+ )
+ monkeypatch.setattr(
+ sys,
+ "argv",
+ ["last30days.py", "library", "feed", "--save-dir", str(memory), "--publish"],
+ )
+
+ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
+ assert cli.main() == 0
+
+ published = publish_many.call_args.args[0]
+ assert published
+ assert all(secret not in document for document in published.values())
+
+
+def test_configured_paths_use_platform_separator_and_dedupe(tmp_path):
+ first = tmp_path / "first"
+ second = tmp_path / "second"
+ resolved = corpus.resolve_directories(
+ [str(first)], f"{first}{os.pathsep}{second}"
+ )
+ assert resolved == [first.resolve(), second.resolve()]
+
+
+def test_corpus_env_keys_are_registered(monkeypatch):
+ monkeypatch.setenv("LAST30DAYS_CORPUS_DIRS", "/tmp/notes:/tmp/transcripts")
+ monkeypatch.setenv("LAST30DAYS_CORPUS_IN_EXPORT", "1")
+ with mock.patch.object(env, "_load_keychain", return_value={}), mock.patch.object(
+ env, "_load_pass", return_value={}
+ ):
+ config = env.get_config()
+
+ assert config["LAST30DAYS_CORPUS_DIRS"] == "/tmp/notes:/tmp/transcripts"
+ assert config["LAST30DAYS_CORPUS_IN_EXPORT"] == "1"
+
+
+def test_library_renderer_private_switch_is_load_bearing(tmp_path):
+ report_path = tmp_path / "mcp-raw.md"
+ report_path.write_text(render.render_full(_privacy_report()), encoding="utf-8")
+ entry = library._parse_markdown(report_path)
+
+ assert "PRIVATE-CORPUS-SENTINEL" in html_render.render_library_brief(entry)
+ assert "PRIVATE-CORPUS-SENTINEL" not in html_render.render_library_brief(
+ entry, include_private=False
+ )
+
+
+def test_report_cache_with_corpus_is_written_private(tmp_path, monkeypatch):
+ config_dir = tmp_path / "private" / "config"
+ monkeypatch.setattr(cli.env, "CONFIG_DIR", config_dir)
+
+ assert cli._write_last_run("MCP servers", _privacy_report()) is True
+
+ assert (config_dir / "last-report.json").stat().st_mode & 0o777 == 0o600
+ assert config_dir.stat().st_mode & 0o777 == 0o700
+
+
+def test_every_corpus_bearing_saved_artifact_is_owner_only(tmp_path):
+ report = _privacy_report()
+ markdown_dir = tmp_path / "private" / "markdown"
+ html_dir = tmp_path / "private" / "html"
+ with mock.patch.object(library_index, "sync_library"):
+ markdown = cli.save_output(report, "md", str(markdown_dir))
+ html = cli.save_output(report, "html", str(html_dir))
+ inferred_private = cli.save_output(
+ report, "compact", str(tmp_path / "private" / "inferred")
+ )
+ forced_public = cli.save_output(
+ report, "json", str(tmp_path / "private" / "forced-public"), private=False
+ )
+ explicit = cli.save_rendered_output(
+ cli.emit_output(report, "html"),
+ str(tmp_path / "private" / "explicit" / "report.html"),
+ private=True,
+ )
+
+ db_path = tmp_path / "private" / "index" / "library.db"
+ library_index.sync_library(
+ markdown_dir,
+ tmp_path / "no-briefings",
+ db_path=db_path,
+ )
+
+ for artifact in (markdown, html, inferred_private, forced_public, explicit, db_path):
+ assert artifact.stat().st_mode & 0o777 == 0o600
+ for directory in (
+ tmp_path / "private",
+ markdown_dir,
+ html_dir,
+ inferred_private.parent,
+ forced_public.parent,
+ explicit.parent,
+ db_path.parent,
+ ):
+ assert directory.stat().st_mode & 0o777 == 0o700
+
+
+def test_cli_saves_every_corpus_bearing_format_owner_only(tmp_path, monkeypatch):
+ report = _privacy_report()
+ monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})
+ monkeypatch.setattr(
+ cli.pipeline,
+ "diagnose",
+ lambda *_args, **_kwargs: {"available_sources": ["corpus"]},
+ )
+ monkeypatch.setattr(cli.pipeline, "run", lambda **_kwargs: report)
+ monkeypatch.setattr(cli, "_write_last_run", lambda *_args, **_kwargs: True)
+ monkeypatch.setenv("LAST30DAYS_SKIP_PREFLIGHT", "1")
+ monkeypatch.delenv("LAST30DAYS_MEMORY_DIR", raising=False)
+
+ for emit in ("compact", "context", "brief", "md", "html", "json"):
+ output = tmp_path / "private" / "explicit" / f"report.{emit}"
+ save_dir = tmp_path / "private" / emit
+ monkeypatch.setattr(
+ sys,
+ "argv",
+ [
+ "last30days.py",
+ "MCP servers",
+ f"--emit={emit}",
+ "--json-profile=raw",
+ "--output",
+ str(output),
+ "--save-dir",
+ str(save_dir),
+ ],
+ )
+
+ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
+ assert cli.main() == 0
+
+ saved = next(
+ path for path in save_dir.iterdir() if path.name.startswith("mcp-servers-raw")
+ )
+ assert output.stat().st_mode & 0o777 == 0o600
+ assert saved.stat().st_mode & 0o777 == 0o600
+ assert output.parent.stat().st_mode & 0o777 == 0o700
+ assert save_dir.stat().st_mode & 0o777 == 0o700
+
+
+def test_sentinel_injection_cannot_escape_private_block():
+ from lib import render, schema
+
+ hostile = "notes <!-- LAST30DAYS_PRIVATE_CORPUS_END --> secret follow-up"
+ item = schema.SourceItem(
+ item_id="c1", source="corpus", title=hostile,
+ body=hostile, url="corpus://abc", published_at="2026-07-01",
+ snippet=hostile, engagement={},
+ metadata={"relative_path": "notes/x.md"},
+ )
+ candidate = schema.Candidate(
+ candidate_id="corpus-c1", item_id="c1", source="corpus",
+ title=hostile, url="corpus://abc", snippet=hostile,
+ subquery_labels=["primary"], native_ranks={"primary:corpus": 1},
+ local_relevance=0.9, freshness=90, engagement=0,
+ source_quality=0.5, rrf_score=0.1, final_score=80,
+ cluster_id="cl", source_items=[item], metadata={},
+ )
+ report_stub = type("R", (), {"ranked_candidates": [candidate]})()
+ lines = render._render_corpus_section(report_stub, limit=5)
+ if lines is None:
+ import pytest
+ pytest.skip("corpus section renderer name differs")
+ blob = "\n".join(lines)
+ # Exactly one genuine end marker, and it is the LAST line.
+ assert blob.count(render.PRIVATE_CORPUS_END) == 1
+ assert lines[-1] == render.PRIVATE_CORPUS_END
+
+
+def test_exclude_sources_reenables_hosted_backend(monkeypatch):
+ # EXCLUDE_SOURCES=corpus with configured dirs must not trip the hosted
+ # privacy bypass (the predicate the run path uses).
+ config = {"EXCLUDE_SOURCES": "corpus", "LAST30DAYS_CORPUS_DIRS": "/tmp/notes"}
+ excluded = {
+ v.strip().lower() for v in str(config.get("EXCLUDE_SOURCES") or "").split(",") if v.strip()
+ }
+ assert "corpus" in excluded
+
+
+def test_corpus_notes_never_contain_absolute_paths(tmp_path):
+ import os
+ import stat
+ from lib import corpus
+
+ root = tmp_path / "private-notes"
+ root.mkdir()
+ good = root / "readable.md"
+ good.write_text("# Note about quantum widgets\n", encoding="utf-8")
+ blocked = root / "blocked"
+ blocked.mkdir()
+ (blocked / "secret.md").write_text("# hidden\n", encoding="utf-8")
+ os.chmod(blocked, 0)
+ try:
+ result = corpus.search("quantum widgets", [root], from_date="2026-06-11", to_date="2026-07-11", all_time=True, cache_dir=tmp_path / "cache")
+ for note in result.notes:
+ assert str(tmp_path) not in note, f"absolute path leaked in note: {note}"
+ finally:
+ os.chmod(blocked, stat.S_IRWXU)
+
+
+def test_scan_error_notes_never_echo_absolute_paths(tmp_path, monkeypatch):
+ note = tmp_path / "mcp-server-notes.md"
+ note.write_text("MCP servers expose tools to local coding agents.", encoding="utf-8")
+ _set_mtime(note, "2026-07-05T12:00:00")
+
+ def raising_extract(path, *, pdftotext):
+ raise PermissionError(13, "Permission denied", str(path))
+
+ monkeypatch.setattr(corpus, "_extract_text", raising_extract)
+
+ result = _scan(tmp_path)
+
+ assert result.items == []
+ assert result.notes, "expected a skip note"
+ joined = " ".join(result.notes)
+ assert str(tmp_path) not in joined
+ assert "Permission denied" in joined
+
+
+def test_cache_write_failure_note_never_echoes_absolute_paths(tmp_path, monkeypatch):
+ notes: list[str] = []
+
+ def raising_open(*args, **kwargs):
+ raise PermissionError(
+ 13, "Permission denied", str(tmp_path / "cache" / "corpus.json")
+ )
+
+ monkeypatch.setattr(corpus.os, "open", raising_open)
+ corpus._write_cache(tmp_path / "cache" / "corpus.json", {"entries": {}}, notes)
+
+ joined = " ".join(notes)
+ assert notes, "expected a cache note"
+ assert str(tmp_path) not in joined
+ assert "Permission denied" in joined