Skip to content

AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary

feat(x): X search over the official X API, plus the Grok Bot X connector (#1124)

Matt Van Hornb15e2eb46a1b
diff --git a/AGENTS.md b/AGENTS.md
index 45eb5749..259e0c37 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -60,7 +60,7 @@ Agents open most PRs. Follow this so `CHANGELOG.md` stops conflicting and versio
- Do not reduce `fail_under` in `pyproject.toml` (`[tool.coverage.report]`) without documenting why in the PR. The coverage gate is a floor meant to rise over time, not to be relaxed when new code is under-tested.
- Every `lib/*.py` call to `log.source_log(...)` must pass `tty_only=False`. The default is `True`, which silently drops every line when stderr isn't a TTY (Claude Code, Codex, CI, captured output) — turning source observability into invisible failure. Enforced by `tests/test_source_log_visibility.py`.
- **CLI-gated optional sources** (Digg via `digg-pp-cli`, YouTube via `yt-dlp`) activate only when `shutil.which` resolves the binary on the **agent subprocess PATH** — not merely when the file exists on disk. First-run setup installs Digg through `@mvanhorn/printing-press-library` (default `$HOME/.local/bin`); Hermes/OpenClaw gateways often need that directory on PATH. Setup must distinguish PATH-visible installs from off-PATH binaries and must not claim "now active" unless the engine gate would pass. See `docs/solutions/integration-issues/digg-cli-agent-path-setup-wizard.md`.
-- **First-run onboarding is consent-driven, model-led, and host-split.** The setup subprocess does only mechanical work (cookie reads, tool installs, GitHub device-auth, and emitting the engine-owned welcome via `--welcome`) — it cannot prompt, so consent lives in `SKILL.md` Step 0. Two flows avoid model-authored prose that Claude Code folds or the model skips: in the **Modal Flow** the welcome pitch is embedded in the setup modal's question (the AskUserQuestion modal is the only always-fully-visible surface — a separate welcome message or `--welcome` Bash run gets buried behind "ctrl+o to expand"); the **Non-Modal Prose Flow** still uses `last30days.py --welcome` (relayed verbatim) since it has no modal. The GitHub device code is surfaced by a two-command split — `setup --github-start` returns the code fast (foreground, copies to clipboard) and `setup --github-poll` waits for authorization (`setup --github` still chains both for back-compat). Step 0 has TWO branches: a **Claude Code Modal Flow** (the restored v3.0.0 `AskUserQuestion`-driven NUX — welcome, Auto/Manual/Skip, cookie consent, ScrapeCreators offer, `INCLUDE_SOURCES` opt-in, first-topic picker) for hosts with modals, and a **Non-Modal Prose Flow** for hosts without (OpenClaw, Codex, Cursor, Gemini CLI, Grok). Both ask before reading cookies, surface the macOS Full Disk Access fix on permission-denied, and offer the ScrapeCreators GitHub signup (10,000 free calls) on every first run. A successful `setup --github` persists `SCRAPECREATORS_API_KEY` automatically (via `setup_wizard.write_api_key`, 0o600) and masks the key in stdout. Do NOT collapse the modal flow back into a bare silent `setup` call or flatten it to prose-only — the guided modals are the feature (they eroded once and were restored). The onboarding contract is locked by `tests/test_onboarding_contract.py`. The Step 5 source opt-in is two tiers, both comment-enabled: **Recommended** (TikTok + Instagram posts AND top comments, plus YouTube comments — `INCLUDE_SOURCES=tiktok,instagram,youtube_comments,tiktok_comments,instagram_comments`) and **Everything** (also Threads + Pinterest). Comments are on by default (posts on → comments on for all three platforms); **Threads and Pinterest are the only opt-in extras**, appearing only in the Step 5 Everything option, never in the welcome or the Step 4 offer. Instagram comments are fetched via ScrapeCreators (`/v2/instagram/post/comments`, ranked by `comment_like_count`) with full vote-weighting parity to YouTube/TikTok (a dedicated `_instagram_engagement` carve-out, the `_VOTE_LOG_REFERENCE`/label/threshold entries). The cross-platform "Top Community Comments" list (`render._render_top_comments`) selects **round-robin by within-platform rank** (every platform's #1, then #2, then #3) so a viral platform can't crowd out a smaller one, and drops the per-platform absolute floor so a less-watched video's killer low-vote comment still surfaces.
+- **First-run onboarding is consent-driven, model-led, and host-split.** The setup subprocess does only mechanical work (cookie reads, tool installs, GitHub device-auth, and emitting the engine-owned welcome via `--welcome`) — it cannot prompt, so consent lives in `SKILL.md` Step 0. Two flows avoid model-authored prose that Claude Code folds or the model skips: in the **Modal Flow** the welcome pitch is embedded in the setup modal's question (the AskUserQuestion modal is the only always-fully-visible surface — a separate welcome message or `--welcome` Bash run gets buried behind "ctrl+o to expand"); the **Non-Modal Prose Flow** still uses `last30days.py --welcome` (relayed verbatim) since it has no modal. The GitHub device code is surfaced by a two-command split — `setup --github-start` returns the code fast (foreground, copies to clipboard) and `setup --github-poll` waits for authorization (`setup --github` still chains both for back-compat). Step 0 has THREE branches: a **Claude Code Modal Flow** (the restored v3.0.0 `AskUserQuestion`-driven NUX — welcome, Auto/Manual/Skip, cookie consent, ScrapeCreators offer, `INCLUDE_SOURCES` opt-in, first-topic picker) for hosts with modals, a **Non-Modal Prose Flow** for hosts without (OpenClaw, Codex, Cursor, Gemini CLI, Grok CLI), and a **Grok Bot Prose Flow** for the Grok Bot host (`LAST30DAYS_HOST=grok-bot`), which leads with the bot's X connector lane (`LAST30DAYS_X_HOST_LANE=1` plus a `--x-posts` envelope the model fetches itself), persists any backup key (`X_BEARER_TOKEN` / `XAI_API_KEY`) only through the engine's `setup --store-key` path, and never reads a browser session. The first two ask before reading cookies and surface the macOS Full Disk Access fix on permission-denied; all three offer the ScrapeCreators GitHub signup (10,000 free calls) on every first run. A successful `setup --github` persists `SCRAPECREATORS_API_KEY` automatically (via `setup_wizard.write_api_key`, 0o600) and masks the key in stdout. Do NOT collapse the modal flow back into a bare silent `setup` call or flatten it to prose-only — the guided modals are the feature (they eroded once and were restored). The onboarding contract is locked by `tests/test_onboarding_contract.py`. The Step 5 source opt-in is two tiers, both comment-enabled: **Recommended** (TikTok + Instagram posts AND top comments, plus YouTube comments — `INCLUDE_SOURCES=tiktok,instagram,youtube_comments,tiktok_comments,instagram_comments`) and **Everything** (also Threads + Pinterest). Comments are on by default (posts on → comments on for all three platforms); **Threads and Pinterest are the only opt-in extras**, appearing only in the Step 5 Everything option, never in the welcome or the Step 4 offer. Instagram comments are fetched via ScrapeCreators (`/v2/instagram/post/comments`, ranked by `comment_like_count`) with full vote-weighting parity to YouTube/TikTok (a dedicated `_instagram_engagement` carve-out, the `_VOTE_LOG_REFERENCE`/label/threshold entries). The cross-platform "Top Community Comments" list (`render._render_top_comments`) selects **round-robin by within-platform rank** (every platform's #1, then #2, then #3) so a viral platform can't crowd out a smaller one, and drops the per-platform absolute floor so a less-watched video's killer low-vote comment still surfaces.
## Security hygiene
- Never commit real API keys, browser cookies, auth tokens, app passwords, access tokens, or `.env` contents.
diff --git a/CONCEPTS.md b/CONCEPTS.md
index d3259bd9..8a1d404d 100644
--- a/CONCEPTS.md
+++ b/CONCEPTS.md
@@ -92,6 +92,28 @@ Identity in the queue is annotate-only: a new topic name that closely matches an
The user-set status on a Topic queue row meaning "I already produced content for this story." Set by marking a topic covered by its exact name; surfaced is the only other status. A resurfacing never un-covers a row, and a new name that fuzzily matches a Covered row is born Covered - so the mark survives the judge (now the hosting model) renaming the same story across runs instead of silently re-pitching it.
+## X source routing
+
+### X backend chain
+
+The ordered list of X backends the Engine tries for one run when none is pinned, failing over to the next when one returns nothing. On an ordinary host the chain puts the free browser-session path first and the key-based backends after it. The chain is defined once and read everywhere it is described or predicted, so what doctor says the run will use and what the run actually uses cannot disagree.
+
+### Opt-in backend
+
+An X backend that is never part of the unpinned X backend chain and runs only when explicitly selected by the backend pin. The point is that a leftover login or an ambient credential on the machine can never silently claim the X lane, or spend metered credits, on a host that never asked for it. On ordinary hosts the Grok CLI and the direct X API bearer are Opt-in backends.
+
+### Extras host
+
+A host class (Linux, a Mac mini, or a machine that opted in explicitly) where the ordinary browser-session extract cannot reach a usable Chrome profile, so the Engine tries additional ways to obtain a complete X session before giving up on the free X path. Membership is decided from named signals only, never inferred from the home directory, PATH, or which Harness is running; a MacBook is never an Extras host unless it opts in.
+
+### Official X policy
+
+The host-conditional X routing rule that applies when the Harness identifies itself as a Grok Bot host through the host signal: the unpinned X backend chain contains only licensed backends (the direct X API bearer, xAI's licensed X search, and the X API through X's own CLI), no browser session is discovered or read, and every onboarding, doctor, and repair message names only those paths. The Engine trusts the host's self-identification and never sniffs the host. A backend pin keeps its exclusive meaning on such a host and is the only way to select a backend outside the official chain.
+
+### Host-fetched lane
+
+An X result the hosting model fetched itself, through its own X connector, and handed to the Engine as a file (the X connector envelope, passed with `--x-posts`), which then serves as the run's X source in place of the X backend chain. The envelope is strict at the top level (a malformed, stale, or off-topic file fails closed), lenient per row (a bad row is dropped and counted), and trusted for nothing beyond a numeric post id and a grammar-valid handle: citations are rebuilt by the Engine, never copied from the file. It is single-serve (one run, consumed once) and accepted on any host; a session-level lane signal tells planning that X is available before the file exists.
+
## Flagged ambiguities
- "Enrichment" is used for two distinct things: Comment-enrichment slots (fetching comments for already-ranked Reddit posts in the Keyless path) and Discovery's Enrichment pass (a full research run per Nomination). Context disambiguates; prefer the full term when writing.
diff --git a/CONFIGURATION.md b/CONFIGURATION.md
index 294a3f9c..ba3d27d0 100644
--- a/CONFIGURATION.md
+++ b/CONFIGURATION.md
@@ -79,12 +79,13 @@ Every completed research pass writes a structured `last-report.json` cache besid
## First-run onboarding
-On the very first `/last30days` run (no `~/.config/last30days/.env`, or `SETUP_COMPLETE` not set), the skill runs a consent-driven onboarding the model drives in chat. It takes one of two forms depending on the host:
+On the very first `/last30days` run (no `~/.config/last30days/.env`, or `SETUP_COMPLETE` not set), the skill runs a consent-driven onboarding the model drives in chat. It takes one of three forms depending on the host:
- **Claude Code Modal Flow** - the restored v3.0.0 guided NUX, used on hosts with `AskUserQuestion` (Claude Code). A welcome message, then modals for Auto/Manual/Skip setup, cookie consent, the ScrapeCreators signup offer, a TikTok/Instagram `INCLUDE_SOURCES` opt-in, and a first-topic picker.
- **Non-Modal Prose Flow** - the same work done conversationally on hosts without modals (OpenClaw, Codex, Cursor, Gemini CLI, Grok, raw CLI).
+- **Grok Bot Prose Flow** - the prose flow on a Grok Bot host (`LAST30DAYS_HOST=grok-bot`, persisted to `.env` by this setup). It has no browser-session step: X is set up through the bot's X connector, with `X_BEARER_TOKEN` or `XAI_API_KEY` as backups (see [Grok Bot](#grok-bot) under Per-client patterns).
-Both share the same consent points:
+The Modal and Non-Modal flows share the same consent points:
1. **Browser cookies** - the model asks before reading anything. On yes it runs `setup --allow-browser-cookies`, which extracts Firefox/Safari cookies (never Chrome unless `FROM_BROWSER=auto` or a named Chromium browser is explicitly configured) to unlock X/Twitter and other logged-in sources, and installs yt-dlp + the keyless Digg CLI. On no it runs setup without `--allow-browser-cookies` (or with `FROM_BROWSER=off`), which skips all cookie reads and still installs the tools.
2. **Full Disk Access (macOS)** - if a cookie read is permission-denied, the model surfaces the System Settings > Privacy & Security > Full Disk Access fix and offers one retry.
@@ -150,7 +151,7 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \
| Techmeme | `techmeme-pp-cli` on PATH (auto-installed via `... install techmeme --cli-only`) | always on if `techmeme-pp-cli` on PATH; searches Techmeme's live archive and keeps only headlines dated within the research window (undated headlines flow through as low-confidence) | yes (free, keyless) |
| Trustpilot | `trustpilot-pp-cli` on PATH (NOT auto-installed; install on demand via `npx -y @mvanhorn/[email protected] install trustpilot --cli-only`) + (`INCLUDE_SOURCES` contains `trustpilot` **or** an explicit `--trustpilot-domain` / plan-level `trustpilot_domain`) | **opt-in, off by default**; `--trustpilot-domain=<domain>` (and per-entity `trustpilot_domain` in `--competitors-plan`) auto-activates the source for that run and bypasses the brand-shape gate. Persist with `INCLUDE_SOURCES=trustpilot` to skip per-run auto-enable. `EXCLUDE_SOURCES=trustpilot` still wins. Bare company names auto-resolve to the review-page domain via the CLI's search only when the source is already active. The session warms once before the search fan-out; a stale session does a ~10s headless-Chrome WAF-cookie harvest (set `LAST30DAYS_TRUSTPILOT_NO_BROWSER=1` to disable in cron/CI) | yes (no API key; cookie-replay after the one-time harvest) |
| Amazon | `brightdata` CLI on PATH **and logged in** (NOT auto-installed: `npm i -g @brightdata/cli` then `brightdata login`) + (`INCLUDE_SOURCES` contains `amazon` **or** `--search` includes `amazon`) | product records with live rating, rating count, and price, plus a capped sample of recent written reviews woven as buyer voice; the emoji footer shows each product's all-time-vs-last-30-days drift | **opt-in, off by default**. Free tier is 5,000 requests/month (~$7.50 equivalent); a typical run spends 4 (1 product search + up to 3 review pulls) regardless of how many reviews come back, since billing is per request. Past the free tier it bills the account balance at $1.50 per 1,000 records (~$0.32 for a default run). `--amazon-query=<keyword>` sets the product keyword when it differs from the topic; `LAST30DAYS_AMAZON_DOMAIN` selects a non-US marketplace. `EXCLUDE_SOURCES=amazon` wins. Never auto-fires: the model requests it per run or the user enables it durably |
-| X / Twitter | one of: a signed-in `grok` CLI (opt-in: `LAST30DAYS_X_BACKEND=grok`), `AUTH_TOKEN` + `CT0` (browser cookies, Bird CLI), `XAI_API_KEY`, `XQUIK_API_KEY`, or `FROM_BROWSER` (cookie-jar auth) | X items in results | grok = Grok plan, opt-in only; cookie-jar / Bird = free; Xquik / xAI = key-based |
+| X / Twitter | one of: `X_BEARER_TOKEN` (official X API v2; opt-in outside Grok Bot: `LAST30DAYS_X_BACKEND=xapi`; covers recent posts, about the last week, unless your X developer project has full-archive access), a signed-in `grok` CLI (opt-in: `LAST30DAYS_X_BACKEND=grok`), `AUTH_TOKEN` + `CT0` (browser cookies, Bird CLI), `XAI_API_KEY`, `XQUIK_API_KEY`, or `FROM_BROWSER` (cookie-jar auth). On a Grok Bot host the bot's X connector serves X first; see [Grok Bot](#grok-bot) | X items in results | X API bearer = your X developer project's credits; grok = Grok plan, opt-in only; cookie-jar / Bird = free; Xquik / xAI = key-based |
| TikTok | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok` | TikTok items | 10K free calls |
| Instagram | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram` | Instagram Reels | 10K free calls; raise `LAST30DAYS_TRANSCRIPT_TIMEOUT` (default 30s) if SC is slow on your network |
| Threads | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `threads` | Threads items | 10K free calls |
@@ -170,11 +171,11 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \
**YouTube transcript tuning.** `LAST30DAYS_YT_SUB_LANGS` controls the comma-separated caption-language priority passed to yt-dlp and defaults to `en,es,pt`. `LAST30DAYS_YT_PLAYER_CLIENT` defaults to `android` so yt-dlp can pass YouTube's web bot-gate without cookies (search, transcripts, and comments); set it empty to disable. When `SCRAPECREATORS_API_KEY` is available, yt-dlp uses one fast attempt before the paid fallback; set `LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT` to the number of seconds allowed for that attempt when a throttled host needs longer than the 12-second default. A VTT completed before the timeout is reused rather than discarded. `LAST30DAYS_YT_SEARCH_TIMEOUT` sets the per-search yt-dlp deadline (default 120s). Comparison-mode fan-out also caps concurrent yt-dlp processes process-wide and caches identical searches within a run so redundant `ytsearch` calls do not self-throttle the same IP.
-**X backend priority (bird first).** The default X backend chain is bird (browser cookies) → xai (API key) → xurl (OAuth2 CLI) → xquik (API key). Cookies beat `XAI_API_KEY` when both are present. A leftover grok login never steals the X lane; see below.
+**X backend priority (bird first).** The default X backend chain is bird (browser cookies) → xai (API key) → xurl (OAuth2 CLI) → xquik (API key). Cookies beat `XAI_API_KEY` when both are present. A leftover grok login never steals the X lane; see below. `xapi` (the official X API v2 with `X_BEARER_TOKEN`) is opt-in on these hosts (`LAST30DAYS_X_BACKEND=xapi`), so an ambient bearer never spends X API credits when the free path comes back empty. **Grok Bot exception.** On a Grok Bot host (`LAST30DAYS_HOST=grok-bot`) the unpinned chain is the official chain instead: xapi (`X_BEARER_TOKEN`) → xai (`XAI_API_KEY`) → xurl (the X API through X's CLI). When the bot's X connector is in the session, its results come first and the chain is not called at all; see [Grok Bot](#grok-bot) under Per-client patterns.
**Grok CLI (opt-in backup).** Install the Grok CLI (`curl -fsSL https://x.ai/cli/install.sh | bash`) and run `grok login`, and X can work with no X account, no browser cookies, and no `XAI_API_KEY`. However, grok is **opt-in only**: a leftover `~/.grok/auth.json` must never steal the X lane. Pin `LAST30DAYS_X_BACKEND=grok` to enable it. It is not "free" in the way the cookie path is: calls draw on your Grok plan, and depth costs several calls per run because the underlying tool caps each search at 10 posts. Results are validated before use — every returned post's ID is decoded to confirm it falls inside the requested date range, because the retrieval is performed by a language model and can otherwise return confident, well-formed posts that were never searched for.
-**X on cookie-less hosts.** Bird (the free X source) scrapes X using your logged-in browser cookies (`AUTH_TOKEN`/`CT0`), which agent hosts like OpenClaw, CI, or headless runs often can't supply — and scraping carries some account risk. On those, set `XQUIK_API_KEY` (or `XAI_API_KEY`) for full, ranked X coverage from a single API key: the same engagement-based ranking, first-party authorship, and handle (from/mentions) lanes the native X source gets. `--diagnose` reports whether the key is working (and flags an unpaid key).
+**X on cookie-less hosts.** Bird (the free X source) scrapes X using your logged-in browser cookies (`AUTH_TOKEN`/`CT0`), which agent hosts like OpenClaw, CI, or headless runs often can't supply — and scraping carries some account risk. On those, set `XQUIK_API_KEY` (or `XAI_API_KEY`) for full, ranked X coverage from a single API key: the same engagement-based ranking, first-party authorship, and handle (from/mentions) lanes the native X source gets. The official X API is the other keyed option: set `X_BEARER_TOKEN` and pin `LAST30DAYS_X_BACKEND=xapi`; it serves the same lanes but covers recent posts, about the last week, unless your X developer project has full-archive access. `--diagnose` reports whether the key is working (and flags an unpaid key as `payment-required`).
**Extra bird cookie lookups on Linux and Mac mini.** On a MacBook the X cookie path is unchanged (Firefox/Safari/Chrome extract, gated by `FROM_BROWSER`). On **extra hosts** the engine adds two more ways to hand bird a complete `auth_token`+`ct0` pair, tried in order (first COMPLETE pair wins; no half-pair merge; nothing is ever written to the `.env` and cookie values are never printed):
@@ -185,7 +186,7 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \
A host counts as an "extra host" when ANY of these hold: `AGENTCOOKIE=on` (explicit opt-in, any OS); the platform is Linux; a Darwin **Mac mini** (`sysctl -n hw.model` prefix `Macmini`); or a Darwin **agentcookie sink** role. The host is never inferred from the home directory, PATH, or Hermes/OpenClaw env — only those signals. A plain MacBook does no agentcookie spawn and opens no CDP socket unless `AGENTCOOKIE=on`.
-CDP endpoint resolution (extra hosts only, no port scan): `BROWSER_CDP_URL` if set, else port `18800` when it answers as Chrome, else `9222` + the X display number. Port `18800` is the last30days extras **NUX convention** — the agent launches a throwaway login Chrome with `SAND_CHROME_REMOTE_DEBUG_PORT=18800` (see SKILL.md's "X on Linux / Grok Bot / Mac mini"), so it is not confused with a daily Chrome profile on `9222`+display (box-chrome's own built-in default). `18800` is tried first but falls through when it yields no complete pair, so a logged-out Chrome there never shadows a logged-in profile; pin `BROWSER_CDP_URL` if a stale session answers there. A Node `--inspect` endpoint is rejected; a Chrome page target is required.
+CDP endpoint resolution (extra hosts only, no port scan): `BROWSER_CDP_URL` if set, else port `18800` when it answers as Chrome, else `9222` + the X display number. Port `18800` is the last30days extras **NUX convention** — the agent launches a throwaway login Chrome with `SAND_CHROME_REMOTE_DEBUG_PORT=18800` (see SKILL.md's "X on Linux / Mac mini"), so it is not confused with a daily Chrome profile on `9222`+display (box-chrome's own built-in default). `18800` is tried first but falls through when it yields no complete pair, so a logged-out Chrome there never shadows a logged-in profile; pin `BROWSER_CDP_URL` if a stale session answers there. A Node `--inspect` endpoint is rejected; a Chrome page target is required.
**Example `.env` skeleton** (placeholders only - replace with your own values):
@@ -214,6 +215,10 @@ INCLUDE_SOURCES=tiktok,instagram
# X authentication (one option only)
AUTH_TOKEN=<your-auth-token>
CT0=<your-ct0-token>
+# OR the official X API v2 bearer. Default on Grok Bot; elsewhere also pin
+# LAST30DAYS_X_BACKEND=xapi. Covers recent posts, about the last week, unless
+# your X developer project has full-archive access.
+# X_BEARER_TOKEN=<your-x-api-bearer-token>
# OR xAI API key (paid)
# XAI_API_KEY=<your-xai-key>
# OR Xquik key-based X search
@@ -451,6 +456,30 @@ python3 skills/last30days/scripts/last30days.py "Listen Labs" --hiring-signals
The engine treats public jobs/careers postings as evidence of focus or priority shifts, not exact roadmap predictions. Standard company runs may include Hiring Signals automatically when multiple current roles support the same interpretation; weak or unavailable hiring evidence is omitted.
+### `--x-posts` flag
+
+`--x-posts <path>` hands the engine an X result the hosting model fetched through its own X connector; it replaces the engine's X fetch for that run and works on any host. The value is a file path only (inline JSON exits `2`): a regular `.json` file in the `last30days-x-posts/1` shape, never read from inside the config dir or a credential store.
+
+| Field | Meaning |
+| --- | --- |
+| `schema`, `generated_at`, `topic`, `window {from, to}`, `provider`, `status` | Envelope header. `status` is `ok`, `partial`, or `error`; `error` is a short category (`credits`, `not-connected`, `unavailable`, `window-unsupported`), never raw tool output. `topic` must match the run topic and `generated_at` must be under 6 hours old, or the run fails closed with exit `2`. |
+| `calls[]` | One entry per connector call: `lane` (`topic`, `from`, `mention`, `related`), `handles` (a subset of the run's `--x-handle` / `--x-related` handles), and `posts`. |
+| `posts[]` | Flat rows with exactly eight fields: `id`, `author_handle`, `created_at`, `text`, `likes`, `reposts`, `replies`, `quotes`. Any other key is ignored and counted. |
+
+Limits: 8 MiB, strict UTF-8, at most 20 calls, 500 rows per call, 1,000 rows in total, 10,000 characters of text per row. Rows are rebuilt from validated parts: the citation is always `https://x.com/<handle>/status/<id>` (a row-supplied URL is never used), rows without an id or text, outside the window, or whose date disagrees with the id are dropped and counted, and an id sequence that looks generated rejects the whole file. The envelope is single-serve for the run. The hosted backend (`LAST30DAYS_API_BASE`) rejects the flag with exit `2`. Comparison runs take the per-entity `x_posts` field of `--competitors-plan` instead; a bare `--x-posts` on a comparison run exits `2`.
+
+```bash
+python3 skills/last30days/scripts/last30days.py "<topic>" --x-posts /tmp/x-posts.json
+```
+
+### `setup --store-key`
+
+`setup --store-key <NAME>` persists one credential to the global `.env` (mode `600`) from a single line on stdin, without echoing it: stdout shows `NAME=****` plus a JSON line `{"persisted": true, "key": "NAME"}`. `NAME` must be one of the credential names the engine loads from `.env` (for example `X_BEARER_TOKEN`, `XAI_API_KEY`, `SCRAPECREATORS_API_KEY`); an unknown name or an empty value exits `2`. Running it again with a new value replaces the stored one (rotating a rejected credential); other lines in the file are untouched.
+
+```bash
+printf '%s\n' "$TOKEN" | python3 skills/last30days/scripts/last30days.py setup --store-key X_BEARER_TOKEN
+```
+
---
## Health check (`doctor`)
@@ -477,7 +506,10 @@ Every live run writes its JSON result to `~/.config/last30days/doctor-cache.json
| --- | --- |
| `LAST30DAYS_DOCTOR_TTL` | Freshness window for `doctor --cached`, in **seconds**. Defaults to `900` (15 minutes). `0` makes every `--cached` call run live. |
| `LAST30DAYS_DOCTOR_PROBE_TIMEOUT` | Per-source deadline (**seconds**) for `doctor --probe` live checks. Defaults to `10`. Caps each concurrent probe so a slow source cannot hang the command. |
-| `LAST30DAYS_X_BACKEND` | Pins the X backend (`bird` / `xai` / `xurl` / `xquik` / `grok`); doctor renders the pin and predicts "will use" accordingly. The unpinned auto chain is bird → xai → xurl → xquik (grok is opt-in only). Pin `grok` to enable it; a leftover `~/.grok/auth.json` is never auto-selected. |
+| `LAST30DAYS_HOST` | Host self-identification. `grok-bot` switches X to the official chain (xapi → xai → xurl) and turns off browser-session discovery; any other value, or unset, leaves every host exactly as today. Persisted to `.env` by first-run setup on Grok Bot and exported per invocation by the skill; doctor prints the resolved value. The engine never infers the host any other way. |
+| `X_BEARER_TOKEN` | App-only bearer for the official X API v2 (`xapi`). First rung of the chain on Grok Bot; opt-in elsewhere via `LAST30DAYS_X_BACKEND=xapi`. Full-archive search is tried first, then recent search, so coverage is recent posts, about the last week, unless your X developer project has full-archive access (the outcome detail says `window truncated to 7 days` when the fallback ran, and `search stopped at the lane deadline; results may be incomplete` when the shared 150s lane budget cut a search short). Exhausted credits (HTTP 402) report as `payment-required`. Doctor checks presence only, never the network. Loaded from `.env`, Keychain, or `pass` like the other keys. |
+| `LAST30DAYS_X_HOST_LANE` | `1` declares that the hosting model's X connector is in this session, so `--diagnose` and planning list `x` as available and the run expects `--x-posts`. Read from the process environment only: a `.env` line is ignored (doctor says so), so a removed connector never leaves a stale declaration. A run with the signal but no `--x-posts` records X as `error` ("connector result not passed"). |
+| `LAST30DAYS_X_BACKEND` | Pins the X backend (`bird` / `xai` / `xurl` / `xquik` / `grok` / `xapi`); doctor renders the pin and predicts "will use" accordingly. The unpinned auto chain is bird → xai → xurl → xquik (grok and xapi are opt-in only). Pin `grok` to enable it; a leftover `~/.grok/auth.json` is never auto-selected. Pin `xapi` to use `X_BEARER_TOKEN` on an ordinary host. On a Grok Bot host the unpinned chain is the official chain (xapi → xai → xurl) and this pin is the only way to select a backend outside it; the pin keeps its exclusive, no-failover meaning there, and doctor names the pinned backend. |
| `AGENTCOOKIE` | `on` opts any host (incl. a MacBook) into the extra bird cookie lookups (agentcookie sidecar + live Chrome CDP); `off` disables the agentcookie sidecar reader. Unset uses host detection (Linux / Mac mini / Darwin sink get the extras). See "Extra bird cookie lookups" above. |
| `BROWSER_CDP_URL` | Explicit Chrome DevTools endpoint (e.g. `http://127.0.0.1:18800`) for the extra-host CDP cookie lookup. Preferred over the `18800` / `9222`+`$DISPLAY` defaults. Extra hosts only. |
| `LAST30DAYS_REDDIT_BACKEND` | `scrapecreators` makes ScrapeCreators the primary Reddit backend; doctor renders Reddit's conditional routing with the pin applied. |
@@ -491,7 +523,7 @@ By default a research run exits `0` even when a source failed mid-run (rate-limi
| Var | Effect |
| --- | --- |
-| `LAST30DAYS_STRICT_EXIT` | Truthy (`1`/`true`/`yes`/`on`): the engine exits `3` when any source outcome is neither `ok`, `no-results`, nor `skipped-unconfigured`. A one-line `strict-exit: degraded sources: ...` note goes to stderr. Default (unset): exit `0`, unchanged behavior. |
+| `LAST30DAYS_STRICT_EXIT` | Truthy (`1`/`true`/`yes`/`on`): the engine exits `3` when any source outcome is neither `ok`, `no-results`, nor `skipped-unconfigured` (so `partial`, `auth-failed`, `payment-required` for exhausted credits, `rate-limited`, and the other failure states all count as degraded). A one-line `strict-exit: degraded sources: ...` note goes to stderr. Default (unset): exit `0`, unchanged behavior. |
Exit codes with the flag on: `0` clean run, `3` completed-but-degraded (report was produced), non-zero others unchanged (hard failures). Same hybrid pattern as `LAST30DAYS_DEBUG` — works shell-exported or in `.env`.
@@ -657,6 +689,22 @@ For competitor-vs-comparisons that recur, a pre-written JSON skeleton per client
Pass as `--competitors-plan @client/competitors-plan.json` (or as a string). See `SKILL.md` section "If QUERY_TYPE = COMPARISON" for the full schema.
+### Grok Bot
+
+On a Grok Bot host the bot exports `LAST30DAYS_HOST=grok-bot` on every engine call (first-run setup also persists it to `.env`), and X search runs through official channels only, in this order:
+
+1. **X connector (primary).** Add the "X for Grok Bot" plugin and connect your X account in Grok Bot settings (it provisions an X developer account for you; paid Grok Bot plans include X API credits). When one of the plugin's post-search tools is in the session, the bot exports `LAST30DAYS_X_HOST_LANE=1`, fetches the posts itself, writes them to a `last30days-x-posts/1` file, and passes it with `--x-posts <path>` (see [`--x-posts` flag](#--x-posts-flag)). Connector calls draw on the credits included with Grok Bot and cover the full research window. `--diagnose` lists `x` as available whenever the lane signal is set; a run with the signal but no `--x-posts` file records X as `error` ("connector result not passed").
+2. **`X_BEARER_TOKEN` (backup).** An app-only bearer from the X developer console, funded by your own X developer project. The engine tries full-archive search first and falls back to recent search, so coverage is recent posts, about the last week, unless your X developer project has full-archive access; the outcome detail says `window truncated to 7 days` when the fallback ran. Exhausted credits (HTTP 402) report as `payment-required`, which `LAST30DAYS_STRICT_EXIT` treats as degraded.
+3. **`XAI_API_KEY` (backup).** xAI's licensed X search from console.x.ai: full window, topic search only (no from/mention handle lanes).
+
+Persist either key without echoing it:
+
+```bash
+printf '%s\n' "$TOKEN" | python3 skills/last30days/scripts/last30days.py setup --store-key X_BEARER_TOKEN
+```
+
+Browser sessions are not read on this host, and no login window is opened; `setup` still installs the free CLIs. Doctor prints the resolved host value, so a missing `LAST30DAYS_HOST` export is visible at a glance.
+
---
## Beta channel
diff --git a/README.de.md b/README.de.md
index f910339b..3253e3d0 100644
--- a/README.de.md
+++ b/README.de.md
@@ -198,7 +198,7 @@ grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
-Mit `--trust` überspringst du die Installationsbestätigung. Aktualisieren kannst du mit `grok plugin update last30days`. Grok liest aus Kompatibilitätsgründen auch die Claude-Code-Manifeste; das native `.grok-plugin/`-Paar ist der bevorzugte Weg – und genau darauf verweist ein offizieller Eintrag im [xAI-Marketplace](https://github.com/xai-org/plugin-marketplace). `npx skills add` bleibt ein gültiger Fallback über alle Hosts hinweg.
+Mit `--trust` überspringst du die Installationsbestätigung. Aktualisieren kannst du mit `grok plugin update last30days`. Grok liest aus Kompatibilitätsgründen auch die Claude-Code-Manifeste; das native `.grok-plugin/`-Paar ist der bevorzugte Weg – und genau darauf verweist ein offizieller Eintrag im [xAI-Marketplace](https://github.com/xai-org/plugin-marketplace). `npx skills add` bleibt ein gültiger Fallback über alle Hosts hinweg. Auf Grok Bot läuft die X-Suche über den X-Connector des Bots, mit der offiziellen X-API (`X_BEARER_TOKEN`) als Fallback.
### Codex, Cursor, Copilot, Gemini CLI und weitere Agent-Skills-Hosts
@@ -293,7 +293,7 @@ Diese Plattformen haben nichts miteinander zu tun. X weiß nicht, was Reddit den
|---------|---------------|------|
| Reddit (mit Kommentaren) + HN + Polymarket + GitHub + StockTwits | Nichts | Kostenlos |
| arXiv + Techmeme | Kostenlose CLIs, die das Erst-Setup automatisch installiert | Kostenlos |
-| X / Twitter | In einem beliebigen Browser bei x.com anmelden, oder `XQUIK_API_KEY` / `XAI_API_KEY` setzen | Browser-Cookies sind kostenlos; Schlüssel hängen vom Anbieter ab |
+| X / Twitter | `X_BEARER_TOKEN` für die offizielle X-API setzen (aktuelle Posts, etwa die letzte Woche, sofern dein X-Entwicklerprojekt keinen Vollarchiv-Zugang hat; Standard auf Grok Bot, anderswo per Opt-in mit `LAST30DAYS_X_BACKEND=xapi`), oder in einem beliebigen Browser bei x.com anmelden, oder `XQUIK_API_KEY` / `XAI_API_KEY` setzen | X-API-Guthaben stammt aus deinem X-Entwicklerprojekt; Browser-Cookies sind kostenlos; andere Schlüssel hängen vom Anbieter ab |
| YouTube | `brew install yt-dlp` | Kostenlos |
| Bluesky | App-Passwort von bsky.app | Kostenlos |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + YouTube-Kommentare | Ein ScrapeCreators-Schlüssel | 10.000 kostenlose Aufrufe, danach nutzungsabhängig |
diff --git a/README.es.md b/README.es.md
index f3bdc760..4b59ff5e 100644
--- a/README.es.md
+++ b/README.es.md
@@ -198,7 +198,7 @@ grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
-Añade `--trust` para saltarte la confirmación de instalación. Actualiza con `grok plugin update last30days`. Grok también lee los manifiestos de Claude Code por compatibilidad; el par nativo `.grok-plugin/` es la vía principal, y es a lo que apunta una entrada oficial en el [marketplace de xAI](https://github.com/xai-org/plugin-marketplace). `npx skills add` sigue siendo una alternativa válida para cualquier host.
+Añade `--trust` para saltarte la confirmación de instalación. Actualiza con `grok plugin update last30days`. Grok también lee los manifiestos de Claude Code por compatibilidad; el par nativo `.grok-plugin/` es la vía principal, y es a lo que apunta una entrada oficial en el [marketplace de xAI](https://github.com/xai-org/plugin-marketplace). `npx skills add` sigue siendo una alternativa válida para cualquier host. En Grok Bot, la búsqueda en X pasa por el conector de X del bot, con la API oficial de X (`X_BEARER_TOKEN`) como respaldo.
### Codex, Cursor, Copilot, Gemini CLI y otros hosts de Agent Skills
@@ -293,7 +293,7 @@ Estas plataformas no tienen ninguna relación entre sí. X no sabe lo que piensa
|---------|---------------|------|
| Reddit (con comentarios) + HN + Polymarket + GitHub + StockTwits | Nada | Gratis |
| arXiv + Techmeme | CLI gratuitas, instaladas automáticamente por la configuración inicial | Gratis |
-| X / Twitter | Inicia sesión en x.com en cualquier navegador, o define `XQUIK_API_KEY` / `XAI_API_KEY` | Las cookies del navegador son gratis; las claves dependen del proveedor |
+| X / Twitter | Define `X_BEARER_TOKEN` para la API oficial de X (posts recientes, aproximadamente la última semana, salvo que tu proyecto de desarrollador de X tenga acceso al archivo completo; es el valor por defecto en Grok Bot y opcional en otros hosts con `LAST30DAYS_X_BACKEND=xapi`), o inicia sesión en x.com en cualquier navegador, o define `XQUIK_API_KEY` / `XAI_API_KEY` | Los créditos de la API de X salen de tu proyecto de desarrollador de X; las cookies del navegador son gratis; las demás claves dependen del proveedor |
| YouTube | `brew install yt-dlp` | Gratis |
| Bluesky | Una contraseña de aplicación de bsky.app | Gratis |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + comentarios de YouTube | Una clave de ScrapeCreators | 10.000 llamadas gratis y luego pago por uso |
diff --git a/README.fr.md b/README.fr.md
index 2076499b..6b68719f 100644
--- a/README.fr.md
+++ b/README.fr.md
@@ -198,7 +198,7 @@ grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
-Ajoutez `--trust` pour sauter la confirmation d'installation. Mettez à jour avec `grok plugin update last30days`. Grok lit aussi les manifestes Claude Code par compatibilité ; la paire native `.grok-plugin/` reste la voie principale, et c'est elle que pointe une entrée officielle dans la [marketplace xAI](https://github.com/xai-org/plugin-marketplace). `npx skills add` reste une solution de repli valable, tous hôtes confondus.
+Ajoutez `--trust` pour sauter la confirmation d'installation. Mettez à jour avec `grok plugin update last30days`. Grok lit aussi les manifestes Claude Code par compatibilité ; la paire native `.grok-plugin/` reste la voie principale, et c'est elle que pointe une entrée officielle dans la [marketplace xAI](https://github.com/xai-org/plugin-marketplace). `npx skills add` reste une solution de repli valable, tous hôtes confondus. Sur Grok Bot, la recherche X passe par le connecteur X du bot, avec l'API X officielle (`X_BEARER_TOKEN`) en secours.
### Codex, Cursor, Copilot, Gemini CLI et autres hôtes Agent Skills
@@ -293,7 +293,7 @@ Ces plateformes n'ont aucune relation entre elles. X ignore ce que pense Reddit.
|---------|---------------|------|
| Reddit (avec les commentaires) + HN + Polymarket + GitHub + StockTwits | Rien | Gratuit |
| arXiv + Techmeme | Des CLI gratuites, installées automatiquement à la configuration initiale | Gratuit |
-| X / Twitter | Connectez-vous à x.com dans n'importe quel navigateur, ou définissez `XQUIK_API_KEY` / `XAI_API_KEY` | Les cookies de navigateur sont gratuits ; les clés dépendent du fournisseur |
+| X / Twitter | Définissez `X_BEARER_TOKEN` pour l'API X officielle (posts récents, environ la dernière semaine, sauf si votre projet développeur X dispose de l'accès aux archives complètes ; par défaut sur Grok Bot, en opt-in ailleurs avec `LAST30DAYS_X_BACKEND=xapi`), ou connectez-vous à x.com dans n'importe quel navigateur, ou définissez `XQUIK_API_KEY` / `XAI_API_KEY` | Les crédits de l'API X proviennent de votre projet développeur X ; les cookies de navigateur sont gratuits ; les autres clés dépendent du fournisseur |
| YouTube | `brew install yt-dlp` | Gratuit |
| Bluesky | Un mot de passe d'application depuis bsky.app | Gratuit |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + commentaires YouTube | Une clé ScrapeCreators | 10 000 appels gratuits, puis paiement à l'usage |
diff --git a/README.ja.md b/README.ja.md
index 71551059..db3cc444 100644
--- a/README.ja.md
+++ b/README.ja.md
@@ -198,7 +198,7 @@ grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
-インストール時の確認を省きたい場合は `--trust` を付けてください。更新は `grok plugin update last30days` です。Grok は互換性のために Claude Code のマニフェストも読みますが、第一の経路はネイティブの `.grok-plugin/` のペアで、[xAI のマーケットプレイス](https://github.com/xai-org/plugin-marketplace)への公式掲載もこちらを指しています。`npx skills add` は、どのホストでも使える代替手段として引き続き有効です。
+インストール時の確認を省きたい場合は `--trust` を付けてください。更新は `grok plugin update last30days` です。Grok は互換性のために Claude Code のマニフェストも読みますが、第一の経路はネイティブの `.grok-plugin/` のペアで、[xAI のマーケットプレイス](https://github.com/xai-org/plugin-marketplace)への公式掲載もこちらを指しています。`npx skills add` は、どのホストでも使える代替手段として引き続き有効です。Grok Bot では、X 検索はボットの X コネクタを通じて実行され、公式 X API(`X_BEARER_TOKEN`)がバックアップになります。
### Codex、Cursor、Copilot、Gemini CLI、その他の Agent Skills ホスト
@@ -292,7 +292,7 @@ Reddit(コメント込み)、Hacker News、Polymarket、GitHub はすぐに使
|---------|---------------|------|
| Reddit(コメント込み)+ HN + Polymarket + GitHub + StockTwits | 不要 | 無料 |
| arXiv + Techmeme | 無料のCLI。初回セットアップが自動でインストールします | 無料 |
-| X / Twitter | 任意のブラウザで x.com にログインするか、`XQUIK_API_KEY` / `XAI_API_KEY` を設定 | ブラウザのクッキーは無料。キーの料金は提供元によります |
+| X / Twitter | 公式 X API 用に `X_BEARER_TOKEN` を設定(最近の投稿、およそ直近 1 週間分。X 開発者プロジェクトにフルアーカイブ検索のアクセスがある場合を除く。Grok Bot では既定、それ以外のホストでは `LAST30DAYS_X_BACKEND=xapi` でオプトイン)するか、任意のブラウザで x.com にログインするか、`XQUIK_API_KEY` / `XAI_API_KEY` を設定 | X API のクレジットはご自身の X 開発者プロジェクトから消費。ブラウザのクッキーは無料。その他のキーの料金は提供元によります |
| YouTube | `brew install yt-dlp` | 無料 |
| Bluesky | bsky.app のアプリパスワード | 無料 |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + YouTube のコメント | ScrapeCreators のキー | 1万リクエストまで無料、以降は従量課金 |
diff --git a/README.md b/README.md
index 8f814f94..2bed26d9 100644
--- a/README.md
+++ b/README.md
@@ -198,7 +198,7 @@ grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
-Add `--trust` to skip the install confirmation. Update with `grok plugin update last30days`. Grok also reads the Claude Code manifests for compatibility; the native `.grok-plugin/` pair is the first-class lane (and what an official [xAI marketplace](https://github.com/xai-org/plugin-marketplace) listing points at). `npx skills add` remains a valid cross-host fallback.
+Add `--trust` to skip the install confirmation. Update with `grok plugin update last30days`. Grok also reads the Claude Code manifests for compatibility; the native `.grok-plugin/` pair is the first-class lane (and what an official [xAI marketplace](https://github.com/xai-org/plugin-marketplace) listing points at). `npx skills add` remains a valid cross-host fallback. On Grok Bot, X search runs through the bot's X connector, with the official X API (`X_BEARER_TOKEN`) as backup.
### Codex, Cursor, Copilot, Gemini CLI, and other Agent Skills hosts
@@ -292,7 +292,7 @@ These platforms don't have relationships with each other. X doesn't know what Re
|---------|---------------|------|
| Reddit (with comments) + HN + Polymarket + GitHub + StockTwits | Nothing | Free |
| arXiv + Techmeme | Free CLIs, auto-installed by first-run setup | Free |
-| X / Twitter | Log into x.com in any browser, or set `XQUIK_API_KEY` / `XAI_API_KEY` | Browser cookies are free; keys are provider-specific |
+| X / Twitter | Set `X_BEARER_TOKEN` for the official X API (recent posts, about the last week, unless your X developer project has full-archive access; the default on Grok Bot, opt-in elsewhere with `LAST30DAYS_X_BACKEND=xapi`), or log into x.com in any browser, or set `XQUIK_API_KEY` / `XAI_API_KEY` | X API credits come from your X developer project; browser cookies are free; other keys are provider-specific |
| YouTube | `brew install yt-dlp` | Free |
| Bluesky | App password from bsky.app | Free |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + YouTube comments | ScrapeCreators key | 10,000 free calls, then PAYG |
diff --git a/README.pt-BR.md b/README.pt-BR.md
index 06f33741..4d7e1373 100644
--- a/README.pt-BR.md
+++ b/README.pt-BR.md
@@ -198,7 +198,7 @@ grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
-Acrescente `--trust` para pular a confirmação de instalação. Atualize com `grok plugin update last30days`. O Grok também lê os manifestos do Claude Code por compatibilidade; o par nativo `.grok-plugin/` é o caminho principal — e é para ele que aponta um registro oficial no [marketplace da xAI](https://github.com/xai-org/plugin-marketplace). O `npx skills add` continua sendo uma alternativa válida em qualquer host.
+Acrescente `--trust` para pular a confirmação de instalação. Atualize com `grok plugin update last30days`. O Grok também lê os manifestos do Claude Code por compatibilidade; o par nativo `.grok-plugin/` é o caminho principal — e é para ele que aponta um registro oficial no [marketplace da xAI](https://github.com/xai-org/plugin-marketplace). O `npx skills add` continua sendo uma alternativa válida em qualquer host. No Grok Bot, a busca no X passa pelo conector de X do bot, com a API oficial do X (`X_BEARER_TOKEN`) como reserva.
### Codex, Cursor, Copilot, Gemini CLI e outros hosts do Agent Skills
@@ -293,7 +293,7 @@ Essas plataformas não têm relação nenhuma entre si. O X não sabe o que o Re
|---------|---------------|------|
| Reddit (com comentários) + HN + Polymarket + GitHub + StockTwits | Nada | De graça |
| arXiv + Techmeme | CLIs gratuitas, instaladas automaticamente pela configuração inicial | De graça |
-| X / Twitter | Faça login em x.com em qualquer navegador, ou defina `XQUIK_API_KEY` / `XAI_API_KEY` | Os cookies do navegador são gratuitos; as chaves dependem do provedor |
+| X / Twitter | Defina `X_BEARER_TOKEN` para a API oficial do X (posts recentes, cerca da última semana, a menos que seu projeto de desenvolvedor do X tenha acesso ao arquivo completo; é o padrão no Grok Bot e opcional em outros hosts com `LAST30DAYS_X_BACKEND=xapi`), ou faça login em x.com em qualquer navegador, ou defina `XQUIK_API_KEY` / `XAI_API_KEY` | Os créditos da API do X vêm do seu projeto de desenvolvedor do X; os cookies do navegador são gratuitos; as demais chaves dependem do provedor |
| YouTube | `brew install yt-dlp` | De graça |
| Bluesky | Uma senha de aplicativo do bsky.app | De graça |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + comentários do YouTube | Uma chave do ScrapeCreators | 10.000 chamadas gratuitas e depois pagamento por uso |
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 49739256..658255ab 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -201,7 +201,7 @@ grok plugin marketplace add mvanhorn/last30days-skill
grok plugin install last30days
```
-加入 `--trust` 可跳过安装确认;使用 `grok plugin update last30days` 更新。为兼容旧机制,Grok 也会读取 Claude Code 的清单文件;原生 `.grok-plugin/` 文件是首选通路,也是 [xAI marketplace](https://github.com/xai-org/plugin-marketplace) 官方目录条目指向的对象。`npx skills add` 仍是有效的跨宿主备用方案。
+加入 `--trust` 可跳过安装确认;使用 `grok plugin update last30days` 更新。为兼容旧机制,Grok 也会读取 Claude Code 的清单文件;原生 `.grok-plugin/` 文件是首选通路,也是 [xAI marketplace](https://github.com/xai-org/plugin-marketplace) 官方目录条目指向的对象。`npx skills add` 仍是有效的跨宿主备用方案。在 Grok Bot 上,X 搜索通过机器人的 X 连接器执行,并以官方 X API(`X_BEARER_TOKEN`)作为备用。
### Codex、Cursor、Copilot、Gemini CLI 与其他 Agent Skills 宿主
@@ -291,7 +291,7 @@ Reddit(含评论)、Hacker News、Polymarket 和 GitHub 无需任何配置
|------|----------------|------|
| Reddit(含评论)+ HN + Polymarket + GitHub + StockTwits | 无 | 免费 |
| arXiv + Techmeme | 免费 CLI,由首次配置自动安装 | 免费 |
-| X / Twitter | 在任意浏览器中登录 x.com,或设置 `XQUIK_API_KEY` / `XAI_API_KEY` | 浏览器 Cookie 免费;密钥费用取决于服务商 |
+| X / Twitter | 设置 `X_BEARER_TOKEN` 使用官方 X API(近期帖子,大约最近一周;除非你的 X 开发者项目拥有完整归档访问权限。Grok Bot 上为默认方式,其他宿主需通过 `LAST30DAYS_X_BACKEND=xapi` 选择启用),或在任意浏览器中登录 x.com,或设置 `XQUIK_API_KEY` / `XAI_API_KEY` | X API 额度来自你自己的 X 开发者项目;浏览器 Cookie 免费;其他密钥费用取决于服务商 |
| YouTube | `brew install yt-dlp` | 免费 |
| Bluesky | 来自 bsky.app 的应用密码 | 免费 |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + YouTube 评论 | ScrapeCreators 密钥 | 前 10,000 次调用免费,之后按量付费 |
diff --git a/changelog.d/+grok-bot-official-x-api.changed.md b/changelog.d/+grok-bot-official-x-api.changed.md
new file mode 100644
index 00000000..4e787eae
--- /dev/null
+++ b/changelog.d/+grok-bot-official-x-api.changed.md
@@ -0,0 +1 @@
+On a Grok Bot host (`LAST30DAYS_HOST=grok-bot`) X search now runs through the bot's X connector first, with the official X API (`X_BEARER_TOKEN`) and xAI's licensed X search (`XAI_API_KEY`) as backups; onboarding, doctor, and repair hints there name only those paths. Credit-exhaustion responses from any source (HTTP 402 or an explicit "insufficient credits" message) now classify as the `payment-required` outcome instead of `error` or `auth-failed`, and `LAST30DAYS_STRICT_EXIT` treats it as degraded (exit `3`).
diff --git a/changelog.d/+x-api-backend-and-host-lane.added.md b/changelog.d/+x-api-backend-and-host-lane.added.md
new file mode 100644
index 00000000..3ed8a938
--- /dev/null
+++ b/changelog.d/+x-api-backend-and-host-lane.added.md
@@ -0,0 +1 @@
+New `xapi` X backend: the official X API v2 with an app-only `X_BEARER_TOKEN`, trying full-archive search first and falling back to recent search (about the last seven days, reported as "window truncated to 7 days"), serving the topic lane and the from/mention/related handle lanes; it is the first rung on Grok Bot and opt-in elsewhere (`LAST30DAYS_X_BACKEND=xapi`). New `--x-posts <path>` flag ingests a `last30days-x-posts/1` envelope of posts the hosting model fetched through its X connector (per-entity `x_posts` in `--competitors-plan`), and `LAST30DAYS_X_HOST_LANE=1` marks X as available for planning while a connector is in the session. New `setup --store-key <NAME>` persists one credential from stdin without echoing it. New `payment-required` source outcome, exported under agent JSON schema `1.3`.
diff --git a/docs/reference/json-export.md b/docs/reference/json-export.md
index 7e7d108a..b6c4711b 100644
--- a/docs/reference/json-export.md
+++ b/docs/reference/json-export.md
@@ -41,7 +41,7 @@ When `LAST30DAYS_API_KEY` and `LAST30DAYS_API_BASE` route a run through a config
| Field | Type | Meaning |
| --- | --- | --- |
-| `schema_version` | string | Agent export contract version. The current version is `1.2`. |
+| `schema_version` | string | Agent export contract version. The current version is `1.3`. |
| `query` | string | The research topic supplied to the engine. |
| `generated_at` | string | UTC generation timestamp in RFC 3339 format. |
| `window_days` | integer | Number of days between the report's start and end dates. |
@@ -67,6 +67,7 @@ Each value distinguishes a clean empty result from incomplete coverage:
| `partial` | The source returned some items before a later failure. |
| `rate-limited` | Retrieval was stopped by a provider rate limit. |
| `auth-failed` | Credentials were missing, rejected, or expired during retrieval. |
+| `payment-required` | The provider rejected the request because the account's credits are exhausted (HTTP 402 or an explicit credit-exhaustion message). Fix is to top up, not to re-authenticate. Added in `1.3`. |
| `unreachable` | The source or network endpoint could not be reached. |
| `timeout` | Retrieval exceeded its time limit. |
| `schema-drift` | The provider response no longer matched the expected shape. |
@@ -108,12 +109,12 @@ Comparison queries use an envelope so each entity keeps its own contract:
```json
{
- "schema_version": "1.2",
+ "schema_version": "1.3",
"comparison": true,
"entities": ["OpenAI", "Anthropic"],
"reports": [
- {"entity": "OpenAI", "report": {"schema_version": "1.2", "query": "OpenAI"}},
- {"entity": "Anthropic", "report": {"schema_version": "1.2", "query": "Anthropic"}}
+ {"entity": "OpenAI", "report": {"schema_version": "1.3", "query": "OpenAI"}},
+ {"entity": "Anthropic", "report": {"schema_version": "1.3", "query": "Anthropic"}}
]
}
```
@@ -126,6 +127,7 @@ The abbreviated reports above only illustrate the envelope; real reports contain
- Any breaking field removal, rename, type change, semantic change, or envelope change requires a major-version bump.
- Backward-compatible field additions may use a minor-version bump. Consumers should ignore fields they do not recognize.
- The checked-in golden snapshot test locks the complete current shape. Contract changes must update the version and snapshot deliberately.
+- `1.3` added the `payment-required` value to `source_status` for credit exhaustion (HTTP 402). Backward-compatible minor bump: consumers that switch on state names should treat it like `auth-failed` (a failure state, not evidence of silence) until they add a branch for it. Messages that previously classified as `auth-failed` or `error` because of a 402 or an "insufficient credits" body now carry this state on every source.
- `1.2` added `candidate_id` to each `results` entry so verdicts can be joined to the result they annotate.
- Discovery `1.1` added `podcast_angle`, `x_article_angle`, `previously_surfaced_count`, `last_surfaced`, and `covered` to each discovery `results` entry — a backward-compatible minor bump; the fields carry their defaults (`null`/`null`/`0`/`null`/`false`) until an angle generator or the topic queue populates them.
- `--json-profile=raw` is outside this compatibility policy because it mirrors internal pipeline dataclasses.
diff --git a/hooks/scripts/check-config.sh b/hooks/scripts/check-config.sh
index 34682489..b967223e 100755
--- a/hooks/scripts/check-config.sh
+++ b/hooks/scripts/check-config.sh
@@ -175,7 +175,7 @@ load_keychain_presence() {
fi
[[ -n "$user" ]] || return 0
- for key in SETUP_COMPLETE OPENAI_API_KEY SCRAPECREATORS_API_KEY AUTH_TOKEN CT0 XAI_API_KEY BSKY_HANDLE EXA_API_KEY; do
+ for key in SETUP_COMPLETE OPENAI_API_KEY SCRAPECREATORS_API_KEY AUTH_TOKEN CT0 XAI_API_KEY X_BEARER_TOKEN BSKY_HANDLE EXA_API_KEY; do
env_var="ENV_${key}"
current="${!env_var:-}"
if [[ -z "$current" ]]; then
@@ -274,6 +274,11 @@ HAS_X=""
if [[ -n "${ENV_AUTH_TOKEN:-${AUTH_TOKEN:-}}" && -n "${ENV_CT0:-${CT0:-}}" ]]; then
HAS_X="yes"
fi
+# X API v2 bearer (official X API backend) is a third way to have X; read the
+# same way XAI_API_KEY is (file first, then process env), presence only.
+if [[ -n "${ENV_X_BEARER_TOKEN:-${X_BEARER_TOKEN:-}}" ]]; then
+ HAS_X="yes"
+fi
HAS_XAI="${ENV_XAI_API_KEY:-${XAI_API_KEY:-}}"
HAS_BSKY="${ENV_BSKY_HANDLE:-${BSKY_HANDLE:-}}"
HAS_EXA="${ENV_EXA_API_KEY:-${EXA_API_KEY:-}}"
diff --git a/mcp/manifest.json b/mcp/manifest.json
index 46d73058..268e43b2 100644
--- a/mcp/manifest.json
+++ b/mcp/manifest.json
@@ -44,7 +44,8 @@
"BSKY_APP_PASSWORD": "${user_config.bsky_app_password}",
"PARALLEL_API_KEY": "${user_config.parallel_api_key}",
"SCRAPECREATORS_API_KEY": "${user_config.scrapecreators_api_key}",
- "OPENROUTER_API_KEY": "${user_config.openrouter_api_key}"
+ "OPENROUTER_API_KEY": "${user_config.openrouter_api_key}",
+ "X_BEARER_TOKEN": "${user_config.x_bearer_token}"
}
}
},
@@ -139,6 +140,13 @@
"description": "OpenRouter API key. Alternative LLM provider gateway for synthesis. Get one at https://openrouter.ai/keys.",
"sensitive": true,
"required": false
+ },
+ "x_bearer_token": {
+ "type": "string",
+ "title": "X_BEARER_TOKEN",
+ "description": "X API v2 app-only bearer token. Searches X through the official X API (recent posts, about the last week, unless your X developer project has full-archive access). Create an app in the X developer console at https://developer.x.com/ and copy its bearer token.",
+ "sensitive": true,
+ "required": false
}
},
"compatibility": {
diff --git a/skills/last30days/SKILL.md b/skills/last30days/SKILL.md
index 702805a8..617f79f3 100644
--- a/skills/last30days/SKILL.md
+++ b/skills/last30days/SKILL.md
@@ -18,6 +18,7 @@ metadata:
- SCRAPECREATORS_API_KEY
- OPENAI_API_KEY
- XAI_API_KEY
+ - X_BEARER_TOKEN
- OPENROUTER_API_KEY
- PERPLEXITY_API_KEY
- PARALLEL_API_KEY
@@ -294,6 +295,8 @@ Resolving this correctly prevents the second-most-common failure mode of this sk
After resolving host web search, run the first-run gate below before anything else.
+**GROK BOT HOST RULE (every invocation, including installs that already have `SETUP_COMPLETE=true`).** If you are running as a Grok Bot (xAI's Grok Bot; do not key this on `CURSOR_AGENT` alone, which Cursor agent chat also sets), export `LAST30DAYS_HOST=grok-bot` in every shell that runs `last30days.py`, the way `LAST30DAYS_NATIVE_SEARCH` is exported. If this session also exposes an X post-search tool from the X connector (the "X for Grok Bot" plugin: search posts, read timelines, check mentions; for example `search_posts_all`), export `LAST30DAYS_X_HOST_LANE=1` in that same shell and follow the X connector recipe in Research Execution (you fetch X through the connector; the engine ingests it via `--x-posts <file>`). On a Grok Bot never run any browser-cookie step, and never place post text unquoted in a shell command: the envelope is written only through a single-quoted heredoc delimiter (`<<'EOF'`) or the tool's own file output. A Grok Bot first run takes the **Grok Bot Prose Flow** in Step 0.
+
**FIRST-RUN GATE — run this Bash command immediately after resolving host web search, before reading the topic or doing any research:**
```bash
@@ -414,7 +417,7 @@ If your Bash call to `last30days.py` does NOT include the FULL pre-flight checkl
# last30days v3.23.1: Research Any Topic from the Last 30 Days
-> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`). X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). On hosts with `uv` and no Python 3.12+, the preflight may install a uv-managed CPython 3.12 (one-time ~28MB download, announced on stderr). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
+> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`). X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars), an X API v2 app-only bearer (X_BEARER_TOKEN, sent only to api.x.com), or a host-provided `--x-posts` envelope the hosting model fetched through its own X connector. Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). On hosts with `uv` and no Python 3.12+, the preflight may install a uv-managed CPython 3.12 (one-time ~28MB download, announced on stderr). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
Research ANY topic across Reddit, X, YouTube, and other sources. Surface what people are actually discussing, recommending, betting on, and debating right now.
@@ -552,6 +555,7 @@ When both `LAST30DAYS_API_KEY` and `LAST30DAYS_API_BASE` are set, the engine run
**Platform split - run exactly ONE branch:**
- **If you HAVE WebSearch and AskUserQuestion (Claude Code):** run the **Claude Code Modal Flow** immediately below.
- **If you do NOT (OpenClaw, Codex, Cursor, Gemini CLI, raw CLI):** run the **Non-Modal Prose Flow** further down. It does the same work conversationally, without modals.
+- **If you are running as a Grok Bot** (the Grok Bot host rule in HOW TO INVOKE): run the **Grok Bot Prose Flow**, the third branch, below the Non-Modal flow. The X connector comes first, any backup key is written only through the engine, and no browser session is read. Cursor stays in the Non-Modal Prose Flow.
---
@@ -590,7 +594,7 @@ Options (give each option the description shown):
The consented `setup --allow-browser-cookies` run extracts cookies (Chrome/Chromium family first via the Keychain with no Full Disk Access, then Firefox and Safari as fallbacks; the winning browser is pinned for future runs only when it is Firefox or Safari, so Chrome never re-triggers the Keychain prompt on later runs) and best-effort installs yt-dlp (YouTube), the free keyless Digg CLI (`digg-pp-cli` via `@mvanhorn/printing-press-library install digg --cli-only`; Digg activates only when the binary is on the **agent subprocess PATH**, typically `$HOME/.local/bin`; setup reports honestly if installed off-PATH; recommend-only if `npx` is unavailable), plus the free keyless arXiv and Techmeme CLIs. Show the user what was found and installed - including whether Digg landed on PATH (active) or off-PATH (installed but not yet active).
-**Extras-host X login (Linux / Grok Bot / Mac mini / Darwin agentcookie sink — a MacBook SKIPS this).** On a MacBook the Keychain/Firefox/Safari extract above is all X needs. On an extras host the local Chrome store can't be decrypted, so that extract finds nothing and X stays empty unless you also capture a LIVE login over CDP. After the cookie-consent setup run: run `"${LAST30DAYS_PYTHON:-python3}" skills/last30days/scripts/box_chrome_login.py` — it prints the exact host-correct command (or launches it with `--exec`), and on a MacBook it prints "no launch needed" and spawns nothing. When `box-chrome` is on PATH it launches a throwaway profile on the last30days extras port **18800** (the last30days convention `SAND_CHROME_REMOTE_DEBUG_PORT=18800`, not box-chrome's default): `CHROME_USER_DATA_DIR=/tmp/last30days-x-chrome SAND_CHROME_REMOTE_DEBUG_PORT=18800 box-chrome --new-window https://x.com/login`. Wait for the x.com login page, then HAND THE DESKTOP to the human to type — do NOT fill or drive the form. After they sign in, append `BROWSER_CDP_URL=http://127.0.0.1:18800` to `.env` (never `AUTH_TOKEN`/`CT0`; keep `AGENTCOOKIE=off` during the harvest), then re-run `setup --allow-browser-cookies` so extras CDP reads the live pair. Full steps and the block/rate-limit stop rule are in **X on Linux / Grok Bot / Mac mini** below. This extras-host login only runs on the consented "Yes - X cookies" path; the Research Continuation Override never skips it when the user said yes to X.
+**Extras-host X login (Linux / Mac mini / Darwin agentcookie sink — a MacBook SKIPS this, and so does a `LAST30DAYS_HOST=grok-bot` host, which never runs it).** On a MacBook the Keychain/Firefox/Safari extract above is all X needs. On an extras host the local Chrome store can't be decrypted, so that extract finds nothing and X stays empty unless you also capture a LIVE login over CDP. After the cookie-consent setup run: run `"${LAST30DAYS_PYTHON:-python3}" skills/last30days/scripts/box_chrome_login.py` — it prints the exact host-correct command (or launches it with `--exec`), and on a MacBook it prints "no launch needed" and spawns nothing. When `box-chrome` is on PATH it launches a throwaway profile on the last30days extras port **18800** (the last30days convention `SAND_CHROME_REMOTE_DEBUG_PORT=18800`, not box-chrome's default): `CHROME_USER_DATA_DIR=/tmp/last30days-x-chrome SAND_CHROME_REMOTE_DEBUG_PORT=18800 box-chrome --new-window https://x.com/login`. Wait for the x.com login page, then HAND THE DESKTOP to the human to type — do NOT fill or drive the form. After they sign in, append `BROWSER_CDP_URL=http://127.0.0.1:18800` to `.env` (never `AUTH_TOKEN`/`CT0`; keep `AGENTCOOKIE=off` during the harvest), then re-run `setup --allow-browser-cookies` so extras CDP reads the live pair. Full steps and the block/rate-limit stop rule are in **X on Linux / Mac mini** below. This extras-host login only runs on the consented "Yes - X cookies" path; the Research Continuation Override never skips it when the user said yes to X.
**macOS Full Disk Access remediation (Safari fallback only).** Chrome and Firefox need no Full Disk Access; only the Safari fallback does. After the `setup` run, inspect its stderr. If it contains `Permission denied reading Cookies.binarycookies` and the platform is macOS, the OS blocked the Safari read - surface the fix instead of swallowing it: `macOS blocked the Safari cookie read. If your x.com login is in Chrome, you don't need this. To use Safari: System Settings > Privacy & Security > Full Disk Access > enable your terminal (or the Claude app), then I can retry.` Offer ONE retry only when no research topic is waiting. If a topic is already waiting or the user skips, continue immediately with available sources.
@@ -655,7 +659,7 @@ For hosts without interactive modal prompts (OpenClaw, Codex, Cursor, Gemini CLI
**3. Cookie consent (ask BEFORE reading anything).** First check if `BROWSER_CONSENT=true` already exists in `~/.config/last30days/.env` (e.g. granted in a prior Claude Code session); if so, skip this prompt and run `setup --allow-browser-cookies` directly. Otherwise ask. Example: `I can read your browser cookies to unlock X/Twitter and other logged-in sources - I check Chrome first (a one-time macOS Keychain prompt may appear; click Always Allow), then Firefox and Safari. Want me to? (yes / no)` **Wait for the answer.**
- On **yes** → run `"${LAST30DAYS_PYTHON:-python3}" skills/last30days/scripts/last30days.py setup --allow-browser-cookies` (and append `BROWSER_CONSENT=true` to `.env` after it completes). Extracts cookies (Chrome/Chromium family first via the Keychain with no Full Disk Access, then Firefox and Safari; only a Firefox/Safari winner is pinned for later runs, so Chrome never re-prompts) and best-effort installs yt-dlp (YouTube), the free keyless Digg CLI (`digg-pp-cli` via `@mvanhorn/printing-press-library install digg --cli-only`; activates only when on the agent subprocess PATH, typically `$HOME/.local/bin`; reports honestly if off-PATH; recommend-only if `npx` is unavailable), plus the free keyless arXiv and Techmeme CLIs.
- - **Extras hosts (Linux / Grok Bot / Mac mini / Darwin agentcookie sink) — a MacBook SKIPS this.** On these hosts the Chrome cookie store can't be decrypted, so the extract above finds nothing and X stays empty unless you capture a LIVE login over CDP. Run `"${LAST30DAYS_PYTHON:-python3}" skills/last30days/scripts/box_chrome_login.py` (prints the host-correct command; `--exec` launches it; a MacBook prints "no launch needed" and spawns nothing). When `box-chrome` is on PATH it launches a throwaway profile on the last30days extras port **18800** (`SAND_CHROME_REMOTE_DEBUG_PORT=18800`, not box-chrome's default): `CHROME_USER_DATA_DIR=/tmp/last30days-x-chrome SAND_CHROME_REMOTE_DEBUG_PORT=18800 box-chrome --new-window https://x.com/login`. Wait for the x.com login page, then HAND THE DESKTOP to the human to type — do NOT drive the form. After they sign in, append `BROWSER_CDP_URL=http://127.0.0.1:18800` to `.env` (never `AUTH_TOKEN`/`CT0`; keep `AGENTCOOKIE=off` during the harvest) and re-run `setup --allow-browser-cookies`. Full steps and the block/rate-limit stop rule: **X on Linux / Grok Bot / Mac mini** below. This extras-host login only runs on the consented **yes** path; a waiting topic never skips it when the user said yes to X.
+ - **Extras hosts (Linux / Mac mini / Darwin agentcookie sink) — a MacBook SKIPS this, and so does a `LAST30DAYS_HOST=grok-bot` host.** On these hosts the Chrome cookie store can't be decrypted, so the extract above finds nothing and X stays empty unless you capture a LIVE login over CDP. Run `"${LAST30DAYS_PYTHON:-python3}" skills/last30days/scripts/box_chrome_login.py` (prints the host-correct command; `--exec` launches it; a MacBook prints "no launch needed" and spawns nothing). When `box-chrome` is on PATH it launches a throwaway profile on the last30days extras port **18800** (`SAND_CHROME_REMOTE_DEBUG_PORT=18800`, not box-chrome's default): `CHROME_USER_DATA_DIR=/tmp/last30days-x-chrome SAND_CHROME_REMOTE_DEBUG_PORT=18800 box-chrome --new-window https://x.com/login`. Wait for the x.com login page, then HAND THE DESKTOP to the human to type — do NOT drive the form. After they sign in, append `BROWSER_CDP_URL=http://127.0.0.1:18800` to `.env` (never `AUTH_TOKEN`/`CT0`; keep `AGENTCOOKIE=off` during the harvest) and re-run `setup --allow-browser-cookies`. Full steps and the block/rate-limit stop rule: **X on Linux / Mac mini** below. This extras-host login only runs on the consented **yes** path; a waiting topic never skips it when the user said yes to X.
- On **no** → run `FROM_BROWSER=off "${LAST30DAYS_PYTHON:-python3}" skills/last30days/scripts/last30days.py setup`. Skips all cookie reads; still installs yt-dlp (YouTube), Digg, arXiv, and Techmeme, still writes `SETUP_COMPLETE`. If the invocation already includes a topic, immediately research it with `--no-browser-cookies`, then resume the deferred onboarding in the same run after the findings: the ScrapeCreators offer (step 5) and source tier (step 5b) if a key is saved. Do not re-ask cookie consent as part of the resume.
**4. Full Disk Access remediation (macOS only).** After `setup`, inspect stderr. If it contains `Permission denied reading Cookies.binarycookies` on macOS, surface: `macOS blocked the cookie read. To enable X/Twitter: System Settings > Privacy & Security > Full Disk Access > enable your terminal (or the Claude app), then I can retry.` Offer ONE retry only when no research topic is waiting. If a topic is already waiting or the user skips, continue immediately with available sources.
@@ -675,6 +679,38 @@ For hosts without interactive modal prompts (OpenClaw, Codex, Cursor, Gemini CLI
---
+### Grok Bot Prose Flow
+
+For a Grok Bot host. The Grok Bot host rule in HOW TO INVOKE applies throughout: `LAST30DAYS_HOST=grok-bot` is exported in every shell that runs the engine, including every command below. Same conversational shape as the Non-Modal Prose Flow, no modals, and the X connector comes first: on this host the engine never reads a browser session and X runs only through official access. Run in order; wait where it says to wait.
+
+**1. Host key + welcome.** Persist the host signal so later runs and `doctor` see it: if `~/.config/last30days/.env` is missing, `mkdir -p ~/.config/last30days && touch ~/.config/last30days/.env`; then append one line `LAST30DAYS_HOST=grok-bot` (append-only with `>>`; never `>`). Then run `"${LAST30DAYS_PYTHON:-python3}" "${SKILL_DIR}/scripts/last30days.py" --welcome` and show its stdout VERBATIM.
+
+**2. Permission preflight.** Run `"${LAST30DAYS_PYTHON:-python3}" "${SKILL_DIR}/scripts/last30days.py" --preflight` and summarize the human-readable result (config source, planned writes, optional commands). It reads nothing from a browser and runs no research.
+
+**3. X connector (primary - check this BEFORE offering any key).** Look for an X post-search tool from the X connector in this session: the "X for Grok Bot" plugin (search posts, read timelines, check mentions), for example `search_posts_all`. Any of its search tools counts; use the one that searches posts by query.
+ - **Present** → tell the user: `X search runs through your X connector on the credits included with Grok Bot, with full 30-day coverage - nothing to configure.` Export `LAST30DAYS_X_HOST_LANE=1` next to `LAST30DAYS_HOST=grok-bot` in every engine shell for this session, and on every research run follow the X connector recipe in Research Execution: one `topic` call sized 10 / 30 / 60 by depth (`--quick` / default / `--deep`) with `-is:retweet` and the window; per `--x-handle`, a `from` call of 8 and a `mention` call of 5; per `--x-related` handle, a `related` call of 3; written to a `last30days-x-posts/1` envelope (`generated_at`, `topic`, `window`, `status`, `calls` tagged `topic` / `from` / `mention` / `related`, each post exactly `id`, `author_handle`, `created_at`, `text`, `likes`, `reposts`, `replies`, `quotes` and nothing else); if the tool rejects the window or count parameters, omit them and write `status: partial` with `error: window-unsupported`; pass the file as `--x-posts <file>`. Skip step 4.
+ - **Absent** → continue to step 4.
+
+**4. Backup key (only when the connector is absent).** Say, then WAIT: `This session has no X connector, so X needs a key. Best fix: add the "X for Grok Bot" plugin and connect X inside Grok Bot (it provisions an X developer account for you, with credits included). Otherwise paste an X_BEARER_TOKEN from the X developer console - recent posts, about the last week, unless your X developer project has full-archive access - or an XAI_API_KEY from console.x.ai. Or skip X for now. (bearer / xai / skip)`
+ - On a pasted key → persist it ONLY through the engine's key-write path, feeding stdin from a single-quoted heredoc. Never an ad-hoc shell write of the value, never interpolate it into a command line:
+
+ ```bash
+ "${LAST30DAYS_PYTHON:-python3}" "${SKILL_DIR}/scripts/last30days.py" setup --store-key X_BEARER_TOKEN <<'KEY_EOF'
+ {PASTED_VALUE}
+ KEY_EOF
+ ```
+
+ Use `setup --store-key XAI_API_KEY` for an xAI key. The engine prints `X_BEARER_TOKEN=****` (or `XAI_API_KEY=****`) plus a JSON `persisted` line; never echo the value back, and confirm only in the masked `NAME=****` form. Running it again with a new value replaces the stored one (that is how a rejected key is rotated). A `"persisted": false` means the write failed: say so and do not claim X is active.
+ - On **skip** → append `X_DECLINED=grok-bot` to `~/.config/last30days/.env` (append-only) so later runs stay quiet about X: no unlock pitch, no second key question. If the invocation already includes a topic, research it right after step 5 and resume steps 6-7 after the findings.
+
+**5. Setup (free CLIs, no browser reads).** Run `"${LAST30DAYS_PYTHON:-python3}" "${SKILL_DIR}/scripts/last30days.py" setup` (the plain form; on this host it reads nothing from a browser). It best-effort installs yt-dlp (YouTube), the Digg CLI, arXiv, and Techmeme and writes `SETUP_COMPLETE=true`. Show what was installed, including whether Digg landed on PATH.
+
+**6. ScrapeCreators offer and source tier.** Run steps 5 and 5b of the Non-Modal Prose Flow exactly as written there (GitHub device-code signup with `setup --github-start` then `setup --github-poll`; the engine persists the key and masks it).
+
+**7. Complete.** Confirm `SETUP_COMPLETE=true` is in `~/.config/last30days/.env` (append it if `setup` did not run), briefly confirm which sources are active (re-run `--preflight`, or safe `--diagnose`, with the host signal and, when the connector is present, `LAST30DAYS_X_HOST_LANE=1` exported; `--diagnose` then lists `x` as served by the connector), and proceed to research.
+
+---
+
### Manual Setup Guide
Shown when a Claude Code user picks "Manual setup", or for anyone who wants to configure by hand. Present as plain text (not blockquoted).
@@ -682,15 +718,18 @@ Shown when a Claude Code user picks "Manual setup", or for anyone who wants to c
The magic of /last30days is Reddit comments + X posts together - and both are free. Add these to `~/.config/last30days/.env`:
**X/Twitter (pick one - the most important source):**
+- `X_BEARER_TOKEN=xxx` - the official X API v2 (`api.x.com`) with an app-only bearer from the X developer console. Recent posts, about a week back on the Basic tier, unless your X developer project has full-archive access. Persist it with `setup --store-key X_BEARER_TOKEN` (value on stdin, masked in output). Outside a Grok Bot host also add `LAST30DAYS_X_BACKEND=xapi` so the engine selects it (`doctor` says so when the bearer is set without it).
- **Grok CLI (no X credential):** install with `curl -fsSL https://x.ai/cli/install.sh | bash`, then `grok login`. No X account, no cookies, no API key. Needs a Grok plan; calls draw on it.
- `FROM_BROWSER=auto` - free. Reads your x.com login cookies live at search time (Firefox/Safari, never saved to disk).
- `XAI_API_KEY=xxx` - no browser access needed. Get a key at api.x.ai. Best for servers.
- `XQUIK_API_KEY=xxx` - keyless-style X via Xquik.
- `AUTH_TOKEN=xxx` + `CT0=xxx` - paste your X cookies manually (x.com → F12 → Application → Cookies).
-**X on Linux / Grok Bot / Mac mini (repair).** These hosts can't decrypt a local Chrome cookie store, so if X returns nothing there, feed bird a cookie pair one of these ways (a MacBook does NOT do any of this — it uses its Keychain / Firefox / Safari extract; do not launch box-chrome on a MacBook):
+**X on a Grok Bot (repair).** On a `LAST30DAYS_HOST=grok-bot` host X runs only through official access, so if X returns nothing there: add the "X for Grok Bot" plugin and connect X inside Grok Bot (the X connector lane; full 30-day coverage on the included credits), or add `X_BEARER_TOKEN` (recent posts, about the last week, unless your X developer project has full-archive access) or `XAI_API_KEY` from console.x.ai through `setup --store-key <NAME>`, or, when the footer reports X API credits exhausted (`payment-required`), top up credits in the X developer console. Nothing in the Linux / Mac mini repair section below applies on a Grok Bot.
+
+**X on Linux / Mac mini (repair).** These hosts (never a `LAST30DAYS_HOST=grok-bot` host) can't decrypt a local Chrome cookie store, so if X returns nothing there, feed bird a cookie pair one of these ways (a MacBook does NOT do any of this — it uses its Keychain / Firefox / Safari extract; do not launch box-chrome on a MacBook):
- **agentcookie sidecar:** install the `agentcookie` CLI so the engine can read your `auth_token`/`ct0` from it automatically. Nothing to configure; `AGENTCOOKIE=off` disables it.
-- **Live Chrome login over CDP (the proven Grok Bot path).** The engine reads a live signed-in Chrome over the DevTools Protocol, but you must LAUNCH that Chrome on the extras port and log in first — `setup --allow-browser-cookies` alone opens no window. Steps (a MacBook skips all of this):
+- **Live Chrome login over CDP (the proven extras-host path).** The engine reads a live signed-in Chrome over the DevTools Protocol, but you must LAUNCH that Chrome on the extras port and log in first — `setup --allow-browser-cookies` alone opens no window. Steps (a MacBook skips all of this):
1. Set `AGENTCOOKIE=off` for this harvest so a sidecar can't mix in a different pair (leave it unset again after success).
2. Launch a throwaway login Chrome on the last30days extras port **18800** — this port is the last30days convention (`SAND_CHROME_REMOTE_DEBUG_PORT=18800`), NOT box-chrome's built-in default (`9222` + the display number). **Launch via the host `box-chrome` wrapper** (which sets `--class=box-chrome`); do NOT launch raw `google-chrome-stable`, and in particular do NOT launch raw Chrome with a custom `--class` — a raw Chrome with `--class=l30d-…` failed where `box-chrome` (class `box-chrome`) succeeded. Do NOT rely on any `GrokAgent` user-agent token: it is not required and may be disabled on the host (`/tmp/sand-ua-token-disabled`), so never tell users it must be present. Run `"${LAST30DAYS_PYTHON:-python3}" skills/last30days/scripts/box_chrome_login.py` to print the exact host-correct command (add `--exec` to launch it); on a MacBook it prints "no launch needed" and spawns nothing. When `box-chrome` is on PATH the command is:
```
@@ -1461,6 +1500,51 @@ Store your plan as `QUERY_PLAN_JSON` - you'll pass it to the script in the next
---
+**Grok Bot X connector recipe (only when `LAST30DAYS_X_HOST_LANE=1` is exported - see the Grok Bot host rule in HOW TO INVOKE).** On a Grok Bot with the X connector, YOU fetch X through the connector before the engine command and hand the engine the file; the engine then calls no X backend, plans `x` in, and the footer's X provenance reads "X via X connector". Do this before the Step 1 command below.
+
+1. **Calls (the connector's post-search tool, e.g. `search_posts_all`).** Window = the engine's date range (`--days`, default 30: `from` is today minus the day count, `to` is today). Depth count = 10 (`--quick`) / 30 (default) / 60 (`--deep`).
+ - One `topic` call: the topic query plus `-is:retweet`, the window, and the depth count.
+ - Per `--x-handle` handle: one `from` call (`from:<handle> -is:retweet`, 8 posts) and one `mention` call (`@<handle> -is:retweet`, 5 posts).
+ - Per `--x-related` handle: one `related` call (`from:<handle> -is:retweet`, 3 posts).
+ - If the tool rejects the window or count parameters, omit them, keep at most the depth count per call, and write `"status": "partial"` with `"error": "window-unsupported"`.
+ - If the connector itself fails, write `"status": "error"` with `"calls": []` and a short category in `error`: `credits`, `not-connected`, or `unavailable` - never raw tool output, never an account or app id.
+2. **Envelope.** Exactly these top-level fields; every post carries the eight flat fields and nothing else (no URLs, media, or author objects); at most the depth count per call. A fresh `generated_at` (the engine rejects an envelope older than 6 hours) and a `topic` identical to the engine's topic string:
+
+```json
+{
+ "schema": "last30days-x-posts/1",
+ "generated_at": "{ISO_8601_UTC_NOW}",
+ "topic": "{TOPIC}",
+ "window": {"from": "{YYYY-MM-DD}", "to": "{YYYY-MM-DD}"},
+ "provider": "x-connector",
+ "status": "ok",
+ "calls": [
+ {"lane": "topic", "handles": [], "posts": [
+ {"id": "1963000000000000000", "author_handle": "someone", "created_at": "2026-09-07T10:00:00Z", "text": "post text", "likes": 12, "reposts": 3, "replies": 1, "quotes": 0}
+ ]},
+ {"lane": "from", "handles": ["{RESOLVED_HANDLE}"], "posts": []},
+ {"lane": "mention", "handles": ["{RESOLVED_HANDLE}"], "posts": []},
+ {"lane": "related", "handles": ["{RELATED_HANDLE}"], "posts": []}
+ ]
+}
+```
+
+ Omit the `from` / `mention` / `related` calls when the run has no `--x-handle` / `--x-related`; `handles` must be the run's own handles. `id` is the post's numeric id as a string; `author_handle` is the username without `@`.
+3. **Write the file - post text is attacker-controlled and never goes unquoted into a shell command.** Use the tool's own file output when it has one; otherwise a single-quoted heredoc (never unquoted) into a `.json` path outside `~/.config`, in the SAME Bash call as the engine command (the trap removes it on exit). Two rules keep a post from closing the heredoc early: emit the envelope as ONE line of compact JSON (newlines inside post text stay escaped as `\n`; never pretty-print), and replace `{X_POSTS_NONCE}` in BOTH sentinel lines with 12 random letters and digits you generate fresh for this run, so no post text can equal the closing line:
+
+```bash
+X_POSTS_DIR=$(mktemp -d "${TMPDIR:-/tmp}/last30days-x-posts.XXXXXX")
+X_POSTS_FILE="$X_POSTS_DIR/x-posts.json"
+trap 'rm -rf "$X_POSTS_DIR"' EXIT
+cat >| "$X_POSTS_FILE" <<'X_POSTS_EOF_{X_POSTS_NONCE}'
+{X_POSTS_ENVELOPE_JSON}
+X_POSTS_EOF_{X_POSTS_NONCE}
+```
+
+ Run it directly in your shell tool, never wrapped in `bash -lc '...'` (same rule as the plan tmpfile).
+4. **Engine.** Add `--x-posts "$X_POSTS_FILE"` to the Step 1 command (the file path only, never inline JSON); every other flag stays as usual. Comparison runs: write one envelope per entity (its `topic` is that entity's name) and put the path in that entity's `--competitors-plan` entry as `"x_posts": "/abs/path/x-posts.json"`; a bare `--x-posts` on a comparison run exits 2. If the engine exits 2 naming the envelope, fix the file it names or re-run without `--x-posts`; X is then absent for that run.
+5. **After the run.** The stats line reads "X via X connector". The engine's one receipt line (accepted / dropped counts, on stderr) is diagnostics only: never narrate it in the deliverable (LAW 9).
+
**Step 1: Run the research script WITH your query plan (FOREGROUND)**
**CRITICAL: Run this command in the FOREGROUND with a 5-minute timeout. Do NOT use run_in_background. The full output contains Reddit, X, AND YouTube data that you need to read completely.**
@@ -2029,7 +2113,7 @@ Headlines should be specific and newsy ("BULLY dropped and it's dominating", "Eu
If the research output contains a `**🔍 Research Coverage:**` block, render it verbatim right before the stats block. This tells the user which core sources are missing and how to unlock them. Do NOT render this block if it is absent from the output (100% coverage = no nudge).
-**Optional X omission:** If X was unavailable because no X authentication was configured, finish the useful findings first. The engine emits one short, non-blocking note: `Optional source omitted: X/Twitter was not enabled; research continued with the available sources.` Render that note once if present. Do not repeat it. Do not open a modal, ask another question, recommend a login, or provide cookie/API setup instructions unless the user explicitly asks to enable X.
+**Optional X omission:** If X was unavailable because no X authentication was configured, finish the useful findings first. The engine emits one short, non-blocking note: `Optional source omitted: X/Twitter was not enabled; research continued with the available sources.` Render that note once if present. Do not repeat it. Do not open a modal, ask another question, recommend a login, or provide cookie/API setup instructions unless the user explicitly asks to enable X. On a Grok Bot host the engine's own X hint (connect X inside Grok Bot, a bearer token, or a credit top-up) is the only guidance; relay it once, unchanged, and the note does not fire at all when the X connector lane served the run.
**THEN - Engine footer pass-through (right before invitation):**
@@ -2278,7 +2362,8 @@ Want another prompt? Just tell me what you're creating next.
**What this skill does:**
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, and as a Reddit search backup when the free Reddit path returns no items (requires SCRAPECREATORS_API_KEY; empty-only by default — see `LAST30DAYS_REDDIT_SC_MIN_ITEMS` / `LAST30DAYS_REDDIT_BACKEND`)
- Legacy: Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery (fallback if no SCRAPECREATORS_API_KEY)
-- Sends search queries to X/Twitter via optional user-provided `AUTH_TOKEN`/`CT0` env vars, explicit browser-cookie opt-in (`FROM_BROWSER` or setup consent), xAI's API (`api.x.ai` by default), Xquik's API (`xquik.com` by default), or the official X API v2 via xurl CLI (OAuth2, auto-detected when installed and authenticated)
+- Sends search queries to X/Twitter via the official X API v2 (`api.x.com`, app-only bearer from `X_BEARER_TOKEN`, sent only in the Authorization header; the engine's default X path on a Grok Bot host next to xAI's API), optional user-provided `AUTH_TOKEN`/`CT0` env vars, explicit browser-cookie opt-in (`FROM_BROWSER` or setup consent), xAI's API (`api.x.ai` by default), Xquik's API (`xquik.com` by default), or the official X API v2 via xurl CLI (OAuth2, auto-detected when installed and authenticated)
+- Accepts a host-provided `--x-posts` envelope (a `.json` file of posts the hosting model fetched through its own X connector) as untrusted input: it replaces the engine's X fetch for that run, every row is validated and its citation URL rebuilt from the post id, and no X backend is called
- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
diff --git a/skills/last30days/scripts/box_chrome_login.py b/skills/last30days/scripts/box_chrome_login.py
index 5c651cd9..525e81ef 100644
--- a/skills/last30days/scripts/box_chrome_login.py
+++ b/skills/last30days/scripts/box_chrome_login.py
@@ -11,7 +11,9 @@ Firefox / Safari extract path).
Contract, matching the rest of the feature:
* **Extras only.** On a MacBook (and any non-extras host) it prints "no launch
- needed" and never spawns a browser, even with ``--exec``.
+ needed" and never spawns a browser, even with ``--exec``. An official-only
+ host (``LAST30DAYS_HOST=grok-bot``, see ``env.x_policy``) gets the same
+ no-launch recipe with a neutral note, even on Linux or a Mac mini.
* Launch on the last30days extras NUX port ``18800``
(``SAND_CHROME_REMOTE_DEBUG_PORT=18800``) so ``chrome_cdp`` finds it. This is
NOT box-chrome's built-in default (``9222`` + the display number).
@@ -44,6 +46,12 @@ EXTRAS_CDP_PORT = chrome_cdp._BOX_CHROME_PORT
DEFAULT_PROFILE_DIR = "/tmp/last30days-x-chrome"
LOGIN_URL = "https://x.com/login"
BOX_CHROME_BIN = "box-chrome"
+# Printed on an official-only host instead of the MacBook note. Names no
+# cookie mechanism: this host searches X through official access only.
+OFFICIAL_HOST_NOTE = (
+ "No launch needed: this host searches X through official access and "
+ "does not use a browser login window."
+)
def build_recipe(
@@ -58,6 +66,23 @@ def build_recipe(
``profile_dir``, ``url``, ``env`` (launch env overrides or None),
``command`` (argv or None), ``note`` (human guidance).
"""
+ # Official-only host (LAST30DAYS_HOST=grok-bot): the same no-launch
+ # recipe as a MacBook, checked BEFORE the extras signals so a Linux or
+ # Mac mini Grok Bot computer never gets a launch command. The note is
+ # neutral on purpose. A LAST30DAYS_X_BACKEND=bird pin re-enables
+ # discovery through env.x_policy, and with it this helper.
+ if not env.x_policy(config).cookie_discovery:
+ return {
+ "applies": False,
+ "box_chrome": None,
+ "port": EXTRAS_CDP_PORT,
+ "profile_dir": profile_dir,
+ "url": url,
+ "env": None,
+ "command": None,
+ "note": OFFICIAL_HOST_NOTE,
+ }
+
if not env.x_extras_enabled(config):
return {
"applies": False,
@@ -119,7 +144,10 @@ def render_recipe(recipe: Dict[str, Any]) -> str:
"""Human-readable recipe. Contains no cookie values (none are read)."""
lines: List[str] = []
if not recipe["applies"]:
- lines.append("[box-chrome login] Not an extras host.")
+ if recipe["note"] == OFFICIAL_HOST_NOTE:
+ lines.append("[login helper] No launch needed on this host.")
+ else:
+ lines.append("[box-chrome login] Not an extras host.")
lines.append(recipe["note"])
return "\n".join(lines)
diff --git a/skills/last30days/scripts/last30days.py b/skills/last30days/scripts/last30days.py
index 10973d46..b1e3989f 100644
--- a/skills/last30days/scripts/last30days.py
+++ b/skills/last30days/scripts/last30days.py
@@ -51,7 +51,7 @@ if os.name == "nt":
SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))
-from lib import competitors as competitors_mod, corpus, dates, discovery_handoff, env, freshness, html_render, http, permission_preflight, pipeline, registers, render, schema, ui
+from lib import competitors as competitors_mod, corpus, dates, discovery_handoff, env, freshness, html_render, http, permission_preflight, pipeline, registers, render, schema, ui, x_envelope
_child_pids: set[int] = set()
_child_pids_lock = threading.Lock()
@@ -755,6 +755,17 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument("--store", action="store_true", help="Persist ranked findings to the SQLite research store")
parser.add_argument("--x-handle", help="X handle for targeted supplemental search")
parser.add_argument("--x-related", help="Comma-separated related X handles (searched with lower weight)")
+ parser.add_argument(
+ "--x-posts",
+ dest="x_posts",
+ metavar="PATH",
+ help=(
+ "Path to a last30days-x-posts/1 JSON envelope of posts the hosting "
+ "model fetched through its X connector; replaces the engine's X "
+ "fetch for this run. A file path only (never inline JSON); on a "
+ "comparison run use the per-entity x_posts field of --competitors-plan."
+ ),
+ )
parser.add_argument("--web-backend", default="auto",
choices=["auto", "brave", "exa", "serper", "parallel", "parallel-mcp", "keyless", "none"],
help="Web search backend (default: auto; parallel-mcp explicitly opts into the "
@@ -904,6 +915,7 @@ def parse_competitors_plan(raw: str | None) -> dict[str, dict]:
known_fields = {
"x_handle", "x_related", "subreddits",
"github_user", "github_repos", "trustpilot_domain", "context",
+ "x_posts",
}
normalized: dict[str, dict] = {}
for entity, entry in parsed.items():
@@ -1242,7 +1254,11 @@ def _write_last_run(
topic: str,
report: "schema.Report",
entity_reports: list[tuple[str, schema.Report]] | None = None,
+ *,
+ x_envelope_sha256: str | None = None,
) -> bool:
+ # ``x_envelope_sha256`` binds the cached report to the --x-posts file it
+ # was built from; _load_last_report_cache misses on any mismatch.
try:
if env.CONFIG_DIR is None:
return False
@@ -1268,6 +1284,7 @@ def _write_last_run(
"topic": topic,
"timestamp": payload["timestamp"],
"comparison": bool(entity_reports),
+ "x_envelope_sha256": x_envelope_sha256 or None,
"reports": [
{"entity": label, "report": schema.to_dict(cached_report)}
for label, cached_report in cached_reports
@@ -1288,6 +1305,8 @@ def _write_last_run(
def _load_last_report_cache(
topic: str | None,
ttl_seconds: int = DEFAULT_REPORT_CACHE_TTL_SECONDS,
+ *,
+ x_envelope_sha256: str | None = None,
) -> tuple[schema.Report, list[tuple[str, schema.Report]] | None, Path] | None:
cache_path = _last_report_cache_path()
if cache_path is None or not cache_path.exists():
@@ -1300,6 +1319,12 @@ def _load_last_report_cache(
return None
if not _is_report_cache_fresh(payload.get("timestamp"), ttl_seconds):
return None
+ # A report built from a --x-posts envelope is only reusable with the
+ # same envelope content; a digest on either side that does not match
+ # the other is a miss.
+ cached_digest = payload.get("x_envelope_sha256") or None
+ if (cached_digest or x_envelope_sha256) and cached_digest != x_envelope_sha256:
+ return None
cached_topic = str(payload.get("topic") or "").strip().lower()
if topic is not None and cached_topic != topic.strip().lower():
return None
@@ -2554,8 +2579,85 @@ SETUP_PASSTHROUGH_FLAGS = {
"--github-start",
"--github-poll",
"--openclaw",
+ "--store-key",
}
+STORE_KEY_FLAG = "--store-key"
+
+
+def _split_store_key(extra_argv: list[str]) -> tuple[bool, str, list[str]]:
+ """Pull ``--store-key <NAME>`` / ``--store-key=<NAME>`` out of ``extra_argv``.
+
+ Returns ``(present, name, remaining)``. ``name`` is "" when the flag has
+ no value; ``remaining`` is every other passthrough token, so the regular
+ allowlist check still applies to them.
+ """
+ present = False
+ name = ""
+ remaining: list[str] = []
+ i = 0
+ while i < len(extra_argv):
+ arg = extra_argv[i]
+ if arg == STORE_KEY_FLAG:
+ present = True
+ if i + 1 < len(extra_argv) and not extra_argv[i + 1].startswith("-"):
+ name = extra_argv[i + 1]
+ i += 2
+ continue
+ i += 1
+ continue
+ if arg.startswith(STORE_KEY_FLAG + "="):
+ present = True
+ name = arg[len(STORE_KEY_FLAG) + 1:]
+ i += 1
+ continue
+ remaining.append(arg)
+ i += 1
+ return present, name, remaining
+
+
+# One credential line: longer than any real token, short enough that a
+# misdirected stream on stdin cannot grow memory.
+STORE_KEY_MAX_BYTES = 64 * 1024
+
+
+def _run_store_key(name: str) -> int:
+ """``setup --store-key <NAME>``: persist one allowlisted credential from stdin.
+
+ Reads exactly one line from stdin (bounded to ``STORE_KEY_MAX_BYTES``),
+ strips whitespace, and writes it to the global ``.env`` as a 0o600 secret
+ through ``setup_wizard.write_api_key``. An existing line for the same
+ name is replaced, so a rejected credential can be rotated by running the
+ command again. The value never reaches stdout or stderr: stdout carries
+ ``NAME=****`` plus a JSON line ``{"persisted": bool, "key": NAME}``. A
+ name outside ``env.KEYCHAIN_KEYS`` or an empty value exits 2 without
+ echoing anything.
+ """
+ from lib import setup_wizard
+
+ if name not in env.KEYCHAIN_KEYS:
+ # Do not enumerate the allowlist here: on an official-only host a
+ # failure hint must not name the legacy credential keys.
+ sys.stderr.write(
+ "[last30days] setup --store-key: unknown or missing key name "
+ "(must be a credential name the engine loads from its .env; "
+ "see CONFIGURATION.md).\n"
+ )
+ return 2
+ value = sys.stdin.readline(STORE_KEY_MAX_BYTES).strip()
+ if not value:
+ sys.stderr.write(
+ f"[last30days] setup --store-key {name}: empty value on stdin; "
+ "pipe the credential as a single line.\n"
+ )
+ return 2
+ persisted = bool(
+ setup_wizard.write_api_key(env.CONFIG_FILE, value, key_name=name, replace=True)
+ )
+ print(f"{name}=****")
+ print(json.dumps({"persisted": persisted, "key": name}))
+ return 0 if persisted else 1
+
SKILL_ONLY_FLAGS = {
"--agent",
}
@@ -2572,10 +2674,107 @@ DOCTOR_PASSTHROUGH_FLAGS = {
}
+def _looks_inline_json(value: str) -> bool:
+ """True when a --x-posts argument is JSON text rather than a path."""
+ stripped = value.strip()
+ return stripped.startswith(("{", "[")) or "\n" in value
+
+
+def _comparison_requested(args: argparse.Namespace, topic: str) -> bool:
+ """Whether this invocation is a comparison run (vs-topic or competitor flags)."""
+ from lib import planner as _planner
+
+ return any(
+ value is not None
+ for value in (args.competitors, args.competitors_list, args.competitors_plan)
+ ) or len(_planner._comparison_entities(topic, uncapped=True)) >= 2
+
+
+def _read_x_envelope(
+ path: str,
+ topic: str,
+ args: argparse.Namespace,
+ *,
+ x_handle: str | None,
+ x_related: list[str] | None,
+) -> x_envelope.Envelope:
+ """Validate a host-fetched X envelope against this run's window and topic."""
+ from_date, to_date = dates.get_date_range(
+ args.lookback_days or 30, as_of_date=args.as_of_date
+ )
+ return x_envelope.read(
+ path,
+ (from_date, to_date),
+ topic,
+ handles=[x_handle] if x_handle else [],
+ related=[h for h in (x_related or []) if h and h.strip()],
+ )
+
+
+def _attach_entity_envelopes(comp_plan: dict[str, dict], args: argparse.Namespace) -> None:
+ """Validate every per-entity ``x_posts`` path in a --competitors-plan.
+
+ Each envelope is checked against its own entity (topic) and that entry's
+ ``x_handle``/``x_related`` handles, and stored on the entry as
+ ``_x_envelope`` for the entity sub-run. Raises EnvelopeContractError.
+ """
+ for entry in comp_plan.values():
+ raw = entry.get("x_posts")
+ if not raw:
+ continue
+ if not isinstance(raw, str) or _looks_inline_json(raw):
+ raise x_envelope.EnvelopeContractError(
+ f"--competitors-plan entry {entry.get('_name', '')!r}: x_posts must "
+ "be a file path to a last30days-x-posts/1 envelope, never inline JSON. "
+ "Rewrite the plan entry, or drop its x_posts field."
+ )
+ related = entry.get("x_related") if isinstance(entry.get("x_related"), list) else None
+ entry["_x_envelope"] = _read_x_envelope(
+ raw, str(entry.get("_name") or ""), args,
+ x_handle=entry.get("x_handle") if isinstance(entry.get("x_handle"), str) else None,
+ x_related=[str(h) for h in related] if related else None,
+ )
+
+
+def _combine_envelope_digests(main_sha256: str | None, entity_sha256: dict[str, str]) -> str | None:
+ """One digest binding the last-report cache to every envelope a run uses.
+
+ A single top-level envelope is bound by its own file digest; per-entity
+ comparison envelopes are folded, name-sorted, into one digest. Both the
+ cache write (validated envelopes) and the cache lookup (planned paths)
+ must go through here so a comparison cache can be reused.
+ """
+ parts: list[str] = []
+ if main_sha256:
+ parts.append(main_sha256)
+ for name in sorted(entity_sha256):
+ parts.append(f"{name}:{entity_sha256[name]}")
+ if not parts:
+ return None
+ if len(parts) == 1 and main_sha256:
+ return main_sha256
+ return hashlib.sha256("|".join(parts).encode("utf-8")).hexdigest()
+
+
+def _x_envelope_digest(
+ main: x_envelope.Envelope | None, comp_plan: dict[str, dict] | None
+) -> str | None:
+ """Digest of the validated envelopes this run used (cache write side)."""
+ entity_sha256 = {
+ name: entry["_x_envelope"].sha256
+ for name, entry in (comp_plan or {}).items()
+ if entry.get("_x_envelope") is not None
+ }
+ return _combine_envelope_digests(main.sha256 if main is not None else None, entity_sha256)
+
+
def _validate_extra_argv(parser: argparse.ArgumentParser, topic: str, extra_argv: list[str]) -> None:
if not extra_argv:
return
if topic.lower() == "setup":
+ # --store-key carries a value token; the name itself is allowlisted
+ # later in _run_store_key, not here.
+ _, _, extra_argv = _split_store_key(extra_argv)
unsupported = [arg for arg in extra_argv if arg not in SETUP_PASSTHROUGH_FLAGS]
if unsupported:
parser.error(
@@ -2927,11 +3126,24 @@ def _main(
topic = " ".join(args.topic).strip()
original_topic = topic
_validate_extra_argv(parser, topic, extra_argv)
+ if args.x_posts is not None and _looks_inline_json(args.x_posts):
+ sys.stderr.write(
+ "[last30days] --x-posts accepts a file path only (inline JSON is not "
+ "accepted); write the envelope to a .json file and pass its path.\n"
+ )
+ return 2
if args.publish and topic.lower() != "library feed":
sys.stderr.write(
"[last30days] --publish is only supported by the 'library feed' command.\n"
)
return 2
+ if topic.lower() == "setup":
+ # Persisting a credential needs no config load (no Keychain / pass
+ # probes, no cookie policy), so it dispatches before get_config.
+ store_key_present, store_key_name, _ = _split_store_key(extra_argv)
+ if store_key_present:
+ return _run_store_key(store_key_name)
+
config = env.get_config(policy=_config_policy_for_args(args, topic, extra_argv))
# One memo per command: comparison mode runs pipeline.run per entity in
# parallel, so the reset must not live inside the pipeline.
@@ -3285,6 +3497,14 @@ def _main(
"the remote API backend only supports --json-profile=raw.\n"
)
return 2
+ if args.x_posts is not None:
+ # The envelope is a local-engine contract; the remote API has no
+ # lane to receive it.
+ sys.stderr.write(
+ "[last30days] --x-posts is not supported by the hosted backend; "
+ "run locally or omit --x-posts.\n"
+ )
+ return 2
from lib import hosted
depth = "deep" if args.deep else "quick" if args.quick else "default"
try:
@@ -3328,7 +3548,33 @@ def _main(
requested_sources,
channels=cli_telegram_sources,
)
- diag = pipeline.diagnose(config, requested_sources, safe=args.diagnose)
+ # Host-fetched X envelope: validated before diagnose so a present
+ # envelope plans X in (available_sources) and a bad one fails closed here.
+ x_posts_envelope: x_envelope.Envelope | None = None
+ if args.x_posts is not None:
+ if not topic:
+ sys.stderr.write("[last30days] --x-posts requires a research topic.\n")
+ return 2
+ if _comparison_requested(args, topic):
+ sys.stderr.write(
+ "[last30days] --x-posts applies to a single-topic run; on a "
+ "comparison run pass each entity's envelope through the "
+ "x_posts field of its --competitors-plan entry.\n"
+ )
+ return 2
+ try:
+ x_posts_envelope = _read_x_envelope(
+ args.x_posts, topic, args,
+ x_handle=args.x_handle,
+ x_related=args.x_related.split(",") if args.x_related else None,
+ )
+ except x_envelope.EnvelopeContractError as exc:
+ sys.stderr.write(f"[last30days] {exc.message}\n")
+ return 2
+ diag = pipeline.diagnose(
+ config, requested_sources, safe=args.diagnose,
+ x_envelope=x_posts_envelope is not None,
+ )
if args.diagnose:
print(json.dumps(diag, indent=2, sort_keys=True))
@@ -3339,6 +3585,17 @@ def _main(
# this runtime-only object out of the safe diagnose configuration contract.
config["_perplexity_paid_budget"] = pipeline.PaidSourceBudget()
+ # Per-entity host-fetched X envelopes are validated here, on the main
+ # thread and BEFORE the report-cache lookup, so a bad or stale one fails
+ # closed (exit 2) instead of silently dropping that entity inside the
+ # fan-out or being served from a cache built while it was still valid.
+ comp_plan = parse_competitors_plan(args.competitors_plan)
+ try:
+ _attach_entity_envelopes(comp_plan, args)
+ except x_envelope.EnvelopeContractError as exc:
+ sys.stderr.write(f"[last30days] {exc.message}\n")
+ return 2
+
if not topic:
parser.print_usage(sys.stderr)
return 2
@@ -3368,6 +3625,7 @@ def _main(
cached = _load_last_report_cache(
topic,
ttl_seconds=_report_cache_ttl_seconds(config),
+ x_envelope_sha256=_x_envelope_digest(x_posts_envelope, comp_plan),
)
if cached is not None:
cached_report, cached_entity_reports, cache_path = cached
@@ -3508,7 +3766,8 @@ def _main(
trustpilot_domain = args.trustpilot_domain.strip() if args.trustpilot_domain else None
comp_enabled, comp_count, comp_explicit = resolve_competitors_args(args)
- comp_plan = parse_competitors_plan(args.competitors_plan)
+ # comp_plan was parsed, and its per-entity envelopes validated, before
+ # the report-cache lookup above.
# Plan-level trustpilot_domain pins are the same user intent as the CLI
# flag (already activated above). Auto-resolve hints must not activate.
@@ -3628,6 +3887,10 @@ def _main(
save_dir=args.save_dir,
corpus_dirs=args.corpus,
corpus_all_time=args.corpus_all_time,
+ x_posts=(
+ comp_plan.get(topic.strip().lower(), {}).get("_x_envelope")
+ if comp_enabled else x_posts_envelope
+ ),
)
r.artifacts["resolved"] = {
"entity": topic,
@@ -3775,6 +4038,7 @@ def _main(
save_dir=args.save_dir,
corpus_dirs=args.corpus,
corpus_all_time=args.corpus_all_time,
+ x_posts=plan_entry.get("_x_envelope"),
)
report.artifacts["resolved"] = resolved_effective
return report
@@ -3808,7 +4072,10 @@ def _main(
report, progress, diag,
suppress_web_promo=bool(external_plan or comp_plan),
)
- _write_last_run(original_topic, report, entity_reports=entity_reports)
+ _write_last_run(
+ original_topic, report, entity_reports=entity_reports,
+ x_envelope_sha256=_x_envelope_digest(x_posts_envelope, comp_plan),
+ )
# LAST30DAYS_STORE env var = persistence default-on. Read both os.environ
# (for shell-exported users) and config (for users who set it in
# ~/.config/last30days/.env, which env.py loads but does not propagate
diff --git a/skills/last30days/scripts/lib/backends.py b/skills/last30days/scripts/lib/backends.py
index b47278d6..18aa8567 100644
--- a/skills/last30days/scripts/lib/backends.py
+++ b/skills/last30days/scripts/lib/backends.py
@@ -29,15 +29,18 @@ say the next run will try" — rendered as "will use". It is not an
observation of what served a past run, and runtime failover can still
diverge mid-run (a present-but-expired paid key passes a presence probe).
-Paid lanes (xai, xquik, serper, and every other API-key backend, including
-ScrapeCreators) probe KEY PRESENCE ONLY: a dict lookup, never a network
-call or credential spend. Binary-backed lanes reuse the U1 dependency
+Paid lanes (xai, xapi, xquik, serper, and every other API-key backend,
+including ScrapeCreators) probe KEY PRESENCE ONLY: a dict lookup, never a
+network call or credential spend. Binary-backed lanes reuse the U1 dependency
probe layer (``health.probe_dependency``) so a stale shim reads as BROKEN,
not available (#692).
This module observes and predicts only. It must never alter which backend
the pipeline actually uses; parity with the pipeline's pre-failover
-selection is asserted in ``tests/test_backend_descriptors.py``.
+selection is asserted in ``tests/test_backend_descriptors.py``. The X chain
+is shaped by ``env.x_policy``: on an official-only host the findings and
+the chain carry only the policy's backends (plus a pinned one), so doctor
+JSON never names a non-official backend there unless it is pinned.
"""
from __future__ import annotations
@@ -74,10 +77,20 @@ _SC_PRESCRIPTION = (
"set SCRAPECREATORS_API_KEY (free 10,000-call signup: "
f"{prescriptions.get('scrapecreators', 'key_missing').fix_cli})"
)
-_X_COOKIES_PRESCRIPTION = (
- "run setup with browser-cookie consent: "
- f"{prescriptions.get('x', 'cookies_missing').fix_cli}"
-)
+
+
+def _x_cookies_prescription(config: Dict[str, Any]) -> str:
+ """Bird's unconfigured fix, routed through the X policy.
+
+ Off an official-only host this is the cookie-consent command; on one
+ the same lookup yields the official-path entry (connector lane, bearer,
+ xAI key), so a pinned-but-unconfigured scraper never prescribes a
+ cookie read there.
+ """
+ entry = prescriptions.for_x(config, "cookies_missing")
+ if entry.failure == "cookies_missing":
+ return f"run setup with browser-cookie consent: {entry.fix_cli}"
+ return f"{entry.fix_nl} (cli: {entry.fix_cli})"
@dataclass
@@ -215,7 +228,7 @@ def _probe_bird(config: Dict[str, Any]) -> BackendFinding:
name="bird",
status=health.MISSING,
detail="X browser cookies (AUTH_TOKEN/CT0) not configured",
- prescription=_X_COOKIES_PRESCRIPTION,
+ prescription=_x_cookies_prescription(config),
requires=requires,
)
if not bird_x.is_bird_installed():
@@ -328,6 +341,36 @@ def _probe_grok(config: Dict[str, Any]) -> BackendFinding:
)
+def _probe_xapi(config: Dict[str, Any]) -> BackendFinding:
+ """xapi = direct X API v2 with an app-only bearer. KEY PRESENCE ONLY.
+
+ Never a network call. The unconfigured fix is the official-path
+ prescription on an official-only host (connector lane, bearer, xAI key,
+ with the about-a-week caveat); elsewhere the plain key hint,
+ since xapi runs there only under an explicit pin.
+ """
+ requires = "X_BEARER_TOKEN (X API v2)"
+ if config.get("X_BEARER_TOKEN"):
+ return BackendFinding(
+ name="xapi",
+ status=health.OK,
+ detail="X_BEARER_TOKEN present",
+ requires=requires,
+ )
+ if env.x_policy(config).official_only:
+ entry = prescriptions.get("x", "bearer_missing")
+ prescription = f"{entry.fix_nl} (cli: {entry.fix_cli})"
+ else:
+ prescription = "set X_BEARER_TOKEN in ~/.config/last30days/.env"
+ return BackendFinding(
+ name="xapi",
+ status=health.MISSING,
+ detail="X_BEARER_TOKEN not set",
+ prescription=prescription,
+ requires=requires,
+ )
+
+
def _probe_xurl(config: Dict[str, Any]) -> BackendFinding:
"""xurl = official X API v2 CLI (OAuth2). Free lane; LOCAL-ONLY probe.
@@ -428,8 +471,10 @@ _X_PROBES: Dict[str, Callable[[Dict[str, Any]], BackendFinding]] = {
"bird": _probe_bird,
"xurl": _probe_xurl,
"xquik": _key_probe("xquik", "XQUIK_API_KEY", "XQUIK_API_KEY (xquik.com)"),
+ # Direct X API v2 with an app-only bearer: key presence only, no network.
+ "xapi": _probe_xapi,
}
-_X_PAID = {"xai", "xquik"}
+_X_PAID = {"xai", "xquik", "xapi"}
# Opt-in backends: never auto-selected; require explicit pin.
_X_OPT_IN = set(env.X_BACKEND_OPT_IN)
@@ -459,6 +504,7 @@ _X_REQUIRES: Dict[str, str] = {
"bird": "X browser cookies (AUTH_TOKEN/CT0) + node",
"xurl": "xurl CLI installed + OAuth2 login",
"xquik": "XQUIK_API_KEY (xquik.com)",
+ "xapi": "X_BEARER_TOKEN (X API v2)",
}
DESCRIPTORS: Dict[str, ChainDescriptor] = {
@@ -555,12 +601,41 @@ def resolve(
multiple binaries are simultaneously hung.
"""
descriptor = get_descriptor(source)
- findings = [
- _run_probe(spec, config) for spec in descriptor.backends
- ]
+ specs, auto_names = _specs_for_policy(descriptor, config)
+ findings = [_run_probe(spec, config) for spec in specs]
if descriptor.mode == MODE_CONDITIONAL:
return _resolve_conditional(descriptor, config, findings)
- return _resolve_alternative(descriptor, config, findings, pin)
+ return _resolve_alternative(descriptor, config, findings, pin, auto_names)
+
+
+def _specs_for_policy(
+ descriptor: ChainDescriptor,
+ config: Dict[str, Any],
+) -> Tuple[List[BackendSpec], set]:
+ """The backends to probe for this host, and which of them auto-select.
+
+ Every source except X keeps its declared backends; auto-selection is the
+ non-opt-in set. For X the answer comes from ``env.x_policy``: on a
+ default host the declared chain (auto order plus opt-in entries for
+ doctor visibility) is unchanged; on an official-only host the findings
+ are the policy's chain in its order, plus the pinned backend when the
+ pin names something outside it, so neither doctor JSON nor the chain
+ string carries a non-official backend unless it is pinned. Observation
+ only: this mirrors ``env.x_backend_chain``, it never alters it.
+ """
+ specs = list(descriptor.backends)
+ if descriptor.source != "x":
+ return specs, {spec.name for spec in specs if not spec.opt_in}
+ policy = env.x_policy(config)
+ auto_names = set(policy.auto_chain)
+ if not policy.official_only:
+ return specs, auto_names
+ by_name = {spec.name: spec for spec in specs}
+ ordered = [by_name[name] for name in policy.auto_chain if name in by_name]
+ pin = env.x_backend_pin(config)
+ if pin in by_name and pin not in auto_names:
+ ordered.append(by_name[pin])
+ return ordered, auto_names
def _run_probe(spec: BackendSpec, config: Dict[str, Any]) -> BackendFinding:
@@ -584,11 +659,14 @@ def _resolve_alternative(
config: Dict[str, Any],
findings: List[BackendFinding],
pin: Optional[str],
+ auto_names: Optional[set] = None,
) -> BackendResolution:
- names = [spec.name for spec in descriptor.backends]
+ names = [f.name for f in findings]
by_name = {f.name: f for f in findings}
- # Track which backends are opt-in (never auto-selected).
- opt_in_names = {spec.name for spec in descriptor.backends if spec.opt_in}
+ if auto_names is None:
+ auto_names = {spec.name for spec in descriptor.backends if not spec.opt_in}
+ # Backends outside the auto set are opt-in here (never auto-selected).
+ opt_in_names = {name for name in names if name not in auto_names}
res = BackendResolution(
source=descriptor.source,
mode=MODE_ALTERNATIVE,
diff --git a/skills/last30days/scripts/lib/doctor.py b/skills/last30days/scripts/lib/doctor.py
index 786e0fe5..325e585c 100644
--- a/skills/last30days/scripts/lib/doctor.py
+++ b/skills/last30days/scripts/lib/doctor.py
@@ -53,7 +53,7 @@ import urllib.request
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional
-from . import backends, brightdata, env, health, http, prescriptions
+from . import backends, brightdata, env, health, http, prescriptions, x_api
from .backends import TIER_ERROR, TIER_OK, TIER_WARN
# Rollup tiers (R1). ok/warn/error are U2's; only "off" is doctor's own.
@@ -195,6 +195,7 @@ KEY_PRESENCE_VARS = (
"SCRAPECREATORS_API_KEY",
"XAI_API_KEY",
"XQUIK_API_KEY",
+ "X_BEARER_TOKEN",
"BRAVE_API_KEY",
"EXA_API_KEY",
"SERPER_API_KEY",
@@ -416,8 +417,58 @@ def _reddit_record(config):
return _chained_record("reddit", config)
+# Official-path wording. The bearer path is never described as
+# parity with the connector lane.
+X_BEARER_CAVEAT = x_api.BEARER_COVERAGE_NOTE
+X_CONNECTOR_NOTE = "will use: X connector (host-fetched at run time)"
+X_CONNECTOR_ARMED = "X connector lane armed"
+
+
+def _x_will_use_note(record: Dict[str, Any], policy: env.XPolicy) -> str:
+ """The will-use line for a predicted X backend, shaped by the policy.
+
+ On an official-only host a pinned backend renders as ``will use: <name>
+ (pinned)`` and nothing else names it (the pin variable is not
+ advertised). Off it the backends summary is kept verbatim.
+ The bearer prediction carries the about-a-week caveat on every host.
+ """
+ name = record["active_backend"]
+ if policy.official_only:
+ qualifiers: List[str] = []
+ if record.get("pinned"):
+ qualifiers.append("pinned")
+ if name == "xapi":
+ qualifiers.append(X_BEARER_CAVEAT)
+ return f"will use: {name}" + (f" ({'; '.join(qualifiers)})" if qualifiers else "")
+ note = record.get("note") or f"will use: {name}"
+ if name == "xapi":
+ if note.endswith(")"):
+ return f"{note[:-1]}; {X_BEARER_CAVEAT})"
+ return f"{note} ({X_BEARER_CAVEAT})"
+ return note
+
+
def _x_record(config):
record = _chained_record("x", config)
+ policy = env.x_policy(config)
+ if policy.official_only and not record.get("pinned"):
+ # The pin is documented in CONFIGURATION.md only; doctor never
+ # advertises the knob on an official-only host.
+ record["pin_var"] = None
+ if record.get("active_backend"):
+ record["note"] = _x_will_use_note(record, policy)
+ return record
+ # The declared X connector lane serves X when no engine backend is
+ # predicted (the same precedence as diagnose.x_backend). Host-independent:
+ # the envelope is accepted anywhere.
+ if env.x_host_lane_declared(config):
+ record["status"] = health.OK
+ record["tier"] = TIER_BY_STATUS[health.OK]
+ record["active_backend"] = "connector"
+ record["note"] = X_CONNECTOR_NOTE
+ record["detail"] = ""
+ record["fix"] = ""
+ return record
# Diagnose/doctor load config in plan_only mode, so browser cookies are not
# extracted and every X backend reads as statically missing -> unconfigured.
# But if bird is installed and FROM_BROWSER will authenticate X at run time,
@@ -427,6 +478,9 @@ def _x_record(config):
# will *attempt* browser auth, not that the session is currently valid -
# keep the note honest and point at the verified key-backed path.
#
+ # Policy-gated: on an official-only host no run-time cookie source
+ # exists unless bird is pinned, and then the note names only the pin.
+ #
# This check MUST come before grok normalization: a pending bird path takes
# precedence over marking X as unconfigured due to an unused grok store.
# Handle both "unconfigured" (all backends missing) and "error" (grok present
@@ -435,10 +489,12 @@ def _x_record(config):
# HOWEVER: pending bird must NOT replace a record that has a configured
# auto-chain backend in ERROR/DEGRADED/BROKEN/TIMEOUT. Same rule as the
# grok normalizer: only upgrade when no auto backend is configured-but-broken.
- pending_bird = env.x_pending_browser_auth(config, local_only=True)
+ auto_chain_names = set(env.x_auto_chain(config))
+ pending_bird = policy.cookie_discovery and env.x_pending_browser_auth(
+ config, local_only=True
+ )
if pending_bird and record["status"] in ("unconfigured", health.ERROR):
backends_list = record.get("backends", [])
- auto_chain_names = {"bird", "xai", "xurl", "xquik"}
auto_backends = [b for b in backends_list if b.get("name") in auto_chain_names]
# Only apply pending-bird upgrade if ALL auto-chain backends are MISSING.
# If any auto backend is configured but broken, keep that error.
@@ -448,10 +504,13 @@ def _x_record(config):
if all_auto_missing:
record["status"] = health.OK
record["tier"] = TIER_BY_STATUS[health.OK]
- record["note"] = (
- "will use: bird (browser cookies; session not verified until a run "
- "- add XAI_API_KEY for a verified, cookie-free path)"
- )
+ if policy.official_only:
+ record["note"] = "will use: bird (pinned)"
+ else:
+ record["note"] = (
+ "will use: bird (browser cookies; session not verified until a run "
+ "- add XAI_API_KEY for a verified, cookie-free path)"
+ )
record["fix"] = ""
return record
#
@@ -460,6 +519,10 @@ def _x_record(config):
# is never auto-selected. Doctor reports it as "available, unused - pin
# LAST30DAYS_X_BACKEND=grok to enable" rather than "will use: grok".
#
+ # Policy-gated: on an official-only host the CLI is not probed
+ # unless pinned and is never offered, so this branch does not run there
+ # (no "pin LAST30DAYS_X_BACKEND=grok" note on a Grok Bot host).
+ #
# R3/R8: When no auto-chain backend is CONFIGURED (all MISSING) but grok has
# any non-MISSING status, X is unconfigured/skipped - NOT broken/auth-failed.
# The tier must be "off" (unconfigured), not "error" (NOT WORKING).
@@ -471,13 +534,13 @@ def _x_record(config):
# Do NOT apply this normalization when pending browser auth would make bird
# usable — check pending_bird first (handled above via early return).
if (
- record["tier"] == TIER_ERROR
+ not policy.official_only
+ and record["tier"] == TIER_ERROR
and not record.get("pinned")
and record.get("active_backend") is None
and not pending_bird
):
backends_list = record.get("backends", [])
- auto_chain_names = {"bird", "xai", "xurl", "xquik"}
auto_backends = [b for b in backends_list if b.get("name") in auto_chain_names]
# Only normalize if ALL auto-chain backends are MISSING (not configured).
# If any auto backend is ERROR/DEGRADED/BROKEN/TIMEOUT, keep that error.
@@ -511,6 +574,21 @@ def _x_record(config):
)
record["fix"] = ""
return record
+ xapi_finding = next(
+ (b for b in backends_list if b.get("name") == "xapi"),
+ None,
+ )
+ if xapi_finding and xapi_finding.get("status") in (health.OK, health.DEGRADED):
+ # Same shape as the grok note: a configured opt-in backend is not
+ # a broken X, it is an unconfigured X with a one-line enable.
+ record["status"] = "unconfigured"
+ record["tier"] = TIER_OFF
+ record["note"] = (
+ "X unconfigured; X_BEARER_TOKEN is set but the X API backend is opt-in "
+ "on this host — pin LAST30DAYS_X_BACKEND=xapi to enable"
+ )
+ record["fix"] = ""
+ return record
return record
@@ -981,6 +1059,48 @@ def load_run_evidence(
# is answerable at a glance without inventing fake sources.
# ---------------------------------------------------------------------------
+def _x_auth_path(config: Dict[str, Any]) -> Dict[str, Any]:
+ """The "X auth path" sub-lane, routed through the X policy.
+
+ Off an official-only host the wording is unchanged (key-backed, cookie
+ path, or nothing armed). On one it names only the official path: the
+ connector lane when declared, else the bearer (with the about-a-week
+ caveat), the xAI key, or xurl; a pinned non-official backend is named
+ once, on the will-use line, never here.
+ """
+ policy = env.x_policy(config)
+ if not policy.official_only:
+ has_key = bool(config.get("XAI_API_KEY") or config.get("XQUIK_API_KEY"))
+ cookie = bool(env.x_pending_browser_auth(config, local_only=True))
+ if has_key:
+ note = "XAI_API_KEY key-backed path (verified, cookie-free)"
+ elif cookie:
+ note = (
+ "browser-cookie path primary; add XAI_API_KEY for a verified "
+ "cookie-free backup"
+ )
+ else:
+ note = "no auth path armed"
+ return {"name": "X auth path", "armed": has_key or cookie, "note": note}
+ if env.x_host_lane_declared(config):
+ return {"name": "X auth path", "armed": True, "note": X_CONNECTOR_ARMED}
+ source = env.get_x_source(config, local_only=True)
+ if source == "xapi":
+ note = f"X_BEARER_TOKEN path ({X_BEARER_CAVEAT})"
+ elif source == "xai":
+ note = "XAI_API_KEY licensed path (console.x.ai)"
+ elif source == "xurl":
+ note = "X API through the xurl CLI (stored OAuth2 login)"
+ elif source:
+ note = "explicit backend pin"
+ else:
+ note = (
+ "no official X path armed (add the X for Grok Bot plugin and connect X in Grok Bot settings, or set "
+ "X_BEARER_TOKEN or XAI_API_KEY)"
+ )
+ return {"name": "X auth path", "armed": bool(source), "note": note}
+
+
def _sub_lanes_for(source: str, config: Dict[str, Any]):
"""Return (backups, comments) metadata for a source, or ([], None)."""
backups: List[Dict[str, Any]] = []
@@ -998,18 +1118,7 @@ def _sub_lanes_for(source: str, config: Dict[str, Any]):
})
comments = {"enabled": bool(env.is_youtube_comments_available(config))}
elif source == "x":
- has_key = bool(config.get("XAI_API_KEY") or config.get("XQUIK_API_KEY"))
- cookie = bool(env.x_pending_browser_auth(config, local_only=True))
- if has_key:
- note = "XAI_API_KEY key-backed path (verified, cookie-free)"
- elif cookie:
- note = (
- "browser-cookie path primary; add XAI_API_KEY for a verified "
- "cookie-free backup"
- )
- else:
- note = "no auth path armed"
- backups.append({"name": "X auth path", "armed": has_key or cookie, "note": note})
+ backups.append(_x_auth_path(config))
elif source == "tiktok":
comments = {"enabled": bool(env.is_tiktok_comments_available(config))}
elif source == "instagram":
@@ -1129,6 +1238,10 @@ def build_report(config: Dict[str, Any]) -> Dict[str, Any]:
"config": {
"global_env": str(env.CONFIG_FILE) if env.CONFIG_FILE else None,
"config_source": config.get("_CONFIG_SOURCE"),
+ # The resolved host value, so a missing export is visible.
+ "host": env.x_policy(config).host or None,
+ # A .env line cannot declare the X connector lane.
+ "host_lane_file_ignored": bool(config.get("_X_HOST_LANE_FILE_IGNORED")),
},
"setup": _setup_block(config),
"permissions": permissions,
@@ -1263,6 +1376,17 @@ def render_text(report: Dict[str, Any]) -> str:
if config_block.get("config_source"):
line += f" (source: {config_block['config_source']})"
lines.append(line)
+ host = config_block.get("host")
+ if host:
+ lines.append(f"host: {host} ({env.X_HOST_VAR})")
+ else:
+ lines.append(f"host: not set ({env.X_HOST_VAR} unset; default X policy)")
+ if config_block.get("host_lane_file_ignored"):
+ lines.append(
+ f"note: a {env.X_HOST_LANE_VAR} line in .env cannot declare the X "
+ "connector lane (ignored); export it in the process environment "
+ "for the session instead"
+ )
setup = report.get("setup") or {}
present = sorted(
@@ -1365,6 +1489,19 @@ def build_postmortem(config: Dict[str, Any]) -> Dict[str, Any]:
}
+def _postmortem_state_label(source: str, outcome: Dict[str, Any]) -> str:
+ """State label for a failed post-mortem line.
+
+ ``payment-required`` reads as "credits exhausted" ("X API credits
+ exhausted" for X) so the fix is obvious at a glance: top up, not re-login.
+ Every other state prints as its raw name.
+ """
+ state = str(outcome.get("state") or "")
+ if state == health.PAYMENT_REQUIRED:
+ return health.credits_exhausted_label(source)
+ return state
+
+
def render_postmortem_text(pm: Dict[str, Any]) -> str:
lines = [f"last30days post-mortem — engine v{pm['engine_version']}"]
if not pm.get("present"):
@@ -1395,7 +1532,7 @@ def render_postmortem_text(pm: Dict[str, Any]) -> str:
lines.append("Failed:")
for source, outcome in failed:
detail = outcome.get("detail") or outcome.get("state")
- lines.append(f" ✕ {source} — {outcome.get('state')}: {detail}")
+ lines.append(f" ✕ {source} — {_postmortem_state_label(source, outcome)}: {detail}")
if outcome.get("fix_hint"):
lines.append(f" fix: {outcome['fix_hint']}")
if partial:
@@ -1469,8 +1606,15 @@ _SECRET_CONFIG_VARS = KEY_PRESENCE_VARS + (
)
# Backend pin vars folded into the config fingerprint. Pin values are
-# backend names (e.g. "bird"), never secrets.
-_FINGERPRINT_PIN_VARS = (env.X_BACKEND_PIN_VAR, env.REDDIT_BACKEND_PIN_VAR)
+# backend names (e.g. "bird"), never secrets. The host key and the X
+# connector lane signal join them: both are non-secret switches that
+# change every X conclusion, so a cached report must not outlive them.
+_FINGERPRINT_PIN_VARS = (
+ env.X_BACKEND_PIN_VAR,
+ env.REDDIT_BACKEND_PIN_VAR,
+ env.X_HOST_VAR,
+ env.X_HOST_LANE_VAR,
+)
# Top-level report keys the renderers read unguarded; a cached report
# missing any of them is treated as corrupt (absent), never rendered.
diff --git a/skills/last30days/scripts/lib/env.py b/skills/last30days/scripts/lib/env.py
index 0606a996..509342c4 100644
--- a/skills/last30days/scripts/lib/env.py
+++ b/skills/last30days/scripts/lib/env.py
@@ -68,6 +68,7 @@ KEYCHAIN_KEYS = (
"TRUTHSOCIAL_TOKEN", "BRAVE_API_KEY", "EXA_API_KEY", "SERPER_API_KEY",
"OPENROUTER_API_KEY", "PERPLEXITY_API_KEY", "PARALLEL_API_KEY", "XQUIK_API_KEY",
"XIAOHONGSHU_API_BASE", "GITHUB_TOKEN", "BRIGHTDATA_API_KEY",
+ "X_BEARER_TOKEN",
)
# pass(1) integration: Linux/Unix analog of the Keychain source. Each key in
@@ -614,9 +615,25 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:
('LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT', None),
('LAST30DAYS_YT_SEARCH_TIMEOUT', None),
('GITHUB_TOKEN', None),
+ # Host self-identification. `grok-bot` switches the X policy
+ # to official-only (see x_policy); the engine never sniffs the host
+ # any other way. Persisted by first-run setup and exported per
+ # invocation by the SKILL.md rule.
+ (X_HOST_VAR, None),
+ # App-only bearer token for the direct X API v2 backend (`xapi`).
+ ('X_BEARER_TOKEN', None),
+ # Per-session X connector lane signal. Read from the process
+ # environment ONLY: a .env line is deliberately ignored (a removed
+ # connector must never leave a stale declaration), so it is handled
+ # in the loop below rather than via merged_env.
+ (X_HOST_LANE_VAR, None),
]
for key, default in keys:
+ if key == X_HOST_LANE_VAR:
+ # Process env only; the .env value never reaches config.
+ config[key] = os.environ.get(key) or default
+ continue
if key == 'LAST30DAYS_YT_PLAYER_CLIENT':
# Empty string is a valid disable; `or` would treat it as unset.
if key in os.environ:
@@ -689,11 +706,22 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:
config['_IGNORED_PROJECT_CONFIG'] = str(ignored_project_env_path)
config['_IGNORED_PROJECT_CONFIG_KEYS'] = ignored_project_keys
config['_BROWSER_COOKIE_MODE'] = policy.browser_cookies
+ # A LAST30DAYS_X_HOST_LANE line in a config file is ignored;
+ # remember that it was there so doctor can say so.
+ config['_X_HOST_LANE_FILE_IGNORED'] = bool(merged_env.get(X_HOST_LANE_VAR))
+ # Evaluated after the host and pin keys are merged: on an official-only
+ # host the browser list is empty unless bird is pinned (x_policy).
config['_BROWSER_COOKIE_BROWSERS'] = cookie_extraction_browsers(config)
if policy.browser_cookies == "read":
_discover_and_apply_x_credentials(config)
+ # Fixture recording (--record-fixtures) must redact a credential that
+ # came from a file, Keychain, or pass, not only one exported in the
+ # shell. No-op outside a recording session.
+ from . import http as _http
+ _http.add_fixture_redactions(_http.config_secret_values(config))
+
return config
@@ -806,7 +834,14 @@ def _discover_and_apply_x_credentials(config: dict[str, Any]) -> None:
On a Mac mini that has already opted into browser reads (FROM_BROWSER set),
the native extract runs BEFORE CDP (R19) — a local Keychain read beats a
debug-port scrape. Never persists cookies; values are never logged.
+
+ On an official-only host (``x_policy``: ``LAST30DAYS_HOST=grok-bot``)
+ this returns before ANY leg, for every cookie domain, unless the pin is
+ ``bird`` (the one path that re-enables discovery for that run).
"""
+ if not x_policy(config).cookie_discovery:
+ return
+
from . import agentcookie, chrome_cdp
def have_pair() -> bool:
@@ -876,8 +911,11 @@ def cookie_extraction_browsers(config: dict[str, Any]) -> list[str]:
Returning the browser list from one place keeps the setup wizard and the
steady-state path on the same policy, so neither surprises the user with an
- unrequested Keychain prompt.
+ unrequested Keychain prompt. On an official-only host (``x_policy``) the
+ list is empty regardless of ``FROM_BROWSER`` unless ``bird`` is pinned.
"""
+ if not x_policy(config).cookie_discovery:
+ return []
silent_browsers = ["firefox", "safari"]
chromium_browsers = ["chrome", "brave", "edge", "vivaldi", "opera", "arc", "chromium"]
known_browsers = silent_browsers + chromium_browsers
@@ -945,23 +983,37 @@ def extract_browser_credentials(config: dict[str, Any]) -> dict[str, str]:
return extracted
+# Auth-origin label per X backend for ``get_x_source_with_method`` (bird's
+# label is the cookie source recorded in ``_AUTH_TOKEN_SOURCE``).
+_X_METHOD_LABELS = {
+ "xai": "xai",
+ "xurl": "oauth2", # xurl CLI (official X API v2, OAuth2, free developer app)
+ "xapi": "bearer",
+ "xquik": "api_key",
+}
+
+
def get_x_source_with_method(config: dict[str, Any]) -> tuple[str | None, str]:
"""Return (source, method) for X search, where method describes the auth origin.
- Order mirrors _X_BACKEND_ORDER: bird first (cookies beat XAI_API_KEY when
- both are present), then xai, then xurl. Grok is opt-in only and is never
- auto-selected here.
+ Walks the policy's unpinned auto chain (``x_auto_chain``): on a default
+ host bird first (cookies beat XAI_API_KEY when both are present), then
+ xai, xurl, xquik; on an official-only host xapi, xai, xurl. Opt-in
+ backends (grok, and xapi off Grok Bot) are never auto-selected here.
"""
- # Bird first: cookies beat XAI_API_KEY when both are present.
- if config.get("AUTH_TOKEN") and config.get("CT0"):
- method = config.get("_AUTH_TOKEN_SOURCE", "env")
- return "bird", method
- if config.get("XAI_API_KEY"):
- return "xai", "xai"
- # Fall back to xurl CLI (official X API v2, OAuth2, free developer app)
- from . import xurl_x
- if xurl_x.is_available():
- return "xurl", "oauth2"
+ has_bird_creds = bool(config.get("AUTH_TOKEN") and config.get("CT0"))
+ for backend in x_auto_chain(config):
+ if backend == "bird":
+ # Cookie presence only: the scraper install is not consulted
+ # here (unlike ``x_backend_chain``), so a fresh cookie-bearing
+ # config reports bird before the binary is checked.
+ if not has_bird_creds:
+ continue
+ elif not _x_backend_available(backend, config, has_bird_creds):
+ continue
+ if backend == "bird":
+ return "bird", config.get("_AUTH_TOKEN_SOURCE", "env")
+ return backend, _X_METHOD_LABELS.get(backend, "none")
return None, "none"
@@ -996,14 +1048,30 @@ def get_reddit_source(config: dict[str, Any]) -> str | None:
# xquik — key-based REST X search (XQUIK_API_KEY)
_X_BACKEND_ORDER = ("bird", "xai", "xurl", "xquik")
-# Opt-in backends: never in the unpinned auto chain; require explicit pin.
-# grok is here because a leftover ~/.grok/auth.json must never steal the X
-# lane. Pin LAST30DAYS_X_BACKEND=grok to enable it.
-_X_BACKEND_OPT_IN = ("grok",)
+# Opt-in backends: never in the default unpinned auto chain; require an
+# explicit pin. grok is here because a leftover ~/.grok/auth.json must never
+# steal the X lane. xapi (direct X API v2 with X_BEARER_TOKEN) is here so an
+# ambient bearer exported for some other tool never spends X API credits
+# every time the cookie scraper comes back empty; on an official-only
+# host it is the first rung of the auto chain instead (see _X_OFFICIAL).
+_X_BACKEND_OPT_IN = ("grok", "xapi")
# All known backends (auto chain + opt-in): valid values for the pin var.
_X_BACKEND_KNOWN = _X_BACKEND_ORDER + _X_BACKEND_OPT_IN
+# Licensed / official backends: the unpinned auto chain on an official-only
+# host. xapi = X API v2 with an app-only bearer, xai = xAI's licensed
+# X search, xurl = the X API through X's own CLI.
+_X_OFFICIAL = ("xapi", "xai", "xurl")
+
+# Host self-identification key and the one value that switches the X
+# policy. The engine trusts this key alone: it never infers the host from
+# the home directory, PATH, platform, or agent env vars.
+X_HOST_VAR = 'LAST30DAYS_HOST'
+GROK_BOT_HOST = 'grok-bot'
+# Per-session X connector lane signal: process env only.
+X_HOST_LANE_VAR = 'LAST30DAYS_X_HOST_LANE'
+
# Public routing definitions for the doctor/backend-descriptor layer
# (lib/backends.py). These are aliases for knowledge this module already
# owns — the declared X chain order and the pin/floor env var names — so
@@ -1011,11 +1079,78 @@ _X_BACKEND_KNOWN = _X_BACKEND_ORDER + _X_BACKEND_OPT_IN
X_BACKEND_ORDER = _X_BACKEND_ORDER
X_BACKEND_OPT_IN = _X_BACKEND_OPT_IN
X_BACKEND_KNOWN = _X_BACKEND_KNOWN
+X_OFFICIAL = _X_OFFICIAL
X_BACKEND_PIN_VAR = 'LAST30DAYS_X_BACKEND'
REDDIT_BACKEND_PIN_VAR = 'LAST30DAYS_REDDIT_BACKEND'
REDDIT_SC_MIN_ITEMS_VAR = 'LAST30DAYS_REDDIT_SC_MIN_ITEMS'
+@dataclass(frozen=True)
+class XPolicy:
+ """The host-conditional X routing rule, resolved once per config.
+
+ ``host`` is the normalized ``LAST30DAYS_HOST`` value; ``official_only``
+ is true on a Grok Bot host; ``auto_chain`` is the unpinned chain
+ (``_X_OFFICIAL`` when official-only, else ``_X_BACKEND_ORDER``);
+ ``cookie_discovery`` is false when official-only unless the pin is
+ ``bird``; ``hint_namespace`` (``official`` or ``default``) is derived
+ from ``official_only`` as a plain string so this module never imports
+ ``prescriptions``.
+ """
+
+ host: str
+ official_only: bool
+ auto_chain: tuple[str, ...]
+ cookie_discovery: bool
+
+ @property
+ def hint_namespace(self) -> str:
+ return 'official' if self.official_only else 'default'
+
+
+def x_backend_pin(config: dict[str, Any]) -> str:
+ """The normalized ``LAST30DAYS_X_BACKEND`` pin value ("" when unset)."""
+ return (config.get(X_BACKEND_PIN_VAR) or '').strip().lower()
+
+
+def x_policy(config: dict[str, Any]) -> XPolicy:
+ """Resolve the X policy from ``LAST30DAYS_HOST`` and the pin.
+
+ This is the ONLY place the Grok Bot host string is compared. It reads
+ just the host key, the pin, and the config dict: no platform, PATH, home
+ directory, or agent env-var inspection (the same rule ``x_extras_enabled``
+ follows), and nothing imported from ``lib``. The pin keeps its exclusive
+ semantics on every host and may name any known backend; a ``bird`` pin
+ is the one path that re-enables cookie discovery on an official-only host.
+ """
+ host = str(config.get(X_HOST_VAR) or '').strip().lower()
+ official_only = host == GROK_BOT_HOST
+ pin = x_backend_pin(config)
+ return XPolicy(
+ host=host,
+ official_only=official_only,
+ auto_chain=_X_OFFICIAL if official_only else _X_BACKEND_ORDER,
+ cookie_discovery=(not official_only) or pin == 'bird',
+ )
+
+
+def x_auto_chain(config: dict[str, Any]) -> list[str]:
+ """The unpinned X auto chain for this host, in failover order."""
+ return list(x_policy(config).auto_chain)
+
+
+def x_host_lane_declared(config: dict[str, Any]) -> bool:
+ """True when the hosting model declared the X connector lane.
+
+ ``get_config`` fills ``LAST30DAYS_X_HOST_LANE`` from the process
+ environment only, so a ``.env`` line never declares the lane.
+ Deliberately NOT ``x_pending_browser_auth``: that predicate is false in
+ cookie-read mode by contract, which would leave the envelope path dead at
+ research time. Host-independent: the envelope is accepted anywhere.
+ """
+ return _truthy(config.get(X_HOST_LANE_VAR))
+
+
def _x_backend_available(
backend: str,
config: dict[str, Any],
@@ -1042,6 +1177,9 @@ def _x_backend_available(
return xurl_x.is_available()
if backend == 'xquik':
return is_xquik_available(config)
+ if backend == 'xapi':
+ # Key presence only (no network); local_only needs no branch.
+ return bool(config.get('X_BEARER_TOKEN'))
return False
@@ -1066,25 +1204,34 @@ def x_backend_chain(config: dict[str, Any], local_only: bool = False) -> list[st
answered from local evidence only (no subprocess spawns that reach the
network — xurl's live `whoami` check is replaced by its on-disk token
store). Research-time callers keep the default live semantics.
+
+ The unpinned walk is ``x_policy(config).auto_chain``: the default order
+ above on every host, or ``_X_OFFICIAL`` (xapi -> xai -> xurl) on an
+ official-only host. The scraper is primed with cookies only when bird
+ ends up in the resulting chain (never on an official-only host unless
+ bird is pinned).
"""
- from . import bird_x
has_bird_creds = bool(config.get('AUTH_TOKEN') and config.get('CT0'))
- if has_bird_creds:
- bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
- preferred = (config.get(X_BACKEND_PIN_VAR) or '').lower()
+ preferred = x_backend_pin(config)
# Pin accepted from _X_BACKEND_KNOWN (auto chain + opt-in like grok).
if preferred in _X_BACKEND_KNOWN:
if _x_backend_available(preferred, config, has_bird_creds, local_only):
- return [preferred]
- return []
-
- # Unpinned: walk only _X_BACKEND_ORDER (bird -> xai -> xurl -> xquik).
- # Opt-in backends like grok are never auto-selected.
- return [
- b for b in _X_BACKEND_ORDER
- if _x_backend_available(b, config, has_bird_creds, local_only)
- ]
+ chain = [preferred]
+ else:
+ chain = []
+ else:
+ # Unpinned: walk the policy's auto chain. Opt-in backends (grok, and
+ # xapi off an official-only host) are never auto-selected.
+ chain = [
+ b for b in x_policy(config).auto_chain
+ if _x_backend_available(b, config, has_bird_creds, local_only)
+ ]
+
+ if 'bird' in chain:
+ from . import bird_x
+ bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
+ return chain
def get_x_source(config: dict[str, Any], local_only: bool = False) -> str | None:
@@ -1129,6 +1276,10 @@ def x_pending_browser_auth(config: dict[str, Any], local_only: bool = False) ->
# run has already attempted extraction and must report its true state.
if config.get('_BROWSER_COOKIE_MODE') == 'read':
return False
+ # Cookie-only predicate: on an official-only host no run-time cookie
+ # source exists unless bird is pinned (x_policy), so nothing is pending.
+ if not x_policy(config).cookie_discovery:
+ return False
if 'x' not in COOKIE_DOMAINS:
return False
from . import bird_x
@@ -1444,10 +1595,24 @@ def get_x_source_status(config: dict[str, Any], probe: bool = False) -> dict[str
"""
from . import bird_x
- if config.get('AUTH_TOKEN') and config.get('CT0'):
+ # Backends this host may run: the policy's auto chain plus a known pin.
+ # Bird is primed/probed and xquik is probed only when they are in that
+ # set, so an official-only host never touches the scraper or the
+ # third-party API unless the backend is pinned.
+ policy = x_policy(config)
+ pin = x_backend_pin(config)
+ considered = set(policy.auto_chain)
+ if pin in _X_BACKEND_KNOWN:
+ considered.add(pin)
+
+ if 'bird' in considered and config.get('AUTH_TOKEN') and config.get('CT0'):
bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
- bird_status = bird_x.get_bird_status()
+ bird_status = dict(bird_x.get_bird_status())
+ if 'bird' not in considered:
+ # Never report the scraper as usable where the policy forbids it.
+ bird_status["authenticated"] = False
xai_available = bool(config.get('XAI_API_KEY'))
+ xapi_available = bool(config.get('X_BEARER_TOKEN'))
# Report the TRUE auth lane (browser / env / keychain) rather than the static
# "env AUTH_TOKEN" label — tokens usually come from live browser cookies, and
@@ -1468,7 +1633,7 @@ def get_x_source_status(config: dict[str, Any], probe: bool = False) -> dict[str
xquik_available = is_xquik_available(config)
xquik_working: bool | None = None
xquik_status = ""
- if xquik_available:
+ if xquik_available and 'xquik' in considered:
if probe:
from . import xquik
xquik_working = xquik.probe_works(get_xquik_token(config))
@@ -1492,36 +1657,24 @@ def get_x_source_status(config: dict[str, Any], probe: bool = False) -> dict[str
# Determine active source. A pin forces a single backend (R4): ANY known
# pin is exclusive, mirroring x_backend_chain's [] semantics. Pinned
- # backend available → that source. Pinned backend unavailable → None.
- # Otherwise, order mirrors _X_BACKEND_ORDER: bird first (cookies beat
- # XAI_API_KEY when both are present), then xai, then xurl, then xquik.
- # Grok is opt-in only and never auto-selected; a leftover ~/.grok/auth.json
- # must not steal the X lane.
- pin = (config.get(X_BACKEND_PIN_VAR) or '').lower()
- if pin and pin in _X_BACKEND_KNOWN:
+ # backend available -> that source. Pinned backend unavailable -> None.
+ # Otherwise walk the policy's auto chain (default: bird first, cookies
+ # beat XAI_API_KEY when both are present, then xai, xurl, xquik;
+ # official-only: xapi, xai, xurl). Opt-in backends are never
+ # auto-selected; a leftover ~/.grok/auth.json must not steal the X lane.
+ usable = {
+ 'bird': bird_status["authenticated"],
+ 'xai': xai_available,
+ 'xurl': xurl_available,
+ 'xquik': xquik_available and xquik_working is not False,
+ 'grok': grok_available,
+ 'xapi': xapi_available,
+ }
+ if pin in _X_BACKEND_KNOWN:
# Pin is exclusive: pinned backend if available, else None (no fallback).
- if pin == 'bird':
- source = 'bird' if bird_status["authenticated"] else None
- elif pin == 'xai':
- source = 'xai' if xai_available else None
- elif pin == 'xurl':
- source = 'xurl' if xurl_available else None
- elif pin == 'xquik':
- source = 'xquik' if (xquik_available and xquik_working is not False) else None
- elif pin == 'grok':
- source = 'grok' if grok_available else None
- else:
- source = None
- elif bird_status["authenticated"]:
- source = 'bird'
- elif xai_available:
- source = 'xai'
- elif xurl_available:
- source = 'xurl'
- elif xquik_available and xquik_working is not False:
- source = 'xquik'
+ source = pin if usable.get(pin) else None
else:
- source = None
+ source = next((b for b in policy.auto_chain if usable.get(b)), None)
return {
"source": source,
@@ -1529,6 +1682,7 @@ def get_x_source_status(config: dict[str, Any], probe: bool = False) -> dict[str
"bird_authenticated": bird_status["authenticated"],
"bird_username": bird_status["username"],
"xai_available": xai_available,
+ "xapi_available": xapi_available,
"grok_available": grok_available,
"xurl_available": xurl_available,
"xquik_available": xquik_available,
diff --git a/skills/last30days/scripts/lib/grok_x.py b/skills/last30days/scripts/lib/grok_x.py
index 2d2ff179..60657ce0 100644
--- a/skills/last30days/scripts/lib/grok_x.py
+++ b/skills/last30days/scripts/lib/grok_x.py
@@ -39,6 +39,18 @@ from typing import Any, Dict, List, Optional, Tuple
from . import log
from .relevance import token_overlap_relevance as _compute_relevance
+# One copy of the snowflake, handle-grammar, and generated-sequence helpers
+# lives in x_api; grok_x keeps its private names for its callers and
+# tests. Model-reported handles are interpolated into post URLs and into the
+# NEXT child prompt, so anything outside the X handle charset is rejected
+# rather than passed through (entity_extract applies the same rule).
+from .x_api import ( # noqa: F401 - re-exported under the same names
+ _HANDLE_RE,
+ _SNOWFLAKE_EPOCH_MS,
+ _clean_handle,
+ _decode_snowflake,
+ _looks_generated,
+)
def _log(msg: str) -> None:
@@ -131,9 +143,6 @@ DEPTH_CONFIG = {
# non-daemon worker can outlive a wall-clock budget.
_TIMEOUT_SECONDS = {"quick": 120, "default": 240, "deep": 360}
-# Twitter/X snowflake epoch (2010-11-04T01:42:54.657Z) in milliseconds.
-_SNOWFLAKE_EPOCH_MS = 1288834974657
-
_AUTH_STORE = Path.home() / ".grok" / "auth.json"
# Substrings that indicate stored credentials. Deliberately format-agnostic:
@@ -348,49 +357,6 @@ def _stage_child_home(workdir: str) -> str:
return home
-def _decode_snowflake(post_id: str) -> Optional[datetime]:
- """Recover a post's creation time from its id, with no network call."""
- try:
- value = int(str(post_id).strip())
- except (TypeError, ValueError):
- return None
- if value <= 0:
- return None
- try:
- return datetime.fromtimestamp(
- ((value >> 22) + _SNOWFLAKE_EPOCH_MS) / 1000, tz=timezone.utc
- )
- except (OverflowError, OSError, ValueError):
- return None
-
-
-def _looks_generated(ids: List[str]) -> bool:
- """True when ids form a near-uniform arithmetic run.
-
- Real ranked results are not evenly spaced in time. A fabricated set often
- is, because the model interpolates a plausible-looking id sequence. Four
- ids is the minimum used here: three gaps are needed before a near-uniform
- step reads as generated rather than coincidental.
- """
- numeric = []
- for pid in ids:
- try:
- numeric.append(int(pid))
- except (TypeError, ValueError):
- return False
- if len(numeric) < 4:
- return False
- numeric.sort()
- gaps = [b - a for a, b in zip(numeric, numeric[1:])]
- if any(g <= 0 for g in gaps):
- return False
- mean = sum(gaps) / len(gaps)
- if mean <= 0:
- return False
- # Every gap within 5% of the mean is not something real timelines do.
- return all(abs(g - mean) / mean < 0.05 for g in gaps)
-
-
# Why the most recent parse returned nothing. Lets _run_query distinguish a
# clean empty window (common, and not worth a second LLM call) from a suspect
# response (fabricated ids, a generated sequence, a self-reported
@@ -405,19 +371,6 @@ _RETRYABLE_REJECTIONS = (
_PLACEHOLDER_HANDLES = {"unknown", "n/a", "none", "null", "example", "user", ""}
-# X's real handle grammar. Model-reported handles are interpolated into post
-# URLs and into the NEXT child prompt, so anything outside this charset is
-# rejected rather than passed through: a poisoned post that steers the child
-# into emitting a crafted handle line would otherwise reach a prompt slot it
-# can close. entity_extract applies the same rule to @mentions.
-_HANDLE_RE = re.compile(r"[A-Za-z0-9_]{1,15}")
-
-
-def _clean_handle(value: str) -> str:
- """Return a grammar-valid handle, or '' when the value is not one."""
- candidate = str(value or "").strip().lstrip("@")
- return candidate if _HANDLE_RE.fullmatch(candidate) else ""
-
_NON_EXECUTION_MARKERS = (
"was not executed",
"not executed in this turn",
diff --git a/skills/last30days/scripts/lib/health.py b/skills/last30days/scripts/lib/health.py
index 7d73f925..69f842c4 100644
--- a/skills/last30days/scripts/lib/health.py
+++ b/skills/last30days/scripts/lib/health.py
@@ -34,11 +34,22 @@ NO_RESULTS = "no-results"
PARTIAL = "partial"
RATE_LIMITED = "rate-limited"
AUTH_FAILED = "auth-failed"
+PAYMENT_REQUIRED = "payment-required" # HTTP 402 / credits exhausted: top up, not re-login
UNREACHABLE = "unreachable"
SCHEMA_DRIFT = "schema-drift"
SKIPPED_UNCONFIGURED = "skipped-unconfigured"
+def credits_exhausted_label(source: str) -> str:
+ """Human label for a ``PAYMENT_REQUIRED`` outcome on ``source``.
+
+ The X source names the API whose credits ran out; every other source
+ gets the generic phrasing so the same label serves render summaries and
+ the doctor post-mortem without each surface hand-writing its own.
+ """
+ return "X API credits exhausted" if source == "x" else "credits exhausted"
+
+
@dataclass
class SourceHealth:
"""Typed outcome for a source or the tool backing it.
diff --git a/skills/last30days/scripts/lib/http.py b/skills/last30days/scripts/lib/http.py
index a5a7e101..7c1b5ce3 100644
--- a/skills/last30days/scripts/lib/http.py
+++ b/skills/last30days/scripts/lib/http.py
@@ -194,10 +194,19 @@ def _scrub_fixture_value(
return value
+_AUTH_SCHEME_RE = re.compile(r"^(?:bearer|basic|token)\s+(\S+)$", re.IGNORECASE)
+
+
def _collect_secret_values(value: Any, *, key: str = "") -> set[str]:
values: set[str] = set()
if key and _is_secret_key(key) and value not in (None, ""):
- values.add(str(value))
+ text = str(value)
+ values.add(text)
+ # "Bearer <token>": the bare token is what a response body or an
+ # adapter error message echoes, so redact it on its own too.
+ scheme = _AUTH_SCHEME_RE.match(text.strip())
+ if scheme:
+ values.add(scheme.group(1))
return values
if isinstance(value, dict):
for child_key, child_value in value.items():
@@ -222,9 +231,56 @@ def _fixture_redactions(
pass
values.update(_collect_secret_values(headers))
values.update(_collect_secret_values(json_data))
+ # Session-wide secret values (process env plus resolved config) so a
+ # bearer loaded from .env, Keychain, or pass is scrubbed at the HTTP
+ # seam even when this request carried it only in a header.
+ with _fixture_lock:
+ state = _fixture_state
+ if state is not None and state.get("redactions"):
+ values.update(state["redactions"])
return frozenset(values)
+def config_secret_values(config: Dict[str, Any]) -> frozenset[str]:
+ """Secret VALUES from a resolved config, for fixture redaction.
+
+ Every key the Keychain/pass loader knows (``env.KEYCHAIN_KEYS``) plus any
+ secret-named key is a credential regardless of which layer supplied it.
+ """
+ values: set[str] = set()
+ try:
+ from . import env as _env
+ secret_keys = set(_env.KEYCHAIN_KEYS)
+ except Exception: # pragma: no cover - env is always importable
+ secret_keys = set()
+ for key, value in (config or {}).items():
+ if not isinstance(value, str) or len(value) < 4:
+ continue
+ if "://" in value:
+ # An endpoint (e.g. an API base URL) is an address, not a credential.
+ continue
+ if key in secret_keys or _is_secret_key(key):
+ values.add(value)
+ return frozenset(values)
+
+
+def add_fixture_redactions(values) -> None:
+ """Register secret values with the active recording session, if any.
+
+ ``env.get_config`` calls this with the resolved config's secrets so a
+ bearer loaded from a file or a credential store is redacted at both the
+ HTTP seam and the module seam. A no-op outside a recording session.
+ """
+ extra = {v for v in values if isinstance(v, str) and len(v) >= 4}
+ if not extra:
+ return
+ with _fixture_lock:
+ state = _fixture_state
+ if state is None or state["mode"] != "record":
+ return
+ state["redactions"] = frozenset(state.get("redactions") or frozenset()) | extra
+
+
def _scrub_fixture_url(url: str) -> str:
try:
parts = urlsplit(url)
@@ -281,7 +337,9 @@ def recording_requests(path: str | Path):
"source_exchanges": [],
# Secret VALUES from the environment, so module-seam recordings
# scrub tokens echoed inside normal string fields (adapter error
- # messages, parsed item text), not just secret-named keys.
+ # messages, parsed item text), not just secret-named keys. The
+ # resolved config's secrets join via add_fixture_redactions once
+ # env.get_config runs inside the session.
"redactions": frozenset(
value
for key, value in os.environ.items()
@@ -600,14 +658,28 @@ def classify_failure(*, status_code: Optional[int] = None, message: str = "") ->
marker in text for marker in ("http 429", "status 429", "rate limit", "too many requests")
):
return health.RATE_LIMITED
- if status_code in (401, 402, 403) or any(
+ # Credit exhaustion is checked before the auth branch: a 402 (or a body
+ # saying the account has no credits) asks the user to top up, not to
+ # re-authenticate. Markers stay narrow on purpose: the bare word "credits"
+ # is not one ("10,000 free credits" is onboarding copy, not a failure).
+ if status_code == 402 or any(
marker in text
for marker in (
- "http 401",
"http 402",
+ "status 402",
+ "payment required",
+ "insufficient credits",
+ "does not have any credits",
+ "out of credits",
+ )
+ ):
+ return health.PAYMENT_REQUIRED
+ if status_code in (401, 403) or any(
+ marker in text
+ for marker in (
+ "http 401",
"http 403",
"status 401",
- "status 402",
"status 403",
"unauthorized",
"forbidden",
diff --git a/skills/last30days/scripts/lib/permission_preflight.py b/skills/last30days/scripts/lib/permission_preflight.py
index 5c21db13..0ddbc9c0 100644
--- a/skills/last30days/scripts/lib/permission_preflight.py
+++ b/skills/last30days/scripts/lib/permission_preflight.py
@@ -22,6 +22,10 @@ PROVIDER_CREDENTIALS = {
"perplexity": "Perplexity API key",
"scrapecreators": "ScrapeCreators API key",
"github": "GitHub token or gh auth",
+ # X API v2 app-only bearer (X_BEARER_TOKEN). Presence is computed from
+ # config inside build(), never through diagnose.providers, whose key set
+ # is frozen by tests/test_diagnose_compat.py.
+ "x_bearer": "X API bearer token",
}
@@ -100,6 +104,10 @@ def build(
"label": PROVIDER_CREDENTIALS["scrapecreators"],
},
"github": {"present": bool(diagnose.get("has_github")), "label": PROVIDER_CREDENTIALS["github"]},
+ "x_bearer": {
+ "present": bool(str(config.get("X_BEARER_TOKEN") or "").strip()),
+ "label": PROVIDER_CREDENTIALS["x_bearer"],
+ },
}
active_endpoint_overrides = sorted(
diff --git a/skills/last30days/scripts/lib/pipeline.py b/skills/last30days/scripts/lib/pipeline.py
index 5bc4b301..e1f352f4 100644
--- a/skills/last30days/scripts/lib/pipeline.py
+++ b/skills/last30days/scripts/lib/pipeline.py
@@ -69,6 +69,8 @@ from . import (
topic_shape,
truthsocial,
trustpilot,
+ x_api,
+ x_envelope,
x_judge,
xai_x,
xiaohongshu_api,
@@ -114,11 +116,12 @@ MAX_SOURCE_FETCHES: dict[str, int] = {
_FAILURE_SPECIFICITY = {
health.AUTH_FAILED: 0,
- health.RATE_LIMITED: 1,
- health.SCHEMA_DRIFT: 2,
- health.TIMEOUT: 3,
- health.UNREACHABLE: 4,
- health.ERROR: 5,
+ health.PAYMENT_REQUIRED: 1,
+ health.RATE_LIMITED: 2,
+ health.SCHEMA_DRIFT: 3,
+ health.TIMEOUT: 4,
+ health.UNREACHABLE: 5,
+ health.ERROR: 6,
}
@@ -239,6 +242,8 @@ def available_sources(
*,
x_pending: bool | None = None,
local_only: bool = False,
+ x_envelope: bool = False,
+ suppress_x_host_lane: bool = False,
) -> list[str]:
"""List the sources the next run can serve.
@@ -246,6 +251,13 @@ def available_sources(
block): availability is answered from local evidence only, so the X
check never spawns xurl's live ``whoami`` network call. Research-time
callers keep the default live semantics.
+
+ X is listed when an engine backend is available, or browser auth is
+ pending, or the hosting model declared the X connector lane
+ (``env.x_host_lane_declared``), or a validated ``--x-posts`` envelope is
+ present for this run (``x_envelope``), in every cookie mode.
+ ``suppress_x_host_lane`` turns only the lane branch off (discovery
+ enrichment passes); an envelope still counts.
"""
available: list[str] = []
# reddit_public needs no API key - always available
@@ -258,6 +270,12 @@ def available_sources(
available.extend(["tiktok", "instagram"])
if env.get_x_source(config, local_only=local_only):
available.append("x")
+ elif x_envelope or (
+ not suppress_x_host_lane and env.x_host_lane_declared(config)
+ ):
+ # Host-fetched X lane: the model passes connector results through
+ # --x-posts, so X is served without an engine backend.
+ available.append("x")
else:
# Safe inspection (--diagnose/--preflight) skips browser-cookie
# extraction, so get_x_source is None even though a real run would
@@ -964,6 +982,10 @@ def enrich_nominations(
lookback_days=lookback_days,
as_of_date=as_of_date,
internal_subrun=True,
+ # Enrichment passes never carry a connector envelope, so the
+ # per-session lane signal must not plan X in and record a
+ # spurious X error on every nominated topic.
+ suppress_x_host_lane=True,
)
# Daemon threads + a semaphore instead of ThreadPoolExecutor: executor
@@ -1790,7 +1812,11 @@ def diagnose(
requested_sources: list[str] | None = None,
*,
safe: bool = False,
+ x_envelope: bool = False,
) -> dict[str, Any]:
+ # ``x_envelope`` is True when a validated --x-posts envelope is present for
+ # this invocation, so available_sources lists x even without a backend
+ # and the optional-source omission note does not fire.
requested_sources = normalize_requested_sources(requested_sources)
google_key = _google_key(config)
x_status = env.get_x_source_status(config, probe=not safe)
@@ -1848,7 +1874,11 @@ def diagnose(
"providers": providers_status,
"local_mode": not reasoning_provider_available,
"reasoning_provider": (config.get("LAST30DAYS_REASONING_PROVIDER") or "auto").lower(),
- "x_backend": x_status["source"],
+ # The host-fetched connector lane serves X when no engine backend
+ # exists and the model declared the lane.
+ "x_backend": x_status["source"] or (
+ "connector" if env.x_host_lane_declared(config) else None
+ ),
"bird_installed": x_status["bird_installed"],
"bird_authenticated": x_status["bird_authenticated"],
"bird_username": x_status["bird_username"],
@@ -1866,7 +1896,8 @@ def diagnose(
# answer X availability from local evidence only. x_pending is
# precomputed by diagnose() to avoid double evaluation.
"available_sources": available_sources(
- config, requested_sources, x_pending=x_pending, local_only=safe
+ config, requested_sources, x_pending=x_pending, local_only=safe,
+ x_envelope=x_envelope,
),
"safe": safe,
"config_source": config.get("_CONFIG_SOURCE"),
@@ -2001,10 +2032,17 @@ def run(
trustpilot_domain_is_hint: bool = False,
hiring_signals_mode: bool = False,
internal_subrun: bool = False,
+ suppress_x_host_lane: bool = False,
save_dir: Path | str | None = None,
corpus_dirs: list[str] | None = None,
corpus_all_time: bool = False,
+ x_posts: x_envelope.Envelope | None = None,
) -> schema.Report:
+ # ``suppress_x_host_lane`` is distinct from ``internal_subrun``: comparison
+ # entities share the latter and must still honor the connector lane;
+ # only discovery enrichment passes set the former.
+ # ``x_posts`` is a validated ``--x-posts`` envelope: when present
+ # it replaces the engine's X fetch for this run and is served once.
# Standalone runs (not competitor/discover sub-runs) own the YouTube
# search-cache lifecycle. Comparison fan-out clears once before submit so
# parallel entity sub-runs can still share in-run hits.
@@ -2031,6 +2069,35 @@ def run(
if corpus_enabled and requested_sources and "corpus" not in requested_sources:
requested_sources = [*requested_sources, "corpus"]
+ # Host-fetched X lane. EXCLUDE_SOURCES=x or a --search list without
+ # x wins: the envelope is ignored with a receipt line and stays unconsumed.
+ envelope = x_posts
+ if envelope is not None and (
+ "x" in excluded_sources
+ or (requested_sources and "x" not in requested_sources)
+ ):
+ log.source_log(
+ "x", "host-fetched X: envelope ignored (x is excluded from this run)",
+ tty_only=False,
+ )
+ envelope = None
+ # The lane signal without an envelope is a broken handoff, not a reason to
+ # spend a backup backend: X records the fixed not-passed outcome.
+ x_lane_missing = (
+ envelope is None
+ and not mock
+ and not suppress_x_host_lane
+ and env.x_host_lane_declared(config)
+ )
+ if envelope is not None or x_lane_missing:
+ # Ride the config dict (the _polymarket_keywords idiom) so the stream
+ # workers and the handle-lane section see it without widening their
+ # signatures. Copy first: comparison entities shallow-copy the shared
+ # config and must never inherit another entity's envelope.
+ config = dict(config)
+ config["_x_envelope"] = envelope
+ config["_x_lane_missing"] = x_lane_missing
+
# Gate StockTwits to ticker/crypto topics. Single chokepoint: when False,
# available_sources() never registers stocktwits, so the planner can't
# assign it (eligible_sources = available ∩ capabilities).
@@ -2048,7 +2115,11 @@ def run(
available = [source for source in available if source != "jobs"]
else:
runtime, reasoning_provider = providers.resolve_runtime(config, depth)
- available = available_sources(config, requested_sources)
+ available = available_sources(
+ config, requested_sources,
+ suppress_x_host_lane=suppress_x_host_lane,
+ x_envelope=envelope is not None,
+ )
if requested_sources:
available = [source for source in available if source in requested_sources]
# Keep an explicitly requested but unconfigured corpus in the plan long
@@ -2164,6 +2235,9 @@ def run(
print("[Planner] (no subqueries in plan)", file=sys.stderr)
bundle = schema.RetrievalBundle(artifacts={"grounding": []})
+ if envelope is not None:
+ # The footer's X provenance reads "via X connector" (render._render_stats).
+ bundle.artifacts["x_provenance"] = "connector"
# Handles the user named explicitly. Available before any retrieval, unlike
# the entity-extracted set, so Phase 1 and quick-depth runs get first-party
# protection too. Without this the exemption reached only the Phase 2
@@ -2740,6 +2814,10 @@ def run(
# marking the source PARTIAL would trip LAST30DAYS_STRICT_EXIT on runs that
# returned good X results.
warnings.extend(bundle.artifacts.get("x_partial_coverage", []))
+ # Backend receipts that are not failures (xapi's truncated window).
+ for stream_artifact in bundle.artifacts.get("grounding", []):
+ if isinstance(stream_artifact, dict):
+ warnings.extend(stream_artifact.get("x_receipts", []))
library_context, library_warning = _load_library_context(
topic=topic,
config=config,
@@ -3543,7 +3621,7 @@ def _legacy_artifact_outcome(
return None
-def _summarize_lane_failures(failures: list[http.HTTPError]) -> str:
+def _summarize_lane_failures(failures: list[http.HTTPError], source: str = "") -> str:
"""One line naming what a source lost to swallowed sub-request failures.
``"3 sub-requests rate-limited (HTTP 429); 1 sub-request blocked (HTTP 403)"``.
@@ -3559,6 +3637,7 @@ def _summarize_lane_failures(failures: list[http.HTTPError]) -> str:
labels = {
health.RATE_LIMITED: "rate-limited",
health.AUTH_FAILED: "blocked",
+ health.PAYMENT_REQUIRED: health.credits_exhausted_label(source),
health.TIMEOUT: "timed out",
health.UNREACHABLE: "unreachable",
health.SCHEMA_DRIFT: "returned an unexpected shape",
@@ -3719,10 +3798,26 @@ def _run_supplemental_searches(
resolved_handles_out: list[str] | None = None,
) -> None:
"""Phase 2: extract entities from Phase 1 results, run targeted supplemental searches."""
- if depth == "quick" or mock:
+ from_date, to_date = date_range
+
+ # Host-fetched X lane: the envelope's lane calls replace the backend
+ # lanes and are served at every depth (the host already paid for them),
+ # before the quick/mock return and before the chain is recomputed.
+ # Extracted-handle promotion is skipped on envelope runs; a declared lane
+ # without an envelope runs no lane at all (the topic stream already
+ # recorded the not-passed outcome).
+ if config.get("_x_lane_missing"):
+ return
+ envelope = config.get("_x_envelope")
+ if envelope is not None:
+ _serve_envelope_lanes(
+ envelope, bundle=bundle, plan=plan, x_handle=x_handle, x_related=x_related,
+ from_date=from_date, to_date=to_date,
+ )
return
- from_date, to_date = date_range
+ if depth == "quick" or mock:
+ return
# Convert SourceItems to dicts for entity_extract. All X items (whatever
# backend fetched them — bird, xai, xurl, xquik) land under the single "x"
@@ -3807,7 +3902,7 @@ def _run_supplemental_searches(
return
# Pick the X handle-search backend: the first handle-capable backend in the
- # chain (bird or xquik). These supplemental from:/mentions lanes are
+ # chain (grok, bird, xapi, or xquik). These supplemental from:/mentions lanes are
# complementary to the topic search, so when the topic primary can't run
# them (xai/xurl have no handle-lane implementation) but a capable backend
# is available, use it rather than skipping Phase 2. bird scrapes X GraphQL
@@ -3819,7 +3914,7 @@ def _run_supplemental_searches(
pinned = runtime.x_search_backend
if pinned:
chain = [pinned] + [b for b in chain if b != pinned]
- primary = next((b for b in chain if b in ("grok", "bird", "xquik")), None)
+ primary = next((b for b in chain if b in ("grok", "bird", "xapi", "xquik")), None)
# Name lane (posts naming the subject in plain text, no @-mention) is
# grok-only for now: it needs phrase-quoting and negation operators the
@@ -3866,6 +3961,42 @@ def _run_supplemental_searches(
def _about_lane(hs: list, count: int) -> tuple[list, bool]:
return bird_x.search_mentions(hs, from_date, count_per=count), False
+ elif primary == "xapi":
+ # Direct X API v2 with the app-only bearer: from:/@ lanes run over
+ # search/all with the recent-search fallback. One budget shared by
+ # every lane below (same shape as the grok lanes): a slow key bounds
+ # the whole supplemental phase, not each call.
+ xapi_token = config.get("X_BEARER_TOKEN") or ""
+ xapi_deadline = time.monotonic() + x_api.LANE_BUDGET_SECONDS
+
+ def _xapi_lane_receipt(lane_warnings: list[str]) -> None:
+ # A deadline stop is incomplete coverage, reported in
+ # report.warnings (the x_partial_coverage artifact), never a
+ # healthy-looking silence and never a source failure.
+ sink = bundle.artifacts.setdefault("x_partial_coverage", [])
+ for note in lane_warnings:
+ line = f"X handle lanes: {note}"
+ if line not in sink:
+ sink.append(line)
+
+ def _from_lane(hs: list, count: int, and_topic: bool = False) -> tuple[list, bool]:
+ # x_api.search_handles doesn't support and_topic; topic ranks only
+ lane_warnings: list[str] = []
+ items = x_api.search_handles(
+ hs, topic, from_date, to_date, count_per=count, token=xapi_token,
+ deadline=xapi_deadline, warnings=lane_warnings,
+ )
+ _xapi_lane_receipt(lane_warnings)
+ return items, False
+
+ def _about_lane(hs: list, count: int) -> tuple[list, bool]:
+ lane_warnings: list[str] = []
+ items = x_api.search_mentions(
+ hs, from_date, to_date, topic=topic, count_per=count, token=xapi_token,
+ deadline=xapi_deadline, warnings=lane_warnings,
+ )
+ _xapi_lane_receipt(lane_warnings)
+ return items, False
elif primary == "xquik":
xquik_token = env.get_xquik_token(config)
@@ -4251,9 +4382,13 @@ def _retry_thin_sources(
)
-def _fetch_x_backend(backend, query, from_date, to_date, depth, config):
+def _fetch_x_backend(backend, query, from_date, to_date, depth, config, warnings=None):
"""Fetch X items from a single backend. Returns (items, error_str).
+ ``warnings``, when given, collects backend receipts that are not
+ failures (xapi's "window truncated to 7 days" after the recent-search
+ fallback) so the X branch can surface them as run artifacts.
+
Backends are tried in priority order by the caller (env.x_backend_chain);
a non-empty error_str signals a hard failure (auth/payment/etc.) so the
caller can fail over to the next backend or surface the error honestly.
@@ -4285,6 +4420,14 @@ def _fetch_x_backend(backend, query, from_date, to_date, depth, config):
elif backend == "xquik":
result = xquik.search_xquik(query, from_date, to_date, depth=depth, token=env.get_xquik_token(config))
items = xquik.parse_xquik_response(result)
+ elif backend == "xapi":
+ result = x_api.search_x(config.get("X_BEARER_TOKEN") or "", query, from_date, to_date, depth=depth)
+ items = result.get("items", []) if isinstance(result, dict) else []
+ warning = result.get("warning") if isinstance(result, dict) else None
+ if warning:
+ print(f"[X] xapi: {warning}", file=sys.stderr)
+ if warnings is not None:
+ warnings.append(f"X: xapi {warning}")
else:
return [], f"unknown X backend: {backend}"
err = result.get("error") if isinstance(result, dict) else ""
@@ -4383,7 +4526,9 @@ def _retrieve_stream(*args, **kwargs) -> tuple[list[dict], dict]:
# the most specific failure state so a later empty filter result
# or the thin-source retry can act on it.
artifact = dict(artifact or {})
- artifact["_source_outcome_detail"] = _summarize_lane_failures(failures)
+ artifact["_source_outcome_detail"] = _summarize_lane_failures(
+ failures, str(kwargs.get("source") or "")
+ )
artifact["_source_outcome_detail_state"] = min(
failures, key=lambda f: _FAILURE_SPECIFICITY.get(f.outcome_state, 9)
).outcome_state
@@ -4392,6 +4537,122 @@ def _retrieve_stream(*args, **kwargs) -> tuple[list[dict], dict]:
return items, artifact
+def _serve_envelope_topic(envelope: x_envelope.Envelope) -> tuple[list[dict], dict]:
+ """Serve the envelope's topic-lane rows once.
+
+ The first X subquery takes the rows and the envelope-status outcome;
+ every later call (a second planner subquery, judge-retry, thin-retry)
+ gets no items and no error, and no backend is ever consulted.
+ """
+ items = envelope.take_topic()
+ if items is None:
+ return [], {}
+ artifact: dict[str, Any] = {}
+ if envelope.warnings:
+ # A narrower host window is a receipt (report.warnings), not a failure.
+ artifact["x_receipts"] = [f"X: {warning}" for warning in envelope.warnings]
+ outcome = envelope.outcome()
+ if outcome is not None:
+ state, detail = outcome
+ artifact.update(_outcome_artifact(state, detail))
+ return items, artifact
+
+
+def _serve_envelope_lanes(
+ envelope: x_envelope.Envelope,
+ *,
+ bundle: schema.RetrievalBundle,
+ plan: schema.QueryPlan,
+ x_handle: str | None,
+ x_related: list[str] | None,
+ from_date: str,
+ to_date: str,
+) -> None:
+ """Serve the envelope's from/mention/related calls into the lane merge.
+
+ Mirrors the backend lanes: primary-handle rows (from + mention) join the
+ primary subquery with first-party handling for the explicit handle and
+ the per-handle lane counts; related rows join ``supplemental-related``
+ at the 0.3 weight. Lane claims were already validated at read time.
+ """
+ calls = envelope.take_lanes()
+ if not calls:
+ return
+ x_slug = "x"
+ existing_urls = {
+ item.url
+ for items in bundle.items_by_source.values()
+ for item in items
+ if item.url
+ }
+ ranking_query = plan.subqueries[0].ranking_query if plan.subqueries else ""
+ primary_label = plan.subqueries[0].label if plan.subqueries else "primary"
+ primary_handles = sorted(
+ {x_handle.lstrip("@").strip().lower()} if x_handle and x_handle.strip() else set()
+ )
+ related_handles = [
+ h.lstrip("@").strip().lower()
+ for h in (x_related or [])
+ if h.strip() and h.lstrip("@").strip().lower() not in primary_handles
+ ]
+
+ def _cap_per_author(posts: list[dict], cap: int) -> list[dict]:
+ seen: Counter[str] = Counter()
+ kept: list[dict] = []
+ for post in posts:
+ author = str(post.get("author_handle") or "").lower()
+ if seen[author] >= cap:
+ continue
+ seen[author] += 1
+ kept.append(post)
+ return kept
+
+ primary_items: list[dict] = []
+ related_items: list[dict] = []
+ for call in calls:
+ if call.lane == "from":
+ primary_items.extend(_cap_per_author(call.posts, FROM_LANE_COUNT_PER))
+ elif call.lane == "mention":
+ primary_items.extend(
+ call.posts[: MENTION_LANE_COUNT_PER * max(1, len(call.handles))]
+ )
+ elif call.lane == "related":
+ related_items.extend(_cap_per_author(call.posts, RELATED_HANDLE_COUNT_PER))
+
+ if primary_items:
+ normalized = _normalize_score_dedupe(
+ x_slug, primary_items, from_date, to_date,
+ freshness_mode=plan.freshness_mode,
+ ranking_query=ranking_query,
+ first_party_handles=primary_handles,
+ )
+ normalized = [item for item in normalized if item.url not in existing_urls]
+ if normalized:
+ bundle.add_items(primary_label, x_slug, normalized)
+ existing_urls.update(item.url for item in normalized if item.url)
+
+ if related_items:
+ normalized = _normalize_score_dedupe(
+ x_slug, related_items, from_date, to_date,
+ freshness_mode=plan.freshness_mode,
+ ranking_query=ranking_query,
+ first_party_handles=related_handles,
+ )
+ normalized = [item for item in normalized if item.url not in existing_urls]
+ if normalized:
+ bundle.add_items("supplemental-related", x_slug, normalized)
+ if not any(sq.label == "supplemental-related" for sq in plan.subqueries):
+ plan.subqueries.append(
+ schema.SubQuery(
+ label="supplemental-related",
+ search_query=", ".join(related_handles),
+ ranking_query=ranking_query,
+ sources=[x_slug],
+ weight=0.3,
+ )
+ )
+
+
def _retrieve_stream_impl(
*,
topic: str,
@@ -4562,6 +4823,15 @@ def _retrieve_stream_impl(
)
return merged, {}
if source == "x":
+ if config.get("_x_lane_missing"):
+ # The model declared the connector lane but passed no envelope.
+ return [], _outcome_artifact(health.ERROR, x_envelope.DETAIL_NOT_PASSED)
+ envelope = config.get("_x_envelope")
+ if envelope is not None:
+ # Host-fetched lane: the envelope replaces the backend chain and
+ # is single-serve, so no backend runs and no judge-retry follows.
+ return _serve_envelope_topic(envelope)
+
# Compile X query from raw_topic (like Reddit/YouTube), not planner's
# search_query which may contain operator strings like "Rome Italy".
x_query = raw_topic or topic or subquery.search_query
@@ -4578,13 +4848,22 @@ def _retrieve_stream_impl(
if not chain:
raise RuntimeError("No X backend is available.")
last_error = ""
+ chain_errors: list[str] = []
items = []
used_backend = None
+ x_warnings: list[str] = []
for i, backend in enumerate(chain):
- items, err = _fetch_x_backend(backend, x_query, from_date, to_date, depth, config)
+ items, err = _fetch_x_backend(
+ backend, x_query, from_date, to_date, depth, config, warnings=x_warnings,
+ )
if items:
if i > 0:
- print(f"[X] primary backend(s) returned nothing; used fallback '{backend}'", file=sys.stderr)
+ # xapi is metered: name the spend when it served as a backup.
+ spend = " (spends X API credits)" if backend == "xapi" else ""
+ print(
+ f"[X] primary backend(s) returned nothing; used fallback '{backend}'{spend}",
+ file=sys.stderr,
+ )
# Check for auth errors before proceeding to judge-retry
if last_error:
# Fallback succeeded after earlier backend failed. Classify
@@ -4625,9 +4904,17 @@ def _retrieve_stream_impl(
break
if err:
last_error = f"{backend}: {err}"
+ chain_errors.append(last_error)
print(f"[X] backend '{backend}' failed ({err}); trying next", file=sys.stderr)
if not items and last_error:
+ # A credit-exhaustion failure earlier in the chain is the most
+ # specific outcome (top up, not re-authenticate); a later
+ # backend's generic failure must not mask it.
+ for candidate in chain_errors:
+ if http.classify_failure(message=candidate) == health.PAYMENT_REQUIRED:
+ last_error = candidate
+ break
state = (
bird_x.classify_run_failure(last_error)
if last_error.startswith("bird:")
@@ -4638,6 +4925,11 @@ def _retrieve_stream_impl(
# Retrieve-judge-retry: judge corpus and retry if off-topic flood.
# Skip retry on quick/mock (same as Phase 2).
artifact = {}
+ if x_warnings:
+ # e.g. xapi's "window truncated to 7 days": a receipt that reaches
+ # report.warnings (see the grounding artifacts walk in
+ # _build_report), never a source failure.
+ artifact["x_receipts"] = list(x_warnings)
if items and depth != "quick" and not mock:
items_for_judge = [
{"author_handle": it.get("author_handle", ""), "text": it.get("text", "")}
diff --git a/skills/last30days/scripts/lib/prescriptions.py b/skills/last30days/scripts/lib/prescriptions.py
index bfae5ebc..fc647995 100644
--- a/skills/last30days/scripts/lib/prescriptions.py
+++ b/skills/last30days/scripts/lib/prescriptions.py
@@ -31,7 +31,8 @@ from __future__ import annotations
from dataclasses import dataclass, replace
from typing import Dict, Optional, Tuple
-from . import health
+from . import env, health
+from .x_api import BEARER_COVERAGE_NOTE
# Direct engine invocation prefix (scripting fallback; the slash-command UX
# is "ask the agent to run setup ...", which is the natural-language form).
@@ -107,6 +108,60 @@ REGISTRY: Dict[Tuple[str, str], Prescription] = dict((
fix_cli="grok login",
anchor="api-keys-env",
),
+ # Official X path (an official-only host per env.x_policy, or an explicit
+ # xapi pin). Copy is limited to the connector lane, X_BEARER_TOKEN,
+ # XAI_API_KEY, and X API credits; the bearer path is described as
+ # about a week, never as parity with the connector. Anchors point at
+ # the CONFIGURATION.md Grok Bot subsection (slug grok-bot).
+ _entry(
+ "x", "bearer_missing",
+ cause=(
+ "no official X path is configured (X connector, X_BEARER_TOKEN, "
+ "or XAI_API_KEY)"
+ ),
+ fix_nl=(
+ "add the X for Grok Bot plugin and connect X in Grok Bot settings (full 30-day coverage), or set "
+ f"X_BEARER_TOKEN from the X developer console ({BEARER_COVERAGE_NOTE}), "
+ "or set XAI_API_KEY from console.x.ai"
+ ),
+ fix_cli="X_BEARER_TOKEN=<your-x-api-bearer-token>",
+ anchor="grok-bot",
+ ),
+ _entry(
+ "x", "bearer_invalid",
+ cause="the X API rejected X_BEARER_TOKEN (401/403)",
+ fix_nl=(
+ "set a valid X_BEARER_TOKEN from the X developer console "
+ f"({BEARER_COVERAGE_NOTE}), or add the X for Grok Bot plugin and connect X in Grok Bot settings "
+ "(full 30-day coverage)"
+ ),
+ fix_cli="X_BEARER_TOKEN=<your-x-api-bearer-token>",
+ anchor="grok-bot",
+ ),
+ _entry(
+ "x", "payment_required",
+ cause="X API credits are exhausted (HTTP 402)",
+ fix_nl=(
+ "top up X API credits in the X developer console, or connect X in "
+ "Grok Bot settings (full 30-day coverage)"
+ ),
+ fix_cli="X_BEARER_TOKEN=<bearer-from-a-project-with-credits>",
+ anchor="grok-bot",
+ ),
+ _entry(
+ "x", "connector_missing",
+ cause=(
+ "the X connector lane was declared but no connector result was "
+ "passed to the engine"
+ ),
+ fix_nl=(
+ "add the X for Grok Bot plugin and connect X in Grok Bot settings (full 30-day coverage) and pass the "
+ "connector's posts with --x-posts, or set X_BEARER_TOKEN from the X "
+ f"developer console ({BEARER_COVERAGE_NOTE})"
+ ),
+ fix_cli=f'{ENGINE_CLI} "<topic>" --x-posts <path-to-x-posts.json>',
+ anchor="grok-bot",
+ ),
_entry(
"scrapecreators", "key_missing",
cause="SCRAPECREATORS_API_KEY is not set",
@@ -242,6 +297,31 @@ def lookup(source: str, failure: str) -> Optional[Prescription]:
return REGISTRY.get((source, failure))
+# Failure names remapped on an official-only host (env.x_policy): every
+# cookie or Grok CLI failure has an official-path counterpart, so no fix
+# line there names cookies, the scraper, or the CLI. Official names
+# pass through unchanged on every host.
+_OFFICIAL_X_FAILURES: Dict[str, str] = {
+ "cookies_missing": "bearer_missing",
+ "cookies_expired": "bearer_invalid",
+ "grok_cli_missing": "bearer_missing",
+ "grok_not_authenticated": "bearer_missing",
+}
+
+
+def for_x(config: dict, failure: str) -> Prescription:
+ """Policy-aware X prescription.
+
+ On an official-only host (``env.x_policy(config).hint_namespace ==
+ "official"``) a default-namespace failure resolves to its official
+ counterpart; elsewhere today's entry is returned unchanged. Never
+ raises (falls through to ``get``'s generic fallback).
+ """
+ if env.x_policy(config or {}).hint_namespace == "official":
+ failure = _OFFICIAL_X_FAILURES.get(failure, failure)
+ return get("x", failure)
+
+
def get(source: str, failure: str) -> Prescription:
"""Return the registered entry, or the generic CONFIGURATION.md fallback.
diff --git a/skills/last30days/scripts/lib/quality_nudge.py b/skills/last30days/scripts/lib/quality_nudge.py
index 6257bc79..826da8e1 100644
--- a/skills/last30days/scripts/lib/quality_nudge.py
+++ b/skills/last30days/scripts/lib/quality_nudge.py
@@ -8,9 +8,9 @@ vocabulary shared with the doctor command, KTD 7); only the trigger
logic and the message framing live here.
"""
-from typing import List
+from typing import List, Optional
-from . import prescriptions
+from . import env, health, http, prescriptions, x_envelope
# Sources whose absence can justify a post-run quality repair. X remains a
@@ -42,12 +42,42 @@ def _is_x_active(config: dict, research_results: dict) -> bool:
def _has_x_credentials(config: dict) -> bool:
- """Return True when any X/Twitter source credential is configured."""
- return bool(
+ """Return True when any X/Twitter source credential is configured.
+
+ ``X_BEARER_TOKEN`` counts where the xapi backend can actually run: an
+ official-only host (it heads the auto chain there) or an explicit xapi
+ pin. An ambient bearer on any other host is not a configured X source,
+ so nothing changes there.
+ """
+ if (
config.get("AUTH_TOKEN")
or config.get("XAI_API_KEY")
or config.get("XQUIK_API_KEY")
- )
+ ):
+ return True
+ if config.get("X_BEARER_TOKEN"):
+ return "xapi" in env.x_auto_chain(config) or env.x_backend_pin(config) == "xapi"
+ return False
+
+
+def _x_error_prescription(config: dict, research_results: dict) -> prescriptions.Prescription:
+ """The fix for a configured X that errored, routed through the X policy.
+
+ On an official-only host a credit-exhaustion error asks for a top-up
+ (``payment_required``) and every other error asks for a valid bearer or
+ the connector (``cookies_expired`` maps to ``bearer_invalid`` there);
+ elsewhere the entry is today's ``cookies_expired``.
+ """
+ message = str(research_results.get("x_error") or "")
+ if message.startswith(x_envelope.DETAIL_NOT_PASSED):
+ # The model declared the X connector lane and passed no envelope:
+ # the fix is the connector, on any host.
+ return prescriptions.for_x(config, "connector_missing")
+ failure = "cookies_expired"
+ if env.x_policy(config).hint_namespace == "official":
+ if http.classify_failure(message=message) == health.PAYMENT_REQUIRED:
+ failure = "payment_required"
+ return prescriptions.for_x(config, failure)
def _has_ytdlp() -> bool:
@@ -240,6 +270,7 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
has_sc = bool(config.get("SCRAPECREATORS_API_KEY"))
active_sources = research_results.get("active_sources") or []
+ x_fix = _x_error_prescription(config, research_results) if "x" in core_errored else None
nudge_text = _build_nudge_text(
core_missing,
core_errored,
@@ -250,6 +281,7 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
bonus_errored=bonus_errored,
has_ytdlp=has_ytdlp,
core_total=scored_source_count,
+ x_fix=x_fix,
) if (core_missing or core_degraded or bonus_errored) else None
return {
@@ -273,6 +305,7 @@ def _build_nudge_text(
bonus_errored: List[str] = None,
has_ytdlp: bool = False,
core_total: int | None = None,
+ x_fix: Optional[prescriptions.Prescription] = None,
) -> str:
"""Build human-readable nudge text describing what was missed or degraded.
@@ -313,7 +346,8 @@ def _build_nudge_text(
# core_missing: unconfigured/declined X is an optional omission and never
# lands here. Surface the repair instead of hiding the outage.
if "x" in core_missing and "x" in core_errored:
- x_fix = prescriptions.get("x", "cookies_expired")
+ if x_fix is None:
+ x_fix = prescriptions.get("x", "cookies_expired")
free_suggestions.append(f"X/Twitter errored - {x_fix.fix_nl}.")
if "youtube" in core_missing:
diff --git a/skills/last30days/scripts/lib/render.py b/skills/last30days/scripts/lib/render.py
index 1b6e0ce3..d69e72bd 100644
--- a/skills/last30days/scripts/lib/render.py
+++ b/skills/last30days/scripts/lib/render.py
@@ -2590,6 +2590,8 @@ def _format_outcome(outcome: schema.SourceOutcome) -> str:
summary += ", some requests rate-limited"
elif state == schema.NO_RESULTS:
summary = "no results"
+ elif state == schema.PAYMENT_REQUIRED:
+ summary = health.credits_exhausted_label(outcome.source)
else:
summary = state
if detail:
@@ -3124,6 +3126,9 @@ def _render_stats(report: schema.Report) -> list[str]:
actor_summary = _top_actor_summary(source, items)
if actor_summary:
parts.append(actor_summary)
+ if source == "x" and report.artifacts.get("x_provenance") == "connector":
+ # Host-fetched lane (--x-posts): name the provenance in the footer.
+ parts.append("via X connector")
lines.append(f"- {_source_label(source)}: {' | '.join(parts)}")
lines.append("")
return lines
diff --git a/skills/last30days/scripts/lib/schema.py b/skills/last30days/scripts/lib/schema.py
index f878c117..5035d2ea 100644
--- a/skills/last30days/scripts/lib/schema.py
+++ b/skills/last30days/scripts/lib/schema.py
@@ -39,7 +39,7 @@ class ProviderRuntime:
reasoning_provider: Literal["gemini", "openai", "xai", "local"]
planner_model: str
rerank_model: str
- x_search_backend: Literal["xai", "grok", "bird", "xurl", "xquik"] | None = None
+ x_search_backend: Literal["xai", "grok", "bird", "xurl", "xquik", "xapi"] | None = None
@dataclass(frozen=True)
@@ -149,6 +149,7 @@ RunOutcomeState = Literal[
"partial",
"rate-limited",
"auth-failed",
+ "payment-required",
"unreachable",
"timeout",
"schema-drift",
@@ -167,6 +168,7 @@ NO_RESULTS = health.NO_RESULTS
PARTIAL = health.PARTIAL
RATE_LIMITED = health.RATE_LIMITED
AUTH_FAILED = health.AUTH_FAILED
+PAYMENT_REQUIRED = health.PAYMENT_REQUIRED
UNREACHABLE = health.UNREACHABLE
SCHEMA_DRIFT = health.SCHEMA_DRIFT
SKIPPED_UNCONFIGURED = health.SKIPPED_UNCONFIGURED
@@ -207,6 +209,7 @@ class SourceOutcome:
PARTIAL,
RATE_LIMITED,
AUTH_FAILED,
+ PAYMENT_REQUIRED,
UNREACHABLE,
SCHEMA_DRIFT,
SKIPPED_UNCONFIGURED,
@@ -689,7 +692,7 @@ def candidate_primary_item(candidate: Candidate) -> SourceItem | None:
return candidate.source_items[0]
-AGENT_EXPORT_SCHEMA_VERSION = "1.2"
+AGENT_EXPORT_SCHEMA_VERSION = "1.3"
def without_sources(report: Report, excluded_sources: set[str]) -> Report:
diff --git a/skills/last30days/scripts/lib/setup_wizard.py b/skills/last30days/scripts/lib/setup_wizard.py
index 138c79e3..df7d2973 100644
--- a/skills/last30days/scripts/lib/setup_wizard.py
+++ b/skills/last30days/scripts/lib/setup_wizard.py
@@ -61,6 +61,12 @@ def render_welcome() -> str:
return _WELCOME_TEXT
+# Neutral note recorded on an official-only host instead of a cookie scan.
+# Deliberately names no cookie mechanism beyond the fact that none is used
+# (the vocabulary rule for Grok Bot onboarding output).
+OFFICIAL_HOST_COOKIE_NOTE = "browser sessions are not read on this host"
+
+
def run_auto_setup(config: Dict[str, Any], *, allow_browser_cookies: bool = False) -> Dict[str, Any]:
"""Perform the auto-setup actions.
@@ -73,7 +79,10 @@ def run_auto_setup(config: Dict[str, Any], *, allow_browser_cookies: bool = Fals
Returns:
Dict with keys:
cookies_found: {source_name: browser_name} for each source where cookies were found
- browser_cookie_scan_attempted: bool (True only after explicit consent)
+ browser_cookie_scan_attempted: bool (True only after explicit consent
+ AND on a host whose X policy permits cookie discovery)
+ cookie_note: present only on an official-only host, where the scan
+ is skipped for every domain (neutral, relayable text)
ytdlp_installed: bool
ytdlp_action: already_installed | installed | install_failed | no_homebrew
digg_installed: bool (True when the engine can resolve digg-pp-cli on PATH)
@@ -83,9 +92,19 @@ def run_auto_setup(config: Dict[str, Any], *, allow_browser_cookies: bool = Fals
digg_stderr: present when digg_action is install_failed
digg_path: present when digg_action is installed_off_path (binary on disk, not on PATH)
"""
- from .env import COOKIE_DOMAINS, cookie_extraction_browsers
+ from .env import COOKIE_DOMAINS, cookie_extraction_browsers, x_policy
cookies_found: Dict[str, str] = {}
+ cookie_note: Optional[str] = None
+
+ # Official-only host (LAST30DAYS_HOST=grok-bot): the consented
+ # cookie scan is skipped for EVERY domain (X and Truth Social alike) and
+ # recorded as not attempted, with a neutral note the caller can relay.
+ # The free CLI installs below still run. A LAST30DAYS_X_BACKEND=bird pin
+ # is the one path that re-enables discovery, via x_policy.
+ if allow_browser_cookies and not x_policy(config).cookie_discovery:
+ allow_browser_cookies = False
+ cookie_note = OFFICIAL_HOST_COOKIE_NOTE
if allow_browser_cookies:
from . import cookie_extract
@@ -164,13 +183,15 @@ def run_auto_setup(config: Dict[str, Any], *, allow_browser_cookies: bool = Fals
# (arxiv, techmeme, trustpilot): {source: {installed, action, ...}}.
"pp_sources": pp_sources,
# Reported, never installed: this CLI spends the user's own metered
- # credits, so acquiring it stays their decision (U5/R11). Passing
+ # credits, so acquiring it stays their decision. Passing
# config matters: a user whose key lives in a .env file or the
# keychain (rather than a `brightdata login` credentials file) is
# active in the engine, and setup must not tell them otherwise.
"brightdata": brightdata_status(config),
"env_written": False,
}
+ if cookie_note:
+ results["cookie_note"] = cookie_note
if ytdlp_action == "install_failed":
results["ytdlp_stderr"] = brew_stderr
if digg_action == "install_failed":
@@ -505,6 +526,42 @@ def _open_secret_append(path: Path):
return os.fdopen(fd, "a", encoding="utf-8")
+def _replace_env_line(env_path: Path, content: str, key_name: str, value: str) -> bool:
+ """Rewrite every ``key_name=`` line of ``content`` with ``value`` as a 0o600 secret.
+
+ The new file is written to a sibling temp path opened at 0o600 and moved
+ over the original, so the secret never has a readable window and a
+ crash mid-write leaves the old file intact.
+ """
+ new_line = f"{key_name}={_format_env_value(value)}"
+ lines = []
+ replaced = False
+ for line in content.splitlines():
+ stripped = line.strip()
+ is_key = (
+ stripped and not stripped.startswith("#") and "=" in stripped
+ and stripped.split("=", 1)[0].strip() == key_name
+ )
+ if is_key:
+ if not replaced:
+ lines.append(new_line)
+ replaced = True
+ continue
+ lines.append(line)
+ if not replaced:
+ lines.append(new_line)
+ tmp_path = env_path.with_name(env_path.name + ".tmp")
+ fd = os.open(tmp_path, os.O_CREAT | os.O_WRONLY | os.O_TRUNC, 0o600)
+ with os.fdopen(fd, "w", encoding="utf-8") as f:
+ f.write("\n".join(lines) + "\n")
+ os.replace(tmp_path, env_path)
+ try:
+ os.chmod(env_path, 0o600)
+ except OSError:
+ pass
+ return True
+
+
def _format_env_value(value: str) -> str:
"""Quote a value so it round-trips through env.load_env_file.
@@ -584,19 +641,28 @@ def write_setup_config(env_path: Path, from_browser: str | None = None) -> bool:
return False
-def write_api_key(env_path: Path, api_key: str, key_name: str = "SCRAPECREATORS_API_KEY") -> bool:
+def write_api_key(
+ env_path: Path,
+ api_key: str,
+ key_name: str = "SCRAPECREATORS_API_KEY",
+ *,
+ replace: bool = False,
+) -> bool:
"""Append an API key to the .env file as a 0o600 secret.
Reuses the same secret-safe write path as ``write_setup_config`` so the
value lands with restrictive permissions and round-trips through
- ``env.load_env_file``. Idempotent: if ``key_name`` is already present in
- the file, nothing is written and the existing value is preserved (we never
- clobber a key the user may have set by hand).
+ ``env.load_env_file``. Idempotent by default: if ``key_name`` is already
+ present in the file, nothing is written and the existing value is
+ preserved (we never clobber a key the user may have set by hand). With
+ ``replace=True`` an existing line is rewritten in place instead, so an
+ explicit ``setup --store-key`` can rotate a rejected credential.
Args:
env_path: Path to the .env file (e.g. ~/.config/last30days/.env).
api_key: The raw key value to persist.
key_name: The env var name to write (default SCRAPECREATORS_API_KEY).
+ replace: Rewrite an existing ``key_name`` line instead of keeping it.
Returns:
True if the key was written or already present, False on error or when
@@ -615,6 +681,8 @@ def write_api_key(env_path: Path, api_key: str, key_name: str = "SCRAPECREATORS_
stripped = line.strip()
if stripped and not stripped.startswith("#") and "=" in stripped:
if stripped.split("=", 1)[0].strip() == key_name:
+ if replace:
+ return _replace_env_line(env_path, existing_content, key_name, api_key)
return True # Already configured; do not duplicate
line = f"{key_name}={_format_env_value(api_key)}\n"
@@ -767,10 +835,19 @@ def get_setup_status_text(results: Dict[str, Any]) -> str:
"requests/month). Install with: npm i -g @brightdata/cli && brightdata login"
)
+ cookie_note = results.get("cookie_note")
+ if cookie_note:
+ # Official-only host: relay the neutral note; say nothing about
+ # browsers. The scan was not attempted, so nothing is "found".
+ lines.append(f" - {cookie_note}")
+
env_written = results.get("env_written", False)
if env_written:
lines.append("")
- lines.append("Configuration saved. Future runs will auto-detect your browsers.")
+ if cookie_note:
+ lines.append("Configuration saved.")
+ else:
+ lines.append("Configuration saved. Future runs will auto-detect your browsers.")
return "\n".join(lines)
diff --git a/skills/last30days/scripts/lib/x_api.py b/skills/last30days/scripts/lib/x_api.py
new file mode 100644
index 00000000..76adf376
--- /dev/null
+++ b/skills/last30days/scripts/lib/x_api.py
@@ -0,0 +1,780 @@
+"""Direct X API v2 backend (``xapi``) and the shared v2 response parser.
+
+``xapi`` searches X through ``api.x.com/2`` with an app-only bearer token
+(``X_BEARER_TOKEN``). Full-archive search is tried first; when the developer
+project is not enrolled for it (HTTP 403 with an enrollment marker), the
+search retries once against recent search with the window clamped to the
+last seven days and the truncation named in the result.
+
+This module also owns the one X API v2 parser (``parse_v2_response``) that
+``xurl_x`` delegates to, plus the snowflake, handle-grammar, and
+generated-sequence helpers that ``grok_x`` imports back.
+
+Security contract: the bearer travels only in the Authorization
+header; every failure becomes an engine-authored fixed string chosen by
+status code plus a marker match on the body, and the response body, reason
+phrase, and headers never enter an error string, a log line, or an
+exception. Topic and handle text is sanitized before it enters a query, and
+the research window is always carried by request parameters.
+"""
+
+from __future__ import annotations
+
+import re
+import time
+from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
+from datetime import datetime, timedelta, timezone
+from typing import Any, Callable, Dict, List, Optional
+
+from . import health, http, log
+from .relevance import token_overlap_relevance as _compute_relevance
+
+_API_BASE = "https://api.x.com/2"
+_SEARCH_ALL_URL = f"{_API_BASE}/tweets/search/all"
+_SEARCH_RECENT_URL = f"{_API_BASE}/tweets/search/recent"
+
+# Depth configurations: number of posts to collect per query (shared with
+# xurl_x, which re-imports this table). The API minimum per page is 10.
+DEPTH_CONFIG = {
+ "quick": 10,
+ "default": 30,
+ "deep": 60,
+}
+
+# X API v2 caps a search query at 512 characters (Basic) and pages at 10..100.
+MAX_QUERY_CHARS = 512
+MIN_PAGE_RESULTS = 10
+MAX_PAGE_RESULTS = 100
+# Recent search reaches back seven days; end_time must be at least ten
+# seconds before now, so a "today" window keeps a small safety margin.
+RECENT_WINDOW_DAYS = 7
+END_TIME_SAFETY_SECONDS = 30
+TIMEOUT_SECONDS = 30
+RETRIES = 2
+# Wall-clock budget for one search (all pages, including the recent-search
+# fallback). A throttled or slow walk returns what it has instead of holding
+# the whole run; the source's per-request timeout still bounds each page.
+DEADLINE_SECONDS = 90
+# One budget shared by every supplemental handle lane of a run (explicit
+# from, extracted from, about, related), started once by the caller: the
+# point is to bound the total, not each part.
+LANE_BUDGET_SECONDS = 150.0
+
+TRUNCATION_DETAIL = "window truncated to 7 days"
+# Receipt when the lane budget stopped a search before ``count`` posts were
+# collected: the posts kept are real, the coverage is not complete.
+DEADLINE_DETAIL = "search stopped at the lane deadline; results may be incomplete"
+
+# The one description of what an app-only bearer can reach. Doctor and the
+# prescriptions quote it; the bearer path is never described as parity with
+# the connector lane.
+BEARER_COVERAGE_NOTE = (
+ "recent posts, about the last week, unless your X developer project has "
+ "full-archive access"
+)
+
+# Engine-authored fixed error strings. Each carries a marker that
+# http.classify_failure recognizes, because the X retrieval branch classifies
+# by message text only.
+ERR_PAYMENT_REQUIRED = "xapi: payment required (X API credits exhausted)"
+ERR_UNAUTHORIZED = "xapi: unauthorized (bearer token rejected)"
+ERR_FORBIDDEN = "xapi: forbidden (bearer token lacks access)"
+ERR_RATE_LIMITED = "xapi: rate limit exceeded (X API)"
+ERR_TIMED_OUT = "xapi: timed out"
+ERR_UNREACHABLE = "xapi: connection error (X API unreachable)"
+ERR_NO_TOKEN = "No X_BEARER_TOKEN configured"
+ERR_EMPTY_QUERY = "xapi: empty query after sanitizing"
+
+# Failures that end every remaining lane for the run (the key itself is the
+# problem, not this request).
+_FATAL_ERRORS = frozenset({ERR_PAYMENT_REQUIRED, ERR_UNAUTHORIZED, ERR_FORBIDDEN})
+
+# A 403 whose body carries one of these is "this project is not enrolled for
+# full-archive search", which recent search can still serve.
+_ENROLLMENT_MARKERS = (
+ "client-not-enrolled",
+ "not enrolled",
+ "enrolled",
+ "access level",
+ "subset of",
+)
+# X-specific body markers for credit exhaustion, matched on any status on
+# top of http.classify_failure's shared payment-required markers: X reports
+# a depleted pay-per-use balance under more than one code and title.
+_X_CREDIT_MARKERS = ("creditsdepleted", "credits depleted")
+
+# Twitter/X snowflake epoch (2010-11-04T01:42:54.657Z) in milliseconds.
+_SNOWFLAKE_EPOCH_MS = 1288834974657
+
+# X's real handle grammar. Handles are interpolated into post URLs and into
+# search operators (from:, @), so anything outside this charset is rejected
+# rather than passed through. entity_extract applies the same rule to
+# @mentions.
+_HANDLE_RE = re.compile(r"[A-Za-z0-9_]{1,15}")
+
+# Grouping syntax that carries no lexical meaning in a topic, mirroring
+# bird_x. Double quotes are handled separately: xapi strips them all and
+# wraps the core in exactly one phrase pair, so a planner quote can never
+# close the phrase early.
+_GROUPING_CHARS = "“”()[]{}"
+_QUOTE_CHARS = "\"„‟"
+_APOSTROPHES = "'‘’"
+
+
+def _log(msg: str) -> None:
+ log.source_log("xapi", msg, tty_only=False)
+
+
+def _utcnow() -> datetime:
+ """Current UTC time; a seam tests patch to pin the window arithmetic."""
+ return datetime.now(timezone.utc)
+
+
+# ---------------------------------------------------------------------------
+# Shared helpers (moved here from grok_x; grok_x imports them back)
+# ---------------------------------------------------------------------------
+
+
+def _decode_snowflake(post_id: str) -> Optional[datetime]:
+ """Recover a post's creation time from its id, with no network call."""
+ try:
+ value = int(str(post_id).strip())
+ except (TypeError, ValueError):
+ return None
+ if value <= 0:
+ return None
+ try:
+ return datetime.fromtimestamp(
+ ((value >> 22) + _SNOWFLAKE_EPOCH_MS) / 1000, tz=timezone.utc
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+
+
+def _looks_generated(ids: List[str]) -> bool:
+ """True when ids form a near-uniform arithmetic run.
+
+ Real ranked results are not evenly spaced in time. A fabricated set often
+ is, because a model interpolates a plausible-looking id sequence. Four
+ ids is the minimum used here: three gaps are needed before a near-uniform
+ step reads as generated rather than coincidental.
+ """
+ numeric = []
+ for pid in ids:
+ try:
+ numeric.append(int(pid))
+ except (TypeError, ValueError):
+ return False
+ if len(numeric) < 4:
+ return False
+ numeric.sort()
+ gaps = [b - a for a, b in zip(numeric, numeric[1:])]
+ if any(g <= 0 for g in gaps):
+ return False
+ mean = sum(gaps) / len(gaps)
+ if mean <= 0:
+ return False
+ # Every gap within 5% of the mean is not something real timelines do.
+ return all(abs(g - mean) / mean < 0.05 for g in gaps)
+
+
+def _clean_handle(value: str) -> str:
+ """Return a grammar-valid handle, or '' when the value is not one."""
+ candidate = str(value or "").strip().lstrip("@")
+ return candidate if _HANDLE_RE.fullmatch(candidate) else ""
+
+
+# ---------------------------------------------------------------------------
+# Parser
+# ---------------------------------------------------------------------------
+
+
+def _safe_int(value: Any) -> Optional[int]:
+ if value is None:
+ return None
+ try:
+ return int(value)
+ except (ValueError, TypeError):
+ return None
+
+
+def parse_v2_response(
+ response: Dict[str, Any],
+ topic: str = "",
+ window: Optional[tuple[str, str]] = None,
+ *,
+ id_prefix: str = "XAPI",
+ index_offset: int = 0,
+ seen_ids: Optional[set[str]] = None,
+) -> List[Dict[str, Any]]:
+ """Parse an X API v2 search response into the engine's X item shape.
+
+ Output matches the XItem schema used by xai_x and bird_x (id, text, url,
+ author_handle, date, engagement, mentioned_handles, why_relevant,
+ relevance) plus ``post_id`` (the numeric snowflake as a string). The
+ citation URL is rebuilt from the id and username; URL fields in the
+ response are never used. A post whose author is not in
+ ``includes.users`` (or whose username fails the handle grammar) keeps
+ the ``https://x.com/i/status/<id>`` form with an empty handle.
+
+ ``window`` is an optional ``(from_date, to_date)`` pair of YYYY-MM-DD
+ strings; dated posts outside it are dropped. ``seen_ids`` and
+ ``index_offset`` let several calls share one accumulator with unique ids.
+ """
+ items: List[Dict[str, Any]] = []
+ if not isinstance(response, dict):
+ return items
+ if "error" in response:
+ _log("error in response; no items parsed")
+ return items
+
+ data = response.get("data") or []
+ if not isinstance(data, list) or not data:
+ return items
+
+ authors: Dict[str, str] = {}
+ for user in (response.get("includes") or {}).get("users") or []:
+ if isinstance(user, dict) and user.get("id") is not None:
+ authors[str(user["id"])] = _clean_handle(user.get("username", ""))
+
+ from .query import leading_mentions
+
+ window_from, window_to = (window or (None, None))
+ seen = seen_ids if seen_ids is not None else set()
+
+ for tweet in data:
+ if not isinstance(tweet, dict):
+ continue
+ tweet_id = str(tweet.get("id") or "").strip()
+ if not tweet_id.isdigit():
+ continue
+ if tweet_id in seen:
+ continue
+
+ username = authors.get(str(tweet.get("author_id") or ""), "")
+ if username:
+ url = f"https://x.com/{username}/status/{tweet_id}"
+ else:
+ url = f"https://x.com/i/status/{tweet_id}"
+
+ note = tweet.get("note_tweet")
+ text = ""
+ if isinstance(note, dict) and note.get("text"):
+ text = str(note["text"])
+ else:
+ text = str(tweet.get("text") or "")
+ text = text.strip()
+
+ engagement: Optional[Dict[str, Any]] = None
+ metrics = tweet.get("public_metrics") or {}
+ if isinstance(metrics, dict) and metrics:
+ engagement = {
+ "likes": metrics.get("like_count", 0),
+ "reposts": metrics.get("retweet_count", 0),
+ "replies": metrics.get("reply_count", 0),
+ "quotes": metrics.get("quote_count", 0),
+ }
+ bookmarks = _safe_int(metrics.get("bookmark_count"))
+ if bookmarks is not None:
+ engagement["bookmarks"] = bookmarks
+ views = _safe_int(metrics.get("impression_count"))
+ if views is not None:
+ engagement["views"] = views
+
+ date: Optional[str] = None
+ created = str(tweet.get("created_at") or "")
+ if created:
+ m = re.match(r"(\d{4}-\d{2}-\d{2})", created)
+ if m:
+ date = m.group(1)
+ if date and window_from and date < window_from:
+ continue
+ if date and window_to and date > window_to:
+ continue
+
+ seen.add(tweet_id)
+ items.append({
+ "id": f"{id_prefix}{index_offset + len(items) + 1}",
+ "text": text[:500],
+ "url": url,
+ "author_handle": username,
+ "date": date,
+ "engagement": engagement,
+ "mentioned_handles": leading_mentions(text),
+ "why_relevant": "",
+ "relevance": _compute_relevance(topic, text) if topic else 0.5,
+ "post_id": tweet_id,
+ })
+
+ return items
+
+
+# ---------------------------------------------------------------------------
+# Query compilation
+# ---------------------------------------------------------------------------
+
+
+def _topic_tokens(topic: str) -> List[str]:
+ """Sanitized topic tokens: no quotes, grouping, operators, or negation."""
+ separators = str.maketrans({char: " " for char in _GROUPING_CHARS + _QUOTE_CHARS})
+ cleaned = str(topic or "").translate(separators)
+ tokens: List[str] = []
+ for token in cleaned.split():
+ clean = token.strip(_APOSTROPHES)
+ if not clean:
+ continue
+ if ":" in clean or clean.startswith("-"):
+ continue
+ if clean.upper() in ("OR", "AND"):
+ continue
+ tokens.append(clean)
+ return tokens
+
+
+def _compile(tokens: List[str]) -> str:
+ return f'"{" ".join(tokens)}" -is:retweet'
+
+
+def build_query(topic: str) -> str:
+ """Compile a topic into one quoted phrase plus ``-is:retweet``.
+
+ Mirrors ``bird_x.build_topic_query`` (grouping characters become spaces,
+ quotes are balanced) and additionally drops every operator-shaped token
+ (anything with a colon, anything starting with ``-``, bare OR/AND) so a
+ topic can never smuggle ``from:``, ``since:``, or a phrase break into the
+ request. The window is never part of the query (see ``_window``). The
+ result stays under ``MAX_QUERY_CHARS``, cut at a token boundary. Returns
+ "" when nothing lexical survives.
+ """
+ tokens = _topic_tokens(topic)
+ if not tokens:
+ return ""
+ compiled = _compile(tokens)
+ while len(compiled) > MAX_QUERY_CHARS and len(tokens) > 1:
+ tokens.pop()
+ compiled = _compile(tokens)
+ if len(compiled) > MAX_QUERY_CHARS:
+ # One token longer than the whole budget: keep as much of it as fits.
+ overhead = len(_compile([""]))
+ tokens = [tokens[0][: MAX_QUERY_CHARS - overhead]]
+ compiled = _compile(tokens)
+ return compiled
+
+
+# ---------------------------------------------------------------------------
+# Window
+# ---------------------------------------------------------------------------
+
+
+def _parse_day(value: str) -> Optional[datetime]:
+ try:
+ return datetime.strptime(str(value)[:10], "%Y-%m-%d").replace(tzinfo=timezone.utc)
+ except (TypeError, ValueError):
+ return None
+
+
+def _iso(when: datetime) -> str:
+ return when.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def _window(from_date: str, to_date: str) -> tuple[str, str]:
+ """``(start_time, end_time)`` for the request, ISO 8601 with Z.
+
+ ``start_time`` is midnight UTC of ``from_date``; ``end_time`` is the end
+ of ``to_date`` or, when that would be now or later ("today"), now minus a
+ safety margin.
+ """
+ now = _utcnow()
+ start = _parse_day(from_date) or (now - timedelta(days=30))
+ end_day = _parse_day(to_date)
+ latest = now - timedelta(seconds=END_TIME_SAFETY_SECONDS)
+ if end_day is None:
+ end = latest
+ else:
+ end = min(end_day + timedelta(hours=23, minutes=59, seconds=59), latest)
+ if start >= end:
+ start = end - timedelta(days=1)
+ return _iso(start), _iso(end)
+
+
+def _recent_floor() -> str:
+ """The earliest ``start_time`` recent search accepts, with a margin."""
+ return _iso(
+ _utcnow() - timedelta(days=RECENT_WINDOW_DAYS) + timedelta(seconds=END_TIME_SAFETY_SECONDS)
+ )
+
+
+# ---------------------------------------------------------------------------
+# Transport
+# ---------------------------------------------------------------------------
+
+
+class _XApiFailure(Exception):
+ """A request failure already reduced to its fixed string.
+
+ ``enrollment`` marks the one 403 that recent search can still serve.
+ The message is always one of the ERR_* literals (or ``xapi: http <n>``),
+ never response text.
+ """
+
+ def __init__(self, message: str, *, enrollment: bool = False):
+ super().__init__(message)
+ self.enrollment = enrollment
+
+
+def _failure_for(exc: http.HTTPError) -> _XApiFailure:
+ status = getattr(exc, "status_code", None)
+ body = str(getattr(exc, "body", "") or "").lower()
+ if (
+ status == 402
+ or http.classify_failure(message=body) == health.PAYMENT_REQUIRED
+ or any(marker in body for marker in _X_CREDIT_MARKERS)
+ ):
+ return _XApiFailure(ERR_PAYMENT_REQUIRED)
+ if status == 401:
+ return _XApiFailure(ERR_UNAUTHORIZED)
+ if status == 403:
+ enrolled = any(marker in body for marker in _ENROLLMENT_MARKERS)
+ return _XApiFailure(ERR_FORBIDDEN, enrollment=enrolled)
+ if status == 429:
+ return _XApiFailure(ERR_RATE_LIMITED)
+ state = getattr(exc, "outcome_state", None)
+ if status == 408 or state == health.TIMEOUT:
+ return _XApiFailure(ERR_TIMED_OUT)
+ if status:
+ return _XApiFailure(f"xapi: http {status}")
+ if state == health.UNREACHABLE:
+ return _XApiFailure(ERR_UNREACHABLE)
+ return _XApiFailure("xapi: request failed (HTTPError)")
+
+
+def _get(
+ token: str, url: str, params: Dict[str, Any], deadline: Optional[float] = None,
+) -> Dict[str, Any]:
+ """One authenticated GET; every failure surfaces as ``_XApiFailure``.
+
+ ``deadline`` is handed to the transport as its wall deadline, so a
+ request started just before it cannot keep the full per-request timeout
+ and retry cycle: the socket wait and every backoff sleep stop at the
+ deadline (``http.DeadlineExceeded`` -> ``ERR_TIMED_OUT``).
+ """
+ try:
+ response = http.get(
+ url,
+ headers={"Authorization": f"Bearer {token}"},
+ params=params,
+ timeout=TIMEOUT_SECONDS,
+ retries=RETRIES,
+ deadline_monotonic=deadline,
+ )
+ except http.HTTPError as exc:
+ raise _failure_for(exc) from None
+ except TimeoutError:
+ raise _XApiFailure(ERR_TIMED_OUT) from None
+ except Exception as exc: # noqa: BLE001 - the message must never leak
+ raise _XApiFailure(f"xapi: request failed ({type(exc).__name__})") from None
+ return response if isinstance(response, dict) else {}
+
+
+def _page_size(count: int) -> int:
+ return max(MIN_PAGE_RESULTS, min(MAX_PAGE_RESULTS, int(count)))
+
+
+def _search_pages(
+ token: str,
+ url: str,
+ params: Dict[str, Any],
+ count: int,
+ deadline: Optional[float] = None,
+) -> Dict[str, Any]:
+ """Follow ``next_token`` until ``count`` posts are collected.
+
+ ``deadline`` is a ``time.monotonic()`` instant: the first page always
+ runs; a later page is skipped once the deadline has passed and the
+ posts collected so far are returned with ``"truncated": True`` so the
+ caller can report the incomplete coverage.
+ """
+ data: List[Dict[str, Any]] = []
+ users: Dict[str, Dict[str, Any]] = {}
+ truncated = False
+ page_params = dict(params)
+ page_params["max_results"] = _page_size(count)
+ # Bound the walk even when every page carries a next_token.
+ max_pages = max(1, -(-count // MIN_PAGE_RESULTS))
+ for page_index in range(max_pages):
+ if page_index and deadline is not None and time.monotonic() >= deadline:
+ _log(f"search deadline ({DEADLINE_SECONDS}s) reached; keeping {len(data)} posts")
+ truncated = True
+ break
+ # A fresh dict per page: the transport must never see a later
+ # page's next_token on an earlier request.
+ try:
+ response = _get(token, url, dict(page_params), deadline)
+ except _XApiFailure as exc:
+ if data and str(exc) == ERR_TIMED_OUT:
+ # The budget ran out mid-walk: the pages already collected
+ # are the result, reported as truncated, not a failure.
+ _log(f"search deadline reached mid-walk; keeping {len(data)} posts")
+ truncated = True
+ break
+ raise
+ page = response.get("data") or []
+ if isinstance(page, list):
+ data.extend(t for t in page if isinstance(t, dict))
+ for user in (response.get("includes") or {}).get("users") or []:
+ if isinstance(user, dict) and user.get("id") is not None:
+ users[str(user["id"])] = user
+ next_token = (response.get("meta") or {}).get("next_token")
+ if not next_token or len(data) >= count:
+ break
+ page_params["next_token"] = next_token
+ return {
+ "data": data[:count],
+ "includes": {"users": list(users.values())},
+ "truncated": truncated,
+ }
+
+
+def _run_search(
+ token: str,
+ query: str,
+ from_date: str,
+ to_date: str,
+ count: int,
+ *,
+ topic: str,
+ id_prefix: str,
+ label: str,
+ deadline: Optional[float] = None,
+) -> Dict[str, Any]:
+ """Full-archive search with the recent-search fallback.
+
+ ``deadline`` (a ``time.monotonic()`` instant) is shared with the caller's
+ other lanes; without one the search gets its own ``DEADLINE_SECONDS``.
+ A shared deadline that has already passed sends no request at all.
+
+ Returns ``{"items": [...]}`` (plus ``"warning"`` when the window was
+ truncated or the deadline stopped the walk early) or ``{"items": [],
+ "error": <fixed string>}``.
+ """
+ start, end = _window(from_date, to_date)
+ params = {
+ "query": query,
+ "start_time": start,
+ "end_time": end,
+ "sort_order": "recency",
+ "expansions": "author_id",
+ "tweet.fields": "created_at,public_metrics,note_tweet,entities",
+ "user.fields": "username",
+ }
+ if deadline is not None and time.monotonic() >= deadline:
+ _log(f"{label}: lane budget ({LANE_BUDGET_SECONDS:.0f}s) exhausted before the search started")
+ return {"items": [], "warning": DEADLINE_DETAIL}
+ _log(f"Searching: {label}")
+ warnings: List[str] = []
+ if deadline is None:
+ deadline = time.monotonic() + DEADLINE_SECONDS
+ try:
+ response = _search_pages(token, _SEARCH_ALL_URL, params, count, deadline)
+ except _XApiFailure as exc:
+ if not exc.enrollment:
+ _log(f"{label}: {exc}")
+ return {"items": [], "error": str(exc)}
+ _log(f"{label}: full-archive search not enrolled; retrying recent search ({TRUNCATION_DETAIL})")
+ floor = _recent_floor()
+ if floor >= end:
+ # The whole window predates what recent search can reach: an
+ # empty, truncated result, never a start_time after end_time.
+ _log(f"{label}: window ends before the recent-search floor; nothing to fetch")
+ return {"items": [], "warning": TRUNCATION_DETAIL}
+ params["start_time"] = max(start, floor)
+ try:
+ response = _search_pages(token, _SEARCH_RECENT_URL, params, count, deadline)
+ except _XApiFailure as exc2:
+ _log(f"{label}: {exc2}")
+ return {"items": [], "error": str(exc2)}
+ warnings.append(TRUNCATION_DETAIL)
+ if response.get("truncated"):
+ warnings.append(DEADLINE_DETAIL)
+ items = parse_v2_response(response, topic, (from_date, to_date), id_prefix=id_prefix)
+ result: Dict[str, Any] = {"items": items}
+ if warnings:
+ result["warning"] = "; ".join(warnings)
+ return result
+
+
+# ---------------------------------------------------------------------------
+# Public search entry points
+# ---------------------------------------------------------------------------
+
+
+def search_x(
+ token: str,
+ query: str,
+ from_date: str,
+ to_date: str,
+ depth: str = "default",
+) -> Dict[str, Any]:
+ """Topic search via X API v2.
+
+ Returns ``{"items": [...]}`` or ``{"items": [], "error": "..."}`` (the
+ xquik shape); ``"warning"`` carries the truncation detail after the
+ recent-search fallback.
+ """
+ if not token:
+ return {"items": [], "error": ERR_NO_TOKEN}
+ count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
+ compiled = build_query(query)
+ if not compiled:
+ _log("topic compiled to an empty query after sanitizing")
+ return {"items": [], "error": ERR_EMPTY_QUERY}
+ return _run_search(
+ token, compiled, from_date, to_date, count,
+ topic=query, id_prefix="XAPI", label="topic",
+ )
+
+
+def is_own_post(url: str, handle: str) -> bool:
+ """True when a post URL is authored by ``handle`` (their own post).
+
+ The ABOUT lanes (here, xquik, the host envelope) drop the subject's own
+ posts so only mentions *by others* remain. Handles both x.com and
+ twitter.com permalinks.
+ """
+ u = (url or "").lower()
+ h = handle.lower().lstrip("@").strip()
+ return bool(h) and (f"x.com/{h}/status" in u or f"twitter.com/{h}/status" in u)
+
+
+
+def _lane_handle(raw: str, lane: str) -> str:
+ handle = _clean_handle(raw)
+ if not handle:
+ _log(f"skipping {lane} lane for a handle outside the X handle grammar")
+ return handle
+
+
+_MAX_LANE_WORKERS = 5
+
+
+def _run_handle_lanes(
+ handles: List[str],
+ search_one: Callable[[str], Dict[str, Any]],
+ *,
+ id_prefix: str,
+ keep: Optional[Callable[[Dict[str, Any], str], bool]] = None,
+ warnings: Optional[List[str]] = None,
+) -> List[Dict[str, Any]]:
+ """Run one search per handle on a bounded pool and merge in handle order.
+
+ Per-handle receipts (``"warning"`` on a result, e.g. a deadline stop)
+ are appended to ``warnings`` once each, so the caller can report
+ incomplete lane coverage instead of presenting it as complete.
+
+ Mirrors ``bird_x.search_handles``: at most five handles in flight. A
+ fatal auth/payment result stops further handles from being scheduled;
+ whatever was already fetched is kept. Results are merged on the calling
+ thread in ``handles`` order, deduped by post id across handles, and
+ renumbered ``<id_prefix><n>`` so the output is deterministic regardless
+ of completion order. ``keep(item, handle)`` filters a handle's items
+ after the post id is recorded, so a dropped post still dedupes later.
+ """
+ results: List[Optional[Dict[str, Any]]] = [None] * len(handles)
+ if handles:
+ max_workers = min(_MAX_LANE_WORKERS, len(handles))
+ fatal = False
+ with ThreadPoolExecutor(max_workers=max_workers) as pool:
+ pending: Dict[Any, int] = {}
+ next_index = 0
+ while pending or (next_index < len(handles) and not fatal):
+ while next_index < len(handles) and not fatal and len(pending) < max_workers:
+ pending[pool.submit(search_one, handles[next_index])] = next_index
+ next_index += 1
+ done, _ = wait(list(pending), return_when=FIRST_COMPLETED)
+ for future in done:
+ index = pending.pop(future)
+ result = future.result()
+ results[index] = result
+ if result.get("error") in _FATAL_ERRORS:
+ fatal = True # the key itself failed; keep what we have
+
+ items: List[Dict[str, Any]] = []
+ seen_ids: set[str] = set()
+ for handle, result in zip(handles, results):
+ note = (result or {}).get("warning")
+ if note and warnings is not None and note not in warnings:
+ warnings.append(note)
+ for item in (result or {}).get("items", []):
+ post_id = item.get("post_id", "")
+ if post_id in seen_ids:
+ continue
+ seen_ids.add(post_id)
+ if keep is not None and not keep(item, handle):
+ continue
+ item["id"] = f"{id_prefix}{len(items) + 1}"
+ items.append(item)
+ return items
+
+
+def search_handles(
+ handles: List[str],
+ topic: str,
+ from_date: str,
+ to_date: str,
+ *,
+ count_per: int = 8,
+ token: str = "",
+ deadline: Optional[float] = None,
+ warnings: Optional[List[str]] = None,
+) -> List[Dict[str, Any]]:
+ """FROM lane: posts authored BY each handle (their own timeline).
+
+ The topic is NOT AND'd into the query; it ranks relevance only (mirrors
+ ``xquik.search_handles``). A handle outside the X grammar skips its lane
+ with a receipt line.
+ """
+ if not token or not handles:
+ return []
+ clean = [h for h in (_lane_handle(raw, "from:") for raw in handles) if h]
+
+ def _search_one(handle: str) -> Dict[str, Any]:
+ return _run_search(
+ token, f"from:{handle} -is:retweet", from_date, to_date, count_per,
+ topic=topic, id_prefix="XF", label=f"from:{handle}", deadline=deadline,
+ )
+
+ return _run_handle_lanes(clean, _search_one, id_prefix="XF", warnings=warnings)
+
+
+def search_mentions(
+ handles: List[str],
+ from_date: str,
+ to_date: str,
+ *,
+ topic: str = "",
+ count_per: int = 5,
+ token: str = "",
+ deadline: Optional[float] = None,
+ warnings: Optional[List[str]] = None,
+) -> List[Dict[str, Any]]:
+ """ABOUT lane: posts mentioning each handle, authored by OTHERS.
+
+ The query is ``@handle -from:handle`` and the handle's own posts are
+ dropped again client-side (``is_own_post``) so only third-party mentions
+ remain.
+ """
+ if not token or not handles:
+ return []
+ clean = [h for h in (_lane_handle(raw, "mention") for raw in handles) if h]
+
+ def _search_one(handle: str) -> Dict[str, Any]:
+ return _run_search(
+ token, f"@{handle} -from:{handle} -is:retweet", from_date, to_date, count_per,
+ topic=topic, id_prefix="XA", label=f"@{handle}", deadline=deadline,
+ )
+
+ return _run_handle_lanes(
+ clean, _search_one, id_prefix="XA",
+ keep=lambda item, handle: not is_own_post(item.get("url", ""), handle),
+ warnings=warnings,
+ )
diff --git a/skills/last30days/scripts/lib/x_envelope.py b/skills/last30days/scripts/lib/x_envelope.py
new file mode 100644
index 00000000..670ba574
--- /dev/null
+++ b/skills/last30days/scripts/lib/x_envelope.py
@@ -0,0 +1,707 @@
+"""Host-fetched X envelope: the ``--x-posts`` lane.
+
+The hosting model fetches posts through its own X connector and writes them
+to a ``.json`` file of flat rows. The engine ingests that file as its X
+source for one run: strict at the envelope level (a malformed, stale, or
+off-topic file fails closed with an exit-2 contract error), lenient per row
+(a bad row is dropped and counted), and trusting nothing a row asserts about
+itself beyond a numeric id and a grammar-valid handle.
+
+Shape (``schema: "last30days-x-posts/1"``)::
+
+ {
+ "schema": "last30days-x-posts/1",
+ "generated_at": "<ISO-8601>",
+ "topic": "<the run topic>",
+ "window": {"from": "YYYY-MM-DD", "to": "YYYY-MM-DD"},
+ "provider": "<free text>",
+ "status": "ok" | "partial" | "error",
+ "error": "<short category: credits | not-connected | unavailable | window-unsupported>",
+ "calls": [{"lane": "topic|from|mention|related", "handles": [...], "posts": [ROW, ...]}]
+ }
+
+A ROW is a flat object with exactly ``id``, ``author_handle``, ``created_at``,
+``text``, ``likes``, ``reposts``, ``replies``, ``quotes``. Any other key is
+ignored (and counted once per row as ``extra-fields``); in particular a
+row-supplied ``url`` is never used for the citation, only cross-checked.
+
+The envelope is single-serve: the first X subquery takes the topic-lane rows
+and the handle-lane section takes the lane calls; later X subqueries,
+judge-retry, and thin-retry get nothing (``take_topic`` / ``take_lanes``
+return ``None`` once consumed).
+
+Security contract: contract errors name the path and the field, never a
+field value; the envelope's raw ``error`` text reaches stderr only under
+``LAST30DAYS_DEBUG``; every outcome detail is one of the fixed strings below.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+import stat
+import threading
+from dataclasses import dataclass, field
+from datetime import date, datetime, timedelta, timezone
+from pathlib import Path
+from typing import Any
+
+from . import env, health, log, schema
+from .relevance import token_overlap_relevance as _compute_relevance
+from .query import leading_mentions
+from .x_api import _clean_handle, _decode_snowflake, _looks_generated, is_own_post
+
+SCHEMA = "last30days-x-posts/1"
+ID_PREFIX = "XHOST"
+
+# Input bounds. Named so the recipe and the receipts can quote them.
+MAX_BYTES = 8 * 1024 * 1024
+MAX_CALLS = 20
+MAX_ROWS_PER_CALL = 500
+MAX_ROWS_TOTAL = 1000
+MAX_TEXT_CHARS = 10_000
+MAX_AGE_SECONDS = 6 * 60 * 60
+MAX_ID_DIGITS = 20
+
+LANES = ("topic", "from", "mention", "related")
+STATUSES = ("ok", "partial", "error")
+ROW_FIELDS = frozenset(
+ {"id", "author_handle", "created_at", "text", "likes", "reposts", "replies", "quotes"}
+)
+COUNTERS = (
+ "out-of-window",
+ "missing-id-text",
+ "uncitable",
+ "date-mismatch",
+ "handle-mismatch",
+ "duplicate",
+ "lane-mismatch",
+ "truncated",
+ "extra-fields",
+)
+
+# Engine-authored fixed outcome details. The envelope's own error text
+# never becomes a detail string.
+DETAIL_CREDITS = "X connector reported no credits"
+DETAIL_NOT_CONNECTED = "X connector not connected"
+DETAIL_UNAVAILABLE = "X connector unavailable"
+DETAIL_ERROR = "X connector error"
+DETAIL_GENERATED = "X connector rows rejected (id sequence looks generated)"
+DETAIL_PARTIAL = "X connector returned partial results"
+DETAIL_NOT_PASSED = "connector result not passed"
+
+CATEGORY_CREDITS = "credits"
+CATEGORY_NOT_CONNECTED = "not-connected"
+CATEGORY_UNAVAILABLE = "unavailable"
+CATEGORY_WINDOW_UNSUPPORTED = "window-unsupported"
+_CATEGORY_GENERATED = "generated"
+
+_REMEDY = (
+ "Rewrite the file to the last30days-x-posts/1 shape, or re-run without --x-posts."
+)
+
+# Directories the envelope must never be read from: the engine's own config
+# dir (which holds .env) and the credential stores of the X tooling and the
+# usual shell neighbours. Checked after realpath so a symlink cannot reach in.
+_CREDENTIAL_STORES = (
+ "~/.xurl",
+ "~/.grok",
+ "~/.config/last30days",
+ "~/.config/gh",
+ "~/.aws",
+ "~/.ssh",
+ "~/.netrc",
+)
+
+# Snowflake ids predate nothing before X's epoch; a decode outside
+# [2010-11-04, now + 1 day] is not a real post id.
+_SNOWFLAKE_FLOOR = datetime(2010, 11, 4, tzinfo=timezone.utc)
+
+# C0 and C1 control characters. Text keeps newlines and tabs; handles and the
+# error field keep nothing.
+_CONTROL_ANY_RE = re.compile(r"[\x00-\x1f\x7f-\x9f]")
+_CONTROL_TEXT_RE = re.compile(r"[\x00-\x08\x0b-\x1f\x7f-\x9f]")
+_PLACEHOLDER_HANDLES = frozenset(
+ {"unknown", "n/a", "na", "none", "null", "user", "anonymous", "placeholder", "handle"}
+)
+_STATUS_URL_RE = re.compile(
+ r"^https?://(?:www\.|mobile\.)?(?:x\.com|twitter\.com)/(?P<handle>[A-Za-z0-9_]{1,15})/status(?:es)?/(?P<id>\d+)",
+ re.IGNORECASE,
+)
+_TOPIC_STRIP = "\"'“”‘’ \t"
+
+
+class EnvelopeContractError(Exception):
+ """The envelope failed its contract: unreadable, out of bounds, malformed,
+ stale, off-topic, or outside the research window. The CLI maps this to
+ exit code 2. The message names the path and the offending field only."""
+
+ def __init__(self, message: str) -> None:
+ super().__init__(message)
+ self.message = message
+
+
+@dataclass
+class EnvelopeCall:
+ """One validated handle-lane call (``from``, ``mention``, or ``related``)."""
+
+ index: int
+ lane: str
+ handles: list[str]
+ posts: list[dict[str, Any]]
+
+
+@dataclass
+class Envelope:
+ """A validated ``--x-posts`` file, ready to be served once."""
+
+ path: str
+ sha256: str
+ status: str
+ error_category: str
+ provider: str
+ window: tuple[str, str]
+ topic_items: list[dict[str, Any]]
+ lane_calls: list[EnvelopeCall]
+ counters: dict[str, int]
+ accepted: int
+ total: int
+ lane_counts: dict[str, int]
+ call_lanes: list[str]
+ warnings: list[str] = field(default_factory=list)
+ notes: list[str] = field(default_factory=list)
+ _lock: threading.Lock = field(default_factory=threading.Lock, repr=False, compare=False)
+ _topic_served: bool = field(default=False, repr=False, compare=False)
+ _lanes_served: bool = field(default=False, repr=False, compare=False)
+
+ def take_topic(self) -> list[dict[str, Any]] | None:
+ """Topic-lane rows on the first call; ``None`` once consumed."""
+ with self._lock:
+ if self._topic_served:
+ return None
+ self._topic_served = True
+ return list(self.topic_items)
+
+ def take_lanes(self) -> list[EnvelopeCall] | None:
+ """Handle-lane calls on the first call; ``None`` once consumed."""
+ with self._lock:
+ if self._lanes_served:
+ return None
+ self._lanes_served = True
+ return list(self.lane_calls)
+
+ def outcome(self) -> tuple[schema.RunOutcomeState, str] | None:
+ """The fixed source outcome for a non-ok envelope, else ``None``."""
+ if self.status == "error":
+ return _error_outcome(self.error_category)
+ if self.status == "partial":
+ detail = DETAIL_PARTIAL
+ if self.error_category:
+ detail += f": {self.error_category}"
+ if self.call_lanes:
+ detail += f" (calls: {', '.join(self.call_lanes)})"
+ return schema.PARTIAL, detail
+ return None
+
+ def receipt(self) -> str:
+ dropped = ", ".join(
+ f"{name} {count}" for name, count in self.counters.items() if count
+ ) or "none"
+ lanes = ", ".join(f"{lane} {self.lane_counts.get(lane, 0)}" for lane in LANES)
+ line = (
+ f"host-fetched X: accepted {self.accepted} of {self.total} "
+ f"(dropped: {dropped}); lanes: {lanes}"
+ )
+ if self.status != "ok":
+ line += f"; status {self.status}"
+ if self.error_category:
+ line += f" ({self.error_category})"
+ if self.notes:
+ line += "; notes: " + "; ".join(self.notes)
+ return line
+
+
+def _error_outcome(category: str) -> tuple[schema.RunOutcomeState, str]:
+ if category == CATEGORY_CREDITS:
+ return schema.PAYMENT_REQUIRED, DETAIL_CREDITS
+ if category == CATEGORY_NOT_CONNECTED:
+ return health.ERROR, DETAIL_NOT_CONNECTED
+ if category == CATEGORY_UNAVAILABLE:
+ return health.ERROR, DETAIL_UNAVAILABLE
+ if category == _CATEGORY_GENERATED:
+ return health.ERROR, DETAIL_GENERATED
+ return health.ERROR, DETAIL_ERROR
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _stat(path: Path) -> os.stat_result:
+ """Seam for tests: the size and type check runs before any read."""
+ return os.stat(path)
+
+
+def _log(msg: str) -> None:
+ log.source_log("x", msg, tty_only=False)
+
+
+def _protected_roots() -> list[Path]:
+ roots: list[Path] = []
+ for candidate in (env.CONFIG_DIR, getattr(env, "CONFIG_FILE", None)):
+ if candidate:
+ roots.append(Path(candidate))
+ for store in _CREDENTIAL_STORES:
+ roots.append(Path(store).expanduser())
+ return [Path(os.path.realpath(root)) for root in roots]
+
+
+def _inside_protected(real: Path) -> bool:
+ for root in _protected_roots():
+ if real == root or root in real.parents:
+ return True
+ return False
+
+
+def _normalize_topic(value: str) -> str:
+ cleaned = _CONTROL_ANY_RE.sub("", str(value or ""))
+ cleaned = cleaned.strip(_TOPIC_STRIP).casefold()
+ return " ".join(cleaned.split())
+
+
+def _parse_day(value: Any) -> date | None:
+ if not isinstance(value, str):
+ return None
+ try:
+ return date.fromisoformat(value.strip()[:10])
+ except ValueError:
+ return None
+
+
+def _parse_iso(value: str) -> datetime | None:
+ text = value.strip()
+ if text.endswith(("Z", "z")):
+ text = text[:-1] + "+00:00"
+ try:
+ parsed = datetime.fromisoformat(text)
+ except ValueError:
+ return None
+ if parsed.tzinfo is None:
+ parsed = parsed.replace(tzinfo=timezone.utc)
+ return parsed
+
+
+def _category(raw: str) -> str:
+ """The short error class: first token, lowercased, [a-z0-9-] only."""
+ cleaned = _CONTROL_ANY_RE.sub("", raw or "").strip().lower()
+ if not cleaned:
+ return ""
+ token = cleaned.split()[0]
+ token = re.sub(r"[^a-z0-9-]", "", token)
+ return token[:40]
+
+
+def _clean_text(value: str) -> str:
+ """Strip C0/C1 controls (newlines and tabs survive) and defang markup.
+
+ Envelope text is model-written and post-authored, so the two Markdown
+ constructs that stay active in the saved report are neutralised without
+ hiding the characters: a link tail ``](`` becomes ``]\\(`` (renders as
+ the same text, never as a link) and an HTML comment opener ``<!--``
+ becomes ``<! --`` (never a comment, so a pasted META marker cannot be
+ promoted). Raw tags are left to the HTML renderer's escaping.
+ """
+ text = value.replace("\r\n", "\n").replace("\r", "\n")
+ text = _CONTROL_TEXT_RE.sub("", text)
+ return text.replace("](", "]\\(").replace("<!--", "<! --")
+
+
+def _int(value: Any) -> int:
+ if isinstance(value, bool):
+ return 0
+ try:
+ return max(0, int(value))
+ except (TypeError, ValueError):
+ return 0
+
+
+def _post_id(value: Any) -> str:
+ if isinstance(value, bool):
+ return ""
+ if isinstance(value, int):
+ candidate = str(value)
+ elif isinstance(value, str):
+ candidate = value.strip()
+ else:
+ return ""
+ if not candidate.isdigit() or len(candidate) > MAX_ID_DIGITS:
+ return ""
+ return candidate
+
+
+def _handle(value: Any) -> str:
+ if not isinstance(value, str) or _CONTROL_ANY_RE.search(value):
+ return ""
+ handle = _clean_handle(value)
+ if handle.lower() in _PLACEHOLDER_HANDLES:
+ return ""
+ return handle
+
+
+def _clean_handles(value: Any) -> list[str] | None:
+ """Grammar-valid, lowercased handles; ``None`` when any entry fails."""
+ if not isinstance(value, list):
+ return None
+ cleaned: list[str] = []
+ for raw in value:
+ handle = _handle(raw)
+ if not handle:
+ return None
+ if handle.lower() not in cleaned:
+ cleaned.append(handle.lower())
+ return cleaned
+
+
+# ---------------------------------------------------------------------------
+# Reader
+# ---------------------------------------------------------------------------
+
+
+# Clock skew a host may legitimately show; anything further ahead is a
+# stamp chosen to outlive the six-hour freshness gate.
+FUTURE_SKEW_SECONDS = 5 * 60
+
+
+def _generated_in_future(value: Any) -> bool:
+ """True when ``generated_at`` is more than ``FUTURE_SKEW_SECONDS`` ahead of now."""
+ try:
+ stamp = datetime.fromisoformat(str(value))
+ except (TypeError, ValueError):
+ return False
+ if stamp.tzinfo is None:
+ stamp = stamp.replace(tzinfo=timezone.utc)
+ ahead = stamp.astimezone(timezone.utc) - datetime.now(timezone.utc)
+ return ahead.total_seconds() > FUTURE_SKEW_SECONDS
+
+
+def read(
+ path: str | os.PathLike[str],
+ window: tuple[str, str],
+ topic: str,
+ handles: list[str] | None = None,
+ related: list[str] | None = None,
+) -> Envelope:
+ """Validate and ingest a ``--x-posts`` file for this run.
+
+ ``window`` is the engine's ``(from_date, to_date)``; ``topic`` is the run
+ topic (or comparison entity); ``handles`` are the run's ``--x-handle``
+ handles and ``related`` its ``--x-related`` handles, which together bound
+ what a call's ``handles`` may claim.
+
+ Raises ``EnvelopeContractError`` (exit 2) for every fail-closed case.
+ """
+ raw_path = os.fspath(path)
+
+ def fail(what: str) -> EnvelopeContractError:
+ return EnvelopeContractError(f"--x-posts file {raw_path} {what}. {_REMEDY}")
+
+ real = Path(os.path.realpath(raw_path))
+ try:
+ info = _stat(real)
+ except OSError as exc:
+ raise fail(f"cannot be read ({type(exc).__name__})") from None
+ if not stat.S_ISREG(info.st_mode):
+ raise fail("is not a regular file")
+ if real.suffix.lower() != ".json":
+ raise fail("must be a .json file")
+ if _inside_protected(real):
+ raise fail("is inside a configuration or credential directory")
+ if info.st_size > MAX_BYTES:
+ raise fail(f"exceeds the {MAX_BYTES // (1024 * 1024)} MiB limit")
+
+ try:
+ data = real.read_bytes()
+ except OSError as exc:
+ raise fail(f"cannot be read ({type(exc).__name__})") from None
+ digest = hashlib.sha256(data).hexdigest()
+ try:
+ text = data.decode("utf-8")
+ except UnicodeDecodeError:
+ raise fail("is not valid UTF-8") from None
+ try:
+ payload = json.loads(text)
+ except (RecursionError, MemoryError):
+ raise fail("is too deeply nested or too large to parse") from None
+ except ValueError:
+ raise fail("is not valid JSON") from None
+
+ if not isinstance(payload, dict):
+ raise fail("must be a JSON object")
+ if payload.get("schema") != SCHEMA:
+ raise fail('has a missing or unsupported "schema" field')
+ status = payload.get("status")
+ if status not in STATUSES:
+ raise fail('has a "status" field that is not ok, partial, or error')
+ if not env.is_timestamp_fresh(payload.get("generated_at"), MAX_AGE_SECONDS):
+ raise fail(
+ 'has a "generated_at" field that is missing, malformed, or older than '
+ f"{MAX_AGE_SECONDS // 3600} hours"
+ )
+ if _generated_in_future(payload.get("generated_at")):
+ raise fail('has a "generated_at" field in the future')
+ if _normalize_topic(payload.get("topic")) != _normalize_topic(topic) or not _normalize_topic(topic):
+ raise fail('has a "topic" field that does not match this run\'s topic')
+
+ from_date, to_date = window
+ window_raw = payload.get("window")
+ if not isinstance(window_raw, dict):
+ raise fail('has a missing or malformed "window" field')
+ window_from = _parse_day(window_raw.get("from"))
+ window_to = _parse_day(window_raw.get("to"))
+ if window_from is None or window_to is None:
+ raise fail('has a "window" field without valid from/to dates')
+ engine_from = _parse_day(from_date) or date.min
+ engine_to = _parse_day(to_date) or date.max
+ if window_to < engine_from:
+ raise fail('has a "window" that ends before the research window starts')
+ warnings: list[str] = []
+ if window_from > engine_from or window_to < engine_to:
+ warnings.append(
+ f"host-fetched X window {window_from.isoformat()}..{window_to.isoformat()} "
+ f"is narrower than the research window {from_date}..{to_date}; "
+ "the research window is authoritative"
+ )
+
+ provider = _CONTROL_ANY_RE.sub("", str(payload.get("provider") or ""))[:64].strip()
+
+ error_raw = payload.get("error")
+ if error_raw is not None and not isinstance(error_raw, str):
+ raise fail('has an "error" field that is not a string')
+ if error_raw and ("\n" in error_raw or "\r" in error_raw):
+ raise fail('has an "error" field that is not a single line')
+ error_category = _category(error_raw or "")
+ if error_raw and log.is_debug():
+ _log(f"envelope error text (debug only): {_CONTROL_ANY_RE.sub('', error_raw)[:500]}")
+
+ calls_raw = payload.get("calls")
+ if not isinstance(calls_raw, list):
+ raise fail('has a "calls" field that is not a list')
+ if len(calls_raw) > MAX_CALLS:
+ raise fail(f'has more than {MAX_CALLS} entries in "calls"')
+ if not calls_raw and status == "ok":
+ raise fail('has an empty "calls" list with status ok')
+
+ allowed = {h.lower() for h in (_clean_handles(list(handles or [])) or [])}
+ related_set = {h.lower() for h in (_clean_handles(list(related or [])) or [])}
+ allowed |= related_set
+
+ counters: dict[str, int] = {name: 0 for name in COUNTERS}
+ lane_counts: dict[str, int] = {lane: 0 for lane in LANES}
+ notes: list[str] = []
+ call_lanes: list[str] = []
+ topic_items: list[dict[str, Any]] = []
+ lane_calls: list[EnvelopeCall] = []
+ seen_ids: set[str] = set()
+ kept_ids: list[str] = []
+ total = 0
+ now_plus = datetime.now(timezone.utc) + timedelta(days=1)
+
+ for index, call in enumerate(calls_raw, start=1):
+ if not isinstance(call, dict):
+ raise fail(f"has a call {index} that is not an object")
+ posts = call.get("posts")
+ if not isinstance(posts, list) or any(not isinstance(row, dict) for row in posts):
+ raise fail(f'has a call {index} whose "posts" is not a list of objects')
+ if len(posts) > MAX_ROWS_PER_CALL:
+ raise fail(f"has a call {index} with more than {MAX_ROWS_PER_CALL} rows")
+ total += len(posts)
+ if total > MAX_ROWS_TOTAL:
+ raise fail(f"has more than {MAX_ROWS_TOTAL} rows in total")
+
+ lane_raw = call.get("lane")
+ lane = lane_raw.strip().lower() if isinstance(lane_raw, str) else ""
+ call_handles = _clean_handles(call.get("handles"))
+ if lane not in LANES:
+ if lane_raw is not None:
+ counters["lane-mismatch"] += 1
+ notes.append(f"call {index}: unknown lane served as topic")
+ lane = "topic"
+ call_handles = []
+ elif lane != "topic":
+ valid = bool(call_handles) and set(call_handles) <= allowed
+ if valid and lane == "related":
+ valid = set(call_handles) <= related_set
+ if not valid:
+ counters["lane-mismatch"] += 1
+ notes.append(
+ f"call {index}: {lane} lane served as topic "
+ "(handles are not in --x-handle/--x-related)"
+ )
+ lane = "topic"
+ call_handles = []
+ else:
+ call_handles = []
+ call_lanes.append(lane)
+
+ kept: list[dict[str, Any]] = []
+ for row in posts:
+ item = _ingest_row(
+ row, topic=topic, from_date=from_date, to_date=to_date,
+ now_plus=now_plus, seen_ids=seen_ids, counters=counters,
+ )
+ if item is None:
+ continue
+ author = item["author_handle"].lower()
+ if lane == "from" or lane == "related":
+ if author not in call_handles:
+ counters["lane-mismatch"] += 1
+ seen_ids.discard(item["post_id"])
+ continue
+ elif lane == "mention":
+ if author in call_handles or any(
+ is_own_post(item["url"], handle) for handle in call_handles
+ ):
+ counters["lane-mismatch"] += 1
+ seen_ids.discard(item["post_id"])
+ continue
+ kept.append(item)
+ kept_ids.append(item["post_id"])
+
+ lane_counts[lane] += len(kept)
+ if lane == "topic":
+ topic_items.extend(kept)
+ else:
+ lane_calls.append(EnvelopeCall(index=index, lane=lane, handles=call_handles, posts=kept))
+
+ accepted = len(kept_ids)
+ if status == "error":
+ # An error envelope carries no evidence: the outcome is the story.
+ topic_items, lane_calls = [], []
+ lane_counts = {lane: 0 for lane in LANES}
+ accepted = 0
+ elif kept_ids and _looks_generated(kept_ids):
+ status = "error"
+ error_category = _CATEGORY_GENERATED
+ topic_items, lane_calls = [], []
+ lane_counts = {lane: 0 for lane in LANES}
+ accepted = 0
+ notes.append("id sequence looks generated; envelope rejected")
+
+ for position, item in enumerate(
+ [*topic_items, *(post for call in lane_calls for post in call.posts)], start=1
+ ):
+ item["id"] = f"{ID_PREFIX}{position}"
+
+ envelope = Envelope(
+ path=raw_path,
+ sha256=digest,
+ status=status,
+ error_category=error_category,
+ provider=provider,
+ window=(window_from.isoformat(), window_to.isoformat()),
+ topic_items=topic_items,
+ lane_calls=lane_calls,
+ counters=counters,
+ accepted=accepted,
+ total=total,
+ lane_counts=lane_counts,
+ call_lanes=call_lanes,
+ warnings=warnings,
+ notes=notes,
+ )
+ _log(envelope.receipt())
+ for warning in warnings:
+ _log(warning)
+ return envelope
+
+
+def _ingest_row(
+ row: dict[str, Any],
+ *,
+ topic: str,
+ from_date: str,
+ to_date: str,
+ now_plus: datetime,
+ seen_ids: set[str],
+ counters: dict[str, int],
+) -> dict[str, Any] | None:
+ """Rebuild one flat row into the engine's X item shape, or drop it.
+
+ Returns ``None`` after incrementing the matching counter. The citation
+ URL is always engine-built from the id and the validated handle.
+ """
+ if set(row) - ROW_FIELDS:
+ counters["extra-fields"] += 1
+
+ post_id = _post_id(row.get("id"))
+ text_raw = row.get("text")
+ text = _clean_text(text_raw) if isinstance(text_raw, str) else ""
+ if not post_id or not text.strip():
+ counters["missing-id-text"] += 1
+ return None
+ text = text.strip()
+ if len(text) > MAX_TEXT_CHARS:
+ text = text[:MAX_TEXT_CHARS]
+ counters["truncated"] += 1
+
+ decoded = _decode_snowflake(post_id)
+ if decoded is None or decoded < _SNOWFLAKE_FLOOR or decoded > now_plus:
+ counters["out-of-window"] += 1
+ return None
+ snowflake_day = decoded.date()
+
+ created_raw = row.get("created_at")
+ if isinstance(created_raw, str) and created_raw.strip():
+ created = _parse_iso(created_raw)
+ if created is not None and abs((created.date() - snowflake_day).days) > 1:
+ counters["date-mismatch"] += 1
+ return None
+
+ day = snowflake_day.isoformat()
+ if (from_date and day < from_date) or (to_date and day > to_date):
+ counters["out-of-window"] += 1
+ return None
+
+ if post_id in seen_ids:
+ counters["duplicate"] += 1
+ return None
+ seen_ids.add(post_id)
+
+ handle = _handle(row.get("author_handle"))
+ url_raw = row.get("url")
+ if isinstance(url_raw, str) and url_raw.strip():
+ match = _STATUS_URL_RE.match(url_raw.strip())
+ url_handle = match.group("handle").lower() if match else ""
+ consistent = bool(match) and match.group("id") == post_id and (
+ url_handle == "i" or not handle or url_handle == handle.lower()
+ )
+ if not consistent:
+ counters["handle-mismatch"] += 1
+ handle = ""
+
+ if handle:
+ url = f"https://x.com/{handle}/status/{post_id}"
+ else:
+ counters["uncitable"] += 1
+ url = f"https://x.com/i/status/{post_id}"
+
+ return {
+ "id": "",
+ "text": text,
+ "url": url,
+ "author_handle": handle,
+ "date": day,
+ "engagement": {
+ "likes": _int(row.get("likes")),
+ "reposts": _int(row.get("reposts")),
+ "replies": _int(row.get("replies")),
+ "quotes": _int(row.get("quotes")),
+ },
+ "mentioned_handles": leading_mentions(text),
+ "why_relevant": "",
+ "relevance": _compute_relevance(topic, text) if topic else 0.5,
+ "post_id": post_id,
+ }
diff --git a/skills/last30days/scripts/lib/xquik.py b/skills/last30days/scripts/lib/xquik.py
index b57816d6..126fcc6a 100644
--- a/skills/last30days/scripts/lib/xquik.py
+++ b/skills/last30days/scripts/lib/xquik.py
@@ -12,6 +12,7 @@ from typing import Any, Dict, List, Optional
from . import http, log
from .relevance import token_overlap_relevance as _compute_relevance
+from .x_api import is_own_post
# Per-process probe cache: (state, reason). state is "unset" until probed, then
# True (funded/working) | False (auth/payment failure) | None (inconclusive).
@@ -148,8 +149,10 @@ def _execute_search(
if status == 402:
# Unpaid key — fatal for the source, and surfaced on the real search
# path (not just --diagnose) so a live run reports it instead of
- # settling silently empty.
- return [], "Xquik key unpaid (402)"
+ # settling silently empty. The X retrieval branch classifies by
+ # message text only, so the detail carries the "payment required"
+ # marker that http.classify_failure maps to PAYMENT_REQUIRED.
+ return [], "Xquik key unpaid: payment required (402)"
if status in (401, 403):
return [], f"Xquik auth failed ({status})"
_log(f"HTTP error for '{label}': {exc}")
@@ -175,15 +178,10 @@ def _execute_search(
return items, None
-def _is_own(url: str, handle: str) -> bool:
- """True when a tweet URL is authored by ``handle`` (their own post).
-
- Used by the ABOUT lane to drop the subject's own tweets so only mentions
- *by others* remain. Handles both x.com and twitter.com permalinks.
- """
- u = (url or "").lower()
- h = handle.lower().lstrip("@").strip()
- return bool(h) and (f"x.com/{h}/status" in u or f"twitter.com/{h}/status" in u)
+# True when a tweet URL is authored by ``handle`` (their own post). Used by
+# the ABOUT lane to drop the subject's own tweets so only mentions *by
+# others* remain; the implementation is ``x_api.is_own_post``.
+_is_own = is_own_post
def search_handles(
@@ -286,7 +284,7 @@ def probe_works(token: str, timeout: int = 8) -> Optional[bool]:
except http.HTTPError as exc:
status = getattr(exc, "status_code", None)
if status == 402:
- _probe_cache = (False, "xquik key unpaid (402)")
+ _probe_cache = (False, "xquik key unpaid: payment required (402)")
elif status in (401, 403):
_probe_cache = (False, f"xquik auth failed ({status})")
else:
diff --git a/skills/last30days/scripts/lib/xurl_x.py b/skills/last30days/scripts/lib/xurl_x.py
index 556b5db5..4b46ca0b 100644
--- a/skills/last30days/scripts/lib/xurl_x.py
+++ b/skills/last30days/scripts/lib/xurl_x.py
@@ -18,8 +18,10 @@ import subprocess
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
-from . import log
-from .relevance import token_overlap_relevance as _compute_relevance
+from . import health, http, log
+from . import x_api
+# One X API v2 depth table and one parser: xurl_x re-imports both.
+from .x_api import DEPTH_CONFIG
# xurl auth status marks a configured app-only bearer as "bearer: ✓".
# Search uses --auth app, so availability must require this — oauth1 alone
@@ -31,13 +33,6 @@ def _log(msg: str) -> None:
log.source_log("xurl", msg, tty_only=False)
-# Depth configurations: number of results to request
-DEPTH_CONFIG = {
- "quick": 10,
- "default": 30,
- "deep": 60,
-}
-
# Memoized availability, mirroring health.py's per-process dependency-probe
# cache: each uncached is_available() check spawns an `xurl auth status`
@@ -155,6 +150,37 @@ def has_stored_auth() -> bool:
return shutil.which("xurl") is not None and stored_auth_status()[0] == AUTH_OK
+# Engine-authored fixed error strings (the same rule as x_api): xurl's
+# stderr can echo the request, the bearer, or an account id, so it never
+# reaches the outcome detail. Each string carries a marker that
+# http.classify_failure recognizes.
+ERR_PAYMENT_REQUIRED = "xurl: payment required (X API credits exhausted)"
+ERR_UNAUTHORIZED = "xurl: unauthorized (bearer token rejected)"
+ERR_FORBIDDEN = "xurl: forbidden (bearer token lacks access)"
+ERR_RATE_LIMITED = "xurl: rate limit exceeded (X API)"
+ERR_FAILED = "xurl: search failed"
+ERR_INVALID_JSON = "xurl: invalid JSON from xurl"
+ERR_NOT_FOUND = "xurl not found in PATH"
+ERR_TIMED_OUT = "xurl search timed out (30s)"
+
+
+def _classify_cli_failure(output: str) -> str:
+ """Map xurl's stderr/stdout to a fixed string via the shared classifier."""
+ text = output or ""
+ state = http.classify_failure(message=text)
+ if state == health.PAYMENT_REQUIRED:
+ return ERR_PAYMENT_REQUIRED
+ if state == health.AUTH_FAILED:
+ # The shared vocabulary has one auth state; split the wording only.
+ lowered = text.lower()
+ if "401" in lowered or "unauthorized" in lowered:
+ return ERR_UNAUTHORIZED
+ return ERR_FORBIDDEN
+ if state == health.RATE_LIMITED:
+ return ERR_RATE_LIMITED
+ return ERR_FAILED
+
+
def search_x(
query: str,
depth: str = "default",
@@ -167,7 +193,8 @@ def search_x(
Returns:
Raw JSON response from X API v2 tweets/search/recent, or a dict
- with an "error" key on failure.
+ with an "error" key holding a fixed string on failure. The CLI's
+ own output reaches only the debug log, never the error.
"""
max_results = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
# X API v2 search/recent requires max_results in 10–100 range
@@ -186,28 +213,32 @@ def search_x(
if result.returncode != 0:
error_text = result.stderr.strip() or result.stdout.strip()
- return {"error": f"xurl search failed: {error_text}"}
+ classified = _classify_cli_failure(error_text)
+ # The raw CLI output can echo the bearer; only the fixed string
+ # and its size reach the debug log.
+ log.debug(f"xurl exit {result.returncode}: {classified} ({len(error_text)} chars)")
+ return {"error": classified}
return json.loads(result.stdout)
except FileNotFoundError:
- return {"error": "xurl not found in PATH"}
+ return {"error": ERR_NOT_FOUND}
except subprocess.TimeoutExpired:
- return {"error": "xurl search timed out (30s)"}
- except json.JSONDecodeError as exc:
- return {"error": f"Invalid JSON from xurl: {exc}"}
+ return {"error": ERR_TIMED_OUT}
+ except json.JSONDecodeError:
+ return {"error": ERR_INVALID_JSON}
except Exception as exc:
- return {"error": f"{type(exc).__name__}: {exc}"}
+ return {"error": f"xurl: {type(exc).__name__}"}
def parse_x_response(
response: Dict[str, Any],
topic: str = "",
) -> List[Dict[str, Any]]:
- """Parse xurl search response into normalized item dicts.
+ """Parse an xurl search response into normalized item dicts.
- Output format matches the existing XItem schema used by xai_x and bird_x:
- id, text, url, author_handle, date, engagement, why_relevant, relevance.
+ Delegates to the shared X API v2 parser (``x_api.parse_v2_response``)
+ so xurl and xapi never drift; only the id prefix differs.
Args:
response: Raw X API v2 response dict from search_x()
@@ -216,62 +247,7 @@ def parse_x_response(
Returns:
List of item dicts. Empty list on error or no results.
"""
- items: List[Dict[str, Any]] = []
-
- if "error" in response:
+ if isinstance(response, dict) and "error" in response:
_log(f"Error in response: {response['error']}")
- return items
-
- data = response.get("data") or []
- if not data:
- return items
-
- # Build author lookup from includes.users
- authors: Dict[str, Dict[str, Any]] = {}
- for user in (response.get("includes") or {}).get("users") or []:
- authors[user["id"]] = user
-
- for i, tweet in enumerate(data):
- author_id = tweet.get("author_id", "")
- author = authors.get(author_id, {})
- username = author.get("username", "")
-
- tweet_id = tweet.get("id", "")
- url = f"https://x.com/{username}/status/{tweet_id}" if username else ""
-
- # Parse public_metrics
- engagement: Optional[Dict[str, Any]] = None
- metrics = tweet.get("public_metrics") or {}
- if metrics:
- engagement = {
- "likes": metrics.get("like_count", 0),
- "reposts": metrics.get("retweet_count", 0),
- "replies": metrics.get("reply_count", 0),
- "quotes": metrics.get("quote_count", 0),
- }
-
- # Parse ISO 8601 date → YYYY-MM-DD
- date: Optional[str] = None
- created = tweet.get("created_at", "")
- if created:
- m = re.match(r"(\d{4}-\d{2}-\d{2})", created)
- if m:
- date = m.group(1)
-
- text = tweet.get("text", "").strip()
-
- # Relevance score via shared token-overlap function
- relevance = _compute_relevance(topic, text) if topic else 0.5
-
- items.append({
- "id": f"XURL{i + 1}",
- "text": text[:500],
- "url": url,
- "author_handle": username,
- "date": date,
- "engagement": engagement,
- "why_relevant": "",
- "relevance": relevance,
- })
-
- return items
+ return []
+ return x_api.parse_v2_response(response, topic, None, id_prefix="XURL")
diff --git a/skills/last30days/scripts/setup-keychain.sh b/skills/last30days/scripts/setup-keychain.sh
index 2d476c3d..696d2233 100755
--- a/skills/last30days/scripts/setup-keychain.sh
+++ b/skills/last30days/scripts/setup-keychain.sh
@@ -42,6 +42,7 @@ ALL_KEYS=(
XIAOHONGSHU_API_BASE
GITHUB_TOKEN
BRIGHTDATA_API_KEY
+ X_BEARER_TOKEN
)
if [[ "${OSTYPE:-}" != darwin* ]]; then
diff --git a/skills/last30days/scripts/setup-pass.sh b/skills/last30days/scripts/setup-pass.sh
index 65799f08..0e995daf 100755
--- a/skills/last30days/scripts/setup-pass.sh
+++ b/skills/last30days/scripts/setup-pass.sh
@@ -44,6 +44,7 @@ ALL_KEYS=(
XIAOHONGSHU_API_BASE
GITHUB_TOKEN
BRIGHTDATA_API_KEY
+ X_BEARER_TOKEN
)
REPLACE=0
diff --git a/tests/fixtures/agent_export_v1.json b/tests/fixtures/agent_export_v1.json
index e2d33738..0743fb0d 100644
--- a/tests/fixtures/agent_export_v1.json
+++ b/tests/fixtures/agent_export_v1.json
@@ -1,5 +1,5 @@
{
- "schema_version": "1.2",
+ "schema_version": "1.3",
"query": "AI coding agents",
"generated_at": "2026-07-10T00:00:00Z",
"window_days": 30,
@@ -9,6 +9,7 @@
"grounding": "unreachable",
"hackernews": "no-results",
"reddit": "ok",
+ "tiktok": "payment-required",
"x": "ok",
"youtube": "rate-limited"
},
diff --git a/tests/test_agent_export.py b/tests/test_agent_export.py
index acd38c38..94a0ff3a 100644
--- a/tests/test_agent_export.py
+++ b/tests/test_agent_export.py
@@ -158,11 +158,12 @@ def _report() -> schema.Report:
"youtube": schema.SourceOutcome(source="youtube", state=schema.RATE_LIMITED),
"grounding": schema.SourceOutcome(source="grounding", state=schema.UNREACHABLE),
"github": schema.SourceOutcome(source="github", state=schema.AUTH_FAILED),
+ "tiktok": schema.SourceOutcome(source="tiktok", state=schema.PAYMENT_REQUIRED),
},
)
-def test_agent_export_matches_v1_2_golden_contract():
+def test_agent_export_matches_v1_3_golden_contract():
expected = json.loads(GOLDEN.read_text(encoding="utf-8"))
assert schema.to_agent_export(_report()) == expected
@@ -177,6 +178,7 @@ def test_agent_export_maps_per_run_source_outcomes_to_states():
"grounding": "unreachable",
"hackernews": "no-results",
"reddit": "ok",
+ "tiktok": "payment-required",
"x": "ok",
"youtube": "rate-limited",
}
diff --git a/tests/test_backend_descriptors.py b/tests/test_backend_descriptors.py
index b445627e..879c48de 100644
--- a/tests/test_backend_descriptors.py
+++ b/tests/test_backend_descriptors.py
@@ -155,15 +155,19 @@ class TestDescriptorRegistry:
assert d.mode == backends.MODE_ALTERNATIVE
# Auto chain order: bird first, grok excluded (opt-in only).
assert env.X_BACKEND_ORDER == ("bird", "xai", "xurl", "xquik")
- # Grok is opt-in only, not in the auto chain.
- assert env.X_BACKEND_OPT_IN == ("grok",)
+ # Grok and xapi are opt-in only off Grok Bot, not in the auto chain.
+ assert env.X_BACKEND_OPT_IN == ("grok", "xapi")
# All known backends (auto + opt-in) for pin validation.
- assert env.X_BACKEND_KNOWN == ("bird", "xai", "xurl", "xquik", "grok")
+ assert env.X_BACKEND_KNOWN == ("bird", "xai", "xurl", "xquik", "grok", "xapi")
# Descriptor includes all backends (auto + opt-in) for doctor visibility.
assert tuple(s.name for s in d.backends) == env.X_BACKEND_ORDER + env.X_BACKEND_OPT_IN
- # Grok is marked opt-in in the descriptor.
+ # Grok and xapi are marked opt-in in the descriptor.
grok_spec = next(s for s in d.backends if s.name == "grok")
assert grok_spec.opt_in is True
+ xapi_spec = next(s for s in d.backends if s.name == "xapi")
+ assert xapi_spec.opt_in is True
+ assert xapi_spec.paid is True
+ assert xapi_spec.requires == "X_BEARER_TOKEN (X API v2)"
# Auto chain backends are NOT marked opt-in.
for name in env.X_BACKEND_ORDER:
spec = next(s for s in d.backends if s.name == name)
@@ -700,6 +704,58 @@ class TestXParityWithPipeline:
bird_installed=True,
)
+ # Host rows (U1): the policy shapes both sides identically.
+
+ def test_parity_grok_bot_bearer(self):
+ self._assert_parity(
+ {"LAST30DAYS_HOST": "grok-bot", "X_BEARER_TOKEN": "dummy-bearer",
+ "AUTH_TOKEN": "dummy-token", "CT0": "dummy-ct0"},
+ bird_installed=True,
+ )
+
+ def test_parity_grok_bot_xai_key(self):
+ self._assert_parity(
+ {"LAST30DAYS_HOST": "grok-bot", "XAI_API_KEY": "dummy-key",
+ "AUTH_TOKEN": "dummy-token", "CT0": "dummy-ct0"},
+ bird_installed=True,
+ )
+
+ def test_parity_grok_bot_cookies_only_unpinned(self):
+ self._assert_parity(
+ {"LAST30DAYS_HOST": "grok-bot", "AUTH_TOKEN": "dummy-token", "CT0": "dummy-ct0",
+ "XQUIK_API_KEY": "dummy-key"},
+ bird_installed=True,
+ )
+
+ def test_parity_grok_bot_bird_pin(self):
+ self._assert_parity(
+ {"LAST30DAYS_HOST": "grok-bot", "LAST30DAYS_X_BACKEND": "bird",
+ "AUTH_TOKEN": "dummy-token", "CT0": "dummy-ct0"},
+ bird_installed=True,
+ )
+
+ def test_parity_grok_bot_grok_pin(self):
+ self._assert_parity(
+ {"LAST30DAYS_HOST": "grok-bot", "LAST30DAYS_X_BACKEND": "grok"},
+ grok_installed=True,
+ grok_authed=True,
+ )
+
+ def test_parity_non_grok_linux_cookies_and_bearer(self):
+ with mock.patch("platform.system", return_value="Linux"):
+ self._assert_parity(
+ {"AUTH_TOKEN": "dummy-token", "CT0": "dummy-ct0",
+ "X_BEARER_TOKEN": "dummy-bearer"},
+ bird_installed=True,
+ )
+
+ def test_parity_macbook_xai_key(self):
+ with (
+ mock.patch("platform.system", return_value="Darwin"),
+ mock.patch.object(env, "_mac_model", return_value="MacBookPro18,2"),
+ ):
+ self._assert_parity({"XAI_API_KEY": "dummy-key"})
+
# ---------------------------------------------------------------------------
# get_x_source_status pin semantics (R4): grok pin forces grok source
diff --git a/tests/test_box_chrome_login.py b/tests/test_box_chrome_login.py
index c2e05701..7c77d0c0 100644
--- a/tests/test_box_chrome_login.py
+++ b/tests/test_box_chrome_login.py
@@ -119,6 +119,51 @@ def test_default_run_prints_but_does_not_spawn():
assert bcl.main([]) == 0
+# --- Official-only host (LAST30DAYS_HOST=grok-bot) --------------------------
+
+GROK_BOT_CONFIG = {"LAST30DAYS_HOST": "grok-bot"}
+
+
+def test_grok_bot_host_gets_no_launch_command_even_when_extras_apply():
+ """U6: an official-only host gets the same no-launch recipe as a MacBook
+ even when the extras signals (Linux / box-chrome on PATH) are present."""
+ with (
+ mock.patch("lib.env.x_extras_enabled", return_value=True),
+ mock.patch("shutil.which", return_value="/usr/local/bin/box-chrome"),
+ ):
+ recipe = bcl.build_recipe(dict(GROK_BOT_CONFIG))
+ assert recipe["applies"] is False
+ assert recipe["command"] is None
+ assert recipe["env"] is None
+ rendered = bcl.render_recipe(recipe).lower()
+ assert "no launch needed" in rendered
+ # R4: nothing in the Grok Bot output names the cookie machinery.
+ for banned in ("cookie", "cdp", "bird", "auth_token", "ct0", "keychain"):
+ assert banned not in rendered, f"{banned!r} leaked into Grok Bot output"
+
+
+def test_main_exec_never_spawns_on_grok_bot_host():
+ with (
+ mock.patch("lib.env.x_extras_enabled", return_value=True),
+ mock.patch("shutil.which", return_value="/usr/local/bin/box-chrome"),
+ mock.patch.object(bcl.env, "get_config", return_value=dict(GROK_BOT_CONFIG)),
+ mock.patch("os.makedirs", side_effect=AssertionError("no profile dir on Grok Bot")),
+ mock.patch("subprocess.Popen", side_effect=AssertionError("no spawn on Grok Bot")),
+ ):
+ assert bcl.main(["--exec"]) == 0
+
+
+def test_linux_and_mac_mini_recipe_unchanged_without_host_signal():
+ """R14/R15: without LAST30DAYS_HOST the extras recipe is what it was."""
+ with (
+ mock.patch("lib.env.x_extras_enabled", return_value=True),
+ mock.patch("shutil.which", return_value="/usr/local/bin/box-chrome"),
+ ):
+ recipe = bcl.build_recipe({})
+ assert recipe["applies"] is True
+ assert recipe["command"] == ["/usr/local/bin/box-chrome", "--new-window", "https://x.com/login"]
+
+
# --- SKILL.md recipe contract ---------------------------------------------
@@ -136,7 +181,10 @@ def test_skill_md_references_helper_in_flows_and_repair():
def test_skill_md_documents_18800_launch_and_macbook_skip():
text = _skill()
assert "SAND_CHROME_REMOTE_DEBUG_PORT=18800" in text
- assert "X on Linux / Grok Bot / Mac mini" in text
+ assert "X on Linux / Mac mini (repair)" in text
+ # The repair recipe is rescoped off Grok Bot (R15): that host has its own
+ # official-only flow and never launches box-chrome.
+ assert "X on Linux / Grok Bot / Mac mini" not in text
# MacBook must be told to skip the box-chrome path.
assert "MacBook SKIPS" in text or "MacBook does NOT" in text
diff --git a/tests/test_check_config_x_bearer.py b/tests/test_check_config_x_bearer.py
new file mode 100644
index 00000000..255057ac
--- /dev/null
+++ b/tests/test_check_config_x_bearer.py
@@ -0,0 +1,115 @@
+"""hooks/scripts/check-config.sh: X_BEARER_TOKEN counts toward HAS_X (U6/R17).
+
+The SessionStart hook counts X as an active source when a cookie pair
+(AUTH_TOKEN + CT0) or XAI_API_KEY is present. The X API bearer is a third
+way to have X, so a bearer set alone must raise the numeric source count by
+exactly one, the same way XAI_API_KEY alone does.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+HOOK = Path(__file__).resolve().parents[1] / "hooks" / "scripts" / "check-config.sh"
+
+_CREDENTIAL_VARS = (
+ "LAST30DAYS_MEMORY_DIR",
+ "SETUP_COMPLETE",
+ "LAST30DAYS_CONFIG_DIR",
+ "LAST30DAYS_TRUST_PROJECT_CONFIG",
+ "OPENAI_API_KEY",
+ "SCRAPECREATORS_API_KEY",
+ "AUTH_TOKEN",
+ "CT0",
+ "XAI_API_KEY",
+ "X_BEARER_TOKEN",
+ "BSKY_HANDLE",
+ "EXA_API_KEY",
+ "EXCLUDE_SOURCES",
+)
+
+
+def _isolated_path(tmp_path: Path) -> str:
+ """PATH with a stub ``security`` so a real Keychain item cannot leak in."""
+ bin_dir = tmp_path / "hook-bin"
+ bin_dir.mkdir(exist_ok=True)
+ security = bin_dir / "security"
+ if not security.exists():
+ security.write_text("#!/bin/sh\nexit 1\n", encoding="utf-8")
+ security.chmod(0o755)
+ return f"{bin_dir}{os.pathsep}{os.environ.get('PATH', '')}"
+
+
+def _run_hook(tmp_path: Path, overrides: dict[str, str]) -> subprocess.CompletedProcess[str]:
+ bash_path = shutil.which("bash")
+ if bash_path is None:
+ pytest.skip("bash not on PATH")
+ env = os.environ.copy()
+ for key in _CREDENTIAL_VARS:
+ env.pop(key, None)
+ cfg_dir = tmp_path / "cfg"
+ cfg_dir.mkdir(exist_ok=True)
+ env["LAST30DAYS_CONFIG_DIR"] = str(cfg_dir) # no .env file inside: env-only
+ env["LAST30DAYS_MEMORY_DIR"] = str(tmp_path / "mem")
+ env["PATH"] = _isolated_path(tmp_path)
+ env["SETUP_COMPLETE"] = "true"
+ env.update(overrides)
+ return subprocess.run(
+ [bash_path, str(HOOK)],
+ capture_output=True,
+ text=True,
+ env=env,
+ cwd=str(tmp_path),
+ timeout=30,
+ check=False,
+ )
+
+
+def _source_count(stdout: str) -> int:
+ match = re.search(r"Ready\s+[—–-]\s+(\d+)\s+sources?\s+active", stdout)
+ assert match, f"could not find source count in hook stdout: {stdout!r}"
+ return int(match.group(1))
+
+
+def test_x_bearer_token_alone_counts_x_as_active(tmp_path: Path):
+ base = _run_hook(tmp_path, {})
+ assert base.returncode == 0, base.stderr
+ bearer = _run_hook(tmp_path, {"X_BEARER_TOKEN": "dummy-x-bearer-not-real"})
+ assert bearer.returncode == 0, bearer.stderr
+ xai = _run_hook(tmp_path, {"XAI_API_KEY": "dummy-xai-not-real"})
+ assert xai.returncode == 0, xai.stderr
+
+ assert _source_count(bearer.stdout) == _source_count(base.stdout) + 1
+ # Same weight as the xAI key: X is one source however it is reached.
+ assert _source_count(bearer.stdout) == _source_count(xai.stdout)
+ # The hook never echoes credential values.
+ assert "dummy-x-bearer-not-real" not in bearer.stdout + bearer.stderr
+
+
+def test_x_bearer_token_from_env_file_counts_x_as_active(tmp_path: Path):
+ cfg_dir = tmp_path / "cfg"
+ cfg_dir.mkdir(exist_ok=True)
+ env_file = cfg_dir / ".env"
+ env_file.write_text("SETUP_COMPLETE=true\nX_BEARER_TOKEN=dummy-x-bearer-not-real\n", encoding="utf-8")
+ env_file.chmod(0o600)
+ with_file = _run_hook(tmp_path, {})
+ assert with_file.returncode == 0, with_file.stderr
+
+ env_file.write_text("SETUP_COMPLETE=true\n", encoding="utf-8")
+ without = _run_hook(tmp_path, {})
+ assert without.returncode == 0, without.stderr
+
+ assert _source_count(with_file.stdout) == _source_count(without.stdout) + 1
+ assert "dummy-x-bearer-not-real" not in with_file.stdout + with_file.stderr
+
+
+def test_bearer_and_xai_key_together_count_x_once(tmp_path: Path):
+ both = _run_hook(tmp_path, {"X_BEARER_TOKEN": "dummy-a", "XAI_API_KEY": "dummy-b"})
+ xai = _run_hook(tmp_path, {"XAI_API_KEY": "dummy-b"})
+ assert _source_count(both.stdout) == _source_count(xai.stdout)
diff --git a/tests/test_cli_v3.py b/tests/test_cli_v3.py
index 4392713e..7ad5433e 100644
--- a/tests/test_cli_v3.py
+++ b/tests/test_cli_v3.py
@@ -64,7 +64,7 @@ class CliV3Tests(unittest.TestCase):
)
self.assertEqual(0, result.returncode, result.stderr)
payload = json.loads(result.stdout)
- self.assertEqual("1.2", payload["schema_version"])
+ self.assertEqual("1.3", payload["schema_version"])
self.assertEqual("test topic", payload["query"])
self.assertIn("results", payload)
self.assertIn("clusters", payload)
diff --git a/tests/test_codex_host_contract.py b/tests/test_codex_host_contract.py
index f9b964c4..2d05c62c 100644
--- a/tests/test_codex_host_contract.py
+++ b/tests/test_codex_host_contract.py
@@ -12,7 +12,9 @@ SKILL_MD = ROOT / "skills" / "last30days" / "SKILL.md"
def _prose_flow() -> str:
text = SKILL_MD.read_text(encoding="utf-8")
start_marker = "### Non-Modal Prose Flow"
- end_marker = "### Manual Setup Guide"
+ # The Grok Bot Prose Flow (third Step 0 branch) sits between the prose
+ # flow and the Manual Setup Guide; slice the prose flow alone.
+ end_marker = "### Grok Bot Prose Flow"
start = text.find(start_marker)
assert start != -1, f"missing section marker: {start_marker}"
end = text.find(end_marker, start)
diff --git a/tests/test_config_x_backends.py b/tests/test_config_x_backends.py
index 01478729..89f1af2b 100644
--- a/tests/test_config_x_backends.py
+++ b/tests/test_config_x_backends.py
@@ -1,25 +1,84 @@
-"""CONFIGURATION.md must not advertise X backends the engine does not have (#942)."""
+"""CONFIGURATION.md must not advertise X backends the engine does not have (#942).
+The X / Twitter row and the X env-var rows are the user-facing surface for the
+X backend set, so they are held to ``env.X_BACKEND_KNOWN`` and to the official
+X path copy (U8): the ``X_BEARER_TOKEN`` bearer path is described as about a
+week, never as parity with the connector lane (R6).
+"""
+
+import re
from pathlib import Path
from lib import env
REPO = Path(__file__).resolve().parents[1]
+BEARER_COVERAGE = (
+ "recent posts, about the last week, unless your X developer project has "
+ "full-archive access"
+)
+
+
+def _configuration() -> str:
+ return (REPO / "CONFIGURATION.md").read_text(encoding="utf-8")
def _x_twitter_row() -> str:
- for line in (REPO / "CONFIGURATION.md").read_text(encoding="utf-8").splitlines():
+ for line in _configuration().splitlines():
if line.startswith("| X / Twitter |"):
return line
raise AssertionError("CONFIGURATION.md is missing the X / Twitter table row")
+def _env_row(var: str) -> str:
+ for line in _configuration().splitlines():
+ if line.startswith(f"| `{var}` |"):
+ return line
+ raise AssertionError(f"CONFIGURATION.md is missing the env-var table row for {var}")
+
+
def test_configuration_x_row_omits_scrapecreators():
row = _x_twitter_row()
assert "SCRAPECREATORS_API_KEY" not in row
assert "ScrapeCreators" not in row
+def test_configuration_x_row_names_only_real_backends():
+ """Every backend the row pins (LAST30DAYS_X_BACKEND=<name>) is a known backend."""
+ row = _x_twitter_row()
+ pinned = re.findall(r"LAST30DAYS_X_BACKEND=([a-z]+)", row)
+ assert pinned, "the X row should show at least one opt-in pin"
+ assert set(pinned) <= set(env.X_BACKEND_KNOWN)
+
+
+def test_configuration_x_row_lists_bearer_with_coverage_caveat():
+ row = _x_twitter_row()
+ assert "X_BEARER_TOKEN" in row
+ assert "LAST30DAYS_X_BACKEND=xapi" in row, "xapi is opt-in outside Grok Bot"
+ assert BEARER_COVERAGE in row
+ # The row must not sell the bearer path as full-window parity.
+ assert "full 30-day" not in row.split("X_BEARER_TOKEN", 1)[1].split("`XAI_API_KEY`", 1)[0]
+
+
+def test_configuration_env_rows_for_the_official_x_path():
+ host = _env_row("LAST30DAYS_HOST")
+ assert "grok-bot" in host
+ bearer = _env_row("X_BEARER_TOKEN")
+ assert BEARER_COVERAGE in bearer
+ lane = _env_row("LAST30DAYS_X_HOST_LANE")
+ # KTD10: the lane signal is read from the process environment only.
+ assert "process environment" in lane
+ assert ".env" in lane and "ignored" in lane
+
+
+def test_configuration_pin_row_names_xapi_and_the_grok_bot_rule():
+ row = _env_row("LAST30DAYS_X_BACKEND")
+ assert "`xapi`" in row
+ # KTD7: on a Grok Bot host the pin is the only way outside the official chain.
+ assert "Grok Bot" in row
+ assert "only way" in row
+ assert "xapi → xai → xurl" in row
+
+
def test_engine_has_no_scrapecreators_x_backend():
assert "scrapecreators" not in env.X_BACKEND_KNOWN
assert env._x_backend_available("scrapecreators", {"SCRAPECREATORS_API_KEY": "k"}, False) is False
diff --git a/tests/test_diagnose_compat.py b/tests/test_diagnose_compat.py
index 3f0d2658..09737466 100644
--- a/tests/test_diagnose_compat.py
+++ b/tests/test_diagnose_compat.py
@@ -106,6 +106,10 @@ PREFLIGHT_PROJECT_CONFIG_KEYS = {"status", "trusted", "ignored_path", "ignored_k
PREFLIGHT_BROWSER_COOKIES_KEYS = {"status", "mode", "browsers", "reads_values"}
PREFLIGHT_CREDENTIALS_KEYS = {
"google", "openai", "xai", "openrouter", "perplexity", "scrapecreators", "github",
+ # X API bearer (X_BEARER_TOKEN): computed inside permission_preflight from
+ # config, never surfaced through diagnose.providers (whose key set above
+ # stays frozen).
+ "x_bearer",
}
PREFLIGHT_NETWORK_KEYS = {
"available_sources", "native_search", "endpoint_overrides", "ignored_endpoint_overrides",
diff --git a/tests/test_doc_security_contract.py b/tests/test_doc_security_contract.py
index f20235c7..a8f2e0c7 100644
--- a/tests/test_doc_security_contract.py
+++ b/tests/test_doc_security_contract.py
@@ -56,6 +56,19 @@ def test_security_copy_avoids_stale_cookie_and_endpoint_claims():
assert "do not read browser-cookie values" in skill
+def test_security_copy_names_official_x_api_and_host_envelope():
+ """The official X API v2 host and the host-provided --x-posts envelope are
+ named in the Security section and the permissions overview."""
+ skill = SKILL_MD.read_text(encoding="utf-8")
+ security = skill[skill.index("## Security & Permissions"):]
+ assert "api.x.com" in security
+ assert "X_BEARER_TOKEN" in security
+ assert "--x-posts" in security
+ overview = skill[skill.index("**Permissions overview:**"):]
+ overview = overview[: overview.index("\n")]
+ assert "api.x.com" in overview
+
+
def test_scrapecreators_copy_uses_canonical_free_call_count():
text = "\n".join(
[
diff --git a/tests/test_doctor.py b/tests/test_doctor.py
index 5aa8b0a0..b8046d3f 100644
--- a/tests/test_doctor.py
+++ b/tests/test_doctor.py
@@ -17,8 +17,10 @@ Covers the plan's U4 scenarios:
import datetime
import io
+import itertools
import json
import os
+import re
import sys
import tempfile
import unittest
@@ -28,7 +30,7 @@ from pathlib import Path
from unittest import mock
import last30days as cli
-from lib import backends, doctor, health, http, prescriptions
+from lib import backends, doctor, env, grok_x, health, http, prescriptions
BIRD_STATUS_OFF = {
"installed": False,
@@ -49,6 +51,7 @@ FAKE_SECRETS = {
"BSKY_APP_PASSWORD": "dummy-bsky-secret-000",
"TRUTHSOCIAL_TOKEN": "dummy-truth-secret-000",
"GITHUB_TOKEN": "dummy-github-secret-000",
+ "X_BEARER_TOKEN": "dummy-x-bearer-secret-000",
}
VALID_TIERS = {"ok", "warn", "off", "error"}
@@ -1283,5 +1286,335 @@ class CliHealth(unittest.TestCase):
self.assertTrue(gh["cli"]["optional"])
+# ---------------------------------------------------------------------------
+# Grok Bot host (official-only X policy): doctor names only the official path
+# ---------------------------------------------------------------------------
+
+# R4 vocabulary that must never appear in an X status or fix line on a Grok
+# Bot host (the pinned backend's own name is the one carve-out).
+GROK_BOT_FORBIDDEN = (
+ "cookie", "cdp", "box-chrome", "bird", "auth_token", "ct0", "xquik",
+ "grok login", "grok cli",
+)
+BEARER_CAVEAT = (
+ "recent posts, about the last week, unless your X developer project has "
+ "full-archive access"
+)
+
+
+def _grok_bot(**over):
+ cfg = {"LAST30DAYS_HOST": "grok-bot"}
+ cfg.update(over)
+ return cfg
+
+
+def _x_text_lines(text):
+ """The X source line plus its indented sub-lane lines from doctor text."""
+ lines = text.splitlines()
+ out = []
+ for i, line in enumerate(lines):
+ if not re.match(r"^\s*\S+\s+x(?:\s|$)", line):
+ continue
+ out.append(line)
+ for follow in lines[i + 1:]:
+ if follow.startswith(" "):
+ out.append(follow)
+ else:
+ break
+ break
+ return out
+
+
+def _grok_signed_in():
+ """A signed-in Grok CLI as the doctor probes see it (filesystem only)."""
+ return [
+ mock.patch("lib.backends.which", lambda n: "/usr/bin/grok" if n == "grok" else None),
+ mock.patch("lib.grok_x.has_stored_auth", return_value=True),
+ mock.patch(
+ "lib.grok_x.stored_auth_status",
+ return_value=(grok_x.AUTH_OK, "signed in", None),
+ ),
+ ]
+
+
+def _build_with(config, patches, **kwargs):
+ with _Hermetic(**kwargs):
+ for p in patches:
+ p.start()
+ try:
+ return doctor.build_report(dict(config))
+ finally:
+ for p in reversed(patches):
+ p.stop()
+
+
+class GrokBotHostDoctor(unittest.TestCase):
+ """AE2, AE2a, AE3: on a Grok Bot host every X line names only the
+ official path (connector lane, X_BEARER_TOKEN, XAI_API_KEY, xurl)."""
+
+ def _assert_official_vocabulary(self, report, allow=()):
+ blob = json.dumps(report["sources"]["x"]).lower()
+ text = "\n".join(_x_text_lines(doctor.render_text(report))).lower()
+ self.assertTrue(text, "doctor text has no X line")
+ for word in GROK_BOT_FORBIDDEN:
+ if word in allow:
+ continue
+ self.assertNotIn(word, blob, f"{word!r} in X JSON record")
+ self.assertNotIn(word, text, f"{word!r} in X text lines")
+
+ def test_nothing_configured_prescribes_bearer_and_names_no_cookie_path(self):
+ config = _grok_bot(
+ AUTH_TOKEN="dummy-auth-token-secret-000",
+ CT0="dummy-ct0-secret-000",
+ XQUIK_API_KEY="dummy-xquik-secret-000",
+ FROM_BROWSER="firefox",
+ AGENTCOOKIE="on",
+ BROWSER_CDP_URL="http://127.0.0.1:9222",
+ )
+ report = _build(config)
+ rec = report["sources"]["x"]
+ self.assertEqual("off", rec["tier"])
+ self.assertEqual("unconfigured", rec["status"])
+ self.assertIsNone(rec["active_backend"])
+ self.assertEqual(["xapi", "xai", "xurl"], [b["name"] for b in rec["backends"]])
+ entry = prescriptions.for_x(config, "cookies_missing")
+ self.assertEqual("bearer_missing", entry.failure)
+ self.assertIn(entry.fix_nl, rec["fix"])
+ self.assertIsNone(rec["pin_var"])
+ self.assertFalse(rec["backups"][0]["armed"])
+ self._assert_official_vocabulary(report)
+ self.assertNotIn("LAST30DAYS_X_BACKEND", json.dumps(rec))
+
+ def test_signed_in_grok_cli_names_neither_cli_nor_pin_variable(self):
+ report = _build_with(_grok_bot(), _grok_signed_in())
+ rec = report["sources"]["x"]
+ self.assertEqual("unconfigured", rec["status"])
+ self.assertEqual("off", rec["tier"])
+ self.assertNotIn("grok", [b["name"] for b in rec["backends"]])
+ blob = json.dumps(rec)
+ x_text = "\n".join(_x_text_lines(doctor.render_text(report)))
+ for surface in (blob, x_text):
+ # "Grok Bot settings" (the connector copy) and "xAI/Grok live
+ # search" (the licensed xai product) are official vocabulary;
+ # the CLI, its login, its store, and the pin knob must be absent.
+ lowered = surface.lower()
+ for word in ("grok cli", "grok login", "~/.grok", "grok binary", "grok --"):
+ self.assertNotIn(word, lowered, word)
+ self.assertNotIn("LAST30DAYS_X_BACKEND", surface)
+ self._assert_official_vocabulary(report)
+
+ def test_bearer_predicts_xapi_with_week_caveat_and_no_network(self):
+ secret = "dummy-x-bearer-secret-000"
+ config = _grok_bot(X_BEARER_TOKEN=secret, AUTH_TOKEN="dummy-auth-token-secret-000",
+ CT0="dummy-ct0-secret-000")
+ patches = [
+ mock.patch("lib.http.get", side_effect=AssertionError("doctor made a network call")),
+ mock.patch("subprocess.run", side_effect=AssertionError("doctor spawned a subprocess")),
+ mock.patch("subprocess.Popen", side_effect=AssertionError("doctor spawned a subprocess")),
+ ]
+ report = _build_with(config, patches)
+ rec = report["sources"]["x"]
+ self.assertEqual("ok", rec["status"])
+ self.assertEqual("xapi", rec["active_backend"])
+ self.assertEqual(f"will use: xapi ({BEARER_CAVEAT})", rec["note"])
+ text = doctor.render_text(report)
+ self.assertIn(f"will use: xapi ({BEARER_CAVEAT})", text)
+ self.assertNotIn(secret, text)
+ self.assertNotIn(secret, doctor.render_json(report))
+ backup = rec["backups"][0]
+ self.assertEqual("X auth path", backup["name"])
+ self.assertTrue(backup["armed"])
+ self.assertIn("X_BEARER_TOKEN", backup["note"])
+ self.assertIn("about the last week", backup["note"])
+ self.assertTrue(report["setup"]["keys_present"]["X_BEARER_TOKEN"])
+ self._assert_official_vocabulary(report)
+
+ def test_xai_key_predicts_xai(self):
+ report = _build(_grok_bot(XAI_API_KEY="dummy-xai-secret-000"))
+ rec = report["sources"]["x"]
+ self.assertEqual("xai", rec["active_backend"])
+ self.assertTrue(rec["note"].startswith("will use: xai"))
+ self.assertIn("XAI_API_KEY", rec["backups"][0]["note"])
+ self._assert_official_vocabulary(report)
+
+ def test_lane_signal_predicts_connector(self):
+ report = _build(_grok_bot(LAST30DAYS_X_HOST_LANE="1"))
+ rec = report["sources"]["x"]
+ self.assertEqual("ok", rec["status"])
+ self.assertEqual("ok", rec["tier"])
+ self.assertEqual("connector", rec["active_backend"])
+ self.assertEqual("will use: X connector (host-fetched at run time)", rec["note"])
+ self.assertEqual("", rec["fix"])
+ backup = rec["backups"][0]
+ self.assertTrue(backup["armed"])
+ self.assertEqual("X connector lane armed", backup["note"])
+ text = "\n".join(_x_text_lines(doctor.render_text(report)))
+ self.assertIn("will use: X connector (host-fetched at run time)", text)
+ self.assertIn("X connector lane armed", text)
+ self._assert_official_vocabulary(report)
+
+ def test_lane_with_bearer_keeps_backend_prediction_and_lane_armed(self):
+ report = _build(_grok_bot(LAST30DAYS_X_HOST_LANE="1", X_BEARER_TOKEN="dummy-x-bearer-secret-000"))
+ rec = report["sources"]["x"]
+ self.assertEqual("xapi", rec["active_backend"])
+ self.assertEqual("X connector lane armed", rec["backups"][0]["note"])
+
+ def test_bird_pin_names_bird_once_as_pinned(self):
+ config = _grok_bot(
+ LAST30DAYS_X_BACKEND="bird",
+ AUTH_TOKEN="dummy-auth-token-secret-000",
+ CT0="dummy-ct0-secret-000",
+ )
+ report = _build_with(
+ config,
+ [mock.patch("lib.bird_x.is_bird_installed", return_value=True)],
+ probe_map={"node": health.OK},
+ )
+ rec = report["sources"]["x"]
+ self.assertEqual("bird", rec["active_backend"])
+ self.assertEqual("will use: bird (pinned)", rec["note"])
+ x_text = "\n".join(_x_text_lines(doctor.render_text(report))).lower()
+ self.assertEqual(1, x_text.count("bird"), x_text)
+ for word in GROK_BOT_FORBIDDEN:
+ if word != "bird":
+ self.assertNotIn(word, x_text, word)
+ self.assertNotIn("LAST30DAYS_X_BACKEND", x_text.upper())
+
+ def test_grok_pin_names_grok_once_as_pinned(self):
+ report = _build_with(_grok_bot(LAST30DAYS_X_BACKEND="grok"), _grok_signed_in())
+ rec = report["sources"]["x"]
+ self.assertEqual("grok", rec["active_backend"])
+ self.assertEqual("will use: grok (pinned)", rec["note"])
+ x_text = "\n".join(_x_text_lines(doctor.render_text(report))).lower()
+ self.assertEqual(1, x_text.count("grok"), x_text)
+ for word in GROK_BOT_FORBIDDEN:
+ self.assertNotIn(word, x_text, word)
+ self.assertNotIn("LAST30DAYS_X_BACKEND", x_text.upper())
+
+ def test_bird_pin_without_cookies_prescribes_only_the_official_path(self):
+ report = _build(_grok_bot(LAST30DAYS_X_BACKEND="bird"))
+ rec = report["sources"]["x"]
+ self.assertIsNone(rec["active_backend"])
+ self.assertIn("X_BEARER_TOKEN", rec["fix"])
+ self.assertNotIn("cookie", rec["fix"].lower())
+ self.assertNotIn("--allow-browser-cookies", rec["fix"])
+
+ def test_host_line_prints_resolved_host(self):
+ text = doctor.render_text(_build(_grok_bot()))
+ self.assertIn("host: grok-bot", text)
+ default = _build({})
+ self.assertIsNone(default["config"]["host"])
+ self.assertIn("host: not set", doctor.render_text(default))
+
+ def test_lane_file_line_is_reported_as_ignored(self):
+ report = _build(_grok_bot(_X_HOST_LANE_FILE_IGNORED=True))
+ text = doctor.render_text(report)
+ self.assertIn("LAST30DAYS_X_HOST_LANE", text)
+ self.assertIn("ignored", text)
+ self.assertEqual("unconfigured", report["sources"]["x"]["status"])
+ self.assertNotIn("LAST30DAYS_X_HOST_LANE", doctor.render_text(_build(_grok_bot())))
+
+ def test_non_grok_host_prescription_unchanged(self):
+ rec = _build({})["sources"]["x"]
+ self.assertIs(prescriptions.get("x", "cookies_missing"), prescriptions.for_x({}, "cookies_missing"))
+ self.assertIn("cookie", rec["fix"].lower())
+ self.assertEqual("LAST30DAYS_X_BACKEND", rec["pin_var"])
+ self.assertIn("no auth path armed", rec["backups"][0]["note"])
+
+ def test_bearer_without_pin_on_default_host_names_the_xapi_pin(self):
+ """xapi is opt-in off Grok Bot: a bare bearer is an unconfigured X
+ with a one-line enable, never a broken X with a cookie fix."""
+ rec = _build({"X_BEARER_TOKEN": "dummy-x-bearer-secret-000"})["sources"]["x"]
+ self.assertEqual("unconfigured", rec["status"])
+ self.assertEqual("off", rec["tier"])
+ self.assertIsNone(rec["active_backend"])
+ self.assertIn("LAST30DAYS_X_BACKEND=xapi", rec["note"])
+ self.assertEqual("", rec["fix"])
+ self.assertNotIn("dummy-x-bearer", json.dumps(rec))
+
+ def test_xapi_pin_on_default_host_carries_the_caveat(self):
+ rec = _build({"LAST30DAYS_X_BACKEND": "xapi", "X_BEARER_TOKEN": "dummy-x-bearer-secret-000"})["sources"]["x"]
+ self.assertEqual("xapi", rec["active_backend"])
+ self.assertIn("about the last week", rec["note"])
+
+
+class HostFingerprintAndCache(unittest.TestCase):
+ """The host key, lane signal, and bearer presence all invalidate a cached
+ report; the bearer value never lands in the cache file."""
+
+ def test_host_and_lane_keys_change_the_fingerprint(self):
+ base = doctor._config_fingerprint({})
+ self.assertNotEqual(base, doctor._config_fingerprint({"LAST30DAYS_HOST": "grok-bot"}))
+ self.assertNotEqual(base, doctor._config_fingerprint({"LAST30DAYS_X_HOST_LANE": "1"}))
+ self.assertNotEqual(base, doctor._config_fingerprint({"X_BEARER_TOKEN": "dummy-x-bearer-secret-000"}))
+ self.assertIn("X_BEARER_TOKEN", doctor.KEY_PRESENCE_VARS)
+ self.assertIn("X_BEARER_TOKEN", doctor._SECRET_CONFIG_VARS)
+
+ def test_cache_file_never_contains_the_bearer(self):
+ secret = "dummy-x-bearer-secret-000"
+ config = _grok_bot(X_BEARER_TOKEN=secret)
+ tmp = Path(tempfile.mkdtemp()) / "doctor-cache.json"
+ with _Hermetic(), mock.patch("lib.doctor.cache_path", return_value=tmp):
+ report = doctor.build_report(dict(config))
+ report["generated_at"] = datetime.datetime.now(datetime.timezone.utc).isoformat()
+ report["from_cache"] = False
+ self.assertTrue(doctor._write_cache(report, config))
+ served = doctor.read_cached_report(config)
+ self.assertNotIn(secret, tmp.read_text(encoding="utf-8"))
+ self.assertIsNotNone(served)
+ self.assertEqual("xapi", served["sources"]["x"]["active_backend"])
+ # A bearer that was tampered into a fake report is refused.
+ report["sources"]["x"]["detail"] = secret
+ with mock.patch("lib.doctor.cache_path", return_value=tmp):
+ self.assertFalse(doctor._write_cache(report, config))
+
+
+class GrokBotParityTable(unittest.TestCase):
+ """host x bearer x lane x pin: doctor's prediction equals the pipeline's
+ pre-failover selection, and Grok Bot output carries no R4 vocabulary."""
+
+ def test_doctor_prediction_matches_pipeline_selection(self):
+ cases = itertools.product(
+ ("", "grok-bot"), (False, True), (False, True), (None, "bird", "grok", "xapi")
+ )
+ for host, bearer, lane, pin in cases:
+ config = {}
+ if host:
+ config["LAST30DAYS_HOST"] = host
+ if bearer:
+ config["X_BEARER_TOKEN"] = "dummy-x-bearer-secret-000"
+ if lane:
+ config["LAST30DAYS_X_HOST_LANE"] = "1"
+ if pin:
+ config["LAST30DAYS_X_BACKEND"] = pin
+ with self.subTest(host=host or "default", bearer=bearer, lane=lane, pin=pin):
+ patches = _grok_signed_in()
+ with _Hermetic():
+ for p in patches:
+ p.start()
+ try:
+ chain = env.x_backend_chain(dict(config), local_only=True)
+ report = doctor.build_report(dict(config))
+ finally:
+ for p in reversed(patches):
+ p.stop()
+ expected = chain[0] if chain else ("connector" if lane else None)
+ rec = report["sources"]["x"]
+ self.assertEqual(expected, rec["active_backend"])
+ if host != "grok-bot":
+ continue
+ x_text = "\n".join(_x_text_lines(doctor.render_text(report))).lower()
+ blob = json.dumps(rec).lower()
+ allow = {pin} if pin in ("bird", "grok") else set()
+ for word in GROK_BOT_FORBIDDEN:
+ if word in allow:
+ continue
+ self.assertNotIn(word, x_text, word)
+ if not allow:
+ self.assertNotIn(word, blob, word)
+ if allow and expected == pin:
+ self.assertLessEqual(x_text.count(pin), 1, x_text)
+ self.assertNotIn("last30days_x_backend", x_text)
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_env_doc_contract.py b/tests/test_env_doc_contract.py
index 9d1ea67e..bf0d3f5b 100644
--- a/tests/test_env_doc_contract.py
+++ b/tests/test_env_doc_contract.py
@@ -11,6 +11,7 @@ ROOT = Path(__file__).resolve().parents[1]
DOC_PATHS = [
ROOT / "skills" / "last30days" / "SKILL.md",
ROOT / "README.md",
+ ROOT / "CONFIGURATION.md",
]
CONFIG_ENV_KEY_RE = re.compile(
r"(?<![A-Z0-9_])(?:"
@@ -19,7 +20,7 @@ CONFIG_ENV_KEY_RE = re.compile(
r"BSKY|TRUTHSOCIAL|BRAVE|EXA|SERPER|OPENROUTER|PERPLEXITY|PARALLEL|"
r"XQUIK|GROQ)_[A-Z0-9_]+|"
r"OPENAI_API_KEY|AUTH_TOKEN|CT0|FROM_BROWSER|INCLUDE_SOURCES|"
- r"EXCLUDE_SOURCES|SETUP_COMPLETE|FUN_LEVEL"
+ r"EXCLUDE_SOURCES|SETUP_COMPLETE|FUN_LEVEL|X_BEARER_TOKEN"
r")(?![A-Z0-9_])"
)
DOC_ONLY_KEYS = {
@@ -28,6 +29,12 @@ DOC_ONLY_KEYS = {
"LAST30DAYS_CACHE_DIR",
"LAST30DAYS_MCP_TIMEOUT",
"LAST30DAYS_PYTHON",
+ # Read from the process environment before or outside get_config
+ # (config-dir override, Keychain/pass source switches), so they are
+ # documented in CONFIGURATION.md without being config keys.
+ "LAST30DAYS_CONFIG_DIR",
+ "LAST30DAYS_PASS_PREFIX",
+ "LAST30DAYS_SKIP_KEYCHAIN",
}
diff --git a/tests/test_env_keychain.py b/tests/test_env_keychain.py
index b1737201..1c6075c6 100644
--- a/tests/test_env_keychain.py
+++ b/tests/test_env_keychain.py
@@ -390,3 +390,10 @@ def test_keychain_keys_match_setup_script():
"lib/env.py::KEYCHAIN_KEYS and scripts/setup-keychain.sh::ALL_KEYS "
f"have drifted.\n python: {python_keys}\n shell: {shell_keys}"
)
+
+
+def test_x_bearer_token_is_a_keychain_key():
+ """U6/R17: the X API bearer is loadable from the Keychain and listed by
+ the setup-keychain.sh helper (parity is enforced above)."""
+ assert "X_BEARER_TOKEN" in env.KEYCHAIN_KEYS
+ assert _parse_all_keys_from_shell(SETUP_KEYCHAIN_SH).count("X_BEARER_TOKEN") == 1
diff --git a/tests/test_env_pass.py b/tests/test_env_pass.py
index ec5c298d..ac406a66 100644
--- a/tests/test_env_pass.py
+++ b/tests/test_env_pass.py
@@ -242,3 +242,10 @@ def test_pass_keys_match_setup_script():
"lib/env.py::KEYCHAIN_KEYS and scripts/setup-pass.sh::ALL_KEYS have "
f"drifted.\n python: {python_keys}\n shell: {shell_keys}"
)
+
+
+def test_x_bearer_token_is_a_pass_key():
+ """U6/R17: the X API bearer is loadable from pass(1) and listed exactly
+ once by the setup-pass.sh helper."""
+ assert "X_BEARER_TOKEN" in env.KEYCHAIN_KEYS
+ assert _parse_all_keys_from_shell(SETUP_PASS_SH).count("X_BEARER_TOKEN") == 1
diff --git a/tests/test_freshness.py b/tests/test_freshness.py
index 480a86e6..e69cc415 100644
--- a/tests/test_freshness.py
+++ b/tests/test_freshness.py
@@ -647,7 +647,7 @@ def test_agent_export_includes_typed_claim_metadata():
exported = schema.to_agent_export(report)
- assert exported["schema_version"] == "1.2"
+ assert exported["schema_version"] == "1.3"
assert exported["freshness_verdicts"][0]["verdict"] == "unsupported"
assert exported["freshness_verdicts"][0]["source_item_id"] == item.item_id
diff --git a/tests/test_grok_bot_host_contract.py b/tests/test_grok_bot_host_contract.py
new file mode 100644
index 00000000..7ad94d06
--- /dev/null
+++ b/tests/test_grok_bot_host_contract.py
@@ -0,0 +1,339 @@
+"""Contract tests for the Grok Bot host slice of SKILL.md (R4, R12, R16; AE9).
+
+On a Grok Bot host the model-facing contract must drive X through the
+official path only: the X connector lane first, the X API bearer or the xAI
+key as backups, keys written only through the engine's ``setup --store-key``
+path, and no browser-session step of any kind. These tests read SKILL.md as
+text - the model's runtime contract - the way tests/test_onboarding_contract.py
+and tests/test_codex_host_contract.py do, and slice the Grok Bot passages so a
+word that is fine elsewhere (the cookie recipes for Linux / Mac mini) cannot
+satisfy or fail an assertion here.
+"""
+
+from __future__ import annotations
+
+import re
+import unittest
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+SKILL_MD = ROOT / "skills" / "last30days" / "SKILL.md"
+CONFIGURATION = ROOT / "CONFIGURATION.md"
+AGENTS_MD = ROOT / "AGENTS.md"
+
+FLOW_HEADING = "### Grok Bot Prose Flow"
+RECIPE_MARKER = "X connector recipe"
+RECIPE_END = "**Step 1: Run the research script"
+
+# R4 vocabulary: none of it may appear in the Grok Bot flow (case-insensitive).
+FORBIDDEN = (
+ "cookie",
+ "box-chrome",
+ "cdp",
+ "auth_token",
+ "ct0",
+ "bird",
+ "18800",
+ "xquik",
+ "grok login",
+ "from_browser",
+ "last30days_x_backend",
+ "askuserquestion",
+)
+
+LANE_TAGS = ("topic", "from", "mention", "related")
+ROW_FIELDS = (
+ "id",
+ "author_handle",
+ "created_at",
+ "text",
+ "likes",
+ "reposts",
+ "replies",
+ "quotes",
+)
+KEY_TOKENS = ("X_BEARER_TOKEN", "XAI_API_KEY", "--x-posts", "envelope", ".json")
+
+HEREDOC_RE = re.compile(r"<<-?\s*([^\s]+)")
+
+
+def _text() -> str:
+ return SKILL_MD.read_text(encoding="utf-8")
+
+
+def _slice_between(text: str, start_marker: str, end_marker: str) -> str:
+ start = text.find(start_marker)
+ assert start != -1, f"missing marker: {start_marker!r}"
+ end = text.find(end_marker, start + len(start_marker))
+ assert end != -1, f"missing end marker after {start_marker!r}: {end_marker!r}"
+ return text[start:end]
+
+
+def _grok_flow(text: str) -> str:
+ """The Grok Bot Prose Flow, from its heading to the next ### heading."""
+ start = text.find(FLOW_HEADING)
+ assert start != -1, f"missing {FLOW_HEADING!r}"
+ match = re.search(r"^### ", text[start + len(FLOW_HEADING):], flags=re.MULTILINE)
+ assert match is not None, "no ### heading follows the Grok Bot Prose Flow"
+ return text[start : start + len(FLOW_HEADING) + match.start()]
+
+
+def _recipe(text: str) -> str:
+ research = text[text.index("## Research Execution") :]
+ return _slice_between(research, RECIPE_MARKER, RECIPE_END)
+
+
+def _guaranteed_band(text: str) -> str:
+ return "\n".join(text.splitlines()[:420])
+
+
+def _extras_passages(text: str) -> dict[str, str]:
+ step0 = _slice_between(text, "## Step 0: First-Run Setup Wizard", "## CRITICAL: Parse User Intent")
+ modal = _slice_between(step0, "### Claude Code Modal Flow", "### Non-Modal Prose Flow")
+ prose = _slice_between(step0, "### Non-Modal Prose Flow", FLOW_HEADING)
+ manual = step0[step0.index("### Manual Setup Guide") :]
+ modal_extras = _slice_between(modal, "**Extras-host X login", "**macOS Full Disk Access")
+ prose_extras = _slice_between(prose, "**Extras hosts", " - On **no**")
+ manual_extras = _slice_between(manual, "**X on Linux / Mac mini (repair).**", "**Reddit (free")
+ return {"modal": modal_extras, "prose": prose_extras, "manual": manual_extras}
+
+
+def _forbidden_hits(slice_text: str) -> list[str]:
+ lowered = slice_text.lower()
+ return [word for word in FORBIDDEN if word in lowered]
+
+
+class TestGrokBotProseFlow(unittest.TestCase):
+ def setUp(self):
+ self.text = _text()
+ self.flow = _grok_flow(self.text)
+
+ def test_flow_is_the_third_step0_branch(self):
+ step0 = _slice_between(
+ self.text, "## Step 0: First-Run Setup Wizard", "## CRITICAL: Parse User Intent"
+ )
+ split = _slice_between(step0, "**Platform split", "### Claude Code Modal Flow")
+ self.assertIn("Grok Bot Prose Flow", split)
+ # Cursor stays a Non-Modal host; it is not routed to the Grok Bot flow.
+ self.assertIn("Cursor", _slice_between(step0, "### Non-Modal Prose Flow", FLOW_HEADING))
+ self.assertNotIn("Cursor", self.flow)
+
+ def test_flow_has_no_r4_vocabulary(self):
+ self.assertEqual([], _forbidden_hits(self.flow))
+
+ def test_flow_names_the_official_contract(self):
+ for token in (
+ "LAST30DAYS_HOST=grok-bot",
+ "LAST30DAYS_X_HOST_LANE=1",
+ "X_BEARER_TOKEN",
+ "XAI_API_KEY",
+ "--x-posts",
+ "search_posts_all",
+ "generated_at",
+ "window-unsupported",
+ "setup --store-key",
+ "about the last week",
+ "SETUP_COMPLETE=true",
+ "X_DECLINED=grok-bot",
+ "--preflight",
+ ):
+ self.assertIn(token, self.flow, token)
+
+ def test_flow_names_the_call_counts_and_lanes(self):
+ self.assertRegex(self.flow, r"10\s*/\s*30\s*/\s*60")
+ self.assertRegex(self.flow, r"\b8\b.*\b5\b.*\b3\b")
+ for lane in LANE_TAGS:
+ self.assertIn(f"`{lane}`", self.flow, lane)
+ for field in ROW_FIELDS:
+ self.assertIn(f"`{field}`", self.flow, field)
+
+ def test_flow_names_the_x_for_grok_bot_plugin(self):
+ """The connector is the marketplace "X for Grok Bot" plugin: the flow
+ names it and keys the lane on its post-search tools, not on one
+ tool name alone (search_posts_all stays as the example)."""
+ self.assertIn('"X for Grok Bot"', self.flow)
+ self.assertIn("search_posts_all", self.flow)
+ rule = _text()[: _text().index("## Step 0")]
+ self.assertIn('"X for Grok Bot"', rule)
+
+ def test_connector_step_precedes_bearer_offer(self):
+ connector = self.flow.index("search_posts_all")
+ bearer = self.flow.index("X_BEARER_TOKEN")
+ self.assertLess(connector, bearer)
+
+ def test_bearer_coverage_caveat_never_implies_parity(self):
+ self.assertIn(
+ "recent posts, about the last week, unless your X developer project has full-archive access",
+ self.flow,
+ )
+ self.assertIn("X developer console", self.flow)
+ self.assertIn("console.x.ai", self.flow)
+
+ def test_key_persistence_only_through_engine_and_masked(self):
+ self.assertIn("setup --store-key", self.flow)
+ self.assertIn("=****", self.flow)
+ self.assertIn("never echo the value back", self.flow)
+ for line in self.flow.splitlines():
+ if not re.search(r"\b(echo|printf)\b", line):
+ continue
+ writes = re.search(r"\b(echo|printf)\b[^\n]*(>>|>|\|)", line)
+ if writes and any(token in line for token in KEY_TOKENS):
+ self.fail(f"a shell write of a key or the envelope: {line.strip()!r}")
+
+ def test_every_heredoc_in_flow_uses_single_quoted_delimiter(self):
+ for delim in HEREDOC_RE.findall(self.flow):
+ self.assertTrue(delim.startswith("'"), f"unquoted heredoc delimiter {delim!r}")
+
+ def test_flow_declines_write_marker_and_has_no_modals(self):
+ self.assertIn("X_DECLINED=grok-bot", self.flow)
+ self.assertNotIn("AskUserQuestion", self.flow)
+
+
+class TestGuaranteedLoadedRule(unittest.TestCase):
+ def setUp(self):
+ self.band = _guaranteed_band(_text())
+
+ def test_rule_lives_in_the_guaranteed_loaded_band(self):
+ self.assertIn("LAST30DAYS_HOST=grok-bot", self.band)
+ self.assertIn("LAST30DAYS_X_HOST_LANE=1", self.band)
+ self.assertIn("never place post text unquoted", self.band)
+ self.assertIn("CURSOR_AGENT", self.band)
+
+ def test_rule_is_not_keyed_on_cursor_agent_alone(self):
+ start = self.band.index("LAST30DAYS_HOST=grok-bot")
+ rule = self.band[max(0, start - 600) : start + 1200]
+ self.assertIn("CURSOR_AGENT", rule)
+ self.assertRegex(rule, r"(?i)not .*CURSOR_AGENT.*alone|CURSOR_AGENT.*alone")
+
+
+class TestConnectorRecipe(unittest.TestCase):
+ def setUp(self):
+ self.recipe = _recipe(_text())
+
+ def test_recipe_precedes_the_engine_command(self):
+ text = _text()
+ research = text[text.index("## Research Execution") :]
+ self.assertLess(research.index(RECIPE_MARKER), research.index(RECIPE_END))
+
+ def test_recipe_names_counts_window_and_status(self):
+ for token in (
+ "search_posts_all",
+ "-is:retweet",
+ "window-unsupported",
+ "partial",
+ "generated_at",
+ "last30days-x-posts/1",
+ "--x-posts",
+ "x_posts",
+ "--competitors-plan",
+ "X via X connector",
+ "LAW 9",
+ "stderr",
+ ):
+ self.assertIn(token, self.recipe, token)
+ self.assertRegex(self.recipe, r"10 .*30 .*60")
+ for lane in LANE_TAGS:
+ self.assertIn(f'"lane": "{lane}"', self.recipe, lane)
+ for field in ROW_FIELDS:
+ self.assertIn(f'"{field}"', self.recipe, field)
+
+ def test_recipe_forbids_raw_tool_output_in_error(self):
+ self.assertIn("never raw tool output", self.recipe)
+ for category in ("credits", "not-connected", "unavailable"):
+ self.assertIn(category, self.recipe)
+
+ def test_recipe_heredocs_are_single_quoted_and_no_echo_writes(self):
+ for delim in HEREDOC_RE.findall(self.recipe):
+ self.assertTrue(delim.startswith("'"), f"unquoted heredoc delimiter {delim!r}")
+ for line in self.recipe.splitlines():
+ if re.search(r"\b(echo|printf)\b[^\n]*(>>|>|\|)", line) and any(
+ token in line for token in KEY_TOKENS
+ ):
+ self.fail(f"a shell write of the envelope: {line.strip()!r}")
+
+ def test_recipe_heredoc_sentinel_is_per_run_and_json_is_one_line(self):
+ """Post text is attacker-controlled: a fixed public sentinel could be
+ echoed by a post to close the heredoc early. The recipe demands a
+ per-run nonce in both sentinel lines and single-line JSON."""
+ delims = HEREDOC_RE.findall(self.recipe)
+ self.assertTrue(any("X_POSTS_EOF_{X_POSTS_NONCE}" in d for d in delims), delims)
+ self.assertNotIn("<<'X_POSTS_EOF'", self.recipe)
+ self.assertIn("\nX_POSTS_EOF_{X_POSTS_NONCE}\n", self.recipe)
+ self.assertIn("ONE line", self.recipe)
+ self.assertIn("random", self.recipe)
+
+ def test_recipe_has_no_r4_vocabulary(self):
+ self.assertEqual([], _forbidden_hits(self.recipe))
+
+
+class TestExtrasPassagesRescoped(unittest.TestCase):
+ def test_extras_passages_no_longer_name_grok_bot(self):
+ for name, passage in _extras_passages(_text()).items():
+ self.assertNotIn("Grok Bot", passage, f"{name} extras passage still names Grok Bot")
+ self.assertIn("grok-bot", passage, f"{name} extras passage does not exclude the grok-bot host")
+
+ def test_manual_repair_heading_rescoped(self):
+ text = _text()
+ self.assertIn("**X on Linux / Mac mini (repair).**", text)
+ self.assertNotIn("X on Linux / Grok Bot / Mac mini", text)
+
+
+class TestManualSetupGuide(unittest.TestCase):
+ def setUp(self):
+ text = _text()
+ step0 = _slice_between(text, "## Step 0: First-Run Setup Wizard", "## CRITICAL: Parse User Intent")
+ self.manual = step0[step0.index("### Manual Setup Guide") :]
+
+ def test_bearer_bullet_comes_first_in_x_section(self):
+ x_section = _slice_between(self.manual, "**X/Twitter (pick one", "**X on Linux / Mac mini (repair).**")
+ bullets = [line for line in x_section.splitlines() if line.startswith("- ")]
+ self.assertTrue(bullets, "no X bullets in the Manual Setup Guide")
+ self.assertIn("X_BEARER_TOKEN", bullets[0])
+ self.assertIn("about a week", bullets[0])
+
+ def test_grok_bot_repair_paragraph_is_official_only(self):
+ para = _slice_between(self.manual, "**X on a Grok Bot (repair).**", "**X on Linux / Mac mini (repair).**")
+ self.assertEqual([], _forbidden_hits(para))
+ for token in ("connect X", "X_BEARER_TOKEN", "about the last week", "XAI_API_KEY", "top up"):
+ self.assertIn(token, para, token)
+
+
+class TestSecurityAndFrontmatter(unittest.TestCase):
+ def test_frontmatter_optional_env_lists_bearer(self):
+ text = _text()
+ frontmatter = text[: text.index("---", 3)]
+ self.assertIn("- X_BEARER_TOKEN", frontmatter)
+
+ def test_security_section_lists_x_api_and_envelope(self):
+ text = _text()
+ security = text[text.index("## Security & Permissions") :]
+ self.assertIn("api.x.com", security)
+ self.assertIn("--x-posts", security)
+ self.assertIn("X connector", security)
+ overview = _slice_between(text, "**Permissions overview:**", "Research ANY topic")
+ self.assertIn("api.x.com", overview)
+
+
+class TestConfigurationGrokBotSubsection(unittest.TestCase):
+ def test_configuration_grok_bot_subsection_has_no_r4_vocabulary(self):
+ text = CONFIGURATION.read_text(encoding="utf-8")
+ match = re.search(r"^(#{2,4}) [^\n]*Grok Bot[^\n]*$", text, flags=re.MULTILINE)
+ if match is None:
+ self.skipTest("CONFIGURATION.md has no Grok Bot subsection yet")
+ level = len(match.group(1))
+ rest = text[match.end() :]
+ nxt = re.search(rf"^#{{1,{level}}} ", rest, flags=re.MULTILINE)
+ section = rest if nxt is None else rest[: nxt.start()]
+ self.assertEqual([], _forbidden_hits(section))
+
+
+class TestAgentsMd(unittest.TestCase):
+ def test_agents_md_names_three_branches(self):
+ text = AGENTS_MD.read_text(encoding="utf-8")
+ self.assertIn("Step 0 has THREE branches", text)
+ self.assertIn("Grok Bot Prose Flow", text)
+ self.assertNotIn("Step 0 has TWO branches", text)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_grok_lanes_pipeline.py b/tests/test_grok_lanes_pipeline.py
index edeb8a01..b3cfc7e7 100644
--- a/tests/test_grok_lanes_pipeline.py
+++ b/tests/test_grok_lanes_pipeline.py
@@ -19,13 +19,26 @@ def _supplements_source():
def test_grok_is_handle_lane_capable():
src = _supplements_source()
- assert '("grok", "bird", "xquik")' in src, (
+ assert '("grok", "bird", "xapi", "xquik")' in src, (
"grok supports from:/@ natively; leaving it out of the capable set "
"silently drops all of Phase 2 for grok users, as it already does for "
- "xai and xurl"
+ "xai and xurl. xapi (X API v2 bearer) runs the same lanes, after "
+ "bird and before xquik (R7)."
)
+def test_xapi_lane_branch_sits_between_bird_and_xquik():
+ src = _supplements_source()
+ grok = src.index('if primary == "grok":')
+ bird = src.index('elif primary == "bird":')
+ xapi = src.index('elif primary == "xapi":')
+ xquik = src.index('elif primary == "xquik":')
+ assert grok < bird < xapi < xquik
+ xapi_block = src[xapi:xquik]
+ assert "x_api.search_handles" in xapi_block
+ assert "x_api.search_mentions" in xapi_block
+
+
def test_all_three_lanes_are_defined_for_grok():
src = _supplements_source()
grok_block = src[src.index('if primary == "grok":'):src.index('elif primary == "bird":')]
diff --git a/tests/test_grok_surfacing.py b/tests/test_grok_surfacing.py
index 6683cb3c..a0a61ac5 100644
--- a/tests/test_grok_surfacing.py
+++ b/tests/test_grok_surfacing.py
@@ -52,11 +52,11 @@ def test_configuration_documents_the_grok_path_as_opt_in():
assert "LAST30DAYS_X_BACKEND=grok" in text
-def test_configuration_pin_row_lists_grok_last():
- """Pin row shows all backends with grok last (opt-in)."""
+def test_configuration_pin_row_lists_opt_ins_last():
+ """Pin row shows all backends with the opt-ins (grok, xapi) last."""
text = (REPO / "CONFIGURATION.md").read_text()
- # New order: bird first, grok last (opt-in).
- assert "`bird` / `xai` / `xurl` / `xquik` / `grok`" in text
+ # Auto-chain order first, then the opt-in backends.
+ assert "`bird` / `xai` / `xurl` / `xquik` / `grok` / `xapi`" in text
def test_configuration_does_not_claim_grok_is_free():
@@ -71,6 +71,19 @@ def test_configuration_documents_bird_first_chain():
assert "bird first" in text.lower() or "bird (browser cookies) → xai" in text.lower()
+def test_configuration_chain_paragraph_keeps_bird_first_and_adds_grok_bot_exception():
+ """Ordinary hosts keep bird first; the Grok Bot exception is the official chain."""
+ text = (REPO / "CONFIGURATION.md").read_text()
+ start = text.index("**X backend priority (bird first).**")
+ paragraph = text[start:text.index("\n", start)]
+ assert "bird (browser cookies) → xai (API key) → xurl (OAuth2 CLI) → xquik (API key)" in paragraph
+ assert "Grok Bot" in paragraph
+ assert "xapi" in paragraph and "xai" in paragraph and "xurl" in paragraph
+ assert "X connector" in paragraph
+ # Grok CLI stays an opt-in backup on every host.
+ assert "leftover grok login never steals the X lane" in paragraph
+
+
def test_changelog_fragments_exist_and_changelog_is_untouched():
frags = list((REPO / "changelog.d").glob("*grok*")) + \
list((REPO / "changelog.d").glob("*bird*"))
@@ -83,6 +96,30 @@ def test_changelog_fragments_exist_and_changelog_is_untouched():
assert frags, "feature PRs add a changelog.d fragment"
+def test_official_x_api_fragments_exist_and_changelog_is_untouched():
+ """U8: two orphan fragments carry the change; CHANGELOG.md is release-owned."""
+ changed = REPO / "changelog.d" / "+grok-bot-official-x-api.changed.md"
+ added = REPO / "changelog.d" / "+x-api-backend-and-host-lane.added.md"
+ changelog = (REPO / "CHANGELOG.md").read_text()
+ if not changed.exists() and not added.exists():
+ # A release consumed the fragments into CHANGELOG.md via towncrier.
+ assert "X connector" in changelog and "xapi" in changelog
+ return
+ assert changed.exists() and added.exists(), "both U8 fragments must ship together"
+ changed_text = changed.read_text().strip()
+ added_text = added.read_text().strip()
+ assert changed_text and added_text
+ # Fragment bodies are not pasted into CHANGELOG.md by hand.
+ assert changed_text.splitlines()[0] not in changelog
+ assert added_text.splitlines()[0] not in changelog
+ for text in (changed_text, added_text):
+ lowered = text.lower()
+ # R20/R21: the fragments never name the legacy backends, cookies,
+ # a pin as an override path, or why the feature exists.
+ for banned in ("bird", "xquik", "grok cli", "cookie", "override", "xai's request", "marketplace"):
+ assert banned not in lowered, f"fragment names {banned!r}"
+
+
# --- SKILL.md unlock surfaces ---------------------------------------------
def _skill_md():
diff --git a/tests/test_health_probe_taxonomy.py b/tests/test_health_probe_taxonomy.py
index b9cfe980..2e66dad8 100644
--- a/tests/test_health_probe_taxonomy.py
+++ b/tests/test_health_probe_taxonomy.py
@@ -324,3 +324,43 @@ class TestWindowsPrintingPressCandidates:
for candidate in candidates:
if candidate.parent != pp_dir:
assert candidate.name == "digg-pp-cli"
+
+
+class TestRunOutcomeTaxonomy:
+ """The per-run outcome vocabulary must stay enumerated in lockstep across
+ ``health`` (constants), ``schema.RunOutcomeState`` (the export Literal),
+ ``SourceOutcome`` validation, and the pipeline's failure-specificity
+ ladder. ``payment-required`` (credit exhaustion, HTTP 402) is its own
+ state, distinct from ``auth-failed``: "top up credits" is a different
+ user action from "re-authenticate".
+ """
+
+ def test_payment_required_constant_and_alias(self):
+ from lib import schema
+
+ assert health.PAYMENT_REQUIRED == "payment-required"
+ assert schema.PAYMENT_REQUIRED == health.PAYMENT_REQUIRED
+
+ def test_literal_and_validation_enumerate_every_run_state(self):
+ import typing
+
+ from lib import schema
+
+ literal_states = set(typing.get_args(schema.RunOutcomeState))
+ assert "payment-required" in literal_states
+ # Every Literal member constructs; the Literal is the export contract.
+ for state in literal_states:
+ schema.SourceOutcome(source="x", state=state)
+ with pytest.raises(ValueError):
+ schema.SourceOutcome(source="x", state="not-a-state")
+
+ def test_payment_required_is_a_failure_state_next_to_auth_failed(self):
+ from lib import pipeline
+
+ ladder = pipeline._FAILURE_SPECIFICITY
+ assert health.PAYMENT_REQUIRED in ladder
+ assert ladder[health.AUTH_FAILED] < ladder[health.PAYMENT_REQUIRED] < ladder[health.RATE_LIMITED]
+
+ def test_credits_exhausted_label_is_source_aware(self):
+ assert health.credits_exhausted_label("x") == "X API credits exhausted"
+ assert health.credits_exhausted_label("reddit") == "credits exhausted"
diff --git a/tests/test_onboarding_contract.py b/tests/test_onboarding_contract.py
index b2870076..56705609 100644
--- a/tests/test_onboarding_contract.py
+++ b/tests/test_onboarding_contract.py
@@ -1,9 +1,10 @@
"""Contract tests for the restored first-run NUX wizard in SKILL.md.
-Step 0 has two branches: a **Claude Code Modal Flow** (AskUserQuestion-driven,
-the restored v3.0.0 NUX) and a **Non-Modal Prose Flow** for hosts without modals
-(OpenClaw, Codex, Cursor, Gemini CLI). These tests assert the structural
-guarantees of both branches, plus the cross-cutting copy rules: the hard
+Step 0 has three branches: a **Claude Code Modal Flow** (AskUserQuestion-driven,
+the restored v3.0.0 NUX), a **Non-Modal Prose Flow** for hosts without modals
+(OpenClaw, Codex, Cursor, Gemini CLI), and a **Grok Bot Prose Flow** (the X
+connector lane first, keys written only through the engine, no browser-session
+reads). These tests assert the structural guarantees of the branches, plus the cross-cutting copy rules: the hard
"Step 0 before Step 1" gate, Digg threaded alongside yt-dlp, the 10,000-free-calls
credit count, and Threads/Pinterest kept out of the onboarding offers. They read
SKILL.md as text - the model's runtime contract - matching
@@ -20,6 +21,7 @@ from lib import setup_wizard
ROOT = Path(__file__).resolve().parents[1]
SKILL_MD = ROOT / "skills" / "last30days" / "SKILL.md"
+AGENTS_MD = ROOT / "AGENTS.md"
class TestOnboardingContract(unittest.TestCase):
@@ -33,18 +35,42 @@ class TestOnboardingContract(unittest.TestCase):
# Branch slices.
modal_start = self.step0.index("### Claude Code Modal Flow")
prose_start = self.step0.index("### Non-Modal Prose Flow")
+ grok_start = self.step0.index("### Grok Bot Prose Flow")
manual_start = self.step0.index("### Manual Setup Guide")
self.modal = self.step0[modal_start:prose_start]
- self.prose = self.step0[prose_start:manual_start]
+ self.prose = self.step0[prose_start:grok_start]
+ self.grok = self.step0[grok_start:manual_start]
self.manual = self.step0[manual_start:]
# --- Platform split + hard gate ---
def test_platform_split_present(self):
- """Step 0 routes modal-capable hosts and prose hosts to distinct flows."""
+ """Step 0 routes modal-capable hosts, prose hosts, and Grok Bot to
+ three distinct flows, in that order."""
self.assertIn("Platform split", self.step0)
self.assertIn("### Claude Code Modal Flow", self.step0)
self.assertIn("### Non-Modal Prose Flow", self.step0)
+ self.assertIn("### Grok Bot Prose Flow", self.step0)
+ split = self.step0[self.step0.index("Platform split"):self.step0.index("### Claude Code Modal Flow")]
+ self.assertIn("Grok Bot Prose Flow", split)
+ self.assertEqual(3, len([h for h in ("### Claude Code Modal Flow", "### Non-Modal Prose Flow", "### Grok Bot Prose Flow") if h in self.step0]))
+ self.assertLess(self.step0.index("### Non-Modal Prose Flow"), self.step0.index("### Grok Bot Prose Flow"))
+
+ def test_agents_md_names_three_step0_branches(self):
+ """AGENTS.md's onboarding rule and this contract move together."""
+ agents = AGENTS_MD.read_text(encoding="utf-8")
+ self.assertIn("Step 0 has THREE branches", agents)
+ self.assertNotIn("Step 0 has TWO branches", agents)
+ for name in ("Claude Code Modal Flow", "Non-Modal Prose Flow", "Grok Bot Prose Flow"):
+ self.assertIn(name, agents, name)
+
+ def test_grok_flow_is_prose_and_connector_first(self):
+ """The third branch has no modals, leads with the X connector, and
+ never routes through a browser-session step."""
+ self.assertNotIn("AskUserQuestion", self.grok)
+ self.assertNotIn("cookie", self.grok.lower())
+ self.assertLess(self.grok.index("search_posts_all"), self.grok.index("X_BEARER_TOKEN"))
+ self.assertIn("setup --store-key", self.grok)
def test_hard_gate_step0_before_step1(self):
"""The erosion-resistant gate that orphaned the wizard in #659 is restored."""
diff --git a/tests/test_permission_preflight.py b/tests/test_permission_preflight.py
index 5d87e414..dda6dc58 100644
--- a/tests/test_permission_preflight.py
+++ b/tests/test_permission_preflight.py
@@ -267,3 +267,31 @@ def test_cli_preflight_json_returns_structured_contract(monkeypatch):
assert payload["local_reads"]["browser_cookies"]["status"] == "off"
assert payload["conditional_writes"] == []
assert payload["safe"] is True
+
+
+DUMMY_X_BEARER = "dummy-x-bearer-token-not-real-000"
+
+
+def test_preflight_reports_x_bearer_presence_as_boolean_without_value():
+ """U6: the X API bearer row is computed from config inside preflight (not
+ through diagnose.providers) and never carries the value."""
+ diag = _diag() # providers dict has no x_bearer entry on purpose
+ present = permission_preflight.build({"X_BEARER_TOKEN": DUMMY_X_BEARER}, diag)
+ absent = permission_preflight.build({}, diag)
+
+ assert present["credentials"]["x_bearer"] == {
+ "present": True,
+ "label": permission_preflight.PROVIDER_CREDENTIALS["x_bearer"],
+ }
+ assert absent["credentials"]["x_bearer"]["present"] is False
+ assert DUMMY_X_BEARER not in json.dumps(present)
+
+ text = permission_preflight.render_text(present)
+ assert permission_preflight.PROVIDER_CREDENTIALS["x_bearer"] in text
+ assert DUMMY_X_BEARER not in text
+ assert "Values are not printed or written by preflight" in text
+
+
+def test_preflight_x_bearer_ignores_whitespace_only_value():
+ diag = _diag()
+ assert permission_preflight.build({"X_BEARER_TOKEN": " "}, diag)["credentials"]["x_bearer"]["present"] is False
diff --git a/tests/test_plugin_contract.py b/tests/test_plugin_contract.py
index b58d7e60..171a7fc5 100644
--- a/tests/test_plugin_contract.py
+++ b/tests/test_plugin_contract.py
@@ -137,5 +137,48 @@ class TestPluginContract(unittest.TestCase):
self.assertNotIn("extensionPath", command)
self.assertNotIn(":-.}", command)
+ def test_mcp_manifest_credential_entries_are_sensitive(self) -> None:
+ # Every user_config entry that names a credential (title ends in _KEY
+ # or _TOKEN, or _PASSWORD) must be marked sensitive so Claude Desktop
+ # stores it in the OS keychain instead of plain config.
+ manifest = _json(ROOT / "mcp" / "manifest.json")
+ user_config = manifest["user_config"]
+ self.assertTrue(user_config)
+ offenders = [
+ key
+ for key, entry in user_config.items()
+ if str(entry.get("title", "")).endswith(("_KEY", "_TOKEN", "_PASSWORD"))
+ and entry.get("sensitive") is not True
+ ]
+ self.assertEqual([], offenders)
+
+ def test_mcp_manifest_declares_x_bearer_token(self) -> None:
+ # U6/R17: the X API bearer is installable through the MCP bundle like
+ # every other credential, wired user_config -> env under the
+ # lowercased-env-var convention mcp/internal/manifest enforces.
+ manifest = _json(ROOT / "mcp" / "manifest.json")
+ entry = manifest["user_config"]["x_bearer_token"]
+ self.assertEqual("X_BEARER_TOKEN", entry["title"])
+ self.assertEqual("string", entry["type"])
+ self.assertIs(True, entry["sensitive"])
+ self.assertIs(False, entry["required"])
+ self.assertIn("X API v2", entry["description"])
+ self.assertIn("bearer", entry["description"].lower())
+ self.assertIn("developer.x.com", entry["description"])
+ env_map = manifest["server"]["mcp_config"]["env"]
+ self.assertEqual("${user_config.x_bearer_token}", env_map["X_BEARER_TOKEN"])
+
+ def test_mcp_manifest_env_and_user_config_cross_reference(self) -> None:
+ # Python-side mirror of mcp/internal/manifest/manifest_test.go so the
+ # invariant is checked by `uv run pytest` too.
+ manifest = _json(ROOT / "mcp" / "manifest.json")
+ env_map = manifest["server"]["mcp_config"]["env"]
+ user_config = manifest["user_config"]
+ for env_name, value in env_map.items():
+ self.assertEqual(f"${{user_config.{env_name.lower()}}}", value)
+ self.assertIn(env_name.lower(), user_config)
+ for key in user_config:
+ self.assertIn(key.upper(), env_map)
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_prescriptions.py b/tests/test_prescriptions.py
index 7dc90d07..6e6ad2cc 100644
--- a/tests/test_prescriptions.py
+++ b/tests/test_prescriptions.py
@@ -42,8 +42,23 @@ SEED_INVENTORY = {
("youtube", "ytdlp_broken"),
("truthsocial", "token_missing"),
("xiaohongshu", "service_unreachable"),
+ # Official X path (Grok Bot host): connector lane, bearer, credits.
+ ("x", "bearer_missing"),
+ ("x", "bearer_invalid"),
+ ("x", "payment_required"),
+ ("x", "connector_missing"),
}
+OFFICIAL_X_FAILURES = (
+ "bearer_missing", "bearer_invalid", "payment_required", "connector_missing",
+)
+# R4 vocabulary the official entries must never carry.
+GROK_BOT_FORBIDDEN = (
+ "cookie", "cdp", "box-chrome", "bird", "auth_token", "ct0", "xquik",
+ "grok login", "grok cli", "x.com",
+)
+GROK_BOT = {"LAST30DAYS_HOST": "grok-bot"}
+
def _configuration_md_slugs():
"""GitHub-style anchor slugs for every CONFIGURATION.md heading."""
@@ -167,6 +182,44 @@ class TestSharedWithQualityNudge:
assert q["nudge_text"] is None
assert q["core_missing"] == []
+ def test_x_error_nudge_on_grok_bot_names_bearer_never_x_login(self):
+ """R4: the nudge after an X error on a Grok Bot host routes through
+ the policy-aware lookup and never says to log into x.com."""
+ config = dict(GROK_BOT, X_BEARER_TOKEN="dummy-bearer")
+ entry = prescriptions.for_x(config, "cookies_expired")
+ assert entry.failure == "bearer_invalid"
+ q = _nudge(
+ config_overrides=config,
+ result_overrides={"x_error": "401 unauthorized"},
+ ytdlp_installed=True,
+ )
+ assert q["core_errored"] == ["x"]
+ assert entry.fix_nl in q["nudge_text"]
+ lowered = q["nudge_text"].lower()
+ for word in GROK_BOT_FORBIDDEN:
+ assert word not in lowered, word
+
+ def test_x_credits_nudge_on_grok_bot_says_top_up(self):
+ config = dict(GROK_BOT, X_BEARER_TOKEN="dummy-bearer")
+ entry = prescriptions.for_x(config, "payment_required")
+ q = _nudge(
+ config_overrides=config,
+ result_overrides={"x_error": "xapi: payment required (X API credits exhausted)"},
+ ytdlp_installed=True,
+ )
+ assert entry.fix_nl in q["nudge_text"]
+ assert "top up" in q["nudge_text"].lower()
+ assert "x.com" not in q["nudge_text"].lower()
+
+ def test_x_error_nudge_off_grok_bot_is_unchanged(self):
+ entry = prescriptions.get("x", "cookies_expired")
+ q = _nudge(
+ config_overrides={"AUTH_TOKEN": "tok123"},
+ result_overrides={"x_error": "xquik: payment required (402)"},
+ ytdlp_installed=True,
+ )
+ assert entry.fix_nl in q["nudge_text"]
+
def test_ytdlp_missing_nudge_uses_registry_cli(self):
entry = prescriptions.get("youtube", "ytdlp_missing")
q = _nudge(config_overrides={"AUTH_TOKEN": "tok123"}, ytdlp_installed=False)
@@ -201,6 +254,57 @@ class TestSharedWithQualityNudge:
assert "yt-dlp: brew" not in source
+# ---------------------------------------------------------------------------
+# Policy-aware X lookup (Grok Bot host names only the official path)
+# ---------------------------------------------------------------------------
+
+class TestPolicyAwareXLookup:
+ def test_default_host_returns_todays_entries(self):
+ assert prescriptions.for_x({}, "cookies_missing") is prescriptions.get("x", "cookies_missing")
+ assert prescriptions.for_x({}, "cookies_expired") is prescriptions.get("x", "cookies_expired")
+ assert prescriptions.for_x({"LAST30DAYS_HOST": "codex"}, "cookies_missing").failure == "cookies_missing"
+
+ def test_grok_bot_maps_cookie_and_grok_failures_to_official_entries(self):
+ assert prescriptions.for_x(GROK_BOT, "cookies_missing").failure == "bearer_missing"
+ assert prescriptions.for_x(GROK_BOT, "cookies_expired").failure == "bearer_invalid"
+ assert prescriptions.for_x(GROK_BOT, "grok_cli_missing").failure == "bearer_missing"
+ assert prescriptions.for_x(GROK_BOT, "grok_not_authenticated").failure == "bearer_missing"
+
+ def test_official_entries_pass_through_on_every_host(self):
+ for failure in OFFICIAL_X_FAILURES:
+ for config in ({}, GROK_BOT):
+ assert prescriptions.for_x(config, failure).failure == failure
+
+ def test_unknown_failure_falls_back_generically_on_grok_bot(self):
+ entry = prescriptions.for_x(GROK_BOT, "flux_capacitor_missing")
+ assert entry.fix_nl == prescriptions.GENERIC_FIX_NL
+
+ def test_official_entries_use_only_official_vocabulary(self):
+ for failure in OFFICIAL_X_FAILURES:
+ entry = prescriptions.get("x", failure)
+ blob = " ".join((entry.cause, entry.fix_nl, entry.fix_cli, *entry.alt_cli)).lower()
+ for word in GROK_BOT_FORBIDDEN:
+ assert word not in blob, (failure, word)
+
+ def test_bearer_copy_carries_the_week_caveat_not_parity(self):
+ for failure in ("bearer_missing", "bearer_invalid", "connector_missing"):
+ nl = prescriptions.get("x", failure).fix_nl
+ assert "X_BEARER_TOKEN" in nl, failure
+ assert "about the last week" in nl, failure
+ assert "full-archive" in nl, failure
+
+ def test_bearer_missing_offers_the_connector_first(self):
+ nl = prescriptions.get("x", "bearer_missing").fix_nl
+ assert nl.index("Grok Bot settings") < nl.index("X_BEARER_TOKEN")
+ assert "30-day" in nl
+ assert "XAI_API_KEY" in nl and "console.x.ai" in nl
+
+ def test_payment_required_says_top_up(self):
+ entry = prescriptions.get("x", "payment_required")
+ assert "top up" in entry.fix_nl.lower()
+ assert "credits" in entry.fix_nl.lower()
+
+
# ---------------------------------------------------------------------------
# Scenario 3: unregistered failure -> generic fallback, no crash
# ---------------------------------------------------------------------------
diff --git a/tests/test_quality_nudge.py b/tests/test_quality_nudge.py
index 1c7c9f3d..5ca2dc4d 100644
--- a/tests/test_quality_nudge.py
+++ b/tests/test_quality_nudge.py
@@ -757,3 +757,65 @@ class TestInstagramSilentFailure:
result_overrides={"instagram_items_count": 0},
)
assert "instagram" in q["bonus_errored"]
+
+
+class TestBearerCredential:
+ """X_BEARER_TOKEN counts as an X credential where the xapi backend can
+ run (a Grok Bot host, or an explicit xapi pin); an ambient bearer on a
+ plain host stays what it is today: not a configured X source."""
+
+ def test_bearer_counts_on_grok_bot(self):
+ q = _compute(
+ config_overrides={"LAST30DAYS_HOST": "grok-bot", "X_BEARER_TOKEN": "dummy-bearer"},
+ ytdlp_installed=True,
+ )
+ assert "x" in q["core_active"]
+ assert q["score_pct"] == 100
+
+ def test_bearer_counts_when_xapi_is_pinned(self):
+ q = _compute(
+ config_overrides={"X_BEARER_TOKEN": "dummy-bearer", "LAST30DAYS_X_BACKEND": "xapi"},
+ ytdlp_installed=True,
+ )
+ assert "x" in q["core_active"]
+
+ def test_ambient_bearer_on_plain_host_is_an_optional_omission(self):
+ q = _compute(config_overrides={"X_BEARER_TOKEN": "dummy-bearer"}, ytdlp_installed=True)
+ assert "x" not in q["core_active"]
+ assert q["core_missing"] == []
+ assert q["nudge_text"] is None
+
+ def test_grok_bot_x_error_nudge_names_bearer_never_x_login(self):
+ q = _compute(
+ config_overrides={"LAST30DAYS_HOST": "grok-bot", "X_BEARER_TOKEN": "dummy-bearer"},
+ result_overrides={"x_error": "401 unauthorized"},
+ ytdlp_installed=True,
+ )
+ assert q["core_errored"] == ["x"]
+ text = q["nudge_text"].lower()
+ assert "x_bearer_token" in text
+ for word in ("x.com", "cookie", "bird", "auth_token", "ct0", "xquik", "grok cli", "grok login"):
+ assert word not in text, word
+
+ def test_declared_lane_without_envelope_prescribes_the_connector(self):
+ from lib import x_envelope
+ q = _compute(
+ config_overrides={"LAST30DAYS_HOST": "grok-bot"},
+ result_overrides={"x_error": x_envelope.DETAIL_NOT_PASSED, "active_sources": ["x"]},
+ ytdlp_installed=True,
+ )
+ assert q["core_errored"] == ["x"]
+ text = q["nudge_text"].lower()
+ assert "connector" in text
+ assert "--x-posts" in text
+ for word in ("x.com", "cookie", "bird", "auth_token", "ct0"):
+ assert word not in text, word
+
+ def test_grok_bot_credit_error_nudge_says_top_up(self):
+ q = _compute(
+ config_overrides={"LAST30DAYS_HOST": "grok-bot", "X_BEARER_TOKEN": "dummy-bearer"},
+ result_overrides={"x_error": "xapi: payment required (X API credits exhausted)"},
+ ytdlp_installed=True,
+ )
+ assert "top up" in q["nudge_text"].lower()
+ assert "x.com" not in q["nudge_text"].lower()
diff --git a/tests/test_render_comparison_multi.py b/tests/test_render_comparison_multi.py
index 26fe1478..8783abb7 100644
--- a/tests/test_render_comparison_multi.py
+++ b/tests/test_render_comparison_multi.py
@@ -274,7 +274,7 @@ class EmitComparisonOutputTests(unittest.TestCase):
self.assertEqual(payload["entities"], ["OpenAI", "Anthropic"])
self.assertEqual(len(payload["reports"]), 2)
self.assertEqual(payload["reports"][0]["entity"], "OpenAI")
- self.assertEqual(payload["schema_version"], "1.2")
+ self.assertEqual(payload["schema_version"], "1.3")
self.assertIn("query", payload["reports"][0]["report"])
def test_compact_and_md_both_route_to_multi(self):
diff --git a/tests/test_security_boundaries.py b/tests/test_security_boundaries.py
index 05325ea0..cf4cdf23 100644
--- a/tests/test_security_boundaries.py
+++ b/tests/test_security_boundaries.py
@@ -43,6 +43,7 @@ def test_diagnose_uses_plan_only_cookie_policy_and_safe_pipeline(monkeypatch):
{"_BROWSER_COOKIE_MODE": "plan_only", "_BROWSER_COOKIE_BROWSERS": ["firefox"]},
None,
safe=True,
+ x_envelope=False,
)
assert json.loads(stdout.getvalue()) == {"ok": True}
diff --git a/tests/test_setup_store_key.py b/tests/test_setup_store_key.py
new file mode 100644
index 00000000..58723ced
--- /dev/null
+++ b/tests/test_setup_store_key.py
@@ -0,0 +1,196 @@
+"""`setup --store-key <NAME>` (U6): persist one allowlisted credential from stdin.
+
+This is the only way the Grok Bot flow persists X_BEARER_TOKEN: the model
+pipes the value on stdin, the engine appends it to the global .env as a 0o600
+secret through setup_wizard.write_api_key, and stdout carries only the masked
+name plus a JSON status line. The value never reaches stdout or stderr; a name
+outside env.KEYCHAIN_KEYS or an empty value exits 2 without echoing anything.
+"""
+
+from __future__ import annotations
+
+import io
+import json
+import os
+import stat
+import sys
+from contextlib import redirect_stderr, redirect_stdout
+from pathlib import Path
+from unittest import mock
+
+import pytest
+
+import last30days as cli
+from lib import env
+
+DUMMY = "dummy-x-bearer-token-not-real-0123456789"
+
+
+def _run(argv: list[str], stdin_text: str, env_path: Path) -> tuple[int, str, str]:
+ stdout, stderr = io.StringIO(), io.StringIO()
+ with (
+ mock.patch.object(cli.env, "CONFIG_FILE", env_path),
+ mock.patch.object(cli.env, "get_config", side_effect=AssertionError("store-key must not load config")),
+ mock.patch.object(sys, "stdin", io.StringIO(stdin_text)),
+ mock.patch.object(sys, "argv", ["last30days.py"] + argv),
+ ):
+ with redirect_stdout(stdout), redirect_stderr(stderr):
+ try:
+ rc = cli.main()
+ except SystemExit as exc: # argparse parser.error path
+ rc = int(exc.code or 0)
+ return rc, stdout.getvalue(), stderr.getvalue()
+
+
+def _mode(path: Path) -> int:
+ return stat.S_IMODE(path.stat().st_mode)
+
+
+ ["setup", "--store-key", "X_BEARER_TOKEN"],
+ ["setup", "--store-key=X_BEARER_TOKEN"],
+])
+def test_store_key_writes_allowlisted_key_at_0600_and_masks_stdout(tmp_path, argv):
+ env_path = tmp_path / "cfg" / ".env"
+ rc, out, err = _run(argv, DUMMY + "\n", env_path)
+
+ assert rc == 0, (out, err)
+ assert env_path.exists()
+ assert _mode(env_path) == 0o600
+ assert env.load_env_file(env_path)["X_BEARER_TOKEN"] == DUMMY
+
+ lines = out.strip().splitlines()
+ assert lines[0] == "X_BEARER_TOKEN=****"
+ assert json.loads(lines[-1]) == {"persisted": True, "key": "X_BEARER_TOKEN"}
+ assert DUMMY not in out
+ assert DUMMY not in err
+
+
+def test_store_key_strips_surrounding_whitespace(tmp_path):
+ env_path = tmp_path / ".env"
+ rc, out, err = _run(["setup", "--store-key", "X_BEARER_TOKEN"], f" {DUMMY}\t\r\n", env_path)
+ assert rc == 0
+ assert env.load_env_file(env_path)["X_BEARER_TOKEN"] == DUMMY
+ assert DUMMY not in out and DUMMY not in err
+
+
+def test_store_key_reads_exactly_one_line(tmp_path):
+ env_path = tmp_path / ".env"
+ rc, out, err = _run(
+ ["setup", "--store-key", "X_BEARER_TOKEN"],
+ DUMMY + "\nsecond-line-must-be-ignored\n",
+ env_path,
+ )
+ assert rc == 0
+ assert env.load_env_file(env_path)["X_BEARER_TOKEN"] == DUMMY
+ assert "second-line" not in env_path.read_text()
+
+
+def test_store_key_replaces_an_existing_value_in_place(tmp_path):
+ """A second store-key rotates the credential: a rejected token must not
+ survive a 'persisted: true' receipt (review finding)."""
+ env_path = tmp_path / ".env"
+ env_path.write_text("SETUP_COMPLETE=true\n# note\nX_BEARER_TOKEN=old-dummy\nXAI_API_KEY=other\n")
+ os.chmod(env_path, 0o600)
+ rotated = "rotated-dummy-value-not-real"
+ rc, out, err = _run(["setup", "--store-key", "X_BEARER_TOKEN"], rotated + "\n", env_path)
+ assert rc == 0
+ assert json.loads(out.strip().splitlines()[-1]) == {"persisted": True, "key": "X_BEARER_TOKEN"}
+ content = env_path.read_text()
+ assert content.count("X_BEARER_TOKEN=") == 1
+ assert "old-dummy" not in content
+ loaded = env.load_env_file(env_path)
+ assert loaded["X_BEARER_TOKEN"] == rotated
+ assert loaded["XAI_API_KEY"] == "other"
+ assert loaded["SETUP_COMPLETE"] == "true"
+ assert "# note" in content
+ assert _mode(env_path) == 0o600
+ assert not (tmp_path / ".env.tmp").exists()
+ assert rotated not in out and rotated not in err
+
+
+def test_write_api_key_default_still_keeps_an_existing_value(tmp_path):
+ from lib import setup_wizard
+ env_path = tmp_path / ".env"
+ assert setup_wizard.write_api_key(env_path, DUMMY, key_name="X_BEARER_TOKEN")
+ assert setup_wizard.write_api_key(env_path, "other-dummy", key_name="X_BEARER_TOKEN")
+ assert env.load_env_file(env_path)["X_BEARER_TOKEN"] == DUMMY
+
+
+def test_store_key_reads_a_bounded_line(tmp_path):
+ env_path = tmp_path / ".env"
+ huge = "a" * (cli.STORE_KEY_MAX_BYTES * 2)
+ rc, out, err = _run(["setup", "--store-key", "X_BEARER_TOKEN"], huge + "\n", env_path)
+ assert rc == 0
+ assert len(env.load_env_file(env_path)["X_BEARER_TOKEN"]) == cli.STORE_KEY_MAX_BYTES
+
+
+def test_store_key_tightens_a_loose_existing_file(tmp_path):
+ env_path = tmp_path / ".env"
+ env_path.write_text("SETUP_COMPLETE=true\n")
+ os.chmod(env_path, 0o644)
+ rc, _, _ = _run(["setup", "--store-key", "X_BEARER_TOKEN"], DUMMY + "\n", env_path)
+ assert rc == 0
+ assert _mode(env_path) == 0o600
+ loaded = env.load_env_file(env_path)
+ assert loaded["SETUP_COMPLETE"] == "true"
+ assert loaded["X_BEARER_TOKEN"] == DUMMY
+
+
[email protected]("name", ["NOT_A_REAL_KEY", "PATH", "x_bearer_token", "X_BEARER_TOKEN=evil"])
+def test_store_key_rejects_name_outside_allowlist_with_exit_2(tmp_path, name):
+ env_path = tmp_path / ".env"
+ rc, out, err = _run(["setup", "--store-key", name], DUMMY + "\n", env_path)
+ assert rc == 2
+ assert not env_path.exists()
+ assert DUMMY not in out and DUMMY not in err
+ assert "store-key" in err
+ # R4: the failure hint must not enumerate legacy credential names.
+ for banned in ("AUTH_TOKEN", "CT0", "XQUIK_API_KEY"):
+ assert banned not in err
+
+
+def test_store_key_without_name_exits_2(tmp_path):
+ env_path = tmp_path / ".env"
+ rc, out, err = _run(["setup", "--store-key"], DUMMY + "\n", env_path)
+ assert rc == 2
+ assert not env_path.exists()
+ assert DUMMY not in out and DUMMY not in err
+
+
[email protected]("stdin_text", ["", "\n", " \n"])
+def test_store_key_rejects_empty_value_with_exit_2(tmp_path, stdin_text):
+ env_path = tmp_path / ".env"
+ rc, out, err = _run(["setup", "--store-key", "X_BEARER_TOKEN"], stdin_text, env_path)
+ assert rc == 2
+ assert not env_path.exists()
+ assert "empty" in err.lower()
+ assert "X_BEARER_TOKEN" in err
+
+
+def test_store_key_works_for_every_allowlisted_name(tmp_path):
+ """The allowlist is env.KEYCHAIN_KEYS, not a bearer-only special case."""
+ env_path = tmp_path / ".env"
+ for name in env.KEYCHAIN_KEYS:
+ rc, out, _ = _run(["setup", "--store-key", name], f"dummy-{name.lower()}-not-real\n", env_path)
+ assert rc == 0, name
+ assert out.strip().splitlines()[0] == f"{name}=****"
+ loaded = env.load_env_file(env_path)
+ assert set(env.KEYCHAIN_KEYS) <= set(loaded)
+ assert _mode(env_path) == 0o600
+
+
+def test_store_key_is_a_declared_setup_passthrough_flag():
+ assert "--store-key" in cli.SETUP_PASSTHROUGH_FLAGS
+
+
+def test_store_key_reports_persist_failure_as_false(tmp_path):
+ env_path = tmp_path / ".env"
+ with mock.patch("lib.setup_wizard.write_api_key", return_value=False) as w:
+ rc, out, err = _run(["setup", "--store-key", "X_BEARER_TOKEN"], DUMMY + "\n", env_path)
+ assert rc == 1
+ w.assert_called_once_with(env_path, DUMMY, key_name="X_BEARER_TOKEN", replace=True)
+ lines = out.strip().splitlines()
+ assert lines[0] == "X_BEARER_TOKEN=****"
+ assert json.loads(lines[-1]) == {"persisted": False, "key": "X_BEARER_TOKEN"}
+ assert DUMMY not in out and DUMMY not in err
diff --git a/tests/test_setup_wizard.py b/tests/test_setup_wizard.py
index 398e8601..4fb285d1 100644
--- a/tests/test_setup_wizard.py
+++ b/tests/test_setup_wizard.py
@@ -159,6 +159,62 @@ class TestRunAutoSetup:
assert tried[0] == "chrome"
+ @patch("lib.cookie_extract.extract_cookies_with_source")
+ @patch("shutil.which")
+ def test_official_only_host_skips_cookie_loop_for_every_domain(self, mock_which, mock_extract):
+ """U6/R5: on an official-only host (LAST30DAYS_HOST=grok-bot) the
+ consented cookie scan is skipped for X AND Truth Social, the result
+ records a neutral note, and the CLI installs (yt-dlp) still run."""
+ mock_extract.return_value = ({"auth_token": "abc", "ct0": "xyz"}, "chrome")
+ mock_which.return_value = "/usr/local/bin/yt-dlp"
+
+ config = {"LAST30DAYS_HOST": "grok-bot"}
+ results = setup_wizard.run_auto_setup(config, allow_browser_cookies=True)
+
+ mock_extract.assert_not_called()
+ assert results["cookies_found"] == {}
+ assert results["browser_cookie_scan_attempted"] is False
+ assert results["cookie_note"] == "browser sessions are not read on this host"
+ # The free CLI installs still run: the yt-dlp probe happened.
+ assert any(call.args[:1] == ("yt-dlp",) for call in mock_which.call_args_list)
+ assert results["ytdlp_installed"] is True
+ assert results["ytdlp_action"] == "already_installed"
+
+ @patch("lib.cookie_extract.extract_cookies_with_source")
+ @patch("shutil.which")
+ def test_official_only_host_status_text_names_no_cookies(self, mock_which, mock_extract):
+ """R4: the setup summary on an official-only host never names cookies."""
+ mock_which.return_value = None
+ results = setup_wizard.run_auto_setup(
+ {"LAST30DAYS_HOST": "grok-bot"}, allow_browser_cookies=True
+ )
+ results["env_written"] = True # what the CLI sets before rendering
+ text = setup_wizard.get_setup_status_text(results)
+ assert setup_wizard.OFFICIAL_HOST_COOKIE_NOTE in text
+ assert "Configuration saved." in text
+ lowered = text.lower()
+ for banned in ("cookies", "cdp", "box-chrome", "bird", "auth_token", "ct0", "xquik", "browsers"):
+ assert banned not in lowered, f"{banned!r} leaked into setup status text: {text!r}"
+
+ def test_status_text_without_note_keeps_browser_tail(self):
+ """R14: non-Grok hosts keep today's status tail byte for byte."""
+ text = setup_wizard.get_setup_status_text({"env_written": True, "ytdlp_action": "already_installed"})
+ assert "Configuration saved. Future runs will auto-detect your browsers." in text
+
+ @patch("lib.cookie_extract.extract_cookies_with_source")
+ @patch("shutil.which")
+ def test_bird_pin_on_official_only_host_re_enables_cookie_scan(self, mock_which, mock_extract):
+ """KTD7: the exclusive LAST30DAYS_X_BACKEND=bird pin is the one path
+ that re-enables discovery on an official-only host."""
+ mock_extract.return_value = None
+ mock_which.return_value = None
+ config = {"LAST30DAYS_HOST": "grok-bot", "LAST30DAYS_X_BACKEND": "bird"}
+ results = setup_wizard.run_auto_setup(config, allow_browser_cookies=True)
+ assert mock_extract.called
+ assert results["browser_cookie_scan_attempted"] is True
+ assert "cookie_note" not in results
+
+
class TestYtdlpAutoInstall:
"""Tests for yt-dlp auto-install via Homebrew in run_auto_setup()."""
diff --git a/tests/test_source_outcomes.py b/tests/test_source_outcomes.py
index 8a78c212..db2a3942 100644
--- a/tests/test_source_outcomes.py
+++ b/tests/test_source_outcomes.py
@@ -58,7 +58,7 @@ def _report(*, source_status=None, items_by_source=None, errors_by_source=None):
[
(http.HTTPError("HTTP 429", status_code=429), schema.RATE_LIMITED),
(http.HTTPError("HTTP 401", status_code=401), schema.AUTH_FAILED),
- (http.HTTPError("HTTP 402", status_code=402), schema.AUTH_FAILED),
+ (http.HTTPError("HTTP 402", status_code=402), schema.PAYMENT_REQUIRED),
(http.HTTPError("HTTP 403", status_code=403), schema.AUTH_FAILED),
(http.HTTPError("Invalid JSON response"), schema.SCHEMA_DRIFT),
(http.HTTPError("Connection error: reset"), schema.UNREACHABLE),
@@ -524,6 +524,17 @@ def test_strict_exit_returns_3_for_degraded_run(capsys):
assert "strict-exit: degraded sources: x" in capsys.readouterr().err
+def test_strict_exit_treats_payment_required_as_degraded(capsys):
+ report = _report(
+ source_status={
+ "x": _outcome("x", schema.PAYMENT_REQUIRED, detail="xapi: payment required")
+ }
+ )
+ rc = cli._strict_exit_code(report, None, {"LAST30DAYS_STRICT_EXIT": "1"})
+ assert rc == 3
+ assert "strict-exit: degraded sources: x" in capsys.readouterr().err
+
+
def test_strict_exit_clean_states_return_0():
report = _report(
source_status={
@@ -601,3 +612,139 @@ def test_finalize_turns_an_empty_ok_source_with_lane_failures_into_that_failure(
assert outcome.state == schema.RATE_LIMITED
assert outcome.detail == "5 sub-requests rate-limited (HTTP 429)"
assert outcome.items_returned == 0
+
+
+# --- payment-required: credit exhaustion is not an auth failure (KTD6) ---
+
+
+def test_classify_failure_402_status_is_payment_required():
+ assert http.classify_failure(status_code=402) == schema.PAYMENT_REQUIRED
+ # A 402 with a non-JSON / empty body still classifies by status alone.
+ err = http.HTTPError("HTTP 402: Payment Required", 402, "<html>upgrade</html>")
+ assert err.outcome_state == schema.PAYMENT_REQUIRED
+
+
+ "message",
+ [
+ "Your enrolled account does not have any credits",
+ "insufficient credits",
+ "ScrapeCreators: Insufficient credits remaining",
+ "xapi: payment required (X API credits exhausted)",
+ "Xquik key unpaid: payment required (402)",
+ "You are out of credits for this billing period",
+ ],
+)
+def test_classify_failure_credit_exhaustion_markers(message):
+ assert http.classify_failure(message=message) == schema.PAYMENT_REQUIRED
+
+
+def test_classify_failure_bare_word_credits_is_not_a_marker():
+ # The onboarding copy mentions "10,000 free credits"; a message that merely
+ # contains the word must not be branded as credit exhaustion.
+ state = http.classify_failure(message="Sign up for 10,000 free credits")
+ assert state != schema.PAYMENT_REQUIRED
+
+
[email protected]("status", [401, 403])
+def test_classify_failure_401_403_stay_auth_failed(status):
+ assert http.classify_failure(status_code=status) == schema.AUTH_FAILED
+ assert http.classify_failure(message=f"HTTP {status}: nope") == schema.AUTH_FAILED
+
+
+def test_payment_required_wins_over_auth_marker_in_same_message():
+ # A credit-exhaustion message that also carries an auth word is about
+ # money, not identity: the payment branch runs before the 401/403 branch.
+ state = http.classify_failure(
+ status_code=403,
+ message="Forbidden: your enrolled account does not have any credits",
+ )
+ assert state == schema.PAYMENT_REQUIRED
+
+
+def test_cross_source_insufficient_credits_reclassifies_from_error():
+ # ScrapeCreators-backed sources surface credit exhaustion as a plain
+ # legacy error string; the shared classifier now types it.
+ artifact = pipeline._result_outcome_artifact(
+ "tiktok", {"items": [], "error": "ScrapeCreators: insufficient credits"}
+ )
+ assert artifact["_source_outcome"]["state"] == schema.PAYMENT_REQUIRED
+ state, attempted = pipeline._classify_source_failure(
+ http.HTTPError("HTTP 402: insufficient credits", status_code=402)
+ )
+ assert (state, attempted) == (schema.PAYMENT_REQUIRED, True)
+
+
+def test_captured_failure_selection_ranks_payment_required_above_rate_limit():
+ pay = http.HTTPError("HTTP 402: Payment Required", status_code=402)
+ rate = http.HTTPError("HTTP 429: Too Many Requests", status_code=429)
+ for failures in ([pay, rate], [rate, pay]):
+ outcome = pipeline._resolve_stream_outcome("x", None, failures)
+ assert outcome["state"] == schema.PAYMENT_REQUIRED
+
+
+def test_lane_failure_summary_names_credit_exhaustion():
+ text = pipeline._summarize_lane_failures(
+ [http.HTTPError("HTTP 402: Payment Required", status_code=402)]
+ )
+ assert text == "1 sub-request credits exhausted (HTTP 402)"
+
+
+def test_render_summary_labels_payment_required_per_source():
+ x_outcome = schema.SourceOutcome(
+ source="x", state=schema.PAYMENT_REQUIRED, detail="xapi: payment required"
+ )
+ other = schema.SourceOutcome(
+ source="tiktok", state=schema.PAYMENT_REQUIRED, detail="insufficient credits"
+ )
+ x_text = render._format_outcome(x_outcome)
+ assert x_text.startswith("X API credits exhausted")
+ assert "xapi: payment required" in x_text
+ assert render._format_outcome(other).startswith("credits exhausted")
+ # The Partial Coverage note carries the label too.
+ report = _report(source_status={"x": x_outcome, "tiktok": other})
+ note = "\n".join(render._render_source_outcome_note(report))
+ assert "X API credits exhausted" in note
+ assert "credits exhausted" in note
+
+
+def test_postmortem_labels_payment_required():
+ from lib import doctor
+
+ pm = {
+ "engine_version": "test",
+ "mode": "postmortem",
+ "present": True,
+ "topic": "t",
+ "at": "2026-07-10T18:22:03Z",
+ "outcomes": {
+ "x": schema.to_dict(
+ schema.SourceOutcome(
+ source="x",
+ state=schema.PAYMENT_REQUIRED,
+ detail="xapi: payment required (X API credits exhausted)",
+ )
+ ),
+ "tiktok": schema.to_dict(
+ schema.SourceOutcome(
+ source="tiktok",
+ state=schema.PAYMENT_REQUIRED,
+ detail="insufficient credits",
+ )
+ ),
+ },
+ }
+ text = doctor.render_postmortem_text(pm)
+ assert "Failed:" in text
+ assert "x — X API credits exhausted" in text
+ assert "tiktok — credits exhausted" in text
+ assert "auth-failed" not in text
+
+
+def test_json_export_carries_payment_required():
+ report = _report(
+ source_status={"x": _outcome("x", schema.PAYMENT_REQUIRED, detail="402")}
+ )
+ assert schema.to_agent_export(report)["source_status"]["x"] == "payment-required"
+ restored = schema.report_from_dict(schema.to_dict(report))
+ assert restored.source_status["x"].state == schema.PAYMENT_REQUIRED
diff --git a/tests/test_x_api.py b/tests/test_x_api.py
new file mode 100644
index 00000000..f930cc28
--- /dev/null
+++ b/tests/test_x_api.py
@@ -0,0 +1,952 @@
+"""xapi: the direct X API v2 backend and the shared v2 parser (U4).
+
+Every test mocks ``lib.http.get`` (or urlopen for the fixture seam); no test
+makes a live X call. The dummy token below is obvious fake data and must
+never appear in any error string, source_log line, or recorded fixture.
+"""
+
+from __future__ import annotations
+
+import io
+import json
+import threading
+from contextlib import redirect_stderr
+from datetime import datetime, timezone
+from unittest import mock
+from unittest.mock import MagicMock
+
+import pytest
+
+from lib import env, health, http, pipeline, schema, x_api, xurl_x
+
+DUMMY_TOKEN = "dummy-x-bearer-secret-000"
+ACCOUNT_ID = "acct-1234567890"
+FIXED_NOW = datetime(2026, 9, 8, 12, 0, 0, tzinfo=timezone.utc)
+FROM = "2026-08-10"
+TO = "2026-09-08"
+
+
+def _tweet(id_, text, author_id="u1", created_at="2026-08-20T12:00:00Z", metrics=None, **extra):
+ t = {"id": id_, "text": text, "author_id": author_id}
+ if created_at:
+ t["created_at"] = created_at
+ if metrics is not None:
+ t["public_metrics"] = metrics
+ t.update(extra)
+ return t
+
+
+def _v2(tweets, users=None, next_token=None):
+ resp = {"data": tweets, "meta": {"result_count": len(tweets)}}
+ if users:
+ resp["includes"] = {"users": users}
+ if next_token:
+ resp["meta"]["next_token"] = next_token
+ return resp
+
+
+def _users(*pairs):
+ return [{"id": uid, "username": name} for uid, name in pairs]
+
+
+def _http_error(status, body="", message=None):
+ return http.HTTPError(message or f"HTTP {status}: Error", status_code=status, body=body)
+
+
+def fixed_now(monkeypatch):
+ monkeypatch.setattr(x_api, "_utcnow", lambda: FIXED_NOW)
+ return FIXED_NOW
+
+
+def get_mock(monkeypatch):
+ m = MagicMock(name="http.get")
+ monkeypatch.setattr(x_api.http, "get", m)
+ return m
+
+
+def _params(call):
+ return call.kwargs.get("params") or {}
+
+
+def _url(call):
+ return call.args[0] if call.args else call.kwargs["url"]
+
+
+# ---------------------------------------------------------------------------
+# parse_v2_response
+# ---------------------------------------------------------------------------
+
+
+class TestParseV2Response:
+ def test_fixture_with_users_parses_all_fields(self):
+ metrics = {
+ "like_count": 42, "retweet_count": 10, "reply_count": 5,
+ "quote_count": 2, "bookmark_count": 7, "impression_count": 9000,
+ }
+ resp = _v2(
+ [_tweet("1956158892141441450", "Claude Code agents are great", metrics=metrics)],
+ users=_users(("u1", "alice")),
+ )
+ items = x_api.parse_v2_response(resp, "Claude Code", (FROM, TO))
+ assert len(items) == 1
+ item = items[0]
+ assert item["id"] == "XAPI1"
+ assert item["post_id"] == "1956158892141441450"
+ assert item["author_handle"] == "alice"
+ assert item["url"] == "https://x.com/alice/status/1956158892141441450"
+ assert item["date"] == "2026-08-20"
+ assert item["engagement"] == {
+ "likes": 42, "reposts": 10, "replies": 5, "quotes": 2,
+ "bookmarks": 7, "views": 9000,
+ }
+ assert item["why_relevant"] == ""
+ assert item["relevance"] > 0.5
+ assert item["mentioned_handles"] == []
+
+ def test_without_users_keeps_item_with_i_status_url(self):
+ resp = _v2([_tweet("999", "text here", author_id="unknown")])
+ items = x_api.parse_v2_response(resp, "text", (FROM, TO))
+ assert len(items) == 1
+ assert items[0]["url"] == "https://x.com/i/status/999"
+ assert items[0]["author_handle"] == ""
+
+ def test_note_tweet_replaces_truncated_text(self):
+ resp = _v2([_tweet("5", "truncated...", note_tweet={"text": "the full long-form post text"})])
+ items = x_api.parse_v2_response(resp, "", None)
+ assert items[0]["text"] == "the full long-form post text"
+
+ def test_url_is_built_from_id_never_from_response_url_fields(self):
+ resp = _v2(
+ [_tweet("7", "hi", url="https://evil.example/phish",
+ entities={"urls": [{"expanded_url": "https://evil.example/x"}]})],
+ users=_users(("u1", "bob")),
+ )
+ items = x_api.parse_v2_response(resp, "", None)
+ assert items[0]["url"] == "https://x.com/bob/status/7"
+ assert "evil.example" not in json.dumps(items)
+
+ def test_username_outside_handle_grammar_falls_back_to_i_status(self):
+ resp = _v2([_tweet("8", "hi")], users=[{"id": "u1", "username": "bad name/../x"}])
+ items = x_api.parse_v2_response(resp, "", None)
+ assert items[0]["author_handle"] == ""
+ assert items[0]["url"] == "https://x.com/i/status/8"
+
+ def test_leading_mentions_become_mentioned_handles(self):
+ resp = _v2([_tweet("9", "@steipete @alice thanks for this")])
+ items = x_api.parse_v2_response(resp, "", None)
+ assert items[0]["mentioned_handles"] == ["steipete", "alice"]
+
+ def test_engagement_none_without_metrics_and_bookmarks_views_only_when_present(self):
+ resp = _v2([
+ _tweet("1", "a"),
+ _tweet("2", "b", metrics={"like_count": 1, "retweet_count": 0, "reply_count": 0, "quote_count": 0}),
+ ])
+ items = x_api.parse_v2_response(resp, "", None)
+ assert items[0]["engagement"] is None
+ assert items[1]["engagement"] == {"likes": 1, "reposts": 0, "replies": 0, "quotes": 0}
+
+ def test_non_numeric_ids_are_dropped(self):
+ resp = _v2([_tweet("abc", "x"), _tweet("12", "y")])
+ items = x_api.parse_v2_response(resp, "", None)
+ assert [i["post_id"] for i in items] == ["12"]
+
+ def test_window_drops_dated_items_outside_it(self):
+ resp = _v2([
+ _tweet("1", "old", created_at="2026-07-01T00:00:00Z"),
+ _tweet("2", "in", created_at="2026-08-20T00:00:00Z"),
+ ])
+ items = x_api.parse_v2_response(resp, "", (FROM, TO))
+ assert [i["post_id"] for i in items] == ["2"]
+
+ def test_error_response_yields_nothing(self):
+ assert x_api.parse_v2_response({"error": "x"}, "", None) == []
+ assert x_api.parse_v2_response({}, "", None) == []
+
+ def test_xurl_delegates_to_shared_parser_and_depth_config(self):
+ resp = _v2([_tweet("3", "hello")], users=_users(("u1", "alice")))
+ items = xurl_x.parse_x_response(resp, topic="hello")
+ assert items[0]["id"] == "XURL1"
+ assert items[0]["url"] == "https://x.com/alice/status/3"
+ assert items[0]["post_id"] == "3"
+ assert xurl_x.DEPTH_CONFIG is x_api.DEPTH_CONFIG
+ assert x_api.DEPTH_CONFIG == {"quick": 10, "default": 30, "deep": 60}
+
+
+# ---------------------------------------------------------------------------
+# Query compilation (R8a)
+# ---------------------------------------------------------------------------
+
+
+class TestBuildQuery:
+ def test_wraps_core_in_one_quote_pair_and_appends_no_retweets(self):
+ q = x_api.build_query('Claude Code "agents"')
+ assert q == '"Claude Code agents" -is:retweet'
+
+ def test_operator_injection_is_stripped(self):
+ q = x_api.build_query('foo" OR from:attacker since:2015-01-01 "')
+ assert q == '"foo" -is:retweet'
+ assert "from:" not in q and "since:" not in q and " OR " not in q
+ assert q.count('"') == 2
+
+ def test_negation_and_grouping_characters_are_stripped(self):
+ q = x_api.build_query("(Peter Steinberger) -steipete “quoted” [x] {y}")
+ assert q == '"Peter Steinberger quoted x y" -is:retweet'
+
+ def test_600_char_topic_compiles_under_512_with_balanced_quotes(self):
+ topic = " ".join(f"word{i}" for i in range(100))
+ assert len(topic) >= 600
+ q = x_api.build_query(topic)
+ assert len(q) <= x_api.MAX_QUERY_CHARS
+ assert q.count('"') == 2
+ assert q.endswith('" -is:retweet')
+ assert not q.startswith('"word') or q.startswith('"word0 ')
+ # Cut at a token boundary: the last kept token is intact.
+ core = q[1:q.index('" -is:retweet')]
+ assert all(tok.startswith("word") and tok[4:].isdigit() for tok in core.split())
+
+ def test_single_giant_token_is_truncated_not_dropped(self):
+ q = x_api.build_query("a" * 700)
+ assert 0 < len(q) <= x_api.MAX_QUERY_CHARS
+ assert q.count('"') == 2
+
+ def test_only_operators_yields_empty_query(self):
+ assert x_api.build_query("from:attacker OR -is:reply lang:en") == ""
+
+
+# ---------------------------------------------------------------------------
+# Request shape (KTD4)
+# ---------------------------------------------------------------------------
+
+
+class TestRequestShape:
+ def test_full_archive_request_shape(self, get_mock, fixed_now):
+ get_mock.return_value = _v2([_tweet("1", "Claude Code rocks")], users=_users(("u1", "a")))
+ result = x_api.search_x(DUMMY_TOKEN, "Claude Code", FROM, TO, depth="quick")
+ assert "error" not in result
+ assert [i["id"] for i in result["items"]] == ["XAPI1"]
+ call = get_mock.call_args
+ assert _url(call) == "https://api.x.com/2/tweets/search/all"
+ p = _params(call)
+ assert p["query"] == '"Claude Code" -is:retweet'
+ assert p["start_time"] == "2026-08-10T00:00:00Z"
+ # to_date is "today" relative to the fixed clock: end_time keeps a
+ # 30 second safety margin before now.
+ assert p["end_time"] == "2026-09-08T11:59:30Z"
+ assert p["sort_order"] == "recency"
+ assert p["max_results"] == 10
+ assert p["expansions"] == "author_id"
+ assert p["tweet.fields"] == "created_at,public_metrics,note_tweet,entities"
+ assert p["user.fields"] == "username"
+ assert "lang:" not in p["query"]
+ assert call.kwargs["headers"]["Authorization"] == f"Bearer {DUMMY_TOKEN}"
+ assert call.kwargs["timeout"] == 30
+ assert call.kwargs["retries"] == 2
+
+ def test_past_end_date_uses_end_of_day(self, get_mock, fixed_now):
+ get_mock.return_value = _v2([])
+ x_api.search_x(DUMMY_TOKEN, "topic", "2026-08-01", "2026-08-31", depth="quick")
+ assert _params(get_mock.call_args)["end_time"] == "2026-08-31T23:59:59Z"
+
+ def test_max_results_clamped_to_10_and_100(self, get_mock, fixed_now, monkeypatch):
+ get_mock.return_value = _v2([])
+ x_api.search_handles(["steipete"], "t", FROM, TO, count_per=3, token=DUMMY_TOKEN)
+ assert _params(get_mock.call_args)["max_results"] == 10
+ monkeypatch.setitem(x_api.DEPTH_CONFIG, "deep", 150)
+ x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="deep")
+ assert _params(get_mock.call_args)["max_results"] == 100
+
+ def test_pagination_stops_at_depth_count(self, get_mock, fixed_now):
+ page = lambda start, n, nxt: _v2(
+ [_tweet(str(1000 + start + i), f"post {i}") for i in range(n)], next_token=nxt,
+ )
+ get_mock.side_effect = [page(0, 20, "p2"), page(20, 20, "p3"), page(40, 20, "p4")]
+ result = x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="default")
+ assert get_mock.call_count == 2, "30 posts reached after two pages; the third is never fetched"
+ assert len(result["items"]) == 30
+ assert "next_token" not in _params(get_mock.call_args_list[0])
+ assert _params(get_mock.call_args_list[1])["next_token"] == "p2"
+ assert len({i["id"] for i in result["items"]}) == 30
+
+ def test_pagination_stops_at_the_wall_clock_deadline(self, get_mock, fixed_now, monkeypatch):
+ page = lambda start, n, nxt: _v2(
+ [_tweet(str(1000 + start + i), f"post {i}") for i in range(n)], next_token=nxt,
+ )
+ get_mock.side_effect = [page(0, 20, "p2"), page(20, 20, "p3"), page(40, 20, "p4")]
+ clock = iter([0.0, x_api.DEADLINE_SECONDS + 1.0, x_api.DEADLINE_SECONDS + 2.0, x_api.DEADLINE_SECONDS + 3.0])
+ monkeypatch.setattr(x_api.time, "monotonic", lambda: next(clock))
+ result = x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="deep")
+ assert get_mock.call_count == 1, "the first page always runs; the second is skipped past the deadline"
+ assert len(result["items"]) == 20
+ assert "error" not in result
+ assert result["warning"] == x_api.DEADLINE_DETAIL
+
+ def test_deadline_reaches_the_transport_and_keeps_pages_collected_before_it(self, get_mock, fixed_now, monkeypatch):
+ """The lane deadline is the transport's wall deadline (no full 30s
+ timeout plus retries past it), and a deadline hit mid-walk returns
+ the pages already collected."""
+ page = lambda start, n, nxt: _v2(
+ [_tweet(str(1000 + start + i), f"post {i}") for i in range(n)], next_token=nxt,
+ )
+ monkeypatch.setattr(x_api.time, "monotonic", lambda: 100.0)
+ get_mock.side_effect = [page(0, 20, "p2"), http.DeadlineExceeded()]
+ result = x_api.search_handles(["steipete"], "t", FROM, TO, count_per=60, token=DUMMY_TOKEN, deadline=130.0)
+ assert get_mock.call_count == 2
+ for call in get_mock.call_args_list:
+ assert call.kwargs["deadline_monotonic"] == 130.0
+ assert len(result) == 20 and "error" not in result[0]
+ get_mock.reset_mock()
+ # The same stop on the topic search is a warning receipt, never a
+ # healthy-looking complete result.
+ get_mock.side_effect = [page(0, 20, "p2"), http.DeadlineExceeded()]
+ topic = x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="deep")
+ assert len(topic["items"]) == 20
+ assert topic["warning"] == x_api.DEADLINE_DETAIL
+ get_mock.reset_mock()
+ get_mock.side_effect = [http.DeadlineExceeded()]
+ assert x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="quick")["error"] == x_api.ERR_TIMED_OUT
+
+ def test_pagination_stops_without_next_token(self, get_mock, fixed_now):
+ get_mock.return_value = _v2([_tweet("1", "only one")])
+ result = x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="deep")
+ assert get_mock.call_count == 1
+ assert len(result["items"]) == 1
+
+ def test_every_request_targets_api_x_com_and_carries_no_token_outside_the_header(self, get_mock, fixed_now):
+ get_mock.side_effect = [
+ _http_error(403, body='{"reason":"client-not-enrolled"}'),
+ _v2([_tweet("1", "a")], next_token="n"),
+ _v2([_tweet("2", "b")]),
+ ]
+ x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="quick")
+ x_api.search_mentions(["steipete"], FROM, TO, topic="t", token=DUMMY_TOKEN)
+ assert get_mock.call_count >= 3
+ for call in get_mock.call_args_list:
+ url = _url(call)
+ assert url.startswith("https://api.x.com/2/tweets/search/")
+ assert DUMMY_TOKEN not in url
+ assert DUMMY_TOKEN not in json.dumps(_params(call))
+ headers = dict(call.kwargs["headers"])
+ headers.pop("Authorization")
+ assert DUMMY_TOKEN not in json.dumps(headers)
+
+ def test_operator_injection_still_carries_engine_window(self, get_mock, fixed_now):
+ get_mock.return_value = _v2([])
+ x_api.search_x(DUMMY_TOKEN, 'foo" OR from:attacker since:2015-01-01 "', FROM, TO)
+ p = _params(get_mock.call_args)
+ assert p["query"] == '"foo" -is:retweet'
+ assert p["start_time"] == "2026-08-10T00:00:00Z"
+ assert p["end_time"] == "2026-09-08T11:59:30Z"
+
+ def test_empty_query_after_sanitizing_makes_no_request(self, get_mock, fixed_now):
+ result = x_api.search_x(DUMMY_TOKEN, "from:attacker OR", FROM, TO)
+ assert result["items"] == []
+ assert result["error"] == x_api.ERR_EMPTY_QUERY
+ get_mock.assert_not_called()
+
+ def test_missing_token_makes_no_request(self, get_mock):
+ result = x_api.search_x("", "topic", FROM, TO)
+ assert result["items"] == [] and result["error"]
+ get_mock.assert_not_called()
+
+
+# ---------------------------------------------------------------------------
+# 403 enrollment fallback (KTD4)
+# ---------------------------------------------------------------------------
+
+
+class TestEnrollmentFallback:
+ def test_enrollment_403_retries_recent_once_with_clamped_window(self, get_mock, fixed_now):
+ get_mock.side_effect = [
+ _http_error(403, body='{"reason":"client-not-enrolled","detail":"...","client_id":"' + ACCOUNT_ID + '"}'),
+ _v2([_tweet("1", "recent post", created_at="2026-09-05T00:00:00Z")], users=_users(("u1", "a"))),
+ ]
+ result = x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="quick")
+ assert get_mock.call_count == 2
+ assert _url(get_mock.call_args_list[0]) == "https://api.x.com/2/tweets/search/all"
+ assert _url(get_mock.call_args_list[1]) == "https://api.x.com/2/tweets/search/recent"
+ p2 = _params(get_mock.call_args_list[1])
+ assert p2["start_time"] == "2026-09-01T12:00:30Z", "start clamped to now minus 7 days"
+ assert p2["end_time"] == "2026-09-08T11:59:30Z"
+ assert "error" not in result
+ assert result["warning"] == x_api.TRUNCATION_DETAIL == "window truncated to 7 days"
+ assert len(result["items"]) == 1
+
+ def test_enrollment_fallback_keeps_start_when_window_already_recent(self, get_mock, fixed_now):
+ get_mock.side_effect = [
+ _http_error(403, body="not enrolled"),
+ _v2([]),
+ ]
+ x_api.search_x(DUMMY_TOKEN, "topic", "2026-09-06", TO, depth="quick")
+ assert _params(get_mock.call_args_list[1])["start_time"] == "2026-09-06T00:00:00Z"
+
+ def test_plain_403_does_not_retry(self, get_mock, fixed_now):
+ get_mock.side_effect = [_http_error(403, body="Forbidden")]
+ result = x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="quick")
+ assert get_mock.call_count == 1
+ assert result["items"] == []
+ assert result["error"] == x_api.ERR_FORBIDDEN
+ assert http.classify_failure(message=result["error"]) == health.AUTH_FAILED
+
+ def test_enrollment_fallback_with_window_before_the_floor_sends_no_request(self, get_mock, fixed_now):
+ """A window that ends before now-7d cannot be served by recent search:
+ no second request with start_time after end_time."""
+ get_mock.side_effect = [_http_error(403, body="client-not-enrolled")]
+ result = x_api.search_x(DUMMY_TOKEN, "topic", "2026-08-01", "2026-08-20", depth="quick")
+ assert get_mock.call_count == 1
+ assert result == {"items": [], "warning": x_api.TRUNCATION_DETAIL}
+
+ def test_second_403_on_recent_is_a_fixed_forbidden_error(self, get_mock, fixed_now):
+ get_mock.side_effect = [
+ _http_error(403, body="client-not-enrolled"),
+ _http_error(403, body="client-not-enrolled " + DUMMY_TOKEN),
+ ]
+ result = x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="quick")
+ assert get_mock.call_count == 2
+ assert result["error"] == x_api.ERR_FORBIDDEN
+
+
+# ---------------------------------------------------------------------------
+# Fixed-string errors (R8, KTD6)
+# ---------------------------------------------------------------------------
+
+# A body that echoes the token and an account id but carries no status
+# marker of its own, so the status code alone decides the fixed string.
+_LEAKY_BODY = json.dumps({
+ "title": "Client Error",
+ "detail": f"request for account {ACCOUNT_ID} rejected; token {DUMMY_TOKEN}",
+ "client_id": ACCOUNT_ID,
+})
+
+
+def _run_capturing_stderr(fn):
+ buf = io.StringIO()
+ with redirect_stderr(buf):
+ with mock.patch("sys.stderr.isatty", return_value=False, create=True):
+ out = fn()
+ return out, buf.getvalue()
+
+
+class TestFixedErrors:
+ @pytest.mark.parametrize("status,expected,state", [
+ (402, x_api.ERR_PAYMENT_REQUIRED, health.PAYMENT_REQUIRED),
+ (403, x_api.ERR_FORBIDDEN, health.AUTH_FAILED),
+ (401, x_api.ERR_UNAUTHORIZED, health.AUTH_FAILED),
+ (429, x_api.ERR_RATE_LIMITED, health.RATE_LIMITED),
+ ])
+ def test_status_maps_to_fixed_string_without_body(self, get_mock, fixed_now, status, expected, state):
+ get_mock.side_effect = _http_error(status, body=_LEAKY_BODY, message=f"HTTP {status}: {ACCOUNT_ID}")
+ result, stderr = _run_capturing_stderr(
+ lambda: x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="quick")
+ )
+ assert result["items"] == []
+ assert result["error"] == expected
+ assert http.classify_failure(message=result["error"]) == state
+ for leak in (DUMMY_TOKEN, ACCOUNT_ID, "Client Error", "request for account"):
+ assert leak not in result["error"]
+ assert leak not in stderr
+ assert "[xapi]" in stderr or "xapi" in stderr
+
+ def test_fixed_strings_are_the_planned_literals(self):
+ assert x_api.ERR_PAYMENT_REQUIRED == "xapi: payment required (X API credits exhausted)"
+ assert x_api.ERR_UNAUTHORIZED == "xapi: unauthorized (bearer token rejected)"
+ assert x_api.ERR_FORBIDDEN == "xapi: forbidden (bearer token lacks access)"
+ assert x_api.ERR_RATE_LIMITED == "xapi: rate limit exceeded (X API)"
+ assert x_api.ERR_TIMED_OUT == "xapi: timed out"
+
+ def test_credit_marker_in_body_maps_to_payment_required_on_any_status(self, get_mock, fixed_now):
+ get_mock.side_effect = _http_error(403, body='{"detail":"insufficient credits"}')
+ result = x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="quick")
+ assert result["error"] == x_api.ERR_PAYMENT_REQUIRED
+
+ def test_other_http_status_is_fixed_http_n(self, get_mock, fixed_now):
+ get_mock.side_effect = _http_error(503, body=_LEAKY_BODY)
+ result = x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="quick")
+ assert result["error"] == "xapi: http 503"
+
+ def test_timeout_is_fixed_string(self, get_mock, fixed_now):
+ get_mock.side_effect = http.HTTPError(
+ f"URL Error: timed out reading {ACCOUNT_ID}", outcome_state=health.TIMEOUT,
+ )
+ result = x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="quick")
+ assert result["error"] == x_api.ERR_TIMED_OUT
+ assert http.classify_failure(message=result["error"]) == health.TIMEOUT
+
+ def test_unexpected_exception_never_carries_its_message(self, get_mock, fixed_now):
+ get_mock.side_effect = ValueError(f"boom {DUMMY_TOKEN} {ACCOUNT_ID}")
+ result, stderr = _run_capturing_stderr(
+ lambda: x_api.search_x(DUMMY_TOKEN, "topic", FROM, TO, depth="quick")
+ )
+ assert result["error"] == "xapi: request failed (ValueError)"
+ assert DUMMY_TOKEN not in stderr and ACCOUNT_ID not in stderr
+
+ @pytest.mark.parametrize("status,state", [(402, health.PAYMENT_REQUIRED), (403, health.AUTH_FAILED)])
+ def test_pipeline_outcome_carries_only_the_fixed_string(self, get_mock, fixed_now, status, state):
+ get_mock.side_effect = _http_error(status, body=_LEAKY_BODY, message=f"HTTP {status}: {ACCOUNT_ID}")
+ sq = schema.SubQuery(label="primary", search_query="q", ranking_query="q?", sources=["x"])
+ runtime = schema.ProviderRuntime(
+ reasoning_provider="mock", planner_model="mock", rerank_model="mock",
+ x_search_backend=None,
+ )
+
+ def run():
+ with mock.patch("lib.env.x_backend_chain", return_value=["xapi"]):
+ with pytest.raises(pipeline.SourceRunError) as ctx:
+ pipeline._retrieve_stream(
+ topic="q", subquery=sq, source="x",
+ config={"X_BEARER_TOKEN": DUMMY_TOKEN},
+ depth="quick", date_range=(FROM, TO),
+ runtime=runtime, mock=False,
+ )
+ return ctx.value
+
+ exc, stderr = _run_capturing_stderr(run)
+ assert exc.outcome_state == state
+ rendered = str(exc)
+ assert DUMMY_TOKEN not in rendered and ACCOUNT_ID not in rendered
+ assert DUMMY_TOKEN not in stderr and ACCOUNT_ID not in stderr
+ detail_state, attempted = pipeline._classify_source_failure(exc)
+ assert (detail_state, attempted) == (state, True)
+
+
+# ---------------------------------------------------------------------------
+# Handle lanes (R7)
+# ---------------------------------------------------------------------------
+
+
+class TestHandleLanes:
+ def test_from_lane_query_excludes_topic(self, get_mock, fixed_now):
+ get_mock.return_value = _v2(
+ [_tweet("1", "anything", created_at="2026-08-20T00:00:00Z")], users=_users(("u1", "steipete")),
+ )
+ items = x_api.search_handles(["@steipete"], "Grok 4", FROM, TO, count_per=8, token=DUMMY_TOKEN)
+ p = _params(get_mock.call_args)
+ assert p["query"] == "from:steipete -is:retweet"
+ assert "Grok" not in p["query"]
+ assert p["start_time"] == "2026-08-10T00:00:00Z"
+ assert [i["id"] for i in items] == ["XF1"]
+ assert items[0]["relevance"] >= 0
+
+ def test_mention_lane_excludes_subjects_own_posts(self, get_mock, fixed_now):
+ get_mock.return_value = _v2(
+ [
+ _tweet("1", "@steipete nice", author_id="fan"),
+ _tweet("2", "my own post", author_id="me"),
+ ],
+ users=_users(("fan", "fanuser"), ("me", "steipete")),
+ )
+ items = x_api.search_mentions(["steipete"], FROM, TO, topic="Grok 4", count_per=5, token=DUMMY_TOKEN)
+ p = _params(get_mock.call_args)
+ assert p["query"] == "@steipete -from:steipete -is:retweet"
+ assert {i["author_handle"] for i in items} == {"fanuser"}
+ assert items[0]["id"].startswith("XA")
+
+ def test_invalid_handle_skips_lane_with_receipt(self, get_mock, fixed_now):
+ items, stderr = _run_capturing_stderr(
+ lambda: x_api.search_handles(["x OR from:elonmusk"], "t", FROM, TO, token=DUMMY_TOKEN)
+ )
+ assert items == []
+ get_mock.assert_not_called()
+ assert "skip" in stderr.lower()
+ mentions, _ = _run_capturing_stderr(
+ lambda: x_api.search_mentions(["bad'; drop", "a" * 16], FROM, TO, token=DUMMY_TOKEN)
+ )
+ assert mentions == []
+ get_mock.assert_not_called()
+
+ def test_item_ids_unique_across_handles(self, get_mock, fixed_now):
+ get_mock.side_effect = [
+ _v2([_tweet("1", "a")], users=_users(("u1", "h1"))),
+ _v2([_tweet("2", "b")], users=_users(("u1", "h2"))),
+ ]
+ items = x_api.search_handles(["h1", "h2"], "topic", FROM, TO, token=DUMMY_TOKEN)
+ ids = [i["id"] for i in items]
+ assert len(ids) == len(set(ids)) == 2
+
+ def test_fatal_auth_failure_stops_remaining_handles(self, get_mock, fixed_now):
+ # Handles fan out on a bounded pool (five in flight), so the handles
+ # already scheduled beside the failing one may still be called; the
+ # ones beyond the pool window are not, and nothing leaks.
+ handles = [f"h{i}" for i in range(1, 8)]
+ get_mock.side_effect = _http_error(401, body=_LEAKY_BODY)
+ items, stderr = _run_capturing_stderr(
+ lambda: x_api.search_handles(handles, "topic", FROM, TO, token=DUMMY_TOKEN)
+ )
+ assert items == []
+ assert 1 <= get_mock.call_count <= x_api._MAX_LANE_WORKERS
+ called = {_params(c)["query"] for c in get_mock.call_args_list}
+ assert called <= {f"from:{h} -is:retweet" for h in handles}
+ assert DUMMY_TOKEN not in stderr and ACCOUNT_ID not in stderr
+
+ def test_handle_results_merge_in_handle_order(self, get_mock, fixed_now):
+ def _by_query(url, headers=None, params=None, **kwargs):
+ handle = params["query"].split(":")[1].split(" ")[0]
+ return _v2([_tweet(str(ord(handle[-1])), handle)], users=_users(("u1", handle)))
+
+ get_mock.side_effect = _by_query
+ handles = ["h3", "h1", "h2"]
+ items = x_api.search_handles(handles, "topic", FROM, TO, token=DUMMY_TOKEN)
+ assert [i["author_handle"] for i in items] == handles
+ assert [i["id"] for i in items] == ["XF1", "XF2", "XF3"]
+
+ def test_transient_failure_continues_to_next_handle(self, get_mock, fixed_now):
+ get_mock.side_effect = [_http_error(500), _v2([_tweet("2", "b")], users=_users(("u1", "h2")))]
+ items = x_api.search_handles(["h1", "h2"], "topic", FROM, TO, token=DUMMY_TOKEN)
+ assert [i["author_handle"] for i in items] == ["h2"]
+
+ def test_no_token_or_handles_returns_empty(self, get_mock):
+ assert x_api.search_handles(["a"], "t", FROM, TO, token="") == []
+ assert x_api.search_handles([], "t", FROM, TO, token=DUMMY_TOKEN) == []
+ assert x_api.search_mentions([], FROM, TO, token=DUMMY_TOKEN) == []
+ get_mock.assert_not_called()
+
+
+# ---------------------------------------------------------------------------
+# Moved helpers (KTD3)
+# ---------------------------------------------------------------------------
+
+
+class TestSharedHelpers:
+ def test_grok_x_imports_helpers_from_x_api(self):
+ from lib import grok_x
+ assert grok_x._decode_snowflake is x_api._decode_snowflake
+ assert grok_x._clean_handle is x_api._clean_handle
+ assert grok_x._looks_generated is x_api._looks_generated
+ assert grok_x._HANDLE_RE is x_api._HANDLE_RE
+
+ def test_snowflake_decodes_to_utc_datetime(self):
+ when = x_api._decode_snowflake("1956158892141441450")
+ assert when is not None and when.tzinfo is not None
+ assert when.year == 2025
+ assert x_api._decode_snowflake("nope") is None
+ assert x_api._decode_snowflake("0") is None
+
+ def test_clean_handle_grammar(self):
+ assert x_api._clean_handle("@steipete") == "steipete"
+ assert x_api._clean_handle("Peter Steinberger") == ""
+ assert x_api._clean_handle("a" * 16) == ""
+ assert x_api._clean_handle("x OR from:elonmusk") == ""
+
+ def test_looks_generated_flags_uniform_runs(self):
+ assert x_api._looks_generated(["100", "200", "300", "400"]) is True
+ assert x_api._looks_generated(["100", "250", "300", "900"]) is False
+ assert x_api._looks_generated(["1", "2"]) is False
+
+
+# ---------------------------------------------------------------------------
+# Pipeline wiring (R7): lane selection and failover line
+# ---------------------------------------------------------------------------
+
+
+def _make_source_item(source, item_id, url, author=None, body=""):
+ return schema.SourceItem(
+ item_id=item_id, source=source, title=f"Item {item_id}", body=body, url=url, author=author,
+ )
+
+
+def _make_plan(topic):
+ return schema.QueryPlan(
+ intent="exploration", freshness_mode="balanced_recent", cluster_mode="topic",
+ raw_topic=topic,
+ subqueries=[schema.SubQuery(
+ label="primary", search_query=topic,
+ ranking_query=f"What recent evidence matters for {topic}?", sources=["x"],
+ )],
+ source_weights={"x": 1.0},
+ )
+
+
+class TestPipelineWiring:
+ def test_chain_xapi_yields_primary_xapi_and_runs_both_lanes(self):
+ bundle = schema.RetrievalBundle()
+ bundle.items_by_source["x"] = [
+ _make_source_item("x", "X1", "https://x.com/analyst1/status/1", author="analyst1", body="AI safety analysis"),
+ _make_source_item("x", "X2", "https://x.com/analyst1/status/2", author="analyst1", body="AI safety research"),
+ ]
+ from_items = [{
+ "id": "XF1", "text": "from analyst1", "url": "https://x.com/analyst1/status/777",
+ "author_handle": "analyst1", "date": "2026-03-15",
+ "engagement": {"likes": 30}, "relevance": 0.8, "why_relevant": "",
+ }]
+ runtime = schema.ProviderRuntime(
+ reasoning_provider="mock", planner_model="mock", rerank_model="mock", x_search_backend=None,
+ )
+ with mock.patch("lib.env.x_backend_chain", return_value=["xapi"]), \
+ mock.patch("lib.entity_extract.extract_entities",
+ return_value={"x_handles": ["analyst1"], "x_hashtags": [], "reddit_subreddits": []}), \
+ mock.patch("lib.x_api.search_handles", return_value=from_items) as from_lane, \
+ mock.patch("lib.x_api.search_mentions", return_value=[]) as about_lane, \
+ mock.patch("lib.bird_x.search_handles") as bird_lane, \
+ mock.patch("lib.xquik.search_handles") as xquik_lane:
+ pipeline._run_supplemental_searches(
+ topic="AI safety", bundle=bundle, plan=_make_plan("AI safety"),
+ config={"X_BEARER_TOKEN": DUMMY_TOKEN}, depth="default",
+ date_range=("2026-02-15", "2026-03-17"), runtime=runtime, mock=False,
+ rate_limited_sources=set(), rate_limit_lock=threading.Lock(),
+ )
+ from_lane.assert_called_once()
+ about_lane.assert_called_once()
+ assert from_lane.call_args.kwargs["token"] == DUMMY_TOKEN
+ assert about_lane.call_args.kwargs["token"] == DUMMY_TOKEN
+ bird_lane.assert_not_called()
+ xquik_lane.assert_not_called()
+ x_urls = {item.url for item in bundle.items_by_source.get("x", [])}
+ assert "https://x.com/analyst1/status/777" in x_urls
+
+ def test_xapi_lanes_share_one_deadline(self):
+ """Explicit-from, extracted-from, about, and related lanes must draw
+ on one wall-clock budget, never a fresh 90s each (review finding)."""
+ bundle = schema.RetrievalBundle()
+ bundle.items_by_source["x"] = [
+ _make_source_item("x", "X1", "https://x.com/analyst1/status/1", author="analyst1", body="AI safety analysis"),
+ ]
+ runtime = schema.ProviderRuntime(
+ reasoning_provider="mock", planner_model="mock", rerank_model="mock", x_search_backend=None,
+ )
+ with mock.patch("lib.env.x_backend_chain", return_value=["xapi"]), \
+ mock.patch("lib.entity_extract.extract_entities",
+ return_value={"x_handles": ["analyst1"], "x_hashtags": [], "reddit_subreddits": []}), \
+ mock.patch("lib.x_api.search_handles", return_value=[]) as from_lane, \
+ mock.patch("lib.x_api.search_mentions", return_value=[]) as about_lane:
+ pipeline._run_supplemental_searches(
+ topic="AI safety", bundle=bundle, plan=_make_plan("AI safety"),
+ config={"X_BEARER_TOKEN": DUMMY_TOKEN}, depth="default",
+ date_range=("2026-02-15", "2026-03-17"), runtime=runtime, mock=False,
+ rate_limited_sources=set(), rate_limit_lock=threading.Lock(),
+ x_handle="steipete", x_related=["peer1"],
+ )
+ calls = from_lane.call_args_list + about_lane.call_args_list
+ assert len(calls) >= 3, "explicit from, extracted from, about, related"
+ deadlines = {c.kwargs.get("deadline") for c in calls}
+ assert len(deadlines) == 1 and None not in deadlines, deadlines
+
+ def test_xapi_lane_deadline_stop_reaches_report_warnings(self):
+ """A lane cut short by the deadline is reported as partial coverage,
+ not presented as complete (review finding)."""
+ bundle = schema.RetrievalBundle()
+ bundle.items_by_source["x"] = [
+ _make_source_item("x", "X1", "https://x.com/analyst1/status/1", author="analyst1", body="AI safety analysis"),
+ ]
+ runtime = schema.ProviderRuntime(
+ reasoning_provider="mock", planner_model="mock", rerank_model="mock", x_search_backend=None,
+ )
+
+ def cut_short(*args, **kwargs):
+ kwargs["warnings"].append(x_api.DEADLINE_DETAIL)
+ return []
+
+ with mock.patch("lib.env.x_backend_chain", return_value=["xapi"]), \
+ mock.patch("lib.entity_extract.extract_entities",
+ return_value={"x_handles": [], "x_hashtags": [], "reddit_subreddits": []}), \
+ mock.patch("lib.x_api.search_handles", side_effect=cut_short), \
+ mock.patch("lib.x_api.search_mentions", side_effect=cut_short):
+ pipeline._run_supplemental_searches(
+ topic="AI safety", bundle=bundle, plan=_make_plan("AI safety"),
+ config={"X_BEARER_TOKEN": DUMMY_TOKEN}, depth="default",
+ date_range=("2026-02-15", "2026-03-17"), runtime=runtime, mock=False,
+ rate_limited_sources=set(), rate_limit_lock=threading.Lock(),
+ x_handle="steipete",
+ )
+ receipts = [w for w in bundle.artifacts.get("x_partial_coverage", []) if x_api.DEADLINE_DETAIL in w]
+ assert receipts == [f"X handle lanes: {x_api.DEADLINE_DETAIL}"], bundle.artifacts.get("x_partial_coverage")
+
+ def test_lane_search_past_a_shared_deadline_sends_no_request(self, get_mock, fixed_now, monkeypatch):
+ monkeypatch.setattr(x_api.time, "monotonic", lambda: 1000.0)
+ notes: list[str] = []
+ assert x_api.search_handles(["steipete", "peer1"], "t", FROM, TO, token=DUMMY_TOKEN, deadline=999.0, warnings=notes) == []
+ assert x_api.search_mentions(["steipete"], FROM, TO, token=DUMMY_TOKEN, deadline=999.0, warnings=notes) == []
+ get_mock.assert_not_called()
+ assert notes == [x_api.DEADLINE_DETAIL], "one receipt per lane call, deduped across handles"
+ get_mock.return_value = _v2([_tweet("1", "hi")], users=_users(("u1", "steipete")))
+ assert len(x_api.search_handles(["steipete"], "t", FROM, TO, token=DUMMY_TOKEN, deadline=1001.0)) == 1
+
+ def test_all_backends_failed_keeps_the_payment_required_state(self):
+ """xapi's 402 must not be masked by a later backend's generic failure:
+ the outcome that reaches doctor says top up, not re-authenticate."""
+ plan = {
+ "intent": "general", "freshness_mode": "balanced_recent", "cluster_mode": "story",
+ "subqueries": [{
+ "label": "primary", "search_query": "topic",
+ "ranking_query": "What are people saying about topic?", "sources": ["x"],
+ }],
+ "source_weights": {"x": 1.0},
+ }
+ answers = iter([
+ ([], x_api.ERR_PAYMENT_REQUIRED),
+ ([], "request failed (HTTPError)"),
+ ])
+ with mock.patch("lib.env.x_backend_chain", return_value=["xapi", "xai"]), \
+ mock.patch("lib.pipeline._fetch_x_backend", side_effect=lambda *a, **k: next(answers)):
+ report = pipeline.run(
+ topic="topic", config={"X_BEARER_TOKEN": DUMMY_TOKEN, "XAI_API_KEY": "dummy-xai"},
+ depth="quick", requested_sources=["x"], mock=False,
+ external_plan=plan, web_backend="none", save_dir="",
+ )
+ assert report.source_status["x"].state == health.PAYMENT_REQUIRED
+ assert "payment required" in report.source_status["x"].detail
+
+ def test_fetch_x_backend_registers_xapi(self, get_mock, fixed_now):
+ get_mock.return_value = _v2([_tweet("1", "hello topic")], users=_users(("u1", "a")))
+ items, err = pipeline._fetch_x_backend(
+ "xapi", "topic", FROM, TO, "quick", {"X_BEARER_TOKEN": DUMMY_TOKEN},
+ )
+ assert err == ""
+ assert [i["id"] for i in items] == ["XAPI1"]
+ assert get_mock.call_args.kwargs["headers"]["Authorization"] == f"Bearer {DUMMY_TOKEN}"
+
+ def test_fetch_x_backend_surfaces_truncation_warning(self, get_mock, fixed_now):
+ get_mock.side_effect = [
+ _http_error(403, body="client-not-enrolled"),
+ _v2([_tweet("1", "hello topic", created_at="2026-09-05T00:00:00Z")]),
+ ]
+ warnings: list[str] = []
+ items, err = pipeline._fetch_x_backend(
+ "xapi", "topic", FROM, TO, "quick", {"X_BEARER_TOKEN": DUMMY_TOKEN}, warnings=warnings,
+ )
+ assert err == "" and len(items) == 1
+ assert warnings and "window truncated to 7 days" in warnings[0]
+
+ def test_truncation_receipt_reaches_the_stream_artifact(self, get_mock, fixed_now):
+ get_mock.side_effect = [
+ _http_error(403, body="client-not-enrolled"),
+ _v2([_tweet("1", "hello topic", created_at="2026-09-05T00:00:00Z")]),
+ ]
+ sq = schema.SubQuery(label="primary", search_query="q", ranking_query="q?", sources=["x"])
+ runtime = schema.ProviderRuntime(
+ reasoning_provider="mock", planner_model="mock", rerank_model="mock", x_search_backend=None,
+ )
+
+ def run():
+ with mock.patch("lib.env.x_backend_chain", return_value=["xapi"]):
+ return pipeline._retrieve_stream(
+ topic="q", subquery=sq, source="x", config={"X_BEARER_TOKEN": DUMMY_TOKEN},
+ depth="quick", date_range=(FROM, TO), runtime=runtime, mock=False,
+ )
+
+ (items, artifact), stderr = _run_capturing_stderr(run)
+ assert len(items) == 1
+ assert artifact["x_receipts"] == ["X: xapi window truncated to 7 days"]
+ assert "window truncated to 7 days" in stderr
+
+ def test_failover_stderr_line_names_xapi_and_credits(self):
+ sq = schema.SubQuery(label="primary", search_query="q", ranking_query="q?", sources=["x"])
+ runtime = schema.ProviderRuntime(
+ reasoning_provider="mock", planner_model="mock", rerank_model="mock", x_search_backend=None,
+ )
+ good = [{"id": "XAPI1", "text": "t", "url": "https://x.com/a/status/1", "author_handle": "a",
+ "date": "2026-08-20", "engagement": None, "relevance": 0.5, "why_relevant": ""}]
+
+ def fetch(backend, *_args, **_kwargs):
+ return (good, "") if backend == "xapi" else ([], "")
+
+ def run():
+ with mock.patch("lib.env.x_backend_chain", return_value=["xai", "xapi"]), \
+ mock.patch("lib.pipeline._fetch_x_backend", side_effect=fetch):
+ return pipeline._retrieve_stream(
+ topic="q", subquery=sq, source="x", config={"X_BEARER_TOKEN": DUMMY_TOKEN},
+ depth="quick", date_range=(FROM, TO), runtime=runtime, mock=False,
+ )
+
+ (items, _artifact), stderr = _run_capturing_stderr(run)
+ assert len(items) == 1
+ assert "used fallback 'xapi'" in stderr
+ assert "X API credits" in stderr
+
+
+# ---------------------------------------------------------------------------
+# Fixture redaction at both seams (R8)
+# ---------------------------------------------------------------------------
+
+
+def _urlopen_response(body: str):
+ response = MagicMock()
+ response.__enter__.return_value = response
+ response.__exit__.return_value = False
+ response.read.return_value = body.encode("utf-8")
+ response.status = 200
+ return response
+
+
+def test_fixture_redacts_bearer_loaded_only_from_env_file(tmp_path, monkeypatch):
+ env_file = tmp_path / ".env"
+ env_file.write_text(f"X_BEARER_TOKEN={DUMMY_TOKEN}\n", encoding="utf-8")
+ env_file.chmod(0o600)
+ monkeypatch.setattr(env, "CONFIG_FILE", env_file)
+ monkeypatch.delenv("X_BEARER_TOKEN", raising=False)
+ monkeypatch.setenv("LAST30DAYS_SKIP_KEYCHAIN", "1")
+ monkeypatch.setattr(env, "_load_pass", lambda *a, **k: {})
+ monkeypatch.setattr(env, "_find_project_env", lambda: None)
+ monkeypatch.setattr(
+ http.urllib.request, "urlopen",
+ lambda *_a, **_k: _urlopen_response(json.dumps({"detail": f"token {DUMMY_TOKEN} rejected"})),
+ )
+ fixture_dir = tmp_path / "fixture"
+
+ with http.recording_requests(fixture_dir):
+ config = env.get_config()
+ assert config["X_BEARER_TOKEN"] == DUMMY_TOKEN
+ http.get(
+ "https://api.x.com/2/tweets/search/all",
+ headers={"Authorization": f"Bearer {DUMMY_TOKEN}"},
+ params={"query": "topic"},
+ )
+ http.fixture_source_record(
+ {"source": "x", "topic": "t", "search_query": "t", "date_range": [FROM, TO], "depth": "quick"},
+ [[{"text": f"echo {DUMMY_TOKEN}"}], {}],
+ )
+
+ payload = json.loads((fixture_dir / "http.json").read_text(encoding="utf-8"))
+ text = json.dumps(payload)
+ assert DUMMY_TOKEN not in text
+ assert "<redacted>" in json.dumps(payload["exchanges"])
+ assert "<redacted>" in json.dumps(payload["source_exchanges"])
+
+
+def test_http_seam_redacts_bare_bearer_without_any_config(tmp_path, monkeypatch):
+ monkeypatch.delenv("X_BEARER_TOKEN", raising=False)
+ monkeypatch.setattr(
+ http.urllib.request, "urlopen",
+ lambda *_a, **_k: _urlopen_response(json.dumps({"echo": DUMMY_TOKEN})),
+ )
+ fixture_dir = tmp_path / "fixture"
+ with http.recording_requests(fixture_dir):
+ http.get("https://api.x.com/2/tweets/search/all", headers={"Authorization": f"Bearer {DUMMY_TOKEN}"})
+ text = (fixture_dir / "http.json").read_text(encoding="utf-8")
+ assert DUMMY_TOKEN not in text
+ assert '"echo": "<redacted>"' in text
+
+
+# ---------------------------------------------------------------------------
+# xurl fixed strings (same rule as xapi)
+# ---------------------------------------------------------------------------
+
+
+class TestXurlFixedStrings:
+ def _run(self, returncode, stderr="", stdout=""):
+ completed = mock.Mock(returncode=returncode, stdout=stdout, stderr=stderr)
+ with mock.patch("subprocess.run", return_value=completed):
+ return xurl_x.search_x("test")
+
+ def test_rate_limited_stderr_with_token_maps_to_fixed_string(self):
+ result = self._run(1, stderr=f"429 rate limit exceeded for {DUMMY_TOKEN} ({ACCOUNT_ID})")
+ assert result["error"] == xurl_x.ERR_RATE_LIMITED
+ assert DUMMY_TOKEN not in result["error"] and ACCOUNT_ID not in result["error"]
+ assert http.classify_failure(message=result["error"]) == health.RATE_LIMITED
+
+ @pytest.mark.parametrize("stderr,expected,state", [
+ ("401 Unauthorized", "ERR_UNAUTHORIZED", health.AUTH_FAILED),
+ ("403 Forbidden: client-not-enrolled", "ERR_FORBIDDEN", health.AUTH_FAILED),
+ ("402 Payment Required", "ERR_PAYMENT_REQUIRED", health.PAYMENT_REQUIRED),
+ ("something else entirely", "ERR_FAILED", health.ERROR),
+ ])
+ def test_status_words_map_to_fixed_strings(self, stderr, expected, state):
+ result = self._run(1, stderr=f"{stderr} {DUMMY_TOKEN}")
+ assert result["error"] == getattr(xurl_x, expected)
+ assert DUMMY_TOKEN not in result["error"]
+ assert http.classify_failure(message=result["error"]) == state
+
+ def test_invalid_json_is_fixed_string(self):
+ result = self._run(0, stdout=f"NOT JSON {DUMMY_TOKEN}")
+ assert result["error"] == xurl_x.ERR_INVALID_JSON
+ assert DUMMY_TOKEN not in result["error"]
diff --git a/tests/test_x_envelope.py b/tests/test_x_envelope.py
new file mode 100644
index 00000000..43dd17c6
--- /dev/null
+++ b/tests/test_x_envelope.py
@@ -0,0 +1,1122 @@
+"""Host-fetched X envelope (``--x-posts``): the U5 contract.
+
+The envelope is a file of flat post rows the hosting model fetched through its
+X connector. It is validated strictly at the top level (fail closed, exit 2)
+and leniently per row (drop and count), every citation is rebuilt from a
+numeric id and a grammar-valid handle, and the envelope replaces the engine's
+X fetch for exactly one serve.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import hashlib
+import io
+import json
+import os
+import re
+import stat
+import sys
+from contextlib import redirect_stderr, redirect_stdout
+from datetime import date, datetime, timedelta, timezone
+from pathlib import Path
+from unittest import mock
+
+import pytest
+
+import last30days as cli
+from lib import dates, env, health, html_render, pipeline, render, schema, x_api, x_envelope
+
+TOPIC = "ai agents"
+FROM, TO = dates.get_date_range(30)
+SUBJECT = "steipete"
+RELATED = "peer1"
+ACCOUNT_ID = "acct-987654321-SECRET"
+
+
+# ---------------------------------------------------------------------------
+# Fixture builders
+# ---------------------------------------------------------------------------
+
+
+def _day(offset: int) -> str:
+ return (date.fromisoformat(TO) - timedelta(days=offset)).isoformat()
+
+
+def _snowflake(day: str, *, hour: int = 12, minute: int = 0, low: int = 0) -> str:
+ when = datetime.strptime(day, "%Y-%m-%d").replace(tzinfo=timezone.utc)
+ when += timedelta(hours=hour, minutes=minute)
+ ms = int(when.timestamp() * 1000)
+ return str(((ms - x_api._SNOWFLAKE_EPOCH_MS) << 22) | (low & ((1 << 22) - 1)))
+
+
+# Day offsets with irregular gaps so a normal fixture never reads as generated.
+_OFFSETS = (1, 2, 4, 9, 11, 18)
+
+
+def _row(
+ n: int,
+ handle: str = SUBJECT,
+ text: str = "ai agents are shipping fast",
+ *,
+ created_at: str | None | object = "auto",
+ likes: int = 10,
+ **extra,
+) -> dict:
+ day = _day(_OFFSETS[n % len(_OFFSETS)])
+ pid = _snowflake(day, hour=8 + n, minute=(n * 17) % 60, low=(n + 1) * 104729)
+ row = {
+ "id": pid,
+ "author_handle": handle,
+ "created_at": f"{day}T{8 + n:02d}:{(n * 17) % 60:02d}:00Z" if created_at == "auto" else created_at,
+ "text": text,
+ "likes": likes,
+ "reposts": 1,
+ "replies": 0,
+ "quotes": 0,
+ }
+ if created_at is None:
+ row.pop("created_at")
+ row.update(extra)
+ return row
+
+
+def _call(lane: str = "topic", handles=(), posts=()) -> dict:
+ return {"lane": lane, "handles": list(handles), "posts": list(posts)}
+
+
+def _envelope(calls, **over) -> dict:
+ payload = {
+ "schema": x_envelope.SCHEMA,
+ "generated_at": datetime.now(timezone.utc).isoformat(),
+ "topic": TOPIC,
+ "window": {"from": FROM, "to": TO},
+ "provider": "x-connector",
+ "status": "ok",
+ "calls": list(calls),
+ }
+ payload.update(over)
+ return payload
+
+
+def _write(tmp_path: Path, payload, name: str = "posts.json") -> str:
+ path = tmp_path / name
+ text = payload if isinstance(payload, str) else json.dumps(payload)
+ path.write_text(text, encoding="utf-8")
+ return str(path)
+
+
+def _read(path: str, *, topic: str = TOPIC, handles=(SUBJECT,), related=(), window=(FROM, TO)):
+ return x_envelope.read(path, window, topic, handles=list(handles), related=list(related))
+
+
+def _basic(tmp_path: Path, **over) -> str:
+ """Topic call with two rows plus a from call with one subject row."""
+ return _write(tmp_path, _envelope([
+ _call("topic", posts=[_row(0, "alice", "ai agents review"), _row(1, "bob", "agents everywhere")]),
+ _call("from", handles=[SUBJECT], posts=[_row(2, SUBJECT, "shipping my agent today")]),
+ ], **over))
+
+
+def _plan(sources=("x",)) -> dict:
+ return {
+ "intent": "general",
+ "freshness_mode": "balanced_recent",
+ "cluster_mode": "story",
+ "subqueries": [{
+ "label": "primary",
+ "search_query": TOPIC,
+ "ranking_query": f"What are people saying about {TOPIC}?",
+ "sources": list(sources),
+ }],
+ "source_weights": {s: 1.0 for s in sources},
+ }
+
+
+def _two_x_plan() -> dict:
+ plan = _plan()
+ plan["subqueries"].append({
+ "label": "angle",
+ "search_query": "agent frameworks",
+ "ranking_query": "Which agent frameworks are people using?",
+ "sources": ["x"],
+ })
+ return plan
+
+
+def _no_backend():
+ with mock.patch("lib.env.x_backend_chain", return_value=[]), \
+ mock.patch(
+ "lib.pipeline._fetch_x_backend",
+ side_effect=AssertionError("no X backend may be called on an envelope run"),
+ ) as fetch:
+ yield fetch
+
+
+def _run(envelope, *, config=None, x_handle=None, x_related=None, depth="default",
+ plan=None, requested=("x",), topic=TOPIC, **kwargs):
+ with _no_backend():
+ return pipeline.run(
+ topic=topic, config=dict(config or {}), depth=depth,
+ requested_sources=list(requested) if requested is not None else None,
+ mock=False, x_handle=x_handle, x_related=x_related,
+ external_plan=plan or _plan(), x_posts=envelope,
+ web_backend="none", save_dir="", **kwargs,
+ )
+
+
+def _capture(fn):
+ err = io.StringIO()
+ with redirect_stderr(err):
+ result = fn()
+ return result, err.getvalue()
+
+
+# ---------------------------------------------------------------------------
+# Ingestion: happy path and row trust (R9, R10, AE5, AE6, AE6a)
+# ---------------------------------------------------------------------------
+
+
+class TestIngestion:
+ def test_valid_envelope_serves_topic_and_from_lanes_without_a_backend(self, tmp_path):
+ envelope, stderr = _capture(lambda: _read(_basic(tmp_path)))
+ assert envelope.accepted == 3 and envelope.total == 3
+ assert envelope.status == "ok"
+ assert [c.lane for c in envelope.lane_calls] == ["from"]
+ assert len(envelope.topic_items) == 2
+ assert "host-fetched X: accepted 3 of 3" in stderr
+ assert "lanes: topic 2, from 1, mention 0, related 0" in stderr
+
+ with _no_backend() as fetch:
+ report = pipeline.run(
+ topic=TOPIC, config={}, depth="default", requested_sources=["x"],
+ mock=False, x_handle=SUBJECT, external_plan=_plan(), x_posts=envelope,
+ web_backend="none", save_dir="",
+ )
+ fetch.assert_not_called()
+ urls = {item.url for item in report.items_by_source["x"]}
+ assert len(urls) == 3
+ assert all(u.startswith("https://x.com/") for u in urls)
+ by_author = {item.author: item for item in report.items_by_source["x"]}
+ assert SUBJECT in by_author, "from-lane row must land under x"
+ assert report.source_status["x"].state == health.OK
+ assert "x" not in report.errors_by_source
+
+ def test_lane_calls_are_served_on_quick_runs(self, tmp_path):
+ """The host already paid for the lanes: --quick must not drop them."""
+ envelope = _read(_basic(tmp_path))
+ report = _run(envelope, x_handle=SUBJECT, depth="quick")
+ by_author = {item.author: item for item in report.items_by_source["x"]}
+ assert SUBJECT in by_author, "from-lane row must land under x on a quick run"
+ assert len(report.items_by_source["x"]) == 3
+
+ def test_mention_lane_rows_reach_the_report(self, tmp_path):
+ envelope = _read(_write(tmp_path, _envelope([
+ _call("topic", posts=[_row(0, "alice", "ai agents review")]),
+ _call("mention", handles=[SUBJECT], posts=[_row(4, "fan", f"@{SUBJECT} love the agent work")]),
+ ])))
+ assert envelope.lane_counts["mention"] == 1
+ report = _run(envelope, x_handle=SUBJECT)
+ by_author = {item.author: item for item in report.items_by_source["x"]}
+ assert "fan" in by_author, "mention-lane row must land under x"
+ assert len(report.items_by_source["x"]) == 2
+
+ def test_from_row_gets_first_party_handling(self, tmp_path):
+ # A subject post that never repeats the topic must survive the
+ # relevance floor: that is what first-party handling means.
+ path = _write(tmp_path, _envelope([
+ _call("topic", posts=[_row(0, "alice", "ai agents review")]),
+ _call("from", handles=[SUBJECT], posts=[_row(2, SUBJECT, "lunch was great")]),
+ ]))
+ envelope = _read(path)
+ report = _run(envelope, x_handle=SUBJECT)
+ authors = {item.author for item in report.items_by_source["x"]}
+ assert SUBJECT in authors
+
+ def test_created_at_disagreeing_with_snowflake_is_date_mismatch(self, tmp_path):
+ # created_at inside the window; the snowflake decodes two months earlier.
+ early = (date.fromisoformat(FROM) - timedelta(days=60)).isoformat()
+ forged = _row(0, "alice")
+ forged["id"] = _snowflake(early, low=99)
+ path = _write(tmp_path, _envelope([_call("topic", posts=[forged, _row(1, "bob")])]))
+ envelope = _read(path)
+ assert envelope.counters["date-mismatch"] == 1
+ assert envelope.accepted == 1
+
+ def test_off_domain_url_is_ignored_and_citation_rebuilt(self, tmp_path):
+ row = _row(0, "alice", url="https://evil.example/alice/status/1")
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ assert envelope.accepted == 1
+ item = envelope.topic_items[0]
+ assert item["url"] == f"https://x.com/i/status/{row['id']}"
+ assert "evil.example" not in json.dumps(envelope.topic_items)
+
+ def test_spoofed_subject_handle_with_off_domain_url_is_not_attributed(self, tmp_path):
+ """AE6a: the row is kept, its citation is rebuilt from the id, the
+ subject attribution is dropped, and handle-mismatch is counted."""
+ row = _row(0, SUBJECT, "totally the subject", url="https://evil.example/x")
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])), handles=[SUBJECT])
+ assert envelope.counters["handle-mismatch"] == 1
+ item = envelope.topic_items[0]
+ assert item["author_handle"] == ""
+ assert item["url"] == f"https://x.com/i/status/{row['id']}"
+
+ def test_url_handle_or_id_disagreement_is_handle_mismatch(self, tmp_path):
+ row = _row(0, SUBJECT)
+ row["url"] = f"https://x.com/someoneelse/status/{row['id']}"
+ other = _row(1, "bob")
+ other["url"] = "https://x.com/bob/status/1234567890"
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row, other])])))
+ assert envelope.counters["handle-mismatch"] == 2
+ assert all(item["author_handle"] == "" for item in envelope.topic_items)
+
+ def test_future_snowflake_is_dropped(self, tmp_path):
+ future = (date.today() + timedelta(days=40)).isoformat()
+ row = _row(0, "alice", created_at=None)
+ row["id"] = _snowflake(future)
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row, _row(1, "bob")])])))
+ assert envelope.accepted == 1
+ assert envelope.counters["out-of-window"] == 1
+
+ def test_uniform_id_sequence_rejects_the_whole_envelope(self, tmp_path):
+ base = int(_snowflake(_day(10)))
+ step = 1 << 30
+ rows = []
+ for i in range(5):
+ row = _row(i, "alice", f"post {i}", created_at=None)
+ row["id"] = str(base + i * step)
+ rows.append(row)
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=rows)])))
+ assert envelope.status == "error"
+ assert envelope.accepted == 0 and envelope.topic_items == []
+ report = _run(envelope)
+ assert report.source_status["x"].state == health.ERROR
+ assert "generated" in (report.source_status["x"].detail or "")
+
+ def test_duplicate_ids_across_calls_first_occurrence_wins(self, tmp_path):
+ first = _row(0, "alice", "first copy")
+ second = dict(first, text="second copy", author_handle="bob")
+ envelope = _read(_write(tmp_path, _envelope([
+ _call("topic", posts=[first]),
+ _call("topic", posts=[second]),
+ ])))
+ assert envelope.counters["duplicate"] == 1
+ assert envelope.accepted == 1
+ assert envelope.topic_items[0]["text"] == "first copy"
+
+ def test_missing_created_at_derives_date_from_snowflake(self, tmp_path):
+ row = _row(0, "alice", created_at=None)
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ assert envelope.accepted == 1
+ assert envelope.topic_items[0]["date"] == _day(_OFFSETS[0])
+
+ def test_unparseable_created_at_is_ignored(self, tmp_path):
+ row = _row(0, "alice", created_at="yesterday-ish")
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ assert envelope.accepted == 1
+ assert envelope.topic_items[0]["date"] == _day(_OFFSETS[0])
+
+ def test_row_outside_engine_window_is_dropped(self, tmp_path):
+ old = (date.fromisoformat(FROM) - timedelta(days=5)).isoformat()
+ row = _row(0, "alice", created_at=f"{old}T10:00:00Z")
+ row["id"] = _snowflake(old, hour=10)
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ assert envelope.accepted == 0
+ assert envelope.counters["out-of-window"] == 1
+
+ def test_missing_id_or_text_is_dropped(self, tmp_path):
+ no_id = _row(0, "alice")
+ no_id.pop("id")
+ no_text = _row(1, "bob", text=" ")
+ non_numeric = _row(2, "carol")
+ non_numeric["id"] = "abc123"
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[no_id, no_text, non_numeric])])))
+ assert envelope.accepted == 0
+ assert envelope.counters["missing-id-text"] == 3
+
+ def test_missing_username_keeps_row_with_i_status_url(self, tmp_path):
+ row = _row(0, "")
+ placeholder = _row(1, "unknown")
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row, placeholder])])))
+ assert envelope.accepted == 2
+ assert envelope.counters["uncitable"] == 2
+ urls = {item["url"] for item in envelope.topic_items}
+ assert urls == {
+ f"https://x.com/i/status/{row['id']}",
+ f"https://x.com/i/status/{placeholder['id']}",
+ }
+
+ def test_handle_with_embedded_newline_is_rejected(self, tmp_path):
+ row = _row(0, "stei\npete")
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ assert envelope.topic_items[0]["author_handle"] == ""
+ assert envelope.counters["uncitable"] == 1
+
+ def test_control_characters_are_stripped_but_newlines_in_text_survive(self, tmp_path):
+ row = _row(0, "alice", text="line one\x07\x1b[31m\nline two…")
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ assert envelope.topic_items[0]["text"] == "line one[31m\nline two"
+
+ def test_markdown_link_tail_and_comment_opener_are_defanged(self, tmp_path):
+ row = _row(0, "alice", text="read [this](https://evil.example) <!-- hidden --> ok")
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ text = envelope.topic_items[0]["text"]
+ assert "](" not in text and "<!--" not in text
+ assert "evil.example" in text, "the characters stay visible as evidence"
+
+ def test_text_over_cap_is_truncated_and_counted(self, tmp_path):
+ row = _row(0, "alice", text="x" * (x_envelope.MAX_TEXT_CHARS + 50))
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ assert envelope.counters["truncated"] == 1
+ assert len(envelope.topic_items[0]["text"]) == x_envelope.MAX_TEXT_CHARS
+
+ def test_extra_keys_are_ignored_and_counted_once(self, tmp_path):
+ row = _row(0, "alice", public_metrics={"like_count": 5}, entities={}, edit_history_tweet_ids=["1"])
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ assert envelope.accepted == 1
+ assert envelope.counters["extra-fields"] == 1
+ item = envelope.topic_items[0]
+ assert item["engagement"]["likes"] == 10
+ assert "public_metrics" not in item
+
+ def test_rows_normalize_through_the_x_item_shape(self, tmp_path):
+ row = _row(0, "alice", "@bob @carol thoughts on ai agents?", likes=7)
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ item = envelope.topic_items[0]
+ assert item["id"].startswith("XHOST")
+ assert item["post_id"] == row["id"]
+ assert item["engagement"] == {"likes": 7, "reposts": 1, "replies": 0, "quotes": 0}
+ assert item["mentioned_handles"] == ["bob", "carol"]
+ assert item["url"] == f"https://x.com/alice/status/{row['id']}"
+ normalized = pipeline._normalize_score_dedupe(
+ "x", [item], FROM, TO, freshness_mode="balanced_recent", ranking_query=TOPIC,
+ )
+ assert normalized and normalized[0].author == "alice"
+ assert normalized[0].url == item["url"]
+
+ def test_digest_is_the_file_sha256(self, tmp_path):
+ path = _basic(tmp_path)
+ envelope = _read(path)
+ assert envelope.sha256 == hashlib.sha256(Path(path).read_bytes()).hexdigest()
+
+
+# ---------------------------------------------------------------------------
+# Rendering safety (AE6a, XSS)
+# ---------------------------------------------------------------------------
+
+
+def _hrefs(html: str) -> set[str]:
+ return set(re.findall(r'href="([^"]+)"', html))
+
+
+class TestRendering:
+ def test_html_has_no_href_outside_x_com(self, tmp_path):
+ row = _row(0, "alice", "see this ai agents thread", url="https://evil.example/alice/status/1")
+ envelope = _read(_write(tmp_path, _envelope([
+ _call("topic", posts=[row, _row(1, "bob", "ai agents everywhere")]),
+ ])))
+ report = _run(envelope)
+ # The HTML page carries evidence links only through the markdown
+ # body, so convert the compact report the way html_render does.
+ rendered = html_render._markdown_to_html(render.render_compact(report))
+ status_links = [h for h in _hrefs(rendered) if "/status/" in h]
+ assert len(status_links) == 2, "the fixture's citations must render as links"
+ for href in status_links:
+ assert href.startswith("https://x.com/"), href
+ assert f"https://x.com/i/status/{row['id']}" in status_links
+ assert "evil.example" not in rendered
+ page = html_render.render_html(report)
+ assert "evil.example" not in page
+ assert all(h.startswith("https://x.com/") for h in _hrefs(page) if "/status/" in h)
+
+ def test_xss_row_is_escaped_in_html_and_inert_in_markdown(self, tmp_path):
+ payload = (
+ '<img src=x onerror=alert(1)> [x](javascript:alert(1)) '
+ '<!-- META: <img src=x onerror=alert(2)> --> ai agents'
+ )
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[_row(0, "alice", payload)])])))
+ report = _run(envelope)
+ assert report.items_by_source["x"], "the row itself is valid evidence"
+ md = render.render_compact(report)
+ assert "onerror" in md, "the text itself is preserved as evidence"
+ assert "](javascript:" not in md
+ assert "<!-- META:" not in md
+ html = html_render._promote_meta_marker(
+ html_render._wrap_engine_footer(html_render._markdown_to_html(md))
+ )
+ assert "<img" not in html
+ assert 'href="javascript:' not in html
+ assert "&lt;img src=x onerror=alert(1)&gt;" in html
+ page = html_render.render_html(report)
+ assert "<img" not in page and 'href="javascript:' not in page
+
+
+# ---------------------------------------------------------------------------
+# Fail-closed input bounds (KTD5): exit 2, path named, no content echoed
+# ---------------------------------------------------------------------------
+
+
+def _assert_contract(path: str, *, must_not_contain=(), **read_kwargs):
+ with pytest.raises(x_envelope.EnvelopeContractError) as ctx:
+ _read(path, **read_kwargs)
+ message = ctx.value.message
+ assert str(path) in message
+ for needle in must_not_contain:
+ assert needle not in message
+ return message
+
+
+class TestInputBounds:
+ def test_posts_not_a_list_of_objects_exits_2(self, tmp_path):
+ path = _write(tmp_path, _envelope([{"lane": "topic", "handles": [], "posts": "SECRETVALUE"}]))
+ _assert_contract(path, must_not_contain=["SECRETVALUE"])
+ path2 = _write(tmp_path, _envelope([_call("topic", posts=["SECRETROW"])]), "b.json")
+ _assert_contract(path2, must_not_contain=["SECRETROW"])
+
+ def test_oversized_file_is_rejected_before_reading(self, tmp_path):
+ path = _basic(tmp_path)
+ real = os.stat(path)
+
+ class _Big:
+ st_mode = real.st_mode
+ st_size = x_envelope.MAX_BYTES + 1
+
+ with mock.patch.object(x_envelope, "_stat", return_value=_Big()), \
+ mock.patch.object(Path, "read_bytes", side_effect=AssertionError("must not read")):
+ _assert_contract(path)
+
+ def test_deeply_nested_json_is_rejected(self, tmp_path):
+ path = _write(tmp_path, "[" * 200000 + "]" * 200000)
+ message = _assert_contract(path)
+ assert "[[[" not in message
+
+ def test_fifo_is_rejected_without_opening(self, tmp_path):
+ fifo = tmp_path / "pipe.json"
+ os.mkfifo(fifo)
+ with mock.patch("builtins.open", side_effect=AssertionError("must not open a FIFO")):
+ _assert_contract(str(fifo))
+
+ def test_non_utf8_is_rejected_without_echo(self, tmp_path):
+ path = tmp_path / "bad.json"
+ path.write_bytes(b'{"schema": "x", "secret": "SECRETVALUE"' + b"\xff\xfe" + b"}")
+ _assert_contract(str(path), must_not_contain=["SECRETVALUE"])
+
+ def test_env_shaped_input_is_rejected(self, tmp_path):
+ path = tmp_path / ".env"
+ path.write_text("X_BEARER_TOKEN=SECRETVALUE\n", encoding="utf-8")
+ _assert_contract(str(path), must_not_contain=["SECRETVALUE"])
+ as_json = tmp_path / "config.json"
+ as_json.write_text('{"X_BEARER_TOKEN": "SECRETVALUE"}', encoding="utf-8")
+ _assert_contract(str(as_json), must_not_contain=["SECRETVALUE"])
+
+ def test_config_dir_and_credential_stores_are_rejected(self, tmp_path, monkeypatch):
+ home = tmp_path / "home"
+ cfg = tmp_path / "cfg"
+ for directory in (home / ".grok", home / ".xurl", cfg):
+ directory.mkdir(parents=True)
+ monkeypatch.setenv("HOME", str(home))
+ monkeypatch.setattr(env, "CONFIG_DIR", cfg)
+ monkeypatch.setattr(env, "CONFIG_FILE", cfg / ".env")
+ for target in (home / ".grok" / "auth.json", home / ".xurl" / "tokens.json", cfg / "posts.json"):
+ target.write_text(json.dumps({"token": "SECRETVALUE"}), encoding="utf-8")
+ _assert_contract(str(target), must_not_contain=["SECRETVALUE"])
+ # A symlink into a store is resolved before the check.
+ link = tmp_path / "link.json"
+ link.symlink_to(home / ".grok" / "auth.json")
+ _assert_contract(str(link), must_not_contain=["SECRETVALUE"])
+
+ def test_wrong_suffix_directory_and_missing_file_are_rejected(self, tmp_path):
+ _assert_contract(_write(tmp_path, _envelope([]), "posts.txt"))
+ _assert_contract(str(tmp_path))
+ _assert_contract(str(tmp_path / "missing.json"))
+
+ def test_call_and_row_caps_are_enforced(self, tmp_path):
+ too_many_calls = _envelope([_call("topic", posts=[_row(0)]) for _ in range(x_envelope.MAX_CALLS + 1)])
+ _assert_contract(_write(tmp_path, too_many_calls, "calls.json"))
+ big_call = _envelope([_call("topic", posts=[_row(i % 6) for i in range(x_envelope.MAX_ROWS_PER_CALL + 1)])])
+ _assert_contract(_write(tmp_path, big_call, "rows.json"))
+ total = _envelope([
+ _call("topic", posts=[_row(i % 6) for i in range(x_envelope.MAX_ROWS_PER_CALL)])
+ for _ in range(3)
+ ])
+ _assert_contract(_write(tmp_path, total, "total.json"))
+
+
+class TestMalformedEnvelope:
+ @pytest.mark.parametrize("mutation", [
+ {"schema": None},
+ {"schema": "last30days-x-posts/2"},
+ {"status": "great"},
+ {"calls": {"lane": "topic"}},
+ {"calls": []},
+ {"generated_at": None},
+ {"window": None},
+ {"topic": None},
+ ])
+ def test_malformed_exits_2_with_two_fix_remedy(self, tmp_path, mutation):
+ payload = _envelope([_call("topic", posts=[_row(0)])])
+ for key, value in mutation.items():
+ if value is None:
+ payload.pop(key)
+ else:
+ payload[key] = value
+ path = _write(tmp_path, payload)
+ message = _assert_contract(path, must_not_contain=["great", "last30days-x-posts/2"])
+ assert "rewrite" in message.lower()
+ assert "--x-posts" in message
+
+ def test_not_json_and_not_an_object_exit_2(self, tmp_path):
+ _assert_contract(_write(tmp_path, "{not json SECRETVALUE", "a.json"), must_not_contain=["SECRETVALUE"])
+ _assert_contract(_write(tmp_path, '["SECRETVALUE"]', "b.json"), must_not_contain=["SECRETVALUE"])
+
+ def test_future_generated_at_exits_2(self, tmp_path):
+ """A stamp ahead of the clock must not outlive the freshness gate."""
+ ahead = (datetime.now(timezone.utc) + timedelta(hours=2)).isoformat()
+ message = _assert_contract(_write(tmp_path, _envelope([_call("topic", posts=[_row(0)])], generated_at=ahead)))
+ assert "future" in message and ahead not in message
+ skew = (datetime.now(timezone.utc) + timedelta(minutes=2)).isoformat()
+ assert _read(_write(tmp_path, _envelope([_call("topic", posts=[_row(0)])], generated_at=skew), "ok.json")).accepted == 1
+
+ def test_stale_generated_at_exits_2(self, tmp_path):
+ stale = (datetime.now(timezone.utc) - timedelta(hours=7)).isoformat()
+ message = _assert_contract(_write(tmp_path, _envelope([_call("topic", posts=[_row(0)])], generated_at=stale)))
+ assert stale not in message
+ fresh = (datetime.now(timezone.utc) - timedelta(hours=5)).isoformat()
+ assert _read(_write(tmp_path, _envelope([_call("topic", posts=[_row(0)])], generated_at=fresh), "ok.json")).accepted == 1
+
+ def test_mismatched_topic_exits_2(self, tmp_path):
+ message = _assert_contract(
+ _write(tmp_path, _envelope([_call("topic", posts=[_row(0)])], topic="rust async SECRETTOPIC")),
+ must_not_contain=["SECRETTOPIC"],
+ )
+ assert "topic" in message
+ # Normalization: case and whitespace do not count as a mismatch.
+ assert _read(_write(tmp_path, _envelope([_call("topic", posts=[_row(0)])], topic=" AI Agents "), "ok.json")).accepted == 1
+
+ def test_window_entirely_before_engine_window_exits_2(self, tmp_path):
+ early_from = (date.fromisoformat(FROM) - timedelta(days=40)).isoformat()
+ early_to = (date.fromisoformat(FROM) - timedelta(days=10)).isoformat()
+ payload = _envelope([_call("topic", posts=[_row(0)])], window={"from": early_from, "to": early_to})
+ _assert_contract(_write(tmp_path, payload))
+
+ def test_narrower_window_is_a_receipt_warning_not_a_failure(self, tmp_path):
+ narrow_from = (date.fromisoformat(FROM) + timedelta(days=10)).isoformat()
+ payload = _envelope([_call("topic", posts=[_row(0)])], window={"from": narrow_from, "to": TO})
+ envelope, stderr = _capture(lambda: _read(_write(tmp_path, payload)))
+ assert envelope.accepted == 1
+ assert envelope.warnings and "window" in envelope.warnings[0]
+ assert "window" in stderr
+ report = _run(envelope)
+ assert any("window" in w for w in report.warnings)
+
+
+# ---------------------------------------------------------------------------
+# Envelope status classes (AE6b, R11)
+# ---------------------------------------------------------------------------
+
+
+class TestStatusOutcomes:
+ def test_error_with_account_identifiers_is_payment_required_with_fixed_detail(self, tmp_path):
+ payload = _envelope([], status="error", error=f"credits exhausted for account {ACCOUNT_ID}")
+ envelope, stderr = _capture(lambda: _read(_write(tmp_path, payload)))
+ assert ACCOUNT_ID not in stderr
+ report, run_stderr = _capture(lambda: _run(envelope))
+ outcome = report.source_status["x"]
+ assert outcome.state == schema.PAYMENT_REQUIRED
+ assert outcome.detail == x_envelope.DETAIL_CREDITS
+ assert ACCOUNT_ID not in run_stderr
+ for rendered in (
+ render.render_compact(report),
+ html_render.render_html(report),
+ json.dumps(schema.to_dict(report)),
+ json.dumps(schema.to_agent_export(report)),
+ render.render_context(report),
+ render.render_brief(report),
+ ):
+ assert ACCOUNT_ID not in rendered
+
+ @pytest.mark.parametrize("category,state,detail", [
+ ("not-connected", health.ERROR, x_envelope.DETAIL_NOT_CONNECTED),
+ ("unavailable", health.ERROR, x_envelope.DETAIL_UNAVAILABLE),
+ ("something-weird", health.ERROR, x_envelope.DETAIL_ERROR),
+ ("credits", schema.PAYMENT_REQUIRED, x_envelope.DETAIL_CREDITS),
+ ])
+ def test_error_categories_map_to_fixed_details(self, tmp_path, category, state, detail):
+ envelope = _read(_write(tmp_path, _envelope([], status="error", error=category)))
+ report = _run(envelope)
+ assert report.source_status["x"].state == state
+ assert report.source_status["x"].detail == detail
+
+ def test_raw_error_text_reaches_stderr_only_under_debug(self, tmp_path, monkeypatch):
+ payload = _envelope([], status="error", error=f"unavailable {ACCOUNT_ID}")
+ monkeypatch.delenv("LAST30DAYS_DEBUG", raising=False)
+ _, quiet = _capture(lambda: _read(_write(tmp_path, payload)))
+ assert ACCOUNT_ID not in quiet
+ monkeypatch.setenv("LAST30DAYS_DEBUG", "1")
+ _, loud = _capture(lambda: _read(_write(tmp_path, payload, "b.json")))
+ assert ACCOUNT_ID in loud
+
+ def test_ok_with_zero_rows_is_no_results_and_omission_note_does_not_fire(self, tmp_path):
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[])])))
+ report = _run(envelope)
+ assert report.source_status["x"].state == schema.NO_RESULTS
+ assert "x" not in report.errors_by_source
+ diag = pipeline.diagnose({}, None, safe=True, x_envelope=True)
+ assert "x" in diag["available_sources"]
+ assert cli._optional_x_omission_text(diag, None) is None
+
+ def test_partial_with_window_unsupported_names_the_unwindowed_call(self, tmp_path):
+ payload = _envelope([
+ _call("topic", posts=[_row(0, "alice", "ai agents review from alice")]),
+ _call("from", handles=[SUBJECT], posts=[_row(2, SUBJECT, "shipping my agent today")]),
+ ], status="partial", error="window-unsupported")
+ envelope, stderr = _capture(lambda: _read(_write(tmp_path, payload)))
+ assert "window-unsupported" in stderr
+ report = _run(envelope, x_handle=SUBJECT)
+ outcome = report.source_status["x"]
+ assert outcome.state == schema.PARTIAL
+ assert "window-unsupported" in (outcome.detail or "")
+ assert "topic" in outcome.detail and "from" in outcome.detail
+ assert len(report.items_by_source["x"]) == 2
+
+
+# ---------------------------------------------------------------------------
+# Lane metadata (R11)
+# ---------------------------------------------------------------------------
+
+
+class TestLanes:
+ def test_from_row_by_foreign_author_is_lane_mismatch(self, tmp_path):
+ envelope = _read(_write(tmp_path, _envelope([
+ _call("from", handles=[SUBJECT], posts=[_row(0, "impostor", "hi"), _row(2, SUBJECT, "mine")]),
+ ])))
+ assert envelope.counters["lane-mismatch"] == 1
+ assert envelope.lane_counts["from"] == 1
+
+ def test_related_lane_claiming_the_primary_handle_is_served_as_topic(self, tmp_path):
+ """The related lane is narrowed to --x-related: a --x-handle handle is
+ allowed for from/mention but not for related."""
+ envelope = _read(
+ _write(tmp_path, _envelope([_call("related", handles=[SUBJECT], posts=[_row(0, SUBJECT, "x")])])),
+ handles=[SUBJECT], related=[RELATED],
+ )
+ assert envelope.lane_counts["related"] == 0
+ assert envelope.lane_counts["topic"] == 1
+ assert envelope.counters["lane-mismatch"] == 1
+ ok = _read(
+ _write(tmp_path, _envelope([_call("related", handles=[RELATED], posts=[_row(0, RELATED, "x")])]), "ok.json"),
+ handles=[SUBJECT], related=[RELATED],
+ )
+ assert ok.lane_counts["related"] == 1
+
+ def test_related_handle_absent_from_x_related_is_served_as_topic(self, tmp_path):
+ envelope, stderr = _capture(lambda: _read(
+ _write(tmp_path, _envelope([_call("related", handles=["stranger"], posts=[_row(0, "stranger", "x")])])),
+ handles=[SUBJECT], related=[RELATED],
+ ))
+ assert envelope.lane_counts["related"] == 0
+ assert envelope.lane_counts["topic"] == 1
+ assert envelope.counters["lane-mismatch"] == 1
+ assert "topic" in stderr and "stranger" not in stderr
+
+ def test_handle_outside_grammar_demotes_the_call(self, tmp_path):
+ envelope = _read(
+ _write(tmp_path, _envelope([_call("from", handles=["bad handle!"], posts=[_row(0, SUBJECT, "x")])])),
+ handles=[SUBJECT],
+ )
+ assert envelope.lane_counts["topic"] == 1
+ assert not envelope.lane_calls
+
+ def test_mention_lane_drops_the_subjects_own_post(self, tmp_path):
+ envelope = _read(_write(tmp_path, _envelope([
+ _call("mention", handles=[SUBJECT], posts=[
+ _row(0, SUBJECT, f"@{SUBJECT} talking to myself"),
+ _row(1, "fan", f"@{SUBJECT} love the work"),
+ ]),
+ ])))
+ assert envelope.lane_counts["mention"] == 1
+ assert envelope.counters["lane-mismatch"] == 1
+
+ def test_related_rows_get_the_related_weight_and_first_party(self, tmp_path):
+ envelope = _read(_write(tmp_path, _envelope([
+ _call("topic", posts=[_row(0, "alice", "ai agents review")]),
+ _call("related", handles=[RELATED], posts=[_row(3, RELATED, "peer news")]),
+ ])), handles=[SUBJECT], related=[RELATED])
+ report = _run(envelope, x_handle=SUBJECT, x_related=[RELATED])
+ related_sq = [sq for sq in report.query_plan.subqueries if sq.label == "supplemental-related"]
+ assert related_sq and related_sq[0].weight == 0.3
+ assert RELATED in {item.author for item in report.items_by_source["x"]}
+
+ def test_from_lane_respects_per_handle_count(self, tmp_path):
+ subjects = (
+ "shipping the agent runtime", "benchmarks for tool calling", "memory layer rewrite",
+ "why evals matter", "latency budget notes", "open sourcing the planner",
+ "hiring for infra", "conference talk recap", "pricing update", "roadmap thread",
+ )
+ rows = [_row(i, SUBJECT, subjects[i]) for i in range(pipeline.FROM_LANE_COUNT_PER + 2)]
+ # Vary ids beyond the offset cycle so the sequence never reads as generated.
+ for i, row in enumerate(rows):
+ row["id"] = _snowflake(_day(1 + (i * 3) % 25), hour=i % 24, minute=(i * 13) % 60, low=i * 7919)
+ row["created_at"] = None
+ row.pop("created_at")
+ envelope = _read(_write(tmp_path, _envelope([_call("from", handles=[SUBJECT], posts=rows)])))
+ assert envelope.accepted == len(rows)
+ report = _run(envelope, x_handle=SUBJECT)
+ assert len(report.items_by_source["x"]) == pipeline.FROM_LANE_COUNT_PER
+
+ def test_extracted_handle_promotion_is_skipped(self, tmp_path):
+ envelope = _read(_basic(tmp_path))
+ with mock.patch("lib.x_judge.promotable_handles", side_effect=AssertionError("no promotion")):
+ report = _run(envelope, x_handle=SUBJECT)
+ assert report.items_by_source["x"]
+
+
+# ---------------------------------------------------------------------------
+# Single-serve and pipeline wiring (KTD5, KTD11, R12)
+# ---------------------------------------------------------------------------
+
+
+class TestPipelineWiring:
+ def test_second_planner_x_subquery_and_thin_retry_get_nothing(self, tmp_path):
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[_row(0, "alice", "ai agents review")])])))
+ calls: list[str] = []
+ original = pipeline._retrieve_stream_impl
+
+ def spy(*args, **kwargs):
+ if kwargs.get("source") == "x":
+ calls.append(kwargs["subquery"].label)
+ return original(*args, **kwargs)
+
+ with mock.patch("lib.pipeline._retrieve_stream_impl", side_effect=spy):
+ report = _run(envelope, plan=_two_x_plan(), config={"_max_source_fetches": 5})
+ assert "primary" in calls and "angle" in calls and "retry" in calls
+ assert len(report.items_by_source["x"]) == 1
+ assert "x" not in report.errors_by_source
+ assert report.source_status["x"].state == health.OK
+
+ def test_available_sources_lists_x_for_envelope_without_backend(self):
+ with mock.patch("lib.env.x_backend_chain", return_value=[]), \
+ mock.patch("lib.env.x_pending_browser_auth", return_value=False):
+ assert "x" in pipeline.available_sources({}, None, x_envelope=True)
+ assert "x" not in pipeline.available_sources({}, None)
+
+ def test_envelope_without_lane_signal_and_no_backend_lands_under_x(self, tmp_path, monkeypatch):
+ monkeypatch.delenv("LAST30DAYS_X_HOST_LANE", raising=False)
+ envelope = _read(_basic(tmp_path))
+ report = _run(envelope, x_handle=SUBJECT, requested=None)
+ assert len(report.items_by_source["x"]) == 3
+ assert "x" in report.query_plan.source_weights or report.items_by_source["x"]
+
+ def test_lane_signal_without_envelope_records_not_passed(self):
+ config = {"LAST30DAYS_X_HOST_LANE": "1"}
+ with mock.patch("lib.env.x_backend_chain", return_value=[]), \
+ mock.patch("lib.env.x_pending_browser_auth", return_value=False):
+ assert "x" in pipeline.available_sources(config, None)
+ with _no_backend() as fetch:
+ report = pipeline.run(
+ topic=TOPIC, config=config, depth="default", requested_sources=["x"],
+ mock=False, external_plan=_plan(), web_backend="none", save_dir="",
+ )
+ fetch.assert_not_called()
+ assert report.source_status["x"].state == health.ERROR
+ assert report.source_status["x"].detail == x_envelope.DETAIL_NOT_PASSED
+
+ def test_lane_signal_with_bearer_still_records_not_passed(self):
+ config = {"LAST30DAYS_X_HOST_LANE": "1", "X_BEARER_TOKEN": "dummy-bearer"}
+ with mock.patch("lib.env.x_backend_chain", return_value=["xapi"]), \
+ mock.patch("lib.pipeline._fetch_x_backend", side_effect=AssertionError("must not fetch")), \
+ mock.patch("lib.x_api.search_handles", side_effect=AssertionError("no lanes")):
+ report = pipeline.run(
+ topic=TOPIC, config=config, depth="default", requested_sources=["x"],
+ mock=False, x_handle=SUBJECT, external_plan=_plan(), web_backend="none", save_dir="",
+ )
+ assert report.source_status["x"].state == health.ERROR
+ assert report.source_status["x"].detail == x_envelope.DETAIL_NOT_PASSED
+
+ def test_discovery_enrichment_pass_with_signal_records_nothing_for_x(self):
+ config = {"LAST30DAYS_X_HOST_LANE": "1"}
+ with mock.patch("lib.env.x_backend_chain", return_value=[]), \
+ mock.patch("lib.env.x_pending_browser_auth", return_value=False):
+ assert "x" not in pipeline.available_sources(config, None, suppress_x_host_lane=True)
+ with _no_backend(), mock.patch("lib.pipeline._retrieve_stream", return_value=([], {})):
+ report = pipeline.run(
+ topic=TOPIC, config=config, depth="default", requested_sources=None,
+ mock=False, external_plan=_plan(("reddit", "x")), web_backend="none", save_dir="",
+ internal_subrun=True, suppress_x_host_lane=True,
+ )
+ assert "x" not in report.errors_by_source
+ assert "x" not in report.source_status
+
+ def test_comparison_entity_pass_with_envelope_still_serves_x(self, tmp_path):
+ envelope = _read(_basic(tmp_path, topic="acme"), topic="acme")
+ report = _run(envelope, topic="acme", x_handle=SUBJECT, internal_subrun=True,
+ config={"LAST30DAYS_X_HOST_LANE": "1"})
+ assert len(report.items_by_source["x"]) == 3
+
+ def test_exclude_sources_x_ignores_the_envelope_with_a_receipt(self, tmp_path):
+ envelope = _read(_basic(tmp_path))
+ report, stderr = _capture(lambda: _run(
+ envelope, config={"EXCLUDE_SOURCES": "x"}, requested=None, plan=_plan(("reddit",)),
+ ))
+ assert "x" not in report.items_by_source
+ assert "envelope ignored" in stderr
+ assert envelope.topic_items, "an ignored envelope is not consumed"
+
+ def test_search_list_without_x_ignores_the_envelope(self, tmp_path):
+ envelope = _read(_basic(tmp_path))
+ with mock.patch("lib.pipeline._retrieve_stream", return_value=([], {})):
+ report, stderr = _capture(lambda: _run(envelope, requested=("reddit",), plan=_plan(("reddit",))))
+ assert "x" not in report.items_by_source
+ assert "envelope ignored" in stderr
+
+ def test_mixed_fixture_fuses_duplicate_url_into_one_candidate(self, tmp_path):
+ row = _row(0, "alice", "ai agents review")
+ envelope = _read(_write(tmp_path, _envelope([_call("topic", posts=[row])])))
+ url = f"https://x.com/alice/status/{row['id']}"
+ # Another source (HN) hands the engine the same post URL: fusion keys
+ # on the normalized URL, so the two copies become one candidate.
+ original = pipeline._retrieve_stream
+
+ def stream(*args, **kwargs):
+ if kwargs.get("source") == "hackernews":
+ return [{
+ "id": "hn1", "title": "ai agents review", "url": url,
+ "points": 40, "num_comments": 3, "date": _day(_OFFSETS[0]),
+ "author": "hnuser", "text": "ai agents review thread",
+ }], {}
+ return original(*args, **kwargs)
+
+ with mock.patch("lib.pipeline._retrieve_stream", side_effect=stream):
+ report = _run(envelope, plan=_plan(("x", "hackernews")), requested=("x", "hackernews"))
+ assert report.items_by_source["x"] and report.items_by_source["hackernews"]
+ candidates = [c for c in report.ranked_candidates if c.url == url]
+ assert len(candidates) == 1
+
+ def test_footer_provenance_reads_via_x_connector(self, tmp_path):
+ envelope = _read(_basic(tmp_path))
+ report = _run(envelope, x_handle=SUBJECT)
+ md = render.render_compact(report)
+ assert "via X connector" in md
+ assert "3 items" in md
+
+ def test_env_file_lane_line_without_process_env_leaves_x_absent(self, tmp_path, monkeypatch):
+ monkeypatch.delenv("LAST30DAYS_X_HOST_LANE", raising=False)
+ config_file = tmp_path / ".env"
+ config_file.write_text("LAST30DAYS_X_HOST_LANE=1\n", encoding="utf-8")
+ config_file.chmod(0o600)
+ monkeypatch.setenv("LAST30DAYS_CONFIG_DIR", str(tmp_path))
+ monkeypatch.setattr(env, "CONFIG_DIR", tmp_path)
+ monkeypatch.setattr(env, "CONFIG_FILE", config_file)
+ for key in ("LAST30DAYS_HOST", "X_BEARER_TOKEN", "LAST30DAYS_X_BACKEND", "XAI_API_KEY",
+ "AUTH_TOKEN", "CT0", "XQUIK_API_KEY", "FROM_BROWSER", "AGENTCOOKIE", "BROWSER_CDP_URL"):
+ monkeypatch.delenv(key, raising=False)
+ with mock.patch.object(env, "_load_keychain", return_value={}), \
+ mock.patch.object(env, "_load_pass", return_value={}), \
+ mock.patch.object(env, "_find_project_env", return_value=None):
+ config = env.get_config()
+ with mock.patch("lib.env.x_backend_chain", return_value=[]), \
+ mock.patch("lib.env.x_pending_browser_auth", return_value=False):
+ assert "x" not in pipeline.available_sources(config, None)
+
+ def test_diagnose_top_keys_unchanged(self):
+ from tests.test_diagnose_compat import DIAGNOSE_TOP_KEYS
+ with mock.patch("lib.env.x_backend_chain", return_value=[]):
+ payload = pipeline.diagnose({}, None, safe=True, x_envelope=True)
+ assert set(payload.keys()) == DIAGNOSE_TOP_KEYS
+ assert "x" in payload["available_sources"]
+
+
+# ---------------------------------------------------------------------------
+# CLI (R9, KTD5)
+# ---------------------------------------------------------------------------
+
+
+_DIAG = {
+ "available_sources": ["reddit", "x"],
+ "x_pending_browser_auth": False,
+ "native_search": False,
+ "bird_installed": False,
+ "bird_authenticated": False,
+ "bird_username": None,
+ "native_web_backend": None,
+ "safe": False,
+}
+
+
+def _cli(argv, tmp_path, *, config=None, run=None, environ=None, real_run=False):
+ out, err = io.StringIO(), io.StringIO()
+ cfg_dir = tmp_path / "cfg"
+ cfg_dir.mkdir(exist_ok=True)
+ stack = contextlib.ExitStack()
+ with stack:
+ stack.enter_context(mock.patch.object(cli.env, "get_config", return_value=dict(config or {})))
+ stack.enter_context(mock.patch.object(cli.env, "CONFIG_DIR", cfg_dir))
+ stack.enter_context(mock.patch.object(cli.env, "CONFIG_FILE", cfg_dir / ".env"))
+ stack.enter_context(mock.patch.object(cli.ui, "ProgressDisplay", return_value=mock.Mock()))
+ stack.enter_context(mock.patch.dict(
+ os.environ, {"LAST30DAYS_SKIP_PREFLIGHT": "1", **(environ or {})}, clear=False,
+ ))
+ stack.enter_context(mock.patch.object(sys, "argv", ["last30days.py", *argv]))
+ if real_run:
+ stack.enter_context(_no_backend())
+ else:
+ stack.enter_context(mock.patch.object(cli.pipeline, "diagnose", return_value=dict(_DIAG)))
+ stack.enter_context(mock.patch.object(
+ cli.pipeline, "run", side_effect=run or AssertionError("pipeline.run must not run"),
+ ))
+ stack.enter_context(redirect_stdout(out))
+ stack.enter_context(redirect_stderr(err))
+ rc = cli.main()
+ return rc, out.getvalue(), err.getvalue()
+
+
+def _fake_run_capturing(store: dict):
+ def fake_run(**kwargs):
+ store.update(kwargs)
+ return _fake_report(kwargs["topic"])
+ return fake_run
+
+
+def _fake_report(topic: str) -> schema.Report:
+ return schema.Report(
+ topic=topic, range_from=FROM, range_to=TO, generated_at="2026-09-08T00:00:00+00:00",
+ provider_runtime=schema.ProviderRuntime(reasoning_provider="local", planner_model="d", rerank_model="l"),
+ query_plan=schema.QueryPlan(intent="general", freshness_mode="balanced_recent", cluster_mode="story",
+ raw_topic=topic, subqueries=[], source_weights={}),
+ clusters=[], ranked_candidates=[], items_by_source={}, errors_by_source={},
+ )
+
+
+class TestCli:
+ def test_flag_is_documented_in_argparse_help(self):
+ flags = {opt for action in cli.build_parser()._actions for opt in action.option_strings}
+ assert "--x-posts" in flags
+
+ def test_inline_json_exits_2(self, tmp_path):
+ rc, _, err = _cli([TOPIC, "--x-posts", '{"schema": "SECRETVALUE"}'], tmp_path)
+ assert rc == 2
+ assert "path" in err and "SECRETVALUE" not in err
+
+ def test_contract_error_exits_2_naming_the_path(self, tmp_path):
+ path = _write(tmp_path, _envelope([_call("topic", posts=[_row(0)])], status="great"))
+ rc, _, err = _cli([TOPIC, "--x-posts", path], tmp_path)
+ assert rc == 2
+ assert path in err and "great" not in err and "--x-posts" in err
+
+ def test_hosted_mode_with_flag_exits_2(self, tmp_path):
+ path = _basic(tmp_path)
+ with mock.patch.object(cli.env, "read_secret_env", return_value="hosted-test-key"), \
+ mock.patch("lib.hosted.run_hosted", side_effect=AssertionError("hosted must not run")):
+ rc, _, err = _cli(
+ [TOPIC, "--x-posts", path], tmp_path,
+ environ={"LAST30DAYS_API_BASE": "https://hosted.example.test"},
+ )
+ assert rc == 2
+ assert "--x-posts" in err
+
+ def test_valid_flag_threads_the_envelope_into_run(self, tmp_path):
+ path = _basic(tmp_path)
+ store: dict = {}
+ rc, _, _ = _cli([TOPIC, "--x-posts", path, "--x-handle", SUBJECT], tmp_path, run=_fake_run_capturing(store))
+ assert rc == 0
+ envelope = store["x_posts"]
+ assert isinstance(envelope, x_envelope.Envelope)
+ assert envelope.accepted == 3
+
+ def test_bare_flag_on_comparison_run_exits_2_naming_the_field(self, tmp_path):
+ path = _basic(tmp_path)
+ rc, _, err = _cli(["acme vs globex", "--x-posts", path], tmp_path)
+ assert rc == 2
+ assert "x_posts" in err and "--competitors-plan" in err
+
+ def test_per_entity_x_posts_in_competitors_plan(self, tmp_path):
+ acme = _write(tmp_path, _envelope([_call("topic", posts=[_row(0, "a", "acme news")])], topic="acme"), "acme.json")
+ globex = _write(tmp_path, _envelope([_call("topic", posts=[_row(1, "b", "globex news")])], topic="globex"), "globex.json")
+ plan = json.dumps({"acme": {"x_posts": acme}, "globex": {"x_posts": globex}})
+ seen: dict[str, dict] = {}
+
+ def fake_run(**kwargs):
+ seen[kwargs["topic"]] = kwargs
+ return _fake_report(kwargs["topic"])
+
+ with mock.patch.object(cli, "emit_comparison_output", return_value="# rendered"):
+ rc, _, err = _cli(["acme vs globex", "--competitors-plan", plan], tmp_path, run=fake_run)
+ assert rc == 0, err
+ assert seen["acme"]["x_posts"].sha256 == hashlib.sha256(Path(acme).read_bytes()).hexdigest()
+ assert seen["globex"]["x_posts"].sha256 == hashlib.sha256(Path(globex).read_bytes()).hexdigest()
+ parsed = cli.parse_competitors_plan(plan)
+ assert parsed["acme"]["x_posts"] == acme
+
+ def test_comparison_pass_rejects_the_other_entitys_envelope(self, tmp_path):
+ acme = _write(tmp_path, _envelope([_call("topic", posts=[_row(0, "a", "acme news")])], topic="acme"), "acme.json")
+ plan = json.dumps({"globex": {"x_posts": acme}})
+ rc, _, err = _cli(["acme vs globex", "--competitors-plan", plan], tmp_path)
+ assert rc == 2
+ assert acme in err and "topic" in err
+
+ def test_comparison_cache_is_reused_only_with_validated_entity_envelopes(self, tmp_path):
+ """The lookup digest is built from the validated per-entity envelopes
+ (same fold as the write side), and a stale entity envelope fails the
+ run closed (exit 2) before any cached output is served."""
+ acme = _write(tmp_path, _envelope([_call("topic", posts=[_row(0, "a", "acme news")])], topic="acme"), "acme.json")
+ globex = _write(tmp_path, _envelope([_call("topic", posts=[_row(1, "b", "globex news")])], topic="globex"), "globex.json")
+ plan = json.dumps({"acme": {"x_posts": acme}, "globex": {"x_posts": globex}})
+ comp_plan = cli.parse_competitors_plan(plan)
+ cli._attach_entity_envelopes(comp_plan, mock.Mock(lookback_days=30, as_of_date=None))
+ digest = cli._x_envelope_digest(None, comp_plan)
+ assert digest and digest != comp_plan["acme"]["_x_envelope"].sha256
+ synth = tmp_path / "synth.md"
+ synth.write_text("# synthesis\n")
+ cfg_dir = tmp_path / "cfg"
+ cfg_dir.mkdir(exist_ok=True)
+ entity_reports = [("acme", _fake_report("acme")), ("globex", _fake_report("globex"))]
+ with mock.patch.object(cli.env, "CONFIG_DIR", cfg_dir):
+ assert cli._write_last_run("acme vs globex", entity_reports[0][1], entity_reports, x_envelope_sha256=digest)
+ argv = ["acme vs globex", "--competitors-plan", plan, "--emit", "html", "--synthesis-file", str(synth)]
+ with mock.patch.object(cli, "_render_save_and_print", return_value=0) as render:
+ rc, _, err = _cli(argv, tmp_path)
+ assert rc == 0, err
+ assert "Reusing cached report data" in err
+ render.assert_called_once()
+ # Same plan, but acme's envelope went stale: fail closed before the cache.
+ stale = (datetime.now(timezone.utc) - timedelta(hours=7)).isoformat()
+ _write(tmp_path, _envelope([_call("topic", posts=[_row(0, "a", "acme news")])], topic="acme", generated_at=stale), "acme.json")
+ with mock.patch.object(cli, "_render_save_and_print", side_effect=AssertionError("must not render")):
+ rc, _, err = _cli(argv, tmp_path)
+ assert rc == 2
+ assert "generated_at" in err and "Reusing cached" not in err
+ main = _read(_basic(tmp_path))
+ assert cli._x_envelope_digest(main, None) == main.sha256
+
+ def test_last_report_cache_misses_on_digest_mismatch(self, tmp_path):
+ path = _basic(tmp_path)
+ envelope = _read(path)
+ report = _fake_report(TOPIC)
+ with mock.patch.object(cli.env, "CONFIG_DIR", tmp_path / "cfg"):
+ (tmp_path / "cfg").mkdir(exist_ok=True)
+ assert cli._write_last_run(TOPIC, report, x_envelope_sha256=envelope.sha256)
+ payload = json.loads((tmp_path / "cfg" / "last-report.json").read_text())
+ assert payload["x_envelope_sha256"] == envelope.sha256
+ assert cli._load_last_report_cache(TOPIC, x_envelope_sha256=envelope.sha256) is not None
+ assert cli._load_last_report_cache(TOPIC, x_envelope_sha256="0" * 64) is None
+ assert cli._load_last_report_cache(TOPIC) is None
+ assert cli._write_last_run(TOPIC, report)
+ assert cli._load_last_report_cache(TOPIC) is not None
+ assert cli._load_last_report_cache(TOPIC, x_envelope_sha256=envelope.sha256) is None
+
+ def test_cli_run_end_to_end_emits_x_citations_only(self, tmp_path):
+ path = _basic(tmp_path)
+ argv = [TOPIC, "--x-posts", path, "--x-handle", SUBJECT, "--search", "x", "--web-backend", "none"]
+ rc, out, err = _cli([*argv, "--emit", "md"], tmp_path, real_run=True)
+ assert rc == 0, err
+ assert "host-fetched X: accepted 3 of 3" in err
+ assert "via X connector" in out
+ assert "3 items" in out
+ citations = re.findall(r"https?://[^\s)\]]+/status/\d+", out)
+ assert len(citations) >= 3 and all(c.startswith("https://x.com/") for c in citations)
+ assert "Optional source omitted" not in out + err
+ rc, html, err = _cli([*argv, "--emit", "html"], tmp_path, real_run=True)
+ assert rc == 0, err
+ assert all(h.startswith("https://x.com/") for h in _hrefs(html) if "/status/" in h)
+ assert "Optional source omitted" not in html + err
diff --git a/tests/test_x_extras_gating.py b/tests/test_x_extras_gating.py
index e4070c91..ac8d02b0 100644
--- a/tests/test_x_extras_gating.py
+++ b/tests/test_x_extras_gating.py
@@ -191,3 +191,113 @@ def test_cdp_used_when_agentcookie_empty_on_extra_host():
env._discover_and_apply_x_credentials(config)
assert config["AUTH_TOKEN"] == "test-auth-token"
assert config["_AUTH_TOKEN_SOURCE"] == "chrome cdp"
+
+
+# --- AE1: Grok Bot host, cookies present, no pin ---------------------------
+
+
+def test_ae1_grok_bot_blocks_scraper_xquik_and_every_cookie_leg():
+ """LAST30DAYS_HOST=grok-bot on Linux with every cookie source armed: the
+ chain has neither bird nor xquik, no sidecar / CDP / browser-store read
+ is made for any cookie domain, and the scraper is never primed."""
+ config = {
+ "LAST30DAYS_HOST": "grok-bot",
+ "FROM_BROWSER": "firefox",
+ "AGENTCOOKIE": "on",
+ "BROWSER_CDP_URL": "http://127.0.0.1:18800",
+ "AUTH_TOKEN": "test-auth-token",
+ "CT0": "test-ct0",
+ "XQUIK_API_KEY": "dummy-xquik-key",
+ }
+ with (
+ mock.patch("platform.system", return_value="Linux"),
+ mock.patch("shutil.which", return_value="/usr/local/bin/agentcookie"),
+ mock.patch("lib.agentcookie.read_x_cookies", side_effect=AssertionError("no sidecar on grok-bot")),
+ mock.patch("lib.chrome_cdp.read_x_cookies", side_effect=AssertionError("no CDP on grok-bot")),
+ mock.patch("lib.cookie_extract.extract_cookies", side_effect=AssertionError("no browser store read on grok-bot")),
+ mock.patch("lib.bird_x.set_credentials", side_effect=AssertionError("scraper must not be primed on grok-bot")),
+ mock.patch("lib.bird_x.is_bird_installed", return_value=True),
+ mock.patch("lib.xurl_x.is_available", return_value=False),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=False),
+ ):
+ env._discover_and_apply_x_credentials(config)
+ assert env.cookie_extraction_browsers(config) == []
+ chain = env.x_backend_chain(config)
+ assert "bird" not in chain
+ assert "xquik" not in chain
+ assert chain == []
+ assert env.get_x_source(config) is None
+ # Preflight reports browser cookies off: no browser resolves.
+ config["_BROWSER_COOKIE_MODE"] = "plan_only"
+ assert env.x_pending_browser_auth(config) is False
+ # X becomes available only through an official path.
+ assert env.x_backend_chain({**config, "X_BEARER_TOKEN": "dummy-bearer"}) == ["xapi"]
+ assert env.x_backend_chain({**config, "XAI_API_KEY": "dummy-xai-key"}) == ["xai"]
+
+
+def test_ae1_get_config_read_mode_on_grok_bot_reports_no_browsers(tmp_path, monkeypatch):
+ """The resolved config's _BROWSER_COOKIE_BROWSERS is empty on a Grok Bot
+ host even with FROM_BROWSER set, and read mode performs no cookie leg."""
+ monkeypatch.setenv("LAST30DAYS_CONFIG_DIR", str(tmp_path))
+ monkeypatch.setattr(env, "CONFIG_DIR", tmp_path)
+ monkeypatch.setattr(env, "CONFIG_FILE", tmp_path / "does-not-exist.env")
+ monkeypatch.chdir(tmp_path)
+ monkeypatch.setenv("LAST30DAYS_HOST", "grok-bot")
+ monkeypatch.setenv("FROM_BROWSER", "firefox")
+ monkeypatch.setenv("AGENTCOOKIE", "on")
+ monkeypatch.delenv("LAST30DAYS_X_BACKEND", raising=False)
+ with (
+ mock.patch.object(env, "_load_keychain", return_value={}),
+ mock.patch.object(env, "_load_pass", return_value={}),
+ mock.patch.object(env, "_find_project_env", return_value=None),
+ mock.patch("platform.system", return_value="Linux"),
+ mock.patch("lib.agentcookie.read_x_cookies", side_effect=AssertionError("no sidecar on grok-bot")),
+ mock.patch("lib.chrome_cdp.read_x_cookies", side_effect=AssertionError("no CDP on grok-bot")),
+ mock.patch("lib.cookie_extract.extract_cookies", side_effect=AssertionError("no browser store read on grok-bot")),
+ ):
+ config = env.get_config(env.ConfigLoadPolicy(browser_cookies="read"))
+ assert config["_BROWSER_COOKIE_BROWSERS"] == []
+ assert config["_BROWSER_COOKIE_MODE"] == "read"
+ assert config.get("_AUTH_TOKEN_SOURCE") is None
+
+
+# --- AE7a: ambient bearer on a non-Grok host never reaches xapi ------------
+
+
+def test_ae7a_non_grok_host_ambient_bearer_never_reaches_xapi():
+ config = {
+ "AUTH_TOKEN": "test-auth-token",
+ "CT0": "test-ct0",
+ "X_BEARER_TOKEN": "dummy-bearer",
+ }
+ with (
+ mock.patch("platform.system", return_value="Linux"),
+ mock.patch("lib.http.get", side_effect=AssertionError("no X API request off Grok Bot")),
+ _stub_backends(),
+ ):
+ chain = env.x_backend_chain(config)
+ # The X loop fails over only along this chain: bird returning nothing
+ # leaves no next rung, so no X API request is possible.
+ assert chain == ["bird"]
+ assert "xapi" not in chain
+ with _stub_backends():
+ assert env.x_backend_chain({**config, "LAST30DAYS_X_BACKEND": "xapi"}) == ["xapi"]
+
+
+# --- AE8: Cursor agent chat without the host key is unchanged --------------
+
+
+def test_ae8_cursor_agent_without_host_key_is_unchanged():
+ config = {"AGENTCOOKIE": "on"}
+ with (
+ mock.patch.dict("os.environ", {"CURSOR_AGENT": "1"}, clear=False),
+ mock.patch("platform.system", return_value="Linux"),
+ mock.patch("lib.agentcookie.read_x_cookies", return_value=dict(_PAIR)) as sidecar,
+ mock.patch("lib.chrome_cdp.read_x_cookies", return_value=None),
+ _no_extract(),
+ ):
+ env._discover_and_apply_x_credentials(config)
+ assert sidecar.called
+ assert config["_AUTH_TOKEN_SOURCE"] == "agentcookie"
+ with _stub_backends():
+ assert env.x_backend_chain(config) == ["bird"]
diff --git a/tests/test_x_policy.py b/tests/test_x_policy.py
new file mode 100644
index 00000000..5edc025f
--- /dev/null
+++ b/tests/test_x_policy.py
@@ -0,0 +1,662 @@
+"""U1: host signal, X policy helper, and chain registration.
+
+One helper in lib/env.py (``x_policy``) owns the official-only rule keyed on
+``LAST30DAYS_HOST=grok-bot``. It shapes the unpinned chain and cookie
+discovery; the ``LAST30DAYS_X_BACKEND`` pin keeps its exclusive semantics on
+every host and may name any known backend. ``LAST30DAYS_X_HOST_LANE`` is the
+per-session lane signal read from the process environment only (a ``.env``
+line is ignored), and ``pipeline.available_sources`` lists X for the lane or
+a validated envelope in every cookie mode.
+
+Only obvious dummy values are used for every credential.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import os
+import re
+from pathlib import Path
+from unittest import mock
+
+import pytest
+
+from lib import backends, env, health, pipeline
+
+ROOT = Path(__file__).resolve().parents[1]
+LIB_DIR = ROOT / "skills" / "last30days" / "scripts" / "lib"
+
+_PAIR = {"auth_token": "test-auth-token", "ct0": "test-ct0"}
+_COOKIES = {"AUTH_TOKEN": "test-auth-token", "CT0": "test-ct0"}
+
+
+def _grok_bot(**over):
+ cfg = {"LAST30DAYS_HOST": "grok-bot"}
+ cfg.update(over)
+ return cfg
+
+
+def _stub_backends(
+ *,
+ bird_installed=True,
+ xurl_available=False,
+ xurl_stored=False,
+ grok_authed=False,
+):
+ """Local-only backend probes so chain resolution touches no network."""
+ stack = contextlib.ExitStack()
+ stack.enter_context(mock.patch("lib.bird_x.is_bird_installed", return_value=bird_installed))
+ stack.enter_context(mock.patch("lib.xurl_x.is_available", return_value=xurl_available))
+ stack.enter_context(mock.patch("lib.xurl_x.has_stored_auth", return_value=xurl_stored))
+ stack.enter_context(mock.patch("lib.grok_x.has_stored_auth", return_value=grok_authed))
+ return stack
+
+
+def _no_cookie_reads():
+ """Every cookie leg raises: the policy must return before any of them."""
+ stack = contextlib.ExitStack()
+ for target in (
+ "lib.agentcookie.read_x_cookies",
+ "lib.chrome_cdp.read_x_cookies",
+ "lib.cookie_extract.extract_cookies",
+ ):
+ stack.enter_context(
+ mock.patch(target, side_effect=AssertionError(f"{target} must not run"))
+ )
+ return stack
+
+
+# ---------------------------------------------------------------------------
+# Registration (KTD1, KTD10)
+# ---------------------------------------------------------------------------
+
+
+def _hermetic_config(tmp_path, monkeypatch, env_file_text: str | None):
+ config_file = tmp_path / ".env"
+ if env_file_text is not None:
+ config_file.write_text(env_file_text, encoding="utf-8")
+ config_file.chmod(0o600)
+ monkeypatch.setenv("LAST30DAYS_CONFIG_DIR", str(tmp_path))
+ monkeypatch.setattr(env, "CONFIG_DIR", tmp_path)
+ monkeypatch.setattr(env, "CONFIG_FILE", config_file)
+ monkeypatch.chdir(tmp_path)
+ for key in (
+ "LAST30DAYS_HOST", "LAST30DAYS_X_HOST_LANE", "X_BEARER_TOKEN",
+ "LAST30DAYS_X_BACKEND", "XAI_API_KEY", "AUTH_TOKEN", "CT0",
+ "XQUIK_API_KEY", "FROM_BROWSER", "AGENTCOOKIE", "BROWSER_CDP_URL",
+ ):
+ monkeypatch.delenv(key, raising=False)
+ return contextlib.ExitStack()
+
+
+def _neutral_sources(stack: contextlib.ExitStack) -> None:
+ stack.enter_context(mock.patch.object(env, "_load_keychain", return_value={}))
+ stack.enter_context(mock.patch.object(env, "_load_pass", return_value={}))
+ stack.enter_context(mock.patch.object(env, "_find_project_env", return_value=None))
+
+
+def test_new_keys_are_registered_in_get_config(tmp_path, monkeypatch):
+ stack = _hermetic_config(tmp_path, monkeypatch, None)
+ with stack:
+ _neutral_sources(stack)
+ config = env.get_config()
+ for key in ("LAST30DAYS_HOST", "X_BEARER_TOKEN", "LAST30DAYS_X_HOST_LANE"):
+ assert key in config, key
+
+
+def test_bearer_token_joins_keychain_keys():
+ assert "X_BEARER_TOKEN" in env.KEYCHAIN_KEYS
+
+
+def test_host_and_bearer_load_from_env_file(tmp_path, monkeypatch):
+ stack = _hermetic_config(
+ tmp_path, monkeypatch,
+ "LAST30DAYS_HOST=grok-bot\nX_BEARER_TOKEN=dummy-bearer\n",
+ )
+ with stack:
+ _neutral_sources(stack)
+ config = env.get_config()
+ assert config["LAST30DAYS_HOST"] == "grok-bot"
+ assert config["X_BEARER_TOKEN"] == "dummy-bearer"
+ assert env.x_policy(config).official_only is True
+
+
+def test_lane_signal_from_process_env_is_declared(tmp_path, monkeypatch):
+ stack = _hermetic_config(tmp_path, monkeypatch, None)
+ monkeypatch.setenv("LAST30DAYS_X_HOST_LANE", "1")
+ with stack:
+ _neutral_sources(stack)
+ config = env.get_config()
+ assert config["LAST30DAYS_X_HOST_LANE"] == "1"
+ assert env.x_host_lane_declared(config) is True
+
+
+def test_lane_signal_in_env_file_only_is_ignored(tmp_path, monkeypatch):
+ """KTD10: a removed connector must never leave a stale .env declaration."""
+ stack = _hermetic_config(tmp_path, monkeypatch, "LAST30DAYS_X_HOST_LANE=1\n")
+ with stack:
+ _neutral_sources(stack)
+ config = env.get_config()
+ assert not config.get("LAST30DAYS_X_HOST_LANE")
+ assert env.x_host_lane_declared(config) is False
+
+
+# ---------------------------------------------------------------------------
+# Policy record (KTD2)
+# ---------------------------------------------------------------------------
+
+
+def test_constants():
+ assert env.X_BACKEND_ORDER == ("bird", "xai", "xurl", "xquik")
+ assert env.X_BACKEND_OPT_IN == ("grok", "xapi")
+ assert env.X_OFFICIAL == ("xapi", "xai", "xurl")
+ assert set(env.X_OFFICIAL) <= set(env.X_BACKEND_KNOWN)
+
+
+def test_policy_default_host_matches_today():
+ policy = env.x_policy({})
+ assert policy.host == ""
+ assert policy.official_only is False
+ assert tuple(policy.auto_chain) == env.X_BACKEND_ORDER
+ assert policy.cookie_discovery is True
+ assert policy.hint_namespace == "default"
+ assert env.x_auto_chain({}) == list(env.X_BACKEND_ORDER)
+
+
+def test_policy_grok_bot_is_official_only():
+ policy = env.x_policy(_grok_bot())
+ assert policy.host == "grok-bot"
+ assert policy.official_only is True
+ assert tuple(policy.auto_chain) == ("xapi", "xai", "xurl")
+ assert policy.cookie_discovery is False
+ assert policy.hint_namespace == "official"
+ assert env.x_auto_chain(_grok_bot()) == ["xapi", "xai", "xurl"]
+
+
+def test_policy_is_frozen():
+ policy = env.x_policy(_grok_bot())
+ with pytest.raises(Exception):
+ policy.official_only = False # type: ignore[misc]
+
+
+def test_policy_never_infers_host_from_agent_env_or_platform():
+ """Only LAST30DAYS_HOST switches the policy (AE8: CURSOR_AGENT is not it)."""
+ with (
+ mock.patch.dict(os.environ, {"CURSOR_AGENT": "1", "GROK_CLI": "1"}, clear=False),
+ mock.patch("platform.system", return_value="Linux"),
+ ):
+ policy = env.x_policy({})
+ assert policy.official_only is False
+ assert tuple(policy.auto_chain) == env.X_BACKEND_ORDER
+ assert policy.cookie_discovery is True
+
+
+def test_other_host_values_are_not_official_only():
+ for host in ("claude-code", "codex", "cursor", "grok", "hermes", "openclaw"):
+ assert env.x_policy({"LAST30DAYS_HOST": host}).official_only is False
+
+
+def test_bird_pin_re_enables_cookie_discovery_on_grok_bot():
+ policy = env.x_policy(_grok_bot(LAST30DAYS_X_BACKEND="bird"))
+ assert policy.official_only is True
+ assert policy.cookie_discovery is True
+
+
+def test_non_bird_pin_keeps_discovery_off_on_grok_bot():
+ for pin in ("xquik", "grok", "xai", "xurl", "xapi"):
+ assert env.x_policy(_grok_bot(LAST30DAYS_X_BACKEND=pin)).cookie_discovery is False
+
+
+def test_grok_bot_literal_compared_only_in_env():
+ """The policy helper is the only place the host string is compared."""
+ pattern = re.compile(r"""==\s*['"]grok-bot['"]|['"]grok-bot['"]\s*==""")
+ offenders = []
+ for path in LIB_DIR.glob("*.py"):
+ if path.name == "env.py":
+ continue
+ if pattern.search(path.read_text(encoding="utf-8")):
+ offenders.append(path.name)
+ assert offenders == []
+
+
+# ---------------------------------------------------------------------------
+# Chain resolution on a Grok Bot host (R1, R3, AE2, AE2a)
+# ---------------------------------------------------------------------------
+
+
+def test_grok_bot_bird_pin_gives_bird_and_discovery_runs():
+ config = _grok_bot(LAST30DAYS_X_BACKEND="bird", AGENTCOOKIE="on")
+ with (
+ mock.patch("platform.system", return_value="Linux"),
+ mock.patch("lib.agentcookie.read_x_cookies", return_value=dict(_PAIR)) as sidecar,
+ mock.patch("lib.chrome_cdp.read_x_cookies", return_value=None),
+ mock.patch.object(env, "extract_browser_credentials", return_value={}),
+ ):
+ env._discover_and_apply_x_credentials(config)
+ assert sidecar.called
+ assert config["AUTH_TOKEN"] == "test-auth-token"
+ with _stub_backends(), mock.patch("lib.bird_x.set_credentials") as prime:
+ assert env.x_backend_chain(config) == ["bird"]
+ assert prime.called
+
+
+def test_grok_bot_xquik_pin_gives_xquik_without_discovery():
+ config = _grok_bot(LAST30DAYS_X_BACKEND="xquik", XQUIK_API_KEY="dummy-key",
+ FROM_BROWSER="firefox", AGENTCOOKIE="on")
+ with (
+ mock.patch("platform.system", return_value="Linux"),
+ _no_cookie_reads(),
+ ):
+ env._discover_and_apply_x_credentials(config)
+ assert "AUTH_TOKEN" not in config
+ with _stub_backends():
+ assert env.x_backend_chain(config) == ["xquik"]
+
+
+def test_grok_bot_grok_pin_with_signed_in_cli_gives_grok():
+ config = _grok_bot(LAST30DAYS_X_BACKEND="grok", **_COOKIES)
+ with _stub_backends(grok_authed=True):
+ assert env.x_backend_chain(config) == ["grok"]
+
+
[email protected]("pin", ["xai", "xurl", "xapi"])
+def test_grok_bot_official_pins_select_only_that_backend(pin):
+ config = _grok_bot(
+ LAST30DAYS_X_BACKEND=pin,
+ X_BEARER_TOKEN="dummy-bearer",
+ XAI_API_KEY="dummy-xai",
+ **_COOKIES,
+ )
+ with _stub_backends(xurl_available=True, xurl_stored=True):
+ assert env.x_backend_chain(config) == [pin]
+ assert env.x_backend_chain(config, local_only=True) == [pin]
+
+
+def test_grok_bot_unpinned_pin_of_unavailable_backend_is_empty():
+ config = _grok_bot(LAST30DAYS_X_BACKEND="xapi", XAI_API_KEY="dummy-xai")
+ with _stub_backends():
+ assert env.x_backend_chain(config) == []
+
+
+def test_grok_bot_bearer_gives_xapi():
+ config = _grok_bot(X_BEARER_TOKEN="dummy-bearer", **_COOKIES, XQUIK_API_KEY="dummy-key")
+ with _stub_backends():
+ assert env.x_backend_chain(config) == ["xapi"]
+ assert env.get_x_source(config) == "xapi"
+
+
+def test_grok_bot_bearer_and_xai_key_gives_xapi_then_xai():
+ config = _grok_bot(X_BEARER_TOKEN="dummy-bearer", XAI_API_KEY="dummy-xai")
+ with _stub_backends():
+ assert env.x_backend_chain(config) == ["xapi", "xai"]
+
+
+def test_grok_bot_bearer_xai_and_xurl_gives_full_official_chain():
+ config = _grok_bot(X_BEARER_TOKEN="dummy-bearer", XAI_API_KEY="dummy-xai", **_COOKIES)
+ with _stub_backends(xurl_available=True, xurl_stored=True):
+ assert env.x_backend_chain(config) == ["xapi", "xai", "xurl"]
+ assert env.x_backend_chain(config, local_only=True) == ["xapi", "xai", "xurl"]
+
+
+def test_grok_bot_cookies_only_unpinned_is_unconfigured_and_never_primes_bird():
+ config = _grok_bot(**_COOKIES, XQUIK_API_KEY="dummy-key")
+ with (
+ _stub_backends(),
+ mock.patch("lib.bird_x.set_credentials",
+ side_effect=AssertionError("bird must not be primed on grok-bot")),
+ ):
+ assert env.x_backend_chain(config) == []
+ assert env.get_x_source(config) is None
+
+
+def test_non_grok_chain_still_primes_bird_when_bird_in_chain():
+ config = dict(_COOKIES)
+ with _stub_backends(), mock.patch("lib.bird_x.set_credentials") as prime:
+ assert env.x_backend_chain(config) == ["bird"]
+ prime.assert_called_once_with("test-auth-token", "test-ct0")
+
+
+def test_non_grok_xapi_is_opt_in_only():
+ config = {"X_BEARER_TOKEN": "dummy-bearer"}
+ with _stub_backends():
+ assert env.x_backend_chain(config) == []
+ assert env.x_backend_chain({**config, "LAST30DAYS_X_BACKEND": "xapi"}) == ["xapi"]
+
+
+# ---------------------------------------------------------------------------
+# get_x_source_status and get_x_source_with_method iterate the policy chain
+# ---------------------------------------------------------------------------
+
+_BIRD_OFF = {"installed": True, "authenticated": False, "username": "", "can_install": True}
+_BIRD_ON = {"installed": True, "authenticated": True, "username": "u", "can_install": True}
+
+
+def test_status_on_grok_bot_picks_xapi_and_never_probes_bird_or_xquik():
+ config = _grok_bot(X_BEARER_TOKEN="dummy-bearer", XQUIK_API_KEY="dummy-key", **_COOKIES)
+ with (
+ mock.patch("lib.bird_x.get_bird_status", return_value=dict(_BIRD_ON)),
+ mock.patch("lib.bird_x.set_credentials",
+ side_effect=AssertionError("bird must not be primed on grok-bot")),
+ mock.patch("lib.bird_x.probe_works",
+ side_effect=AssertionError("bird must not be probed on grok-bot")),
+ mock.patch("lib.xquik.probe_works",
+ side_effect=AssertionError("xquik must not be probed on grok-bot")),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=False),
+ mock.patch("lib.xurl_x.is_available", return_value=False),
+ ):
+ status = env.get_x_source_status(config, probe=True)
+ assert status["source"] == "xapi"
+ assert status["xapi_available"] is True
+
+
+def test_status_on_grok_bot_cookies_only_is_none_but_bird_pin_is_bird():
+ with (
+ mock.patch("lib.bird_x.get_bird_status", return_value=dict(_BIRD_ON)),
+ mock.patch("lib.bird_x.set_credentials"),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=False),
+ ):
+ assert env.get_x_source_status(_grok_bot(**_COOKIES))["source"] is None
+ pinned = env.get_x_source_status(_grok_bot(**_COOKIES, LAST30DAYS_X_BACKEND="bird"))
+ assert pinned["source"] == "bird"
+
+
+def test_status_on_grok_bot_grok_pin_is_grok():
+ with (
+ mock.patch("lib.bird_x.get_bird_status", return_value=dict(_BIRD_OFF)),
+ mock.patch("lib.grok_x.has_stored_auth", return_value=True),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=False),
+ ):
+ status = env.get_x_source_status(_grok_bot(LAST30DAYS_X_BACKEND="grok"))
+ assert status["source"] == "grok"
+
+
+def test_status_non_grok_ladder_unchanged():
+ with (
+ mock.patch("lib.bird_x.get_bird_status", return_value=dict(_BIRD_ON)),
+ mock.patch("lib.bird_x.set_credentials"),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=True),
+ ):
+ assert env.get_x_source_status({**_COOKIES, "XAI_API_KEY": "k"})["source"] == "bird"
+ with (
+ mock.patch("lib.bird_x.get_bird_status", return_value=dict(_BIRD_OFF)),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=True),
+ ):
+ assert env.get_x_source_status({"XAI_API_KEY": "k"})["source"] == "xai"
+ assert env.get_x_source_status({})["source"] == "xurl"
+ with (
+ mock.patch("lib.bird_x.get_bird_status", return_value=dict(_BIRD_OFF)),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=False),
+ ):
+ assert env.get_x_source_status({"XQUIK_API_KEY": "k"})["source"] == "xquik"
+ # xapi is opt-in on a non-Grok host: never the unpinned source.
+ assert env.get_x_source_status({"X_BEARER_TOKEN": "b"})["source"] is None
+ pinned = env.get_x_source_status(
+ {"X_BEARER_TOKEN": "b", "LAST30DAYS_X_BACKEND": "xapi"}
+ )
+ assert pinned["source"] == "xapi"
+
+
+def test_source_with_method_iterates_policy_chain():
+ with mock.patch("lib.xurl_x.is_available", return_value=False):
+ assert env.get_x_source_with_method({**_COOKIES, "XAI_API_KEY": "k"}) == ("bird", "env")
+ assert env.get_x_source_with_method({"XAI_API_KEY": "k"}) == ("xai", "xai")
+ assert env.get_x_source_with_method(
+ _grok_bot(X_BEARER_TOKEN="b", **_COOKIES)
+ ) == ("xapi", "bearer")
+ assert env.get_x_source_with_method(_grok_bot(**_COOKIES)) == (None, "none")
+
+
+# ---------------------------------------------------------------------------
+# Cookie discovery, browsers, and pending-auth on a Grok Bot host (R2)
+# ---------------------------------------------------------------------------
+
+
+def test_discovery_returns_before_every_leg_on_grok_bot():
+ config = _grok_bot(FROM_BROWSER="firefox", AGENTCOOKIE="on",
+ BROWSER_CDP_URL="http://127.0.0.1:18800")
+ with (
+ mock.patch("platform.system", return_value="Linux"),
+ _no_cookie_reads(),
+ mock.patch.object(env, "extract_browser_credentials",
+ side_effect=AssertionError("no browser extract on grok-bot")),
+ ):
+ env._discover_and_apply_x_credentials(config)
+ assert "AUTH_TOKEN" not in config
+ assert "TRUTHSOCIAL_TOKEN" not in config
+
+
+def test_cookie_extraction_browsers_empty_on_grok_bot_unless_bird_pinned():
+ assert env.cookie_extraction_browsers(_grok_bot(FROM_BROWSER="firefox")) == []
+ assert env.cookie_extraction_browsers(_grok_bot(FROM_BROWSER="auto")) == []
+ assert env.cookie_extraction_browsers(
+ _grok_bot(FROM_BROWSER="firefox", LAST30DAYS_X_BACKEND="bird")
+ ) == ["firefox"]
+ assert env.cookie_extraction_browsers({"FROM_BROWSER": "firefox"}) == ["firefox"]
+
+
+def test_pending_browser_auth_false_on_grok_bot_unless_bird_pinned():
+ cfg = _grok_bot(FROM_BROWSER="chrome", _BROWSER_COOKIE_MODE="plan_only")
+ with (
+ mock.patch("lib.env.get_x_source", return_value=None),
+ mock.patch("lib.bird_x.is_bird_installed", return_value=True),
+ ):
+ assert env.x_pending_browser_auth(cfg) is False
+ assert env.x_pending_browser_auth({**cfg, "LAST30DAYS_X_BACKEND": "bird"}) is True
+
+
+# ---------------------------------------------------------------------------
+# Lane and envelope availability (R12, R13, KTD11, AE5a)
+# ---------------------------------------------------------------------------
+
+
+def _no_engine_backend():
+ stack = contextlib.ExitStack()
+ stack.enter_context(mock.patch("lib.env.get_x_source", return_value=None))
+ stack.enter_context(mock.patch("lib.env.x_pending_browser_auth", return_value=False))
+ return stack
+
+
+def test_lane_signal_lists_x_in_read_mode():
+ cfg = {"LAST30DAYS_X_HOST_LANE": "1", "_BROWSER_COOKIE_MODE": "read"}
+ with _no_engine_backend():
+ assert "x" in pipeline.available_sources(cfg)
+ assert pipeline.available_sources(cfg).count("x") == 1
+
+
+def test_lane_signal_from_process_env_lists_x_but_env_file_line_does_not(tmp_path, monkeypatch):
+ stack = _hermetic_config(tmp_path, monkeypatch, "LAST30DAYS_X_HOST_LANE=1\n")
+ read = env.ConfigLoadPolicy(browser_cookies="read")
+ with stack:
+ _neutral_sources(stack)
+ stack.enter_context(mock.patch.object(env, "_discover_and_apply_x_credentials"))
+ file_only = env.get_config(read)
+ monkeypatch.setenv("LAST30DAYS_X_HOST_LANE", "1")
+ exported = env.get_config(read)
+ with _no_engine_backend():
+ assert "x" not in pipeline.available_sources(file_only)
+ assert "x" in pipeline.available_sources(exported)
+
+
+def test_envelope_without_lane_signal_lists_x():
+ cfg = {"_BROWSER_COOKIE_MODE": "read"}
+ with _no_engine_backend():
+ assert "x" not in pipeline.available_sources(cfg)
+ assert "x" in pipeline.available_sources(cfg, x_envelope=True)
+
+
+def test_lane_available_in_every_cookie_mode():
+ for mode in ("off", "read", "plan_only"):
+ cfg = {"LAST30DAYS_X_HOST_LANE": "1", "_BROWSER_COOKIE_MODE": mode}
+ with _no_engine_backend():
+ assert "x" in pipeline.available_sources(cfg), mode
+
+
+def test_suppress_x_host_lane_hides_lane_but_not_envelope():
+ cfg = {"LAST30DAYS_X_HOST_LANE": "1"}
+ with _no_engine_backend():
+ assert "x" not in pipeline.available_sources(cfg, suppress_x_host_lane=True)
+ assert "x" in pipeline.available_sources(cfg, x_envelope=True, suppress_x_host_lane=True)
+
+
+def test_lane_signal_uses_precomputed_x_pending_without_calling_predicate():
+ cfg = {"LAST30DAYS_X_HOST_LANE": "1"}
+ with (
+ mock.patch("lib.env.get_x_source", return_value=None),
+ mock.patch("lib.env.x_pending_browser_auth",
+ side_effect=AssertionError("must use precomputed x_pending")),
+ ):
+ assert "x" in pipeline.available_sources(cfg, x_pending=False)
+
+
+class _Stop(Exception):
+ pass
+
+
+def _capture_available_sources_kwargs(**run_kwargs) -> dict:
+ """Drive pipeline.run up to its available_sources call and capture the kwargs."""
+ captured: dict = {}
+
+ def fake(config, requested_sources=None, **kw):
+ captured.update(kw)
+ raise _Stop()
+
+ with mock.patch.object(pipeline, "available_sources", side_effect=fake):
+ with pytest.raises(_Stop):
+ pipeline.run(topic="x policy", config={}, depth="quick", **run_kwargs)
+ return captured
+
+
+def test_run_passes_suppress_flag_through_and_defaults_false():
+ assert _capture_available_sources_kwargs().get("suppress_x_host_lane") is False
+ assert _capture_available_sources_kwargs(
+ suppress_x_host_lane=True
+ ).get("suppress_x_host_lane") is True
+
+
+def test_comparison_internal_subrun_does_not_suppress_lane():
+ captured = _capture_available_sources_kwargs(internal_subrun=True)
+ assert captured.get("suppress_x_host_lane") is False
+
+
+def test_discovery_enrichment_pass_suppresses_lane():
+ calls: list[dict] = []
+
+ def fake_run(**kwargs):
+ calls.append(kwargs)
+ raise RuntimeError("stop")
+
+ nomination = pipeline.Nomination(name="Agent SDK Wars", seed_score=1.0)
+ with mock.patch.object(pipeline, "run", side_effect=fake_run):
+ pipeline.enrich_nominations([nomination], config={}, max_workers=1, budget_seconds=5)
+ assert len(calls) == 1
+ assert calls[0]["internal_subrun"] is True
+ assert calls[0]["suppress_x_host_lane"] is True
+
+
+# ---------------------------------------------------------------------------
+# diagnose reports the lane as the connector backend
+# ---------------------------------------------------------------------------
+
+
+def test_diagnose_reports_connector_when_lane_declared_and_no_backend():
+ cfg = {"LAST30DAYS_X_HOST_LANE": "1", "_BROWSER_COOKIE_MODE": "plan_only"}
+ with (
+ mock.patch("lib.bird_x.get_bird_status", return_value=dict(_BIRD_OFF)),
+ mock.patch("lib.bird_x.is_bird_installed", return_value=False),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=False),
+ ):
+ diag = pipeline.diagnose(cfg, safe=True)
+ assert diag["x_backend"] == "connector"
+ assert "x" in diag["available_sources"]
+
+
+def test_diagnose_keeps_engine_backend_when_lane_declared_with_key():
+ cfg = {"LAST30DAYS_X_HOST_LANE": "1", "XAI_API_KEY": "dummy-xai",
+ "_BROWSER_COOKIE_MODE": "plan_only"}
+ with (
+ mock.patch("lib.bird_x.get_bird_status", return_value=dict(_BIRD_OFF)),
+ mock.patch("lib.bird_x.is_bird_installed", return_value=False),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=False),
+ ):
+ diag = pipeline.diagnose(cfg, safe=True)
+ assert diag["x_backend"] == "xai"
+
+
+def test_diagnose_without_lane_or_backend_is_none():
+ cfg = {"_BROWSER_COOKIE_MODE": "plan_only"}
+ with (
+ mock.patch("lib.bird_x.get_bird_status", return_value=dict(_BIRD_OFF)),
+ mock.patch("lib.bird_x.is_bird_installed", return_value=False),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=False),
+ ):
+ diag = pipeline.diagnose(cfg, safe=True)
+ assert diag["x_backend"] is None
+ assert "x" not in diag["available_sources"]
+
+
+# ---------------------------------------------------------------------------
+# backends.resolve filters findings and chain by policy (R4 JSON surface)
+# ---------------------------------------------------------------------------
+
+
+def _node_ok(name, timeout=health.PROBE_TIMEOUT):
+ return health.DependencyProbe(name=name, status=health.OK, detail=f"{name} 1.0.0")
+
+
+def _resolve_x(config):
+ with (
+ mock.patch("lib.bird_x.is_bird_installed", return_value=True),
+ mock.patch("lib.bird_x.set_credentials", lambda *a, **k: None),
+ mock.patch("lib.backends.which", return_value=None),
+ mock.patch("lib.health.probe_dependency", _node_ok),
+ mock.patch("lib.xurl_x.has_stored_auth", return_value=False),
+ mock.patch("lib.xurl_x.is_available", return_value=False),
+ ):
+ return backends.resolve("x", config)
+
+
+def test_resolve_on_grok_bot_carries_only_official_backends():
+ res = _resolve_x(_grok_bot(**_COOKIES, XQUIK_API_KEY="dummy-key"))
+ assert res.chain == ["xapi", "xai", "xurl"]
+ assert [f.name for f in res.findings] == ["xapi", "xai", "xurl"]
+ assert res.active_backend is None
+ blob = str([(f.name, f.detail, f.prescription, f.requires) for f in res.findings])
+ for word in ("AUTH_TOKEN", "CT0", "cookie", "XQUIK", "grok CLI", "grok login"):
+ assert word.lower() not in blob.lower(), word
+
+
+def test_resolve_on_grok_bot_predicts_xapi_with_bearer():
+ res = _resolve_x(_grok_bot(X_BEARER_TOKEN="dummy-bearer", **_COOKIES))
+ assert res.active_backend == "xapi"
+ assert res.pinned is False
+ assert res.summary.startswith("will use: xapi")
+
+
+def test_resolve_on_grok_bot_bird_pin_names_bird_only_as_pinned():
+ res = _resolve_x(_grok_bot(**_COOKIES, LAST30DAYS_X_BACKEND="bird"))
+ assert res.pinned is True and res.pin == "bird"
+ assert res.active_backend == "bird"
+ assert "bird" in res.chain
+ assert "xquik" not in res.chain and "grok" not in res.chain
+
+
+def test_resolve_non_grok_chain_unchanged():
+ res = _resolve_x(dict(_COOKIES))
+ assert res.chain == list(env.X_BACKEND_ORDER + env.X_BACKEND_OPT_IN)
+ assert res.active_backend == "bird"
+ # xapi is opt-in off Grok Bot: an ambient bearer never wins the prediction.
+ res = _resolve_x({"X_BEARER_TOKEN": "dummy-bearer"})
+ assert res.active_backend is None
+ res = _resolve_x({"X_BEARER_TOKEN": "dummy-bearer", "LAST30DAYS_X_BACKEND": "xapi"})
+ assert res.active_backend == "xapi" and res.pinned is True
+
+
+def test_xapi_probe_is_key_presence_only():
+ finding = backends._X_PROBES["xapi"]({"X_BEARER_TOKEN": "dummy-bearer"})
+ assert finding.status == "ok"
+ assert finding.requires == "X_BEARER_TOKEN (X API v2)"
+ assert "dummy-bearer" not in finding.detail
+ missing = backends._X_PROBES["xapi"]({})
+ assert missing.status == "missing"
+ assert "xapi" in backends._X_PAID
+ assert backends._X_REQUIRES["xapi"] == "X_BEARER_TOKEN (X API v2)"
diff --git a/tests/test_xquik.py b/tests/test_xquik.py
index f08fab7f..fcf9b818 100644
--- a/tests/test_xquik.py
+++ b/tests/test_xquik.py
@@ -1,6 +1,7 @@
import unittest
from unittest.mock import patch
+from lib import health
from lib.xquik import (
DEPTH_CONFIG,
_parse_tweet,
@@ -248,6 +249,34 @@ class TestSearchXquik(unittest.TestCase):
result = search_xquik("test", "2026-01-01", "2026-03-01", token="unpaid-key")
self.assertEqual(result["items"], [])
self.assertIn("unpaid", result.get("error", "").lower())
+ # The X retrieval branch classifies by message text only, so the fixed
+ # detail must carry a payment-required classifier marker (KTD6).
+ self.assertEqual("Xquik key unpaid: payment required (402)", result["error"])
+ self.assertEqual(
+ http_mod.classify_failure(message=result["error"]), health.PAYMENT_REQUIRED
+ )
+
+ @patch("lib.env.x_backend_chain", return_value=["xquik"])
+ @patch("lib.xquik.http.get")
+ def test_unpaid_402_yields_payment_required_source_outcome(self, mock_get, _chain):
+ # End to end through the X backend loop: a 402 from xquik as the sole
+ # backend settles the x source as payment-required, not auth-failed.
+ from lib import http as http_mod, pipeline, schema
+ mock_get.side_effect = http_mod.HTTPError("Payment Required", status_code=402)
+ sq = schema.SubQuery(label="primary", search_query="q", ranking_query="q?", sources=["x"])
+ runtime = schema.ProviderRuntime(
+ reasoning_provider="mock", planner_model="mock", rerank_model="mock",
+ x_search_backend=None,
+ )
+ with self.assertRaises(pipeline.SourceRunError) as ctx:
+ pipeline._retrieve_stream(
+ topic="q", subquery=sq, source="x", config={"XQUIK_API_KEY": "k"},
+ depth="default", date_range=("2026-05-19", "2026-06-18"),
+ runtime=runtime, mock=False,
+ )
+ self.assertEqual(health.PAYMENT_REQUIRED, ctx.exception.outcome_state)
+ state, attempted = pipeline._classify_source_failure(ctx.exception)
+ self.assertEqual((health.PAYMENT_REQUIRED, True), (state, attempted))
@patch("lib.xquik.http.get")
def test_empty_tweets_list(self, mock_get):
diff --git a/tests/test_xurl_x.py b/tests/test_xurl_x.py
index be6751c0..931b1201 100644
--- a/tests/test_xurl_x.py
+++ b/tests/test_xurl_x.py
@@ -192,19 +192,26 @@ class TestSearchX(unittest.TestCase):
result = xurl_x.search_x("hello world")
self.assertEqual(result["data"][0]["id"], "1")
- def test_returns_error_on_non_zero_exit(self):
- completed = mock.Mock(returncode=1, stdout="", stderr="rate limit exceeded")
+ def test_returns_fixed_error_on_non_zero_exit(self):
+ # xurl's stderr may echo the request or the bearer; only an
+ # engine-authored fixed string (with a classifier marker) survives.
+ completed = mock.Mock(
+ returncode=1, stdout="",
+ stderr="rate limit exceeded for dummy-x-bearer-secret-000",
+ )
with mock.patch("subprocess.run", return_value=completed):
result = xurl_x.search_x("test")
- self.assertIn("error", result)
- self.assertIn("rate limit exceeded", result["error"])
+ self.assertEqual(xurl_x.ERR_RATE_LIMITED, result["error"])
+ self.assertIn("rate limit", result["error"])
+ self.assertNotIn("dummy-x-bearer-secret-000", result["error"])
- def test_returns_error_on_invalid_json(self):
- completed = mock.Mock(returncode=0, stdout="NOT JSON")
+ def test_returns_fixed_error_on_invalid_json(self):
+ completed = mock.Mock(returncode=0, stdout="NOT JSON dummy-x-bearer-secret-000")
with mock.patch("subprocess.run", return_value=completed):
result = xurl_x.search_x("test")
- self.assertIn("error", result)
- self.assertIn("Invalid JSON", result["error"])
+ self.assertEqual(xurl_x.ERR_INVALID_JSON, result["error"])
+ self.assertIn("invalid JSON", result["error"])
+ self.assertNotIn("dummy-x-bearer-secret-000", result["error"])
def test_returns_error_when_not_installed(self):
with mock.patch("subprocess.run", side_effect=FileNotFoundError):
@@ -345,11 +352,14 @@ class TestParseXResponse(unittest.TestCase):
items = xurl_x.parse_x_response(resp)
self.assertEqual(items[0]["relevance"], 0.5)
- def test_url_empty_when_no_username(self):
- # author_id not in includes.users → username=""
+ def test_url_falls_back_to_i_status_when_no_username(self):
+ # author_id not in includes.users → username="" but the post is
+ # kept with the id-only citation form (shared x_api parser).
resp = _make_api_response(tweets=[self._tweet("999", "text", "unknown_uid")])
items = xurl_x.parse_x_response(resp)
- self.assertEqual(items[0]["url"], "")
+ self.assertEqual(items[0]["url"], "https://x.com/i/status/999")
+ self.assertEqual(items[0]["author_handle"], "")
+ self.assertEqual(items[0]["post_id"], "999")
def test_multiple_tweets_parsed(self):
tweets = [self._tweet(str(i), f"tweet {i}", "u1") for i in range(5)]
@@ -377,6 +387,10 @@ class TestDepthConfig(unittest.TestCase):
for depth in ("quick", "default", "deep"):
self.assertIn(depth, xurl_x.DEPTH_CONFIG)
+ def test_depth_config_is_shared_with_x_api(self):
+ from lib import x_api
+ self.assertIs(x_api.DEPTH_CONFIG, xurl_x.DEPTH_CONFIG)
+
def test_deep_greater_than_quick(self):
self.assertGreater(
xurl_x.DEPTH_CONFIG["deep"],