Skip to content

armgit / avibe

Public Imported from GitHub
0

Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.

feat(skills): add agent-prompt-audit built-in skill (#2185)

Call White0ca51fa8d0b9
diff --git a/skills/agent-prompt-audit/SKILL.md b/skills/agent-prompt-audit/SKILL.md
new file mode 100644
index 000000000..3d19d1332
--- /dev/null
+++ b/skills/agent-prompt-audit/SKILL.md
@@ -0,0 +1,174 @@
+---
+name: agent-prompt-audit
+slug: agent-prompt-audit
+description: Audit and improve the prompt surface of Avibe Agents across backends (Claude, Codex/GPT, OpenCode) — global and project rules, Agent system prompts, Skills, delegation briefs, and Task and Watch messages — using real run evidence. Use when an Agent misbehaves (stalls, over-asks, over-reaches, ignores or over-applies a rule), after a model or backend change, or when the user asks to review, clean up, or tighten prompts.
+version: 0.1.0
+---
+
+# Agent Prompt Audit
+
+An Agent's behavior comes from every piece of text that reaches it over its
+lifecycle, not just its system prompt. The audit's job is to find which text
+causes the behavior the user sees, on the backend and model that actually ran
+it, and to propose the smallest change that fixes it. Judge each instruction
+by what it does to behavior, not by its length: sometimes the fix is adding a
+missing reason or exit, and a clean surface is a valid result.
+
+Deliver a report of findings — each with its evidence, confidence, and a
+concrete proposed change — and apply changes only when asked.
+
+## Core ideas
+
+**Evidence over reading.** What Agents actually did beats what the text seems
+to say. Start from real runs and user corrections ("you stopped", "why didn't
+you report", "don't ask me that"), trace each symptom to the line that caused
+it or the missing line that would have prevented it, and use `git blame` to
+learn what incident a rule was written for and whether it still happens.
+A finding without evidence or documented model behavior is a flag, not a fix.
+
+**Context is kept; constraints must earn their place.** Facts only the author
+knows — environment, contracts, ownership, quality bar, the reason behind a
+rule — are what prompts are for. Behavioral constraints are what go stale.
+Keep exact scripts where one sequence is safe (destructive commands, auth,
+merge gates, key custody), prohibitions against failures that still reproduce,
+and the scope bounds that make autonomy safe.
+
+**Say intent and reason, not pressure or method.** Caps, `MUST/NEVER`, and
+emphasis without a reason make current models rigid; step scripts for judgment
+work and strategy coaching usually do worse than the model's own plan; fixed
+formats, word caps, and "don't narrate" rules produce silence or starved
+answers. Fossils — named-model workarounds, incident numbers as authority,
+"now/no longer" phrasing, one session's stumble made permanent — should become
+the current rule they stand for.
+
+**Every stop needs an exit.** Agents run across turns, wake on callbacks, and
+hand work to each other, so the costliest defects are lifecycle gaps: a
+"stop/wait" with no statement of what the turn produces instead, asking
+permission for reversible in-scope steps, continuing without bounds after
+repeated failure, waiting with no durable waiter or expiry meaning, briefs
+missing the goal or report target, callbacks that say "done" without the
+result, and Task/Watch messages that restate rules on every fire.
+
+**One home per rule, at the layer whose timing fits.** Always-loaded and
+recurring text has the most leverage and deserves the most scrutiny.
+Duplicates that disagree force the Agent to guess; keep the mechanism in one
+place and a principle or pointer elsewhere. Agreeing fallbacks are fine. Long
+procedures belong in on-demand Skills, not always-loaded rules.
+
+**Shared text runs on every backend.** GPT/Codex tend to follow a bare
+prohibition or stop literally, so they need scope and exit conditions; strong
+Claude models tend to over-reach, so they need scope bounds and a definition
+of done; tool names and native mechanics dangle on other backends. Take
+model-specific behavior from the vendor's current docs, and lower confidence
+when you cannot reach them.
+
+**A removal is a hypothesis.** For contested changes, compare behavior before
+and after with a scratch run on the target that produced the failure, and read
+the transcript rather than asking the model whether it needs the rule.
+
+## Where the surface lives
+
+Verify against the current machine; these are starting points. Each backend
+also reads its own native configuration — config directories moved by
+environment variables (`CLAUDE_CONFIG_DIR`, `CODEX_HOME`, OpenCode's config
+path), and native subagent definitions such as `.claude/agents/`,
+`.codex/agents/`, or OpenCode agents — so resolve what the target backend
+actually loads rather than assuming default paths.
+
+| Layer | Where | How it changes |
+| --- | --- | --- |
+| Avibe runtime prompt | `vibe debug prompt export --format json` lists every source; `vibe debug prompt export --format json --context-file <file>` renders a composition from the inputs you supply (backend, Agent instructions, Skill directory, context), so it approximates the target only as well as those inputs match (history in the Avibe repo `core/prompts/`, if checked out) | Proposal to the Avibe repository |
+| Global rules and native backend config | `~/.claude/CLAUDE.md`, `~/.codex/AGENTS.md`, Codex `developer_instructions` in `$CODEX_HOME/config.toml` (default `~/.codex`), OpenCode `instructions` in global or project `opencode.json[c]`, … | Edit the source if the file is generated or imports others |
+| Project rules | nearest `AGENTS.md` / `CLAUDE.md` chain | The repository's own delivery process |
+| Agent system prompt, model, effort | `vibe agent show <name> --json` | `vibe agent update <name> --system-prompt-file <file>` |
+| Skills | user skill dirs (follow symlinks), Avibe `skills/`, project `.agents/skills/` | The directory's owner |
+| Task and Watch messages (re-sent every fire) | `vibe task list` / `vibe watch list` for ids, then `vibe task show <id>` / `vibe watch show <id>` for the full text | `vibe task update`, `vibe watch update` |
+| User preferences (read on demand) | `~/.avibe/state/user_preferences.md`; inspect only the reported user's part, and only when the transcript shows it was read | The user |
+| Delegation briefs and callbacks | `agent_runs.message` / `result_text` | The prompt or Skill that writes them |
+
+Only Skill descriptions on the first catalog page are loaded every turn;
+later pages, `disable-model-invocation` Skills, Skill bodies, and references
+load on demand. Project Skills in that catalog resolve from the target
+Session's working directory, not yours, so read them from there.
+
+## Finding evidence
+
+Resolve the actual target (backend, model, effort) from the run record, not
+the Agent's current definition. A Session's model and effort can change during
+its life and ordinary IM turns have no run record, so treat the Session row as
+the current setting and mark the target unconfirmed if it may have changed.
+Attribute a symptom only to prompt text that existed when it ran. Each run's
+`prompt` and `message` in `vibe runs show` snapshot what a Task or Watch
+actually sent; file-owned text needs Git or release history matching the run;
+Agent system prompts have no history. A long-lived Codex thread also keeps
+earlier injected prompt snapshots in its native history, so text since removed
+may still have been in view. Where the text may have changed since the run and
+no history covers it, say the attribution is unconfirmed.
+
+`vibe runs show <id>` gives one run's prompt, result, and callback state;
+`vibe data query` is read-only SQLite over `agent_sessions`, `agent_runs`, and
+`messages`. Keep evidence to the Session the user reported, plus any others they point
+to — a channel's `scope_id` can hold other people's threads; the user's
+own corrections in `messages` are usually the sharpest evidence. For a
+recurring Task or Watch, `vibe runs list --definition-id <id>` gathers its
+fires across per-run Sessions. A stuck delegated turn stays `running`, so include
+long-running rows when the complaint is a stall; ordinary IM turns have no
+`agent_runs` row, so read that Session's messages instead. A delegated run can succeed while
+its report never arrives; check `callback_status` and `callback_error` when the
+complaint is a missing result. Two starting points, each run
+with `vibe data query --sql-file <file>` (or `--sql-file -` for stdin):
+
+```sql
+-- The reported session and its current backend, model, and effort
+select id, scope_id, agent_name, agent_backend, model, reasoning_effort, status
+from agent_sessions where id = '<session>';
+
+-- Recent failed, cancelled, or silent Agent runs in that session
+select id, run_type, status, model, created_at
+from agent_runs
+where session_id = '<session>'
+ and run_type in ('agent_run','scheduled','watch','webhook','task_escalation')
+ and exit_code is null -- command-backed Tasks record an exit code instead
+ and (status in ('failed','canceled','cancelled')
+ or (status in ('succeeded','completed') and coalesce(trim(result_text),'') = ''))
+order by created_at desc;
+```
+
+Quote the minimum excerpt and redact secrets and unrelated private content.
+
+A before/after probe spends the user's account and writes session state, so
+propose it in the report unless the user asked for verification. A useful probe
+reproduces the original conditions — same backend, model, and effort, and only
+the context before the failing turn — rather than forking a session that
+already holds the failure and its correction.
+
+## From symptom to likely cause
+
+User complaints map to recurring prompt defects. Treat these as leads to
+check against the transcript, not verdicts.
+
+| What the user sees | Where to look first |
+| --- | --- |
+| Agent stopped or went quiet mid-task | A "stop / wait / do not proceed" with no stated exit; "don't narrate" or "report only at the end"; a wait with no durable Watch or expiry meaning |
+| Keeps asking for permission | "Ask before…" with no threshold separating reversible in-scope steps from irreversible or outward-facing ones |
+| Did far more than asked | Autonomy with no scope bound or definition of done, most often on strong Claude models |
+| Followed a rule where it made no sense | A bare prohibition with no reason or scope, most often on GPT/Codex; pressure language (caps, `MUST/NEVER`) |
+| Behaves differently across Agents or backends | The same rule at different strengths in different layers; backend-specific tool names in shared text |
+| Delegated work came back unusable | A brief missing goal, acceptance evidence, or report target; a callback that says "done" without the result |
+| Recurring Task or Watch runs drift or repeat themselves | The fire message restates loaded rules, names finished work, or asks for output the recipient cannot act on |
+| Stale commands, paths, or answers | Facts that no longer match the CLI or code; fossils like named-model workarounds or "now / no longer" phrasing |
+
+## Report
+
+Open with counts and up to three findings that matter most; zero findings is a valid report. For each
+finding: location, the evidence excerpt, which idea above it violates and why
+on which target, confidence (high: reproduced in transcripts or documented;
+medium: consistent known behavior; low: heuristic, flag only), and the
+proposed change — a file hunk, or a before/after payload plus the update
+command for text stored in Avibe state. Rewrite rather than delete when the
+concern is still live, and complete each removal across duplicates, tests, and
+mirrors.
+
+State each finding's confidence once and the audit's overall limits once;
+repeating caveats in every paragraph buries the findings. Read-only checks,
+such as `--help` or reading a file, settle a doubt faster than flagging it.
diff --git a/tests/test_harness_skill_guidance.py b/tests/test_harness_skill_guidance.py
index bce9a3b04..57854ab6d 100644
--- a/tests/test_harness_skill_guidance.py
+++ b/tests/test_harness_skill_guidance.py
@@ -300,3 +300,26 @@ def test_pr_delivery_loop_keeps_one_optional_reference_out_of_routine_loading()
reference = (directory / references[0]).read_text()
assert reference.startswith("# Delivery Rationale and Examples")
assert reference not in rendered
+
+
+def test_agent_prompt_audit_examples_parse_against_the_real_cli() -> None:
+ """The audit Skill teaches live `vibe` calls.
+
+ Each backticked example that carries a flag or placeholder is parsed with
+ its `<placeholder>` and `...` slots filled, so a renamed command or flag
+ fails here instead of in an audit run.
+ """
+
+ body = _read("skills/agent-prompt-audit/SKILL.md")
+ examples = sorted(
+ {example for example in re.findall(r"`(vibe [^`]+)`", body) if "--" in example or "<" in example}
+ )
+ assert examples, "no embedded vibe examples found — did the regex drift?"
+
+ parser = cli.build_parser()
+ for example in examples:
+ argv = shlex.split(re.sub(r"<[^>]+>|\.\.\.", "x", example))[1:]
+ try:
+ parser.parse_args(argv)
+ except SystemExit as exc:
+ raise AssertionError(f"agent-prompt-audit teaches an unparseable command: {example!r}") from exc
diff --git a/tests/test_prompt_studio_catalog.py b/tests/test_prompt_studio_catalog.py
index 3c0ffb390..e1b5ea75a 100644
--- a/tests/test_prompt_studio_catalog.py
+++ b/tests/test_prompt_studio_catalog.py
@@ -85,6 +85,7 @@ def test_studio_catalog_contains_runtime_modules_and_builtin_skills() -> None:
(document["name"] for document in skills), key=lambda name: (name.casefold(), name)
)
assert {document["name"] for document in skills} == {
+ "agent-prompt-audit",
"background-watch-hook",
"use-avibe",
"use-avibe-harness",