The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
agent-improve: bot self-retrospectives (#73130)
Nathan Voxland35b9869301bd
diff --git a/.claude/commands/agent-improve.md b/.claude/commands/agent-improve.mdnew file mode 100644index 00000000000..26232c92ce9--- /dev/null+++ b/.claude/commands/agent-improve.md@@ -0,0 +1,82 @@+Review this conversation and generate a structured improvement report.++**Precondition:** If the conversation contains no substantive task work prior to this invocation (no edits, no non-trivial tool calls, no user task beyond invoking `/agent-improve`), respond with a single line stating there is nothing to analyze and exit without generating `OUTPUT_DIR` or writing `SUGGESTIONS.md`. Do not run `bin/mage -bot-timestamp` in that case.++Generate `TIMESTAMP` by running `./bin/mage -bot-timestamp` (do **NOT** use `date` directly), then set `OUTPUT_DIR=.bot/agent-improve/<TIMESTAMP>` and save the report to `<OUTPUT_DIR>/SUGGESTIONS.md`. All artifacts for the run live directly under `<OUTPUT_DIR>/` so parallel `/agent-improve` invocations in the same checkout never collide.++## What to analyze++Read back through the full conversation history from this session. Look at every tool call, every error, every retry, every place you or the user got stuck or went in circles. Categorize what you find.++## Categories to evaluate++### 1. Tool issues+- Tools that failed, returned unexpected results, or required workarounds+- Tools you tried to use but couldn't (missing permissions, wrong arguments, etc.)+- Tools you used inefficiently (e.g., multiple calls where one would suffice)++### 2. Prompt gaps+- Steps that were missing from the initial prompt that had to be figured out on the fly+- Instructions that were ambiguous or led down the wrong path+- Steps that were in the wrong order or had missing prerequisites+- Situations where clarification was needed that should have been covered in the prompt++### 3. Knowledge gaps+- Facts about the codebase, architecture, or conventions discovered through trial and error+- Incorrect assumptions about how the code works+- Database schemas, API endpoints, config options, or other reference info that would have saved time if documented upfront++### 4. Memory-worthy patterns+- Stable lessons learned that would benefit future sessions (not one-off fixes)+- Common failure modes and their solutions+- Patterns about how this codebase works that aren't obvious from reading the code++### 5. Workflow inefficiencies+- Places where work was repeated unnecessarily+- Dead ends — what signal should have indicated earlier to change approach?+- Test/verify cycles that could have been shorter++## Report format++Write `<OUTPUT_DIR>/SUGGESTIONS.md` with this structure:++```markdown+# Session Improvement Report++**Session:** <brief description of what was being done>+**Date:** <today's date>+**Outcome:** <succeeded / partially succeeded / failed>+**Transcript:** <absolute path to this conversation's transcript JSONL file — derive the project dir by replacing `/` with `-` in the absolute working directory and prefixing `~/.claude/projects/`, then `ls -lt <that dir>/*.jsonl | head -1`>++## Summary++<2-3 sentence summary of the session and the main improvement themes>++## Suggestions++### <Category name>++#### <Short title for the suggestion>++**What happened:** <Describe the specific problem encountered>+**Where to fix:** <Exact file path and section/line range>+**Proposed change:** <The actual text to add, remove, or modify — be specific enough that someone can apply it directly>+**Impact:** <What this would have saved — time, retries, user interruptions, etc.>++(Repeat for each suggestion, grouped by category)++## Quick wins++<Bulleted list of the 3-5 highest-impact, lowest-effort changes>+```++## Rules++- **Be specific** — include exact file paths, section names, line ranges, and actual proposed text changes. Vague suggestions like "improve error handling" are useless.+- **Be actionable** — every suggestion must propose a concrete fix that can be applied. If you can't propose a specific change, it's not ready to be a suggestion.+- **Be honest** — if the mistake was due to reasoning (not a prompt gap), say so. Not everything is the prompt's fault.+- **Prioritize by impact** — order suggestions within each category by how much time/frustration they would save in future sessions.+- **Include prompt text diffs** — for prompt gap suggestions, show the exact text you'd add to the relevant prompt or agent file as a before/after or insertion.+- **Cover the orchestrator too** — if the issue was in how the session was set up (database choice, prompt generation, environment config), suggest changes to the relevant `.claude/commands/*.md` file as well.+- **Don't suggest things that are already documented** — if the answer was in the prompt and was just missed, that's a different problem than a missing instruction. Note which it was.+- **Print the absolute path** — after writing the file, print the full absolute path to the suggestions file so the user can easily find it.diff --git a/.claude/commands/autobot.md b/.claude/commands/autobot.mdindex 57ac4c151de..223d17179f9 100644--- a/.claude/commands/autobot.md+++ b/.claude/commands/autobot.md@@ -8,6 +8,8 @@ The user provided: `$ARGUMENTS`Parse as: `<branch-name-or-pr-env-url> [from <base-branch>] <inner-command> [inner-args...]`+**Arg-order sanity check:** If the first token starts with `/` (e.g. the user wrote `/autobot /qabot my-branch`), stop immediately and tell the user the correct order is `/autobot <branch> [from <base>] /<command> [args]`. Do not try to guess.+The first token can be either a branch name OR a PR preview environment URL:- **Branch name** (e.g., `master`, `my-feature-branch`) — launches a local dev environment in a worktree (normal mode).@@ -40,6 +42,8 @@ Extract the bot name from the inner command by stripping the leading `/` (e.g.,### 2. Preflight checks+**Where inner-bot files live:** Inner-bot slash commands are at `.claude/commands/<bot-name>*.md` (e.g. `.claude/commands/qabot.md`, `.claude/commands/qabot-discover.md`). They are NOT under `.claude/skills/`. Read them directly with `Read`; do not `find`/`grep` `.claude/skills/` looking for them.+#### Autobot infrastructure checksVerify these are available (stop if any fail):@@ -58,6 +62,8 @@ Generate a timestamp in `YYYYMMDD-HHMMSS` format (use the current wall-clock tim- `TIMESTAMP=<YYYYMMDD-HHMMSS>`- `OUTPUT_DIR=.bot/<BOT_NAME>/<TIMESTAMP>`+Run `mkdir -p <OUTPUT_DIR>` before invoking discover so subsequent file copies / writes never fail on a missing parent directory. The discover skill must be the only producer of artifacts inside `<OUTPUT_DIR>` — do not pre-copy `linear-context.txt` from a previous run; let discover regenerate it.+Run the `/<bot-name>-discover` skill, passing the inner-args plus `--output-dir <OUTPUT_DIR>`. For example, if the command is `/fixbot MB-12345`, run `/fixbot-discover MB-12345 --output-dir <OUTPUT_DIR>`.This will:diff --git a/.claude/commands/fixbot.md b/.claude/commands/fixbot.mdindex bda296d1cf4..51b7931d0fb 100644--- a/.claude/commands/fixbot.md+++ b/.claude/commands/fixbot.md@@ -10,7 +10,7 @@ Set:- `TIMESTAMP=<YYYYMMDD-HHMMSS>`- `OUTPUT_DIR=.bot/fixbot/<TIMESTAMP>`-Every file this run writes — including discover artifacts — lives under `<OUTPUT_DIR>/`. There must be **no shared paths across runs**, so multiple `/fixbot` invocations in the same repo do not collide.+Run `mkdir -p <OUTPUT_DIR>` immediately so any subsequent `cp`/`Write` calls into it succeed. Every file this run writes — including discover artifacts — lives under `<OUTPUT_DIR>/`. There must be **no shared paths across runs**, so multiple `/fixbot` invocations in the same repo do not collide. Do not copy artifacts (e.g. `linear-context.txt`) from a previous run's directory; let discover regenerate them.### 2. Gather contextdiff --git a/.claude/commands/reprobot.md b/.claude/commands/reprobot.mdindex 05c03773579..8e0be0fe32a 100644--- a/.claude/commands/reprobot.md+++ b/.claude/commands/reprobot.md@@ -10,7 +10,7 @@ Set:- `TIMESTAMP=<YYYYMMDD-HHMMSS>`- `OUTPUT_DIR=.bot/reprobot/<TIMESTAMP>`-Every file this run writes — including discover artifacts — lives under `<OUTPUT_DIR>/`. There must be **no shared paths across runs**, so multiple `/reprobot` invocations in the same repo do not collide.+Run `mkdir -p <OUTPUT_DIR>` immediately so subsequent `cp`/`Write` calls into it succeed. Every file this run writes — including discover artifacts — lives under `<OUTPUT_DIR>/`. There must be **no shared paths across runs**, so multiple `/reprobot` invocations in the same repo do not collide. Do not copy artifacts from a previous run's directory; let discover regenerate them.### 2. Gather contextdiff --git a/dev/bot/common/playwright-guide.md b/.claude/skills/playwright-mcp-metabase/SKILL.mdsimilarity index 76%rename from dev/bot/common/playwright-guide.mdrename to .claude/skills/playwright-mcp-metabase/SKILL.mdindex 8a9e00634b5..9a9ea2c4c14 100644--- a/dev/bot/common/playwright-guide.md+++ b/.claude/skills/playwright-mcp-metabase/SKILL.md@@ -1,10 +1,15 @@-## Browser Automation with Playwright MCP+---+name: playwright-mcp-metabase+description: Drive Metabase's UI with the Playwright MCP browser tools (mcp__playwright__browser_*). Covers the snapshot/act/check pattern, Mantine component pitfalls (Menu race, Select/MultiSelect, the Escape-closes-modal trap, portal scoping), and Metabase-specific login flows. Use whenever a session needs to interact with the Metabase UI through Playwright MCP.+---++# Browser Automation with Playwright MCP (Metabase)A Playwright MCP server is configured in `.mcp.json`. Load the tool schemas first with `ToolSearch`: `select:mcp__playwright__browser_navigate,mcp__playwright__browser_snapshot,mcp__playwright__browser_click,mcp__playwright__browser_fill,mcp__playwright__browser_type,mcp__playwright__browser_press_key,mcp__playwright__browser_hover,mcp__playwright__browser_take_screenshot,mcp__playwright__browser_close`If ToolSearch says "MCP servers still connecting," wait a few seconds and retry — the server takes a moment to start on first use.-### Core pattern: Snapshot → Act → Check+## Core pattern: Snapshot → Act → Check1. **`browser_snapshot`** — see what's on screen, get element refs2. **Act** — `browser_click`, `browser_fill`, `browser_type`, etc. using refs from the snapshot@@ -22,7 +27,7 @@ If ToolSearch says "MCP servers still connecting," wait a few seconds and retry**Element refs go stale after every action.** Use refs from the most recent snapshot or inline response — never from an earlier one.-### How to interact with Metabase's UI components+## How to interact with Metabase's UI componentsMetabase uses Mantine UI components. Most interactions work with a plain `browser_click`. The hover-before-click pattern is only needed for specific component types.@@ -47,14 +52,18 @@ You can also type into the input to filter options before clicking.**Modals and dialogs:**To dismiss: `browser_click` the close/action button, or `browser_press_key` with `Escape`.-### Login+**⚠️ Mantine `Escape`-inside-modal trap:** Pressing `Escape` while focus is inside a Mantine popover/Select/MultiSelect that is *itself* nested in a modal closes the **entire modal** and discards every form field you've already filled. To dismiss just an open dropdown, prefer `browser_click` outside the dropdown (or click another field). Only press `Escape` when you actually want the whole modal gone.++**Mantine portals + scoped snapshots:** Mantine renders dropdowns, popovers, and tooltips into a portal at `<body>` level — they are NOT children of the `[role=dialog]` element. If you scope `browser_snapshot` to the modal, dropdown options will be missing. Take a full-page snapshot when interacting with portaled content.++## Login1. `browser_navigate` to `http://localhost:$MB_JETTY_PORT/auth/login`2. `browser_snapshot` → `browser_fill` email and password → `browser_click` sign-in button3. **`browser_navigate` to `http://localhost:$MB_JETTY_PORT/`** — always navigate explicitly to the home page after login. Do NOT rely on the login redirect alone. The redirect can leave the browser session in a state where clicks don't register.4. `browser_snapshot` to confirm you're logged in-### Quick login via API (alternative)+## Quick login via API (alternative)Get a session token via API and set it as a cookie — faster than filling the login form:```bash@@ -66,11 +75,11 @@ document.cookie = 'metabase.SESSION=<token>;path=/'```Then `browser_navigate` to the target page. Use the credentials from `./bin/mage -bot-server-info`.-### When clicks don't work+## When clicks don't work1. Take a fresh snapshot (async rendering). 2. Try hover + click (menu triggers). 3. Try keyboard (focus + Enter). 4. After 3 attempts, report and move on.-### Rules+## Rules- Always use `http://localhost:$MB_JETTY_PORT`- Close the browser when donediff --git a/dev/bot/fixbot-agent.md b/dev/bot/fixbot-agent.mdindex 2ae6fde149e..792b522661f 100644--- a/dev/bot/fixbot-agent.md+++ b/dev/bot/fixbot-agent.md@@ -135,7 +135,9 @@ After submitting the pull request, monitor the pull request until it passes. NOT1. Run `/cibot` to monitor CI results and handle failures-{{FILE:dev/bot/common/playwright-guide.md}}+## Browser Automation++For browser interaction with Metabase, use the **`playwright-mcp-metabase`** skill — it auto-loads on Playwright MCP tool use and documents the snapshot/act/check pattern, Mantine component pitfalls (Menu race, Select/MultiSelect, the Escape-closes-modal trap, portal scoping), and Metabase login flows.**This is optional for fixbot.** For purely backend fixes where the issue and verification are API-level or logic-level, you don't need the browser at all — automated tests and nREPL are sufficient. Use the Playwright MCP tools when the issue involves UI behavior, when you need to see what the user sees, or when the user reports something that doesn't match what you'd expect from the code.diff --git a/dev/bot/reprobot-agent.md b/dev/bot/reprobot-agent.mdindex 4f270b2e998..c35e5cea48d 100644--- a/dev/bot/reprobot-agent.md+++ b/dev/bot/reprobot-agent.md@@ -193,7 +193,9 @@ Show the user the report path and a brief summary using a banner:----{{FILE:dev/bot/common/playwright-guide.md}}+## Browser Automation++For browser interaction with Metabase, use the **`playwright-mcp-metabase`** skill — it auto-loads on Playwright MCP tool use and documents the snapshot/act/check pattern, Mantine component pitfalls (Menu race, Select/MultiSelect, the Escape-closes-modal trap, portal scoping), and Metabase login flows.## Important Rulesdiff --git a/mage/src/mage/bot/autobot.clj b/mage/src/mage/bot/autobot.cljindex 295b428f940..834d94837bb 100644--- a/mage/src/mage/bot/autobot.clj+++ b/mage/src/mage/bot/autobot.clj@@ -250,7 +250,7 @@(shell/sh* {:quiet? true}"tmux" "list-sessions" "-F" "#{session_name}")others (when (zero? exit)- (->> (str/split-lines (or out ""))+ (->> (or out [])(remove #(or (str/blank? %) (= % bootstrap)))seq))](when others