Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(auto-improve): add eval gate examples

AkitaOnRails6b48cbe0ddce
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 67096294..5c23938e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -31,6 +31,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`POST /admin/auto-improve/report` and `ai-memory auto-improve-report`, with
JSON and human CLI output for recent run counts, proposal outcomes, terminal
rates, and operational findings without staging pending proposals.
+- Added `docs/auto-improve-eval-gates.md` plus dependency-free Python and shell
+ scorer templates for `[auto_improve.eval]` proposal gates.
- Hook spool drains now use `POST /hook/batch` when the server supports it,
grouping compatible queued lifecycle events into bounded batches to amortize
remote request latency. Older servers fall back to the existing per-event
diff --git a/README.md b/README.md
index b21a9d01..1e2c6e44 100644
--- a/README.md
+++ b/README.md
@@ -143,7 +143,9 @@ priors are at the [bottom](#influences-and-prior-art).
available for manual catch-up or targeted reruns. `ai-memory
auto-improve-report --workspace <w> --project <p>` returns a read-only
telemetry report for recent auto-improvement outcomes without staging or
- creating proposals.
+ creating proposals. See
+ [`docs/auto-improve-eval-gates.md`](docs/auto-improve-eval-gates.md) for
+ example executable eval scorers.
Existing installs do not need per-project migration. The scheduler initializes
a per-project first-run watermark so historical sessions are not reviewed
diff --git a/docs/auto-improve-eval-gates.md b/docs/auto-improve-eval-gates.md
new file mode 100644
index 00000000..0ac18af1
--- /dev/null
+++ b/docs/auto-improve-eval-gates.md
@@ -0,0 +1,93 @@
+# Auto-Improve Eval Gates
+
+`[auto_improve.eval]` lets operators guard high-impact auto-improvement
+proposals with a small executable scorer. The scorer runs after LLM validation
+and before staging or auto-approval. Hooks never run eval commands.
+
+## Configuration
+
+```toml
+[auto_improve.eval]
+enabled = true
+command = "python3 docs/examples/auto-improve-eval/score_proposal.py"
+timeout_secs = 30
+targets = ["_rules", "procedures"]
+min_delta = 0.0
+```
+
+The command is split on whitespace and executed directly, not through a shell.
+Use a wrapper script if you need quoting, environment setup, or multiple
+commands.
+
+## Request contract
+
+The scorer receives one JSON object on stdin:
+
+```json
+{
+ "path": "procedures/release.md",
+ "kind": "procedure",
+ "operation": "update",
+ "edit_mode": "patch",
+ "title": "Release Procedure",
+ "confidence": 0.91,
+ "rationale": "Capture the repeated release checklist.",
+ "before_body": "# Release Procedure\n\n## Steps\n- Run tests\n",
+ "after_body": "# Release Procedure\n\n## Steps\n- Run tests\n- Run deploy smoke checks\n",
+ "expected_base_body_sha256": "..."
+}
+```
+
+`before_body` is empty for create proposals. `expected_base_body_sha256` is
+present only for patch proposals that were materialized against a known base.
+
+## Response contract
+
+The scorer must print one JSON object to stdout:
+
+```json
+{ "score_before": 0.72, "score_after": 0.76, "passed": true }
+```
+
+Fields:
+
+- `passed` is required. `false` rejects the targeted proposal.
+- `score_before` and `score_after` are optional. When both are present,
+ `score_after - score_before` must be at least `min_delta`.
+- `reason` is optional and should explain a rejection in one short sentence.
+
+Command errors, timeouts, invalid JSON, missing `passed`, `passed = false`, and
+insufficient score delta all fail closed for the targeted proposal. Other
+proposals in the same run can still proceed.
+
+## Scorer design rules
+
+- Keep scorers deterministic, fast, and side-effect-free.
+- Read only stdin and local project files that are safe to inspect.
+- Do not call LLMs, mutate files, run deploys, or depend on network services.
+- Return bounded reasons; ai-memory caps captured eval evidence.
+- Prefer simple checks that match the target path: heading structure for
+ procedures, forbidden placeholders for `_rules`, or project-specific smoke
+ assertions for critical docs.
+
+## Examples
+
+This repository includes two dependency-free templates:
+
+- [`docs/examples/auto-improve-eval/score_proposal.py`](examples/auto-improve-eval/score_proposal.py)
+ — Python scorer that checks basic structure and placeholders.
+- [`docs/examples/auto-improve-eval/score_proposal.sh`](examples/auto-improve-eval/score_proposal.sh)
+ — POSIX shell wrapper around an embedded Python scorer for hosts that prefer a
+ script entrypoint.
+
+Try them with the sample payload:
+
+```bash
+python3 docs/examples/auto-improve-eval/score_proposal.py \
+ < docs/examples/auto-improve-eval/sample-proposal.json
+
+sh docs/examples/auto-improve-eval/score_proposal.sh \
+ < docs/examples/auto-improve-eval/sample-proposal.json
+```
+
+Both print compact JSON suitable for ai-memory's eval gate.
diff --git a/docs/auto-improvement-loop.md b/docs/auto-improvement-loop.md
index 30c313f4..b1e2ef48 100644
--- a/docs/auto-improvement-loop.md
+++ b/docs/auto-improvement-loop.md
@@ -193,7 +193,9 @@ timeouts, invalid JSON, `passed = false`, missing `passed`, or score deltas belo
`min_delta` reject only that targeted proposal. Non-targeted proposals bypass the
gate. If all proposals fail eval, the run is still staged with zero proposals and
the rejected candidates so the rejection buffer can remember the failed attempt.
-Hooks never run the eval command.
+Hooks never run the eval command. See
+[`auto-improve-eval-gates.md`](auto-improve-eval-gates.md) for the full stdin /
+stdout contract and example scorer scripts.
## Proposed Page Targets
diff --git a/docs/examples/auto-improve-eval/README.md b/docs/examples/auto-improve-eval/README.md
new file mode 100644
index 00000000..95d5afa6
--- /dev/null
+++ b/docs/examples/auto-improve-eval/README.md
@@ -0,0 +1,27 @@
+# Auto-Improve Eval Example Scorers
+
+These examples are intentionally small, deterministic, and dependency-free.
+They read one proposal JSON object from stdin and print one eval response JSON
+object to stdout.
+
+Use them as templates, not as universal quality gates. Real projects should
+replace the checks with project-specific invariants that are safe to run before
+an auto-improvement proposal is staged.
+
+```toml
+[auto_improve.eval]
+enabled = true
+command = "python3 docs/examples/auto-improve-eval/score_proposal.py"
+timeout_secs = 30
+targets = ["_rules", "procedures"]
+min_delta = 0.0
+```
+
+Smoke test:
+
+```bash
+python3 docs/examples/auto-improve-eval/score_proposal.py < \
+ docs/examples/auto-improve-eval/sample-proposal.json
+sh docs/examples/auto-improve-eval/score_proposal.sh < \
+ docs/examples/auto-improve-eval/sample-proposal.json
+```
diff --git a/docs/examples/auto-improve-eval/sample-proposal.json b/docs/examples/auto-improve-eval/sample-proposal.json
new file mode 100644
index 00000000..f240e44b
--- /dev/null
+++ b/docs/examples/auto-improve-eval/sample-proposal.json
@@ -0,0 +1,12 @@
+{
+ "path": "procedures/release.md",
+ "kind": "procedure",
+ "operation": "update",
+ "edit_mode": "full_page",
+ "title": "Release Procedure",
+ "confidence": 0.91,
+ "rationale": "Capture the release smoke-test checklist.",
+ "before_body": "# Release Procedure\n\n## Steps\n- Run tests\n",
+ "after_body": "# Release Procedure\n\n## Purpose\nShip a verified ai-memory build.\n\n## Steps\n- Run the full local gate.\n- Deploy to the homelab service.\n- Check /admin/status.\n",
+ "expected_base_body_sha256": null
+}
diff --git a/docs/examples/auto-improve-eval/score_proposal.py b/docs/examples/auto-improve-eval/score_proposal.py
new file mode 100644
index 00000000..d573a3de
--- /dev/null
+++ b/docs/examples/auto-improve-eval/score_proposal.py
@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+"""Example ai-memory auto-improve eval scorer.
+
+Reads proposal JSON from stdin and emits:
+ {"passed": bool, "score_before": float, "score_after": float, "reason": str?}
+
+The checks are intentionally simple and deterministic:
+- procedures should have a Purpose/Steps shape;
+- rules should avoid placeholders and use imperative language;
+- all targeted docs should avoid TODO placeholders.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import sys
+
+
+def score_body(path: str, body: str) -> tuple[float, list[str]]:
+ score = 0.0
+ reasons: list[str] = []
+ lowered = body.lower()
+
+ if re.search(r"^#\s+\S", body, re.MULTILINE):
+ score += 0.25
+ else:
+ reasons.append("missing H1 heading")
+
+ if "todo" not in lowered and "tbd" not in lowered:
+ score += 0.20
+ else:
+ reasons.append("contains TODO/TBD placeholder")
+
+ if path.startswith("procedures/"):
+ if "## purpose" in lowered or "## overview" in lowered:
+ score += 0.20
+ else:
+ reasons.append("procedure lacks Purpose/Overview section")
+ if "## steps" in lowered or "## checklist" in lowered:
+ score += 0.25
+ else:
+ reasons.append("procedure lacks Steps/Checklist section")
+ elif path.startswith("_rules/"):
+ if re.search(r"\b(always|never|must|do not|prefer)\b", lowered):
+ score += 0.35
+ else:
+ reasons.append("rule lacks imperative guidance")
+ if len(body.strip()) <= 2400:
+ score += 0.10
+ else:
+ reasons.append("rule is too long for an agent instruction")
+ else:
+ score += 0.45
+
+ return min(score, 1.0), reasons
+
+
+def main() -> int:
+ try:
+ payload = json.load(sys.stdin)
+ except json.JSONDecodeError as exc:
+ print(json.dumps({"passed": False, "reason": f"invalid input JSON: {exc}"}))
+ return 0
+
+ path = str(payload.get("path", ""))
+ before_body = str(payload.get("before_body", ""))
+ after_body = str(payload.get("after_body", ""))
+
+ score_before, _ = score_body(path, before_body)
+ score_after, reasons = score_body(path, after_body)
+ passed = score_after >= 0.70 and score_after >= score_before
+
+ response: dict[str, object] = {
+ "passed": passed,
+ "score_before": round(score_before, 4),
+ "score_after": round(score_after, 4),
+ }
+ if not passed:
+ response["reason"] = "; ".join(reasons) or "score did not improve"
+
+ print(json.dumps(response, separators=(",", ":")))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/docs/examples/auto-improve-eval/score_proposal.sh b/docs/examples/auto-improve-eval/score_proposal.sh
new file mode 100644
index 00000000..251c7b48
--- /dev/null
+++ b/docs/examples/auto-improve-eval/score_proposal.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+# POSIX-shell entrypoint for ai-memory auto-improve eval gates.
+# It embeds a tiny Python scorer so the configured command can be a single
+# script path while keeping JSON parsing correct.
+
+python3 -c '
+import json, re, sys
+
+try:
+ payload = json.load(sys.stdin)
+except json.JSONDecodeError as exc:
+ print(json.dumps({"passed": False, "reason": f"invalid input JSON: {exc}"}))
+ raise SystemExit(0)
+
+path = str(payload.get("path", ""))
+body = str(payload.get("after_body", ""))
+lowered = body.lower()
+checks = [
+ bool(re.search(r"^#\s+\S", body, re.MULTILINE)),
+ "todo" not in lowered and "tbd" not in lowered,
+]
+if path.startswith("procedures/"):
+ checks.extend([
+ "## steps" in lowered or "## checklist" in lowered,
+ "## purpose" in lowered or "## overview" in lowered,
+ ])
+elif path.startswith("_rules/"):
+ checks.append(bool(re.search(r"\b(always|never|must|do not|prefer)\b", lowered)))
+
+score_after = sum(1 for ok in checks if ok) / len(checks)
+passed = score_after >= 0.75
+response = {"passed": passed, "score_after": round(score_after, 4)}
+if not passed:
+ response["reason"] = "proposal did not satisfy the example structure checks"
+print(json.dumps(response, separators=(",", ":")))
+'