Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs(auto-improve): add eval gate examples
AkitaOnRails6b48cbe0ddce
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 67096294..5c23938e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -31,6 +31,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`POST /admin/auto-improve/report` and `ai-memory auto-improve-report`, withJSON and human CLI output for recent run counts, proposal outcomes, terminalrates, and operational findings without staging pending proposals.+- Added `docs/auto-improve-eval-gates.md` plus dependency-free Python and shell+ scorer templates for `[auto_improve.eval]` proposal gates.- Hook spool drains now use `POST /hook/batch` when the server supports it,grouping compatible queued lifecycle events into bounded batches to amortizeremote request latency. Older servers fall back to the existing per-eventdiff --git a/README.md b/README.mdindex b21a9d01..1e2c6e44 100644--- a/README.md+++ b/README.md@@ -143,7 +143,9 @@ priors are at the [bottom](#influences-and-prior-art).available for manual catch-up or targeted reruns. `ai-memoryauto-improve-report --workspace <w> --project <p>` returns a read-onlytelemetry report for recent auto-improvement outcomes without staging or- creating proposals.+ creating proposals. See+ [`docs/auto-improve-eval-gates.md`](docs/auto-improve-eval-gates.md) for+ example executable eval scorers.Existing installs do not need per-project migration. The scheduler initializesa per-project first-run watermark so historical sessions are not revieweddiff --git a/docs/auto-improve-eval-gates.md b/docs/auto-improve-eval-gates.mdnew file mode 100644index 00000000..0ac18af1--- /dev/null+++ b/docs/auto-improve-eval-gates.md@@ -0,0 +1,93 @@+# Auto-Improve Eval Gates++`[auto_improve.eval]` lets operators guard high-impact auto-improvement+proposals with a small executable scorer. The scorer runs after LLM validation+and before staging or auto-approval. Hooks never run eval commands.++## Configuration++```toml+[auto_improve.eval]+enabled = true+command = "python3 docs/examples/auto-improve-eval/score_proposal.py"+timeout_secs = 30+targets = ["_rules", "procedures"]+min_delta = 0.0+```++The command is split on whitespace and executed directly, not through a shell.+Use a wrapper script if you need quoting, environment setup, or multiple+commands.++## Request contract++The scorer receives one JSON object on stdin:++```json+{+ "path": "procedures/release.md",+ "kind": "procedure",+ "operation": "update",+ "edit_mode": "patch",+ "title": "Release Procedure",+ "confidence": 0.91,+ "rationale": "Capture the repeated release checklist.",+ "before_body": "# Release Procedure\n\n## Steps\n- Run tests\n",+ "after_body": "# Release Procedure\n\n## Steps\n- Run tests\n- Run deploy smoke checks\n",+ "expected_base_body_sha256": "..."+}+```++`before_body` is empty for create proposals. `expected_base_body_sha256` is+present only for patch proposals that were materialized against a known base.++## Response contract++The scorer must print one JSON object to stdout:++```json+{ "score_before": 0.72, "score_after": 0.76, "passed": true }+```++Fields:++- `passed` is required. `false` rejects the targeted proposal.+- `score_before` and `score_after` are optional. When both are present,+ `score_after - score_before` must be at least `min_delta`.+- `reason` is optional and should explain a rejection in one short sentence.++Command errors, timeouts, invalid JSON, missing `passed`, `passed = false`, and+insufficient score delta all fail closed for the targeted proposal. Other+proposals in the same run can still proceed.++## Scorer design rules++- Keep scorers deterministic, fast, and side-effect-free.+- Read only stdin and local project files that are safe to inspect.+- Do not call LLMs, mutate files, run deploys, or depend on network services.+- Return bounded reasons; ai-memory caps captured eval evidence.+- Prefer simple checks that match the target path: heading structure for+ procedures, forbidden placeholders for `_rules`, or project-specific smoke+ assertions for critical docs.++## Examples++This repository includes two dependency-free templates:++- [`docs/examples/auto-improve-eval/score_proposal.py`](examples/auto-improve-eval/score_proposal.py)+ — Python scorer that checks basic structure and placeholders.+- [`docs/examples/auto-improve-eval/score_proposal.sh`](examples/auto-improve-eval/score_proposal.sh)+ — POSIX shell wrapper around an embedded Python scorer for hosts that prefer a+ script entrypoint.++Try them with the sample payload:++```bash+python3 docs/examples/auto-improve-eval/score_proposal.py \+ < docs/examples/auto-improve-eval/sample-proposal.json++sh docs/examples/auto-improve-eval/score_proposal.sh \+ < docs/examples/auto-improve-eval/sample-proposal.json+```++Both print compact JSON suitable for ai-memory's eval gate.diff --git a/docs/auto-improvement-loop.md b/docs/auto-improvement-loop.mdindex 30c313f4..b1e2ef48 100644--- a/docs/auto-improvement-loop.md+++ b/docs/auto-improvement-loop.md@@ -193,7 +193,9 @@ timeouts, invalid JSON, `passed = false`, missing `passed`, or score deltas belo`min_delta` reject only that targeted proposal. Non-targeted proposals bypass thegate. If all proposals fail eval, the run is still staged with zero proposals andthe rejected candidates so the rejection buffer can remember the failed attempt.-Hooks never run the eval command.+Hooks never run the eval command. See+[`auto-improve-eval-gates.md`](auto-improve-eval-gates.md) for the full stdin /+stdout contract and example scorer scripts.## Proposed Page Targetsdiff --git a/docs/examples/auto-improve-eval/README.md b/docs/examples/auto-improve-eval/README.mdnew file mode 100644index 00000000..95d5afa6--- /dev/null+++ b/docs/examples/auto-improve-eval/README.md@@ -0,0 +1,27 @@+# Auto-Improve Eval Example Scorers++These examples are intentionally small, deterministic, and dependency-free.+They read one proposal JSON object from stdin and print one eval response JSON+object to stdout.++Use them as templates, not as universal quality gates. Real projects should+replace the checks with project-specific invariants that are safe to run before+an auto-improvement proposal is staged.++```toml+[auto_improve.eval]+enabled = true+command = "python3 docs/examples/auto-improve-eval/score_proposal.py"+timeout_secs = 30+targets = ["_rules", "procedures"]+min_delta = 0.0+```++Smoke test:++```bash+python3 docs/examples/auto-improve-eval/score_proposal.py < \+ docs/examples/auto-improve-eval/sample-proposal.json+sh docs/examples/auto-improve-eval/score_proposal.sh < \+ docs/examples/auto-improve-eval/sample-proposal.json+```diff --git a/docs/examples/auto-improve-eval/sample-proposal.json b/docs/examples/auto-improve-eval/sample-proposal.jsonnew file mode 100644index 00000000..f240e44b--- /dev/null+++ b/docs/examples/auto-improve-eval/sample-proposal.json@@ -0,0 +1,12 @@+{+ "path": "procedures/release.md",+ "kind": "procedure",+ "operation": "update",+ "edit_mode": "full_page",+ "title": "Release Procedure",+ "confidence": 0.91,+ "rationale": "Capture the release smoke-test checklist.",+ "before_body": "# Release Procedure\n\n## Steps\n- Run tests\n",+ "after_body": "# Release Procedure\n\n## Purpose\nShip a verified ai-memory build.\n\n## Steps\n- Run the full local gate.\n- Deploy to the homelab service.\n- Check /admin/status.\n",+ "expected_base_body_sha256": null+}diff --git a/docs/examples/auto-improve-eval/score_proposal.py b/docs/examples/auto-improve-eval/score_proposal.pynew file mode 100644index 00000000..d573a3de--- /dev/null+++ b/docs/examples/auto-improve-eval/score_proposal.py@@ -0,0 +1,87 @@+#!/usr/bin/env python3+"""Example ai-memory auto-improve eval scorer.++Reads proposal JSON from stdin and emits:+ {"passed": bool, "score_before": float, "score_after": float, "reason": str?}++The checks are intentionally simple and deterministic:+- procedures should have a Purpose/Steps shape;+- rules should avoid placeholders and use imperative language;+- all targeted docs should avoid TODO placeholders.+"""++from __future__ import annotations++import json+import re+import sys+++def score_body(path: str, body: str) -> tuple[float, list[str]]:+ score = 0.0+ reasons: list[str] = []+ lowered = body.lower()++ if re.search(r"^#\s+\S", body, re.MULTILINE):+ score += 0.25+ else:+ reasons.append("missing H1 heading")++ if "todo" not in lowered and "tbd" not in lowered:+ score += 0.20+ else:+ reasons.append("contains TODO/TBD placeholder")++ if path.startswith("procedures/"):+ if "## purpose" in lowered or "## overview" in lowered:+ score += 0.20+ else:+ reasons.append("procedure lacks Purpose/Overview section")+ if "## steps" in lowered or "## checklist" in lowered:+ score += 0.25+ else:+ reasons.append("procedure lacks Steps/Checklist section")+ elif path.startswith("_rules/"):+ if re.search(r"\b(always|never|must|do not|prefer)\b", lowered):+ score += 0.35+ else:+ reasons.append("rule lacks imperative guidance")+ if len(body.strip()) <= 2400:+ score += 0.10+ else:+ reasons.append("rule is too long for an agent instruction")+ else:+ score += 0.45++ return min(score, 1.0), reasons+++def main() -> int:+ try:+ payload = json.load(sys.stdin)+ except json.JSONDecodeError as exc:+ print(json.dumps({"passed": False, "reason": f"invalid input JSON: {exc}"}))+ return 0++ path = str(payload.get("path", ""))+ before_body = str(payload.get("before_body", ""))+ after_body = str(payload.get("after_body", ""))++ score_before, _ = score_body(path, before_body)+ score_after, reasons = score_body(path, after_body)+ passed = score_after >= 0.70 and score_after >= score_before++ response: dict[str, object] = {+ "passed": passed,+ "score_before": round(score_before, 4),+ "score_after": round(score_after, 4),+ }+ if not passed:+ response["reason"] = "; ".join(reasons) or "score did not improve"++ print(json.dumps(response, separators=(",", ":")))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/docs/examples/auto-improve-eval/score_proposal.sh b/docs/examples/auto-improve-eval/score_proposal.shnew file mode 100644index 00000000..251c7b48--- /dev/null+++ b/docs/examples/auto-improve-eval/score_proposal.sh@@ -0,0 +1,36 @@+#!/bin/sh+# POSIX-shell entrypoint for ai-memory auto-improve eval gates.+# It embeds a tiny Python scorer so the configured command can be a single+# script path while keeping JSON parsing correct.++python3 -c '+import json, re, sys++try:+ payload = json.load(sys.stdin)+except json.JSONDecodeError as exc:+ print(json.dumps({"passed": False, "reason": f"invalid input JSON: {exc}"}))+ raise SystemExit(0)++path = str(payload.get("path", ""))+body = str(payload.get("after_body", ""))+lowered = body.lower()+checks = [+ bool(re.search(r"^#\s+\S", body, re.MULTILINE)),+ "todo" not in lowered and "tbd" not in lowered,+]+if path.startswith("procedures/"):+ checks.extend([+ "## steps" in lowered or "## checklist" in lowered,+ "## purpose" in lowered or "## overview" in lowered,+ ])+elif path.startswith("_rules/"):+ checks.append(bool(re.search(r"\b(always|never|must|do not|prefer)\b", lowered)))++score_after = sum(1 for ok in checks if ok) / len(checks)+passed = score_after >= 0.75+response = {"passed": passed, "score_after": round(score_after, 4)}+if not passed:+ response["reason"] = "proposal did not satisfy the example structure checks"+print(json.dumps(response, separators=(",", ":")))+'