Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs: make the example eval scorers language-neutral and Python-free in sh
Renato Regisbba7d54eeee4
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 10e3f4b7..f6ec6fc5 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -138,6 +138,13 @@ jobs:elseecho "no tests under $dir"fi+ # The two example eval scorers must keep giving the same verdict; the+ # suite runs the Python one and the sh + awk one on the same proposals.+ - name: Auto-improve eval example scorer tests+ if: matrix.os == 'ubuntu-latest'+ env:+ PYTHONDONTWRITEBYTECODE: "1"+ run: uv run --no-project python -m unittest discover -s docs/examples/auto-improve-eval -p 'test_*.py' -v# The capture policy every generated OpenCode/OMP/Pi/OpenClaw plugin# embeds is TypeScript; only Node can execute it against the shared# capture-policy fixture. The test is ignored locally, where Node is notdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex c6a02aca..2b6a96f3 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -34,6 +34,18 @@ version in [`docs/changelog/`](docs/changelog/):rejected as containing no extractable text when that was its only cue.WebVTT allows any number of blank lines between blocks; the reader now skipsthem before reading a block.+- The example auto-improve eval scorers in `docs/examples/auto-improve-eval/`+ no longer reject pages written in a language other than English. They+ required `## Purpose`/`## Overview` and `## Steps`/`## Checklist` headings in+ procedures and English imperatives in rules, and read the Portuguese and+ Spanish word "todo" as a TODO placeholder, so a correct procedure in+ Portuguese or Japanese failed. They now check structure only: an H1, an+ upper-case TODO/TBD, a `##` section and a list of steps, text beyond the+ heading and its length counted in characters. `score_proposal.sh` no longer+ embeds Python: it is the same scorer in POSIX `sh` and `awk`, so it runs in+ the official Docker image, which has no `python3`. The docs no longer call+ the Python scorer dependency-free and say how to point the eval command at a+ script inside the container (#PRNUM).## [3.2.2] - 2026-10-22diff --git a/docs/auto-improve-eval-gates.md b/docs/auto-improve-eval-gates.mdindex 5609cc56..e2da5c85 100644--- a/docs/auto-improve-eval-gates.md+++ b/docs/auto-improve-eval-gates.md@@ -9,7 +9,7 @@ and before staging or auto-approval. Hooks never run eval commands.```toml[auto_improve.eval]enabled = true-command = "python3 docs/examples/auto-improve-eval/score_proposal.py"+command = "sh docs/examples/auto-improve-eval/score_proposal.sh"timeout_secs = 30targets = ["_rules", "procedures"]min_delta = 0.0@@ -109,16 +109,27 @@ to pending evidence; reading a report cannot restore independent observations.- Prefer simple checks that match the target path: heading structure forprocedures, forbidden placeholders for `_rules`, or project-specific smokeassertions for critical docs.+- Check structure, not wording. Pages are written in whatever language the+ team works in, so a check for an English section name or keyword rejects+ every proposal written in another language.## Examples-This repository includes two dependency-free templates:+This repository includes two templates that apply the same checks: an H1,+no TODO/TBD placeholder, a `##` section and a list of steps for procedures, and+text beyond the heading within 2,400 characters for rules. None of them looks+at section names or wording, so pages in any language score the same way.- [`docs/examples/auto-improve-eval/score_proposal.py`](examples/auto-improve-eval/score_proposal.py):- Python scorer that checks basic structure and placeholders.+ Python scorer. It needs `python3` on the host that runs the server.- [`docs/examples/auto-improve-eval/score_proposal.sh`](examples/auto-improve-eval/score_proposal.sh):- POSIX shell wrapper around an embedded Python scorer for hosts that prefer a- script entrypoint.+ the same scorer in POSIX `sh` and `awk`, with no other dependency.++The official Docker image ships no `python3`, so use the shell scorer there.+The command runs inside the container, from its working directory `/data`, and+the image does not include `docs/`: copy the script into the data volume and+point the command at that copy, for example+`command = "sh /data/eval/score_proposal.sh"`.Try them with the sample payload:diff --git a/docs/examples/auto-improve-eval/README.md b/docs/examples/auto-improve-eval/README.mdindex 95d5afa6..11261bfb 100644--- a/docs/examples/auto-improve-eval/README.md+++ b/docs/examples/auto-improve-eval/README.md@@ -1,8 +1,11 @@# Auto-Improve Eval Example Scorers-These examples are intentionally small, deterministic, and dependency-free.-They read one proposal JSON object from stdin and print one eval response JSON-object to stdout.+These examples are intentionally small and deterministic. They read one+proposal JSON object from stdin and print one eval response JSON object to+stdout. `score_proposal.py` needs `python3`; `score_proposal.sh` applies the same+checks with only POSIX `sh` and `awk`, so it also runs inside the official Docker+image, which has no Python. Neither matches section names or wording, so pages+written in any language score the same way.Use them as templates, not as universal quality gates. Real projects shouldreplace the checks with project-specific invariants that are safe to run before@@ -11,7 +14,7 @@ an auto-improvement proposal is staged.```toml[auto_improve.eval]enabled = true-command = "python3 docs/examples/auto-improve-eval/score_proposal.py"+command = "sh docs/examples/auto-improve-eval/score_proposal.sh"timeout_secs = 30targets = ["_rules", "procedures"]min_delta = 0.0@@ -25,3 +28,10 @@ python3 docs/examples/auto-improve-eval/score_proposal.py < \sh docs/examples/auto-improve-eval/score_proposal.sh < \docs/examples/auto-improve-eval/sample-proposal.json```++Both scorers must give the same verdict; `test_score_proposal.py` runs them on+the same proposals, including pages in Portuguese and Japanese:++```bash+python3 -m unittest discover -s docs/examples/auto-improve-eval -p 'test_*.py'+```diff --git a/docs/examples/auto-improve-eval/score_proposal.py b/docs/examples/auto-improve-eval/score_proposal.pyindex d573a3de..46ef2bf3 100644--- a/docs/examples/auto-improve-eval/score_proposal.py+++ b/docs/examples/auto-improve-eval/score_proposal.py@@ -4,10 +4,16 @@Reads proposal JSON from stdin and emits:{"passed": bool, "score_before": float, "score_after": float, "reason": str?}-The checks are intentionally simple and deterministic:-- procedures should have a Purpose/Steps shape;-- rules should avoid placeholders and use imperative language;-- all targeted docs should avoid TODO placeholders.+The checks are intentionally simple, deterministic, and independent of the+language the wiki is written in (section names and wording are never matched):+- procedures should have at least one `##` section and at least one list item+ for their steps;+- rules should carry text beyond their heading and stay short enough for an+ agent instruction;+- all targeted docs should have an H1 and avoid TODO/TBD placeholders (matched+ as whole upper-case words, so a word like "todo" in prose is not one).++`score_proposal.sh` applies the same checks with `sh` and `awk` only."""from __future__ import annotations@@ -16,37 +22,46 @@ import jsonimport reimport sys+LIST_ITEM = re.compile(r"^[ \t]*(?:[-*+]|[0-9]+[.)])[ \t]+\S", re.MULTILINE)+PLACEHOLDER = re.compile(r"(?<![A-Za-z0-9_])(?:TODO|TBD)(?![A-Za-z0-9_])")+++def has_text_beyond_headings(body: str) -> bool:+ return any(+ line.strip(" \t") and not line.lstrip(" \t").startswith("#")+ for line in body.split("\n")+ )+def score_body(path: str, body: str) -> tuple[float, list[str]]:score = 0.0reasons: list[str] = []- lowered = body.lower()- if re.search(r"^#\s+\S", body, re.MULTILINE):+ if re.search(r"^#[ \t]+\S", body, re.MULTILINE):score += 0.25else:reasons.append("missing H1 heading")- if "todo" not in lowered and "tbd" not in lowered:+ if not PLACEHOLDER.search(body):score += 0.20else:reasons.append("contains TODO/TBD placeholder")if path.startswith("procedures/"):- if "## purpose" in lowered or "## overview" in lowered:+ if re.search(r"^##[ \t]+\S", body, re.MULTILINE):score += 0.20else:- reasons.append("procedure lacks Purpose/Overview section")- if "## steps" in lowered or "## checklist" in lowered:+ reasons.append("procedure has no ## section")+ if LIST_ITEM.search(body):score += 0.25else:- reasons.append("procedure lacks Steps/Checklist section")+ reasons.append("procedure has no list of steps")elif path.startswith("_rules/"):- if re.search(r"\b(always|never|must|do not|prefer)\b", lowered):+ if has_text_beyond_headings(body):score += 0.35else:- reasons.append("rule lacks imperative guidance")- if len(body.strip()) <= 2400:+ reasons.append("rule has no text beyond its heading")+ if len(body.strip(" \t\r\n")) <= 2400:score += 0.10else:reasons.append("rule is too long for an agent instruction")diff --git a/docs/examples/auto-improve-eval/score_proposal.sh b/docs/examples/auto-improve-eval/score_proposal.shindex 251c7b48..aab34dfd 100644--- a/docs/examples/auto-improve-eval/score_proposal.sh+++ b/docs/examples/auto-improve-eval/score_proposal.sh@@ -1,36 +1,136 @@#!/bin/sh-# POSIX-shell entrypoint for ai-memory auto-improve eval gates.-# It embeds a tiny Python scorer so the configured command can be a single-# script path while keeping JSON parsing correct.--python3 -c '-import json, re, sys--try:- payload = json.load(sys.stdin)-except json.JSONDecodeError as exc:- print(json.dumps({"passed": False, "reason": f"invalid input JSON: {exc}"}))- raise SystemExit(0)--path = str(payload.get("path", ""))-body = str(payload.get("after_body", ""))-lowered = body.lower()-checks = [- bool(re.search(r"^#\s+\S", body, re.MULTILINE)),- "todo" not in lowered and "tbd" not in lowered,-]-if path.startswith("procedures/"):- checks.extend([- "## steps" in lowered or "## checklist" in lowered,- "## purpose" in lowered or "## overview" in lowered,- ])-elif path.startswith("_rules/"):- checks.append(bool(re.search(r"\b(always|never|must|do not|prefer)\b", lowered)))--score_after = sum(1 for ok in checks if ok) / len(checks)-passed = score_after >= 0.75-response = {"passed": passed, "score_after": round(score_after, 4)}-if not passed:- response["reason"] = "proposal did not satisfy the example structure checks"-print(json.dumps(response, separators=(",", ":")))+# POSIX sh + awk port of score_proposal.py for ai-memory auto-improve eval gates.+# It needs no Python, so it also runs inside the official Docker image, which+# ships only a POSIX shell and awk. The checks never match section names or+# wording, so a wiki written in any language scores the same way.++LC_ALL=C exec awk '+function fail(reason) {+ printf "{\"passed\":false,\"reason\":\"%s\"}\n", reason+ exit 0+}++function skip_space() {+ while (pos <= len && index(" \t\r\n", substr(doc, pos, 1)) > 0) pos+++}++# Decodes the JSON string starting at doc[pos] (the opening quote) and leaves+# pos after the closing quote. It finds the closing quote first and decodes the+# whole run with gsub: appending one character at a time copies the string on+# every step, which took 22 s on a 1 MB page. \uXXXX becomes "?": the checks+# only look at ASCII markup, and the server writes non-ASCII text as raw UTF-8.+function read_string( start, c, raw) {+ start = ++pos+ while (pos <= len) {+ c = substr(doc, pos, 1)+ if (c == "\"") break+ pos += c == "\\" ? 2 : 1+ }+ if (pos > len) fail("invalid input JSON: unterminated string")+ raw = substr(doc, start, pos - start)+ pos+++ gsub(/\\\\/, "\001", raw)+ gsub(/\\n/, "\n", raw)+ gsub(/\\t/, "\t", raw)+ gsub(/\\r/, "\r", raw)+ gsub(/\\[bf]/, " ", raw)+ gsub(/\\u[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]/, "?", raw)+ gsub(/\\"/, "\"", raw)+ gsub(/\\\//, "/", raw)+ gsub(/\001/, "\\", raw)+ return raw+}++# Skips a non-string value (number, literal, nested object or array).+function skip_value( depth, c) {+ depth = 0+ while (pos <= len) {+ c = substr(doc, pos, 1)+ if (c == "\"") { read_string(); continue }+ if (c == "{" || c == "[") depth+++ else if (c == "}" || c == "]") { if (depth == 0) return; depth-- }+ else if (c == "," && depth == 0) return+ pos+++ }+}++function read_top_level_fields( key, c) {+ pos = 1+ skip_space()+ if (substr(doc, pos, 1) != "{") fail("invalid input JSON: expected an object")+ pos+++ while (pos <= len) {+ skip_space()+ c = substr(doc, pos, 1)+ if (c == "}") return+ if (c == ",") { pos++; continue }+ if (c != "\"") fail("invalid input JSON: expected a key")+ key = read_string()+ skip_space()+ if (substr(doc, pos, 1) != ":") fail("invalid input JSON: expected a colon")+ pos+++ skip_space()+ if (substr(doc, pos, 1) == "\"") field[key] = read_string()+ else skip_value()+ }+ fail("invalid input JSON: unterminated object")+}++function utf8_length(text, copy, continuation_bytes) {+ copy = text+ continuation_bytes = gsub(/[\200-\277]/, "", copy)+ return length(text) - continuation_bytes+}++function trimmed(text) {+ sub(/^[ \t\r\n]+/, "", text)+ sub(/[ \t\r\n]+$/, "", text)+ return text+}++# Same checks and weights as score_proposal.py. Sets reasons as a side effect.+function score_body(path, body, lines, n, i, line, h1, h2, item, prose, placeholder, score) {+ n = split(body, lines, "\n")+ for (i = 1; i <= n; i++) {+ line = lines[i]+ if (line ~ /^#[ \t]+[^ \t]/) h1 = 1+ if (line ~ /^##[ \t]+[^ \t]/) h2 = 1+ if (line ~ /^[ \t]*([-*+]|[0-9]+[.)])[ \t]+[^ \t]/) item = 1+ if (line ~ /[^ \t]/ && line !~ /^[ \t]*#/) prose = 1+ if (line ~ /(^|[^A-Za-z0-9_])(TODO|TBD)([^A-Za-z0-9_]|$)/) placeholder = 1+ }+ score = 0+ reasons = ""+ if (h1) score += 0.25; else add_reason("missing H1 heading")+ if (!placeholder) score += 0.20; else add_reason("contains TODO/TBD placeholder")+ if (path ~ /^procedures\//) {+ if (h2) score += 0.20; else add_reason("procedure has no ## section")+ if (item) score += 0.25; else add_reason("procedure has no list of steps")+ } else if (path ~ /^_rules\//) {+ if (prose) score += 0.35; else add_reason("rule has no text beyond its heading")+ if (utf8_length(trimmed(body)) <= 2400) score += 0.10+ else add_reason("rule is too long for an agent instruction")+ } else {+ score += 0.45+ }+ return score > 1 ? 1 : score+}++function add_reason(reason) {+ reasons = reasons == "" ? reason : reasons "; " reason+}++{ doc = doc $0 "\n" }++END {+ len = length(doc)+ read_top_level_fields()+ score_before = score_body(field["path"], field["before_body"])+ score_after = score_body(field["path"], field["after_body"])+ passed = score_after >= 0.70 && score_after >= score_before+ printf "{\"passed\":%s,\"score_before\":%.4f,\"score_after\":%.4f", \+ passed ? "true" : "false", score_before, score_after+ if (!passed) printf ",\"reason\":\"%s\"", reasons == "" ? "score did not improve" : reasons+ printf "}\n"+}'diff --git a/docs/examples/auto-improve-eval/test_score_proposal.py b/docs/examples/auto-improve-eval/test_score_proposal.pynew file mode 100644index 00000000..d5f827ec--- /dev/null+++ b/docs/examples/auto-improve-eval/test_score_proposal.py@@ -0,0 +1,102 @@+"""Tests for the example eval scorers: both give the same verdict, in any language.++Run with: python3 -m unittest discover -s docs/examples/auto-improve-eval -p 'test_*.py'+"""++from __future__ import annotations++import json+import subprocess+import sys+import unittest+from pathlib import Path++HERE = Path(__file__).resolve().parent+SCORERS = {+ "python": [sys.executable, "-I", str(HERE / "score_proposal.py")],+ "sh": ["sh", str(HERE / "score_proposal.sh")],+}+PT_PROCEDURE = (+ "# Procedimento de publicação\n\n## Objetivo\n"+ "Publicar uma versão verificada; todo o fluxo é revisado.\n\n"+ "## Passos\n- Rodar todos os testes.\n- Conferir o /admin/status.\n"+)+++def proposal(path: str, after_body: str, before_body: str = "") -> bytes:+ sample = json.loads((HERE / "sample-proposal.json").read_text(encoding="utf-8"))+ sample.update(path=path, after_body=after_body, before_body=before_body)+ return json.dumps(sample, ensure_ascii=False).encode("utf-8")+++def score(scorer: str, stdin: bytes) -> dict[str, object]:+ completed = subprocess.run(+ SCORERS[scorer], input=stdin, capture_output=True, check=True, timeout=30+ )+ return json.loads(completed.stdout)+++class ScorerParityTest(unittest.TestCase):+ def assert_verdict(self, stdin: bytes, passed: bool) -> None:+ results = {name: score(name, stdin) for name in SCORERS}+ for name, result in results.items():+ self.assertIs(result["passed"], passed, f"{name}: {result}")+ python, shell = results["python"], results["sh"]+ for field in ("score_before", "score_after", "reason"):+ self.assertEqual(python.get(field), shell.get(field), field)++ def test_the_sample_proposal_passes(self) -> None:+ self.assert_verdict((HERE / "sample-proposal.json").read_bytes(), True)++ def test_a_portuguese_procedure_passes_and_todo_in_prose_is_no_placeholder(+ self,+ ) -> None:+ self.assert_verdict(proposal("procedures/publicacao.md", PT_PROCEDURE), True)++ def test_a_japanese_procedure_passes(self) -> None:+ body = "# リリース手順\n\n## 手順\n1. テストを実行する\n2) デプロイする\n"+ self.assert_verdict(proposal("procedures/release.md", body), True)++ def test_a_portuguese_rule_passes(self) -> None:+ body = "# Testes\n\nSempre rode a suíte completa antes de publicar.\n"+ self.assert_verdict(proposal("_rules/testes.md", body), True)++ def test_rule_length_counts_characters_not_bytes(self) -> None:+ body = "# Regra\n\n" + "ação " * 470 + "\n"+ self.assertEqual(+ score("sh", proposal("_rules/longa.md", body))["score_after"], 0.9+ )+ self.assert_verdict(proposal("_rules/longa.md", body), True)++ def test_a_procedure_without_a_list_of_steps_fails(self) -> None:+ body = "# Procedimento\n\n## Passos\nRodar os testes e publicar.\n"+ self.assert_verdict(proposal("procedures/x.md", body), False)++ def test_an_upper_case_placeholder_without_h1_fails(self) -> None:+ body = "Sem título\n\n## Passos\n- TODO: completar\n"+ self.assert_verdict(proposal("procedures/x.md", body), False)++ def test_a_rewrite_that_scores_lower_than_the_page_fails(self) -> None:+ body = "# P\n\n## S\nsem lista\n"+ self.assert_verdict(proposal("procedures/x.md", body, PT_PROCEDURE), False)++ def test_escaped_json_and_nested_keys_are_read_as_the_server_sends_them(+ self,+ ) -> None:+ sample = json.loads(proposal("_rules/a.md", '# Regra\n\nUse "x" \\ e\ttab.\n'))+ nested = {"meta": {"path": "procedures/other", "list": [1, {"a": "}"}]}}+ stdin = json.dumps({**nested, **sample}, ensure_ascii=True).encode()+ self.assert_verdict(stdin, True)++ def test_an_escaped_backslash_before_n_is_not_read_as_a_line_break(self) -> None:+ # Decoded as a line break, the "# x" after C:\n would be an H1 and pass.+ body = "Caminho C:\\n# x sem título\n"+ self.assert_verdict(proposal("_rules/a.md", body), False)++ def test_a_non_json_input_fails_closed_in_the_shell_scorer(self) -> None:+ result = score("sh", b'{"path": "procedures/x", "after_body": "# a\n')+ self.assertIs(result["passed"], False)+++if __name__ == "__main__":+ unittest.main()