Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs: make the example eval scorers language-neutral and Python-free in sh

Renato Regisbba7d54eeee4
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 10e3f4b7..f6ec6fc5 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -138,6 +138,13 @@ jobs:
else
echo "no tests under $dir"
fi
+ # The two example eval scorers must keep giving the same verdict; the
+ # suite runs the Python one and the sh + awk one on the same proposals.
+ - name: Auto-improve eval example scorer tests
+ if: matrix.os == 'ubuntu-latest'
+ env:
+ PYTHONDONTWRITEBYTECODE: "1"
+ run: uv run --no-project python -m unittest discover -s docs/examples/auto-improve-eval -p 'test_*.py' -v
# The capture policy every generated OpenCode/OMP/Pi/OpenClaw plugin
# embeds is TypeScript; only Node can execute it against the shared
# capture-policy fixture. The test is ignored locally, where Node is not
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c6a02aca..2b6a96f3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -34,6 +34,18 @@ version in [`docs/changelog/`](docs/changelog/):
rejected as containing no extractable text when that was its only cue.
WebVTT allows any number of blank lines between blocks; the reader now skips
them before reading a block.
+- The example auto-improve eval scorers in `docs/examples/auto-improve-eval/`
+ no longer reject pages written in a language other than English. They
+ required `## Purpose`/`## Overview` and `## Steps`/`## Checklist` headings in
+ procedures and English imperatives in rules, and read the Portuguese and
+ Spanish word "todo" as a TODO placeholder, so a correct procedure in
+ Portuguese or Japanese failed. They now check structure only: an H1, an
+ upper-case TODO/TBD, a `##` section and a list of steps, text beyond the
+ heading and its length counted in characters. `score_proposal.sh` no longer
+ embeds Python: it is the same scorer in POSIX `sh` and `awk`, so it runs in
+ the official Docker image, which has no `python3`. The docs no longer call
+ the Python scorer dependency-free and say how to point the eval command at a
+ script inside the container (#PRNUM).
## [3.2.2] - 2026-10-22
diff --git a/docs/auto-improve-eval-gates.md b/docs/auto-improve-eval-gates.md
index 5609cc56..e2da5c85 100644
--- a/docs/auto-improve-eval-gates.md
+++ b/docs/auto-improve-eval-gates.md
@@ -9,7 +9,7 @@ and before staging or auto-approval. Hooks never run eval commands.
```toml
[auto_improve.eval]
enabled = true
-command = "python3 docs/examples/auto-improve-eval/score_proposal.py"
+command = "sh docs/examples/auto-improve-eval/score_proposal.sh"
timeout_secs = 30
targets = ["_rules", "procedures"]
min_delta = 0.0
@@ -109,16 +109,27 @@ to pending evidence; reading a report cannot restore independent observations.
- Prefer simple checks that match the target path: heading structure for
procedures, forbidden placeholders for `_rules`, or project-specific smoke
assertions for critical docs.
+- Check structure, not wording. Pages are written in whatever language the
+ team works in, so a check for an English section name or keyword rejects
+ every proposal written in another language.
## Examples
-This repository includes two dependency-free templates:
+This repository includes two templates that apply the same checks: an H1,
+no TODO/TBD placeholder, a `##` section and a list of steps for procedures, and
+text beyond the heading within 2,400 characters for rules. None of them looks
+at section names or wording, so pages in any language score the same way.
- [`docs/examples/auto-improve-eval/score_proposal.py`](examples/auto-improve-eval/score_proposal.py):
- Python scorer that checks basic structure and placeholders.
+ Python scorer. It needs `python3` on the host that runs the server.
- [`docs/examples/auto-improve-eval/score_proposal.sh`](examples/auto-improve-eval/score_proposal.sh):
- POSIX shell wrapper around an embedded Python scorer for hosts that prefer a
- script entrypoint.
+ the same scorer in POSIX `sh` and `awk`, with no other dependency.
+
+The official Docker image ships no `python3`, so use the shell scorer there.
+The command runs inside the container, from its working directory `/data`, and
+the image does not include `docs/`: copy the script into the data volume and
+point the command at that copy, for example
+`command = "sh /data/eval/score_proposal.sh"`.
Try them with the sample payload:
diff --git a/docs/examples/auto-improve-eval/README.md b/docs/examples/auto-improve-eval/README.md
index 95d5afa6..11261bfb 100644
--- a/docs/examples/auto-improve-eval/README.md
+++ b/docs/examples/auto-improve-eval/README.md
@@ -1,8 +1,11 @@
# Auto-Improve Eval Example Scorers
-These examples are intentionally small, deterministic, and dependency-free.
-They read one proposal JSON object from stdin and print one eval response JSON
-object to stdout.
+These examples are intentionally small and deterministic. They read one
+proposal JSON object from stdin and print one eval response JSON object to
+stdout. `score_proposal.py` needs `python3`; `score_proposal.sh` applies the same
+checks with only POSIX `sh` and `awk`, so it also runs inside the official Docker
+image, which has no Python. Neither matches section names or wording, so pages
+written in any language score the same way.
Use them as templates, not as universal quality gates. Real projects should
replace the checks with project-specific invariants that are safe to run before
@@ -11,7 +14,7 @@ an auto-improvement proposal is staged.
```toml
[auto_improve.eval]
enabled = true
-command = "python3 docs/examples/auto-improve-eval/score_proposal.py"
+command = "sh docs/examples/auto-improve-eval/score_proposal.sh"
timeout_secs = 30
targets = ["_rules", "procedures"]
min_delta = 0.0
@@ -25,3 +28,10 @@ python3 docs/examples/auto-improve-eval/score_proposal.py < \
sh docs/examples/auto-improve-eval/score_proposal.sh < \
docs/examples/auto-improve-eval/sample-proposal.json
```
+
+Both scorers must give the same verdict; `test_score_proposal.py` runs them on
+the same proposals, including pages in Portuguese and Japanese:
+
+```bash
+python3 -m unittest discover -s docs/examples/auto-improve-eval -p 'test_*.py'
+```
diff --git a/docs/examples/auto-improve-eval/score_proposal.py b/docs/examples/auto-improve-eval/score_proposal.py
index d573a3de..46ef2bf3 100644
--- a/docs/examples/auto-improve-eval/score_proposal.py
+++ b/docs/examples/auto-improve-eval/score_proposal.py
@@ -4,10 +4,16 @@
Reads proposal JSON from stdin and emits:
{"passed": bool, "score_before": float, "score_after": float, "reason": str?}
-The checks are intentionally simple and deterministic:
-- procedures should have a Purpose/Steps shape;
-- rules should avoid placeholders and use imperative language;
-- all targeted docs should avoid TODO placeholders.
+The checks are intentionally simple, deterministic, and independent of the
+language the wiki is written in (section names and wording are never matched):
+- procedures should have at least one `##` section and at least one list item
+ for their steps;
+- rules should carry text beyond their heading and stay short enough for an
+ agent instruction;
+- all targeted docs should have an H1 and avoid TODO/TBD placeholders (matched
+ as whole upper-case words, so a word like "todo" in prose is not one).
+
+`score_proposal.sh` applies the same checks with `sh` and `awk` only.
"""
from __future__ import annotations
@@ -16,37 +22,46 @@ import json
import re
import sys
+LIST_ITEM = re.compile(r"^[ \t]*(?:[-*+]|[0-9]+[.)])[ \t]+\S", re.MULTILINE)
+PLACEHOLDER = re.compile(r"(?<![A-Za-z0-9_])(?:TODO|TBD)(?![A-Za-z0-9_])")
+
+
+def has_text_beyond_headings(body: str) -> bool:
+ return any(
+ line.strip(" \t") and not line.lstrip(" \t").startswith("#")
+ for line in body.split("\n")
+ )
+
def score_body(path: str, body: str) -> tuple[float, list[str]]:
score = 0.0
reasons: list[str] = []
- lowered = body.lower()
- if re.search(r"^#\s+\S", body, re.MULTILINE):
+ if re.search(r"^#[ \t]+\S", body, re.MULTILINE):
score += 0.25
else:
reasons.append("missing H1 heading")
- if "todo" not in lowered and "tbd" not in lowered:
+ if not PLACEHOLDER.search(body):
score += 0.20
else:
reasons.append("contains TODO/TBD placeholder")
if path.startswith("procedures/"):
- if "## purpose" in lowered or "## overview" in lowered:
+ if re.search(r"^##[ \t]+\S", body, re.MULTILINE):
score += 0.20
else:
- reasons.append("procedure lacks Purpose/Overview section")
- if "## steps" in lowered or "## checklist" in lowered:
+ reasons.append("procedure has no ## section")
+ if LIST_ITEM.search(body):
score += 0.25
else:
- reasons.append("procedure lacks Steps/Checklist section")
+ reasons.append("procedure has no list of steps")
elif path.startswith("_rules/"):
- if re.search(r"\b(always|never|must|do not|prefer)\b", lowered):
+ if has_text_beyond_headings(body):
score += 0.35
else:
- reasons.append("rule lacks imperative guidance")
- if len(body.strip()) <= 2400:
+ reasons.append("rule has no text beyond its heading")
+ if len(body.strip(" \t\r\n")) <= 2400:
score += 0.10
else:
reasons.append("rule is too long for an agent instruction")
diff --git a/docs/examples/auto-improve-eval/score_proposal.sh b/docs/examples/auto-improve-eval/score_proposal.sh
index 251c7b48..aab34dfd 100644
--- a/docs/examples/auto-improve-eval/score_proposal.sh
+++ b/docs/examples/auto-improve-eval/score_proposal.sh
@@ -1,36 +1,136 @@
#!/bin/sh
-# POSIX-shell entrypoint for ai-memory auto-improve eval gates.
-# It embeds a tiny Python scorer so the configured command can be a single
-# script path while keeping JSON parsing correct.
-
-python3 -c '
-import json, re, sys
-
-try:
- payload = json.load(sys.stdin)
-except json.JSONDecodeError as exc:
- print(json.dumps({"passed": False, "reason": f"invalid input JSON: {exc}"}))
- raise SystemExit(0)
-
-path = str(payload.get("path", ""))
-body = str(payload.get("after_body", ""))
-lowered = body.lower()
-checks = [
- bool(re.search(r"^#\s+\S", body, re.MULTILINE)),
- "todo" not in lowered and "tbd" not in lowered,
-]
-if path.startswith("procedures/"):
- checks.extend([
- "## steps" in lowered or "## checklist" in lowered,
- "## purpose" in lowered or "## overview" in lowered,
- ])
-elif path.startswith("_rules/"):
- checks.append(bool(re.search(r"\b(always|never|must|do not|prefer)\b", lowered)))
-
-score_after = sum(1 for ok in checks if ok) / len(checks)
-passed = score_after >= 0.75
-response = {"passed": passed, "score_after": round(score_after, 4)}
-if not passed:
- response["reason"] = "proposal did not satisfy the example structure checks"
-print(json.dumps(response, separators=(",", ":")))
+# POSIX sh + awk port of score_proposal.py for ai-memory auto-improve eval gates.
+# It needs no Python, so it also runs inside the official Docker image, which
+# ships only a POSIX shell and awk. The checks never match section names or
+# wording, so a wiki written in any language scores the same way.
+
+LC_ALL=C exec awk '
+function fail(reason) {
+ printf "{\"passed\":false,\"reason\":\"%s\"}\n", reason
+ exit 0
+}
+
+function skip_space() {
+ while (pos <= len && index(" \t\r\n", substr(doc, pos, 1)) > 0) pos++
+}
+
+# Decodes the JSON string starting at doc[pos] (the opening quote) and leaves
+# pos after the closing quote. It finds the closing quote first and decodes the
+# whole run with gsub: appending one character at a time copies the string on
+# every step, which took 22 s on a 1 MB page. \uXXXX becomes "?": the checks
+# only look at ASCII markup, and the server writes non-ASCII text as raw UTF-8.
+function read_string( start, c, raw) {
+ start = ++pos
+ while (pos <= len) {
+ c = substr(doc, pos, 1)
+ if (c == "\"") break
+ pos += c == "\\" ? 2 : 1
+ }
+ if (pos > len) fail("invalid input JSON: unterminated string")
+ raw = substr(doc, start, pos - start)
+ pos++
+ gsub(/\\\\/, "\001", raw)
+ gsub(/\\n/, "\n", raw)
+ gsub(/\\t/, "\t", raw)
+ gsub(/\\r/, "\r", raw)
+ gsub(/\\[bf]/, " ", raw)
+ gsub(/\\u[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]/, "?", raw)
+ gsub(/\\"/, "\"", raw)
+ gsub(/\\\//, "/", raw)
+ gsub(/\001/, "\\", raw)
+ return raw
+}
+
+# Skips a non-string value (number, literal, nested object or array).
+function skip_value( depth, c) {
+ depth = 0
+ while (pos <= len) {
+ c = substr(doc, pos, 1)
+ if (c == "\"") { read_string(); continue }
+ if (c == "{" || c == "[") depth++
+ else if (c == "}" || c == "]") { if (depth == 0) return; depth-- }
+ else if (c == "," && depth == 0) return
+ pos++
+ }
+}
+
+function read_top_level_fields( key, c) {
+ pos = 1
+ skip_space()
+ if (substr(doc, pos, 1) != "{") fail("invalid input JSON: expected an object")
+ pos++
+ while (pos <= len) {
+ skip_space()
+ c = substr(doc, pos, 1)
+ if (c == "}") return
+ if (c == ",") { pos++; continue }
+ if (c != "\"") fail("invalid input JSON: expected a key")
+ key = read_string()
+ skip_space()
+ if (substr(doc, pos, 1) != ":") fail("invalid input JSON: expected a colon")
+ pos++
+ skip_space()
+ if (substr(doc, pos, 1) == "\"") field[key] = read_string()
+ else skip_value()
+ }
+ fail("invalid input JSON: unterminated object")
+}
+
+function utf8_length(text, copy, continuation_bytes) {
+ copy = text
+ continuation_bytes = gsub(/[\200-\277]/, "", copy)
+ return length(text) - continuation_bytes
+}
+
+function trimmed(text) {
+ sub(/^[ \t\r\n]+/, "", text)
+ sub(/[ \t\r\n]+$/, "", text)
+ return text
+}
+
+# Same checks and weights as score_proposal.py. Sets reasons as a side effect.
+function score_body(path, body, lines, n, i, line, h1, h2, item, prose, placeholder, score) {
+ n = split(body, lines, "\n")
+ for (i = 1; i <= n; i++) {
+ line = lines[i]
+ if (line ~ /^#[ \t]+[^ \t]/) h1 = 1
+ if (line ~ /^##[ \t]+[^ \t]/) h2 = 1
+ if (line ~ /^[ \t]*([-*+]|[0-9]+[.)])[ \t]+[^ \t]/) item = 1
+ if (line ~ /[^ \t]/ && line !~ /^[ \t]*#/) prose = 1
+ if (line ~ /(^|[^A-Za-z0-9_])(TODO|TBD)([^A-Za-z0-9_]|$)/) placeholder = 1
+ }
+ score = 0
+ reasons = ""
+ if (h1) score += 0.25; else add_reason("missing H1 heading")
+ if (!placeholder) score += 0.20; else add_reason("contains TODO/TBD placeholder")
+ if (path ~ /^procedures\//) {
+ if (h2) score += 0.20; else add_reason("procedure has no ## section")
+ if (item) score += 0.25; else add_reason("procedure has no list of steps")
+ } else if (path ~ /^_rules\//) {
+ if (prose) score += 0.35; else add_reason("rule has no text beyond its heading")
+ if (utf8_length(trimmed(body)) <= 2400) score += 0.10
+ else add_reason("rule is too long for an agent instruction")
+ } else {
+ score += 0.45
+ }
+ return score > 1 ? 1 : score
+}
+
+function add_reason(reason) {
+ reasons = reasons == "" ? reason : reasons "; " reason
+}
+
+{ doc = doc $0 "\n" }
+
+END {
+ len = length(doc)
+ read_top_level_fields()
+ score_before = score_body(field["path"], field["before_body"])
+ score_after = score_body(field["path"], field["after_body"])
+ passed = score_after >= 0.70 && score_after >= score_before
+ printf "{\"passed\":%s,\"score_before\":%.4f,\"score_after\":%.4f", \
+ passed ? "true" : "false", score_before, score_after
+ if (!passed) printf ",\"reason\":\"%s\"", reasons == "" ? "score did not improve" : reasons
+ printf "}\n"
+}
'
diff --git a/docs/examples/auto-improve-eval/test_score_proposal.py b/docs/examples/auto-improve-eval/test_score_proposal.py
new file mode 100644
index 00000000..d5f827ec
--- /dev/null
+++ b/docs/examples/auto-improve-eval/test_score_proposal.py
@@ -0,0 +1,102 @@
+"""Tests for the example eval scorers: both give the same verdict, in any language.
+
+Run with: python3 -m unittest discover -s docs/examples/auto-improve-eval -p 'test_*.py'
+"""
+
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+SCORERS = {
+ "python": [sys.executable, "-I", str(HERE / "score_proposal.py")],
+ "sh": ["sh", str(HERE / "score_proposal.sh")],
+}
+PT_PROCEDURE = (
+ "# Procedimento de publicação\n\n## Objetivo\n"
+ "Publicar uma versão verificada; todo o fluxo é revisado.\n\n"
+ "## Passos\n- Rodar todos os testes.\n- Conferir o /admin/status.\n"
+)
+
+
+def proposal(path: str, after_body: str, before_body: str = "") -> bytes:
+ sample = json.loads((HERE / "sample-proposal.json").read_text(encoding="utf-8"))
+ sample.update(path=path, after_body=after_body, before_body=before_body)
+ return json.dumps(sample, ensure_ascii=False).encode("utf-8")
+
+
+def score(scorer: str, stdin: bytes) -> dict[str, object]:
+ completed = subprocess.run(
+ SCORERS[scorer], input=stdin, capture_output=True, check=True, timeout=30
+ )
+ return json.loads(completed.stdout)
+
+
+class ScorerParityTest(unittest.TestCase):
+ def assert_verdict(self, stdin: bytes, passed: bool) -> None:
+ results = {name: score(name, stdin) for name in SCORERS}
+ for name, result in results.items():
+ self.assertIs(result["passed"], passed, f"{name}: {result}")
+ python, shell = results["python"], results["sh"]
+ for field in ("score_before", "score_after", "reason"):
+ self.assertEqual(python.get(field), shell.get(field), field)
+
+ def test_the_sample_proposal_passes(self) -> None:
+ self.assert_verdict((HERE / "sample-proposal.json").read_bytes(), True)
+
+ def test_a_portuguese_procedure_passes_and_todo_in_prose_is_no_placeholder(
+ self,
+ ) -> None:
+ self.assert_verdict(proposal("procedures/publicacao.md", PT_PROCEDURE), True)
+
+ def test_a_japanese_procedure_passes(self) -> None:
+ body = "# リリース手順\n\n## 手順\n1. テストを実行する\n2) デプロイする\n"
+ self.assert_verdict(proposal("procedures/release.md", body), True)
+
+ def test_a_portuguese_rule_passes(self) -> None:
+ body = "# Testes\n\nSempre rode a suíte completa antes de publicar.\n"
+ self.assert_verdict(proposal("_rules/testes.md", body), True)
+
+ def test_rule_length_counts_characters_not_bytes(self) -> None:
+ body = "# Regra\n\n" + "ação " * 470 + "\n"
+ self.assertEqual(
+ score("sh", proposal("_rules/longa.md", body))["score_after"], 0.9
+ )
+ self.assert_verdict(proposal("_rules/longa.md", body), True)
+
+ def test_a_procedure_without_a_list_of_steps_fails(self) -> None:
+ body = "# Procedimento\n\n## Passos\nRodar os testes e publicar.\n"
+ self.assert_verdict(proposal("procedures/x.md", body), False)
+
+ def test_an_upper_case_placeholder_without_h1_fails(self) -> None:
+ body = "Sem título\n\n## Passos\n- TODO: completar\n"
+ self.assert_verdict(proposal("procedures/x.md", body), False)
+
+ def test_a_rewrite_that_scores_lower_than_the_page_fails(self) -> None:
+ body = "# P\n\n## S\nsem lista\n"
+ self.assert_verdict(proposal("procedures/x.md", body, PT_PROCEDURE), False)
+
+ def test_escaped_json_and_nested_keys_are_read_as_the_server_sends_them(
+ self,
+ ) -> None:
+ sample = json.loads(proposal("_rules/a.md", '# Regra\n\nUse "x" \\ e\ttab.\n'))
+ nested = {"meta": {"path": "procedures/other", "list": [1, {"a": "}"}]}}
+ stdin = json.dumps({**nested, **sample}, ensure_ascii=True).encode()
+ self.assert_verdict(stdin, True)
+
+ def test_an_escaped_backslash_before_n_is_not_read_as_a_line_break(self) -> None:
+ # Decoded as a line break, the "# x" after C:\n would be an H1 and pass.
+ body = "Caminho C:\\n# x sem título\n"
+ self.assert_verdict(proposal("_rules/a.md", body), False)
+
+ def test_a_non_json_input_fails_closed_in_the_shell_scorer(self) -> None:
+ result = score("sh", b'{"path": "procedures/x", "after_body": "# a\n')
+ self.assertIs(result["passed"], False)
+
+
+if __name__ == "__main__":
+ unittest.main()