Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Auto-Improve Eval Gates
[auto_improve.eval] lets operators guard high-impact auto-improvement
proposals with a small executable scorer. The scorer runs after LLM validation
and before staging or auto-approval. Hooks never run eval commands.
Configuration
[auto_improve.eval]
enabled = true
command = "sh docs/examples/auto-improve-eval/score_proposal.sh"
timeout_secs = 30
targets = ["_rules", "procedures"]
min_delta = 0.0
The command is split on whitespace and executed directly, not through a shell. Use a wrapper script if you need quoting, environment setup, or multiple commands.
The built-in scorer
Writing, mounting and maintaining a scorer is optional: builtin = "structure"
runs the checks in process, so the gate needs no interpreter, no script inside
the container, and no timeout.
[auto_improve.eval]
enabled = true
builtin = "structure"
targets = ["_rules", "procedures"]
min_delta = 0.0
It checks, never matching a section name or a wording (so a wiki in any language scores the same):
- an H1 heading;
- no upper-case
TODO/TBDplaceholder — a hard failure, not a deduction; the Portuguese word "todo" in prose is not a placeholder; - for
procedures/, a##section and a list of steps; - for
_rules/, text beyond the heading, within 2,400 characters.
A proposal passes at score_after >= 0.70 and no lower than score_before.
eval_results records checker_name = "builtin:structure" and an invocation
digest over the scorer name and the binary version, instead of a command
string.
builtin and command are mutually exclusive: setting both is a startup
error rather than one being silently ignored. command keeps working
unchanged for project-specific scorers.
The eval policy is owned by the server process. ai-memory auto-improve and
direct HTTP requests cannot replace the configured command, targets, timeout,
or score threshold. When a CLI talks to a remote server, configure
[auto_improve.eval] in that server's config; older clients that send an
eval request field receive 422 Unprocessable Entity and must be upgraded.
Request contract
The scorer receives one JSON object on stdin:
{
"path": "procedures/release.md",
"kind": "procedure",
"operation": "update",
"edit_mode": "patch",
"title": "Release Procedure",
"confidence": 0.91,
"rationale": "Capture the repeated release checklist.",
"before_body": "# Release Procedure\n\n## Steps\n- Run tests\n",
"after_body": "# Release Procedure\n\n## Steps\n- Run tests\n- Run deploy smoke checks\n",
"expected_base_body_sha256": "..."
}
before_body is empty for create proposals. expected_base_body_sha256 is
present only for patch proposals that were materialized against a known base.
Response contract
The scorer must print one JSON object to stdout:
{ "score_before": 0.72, "score_after": 0.76, "passed": true }
Fields:
passedis required.falserejects the targeted proposal.score_beforeandscore_afterare optional. When both are present,score_after - score_beforemust be at leastmin_delta.reasonis optional and should explain a rejection in one short sentence.
Command errors, timeouts, invalid JSON, missing passed, passed = false, and
insufficient score delta all fail closed for the targeted proposal. Other
proposals in the same run can still proceed.
Recorded results
The review report includes optional eval_results, one record for each targeted
proposal evaluated by the existing gate. The server generates each eval_id as a
UUID v7 for that scorer attempt, independently of the review run ID. Its status is
success when the response passes the existing criteria, rejected when a valid
response has passed = false or insufficient score delta, failure for command,
I/O, or response parsing errors, and timeout when the deadline expires.
Non-targeted proposals and disabled evals produce no records.
Each record contains the target path, checker_name (the program's basename),
checker_invocation_digest (SHA-256 of the configured command string), and
proposal_sha256 (SHA-256 of the serialized JSON request sent on stdin).
before_body_sha256 and after_body_sha256 hash the exact UTF-8 bodies in that
request. Optional materialized_base_body_sha256 identifies the server-resolved
base used to materialize a patch, which can differ from the current before_body
if the page changed before eval. These records do not certify a later page version.
The invocation digest does not hash executable or script contents and cannot
prove which checker code ran. No executable is read to produce these records.
Scores and passed come only from the scorer response. Reasons are scrubbed with
the built-in sanitizer before being capped at 2,048 characters. They remain
untrusted, attributed data. Extra scorer/model fields cannot supply eval IDs,
digests, or independent results; ordinary evidence quotes remain ordinary quotes.
Staging stores all results in the existing run config JSON. For an accepted
proposal, its eval result is also attached to the auto_improve_eval evidence
entry in the pending sidecar. The sidecar remains a snapshot at staging time,
without a new approval or execution step. Reports without eval_results and
older evidence entries without eval_result remain readable.
Deserializing a review report always discards eval_results, including forged
or malformed values. Only results held by the live server review can be attached
to pending evidence; reading a report cannot restore independent observations.
Scorer design rules
- Keep scorers deterministic, fast, and side-effect-free.
- Read only stdin and local project files that are safe to inspect.
- Do not call LLMs, mutate files, run deploys, or depend on network services.
- Return bounded reasons; ai-memory caps captured eval evidence.
- Prefer simple checks that match the target path: heading structure for
procedures, forbidden placeholders for
_rules, or project-specific smoke assertions for critical docs. - Check structure, not wording. Pages are written in whatever language the team works in, so a check for an English section name or keyword rejects every proposal written in another language.
Examples
This repository includes two templates that apply the same checks: an H1,
no TODO/TBD placeholder, a ## section and a list of steps for procedures, and
text beyond the heading within 2,400 characters for rules. None of them looks
at section names or wording, so pages in any language score the same way.
docs/examples/auto-improve-eval/score_proposal.py: Python scorer. It needspython3on the host that runs the server.docs/examples/auto-improve-eval/score_proposal.sh: the same scorer in POSIXshandawk, with no other dependency.
The official Docker image ships no python3, so use the shell scorer there.
The command runs inside the container, from its working directory /data, and
the image does not include docs/: copy the script into the data volume and
point the command at that copy, for example
command = "sh /data/eval/score_proposal.sh".
Try them with the sample payload:
python3 docs/examples/auto-improve-eval/score_proposal.py \
< docs/examples/auto-improve-eval/sample-proposal.json
sh docs/examples/auto-improve-eval/score_proposal.sh \
< docs/examples/auto-improve-eval/sample-proposal.json
Both print compact JSON suitable for ai-memory's eval gate.