Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust
7.4 KB
Raw Blame

Auto-Improve Eval Gates

[auto_improve.eval] lets operators guard high-impact auto-improvement proposals with a small executable scorer. The scorer runs after LLM validation and before staging or auto-approval. Hooks never run eval commands.

Configuration

[auto_improve.eval]
enabled = true
command = "sh docs/examples/auto-improve-eval/score_proposal.sh"
timeout_secs = 30
targets = ["_rules", "procedures"]
min_delta = 0.0

The command is split on whitespace and executed directly, not through a shell. Use a wrapper script if you need quoting, environment setup, or multiple commands.

The built-in scorer

Writing, mounting and maintaining a scorer is optional: builtin = "structure" runs the checks in process, so the gate needs no interpreter, no script inside the container, and no timeout.

[auto_improve.eval]
enabled = true
builtin = "structure"
targets = ["_rules", "procedures"]
min_delta = 0.0

It checks, never matching a section name or a wording (so a wiki in any language scores the same):

  • an H1 heading;
  • no upper-case TODO/TBD placeholder — a hard failure, not a deduction; the Portuguese word "todo" in prose is not a placeholder;
  • for procedures/, a ## section and a list of steps;
  • for _rules/, text beyond the heading, within 2,400 characters.

A proposal passes at score_after >= 0.70 and no lower than score_before. eval_results records checker_name = "builtin:structure" and an invocation digest over the scorer name and the binary version, instead of a command string.

builtin and command are mutually exclusive: setting both is a startup error rather than one being silently ignored. command keeps working unchanged for project-specific scorers.

The eval policy is owned by the server process. ai-memory auto-improve and direct HTTP requests cannot replace the configured command, targets, timeout, or score threshold. When a CLI talks to a remote server, configure [auto_improve.eval] in that server's config; older clients that send an eval request field receive 422 Unprocessable Entity and must be upgraded.

Request contract

The scorer receives one JSON object on stdin:

{
  "path": "procedures/release.md",
  "kind": "procedure",
  "operation": "update",
  "edit_mode": "patch",
  "title": "Release Procedure",
  "confidence": 0.91,
  "rationale": "Capture the repeated release checklist.",
  "before_body": "# Release Procedure\n\n## Steps\n- Run tests\n",
  "after_body": "# Release Procedure\n\n## Steps\n- Run tests\n- Run deploy smoke checks\n",
  "expected_base_body_sha256": "..."
}

before_body is empty for create proposals. expected_base_body_sha256 is present only for patch proposals that were materialized against a known base.

Response contract

The scorer must print one JSON object to stdout:

{ "score_before": 0.72, "score_after": 0.76, "passed": true }

Fields:

  • passed is required. false rejects the targeted proposal.
  • score_before and score_after are optional. When both are present, score_after - score_before must be at least min_delta.
  • reason is optional and should explain a rejection in one short sentence.

Command errors, timeouts, invalid JSON, missing passed, passed = false, and insufficient score delta all fail closed for the targeted proposal. Other proposals in the same run can still proceed.

Recorded results

The review report includes optional eval_results, one record for each targeted proposal evaluated by the existing gate. The server generates each eval_id as a UUID v7 for that scorer attempt, independently of the review run ID. Its status is success when the response passes the existing criteria, rejected when a valid response has passed = false or insufficient score delta, failure for command, I/O, or response parsing errors, and timeout when the deadline expires. Non-targeted proposals and disabled evals produce no records.

Each record contains the target path, checker_name (the program's basename), checker_invocation_digest (SHA-256 of the configured command string), and proposal_sha256 (SHA-256 of the serialized JSON request sent on stdin). before_body_sha256 and after_body_sha256 hash the exact UTF-8 bodies in that request. Optional materialized_base_body_sha256 identifies the server-resolved base used to materialize a patch, which can differ from the current before_body if the page changed before eval. These records do not certify a later page version. The invocation digest does not hash executable or script contents and cannot prove which checker code ran. No executable is read to produce these records.

Scores and passed come only from the scorer response. Reasons are scrubbed with the built-in sanitizer before being capped at 2,048 characters. They remain untrusted, attributed data. Extra scorer/model fields cannot supply eval IDs, digests, or independent results; ordinary evidence quotes remain ordinary quotes.

Staging stores all results in the existing run config JSON. For an accepted proposal, its eval result is also attached to the auto_improve_eval evidence entry in the pending sidecar. The sidecar remains a snapshot at staging time, without a new approval or execution step. Reports without eval_results and older evidence entries without eval_result remain readable. Deserializing a review report always discards eval_results, including forged or malformed values. Only results held by the live server review can be attached to pending evidence; reading a report cannot restore independent observations.

Scorer design rules

  • Keep scorers deterministic, fast, and side-effect-free.
  • Read only stdin and local project files that are safe to inspect.
  • Do not call LLMs, mutate files, run deploys, or depend on network services.
  • Return bounded reasons; ai-memory caps captured eval evidence.
  • Prefer simple checks that match the target path: heading structure for procedures, forbidden placeholders for _rules, or project-specific smoke assertions for critical docs.
  • Check structure, not wording. Pages are written in whatever language the team works in, so a check for an English section name or keyword rejects every proposal written in another language.

Examples

This repository includes two templates that apply the same checks: an H1, no TODO/TBD placeholder, a ## section and a list of steps for procedures, and text beyond the heading within 2,400 characters for rules. None of them looks at section names or wording, so pages in any language score the same way.

The official Docker image ships no python3, so use the shell scorer there. The command runs inside the container, from its working directory /data, and the image does not include docs/: copy the script into the data volume and point the command at that copy, for example command = "sh /data/eval/score_proposal.sh".

Try them with the sample payload:

python3 docs/examples/auto-improve-eval/score_proposal.py \
  < docs/examples/auto-improve-eval/sample-proposal.json

sh docs/examples/auto-improve-eval/score_proposal.sh \
  < docs/examples/auto-improve-eval/sample-proposal.json

Both print compact JSON suitable for ai-memory's eval gate.