Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

test(evals): pin the published seed's sha256; note harness vs server differences

AkitaOnRails8f417180a82d
diff --git a/evals/README.md b/evals/README.md
index 8b47de9d..1e2970ac 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -185,7 +185,17 @@ output with no high-signal term, or past the excerpt cap of a pasted
prompt). Where a fact sits is a choice, so this session's
`projection_loss` describes the fixture, not real sessions; what it
supports is a before/after comparison. `report.json` records the seed
-and the sha256 of the generated JSON next to the numbers.
+and the sha256 of the generated JSON next to the numbers; a test pins the
+sha256 of seed 1215, so a generator change that alters it has to update the
+pin and say so beside any republished number.
+
+Two differences from a live server: fixtures and the generated session reach
+the projection without passing through the capture sanitizer (they are
+synthetic, and the injection fixture keeps its planted strings on purpose),
+and they are projected at the harness's 300,000-character default budget,
+not the server's 240,000-character one (100,000 tokens × 3 characters ×
+the 0.8 safety margin). Compare runs against each other, not against a
+server's prompts.
Zero-LLM, no provider:
diff --git a/evals/src/generate.rs b/evals/src/generate.rs
index 3f7635f5..fc228d98 100644
--- a/evals/src/generate.rs
+++ b/evals/src/generate.rs
@@ -420,6 +420,19 @@ mod tests {
assert_ne!(a, c, "the seed must reach the output");
}
+ /// Published runs cite the sha256 of `session(1215)`. A change to the
+ /// generator (a template, a formatting tweak, the RNG) that alters those
+ /// bytes must update this pin and say so next to any republished number,
+ /// rather than silently changing what the old hash names.
+ #[test]
+ fn the_published_seed_keeps_its_sha256() {
+ let (_, sha) = dump(&session(DEFAULT_SEED)).unwrap();
+ assert_eq!(
+ sha,
+ "f25fbc436edf70647ff54b910dfa5ab72259e1b45f1c072db63b9e23d01879d4"
+ );
+ }
+
#[test]
fn every_fact_is_planted_exactly_once_and_nowhere_in_the_prompt() {
let fx = session(DEFAULT_SEED);