Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(consolidate): correct the small-context budget examples and floor

yubr-zzh6ac6bb9bb201
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6bf29b1a..c71fd2d7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
+- Fixed consolidation sending the model a prompt with no evidence when
+ `consolidation.max_input_tokens` sat at the validated floor. The floor is
+ meant to guarantee room for observations after the fixed system prompt, page
+ conventions and structured-output schema, but it was checked against the raw
+ token count while the builders run on a shrunken character budget
+ (`max_input_tokens` times a chars-per-token estimate times
+ `input_token_safety_margin`, default 0.8). At the old 6,500 floor the batch
+ prompt had a 15,600-character budget against a 18,997-character fixed part,
+ so every observation became an omission marker and the config was accepted
+ anyway. The floor is now 8,350 tokens, derived from the real builders at the
+ default margin, and the startup check compares the configured budget at its
+ own margin with that requirement at the default, so lowering the margin
+ demands a proportionally larger `max_input_tokens` instead of silently
+ starving the prompt. An install configured between 6,500 and 8,350 tokens now
+ fails at startup with a message naming both numbers, where it previously
+ started and produced evidence-free consolidations. (#1234)
- Replaced the unmaintained YAML library (`serde_yaml` and its
`unsafe-libyaml` parser, RUSTSEC-2024-0449) with `serde-saphyr`. The server
reads existing page frontmatter to exactly the same values as before. One
diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rs
index 27e3632f..ab87e3de 100644
--- a/crates/ai-memory-cli/src/config.rs
+++ b/crates/ai-memory-cli/src/config.rs
@@ -3973,6 +3973,33 @@ max_input_tokens = {tokens}
);
}
+ /// The small-context examples in `docs/install.md` and
+ /// `docs/llm-providers.md` must be configs that actually load, so a doc
+ /// example cannot go stale the way `max_input_tokens = 6500` did once the
+ /// floor moved (#1234). These are the literal values those docs quote.
+ #[test]
+ fn documented_small_context_examples_load() {
+ for (doc, tokens, output) in [
+ ("docs/install.md", 10_000, 1_000),
+ ("docs/llm-providers.md", 10_000, 1_000),
+ ] {
+ let tmp = TempDir::new().unwrap();
+ let config_path = tmp.path().join("config.toml");
+ std::fs::write(
+ &config_path,
+ format!(
+ "[consolidation]
+max_input_tokens = {tokens}
+max_output_tokens = {output}
+"
+ ),
+ )
+ .unwrap();
+ Config::load(Some(&config_path), Some(tmp.path().to_path_buf()))
+ .unwrap_or_else(|error| panic!("{doc}'s example no longer loads: {error:#}"));
+ }
+ }
+
/// A small-context provider needs both sides of the context allocation to
/// survive the config round-trip. The budget is derived from the floor
/// rather than written out, so the test keeps meaning "the smallest
diff --git a/docs/install.md b/docs/install.md
index 16670884..dd774fa5 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -2309,7 +2309,7 @@ the real context window, with additional headroom for tokenizer variance:
```bash
# e.g. a model loaded with an 8k context window
--e AI_MEMORY_CONSOLIDATION__MAX_INPUT_TOKENS=6500
+-e AI_MEMORY_CONSOLIDATION__MAX_INPUT_TOKENS=10000
-e AI_MEMORY_CONSOLIDATION__MAX_OUTPUT_TOKENS=1000
```
@@ -2320,9 +2320,12 @@ schema, and provider-envelope reserve. Tokenizers differ, so this is a
conservative estimate rather than an exact provider token count. An automatic
checkpoint provider failure degrades to a rule-based page rather than losing
the checkpoint, but right-sized limits are what allow LLM consolidation to
-succeed. Startup rejects input targets below 6,000 and output limits below
-1,000 because the batch schema and a useful response cannot fit reliably below
-those floors.
+succeed. Startup rejects a budget that leaves too little room for observations
+after the fixed prompts and the structured-output schema -- 8,350 input tokens
+at the default `input_token_safety_margin` of 0.8 -- and rejects output limits
+below 1,000, because the batch schema and a useful response cannot fit reliably
+below those floors. Lowering the safety margin shrinks the character budget, so
+a smaller margin needs a proportionally larger `max_input_tokens`.
---
diff --git a/docs/llm-providers.md b/docs/llm-providers.md
index 2db90351..9f4b83d6 100644
--- a/docs/llm-providers.md
+++ b/docs/llm-providers.md
@@ -357,7 +357,7 @@ headroom because provider tokenizers differ:
```toml
[consolidation]
-max_input_tokens = 6500
+max_input_tokens = 10000
max_output_tokens = 1000
```
@@ -366,7 +366,9 @@ The equivalent environment variables are
`AI_MEMORY_CONSOLIDATION__MAX_OUTPUT_TOKENS`. Provider failures during an
automatic PreCompact/PostCompaction checkpoint fall back to the deterministic
rule-based page; admission, storage, and scope errors still fail closed. The
-validated minimums are 6,500 input and 1,000 output tokens.
+validated minimums are 8,350 input and 1,000 output tokens at the default
+`input_token_safety_margin` of 0.8; a lower margin shrinks the character budget
+and needs a proportionally larger input target.
Every chat provider bounds each completion request at 300 seconds
(`AI_MEMORY_LLM_TIMEOUT_SECS` to override, or `llm_timeout_secs = 900` in