Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(llm): document compat strict mode

AkitaOnRails64fd83a1eb09
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 75a44ec5..faa21be2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
+- `openai-compat` LLM providers can now opt into strict JSON Schema structured
+ output with `AI_MEMORY_LLM_COMPAT_STRICT=true`. Strict mode sends
+ `response_format=json_schema` first for compatible Ollama, vLLM, LM Studio,
+ llama.cpp, and gateway endpoints, while the tolerant JSON-object parser
+ remains the default and the fallback for strict raw-call failures ([#70]).
- The read-only web browser now renders `[[wiki links]]` as clickable internal
links to the target page. Supports `[[path]]`, `[[path|label]]`,
`[[project:path]]`, and `[[workspace/project:path]]`, resolved against the
diff --git a/README.md b/README.md
index 5241aa59..ee4acf6e 100644
--- a/README.md
+++ b/README.md
@@ -441,8 +441,9 @@ also set `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN` on the server.
> `openai-compat`, if consolidation fails on large sessions** with
> `did not contain a JSON object` or `serde: unknown variant`, set
> `AI_MEMORY_LLM_COMPAT_STRICT=true`. It sends `response_format=json_schema`
-> (strict) so the engine constrains output to the schema, with a fallback to
-> the default tolerant parser when the engine ignores it. Off by default.
+> (strict) so capable engines constrain output to the schema. If the strict
+> raw call fails, ai-memory falls back to the default tolerant parser. Off by
+> default.
Embeddings are optional and separate from the LLM provider. Set
`AI_MEMORY_EMBEDDING_PROVIDER=openai`, `voyage`, `google`, or `gemini` when
diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rs
index 389b890e..8e2cf616 100644
--- a/crates/ai-memory-cli/src/config.rs
+++ b/crates/ai-memory-cli/src/config.rs
@@ -65,9 +65,8 @@ pub struct Config {
/// parser stays the default for older local engines that ignore
/// `response_format`. Modern engines (recent Ollama, vLLM, LM Studio,
/// llama.cpp) honour structured output; this lets the operator opt in.
- /// When the strict call fails, the provider falls back to the tolerant
- /// parser, so enabling it never regresses below the default. Set with
- /// `AI_MEMORY_LLM_COMPAT_STRICT=true`.
+ /// If the strict raw call fails, the provider falls back to the tolerant
+ /// parser. Set with `AI_MEMORY_LLM_COMPAT_STRICT=true`.
pub llm_compat_strict: bool,
/// Opt-in: run LLM consolidation on SessionEnd (in addition to the
/// always-written heuristic session page), when an LLM provider is
@@ -815,6 +814,7 @@ mod tests {
provider.auth.require_api_key().unwrap().expose_secret(),
"sk-test-key"
);
+ assert!(!provider.compat_strict);
}
#[test]
@@ -854,6 +854,27 @@ mod tests {
assert!(auth.optional_api_key().is_none());
}
+ #[test]
+ fn openai_compat_provider_threads_strict_flag() {
+ let cfg = Config {
+ llm_provider: Some("openai-compat".into()),
+ llm_model: Some("qwen3:32b".into()),
+ llm_base_url: Some("http://localhost:11434/v1".into()),
+ llm_compat_strict: true,
+ ..Config::default()
+ };
+
+ let provider = cfg.llm_provider_config().unwrap().unwrap();
+
+ assert_eq!(provider.provider, ProviderChoice::OpenAiCompat);
+ assert_eq!(provider.model, "qwen3:32b");
+ assert_eq!(
+ provider.base_url.as_deref(),
+ Some("http://localhost:11434/v1")
+ );
+ assert!(provider.compat_strict);
+ }
+
#[test]
fn openai_oauth_provider_uses_data_dir_token_file() {
let tmp = TempDir::new().unwrap();
diff --git a/crates/ai-memory-llm/src/openai_compat.rs b/crates/ai-memory-llm/src/openai_compat.rs
index cad1f75e..1be06797 100644
--- a/crates/ai-memory-llm/src/openai_compat.rs
+++ b/crates/ai-memory-llm/src/openai_compat.rs
@@ -2,9 +2,10 @@
//!
//! Uses the same wire format as [`crate::openai::OpenAiProvider`] but
//! with a configurable base URL (and no key required for most local
-//! deployments). Structured output falls back to "parse first JSON
-//! object out of the text" because most local engines lack reliable
-//! `response_format` honour.
+//! deployments). Structured output defaults to "parse first JSON object
+//! out of the text" because many local engines lack reliable
+//! `response_format` honour; operators can opt into strict
+//! `response_format=json_schema` for engines that support it.
use std::sync::LazyLock;
@@ -126,10 +127,9 @@ impl LlmProvider for OpenAiCompatProvider {
// response_format. The dialect stays Compat, so the api.openai.com
// token/temperature quirks do NOT leak into the local engine.
//
- // If the strict call fails (engine that ignores the schema, truncated
- // JSON, etc.) we fall through to the tolerant parser below instead of
- // propagating the error — so enabling the flag never regresses below
- // the default behaviour.
+ // If the strict raw call fails (engine that rejects the schema,
+ // truncated JSON, etc.) we fall through to the tolerant parser below
+ // instead of propagating that raw-call error.
if self.strict {
let mut strict_schema = schema.clone();
enforce_strict_object_schemas(&mut strict_schema);
diff --git a/docs/deploy.md b/docs/deploy.md
index c18c9643..8fb7c540 100644
--- a/docs/deploy.md
+++ b/docs/deploy.md
@@ -170,12 +170,14 @@ alternatives:
> responses with the strict-JSON consolidation prompt. If you must use one, turn
> reasoning off.
-ai-memory's consolidator uses OpenAI's `json_schema` strict mode for
+ai-memory's hosted OpenAI-family providers use `json_schema` strict mode for
structured output. The OpenAI provider normalizes schemars output into
-OpenAI's supported subset (`additionalProperties: false`, complete
-`required`, generated enum `anyOf`, and plain `$ref` nodes). Most modern
-models honour this, but if you switch to a niche local model, run a quick
-`ai-memory llm-test` (with a structured prompt) before trusting it.
+OpenAI's supported subset (`additionalProperties: false`, complete `required`,
+generated enum `anyOf`, and plain `$ref` nodes). For `openai-compat` local or
+gateway endpoints, the tolerant parser stays the default; set
+`AI_MEMORY_LLM_COMPAT_STRICT=true` only after confirming the endpoint honours
+OpenAI-style `response_format=json_schema`. If you switch to a niche local
+model, run a quick `ai-memory llm-test` before trusting it.
## Backups
diff --git a/docs/install.md b/docs/install.md
index 2e1da430..2ab70a0f 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -806,6 +806,18 @@ required. For OpenRouter (Kimi, DeepSeek, etc.):
-e LLM_API_KEY=sk-or-v1-...
```
+Modern Ollama, vLLM, LM Studio, llama.cpp, and gateway endpoints may honour
+OpenAI-style `response_format=json_schema`. If the tolerant default parser fails
+with errors such as `did not contain a JSON object` or `serde: unknown variant`,
+try strict compat mode:
+
+```bash
+-e AI_MEMORY_LLM_COMPAT_STRICT=true
+```
+
+Strict mode is opt-in. ai-memory sends the schema-constrained request first and
+falls back to the tolerant parser only when that raw strict call fails.
+
---
## Subcommand reference
diff --git a/docs/llm-provider-comparison.md b/docs/llm-provider-comparison.md
index 0411141e..20898cb4 100644
--- a/docs/llm-provider-comparison.md
+++ b/docs/llm-provider-comparison.md
@@ -162,10 +162,13 @@ synonym, never code fences". Local instruction-tuned models -
especially when there's no `response_format: json_schema`
support to enforce - will drift to whatever feels natural.
-Compounding this: openai-compat providers (Ollama, OpenRouter
-passthrough) do **not** expose strict-mode JSON-schema
-validation. The schema is descriptive, not coercive. So the
-prompt has to do the load-bearing work.
+Compounding this at the time of the run: openai-compat providers
+(Ollama, OpenRouter passthrough) were using ai-memory's tolerant
+parser path, so the schema was descriptive, not coercive. ai-memory
+now has opt-in `AI_MEMORY_LLM_COMPAT_STRICT=true` for compatible
+engines that honour OpenAI-style `response_format=json_schema`, but
+the prompt still has to do the load-bearing work when strict mode is
+off or the strict raw call falls back.
## The fix