Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs(llm): document compat strict mode
AkitaOnRails64fd83a1eb09
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 75a44ec5..faa21be2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Added+- `openai-compat` LLM providers can now opt into strict JSON Schema structured+ output with `AI_MEMORY_LLM_COMPAT_STRICT=true`. Strict mode sends+ `response_format=json_schema` first for compatible Ollama, vLLM, LM Studio,+ llama.cpp, and gateway endpoints, while the tolerant JSON-object parser+ remains the default and the fallback for strict raw-call failures ([#70]).- The read-only web browser now renders `[[wiki links]]` as clickable internallinks to the target page. Supports `[[path]]`, `[[path|label]]`,`[[project:path]]`, and `[[workspace/project:path]]`, resolved against thediff --git a/README.md b/README.mdindex 5241aa59..ee4acf6e 100644--- a/README.md+++ b/README.md@@ -441,8 +441,9 @@ also set `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN` on the server.> `openai-compat`, if consolidation fails on large sessions** with> `did not contain a JSON object` or `serde: unknown variant`, set> `AI_MEMORY_LLM_COMPAT_STRICT=true`. It sends `response_format=json_schema`-> (strict) so the engine constrains output to the schema, with a fallback to-> the default tolerant parser when the engine ignores it. Off by default.+> (strict) so capable engines constrain output to the schema. If the strict+> raw call fails, ai-memory falls back to the default tolerant parser. Off by+> default.Embeddings are optional and separate from the LLM provider. Set`AI_MEMORY_EMBEDDING_PROVIDER=openai`, `voyage`, `google`, or `gemini` whendiff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rsindex 389b890e..8e2cf616 100644--- a/crates/ai-memory-cli/src/config.rs+++ b/crates/ai-memory-cli/src/config.rs@@ -65,9 +65,8 @@ pub struct Config {/// parser stays the default for older local engines that ignore/// `response_format`. Modern engines (recent Ollama, vLLM, LM Studio,/// llama.cpp) honour structured output; this lets the operator opt in.- /// When the strict call fails, the provider falls back to the tolerant- /// parser, so enabling it never regresses below the default. Set with- /// `AI_MEMORY_LLM_COMPAT_STRICT=true`.+ /// If the strict raw call fails, the provider falls back to the tolerant+ /// parser. Set with `AI_MEMORY_LLM_COMPAT_STRICT=true`.pub llm_compat_strict: bool,/// Opt-in: run LLM consolidation on SessionEnd (in addition to the/// always-written heuristic session page), when an LLM provider is@@ -815,6 +814,7 @@ mod tests {provider.auth.require_api_key().unwrap().expose_secret(),"sk-test-key");+ assert!(!provider.compat_strict);}#[test]@@ -854,6 +854,27 @@ mod tests {assert!(auth.optional_api_key().is_none());}+ #[test]+ fn openai_compat_provider_threads_strict_flag() {+ let cfg = Config {+ llm_provider: Some("openai-compat".into()),+ llm_model: Some("qwen3:32b".into()),+ llm_base_url: Some("http://localhost:11434/v1".into()),+ llm_compat_strict: true,+ ..Config::default()+ };++ let provider = cfg.llm_provider_config().unwrap().unwrap();++ assert_eq!(provider.provider, ProviderChoice::OpenAiCompat);+ assert_eq!(provider.model, "qwen3:32b");+ assert_eq!(+ provider.base_url.as_deref(),+ Some("http://localhost:11434/v1")+ );+ assert!(provider.compat_strict);+ }+#[test]fn openai_oauth_provider_uses_data_dir_token_file() {let tmp = TempDir::new().unwrap();diff --git a/crates/ai-memory-llm/src/openai_compat.rs b/crates/ai-memory-llm/src/openai_compat.rsindex cad1f75e..1be06797 100644--- a/crates/ai-memory-llm/src/openai_compat.rs+++ b/crates/ai-memory-llm/src/openai_compat.rs@@ -2,9 +2,10 @@//!//! Uses the same wire format as [`crate::openai::OpenAiProvider`] but//! with a configurable base URL (and no key required for most local-//! deployments). Structured output falls back to "parse first JSON-//! object out of the text" because most local engines lack reliable-//! `response_format` honour.+//! deployments). Structured output defaults to "parse first JSON object+//! out of the text" because many local engines lack reliable+//! `response_format` honour; operators can opt into strict+//! `response_format=json_schema` for engines that support it.use std::sync::LazyLock;@@ -126,10 +127,9 @@ impl LlmProvider for OpenAiCompatProvider {// response_format. The dialect stays Compat, so the api.openai.com// token/temperature quirks do NOT leak into the local engine.//- // If the strict call fails (engine that ignores the schema, truncated- // JSON, etc.) we fall through to the tolerant parser below instead of- // propagating the error — so enabling the flag never regresses below- // the default behaviour.+ // If the strict raw call fails (engine that rejects the schema,+ // truncated JSON, etc.) we fall through to the tolerant parser below+ // instead of propagating that raw-call error.if self.strict {let mut strict_schema = schema.clone();enforce_strict_object_schemas(&mut strict_schema);diff --git a/docs/deploy.md b/docs/deploy.mdindex c18c9643..8fb7c540 100644--- a/docs/deploy.md+++ b/docs/deploy.md@@ -170,12 +170,14 @@ alternatives:> responses with the strict-JSON consolidation prompt. If you must use one, turn> reasoning off.-ai-memory's consolidator uses OpenAI's `json_schema` strict mode for+ai-memory's hosted OpenAI-family providers use `json_schema` strict mode forstructured output. The OpenAI provider normalizes schemars output into-OpenAI's supported subset (`additionalProperties: false`, complete-`required`, generated enum `anyOf`, and plain `$ref` nodes). Most modern-models honour this, but if you switch to a niche local model, run a quick-`ai-memory llm-test` (with a structured prompt) before trusting it.+OpenAI's supported subset (`additionalProperties: false`, complete `required`,+generated enum `anyOf`, and plain `$ref` nodes). For `openai-compat` local or+gateway endpoints, the tolerant parser stays the default; set+`AI_MEMORY_LLM_COMPAT_STRICT=true` only after confirming the endpoint honours+OpenAI-style `response_format=json_schema`. If you switch to a niche local+model, run a quick `ai-memory llm-test` before trusting it.## Backupsdiff --git a/docs/install.md b/docs/install.mdindex 2e1da430..2ab70a0f 100644--- a/docs/install.md+++ b/docs/install.md@@ -806,6 +806,18 @@ required. For OpenRouter (Kimi, DeepSeek, etc.):-e LLM_API_KEY=sk-or-v1-...```+Modern Ollama, vLLM, LM Studio, llama.cpp, and gateway endpoints may honour+OpenAI-style `response_format=json_schema`. If the tolerant default parser fails+with errors such as `did not contain a JSON object` or `serde: unknown variant`,+try strict compat mode:++```bash+-e AI_MEMORY_LLM_COMPAT_STRICT=true+```++Strict mode is opt-in. ai-memory sends the schema-constrained request first and+falls back to the tolerant parser only when that raw strict call fails.+---## Subcommand referencediff --git a/docs/llm-provider-comparison.md b/docs/llm-provider-comparison.mdindex 0411141e..20898cb4 100644--- a/docs/llm-provider-comparison.md+++ b/docs/llm-provider-comparison.md@@ -162,10 +162,13 @@ synonym, never code fences". Local instruction-tuned models -especially when there's no `response_format: json_schema`support to enforce - will drift to whatever feels natural.-Compounding this: openai-compat providers (Ollama, OpenRouter-passthrough) do **not** expose strict-mode JSON-schema-validation. The schema is descriptive, not coercive. So the-prompt has to do the load-bearing work.+Compounding this at the time of the run: openai-compat providers+(Ollama, OpenRouter passthrough) were using ai-memory's tolerant+parser path, so the schema was descriptive, not coercive. ai-memory+now has opt-in `AI_MEMORY_LLM_COMPAT_STRICT=true` for compatible+engines that honour OpenAI-style `response_format=json_schema`, but+the prompt still has to do the load-bearing work when strict mode is+off or the strict raw call falls back.## The fix