Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(llm): constrain compat structured output by default (#292)
AkitaOnRails75ff81f49d6c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f185d723..91523e04 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Fixed+- OpenAI-compatible providers now send each structured operation's JSON Schema+ through `response_format=json_schema` by default, so local models cannot+ replace consolidation JSON with prose or omit required fields. Explicit+ structured-output capability rejections fall back to the tolerant parser;+ other HTTP failures still propagate, and+ `AI_MEMORY_LLM_COMPAT_STRICT=false` remains the compatibility opt-out. (#292)- Recognized Antigravity CLI's native file/edit and search tools, applied pathexclusions to its `TargetFile` operations, and captured bounded successfuledit content from `toolCall.args` when the hook omits an output field. Genericdiff --git a/README.md b/README.mdindex f8585930..a9e34b54 100644--- a/README.md+++ b/README.md@@ -715,13 +715,12 @@ also set `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN` on the server.> high-effort thinking models for your coding agent.> [!TIP]-> **On a local engine (Ollama, vLLM, LM Studio, llama.cpp) with-> `openai-compat`, if consolidation fails on large sessions** with-> `did not contain a JSON object` or `serde: unknown variant`, set-> `AI_MEMORY_LLM_COMPAT_STRICT=true`. It sends `response_format=json_schema`-> (strict) so capable engines constrain output to the schema. If the strict-> raw call fails, ai-memory falls back to the default tolerant parser. Off by-> default.+> **OpenAI-compatible structured output is schema-constrained by default.**+> ai-memory sends each operation's JSON Schema through+> `response_format=json_schema`, which recent Ollama, vLLM, LM Studio, and+> llama.cpp releases honour. It falls back to the tolerant parser when an+> endpoint explicitly rejects that field or returns a malformed shape. Set+> `AI_MEMORY_LLM_COMPAT_STRICT=false` only for an incompatible endpoint.Embeddings are optional and separate from the LLM provider. Set`AI_MEMORY_EMBEDDING_PROVIDER=openai`, `voyage`, `google`, or `gemini` whendiff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rsindex 6e061b7d..a2613dc8 100644--- a/crates/ai-memory-cli/src/config.rs+++ b/crates/ai-memory-cli/src/config.rs@@ -80,14 +80,11 @@ pub struct Config {pub llm_model: Option<String>,/// Optional LLM base URL override.pub llm_base_url: Option<String>,- /// Opt-in: send `response_format=json_schema` (strict) to the- /// `openai-compat` provider instead of asking for prose JSON and- /// extracting the first balanced object. Off by default — the tolerant- /// parser stays the default for older local engines that ignore- /// `response_format`. Modern engines (recent Ollama, vLLM, LM Studio,- /// llama.cpp) honour structured output; this lets the operator opt in.- /// If the strict raw call fails, the provider falls back to the tolerant- /// parser. Set with `AI_MEMORY_LLM_COMPAT_STRICT=true`.+ /// Send `response_format=json_schema` (strict) to the `openai-compat`+ /// provider instead of relying on prose instructions and extracting the+ /// first balanced object. On by default because every structured call+ /// already supplies its own schema. Set+ /// `AI_MEMORY_LLM_COMPAT_STRICT=false` for an incompatible endpoint.pub llm_compat_strict: bool,/// Opt-in: run LLM consolidation on SessionEnd (in addition to the/// always-written heuristic session page), when an LLM provider is@@ -413,7 +410,7 @@ impl Default for Config {llm_provider: None,llm_model: None,llm_base_url: None,- llm_compat_strict: false,+ llm_compat_strict: true,consolidate_on_session_end: false,capture_assistant: false,embedding_provider: None,@@ -1270,7 +1267,7 @@ mod tests {provider.auth.require_api_key().unwrap().expose_secret(),"sk-test-key");- assert!(!provider.compat_strict);+ assert!(provider.compat_strict);}#[test]@@ -1327,12 +1324,11 @@ mod tests {}#[test]- fn openai_compat_provider_threads_strict_flag() {- let cfg = Config {+ fn openai_compat_provider_defaults_strict_and_allows_opt_out() {+ let mut cfg = Config {llm_provider: Some("openai-compat".into()),llm_model: Some("qwen3:32b".into()),llm_base_url: Some("http://localhost:11434/v1".into()),- llm_compat_strict: true,..Config::default()};@@ -1345,6 +1341,10 @@ mod tests {Some("http://localhost:11434/v1"));assert!(provider.compat_strict);++ cfg.llm_compat_strict = false;+ let provider = cfg.llm_provider_config().unwrap().unwrap();+ assert!(!provider.compat_strict);}#[test]diff --git a/crates/ai-memory-llm/src/factory.rs b/crates/ai-memory-llm/src/factory.rsindex 59a59f7c..1edc9805 100644--- a/crates/ai-memory-llm/src/factory.rs+++ b/crates/ai-memory-llm/src/factory.rs@@ -94,10 +94,10 @@ pub struct ProviderConfig {pub auth: ProviderAuth,/// Base URL override (required for OpenAI-compat).pub base_url: Option<String>,- /// Opt-in strict mode for the `openai-compat` provider: send+ /// Strict mode for the `openai-compat` provider: send/// `response_format=json_schema` instead of the tolerant prose-JSON- /// parser. Ignored by every other provider. Sourced once from- /// `AI_MEMORY_LLM_COMPAT_STRICT` by `Config::load`.+ /// parser. Enabled by default and ignored by every other provider.+ /// Sourced once from `AI_MEMORY_LLM_COMPAT_STRICT` by `Config::load`.pub compat_strict: bool,}diff --git a/crates/ai-memory-llm/src/openai_compat.rs b/crates/ai-memory-llm/src/openai_compat.rsindex 69d0fd0b..3bed1122 100644--- a/crates/ai-memory-llm/src/openai_compat.rs+++ b/crates/ai-memory-llm/src/openai_compat.rs@@ -119,7 +119,7 @@ impl LlmProvider for OpenAiCompatProvider {request: ChatRequest,schema: serde_json::Value,) -> LlmResult<serde_json::Value> {- // Strict mode (opt-in; see `strict` field): modern local engines+ // Strict mode: modern local engines// honour `response_format=json_schema`. Normalise the schema// to the strict subset (additionalProperties:false, full required, no// oneOf / no $ref siblings) — the same rewrite the Official dialect@@ -134,12 +134,10 @@ impl LlmProvider for OpenAiCompatProvider {// silently drop it.//// Fallback path:- // Fallback only on *parse-shape* failures — the engine returned a- // response but it wasn't a valid JSON object (the case the tolerant- // path was designed for). On `LlmError::Provider` (HTTP 4xx/5xx,- // auth, rate-limit) and transport-level failures, propagate the- // error: a `complete()` retry inside the fallback would hit the- // same wall and double the latency / token spend for no gain.+ // Fallback on *parse-shape* failures, or when a compatibility endpoint+ // explicitly rejects `response_format` / `json_schema` before model+ // execution. Other provider and transport failures propagate: a+ // retry would hit the same wall and may double latency or token spend.//// The fallback itself is a SECOND HTTP call to the model — the// strict raw call returns parsed JSON or an error, not raw text,@@ -161,6 +159,9 @@ impl LlmProvider for OpenAiCompatProvider {Err(err) if is_parse_shape_error(&err) => {debug!(error = %err, "compat strict parse-shape mismatch, falling back to tolerant parser");}+ Err(err) if is_response_format_rejection(&err) => {+ debug!(error = %err, "compat endpoint rejected response_format, falling back to tolerant parser");+ }Err(err) => {// Transport / 5xx / auth / rate-limit: a tolerant retry// would hit the same failure. Propagate so the caller@@ -213,6 +214,23 @@ fn is_parse_shape_error(err: &LlmError) -> bool {matches!(err, LlmError::UnexpectedShape(_) | LlmError::Serde(_))}+/// True only when a compatibility endpoint rejected structured-output syntax+/// before generation. Generic 4xx errors are deliberately excluded so auth,+/// quota, model, and malformed-request failures retain their original cause.+fn is_response_format_rejection(err: &LlmError) -> bool {+ let LlmError::Provider { status, body } = err else {+ return false;+ };+ if !matches!(status, 400 | 422) {+ return false;+ }+ let body = body.to_ascii_lowercase();+ body.contains("response_format")+ || body.contains("json_schema")+ || body.contains("structured output")+ || body.contains("structured-output")+}+/// Find the first balanced `{...}` object in a string, skipping/// braces that appear inside JSON string literals.///@@ -259,8 +277,8 @@ fn first_json_object(s: &str) -> Option<&str> {mod tests {use super::*;- /// `new` defaults strict off; `with_strict` is the only way to turn it- /// on, mirroring how the factory threads `ProviderConfig::compat_strict`.+ /// The low-level constructor stays tolerant so wrappers can choose their+ /// own compatibility policy; runtime configuration supplies the default.#[test]fn strict_defaults_off_and_can_be_overridden() {let p = OpenAiCompatProvider::new("http://localhost:11434/v1", None, "mistral-nemo")@@ -270,12 +288,28 @@ mod tests {assert!(p.strict);}- /// The strict path's fallback policy lives in `is_parse_shape_error`:- /// fall back ONLY when the upstream returned a response in the wrong- /// shape (the tolerant prose-JSON parser has a real chance). Propagate- /// transport / HTTP-status / auth errors — a tolerant retry would just- /// hit the same wall and double cost. Regression guard for the audit- /// finding that the original strict-fallback caught every `Err(_)`.+ #[test]+ fn only_explicit_response_format_rejections_are_retryable() {+ assert!(is_response_format_rejection(&LlmError::Provider {+ status: 400,+ body: "unsupported parameter: response_format".into(),+ }));+ assert!(is_response_format_rejection(&LlmError::Provider {+ status: 422,+ body: "json_schema is not supported".into(),+ }));+ assert!(!is_response_format_rejection(&LlmError::Provider {+ status: 400,+ body: "unknown model".into(),+ }));+ assert!(!is_response_format_rejection(&LlmError::Provider {+ status: 401,+ body: "response_format requires authentication".into(),+ }));+ }++ /// Parse-shape failures are retryable. Provider capability rejection is+ /// classified separately; generic provider and transport failures are not.#[test]fn is_parse_shape_error_classifies_correctly() {// Shape failures → fall back.diff --git a/crates/ai-memory-llm/tests/openai_compat_strict.rs b/crates/ai-memory-llm/tests/openai_compat_strict.rsindex f061a01a..253ad52a 100644--- a/crates/ai-memory-llm/tests/openai_compat_strict.rs+++ b/crates/ai-memory-llm/tests/openai_compat_strict.rs@@ -4,9 +4,10 @@//! The strict path is the load-bearing addition from PR #70: when//! `compat_strict=true`, the provider sends `response_format=json_schema`//! first; on a *parse-shape* failure (HTTP 200 with bad JSON, or 200-//! with a non-object) it falls back to the tolerant prose-JSON parser-//! the default mode uses. On transport / HTTP-status / auth failures-//! it propagates without retry. These tests assert those four outcomes+//! with a non-object), or an explicit structured-output capability+//! rejection, it falls back to the tolerant prose-JSON parser. Other+//! transport / HTTP-status / auth failures propagate without retry.+//! These tests assert those outcomes//! end-to-end against synthesised upstream responses.//!//! Why not unit-test in `openai_compat.rs`: the strict path delegates@@ -229,6 +230,57 @@ async fn strict_upstream_401_propagates_without_fallback_retry() {);}+/// Older compatibility endpoints may reject the structured-output field+/// before running the model. That specific capability failure is safe to+/// retry without `response_format`; a generic 400 remains non-retryable.+#[tokio::test]+async fn strict_response_format_rejection_falls_back_to_tolerant_parser() {+ let server = MockServer::start().await;+ let counter = Arc::new(AtomicUsize::new(0));+ let counter_clone = counter.clone();+ Mock::given(method("POST"))+ .and(path("/v1/chat/completions"))+ .respond_with(move |req: &Request| {+ counter_clone.fetch_add(1, Ordering::SeqCst);+ let request: serde_json::Value =+ serde_json::from_slice(&req.body).expect("request body is JSON");+ if request.get("response_format").is_some() {+ ResponseTemplate::new(400).set_body_string("unsupported parameter: response_format")+ } else {+ ResponseTemplate::new(200).set_body_json(body_with_content("result: {\"ok\":true}"))+ }+ })+ .mount(&server)+ .await;++ let provider = strict_provider(server.uri());+ let value = provider+ .complete_structured_raw(tiny_request(), tiny_schema())+ .await+ .expect("capability rejection must use tolerant fallback");+ assert_eq!(value, json!({"ok": true}));+ assert_eq!(counter.load(Ordering::SeqCst), 2);+}++#[tokio::test]+async fn strict_generic_400_propagates_without_fallback_retry() {+ let server = MockServer::start().await;+ let responder = CountingResponder::raw(400, "unknown model");+ Mock::given(method("POST"))+ .and(path("/v1/chat/completions"))+ .respond_with(responder.clone())+ .mount(&server)+ .await;++ let provider = strict_provider(server.uri());+ let err = provider+ .complete_structured_raw(tiny_request(), tiny_schema())+ .await+ .expect_err("generic 400 must surface");+ assert!(err.to_string().contains("400"));+ assert_eq!(responder.count(), 1);+}+/// Reasoning models (DeepSeek / Qwen3 / MiniMax M2.7) embed/// `<think>…</think>` inside `content` before the JSON. Strict-mode/// users on those models should keep the flag OFF; this testdiff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 8e02111f..91c893bf 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -478,6 +478,7 @@ AI_MEMORY_LLM_PROVIDER anthropic | openai | openai-oauth | copilot | geminiAI_MEMORY_LLM_MODEL optional when the provider has a default; e.g. claude-haiku-4-5, gpt-5.4-miniANTHROPIC_API_KEY / OPENAI_API_KEY / GEMINI_API_KEY / LLM_API_KEYAI_MEMORY_LLM_BASE_URL for openai-compat (Ollama, vLLM)+AI_MEMORY_LLM_COMPAT_STRICT true by default; false disables response_format=json_schemaCOPILOT_GITHUB_TOKEN optional GitHub token for copilotGITHUB_COPILOT_API_TOKEN optional pre-minted Copilot API tokenCOPILOT_API_URL optional Copilot API base URL overridediff --git a/docs/deploy.md b/docs/deploy.mdindex d569ce14..5f657d85 100644--- a/docs/deploy.md+++ b/docs/deploy.md@@ -174,11 +174,11 @@ alternatives:ai-memory's hosted OpenAI-family providers use `json_schema` strict mode forstructured output. The OpenAI provider normalizes schemars output intoOpenAI's supported subset (`additionalProperties: false`, complete `required`,-generated enum `anyOf`, and plain `$ref` nodes). For `openai-compat` local or-gateway endpoints, the tolerant parser stays the default; set-`AI_MEMORY_LLM_COMPAT_STRICT=true` only after confirming the endpoint honours-OpenAI-style `response_format=json_schema`. If you switch to a niche local-model, run a quick `ai-memory llm-test` before trusting it.+generated enum `anyOf`, and plain `$ref` nodes). `openai-compat` local and+gateway endpoints use the same schema-constrained request by default, with a+tolerant fallback for explicit capability rejection or malformed output. Set+`AI_MEMORY_LLM_COMPAT_STRICT=false` for an incompatible endpoint. If you switch+to a niche local model, run a quick `ai-memory llm-test` before trusting it.## Backupsdiff --git a/docs/install.md b/docs/install.mdindex 21571522..03061b4d 100644--- a/docs/install.md+++ b/docs/install.md@@ -1187,17 +1187,20 @@ through the generic compatibility credential:Replace the model with another current Atlas model id when needed. ai-memorydoes not select a default for hosted compatibility endpoints.-Modern Ollama, vLLM, LM Studio, llama.cpp, and gateway endpoints may honour-OpenAI-style `response_format=json_schema`. If the tolerant default parser fails-with errors such as `did not contain a JSON object` or `serde: unknown variant`,-try strict compat mode:+OpenAI-compatible structured calls use the operation's JSON Schema by default:```bash-e AI_MEMORY_LLM_COMPAT_STRICT=true```-Strict mode is opt-in. ai-memory sends the schema-constrained request first and-falls back to the tolerant parser only when that raw strict call fails.+Modern Ollama, vLLM, LM Studio, llama.cpp, and gateway endpoints honour this+OpenAI-style `response_format=json_schema` request. ai-memory retries with its+tolerant parser when an endpoint explicitly rejects the structured-output field+or returns a malformed response shape. For an incompatible endpoint, opt out:++```bash+-e AI_MEMORY_LLM_COMPAT_STRICT=false+```---diff --git a/docs/llm-provider-comparison.md b/docs/llm-provider-comparison.mdindex 4610a619..43b0f494 100644--- a/docs/llm-provider-comparison.md+++ b/docs/llm-provider-comparison.md@@ -167,16 +167,15 @@ Compounding this at the time of the run: openai-compat providersparser path, so the schema was descriptive, not coercive. Theprovider opt-in for coercive structured output is documented below.-#### Opting into strict structured output for openai-compat+#### Strict structured output for openai-compat-Set `AI_MEMORY_LLM_COMPAT_STRICT=true` to make `openai-compat` send-`response_format={ type: "json_schema", strict: true }` first. On a+`openai-compat` sends+`response_format={ type: "json_schema", strict: true }` by default. On aparse-shape failure (the upstream returned a response but it wasn't a-valid JSON object), ai-memory falls back to the tolerant parser the-default mode uses; on HTTP 4xx / 5xx / auth / transport errors the-strict call propagates without retry (a tolerant retry would just hit-the same wall and double cost — see `is_parse_shape_error` in-`crates/ai-memory-llm/src/openai_compat.rs`).+valid JSON object), or an explicit 400/422 rejection naming+`response_format`, `json_schema`, or structured output, ai-memory falls back to+the tolerant parser. Other HTTP / auth / transport errors propagate without+retry. Set `AI_MEMORY_LLM_COMPAT_STRICT=false` to opt out.**Cost.** Strict mode is one HTTP call when the upstream honours`response_format`. When it doesn't, you pay a second call for the@@ -184,9 +183,10 @@ tolerant fallback. Pick by engine:| Engine class | Setting ||---|---|-| Modern Ollama / vLLM / LM Studio honouring `response_format=json_schema` | `AI_MEMORY_LLM_COMPAT_STRICT=true` (one call, schema-constrained) |-| Reasoning models with `<think>…</think>` inside `content` (DeepSeek-R1, Qwen3-Thinking, MiniMax M2) | Leave OFF. Strict-then-fallback burns a call per consolidation; the tolerant path already strips `<think>` before parsing |-| Older engines / proxies that ignore `response_format` | Leave OFF. Strict adds latency and recovers nothing |+| Modern Ollama / vLLM / LM Studio honouring `response_format=json_schema` | Keep the default (one call, schema-constrained) |+| Reasoning models with `<think>…</think>` inside `content` (DeepSeek-R1, Qwen3-Thinking, MiniMax M2) | Set `AI_MEMORY_LLM_COMPAT_STRICT=false` if the strict-then-fallback double call is frequent |+| Older engines / proxies that reject `response_format` explicitly | Keep the default; ai-memory retries without it |+| Older engines / proxies that mishandle the field without a recognizable rejection | Set `AI_MEMORY_LLM_COMPAT_STRICT=false` |The prompt still has to do the load-bearing work when strict mode isoff or the strict call falls back.@@ -703,10 +703,9 @@ Re-run this harness when any of the following changes:drops for Ollama)- A new fixture is added to `evals/fixtures/`- The home server hardware changes-- A new local engine ships first-class- `response_format=json_schema` support, making- `AI_MEMORY_LLM_COMPAT_STRICT=true` worth re-benchmarking against- the tolerant-parser baseline+- A local engine changes its `response_format=json_schema` implementation,+ making a fresh default-vs-`AI_MEMORY_LLM_COMPAT_STRICT=false` comparison+ worthwhile## How to reproducediff --git a/evals/src/main.rs b/evals/src/main.rsindex 144db9ac..23a214ed 100644--- a/evals/src/main.rs+++ b/evals/src/main.rs@@ -368,9 +368,9 @@ impl From<ResolvedConfig> for ProviderConfig {model: r.model,auth: r.auth,base_url: r.base_url,- // The eval harness benchmarks the default tolerant path; strict- // mode is an operator opt-in not modeled by these comparisons.- compat_strict: false,+ // Match the product default so provider comparisons exercise the+ // same schema-constrained path operators receive.+ compat_strict: true,}}}