Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(llm): constrain compat structured output by default (#292)

AkitaOnRails75ff81f49d6c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f185d723..91523e04 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
+- OpenAI-compatible providers now send each structured operation's JSON Schema
+ through `response_format=json_schema` by default, so local models cannot
+ replace consolidation JSON with prose or omit required fields. Explicit
+ structured-output capability rejections fall back to the tolerant parser;
+ other HTTP failures still propagate, and
+ `AI_MEMORY_LLM_COMPAT_STRICT=false` remains the compatibility opt-out. (#292)
- Recognized Antigravity CLI's native file/edit and search tools, applied path
exclusions to its `TargetFile` operations, and captured bounded successful
edit content from `toolCall.args` when the hook omits an output field. Generic
diff --git a/README.md b/README.md
index f8585930..a9e34b54 100644
--- a/README.md
+++ b/README.md
@@ -715,13 +715,12 @@ also set `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN` on the server.
> high-effort thinking models for your coding agent.
> [!TIP]
-> **On a local engine (Ollama, vLLM, LM Studio, llama.cpp) with
-> `openai-compat`, if consolidation fails on large sessions** with
-> `did not contain a JSON object` or `serde: unknown variant`, set
-> `AI_MEMORY_LLM_COMPAT_STRICT=true`. It sends `response_format=json_schema`
-> (strict) so capable engines constrain output to the schema. If the strict
-> raw call fails, ai-memory falls back to the default tolerant parser. Off by
-> default.
+> **OpenAI-compatible structured output is schema-constrained by default.**
+> ai-memory sends each operation's JSON Schema through
+> `response_format=json_schema`, which recent Ollama, vLLM, LM Studio, and
+> llama.cpp releases honour. It falls back to the tolerant parser when an
+> endpoint explicitly rejects that field or returns a malformed shape. Set
+> `AI_MEMORY_LLM_COMPAT_STRICT=false` only for an incompatible endpoint.
Embeddings are optional and separate from the LLM provider. Set
`AI_MEMORY_EMBEDDING_PROVIDER=openai`, `voyage`, `google`, or `gemini` when
diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rs
index 6e061b7d..a2613dc8 100644
--- a/crates/ai-memory-cli/src/config.rs
+++ b/crates/ai-memory-cli/src/config.rs
@@ -80,14 +80,11 @@ pub struct Config {
pub llm_model: Option<String>,
/// Optional LLM base URL override.
pub llm_base_url: Option<String>,
- /// Opt-in: send `response_format=json_schema` (strict) to the
- /// `openai-compat` provider instead of asking for prose JSON and
- /// extracting the first balanced object. Off by default — the tolerant
- /// parser stays the default for older local engines that ignore
- /// `response_format`. Modern engines (recent Ollama, vLLM, LM Studio,
- /// llama.cpp) honour structured output; this lets the operator opt in.
- /// If the strict raw call fails, the provider falls back to the tolerant
- /// parser. Set with `AI_MEMORY_LLM_COMPAT_STRICT=true`.
+ /// Send `response_format=json_schema` (strict) to the `openai-compat`
+ /// provider instead of relying on prose instructions and extracting the
+ /// first balanced object. On by default because every structured call
+ /// already supplies its own schema. Set
+ /// `AI_MEMORY_LLM_COMPAT_STRICT=false` for an incompatible endpoint.
pub llm_compat_strict: bool,
/// Opt-in: run LLM consolidation on SessionEnd (in addition to the
/// always-written heuristic session page), when an LLM provider is
@@ -413,7 +410,7 @@ impl Default for Config {
llm_provider: None,
llm_model: None,
llm_base_url: None,
- llm_compat_strict: false,
+ llm_compat_strict: true,
consolidate_on_session_end: false,
capture_assistant: false,
embedding_provider: None,
@@ -1270,7 +1267,7 @@ mod tests {
provider.auth.require_api_key().unwrap().expose_secret(),
"sk-test-key"
);
- assert!(!provider.compat_strict);
+ assert!(provider.compat_strict);
}
#[test]
@@ -1327,12 +1324,11 @@ mod tests {
}
#[test]
- fn openai_compat_provider_threads_strict_flag() {
- let cfg = Config {
+ fn openai_compat_provider_defaults_strict_and_allows_opt_out() {
+ let mut cfg = Config {
llm_provider: Some("openai-compat".into()),
llm_model: Some("qwen3:32b".into()),
llm_base_url: Some("http://localhost:11434/v1".into()),
- llm_compat_strict: true,
..Config::default()
};
@@ -1345,6 +1341,10 @@ mod tests {
Some("http://localhost:11434/v1")
);
assert!(provider.compat_strict);
+
+ cfg.llm_compat_strict = false;
+ let provider = cfg.llm_provider_config().unwrap().unwrap();
+ assert!(!provider.compat_strict);
}
#[test]
diff --git a/crates/ai-memory-llm/src/factory.rs b/crates/ai-memory-llm/src/factory.rs
index 59a59f7c..1edc9805 100644
--- a/crates/ai-memory-llm/src/factory.rs
+++ b/crates/ai-memory-llm/src/factory.rs
@@ -94,10 +94,10 @@ pub struct ProviderConfig {
pub auth: ProviderAuth,
/// Base URL override (required for OpenAI-compat).
pub base_url: Option<String>,
- /// Opt-in strict mode for the `openai-compat` provider: send
+ /// Strict mode for the `openai-compat` provider: send
/// `response_format=json_schema` instead of the tolerant prose-JSON
- /// parser. Ignored by every other provider. Sourced once from
- /// `AI_MEMORY_LLM_COMPAT_STRICT` by `Config::load`.
+ /// parser. Enabled by default and ignored by every other provider.
+ /// Sourced once from `AI_MEMORY_LLM_COMPAT_STRICT` by `Config::load`.
pub compat_strict: bool,
}
diff --git a/crates/ai-memory-llm/src/openai_compat.rs b/crates/ai-memory-llm/src/openai_compat.rs
index 69d0fd0b..3bed1122 100644
--- a/crates/ai-memory-llm/src/openai_compat.rs
+++ b/crates/ai-memory-llm/src/openai_compat.rs
@@ -119,7 +119,7 @@ impl LlmProvider for OpenAiCompatProvider {
request: ChatRequest,
schema: serde_json::Value,
) -> LlmResult<serde_json::Value> {
- // Strict mode (opt-in; see `strict` field): modern local engines
+ // Strict mode: modern local engines
// honour `response_format=json_schema`. Normalise the schema
// to the strict subset (additionalProperties:false, full required, no
// oneOf / no $ref siblings) — the same rewrite the Official dialect
@@ -134,12 +134,10 @@ impl LlmProvider for OpenAiCompatProvider {
// silently drop it.
//
// Fallback path:
- // Fallback only on *parse-shape* failures — the engine returned a
- // response but it wasn't a valid JSON object (the case the tolerant
- // path was designed for). On `LlmError::Provider` (HTTP 4xx/5xx,
- // auth, rate-limit) and transport-level failures, propagate the
- // error: a `complete()` retry inside the fallback would hit the
- // same wall and double the latency / token spend for no gain.
+ // Fallback on *parse-shape* failures, or when a compatibility endpoint
+ // explicitly rejects `response_format` / `json_schema` before model
+ // execution. Other provider and transport failures propagate: a
+ // retry would hit the same wall and may double latency or token spend.
//
// The fallback itself is a SECOND HTTP call to the model — the
// strict raw call returns parsed JSON or an error, not raw text,
@@ -161,6 +159,9 @@ impl LlmProvider for OpenAiCompatProvider {
Err(err) if is_parse_shape_error(&err) => {
debug!(error = %err, "compat strict parse-shape mismatch, falling back to tolerant parser");
}
+ Err(err) if is_response_format_rejection(&err) => {
+ debug!(error = %err, "compat endpoint rejected response_format, falling back to tolerant parser");
+ }
Err(err) => {
// Transport / 5xx / auth / rate-limit: a tolerant retry
// would hit the same failure. Propagate so the caller
@@ -213,6 +214,23 @@ fn is_parse_shape_error(err: &LlmError) -> bool {
matches!(err, LlmError::UnexpectedShape(_) | LlmError::Serde(_))
}
+/// True only when a compatibility endpoint rejected structured-output syntax
+/// before generation. Generic 4xx errors are deliberately excluded so auth,
+/// quota, model, and malformed-request failures retain their original cause.
+fn is_response_format_rejection(err: &LlmError) -> bool {
+ let LlmError::Provider { status, body } = err else {
+ return false;
+ };
+ if !matches!(status, 400 | 422) {
+ return false;
+ }
+ let body = body.to_ascii_lowercase();
+ body.contains("response_format")
+ || body.contains("json_schema")
+ || body.contains("structured output")
+ || body.contains("structured-output")
+}
+
/// Find the first balanced `{...}` object in a string, skipping
/// braces that appear inside JSON string literals.
///
@@ -259,8 +277,8 @@ fn first_json_object(s: &str) -> Option<&str> {
mod tests {
use super::*;
- /// `new` defaults strict off; `with_strict` is the only way to turn it
- /// on, mirroring how the factory threads `ProviderConfig::compat_strict`.
+ /// The low-level constructor stays tolerant so wrappers can choose their
+ /// own compatibility policy; runtime configuration supplies the default.
#[test]
fn strict_defaults_off_and_can_be_overridden() {
let p = OpenAiCompatProvider::new("http://localhost:11434/v1", None, "mistral-nemo")
@@ -270,12 +288,28 @@ mod tests {
assert!(p.strict);
}
- /// The strict path's fallback policy lives in `is_parse_shape_error`:
- /// fall back ONLY when the upstream returned a response in the wrong
- /// shape (the tolerant prose-JSON parser has a real chance). Propagate
- /// transport / HTTP-status / auth errors — a tolerant retry would just
- /// hit the same wall and double cost. Regression guard for the audit
- /// finding that the original strict-fallback caught every `Err(_)`.
+ #[test]
+ fn only_explicit_response_format_rejections_are_retryable() {
+ assert!(is_response_format_rejection(&LlmError::Provider {
+ status: 400,
+ body: "unsupported parameter: response_format".into(),
+ }));
+ assert!(is_response_format_rejection(&LlmError::Provider {
+ status: 422,
+ body: "json_schema is not supported".into(),
+ }));
+ assert!(!is_response_format_rejection(&LlmError::Provider {
+ status: 400,
+ body: "unknown model".into(),
+ }));
+ assert!(!is_response_format_rejection(&LlmError::Provider {
+ status: 401,
+ body: "response_format requires authentication".into(),
+ }));
+ }
+
+ /// Parse-shape failures are retryable. Provider capability rejection is
+ /// classified separately; generic provider and transport failures are not.
#[test]
fn is_parse_shape_error_classifies_correctly() {
// Shape failures → fall back.
diff --git a/crates/ai-memory-llm/tests/openai_compat_strict.rs b/crates/ai-memory-llm/tests/openai_compat_strict.rs
index f061a01a..253ad52a 100644
--- a/crates/ai-memory-llm/tests/openai_compat_strict.rs
+++ b/crates/ai-memory-llm/tests/openai_compat_strict.rs
@@ -4,9 +4,10 @@
//! The strict path is the load-bearing addition from PR #70: when
//! `compat_strict=true`, the provider sends `response_format=json_schema`
//! first; on a *parse-shape* failure (HTTP 200 with bad JSON, or 200
-//! with a non-object) it falls back to the tolerant prose-JSON parser
-//! the default mode uses. On transport / HTTP-status / auth failures
-//! it propagates without retry. These tests assert those four outcomes
+//! with a non-object), or an explicit structured-output capability
+//! rejection, it falls back to the tolerant prose-JSON parser. Other
+//! transport / HTTP-status / auth failures propagate without retry.
+//! These tests assert those outcomes
//! end-to-end against synthesised upstream responses.
//!
//! Why not unit-test in `openai_compat.rs`: the strict path delegates
@@ -229,6 +230,57 @@ async fn strict_upstream_401_propagates_without_fallback_retry() {
);
}
+/// Older compatibility endpoints may reject the structured-output field
+/// before running the model. That specific capability failure is safe to
+/// retry without `response_format`; a generic 400 remains non-retryable.
+#[tokio::test]
+async fn strict_response_format_rejection_falls_back_to_tolerant_parser() {
+ let server = MockServer::start().await;
+ let counter = Arc::new(AtomicUsize::new(0));
+ let counter_clone = counter.clone();
+ Mock::given(method("POST"))
+ .and(path("/v1/chat/completions"))
+ .respond_with(move |req: &Request| {
+ counter_clone.fetch_add(1, Ordering::SeqCst);
+ let request: serde_json::Value =
+ serde_json::from_slice(&req.body).expect("request body is JSON");
+ if request.get("response_format").is_some() {
+ ResponseTemplate::new(400).set_body_string("unsupported parameter: response_format")
+ } else {
+ ResponseTemplate::new(200).set_body_json(body_with_content("result: {\"ok\":true}"))
+ }
+ })
+ .mount(&server)
+ .await;
+
+ let provider = strict_provider(server.uri());
+ let value = provider
+ .complete_structured_raw(tiny_request(), tiny_schema())
+ .await
+ .expect("capability rejection must use tolerant fallback");
+ assert_eq!(value, json!({"ok": true}));
+ assert_eq!(counter.load(Ordering::SeqCst), 2);
+}
+
+#[tokio::test]
+async fn strict_generic_400_propagates_without_fallback_retry() {
+ let server = MockServer::start().await;
+ let responder = CountingResponder::raw(400, "unknown model");
+ Mock::given(method("POST"))
+ .and(path("/v1/chat/completions"))
+ .respond_with(responder.clone())
+ .mount(&server)
+ .await;
+
+ let provider = strict_provider(server.uri());
+ let err = provider
+ .complete_structured_raw(tiny_request(), tiny_schema())
+ .await
+ .expect_err("generic 400 must surface");
+ assert!(err.to_string().contains("400"));
+ assert_eq!(responder.count(), 1);
+}
+
/// Reasoning models (DeepSeek / Qwen3 / MiniMax M2.7) embed
/// `<think>…</think>` inside `content` before the JSON. Strict-mode
/// users on those models should keep the flag OFF; this test
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 8e02111f..91c893bf 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -478,6 +478,7 @@ AI_MEMORY_LLM_PROVIDER anthropic | openai | openai-oauth | copilot | gemini
AI_MEMORY_LLM_MODEL optional when the provider has a default; e.g. claude-haiku-4-5, gpt-5.4-mini
ANTHROPIC_API_KEY / OPENAI_API_KEY / GEMINI_API_KEY / LLM_API_KEY
AI_MEMORY_LLM_BASE_URL for openai-compat (Ollama, vLLM)
+AI_MEMORY_LLM_COMPAT_STRICT true by default; false disables response_format=json_schema
COPILOT_GITHUB_TOKEN optional GitHub token for copilot
GITHUB_COPILOT_API_TOKEN optional pre-minted Copilot API token
COPILOT_API_URL optional Copilot API base URL override
diff --git a/docs/deploy.md b/docs/deploy.md
index d569ce14..5f657d85 100644
--- a/docs/deploy.md
+++ b/docs/deploy.md
@@ -174,11 +174,11 @@ alternatives:
ai-memory's hosted OpenAI-family providers use `json_schema` strict mode for
structured output. The OpenAI provider normalizes schemars output into
OpenAI's supported subset (`additionalProperties: false`, complete `required`,
-generated enum `anyOf`, and plain `$ref` nodes). For `openai-compat` local or
-gateway endpoints, the tolerant parser stays the default; set
-`AI_MEMORY_LLM_COMPAT_STRICT=true` only after confirming the endpoint honours
-OpenAI-style `response_format=json_schema`. If you switch to a niche local
-model, run a quick `ai-memory llm-test` before trusting it.
+generated enum `anyOf`, and plain `$ref` nodes). `openai-compat` local and
+gateway endpoints use the same schema-constrained request by default, with a
+tolerant fallback for explicit capability rejection or malformed output. Set
+`AI_MEMORY_LLM_COMPAT_STRICT=false` for an incompatible endpoint. If you switch
+to a niche local model, run a quick `ai-memory llm-test` before trusting it.
## Backups
diff --git a/docs/install.md b/docs/install.md
index 21571522..03061b4d 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -1187,17 +1187,20 @@ through the generic compatibility credential:
Replace the model with another current Atlas model id when needed. ai-memory
does not select a default for hosted compatibility endpoints.
-Modern Ollama, vLLM, LM Studio, llama.cpp, and gateway endpoints may honour
-OpenAI-style `response_format=json_schema`. If the tolerant default parser fails
-with errors such as `did not contain a JSON object` or `serde: unknown variant`,
-try strict compat mode:
+OpenAI-compatible structured calls use the operation's JSON Schema by default:
```bash
-e AI_MEMORY_LLM_COMPAT_STRICT=true
```
-Strict mode is opt-in. ai-memory sends the schema-constrained request first and
-falls back to the tolerant parser only when that raw strict call fails.
+Modern Ollama, vLLM, LM Studio, llama.cpp, and gateway endpoints honour this
+OpenAI-style `response_format=json_schema` request. ai-memory retries with its
+tolerant parser when an endpoint explicitly rejects the structured-output field
+or returns a malformed response shape. For an incompatible endpoint, opt out:
+
+```bash
+-e AI_MEMORY_LLM_COMPAT_STRICT=false
+```
---
diff --git a/docs/llm-provider-comparison.md b/docs/llm-provider-comparison.md
index 4610a619..43b0f494 100644
--- a/docs/llm-provider-comparison.md
+++ b/docs/llm-provider-comparison.md
@@ -167,16 +167,15 @@ Compounding this at the time of the run: openai-compat providers
parser path, so the schema was descriptive, not coercive. The
provider opt-in for coercive structured output is documented below.
-#### Opting into strict structured output for openai-compat
+#### Strict structured output for openai-compat
-Set `AI_MEMORY_LLM_COMPAT_STRICT=true` to make `openai-compat` send
-`response_format={ type: "json_schema", strict: true }` first. On a
+`openai-compat` sends
+`response_format={ type: "json_schema", strict: true }` by default. On a
parse-shape failure (the upstream returned a response but it wasn't a
-valid JSON object), ai-memory falls back to the tolerant parser the
-default mode uses; on HTTP 4xx / 5xx / auth / transport errors the
-strict call propagates without retry (a tolerant retry would just hit
-the same wall and double cost — see `is_parse_shape_error` in
-`crates/ai-memory-llm/src/openai_compat.rs`).
+valid JSON object), or an explicit 400/422 rejection naming
+`response_format`, `json_schema`, or structured output, ai-memory falls back to
+the tolerant parser. Other HTTP / auth / transport errors propagate without
+retry. Set `AI_MEMORY_LLM_COMPAT_STRICT=false` to opt out.
**Cost.** Strict mode is one HTTP call when the upstream honours
`response_format`. When it doesn't, you pay a second call for the
@@ -184,9 +183,10 @@ tolerant fallback. Pick by engine:
| Engine class | Setting |
|---|---|
-| Modern Ollama / vLLM / LM Studio honouring `response_format=json_schema` | `AI_MEMORY_LLM_COMPAT_STRICT=true` (one call, schema-constrained) |
-| Reasoning models with `<think>…</think>` inside `content` (DeepSeek-R1, Qwen3-Thinking, MiniMax M2) | Leave OFF. Strict-then-fallback burns a call per consolidation; the tolerant path already strips `<think>` before parsing |
-| Older engines / proxies that ignore `response_format` | Leave OFF. Strict adds latency and recovers nothing |
+| Modern Ollama / vLLM / LM Studio honouring `response_format=json_schema` | Keep the default (one call, schema-constrained) |
+| Reasoning models with `<think>…</think>` inside `content` (DeepSeek-R1, Qwen3-Thinking, MiniMax M2) | Set `AI_MEMORY_LLM_COMPAT_STRICT=false` if the strict-then-fallback double call is frequent |
+| Older engines / proxies that reject `response_format` explicitly | Keep the default; ai-memory retries without it |
+| Older engines / proxies that mishandle the field without a recognizable rejection | Set `AI_MEMORY_LLM_COMPAT_STRICT=false` |
The prompt still has to do the load-bearing work when strict mode is
off or the strict call falls back.
@@ -703,10 +703,9 @@ Re-run this harness when any of the following changes:
drops for Ollama)
- A new fixture is added to `evals/fixtures/`
- The home server hardware changes
-- A new local engine ships first-class
- `response_format=json_schema` support, making
- `AI_MEMORY_LLM_COMPAT_STRICT=true` worth re-benchmarking against
- the tolerant-parser baseline
+- A local engine changes its `response_format=json_schema` implementation,
+ making a fresh default-vs-`AI_MEMORY_LLM_COMPAT_STRICT=false` comparison
+ worthwhile
## How to reproduce
diff --git a/evals/src/main.rs b/evals/src/main.rs
index 144db9ac..23a214ed 100644
--- a/evals/src/main.rs
+++ b/evals/src/main.rs
@@ -368,9 +368,9 @@ impl From<ResolvedConfig> for ProviderConfig {
model: r.model,
auth: r.auth,
base_url: r.base_url,
- // The eval harness benchmarks the default tolerant path; strict
- // mode is an operator opt-in not modeled by these comparisons.
- compat_strict: false,
+ // Match the product default so provider comparisons exercise the
+ // same schema-constrained path operators receive.
+ compat_strict: true,
}
}
}