Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(llm): disable Gemini Flash thinking budget

AkitaOnRailsbf55fdb6c32c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a190d468..d9fdb2d4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,6 +22,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Experimental native Windows support: PowerShell Docker wrapper,
`ai-memory.cmd`, `.ps1` lifecycle hooks in parity with `.sh` hooks, Windows
Tailwind hash/download support, and [`docs/windows.md`](docs/windows.md).
+- Google Gemini LLM provider via `AI_MEMORY_LLM_PROVIDER=gemini`, with
+ `gemini-2.5-flash` as the default hosted Google model and `GEMINI_API_KEY`
+ / `GOOGLE_API_KEY` support.
- README support matrix for operating systems, agent integrations, LLM
providers, and embedding providers.
@@ -33,6 +36,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- Hyphenated FTS5 queries such as `ai-memory` are normalized safely instead of
being parsed as column operators.
+- Gemini 2.5 Flash requests disable default dynamic thinking so hidden thought
+ tokens do not consume `maxOutputTokens` and truncate strict JSON responses.
## [0.1.3] - 2026-05-24
diff --git a/README.md b/README.md
index 8a04a1c5..6d422402 100644
--- a/README.md
+++ b/README.md
@@ -30,7 +30,7 @@
| Oh My Pi / OMP | Supported | `pi` / `omp` aliases for MCP config + TypeScript extension. |
| Claude Desktop | MCP-only | Uses `mcp-remote`; no lifecycle hooks. |
| OpenClaw | MCP-only | OpenClaw has MCP/webhook ingress but no lifecycle hook surface. |
-| LLM providers | Supported | Anthropic, OpenAI, and OpenAI-compatible endpoints. |
+| LLM providers | Supported | Anthropic, OpenAI, Gemini, and OpenAI-compatible endpoints. |
| Embedding providers | Supported | OpenAI and Voyage. |
## What it is
diff --git a/crates/ai-memory-llm/src/gemini.rs b/crates/ai-memory-llm/src/gemini.rs
index d82e0227..12732ce2 100644
--- a/crates/ai-memory-llm/src/gemini.rs
+++ b/crates/ai-memory-llm/src/gemini.rs
@@ -87,6 +87,8 @@ struct GeminiSystem<'a> {
struct GeminiGenerationConfig {
#[serde(rename = "maxOutputTokens")]
max_output_tokens: u32,
+ #[serde(rename = "thinkingConfig", skip_serializing_if = "Option::is_none")]
+ thinking_config: Option<GeminiThinkingConfig>,
#[serde(skip_serializing_if = "Option::is_none")]
temperature: Option<f32>,
#[serde(rename = "responseMimeType", skip_serializing_if = "Option::is_none")]
@@ -95,6 +97,12 @@ struct GeminiGenerationConfig {
response_schema: Option<serde_json::Value>,
}
+#[derive(Debug, Serialize)]
+struct GeminiThinkingConfig {
+ #[serde(rename = "thinkingBudget")]
+ thinking_budget: i32,
+}
+
#[derive(Debug, Deserialize)]
struct GeminiResponse {
#[serde(default)]
@@ -189,6 +197,7 @@ impl GeminiProvider {
system_instruction,
generation_config: GeminiGenerationConfig {
max_output_tokens: request.max_tokens,
+ thinking_config: default_thinking_config_for(&self.model),
temperature: request.temperature,
response_mime_type,
response_schema,
@@ -236,6 +245,14 @@ impl GeminiProvider {
}
}
+fn default_thinking_config_for(model: &str) -> Option<GeminiThinkingConfig> {
+ let model = model.to_ascii_lowercase();
+ if model.contains("gemini-2.5-flash") {
+ return Some(GeminiThinkingConfig { thinking_budget: 0 });
+ }
+ None
+}
+
fn first_text(response: &GeminiResponse) -> Option<String> {
let candidate = response.candidates.first()?;
let content = candidate.content.as_ref()?;
@@ -244,7 +261,11 @@ fn first_text(response: &GeminiResponse) -> Option<String> {
.iter()
.filter_map(|p| p.text.as_deref())
.collect();
- if joined.is_empty() { None } else { Some(joined) }
+ if joined.is_empty() {
+ None
+ } else {
+ Some(joined)
+ }
}
/// Normalise a schemars-generated JSON Schema into the Gemini-supported
@@ -403,7 +424,10 @@ mod tests {
.unwrap();
assert_eq!(item.get("type").unwrap(), "object");
assert!(!item.contains_key("additionalProperties"));
- assert_eq!(item.get("properties").unwrap().get("path").unwrap(), &json!({"type": "string"}));
+ assert_eq!(
+ item.get("properties").unwrap().get("path").unwrap(),
+ &json!({"type": "string"})
+ );
}
#[test]
@@ -435,4 +459,25 @@ mod tests {
let prepared = prepare_schema_for_gemini(schema.clone()).unwrap();
assert_eq!(prepared, schema);
}
+
+ #[test]
+ fn build_request_disables_default_thinking_for_25_flash() {
+ let provider =
+ GeminiProvider::new(SecretString::from("test-key"), "gemini-2.5-flash").unwrap();
+ let request = ChatRequest::user_prompt("emit json");
+ let body = serde_json::to_value(provider.build_request(&request, None)).unwrap();
+ assert_eq!(
+ body.pointer("/generationConfig/thinkingConfig/thinkingBudget"),
+ Some(&json!(0))
+ );
+ }
+
+ #[test]
+ fn build_request_omits_thinking_config_for_non_flash_models() {
+ let provider =
+ GeminiProvider::new(SecretString::from("test-key"), "gemini-2.5-pro").unwrap();
+ let request = ChatRequest::user_prompt("emit json");
+ let body = serde_json::to_value(provider.build_request(&request, None)).unwrap();
+ assert!(body.pointer("/generationConfig/thinkingConfig").is_none());
+ }
}
diff --git a/docs/install.md b/docs/install.md
index 95ce7e63..4e007b17 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -297,7 +297,7 @@ If you set only the provider, ai-memory picks a sensible default:
|---|---|---|
| `AI_MEMORY_LLM_PROVIDER=anthropic` | `claude-haiku-4-5` | **Recommended default.** Best balance of speed, restraint, and classification quality. Not a reasoning model. Consistently classifies durable project rules as `kind: rule`. |
| `AI_MEMORY_LLM_PROVIDER=openai` | `gpt-5.4-mini` | Cheaper + faster alternative. Same parse reliability; mild over-classification on thin sessions. |
-| `AI_MEMORY_LLM_PROVIDER=gemini` | `gemini-2.5-flash` | Google's hosted option with a generous free tier — useful if you don't have an Anthropic / OpenAI key handy. Set `GEMINI_API_KEY` (or `GOOGLE_API_KEY`). |
+| `AI_MEMORY_LLM_PROVIDER=gemini` | `gemini-2.5-flash` | Google's hosted option with a generous free tier. ai-memory disables Gemini 2.5 Flash's default dynamic thinking so hidden thought tokens do not truncate strict JSON. Set `GEMINI_API_KEY` (or `GOOGLE_API_KEY`). |
| `AI_MEMORY_EMBEDDING_PROVIDER=openai` | `text-embedding-3-small` (1536-dim) | 5× cheaper than `-3-large` with marginal recall loss. |
| `AI_MEMORY_EMBEDDING_PROVIDER=voyage` | `voyage-3` (1024-dim) | Voyage's current general-purpose recommendation. |