Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(llm-test): explain anthropic-oauth's 429 for subscription tokens (#1170)

Gabriel Alencar651e0c63f0bc
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b43a8edc..45fdc0ee 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
+- Fixed `llm-test --provider anthropic-oauth` reporting Anthropic's refusal of a
+ `claude setup-token` token as a plain rate limit: that 429 now carries a hint
+ that it is not a real rate limit and names the alternatives, and
+ `docs/llm-providers.md` documents the behavior. (#1170)
- Fixed Cursor tool calls being stored with no title or content: Cursor's own
hooks, and the Claude Code hooks Cursor also runs, send Claude's
`tool_name`/`tool_input` fields, but Cursor was missing from the tool-capture
diff --git a/crates/ai-memory-cli/src/commands/llm_test.rs b/crates/ai-memory-cli/src/commands/llm_test.rs
index b43330c8..3d1c1b18 100644
--- a/crates/ai-memory-cli/src/commands/llm_test.rs
+++ b/crates/ai-memory-cli/src/commands/llm_test.rs
@@ -1,6 +1,6 @@
//! `ai-memory llm-test` — smoke test an LLM provider end-to-end.
-use ai_memory_llm::{ChatRequest, ProviderChoice, ProviderConfig, build_provider};
+use ai_memory_llm::{ChatRequest, LlmError, ProviderChoice, ProviderConfig, build_provider};
use anyhow::{Context, Result};
use tracing::info;
@@ -48,12 +48,17 @@ pub async fn run(config: &Config, args: LlmTestArgs) -> Result<()> {
}),
)
.await
+ .map_err(|e| with_oauth_hint(provider, e))
.context("calling provider for structured output")?;
println!("--- model: {} ---", client.model());
println!("{}", serde_json::to_string_pretty(&value)?);
return Ok(());
}
- let resp = client.complete(request).await.context("calling provider")?;
+ let resp = client
+ .complete(request)
+ .await
+ .map_err(|e| with_oauth_hint(provider, e))
+ .context("calling provider")?;
println!("--- model: {} ---", resp.model);
if let Some(u) = resp.usage {
@@ -66,6 +71,24 @@ pub async fn run(config: &Config, args: LlmTestArgs) -> Result<()> {
Ok(())
}
+/// Anthropic answers a subscription token it will not serve with a bare 429
+/// that reads like a real rate limit (#1170).
+const ANTHROPIC_OAUTH_429_HINT: &str = "Anthropic often answers `claude setup-token` \
+tokens with this 429 for any client that is not Claude Code; it is not a real \
+rate limit. Use `--provider anthropic` with an API key or an OpenAI-compatible \
+gateway (see docs/llm-providers.md).";
+
+fn with_oauth_hint(provider: ProviderChoice, err: LlmError) -> anyhow::Error {
+ let is_oauth_429 = provider == ProviderChoice::AnthropicOAuth
+ && matches!(err, LlmError::Provider { status: 429, .. });
+ let err = anyhow::Error::new(err);
+ if is_oauth_429 {
+ err.context(ANTHROPIC_OAUTH_429_HINT)
+ } else {
+ err
+ }
+}
+
/// Use the same sampling value as bootstrap and consolidation so this smoke
/// test exercises provider-specific request normalization.
fn representative_request(prompt: String) -> ChatRequest {
@@ -110,6 +133,35 @@ mod tests {
);
}
+ #[test]
+ fn anthropic_oauth_429_carries_the_hint() {
+ let err = with_oauth_hint(
+ ProviderChoice::AnthropicOAuth,
+ LlmError::Provider {
+ status: 429,
+ body: "rate_limit_error".into(),
+ },
+ );
+ assert!(format!("{err:#}").contains("not a real rate limit"));
+ }
+
+ #[test]
+ fn other_providers_and_statuses_carry_no_hint() {
+ for (provider, status) in [
+ (ProviderChoice::Anthropic, 429),
+ (ProviderChoice::AnthropicOAuth, 500),
+ ] {
+ let err = with_oauth_hint(
+ provider,
+ LlmError::Provider {
+ status,
+ body: String::new(),
+ },
+ );
+ assert!(!format!("{err:#}").contains("not a real rate limit"));
+ }
+ }
+
#[test]
fn llm_test_exercises_pipeline_sampling_compatibility() {
let request = representative_request("diagnostic".into());
diff --git a/docs/llm-providers.md b/docs/llm-providers.md
index 21d98e7c..d135f564 100644
--- a/docs/llm-providers.md
+++ b/docs/llm-providers.md
@@ -304,6 +304,13 @@ wrappers forward either token by name to short-lived helper commands such as
`llm-test`; configure the long-lived server container separately as shown in the
installation guide.
+Anthropic may also refuse a subscription token outright: with a `claude
+setup-token` token, every request whose system prompt does not identify itself
+as Claude Code can come back as `429 rate_limit_error` with the body `"Error"`,
+even though the same token works from `claude -p` (#1170). That 429 is not a
+real rate limit, and ai-memory does not impersonate Claude Code to get around
+it. Use `anthropic` with an API key, or an OpenAI-compatible gateway, instead.
+
For both Anthropic providers, ai-memory omits `temperature` for Claude
4.7 and later models and Claude Mythos Preview because those models reject
non-default sampling parameters. `llm-test` sends the same representative 0.2