Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

evals: cover Anthropic tool input objects

Louis Beaumontb56175c2a772
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.md
index 3bd4c4836..3311c22cf 100644
--- a/evals/coding-agent/README.md
+++ b/evals/coding-agent/README.md
@@ -9,7 +9,7 @@ This is an agent eval suite, not a unit-test suite. Every case contains:
- saved prompt, transcript, candidate patch, grader output, runtime fingerprint, and result;
- repeated-trial reporting with success rate, `pass@k`, and `pass^k`.
-The current app corpus contains 35 git-mined regressions. See
+The current app corpus contains 36 git-mined regressions. See
[DESIGN.md](./DESIGN.md) for the Anthropic guidance, source contract, and
history-mining workflow. The companion website manifest contributes 20 more.
diff --git a/evals/coding-agent/cases.json b/evals/coding-agent/cases.json
index 0071dfbeb..62f1c0fb7 100644
--- a/evals/coding-agent/cases.json
+++ b/evals/coding-agent/cases.json
@@ -1,7 +1,7 @@
{
"schema_version": 1,
"suite": "screenpipe-app-coding-agent-regressions",
- "dataset_version": "2026-09-05.1",
+ "dataset_version": "2026-09-06.1",
"suite_type": "regression",
"owner": "screenpipe engineering",
"policy": {
@@ -1021,6 +1021,42 @@
],
"command": "cd packages/ai-gateway && bun test src/test/openai-param-retry.unit.test.ts --timeout=10000"
}
+ },
+ {
+ "id": "ai-gateway-anthropic-tool-input-object",
+ "name": "Normalize Anthropic tool-use inputs to objects",
+ "source": {
+ "kind": "git_regression",
+ "fix_commit": "b06e1ffa467989bb075540e0d2e86ae36dc03683",
+ "pull_request": "https://github.com/screenpipe/screenpipe/pull/6905"
+ },
+ "base_ref": "1a645b0ba4d0c51e00a2665e871050800ae6fac5",
+ "oracle_ref": "b06e1ffa467989bb075540e0d2e86ae36dc03683",
+ "oracle_paths": [
+ "packages/ai-gateway/src/providers/anthropic.ts"
+ ],
+ "gate": "advisory",
+ "trigger_paths": [
+ "packages/ai-gateway/src/providers/anthropic.ts",
+ "packages/ai-gateway/src/types.ts",
+ "packages/ai-gateway/src/handlers/chat.ts"
+ ],
+ "tags": ["regression", "escaped-failure", "ai-gateway", "anthropic", "tool-use", "schema", "compatibility"],
+ "prompt": "OpenAI-compatible assistant history can contain tool-call arguments that parse as JSON null, an array, a number, or a string. Forwarding those values as Anthropic tool-use input causes the provider to reject the request because tool input must be an object. Normalize every parsed non-object tool input to an empty object before Anthropic dispatch. Preserve valid object arguments unchanged, keep malformed JSON fail-safe, and preserve tool call identifiers and names, tool results, ordinary text and image messages, streaming behavior, and provider routing.",
+ "agent_timeout_seconds": 900,
+ "grader_dependency_links": [
+ { "source_path": "packages/ai-gateway/node_modules", "destination_path": "packages/ai-gateway/node_modules" }
+ ],
+ "grader": {
+ "timeout_seconds": 300,
+ "fixtures": [
+ {
+ "source_ref": "b06e1ffa467989bb075540e0d2e86ae36dc03683",
+ "source_path": "packages/ai-gateway/src/test/anthropic-proxy.unit.test.ts"
+ }
+ ],
+ "command": "cd packages/ai-gateway && bun test src/test/anthropic-proxy.unit.test.ts --timeout=10000"
+ }
}
]
}