YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
evals: cover Anthropic tool input objects
Louis Beaumontb56175c2a772
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.mdindex 3bd4c4836..3311c22cf 100644--- a/evals/coding-agent/README.md+++ b/evals/coding-agent/README.md@@ -9,7 +9,7 @@ This is an agent eval suite, not a unit-test suite. Every case contains:- saved prompt, transcript, candidate patch, grader output, runtime fingerprint, and result;- repeated-trial reporting with success rate, `pass@k`, and `pass^k`.-The current app corpus contains 35 git-mined regressions. See+The current app corpus contains 36 git-mined regressions. See[DESIGN.md](./DESIGN.md) for the Anthropic guidance, source contract, andhistory-mining workflow. The companion website manifest contributes 20 more.diff --git a/evals/coding-agent/cases.json b/evals/coding-agent/cases.jsonindex 0071dfbeb..62f1c0fb7 100644--- a/evals/coding-agent/cases.json+++ b/evals/coding-agent/cases.json@@ -1,7 +1,7 @@{"schema_version": 1,"suite": "screenpipe-app-coding-agent-regressions",- "dataset_version": "2026-09-05.1",+ "dataset_version": "2026-09-06.1","suite_type": "regression","owner": "screenpipe engineering","policy": {@@ -1021,6 +1021,42 @@],"command": "cd packages/ai-gateway && bun test src/test/openai-param-retry.unit.test.ts --timeout=10000"}+ },+ {+ "id": "ai-gateway-anthropic-tool-input-object",+ "name": "Normalize Anthropic tool-use inputs to objects",+ "source": {+ "kind": "git_regression",+ "fix_commit": "b06e1ffa467989bb075540e0d2e86ae36dc03683",+ "pull_request": "https://github.com/screenpipe/screenpipe/pull/6905"+ },+ "base_ref": "1a645b0ba4d0c51e00a2665e871050800ae6fac5",+ "oracle_ref": "b06e1ffa467989bb075540e0d2e86ae36dc03683",+ "oracle_paths": [+ "packages/ai-gateway/src/providers/anthropic.ts"+ ],+ "gate": "advisory",+ "trigger_paths": [+ "packages/ai-gateway/src/providers/anthropic.ts",+ "packages/ai-gateway/src/types.ts",+ "packages/ai-gateway/src/handlers/chat.ts"+ ],+ "tags": ["regression", "escaped-failure", "ai-gateway", "anthropic", "tool-use", "schema", "compatibility"],+ "prompt": "OpenAI-compatible assistant history can contain tool-call arguments that parse as JSON null, an array, a number, or a string. Forwarding those values as Anthropic tool-use input causes the provider to reject the request because tool input must be an object. Normalize every parsed non-object tool input to an empty object before Anthropic dispatch. Preserve valid object arguments unchanged, keep malformed JSON fail-safe, and preserve tool call identifiers and names, tool results, ordinary text and image messages, streaming behavior, and provider routing.",+ "agent_timeout_seconds": 900,+ "grader_dependency_links": [+ { "source_path": "packages/ai-gateway/node_modules", "destination_path": "packages/ai-gateway/node_modules" }+ ],+ "grader": {+ "timeout_seconds": 300,+ "fixtures": [+ {+ "source_ref": "b06e1ffa467989bb075540e0d2e86ae36dc03683",+ "source_path": "packages/ai-gateway/src/test/anthropic-proxy.unit.test.ts"+ }+ ],+ "command": "cd packages/ai-gateway && bun test src/test/anthropic-proxy.unit.test.ts --timeout=10000"+ }}]}