Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(evals): use schema-compatible model preflight prompt (#18252)

Tobias Wochingerd29bd6acd644
diff --git a/packages/shared/src/server/llm/testModelCall.ts b/packages/shared/src/server/llm/testModelCall.ts
index adff7165bb..753dcc7be2 100644
--- a/packages/shared/src/server/llm/testModelCall.ts
+++ b/packages/shared/src/server/llm/testModelCall.ts
@@ -43,7 +43,7 @@ export const testModelCall = async ({
{
role: ChatMessageRole.User,
content:
- 'Extract a score (1-5) and reasoning from this text: "This is a test. It worked perfectly because it matched all passing criteria."',
+ "Generate one example response that conforms to the provided JSON schema. Respect the type, allowed values, and bounds of every field. Keep any reasoning brief.",
type: ChatMessageType.User,
},
],