Skip to content

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

fix(platform): serve memory calls in scripted journeys (#15948)

Nicholas Tindleddc53b78c977
diff --git a/autogpt_platform/docker-compose.journeys.yml b/autogpt_platform/docker-compose.journeys.yml
index 73c738f332..5d2e2a8135 100644
--- a/autogpt_platform/docker-compose.journeys.yml
+++ b/autogpt_platform/docker-compose.journeys.yml
@@ -20,9 +20,10 @@ x-scripted-model-env: &scripted-model-env
CHAT_USE_CLAUDE_CODE_SUBSCRIPTION: "false"
CHAT_BASE_URL: http://scripted_model:4010/api/v1
CHAT_API_KEY: "scripted-model-accepts-any-key" # pragma: allowlist secret
- # Memory would otherwise send that key to openrouter.ai and api.openai.com.
- # Kept inside the stack, its calls fail
- # fast and the turn goes on without memory, as it does on any failure.
+ # Keep memory inside the stack. The fake returns schema-valid empty
+ # extractions and deterministic embeddings so journeys exercise memory
+ # without inventing facts or sending test conversations to an external
+ # provider.
GRAPHITI_LLM_BASE_URL: http://scripted_model:4010/api/v1
GRAPHITI_EMBEDDER_BASE_URL: http://scripted_model:4010/api/v1
GRAPHITI_EMBEDDER_API_KEY: "scripted-model-accepts-any-key" # pragma: allowlist secret
diff --git a/autogpt_platform/scripted_model/README.md b/autogpt_platform/scripted_model/README.md
index f6cfe2affe..1dda304e26 100644
--- a/autogpt_platform/scripted_model/README.md
+++ b/autogpt_platform/scripted_model/README.md
@@ -10,15 +10,17 @@ and local test stacks. Self-hosted and hosted deployments never start it.
## How the platform reaches it
-`docker-compose.journeys.yml` puts the backend in OpenRouter mode with
-`CHAT_BASE_URL=http://scripted_model:4010/api/v1`, so both engines and every
-side call come here:
+`docker-compose.journeys.yml` points the chat, memory LLM and memory embedder
+base URLs at `http://scripted_model:4010/api/v1`, so both engines and their
+side calls come here:
| Caller | Request | Dialect |
| --- | --- | --- |
| SDK engine (the Claude Agent SDK's CLI, the default) | `POST /api/v1/messages` | Anthropic Messages, streamed |
| Baseline engine (`FORCE_FLAG_COPILOT_SDK=false`) | `POST /api/v1/chat/completions` | OpenAI chat completions, streamed |
| Side calls (session title) | `POST /api/v1/chat/completions` | non-streamed, no tools |
+| Memory extraction | `POST /api/v1/responses` | OpenAI Responses, structured output |
+| Memory search | `POST /api/v1/embeddings` | OpenAI Embeddings |
The CLI also sends `HEAD /api/api/hello` to check it can reach the server,
and may ask `POST /api/v1/messages/count_tokens`; both are answered.
@@ -84,6 +86,14 @@ round or retry), or a tool the platform didn't offer. A journey fails loudly
instead of waiting on a reply that never comes. A request that offers no tools
is a side call and gets the script's `title`.
+Memory extraction requests get a minimal value for the object, array and scalar
+schema forms Graphiti requests. Lists are empty and scalar fields use neutral
+values, so the memory ingestion pipeline runs successfully without making up
+facts about the test conversation. These calls are journaled as
+`openai-responses` side calls. Memory searches get deterministic, nonzero
+1536-dimensional vectors like `text-embedding-3-small`; Graphiti consumes its
+configured 1024 dimensions. They are journaled as `openai-embeddings`.
+
## Cost
The server keeps the provider's invoice, priced in `invoice.py` from
@@ -95,16 +105,18 @@ Anthropic's list prices (the same as `backend/copilot/anthropic_rates.json`):
Message ids are `msg_…`, not OpenRouter's `gen-…`, so the platform doesn't
try to look the cost up on openrouter.ai.
-Each journal entry records `usage` and `cost_usd`, so a journey can check that
-the platform recorded exactly what the provider billed.
+Each scripted chat-reply entry records `usage` and `cost_usd`, so a journey can
+check that the platform recorded exactly what the provider billed.
## Journal
`GET http://localhost:4010/__scripted__/journal?nonce=<nonce>` (or
-`?script=<name>`) lists every request: dialect, model, which turn answered it,
-the tool results it carried, expectation checks, usage, cost, and whether the
-whole reply reached the platform (`completed`). Refused requests are there
-with their reason. The container log has one line per request.
+`?script=<name>`) lists a scripted run's chat requests: dialect, model, which
+turn answered it, tool results, expectation checks, usage, cost, and whether
+the whole reply reached the platform (`completed`). Refused scripted turns are
+there with their reason. Unfiltered reads also include memory side calls and
+unexpected requests; memory calls deliberately have no script or nonce, usage,
+or billed cost. The container log has one line per request.
## Run it
diff --git a/autogpt_platform/scripted_model/scripted_model/embeddings.py b/autogpt_platform/scripted_model/scripted_model/embeddings.py
new file mode 100644
index 0000000000..99f6be74c5
--- /dev/null
+++ b/autogpt_platform/scripted_model/scripted_model/embeddings.py
@@ -0,0 +1,69 @@
+"""OpenAI Embeddings payloads for deterministic memory searches."""
+
+import json
+from typing import Any
+
+EMBEDDING_DIMENSIONS = 1536
+
+
+class EmbeddingError(ValueError):
+ pass
+
+
+def response(body: dict[str, Any]) -> dict[str, Any]:
+ model = body.get("model")
+ if not isinstance(model, str) or not model:
+ raise EmbeddingError("model must be a non-empty string")
+
+ count = _input_count(body.get("input"))
+ dimensions = body.get("dimensions", EMBEDDING_DIMENSIONS)
+ if (
+ not isinstance(dimensions, int)
+ or isinstance(dimensions, bool)
+ or dimensions < 1
+ ):
+ raise EmbeddingError("dimensions must be a positive integer")
+
+ tokens = max(1, len(json.dumps(body.get("input"))) // 4)
+ vector = [0.1] * dimensions
+ return {
+ "object": "list",
+ "data": [
+ {"object": "embedding", "embedding": vector, "index": index}
+ for index in range(count)
+ ],
+ "model": model,
+ "usage": {"prompt_tokens": tokens, "total_tokens": tokens},
+ }
+
+
+def error(message_text: str) -> dict[str, Any]:
+ return {
+ "error": {
+ "message": f"scripted-model: {message_text}",
+ "type": "invalid_request_error",
+ "param": None,
+ "code": "scripted_model",
+ }
+ }
+
+
+def _input_count(value: Any) -> int:
+ if isinstance(value, str):
+ return 1
+ if not isinstance(value, list) or not value:
+ raise EmbeddingError("input must be a string or a non-empty array")
+ if all(isinstance(item, str) for item in value):
+ return len(value)
+ if all(isinstance(item, int) and not isinstance(item, bool) for item in value):
+ return 1
+ if all(
+ isinstance(item, list)
+ and item
+ and all(
+ isinstance(token, int) and not isinstance(token, bool) for token in item
+ )
+ for item in value
+ ):
+ return len(value)
+ raise EmbeddingError("input array has unsupported values")
diff --git a/autogpt_platform/scripted_model/scripted_model/journal.py b/autogpt_platform/scripted_model/scripted_model/journal.py
index fd63d2994e..b83e75bddc 100644
--- a/autogpt_platform/scripted_model/scripted_model/journal.py
+++ b/autogpt_platform/scripted_model/scripted_model/journal.py
@@ -63,6 +63,30 @@ class Journal:
self._entries.append(entry)
return entry
+ def record_side_call(
+ self, request: Request, model: str, text: str
+ ) -> dict[str, Any]:
+ with self._lock:
+ entry = {
+ "id": self._take_id(),
+ "at": time.time(),
+ "dialect": request.dialect,
+ "path": request.path,
+ "model": model,
+ "script": None,
+ "nonce": None,
+ "tool_results": [],
+ "kind": "side_call",
+ "turn": None,
+ "turns": None,
+ "text": text,
+ "tool_calls": [],
+ "checks": [],
+ "completed": None,
+ }
+ self._entries.append(entry)
+ return entry
+
def finish(self, entry: dict[str, Any], completed: bool) -> None:
"""Mark whether the whole reply reached the platform."""
with self._lock:
diff --git a/autogpt_platform/scripted_model/scripted_model/responses.py b/autogpt_platform/scripted_model/scripted_model/responses.py
new file mode 100644
index 0000000000..4bc9b6f2f1
--- /dev/null
+++ b/autogpt_platform/scripted_model/scripted_model/responses.py
@@ -0,0 +1,339 @@
+"""OpenAI Responses payloads for structured-output side calls."""
+
+import json
+import math
+import time
+import uuid
+from fractions import Fraction
+from typing import Any
+
+
+class ResponseError(ValueError):
+ pass
+
+
+def response(body: dict[str, Any]) -> tuple[dict[str, Any], str]:
+ model = body.get("model")
+ if not isinstance(model, str) or not model:
+ raise ResponseError("model must be a non-empty string")
+
+ text = body.get("text")
+ response_format = text.get("format") if isinstance(text, dict) else None
+ schema = (
+ response_format.get("schema") if isinstance(response_format, dict) else None
+ )
+ if not isinstance(schema, dict):
+ raise ResponseError("text.format.schema must be a JSON object")
+
+ output_text = json.dumps(_schema_value(schema, schema), separators=(",", ":"))
+ input_tokens = max(1, len(json.dumps(body)) // 4)
+ output_tokens = max(1, len(output_text) // 4)
+ response_id = f"resp_{uuid.uuid4().hex}"
+
+ return (
+ {
+ "id": response_id,
+ "created_at": int(time.time()),
+ "model": model,
+ "object": "response",
+ "output": [
+ {
+ "id": f"msg_{uuid.uuid4().hex}",
+ "content": [
+ {
+ "annotations": [],
+ "text": output_text,
+ "type": "output_text",
+ }
+ ],
+ "role": "assistant",
+ "status": "completed",
+ "type": "message",
+ }
+ ],
+ "parallel_tool_calls": True,
+ "status": "completed",
+ "tool_choice": "auto",
+ "tools": [],
+ "usage": {
+ "input_tokens": input_tokens,
+ "input_tokens_details": {"cached_tokens": 0},
+ "output_tokens": output_tokens,
+ "output_tokens_details": {"reasoning_tokens": 0},
+ "total_tokens": input_tokens + output_tokens,
+ },
+ },
+ output_text,
+ )
+
+
+def error(message_text: str) -> dict[str, Any]:
+ return {
+ "error": {
+ "message": f"scripted-model: {message_text}",
+ "type": "invalid_request_error",
+ "param": None,
+ "code": "scripted_model",
+ }
+ }
+
+
+def _schema_value(schema: dict[str, Any], root: dict[str, Any], depth: int = 0) -> Any:
+ if depth > 50:
+ raise ResponseError("JSON schema is too deeply nested")
+
+ reference = schema.get("$ref")
+ if isinstance(reference, str):
+ resolved = _resolve_reference(reference, root)
+ schema = {
+ **resolved,
+ **{key: value for key, value in schema.items() if key != "$ref"},
+ }
+
+ if "const" in schema:
+ return schema["const"]
+ enum = schema.get("enum")
+ if isinstance(enum, list):
+ if not enum:
+ raise ResponseError("JSON schema contains an empty enum")
+ return enum[0]
+ if "default" in schema:
+ return schema["default"]
+
+ for union in ("anyOf", "oneOf"):
+ options = schema.get(union)
+ if isinstance(options, list) and options:
+ option = options[0]
+ if not isinstance(option, dict):
+ raise ResponseError(f"{union} entries must be JSON objects")
+ return _schema_value(option, root, depth + 1)
+
+ all_of = schema.get("allOf")
+ if isinstance(all_of, list) and all_of:
+ values = [
+ _schema_value(option, root, depth + 1)
+ for option in all_of
+ if isinstance(option, dict)
+ ]
+ if values and all(isinstance(value, dict) for value in values):
+ return {key: value for item in values for key, value in item.items()}
+ if values:
+ return values[0]
+
+ schema_type = schema.get("type")
+ if isinstance(schema_type, list):
+ schema_type = next((item for item in schema_type if item != "null"), "null")
+ if schema_type is None:
+ if "properties" in schema:
+ schema_type = "object"
+ elif "items" in schema:
+ schema_type = "array"
+
+ if schema_type == "object":
+ properties = schema.get("properties", {})
+ required = schema.get("required", [])
+ if not isinstance(properties, dict) or not isinstance(required, list):
+ raise ResponseError("object schema properties and required must be valid")
+ result = {}
+ for key in required:
+ if not isinstance(key, str):
+ raise ResponseError(f"required property {key!r} has no schema")
+ property_schema = properties.get(key)
+ if not isinstance(property_schema, dict):
+ raise ResponseError(f"required property {key!r} has no schema")
+ result[key] = _schema_value(property_schema, root, depth + 1)
+ return result
+ if schema_type == "array":
+ items = schema.get("items", {})
+ if not isinstance(items, dict):
+ raise ResponseError("array items must be a JSON object")
+ minimum = schema.get("minItems", 0)
+ count = minimum if isinstance(minimum, int) and minimum > 0 else 0
+ return [_schema_value(items, root, depth + 1) for _ in range(count)]
+ if schema_type == "string":
+ minimum = schema.get("minLength", 0)
+ return "x" * minimum if isinstance(minimum, int) and minimum > 0 else ""
+ if schema_type == "integer":
+ return _bounded_number(schema, integer=True)
+ if schema_type == "number":
+ return _bounded_number(schema, integer=False)
+ if schema_type == "boolean":
+ return False
+ if schema_type == "null" or schema_type is None:
+ return None
+ raise ResponseError(f"unsupported JSON schema type {schema_type!r}")
+
+
+def _bounded_number(schema: dict[str, Any], *, integer: bool) -> int | float:
+ minimum, minimum_exclusive = _lower_bound(schema)
+ maximum, maximum_exclusive = _upper_bound(schema)
+ multiple = _numeric_constraint(schema, "multipleOf")
+
+ if multiple is not None and multiple <= 0:
+ raise ResponseError("numeric schema multipleOf must be positive")
+
+ if integer:
+ if minimum is not None:
+ minimum = Fraction(
+ _floor_fraction(minimum) + 1
+ if minimum_exclusive
+ else _ceil_fraction(minimum)
+ )
+ minimum_exclusive = False
+ if maximum is not None:
+ maximum = Fraction(
+ _ceil_fraction(maximum) - 1
+ if maximum_exclusive
+ else _floor_fraction(maximum)
+ )
+ maximum_exclusive = False
+ if multiple is not None:
+ multiple = Fraction(multiple.numerator)
+
+ if (
+ minimum is not None
+ and maximum is not None
+ and (
+ minimum > maximum
+ or (minimum == maximum and (minimum_exclusive or maximum_exclusive))
+ )
+ ):
+ raise ResponseError("numeric schema has no value within its bounds")
+
+ if multiple is not None:
+ minimum_index = (
+ _floor_fraction(minimum / multiple) + 1
+ if minimum is not None and minimum_exclusive
+ else _ceil_fraction(minimum / multiple) if minimum is not None else None
+ )
+ maximum_index = (
+ _ceil_fraction(maximum / multiple) - 1
+ if maximum is not None and maximum_exclusive
+ else _floor_fraction(maximum / multiple) if maximum is not None else None
+ )
+ if (
+ minimum_index is not None
+ and maximum_index is not None
+ and minimum_index > maximum_index
+ ):
+ raise ResponseError(
+ "numeric schema has no value satisfying its constraints"
+ )
+ index = 0
+ if minimum_index is not None and index < minimum_index:
+ index = minimum_index
+ if maximum_index is not None and index > maximum_index:
+ index = maximum_index
+ value = Fraction(index) * multiple
+ else:
+ value = _unscaled_value(
+ minimum,
+ minimum_exclusive,
+ maximum,
+ maximum_exclusive,
+ )
+
+ if not _within_bounds(
+ value,
+ minimum,
+ minimum_exclusive,
+ maximum,
+ maximum_exclusive,
+ ):
+ raise ResponseError("numeric schema has no value satisfying its constraints")
+ if integer or value.denominator == 1:
+ return value.numerator
+ try:
+ return float(value)
+ except OverflowError as error:
+ raise ResponseError("numeric schema value is too large") from error
+
+
+def _lower_bound(schema: dict[str, Any]) -> tuple[Fraction | None, bool]:
+ minimum = _numeric_constraint(schema, "minimum")
+ exclusive = _numeric_constraint(schema, "exclusiveMinimum")
+ if exclusive is not None and (minimum is None or exclusive >= minimum):
+ return exclusive, True
+ return minimum, False
+
+
+def _upper_bound(schema: dict[str, Any]) -> tuple[Fraction | None, bool]:
+ maximum = _numeric_constraint(schema, "maximum")
+ exclusive = _numeric_constraint(schema, "exclusiveMaximum")
+ if exclusive is not None and (maximum is None or exclusive <= maximum):
+ return exclusive, True
+ return maximum, False
+
+
+def _unscaled_value(
+ minimum: Fraction | None,
+ minimum_exclusive: bool,
+ maximum: Fraction | None,
+ maximum_exclusive: bool,
+) -> Fraction:
+ zero = Fraction(0)
+ if _within_bounds(
+ zero,
+ minimum,
+ minimum_exclusive,
+ maximum,
+ maximum_exclusive,
+ ):
+ return zero
+ if minimum is not None and minimum >= zero:
+ if not minimum_exclusive:
+ return minimum
+ return minimum + 1 if maximum is None else (minimum + maximum) / 2
+ if maximum is None:
+ raise ResponseError("numeric schema has no value within its bounds")
+ if not maximum_exclusive:
+ return maximum
+ return maximum - 1 if minimum is None else (minimum + maximum) / 2
+
+
+def _within_bounds(
+ value: Fraction,
+ minimum: Fraction | None,
+ minimum_exclusive: bool,
+ maximum: Fraction | None,
+ maximum_exclusive: bool,
+) -> bool:
+ if minimum is not None and (
+ value < minimum or (minimum_exclusive and value == minimum)
+ ):
+ return False
+ return not (
+ maximum is not None
+ and (value > maximum or (maximum_exclusive and value == maximum))
+ )
+
+
+def _numeric_constraint(schema: dict[str, Any], key: str) -> Fraction | None:
+ value = schema.get(key)
+ if isinstance(value, bool) or not isinstance(value, (int, float)):
+ return None
+ if isinstance(value, float) and not math.isfinite(value):
+ raise ResponseError(f"numeric schema {key} must be finite")
+ return Fraction(str(value))
+
+
+def _ceil_fraction(value: Fraction) -> int:
+ return -((-value.numerator) // value.denominator)
+
+
+def _floor_fraction(value: Fraction) -> int:
+ return value.numerator // value.denominator
+
+
+def _resolve_reference(reference: str, root: dict[str, Any]) -> dict[str, Any]:
+ if not reference.startswith("#/"):
+ raise ResponseError(f"unsupported JSON schema reference {reference!r}")
+ value: Any = root
+ for part in reference[2:].split("/"):
+ key = part.replace("~1", "/").replace("~0", "~")
+ if not isinstance(value, dict) or key not in value:
+ raise ResponseError(f"unresolved JSON schema reference {reference!r}")
+ value = value[key]
+ if not isinstance(value, dict):
+ raise ResponseError(f"JSON schema reference {reference!r} is not an object")
+ return value
diff --git a/autogpt_platform/scripted_model/scripted_model/server.py b/autogpt_platform/scripted_model/scripted_model/server.py
index 586ecbf2e9..13680047cc 100644
--- a/autogpt_platform/scripted_model/scripted_model/server.py
+++ b/autogpt_platform/scripted_model/scripted_model/server.py
@@ -11,7 +11,7 @@ from pathlib import Path
from typing import Any, cast
from urllib.parse import parse_qs, urlparse
-from . import chat_completions, messages
+from . import chat_completions, embeddings, messages, responses
from .journal import Journal, Request
from .planner import Conversation, Plan, Reply, plan
from .scripts import ScriptError
@@ -95,9 +95,63 @@ class Handler(BaseHTTPRequestHandler):
self._answer(ANTHROPIC)
elif path.endswith("/chat/completions"):
self._answer(OPENAI)
+ elif path.endswith("/embeddings"):
+ self._embedding_response()
+ elif path.endswith("/responses"):
+ self._structured_response()
else:
self._unexpected()
+ def _embedding_response(self) -> None:
+ body = self._read_json()
+ if not isinstance(body, dict):
+ logger.error(
+ "unreadable openai-embeddings request: body is not a JSON object"
+ )
+ self._send_json(
+ HTTPStatus.BAD_REQUEST, embeddings.error("body is not a JSON object")
+ )
+ return
+ try:
+ payload = embeddings.response(body)
+ except embeddings.EmbeddingError as error:
+ logger.error("unreadable openai-embeddings request: %s", error)
+ self._send_json(HTTPStatus.BAD_REQUEST, embeddings.error(str(error)))
+ return
+
+ model = str(body["model"])
+ entry = self.model_server.journal.record_side_call(
+ Request("openai-embeddings", self.path), model, ""
+ )
+ logger.info("openai-embeddings side call (%s)", model)
+ self._send_json(HTTPStatus.OK, payload)
+ self.model_server.journal.finish(entry, completed=True)
+
+ def _structured_response(self) -> None:
+ body = self._read_json()
+ if not isinstance(body, dict):
+ logger.error(
+ "unreadable openai-responses request: body is not a JSON object"
+ )
+ self._send_json(
+ HTTPStatus.BAD_REQUEST, responses.error("body is not a JSON object")
+ )
+ return
+ try:
+ payload, output_text = responses.response(body)
+ except responses.ResponseError as error:
+ logger.error("unreadable openai-responses request: %s", error)
+ self._send_json(HTTPStatus.BAD_REQUEST, responses.error(str(error)))
+ return
+
+ model = str(body["model"])
+ entry = self.model_server.journal.record_side_call(
+ Request("openai-responses", self.path), model, output_text
+ )
+ logger.info("openai-responses side call (%s)", model)
+ self._send_json(HTTPStatus.OK, payload)
+ self.model_server.journal.finish(entry, completed=True)
+
def _answer(self, dialect: Dialect) -> None:
body = self._read_json()
if not isinstance(body, dict):
diff --git a/autogpt_platform/scripted_model/tests/test_scripted_model.py b/autogpt_platform/scripted_model/tests/test_scripted_model.py
index 2a0c40c9b4..c14c25fc27 100644
--- a/autogpt_platform/scripted_model/tests/test_scripted_model.py
+++ b/autogpt_platform/scripted_model/tests/test_scripted_model.py
@@ -5,19 +5,32 @@ does, and the OpenAI client is the baseline engine's own. A script that round
trips through both is one the copilot can follow.
"""
+import asyncio
import json
+import logging
import re
import threading
import urllib.error
import urllib.request
from pathlib import Path
+from typing import Literal
import anthropic
import openai
import pytest
-
+from graphiti_core.embedder import OpenAIEmbedder, OpenAIEmbedderConfig
+from graphiti_core.embedder.client import EMBEDDING_DIM
+from graphiti_core.llm_client.config import LLMConfig
+from graphiti_core.prompts.extract_edges import ExtractedEdges
+from graphiti_core.prompts.extract_nodes import ExtractedEntities
+from graphiti_core.prompts.models import Message
+from pydantic import BaseModel, Field
+from scripted_model.responses import ResponseError
+from scripted_model.responses import response as structured_response
from scripted_model.scripts import parse_script
+from backend.copilot.graphiti.llm_client import GraphitiOpenAIClient
+
JOURNEY_SCRIPTS = (
Path(__file__).resolve().parents[2]
/ "frontend/src/playwright/journeys/model-scripts"
@@ -61,6 +74,37 @@ CHAT_TOOLS = [
]
+class StructuredSummary(BaseModel):
+ label: str
+ count: int = Field(ge=1)
+
+
+class StructuredExtraction(BaseModel):
+ entities: list[str]
+ summary: StructuredSummary
+ accepted: bool
+
+
+class ConstrainedExtraction(BaseModel):
+ kind: Literal["fixed"]
+ status: Literal["ready", "waiting"]
+ labels: list[str] = Field(min_length=1)
+ code: str = Field(min_length=2)
+ score: int | None
+ ratio: float = Field(ge=0.25)
+
+
+class MaximumConstrainedExtraction(BaseModel):
+ integer: int = Field(le=-1, multiple_of=2)
+ number: float = Field(le=-0.25, multiple_of=0.5)
+
+
+class ExclusiveConstrainedExtraction(BaseModel):
+ integer: int = Field(gt=0, lt=3, multiple_of=2)
+ number: float = Field(gt=0.25, lt=0.75, multiple_of=0.25)
+ negative: int = Field(lt=-1, multiple_of=2)
+
+
def test_messages_follow_the_script_through_a_tool_round(scripted_model):
scripted_model.add_script("lookup", LOOKUP)
client = _anthropic(scripted_model)
@@ -164,6 +208,230 @@ def test_chat_completions_follow_the_script_and_bill_like_openrouter(
assert scripted_model.journal(nonce="chat-run")[-1]["checks"][0]["ok"]
+def test_responses_parse_returns_schema_valid_structured_output(scripted_model, caplog):
+ caplog.set_level(logging.INFO, logger="scripted_model")
+ response = _openai(scripted_model).responses.parse(
+ model="gpt-6-luna",
+ input=[
+ {
+ "role": "user",
+ "content": "Extract entities. [[script:lookup#memory-side-call]]",
+ }
+ ],
+ text_format=StructuredExtraction,
+ reasoning={"effort": "none"},
+ )
+
+ content = response.output[0].content[0]
+ assert content.parsed == StructuredExtraction(
+ entities=[],
+ summary=StructuredSummary(label="", count=1),
+ accepted=False,
+ )
+ assert json.loads(response.output_text) == content.parsed.model_dump()
+ assert response.usage is not None
+ assert response.usage.total_tokens > 0
+
+ (entry,) = scripted_model.journal()
+ assert entry["kind"] == "side_call"
+ assert entry["dialect"] == "openai-responses"
+ assert entry["completed"]
+ assert scripted_model.journal(nonce="memory-side-call") == []
+ assert "openai-responses side call (gpt-6-luna)" in caplog.messages
+
+
+def test_responses_parse_honors_supported_schema_constraints(scripted_model):
+ response = _openai(scripted_model).responses.parse(
+ model="gpt-6-luna",
+ input="Extract constrained values.",
+ text_format=ConstrainedExtraction,
+ )
+
+ assert response.output_parsed == ConstrainedExtraction(
+ kind="fixed",
+ status="ready",
+ labels=[""],
+ code="xx",
+ score=0,
+ ratio=0.25,
+ )
+
+
+def test_responses_parse_honors_numeric_constraints(scripted_model):
+ response = _openai(scripted_model).responses.parse(
+ model="gpt-6-luna",
+ input="Extract upper-bounded values.",
+ text_format=MaximumConstrainedExtraction,
+ )
+
+ assert response.output_parsed == MaximumConstrainedExtraction(
+ integer=-2,
+ number=-0.5,
+ )
+
+
+def test_responses_parse_honors_exclusive_numeric_constraints(scripted_model):
+ response = _openai(scripted_model).responses.parse(
+ model="gpt-6-luna",
+ input="Extract exclusively bounded values.",
+ text_format=ExclusiveConstrainedExtraction,
+ )
+
+ assert response.output_parsed == ExclusiveConstrainedExtraction(
+ integer=2,
+ number=0.5,
+ negative=-2,
+ )
+
+
+def test_responses_rejects_unsatisfiable_numeric_constraints():
+ with pytest.raises(ResponseError, match="no value satisfying its constraints"):
+ structured_response(
+ {
+ "model": "gpt-6-luna",
+ "text": {
+ "format": {
+ "schema": {
+ "type": "object",
+ "properties": {
+ "value": {
+ "type": "integer",
+ "minimum": 1,
+ "maximum": 1,
+ "multipleOf": 2,
+ }
+ },
+ "required": ["value"],
+ }
+ }
+ },
+ }
+ )
+
+
+def test_responses_rejects_unsatisfiable_exclusive_numeric_constraints():
+ with pytest.raises(ResponseError, match="no value"):
+ structured_response(
+ {
+ "model": "gpt-6-luna",
+ "text": {
+ "format": {
+ "schema": {
+ "type": "object",
+ "properties": {
+ "value": {
+ "type": "integer",
+ "exclusiveMinimum": 0,
+ "exclusiveMaximum": 1,
+ }
+ },
+ "required": ["value"],
+ }
+ }
+ },
+ }
+ )
+
+
+def test_responses_rejects_a_non_object_body(scripted_model):
+ request = urllib.request.Request(
+ f"{scripted_model.url}/api/v1/responses",
+ data=b"[]",
+ headers={"Content-Type": "application/json"},
+ )
+
+ with pytest.raises(urllib.error.HTTPError) as refused:
+ urllib.request.urlopen(request, timeout=5)
+
+ assert refused.value.code == 400
+ assert "body is not a JSON object" in refused.value.read().decode()
+
+
+def test_responses_rejects_a_non_string_required_key(scripted_model):
+ request = urllib.request.Request(
+ f"{scripted_model.url}/api/v1/responses",
+ data=json.dumps(
+ {
+ "model": "gpt-6-luna",
+ "text": {
+ "format": {
+ "schema": {
+ "type": "object",
+ "properties": {},
+ "required": [[]],
+ }
+ }
+ },
+ }
+ ).encode(),
+ headers={"Content-Type": "application/json"},
+ )
+
+ with pytest.raises(urllib.error.HTTPError) as refused:
+ urllib.request.urlopen(request, timeout=5)
+
+ assert refused.value.code == 400
+ assert "required property [] has no schema" in refused.value.read().decode()
+
+
+def test_responses_preserve_large_numeric_minimums():
+ minimum = 10**400
+ _, output_text = structured_response(
+ {
+ "model": "gpt-6-luna",
+ "text": {
+ "format": {
+ "schema": {
+ "type": "object",
+ "properties": {"value": {"type": "number", "minimum": minimum}},
+ "required": ["value"],
+ }
+ }
+ },
+ }
+ )
+
+ assert json.loads(output_text) == {"value": minimum}
+
+
+def test_graphiti_client_parses_its_real_extraction_schemas(scripted_model):
+ nodes, edges = asyncio.run(_graphiti_extract(scripted_model.url))
+
+ assert nodes == {"extracted_entities": []}
+ assert edges == {"edges": []}
+ assert [entry["dialect"] for entry in scripted_model.journal()] == [
+ "openai-responses",
+ "openai-responses",
+ ]
+
+
+def test_embeddings_return_graphiti_sized_vectors(scripted_model, caplog):
+ caplog.set_level(logging.INFO, logger="scripted_model")
+ response = _openai(scripted_model).embeddings.create(
+ model="text-embedding-3-small",
+ input=["today", "tomorrow"],
+ )
+
+ assert [item.index for item in response.data] == [0, 1]
+ assert all(len(item.embedding) == 1536 for item in response.data)
+ assert all(any(value != 0 for value in item.embedding) for item in response.data)
+
+ graphiti_embedding = asyncio.run(_graphiti_embedding(scripted_model.url))
+ assert len(graphiti_embedding) == EMBEDDING_DIM
+
+ entries = scripted_model.journal()
+ assert [entry["dialect"] for entry in entries] == [
+ "openai-embeddings",
+ "openai-embeddings",
+ ]
+ assert all(entry["kind"] == "side_call" for entry in entries)
+ assert all(entry["completed"] for entry in entries)
+ assert (
+ caplog.messages.count("openai-embeddings side call (text-embedding-3-small)")
+ == 2
+ )
+
+
def test_a_failed_expectation_is_recorded_with_what_the_tool_said(scripted_model):
scripted_model.add_script("lookup", LOOKUP)
messages = [
@@ -375,6 +643,44 @@ def _openai(server) -> openai.OpenAI:
return openai.OpenAI(base_url=f"{server.url}/api/v1", api_key="any", max_retries=0)
+async def _graphiti_extract(base_url: str):
+ client = GraphitiOpenAIClient(
+ config=LLMConfig(
+ api_key="any",
+ model="gpt-6-luna",
+ base_url=f"{base_url}/api/v1",
+ )
+ )
+ try:
+ nodes = await client.generate_response(
+ [Message(role="user", content="Remember nothing from this turn.")],
+ response_model=ExtractedEntities,
+ prompt_name="extract_nodes.extract_message",
+ )
+ edges = await client.generate_response(
+ [Message(role="user", content="Relate nothing from this turn.")],
+ response_model=ExtractedEdges,
+ prompt_name="extract_edges.edge",
+ )
+ return nodes, edges
+ finally:
+ await client.client.close()
+
+
+async def _graphiti_embedding(base_url: str):
+ embedder = OpenAIEmbedder(
+ config=OpenAIEmbedderConfig(
+ api_key="any",
+ embedding_model="text-embedding-3-small",
+ base_url=f"{base_url}/api/v1",
+ )
+ )
+ try:
+ return await embedder.create("today")
+ finally:
+ await embedder.client.close()
+
+
def _stream(client: anthropic.Anthropic, messages: list[dict], system=None):
extra = {"system": system} if system else {}
with client.messages.stream(