Skip to content

CubeLoop — a Pythonic, async-native agent harness core designed for high performance, readability, and production-grade persistence.

docs: add Deferred Tool Groups guide

xf gongfaa4f43e170f
diff --git a/website/docs/guides/middleware/deferred-tools.md b/website/docs/guides/middleware/deferred-tools.md
new file mode 100644
index 0000000..730a9eb
--- /dev/null
+++ b/website/docs/guides/middleware/deferred-tools.md
@@ -0,0 +1,265 @@
+---
+title: Deferred Tool Groups
+description: "Hide MCP tool schemas from the model by default, expanding them on demand to reduce context bloat."
+---
+
+# Deferred Tool Groups
+
+When an agent connects to many MCP servers, their combined tool schemas
+can consume thousands of tokens of context on every turn — even if the
+model only needs one or two groups for the current task.
+`DeferredToolGroup` solves this by replacing full schemas with a compact
+catalog, letting the model expand groups on demand.
+
+## How it works
+
+1. At construction time, the agent's system prompt includes a short
+ catalog — one line per group with a description and tool list.
+2. The model sees a built-in `expand_tools` tool it can call to load a
+ group (or specific tools within a group).
+3. On expansion, the loader runs once, the tools are injected into the
+ live tool set, and their schemas are appended to the system prompt.
+
+```
+# Deferred tool groups
+
+These tool groups are available but not yet loaded. Call `expand_tools(group_id)`
+to load a group's tools for the rest of this conversation.
+
+- `mcp:github` — GitHub: Issues, PRs, repos, code search (4 tools)
+ create_issue, search_repos, create_pr, list_comments
+- `mcp:linear` — Linear: Project management and issue tracking (6 tools)
+ create_issue, update_issue, list_projects, ...
+```
+
+## Basic setup
+
+Pass `deferred_tool_groups` to `Agent`. The middleware is created
+automatically — no manual wiring needed:
+
+```python
+from cubepi import Agent
+from cubepi.deferred import DeferredToolGroup
+
+github_group = DeferredToolGroup(
+ group_id="mcp:github",
+ display_name="GitHub",
+ description="Issues, PRs, repos, code search",
+ tool_names=["create_issue", "search_repos", "create_pr", "list_comments"],
+ loader=github_mcp.load_tools, # async () -> list[AgentTool]
+)
+
+linear_group = DeferredToolGroup(
+ group_id="mcp:linear",
+ display_name="Linear",
+ description="Project management and issue tracking",
+ tool_names=["create_issue", "update_issue", "list_projects"],
+ loader=linear_mcp.load_tools,
+)
+
+agent = Agent(
+ model=provider.model("claude-sonnet-4-6"),
+ tools=[search_tool, calculator], # always-available tools
+ deferred_tool_groups=[github_group, linear_group],
+)
+```
+
+### `DeferredToolGroup` fields
+
+| Field | Type | Description |
+|---|---|---|
+| `group_id` | `str` | Unique identifier the model uses in `expand_tools` calls (e.g. `"mcp:github"`) |
+| `display_name` | `str` | Human-readable label shown in the catalog |
+| `description` | `str` | One-line summary of the group's capabilities |
+| `tool_names` | `list[str]` | Tool names shown in the catalog |
+| `loader` | `async () -> list[AgentTool]` | Callback that returns the full tool set for this group |
+
+## The `expand_tools` tool
+
+The model calls `expand_tools` to load a group's tools. Two modes:
+
+```
+# Expand everything in the group
+expand_tools(group_id="mcp:github")
+
+# Expand specific tools only
+expand_tools(group_id="mcp:github", tool_names=["create_issue", "search_repos"])
+```
+
+The tool returns a structured result:
+
+```json
+{
+ "group_id": "mcp:github",
+ "expanded": true,
+ "tool_names": ["create_issue", "search_repos", "create_pr", "list_comments"],
+ "remaining": 0
+}
+```
+
+After expansion, the tools are immediately available for the model to
+call in the same turn (via the `after_tool_call` hook).
+
+### Selective expansion
+
+The model can expand a group incrementally — requesting one or two tools
+now and more later:
+
+```
+expand_tools(group_id="mcp:github", tool_names=["create_issue"])
+# → remaining: 3
+
+# later...
+expand_tools(group_id="mcp:github", tool_names=["search_repos"])
+# → remaining: 2
+```
+
+Already-expanded tools are idempotent — re-requesting them is a no-op.
+
+### Loader caching
+
+The `loader` callback is invoked exactly **once per group per run**.
+The first `expand_tools` call triggers it; subsequent selective
+expansions filter from the cached result. If the loader fails, the
+error is returned to the model and the group remains unexpanded.
+
+## Prompt-cache stability
+
+The system prompt is designed for prompt-cache prefix stability:
+
+- **Catalog** is sorted by `group_id` alphabetically — input order
+ doesn't matter, the rendered text is byte-stable.
+- **Expanded schemas** are appended in expansion order (the order the
+ model called `expand_tools`), never reordered. Each new expansion
+ appends to the end, preserving the existing prefix.
+
+This means the LLM API's prompt cache remains valid across turns: the
+system prompt only grows, and only at the end.
+
+## Expansion state
+
+The middleware tracks which groups are expanded in `ctx.extra`:
+
+```python
+ctx.extra["expanded_groups"] = {
+ "mcp:github": None, # fully expanded (None = all tools)
+ "mcp:linear": ["create_issue"], # partially expanded
+ # mcp:slack not present = unexpanded
+}
+```
+
+This state survives checkpointing and can be used for cross-run replay
+(see below).
+
+## Cross-run replay
+
+When resuming a conversation from a previous run, you need to restore
+the expansion state so the model has the same tools available.
+`prepare_resumed_state` handles this:
+
+```python
+from cubepi.deferred import DeferredToolsMiddleware
+
+# saved_extra is the persisted ctx.extra from the previous run
+resumed = await DeferredToolsMiddleware.prepare_resumed_state(
+ groups=all_groups,
+ expanded=saved_extra["expanded_groups"],
+)
+
+agent = Agent(
+ model=model,
+ tools=[*builtin_tools, *resumed.pre_loaded_tools],
+ deferred_tool_groups=resumed.remaining_groups,
+)
+```
+
+`prepare_resumed_state` returns a `ResumedState` with:
+
+| Field | Description |
+|---|---|
+| `pre_loaded_tools` | Tools from previously-expanded groups, ready to use |
+| `remaining_groups` | Groups that were never expanded or only partially expanded |
+| `expanded_schemas` | Schema data for the system prompt (pass to `resumed_schemas` for advanced use) |
+
+Fully expanded groups are loaded and removed from the deferred set.
+Partially expanded groups load the selected tools but stay deferrable
+(the model can still expand the rest).
+
+### Restoring schema text
+
+The `Agent(deferred_tool_groups=...)` shorthand handles the common case.
+For full prompt-cache continuity — where the resumed run's system prompt
+must match the previous run's final state byte-for-byte — construct the
+middleware directly with `resumed_schemas`:
+
+```python
+mw = DeferredToolsMiddleware(
+ groups=resumed.remaining_groups,
+ extra_ref=lambda: agent_extra,
+ resumed_schemas=resumed.expanded_schemas,
+)
+
+agent = Agent(
+ model=model,
+ tools=[*builtin_tools, *resumed.pre_loaded_tools],
+ middleware=[mw],
+)
+```
+
+## Advanced: constructing the middleware directly
+
+For full control over the catalog header, cross-run schema seeding, or
+other middleware options, construct `DeferredToolsMiddleware` yourself:
+
+```python
+from cubepi.deferred import DeferredToolsMiddleware
+
+mw = DeferredToolsMiddleware(
+ groups=[github_group, linear_group],
+ extra_ref=lambda: agent_extra,
+ catalog_header="# Available integrations\n\nExpand with expand_tools().",
+ resumed_schemas=None, # or pass schemas from a previous run
+)
+
+agent = Agent(
+ model=model,
+ tools=[search_tool],
+ middleware=[mw],
+)
+```
+
+### Constructor parameters
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `groups` | `list[DeferredToolGroup]` | required | Groups to defer |
+| `extra_ref` | `() -> dict` | required | Returns the live `ctx.extra` dict |
+| `catalog_header` | `str` | *(built-in)* | Header text for the catalog section |
+| `resumed_schemas` | `list[tuple[str, list[dict]]] \| None` | `None` | Schema data to seed from a previous run |
+
+When using the `Agent(deferred_tool_groups=...)` shorthand, `extra_ref`
+is automatically bound to `self._extra`.
+
+## When to use it
+
+**Good fit:**
+
+- Agent has access to 5+ MCP servers but typically uses 1–2 per conversation.
+- Tool schemas are large (many parameters, long descriptions).
+- You want to keep prompt-cache hit rates high across turns.
+
+**Skip it when:**
+
+- The agent has only a few tools — the overhead of the catalog and
+ `expand_tools` call isn't worth it.
+- All tools are needed on every turn — deferring just adds a round trip.
+- Tool schemas are small — the context savings are minimal.
+
+## See also
+
+- [Loading MCP Tools](../mcp/loading) — how to get `AgentTool` lists from
+ MCP servers.
+- [The 8 Hooks](./hooks) — the middleware hooks that power deferred tools
+ (`transform_system_prompt`, `after_tool_call`).
+- [Composition](./composition) — how middleware composes when stacked with
+ other middleware.
diff --git a/website/sidebars.ts b/website/sidebars.ts
index b83b226..8a89df5 100644
--- a/website/sidebars.ts
+++ b/website/sidebars.ts
@@ -45,6 +45,7 @@ const sidebars: SidebarsConfig = {
'guides/middleware/compaction',
'guides/middleware/subagents',
'guides/middleware/todo',
+ 'guides/middleware/deferred-tools',
]},
{ type: 'category', label: 'Human-in-the-Loop', items: [
'guides/hitl/overview',