Skip to content

Self-improving AI Employees for everyone

fix(connectors): a server's $ref never makes the backend fetch a URL

trohitga82af6765282
diff --git a/CLAUDE.md b/CLAUDE.md
index 689a3133..00b74c8e 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -33,7 +33,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[Email Service](./docs-internal/email_service.md)** | IMAP/SMTP integration via Himalaya CLI with EmailService orchestrator, provider presets, custom credential fallback, and polling triggers |
| **[Browser](./docs-internal/browser.md)** | Current `browser` node: installed Chrome/Edge/Chromium in a dedicated profile rendered inside the workspace by default (`BROWSER_RUNTIME=system`, `BROWSER_HEADLESS=true`) and pinned browser-use CLI. Chrome for Testing requires explicit `BROWSER_RUNTIME=testing`. System mode retains the native user agent and OS password-store/keychain integration; basic/mock keychain flags are restricted to testing mode. Includes persistent employee/shared profiles (the Credentials / Connectors "Web browser" panel adds, deletes and imports logins from a session file; the node's profile dropdown lists only the caller's profiles, via `current_load_options_principal()`), session leases, agent/user handoff, network policy, screenshots as workspace FileRefs and authenticated `/ws/browser` live viewing. Plugin-owned installation is lazy (`_install_chrome.py`, `_install_bu.py`); importing or shutting down the plugin never installs it. Replaces the former agent-browser and separate `browserHarness` implementations. |
| **[Browser Workspace](./docs-internal/browser_workspace.md)** | Home and Dev share `WorkspaceTabs` and `BrowserWorkspace`: Browser / Canvas / Mobile, capability-based discovery, binary JPEG frames, bounded frame credit and decoding, Take control / Hand back, the agent's cursor (`agent_action`: where each click, hover, type or select happened, beside the agent's result and never during a protected login), visibility cleanup and payload-free opt-in diagnostics. The Browser node's parameter panel shows the same viewer, and the Dev dock opens on its Browser tab when an agent in the open workflow asks for help (`browser_updated`). Mobile shows the managed local Android phone. |
-| **[MCP Connectors](./docs-internal/mcp_connectors.md)** | Custom connectors: the owner adds an outside MCP server from Connectors → Add (URL, name, sign-in: none, bearer, one header, or OAuth), and OpenCompany is its MCP *client* (`server/nodes/mcp/`, official `mcp` SDK; unrelated to the CLI agents' `/mcp/ide` server). The server reads the tools before keeping anything: streamable HTTP, then SSE; our own httpx client with redirects off; host and resolved addresses of every request (an OAuth sign-in's to its authorization server too) checked against `services/netpolicy.py` (shared with the Browser node: never cloud metadata or OpenCompany's own ports, plain http only on the owner's network), within `DISCOVER_S` (45 s, under the app's 60 s probe wait). Kept as `mcp:<slug>` (sign-in, encrypted; plain `model_params._mcp`: tools as accepted, per-tool `{enabled, ask}`, a waiting refresh) and `mcp:<slug>_proxy` (URL). A tool no model could call is kept with the reason, never renamed. Each connector is its own card through the generic `Credential.catalogue_entries` hook and the `_mcp_connector` template. Commands `mcp_connector_add` / `_sign_in` / `_test` / `_refresh` / `_review` / `_set_tool` / `_remove`; Refresh holds every change for Accept. Client: `AddConnectorForm.tsx` and `panels/McpConnectorPanel.tsx`. The `mcpConnector` node on an agent's Tools gives one tool per tool that is on (`ToolNode.tool_bindings`, named `<slug>__<tool>`, run with its binding's locked `mcp_tool` / `mcp_ask` / labels; arguments checked with jsonschema; Ask first per tool through the node's `approval` spec); CLI agents leave it out and RLM refuses it. OAuth (`_oauth.py`) is the SDK's `OAuthClientProvider`: Add or Sign in again answers `sign_in_url`, the sign-in finishes in the background through `/api/mcp/oauth/callback`, tokens go to the OAuth token store under `mcp:<slug>` with their expiry (`store_oauth_tokens(..., expiry=)`), and each connection restores the expiry (read uncached with `get_stored_oauth_tokens`) and the endpoints before first use, so the SDK refreshes instead of opening a sign-in; a refused refresh says Sign in again. Each connector is also an app a hire can use: `connector_apps`, registered with `services/employees/apps.register_app_source`, is read by `Connections.apps` with `employee_apps.json`'s apps (a saved one counts as connected), `mcpConnector` is on the hire allowlist, and the employee's card reads the connector from the node's `app_field` (`mcp_connector`). |
+| **[MCP Connectors](./docs-internal/mcp_connectors.md)** | Custom connectors: the owner adds an outside MCP server from Connectors → Add (URL, name, sign-in: none, bearer, one header, or OAuth), and OpenCompany is its MCP *client* (`server/nodes/mcp/`, official `mcp` SDK; unrelated to the CLI agents' `/mcp/ide` server). The server reads the tools before keeping anything: streamable HTTP, then SSE; our own httpx client with redirects off; host and resolved addresses of every request (an OAuth sign-in's to its authorization server too) checked against `services/netpolicy.py` (shared with the Browser node: never cloud metadata or OpenCompany's own ports, plain http only on the owner's network), within `DISCOVER_S` (45 s, under the app's 60 s probe wait). Kept as `mcp:<slug>` (sign-in, encrypted; plain `model_params._mcp`: tools as accepted, per-tool `{enabled, ask}`, a waiting refresh) and `mcp:<slug>_proxy` (URL). A tool no model could call is kept with the reason, never renamed. Each connector is its own card through the generic `Credential.catalogue_entries` hook and the `_mcp_connector` template. Commands `mcp_connector_add` / `_sign_in` / `_test` / `_refresh` / `_review` / `_set_tool` / `_remove`; Refresh holds every change for Accept. Client: `AddConnectorForm.tsx` and `panels/McpConnectorPanel.tsx`. The `mcpConnector` node on an agent's Tools gives one tool per tool that is on (`ToolNode.tool_bindings`, named `<slug>__<tool>`, run with its binding's locked `mcp_tool` / `mcp_ask` / labels; arguments checked with jsonschema, a `$ref` followed only inside its schema: a tool whose inputs refer to another document is unusable, and a call goes as a plain `tools/call`, so the SDK never checks a result against an output schema the server sends; Ask first per tool through the node's `approval` spec); CLI agents leave it out and RLM refuses it. OAuth (`_oauth.py`) is the SDK's `OAuthClientProvider`: Add or Sign in again answers `sign_in_url`, the sign-in finishes in the background through `/api/mcp/oauth/callback`, tokens go to the OAuth token store under `mcp:<slug>` with their expiry (`store_oauth_tokens(..., expiry=)`), and each connection restores the expiry (read uncached with `get_stored_oauth_tokens`) and the endpoints before first use, so the SDK refreshes instead of opening a sign-in; a refused refresh says Sign in again. Each connector is also an app a hire can use: `connector_apps`, registered with `services/employees/apps.register_app_source`, is read by `Connections.apps` with `employee_apps.json`'s apps (a saved one counts as connected), `mcpConnector` is on the hire allowlist, and the employee's card reads the connector from the node's `app_field` (`mcp_connector`). |
| **[WhatsApp Business Service](./docs-internal/whatsapp_business_service.md)** | Official Meta Cloud API plugin in [`server/nodes/whatsapp_business/`](./server/nodes/whatsapp_business/) — **distinct from `nodes/whatsapp/`**, which is a personal account over an unofficial Go bridge; they share no node type, credential or palette group. Four nodes: `whatsappBusinessSend` (10 ops — Meta models message type as a *field* on one `/messages` endpoint, so text / media / template / interactive are operations, not nodes), `whatsappBusinessMedia` (the one genuinely separate endpoint family), and the `Receive` / `Status` triggers (kept separate because a canary trigger registers exactly one CloudEvents type). `phone_number_id` is deliberately **absent from Params** and comes from the credential only, because it selects the sending identity — `server_controlled_fields` would keep a model-set value out on every tool path, but an ActionNode's tool schema is its whole `Params` (`ToolInput` is ToolNode-only), so a declared field would still appear in the schema the model sees. Webhook via `register_webhook_source` on the shared catch-all, `X-Hub-Signature-256` HMAC failing closed, composite `wamid:status` dedup ids so `sent→delivered→read` don't collapse. |
| **[Stripe Service](./docs-internal/stripe_service.md)** | Stripe CLI integration — `stripeAction` (CLI pass-through) + `stripeReceive` (signed-webhook trigger, on a canvas Run and deployed: every Stripe event travels under one CloudEvents type, `com.opencompany.stripe.event.received`, with the Stripe type in `subject` and `data.event_type`, where the node's own `build_filter` reads it; the source shapes the payload once, wakes the in-process waiter through `WebhookSource.handle` and reaches deployed listeners through `dispatch.emit`; deploying starts `stripe listen` through the generic `TriggerNode.prepare_deployment` hook, at Start and at the boot re-arm). Reference plugin for the Wave 12 event framework AND the **CLI-managed-auth pattern**: browser OAuth via `stripe login --non-interactive` + `--complete <url>` (URL extracted from `next_step` shell command via `shlex.split`) + auto-installed binary (`ensure_stripe_cli` from GitHub releases) + marker-token reuse of `auth_service.store_oauth_tokens` + CloudEvents-shaped `broadcast_credential_event("credential.oauth.connected" \| ".disconnected", provider=...)` — zero per-provider hardcoding in the frontend. Daemon stdout/stderr ingested via `ProcessService`'s `line_handler` callback (no log-file tailing); credential gate routed through `await self.has_credential()` so non-api-key auth (`is_logged_in()`) plugs in via subclass override. |
| **[Vercel Service](./docs-internal/vercel_service.md)** | Vercel CLI integration — `vercelAction` (deploy / inspect / list + `custom` CLI passthrough; AI tool `vercel`). Second reference for the CLI-managed-auth pattern: the **device-flow variant** (`vercel login` is one blocking process — no two-step `--complete`; direct `create_subprocess_exec` + chunk-based banner read for the code-embedding URL, pumps drain pipes for the process lifetime, success gate = auth.json mtime advance + sniff). Dual auth: optional `vercel_token` field (injected as `VERCEL_TOKEN` env, never argv) OR CLI login with marker tokens. All invocations pin `--global-config <DATA_DIR>/vercel/` (the `CLAUDE_CONFIG_DIR` isolation idiom). `bun add` into the shared packages tree (pinned version, via `core/js_runtime.add_package`; the CLI runs on bun). First-deploy project guard raises `NodeUserError` instead of letting Vercel derive an invalid name from the workspace dir. Shipped with the generic OAuthConnect change: Login gates on **required** fields only, so optional fields never block CLI login. |
diff --git a/docs-internal/mcp_connectors.md b/docs-internal/mcp_connectors.md
index be55172c..42e6f7b6 100644
--- a/docs-internal/mcp_connectors.md
+++ b/docs-internal/mcp_connectors.md
@@ -89,7 +89,11 @@ A tool no model could call is kept with the reason and never renamed:
- its name repeats another;
- `<slug>__<tool>` breaks the strictest provider's rule,
`^[A-Za-z0-9_-]{1,64}$`;
-- its input schema is not an object, or is not valid JSON Schema.
+- its input schema is not an object, or is not valid JSON Schema;
+- its input schema refers to another document: a `$ref` that does not start
+ with `#`. A `$ref` is followed only inside its schema, because jsonschema,
+ given no registry, fetches whatever URL one names, from the backend and
+ outside the address checks above.
**Settings.** Each usable tool has `{enabled, ask}`. It starts on, and asks
first unless the server marks it read-only. The owner changes both per tool.
@@ -195,11 +199,14 @@ settings, which are locked so the model never sets them: `mcp_tool`, `mcp_ask`
- **A call**: the connector is read again, so a tool the server no longer
offers or the owner turned off since the run began is refused. The
- arguments are checked against the tool's input schema (jsonschema), then
- the tool is called over the connector's transport within `CALL_TIMEOUT_S`
- (300 s). The model gets the answer's text (each part that is not text is
- named) and its structured result; an answer marked as an error comes back
- as the tool's error.
+ arguments are checked against the tool's input schema (jsonschema with an
+ empty registry, so its `$ref`s resolve only inside it), then the tool is
+ called over the connector's transport within `CALL_TIMEOUT_S` (300 s), as a
+ plain `tools/call` request: the SDK's `ClientSession.call_tool` would check
+ the result against the output schema the server sends at that moment, and
+ follow any `$ref` in it. The model gets the answer's text (each part that
+ is not text is named) and its structured result; an answer marked as an
+ error comes back as the tool's error.
- **Ask first**: the node's `approval` spec holds a call while the employee
asks first, unless its tool is set not to (`mcp_ask`). A call whose binding
says nothing asks first. The card names the connector and the tool, and
diff --git a/server/nodes/mcp/_client.py b/server/nodes/mcp/_client.py
index 25a56bc5..a9ab31bc 100644
--- a/server/nodes/mcp/_client.py
+++ b/server/nodes/mcp/_client.py
@@ -18,6 +18,14 @@ addresses are checked per request, not pinned for the connection.
A tool as an employee sees it is ``<connector>__<tool>``. A tool whose name
or input description no model could use is kept with the reason, never
renamed (``unusable_reason``).
+
+A ``$ref`` in a server's schemas is followed only inside the schema: given
+no registry, jsonschema fetches whatever URL a ``$ref`` names, from this
+backend and outside the checks above. So arguments are checked with an
+empty registry (``check_arguments``), a tool whose inputs refer to another
+document is kept unusable, and a call goes as a plain ``tools/call``
+(``call_tool``): the SDK's own ``call_tool`` checks each result against the
+output schema the server sends at that moment.
"""
from __future__ import annotations
@@ -351,14 +359,19 @@ class CallResult:
def check_arguments(schema: Dict[str, Any], arguments: Dict[str, Any]) -> None:
- """Refuse arguments the tool's input schema does not accept."""
+ """Refuse arguments the tool's input schema does not accept. Its
+ ``$ref``s resolve inside the schema only (an empty registry)."""
from jsonschema import ValidationError
from jsonschema.validators import validator_for
+ from referencing import Registry
+ from referencing.exceptions import Unresolvable
try:
- validator_for(schema)(schema).validate(arguments)
+ validator_for(schema)(schema, registry=Registry()).validate(arguments)
except ValidationError as exc:
raise ConnectorError(f"The arguments don't fit the tool: {exc.message}") from None
+ except Unresolvable:
+ raise ConnectorError("The tool's inputs refer to a schema it doesn't include, so they can't be checked.") from None
def result_text(result: "CallToolResult") -> str:
@@ -387,12 +400,20 @@ async def call_tool(
http_transport: Optional[httpx.AsyncBaseTransport] = None,
auth: Optional[httpx.Auth] = None,
) -> CallResult:
- """Call one of the server's tools, within ``CALL_TIMEOUT_S``."""
+ """Call one of the server's tools, within ``CALL_TIMEOUT_S``, as a plain
+ ``tools/call``: ``ClientSession.call_tool`` would check the result
+ against an output schema the server sends at call time (see the module
+ docstring)."""
+ from mcp import types
+
host = await check_server(url)
+ request = types.ClientRequest(types.CallToolRequest(params=types.CallToolRequestParams(name=name, arguments=arguments)))
try:
async with asyncio.timeout(CALL_TIMEOUT_S):
async with open_session(url, sign_in, transport, http_transport=http_transport, auth=auth) as (session, _init):
- result = await session.call_tool(name, arguments, read_timeout_seconds=timedelta(seconds=CALL_TIMEOUT_S))
+ result = await session.send_request(
+ request, types.CallToolResult, request_read_timeout_seconds=timedelta(seconds=CALL_TIMEOUT_S)
+ )
except Exception as exc: # noqa: BLE001 - described below, in words for the owner
raise ConnectorError(describe(exc, host)) from None
return CallResult(text=result_text(result), structured=result.structuredContent, is_error=bool(result.isError))
@@ -458,9 +479,23 @@ def unusable_reason(slug: str, tool: Dict[str, Any], taken: set) -> Optional[str
validator_for(schema).check_schema(schema)
except SchemaError:
return "Its inputs aren't valid JSON Schema."
+ if _refers_elsewhere(schema):
+ return "Its inputs refer to another document."
return None
+def _refers_elsewhere(schema: Any) -> bool:
+ """Whether a ``$ref`` in ``schema`` names something outside it."""
+ if isinstance(schema, dict):
+ ref = schema.get("$ref")
+ if isinstance(ref, str) and not ref.startswith("#"):
+ return True
+ return any(_refers_elsewhere(value) for value in schema.values())
+ if isinstance(schema, list):
+ return any(_refers_elsewhere(value) for value in schema)
+ return False
+
+
def snapshots(slug: str, tools: List["Tool"]) -> List[Dict[str, Any]]:
"""The tools as kept, each saying whether it can be used and why not."""
out: List[Dict[str, Any]] = []
diff --git a/server/tests/nodes/test_mcp_connector.py b/server/tests/nodes/test_mcp_connector.py
index 58622a1c..d1d3b0c0 100644
--- a/server/tests/nodes/test_mcp_connector.py
+++ b/server/tests/nodes/test_mcp_connector.py
@@ -7,7 +7,9 @@ from __future__ import annotations
import contextlib
import functools
import json
+import threading
from datetime import datetime, timedelta, timezone
+from http.server import BaseHTTPRequestHandler, HTTPServer
from types import SimpleNamespace
from typing import Any, Dict, List, Tuple
from unittest.mock import AsyncMock
@@ -19,7 +21,7 @@ from mcp.server.fastmcp import FastMCP
from mcp.types import Tool, ToolAnnotations
from nodes.mcp import _client, _handlers, _oauth, mcp_connector
-from nodes.mcp._client import ConnectorError, SignIn, check_server, snapshots
+from nodes.mcp._client import ConnectorError, SignIn, check_arguments, check_server, snapshots
from nodes.mcp._credentials import McpConnectorCredential
from nodes.mcp._store import new_slug
from nodes.mcp.mcp_connector import McpConnectorNode
@@ -216,6 +218,8 @@ def test_a_tool_no_model_could_call_is_kept_with_the_reason():
Tool(name="x" * 60, inputSchema={"type": "object"}),
Tool(name="listing", inputSchema={"type": "array"}),
Tool(name="broken", inputSchema={"type": "object", "properties": {"a": {"type": "nonsense"}}}),
+ Tool(name="elsewhere", inputSchema={"type": "object", "properties": {"q": {"$ref": "https://example.com/q.json"}}}),
+ Tool(name="inside", inputSchema={"type": "object", "$defs": {"q": {"type": "string"}}, "properties": {"q": {"$ref": "#/$defs/q"}}}),
Tool(name="ok", inputSchema={"type": "object"}),
]
kept = snapshots("orders", tools)
@@ -226,6 +230,8 @@ def test_a_tool_no_model_could_call_is_kept_with_the_reason():
("x" * 60, False),
("listing", False),
("broken", False),
+ ("elsewhere", False),
+ ("inside", True),
("ok", False),
]
assert [tool["reason"] for tool in kept][1:] == [
@@ -233,10 +239,81 @@ def test_a_tool_no_model_could_call_is_kept_with_the_reason():
"Its name is too long, or has characters a model can't use.",
"Its inputs aren't described as an object.",
"Its inputs aren't valid JSON Schema.",
+ "Its inputs refer to another document.",
+ None,
"Another tool on this server has the same name.",
]
+def elsewhere():
+ """A web server a schema's ``$ref`` can point at: it serves a schema and
+ keeps the path of each request it gets."""
+ hits: List[str] = []
+
+ class Handler(BaseHTTPRequestHandler):
+ def do_GET(self): # noqa: N802 - the stdlib's name
+ hits.append(self.path)
+ body = b'{"type": "integer"}'
+ self.send_response(200)
+ self.send_header("Content-Type", "application/schema+json")
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, *args):
+ pass
+
+ server = HTTPServer(("127.0.0.1", 0), Handler)
+ threading.Thread(target=server.serve_forever, daemon=True).start()
+ yield SimpleNamespace(url=f"http://127.0.0.1:{server.server_port}/schema.json", hits=hits)
+ server.shutdown()
+ server.server_close()
+
+
+class OutputSchemaServer(httpx.AsyncBaseTransport):
+ """A bare MCP server (JSON-RPC over HTTP) with one tool, ``lookup``,
+ whose output schema is ``output_schema``."""
+
+ def __init__(self, output_schema: Dict[str, Any]) -> None:
+ self.output_schema = output_schema
+
+ async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
+ if request.method != "POST":
+ return httpx.Response(405, request=request)
+ message = json.loads(await request.aread())
+ if "id" not in message: # a notification
+ return httpx.Response(202, request=request)
+ if message["method"] == "initialize":
+ version = message["params"]["protocolVersion"]
+ result = {"protocolVersion": version, "capabilities": {"tools": {}}, "serverInfo": {"name": "Orders", "version": "1"}}
+ elif message["method"] == "tools/list":
+ result = {"tools": [{"name": "lookup", "inputSchema": {"type": "object"}, "outputSchema": self.output_schema}]}
+ else:
+ result = {"content": [{"type": "text", "text": "found"}], "structuredContent": {"order": "A1"}, "isError": False}
+ return httpx.Response(200, json={"jsonrpc": "2.0", "id": message["id"], "result": result}, request=request)
+
+
+class TestARefIsFollowedOnlyInsideItsSchema:
+ """A server's ``$ref`` never makes the backend fetch anything."""
+
+ def test_arguments_are_checked_without_fetching_a_ref(self, elsewhere):
+ with pytest.raises(ConnectorError, match="refer to a schema it doesn't include"):
+ check_arguments({"type": "object", "properties": {"q": {"$ref": elsewhere.url}}}, {"q": "x"})
+ assert elsewhere.hits == []
+ # A reference inside the schema still checks.
+ inside = {"type": "object", "$defs": {"q": {"type": "string"}}, "properties": {"q": {"$ref": "#/$defs/q"}}}
+ check_arguments(inside, {"q": "x"})
+ with pytest.raises(ConnectorError, match="don't fit the tool"):
+ check_arguments(inside, {"q": 1})
+
+ async def test_a_result_is_not_checked_against_a_schema_the_server_sends(self, elsewhere):
+ server = OutputSchemaServer({"type": "object", "properties": {"order": {"$ref": elsewhere.url}}})
+ result = await _client.call_tool(URL, SignIn(), "streamable_http", "lookup", {"order": "A1"}, http_transport=server)
+ assert (result.text, result.structured, result.is_error) == ("found", {"order": "A1"}, False)
+ assert elsewhere.hits == []
+
+
class TestTheConnectorsPage:
async def test_add_reads_the_server_and_keeps_it(self, auth, monkeypatch):
async with serving(monkeypatch, orders_server()) as transport: