Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(connectors): a server's $ref never makes the backend fetch a URL
trohitga82af6765282
diff --git a/CLAUDE.md b/CLAUDE.mdindex 689a3133..00b74c8e 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -33,7 +33,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[Email Service](./docs-internal/email_service.md)** | IMAP/SMTP integration via Himalaya CLI with EmailService orchestrator, provider presets, custom credential fallback, and polling triggers || **[Browser](./docs-internal/browser.md)** | Current `browser` node: installed Chrome/Edge/Chromium in a dedicated profile rendered inside the workspace by default (`BROWSER_RUNTIME=system`, `BROWSER_HEADLESS=true`) and pinned browser-use CLI. Chrome for Testing requires explicit `BROWSER_RUNTIME=testing`. System mode retains the native user agent and OS password-store/keychain integration; basic/mock keychain flags are restricted to testing mode. Includes persistent employee/shared profiles (the Credentials / Connectors "Web browser" panel adds, deletes and imports logins from a session file; the node's profile dropdown lists only the caller's profiles, via `current_load_options_principal()`), session leases, agent/user handoff, network policy, screenshots as workspace FileRefs and authenticated `/ws/browser` live viewing. Plugin-owned installation is lazy (`_install_chrome.py`, `_install_bu.py`); importing or shutting down the plugin never installs it. Replaces the former agent-browser and separate `browserHarness` implementations. || **[Browser Workspace](./docs-internal/browser_workspace.md)** | Home and Dev share `WorkspaceTabs` and `BrowserWorkspace`: Browser / Canvas / Mobile, capability-based discovery, binary JPEG frames, bounded frame credit and decoding, Take control / Hand back, the agent's cursor (`agent_action`: where each click, hover, type or select happened, beside the agent's result and never during a protected login), visibility cleanup and payload-free opt-in diagnostics. The Browser node's parameter panel shows the same viewer, and the Dev dock opens on its Browser tab when an agent in the open workflow asks for help (`browser_updated`). Mobile shows the managed local Android phone. |-| **[MCP Connectors](./docs-internal/mcp_connectors.md)** | Custom connectors: the owner adds an outside MCP server from Connectors → Add (URL, name, sign-in: none, bearer, one header, or OAuth), and OpenCompany is its MCP *client* (`server/nodes/mcp/`, official `mcp` SDK; unrelated to the CLI agents' `/mcp/ide` server). The server reads the tools before keeping anything: streamable HTTP, then SSE; our own httpx client with redirects off; host and resolved addresses of every request (an OAuth sign-in's to its authorization server too) checked against `services/netpolicy.py` (shared with the Browser node: never cloud metadata or OpenCompany's own ports, plain http only on the owner's network), within `DISCOVER_S` (45 s, under the app's 60 s probe wait). Kept as `mcp:<slug>` (sign-in, encrypted; plain `model_params._mcp`: tools as accepted, per-tool `{enabled, ask}`, a waiting refresh) and `mcp:<slug>_proxy` (URL). A tool no model could call is kept with the reason, never renamed. Each connector is its own card through the generic `Credential.catalogue_entries` hook and the `_mcp_connector` template. Commands `mcp_connector_add` / `_sign_in` / `_test` / `_refresh` / `_review` / `_set_tool` / `_remove`; Refresh holds every change for Accept. Client: `AddConnectorForm.tsx` and `panels/McpConnectorPanel.tsx`. The `mcpConnector` node on an agent's Tools gives one tool per tool that is on (`ToolNode.tool_bindings`, named `<slug>__<tool>`, run with its binding's locked `mcp_tool` / `mcp_ask` / labels; arguments checked with jsonschema; Ask first per tool through the node's `approval` spec); CLI agents leave it out and RLM refuses it. OAuth (`_oauth.py`) is the SDK's `OAuthClientProvider`: Add or Sign in again answers `sign_in_url`, the sign-in finishes in the background through `/api/mcp/oauth/callback`, tokens go to the OAuth token store under `mcp:<slug>` with their expiry (`store_oauth_tokens(..., expiry=)`), and each connection restores the expiry (read uncached with `get_stored_oauth_tokens`) and the endpoints before first use, so the SDK refreshes instead of opening a sign-in; a refused refresh says Sign in again. Each connector is also an app a hire can use: `connector_apps`, registered with `services/employees/apps.register_app_source`, is read by `Connections.apps` with `employee_apps.json`'s apps (a saved one counts as connected), `mcpConnector` is on the hire allowlist, and the employee's card reads the connector from the node's `app_field` (`mcp_connector`). |+| **[MCP Connectors](./docs-internal/mcp_connectors.md)** | Custom connectors: the owner adds an outside MCP server from Connectors → Add (URL, name, sign-in: none, bearer, one header, or OAuth), and OpenCompany is its MCP *client* (`server/nodes/mcp/`, official `mcp` SDK; unrelated to the CLI agents' `/mcp/ide` server). The server reads the tools before keeping anything: streamable HTTP, then SSE; our own httpx client with redirects off; host and resolved addresses of every request (an OAuth sign-in's to its authorization server too) checked against `services/netpolicy.py` (shared with the Browser node: never cloud metadata or OpenCompany's own ports, plain http only on the owner's network), within `DISCOVER_S` (45 s, under the app's 60 s probe wait). Kept as `mcp:<slug>` (sign-in, encrypted; plain `model_params._mcp`: tools as accepted, per-tool `{enabled, ask}`, a waiting refresh) and `mcp:<slug>_proxy` (URL). A tool no model could call is kept with the reason, never renamed. Each connector is its own card through the generic `Credential.catalogue_entries` hook and the `_mcp_connector` template. Commands `mcp_connector_add` / `_sign_in` / `_test` / `_refresh` / `_review` / `_set_tool` / `_remove`; Refresh holds every change for Accept. Client: `AddConnectorForm.tsx` and `panels/McpConnectorPanel.tsx`. The `mcpConnector` node on an agent's Tools gives one tool per tool that is on (`ToolNode.tool_bindings`, named `<slug>__<tool>`, run with its binding's locked `mcp_tool` / `mcp_ask` / labels; arguments checked with jsonschema, a `$ref` followed only inside its schema: a tool whose inputs refer to another document is unusable, and a call goes as a plain `tools/call`, so the SDK never checks a result against an output schema the server sends; Ask first per tool through the node's `approval` spec); CLI agents leave it out and RLM refuses it. OAuth (`_oauth.py`) is the SDK's `OAuthClientProvider`: Add or Sign in again answers `sign_in_url`, the sign-in finishes in the background through `/api/mcp/oauth/callback`, tokens go to the OAuth token store under `mcp:<slug>` with their expiry (`store_oauth_tokens(..., expiry=)`), and each connection restores the expiry (read uncached with `get_stored_oauth_tokens`) and the endpoints before first use, so the SDK refreshes instead of opening a sign-in; a refused refresh says Sign in again. Each connector is also an app a hire can use: `connector_apps`, registered with `services/employees/apps.register_app_source`, is read by `Connections.apps` with `employee_apps.json`'s apps (a saved one counts as connected), `mcpConnector` is on the hire allowlist, and the employee's card reads the connector from the node's `app_field` (`mcp_connector`). || **[WhatsApp Business Service](./docs-internal/whatsapp_business_service.md)** | Official Meta Cloud API plugin in [`server/nodes/whatsapp_business/`](./server/nodes/whatsapp_business/) — **distinct from `nodes/whatsapp/`**, which is a personal account over an unofficial Go bridge; they share no node type, credential or palette group. Four nodes: `whatsappBusinessSend` (10 ops — Meta models message type as a *field* on one `/messages` endpoint, so text / media / template / interactive are operations, not nodes), `whatsappBusinessMedia` (the one genuinely separate endpoint family), and the `Receive` / `Status` triggers (kept separate because a canary trigger registers exactly one CloudEvents type). `phone_number_id` is deliberately **absent from Params** and comes from the credential only, because it selects the sending identity — `server_controlled_fields` would keep a model-set value out on every tool path, but an ActionNode's tool schema is its whole `Params` (`ToolInput` is ToolNode-only), so a declared field would still appear in the schema the model sees. Webhook via `register_webhook_source` on the shared catch-all, `X-Hub-Signature-256` HMAC failing closed, composite `wamid:status` dedup ids so `sent→delivered→read` don't collapse. || **[Stripe Service](./docs-internal/stripe_service.md)** | Stripe CLI integration — `stripeAction` (CLI pass-through) + `stripeReceive` (signed-webhook trigger, on a canvas Run and deployed: every Stripe event travels under one CloudEvents type, `com.opencompany.stripe.event.received`, with the Stripe type in `subject` and `data.event_type`, where the node's own `build_filter` reads it; the source shapes the payload once, wakes the in-process waiter through `WebhookSource.handle` and reaches deployed listeners through `dispatch.emit`; deploying starts `stripe listen` through the generic `TriggerNode.prepare_deployment` hook, at Start and at the boot re-arm). Reference plugin for the Wave 12 event framework AND the **CLI-managed-auth pattern**: browser OAuth via `stripe login --non-interactive` + `--complete <url>` (URL extracted from `next_step` shell command via `shlex.split`) + auto-installed binary (`ensure_stripe_cli` from GitHub releases) + marker-token reuse of `auth_service.store_oauth_tokens` + CloudEvents-shaped `broadcast_credential_event("credential.oauth.connected" \| ".disconnected", provider=...)` — zero per-provider hardcoding in the frontend. Daemon stdout/stderr ingested via `ProcessService`'s `line_handler` callback (no log-file tailing); credential gate routed through `await self.has_credential()` so non-api-key auth (`is_logged_in()`) plugs in via subclass override. || **[Vercel Service](./docs-internal/vercel_service.md)** | Vercel CLI integration — `vercelAction` (deploy / inspect / list + `custom` CLI passthrough; AI tool `vercel`). Second reference for the CLI-managed-auth pattern: the **device-flow variant** (`vercel login` is one blocking process — no two-step `--complete`; direct `create_subprocess_exec` + chunk-based banner read for the code-embedding URL, pumps drain pipes for the process lifetime, success gate = auth.json mtime advance + sniff). Dual auth: optional `vercel_token` field (injected as `VERCEL_TOKEN` env, never argv) OR CLI login with marker tokens. All invocations pin `--global-config <DATA_DIR>/vercel/` (the `CLAUDE_CONFIG_DIR` isolation idiom). `bun add` into the shared packages tree (pinned version, via `core/js_runtime.add_package`; the CLI runs on bun). First-deploy project guard raises `NodeUserError` instead of letting Vercel derive an invalid name from the workspace dir. Shipped with the generic OAuthConnect change: Login gates on **required** fields only, so optional fields never block CLI login. |diff --git a/docs-internal/mcp_connectors.md b/docs-internal/mcp_connectors.mdindex be55172c..42e6f7b6 100644--- a/docs-internal/mcp_connectors.md+++ b/docs-internal/mcp_connectors.md@@ -89,7 +89,11 @@ A tool no model could call is kept with the reason and never renamed:- its name repeats another;- `<slug>__<tool>` breaks the strictest provider's rule,`^[A-Za-z0-9_-]{1,64}$`;-- its input schema is not an object, or is not valid JSON Schema.+- its input schema is not an object, or is not valid JSON Schema;+- its input schema refers to another document: a `$ref` that does not start+ with `#`. A `$ref` is followed only inside its schema, because jsonschema,+ given no registry, fetches whatever URL one names, from the backend and+ outside the address checks above.**Settings.** Each usable tool has `{enabled, ask}`. It starts on, and asksfirst unless the server marks it read-only. The owner changes both per tool.@@ -195,11 +199,14 @@ settings, which are locked so the model never sets them: `mcp_tool`, `mcp_ask`- **A call**: the connector is read again, so a tool the server no longeroffers or the owner turned off since the run began is refused. The- arguments are checked against the tool's input schema (jsonschema), then- the tool is called over the connector's transport within `CALL_TIMEOUT_S`- (300 s). The model gets the answer's text (each part that is not text is- named) and its structured result; an answer marked as an error comes back- as the tool's error.+ arguments are checked against the tool's input schema (jsonschema with an+ empty registry, so its `$ref`s resolve only inside it), then the tool is+ called over the connector's transport within `CALL_TIMEOUT_S` (300 s), as a+ plain `tools/call` request: the SDK's `ClientSession.call_tool` would check+ the result against the output schema the server sends at that moment, and+ follow any `$ref` in it. The model gets the answer's text (each part that+ is not text is named) and its structured result; an answer marked as an+ error comes back as the tool's error.- **Ask first**: the node's `approval` spec holds a call while the employeeasks first, unless its tool is set not to (`mcp_ask`). A call whose bindingsays nothing asks first. The card names the connector and the tool, anddiff --git a/server/nodes/mcp/_client.py b/server/nodes/mcp/_client.pyindex 25a56bc5..a9ab31bc 100644--- a/server/nodes/mcp/_client.py+++ b/server/nodes/mcp/_client.py@@ -18,6 +18,14 @@ addresses are checked per request, not pinned for the connection.A tool as an employee sees it is ``<connector>__<tool>``. A tool whose nameor input description no model could use is kept with the reason, neverrenamed (``unusable_reason``).++A ``$ref`` in a server's schemas is followed only inside the schema: given+no registry, jsonschema fetches whatever URL a ``$ref`` names, from this+backend and outside the checks above. So arguments are checked with an+empty registry (``check_arguments``), a tool whose inputs refer to another+document is kept unusable, and a call goes as a plain ``tools/call``+(``call_tool``): the SDK's own ``call_tool`` checks each result against the+output schema the server sends at that moment."""from __future__ import annotations@@ -351,14 +359,19 @@ class CallResult:def check_arguments(schema: Dict[str, Any], arguments: Dict[str, Any]) -> None:- """Refuse arguments the tool's input schema does not accept."""+ """Refuse arguments the tool's input schema does not accept. Its+ ``$ref``s resolve inside the schema only (an empty registry)."""from jsonschema import ValidationErrorfrom jsonschema.validators import validator_for+ from referencing import Registry+ from referencing.exceptions import Unresolvabletry:- validator_for(schema)(schema).validate(arguments)+ validator_for(schema)(schema, registry=Registry()).validate(arguments)except ValidationError as exc:raise ConnectorError(f"The arguments don't fit the tool: {exc.message}") from None+ except Unresolvable:+ raise ConnectorError("The tool's inputs refer to a schema it doesn't include, so they can't be checked.") from Nonedef result_text(result: "CallToolResult") -> str:@@ -387,12 +400,20 @@ async def call_tool(http_transport: Optional[httpx.AsyncBaseTransport] = None,auth: Optional[httpx.Auth] = None,) -> CallResult:- """Call one of the server's tools, within ``CALL_TIMEOUT_S``."""+ """Call one of the server's tools, within ``CALL_TIMEOUT_S``, as a plain+ ``tools/call``: ``ClientSession.call_tool`` would check the result+ against an output schema the server sends at call time (see the module+ docstring)."""+ from mcp import types+host = await check_server(url)+ request = types.ClientRequest(types.CallToolRequest(params=types.CallToolRequestParams(name=name, arguments=arguments)))try:async with asyncio.timeout(CALL_TIMEOUT_S):async with open_session(url, sign_in, transport, http_transport=http_transport, auth=auth) as (session, _init):- result = await session.call_tool(name, arguments, read_timeout_seconds=timedelta(seconds=CALL_TIMEOUT_S))+ result = await session.send_request(+ request, types.CallToolResult, request_read_timeout_seconds=timedelta(seconds=CALL_TIMEOUT_S)+ )except Exception as exc: # noqa: BLE001 - described below, in words for the ownerraise ConnectorError(describe(exc, host)) from Nonereturn CallResult(text=result_text(result), structured=result.structuredContent, is_error=bool(result.isError))@@ -458,9 +479,23 @@ def unusable_reason(slug: str, tool: Dict[str, Any], taken: set) -> Optional[strvalidator_for(schema).check_schema(schema)except SchemaError:return "Its inputs aren't valid JSON Schema."+ if _refers_elsewhere(schema):+ return "Its inputs refer to another document."return None+def _refers_elsewhere(schema: Any) -> bool:+ """Whether a ``$ref`` in ``schema`` names something outside it."""+ if isinstance(schema, dict):+ ref = schema.get("$ref")+ if isinstance(ref, str) and not ref.startswith("#"):+ return True+ return any(_refers_elsewhere(value) for value in schema.values())+ if isinstance(schema, list):+ return any(_refers_elsewhere(value) for value in schema)+ return False++def snapshots(slug: str, tools: List["Tool"]) -> List[Dict[str, Any]]:"""The tools as kept, each saying whether it can be used and why not."""out: List[Dict[str, Any]] = []diff --git a/server/tests/nodes/test_mcp_connector.py b/server/tests/nodes/test_mcp_connector.pyindex 58622a1c..d1d3b0c0 100644--- a/server/tests/nodes/test_mcp_connector.py+++ b/server/tests/nodes/test_mcp_connector.py@@ -7,7 +7,9 @@ from __future__ import annotationsimport contextlibimport functoolsimport json+import threadingfrom datetime import datetime, timedelta, timezone+from http.server import BaseHTTPRequestHandler, HTTPServerfrom types import SimpleNamespacefrom typing import Any, Dict, List, Tuplefrom unittest.mock import AsyncMock@@ -19,7 +21,7 @@ from mcp.server.fastmcp import FastMCPfrom mcp.types import Tool, ToolAnnotationsfrom nodes.mcp import _client, _handlers, _oauth, mcp_connector-from nodes.mcp._client import ConnectorError, SignIn, check_server, snapshots+from nodes.mcp._client import ConnectorError, SignIn, check_arguments, check_server, snapshotsfrom nodes.mcp._credentials import McpConnectorCredentialfrom nodes.mcp._store import new_slugfrom nodes.mcp.mcp_connector import McpConnectorNode@@ -216,6 +218,8 @@ def test_a_tool_no_model_could_call_is_kept_with_the_reason():Tool(name="x" * 60, inputSchema={"type": "object"}),Tool(name="listing", inputSchema={"type": "array"}),Tool(name="broken", inputSchema={"type": "object", "properties": {"a": {"type": "nonsense"}}}),+ Tool(name="elsewhere", inputSchema={"type": "object", "properties": {"q": {"$ref": "https://example.com/q.json"}}}),+ Tool(name="inside", inputSchema={"type": "object", "$defs": {"q": {"type": "string"}}, "properties": {"q": {"$ref": "#/$defs/q"}}}),Tool(name="ok", inputSchema={"type": "object"}),]kept = snapshots("orders", tools)@@ -226,6 +230,8 @@ def test_a_tool_no_model_could_call_is_kept_with_the_reason():("x" * 60, False),("listing", False),("broken", False),+ ("elsewhere", False),+ ("inside", True),("ok", False),]assert [tool["reason"] for tool in kept][1:] == [@@ -233,10 +239,81 @@ def test_a_tool_no_model_could_call_is_kept_with_the_reason():"Its name is too long, or has characters a model can't use.","Its inputs aren't described as an object.","Its inputs aren't valid JSON Schema.",+ "Its inputs refer to another document.",+ None,"Another tool on this server has the same name.",]+def elsewhere():+ """A web server a schema's ``$ref`` can point at: it serves a schema and+ keeps the path of each request it gets."""+ hits: List[str] = []++ class Handler(BaseHTTPRequestHandler):+ def do_GET(self): # noqa: N802 - the stdlib's name+ hits.append(self.path)+ body = b'{"type": "integer"}'+ self.send_response(200)+ self.send_header("Content-Type", "application/schema+json")+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def log_message(self, *args):+ pass++ server = HTTPServer(("127.0.0.1", 0), Handler)+ threading.Thread(target=server.serve_forever, daemon=True).start()+ yield SimpleNamespace(url=f"http://127.0.0.1:{server.server_port}/schema.json", hits=hits)+ server.shutdown()+ server.server_close()+++class OutputSchemaServer(httpx.AsyncBaseTransport):+ """A bare MCP server (JSON-RPC over HTTP) with one tool, ``lookup``,+ whose output schema is ``output_schema``."""++ def __init__(self, output_schema: Dict[str, Any]) -> None:+ self.output_schema = output_schema++ async def handle_async_request(self, request: httpx.Request) -> httpx.Response:+ if request.method != "POST":+ return httpx.Response(405, request=request)+ message = json.loads(await request.aread())+ if "id" not in message: # a notification+ return httpx.Response(202, request=request)+ if message["method"] == "initialize":+ version = message["params"]["protocolVersion"]+ result = {"protocolVersion": version, "capabilities": {"tools": {}}, "serverInfo": {"name": "Orders", "version": "1"}}+ elif message["method"] == "tools/list":+ result = {"tools": [{"name": "lookup", "inputSchema": {"type": "object"}, "outputSchema": self.output_schema}]}+ else:+ result = {"content": [{"type": "text", "text": "found"}], "structuredContent": {"order": "A1"}, "isError": False}+ return httpx.Response(200, json={"jsonrpc": "2.0", "id": message["id"], "result": result}, request=request)+++class TestARefIsFollowedOnlyInsideItsSchema:+ """A server's ``$ref`` never makes the backend fetch anything."""++ def test_arguments_are_checked_without_fetching_a_ref(self, elsewhere):+ with pytest.raises(ConnectorError, match="refer to a schema it doesn't include"):+ check_arguments({"type": "object", "properties": {"q": {"$ref": elsewhere.url}}}, {"q": "x"})+ assert elsewhere.hits == []+ # A reference inside the schema still checks.+ inside = {"type": "object", "$defs": {"q": {"type": "string"}}, "properties": {"q": {"$ref": "#/$defs/q"}}}+ check_arguments(inside, {"q": "x"})+ with pytest.raises(ConnectorError, match="don't fit the tool"):+ check_arguments(inside, {"q": 1})++ async def test_a_result_is_not_checked_against_a_schema_the_server_sends(self, elsewhere):+ server = OutputSchemaServer({"type": "object", "properties": {"order": {"$ref": elsewhere.url}}})+ result = await _client.call_tool(URL, SignIn(), "streamable_http", "lookup", {"order": "A1"}, http_transport=server)+ assert (result.text, result.structured, result.is_error) == ("found", {"order": "A1"}, False)+ assert elsewhere.hits == []++class TestTheConnectorsPage:async def test_add_reads_the_server_and_keeps_it(self, auth, monkeypatch):async with serving(monkeypatch, orders_server()) as transport: