Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
TypeScript agentic-aiagentsaiai-agentsaiagentsdeveloper-toolsfunction-callinggpt-4javascriptjsllmllmopsmcppythonremote-mcp-serverssetypescript
fix(harness): unblock python examples parity runs (#4228)
Alberto Schiabel5b898e7957c3
diff --git a/.gitignore b/.gitignoreindex d20273ad8..6ce164ae4 100644--- a/.gitignore+++ b/.gitignore@@ -75,3 +75,6 @@ docs/.workflow-data/*# Local agent / tooling state (regenerated per machine).pi/.playwright-mcp/++# Examples harness sweep output (results, traces, candidate client artifacts)+.artifacts/diff --git a/harness/README.md b/harness/README.mdnew file mode 100644index 000000000..437a9f6ec--- /dev/null+++ b/harness/README.md@@ -0,0 +1,82 @@+# Examples harness++Runs the entrypoints inventoried in `examples-manifest.json` against a live+Composio backend, records every backend call each entry makes, and compares two+runs call-for-call.++- `harness/run.mjs` — sweeps entries, writes `results.jsonl` + one trace file per+ entry under `.artifacts/examples-parity/<run-id>/`.+- `harness/parity.mjs` — compares two run directories.+- `harness/trace/register.mjs`, `harness/trace-py/sitecustomize.py` — the fetch+ and httpx tracers, injected by the runner. Examples never reference+ `COMPOSIO_TRACE_FILE` themselves.+- `scripts/examples-provision.mjs` — provisions the auth configs and connected+ accounts the tier-2/3 entries need, and prints them as `COMPOSIO_EXAMPLES_*`+ exports.++## Backend selection++`COMPOSIO_BASE_URL` picks the backend and defaults to staging. Any bare `https`+root is accepted; a URL carrying a path, query, fragment, or embedded+credentials is refused, because the tracers pin the backend host and the+provisioner appends its own paths. `.github/workflows/examples-live.yml` sets+staging explicitly, so CI is unaffected by the default.++Point this at a project whose data you are willing to have the examples touch.+`scripts/examples-provision.mjs --gc` deletes examples-owned resources older+than 24h across the whole project — never run it against a project you care+about. The outbound-email denylist is enforced in the tracers and is not+overridable per run; `--llm mock` additionally keeps model traffic on a local+`aimock` server so no agent can decide to write something.++## Comparing a client bump++A parity run is two sweeps over the same entry ids, one on each side of the+change, compared by traced `(method, path-template)` pairs.++```bash+# Provisioned ids for the project you are sweeping. Capture, then eval — a+# failed provisioning run must not be swallowed.+out=$(node scripts/examples-provision.mjs) && eval "$out"++# Baseline: the pinned client, from a checkout of the base branch.+node harness/run.mjs sweep --client baseline --lang py --llm mock --ids "$IDS"++# Candidate: the same entries with the candidate client swapped in.+COMPOSIO_CLIENT_WHEEL=/abs/path/composio_client-<version>-py3-none-any.whl \+ node harness/run.mjs sweep --client candidate --lang py --llm mock --ids "$IDS"++node harness/parity.mjs <baseline-run-dir> <candidate-run-dir>+```++Pass `--ids` explicitly rather than relying on the default selection, so both+sides run the same set even when the two checkouts disagree about the manifest.++The candidate client comes from a **local artifact**, not a version spec:+`COMPOSIO_CLIENT_TARBALL` for TypeScript, `COMPOSIO_CLIENT_WHEEL` for Python.+Fetch the Python one with `pip download composio-client==<version> --no-deps`.+Both swaps abort the sweep if the client the project resolves does not actually+change, and both restore the files they touched when the sweep ends.++For Python the runner also drops the project's exact `composio-client==` pin for+the duration of a candidate sweep. Several entries install the local `./python`+project through `pyWith`, and uv cannot satisfy that pin and the candidate wheel+at once — without this, those entries fail to resolve and go red for a packaging+reason, quietly shrinking the comparison.++`parity.mjs` only compares entries green in **both** runs; anything red, skipped,+or missing on either side becomes `parity: false` with a reason rather than+failing the comparator. Read `compared` alongside `parityGreen`, and check that+the traces are non-empty — parity over two empty traces holds vacuously.++## Verifying the harness itself++```bash+node harness/run.mjs selftest+```++Covers backend-URL handling, the candidate-swap guards, the known-good and+known-bad fixtures, both tracers, and the comparator's own accept/reject+behaviour. `node harness/run.mjs neg` is the complementary check on the examples:+every entry must go red under garbage credentials, so an entry that swallows its+errors cannot pass as coverage.diff --git a/harness/backend-url.mjs b/harness/backend-url.mjsnew file mode 100644index 000000000..f6fa83511--- /dev/null+++ b/harness/backend-url.mjs@@ -0,0 +1,30 @@+export const STAGING_BASE_URL = 'https://staging-backend.composio.dev';++// Resolves the backend the examples run against: whatever COMPOSIO_BASE_URL+// says, defaulting to staging. The structural checks are not about which+// backend it is — a base URL carrying a path, query, fragment, or embedded+// credentials means the caller meant something else, and the origin+// normalisation below only holds for a bare root.+export const resolveBackendBaseUrl = (+ value = process.env.COMPOSIO_BASE_URL ?? STAGING_BASE_URL+) => {+ let url;+ try {+ url = new URL(value);+ } catch {+ throw new Error(`refusing invalid COMPOSIO_BASE_URL: ${value}`);+ }++ const isBareHttpsRoot =+ url.protocol === 'https:' &&+ url.pathname === '/' &&+ !url.search &&+ !url.hash &&+ !url.username &&+ !url.password;+ if (!isBareHttpsRoot) {+ throw new Error(`refusing malformed COMPOSIO_BASE_URL: ${value}`);+ }++ return url.origin;+};diff --git a/harness/run.mjs b/harness/run.mjsindex bf159fa6a..e9cfb086a 100755--- a/harness/run.mjs+++ b/harness/run.mjs@@ -12,7 +12,7 @@ import { spawn } from 'node:child_process';import { mkdirSync, readFileSync, writeFileSync, appendFileSync, existsSync, rmSync } from 'node:fs';import { dirname, join, relative, resolve } from 'node:path';import { fileURLToPath } from 'node:url';-import { requireStagingBaseUrl, STAGING_BASE_URL } from './staging-backend.mjs';+import { resolveBackendBaseUrl, STAGING_BASE_URL } from './backend-url.mjs';const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');const ARTIFACTS = join(ROOT, '.artifacts', 'examples-parity');@@ -67,7 +67,7 @@ const selectEntries = () => {const baseUrl = () => {try {- return requireStagingBaseUrl();+ return resolveBackendBaseUrl();} catch (error) {fail(error instanceof Error ? error.message : String(error));}@@ -315,6 +315,35 @@ const verifyPyCandidate = async (wheel) => {return version;};+const PY_PROJECT = join(ROOT, 'python', 'pyproject.toml');+const UV_LOCK = join(ROOT, 'uv.lock');+const PY_CLIENT_PIN = /^(\s*)"composio-client==[^"]+",$/m;++/**+ * Drop the exact `composio-client==` pin from the python project.+ *+ * A candidate sweep layers the wheel onto every entry with `uv run --with`, but+ * several entries also install the local `./python` project, whose exact pin the+ * wheel contradicts — uv then refuses the whole overlay ("no solution found")+ * and those entries go red for a packaging reason rather than a client one.+ * Without the pin the wheel is the only `composio-client` in play. Baseline+ * sweeps never call this, and restorePyBaseline() writes the pre-run files back+ * either way.+ */+const relaxPyClientPin = (toml) => {+ if (!PY_CLIENT_PIN.test(toml)) {+ fail(`no exact composio-client pin found in ${relative(ROOT, PY_PROJECT)}`);+ }+ return toml.replace(PY_CLIENT_PIN, '$1"composio-client",');+};++const snapshotPyBaseline = () => snapshotFiles([PY_PROJECT, UV_LOCK]);++const restorePyBaseline = async (snapshot) => {+ restoreFiles(snapshot);+ await sh(['uv', 'sync', '--project', 'python', '--frozen']);+};+// --- llm mock ---------------------------------------------------------------/** Start aimock serving harness/llm-mock/fixtures; resolve once it answers. */@@ -365,6 +394,7 @@ const cmdSweep = async () => {mkdirSync(join(runDir, 'traces'), { recursive: true });let tsBaselineSnapshot;+ let pyBaselineSnapshot;let stopLlmMock;const pyWheel = process.env.COMPOSIO_CLIENT_WHEEL;try {@@ -381,6 +411,8 @@ const cmdSweep = async () => {if (!pyWheel) fail('candidate sweep with py entries needs COMPOSIO_CLIENT_WHEEL');const v = await verifyPyCandidate(pyWheel);console.log(`candidate composio-client (py) resolved: ${v}`);+ pyBaselineSnapshot = snapshotPyBaseline();+ writeFileSync(PY_PROJECT, relaxPyClientPin(readFileSync(PY_PROJECT, 'utf8')));for (const e of entries) if (e.lang === 'py') e.pyWith = [...(e.pyWith ?? []), pyWheel];}}@@ -402,6 +434,7 @@ const cmdSweep = async () => {} finally {if (stopLlmMock) stopLlmMock();if (tsBaselineSnapshot) await restoreTsBaseline(tsBaselineSnapshot);+ if (pyBaselineSnapshot) await restorePyBaseline(pyBaselineSnapshot);}};@@ -438,15 +471,29 @@ const cmdSelftest = async () => {if (!ok) failures += 1;};- // 0. backend safety: the harness accepts only the canonical staging root.- check('staging backend is accepted', requireStagingBaseUrl() === STAGING_BASE_URL);- let productionRejected = false;- try {- requireStagingBaseUrl('https://backend.composio.dev');- } catch {- productionRejected = true;- }- check('production backend is rejected', productionRejected);+ // 0. backend URL: staging is the default, any other bare https root is+ // honoured as given, and anything structurally unusable is refused.+ check('staging backend is the default', resolveBackendBaseUrl(undefined) === STAGING_BASE_URL);+ check(+ 'a non-staging bare https root is honoured',+ resolveBackendBaseUrl('https://backend.composio.dev') === 'https://backend.composio.dev'+ );+ const refuses = (value) => {+ try {+ resolveBackendBaseUrl(value);+ return false;+ } catch {+ return true;+ }+ };+ check('a base URL carrying a path is refused', refuses('https://backend.composio.dev/api/v3'));+ check('a base URL carrying a query is refused', refuses('https://backend.composio.dev/?x=1'));+ check(+ 'a base URL carrying credentials is refused',+ refuses('https://user:[email protected]')+ );+ check('a plaintext base URL is refused', refuses('http://backend.composio.dev'));+ check('an unparseable base URL is refused', refuses('not-a-url'));let cleanupRan = false;let cleanupError;@@ -471,6 +518,28 @@ const cmdSelftest = async () => {restoreFiles(cleanupSnapshot);check('cleanup restores pre-run file contents', readFileSync(cleanupFixture, 'utf8') === 'user contents');+ // The python candidate swap only resolves once the project's exact+ // composio-client pin is out of the way; see relaxPyClientPin.+ const pinnedProject = readFileSync(PY_PROJECT, 'utf8');+ const relaxedProject = relaxPyClientPin(pinnedProject);+ check('the python project pins composio-client exactly', /"composio-client==/.test(pinnedProject));+ check(+ 'relaxing the pin keeps composio-client a dependency',+ /^\s*"composio-client",$/m.test(relaxedProject) && !/"composio-client==/.test(relaxedProject)+ );+ check(+ 'relaxing the pin touches nothing else',+ relaxedProject.replace(/^\s*"composio-client",$/m, '') ===+ pinnedProject.replace(/^\s*"composio-client==[^"]+",$/m, '')+ );+ let unpinnedProjectRejected = false;+ try {+ relaxPyClientPin(relaxedProject);+ } catch {+ unpinnedProjectRejected = true;+ }+ check('relaxing an already-unpinned project is refused', unpinnedProjectRejected);+// 1. known-good: error-handling-demo (no backend, no keys).const good = loadManifest().find((e) => e.id === 'ts/error-handling-demo/index');const { env: goodEnv } = buildEnv(good, runDir);diff --git a/harness/staging-backend.mjs b/harness/staging-backend.mjsdeleted file mode 100644index 966e72469..000000000--- a/harness/staging-backend.mjs+++ /dev/null@@ -1,25 +0,0 @@-export const STAGING_BASE_URL = 'https://staging-backend.composio.dev';--export const requireStagingBaseUrl = (- value = process.env.COMPOSIO_BASE_URL ?? STAGING_BASE_URL-) => {- let url;- try {- url = new URL(value);- } catch {- throw new Error(`refusing invalid COMPOSIO_BASE_URL: ${value}`);- }-- const isStagingRoot =- url.origin === STAGING_BASE_URL &&- url.pathname === '/' &&- !url.search &&- !url.hash &&- !url.username &&- !url.password;- if (!isStagingRoot) {- throw new Error(`refusing non-staging COMPOSIO_BASE_URL: ${value}`);- }-- return STAGING_BASE_URL;-};diff --git a/harness/trace-py/sitecustomize.py b/harness/trace-py/sitecustomize.pyindex 634e323ab..8054916d5 100644--- a/harness/trace-py/sitecustomize.py+++ b/harness/trace-py/sitecustomize.py@@ -13,11 +13,15 @@ _TRACE = os.environ.get("COMPOSIO_TRACE_FILE")_STAGING_BASE_URL = "https://staging-backend.composio.dev"if _TRACE:+ # Mirrors harness/backend-url.mjs: COMPOSIO_BASE_URL selects the backend and+ # defaults to staging. The structural checks stay — a base URL carrying a+ # path, query, fragment, or embedded credentials means the caller meant+ # something else, and the host pin below needs a bare root to be meaningful._base_url = os.environ.get("COMPOSIO_BASE_URL", _STAGING_BASE_URL)_parsed_base_url = urlsplit(_base_url)if not (_parsed_base_url.scheme == "https"- and _parsed_base_url.hostname == "staging-backend.composio.dev"+ and _parsed_base_url.hostnameand _parsed_base_url.path in {"", "/"}and not _parsed_base_url.queryand not _parsed_base_url.fragment@@ -25,7 +29,7 @@ if _TRACE:and not _parsed_base_url.password):print(- f"sitecustomize: refusing non-staging COMPOSIO_BASE_URL: {_base_url}",+ f"sitecustomize: refusing malformed COMPOSIO_BASE_URL: {_base_url}",file=sys.stderr,)os._exit(78)diff --git a/harness/trace/register.mjs b/harness/trace/register.mjsindex 7c0e6adc6..457a7f6d1 100644--- a/harness/trace/register.mjs+++ b/harness/trace/register.mjs@@ -1,6 +1,6 @@// Harness-owned fetch tracer. Injected via NODE_OPTIONS=--import; examples// must never reference COMPOSIO_TRACE_FILE themselves (lint-enforced).-import { requireStagingBaseUrl } from '../staging-backend.mjs';+import { resolveBackendBaseUrl } from '../backend-url.mjs';const traceFile = process.env.COMPOSIO_TRACE_FILE;@@ -13,7 +13,7 @@ if (traceFile) {'generativelanguage.googleapis.com',]);- const backendHost = new URL(requireStagingBaseUrl()).hostname;+ const backendHost = new URL(resolveBackendBaseUrl()).hostname;const ID_SEG =/^(ca_|ac_|ti_|tr_|trs_|sess_|auth_|req_|proj_|org_)[\w-]+$|^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$|^\d+$/i;diff --git a/scripts/examples-provision.mjs b/scripts/examples-provision.mjsindex f62773c0c..7950d7bee 100644--- a/scripts/examples-provision.mjs+++ b/scripts/examples-provision.mjs@@ -30,11 +30,11 @@// values are ever printed. The API-key demo value stored for serpapi is a// deliberately fake placeholder, not a real key.-import { requireStagingBaseUrl } from '../harness/staging-backend.mjs';+import { resolveBackendBaseUrl } from '../harness/backend-url.mjs';let BASE_URL;try {- BASE_URL = requireStagingBaseUrl();+ BASE_URL = resolveBackendBaseUrl();} catch (error) {console.error(error instanceof Error ? error.message : String(error));process.exit(1);@@ -45,8 +45,10 @@ const INITIATE_MISSING = process.argv.includes('--initiate-missing');const GC = process.argv.includes('--gc');const DRY_RUN = process.argv.includes('--dry-run');-// The examples only ever run against staging with a disposable project key.-// requireStagingBaseUrl refuses every other host before the key is read or sent.+// COMPOSIO_BASE_URL selects the backend and defaults to staging;+// resolveBackendBaseUrl refuses a structurally unusable one before the key is+// read or sent. Point this at a project whose resources you are willing to have+// this script create — and, under --gc, delete.if (!API_KEY) {console.error('COMPOSIO_API_KEY is required (dedicated examples-project key)');process.exit(1);