Skip to content

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

fix(harness): unblock python examples parity runs (#4228)

Alberto Schiabel5b898e7957c3
diff --git a/.gitignore b/.gitignore
index d20273ad8..6ce164ae4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -75,3 +75,6 @@ docs/.workflow-data/*
# Local agent / tooling state (regenerated per machine)
.pi/
.playwright-mcp/
+
+# Examples harness sweep output (results, traces, candidate client artifacts)
+.artifacts/
diff --git a/harness/README.md b/harness/README.md
new file mode 100644
index 000000000..437a9f6ec
--- /dev/null
+++ b/harness/README.md
@@ -0,0 +1,82 @@
+# Examples harness
+
+Runs the entrypoints inventoried in `examples-manifest.json` against a live
+Composio backend, records every backend call each entry makes, and compares two
+runs call-for-call.
+
+- `harness/run.mjs` — sweeps entries, writes `results.jsonl` + one trace file per
+ entry under `.artifacts/examples-parity/<run-id>/`.
+- `harness/parity.mjs` — compares two run directories.
+- `harness/trace/register.mjs`, `harness/trace-py/sitecustomize.py` — the fetch
+ and httpx tracers, injected by the runner. Examples never reference
+ `COMPOSIO_TRACE_FILE` themselves.
+- `scripts/examples-provision.mjs` — provisions the auth configs and connected
+ accounts the tier-2/3 entries need, and prints them as `COMPOSIO_EXAMPLES_*`
+ exports.
+
+## Backend selection
+
+`COMPOSIO_BASE_URL` picks the backend and defaults to staging. Any bare `https`
+root is accepted; a URL carrying a path, query, fragment, or embedded
+credentials is refused, because the tracers pin the backend host and the
+provisioner appends its own paths. `.github/workflows/examples-live.yml` sets
+staging explicitly, so CI is unaffected by the default.
+
+Point this at a project whose data you are willing to have the examples touch.
+`scripts/examples-provision.mjs --gc` deletes examples-owned resources older
+than 24h across the whole project — never run it against a project you care
+about. The outbound-email denylist is enforced in the tracers and is not
+overridable per run; `--llm mock` additionally keeps model traffic on a local
+`aimock` server so no agent can decide to write something.
+
+## Comparing a client bump
+
+A parity run is two sweeps over the same entry ids, one on each side of the
+change, compared by traced `(method, path-template)` pairs.
+
+```bash
+# Provisioned ids for the project you are sweeping. Capture, then eval — a
+# failed provisioning run must not be swallowed.
+out=$(node scripts/examples-provision.mjs) && eval "$out"
+
+# Baseline: the pinned client, from a checkout of the base branch.
+node harness/run.mjs sweep --client baseline --lang py --llm mock --ids "$IDS"
+
+# Candidate: the same entries with the candidate client swapped in.
+COMPOSIO_CLIENT_WHEEL=/abs/path/composio_client-<version>-py3-none-any.whl \
+ node harness/run.mjs sweep --client candidate --lang py --llm mock --ids "$IDS"
+
+node harness/parity.mjs <baseline-run-dir> <candidate-run-dir>
+```
+
+Pass `--ids` explicitly rather than relying on the default selection, so both
+sides run the same set even when the two checkouts disagree about the manifest.
+
+The candidate client comes from a **local artifact**, not a version spec:
+`COMPOSIO_CLIENT_TARBALL` for TypeScript, `COMPOSIO_CLIENT_WHEEL` for Python.
+Fetch the Python one with `pip download composio-client==<version> --no-deps`.
+Both swaps abort the sweep if the client the project resolves does not actually
+change, and both restore the files they touched when the sweep ends.
+
+For Python the runner also drops the project's exact `composio-client==` pin for
+the duration of a candidate sweep. Several entries install the local `./python`
+project through `pyWith`, and uv cannot satisfy that pin and the candidate wheel
+at once — without this, those entries fail to resolve and go red for a packaging
+reason, quietly shrinking the comparison.
+
+`parity.mjs` only compares entries green in **both** runs; anything red, skipped,
+or missing on either side becomes `parity: false` with a reason rather than
+failing the comparator. Read `compared` alongside `parityGreen`, and check that
+the traces are non-empty — parity over two empty traces holds vacuously.
+
+## Verifying the harness itself
+
+```bash
+node harness/run.mjs selftest
+```
+
+Covers backend-URL handling, the candidate-swap guards, the known-good and
+known-bad fixtures, both tracers, and the comparator's own accept/reject
+behaviour. `node harness/run.mjs neg` is the complementary check on the examples:
+every entry must go red under garbage credentials, so an entry that swallows its
+errors cannot pass as coverage.
diff --git a/harness/backend-url.mjs b/harness/backend-url.mjs
new file mode 100644
index 000000000..f6fa83511
--- /dev/null
+++ b/harness/backend-url.mjs
@@ -0,0 +1,30 @@
+export const STAGING_BASE_URL = 'https://staging-backend.composio.dev';
+
+// Resolves the backend the examples run against: whatever COMPOSIO_BASE_URL
+// says, defaulting to staging. The structural checks are not about which
+// backend it is — a base URL carrying a path, query, fragment, or embedded
+// credentials means the caller meant something else, and the origin
+// normalisation below only holds for a bare root.
+export const resolveBackendBaseUrl = (
+ value = process.env.COMPOSIO_BASE_URL ?? STAGING_BASE_URL
+) => {
+ let url;
+ try {
+ url = new URL(value);
+ } catch {
+ throw new Error(`refusing invalid COMPOSIO_BASE_URL: ${value}`);
+ }
+
+ const isBareHttpsRoot =
+ url.protocol === 'https:' &&
+ url.pathname === '/' &&
+ !url.search &&
+ !url.hash &&
+ !url.username &&
+ !url.password;
+ if (!isBareHttpsRoot) {
+ throw new Error(`refusing malformed COMPOSIO_BASE_URL: ${value}`);
+ }
+
+ return url.origin;
+};
diff --git a/harness/run.mjs b/harness/run.mjs
index bf159fa6a..e9cfb086a 100755
--- a/harness/run.mjs
+++ b/harness/run.mjs
@@ -12,7 +12,7 @@ import { spawn } from 'node:child_process';
import { mkdirSync, readFileSync, writeFileSync, appendFileSync, existsSync, rmSync } from 'node:fs';
import { dirname, join, relative, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
-import { requireStagingBaseUrl, STAGING_BASE_URL } from './staging-backend.mjs';
+import { resolveBackendBaseUrl, STAGING_BASE_URL } from './backend-url.mjs';
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
const ARTIFACTS = join(ROOT, '.artifacts', 'examples-parity');
@@ -67,7 +67,7 @@ const selectEntries = () => {
const baseUrl = () => {
try {
- return requireStagingBaseUrl();
+ return resolveBackendBaseUrl();
} catch (error) {
fail(error instanceof Error ? error.message : String(error));
}
@@ -315,6 +315,35 @@ const verifyPyCandidate = async (wheel) => {
return version;
};
+const PY_PROJECT = join(ROOT, 'python', 'pyproject.toml');
+const UV_LOCK = join(ROOT, 'uv.lock');
+const PY_CLIENT_PIN = /^(\s*)"composio-client==[^"]+",$/m;
+
+/**
+ * Drop the exact `composio-client==` pin from the python project.
+ *
+ * A candidate sweep layers the wheel onto every entry with `uv run --with`, but
+ * several entries also install the local `./python` project, whose exact pin the
+ * wheel contradicts — uv then refuses the whole overlay ("no solution found")
+ * and those entries go red for a packaging reason rather than a client one.
+ * Without the pin the wheel is the only `composio-client` in play. Baseline
+ * sweeps never call this, and restorePyBaseline() writes the pre-run files back
+ * either way.
+ */
+const relaxPyClientPin = (toml) => {
+ if (!PY_CLIENT_PIN.test(toml)) {
+ fail(`no exact composio-client pin found in ${relative(ROOT, PY_PROJECT)}`);
+ }
+ return toml.replace(PY_CLIENT_PIN, '$1"composio-client",');
+};
+
+const snapshotPyBaseline = () => snapshotFiles([PY_PROJECT, UV_LOCK]);
+
+const restorePyBaseline = async (snapshot) => {
+ restoreFiles(snapshot);
+ await sh(['uv', 'sync', '--project', 'python', '--frozen']);
+};
+
// --- llm mock ---------------------------------------------------------------
/** Start aimock serving harness/llm-mock/fixtures; resolve once it answers. */
@@ -365,6 +394,7 @@ const cmdSweep = async () => {
mkdirSync(join(runDir, 'traces'), { recursive: true });
let tsBaselineSnapshot;
+ let pyBaselineSnapshot;
let stopLlmMock;
const pyWheel = process.env.COMPOSIO_CLIENT_WHEEL;
try {
@@ -381,6 +411,8 @@ const cmdSweep = async () => {
if (!pyWheel) fail('candidate sweep with py entries needs COMPOSIO_CLIENT_WHEEL');
const v = await verifyPyCandidate(pyWheel);
console.log(`candidate composio-client (py) resolved: ${v}`);
+ pyBaselineSnapshot = snapshotPyBaseline();
+ writeFileSync(PY_PROJECT, relaxPyClientPin(readFileSync(PY_PROJECT, 'utf8')));
for (const e of entries) if (e.lang === 'py') e.pyWith = [...(e.pyWith ?? []), pyWheel];
}
}
@@ -402,6 +434,7 @@ const cmdSweep = async () => {
} finally {
if (stopLlmMock) stopLlmMock();
if (tsBaselineSnapshot) await restoreTsBaseline(tsBaselineSnapshot);
+ if (pyBaselineSnapshot) await restorePyBaseline(pyBaselineSnapshot);
}
};
@@ -438,15 +471,29 @@ const cmdSelftest = async () => {
if (!ok) failures += 1;
};
- // 0. backend safety: the harness accepts only the canonical staging root.
- check('staging backend is accepted', requireStagingBaseUrl() === STAGING_BASE_URL);
- let productionRejected = false;
- try {
- requireStagingBaseUrl('https://backend.composio.dev');
- } catch {
- productionRejected = true;
- }
- check('production backend is rejected', productionRejected);
+ // 0. backend URL: staging is the default, any other bare https root is
+ // honoured as given, and anything structurally unusable is refused.
+ check('staging backend is the default', resolveBackendBaseUrl(undefined) === STAGING_BASE_URL);
+ check(
+ 'a non-staging bare https root is honoured',
+ resolveBackendBaseUrl('https://backend.composio.dev') === 'https://backend.composio.dev'
+ );
+ const refuses = (value) => {
+ try {
+ resolveBackendBaseUrl(value);
+ return false;
+ } catch {
+ return true;
+ }
+ };
+ check('a base URL carrying a path is refused', refuses('https://backend.composio.dev/api/v3'));
+ check('a base URL carrying a query is refused', refuses('https://backend.composio.dev/?x=1'));
+ check(
+ 'a base URL carrying credentials is refused',
+ refuses('https://user:[email protected]')
+ );
+ check('a plaintext base URL is refused', refuses('http://backend.composio.dev'));
+ check('an unparseable base URL is refused', refuses('not-a-url'));
let cleanupRan = false;
let cleanupError;
@@ -471,6 +518,28 @@ const cmdSelftest = async () => {
restoreFiles(cleanupSnapshot);
check('cleanup restores pre-run file contents', readFileSync(cleanupFixture, 'utf8') === 'user contents');
+ // The python candidate swap only resolves once the project's exact
+ // composio-client pin is out of the way; see relaxPyClientPin.
+ const pinnedProject = readFileSync(PY_PROJECT, 'utf8');
+ const relaxedProject = relaxPyClientPin(pinnedProject);
+ check('the python project pins composio-client exactly', /"composio-client==/.test(pinnedProject));
+ check(
+ 'relaxing the pin keeps composio-client a dependency',
+ /^\s*"composio-client",$/m.test(relaxedProject) && !/"composio-client==/.test(relaxedProject)
+ );
+ check(
+ 'relaxing the pin touches nothing else',
+ relaxedProject.replace(/^\s*"composio-client",$/m, '') ===
+ pinnedProject.replace(/^\s*"composio-client==[^"]+",$/m, '')
+ );
+ let unpinnedProjectRejected = false;
+ try {
+ relaxPyClientPin(relaxedProject);
+ } catch {
+ unpinnedProjectRejected = true;
+ }
+ check('relaxing an already-unpinned project is refused', unpinnedProjectRejected);
+
// 1. known-good: error-handling-demo (no backend, no keys).
const good = loadManifest().find((e) => e.id === 'ts/error-handling-demo/index');
const { env: goodEnv } = buildEnv(good, runDir);
diff --git a/harness/staging-backend.mjs b/harness/staging-backend.mjs
deleted file mode 100644
index 966e72469..000000000
--- a/harness/staging-backend.mjs
+++ /dev/null
@@ -1,25 +0,0 @@
-export const STAGING_BASE_URL = 'https://staging-backend.composio.dev';
-
-export const requireStagingBaseUrl = (
- value = process.env.COMPOSIO_BASE_URL ?? STAGING_BASE_URL
-) => {
- let url;
- try {
- url = new URL(value);
- } catch {
- throw new Error(`refusing invalid COMPOSIO_BASE_URL: ${value}`);
- }
-
- const isStagingRoot =
- url.origin === STAGING_BASE_URL &&
- url.pathname === '/' &&
- !url.search &&
- !url.hash &&
- !url.username &&
- !url.password;
- if (!isStagingRoot) {
- throw new Error(`refusing non-staging COMPOSIO_BASE_URL: ${value}`);
- }
-
- return STAGING_BASE_URL;
-};
diff --git a/harness/trace-py/sitecustomize.py b/harness/trace-py/sitecustomize.py
index 634e323ab..8054916d5 100644
--- a/harness/trace-py/sitecustomize.py
+++ b/harness/trace-py/sitecustomize.py
@@ -13,11 +13,15 @@ _TRACE = os.environ.get("COMPOSIO_TRACE_FILE")
_STAGING_BASE_URL = "https://staging-backend.composio.dev"
if _TRACE:
+ # Mirrors harness/backend-url.mjs: COMPOSIO_BASE_URL selects the backend and
+ # defaults to staging. The structural checks stay — a base URL carrying a
+ # path, query, fragment, or embedded credentials means the caller meant
+ # something else, and the host pin below needs a bare root to be meaningful.
_base_url = os.environ.get("COMPOSIO_BASE_URL", _STAGING_BASE_URL)
_parsed_base_url = urlsplit(_base_url)
if not (
_parsed_base_url.scheme == "https"
- and _parsed_base_url.hostname == "staging-backend.composio.dev"
+ and _parsed_base_url.hostname
and _parsed_base_url.path in {"", "/"}
and not _parsed_base_url.query
and not _parsed_base_url.fragment
@@ -25,7 +29,7 @@ if _TRACE:
and not _parsed_base_url.password
):
print(
- f"sitecustomize: refusing non-staging COMPOSIO_BASE_URL: {_base_url}",
+ f"sitecustomize: refusing malformed COMPOSIO_BASE_URL: {_base_url}",
file=sys.stderr,
)
os._exit(78)
diff --git a/harness/trace/register.mjs b/harness/trace/register.mjs
index 7c0e6adc6..457a7f6d1 100644
--- a/harness/trace/register.mjs
+++ b/harness/trace/register.mjs
@@ -1,6 +1,6 @@
// Harness-owned fetch tracer. Injected via NODE_OPTIONS=--import; examples
// must never reference COMPOSIO_TRACE_FILE themselves (lint-enforced).
-import { requireStagingBaseUrl } from '../staging-backend.mjs';
+import { resolveBackendBaseUrl } from '../backend-url.mjs';
const traceFile = process.env.COMPOSIO_TRACE_FILE;
@@ -13,7 +13,7 @@ if (traceFile) {
'generativelanguage.googleapis.com',
]);
- const backendHost = new URL(requireStagingBaseUrl()).hostname;
+ const backendHost = new URL(resolveBackendBaseUrl()).hostname;
const ID_SEG =
/^(ca_|ac_|ti_|tr_|trs_|sess_|auth_|req_|proj_|org_)[\w-]+$|^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$|^\d+$/i;
diff --git a/scripts/examples-provision.mjs b/scripts/examples-provision.mjs
index f62773c0c..7950d7bee 100644
--- a/scripts/examples-provision.mjs
+++ b/scripts/examples-provision.mjs
@@ -30,11 +30,11 @@
// values are ever printed. The API-key demo value stored for serpapi is a
// deliberately fake placeholder, not a real key.
-import { requireStagingBaseUrl } from '../harness/staging-backend.mjs';
+import { resolveBackendBaseUrl } from '../harness/backend-url.mjs';
let BASE_URL;
try {
- BASE_URL = requireStagingBaseUrl();
+ BASE_URL = resolveBackendBaseUrl();
} catch (error) {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
@@ -45,8 +45,10 @@ const INITIATE_MISSING = process.argv.includes('--initiate-missing');
const GC = process.argv.includes('--gc');
const DRY_RUN = process.argv.includes('--dry-run');
-// The examples only ever run against staging with a disposable project key.
-// requireStagingBaseUrl refuses every other host before the key is read or sent.
+// COMPOSIO_BASE_URL selects the backend and defaults to staging;
+// resolveBackendBaseUrl refuses a structurally unusable one before the key is
+// read or sent. Point this at a project whose resources you are willing to have
+// this script create — and, under --gc, delete.
if (!API_KEY) {
console.error('COMPOSIO_API_KEY is required (dedicated examples-project key)');
process.exit(1);