Skip to content

armgit / Ix

Public Imported from GitHub
0

Understand any codebase instantly. System intelligence for codebases, built for humans and AI.

ci(e2e): run the correctness harness on every PR against a pinned backend

KageBinarye3078be6da83
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
index 65000dae..ad9753c9 100644
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -8,6 +8,14 @@ updates:
actions:
patterns: ["*"]
+ # The backend image the real-backend harness runs against, pinned by digest
+ # in ix-cli/test/e2e/compose.e2e.yml. A bump PR runs the scenarios against the
+ # new release before any CLI PR sees it.
+ - package-ecosystem: docker-compose
+ directory: /ix-cli/test/e2e
+ schedule:
+ interval: weekly
+
- package-ecosystem: npm
directory: /ix-cli
schedule:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index af95550d..2729e71a 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -295,9 +295,36 @@ jobs:
echo "::endgroup::"
node -e "const d=require('./doctor-unmapped.json');const c=(d.checks||[]).find(c=>c.name==='Workspace for this directory');if(!c){console.error('no workspace check in doctor output');process.exit(1)}if(c.ok||d.healthy){console.error('doctor reported healthy in a directory with no workspace');process.exit(1)}console.log('unmapped cwd reported:',c.detail)"
+ # The correctness harness (ix-cli/test/e2e): edit sequences against a
+ # backend pinned by digest in compose.e2e.yml, not the :latest the smoke
+ # step uses, so a backend release cannot turn an unrelated CLI PR red.
+ # Its own ports (8290/8729), next to the smoke stack. Scenarios that fail
+ # until a fix lands are marked failsUntil([...]) and pass while they fail.
+ - name: Start harness backend (pinned)
+ working-directory: ix-cli
+ run: npm run e2e:up
+
+ - name: Correctness scenarios (incremental equals fresh)
+ working-directory: ix-cli
+ env:
+ IX_E2E: "1"
+ IX_E2E_DIFF_DIR: ${{ runner.temp }}/e2e-diffs
+ # dist/ is already built above; test:e2e would rebuild it.
+ run: npx vitest run --config vitest.e2e.config.ts
+
+ - name: Upload graph diffs on failure
+ if: failure()
+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+ with:
+ name: e2e-graph-diffs
+ path: ${{ runner.temp }}/e2e-diffs
+ if-no-files-found: ignore
+
- name: Backend logs on failure
if: failure()
- run: docker compose -f docker-compose.standalone.yml logs --tail 200
+ run: |
+ docker compose -f docker-compose.standalone.yml logs --tail 200
+ (cd ix-cli && node test/e2e/stack.mjs logs)
# ── API-reference parity gate ────────────────────────────────────────────
# docs/api/openapi.yaml claims to be generated from the client source; this
diff --git a/ix-cli/package.json b/ix-cli/package.json
index 40a57654..734bd051 100644
--- a/ix-cli/package.json
+++ b/ix-cli/package.json
@@ -21,8 +21,8 @@
"test:unit": "vitest run",
"test:coverage": "vitest run --coverage",
"test:e2e": "npm run build && vitest run --config vitest.e2e.config.ts",
- "e2e:up": "docker compose -f test/e2e/compose.e2e.yml up -d --wait --wait-timeout 240",
- "e2e:down": "docker compose -f test/e2e/compose.e2e.yml down -v",
+ "e2e:up": "node test/e2e/stack.mjs up",
+ "e2e:down": "node test/e2e/stack.mjs down",
"smoke": "node test/parser.smoke.mjs",
"typecheck": "tsc --noEmit",
"lint": "eslint src",
diff --git a/ix-cli/test/e2e/README.md b/ix-cli/test/e2e/README.md
index 23820d2a..02ab2231 100644
--- a/ix-cli/test/e2e/README.md
+++ b/ix-cli/test/e2e/README.md
@@ -36,7 +36,7 @@ endpoint that is not loopback or that uses a port of a stack with real data:
| `IX_E2E_PROJECT` | `ix-e2e` | compose project name |
| `IX_E2E_BACKEND_PORT` | `8290` | compose and harness: host port of the memory-layer |
| `IX_E2E_ARANGO_PORT` | `8729` | compose and harness: host port of ArangoDB |
-| `IX_E2E_BACKEND_IMAGE` | `ghcr.io/ix-infrastructure/ix-memory-layer:1.0.31@sha256:7f972fb0…` | compose: the backend under test |
+| `IX_E2E_BACKEND_IMAGE` | the pin in `compose.e2e.yml` (1.0.31 by digest) | `stack.mjs`: run this image instead |
| `IX_E2E_ENDPOINT` | `http://127.0.0.1:$IX_E2E_BACKEND_PORT` | harness: backend URL, overrides the port |
| `IX_E2E_ARANGO_URL` | `http://127.0.0.1:$IX_E2E_ARANGO_PORT` | harness and `graph-sig.mjs`: ArangoDB URL |
| `IX_E2E_DIFF_DIR` | unset | harness: write every graph difference to `<dir>/<scenario>.txt` |
@@ -44,6 +44,11 @@ endpoint that is not loopback or that uses a port of a stack with real data:
Set the same `IX_E2E_*` values for `e2e:up`, `test:e2e` and `e2e:down`.
+`e2e:up` and `e2e:down` run `stack.mjs`, which calls `docker compose` on
+`compose.e2e.yml` and adds `compose.image-override.yml` when
+`IX_E2E_BACKEND_IMAGE` is set. The pin itself is a literal line so Dependabot
+can propose bumps (`.github/dependabot.yml`, ecosystem `docker-compose`).
+
### Several stacks at once
Each stack needs its own project name and ports. One stack is capped at about
@@ -108,6 +113,14 @@ crashed `ix map` or an unreachable database, fails the run either way.
To see what a marked scenario still gets wrong, set `IX_E2E_DIFF_DIR`.
+## In CI
+
+Job `e2e` in `.github/workflows/ci.yml` runs the scenarios on every PR after
+the smoke test, against the pinned image on the default ports, and is part of
+`CI Passed`. On failure it uploads the graph diffs (`e2e-graph-diffs`) and both
+stacks' logs. A backend fix reaches this job only once its image is released
+and the pin is moved to it, normally by the Dependabot PR.
+
## Fixtures
`fixtures/polyglot/` has 23 files: Python with cross-file calls (`app/`),
diff --git a/ix-cli/test/e2e/compose.e2e.yml b/ix-cli/test/e2e/compose.e2e.yml
index 15268d38..08e6a7dd 100644
--- a/ix-cli/test/e2e/compose.e2e.yml
+++ b/ix-cli/test/e2e/compose.e2e.yml
@@ -3,11 +3,12 @@
# Throwaway backend for the real-backend harness (test/e2e). Every scenario
# resets this database, so it gets its own compose project and ports and must
# never be pointed at a stack anything else uses. See test/e2e/README.md.
+# Start and stop it with `npm run e2e:up` / `npm run e2e:down` (stack.mjs).
#
# IX_E2E_PROJECT compose project name (default ix-e2e)
# IX_E2E_BACKEND_PORT host port of memory-layer (default 8290)
# IX_E2E_ARANGO_PORT host port of ArangoDB (default 8729)
-# IX_E2E_BACKEND_IMAGE memory-layer image (default: v1.0.31, pinned by digest)
+# IX_E2E_BACKEND_IMAGE memory-layer image (default: the pin below; see stack.mjs)
#
# Memory is capped: ArangoDB's caches otherwise size themselves from host RAM
# (a 9 GB block cache on a 30 GB host). The whole stack stays under ~2.7 GB.
@@ -47,7 +48,9 @@ services:
retries: 15
memory-layer:
- image: ${IX_E2E_BACKEND_IMAGE:-ghcr.io/ix-infrastructure/ix-memory-layer:1.0.31@sha256:7f972fb084d9667ef1b7cd7e7c40b24aad76fb4cb38d1cc07a0319444c0ab979}
+ # Literal, so Dependabot can propose bumps. Set IX_E2E_BACKEND_IMAGE to run
+ # another image (stack.mjs then adds compose.image-override.yml).
+ image: ghcr.io/ix-infrastructure/ix-memory-layer:1.0.31@sha256:7f972fb084d9667ef1b7cd7e7c40b24aad76fb4cb38d1cc07a0319444c0ab979
networks: [backend]
ports:
- "127.0.0.1:${IX_E2E_BACKEND_PORT:-8290}:8090"
diff --git a/ix-cli/test/e2e/compose.image-override.yml b/ix-cli/test/e2e/compose.image-override.yml
new file mode 100644
index 00000000..68fbf5d2
--- /dev/null
+++ b/ix-cli/test/e2e/compose.image-override.yml
@@ -0,0 +1,9 @@
+# Copyright 2026 Ix Infrastructure Inc.
+#
+# Added by stack.mjs only when IX_E2E_BACKEND_IMAGE is set: runs that image
+# instead of the pinned one in compose.e2e.yml (for example a backend built
+# from an Ix-memory branch).
+
+services:
+ memory-layer:
+ image: ${IX_E2E_BACKEND_IMAGE:?set IX_E2E_BACKEND_IMAGE}
diff --git a/ix-cli/test/e2e/stack.mjs b/ix-cli/test/e2e/stack.mjs
new file mode 100644
index 00000000..03122d69
--- /dev/null
+++ b/ix-cli/test/e2e/stack.mjs
@@ -0,0 +1,30 @@
+// Copyright 2026 Ix Infrastructure Inc.
+
+// Starts or stops the harness stack: `node test/e2e/stack.mjs up|down|logs`.
+//
+// compose.e2e.yml pins the backend image as a literal line so Dependabot can
+// propose bumps. To test another image, set IX_E2E_BACKEND_IMAGE: this script
+// then adds compose.image-override.yml, which swaps it in.
+
+/* global process */
+
+import { spawnSync } from "node:child_process";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+const here = dirname(fileURLToPath(import.meta.url));
+const files = ["-f", join(here, "compose.e2e.yml")];
+if (process.env.IX_E2E_BACKEND_IMAGE) files.push("-f", join(here, "compose.image-override.yml"));
+
+const actions = {
+ up: ["up", "-d", "--wait", "--wait-timeout", "240"],
+ down: ["down", "-v"],
+ logs: ["logs", "--tail", "200"],
+};
+const action = actions[process.argv[2]];
+if (!action) {
+ process.stderr.write("usage: node test/e2e/stack.mjs up|down|logs\n");
+ process.exit(2);
+}
+const res = spawnSync("docker", ["compose", ...files, ...action], { stdio: "inherit" });
+process.exit(res.status ?? 1);