Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

Seed the test memo from trusted CI runs and document it

[._.]/ Adam Eivyb46cccfcab65
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index ce3e12fbb3..2fde6b13fc 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -354,9 +354,25 @@ jobs:
if: needs.impact.outputs.slashdo == 'true'
run: git submodule update --init lib/slashdo
+ # The test memo (docs/GITHUB_ACTIONS.md "Test memo"): the runner skips a
+ # selected test file whose every input — loaded modules, fixtures,
+ # package versions, Node/platform, gating env — is byte-identical to one
+ # a trusted green run verified. Pull requests only read. The scheduled,
+ # dispatched and called full runs save, so the cache holds nothing a PR
+ # branch wrote. Each job and shard keeps its own lineage under the key
+ # prefix; a restore takes the newest.
+ - name: Restore the test memo
+ uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+ with:
+ path: ${{ runner.temp }}/portos-test-memo
+ key: test-memo-${{ runner.os }}-${{ runner.arch }}-server-${{ matrix.shard }}of${{ strategy.job-total }}-${{ github.run_id }}
+ restore-keys: |
+ test-memo-${{ runner.os }}-${{ runner.arch }}-server-${{ matrix.shard }}of${{ strategy.job-total }}-
+
- name: Run server tests
if: needs.impact.outputs.server_mode != 'skip'
env:
+ PORTOS_TEST_MEMO_DIR: ${{ runner.temp }}/portos-test-memo
PORTOS_CI_BROWSER_OWNER: unit
CI_TEST_MODE: ${{ needs.impact.outputs.server_mode }}
CI_TEST_FILES: ${{ needs.impact.outputs.server_files }}
@@ -371,6 +387,15 @@ jobs:
PGPASSWORD: portos
run: node scripts/run-ci-tests.js server
+ # Only trusted full runs on the default/release branches seed the memo;
+ # a red shard still saves the files that passed (entries are per file).
+ - name: Save the test memo
+ if: github.event_name != 'pull_request' && !cancelled()
+ uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+ with:
+ path: ${{ runner.temp }}/portos-test-memo
+ key: test-memo-${{ runner.os }}-${{ runner.arch }}-server-${{ matrix.shard }}of${{ strategy.job-total }}-${{ github.run_id }}
+
# Smoke-boot does not need Postgres: NODE_ENV=test selects the file
# backend. It does need the native rebuild (server boot loads node-pty),
# which is why it lives on this job rather than a third installer.
@@ -456,15 +481,33 @@ jobs:
if: needs.impact.outputs.client_mode != 'skip'
run: bash scripts/ci-warm-test-chrome.sh
+ # Same contract as the server job's memo steps.
+ - name: Restore the test memo
+ if: needs.impact.outputs.client_mode != 'skip'
+ uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+ with:
+ path: ${{ runner.temp }}/portos-test-memo
+ key: test-memo-${{ runner.os }}-${{ runner.arch }}-client-${{ matrix.shard }}of${{ strategy.job-total }}-${{ github.run_id }}
+ restore-keys: |
+ test-memo-${{ runner.os }}-${{ runner.arch }}-client-${{ matrix.shard }}of${{ strategy.job-total }}-
+
- name: Run client tests
if: needs.impact.outputs.client_mode != 'skip'
env:
+ PORTOS_TEST_MEMO_DIR: ${{ runner.temp }}/portos-test-memo
CI_TEST_MODE: ${{ needs.impact.outputs.client_mode }}
CI_TEST_FILES: ${{ needs.impact.outputs.client_files }}
CI_TEST_SOURCES: ${{ needs.impact.outputs.client_sources }}
CI_SHARD: ${{ matrix.shard }}/${{ strategy.job-total }}
run: node scripts/run-ci-tests.js client
+ - name: Save the test memo
+ if: needs.impact.outputs.client_mode != 'skip' && github.event_name != 'pull_request' && !cancelled()
+ uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+ with:
+ path: ${{ runner.temp }}/portos-test-memo
+ key: test-memo-${{ runner.os }}-${{ runner.arch }}-client-${{ matrix.shard }}of${{ strategy.job-total }}-${{ github.run_id }}
+
- name: Build client
if: needs.impact.outputs.build == 'true' && matrix.shard == 1
run: npm run build --prefix client
@@ -846,6 +889,16 @@ jobs:
if: steps.client-modules.outputs.cache-hit != 'true'
run: npm ci --prefix client
+ # Same contract as the server job's memo steps. The browser profile is
+ # part of every key, so a smoke verdict never stands in for a full one.
+ - name: Restore the test memo
+ uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+ with:
+ path: ${{ runner.temp }}/portos-test-memo
+ key: test-memo-${{ runner.os }}-${{ runner.arch }}-browser-${{ github.run_id }}
+ restore-keys: |
+ test-memo-${{ runner.os }}-${{ runner.arch }}-browser-
+
# Fail-closed: PORTOS_REQUIRE_BROWSER_SUITES makes the suites' canRun guard
# throw (see server/lib/browserSuiteGate.js) when Chrome, ffmpeg or the
# client bundler is missing, so a skip can never pass this step.
@@ -854,6 +907,7 @@ jobs:
# this 15m step limit is only a fallback before the 20m job boundary.
timeout-minutes: 15
env:
+ PORTOS_TEST_MEMO_DIR: ${{ runner.temp }}/portos-test-memo
PORTOS_BROWSER_TEST_PROFILE: ${{ github.event_name == 'pull_request' && github.base_ref != 'release' && inputs.full != true && 'smoke' || 'full' }}
PORTOS_CI_DIAGNOSTICS_DIR: ${{ runner.temp }}/portos-ci-diagnostics
CI_TEST_MODE: files
@@ -862,6 +916,13 @@ jobs:
PORTOS_TEST_QUIET: 1
run: node scripts/watch-ci-tests.js server
+ - name: Save the test memo
+ if: github.event_name != 'pull_request' && !cancelled()
+ uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+ with:
+ path: ${{ runner.temp }}/portos-test-memo
+ key: test-memo-${{ runner.os }}-${{ runner.arch }}-browser-${{ github.run_id }}
+
- name: Preserve browser test failure diagnostics
if: failure()
continue-on-error: true
@@ -987,12 +1048,23 @@ jobs:
- name: Check server entry-point syntax
run: node --check server/index.js
+ # Same contract as the server job's memo steps; the platform is part of
+ # every key, so Windows reads only Windows verdicts.
+ - name: Restore the test memo
+ uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+ with:
+ path: ${{ runner.temp }}/portos-test-memo
+ key: test-memo-${{ runner.os }}-${{ runner.arch }}-windows-server-${{ matrix.shard }}of${{ strategy.job-total }}-${{ github.run_id }}
+ restore-keys: |
+ test-memo-${{ runner.os }}-${{ runner.arch }}-windows-server-${{ matrix.shard }}of${{ strategy.job-total }}-
+
- name: Run server tests on Windows
# A step timeout FAILS the step (log kept, `if: failure()` reporting
# runs); the job-level timeout would cancel the job and lose the log
# (#10415). Observed healthy runs take 2-10 minutes.
timeout-minutes: 15
env:
+ PORTOS_TEST_MEMO_DIR: ${{ runner.temp }}/portos-test-memo
# Outside the launcher's disposable temp root; recoverable after exit.
PORTOS_CI_DIAGNOSTICS_DIR: ${{ runner.temp }}/portos-ci-diagnostics
CI_TEST_MODE: ${{ needs.impact.outputs.windows_mode }}
@@ -1006,6 +1078,13 @@ jobs:
PORTOS_CI_MAX_WORKERS: 2
run: node scripts/run-ci-tests.js server
+ - name: Save the test memo
+ if: github.event_name != 'pull_request' && !cancelled()
+ uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+ with:
+ path: ${{ runner.temp }}/portos-test-memo
+ key: test-memo-${{ runner.os }}-${{ runner.arch }}-windows-server-${{ matrix.shard }}of${{ strategy.job-total }}-${{ github.run_id }}
+
- name: Preserve Windows test failure diagnostics
if: failure()
continue-on-error: true
diff --git a/AGENTS.md b/AGENTS.md
index 49e96fda15..8ceb143815 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -56,6 +56,8 @@ npm test --prefix client # Vitest (happy-dom) — component/unit tests
npm run test:db # DB-backed suites → portos_test ONLY (see Security Model)
npm run pregate # BEFORE EVERY PUSH — CI's plan run locally; receipt in .cache/pregate/report.json
+# pregate and CI skip test files whose every input already passed (the test
+# memo, docs/GITHUB_ACTIONS.md "Test memo"); PORTOS_TEST_MEMO=off runs them all.
```
**Run `npm run pregate` before every push and again after rebasing onto a moved base.** It uses CI's planner and runners, including tree-wide guards for import boundaries, catalog merges, and generated manifests. The local plan includes both committed and uncommitted changes, including untracked files.
diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md
index 0d94a808b4..c86c91831a 100644
--- a/docs/CONTRIBUTING.md
+++ b/docs/CONTRIBUTING.md
@@ -253,8 +253,18 @@ For server watch mode, run this alternative from the repository root:
npm run test:watch --prefix server
```
+`npm run pregate` (and CI, through the same runner) also consults the test
+memo: a selected test file whose every input — the modules it loads, the
+fixtures and package versions it reaches, the Node major and platform, and
+the gating environment variables — is byte-identical to a run that already
+passed is skipped, and new green results are recorded under
+`~/.portos/test-memo/`. Tree-wide scanners and anything that reads repository
+state are never memoized. `PORTOS_TEST_MEMO=off` disables it, and
+`PORTOS_TEST_MEMO_VERBOSE=1` lists the skipped files; see
+[GITHUB_ACTIONS.md](./GITHUB_ACTIONS.md#test-memo).
+
Pull requests into `main` run the tests for affected feature directories, with
-conservative fallbacks to Vitest related-test mode or the complete suite. The
+conservative fallbacks to the runner's related-test mode or the complete suite. The
pull request into `release`, nightly CI, and manual CI runs always run the
complete server, client, DB, lint, build, and smoke checks. Pushes to `main`
run nothing — the pull request gate already covered that tree. Release
diff --git a/docs/GITHUB_ACTIONS.md b/docs/GITHUB_ACTIONS.md
index b1b3114473..79e4697583 100644
--- a/docs/GITHUB_ACTIONS.md
+++ b/docs/GITHUB_ACTIONS.md
@@ -65,8 +65,8 @@ A release therefore pays for one full run (on its PR), not three.
PRs into `main` use `scripts/ci-test-plan.js` to classify the changed files
before installing dependencies. Directory-scoped features add their server and
-client feature tests, and every behavioral source (nested or flat) uses
-Vitest's import-graph-aware `related` mode, fed the changed behavioral source
+client feature tests, and every behavioral source (nested or flat) uses the
+runner's import-graph-aware `related` mode, fed the changed behavioral source
paths plus the planner's explicit test files. The planner deliberately chooses full CI for shared
composition roots, test configuration, dependency manifests, workflow changes,
unknown artifacts, or wide diffs.
@@ -452,8 +452,9 @@ pull request is the merge ref plus both of its parents — and the
first parent *is* the base-branch commit the pull request is diffed against.
`scripts/ci-base-sha.js` reads it (`HEAD^1`) and exports `CI_BASE_SHA` for the
rest of the job, so the planner's `git diff <base>...HEAD` resolves without
-deeper history. The planner passes the resulting source paths directly to
-`vitest related`; Vitest no longer performs its own Git diff.
+deeper history. The planner passes the resulting source paths to the runner,
+which resolves them through its own module graph; Vitest no longer performs
+its own Git diff.
Reading the base off the checkout rather than `github.event.pull_request.base.sha`
is also more correct: GitHub rebuilds the merge ref when the base branch moves,
@@ -771,14 +772,89 @@ failing closed to the full suite for a script nothing names. A `.py` outside
`scripts/` is still unclassified.
Targeted `files` plans run the planner's exact test files once. `related` plans
-run `vitest related` once with changed behavioral source paths and the cheap
-structural/repository contract files as inputs. Vitest treats a test-file input
-as directly selected, so contracts and changed tests share the import-graph run
-without being repeated in a second process.
+resolve the changed behavioral source paths through the runner's own module
+graph (`scripts/lib/testMemo.js`: static, dynamic and `vi.mock` edges, the
+same edges `vitest related` follows): a test is selected when a changed
+source is in its closure or when one of its edges cannot be resolved, plus
+the cheap structural/repository contract files, in one Vitest invocation.
+The exact list reaches Vitest through a selection file
+(`PORTOS_CI_TEST_SELECTION`, read by both workspace configs) rather than
+command-line selectors, because a related set can exceed the Windows
+command-line limit. Computing the set in the runner is what lets the test
+memo below subtract the files a trusted green run already verified.
There is no buffered discovery pass: the old `vitest list --changed` path could
spend minutes printing every test name, overflow Node's buffer, discard the
result, and rerun the same graph.
+### Test memo
+
+`scripts/run-ci-tests.js` skips a selected test file whose every input is
+byte-identical to one a trusted green run already verified, locally under
+`npm run pregate` and in CI through the same code path. Each selected file
+gets two content-addressed keys (`scripts/lib/testMemo.js`):
+
+- the **floor** key hashes the working-tree bytes of the file's static +
+ dynamic module closure (`import`/`export … from`, `require()`, `import()`,
+ `vi.mock`/`vi.doMock`/`vi.importActual`/`vi.importMock`/`vi.unmock`, all
+ with literal specifiers);
+- the **recorded** key hashes the modules the file actually loaded in the
+ green run that produced its entry (Vitest's
+ `experimental.recordDependencies` depfile). A module the test mocks away or
+ never reaches cannot change its outcome, so a change to the lazily
+ imported server cluster invalidates only the suites that loaded it; a new
+ edge still invalidates because the importer's own bytes changed.
+
+Both keys also cover: every non-JS file a member names through a literal path
+anchored at `import.meta.url`, `__dirname`, `process.cwd()` or an identifier
+bound to one of those (a literal directory contributes its whole tree, up to
+400 files; snapshot files and the planner's `FIXTURE_OWNERS` directories are
+added for the test itself); the installed version of every package the
+members reach (so a lockfile bump invalidates exactly the suites that reach
+the bumped packages); the workspace Vitest config closure, setup files,
+`scripts/vitestCiPool.js`, `package.json` and lockfile; the Node major,
+`process.platform`, `process.arch`; and `TEST_MEMO_ENV` (`CI`, `NODE_ENV`,
+`PORTOS_BROWSER_TEST_PROFILE`, `PORTOS_CI_BROWSER_OWNER`,
+`PORTOS_REQUIRE_BROWSER_SUITES`, `PORTOS_REQUIRE_LINUX_CONTAINMENT`,
+`PORTOS_TEST_QUIET`, `CI_EXPECT_SLASHDO_SUBMODULE`, …).
+
+**Never memoized (fails closed).** A file in the planner's `ALWAYS_RUN_TESTS`
+or in `TEST_MEMO_NEVER` (the tree-wide convention and catalog scanners); a
+file whose closure contains the `// test-memo: never` marker; a test file or
+`scripts/` module that spawns `git` (repository state is not in the key —
+product modules that run git on their own targets are under the test's
+control); a member that reads a path the analyzer cannot resolve
+(`import.meta.glob`, anything under `.git/`, a literal directory with more
+than 400 files); a bare specifier that resolves to no installed package; and,
+for the floor key only, a non-literal `import()`/`require()` or an
+unparseable member. A recorded set that reaches outside the repository is
+not memoizable either. Every unknown construct lands in one of those buckets
+rather than silently narrowing the key.
+
+**Store.** `~/.portos/test-memo/` (override with `PORTOS_TEST_MEMO_DIR`):
+`entries/<key[0..2]>/<key>.json` holds `{ test, workspace, verifiedAt,
+source: 'local'|'ci', commit }`, `records/<workspace>/<sha256(test)>.json`
+the loaded-module list. Writes are temp-file-plus-rename so concurrent
+worktree pregates never tear a file; any read error is a miss. Entries are
+written only for files Vitest's JSON report shows passed with no failed and
+no skipped case (a browser profile's name pattern is the one accepted skip
+reason). The runner prints one line —
+`🧠 Test memo: skipped N of M selected files (closure unchanged); running K.`
+— and lists the skipped files under `PORTOS_TEST_MEMO_VERBOSE=1`. When every
+selected file is memoized nothing is launched and the job exits 0.
+
+**CI seeding.** The server, client, browser and Windows jobs restore
+`${{ runner.temp }}/portos-test-memo` with `actions/cache/restore` under a
+per-job, per-shard key prefix (`test-memo-<os>-<arch>-<job>-<shard>of<n>-`).
+Pull requests only read. The scheduled nightly, `workflow_dispatch` and
+`workflow_call` runs on the default/release branches save a new entry keyed
+by run id, so the cache holds nothing a pull-request branch wrote and every
+PR restores the newest trusted lineage. The DB job does not go through
+`run-ci-tests.js` and is not memoized.
+
+**Disable.** `PORTOS_TEST_MEMO=off` turns reads and writes off together and
+restores the pre-memo behavior. A test spawned under Vitest (`VITEST` set)
+never touches the developer's store unless it sets `PORTOS_TEST_MEMO_DIR`.
+
No third-party change-filter action is used. The planner passes test paths as a
JSON argument array to `spawnSync`, never through shell interpolation.