Skip to content

Local code intelligence for developers and AI agents. Connect code, docs and infrastructure in a persistent graph, with CBM Lens, MCP tools, agent integrations and change-aware test selection.

ci: report-only test-impact shadow job on PRs

Martin Vogela8d5a8154f9d
diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml
index 0fd09b878..d6d966912 100644
--- a/.github/workflows/pr.yml
+++ b/.github/workflows/pr.yml
@@ -212,6 +212,49 @@ jobs:
with:
mode: pr
+ # ── Test-impact SHADOW, REPORT-ONLY: predicts from the code graph which C
+ # suites this change can affect (the latest released binary indexes the
+ # PR checkout; detect_changes at depth 15 against the base, mapped to
+ # suites, with the selector's run-all triggers) and uploads the
+ # prediction, so graph-based selection can be judged against what really
+ # failed before anything gates on it. continue-on-error and NOT in
+ # ci-ok's needs: it can never block or pass a merge. ──
+ test-impact-shadow:
+ runs-on: ubuntu-latest
+ timeout-minutes: 20
+ continue-on-error: true
+ steps:
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ with:
+ # The PR merge commit and its base parent: the diff to predict for.
+ fetch-depth: 2
+ persist-credentials: false
+ - name: Fetch the latest released binary
+ id: release
+ continue-on-error: true
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: scripts/ci/test-impact-shadow.sh --fetch-release "$RUNNER_TEMP/cbm-bin"
+ - name: Build the binary from this checkout (download failed)
+ if: ${{ steps.release.outcome == 'failure' }}
+ run: |
+ sudo apt-get update && sudo apt-get install -y zlib1g-dev
+ scripts/build.sh CC=gcc CXX=g++
+ mkdir -p "$RUNNER_TEMP/cbm-bin"
+ cp build/c/codebase-memory-mcp "$RUNNER_TEMP/cbm-bin/"
+ - name: Predict the impacted suites
+ run: scripts/ci/test-impact-shadow.sh --binary "$RUNNER_TEMP/cbm-bin/codebase-memory-mcp" --out "$RUNNER_TEMP/test-impact"
+ - name: Upload the prediction
+ if: ${{ always() }}
+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+ with:
+ name: test-impact-prediction
+ path: |
+ ${{ runner.temp }}/test-impact/prediction.json
+ ${{ runner.temp }}/test-impact/selection.json
+ ${{ runner.temp }}/test-impact/changed-files.txt
+ if-no-files-found: warn
+
# ── Docs-only PRs (tier T0a) select no test leg, so this runs the static
# contract steps of the canonical test leg on their own (they police
# docs surfaces too: language counts, version metadata). Every test leg
diff --git a/scripts/ci/README.md b/scripts/ci/README.md
index 3e74b86d7..f38cf36a8 100644
--- a/scripts/ci/README.md
+++ b/scripts/ci/README.md
@@ -15,6 +15,7 @@ CI and the local infrastructure — both of which the venue-parity contract
| `generate-sbom.py` | The release SPDX SBOM (vendored versions reviewable here, diffable by vendoring PRs — was inline YAML). | `release.yml` |
| `require-all-green.sh` | The aggregate gate: fail unless every needed job succeeded or legitimately skipped (was inline YAML) — and, under lane selection, every SELECTED lane ran and succeeded (skipped/cancelled/missing = red; unselected lanes may skip). | `pr.yml ci-ok` |
| `select-lanes.sh` | Decide which PR lanes a change set runs (tier, lanes, full, reasons) from its file list — the R1 rule set, fail-safe to every lane. Pinned by `tests/test_select_lanes.sh` and the September 2026 history replay (`scripts/test-impact/replay-selector.sh`). | `pr.yml changes` |
+| `test-impact-shadow.sh` | The report-only test-impact shadow: fetch the latest released binary (checksum-verified) or take a given one, index this checkout into a throwaway cache, run `detect_changes` against the base and map it to suites (`scripts/test-impact/predict.py`). Never gates. | `pr.yml test-impact-shadow` |
| `verify-shard-union.sh` | Prove sharded test legs lost nothing: shard count agreement, indices 1..n, identical suite lists, union of slices == full list (was inline YAML). No manifests is green only when the lane selection holds no test leg. | `_test.yml` shard-completeness |
| `prepare-release-candidates.sh` | Copy one linker output into stripped/unstripped candidates, finalize signatures, composition-check them without execution, and record their hashes. | `_build.yml`, local artifact smoke |
| `stage-release-candidates.py` | Admit exactly eight candidate artifacts / sixteen byte-distinct binaries into the content-addressed VirusTotal scan set. | `_build.yml` |
diff --git a/scripts/ci/test-impact-shadow.sh b/scripts/ci/test-impact-shadow.sh
new file mode 100755
index 000000000..a6f6f369a
--- /dev/null
+++ b/scripts/ci/test-impact-shadow.sh
@@ -0,0 +1,168 @@
+#!/usr/bin/env bash
+# test-impact-shadow.sh — the PR test-impact SHADOW: predict from the code graph
+# which C test suites a change can affect, and publish the prediction. It is
+# REPORT-ONLY: pr.yml runs it continue-on-error and ci-ok does not need it, so
+# it can never block or pass a merge. Its predictions accumulate next to the
+# lanes that actually ran and failed, which is how graph-based selection gets
+# evaluated before anyone gates on it.
+#
+# The binary is the latest PUBLISHED release, not this checkout's build: a few
+# seconds of download instead of minutes of compile, and a PR that breaks the
+# indexer cannot corrupt its own prediction. pr.yml builds the checkout only
+# if the download fails.
+#
+# Usage: scripts/ci/test-impact-shadow.sh --fetch-release DIR [--repo OWNER/REPO]
+# Download the latest release's archive for this host + checksums.txt
+# into DIR, verify the SHA-256, extract DIR/codebase-memory-mcp.
+# Needs GH_TOKEN (read-only); $GH overrides the gh command.
+# scripts/ci/test-impact-shadow.sh --binary PATH --out DIR [--base REV] [--depth N]
+# Index this checkout into a throwaway cache (HOME, XDG dirs and
+# CBM_CACHE_DIR under DIR/work, a private CBM_RUNTIME_DIR under /tmp:
+# the daemon's socket path must fit sun_path), run
+# detect_changes against REV (default HEAD^1, the base side of a PR
+# merge commit) at depth N (default 15), and let
+# scripts/test-impact/predict.py map it to suites with the lane
+# selector's run-all triggers. Writes DIR/prediction.json and
+# DIR/summary.md (appended to $GITHUB_STEP_SUMMARY when set). The
+# graph is only consulted when C sources changed.
+# Exit: 0 = prediction written · 1 = the tooling broke · 2 = usage error.
+set -euo pipefail
+
+usage() { sed -n '2,28p' "$0" | sed 's/^# \{0,1\}//'; }
+
+FETCH_DIR=""
+REPO="${GITHUB_REPOSITORY:-DeusData/codebase-memory-mcp}"
+BINARY=""
+OUT=""
+BASE="HEAD^1"
+DEPTH=15
+while [ $# -gt 0 ]; do
+ case "$1" in
+ -h | --help)
+ usage
+ exit 0
+ ;;
+ --fetch-release | --repo | --binary | --out | --base | --depth)
+ if [ $# -lt 2 ]; then
+ echo "test-impact-shadow.sh: $1 needs a value. Please consult --help." >&2
+ exit 2
+ fi
+ case "$1" in
+ --fetch-release) FETCH_DIR="$2" ;;
+ --repo) REPO="$2" ;;
+ --binary) BINARY="$2" ;;
+ --out) OUT="$2" ;;
+ --base) BASE="$2" ;;
+ --depth) DEPTH="$2" ;;
+ esac
+ shift
+ ;;
+ *)
+ echo "test-impact-shadow.sh: unknown argument '$1'. Please consult --help." >&2
+ exit 2
+ ;;
+ esac
+ shift
+done
+
+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+
+sha256_check() { # file with "<sha> <name>" lines, run in the archive's dir
+ if command -v sha256sum >/dev/null 2>&1; then
+ sha256sum -c "$1"
+ else
+ shasum -a 256 -c "$1"
+ fi
+}
+
+if [ -n "$FETCH_DIR" ]; then
+ case "$(uname -s)/$(uname -m)" in
+ Linux/x86_64) asset=codebase-memory-mcp-linux-amd64.tar.gz ;;
+ Linux/aarch64 | Linux/arm64) asset=codebase-memory-mcp-linux-arm64.tar.gz ;;
+ Darwin/arm64) asset=codebase-memory-mcp-darwin-arm64.tar.gz ;;
+ Darwin/x86_64) asset=codebase-memory-mcp-darwin-amd64.tar.gz ;;
+ *)
+ echo "test-impact-shadow.sh: no release archive for $(uname -s)/$(uname -m)" >&2
+ exit 1
+ ;;
+ esac
+ read -r -a gh_cmd <<<"${GH:-gh}"
+ mkdir -p "$FETCH_DIR"
+ # No tag: the latest release.
+ "${gh_cmd[@]}" release download --repo "$REPO" --pattern "$asset" \
+ --pattern checksums.txt --dir "$FETCH_DIR" --clobber
+ grep " $asset\$" "$FETCH_DIR/checksums.txt" >"$FETCH_DIR/asset.sha256"
+ (cd "$FETCH_DIR" && sha256_check asset.sha256)
+ tar -xzf "$FETCH_DIR/$asset" -C "$FETCH_DIR" codebase-memory-mcp
+ echo "release binary: $FETCH_DIR/codebase-memory-mcp ($asset, SHA-256 verified)"
+ exit 0
+fi
+
+if [ -z "$BINARY" ] || [ -z "$OUT" ]; then
+ echo "test-impact-shadow.sh: --binary and --out are required. Please consult --help." >&2
+ exit 2
+fi
+case "$DEPTH" in
+'' | *[!0-9]*)
+ echo "test-impact-shadow.sh: --depth must be a number. Please consult --help." >&2
+ exit 2
+ ;;
+esac
+BINARY="$(cd "$(dirname "$BINARY")" && pwd)/$(basename "$BINARY")"
+PREDICT="$ROOT/scripts/test-impact/predict.py"
+mkdir -p "$OUT"
+OUT="$(cd "$OUT" && pwd)"
+WORK="$OUT/work"
+rm -rf "$WORK"
+mkdir -p "$WORK/home/.config" "$WORK/home/.cache" "$WORK/cache"
+# <runtime>/cbm-daemon-<uid>/cbm-<key>.sock adds ~42 bytes to this path, and
+# sun_path holds 104 (macOS) / 108: a deep $OUT or $TMPDIR does not fit.
+RUNTIME=$(mktemp -d /tmp/cbm-rt.XXXXXX)
+trap 'rm -rf "$RUNTIME"' EXIT
+
+BASE_SHA=$(git -C "$ROOT" rev-parse --verify "$BASE^{commit}")
+git -C "$ROOT" diff --name-only --no-renames "$BASE_SHA" HEAD >"$OUT/changed-files.txt"
+bash "$ROOT/scripts/ci/select-lanes.sh" "$OUT/changed-files.txt" >"$OUT/selection.json"
+
+# From here on nothing may touch the host's real config, cache or daemon.
+export HOME="$WORK/home"
+export XDG_CONFIG_HOME="$WORK/home/.config"
+export XDG_CACHE_HOME="$WORK/home/.cache"
+export CBM_CACHE_DIR="$WORK/cache"
+export CBM_RUNTIME_DIR="$RUNTIME"
+
+VERSION=$("$BINARY" --version 2>/dev/null | head -1 || true)
+DETECT_ARGS=()
+T_INDEX="-"
+T_DETECT="-"
+if [ "$(python3 "$PREDICT" --needs-graph --changed "$OUT/changed-files.txt")" = yes ]; then
+ python3 -c 'import json, sys; json.dump({"repo_path": sys.argv[1]}, open(sys.argv[2], "w"))' \
+ "$ROOT" "$WORK/index-args.json"
+ t0=$(date +%s)
+ "$BINARY" cli --quiet index_repository --args-file "$WORK/index-args.json" >"$WORK/index.json"
+ t1=$(date +%s)
+ python3 -c '
+import json, sys
+project = json.load(open(sys.argv[1]))["project"]
+json.dump({"project": project, "base_branch": sys.argv[2], "depth": int(sys.argv[3]),
+ "limit": 5000, "changed_limit": 5000, "module_limit": 0,
+ "max_output_tokens": 1000000, "format": "json"}, open(sys.argv[4], "w"))
+' "$WORK/index.json" "$BASE_SHA" "$DEPTH" "$WORK/detect-args.json"
+ "$BINARY" cli --quiet detect_changes --args-file "$WORK/detect-args.json" >"$WORK/detect.json"
+ t2=$(date +%s)
+ # The CLI started a private daemon under $RUNTIME; stop it (best effort).
+ "$BINARY" daemon stop >/dev/null 2>&1 || true
+ T_INDEX="$((t1 - t0))s"
+ T_DETECT="$((t2 - t1))s"
+ DETECT_ARGS=(--detect "$WORK/detect.json")
+fi
+
+META=$(python3 -c 'import json, sys; print(json.dumps(dict(zip(sys.argv[1::2], sys.argv[2::2]))))' \
+ binary "${VERSION:-unknown}" base "$BASE_SHA" "index time" "$T_INDEX" "detect time" "$T_DETECT")
+python3 "$PREDICT" --root "$ROOT" --changed "$OUT/changed-files.txt" \
+ --selection "$OUT/selection.json" ${DETECT_ARGS[@]+"${DETECT_ARGS[@]}"} --depth "$DEPTH" \
+ --meta "$META" --out "$OUT/prediction.json" --summary "$OUT/summary.md"
+cat "$OUT/summary.md"
+if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+ cat "$OUT/summary.md" >>"$GITHUB_STEP_SUMMARY"
+fi
diff --git a/scripts/test-impact/predict.py b/scripts/test-impact/predict.py
new file mode 100644
index 000000000..7658ea2b2
--- /dev/null
+++ b/scripts/test-impact/predict.py
@@ -0,0 +1,232 @@
+#!/usr/bin/env python3
+"""predict.py — map a detect_changes impact set to the C test suites it can
+affect: the PR test-impact SHADOW (scripts/ci/test-impact-shadow.sh).
+Report-only; nothing here decides a merge.
+
+The mapping is the one the September 2026 test-impact replay measured
+(detect_changes at depth 15 plus rule fallbacks, "dcF@15"):
+ - impacted test FUNCTIONS (graph rows in tests/*.c) map to the suites that
+ RUN_TEST them, resolved in their own file first; a nested suite (RUN_SUITE
+ inside another) resolves to the registered suite that runs it; only suites
+ registered in tests/test_main.c count, perf suites excluded;
+ - a changed tests/*.c file always selects its whole suite;
+ - a changed fixture (tests/fixtures/<key>/...) selects the suites whose test
+ files name <key>; a changed grammar selects the grammar suites;
+ - run-all: the lane selector's full, a changed test header or
+ test-infrastructure file, a saturated or truncated impact set, or a C
+ change the graph was not asked about -> every suite.
+
+Usage: predict.py --root DIR --changed FILE --selection FILE [--detect FILE]
+ [--depth N] [--meta JSON] --out FILE [--summary FILE]
+ predict.py --needs-graph --changed FILE (prints yes|no)
+"""
+import argparse
+import json
+import os
+import re
+import sys
+
+GRAMMAR_SUITES = re.compile(r"^(grammar_|lang_contract|parse_coverage|extraction|matrix_|"
+ r"edge_types_probe|convergence_probe|ac$|language$)")
+C_SOURCE = re.compile(r"\.(c|h|cc|cpp|inc)$")
+SUITE_DEF = re.compile(r"^\s*(?:SUITE\((\w+)\)|void\s+suite_(\w+)\s*\(\s*void\s*\))\s*\{", re.M)
+RUN_TEST = re.compile(r"\bRUN_TEST\((\w+)\)")
+RUN_SUITE = re.compile(r"\bRUN_SUITE\((\w+)\)")
+
+
+def needs_graph(changed):
+ """A C source change the graph can place (prod or test code)."""
+ return any(C_SOURCE.search(p) and (p.startswith(("src/", "tests/")) or (
+ p.startswith("internal/cbm/") and not p.startswith("internal/cbm/vendored/")))
+ for p in changed)
+
+
+def suitemap(root):
+ """file -> suites defined there, test -> suites running it, suite universe,
+ and the resolver from any suite to the registered suites that run it."""
+ file_suites, test_suites, sources, parents = {}, {}, {}, {}
+ for directory, dirs, names in os.walk(os.path.join(root, "tests")):
+ dirs[:] = sorted(d for d in dirs if not (directory == os.path.join(root, "tests")
+ and d == "fixtures"))
+ for name in sorted(names):
+ if not name.endswith(".c"):
+ continue
+ path = os.path.join(directory, name)
+ rel = os.path.relpath(path, root).replace(os.sep, "/")
+ with open(path, encoding="utf-8", errors="replace") as fh:
+ src = fh.read()
+ sources[rel] = src
+ for m in SUITE_DEF.finditer(src):
+ suite = m.group(1) or m.group(2)
+ start = depth = m.end() - 1
+ end = start
+ depth = 0
+ while end < len(src): # the body is the balanced { } block
+ if src[end] == "{":
+ depth += 1
+ elif src[end] == "}":
+ depth -= 1
+ if depth == 0:
+ break
+ end += 1
+ file_suites.setdefault(rel, []).append(suite)
+ for test in RUN_TEST.findall(src[start:end]):
+ test_suites.setdefault(test, [])
+ if suite not in test_suites[test]:
+ test_suites[test].append(suite)
+ for child in RUN_SUITE.findall(src[start:end]):
+ parents.setdefault(child, set()).add(suite)
+ main = sources.get("tests/test_main.c", "")
+ registered = set(re.findall(r"RUN_SELECTED_SUITE(?:_PERF)?\((\w+)\)", main))
+ universe = registered - set(re.findall(r"RUN_SELECTED_SUITE_PERF\((\w+)\)", main))
+
+ def runnable(suite, seen=frozenset()):
+ """A registered suite runs itself; a nested one runs inside its parents."""
+ if suite in registered:
+ return {suite} & universe
+ out = set()
+ for parent in parents.get(suite, ()):
+ if parent not in seen:
+ out |= runnable(parent, seen | {suite})
+ return out
+ return file_suites, test_suites, universe, sources, runnable
+
+
+def predict(root, changed, selection, detect, depth):
+ file_suites, test_suites, universe, sources, runnable = suitemap(root)
+ reasons, run_all = [], False
+ if selection.get("full"):
+ run_all = True
+ reasons.append(f"run-all: the lane selector ran everything ({selection.get('tier')})")
+ for p in changed:
+ if p.startswith("tests/") and not p.startswith("tests/fixtures/") and p.endswith(".h"):
+ run_all = True
+ reasons.append(f"run-all: test header {p}")
+ elif p.startswith("test-infrastructure/"):
+ run_all = True
+ reasons.append(f"run-all: test infrastructure {p}")
+
+ graph = {"consulted": detect is not None}
+ picked = {} # suite -> "ALL" | set(tests)
+ if detect is not None:
+ rows = detect.get("impacted", [])
+ total = detect.get("impacted_total", len(rows))
+ shown = detect.get("impacted_shown", len(rows))
+ graph.update(impacted_total=total, impacted_shown=shown,
+ saturated=bool(detect.get("engine_saturated")))
+ if graph["saturated"]:
+ run_all = True
+ reasons.append("run-all: detect_changes saturated")
+ if total > len(rows) or shown > len(rows):
+ run_all = True
+ reasons.append(f"run-all: impact set truncated ({len(rows)} of {total} rows)")
+ tests = 0
+ for r in rows:
+ fp, name = r.get("file", ""), r.get("qn", "").rsplit(".", 1)[-1]
+ if not fp.startswith("tests/") or r.get("label") != "Function" or r.get("hop", 0) > depth:
+ continue
+ local = file_suites.get(fp, [])
+ if name in local:
+ continue # the SUITE function itself
+ candidates = test_suites.get(name, [])
+ for suite in [s for s in candidates if s in local] or candidates:
+ for runner in runnable(suite):
+ picked.setdefault(runner, set()).add(name)
+ tests += 1
+ graph["test_rows"] = tests
+ elif needs_graph(changed):
+ run_all = True
+ reasons.append("run-all: C change but no impact set (graph not consulted)")
+
+ for p in changed:
+ if p.startswith("tests/fixtures/"): # fixture sources are data, not tests
+ key = p.split("/")[2] if p.count("/") >= 2 else p
+ for f, src in sources.items():
+ if key in src:
+ for suite in file_suites.get(f, []):
+ for runner in runnable(suite):
+ picked[runner] = "ALL"
+ reasons.append(f"changed fixture {p}: suites whose tests name '{key}'")
+ elif p.startswith("tests/") and p.endswith(".c"):
+ for suite in file_suites.get(p, []):
+ for runner in runnable(suite):
+ picked[runner] = "ALL"
+ reasons.append(f"changed test file {p}: suite {runner}")
+ elif p.startswith("internal/cbm/vendored/grammars/"):
+ for suite in universe:
+ if GRAMMAR_SUITES.match(suite):
+ picked[suite] = "ALL"
+ reasons.append(f"changed grammar {p}: grammar suites")
+
+ if run_all:
+ mode, suites = "all", {s: "ALL" for s in sorted(universe)}
+ else:
+ suites = {s: v if v == "ALL" else sorted(v) for s, v in sorted(picked.items())}
+ mode = "graph" if (graph["consulted"] or suites) else "none"
+ return {"version": 1, "mode": mode, "depth": depth,
+ "selector": {"tier": selection.get("tier"), "full": bool(selection.get("full"))},
+ "universe": len(universe), "selected_suites": len(suites), "suites": suites,
+ "graph": graph, "reasons": sorted(set(reasons))}
+
+
+def summary(result, meta):
+ g = result["graph"]
+ whole = sum(1 for v in result["suites"].values() if v == "ALL")
+ tests = sum(len(v) for v in result["suites"].values() if v != "ALL")
+ lines = ["### Test-impact shadow (report-only, never gates)", "",
+ "| | |", "|---|---|",
+ f"| selector | {result['selector']['tier']} (full: {'yes' if result['selector']['full'] else 'no'}) |",
+ f"| prediction | **{result['mode']}**: {result['selected_suites']} of {result['universe']} "
+ f"suites ({whole} whole, {tests} tests in the rest) |"]
+ if g.get("consulted"):
+ lines.append(f"| graph | {g.get('impacted_shown')} of {g.get('impacted_total')} impacted rows at "
+ f"depth {result['depth']}, {g.get('test_rows')} test rows"
+ f"{', saturated' if g.get('saturated') else ''} |")
+ for key in sorted(meta):
+ lines.append(f"| {key} | {meta[key]} |")
+ if result["reasons"]:
+ lines += ["", "Reasons: " + "; ".join(result["reasons"][:12])]
+ if result["mode"] == "graph" and result["suites"]:
+ lines += ["", "Suites: " + ", ".join(result["suites"])[:1500]]
+ return "\n".join(lines) + "\n"
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+ ap.add_argument("--needs-graph", action="store_true")
+ ap.add_argument("--root")
+ ap.add_argument("--changed", required=True)
+ ap.add_argument("--selection")
+ ap.add_argument("--detect")
+ ap.add_argument("--depth", type=int, default=15)
+ ap.add_argument("--meta", default="{}")
+ ap.add_argument("--out")
+ ap.add_argument("--summary")
+ a = ap.parse_args()
+ with open(a.changed, encoding="utf-8") as fh:
+ changed = sorted({line.strip() for line in fh if line.strip()})
+ if a.needs_graph:
+ print("yes" if needs_graph(changed) else "no")
+ return 0
+ if not (a.root and a.selection and a.out):
+ ap.error("--root, --selection and --out are required")
+ with open(a.selection, encoding="utf-8") as fh:
+ selection = json.load(fh)
+ detect = None
+ if a.detect:
+ with open(a.detect, encoding="utf-8") as fh:
+ detect = json.load(fh)
+ result = predict(a.root, changed, selection, detect, a.depth)
+ meta = json.loads(a.meta)
+ result["meta"] = meta
+ with open(a.out, "w", encoding="utf-8") as fh:
+ json.dump(result, fh, indent=1, sort_keys=True)
+ fh.write("\n")
+ if a.summary:
+ with open(a.summary, "w", encoding="utf-8") as fh:
+ fh.write(summary(result, meta))
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/test.sh b/scripts/test.sh
index bcac86d01..bfc299b15 100755
--- a/scripts/test.sh
+++ b/scripts/test.sh
@@ -360,6 +360,8 @@ echo "=== Step 0z3: lane-aware aggregate gates (ci-ok, shard union) ==="
bash "$ROOT/tests/test_lane_gate_contract.sh"
echo "=== Step 0z4: lane wiring (PRs select, dry run and release run all) ==="
bash "$ROOT/tests/test_lane_wiring_contract.sh"
+echo "=== Step 0z5: test-impact shadow prediction contract ==="
+bash "$ROOT/tests/test_test_impact_predict.sh"
if [ "$CONTRACTS_ONLY" -eq 1 ]; then
echo "=== test.sh: contracts-only — every contract step passed ==="
diff --git a/tests/test_lane_wiring_contract.sh b/tests/test_lane_wiring_contract.sh
index 8aa9e0921..171e61b34 100755
--- a/tests/test_lane_wiring_contract.sh
+++ b/tests/test_lane_wiring_contract.sh
@@ -8,7 +8,7 @@
# 2. pr.yml feeds the selector's lanes to every reusable workflow, runs the
# docs-only contracts job, and hands ci-ok both the selection and the
# run's job list (actions: read) -- ci-ok needs `contracts` but never the
-# report-only memwaste / shadow jobs.
+# report-only memwaste / shadow jobs, and the shadow is continue-on-error.
# 3. every lane of scripts/ci/select-lanes.sh --list-lanes is served by a
# job: matrix lanes as a "lane":"<name>" tag in the leg data (the tag
# ci-ok reads back from the job name), single lanes by an `if:` that
@@ -98,6 +98,11 @@ for needle, why in (("LANES: ${{ needs.changes.outputs.lanes }}", "the selection
("jobs?filter=latest", "the latest attempt of every job"),
("scripts/ci/require-all-green.sh", "the canonical gate")):
require(needle in ci_ok, f"ci-ok must use {why} ({needle})")
+shadow = pr.get("test-impact-shadow", "")
+require("scripts/ci/test-impact-shadow.sh --binary" in shadow
+ and re.search(r"(?m)^ continue-on-error: true$", shadow) is not None,
+ "pr.yml test-impact-shadow must exist, run the canonical script and be "
+ "continue-on-error (report-only)")
test_wf = jobs(text("_test.yml"))
require('--lanes "$LANES"' in test_wf.get("shard-completeness", ""),
"shard-completeness must pass the selection to verify-shard-union.sh")
diff --git a/tests/test_test_impact_predict.sh b/tests/test_test_impact_predict.sh
new file mode 100755
index 000000000..598988830
--- /dev/null
+++ b/tests/test_test_impact_predict.sh
@@ -0,0 +1,139 @@
+#!/usr/bin/env bash
+# Test-impact prediction contract: scripts/test-impact/predict.py turns a
+# detect_changes impact set into the C test suites a change can affect. It
+# feeds the PR test-impact SHADOW job -- report-only, never a gate -- but a
+# shadow that silently under-predicts would make its own evaluation lie, so
+# the mapping and its fallbacks are pinned on a synthetic tree:
+# - impacted test functions map to their registered suites (perf excluded;
+# a nested suite resolves to the registered suite that runs it);
+# - a changed test file always selects its whole suite;
+# - a changed fixture selects the suites whose tests name it;
+# - run-all: the selector's full, a test header, a saturated or truncated
+# impact set -> every suite;
+# - no C change -> the graph is not consulted, nothing is predicted.
+#
+# Usage: tests/test_test_impact_predict.sh [repo-root]
+
+set -euo pipefail
+
+ROOT="${1:-$(cd "$(dirname "$0")/.." && pwd)}"
+PREDICT="$ROOT/scripts/test-impact/predict.py"
+if [ ! -f "$PREDICT" ]; then
+ echo "FAIL: $PREDICT not found" >&2
+ exit 1
+fi
+WORK=$(mktemp -d "${TMPDIR:-/tmp}/cbm-impact-predict.XXXXXX")
+trap 'rm -rf "$WORK"' EXIT
+
+python3 - "$PREDICT" "$WORK" <<'PY'
+import json
+import pathlib
+import subprocess
+import sys
+
+predict, work = sys.argv[1], pathlib.Path(sys.argv[2])
+tree = work / "tree"
+(tree / "tests").mkdir(parents=True)
+(tree / "tests" / "test_main.c").write_text(
+ "RUN_SELECTED_SUITE(alpha);\nRUN_SELECTED_SUITE(beta);\n"
+ "RUN_SELECTED_SUITE(gamma);\nRUN_SELECTED_SUITE_PERF(delta);\n")
+(tree / "tests" / "test_alpha.c").write_text(
+ "TEST(a1) { }\nTEST(a2) { }\nSUITE(alpha) {\n RUN_TEST(a1);\n RUN_TEST(a2);\n}\n")
+(tree / "tests" / "test_beta.c").write_text(
+ 'TEST(b1) { load("tests/fixtures/kx/in.c"); }\nSUITE(beta) {\n RUN_TEST(b1);\n}\n')
+(tree / "tests" / "test_gamma.c").write_text(
+ "TEST(g1) { }\nvoid suite_gamma(void) {\n RUN_TEST(g1);\n}\n")
+(tree / "tests" / "test_delta.c").write_text("TEST(d1) { }\nSUITE(delta) {\n RUN_TEST(d1);\n}\n")
+# A nested suite runs only inside a registered parent (RUN_SUITE in its body).
+(tree / "tests" / "test_nested.c").write_text(
+ "TEST(n1) { }\nSUITE(inner) {\n RUN_TEST(n1);\n}\nSUITE(outer) {\n RUN_SUITE(inner);\n}\n")
+(tree / "tests" / "test_main.c").write_text(
+ (tree / "tests" / "test_main.c").read_text() + "RUN_SELECTED_SUITE(outer);\n")
+failures = []
+
+
+def row(qn, label, file, hop):
+ return {"qn": qn, "label": label, "file": file, "hop": hop}
+
+
+def predict_for(name, changed, selection, detect=None):
+ case = work / name
+ case.mkdir()
+ (case / "changed.txt").write_text("".join(f + "\n" for f in changed))
+ (case / "selection.json").write_text(json.dumps(selection))
+ args = [sys.executable, predict, "--root", str(tree), "--changed", str(case / "changed.txt"),
+ "--selection", str(case / "selection.json"), "--out", str(case / "p.json"),
+ "--summary", str(case / "s.md")]
+ if detect is not None:
+ (case / "detect.json").write_text(json.dumps(detect))
+ args += ["--detect", str(case / "detect.json")]
+ proc = subprocess.run(args, capture_output=True, text=True)
+ if proc.returncode != 0:
+ failures.append(f"{name}: predict.py exited {proc.returncode}: {proc.stderr.strip()}")
+ return None
+ return json.loads((case / "p.json").read_text())
+
+
+def expect(name, got, mode, suites):
+ if got is None:
+ return
+ if got["mode"] != mode:
+ failures.append(f"{name}: mode {got['mode']}, expected {mode}")
+ if got["suites"] != suites:
+ failures.append(f"{name}: suites {got['suites']}, expected {suites}")
+
+
+NARROW = {"tier": "T3-mid", "full": False}
+graph = {"impacted_total": 3, "impacted_shown": 3, "impacted": [
+ row("p.src.store.store_open", "Function", "src/store/store.c", 1),
+ row("p.tests.test_alpha.a1", "Function", "tests/test_alpha.c", 2),
+ row("p.tests.test_alpha.alpha", "Function", "tests/test_alpha.c", 3),
+ row("p.tests.test_delta.d1", "Function", "tests/test_delta.c", 2)]}
+
+expect("graph rows map to suites, perf excluded",
+ predict_for("graph", ["src/store/store.c"], NARROW, graph), "graph", {"alpha": ["a1"]})
+expect("a test in a nested suite maps to its registered parent",
+ predict_for("nested", ["src/store/store.c"], NARROW,
+ {"impacted_total": 1, "impacted_shown": 1,
+ "impacted": [row("p.tests.test_nested.n1", "Function", "tests/test_nested.c", 2)]}),
+ "graph", {"outer": ["n1"]})
+expect("a changed test file selects its whole suite",
+ predict_for("changed-test", ["src/store/store.c", "tests/test_gamma.c"], NARROW, graph),
+ "graph", {"alpha": ["a1"], "gamma": "ALL"})
+expect("a changed fixture selects the suites that name it",
+ predict_for("fixture", ["tests/fixtures/kx/in.c"], {"tier": "T1-tests", "full": False},
+ {"impacted_total": 0, "impacted_shown": 0, "impacted": []}),
+ "graph", {"beta": "ALL"})
+ALL = {"alpha": "ALL", "beta": "ALL", "gamma": "ALL", "outer": "ALL"}
+expect("the selector's full runs every suite",
+ predict_for("full", ["Makefile.cbm", "src/store/store.c"], {"tier": "T5-core", "full": True}, graph),
+ "all", ALL)
+expect("a changed test header runs every suite",
+ predict_for("test-header", ["tests/test_helpers.h"], {"tier": "T1-tests", "full": False}, graph),
+ "all", ALL)
+expect("a saturated impact set runs every suite",
+ predict_for("saturated", ["src/store/store.c"], NARROW, dict(graph, engine_saturated=True)),
+ "all", ALL)
+expect("a truncated impact set runs every suite",
+ predict_for("truncated", ["src/store/store.c"], NARROW, dict(graph, impacted_total=9000)),
+ "all", ALL)
+expect("no C change predicts nothing",
+ predict_for("docs", ["README.md"], {"tier": "T0a-docs", "full": False}), "none", {})
+expect("a C change without a graph is not a narrow prediction",
+ predict_for("no-graph", ["src/store/store.c"], NARROW), "all", ALL)
+
+for changed, want in ((["README.md"], "no"), (["src/store/store.c"], "yes"),
+ (["tests/test_alpha.c"], "yes")):
+ (work / "needs.txt").write_text("".join(f + "\n" for f in changed))
+ out = subprocess.run([sys.executable, predict, "--needs-graph", "--changed",
+ str(work / "needs.txt")], capture_output=True, text=True).stdout.strip()
+ if out != want:
+ failures.append(f"--needs-graph for {changed}: {out!r}, expected {want!r}")
+
+if failures:
+ print("TEST-IMPACT PREDICTION CONTRACT VIOLATED:")
+ for failure in failures:
+ print(f" {failure}")
+ sys.exit(1)
+print("test-impact prediction contract OK (mapping, fallbacks, run-all triggers)")
+PY
diff --git a/tests/test_venue_parity_contract.sh b/tests/test_venue_parity_contract.sh
index 031e9f6d4..c09a0144d 100755
--- a/tests/test_venue_parity_contract.sh
+++ b/tests/test_venue_parity_contract.sh
@@ -393,6 +393,7 @@ scripts/ci/preflight-docker.sh
scripts/ci/require-all-green.sh
scripts/ci/verify-shard-union.sh
scripts/ci/select-lanes.sh
+scripts/ci/test-impact-shadow.sh
scripts/ci/generate-sbom.py
scripts/package-release.sh
scripts/ci/smoke-artifact.sh
@@ -438,6 +439,7 @@ scripts/smoke-invariants.sh
scripts/fuzz.sh
scripts/memwaste.sh
scripts/ci/select-lanes.sh
+scripts/ci/test-impact-shadow.sh
"
for entry in $STRICT_ENTRIES; do
out=$(cd "$ROOT" && bash "$entry" --definitely-not-a-flag 2>&1) && rc=0 || rc=$?