Local code intelligence for developers and AI agents. Connect code, docs and infrastructure in a persistent graph, with CBM Lens, MCP tools, agent integrations and change-aware test selection.
C aiderastclaude-codecode-analysiscode-intelligencecodexcursorcypherdeveloper-toolsgemini-cligraph-visualizationkilocodeknowledge-graphmcpmcp-servermodel-context-protocolopencodesqlitetree-sitterwindsurf
ci: report-only test-impact shadow job on PRs
Martin Vogela8d5a8154f9d
diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.ymlindex 0fd09b878..d6d966912 100644--- a/.github/workflows/pr.yml+++ b/.github/workflows/pr.yml@@ -212,6 +212,49 @@ jobs:with:mode: pr+ # ── Test-impact SHADOW, REPORT-ONLY: predicts from the code graph which C+ # suites this change can affect (the latest released binary indexes the+ # PR checkout; detect_changes at depth 15 against the base, mapped to+ # suites, with the selector's run-all triggers) and uploads the+ # prediction, so graph-based selection can be judged against what really+ # failed before anything gates on it. continue-on-error and NOT in+ # ci-ok's needs: it can never block or pass a merge. ──+ test-impact-shadow:+ runs-on: ubuntu-latest+ timeout-minutes: 20+ continue-on-error: true+ steps:+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ with:+ # The PR merge commit and its base parent: the diff to predict for.+ fetch-depth: 2+ persist-credentials: false+ - name: Fetch the latest released binary+ id: release+ continue-on-error: true+ env:+ GH_TOKEN: ${{ github.token }}+ run: scripts/ci/test-impact-shadow.sh --fetch-release "$RUNNER_TEMP/cbm-bin"+ - name: Build the binary from this checkout (download failed)+ if: ${{ steps.release.outcome == 'failure' }}+ run: |+ sudo apt-get update && sudo apt-get install -y zlib1g-dev+ scripts/build.sh CC=gcc CXX=g+++ mkdir -p "$RUNNER_TEMP/cbm-bin"+ cp build/c/codebase-memory-mcp "$RUNNER_TEMP/cbm-bin/"+ - name: Predict the impacted suites+ run: scripts/ci/test-impact-shadow.sh --binary "$RUNNER_TEMP/cbm-bin/codebase-memory-mcp" --out "$RUNNER_TEMP/test-impact"+ - name: Upload the prediction+ if: ${{ always() }}+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1+ with:+ name: test-impact-prediction+ path: |+ ${{ runner.temp }}/test-impact/prediction.json+ ${{ runner.temp }}/test-impact/selection.json+ ${{ runner.temp }}/test-impact/changed-files.txt+ if-no-files-found: warn+# ── Docs-only PRs (tier T0a) select no test leg, so this runs the static# contract steps of the canonical test leg on their own (they police# docs surfaces too: language counts, version metadata). Every test legdiff --git a/scripts/ci/README.md b/scripts/ci/README.mdindex 3e74b86d7..f38cf36a8 100644--- a/scripts/ci/README.md+++ b/scripts/ci/README.md@@ -15,6 +15,7 @@ CI and the local infrastructure — both of which the venue-parity contract| `generate-sbom.py` | The release SPDX SBOM (vendored versions reviewable here, diffable by vendoring PRs — was inline YAML). | `release.yml` || `require-all-green.sh` | The aggregate gate: fail unless every needed job succeeded or legitimately skipped (was inline YAML) — and, under lane selection, every SELECTED lane ran and succeeded (skipped/cancelled/missing = red; unselected lanes may skip). | `pr.yml ci-ok` || `select-lanes.sh` | Decide which PR lanes a change set runs (tier, lanes, full, reasons) from its file list — the R1 rule set, fail-safe to every lane. Pinned by `tests/test_select_lanes.sh` and the September 2026 history replay (`scripts/test-impact/replay-selector.sh`). | `pr.yml changes` |+| `test-impact-shadow.sh` | The report-only test-impact shadow: fetch the latest released binary (checksum-verified) or take a given one, index this checkout into a throwaway cache, run `detect_changes` against the base and map it to suites (`scripts/test-impact/predict.py`). Never gates. | `pr.yml test-impact-shadow` || `verify-shard-union.sh` | Prove sharded test legs lost nothing: shard count agreement, indices 1..n, identical suite lists, union of slices == full list (was inline YAML). No manifests is green only when the lane selection holds no test leg. | `_test.yml` shard-completeness || `prepare-release-candidates.sh` | Copy one linker output into stripped/unstripped candidates, finalize signatures, composition-check them without execution, and record their hashes. | `_build.yml`, local artifact smoke || `stage-release-candidates.py` | Admit exactly eight candidate artifacts / sixteen byte-distinct binaries into the content-addressed VirusTotal scan set. | `_build.yml` |diff --git a/scripts/ci/test-impact-shadow.sh b/scripts/ci/test-impact-shadow.shnew file mode 100755index 000000000..a6f6f369a--- /dev/null+++ b/scripts/ci/test-impact-shadow.sh@@ -0,0 +1,168 @@+#!/usr/bin/env bash+# test-impact-shadow.sh — the PR test-impact SHADOW: predict from the code graph+# which C test suites a change can affect, and publish the prediction. It is+# REPORT-ONLY: pr.yml runs it continue-on-error and ci-ok does not need it, so+# it can never block or pass a merge. Its predictions accumulate next to the+# lanes that actually ran and failed, which is how graph-based selection gets+# evaluated before anyone gates on it.+#+# The binary is the latest PUBLISHED release, not this checkout's build: a few+# seconds of download instead of minutes of compile, and a PR that breaks the+# indexer cannot corrupt its own prediction. pr.yml builds the checkout only+# if the download fails.+#+# Usage: scripts/ci/test-impact-shadow.sh --fetch-release DIR [--repo OWNER/REPO]+# Download the latest release's archive for this host + checksums.txt+# into DIR, verify the SHA-256, extract DIR/codebase-memory-mcp.+# Needs GH_TOKEN (read-only); $GH overrides the gh command.+# scripts/ci/test-impact-shadow.sh --binary PATH --out DIR [--base REV] [--depth N]+# Index this checkout into a throwaway cache (HOME, XDG dirs and+# CBM_CACHE_DIR under DIR/work, a private CBM_RUNTIME_DIR under /tmp:+# the daemon's socket path must fit sun_path), run+# detect_changes against REV (default HEAD^1, the base side of a PR+# merge commit) at depth N (default 15), and let+# scripts/test-impact/predict.py map it to suites with the lane+# selector's run-all triggers. Writes DIR/prediction.json and+# DIR/summary.md (appended to $GITHUB_STEP_SUMMARY when set). The+# graph is only consulted when C sources changed.+# Exit: 0 = prediction written · 1 = the tooling broke · 2 = usage error.+set -euo pipefail++usage() { sed -n '2,28p' "$0" | sed 's/^# \{0,1\}//'; }++FETCH_DIR=""+REPO="${GITHUB_REPOSITORY:-DeusData/codebase-memory-mcp}"+BINARY=""+OUT=""+BASE="HEAD^1"+DEPTH=15+while [ $# -gt 0 ]; do+ case "$1" in+ -h | --help)+ usage+ exit 0+ ;;+ --fetch-release | --repo | --binary | --out | --base | --depth)+ if [ $# -lt 2 ]; then+ echo "test-impact-shadow.sh: $1 needs a value. Please consult --help." >&2+ exit 2+ fi+ case "$1" in+ --fetch-release) FETCH_DIR="$2" ;;+ --repo) REPO="$2" ;;+ --binary) BINARY="$2" ;;+ --out) OUT="$2" ;;+ --base) BASE="$2" ;;+ --depth) DEPTH="$2" ;;+ esac+ shift+ ;;+ *)+ echo "test-impact-shadow.sh: unknown argument '$1'. Please consult --help." >&2+ exit 2+ ;;+ esac+ shift+done++ROOT="$(cd "$(dirname "$0")/../.." && pwd)"++sha256_check() { # file with "<sha> <name>" lines, run in the archive's dir+ if command -v sha256sum >/dev/null 2>&1; then+ sha256sum -c "$1"+ else+ shasum -a 256 -c "$1"+ fi+}++if [ -n "$FETCH_DIR" ]; then+ case "$(uname -s)/$(uname -m)" in+ Linux/x86_64) asset=codebase-memory-mcp-linux-amd64.tar.gz ;;+ Linux/aarch64 | Linux/arm64) asset=codebase-memory-mcp-linux-arm64.tar.gz ;;+ Darwin/arm64) asset=codebase-memory-mcp-darwin-arm64.tar.gz ;;+ Darwin/x86_64) asset=codebase-memory-mcp-darwin-amd64.tar.gz ;;+ *)+ echo "test-impact-shadow.sh: no release archive for $(uname -s)/$(uname -m)" >&2+ exit 1+ ;;+ esac+ read -r -a gh_cmd <<<"${GH:-gh}"+ mkdir -p "$FETCH_DIR"+ # No tag: the latest release.+ "${gh_cmd[@]}" release download --repo "$REPO" --pattern "$asset" \+ --pattern checksums.txt --dir "$FETCH_DIR" --clobber+ grep " $asset\$" "$FETCH_DIR/checksums.txt" >"$FETCH_DIR/asset.sha256"+ (cd "$FETCH_DIR" && sha256_check asset.sha256)+ tar -xzf "$FETCH_DIR/$asset" -C "$FETCH_DIR" codebase-memory-mcp+ echo "release binary: $FETCH_DIR/codebase-memory-mcp ($asset, SHA-256 verified)"+ exit 0+fi++if [ -z "$BINARY" ] || [ -z "$OUT" ]; then+ echo "test-impact-shadow.sh: --binary and --out are required. Please consult --help." >&2+ exit 2+fi+case "$DEPTH" in+'' | *[!0-9]*)+ echo "test-impact-shadow.sh: --depth must be a number. Please consult --help." >&2+ exit 2+ ;;+esac+BINARY="$(cd "$(dirname "$BINARY")" && pwd)/$(basename "$BINARY")"+PREDICT="$ROOT/scripts/test-impact/predict.py"+mkdir -p "$OUT"+OUT="$(cd "$OUT" && pwd)"+WORK="$OUT/work"+rm -rf "$WORK"+mkdir -p "$WORK/home/.config" "$WORK/home/.cache" "$WORK/cache"+# <runtime>/cbm-daemon-<uid>/cbm-<key>.sock adds ~42 bytes to this path, and+# sun_path holds 104 (macOS) / 108: a deep $OUT or $TMPDIR does not fit.+RUNTIME=$(mktemp -d /tmp/cbm-rt.XXXXXX)+trap 'rm -rf "$RUNTIME"' EXIT++BASE_SHA=$(git -C "$ROOT" rev-parse --verify "$BASE^{commit}")+git -C "$ROOT" diff --name-only --no-renames "$BASE_SHA" HEAD >"$OUT/changed-files.txt"+bash "$ROOT/scripts/ci/select-lanes.sh" "$OUT/changed-files.txt" >"$OUT/selection.json"++# From here on nothing may touch the host's real config, cache or daemon.+export HOME="$WORK/home"+export XDG_CONFIG_HOME="$WORK/home/.config"+export XDG_CACHE_HOME="$WORK/home/.cache"+export CBM_CACHE_DIR="$WORK/cache"+export CBM_RUNTIME_DIR="$RUNTIME"++VERSION=$("$BINARY" --version 2>/dev/null | head -1 || true)+DETECT_ARGS=()+T_INDEX="-"+T_DETECT="-"+if [ "$(python3 "$PREDICT" --needs-graph --changed "$OUT/changed-files.txt")" = yes ]; then+ python3 -c 'import json, sys; json.dump({"repo_path": sys.argv[1]}, open(sys.argv[2], "w"))' \+ "$ROOT" "$WORK/index-args.json"+ t0=$(date +%s)+ "$BINARY" cli --quiet index_repository --args-file "$WORK/index-args.json" >"$WORK/index.json"+ t1=$(date +%s)+ python3 -c '+import json, sys+project = json.load(open(sys.argv[1]))["project"]+json.dump({"project": project, "base_branch": sys.argv[2], "depth": int(sys.argv[3]),+ "limit": 5000, "changed_limit": 5000, "module_limit": 0,+ "max_output_tokens": 1000000, "format": "json"}, open(sys.argv[4], "w"))+' "$WORK/index.json" "$BASE_SHA" "$DEPTH" "$WORK/detect-args.json"+ "$BINARY" cli --quiet detect_changes --args-file "$WORK/detect-args.json" >"$WORK/detect.json"+ t2=$(date +%s)+ # The CLI started a private daemon under $RUNTIME; stop it (best effort).+ "$BINARY" daemon stop >/dev/null 2>&1 || true+ T_INDEX="$((t1 - t0))s"+ T_DETECT="$((t2 - t1))s"+ DETECT_ARGS=(--detect "$WORK/detect.json")+fi++META=$(python3 -c 'import json, sys; print(json.dumps(dict(zip(sys.argv[1::2], sys.argv[2::2]))))' \+ binary "${VERSION:-unknown}" base "$BASE_SHA" "index time" "$T_INDEX" "detect time" "$T_DETECT")+python3 "$PREDICT" --root "$ROOT" --changed "$OUT/changed-files.txt" \+ --selection "$OUT/selection.json" ${DETECT_ARGS[@]+"${DETECT_ARGS[@]}"} --depth "$DEPTH" \+ --meta "$META" --out "$OUT/prediction.json" --summary "$OUT/summary.md"+cat "$OUT/summary.md"+if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then+ cat "$OUT/summary.md" >>"$GITHUB_STEP_SUMMARY"+fidiff --git a/scripts/test-impact/predict.py b/scripts/test-impact/predict.pynew file mode 100644index 000000000..7658ea2b2--- /dev/null+++ b/scripts/test-impact/predict.py@@ -0,0 +1,232 @@+#!/usr/bin/env python3+"""predict.py — map a detect_changes impact set to the C test suites it can+affect: the PR test-impact SHADOW (scripts/ci/test-impact-shadow.sh).+Report-only; nothing here decides a merge.++The mapping is the one the September 2026 test-impact replay measured+(detect_changes at depth 15 plus rule fallbacks, "dcF@15"):+ - impacted test FUNCTIONS (graph rows in tests/*.c) map to the suites that+ RUN_TEST them, resolved in their own file first; a nested suite (RUN_SUITE+ inside another) resolves to the registered suite that runs it; only suites+ registered in tests/test_main.c count, perf suites excluded;+ - a changed tests/*.c file always selects its whole suite;+ - a changed fixture (tests/fixtures/<key>/...) selects the suites whose test+ files name <key>; a changed grammar selects the grammar suites;+ - run-all: the lane selector's full, a changed test header or+ test-infrastructure file, a saturated or truncated impact set, or a C+ change the graph was not asked about -> every suite.++Usage: predict.py --root DIR --changed FILE --selection FILE [--detect FILE]+ [--depth N] [--meta JSON] --out FILE [--summary FILE]+ predict.py --needs-graph --changed FILE (prints yes|no)+"""+import argparse+import json+import os+import re+import sys++GRAMMAR_SUITES = re.compile(r"^(grammar_|lang_contract|parse_coverage|extraction|matrix_|"+ r"edge_types_probe|convergence_probe|ac$|language$)")+C_SOURCE = re.compile(r"\.(c|h|cc|cpp|inc)$")+SUITE_DEF = re.compile(r"^\s*(?:SUITE\((\w+)\)|void\s+suite_(\w+)\s*\(\s*void\s*\))\s*\{", re.M)+RUN_TEST = re.compile(r"\bRUN_TEST\((\w+)\)")+RUN_SUITE = re.compile(r"\bRUN_SUITE\((\w+)\)")+++def needs_graph(changed):+ """A C source change the graph can place (prod or test code)."""+ return any(C_SOURCE.search(p) and (p.startswith(("src/", "tests/")) or (+ p.startswith("internal/cbm/") and not p.startswith("internal/cbm/vendored/")))+ for p in changed)+++def suitemap(root):+ """file -> suites defined there, test -> suites running it, suite universe,+ and the resolver from any suite to the registered suites that run it."""+ file_suites, test_suites, sources, parents = {}, {}, {}, {}+ for directory, dirs, names in os.walk(os.path.join(root, "tests")):+ dirs[:] = sorted(d for d in dirs if not (directory == os.path.join(root, "tests")+ and d == "fixtures"))+ for name in sorted(names):+ if not name.endswith(".c"):+ continue+ path = os.path.join(directory, name)+ rel = os.path.relpath(path, root).replace(os.sep, "/")+ with open(path, encoding="utf-8", errors="replace") as fh:+ src = fh.read()+ sources[rel] = src+ for m in SUITE_DEF.finditer(src):+ suite = m.group(1) or m.group(2)+ start = depth = m.end() - 1+ end = start+ depth = 0+ while end < len(src): # the body is the balanced { } block+ if src[end] == "{":+ depth += 1+ elif src[end] == "}":+ depth -= 1+ if depth == 0:+ break+ end += 1+ file_suites.setdefault(rel, []).append(suite)+ for test in RUN_TEST.findall(src[start:end]):+ test_suites.setdefault(test, [])+ if suite not in test_suites[test]:+ test_suites[test].append(suite)+ for child in RUN_SUITE.findall(src[start:end]):+ parents.setdefault(child, set()).add(suite)+ main = sources.get("tests/test_main.c", "")+ registered = set(re.findall(r"RUN_SELECTED_SUITE(?:_PERF)?\((\w+)\)", main))+ universe = registered - set(re.findall(r"RUN_SELECTED_SUITE_PERF\((\w+)\)", main))++ def runnable(suite, seen=frozenset()):+ """A registered suite runs itself; a nested one runs inside its parents."""+ if suite in registered:+ return {suite} & universe+ out = set()+ for parent in parents.get(suite, ()):+ if parent not in seen:+ out |= runnable(parent, seen | {suite})+ return out+ return file_suites, test_suites, universe, sources, runnable+++def predict(root, changed, selection, detect, depth):+ file_suites, test_suites, universe, sources, runnable = suitemap(root)+ reasons, run_all = [], False+ if selection.get("full"):+ run_all = True+ reasons.append(f"run-all: the lane selector ran everything ({selection.get('tier')})")+ for p in changed:+ if p.startswith("tests/") and not p.startswith("tests/fixtures/") and p.endswith(".h"):+ run_all = True+ reasons.append(f"run-all: test header {p}")+ elif p.startswith("test-infrastructure/"):+ run_all = True+ reasons.append(f"run-all: test infrastructure {p}")++ graph = {"consulted": detect is not None}+ picked = {} # suite -> "ALL" | set(tests)+ if detect is not None:+ rows = detect.get("impacted", [])+ total = detect.get("impacted_total", len(rows))+ shown = detect.get("impacted_shown", len(rows))+ graph.update(impacted_total=total, impacted_shown=shown,+ saturated=bool(detect.get("engine_saturated")))+ if graph["saturated"]:+ run_all = True+ reasons.append("run-all: detect_changes saturated")+ if total > len(rows) or shown > len(rows):+ run_all = True+ reasons.append(f"run-all: impact set truncated ({len(rows)} of {total} rows)")+ tests = 0+ for r in rows:+ fp, name = r.get("file", ""), r.get("qn", "").rsplit(".", 1)[-1]+ if not fp.startswith("tests/") or r.get("label") != "Function" or r.get("hop", 0) > depth:+ continue+ local = file_suites.get(fp, [])+ if name in local:+ continue # the SUITE function itself+ candidates = test_suites.get(name, [])+ for suite in [s for s in candidates if s in local] or candidates:+ for runner in runnable(suite):+ picked.setdefault(runner, set()).add(name)+ tests += 1+ graph["test_rows"] = tests+ elif needs_graph(changed):+ run_all = True+ reasons.append("run-all: C change but no impact set (graph not consulted)")++ for p in changed:+ if p.startswith("tests/fixtures/"): # fixture sources are data, not tests+ key = p.split("/")[2] if p.count("/") >= 2 else p+ for f, src in sources.items():+ if key in src:+ for suite in file_suites.get(f, []):+ for runner in runnable(suite):+ picked[runner] = "ALL"+ reasons.append(f"changed fixture {p}: suites whose tests name '{key}'")+ elif p.startswith("tests/") and p.endswith(".c"):+ for suite in file_suites.get(p, []):+ for runner in runnable(suite):+ picked[runner] = "ALL"+ reasons.append(f"changed test file {p}: suite {runner}")+ elif p.startswith("internal/cbm/vendored/grammars/"):+ for suite in universe:+ if GRAMMAR_SUITES.match(suite):+ picked[suite] = "ALL"+ reasons.append(f"changed grammar {p}: grammar suites")++ if run_all:+ mode, suites = "all", {s: "ALL" for s in sorted(universe)}+ else:+ suites = {s: v if v == "ALL" else sorted(v) for s, v in sorted(picked.items())}+ mode = "graph" if (graph["consulted"] or suites) else "none"+ return {"version": 1, "mode": mode, "depth": depth,+ "selector": {"tier": selection.get("tier"), "full": bool(selection.get("full"))},+ "universe": len(universe), "selected_suites": len(suites), "suites": suites,+ "graph": graph, "reasons": sorted(set(reasons))}+++def summary(result, meta):+ g = result["graph"]+ whole = sum(1 for v in result["suites"].values() if v == "ALL")+ tests = sum(len(v) for v in result["suites"].values() if v != "ALL")+ lines = ["### Test-impact shadow (report-only, never gates)", "",+ "| | |", "|---|---|",+ f"| selector | {result['selector']['tier']} (full: {'yes' if result['selector']['full'] else 'no'}) |",+ f"| prediction | **{result['mode']}**: {result['selected_suites']} of {result['universe']} "+ f"suites ({whole} whole, {tests} tests in the rest) |"]+ if g.get("consulted"):+ lines.append(f"| graph | {g.get('impacted_shown')} of {g.get('impacted_total')} impacted rows at "+ f"depth {result['depth']}, {g.get('test_rows')} test rows"+ f"{', saturated' if g.get('saturated') else ''} |")+ for key in sorted(meta):+ lines.append(f"| {key} | {meta[key]} |")+ if result["reasons"]:+ lines += ["", "Reasons: " + "; ".join(result["reasons"][:12])]+ if result["mode"] == "graph" and result["suites"]:+ lines += ["", "Suites: " + ", ".join(result["suites"])[:1500]]+ return "\n".join(lines) + "\n"+++def main():+ ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])+ ap.add_argument("--needs-graph", action="store_true")+ ap.add_argument("--root")+ ap.add_argument("--changed", required=True)+ ap.add_argument("--selection")+ ap.add_argument("--detect")+ ap.add_argument("--depth", type=int, default=15)+ ap.add_argument("--meta", default="{}")+ ap.add_argument("--out")+ ap.add_argument("--summary")+ a = ap.parse_args()+ with open(a.changed, encoding="utf-8") as fh:+ changed = sorted({line.strip() for line in fh if line.strip()})+ if a.needs_graph:+ print("yes" if needs_graph(changed) else "no")+ return 0+ if not (a.root and a.selection and a.out):+ ap.error("--root, --selection and --out are required")+ with open(a.selection, encoding="utf-8") as fh:+ selection = json.load(fh)+ detect = None+ if a.detect:+ with open(a.detect, encoding="utf-8") as fh:+ detect = json.load(fh)+ result = predict(a.root, changed, selection, detect, a.depth)+ meta = json.loads(a.meta)+ result["meta"] = meta+ with open(a.out, "w", encoding="utf-8") as fh:+ json.dump(result, fh, indent=1, sort_keys=True)+ fh.write("\n")+ if a.summary:+ with open(a.summary, "w", encoding="utf-8") as fh:+ fh.write(summary(result, meta))+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/test.sh b/scripts/test.shindex bcac86d01..bfc299b15 100755--- a/scripts/test.sh+++ b/scripts/test.sh@@ -360,6 +360,8 @@ echo "=== Step 0z3: lane-aware aggregate gates (ci-ok, shard union) ==="bash "$ROOT/tests/test_lane_gate_contract.sh"echo "=== Step 0z4: lane wiring (PRs select, dry run and release run all) ==="bash "$ROOT/tests/test_lane_wiring_contract.sh"+echo "=== Step 0z5: test-impact shadow prediction contract ==="+bash "$ROOT/tests/test_test_impact_predict.sh"if [ "$CONTRACTS_ONLY" -eq 1 ]; thenecho "=== test.sh: contracts-only — every contract step passed ==="diff --git a/tests/test_lane_wiring_contract.sh b/tests/test_lane_wiring_contract.shindex 8aa9e0921..171e61b34 100755--- a/tests/test_lane_wiring_contract.sh+++ b/tests/test_lane_wiring_contract.sh@@ -8,7 +8,7 @@# 2. pr.yml feeds the selector's lanes to every reusable workflow, runs the# docs-only contracts job, and hands ci-ok both the selection and the# run's job list (actions: read) -- ci-ok needs `contracts` but never the-# report-only memwaste / shadow jobs.+# report-only memwaste / shadow jobs, and the shadow is continue-on-error.# 3. every lane of scripts/ci/select-lanes.sh --list-lanes is served by a# job: matrix lanes as a "lane":"<name>" tag in the leg data (the tag# ci-ok reads back from the job name), single lanes by an `if:` that@@ -98,6 +98,11 @@ for needle, why in (("LANES: ${{ needs.changes.outputs.lanes }}", "the selection("jobs?filter=latest", "the latest attempt of every job"),("scripts/ci/require-all-green.sh", "the canonical gate")):require(needle in ci_ok, f"ci-ok must use {why} ({needle})")+shadow = pr.get("test-impact-shadow", "")+require("scripts/ci/test-impact-shadow.sh --binary" in shadow+ and re.search(r"(?m)^ continue-on-error: true$", shadow) is not None,+ "pr.yml test-impact-shadow must exist, run the canonical script and be "+ "continue-on-error (report-only)")test_wf = jobs(text("_test.yml"))require('--lanes "$LANES"' in test_wf.get("shard-completeness", ""),"shard-completeness must pass the selection to verify-shard-union.sh")diff --git a/tests/test_test_impact_predict.sh b/tests/test_test_impact_predict.shnew file mode 100755index 000000000..598988830--- /dev/null+++ b/tests/test_test_impact_predict.sh@@ -0,0 +1,139 @@+#!/usr/bin/env bash+# Test-impact prediction contract: scripts/test-impact/predict.py turns a+# detect_changes impact set into the C test suites a change can affect. It+# feeds the PR test-impact SHADOW job -- report-only, never a gate -- but a+# shadow that silently under-predicts would make its own evaluation lie, so+# the mapping and its fallbacks are pinned on a synthetic tree:+# - impacted test functions map to their registered suites (perf excluded;+# a nested suite resolves to the registered suite that runs it);+# - a changed test file always selects its whole suite;+# - a changed fixture selects the suites whose tests name it;+# - run-all: the selector's full, a test header, a saturated or truncated+# impact set -> every suite;+# - no C change -> the graph is not consulted, nothing is predicted.+#+# Usage: tests/test_test_impact_predict.sh [repo-root]++set -euo pipefail++ROOT="${1:-$(cd "$(dirname "$0")/.." && pwd)}"+PREDICT="$ROOT/scripts/test-impact/predict.py"+if [ ! -f "$PREDICT" ]; then+ echo "FAIL: $PREDICT not found" >&2+ exit 1+fi+WORK=$(mktemp -d "${TMPDIR:-/tmp}/cbm-impact-predict.XXXXXX")+trap 'rm -rf "$WORK"' EXIT++python3 - "$PREDICT" "$WORK" <<'PY'+import json+import pathlib+import subprocess+import sys++predict, work = sys.argv[1], pathlib.Path(sys.argv[2])+tree = work / "tree"+(tree / "tests").mkdir(parents=True)+(tree / "tests" / "test_main.c").write_text(+ "RUN_SELECTED_SUITE(alpha);\nRUN_SELECTED_SUITE(beta);\n"+ "RUN_SELECTED_SUITE(gamma);\nRUN_SELECTED_SUITE_PERF(delta);\n")+(tree / "tests" / "test_alpha.c").write_text(+ "TEST(a1) { }\nTEST(a2) { }\nSUITE(alpha) {\n RUN_TEST(a1);\n RUN_TEST(a2);\n}\n")+(tree / "tests" / "test_beta.c").write_text(+ 'TEST(b1) { load("tests/fixtures/kx/in.c"); }\nSUITE(beta) {\n RUN_TEST(b1);\n}\n')+(tree / "tests" / "test_gamma.c").write_text(+ "TEST(g1) { }\nvoid suite_gamma(void) {\n RUN_TEST(g1);\n}\n")+(tree / "tests" / "test_delta.c").write_text("TEST(d1) { }\nSUITE(delta) {\n RUN_TEST(d1);\n}\n")+# A nested suite runs only inside a registered parent (RUN_SUITE in its body).+(tree / "tests" / "test_nested.c").write_text(+ "TEST(n1) { }\nSUITE(inner) {\n RUN_TEST(n1);\n}\nSUITE(outer) {\n RUN_SUITE(inner);\n}\n")+(tree / "tests" / "test_main.c").write_text(+ (tree / "tests" / "test_main.c").read_text() + "RUN_SELECTED_SUITE(outer);\n")+failures = []+++def row(qn, label, file, hop):+ return {"qn": qn, "label": label, "file": file, "hop": hop}+++def predict_for(name, changed, selection, detect=None):+ case = work / name+ case.mkdir()+ (case / "changed.txt").write_text("".join(f + "\n" for f in changed))+ (case / "selection.json").write_text(json.dumps(selection))+ args = [sys.executable, predict, "--root", str(tree), "--changed", str(case / "changed.txt"),+ "--selection", str(case / "selection.json"), "--out", str(case / "p.json"),+ "--summary", str(case / "s.md")]+ if detect is not None:+ (case / "detect.json").write_text(json.dumps(detect))+ args += ["--detect", str(case / "detect.json")]+ proc = subprocess.run(args, capture_output=True, text=True)+ if proc.returncode != 0:+ failures.append(f"{name}: predict.py exited {proc.returncode}: {proc.stderr.strip()}")+ return None+ return json.loads((case / "p.json").read_text())+++def expect(name, got, mode, suites):+ if got is None:+ return+ if got["mode"] != mode:+ failures.append(f"{name}: mode {got['mode']}, expected {mode}")+ if got["suites"] != suites:+ failures.append(f"{name}: suites {got['suites']}, expected {suites}")+++NARROW = {"tier": "T3-mid", "full": False}+graph = {"impacted_total": 3, "impacted_shown": 3, "impacted": [+ row("p.src.store.store_open", "Function", "src/store/store.c", 1),+ row("p.tests.test_alpha.a1", "Function", "tests/test_alpha.c", 2),+ row("p.tests.test_alpha.alpha", "Function", "tests/test_alpha.c", 3),+ row("p.tests.test_delta.d1", "Function", "tests/test_delta.c", 2)]}++expect("graph rows map to suites, perf excluded",+ predict_for("graph", ["src/store/store.c"], NARROW, graph), "graph", {"alpha": ["a1"]})+expect("a test in a nested suite maps to its registered parent",+ predict_for("nested", ["src/store/store.c"], NARROW,+ {"impacted_total": 1, "impacted_shown": 1,+ "impacted": [row("p.tests.test_nested.n1", "Function", "tests/test_nested.c", 2)]}),+ "graph", {"outer": ["n1"]})+expect("a changed test file selects its whole suite",+ predict_for("changed-test", ["src/store/store.c", "tests/test_gamma.c"], NARROW, graph),+ "graph", {"alpha": ["a1"], "gamma": "ALL"})+expect("a changed fixture selects the suites that name it",+ predict_for("fixture", ["tests/fixtures/kx/in.c"], {"tier": "T1-tests", "full": False},+ {"impacted_total": 0, "impacted_shown": 0, "impacted": []}),+ "graph", {"beta": "ALL"})+ALL = {"alpha": "ALL", "beta": "ALL", "gamma": "ALL", "outer": "ALL"}+expect("the selector's full runs every suite",+ predict_for("full", ["Makefile.cbm", "src/store/store.c"], {"tier": "T5-core", "full": True}, graph),+ "all", ALL)+expect("a changed test header runs every suite",+ predict_for("test-header", ["tests/test_helpers.h"], {"tier": "T1-tests", "full": False}, graph),+ "all", ALL)+expect("a saturated impact set runs every suite",+ predict_for("saturated", ["src/store/store.c"], NARROW, dict(graph, engine_saturated=True)),+ "all", ALL)+expect("a truncated impact set runs every suite",+ predict_for("truncated", ["src/store/store.c"], NARROW, dict(graph, impacted_total=9000)),+ "all", ALL)+expect("no C change predicts nothing",+ predict_for("docs", ["README.md"], {"tier": "T0a-docs", "full": False}), "none", {})+expect("a C change without a graph is not a narrow prediction",+ predict_for("no-graph", ["src/store/store.c"], NARROW), "all", ALL)++for changed, want in ((["README.md"], "no"), (["src/store/store.c"], "yes"),+ (["tests/test_alpha.c"], "yes")):+ (work / "needs.txt").write_text("".join(f + "\n" for f in changed))+ out = subprocess.run([sys.executable, predict, "--needs-graph", "--changed",+ str(work / "needs.txt")], capture_output=True, text=True).stdout.strip()+ if out != want:+ failures.append(f"--needs-graph for {changed}: {out!r}, expected {want!r}")++if failures:+ print("TEST-IMPACT PREDICTION CONTRACT VIOLATED:")+ for failure in failures:+ print(f" {failure}")+ sys.exit(1)+print("test-impact prediction contract OK (mapping, fallbacks, run-all triggers)")+PYdiff --git a/tests/test_venue_parity_contract.sh b/tests/test_venue_parity_contract.shindex 031e9f6d4..c09a0144d 100755--- a/tests/test_venue_parity_contract.sh+++ b/tests/test_venue_parity_contract.sh@@ -393,6 +393,7 @@ scripts/ci/preflight-docker.shscripts/ci/require-all-green.shscripts/ci/verify-shard-union.shscripts/ci/select-lanes.sh+scripts/ci/test-impact-shadow.shscripts/ci/generate-sbom.pyscripts/package-release.shscripts/ci/smoke-artifact.sh@@ -438,6 +439,7 @@ scripts/smoke-invariants.shscripts/fuzz.shscripts/memwaste.shscripts/ci/select-lanes.sh+scripts/ci/test-impact-shadow.sh"for entry in $STRICT_ENTRIES; doout=$(cd "$ROOT" && bash "$entry" --definitely-not-a-flag 2>&1) && rc=0 || rc=$?