Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Python agentagent-osaiai-agentsai-partnerchatopsclaudeclaude-codecodexdevtoolsdiscord-botlark-botllmlocal-firstlong-term-memoryopencodeslack-botvibe-codingvibeywechat
fix(ci): bound authority scans to live repository sources (#1922)
Call White627ed97aebe2
diff --git a/docs/plans/ci-authority-source-boundary.md b/docs/plans/ci-authority-source-boundary.mdnew file mode 100644index 000000000..e57a4840c--- /dev/null+++ b/docs/plans/ci-authority-source-boundary.md@@ -0,0 +1,64 @@+# Authority Scanner Source Boundary++## Scope and Decision++The final CI quality item changes only the Model Hub authority scanner and its+contracts. PR #1921 has already merged and its exact-source master CI passed.+No application behavior, authority policy, workflow, dependency, test selection,+timeout, runner count, or shard weights change. Dedicated CI optimization stops+after this item and its merged-source gates.++The old recursive scans include ignored virtual environments, build outputs,+and nested worktrees. A bounded local profile read 15,795 distinct files and+parsed 14,493 Python paths, including 13,637 under a private virtual environment.+Only nine parses repeated a filename: source scope, not AST caching alone, is+the main defect. The original two CI authority checks each took about 10 seconds+on both the #1921 PR and master. Local profiled durations are not CI speedups.++## Contracts++- Repository discovery uses Git's tracked plus untracked, non-ignored files.+ Tracked files remain in scope even when an ignore rule matches them. New+ legitimate source files do not need staging or a known top-level directory.+- Importer discovery and all registry glob scans use the same boundary and Git+ glob semantics, including root files matched by a leading `**/`.+- This repository development checker requires Git and an exact checkout root,+ including linked worktrees. Enumeration failure cannot produce a passing+ verdict. Explicit registered inputs remain required regardless of ignore rules.+ Source reads may not escape the root through parent paths or symlinks.+- File bytes and selected consumer ASTs are reused only within one check's+ `AuthorityInput`. Bulk importer trees are not retained. A new check gets new+ source discovery, bytes, ASTs, findings, and a content fingerprint.+- JSON is decoded anew from invocation-local bytes; callers and the existing+ persisted-version mutation probe cannot share a mutable cached JSON object.+- Live-file checks, ownership findings, all normative absence/version checks,+ and the existing `AuthorityInput.json` mutation contract remain effective.++Git excludes ignored untracked dependencies and does not recursively inspect+submodules as if their implementation belonged to this repository. This is an+explicit source-ownership boundary, not a security sandbox or an atomic snapshot+of concurrent filesystem edits. Files observed during one invocation keep their+first-read bytes; a later invocation observes changes again.++## Validation++- Eleven new boundary/cache cases and the unchanged Model Hub config/routing+ consumers passed together: 303 cases in 11.37 seconds. Another run covered the+ eleven contracts plus 78 workflow/shard/private-fixture cases: 89 passed.+- Diagnostic-only mutations restored the old filesystem scope, disabled byte+ reuse, and disabled AST reuse separately. Each failed its targeted contract.+ The byte-reuse mutation initially revealed an insufficient test assertion;+ the contract now exercises both text and AST consumers of the same input.+- The original live checker caught a synthetic fixture version in the new,+ still-untracked test file. The fixture now generates its own version parameter;+ no version policy, file exclusion, or original assertion was weakened.+- A completed local fixed-check profile returned a clean verdict in 8.65 seconds,+ versus the original 28.04-second profile. It read 2,121 files exactly once and+ parsed 855 Python files once each; three additional AST calls came from literal+ evaluation. These local profiled samples do not establish hosted CI savings.+- Ruff 0.4.9, dependency integrity (83 packages), and whitespace checks passed.++Full PR CI and exact merged-source master CI remain required, including all 17+checks, every selected file/exit/metrics record, real distribution/install/upgrade+tests, Windows, and the UI browser gate. No additional optimization phase is+authorized by this plan.diff --git a/scripts/check_model_hub_authorities.py b/scripts/check_model_hub_authorities.pyindex ce667b576..63ddc008b 100644--- a/scripts/check_model_hub_authorities.py+++ b/scripts/check_model_hub_authorities.py@@ -8,7 +8,9 @@ import astimport fnmatchimport hashlibimport json+import osimport re+import subprocessfrom pathlib import Pathfrom typing import Any@@ -20,14 +22,23 @@ REGISTRY = CONTRACTS / "mirror-registry.json"class AuthorityInput:def __init__(self, root: Path) -> None:- self.root = root+ self.root = root.resolve()self.files: dict[Path, bytes] = {}+ self._trees: dict[str, ast.Module] = {}+ self._globs: dict[str, tuple[Path, ...]] = {}+ self._checkout_verified = False- def bytes(self, relative: str) -> bytes:+ def _path(self, relative: str) -> Path:path = self.root / relative- payload = path.read_bytes()- self.files[path] = payload- return payload+ if not path.resolve().is_relative_to(self.root):+ raise ValueError(f"authority input escapes checkout: {relative}")+ return path++ def bytes(self, relative: str) -> bytes:+ path = self._path(relative)+ if path not in self.files:+ self.files[path] = path.read_bytes()+ return self.files[path]def text(self, relative: str) -> str:return self.bytes(relative).decode("utf-8")@@ -35,6 +46,36 @@ class AuthorityInput:def json(self, relative: str) -> Any:return json.loads(self.text(relative))+ def python_tree(self, relative: str, *, retain: bool = True) -> ast.Module:+ if relative not in self._trees:+ tree = ast.parse(self.bytes(relative), filename=relative)+ if not retain:+ return tree+ self._trees[relative] = tree+ return self._trees[relative]++ def _git(self, *args: str) -> bytes:+ return subprocess.run(+ ["git", "-C", str(self.root), *args],+ check=True, capture_output=True, timeout=30,+ ).stdout++ def glob(self, pattern: str) -> tuple[Path, ...]:+ if not self._checkout_verified:+ checkout = Path(os.fsdecode(self._git("rev-parse", "--show-toplevel")).rstrip("\n"))+ if checkout.resolve() != self.root:+ raise ValueError("authority source root must be the Git checkout root")+ self._checkout_verified = True+ if pattern not in self._globs:+ # Git owns ignore/pathspec semantics; include unstaged new consumers.+ names = self._git(+ "ls-files", "--cached", "--others", "--exclude-standard", "-z",+ "--", f":(glob){pattern}",+ ).split(b"\0")+ paths = (self._path(os.fsdecode(name)) for name in sorted(set(names)) if name)+ self._globs[pattern] = tuple(path for path in paths if path.is_file())+ return self._globs[pattern]+def fingerprint(self) -> str:digest = hashlib.sha256()for path, payload in sorted(self.files.items()):@@ -233,7 +274,7 @@ def _literal_strings(node: ast.AST) -> set[str]:def _python_test_literals(source: AuthorityInput, relative: str, name: str) -> set[str]:- tree = ast.parse(source.text(relative), filename=relative)+ tree = source.python_tree(relative)for node in tree.body:if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == name:return _literal_strings(node)@@ -241,7 +282,7 @@ def _python_test_literals(source: AuthorityInput, relative: str, name: str) -> sdef _python_literal_annotation(source: AuthorityInput, spec: dict[str, Any]) -> set[str]:- tree = ast.parse(source.text(spec["file"]), filename=spec["file"])+ tree = source.python_tree(spec["file"])for node in tree.body:if not isinstance(node, ast.ClassDef) or node.name != spec["class"]:continue@@ -256,7 +297,7 @@ def _python_literal_annotation(source: AuthorityInput, spec: dict[str, Any]) ->def _python_string_assignment(source: AuthorityInput, spec: dict[str, Any]) -> set[str]:- tree = ast.parse(source.text(spec["file"]), filename=spec["file"])+ tree = source.python_tree(spec["file"])for node in tree.body:if not isinstance(node, (ast.Assign, ast.AnnAssign)):continue@@ -485,12 +526,12 @@ def _binding_lane_rows(source: AuthorityInput, check: dict[str, Any]) -> list[stdef _python_importers(source: AuthorityInput, check: dict[str, Any]) -> set[str]:importers: set[str] = set()- for path in sorted(source.root.rglob("*.py")):+ for path in source.glob("**/*.py"):relative = path.relative_to(source.root).as_posix()if any(fnmatch.fnmatch(relative, pattern) for pattern in check.get("exclude_globs", ())):continuetry:- tree = ast.parse(source.bytes(relative), filename=relative)+ tree = source.python_tree(relative, retain=False)except (SyntaxError, UnicodeDecodeError):continueif any(@@ -503,6 +544,7 @@ def _python_importers(source: AuthorityInput, check: dict[str, Any]) -> set[str]def check(root: Path = ROOT) -> dict[str, Any]:source = AuthorityInput(root)+ root = source.rootregistry = source.json("docs/plans/model-hub-contracts/mirror-registry.json")findings: list[dict[str, Any]] = []@@ -606,7 +648,7 @@ def check(root: Path = ROOT) -> dict[str, Any]:)matched_literal_files: set[str] = set()for pattern in version_closure.get("literal_globs", ()):- for path in sorted(root.glob(pattern)):+ for path in source.glob(pattern):if not path.is_file():continuerelative = path.relative_to(root).as_posix()@@ -662,7 +704,7 @@ def check(root: Path = ROOT) -> dict[str, Any]:for absence in registry.get("schema_absence_checks", ()):terms = _normative_absence_terms(source, absence)for pattern in absence["scope_globs"]:- for path in sorted(root.glob(pattern)):+ for path in source.glob(pattern):relative = path.relative_to(root).as_posix()tokens = _schema_decision_tokens(source, relative)for term in sorted(terms & tokens):@@ -711,7 +753,7 @@ def check(root: Path = ROOT) -> dict[str, Any]:})for pattern in boundary["forbidden_ui_globs"]:- for path in sorted(root.glob(pattern)):+ for path in source.glob(pattern):relative = path.relative_to(root).as_posix()if value in source.text(relative):findings.append(@@ -721,7 +763,7 @@ def check(root: Path = ROOT) -> dict[str, Any]:for absence in registry.get("repo_absence_checks", ()):term = "-".join(absence["term_parts"])for pattern in absence["scope_globs"]:- for path in sorted(root.glob(pattern)):+ for path in source.glob(pattern):if not path.is_file():continuerelative = path.relative_to(root).as_posix()diff --git a/tests/test_model_hub_authorities.py b/tests/test_model_hub_authorities.pynew file mode 100644index 000000000..8bd54f3a9--- /dev/null+++ b/tests/test_model_hub_authorities.py@@ -0,0 +1,250 @@+"""Source ownership and invocation-local reuse for the live authority checker."""++import ast+from collections import Counter+import json+from pathlib import Path+import subprocess++import pytest++from scripts.check_model_hub_authorities import (+ AuthorityInput,+ _python_importers,+ _python_literal_annotation,+ _python_string_assignment,+ _python_test_literals,+ check,+)+++_FIXTURE_VERSION = 1+++def source_repo(tmp_path):+ root = tmp_path / "checkout"+ root.mkdir()+ subprocess.run(["git", "init", "-q", str(root)], check=True)+ return root+++def _write(root, relative, text):+ path = root / relative+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(text, encoding="utf-8")+ return path+++def _registry(root):+ registry = {+ "contract_version": _FIXTURE_VERSION,+ "input_generation": {"mode": "same_run_live_files"},+ "authority_table_discovery": {"files": []},+ "entries": [],+ "decision_tables": [],+ "ownership_checks": [{+ "id": "ownership", "kind": "python_importers_have_exactly_one_lane",+ "module": "pkg.owner", "lane_table_file": "lanes.md",+ "lane_table_heading": "Implementation lanes",+ }],+ "repo_absence_checks": [{+ "id": "retired", "term_parts": ["retired", "fixture"],+ "scope_globs": ["**/*.py", "**/*.md", "**/*.json", "**/*.ts", "**/*.tsx"],+ }],+ }+ _write(root, "lanes.md", "Implementation lanes\n| Lane | Files |\n| --- | --- |\n")+ _write(root, "docs/plans/model-hub-contracts/mirror-registry.json", json.dumps(registry))+ return registry+++def test_discovery_preserves_live_source_ownership_and_ignores_artifacts(source_repo):+ root = source_repo+ artifacts = [".venv", "build", "dist", "node_modules", ".worktrees"]+ _write(root, ".gitignore", "\n".join(f"{directory}/" for directory in artifacts))+ source_names = {"root.py", "new_lane/new.py", "odd name\nconsumer.py", "build/tracked.py"}+ for name in source_names:+ _write(root, name, "from pkg.owner import value\n")+ subprocess.run(["git", "-C", str(root), "add", "-f", "build/tracked.py"], check=True)+ for directory in artifacts:+ _write(root, f"{directory}/copied.py", "from pkg.owner import value\n")+ _write(root, "invalid.py", "this is not Python !!!")+ deleted = _write(root, "deleted.py", "from pkg.owner import value\n")+ subprocess.run(["git", "-C", str(root), "add", "deleted.py"], check=True)+ deleted.unlink()++ source = AuthorityInput(root)+ assert {p.relative_to(root).as_posix() for p in source.glob("**/*.py")} == source_names | {"invalid.py"}+ assert _python_importers(source, {"module": "pkg.owner"}) == source_names+ assert _python_importers(source, {+ "module": "pkg.owner", "exclude_globs": ["new_lane/**"],+ }) == source_names - {"new_lane/new.py"}+++def test_all_registry_globs_use_the_checkout_boundary(source_repo, monkeypatch):+ root = source_repo+ registry = _registry(root)+ registry["contract_version_closure"] = {"literal_globs": ["**/*.py"]}+ registry["schema_absence_checks"] = [{+ "id": "schema", "source_file": "normative.md", "start_marker": "Forbidden:",+ "scope_globs": ["**/*.schema.json"],+ }]+ registry["api_boundary_only_errors"] = [{+ "value": "boundary_only", "contract_file": "normative.md",+ "negative_test_file": "consumer.py", "negative_test": "test_boundary",+ "forbidden_ui_globs": ["**/*.tsx"],+ }]+ _write(root, "docs/plans/model-hub-contracts/mirror-registry.json", json.dumps(registry))+ _write(root, "normative.md", "Forbidden: `retired_field`\n\nboundary_only\n")+ _write(root, "input.schema.json", '{"properties": {"kept": {"type": "string"}}}')+ _write(root, "consumer.py", f"CONTRACT_VERSION = {_FIXTURE_VERSION}\ndef test_boundary():\n assert 'boundary_only'\n")+ _write(root, ".gitignore", ".venv/\n")+ _write(root, ".venv/copy.py", "from pkg.owner import value\n")+ retired = "-".join(registry["repo_absence_checks"][0]["term_parts"])+ for suffix in ("py", "md", "json", "ts", "tsx"):+ _write(root, f".venv/copy.{suffix}", json.dumps(retired))++ original_glob = Path.glob++ def no_unbounded_root_glob(path, pattern):+ assert path != root, f"unbounded registry scan: {pattern}"+ return original_glob(path, pattern)++ monkeypatch.setattr(Path, "glob", no_unbounded_root_glob)+ baseline = check(root)+ assert baseline["ok"], baseline["findings"]+ for suffix in ("py", "md", "json", "ts", "tsx"):+ _write(root, f"future_lane/live.{suffix}", json.dumps(retired))+ result = check(root)+ assert not result["ok"]+ assert {finding["file"] for finding in result["findings"]} == {+ f"future_lane/live.{suffix}" for suffix in ("py", "md", "json", "ts", "tsx")+ }+ assert result["input_fingerprint"] != baseline["input_fingerprint"]+++def test_new_checks_observe_tracked_edits_and_new_importers(source_repo):+ root = source_repo+ _registry(root)+ tracked = _write(root, "consumer.py", "value = 1\n")+ subprocess.run(["git", "-C", str(root), "add", "consumer.py"], check=True)+ baseline = check(root)+ assert baseline["ok"], baseline["findings"]++ tracked.write_text("from pkg.owner import value\n", encoding="utf-8")+ _write(root, "new_lane/untracked.py", "from pkg.owner import value\n")+ changed = check(root)+ assert not changed["ok"]+ assert changed["input_fingerprint"] != baseline["input_fingerprint"]+ assert changed["findings"] == [+ {"kind": "ownership_cardinality", "domain": "ownership", "path": name, "owners": 0}+ for name in ("consumer.py", "new_lane/untracked.py")+ ]+++def test_reuses_first_read_and_consumer_tree_only_within_one_input(source_repo, monkeypatch):+ root = source_repo+ original_text = (+ "from pkg.owner import value\n"+ "VERSION = 'first'\n"+ "class Input:\n reason: Literal['first']\n"+ "def test_example():\n assert value == 'first'\n"+ )+ path = _write(root, "consumer.py", original_text)+ _write(root, "bulk.py", "from pkg.owner import value\n")+ reads, parses = Counter(), Counter()+ original_read, original_parse = Path.read_bytes, ast.parse++ def read(file):+ reads[file] += 1+ return original_read(file)++ def parse(payload, filename="<unknown>", *args, **kwargs):+ parses[filename] += 1+ return original_parse(payload, filename, *args, **kwargs)++ monkeypatch.setattr(Path, "read_bytes", read)+ monkeypatch.setattr(ast, "parse", parse)+ source = AuthorityInput(root)+ assignment = {"file": "consumer.py", "name": "VERSION"}+ assert source.text("consumer.py") == original_text+ assert _python_string_assignment(source, assignment) == {"first"}+ assert _python_literal_annotation(source, {+ "file": "consumer.py", "class": "Input", "field": "reason",+ }) == {"first"}+ assert _python_test_literals(source, "consumer.py", "test_example") == {"first"}+ assert _python_importers(source, {"module": "pkg.owner"}) == {"consumer.py", "bulk.py"}+ assert reads[path] == parses["consumer.py"] == 1+ assert set(source._trees) == {"consumer.py"}+ fingerprint = source.fingerprint()++ path.write_text(original_text.replace("first", "second"), encoding="utf-8")+ assert source.text("consumer.py") == original_text+ assert _python_string_assignment(source, assignment) == {"first"}+ assert source.fingerprint() == fingerprint+ fresh = AuthorityInput(root)+ assert _python_string_assignment(fresh, assignment) == {"second"}+ assert reads[path] == parses["consumer.py"] == 2+ assert fresh.fingerprint() != fingerprint+++def test_json_mutations_do_not_change_invocation_input(source_repo):+ _write(source_repo, "input.json", '{"values": [1]}')+ source = AuthorityInput(source_repo)+ payload = source.json("input.json")+ fingerprint = source.fingerprint()+ payload["values"].append(2)+ assert source.json("input.json") == {"values": [1]}+ assert source.fingerprint() == fingerprint+++def test_source_discovery_is_reused_only_within_one_input(source_repo, monkeypatch):+ original_git = AuthorityInput._git+ queries = []++ def git(source, *args):+ queries.append(args)+ return original_git(source, *args)++ monkeypatch.setattr(AuthorityInput, "_git", git)+ source = AuthorityInput(source_repo)+ assert source.glob("**/*.py") == ()+ new_path = _write(source_repo, "new.py", "value = 1\n")+ assert source.glob("**/*.py") == ()+ assert sum(args[0] == "ls-files" for args in queries) == 1+ assert AuthorityInput(source_repo).glob("**/*.py") == (new_path,)+ assert sum(args[0] == "ls-files" for args in queries) == 2+++def test_git_enumeration_failure_never_becomes_a_passing_check(source_repo, monkeypatch):+ _registry(source_repo)++ def unavailable(*args):+ raise subprocess.CalledProcessError(128, "git")++ monkeypatch.setattr(AuthorityInput, "_git", unavailable)+ with pytest.raises(subprocess.CalledProcessError):+ check(source_repo)+++def test_source_root_must_be_the_checkout_not_a_parent_repository_subdirectory(source_repo):+ child = source_repo / "child"+ child.mkdir()+ with pytest.raises(ValueError, match="checkout root"):+ AuthorityInput(child).glob("**/*.py")+++def test_archive_without_git_cannot_silently_pass(tmp_path):+ _registry(tmp_path)+ with pytest.raises(subprocess.CalledProcessError):+ check(tmp_path)++[email protected]("relative", ["../outside.py", "escape.py"])+def test_authority_inputs_cannot_escape_checkout(source_repo, relative):+ outside = _write(source_repo.parent, "outside.py", "value = 1\n")+ (source_repo / "escape.py").symlink_to(outside)+ with pytest.raises(ValueError, match="escapes checkout"):+ AuthorityInput(source_repo).bytes(relative)+ with pytest.raises(ValueError, match="escapes checkout"):+ AuthorityInput(source_repo).glob("**/*.py")