Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(hogli): add a verification ladder skill and hogli verify --changed (#122352)

Georges-Antoine Assi7745f1da27ac
diff --git a/.agents/skills/verifying-changes/SKILL.md b/.agents/skills/verifying-changes/SKILL.md
new file mode 100644
index 000000000000..019630be64ae
--- /dev/null
+++ b/.agents/skills/verifying-changes/SKILL.md
@@ -0,0 +1,72 @@
+---
+name: verifying-changes
+description: >
+ Picks the cheapest check that is enough to confirm a change works, and says when a slower one is needed.
+ Use before you report a task done, before you start Storybook, the dev stack or a
+ Playwright run to check your own work, and when you are unsure how much verification a diff needs.
+ Maps each kind of change, such as pure logic, DRF serializer or viewset, migration, kea logic, React component,
+ CSS, Django template, CI workflow, SQL or HogQL, or docs and skills, to the layer that is enough and the layer that is wasted.
+ `hogli verify --changed` runs the cheap layers for the current diff and names the slower ones left to run yourself.
+ Trigger terms: verify, how do I test this, is this enough testing, start the stack, render it, run e2e.
+---
+
+# Verifying changes
+
+Verification has six layers, and each one costs roughly ten times the one before it.
+Start with the cheapest, and move up only when the current layer cannot see what the diff changes.
+
+| Layer | Check | Command |
+| ----- | ---------------------------- | ------------------------------------------------------------------------------------------------ |
+| 1 | Lint, format, typecheck | `hogli verify --changed`, or `ruff check` and `pnpm --filter=@posthog/frontend typescript:check` |
+| 2 | The nearest unit test | `hogli test <file>`, `hogli test --changed` |
+| 3 | Serializer and OpenAPI regen | `hogli up -d`, `hogli build:openapi`, then commit the generated diff |
+| 4 | Storybook render | `pnpm storybook`, open the affected story, compare before and after |
+| 5 | The running stack | `hogli up -d`, `hogli wait`, then exercise the flow |
+| 6 | Playwright or a recorded run | `hogli test playwright/<spec>.spec.ts` against a running stack |
+
+`hogli verify --changed` classifies the diff against master, runs layers 1 and 2, and lists any higher layer you still have to run yourself.
+It does not report the diff as verified when a check was skipped, when a check only printed advice, or when no test covers the changed files.
+mypy is one of those advice-only checks. It runs repo-wide, so `hogli verify` prints its command instead of running it.
+Use `--plan` to see the classification without running anything.
+
+## Decision table
+
+| Change | Enough | Wasted |
+| -------------------------- | --------------------------------------------------- | ---------------------------------------- |
+| Pure logic, Python or TS | 2: nearest unit test | 5: stack |
+| DRF serializer or viewset | 3: OpenAPI regen, plus the endpoint test at layer 2 | 6: Playwright |
+| Migration | 2: migration test, plus `hogli migrations:check` | 4: Storybook |
+| Kea logic | 2: logic test, plus typecheck | 4: Storybook |
+| React component | 4: Storybook render | 5: stack, unless the data wiring changed |
+| CSS only | 4: Storybook render | 2: unit tests |
+| CI workflow | 1: `hogli lint:workflows`, `hogli ci:plan` | 2: unit tests |
+| SQL or HogQL | 2: the ClickHouse-backed query test | 5: stack |
+| Docs or skills | 1: format, `hogli lint:skills` | 2 and above |
+| Django template | 5: render the page in the running stack | 6: Playwright |
+| Settings, URLs, middleware | 5: running stack | 6: Playwright |
+
+A diff that mixes kinds needs the highest layer among them, and only for the files of that kind.
+
+## Other rules still apply
+
+- Reading the code does not confirm visual or UX work, as the root `AGENTS.md` says. A component or CSS change always needs at least a Storybook render.
+- Layer 2 says where a test runs. Use `/writing-tests` to decide whether to add or change one. When a regression is hard to reach cheaply, extract the logic and test it at a lower layer.
+- `hogli ci:preflight` stays the pre-push check, see `/running-ci-preflight`. `hogli verify` reuses its lint and format checks, so a pass here usually means a pass there too.
+
+## When the stack is required
+
+Start the stack only when the behavior spans more than one process and no layer 2 test runs both sides:
+
+- Settings, URL routing, middleware, or the Content Security Policy, because only the browser applies them.
+- A Django template, because only Django renders it.
+- A change that a person has to see in the app: a new scene, a navigation change, or data flowing from ingestion to a UI.
+- A Celery task, Temporal workflow, or plugin-server path whose wiring changed. A change to its logic alone stays at layer 2.
+- A Playwright spec you changed, because it runs against the stack.
+
+Everything else can skip the stack.
+Django and ClickHouse-backed tests need the Postgres and ClickHouse services but not the web app, so `hogli up -d` plus a scoped `hogli test` still counts as layer 2.
+
+## Report what you ran
+
+Say which layer you stopped at and why the next one was not needed.
+If you could not run a required layer, say so and name the command, so a reviewer knows what is unverified.
diff --git a/AGENTS.md b/AGENTS.md
index 166696bbcc59..a3a750ccae60 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -314,6 +314,7 @@ ALWAYS invoke the matching skill **first** — do not skip it, and do not attemp
- `/writing-dataclasses` — adding or changing a Python dataclass, replacing a tuple or `dict[str, Any]` payload, or passing a dataclass or facade contract through internal layers
- `/announcing-behavior-changes` — shipping a fix that changes what an existing user sees (a metric moves, a count drops, a range resolves differently), or adding, reviewing, or removing an in-app change notice
+- `/verifying-changes` — choosing how to confirm a change works before reporting it done, or before starting Storybook, the dev stack, or a Playwright run to check your own work
- `/merging-prs` — merging a PR, or babysitting one through the Trunk merge queue
- `/stacking-prs` — creating, restacking, adopting, or landing a stack of PRs (`gh stack`)
- `/implementing-mcp-tools` — adding/modifying endpoints or `tools.yaml`
diff --git a/hogli.yaml b/hogli.yaml
index 83eb87dcdc19..981c9f0d0afc 100644
--- a/hogli.yaml
+++ b/hogli.yaml
@@ -800,6 +800,9 @@ tests:
test:
click: hogli_commands.test_runner:test_command
description: Auto-detect test type (Python, Jest, Playwright, Rust, Go) and run it
+ verify:
+ click: hogli_commands.verify_layers:verify
+ description: Run the cheapest checks that cover your diff (lint, typecheck, nearest tests), then name the next potential layer
test:python:
cmd: pytest
description: Run Python test suite
diff --git a/tools/hogli-commands/hogli_commands/test_runner.py b/tools/hogli-commands/hogli_commands/test_runner.py
index 64371a27d4fb..248447104afb 100644
--- a/tools/hogli-commands/hogli_commands/test_runner.py
+++ b/tools/hogli-commands/hogli_commands/test_runner.py
@@ -726,8 +726,19 @@ def _run_grouped(detected: list[tuple[str, TestRunConfig]], extra_args: list[str
_run(cfg.command + extra_args, env=cfg.env_or_none, cwd=cfg.cwd)
-def _run_changed(extra_args: list[str]) -> None:
- """Find changed test files and tests for changed source files, then run them."""
+@dataclass(frozen=True, kw_only=True, slots=True)
+class ChangedTests:
+ directly_changed: set[str]
+ discovered: set[str]
+ skipped_playwright_specs: set[str]
+
+ @property
+ def all_files(self) -> list[str]:
+ return sorted(self.directly_changed | self.discovered)
+
+
+def changed_test_files(*, skip_playwright: bool = False) -> ChangedTests:
+ """The changed test files, and the test files found for changed source files."""
changed = _get_changed_files()
directly_changed = {f for f in changed if _is_test_file(f)}
@@ -737,7 +748,26 @@ def _run_changed(extra_args: list[str]) -> None:
if not _is_test_file(f):
discovered.update(_find_test_files_for_source(f))
- all_test_files = sorted(directly_changed | discovered)
+ playwright_specs: set[str] = set()
+ if skip_playwright:
+ playwright_specs = {f for f in directly_changed | discovered if f.startswith("playwright/")}
+ directly_changed -= playwright_specs
+ discovered -= playwright_specs
+
+ return ChangedTests(
+ directly_changed=directly_changed, discovered=discovered, skipped_playwright_specs=playwright_specs
+ )
+
+
+def _run_changed(extra_args: list[str], skip_playwright: bool = False) -> None:
+ """Find changed test files and tests for changed source files, then run them."""
+ tests = changed_test_files(skip_playwright=skip_playwright)
+ if tests.skipped_playwright_specs:
+ click.secho(f"Skipping {len(tests.skipped_playwright_specs)} Playwright spec(s).", fg="yellow")
+ directly_changed = tests.directly_changed
+ discovered = tests.discovered
+
+ all_test_files = tests.all_files
if not all_test_files:
click.secho("No test files found for changes on this branch.", fg="yellow")
@@ -828,16 +858,21 @@ def _run_watch(file_path: str, extra_args: list[str]) -> None:
)
@click.argument("file_path", required=False, type=click.Path())
@click.option("--changed", is_flag=True, help="Run tests changed on this branch, plus tests for changed source files")
+ "--skip-playwright",
+ is_flag=True,
+ help="With --changed, leave out Playwright specs, which need a running stack",
+)
@click.option("--watch", is_flag=True, help="Re-run tests on file changes (Python and Jest)")
@click.pass_context
-def test_command(ctx: click.Context, file_path: str | None, changed: bool, watch: bool) -> None:
+def test_command(ctx: click.Context, file_path: str | None, changed: bool, skip_playwright: bool, watch: bool) -> None:
"""Auto-detect test type and run the correct test runner."""
if changed:
if file_path:
raise click.UsageError("Cannot combine --changed with a file path.")
if watch:
raise click.UsageError("Cannot combine --changed with --watch.")
- _run_changed(list(ctx.args))
+ _run_changed(list(ctx.args), skip_playwright=skip_playwright)
return
if not file_path:
diff --git a/tools/hogli-commands/hogli_commands/tests/test_test_runner.py b/tools/hogli-commands/hogli_commands/tests/test_test_runner.py
index 8f4880b37879..642209f42fa6 100644
--- a/tools/hogli-commands/hogli_commands/tests/test_test_runner.py
+++ b/tools/hogli-commands/hogli_commands/tests/test_test_runner.py
@@ -515,6 +515,17 @@ class TestRunChanged:
command = mock_run.call_args[0][0]
assert command.count("posthog/api/test/test_comments.py") == 1
+ @patch("hogli_commands.test_runner._run")
+ @patch("hogli_commands.test_runner._get_changed_files")
+ def test_skip_playwright_leaves_out_specs(self, mock_changed, mock_run) -> None:
+ mock_changed.return_value = ["playwright/e2e/dashboards.spec.ts", "posthog/api/test/test_user.py"]
+ _run_changed([], skip_playwright=True)
+
+ commands = [call[0][0] for call in mock_run.call_args_list]
+ assert len(commands) == 1
+ assert commands[0][0] == "pytest"
+ assert "playwright/e2e/dashboards.spec.ts" not in commands[0]
+
@patch("hogli_commands.test_runner._get_changed_files")
def test_no_changed_files_exits(self, mock_changed) -> None:
mock_changed.return_value = ["posthog/api/views.py"]
diff --git a/tools/hogli-commands/hogli_commands/tests/test_verify_layers.py b/tools/hogli-commands/hogli_commands/tests/test_verify_layers.py
new file mode 100644
index 000000000000..f97c35e6ea6c
--- /dev/null
+++ b/tools/hogli-commands/hogli_commands/tests/test_verify_layers.py
@@ -0,0 +1,97 @@
+from __future__ import annotations
+
+import pytest
+from unittest.mock import patch
+
+from click.testing import CliRunner
+from hogli_commands.test_runner import ChangedTests
+from hogli_commands.verify_layers import (
+ ChangeKind,
+ ComplexityLayer,
+ _print_manual_layers,
+ classify,
+ plan_verification,
+ verify,
+)
+
+
+class TestClassify:
+ @pytest.mark.parametrize(
+ "path,expected_kind",
+ [
+ ("posthog/api/test/test_insight.py", ChangeKind.TEST),
+ ("posthog/api/test/__snapshots__/test_insight.ambr", ChangeKind.TEST),
+ ("frontend/src/scenes/insights/Insight.test.tsx", ChangeKind.TEST),
+ ("playwright/e2e/dashboards.spec.ts", ChangeKind.END_TO_END_TEST),
+ ("posthog/api/insight.py", ChangeKind.API_SCHEMA),
+ ("products/surveys/backend/api/survey.py", ChangeKind.API_SCHEMA),
+ ("posthog/migrations/0999_add_field.py", ChangeKind.MIGRATION),
+ ("frontend/src/scenes/insights/insightLogic.tsx", ChangeKind.KEA_LOGIC),
+ ("frontend/src/scenes/insights/Insight.tsx", ChangeKind.REACT_COMPONENT),
+ ("frontend/src/scenes/insights/Insight.stories.tsx", ChangeKind.REACT_COMPONENT),
+ ("frontend/src/styles/base.scss", ChangeKind.CSS),
+ ("posthog/hogql_queries/insights/trends/trends_query_runner.py", ChangeKind.SQL_OR_HOGQL),
+ (".github/workflows/ci-backend.yml", ChangeKind.CI_WORKFLOW),
+ (".agents/skills/verifying-changes/SKILL.md", ChangeKind.DOCS_OR_SKILLS),
+ ("products/surveys/skills/creating-surveys/references/guide.md", ChangeKind.DOCS_OR_SKILLS),
+ ("posthog/settings/web.py", ChangeKind.APP_WIRING),
+ ("products/surveys/backend/routes.py", ChangeKind.APP_WIRING),
+ ("posthog/csp_middleware.py", ChangeKind.APP_WIRING),
+ ("posthog/models/team/team.py", ChangeKind.PURE_LOGIC),
+ ("docs/onboarding/web-analytics/react.tsx", ChangeKind.REACT_COMPONENT),
+ (".agents/skills/verifying-changes/scripts/check.py", ChangeKind.PURE_LOGIC),
+ (".github/actions/paths-filter/action.yml", ChangeKind.CI_WORKFLOW),
+ (".github/actions/paths-filter/__tests__/filter.test.ts", ChangeKind.TEST),
+ (".github/actions/paths-filter/src/filter.ts", ChangeKind.PURE_LOGIC),
+ ("posthog/templates/authorize_and_link.html", ChangeKind.DJANGO_TEMPLATE),
+ ],
+ )
+ def test_kind(self, path: str, expected_kind: ChangeKind) -> None:
+ assert classify(path).kind == expected_kind
+
+
+class TestPlanVerification:
+ @pytest.mark.parametrize(
+ "paths,runs_unit_tests,manual_layers",
+ [
+ (["docs/internal/monorepo-layout.md", ".github/workflows/ci-backend.yml"], False, ()),
+ (["posthog/models/team/team.py", "posthog/api/test/test_team.py"], True, ()),
+ (
+ ["frontend/src/scenes/insights/Insight.tsx", "posthog/api/insight.py", "posthog/settings/web.py"],
+ True,
+ (ComplexityLayer.OPENAPI, ComplexityLayer.STORYBOOK, ComplexityLayer.STACK),
+ ),
+ ],
+ )
+ def test_layers(self, paths: list[str], runs_unit_tests: bool, manual_layers: tuple[ComplexityLayer, ...]) -> None:
+ plan = plan_verification(paths)
+ assert plan.runs_unit_tests is runs_unit_tests
+ assert plan.manual_layers == manual_layers
+
+ @pytest.mark.parametrize(
+ "outstanding,claims_enough",
+ [((), True), (("[markdown-format] was skipped: needs node. Set up what it needs.",), False)],
+ )
+ def test_outstanding_check_does_not_claim_enough(
+ self, capsys: pytest.CaptureFixture[str], outstanding: tuple[str, ...], claims_enough: bool
+ ) -> None:
+ _print_manual_layers(plan_verification(["docs/internal/monorepo-layout.md"]), outstanding)
+ assert ("are enough for this diff" in capsys.readouterr().out) is claims_enough
+
+
+class TestVerifyCommand:
+ def test_an_advisory_check_and_no_tests_leave_the_diff_unverified(self) -> None:
+ no_tests = ChangedTests(directly_changed=set(), discovered=set(), skipped_playwright_specs=set())
+ with (
+ patch("hogli_commands.verify_layers.changed_files", return_value=["posthog/models/team/team.py"]),
+ patch("hogli_commands.verify_layers._run_diff_check", return_value=("advisory", "run mypy")),
+ patch("hogli_commands.verify_layers.changed_test_files", return_value=no_tests),
+ patch("hogli_commands.verify_layers.subprocess.run") as run,
+ ):
+ result = CliRunner().invoke(verify, ["--changed"])
+
+ assert result.exit_code == 0
+ assert "[type-check] did not run: run mypy." in result.output
+ assert "No tests were found for the changed files" in result.output
+ assert "are enough for this diff" not in result.output
+ run.assert_not_called()
diff --git a/tools/hogli-commands/hogli_commands/verify_layers.py b/tools/hogli-commands/hogli_commands/verify_layers.py
new file mode 100644
index 000000000000..de904e1676f6
--- /dev/null
+++ b/tools/hogli-commands/hogli_commands/verify_layers.py
@@ -0,0 +1,278 @@
+"""``hogli verify``: run the cheapest checks that are enough for the current diff.
+
+The layers and the decision table come from ``.agents/skills/verifying-changes/SKILL.md``.
+Keep the two in sync.
+"""
+
+from __future__ import annotations
+
+import sys
+import subprocess
+import dataclasses
+from collections.abc import Callable, Iterable
+from dataclasses import dataclass
+from enum import IntEnum, StrEnum
+
+import click
+from hogli.manifest import REPO_ROOT
+
+from hogli_commands.build import TRIGGERS as BUILD_TRIGGERS
+from hogli_commands.change_detection import changed_files, matches_globs
+from hogli_commands.ci_preflight import (
+ _COLOR,
+ _ICON,
+ DIFF_CHECKS,
+ DiffCheck,
+ Outcome,
+ Scope,
+ _capability_met,
+ _run_diff_check,
+)
+from hogli_commands.test_runner import changed_test_files
+
+
+class ComplexityLayer(IntEnum):
+ STATIC = 1
+ UNIT_TEST = 2
+ OPENAPI = 3
+ STORYBOOK = 4
+ STACK = 5
+ END_TO_END = 6
+
+
+class ChangeKind(StrEnum):
+ DOCS_OR_SKILLS = "docs or skills"
+ CI_WORKFLOW = "CI workflow"
+ MIGRATION = "migration"
+ END_TO_END_TEST = "Playwright spec"
+ TEST = "test"
+ API_SCHEMA = "DRF serializer or viewset"
+ SQL_OR_HOGQL = "SQL or HogQL"
+ KEA_LOGIC = "kea logic"
+ CSS = "CSS"
+ REACT_COMPONENT = "React component"
+ DJANGO_TEMPLATE = "Django template"
+ APP_WIRING = "settings, URLs or middleware"
+ PURE_LOGIC = "pure logic"
+
+
+@dataclass(frozen=True, kw_only=True, slots=True)
+class KindRule:
+ kind: ChangeKind
+ globs: tuple[str, ...]
+ sufficient: ComplexityLayer
+
+
+# First match wins, so a narrow kind sits above a broad one that shares its files.
+KIND_RULES: tuple[KindRule, ...] = (
+ KindRule(
+ kind=ChangeKind.DOCS_OR_SKILLS,
+ globs=("*.md", "*.mdx"),
+ sufficient=ComplexityLayer.STATIC,
+ ),
+ KindRule(
+ kind=ChangeKind.CI_WORKFLOW,
+ globs=(".github/workflows/*", ".github/actions/*.yml", ".github/actions/*.yaml"),
+ sufficient=ComplexityLayer.STATIC,
+ ),
+ KindRule(kind=ChangeKind.MIGRATION, globs=("*/migrations/*.py",), sufficient=ComplexityLayer.UNIT_TEST),
+ KindRule(kind=ChangeKind.END_TO_END_TEST, globs=("playwright/*",), sufficient=ComplexityLayer.END_TO_END),
+ KindRule(
+ kind=ChangeKind.TEST,
+ globs=("test_*.py", "*/test_*.py", "*.test.ts", "*.test.tsx", "*/test/*", "*/tests/*", "*/__snapshots__/*"),
+ sufficient=ComplexityLayer.UNIT_TEST,
+ ),
+ KindRule(kind=ChangeKind.API_SCHEMA, globs=BUILD_TRIGGERS["build:openapi"], sufficient=ComplexityLayer.OPENAPI),
+ KindRule(
+ kind=ChangeKind.SQL_OR_HOGQL,
+ globs=("*.sql", "posthog/hogql/*", "posthog/hogql_queries/*", "products/*/backend/hogql_queries/*"),
+ sufficient=ComplexityLayer.UNIT_TEST,
+ ),
+ KindRule(kind=ChangeKind.KEA_LOGIC, globs=("*Logic.ts", "*Logic.tsx"), sufficient=ComplexityLayer.UNIT_TEST),
+ KindRule(kind=ChangeKind.CSS, globs=("*.css", "*.scss"), sufficient=ComplexityLayer.STORYBOOK),
+ KindRule(kind=ChangeKind.REACT_COMPONENT, globs=("*.tsx",), sufficient=ComplexityLayer.STORYBOOK),
+ KindRule(kind=ChangeKind.DJANGO_TEMPLATE, globs=("*/templates/*.html",), sufficient=ComplexityLayer.STACK),
+ KindRule(
+ kind=ChangeKind.APP_WIRING,
+ globs=(
+ "posthog/settings/*",
+ "posthog/urls.py",
+ "products/*/backend/routes.py",
+ "posthog/middleware.py",
+ "posthog/csp_middleware.py",
+ "docker-compose*.yml",
+ ),
+ sufficient=ComplexityLayer.STACK,
+ ),
+)
+
+FALLBACK_RULE = KindRule(kind=ChangeKind.PURE_LOGIC, globs=("*",), sufficient=ComplexityLayer.UNIT_TEST)
+
+COMPLEXITY_LAYERS: dict[ComplexityLayer, str] = {
+ ComplexityLayer.OPENAPI: "Start the services with `hogli up -d`, run `hogli build:openapi`, and commit the generated types.",
+ ComplexityLayer.STORYBOOK: "Render the affected stories with `pnpm storybook` and compare them before and after.",
+ ComplexityLayer.STACK: "Start the stack with `hogli up -d` and `hogli wait`, then exercise the change in the app.",
+ ComplexityLayer.END_TO_END: "Run the changed Playwright specs with `hogli test <spec>` against a running stack.",
+}
+
+STATIC_CHECK_KEYS = (
+ "ruff-lint",
+ "ruff-format",
+ "type-check",
+ "frontend-format",
+ "markdown-format",
+ "desktop-biome",
+ "agent-biome",
+ "workflow-lint",
+ "migrations",
+)
+
+
+def _streamed(command: list[str]) -> Callable[[Scope], Outcome]:
+ # Preflight keeps three lines of a failure, and the pnpm banner fills them, so the compiler
+ # errors and the typecheck wrapper's recovery steps print straight to the terminal instead.
+ def run(_scope: Scope) -> Outcome:
+ if not _capability_met("node"):
+ return "skipped", "needs node"
+ returncode = subprocess.run(command, cwd=REPO_ROOT).returncode
+ return ("pass", "ok") if returncode == 0 else ("fail", f"exit {returncode}, see the output above")
+
+ return run
+
+
+FRONTEND_TS = ("frontend/src/*.ts", "frontend/src/*.tsx", "products/*/frontend/*.ts", "products/*/frontend/*.tsx")
+
+VERIFY_ONLY_CHECKS: tuple[DiffCheck, ...] = (
+ DiffCheck(
+ key="oxlint",
+ label="frontend lint (oxlint)",
+ triggers=list(FRONTEND_TS),
+ verify=["pnpm", "exec", "oxlint", "--quiet"],
+ requires=("node",),
+ takes_files=True,
+ ),
+ DiffCheck(
+ key="typescript",
+ label="frontend typecheck (tsgo)",
+ triggers=list(FRONTEND_TS),
+ verify=None,
+ run=_streamed(["pnpm", "--filter=@posthog/frontend", "typescript:check"]),
+ ),
+ DiffCheck(
+ key="skills-lint",
+ label="agent skill sources",
+ triggers=[".agents/skills/*", "products/*/skills/*"],
+ verify=["hogli", "lint:skills"],
+ requires=("python-env",),
+ ),
+)
+
+
+@dataclass(frozen=True, kw_only=True, slots=True)
+class VerifyPlan:
+ files_by_kind: dict[ChangeKind, list[str]]
+ runs_unit_tests: bool
+ manual_layers: tuple[ComplexityLayer, ...]
+
+
+def classify(path: str) -> KindRule:
+ return next((rule for rule in KIND_RULES if matches_globs(path, rule.globs)), FALLBACK_RULE)
+
+
+def plan_verification(paths: Iterable[str]) -> VerifyPlan:
+ files_by_kind: dict[ChangeKind, list[str]] = {}
+ sufficient: set[ComplexityLayer] = set()
+ for path in paths:
+ rule = classify(path)
+ files_by_kind.setdefault(rule.kind, []).append(path)
+ sufficient.add(rule.sufficient)
+ return VerifyPlan(
+ files_by_kind=files_by_kind,
+ runs_unit_tests=any(layer >= ComplexityLayer.UNIT_TEST for layer in sufficient),
+ manual_layers=tuple(sorted(layer for layer in sufficient if layer in COMPLEXITY_LAYERS)),
+ )
+
+
+def _static_checks(files: list[str]) -> list[DiffCheck]:
+ preflight_checks = [check for check in DIFF_CHECKS if check.key in STATIC_CHECK_KEYS]
+ triggered = []
+ for check in [*preflight_checks, *VERIFY_ONLY_CHECKS]:
+ matched = [path for path in files if matches_globs(path, check.triggers)]
+ if matched:
+ triggered.append(dataclasses.replace(check, matched=matched))
+ return triggered
+
+
+def _print_plan(plan: VerifyPlan) -> None:
+ for kind, paths in plan.files_by_kind.items():
+ click.echo(f" {kind}: {len(paths)} file(s)")
+ click.echo()
+
+
+def _print_manual_layers(plan: VerifyPlan, outstanding: tuple[str, ...] = ()) -> None:
+ covered = "Lint, format, typecheck and the nearest tests" if plan.runs_unit_tests else "Lint, format and typecheck"
+ if outstanding:
+ click.secho(" This diff is not verified yet. Do these, then run `hogli verify --changed` again:", fg="yellow")
+ for step in outstanding:
+ click.echo(f" - {step}")
+ if not plan.manual_layers:
+ if not outstanding:
+ click.secho(
+ f" {covered} are enough for this diff. Slower checks would not find anything more.", fg="green"
+ )
+ return
+ click.secho(f" {covered} are not enough for this diff. Run these yourself, cheapest first:", bold=True)
+ for layer in plan.manual_layers:
+ click.echo(f" {layer.value}. {COMPLEXITY_LAYERS[layer]}")
+
+
+ name="verify",
+ help="Run the cheapest checks that cover your diff (lint, typecheck, nearest tests), then name the next potential layer",
+)
[email protected]("--changed", is_flag=True, help="Verify the files changed on this branch, plus uncommitted work.")
[email protected]("--plan", "plan_only", is_flag=True, help="Show how the diff is classified without running checks.")
+def verify(changed: bool, plan_only: bool) -> None:
+ if not changed:
+ raise click.UsageError("Pass --changed to verify the files changed on this branch.")
+ files = changed_files()
+ if not files:
+ click.secho("No changed files on this branch.", fg="yellow")
+ return
+ plan = plan_verification(files)
+ click.secho(f"\n hogli verify: {len(files)} changed file(s)\n", bold=True)
+ _print_plan(plan)
+ if plan_only:
+ _print_manual_layers(plan)
+ return
+
+ click.secho(" Layer 1: lint, format, typecheck", bold=True)
+ failures = 0
+ outstanding: list[str] = []
+ for check in _static_checks(files):
+ status, detail = _run_diff_check(check, do_fix=False, against=None, strict=False, changed=files)
+ failures += status == "fail"
+ if status == "skipped":
+ outstanding.append(f"[{check.key}] was skipped: {detail}. Set up what it needs.")
+ elif status == "advisory":
+ outstanding.append(f"[{check.key}] did not run: {detail}.")
+ click.secho(f" {_ICON[status]} [{check.key}] {check.label}", fg=_COLOR[status])
+ click.echo(f" {detail}")
+ click.echo()
+ if failures:
+ click.secho(" Fix the failures above, then run `hogli verify --changed` again.", fg="red")
+ sys.exit(1)
+
+ if plan.runs_unit_tests and not changed_test_files(skip_playwright=True).all_files:
+ outstanding.append(
+ "No tests were found for the changed files. Pick the nearest test and run `hogli test <file>`."
+ )
+ elif plan.runs_unit_tests:
+ click.secho(" Layer 2: hogli test --changed --skip-playwright", bold=True)
+ test_command = [sys.executable, "-m", "hogli", "test", "--changed", "--skip-playwright"]
+ if subprocess.run(test_command, cwd=REPO_ROOT).returncode != 0:
+ click.secho("\n Tests failed. Fix them before you run slower checks.", fg="red")
+ sys.exit(1)
+ click.echo()
+
+ _print_manual_layers(plan, tuple(outstanding))