Skip to content

ServerKit is a lightweight, modern server control panel for managing web applications, databases, and services on your VPS or dedicated server. Built with Python/Flask backend and React frontend.

ci(tests): add test-count ratchet + changelog recovery-parity (plan 42 ph6)

Juan Denis297c1e586423
diff --git a/.github/workflows/backend-ci.yml b/.github/workflows/backend-ci.yml
index 8eddb2c0..34b1142f 100644
--- a/.github/workflows/backend-ci.yml
+++ b/.github/workflows/backend-ci.yml
@@ -35,6 +35,13 @@ jobs:
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest
+ - name: Test-count ratchet
+ # Fails if collected-test count drops below backend/tests/BASELINE_COUNT.
+ # Converts "code + its tests vanished together" (the 2026-07 data loss,
+ # where a green suite hid ~363 lost tests) from invisible to a red X.
+ # Lowering the floor requires editing BASELINE_COUNT in the same commit.
+ working-directory: backend
+ run: python tests/check_test_count.py
- name: Run tests
working-directory: backend
run: python -m pytest -v
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 60258529..8ae8c7ea 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -127,6 +127,25 @@ awaiting a stable release:
with high fidelity from surviving compiled bytecode, the surviving tests, and
the plan docs. Backend boots clean, `flask db upgrade` reaches head on a fresh
DB, and the full test suite is green again.
+- **Recovery parity audit** — a follow-up audit found the recovery rebuild had
+ silently dropped more than the above: because code and its proving tests died
+ *together*, the suite stayed green while whole slices vanished. Restored the
+ Cloudflare Round 2 engine (DNSSEC status, Origin CA issue/install/revoke,
+ redirect & transform rules, the per-zone operations activity ledger, and the
+ per-product token-scope probe), the email-bounce webhook + auto-mute slice,
+ the DNS-cutover explicit-records staging path (a `NO_PROVIDER` now names the
+ provider), the Server **Survey** tab and its (previously 404-ing) API routes,
+ the cron run-history styling, and 18 lost backend test suites. Where a restored
+ test proved its feature still hollow, it is skipped with a reason naming the
+ exact missing symbol so the gap stays visible.
+- **Survey API routes returned 404** — `app/api/survey.py` was defined but never
+ registered after the rebuild; the server-detail Survey tab and management-mode
+ endpoints are reachable again.
+- **Test-count ratchet** — Backend CI now fails if the collected-test count drops
+ below a checked-in floor (`backend/tests/BASELINE_COUNT`). Lowering the floor
+ requires editing that file in the same commit, so a silent test loss (the exact
+ failure mode that hid the recovery gaps above) surfaces as a red X instead of a
+ still-green suite.
### Removed
diff --git a/backend/tests/BASELINE_COUNT b/backend/tests/BASELINE_COUNT
new file mode 100644
index 00000000..70081150
--- /dev/null
+++ b/backend/tests/BASELINE_COUNT
@@ -0,0 +1 @@
+2475
diff --git a/backend/tests/check_test_count.py b/backend/tests/check_test_count.py
new file mode 100644
index 00000000..8e68e110
--- /dev/null
+++ b/backend/tests/check_test_count.py
@@ -0,0 +1,84 @@
+#!/usr/bin/env python3
+"""Test-count ratchet — the guard the 2026-07 data loss taught us to want.
+
+Back then code and its proving tests died TOGETHER, so the suite stayed green
+(2572 -> 2209 passing) while whole features silently vanished — nobody diffs a
+green number. This checks the collected-test count against a checked-in floor
+(``backend/tests/BASELINE_COUNT``) and FAILS if collection has dropped below it.
+
+- Raising the floor is expected as tests are added — run with ``--update`` to
+ bump ``BASELINE_COUNT`` to the current count (or just edit the file).
+- Lowering it requires editing ``BASELINE_COUNT`` in the same commit — an
+ explicit, reviewable statement that tests were intentionally removed, instead
+ of a silent regression.
+
+Usage:
+ python tests/check_test_count.py # verify (CI mode) — exit 1 if below
+ python tests/check_test_count.py --update # write current count as the new floor
+
+Run from the ``backend/`` directory.
+"""
+import os
+import re
+import subprocess
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+BASELINE_FILE = os.path.join(HERE, 'BASELINE_COUNT')
+
+
+def read_baseline():
+ try:
+ with open(BASELINE_FILE) as f:
+ return int(f.read().strip())
+ except (OSError, ValueError):
+ return 0
+
+
+def collect_count():
+ """Return the number of tests pytest collects, without running them."""
+ proc = subprocess.run(
+ [sys.executable, '-m', 'pytest', '--collect-only', '-q',
+ '-p', 'no:cacheprovider'],
+ cwd=os.path.dirname(HERE), # backend/
+ capture_output=True, text=True,
+ )
+ out = proc.stdout + proc.stderr
+ # pytest prints e.g. "2475 tests collected in 6.55s" (or "1 test collected")
+ m = re.search(r'(\d+)\s+tests?\s+collected', out)
+ if not m:
+ sys.stderr.write('Could not parse a collected-test count from pytest.\n')
+ sys.stderr.write(out[-2000:] + '\n')
+ sys.exit(2)
+ return int(m.group(1))
+
+
+def main():
+ update = '--update' in sys.argv[1:]
+ collected = collect_count()
+ baseline = read_baseline()
+
+ if update:
+ with open(BASELINE_FILE, 'w') as f:
+ f.write(f'{collected}\n')
+ print(f'BASELINE_COUNT updated: {baseline} -> {collected}')
+ return 0
+
+ print(f'collected={collected} baseline={baseline}')
+ if collected < baseline:
+ sys.stderr.write(
+ f'\nTEST-COUNT RATCHET FAILED: collected {collected} tests, but the '
+ f'floor is {baseline}.\n'
+ 'Tests disappeared. If this is an intentional removal, lower '
+ f'{os.path.relpath(BASELINE_FILE)} in THIS commit; otherwise a '
+ 'feature (and its proving tests) went missing — investigate before '
+ 'merging.\n')
+ return 1
+ if collected > baseline:
+ print(f'note: {collected - baseline} new test(s) since the floor was '
+ 'set — run this with --update to raise BASELINE_COUNT.')
+ return 0
+
+
+if __name__ == '__main__':
+ sys.exit(main())