ServerKit is a lightweight, modern server control panel for managing web applications, databases, and services on your VPS or dedicated server. Built with Python/Flask backend and React frontend.
Python astracloudflarecontrol-panelcyberpaneldnsfablehosting-control-panelhosting-panelkimi-k3server-managementserverkitubuntu-web-panelwebserver
docs(plan-77): completion record — decisions, scope calls, retained tail
Juan Denisa3e907e97fff
diff --git a/backend/tests/BASELINE_COUNT b/backend/tests/BASELINE_COUNTindex 5ee7972c..8f22b44a 100644--- a/backend/tests/BASELINE_COUNT+++ b/backend/tests/BASELINE_COUNT@@ -1 +1 @@-4460+4591diff --git a/docs/plans/77_MODELS_REALTIME_AND_VOCABULARY_PLAN.md b/docs/plans/77_MODELS_REALTIME_AND_VOCABULARY_PLAN.mdnew file mode 100644index 00000000..c160f1de--- /dev/null+++ b/docs/plans/77_MODELS_REALTIME_AND_VOCABULARY_PLAN.md@@ -0,0 +1,314 @@+# Plan 77 — One shape: models, realtime, and the shared vocabulary++## Position among the plans++Plan 75 owns the **services/system doors** (subprocess, docker, nginx, SQL+exec — Round 2 §G). Plan 76 owns the **boundary contracts** (auth, errors,+schemas, server state, forms, primitives). This plan owns what neither does:+the **shape of the data itself** — how models serialize, timestamp, encrypt,+and name their states — and the **realtime layer** that carries those shapes+to the browser. Same rule as its siblings:++> For each shape, one definition; migrate writers to it; ratchet so a second+> definition cannot appear.++Source: the 2026-08-18 11-agent duplication audit ("Twenty Doors" artifact;+memory `project_dry_audit_twenty_doors`). All counts grep-verified on+2026-08-18. One finding is a **live bug shipping today** and leads the+ordering (§A).++---++## A. Fix first — the agent handshake fork already shipped a bug++`agent_gateway.on_auth` (WS) and `api/agent_poll.py /connect` (HTTP long-poll+fallback) hand-roll the same ~70-line sequence twice: rate-limit → HMAC verify+→ allowed-IPs + anomaly tracking → new-IP check → version parse → register →+failure handling. `agent_poll.py` even comments "Mirrors the on_auth handler".+Drift has shipped twice:++1. Historically: the `/connect` rate limit was missing (the credential-stuffing+ bypass documented at `agent_poll.py:57-60`, patched back in).+2. **Now, verified:** `TunnelBrokerService.schedule_reconcile` is called from+ exactly one place — `agent_gateway.py:316`, the WS path. A WireGuard-capable+ agent connected via long-poll **never self-heals its tunnels after a+ restart.**++**A1 (immediate, smallest diff):** invoke the tunnel-reconcile capability hook+from the poll path's state ingest. A proving test: capability payload arriving+via poll triggers reconcile exactly as via WS.++**A2 (structural):** extract transport-neutral services on the registry —+`agent_registry.authenticate_and_register(payload, ip, user_agent, transport)`+returning `(session_token, server, error)`, and+`agent_registry.ingest_agent_state(agent, metrics, sysinfo, capabilities)`+owning capability side-effects. `on_auth` and `/connect`+`/poll` become thin+adapters translating errors to emit/disconnect vs jsonify/status. The registry+already proves the pattern with its dual-transport `send_command`.++**Gate:** both transports run the same handshake test matrix (bad HMAC, blocked+IP, rate limit, new-IP anomaly, capability side-effects); a ratchet asserts the+shared sequence has no second implementation.++---++## B. Model mixin suite++`backend/app/models/mixins.py` exists (SoftDeleteMixin lives there). It gains+the mechanical shapes below. **Reconciliation with plan 75 §F, which correctly+ruled "never consolidate `to_dict`":** that ruling protects per-entity+*difference*. What follows removes only the *mechanical fragment* inside those+methods, judged by plan 75's own test — "would a single change otherwise have+to be made in N places?"++| Shape | Measured | The single change that proves it |+|---|---|---|+| B1 `TimestampMixin` | 104 `created_at` + 69 `updated_at` definitions, ~270 `datetime.utcnow` references | `utcnow` is deprecated on Python 3.13 (this repo already fought Trixie/3.13 once); the tz-aware migration becomes a 1-file edit instead of ~270. Columns are byte-identical — **no schema migration needed**, adopt opportunistically |+| B2 `SerializableMixin` | 138 `to_dict()` methods across 90 files; 266 identical `self.X.isoformat() if self.X else None` ternaries | a datetime-format or field-masking convention change lands once. Introspect `__table__.columns`, auto-isoformat DateTime, class-level `__serialize_exclude__` (for `*_encrypted`/token columns), an `extra()` hook for computed fields. **Models with genuinely custom shapes keep their hand-written `to_dict` — the mixin is opt-in, never forced** |+| B3 `uuid_pk()` factory | the String(36) uuid4 lambda pasted 13× across 8 files, plus one incompatible `uuid4().hex` variant (`ai.py` — 32-char, no dashes) | the format decision is made once; a third format that breaks FK joins to String(36) columns cannot appear |+| B4 finish `JsonColumnMixin` | plan 75 §F2 built it (28 adopters) but **21 model files still hand-roll `json.loads`**, most without the mixin's corruption tolerance — a corrupt row 500s every serializing endpoint | pure adoption debt; ratchet: no `json.loads(` in models/ outside the mixin |+| B5 SoftDelete roadmap | 3 adopters (application, domain, saved_view) despite the docstring calling it the one deletion pattern | **not a sweep** — adoption is deliberately per-model (mixin + migration + recycle-bin registration + partial-unique-index, per migration 083). Rank by regret: `Server` (fleet enrollment + HMAC secrets) and `ManagedDatabase` first |++**Gate:** serializer parity tests on converted models (old dict == new dict,+key for key) before each swap; the B4 ratchet.++---++## C. One crypto path — `EncryptedSecret`++17 `*_encrypted` columns across 16 model files, handled three ways:++- hand-rolled accessor pairs — `server.py:208-240` **swallows encrypt failures+ with `print()` and silently stores nothing** (a live data-loss path);+- a completely separate class-level Fernet stack in `env_variable.py:42-83`+ (key derived from `SECRET_KEY`) that `shared_resource.py` reaches into+ cross-model;+- no accessors at all (oauth_identity, cloudflare_tunnel,+ registrar_connection) — pushing crypto calls into 12 service files.++**C1:** one `EncryptedSecret` descriptor (or `encrypted_column(name)` factory)+in `models/mixins.py` wrapping `utils/crypto`: assignment encrypts —+**raising, never printing, on failure** — read decrypts, and it standardizes+the `has_secret` masking already duplicated in three `to_dict`s.++**C2:** fold the parallel Fernet stack into it so there is one key-derivation+path. Migration is dual-read (try new path, fall back to old), encrypt-on-write+— **never a bulk decrypt-all→re-encrypt-all pass**: the PR #94 postmortem+records that exact approach double-wrapping credentials.++**Gate:** round-trip tests per column; a test that an encrypt failure raises;+ratchet: no `import` of `utils/crypto` from services for model-owned columns.++---++## D. One status vocabulary — database to pixels++The same terminal state is spelled four ways across the run-shaped models+(`success` in cron_run/backup_run, `completed` in db_snapshot and+environment_activity, `done` in sandbox runs, `succeeded` once) — 55 free-text+status columns, only ~6 files with constants. The frontend mirrors the chaos:+**19 files define their own status→color map, so "running" renders cyan,+green, or amber depending on the page**, split across two disjoint tone+vocabularies (ds/Pill kinds vs ui/badge variants); SCSS re-rolls the+state-color triple 83× across 32 files.++- **D1** `backend/app/models/status.py`: canonical lifecycle constants+ (PENDING / QUEUED / RUNNING / SUCCESS / FAILED / CANCELLED + a `TERMINAL`+ frozenset). New columns and API filters import it; existing domains migrate+ per-domain with a data migration when stored strings change (see open+ questions).+- **D2** `RunLifecycleMixin`: `mark_running()` / `mark_succeeded()` /+ `mark_failed()` / `mark_cancelled()` setting canonical status ++ started_at/completed_at/duration. Ten run models hand-write these+ transitions today (`deployment_job_service` writes `completed_at` in 11+ places). This is the model half of the already-open runs-consolidation item.+- **D3** frontend `ds/status.js`: one `statusKind(status)` map covering the+ union of job/deploy/service/backup states; StatusBadge, Pill call sites, and+ the connections tones all consume it; unify the two tone vocabularies so one+ table drives both primitives.+- **D4** SCSS `@mixin status-variant($color, $bg, $border)` + one canonical+ pill class; pages position the pill, never recolor it. Deletes the 83+ re-rolled blocks.++**Gate:** a parity test that every status string a run model can write has an+entry in status.py, and every status.py entry has a `statusKind`; ratchet on+per-page STATUS_KIND/getStatusColor definitions.++---++## E. One realtime door++Five parallel log/progress streaming implementations exist: `RunLogStream`+(persist + batch emit + `after_id` resync — the best, but hard-coupled to+`DeploymentJob`), two hand-rolled paths inside `sockets.py` (file logs;+container logs with no persistence/resync), the agent `on_stream` rebroadcast,+and a **dead `pipeline_*` channel with zero callers**. Meanwhile the unified+Job runtime has no live channel at all — Jobs, sandbox runs, site imports, and+backups each poll bespoke REST endpoints. Channel plumbing repeats too: 8+subscribe/unsubscribe pairs with four registry shapes and inconsistent auth+re-checks; room-name grammar lives in ~25 scattered f-strings across+sockets.py, agent_gateway.py, notifications, and is re-derived by hand in+frontend template literals.++- **E1** Generalize `RunLogStream` to a run-scoped envelope keyed by+ `(run_kind, run_id)`: one room grammar `run_<kind>_<id>`, one batched+ persisted `run_log` + `run_status` event pair, one `GET .../logs?after_id`+ polling twin. Accept any model implementing a small RunLike protocol+ (natural fit: anything carrying D2's `RunLifecycleMixin`). Adopt for unified+ Jobs, sandbox runs, site imports, backups. Retire the dead `pipeline_*`+ scaffolding.+- **E2** Declarative channel registry in sockets.py:+ `register_channel(name, room_fn, auth=predicate, on_first_subscribe=...,+ on_last_unsubscribe=...)` generating the handler pairs with one auth model.+- **E3** `backend/app/sockets_rooms.py` (or `rooms.py`) owning every room-name+ builder; mirrored event-name constants (`backend/app/constants.py` ↔+ `frontend/src/constants/events.js`) so a renamed event cannot fail silently.+- **E4** frontend `useServerStream(room, event, handler)` wrapping the+ join/listen/cleanup dance currently hand-rolled in 3 components.+- **E5** `BackgroundLoop` helper (Event-based stop, idempotent start/stop,+ app-context, logger error handling, auto-registration in+ `thread_ownership`): the 13 classified LIFECYCLE_PROCESS_LOOP threads each+ hand-roll the loop today (bool flags vs Event; sockets.py reports errors via+ `print()` — unreadable in prod).++**Gate:** stream contract test (emit → persisted row → `after_id` resync+returns it); channel auth test per registered channel; ratchet on raw+`socketio.on('subscribe_` handler definitions outside the registry.++---++## F. Shared infra utilities — five re-inventions, one home each++| Utility | Measured re-invention | One door |+|---|---|---|+| F1 remote catalog engine | the fetch → TTL → last-good → bundled-fallback pipeline is re-implemented in **4 services** (registry, theme_registry, template remote, security_feed), ~80–150 lines each; this family has already drifted in production (bundled-index and proxy-rewrite incidents) | `utils/remote_index.CachedRemoteIndex(env_var, default_url, bundled_path, ttl, error_ttl, normalize_fn)`; services become thin normalizers; future catalogs get the hardened behavior free |+| F2 TTL caches | 7 services hand-roll module-level TTL dicts despite `CacheService` (one file has two caches *and* imports cache_service) | a `ttl_cached(ttl, key_fn=None)` decorator on CacheService; Redis benefits apply wherever configured |+| F3 sensitive-data redaction | **4 diverging keyword lists** (utils filter, audit_service, middleware/audit — the only one covering totp/otp/csrf/session — telemetry) + 3 extra maskers; a keyword added to one list is silently not redacted by the other three sinks | `utils/sensitive_data_filter.py` exports the union `SENSITIVE_KEY_PARTS`, `is_sensitive_key()`, `mask_payload()`; the four sinks import it |+| F4 env access | 70 call-time `os.environ.get` across 28 app files bypass config.py; truthiness parsed in **3 incompatible variants** — `SERVERKIT_ALLOW_PRIVATE_DOWNLOADS=on` silently reads false in plugin_service | `utils/env.py` with `env_str`/`env_bool`/`env_int` (env_bool = config.py's `_bool`, moved out and imported back) |++**Gate:** per-utility unit tests; ratchet greps (no new module-level+`_cache = {}` + timestamp pairs; no new `SENSITIVE` lists; no+`.lower() in (` truthiness tuples outside utils/env.py).++---++## G. Test one-doors++The seed abstractions already exist in `conftest.py` — underscore-private.++- **G1** Promote `_mk_scope_user`/`_scope_token` to public+ `make_user(db, username, role='developer', **overrides)` and+ `headers_for(user)`; add `viewer_headers`/`developer_headers` persona+ fixtures beside the existing admin `auth_headers`. Today **45 test files+ mint their own JWT headers**; role is written three ways and passwords set+ three ways.+- **G2** `backend/tests/factories.py`: `make_application`, `make_server`,+ `make_workspace` with defaults + `**overrides`. `Application(` is constructed+ directly in **67 files**; `_seed_app` is byte-identical in at least two,+ the server fixture in three. Every schema change is currently absorbed by+ dozens of hand-rolled seed blocks.+- **G3** Guard against direct `create_app()`: **16 files** boot it around the+ plan-64 session-scoped fixture (one carries a justifying comment that is now+ false), re-exposing the state-leak classes conftest documents. Add a+ session-scoped `route_rules` fixture so no test boots an app just to read+ `url_map`, and extend `test_fixture_scope_guard.py` to assert no module+ calls `create_app` outside conftest without the `fresh_app` pattern.+- **G4** Promote `assert_requires_auth` / `assert_admin_only` (the shape exists+ file-private in `test_raw_infra_authz.py`) so per-endpoint 401/403 micro-tests+ (45 functions across 36 files) become one-liners.++The shared **subprocess stub kit is plan 75 G7**, not here — it must move in+lockstep with the `run_checked` migration.++**Gate:** the G3 guard test; migrate byte-identical helper files first as the+proof, then ratchet new-file usage via a conftest-level grep test.++---++## Ordering++| # | Workstream | Effort | Why |+|---|---|---|---|+| 1 | A1 — poll-path tunnel reconcile | trivial | **live bug**; a WireGuard agent on the fallback transport does not self-heal |+| 2 | C1 — `EncryptedSecret`, `server.py` first | low | the `print()`-swallowed encrypt failure is silent data loss |+| 3 | G1/G2 — personas + factories | low | enabler: every later migration writes tests through them |+| 4 | B1/B3/B4 — Timestamp, uuid_pk, JsonColumn finish | low | mechanical, no schema migrations, utcnow debt shrinks |+| 5 | D1/D2 — status constants + RunLifecycleMixin | medium | unblocks runs-consolidation and E1's RunLike protocol |+| 6 | A2 — transport-neutral handshake | medium | makes A1 structural instead of patched |+| 7 | E1–E4 — stream envelope + channels | high | the largest piece; lands on top of D2 |+| 8 | B2 — SerializableMixin, opportunistic | medium | parity-tested swaps, never forced |+| 9 | F1–F4 — infra utilities | low each | independent; F3 (redaction) first for the security surface |+| 10 | D3/D4 — frontend status map + SCSS variant | medium | after D1 so the vocabulary is settled |+| 11 | C2 — Fernet fold-in | medium | dual-read migration, scheduled alone |+| 12 | B5 — SoftDelete per-model adoptions | per-model | each is a deliberate product decision |++## Deliberately not in scope++- API/validation/auth boundary contracts — plan 76.+- Subprocess/docker/nginx/SQL-exec doors — plan 75 Round 2.+- Consolidating `to_dict` bodies that express genuine per-entity shape (plan 75+ §F's ruling stands; B2 removes only the mechanical fragment, opt-in).+- A generic runs UI — D2/E1 make one *possible*; building it is its own plan.+- Changing stored status strings wholesale — per-domain, with data migrations.++## Open questions++- **D1 stored-string migration:** map legacy spellings at the read edge+ (cheap, but the lie persists in the DB) or data-migrate per domain (clean,+ but each is a migration + deploy consideration)? Leaning per-domain+ migrations, smallest tables first.+- **C2 key derivation:** env_variable's Fernet derives from `SECRET_KEY`;+ `utils/crypto` has its own scheme. Which survives? Rotating `SECRET_KEY`+ must not brick stored env vars — the dual-read window may need to be+ permanent for rows never rewritten.+- **B2 field-selection interplay:** `api/_query.py` already does response+ field selection. Does `SerializableMixin` feed it (mixin produces the full+ dict, _query filters) or replace part of it? Decide before converting the+ first model.+- **E1 event compatibility:** existing `deploy_log`/`deploy_status` listeners+ ship in the frontend today. Dual-emit during migration, or migrate the+ deploy console in the same commit?++---++## Completion record — 2026-08-19++Executed in full (one session, ~30 commits on `dev`, +131 collected tests →+BASELINE_COUNT 4591). Every workstream landed with its gate; every ratchet+named below is a live test.++**Open questions, resolved:**+- *D1 stored strings*: read-edge `normalize()` shipped; per-domain data+ migrations remain deliberate follow-ups. `RunLifecycleMixin` carries+ per-model spelling overrides (DeploymentJob keeps `succeeded`) so adoption+ never silently rewrites a domain's vocabulary.+- *C2 key derivation*: `SERVERKIT_ENCRYPTION_KEY` (utils/crypto) survives.+ env_variable + sso_service write through it and dual-read (one path first,+ legacy `SECRET_KEY` Fernet second). The dual-read window is permanent —+ rows never rewritten decrypt forever; no bulk re-encrypt pass exists.+- *B2 field selection*: the mixin produces the full dict; `api/_query.py`+ keeps owning response shaping. (Hook is `serialize_extra()` — ServerMetrics+ has a column literally named `extra`.)+- *E1 event compatibility*: dual-emit. `run_log`/`run_status` are canonical;+ the deploy kind also emits legacy `deploy_log`/`deploy_status` until the+ Deploy Console migrates to the envelope.++**Deliberate scope calls (with owners for the tail):**+- B5: ManagedDatabase adopted (migration 088; delete keeps content+policy,+ purge never drops data, `?drop=true` stays destructive). **Server (98+ query sites, fleet enrollment + HMAC secrets) is the next ranked adoption+ and needs its own pass.**+- F1: registry + themes ride `CachedRemoteIndex`; template_service (keyed+ per-repo cache w/ targeted invalidation) and security_feed (DB-durable+ last-good for a daily job) are documented non-adopters.+- E2: `subscribe_logs`/`subscribe_container_logs` stay raw (per-sid OS+ resources), frozen by the channel-registry ratchet until they migrate.+- D3: `running` is cyan for run-shaped surfaces; `serviceStatusKind` (in the+ same file) keeps steady-state surfaces green. Status-page/WAF/metric-series+ maps are domain vocabularies, allowlisted in `check-status-one-door.mjs`.++**Known pre-existing failures (not from this plan, all Windows-env):**+sbin resolution (×26), host_inventory (×7), firewall_detection (×3),+fleet_doctor DNS, getloadavg. Full Windows runs must+`--deselect tests/test_micro_cache.py::test_purge_wipes_and_recreates_cache_dirs`+(fails on `os.geteuid` under a patched `os.name` and aborts pytest).