Skip to content

ServerKit is a lightweight, modern server control panel for managing web applications, databases, and services on your VPS or dedicated server. Built with Python/Flask backend and React frontend.

docs(wordpress): split #34 into an executable per-site edge + scaling spec

Juan Denis4a637c718154
diff --git a/docs/HORIZONTAL_SCALING_SPEC.md b/docs/HORIZONTAL_SCALING_SPEC.md
new file mode 100644
index 00000000..dac34386
--- /dev/null
+++ b/docs/HORIZONTAL_SCALING_SPEC.md
@@ -0,0 +1,218 @@
+# ServerKit — Per-Site Edge & Horizontal Scaling Spec
+
+*Splitting the #34 epic — and specifying the **keystone** it shares with #21, #15-TLS, #30-jail, and #8/#25.*
+
+> **Status:** design only. No code shipped. This doc exists so #34 (and its
+> prerequisites) can be executed in focused passes **on a real Linux/Docker
+> host**, where the infra can actually be verified — which is exactly why it
+> wasn't built blind from the Windows dev environment.
+
+---
+
+## 1. Why this doc
+
+`docs/WORDPRESS_ROADMAP.md` defers **#34 (horizontal scaling)** as *"XL + blocked on
+the same per-site reverse-proxy/upstream layer as #21."* That deferral is correct,
+but it undersells how much already exists. A capability sweep of the nginx/SSL/storage
+services shows the **load-balancing, rate-limiting, caching, per-vhost-log, wildcard-TLS,
+and S3-upload primitives are already built** — they're just bound to the generic
+`Application`/domain model, never to a `WordPressSite` running as a `localhost:PORT`
+container.
+
+So the real work is three layers, in order:
+
+1. **The keystone** — a per-site *edge* (public domain → reverse-proxy vhost → the WP
+ container) with auto-TLS. This is the missing infrastructure that blocks **#21, #15-TLS,
+ #30-jail, #8-backend, and the original #25-nginx design** — and it is #34's prerequisite.
+2. **Stateless WP** — shared media + shared object cache so replicas are interchangeable.
+3. **Replicas + upstream LB** — multiple WP containers behind the keystone vhost.
+
+This doc specs all three as numbered phases (`K0–K4` keystone, `H0–H4` scaling), each with
+**Goal / Do / Reuse / Acceptance / Verifiable-here? / Risk**, plus a dependency graph and a
+real-host verification checklist.
+
+---
+
+## 2. Current reality (verified against the code)
+
+| Fact | Evidence |
+|---|---|
+| A managed WP site is a single `wordpress:${VERSION}-apache` container publishing `${HTTP_PORT}:80` straight to the host; per-site MySQL + per-site Redis (#23). **No domain, no per-site vhost at create.** | `backend/templates/wordpress.yaml` (`ports: "${HTTP_PORT}:80"`) |
+| `site.url` falls back to the host port unless a domain is attached. | `wordpress_service.py` `_enrich_site_data` (#3) |
+| **A container-proxying vhost primitive already exists**, incl. HTTPS via cert + HTTP→HTTPS redirect. | `nginx_service.py:391` `create_site(..., ssl_cert, ssl_key)`, `:450` `_with_ssl`, `:535` `add_ssl_to_site` |
+| **Upstream load-balancing, `limit_req`, `proxy_cache`, and per-vhost log reads already exist** — bound to a generic `domain`/`data` dict, not to a WP site. | `nginx_advanced_service.py:31` `create_reverse_proxy` (`upstreams`→`upstream{}` at `:47`, `proxy_pass http://{upstream}` at `:95/:113`, `limit_req_zone` at `:61`, `proxy_cache_path` at `:68`), `:173` `get_vhost_logs`, `:196` `get_load_balancing_methods` |
+| Wildcard cert issuance + auto-renew already exist. | `advanced_ssl_service.py:39` `issue_wildcard_cert(domain, dns_provider, credentials, email)`, `ssl_service.py:256` `setup_auto_renewal` |
+| Subdomain generation for environments already exists. | `environment_domain_service.py:61` `generate_domain(production_domain, env_type, …)` |
+| S3/B2/MinIO directory upload already exists (for media offload). | `storage_provider_service.py:260` `upload_directory(local_dir, remote_prefix)` |
+
+**Conclusion:** the gap is *wiring + a public base-domain + statelessness*, not greenfield nginx
+work. That makes both the keystone and #34 far smaller than "XL from zero" implies — but every
+phase touches nginx apply/reload, cert issuance, or container orchestration, **none of which is
+runtime-verifiable on Windows.**
+
+---
+
+## 3. Dependency graph
+
+```
+ ┌─────────────────────────── Part 1: KEYSTONE (per-site edge) ───────────────────────────┐
+ │ K0 public base-domain + wildcard DNS/TLS │
+ │ │ │
+ │ ▼ │
+ │ K1 per-site reverse-proxy vhost ──► K2 auto-TLS per site ──► K3 preview URL (#21) │
+ │ │ │ │
+ │ ▼ ▼ │
+ │ K4 per-site access log + limit_req ──► unblocks #30-jail, restores #25-nginx │
+ └────────┬────────────────────────────────────────────────────────────────────────────┘
+ │ (also closes: #15-TLS, #8-backend SSL wiring, #3 already done)
+ ▼
+ ┌─────────────────────────── Part 2: #34 HORIZONTAL SCALING ─────────────────────────────┐
+ │ H0 stateless WP (shared media + shared object cache) │
+ │ ▼ │
+ │ H1 replica orchestration (internal network; edge publishes, not the container) │
+ │ ▼ │
+ │ H2 upstream LB wiring ──► H3 health/rolling ──► H4 autoscale signal (optional) │
+ └────────────────────────────────────────────────────────────────────────────────────────┘
+```
+
+**Do K0→K2 first** — it alone closes #15-TLS, #8-backend, and #21 (with K3), independent of #34.
+**#34 cannot start before H0**, and H0/H1 require the keystone's edge to exist (replicas can't each
+publish a host port).
+
+---
+
+## 4. Part 1 — The keystone: per-site edge
+
+### K0 — Panel-wide public base domain + wildcard DNS/TLS `[M]`
+- **Goal:** a configurable base domain (e.g. `apps.example.com`) with `*.apps.example.com` DNS and a wildcard cert, so *any* managed site can get a real public URL. This is the single setting whose absence makes every public-URL feature impossible today.
+- **Do:** add a `base_domain` + DNS-provider binding to settings; on save, issue/refresh the wildcard cert and schedule renewal. Validate the wildcard record resolves before declaring it ready.
+- **Reuse:** `advanced_ssl_service.issue_wildcard_cert` (`:39`), `ssl_service.setup_auto_renewal` (`:256`), the existing DNS-provider credentials (Connections / `provider_secret` encryption).
+- **Acceptance:** with a base domain configured, `dig *.base` resolves and a valid `*.base` cert exists.
+- **Verifiable here?** Config/model/validation logic yes; DNS + cert issuance **real host only**.
+- **Risk:** registrar/DNS-provider variance; wildcard issuance needs DNS-01 (already supported). If no public base domain is set, the whole edge degrades gracefully to today's `localhost:PORT` behavior — keep that fallback.
+
+### K1 — Per-site reverse-proxy vhost at create/attach `[L]`
+- **Goal:** put an nginx vhost in front of each managed WP container so it answers on a real hostname instead of `localhost:PORT`.
+- **Do:** on create (and on domain-attach), call `NginxService.create_site(name, app_type='docker', domains=[hostname], port=<container published port>)` + `enable_site` + `reload`. Persist the hostname on `WordPressSite`. Make WP proxy-aware: set `WP_HOME`/`WP_SITEURL` to the hostname, trust `X-Forwarded-Proto`/`-For` in `wp-config`, and run the serialized-safe `search-replace` (#5/#9 primitive) from `localhost:PORT` → the hostname.
+- **Reuse:** `nginx_service.create_site/enable_site/reload` (`:391/:483/:288`), `db_sync_service` search-replace, the `domains.py` attach flow (#8 already wires domain-attach for generic docker apps).
+- **Acceptance:** a freshly created site is reachable at `http://<hostname>` and `wp-admin` loads with correct URLs (no redirect loop, no mixed `localhost` links).
+- **Verifiable here?** Template rendering + the `wp-config`/search-replace logic yes; nginx apply + live HTTP **real host only**.
+- **Risk:** the classic WP-behind-proxy redirect loop (needs `X-Forwarded-Proto` handling); a port is still published in K1 (internal-only comes in H1).
+
+### K2 — Auto-TLS per site `[M]`
+- **Goal:** every site is HTTPS by default.
+- **Do:** for a `*.base` subdomain, serve the K0 wildcard cert via `create_site(ssl_cert=…, ssl_key=…)` (the `_with_ssl` path already does the 80→443 redirect). For an attached custom domain, issue a per-host LE cert. Add `ssl_status` to `WordPressSite` and surface grade/expiry (the #8 `SiteSSLPanel` already renders this once a domain is routed).
+- **Reuse:** `nginx_service.create_site(ssl_cert, ssl_key)` (`:391`) / `add_ssl_to_site` (`:535`), `ssl_service`/`advanced_ssl_service`, the #8 frontend.
+- **Acceptance:** the site loads over HTTPS with a valid cert; HTTP redirects to HTTPS.
+- **Verifiable here?** `ssl_status` model + wiring yes; cert + TLS handshake **real host only**.
+- **Risk:** custom-domain certs need DNS pointed first — keep a "pending DNS" state.
+
+### K3 — Preview URL before DNS (#21) `[S–M, on top of K0–K2]`
+- **Goal:** a working HTTPS preview link the instant a site exists, pre-DNS.
+- **Do:** mint `<site|hash>.preview.<base>` via the K0 wildcard (covered by the wildcard cert, so no per-host issuance). Optionally a basic-auth gate (the env-pipeline already has basic-auth, P4). This **is** #21 once K0–K2 exist.
+- **Reuse:** `environment_domain_service.generate_domain` (`:61`), K0 wildcard, `create_site`.
+- **Acceptance:** every new site shows an immediately-working `https://….preview.base` link.
+- **Verifiable here?** Hostname generation yes; live preview **real host only**.
+
+### K4 — Per-site access log + `limit_req` jail (restores #25-nginx, unblocks #30-jail) `[M]`
+- **Goal:** a per-site nginx access log on the host + brute-force protection — the exact signals #25 and #30 had to defer because the container model exposes none.
+- **Do:** emit a per-site `access_log` path in the K1 vhost. Add a `limit_req` zone for `wp-login.php`/`xmlrpc.php` via `create_reverse_proxy` (it already emits `limit_req_zone`/`limit_req`). Point #25's analytics at `get_vhost_logs` (a real per-vhost log) instead of/alongside the `docker logs` source it fell back to; this also restores `%D` response-time once we own the `LogFormat`.
+- **Reuse:** `nginx_advanced_service.create_reverse_proxy` (`:31`, `limit_req` at `:61/:85`), `get_vhost_logs` (`:173`), `security_service` Fail2ban (optional, watching the new log).
+- **Acceptance:** repeated `wp-login` hits get throttled; per-site visits/5xx/slow-pages come from the vhost log.
+- **Verifiable here?** Log-parsing + zone-rendering yes; live throttling + log emission **real host only**.
+
+**After Part 1:** #15-TLS ✅, #21 ✅, #8-backend ✅, #30-jail ✅, #25 (response-time/slow-pages) ✅ — independent of #34.
+
+---
+
+## 5. Part 2 — #34 horizontal scaling
+
+> Built entirely on the keystone. The mental shift: in H1 the WP container stops publishing a
+> host port; the **edge vhost (K1)** becomes the only public entry and load-balances across
+> internal replicas.
+
+### H0 — Make WP stateless `[L]`
+- **Goal:** replicas must be interchangeable — no replica-local state.
+- **What's already fine:** the **object cache is already shareable** — the #23 Redis is one service in the site's compose stack, so N WP replicas pointing at the same `redis` service share it automatically. No change needed there.
+- **The real statefulness is two things:**
+ 1. **Uploads / `wp-content/uploads`** (local disk per replica). **Decision (open):**
+ - *Option A — S3/object offload* (a media-offload plugin writing to the site's configured `storage_provider`). Pro: truly stateless, scales infinitely. Con: a plugin dependency (mild tension with the WP-CLI-over-plugin thesis), egress cost, and rewriting existing media URLs. Reuse `storage_provider_service.upload_directory` (`:260`) + creds.
+ - *Option B — shared volume* (NFS/CIFS or a Docker named volume shared across replicas on one host). Pro: no plugin, transparent. Con: needs shared-FS infra; single-host only unless NFS; locking edge-cases.
+ - **Recommendation:** Option A for multi-host scale; Option B as a single-host MVP. Spec both; pick at execution.
+ 2. **On-disk page cache** — if #22 landed a *disk-writing* page-cache plugin, that's replica-local and must be moved to Redis or disabled under LB. Verify #22's cache backend before H1.
+- **Acceptance:** uploading media on replica A is immediately served by replica B.
+- **Verifiable here?** Offload-config + URL-rewrite logic yes; cross-replica consistency **real host only**.
+- **Risk:** media migration for *existing* sites; plugin licensing for offload.
+
+### H1 — Replica orchestration `[L]`
+- **Goal:** run N WP containers for one site.
+- **Do:** change the stack so the WP service is replica-able (`deploy.replicas` under swarm, or N enumerated services / `docker compose up --scale` on a compose host) on an **internal** network — **drop the `${HTTP_PORT}:80` host publish**; only the edge nginx reaches them. This is a **breaking change to the single-container model**, so it needs a per-site opt-in + a migration path (existing sites stay single-container until "Enable scaling").
+- **Reuse:** the `wordpress.yaml` template + the `_ensure_*_in_stack` additive-compose pattern (#23 redis injection is the proven precedent).
+- **Acceptance:** `docker ps` shows N WP replicas for the site, none publishing a host port.
+- **Verifiable here?** Template rendering yes; actual replicas **real host only**.
+- **Risk:** the publish→internal flip must be coordinated with K1 (edge must exist first or the site goes dark); MySQL stays single (not replicated here — that's a separate epic).
+
+### H2 — Upstream load-balancing wiring `[M]`
+- **Goal:** the edge vhost balances across the H1 replicas.
+- **Do:** upgrade the K1 vhost to an upstream-backed one: feed the replica endpoints to `create_reverse_proxy(upstreams=[…], method=…)`. Anonymous WP is stateless → round-robin; logged-in/cart needs affinity → `ip_hash` or cookie-sticky (expose via `get_load_balancing_methods`).
+- **Reuse:** `nginx_advanced_service.create_reverse_proxy` (`upstream{}` at `:47`, `proxy_pass` at `:95`), `get_load_balancing_methods` (`:196`).
+- **Acceptance:** requests distribute across replicas; killing one replica doesn't drop traffic.
+- **Verifiable here?** Upstream-config rendering yes; live balancing/failover **real host only**.
+- **Risk:** session affinity for WooCommerce carts; cache coherence across replicas (mitigated by H0 shared Redis).
+
+### H3 — Health checks + rolling deploys `[M]`
+- **Goal:** safe scale up/down and zero-downtime deploys.
+- **Do:** per-replica health probe; drain a replica out of the upstream before stop; a scale API on the `WordPressSite`. Tie deploys (#13/#29) into a rolling pattern: update one replica, health-check, proceed.
+- **Reuse:** `environment_health_service` (#26 poller), `nginx_advanced_service` (rewrite upstream on scale), #29 safe-update health gate.
+- **Acceptance:** scaling and deploys cause no failed requests.
+- **Verifiable here?** Orchestration logic yes; zero-downtime claim **real host only**.
+
+### H4 — Autoscale signal (optional) `[M]`
+- **Goal:** scale on load.
+- **Do:** drive H3's scale API from the #25 per-site traffic metrics (now real, via K4). Thresholds + cooldown.
+- **Reuse:** #25 analytics, H3 scale API.
+- **Acceptance:** sustained load adds a replica; quiet removes one.
+
+---
+
+## 6. Verifiable on Windows vs. needs a real Docker host
+
+| Verifiable here (pytest / unit) | Real Linux+Docker host only |
+|---|---|
+| `WordPressSite` schema additions (hostname, ssl_status, scaling flags, replica count) | nginx config **apply + reload** |
+| nginx **template/string rendering** (vhost, upstream, limit_req, ssl block) | Let's Encrypt / wildcard **cert issuance** |
+| `wp-config` mutation + serialized search-replace logic | live HTTP / HTTPS, redirect-loop checks |
+| service unit logic with mocked shell/Docker | **container replicas**, `--scale`, internal networking |
+| API routes + RBAC + workspace scoping | **load-balancing / failover** behavior |
+| media-offload config + URL-rewrite logic | cross-replica media + cache consistency |
+| `limit_req` zone generation | actual brute-force **throttling** |
+
+→ Each real-host item gets a **manual verification checklist** in the executing PR; nothing infra
+ships claimed-working until exercised on a host (per the project's verify-before-claiming bar).
+
+---
+
+## 7. Sequencing recommendation
+
+1. **K0 → K1 → K2** — the highest-value slice. Closes **#15-TLS, #8-backend**, and (with **K3**) **#21**. Ship and verify on a host *before* touching #34.
+2. **K4** — closes **#30-jail** and restores **#25** response-time/slow-pages.
+3. **H0 → H1 → H2** — the #34 MVP (stateless + replicas + LB). Gate H1 behind a per-site "Enable scaling" opt-in so the single-container model stays the default.
+4. **H3 → H4** — production hardening + autoscale.
+
+Each arrow is a separately shippable, separately verifiable PR. **#34's "Done when" (N replicas behind
+a balancer with consistent media + cache) is reached at the end of H2.**
+
+---
+
+## 8. Open decisions (resolve at execution)
+
+- **Media:** S3 offload (Option A) vs shared volume (Option B) — pick per deployment topology (multi-host ⇒ A).
+- **Port model:** the K1-publishes-a-port → H1-internal-only flip is a breaking change; needs a migration for existing sites and a clear opt-in.
+- **MySQL:** stays single-instance in this spec. DB replication/clustering is a distinct epic, explicitly out of scope for #34.
+- **Panel vs sites:** the *panel's* single-gevent-worker constraint (agent gateway, see `CLAUDE.md`/`ARCHITECTURE.md`) is unrelated — this LB is for the **managed sites**, not the panel. Don't conflate.
+- **Plugin thesis:** media-offload (Option A) and any disk→Redis page-cache move may need plugins; weigh against the WP-CLI-over-plugin preference.
+
+---
+
+*Cross-references: `docs/WORDPRESS_ROADMAP.md` #34, #21, #15, #8, #25, #30, #22, #23. This spec is the
+"split before starting" the roadmap asks for on every `XL`.*
diff --git a/docs/WORDPRESS_ROADMAP.md b/docs/WORDPRESS_ROADMAP.md
index 058520c0..ad0ec4ab 100644
--- a/docs/WORDPRESS_ROADMAP.md
+++ b/docs/WORDPRESS_ROADMAP.md
@@ -306,12 +306,13 @@ Highest-effort, broadest surface — but by now the `WordPressSite` is a first-c
- **Landed (per-site ACL — backend):** per-resource **sharing** — an owner/admin can grant a specific user access to a specific application (transitively its WordPress site / DB / domains) without that user owning it (access-level; per-grant view/edit roles reserved for later via a `role` column). New `ResourceGrant` model (Alembic **016**, unique per user×resource) + `ResourceGrantService` (grant/revoke/list/`user_has_grant`/`granted_ids`). **Enforcement at the cleanest seams:** `WorkspaceService.scope_query` gained `grant_resource_type` so granted apps appear in the apps list (own **OR** granted, then workspace-narrowed); `get_app` and — crucially — the WordPress `_owner_or_admin_app` helper became grant-aware, so **one change opens every WP per-site route** to a grantee. Grant-management API on `apps.py` (`GET/POST/DELETE /apps/<id>/grants`, owner-or-admin; can't grant to the owner; revoke is resource-scoped so you can't revoke another app's grant by id). Also uses `user.id` (int) not the stringified `get_jwt_identity()`. Verified: **4 new ACL tests** (visibility via list, get_app, WP route access, management perms + idempotency + revoke), full suite **127 passed**, end-to-end `alembic upgrade` to `016`. **Sharing UI (shipped):** a per-app **Share** action in the Workspaces "Resources" modal opens an inline sub-view (grant a user → access list with revoke; the owner and existing grantees are filtered out of the add-list); client methods `getAppGrants`/`grantAppAccess`/`revokeAppAccess`. Verified: frontend lint-clean + build. **Viewer/editor roles + operate access (shipped):** grants now carry a real **role** — `editor` (view + operate) or `viewer` (read-only) — and the generic app **operate** endpoints became grant-aware. Two helpers split it: `_can_access_app` (owner/admin/**any** grant) gates reads (`get_app`, `/linked`, `/logs`, `/container-logs`, `/containers`, `/status`), `_can_edit_app` (owner/admin/**editor** grant) gates mutations (`/link`, `/unlink`, `/environment`, `PUT`, `/start`, `/stop`, `/restart`); `ResourceGrantService.grant_role` backs the split. **Delete stays owner-or-admin** (an editor can't delete — tested). The grant POST validates `role ∈ {viewer, editor}` (default editor); the Sharing UI gained a role selector. The same sweep **fixed the latent int-vs-str ownership bug** across ~14 app endpoints (they compared `app.user_id` to the stringified `get_jwt_identity()` — now `user.id`). Verified: **3 more ACL tests** (viewer read-only + can't-operate, editor operates + can't-delete, role validation/default), full suite **130 passed**, frontend lint-clean + build. **Cross-blueprint sweep (shipped):** the per-resource grant now extends to every app-related blueprint, not just apps.py + WordPress. The two access helpers were promoted to a shared seam (`ResourceGrantService.can_access_app` / `can_edit_app`; `apps.py` delegates), and **33 ownership checks across 6 blueprints** were made grant-aware, categorized read-vs-mutate (via a parallel mapping workflow): `builds.py` (12), `domains.py` (7, incl. SSL enable/disable/renew + domain CRUD = editor), `deploy.py` (6), `private_urls.py` (5), `templates.py` (2), `databases.py` (1). Reads → any grant; mutates → editor grant. The sweep also **fixed the same latent int-vs-str ownership bug** in all 33 (now `user.id`). Verified: a cross-blueprint integration test (grantee reaches `builds` deployments; stranger 403), full suite **131 passed**. **`python.py` (shipped):** re-reading showed every mutating endpoint there is independently `@admin_required` (runs before the shared `get_app_or_404` gate), so the gate only governs the 4 read GETs — pointing it at `can_access_app` safely opens those to grantees while the admin-only mutations are untouched. No per-method split needed. With this, **every app-related blueprint is grant-aware.** Verified: a python read-gate test (stranger 403, grantee passes), full suite **132 passed**.
- **Landed (domains/databases scoping + role reconciliation):** the two named remaining scoping slices, plus the reconciliation primitive that makes the per-workspace role *mean* something. **(1) Domains** — `get_domains` now scopes through the parent `Application` (domains carry no `workspace_id` of their own): it resolves the active workspace and lists the domains of the apps `scope_query` returns (own + granted, workspace-narrowed). With no workspace context this preserves prior behavior (admin → all domains, owner → own apps' domains) *and* now also surfaces **granted** apps' domains — closing a latent gap where the per-domain routes authorized via `can_access_app` but the list ignored grants. Since `Domain.application_id` is non-nullable, the admin-no-context path is byte-identical to the old `Domain.query.all()`. **(2) Databases** — only the **Docker-databases** enumeration (`/docker/databases`) is app-linked and therefore workspace-scopable; it now drives its app iteration through the same `scope_query` (replacing a manual `can_access_app` loop with the equivalent query-level filter + workspace narrowing). The **host-level MySQL/PostgreSQL/SQLite** endpoints are server-global system resources with no app/workspace linkage (and already admin-gated for mutation), so they're **intentionally left unscoped** — documented inline. **(3) Role reconciliation** — new `WorkspaceService.effective_role(user, workspace_id)` + `can_write_in_workspace`: reconcile the user's **global** `User.role` with their **per-workspace** `WorkspaceMember.role`, **narrow-only** (mirrors `scope_query`) — a workspace role can only *reduce* capability, never elevate. A system admin is never capped (no workspace lock-out); a workspace **viewer** caps a global developer to read-only *in that workspace*; any other membership leaves the global role intact. Enforced at the resource-stamping write seams (apps + servers `create` and `…/workspace` reassign): a workspace viewer gets a **403** creating/moving resources into that workspace — checked **only** when a workspace context is active, so no-context behavior is unchanged. The reconciled role is surfaced on `GET /workspaces` (`my_role` / `my_effective_role`) so the frontend can show read-only state. **No schema/migration** (pure query + capability logic on existing columns) and **no frontend change** (the WP hub/apps/domains/databases already fetch through the `X-Workspace-Id`-injecting client; the 403s surface via the existing toast). Verified: backend `py_compile`, **6 new pytest tests** (effective-role reconciliation incl. the never-elevate + admin-never-capped cases, viewer-blocked create + reassign, domains scoping incl. grant-visibility + admin-sees-all, docker-databases scoping, list-workspaces surfacing), full suite **193 passed / 12 skipped**. *(Still remaining for the dedicated milestone: server-side workspace-scoped **logo** swap, and the broad per-route write-enforcement of the viewer-cap beyond the create/reassign seams — an enforcement sweep across the mutating routes, the same XL surface as the rest of #33.)* Not runtime-exercised against the live frontend (backend + API-level tests only).
-### #34 — Horizontal scaling `[XL]` ❌ — ⏸️ Deferred (XL + blocked on the same per-site reverse-proxy/upstream layer as #21)
+### #34 — Horizontal scaling `[XL]` ❌ — ⏸️ Deferred, now **split into a spec** ([docs/HORIZONTAL_SCALING_SPEC.md](HORIZONTAL_SCALING_SPEC.md)) — XL + blocked on the same per-site reverse-proxy/upstream layer as #21
- **Today:** `nginx_advanced_service` has load-balancing primitives (round-robin/least-conn/IP-hash) but WP containers are stateful (local uploads, no shared object cache).
- **Do:** Make WP stateless — shared media (uploads on a shared volume or S3 offload), shared Redis object cache (#23), then nginx `upstream` load balancing across replicas with sticky handling where needed.
- **Reuse:** `nginx_advanced_service`, #23, `storage_provider_service`.
- **Done when:** A site can run N replicas behind a balancer with consistent media + cache.
- **Assessment / why deferred:** **`XL`** *and* infra-blocked. Managed sites are single `wordpress:*-apache` containers published directly on `localhost:PORT` with **no per-site nginx in front** — the very reverse-proxy layer #34's `upstream` load-balancing needs, and the same missing infra that blocks **#21** (preview URLs) and that #8's backend/#25's "nginx" premise also ran into. Beyond the balancer, making WP **stateless** is itself a large build: shared media (an S3/offload plugin or a shared uploads volume across replicas, via `storage_provider_service`), a shared Redis object cache (the **#23** redis sidecar exists but is per-site, not shared across replicas), and replica orchestration in the compose stack. So #34 needs (a) the per-site reverse-proxy/routing layer to exist first (a prerequisite shared with #21), then (b) a stateless-WP refactor, then (c) the `upstream` wiring — a milestone-sized effort gated on infrastructure not yet built. Deferred. No code shipped.
+- **Spec (landed):** the epic is now split into [docs/HORIZONTAL_SCALING_SPEC.md](HORIZONTAL_SCALING_SPEC.md). A capability sweep found the hard parts **already exist** — `NginxAdvancedService.create_reverse_proxy` already emits `upstream{}` load-balancing + `limit_req` + `proxy_cache` + per-vhost log reads (`nginx_advanced_service.py:31,47,61,68,173`), `NginxService.create_site` already renders a container-proxying vhost with cert + 80→443 redirect (`nginx_service.py:391,450`), and wildcard TLS / S3 directory upload are built (`advanced_ssl_service.py:39`, `storage_provider_service.py:260`). So #34 is **wiring + a public base-domain + stateless WP**, not greenfield nginx. The spec defines a shared **keystone** (K0–K4: public base-domain → per-site reverse-proxy vhost → auto-TLS → preview URL → per-site access-log/`limit_req`) that *also* closes **#21, #15-TLS, #8-backend, #30-jail, and #25's response-time/slow-pages**, then **#34 proper** (H0–H4: stateless WP via shared media + already-shared #23 Redis → internal-network replicas → upstream LB → rolling/health → optional autoscale). Each phase is a separately shippable + separately verifiable PR, with an explicit "verifiable on Windows vs needs a real Docker host" split so the infra is exercised before it's claimed working. Recommended order: K0→K2 (closes #15/#8/#21) before any #34 work; #34 MVP done at H2.
### #35 — WordPress workflow nodes, events & CLI `[L]` 🟡 — ✅ Done (WP lifecycle events emitted + subscribable; workflow action-nodes + CLI deferred)
- **Today:** The event/webhook + workflow engine can't see a `WordPressSite` — `EVENT_CATALOG` has no WP lifecycle events, WP operations emit nothing, the workflow builder has no WP node types, and there's no `serverkit wp` CLI.