Skip to content

ServerKit is a lightweight, modern server control panel for managing web applications, databases, and services on your VPS or dedicated server. Built with Python/Flask backend and React frontend.

docs(deploy): document the Deploy Console; update CHANGELOG and ROADMAP

Juan Denis6bf8c87b9dbf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e75eeff2..3d8d1e40 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -20,6 +20,23 @@ awaiting a stable release:
### Added
+- **The Deploy Console — publish anything, watch it live, debug it from the UI.**
+ Installing a template, deploying a repo, redeploying a service, or uploading a
+ build now takes you to one full-page live console at `/deployments/<id>`: a
+ terminal-style log that streams the real pull/build/start output line-by-line
+ (not 1.5-second snapshots), a step rail that checks off with per-step timings, a
+ live elapsed timer, and follow/wrap/timestamps/level-filter/search/copy/download
+ controls. When a deploy fails, the page pins an error card with the real failure
+ tail (the actual build output, not a stripped one-liner), a plain-language hint
+ for common cases (port in use, image pull denied, OOM, missing env var, npm/pip
+ failures…), and a one-click **Retry** — so you can read the failure and fix your
+ env var / port / Dockerfile and retry entirely from the browser, no SSH. Every
+ install and deploy flow — templates, repo services, manual redeploys, the Build
+ tab — now rides the same job + log + console machinery, and the `/deployments`
+ index deep-links every run to its console. Updates push over websockets with a
+ transparent 2-second polling fallback, so the console keeps working (and says so)
+ when live updates are unavailable. See [docs/DEPLOY_CONSOLE.md](docs/DEPLOY_CONSOLE.md).
+
- **Web Analytics — a native, privacy-first analytics extension
(`serverkit-analytics`).** Self-hosted, first-party website analytics for the
sites this panel manages, built on the extension platform. A cookieless
diff --git a/ROADMAP.md b/ROADMAP.md
index c75afd00..5e22526b 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -10,6 +10,7 @@ This document outlines the development roadmap for ServerKit. Features are organ
### Recently Completed (v1.7.x)
+- **Deploy Console & Live Run Logs** - Every install and deploy lands on one full-page live console: streaming build output, a step timeline with per-step durations and a live elapsed timer, and failure cards showing the real error tail with plain-language hints and one-click retry. Backed by a unified batched run-log layer (database persistence + WebSocket push with a polling fallback).
- **New Service Wizard & Template Catalog** - Three-step service creation (Source / Connect / Review) backed by a server-side template catalog with one-click compose templates and Git-repo templates.
- **Extensions Platform Maturation** - Marketplace registry backed by standalone extension repos (Kubernetes, Automations, Analytics, Mail, GPU, CrowdSec, DNS server, and more), with built-in features progressively extracted to keep the default panel lean.
- **Privacy-First Web Analytics** - Cookieless, self-hosted analytics extension with WordPress and nginx integration and optional geo lookup.
@@ -597,10 +598,10 @@ platform infrastructure; new capability areas ship as Marketplace extensions.
The publish pipeline is becoming ServerKit's signature experience: deploy anything, watch it
happen live, and debug failures entirely from the UI.
-- [ ] Full-page live deploy console for every install and deploy (templates, Git repos, uploads, redeploys)
-- [ ] Streaming build output with a step timeline, per-step durations, and live elapsed time
-- [ ] Failure cards showing the real error tail with plain-language hints and one-click retry
-- [ ] Unified batched run-log layer (database persistence + WebSocket push, polling fallback)
+- [x] Full-page live deploy console for every install and deploy (templates, Git repos, uploads, redeploys)
+- [x] Streaming build output with a step timeline, per-step durations, and live elapsed time
+- [x] Failure cards showing the real error tail with plain-language hints and one-click retry
+- [x] Unified batched run-log layer (database persistence + WebSocket push, polling fallback)
## Extension Platform
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 931d3762..92bc045a 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -508,6 +508,35 @@ drive handlers directly.
---
+## Deploy Console & run logs
+
+Every install and deploy is a `DeploymentJob` run by the single `JobConsumer`, and
+its live output is surfaced by the full-page **Deploy Console** at
+`/deployments/<jobId>`. All deploy-path logging funnels through one seam,
+`app/services/run_log_service.py` (`RunLogStream`):
+
+- **Batched persistence** — one DB commit + one socket emit per flush (50 lines,
+ 300 ms, a step change, or close) instead of a commit per line; a hard 5000-row cap
+ per job. Every line is a `DeploymentJobLog` row.
+- **Truthful failure tail** — an 80-line in-memory ring buffer persisted to the job's
+ `result` JSON on failure, alongside `step_timings` and a matched plain-language
+ `hint` (no schema migration).
+- **Clean text** — ANSI escapes and `\r` progress overwrites are stripped at the seam;
+ builds run with `--ansi never --progress plain` (`DockerService.compose_up_streaming`).
+- **Crash-proof** — `RunLogStream.log()` never raises; a flush failure degrades to a
+ buffered-drop + one telemetry event. Job failure marking never depends on the log
+ path.
+
+The read side stays the source of truth: `GET /deployment-jobs/<id>/logs?after_id=`
+for incremental polling, plus a `deploy_log` / `deploy_status` Socket.IO channel
+(room `deploy_{job_id}`) as an accelerator. The frontend `useDeployJobStream` hook
+de-dupes by row id and re-syncs with `after_id` on reconnect, with a 2-second poll
+fallback — so the console works with sockets disabled. Emits are in-process only,
+consistent with the single-worker gateway constraint below. See
+[DEPLOY_CONSOLE.md](DEPLOY_CONSOLE.md).
+
+---
+
## Notifications Bus
![Notifications bus: a non-blocking notify.send(event, to, data) call writes a durable
diff --git a/docs/DEPLOY_CONSOLE.md b/docs/DEPLOY_CONSOLE.md
new file mode 100644
index 00000000..8407a654
--- /dev/null
+++ b/docs/DEPLOY_CONSOLE.md
@@ -0,0 +1,82 @@
+# The Deploy Console
+
+Publishing something on ServerKit — a template, a repository, an upload, a manual
+redeploy — should be the signature experience of the product. Other panels become
+black boxes the moment something goes wrong: you click deploy, a spinner spins, and
+when it fails you get a one-line error. ServerKit's promise is the opposite: **you
+can always see what the machine is doing, and when it breaks you can debug it from
+the UI without SSH.**
+
+The Deploy Console delivers that. One click (or one wizard) takes you to a single
+full-page live console at **`/deployments/<jobId>`** that shows the whole deploy
+happening in real time.
+
+## What you see
+
+- **Header** — a humanized title ("Installing Umami Analytics", "Deploying my-api"),
+ a status pill (Queued / Running / Succeeded / Failed), a step counter, and a live
+ elapsed timer.
+- **Step rail** — the job plan as a checklist: ✓ with per-step duration when done,
+ a spinner on the current step, ✗ on the failed one, ○ pending. Click a step to
+ scroll the log to its first line.
+- **Log pane** — a dark, monospace, terminal-style surface that streams real build
+ output line-by-line, level-colored, with **Follow / Wrap / Timestamps**, a level
+ filter, client-side **Search**, **Copy** and **Download .txt**. Follow disengages
+ when you scroll up and re-engages via a "Jump to live" chip.
+- **On failure** — a pinned error card shows the failed step, the **failure tail**
+ (the last ~80 lines of the *real* build output, not a stripped stderr fragment),
+ the one-line error, and — when a heuristic matches — a plain-language **hint**
+ ("Another service is already using this port — change the port mapping and retry").
+ Actions: **Retry deploy** (clones the job and swaps the console to the new run),
+ **Copy error**, and **Ask AI** (opens the assistant seeded with the tail).
+- **On success** — a completion banner with total duration, per-step timings, and
+ the payoff actions: **Open app**, **View service**, **View runtime logs**.
+- **Degraded** — if the websocket is unavailable the console says so and keeps
+ working on a 2-second poll. It is never blank and never silently frozen.
+- **History** — opening the console for a finished job renders everything from the
+ database (full log, timings, error card). Console URLs are permanent, shareable
+ links to a deploy.
+
+## Every flow ends up here
+
+Template one-click installs, Git-repo service creation, uploads with auto-deploy,
+manual redeploys, and the Build tab's "Deploy Now" all enqueue a **DeploymentJob**
+and navigate to the console. The `/deployments` index lists every past and active
+run and deep-links each row to its console.
+
+## How it works (architecture)
+
+- **`RunLogStream`** (`backend/app/services/run_log_service.py`) is the single write
+ seam for all deploy-path logging. It batches writes (one DB commit + one socket
+ emit per flush — on 50 lines, 300 ms, a step change, or close), keeps a truthful
+ 80-line in-memory failure tail, records per-step timings, sanitizes ANSI escapes
+ and `\r` progress overwrites, caps persisted rows at 5000 per job, and matches a
+ failure hint. `RunLogStream.log()` never raises — a stuck "running" job is worse
+ than a missing log line.
+- **Persistence is the source of truth.** Every line is a `DeploymentJobLog` row;
+ step timings, the failure tail and the hint live in the job's `result` JSON (no
+ schema migration). The read API exposes incremental polling via
+ `GET /deployment-jobs/<id>/logs?after_id=`.
+- **Sockets are an accelerator.** `RunLogStream` emits `deploy_log` (a batch of
+ persisted lines, each carrying its DB id) and `deploy_status` (the job summary) to
+ room `deploy_{job_id}`. The frontend hook `useDeployJobStream` boots from a full
+ snapshot, prefers the socket channel, de-dupes by id, re-syncs with `after_id` on
+ every reconnect, and falls back to a 2-second poll — so the console is 100%
+ functional with sockets disabled. In-process emits only (single-worker gateway
+ constraint; see [ARCHITECTURE.md](ARCHITECTURE.md)).
+- **Streaming builds.** `DockerService.compose_up_streaming` runs compose with
+ `--ansi never --progress plain` under `Popen` and delivers output line-by-line, so
+ template installs stream the real pull/build transcript and a failure persists a
+ meaningful tail — historically the biggest reason install errors were unreadable.
+- **Every deploy is a job.** `POST /apps/<id>/deploy` returns `202 {deploy_job_id}`
+ (or runs synchronously with `?wait=true` for CLI/tests). Failed jobs can be retried
+ via `POST /deployment-jobs/<id>/retry`, which clones the job and enqueues a fresh
+ run on a new console URL.
+
+## Follow-ups (not yet shipped)
+
+- WordPress site creation adopting the console (via the plugins SDK).
+- Agent-side line streaming for remote-server installs (a protocol addition in the
+ `serverkit-agent` repo); until then remote installs render as a labeled output
+ block.
+- A cancel-running-deploy button (needs safe subprocess-kill semantics).