Skip to content

ServerKit is a lightweight, modern server control panel for managing web applications, databases, and services on your VPS or dedicated server. Built with Python/Flask backend and React frontend.

Add Multipass-based E2E test harness

Juan Denisabb8889c8395
diff --git a/scripts/test/.gitignore b/scripts/test/.gitignore
new file mode 100644
index 00000000..43d56d56
--- /dev/null
+++ b/scripts/test/.gitignore
@@ -0,0 +1,3 @@
+output/
+*.tar.gz
+__pycache__/
diff --git a/scripts/test/README.md b/scripts/test/README.md
new file mode 100644
index 00000000..33872cc7
--- /dev/null
+++ b/scripts/test/README.md
@@ -0,0 +1,129 @@
+# ServerKit E2E Test Harness
+
+One-click full-stack test that spins up fresh Linux VMs on your Windows box,
+installs ServerKit from your **local working tree** (uncommitted changes
+included), and runs an API harness against the live panel. Aggregates
+everything into a single HTML report.
+
+Designed for the "I'm about to merge a huge PR and want to be sure the
+installer actually works on multiple distros" use case.
+
+## Prerequisites (one-time)
+
+1. **Multipass** — https://multipass.run/download/windows
+ ```powershell
+ winget install Canonical.Multipass
+ ```
+2. **Python 3** on PATH (for the HTML report generator; tests run inside the VMs).
+3. **WSL2 / Hyper-V** enabled (Multipass uses this — installer prompts you).
+
+## Usage
+
+From the repo root, in PowerShell:
+
+```powershell
+.\scripts\test\full-stack-test.ps1
+```
+
+That's it. Goes and makes coffee. ~1-2 hours on first run (cloud images
+download), 15-30 min after that. When it finishes, your browser opens to the
+HTML report.
+
+### Options
+
+```powershell
+# Only test Ubuntu 24.04
+.\scripts\test\full-stack-test.ps1 -Only ubuntu24
+
+# Keep VMs running so you can shell in and poke around
+.\scripts\test\full-stack-test.ps1 -Keep
+
+# Bigger VMs (default 2 CPU / 4 GB RAM / 15 GB disk)
+.\scripts\test\full-stack-test.ps1 -Cpus 4 -MemoryGB 8
+```
+
+### Agent pairing test (optional)
+
+After running with `-Keep`:
+
+```powershell
+.\scripts\test\agent-test.ps1
+```
+
+Exercises the agent <-> panel pairing API end-to-end (enroll → claim).
+
+## What it does
+
+1. Tar your local repo (excluding `.git`, `node_modules`, venvs, `dist/`).
+2. `multipass launch` three VMs in parallel: Ubuntu 22.04, Ubuntu 24.04, Debian 12.
+3. On each VM, in parallel:
+ - Upload the tarball + `vm-install.sh`.
+ - Run `install.sh` (real public installer — clones from GitHub, installs
+ Python, Node, Docker, nginx, builds frontend, starts systemd unit).
+ - Overlay your local working tree on top of `/opt/serverkit`, rebuild
+ frontend, restart `serverkit` systemd unit. This is what makes us test
+ **your code**, not what's on `main`.
+ - Wait for `/api/v1/system/health` to return 200.
+ - Push the pytest harness and run it against the live panel.
+ - Capture install log + journalctl regardless of outcome.
+4. Generate one self-contained `report.html` with green/red per VM, per test,
+ plus full logs inline.
+5. Tear down VMs (unless `-Keep`).
+
+## Output
+
+```
+scripts/test/output/<run-id>/
+ report.html <- open this
+ serverkit-src.tar.gz
+ sk-test-ubuntu22-<id>/
+ install.log
+ vm-install.log
+ journalctl.log
+ install-status ("OK" or "FAIL")
+ pytest.log
+ pytest-report.json
+ sk-test-ubuntu24-<id>/ ...
+ sk-test-debian12-<id>/ ...
+```
+
+## Extending the harness
+
+Add new tests under `harness/test_*.py` — they're plain pytest using a
+session-scoped `admin_token` fixture and a `base_url` fixture. The orchestrator
+auto-copies every file in `harness/` to each VM, so just dropping in a new
+`test_05_whatever.py` is enough.
+
+Currently covered:
+- `test_01_health.py` — backend health, frontend reachable
+- `test_02_auth.py` — setup-status, register, login, JWT-authed request
+- `test_03_plugins.py` — list plugins (install-from-URL test is `@skipif`'d
+ until a stable test plugin repo exists; flip when ready)
+- `test_04_smoke.py` — sample of authed endpoints must not return 5xx
+
+## Limitations
+
+- Doesn't test bare-metal-only stuff (hardware drivers, real partitioning).
+- Agent ARM64 MSI installer isn't exercised (no ARM hardware).
+- UI smoke (Playwright) is not wired up yet — easy to add as a follow-up if
+ the API surface stops catching everything.
+- Fedora/Rocky not in the default distro list because Multipass focuses on
+ Ubuntu; add via Vagrant + libvirt if you need those.
+
+## Troubleshooting
+
+**"multipass: command not found"** — install Multipass and restart PowerShell.
+
+**Launch fails with timeout** — first launch downloads ~600 MB per distro
+image; let it run. If it times out repeatedly, increase Multipass timeout:
+`multipass set local.driver=hyperv`.
+
+**Health check never passes** — open `report.html`, expand "Install log" and
+"journalctl" for that VM. Most common: missing system package on a distro the
+installer doesn't handle, or `frontend build` OOM (bump `-MemoryGB`).
+
+**Want to debug a failing VM** — re-run with `-Keep`, then:
+```powershell
+multipass shell sk-test-ubuntu24-<id>
+sudo journalctl -u serverkit -f
+```
diff --git a/scripts/test/agent-test.ps1 b/scripts/test/agent-test.ps1
new file mode 100644
index 00000000..db21d5cf
--- /dev/null
+++ b/scripts/test/agent-test.ps1
@@ -0,0 +1,92 @@
+#requires -Version 5.1
+<#
+.SYNOPSIS
+ Test Windows agent pairing against one of the test VMs.
+
+.DESCRIPTION
+ Optional follow-on to full-stack-test.ps1 (which must be run with -Keep).
+ - Discovers a running sk-test-* VM
+ - Registers admin + obtains JWT
+ - Builds the agent (go build) and runs it against the VM
+ - Verifies pairing short-code flow and that capabilities arrive
+
+ This is intentionally minimal — it exercises the pairing happy path so
+ regressions in the agent <-> panel handshake get caught.
+
+.PARAMETER VmName
+ Specific VM to target. Default: first sk-test-* VM found.
+#>
+[CmdletBinding()]
+param(
+ [string] $VmName
+)
+
+$ErrorActionPreference = 'Stop'
+$RepoRoot = (Resolve-Path "$PSScriptRoot\..\..").Path
+
+if (-not (Get-Command multipass -ErrorAction SilentlyContinue)) {
+ Write-Error "Multipass not installed."
+ exit 2
+}
+if (-not (Get-Command go -ErrorAction SilentlyContinue)) {
+ Write-Error "Go is required to build the agent."
+ exit 2
+}
+
+if (-not $VmName) {
+ $VmName = (& multipass list --format csv | Select-String '^sk-test-' | Select-Object -First 1).ToString().Split(',')[0]
+}
+if (-not $VmName) {
+ Write-Error "No sk-test-* VM found. Run full-stack-test.ps1 -Keep first."
+ exit 2
+}
+
+$vmIp = (& multipass info $VmName --format csv | Select-String "^$VmName,").ToString().Split(',')[2]
+Write-Host "Targeting VM: $VmName ($vmIp)" -ForegroundColor Cyan
+
+# 1. Register admin and grab token
+$cred = @{
+ email = "agent-test-$(Get-Random -Maximum 99999)@test.local"
+ username = "agentuser$(Get-Random -Maximum 99999)"
+ password = 'Test12345!'
+}
+$resp = Invoke-RestMethod -Uri "http://${vmIp}:5000/api/v1/auth/register" `
+ -Method Post -ContentType 'application/json' -Body ($cred | ConvertTo-Json)
+$token = $resp.access_token
+Write-Host " ✓ Admin registered, got JWT" -ForegroundColor Green
+
+# 2. Build agent
+$agentDir = Join-Path $RepoRoot 'agent'
+if (-not (Test-Path $agentDir)) { Write-Error "agent/ not found at $agentDir"; exit 2 }
+$agentBin = Join-Path $env:TEMP "serverkit-agent-test.exe"
+Push-Location $agentDir
+try {
+ Write-Host " Building agent..." -ForegroundColor Yellow
+ & go build -o $agentBin ./cmd/serverkit-agent
+ if ($LASTEXITCODE -ne 0) { throw "go build failed" }
+} finally { Pop-Location }
+Write-Host " ✓ Agent built: $agentBin" -ForegroundColor Green
+
+# 3. Generate pairing short-code by enrolling from "agent side"
+# (Normally the agent does this itself; here we drive it explicitly to test API.)
+$enrollBody = @{
+ pubkey = ([System.BitConverter]::ToString((New-Object byte[] 32 | ForEach-Object { Get-Random -Maximum 256 })) -replace '-','').ToLower()
+ passphrase = 'test-passphrase'
+ machine_id = "test-$(Get-Random -Maximum 999999)"
+ system_info = @{ os = 'Windows'; hostname = $env:COMPUTERNAME }
+} | ConvertTo-Json -Depth 5
+
+$enroll = Invoke-RestMethod -Uri "http://${vmIp}:5000/api/v1/pairing/enroll" `
+ -Method Post -ContentType 'application/json' -Body $enrollBody
+Write-Host " ✓ Enrolled, short-code: $($enroll.code)" -ForegroundColor Green
+
+# 4. Operator-side claim
+$claim = Invoke-RestMethod -Uri "http://${vmIp}:5000/api/v1/pairing/claim" `
+ -Method Post `
+ -Headers @{ Authorization = "Bearer $token" } `
+ -ContentType 'application/json' `
+ -Body (@{ code = $enroll.code; passphrase = 'test-passphrase'; name = $VmName } | ConvertTo-Json)
+Write-Host " ✓ Server claimed, id=$($claim.server_id)" -ForegroundColor Green
+
+Write-Host "`nPairing API end-to-end OK." -ForegroundColor Green
+Write-Host "Note: this exercises the API handshake; full agent process lifecycle test is TODO."
diff --git a/scripts/test/cloud-init/base.yaml b/scripts/test/cloud-init/base.yaml
new file mode 100644
index 00000000..78ec5f0e
--- /dev/null
+++ b/scripts/test/cloud-init/base.yaml
@@ -0,0 +1,23 @@
+#cloud-config
+# Minimal bootstrap for ServerKit E2E test VMs.
+# The harness uploads the local working tree and runs vm-install.sh afterward.
+package_update: true
+package_upgrade: false
+
+packages:
+ - curl
+ - git
+ - rsync
+ - sudo
+ - ca-certificates
+
+write_files:
+ - path: /etc/profile.d/serverkit-test.sh
+ permissions: '0644'
+ content: |
+ export DEBIAN_FRONTEND=noninteractive
+ export NEEDRESTART_MODE=a
+
+runcmd:
+ - mkdir -p /opt/serverkit-src
+ - chown ubuntu:ubuntu /opt/serverkit-src 2>/dev/null || true
diff --git a/scripts/test/full-stack-test.ps1 b/scripts/test/full-stack-test.ps1
new file mode 100644
index 00000000..30ca3a00
--- /dev/null
+++ b/scripts/test/full-stack-test.ps1
@@ -0,0 +1,272 @@
+#requires -Version 5.1
+<#
+.SYNOPSIS
+ One-click full-stack E2E test for ServerKit on Windows.
+
+.DESCRIPTION
+ Spins up fresh Ubuntu/Debian VMs via Multipass in parallel, uploads the
+ current local working tree (including uncommitted changes), runs the
+ installer, then runs a pytest harness against the live API. Aggregates
+ everything into a single HTML report.
+
+ Designed to be slow but unattended — 1-2 hours is expected on first run
+ (cloud images download). Subsequent runs are ~15-30 min.
+
+.PARAMETER Distros
+ Which distros to test. Default: ubuntu22, ubuntu24, debian12.
+
+.PARAMETER Keep
+ Don't tear down VMs at the end (for debugging).
+
+.PARAMETER Only
+ Comma-separated subset of distros to run (e.g. -Only "ubuntu24").
+
+.PARAMETER Cpus
+ CPUs per VM (default 2).
+
+.PARAMETER MemoryGB
+ Memory per VM in GB (default 4).
+
+.PARAMETER DiskGB
+ Disk per VM in GB (default 15).
+
+.EXAMPLE
+ .\scripts\test\full-stack-test.ps1
+
+.EXAMPLE
+ .\scripts\test\full-stack-test.ps1 -Only ubuntu24 -Keep
+#>
+[CmdletBinding()]
+param(
+ [string[]] $Distros = @('ubuntu22','ubuntu24','debian12'),
+ [string] $Only,
+ [switch] $Keep,
+ [int] $Cpus = 2,
+ [int] $MemoryGB = 4,
+ [int] $DiskGB = 15
+)
+
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
+# --- distro -> multipass image map ---------------------------------------
+$ImageMap = @{
+ 'ubuntu22' = '22.04'
+ 'ubuntu24' = '24.04'
+ 'debian12' = 'daily:debian12' # not always available; fallback to 22.04 if launch fails
+}
+
+if ($Only) {
+ $requested = $Only -split ','
+ $Distros = $Distros | Where-Object { $_ -in $requested }
+}
+if (-not $Distros) {
+ Write-Error "No distros selected."
+ exit 2
+}
+
+# --- paths ---------------------------------------------------------------
+$RepoRoot = (Resolve-Path "$PSScriptRoot\..\..").Path
+$RunId = Get-Date -Format 'yyyyMMdd-HHmmss'
+$OutDir = Join-Path $PSScriptRoot "output\$RunId"
+$null = New-Item -ItemType Directory -Force -Path $OutDir
+
+Write-Host "========================================" -ForegroundColor Cyan
+Write-Host " ServerKit E2E Test — run $RunId" -ForegroundColor Cyan
+Write-Host " Repo: $RepoRoot" -ForegroundColor Cyan
+Write-Host " Distros: $($Distros -join ', ')" -ForegroundColor Cyan
+Write-Host " Output: $OutDir" -ForegroundColor Cyan
+Write-Host "========================================" -ForegroundColor Cyan
+
+# --- prerequisites -------------------------------------------------------
+if (-not (Get-Command multipass -ErrorAction SilentlyContinue)) {
+ Write-Host ""
+ Write-Host "Multipass is not installed." -ForegroundColor Red
+ Write-Host "Install from: https://multipass.run/download/windows"
+ Write-Host "Or via winget: winget install Canonical.Multipass"
+ exit 2
+}
+if (-not (Get-Command python -ErrorAction SilentlyContinue)) {
+ Write-Warning "python not on PATH — report.py won't run automatically; will print raw paths."
+}
+
+# --- build source tarball once -------------------------------------------
+$Tarball = Join-Path $OutDir 'serverkit-src.tar.gz'
+Write-Host "`n[1/4] Packing local working tree -> $Tarball" -ForegroundColor Yellow
+
+# Use tar (Windows 10+ has bsdtar). Exclude heavy/irrelevant dirs.
+$excludes = @(
+ '--exclude=./.git',
+ '--exclude=./backend/venv',
+ '--exclude=./backend/.venv',
+ '--exclude=./backend/.venv-wsl',
+ '--exclude=./backend/instance',
+ '--exclude=./frontend/node_modules',
+ '--exclude=./frontend/dist',
+ '--exclude=./scripts/test/output'
+)
+Push-Location $RepoRoot
+try {
+ & tar -czf $Tarball @excludes -C $RepoRoot .
+ if ($LASTEXITCODE -ne 0) { throw "tar failed (exit $LASTEXITCODE)" }
+} finally {
+ Pop-Location
+}
+$tarSize = (Get-Item $Tarball).Length / 1MB
+Write-Host (" Tarball: {0:N1} MB" -f $tarSize)
+
+# --- launch VMs in parallel ---------------------------------------------
+Write-Host "`n[2/4] Launching $($Distros.Count) VM(s) in parallel..." -ForegroundColor Yellow
+
+$cloudInit = Join-Path $PSScriptRoot 'cloud-init\base.yaml'
+$vmInstall = Join-Path $PSScriptRoot 'vm-install.sh'
+
+$jobs = foreach ($d in $Distros) {
+ $vmName = "sk-test-$d-$RunId"
+ $image = $ImageMap[$d]
+ if (-not $image) { Write-Warning "Unknown distro $d, skipping"; continue }
+
+ Start-Job -Name $vmName -ArgumentList $vmName,$image,$cloudInit,$Cpus,$MemoryGB,$DiskGB -ScriptBlock {
+ param($name,$image,$cloudInit,$cpu,$mem,$disk)
+ $log = @()
+ $log += "Launching $name (image=$image, cpu=$cpu, mem=${mem}G, disk=${disk}G)"
+ & multipass launch $image `
+ --name $name `
+ --cpus $cpu `
+ --memory "${mem}G" `
+ --disk "${disk}G" `
+ --cloud-init $cloudInit 2>&1 | ForEach-Object { $log += $_ }
+ @{ Name=$name; Image=$image; RC=$LASTEXITCODE; Log=$log }
+ }
+}
+
+$launchResults = $jobs | Wait-Job | Receive-Job
+$jobs | Remove-Job
+
+$liveVMs = @()
+foreach ($r in $launchResults) {
+ if ($r.RC -eq 0) {
+ Write-Host " ✓ $($r.Name) launched" -ForegroundColor Green
+ $liveVMs += $r.Name
+ } else {
+ Write-Host " ✗ $($r.Name) failed to launch (rc=$($r.RC))" -ForegroundColor Red
+ $r.Log | ForEach-Object { Write-Host " $_" -ForegroundColor DarkGray }
+ }
+}
+
+if (-not $liveVMs) {
+ Write-Error "No VMs launched successfully."
+ exit 1
+}
+
+# --- per-VM: transfer source + run installer + harness in parallel ------
+Write-Host "`n[3/4] Installing + testing on $($liveVMs.Count) VM(s) in parallel..." -ForegroundColor Yellow
+Write-Host " (This typically takes 20-60 min per VM. Go make coffee.)" -ForegroundColor DarkGray
+
+$harnessDir = Join-Path $PSScriptRoot 'harness'
+
+$testJobs = foreach ($vm in $liveVMs) {
+ Start-Job -Name "test-$vm" -ArgumentList $vm,$Tarball,$vmInstall,$harnessDir,$OutDir -ScriptBlock {
+ param($vm,$tarball,$vmInstall,$harnessDir,$outDir)
+
+ $vmOut = Join-Path $outDir $vm
+ New-Item -ItemType Directory -Force -Path $vmOut | Out-Null
+ $installLog = Join-Path $vmOut 'install.log'
+ $statusFile = Join-Path $vmOut 'install-status'
+
+ function Mp { & multipass @args 2>&1 }
+
+ # 1. Transfer tarball + vm-install.sh
+ Mp transfer $tarball "${vm}:/tmp/serverkit-src.tar.gz" | Out-Null
+ Mp transfer $vmInstall "${vm}:/tmp/vm-install.sh" | Out-Null
+
+ # 2. Extract on VM
+ Mp exec $vm -- sudo mkdir -p /opt/serverkit-src | Out-Null
+ Mp exec $vm -- sudo tar -xzf /tmp/serverkit-src.tar.gz -C /opt/serverkit-src | Out-Null
+ Mp exec $vm -- sudo chmod +x /tmp/vm-install.sh | Out-Null
+
+ # 3. Run install (long)
+ $installOut = Mp exec $vm -- sudo bash /tmp/vm-install.sh
+ $installRC = $LASTEXITCODE
+ $installOut | Out-File -FilePath $installLog -Encoding utf8
+
+ # Pull the canonical install log + status from VM
+ Mp transfer "${vm}:/var/log/serverkit-test-install.log" "$vmOut\vm-install.log" 2>&1 | Out-Null
+ Mp transfer "${vm}:/tmp/serverkit-install-status" $statusFile 2>&1 | Out-Null
+ if (-not (Test-Path $statusFile)) {
+ $fallback = if ($installRC -eq 0) { 'OK' } else { 'FAIL' }
+ $fallback | Out-File $statusFile -Encoding ascii
+ }
+
+ # 4. Capture journalctl for backend regardless of install outcome
+ Mp exec $vm -- sudo journalctl -u serverkit --no-pager -n 500 2>$null `
+ | Out-File (Join-Path $vmOut 'journalctl.log') -Encoding utf8
+
+ # 5. If install ok, copy harness and run pytest
+ $statusContent = (Get-Content $statusFile -Raw).Trim()
+ if ($statusContent -eq 'OK') {
+ # Push harness
+ Mp exec $vm -- sudo mkdir -p /opt/serverkit-test | Out-Null
+ Mp exec $vm -- sudo chown ubuntu:ubuntu /opt/serverkit-test 2>$null | Out-Null
+ Get-ChildItem $harnessDir -File | ForEach-Object {
+ Mp transfer $_.FullName "${vm}:/opt/serverkit-test/$($_.Name)" | Out-Null
+ }
+ # Install pytest in the panel's venv (already has python)
+ Mp exec $vm -- sudo /opt/serverkit/venv/bin/pip install -r /opt/serverkit-test/requirements.txt 2>&1 `
+ | Out-File (Join-Path $vmOut 'harness-deps.log') -Encoding utf8
+
+ $reportJson = '/opt/serverkit-test/pytest-report.json'
+ Mp exec $vm -- sudo /opt/serverkit/venv/bin/python -m pytest /opt/serverkit-test `
+ --json-report --json-report-file=$reportJson -v 2>&1 `
+ | Out-File (Join-Path $vmOut 'pytest.log') -Encoding utf8
+ Mp transfer "${vm}:$reportJson" (Join-Path $vmOut 'pytest-report.json') 2>&1 | Out-Null
+ }
+
+ @{ Name=$vm; Status=$statusContent }
+ }
+}
+
+$results = $testJobs | Wait-Job | Receive-Job
+$testJobs | Remove-Job
+
+foreach ($r in $results) {
+ $color = if ($r.Status -eq 'OK') { 'Green' } else { 'Red' }
+ Write-Host " $($r.Name): install=$($r.Status)" -ForegroundColor $color
+}
+
+# --- generate report -----------------------------------------------------
+Write-Host "`n[4/4] Generating HTML report..." -ForegroundColor Yellow
+$reportHtml = $null
+$reportPy = Join-Path $PSScriptRoot 'report.py'
+if (Get-Command python -ErrorAction SilentlyContinue) {
+ $reportHtml = & python $reportPy $OutDir
+ Write-Host " Report: $reportHtml" -ForegroundColor Green
+} else {
+ Write-Host " Skipped (python not on PATH). Raw output in: $OutDir" -ForegroundColor Yellow
+}
+
+# --- teardown ------------------------------------------------------------
+if ($Keep) {
+ Write-Host "`n-Keep set: VMs left running. Connect with: multipass shell <name>" -ForegroundColor Yellow
+ $liveVMs | ForEach-Object { Write-Host " - $_" }
+} else {
+ Write-Host "`nTearing down VMs..." -ForegroundColor Yellow
+ foreach ($vm in $liveVMs) {
+ & multipass delete $vm 2>&1 | Out-Null
+ }
+ & multipass purge 2>&1 | Out-Null
+}
+
+# --- final summary -------------------------------------------------------
+$failed = @($results | Where-Object { $_.Status -ne 'OK' })
+Write-Host ""
+if ($failed.Count -eq 0) {
+ Write-Host "ALL GREEN — $($liveVMs.Count) VM(s) installed and passed tests." -ForegroundColor Green
+ if ($reportHtml) { Start-Process $reportHtml }
+ exit 0
+} else {
+ Write-Host "FAILURES on $($failed.Count) VM(s):" -ForegroundColor Red
+ $failed | ForEach-Object { Write-Host " - $($_.Name)" -ForegroundColor Red }
+ if ($reportHtml) { Start-Process $reportHtml }
+ exit 1
+}
diff --git a/scripts/test/harness/conftest.py b/scripts/test/harness/conftest.py
new file mode 100644
index 00000000..df67bb8b
--- /dev/null
+++ b/scripts/test/harness/conftest.py
@@ -0,0 +1,68 @@
+import os
+import time
+import uuid
+import requests
+import pytest
+
+BASE_URL = os.environ.get("SERVERKIT_URL", "http://127.0.0.1:5000")
+TIMEOUT = 15
+
+
[email protected](scope="session")
+def base_url():
+ return BASE_URL
+
+
[email protected](scope="session")
+def http():
+ s = requests.Session()
+ s.headers.update({"Accept": "application/json"})
+ return s
+
+
[email protected](scope="session")
+def admin_credentials():
+ suffix = uuid.uuid4().hex[:8]
+ return {
+ "email": f"e2e-{suffix}@test.local",
+ "username": f"e2e_{suffix}",
+ "password": "Test12345!",
+ }
+
+
[email protected](scope="session")
+def admin_token(http, base_url, admin_credentials):
+ """Register the first user (becomes admin) or log in if already exists."""
+ # Wait briefly for backend in case test runs immediately after install
+ for _ in range(30):
+ try:
+ r = http.get(f"{base_url}/api/v1/system/health", timeout=2)
+ if r.status_code == 200:
+ break
+ except requests.RequestException:
+ time.sleep(1)
+
+ r = http.post(
+ f"{base_url}/api/v1/auth/register",
+ json=admin_credentials,
+ timeout=TIMEOUT,
+ )
+ if r.status_code in (200, 201):
+ return r.json()["access_token"]
+
+ # Fallback: maybe already registered (re-run on same VM)
+ r = http.post(
+ f"{base_url}/api/v1/auth/login",
+ json={
+ "email": admin_credentials["email"],
+ "password": admin_credentials["password"],
+ },
+ timeout=TIMEOUT,
+ )
+ r.raise_for_status()
+ return r.json()["access_token"]
+
+
+def auth_headers(admin_token):
+ return {"Authorization": f"Bearer {admin_token}"}
diff --git a/scripts/test/harness/requirements.txt b/scripts/test/harness/requirements.txt
new file mode 100644
index 00000000..daf0f940
--- /dev/null
+++ b/scripts/test/harness/requirements.txt
@@ -0,0 +1,3 @@
+pytest>=8.0
+pytest-json-report>=1.5
+requests>=2.31
diff --git a/scripts/test/harness/test_01_health.py b/scripts/test/harness/test_01_health.py
new file mode 100644
index 00000000..1e03a335
--- /dev/null
+++ b/scripts/test/harness/test_01_health.py
@@ -0,0 +1,15 @@
+import requests
+
+
+def test_backend_health(http, base_url):
+ r = http.get(f"{base_url}/api/v1/system/health", timeout=10)
+ assert r.status_code == 200, r.text
+ body = r.json()
+ assert body.get("status") == "healthy", body
+
+
+def test_frontend_reachable(base_url):
+ # Frontend served by nginx on :80 (not the same as backend BASE_URL)
+ url = base_url.replace(":5000", "").rstrip("/") or "http://127.0.0.1"
+ r = requests.get(url, timeout=10, allow_redirects=True)
+ assert r.status_code < 500, f"frontend returned {r.status_code}"
diff --git a/scripts/test/harness/test_02_auth.py b/scripts/test/harness/test_02_auth.py
new file mode 100644
index 00000000..2a0a314c
--- /dev/null
+++ b/scripts/test/harness/test_02_auth.py
@@ -0,0 +1,29 @@
+def test_setup_status_endpoint(http, base_url):
+ r = http.get(f"{base_url}/api/v1/auth/setup-status", timeout=10)
+ assert r.status_code == 200, r.text
+ body = r.json()
+ assert "needs_setup" in body, body
+
+
+def test_admin_register_and_login(admin_token, http, base_url, admin_credentials):
+ # admin_token fixture already did the register; verify login works too
+ assert admin_token, "no token from register"
+ r = http.post(
+ f"{base_url}/api/v1/auth/login",
+ json={
+ "email": admin_credentials["email"],
+ "password": admin_credentials["password"],
+ },
+ timeout=10,
+ )
+ assert r.status_code == 200, r.text
+ assert "access_token" in r.json()
+
+
+def test_authed_request(http, base_url, auth_headers):
+ r = http.get(
+ f"{base_url}/api/v1/auth/setup-status",
+ headers=auth_headers,
+ timeout=10,
+ )
+ assert r.status_code == 200, r.text
diff --git a/scripts/test/harness/test_03_plugins.py b/scripts/test/harness/test_03_plugins.py
new file mode 100644
index 00000000..bdad3c17
--- /dev/null
+++ b/scripts/test/harness/test_03_plugins.py
@@ -0,0 +1,47 @@
+import pytest
+
+
+def test_list_plugins_empty(http, base_url, auth_headers):
+ r = http.get(f"{base_url}/api/v1/plugins", headers=auth_headers, timeout=15)
+ assert r.status_code == 200, r.text
+ body = r.json()
+ # accept either {"plugins": [...]} or a bare list
+ if isinstance(body, dict):
+ assert "plugins" in body
+ else:
+ assert isinstance(body, list)
+
+
+ True,
+ reason="install-from-URL hits GitHub; enable when a stable test plugin repo exists",
+)
+def test_install_plugin_from_url(http, base_url, auth_headers):
+ # Placeholder — wire to a known-good test plugin repo once one exists.
+ r = http.post(
+ f"{base_url}/api/v1/plugins/install",
+ headers=auth_headers,
+ json={"url": "https://github.com/jhd3197/serverkit-gui"},
+ timeout=120,
+ )
+ assert r.status_code in (200, 201), r.text
+ plugin = r.json()
+ pid = plugin["id"]
+
+ # disable
+ r = http.post(
+ f"{base_url}/api/v1/plugins/{pid}/disable", headers=auth_headers, timeout=15
+ )
+ assert r.status_code == 200, r.text
+
+ # enable
+ r = http.post(
+ f"{base_url}/api/v1/plugins/{pid}/enable", headers=auth_headers, timeout=15
+ )
+ assert r.status_code == 200, r.text
+
+ # uninstall
+ r = http.delete(
+ f"{base_url}/api/v1/plugins/{pid}", headers=auth_headers, timeout=30
+ )
+ assert r.status_code in (200, 204), r.text
diff --git a/scripts/test/harness/test_04_smoke.py b/scripts/test/harness/test_04_smoke.py
new file mode 100644
index 00000000..fd009800
--- /dev/null
+++ b/scripts/test/harness/test_04_smoke.py
@@ -0,0 +1,19 @@
+"""Lightweight smoke tests on a sampling of authed endpoints.
+Goal: catch import-time failures and 500s from blueprint registration."""
+
+import pytest
+
+# Endpoints we expect to exist and return something sane (200 or 4xx, not 5xx).
+# Don't expand wildly — keep this focused on "did the blueprint load."
+ENDPOINTS = [
+ "/api/v1/servers",
+ "/api/v1/plugins",
+ "/api/v1/system/health",
+ "/api/v1/auth/setup-status",
+]
+
+
[email protected]("path", ENDPOINTS)
+def test_endpoint_no_5xx(http, base_url, auth_headers, path):
+ r = http.get(f"{base_url}{path}", headers=auth_headers, timeout=15)
+ assert r.status_code < 500, f"{path} returned {r.status_code}: {r.text[:300]}"
diff --git a/scripts/test/report.py b/scripts/test/report.py
new file mode 100644
index 00000000..fc5dc65e
--- /dev/null
+++ b/scripts/test/report.py
@@ -0,0 +1,172 @@
+#!/usr/bin/env python3
+"""Aggregate per-VM results into a single self-contained HTML report.
+
+Inputs (from scripts/test/output/<run-id>/):
+ <vm-name>/install.log — captured cloud-init / install output
+ <vm-name>/install-status — "OK" or "FAIL"
+ <vm-name>/pytest-report.json — pytest-json-report output (may be missing if install failed)
+ <vm-name>/journalctl.log — backend logs (best-effort)
+
+Output:
+ report.html in the run directory.
+"""
+import json
+import sys
+import html
+from pathlib import Path
+from datetime import datetime
+
+
+def esc(s):
+ return html.escape(str(s) if s is not None else "")
+
+
+def load_json(path):
+ try:
+ return json.loads(path.read_text(encoding="utf-8"))
+ except Exception:
+ return None
+
+
+def read_text(path, limit=200_000):
+ try:
+ data = path.read_text(encoding="utf-8", errors="replace")
+ if len(data) > limit:
+ data = "...[truncated]...\n" + data[-limit:]
+ return data
+ except Exception:
+ return "(file missing)"
+
+
+def vm_summary(vm_dir: Path):
+ name = vm_dir.name
+ status_file = vm_dir / "install-status"
+ install_ok = status_file.exists() and status_file.read_text().strip() == "OK"
+ pytest_json = load_json(vm_dir / "pytest-report.json")
+
+ tests = []
+ pass_count = fail_count = skip_count = 0
+ if pytest_json and "tests" in pytest_json:
+ for t in pytest_json["tests"]:
+ outcome = t.get("outcome", "unknown")
+ tests.append((t.get("nodeid", "?"), outcome, t.get("call", {}).get("longrepr", "")))
+ if outcome == "passed":
+ pass_count += 1
+ elif outcome == "failed":
+ fail_count += 1
+ elif outcome == "skipped":
+ skip_count += 1
+
+ overall = "PASS" if (install_ok and fail_count == 0 and (pass_count > 0 or pytest_json is None)) else "FAIL"
+ if not install_ok:
+ overall = "FAIL"
+
+ return {
+ "name": name,
+ "install_ok": install_ok,
+ "overall": overall,
+ "tests": tests,
+ "pass": pass_count,
+ "fail": fail_count,
+ "skip": skip_count,
+ "install_log": read_text(vm_dir / "install.log"),
+ "journal": read_text(vm_dir / "journalctl.log", limit=50_000),
+ }
+
+
+HTML_TEMPLATE = """<!doctype html>
+<html><head><meta charset="utf-8"><title>ServerKit E2E Report — {ts}</title>
+<style>
+ body{{font-family:-apple-system,Segoe UI,sans-serif;margin:24px;background:#0f1115;color:#e6e6e6}}
+ h1{{margin-top:0}}
+ .vm{{background:#1a1d24;border-radius:8px;padding:16px;margin:16px 0;border-left:6px solid #555}}
+ .vm.pass{{border-left-color:#3fb950}}
+ .vm.fail{{border-left-color:#f85149}}
+ .badge{{display:inline-block;padding:2px 10px;border-radius:12px;font-size:12px;font-weight:600}}
+ .badge.pass{{background:#1f6f33;color:#d2f5d8}}
+ .badge.fail{{background:#7a2a26;color:#fdd}}
+ .badge.skip{{background:#3a3a3a;color:#bbb}}
+ details{{margin-top:8px;background:#0c0e13;border-radius:6px;padding:8px}}
+ summary{{cursor:pointer;font-weight:600}}
+ pre{{white-space:pre-wrap;word-break:break-word;font-size:12px;background:#000;padding:8px;border-radius:4px;max-height:400px;overflow:auto}}
+ table{{border-collapse:collapse;width:100%;margin-top:8px;font-size:13px}}
+ td,th{{padding:4px 8px;border-bottom:1px solid #2a2d34;text-align:left}}
+ .counts span{{margin-right:12px}}
+</style></head><body>
+<h1>ServerKit E2E Test Report</h1>
+<p>Run: <code>{run_id}</code> · {ts}</p>
+<p>Overall: <span class="badge {overall_cls}">{overall}</span>
+ &nbsp; VMs: {n_vms} &nbsp; Passed: {n_pass} &nbsp; Failed: {n_fail}</p>
+{vm_sections}
+</body></html>"""
+
+VM_TEMPLATE = """<div class="vm {cls}">
+ <h2>{name} <span class="badge {cls}">{overall}</span></h2>
+ <div class="counts">
+ <span>Install: <b>{install}</b></span>
+ <span>Passed: <b>{p}</b></span>
+ <span>Failed: <b>{f}</b></span>
+ <span>Skipped: <b>{s}</b></span>
+ </div>
+ {tests_table}
+ <details><summary>Install log</summary><pre>{install_log}</pre></details>
+ <details><summary>journalctl -u serverkit</summary><pre>{journal}</pre></details>
+</div>"""
+
+
+def render(run_dir: Path) -> Path:
+ vm_dirs = sorted([p for p in run_dir.iterdir() if p.is_dir()])
+ summaries = [vm_summary(d) for d in vm_dirs]
+
+ n_pass = sum(1 for s in summaries if s["overall"] == "PASS")
+ n_fail = sum(1 for s in summaries if s["overall"] != "PASS")
+ overall = "PASS" if n_fail == 0 and n_pass > 0 else "FAIL"
+
+ vm_sections = []
+ for s in summaries:
+ cls = "pass" if s["overall"] == "PASS" else "fail"
+ if s["tests"]:
+ rows = "".join(
+ f"<tr><td>{esc(nid)}</td><td><span class='badge {oc}'>{esc(oc)}</span></td>"
+ f"<td><pre style='max-height:120px'>{esc(repr_)}</pre></td></tr>"
+ for nid, oc, repr_ in s["tests"]
+ )
+ tests_table = f"<table><tr><th>Test</th><th>Result</th><th>Details</th></tr>{rows}</table>"
+ else:
+ tests_table = "<p><i>No pytest results (install likely failed before harness ran).</i></p>"
+
+ vm_sections.append(VM_TEMPLATE.format(
+ cls=cls,
+ name=esc(s["name"]),
+ overall=esc(s["overall"]),
+ install="OK" if s["install_ok"] else "FAIL",
+ p=s["pass"], f=s["fail"], s=s["skip"],
+ tests_table=tests_table,
+ install_log=esc(s["install_log"]),
+ journal=esc(s["journal"]),
+ ))
+
+ out = run_dir / "report.html"
+ out.write_text(HTML_TEMPLATE.format(
+ ts=datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
+ run_id=esc(run_dir.name),
+ overall=overall,
+ overall_cls="pass" if overall == "PASS" else "fail",
+ n_vms=len(summaries),
+ n_pass=n_pass,
+ n_fail=n_fail,
+ vm_sections="\n".join(vm_sections),
+ ), encoding="utf-8")
+ return out
+
+
+if __name__ == "__main__":
+ if len(sys.argv) != 2:
+ print("usage: report.py <run-output-dir>", file=sys.stderr)
+ sys.exit(2)
+ run_dir = Path(sys.argv[1]).resolve()
+ if not run_dir.is_dir():
+ print(f"not a directory: {run_dir}", file=sys.stderr)
+ sys.exit(2)
+ out = render(run_dir)
+ print(str(out))
diff --git a/scripts/test/vm-install.sh b/scripts/test/vm-install.sh
new file mode 100644
index 00000000..68a783b4
--- /dev/null
+++ b/scripts/test/vm-install.sh
@@ -0,0 +1,85 @@
+#!/bin/bash
+# Runs inside the test VM.
+# Strategy:
+# 1. Run install.sh from the uploaded local source (so it bootstraps deps).
+# 2. After install, replace /opt/serverkit with the uploaded local working
+# tree (overlay), rebuild frontend, reinstall Python deps, restart.
+# 3. Probe health endpoint and exit non-zero if anything failed.
+#
+# This way we test the EXACT local code, including uncommitted changes,
+# instead of whatever is on origin/main.
+set -u # NB: not -e — we want to capture failures and still emit a report
+
+SRC=/opt/serverkit-src
+INSTALL_DIR=/opt/serverkit
+LOG=/var/log/serverkit-test-install.log
+
+log() { echo "[vm-install] $*" | tee -a "$LOG"; }
+fail() { echo "[vm-install] FAIL: $*" | tee -a "$LOG"; exit 1; }
+
+mkdir -p "$(dirname "$LOG")"
+: > "$LOG"
+
+[ -d "$SRC" ] || fail "source dir $SRC missing — multipass transfer broken"
+[ -f "$SRC/install.sh" ] || fail "install.sh not found in source"
+
+log "Step 1/4: running install.sh (this clones origin/main, installs deps)"
+# install.sh clones from GitHub — we let it, then overlay our local code.
+bash "$SRC/install.sh" >> "$LOG" 2>&1
+INSTALL_RC=$?
+log "install.sh exit=$INSTALL_RC"
+
+if [ ! -d "$INSTALL_DIR" ]; then
+ fail "install.sh did not create $INSTALL_DIR (rc=$INSTALL_RC)"
+fi
+
+log "Step 2/4: overlaying local working tree onto $INSTALL_DIR"
+# Preserve .env / instance / nginx ssl from the install
+rsync -a \
+ --exclude='.env' \
+ --exclude='backend/instance/' \
+ --exclude='nginx/ssl/' \
+ --exclude='backend/venv/' \
+ --exclude='backend/.venv/' \
+ --exclude='backend/.venv-wsl/' \
+ --exclude='frontend/node_modules/' \
+ --exclude='frontend/dist/' \
+ --exclude='.git/' \
+ "$SRC/" "$INSTALL_DIR/" >> "$LOG" 2>&1 || fail "rsync overlay failed"
+
+log "Step 3/4: rebuild + restart with local code"
+cd "$INSTALL_DIR" || fail "cd $INSTALL_DIR"
+
+# Reinstall Python deps in case requirements.txt changed
+if [ -d "$INSTALL_DIR/venv" ]; then
+ "$INSTALL_DIR/venv/bin/pip" install -r "$INSTALL_DIR/backend/requirements.txt" >> "$LOG" 2>&1 \
+ || log "WARN: pip install had non-zero exit"
+fi
+
+# Rebuild frontend
+cd "$INSTALL_DIR/frontend" || fail "cd frontend"
+NODE_OPTIONS="--max-old-space-size=1024" npm run build >> "$LOG" 2>&1 \
+ || fail "frontend build failed"
+
+cd "$INSTALL_DIR" || true
+docker compose build >> "$LOG" 2>&1 || log "WARN: docker compose build had non-zero exit"
+docker compose up -d >> "$LOG" 2>&1 || log "WARN: docker compose up had non-zero exit"
+
+systemctl restart serverkit >> "$LOG" 2>&1 || fail "systemctl restart serverkit failed"
+systemctl restart nginx >> "$LOG" 2>&1 || log "WARN: nginx restart had non-zero exit"
+
+log "Step 4/4: waiting for health endpoint"
+for i in $(seq 1 30); do
+ if curl -fsS http://127.0.0.1:5000/api/v1/system/health > /dev/null 2>&1; then
+ log "Backend healthy after ${i}s"
+ echo "OK" > /tmp/serverkit-install-status
+ exit 0
+ fi
+ sleep 1
+done
+
+log "Backend never became healthy"
+systemctl status serverkit --no-pager >> "$LOG" 2>&1 || true
+journalctl -u serverkit --no-pager -n 100 >> "$LOG" 2>&1 || true
+echo "FAIL" > /tmp/serverkit-install-status
+exit 1