Skip to content

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

fix: server-mode metrics count only successful requests, so requestsFailed is structurally always 0 (PIN-093)

Luigi Agostic13a1b3af8a2
diff --git a/docs/reference/health.md b/docs/reference/health.md
index 04820ff6..184f8bc0 100644
--- a/docs/reference/health.md
+++ b/docs/reference/health.md
@@ -2,6 +2,13 @@
Check server status and availability.
+`/health` is deliberately **not** the endpoint to compare across modes: in server
+mode it is the orchestrator's own envelope, and an orchestrator has facts
+(`instances`, `profiles`, `defaultInstance`) a bridge does not. Failure and crash
+telemetry is served in the same shape by both modes on
+[`/metrics`](./metrics.md) — use that when comparing a bridge repro against a
+server-mode problem.
+
## Bridge Mode
```bash
@@ -15,7 +22,7 @@ pinchtab health
pinchtab health --json # Full JSON response
```
-Bridge-mode health may also include:
+Bridge-mode health also reports `version`, and may include:
- `crashLogs`
- `failures`
diff --git a/docs/reference/index.md b/docs/reference/index.md
index bc374285..7812aab2 100644
--- a/docs/reference/index.md
+++ b/docs/reference/index.md
@@ -19,6 +19,7 @@ Note: this index is generated and should be checked against the code when comman
- [Hover](./hover.md)
- [Instances](./instances.md)
- [Keyboard](./keyboard.md)
+- [Metrics](./metrics.md)
- [Mouse](./mouse.md)
- [Navigate](./navigate.md)
- [PDF](./pdf.md)
diff --git a/docs/reference/metrics.md b/docs/reference/metrics.md
new file mode 100644
index 00000000..1dcbe6e3
--- /dev/null
+++ b/docs/reference/metrics.md
@@ -0,0 +1,80 @@
+# Metrics
+
+Runtime counters, recent failures and crash diagnostics.
+
+```bash
+curl -H "Authorization: Bearer $TOKEN" http://localhost:9867/metrics
+```
+
+## Two layers, never summed
+
+Server mode runs **two processes**, and each keeps its own counters:
+
+| Layer | Process | Sees |
+|-------|---------|------|
+| `frontDoor` | the orchestrator that owns the port | auth rejections, unrouted paths, everything before a request is forwarded |
+| `instance` | a browser-control child (also what `pinchtab bridge` runs standalone) | every request that reached a browser |
+
+Every response says which layer its numbers describe:
+
+```json
+{
+ "layer": "frontDoor",
+ "metrics": { "requestsTotal": 41, "requestsFailed": 7, "avgLatencyMs": 3.1, "...": "..." },
+ "failures": {
+ "layer": "frontDoor",
+ "requestsFailed": 7,
+ "recent": [
+ { "time": "...", "requestId": "a1b2c3", "method": "GET", "path": "/tabs",
+ "status": 401, "type": "http_error", "layer": "frontDoor" }
+ ]
+ },
+ "crashes": { "total": 0, "recent": [] }
+}
+```
+
+**The two layers are never added together.** A single `requestsTotal` covering both
+would mean two things at once, and an operator cannot act on that number: a spike
+would not say whether clients are being turned away at the door or the browser is
+failing. Read each layer at its own endpoint instead:
+
+| Endpoint | Answered by | Reports |
+|----------|-------------|---------|
+| `GET /metrics` | front door (server mode) / the bridge itself (bridge mode) | that process's own counters |
+| `GET /instances/{id}/metrics` | proxied to that instance | that instance's counters |
+| `GET /instances/metrics` | front door | per-instance **browser memory**, not request counters |
+| `GET /tabs/{id}/metrics` | proxied to the owning instance | one tab's browser measurements |
+
+`failures.recent` repeats `layer` on each event as well as on the block, so an
+event pasted into a bug report still says where it came from.
+
+`failures` and `crashes` are always present, empty when there is nothing to
+report. A key that appeared only after the first failure could not be relied on by
+anything watching for one.
+
+## What the two modes share, and what they do not
+
+Shared, by construction — both modes serve the same diagnostics payload:
+
+- `metrics` (request counters, latency, Go runtime figures)
+- `failures` with `requestsFailed` and `recent`
+- `crashes` with `total` and `recent`
+- `version` on `/health`
+
+Deliberately different:
+
+- **`/metrics` is answered locally in both modes; `/health` is not.** In server mode
+ `/health` is the orchestrator's own envelope (`instances`, `profiles`,
+ `defaultInstance`, `restartRequired`), because those facts exist only at the front
+ door. A bridge has no instances to report. The two `/health` bodies therefore
+ differ on purpose, and `/metrics` is the endpoint to compare across modes.
+- **`memory` appears only on an instance's `/metrics`**, since only a process
+ holding a browser can measure one.
+- **Browser crash events are recorded by whichever process owns the browser**, so in
+ server mode a crash shows up under the instance layer, not the front door — the
+ front door's `crashes` block stays empty by design.
+
+## Related Pages
+
+- [Health](./health.md)
+- [Instances](./instances.md)
diff --git a/internal/handlers/health_tabs.go b/internal/handlers/health_tabs.go
index ac3f3f65..83c2b8c0 100644
--- a/internal/handlers/health_tabs.go
+++ b/internal/handlers/health_tabs.go
@@ -42,12 +42,17 @@ func (h *Handlers) HandleHealth(w http.ResponseWriter, r *http.Request) {
}
resp := map[string]any{"status": "ok", "tabs": len(targets)}
+ // Server-mode /health reports version; bridge /health did not, so a bridge
+ // bug report could not state which build produced it.
+ if h.Version != "" {
+ resp["version"] = h.Version
+ }
if crashLogs := h.Bridge.GetCrashLogs(); len(crashLogs) > 0 {
resp["crashLogs"] = crashLogs
}
if hasFailureDiagnostics() {
- resp["failures"] = FailureSnapshot()
+ resp["failures"] = FailureSnapshot(LayerInstance)
}
if bridge.HasCrashDiagnostics() {
resp["crashes"] = bridge.CrashSnapshot()
@@ -89,14 +94,11 @@ func (h *Handlers) HandleBrowserRestart(w http.ResponseWriter, r *http.Request)
httpx.JSON(w, 200, map[string]string{"status": "browser_restarted"})
}
+// HandleMetrics reports this process's own counters. In server mode that is an
+// instance child: the orchestrator front door answers its own /metrics, so a
+// client can read either layer without the two ever being summed.
func (h *Handlers) HandleMetrics(w http.ResponseWriter, r *http.Request) {
- result := map[string]any{"metrics": SnapshotMetrics()}
- if hasFailureDiagnostics() {
- result["failures"] = FailureSnapshot()
- }
- if bridge.HasCrashDiagnostics() {
- result["crashes"] = bridge.CrashSnapshot()
- }
+ result := DiagnosticsSnapshot(LayerInstance)
if h.Bridge != nil {
if mem, err := h.Bridge.GetAggregatedMemoryMetrics(); err == nil && mem != nil {
diff --git a/internal/handlers/middleware_test.go b/internal/handlers/middleware_test.go
index 977f307d..232914ae 100644
--- a/internal/handlers/middleware_test.go
+++ b/internal/handlers/middleware_test.go
@@ -856,18 +856,18 @@ func TestLoggingMiddleware_RecordsFailure(t *testing.T) {
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
- snap := FailureSnapshot()
+ snap := FailureSnapshot(LayerInstance)
if got := snap["requestsFailed"].(uint64); got != 1 {
t.Fatalf("requestsFailed = %d, want 1", got)
}
- recent, ok := snap["recent"].([]FailureEvent)
+ recent, ok := snap["recent"].([]map[string]any)
if !ok || len(recent) != 1 {
t.Fatalf("recent failures = %#v, want 1 event", snap["recent"])
}
- if recent[0].Path != "/boom" {
- t.Fatalf("recent path = %q, want /boom", recent[0].Path)
+ if recent[0]["path"] != "/boom" {
+ t.Fatalf("recent path = %v, want /boom", recent[0]["path"])
}
- if recent[0].RequestID == "" {
+ if recent[0]["requestId"] == "" {
t.Fatal("expected request id on failure event")
}
}
diff --git a/internal/handlers/observability.go b/internal/handlers/observability.go
index caa15293..23cac0af 100644
--- a/internal/handlers/observability.go
+++ b/internal/handlers/observability.go
@@ -4,6 +4,8 @@ import (
"sync"
"sync/atomic"
"time"
+
+ "github.com/pinchtab/pinchtab/internal/bridge"
)
const maxRecentFailures = 20
@@ -33,18 +35,60 @@ func recordFailureEvent(ev FailureEvent) {
}
}
-// FailureSnapshot returns recent failure diagnostics for /health and /metrics.
-func FailureSnapshot() map[string]any {
+// Layer names the process a set of counters describes. Server mode runs TWO
+// counter sets in two processes — the orchestrator front door, which is the only
+// one that sees auth rejections and unrouted paths, and each browser-control
+// instance — so every number a client reads has to say which one it came from.
+// They are never summed: one requestsTotal meaning two layers is a number an
+// operator cannot act on.
+const (
+ LayerFrontDoor = "frontDoor"
+ LayerInstance = "instance"
+)
+
+// FailureSnapshot returns recent failure diagnostics for /health and /metrics,
+// labelled with the layer that recorded them. The label is repeated on each event
+// as well as on the block, so an event stays self-describing when it is copied
+// out of the response it arrived in.
+func FailureSnapshot(layer string) map[string]any {
failureMu.Lock()
recent := append([]FailureEvent(nil), recentFailures...)
failureMu.Unlock()
+ events := make([]map[string]any, 0, len(recent))
+ for _, ev := range recent {
+ events = append(events, map[string]any{
+ "time": ev.Time,
+ "requestId": ev.RequestID,
+ "method": ev.Method,
+ "path": ev.Path,
+ "status": ev.Status,
+ "type": ev.Type,
+ "layer": layer,
+ })
+ }
+
return map[string]any{
+ "layer": layer,
"requestsFailed": atomic.LoadUint64(&metricRequestsFailed),
- "recent": recent,
+ "recent": events,
}
}
+// DiagnosticsSnapshot is the whole observability payload for one process: its own
+// counters plus the failure and crash logs it recorded. Both modes serve it, so
+// the two cannot drift into exposing different keys — the divergence this closes
+// was exactly that.
+func DiagnosticsSnapshot(layer string) map[string]any {
+ snapshot := map[string]any{
+ "layer": layer,
+ "metrics": SnapshotMetrics(),
+ }
+ snapshot["failures"] = FailureSnapshot(layer)
+ snapshot["crashes"] = bridge.CrashSnapshot()
+ return snapshot
+}
+
func hasFailureDiagnostics() bool {
failureMu.Lock()
recentCount := len(recentFailures)
@@ -52,6 +96,13 @@ func hasFailureDiagnostics() bool {
return atomic.LoadUint64(&metricRequestsFailed) > 0 || recentCount > 0
}
+// ResetObservabilityForTests clears the process counters and failure log. It is
+// exported because the front-door chain lives in internal/server, and the
+// counters it must be proven to move are owned here.
+func ResetObservabilityForTests() {
+ resetObservabilityForTests()
+}
+
func resetObservabilityForTests() {
atomic.StoreUint64(&metricRequestsTotal, 0)
atomic.StoreUint64(&metricRequestsFailed, 0)
diff --git a/internal/handlers/observability_layers_test.go b/internal/handlers/observability_layers_test.go
new file mode 100644
index 00000000..8d9d38f3
--- /dev/null
+++ b/internal/handlers/observability_layers_test.go
@@ -0,0 +1,102 @@
+package handlers
+
+import (
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+
+ "github.com/pinchtab/pinchtab/internal/config"
+)
+
+// Server mode runs two counter sets in two processes. Moving the bare /metrics to
+// the front door must not make the other layer invisible in turn: an instance
+// still reports its OWN traffic, and says so.
+func TestInstanceMetricsStillReportItsOwnTrafficAndLayer(t *testing.T) {
+ resetObservabilityForTests()
+ t.Cleanup(resetObservabilityForTests)
+
+ counted := LoggingMiddleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(http.StatusInternalServerError)
+ }))
+ for i := 0; i < 3; i++ {
+ counted.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/boom", nil))
+ }
+
+ h := New(&mockBridge{}, &config.RuntimeConfig{}, nil, nil, nil)
+ rec := httptest.NewRecorder()
+ h.HandleMetrics(rec, httptest.NewRequest("GET", "/metrics", nil))
+ if rec.Code != http.StatusOK {
+ t.Fatalf("GET /metrics = %d: %s", rec.Code, rec.Body.String())
+ }
+
+ var body map[string]any
+ if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
+ t.Fatalf("decode metrics: %v", err)
+ }
+ if body["layer"] != LayerInstance {
+ t.Errorf("layer = %v, want %q", body["layer"], LayerInstance)
+ }
+ metrics, ok := body["metrics"].(map[string]any)
+ if !ok {
+ t.Fatalf("metrics block missing: %v", body)
+ }
+ if got := metrics["requestsTotal"].(float64); got != 3 {
+ t.Errorf("requestsTotal = %v, want the 3 requests this process served", got)
+ }
+ failures, ok := body["failures"].(map[string]any)
+ if !ok {
+ t.Fatalf("failures block missing: %v", body)
+ }
+ if got := failures["requestsFailed"].(float64); got != 3 {
+ t.Errorf("requestsFailed = %v, want 3", got)
+ }
+ if failures["layer"] != LayerInstance {
+ t.Errorf("failures.layer = %v, want %q", failures["layer"], LayerInstance)
+ }
+}
+
+// Both modes now serve the same diagnostic keys, which is what makes bridge mode
+// usable for reproducing a server-mode problem. A shape that differs per mode is
+// the drift this card is closing.
+func TestBothModesExposeTheSameDiagnosticKeys(t *testing.T) {
+ resetObservabilityForTests()
+ t.Cleanup(resetObservabilityForTests)
+
+ frontDoor := DiagnosticsSnapshot(LayerFrontDoor)
+ instance := DiagnosticsSnapshot(LayerInstance)
+
+ for key := range frontDoor {
+ if _, ok := instance[key]; !ok {
+ t.Errorf("instance diagnostics missing %q, which the front door reports", key)
+ }
+ }
+ for key := range instance {
+ if _, ok := frontDoor[key]; !ok {
+ t.Errorf("front-door diagnostics missing %q, which an instance reports", key)
+ }
+ }
+ if frontDoor["layer"] == instance["layer"] {
+ t.Error("both layers report the same label; a client cannot tell the two counter sets apart")
+ }
+}
+
+// A bridge bug report has to be able to state which build produced it.
+func TestBridgeHealthReportsVersion(t *testing.T) {
+ h := New(&mockBridge{}, &config.RuntimeConfig{}, nil, nil, nil)
+ h.Version = "9.9.9-test"
+
+ rec := httptest.NewRecorder()
+ h.HandleHealth(rec, httptest.NewRequest("GET", "/health", nil))
+ if rec.Code != http.StatusOK {
+ t.Fatalf("GET /health = %d: %s", rec.Code, rec.Body.String())
+ }
+
+ var body map[string]any
+ if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
+ t.Fatalf("decode health: %v", err)
+ }
+ if body["version"] != "9.9.9-test" {
+ t.Errorf("version = %v, want the build the bridge is running", body["version"])
+ }
+}
diff --git a/internal/orchestrator/handlers.go b/internal/orchestrator/handlers.go
index 412f5d8a..d67e0696 100644
--- a/internal/orchestrator/handlers.go
+++ b/internal/orchestrator/handlers.go
@@ -62,6 +62,10 @@ func (o *Orchestrator) registerHandlers(mux *http.ServeMux, skipLaunch bool) {
mux.HandleFunc("GET /instances/{id}", o.handleGetInstance)
mux.HandleFunc("GET /instances/tabs", o.handleAllTabs)
mux.HandleFunc("GET /instances/metrics", o.handleAllMetrics)
+ // The front door answers the bare /metrics with its OWN counters, so an
+ // instance's request counters need a path of their own; without it, moving
+ // /metrics to the front door would just make the other layer invisible.
+ mux.HandleFunc("GET /instances/{id}/metrics", o.proxyToInstance)
if !skipLaunch {
mux.HandleFunc("POST /instances/start", o.handleStartInstance)
mux.HandleFunc("POST /instances/launch", o.handleLaunchByName)
diff --git a/internal/routes/routes.go b/internal/routes/routes.go
index cbb5a4ef..b3f79710 100644
--- a/internal/routes/routes.go
+++ b/internal/routes/routes.go
@@ -61,6 +61,18 @@ type Endpoint struct {
TabScoped bool // true = auto-generates /tabs/{id}/... variant
}
+// frontDoorLocalPaths are the shorthand paths the orchestrator front door answers
+// ITSELF instead of proxying to an instance. Proxying a diagnostic endpoint hides
+// the front door's own numbers: it is the only layer that sees auth rejections and
+// unrouted paths, so forwarding the read leaves no way for a client to see them.
+// Declared here rather than as an Endpoint field because the catalogue below is
+// written positionally, and a new field would rewrite every unrelated entry.
+// Tab-scoped variants are unaffected: /tabs/{id}/metrics is a browser measurement
+// and still belongs to the instance.
+var frontDoorLocalPaths = map[string]bool{
+ "/metrics": true,
+}
+
// Route returns the "METHOD /path" string used for mux registration.
func (e Endpoint) Route() string {
return e.Method + " " + e.Path
@@ -198,12 +210,27 @@ func Core() []Endpoint {
return out
}
-// ShorthandRoutes returns all non-capability-gated shorthand routes
-// as "METHOD /path" strings, suitable for mux registration.
+// ShorthandRoutes returns the non-capability-gated shorthand routes the front
+// door PROXIES, as "METHOD /path" strings suitable for mux registration.
+// FrontDoorLocal endpoints are excluded: the front door registers its own
+// handler for those, and registering both would panic on a duplicate pattern.
func ShorthandRoutes() []string {
var routes []string
for _, ep := range coreEndpoints {
- if ep.Capability == CapNone {
+ if ep.Capability == CapNone && !frontDoorLocalPaths[ep.Path] {
+ routes = append(routes, ep.Route())
+ }
+ }
+ return routes
+}
+
+// FrontDoorLocalRoutes returns the routes the front door must register itself.
+// It is the other half of ShorthandRoutes: together they cover every CapNone
+// endpoint exactly once, which the routes test pins.
+func FrontDoorLocalRoutes() []string {
+ var routes []string
+ for _, ep := range coreEndpoints {
+ if ep.Capability == CapNone && frontDoorLocalPaths[ep.Path] {
routes = append(routes, ep.Route())
}
}
diff --git a/internal/server/front_door_metrics.go b/internal/server/front_door_metrics.go
new file mode 100644
index 00000000..c1c9f402
--- /dev/null
+++ b/internal/server/front_door_metrics.go
@@ -0,0 +1,52 @@
+package server
+
+import (
+ "net/http"
+
+ "github.com/pinchtab/pinchtab/internal/activity"
+ "github.com/pinchtab/pinchtab/internal/browsersession"
+ "github.com/pinchtab/pinchtab/internal/config"
+ "github.com/pinchtab/pinchtab/internal/handlers"
+ "github.com/pinchtab/pinchtab/internal/httpx"
+ "github.com/pinchtab/pinchtab/internal/routes"
+ "github.com/pinchtab/pinchtab/internal/session"
+)
+
+// FrontDoorHandler is the orchestrator front door's whole middleware chain. It is
+// a named function rather than an expression inside RunDashboard because the
+// counters live in the middleware and the rejections that were invisible happen
+// ABOVE the mux: proving they are counted needs the real chain, and the isolated
+// middleware already behaved correctly, which is why nothing caught this.
+func FrontDoorHandler(
+ cfg *config.RuntimeConfig,
+ recorder activity.Recorder,
+ sessions *browsersession.Manager,
+ sessionStore *session.Store,
+ mux http.Handler,
+) http.Handler {
+ return handlers.StripInternalHeadersMiddleware(
+ handlers.RequestIDMiddleware(
+ activity.Middleware(
+ recorder,
+ "server",
+ handlers.SecurityHeadersMiddleware(cfg,
+ handlers.LoggingMiddleware(handlers.RateLimitMiddleware(handlers.CorsMiddleware(cfg, handlers.AuthMiddlewareWithSessions(cfg, sessions, sessionStore, mux)))),
+ ),
+ ),
+ ),
+ )
+}
+
+// registerFrontDoorMetrics serves the front door's OWN counters at /metrics.
+// It used to be a proxied route, so a client read the instance's numbers and the
+// front-door counters — the only ones that see auth rejections and unrouted paths
+// — were unreachable by any client. The instance figures did not become
+// unreachable in exchange: an instance still answers its own /metrics, reachable
+// through GET /instances/{id}/metrics.
+func registerFrontDoorMetrics(mux *http.ServeMux) {
+ for _, route := range routes.FrontDoorLocalRoutes() {
+ mux.HandleFunc(route, func(w http.ResponseWriter, r *http.Request) {
+ httpx.JSON(w, http.StatusOK, handlers.DiagnosticsSnapshot(handlers.LayerFrontDoor))
+ })
+ }
+}
diff --git a/internal/server/front_door_metrics_test.go b/internal/server/front_door_metrics_test.go
new file mode 100644
index 00000000..03d877af
--- /dev/null
+++ b/internal/server/front_door_metrics_test.go
@@ -0,0 +1,186 @@
+package server
+
+import (
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+
+ "github.com/pinchtab/pinchtab/internal/config"
+ "github.com/pinchtab/pinchtab/internal/handlers"
+ "github.com/pinchtab/pinchtab/internal/routes"
+)
+
+const frontDoorToken = "tok-front-door-1234567890"
+
+// frontDoorFixture mounts the REAL front-door chain over a mux carrying the real
+// local /metrics route. The rejections this card is about happen ABOVE the mux,
+// in the auth middleware, so a test that exercised LoggingMiddleware on its own
+// would pass while the product stayed blind — which is exactly what happened.
+func frontDoorFixture(t *testing.T) http.Handler {
+ t.Helper()
+ handlers.ResetObservabilityForTests()
+ t.Cleanup(handlers.ResetObservabilityForTests)
+
+ cfg := &config.RuntimeConfig{Token: frontDoorToken, Bind: "127.0.0.1", Port: "9867"}
+ mux := http.NewServeMux()
+ registerFrontDoorMetrics(mux)
+ mux.HandleFunc("GET /tabs", func(w http.ResponseWriter, r *http.Request) {
+ _, _ = w.Write([]byte(`{"tabs":[]}`))
+ })
+ return FrontDoorHandler(cfg, nil, nil, nil, mux)
+}
+
+func readFrontDoorMetrics(t *testing.T, handler http.Handler) map[string]any {
+ t.Helper()
+ req := httptest.NewRequest("GET", "/metrics", nil)
+ req.Header.Set("Authorization", "Bearer "+frontDoorToken)
+ rec := httptest.NewRecorder()
+ handler.ServeHTTP(rec, req)
+ if rec.Code != http.StatusOK {
+ t.Fatalf("GET /metrics = %d: %s", rec.Code, rec.Body.String())
+ }
+ var body map[string]any
+ if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
+ t.Fatalf("decode metrics: %v (body %s)", err, rec.Body.String())
+ }
+ return body
+}
+
+// The reported defect: N bad-token requests left requestsFailed at 0 forever,
+// because /metrics was proxied to the instance and the front-door counters — the
+// only ones that see an auth rejection — were unreachable by any client.
+//
+// The assertion is EXACT, not >=: an inequality passes on double counting, which
+// is the specific risk when two layers of counters exist.
+func TestFrontDoorMetricsCountRejectedRequestsExactly(t *testing.T) {
+ handler := frontDoorFixture(t)
+
+ const rejected = 5
+ for i := 0; i < rejected; i++ {
+ req := httptest.NewRequest("GET", "/tabs", nil)
+ req.Header.Set("Authorization", "Bearer WRONG")
+ rec := httptest.NewRecorder()
+ handler.ServeHTTP(rec, req)
+ if rec.Code != http.StatusUnauthorized {
+ t.Fatalf("request %d = %d, want 401: %s", i, rec.Code, rec.Body.String())
+ }
+ }
+ // An unrouted path is the other case that never reaches an instance.
+ for i := 0; i < 2; i++ {
+ req := httptest.NewRequest("GET", fmt.Sprintf("/zzz%d", i), nil)
+ req.Header.Set("Authorization", "Bearer "+frontDoorToken)
+ rec := httptest.NewRecorder()
+ handler.ServeHTTP(rec, req)
+ if rec.Code != http.StatusNotFound {
+ t.Fatalf("unrouted request %d = %d, want 404", i, rec.Code)
+ }
+ }
+
+ body := readFrontDoorMetrics(t, handler)
+ if body["layer"] != handlers.LayerFrontDoor {
+ t.Fatalf("layer = %v, want %q — a number that does not say which process it describes is not actionable", body["layer"], handlers.LayerFrontDoor)
+ }
+
+ failures, ok := body["failures"].(map[string]any)
+ if !ok {
+ t.Fatalf("no failures block in front-door metrics: %v", body)
+ }
+ if got := failures["requestsFailed"].(float64); got != rejected+2 {
+ t.Errorf("requestsFailed = %v, want exactly %d", got, rejected+2)
+ }
+ recent, ok := failures["recent"].([]any)
+ if !ok {
+ t.Fatalf("failures.recent missing: %v", failures)
+ }
+ if len(recent) != rejected+2 {
+ t.Fatalf("failures.recent has %d events, want exactly %d", len(recent), rejected+2)
+ }
+
+ // Every event must carry the four fields an operator needs to act, plus the
+ // layer that produced it.
+ unauthorized := 0
+ for i, raw := range recent {
+ event, ok := raw.(map[string]any)
+ if !ok {
+ t.Fatalf("event %d is not an object: %v", i, raw)
+ }
+ for _, key := range []string{"method", "path", "status", "requestId", "layer"} {
+ if event[key] == nil || event[key] == "" {
+ t.Errorf("event %d missing %q: %v", i, key, event)
+ }
+ }
+ if event["layer"] != handlers.LayerFrontDoor {
+ t.Errorf("event %d layer = %v, want %q", i, event["layer"], handlers.LayerFrontDoor)
+ }
+ if event["status"].(float64) == http.StatusUnauthorized {
+ unauthorized++
+ if event["path"] != "/tabs" {
+ t.Errorf("401 event %d path = %v, want /tabs", i, event["path"])
+ }
+ if event["method"] != "GET" {
+ t.Errorf("401 event %d method = %v, want GET", i, event["method"])
+ }
+ }
+ }
+ if unauthorized != rejected {
+ t.Errorf("recorded %d auth rejections, want %d", unauthorized, rejected)
+ }
+}
+
+// crashes.recent must be present in server mode too, and with a stable shape: a
+// key that only appears after the first crash cannot be relied on by a client
+// watching for one.
+func TestFrontDoorMetricsAlwaysCarryFailureAndCrashBlocks(t *testing.T) {
+ body := readFrontDoorMetrics(t, frontDoorFixture(t))
+
+ for _, key := range []string{"metrics", "failures", "crashes", "layer"} {
+ if _, ok := body[key]; !ok {
+ t.Errorf("front-door metrics missing %q on a clean server: %v", key, body)
+ }
+ }
+ crashes, ok := body["crashes"].(map[string]any)
+ if !ok {
+ t.Fatalf("crashes block is not an object: %v", body["crashes"])
+ }
+ if _, ok := crashes["recent"]; !ok {
+ t.Errorf("crashes.recent missing: %v", crashes)
+ }
+ failures := body["failures"].(map[string]any)
+ if got := failures["requestsFailed"].(float64); got != 0 {
+ t.Errorf("a clean front door reports requestsFailed = %v, want 0", got)
+ }
+}
+
+// The front door can only answer /metrics itself if the proxy registration stops
+// claiming it — registering both patterns panics. This pins the split rather than
+// the spelling: every CapNone endpoint belongs to exactly one of the two sets.
+func TestMetricsIsFrontDoorLocalAndNotProxied(t *testing.T) {
+ local := routes.FrontDoorLocalRoutes()
+ proxied := routes.ShorthandRoutes()
+
+ if len(local) == 0 {
+ t.Fatal("no front-door-local routes; the front door would proxy its own diagnostics again")
+ }
+ inLocal := map[string]bool{}
+ for _, route := range local {
+ inLocal[route] = true
+ }
+ for _, route := range proxied {
+ if inLocal[route] {
+ t.Errorf("%q is registered both locally and as a proxy route; mux registration would panic", route)
+ }
+ }
+ if !inLocal["GET /metrics"] {
+ t.Errorf("GET /metrics must be answered by the front door: local routes = %v", local)
+ }
+
+ // The tab-scoped variant is a browser measurement and must STILL proxy.
+ for _, route := range routes.TabScopedRoutes() {
+ if route == "GET /tabs/{id}/metrics" {
+ return
+ }
+ }
+ t.Error("GET /tabs/{id}/metrics is no longer proxied; per-tab metrics belong to the instance")
+}
diff --git a/internal/server/server.go b/internal/server/server.go
index 38a974b3..ab517d76 100644
--- a/internal/server/server.go
+++ b/internal/server/server.go
@@ -300,6 +300,7 @@ func RunDashboard(cfg *config.RuntimeConfig, version string) {
}
mux.HandleFunc("GET /health", configAPI.HandleHealth)
+ registerFrontDoorMetrics(mux)
mux.HandleFunc("GET /health/background", func(w http.ResponseWriter, r *http.Request) {
httpx.JSON(w, http.StatusOK, map[string]string{
"status": "ok",
@@ -309,17 +310,7 @@ func RunDashboard(cfg *config.RuntimeConfig, version string) {
})
})
- handler := handlers.StripInternalHeadersMiddleware(
- handlers.RequestIDMiddleware(
- activity.Middleware(
- liveActivity,
- "server",
- handlers.SecurityHeadersMiddleware(cfg,
- handlers.LoggingMiddleware(handlers.RateLimitMiddleware(handlers.CorsMiddleware(cfg, handlers.AuthMiddlewareWithSessions(cfg, sessions, sessionStore, mux)))),
- ),
- ),
- ),
- )
+ handler := FrontDoorHandler(cfg, liveActivity, sessions, sessionStore, mux)
if cfg.VerboseBanner {
cli.LogSecurityWarnings(cfg)
}