The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
fix: minor consistency fixes
narrow-adventuredaed7134ccec
diff --git a/CLAUDE.md b/CLAUDE.mdindex 25574974..5c9c2705 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -125,7 +125,7 @@ func (c *AuthController) Register(ctx *gin.Context) {- Commits on status codes: 200, 201, 303- Rolls back on all other status codes or panics-**Body buffering:** `Transactional` reads the request body into memory (cap `maxTransactionalBodyBytes`, 8MB) before it calls `db.DB.Begin()`, answering 413/408/400 without a transaction when that fails; the handler's later bind is served from memory. This is what keeps a slow-drip body from holding the single SQLite main-DB connection, and it covers every transactional route without each one remembering a special middleware. Routes that want a tighter cap put `middleware.BufferAuthBody` (64KB) in front of `Transactional`, which then skips the body it finds already buffered; the anonymous auth endpoints do this. The self-transacting OAuth routes (`/auth/device/*`, `/auth/token`, `/auth/logout`) use `BufferAuthBody` on its own for the same cap. Every plain JSON bind site in the controllers answers through `middleware.RejectBindError(c, err, fallback)`, which maps `MaxBytesError` to 413 and the body guard's timeout to 408 with fixed messages and otherwise returns 400 with the fallback; the guard itself swaps the raw deadline error (a `*net.OpError` naming the listener address) for `middleware.ErrBodyTimedOut` before it leaves `Read`, so no handler can echo the address.+**Body buffering:** `Transactional` reads the request body into memory (cap `maxTransactionalBodyBytes`, 8MB) before it calls `db.DB.Begin()`, answering 413/408/400 without a transaction when that fails; the handler's later bind is served from memory. This is what keeps a slow-drip body from holding the single SQLite main-DB connection, and it covers every transactional route without each one remembering a special middleware. Routes that want a tighter cap put `middleware.BufferAuthBody` (64KB) in front of `Transactional`, which then skips the body it finds already buffered; the anonymous auth endpoints do this. The self-transacting OAuth routes (`/auth/device/*`, `/auth/token`, `/auth/logout`) use `BufferAuthBody` on its own for the same cap. `UseAppAuth` caps every dashboard route's body at the same 8MB with `MaxBytesReader`, so the non-transactional telemetry reads are bounded too. Every plain JSON bind site in the controllers answers through `middleware.RejectBindError(c, err, fallback)`, which maps `MaxBytesError` to 413 and the body guard's timeout to 408 with fixed messages and otherwise returns 400 with the fallback (the ingest routes use `middleware.RejectIngestBindError`, the same mapping except that the timeout becomes 503 + `Retry-After`, because OTLP exporters retry 503 but treat 408 as permanent; the Traceway SDKs re-queue a failed batch whatever the status); the guard itself swaps the raw deadline error (a `*net.OpError` naming the listener address) for `middleware.ErrBodyTimedOut` before it leaves `Read`, so no handler can echo the address.**Preference:** For CRUD controller methods, always prefer using `middleware.Transactional` in the route + `db.GetTx(ctx)` in the controller over `pgdb.ExecuteTransaction`. The middleware approach keeps controllers flat, avoids nested closures, and follows the established pattern. (The tx getter lives in the `db` package: `db.GetTx(ctx)`, not `middleware.GetTx`.)@@ -596,7 +596,7 @@ backend/| Synthetic runner API | `UseRunnerAuth` only (shared SYNTHETICS_RUNNER_SECRET bearer + self-registration by X-Traceway-Runner-Name; no Transactional — poll holds the request up to 25s) || Public status page | `RateLimitPerIP` only |-**Global middleware** (registered in `cmd/run.go` ahead of the routes): `middleware.GuardBodyReads` (every request body gets the progress guard, 30s idle / 10 min total / 1 KB/s floor after 20s, answering through `middleware.BodyReadError` as 408; routes with their own budget call `GuardBodyRead` again, which re-arms the same guard rather than wrapping twice, so uploads get 30s/30 min, `/api/report` 20s/5 min and OTLP 20s/2 min. The guard clears the socket read deadline the moment the body is complete: Go's background connection read would otherwise time out and cancel the request context of any handler that outlives the idle window. The `http.Server` `ReadTimeout` is a 1h backstop that only applies while a body is still arriving (the guard lifts it for bodiless requests too, so a long-lived handler such as the MCP `GET /mcp` stream is not cancelled after an hour), `ReadHeaderTimeout` is 15s), `configureClientIP` (gin's trusted proxies from `Cfg.TrustedProxyList`, plus `TrustedPlatform` from `TRUSTED_PROXY_HEADER`; a malformed list panics), `warnOnUntrustedForwardedFor` (list mode only: one-time log when a forwarding header arrives from a peer outside `TRUSTED_PROXIES`), and `middleware.SecurityHeaders` (`X-Frame-Options: DENY`, CSP `frame-ancestors 'none'; base-uri 'self'; form-action 'self'`, `Referrer-Policy`, `nosniff`). The two frame headers are skipped for any `/status/*` or `/api/status/*` path so public status pages stay embeddable in an iframe. A status page served at the root of a custom domain is not exempt: the same root serves the logged-in dashboard on every hostname the backend answers on, so it must not be frameable there.+**Global middleware** (registered in `cmd/run.go` ahead of the routes): `middleware.GuardBodyReads` (every request body gets the progress guard, 30s idle / 10 min total / 1 KB/s floor after 20s, answering through `middleware.BodyReadError` as 408; routes with their own budget call `GuardBodyRead` again, which re-arms the same guard rather than wrapping twice, so uploads get 30s/30 min, `/api/report` 20s/10 min and OTLP 20s/2 min. The guard clears the socket read deadline the moment the body is complete: Go's background connection read would otherwise time out and cancel the request context of any handler that outlives the idle window. The `http.Server` `ReadTimeout` is a 1h backstop that only applies while a body is still arriving (the guard lifts it for bodiless requests too, so a long-lived handler such as the MCP `GET /mcp` stream is not cancelled after an hour), `ReadHeaderTimeout` is 15s), `configureClientIP` (gin's trusted proxies from `Cfg.TrustedProxyList`, plus `TrustedPlatform` from `TRUSTED_PROXY_HEADER`; a malformed list panics), `warnOnUntrustedForwardedFor` (list mode only: one-time log when a forwarding header arrives from a peer outside `TRUSTED_PROXIES`), and `middleware.SecurityHeaders` (`X-Frame-Options: DENY`, CSP `frame-ancestors 'none'; base-uri 'self'; form-action 'self'`, `Referrer-Policy`, `nosniff`). The two frame headers are skipped for any `/status/*` or `/api/status/*` path so public status pages stay embeddable in an iframe. A status page served at the root of a custom domain is not exempt: the same root serves the logged-in dashboard on every hostname the backend answers on, so it must not be frameable there.### API Endpoints@@ -664,7 +664,7 @@ backend/| GET | `/api/metrics/application` | App | Application metrics || GET | `/api/metrics/stats` | App | Stats metrics || GET | `/api/metrics/server` | App | Server metrics |-| POST | `/api/metrics/query` | App | Custom metric queries. Bounded in `metric_query.controller.go`: 256KB body, max 50 queries and 20 tag filters each, range at most 731 days and `to > from` (422), buckets widened so no series exceeds 2000 points (the effective `intervalMinutes` is returned at the top level of the response), at most 200 groups per query (the repositories return one extra so the result carries `truncatedGroups: true`), and a 30s deadline for the whole request that answers 504. The discovery endpoints validate their `from`/`to` the same way |+| POST | `/api/metrics/query` | App | Custom metric queries. Bounded in `metric_query.controller.go`: 256KB body, max 50 queries and 20 tag filters each, range at most 731 days and `to >= from` (422), buckets widened so no series exceeds 2000 points (the effective `intervalMinutes` is returned at the top level of the response), at most 200 groups per query (the repositories return one extra so the result carries `truncatedGroups: true`), and a 30s deadline for the whole request that answers 504. The discovery endpoints validate their `from`/`to` the same way || GET | `/api/metrics/discover` | App | Discover available metrics || GET | `/api/metrics/discover/tags` | App | Discover metric tags || GET | `/api/metrics/discover/instances` | App | Distinct `server_name` values in a range (dashboard instance filter) |diff --git a/backend/app/controllers/clientcontrollers/client.controller.go b/backend/app/controllers/clientcontrollers/client.controller.goindex 9d446048..6eeaf3dd 100644--- a/backend/app/controllers/clientcontrollers/client.controller.go+++ b/backend/app/controllers/clientcontrollers/client.controller.go@@ -112,7 +112,7 @@ func (e clientController) Report(c *gin.Context) {var request ReportRequestif err := c.ShouldBindBodyWithJSON(&request); err != nil {parseSpan.End()- middleware.RejectBindError(c, err, err.Error())+ middleware.RejectIngestBindError(c, err, err.Error())return}parseSpan.End()diff --git a/backend/app/controllers/clientcontrollers/profile_ingest.controller.go b/backend/app/controllers/clientcontrollers/profile_ingest.controller.goindex 2c41689b..04034cd5 100644--- a/backend/app/controllers/clientcontrollers/profile_ingest.controller.go+++ b/backend/app/controllers/clientcontrollers/profile_ingest.controller.go@@ -38,7 +38,7 @@ func (e profileIngestController) Ingest(c *gin.Context) {body, err := io.ReadAll(c.Request.Body)if err != nil {- middleware.RejectBindError(c, err, "failed to read request body")+ middleware.RejectIngestBindError(c, err, "failed to read request body")return}diff --git a/backend/app/controllers/metric_query.controller.go b/backend/app/controllers/metric_query.controller.goindex 9afdc0de..484c72e5 100644--- a/backend/app/controllers/metric_query.controller.go+++ b/backend/app/controllers/metric_query.controller.go@@ -71,8 +71,8 @@ func validateMetricQuery(req *MetricQueryRequest) string {}func validateMetricRange(from, to time.Time) string {- if !to.After(from) {- return "to must be after from"+ if to.Before(from) {+ return "to must not be before from"}if to.Sub(from) > metricQueryMaxRange {return fmt.Sprintf("the time range must be at most %d days", int(metricQueryMaxRange.Hours()/24))@@ -136,7 +136,7 @@ func (c *metricQueryController) Query(ctx *gin.Context) {return}- queryCtx, cancel := context.WithTimeout(ctx, metricQueryTimeout)+ queryCtx, cancel := context.WithTimeout(ctx.Request.Context(), metricQueryTimeout)defer cancel()unitMap := make(map[string]string)diff --git a/backend/app/controllers/metric_query_bounds_test.go b/backend/app/controllers/metric_query_bounds_test.goindex 667c3bf4..245dd72a 100644--- a/backend/app/controllers/metric_query_bounds_test.go+++ b/backend/app/controllers/metric_query_bounds_test.go@@ -51,7 +51,7 @@ func TestValidateMetricQueryBounds(t *testing.T) {{"ordinary range", MetricQueryRequest{Queries: one, From: base, To: base.Add(time.Hour)}, false},{"at the range cap", MetricQueryRequest{Queries: one, From: base, To: base.Add(metricQueryMaxRange)}, false},{"reversed", MetricQueryRequest{Queries: one, From: base.Add(time.Hour), To: base}, true},- {"equal", MetricQueryRequest{Queries: one, From: base, To: base}, true},+ {"equal", MetricQueryRequest{Queries: one, From: base, To: base}, false},{"over the range cap", MetricQueryRequest{Queries: one, From: base, To: base.Add(metricQueryMaxRange + time.Hour)}, true},{"absurd timestamps", MetricQueryRequest{Queries: one, From: time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC), To: time.Date(9999, 1, 1, 0, 0, 0, 0, time.UTC)}, true},{"too many tag filters", MetricQueryRequest{Queries: []MetricQueryItem{{Name: "cpu", TagFilters: manyFilters}}, From: base, To: base.Add(time.Hour)}, true},diff --git a/backend/app/controllers/organization_overview.controller.go b/backend/app/controllers/organization_overview.controller.goindex 4af2ebf8..36d5a5f5 100644--- a/backend/app/controllers/organization_overview.controller.go+++ b/backend/app/controllers/organization_overview.controller.go@@ -13,6 +13,7 @@ import ("github.com/tracewayapp/traceway/backend/app/middleware""github.com/tracewayapp/traceway/backend/app/models""github.com/tracewayapp/traceway/backend/app/repositories/telemetry"+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared""github.com/tracewayapp/traceway/backend/app/repositories/transactional""github.com/gin-gonic/gin"@@ -159,16 +160,34 @@ func sumDeviceRates(series map[string][]models.TimeSeriesPoint) (float64, bool)// system.network.io is a cumulative counter per device, so summing samples// inside a bucket (or across devices) makes the bucket delta meaningless. The// last sample per device per minute, differenced and then summed, is the rate.-func loadNetworkRate(ctx context.Context, projectId uuid.UUID, serverName, direction string, since, to time.Time) (*float64, error) {- series, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, projectId, models.MetricNameSystemNetworkIO, since, to, 1, "last", map[string]string{"direction": direction, "server_name": serverName}, "device", 0)+func loadNetworkRates(ctx context.Context, projectId uuid.UUID, direction string, since, to time.Time) (map[string]float64, error) {+ series, err := telemetry.MetricPointRepository.QueryTimeSeriesByTags(ctx, projectId, models.MetricNameSystemNetworkIO, since, to, 1, "last", map[string]string{"direction": direction}, []string{"device", "server_name"}, 0)if err != nil {return nil, err}- rate, ok := sumDeviceRates(series)- if !ok {- return nil, nil+ return networkRatesByServer(series), nil+}++func networkRatesByServer(series map[string][]models.TimeSeriesPoint) map[string]float64 {+ byServer := map[string]map[string][]models.TimeSeriesPoint{}+ for key, points := range series {+ parts := shared.SplitGroupKey(key)+ if len(parts) != 2 {+ continue+ }+ device, server := parts[0], parts[1]+ if byServer[server] == nil {+ byServer[server] = map[string][]models.TimeSeriesPoint{}+ }+ byServer[server][device] = points+ }+ rates := make(map[string]float64, len(byServer))+ for server, devices := range byServer {+ if rate, ok := sumDeviceRates(devices); ok {+ rates[server] = rate+ }}- return pointerTo(rate), nil+ return rates}func applyServerMetadata(row *orgServerRow, tags map[string]string) {@@ -248,6 +267,14 @@ func loadProjectServerRows(ctx *gin.Context, project *models.Project, dashboardIreturn nil, err}networkTo := now.Truncate(time.Minute).Add(-time.Millisecond)+ receive, err := loadNetworkRates(ctx, project.Id, "receive", since, networkTo)+ if err != nil {+ return nil, err+ }+ transmit, err := loadNetworkRates(ctx, project.Id, "transmit", since, networkTo)+ if err != nil {+ return nil, err+ }for _, point := range latestOtel {row := rows[point.ServerName]@@ -261,11 +288,11 @@ func loadProjectServerRows(ctx *gin.Context, project *models.Project, dashboardIif value, ok := lastSeriesValue(filesystemUsed, point.ServerName); ok {row.DiskPct = pointerTo(otelFractionToPercent(value))}- if row.NetworkRxBps, err = loadNetworkRate(ctx, project.Id, point.ServerName, "receive", since, networkTo); err != nil {- return nil, err+ if rate, ok := receive[point.ServerName]; ok {+ row.NetworkRxBps = pointerTo(rate)}- if row.NetworkTxBps, err = loadNetworkRate(ctx, project.Id, point.ServerName, "transmit", since, networkTo); err != nil {- return nil, err+ if rate, ok := transmit[point.ServerName]; ok {+ row.NetworkTxBps = pointerTo(rate)}}}diff --git a/backend/app/controllers/organization_overview_test.go b/backend/app/controllers/organization_overview_test.goindex e93dc8f9..d7651011 100644--- a/backend/app/controllers/organization_overview_test.go+++ b/backend/app/controllers/organization_overview_test.go@@ -6,6 +6,7 @@ import ("time""github.com/tracewayapp/traceway/backend/app/models"+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared")func TestCpuUsageTrendConvertsIdleFraction(t *testing.T) {@@ -67,3 +68,24 @@ func TestOtelFractionToPercent(t *testing.T) {}}}++func TestNetworkRatesByServer(t *testing.T) {+ now := time.Now().UTC()+ sep := shared.GroupKeySeparator+ rates := networkRatesByServer(map[string][]models.TimeSeriesPoint{+ "eth0" + sep + "web-1": {{Timestamp: now, Value: 1_000}, {Timestamp: now.Add(time.Minute), Value: 7_000}},+ "lo" + sep + "web-1": {{Timestamp: now, Value: 10}, {Timestamp: now.Add(time.Minute), Value: 610}},+ "eth0" + sep + "web-2": {{Timestamp: now, Value: 500}, {Timestamp: now.Add(time.Minute), Value: 1_100}},+ "fresh" + sep + "web-3": {{Timestamp: now.Add(time.Minute), Value: 42}},+ "__all__": {{Timestamp: now, Value: 1}, {Timestamp: now.Add(time.Minute), Value: 2}},+ })+ if len(rates) != 2 {+ t.Fatalf("rates = %v, want web-1 and web-2 only", rates)+ }+ if rates["web-1"] != 110 {+ t.Fatalf("web-1 = %v, want 110 bytes/s (100 eth0 + 10 lo)", rates["web-1"])+ }+ if rates["web-2"] != 10 {+ t.Fatalf("web-2 = %v, want 10 bytes/s", rates["web-2"])+ }+}diff --git a/backend/app/controllers/otelcontrollers/codec.go b/backend/app/controllers/otelcontrollers/codec.goindex 56de56cf..8d744a7c 100644--- a/backend/app/controllers/otelcontrollers/codec.go+++ b/backend/app/controllers/otelcontrollers/codec.go@@ -64,15 +64,11 @@ func isMaxBytes(err error) bool {}func writeDecodeError(c *gin.Context, err error) {- if status, message, ok := middleware.BodyReadError(err); ok {- c.JSON(status, gin.H{"error": message})- return- }- status := http.StatusBadRequestif errors.Is(err, errBodyTooLarge) {- status = http.StatusRequestEntityTooLarge+ c.JSON(http.StatusRequestEntityTooLarge, gin.H{"error": err.Error()})+ return}- c.JSON(status, gin.H{"error": err.Error()})+ middleware.RejectIngestBindError(c, err, err.Error())}func isProtobuf(c *gin.Context) bool {diff --git a/backend/app/middleware/body_error.go b/backend/app/middleware/body_error.goindex 07e834a7..a47f7a8d 100644--- a/backend/app/middleware/body_error.go+++ b/backend/app/middleware/body_error.go@@ -26,3 +26,18 @@ func RejectBindError(c *gin.Context, err error, fallback string) {}c.JSON(status, gin.H{"error": message})}++// OTLP exporters retry a 503 but treat a 408 as permanent and drop the batch,+// so a stalled ingest body is turned away as retryable. The Traceway SDKs+// re-queue a failed batch whatever the status, so this changes nothing for them.+func RejectIngestBindError(c *gin.Context, err error, fallback string) {+ status, message, ok := BodyReadError(err)+ if !ok {+ status, message = http.StatusBadRequest, fallback+ }+ if status == http.StatusRequestTimeout {+ status = http.StatusServiceUnavailable+ c.Header("Retry-After", "2")+ }+ c.JSON(status, gin.H{"error": message})+}diff --git a/backend/app/middleware/body_error_test.go b/backend/app/middleware/body_error_test.goindex e384a68d..f91c1dd6 100644--- a/backend/app/middleware/body_error_test.go+++ b/backend/app/middleware/body_error_test.go@@ -85,3 +85,34 @@ func TestRejectBindError(t *testing.T) {})}}++func TestRejectIngestBindError(t *testing.T) {+ gin.SetMode(gin.TestMode)+ cases := []struct {+ name string+ err error+ wantStatus int+ wantRetryAfter string+ }{+ {"body too slow becomes retryable", ErrBodyTooSlow, http.StatusServiceUnavailable, "2"},+ {"read deadline becomes retryable", ErrBodyTimedOut, http.StatusServiceUnavailable, "2"},+ {"max bytes stays 413", &http.MaxBytesError{Limit: 64}, http.StatusRequestEntityTooLarge, ""},+ {"plain bind error stays 400", errors.New("bad json"), http.StatusBadRequest, ""},+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ rec := httptest.NewRecorder()+ c, _ := gin.CreateTestContext(rec)+ c.Request = httptest.NewRequest(http.MethodPost, "/", nil)++ RejectIngestBindError(c, tc.err, "Invalid request body")++ if rec.Code != tc.wantStatus {+ t.Fatalf("status = %d, want %d (body %q)", rec.Code, tc.wantStatus, rec.Body.String())+ }+ if got := rec.Header().Get("Retry-After"); got != tc.wantRetryAfter {+ t.Fatalf("Retry-After = %q, want %q", got, tc.wantRetryAfter)+ }+ })+ }+}diff --git a/backend/app/middleware/rate_limit.middleware.go b/backend/app/middleware/rate_limit.middleware.goindex 7c8f7a2f..2d39a17f 100644--- a/backend/app/middleware/rate_limit.middleware.go+++ b/backend/app/middleware/rate_limit.middleware.go@@ -3,6 +3,7 @@ package middlewareimport ("bytes""encoding/json"+ "fmt""io""math""net/http"@@ -74,7 +75,7 @@ func rejectRateLimited(c *gin.Context, retryAfter time.Duration) {seconds = 1}c.Header("Retry-After", strconv.Itoa(seconds))- c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{"error": "slow_down"})+ c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{"error": fmt.Sprintf("Too many requests, try again in %ds", seconds)})}func rateLimitWithKey(maxRequests int, window time.Duration, keyOf func(c *gin.Context) string) gin.HandlerFunc {diff --git a/backend/app/middleware/read_deadline.go b/backend/app/middleware/read_deadline.goindex 417b7ef3..a0ee575d 100644--- a/backend/app/middleware/read_deadline.go+++ b/backend/app/middleware/read_deadline.go@@ -38,10 +38,17 @@ type progressBody struct {lastSet time.Timestart time.Timeread int64+ started booldone bool}func (p *progressBody) Read(b []byte) (int, error) {+ // The clock starts at the first read: everything before it (auth, a+ // queued main-DB transaction) is the server's time, not the client's.+ if !p.started {+ p.started = true+ p.restart(time.Now(), p.idle, p.total)+ }n, err := p.rc.Read(b)if errors.Is(err, os.ErrDeadlineExceeded) {err = ErrBodyTimedOutdiff --git a/backend/app/middleware/use_app_auth.middleware.go b/backend/app/middleware/use_app_auth.middleware.goindex b0a9627a..0649f208 100644--- a/backend/app/middleware/use_app_auth.middleware.go+++ b/backend/app/middleware/use_app_auth.middleware.go@@ -22,6 +22,11 @@ const patTouchInterval = time.Minutefunc InitUseAppAuth() {UseAppAuth = func(c *gin.Context) {+ // Every dashboard route binds JSON that encoding/json buffers whole, so+ // the body is capped once here rather than at each bind site.+ if c.Request.Body != nil && c.Request.Body != http.NoBody {+ c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxTransactionalBodyBytes)+ }authHeader := c.GetHeader("Authorization")if !strings.HasPrefix(authHeader, "Bearer ") {diff --git a/backend/app/middleware/use_gzip_decompress.middleware.go b/backend/app/middleware/use_gzip_decompress.middleware.goindex bfe0a08e..13492c65 100644--- a/backend/app/middleware/use_gzip_decompress.middleware.go+++ b/backend/app/middleware/use_gzip_decompress.middleware.go@@ -26,7 +26,7 @@ func UseGzip(c *gin.Context) {const (reportBodyIdle = 20 * time.Second- reportBodyTotal = 5 * time.Minute+ reportBodyTotal = DefaultBodyTotal)func useGzipLimited(c *gin.Context, maxBytes int64) {@@ -46,7 +46,8 @@ func useGzipLimited(c *gin.Context, maxBytes int64) {gzReader, err := gzip.NewReader(body)if err != nil {- c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": "Invalid gzip"})+ RejectIngestBindError(c, err, "Invalid gzip")+ c.Abort()return}c.Request.Body = http.MaxBytesReader(c.Writer, gzReader, maxBytes)diff --git a/backend/app/repositories/telemetry/clickhouse/metric_point.repository.go b/backend/app/repositories/telemetry/clickhouse/metric_point.repository.goindex b6daae00..709f2773 100644--- a/backend/app/repositories/telemetry/clickhouse/metric_point.repository.go+++ b/backend/app/repositories/telemetry/clickhouse/metric_point.repository.go@@ -30,6 +30,20 @@ func (r *metricPointRepository) InsertAsync(ctx context.Context, points []models}func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {+ var groupKeys []string+ if groupBy != "" {+ groupKeys = []string{groupBy}+ }+ return r.queryTimeSeries(ctx, projectId, name, from, to, intervalMinutes, aggregation, tagFilters, groupKeys, maxGroups)+}++// QueryTimeSeriesByTags groups by every tag in groupBy at once. Result keys are+// the tag values in that order joined by shared.GroupKeySeparator.+func (r *metricPointRepository) QueryTimeSeriesByTags(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy []string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {+ return r.queryTimeSeries(ctx, projectId, name, from, to, intervalMinutes, aggregation, tagFilters, groupBy, maxGroups)+}++func (r *metricPointRepository) queryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupKeys []string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {table := selectTable(to.Sub(from))if aggregation == "last" {table = "metric_points"@@ -41,10 +55,19 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uargs := []interface{}{intervalMinutes}- hasGroupBy := groupBy != ""+ hasGroupBy := len(groupKeys) > 0if hasGroupBy {- query += ", tags[?] AS group_key"- args = append(args, groupBy)+ for i, key := range groupKeys {+ if i > 0 {+ query += " || ? || "+ args = append(args, shared.GroupKeySeparator)+ } else {+ query += ", "+ }+ query += "tags[?]"+ args = append(args, key)+ }+ query += " AS group_key"}query += ", " + aggFunc + " AS agg_value FROM " + table + " WHERE project_id = ? AND name = ? AND recorded_at >= ? AND recorded_at <= ?"@@ -99,6 +122,9 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uValue: value,})}+ if err := rows.Err(); err != nil {+ return nil, err+ }return result, nil}diff --git a/backend/app/repositories/telemetry/duckdb/metric_point.repository.go b/backend/app/repositories/telemetry/duckdb/metric_point.repository.goindex 6b321e1c..454133d5 100644--- a/backend/app/repositories/telemetry/duckdb/metric_point.repository.go+++ b/backend/app/repositories/telemetry/duckdb/metric_point.repository.go@@ -62,13 +62,31 @@ func (r *metricPointRepository) InsertAsync(ctx context.Context, points []models}func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {+ var groupKeys []string+ if groupBy != "" {+ groupKeys = []string{groupBy}+ }+ return r.queryTimeSeries(ctx, projectId, name, from, to, intervalMinutes, aggregation, tagFilters, groupKeys, maxGroups)+}++// QueryTimeSeriesByTags groups by every tag in groupBy at once. Result keys are+// the tag values in that order joined by shared.GroupKeySeparator.+func (r *metricPointRepository) QueryTimeSeriesByTags(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy []string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {+ return r.queryTimeSeries(ctx, projectId, name, from, to, intervalMinutes, aggregation, tagFilters, groupBy, maxGroups)+}++func (r *metricPointRepository) queryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupKeys []string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {secs := intervalMinutes * 60aggFunc := duckdbAggregationFunc(aggregation)- hasGroupBy := groupBy != ""+ hasGroupBy := len(groupKeys) > 0selectClause := "SELECT " + timeBucketExpr("recorded_at", secs) + " AS bucket"if hasGroupBy {- selectClause += ", json_extract_string(tags, '$.\"' || :group_by || '\"') AS group_key"+ exprs := make([]string, len(groupKeys))+ for i := range groupKeys {+ exprs[i] = fmt.Sprintf("json_extract_string(tags, '$.\"' || :group_by_%d || '\"')", i)+ }+ selectClause += ", " + strings.Join(exprs, " || :group_sep || ") + " AS group_key"}selectClause += ", " + aggFunc + " AS agg_value FROM metric_points WHERE project_id = :project_id AND name = :name AND recorded_at >= :from AND recorded_at <= :to"@@ -78,8 +96,11 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId u"from": from.UTC(),"to": to.UTC(),}- if hasGroupBy {- params["group_by"] = groupBy+ for i, key := range groupKeys {+ params[fmt.Sprintf("group_by_%d", i)] = key+ }+ if len(groupKeys) > 1 {+ params["group_sep"] = shared.GroupKeySeparator}filterClauses := ""@@ -144,6 +165,9 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uValue: value,})}+ if err := rows.Err(); err != nil {+ return nil, err+ }return result, nil}diff --git a/backend/app/repositories/telemetry/metric_point_repository_test.go b/backend/app/repositories/telemetry/metric_point_repository_test.goindex 2416d0e4..af8d6427 100644--- a/backend/app/repositories/telemetry/metric_point_repository_test.go+++ b/backend/app/repositories/telemetry/metric_point_repository_test.go@@ -7,6 +7,7 @@ import ("github.com/google/uuid""github.com/tracewayapp/traceway/backend/app/models"+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared")func TestMetricPointRepository_InsertAndQueryTimeSeries(t *testing.T) {@@ -530,3 +531,65 @@ func TestMetricPointRepository_QueryTimeSeries_LastPerDeviceWithServerFilter(t *t.Fatalf("bucket gap = %v, want 1m", gap)}}++func TestMetricPointRepository_QueryTimeSeries_CompositeGroupBy(t *testing.T) {+ setupTestDB(t)+ ctx := context.Background()+ projectId := uuid.New()+ base := time.Now().UTC().Truncate(time.Hour)+ tags := func(server, device string) map[string]string {+ return map[string]string{"server_name": server, "device": device, "direction": "receive"}+ }++ points := []models.MetricPoint{+ makeMetricPoint(projectId, "system.network.io", 1000, tags("web-1", "eth0"), base),+ makeMetricPoint(projectId, "system.network.io", 2200, tags("web-1", "eth0"), base.Add(time.Minute)),+ makeMetricPoint(projectId, "system.network.io", 10, tags("web-1", "lo"), base),+ makeMetricPoint(projectId, "system.network.io", 20, tags("web-1", "lo"), base.Add(time.Minute)),+ makeMetricPoint(projectId, "system.network.io", 500, tags("web-2", "eth0"), base),+ makeMetricPoint(projectId, "system.network.io", 900, tags("web-2", "eth0"), base.Add(time.Minute)),+ }+ if err := MetricPointRepository.InsertAsync(ctx, points); err != nil {+ t.Fatalf("InsertAsync failed: %v", err)+ }++ result, err := MetricPointRepository.QueryTimeSeriesByTags(ctx, projectId, "system.network.io", base.Add(-time.Minute), base.Add(2*time.Minute), 1, "last", map[string]string{"direction": "receive"}, []string{"device", "server_name"}, 0)+ if err != nil {+ t.Fatalf("QueryTimeSeriesByTags failed: %v", err)+ }+ sep := shared.GroupKeySeparator+ want := map[string][2]float64{"eth0" + sep + "web-1": {1000, 2200}, "lo" + sep + "web-1": {10, 20}, "eth0" + sep + "web-2": {500, 900}}+ if len(result) != len(want) {+ t.Fatalf("expected one series per device and server, got %v", result)+ }+ for key, values := range want {+ if len(result[key]) != 2 {+ t.Fatalf("group %s: expected 2 buckets, got %d", key, len(result[key]))+ }+ assertApproxEqual(t, key+" bucket 0", result[key][0].Value, values[0], 0.1)+ assertApproxEqual(t, key+" bucket 1", result[key][1].Value, values[1], 0.1)+ }+}++func TestMetricPointRepository_QueryTimeSeries_GroupByKeyWithComma(t *testing.T) {+ setupTestDB(t)+ ctx := context.Background()+ projectId := uuid.New()+ base := time.Now().UTC().Truncate(time.Hour)++ points := []models.MetricPoint{+ makeMetricPoint(projectId, "cpu.used_pcnt", 50.0, map[string]string{"a,b": "x", "a": "wrong"}, base),+ makeMetricPoint(projectId, "cpu.used_pcnt", 80.0, map[string]string{"a,b": "y", "a": "wrong"}, base),+ }+ if err := MetricPointRepository.InsertAsync(ctx, points); err != nil {+ t.Fatalf("InsertAsync failed: %v", err)+ }++ result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "cpu.used_pcnt", base.Add(-time.Minute), base.Add(time.Hour), 60, "avg", nil, "a,b", 0)+ if err != nil {+ t.Fatalf("QueryTimeSeries failed: %v", err)+ }+ if len(result) != 2 || len(result["x"]) != 1 || len(result["y"]) != 1 {+ t.Fatalf("a comma in groupBy must name one tag key literally, got %v", result)+ }+}diff --git a/backend/app/repositories/telemetry/shared/metric_limits.go b/backend/app/repositories/telemetry/shared/metric_limits.goindex 302858a3..ddb39b03 100644--- a/backend/app/repositories/telemetry/shared/metric_limits.go+++ b/backend/app/repositories/telemetry/shared/metric_limits.go@@ -1,5 +1,7 @@package shared+import "strings"+const MaxTimeSeriesBuckets = 2000func TimeSeriesRowLimit(maxGroups int) int {@@ -19,3 +21,11 @@ func GroupCapReached[T any](result map[string][]T, groupKey string, maxGroups inresult[groupKey] = nilreturn true}++// GroupKeySeparator joins the tag values of a QueryTimeSeriesByTags result key.+// A control character, so no tag value can be mistaken for a boundary.+const GroupKeySeparator = "\x1f"++func SplitGroupKey(key string) []string {+ return strings.Split(key, GroupKeySeparator)+}diff --git a/backend/app/repositories/telemetry/sqlite/metric_point.repository.go b/backend/app/repositories/telemetry/sqlite/metric_point.repository.goindex 1cf32123..641fbda9 100644--- a/backend/app/repositories/telemetry/sqlite/metric_point.repository.go+++ b/backend/app/repositories/telemetry/sqlite/metric_point.repository.go@@ -69,14 +69,32 @@ func (r *metricPointRepository) InsertAsync(ctx context.Context, points []models}func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {+ var groupKeys []string+ if groupBy != "" {+ groupKeys = []string{groupBy}+ }+ return r.queryTimeSeries(ctx, projectId, name, from, to, intervalMinutes, aggregation, tagFilters, groupKeys, maxGroups)+}++// QueryTimeSeriesByTags groups by every tag in groupBy at once. Result keys are+// the tag values in that order joined by shared.GroupKeySeparator.+func (r *metricPointRepository) QueryTimeSeriesByTags(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy []string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {+ return r.queryTimeSeries(ctx, projectId, name, from, to, intervalMinutes, aggregation, tagFilters, groupBy, maxGroups)+}++func (r *metricPointRepository) queryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupKeys []string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {secs := intervalMinutes * 60aggFunc := sqliteAggregationFunc(aggregation)- hasGroupBy := groupBy != ""+ hasGroupBy := len(groupKeys) > 0isLast := aggregation == "last"selectClause := fmt.Sprintf("SELECT datetime((strftime('%%s', recorded_at) / %d) * %d, 'unixepoch') AS bucket", secs, secs)if hasGroupBy {- selectClause += ", json_extract(tags, '$.\"' || :group_by || '\"') AS group_key"+ exprs := make([]string, len(groupKeys))+ for i := range groupKeys {+ exprs[i] = fmt.Sprintf("json_extract(tags, '$.\"' || :group_by_%d || '\"')", i)+ }+ selectClause += ", " + strings.Join(exprs, " || :group_sep || ") + " AS group_key"}selectClause += ", " + aggFunc + " AS agg_value"if isLast {@@ -90,8 +108,11 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId u"from": sqlitetypes.NewSQLiteTime(from),"to": sqlitetypes.NewSQLiteTime(to),}- if hasGroupBy {- params["group_by"] = groupBy+ for i, key := range groupKeys {+ params[fmt.Sprintf("group_by_%d", i)] = key+ }+ if len(groupKeys) > 1 {+ params["group_sep"] = shared.GroupKeySeparator}filterClauses := ""@@ -164,6 +185,9 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uValue: value,})}+ if err := rows.Err(); err != nil {+ return nil, err+ }return result, nil}diff --git a/cli/cmd/traceway/metrics.go b/cli/cmd/traceway/metrics.goindex 0885927b..b0e86ab2 100644--- a/cli/cmd/traceway/metrics.go+++ b/cli/cmd/traceway/metrics.go@@ -109,11 +109,10 @@ func runMetricsQuery(cmd *cobra.Command, _ []string) error {if len(pts) > 0 {latest = fmt.Sprintf("%g", pts[len(pts)-1].Value)}- cell := group- if r.TruncatedGroups {- cell += " (truncated)"- }- _, _ = fmt.Fprintf(tw, "%s\t%s\t%s\t%d\t%s\n", r.Name, pickStr(r.Unit, "-"), cell, len(pts), latest)+ _, _ = fmt.Fprintf(tw, "%s\t%s\t%s\t%d\t%s\n", r.Name, pickStr(r.Unit, "-"), group, len(pts), latest)+ }+ if r.TruncatedGroups {+ _, _ = fmt.Fprintf(tw, "%s\t%s\t(more groups not shown)\t-\t-\n", r.Name, pickStr(r.Unit, "-"))}}return tw.Flush()diff --git a/docs/pages/server/index.mdx b/docs/pages/server/index.mdxindex 3218a7c3..cb572e2c 100644--- a/docs/pages/server/index.mdx+++ b/docs/pages/server/index.mdx@@ -1,9 +1,14 @@+import { Callout } from "nextra/components";import { DeploymentCards } from "../../components/DeploymentCards";# Self HostTraceway is fully self-hosted. You run the entire platform on your own infrastructure. This section covers how to get Traceway up and running.+<Callout type="warning">+ Upgrading an instance that sits behind a reverse proxy, ingress, or CDN? Read the [request limits upgrade notes](/server/upgrade-request-limits) first. The server now trusts `X-Forwarded-For` from loopback only, and until `TRUSTED_PROXIES` lists every hop, all of your users share one rate-limit bucket.+</Callout>+## Deployment Options<DeploymentCards />diff --git a/docs/pages/server/upgrade-request-limits.mdx b/docs/pages/server/upgrade-request-limits.mdxindex 0395f7ce..ef5fd211 100644--- a/docs/pages/server/upgrade-request-limits.mdx+++ b/docs/pages/server/upgrade-request-limits.mdx@@ -54,7 +54,7 @@ The ingest admission gate (`INGEST_MAX_CONCURRENT`, default 2 x CPU cores with a## Server timeouts-The server now closes connections whose headers or body do not finish arriving. Headers must arrive within 15 seconds. Every request body has a progress deadline: no more than 30 seconds between bytes, 10 minutes in total. Routes that take large bodies have their own budget: uploads may take up to 30 minutes as long as data keeps flowing, `/api/report` up to 5 minutes and OTLP up to 2 minutes, both with 20 seconds between bytes instead of 30. Time spent waiting in an admission gate does not count against any of these. A client that averages under 1 KB/s after the first 20 seconds is cut off with `408`; a body that has fully arrived is never rejected for being slow. Idle keep-alive connections are closed after 2 minutes.+The server now closes connections whose headers or body do not finish arriving. Headers must arrive within 15 seconds. Every request body has a progress deadline: no more than 30 seconds between bytes, 10 minutes in total. Routes that take large bodies have their own budget: uploads may take up to 30 minutes as long as data keeps flowing, `/api/report` up to 10 minutes and OTLP up to 2 minutes, both with 20 seconds between bytes instead of 30. Time spent waiting in an admission gate does not count against any of these. A client that averages under 1 KB/s after the first 20 seconds is cut off with `408` (`503` with `Retry-After` on `/api/report`, `/api/profiles/ingest` and the OTLP endpoints, so OTLP exporters retry instead of dropping the batch); a body that has fully arrived is never rejected for being slow. Idle keep-alive connections are closed after 2 minutes.## Public status pages in iframes@@ -62,7 +62,7 @@ Every response now carries `X-Frame-Options: DENY` and a `frame-ancestors 'none'## Metric query limits-`POST /api/metrics/query` enforces: at most 50 queries per request and 20 tag filters per query, a time range of at most 731 days with `to` after `from` (`422`), buckets widened so no series exceeds 2000 points (the response reports the effective `intervalMinutes`), at most 200 groups per query (the result carries `truncatedGroups: true` when cut), and a 30-second deadline for the whole request that answers `504`. Dashboards built in the UI stay well inside these limits.+`POST /api/metrics/query` enforces: at most 50 queries per request and 20 tag filters per query, a time range of at most 731 days with `to` not before `from` (equal timestamps are accepted; `422` otherwise), buckets widened so no series exceeds 2000 points (the response reports the effective `intervalMinutes`), at most 200 groups per query (the result carries `truncatedGroups: true` when cut), and a 30-second deadline for the whole request that answers `504`. Dashboards built in the UI stay well inside these limits.## Transactional requestsdiff --git a/frontend/src/lib/components/dashboard/widget-renderer.svelte b/frontend/src/lib/components/dashboard/widget-renderer.svelteindex ff8e7d68..445ccab1 100644--- a/frontend/src/lib/components/dashboard/widget-renderer.svelte+++ b/frontend/src/lib/components/dashboard/widget-renderer.svelte@@ -50,6 +50,7 @@let singleValue = $state<number | null>(null);let resolvedUnit = $state('');let truncated = $state(false);+ let rangeError = $state('');let loadSequence = 0;const colors = [@@ -142,6 +143,7 @@);if (sequence !== loadSequence) return;truncated = response.results.some((r) => r.truncatedGroups);+ rangeError = '';const units = new SvelteSet<string>();const usedKeys = new SvelteSet<string>();@@ -174,9 +176,10 @@const aggregation = widget.config.sources?.[0]?.aggregation || 'avg';singleValue = newSeries.length > 0 ? reduceSeries(newSeries[0].data, aggregation) : null;}- } catch {+ } catch (e) {if (sequence !== loadSequence) return;// Retain the last successful snapshot during transient query failures.+ rangeError = (e as { status?: number }).status === 422 ? (e as Error).message : '';} finally {if (sequence === loadSequence) loading = false;}@@ -374,4 +377,7 @@{#if truncated && !loading}<div class="px-2 pt-1 text-xs text-muted-foreground">Showing the first 200 groups</div>{/if}+ {#if rangeError && !loading}+ <div class="px-2 pt-1 text-xs text-destructive">{rangeError}</div>+ {/if}</div>diff --git a/frontend/src/routes/endpoints/+page.svelte b/frontend/src/routes/endpoints/+page.svelteindex 0add1b93..96270562 100644--- a/frontend/src/routes/endpoints/+page.svelte+++ b/frontend/src/routes/endpoints/+page.svelte@@ -96,6 +96,8 @@let endpoints = $state<EndpointStats[]>([]);let loading = $state(true);+ let loadSequence = 0;+ let chartSequence = 0;let error = $state('');// Chart state@@ -258,6 +260,7 @@}async function loadChartData() {+ const sequence = ++chartSequence;chartLoading = true;try {@@ -276,6 +279,7 @@},{ projectId: projectsState.currentProjectId ?? undefined })) as ChartResponse;+ if (sequence !== chartSequence) return;chartEndpoints = response.endpoints || [];chartSeries = (response.series || []).map(@@ -286,15 +290,17 @@}));} catch (e) {+ if (sequence !== chartSequence) return;console.error('Failed to load chart data:', e);chartEndpoints = [];chartSeries = [];} finally {- chartLoading = false;+ if (sequence === chartSequence) chartLoading = false;}}async function loadData(pushToHistory = true) {+ const sequence = ++loadSequence;loading = true;error = '';@@ -325,15 +331,17 @@const response = await api.post('/endpoints/grouped', requestBody, {projectId: projectsState.currentProjectId ?? undefined});+ if (sequence !== loadSequence) return;endpoints = response.data || [];total = response.pagination.total;totalPages = response.pagination.totalPages;} catch (e) {+ if (sequence !== loadSequence) return;console.error(e);error = getErrorMessage(e) || 'Failed to load data';} finally {- loading = false;+ if (sequence === loadSequence) loading = false;}// Also load chart datadiff --git a/frontend/src/routes/organization/+page.svelte b/frontend/src/routes/organization/+page.svelteindex e46eb84d..4bab7f63 100644--- a/frontend/src/routes/organization/+page.svelte+++ b/frontend/src/routes/organization/+page.svelte@@ -52,7 +52,6 @@const redirectHref = $derived.by(() => {if (currentOrganizationId === null) return null;if (organizationProjects.length === 1) return `/?projectId=${organizationProjects[0].id}`;- if (organizationProjects.length === 0) return '/setup';return null;});@@ -126,6 +125,8 @@{#if currentOrganizationId === null}<EmptyState message="You are not a member of any organization yet." />+ {:else if organizationProjects.length === 0}+ <EmptyState message="This organization has no projects yet." />{:else if activeTab === 'overview'}<OverviewTab organizationId={currentOrganizationId} {hasBackendProjects} />{:else if activeTab === 'issues'}diff --git a/frontend/src/routes/tasks/+page.svelte b/frontend/src/routes/tasks/+page.svelteindex 9e7e6a68..3445ecb6 100644--- a/frontend/src/routes/tasks/+page.svelte+++ b/frontend/src/routes/tasks/+page.svelte@@ -53,6 +53,7 @@let tasks = $state<TaskStats[]>([]);let loading = $state(true);+ let loadSequence = 0;let error = $state('');// Pagination State@@ -164,6 +165,7 @@}async function loadData(pushToHistory = true) {+ const sequence = ++loadSequence;loading = true;error = '';@@ -195,15 +197,17 @@const response = await api.post('/tasks/grouped', requestBody, {projectId: projectsState.currentProjectId ?? undefined});+ if (sequence !== loadSequence) return;tasks = response.data || [];total = response.pagination.total;totalPages = response.pagination.totalPages;} catch (e) {+ if (sequence !== loadSequence) return;console.error(e);error = getErrorMessage(e) || 'Failed to load data';} finally {- loading = false;+ if (sequence === loadSequence) loading = false;}}