The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
feat: rate limits on payload size, timing and gzip size (#300)
narrow-adventure4e18b2ecca5a
diff --git a/CLAUDE.md b/CLAUDE.mdindex 2ca99993..42f783ee 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -119,6 +119,8 @@ func (c *AuthController) Register(ctx *gin.Context) {- Commits on status codes: 200, 201, 303- Rolls back on all other status codes or panics+**Body buffering:** `Transactional` reads the request body into memory (cap `maxTransactionalBodyBytes`, 8MB) before it calls `db.DB.Begin()`, answering 413/408/400 without a transaction when that fails; the handler's later bind is served from memory. This is what keeps a slow-drip body from holding the single SQLite main-DB connection, and it covers every transactional route without each one remembering a special middleware. Routes that want a tighter cap put `middleware.BufferAuthBody` (64KB) in front of `Transactional`, which then skips the body it finds already buffered; the anonymous auth endpoints do this. The self-transacting OAuth routes (`/auth/device/*`, `/auth/token`, `/auth/logout`) use `BufferAuthBody` on its own for the same cap. Every other bind failure that is really a body problem goes through `middleware.RejectBindError`, which maps `MaxBytesError` to 413 and the guard's deadline errors to 408 with fixed messages (the deadline error is a `*net.OpError` naming the listener address, so it is never echoed).+**Preference:** For CRUD controller methods, always prefer using `middleware.Transactional` in the route + `db.GetTx(ctx)` in the controller over `pgdb.ExecuteTransaction`. The middleware approach keeps controllers flat, avoids nested closures, and follows the established pattern. (The tx getter lives in the `db` package: `db.GetTx(ctx)`, not `middleware.GetTx`.)#### Repository Pattern@@ -242,8 +244,10 @@ if err != nil {```JWT_SECRET=<min 32 char secret for JWT signing>APP_BASE_URL= # public origin of this server (e.g. https://traceway.example.com). Used as the OAuth issuer / device verification URL and SSO redirect base. If unset, the device-auth + well-known endpoints derive it per-request from the Host / X-Forwarded-* headers; set it explicitly behind a proxy that doesn't forward Host.-TRUSTED_PROXIES=127.0.0.1,::1 # comma-separated IPs/CIDRs of reverse proxies whose X-Forwarded-For gin trusts when resolving c.ClientIP(), the per-IP rate-limit key (cmd/run.go SetTrustedProxies via cfg.TrustedProxyList; default loopback so a direct client cannot spoof it, "*" restores gin's trust-everything behavior). The unauthenticated credential endpoints sit behind middleware.RateLimitPerIP — budgets live in routes.go — placed before Transactional so throttled requests never open a transaction.-TRUSTED_PROXY_HEADER= # optional header trusted verbatim as the client IP (gin TrustedPlatform, e.g. X-Real-IP behind ingress-nginx, CF-Connecting-IP behind Cloudflare); takes precedence over TRUSTED_PROXIES. Only safe when every request path goes through a proxy that overwrites the header+TRUSTED_PROXIES= # comma-separated IPs/CIDRs whose X-Forwarded-For/X-Real-IP gin trusts for c.ClientIP() (every per-IP rate limiter keys on it, and `/api/report` stores it on every session as `client.ip`). Unset = loopback only, so no peer can spoof its IP. Private ranges are deliberately NOT trusted by default: on a Docker/k8s network every peer has a private address, the clients included when the port is published directly, so trusting RFC1918 would let any of them forge XFF past every per-IP limiter. Set it to the CIDRs of whatever sits in front (a proxy container's compose network, the ingress pod range, the CDN ranges); otherwise every visitor shares one rate-limit bucket keyed on the proxy. The value REPLACES the default, so it must name every hop in the chain, not just the outermost (a CDN-only list leaves a private ingress untrusted and disables XFF entirely); `*` trusts every peer. A malformed entry panics at startup (`configureClientIP`, cmd/run.go). The first request carrying X-Forwarded-For/X-Real-IP from a peer outside the list logs a one-time warning naming that peer (`warnOnUntrustedForwardedFor`, cmd/run.go), which is how a forgotten proxy hop shows up. Parsing lives in `Cfg.TrustedProxyList` (config.go), covered by config_test.go.+TRUSTED_PROXY_HEADER= # optional header taken verbatim as the client IP on every request (gin TrustedPlatform: X-Real-IP behind ingress-nginx, CF-Connecting-IP behind Cloudflare); it wins over TRUSTED_PROXIES and turns the untrusted-forwarder warning off. Only safe when every request provably passes through a proxy that overwrites the header.+UPLOAD_MAX_CONCURRENT=4 # concurrent /api/sourcemaps/upload + /api/symbols/upload requests; excess waits 30s then gets 503. Separate from INGEST_MAX_CONCURRENT so a CI upload burst cannot starve telemetry ingest. Each upload holds a whole file in memory while parsing, so this bounds peak upload memory at roughly concurrency x largest file. It is a count-based gate, not a memory budget: on a memory-capped container size it from the limit rather than trusting the default. Same gate and bounded waiting room as ingest, its own instance. The source-map warm-up that follows an upload (`services.GenerateTWArtifacts`) runs outside this gate on its own 2-worker pool with a 64-job queue: the stale `.tw` artifact is deleted and the cache invalidated inline, the rebuild is queued, identical pending jobs are coalesced, and a full queue drops the warm-up (rate-limited `CaptureException`) because lookups build a missing artifact on demand anyway.+REPORT_MAX_BODY_MB=64 # cap on the DECOMPRESSED /api/report and /api/profiles/ingest body (middleware.UseGzip bounds both the raw and gunzipped stream, so gzip cannot raise it). Deliberately far above the 10MB OTLP cap: session-replay frames carry rrweb segments and are a different size class, and the shipped browser SDKs re-queue a rejected batch forever, so a reachable cap is a permanent wedge rather than shed load. Overruns answer 413.CLICKHOUSE_SERVER=localhost:9000CLICKHOUSE_DATABASE=tracewayCLICKHOUSE_USERNAME=default@@ -262,7 +266,7 @@ DUCKDB_THREADS= # e.g. 4. Unset = DuckDB auto-tunes (= corDUCKDB_CHECKPOINT_THRESHOLD= # e.g. 256MB. Unset = DuckDB default (16MB). Raise under sustained ingest to reduce WAL checkpoint stalls; costs a larger WAL and longer restart replay.# Ingest admission gate (all telemetry ingest endpoints: /api/report, /api/profiles/ingest, /api/otel/*)-INGEST_MAX_CONCURRENT= # max concurrently processed ingest requests. Unset = 2×CPU cores, min 4. Bounds ingest memory so overload sheds load with 503s instead of the process being OOM-killed (on DuckDB an OOM death is followed by a minutes-long WAL-replay stall on restart).+INGEST_MAX_CONCURRENT= # max concurrently processed ingest requests. Unset = 2×CPU cores, min 4. Bounds ingest memory so overload sheds load with 503s instead of the process being OOM-killed (on DuckDB an OOM death is followed by a minutes-long WAL-replay stall on restart). The gate (`newAdmissionGate` in `middleware/ingest_admission.go`, a buffered-channel semaphore) also bounds waiters at 4×capacity (min 16); beyond that a request gets an immediate 503 instead of parking a goroutine and a timer for the wait window.INGEST_ADMISSION_WAIT_SECONDS=5 # how long a request may wait for a slot before the 503 + Retry-After; 0 = reject immediately when saturated@@ -574,11 +578,13 @@ backend/| PostgreSQL CRUD (read) | `UseAppAuth, RequireProjectAccess, Transactional` || PostgreSQL CRUD (write) | `UseAppAuth, RequireProjectAccess, RequireWriteAccess, Transactional` || Admin org management | `UseAppAuth, RequireAdminAccess, Transactional` |-| Public (auth/invitations) | `Transactional` only |+| Public (auth/invitations) | `RateLimitPerIP`, `BufferAuthBody`, `Transactional` (limiter first so a throttled request never reads a body or opens a transaction) || Client SDK ingestion | `CORSReport, UseClientAuth, UseGzip` || Synthetic runner API | `UseRunnerAuth` only (shared SYNTHETICS_RUNNER_SECRET bearer + self-registration by X-Traceway-Runner-Name; no Transactional — poll holds the request up to 25s) || Public status page | `RateLimitPerIP` only |+**Global middleware** (registered in `cmd/run.go` ahead of the routes): `middleware.GuardBodyReads` (every request body gets the progress guard, 30s idle / 10 min total / 1 KB/s floor after 20s, answering through `middleware.BodyReadError` as 408; routes with their own budget call `GuardBodyRead` again, which re-arms the same guard rather than wrapping twice, so uploads get 30s/30 min, `/api/report` 20s/5 min and OTLP 20s/2 min. The guard clears the socket read deadline the moment the body is complete: Go's background connection read would otherwise time out and cancel the request context of any handler that outlives the idle window. The `http.Server` `ReadTimeout` is a 1h backstop that only applies while a body is still arriving, `ReadHeaderTimeout` is 15s), `configureClientIP` (gin's trusted proxies from `Cfg.TrustedProxyList`, plus `TrustedPlatform` from `TRUSTED_PROXY_HEADER`; a malformed list panics), `warnOnUntrustedForwardedFor` (list mode only: one-time log when a forwarding header arrives from a peer outside `TRUSTED_PROXIES`), and `middleware.SecurityHeaders` (`X-Frame-Options: DENY`, CSP `frame-ancestors 'none'; base-uri 'self'; form-action 'self'`, `Referrer-Policy`, `nosniff`). The two frame headers are skipped for any `/status/*` or `/api/status/*` path so public status pages stay embeddable in an iframe. A status page served at the root of a custom domain is not exempt: the same root serves the logged-in dashboard on every hostname the backend answers on, so it must not be frameable there.+### API Endpoints**Client SDK Ingestion**@@ -604,8 +610,8 @@ backend/| Method | Endpoint | Auth | Purpose ||--------|----------|------|---------|| POST | `/api/auth/device/authorize` | None | Start device flow; returns device/user code + verification URL |-| POST | `/api/auth/device/token` | None | Poll for the token (grant `device_code`); also accepts `refresh_token` |-| POST | `/api/auth/token` | None | Token endpoint: `device_code` or `refresh_token` grant (JSON or form-encoded) |+| POST | `/api/auth/device/token` | None | Poll for the token (grant `device_code`); also accepts `refresh_token`. Shares a 60/min per-IP limiter with `/api/auth/token`; over it a `device_code` grant answers the RFC 8628 `400 slow_down` (`RateLimitOAuthTokenPerIP` peeks `grant_type` from the buffered body), so the shipped CLI backs off instead of aborting the login, while every other grant gets a plain `429` + `Retry-After` because `slow_down` is undefined for them and OAuth clients treat an unknown 400 as terminal. Body capped at 64KB by `BufferAuthBody` like the other anonymous auth routes |+| POST | `/api/auth/token` | None | Token endpoint: `device_code`, `refresh_token` or `authorization_code` grant (JSON or form-encoded). Same shared limiter, grant-aware rejection and 64KB body cap as the device token route || POST | `/api/auth/logout` | None | Revoke the presented refresh token's family (idempotent) || GET | `/api/device` | App | Look up a user code for the approval screen || POST | `/api/device/approve` | App | Approve a pending device authorization (tokens carry only the approving user's own role, so no write guard) |@@ -645,7 +651,7 @@ backend/| GET | `/api/metrics/application` | App | Application metrics || GET | `/api/metrics/stats` | App | Stats metrics || GET | `/api/metrics/server` | App | Server metrics |-| POST | `/api/metrics/query` | App | Custom metric queries |+| POST | `/api/metrics/query` | App | Custom metric queries. Bounded in `metric_query.controller.go`: 256KB body, max 50 queries and 20 tag filters each, range at most 731 days and `to > from` (422), buckets widened so no series exceeds 2000 points, at most 200 groups per query (the repositories return one extra so the result carries `truncatedGroups: true`), and a 30s deadline for the whole request that answers 504. The discovery endpoints validate their `from`/`to` the same way || GET | `/api/metrics/discover` | App | Discover available metrics || GET | `/api/metrics/discover/tags` | App | Discover metric tags || GET | `/api/metrics/discover/instances` | App | Distinct `server_name` values in a range (dashboard instance filter) |diff --git a/backend/.env.docker b/backend/.env.dockerindex e9799105..30476720 100644--- a/backend/.env.docker+++ b/backend/.env.docker@@ -17,3 +17,12 @@ GIN_MODE=release# SQLITE_RETENTION_DAYS only takes effect when DB_TYPE=sqlite.#SESSION_RECORDING_RETENTION_DAYS=30#SQLITE_RETENTION_DAYS=30++# Proxies whose X-Forwarded-For is trusted for the real client IP, which the+# per-IP rate limits key on. Only loopback is trusted by default, so list+# every hop in front of Traceway (proxy container, ingress, CDN).+#TRUSTED_PROXIES=173.245.48.0/20,103.21.244.0/22,10.0.0.0/8+#TRUSTED_PROXY_HEADER=CF-Connecting-IP++# Cap on the decompressed /api/report body in megabytes. Default 64.+#REPORT_MAX_BODY_MB=64diff --git a/backend/app/config/config.go b/backend/app/config/config.goindex 83f35059..ab460099 100644--- a/backend/app/config/config.go+++ b/backend/app/config/config.go@@ -93,6 +93,7 @@ type Cfg struct {APIOnly stringPorts stringTurnstileSecretKey string+ ReportMaxBodyMB stringGoogleClientID stringGoogleClientSecret string@@ -132,6 +133,16 @@ func PollSeconds(value string, defaultSeconds int) time.Duration {return time.Duration(seconds) * time.Second}+const maxConfigurableMB = 1 << 20++func SizeMB(value string, defaultMB int) int64 {+ mb := int64(defaultMB)+ if parsed, err := strconv.ParseInt(strings.TrimSpace(value), 10, 64); err == nil && parsed > 0 && parsed <= maxConfigurableMB {+ mb = parsed+ }+ return mb * 1024 * 1024+}+// PasswordLoginDisabled reports whether the instance is SSO-only// (DISABLE_PASSWORD_LOGIN=true), which turns off password login,// registration, and the password-reset flow.@@ -253,6 +264,7 @@ func LoadFromEnv() *Cfg {APIOnly: os.Getenv("API_ONLY"),Ports: os.Getenv("PORTS"),TurnstileSecretKey: os.Getenv("TURNSTILE_SECRET_KEY"),+ ReportMaxBodyMB: os.Getenv("REPORT_MAX_BODY_MB"),GoogleClientID: os.Getenv("GOOGLE_CLIENT_ID"),GoogleClientSecret: os.Getenv("GOOGLE_CLIENT_SECRET"),diff --git a/backend/app/controllers/clientcontrollers/client.controller.go b/backend/app/controllers/clientcontrollers/client.controller.goindex 3f42b4fd..9d446048 100644--- a/backend/app/controllers/clientcontrollers/client.controller.go+++ b/backend/app/controllers/clientcontrollers/client.controller.go@@ -112,7 +112,7 @@ func (e clientController) Report(c *gin.Context) {var request ReportRequestif err := c.ShouldBindBodyWithJSON(&request); err != nil {parseSpan.End()- c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})+ middleware.RejectBindError(c, err, err.Error())return}parseSpan.End()diff --git a/backend/app/controllers/clientcontrollers/profile_ingest.controller.go b/backend/app/controllers/clientcontrollers/profile_ingest.controller.goindex 9cacdfd3..2c41689b 100644--- a/backend/app/controllers/clientcontrollers/profile_ingest.controller.go+++ b/backend/app/controllers/clientcontrollers/profile_ingest.controller.go@@ -38,7 +38,7 @@ func (e profileIngestController) Ingest(c *gin.Context) {body, err := io.ReadAll(c.Request.Body)if err != nil {- c.JSON(http.StatusBadRequest, gin.H{"error": "failed to read request body"})+ middleware.RejectBindError(c, err, "failed to read request body")return}diff --git a/backend/app/controllers/clientcontrollers/report_limit_test.go b/backend/app/controllers/clientcontrollers/report_limit_test.gonew file mode 100644index 00000000..20e32f05--- /dev/null+++ b/backend/app/controllers/clientcontrollers/report_limit_test.go@@ -0,0 +1,70 @@+package clientcontrollers++import (+ "bytes"+ "compress/gzip"+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"++ "github.com/gin-gonic/gin"+ "github.com/google/uuid"+ "github.com/tracewayapp/traceway/backend/app/config"+ "github.com/tracewayapp/traceway/backend/app/middleware"+)++func TestReportAnswers413ForOversizedBody(t *testing.T) {+ config.Init(&config.Cfg{ReportMaxBodyMB: "1"})++ gzipBody := func(t *testing.T, payload []byte) []byte {+ t.Helper()+ var buf bytes.Buffer+ zw := gzip.NewWriter(&buf)+ if _, err := zw.Write(payload); err != nil {+ t.Fatal(err)+ }+ if err := zw.Close(); err != nil {+ t.Fatal(err)+ }+ return buf.Bytes()+ }++ oversized := []byte(`{"appVersion":"1.0.0","serverName":"t","collectionFrames":[{"traces":[],"metrics":[],"stackTraces":[{"stackTrace":"` ++ strings.Repeat("x", 2<<20) + `","recordedAt":"2026-01-15T10:30:00Z","attributes":{},"isMessage":false,"isTask":false}]}]}`)++ cases := []struct {+ name string+ body []byte+ encoding string+ want int+ }{+ {"oversized gzipped body", gzipBody(t, oversized), "gzip", http.StatusRequestEntityTooLarge},+ {"oversized plain body", oversized, "", http.StatusRequestEntityTooLarge},+ {"malformed json is still a 400", []byte(`{"nope"`), "", http.StatusBadRequest},+ }++ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ gin.SetMode(gin.TestMode)+ router := gin.New()+ router.POST("/api/report",+ func(c *gin.Context) { c.Set(middleware.ProjectIdContextKey, uuid.New()) },+ middleware.UseGzip,+ ClientController.Report,+ )++ req := httptest.NewRequest(http.MethodPost, "/api/report", bytes.NewReader(tc.body))+ req.Header.Set("Content-Type", "application/json")+ if tc.encoding != "" {+ req.Header.Set("Content-Encoding", tc.encoding)+ }+ rec := httptest.NewRecorder()+ router.ServeHTTP(rec, req)++ if rec.Code != tc.want {+ t.Fatalf("got %d %s, want %d", rec.Code, rec.Body.String(), tc.want)+ }+ })+ }+}diff --git a/backend/app/controllers/contact_method.controller.go b/backend/app/controllers/contact_method.controller.goindex f5351083..53fa7ccb 100644--- a/backend/app/controllers/contact_method.controller.go+++ b/backend/app/controllers/contact_method.controller.go@@ -8,6 +8,7 @@ import ("errors""fmt""net/http"+ netmail "net/mail""strconv""strings""time"@@ -215,12 +216,15 @@ func (c *contactMethodController) Update(ctx *gin.Context) {if request.MethodType == "" {request.MethodType = method.MethodType}+ configProvided := request.Config != nilif request.Config == nil {request.Config = json.RawMessage(method.Config)}- if message := validateContactMethod(request.MethodType, request.Config); message != "" {- ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})- return+ if configProvided || request.MethodType != method.MethodType {+ if message := validateContactMethod(request.MethodType, request.Config); message != "" {+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})+ return+ }}// Re-pointing an sms method at a different number voids the verification.@@ -479,8 +483,10 @@ func validateContactMethod(methodType string, methodConfig json.RawMessage) strireturn "Invalid email configuration."}}- if parsed.Email != "" && !strings.Contains(parsed.Email, "@") {- return "The email address is not valid."+ if parsed.Email != "" {+ if _, err := netmail.ParseAddress(parsed.Email); err != nil {+ return "The email address is not valid."+ }}if len(parsed.Email) > maxContactEmailLength {return "The email address is too long."diff --git a/backend/app/controllers/dashboards.controller.go b/backend/app/controllers/dashboards.controller.goindex 0843f186..2ddba543 100644--- a/backend/app/controllers/dashboards.controller.go+++ b/backend/app/controllers/dashboards.controller.go@@ -28,8 +28,18 @@ const maxDashboardNameLength = 100const maxDashboardBodyBytes = 5 << 20+const maxJSONBodyBytes = 256 << 10+func limitRequestBody(ctx *gin.Context) {- ctx.Request.Body = http.MaxBytesReader(ctx.Writer, ctx.Request.Body, maxDashboardBodyBytes)+ limitRequestBodyTo(ctx, maxDashboardBodyBytes)+}++func limitJSONBody(ctx *gin.Context) {+ limitRequestBodyTo(ctx, maxJSONBodyBytes)+}++func limitRequestBodyTo(ctx *gin.Context, maxBytes int64) {+ ctx.Request.Body = http.MaxBytesReader(ctx.Writer, ctx.Request.Body, maxBytes)}func definitionProvided(raw json.RawMessage) bool {diff --git a/backend/app/controllers/exception_stack_trace.controller.go b/backend/app/controllers/exception_stack_trace.controller.goindex 9d5ce1a3..741c919d 100644--- a/backend/app/controllers/exception_stack_trace.controller.go+++ b/backend/app/controllers/exception_stack_trace.controller.go@@ -36,6 +36,8 @@ type ExceptionSearchRequest struct {IncludeArchived bool `json:"includeArchived"`}+const maxBulkIssueHashes = 100+type ArchiveRequest struct {Hashes []string `json:"hashes"`// ResolvePages also resolves any unresolved on-call pages that were opened@@ -218,9 +220,11 @@ func (e exceptionStackTraceController) ArchiveExceptions(c *gin.Context) {return}+ limitJSONBody(c)+var request ArchiveRequestif err := c.ShouldBindJSON(&request); err != nil {- c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})+ middleware.RejectBindError(c, err, err.Error())return}@@ -228,6 +232,10 @@ func (e exceptionStackTraceController) ArchiveExceptions(c *gin.Context) {c.JSON(http.StatusBadRequest, gin.H{"error": "hashes array is required"})return}+ if len(request.Hashes) > maxBulkIssueHashes {+ c.JSON(http.StatusUnprocessableEntity, gin.H{"error": fmt.Sprintf("a maximum of %d issues can be archived at once", maxBulkIssueHashes)})+ return+ }err = telemetry.ExceptionStackTraceRepository.ArchiveByHashes(c, projectId, request.Hashes)if err != nil {@@ -274,9 +282,11 @@ func (e exceptionStackTraceController) UnarchiveExceptions(c *gin.Context) {return}+ limitJSONBody(c)+var request ArchiveRequestif err := c.ShouldBindJSON(&request); err != nil {- c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})+ middleware.RejectBindError(c, err, err.Error())return}@@ -284,6 +294,10 @@ func (e exceptionStackTraceController) UnarchiveExceptions(c *gin.Context) {c.JSON(http.StatusBadRequest, gin.H{"error": "hashes array is required"})return}+ if len(request.Hashes) > maxBulkIssueHashes {+ c.JSON(http.StatusUnprocessableEntity, gin.H{"error": fmt.Sprintf("a maximum of %d issues can be unarchived at once", maxBulkIssueHashes)})+ return+ }err = telemetry.ExceptionStackTraceRepository.UnarchiveByHashes(c, projectId, request.Hashes)if err != nil {diff --git a/backend/app/controllers/metric_query.controller.go b/backend/app/controllers/metric_query.controller.goindex 30e7bf8a..3fc50989 100644--- a/backend/app/controllers/metric_query.controller.go+++ b/backend/app/controllers/metric_query.controller.go@@ -1,7 +1,9 @@package controllersimport (+ "context""database/sql"+ "errors""fmt""net/http""strconv"@@ -11,6 +13,7 @@ import ("github.com/tracewayapp/traceway/backend/app/middleware""github.com/tracewayapp/traceway/backend/app/models""github.com/tracewayapp/traceway/backend/app/repositories/telemetry"+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared""github.com/tracewayapp/traceway/backend/app/repositories/transactional""github.com/gin-gonic/gin"@@ -20,6 +23,74 @@ import (type metricQueryController struct{}+const (+ metricQueryMaxQueries = 50+ metricQueryMaxBuckets = shared.MaxTimeSeriesBuckets+ metricQueryMaxGroups = 200+ metricQueryMaxRange = 731 * 24 * time.Hour+ metricQueryMaxTagFilters = 20+ metricQueryTimeout = 30 * time.Second+ metricQueryMaxIntervalMinutes = 366 * 24 * 60+)++func minIntervalMinutes(rangeSeconds, maxBuckets int) int {+ if maxBuckets < 2 {+ maxBuckets = 2+ }+ per := 60 * (maxBuckets - 1)+ iv := (rangeSeconds + per - 1) / per+ if iv < 1 {+ iv = 1+ }+ return iv+}++func validateMetricQuery(req *MetricQueryRequest) string {+ if len(req.Queries) > metricQueryMaxQueries {+ return fmt.Sprintf("a maximum of %d queries can be requested at once", metricQueryMaxQueries)+ }+ if msg := validateMetricRange(req.From, req.To); msg != "" {+ return msg+ }+ for _, q := range req.Queries {+ if len(q.TagFilters) > metricQueryMaxTagFilters {+ return fmt.Sprintf("a maximum of %d tag filters can be applied per query", metricQueryMaxTagFilters)+ }+ }++ if req.IntervalMinutes <= 0 {+ req.IntervalMinutes = calculateIntervalMinutes(req.To.Sub(req.From))+ }+ if req.IntervalMinutes > metricQueryMaxIntervalMinutes {+ return fmt.Sprintf("intervalMinutes must be at most %d", metricQueryMaxIntervalMinutes)+ }+ if minInterval := minIntervalMinutes(int(req.To.Sub(req.From).Seconds()), metricQueryMaxBuckets); req.IntervalMinutes < minInterval {+ req.IntervalMinutes = minInterval+ }+ return ""+}++func validateMetricRange(from, to time.Time) string {+ if !to.After(from) {+ return "to must be after from"+ }+ if to.Sub(from) > metricQueryMaxRange {+ return fmt.Sprintf("the time range must be at most %d days", int(metricQueryMaxRange.Hours()/24))+ }+ return ""+}++func trimGroupSeries(series map[string][]models.TimeSeriesPoint) bool {+ truncated := false+ for k, points := range series {+ if points == nil {+ delete(series, k)+ truncated = true+ }+ }+ return truncated+}+type MetricQueryRequest struct {Queries []MetricQueryItem `json:"queries" binding:"required,min=1"`From time.Time `json:"from" binding:"required"`@@ -39,9 +110,10 @@ type MetricQueryResponse struct {}type MetricQueryResult struct {- Name string `json:"name"`- Unit string `json:"unit"`- Series map[string][]models.TimeSeriesPoint `json:"series"`+ Name string `json:"name"`+ Unit string `json:"unit"`+ Series map[string][]models.TimeSeriesPoint `json:"series"`+ TruncatedGroups bool `json:"truncatedGroups,omitempty"`}func (c *metricQueryController) Query(ctx *gin.Context) {@@ -51,16 +123,21 @@ func (c *metricQueryController) Query(ctx *gin.Context) {return}+ limitJSONBody(ctx)+var req MetricQueryRequestif err := ctx.ShouldBindJSON(&req); err != nil {- ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})+ middleware.RejectBindError(ctx, err, "Invalid request body")return}-- if req.IntervalMinutes <= 0 {- req.IntervalMinutes = calculateIntervalMinutes(req.To.Sub(req.From))+ if msg := validateMetricQuery(&req); msg != "" {+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": msg})+ return}+ queryCtx, cancel := context.WithTimeout(ctx, metricQueryTimeout)+ defer cancel()+unitMap := make(map[string]string)registry, regErr := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.MetricRegistry, error) {return transactional.MetricRegistryRepository.FindByProject(tx, projectId)@@ -90,14 +167,19 @@ func (c *metricQueryController) Query(ctx *gin.Context) {}span := traceway.StartSpan(ctx, spanName)series, err := telemetry.MetricPointRepository.QueryTimeSeries(- ctx, projectId, q.Name, req.From, req.To,- req.IntervalMinutes, agg, q.TagFilters, q.GroupBy,+ queryCtx, projectId, q.Name, req.From, req.To,+ req.IntervalMinutes, agg, q.TagFilters, q.GroupBy, metricQueryMaxGroups,)span.End()if err != nil {+ if errors.Is(err, context.DeadlineExceeded) || errors.Is(queryCtx.Err(), context.DeadlineExceeded) {+ ctx.JSON(http.StatusGatewayTimeout, gin.H{"error": fmt.Sprintf("metric query timed out after %s; narrow the time range or send fewer queries", metricQueryTimeout)})+ return+ }ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query metric %s: %w", q.Name, err))return}+ truncated := trimGroupSeries(series)resultUnit := unitMap[q.Name]if agg == "count" {@@ -105,9 +187,10 @@ func (c *metricQueryController) Query(ctx *gin.Context) {}results = append(results, MetricQueryResult{- Name: q.Name,- Unit: resultUnit,- Series: series,+ Name: q.Name,+ Unit: resultUnit,+ Series: series,+ TruncatedGroups: truncated,})}@@ -122,7 +205,7 @@ type DiscoverInstancesResponse struct {Instances []string `json:"instances"`}-func parseMetricDiscoveryRange(ctx *gin.Context) (time.Time, time.Time) {+func parseMetricDiscoveryRange(ctx *gin.Context) (time.Time, time.Time, string) {from := time.Now().AddDate(0, 0, -7)to := time.Now()@@ -137,7 +220,7 @@ func parseMetricDiscoveryRange(ctx *gin.Context) (time.Time, time.Time) {}}- return from, to+ return from, to, validateMetricRange(from, to)}func (c *metricQueryController) Discover(ctx *gin.Context) {@@ -147,7 +230,11 @@ func (c *metricQueryController) Discover(ctx *gin.Context) {return}- from, to := parseMetricDiscoveryRange(ctx)+ from, to, rangeMsg := parseMetricDiscoveryRange(ctx)+ if rangeMsg != "" {+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": rangeMsg})+ return+ }span := traceway.StartSpan(ctx, "discover metrics")discovered, err := telemetry.MetricPointRepository.DiscoverMetrics(ctx, projectId, from, to)@@ -187,7 +274,11 @@ func (c *metricQueryController) DiscoverInstances(ctx *gin.Context) {return}- from, to := parseMetricDiscoveryRange(ctx)+ from, to, rangeMsg := parseMetricDiscoveryRange(ctx)+ if rangeMsg != "" {+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": rangeMsg})+ return+ }span := traceway.StartSpan(ctx, "discover instances")instances, err := telemetry.MetricPointRepository.GetDistinctServers(ctx, projectId, from, to)span.End()diff --git a/backend/app/controllers/metric_query_bounds_test.go b/backend/app/controllers/metric_query_bounds_test.gonew file mode 100644index 00000000..667c3bf4--- /dev/null+++ b/backend/app/controllers/metric_query_bounds_test.go@@ -0,0 +1,76 @@+package controllers++import (+ "fmt"+ "testing"+ "time"+)++func bucketsFor(fromSec, rangeSec, intervalMinutes int) int {+ secs := intervalMinutes * 60+ return (fromSec+rangeSec)/secs - fromSec/secs + 1+}++func TestMinIntervalMinutesNeverExceedsTheCap(t *testing.T) {+ worst := 0+ for rangeMin := 1; rangeMin <= 20000; rangeMin++ {+ rangeSec := rangeMin * 60+ iv := minIntervalMinutes(rangeSec, metricQueryMaxBuckets)+ if iv < 1 {+ t.Fatalf("range %dm produced interval %d", rangeMin, iv)+ }+ for _, off := range []int{0, 1, 17, 31, 59, iv*60 - 1} {+ if n := bucketsFor(off, rangeSec, iv); n > worst {+ worst = n+ }+ }+ }+ if worst > metricQueryMaxBuckets {+ t.Fatalf("worst-case bucket count %d exceeds the %d cap", worst, metricQueryMaxBuckets)+ }+}++func TestValidateMetricQueryBounds(t *testing.T) {+ base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)+ one := []MetricQueryItem{{Name: "cpu"}}++ manyFilters := map[string]string{}+ for i := 0; i <= metricQueryMaxTagFilters; i++ {+ manyFilters[fmt.Sprintf("k%d", i)] = "v"+ }+ manyQueries := make([]MetricQueryItem, metricQueryMaxQueries+1)+ for i := range manyQueries {+ manyQueries[i] = MetricQueryItem{Name: "cpu"}+ }++ cases := []struct {+ name string+ req MetricQueryRequest+ rejects bool+ }{+ {"ordinary range", MetricQueryRequest{Queries: one, From: base, To: base.Add(time.Hour)}, false},+ {"at the range cap", MetricQueryRequest{Queries: one, From: base, To: base.Add(metricQueryMaxRange)}, false},+ {"reversed", MetricQueryRequest{Queries: one, From: base.Add(time.Hour), To: base}, true},+ {"equal", MetricQueryRequest{Queries: one, From: base, To: base}, true},+ {"over the range cap", MetricQueryRequest{Queries: one, From: base, To: base.Add(metricQueryMaxRange + time.Hour)}, true},+ {"absurd timestamps", MetricQueryRequest{Queries: one, From: time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC), To: time.Date(9999, 1, 1, 0, 0, 0, 0, time.UTC)}, true},+ {"too many tag filters", MetricQueryRequest{Queries: []MetricQueryItem{{Name: "cpu", TagFilters: manyFilters}}, From: base, To: base.Add(time.Hour)}, true},+ {"too many queries", MetricQueryRequest{Queries: manyQueries, From: base, To: base.Add(time.Hour)}, true},+ {"interval over the ceiling", MetricQueryRequest{Queries: one, From: base, To: base.Add(time.Hour), IntervalMinutes: metricQueryMaxIntervalMinutes + 1}, true},+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ req := tc.req+ msg := validateMetricQuery(&req)+ if tc.rejects && msg == "" {+ t.Fatal("expected a validation message")+ }+ if !tc.rejects && msg != "" {+ t.Fatalf("unexpected rejection: %s", msg)+ }+ if !tc.rejects && req.IntervalMinutes < 1 {+ t.Fatalf("interval was not defaulted: %d", req.IntervalMinutes)+ }+ })+ }+}diff --git a/backend/app/controllers/multipart_upload.go b/backend/app/controllers/multipart_upload.gonew file mode 100644index 00000000..3e8d3bfa--- /dev/null+++ b/backend/app/controllers/multipart_upload.go@@ -0,0 +1,68 @@+package controllers++import (+ "errors"+ "fmt"+ "io"+ "mime/multipart"+ "net/http"+ "time"++ "github.com/gin-gonic/gin"+ "github.com/tracewayapp/traceway/backend/app/middleware"+)++const (+ maxUploadBytes = 250 << 20+ uploadMultipartMemory = 8 << 20+ uploadBodyIdle = 30 * time.Second+ uploadBodyTotal = 30 * time.Minute+)++type countingReadCloser struct {+ rc io.ReadCloser+ n int64+}++func (c *countingReadCloser) Read(p []byte) (int, error) {+ n, err := c.rc.Read(p)+ c.n += int64(n)+ return n, err+}++func (c *countingReadCloser) Close() error { return c.rc.Close() }++func parseMultipartCapped(c *gin.Context, maxBytes, maxMemory int64) bool {+ middleware.GuardBodyRead(c, uploadBodyIdle, uploadBodyTotal)++ if c.Request.ContentLength > maxBytes {+ c.JSON(http.StatusRequestEntityTooLarge, gin.H{"error": fmt.Sprintf("Total upload size exceeds %dMB", maxBytes>>20)})+ return false+ }++ counter := &countingReadCloser{rc: c.Request.Body}+ c.Request.Body = http.MaxBytesReader(c.Writer, counter, maxBytes)++ err := c.Request.ParseMultipartForm(maxMemory)+ if err == nil {+ if _, err = io.Copy(io.Discard, c.Request.Body); err == nil {+ return true+ }+ c.Request.MultipartForm.RemoveAll()+ }++ var maxBytesErr *http.MaxBytesError+ switch {+ case errors.As(err, &maxBytesErr) || counter.n > maxBytes:+ c.JSON(http.StatusRequestEntityTooLarge, gin.H{"error": fmt.Sprintf("Total upload size exceeds %dMB", maxBytes>>20)})+ case errors.Is(err, multipart.ErrMessageTooLarge):+ c.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Too many files in one request; split the upload into smaller batches"})+ default:+ if status, message, ok := middleware.BodyReadError(err); ok {+ c.JSON(status, gin.H{"error": message})+ } else {+ c.JSON(http.StatusBadRequest, gin.H{"error": "Failed to parse multipart form"})+ }+ }+ return false+}diff --git a/backend/app/controllers/multipart_upload_test.go b/backend/app/controllers/multipart_upload_test.gonew file mode 100644index 00000000..7d1b66d1--- /dev/null+++ b/backend/app/controllers/multipart_upload_test.go@@ -0,0 +1,134 @@+package controllers++import (+ "bytes"+ "fmt"+ "io"+ "mime/multipart"+ "net/http"+ "net/http/httptest"+ "testing"++ "github.com/gin-gonic/gin"+ "github.com/tracewayapp/traceway/backend/app/middleware"+)++func multipartBody(t *testing.T, parts int, size int) (*bytes.Buffer, string) {+ t.Helper()+ var body bytes.Buffer+ w := multipart.NewWriter(&body)+ for i := 0; i < parts; i++ {+ fw, err := w.CreateFormFile("files", fmt.Sprintf("f%d.map", i))+ if err != nil {+ t.Fatal(err)+ }+ if _, err := fw.Write(bytes.Repeat([]byte("x"), size)); err != nil {+ t.Fatal(err)+ }+ }+ if err := w.Close(); err != nil {+ t.Fatal(err)+ }+ return &body, w.FormDataContentType()+}++func runParseCapped(t *testing.T, body []byte, contentType string, contentLength, maxBytes int64) (bool, *httptest.ResponseRecorder) {+ t.Helper()+ gin.SetMode(gin.TestMode)+ rec := httptest.NewRecorder()+ c, _ := gin.CreateTestContext(rec)+ c.Request = httptest.NewRequest(http.MethodPost, "/api/sourcemaps/upload", bytes.NewReader(body))+ c.Request.Header.Set("Content-Type", contentType)+ c.Request.ContentLength = contentLength+ return parseMultipartCapped(c, maxBytes, 1<<20), rec+}++func TestParseMultipartCappedRefusesADeclaredLengthOverTheCapBeforeReading(t *testing.T) {+ body, ct := multipartBody(t, 1, 100)+ raw := body.Bytes()+ ok, rec := runParseCapped(t, raw, ct, int64(len(raw)), int64(len(raw)-1))+ if ok || rec.Code != http.StatusRequestEntityTooLarge {+ t.Fatalf("got ok=%v %d %s, want 413 from the Content-Length precheck", ok, rec.Code, rec.Body.String())+ }+}++func TestParseMultipartCappedAlwaysReports413WhenTruncated(t *testing.T) {+ body, ct := multipartBody(t, 8, 100)+ raw := body.Bytes()++ misreported := 0+ for limit := 0; limit < len(raw); limit++ {+ ok, rec := runParseCapped(t, raw, ct, -1, int64(limit))+ if ok {+ t.Fatalf("limit %d < body %d must not parse", limit, len(raw))+ }+ if rec.Code != http.StatusRequestEntityTooLarge {+ misreported+++ if misreported <= 3 {+ t.Errorf("cut at %d/%d: got %d %s, want 413", limit, len(raw), rec.Code, rec.Body.String())+ }+ }+ }+ if misreported > 0 {+ t.Fatalf("%d of %d truncation points misreported as something other than 413", misreported, len(raw))+ }+}++func TestParseMultipartCappedRejectsUnknownLengthEpilogueOverTheCap(t *testing.T) {+ body, ct := multipartBody(t, 1, 100)+ parts := body.Len()+ raw := append(body.Bytes(), bytes.Repeat([]byte("e"), 1<<20)...)++ ok, rec := runParseCapped(t, raw, ct, -1, int64(parts+4096))+ if ok {+ t.Fatal("a 1MB epilogue slipped past a cap a few KB above the parts")+ }+ if rec.Code != http.StatusRequestEntityTooLarge {+ t.Fatalf("got %d %s, want 413", rec.Code, rec.Body.String())+ }+}++type failAfterReader struct {+ r io.Reader+ err error+}++func (f *failAfterReader) Read(p []byte) (int, error) {+ n, err := f.r.Read(p)+ if err == io.EOF {+ return n, f.err+ }+ return n, err+}++func TestParseMultipartCappedReportsASlowBodyAs408(t *testing.T) {+ body, ct := multipartBody(t, 2, 100)+ raw := body.Bytes()++ gin.SetMode(gin.TestMode)+ rec := httptest.NewRecorder()+ c, _ := gin.CreateTestContext(rec)+ c.Request = httptest.NewRequest(http.MethodPost, "/api/sourcemaps/upload", io.NopCloser(&failAfterReader{r: bytes.NewReader(raw[:len(raw)/2]), err: middleware.ErrBodyTooSlow}))+ c.Request.Header.Set("Content-Type", ct)+ c.Request.ContentLength = -1++ if parseMultipartCapped(c, int64(len(raw))*2, 1<<20) {+ t.Fatal("a body cut by the throughput floor must not parse")+ }+ if rec.Code != http.StatusRequestTimeout {+ t.Fatalf("got %d %s, want 408", rec.Code, rec.Body.String())+ }+}++func TestParseMultipartCappedReportsTooManyPartsAs422(t *testing.T) {+ body, ct := multipartBody(t, 1001, 1)+ raw := body.Bytes()++ ok, rec := runParseCapped(t, raw, ct, int64(len(raw)), int64(len(raw))*2)+ if ok {+ t.Fatal("1001 parts must not parse")+ }+ if rec.Code != http.StatusUnprocessableEntity {+ t.Fatalf("got %d %s, want 422 telling the client to batch", rec.Code, rec.Body.String())+ }+}diff --git a/backend/app/controllers/organization_overview.controller.go b/backend/app/controllers/organization_overview.controller.goindex a3e881c6..7d28fd0a 100644--- a/backend/app/controllers/organization_overview.controller.go+++ b/backend/app/controllers/organization_overview.controller.go@@ -206,23 +206,23 @@ func loadProjectServerRows(ctx *gin.Context, project *models.Project, dashboardI}if len(latestOtel) > 0 {- cpuIdle, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, project.Id, models.MetricNameSystemCPUUtilization, since, now, 2, "avg", map[string]string{"state": "idle"}, "server_name")+ cpuIdle, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, project.Id, models.MetricNameSystemCPUUtilization, since, now, 2, "avg", map[string]string{"state": "idle"}, "server_name", 0)if err != nil {return nil, err}- memoryUsed, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, project.Id, models.MetricNameSystemMemoryUtilization, since, now, 2, "avg", map[string]string{"state": "used"}, "server_name")+ memoryUsed, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, project.Id, models.MetricNameSystemMemoryUtilization, since, now, 2, "avg", map[string]string{"state": "used"}, "server_name", 0)if err != nil {return nil, err}- filesystemUsed, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, project.Id, models.MetricNameSystemFilesystemUtilization, since, now, 2, "max", map[string]string{"state": "used"}, "server_name")+ filesystemUsed, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, project.Id, models.MetricNameSystemFilesystemUtilization, since, now, 2, "max", map[string]string{"state": "used"}, "server_name", 0)if err != nil {return nil, err}- networkReceive, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, project.Id, models.MetricNameSystemNetworkIO, since, now, 1, "sum", map[string]string{"direction": "receive"}, "server_name")+ networkReceive, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, project.Id, models.MetricNameSystemNetworkIO, since, now, 1, "sum", map[string]string{"direction": "receive"}, "server_name", 0)if err != nil {return nil, err}- networkTransmit, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, project.Id, models.MetricNameSystemNetworkIO, since, now, 1, "sum", map[string]string{"direction": "transmit"}, "server_name")+ networkTransmit, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, project.Id, models.MetricNameSystemNetworkIO, since, now, 1, "sum", map[string]string{"direction": "transmit"}, "server_name", 0)if err != nil {return nil, err}diff --git a/backend/app/controllers/otelcontrollers/codec.go b/backend/app/controllers/otelcontrollers/codec.goindex 5eb3d029..56de56cf 100644--- a/backend/app/controllers/otelcontrollers/codec.go+++ b/backend/app/controllers/otelcontrollers/codec.go@@ -2,12 +2,15 @@ package otelcontrollersimport ("compress/gzip"+ "errors""fmt""io""net/http""strings"+ "time""github.com/gin-gonic/gin"+ "github.com/tracewayapp/traceway/backend/app/middleware"collogspb "go.opentelemetry.io/proto/otlp/collector/logs/v1"colmetricspb "go.opentelemetry.io/proto/otlp/collector/metrics/v1"colprofilespb "go.opentelemetry.io/proto/otlp/collector/profiles/v1development"@@ -18,17 +21,58 @@ import (const maxBodySize = 10 * 1024 * 1024 // 10MB+var errBodyTooLarge = errors.New("request body exceeds the 10MB limit")++const (+ otlpBodyIdle = 20 * time.Second+ otlpBodyTotal = 2 * time.Minute+)+func readBody(c *gin.Context) ([]byte, error) {- var reader io.Reader = c.Request.Body+ middleware.GuardBodyRead(c, otlpBodyIdle, otlpBodyTotal)++ raw := http.MaxBytesReader(c.Writer, c.Request.Body, maxBodySize)++ var reader io.Reader = rawif strings.EqualFold(c.GetHeader("Content-Encoding"), "gzip") {- gr, err := gzip.NewReader(c.Request.Body)+ gr, err := gzip.NewReader(raw)if err != nil {+ if isMaxBytes(err) {+ return nil, errBodyTooLarge+ }return nil, fmt.Errorf("failed to create gzip reader: %w", err)}defer gr.Close()reader = gr}- return io.ReadAll(io.LimitReader(reader, maxBodySize))+ body, err := io.ReadAll(io.LimitReader(reader, maxBodySize+1))+ if err != nil {+ if isMaxBytes(err) {+ return nil, errBodyTooLarge+ }+ return nil, err+ }+ if len(body) > maxBodySize {+ return nil, errBodyTooLarge+ }+ return body, nil+}++func isMaxBytes(err error) bool {+ var maxBytesErr *http.MaxBytesError+ return errors.As(err, &maxBytesErr)+}++func writeDecodeError(c *gin.Context, err error) {+ if status, message, ok := middleware.BodyReadError(err); ok {+ c.JSON(status, gin.H{"error": message})+ return+ }+ status := http.StatusBadRequest+ if errors.Is(err, errBodyTooLarge) {+ status = http.StatusRequestEntityTooLarge+ }+ c.JSON(status, gin.H{"error": err.Error()})}func isProtobuf(c *gin.Context) bool {diff --git a/backend/app/controllers/otelcontrollers/codec_limits_test.go b/backend/app/controllers/otelcontrollers/codec_limits_test.gonew file mode 100644index 00000000..bc243142--- /dev/null+++ b/backend/app/controllers/otelcontrollers/codec_limits_test.go@@ -0,0 +1,82 @@+package otelcontrollers++import (+ "bytes"+ "compress/gzip"+ "errors"+ "io"+ "net/http"+ "net/http/httptest"+ "testing"++ "github.com/gin-gonic/gin"+)++type countingBody struct {+ r io.Reader+ n int64+}++func (c *countingBody) Read(p []byte) (int, error) {+ n, err := c.r.Read(p)+ c.n += int64(n)+ return n, err+}++func (c *countingBody) Close() error { return nil }++func gzipMembers(t *testing.T, minBytes int) []byte {+ t.Helper()+ var member bytes.Buffer+ zw := gzip.NewWriter(&member)+ if err := zw.Close(); err != nil {+ t.Fatal(err)+ }+ var buf bytes.Buffer+ for buf.Len() < minBytes {+ buf.Write(member.Bytes())+ }+ return buf.Bytes()+}++func readBodyWith(t *testing.T, encoding string, body []byte) ([]byte, error, int64) {+ t.Helper()+ gin.SetMode(gin.TestMode)+ c, _ := gin.CreateTestContext(httptest.NewRecorder())+ c.Request = httptest.NewRequest(http.MethodPost, "/api/otel/v1/traces", nil)+ counter := &countingBody{r: bytes.NewReader(body)}+ c.Request.Body = counter+ if encoding != "" {+ c.Request.Header.Set("Content-Encoding", encoding)+ }+ out, err := readBody(c)+ return out, err, counter.n+}++func TestReadBodyCapsRawGzipStream(t *testing.T) {+ body := gzipMembers(t, maxBodySize*4)++ out, err, consumed := readBodyWith(t, "gzip", body)++ if !errors.Is(err, errBodyTooLarge) {+ t.Fatalf("expected errBodyTooLarge, got err=%v out=%d bytes", err, len(out))+ }+ if consumed > maxBodySize+4096 {+ t.Fatalf("read %d raw bytes despite a %d-byte cap", consumed, maxBodySize)+ }+}++func TestReadBodyStillCapsDecompressedSize(t *testing.T) {+ var gz bytes.Buffer+ zw := gzip.NewWriter(&gz)+ if _, err := zw.Write(bytes.Repeat([]byte("A"), maxBodySize*3)); err != nil {+ t.Fatal(err)+ }+ if err := zw.Close(); err != nil {+ t.Fatal(err)+ }++ if _, err, _ := readBodyWith(t, "gzip", gz.Bytes()); !errors.Is(err, errBodyTooLarge) {+ t.Fatalf("expected errBodyTooLarge, got %v", err)+ }+}diff --git a/backend/app/controllers/otelcontrollers/otel.controller.go b/backend/app/controllers/otelcontrollers/otel.controller.goindex 8dd1edac..e552f730 100644--- a/backend/app/controllers/otelcontrollers/otel.controller.go+++ b/backend/app/controllers/otelcontrollers/otel.controller.go@@ -79,7 +79,7 @@ func (o otelController) ExportTraces(c *gin.Context) {}req, bodyBytes, err := decodeTraceRequest(c)if err != nil {- c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})+ writeDecodeError(c, err)return}@@ -224,7 +224,7 @@ func (o otelController) ExportMetrics(c *gin.Context) {req, bodyBytes, err := decodeMetricsRequest(c)if err != nil {- c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})+ writeDecodeError(c, err)return}@@ -296,7 +296,7 @@ func (o otelController) ExportLogs(c *gin.Context) {req, bodyBytes, err := decodeLogsRequest(c)if err != nil {- c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})+ writeDecodeError(c, err)return}@@ -358,7 +358,7 @@ func (o otelController) ExportProfiles(c *gin.Context) {payload, bodyBytes, err := decodeProfilesPayload(c)if err != nil {- c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})+ writeDecodeError(c, err)return}diff --git a/backend/app/controllers/page.controller.go b/backend/app/controllers/page.controller.goindex d7571416..b15ee31b 100644--- a/backend/app/controllers/page.controller.go+++ b/backend/app/controllers/page.controller.go@@ -2,6 +2,7 @@ package controllersimport ("database/sql"+ "fmt""net/http""strconv""time"@@ -240,6 +241,8 @@ type bulkAcknowledgeRequest struct {Ids []int `json:"ids"`}+const maxBulkPageIds = 100+// BulkAcknowledge acknowledges every selected page that is still open. Pages// from other projects or in another state are skipped, not errors: bulk// actions race the escalator and other responders by nature.@@ -256,6 +259,10 @@ func (c *pageController) BulkAcknowledge(ctx *gin.Context) {ctx.JSON(http.StatusBadRequest, gin.H{"error": "ids array is required"})return}+ if len(request.Ids) > maxBulkPageIds {+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": fmt.Sprintf("a maximum of %d pages can be acknowledged at once", maxBulkPageIds)})+ return+ }now := time.Now().UTC()acknowledged := 0@@ -303,6 +310,10 @@ func (c *pageController) BulkResolve(ctx *gin.Context) {ctx.JSON(http.StatusBadRequest, gin.H{"error": "ids array is required"})return}+ if len(request.Ids) > maxBulkPageIds {+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": fmt.Sprintf("a maximum of %d pages can be resolved at once", maxBulkPageIds)})+ return+ }now := time.Now().UTC()resolved := 0@@ -376,6 +387,10 @@ func (c *pageController) UnresolvedForIssues(ctx *gin.Context) {ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})return}+ if len(request.Hashes) > maxBulkIssueHashes {+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": fmt.Sprintf("a maximum of %d issues can be checked at once", maxBulkIssueHashes)})+ return+ }count := 0for _, hash := range request.Hashes {diff --git a/backend/app/controllers/pat.controller.go b/backend/app/controllers/pat.controller.goindex ebb200ba..fd12f155 100644--- a/backend/app/controllers/pat.controller.go+++ b/backend/app/controllers/pat.controller.go@@ -37,6 +37,10 @@ func (c *patController) Create(ctx *gin.Context) {ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Name must be 100 characters or fewer"})return}+ if req.ExpiresInDays != nil && (*req.ExpiresInDays < 1 || *req.ExpiresInDays > 3650) {+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Expiry must be between 1 and 3650 days"})+ return+ }token := authserver.NewPATToken()id := uuid.New().String()diff --git a/backend/app/controllers/routes.go b/backend/app/controllers/routes.goindex 3c45f42f..907166fe 100644--- a/backend/app/controllers/routes.go+++ b/backend/app/controllers/routes.go@@ -177,8 +177,8 @@ func RegisterControllers(router *gin.RouterGroup) {router.POST("/exception-stack-traces/:hash", middleware.UseAppAuth, middleware.RequireProjectAccess, ExceptionStackTraceController.FindByHash)// Keep the limiter before Transactional so rejected requests don't open a transaction.- router.POST("/login", middleware.RateLimitPerIP(10, time.Minute), middleware.Transactional, AuthController.Login)- router.POST("/register", middleware.RateLimitPerIP(10, time.Minute), middleware.Transactional, AuthController.Register)+ router.POST("/login", middleware.RateLimitPerIP(10, time.Minute), middleware.BufferAuthBody, middleware.Transactional, AuthController.Login)+ router.POST("/register", middleware.RateLimitPerIP(10, time.Minute), middleware.BufferAuthBody, middleware.Transactional, AuthController.Register)router.GET("/me/login-bundle", middleware.UseAppAuth, middleware.Transactional, AuthController.LoginBundle)router.GET("/auth/providers", OAuthController.ListProviders)@@ -186,18 +186,18 @@ func RegisterControllers(router *gin.RouterGroup) {router.GET("/auth/callback/:provider", middleware.RateLimitPerIP(20, time.Minute), middleware.Transactional, OAuthController.Callback)router.POST("/auth/finish-setup", middleware.UseAppAuth, middleware.Transactional, OAuthController.FinishSetup)- router.Match(postPreflight, "/auth/device/authorize", middleware.OAuthCors, middleware.RateLimitPerIP(10, time.Minute), DeviceAuthController.Authorize)+ router.Match(postPreflight, "/auth/device/authorize", middleware.OAuthCors, middleware.RateLimitPerIP(10, time.Minute), middleware.BufferAuthBody, DeviceAuthController.Authorize)// One shared limiter instance so both token routes draw from the same budget;// slightly higher 60/min limit to leave headroom for concurrent device flow polling behind NAT- tokenRateLimit := middleware.RateLimitPerIP(60, time.Minute)- router.Match(postPreflight, "/auth/device/token", middleware.OAuthCors, tokenRateLimit, DeviceAuthController.Token)- router.Match(postPreflight, "/auth/token", middleware.OAuthCors, tokenRateLimit, DeviceAuthController.Token)- router.Match(postPreflight, "/auth/logout", middleware.OAuthCors, middleware.RateLimitPerIP(20, time.Minute), DeviceAuthController.Logout)+ tokenRateLimit := middleware.RateLimitOAuthTokenPerIP(60, time.Minute)+ router.Match(postPreflight, "/auth/device/token", middleware.OAuthCors, tokenRateLimit, middleware.BufferAuthBody, DeviceAuthController.Token)+ router.Match(postPreflight, "/auth/token", middleware.OAuthCors, tokenRateLimit, middleware.BufferAuthBody, DeviceAuthController.Token)+ router.Match(postPreflight, "/auth/logout", middleware.OAuthCors, middleware.RateLimitPerIP(20, time.Minute), middleware.BufferAuthBody, DeviceAuthController.Logout)router.GET("/device", middleware.UseAppAuth, DeviceAuthController.Lookup)router.POST("/device/approve", middleware.UseAppAuth, middleware.Transactional, DeviceAuthController.Approve)router.POST("/device/deny", middleware.UseAppAuth, middleware.Transactional, DeviceAuthController.Deny)- router.Match(postPreflight, "/oauth/register", middleware.OAuthCors, middleware.RateLimitPerIP(10, time.Minute), middleware.Transactional, OauthAuthorizeController.Register)+ router.Match(postPreflight, "/oauth/register", middleware.OAuthCors, middleware.RateLimitPerIP(10, time.Minute), middleware.BufferAuthBody, middleware.Transactional, OauthAuthorizeController.Register)router.GET("/oauth/client", middleware.UseAppAuth, OauthAuthorizeController.Lookup)router.POST("/oauth/approve", middleware.UseAppAuth, middleware.Transactional, OauthAuthorizeController.Approve)router.POST("/oauth/deny", middleware.UseAppAuth, OauthAuthorizeController.Deny)@@ -223,9 +223,9 @@ func RegisterControllers(router *gin.RouterGroup) {}// slightly tighter limit since every accepted request sends an email- router.POST("/forgot-password", middleware.RateLimitPerIP(5, time.Minute), middleware.Transactional, PasswordResetController.ForgotPassword)+ router.POST("/forgot-password", middleware.RateLimitPerIP(5, time.Minute), middleware.BufferAuthBody, middleware.Transactional, PasswordResetController.ForgotPassword)router.GET("/password-reset/:token", middleware.RateLimitPerIP(30, time.Minute), PasswordResetController.ValidateToken)- router.POST("/password-reset/:token", middleware.RateLimitPerIP(10, time.Minute), middleware.Transactional, PasswordResetController.ResetPassword)+ router.POST("/password-reset/:token", middleware.RateLimitPerIP(10, time.Minute), middleware.BufferAuthBody, middleware.Transactional, PasswordResetController.ResetPassword)router.GET("/organizations/:organizationId/settings", middleware.UseAppAuth, middleware.RequireAdminAccess, OrganizationController.GetSettings)router.PUT("/organizations/:organizationId/settings", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, OrganizationController.UpdateSettings)@@ -241,12 +241,12 @@ func RegisterControllers(router *gin.RouterGroup) {router.DELETE("/organizations/:organizationId/invitations/:id", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, InvitationController.RevokeInvitation)router.GET("/invitations/:token", middleware.RateLimitPerIP(30, time.Minute), InvitationController.GetInvitationInfo)- router.POST("/invitations/:token/accept", middleware.RateLimitPerIP(10, time.Minute), middleware.Transactional, InvitationController.AcceptInvitation)+ router.POST("/invitations/:token/accept", middleware.RateLimitPerIP(10, time.Minute), middleware.BufferAuthBody, middleware.Transactional, InvitationController.AcceptInvitation)router.POST("/invitations/:token/accept-existing", middleware.UseAppAuth, middleware.Transactional, InvitationController.AcceptExistingUser)router.POST("/projects/source-map-token", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.RequireWriteAccess, ProjectController.GenerateSourceMapToken)- router.POST("/sourcemaps/upload", middleware.UseSourceMapAuth, SourceMapController.Upload)- router.POST("/symbols/upload", middleware.UseSourceMapAuth, SymbolsController.Upload)+ router.POST("/sourcemaps/upload", middleware.UseSourceMapAuth, middleware.UploadAdmission, SourceMapController.Upload)+ router.POST("/symbols/upload", middleware.UseSourceMapAuth, middleware.UploadAdmission, SymbolsController.Upload)router.GET("/notification-channels", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, NotificationChannelController.List)router.POST("/notification-channels", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.RequireWriteAccess, middleware.Transactional, NotificationChannelController.Create)diff --git a/backend/app/controllers/sourcemap.controller.go b/backend/app/controllers/sourcemap.controller.goindex c8f19afd..955b106e 100644--- a/backend/app/controllers/sourcemap.controller.go+++ b/backend/app/controllers/sourcemap.controller.go@@ -24,10 +24,10 @@ func (s sourceMapController) Upload(c *gin.Context) {return}- if err := c.Request.ParseMultipartForm(50 << 20); err != nil {- c.JSON(http.StatusBadRequest, gin.H{"error": "Failed to parse multipart form"})+ if !parseMultipartCapped(c, maxUploadBytes, uploadMultipartMemory) {return}+ defer c.Request.MultipartForm.RemoveAll()files := c.Request.MultipartForm.File["files"]if len(files) == 0 {diff --git a/backend/app/controllers/symbols.controller.go b/backend/app/controllers/symbols.controller.goindex 4b03be72..d1f58306 100644--- a/backend/app/controllers/symbols.controller.go+++ b/backend/app/controllers/symbols.controller.go@@ -28,10 +28,10 @@ func (s symbolsController) Upload(c *gin.Context) {return}- if err := c.Request.ParseMultipartForm(200 << 20); err != nil {- c.JSON(http.StatusBadRequest, gin.H{"error": "Failed to parse multipart form"})+ if !parseMultipartCapped(c, maxUploadBytes, uploadMultipartMemory) {return}+ defer c.Request.MultipartForm.RemoveAll()files := c.Request.MultipartForm.File["files"]if len(files) == 0 {diff --git a/backend/app/middleware/admission_test.go b/backend/app/middleware/admission_test.gonew file mode 100644index 00000000..612f65f6--- /dev/null+++ b/backend/app/middleware/admission_test.go@@ -0,0 +1,90 @@+package middleware++import (+ "net/http"+ "net/http/httptest"+ "sync"+ "testing"+ "time"++ "github.com/gin-gonic/gin"+)++func admissionRouter(gate gin.HandlerFunc, release chan struct{}) *gin.Engine {+ gin.SetMode(gin.TestMode)+ r := gin.New()+ r.POST("/x", gate, func(c *gin.Context) {+ if release != nil {+ <-release+ }+ c.Status(http.StatusOK)+ })+ return r+}++func post(r *gin.Engine) int {+ rec := httptest.NewRecorder()+ r.ServeHTTP(rec, httptest.NewRequest(http.MethodPost, "/x", nil))+ return rec.Code+}++func TestAdmissionGateAdmitsUpToCapacity(t *testing.T) {+ release := make(chan struct{})+ r := admissionRouter(newAdmissionGate(2, 10*time.Millisecond, "busy", nil), release)++ var wg sync.WaitGroup+ codes := make([]int, 3)+ for i := range codes {+ wg.Add(1)+ go func(i int) {+ defer wg.Done()+ codes[i] = post(r)+ }(i)+ }+ time.Sleep(150 * time.Millisecond)+ close(release)+ wg.Wait()++ ok, rejected := 0, 0+ for _, c := range codes {+ switch c {+ case http.StatusOK:+ ok+++ case http.StatusServiceUnavailable:+ rejected+++ }+ }+ if ok != 2 || rejected != 1 {+ t.Fatalf("capacity 2 admitted %d and rejected %d, want 2 and 1 (codes %v)", ok, rejected, codes)+ }+}++func TestAdmissionGateBoundsWaiters(t *testing.T) {+ release := make(chan struct{})+ defer close(release)+ r := admissionRouter(newAdmissionGate(1, time.Second, "busy", nil), release)++ codes := make(chan int, 32)+ for i := 0; i < 1+16+3; i++ {+ go func() { codes <- post(r) }()+ }++ immediate := 0+ timeout := time.After(300 * time.Millisecond)+ for immediate < 3 {+ select {+ case code := <-codes:+ if code != http.StatusServiceUnavailable {+ t.Fatalf("got %d before any slot was released, want 503", code)+ }+ immediate+++ case <-timeout:+ t.Fatalf("only %d requests were rejected immediately, want 3", immediate)+ }+ }+ select {+ case code := <-codes:+ t.Fatalf("a request answered %d before the wait window ended", code)+ case <-time.After(100 * time.Millisecond):+ }+}diff --git a/backend/app/middleware/body_error.go b/backend/app/middleware/body_error.gonew file mode 100644index 00000000..07e834a7--- /dev/null+++ b/backend/app/middleware/body_error.go@@ -0,0 +1,28 @@+package middleware++import (+ "errors"+ "net/http"+ "os"++ "github.com/gin-gonic/gin"+)++func BodyReadError(err error) (status int, message string, ok bool) {+ var maxBytesErr *http.MaxBytesError+ switch {+ case errors.As(err, &maxBytesErr):+ return http.StatusRequestEntityTooLarge, "Request body too large", true+ case errors.Is(err, ErrBodyTooSlow), errors.Is(err, os.ErrDeadlineExceeded):+ return http.StatusRequestTimeout, "Request body arrived too slowly", true+ }+ return 0, "", false+}++func RejectBindError(c *gin.Context, err error, fallback string) {+ status, message, ok := BodyReadError(err)+ if !ok {+ status, message = http.StatusBadRequest, fallback+ }+ c.JSON(status, gin.H{"error": message})+}diff --git a/backend/app/middleware/buffer_body.go b/backend/app/middleware/buffer_body.gonew file mode 100644index 00000000..4908c2dd--- /dev/null+++ b/backend/app/middleware/buffer_body.go@@ -0,0 +1,45 @@+package middleware++import (+ "bytes"+ "io"+ "net/http"++ "github.com/gin-gonic/gin"+)++const maxAuthBodyBytes = 64 << 10++const maxTransactionalBodyBytes = 8 << 20++const bodyBufferedContextKey = "bodyBuffered"++func BufferAuthBody(c *gin.Context) {+ if bufferRequestBody(c, maxAuthBodyBytes) {+ c.Next()+ }+}++func bufferRequestBody(c *gin.Context, maxBytes int64) bool {+ if c.Request == nil || c.Request.Body == nil || c.Request.Body == http.NoBody {+ return true+ }+ if c.GetBool(bodyBufferedContextKey) {+ return true+ }++ data, err := io.ReadAll(http.MaxBytesReader(c.Writer, c.Request.Body, maxBytes))+ if err != nil {+ status, message, ok := BodyReadError(err)+ if !ok {+ status, message = http.StatusBadRequest, "Could not read request body"+ }+ c.AbortWithStatusJSON(status, gin.H{"error": message})+ return false+ }++ c.Request.Body = io.NopCloser(bytes.NewReader(data))+ c.Request.ContentLength = int64(len(data))+ c.Set(bodyBufferedContextKey, true)+ return true+}diff --git a/backend/app/middleware/buffer_body_test.go b/backend/app/middleware/buffer_body_test.gonew file mode 100644index 00000000..a00250a4--- /dev/null+++ b/backend/app/middleware/buffer_body_test.go@@ -0,0 +1,97 @@+//go:build !transactional_pg && !telemetry_ch && !telemetry_duckdb++package middleware++import (+ "database/sql"+ "fmt"+ "net"+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"+ "time"++ "github.com/gin-gonic/gin"+ "github.com/tracewayapp/traceway/backend/app/db"+ _ "modernc.org/sqlite"+)++func TestTransactionalBuffersBodyBeforeOpeningTransaction(t *testing.T) {+ gin.SetMode(gin.TestMode)++ conn, err := sql.Open("sqlite", ":memory:")+ if err != nil {+ t.Fatal(err)+ }+ conn.SetMaxOpenConns(1)+ defer conn.Close()++ prev := db.DB+ db.DB = conn+ t.Cleanup(func() { db.DB = prev })++ r := gin.New()+ r.POST("/personal-access-tokens", Transactional, func(c *gin.Context) {+ var body map[string]any+ _ = c.ShouldBindJSON(&body)+ c.JSON(http.StatusOK, gin.H{"ok": true})+ })+ r.GET("/other", Transactional, func(c *gin.Context) { c.JSON(http.StatusOK, gin.H{"ok": true}) })++ ln, err := net.Listen("tcp", "127.0.0.1:0")+ if err != nil {+ t.Fatal(err)+ }+ srv := &http.Server{Handler: r, ReadTimeout: 3 * time.Second}+ go srv.Serve(ln)+ defer srv.Close()+ addr := ln.Addr().String()++ slow, err := net.Dial("tcp", addr)+ if err != nil {+ t.Fatal(err)+ }+ defer slow.Close()+ fmt.Fprint(slow, "POST /personal-access-tokens HTTP/1.1\r\nHost: x\r\nContent-Type: application/json\r\nContent-Length: 2000\r\n\r\n")+ fmt.Fprint(slow, `{"name":"`)+ go func() {+ for i := 0; i < 1000; i++ {+ if _, err := slow.Write([]byte("a")); err != nil {+ return+ }+ time.Sleep(50 * time.Millisecond)+ }+ }()+ time.Sleep(300 * time.Millisecond)++ start := time.Now()+ resp, err := (&http.Client{Timeout: 5 * time.Second}).Get("http://" + addr + "/other")+ elapsed := time.Since(start)+ if err != nil {+ t.Fatalf("unrelated request failed after %v: %v", elapsed, err)+ }+ resp.Body.Close()++ if elapsed > time.Second {+ t.Fatalf("slow body held the single main-DB connection for %v", elapsed)+ }+}++func TestTransactionalRejectsOversizedBodyWithoutATransaction(t *testing.T) {+ gin.SetMode(gin.TestMode)++ prev := db.DB+ db.DB = nil+ t.Cleanup(func() { db.DB = prev })++ r := gin.New()+ r.POST("/x", Transactional, func(c *gin.Context) { c.Status(http.StatusOK) })++ body := strings.NewReader(strings.Repeat("a", maxTransactionalBodyBytes+1))+ rec := httptest.NewRecorder()+ r.ServeHTTP(rec, httptest.NewRequest(http.MethodPost, "/x", body))+ if rec.Code != http.StatusRequestEntityTooLarge {+ t.Fatalf("got %d, want 413", rec.Code)+ }+}diff --git a/backend/app/middleware/ingest_admission.go b/backend/app/middleware/ingest_admission.goindex f9b9cb09..83a16dd4 100644--- a/backend/app/middleware/ingest_admission.go+++ b/backend/app/middleware/ingest_admission.go@@ -22,21 +22,41 @@ import (// a connection; a request that cannot get a slot within the wait window is// rejected with 503 + Retry-After, which SDKs and OTLP collectors retry.func newIngestAdmission(capacity int, wait time.Duration) gin.HandlerFunc {+ return newAdmissionGate(capacity, wait, "ingest saturated, retry later", db.RecordIngestRejected)+}++func newAdmissionGate(capacity int, wait time.Duration, message string, onReject func()) gin.HandlerFunc {slots := make(chan struct{}, capacity)+ waiters := make(chan struct{}, max(4*capacity, 16))+ reject := func(c *gin.Context) {+ if onReject != nil {+ onReject()+ }+ c.Header("Retry-After", "2")+ c.AbortWithStatusJSON(http.StatusServiceUnavailable, gin.H{"error": message})+ }return func(c *gin.Context) {select {case slots <- struct{}{}:default:+ select {+ case waiters <- struct{}{}:+ default:+ reject(c)+ return+ }timer := time.NewTimer(wait)- defer timer.Stop()select {case slots <- struct{}{}:+ <-waiters+ timer.Stop()case <-timer.C:- db.RecordIngestRejected()- c.Header("Retry-After", "2")- c.AbortWithStatusJSON(http.StatusServiceUnavailable, gin.H{"error": "ingest saturated, retry later"})+ <-waiters+ reject(c)returncase <-c.Request.Context().Done():+ <-waiters+ timer.Stop()c.Abort()return}@@ -78,3 +98,21 @@ var ingestAdmission = sync.OnceValue(func() gin.HandlerFunc {func IngestAdmission(c *gin.Context) {ingestAdmission()(c)}++var uploadAdmission = sync.OnceValue(func() gin.HandlerFunc {+ return newAdmissionGate(uploadCapacityFromEnv(), 30*time.Second, "upload queue saturated, retry later", nil)+})++func uploadCapacityFromEnv() int {+ n := 4+ if v := os.Getenv("UPLOAD_MAX_CONCURRENT"); v != "" {+ if parsed, err := strconv.Atoi(v); err == nil && parsed > 0 {+ n = parsed+ }+ }+ return n+}++func UploadAdmission(c *gin.Context) {+ uploadAdmission()(c)+}diff --git a/backend/app/middleware/rate_limit.middleware.go b/backend/app/middleware/rate_limit.middleware.goindex 463b5647..7c8f7a2f 100644--- a/backend/app/middleware/rate_limit.middleware.go+++ b/backend/app/middleware/rate_limit.middleware.go@@ -1,8 +1,14 @@package middlewareimport (+ "bytes"+ "encoding/json"+ "io"+ "math""net/http"+ "net/url""strconv"+ "strings""sync""time"@@ -36,6 +42,11 @@ func NewFixedWindowLimiter(maxRequests int, window time.Duration) *FixedWindowLi// Allow consumes one slot for key and reports whether the request is within// the limit.func (l *FixedWindowLimiter) Allow(key string) bool {+ allowed, _ := l.Take(key)+ return allowed+}++func (l *FixedWindowLimiter) Take(key string) (bool, time.Duration) {now := time.Now()l.mu.Lock()@@ -51,20 +62,79 @@ func (l *FixedWindowLimiter) Allow(key string) bool {l.buckets[key] = b}b.count++- return b.count <= l.maxRequests+ if b.count <= l.maxRequests {+ return true, 0+ }+ return false, b.windowStart.Add(l.window).Sub(now)+}++func rejectRateLimited(c *gin.Context, retryAfter time.Duration) {+ seconds := int(math.Ceil(retryAfter.Seconds()))+ if seconds < 1 {+ seconds = 1+ }+ c.Header("Retry-After", strconv.Itoa(seconds))+ c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{"error": "slow_down"})}func rateLimitWithKey(maxRequests int, window time.Duration, keyOf func(c *gin.Context) string) gin.HandlerFunc {limiter := NewFixedWindowLimiter(maxRequests, window)return func(c *gin.Context) {- if !limiter.Allow(keyOf(c)) {- c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{"error": "slow_down"})+ allowed, retryAfter := limiter.Take(keyOf(c))+ if !allowed {+ rejectRateLimited(c, retryAfter)return}c.Next()}}+func RateLimitOAuthTokenPerIP(maxRequests int, window time.Duration) gin.HandlerFunc {+ limiter := NewFixedWindowLimiter(maxRequests, window)+ return func(c *gin.Context) {+ allowed, retryAfter := limiter.Take(c.ClientIP())+ if allowed {+ c.Next()+ return+ }+ grant, ok := oauthGrantType(c)+ if !ok {+ return+ }+ if grant == "device_code" || grant == "urn:ietf:params:oauth:grant-type:device_code" {+ c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": "slow_down"})+ return+ }+ rejectRateLimited(c, retryAfter)+ }+}++func oauthGrantType(c *gin.Context) (string, bool) {+ if !bufferRequestBody(c, maxAuthBodyBytes) {+ return "", false+ }+ if c.Request.Body == nil || c.Request.Body == http.NoBody {+ return "", true+ }+ data, err := io.ReadAll(c.Request.Body)+ c.Request.Body = io.NopCloser(bytes.NewReader(data))+ if err != nil {+ return "", true+ }+ if strings.HasPrefix(c.ContentType(), "application/json") {+ var body struct {+ GrantType string `json:"grant_type"`+ }+ _ = json.Unmarshal(data, &body)+ return body.GrantType, true+ }+ values, err := url.ParseQuery(string(data))+ if err != nil {+ return "", true+ }+ return values.Get("grant_type"), true+}+// RateLimitPerIP returns a fixed-window per-IP rate limiter for unauthenticated// endpoints that persist state per request (e.g. the device-authorize endpoint,// which inserts a main-DB row per call).diff --git a/backend/app/middleware/rate_limit_test.go b/backend/app/middleware/rate_limit_test.gonew file mode 100644index 00000000..37d9a917--- /dev/null+++ b/backend/app/middleware/rate_limit_test.go@@ -0,0 +1,92 @@+package middleware++import (+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"+ "time"++ "github.com/gin-gonic/gin"+)++func tokenRouter(limit int) *gin.Engine {+ gin.SetMode(gin.TestMode)+ r := gin.New()+ r.POST("/token", RateLimitOAuthTokenPerIP(limit, time.Minute), func(c *gin.Context) { c.Status(http.StatusOK) })+ return r+}++func postToken(r *gin.Engine, contentType, body string) *httptest.ResponseRecorder {+ rec := httptest.NewRecorder()+ req := httptest.NewRequest(http.MethodPost, "/token", strings.NewReader(body))+ req.RemoteAddr = "203.0.113.5:1234"+ if contentType != "" {+ req.Header.Set("Content-Type", contentType)+ }+ r.ServeHTTP(rec, req)+ return rec+}++func TestOAuthTokenRateLimitAnswersDevicePollsInTheRFCShape(t *testing.T) {+ for _, tc := range []struct{ name, contentType, body string }{+ {"json short grant", "application/json", `{"grant_type":"device_code","device_code":"x"}`},+ {"json urn grant", "application/json", `{"grant_type":"urn:ietf:params:oauth:grant-type:device_code","device_code":"x"}`},+ {"form urn grant", "application/x-www-form-urlencoded", "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code&device_code=x"},+ } {+ t.Run(tc.name, func(t *testing.T) {+ r := tokenRouter(1)+ if rec := postToken(r, tc.contentType, tc.body); rec.Code != http.StatusOK {+ t.Fatalf("first request got %d, want 200", rec.Code)+ }+ rec := postToken(r, tc.contentType, tc.body)+ if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), `"slow_down"`) {+ t.Fatalf("over the limit got %d %s, want 400 with error slow_down", rec.Code, rec.Body.String())+ }+ })+ }+}++func TestOAuthTokenRateLimitAnswersOtherGrantsWith429(t *testing.T) {+ for _, tc := range []struct{ name, contentType, body string }{+ {"json refresh", "application/json", `{"grant_type":"refresh_token","refresh_token":"x"}`},+ {"form refresh", "application/x-www-form-urlencoded", "grant_type=refresh_token&refresh_token=x"},+ {"form code exchange", "application/x-www-form-urlencoded", "grant_type=authorization_code&code=x"},+ {"no body", "", ""},+ } {+ t.Run(tc.name, func(t *testing.T) {+ r := tokenRouter(1)+ if rec := postToken(r, tc.contentType, tc.body); rec.Code != http.StatusOK {+ t.Fatalf("first request got %d, want 200", rec.Code)+ }+ rec := postToken(r, tc.contentType, tc.body)+ if rec.Code != http.StatusTooManyRequests {+ t.Fatalf("over the limit got %d %s, want 429", rec.Code, rec.Body.String())+ }+ if rec.Header().Get("Retry-After") == "" {+ t.Fatal("429 must carry Retry-After")+ }+ })+ }+}++func TestRateLimitPerIPCarriesRetryAfter(t *testing.T) {+ gin.SetMode(gin.TestMode)+ r := gin.New()+ r.POST("/login", RateLimitPerIP(1, time.Minute), func(c *gin.Context) { c.Status(http.StatusOK) })+ post := func() *httptest.ResponseRecorder {+ rec := httptest.NewRecorder()+ req := httptest.NewRequest(http.MethodPost, "/login", nil)+ req.RemoteAddr = "203.0.113.5:1234"+ r.ServeHTTP(rec, req)+ return rec+ }+ post()+ rec := post()+ if rec.Code != http.StatusTooManyRequests {+ t.Fatalf("got %d, want 429", rec.Code)+ }+ if got := rec.Header().Get("Retry-After"); got == "" || got == "0" {+ t.Fatalf("Retry-After = %q, want the seconds until the window resets", got)+ }+}diff --git a/backend/app/middleware/read_deadline.go b/backend/app/middleware/read_deadline.gonew file mode 100644index 00000000..490850fc--- /dev/null+++ b/backend/app/middleware/read_deadline.go@@ -0,0 +1,111 @@+package middleware++import (+ "errors"+ "io"+ "net/http"+ "time"++ "github.com/gin-gonic/gin"+)++const minThroughputBytesPerSec = 1 << 10++var throughputGracePeriod = 20 * time.Second++var ErrBodyTooSlow = errors.New("request body delivered too slowly")++const (+ DefaultBodyIdle = 30 * time.Second+ DefaultBodyTotal = 10 * time.Minute+)++const bodyGuardContextKey = "bodyGuard"++type progressBody struct {+ rc io.ReadCloser+ ctl *http.ResponseController+ declared int64+ idle time.Duration+ deadline time.Time+ lastSet time.Time+ start time.Time+ read int64+ done bool+}++func (p *progressBody) Read(b []byte) (int, error) {+ n, err := p.rc.Read(b)+ if p.done {+ return n, err+ }+ p.read += int64(n)+ if err == io.EOF || (p.declared >= 0 && p.read >= p.declared) {+ p.done = true+ _ = p.ctl.SetReadDeadline(time.Time{})+ return n, err+ }+ if n == 0 {+ return n, err+ }+ now := time.Now()+ if elapsed := now.Sub(p.start); elapsed > throughputGracePeriod && float64(p.read)/elapsed.Seconds() < minThroughputBytesPerSec {+ return n, ErrBodyTooSlow+ }+ if now.Sub(p.lastSet) >= p.idle/4 {+ p.arm(now)+ }+ return n, err+}++func (p *progressBody) arm(now time.Time) {+ next := now.Add(p.idle)+ if next.After(p.deadline) {+ next = p.deadline+ }+ _ = p.ctl.SetReadDeadline(next)+ p.lastSet = now+}++func (p *progressBody) Close() error { return p.rc.Close() }++func GuardBodyRead(c *gin.Context, idle, total time.Duration) {+ if c.Request == nil || c.Request.Body == nil || c.Request.Body == http.NoBody {+ return+ }+ now := time.Now()+ if existing, ok := c.Get(bodyGuardContextKey); ok {+ if p, ok := existing.(*progressBody); ok {+ if p.done {+ return+ }+ p.idle = idle+ p.deadline = now.Add(total)+ p.arm(now)+ return+ }+ }++ ctl := http.NewResponseController(c.Writer)+ p := &progressBody{+ rc: c.Request.Body,+ ctl: ctl,+ declared: c.Request.ContentLength,+ idle: idle,+ deadline: now.Add(total),+ start: now,+ }+ if err := ctl.SetReadDeadline(now.Add(min(idle, total))); err != nil {+ return+ }+ p.lastSet = now+ c.Request.Body = p+ c.Set(bodyGuardContextKey, p)+}++func GuardBodyReads(idle, total time.Duration) gin.HandlerFunc {+ return func(c *gin.Context) {+ GuardBodyRead(c, idle, total)+ c.Next()+ }+}diff --git a/backend/app/middleware/security_headers.middleware.go b/backend/app/middleware/security_headers.middleware.gonew file mode 100644index 00000000..925fd311--- /dev/null+++ b/backend/app/middleware/security_headers.middleware.go@@ -0,0 +1,20 @@+package middleware++import (+ "strings"++ "github.com/gin-gonic/gin"+)++func SecurityHeaders(c *gin.Context) {+ path := c.Request.URL.Path+ if strings.HasPrefix(path, "/status/") || strings.HasPrefix(path, "/api/status/") {+ c.Header("Content-Security-Policy", "base-uri 'self'; form-action 'self'")+ } else {+ c.Header("X-Frame-Options", "DENY")+ c.Header("Content-Security-Policy", "frame-ancestors 'none'; base-uri 'self'; form-action 'self'")+ }+ c.Header("Referrer-Policy", "strict-origin-when-cross-origin")+ c.Header("X-Content-Type-Options", "nosniff")+ c.Next()+}diff --git a/backend/app/middleware/security_headers_test.go b/backend/app/middleware/security_headers_test.gonew file mode 100644index 00000000..6a740188--- /dev/null+++ b/backend/app/middleware/security_headers_test.go@@ -0,0 +1,50 @@+package middleware++import (+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"++ "github.com/gin-gonic/gin"+)++func TestSecurityHeadersExemptPublicStatusPagesFromFraming(t *testing.T) {+ gin.SetMode(gin.TestMode)+ r := gin.New()+ r.Use(SecurityHeaders)+ ok := func(c *gin.Context) { c.Status(http.StatusOK) }+ r.GET("/api/projects", ok)+ r.GET("/api/status/:slug", ok)+ r.GET("/api/status/:slug/logo", ok)+ r.NoRoute(ok)++ cases := []struct {+ path string+ embeddable bool+ }{+ {"/api/projects", false},+ {"/", false},+ {"/issues", false},+ {"/status/api", true},+ {"/api/status/api", true},+ {"/api/status/api/logo", true},+ }+ for _, tc := range cases {+ rec := httptest.NewRecorder()+ r.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, tc.path, nil))+ h := rec.Header()+ csp := h.Get("Content-Security-Policy")+ blocked := h.Get("X-Frame-Options") == "DENY" && strings.Contains(csp, "frame-ancestors 'none'")+ allowed := h.Get("X-Frame-Options") == "" && !strings.Contains(csp, "frame-ancestors")+ if tc.embeddable && !allowed {+ t.Errorf("%s: must be embeddable, got X-Frame-Options=%q CSP=%q", tc.path, h.Get("X-Frame-Options"), csp)+ }+ if !tc.embeddable && !blocked {+ t.Errorf("%s: framing must be blocked, got X-Frame-Options=%q CSP=%q", tc.path, h.Get("X-Frame-Options"), csp)+ }+ if h.Get("X-Content-Type-Options") != "nosniff" || !strings.Contains(csp, "base-uri 'self'") {+ t.Errorf("%s: the rest of the policy must survive", tc.path)+ }+ }+}diff --git a/backend/app/middleware/throughput_test.go b/backend/app/middleware/throughput_test.gonew file mode 100644index 00000000..466afb27--- /dev/null+++ b/backend/app/middleware/throughput_test.go@@ -0,0 +1,177 @@+package middleware++import (+ "errors"+ "fmt"+ "io"+ "net"+ "net/http"+ "testing"+ "time"++ "github.com/gin-gonic/gin"+)++func guardedServer(t *testing.T, idle, total time.Duration) (string, chan error) {+ t.Helper()+ gin.SetMode(gin.TestMode)++ prev := throughputGracePeriod+ throughputGracePeriod = 500 * time.Millisecond+ t.Cleanup(func() { throughputGracePeriod = prev })++ readErrs := make(chan error, 8)+ r := gin.New()+ r.POST("/ingest", func(c *gin.Context) {+ GuardBodyRead(c, idle, total)+ _, err := io.Copy(io.Discard, c.Request.Body)+ readErrs <- err+ c.Status(http.StatusOK)+ })++ ln, err := net.Listen("tcp", "127.0.0.1:0")+ if err != nil {+ t.Fatal(err)+ }+ srv := &http.Server{Handler: r, ReadHeaderTimeout: 2 * time.Second, ReadTimeout: 3 * time.Second}+ go srv.Serve(ln)+ t.Cleanup(func() { srv.Close() })+ return ln.Addr().String(), readErrs+}++func TestGuardBodyReadRejectsADripClient(t *testing.T) {+ addr, readErrs := guardedServer(t, 2*time.Second, time.Minute)++ conn, err := net.Dial("tcp", addr)+ if err != nil {+ t.Fatal(err)+ }+ defer conn.Close()+ fmt.Fprint(conn, "POST /ingest HTTP/1.1\r\nHost: x\r\nContent-Length: 10000000\r\n\r\n")++ start := time.Now()+ go func() {+ for i := 0; i < 300; i++ {+ if _, err := conn.Write([]byte("a")); err != nil {+ return+ }+ time.Sleep(100 * time.Millisecond)+ }+ }()++ select {+ case err := <-readErrs:+ if !errors.Is(err, ErrBodyTooSlow) {+ t.Fatalf("read ended with %v, want ErrBodyTooSlow", err)+ }+ if elapsed := time.Since(start); elapsed > throughputGracePeriod+3*time.Second {+ t.Fatalf("drip client held the slot for %v", elapsed)+ }+ case <-time.After(throughputGracePeriod + 5*time.Second):+ t.Fatal("drip client was never cut off")+ }+}++func TestGuardBodyReadAcceptsASlowButRealClient(t *testing.T) {+ addr, readErrs := guardedServer(t, 3*time.Second, time.Minute)++ const total = 64 << 10+ conn, err := net.Dial("tcp", addr)+ if err != nil {+ t.Fatal(err)+ }+ defer conn.Close()+ fmt.Fprintf(conn, "POST /ingest HTTP/1.1\r\nHost: x\r\nContent-Length: %d\r\n\r\n", total)++ chunk := make([]byte, 8<<10)+ go func() {+ for sent := 0; sent < total; sent += len(chunk) {+ if _, err := conn.Write(chunk); err != nil {+ return+ }+ time.Sleep(100 * time.Millisecond)+ }+ }()++ select {+ case err := <-readErrs:+ if err != nil {+ t.Fatalf("a legitimate 80KB/s client was rejected: %v", err)+ }+ case <-time.After(10 * time.Second):+ t.Fatal("timed out")+ }+}++func TestGuardBodyReadAcceptsACompleteBodyUnderTheFloor(t *testing.T) {+ addr, readErrs := guardedServer(t, 3*time.Second, time.Minute)++ const total = 1 << 10+ conn, err := net.Dial("tcp", addr)+ if err != nil {+ t.Fatal(err)+ }+ defer conn.Close()+ fmt.Fprintf(conn, "POST /ingest HTTP/1.1\r\nHost: x\r\nContent-Length: %d\r\n\r\n", total)++ go func() {+ half := make([]byte, total/2)+ if _, err := conn.Write(half); err != nil {+ return+ }+ time.Sleep(1200 * time.Millisecond)+ conn.Write(half)+ }()++ select {+ case err := <-readErrs:+ if err != nil {+ t.Fatalf("a complete slow body was rejected: %v", err)+ }+ case <-time.After(10 * time.Second):+ t.Fatal("timed out")+ }+}++func TestGuardBodyReadClearsTheDeadlineOnceTheBodyIsComplete(t *testing.T) {+ gin.SetMode(gin.TestMode)+ ctxErrs := make(chan error, 1)+ r := gin.New()+ r.POST("/slow-handler", func(c *gin.Context) {+ GuardBodyRead(c, 300*time.Millisecond, time.Minute)+ if _, err := io.Copy(io.Discard, c.Request.Body); err != nil {+ ctxErrs <- err+ return+ }+ select {+ case <-c.Request.Context().Done():+ ctxErrs <- c.Request.Context().Err()+ case <-time.After(time.Second):+ ctxErrs <- nil+ }+ c.Status(http.StatusOK)+ })+ ln, err := net.Listen("tcp", "127.0.0.1:0")+ if err != nil {+ t.Fatal(err)+ }+ srv := &http.Server{Handler: r}+ go srv.Serve(ln)+ t.Cleanup(func() { srv.Close() })++ conn, err := net.Dial("tcp", ln.Addr().String())+ if err != nil {+ t.Fatal(err)+ }+ defer conn.Close()+ const total = 2 << 10+ fmt.Fprintf(conn, "POST /slow-handler HTTP/1.1\r\nHost: x\r\nContent-Length: %d\r\n\r\n", total)+ half := make([]byte, total/2)+ conn.Write(half)+ time.Sleep(150 * time.Millisecond)+ conn.Write(half)++ if err := <-ctxErrs; err != nil {+ t.Fatalf("a handler outliving the idle window after the body arrived lost its context: %v", err)+ }+}diff --git a/backend/app/middleware/transactional.middleware.go b/backend/app/middleware/transactional.middleware.goindex 87707a24..dd40a849 100644--- a/backend/app/middleware/transactional.middleware.go+++ b/backend/app/middleware/transactional.middleware.go@@ -10,6 +10,10 @@ import ()func Transactional(c *gin.Context) {+ if !bufferRequestBody(c, maxTransactionalBodyBytes) {+ return+ }+txHandle, err := db.DB.Begin()if err != nil {diff --git a/backend/app/middleware/use_gzip_decompress.middleware.go b/backend/app/middleware/use_gzip_decompress.middleware.goindex 47c73bde..bfe0a08e 100644--- a/backend/app/middleware/use_gzip_decompress.middleware.go+++ b/backend/app/middleware/use_gzip_decompress.middleware.go@@ -3,26 +3,52 @@ package middlewareimport ("compress/gzip""net/http"+ "sync"+ "time""github.com/gin-gonic/gin"+ "github.com/tracewayapp/traceway/backend/app/config")+const defaultReportMaxBodyMB = 64++var reportMaxBodyBytes = sync.OnceValue(func() int64 {+ value := ""+ if config.Config != nil {+ value = config.Config.ReportMaxBodyMB+ }+ return config.SizeMB(value, defaultReportMaxBodyMB)+})+func UseGzip(c *gin.Context) {+ useGzipLimited(c, reportMaxBodyBytes())+}++const (+ reportBodyIdle = 20 * time.Second+ reportBodyTotal = 5 * time.Minute+)++func useGzipLimited(c *gin.Context, maxBytes int64) {+ GuardBodyRead(c, reportBodyIdle, reportBodyTotal)+// Decompress when Content-Encoding announces gzip; otherwise pass the// body through untouched. The pagehide / keepalive code path in the SDK// has to dispatch the request synchronously inside the unload handler,// which means it can't `await` the async CompressionStream — those// requests arrive as plain JSON and we accept them as-is.+ body := http.MaxBytesReader(c.Writer, c.Request.Body, maxBytes)if c.GetHeader("Content-Encoding") != "gzip" {+ c.Request.Body = bodyc.Next()return}- gzReader, err := gzip.NewReader(c.Request.Body)+ gzReader, err := gzip.NewReader(body)if err != nil {c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": "Invalid gzip"})return}- c.Request.Body = gzReader+ c.Request.Body = http.MaxBytesReader(c.Writer, gzReader, maxBytes)c.Next()}diff --git a/backend/app/middleware/use_gzip_decompress_test.go b/backend/app/middleware/use_gzip_decompress_test.gonew file mode 100644index 00000000..5c37812f--- /dev/null+++ b/backend/app/middleware/use_gzip_decompress_test.go@@ -0,0 +1,105 @@+package middleware++import (+ "bytes"+ "compress/gzip"+ "errors"+ "io"+ "net/http"+ "net/http/httptest"+ "testing"++ "github.com/gin-gonic/gin"+)++const testLimit = 64 * 1024++func gzipped(t *testing.T, data []byte) []byte {+ t.Helper()+ var buf bytes.Buffer+ zw := gzip.NewWriter(&buf)+ if _, err := zw.Write(data); err != nil {+ t.Fatal(err)+ }+ if err := zw.Close(); err != nil {+ t.Fatal(err)+ }+ return buf.Bytes()+}++func emptyGzipStream(t *testing.T, minBytes int) []byte {+ t.Helper()+ member := gzipped(t, nil)+ var buf bytes.Buffer+ for buf.Len() <= minBytes {+ buf.Write(member)+ }+ return buf.Bytes()+}++func runUseGzip(t *testing.T, encoding string, body []byte) *gin.Context {+ t.Helper()+ gin.SetMode(gin.TestMode)+ c, _ := gin.CreateTestContext(httptest.NewRecorder())+ c.Request = httptest.NewRequest(http.MethodPost, "/api/report", bytes.NewReader(body))+ if encoding != "" {+ c.Request.Header.Set("Content-Encoding", encoding)+ }+ useGzipLimited(c, testLimit)+ return c+}++func TestUseGzipCapsBodySize(t *testing.T) {+ cases := []struct {+ name string+ encoding string+ body []byte+ wantTooLarge bool+ }{+ {"gzip bomb capped at decompressed limit", "gzip", gzipped(t, bytes.Repeat([]byte("A"), testLimit*100)), true},+ {"plain oversized body capped", "", bytes.Repeat([]byte("A"), testLimit+1), true},+ {"oversized raw gzip body capped", "gzip", emptyGzipStream(t, testLimit), true},+ {"small gzip body passes", "gzip", gzipped(t, []byte(`{"ok":true}`)), false},+ {"small plain body passes", "", []byte(`{"ok":true}`), false},+ }++ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ c := runUseGzip(t, tc.encoding, tc.body)+ if c.IsAborted() {+ t.Fatal("middleware aborted on a well-formed body")+ }++ n, err := io.Copy(io.Discard, c.Request.Body)++ var maxBytesErr *http.MaxBytesError+ if tc.wantTooLarge {+ if !errors.As(err, &maxBytesErr) {+ t.Fatalf("expected MaxBytesError after %d bytes, got %v", n, err)+ }+ return+ }+ if err != nil {+ t.Fatalf("unexpected error: %v", err)+ }+ })+ }+}++func TestUseGzipCapsRawBodyNotJustDecompressed(t *testing.T) {+ body := emptyGzipStream(t, testLimit)+ if len(body) <= testLimit {+ t.Fatalf("fixture must exceed the raw cap, got %d bytes for limit %d", len(body), testLimit)+ }++ c := runUseGzip(t, "gzip", body)+ n, err := io.Copy(io.Discard, c.Request.Body)++ var maxBytesErr *http.MaxBytesError+ if !errors.As(err, &maxBytesErr) {+ t.Fatalf("raw body over the cap must be rejected, got err=%v", err)+ }+ if n != 0 {+ t.Fatalf("fixture should decompress to nothing, got %d bytes", n)+ }+}diff --git a/backend/app/notifications/adapter_email.go b/backend/app/notifications/adapter_email.goindex 3b1b4348..e6bdb83f 100644--- a/backend/app/notifications/adapter_email.go+++ b/backend/app/notifications/adapter_email.go@@ -3,6 +3,7 @@ package notificationsimport ("context""fmt"+ netmail "net/mail""strings""github.com/tracewayapp/traceway/backend/app/models"@@ -23,7 +24,7 @@ func (a *EmailAdapter) Validate() error {return fmt.Errorf("maximum 10 recipients allowed")}for _, r := range a.Recipients {- if !strings.Contains(r, "@") {+ if _, err := netmail.ParseAddress(r); err != nil || strings.ContainsAny(r, "\r\n") {return fmt.Errorf("invalid email address: %s", r)}}diff --git a/backend/app/repositories/telemetry/clickhouse/metric_point.repository.go b/backend/app/repositories/telemetry/clickhouse/metric_point.repository.goindex 4d4b391a..b6daae00 100644--- a/backend/app/repositories/telemetry/clickhouse/metric_point.repository.go+++ b/backend/app/repositories/telemetry/clickhouse/metric_point.repository.go@@ -6,6 +6,7 @@ import ("context""github.com/tracewayapp/traceway/backend/app/chdb""github.com/tracewayapp/traceway/backend/app/models"+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared""time""github.com/ClickHouse/clickhouse-go/v2/lib/driver"@@ -28,7 +29,7 @@ func (r *metricPointRepository) InsertAsync(ctx context.Context, points []models})}-func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy string) (map[string][]models.TimeSeriesPoint, error) {+func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {table := selectTable(to.Sub(from))if aggregation == "last" {table = "metric_points"@@ -56,9 +57,14 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uquery += " GROUP BY bucket"if hasGroupBy {- query += ", group_key"+ query += ", group_key ORDER BY group_key ASC, bucket ASC"+ if maxGroups > 0 {+ query += " LIMIT ?"+ args = append(args, shared.TimeSeriesRowLimit(maxGroups))+ }+ } else {+ query += " ORDER BY bucket ASC"}- query += " ORDER BY bucket ASC"rows, err := chdb.Conn.Query(ctx, query, args...)if err != nil {@@ -85,6 +91,9 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uif groupKey == "" {groupKey = "(empty)"}+ if shared.GroupCapReached(result, groupKey, maxGroups) {+ break+ }result[groupKey] = append(result[groupKey], models.TimeSeriesPoint{Timestamp: bucket,Value: value,diff --git a/backend/app/repositories/telemetry/duckdb/metric_point.repository.go b/backend/app/repositories/telemetry/duckdb/metric_point.repository.goindex c8e399e9..6b321e1c 100644--- a/backend/app/repositories/telemetry/duckdb/metric_point.repository.go+++ b/backend/app/repositories/telemetry/duckdb/metric_point.repository.go@@ -61,7 +61,7 @@ func (r *metricPointRepository) InsertAsync(ctx context.Context, points []models})}-func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy string) (map[string][]models.TimeSeriesPoint, error) {+func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {secs := intervalMinutes * 60aggFunc := duckdbAggregationFunc(aggregation)hasGroupBy := groupBy != ""@@ -93,9 +93,14 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uquery := selectClause + filterClauses + " GROUP BY bucket"if hasGroupBy {- query += ", group_key"+ query += ", group_key ORDER BY group_key ASC, bucket ASC"+ if maxGroups > 0 {+ query += " LIMIT :row_limit"+ params["row_limit"] = shared.TimeSeriesRowLimit(maxGroups)+ }+ } else {+ query += " ORDER BY bucket ASC"}- query += " ORDER BY bucket ASC"parsedQuery, args, err := lit.ParseNamedQuery(db.Driver, query, params)if err != nil {@@ -131,6 +136,9 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uif groupKey == "" {groupKey = "(empty)"}+ if shared.GroupCapReached(result, groupKey, maxGroups) {+ break+ }result[groupKey] = append(result[groupKey], models.TimeSeriesPoint{Timestamp: bucket,Value: value,diff --git a/backend/app/repositories/telemetry/metric_point.repository_duckdb_test.go b/backend/app/repositories/telemetry/metric_point.repository_duckdb_test.goindex 0bfe77f1..8d2869a2 100644--- a/backend/app/repositories/telemetry/metric_point.repository_duckdb_test.go+++ b/backend/app/repositories/telemetry/metric_point.repository_duckdb_test.go@@ -70,7 +70,7 @@ func TestDuckDBMetricPointsRoundTrip(t *testing.T) {}// time_bucket over a 1-minute window collapses all three points into one bucket.- series, err := repo.QueryTimeSeries(ctx, project, "cpu.used_pcnt", from, to, 1, "avg", nil, "")+ series, err := repo.QueryTimeSeries(ctx, project, "cpu.used_pcnt", from, to, 1, "avg", nil, "", 0)if err != nil {t.Fatalf("QueryTimeSeries: %v", err)}@@ -79,7 +79,7 @@ func TestDuckDBMetricPointsRoundTrip(t *testing.T) {}// json_extract_string group-by splits the same window by server tag.- grouped, err := repo.QueryTimeSeries(ctx, project, "cpu.used_pcnt", from, to, 1, "avg", nil, "server_name")+ grouped, err := repo.QueryTimeSeries(ctx, project, "cpu.used_pcnt", from, to, 1, "avg", nil, "server_name", 0)if err != nil {t.Fatalf("QueryTimeSeries grouped: %v", err)}diff --git a/backend/app/repositories/telemetry/metric_point_repository_test.go b/backend/app/repositories/telemetry/metric_point_repository_test.goindex 672897c7..d0ba12d6 100644--- a/backend/app/repositories/telemetry/metric_point_repository_test.go+++ b/backend/app/repositories/telemetry/metric_point_repository_test.go@@ -26,7 +26,7 @@ func TestMetricPointRepository_InsertAndQueryTimeSeries(t *testing.T) {t.Fatalf("InsertAsync failed: %v", err)}- result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "cpu.used_pcnt", base.Add(-time.Minute), base.Add(time.Hour), 30, "avg", nil, "")+ result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "cpu.used_pcnt", base.Add(-time.Minute), base.Add(time.Hour), 30, "avg", nil, "", 0)if err != nil {t.Fatalf("QueryTimeSeries failed: %v", err)}@@ -46,6 +46,55 @@ func TestMetricPointRepository_InsertAndQueryTimeSeries(t *testing.T) {assertApproxEqual(t, "second bucket avg", series[1].Value, 70.0, 0.1)}+func TestMetricPointRepository_QueryTimeSeries_GroupCap(t *testing.T) {+ setupTestDB(t)+ ctx := context.Background()+ projectId := uuid.New()+ base := time.Now().UTC().Truncate(time.Hour)++ var points []models.MetricPoint+ for _, server := range []string{"web-5", "web-1", "web-3", "web-2", "web-4"} {+ for i := 0; i < 3; i++ {+ points = append(points, makeMetricPoint(projectId, "cpu.used_pcnt", float64(10*i), map[string]string{"server_name": server}, base.Add(time.Duration(i)*10*time.Minute)))+ }+ }+ if err := MetricPointRepository.InsertAsync(ctx, points); err != nil {+ t.Fatalf("InsertAsync failed: %v", err)+ }++ capped, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "cpu.used_pcnt", base.Add(-time.Minute), base.Add(time.Hour), 10, "avg", nil, "server_name", 2)+ if err != nil {+ t.Fatalf("QueryTimeSeries failed: %v", err)+ }+ if len(capped) != 3 {+ t.Fatalf("expected 2 groups plus the marker, got %d: %v", len(capped), capped)+ }+ for _, server := range []string{"web-1", "web-2"} {+ if len(capped[server]) != 3 {+ t.Fatalf("group %s should be complete with 3 buckets, got %d", server, len(capped[server]))+ }+ }+ if marker, ok := capped["web-3"]; !ok || len(marker) != 0 {+ t.Fatalf("expected an empty marker for the first group past the cap, got %v", capped)+ }++ uncapped, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "cpu.used_pcnt", base.Add(-time.Minute), base.Add(time.Hour), 10, "avg", nil, "server_name", 0)+ if err != nil {+ t.Fatalf("QueryTimeSeries failed: %v", err)+ }+ if len(uncapped) != 5 {+ t.Fatalf("cap 0 must return every group, got %d", len(uncapped))+ }++ wide, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "cpu.used_pcnt", base.Add(-time.Minute), base.Add(time.Hour), 10, "avg", nil, "server_name", 5)+ if err != nil {+ t.Fatalf("QueryTimeSeries failed: %v", err)+ }+ if len(wide) != 5 {+ t.Fatalf("a cap equal to the cardinality must not add a marker, got %d", len(wide))+ }+}+func TestMetricPointRepository_QueryTimeSeries_WithGroupBy(t *testing.T) {setupTestDB(t)ctx := context.Background()@@ -62,7 +111,7 @@ func TestMetricPointRepository_QueryTimeSeries_WithGroupBy(t *testing.T) {t.Fatalf("InsertAsync failed: %v", err)}- result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "cpu.used_pcnt", base.Add(-time.Minute), base.Add(time.Hour), 60, "avg", nil, "server_name")+ result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "cpu.used_pcnt", base.Add(-time.Minute), base.Add(time.Hour), 60, "avg", nil, "server_name", 0)if err != nil {t.Fatalf("QueryTimeSeries with groupBy failed: %v", err)}@@ -103,7 +152,7 @@ func TestMetricPointRepository_QueryTimeSeries_WithTagFilter(t *testing.T) {}filters := map[string]string{"server_name": "web-1"}- result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "cpu.used_pcnt", base.Add(-time.Minute), base.Add(time.Hour), 60, "avg", filters, "")+ result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "cpu.used_pcnt", base.Add(-time.Minute), base.Add(time.Hour), 60, "avg", filters, "", 0)if err != nil {t.Fatalf("QueryTimeSeries with tag filter failed: %v", err)}@@ -350,7 +399,7 @@ func TestMetricPointRepository_QueryTimeSeries_Aggregations(t *testing.T) {to := base.Add(time.Hour)// Test min aggregation- result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "response_time", from, to, 60, "min", nil, "")+ result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "response_time", from, to, 60, "min", nil, "", 0)if err != nil {t.Fatalf("QueryTimeSeries min failed: %v", err)}@@ -361,7 +410,7 @@ func TestMetricPointRepository_QueryTimeSeries_Aggregations(t *testing.T) {assertApproxEqual(t, "min", series[0].Value, 100.0, 0.1)// Test max aggregation- result, err = MetricPointRepository.QueryTimeSeries(ctx, projectId, "response_time", from, to, 60, "max", nil, "")+ result, err = MetricPointRepository.QueryTimeSeries(ctx, projectId, "response_time", from, to, 60, "max", nil, "", 0)if err != nil {t.Fatalf("QueryTimeSeries max failed: %v", err)}@@ -369,7 +418,7 @@ func TestMetricPointRepository_QueryTimeSeries_Aggregations(t *testing.T) {assertApproxEqual(t, "max", series[0].Value, 300.0, 0.1)// Test sum aggregation- result, err = MetricPointRepository.QueryTimeSeries(ctx, projectId, "response_time", from, to, 60, "sum", nil, "")+ result, err = MetricPointRepository.QueryTimeSeries(ctx, projectId, "response_time", from, to, 60, "sum", nil, "", 0)if err != nil {t.Fatalf("QueryTimeSeries sum failed: %v", err)}@@ -396,7 +445,7 @@ func TestMetricPointRepository_QueryTimeSeries_LastAggregation(t *testing.T) {from := base.Add(-time.Minute)to := base.Add(time.Hour)- result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "response_time", from, to, 60, "last", nil, "")+ result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "response_time", from, to, 60, "last", nil, "", 0)if err != nil {t.Fatalf("QueryTimeSeries last failed: %v", err)}@@ -428,7 +477,7 @@ func TestMetricPointRepository_QueryTimeSeries_LastAggregationWithGroupBy(t *tesfrom := base.Add(-time.Minute)to := base.Add(time.Hour)- result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "disk.used", from, to, 60, "last", nil, "disk")+ result, err := MetricPointRepository.QueryTimeSeries(ctx, projectId, "disk.used", from, to, 60, "last", nil, "disk", 0)if err != nil {t.Fatalf("QueryTimeSeries last with groupBy failed: %v", err)}diff --git a/backend/app/repositories/telemetry/shared/metric_limits.go b/backend/app/repositories/telemetry/shared/metric_limits.gonew file mode 100644index 00000000..302858a3--- /dev/null+++ b/backend/app/repositories/telemetry/shared/metric_limits.go@@ -0,0 +1,21 @@+package shared++const MaxTimeSeriesBuckets = 2000++func TimeSeriesRowLimit(maxGroups int) int {+ return (maxGroups + 1) * MaxTimeSeriesBuckets+}++func GroupCapReached[T any](result map[string][]T, groupKey string, maxGroups int) bool {+ if maxGroups <= 0 {+ return false+ }+ if _, seen := result[groupKey]; seen {+ return false+ }+ if len(result) < maxGroups {+ return false+ }+ result[groupKey] = nil+ return true+}diff --git a/backend/app/repositories/telemetry/sqlite/metric_point.repository.go b/backend/app/repositories/telemetry/sqlite/metric_point.repository.goindex 37cad2e2..1cf32123 100644--- a/backend/app/repositories/telemetry/sqlite/metric_point.repository.go+++ b/backend/app/repositories/telemetry/sqlite/metric_point.repository.go@@ -68,7 +68,7 @@ func (r *metricPointRepository) InsertAsync(ctx context.Context, points []modelsreturn tx.Commit()}-func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy string) (map[string][]models.TimeSeriesPoint, error) {+func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uuid.UUID, name string, from, to time.Time, intervalMinutes int, aggregation string, tagFilters map[string]string, groupBy string, maxGroups int) (map[string][]models.TimeSeriesPoint, error) {secs := intervalMinutes * 60aggFunc := sqliteAggregationFunc(aggregation)hasGroupBy := groupBy != ""@@ -105,9 +105,14 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uquery := selectClause + filterClauses + " GROUP BY bucket"if hasGroupBy {- query += ", group_key"+ query += ", group_key ORDER BY group_key ASC, bucket ASC"+ if maxGroups > 0 {+ query += " LIMIT :row_limit"+ params["row_limit"] = shared.TimeSeriesRowLimit(maxGroups)+ }+ } else {+ query += " ORDER BY bucket ASC"}- query += " ORDER BY bucket ASC"parsedQuery, args, err := lit.ParseNamedQuery(db.Driver, query, params)if err != nil {@@ -151,6 +156,9 @@ func (r *metricPointRepository) QueryTimeSeries(ctx context.Context, projectId uif groupKey == "" {groupKey = "(empty)"}+ if shared.GroupCapReached(result, groupKey, maxGroups) {+ break+ }result[groupKey] = append(result[groupKey], models.TimeSeriesPoint{Timestamp: bucket,Value: value,diff --git a/backend/app/services/email.service.go b/backend/app/services/email.service.goindex f0532eb8..87f5d1ec 100644--- a/backend/app/services/email.service.go+++ b/backend/app/services/email.service.go@@ -11,6 +11,7 @@ import ("mime/multipart""mime/quotedprintable""net"+ netmail "net/mail""net/smtp""net/textproto""strings"@@ -84,13 +85,36 @@ func RenderEmail(email Email) (string, error) {return out.String(), nil}+func sanitizeHeaderValues(values []string) []string {+ out := make([]string, len(values))+ for i, v := range values {+ out[i] = strings.NewReplacer("\r", "", "\n", "").Replace(v)+ }+ return out+}++func envelopeAddresses(recipients []string) []string {+ out := make([]string, 0, len(recipients))+ for _, r := range recipients {+ if parsed, err := netmail.ParseAddress(r); err == nil {+ out = append(out, parsed.Address)+ continue+ }+ out = append(out, strings.TrimSpace(r))+ }+ return out+}+func SendEmail(ctx context.Context, email Email) error {if len(email.To) == 0 {return fmt.Errorf("no recipients")}+ recipients := sanitizeHeaderValues(email.To)+ subject := sanitizeHeaderValues([]string{email.Subject})[0]+if !emailEnabled() {- config.Logf("[EMAIL LOG] To: %s\nSubject: %s\nBody:\n%s", strings.Join(email.To, ", "), email.Subject, email.Text)+ config.Logf("[EMAIL LOG] To: %s\nSubject: %s\nBody:\n%s", strings.Join(recipients, ", "), subject, email.Text)return nil}@@ -102,8 +126,8 @@ func SendEmail(ctx context.Context, email Email) error {from := config.Config.SMTPFromvar buf bytes.Bufferfmt.Fprintf(&buf, "From: %s\r\n", from)- fmt.Fprintf(&buf, "To: %s\r\n", strings.Join(email.To, ", "))- fmt.Fprintf(&buf, "Subject: %s\r\n", mime.QEncoding.Encode("utf-8", email.Subject))+ fmt.Fprintf(&buf, "To: %s\r\n", strings.Join(recipients, ", "))+ fmt.Fprintf(&buf, "Subject: %s\r\n", mime.QEncoding.Encode("utf-8", subject))fmt.Fprintf(&buf, "Date: %s\r\n", time.Now().Format(time.RFC1123Z))buf.WriteString("MIME-Version: 1.0\r\n")@@ -132,7 +156,11 @@ func SendEmail(ctx context.Context, email Email) error {return err}- return sendSMTP(ctx, from, email.To, buf.Bytes())+ envelopeFrom := from+ if parsed, err := netmail.ParseAddress(from); err == nil {+ envelopeFrom = parsed.Address+ }+ return sendSMTP(ctx, envelopeFrom, envelopeAddresses(recipients), buf.Bytes())}func sendSMTP(ctx context.Context, from string, to []string, msg []byte) error {diff --git a/backend/app/services/email_envelope_test.go b/backend/app/services/email_envelope_test.gonew file mode 100644index 00000000..631b2e17--- /dev/null+++ b/backend/app/services/email_envelope_test.go@@ -0,0 +1,103 @@+package services++import (+ "bufio"+ "context"+ "fmt"+ "net"+ "strings"+ "testing"++ "github.com/tracewayapp/traceway/backend/app/config"+)++func fakeSMTP(t *testing.T) (host string, port string, got *[]string) {+ t.Helper()+ ln, err := net.Listen("tcp", "127.0.0.1:0")+ if err != nil {+ t.Fatal(err)+ }+ lines := []string{}+ done := make(chan struct{})+ go func() {+ defer close(done)+ conn, err := ln.Accept()+ if err != nil {+ return+ }+ defer conn.Close()+ r := bufio.NewReader(conn)+ fmt.Fprint(conn, "220 fake ESMTP\r\n")+ inData := false+ for {+ line, err := r.ReadString('\n')+ if err != nil {+ return+ }+ line = strings.TrimRight(line, "\r\n")+ if inData {+ if line == "." {+ inData = false+ fmt.Fprint(conn, "250 OK\r\n")+ }+ continue+ }+ switch {+ case strings.HasPrefix(line, "EHLO"), strings.HasPrefix(line, "HELO"):+ fmt.Fprint(conn, "250-fake\r\n250 OK\r\n")+ case strings.HasPrefix(line, "MAIL FROM"), strings.HasPrefix(line, "RCPT TO"):+ lines = append(lines, line)+ fmt.Fprint(conn, "250 OK\r\n")+ case strings.HasPrefix(line, "DATA"):+ inData = true+ fmt.Fprint(conn, "354 send it\r\n")+ case strings.HasPrefix(line, "QUIT"):+ fmt.Fprint(conn, "221 bye\r\n")+ return+ default:+ fmt.Fprint(conn, "250 OK\r\n")+ }+ }+ }()+ t.Cleanup(func() { ln.Close(); <-done })+ h, p, _ := net.SplitHostPort(ln.Addr().String())+ return h, p, &lines+}++func TestSendEmailUsesBareAddressesInTheEnvelope(t *testing.T) {+ host, port, captured := fakeSMTP(t)++ prev := config.Config+ config.Init(&config.Cfg{+ SMTPEnabled: "true", SMTPHost: host, SMTPPort: port,+ SMTPFrom: "Traceway Alerts <[email protected]>",+ })+ t.Cleanup(func() { config.Init(prev) })++ err := SendEmail(context.Background(), Email{+ Subject: "Test",+ Template: "invitation",+ Title: "Test",+ Text: "test",+ Data: invitationData{InviterName: "Dana", OrgName: "Acme"},+ })+ if err != nil {+ t.Fatalf("SendEmail: %v", err)+ }++ want := []string{+ "MAIL FROM:<[email protected]>",+ "RCPT TO:<[email protected]>",+ "RCPT TO:<[email protected]>",+ "RCPT TO:<[email protected]>",+ }+ for i, w := range want {+ if i >= len(*captured) {+ t.Fatalf("missing command %d, got %v", i, *captured)+ }+ if (*captured)[i] != w {+ t.Errorf("command %d:\n got %q\nwant %q", i, (*captured)[i], w)+ }+ }+}diff --git a/backend/app/services/sourcemap_cache_test.go b/backend/app/services/sourcemap_cache_test.goindex d7b4a962..6fd161af 100644--- a/backend/app/services/sourcemap_cache_test.go+++ b/backend/app/services/sourcemap_cache_test.go@@ -130,6 +130,7 @@ func TestInvalidateSourceMapEvictsStaleResolver(t *testing.T) {defer func() { storage.Store = prev }()cs := &countingStorage{reads: map[string]int{}, data: map[string][]byte{}}storage.Store = cs+ defer awaitTWWarmups(t)projectId := uuid.New()mapKey := fmt.Sprintf("sourcemaps/%s/stable.js.map", projectId)diff --git a/backend/app/services/sourcemap_disk_cache_test.go b/backend/app/services/sourcemap_disk_cache_test.goindex 79bb043c..475a765f 100644--- a/backend/app/services/sourcemap_disk_cache_test.go+++ b/backend/app/services/sourcemap_disk_cache_test.go@@ -287,6 +287,7 @@ func TestGenerateTWArtifacts(t *testing.T) {seedFixture(t, cs, prefix+"minified.js", "testdata/sourcemapcache/simple/minified.js")GenerateTWArtifacts(context.Background(), projectId, []string{"minified.js.map", "minified.js"})+ awaitTWWarmups(t)twBytes, ok := cs.data[prefix+"minified.js.tw"]if !ok {@@ -306,6 +307,7 @@ func TestGenerateTWArtifactsDeletesStaleOnFailure(t *testing.T) {cs.data[prefix+"minified.js.map"] = []byte("{not a valid source map")GenerateTWArtifacts(context.Background(), projectId, []string{"minified.js.map"})+ awaitTWWarmups(t)if _, ok := cs.data[prefix+"minified.js.tw"]; ok {t.Fatal("stale tw artifact must be deleted even when regeneration fails")diff --git a/backend/app/services/sourcemap_tw.go b/backend/app/services/sourcemap_tw.goindex 2b597d87..e71482ff 100644--- a/backend/app/services/sourcemap_tw.go+++ b/backend/app/services/sourcemap_tw.go@@ -5,6 +5,9 @@ import ("errors""fmt""strings"+ "sync"+ "sync/atomic"+ "time""github.com/tracewayapp/traceway/backend/app/storage"@@ -12,6 +15,27 @@ import (traceway "go.tracewayapp.com")+const (+ twWorkers = 2+ twQueueDepth = 64+ twJobTimeout = 2 * time.Minute+)++type twJob struct {+ projectId uuid.UUID+ base string+}++var (+ twQueue = make(chan twJob, twQueueDepth)+ twWorkersOnce sync.Once+ twPendingMu sync.Mutex+ twPending = map[twJob]struct{}{}+ twRunning atomic.Int32+ twDroppedAt time.Time+ twRun = warmTWArtifact+)+func GenerateTWArtifacts(ctx context.Context, projectId uuid.UUID, fileNames []string) {ctx = context.WithoutCancel(ctx)@@ -21,19 +45,70 @@ func GenerateTWArtifacts(ctx context.Context, projectId uuid.UUID, fileNames []s}for base := range bases {- mapKey := SourceMapStorageKey(projectId, base+".map")- bundleKey := SourceMapStorageKey(projectId, base)- twKey := twKeyFor(mapKey)-+ twKey := twKeyFor(SourceMapStorageKey(projectId, base+".map"))if err := storage.Store.Delete(ctx, twKey); err != nil {traceway.CaptureException(fmt.Errorf("tw generation: failed to delete stale tw artifact (key=%s): %w", twKey, err))}InvalidateSourceMap(projectId, base)- _, done, err := sharedCache.Get(ctx, twKey, loadSourceMapBlob(mapKey, bundleKey))- if err != nil && !errors.Is(err, storage.ErrNotFound) {- traceway.CaptureException(fmt.Errorf("tw generation: failed to warm resolver (key=%s): %w", mapKey, err))- }- done()+ enqueueTWWarmup(twJob{projectId: projectId, base: base})+ }+}++func enqueueTWWarmup(job twJob) bool {+ twWorkersOnce.Do(startTWWorkers)++ twPendingMu.Lock()+ defer twPendingMu.Unlock()+ if _, pending := twPending[job]; pending {+ return false+ }+ select {+ case twQueue <- job:+ twPending[job] = struct{}{}+ return true+ default:+ reportTWDrop(job)+ return false+ }+}++func startTWWorkers() {+ for i := 0; i < twWorkers; i++ {+ go func() {+ for job := range twQueue {+ twRunning.Add(1)+ twPendingMu.Lock()+ delete(twPending, job)+ twPendingMu.Unlock()+ func() {+ defer twRunning.Add(-1)+ defer traceway.Recover()+ twRun(job)+ }()+ }+ }()+ }+}++func warmTWArtifact(job twJob) {+ ctx, cancel := context.WithTimeout(context.Background(), twJobTimeout)+ defer cancel()++ mapKey := SourceMapStorageKey(job.projectId, job.base+".map")+ bundleKey := SourceMapStorageKey(job.projectId, job.base)+ _, done, err := sharedCache.Get(ctx, twKeyFor(mapKey), loadSourceMapBlob(mapKey, bundleKey))+ if err != nil && !errors.Is(err, storage.ErrNotFound) {+ traceway.CaptureException(fmt.Errorf("tw generation: failed to warm resolver (key=%s): %w", mapKey, err))+ }+ done()+}++func reportTWDrop(job twJob) {+ now := time.Now()+ if now.Sub(twDroppedAt) < time.Minute {+ return}+ twDroppedAt = now+ traceway.CaptureException(fmt.Errorf("tw generation: warm-up queue full (%d jobs), dropped %s/%s; the artifact will be built on first lookup", twQueueDepth, job.projectId, job.base))}diff --git a/backend/app/services/sourcemap_tw_bound_test.go b/backend/app/services/sourcemap_tw_bound_test.gonew file mode 100644index 00000000..bba91619--- /dev/null+++ b/backend/app/services/sourcemap_tw_bound_test.go@@ -0,0 +1,90 @@+package services++import (+ "fmt"+ "sync/atomic"+ "testing"+ "time"++ "github.com/google/uuid"+)++func awaitTWWarmups(t *testing.T) {+ t.Helper()+ deadline := time.Now().Add(10 * time.Second)+ for time.Now().Before(deadline) {+ twPendingMu.Lock()+ pending := len(twPending)+ twPendingMu.Unlock()+ if pending == 0 && len(twQueue) == 0 && twRunning.Load() == 0 {+ return+ }+ time.Sleep(5 * time.Millisecond)+ }+ t.Fatal("source-map warm-up queue did not drain")+}++func TestTWWarmupQueueIsBoundedAndCoalesced(t *testing.T) {+ var live, peak int64+ gate := make(chan struct{})+ started := make(chan twJob, twWorkers+twQueueDepth+8)++ prev := twRun+ twRun = func(job twJob) {+ n := atomic.AddInt64(&live, 1)+ for {+ old := atomic.LoadInt64(&peak)+ if n <= old || atomic.CompareAndSwapInt64(&peak, old, n) {+ break+ }+ }+ started <- job+ <-gate+ atomic.AddInt64(&live, -1)+ }+ t.Cleanup(func() { twRun = prev })+ defer func() {+ close(gate)+ awaitTWWarmups(t)+ }()++ project := uuid.New()+ job := func(i int) twJob { return twJob{projectId: project, base: fmt.Sprintf("bundle-%d", i)} }++ for i := 0; i < twWorkers; i++ {+ if !enqueueTWWarmup(job(i)) {+ t.Fatalf("job %d was not accepted on an idle pool", i)+ }+ }+ for i := 0; i < twWorkers; i++ {+ select {+ case <-started:+ case <-time.After(time.Second):+ t.Fatal("workers did not start")+ }+ }+ for i := twWorkers; i < twWorkers+twQueueDepth; i++ {+ if !enqueueTWWarmup(job(i)) {+ t.Fatalf("job %d was dropped with room left in the queue", i)+ }+ }++ done := make(chan bool, 1)+ go func() { done <- enqueueTWWarmup(job(10_000)) }()+ select {+ case queued := <-done:+ if queued {+ t.Fatal("a job past the queue depth was accepted")+ }+ case <-time.After(time.Second):+ t.Fatal("enqueue blocked on a full queue")+ }++ if enqueueTWWarmup(job(twWorkers + 1)) {+ t.Fatal("a pending job was queued a second time")+ }++ if peak > int64(twWorkers) {+ t.Fatalf("peak concurrent warm-ups %d exceeded the %d worker bound", peak, twWorkers)+ }+}diff --git a/backend/cmd/run.go b/backend/cmd/run.goindex d2b91b0d..a0f490aa 100644--- a/backend/cmd/run.go+++ b/backend/cmd/run.go@@ -4,10 +4,12 @@ import ("context""fmt""io/fs"+ "net""net/http""os""strconv""strings"+ "sync""time""github.com/tracewayapp/traceway/backend/app/backfill"@@ -182,9 +184,15 @@ func Run(opts ...Option) {router = gin.Default()}+ router.Use(middleware.GuardBodyReads(middleware.DefaultBodyIdle, middleware.DefaultBodyTotal))+if err := configureClientIP(router, cfg); err != nil {panic(fmt.Errorf("invalid TRUSTED_PROXIES: %w", err))}+ if cfg.TrustedProxyHeader == "" {+ router.Use(warnOnUntrustedForwardedFor(cfg.TrustedProxyList()))+ }+ router.Use(middleware.SecurityHeaders)if monitoringTracewayUrl := cfg.MonitoringTracewayURL; monitoringTracewayUrl != "" {twmw := tracewaygin.New(@@ -282,16 +290,82 @@ func Run(opts ...Option) {port := ":" + portsList[i]config.Logln("Starting server on " + port)- if err := router.Run(port); err != nil {- panic(fmt.Errorf("Error starting server on port %s: %v", port, err))- }+ serveHTTP(router, port)}()}}notifySystemd()- if err := router.Run(":" + portsList[0]); err != nil {- panic(fmt.Errorf("Error starting server on port %s: %v", portsList[0], err))+ serveHTTP(router, ":"+portsList[0])+}++func warnOnUntrustedForwardedFor(proxies []string) gin.HandlerFunc {+ detect := newUntrustedForwarderDetector(proxies)+ return func(c *gin.Context) {+ if peer, first := detect(c); first {+ config.Logf("X-Forwarded-For received from %s, which is not in TRUSTED_PROXIES. The header is ignored, so per-IP rate limits and session client IPs see that address instead of the real client. If %s is your proxy or CDN, add its range to TRUSTED_PROXIES.", peer, peer)+ }+ c.Next()+ }+}++func newUntrustedForwarderDetector(proxies []string) func(*gin.Context) (string, bool) {+ nets := trustedProxyNets(proxies)+ var once sync.Once+ return func(c *gin.Context) (string, bool) {+ if c.GetHeader("X-Forwarded-For") == "" && c.GetHeader("X-Real-IP") == "" {+ return "", false+ }+ peer := c.RemoteIP()+ ip := net.ParseIP(peer)+ if ip == nil || containsIP(nets, ip) {+ return "", false+ }+ first := false+ once.Do(func() { first = true })+ return peer, first+ }+}++func trustedProxyNets(proxies []string) []*net.IPNet {+ var nets []*net.IPNet+ for _, p := range proxies {+ if _, n, err := net.ParseCIDR(p); err == nil {+ nets = append(nets, n)+ continue+ }+ if ip := net.ParseIP(p); ip != nil {+ bits := 32+ if ip.To4() == nil {+ bits = 128+ }+ nets = append(nets, &net.IPNet{IP: ip, Mask: net.CIDRMask(bits, bits)})+ }+ }+ return nets+}++func containsIP(nets []*net.IPNet, ip net.IP) bool {+ for _, n := range nets {+ if n.Contains(ip) {+ return true+ }+ }+ return false+}++const serverReadTimeout = time.Hour++func serveHTTP(router *gin.Engine, port string) {+ srv := &http.Server{+ Addr: port,+ Handler: router,+ ReadHeaderTimeout: 15 * time.Second,+ ReadTimeout: serverReadTimeout,+ IdleTimeout: 2 * time.Minute,+ }+ if err := srv.ListenAndServe(); err != nil {+ panic(fmt.Errorf("Error starting server on port %s: %v", port, err))}}@@ -340,6 +414,7 @@ func applyEnvOverrides(cfg *config.Cfg) {{"TWILIO_FROM_NUMBER", &cfg.TwilioFromNumber},{"TWILIO_MESSAGING_SERVICE_SID", &cfg.TwilioMessagingServiceSID},{"ALLOW_PRIVATE_NOTIFICATION_TARGETS", &cfg.AllowPrivateNotificationTargets},+ {"REPORT_MAX_BODY_MB", &cfg.ReportMaxBodyMB},{"OAUTH_SESSION_SECRET", &cfg.OAuthSessionSecret},{"GOOGLE_CLIENT_ID", &cfg.GoogleClientID},{"GOOGLE_CLIENT_SECRET", &cfg.GoogleClientSecret},diff --git a/backend/cmd/run_test.go b/backend/cmd/run_test.goindex 63c5216b..810c6fd7 100644--- a/backend/cmd/run_test.go+++ b/backend/cmd/run_test.go@@ -1,6 +1,7 @@package cmdimport (+ "net""net/http""net/http/httptest""testing"@@ -70,3 +71,29 @@ func clientIPRouter(t *testing.T, cfg *config.Cfg) *gin.Engine {})return router}++func TestUntrustedForwarderDetectorFiresOnceForPeersOutsideTheList(t *testing.T) {+ gin.SetMode(gin.TestMode)+ detect := newUntrustedForwarderDetector([]string{"10.0.0.0/8", "::1"})+ check := func(peer, xff string) (string, bool) {+ c, _ := gin.CreateTestContext(httptest.NewRecorder())+ c.Request = httptest.NewRequest(http.MethodGet, "/", nil)+ c.Request.RemoteAddr = net.JoinHostPort(peer, "1234")+ if xff != "" {+ c.Request.Header.Set("X-Forwarded-For", xff)+ }+ return detect(c)+ }+ if _, first := check("203.0.113.9", ""); first {+ t.Fatal("fired without a forwarding header")+ }+ if _, first := check("10.1.2.3", "198.51.100.20"); first {+ t.Fatal("fired for a trusted peer")+ }+ if peer, first := check("172.17.0.1", "198.51.100.20"); !first || peer != "172.17.0.1" {+ t.Fatalf("got (%q, %v), want the untrusted peer reported", peer, first)+ }+ if _, first := check("172.17.0.1", "198.51.100.20"); first {+ t.Fatal("fired a second time")+ }+}diff --git a/cli/cmd/traceway/login_web.go b/cli/cmd/traceway/login_web.goindex 684f0769..47e30e40 100644--- a/cli/cmd/traceway/login_web.go+++ b/cli/cmd/traceway/login_web.go@@ -100,7 +100,7 @@ func runLoginWeb(cmd *cobra.Command) error {switch {case errors.Is(err, client.ErrAuthorizationPending):continue- case errors.Is(err, client.ErrSlowDown):+ case errors.Is(err, client.ErrSlowDown), errors.Is(err, client.ErrRateLimited):interval += 5continuecase errors.Is(err, client.ErrAccessDenied):diff --git a/docker-compose.browser.yml b/docker-compose.browser.ymlindex 4aab7797..4fc455b0 100644--- a/docker-compose.browser.yml+++ b/docker-compose.browser.yml@@ -12,6 +12,11 @@ services:environment:JWT_SECRET: "change-this-to-a-secure-secret-at-least-32-chars"# APP_BASE_URL: "https://traceway.example.com"+ # Proxies whose X-Forwarded-For is trusted for the real client IP (the+ # per-IP rate limits key on it). Only loopback is trusted by default, so+ # list every hop in front of Traceway: a proxy container, an ingress, a CDN.+ # TRUSTED_PROXIES: "172.18.0.0/16"+ # TRUSTED_PROXY_HEADER: "X-Real-IP"# DuckDB sizes its memory budget from host RAM, not the container limit.# On any memory-capped deployment set this to roughly half thediff --git a/docker-compose.duckdb.yml b/docker-compose.duckdb.ymlindex 258b7985..030e634b 100644--- a/docker-compose.duckdb.yml+++ b/docker-compose.duckdb.yml@@ -8,6 +8,11 @@ services:environment:JWT_SECRET: "change-this-to-a-secure-secret-at-least-32-chars"# APP_BASE_URL: "https://traceway.example.com"+ # Proxies whose X-Forwarded-For is trusted for the real client IP (the+ # per-IP rate limits key on it). Only loopback is trusted by default, so+ # list every hop in front of Traceway: a proxy container, an ingress, a CDN.+ # TRUSTED_PROXIES: "172.18.0.0/16"+ # TRUSTED_PROXY_HEADER: "X-Real-IP"# DuckDB sizes its memory budget from host RAM, not the container limit.# On any memory-capped deployment set this to roughly half thediff --git a/docker-compose.sqlite.yml b/docker-compose.sqlite.ymlindex de4b42b6..7c32be60 100644--- a/docker-compose.sqlite.yml+++ b/docker-compose.sqlite.yml@@ -8,6 +8,11 @@ services:environment:JWT_SECRET: "change-this-to-a-secure-secret-at-least-32-chars"# APP_BASE_URL: "https://traceway.example.com"+ # Proxies whose X-Forwarded-For is trusted for the real client IP (the+ # per-IP rate limits key on it). Only loopback is trusted by default, so+ # list every hop in front of Traceway: a proxy container, an ingress, a CDN.+ # TRUSTED_PROXIES: "172.18.0.0/16"+ # TRUSTED_PROXY_HEADER: "X-Real-IP"# Optional retention knobs. Defaults are 30 days; set to "0" to disable.# SQLITE_RETENTION_DAYS prunes telemetry rows; SESSION_RECORDING_RETENTION_DAYSdiff --git a/docker-compose.yml b/docker-compose.ymlindex df9e19ba..7fd4765a 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -18,6 +18,11 @@ services:POSTGRES_PASSWORD: "traceway"POSTGRES_SSLMODE: "disable"JWT_SECRET: "change-this-to-a-secure-secret-at-least-32-chars"+ # Proxies whose X-Forwarded-For is trusted for the real client IP (the+ # per-IP rate limits key on it). Only loopback is trusted by default, so+ # list every hop in front of Traceway: a proxy container, an ingress, a CDN.+ # TRUSTED_PROXIES: "172.18.0.0/16"+ # TRUSTED_PROXY_HEADER: "X-Real-IP"# Optional. Days to keep on-disk session recordings under STORAGE_PATH/recordings/.# Worker runs hourly and on startup; 0 disables; no effect when STORAGE_TYPE=s3.# SESSION_RECORDING_RETENTION_DAYS: "30"diff --git a/docs/pages/client/js-sdk/sourcemap-upload.mdx b/docs/pages/client/js-sdk/sourcemap-upload.mdxindex 51c384cf..7380e60e 100644--- a/docs/pages/client/js-sdk/sourcemap-upload.mdx+++ b/docs/pages/client/js-sdk/sourcemap-upload.mdx@@ -67,7 +67,9 @@ Flags take precedence over environment variables when both are set.## Limits-Individual source map files must not exceed **50MB**.+Individual source map files must not exceed **50MB**, and one request must not exceed **250MB** and **1000 files** (maps and bundles together) in total.++The CLI posts every map and its sibling bundle in a single request, so the total is what a large build actually hits. A request over the total answers `413`; a single file over 50MB answers `400`.## Maps Without Source File Names@@ -140,6 +142,8 @@ The contract:|----------|--------|| Accepted extensions | `.map`, `.js`, `.cjs`, `.mjs`. Files with any other extension are silently skipped, not rejected. || Size limit | 50MB per file. A larger file fails the request with `400` and an error message. |+| Total request size | 250MB across all files in one request. A larger body fails with `413`. |+| File count | 1000 files per request, counting the bundles uploaded next to the maps. More than that fails with `422`; upload the build in smaller batches. || Auth failure | Missing or invalid token returns `401`. || Response | `200` with `{"uploaded": N}`, counting only the stored files (skipped files are excluded). || Debug IDs | Detected server-side in the uploaded content; no extra fields or flags needed. |diff --git a/docs/pages/client/otel/index.mdx b/docs/pages/client/otel/index.mdxindex 99a0a55f..0ba447b5 100644--- a/docs/pages/client/otel/index.mdx+++ b/docs/pages/client/otel/index.mdx@@ -56,7 +56,7 @@ Every OTel ecosystem ships the API, the SDK and the OTLP exporter as separate pa| **Protocol** | OTLP/HTTP only, Protobuf or JSON. **gRPC is not supported**, so `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` is required || **Compression** | Gzip supported (`Content-Encoding: gzip`) || **Content-Type** | `application/x-protobuf` (or `application/protobuf`) for OTLP/Protobuf. Anything else, including a missing header, is parsed as OTLP/JSON |-| **Max body size** | 10 MB **after decompression**. Gzip does not raise it |+| **Max body size** | 10 MB, applied to the compressed body **and** to the decompressed output. Gzip does not raise it. A larger export is rejected with `413` || **Auth failure** | `401` with an empty body. The header value must literally start with `Bearer ` |Traceway serves OTLP over HTTP only. There is no gRPC endpoint, so set your exporter's protocol to `http/protobuf` (or `http/json`) explicitly. Several SDKs default to gRPC when the variable is unset: with `OTEL_TRACES_EXPORTER=otlp` and no `OTEL_EXPORTER_OTLP_PROTOCOL`, the Python SDK resolves to gRPC, and the Java agent does the same. The failure is silent. The app runs, exits 0, prints no warning, and every export lands nowhere.@@ -67,7 +67,7 @@ OTEL_EXPORTER_OTLP_PROTOCOL=http/protobufSet that wherever you configure the exporter. The one exception is PHP, which needs `http/json` instead.-An export larger than 10 MB is truncated and then fails to parse, so Traceway answers `400 Bad Request` with a decoding error. OTLP exporters treat 4xx as permanent and drop the batch instead of retrying, so that data is lost. Keep batches comfortably under the limit by capping the exporter's batch size: `maxExportBatchSize` in the JS SDK, `max_export_batch_size` in Python, `send_batch_max_size` on the Collector's `batch` processor. The SDK default of 512 spans per export is far below the limit.+An export larger than 10 MB after decompression is rejected outright with `413 Request Entity Too Large` and `{"error":"request body exceeds the 10MB limit"}`. Nothing is truncated and nothing is partially ingested. OTLP exporters treat 4xx as permanent and drop the batch instead of retrying, so that data is lost. Keep batches comfortably under the limit by capping the exporter's batch size: `maxExportBatchSize` in the JS SDK, `max_export_batch_size` in Python, `send_batch_max_size` on the Collector's `batch` processor. The SDK default of 512 spans per export is far below the limit.If ingest is saturated, Traceway answers `503` with a `Retry-After` header instead. That one is retryable, and every OTLP exporter and the Collector will back off and resend.@@ -311,6 +311,7 @@ curl -i -X POST https://<your-instance>/api/otel/v1/traces \- `200` with the body `{}` means the endpoint and token are good. The problem is in your SDK, or in how your spans are shaped.- `401` with an empty body means a bad token, or a header that does not start with `Bearer `.- `503` with `Retry-After` means ingest is saturated. Exporters retry this automatically.+- `413` means the batch exceeded 10 MB after decompression. Nothing was ingested. Lower the exporter's batch size.- `404` means the path under `/api` is wrong, for example `/api/v1/traces` instead of `/api/otel/v1/traces`.- `200` with an HTML body means the `/api` prefix is missing. The dashboard is served from the same origin, so its single-page fallback answers `/otel/v1/traces` with the dashboard itself and it looks like a success. Check that the response `Content-Type` is `application/json`, not `text/html`. The base is `/api/otel`, not `/otel`.@@ -328,7 +329,7 @@ curl -i -X POST https://<your-instance>/api/otel/v1/traces \| Logs exist but the endpoint's Logs card is empty | The endpoint was promoted from a non-root span, so it is keyed by span id and not by the trace id the logs carry | Search the trace id on the Logs page || A whole metric family is missing | It is an ExponentialHistogram or a Summary. Both are dropped without an error | Switch to explicit-bucket histograms, or convert in the Collector || Metrics arrive but every series looks identical | The distinguishing attribute is on the Resource and is not in the allowlist | Emit it as a data-point attribute instead. See [Metrics](/client/otel/metrics) |-| Large exports vanish and the exporter logs a 400 | The batch exceeded 10 MB after decompression, was truncated, and failed to parse | Lower the exporter's batch size |+| Large exports vanish and the exporter logs a 413 | The batch exceeded 10 MB after decompression and was rejected outright | Lower the exporter's batch size |**Turn on the SDK's own diagnostics.** Most silent client-side failures, including a mis-constructed exporter or processor, only surface here:diff --git a/docs/pages/protocol/index.md b/docs/pages/protocol/index.mdindex f8ab855c..0668dfaf 100644--- a/docs/pages/protocol/index.md+++ b/docs/pages/protocol/index.md@@ -52,6 +52,8 @@ Parse by splitting on the first `@`:The request body **should** be gzip-compressed JSON for the regular sync path. The backend's gzip middleware decompresses when `Content-Encoding: gzip` is present and passes the body through unchanged when it is absent. Browser SDKs use the uncompressed path on the page-unload (`pagehide`) flush so the request can dispatch synchronously inside the unload handler; `CompressionStream` is async and would yield control before the request leaves. Servers MUST therefore accept both compressed and uncompressed JSON on `/api/report`.+The body is capped **after decompression**, so gzip does not raise the ceiling. The default is 64 MB, tunable with `REPORT_MAX_BODY_MB`. A body over the cap answers `413` and is not ingested. See [Response Handling](#response-handling).+## Request Payload### Top-Level Structure@@ -469,8 +471,9 @@ When a trace is not sampled, do not send the trace or any associated exceptions.| Status | Meaning | Action ||--------|---------|--------|| `200` | Success | Remove sent frames from the send queue |-| `400` | Missing `Content-Encoding: gzip` or malformed JSON | Fix the request; do not retry as-is |+| `400` | Malformed JSON, or a body that claims `Content-Encoding: gzip` but is not gzip | Fix the request; do not retry as-is || `401` | Invalid or expired token | Check the project token configuration |+| `413` | Payload too large after decompression | Drop or split the batch. Do **not** retry it as-is: the same body fails every time, and re-queueing it wedges the client permanently || `429` | Rate limited | Back off and retry on the next interval || Any other | Server error or network issue | Retry on the next interval |diff --git a/docs/pages/server/_meta.json b/docs/pages/server/_meta.jsonindex f72ca600..e0596ec7 100644--- a/docs/pages/server/_meta.json+++ b/docs/pages/server/_meta.json@@ -11,5 +11,6 @@"storage": "Blob Storage","haloy": "Haloy","capacity-monitoring": "Capacity & Self-Monitoring",- "upgrade-1-8": "Upgrading to 1.8"+ "upgrade-1-8": "Upgrading to 1.8",+ "upgrade-request-limits": "Upgrade Notes: Request Limits"}diff --git a/docs/pages/server/all-in-one.mdx b/docs/pages/server/all-in-one.mdxindex 397c0144..7706e9a9 100644--- a/docs/pages/server/all-in-one.mdx+++ b/docs/pages/server/all-in-one.mdx@@ -62,8 +62,11 @@ All environment variables have working defaults, no configuration required. Over| `JWT_SECRET` | *(built-in default, **override in production**)* | Signs all authentication tokens: dashboard sessions and CLI/MCP device logins. The built-in default is public; set a strong unique secret (min 32 chars, `openssl rand -hex 32`) and keep it stable. See [CLI Authentication](/learn/cli-auth#set-a-strong-jwt_secret). || `GIN_MODE` | `release` | Gin framework mode || `APP_BASE_URL` | *(none)* | Public URL of your Traceway instance (e.g. `https://traceway.example.com`). Used for SDK setup instructions, email links, OAuth callback URLs, and the CLI/MCP OAuth issuer plus device-login verification URL. If unset, the device-auth and `/.well-known` endpoints derive it per-request from the `Host` / `X-Forwarded-*` headers; set it explicitly behind a proxy that rewrites `Host`. See [CLI Authentication](/learn/cli-auth). |-| `TRUSTED_PROXIES` | `127.0.0.1,::1` | Comma-separated IPs/CIDRs of reverse proxies whose `X-Forwarded-For` is trusted when resolving the client IP for the per-IP rate limits (login, registration, password reset, device auth, public status pages). The loopback default covers a proxy on the same host; add your proxy's address or network (e.g. `172.16.0.0/12` for Docker bridge networks) when it connects from elsewhere, or `*` to trust any peer — which lets clients spoof their IP and rotate out of the limits. |-| `TRUSTED_PROXY_HEADER` | *(none)* | A header trusted verbatim as the client IP, taking precedence over `TRUSTED_PROXIES` — e.g. `X-Real-IP` behind ingress-nginx or `CF-Connecting-IP` behind Cloudflare. Only safe when every request provably passes through a proxy that overwrites the header; a directly reachable backend lets clients set it freely. |+| `TRUSTED_PROXIES` | *(loopback)* | Comma-separated CIDRs of the proxies in front of Traceway whose `X-Forwarded-For` / `X-Real-IP` is trusted for the real client IP. Every per-IP rate limit keys on it, and `/api/report` stores it as each session's `client.ip`. Unset trusts loopback only, so a proxy container on a Docker network or an in-cluster ingress has to be listed before its header counts. A value replaces the default, so list every hop between the client and this server, not just the outermost one: naming only a CDN leaves a private ingress untrusted and turns `X-Forwarded-For` off entirely. `*` trusts every peer and lets clients spoof their address. Until the proxy is listed, every visitor shares one rate-limit bucket keyed on the proxy, and the server logs a warning the first time it sees a forwarding header from a peer it does not trust. |+| `TRUSTED_PROXY_HEADER` | *(none)* | A header taken as the client IP on every request, ahead of `TRUSTED_PROXIES`: `X-Real-IP` behind ingress-nginx, `CF-Connecting-IP` behind Cloudflare. Only safe when every request provably passes through a proxy that overwrites the header; a directly reachable backend lets clients set it freely. |+| `REPORT_MAX_BODY_MB` | `64` | Cap on the decompressed `/api/report` and `/api/profiles/ingest` body in megabytes. Gzip does not raise it. A body over the cap answers `413`. |+| `INGEST_MAX_CONCURRENT` | *(2 x CPU cores, min 4)* | Telemetry ingest requests processed at once (`/api/report`, `/api/profiles/ingest`, `/api/otel/*`). Excess requests wait up to `INGEST_ADMISSION_WAIT_SECONDS` (default 5) and then get `503` with `Retry-After`; past a bounded waiting room (four times the capacity, at least 16) they are turned away immediately. Each admitted request can hold a decoded body, so on a memory-capped container size this from the memory limit rather than the CPU count. |+| `UPLOAD_MAX_CONCURRENT` | `4` | Source map and symbol uploads processed at once. Each holds a whole file in memory while it is parsed. Excess uploads wait 30 seconds and then get `503`. Separate from the ingest pool so a CI burst cannot starve telemetry. || `STORAGE_TYPE` | `local` | Blob storage backend for source maps, session recordings, and AI traces: `local` or `s3`. See [Blob Storage](/server/storage) for the S3 variables. || `STORAGE_PATH` | `./storage` | Folder for local blob storage, `/app/storage` in this image. Mount a volume there. Ignored when `STORAGE_TYPE=s3`. || `SESSION_RECORDING_RETENTION_DAYS` | `30` | Days to keep session recording files on disk under `STORAGE_PATH/recordings/`. Files older than the TTL are deleted hourly and on startup. No effect when `STORAGE_TYPE=s3`. Set to `0` to disable. |diff --git a/docs/pages/server/docker-compose.mdx b/docs/pages/server/docker-compose.mdxindex 52f0fb15..d19cc37e 100644--- a/docs/pages/server/docker-compose.mdx+++ b/docs/pages/server/docker-compose.mdx@@ -100,8 +100,11 @@ Edit the environment variables in your compose file as needed:| `POSTGRES_SSLMODE` | `disable` | PostgreSQL SSL mode || `JWT_SECRET` | `change-this-to-a-secure-secret-at-least-32-chars` | Signs all authentication tokens (dashboard sessions and CLI/MCP device logins). Replace the placeholder with a strong, unique secret of at least 32 chars (`openssl rand -hex 32`) and keep it stable. See [CLI Authentication](/learn/cli-auth#set-a-strong-jwt_secret). || `APP_BASE_URL` | *(none)* | Public URL of your Traceway instance (e.g. `https://traceway.example.com`). Used for SDK setup instructions, email links, OAuth callback URLs, and the CLI/MCP OAuth issuer plus device-login verification URL. If unset, the device-auth and `/.well-known` endpoints derive it per-request from the `Host` / `X-Forwarded-*` headers; set it explicitly behind a proxy that rewrites `Host`. See [CLI Authentication](/learn/cli-auth). |-| `TRUSTED_PROXIES` | `127.0.0.1,::1` | Comma-separated IPs/CIDRs of reverse proxies whose `X-Forwarded-For` is trusted when resolving the client IP for the per-IP rate limits (login, registration, password reset, device auth, public status pages). The loopback default covers a proxy on the same host; add your proxy's address or network (e.g. `172.16.0.0/12` for Docker bridge networks) when it connects from elsewhere, or `*` to trust any peer — which lets clients spoof their IP and rotate out of the limits. |-| `TRUSTED_PROXY_HEADER` | *(none)* | A header trusted verbatim as the client IP, taking precedence over `TRUSTED_PROXIES` — e.g. `X-Real-IP` behind ingress-nginx or `CF-Connecting-IP` behind Cloudflare. Only safe when every request provably passes through a proxy that overwrites the header; a directly reachable backend lets clients set it freely. |+| `TRUSTED_PROXIES` | *(loopback)* | Comma-separated CIDRs of the proxies in front of Traceway whose `X-Forwarded-For` / `X-Real-IP` is trusted for the real client IP. Every per-IP rate limit keys on it, and `/api/report` stores it as each session's `client.ip`. Unset trusts loopback only, so a proxy container on a Docker network or an in-cluster ingress has to be listed before its header counts. A value replaces the default, so list every hop between the client and this server, not just the outermost one: naming only a CDN leaves a private ingress untrusted and turns `X-Forwarded-For` off entirely. `*` trusts every peer and lets clients spoof their address. Until the proxy is listed, every visitor shares one rate-limit bucket keyed on the proxy, and the server logs a warning the first time it sees a forwarding header from a peer it does not trust. |+| `TRUSTED_PROXY_HEADER` | *(none)* | A header taken as the client IP on every request, ahead of `TRUSTED_PROXIES`: `X-Real-IP` behind ingress-nginx, `CF-Connecting-IP` behind Cloudflare. Only safe when every request provably passes through a proxy that overwrites the header; a directly reachable backend lets clients set it freely. |+| `REPORT_MAX_BODY_MB` | `64` | Cap on the decompressed `/api/report` and `/api/profiles/ingest` body in megabytes. Gzip does not raise it. A body over the cap answers `413`. |+| `INGEST_MAX_CONCURRENT` | *(2 x CPU cores, min 4)* | Telemetry ingest requests processed at once (`/api/report`, `/api/profiles/ingest`, `/api/otel/*`). Excess requests wait up to `INGEST_ADMISSION_WAIT_SECONDS` (default 5) and then get `503` with `Retry-After`; past a bounded waiting room (four times the capacity, at least 16) they are turned away immediately. Each admitted request can hold a decoded body, so on a memory-capped container size this from the memory limit rather than the CPU count. |+| `UPLOAD_MAX_CONCURRENT` | `4` | Source map and symbol uploads processed at once. Each holds a whole file in memory while it is parsed. Excess uploads wait 30 seconds and then get `503`. Separate from the ingest pool so a CI burst cannot starve telemetry. || `STORAGE_TYPE` | `local` | Blob storage backend for source maps, session recordings, and AI traces: `local` or `s3`. See [Blob Storage](/server/storage) for the S3 variables. || `STORAGE_PATH` | `./storage` | Folder for local blob storage, `/app/storage` in the minimal image. Mount a volume there. Ignored when `STORAGE_TYPE=s3`. || `SESSION_RECORDING_RETENTION_DAYS` | `30` | Days to keep session recording files on disk under `STORAGE_PATH/recordings/`. A worker deletes older files hourly and on startup. No effect when `STORAGE_TYPE=s3`. Set to `0` to disable. |diff --git a/docs/pages/server/haloy.mdx b/docs/pages/server/haloy.mdxindex 4352d5e4..f19d464d 100644--- a/docs/pages/server/haloy.mdx+++ b/docs/pages/server/haloy.mdx@@ -164,6 +164,11 @@ Point your SDK's connection string at your Traceway domain:| Variable | Default | Description ||----------|---------|-------------|| `APP_BASE_URL` | `https://traceway.yourdomain.com` | Public URL of the Traceway instance. Also the CLI/MCP OAuth issuer and device-login verification URL. See [CLI Authentication](/learn/cli-auth). |+| `TRUSTED_PROXIES` | *(loopback)* | Comma-separated CIDRs of the proxies in front of Traceway whose `X-Forwarded-For` / `X-Real-IP` is trusted for the real client IP. Every per-IP rate limit keys on it, and `/api/report` stores it as each session's `client.ip`. Unset trusts loopback only, so a proxy container on a Docker network or an in-cluster ingress has to be listed before its header counts. A value replaces the default, so list every hop between the client and this server, not just the outermost one: naming only a CDN leaves a private ingress untrusted and turns `X-Forwarded-For` off entirely. `*` trusts every peer and lets clients spoof their address. Until the proxy is listed, every visitor shares one rate-limit bucket keyed on the proxy, and the server logs a warning the first time it sees a forwarding header from a peer it does not trust. Haloy fronts the app with its own proxy, so list the range that proxy connects from (and your CDN's, if any). |+| `TRUSTED_PROXY_HEADER` | *(none)* | A header taken as the client IP on every request, ahead of `TRUSTED_PROXIES`: `X-Real-IP` behind ingress-nginx, `CF-Connecting-IP` behind Cloudflare. Only safe when every request provably passes through a proxy that overwrites the header; a directly reachable backend lets clients set it freely. |+| `REPORT_MAX_BODY_MB` | `64` | Cap on the decompressed `/api/report` and `/api/profiles/ingest` body in megabytes. Gzip does not raise it. A body over the cap answers `413`. |+| `INGEST_MAX_CONCURRENT` | *(2 x CPU cores, min 4)* | Telemetry ingest requests processed at once (`/api/report`, `/api/profiles/ingest`, `/api/otel/*`). Excess requests wait up to `INGEST_ADMISSION_WAIT_SECONDS` (default 5) and then get `503` with `Retry-After`; past a bounded waiting room (four times the capacity, at least 16) they are turned away immediately. Each admitted request can hold a decoded body, so on a memory-capped container size this from the memory limit rather than the CPU count. |+| `UPLOAD_MAX_CONCURRENT` | `4` | Source map and symbol uploads processed at once. Each holds a whole file in memory while it is parsed. Excess uploads wait 30 seconds and then get `503`. Separate from the ingest pool so a CI burst cannot starve telemetry. || `CLICKHOUSE_SERVER` | `clickhouse:9000` | ClickHouse host:port || `CLICKHOUSE_DATABASE` | `traceway` | ClickHouse database name || `CLICKHOUSE_USERNAME` | `default` | ClickHouse username |diff --git a/docs/pages/server/minimal.mdx b/docs/pages/server/minimal.mdxindex b721b60e..658cfd11 100644--- a/docs/pages/server/minimal.mdx+++ b/docs/pages/server/minimal.mdx@@ -58,8 +58,11 @@ Then use `traceway:minimal` instead of `ghcr.io/tracewayapp/traceway:minimal` in| `POSTGRES_SSLMODE` | PostgreSQL SSL mode | `disable` || `JWT_SECRET` | Signs all authentication tokens (dashboard sessions and CLI/MCP device logins). Use a strong, unique secret of at least 32 chars; keep it stable. See [CLI Authentication](/learn/cli-auth#set-a-strong-jwt_secret). | `your-secret-here` || `APP_BASE_URL` | Public URL of your instance. Used for SDK setup instructions, email links, OAuth callback URLs, and the CLI/MCP OAuth issuer plus device-login verification URL. If unset, derived per-request from `Host` / `X-Forwarded-*`. | `https://traceway.example.com` |-| `TRUSTED_PROXIES` | Comma-separated IPs/CIDRs of reverse proxies whose `X-Forwarded-For` is trusted when resolving the client IP for the per-IP rate limits. Defaults to loopback (`127.0.0.1,::1`); add your proxy's address or network when it connects from elsewhere, or `*` to trust any peer (lets clients spoof their IP past the limits). | `172.16.0.0/12` |-| `TRUSTED_PROXY_HEADER` | A header trusted verbatim as the client IP, taking precedence over `TRUSTED_PROXIES` — e.g. behind ingress-nginx or Cloudflare. Only safe when every request provably passes through a proxy that overwrites the header. | `X-Real-IP` |+| `TRUSTED_PROXIES` | Comma-separated CIDRs of the proxies in front of Traceway whose `X-Forwarded-For` / `X-Real-IP` is trusted for the real client IP. Every per-IP rate limit keys on it, and `/api/report` stores it as each session's `client.ip`. Unset trusts loopback only, so a proxy container on a Docker network or an in-cluster ingress has to be listed before its header counts. A value replaces the default, so list every hop between the client and this server, not just the outermost one: naming only a CDN leaves a private ingress untrusted and turns `X-Forwarded-For` off entirely. `*` trusts every peer and lets clients spoof their address. Until the proxy is listed, every visitor shares one rate-limit bucket keyed on the proxy, and the server logs a warning the first time it sees a forwarding header from a peer it does not trust. | `10.0.0.0/8,192.168.0.0/16` |+| `TRUSTED_PROXY_HEADER` | A header taken as the client IP on every request, ahead of `TRUSTED_PROXIES`: `X-Real-IP` behind ingress-nginx, `CF-Connecting-IP` behind Cloudflare. Only safe when every request provably passes through a proxy that overwrites the header; a directly reachable backend lets clients set it freely. | `X-Real-IP` |+| `REPORT_MAX_BODY_MB` | Cap on the decompressed `/api/report` and `/api/profiles/ingest` body in megabytes. Gzip does not raise it. A body over the cap answers `413`. | `64` |+| `INGEST_MAX_CONCURRENT` | Telemetry ingest requests processed at once (`/api/report`, `/api/profiles/ingest`, `/api/otel/*`). Defaults to 2 x CPU cores with a floor of 4. Excess requests wait up to `INGEST_ADMISSION_WAIT_SECONDS` (default 5) and then get `503` with `Retry-After`; past a bounded waiting room they are turned away immediately. Each admitted request can hold a decoded body, so on a memory-capped container size this from the memory limit. | `8` |+| `UPLOAD_MAX_CONCURRENT` | Source map and symbol uploads processed at once. Each holds a whole file in memory while it is parsed. Excess uploads wait 30 seconds and then get `503`. Separate from the ingest pool so a CI burst cannot starve telemetry. | `4` || `STORAGE_TYPE` | Blob storage backend for source maps, session recordings, and AI traces: `local` (default) or `s3`. See [Blob Storage](/server/storage) for the S3 variables. | `local` || `STORAGE_PATH` | Folder for local blob storage. Defaults to `./storage`, which is `/app/storage` in this image. Mount a volume there. Ignored when `STORAGE_TYPE=s3`. | `/app/storage` || `SESSION_RECORDING_RETENTION_DAYS` | Days to keep on-disk session recordings under `STORAGE_PATH/recordings/`. Defaults to `30`. Worker runs hourly and on startup. `0` disables; no effect when `STORAGE_TYPE=s3`. | `30` |diff --git a/docs/pages/server/sqlite.mdx b/docs/pages/server/sqlite.mdxindex 96e41d24..ec72c527 100644--- a/docs/pages/server/sqlite.mdx+++ b/docs/pages/server/sqlite.mdx@@ -62,8 +62,11 @@ If `STORAGE_TYPE` is unset (or set to `local`), blobs are written to `/data/stor|----------|---------|-------------|| `JWT_SECRET` | *(required)* | Signs all authentication tokens (dashboard sessions and CLI/MCP device logins). Use a strong, unique secret of at least 32 characters (`openssl rand -hex 32`) and keep it stable. See [CLI Authentication](/learn/cli-auth#set-a-strong-jwt_secret). || `APP_BASE_URL` | *(unset)* | Public URL of your instance. Used for SDK setup snippets, password-reset emails, OAuth callback URLs, and the CLI/MCP OAuth issuer plus device-login verification URL. If unset, the device-auth and `/.well-known` endpoints derive it per-request from the `Host` / `X-Forwarded-*` headers. See [CLI Authentication](/learn/cli-auth) |-| `TRUSTED_PROXIES` | `127.0.0.1,::1` | Comma-separated IPs/CIDRs of reverse proxies whose `X-Forwarded-For` is trusted when resolving the client IP for the per-IP rate limits (login, registration, password reset, device auth, public status pages). The loopback default covers a proxy on the same host; add your proxy's address or network (e.g. `172.16.0.0/12` for Docker bridge networks) when it connects from elsewhere, or `*` to trust any peer — which lets clients spoof their IP and rotate out of the limits |-| `TRUSTED_PROXY_HEADER` | *(none)* | A header trusted verbatim as the client IP, taking precedence over `TRUSTED_PROXIES` — e.g. `X-Real-IP` behind ingress-nginx or `CF-Connecting-IP` behind Cloudflare. Only safe when every request provably passes through a proxy that overwrites the header |+| `TRUSTED_PROXIES` | *(loopback)* | Comma-separated CIDRs of the proxies in front of Traceway whose `X-Forwarded-For` / `X-Real-IP` is trusted for the real client IP. Every per-IP rate limit keys on it, and `/api/report` stores it as each session's `client.ip`. Unset trusts loopback only, so a proxy container on a Docker network or an in-cluster ingress has to be listed before its header counts. A value replaces the default, so list every hop between the client and this server, not just the outermost one: naming only a CDN leaves a private ingress untrusted and turns `X-Forwarded-For` off entirely. `*` trusts every peer and lets clients spoof their address. Until the proxy is listed, every visitor shares one rate-limit bucket keyed on the proxy, and the server logs a warning the first time it sees a forwarding header from a peer it does not trust. |+| `TRUSTED_PROXY_HEADER` | *(none)* | A header taken as the client IP on every request, ahead of `TRUSTED_PROXIES`: `X-Real-IP` behind ingress-nginx, `CF-Connecting-IP` behind Cloudflare. Only safe when every request provably passes through a proxy that overwrites the header; a directly reachable backend lets clients set it freely. |+| `REPORT_MAX_BODY_MB` | `64` | Cap on the decompressed `/api/report` and `/api/profiles/ingest` body in megabytes. Gzip does not raise it. A body over the cap answers `413`. |+| `INGEST_MAX_CONCURRENT` | *(2 x CPU cores, min 4)* | Telemetry ingest requests processed at once (`/api/report`, `/api/profiles/ingest`, `/api/otel/*`). Excess requests wait up to `INGEST_ADMISSION_WAIT_SECONDS` (default 5) and then get `503` with `Retry-After`; past a bounded waiting room (four times the capacity, at least 16) they are turned away immediately. Each admitted request can hold a decoded body, so on a memory-capped container size this from the memory limit rather than the CPU count. |+| `UPLOAD_MAX_CONCURRENT` | `4` | Source map and symbol uploads processed at once. Each holds a whole file in memory while it is parsed. Excess uploads wait 30 seconds and then get `503`. Separate from the ingest pool so a CI burst cannot starve telemetry. || `SQLITE_PATH` | `/data/traceway.db` | Main SQLite file. Telemetry DB is derived as `<path>_telemetry.db` || `STORAGE_TYPE` | `local` | Blob storage backend. `local` or `s3` || `STORAGE_PATH` | `/data/storage` | Folder for local blob storage. Ignored when `STORAGE_TYPE=s3` |diff --git a/docs/pages/server/upgrade-request-limits.mdx b/docs/pages/server/upgrade-request-limits.mdxnew file mode 100644index 00000000..2df8a057--- /dev/null+++ b/docs/pages/server/upgrade-request-limits.mdx@@ -0,0 +1,69 @@+# Upgrade notes: request limits and trusted proxies++This release hardens the server against slow, oversized, and unfair requests. Most instances upgrade without changing anything. The sections below cover the changes that can affect an existing deployment, roughly in order of how likely they are to matter.++## Trusted proxies++Traceway used to accept `X-Forwarded-For` from any peer. It now trusts only loopback by default. A reverse proxy running on the same host as a plain process keeps working; anything else has to be listed. Private networks are deliberately not trusted: on a Docker or Kubernetes network every peer has a private address, including the clients themselves when the port is published directly, so trusting those ranges would let any of them forge the header past the per-IP limits.++You are affected if any hop between your users and Traceway is not on loopback:++- A proxy container on the same Docker network (nginx, Caddy, Traefik).+- An in-cluster ingress. List its pod range.+- A CDN such as Cloudflare talking to Traceway directly.+- A CDN in front of a private ingress (`CDN -> nginx -> Traceway`). List both.+- An external load balancer with a public IP.++Symptoms after the upgrade:++- Public status pages answer `429` once one proxy address exceeds 30 requests a minute. Acknowledge links and CLI device logins hit their per-IP limits the same way.+- Sessions show the proxy's address as `client.ip`.+- The server log contains a line starting with `X-Forwarded-For received from <address>, which is not in TRUSTED_PROXIES`. That address is the hop you need to list.++Fix: set `TRUSTED_PROXIES` to every hop in the chain, not just the outermost one. The value replaces the defaults, so include the private range your ingress uses as well:++```+TRUSTED_PROXIES=173.245.48.0/20,103.21.244.0/22,10.0.0.0/8+```++`*` trusts every peer, which is the behaviour before this setting existed and lets clients spoof their address. The full list of Cloudflare ranges is at https://www.cloudflare.com/ips/.++`TRUSTED_PROXY_HEADER` is the alternative for a proxy that sets a dedicated header: `X-Real-IP` behind ingress-nginx, `CF-Connecting-IP` behind Cloudflare. Its value is taken as the client IP on every request, ahead of `TRUSTED_PROXIES`. Only use it when nothing can reach the backend without passing through that proxy, because a directly reachable backend lets clients set the header themselves.++## Per-IP limits on the authentication endpoints++The unauthenticated endpoints now have fixed-window per-IP limits: `/login` and `/register` 10 a minute, SSO start and callback 20, the token endpoints (`/auth/token`, `/auth/device/token`) 60 shared, `/auth/logout` 20, `/forgot-password` 5, password-reset validation 30 and consumption 10, invitation lookup 30 and acceptance 10. Over the limit answers `429` with a `Retry-After` header. The one exception is a `device_code` poll on the token endpoints, which answers the RFC 8628 shape `400 {"error":"slow_down"}` so device-flow pollers (the CLI, MCP clients) back off and keep polling instead of failing the login; refresh and authorization-code grants get the `429` like everything else, since `slow_down` means nothing to them. The anonymous auth endpoints also cap their bodies at 64 KB. Limits are per process and reset on restart.++They key on the client IP, which is why the trusted-proxy setting above matters: behind a proxy Traceway does not trust, every user shares one bucket of 10 logins a minute. Teams behind one office NAT share a bucket by design; raise the numbers in `routes.go` if that is too tight for you.++## Upload limits++Source map and symbol uploads now have a total cap of 250 MB per request. A larger body answers `413`. A request with more than 1000 files (the number of parts Go's multipart reader accepts, so nothing that uploaded before is refused now) answers `422` instead of the old `400`. `@tracewayapp/sourcemap-upload` posts a whole build, maps and bundles, in one request; a build over 1000 files needs to be uploaded in batches. Chunked uploads without a `Content-Length` are held to the same total.++Concurrent uploads are limited by `UPLOAD_MAX_CONCURRENT` (default 4). Excess uploads wait 30 seconds and then get `503`; CI retries handle that.++## Ingest body caps++`/api/report` and `/api/profiles/ingest` accept at most 64 MB after decompression, tunable with `REPORT_MAX_BODY_MB`. A larger body answers `413` and is not ingested. Gzip does not raise the cap.++OTLP endpoints keep their 10 MB cap, but an oversized batch is now rejected with `413` instead of being truncated into a `400`. Exporters treat both as permanent and drop the batch, so the change is only in what the log says.++## Ingest concurrency++The ingest admission gate (`INGEST_MAX_CONCURRENT`, default 2 x CPU cores with a floor of 4) now bounds its waiting room as well as its active slots. Requests that cannot get a slot wait in a room of four times the capacity (at least 16); beyond that they are turned away immediately with `503` and `Retry-After`, which SDKs and OTLP exporters retry. Before, every waiting request parked a goroutine, a timer and a connection for the full wait window with no limit.++## Server timeouts++The server now closes connections whose headers or body do not finish arriving. Headers must arrive within 15 seconds. Every request body has a progress deadline: no more than 30 seconds between bytes, 10 minutes in total. Routes that take large bodies have their own budget: uploads may take up to 30 minutes as long as data keeps flowing, `/api/report` up to 5 minutes, OTLP up to 2 minutes. A client that averages under 1 KB/s after the first 20 seconds is cut off with `408`; a body that has fully arrived is never rejected for being slow. Idle keep-alive connections are closed after 2 minutes.++## Public status pages in iframes++Every response now carries `X-Frame-Options: DENY` and a `frame-ancestors 'none'` policy, except public status pages at `/status/<slug>`, which stay embeddable. A status page served at the root of a custom domain is not exempt, because that same root serves the dashboard on every hostname the backend answers on. To embed a status page, use its `/status/<slug>` URL.++## Metric query limits++`POST /api/metrics/query` enforces: at most 50 queries per request and 20 tag filters per query, a time range of at most 731 days with `to` after `from` (`422`), buckets widened so no series exceeds 2000 points, at most 200 groups per query (the result carries `truncatedGroups: true` when cut), and a 30-second deadline for the whole request that answers `504`. Dashboards built in the UI stay well inside these limits.++## Transactional requests++Endpoints that write to the main database now read the whole request body (up to 8 MB) before opening their transaction. Bodies above that answer `413`. No documented endpoint takes a body that large; the biggest is dashboard import at 5 MB.diff --git a/docs/pages/symbolicator/android.mdx b/docs/pages/symbolicator/android.mdxindex baeeb697..983b9d84 100644--- a/docs/pages/symbolicator/android.mdx+++ b/docs/pages/symbolicator/android.mdx@@ -97,7 +97,7 @@ Content-Type: multipart/form-data| Field | Required | Notes ||---|---|---|-| `files` | yes | The `mapping.txt` file. 200 MB per file. |+| `files` | yes | The `mapping.txt` file. 200 MB per file, 250 MB and 1000 files total per request (a larger body answers `413`, too many files `422`). || `proguard_uuid` | yes | The build UUID. The mapping is stored as `<uuid>.txt`. |The backend recognizes an R8 mapping by its header and class lines. The stored key is:diff --git a/docs/pages/symbolicator/dart.mdx b/docs/pages/symbolicator/dart.mdxindex eec5fb31..983b9cf7 100644--- a/docs/pages/symbolicator/dart.mdx+++ b/docs/pages/symbolicator/dart.mdx@@ -87,7 +87,7 @@ Content-Type: multipart/form-data| Field | Required | Notes ||---|---|---|-| `files` | yes | One or more `.symbols` files. Non-`.symbols` entries are ignored. 200 MB per file. |+| `files` | yes | One or more `.symbols` files. Non-`.symbols` entries are ignored. 200 MB per file, 250 MB and 1000 files total per request (a larger body answers `413`, too many files `422`). || `arch` | sometimes | The architecture token (`arm64`, `arm`, `x64`, `ia32`). Optional when the filename ends in `-<arch>.symbols`, which Flutter's output already does. || `debug_id` | sometimes | Required only when the file has no build-id note (Apple). When the file does have one, this must match it if provided. |diff --git a/docs/pages/symbolicator/ios.mdx b/docs/pages/symbolicator/ios.mdxindex a281e69d..8d19289c 100644--- a/docs/pages/symbolicator/ios.mdx+++ b/docs/pages/symbolicator/ios.mdx@@ -64,7 +64,7 @@ curl -H "Authorization: Bearer $TRACEWAY_UPLOAD_TOKEN" \| Field | Required | Notes ||---|---|---|-| `files` | yes | One or more Mach-O dSYM binaries. Repeatable to upload several at once. 200 MB per file. |+| `files` | yes | One or more Mach-O dSYM binaries. Repeatable to upload several at once. 200 MB per file, 250 MB and 1000 files total per request (a larger body answers `413`, too many files `422`). |Unlike the Dart upload, iOS needs no `arch` or `debug_id` form fields. The backend auto-detects the Mach-O, reads the `LC_UUID` straight from the binary, and for a fat dSYM stores one entry per architecture slice. The stored key is content-addressed by UUID:diff --git a/helm/traceway/templates/configmap.yaml b/helm/traceway/templates/configmap.yamlindex 6c14589f..5ea7cdd3 100644--- a/helm/traceway/templates/configmap.yaml+++ b/helm/traceway/templates/configmap.yaml@@ -53,6 +53,15 @@ data:{{- if .Values.app.trustedProxyHeader }}TRUSTED_PROXY_HEADER: {{ .Values.app.trustedProxyHeader | quote }}{{- end }}+ {{- if .Values.app.reportMaxBodyMB }}+ REPORT_MAX_BODY_MB: {{ .Values.app.reportMaxBodyMB | quote }}+ {{- end }}+ {{- if .Values.app.ingestMaxConcurrent }}+ INGEST_MAX_CONCURRENT: {{ .Values.app.ingestMaxConcurrent | quote }}+ {{- end }}+ {{- if .Values.app.uploadMaxConcurrent }}+ UPLOAD_MAX_CONCURRENT: {{ .Values.app.uploadMaxConcurrent | quote }}+ {{- end }}{{- if .Values.app.apiOnly }}API_ONLY: {{ .Values.app.apiOnly | quote }}{{- end }}diff --git a/helm/traceway/values.yaml b/helm/traceway/values.yamlindex 4c45ecf2..6557c87e 100644--- a/helm/traceway/values.yaml+++ b/helm/traceway/values.yaml@@ -73,6 +73,14 @@ app:trustedProxyHeader: ""# Optional. Set to "true" to serve only the API without the embedded frontend.apiOnly: ""+ # Optional. Cap on the decompressed /api/report body in megabytes. Default 64.+ reportMaxBodyMB: ""+ # Optional. Telemetry ingest requests processed at once. Default is 2x CPU+ # count with a floor of 4. Each admitted request can hold a decoded body, so+ # on a memory-capped container size this from the memory limit instead.+ ingestMaxConcurrent: ""+ # Optional. Source map and symbol uploads processed at once. Default 4.+ uploadMaxConcurrent: ""# ClickHouse configuration — required when variant=minimalclickhouse:diff --git a/skills/traceway-setup/SKILL.md b/skills/traceway-setup/SKILL.mdindex 9700c657..cecb7cc1 100644--- a/skills/traceway-setup/SKILL.md+++ b/skills/traceway-setup/SKILL.md@@ -318,7 +318,7 @@ SDKs append `/v1/traces`, `/v1/metrics`, `/v1/logs` to the endpoint automaticallWhen Step 3's apply script wrote the env files, the plan's variables already hold real values locally, so wire the exporter to exactly those names (e.g. `Authorization=Bearer ${TRACEWAY_BACKEND_TOKEN}`) and run live verification immediately. The script writes only each project's one plan variable: companion variables the integration reads, like `TRACEWAY_URL` for the instance URL, are yours to add to the same env file now. On the manual fallback the user populates the variables first.-Constraints: OTLP/HTTP only (protobuf or JSON). OTLP/gRPC is not supported, there is no listener on port 4317. `Content-Encoding: gzip` is fine. The body is read up to 10 MB and the rest is truncated, so an oversized batch answers `400 failed to unmarshal` rather than `413`. A wrong or missing token answers `401`, and any other path answers `404`. All three are invisible from the application: most SDKs log exporter failures at debug level only, so when nothing arrives, turn the SDK's own diagnostic logging on first.+Constraints: OTLP/HTTP only (protobuf or JSON). OTLP/gRPC is not supported, there is no listener on port 4317. `Content-Encoding: gzip` is fine. The body is capped at 10 MB after decompression, and an oversized batch is rejected outright with `413` and `{"error":"request body exceeds the 10MB limit"}`. Nothing is ingested partially. A wrong or missing token answers `401`, and any other path answers `404`. All three are invisible from the application: most SDKs log exporter failures at debug level only, so when nothing arrives, turn the SDK's own diagnostic logging on first.### Node.js examplediff --git a/skills/traceway-setup/data-model.md b/skills/traceway-setup/data-model.mdindex e01cc792..d2e2c72c 100644--- a/skills/traceway-setup/data-model.md+++ b/skills/traceway-setup/data-model.md@@ -12,7 +12,7 @@ This is the framework-agnostic reference for connecting any OpenTelemetry projec| Profiles | `POST https://<instance>/api/otel/v1development/profiles` (OTLP profiles, the development signal) || Auth | `Authorization: Bearer <project-token>` header on every request || Encoding | OTLP/HTTP: both `application/x-protobuf` and `application/json` are accepted. OTLP/gRPC is NOT supported. |-| Compression | `Content-Encoding: gzip` is supported (exporters default to no compression; enabling `compression: gzip` is fine). Max request body: 10 MB. |+| Compression | `Content-Encoding: gzip` is supported (exporters default to no compression; enabling `compression: gzip` is fine). Max request body: 10 MB after decompression; larger bodies answer `413`. |Anything whose `Content-Type` is not `application/x-protobuf` or `application/protobuf` is parsed as OTLP/JSON, so a missing `Content-Type` header still works for JSON exporters. When ingest is saturated the endpoints answer `503` with a `Retry-After` header. That is a back-pressure signal to retry, not a misconfiguration.