Skip to content

The only tool you need to know what is happening and how to fix it.

SLOs (#422)

narrow-adventure591a08cf405d
diff --git a/CLAUDE.md b/CLAUDE.md
index ec6c26d6..274f7f0d 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -296,6 +296,9 @@ EMAIL_PREVIEW_ENABLED=false # "true" registers GET /api/email-preview[
NOTIFICATION_POLL_SECONDS=60 # polled rule evaluation interval; minimum 5, invalid values fall back to 60
ONCALL_POLL_SECONDS=30 # on-call escalation worker interval; minimum 5, invalid values fall back to 30. Kept separate from NOTIFICATION_POLL_SECONDS so raising rule-evaluation intervals never delays paging. A buffered Wake() channel makes freshly opened pages notify L1 near-instantly regardless of this interval.
OUTBOX_POLL_SECONDS=15 # notification outbox drain interval; minimum 5, invalid values fall back to 15. The outbox (backend/app/outbox, notification_outbox table in the main DB) is the persist-then-send layer for ALL notifications: rule dispatch and the escalator only enqueue (with an adapter-config snapshot) inside their transactions; the drain worker sends with retries (backoff 1m/5m/15m/60m, 5 attempts, then terminal failed + CaptureException). Crash-safe at-least-once: stale 'sending' rows are reclaimed after 5 min, cancelled rows can never resurrect (guarded status transitions), ack/resolve cancels queued page deliveries via outbox.CancelByKey. Cooldown and event-rule dedup record at enqueue commit (the durable promise), and fired_notifications is written at the terminal outcome. /api/health/deep exposes an `outbox` block; `traceway.outbox.*` metrics are emitted when monitoring is on; terminal rows are pruned daily (sent/cancelled 7d, failed 30d).
+SLO_POLL_SECONDS=60 # SLO worker interval (backend/app/slo): recomputes the last 10 minutes of every enabled SLO into the telemetry `slo_buckets` table (every 15 minutes, in a slot derived from the SLO id and decided against its persisted `computed_at` (`passDue`, `slo/schedule.go`) so passes are spread over ticks and a restart or shard move neither skips nor bunches them, the other kinds also recompute the last 75 minutes, for long requests and retried exports, and session and task SLOs their exception slack plus 25 minutes, since a crash or failure lands after the bucket of its start; once a day session SLOs recompute the last 72 hours through the gap fill, for native crashes uploaded on the next launch), computes `SLO_COMPUTE_WORKERS` (default 4, max 64) SLOs at a time with a per-SLO `SLO_COMPUTE_TIMEOUT_SECONDS` (default 30; a timed-out SLO keeps its old snapshot and resumes from `last_bucket_at` on the next tick), works through the SLOs in groups of 200 so memory stays bounded and each group's buckets go out in one batched insert stamped at write time (ClickHouse makes a part of every insert); the window sum is incremental (`slo/snapshot_cache.go`): the part of the window before an anchor 7h back (73h for session SLOs) is cached in memory and only loses the buckets that slide out, the recent part is read every tick, and the whole window is re-read when the cache is missing, the revision or `computed_at` differs from what this process wrote (an edit, or another instance computed it), a bucket was written before the anchor, or once an hour in the SLO's slot; an SLO with no `last_bucket_at` (new, or reset by a definition change) has its buckets deleted by the worker before anything is written, never by the controller, so a delete cannot land after the new buckets; backfills older history in 6h chunks (24h for metric SLIs so ClickHouse reads the 1m rollups) under one 5s budget shared by every SLO of the tick (start rotated per tick), records an outage longer than the 24h catch-up as a `gap_from`/`gap_to` range it fills under the same budget, writes the snapshot columns on `slos` only while the row is still at the `revision` it loaded (an edit during a tick makes it drop its result; the reset of a definition change clears the buckets on the next tick), on Postgres splits the SLOs over 64 shards (`id % 64`) leased per instance through `slo_workers` (heartbeat) and `slo_shard_leases` (pg/0157, pg/0158; `slo/shards.go`): each tick an instance renews its leases, releases any above its fair share (`ceil(64 / live instances)`) and claims free or expired ones, renews again between groups, leases last max(5 min, 3x the poll) and a stopped instance's shards move once it ages out, shard 0's owner runs the daily prune; SQLite owns all shards, counts metric slices without samples as bad only from `sli_since` (set on create and on every definition change), rounds slo_burn_rate windows up to whole slices and clears it when either window drops below the threshold, slo_budget_consumed reads the snapshot's `budget_remaining` instead of re-summing the window, rule state moves through `NotificationRuleStateRepository.Transition` (a conditional update, so two instances never both notify), both slo_* rules hold their state while the SLO has no `last_bucket_at` (a definition change being recomputed) so a reset never sends a false recovery, a recovery notice does not record a cooldown fire, prunes buckets older than 120 days daily, and reports the `slo` block of `/api/health/deep` (totals incl. timeouts, `runningForSec`, owned shards and instances, the last tick's phase timings with its slowest SLO, and `freshness` read from the main DB: enabled, stale > 5 min, max staleness, backfill pending), which `monitoring.StartSLOReporter` emits every 30s as `traceway.slo.*` metrics when self-monitoring is on (documented in `docs/pages/server/capacity-monitoring.mdx`); minimum 5, invalid values fall back to 60. `slo.Wake()` runs a tick right after an SLO is created or changed. Buckets are deliberately NOT in the telemetry retention list; slo_* notification rules read them (never raw data) and keep their firing state in `notification_rule_states` so they can send recovery notices across restarts.
+SLO_COMPUTE_WORKERS=4 # SLOs computed at once per instance (max 64)
+SLO_COMPUTE_TIMEOUT_SECONDS=30 # per-SLO budget for its raw queries and window sums in one tick; a timeout is counted (traceway.slo.timeouts.delta), captured as an exception naming the SLO, and retried next tick
# Synthetics (synthetic uptime monitoring: backend/app/synthetics, /synthetics frontend route)
SYNTHETICS_POLL_SECONDS=15 # scheduler tick for due checks; minimum 5, invalid values fall back to 15. The scheduler enqueues due checks into the check_runs queue (main DB, outbox-style guarded claims, advisory lock 824737004) and records expired queued runs as `missed` in telemetry — a probe is never executed late. In-process executors claim http/tcp runs always, browser runs only when mode=embedded.
@@ -549,6 +552,8 @@ const handleClick = createRowClickHandler('/issues/abc123', 'preset', 'from', 't
/monitors/status-pages/[pageId]/incidents Paginated incident management for one status page (timeline updates, titles, manual resolve/delete, post-mortem links); reads for members, mutations admin-gated in the UI
/monitors/[checkId] Monitor detail (latency chart, uptime bars, runs w/ result filter, incidents w/ titles + post-mortem links)
/monitors/post-mortems/[id] Post-mortem editor/viewer (@milkdown/crepe WYSIWYG with a Rich text/Markdown raw toggle, tag pills + Add-tags dialog, incident link pill + searchable picker, Activity sheet; write gating follows the effective project role). Project-scoped: opening a doc under a different project 404s. Creation is a title-only dialog (new-post-mortem-dialog.svelte, openable from the tab, monitor detail incidents, and the status page incidents page with an optional incident pre-link) that POSTs immediately and navigates here; /monitors/post-mortems/new redirects to the tab
+/slos SLOs (service level objectives, `backend/app/slo`): one row per objective with kind badge, SLI summary, target, window, current SLI, error-budget bar, 1h burn rate and status; sort key `slos`; sidebar badge = breached count; frontend and mobile projects see only the `session` kind (crash-free sessions, `sessions` joined to `exceptions_v2.session_id`), backend projects the other five
+/slos/[id] SLO detail: SLI and error-budget charts over a range clamped to the window (default = the whole window), burn-rate tiles, a coverage note while history backfills, a "View bad events" deep link per kind, and the alert rules that reference the SLO
/status/[slug] Public status page (light standalone design, no auth, raw fetch, listed in isPublicPath)
/connection SDK integration guide
```
@@ -873,6 +878,15 @@ Checks (`synthetic_checks`, main DB) are http/tcp/browser probes with per-check
| GET/PUT/DELETE | `/api/post-mortems/:id` | App / +Write / +Write | Full document with author names (contentMd) / update (records a `post_mortem_events` diff row; no-op saves change nothing; an optional `expectedUpdatedAt` answers 409 when the document changed since that read; the row is locked first (`LockForUpdate`) so concurrent Postgres updates cannot both pass the check. The CLI and MCP send the revision the caller's edit is based on, required when they replace the body (`--expected-updated-at` / `expected_updated_at`), and fall back to their own fresh read for title, tag and incident-only edits) / delete |
| GET | `/api/post-mortems/:id/activity` | App | Edit log for the Activity panel (who created/edited what, joined user names) |
+**SLOs** (service level objectives; worker `backend/app/slo`, controller `slo.controller.go`, telemetry table `slo_buckets`, main-DB tables `slos` and `notification_rule_states`; docs `docs/pages/learn/slos.mdx`)
+| Method | Endpoint | Auth | Purpose |
+|--------|----------|------|---------|
+| GET/POST | `/api/slos` | App / App+Write | List (snapshot columns included, disabled ones too) / create. `sliKind` is endpoint, span, task, monitor, metric or session with a per-kind `sliConfig` (validated and canonicalized in `validateSLIDefinition`), `targetPercent` in (0,100), `windowDays` 7, 30 or 90, `bucketMinutes` only for metric (1, 5, 15 or 60; other kinds are forced to 1). Create and update call `OnCommit(slo.Wake)` |
+| GET/PUT/DELETE | `/api/slos/:id` | App / +Write / +Write | `{slo, rules}` (the project rules of type `slo_*` whose config names it) / update (a change of kind, config or slice length resets the snapshot, and the worker deletes the old buckets on its next tick; target and window changes keep the history) / delete (auto-resolves pages of referencing rules through the `slo:<id>` dedup token, deletes those rules, deletes the buckets post-commit) |
+| POST | `/api/slos/:id/series` | App | `{buckets: [{bucket, good, total}]}` re-bucketed to `intervalMinutes` for `{fromDate, toDate}` (max 91 days); telemetry read, no Transactional, scoped by project through the bucket key |
+| POST | `/api/slos/preview` | App | `{good, total, sli, from, to}` of a definition over the last 24h straight from raw telemetry (the sheet's Preview button) |
+| GET | `/api/slos/breached-count` | App | Sidebar badge |
+
**Logs**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
diff --git a/README.md b/README.md
index b3336aca..f6758dc8 100644
--- a/README.md
+++ b/README.md
@@ -54,6 +54,7 @@ Traceway is an **OpenTelemetry-native** observability platform. It combines **lo
- **Session Replay**: Watch what the user did right before the error. Available for web (any JS framework) and Flutter.
- **AI Observability**: LLM cost, tokens, latency, and full conversations across providers (OpenRouter and any OTel-compatible AI gateway). Calls group into conversations via `gen_ai.conversation.id`, tool calls are parsed from completions and rendered in the chat view, and multi-language content flagging catches conversations containing terms you care about. Per-customer analytics (conversation length, cost per conversation) key on `user.id`: set it to a stable customer identifier such as your account or tenant id, the same value across all of that user's conversations, never a session id.
- **On-Call & Paging**: Rotation schedules with layers and overrides, escalation policies, and pages that escalate until someone acknowledges. Delivered via email, Slack, Pushover, Telegram, or SMS, with one-click acknowledge links that need no login.
+- **SLOs**: Service level objectives over any indicator Traceway already stores: endpoint availability or latency, any span filter (a queue consumer, an outbound call), task success, monitor uptime, a metric held under a limit per time slice, or crash-free sessions for browser and mobile projects. Rolling 7, 30 or 90 day windows, an error budget, burn rates, and multi-window burn-rate alerts with recovery notices, all computed from precomputed buckets rather than raw scans.
- **Fleet Overview**: One organization-wide page above every project: each reporting instance with live CPU, memory, disk and network, plus issues, monitors and open pages across all projects. Kubernetes nodes group by cluster, and a single `kubectl apply` instruments a whole cluster.
Plus: background-task (job) monitoring, configurable alerts (Slack / GitHub / email / webhook / Pushover / Telegram), multi-tenant orgs with role-based access, and a per-endpoint slow-threshold override.
@@ -300,6 +301,7 @@ Full documentation at **[docs.tracewayapp.com](https://docs.tracewayapp.com)**:
- [**Embedded Mode**](https://docs.tracewayapp.com/learn/embedded-mode): Run Traceway inside your Go app
- [**Kubernetes**](https://docs.tracewayapp.com/learn/kubernetes): Instrument a whole cluster with the manifests in [`examples/kubernetes`](./examples/kubernetes)
- [**Organization Overview**](https://docs.tracewayapp.com/learn/organization-overview): Servers, issues, monitors, and on-call pages across every project, one sidebar click apart
+- [**SLOs**](https://docs.tracewayapp.com/learn/slos): Objectives, error budgets and burn-rate alerting over endpoints, spans, tasks, monitors, metrics and crash-free sessions
## Community
diff --git a/backend/app/config/config.go b/backend/app/config/config.go
index 6f055d7a..4f03b2ab 100644
--- a/backend/app/config/config.go
+++ b/backend/app/config/config.go
@@ -64,6 +64,9 @@ type Cfg struct {
NotificationPollSeconds string
OncallPollSeconds string
OutboxPollSeconds string
+ SLOPollSeconds string
+ SLOComputeWorkers string
+ SLOComputeTimeoutSecs string
SyntheticsPollSeconds string
SyntheticsBrowserMode string
@@ -264,6 +267,9 @@ func LoadFromEnv() *Cfg {
NotificationPollSeconds: os.Getenv("NOTIFICATION_POLL_SECONDS"),
OncallPollSeconds: os.Getenv("ONCALL_POLL_SECONDS"),
OutboxPollSeconds: os.Getenv("OUTBOX_POLL_SECONDS"),
+ SLOPollSeconds: os.Getenv("SLO_POLL_SECONDS"),
+ SLOComputeWorkers: os.Getenv("SLO_COMPUTE_WORKERS"),
+ SLOComputeTimeoutSecs: os.Getenv("SLO_COMPUTE_TIMEOUT_SECONDS"),
SyntheticsPollSeconds: os.Getenv("SYNTHETICS_POLL_SECONDS"),
SyntheticsBrowserMode: os.Getenv("SYNTHETICS_BROWSER_MODE"),
diff --git a/backend/app/controllers/health_deep.go b/backend/app/controllers/health_deep.go
index d1398ca1..3173d063 100644
--- a/backend/app/controllers/health_deep.go
+++ b/backend/app/controllers/health_deep.go
@@ -7,6 +7,7 @@ import (
"github.com/gin-gonic/gin"
"github.com/tracewayapp/traceway/backend/app/db"
"github.com/tracewayapp/traceway/backend/app/outbox"
+ "github.com/tracewayapp/traceway/backend/app/slo"
"github.com/tracewayapp/traceway/backend/app/synthetics"
traceway "go.tracewayapp.com"
)
@@ -45,6 +46,7 @@ type HealthDeepResponse struct {
Engine *db.TelemetryEngineStats `json:"engine,omitempty"`
Outbox *outbox.HealthStats `json:"outbox,omitempty"`
Synthetics *synthetics.HealthStats `json:"synthetics,omitempty"`
+ SLO *slo.HealthStats `json:"slo,omitempty"`
}
// 503 means the configured telemetry backend is down. On the embedded
@@ -74,6 +76,11 @@ func (h healthDeepController) Get(c *gin.Context) {
} else {
traceway.CaptureException(fmt.Errorf("failed to load synthetics health: %w", err))
}
+ sloStats, err := slo.HealthSnapshot(c.Request.Context())
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to load slo health: %w", err))
+ }
+ resp.SLO = sloStats
if resp.TelemetryBackend == "clickhouse" && !resp.CHReachable {
c.JSON(http.StatusServiceUnavailable, resp)
diff --git a/backend/app/controllers/notification_rule.controller.go b/backend/app/controllers/notification_rule.controller.go
index 862fdd2a..4f93c1bc 100644
--- a/backend/app/controllers/notification_rule.controller.go
+++ b/backend/app/controllers/notification_rule.controller.go
@@ -1,19 +1,23 @@
package controllers
import (
+ "database/sql"
"encoding/json"
"errors"
"io"
"net/http"
+ "reflect"
"strconv"
"strings"
"time"
"github.com/gin-gonic/gin"
+ "github.com/google/uuid"
"github.com/tracewayapp/traceway/backend/app/db"
"github.com/tracewayapp/traceway/backend/app/middleware"
"github.com/tracewayapp/traceway/backend/app/models"
"github.com/tracewayapp/traceway/backend/app/notifications"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
"github.com/tracewayapp/traceway/backend/app/repositories/transactional"
traceway "go.tracewayapp.com"
)
@@ -41,6 +45,8 @@ var validRuleTypes = map[string]bool{
"ai_conversation_cost": true,
"ai_flagged_content": true,
"check_down": true,
+ "slo_burn_rate": true,
+ "slo_budget_consumed": true,
}
func (ctrl *notificationRuleController) List(ctx *gin.Context) {
@@ -121,6 +127,15 @@ func (ctrl *notificationRuleController) Create(ctx *gin.Context) {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Channel not found in this project."})
return
}
+ message, err := validateSLORuleConfig(tx, projectId, req.RuleType, req.Config)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate the SLO rule: %w", err))
+ return
+ }
+ if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
userId := middleware.GetUserId(ctx)
var createdBy *int
@@ -216,12 +231,29 @@ func (ctrl *notificationRuleController) Update(ctx *gin.Context) {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Channel not found in this project."})
return
}
+ message, err := validateSLORuleConfig(tx, projectId, req.RuleType, req.Config)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate the SLO rule: %w", err))
+ return
+ }
+ if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
cooldown := req.CooldownMinutes
if cooldown <= 0 {
cooldown = 15
}
+ conditionChanged := existing.RuleType != req.RuleType || existing.ChannelId != req.ChannelId || !sameRuleConfig(existing.Config, req.Config)
+ if conditionChanged {
+ if err := resolveSLORulePage(tx, existing); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to resolve the page of the edited rule: %w", err))
+ return
+ }
+ }
+
existing.ChannelId = req.ChannelId
existing.Name = req.Name
existing.RuleType = req.RuleType
@@ -234,7 +266,13 @@ func (ctrl *notificationRuleController) Update(ctx *gin.Context) {
ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to update notification rule: %w", err))
return
}
- notifications.ClearRuleState(id)
+ if conditionChanged {
+ notifications.ClearRuleState(id)
+ if err := transactional.NotificationRuleStateRepository.Delete(tx, id); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to reset notification rule state: %w", err))
+ return
+ }
+ }
ctx.JSON(http.StatusOK, existing)
}
@@ -264,6 +302,10 @@ func (ctrl *notificationRuleController) Delete(ctx *gin.Context) {
return
}
+ if err := resolveSLORulePage(tx, existing); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to resolve the page of the deleted rule: %w", err))
+ return
+ }
if err := transactional.NotificationRuleRepository.Delete(tx, id); err != nil {
ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to delete notification rule: %w", err))
return
@@ -313,6 +355,12 @@ func (ctrl *notificationRuleController) Toggle(ctx *gin.Context) {
if req.Enabled != nil {
newEnabled = *req.Enabled
}
+ if !newEnabled {
+ if err := resolveSLORulePage(tx, existing); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to resolve the page of the disabled rule: %w", err))
+ return
+ }
+ }
if err := transactional.NotificationRuleRepository.UpdateEnabled(tx, id, newEnabled); err != nil {
ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to toggle notification rule: %w", err))
return
@@ -373,3 +421,74 @@ func (ctrl *notificationRuleController) Snooze(ctx *gin.Context) {
}
var NotificationRuleController = notificationRuleController{}
+
+func validateSLORuleConfig(tx *sql.Tx, projectId uuid.UUID, ruleType string, config json.RawMessage) (string, error) {
+ switch ruleType {
+ case "slo_burn_rate":
+ var cfg struct {
+ SloId int `json:"sloId"`
+ LongWindowMinutes int `json:"longWindowMinutes"`
+ ShortWindowMinutes int `json:"shortWindowMinutes"`
+ BurnRateThreshold float64 `json:"burnRateThreshold"`
+ }
+ if err := json.Unmarshal(config, &cfg); err != nil {
+ return "The rule configuration is not valid JSON.", nil
+ }
+ if cfg.LongWindowMinutes < 5 || cfg.LongWindowMinutes > 10080 {
+ return "Long window must be between 5 minutes and 7 days.", nil
+ }
+ if cfg.ShortWindowMinutes < 1 || cfg.ShortWindowMinutes >= cfg.LongWindowMinutes {
+ return "Short window must be at least 1 minute and shorter than the long window.", nil
+ }
+ if cfg.BurnRateThreshold <= 0 {
+ return "Burn rate threshold must be above 0.", nil
+ }
+ return validateRuleSLO(tx, projectId, cfg.SloId)
+ case "slo_budget_consumed":
+ var cfg struct {
+ SloId int `json:"sloId"`
+ ConsumedPercent float64 `json:"consumedPercent"`
+ }
+ if err := json.Unmarshal(config, &cfg); err != nil {
+ return "The rule configuration is not valid JSON.", nil
+ }
+ if cfg.ConsumedPercent <= 0 || cfg.ConsumedPercent > 100 {
+ return "Budget consumed must be between 1 and 100 percent.", nil
+ }
+ return validateRuleSLO(tx, projectId, cfg.SloId)
+ }
+ return "", nil
+}
+
+func resolveSLORulePage(tx *sql.Tx, rule *models.NotificationRule) error {
+ sloId := sloRuleTarget(rule.RuleType, rule.Config)
+ if sloId == 0 {
+ return nil
+ }
+ if _, err := oncall.AutoResolveByDedupKeyTx(tx, rule.Id, notifications.SLODedupToken(sloId), time.Now().UTC()); err != nil {
+ return err
+ }
+ return transactional.NotificationRuleStateRepository.Delete(tx, rule.Id)
+}
+
+func sameRuleConfig(a, b json.RawMessage) bool {
+ var left, right any
+ if json.Unmarshal(a, &left) != nil || json.Unmarshal(b, &right) != nil {
+ return false
+ }
+ return reflect.DeepEqual(left, right)
+}
+
+func validateRuleSLO(tx *sql.Tx, projectId uuid.UUID, sloId int) (string, error) {
+ if sloId < 1 {
+ return "Select an SLO.", nil
+ }
+ entity, err := transactional.SLORepository.FindByIdForProject(tx, sloId, projectId)
+ if err != nil {
+ return "", err
+ }
+ if entity == nil {
+ return "SLO not found in this project.", nil
+ }
+ return "", nil
+}
diff --git a/backend/app/controllers/routes.go b/backend/app/controllers/routes.go
index c0d2933d..9e6a4ee1 100644
--- a/backend/app/controllers/routes.go
+++ b/backend/app/controllers/routes.go
@@ -283,6 +283,15 @@ func RegisterControllers(router *gin.RouterGroup) {
router.POST("/synthetics/checks/:id/series", middleware.UseAppAuth, middleware.RequireProjectAccess, SyntheticCheckController.Series)
router.GET("/synthetics/open-count", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, SyntheticCheckController.OpenCount)
+ router.GET("/slos", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, SLOController.List)
+ router.POST("/slos", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.RequireWriteAccess, middleware.Transactional, SLOController.Create)
+ router.GET("/slos/breached-count", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, SLOController.BreachedCount)
+ router.POST("/slos/preview", middleware.UseAppAuth, middleware.RequireProjectAccess, SLOController.Preview)
+ router.GET("/slos/:id", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, SLOController.Get)
+ router.PUT("/slos/:id", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.RequireWriteAccess, middleware.Transactional, SLOController.Update)
+ router.DELETE("/slos/:id", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.RequireWriteAccess, middleware.Transactional, SLOController.Delete)
+ router.POST("/slos/:id/series", middleware.UseAppAuth, middleware.RequireProjectAccess, SLOController.Series)
+
// No Transactional on either runner route: Poll holds the request for up
// to 25s (a held transaction would starve the single-connection SQLite
// main DB) and Result finalizes through ProcessOutcome, which manages its
diff --git a/backend/app/controllers/slo.controller.go b/backend/app/controllers/slo.controller.go
new file mode 100644
index 00000000..e53f0680
--- /dev/null
+++ b/backend/app/controllers/slo.controller.go
@@ -0,0 +1,585 @@
+package controllers
+
+import (
+ "bytes"
+ "context"
+ "database/sql"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/notifications"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ "github.com/tracewayapp/traceway/backend/app/slo"
+
+ "github.com/google/uuid"
+ traceway "go.tracewayapp.com"
+)
+
+type sloController struct{}
+
+var SLOController = &sloController{}
+
+const (
+ sloPreviewWindow = 24 * time.Hour
+ sloMaxSeriesRange = 91 * 24 * time.Hour
+ sloDefaultInterval = 60
+ sloMaxSeriesPoints = 2000
+)
+
+type sloRequest struct {
+ Name string `json:"name"`
+ Description string `json:"description"`
+ SliKind string `json:"sliKind"`
+ SliConfig json.RawMessage `json:"sliConfig"`
+ TargetPercent float64 `json:"targetPercent"`
+ WindowDays int `json:"windowDays"`
+ BucketMinutes int `json:"bucketMinutes"`
+ Enabled *bool `json:"enabled"`
+}
+
+type sloSeriesRequest struct {
+ FromDate string `json:"fromDate"`
+ ToDate string `json:"toDate"`
+ IntervalMinutes int `json:"intervalMinutes"`
+}
+
+type sloPreviewRequest struct {
+ SliKind string `json:"sliKind"`
+ SliConfig json.RawMessage `json:"sliConfig"`
+ BucketMinutes int `json:"bucketMinutes"`
+}
+
+type sloDefinition struct {
+ Kind string
+ Config json.RawMessage
+ BucketMinutes int
+}
+
+func validateSLIDefinition(tx *sql.Tx, projectId uuid.UUID, kind string, raw json.RawMessage, bucketMinutes int) (sloDefinition, string, error) {
+ def := sloDefinition{Kind: kind, BucketMinutes: 1}
+ if !models.ValidSLIKind(kind) {
+ return def, "SLI kind must be one of: endpoint, span, task, monitor, metric, session.", nil
+ }
+ if len(raw) == 0 {
+ return def, "SLI configuration is required.", nil
+ }
+ cfg, err := models.ParseSLIConfig(kind, raw)
+ if err != nil {
+ return def, "The SLI configuration is not valid JSON.", nil
+ }
+ var canonical any
+ switch kind {
+ case models.SLIKindEndpoint:
+ c := cfg.Endpoint
+ c.Endpoint = strings.TrimSpace(c.Endpoint)
+ if message := validateGoodCondition(c.Good, c.LatencyThresholdMs); message != "" {
+ return def, message, nil
+ }
+ if c.Good != models.SLOGoodLatency {
+ c.LatencyThresholdMs = 0
+ }
+ canonical = c
+ case models.SLIKindSpan:
+ c := cfg.Span
+ c.ServiceName = strings.TrimSpace(c.ServiceName)
+ c.Name = strings.TrimSpace(c.Name)
+ if c.Kind != nil && (*c.Kind < 0 || *c.Kind > 5) {
+ return def, "Span kind must be between 0 and 5.", nil
+ }
+ if len(c.Attributes) > models.SLOMaxSpanAttributeFilters {
+ return def, fmt.Sprintf("At most %d attribute filters are allowed.", models.SLOMaxSpanAttributeFilters), nil
+ }
+ attributes := make([]models.SpanSLIAttribute, 0, len(c.Attributes))
+ for _, attribute := range c.Attributes {
+ key := strings.TrimSpace(attribute.Key)
+ if key == "" {
+ return def, "Attribute filters need a key.", nil
+ }
+ if strings.ContainsAny(key, `"\`) || len(key) > 256 {
+ return def, "Attribute keys cannot contain quotes or backslashes.", nil
+ }
+ if attribute.Value == "" {
+ return def, "An attribute filter needs a value.", nil
+ }
+ attributes = append(attributes, models.SpanSLIAttribute{Key: key, Value: attribute.Value})
+ }
+ c.Attributes = attributes
+ if message := validateGoodCondition(c.Good, c.LatencyThresholdMs); message != "" {
+ return def, message, nil
+ }
+ if c.Good != models.SLOGoodLatency {
+ c.LatencyThresholdMs = 0
+ }
+ canonical = c
+ case models.SLIKindTask:
+ cfg.Task.TaskName = strings.TrimSpace(cfg.Task.TaskName)
+ canonical = cfg.Task
+ case models.SLIKindMonitor:
+ c := cfg.Monitor
+ if len(c.CheckIds) == 0 {
+ return def, "Select at least one monitor.", nil
+ }
+ if len(c.CheckIds) > models.SLOMaxMonitors {
+ return def, fmt.Sprintf("At most %d monitors are allowed.", models.SLOMaxMonitors), nil
+ }
+ seen := map[int]bool{}
+ ids := make([]int, 0, len(c.CheckIds))
+ for _, id := range c.CheckIds {
+ if id < 1 || seen[id] {
+ continue
+ }
+ seen[id] = true
+ if tx != nil {
+ check, err := transactional.SyntheticCheckRepository.FindByIdForProject(tx, id, projectId)
+ if err != nil {
+ return def, "", err
+ }
+ if check == nil {
+ return def, fmt.Sprintf("Monitor %d was not found in this project.", id), nil
+ }
+ }
+ ids = append(ids, id)
+ }
+ if len(ids) == 0 {
+ return def, "Select at least one monitor.", nil
+ }
+ c.CheckIds = ids
+ canonical = c
+ case models.SLIKindMetric:
+ c := cfg.Metric
+ c.MetricName = strings.TrimSpace(c.MetricName)
+ if c.MetricName == "" {
+ return def, "Metric name is required.", nil
+ }
+ if !models.SLOMetricAggregations[c.Aggregation] {
+ return def, "Aggregation must be one of: avg, min, max, sum, last.", nil
+ }
+ if !models.SLOMetricOperators[c.Operator] {
+ return def, "Operator must be one of: gt, gte, lt, lte.", nil
+ }
+ var threshold struct {
+ Threshold *float64 `json:"threshold"`
+ }
+ if json.Unmarshal(raw, &threshold) != nil || threshold.Threshold == nil {
+ return def, "Threshold is required.", nil
+ }
+ if c.Tags == nil {
+ c.Tags = map[string]string{}
+ }
+ if len(c.Tags) > models.SLOMaxMetricTagFilters {
+ return def, fmt.Sprintf("At most %d tag filters are allowed.", models.SLOMaxMetricTagFilters), nil
+ }
+ validSlice := false
+ for _, minutes := range models.SLOMetricSliceMinutes {
+ if bucketMinutes == minutes {
+ validSlice = true
+ }
+ }
+ if !validSlice {
+ return def, "Slice length must be 1, 5, 15 or 60 minutes.", nil
+ }
+ def.BucketMinutes = bucketMinutes
+ canonical = c
+ case models.SLIKindSession:
+ cfg.Session.AppVersion = strings.TrimSpace(cfg.Session.AppVersion)
+ canonical = cfg.Session
+ }
+ encoded, err := json.Marshal(canonical)
+ if err != nil {
+ return def, "", err
+ }
+ def.Config = encoded
+ return def, "", nil
+}
+
+func validateGoodCondition(good string, latencyThresholdMs float64) string {
+ switch good {
+ case models.SLOGoodAvailability:
+ return ""
+ case models.SLOGoodLatency:
+ if latencyThresholdMs <= 0 {
+ return "Latency threshold must be above 0 ms."
+ }
+ return ""
+ }
+ return "Good condition must be availability or latency."
+}
+
+func validateSLORequest(tx *sql.Tx, projectId uuid.UUID, req *sloRequest) (sloDefinition, string, error) {
+ req.Name = strings.TrimSpace(req.Name)
+ req.Description = strings.TrimSpace(req.Description)
+ if req.Name == "" {
+ return sloDefinition{}, "Name is required.", nil
+ }
+ if len(req.Name) > 200 {
+ return sloDefinition{}, "Name must be 200 characters or fewer.", nil
+ }
+ if len(req.Description) > 2000 {
+ return sloDefinition{}, "Description must be 2000 characters or fewer.", nil
+ }
+ if req.TargetPercent <= 0 || req.TargetPercent >= 100 {
+ return sloDefinition{}, "Target must be above 0 and below 100 percent.", nil
+ }
+ validWindow := false
+ for _, days := range models.SLOWindowDays {
+ if req.WindowDays == days {
+ validWindow = true
+ }
+ }
+ if !validWindow {
+ return sloDefinition{}, "Window must be 7, 30 or 90 days.", nil
+ }
+ return validateSLIDefinition(tx, projectId, req.SliKind, req.SliConfig, req.BucketMinutes)
+}
+
+func sloIdFromPath(ctx *gin.Context) (projectId uuid.UUID, sloId int, ok bool) {
+ pid, err := middleware.GetProjectId(ctx)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return pid, 0, false
+ }
+ id, err := strconv.Atoi(ctx.Param("id"))
+ if err != nil || id < 1 {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid SLO id"})
+ return pid, 0, false
+ }
+ return pid, id, true
+}
+
+func sloRuleTarget(ruleType string, config json.RawMessage) int {
+ if !strings.HasPrefix(ruleType, "slo_") {
+ return 0
+ }
+ var cfg struct {
+ SloId int `json:"sloId"`
+ }
+ if err := json.Unmarshal(config, &cfg); err != nil {
+ return 0
+ }
+ return cfg.SloId
+}
+
+func (ctrl *sloController) List(ctx *gin.Context) {
+ projectId, err := middleware.GetProjectId(ctx)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ tx := db.GetTx(ctx)
+ slos, err := transactional.SLORepository.FindByProject(tx, projectId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list SLOs: %w", err))
+ return
+ }
+ if slos == nil {
+ slos = []*models.SLO{}
+ }
+ ctx.JSON(http.StatusOK, gin.H{"slos": slos})
+}
+
+func (ctrl *sloController) Create(ctx *gin.Context) {
+ projectId, err := middleware.GetProjectId(ctx)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ var req sloRequest
+ if err := ctx.ShouldBindJSON(&req); err != nil {
+ middleware.RejectBindError(ctx, err, "Invalid request body")
+ return
+ }
+ tx := db.GetTx(ctx)
+ def, message, err := validateSLORequest(tx, projectId, &req)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate SLO: %w", err))
+ return
+ }
+ if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ now := time.Now().UTC()
+ enabled := true
+ if req.Enabled != nil {
+ enabled = *req.Enabled
+ }
+ userId := middleware.GetUserId(ctx)
+ var createdBy *int
+ if userId > 0 {
+ createdBy = &userId
+ }
+ entity := &models.SLO{
+ ProjectId: projectId,
+ Name: req.Name,
+ Description: req.Description,
+ SliKind: def.Kind,
+ SliConfig: models.JSONText(def.Config),
+ TargetPercent: req.TargetPercent,
+ WindowDays: req.WindowDays,
+ BucketMinutes: def.BucketMinutes,
+ Enabled: enabled,
+ Status: models.SLOStatusNoData,
+ SliSince: &now,
+ CreatedBy: createdBy,
+ CreatedAt: now,
+ UpdatedAt: now,
+ }
+ id, err := transactional.SLORepository.Create(tx, entity)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to create SLO: %w", err))
+ return
+ }
+ entity.Id = id
+ middleware.OnCommit(ctx, slo.Wake)
+ ctx.JSON(http.StatusCreated, entity)
+}
+
+func (ctrl *sloController) Get(ctx *gin.Context) {
+ projectId, sloId, ok := sloIdFromPath(ctx)
+ if !ok {
+ return
+ }
+ tx := db.GetTx(ctx)
+ entity, err := transactional.SLORepository.FindByIdForProject(tx, sloId, projectId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load SLO: %w", err))
+ return
+ }
+ if entity == nil {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "SLO not found"})
+ return
+ }
+ rules, err := transactional.NotificationRuleRepository.FindByProjectWithChannel(tx, projectId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load notification rules for SLO: %w", err))
+ return
+ }
+ referencing := make([]*models.NotificationRuleWithChannel, 0)
+ for _, rule := range rules {
+ if sloRuleTarget(rule.RuleType, rule.Config) == sloId {
+ referencing = append(referencing, rule)
+ }
+ }
+ ctx.JSON(http.StatusOK, gin.H{"slo": entity, "rules": referencing})
+}
+
+func (ctrl *sloController) Update(ctx *gin.Context) {
+ projectId, sloId, ok := sloIdFromPath(ctx)
+ if !ok {
+ return
+ }
+ var req sloRequest
+ if err := ctx.ShouldBindJSON(&req); err != nil {
+ middleware.RejectBindError(ctx, err, "Invalid request body")
+ return
+ }
+ tx := db.GetTx(ctx)
+ existing, err := transactional.SLORepository.FindByIdForProject(tx, sloId, projectId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load SLO: %w", err))
+ return
+ }
+ if existing == nil {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "SLO not found"})
+ return
+ }
+ def, message, err := validateSLORequest(tx, projectId, &req)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate SLO: %w", err))
+ return
+ }
+ if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ definitionChanged := existing.SliKind != def.Kind || existing.BucketMinutes != def.BucketMinutes || !sameSLIConfig(existing, def)
+
+ existing.Name = req.Name
+ existing.Description = req.Description
+ existing.SliKind = def.Kind
+ existing.SliConfig = models.JSONText(def.Config)
+ existing.TargetPercent = req.TargetPercent
+ existing.WindowDays = req.WindowDays
+ existing.BucketMinutes = def.BucketMinutes
+ if req.Enabled != nil {
+ existing.Enabled = *req.Enabled
+ }
+ existing.UpdatedAt = time.Now().UTC()
+ if err := transactional.SLORepository.Update(tx, existing); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to update SLO: %w", err))
+ return
+ }
+ if definitionChanged {
+ if err := transactional.SLORepository.ResetHistory(tx, sloId, existing.UpdatedAt); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to reset SLO history: %w", err))
+ return
+ }
+ }
+ middleware.OnCommit(ctx, slo.Wake)
+ updated, err := transactional.SLORepository.FindByIdForProject(tx, sloId, projectId)
+ if err != nil || updated == nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to reload SLO: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, updated)
+}
+
+func sameSLIConfig(existing *models.SLO, def sloDefinition) bool {
+ stored, message, _ := validateSLIDefinition(nil, existing.ProjectId, existing.SliKind, json.RawMessage(existing.SliConfig), existing.BucketMinutes)
+ if message != "" {
+ return false
+ }
+ return bytes.Equal(stored.Config, def.Config)
+}
+
+func (ctrl *sloController) Delete(ctx *gin.Context) {
+ projectId, sloId, ok := sloIdFromPath(ctx)
+ if !ok {
+ return
+ }
+ tx := db.GetTx(ctx)
+ entity, err := transactional.SLORepository.FindByIdForProject(tx, sloId, projectId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load SLO: %w", err))
+ return
+ }
+ if entity == nil {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "SLO not found"})
+ return
+ }
+ rules, err := transactional.NotificationRuleRepository.FindByProjectWithChannel(tx, projectId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load notification rules for SLO cleanup: %w", err))
+ return
+ }
+ now := time.Now().UTC()
+ for _, rule := range rules {
+ if sloRuleTarget(rule.RuleType, rule.Config) != sloId {
+ continue
+ }
+ if _, err := oncall.AutoResolveByDedupKeyTx(tx, rule.Id, notifications.SLODedupToken(sloId), now); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to resolve pages for deleted SLO: %w", err))
+ return
+ }
+ if err := transactional.NotificationRuleRepository.Delete(tx, rule.Id); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to delete rule of deleted SLO: %w", err))
+ return
+ }
+ notifications.ClearRuleState(rule.Id)
+ }
+ if err := transactional.SLORepository.Delete(tx, sloId, projectId); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to delete SLO: %w", err))
+ return
+ }
+ middleware.OnCommit(ctx, func() {
+ if err := telemetry.SLOBucketRepository.DeleteBySLO(context.Background(), projectId, sloId); err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to delete buckets of deleted SLO %d: %w", sloId, err))
+ }
+ })
+ ctx.JSON(http.StatusOK, gin.H{"deleted": true})
+}
+
+func (ctrl *sloController) Series(ctx *gin.Context) {
+ projectId, sloId, ok := sloIdFromPath(ctx)
+ if !ok {
+ return
+ }
+ var req sloSeriesRequest
+ if err := ctx.ShouldBindJSON(&req); err != nil {
+ middleware.RejectBindError(ctx, err, "Invalid request body")
+ return
+ }
+ from, to, ok := parseCheckRange(ctx, req.FromDate, req.ToDate)
+ if !ok {
+ return
+ }
+ if !to.After(from) {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "toDate must be after fromDate."})
+ return
+ }
+ if to.Sub(from) > sloMaxSeriesRange {
+ from = to.Add(-sloMaxSeriesRange)
+ }
+ interval := req.IntervalMinutes
+ if interval < 1 {
+ interval = sloDefaultInterval
+ }
+ if interval > 1440 {
+ interval = 1440
+ }
+ if minimum := int((to.Sub(from).Minutes() + sloMaxSeriesPoints - 1) / sloMaxSeriesPoints); interval < minimum {
+ interval = minimum
+ }
+ points, err := telemetry.SLOBucketRepository.Series(ctx, projectId, sloId, from, to, interval)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load SLO series: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"buckets": points})
+}
+
+func (ctrl *sloController) Preview(ctx *gin.Context) {
+ projectId, err := middleware.GetProjectId(ctx)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ var req sloPreviewRequest
+ if err := ctx.ShouldBindJSON(&req); err != nil {
+ middleware.RejectBindError(ctx, err, "Invalid request body")
+ return
+ }
+ def, message, err := validateSLIDefinition(nil, projectId, req.SliKind, req.SliConfig, req.BucketMinutes)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate the SLO preview: %w", err))
+ return
+ }
+ if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ draft := &models.SLO{ProjectId: projectId, SliKind: def.Kind, SliConfig: models.JSONText(def.Config), BucketMinutes: def.BucketMinutes}
+ to := time.Now().UTC().Truncate(draft.BucketDuration())
+ draft.SliSince = &to
+ from := to.Add(-sloPreviewWindow)
+ points, err := slo.ComputeBuckets(ctx, draft, from, to)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to preview SLO: %w", err))
+ return
+ }
+ var window slo.Window
+ for _, point := range points {
+ window.Good += point.Good
+ window.Total += point.Total
+ }
+ response := gin.H{"good": window.Good, "total": window.Total, "sli": nil, "from": from, "to": to}
+ if sli, ok := window.SLI(); ok {
+ response["sli"] = sli * 100
+ }
+ ctx.JSON(http.StatusOK, response)
+}
+
+func (ctrl *sloController) BreachedCount(ctx *gin.Context) {
+ projectId, err := middleware.GetProjectId(ctx)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ tx := db.GetTx(ctx)
+ count, err := transactional.SLORepository.CountBreachedByProject(tx, projectId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to count breached SLOs: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"count": count})
+}
diff --git a/backend/app/controllers/slo_rule_page_sqlite_test.go b/backend/app/controllers/slo_rule_page_sqlite_test.go
new file mode 100644
index 00000000..9c6d381c
--- /dev/null
+++ b/backend/app/controllers/slo_rule_page_sqlite_test.go
@@ -0,0 +1,78 @@
+//go:build !transactional_pg && !telemetry_ch && !telemetry_duckdb
+
+package controllers
+
+import (
+ "database/sql"
+ "fmt"
+ "testing"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+func TestResolveSLORulePageClosesTheAlertPage(t *testing.T) {
+ dbtest.SetupSQLite(t)
+ orgId := createTestOrg(t, "org")
+ now := time.Now().UTC()
+ type seeded struct {
+ rule *models.NotificationRule
+ pageId int
+ }
+ s, err := db.ExecuteTransaction(func(tx *sql.Tx) (seeded, error) {
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, "api", "gin", orgId)
+ if err != nil {
+ return seeded{}, err
+ }
+ channel := &models.NotificationChannel{ProjectId: project.Id, Name: "Pager", ChannelType: "escalation", Config: []byte(`{"policyId":1}`), Enabled: true, CreatedAt: now, UpdatedAt: now}
+ channelId, err := transactional.NotificationChannelRepository.Create(tx, channel)
+ if err != nil {
+ return seeded{}, err
+ }
+ rule := &models.NotificationRule{ProjectId: project.Id, ChannelId: channelId, Name: "burn", RuleType: "slo_burn_rate", Config: []byte(`{"sloId":7}`), Enabled: true, CooldownMinutes: 15, CreatedAt: now, UpdatedAt: now}
+ if rule.Id, err = transactional.NotificationRuleRepository.Create(tx, rule); err != nil {
+ return seeded{}, err
+ }
+ page := &models.Page{OrganizationId: orgId, ProjectId: project.Id, RuleId: &rule.Id, RuleName: rule.Name, RuleType: rule.RuleType, Subject: "burn", Severity: "warning", Urgency: "high", Status: "open",
+ DedupKey: fmt.Sprintf("%d|slo:7", rule.Id), EventCount: 1, LastEventAt: now, PolicySnapshot: models.JSONText(`{}`), CreatedAt: now, UpdatedAt: now}
+ pageId, err := transactional.PageRepository.Create(tx, page)
+ if err != nil {
+ return seeded{}, err
+ }
+ _, err = transactional.NotificationRuleStateRepository.Transition(tx, rule.Id, true, now)
+ return seeded{rule, pageId}, err
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ if _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ return struct{}{}, resolveSLORulePage(tx, s.rule)
+ }); err != nil {
+ t.Fatal(err)
+ }
+
+ page, state, err := func() (*models.Page, *models.NotificationRuleState, error) {
+ var page *models.Page
+ state, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.NotificationRuleState, error) {
+ var err error
+ if page, err = transactional.PageRepository.FindById(tx, s.pageId); err != nil {
+ return nil, err
+ }
+ return transactional.NotificationRuleStateRepository.Find(tx, s.rule.Id)
+ })
+ return page, state, err
+ }()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if page == nil || page.Status != "resolved" {
+ t.Fatalf("the page the alert opened must be resolved: %+v", page)
+ }
+ if state != nil {
+ t.Fatalf("the firing state must be forgotten: %+v", state)
+ }
+}
diff --git a/backend/app/controllers/slo_validation_test.go b/backend/app/controllers/slo_validation_test.go
new file mode 100644
index 00000000..8022c786
--- /dev/null
+++ b/backend/app/controllers/slo_validation_test.go
@@ -0,0 +1,70 @@
+package controllers
+
+import (
+ "encoding/json"
+ "testing"
+
+ "github.com/google/uuid"
+)
+
+func TestValidateSLIDefinitionRequiresAMetricThreshold(t *testing.T) {
+ base := map[string]any{"metricName": "cpu.used_pcnt", "aggregation": "avg", "operator": "lt"}
+ for name, threshold := range map[string]any{"missing": nil, "zero": 0.0, "set": 90.0} {
+ cfg := map[string]any{}
+ for k, v := range base {
+ cfg[k] = v
+ }
+ if name != "missing" {
+ cfg["threshold"] = threshold
+ }
+ raw, _ := json.Marshal(cfg)
+ _, message, _ := validateSLIDefinition(nil, uuid.New(), "metric", raw, 5)
+ if name == "missing" && message != "Threshold is required." {
+ t.Fatalf("missing threshold: got %q", message)
+ }
+ if name != "missing" && message != "" {
+ t.Fatalf("%s threshold: got %q", name, message)
+ }
+ }
+
+ raw := json.RawMessage(`{"metricName":"cpu.used_pcnt","aggregation":"avg","operator":"lt","threshold":null}`)
+ if _, message, _ := validateSLIDefinition(nil, uuid.New(), "metric", raw, 5); message != "Threshold is required." {
+ t.Fatalf("null threshold: got %q", message)
+ }
+}
+
+func TestSameRuleConfigIgnoresKeyOrder(t *testing.T) {
+ a := json.RawMessage(`{"sloId":1,"longWindowMinutes":60,"shortWindowMinutes":5,"burnRateThreshold":14.4}`)
+ b := json.RawMessage(`{"burnRateThreshold":14.4,"shortWindowMinutes":5,"longWindowMinutes":60,"sloId":1}`)
+ if !sameRuleConfig(a, b) {
+ t.Fatal("reordered keys should compare equal")
+ }
+ c := json.RawMessage(`{"sloId":1,"longWindowMinutes":60,"shortWindowMinutes":5,"burnRateThreshold":6}`)
+ if sameRuleConfig(a, c) {
+ t.Fatal("a changed threshold should compare different")
+ }
+}
+
+func TestValidateSLIDefinitionSpanAttributesMatchTheExplorer(t *testing.T) {
+ cases := map[string]string{
+ `{"attributes":[{"key":"tenant","value":""}],"good":"availability"}`: "An attribute filter needs a value.",
+ `{"attributes":[{"key":"te\"nant","value":"a"}],"good":"availability"}`: "Attribute keys cannot contain quotes or backslashes.",
+ `{"attributes":[{"key":"tenant","value":"acme"}],"good":"availability"}`: "",
+ }
+ for raw, want := range cases {
+ if _, got, _ := validateSLIDefinition(nil, uuid.New(), "span", json.RawMessage(raw), 1); got != want {
+ t.Fatalf("%s: got %q, want %q", raw, got, want)
+ }
+ }
+}
+
+func TestValidateSLIDefinitionCapsMonitors(t *testing.T) {
+ ids := make([]int, 51)
+ for i := range ids {
+ ids[i] = i + 1
+ }
+ raw, _ := json.Marshal(map[string]any{"checkIds": ids})
+ if _, got, _ := validateSLIDefinition(nil, uuid.New(), "monitor", raw, 1); got != "At most 50 monitors are allowed." {
+ t.Fatalf("got %q", got)
+ }
+}
diff --git a/backend/app/middleware/agent_routes.go b/backend/app/middleware/agent_routes.go
index f8e5f95c..a72e3a59 100644
--- a/backend/app/middleware/agent_routes.go
+++ b/backend/app/middleware/agent_routes.go
@@ -90,6 +90,10 @@ var agentRoutes = map[string]agentAccess{
"POST /api/synthetics/checks/:id/results": agentRead,
"POST /api/synthetics/checks/:id/series": agentRead,
"GET /api/synthetics/open-count": agentRead,
+ "GET /api/slos": agentRead,
+ "GET /api/slos/:id": agentRead,
+ "POST /api/slos/:id/series": agentRead,
+ "GET /api/slos/breached-count": agentRead,
"POST /api/synthetics/checks/:id/run": agentWrite,
"GET /api/organizations/:organizationId/overview/servers": agentRead,
diff --git a/backend/app/migrations/ch/0101_create_slo_buckets.up.sql b/backend/app/migrations/ch/0101_create_slo_buckets.up.sql
new file mode 100644
index 00000000..474a8168
--- /dev/null
+++ b/backend/app/migrations/ch/0101_create_slo_buckets.up.sql
@@ -0,0 +1,10 @@
+CREATE TABLE IF NOT EXISTS slo_buckets (
+ project_id UUID,
+ slo_id Int64,
+ bucket_start DateTime,
+ good Int64,
+ total Int64,
+ computed_at DateTime64(3)
+) ENGINE = ReplacingMergeTree(computed_at)
+PARTITION BY toYYYYMM(bucket_start)
+ORDER BY (project_id, slo_id, bucket_start)
diff --git a/backend/app/migrations/ch/0102_add_ttl_slo_buckets.up.sql b/backend/app/migrations/ch/0102_add_ttl_slo_buckets.up.sql
new file mode 100644
index 00000000..982c5d9a
--- /dev/null
+++ b/backend/app/migrations/ch/0102_add_ttl_slo_buckets.up.sql
@@ -0,0 +1 @@
+ALTER TABLE slo_buckets MODIFY TTL toDateTime(bucket_start) + INTERVAL 120 DAY
diff --git a/backend/app/migrations/duckdb_telemetry/0009_create_slo_buckets.up.sql b/backend/app/migrations/duckdb_telemetry/0009_create_slo_buckets.up.sql
new file mode 100644
index 00000000..513177b6
--- /dev/null
+++ b/backend/app/migrations/duckdb_telemetry/0009_create_slo_buckets.up.sql
@@ -0,0 +1,9 @@
+CREATE TABLE IF NOT EXISTS slo_buckets (
+ project_id VARCHAR NOT NULL,
+ slo_id BIGINT NOT NULL,
+ bucket_start TIMESTAMP NOT NULL,
+ good BIGINT NOT NULL DEFAULT 0,
+ total BIGINT NOT NULL DEFAULT 0,
+ computed_at TIMESTAMP NOT NULL,
+ UNIQUE(project_id, slo_id, bucket_start)
+);
diff --git a/backend/app/migrations/pg/0150_create_slos.up.sql b/backend/app/migrations/pg/0150_create_slos.up.sql
new file mode 100644
index 00000000..bee31cd6
--- /dev/null
+++ b/backend/app/migrations/pg/0150_create_slos.up.sql
@@ -0,0 +1,23 @@
+CREATE TABLE IF NOT EXISTS slos (
+ id SERIAL PRIMARY KEY,
+ project_id UUID NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+ name VARCHAR(200) NOT NULL,
+ description TEXT NOT NULL DEFAULT '',
+ sli_kind VARCHAR(20) NOT NULL,
+ sli_config JSONB NOT NULL DEFAULT '{}',
+ target_percent DOUBLE PRECISION NOT NULL,
+ window_days INT NOT NULL DEFAULT 30,
+ bucket_minutes INT NOT NULL DEFAULT 1,
+ enabled BOOLEAN NOT NULL DEFAULT TRUE,
+ status VARCHAR(20) NOT NULL DEFAULT 'no_data',
+ current_sli DOUBLE PRECISION,
+ budget_remaining DOUBLE PRECISION,
+ burn_rate_1h DOUBLE PRECISION,
+ burn_rate_6h DOUBLE PRECISION,
+ computed_at TIMESTAMPTZ,
+ history_from TIMESTAMPTZ,
+ last_bucket_at TIMESTAMPTZ,
+ created_by INT,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+)
diff --git a/backend/app/migrations/pg/0151_create_slos_project_idx.up.sql b/backend/app/migrations/pg/0151_create_slos_project_idx.up.sql
new file mode 100644
index 00000000..207d0432
--- /dev/null
+++ b/backend/app/migrations/pg/0151_create_slos_project_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS slos_project_idx ON slos (project_id)
diff --git a/backend/app/migrations/pg/0152_create_notification_rule_states.up.sql b/backend/app/migrations/pg/0152_create_notification_rule_states.up.sql
new file mode 100644
index 00000000..544a2a3e
--- /dev/null
+++ b/backend/app/migrations/pg/0152_create_notification_rule_states.up.sql
@@ -0,0 +1,7 @@
+CREATE TABLE IF NOT EXISTS notification_rule_states (
+ id SERIAL PRIMARY KEY,
+ rule_id INT NOT NULL UNIQUE REFERENCES notification_rules(id) ON DELETE CASCADE,
+ firing BOOLEAN NOT NULL DEFAULT FALSE,
+ since TIMESTAMPTZ,
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+)
diff --git a/backend/app/migrations/pg/0153_add_revision_to_slos.up.sql b/backend/app/migrations/pg/0153_add_revision_to_slos.up.sql
new file mode 100644
index 00000000..a369b01c
--- /dev/null
+++ b/backend/app/migrations/pg/0153_add_revision_to_slos.up.sql
@@ -0,0 +1 @@
+ALTER TABLE slos ADD COLUMN revision INT NOT NULL DEFAULT 0
diff --git a/backend/app/migrations/pg/0154_add_sli_since_to_slos.up.sql b/backend/app/migrations/pg/0154_add_sli_since_to_slos.up.sql
new file mode 100644
index 00000000..dc783f42
--- /dev/null
+++ b/backend/app/migrations/pg/0154_add_sli_since_to_slos.up.sql
@@ -0,0 +1 @@
+ALTER TABLE slos ADD COLUMN sli_since TIMESTAMPTZ
diff --git a/backend/app/migrations/pg/0155_add_gap_from_to_slos.up.sql b/backend/app/migrations/pg/0155_add_gap_from_to_slos.up.sql
new file mode 100644
index 00000000..923b6884
--- /dev/null
+++ b/backend/app/migrations/pg/0155_add_gap_from_to_slos.up.sql
@@ -0,0 +1 @@
+ALTER TABLE slos ADD COLUMN gap_from TIMESTAMPTZ
diff --git a/backend/app/migrations/pg/0156_add_gap_to_to_slos.up.sql b/backend/app/migrations/pg/0156_add_gap_to_to_slos.up.sql
new file mode 100644
index 00000000..7ff7e1f1
--- /dev/null
+++ b/backend/app/migrations/pg/0156_add_gap_to_to_slos.up.sql
@@ -0,0 +1 @@
+ALTER TABLE slos ADD COLUMN gap_to TIMESTAMPTZ
diff --git a/backend/app/migrations/pg/0157_create_slo_workers.up.sql b/backend/app/migrations/pg/0157_create_slo_workers.up.sql
new file mode 100644
index 00000000..66e555a4
--- /dev/null
+++ b/backend/app/migrations/pg/0157_create_slo_workers.up.sql
@@ -0,0 +1,4 @@
+CREATE TABLE IF NOT EXISTS slo_workers (
+ owner TEXT PRIMARY KEY,
+ seen_at TIMESTAMPTZ NOT NULL
+)
diff --git a/backend/app/migrations/pg/0158_create_slo_shard_leases.up.sql b/backend/app/migrations/pg/0158_create_slo_shard_leases.up.sql
new file mode 100644
index 00000000..5a7363f2
--- /dev/null
+++ b/backend/app/migrations/pg/0158_create_slo_shard_leases.up.sql
@@ -0,0 +1,5 @@
+CREATE TABLE IF NOT EXISTS slo_shard_leases (
+ shard INTEGER PRIMARY KEY,
+ owner TEXT NOT NULL,
+ expires_at TIMESTAMPTZ NOT NULL
+)
diff --git a/backend/app/migrations/sqlite/0081_create_slos.up.sql b/backend/app/migrations/sqlite/0081_create_slos.up.sql
new file mode 100644
index 00000000..29140675
--- /dev/null
+++ b/backend/app/migrations/sqlite/0081_create_slos.up.sql
@@ -0,0 +1,33 @@
+CREATE TABLE IF NOT EXISTS slos (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+ name TEXT NOT NULL,
+ description TEXT NOT NULL DEFAULT '',
+ sli_kind TEXT NOT NULL,
+ sli_config TEXT NOT NULL DEFAULT '{}',
+ target_percent REAL NOT NULL,
+ window_days INTEGER NOT NULL DEFAULT 30,
+ bucket_minutes INTEGER NOT NULL DEFAULT 1,
+ enabled INTEGER NOT NULL DEFAULT 1,
+ status TEXT NOT NULL DEFAULT 'no_data',
+ current_sli REAL,
+ budget_remaining REAL,
+ burn_rate_1h REAL,
+ burn_rate_6h REAL,
+ computed_at DATETIME,
+ history_from DATETIME,
+ last_bucket_at DATETIME,
+ created_by INTEGER,
+ created_at DATETIME NOT NULL,
+ updated_at DATETIME NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS slos_project_idx ON slos (project_id);
+
+CREATE TABLE IF NOT EXISTS notification_rule_states (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ rule_id INTEGER NOT NULL UNIQUE REFERENCES notification_rules(id) ON DELETE CASCADE,
+ firing INTEGER NOT NULL DEFAULT 0,
+ since DATETIME,
+ updated_at DATETIME NOT NULL
+);
diff --git a/backend/app/migrations/sqlite/0082_add_slo_revision_and_gaps.up.sql b/backend/app/migrations/sqlite/0082_add_slo_revision_and_gaps.up.sql
new file mode 100644
index 00000000..bcf5fa3d
--- /dev/null
+++ b/backend/app/migrations/sqlite/0082_add_slo_revision_and_gaps.up.sql
@@ -0,0 +1,7 @@
+ALTER TABLE slos ADD COLUMN revision INTEGER NOT NULL DEFAULT 0;
+
+ALTER TABLE slos ADD COLUMN sli_since DATETIME;
+
+ALTER TABLE slos ADD COLUMN gap_from DATETIME;
+
+ALTER TABLE slos ADD COLUMN gap_to DATETIME;
diff --git a/backend/app/migrations/sqlite_telemetry/0028_create_slo_buckets.up.sql b/backend/app/migrations/sqlite_telemetry/0028_create_slo_buckets.up.sql
new file mode 100644
index 00000000..394c9214
--- /dev/null
+++ b/backend/app/migrations/sqlite_telemetry/0028_create_slo_buckets.up.sql
@@ -0,0 +1,9 @@
+CREATE TABLE IF NOT EXISTS slo_buckets (
+ project_id TEXT NOT NULL,
+ slo_id INTEGER NOT NULL,
+ bucket_start DATETIME NOT NULL,
+ good INTEGER NOT NULL DEFAULT 0,
+ total INTEGER NOT NULL DEFAULT 0,
+ computed_at DATETIME NOT NULL,
+ UNIQUE(project_id, slo_id, bucket_start)
+);
diff --git a/backend/app/models/models.go b/backend/app/models/models.go
index 2182b026..81a94aa3 100644
--- a/backend/app/models/models.go
+++ b/backend/app/models/models.go
@@ -71,6 +71,9 @@ func Init(driver lit.Driver) {
lit.RegisterModel[OutboxRuleEnqueue](driver)
lit.RegisterModel[OutboxHealthCounts](driver)
lit.RegisterModel[SyntheticCheck](driver)
+ lit.RegisterModel[SLO](driver)
+ lit.RegisterModel[SLOFreshness](driver)
+ lit.RegisterModel[NotificationRuleState](driver)
lit.RegisterModel[CheckRun](driver)
lit.RegisterModel[CheckRunHealthCounts](driver)
lit.RegisterModel[CheckIncident](driver)
diff --git a/backend/app/models/slo.model.go b/backend/app/models/slo.model.go
new file mode 100644
index 00000000..17632808
--- /dev/null
+++ b/backend/app/models/slo.model.go
@@ -0,0 +1,210 @@
+package models
+
+import (
+ "encoding/json"
+ "fmt"
+ "time"
+
+ "github.com/google/uuid"
+)
+
+const (
+ SLIKindEndpoint = "endpoint"
+ SLIKindSpan = "span"
+ SLIKindTask = "task"
+ SLIKindMonitor = "monitor"
+ SLIKindMetric = "metric"
+ SLIKindSession = "session"
+
+ SLOStatusOk = "ok"
+ SLOStatusBreached = "breached"
+ SLOStatusNoData = "no_data"
+
+ SLOGoodAvailability = "availability"
+ SLOGoodLatency = "latency"
+
+ SLOMaxSpanAttributeFilters = 10
+ SLOMaxMonitors = 50
+ SLOMaxMetricTagFilters = 20
+)
+
+var (
+ SLOWindowDays = []int{7, 30, 90}
+ SLOMetricSliceMinutes = []int{1, 5, 15, 60}
+ SLOMetricAggregations = map[string]bool{"avg": true, "min": true, "max": true, "sum": true, "last": true}
+ SLOMetricOperators = map[string]bool{"gt": true, "gte": true, "lt": true, "lte": true}
+)
+
+type SLO struct {
+ Id int `json:"id" lit:"id"`
+ ProjectId uuid.UUID `json:"projectId" lit:"project_id"`
+ Name string `json:"name" lit:"name"`
+ Description string `json:"description" lit:"description"`
+ SliKind string `json:"sliKind" lit:"sli_kind"`
+ SliConfig JSONText `json:"sliConfig" lit:"sli_config"`
+ TargetPercent float64 `json:"targetPercent" lit:"target_percent"`
+ WindowDays int `json:"windowDays" lit:"window_days"`
+ BucketMinutes int `json:"bucketMinutes" lit:"bucket_minutes"`
+ Enabled bool `json:"enabled" lit:"enabled"`
+ Status string `json:"status" lit:"status"`
+ CurrentSli *float64 `json:"currentSli" lit:"current_sli"`
+ BudgetRemaining *float64 `json:"budgetRemaining" lit:"budget_remaining"`
+ BurnRate1h *float64 `json:"burnRate1h" lit:"burn_rate_1h"`
+ BurnRate6h *float64 `json:"burnRate6h" lit:"burn_rate_6h"`
+ ComputedAt *time.Time `json:"computedAt" lit:"computed_at"`
+ HistoryFrom *time.Time `json:"historyFrom" lit:"history_from"`
+ LastBucketAt *time.Time `json:"lastBucketAt" lit:"last_bucket_at"`
+ SliSince *time.Time `json:"sliSince" lit:"sli_since"`
+ GapFrom *time.Time `json:"-" lit:"gap_from"`
+ GapTo *time.Time `json:"-" lit:"gap_to"`
+ Revision int `json:"revision" lit:"revision"`
+ CreatedBy *int `json:"createdBy" lit:"created_by"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+ UpdatedAt time.Time `json:"updatedAt" lit:"updated_at"`
+}
+
+func (s *SLO) Target() float64 {
+ return s.TargetPercent / 100
+}
+
+func (s *SLO) BucketDuration() time.Duration {
+ if s.BucketMinutes < 1 {
+ return time.Minute
+ }
+ return time.Duration(s.BucketMinutes) * time.Minute
+}
+
+func (s *SLO) MissingCountsFrom() time.Time {
+ if s.SliSince != nil {
+ return s.SliSince.UTC()
+ }
+ return s.CreatedAt.UTC()
+}
+
+func (s *SLO) Window() time.Duration {
+ return time.Duration(s.WindowDays) * 24 * time.Hour
+}
+
+type SLOSnapshot struct {
+ Status string
+ CurrentSli *float64
+ BudgetRemaining *float64
+ BurnRate1h *float64
+ BurnRate6h *float64
+ ComputedAt time.Time
+ HistoryFrom *time.Time
+ LastBucketAt *time.Time
+ GapFrom *time.Time
+ GapTo *time.Time
+}
+
+type SLOFreshness struct {
+ Id int `lit:"id"`
+ WindowDays int `lit:"window_days"`
+ ComputedAt *time.Time `lit:"computed_at"`
+ HistoryFrom *time.Time `lit:"history_from"`
+ LastBucketAt *time.Time `lit:"last_bucket_at"`
+ GapFrom *time.Time `lit:"gap_from"`
+ CreatedAt time.Time `lit:"created_at"`
+}
+
+type NotificationRuleState struct {
+ Id int `json:"id" lit:"id"`
+ RuleId int `json:"ruleId" lit:"rule_id"`
+ Firing bool `json:"firing" lit:"firing"`
+ Since *time.Time `json:"since" lit:"since"`
+ UpdatedAt time.Time `json:"updatedAt" lit:"updated_at"`
+}
+
+type EndpointSLIConfig struct {
+ Endpoint string `json:"endpoint"`
+ Good string `json:"good"`
+ LatencyThresholdMs float64 `json:"latencyThresholdMs"`
+}
+
+type SpanSLIAttribute struct {
+ Key string `json:"key"`
+ Value string `json:"value"`
+}
+
+type SpanSLIConfig struct {
+ ServiceName string `json:"serviceName"`
+ Name string `json:"name"`
+ Kind *int32 `json:"kind"`
+ Attributes []SpanSLIAttribute `json:"attributes"`
+ Good string `json:"good"`
+ LatencyThresholdMs float64 `json:"latencyThresholdMs"`
+}
+
+type TaskSLIConfig struct {
+ TaskName string `json:"taskName"`
+}
+
+type MonitorSLIConfig struct {
+ CheckIds []int `json:"checkIds"`
+}
+
+type SessionSLIConfig struct {
+ AppVersion string `json:"appVersion"`
+ IncludeMessages bool `json:"includeMessages"`
+}
+
+type MetricSLIConfig struct {
+ MetricName string `json:"metricName"`
+ Tags map[string]string `json:"tags"`
+ Aggregation string `json:"aggregation"`
+ Operator string `json:"operator"`
+ Threshold float64 `json:"threshold"`
+ MissingIsBad bool `json:"missingIsBad"`
+}
+
+type SLIConfig struct {
+ Endpoint *EndpointSLIConfig
+ Span *SpanSLIConfig
+ Task *TaskSLIConfig
+ Monitor *MonitorSLIConfig
+ Metric *MetricSLIConfig
+ Session *SessionSLIConfig
+}
+
+func ParseSLIConfig(kind string, raw []byte) (*SLIConfig, error) {
+ if len(raw) == 0 {
+ raw = []byte("{}")
+ }
+ cfg := &SLIConfig{}
+ var err error
+ switch kind {
+ case SLIKindEndpoint:
+ cfg.Endpoint = &EndpointSLIConfig{}
+ err = json.Unmarshal(raw, cfg.Endpoint)
+ case SLIKindSpan:
+ cfg.Span = &SpanSLIConfig{}
+ err = json.Unmarshal(raw, cfg.Span)
+ case SLIKindTask:
+ cfg.Task = &TaskSLIConfig{}
+ err = json.Unmarshal(raw, cfg.Task)
+ case SLIKindMonitor:
+ cfg.Monitor = &MonitorSLIConfig{}
+ err = json.Unmarshal(raw, cfg.Monitor)
+ case SLIKindMetric:
+ cfg.Metric = &MetricSLIConfig{}
+ err = json.Unmarshal(raw, cfg.Metric)
+ case SLIKindSession:
+ cfg.Session = &SessionSLIConfig{}
+ err = json.Unmarshal(raw, cfg.Session)
+ default:
+ return nil, fmt.Errorf("unknown SLI kind %q", kind)
+ }
+ if err != nil {
+ return nil, fmt.Errorf("invalid %s SLI config: %w", kind, err)
+ }
+ return cfg, nil
+}
+
+func ValidSLIKind(kind string) bool {
+ switch kind {
+ case SLIKindEndpoint, SLIKindSpan, SLIKindTask, SLIKindMonitor, SLIKindMetric, SLIKindSession:
+ return true
+ }
+ return false
+}
diff --git a/backend/app/monitoring/slo_reporter.go b/backend/app/monitoring/slo_reporter.go
new file mode 100644
index 00000000..8c674f3b
--- /dev/null
+++ b/backend/app/monitoring/slo_reporter.go
@@ -0,0 +1,83 @@
+package monitoring
+
+import (
+ "context"
+ "fmt"
+ "time"
+
+ traceway "go.tracewayapp.com"
+
+ "github.com/tracewayapp/traceway/backend/app/slo"
+)
+
+const sloReportInterval = 30 * time.Second
+
+type sloBaselines struct {
+ computed uint64
+ errors uint64
+ timeouts uint64
+ lastTick time.Time
+ first bool
+}
+
+func StartSLOReporter(ctx context.Context) {
+ go func() {
+ defer traceway.Recover()
+
+ ticker := time.NewTicker(sloReportInterval)
+ defer ticker.Stop()
+
+ baselines := &sloBaselines{first: true}
+ reportSLOOnce(ctx, baselines)
+
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-ticker.C:
+ reportSLOOnce(ctx, baselines)
+ }
+ }
+ }()
+}
+
+func reportSLOOnce(ctx context.Context, baselines *sloBaselines) {
+ stats, err := slo.HealthSnapshot(ctx)
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to snapshot slo health for metrics: %w", err))
+ }
+ if stats == nil {
+ return
+ }
+ if stats.Freshness != nil {
+ traceway.CaptureMetric("traceway.slo.enabled", float64(stats.Freshness.Enabled))
+ traceway.CaptureMetric("traceway.slo.stale", float64(stats.Freshness.Stale))
+ traceway.CaptureMetric("traceway.slo.max_staleness_sec", stats.Freshness.MaxStalenessSec)
+ traceway.CaptureMetric("traceway.slo.backfill_pending", float64(stats.Freshness.BackfillPending))
+ }
+ traceway.CaptureMetric("traceway.slo.tick.running_sec", stats.RunningForSec)
+ traceway.CaptureMetric("traceway.slo.shards_owned", float64(stats.OwnedShards))
+ traceway.CaptureMetric("traceway.slo.instances", float64(stats.Instances))
+ if tick := stats.LastTick; tick != nil && !tick.At.Equal(baselines.lastTick) {
+ traceway.CaptureMetric("traceway.slo.tick.duration_ms", tick.DurationMs)
+ traceway.CaptureMetric("traceway.slo.tick.compute_ms", tick.ComputeMs)
+ traceway.CaptureMetric("traceway.slo.tick.store_ms", tick.StoreMs)
+ traceway.CaptureMetric("traceway.slo.tick.snapshot_ms", tick.SnapshotMs)
+ traceway.CaptureMetric("traceway.slo.tick.slos", float64(tick.SLOs))
+ traceway.CaptureMetric("traceway.slo.tick.buckets", float64(tick.Buckets))
+ traceway.CaptureMetric("traceway.slo.tick.full_snapshots", float64(tick.FullSnapshots))
+ if tick.Slowest != nil {
+ traceway.CaptureMetricWithTags("traceway.slo.tick.slowest_ms", tick.Slowest.DurationMs, map[string]string{"kind": tick.Slowest.Kind})
+ }
+ baselines.lastTick = tick.At
+ }
+ if !baselines.first {
+ traceway.CaptureMetric("traceway.slo.computed.delta", float64(stats.ComputedTotal-baselines.computed))
+ traceway.CaptureMetric("traceway.slo.errors.delta", float64(stats.ErrorsTotal-baselines.errors))
+ traceway.CaptureMetric("traceway.slo.timeouts.delta", float64(stats.TimeoutsTotal-baselines.timeouts))
+ }
+ baselines.computed = stats.ComputedTotal
+ baselines.errors = stats.ErrorsTotal
+ baselines.timeouts = stats.TimeoutsTotal
+ baselines.first = false
+}
diff --git a/backend/app/notifications/dispatch.go b/backend/app/notifications/dispatch.go
index 893f905e..b413e76f 100644
--- a/backend/app/notifications/dispatch.go
+++ b/backend/app/notifications/dispatch.go
@@ -94,7 +94,9 @@ func dispatch(rule *models.NotificationRuleWithChannel, msg Message) bool {
return false
}
- cooldowns.recordFire(rule.Id)
+ if !msg.Recovered {
+ cooldowns.recordFire(rule.Id)
+ }
outbox.Wake()
return true
}
diff --git a/backend/app/notifications/evaluator.go b/backend/app/notifications/evaluator.go
index 14c1b14c..34f1f80e 100644
--- a/backend/app/notifications/evaluator.go
+++ b/backend/app/notifications/evaluator.go
@@ -77,7 +77,7 @@ func evaluatePolledRules(ctx context.Context) {
continue
}
- if !cooldowns.canFire(rule.Id, rule.CooldownMinutes) {
+ if !statefulRuleTypes[rule.RuleType] && !cooldowns.canFire(rule.Id, rule.CooldownMinutes) {
continue
}
@@ -106,13 +106,17 @@ func evaluatePolledRules(ctx context.Context) {
continue
}
+ if result != nil && result.Recovered {
+ dispatchRecovery(rule, result.Message)
+ continue
+ }
if result != nil && result.Fired {
if len(result.Messages) > 0 {
for _, msg := range result.Messages {
dispatch(rule, msg)
}
- } else {
- dispatch(rule, result.Message)
+ } else if !dispatch(rule, result.Message) && statefulRuleTypes[rule.RuleType] {
+ rollBackRuleState(rule.Id)
}
}
}
diff --git a/backend/app/notifications/evaluator_polled.go b/backend/app/notifications/evaluator_polled.go
index 78783b7d..58a50597 100644
--- a/backend/app/notifications/evaluator_polled.go
+++ b/backend/app/notifications/evaluator_polled.go
@@ -14,9 +14,10 @@ import (
)
type EvalResult struct {
- Fired bool
- Message Message
- Messages []Message
+ Fired bool
+ Recovered bool
+ Message Message
+ Messages []Message
}
type RuleEvaluator func(ctx context.Context, rule *models.NotificationRule, projectId uuid.UUID) (*EvalResult, error)
@@ -36,6 +37,8 @@ var polledEvaluators = map[string]RuleEvaluator{
"impact_score_critical": evaluateImpactScoreCritical,
"impact_score_high": evaluateImpactScoreHigh,
"impact_score_medium": evaluateImpactScoreMedium,
+ "slo_burn_rate": evaluateSLOBurnRate,
+ "slo_budget_consumed": evaluateSLOBudgetConsumed,
}
// --- Error Rate Threshold ---
diff --git a/backend/app/notifications/evaluator_polled_sqlite.go b/backend/app/notifications/evaluator_polled_sqlite.go
index 4fe874f1..a94ed7d9 100644
--- a/backend/app/notifications/evaluator_polled_sqlite.go
+++ b/backend/app/notifications/evaluator_polled_sqlite.go
@@ -16,9 +16,10 @@ import (
)
type EvalResult struct {
- Fired bool
- Message Message
- Messages []Message
+ Fired bool
+ Recovered bool
+ Message Message
+ Messages []Message
}
type RuleEvaluator func(ctx context.Context, rule *models.NotificationRule, projectId uuid.UUID) (*EvalResult, error)
@@ -38,6 +39,8 @@ var polledEvaluators = map[string]RuleEvaluator{
"impact_score_critical": evaluateImpactScoreCritical,
"impact_score_high": evaluateImpactScoreHigh,
"impact_score_medium": evaluateImpactScoreMedium,
+ "slo_burn_rate": evaluateSLOBurnRate,
+ "slo_budget_consumed": evaluateSLOBudgetConsumed,
}
// --- Error Rate Threshold ---
diff --git a/backend/app/notifications/messages.go b/backend/app/notifications/messages.go
index 0cc6a992..cd1bf949 100644
--- a/backend/app/notifications/messages.go
+++ b/backend/app/notifications/messages.go
@@ -563,3 +563,75 @@ func TestContactMethodMessage() Message {
},
}
}
+
+func buildSLOBurnRateMessage(entity *models.SLO, longBurn, shortBurn float64, longMinutes, shortMinutes int, threshold float64, projectName string) Message {
+ severity := SeverityWarning
+ if longBurn >= threshold*2 {
+ severity = SeverityCritical
+ }
+ headline := fmt.Sprintf("The SLO %q is burning its error budget at %.1fx over the last %s and %.1fx over the last %s (threshold %.1fx).",
+ entity.Name, longBurn, windowLabel(longMinutes), shortBurn, windowLabel(shortMinutes), threshold)
+ return Message{
+ Subject: fmt.Sprintf("[%s] SLO %q burn rate %.1fx", projectName, entity.Name, longBurn),
+ Body: headline,
+ Severity: severity,
+ URL: fmt.Sprintf("/slos/%d", entity.Id),
+ DedupToken: SLODedupToken(entity.Id),
+ Email: alertEmail(models.EmailAlert{
+ ProjectName: projectName,
+ Headline: headline,
+ ScopeLabel: "SLO",
+ Scope: entity.Name,
+ Observed: fmt.Sprintf("%.1fx over %s, %.1fx over %s", longBurn, windowLabel(longMinutes), shortBurn, windowLabel(shortMinutes)),
+ Threshold: fmt.Sprintf("%.1fx", threshold),
+ WindowMins: longMinutes,
+ }),
+ }
+}
+
+func buildSLOBudgetConsumedMessage(entity *models.SLO, consumed, threshold float64, projectName string) Message {
+ severity := SeverityWarning
+ if consumed >= 100 {
+ severity = SeverityCritical
+ }
+ headline := fmt.Sprintf("The SLO %q has consumed %.1f%% of its %d-day error budget (threshold %.0f%%).", entity.Name, consumed, entity.WindowDays, threshold)
+ subject := fmt.Sprintf("[%s] SLO %q budget %.0f%% consumed", projectName, entity.Name, consumed)
+ if consumed >= 100 {
+ headline = fmt.Sprintf("The SLO %q has exhausted its %d-day error budget: %.0f%% consumed against a threshold of %.0f%%.", entity.Name, entity.WindowDays, consumed, threshold)
+ subject = fmt.Sprintf("[%s] SLO %q error budget exhausted", projectName, entity.Name)
+ }
+ return Message{
+ Subject: subject,
+ Body: headline,
+ Severity: severity,
+ URL: fmt.Sprintf("/slos/%d", entity.Id),
+ DedupToken: SLODedupToken(entity.Id),
+ Email: alertEmail(models.EmailAlert{
+ ProjectName: projectName,
+ Headline: headline,
+ ScopeLabel: "SLO",
+ Scope: entity.Name,
+ Observed: fmt.Sprintf("%.1f%% consumed", consumed),
+ Threshold: fmt.Sprintf("%.0f%%", threshold),
+ WindowMins: entity.WindowDays * 1440,
+ }),
+ }
+}
+
+func buildSLORecoveredMessage(entity *models.SLO, detail string, projectName string) Message {
+ headline := fmt.Sprintf("The SLO %q recovered. %s", entity.Name, detail)
+ return Message{
+ Subject: fmt.Sprintf("[%s] SLO %q recovered", projectName, entity.Name),
+ Body: headline,
+ Severity: SeverityInfo,
+ Recovered: true,
+ URL: fmt.Sprintf("/slos/%d", entity.Id),
+ DedupToken: SLODedupToken(entity.Id),
+ Email: alertEmail(models.EmailAlert{
+ ProjectName: projectName,
+ Headline: headline,
+ ScopeLabel: "SLO",
+ Scope: entity.Name,
+ }),
+ }
+}
diff --git a/backend/app/notifications/slo_rules.go b/backend/app/notifications/slo_rules.go
new file mode 100644
index 00000000..edce8aff
--- /dev/null
+++ b/backend/app/notifications/slo_rules.go
@@ -0,0 +1,233 @@
+package notifications
+
+import (
+ "context"
+ "database/sql"
+ "encoding/json"
+ "fmt"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ "github.com/tracewayapp/traceway/backend/app/slo"
+ traceway "go.tracewayapp.com"
+)
+
+var statefulRuleTypes = map[string]bool{
+ "slo_burn_rate": true,
+ "slo_budget_consumed": true,
+}
+
+const (
+ defaultBurnLongWindowMinutes = 60
+ defaultBurnShortWindowMinutes = 5
+ defaultBurnRateThreshold = 14.4
+ defaultBudgetConsumedPercent = 80
+)
+
+func SLODedupToken(sloId int) string {
+ return fmt.Sprintf("slo:%d", sloId)
+}
+
+type sloBurnRateConfig struct {
+ SloId int `json:"sloId"`
+ LongWindowMinutes int `json:"longWindowMinutes"`
+ ShortWindowMinutes int `json:"shortWindowMinutes"`
+ BurnRateThreshold float64 `json:"burnRateThreshold"`
+}
+
+type sloBudgetConsumedConfig struct {
+ SloId int `json:"sloId"`
+ ConsumedPercent float64 `json:"consumedPercent"`
+}
+
+func loadRuleSLO(sloId int, projectId uuid.UUID) (*models.SLO, error) {
+ entity, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.SLO, error) {
+ return transactional.SLORepository.FindByIdForProject(tx, sloId, projectId)
+ })
+ if err != nil {
+ return nil, err
+ }
+ if entity == nil {
+ return nil, fmt.Errorf("SLO %d not found in project", sloId)
+ }
+ return entity, nil
+}
+
+func evaluateSLOBurnRate(ctx context.Context, rule *models.NotificationRule, projectId uuid.UUID) (*EvalResult, error) {
+ var cfg sloBurnRateConfig
+ if err := json.Unmarshal(rule.Config, &cfg); err != nil {
+ return nil, fmt.Errorf("invalid slo_burn_rate config: %w", err)
+ }
+ if cfg.LongWindowMinutes <= 0 {
+ cfg.LongWindowMinutes = defaultBurnLongWindowMinutes
+ }
+ if cfg.ShortWindowMinutes <= 0 {
+ cfg.ShortWindowMinutes = defaultBurnShortWindowMinutes
+ }
+ if cfg.BurnRateThreshold <= 0 {
+ cfg.BurnRateThreshold = defaultBurnRateThreshold
+ }
+ entity, err := loadRuleSLO(cfg.SloId, projectId)
+ if err != nil {
+ return nil, err
+ }
+ now := time.Now().UTC()
+ if !entity.Enabled {
+ return recoverDisabledSLO(rule, entity, projectId, now)
+ }
+ if entity.LastBucketAt == nil {
+ return &EvalResult{}, nil
+ }
+ cfg.LongWindowMinutes = sliceAligned(cfg.LongWindowMinutes, entity.BucketMinutes)
+ cfg.ShortWindowMinutes = sliceAligned(cfg.ShortWindowMinutes, entity.BucketMinutes)
+
+ long, err := slo.WindowSums(ctx, entity, time.Duration(cfg.LongWindowMinutes)*time.Minute, now)
+ if err != nil {
+ return nil, err
+ }
+ short, err := slo.WindowSums(ctx, entity, time.Duration(cfg.ShortWindowMinutes)*time.Minute, now)
+ if err != nil {
+ return nil, err
+ }
+ target := entity.Target()
+ longBurn := slo.BurnRate(long, target)
+ shortBurn := slo.BurnRate(short, target)
+ met := long.Total > 0 && longBurn >= cfg.BurnRateThreshold && shortBurn >= cfg.BurnRateThreshold
+ clear := longBurn < cfg.BurnRateThreshold || shortBurn < cfg.BurnRateThreshold
+
+ fire, recover, err := transitionRuleState(rule, met, clear, now)
+ if err != nil {
+ return nil, err
+ }
+ switch {
+ case fire:
+ return &EvalResult{Fired: true, Message: buildSLOBurnRateMessage(entity, longBurn, shortBurn, cfg.LongWindowMinutes, cfg.ShortWindowMinutes, cfg.BurnRateThreshold, getProjectName(projectId))}, nil
+ case recover:
+ detail := fmt.Sprintf("The burn rate is back to %.1fx over the last %s and %.1fx over the last %s (threshold %.1fx).", longBurn, windowLabel(cfg.LongWindowMinutes), shortBurn, windowLabel(cfg.ShortWindowMinutes), cfg.BurnRateThreshold)
+ return &EvalResult{Recovered: true, Message: buildSLORecoveredMessage(entity, detail, getProjectName(projectId))}, nil
+ }
+ return &EvalResult{}, nil
+}
+
+func evaluateSLOBudgetConsumed(_ context.Context, rule *models.NotificationRule, projectId uuid.UUID) (*EvalResult, error) {
+ var cfg sloBudgetConsumedConfig
+ if err := json.Unmarshal(rule.Config, &cfg); err != nil {
+ return nil, fmt.Errorf("invalid slo_budget_consumed config: %w", err)
+ }
+ if cfg.ConsumedPercent <= 0 {
+ cfg.ConsumedPercent = defaultBudgetConsumedPercent
+ }
+ entity, err := loadRuleSLO(cfg.SloId, projectId)
+ if err != nil {
+ return nil, err
+ }
+ now := time.Now().UTC()
+ if !entity.Enabled {
+ return recoverDisabledSLO(rule, entity, projectId, now)
+ }
+ if entity.LastBucketAt == nil {
+ return &EvalResult{}, nil
+ }
+
+ ok := entity.BudgetRemaining != nil
+ consumed := 0.0
+ if ok {
+ consumed = 100 - *entity.BudgetRemaining
+ }
+ met := ok && consumed >= cfg.ConsumedPercent
+ clear := !ok || consumed < cfg.ConsumedPercent
+
+ fire, recover, err := transitionRuleState(rule, met, clear, now)
+ if err != nil {
+ return nil, err
+ }
+ switch {
+ case fire:
+ return &EvalResult{Fired: true, Message: buildSLOBudgetConsumedMessage(entity, consumed, cfg.ConsumedPercent, getProjectName(projectId))}, nil
+ case recover:
+ detail := fmt.Sprintf("%.1f%% of the %d-day error budget is consumed, below the %.0f%% line again.", consumed, entity.WindowDays, cfg.ConsumedPercent)
+ if !ok {
+ detail = fmt.Sprintf("The SLO has no data in its %d-day window.", entity.WindowDays)
+ }
+ return &EvalResult{Recovered: true, Message: buildSLORecoveredMessage(entity, detail, getProjectName(projectId))}, nil
+ }
+ return &EvalResult{}, nil
+}
+
+func sliceAligned(minutes, bucketMinutes int) int {
+ if bucketMinutes <= 1 {
+ return minutes
+ }
+ return (minutes + bucketMinutes - 1) / bucketMinutes * bucketMinutes
+}
+
+func recoverDisabledSLO(rule *models.NotificationRule, entity *models.SLO, projectId uuid.UUID, now time.Time) (*EvalResult, error) {
+ _, recover, err := transitionRuleState(rule, false, true, now)
+ if err != nil || !recover {
+ return &EvalResult{}, err
+ }
+ return &EvalResult{Recovered: true, Message: buildSLORecoveredMessage(entity, "The SLO was disabled.", getProjectName(projectId))}, nil
+}
+
+func transitionRuleState(rule *models.NotificationRule, met bool, clear bool, now time.Time) (fire bool, recover bool, err error) {
+ state, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.NotificationRuleState, error) {
+ return transactional.NotificationRuleStateRepository.Find(tx, rule.Id)
+ })
+ if err != nil {
+ return false, false, err
+ }
+ firing := state != nil && state.Firing
+ switch {
+ case !firing && met:
+ if !cooldowns.canFire(rule.Id, rule.CooldownMinutes) {
+ return false, false, nil
+ }
+ fire = true
+ case firing && clear:
+ recover = true
+ default:
+ return false, false, nil
+ }
+ changed, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return transactional.NotificationRuleStateRepository.Transition(tx, rule.Id, fire, now)
+ })
+ if err != nil || !changed {
+ return false, false, err
+ }
+ return fire, recover, nil
+}
+
+func rollBackRuleState(ruleId int) {
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return transactional.NotificationRuleStateRepository.Transition(tx, ruleId, false, time.Now().UTC())
+ })
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to roll back the state of rule %d after a failed send: %w", ruleId, err))
+ }
+}
+
+func dispatchRecovery(rule *models.NotificationRuleWithChannel, msg Message) {
+ if rule.ChannelType == "escalation" {
+ if pageResolver == nil {
+ return
+ }
+ if err := pageResolver(rule.Id, msg.DedupToken); err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to auto-resolve page for recovered rule %d: %w", rule.Id, err))
+ }
+ return
+ }
+ dispatch(rule, msg)
+}
+
+func windowLabel(minutes int) string {
+ switch {
+ case minutes%1440 == 0:
+ return fmt.Sprintf("%dd", minutes/1440)
+ case minutes%60 == 0:
+ return fmt.Sprintf("%dh", minutes/60)
+ }
+ return fmt.Sprintf("%dm", minutes)
+}
diff --git a/backend/app/notifications/slo_rules_sqlite_test.go b/backend/app/notifications/slo_rules_sqlite_test.go
new file mode 100644
index 00000000..b9a17fe0
--- /dev/null
+++ b/backend/app/notifications/slo_rules_sqlite_test.go
@@ -0,0 +1,393 @@
+//go:build !transactional_pg && !telemetry_ch && !telemetry_duckdb
+
+package notifications
+
+import (
+ "context"
+ "database/sql"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+type sloRuleFixture struct {
+ ProjectId uuid.UUID
+ SLO *models.SLO
+ Rule *models.NotificationRule
+}
+
+func setupSLORuleDB(t *testing.T, ruleType, config string, cooldownMinutes int) *sloRuleFixture {
+ t.Helper()
+ dbtest.SetupSQLite(t)
+ t.Cleanup(func() {
+ cooldowns.mu.Lock()
+ cooldowns.fired = make(map[int]time.Time)
+ cooldowns.mu.Unlock()
+ })
+ fixture := &sloRuleFixture{}
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ project, err := transactional.ProjectRepository.Create(tx, "api", "gin")
+ if err != nil {
+ return struct{}{}, err
+ }
+ fixture.ProjectId = project.Id
+ now := time.Now().UTC()
+ entity := &models.SLO{
+ ProjectId: project.Id, Name: "Checkout availability", SliKind: models.SLIKindEndpoint, SliConfig: models.JSONText(`{"endpoint":"","good":"availability"}`),
+ TargetPercent: 99, WindowDays: 7, BucketMinutes: 1, Enabled: true, Status: models.SLOStatusNoData, LastBucketAt: &now, CreatedAt: now, UpdatedAt: now,
+ }
+ sloId, err := transactional.SLORepository.Create(tx, entity)
+ if err != nil {
+ return struct{}{}, err
+ }
+ entity.Id = sloId
+ fixture.SLO = entity
+ channel := &models.NotificationChannel{ProjectId: project.Id, Name: "Ops", ChannelType: "slack", Config: []byte(`{"webhookUrl":"https://hooks.example.com/x"}`), Enabled: true, CreatedAt: now, UpdatedAt: now}
+ channelId, err := transactional.NotificationChannelRepository.Create(tx, channel)
+ if err != nil {
+ return struct{}{}, err
+ }
+ rule := &models.NotificationRule{ProjectId: project.Id, ChannelId: channelId, Name: "burn", RuleType: ruleType, Config: []byte(config), Enabled: true, CooldownMinutes: cooldownMinutes, CreatedAt: now, UpdatedAt: now}
+ ruleId, err := transactional.NotificationRuleRepository.Create(tx, rule)
+ if err != nil {
+ return struct{}{}, err
+ }
+ rule.Id = ruleId
+ fixture.Rule = rule
+ return struct{}{}, nil
+ })
+ if err != nil {
+ t.Fatalf("seed: %v", err)
+ }
+ return fixture
+}
+
+func seedBuckets(t *testing.T, fixture *sloRuleFixture, minutes int, good, total int64) {
+ t.Helper()
+ end := time.Now().UTC().Truncate(time.Minute)
+ buckets := make([]shared.SLOBucket, 0, minutes)
+ for i := 1; i <= minutes; i++ {
+ buckets = append(buckets, shared.SLOBucket{ProjectId: fixture.ProjectId, SLOId: fixture.SLO.Id, BucketStart: end.Add(-time.Duration(i) * time.Minute), Good: good, Total: total, ComputedAt: end})
+ }
+ if err := telemetry.SLOBucketRepository.Upsert(context.Background(), buckets); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func ruleState(t *testing.T, ruleId int) bool {
+ t.Helper()
+ state, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.NotificationRuleState, error) {
+ return transactional.NotificationRuleStateRepository.Find(tx, ruleId)
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ return state != nil && state.Firing
+}
+
+func TestSLOBurnRateFiresOnceAndRecovers(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_burn_rate", `{"sloId":1,"longWindowMinutes":60,"shortWindowMinutes":5,"burnRateThreshold":6}`, 0)
+ ctx := context.Background()
+ seedBuckets(t, fixture, 60, 90, 100)
+
+ result, err := evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.Fired || result.Recovered {
+ t.Fatalf("expected a fire: %+v", result)
+ }
+ if result.Message.Severity != SeverityWarning || result.Message.DedupToken != SLODedupToken(fixture.SLO.Id) || result.Message.URL != "/slos/1" {
+ t.Fatalf("message: %+v", result.Message)
+ }
+ if !ruleState(t, fixture.Rule.Id) {
+ t.Fatal("firing state not persisted")
+ }
+
+ result, err = evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if result.Fired || result.Recovered {
+ t.Fatalf("a burning SLO must not fire again: %+v", result)
+ }
+
+ seedBuckets(t, fixture, 60, 100, 100)
+ result, err = evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.Recovered || result.Fired || !result.Message.Recovered || result.Message.Severity != SeverityInfo {
+ t.Fatalf("expected a recovery: %+v", result)
+ }
+ if ruleState(t, fixture.Rule.Id) {
+ t.Fatal("firing state not cleared")
+ }
+ result, err = evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || result.Fired || result.Recovered {
+ t.Fatalf("a recovered SLO is quiet: %+v, %v", result, err)
+ }
+}
+
+func TestSLOBurnRateCriticalAndShortWindow(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_burn_rate", `{"sloId":1,"longWindowMinutes":60,"shortWindowMinutes":5,"burnRateThreshold":6}`, 0)
+ ctx := context.Background()
+ seedBuckets(t, fixture, 60, 80, 100)
+ end := time.Now().UTC().Truncate(time.Minute)
+ var clean []shared.SLOBucket
+ for i := 1; i <= 5; i++ {
+ clean = append(clean, shared.SLOBucket{ProjectId: fixture.ProjectId, SLOId: fixture.SLO.Id, BucketStart: end.Add(-time.Duration(i) * time.Minute), Good: 100, Total: 100, ComputedAt: end})
+ }
+ if err := telemetry.SLOBucketRepository.Upsert(ctx, clean); err != nil {
+ t.Fatal(err)
+ }
+ result, err := evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || result.Fired {
+ t.Fatalf("short window must also burn: %+v, %v", result, err)
+ }
+ seedBuckets(t, fixture, 60, 80, 100)
+ result, err = evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || !result.Fired || result.Message.Severity != SeverityCritical {
+ t.Fatalf("20x against 6x is critical: %+v, %v", result, err)
+ }
+}
+
+func TestSLOBurnRateHonoursCooldown(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_burn_rate", `{"sloId":1,"longWindowMinutes":60,"shortWindowMinutes":5,"burnRateThreshold":6}`, 15)
+ ctx := context.Background()
+ seedBuckets(t, fixture, 60, 90, 100)
+ cooldowns.recordFire(fixture.Rule.Id)
+
+ result, err := evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || result.Fired {
+ t.Fatalf("cooldown suppresses the fire: %+v, %v", result, err)
+ }
+ if ruleState(t, fixture.Rule.Id) {
+ t.Fatal("a suppressed fire must not be recorded as firing")
+ }
+ cooldowns.mu.Lock()
+ delete(cooldowns.fired, fixture.Rule.Id)
+ cooldowns.mu.Unlock()
+ result, err = evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || !result.Fired {
+ t.Fatalf("fires once the cooldown passed: %+v, %v", result, err)
+ }
+}
+
+func TestSLORecoveryDoesNotRestartTheCooldown(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_burn_rate", `{"sloId":1,"longWindowMinutes":60,"shortWindowMinutes":5,"burnRateThreshold":6}`, 15)
+ ctx := context.Background()
+ seedBuckets(t, fixture, 60, 90, 100)
+ evaluatePolledRules(ctx)
+ if !ruleState(t, fixture.Rule.Id) {
+ t.Fatal("expected the first fire")
+ }
+ cooldowns.mu.Lock()
+ cooldowns.fired[fixture.Rule.Id] = time.Now().Add(-40 * time.Minute)
+ cooldowns.mu.Unlock()
+
+ seedBuckets(t, fixture, 60, 100, 100)
+ evaluatePolledRules(ctx)
+ if ruleState(t, fixture.Rule.Id) {
+ t.Fatal("expected the recovery")
+ }
+ seedBuckets(t, fixture, 60, 90, 100)
+ evaluatePolledRules(ctx)
+ if !ruleState(t, fixture.Rule.Id) {
+ t.Fatal("a burn 40 minutes after the last alert must fire again; the recovery notice is not a fire")
+ }
+}
+
+func setBudgetRemaining(t *testing.T, fixture *sloRuleFixture, remaining *float64) {
+ t.Helper()
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ _, err := tx.Exec("UPDATE slos SET budget_remaining = ? WHERE id = ?", remaining, fixture.SLO.Id)
+ return struct{}{}, err
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+}
+
+func budget(remaining float64) *float64 {
+ return &remaining
+}
+
+func TestSLOBudgetConsumedFiresAndRecovers(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_budget_consumed", `{"sloId":1,"consumedPercent":80}`, 0)
+ ctx := context.Background()
+ setBudgetRemaining(t, fixture, budget(15))
+
+ result, err := evaluateSLOBudgetConsumed(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || !result.Fired || result.Message.Severity != SeverityWarning {
+ t.Fatalf("expected a warning fire: %+v, %v", result, err)
+ }
+ setBudgetRemaining(t, fixture, budget(50))
+ result, err = evaluateSLOBudgetConsumed(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || !result.Recovered {
+ t.Fatalf("half the budget back is a recovery: %+v, %v", result, err)
+ }
+ setBudgetRemaining(t, fixture, budget(-100))
+ result, err = evaluateSLOBudgetConsumed(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || !result.Fired || result.Message.Severity != SeverityCritical {
+ t.Fatalf("an exhausted budget is critical: %+v, %v", result, err)
+ }
+}
+
+func TestSLOBurnRateShortWindowCoversOneSlice(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_burn_rate", `{"sloId":1,"longWindowMinutes":60,"shortWindowMinutes":5,"burnRateThreshold":6}`, 0)
+ ctx := context.Background()
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ _, err := tx.Exec("UPDATE slos SET bucket_minutes = 60 WHERE id = ?", fixture.SLO.Id)
+ return struct{}{}, err
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ lastSlice := time.Now().UTC().Truncate(time.Hour).Add(-time.Hour)
+ bucket := shared.SLOBucket{ProjectId: fixture.ProjectId, SLOId: fixture.SLO.Id, BucketStart: lastSlice, Good: 0, Total: 1, ComputedAt: time.Now().UTC()}
+ if err := telemetry.SLOBucketRepository.Upsert(ctx, []shared.SLOBucket{bucket}); err != nil {
+ t.Fatal(err)
+ }
+
+ result, err := evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.Fired {
+ t.Fatalf("a 5m short window on hourly slices must read the last slice and fire: %+v", result)
+ }
+}
+
+func TestSLORuleRecoversWhenSLODisabled(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_burn_rate", `{"sloId":1,"longWindowMinutes":60,"shortWindowMinutes":5,"burnRateThreshold":6}`, 0)
+ ctx := context.Background()
+ seedBuckets(t, fixture, 60, 90, 100)
+ if result, err := evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId); err != nil || !result.Fired {
+ t.Fatalf("expected a fire: %+v %v", result, err)
+ }
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ _, err := tx.Exec("UPDATE slos SET enabled = 0 WHERE id = ?", fixture.SLO.Id)
+ return struct{}{}, err
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ result, err := evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.Recovered || ruleState(t, fixture.Rule.Id) {
+ t.Fatalf("disabling the SLO must end the firing episode: %+v", result)
+ }
+ result, err = evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || result.Recovered || result.Fired {
+ t.Fatalf("a disabled SLO stays quiet afterwards: %+v %v", result, err)
+ }
+}
+
+func TestSLOBudgetConsumedRecoveryWithoutData(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_budget_consumed", `{"sloId":1,"consumedPercent":80}`, 0)
+ ctx := context.Background()
+ setBudgetRemaining(t, fixture, budget(10))
+ if result, err := evaluateSLOBudgetConsumed(ctx, fixture.Rule, fixture.ProjectId); err != nil || !result.Fired {
+ t.Fatalf("expected a fire: %+v, %v", result, err)
+ }
+ setBudgetRemaining(t, fixture, nil)
+ result, err := evaluateSLOBudgetConsumed(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || !result.Recovered {
+ t.Fatalf("an SLO without data recovers: %+v, %v", result, err)
+ }
+ if !strings.Contains(result.Message.Body, "no data") || strings.Contains(result.Message.Body, "100.0%") {
+ t.Fatalf("the recovery must not claim the budget is consumed: %q", result.Message.Body)
+ }
+}
+
+func TestSLORuleHoldsWhileTheHistoryIsRecomputed(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_budget_consumed", `{"sloId":1,"consumedPercent":80}`, 0)
+ ctx := context.Background()
+ setBudgetRemaining(t, fixture, budget(10))
+ if result, err := evaluateSLOBudgetConsumed(ctx, fixture.Rule, fixture.ProjectId); err != nil || !result.Fired {
+ t.Fatalf("expected a fire: %+v, %v", result, err)
+ }
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ return struct{}{}, transactional.SLORepository.ResetHistory(tx, fixture.SLO.Id, time.Now().UTC())
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ result, err := evaluateSLOBudgetConsumed(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || result.Recovered || result.Fired {
+ t.Fatalf("a definition change must not send a recovery before the worker recomputed the SLO: %+v, %v", result, err)
+ }
+ if !ruleState(t, fixture.Rule.Id) {
+ t.Fatal("the rule keeps firing until the recomputed snapshot says otherwise")
+ }
+}
+
+func TestSLOBurnRateRecoversOnTheShortWindow(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_burn_rate", `{"sloId":1,"longWindowMinutes":60,"shortWindowMinutes":5,"burnRateThreshold":6}`, 0)
+ ctx := context.Background()
+ seedBuckets(t, fixture, 60, 90, 100)
+ if result, err := evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId); err != nil || !result.Fired {
+ t.Fatalf("expected a fire: %+v, %v", result, err)
+ }
+ seedBuckets(t, fixture, 5, 100, 100)
+ result, err := evaluateSLOBurnRate(ctx, fixture.Rule, fixture.ProjectId)
+ if err != nil || !result.Recovered {
+ t.Fatalf("a clean short window ends the episode although the hour still burns: %+v, %v", result, err)
+ }
+}
+
+func TestSLORuleStateTransitionsOnce(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_burn_rate", `{"sloId":1}`, 0)
+ now := time.Now().UTC()
+ transition := func(firing bool) bool {
+ changed, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return transactional.NotificationRuleStateRepository.Transition(tx, fixture.Rule.Id, firing, now)
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ return changed
+ }
+ if !transition(true) || transition(true) {
+ t.Fatal("only the first of two instances may record the fire")
+ }
+ if !transition(false) || transition(false) {
+ t.Fatal("only the first of two instances may record the recovery")
+ }
+}
+
+func TestSLORuleRetriesWhenTheSendFails(t *testing.T) {
+ fixture := setupSLORuleDB(t, "slo_burn_rate", `{"sloId":1,"longWindowMinutes":60,"shortWindowMinutes":5,"burnRateThreshold":6}`, 0)
+ seedBuckets(t, fixture, 60, 90, 100)
+ if _, err := db.DB.Exec(`UPDATE notification_channels SET channel_type = 'escalation', config = '{"policyId":1}'`); err != nil {
+ t.Fatal(err)
+ }
+ previous := pageOpener
+ pageOpener = nil
+ t.Cleanup(func() { pageOpener = previous })
+
+ evaluatePolledRules(context.Background())
+ if ruleState(t, fixture.Rule.Id) {
+ t.Fatal("a rule whose page could not be opened must not stay firing")
+ }
+
+ result, err := evaluateSLOBurnRate(context.Background(), fixture.Rule, fixture.ProjectId)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.Fired {
+ t.Fatalf("the next poll should try again: %+v", result)
+ }
+}
diff --git a/backend/app/repositories/telemetry/check_result_bounds_test.go b/backend/app/repositories/telemetry/check_result_bounds_test.go
new file mode 100644
index 00000000..b12bcedb
--- /dev/null
+++ b/backend/app/repositories/telemetry/check_result_bounds_test.go
@@ -0,0 +1,52 @@
+package telemetry
+
+import (
+ "context"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func TestCheckResultRepository_CountsResultsInTheFirstSecondOfARange(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project := uuid.New()
+ from := time.Now().UTC().Truncate(time.Minute).Add(-10 * time.Minute)
+ to := from.Add(time.Minute)
+ for i, offset := range []time.Duration{500 * time.Millisecond, 30 * time.Second} {
+ at := from.Add(offset)
+ if err := CheckResultRepository.Insert(ctx, shared.CheckResult{ProjectId: project, CheckId: 3, RunId: i + 1, CheckType: "http", RecordedAt: at, ExecutedAt: at, Status: "down"}); err != nil {
+ t.Fatal(err)
+ }
+ }
+
+ points, err := CheckResultRepository.SeriesByCheck(ctx, project, 3, from, to, 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var total int64
+ for _, point := range points {
+ total += int64(point.Total)
+ }
+ if total != 2 {
+ t.Fatalf("series counted %d results, want 2: %+v", total, points)
+ }
+
+ aggregates, err := CheckResultRepository.AggregatesByProject(ctx, project, from, to)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if aggregates[3] == nil || aggregates[3].Total != 2 {
+ t.Fatalf("aggregates: %+v", aggregates[3])
+ }
+
+ entries, count, err := CheckResultRepository.FindByCheck(ctx, project, 3, &from, &to, "", 1, 10)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if count != 2 || len(entries) != 2 {
+ t.Fatalf("history listed %d of %d results, want 2", len(entries), count)
+ }
+}
diff --git a/backend/app/repositories/telemetry/clickhouse/otel_span.repository.go b/backend/app/repositories/telemetry/clickhouse/otel_span.repository.go
index e0b3ff9e..87d2fa3a 100644
--- a/backend/app/repositories/telemetry/clickhouse/otel_span.repository.go
+++ b/backend/app/repositories/telemetry/clickhouse/otel_span.repository.go
@@ -221,6 +221,9 @@ var clickhouseSearchDialect = shared.OtelSearchDialect{
NameLike: "positionCaseInsensitive(name, ?) > 0",
Attribute: "span_attributes[?] = ?",
StartOrder: "start_time_unix_nano",
+ Bucket: epochBucket,
+ Count: "toInt64(count())",
+ CountIf: countIf,
ReadSetting: otelSearchSettings,
}
diff --git a/backend/app/repositories/telemetry/clickhouse/slo_bucket.repository.go b/backend/app/repositories/telemetry/clickhouse/slo_bucket.repository.go
new file mode 100644
index 00000000..c4e89bcd
--- /dev/null
+++ b/backend/app/repositories/telemetry/clickhouse/slo_bucket.repository.go
@@ -0,0 +1,71 @@
+//go:build telemetry_ch
+
+package clickhouse
+
+import (
+ "context"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/chdb"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+type sloBucketRepository struct{}
+
+func (r *sloBucketRepository) Upsert(ctx context.Context, buckets []shared.SLOBucket) error {
+ if len(buckets) == 0 {
+ return nil
+ }
+ batch, err := chdb.Conn.PrepareBatch(chdb.BatchCtx(), "INSERT INTO slo_buckets (project_id, slo_id, bucket_start, good, total, computed_at)")
+ if err != nil {
+ return err
+ }
+ for _, b := range buckets {
+ if err := batch.Append(b.ProjectId, int64(b.SLOId), b.BucketStart.UTC(), b.Good, b.Total, b.ComputedAt.UTC()); err != nil {
+ return err
+ }
+ }
+ return batch.Send()
+}
+
+func (r *sloBucketRepository) SumRange(ctx context.Context, projectId uuid.UUID, sloId int, from, to time.Time) (int64, int64, error) {
+ var good, total int64
+ err := chdb.Conn.QueryRow(ctx,
+ "SELECT toInt64(sum(good)), toInt64(sum(total)) FROM slo_buckets FINAL WHERE project_id = ? AND slo_id = ? AND bucket_start >= ? AND bucket_start < ?",
+ projectId, int64(sloId), from.UTC(), to.UTC()).Scan(&good, &total)
+ return good, total, err
+}
+
+func (r *sloBucketRepository) SnapshotSums(ctx context.Context, projectId uuid.UUID, sloId int, from, to, sixHoursFrom, hourFrom time.Time) (shared.SLOSnapshotSums, error) {
+ var sums shared.SLOSnapshotSums
+ err := chdb.Conn.QueryRow(ctx,
+ "SELECT toInt64(sum(good)), toInt64(sum(total)), toInt64(sumIf(good, bucket_start >= ?)), toInt64(sumIf(total, bucket_start >= ?)), toInt64(sumIf(good, bucket_start >= ?)), toInt64(sumIf(total, bucket_start >= ?)) "+
+ "FROM slo_buckets FINAL WHERE project_id = ? AND slo_id = ? AND bucket_start >= ? AND bucket_start < ?",
+ sixHoursFrom.UTC(), sixHoursFrom.UTC(), hourFrom.UTC(), hourFrom.UTC(), projectId, int64(sloId), from.UTC(), to.UTC()).
+ Scan(&sums.Window.Good, &sums.Window.Total, &sums.LastSixHours.Good, &sums.LastSixHours.Total, &sums.LastHour.Good, &sums.LastHour.Total)
+ return sums, err
+}
+
+func (r *sloBucketRepository) Series(ctx context.Context, projectId uuid.UUID, sloId int, from, to time.Time, intervalMinutes int) ([]shared.SLOBucketPoint, error) {
+ query := "SELECT " + epochBucket("bucket_start", intervalMinutes*60) + " AS bucket, toInt64(sum(total)), toInt64(sum(good)) FROM slo_buckets FINAL " +
+ "WHERE project_id = ? AND slo_id = ? AND bucket_start >= ? AND bucket_start < ? GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := chdb.Conn.Query(ctx, query, projectId, int64(sloId), from.UTC(), to.UTC())
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
+
+func (r *sloBucketRepository) DeleteBySLO(ctx context.Context, projectId uuid.UUID, sloId int) error {
+ return chdb.Conn.Exec(ctx,
+ "INSERT INTO slo_buckets (project_id, slo_id, bucket_start, good, total, computed_at) SELECT project_id, slo_id, bucket_start, 0, 0, fromUnixTimestamp64Milli(?) FROM slo_buckets FINAL WHERE project_id = ? AND slo_id = ? AND (good != 0 OR total != 0)",
+ time.Now().UnixMilli(), projectId, int64(sloId))
+}
+
+func (r *sloBucketRepository) PruneBefore(ctx context.Context, cutoff time.Time) error {
+ return nil
+}
+
+var SLOBucketRepository = &sloBucketRepository{}
diff --git a/backend/app/repositories/telemetry/clickhouse/slo_counts.go b/backend/app/repositories/telemetry/clickhouse/slo_counts.go
new file mode 100644
index 00000000..518f53fd
--- /dev/null
+++ b/backend/app/repositories/telemetry/clickhouse/slo_counts.go
@@ -0,0 +1,80 @@
+//go:build telemetry_ch
+
+package clickhouse
+
+import (
+ "context"
+ "fmt"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/chdb"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func epochBucket(column string, seconds int) string {
+ return fmt.Sprintf("toInt64(toUnixTimestamp(toStartOfInterval(%s, INTERVAL %d second)))", column, seconds)
+}
+
+func countIf(condition string) string {
+ return "toInt64(countIf(" + condition + "))"
+}
+
+func distinctIf(condition string) string {
+ return "toInt64(uniqExactIf(id, " + condition + "))"
+}
+
+func (e *endpointRepository) SLICounts(ctx context.Context, projectId uuid.UUID, spec shared.EndpointSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ good := "status_code < 500"
+ var args []any
+ where := ""
+ if spec.Latency {
+ good = "duration <= ?"
+ args = append(args, int64(spec.LatencyThreshold))
+ where += " AND is_stream = 0"
+ }
+ args = append(args, projectId, from.UTC(), to.UTC())
+ if spec.Endpoint != "" {
+ where += " AND endpoint = ?"
+ args = append(args, spec.Endpoint)
+ }
+ query := "SELECT " + epochBucket("recorded_at", bucketMinutes*60) + " AS bucket, toInt64(uniqExact(id)), " + distinctIf(good) +
+ " FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?" + where + " GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := chdb.Conn.Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
+
+func (r *taskRepository) SLICounts(ctx context.Context, projectId uuid.UUID, spec shared.TaskSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ failed := "(trace_id, span_id) IN (SELECT trace_id, span_id FROM exceptions_v2 WHERE project_id = ? AND trace_type = 'task' AND recorded_at >= ? AND recorded_at <= ?)"
+ args := []any{projectId, from.UTC(), to.Add(shared.TaskExceptionSlack).UTC(), projectId, from.UTC(), to.UTC()}
+ where := ""
+ if spec.TaskName != "" {
+ where = " AND task_name = ?"
+ args = append(args, spec.TaskName)
+ }
+ query := "SELECT " + epochBucket("recorded_at", bucketMinutes*60) + " AS bucket, toInt64(uniqExact(id)), " + distinctIf("NOT ("+failed+")") +
+ " FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?" + where + " GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := chdb.Conn.Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
+
+func (r *otelSpanRepository) SLICounts(ctx context.Context, spec shared.SpanSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ search := spec.Filter
+ search.From, search.To = from, to
+ good, goodArgs := shared.SpanGoodCondition(spec)
+ query, args := shared.OtelSLICountQuery(search, good, goodArgs, bucketMinutes*60, clickhouseSearchDialect)
+ rows, err := chdb.Conn.Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
diff --git a/backend/app/repositories/telemetry/clickhouse/slo_session_counts.go b/backend/app/repositories/telemetry/clickhouse/slo_session_counts.go
new file mode 100644
index 00000000..05768a90
--- /dev/null
+++ b/backend/app/repositories/telemetry/clickhouse/slo_session_counts.go
@@ -0,0 +1,34 @@
+//go:build telemetry_ch
+
+package clickhouse
+
+import (
+ "context"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/chdb"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func (r *sessionRepository) SLICounts(ctx context.Context, projectId uuid.UUID, spec shared.SessionSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ failed := "id IN (SELECT assumeNotNull(session_id) FROM exceptions_v2 WHERE project_id = ? AND session_id IS NOT NULL AND recorded_at >= ? AND recorded_at <= ?"
+ if !spec.IncludeMessages {
+ failed += " AND is_message = 0"
+ }
+ failed += ")"
+ args := []any{projectId, from.UTC(), to.Add(shared.SessionExceptionSlack).UTC(), projectId, from.UTC(), to.UTC()}
+ where := ""
+ if spec.AppVersion != "" {
+ where = " AND app_version = ?"
+ args = append(args, spec.AppVersion)
+ }
+ query := "SELECT " + epochBucket("started_at", bucketMinutes*60) + " AS bucket, toInt64(count()), " + countIf("NOT ("+failed+")") +
+ " FROM sessions FINAL WHERE project_id = ? AND started_at >= ? AND started_at <= ?" + where + " GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := chdb.Conn.Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
diff --git a/backend/app/repositories/telemetry/duckdb/otel_span.repository.go b/backend/app/repositories/telemetry/duckdb/otel_span.repository.go
index 82d90d22..905ae972 100644
--- a/backend/app/repositories/telemetry/duckdb/otel_span.repository.go
+++ b/backend/app/repositories/telemetry/duckdb/otel_span.repository.go
@@ -146,6 +146,9 @@ var duckdbSearchDialect = shared.OtelSearchDialect{
NameLike: "contains(lower(name), lower(?))",
Attribute: `json_extract_string(span_attributes, '$."' || ? || '"') = ?`,
StartOrder: "start_time_unix_nano",
+ Bucket: epochBucket,
+ Count: "CAST(COUNT(*) AS BIGINT)",
+ CountIf: sumIf,
}
func (r *otelSpanRepository) Search(ctx context.Context, search shared.OtelSpanSearch) ([]models.OtelSpan, uint64, error) {
diff --git a/backend/app/repositories/telemetry/duckdb/slo_bucket.repository.go b/backend/app/repositories/telemetry/duckdb/slo_bucket.repository.go
new file mode 100644
index 00000000..635acd9a
--- /dev/null
+++ b/backend/app/repositories/telemetry/duckdb/slo_bucket.repository.go
@@ -0,0 +1,77 @@
+//go:build telemetry_duckdb
+
+package duckdb
+
+import (
+ "context"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+type sloBucketRepository struct{}
+
+const sloBucketUpsert = "INSERT INTO slo_buckets (project_id, slo_id, bucket_start, good, total, computed_at) VALUES (?, ?, ?, ?, ?, ?) " +
+ "ON CONFLICT (project_id, slo_id, bucket_start) DO UPDATE SET good = excluded.good, total = excluded.total, computed_at = excluded.computed_at"
+
+func (r *sloBucketRepository) Upsert(ctx context.Context, buckets []shared.SLOBucket) error {
+ if len(buckets) == 0 {
+ return nil
+ }
+ tx, err := db.TelemetryDB.BeginTx(ctx, nil)
+ if err != nil {
+ return err
+ }
+ defer tx.Rollback()
+ for _, b := range buckets {
+ if _, err := tx.ExecContext(ctx, sloBucketUpsert, b.ProjectId.String(), int64(b.SLOId), b.BucketStart.UTC(), b.Good, b.Total, b.ComputedAt.UTC()); err != nil {
+ return err
+ }
+ }
+ return tx.Commit()
+}
+
+func (r *sloBucketRepository) SumRange(ctx context.Context, projectId uuid.UUID, sloId int, from, to time.Time) (int64, int64, error) {
+ var good, total int64
+ err := db.TelemetryDB.QueryRowContext(ctx,
+ "SELECT COALESCE(CAST(SUM(good) AS BIGINT), 0), COALESCE(CAST(SUM(total) AS BIGINT), 0) FROM slo_buckets WHERE project_id = ? AND slo_id = ? AND bucket_start >= ? AND bucket_start < ?",
+ projectId.String(), int64(sloId), from.UTC(), to.UTC()).Scan(&good, &total)
+ return good, total, err
+}
+
+func (r *sloBucketRepository) SnapshotSums(ctx context.Context, projectId uuid.UUID, sloId int, from, to, sixHoursFrom, hourFrom time.Time) (shared.SLOSnapshotSums, error) {
+ var sums shared.SLOSnapshotSums
+ six, hour := sixHoursFrom.UTC(), hourFrom.UTC()
+ err := db.TelemetryDB.QueryRowContext(ctx,
+ "SELECT COALESCE(CAST(SUM(good) AS BIGINT), 0), COALESCE(CAST(SUM(total) AS BIGINT), 0), COALESCE(CAST(SUM(CASE WHEN bucket_start >= ? THEN good ELSE 0 END) AS BIGINT), 0), COALESCE(CAST(SUM(CASE WHEN bucket_start >= ? THEN total ELSE 0 END) AS BIGINT), 0), "+
+ "COALESCE(CAST(SUM(CASE WHEN bucket_start >= ? THEN good ELSE 0 END) AS BIGINT), 0), COALESCE(CAST(SUM(CASE WHEN bucket_start >= ? THEN total ELSE 0 END) AS BIGINT), 0) "+
+ "FROM slo_buckets WHERE project_id = ? AND slo_id = ? AND bucket_start >= ? AND bucket_start < ?",
+ six, six, hour, hour, projectId.String(), int64(sloId), from.UTC(), to.UTC()).
+ Scan(&sums.Window.Good, &sums.Window.Total, &sums.LastSixHours.Good, &sums.LastSixHours.Total, &sums.LastHour.Good, &sums.LastHour.Total)
+ return sums, err
+}
+
+func (r *sloBucketRepository) Series(ctx context.Context, projectId uuid.UUID, sloId int, from, to time.Time, intervalMinutes int) ([]shared.SLOBucketPoint, error) {
+ query := "SELECT " + epochBucket("bucket_start", intervalMinutes*60) + " AS bucket, CAST(SUM(total) AS BIGINT), CAST(SUM(good) AS BIGINT) FROM slo_buckets " +
+ "WHERE project_id = ? AND slo_id = ? AND bucket_start >= ? AND bucket_start < ? GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, projectId.String(), int64(sloId), from.UTC(), to.UTC())
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
+
+func (r *sloBucketRepository) DeleteBySLO(ctx context.Context, projectId uuid.UUID, sloId int) error {
+ _, err := db.TelemetryDB.ExecContext(ctx, "DELETE FROM slo_buckets WHERE project_id = ? AND slo_id = ?", projectId.String(), int64(sloId))
+ return err
+}
+
+func (r *sloBucketRepository) PruneBefore(ctx context.Context, cutoff time.Time) error {
+ _, err := db.TelemetryDB.ExecContext(ctx, "DELETE FROM slo_buckets WHERE bucket_start < ?", cutoff.UTC())
+ return err
+}
+
+var SLOBucketRepository = &sloBucketRepository{}
diff --git a/backend/app/repositories/telemetry/duckdb/slo_counts.go b/backend/app/repositories/telemetry/duckdb/slo_counts.go
new file mode 100644
index 00000000..9c82f496
--- /dev/null
+++ b/backend/app/repositories/telemetry/duckdb/slo_counts.go
@@ -0,0 +1,81 @@
+//go:build telemetry_duckdb
+
+package duckdb
+
+import (
+ "context"
+ "fmt"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func epochBucket(column string, seconds int) string {
+ return fmt.Sprintf("CAST(epoch(%s) AS BIGINT)", timeBucketExpr(column, seconds))
+}
+
+func sumIf(condition string) string {
+ return "CAST(SUM(CASE WHEN " + condition + " THEN 1 ELSE 0 END) AS BIGINT)"
+}
+
+func distinctIf(condition string) string {
+ return "CAST(COUNT(DISTINCT CASE WHEN " + condition + " THEN id END) AS BIGINT)"
+}
+
+func (e *endpointRepository) SLICounts(ctx context.Context, projectId uuid.UUID, spec shared.EndpointSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ good := "status_code < 500"
+ var args []any
+ where := ""
+ if spec.Latency {
+ good = "duration <= ?"
+ args = append(args, int64(spec.LatencyThreshold))
+ where += " AND is_stream = 0"
+ }
+ args = append(args, projectId.String(), from.UTC(), to.UTC())
+ if spec.Endpoint != "" {
+ where += " AND endpoint = ?"
+ args = append(args, spec.Endpoint)
+ }
+ query := "SELECT " + epochBucket("recorded_at", bucketMinutes*60) + " AS bucket, CAST(COUNT(DISTINCT id) AS BIGINT), " + distinctIf(good) +
+ " FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?" + where + " GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
+
+func (r *taskRepository) SLICounts(ctx context.Context, projectId uuid.UUID, spec shared.TaskSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ pid := projectId.String()
+ failed := "(trace_id, span_id) IN (SELECT trace_id, span_id FROM exceptions_v2 WHERE project_id = ? AND trace_type = 'task' AND recorded_at >= ? AND recorded_at <= ?)"
+ args := []any{pid, from.UTC(), to.Add(shared.TaskExceptionSlack).UTC(), pid, from.UTC(), to.UTC()}
+ where := ""
+ if spec.TaskName != "" {
+ where = " AND task_name = ?"
+ args = append(args, spec.TaskName)
+ }
+ query := "SELECT " + epochBucket("recorded_at", bucketMinutes*60) + " AS bucket, CAST(COUNT(DISTINCT id) AS BIGINT), " + distinctIf("NOT ("+failed+")") +
+ " FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?" + where + " GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
+
+func (r *otelSpanRepository) SLICounts(ctx context.Context, spec shared.SpanSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ search := spec.Filter
+ search.From, search.To = from, to
+ good, goodArgs := shared.SpanGoodCondition(spec)
+ query, args := shared.OtelSLICountQuery(search, good, goodArgs, bucketMinutes*60, duckdbSearchDialect)
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
diff --git a/backend/app/repositories/telemetry/duckdb/slo_session_counts.go b/backend/app/repositories/telemetry/duckdb/slo_session_counts.go
new file mode 100644
index 00000000..6710a36f
--- /dev/null
+++ b/backend/app/repositories/telemetry/duckdb/slo_session_counts.go
@@ -0,0 +1,35 @@
+//go:build telemetry_duckdb
+
+package duckdb
+
+import (
+ "context"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func (r *sessionRepository) SLICounts(ctx context.Context, projectId uuid.UUID, spec shared.SessionSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ pid := projectId.String()
+ failed := "id IN (SELECT session_id FROM exceptions_v2 WHERE project_id = ? AND session_id IS NOT NULL AND session_id != '' AND recorded_at >= ? AND recorded_at <= ?"
+ if !spec.IncludeMessages {
+ failed += " AND is_message = 0"
+ }
+ failed += ")"
+ args := []any{pid, from.UTC(), to.Add(shared.SessionExceptionSlack).UTC(), pid, from.UTC(), to.UTC()}
+ where := ""
+ if spec.AppVersion != "" {
+ where = " AND app_version = ?"
+ args = append(args, spec.AppVersion)
+ }
+ query := "SELECT " + epochBucket("started_at", bucketMinutes*60) + " AS bucket, CAST(COUNT(*) AS BIGINT), " + sumIf("NOT ("+failed+")") +
+ " FROM sessions WHERE project_id = ? AND started_at >= ? AND started_at <= ?" + where + " GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_search.go b/backend/app/repositories/telemetry/shared/otel_search.go
index 1b208eec..3a00e8b1 100644
--- a/backend/app/repositories/telemetry/shared/otel_search.go
+++ b/backend/app/repositories/telemetry/shared/otel_search.go
@@ -33,6 +33,9 @@ type OtelSearchDialect struct {
StartOrder string
ReadSetting string
WinnerOrder string
+ Bucket func(column string, seconds int) string
+ Count string
+ CountIf func(condition string) string
}
var otelSearchOrders = map[string]string{
diff --git a/backend/app/repositories/telemetry/shared/slo.go b/backend/app/repositories/telemetry/shared/slo.go
new file mode 100644
index 00000000..01228d26
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/slo.go
@@ -0,0 +1,98 @@
+package shared
+
+import (
+ "time"
+
+ "github.com/google/uuid"
+)
+
+type SLOBucket struct {
+ ProjectId uuid.UUID
+ SLOId int
+ BucketStart time.Time
+ Good int64
+ Total int64
+ ComputedAt time.Time
+}
+
+type SLOCount struct {
+ Good int64
+ Total int64
+}
+
+type SLOSnapshotSums struct {
+ Window SLOCount
+ LastSixHours SLOCount
+ LastHour SLOCount
+}
+
+type SLOBucketPoint struct {
+ Bucket time.Time `json:"bucket"`
+ Good int64 `json:"good"`
+ Total int64 `json:"total"`
+}
+
+type EndpointSLISpec struct {
+ Endpoint string
+ Latency bool
+ LatencyThreshold time.Duration
+}
+
+type TaskSLISpec struct {
+ TaskName string
+}
+
+type SpanSLISpec struct {
+ Filter OtelSpanSearch
+ Latency bool
+ LatencyThreshold time.Duration
+}
+
+type SessionSLISpec struct {
+ AppVersion string
+ IncludeMessages bool
+}
+
+const SessionExceptionSlack = 2 * time.Hour
+
+const TaskExceptionSlack = 6 * time.Hour
+
+type bucketRows interface {
+ Next() bool
+ Scan(dest ...any) error
+ Err() error
+}
+
+func ScanSLOBucketPoints(rows bucketRows, from, to time.Time) ([]SLOBucketPoint, error) {
+ points := make([]SLOBucketPoint, 0)
+ for rows.Next() {
+ var bucket, total, good int64
+ if err := rows.Scan(&bucket, &total, &good); err != nil {
+ return nil, err
+ }
+ at := time.Unix(bucket, 0).UTC()
+ if !at.Before(to) {
+ continue
+ }
+ if at.Before(from) {
+ at = from
+ }
+ points = append(points, SLOBucketPoint{Bucket: at, Good: good, Total: total})
+ }
+ return points, rows.Err()
+}
+
+func SpanGoodCondition(spec SpanSLISpec) (string, []any) {
+ if spec.Latency {
+ return "duration <= ?", []any{int64(spec.LatencyThreshold)}
+ }
+ return "status_code != 2", nil
+}
+
+func OtelSLICountQuery(search OtelSpanSearch, good string, goodArgs []any, bucketSeconds int, dialect OtelSearchDialect) (string, []any) {
+ predicate, args := otelSearchPredicate(search, dialect)
+ winners := OtelWinningRows("recorded_at, duration, status_code", predicate, dialect.WinnerOrder)
+ query := "SELECT " + dialect.Bucket("recorded_at", bucketSeconds) + " AS bucket, " + dialect.Count + ", " + dialect.CountIf(good) +
+ " FROM " + winners + " GROUP BY bucket ORDER BY bucket ASC" + dialect.ReadSetting
+ return query, append(append([]any{}, goodArgs...), args...)
+}
diff --git a/backend/app/repositories/telemetry/slo_repository_test.go b/backend/app/repositories/telemetry/slo_repository_test.go
new file mode 100644
index 00000000..7a51757a
--- /dev/null
+++ b/backend/app/repositories/telemetry/slo_repository_test.go
@@ -0,0 +1,339 @@
+package telemetry
+
+import (
+ "context"
+ "encoding/hex"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func sloTestBase() time.Time {
+ return time.Now().UTC().Truncate(time.Hour).Add(-2 * time.Hour)
+}
+
+func bucketAt(points []shared.SLOBucketPoint, at time.Time) (shared.SLOBucketPoint, bool) {
+ for _, point := range points {
+ if point.Bucket.Equal(at) {
+ return point, true
+ }
+ }
+ return shared.SLOBucketPoint{}, false
+}
+
+func expectBucket(t *testing.T, points []shared.SLOBucketPoint, at time.Time, good, total int64) {
+ t.Helper()
+ point, ok := bucketAt(points, at)
+ if !ok {
+ t.Fatalf("no bucket at %s in %+v", at, points)
+ }
+ if point.Good != good || point.Total != total {
+ t.Fatalf("bucket %s = good %d / total %d, want %d / %d", at, point.Good, point.Total, good, total)
+ }
+}
+
+func TestSLOBucketRepository_UpsertSumSeries(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project := uuid.New()
+ base := sloTestBase()
+ now := time.Now().UTC().Add(-time.Minute)
+ rows := []shared.SLOBucket{
+ {ProjectId: project, SLOId: 7, BucketStart: base, Good: 90, Total: 100, ComputedAt: now},
+ {ProjectId: project, SLOId: 7, BucketStart: base.Add(time.Minute), Good: 100, Total: 100, ComputedAt: now},
+ {ProjectId: project, SLOId: 7, BucketStart: base.Add(2 * time.Minute), Good: 50, Total: 100, ComputedAt: now},
+ {ProjectId: project, SLOId: 8, BucketStart: base, Good: 1, Total: 1, ComputedAt: now},
+ }
+ if err := SLOBucketRepository.Upsert(ctx, rows); err != nil {
+ t.Fatal(err)
+ }
+ if err := SLOBucketRepository.Upsert(ctx, []shared.SLOBucket{{ProjectId: project, SLOId: 7, BucketStart: base, Good: 95, Total: 100, ComputedAt: now.Add(time.Second)}}); err != nil {
+ t.Fatal(err)
+ }
+
+ good, total, err := SLOBucketRepository.SumRange(ctx, project, 7, base, base.Add(3*time.Minute))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if good != 245 || total != 300 {
+ t.Fatalf("sum = %d/%d, want 245/300", good, total)
+ }
+ good, total, err = SLOBucketRepository.SumRange(ctx, project, 7, base, base.Add(2*time.Minute))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if good != 195 || total != 200 {
+ t.Fatalf("upper bound is exclusive: sum = %d/%d, want 195/200", good, total)
+ }
+
+ series, err := SLOBucketRepository.Series(ctx, project, 7, base, base.Add(4*time.Minute), 2)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(series) != 2 {
+ t.Fatalf("series re-bucketed to 2 minutes: %+v", series)
+ }
+ expectBucket(t, series, base, 195, 200)
+ expectBucket(t, series, base.Add(2*time.Minute), 50, 100)
+
+ series, err = SLOBucketRepository.Series(ctx, project, 7, base.Add(time.Minute), base.Add(4*time.Minute), 2)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, series, base.Add(time.Minute), 100, 100)
+ expectBucket(t, series, base.Add(2*time.Minute), 50, 100)
+
+ sums, err := SLOBucketRepository.SnapshotSums(ctx, project, 7, base, base.Add(3*time.Minute), base.Add(time.Minute), base.Add(2*time.Minute))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if sums.Window != (shared.SLOCount{Good: 245, Total: 300}) || sums.LastSixHours != (shared.SLOCount{Good: 150, Total: 200}) || sums.LastHour != (shared.SLOCount{Good: 50, Total: 100}) {
+ t.Fatalf("snapshot sums = %+v", sums)
+ }
+
+ if err := SLOBucketRepository.DeleteBySLO(ctx, project, 7); err != nil {
+ t.Fatal(err)
+ }
+ good, total, err = SLOBucketRepository.SumRange(ctx, project, 7, base, base.Add(3*time.Minute))
+ if err != nil || good != 0 || total != 0 {
+ t.Fatalf("buckets survive delete: %d/%d, %v", good, total, err)
+ }
+ good, total, err = SLOBucketRepository.SumRange(ctx, project, 8, base, base.Add(time.Minute))
+ if err != nil || good != 1 || total != 1 {
+ t.Fatalf("delete touched another SLO: %d/%d, %v", good, total, err)
+ }
+ if err := SLOBucketRepository.Upsert(ctx, []shared.SLOBucket{{ProjectId: project, SLOId: 7, BucketStart: base.Add(time.Minute), Good: 1, Total: 1, ComputedAt: now}}); err != nil {
+ t.Fatal(err)
+ }
+ if err := SLOBucketRepository.Upsert(ctx, []shared.SLOBucket{{ProjectId: project, SLOId: 7, BucketStart: base, Good: 3, Total: 4, ComputedAt: time.Now().UTC().Add(time.Second)}}); err != nil {
+ t.Fatal(err)
+ }
+ good, total, err = SLOBucketRepository.SumRange(ctx, project, 7, base, base.Add(3*time.Minute))
+ wantGood, wantTotal := int64(4), int64(5)
+ if db.TelemetryBackendName() == "clickhouse" {
+ wantGood, wantTotal = 3, 4
+ }
+ if err != nil || good != wantGood || total != wantTotal {
+ t.Fatalf("after delete: %d/%d (%v), want %d/%d", good, total, err, wantGood, wantTotal)
+ }
+ if db.TelemetryBackendName() != "clickhouse" {
+ if err := SLOBucketRepository.PruneBefore(ctx, base.Add(time.Minute)); err != nil {
+ t.Fatal(err)
+ }
+ good, total, err = SLOBucketRepository.SumRange(ctx, project, 8, base, base.Add(time.Minute))
+ if err != nil || total != 0 {
+ t.Fatalf("prune kept an old bucket: %d/%d, %v", good, total, err)
+ }
+ }
+}
+
+func TestSLOBucketRepository_DeleteRightAfterTheWrite(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project := uuid.New()
+ base := sloTestBase()
+ var rows []shared.SLOBucket
+ for minute := 0; minute < 30; minute++ {
+ rows = append(rows, shared.SLOBucket{ProjectId: project, SLOId: 3, BucketStart: base.Add(time.Duration(minute) * time.Minute), Good: 1, Total: 1, ComputedAt: time.Now().UTC()})
+ }
+ for attempt := 0; attempt < 3; attempt++ {
+ if err := SLOBucketRepository.Upsert(ctx, rows); err != nil {
+ t.Fatal(err)
+ }
+ if err := SLOBucketRepository.DeleteBySLO(ctx, project, 3); err != nil {
+ t.Fatal(err)
+ }
+ if _, total, err := SLOBucketRepository.SumRange(ctx, project, 3, base, base.Add(time.Hour)); err != nil || total != 0 {
+ t.Fatalf("attempt %d: a delete in the same second as the write left %d events (%v)", attempt, total, err)
+ }
+ for i := range rows {
+ rows[i].ComputedAt = time.Now().UTC()
+ }
+ }
+}
+
+func TestEndpointRepository_SLICounts(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project := uuid.New()
+ base := sloTestBase()
+ first, second := base, base.Add(time.Minute)
+ endpoints := []models.Endpoint{
+ makeEndpoint(project, "GET /users", 100*time.Millisecond, 200, first.Add(5*time.Second)),
+ makeEndpoint(project, "GET /users", 100*time.Millisecond, 200, first.Add(10*time.Second)),
+ makeEndpoint(project, "GET /users", 100*time.Millisecond, 404, first.Add(15*time.Second)),
+ makeEndpoint(project, "GET /users", 100*time.Millisecond, 500, first.Add(20*time.Second)),
+ makeEndpoint(project, "GET /orders", 100*time.Millisecond, 200, second.Add(5*time.Second)),
+ makeEndpoint(project, "GET /orders", 900*time.Millisecond, 200, second.Add(10*time.Second)),
+ makeEndpoint(project, "GET /users", 100*time.Millisecond, 200, second.Add(40*time.Second)),
+ makeEndpoint(project, "GET /users", 100*time.Millisecond, 200, base.Add(2*time.Minute)),
+ }
+ stream := makeEndpoint(project, "GET /events", 30*time.Second, 200, second.Add(20*time.Second))
+ stream.IsStream = true
+ retried := endpoints[3]
+ endpoints = append(endpoints, stream, retried)
+ if err := EndpointRepository.InsertAsync(ctx, endpoints); err != nil {
+ t.Fatal(err)
+ }
+
+ availability, err := EndpointRepository.SLICounts(ctx, project, shared.EndpointSLISpec{}, base, base.Add(2*time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(availability) != 2 {
+ t.Fatalf("two buckets inside [from, to): %+v", availability)
+ }
+ expectBucket(t, availability, first, 3, 4)
+ expectBucket(t, availability, second, 4, 4)
+
+ latency, err := EndpointRepository.SLICounts(ctx, project, shared.EndpointSLISpec{Latency: true, LatencyThreshold: 500 * time.Millisecond}, base, base.Add(2*time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, latency, second, 2, 3)
+
+ named, err := EndpointRepository.SLICounts(ctx, project, shared.EndpointSLISpec{Endpoint: "GET /orders"}, base, base.Add(2*time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(named) != 1 {
+ t.Fatalf("endpoint filter: %+v", named)
+ }
+ expectBucket(t, named, second, 2, 2)
+
+ wide, err := EndpointRepository.SLICounts(ctx, project, shared.EndpointSLISpec{}, base, base.Add(10*time.Minute), 5)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, wide, base.Truncate(5*time.Minute), 8, 9)
+}
+
+func TestTaskRepository_SLICounts(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project := uuid.New()
+ base := sloTestBase()
+ failing := makeTask(project, "send-email", time.Second, base.Add(10*time.Second))
+ failing.TraceId, failing.SpanId = hexId(uuid.New()), hexId(uuid.New())
+ healthy := makeTask(project, "send-email", time.Second, base.Add(20*time.Second))
+ healthy.TraceId, healthy.SpanId = hexId(uuid.New()), hexId(uuid.New())
+ other := makeTask(project, "rebuild-index", time.Second, base.Add(30*time.Second))
+ other.TraceId, other.SpanId = hexId(uuid.New()), hexId(uuid.New())
+ if err := TaskRepository.InsertAsync(ctx, []models.Task{failing, healthy, other, healthy}); err != nil {
+ t.Fatal(err)
+ }
+ onTask := makeException(project, "hash-a", "boom", base.Add(5*time.Minute))
+ onTask.TraceType, onTask.TraceId, onTask.SpanId = "task", failing.TraceId, failing.SpanId
+ onChild := makeException(project, "hash-b", "caught", base.Add(15*time.Second))
+ onChild.TraceType, onChild.TraceId, onChild.SpanId = "task", healthy.TraceId, hexId(uuid.New())
+ if err := ExceptionStackTraceRepository.InsertAsync(ctx, []models.ExceptionStackTrace{onTask, onChild}); err != nil {
+ t.Fatal(err)
+ }
+
+ all, err := TaskRepository.SLICounts(ctx, project, shared.TaskSLISpec{}, base, base.Add(time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, all, base, 2, 3)
+
+ named, err := TaskRepository.SLICounts(ctx, project, shared.TaskSLISpec{TaskName: "send-email"}, base, base.Add(time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, named, base, 1, 2)
+}
+
+func sloTestSpan(project uuid.UUID, name, service string, kind, status int32, duration time.Duration, at time.Time, attributes map[string]string) models.OtelSpan {
+ trace, id := uuid.New(), uuid.New()
+ return models.OtelSpan{Span: models.Span{
+ ProjectId: project, TraceId: hex.EncodeToString(trace[:]), SpanId: hex.EncodeToString(id[8:]),
+ Name: name, ServiceName: service, SpanKind: kind, StatusCode: status,
+ StartTime: at, RecordedAt: at, Duration: duration, Attributes: attributes,
+ }}
+}
+
+func TestOtelSpanRepository_SLICounts(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project := uuid.New()
+ base := sloTestBase()
+ checkout := sloTestSpan(project, "POST /checkout", "api", 2, 0, 100*time.Millisecond, base.Add(5*time.Second), map[string]string{"http.route": "/checkout"})
+ failed := sloTestSpan(project, "POST /checkout", "api", 2, 2, 100*time.Millisecond, base.Add(10*time.Second), map[string]string{"http.route": "/checkout"})
+ slow := sloTestSpan(project, "GET /cart", "api", 2, 0, 2*time.Second, base.Add(15*time.Second), map[string]string{"http.route": "/cart"})
+ worker := sloTestSpan(project, "process", "worker", 5, 0, 100*time.Millisecond, base.Add(20*time.Second), nil)
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{checkout, failed, slow, worker}); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{failed}); err != nil {
+ t.Fatal(err)
+ }
+
+ all, err := OtelSpanRepository.SLICounts(ctx, shared.SpanSLISpec{Filter: shared.OtelSpanSearch{ProjectId: project}}, base, base.Add(time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, all, base, 3, 4)
+
+ api, err := OtelSpanRepository.SLICounts(ctx, shared.SpanSLISpec{Filter: shared.OtelSpanSearch{ProjectId: project, Service: "api"}}, base, base.Add(time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, api, base, 2, 3)
+
+ route, err := OtelSpanRepository.SLICounts(ctx, shared.SpanSLISpec{Filter: shared.OtelSpanSearch{ProjectId: project, Attributes: []shared.OtelAttributeFilter{{Key: "http.route", Value: "/checkout"}}}}, base, base.Add(time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, route, base, 1, 2)
+
+ latency, err := OtelSpanRepository.SLICounts(ctx, shared.SpanSLISpec{Filter: shared.OtelSpanSearch{ProjectId: project, Name: "/"}, Latency: true, LatencyThreshold: time.Second}, base, base.Add(time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, latency, base, 2, 3)
+}
+
+func TestSessionRepository_SLICounts(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project := uuid.New()
+ base := sloTestBase()
+ clean := models.Session{Id: uuid.New(), ProjectId: project, StartedAt: base.Add(5 * time.Second), AppVersion: "1.4.2", Attributes: map[string]string{}}
+ crashed := models.Session{Id: uuid.New(), ProjectId: project, StartedAt: base.Add(10 * time.Second), AppVersion: "1.4.2", Attributes: map[string]string{}}
+ logged := models.Session{Id: uuid.New(), ProjectId: project, StartedAt: base.Add(15 * time.Second), AppVersion: "1.4.1", Attributes: map[string]string{}}
+ later := models.Session{Id: uuid.New(), ProjectId: project, StartedAt: base.Add(70 * time.Second), AppVersion: "1.4.2", Attributes: map[string]string{}}
+ if err := SessionRepository.Upsert(ctx, []models.Session{clean, crashed, logged, later}); err != nil {
+ t.Fatal(err)
+ }
+ crash := makeException(project, "crash", "boom", base.Add(20*time.Minute))
+ crash.SessionId = &crashed.Id
+ message := makeException(project, "note", "captured message", base.Add(40*time.Second))
+ message.SessionId = &logged.Id
+ message.IsMessage = true
+ if err := ExceptionStackTraceRepository.InsertAsync(ctx, []models.ExceptionStackTrace{crash, message}); err != nil {
+ t.Fatal(err)
+ }
+
+ all, err := SessionRepository.SLICounts(ctx, project, shared.SessionSLISpec{}, base, base.Add(2*time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, all, base, 2, 3)
+ expectBucket(t, all, base.Add(time.Minute), 1, 1)
+
+ strict, err := SessionRepository.SLICounts(ctx, project, shared.SessionSLISpec{IncludeMessages: true}, base, base.Add(time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, strict, base, 1, 3)
+
+ version, err := SessionRepository.SLICounts(ctx, project, shared.SessionSLISpec{AppVersion: "1.4.2"}, base, base.Add(time.Minute), 1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expectBucket(t, version, base, 1, 2)
+}
diff --git a/backend/app/repositories/telemetry/sqlite/check_result.repository.go b/backend/app/repositories/telemetry/sqlite/check_result.repository.go
index d050ee3c..64109bd9 100644
--- a/backend/app/repositories/telemetry/sqlite/check_result.repository.go
+++ b/backend/app/repositories/telemetry/sqlite/check_result.repository.go
@@ -78,11 +78,11 @@ func (r *checkResultRepository) FindByCheck(ctx context.Context, projectId uuid.
if from != nil {
where += " AND recorded_at >= ?"
- args = append(args, from.UTC().Format(time.RFC3339Nano))
+ args = append(args, sqlitetypes.NewSQLiteTime(*from))
}
if to != nil {
where += " AND recorded_at <= ?"
- args = append(args, to.UTC().Format(time.RFC3339Nano))
+ args = append(args, sqlitetypes.NewSQLiteTime(*to))
}
var total int64
@@ -134,7 +134,7 @@ func (r *checkResultRepository) AggregatesByProject(ctx context.Context, project
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY check_id`
- rows, err := db.TelemetryDB.QueryContext(ctx, query, projectId.String(), from.UTC().Format(time.RFC3339Nano), to.UTC().Format(time.RFC3339Nano))
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, projectId.String(), sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to))
if err != nil {
return nil, err
}
@@ -166,7 +166,7 @@ func (r *checkResultRepository) SeriesByCheck(ctx context.Context, projectId uui
WHERE project_id = ? AND check_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY bucket ORDER BY bucket ASC`
- rows, err := db.TelemetryDB.QueryContext(ctx, query, bucketSeconds, bucketSeconds, projectId.String(), checkId, from.UTC().Format(time.RFC3339Nano), to.UTC().Format(time.RFC3339Nano))
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, bucketSeconds, bucketSeconds, projectId.String(), checkId, sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to))
if err != nil {
return nil, err
}
diff --git a/backend/app/repositories/telemetry/sqlite/otel_span.repository.go b/backend/app/repositories/telemetry/sqlite/otel_span.repository.go
index b88bd2af..02a09ebb 100644
--- a/backend/app/repositories/telemetry/sqlite/otel_span.repository.go
+++ b/backend/app/repositories/telemetry/sqlite/otel_span.repository.go
@@ -153,6 +153,9 @@ var sqliteSearchDialect = shared.OtelSearchDialect{
NameLike: "INSTR(LOWER(name), LOWER(?)) > 0",
Attribute: `json_extract(span_attributes, '$."' || ? || '"') = ?`,
StartOrder: "CAST(start_time_unix_nano AS INTEGER)",
+ Bucket: epochBucket,
+ Count: "COUNT(*)",
+ CountIf: sumIf,
}
func (r *otelSpanRepository) Search(ctx context.Context, search shared.OtelSpanSearch) ([]models.OtelSpan, uint64, error) {
diff --git a/backend/app/repositories/telemetry/sqlite/slo_bucket.repository.go b/backend/app/repositories/telemetry/sqlite/slo_bucket.repository.go
new file mode 100644
index 00000000..9d704a68
--- /dev/null
+++ b/backend/app/repositories/telemetry/sqlite/slo_bucket.repository.go
@@ -0,0 +1,78 @@
+//go:build !telemetry_ch && !telemetry_duckdb
+
+package sqlite
+
+import (
+ "context"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/sqlitetypes"
+)
+
+type sloBucketRepository struct{}
+
+const sloBucketUpsert = "INSERT INTO slo_buckets (project_id, slo_id, bucket_start, good, total, computed_at) VALUES (?, ?, ?, ?, ?, ?) " +
+ "ON CONFLICT(project_id, slo_id, bucket_start) DO UPDATE SET good = excluded.good, total = excluded.total, computed_at = excluded.computed_at"
+
+func (r *sloBucketRepository) Upsert(ctx context.Context, buckets []shared.SLOBucket) error {
+ if len(buckets) == 0 {
+ return nil
+ }
+ tx, err := db.TelemetryDB.BeginTx(ctx, nil)
+ if err != nil {
+ return err
+ }
+ defer tx.Rollback()
+ for _, b := range buckets {
+ if _, err := tx.ExecContext(ctx, sloBucketUpsert, b.ProjectId.String(), b.SLOId, sqlitetypes.NewSQLiteTime(b.BucketStart), b.Good, b.Total, sqlitetypes.NewSQLiteTime(b.ComputedAt)); err != nil {
+ return err
+ }
+ }
+ return tx.Commit()
+}
+
+func (r *sloBucketRepository) SumRange(ctx context.Context, projectId uuid.UUID, sloId int, from, to time.Time) (int64, int64, error) {
+ var good, total int64
+ err := db.TelemetryDB.QueryRowContext(ctx,
+ "SELECT COALESCE(SUM(good), 0), COALESCE(SUM(total), 0) FROM slo_buckets WHERE project_id = ? AND slo_id = ? AND bucket_start >= ? AND bucket_start < ?",
+ projectId.String(), sloId, sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to)).Scan(&good, &total)
+ return good, total, err
+}
+
+func (r *sloBucketRepository) SnapshotSums(ctx context.Context, projectId uuid.UUID, sloId int, from, to, sixHoursFrom, hourFrom time.Time) (shared.SLOSnapshotSums, error) {
+ var sums shared.SLOSnapshotSums
+ six, hour := sqlitetypes.NewSQLiteTime(sixHoursFrom), sqlitetypes.NewSQLiteTime(hourFrom)
+ err := db.TelemetryDB.QueryRowContext(ctx,
+ "SELECT COALESCE(SUM(good), 0), COALESCE(SUM(total), 0), COALESCE(SUM(CASE WHEN bucket_start >= ? THEN good ELSE 0 END), 0), COALESCE(SUM(CASE WHEN bucket_start >= ? THEN total ELSE 0 END), 0), "+
+ "COALESCE(SUM(CASE WHEN bucket_start >= ? THEN good ELSE 0 END), 0), COALESCE(SUM(CASE WHEN bucket_start >= ? THEN total ELSE 0 END), 0) "+
+ "FROM slo_buckets WHERE project_id = ? AND slo_id = ? AND bucket_start >= ? AND bucket_start < ?",
+ six, six, hour, hour, projectId.String(), sloId, sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to)).
+ Scan(&sums.Window.Good, &sums.Window.Total, &sums.LastSixHours.Good, &sums.LastSixHours.Total, &sums.LastHour.Good, &sums.LastHour.Total)
+ return sums, err
+}
+
+func (r *sloBucketRepository) Series(ctx context.Context, projectId uuid.UUID, sloId int, from, to time.Time, intervalMinutes int) ([]shared.SLOBucketPoint, error) {
+ query := "SELECT " + epochBucket("bucket_start", intervalMinutes*60) + " AS bucket, SUM(total), SUM(good) FROM slo_buckets " +
+ "WHERE project_id = ? AND slo_id = ? AND bucket_start >= ? AND bucket_start < ? GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, projectId.String(), sloId, sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to))
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
+
+func (r *sloBucketRepository) DeleteBySLO(ctx context.Context, projectId uuid.UUID, sloId int) error {
+ _, err := db.TelemetryDB.ExecContext(ctx, "DELETE FROM slo_buckets WHERE project_id = ? AND slo_id = ?", projectId.String(), sloId)
+ return err
+}
+
+func (r *sloBucketRepository) PruneBefore(ctx context.Context, cutoff time.Time) error {
+ _, err := db.TelemetryDB.ExecContext(ctx, "DELETE FROM slo_buckets WHERE bucket_start < ?", sqlitetypes.NewSQLiteTime(cutoff))
+ return err
+}
+
+var SLOBucketRepository = &sloBucketRepository{}
diff --git a/backend/app/repositories/telemetry/sqlite/slo_counts.go b/backend/app/repositories/telemetry/sqlite/slo_counts.go
new file mode 100644
index 00000000..03421bf1
--- /dev/null
+++ b/backend/app/repositories/telemetry/sqlite/slo_counts.go
@@ -0,0 +1,82 @@
+//go:build !telemetry_ch && !telemetry_duckdb
+
+package sqlite
+
+import (
+ "context"
+ "fmt"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/sqlitetypes"
+)
+
+func epochBucket(column string, seconds int) string {
+ return fmt.Sprintf("(CAST(strftime('%%s', %s) AS INTEGER) / %d) * %d", column, seconds, seconds)
+}
+
+func sumIf(condition string) string {
+ return "SUM(CASE WHEN " + condition + " THEN 1 ELSE 0 END)"
+}
+
+func distinctIf(condition string) string {
+ return "COUNT(DISTINCT CASE WHEN " + condition + " THEN id END)"
+}
+
+func (e *endpointRepository) SLICounts(ctx context.Context, projectId uuid.UUID, spec shared.EndpointSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ good := "status_code < 500"
+ var args []any
+ where := ""
+ if spec.Latency {
+ good = "duration <= ?"
+ args = append(args, int64(spec.LatencyThreshold))
+ where += " AND is_stream = 0"
+ }
+ args = append(args, projectId.String(), sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to))
+ if spec.Endpoint != "" {
+ where += " AND endpoint = ?"
+ args = append(args, spec.Endpoint)
+ }
+ query := "SELECT " + epochBucket("recorded_at", bucketMinutes*60) + " AS bucket, COUNT(DISTINCT id), " + distinctIf(good) +
+ " FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?" + where + " GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
+
+func (r *taskRepository) SLICounts(ctx context.Context, projectId uuid.UUID, spec shared.TaskSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ pid := projectId.String()
+ failed := "(trace_id, span_id) IN (SELECT trace_id, span_id FROM exceptions_v2 WHERE project_id = ? AND trace_type = 'task' AND recorded_at >= ? AND recorded_at <= ?)"
+ args := []any{pid, sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to.Add(shared.TaskExceptionSlack)), pid, sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to)}
+ where := ""
+ if spec.TaskName != "" {
+ where = " AND task_name = ?"
+ args = append(args, spec.TaskName)
+ }
+ query := "SELECT " + epochBucket("recorded_at", bucketMinutes*60) + " AS bucket, COUNT(DISTINCT id), " + distinctIf("NOT ("+failed+")") +
+ " FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?" + where + " GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
+
+func (r *otelSpanRepository) SLICounts(ctx context.Context, spec shared.SpanSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ search := spec.Filter
+ search.From, search.To = from, to
+ good, goodArgs := shared.SpanGoodCondition(spec)
+ query, args := shared.OtelSLICountQuery(search, good, goodArgs, bucketMinutes*60, sqliteSearchDialect)
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
diff --git a/backend/app/repositories/telemetry/sqlite/slo_session_counts.go b/backend/app/repositories/telemetry/sqlite/slo_session_counts.go
new file mode 100644
index 00000000..4501cc32
--- /dev/null
+++ b/backend/app/repositories/telemetry/sqlite/slo_session_counts.go
@@ -0,0 +1,36 @@
+//go:build !telemetry_ch && !telemetry_duckdb
+
+package sqlite
+
+import (
+ "context"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/sqlitetypes"
+)
+
+func (r *sessionRepository) SLICounts(ctx context.Context, projectId uuid.UUID, spec shared.SessionSLISpec, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ pid := projectId.String()
+ failed := "id IN (SELECT session_id FROM exceptions_v2 WHERE project_id = ? AND session_id IS NOT NULL AND session_id != '' AND recorded_at >= ? AND recorded_at <= ?"
+ if !spec.IncludeMessages {
+ failed += " AND is_message = 0"
+ }
+ failed += ")"
+ args := []any{pid, sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to.Add(shared.SessionExceptionSlack)), pid, sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to)}
+ where := ""
+ if spec.AppVersion != "" {
+ where = " AND app_version = ?"
+ args = append(args, spec.AppVersion)
+ }
+ query := "SELECT " + epochBucket("started_at", bucketMinutes*60) + " AS bucket, COUNT(*), " + sumIf("NOT ("+failed+")") +
+ " FROM sessions WHERE project_id = ? AND started_at >= ? AND started_at <= ?" + where + " GROUP BY bucket ORDER BY bucket ASC"
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanSLOBucketPoints(rows, from, to)
+}
diff --git a/backend/app/repositories/telemetry/telemetry_ch.go b/backend/app/repositories/telemetry/telemetry_ch.go
index 878131f7..596ebfef 100644
--- a/backend/app/repositories/telemetry/telemetry_ch.go
+++ b/backend/app/repositories/telemetry/telemetry_ch.go
@@ -18,4 +18,5 @@ var (
OtelSpanRepository = ch.OtelSpanRepository
LegacyRepository = ch.LegacyRepository
TaskRepository = ch.TaskRepository
+ SLOBucketRepository = ch.SLOBucketRepository
)
diff --git a/backend/app/repositories/telemetry/telemetry_duckdb.go b/backend/app/repositories/telemetry/telemetry_duckdb.go
index 764c37d2..3d2c518c 100644
--- a/backend/app/repositories/telemetry/telemetry_duckdb.go
+++ b/backend/app/repositories/telemetry/telemetry_duckdb.go
@@ -18,4 +18,5 @@ var (
OtelSpanRepository = duckdbrepo.OtelSpanRepository
LegacyRepository = duckdbrepo.LegacyRepository
TaskRepository = duckdbrepo.TaskRepository
+ SLOBucketRepository = duckdbrepo.SLOBucketRepository
)
diff --git a/backend/app/repositories/telemetry/telemetry_shared.go b/backend/app/repositories/telemetry/telemetry_shared.go
index 2227bbc3..af2ec263 100644
--- a/backend/app/repositories/telemetry/telemetry_shared.go
+++ b/backend/app/repositories/telemetry/telemetry_shared.go
@@ -13,6 +13,14 @@ type (
LogAttributeFilter = shared.LogAttributeFilter
LogSearchParams = shared.LogSearchParams
SpanLookup = shared.SpanLookup
+ SLOBucket = shared.SLOBucket
+ SLOBucketPoint = shared.SLOBucketPoint
+ EndpointSLISpec = shared.EndpointSLISpec
+ TaskSLISpec = shared.TaskSLISpec
+ SpanSLISpec = shared.SpanSLISpec
+ SessionSLISpec = shared.SessionSLISpec
+ OtelSpanSearch = shared.OtelSpanSearch
+ OtelAttributeFilter = shared.OtelAttributeFilter
)
var (
diff --git a/backend/app/repositories/telemetry/telemetry_sqlite.go b/backend/app/repositories/telemetry/telemetry_sqlite.go
index 4cd7d8ef..368ec7bd 100644
--- a/backend/app/repositories/telemetry/telemetry_sqlite.go
+++ b/backend/app/repositories/telemetry/telemetry_sqlite.go
@@ -18,4 +18,5 @@ var (
OtelSpanRepository = sqliterepo.OtelSpanRepository
LegacyRepository = sqliterepo.LegacyRepository
TaskRepository = sqliterepo.TaskRepository
+ SLOBucketRepository = sqliterepo.SLOBucketRepository
)
diff --git a/backend/app/repositories/transactional/pg/notification_rule_state.repository.go b/backend/app/repositories/transactional/pg/notification_rule_state.repository.go
new file mode 100644
index 00000000..eaf5604f
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/notification_rule_state.repository.go
@@ -0,0 +1,55 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type notificationRuleStateRepository struct{}
+
+func (r *notificationRuleStateRepository) Find(tx *sql.Tx, ruleId int) (*models.NotificationRuleState, error) {
+ return lit.SelectSingleNamed[models.NotificationRuleState](
+ tx,
+ "SELECT id, rule_id, firing, since, updated_at FROM notification_rule_states WHERE rule_id = :rule_id",
+ lit.P{"rule_id": ruleId},
+ )
+}
+
+func (r *notificationRuleStateRepository) Transition(tx *sql.Tx, ruleId int, firing bool, now time.Time) (bool, error) {
+ var since any
+ if firing {
+ since = now.UTC()
+ }
+ params := lit.P{"rule_id": ruleId, "firing": firing, "since": since, "updated_at": now.UTC()}
+ changed, err := execAffected(tx, "UPDATE notification_rule_states SET firing = :firing, since = :since, updated_at = :updated_at WHERE rule_id = :rule_id AND firing <> :firing", params)
+ if err != nil || changed {
+ return changed, err
+ }
+ return execAffected(tx, "INSERT INTO notification_rule_states (rule_id, firing, since, updated_at) VALUES (:rule_id, :firing, :since, :updated_at) ON CONFLICT(rule_id) DO NOTHING", params)
+}
+
+func execAffected(tx *sql.Tx, namedQuery string, params lit.P) (bool, error) {
+ query, args, err := lit.ParseNamedQuery(db.Driver, namedQuery, params)
+ if err != nil {
+ return false, err
+ }
+ result, err := tx.Exec(query, args...)
+ if err != nil {
+ return false, err
+ }
+ affected, err := result.RowsAffected()
+ return affected > 0, err
+}
+
+func (r *notificationRuleStateRepository) Delete(tx *sql.Tx, ruleId int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM notification_rule_states WHERE rule_id = :rule_id", lit.P{"rule_id": ruleId})
+}
+
+var NotificationRuleStateRepository = &notificationRuleStateRepository{}
diff --git a/backend/app/repositories/transactional/pg/slo.repository.go b/backend/app/repositories/transactional/pg/slo.repository.go
new file mode 100644
index 00000000..af363727
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/slo.repository.go
@@ -0,0 +1,162 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+)
+
+type sloRepository struct{}
+
+const sloColumns = "id, project_id, name, description, sli_kind, sli_config, target_percent, window_days, bucket_minutes, enabled, status, current_sli, budget_remaining, burn_rate_1h, burn_rate_6h, computed_at, history_from, last_bucket_at, sli_since, gap_from, gap_to, revision, created_by, created_at, updated_at"
+
+func nullableTime(t *time.Time) any {
+ if t == nil {
+ return nil
+ }
+ return t.UTC()
+}
+
+func nullableFloat(f *float64) any {
+ if f == nil {
+ return nil
+ }
+ return *f
+}
+
+func (r *sloRepository) Create(tx *sql.Tx, slo *models.SLO) (int, error) {
+ return lit.Insert[models.SLO](tx, slo)
+}
+
+func (r *sloRepository) Update(tx *sql.Tx, slo *models.SLO) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE slos SET name = :name, description = :description, sli_kind = :sli_kind, sli_config = :sli_config, target_percent = :target_percent, window_days = :window_days, bucket_minutes = :bucket_minutes, enabled = :enabled, updated_at = :updated_at, revision = revision + 1 WHERE id = :id",
+ lit.P{
+ "name": slo.Name,
+ "description": slo.Description,
+ "sli_kind": slo.SliKind,
+ "sli_config": slo.SliConfig,
+ "target_percent": slo.TargetPercent,
+ "window_days": slo.WindowDays,
+ "bucket_minutes": slo.BucketMinutes,
+ "enabled": slo.Enabled,
+ "updated_at": slo.UpdatedAt.UTC(),
+ "id": slo.Id,
+ },
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+func (r *sloRepository) ResetHistory(tx *sql.Tx, id int, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE slos SET status = 'no_data', current_sli = NULL, budget_remaining = NULL, burn_rate_1h = NULL, burn_rate_6h = NULL, computed_at = NULL, history_from = NULL, last_bucket_at = NULL, gap_from = NULL, gap_to = NULL, sli_since = :sli_since WHERE id = :id",
+ lit.P{"id": id, "sli_since": now.UTC()},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+func (r *sloRepository) UpdateSnapshot(tx *sql.Tx, id int, revision int, snapshot models.SLOSnapshot) (bool, error) {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE slos SET status = :status, current_sli = :current_sli, budget_remaining = :budget_remaining, burn_rate_1h = :burn_rate_1h, burn_rate_6h = :burn_rate_6h, computed_at = :computed_at, history_from = :history_from, last_bucket_at = :last_bucket_at, gap_from = :gap_from, gap_to = :gap_to WHERE id = :id AND revision = :revision",
+ lit.P{
+ "status": snapshot.Status,
+ "current_sli": nullableFloat(snapshot.CurrentSli),
+ "budget_remaining": nullableFloat(snapshot.BudgetRemaining),
+ "burn_rate_1h": nullableFloat(snapshot.BurnRate1h),
+ "burn_rate_6h": nullableFloat(snapshot.BurnRate6h),
+ "computed_at": snapshot.ComputedAt.UTC(),
+ "history_from": nullableTime(snapshot.HistoryFrom),
+ "last_bucket_at": nullableTime(snapshot.LastBucketAt),
+ "gap_from": nullableTime(snapshot.GapFrom),
+ "gap_to": nullableTime(snapshot.GapTo),
+ "id": id,
+ "revision": revision,
+ },
+ )
+ if err != nil {
+ return false, err
+ }
+ result, err := tx.Exec(query, args...)
+ if err != nil {
+ return false, err
+ }
+ affected, err := result.RowsAffected()
+ return affected > 0, err
+}
+
+func (r *sloRepository) Delete(tx *sql.Tx, id int, projectId uuid.UUID) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM slos WHERE id = :id AND project_id = :project_id", lit.P{"id": id, "project_id": projectId})
+}
+
+func (r *sloRepository) FindById(tx *sql.Tx, id int) (*models.SLO, error) {
+ return lit.SelectSingleNamed[models.SLO](
+ tx,
+ "SELECT "+sloColumns+" FROM slos WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *sloRepository) FindByIdForProject(tx *sql.Tx, id int, projectId uuid.UUID) (*models.SLO, error) {
+ return lit.SelectSingleNamed[models.SLO](
+ tx,
+ "SELECT "+sloColumns+" FROM slos WHERE id = :id AND project_id = :project_id",
+ lit.P{"id": id, "project_id": projectId},
+ )
+}
+
+func (r *sloRepository) FindByProject(tx *sql.Tx, projectId uuid.UUID) ([]*models.SLO, error) {
+ return lit.SelectNamed[models.SLO](
+ tx,
+ "SELECT "+sloColumns+" FROM slos WHERE project_id = :project_id ORDER BY name ASC, id ASC",
+ lit.P{"project_id": projectId},
+ )
+}
+
+func (r *sloRepository) FindEnabled(tx *sql.Tx) ([]*models.SLO, error) {
+ return lit.SelectNamed[models.SLO](
+ tx,
+ "SELECT "+sloColumns+" FROM slos WHERE enabled = true ORDER BY id ASC",
+ lit.P{},
+ )
+}
+
+func (r *sloRepository) FindEnabledFreshness(tx *sql.Tx) ([]*models.SLOFreshness, error) {
+ return lit.SelectNamed[models.SLOFreshness](
+ tx,
+ "SELECT id, window_days, computed_at, history_from, last_bucket_at, gap_from, created_at FROM slos WHERE enabled = true",
+ lit.P{},
+ )
+}
+
+func (r *sloRepository) CountBreachedByProject(tx *sql.Tx, projectId uuid.UUID) (int, error) {
+ result, err := lit.SelectSingleNamed[models.CountResult](
+ tx,
+ "SELECT COUNT(*) as count FROM slos WHERE project_id = :project_id AND enabled = true AND status = 'breached'",
+ lit.P{"project_id": projectId},
+ )
+ if err != nil {
+ return 0, err
+ }
+ if result == nil {
+ return 0, nil
+ }
+ return result.Count, nil
+}
+
+var SLORepository = &sloRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/notification_rule_state.repository.go b/backend/app/repositories/transactional/sqlite/notification_rule_state.repository.go
new file mode 100644
index 00000000..0b58d4be
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/notification_rule_state.repository.go
@@ -0,0 +1,55 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type notificationRuleStateRepository struct{}
+
+func (r *notificationRuleStateRepository) Find(tx *sql.Tx, ruleId int) (*models.NotificationRuleState, error) {
+ return lit.SelectSingleNamed[models.NotificationRuleState](
+ tx,
+ "SELECT id, rule_id, firing, since, updated_at FROM notification_rule_states WHERE rule_id = :rule_id",
+ lit.P{"rule_id": ruleId},
+ )
+}
+
+func (r *notificationRuleStateRepository) Transition(tx *sql.Tx, ruleId int, firing bool, now time.Time) (bool, error) {
+ var since any
+ if firing {
+ since = now.UTC()
+ }
+ params := lit.P{"rule_id": ruleId, "firing": firing, "since": since, "updated_at": now.UTC()}
+ changed, err := execAffected(tx, "UPDATE notification_rule_states SET firing = :firing, since = :since, updated_at = :updated_at WHERE rule_id = :rule_id AND firing <> :firing", params)
+ if err != nil || changed {
+ return changed, err
+ }
+ return execAffected(tx, "INSERT INTO notification_rule_states (rule_id, firing, since, updated_at) VALUES (:rule_id, :firing, :since, :updated_at) ON CONFLICT(rule_id) DO NOTHING", params)
+}
+
+func execAffected(tx *sql.Tx, namedQuery string, params lit.P) (bool, error) {
+ query, args, err := lit.ParseNamedQuery(db.Driver, namedQuery, params)
+ if err != nil {
+ return false, err
+ }
+ result, err := tx.Exec(query, args...)
+ if err != nil {
+ return false, err
+ }
+ affected, err := result.RowsAffected()
+ return affected > 0, err
+}
+
+func (r *notificationRuleStateRepository) Delete(tx *sql.Tx, ruleId int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM notification_rule_states WHERE rule_id = :rule_id", lit.P{"rule_id": ruleId})
+}
+
+var NotificationRuleStateRepository = &notificationRuleStateRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/slo.repository.go b/backend/app/repositories/transactional/sqlite/slo.repository.go
new file mode 100644
index 00000000..8182304a
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/slo.repository.go
@@ -0,0 +1,162 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+)
+
+type sloRepository struct{}
+
+const sloColumns = "id, project_id, name, description, sli_kind, sli_config, target_percent, window_days, bucket_minutes, enabled, status, current_sli, budget_remaining, burn_rate_1h, burn_rate_6h, computed_at, history_from, last_bucket_at, sli_since, gap_from, gap_to, revision, created_by, created_at, updated_at"
+
+func nullableTime(t *time.Time) any {
+ if t == nil {
+ return nil
+ }
+ return t.UTC()
+}
+
+func nullableFloat(f *float64) any {
+ if f == nil {
+ return nil
+ }
+ return *f
+}
+
+func (r *sloRepository) Create(tx *sql.Tx, slo *models.SLO) (int, error) {
+ return lit.Insert[models.SLO](tx, slo)
+}
+
+func (r *sloRepository) Update(tx *sql.Tx, slo *models.SLO) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE slos SET name = :name, description = :description, sli_kind = :sli_kind, sli_config = :sli_config, target_percent = :target_percent, window_days = :window_days, bucket_minutes = :bucket_minutes, enabled = :enabled, updated_at = :updated_at, revision = revision + 1 WHERE id = :id",
+ lit.P{
+ "name": slo.Name,
+ "description": slo.Description,
+ "sli_kind": slo.SliKind,
+ "sli_config": slo.SliConfig,
+ "target_percent": slo.TargetPercent,
+ "window_days": slo.WindowDays,
+ "bucket_minutes": slo.BucketMinutes,
+ "enabled": slo.Enabled,
+ "updated_at": slo.UpdatedAt.UTC(),
+ "id": slo.Id,
+ },
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+func (r *sloRepository) ResetHistory(tx *sql.Tx, id int, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE slos SET status = 'no_data', current_sli = NULL, budget_remaining = NULL, burn_rate_1h = NULL, burn_rate_6h = NULL, computed_at = NULL, history_from = NULL, last_bucket_at = NULL, gap_from = NULL, gap_to = NULL, sli_since = :sli_since WHERE id = :id",
+ lit.P{"id": id, "sli_since": now.UTC()},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+func (r *sloRepository) UpdateSnapshot(tx *sql.Tx, id int, revision int, snapshot models.SLOSnapshot) (bool, error) {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE slos SET status = :status, current_sli = :current_sli, budget_remaining = :budget_remaining, burn_rate_1h = :burn_rate_1h, burn_rate_6h = :burn_rate_6h, computed_at = :computed_at, history_from = :history_from, last_bucket_at = :last_bucket_at, gap_from = :gap_from, gap_to = :gap_to WHERE id = :id AND revision = :revision",
+ lit.P{
+ "status": snapshot.Status,
+ "current_sli": nullableFloat(snapshot.CurrentSli),
+ "budget_remaining": nullableFloat(snapshot.BudgetRemaining),
+ "burn_rate_1h": nullableFloat(snapshot.BurnRate1h),
+ "burn_rate_6h": nullableFloat(snapshot.BurnRate6h),
+ "computed_at": snapshot.ComputedAt.UTC(),
+ "history_from": nullableTime(snapshot.HistoryFrom),
+ "last_bucket_at": nullableTime(snapshot.LastBucketAt),
+ "gap_from": nullableTime(snapshot.GapFrom),
+ "gap_to": nullableTime(snapshot.GapTo),
+ "id": id,
+ "revision": revision,
+ },
+ )
+ if err != nil {
+ return false, err
+ }
+ result, err := tx.Exec(query, args...)
+ if err != nil {
+ return false, err
+ }
+ affected, err := result.RowsAffected()
+ return affected > 0, err
+}
+
+func (r *sloRepository) Delete(tx *sql.Tx, id int, projectId uuid.UUID) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM slos WHERE id = :id AND project_id = :project_id", lit.P{"id": id, "project_id": projectId})
+}
+
+func (r *sloRepository) FindById(tx *sql.Tx, id int) (*models.SLO, error) {
+ return lit.SelectSingleNamed[models.SLO](
+ tx,
+ "SELECT "+sloColumns+" FROM slos WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *sloRepository) FindByIdForProject(tx *sql.Tx, id int, projectId uuid.UUID) (*models.SLO, error) {
+ return lit.SelectSingleNamed[models.SLO](
+ tx,
+ "SELECT "+sloColumns+" FROM slos WHERE id = :id AND project_id = :project_id",
+ lit.P{"id": id, "project_id": projectId},
+ )
+}
+
+func (r *sloRepository) FindByProject(tx *sql.Tx, projectId uuid.UUID) ([]*models.SLO, error) {
+ return lit.SelectNamed[models.SLO](
+ tx,
+ "SELECT "+sloColumns+" FROM slos WHERE project_id = :project_id ORDER BY name ASC, id ASC",
+ lit.P{"project_id": projectId},
+ )
+}
+
+func (r *sloRepository) FindEnabled(tx *sql.Tx) ([]*models.SLO, error) {
+ return lit.SelectNamed[models.SLO](
+ tx,
+ "SELECT "+sloColumns+" FROM slos WHERE enabled = true ORDER BY id ASC",
+ lit.P{},
+ )
+}
+
+func (r *sloRepository) FindEnabledFreshness(tx *sql.Tx) ([]*models.SLOFreshness, error) {
+ return lit.SelectNamed[models.SLOFreshness](
+ tx,
+ "SELECT id, window_days, computed_at, history_from, last_bucket_at, gap_from, created_at FROM slos WHERE enabled = true",
+ lit.P{},
+ )
+}
+
+func (r *sloRepository) CountBreachedByProject(tx *sql.Tx, projectId uuid.UUID) (int, error) {
+ result, err := lit.SelectSingleNamed[models.CountResult](
+ tx,
+ "SELECT COUNT(*) as count FROM slos WHERE project_id = :project_id AND enabled = true AND status = 'breached'",
+ lit.P{"project_id": projectId},
+ )
+ if err != nil {
+ return 0, err
+ }
+ if result == nil {
+ return 0, nil
+ }
+ return result.Count, nil
+}
+
+var SLORepository = &sloRepository{}
diff --git a/backend/app/repositories/transactional/transactional_pg.go b/backend/app/repositories/transactional/transactional_pg.go
index b950e9ec..0d8bba05 100644
--- a/backend/app/repositories/transactional/transactional_pg.go
+++ b/backend/app/repositories/transactional/transactional_pg.go
@@ -5,39 +5,41 @@ package transactional
import pgrepo "github.com/tracewayapp/traceway/backend/app/repositories/transactional/pg"
var (
- AgentWriteEventRepository = pgrepo.AgentWriteEventRepository
- AuthorizationCodeRepository = pgrepo.AuthorizationCodeRepository
- CheckIncidentRepository = pgrepo.CheckIncidentRepository
- CheckRunRepository = pgrepo.CheckRunRepository
- DashboardRepository = pgrepo.DashboardRepository
- DashboardTemplateRepository = pgrepo.DashboardTemplateRepository
- DeviceAuthorizationRepository = pgrepo.DeviceAuthorizationRepository
- EscalationPolicyRepository = pgrepo.EscalationPolicyRepository
- IncidentUpdateRepository = pgrepo.IncidentUpdateRepository
- InvitationRepository = pgrepo.InvitationRepository
- MetricRegistryRepository = pgrepo.MetricRegistryRepository
- NotificationChannelRepository = pgrepo.NotificationChannelRepository
- NotificationRuleRepository = pgrepo.NotificationRuleRepository
- OncallOverrideRepository = pgrepo.OncallOverrideRepository
- OncallScheduleRepository = pgrepo.OncallScheduleRepository
- OAuthSessionRepository = pgrepo.OAuthSessionRepository
- OauthClientRepository = pgrepo.OauthClientRepository
- OrganizationRepository = pgrepo.OrganizationRepository
- OutboxRepository = pgrepo.OutboxRepository
- PageNotificationRepository = pgrepo.PageNotificationRepository
- PageRepository = pgrepo.PageRepository
- PersonalAccessTokenRepository = pgrepo.PersonalAccessTokenRepository
- PostMortemRepository = pgrepo.PostMortemRepository
- ProjectRepository = pgrepo.ProjectRepository
- ProjectUserRoleRepository = pgrepo.ProjectUserRoleRepository
- RefreshTokenRepository = pgrepo.RefreshTokenRepository
- SetupPlanRepository = pgrepo.SetupPlanRepository
- SetupTokenRepository = pgrepo.SetupTokenRepository
- StatusPageRepository = pgrepo.StatusPageRepository
- SyntheticCheckRepository = pgrepo.SyntheticCheckRepository
- SyntheticRunnerRepository = pgrepo.SyntheticRunnerRepository
- TeamRepository = pgrepo.TeamRepository
- UserContactMethodRepository = pgrepo.UserContactMethodRepository
- UserNotificationRuleRepository = pgrepo.UserNotificationRuleRepository
- UserRepository = pgrepo.UserRepository
+ AgentWriteEventRepository = pgrepo.AgentWriteEventRepository
+ AuthorizationCodeRepository = pgrepo.AuthorizationCodeRepository
+ CheckIncidentRepository = pgrepo.CheckIncidentRepository
+ CheckRunRepository = pgrepo.CheckRunRepository
+ DashboardRepository = pgrepo.DashboardRepository
+ DashboardTemplateRepository = pgrepo.DashboardTemplateRepository
+ DeviceAuthorizationRepository = pgrepo.DeviceAuthorizationRepository
+ EscalationPolicyRepository = pgrepo.EscalationPolicyRepository
+ IncidentUpdateRepository = pgrepo.IncidentUpdateRepository
+ InvitationRepository = pgrepo.InvitationRepository
+ MetricRegistryRepository = pgrepo.MetricRegistryRepository
+ NotificationChannelRepository = pgrepo.NotificationChannelRepository
+ NotificationRuleRepository = pgrepo.NotificationRuleRepository
+ NotificationRuleStateRepository = pgrepo.NotificationRuleStateRepository
+ OncallOverrideRepository = pgrepo.OncallOverrideRepository
+ OncallScheduleRepository = pgrepo.OncallScheduleRepository
+ OAuthSessionRepository = pgrepo.OAuthSessionRepository
+ OauthClientRepository = pgrepo.OauthClientRepository
+ OrganizationRepository = pgrepo.OrganizationRepository
+ OutboxRepository = pgrepo.OutboxRepository
+ PageNotificationRepository = pgrepo.PageNotificationRepository
+ PageRepository = pgrepo.PageRepository
+ PersonalAccessTokenRepository = pgrepo.PersonalAccessTokenRepository
+ PostMortemRepository = pgrepo.PostMortemRepository
+ ProjectRepository = pgrepo.ProjectRepository
+ ProjectUserRoleRepository = pgrepo.ProjectUserRoleRepository
+ RefreshTokenRepository = pgrepo.RefreshTokenRepository
+ SetupPlanRepository = pgrepo.SetupPlanRepository
+ SetupTokenRepository = pgrepo.SetupTokenRepository
+ SLORepository = pgrepo.SLORepository
+ StatusPageRepository = pgrepo.StatusPageRepository
+ SyntheticCheckRepository = pgrepo.SyntheticCheckRepository
+ SyntheticRunnerRepository = pgrepo.SyntheticRunnerRepository
+ TeamRepository = pgrepo.TeamRepository
+ UserContactMethodRepository = pgrepo.UserContactMethodRepository
+ UserNotificationRuleRepository = pgrepo.UserNotificationRuleRepository
+ UserRepository = pgrepo.UserRepository
)
diff --git a/backend/app/repositories/transactional/transactional_sqlite.go b/backend/app/repositories/transactional/transactional_sqlite.go
index bfa5d597..c857b46d 100644
--- a/backend/app/repositories/transactional/transactional_sqlite.go
+++ b/backend/app/repositories/transactional/transactional_sqlite.go
@@ -5,39 +5,41 @@ package transactional
import sqliterepo "github.com/tracewayapp/traceway/backend/app/repositories/transactional/sqlite"
var (
- AgentWriteEventRepository = sqliterepo.AgentWriteEventRepository
- AuthorizationCodeRepository = sqliterepo.AuthorizationCodeRepository
- CheckIncidentRepository = sqliterepo.CheckIncidentRepository
- CheckRunRepository = sqliterepo.CheckRunRepository
- DashboardRepository = sqliterepo.DashboardRepository
- DashboardTemplateRepository = sqliterepo.DashboardTemplateRepository
- DeviceAuthorizationRepository = sqliterepo.DeviceAuthorizationRepository
- EscalationPolicyRepository = sqliterepo.EscalationPolicyRepository
- IncidentUpdateRepository = sqliterepo.IncidentUpdateRepository
- InvitationRepository = sqliterepo.InvitationRepository
- MetricRegistryRepository = sqliterepo.MetricRegistryRepository
- NotificationChannelRepository = sqliterepo.NotificationChannelRepository
- NotificationRuleRepository = sqliterepo.NotificationRuleRepository
- OncallOverrideRepository = sqliterepo.OncallOverrideRepository
- OncallScheduleRepository = sqliterepo.OncallScheduleRepository
- OAuthSessionRepository = sqliterepo.OAuthSessionRepository
- OauthClientRepository = sqliterepo.OauthClientRepository
- OrganizationRepository = sqliterepo.OrganizationRepository
- OutboxRepository = sqliterepo.OutboxRepository
- PageNotificationRepository = sqliterepo.PageNotificationRepository
- PageRepository = sqliterepo.PageRepository
- PersonalAccessTokenRepository = sqliterepo.PersonalAccessTokenRepository
- PostMortemRepository = sqliterepo.PostMortemRepository
- ProjectRepository = sqliterepo.ProjectRepository
- ProjectUserRoleRepository = sqliterepo.ProjectUserRoleRepository
- RefreshTokenRepository = sqliterepo.RefreshTokenRepository
- SetupPlanRepository = sqliterepo.SetupPlanRepository
- SetupTokenRepository = sqliterepo.SetupTokenRepository
- StatusPageRepository = sqliterepo.StatusPageRepository
- SyntheticCheckRepository = sqliterepo.SyntheticCheckRepository
- SyntheticRunnerRepository = sqliterepo.SyntheticRunnerRepository
- TeamRepository = sqliterepo.TeamRepository
- UserContactMethodRepository = sqliterepo.UserContactMethodRepository
- UserNotificationRuleRepository = sqliterepo.UserNotificationRuleRepository
- UserRepository = sqliterepo.UserRepository
+ AgentWriteEventRepository = sqliterepo.AgentWriteEventRepository
+ AuthorizationCodeRepository = sqliterepo.AuthorizationCodeRepository
+ CheckIncidentRepository = sqliterepo.CheckIncidentRepository
+ CheckRunRepository = sqliterepo.CheckRunRepository
+ DashboardRepository = sqliterepo.DashboardRepository
+ DashboardTemplateRepository = sqliterepo.DashboardTemplateRepository
+ DeviceAuthorizationRepository = sqliterepo.DeviceAuthorizationRepository
+ EscalationPolicyRepository = sqliterepo.EscalationPolicyRepository
+ IncidentUpdateRepository = sqliterepo.IncidentUpdateRepository
+ InvitationRepository = sqliterepo.InvitationRepository
+ MetricRegistryRepository = sqliterepo.MetricRegistryRepository
+ NotificationChannelRepository = sqliterepo.NotificationChannelRepository
+ NotificationRuleRepository = sqliterepo.NotificationRuleRepository
+ NotificationRuleStateRepository = sqliterepo.NotificationRuleStateRepository
+ OncallOverrideRepository = sqliterepo.OncallOverrideRepository
+ OncallScheduleRepository = sqliterepo.OncallScheduleRepository
+ OAuthSessionRepository = sqliterepo.OAuthSessionRepository
+ OauthClientRepository = sqliterepo.OauthClientRepository
+ OrganizationRepository = sqliterepo.OrganizationRepository
+ OutboxRepository = sqliterepo.OutboxRepository
+ PageNotificationRepository = sqliterepo.PageNotificationRepository
+ PageRepository = sqliterepo.PageRepository
+ PersonalAccessTokenRepository = sqliterepo.PersonalAccessTokenRepository
+ PostMortemRepository = sqliterepo.PostMortemRepository
+ ProjectRepository = sqliterepo.ProjectRepository
+ ProjectUserRoleRepository = sqliterepo.ProjectUserRoleRepository
+ RefreshTokenRepository = sqliterepo.RefreshTokenRepository
+ SetupPlanRepository = sqliterepo.SetupPlanRepository
+ SetupTokenRepository = sqliterepo.SetupTokenRepository
+ SLORepository = sqliterepo.SLORepository
+ StatusPageRepository = sqliterepo.StatusPageRepository
+ SyntheticCheckRepository = sqliterepo.SyntheticCheckRepository
+ SyntheticRunnerRepository = sqliterepo.SyntheticRunnerRepository
+ TeamRepository = sqliterepo.TeamRepository
+ UserContactMethodRepository = sqliterepo.UserContactMethodRepository
+ UserNotificationRuleRepository = sqliterepo.UserNotificationRuleRepository
+ UserRepository = sqliterepo.UserRepository
)
diff --git a/backend/app/slo/budget.go b/backend/app/slo/budget.go
new file mode 100644
index 00000000..d9ac09d9
--- /dev/null
+++ b/backend/app/slo/budget.go
@@ -0,0 +1,72 @@
+package slo
+
+import (
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+type Window struct {
+ Good int64
+ Total int64
+}
+
+func (w Window) Bad() int64 {
+ return w.Total - w.Good
+}
+
+func (w Window) SLI() (float64, bool) {
+ if w.Total <= 0 {
+ return 0, false
+ }
+ return float64(w.Good) / float64(w.Total), true
+}
+
+func BudgetRemaining(w Window, target float64) (float64, bool) {
+ sli, ok := w.SLI()
+ if !ok || target >= 1 {
+ return 0, false
+ }
+ return 100 * (1 - (1-sli)/(1-target)), true
+}
+
+func BurnRate(w Window, target float64) float64 {
+ if w.Total <= 0 || target >= 1 {
+ return 0
+ }
+ return (float64(w.Bad()) / float64(w.Total)) / (1 - target)
+}
+
+func Status(w Window, target float64) string {
+ sli, ok := w.SLI()
+ if !ok {
+ return models.SLOStatusNoData
+ }
+ if sli < target {
+ return models.SLOStatusBreached
+ }
+ return models.SLOStatusOk
+}
+
+func Snapshot(window, lastHour, lastSixHours Window, target float64, now time.Time, historyFrom, lastBucketAt *time.Time) models.SLOSnapshot {
+ snapshot := models.SLOSnapshot{
+ Status: Status(window, target),
+ ComputedAt: now,
+ HistoryFrom: historyFrom,
+ LastBucketAt: lastBucketAt,
+ }
+ if sli, ok := window.SLI(); ok {
+ percent := sli * 100
+ snapshot.CurrentSli = &percent
+ }
+ if remaining, ok := BudgetRemaining(window, target); ok {
+ snapshot.BudgetRemaining = &remaining
+ }
+ if window.Total > 0 {
+ burn1h := BurnRate(lastHour, target)
+ burn6h := BurnRate(lastSixHours, target)
+ snapshot.BurnRate1h = &burn1h
+ snapshot.BurnRate6h = &burn6h
+ }
+ return snapshot
+}
diff --git a/backend/app/slo/budget_test.go b/backend/app/slo/budget_test.go
new file mode 100644
index 00000000..0a879c5c
--- /dev/null
+++ b/backend/app/slo/budget_test.go
@@ -0,0 +1,64 @@
+package slo
+
+import (
+ "math"
+ "testing"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+func near(got, want float64) bool {
+ return math.Abs(got-want) < 1e-6
+}
+
+func TestBudgetMath(t *testing.T) {
+ target := 0.999
+ window := Window{Good: 999_500, Total: 1_000_000}
+ sli, ok := window.SLI()
+ if !ok || !near(sli, 0.9995) {
+ t.Fatalf("sli = %v, %v", sli, ok)
+ }
+ remaining, ok := BudgetRemaining(window, target)
+ if !ok || !near(remaining, 50) {
+ t.Fatalf("half the budget left: got %v", remaining)
+ }
+ overspent := Window{Good: 998_800, Total: 1_000_000}
+ remaining, _ = BudgetRemaining(overspent, target)
+ if !near(remaining, -20) {
+ t.Fatalf("1200 bad of 1000 allowed is -20%%: got %v", remaining)
+ }
+ if burn := BurnRate(Window{Good: 9_856, Total: 10_000}, target); !near(burn, 14.4) {
+ t.Fatalf("144 bad per 10000 at 99.9%% is 14.4x: got %v", burn)
+ }
+ if burn := BurnRate(Window{}, target); burn != 0 {
+ t.Fatalf("no traffic burns nothing: got %v", burn)
+ }
+ if _, ok := BudgetRemaining(window, 1); ok {
+ t.Fatal("a 100% target has no budget to measure")
+ }
+}
+
+func TestStatus(t *testing.T) {
+ if got := Status(Window{}, 0.99); got != models.SLOStatusNoData {
+ t.Fatalf("empty window: %s", got)
+ }
+ if got := Status(Window{Good: 99, Total: 100}, 0.99); got != models.SLOStatusOk {
+ t.Fatalf("exactly on target is ok: %s", got)
+ }
+ if got := Status(Window{Good: 98, Total: 100}, 0.99); got != models.SLOStatusBreached {
+ t.Fatalf("below target is breached: %s", got)
+ }
+}
+
+func TestSnapshot(t *testing.T) {
+ now := time.Now().UTC()
+ empty := Snapshot(Window{}, Window{}, Window{}, 0.99, now, nil, nil)
+ if empty.Status != models.SLOStatusNoData || empty.CurrentSli != nil || empty.BudgetRemaining != nil || empty.BurnRate1h != nil {
+ t.Fatalf("empty snapshot carries values: %+v", empty)
+ }
+ full := Snapshot(Window{Good: 995, Total: 1000}, Window{Good: 90, Total: 100}, Window{Good: 100, Total: 100}, 0.99, now, &now, &now)
+ if full.Status != models.SLOStatusOk || !near(*full.CurrentSli, 99.5) || !near(*full.BudgetRemaining, 50) || !near(*full.BurnRate1h, 10) || !near(*full.BurnRate6h, 0) {
+ t.Fatalf("snapshot: %+v", full)
+ }
+}
diff --git a/backend/app/slo/compute.go b/backend/app/slo/compute.go
new file mode 100644
index 00000000..4bb5ecc8
--- /dev/null
+++ b/backend/app/slo/compute.go
@@ -0,0 +1,134 @@
+package slo
+
+import (
+ "context"
+ "fmt"
+ "sort"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func ComputeBuckets(ctx context.Context, slo *models.SLO, from, to time.Time) ([]shared.SLOBucketPoint, error) {
+ cfg, err := models.ParseSLIConfig(slo.SliKind, slo.SliConfig)
+ if err != nil {
+ return nil, err
+ }
+ bucketMinutes := slo.BucketMinutes
+ if bucketMinutes < 1 {
+ bucketMinutes = 1
+ }
+ switch slo.SliKind {
+ case models.SLIKindEndpoint:
+ spec := shared.EndpointSLISpec{
+ Endpoint: cfg.Endpoint.Endpoint,
+ Latency: cfg.Endpoint.Good == models.SLOGoodLatency,
+ LatencyThreshold: millis(cfg.Endpoint.LatencyThresholdMs),
+ }
+ return telemetry.EndpointRepository.SLICounts(ctx, slo.ProjectId, spec, from, to, bucketMinutes)
+ case models.SLIKindSpan:
+ spec := shared.SpanSLISpec{
+ Filter: spanFilter(slo.ProjectId, cfg.Span),
+ Latency: cfg.Span.Good == models.SLOGoodLatency,
+ LatencyThreshold: millis(cfg.Span.LatencyThresholdMs),
+ }
+ return telemetry.OtelSpanRepository.SLICounts(ctx, spec, from, to, bucketMinutes)
+ case models.SLIKindTask:
+ return telemetry.TaskRepository.SLICounts(ctx, slo.ProjectId, shared.TaskSLISpec{TaskName: cfg.Task.TaskName}, from, to, bucketMinutes)
+ case models.SLIKindMonitor:
+ return monitorBuckets(ctx, slo, cfg.Monitor, from, to, bucketMinutes)
+ case models.SLIKindMetric:
+ return metricBuckets(ctx, slo, cfg.Metric, from, to, bucketMinutes)
+ case models.SLIKindSession:
+ spec := shared.SessionSLISpec{AppVersion: cfg.Session.AppVersion, IncludeMessages: cfg.Session.IncludeMessages}
+ return telemetry.SessionRepository.SLICounts(ctx, slo.ProjectId, spec, from, to, bucketMinutes)
+ }
+ return nil, fmt.Errorf("unknown SLI kind %q", slo.SliKind)
+}
+
+func millis(ms float64) time.Duration {
+ return time.Duration(ms * float64(time.Millisecond))
+}
+
+func spanFilter(projectId uuid.UUID, cfg *models.SpanSLIConfig) shared.OtelSpanSearch {
+ filter := shared.OtelSpanSearch{ProjectId: projectId, Service: cfg.ServiceName, Name: cfg.Name, Kind: cfg.Kind}
+ for _, attribute := range cfg.Attributes {
+ filter.Attributes = append(filter.Attributes, shared.OtelAttributeFilter{Key: attribute.Key, Value: attribute.Value})
+ }
+ return filter
+}
+
+func monitorBuckets(ctx context.Context, slo *models.SLO, cfg *models.MonitorSLIConfig, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ sums := map[int64]*shared.SLOBucketPoint{}
+ for _, checkId := range cfg.CheckIds {
+ points, err := telemetry.CheckResultRepository.SeriesByCheck(ctx, slo.ProjectId, checkId, from, to, bucketMinutes)
+ if err != nil {
+ return nil, err
+ }
+ for _, point := range points {
+ if point.Bucket.Before(from) || !point.Bucket.Before(to) {
+ continue
+ }
+ key := point.Bucket.Unix()
+ sum, ok := sums[key]
+ if !ok {
+ sum = &shared.SLOBucketPoint{Bucket: point.Bucket.UTC()}
+ sums[key] = sum
+ }
+ sum.Good += point.Up
+ sum.Total += point.Total - point.Missed
+ }
+ }
+ result := make([]shared.SLOBucketPoint, 0, len(sums))
+ for _, sum := range sums {
+ result = append(result, *sum)
+ }
+ sort.Slice(result, func(i, j int) bool { return result[i].Bucket.Before(result[j].Bucket) })
+ return result, nil
+}
+
+func metricBuckets(ctx context.Context, slo *models.SLO, cfg *models.MetricSLIConfig, from, to time.Time, bucketMinutes int) ([]shared.SLOBucketPoint, error) {
+ series, err := telemetry.MetricPointRepository.QueryTimeSeries(ctx, slo.ProjectId, cfg.MetricName, from, to, bucketMinutes, cfg.Aggregation, cfg.Tags, "", 1)
+ if err != nil {
+ return nil, err
+ }
+ values := map[int64]float64{}
+ for _, points := range series {
+ for _, point := range points {
+ values[point.Timestamp.Unix()] = point.Value
+ }
+ }
+ bucket := time.Duration(bucketMinutes) * time.Minute
+ missingFrom := slo.MissingCountsFrom()
+ result := make([]shared.SLOBucketPoint, 0, int(to.Sub(from)/bucket))
+ for at := from; at.Before(to); at = at.Add(bucket) {
+ point := shared.SLOBucketPoint{Bucket: at.UTC()}
+ if value, ok := values[at.Unix()]; ok {
+ point.Total = 1
+ if metricSliceGood(value, cfg.Operator, cfg.Threshold) {
+ point.Good = 1
+ }
+ } else if cfg.MissingIsBad && !at.Before(missingFrom) {
+ point.Total = 1
+ }
+ result = append(result, point)
+ }
+ return result, nil
+}
+
+func metricSliceGood(value float64, operator string, threshold float64) bool {
+ switch operator {
+ case "gt":
+ return value > threshold
+ case "gte":
+ return value >= threshold
+ case "lt":
+ return value < threshold
+ case "lte":
+ return value <= threshold
+ }
+ return false
+}
diff --git a/backend/app/slo/health.go b/backend/app/slo/health.go
new file mode 100644
index 00000000..4a56d416
--- /dev/null
+++ b/backend/app/slo/health.go
@@ -0,0 +1,141 @@
+package slo
+
+import (
+ "context"
+ "database/sql"
+ "sync"
+ "sync/atomic"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+const staleAfter = 5 * time.Minute
+
+var (
+ computedTotal uint64
+ errorsTotal uint64
+ timeoutsTotal uint64
+ lastTickUnix atomic.Int64
+ tickStarted atomic.Int64
+
+ lastTickMu sync.Mutex
+ lastTick *TickStats
+)
+
+type SlowestSLO struct {
+ Id int `json:"id"`
+ ProjectId uuid.UUID `json:"projectId"`
+ Kind string `json:"kind"`
+ DurationMs float64 `json:"durationMs"`
+}
+
+type TickStats struct {
+ At time.Time `json:"at"`
+ DurationMs float64 `json:"durationMs"`
+ ComputeMs float64 `json:"computeMs"`
+ StoreMs float64 `json:"storeMs"`
+ SnapshotMs float64 `json:"snapshotMs"`
+ SLOs int `json:"slos"`
+ Errors int `json:"errors"`
+ Timeouts int `json:"timeouts"`
+ FullSnapshots int `json:"fullSnapshots"`
+ Buckets int `json:"buckets"`
+ Slowest *SlowestSLO `json:"slowest"`
+}
+
+type Freshness struct {
+ Enabled int `json:"enabled"`
+ Stale int `json:"stale"`
+ MaxStalenessSec float64 `json:"maxStalenessSec"`
+ BackfillPending int `json:"backfillPending"`
+}
+
+type HealthStats struct {
+ ComputedTotal uint64 `json:"computedTotal"`
+ ErrorsTotal uint64 `json:"errorsTotal"`
+ TimeoutsTotal uint64 `json:"timeoutsTotal"`
+ LastTickAt *time.Time `json:"lastTickAt"`
+ RunningForSec float64 `json:"runningForSec"`
+ OwnedShards int `json:"ownedShards"`
+ Instances int `json:"instances"`
+ LastTick *TickStats `json:"lastTick"`
+ Freshness *Freshness `json:"freshness"`
+}
+
+func HealthSnapshot(ctx context.Context) (*HealthStats, error) {
+ stats := &HealthStats{
+ ComputedTotal: atomic.LoadUint64(&computedTotal),
+ ErrorsTotal: atomic.LoadUint64(&errorsTotal),
+ TimeoutsTotal: atomic.LoadUint64(&timeoutsTotal),
+ OwnedShards: int(ownedShards.Load()),
+ Instances: int(knownInstances.Load()),
+ }
+ if unix := lastTickUnix.Load(); unix > 0 {
+ at := time.Unix(unix, 0).UTC()
+ stats.LastTickAt = &at
+ }
+ if started := tickStarted.Load(); started > 0 {
+ stats.RunningForSec = time.Since(time.Unix(0, started)).Seconds()
+ }
+ lastTickMu.Lock()
+ if lastTick != nil {
+ copied := *lastTick
+ stats.LastTick = &copied
+ }
+ lastTickMu.Unlock()
+ freshness, err := ReadFreshness(ctx, time.Now().UTC())
+ if err != nil {
+ return stats, err
+ }
+ stats.Freshness = freshness
+ return stats, nil
+}
+
+// ReadFreshness reads every enabled SLO's last computation from the main
+// database, so it covers all instances and also catches a worker that stopped.
+func ReadFreshness(ctx context.Context, now time.Time) (*Freshness, error) {
+ rows, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.SLOFreshness, error) {
+ return transactional.SLORepository.FindEnabledFreshness(tx)
+ })
+ if err != nil {
+ return nil, err
+ }
+ freshness := &Freshness{Enabled: len(rows)}
+ for _, row := range rows {
+ last := row.CreatedAt
+ if row.ComputedAt != nil {
+ last = *row.ComputedAt
+ }
+ staleness := now.Sub(last).Seconds()
+ if staleness < 0 {
+ staleness = 0
+ }
+ if staleness > freshness.MaxStalenessSec {
+ freshness.MaxStalenessSec = staleness
+ }
+ if row.ComputedAt == nil || now.Sub(*row.ComputedAt) > staleAfter {
+ freshness.Stale++
+ }
+ if backfillPending(row) {
+ freshness.BackfillPending++
+ }
+ }
+ return freshness, nil
+}
+
+func backfillPending(row *models.SLOFreshness) bool {
+ if row.GapFrom != nil || row.HistoryFrom == nil || row.LastBucketAt == nil {
+ return true
+ }
+ return row.HistoryFrom.After(row.LastBucketAt.Add(-time.Duration(row.WindowDays) * 24 * time.Hour))
+}
+
+func recordTick(stats *TickStats) {
+ lastTickMu.Lock()
+ lastTick = stats
+ lastTickMu.Unlock()
+}
diff --git a/backend/app/slo/schedule.go b/backend/app/slo/schedule.go
new file mode 100644
index 00000000..e830b6ff
--- /dev/null
+++ b/backend/app/slo/schedule.go
@@ -0,0 +1,50 @@
+package slo
+
+import (
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+func slotOffset(sloId int, interval time.Duration) time.Duration {
+ seconds := uint64(interval / time.Second)
+ if seconds == 0 {
+ return 0
+ }
+ return time.Duration((uint64(sloId)*2654435761)%seconds) * time.Second
+}
+
+func slotIndex(t time.Time, offset, interval time.Duration) int64 {
+ n := t.UnixNano() - int64(offset)
+ d := int64(interval)
+ q := n / d
+ if n%d != 0 && n < 0 {
+ q--
+ }
+ return q
+}
+
+// passDue reports whether the SLO's own slot of the interval falls in
+// (since, now]. Slots are spread by SLO id and keyed on the persisted
+// computed_at, so a restart or a shard moving to another instance neither
+// skips a pass nor runs every SLO's pass on the same tick.
+func passDue(sloId int, interval time.Duration, since *time.Time, now time.Time) bool {
+ if since == nil {
+ return true
+ }
+ if !now.After(*since) {
+ return false
+ }
+ offset := slotOffset(sloId, interval)
+ return slotIndex(now, offset, interval) > slotIndex(*since, offset, interval)
+}
+
+func recomputeHorizon(slo *models.SLO, now time.Time) time.Duration {
+ if slo.SliKind == models.SLIKindSession && passDue(slo.Id, lateCrashInterval, slo.ComputedAt, now) {
+ return lateCrashHorizon
+ }
+ if passDue(slo.Id, lateArrivalInterval, slo.ComputedAt, now) {
+ return lateArrivalHorizon(slo.SliKind)
+ }
+ return lateDataWindow
+}
diff --git a/backend/app/slo/schedule_test.go b/backend/app/slo/schedule_test.go
new file mode 100644
index 00000000..4ee6bfec
--- /dev/null
+++ b/backend/app/slo/schedule_test.go
@@ -0,0 +1,53 @@
+package slo
+
+import (
+ "testing"
+ "time"
+)
+
+func TestPassDueRunsEverySLOOncePerIntervalSpreadOverTicks(t *testing.T) {
+ start := time.Date(2026, 10, 27, 12, 0, 0, 0, time.UTC)
+ const slos = 900
+ runs := make([]int, slos)
+ busiestTick := 0
+ for minute := 1; minute <= 60; minute++ {
+ since, now := start.Add(time.Duration(minute-1)*time.Minute), start.Add(time.Duration(minute)*time.Minute)
+ due := 0
+ for id := 1; id <= slos; id++ {
+ if passDue(id, lateArrivalInterval, &since, now) {
+ runs[id-1]++
+ due++
+ }
+ }
+ busiestTick = max(busiestTick, due)
+ }
+ for id, count := range runs {
+ if count != 4 {
+ t.Fatalf("SLO %d ran its 15-minute pass %d times in an hour, want 4", id+1, count)
+ }
+ }
+ if busiestTick > slos/5 {
+ t.Fatalf("%d of %d SLOs ran their pass on the same tick; the passes are not spread", busiestTick, slos)
+ }
+}
+
+func TestPassDueAfterARestartDoesNotRunEveryPass(t *testing.T) {
+ now := time.Date(2026, 10, 27, 12, 0, 30, 0, time.UTC)
+ since := now.Add(-time.Minute)
+ due := 0
+ for id := 1; id <= 1000; id++ {
+ if passDue(id, lateCrashInterval, &since, now) {
+ due++
+ }
+ }
+ if due > 5 {
+ t.Fatalf("%d of 1000 daily passes ran on one tick after a restart", due)
+ }
+ if !passDue(1, lateCrashInterval, nil, now) {
+ t.Fatal("an SLO that was never computed runs every pass")
+ }
+ longAgo := now.Add(-25 * time.Hour)
+ if !passDue(1, lateCrashInterval, &longAgo, now) {
+ t.Fatal("an SLO that was not computed for a whole interval runs its pass")
+ }
+}
diff --git a/backend/app/slo/shards.go b/backend/app/slo/shards.go
new file mode 100644
index 00000000..af7a4980
--- /dev/null
+++ b/backend/app/slo/shards.go
@@ -0,0 +1,136 @@
+package slo
+
+import (
+ "context"
+ "fmt"
+ "sync/atomic"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+)
+
+// SLOs are split into shardCount shards by id. On Postgres every backend
+// instance leases a fair share of the shards, so the work spreads across
+// instances instead of one advisory-lock holder computing every tenant.
+const shardCount = 64
+
+var (
+ workerId = uuid.NewString()
+ ownedShards atomic.Int64
+ knownInstances atomic.Int64
+)
+
+type shardSet struct {
+ all bool
+ owned map[int]bool
+}
+
+func (s shardSet) owns(sloId int) bool {
+ return s.all || s.owned[sloId%shardCount]
+}
+
+func (s shardSet) size() int {
+ if s.all {
+ return shardCount
+ }
+ return len(s.owned)
+}
+
+func leaseDuration() time.Duration {
+ if d := 3 * pollInterval(); d > 5*time.Minute {
+ return d
+ }
+ return 5 * time.Minute
+}
+
+func claimShards(ctx context.Context) (shardSet, error) {
+ if db.IsSQLite() {
+ ownedShards.Store(shardCount)
+ knownInstances.Store(1)
+ return shardSet{all: true}, nil
+ }
+ lease := leaseDuration().Seconds()
+ if _, err := db.DB.ExecContext(ctx, "INSERT INTO slo_workers (owner, seen_at) VALUES ($1, now()) ON CONFLICT (owner) DO UPDATE SET seen_at = now()", workerId); err != nil {
+ return shardSet{}, fmt.Errorf("register slo worker: %w", err)
+ }
+ if _, err := db.DB.ExecContext(ctx, "DELETE FROM slo_workers WHERE seen_at < now() - make_interval(secs => $1)", 2*lease); err != nil {
+ return shardSet{}, fmt.Errorf("prune slo workers: %w", err)
+ }
+ var instances int
+ if err := db.DB.QueryRowContext(ctx, "SELECT COUNT(*) FROM slo_workers WHERE seen_at > now() - make_interval(secs => $1)", lease).Scan(&instances); err != nil {
+ return shardSet{}, fmt.Errorf("count slo workers: %w", err)
+ }
+ if instances < 1 {
+ instances = 1
+ }
+ owned, err := queryShards(ctx, "UPDATE slo_shard_leases SET expires_at = now() + make_interval(secs => $2) WHERE owner = $1 RETURNING shard", workerId, lease)
+ if err != nil {
+ return shardSet{}, fmt.Errorf("renew slo shard leases: %w", err)
+ }
+ target := (shardCount + instances - 1) / instances
+ for shard := range owned {
+ if len(owned) <= target {
+ break
+ }
+ if _, err := db.DB.ExecContext(ctx, "DELETE FROM slo_shard_leases WHERE shard = $1 AND owner = $2", shard, workerId); err != nil {
+ return shardSet{}, fmt.Errorf("release slo shard %d: %w", shard, err)
+ }
+ delete(owned, shard)
+ }
+ if missing := target - len(owned); missing > 0 {
+ claimed, err := queryShards(ctx,
+ "INSERT INTO slo_shard_leases (shard, owner, expires_at) "+
+ "SELECT s, $1, now() + make_interval(secs => $2) FROM generate_series(0, $3 - 1) AS s "+
+ "WHERE NOT EXISTS (SELECT 1 FROM slo_shard_leases l WHERE l.shard = s AND l.expires_at > now()) "+
+ "ORDER BY random() LIMIT $4 "+
+ "ON CONFLICT (shard) DO UPDATE SET owner = EXCLUDED.owner, expires_at = EXCLUDED.expires_at WHERE slo_shard_leases.expires_at <= now() "+
+ "RETURNING shard",
+ workerId, lease, shardCount, missing)
+ if err != nil {
+ return shardSet{}, fmt.Errorf("claim slo shards: %w", err)
+ }
+ for shard := range claimed {
+ owned[shard] = true
+ }
+ }
+ ownedShards.Store(int64(len(owned)))
+ knownInstances.Store(int64(instances))
+ return shardSet{owned: owned}, nil
+}
+
+func renewShards(ctx context.Context) error {
+ if db.IsSQLite() {
+ return nil
+ }
+ _, err := db.DB.ExecContext(ctx, "UPDATE slo_shard_leases SET expires_at = now() + make_interval(secs => $2) WHERE owner = $1", workerId, leaseDuration().Seconds())
+ return err
+}
+
+func releaseShards() {
+ if db.IsSQLite() {
+ return
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
+ _, _ = db.DB.ExecContext(ctx, "DELETE FROM slo_shard_leases WHERE owner = $1", workerId)
+ _, _ = db.DB.ExecContext(ctx, "DELETE FROM slo_workers WHERE owner = $1", workerId)
+ ownedShards.Store(0)
+}
+
+func queryShards(ctx context.Context, query string, args ...any) (map[int]bool, error) {
+ rows, err := db.DB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ shards := map[int]bool{}
+ for rows.Next() {
+ var shard int
+ if err := rows.Scan(&shard); err != nil {
+ return nil, err
+ }
+ shards[shard] = true
+ }
+ return shards, rows.Err()
+}
diff --git a/backend/app/slo/shards_pg_test.go b/backend/app/slo/shards_pg_test.go
new file mode 100644
index 00000000..527b626c
--- /dev/null
+++ b/backend/app/slo/shards_pg_test.go
@@ -0,0 +1,111 @@
+//go:build transactional_pg
+
+package slo
+
+import (
+ "context"
+ "database/sql"
+ "fmt"
+ "os"
+ "strings"
+ "testing"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+ "github.com/tracewayapp/traceway/backend/app/config"
+ "github.com/tracewayapp/traceway/backend/app/db"
+)
+
+func setupShardDB(t *testing.T) {
+ t.Helper()
+ password := os.Getenv("TEST_POSTGRES_PASSWORD")
+ if password == "" {
+ t.Skip("TEST_POSTGRES_PASSWORD is not set")
+ }
+ host := os.Getenv("TEST_POSTGRES_HOST")
+ if host == "" {
+ host = "localhost"
+ }
+ base := fmt.Sprintf("host=%s user=%s password=%s dbname=%s sslmode=disable", host, os.Getenv("TEST_POSTGRES_USERNAME"), password, os.Getenv("TEST_POSTGRES_DATABASE"))
+ admin, err := sql.Open("postgres", base)
+ if err != nil {
+ t.Fatal(err)
+ }
+ schema := "slo_shards_" + strings.ReplaceAll(uuid.NewString(), "-", "")
+ if _, err := admin.Exec("CREATE SCHEMA " + schema); err != nil {
+ t.Fatal(err)
+ }
+ conn, err := sql.Open("postgres", base+" search_path="+schema)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, file := range []string{"0157_create_slo_workers.up.sql", "0158_create_slo_shard_leases.up.sql"} {
+ migration, err := os.ReadFile("../migrations/pg/" + file)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := conn.Exec(string(migration)); err != nil {
+ t.Fatal(err)
+ }
+ }
+ previousDB, previousDriver, previousConfig, previousWorker := db.DB, db.Driver, config.Config, workerId
+ db.DB, db.Driver = conn, lit.PostgreSQL
+ config.Init(&config.Cfg{})
+ t.Cleanup(func() {
+ conn.Close()
+ _, _ = admin.Exec("DROP SCHEMA " + schema + " CASCADE")
+ admin.Close()
+ db.DB, db.Driver, config.Config, workerId = previousDB, previousDriver, previousConfig, previousWorker
+ })
+}
+
+func claimAs(t *testing.T, worker string) shardSet {
+ t.Helper()
+ workerId = worker
+ shards, err := claimShards(context.Background())
+ if err != nil {
+ t.Fatalf("claim as %s: %v", worker, err)
+ }
+ return shards
+}
+
+func TestShardsSplitBetweenInstancesAndMoveWhenOneStops(t *testing.T) {
+ setupShardDB(t)
+ if got := claimAs(t, "a").size(); got != shardCount {
+ t.Fatalf("a single instance owns every shard, got %d", got)
+ }
+ if got := claimAs(t, "b").size(); got != 0 {
+ t.Fatalf("a new instance waits for shards to be released, got %d", got)
+ }
+ a := claimAs(t, "a")
+ b := claimAs(t, "b")
+ if a.size() != shardCount/2 || b.size() != shardCount/2 {
+ t.Fatalf("two instances split the shards: a=%d b=%d", a.size(), b.size())
+ }
+ for shard := range a.owned {
+ if b.owned[shard] {
+ t.Fatalf("shard %d is leased twice", shard)
+ }
+ }
+ for id := 1; id <= 200; id++ {
+ if a.owns(id) == b.owns(id) {
+ t.Fatalf("SLO %d must belong to exactly one instance", id)
+ }
+ }
+
+ if _, err := db.DB.Exec("UPDATE slo_workers SET seen_at = now() - interval '1 hour' WHERE owner = 'b'"); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := db.DB.Exec("UPDATE slo_shard_leases SET expires_at = now() - interval '1 second' WHERE owner = 'b'"); err != nil {
+ t.Fatal(err)
+ }
+ if got := claimAs(t, "a").size(); got != shardCount {
+ t.Fatalf("the remaining instance takes over the stopped one's shards, got %d", got)
+ }
+
+ workerId = "a"
+ releaseShards()
+ if got := claimAs(t, "c").size(); got != shardCount {
+ t.Fatalf("released shards are free at once, got %d", got)
+ }
+}
diff --git a/backend/app/slo/slo_sqlite_test.go b/backend/app/slo/slo_sqlite_test.go
new file mode 100644
index 00000000..488dcdb8
--- /dev/null
+++ b/backend/app/slo/slo_sqlite_test.go
@@ -0,0 +1,377 @@
+//go:build !transactional_pg && !telemetry_ch && !telemetry_duckdb
+
+package slo
+
+import (
+ "context"
+ "database/sql"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+func setupSLODB(t *testing.T) uuid.UUID {
+ t.Helper()
+ dbtest.SetupSQLite(t)
+ windowCaches = map[int]*windowCache{}
+ project, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.Project, error) {
+ return transactional.ProjectRepository.Create(tx, "slo-test", "gin")
+ })
+ if err != nil {
+ t.Fatalf("create project: %v", err)
+ }
+ return project.Id
+}
+
+func createTestSLO(t *testing.T, projectId uuid.UUID, kind, config string, target float64, windowDays, bucketMinutes int) *models.SLO {
+ t.Helper()
+ now := time.Now().UTC()
+ entity := &models.SLO{
+ ProjectId: projectId, Name: "test", SliKind: kind, SliConfig: models.JSONText(config),
+ TargetPercent: target, WindowDays: windowDays, BucketMinutes: bucketMinutes, Enabled: true,
+ Status: models.SLOStatusNoData, CreatedAt: now, UpdatedAt: now,
+ }
+ id, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ return transactional.SLORepository.Create(tx, entity)
+ })
+ if err != nil {
+ t.Fatalf("create slo: %v", err)
+ }
+ entity.Id = id
+ return entity
+}
+
+func reloadSLO(t *testing.T, id int) *models.SLO {
+ t.Helper()
+ entity, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.SLO, error) {
+ return transactional.SLORepository.FindById(tx, id)
+ })
+ if err != nil || entity == nil {
+ t.Fatalf("reload slo %d: %v", id, err)
+ }
+ return entity
+}
+
+func endpointAt(projectId uuid.UUID, status int16, at time.Time) models.Endpoint {
+ return models.Endpoint{Id: uuid.New(), ProjectId: projectId, Endpoint: "GET /users", Duration: 50 * time.Millisecond, RecordedAt: at, StatusCode: status, ClientIP: "127.0.0.1"}
+}
+
+func TestWorkerComputesEndpointAvailability(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Minute).Add(30 * time.Second)
+ var endpoints []models.Endpoint
+ for minute := 1; minute <= 20; minute++ {
+ at := now.Add(-time.Duration(minute) * time.Minute)
+ for i := 0; i < 9; i++ {
+ endpoints = append(endpoints, endpointAt(projectId, 200, at.Add(time.Duration(i)*time.Second)))
+ }
+ endpoints = append(endpoints, endpointAt(projectId, 500, at.Add(9*time.Second)))
+ }
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, endpoints); err != nil {
+ t.Fatal(err)
+ }
+ entity := createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"","good":"availability"}`, 99, 7, 1)
+
+ RunOnce(ctx, now)
+
+ got := reloadSLO(t, entity.Id)
+ if got.Status != models.SLOStatusBreached {
+ t.Fatalf("90%% availability against a 99%% target is breached, got %q", got.Status)
+ }
+ if got.CurrentSli == nil || *got.CurrentSli < 89.9 || *got.CurrentSli > 90.1 {
+ t.Fatalf("current sli: %v", got.CurrentSli)
+ }
+ if got.BudgetRemaining == nil || *got.BudgetRemaining > -800 {
+ t.Fatalf("budget is overspent tenfold: %v", got.BudgetRemaining)
+ }
+ lastClosed := now.Truncate(time.Minute)
+ if got.LastBucketAt == nil || !got.LastBucketAt.Equal(lastClosed) {
+ t.Fatalf("last bucket = %v, want %v", got.LastBucketAt, lastClosed)
+ }
+ windowStart := lastClosed.Add(-7 * 24 * time.Hour)
+ if got.HistoryFrom == nil || got.HistoryFrom.After(windowStart) {
+ t.Fatalf("backfill of an empty week should finish in one tick: history from %v", got.HistoryFrom)
+ }
+ good, total, err := telemetry.SLOBucketRepository.SumRange(ctx, projectId, entity.Id, windowStart, lastClosed)
+ if err != nil || good != 180 || total != 200 {
+ t.Fatalf("window sums = %d/%d (%v), want 180/200", good, total, err)
+ }
+
+ late := endpointAt(projectId, 200, now.Add(-3*time.Minute).Add(20*time.Second))
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{late}); err != nil {
+ t.Fatal(err)
+ }
+ RunOnce(ctx, now.Add(time.Minute))
+ RunOnce(ctx, now.Add(2*time.Minute))
+ good, total, err = telemetry.SLOBucketRepository.SumRange(ctx, projectId, entity.Id, windowStart, lastClosed.Add(2*time.Minute))
+ if err != nil || good != 181 || total != 201 {
+ t.Fatalf("window sums after late data = %d/%d (%v), want 181/201", good, total, err)
+ }
+}
+
+func TestWorkerMetricTimeslices(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(5 * time.Minute).Add(time.Minute)
+ lastClosed := now.Truncate(5 * time.Minute)
+ points := []models.MetricPoint{
+ {ProjectId: projectId, Name: "cpu.used_pcnt", Value: 40, Tags: map[string]string{}, RecordedAt: lastClosed.Add(-5 * time.Minute).Add(10 * time.Second)},
+ {ProjectId: projectId, Name: "cpu.used_pcnt", Value: 95, Tags: map[string]string{}, RecordedAt: lastClosed.Add(-10 * time.Minute).Add(10 * time.Second)},
+ {ProjectId: projectId, Name: "cpu.used_pcnt", Value: 10, Tags: map[string]string{}, RecordedAt: lastClosed.Add(-15 * time.Minute).Add(10 * time.Second)},
+ }
+ if err := telemetry.MetricPointRepository.InsertAsync(ctx, points); err != nil {
+ t.Fatal(err)
+ }
+ strict := createTestSLO(t, projectId, models.SLIKindMetric, `{"metricName":"cpu.used_pcnt","tags":{},"aggregation":"avg","operator":"lt","threshold":90,"missingIsBad":true}`, 99, 7, 5)
+ lenient := createTestSLO(t, projectId, models.SLIKindMetric, `{"metricName":"cpu.used_pcnt","tags":{},"aggregation":"avg","operator":"lt","threshold":90,"missingIsBad":false}`, 50, 7, 5)
+ since := lastClosed.Add(-30 * time.Minute)
+ strict.SliSince = &since
+ setSLISince(t, strict.Id, since)
+
+ RunOnce(ctx, now)
+
+ windowStart := lastClosed.Add(-7 * 24 * time.Hour)
+ good, total, err := telemetry.SLOBucketRepository.SumRange(ctx, projectId, strict.Id, windowStart, lastClosed)
+ if err != nil || total != 6 || good != 2 {
+ t.Fatalf("empty slices are bad only since the definition took effect: %d/%d (%v), want 2/6", good, total, err)
+ }
+ good, total, err = telemetry.SLOBucketRepository.SumRange(ctx, projectId, lenient.Id, windowStart, lastClosed)
+ if err != nil || total != 3 || good != 2 {
+ t.Fatalf("empty slices do not count: %d/%d (%v), want 2/3", good, total, err)
+ }
+ if got := reloadSLO(t, lenient.Id); got.Status != models.SLOStatusOk {
+ t.Fatalf("2 of 3 slices good against a 50%% target is ok, got %q", got.Status)
+ }
+}
+
+func TestWorkerMonitorUptime(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Minute).Add(30 * time.Second)
+ at := now.Add(-5 * time.Minute)
+ for i, status := range []string{"up", "up", "down", "missed", "up"} {
+ result := shared.CheckResult{ProjectId: projectId, CheckId: 3, RunId: i + 1, CheckType: "http", RecordedAt: at.Add(time.Duration(i) * 10 * time.Second), ExecutedAt: at, Status: status, LatencyMs: 12}
+ if err := telemetry.CheckResultRepository.Insert(ctx, result); err != nil {
+ t.Fatal(err)
+ }
+ }
+ foreign := shared.CheckResult{ProjectId: projectId, CheckId: 4, RunId: 9, CheckType: "http", RecordedAt: at, ExecutedAt: at, Status: "down"}
+ if err := telemetry.CheckResultRepository.Insert(ctx, foreign); err != nil {
+ t.Fatal(err)
+ }
+ entity := createTestSLO(t, projectId, models.SLIKindMonitor, `{"checkIds":[3]}`, 90, 7, 1)
+
+ RunOnce(ctx, now)
+
+ got := reloadSLO(t, entity.Id)
+ if got.CurrentSli == nil || *got.CurrentSli != 75 {
+ t.Fatalf("3 up of 4 measured probes is 75%%: %v", got.CurrentSli)
+ }
+ if got.Status != models.SLOStatusBreached {
+ t.Fatalf("75%% against 90%% is breached: %q", got.Status)
+ }
+}
+
+func setSLISince(t *testing.T, id int, since time.Time) {
+ t.Helper()
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ _, err := tx.Exec("UPDATE slos SET sli_since = ? WHERE id = ?", since.UTC(), id)
+ return struct{}{}, err
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestWorkerDiscardsBucketsOfAnEditDuringTheTick(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Minute).Add(30 * time.Second)
+ var endpoints []models.Endpoint
+ for minute := 1; minute <= 120; minute++ {
+ at := now.Add(-time.Duration(minute) * time.Minute)
+ bad := endpointAt(projectId, 500, at)
+ bad.Endpoint = "GET /old"
+ good := endpointAt(projectId, 200, at)
+ good.Endpoint = "GET /new"
+ endpoints = append(endpoints, bad, good)
+ }
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, endpoints); err != nil {
+ t.Fatal(err)
+ }
+ entity := createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"GET /old","good":"availability"}`, 99, 7, 1)
+ loadedByTheTick := reloadSLO(t, entity.Id)
+
+ edited := reloadSLO(t, entity.Id)
+ edited.SliConfig = models.JSONText(`{"endpoint":"GET /new","good":"availability"}`)
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ if err := transactional.SLORepository.Update(tx, edited); err != nil {
+ return struct{}{}, err
+ }
+ return struct{}{}, transactional.SLORepository.ResetHistory(tx, edited.Id, now)
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := telemetry.SLOBucketRepository.DeleteBySLO(ctx, projectId, entity.Id); err != nil {
+ t.Fatal(err)
+ }
+
+ computeAll(ctx, []*models.SLO{loadedByTheTick}, now)
+ if got := reloadSLO(t, entity.Id); got.LastBucketAt != nil {
+ t.Fatalf("the stale tick must not write its snapshot over the reset: %v", got.LastBucketAt)
+ }
+ RunOnce(ctx, now.Add(time.Minute))
+
+ got := reloadSLO(t, entity.Id)
+ if got.CurrentSli == nil || *got.CurrentSli != 100 {
+ t.Fatalf("the new definition only sees 200s: %v", got.CurrentSli)
+ }
+}
+
+func TestWorkerKeepsBucketsWhenOnlyTheTargetChangedDuringTheTick(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Minute).Add(30 * time.Second)
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{endpointAt(projectId, 200, now.Add(-2*time.Minute))}); err != nil {
+ t.Fatal(err)
+ }
+ entity := createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"","good":"availability"}`, 99, 7, 1)
+ loadedByTheTick := reloadSLO(t, entity.Id)
+ edited := reloadSLO(t, entity.Id)
+ edited.TargetPercent = 95
+ if _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ return struct{}{}, transactional.SLORepository.Update(tx, edited)
+ }); err != nil {
+ t.Fatal(err)
+ }
+
+ computeAll(ctx, []*models.SLO{loadedByTheTick}, now)
+ _, total, err := telemetry.SLOBucketRepository.SumRange(ctx, projectId, entity.Id, now.Add(-time.Hour), now)
+ if err != nil || total != 1 {
+ t.Fatalf("buckets of an unchanged definition stay: %d (%v)", total, err)
+ }
+}
+
+func TestWorkerFillsTheGapOfAnOutage(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Minute).Add(30 * time.Second)
+ var endpoints []models.Endpoint
+ for hour := 1; hour <= 96; hour++ {
+ endpoints = append(endpoints, endpointAt(projectId, 500, now.Add(-time.Duration(hour)*time.Hour)))
+ }
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, endpoints); err != nil {
+ t.Fatal(err)
+ }
+ entity := createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"","good":"availability"}`, 99, 7, 1)
+ computeAll(ctx, []*models.SLO{entity}, now.Add(-72*time.Hour))
+
+ RunOnce(ctx, now)
+
+ got := reloadSLO(t, entity.Id)
+ if got.GapFrom != nil || got.GapTo != nil {
+ t.Fatalf("the gap is filled within one budget: %v %v", got.GapFrom, got.GapTo)
+ }
+ window, err := WindowSums(ctx, got, 7*24*time.Hour, now)
+ if err != nil || window.Total != 96 {
+ t.Fatalf("every request of the window counts after an outage: %d (%v)", window.Total, err)
+ }
+}
+
+func TestWorkerCountsASessionCrashThatArrivesLate(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Minute).Add(30 * time.Second)
+ start := now.Add(-2 * time.Minute)
+ sessions := make([]models.Session, 0, 10)
+ for i := 0; i < 10; i++ {
+ sessions = append(sessions, models.Session{Id: uuid.New(), ProjectId: projectId, StartedAt: start.Add(time.Duration(i) * time.Second), Attributes: map[string]string{}})
+ }
+ if err := telemetry.SessionRepository.Upsert(ctx, sessions); err != nil {
+ t.Fatal(err)
+ }
+ entity := createTestSLO(t, projectId, models.SLIKindSession, `{}`, 99, 7, 1)
+ for minute := 0; minute < 30; minute++ {
+ RunOnce(ctx, now.Add(time.Duration(minute)*time.Minute))
+ }
+ crash := models.ExceptionStackTrace{Id: uuid.New(), ProjectId: projectId, ExceptionHash: "crash", StackTrace: "boom", RecordedAt: now.Add(30 * time.Minute), SessionId: &sessions[0].Id}
+ if err := telemetry.ExceptionStackTraceRepository.InsertAsync(ctx, []models.ExceptionStackTrace{crash}); err != nil {
+ t.Fatal(err)
+ }
+ for minute := 30; minute < 30+int(lateArrivalInterval/time.Minute); minute++ {
+ RunOnce(ctx, now.Add(time.Duration(minute)*time.Minute))
+ }
+
+ got := reloadSLO(t, entity.Id)
+ if got.CurrentSli == nil || *got.CurrentSli != 90 {
+ t.Fatalf("a crash 30 minutes into a session must still count: %v", derefSLI(got.CurrentSli))
+ }
+}
+
+func TestWorkerCountsAMobileCrashUploadedTheNextDay(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Minute).Add(-48 * time.Hour).Add(30 * time.Second)
+ start := now.Add(-2 * time.Minute)
+ sessions := make([]models.Session, 0, 10)
+ for i := 0; i < 10; i++ {
+ sessions = append(sessions, models.Session{Id: uuid.New(), ProjectId: projectId, StartedAt: start.Add(time.Duration(i) * time.Second), Attributes: map[string]string{}})
+ }
+ if err := telemetry.SessionRepository.Upsert(ctx, sessions); err != nil {
+ t.Fatal(err)
+ }
+ entity := createTestSLO(t, projectId, models.SLIKindSession, `{}`, 99, 7, 1)
+ RunOnce(ctx, now)
+ RunOnce(ctx, now.Add(time.Hour))
+
+ crash := models.ExceptionStackTrace{Id: uuid.New(), ProjectId: projectId, ExceptionHash: "crash", StackTrace: "SIGSEGV", RecordedAt: start.Add(40 * time.Second), SessionId: &sessions[0].Id}
+ if err := telemetry.ExceptionStackTraceRepository.InsertAsync(ctx, []models.ExceptionStackTrace{crash}); err != nil {
+ t.Fatal(err)
+ }
+ RunOnce(ctx, now.Add(25*time.Hour))
+
+ got := reloadSLO(t, entity.Id)
+ if got.CurrentSli == nil || *got.CurrentSli != 90 {
+ t.Fatalf("a crash uploaded on the next launch must still count: %v", derefSLI(got.CurrentSli))
+ }
+}
+
+func TestWorkerWritesEveryBucketOfATickInOneInsert(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Minute).Add(30 * time.Second)
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{endpointAt(projectId, 200, now.Add(-2*time.Minute)), endpointAt(projectId, 500, now.Add(-2*time.Minute))}); err != nil {
+ t.Fatal(err)
+ }
+ first := createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"","good":"availability"}`, 99, 7, 1)
+ second := createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"GET /users","good":"availability"}`, 40, 7, 1)
+ results := []*tickResult{
+ computeBuckets(ctx, first, now, time.Now().Add(backfillBudget), lateDataWindow),
+ computeBuckets(ctx, second, now, time.Now().Add(backfillBudget), lateDataWindow),
+ }
+ if err := storeBuckets(ctx, results); err != nil {
+ t.Fatal(err)
+ }
+ for _, entity := range []*models.SLO{first, second} {
+ good, total, err := telemetry.SLOBucketRepository.SumRange(ctx, projectId, entity.Id, now.Add(-time.Hour), now)
+ if err != nil || good != 1 || total != 2 {
+ t.Fatalf("SLO %d: %d/%d (%v), want 1/2", entity.Id, good, total, err)
+ }
+ }
+}
+
+func derefSLI(sli *float64) any {
+ if sli == nil {
+ return nil
+ }
+ return *sli
+}
diff --git a/backend/app/slo/snapshot_cache.go b/backend/app/slo/snapshot_cache.go
new file mode 100644
index 00000000..6adc2524
--- /dev/null
+++ b/backend/app/slo/snapshot_cache.go
@@ -0,0 +1,123 @@
+package slo
+
+import (
+ "context"
+ "sync"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+)
+
+const snapshotRefreshInterval = time.Hour
+
+// The window sum is split at an anchor: the stable part before it is kept in
+// memory and only loses the buckets that slide out of the window, the recent
+// part after it, where recomputes and late data land, is read every tick.
+// Any write before the anchor, an edit, or a snapshot written by another
+// instance forces a full read, and every SLO is read in full once an hour.
+type windowCache struct {
+ revision int
+ computedAt time.Time
+ anchor time.Time
+ windowStart time.Time
+ stable Window
+}
+
+var (
+ windowCacheMu sync.Mutex
+ windowCaches = map[int]*windowCache{}
+)
+
+func recentSpan(kind string) time.Duration {
+ if kind == models.SLIKindSession {
+ return lateCrashHorizon + time.Hour
+ }
+ return 7 * time.Hour
+}
+
+type windowSums struct {
+ window, lastHour, lastSixHours Window
+ cache *windowCache
+ full bool
+}
+
+func readWindowSums(ctx context.Context, result *tickResult, now time.Time) (windowSums, error) {
+ slo := result.slo
+ lastClosed := result.lastClosed
+ sixFrom, hourFrom := lastClosed.Add(-6*time.Hour), lastClosed.Add(-time.Hour)
+
+ windowCacheMu.Lock()
+ cached := windowCaches[slo.Id]
+ windowCacheMu.Unlock()
+
+ full := cached == nil ||
+ cached.revision != slo.Revision ||
+ slo.ComputedAt == nil ||
+ !sameInstant(cached.computedAt, *slo.ComputedAt) ||
+ result.windowStart.Before(cached.windowStart) ||
+ (result.earliestWrite != nil && result.earliestWrite.Before(cached.anchor)) ||
+ sixFrom.Before(cached.anchor) ||
+ lastClosed.Sub(cached.anchor) > recentSpan(slo.SliKind)+2*snapshotRefreshInterval ||
+ passDue(slo.Id, snapshotRefreshInterval, slo.ComputedAt, now)
+
+ next := &windowCache{revision: slo.Revision, computedAt: now, windowStart: result.windowStart}
+ if full {
+ next.anchor = lastClosed.Add(-recentSpan(slo.SliKind)).Truncate(slo.BucketDuration())
+ if next.anchor.Before(result.windowStart) {
+ next.anchor = result.windowStart
+ }
+ stable, err := sumRange(ctx, slo, result.windowStart, next.anchor)
+ if err != nil {
+ return windowSums{}, err
+ }
+ next.stable = stable
+ } else {
+ next.anchor = cached.anchor
+ next.stable = cached.stable
+ if result.windowStart.After(cached.windowStart) {
+ dropped, err := sumRange(ctx, slo, cached.windowStart, result.windowStart)
+ if err != nil {
+ return windowSums{}, err
+ }
+ next.stable.Good -= dropped.Good
+ next.stable.Total -= dropped.Total
+ }
+ }
+ recent, err := telemetry.SLOBucketRepository.SnapshotSums(ctx, slo.ProjectId, slo.Id, next.anchor, lastClosed, sixFrom, hourFrom)
+ if err != nil {
+ return windowSums{}, err
+ }
+ return windowSums{
+ window: Window{Good: next.stable.Good + recent.Window.Good, Total: next.stable.Total + recent.Window.Total},
+ lastHour: window(recent.LastHour),
+ lastSixHours: window(recent.LastSixHours),
+ cache: next,
+ full: full,
+ }, nil
+}
+
+func storeWindowCache(sloId int, cache *windowCache) {
+ windowCacheMu.Lock()
+ defer windowCacheMu.Unlock()
+ if cache == nil {
+ delete(windowCaches, sloId)
+ return
+ }
+ windowCaches[sloId] = cache
+}
+
+func pruneWindowCaches(keep map[int]bool) {
+ windowCacheMu.Lock()
+ defer windowCacheMu.Unlock()
+ for id := range windowCaches {
+ if !keep[id] {
+ delete(windowCaches, id)
+ }
+ }
+}
+
+func sameInstant(a, b time.Time) bool {
+ d := a.Sub(b)
+ return d > -time.Millisecond && d < time.Millisecond
+}
diff --git a/backend/app/slo/worker.go b/backend/app/slo/worker.go
new file mode 100644
index 00000000..7fcb6aa6
--- /dev/null
+++ b/backend/app/slo/worker.go
@@ -0,0 +1,464 @@
+package slo
+
+import (
+ "context"
+ "database/sql"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+ "sync"
+ "sync/atomic"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/config"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ traceway "go.tracewayapp.com"
+)
+
+const (
+ lateDataWindow = 10 * time.Minute
+
+ lateArrivalInterval = 15 * time.Minute
+ lateExportSlack = time.Hour
+ lateCrashInterval = 24 * time.Hour
+ lateCrashHorizon = 72 * time.Hour
+
+ maxCatchUp = 24 * time.Hour
+
+ backfillChunk = 6 * time.Hour
+ metricBackfillChunk = 24 * time.Hour
+ backfillBudget = 5 * time.Second
+ maxBackfillChunks = 48
+
+ defaultComputeWorkers = 4
+ maxComputeWorkers = 64
+ defaultComputeTimeout = 30
+ computeGroupSize = 200
+ insertBatchRows = 50_000
+
+ bucketRetention = 120 * 24 * time.Hour
+ pruneInterval = 24 * time.Hour
+)
+
+var (
+ wakeCh = make(chan struct{}, 1)
+ lastPruneAt time.Time
+ tickCount int
+)
+
+func Wake() {
+ select {
+ case wakeCh <- struct{}{}:
+ default:
+ }
+}
+
+func pollInterval() time.Duration {
+ return config.PollSeconds(config.Config.SLOPollSeconds, 60)
+}
+
+func computeWorkers() int {
+ return boundedSetting(config.Config.SLOComputeWorkers, defaultComputeWorkers, maxComputeWorkers)
+}
+
+func computeTimeout() time.Duration {
+ return time.Duration(boundedSetting(config.Config.SLOComputeTimeoutSecs, defaultComputeTimeout, 3600)) * time.Second
+}
+
+func boundedSetting(value string, fallback, max int) int {
+ parsed, err := strconv.Atoi(strings.TrimSpace(value))
+ if err != nil || parsed < 1 {
+ return fallback
+ }
+ if parsed > max {
+ return max
+ }
+ return parsed
+}
+
+func Start(ctx context.Context) {
+ go func() {
+ defer traceway.Recover()
+ defer releaseShards()
+ ticker := time.NewTicker(pollInterval())
+ defer ticker.Stop()
+ RunOnce(ctx, time.Now().UTC())
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-ticker.C:
+ case <-wakeCh:
+ }
+ RunOnce(ctx, time.Now().UTC())
+ }
+ }()
+}
+
+func RunOnce(ctx context.Context, now time.Time) {
+ shards, err := claimShards(ctx)
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to lease SLO shards: %w", err))
+ return
+ }
+ if shards.size() == 0 {
+ pruneWindowCaches(nil)
+ return
+ }
+ started := time.Now()
+ tickStarted.Store(started.UnixNano())
+ defer tickStarted.Store(0)
+ enabled, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.SLO, error) {
+ return transactional.SLORepository.FindEnabled(tx)
+ })
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to load enabled SLOs: %w", err))
+ return
+ }
+ slos := make([]*models.SLO, 0, len(enabled))
+ keep := make(map[int]bool, len(enabled))
+ for _, slo := range enabled {
+ if shards.owns(slo.Id) {
+ slos = append(slos, slo)
+ keep[slo.Id] = true
+ }
+ }
+ pruneWindowCaches(keep)
+ if len(slos) > 0 {
+ offset := tickCount % len(slos)
+ slos = append(slos[offset:], slos[:offset]...)
+ }
+ tickCount++
+ stats := computeAll(ctx, slos, now)
+ stats.At = now
+ stats.DurationMs = millisSince(started)
+ recordTick(stats)
+ lastTickUnix.Store(now.Unix())
+ if shards.owns(0) && now.Sub(lastPruneAt) >= pruneInterval {
+ if err := telemetry.SLOBucketRepository.PruneBefore(ctx, now.Add(-bucketRetention)); err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to prune slo buckets: %w", err))
+ } else {
+ lastPruneAt = now
+ }
+ }
+}
+
+type tickResult struct {
+ slo *models.SLO
+ buckets []shared.SLOBucketPoint
+ lastClosed time.Time
+ windowStart time.Time
+ historyFrom time.Time
+ gapFrom *time.Time
+ gapTo *time.Time
+ earliestWrite *time.Time
+ backfillErr error
+ err error
+ elapsed time.Duration
+ timedOut bool
+ fullSnapshot bool
+}
+
+func computeAll(ctx context.Context, slos []*models.SLO, now time.Time) *TickStats {
+ stats := &TickStats{SLOs: len(slos)}
+ deadline := time.Now().Add(backfillBudget)
+ workers, timeout := computeWorkers(), computeTimeout()
+ for start := 0; start < len(slos); start += computeGroupSize {
+ end := min(start+computeGroupSize, len(slos))
+ computeGroup(ctx, slos[start:end], now, deadline, workers, timeout, stats)
+ if end < len(slos) {
+ if err := renewShards(ctx); err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to renew SLO shard leases: %w", err))
+ }
+ }
+ }
+ return stats
+}
+
+func computeGroup(ctx context.Context, group []*models.SLO, now, deadline time.Time, workers int, timeout time.Duration, stats *TickStats) {
+ results := make([]*tickResult, len(group))
+ phase := time.Now()
+ forEach(len(group), workers, func(i int) {
+ began := time.Now()
+ sloCtx, cancel := context.WithTimeout(ctx, timeout)
+ defer cancel()
+ result := computeBuckets(sloCtx, group[i], now, deadline, recomputeHorizon(group[i], now))
+ result.timedOut = timedOut(sloCtx, result.err) || timedOut(sloCtx, result.backfillErr)
+ result.elapsed = time.Since(began)
+ results[i] = result
+ })
+ stats.ComputeMs += millisSince(phase)
+
+ phase = time.Now()
+ if err := storeBuckets(ctx, results); err != nil {
+ for _, result := range results {
+ if result != nil && result.err == nil {
+ result.err = err
+ }
+ }
+ }
+ stats.StoreMs += millisSince(phase)
+
+ phase = time.Now()
+ forEach(len(results), workers, func(i int) {
+ result := results[i]
+ if result == nil || result.err != nil {
+ return
+ }
+ began := time.Now()
+ sloCtx, cancel := context.WithTimeout(ctx, timeout)
+ defer cancel()
+ result.err = writeSnapshot(sloCtx, result, now)
+ result.timedOut = result.timedOut || timedOut(sloCtx, result.err)
+ result.elapsed += time.Since(began)
+ })
+ stats.SnapshotMs += millisSince(phase)
+
+ for i, result := range results {
+ if result == nil {
+ atomic.AddUint64(&errorsTotal, 1)
+ stats.Errors++
+ traceway.CaptureException(fmt.Errorf("failed to compute SLO %d (%s): panicked", group[i].Id, group[i].SliKind))
+ continue
+ }
+ stats.Buckets += len(result.buckets)
+ if result.fullSnapshot {
+ stats.FullSnapshots++
+ }
+ if stats.Slowest == nil || durationMs(result.elapsed) > stats.Slowest.DurationMs {
+ stats.Slowest = &SlowestSLO{Id: result.slo.Id, ProjectId: result.slo.ProjectId, Kind: result.slo.SliKind, DurationMs: durationMs(result.elapsed)}
+ }
+ if result.timedOut {
+ atomic.AddUint64(&timeoutsTotal, 1)
+ stats.Timeouts++
+ }
+ if result.err != nil {
+ atomic.AddUint64(&errorsTotal, 1)
+ stats.Errors++
+ if result.timedOut {
+ traceway.CaptureException(fmt.Errorf("SLO %d (%s) of project %s timed out after %s: %w", result.slo.Id, result.slo.SliKind, result.slo.ProjectId, timeout, result.err))
+ } else {
+ traceway.CaptureException(fmt.Errorf("failed to compute SLO %d (%s): %w", result.slo.Id, result.slo.SliKind, result.err))
+ }
+ continue
+ }
+ atomic.AddUint64(&computedTotal, 1)
+ }
+}
+
+func timedOut(ctx context.Context, err error) bool {
+ return err != nil && (errors.Is(err, context.DeadlineExceeded) || errors.Is(ctx.Err(), context.DeadlineExceeded))
+}
+
+func durationMs(d time.Duration) float64 {
+ return float64(d) / float64(time.Millisecond)
+}
+
+func millisSince(t time.Time) float64 {
+ return durationMs(time.Since(t))
+}
+
+func forEach(n, workers int, fn func(i int)) {
+ var wg sync.WaitGroup
+ slots := make(chan struct{}, workers)
+ for i := 0; i < n; i++ {
+ wg.Add(1)
+ slots <- struct{}{}
+ go func(i int) {
+ defer wg.Done()
+ defer func() { <-slots }()
+ defer traceway.Recover()
+ fn(i)
+ }(i)
+ }
+ wg.Wait()
+}
+
+func lateArrivalHorizon(kind string) time.Duration {
+ switch kind {
+ case models.SLIKindSession:
+ return shared.SessionExceptionSlack + lateArrivalInterval + lateDataWindow
+ case models.SLIKindTask:
+ return shared.TaskExceptionSlack + lateArrivalInterval + lateDataWindow
+ }
+ return lateExportSlack + lateArrivalInterval
+}
+
+func computeBuckets(ctx context.Context, slo *models.SLO, now, deadline time.Time, horizon time.Duration) *tickResult {
+ result := &tickResult{slo: slo}
+ if slo.LastBucketAt == nil {
+ if err := telemetry.SLOBucketRepository.DeleteBySLO(ctx, slo.ProjectId, slo.Id); err != nil {
+ result.err = err
+ return result
+ }
+ }
+ collect := func(from, to time.Time) error {
+ points, err := ComputeBuckets(ctx, slo, from, to)
+ result.buckets = append(result.buckets, points...)
+ return err
+ }
+ bucket := slo.BucketDuration()
+ lastClosed := now.Truncate(bucket)
+ windowStart := lastClosed.Add(-slo.Window())
+ chunk := backfillChunk
+ if slo.SliKind == models.SLIKindMetric {
+ chunk = metricBackfillChunk
+ }
+
+ from := lastClosed.Add(-maxCatchUp).Truncate(bucket)
+ gapFrom, gapTo := copyTime(slo.GapFrom), copyTime(slo.GapTo)
+ if slo.LastBucketAt != nil {
+ resume := slo.LastBucketAt.UTC().Add(-horizon).Truncate(bucket)
+ switch {
+ case resume.After(from):
+ from = resume
+ case resume.Before(from):
+ if gapFrom == nil || resume.Before(*gapFrom) {
+ gapFrom = &resume
+ }
+ gapTo = copyTime(&from)
+ }
+ }
+ for chunkFrom := from; chunkFrom.Before(lastClosed); chunkFrom = chunkFrom.Add(backfillChunk) {
+ chunkTo := chunkFrom.Add(backfillChunk)
+ if chunkTo.After(lastClosed) {
+ chunkTo = lastClosed
+ }
+ if err := collect(chunkFrom, chunkTo); err != nil {
+ result.err = err
+ return result
+ }
+ }
+
+ if gapFrom != nil && gapFrom.Before(windowStart) {
+ gapFrom = copyTime(&windowStart)
+ }
+ for chunks := 0; gapFrom != nil && gapTo != nil && gapTo.After(*gapFrom) && chunks < maxBackfillChunks && time.Now().Before(deadline); chunks++ {
+ chunkFrom := gapTo.Add(-chunk)
+ if chunkFrom.Before(*gapFrom) {
+ chunkFrom = *gapFrom
+ }
+ if result.backfillErr = collect(chunkFrom, *gapTo); result.backfillErr != nil {
+ break
+ }
+ gapTo = &chunkFrom
+ }
+ if gapFrom == nil || gapTo == nil || !gapTo.After(*gapFrom) {
+ gapFrom, gapTo = nil, nil
+ }
+
+ historyFrom := from
+ if slo.HistoryFrom != nil && slo.HistoryFrom.Before(historyFrom) {
+ historyFrom = slo.HistoryFrom.UTC()
+ }
+ for chunks := 0; result.backfillErr == nil && historyFrom.After(windowStart) && chunks < maxBackfillChunks && time.Now().Before(deadline); chunks++ {
+ chunkFrom := historyFrom.Add(-chunk)
+ if chunkFrom.Before(windowStart) {
+ chunkFrom = windowStart
+ }
+ if result.backfillErr = collect(chunkFrom, historyFrom); result.backfillErr != nil {
+ break
+ }
+ historyFrom = chunkFrom
+ }
+
+ result.lastClosed, result.windowStart, result.historyFrom = lastClosed, windowStart, historyFrom
+ result.gapFrom, result.gapTo = gapFrom, gapTo
+ return result
+}
+
+func storeBuckets(ctx context.Context, results []*tickResult) error {
+ batch := make([]shared.SLOBucket, 0, insertBatchRows)
+ flush := func() error {
+ if len(batch) == 0 {
+ return nil
+ }
+ stamp := time.Now().UTC()
+ for i := range batch {
+ batch[i].ComputedAt = stamp
+ }
+ err := telemetry.SLOBucketRepository.Upsert(ctx, batch)
+ batch = batch[:0]
+ return err
+ }
+ for _, result := range results {
+ if result == nil || result.err != nil {
+ continue
+ }
+ for _, point := range result.buckets {
+ if result.earliestWrite == nil || point.Bucket.Before(*result.earliestWrite) {
+ at := point.Bucket
+ result.earliestWrite = &at
+ }
+ batch = append(batch, shared.SLOBucket{ProjectId: result.slo.ProjectId, SLOId: result.slo.Id, BucketStart: point.Bucket, Good: point.Good, Total: point.Total})
+ if len(batch) == insertBatchRows {
+ if err := flush(); err != nil {
+ return err
+ }
+ }
+ }
+ }
+ return flush()
+}
+
+func writeSnapshot(ctx context.Context, result *tickResult, now time.Time) error {
+ slo := result.slo
+ sums, err := readWindowSums(ctx, result, now)
+ if err != nil {
+ storeWindowCache(slo.Id, nil)
+ return err
+ }
+ result.fullSnapshot = sums.full
+ snapshot := Snapshot(sums.window, sums.lastHour, sums.lastSixHours, slo.Target(), now, &result.historyFrom, &result.lastClosed)
+ snapshot.GapFrom, snapshot.GapTo = result.gapFrom, result.gapTo
+ applied, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return transactional.SLORepository.UpdateSnapshot(tx, slo.Id, slo.Revision, snapshot)
+ })
+ if err != nil {
+ storeWindowCache(slo.Id, nil)
+ return err
+ }
+ if !applied {
+ storeWindowCache(slo.Id, nil)
+ return discardStale(ctx, slo)
+ }
+ storeWindowCache(slo.Id, sums.cache)
+ return result.backfillErr
+}
+
+func window(count shared.SLOCount) Window {
+ return Window{Good: count.Good, Total: count.Total}
+}
+
+func discardStale(ctx context.Context, stale *models.SLO) error {
+ current, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.SLO, error) {
+ return transactional.SLORepository.FindById(tx, stale.Id)
+ })
+ if err != nil || current != nil {
+ return err
+ }
+ return telemetry.SLOBucketRepository.DeleteBySLO(ctx, stale.ProjectId, stale.Id)
+}
+
+func copyTime(t *time.Time) *time.Time {
+ if t == nil {
+ return nil
+ }
+ value := t.UTC()
+ return &value
+}
+
+func sumRange(ctx context.Context, slo *models.SLO, from, to time.Time) (Window, error) {
+ good, total, err := telemetry.SLOBucketRepository.SumRange(ctx, slo.ProjectId, slo.Id, from, to)
+ return Window{Good: good, Total: total}, err
+}
+
+func WindowSums(ctx context.Context, slo *models.SLO, length time.Duration, now time.Time) (Window, error) {
+ to := now.Truncate(slo.BucketDuration())
+ return sumRange(ctx, slo, to.Add(-length), to)
+}
diff --git a/backend/app/slo/worker_stats_sqlite_test.go b/backend/app/slo/worker_stats_sqlite_test.go
new file mode 100644
index 00000000..1e8b1c69
--- /dev/null
+++ b/backend/app/slo/worker_stats_sqlite_test.go
@@ -0,0 +1,137 @@
+//go:build !transactional_pg && !telemetry_ch && !telemetry_duckdb
+
+package slo
+
+import (
+ "context"
+ "database/sql"
+ "testing"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+func TestIncrementalWindowSumMatchesAFullRead(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ start := time.Now().UTC().Truncate(time.Hour).Add(-6 * time.Hour).Add(30 * time.Second)
+ window := 7 * 24 * time.Hour
+ var seed []models.Endpoint
+ for minute := -90; minute < 180; minute++ {
+ at := start.Add(-window).Add(time.Duration(minute) * time.Minute)
+ seed = append(seed, endpointAt(projectId, 200, at), endpointAt(projectId, 500, at))
+ }
+ for minute := 1; minute < 600; minute += 7 {
+ seed = append(seed, endpointAt(projectId, 200, start.Add(-time.Duration(minute)*time.Minute)))
+ }
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, seed); err != nil {
+ t.Fatal(err)
+ }
+ entity := createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"","good":"availability"}`, 99, 7, 1)
+
+ incremental := 0
+ for minute := 0; minute < 180; minute++ {
+ now := start.Add(time.Duration(minute) * time.Minute)
+ var late []models.Endpoint
+ if minute%5 == 0 {
+ late = append(late, endpointAt(projectId, 500, now.Add(-3*time.Minute)))
+ }
+ if minute == 90 {
+ late = append(late, endpointAt(projectId, 500, now.Add(-20*time.Hour)))
+ }
+ if len(late) > 0 {
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, late); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if minute == 91 {
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ _, err := tx.Exec("UPDATE slos SET last_bucket_at = ? WHERE id = ?", now.Add(-21*time.Hour).Truncate(time.Minute), entity.Id)
+ return struct{}{}, err
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ }
+ RunOnce(ctx, now)
+ if lastTick.FullSnapshots == 0 {
+ incremental++
+ }
+
+ got := reloadSLO(t, entity.Id)
+ lastClosed := now.Truncate(time.Minute)
+ full, err := sumRange(ctx, got, lastClosed.Add(-window), lastClosed)
+ if err != nil {
+ t.Fatal(err)
+ }
+ sli, _ := full.SLI()
+ if got.CurrentSli == nil || *got.CurrentSli != sli*100 {
+ t.Fatalf("minute %d: incremental SLI %v, full read %v (%d/%d)", minute, derefSLI(got.CurrentSli), sli*100, full.Good, full.Total)
+ }
+ }
+ if incremental < 150 {
+ t.Fatalf("only %d of 180 ticks used the cached stable part of the window", incremental)
+ }
+}
+
+func TestTickStatsCountTimeoutsAndTheSlowestSLO(t *testing.T) {
+ projectId := setupSLODB(t)
+ now := time.Now().UTC().Truncate(time.Minute).Add(30 * time.Second)
+ first := createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"","good":"availability"}`, 99, 7, 1)
+ second := createTestSLO(t, projectId, models.SLIKindTask, `{}`, 99, 7, 1)
+ expired, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Second))
+ defer cancel()
+ before := timeoutsTotal
+
+ stats := computeAll(expired, []*models.SLO{first, second}, now)
+ if stats.Timeouts != 2 || stats.Errors != 2 || timeoutsTotal-before != 2 {
+ t.Fatalf("both SLOs time out: %+v", stats)
+ }
+ if stats.Slowest == nil || (stats.Slowest.Id != first.Id && stats.Slowest.Id != second.Id) {
+ t.Fatalf("the slowest SLO is reported: %+v", stats.Slowest)
+ }
+ if reloadSLO(t, first.Id).ComputedAt != nil {
+ t.Fatal("a timed-out SLO keeps its old snapshot")
+ }
+}
+
+func TestFreshnessReportsStaleAndPendingSLOs(t *testing.T) {
+ projectId := setupSLODB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Minute).Add(30 * time.Second)
+ fresh := createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"","good":"availability"}`, 99, 7, 1)
+ stale := createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"","good":"availability"}`, 99, 7, 1)
+ createTestSLO(t, projectId, models.SLIKindEndpoint, `{"endpoint":"","good":"availability"}`, 99, 7, 1)
+ RunOnce(ctx, now)
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ _, err := tx.Exec("UPDATE slos SET computed_at = ? WHERE id = ?", now.Add(-10*time.Minute), stale.Id)
+ if err != nil {
+ return struct{}{}, err
+ }
+ _, err = tx.Exec("UPDATE slos SET computed_at = NULL, history_from = NULL WHERE id NOT IN (?, ?)", fresh.Id, stale.Id)
+ return struct{}{}, err
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ freshness, err := ReadFreshness(ctx, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if freshness.Enabled != 3 || freshness.Stale != 2 || freshness.BackfillPending != 1 {
+ t.Fatalf("freshness = %+v", freshness)
+ }
+ if freshness.MaxStalenessSec < 600 {
+ t.Fatalf("the oldest computation is at least ten minutes old: %+v", freshness)
+ }
+ entities, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.SLO, error) {
+ return transactional.SLORepository.FindEnabled(tx)
+ })
+ if err != nil || len(entities) != 3 {
+ t.Fatalf("enabled SLOs: %d, %v", len(entities), err)
+ }
+}
diff --git a/backend/cmd/run.go b/backend/cmd/run.go
index 130b5cda..14ecfe79 100644
--- a/backend/cmd/run.go
+++ b/backend/cmd/run.go
@@ -31,6 +31,7 @@ import (
"github.com/tracewayapp/traceway/backend/app/retention"
"github.com/tracewayapp/traceway/backend/app/services"
"github.com/tracewayapp/traceway/backend/app/services/mcpmount"
+ "github.com/tracewayapp/traceway/backend/app/slo"
"github.com/tracewayapp/traceway/backend/app/sourcemapbackfill"
"github.com/tracewayapp/traceway/backend/app/storage"
"github.com/tracewayapp/traceway/backend/app/symbolicator/sourcemap/scopes"
@@ -189,6 +190,7 @@ func Run(opts ...Option) {
oncall.StartEscalator(ctx)
notifications.StartEvaluator(ctx)
synthetics.Start(ctx)
+ slo.Start(ctx)
retention.Start(ctx)
recordings.Start(ctx)
sourcemapbackfill.Start(ctx)
@@ -242,6 +244,7 @@ func Run(opts ...Option) {
monitoring.StartTelemetryDBReporter(ctx)
monitoring.StartOutboxReporter(ctx)
monitoring.StartSyntheticsReporter(ctx)
+ monitoring.StartSLOReporter(ctx)
}
router.GET("/health", func(c *gin.Context) {
@@ -448,6 +451,9 @@ func applyEnvOverrides(cfg *config.Cfg) {
{"SMTP_FROM", &cfg.SMTPFrom},
{"ONCALL_POLL_SECONDS", &cfg.OncallPollSeconds},
{"OUTBOX_POLL_SECONDS", &cfg.OutboxPollSeconds},
+ {"SLO_POLL_SECONDS", &cfg.SLOPollSeconds},
+ {"SLO_COMPUTE_WORKERS", &cfg.SLOComputeWorkers},
+ {"SLO_COMPUTE_TIMEOUT_SECONDS", &cfg.SLOComputeTimeoutSecs},
{"SYNTHETICS_POLL_SECONDS", &cfg.SyntheticsPollSeconds},
{"SYNTHETICS_BROWSER_MODE", &cfg.SyntheticsBrowserMode},
{"SYNTHETICS_BROWSER_SANDBOX", &cfg.SyntheticsBrowserSandbox},
diff --git a/cli/pkg/mcpserver/knowledge/learn/alerts.md b/cli/pkg/mcpserver/knowledge/learn/alerts.md
index adca95f4..670e3239 100644
--- a/cli/pkg/mcpserver/knowledge/learn/alerts.md
+++ b/cli/pkg/mcpserver/knowledge/learn/alerts.md
@@ -8,7 +8,7 @@ Alerts are how Traceway tells people that something happened. They live under **
| **Rules** | When to notify. Each rule watches one condition and sends to exactly one channel. A channel can serve many rules. |
| **History** | Every notification that was sent, with its status |
-Rules and channels belong to a project and only see that project's data. Channels have their own On/Off switch, and a rule only runs while its channel is on. Changing them needs write access on the project. Browser and mobile projects only offer the New Issue rule type.
+Rules and channels belong to a project and only see that project's data. Channels have their own On/Off switch, and a rule only runs while its channel is on. Changing them needs write access on the project. Browser and mobile projects only offer the New Issue rule type and the two SLO rule types.
## Channels
@@ -61,6 +61,8 @@ The numbers in brackets are what the evaluator uses when a setting is missing. T
| **AI Conversation Cost** | One conversation's cost over 24 hours reaches the threshold | $5 | Warning, Critical at 3x |
| **AI Flagged Content** | An AI call matches a flagged term. The terms come from the project's AI settings. | Optional term list (empty means any flagged match) | Warning |
| **Monitor Down** | Any monitor in the project goes down, plus a recovery notice when it comes back | None | Critical, recovery Info |
+| **SLO Burn Rate** | An SLO spends its error budget faster than the threshold over both the long and the short window, plus a recovery notice once either window calms down. Fires on the transition only. | SLO, long window 60 min, short window 5 min, threshold 14.4x; presets Fast 14.4x 1h/5m, Slow 6x 6h/30m, Long 1x 3d/6h; both windows round up to whole slices of a metric SLO | Warning, Critical at 2x the threshold, recovery Info |
+| **SLO Budget Consumed** | An SLO has used at least a share of its window's error budget, plus a recovery notice when the budget recovers. Fires on the transition only. | SLO, consumed 80% | Warning, Critical at 100%, recovery Info |
The aggregation in Metric Threshold spans every series of the metric with no tag filter, so prefer `max` (or `min` for lower-is-bad metrics) where one bad host could hide in an average.
@@ -71,13 +73,14 @@ Apdex uses fixed bands: satisfied up to 750 ms, tolerating up to 1500 ms, anythi
- **On ingest**: New Issue, Error Regression, AI Trace Cost, AI Conversation Cost, AI Flagged Content. These fire within seconds.
- **On state change**: Monitor Down, the moment a probe flips the monitor.
- **Polled**: everything else, every 60 seconds by default (self-hosted: `NOTIFICATION_POLL_SECONDS`).
+- **SLO rules** are polled too, but read the SLO's precomputed buckets (refreshed every `SLO_POLL_SECONDS`, 60 by default) instead of raw telemetry, and they are evaluated during their cooldown so a recovery notice is never delayed by it. A recovery notice does not restart the cooldown.
Cooldowns differ by kind:
- **Polled rules** have one cooldown per rule. After it fires, it is not evaluated again until the cooldown ends. This cooldown survives restarts.
- **Ingest and monitor rules** cool down per item: per exception hash, trace name, conversation or monitor. Two different new errors a minute apart both alert. The same error does not alert again within its cooldown. This memory is in process only, so a restart can let the same item alert once more.
- The cooldown cannot be 0. Empty or 0 becomes 15 minutes.
-- **Editing** a rule resets its cooldown and its per-item memory, so the new settings apply at once. Toggling it does not.
+- **Changing** a rule's type, channel or condition resets its cooldown and its per-item memory, so the new settings apply at once. Renaming it, changing its cooldown or severity, or toggling it does not.
Delivery goes through a durable queue. Failed sends are retried after 1, 5, 15 and 60 minutes, five attempts in all. A queued message keeps the channel settings it was sent with.
diff --git a/cli/pkg/mcpserver/tools_operations.go b/cli/pkg/mcpserver/tools_operations.go
index d7037d03..5a2462cf 100644
--- a/cli/pkg/mcpserver/tools_operations.go
+++ b/cli/pkg/mcpserver/tools_operations.go
@@ -27,7 +27,7 @@ func (s *server) addOperationReadTools(srv *mcp.Server) {
mcp.AddTool(srv, &mcp.Tool{
Name: "list_notification_rules",
OutputSchema: passThroughSchema,
- Description: "List the project's alert rules: id, name, ruleType (new_error, error_regression, error_rate_threshold, error_count_threshold, endpoint_p95_threshold, endpoint_p99_threshold, endpoint_error_rate, apdex_drop, throughput_drop, metric_threshold, no_data, task_duration_threshold, task_failure_rate, impact_score_*, ai_trace_cost, ai_conversation_cost, ai_flagged_content, check_down), enabled, snoozedUntil, threshold config and the channels they notify.",
+ Description: "List the project's alert rules: id, name, ruleType (new_error, error_regression, error_rate_threshold, error_count_threshold, endpoint_p95_threshold, endpoint_p99_threshold, endpoint_error_rate, apdex_drop, throughput_drop, metric_threshold, no_data, task_duration_threshold, task_failure_rate, impact_score_*, ai_trace_cost, ai_conversation_cost, ai_flagged_content, check_down, slo_burn_rate, slo_budget_consumed), enabled, snoozedUntil, threshold config and the channels they notify.",
Annotations: readOnly(),
}, s.listNotificationRules)
diff --git a/docs/pages/learn/_meta.json b/docs/pages/learn/_meta.json
index bbcb9789..4f60156d 100644
--- a/docs/pages/learn/_meta.json
+++ b/docs/pages/learn/_meta.json
@@ -20,6 +20,7 @@
"auto-fix": "Auto-Fix",
"on-call": "On-Call",
"monitors": "Monitors",
+ "slos": "SLOs",
"incidents": "Incidents & Post-Mortems",
"sso": "SSO",
"cli": "CLI",
diff --git a/docs/pages/learn/alerts.mdx b/docs/pages/learn/alerts.mdx
index 58b5e1ba..7e33545b 100644
--- a/docs/pages/learn/alerts.mdx
+++ b/docs/pages/learn/alerts.mdx
@@ -125,7 +125,7 @@ A rule connects a condition to a channel. Every rule has:
| Cooldown | Minimum time between notifications for the same condition (default 15 minutes) |
| Severity | Auto (derived from the condition) or an explicit override |
-Rules belong to a project and only evaluate that project's data. Frontend projects only support the New Issue rule type; all other types monitor backend telemetry.
+Rules belong to a project and only evaluate that project's data. Frontend projects support the New Issue rule type and the two SLO rule types; all other types monitor backend telemetry.
## Rule Types
@@ -148,6 +148,8 @@ Rules belong to a project and only evaluate that project's data. Frontend projec
| [AI Conversation Cost](#ai-conversation-cost) | A conversation's cumulative 24-hour cost exceeds a threshold | On ingest |
| [AI Flagged Content](#ai-flagged-content) | An AI conversation matches a flagged content term | On ingest |
| [Monitor Down](#monitor-down) | A monitor crosses its failure threshold | On state change |
+| [SLO Burn Rate](#slo-burn-rate) | An SLO burns its error budget faster than a threshold over both a long and a short window | Polled |
+| [SLO Budget Consumed](#slo-budget-consumed) | An SLO has consumed at least a share of its window's error budget | Polled |
### New Issue
@@ -316,6 +318,28 @@ Fires when a [monitor](/learn/monitors) transitions to down, which happens only
There are no rule-specific options. Pointing the rule at an escalation channel opens an [on-call page](/learn/on-call) instead of sending a message: the page deduplicates per monitor, re-fires never restart its escalation clock, and recovery resolves it automatically.
+### SLO Burn Rate
+
+Fires when an [SLO](/learn/slos) spends its error budget faster than the threshold over both the long and the short window, and sends a recovery notice once either window drops below it. Both windows must burn: the short one keeps a burn that already ended from paging, the long one keeps a one-minute blip from paging. The short window also ends the alert soon after the burn stops, so the Long preset does not stay firing for days. The rule fires once when the condition starts and again only after it cleared, so it never re-pages every poll. The recovery notice is always Info.
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| SLO | required | The SLO to watch |
+| Long Window Minutes | 60 | Lookback of the long window |
+| Short Window Minutes | 5 | Lookback of the short window; must be shorter than the long one. On a metric SLO with longer slices, both windows are rounded up to whole slices |
+| Burn Rate Threshold | 14.4 | Multiple of the rate that spends the budget exactly over the SLO window |
+
+The form offers three presets: Fast (14.4x over 1h / 5m, pages when 2% of a 30-day budget burns in an hour), Slow (6x over 6h / 30m) and Long (1x over 3d / 6h). Pointing the rule at an escalation channel opens an [on-call page](/learn/on-call) that the recovery resolves automatically.
+
+### SLO Budget Consumed
+
+Fires when an [SLO](/learn/slos) has consumed at least the configured share of its window's error budget, and sends a recovery notice when the budget climbs back above the line. Like SLO Burn Rate it fires on the transition only.
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| SLO | required | The SLO to watch |
+| Consumed % | 80 | Share of the error budget that has to be gone |
+
## Evaluation Model and Cost
Rules are evaluated in one of three ways:
@@ -345,10 +369,12 @@ Each rule type has a different evaluation cost against the telemetry database. T
| Throughput Drop | Two counts: lookback plus baseline window | Low |
| Task Failure Rate | Two queries joining tasks and exceptions over the window | Moderate |
| Impact Score | One scan of the trailing 24 hours of requests | Highest |
+| SLO Burn Rate | Two sums over the SLO's precomputed buckets, no raw telemetry | Minimal |
+| SLO Budget Consumed | One sum over the SLO's precomputed buckets | Minimal |
Cost-relevant behavior:
-- Rules that are snoozed or still in cooldown are skipped entirely, including their queries. Longer cooldowns directly reduce evaluation load.
+- Rules that are snoozed or still in cooldown are skipped entirely, including their queries. Longer cooldowns directly reduce evaluation load. SLO rules are the exception: they are evaluated during their cooldown so a recovery notice is never delayed, which costs little since they only read precomputed buckets.
- The three Impact Score levels share one 24-hour scan per project per poll cycle through a 30-second cache, so adding all three costs the same as adding one.
- On-ingest rules deduplicate per error hash, so a busy error stops generating lookups once it has alerted until its cooldown expires.
- Polled query cost scales with the lookback window. The defaults keep windows short; a 24-hour lookback on a high-traffic project costs proportionally more.
@@ -358,10 +384,10 @@ Cost-relevant behavior:
**Cooldown** prevents a rule from firing repeatedly for the same ongoing condition. The default is 15 minutes, configurable per rule.
-- For polled rules the cooldown is rule-wide: after a notification is delivered, the rule is not evaluated again until the cooldown expires.
+- For polled rules the cooldown is rule-wide: after a notification is delivered, the rule is not evaluated again until the cooldown expires. SLO rules keep being evaluated, so their recovery notice is sent at once; the cooldown only holds back a new fire, and a recovery notice does not restart it.
- For on-ingest rules the cooldown is per condition: per error hash for New Issue and Error Regression, per trace name for AI Trace Cost. Different errors can alert back-to-back; the same error cannot.
-Rule-level cooldowns are persisted with the notification history, so a server restart does not cause every rule to re-fire. Per-error cooldown state is held in memory and resets on restart, meaning an already-alerted error can alert once more after a restart. Editing a rule resets its cooldown and per-error state so the new configuration takes effect immediately.
+Rule-level cooldowns are persisted with the notification history, so a server restart does not cause every rule to re-fire. Per-error cooldown state is held in memory and resets on restart, meaning an already-alerted error can alert once more after a restart. Changing a rule's type, channel or condition resets its cooldown and per-error state so the new configuration takes effect immediately. Renaming it or changing its cooldown or severity keeps them.
**Snooze** temporarily mutes a rule for a set duration from the dashboard. Snoozed rules are not evaluated at all. Use it during deployments, maintenance windows, or known incidents.
@@ -388,6 +414,8 @@ Each rule has a **Severity** setting that controls the severity attached to its
| Metric Threshold | Warning; Critical when the value deviates from the threshold by more than 20% |
| AI Trace Cost | Warning; Critical at three times the threshold |
| Monitor Down | Critical; the recovery notice is Info |
+| SLO Burn Rate | Warning; Critical when the long window burns at twice the threshold; the recovery notice is Info |
+| SLO Budget Consumed | Warning; Critical once the whole budget is consumed; the recovery notice is Info |
You can override the automatic severity by setting it explicitly on a rule to **Critical**, **Warning**, or **Info**. This is useful when you want a normally-warning condition to page as critical, or to downgrade noisy rules to info.
diff --git a/docs/pages/learn/index.mdx b/docs/pages/learn/index.mdx
index bcd4f1bd..98e2a4fc 100644
--- a/docs/pages/learn/index.mdx
+++ b/docs/pages/learn/index.mdx
@@ -38,6 +38,10 @@ Endpoints, Tasks, and AI Traces all carry an `is_root` flag. A row is **non-root
See [Data Flow](/learn/data-flow) for how these connect during request processing, task execution, and metrics collection, and [Project Structure](/learn/projects) for how to split an application across Traceway projects before you instrument it.
+## Objectives
+
+An [SLO](/learn/slos) holds any of the above to a target: the share of good requests of an endpoint, of spans matching a filter, of task runs, of monitor probes, of metric time slices, or of crash-free sessions, over a rolling 7, 30 or 90 day window. The worker folds the raw rows into per-minute good/total buckets once, so the current SLI, the error budget and the burn rates are sums over buckets rather than scans, and the burn-rate [alert rules](/learn/alerts#slo-burn-rate) read the same buckets.
+
## Above a single project
Everything above is scoped to one project. The [Organization Overview](/learn/organization-overview) sits above them all: every server reporting into the organization, every recently active issue, every monitor, and every open on-call page, each one a sidebar click apart. It is how a fleet of servers or a [Kubernetes cluster](/learn/kubernetes) is read at a glance before you drill into the project that owns the problem.
diff --git a/docs/pages/learn/slos.mdx b/docs/pages/learn/slos.mdx
new file mode 100644
index 00000000..e2c2bf82
--- /dev/null
+++ b/docs/pages/learn/slos.mdx
@@ -0,0 +1,90 @@
+# SLOs
+
+A service level objective (SLO) holds a service level indicator (SLI) to a target over a rolling window. The SLI is the share of good events: requests that succeeded, spans that were fast enough, task runs without an exception, probes that came back up, metric slices that stayed under a limit, or sessions that ended without a crash. The target says how many of them have to be good, such as 99.9% over the last 30 days. Everything the target leaves over is the error budget, and the burn rate says how fast that budget is being spent.
+
+SLOs live under **SLOs** in the sidebar, one list per project. The sidebar badge counts the SLOs that are currently breached. Each row shows the current SLI, how much of the error budget is left, the burn rate over the last hour and a status. The detail page adds the burn rate over the last six hours. The statuses are:
+
+| Status | Meaning |
+|--------|---------|
+| Meeting | The SLI over the window is at or above the target |
+| Breached | The SLI is below the target, so the error budget is overspent |
+| No data | Nothing matched the indicator in the window yet |
+| Paused | The SLO is disabled; nothing is computed, and an alert that was firing on it sends its recovery notice |
+
+## SLI kinds
+
+| Kind | Counts | Good when | View bad events opens |
+|------|--------|-----------|-----------------------|
+| Endpoint | Requests of one endpoint, or of every endpoint in the project | The status is below 500, or the request took at most a threshold | The endpoint page, or the endpoints list for all endpoints |
+| Span | Spans matching the same filters the [span explorer](/learn/spans) offers: service, name, kind and exact attribute values | The span status is not error, or the span took at most a threshold | The span explorer, prefiltered |
+| Task | Background task runs, all tasks or one by name | No exception or captured message was recorded on the run's own span | The task page, or the tasks list for all tasks |
+| Monitor | Probes of the selected [monitors](/learn/monitors) | The probe came back up; missed probes do not count | The monitor page, or the monitors list for several monitors |
+| Metric | Fixed time slices of a metric (1, 5, 15 or 60 minutes), optionally filtered by tags | The slice's aggregate (average, min, max, sum or last value) satisfies the comparison | Dashboards |
+| Session | Sessions reported by the SDK, all versions or one app version | No exception was recorded in the session; captured messages count only if you say so | The sessions page |
+
+Browser and mobile projects (React, Vue, Svelte, jQuery, React Native, Flutter, Android, iOS) do not report spans, so they get the session kind alone: the classic crash-free sessions objective, optionally per app version. Backend projects get the other five.
+
+The session kind counts the sessions an SDK reports. The browser SDK reports one per visit when [always-on session recording](/client/js-sdk/session-replay#always-on-session-recording) is on (`recordAllSessions: true`). Without it, and in the mobile SDKs today, no sessions are reported and a session SLO stays at No data.
+
+The span kind is the general one. A queue consumer, an outbound call to a payment provider, or a GraphQL resolver can each carry their own SLO as long as the spans carry an attribute that identifies them. Endpoints and tasks are promoted from spans, so a span SLO covers them too.
+
+The metric kind is a timeslice SLI: every slice of the window is one unit, which is what Datadog and Elastic call time slice or timeslices budgeting. A slice without any sample is ignored by default. Turn on **Slices without data count as bad** for a metric whose absence means the thing it measures is gone, such as a heartbeat. Empty slices count only from the moment the SLO was saved, or its indicator last changed. The backfill of older history never turns the time before the metric existed, or past its retention, into bad slices.
+
+Endpoint latency indicators leave streaming endpoints out, since a connection lifetime is not a response time. They also ignore the status code, so a fast 500 counts as good; pair a latency SLO with an availability one.
+
+When **Drop healthy healthcheck requests** is on (the default, see [Healthchecks](/learn/healthchecks)), healthy probes are never stored, so an endpoint SLO only sees the failing ones. To hold a health endpoint to a target, use a monitor SLO, or turn the toggle off for that project.
+
+The sheet has a **Preview last 24h** button that runs the indicator straight against your telemetry, so you can see the share of good events and the counts before saving. Like a newly saved SLO, the preview does not count empty metric slices as bad.
+
+An SLO can watch at most 50 monitors, 20 metric tags and 10 span attribute filters.
+
+## Windows and the error budget
+
+The window is rolling: 7, 30 or 90 days ending now. The error budget is the share of events the target allows to be bad. With a 99.9% target and 1,000,000 requests in the window, 1,000 of them may fail. 500 failures leave half the budget; 1,200 failures put the budget at minus 20% and the SLO in breach.
+
+The burn rate is the observed bad rate divided by the allowed bad rate. A burn rate of 1 spends the budget exactly at the end of the window. 14.4 exhausts a 30-day budget in about two days. The two burn rate tiles use the last hour and the last six hours; a window without events burns nothing.
+
+Changing the target or the window applies at once and keeps the history, since both are applied on read. Changing what the indicator measures, or the metric slice length, recomputes the history from scratch; the sheet warns before you save.
+
+## Burn-rate alerting
+
+Two [alert rule](/learn/alerts) types read the SLO:
+
+- **SLO Burn Rate** fires when the budget burns faster than a threshold over both a long and a short window. The short window stops a burn that already ended from paging, the long window stops a one-minute blip from paging. It sends a recovery notice once either window drops below the threshold, so the alert ends soon after the burn stops.
+- **SLO Budget Consumed** fires when a share of the window's budget or more is gone, and sends a recovery notice when the budget climbs back above the line.
+
+The presets follow the Google SRE workbook:
+
+| Preset | Long / short window | Threshold | Share of a 30-day budget burned when it fires | Time to exhaustion at that rate |
+|--------|---------------------|-----------|-----------------------------------------------|---------------------------------|
+| Fast | 1h / 5m | 14.4x | 2% | about 2 days |
+| Slow | 6h / 30m | 6x | 5% | 5 days |
+| Long | 3d / 6h | 1x | 10% | 30 days |
+
+Point the Fast preset at a paging channel and the Slow and Long ones at chat. Add SLO Budget Consumed at 80% as the planning signal that says the next deploy should be a careful one. Pointing either rule at an escalation channel opens an [on-call page](/learn/on-call) that the recovery resolves automatically. The rules fire once when the condition starts and again only after it cleared, so they do not re-page every poll. A rule is still evaluated during its cooldown so the recovery is never delayed; the cooldown only holds back a new fire, and a recovery notice does not restart it. Changing a rule's type, channel or condition, disabling it or deleting it resolves the page it opened, and so does disabling or deleting its SLO. Deleting an SLO also deletes the rules that read it. While a changed SLO is being recomputed, its rules keep their state, so a definition change does not send a false recovery.
+
+On a metric SLO with slices longer than a minute, both windows are rounded up to whole slices, so a 5-minute short window over 60-minute slices reads the last hour. The rule form says so when it happens.
+
+## How history is computed
+
+A background worker computes the indicator into buckets of one minute (or the metric slice length) and stores them next to the telemetry, so the SLI, the budget and the burn rates are sums over buckets rather than scans of raw data. Every poll recomputes the last ten minutes, so data that arrives a little late still lands in its bucket, and backfills older history in chunks until the whole window is covered. A new 90-day SLO fills over a few minutes; the detail page says how much of the range the history covers while it does. All SLOs share one five-second backfill budget per poll, so adding many at once never delays the refresh of the others.
+
+Data that arrives late is picked up too. Every 15 minutes, at a moment that differs per SLO so they never all run together, the worker recomputes the last hour and a quarter, which catches requests longer than ten minutes (a span is counted at its start but exported at its end) and batches an exporter retried after an outage. Sessions and task runs are counted at their start but turn bad when their exception arrives, so their buckets stay open longer: every 15 minutes the worker recomputes about the last two and a half hours of a session SLO and the last six and a half hours of a task SLO. A crash that is uploaded on the next app launch is caught too: once a day the worker recomputes the last three days of every session SLO, so a crash counts when the app is opened again within two days, and usually up to three.
+
+If the backend was down, the first poll after it recomputes the last 24 hours, and the rest of the gap is filled from the raw telemetry within the same backfill budget.
+
+Buckets are kept for 120 days whatever the raw telemetry retention is. On SQLite and DuckDB the raw retention defaults to 30 days (`SQLITE_RETENTION_DAYS`, `DUCKDB_RETENTION_DAYS`), so a 90-day window backfills only what is still retained, and buckets already computed survive the pruning of the raw rows. On ClickHouse, metric indicators backfill from the 1-minute rollups, which keep 30 days. The last value aggregation reads the raw points instead, which keep 7 days.
+
+## Configuration
+
+```bash
+SLO_POLL_SECONDS=60 # how often buckets are refreshed and statuses recomputed; values below 5 fall back to 60
+SLO_COMPUTE_WORKERS=4 # SLOs computed at once per instance (max 64)
+SLO_COMPUTE_TIMEOUT_SECONDS=30 # an SLO whose queries take longer is skipped for that tick and retried on the next
+```
+
+With several backend instances on Postgres, the instances split the SLOs between them: SLOs are spread over 64 shards, and each instance leases an equal share. When an instance stops, the others take over its shards within about five minutes.
+
+Each poll re-reads only the recent hours of an SLO's buckets and keeps the older part of the window in memory, so a 90-day SLO costs about as much as a 7-day one. The whole window is re-read once an hour per SLO, and whenever older buckets change.
+
+To watch the worker, turn on [self-monitoring](/server/capacity-monitoring#slo-worker-every-30s). It reports how far behind the SLOs are, how long ticks take and which SLO was slowest, so you can alert when SLOs fall behind. `/api/health/deep` shows the same numbers in its `slo` block.
diff --git a/docs/pages/server/capacity-monitoring.mdx b/docs/pages/server/capacity-monitoring.mdx
index f9c68625..a1e3f43f 100644
--- a/docs/pages/server/capacity-monitoring.mdx
+++ b/docs/pages/server/capacity-monitoring.mdx
@@ -87,6 +87,24 @@ Not emitted in SQLite mode.
| `traceway.sourcemap.disk.entries` / `disk.bytes` | Local `.tw` files cached on disk and their total size (only reported when `SOURCEMAP_CACHE_TYPE=disk`) |
| `traceway.sourcemap.disk.hits.delta` / `disk.evictions.delta` | Memory misses served by a local `.tw` file, and local files evicted by the byte cap, since the last tick |
+### SLO worker (every 30s)
+
+The [SLO](/learn/slos) worker recomputes every enabled SLO each `SLO_POLL_SECONDS`. The freshness metrics are read from the main database, so every instance reports the same values, and they also fire when the worker has stopped entirely. The tick metrics describe the last tick of the reporting instance.
+
+| Metric | Meaning |
+|--------|---------|
+| `traceway.slo.enabled` | Enabled SLOs across all projects |
+| `traceway.slo.max_staleness_sec` | Seconds since the least recently computed SLO was computed. It sits near `SLO_POLL_SECONDS` when the worker keeps up, and grows when ticks fall behind or an SLO keeps failing |
+| `traceway.slo.stale` | SLOs not computed in the last 5 minutes |
+| `traceway.slo.backfill_pending` | SLOs whose history does not cover their window yet, or that still have an outage gap to fill |
+| `traceway.slo.tick.running_sec` | How long the current tick has been running, 0 between ticks. A value that keeps growing is a stuck tick |
+| `traceway.slo.tick.duration_ms` | Duration of the last tick, split into `tick.compute_ms` (raw telemetry queries), `tick.store_ms` (bucket inserts) and `tick.snapshot_ms` (window sums and status updates) |
+| `traceway.slo.tick.slos` / `tick.buckets` | SLOs this instance computed in the last tick, and the buckets it wrote |
+| `traceway.slo.tick.full_snapshots` | SLOs whose whole window was re-read in the last tick. The rest only read their recent hours, so this should stay a small share of `tick.slos` |
+| `traceway.slo.tick.slowest_ms` | The slowest SLO of the last tick, tagged `kind`. Its id and project are in the `slo.lastTick.slowest` block of `/api/health/deep` |
+| `traceway.slo.computed.delta` / `errors.delta` / `timeouts.delta` | SLOs computed, failed, and stopped by `SLO_COMPUTE_TIMEOUT_SECONDS` since the last report. Each failure and timeout is also captured as an exception naming the SLO |
+| `traceway.slo.shards_owned` / `instances` | SLO shards leased by this instance, and the backend instances sharing them (Postgres builds; a single instance owns all 64) |
+
## Source map cache sizing
Symbolication resolves stack frames against object storage at `sourcemaps/{projectId}/{filename}`, with one in-memory LRU of built resolvers in front of it. A resolver keeps only a flat token table plus interned file and function names; the raw `.map` and bundle bytes are held only while building. Concurrent requests for the same map share one storage read and one build, so a given map exists in memory at most once. The cache cap is on the estimated in-memory size, not the raw file size.
@@ -139,6 +157,9 @@ Two more rule types complete the picture:
| `traceway.ch.disk.used_pcnt` | max | gt | 80 | 15 | warning | ClickHouse disk filling up |
| `traceway.ch.pool.open` | avg | gte | 15 | 10 | warning | Connection pool saturated; check `insert_ms` next |
| `traceway.recordings.dropped.delta` | max | gt | 0 | 5 | warning | Recording segments being dropped (ingestion overload) |
+| `traceway.slo.max_staleness_sec` | max | gt | 600 | 10 | warning | SLOs are 10 minutes behind: ticks take too long or the worker stopped. Check `tick.duration_ms` and `tick.slowest_ms` |
+| `traceway.slo.timeouts.delta` | max | gt | 0 | 15 | warning | An SLO query hit `SLO_COMPUTE_TIMEOUT_SECONDS`; the exception names it |
+| `traceway.slo.tick.running_sec` | max | gt | 300 | 10 | warning | A single SLO tick has been running for 5 minutes |
| `traceway.sourcemap.load_failures.delta` | max | gt | 0 | 10 | warning | Source map loads failing (storage errors/timeouts); traces stored unsymbolicated |
| `cpu.used_pcnt` | avg | gt | 80 | 10 | warning | Backend CPU-bound |
| `traceway.proc.rss_mb` | avg | gt | 80% of host RAM | 10 | warning | Memory pressure / OOM risk (e.g. 800 on a 1 GB box) |
diff --git a/frontend/src/lib/components/app-sidebar.svelte b/frontend/src/lib/components/app-sidebar.svelte
index 0c3742bb..1ac03a79 100644
--- a/frontend/src/lib/components/app-sidebar.svelte
+++ b/frontend/src/lib/components/app-sidebar.svelte
@@ -3,6 +3,7 @@
import { useSidebar } from '$lib/components/ui/sidebar';
import { oncallState } from '$lib/state/oncall.svelte';
import { monitorsState } from '$lib/state/monitors.svelte';
+ import { slosState } from '$lib/state/slos.svelte';
import { organizationContext } from '$lib/state/organization-context.svelte';
import {
Activity,
@@ -17,6 +18,7 @@
Flame,
FolderKanban,
Gauge,
+ Goal,
ListEnd,
PhoneCall,
Server,
@@ -90,6 +92,7 @@
title: 'Monitors',
stickyParams: ['preset', 'from', 'to']
},
+ { Icon: Goal, href: '/slos', title: 'SLOs', stickyParams: [] },
{ Icon: Bell, href: '/notifications', title: 'Alerts', stickyParams: ['preset', 'from', 'to'] },
{ Icon: PhoneCall, href: '/on-call', title: 'On-Call', stickyParams: [] },
{ Icon: Link2, href: '/connection', title: 'Connection', stickyParams: [] }
@@ -206,6 +209,9 @@
? organizationContext.openPagesCount
: oncallState.openPagesCount;
}
+ if (item.title === 'SLOs') {
+ return organizationContext.active ? 0 : slosState.breachedCount;
+ }
return 0;
}
@@ -215,11 +221,14 @@
const projectId = projectsState.currentProjectId;
if (!projectId) return;
const hasMonitors = sidebarItems.some((item) => item.title === 'Monitors');
+ const hasSlos = sidebarItems.some((item) => item.title === 'SLOs');
oncallState.refreshOpenCount();
if (hasMonitors) monitorsState.refreshDownCount();
+ if (hasSlos) slosState.refreshBreachedCount();
const interval = setInterval(() => {
oncallState.refreshOpenCount();
if (hasMonitors) monitorsState.refreshDownCount();
+ if (hasSlos) slosState.refreshBreachedCount();
}, 60000);
return () => clearInterval(interval);
});
diff --git a/frontend/src/lib/components/dashboard/d3-line-chart.svelte b/frontend/src/lib/components/dashboard/d3-line-chart.svelte
index dfbef75a..557ed67b 100644
--- a/frontend/src/lib/components/dashboard/d3-line-chart.svelte
+++ b/frontend/src/lib/components/dashboard/d3-line-chart.svelte
@@ -18,6 +18,12 @@
color: string;
};
+ type ReferenceLine = {
+ value: number;
+ label?: string;
+ color?: string;
+ };
+
type ServerData = {
serverName: string;
trend: DataPoint[];
@@ -38,7 +44,9 @@
areaFill = false,
sharedHoverTime = null,
isSourceChart = false,
- onHoverTimeChange
+ onHoverTimeChange,
+ referenceLines = [],
+ yDomain = null
} = $props<{
series: Series[];
xDomain?: [Date, Date] | null;
@@ -55,6 +63,8 @@
sharedHoverTime?: Date | null;
isSourceChart?: boolean;
onHoverTimeChange?: (time: Date | null) => void;
+ referenceLines?: ReferenceLine[];
+ yDomain?: [number, number] | null;
}>();
const tz = $derived(getTimezone());
@@ -97,12 +107,14 @@
const yMin = $derived(() => {
const minVal = min(allData, (d: DataPoint) => d.value);
- return typeof minVal === 'number' ? minVal : 0;
+ const base = typeof minVal === 'number' ? minVal : 0;
+ return referenceLines.reduce((low: number, r: ReferenceLine) => Math.min(low, r.value), base);
});
const yMax = $derived(() => {
const maxVal = max(allData, (d: DataPoint) => d.value);
- return typeof maxVal === 'number' ? maxVal : 0;
+ const base = typeof maxVal === 'number' ? maxVal : 0;
+ return referenceLines.reduce((high: number, r: ReferenceLine) => Math.max(high, r.value), base);
});
// Add padding to y domain
@@ -113,7 +125,11 @@
// Scales
const xScale = $derived(scaleUtc().domain([xMin, xMax]).range([0, chartWidth]));
- const yScale = $derived(scaleLinear().domain([yDomainMin, yDomainMax]).range([chartHeight, 0]));
+ const yScale = $derived(
+ scaleLinear()
+ .domain(yDomain ?? [yDomainMin, yDomainMax])
+ .range([chartHeight, 0])
+ );
// Generate Y axis ticks
const yTicks = $derived(() => yScale.ticks(4));
@@ -498,6 +514,32 @@
</text>
{/each}
+ {#each referenceLines as reference, __index (__index)}
+ {@const referenceY = yScale(reference.value)}
+ {#if referenceY >= 0 && referenceY <= chartHeight}
+ <line
+ x1={0}
+ y1={referenceY}
+ x2={chartWidth}
+ y2={referenceY}
+ stroke={reference.color ?? 'var(--destructive)'}
+ stroke-width="1"
+ stroke-dasharray="6 3"
+ />
+ {#if reference.label}
+ <text
+ x={chartWidth - 4}
+ y={referenceY - 4}
+ text-anchor="end"
+ fill={reference.color ?? 'var(--destructive)'}
+ font-size="10"
+ >
+ {reference.label}
+ </text>
+ {/if}
+ {/if}
+ {/each}
+
<!-- Y axis unit label -->
{#if yAxisUnit() && !unitOnTopTick}
<text x={-8} y={-4} text-anchor="end" fill="#9ca3af" font-size="10">
diff --git a/frontend/src/lib/components/slos/slo-kind-badge.svelte b/frontend/src/lib/components/slos/slo-kind-badge.svelte
new file mode 100644
index 00000000..0d8ec2dd
--- /dev/null
+++ b/frontend/src/lib/components/slos/slo-kind-badge.svelte
@@ -0,0 +1,26 @@
+<script lang="ts">
+ import { SLI_KIND_LABELS, type SliKind } from '$lib/state/slos.svelte';
+
+ interface Props {
+ kind: SliKind;
+ }
+
+ let { kind }: Props = $props();
+
+ const kindClasses: Record<SliKind, string> = {
+ endpoint: 'bg-blue-500/15 text-blue-600 dark:text-blue-400',
+ span: 'bg-violet-500/15 text-violet-600 dark:text-violet-400',
+ task: 'bg-cyan-500/15 text-cyan-700 dark:text-cyan-400',
+ monitor: 'bg-amber-500/15 text-amber-700 dark:text-amber-400',
+ metric: 'bg-emerald-500/15 text-emerald-700 dark:text-emerald-400',
+ session: 'bg-rose-500/15 text-rose-600 dark:text-rose-400'
+ };
+</script>
+
+<span
+ class="inline-flex items-center gap-1 rounded-full px-2 py-0.5 text-xs font-medium tracking-wide uppercase {kindClasses[
+ kind
+ ] ?? 'bg-muted text-muted-foreground'}"
+>
+ {SLI_KIND_LABELS[kind] ?? kind}
+</span>
diff --git a/frontend/src/lib/components/slos/slo-status-badge.svelte b/frontend/src/lib/components/slos/slo-status-badge.svelte
new file mode 100644
index 00000000..cfaf56e1
--- /dev/null
+++ b/frontend/src/lib/components/slos/slo-status-badge.svelte
@@ -0,0 +1,21 @@
+<script lang="ts">
+ import type { SloStatus } from '$lib/state/slos.svelte';
+ import StatusPill from '$lib/components/traceway/status-pill.svelte';
+
+ interface Props {
+ status: SloStatus;
+ enabled?: boolean;
+ }
+
+ let { status, enabled = true }: Props = $props();
+</script>
+
+{#if !enabled}
+ <StatusPill tone="neutral" dot label="Paused" />
+{:else if status === 'ok'}
+ <StatusPill tone="success" dot label="Meeting" />
+{:else if status === 'breached'}
+ <StatusPill tone="danger" dot label="Breached" />
+{:else}
+ <StatusPill tone="neutral" dot label="No data" />
+{/if}
diff --git a/frontend/src/lib/state/slos.svelte.ts b/frontend/src/lib/state/slos.svelte.ts
new file mode 100644
index 00000000..5c73b8cd
--- /dev/null
+++ b/frontend/src/lib/state/slos.svelte.ts
@@ -0,0 +1,138 @@
+import { api } from '$lib/api';
+import { projectsState } from '$lib/state/projects.svelte';
+import type { NotificationRuleConfig } from '$lib/types/notifications';
+
+export type SliKind = 'endpoint' | 'span' | 'task' | 'monitor' | 'metric' | 'session';
+export type SloStatus = 'ok' | 'breached' | 'no_data';
+export type GoodCondition = 'availability' | 'latency';
+export type MetricAggregation = 'avg' | 'min' | 'max' | 'sum' | 'last';
+export type MetricOperator = 'gt' | 'gte' | 'lt' | 'lte';
+
+export interface EndpointSliConfig {
+ endpoint: string;
+ good: GoodCondition;
+ latencyThresholdMs: number;
+}
+
+export interface SpanSliAttribute {
+ key: string;
+ value: string;
+}
+
+export interface SpanSliConfig {
+ serviceName: string;
+ name: string;
+ kind: number | null;
+ attributes: SpanSliAttribute[];
+ good: GoodCondition;
+ latencyThresholdMs: number;
+}
+
+export interface TaskSliConfig {
+ taskName: string;
+}
+
+export interface MonitorSliConfig {
+ checkIds: number[];
+}
+
+export interface MetricSliConfig {
+ metricName: string;
+ tags: Record<string, string>;
+ aggregation: MetricAggregation;
+ operator: MetricOperator;
+ threshold: number | null;
+ missingIsBad: boolean;
+}
+
+export interface SessionSliConfig {
+ appVersion: string;
+ includeMessages: boolean;
+}
+
+export type SliConfig =
+ | EndpointSliConfig
+ | SpanSliConfig
+ | TaskSliConfig
+ | MonitorSliConfig
+ | MetricSliConfig
+ | SessionSliConfig;
+
+export interface SLO {
+ id: number;
+ projectId: string;
+ name: string;
+ description: string;
+ sliKind: SliKind;
+ sliConfig: SliConfig;
+ targetPercent: number;
+ windowDays: number;
+ bucketMinutes: number;
+ enabled: boolean;
+ status: SloStatus;
+ currentSli: number | null;
+ budgetRemaining: number | null;
+ burnRate1h: number | null;
+ burnRate6h: number | null;
+ computedAt: string | null;
+ historyFrom: string | null;
+ lastBucketAt: string | null;
+ createdAt: string;
+ updatedAt: string;
+}
+
+export interface SloSeriesBucket {
+ bucket: string;
+ good: number;
+ total: number;
+}
+
+export interface SloPreview {
+ good: number;
+ total: number;
+ sli: number | null;
+ from: string;
+ to: string;
+}
+
+export interface SloAlertRule {
+ id: number;
+ name: string;
+ ruleType: string;
+ config: NotificationRuleConfig;
+ enabled: boolean;
+ snoozedUntil: string | null;
+ channelName: string;
+ channelType: string;
+}
+
+export const SLI_KIND_LABELS: Record<SliKind, string> = {
+ endpoint: 'Endpoint',
+ span: 'Span',
+ task: 'Task',
+ monitor: 'Monitor',
+ metric: 'Metric',
+ session: 'Session'
+};
+
+export const BACKEND_SLI_KINDS: SliKind[] = ['endpoint', 'span', 'task', 'monitor', 'metric'];
+export const FRONTEND_SLI_KINDS: SliKind[] = ['session'];
+
+export const SLO_WINDOW_OPTIONS = [
+ { value: '7', label: '7 days' },
+ { value: '30', label: '30 days' },
+ { value: '90', label: '90 days' }
+];
+
+class SlosState {
+ breachedCount = $state(0);
+
+ async refreshBreachedCount() {
+ const res = await api
+ .get('/slos/breached-count', { projectId: projectsState.currentProjectId ?? undefined })
+ .catch(() => null);
+ if (res) this.breachedCount = res.count ?? 0;
+ }
+}
+
+export const slosState = new SlosState();
diff --git a/frontend/src/lib/types/notifications.ts b/frontend/src/lib/types/notifications.ts
index 4efda066..115b9268 100644
--- a/frontend/src/lib/types/notifications.ts
+++ b/frontend/src/lib/types/notifications.ts
@@ -49,4 +49,9 @@ export type NotificationRuleConfig = {
traceName?: string;
thresholdCost?: number;
terms?: string[];
+ sloId?: number;
+ longWindowMinutes?: number;
+ shortWindowMinutes?: number;
+ burnRateThreshold?: number;
+ consumedPercent?: number;
};
diff --git a/frontend/src/lib/utils/slo-links.test.ts b/frontend/src/lib/utils/slo-links.test.ts
new file mode 100644
index 00000000..ea8559b3
--- /dev/null
+++ b/frontend/src/lib/utils/slo-links.test.ts
@@ -0,0 +1,72 @@
+import { describe, expect, it } from 'vitest';
+import { badEventsHref } from './slo-links';
+import { readSpanSearchFilters } from './span-explorer';
+
+describe('badEventsHref', () => {
+ it('opens the span explorer with the SLI filters and the error status', () => {
+ const href = badEventsHref({
+ sliKind: 'span',
+ sliConfig: {
+ serviceName: 'api',
+ name: 'POST /checkout',
+ kind: 2,
+ attributes: [{ key: 'http.route', value: '/checkout' }],
+ good: 'availability',
+ latencyThresholdMs: 0
+ }
+ });
+ expect(href.startsWith('/spans?')).toBe(true);
+ const filters = readSpanSearchFilters(href.slice(href.indexOf('?')));
+ expect(filters.service).toBe('api');
+ expect(filters.name).toBe('POST /checkout');
+ expect(filters.kind).toBe('2');
+ expect(filters.status).toBe('2');
+ expect(filters.attributes).toEqual([{ key: 'http.route', value: '/checkout' }]);
+ });
+
+ it('filters slow spans for a latency SLI', () => {
+ const href = badEventsHref({
+ sliKind: 'span',
+ sliConfig: {
+ serviceName: '',
+ name: '',
+ kind: null,
+ attributes: [],
+ good: 'latency',
+ latencyThresholdMs: 800
+ }
+ });
+ const filters = readSpanSearchFilters(href.slice(href.indexOf('?')));
+ expect(filters.status).toBe('');
+ expect(filters.minMs).toBe('800');
+ });
+
+ it('links the other kinds to their pages', () => {
+ expect(
+ badEventsHref({
+ sliKind: 'endpoint',
+ sliConfig: { endpoint: 'GET /users', good: 'availability', latencyThresholdMs: 0 }
+ })
+ ).toBe('/endpoints/GET%20%2Fusers?preset=24h');
+ expect(badEventsHref({ sliKind: 'task', sliConfig: { taskName: '' } })).toBe(
+ '/tasks?preset=24h'
+ );
+ expect(badEventsHref({ sliKind: 'monitor', sliConfig: { checkIds: [7] } })).toBe('/monitors/7');
+ expect(
+ badEventsHref({ sliKind: 'session', sliConfig: { appVersion: '', includeMessages: false } })
+ ).toBe('/sessions?preset=24h');
+ expect(
+ badEventsHref({
+ sliKind: 'metric',
+ sliConfig: {
+ metricName: 'x',
+ tags: {},
+ aggregation: 'avg',
+ operator: 'lt',
+ threshold: 1,
+ missingIsBad: false
+ }
+ })
+ ).toBe('/dashboards');
+ });
+});
diff --git a/frontend/src/lib/utils/slo-links.ts b/frontend/src/lib/utils/slo-links.ts
new file mode 100644
index 00000000..50d305b7
--- /dev/null
+++ b/frontend/src/lib/utils/slo-links.ts
@@ -0,0 +1,71 @@
+import type { SLO, SpanSliConfig } from '$lib/state/slos.svelte';
+import { spanSearchUrlParams } from '$lib/utils/span-explorer';
+
+const BAD_EVENTS_PRESET = '24h';
+
+export function badEventsLabel(slo: Pick<SLO, 'sliKind'>): string {
+ switch (slo.sliKind) {
+ case 'endpoint':
+ return 'View requests';
+ case 'span':
+ return 'View bad spans';
+ case 'task':
+ return 'View task runs';
+ case 'monitor':
+ return 'Open monitor';
+ case 'session':
+ return 'View sessions';
+ default:
+ return 'Open dashboards';
+ }
+}
+
+export function badEventsHref(slo: Pick<SLO, 'sliKind' | 'sliConfig'>): string {
+ const cfg = slo.sliConfig as unknown as Record<string, unknown>;
+ switch (slo.sliKind) {
+ case 'endpoint': {
+ const endpoint = cfg.endpoint as string;
+ return endpoint
+ ? `/endpoints/${encodeURIComponent(endpoint)}?preset=${BAD_EVENTS_PRESET}`
+ : `/endpoints?preset=${BAD_EVENTS_PRESET}`;
+ }
+ case 'span': {
+ const span = slo.sliConfig as SpanSliConfig;
+ const params = spanSearchUrlParams({
+ name: span.name ?? '',
+ service: span.serviceName ?? '',
+ kind: span.kind === null || span.kind === undefined ? '' : String(span.kind),
+ status: span.good === 'availability' ? '2' : '',
+ minMs: span.good === 'latency' ? String(span.latencyThresholdMs) : '',
+ maxMs: '',
+ traceId: '',
+ attributes: span.attributes ?? []
+ });
+ const search = new URLSearchParams();
+ for (const [key, value] of Object.entries(params)) {
+ if (value === undefined) continue;
+ if (Array.isArray(value)) {
+ for (const item of value) search.append(key, item);
+ } else {
+ search.set(key, value);
+ }
+ }
+ search.set('preset', BAD_EVENTS_PRESET);
+ return `/spans?${search.toString()}`;
+ }
+ case 'task': {
+ const taskName = cfg.taskName as string;
+ return taskName
+ ? `/tasks/${encodeURIComponent(taskName)}?preset=${BAD_EVENTS_PRESET}`
+ : `/tasks?preset=${BAD_EVENTS_PRESET}`;
+ }
+ case 'session':
+ return `/sessions?preset=${BAD_EVENTS_PRESET}`;
+ case 'monitor': {
+ const ids = (cfg.checkIds as number[] | undefined) ?? [];
+ return ids.length === 1 ? `/monitors/${ids[0]}` : '/monitors';
+ }
+ default:
+ return '/dashboards';
+ }
+}
diff --git a/frontend/src/lib/utils/slo-math.test.ts b/frontend/src/lib/utils/slo-math.test.ts
new file mode 100644
index 00000000..97274b80
--- /dev/null
+++ b/frontend/src/lib/utils/slo-math.test.ts
@@ -0,0 +1,136 @@
+import { describe, expect, it } from 'vitest';
+import {
+ budgetTone,
+ burnTone,
+ calculateSloInterval,
+ clampRangeToWindow,
+ defaultPresetForWindow,
+ errorBudgetSeries,
+ formatBudgetRemaining,
+ formatSloPercent,
+ sliSeries,
+ sliSummary
+} from './slo-math';
+
+const buckets = [
+ { bucket: '2026-10-01T00:00:00Z', good: 990, total: 1000 },
+ { bucket: '2026-10-01T01:00:00Z', good: 1000, total: 1000 },
+ { bucket: '2026-10-01T02:00:00Z', good: 0, total: 0 },
+ { bucket: '2026-10-01T03:00:00Z', good: 980, total: 1000 }
+];
+
+describe('slo-math', () => {
+ it('builds the SLI series from non-empty buckets only', () => {
+ const points = sliSeries(buckets);
+ expect(points).toHaveLength(3);
+ expect(points[0].value).toBeCloseTo(99);
+ expect(points[2].value).toBeCloseTo(98);
+ });
+
+ it('accumulates budget consumption against the range budget', () => {
+ const points = errorBudgetSeries(buckets, 99);
+ expect(points).toHaveLength(4);
+ expect(points[0].value).toBeCloseTo((10 / 30) * 100);
+ expect(points[3].value).toBeCloseTo(100);
+ });
+
+ it('has no budget series without traffic or without a budget', () => {
+ expect(errorBudgetSeries([], 99)).toEqual([]);
+ expect(errorBudgetSeries(buckets, 100)).toEqual([]);
+ });
+
+ it('tones the budget', () => {
+ expect(budgetTone(null)).toBe('neutral');
+ expect(budgetTone(-5)).toBe('danger');
+ expect(budgetTone(10)).toBe('warning');
+ expect(budgetTone(60)).toBe('success');
+ });
+
+ it('keeps at least the target precision', () => {
+ expect(formatSloPercent(99.9512, 99.9)).toBe('99.95%');
+ expect(formatSloPercent(99.9512, 99.999)).toBe('99.951%');
+ expect(formatSloPercent(null, 99.9)).toBe('—');
+ expect(formatSloPercent(99.8996, 99.9)).toBe('99.89%');
+ expect(formatSloPercent(99.95, 99.9)).toBe('99.95%');
+ expect(formatSloPercent(100, 99.9)).toBe('100.00%');
+ });
+
+ it('maps windows to presets and clamps ranges', () => {
+ expect(defaultPresetForWindow(7)).toBe('7d');
+ expect(defaultPresetForWindow(30)).toBe('1M');
+ expect(defaultPresetForWindow(90)).toBe('3M');
+ const now = new Date('2026-10-25T12:00:00Z');
+ const clamped = clampRangeToWindow(new Date('2026-07-01T00:00:00Z'), now, 7, now);
+ expect(clamped.clamped).toBe(true);
+ expect(clamped.from.toISOString()).toBe('2026-10-18T12:00:00.000Z');
+ const kept = clampRangeToWindow(new Date('2026-10-24T00:00:00Z'), now, 7, now);
+ expect(kept.clamped).toBe(false);
+ const endOfToday = new Date(now.getTime() + 11 * 60 * 60 * 1000);
+ const trimmed = clampRangeToWindow(new Date('2026-10-24T00:00:00Z'), endOfToday, 7, now);
+ expect(trimmed.clamped).toBe(false);
+ expect(trimmed.to.toISOString()).toBe(now.toISOString());
+ });
+
+ it('never buckets finer than the SLO slices', () => {
+ const from = new Date('2026-10-25T00:00:00Z');
+ const to = new Date('2026-10-25T01:00:00Z');
+ expect(calculateSloInterval(from, to, 1)).toBe(1);
+ expect(calculateSloInterval(from, to, 15)).toBe(15);
+ expect(calculateSloInterval(new Date('2026-07-25T00:00:00Z'), to, 1)).toBe(1440);
+ });
+
+ it('summarizes each kind', () => {
+ expect(
+ sliSummary({
+ sliKind: 'endpoint',
+ bucketMinutes: 1,
+ sliConfig: { endpoint: '', good: 'latency', latencyThresholdMs: 500 }
+ })
+ ).toBe('all endpoints · latency ≤ 500ms');
+ expect(
+ sliSummary({
+ sliKind: 'span',
+ bucketMinutes: 1,
+ sliConfig: {
+ serviceName: 'api',
+ name: '',
+ kind: 2,
+ attributes: [{ key: 'http.route', value: '/x' }],
+ good: 'availability',
+ latencyThresholdMs: 0
+ }
+ })
+ ).toBe('api · Server · http.route=/x · availability');
+ expect(
+ sliSummary({
+ sliKind: 'metric',
+ bucketMinutes: 5,
+ sliConfig: {
+ metricName: 'cpu',
+ tags: {},
+ aggregation: 'avg',
+ operator: 'lt',
+ threshold: 90,
+ missingIsBad: true
+ }
+ })
+ ).toBe('avg(cpu) < 90 · 5m slices');
+ expect(
+ sliSummary({ sliKind: 'monitor', bucketMinutes: 1, sliConfig: { checkIds: [1, 2] } })
+ ).toBe('2 monitors');
+ });
+
+ it('colours any burn that outpaces the window as a warning', () => {
+ expect(burnTone(0.8)).toBe('success');
+ expect(burnTone(1.3)).toBe('warning');
+ expect(burnTone(14.4)).toBe('danger');
+ expect(burnTone(null)).toBe('neutral');
+ });
+
+ it('shows an overspent budget at its real size', () => {
+ expect(formatBudgetRemaining(33.94)).toBe('33.9%');
+ expect(formatBudgetRemaining(-100)).toBe('-100%');
+ expect(formatBudgetRemaining(-9900)).toBe('-9,900%');
+ expect(formatBudgetRemaining(null)).toBe('—');
+ });
+});
diff --git a/frontend/src/lib/utils/slo-math.ts b/frontend/src/lib/utils/slo-math.ts
new file mode 100644
index 00000000..9a642f02
--- /dev/null
+++ b/frontend/src/lib/utils/slo-math.ts
@@ -0,0 +1,226 @@
+import type { SLO, SloSeriesBucket } from '$lib/state/slos.svelte';
+import { spanKindLabel } from '$lib/utils/span-fields';
+
+export type SloTone = 'success' | 'warning' | 'danger' | 'neutral';
+
+export interface ChartPoint {
+ timestamp: Date;
+ value: number;
+}
+
+export interface BurnRatePreset {
+ id: string;
+ label: string;
+ longWindowMinutes: number;
+ shortWindowMinutes: number;
+ burnRateThreshold: number;
+}
+
+export const BURN_RATE_PRESETS: BurnRatePreset[] = [
+ {
+ id: 'fast',
+ label: 'Fast',
+ longWindowMinutes: 60,
+ shortWindowMinutes: 5,
+ burnRateThreshold: 14.4
+ },
+ {
+ id: 'slow',
+ label: 'Slow',
+ longWindowMinutes: 360,
+ shortWindowMinutes: 30,
+ burnRateThreshold: 6
+ },
+ {
+ id: 'long',
+ label: 'Long',
+ longWindowMinutes: 4320,
+ shortWindowMinutes: 360,
+ burnRateThreshold: 1
+ }
+];
+
+export const SLO_BUCKET_RETENTION_DAYS = 120;
+
+const INTERVAL_STEPS = [1, 5, 15, 30, 60, 120, 240, 480, 720, 1440];
+
+export function sliSeries(buckets: SloSeriesBucket[]): ChartPoint[] {
+ return buckets
+ .filter((bucket) => bucket.total > 0)
+ .map((bucket) => ({
+ timestamp: new Date(bucket.bucket),
+ value: (bucket.good / bucket.total) * 100
+ }));
+}
+
+export function errorBudgetSeries(buckets: SloSeriesBucket[], targetPercent: number): ChartPoint[] {
+ const total = buckets.reduce((sum, bucket) => sum + bucket.total, 0);
+ const allowedBad = (1 - targetPercent / 100) * total;
+ if (total === 0 || allowedBad <= 0) return [];
+ let bad = 0;
+ return buckets.map((bucket) => {
+ bad += bucket.total - bucket.good;
+ return { timestamp: new Date(bucket.bucket), value: (bad / allowedBad) * 100 };
+ });
+}
+
+export function budgetTone(remaining: number | null): SloTone {
+ if (remaining === null) return 'neutral';
+ if (remaining < 0) return 'danger';
+ if (remaining < 25) return 'warning';
+ return 'success';
+}
+
+export function burnTone(rate: number | null): SloTone {
+ if (rate === null) return 'neutral';
+ if (rate >= BURN_RATE_PRESETS[0].burnRateThreshold) return 'danger';
+ if (rate >= 1) return 'warning';
+ return 'success';
+}
+
+export function toneTextClass(tone: SloTone): string {
+ switch (tone) {
+ case 'success':
+ return 'text-green-600 dark:text-green-400';
+ case 'warning':
+ return 'text-amber-600 dark:text-amber-400';
+ case 'danger':
+ return 'text-red-600 dark:text-red-400';
+ default:
+ return 'text-muted-foreground';
+ }
+}
+
+export function toneBarClass(tone: SloTone): string {
+ switch (tone) {
+ case 'success':
+ return 'bg-green-500';
+ case 'warning':
+ return 'bg-amber-500';
+ case 'danger':
+ return 'bg-red-500';
+ default:
+ return 'bg-muted-foreground/40';
+ }
+}
+
+function decimalsOf(value: number): number {
+ const text = String(value);
+ const dot = text.indexOf('.');
+ return dot === -1 ? 0 : text.length - dot - 1;
+}
+
+export function formatSloPercent(value: number | null, targetPercent: number): string {
+ if (value === null || Number.isNaN(value)) return '—';
+ const decimals = Math.max(2, decimalsOf(targetPercent));
+ const factor = 10 ** decimals;
+ return `${(Math.floor(value * factor + 1e-6) / factor).toFixed(decimals)}%`;
+}
+
+export function formatBurnRate(rate: number | null): string {
+ if (rate === null) return '—';
+ return `${rate.toFixed(rate >= 10 ? 0 : 1)}x`;
+}
+
+export function defaultPresetForWindow(windowDays: number): string {
+ if (windowDays <= 7) return '7d';
+ if (windowDays <= 30) return '1M';
+ return '3M';
+}
+
+export function clampRangeToWindow(
+ from: Date,
+ to: Date,
+ windowDays: number,
+ now: Date
+): { from: Date; to: Date; clamped: boolean } {
+ const windowMs = windowDays * 24 * 60 * 60 * 1000;
+ let clamped = false;
+ const clampedTo = to.getTime() > now.getTime() ? now : to;
+ let clampedFrom = from;
+ const earliest = new Date(clampedTo.getTime() - windowMs);
+ if (clampedFrom.getTime() < earliest.getTime()) {
+ clampedFrom = earliest;
+ clamped = true;
+ }
+ return { from: clampedFrom, to: clampedTo, clamped };
+}
+
+export function calculateSloInterval(from: Date, to: Date, bucketMinutes: number): number {
+ const raw = (to.getTime() - from.getTime()) / 60000 / 150;
+ for (const step of INTERVAL_STEPS) {
+ if (step >= raw - 0.05 && step >= bucketMinutes) return step;
+ }
+ return 1440;
+}
+
+export function sliChartDomain(points: ChartPoint[], targetPercent: number): [number, number] {
+ let low = targetPercent;
+ for (const point of points) {
+ if (point.value < low) low = point.value;
+ }
+ const pad = Math.max(0.05, (100 - low) * 0.15);
+ return [Math.max(0, low - pad), 100];
+}
+
+export function budgetChartDomain(points: ChartPoint[]): [number, number] {
+ let high = 100;
+ for (const point of points) {
+ if (point.value > high) high = point.value;
+ }
+ return [0, Math.min(high, 250) * 1.05];
+}
+
+function goodLabel(good: string, latencyThresholdMs: number): string {
+ return good === 'latency' ? `latency ≤ ${latencyThresholdMs}ms` : 'availability';
+}
+
+export function sliSummary(slo: Pick<SLO, 'sliKind' | 'sliConfig' | 'bucketMinutes'>): string {
+ const cfg = slo.sliConfig as unknown as Record<string, unknown>;
+ switch (slo.sliKind) {
+ case 'endpoint': {
+ const endpoint = (cfg.endpoint as string) || 'all endpoints';
+ return `${endpoint} · ${goodLabel(cfg.good as string, cfg.latencyThresholdMs as number)}`;
+ }
+ case 'span': {
+ const parts: string[] = [];
+ if (cfg.serviceName) parts.push(cfg.serviceName as string);
+ if (cfg.name) parts.push(`name ~ ${cfg.name as string}`);
+ if (cfg.kind !== null && cfg.kind !== undefined)
+ parts.push(spanKindLabel(cfg.kind as number));
+ const attributes = (cfg.attributes as SpanSliAttributeLike[] | undefined) ?? [];
+ if (attributes.length === 1) parts.push(`${attributes[0].key}=${attributes[0].value}`);
+ else if (attributes.length > 1) parts.push(`${attributes.length} attributes`);
+ if (parts.length === 0) parts.push('all spans');
+ parts.push(goodLabel(cfg.good as string, cfg.latencyThresholdMs as number));
+ return parts.join(' · ');
+ }
+ case 'task':
+ return (cfg.taskName as string) || 'all tasks';
+ case 'monitor': {
+ const ids = (cfg.checkIds as number[] | undefined) ?? [];
+ return ids.length === 1 ? '1 monitor' : `${ids.length} monitors`;
+ }
+ case 'session': {
+ const version = (cfg.appVersion as string) || 'all versions';
+ return `${version} · crash-free sessions${cfg.includeMessages ? ' (messages count)' : ''}`;
+ }
+ case 'metric': {
+ const operator =
+ { gt: '>', gte: '≥', lt: '<', lte: '≤' }[cfg.operator as string] ?? cfg.operator;
+ return `${cfg.aggregation as string}(${cfg.metricName as string}) ${operator} ${cfg.threshold as number} · ${slo.bucketMinutes}m slices`;
+ }
+ }
+ return slo.sliKind;
+}
+
+interface SpanSliAttributeLike {
+ key: string;
+ value: string;
+}
+
+export function formatBudgetRemaining(remaining: number | null): string {
+ if (remaining === null || Number.isNaN(remaining)) return '—';
+ if (remaining <= -100) return `${Math.round(remaining).toLocaleString('en-US')}%`;
+ return `${remaining.toFixed(1)}%`;
+}
diff --git a/frontend/src/routes/notifications/rule-dialog.svelte b/frontend/src/routes/notifications/rule-dialog.svelte
index 3590594d..9c358a6f 100644
--- a/frontend/src/routes/notifications/rule-dialog.svelte
+++ b/frontend/src/routes/notifications/rule-dialog.svelte
@@ -12,6 +12,10 @@
import { projectsState, isFrontendFramework } from '$lib/state/projects.svelte';
import { ruleTypeOptions } from './rule-types';
import type { NotificationChannelConfig, NotificationRuleConfig } from '$lib/types/notifications';
+ import { BURN_RATE_PRESETS } from '$lib/utils/slo-math';
+ import { SLI_KIND_LABELS, type SLO } from '$lib/state/slos.svelte';
+ import { WarningCallout } from '$lib/components/ui/warning-callout';
+ import { resolve } from '$app/paths';
interface NotificationChannel {
id: number;
@@ -85,15 +89,83 @@
let thresholdCost = $state(0.5);
let conversationThresholdCost = $state(5);
let flaggedTermsFilter = $state('');
+ let sloId = $state('');
+ let longWindowMinutes = $state(60);
+ let shortWindowMinutes = $state(5);
+ let burnRateThreshold = $state(14.4);
+ let consumedPercent = $state(80);
+ let slos = $state<SLO[]>([]);
+ let slosLoaded = $state(false);
+ let slosStatus = $state<'loading' | 'loaded' | 'error'>('loading');
const isEditing = $derived(rule !== null);
+ const isSloRule = $derived(ruleType === 'slo_burn_rate' || ruleType === 'slo_budget_consumed');
+ const burnPreset = $derived(
+ BURN_RATE_PRESETS.find(
+ (p) =>
+ p.longWindowMinutes === longWindowMinutes &&
+ p.shortWindowMinutes === shortWindowMinutes &&
+ p.burnRateThreshold === burnRateThreshold
+ )?.id ?? 'custom'
+ );
+
+ const selectedSlo = $derived(slos.find((s) => String(s.id) === sloId));
+ const sloSliceMinutes = $derived(selectedSlo?.bucketMinutes ?? 1);
+ const roundsBurnWindows = $derived(
+ sloSliceMinutes > 1 &&
+ (shortWindowMinutes % sloSliceMinutes !== 0 || longWindowMinutes % sloSliceMinutes !== 0)
+ );
+
+ function roundUpToSlice(minutes: number): number {
+ return Math.ceil(minutes / sloSliceMinutes) * sloSliceMinutes;
+ }
+
+ async function loadSlos() {
+ slosStatus = 'loading';
+ try {
+ const response = await api.get('/slos', {
+ projectId: projectsState.currentProjectId ?? undefined
+ });
+ slos = response.slos ?? [];
+ slosStatus = 'loaded';
+ } catch {
+ slos = [];
+ slosStatus = 'error';
+ }
+ }
+
+ $effect(() => {
+ if (open && isSloRule && !slosLoaded) {
+ slosLoaded = true;
+ loadSlos();
+ }
+ if (!open) slosLoaded = false;
+ });
+
+ function windowLabel(minutes: number): string {
+ if (minutes >= 1440) return `${minutes / 1440}d`;
+ if (minutes >= 60) return `${minutes / 60}h`;
+ return `${minutes}m`;
+ }
+
+ function applyBurnPreset(id: string) {
+ const preset = BURN_RATE_PRESETS.find((p) => p.id === id);
+ if (!preset) return;
+ longWindowMinutes = preset.longWindowMinutes;
+ shortWindowMinutes = preset.shortWindowMinutes;
+ burnRateThreshold = preset.burnRateThreshold;
+ }
+
const isFrontendProject = $derived(
!!projectsState.currentProject && isFrontendFramework(projectsState.currentProject.framework)
);
+ const frontendRuleTypes = new Set(['new_error', 'slo_burn_rate', 'slo_budget_consumed']);
const visibleRuleTypes = $derived(
- isFrontendProject ? [{ value: 'new_error', label: 'New Issue' }] : ruleTypeOptions
+ isFrontendProject
+ ? ruleTypeOptions.filter((o) => frontendRuleTypes.has(o.value))
+ : ruleTypeOptions
);
$effect(() => {
@@ -120,6 +192,10 @@
error_regression: 'Fires when an error that was previously archived (resolved) occurs again.',
check_down:
'Fires when a monitor transitions to down (after its failure threshold), and sends a recovery notice when it comes back up. Applies to every monitor in this project.',
+ slo_burn_rate:
+ 'Fires when an SLO spends its error budget faster than the threshold over both the long and the short window (multi-window burn rate), and sends a recovery notice once the burn slows down.',
+ slo_budget_consumed:
+ "Fires when an SLO has used more than the configured share of its window's error budget, and sends a recovery notice when the budget recovers.",
error_rate_threshold:
'Fires when the percentage of requests returning 5xx across all endpoints exceeds the threshold within the lookback window.',
error_count_threshold:
@@ -197,6 +273,11 @@
ignorePatterns = '';
traceName = '*';
thresholdCost = 0.5;
+ sloId = '';
+ longWindowMinutes = 60;
+ shortWindowMinutes = 5;
+ burnRateThreshold = 14.4;
+ consumedPercent = 80;
}
function populateFromRule(r: NotificationRule) {
@@ -279,6 +360,16 @@
case 'ai_flagged_content':
flaggedTermsFilter = (cfg.terms || []).join(', ');
break;
+ case 'slo_burn_rate':
+ sloId = cfg.sloId ? String(cfg.sloId) : '';
+ longWindowMinutes = cfg.longWindowMinutes ?? 60;
+ shortWindowMinutes = cfg.shortWindowMinutes ?? 5;
+ burnRateThreshold = cfg.burnRateThreshold ?? 14.4;
+ break;
+ case 'slo_budget_consumed':
+ sloId = cfg.sloId ? String(cfg.sloId) : '';
+ consumedPercent = cfg.consumedPercent ?? 80;
+ break;
}
}
@@ -327,6 +418,15 @@
.filter((t) => t);
return { terms };
}
+ case 'slo_burn_rate':
+ return {
+ sloId: parseInt(sloId) || 0,
+ longWindowMinutes,
+ shortWindowMinutes,
+ burnRateThreshold
+ };
+ case 'slo_budget_consumed':
+ return { sloId: parseInt(sloId) || 0, consumedPercent };
default:
return {};
}
@@ -583,6 +683,105 @@
<Label for="cfg-lookback-td">Lookback (minutes)</Label>
<Input id="cfg-lookback-td" type="number" bind:value={lookbackMinutes} min="1" />
</div>
+ {:else if ruleType === 'slo_burn_rate' || ruleType === 'slo_budget_consumed'}
+ <div class="space-y-2">
+ <Label>SLO</Label>
+ <Select.Root type="single" bind:value={sloId}>
+ <Select.Trigger class="w-full">
+ {slos.find((s) => String(s.id) === sloId)?.name || 'Select an SLO'}
+ </Select.Trigger>
+ <Select.Content class="max-h-64 overflow-y-auto">
+ {#each slos as s (s.id)}
+ <Select.Item value={String(s.id)} label={s.name}>
+ {s.name}
+ <span class="ml-2 text-xs text-muted-foreground">
+ {SLI_KIND_LABELS[s.sliKind]} · {s.targetPercent}% / {s.windowDays}d
+ </span>
+ </Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ {#if slosStatus === 'error'}
+ <p class="text-xs text-destructive">
+ The SLOs could not be loaded.
+ <button
+ type="button"
+ class="underline underline-offset-2"
+ onclick={() => loadSlos()}
+ >
+ Retry
+ </button>
+ </p>
+ {:else if slosStatus === 'loaded' && slos.length === 0}
+ <p class="text-xs text-muted-foreground">
+ No SLOs in this project yet. Create one on the
+ <a class="text-primary underline-offset-2 hover:underline" href={resolve('/slos')}
+ >SLOs page</a
+ >.
+ </p>
+ {/if}
+ </div>
+ {#if ruleType === 'slo_burn_rate'}
+ <div class="space-y-2">
+ <Label>Preset</Label>
+ <div class="flex flex-wrap gap-2">
+ {#each BURN_RATE_PRESETS as preset (preset.id)}
+ <Button
+ type="button"
+ size="sm"
+ variant={burnPreset === preset.id ? 'secondary' : 'outline'}
+ onclick={() => applyBurnPreset(preset.id)}
+ >
+ {preset.label}
+ {preset.burnRateThreshold}x · {windowLabel(preset.longWindowMinutes)} / {windowLabel(
+ preset.shortWindowMinutes
+ )}
+ </Button>
+ {/each}
+ </div>
+ </div>
+ <div class="space-y-2">
+ <Label for="cfg-burn-long">Long window (minutes)</Label>
+ <Input id="cfg-burn-long" type="number" bind:value={longWindowMinutes} min="5" />
+ </div>
+ <div class="space-y-2">
+ <Label for="cfg-burn-short">Short window (minutes)</Label>
+ <Input id="cfg-burn-short" type="number" bind:value={shortWindowMinutes} min="1" />
+ </div>
+ <div class="space-y-2">
+ <Label for="cfg-burn-threshold">Burn rate threshold (x)</Label>
+ <Input
+ id="cfg-burn-threshold"
+ type="number"
+ bind:value={burnRateThreshold}
+ step="0.1"
+ min="0.1"
+ />
+ <p class="text-xs text-muted-foreground">
+ Fires when both windows burn at or above this multiple of the rate that spends the
+ budget exactly over the SLO window.
+ </p>
+ </div>
+ {#if roundsBurnWindows}
+ <WarningCallout title="Windows are rounded up to whole slices">
+ This SLO is measured in {sloSliceMinutes}-minute slices, so the rule reads the last
+ {windowLabel(roundUpToSlice(shortWindowMinutes))} and {windowLabel(
+ roundUpToSlice(longWindowMinutes)
+ )}.
+ </WarningCallout>
+ {/if}
+ {:else}
+ <div class="space-y-2">
+ <Label for="cfg-budget-consumed">Budget consumed (%)</Label>
+ <Input
+ id="cfg-budget-consumed"
+ type="number"
+ bind:value={consumedPercent}
+ min="1"
+ max="100"
+ />
+ </div>
+ {/if}
{:else if ruleType === 'task_failure_rate'}
<div class="space-y-2">
<Label for="cfg-task-f">Task Name (* for all)</Label>
diff --git a/frontend/src/routes/notifications/rule-types.ts b/frontend/src/routes/notifications/rule-types.ts
index ef17417b..2a581740 100644
--- a/frontend/src/routes/notifications/rule-types.ts
+++ b/frontend/src/routes/notifications/rule-types.ts
@@ -18,7 +18,9 @@ export const ruleTypeOptions = [
{ value: 'ai_trace_cost', label: 'AI Trace Cost' },
{ value: 'ai_conversation_cost', label: 'AI Conversation Cost' },
{ value: 'ai_flagged_content', label: 'AI Flagged Content' },
- { value: 'check_down', label: 'Monitor Down' }
+ { value: 'check_down', label: 'Monitor Down' },
+ { value: 'slo_burn_rate', label: 'SLO Burn Rate' },
+ { value: 'slo_budget_consumed', label: 'SLO Budget Consumed' }
];
export const ruleTypeLabels: Record<string, string> = Object.fromEntries(
diff --git a/frontend/src/routes/slos/+page.svelte b/frontend/src/routes/slos/+page.svelte
new file mode 100644
index 00000000..782960d8
--- /dev/null
+++ b/frontend/src/routes/slos/+page.svelte
@@ -0,0 +1,373 @@
+<script lang="ts">
+ import { getErrorMessage } from '$lib/utils/errors';
+ import { onMount, onDestroy } from 'svelte';
+ import { api } from '$lib/api';
+ import * as Table from '$lib/components/ui/table';
+ import { Button } from '$lib/components/ui/button';
+ import { LoadingCircle } from '$lib/components/ui/loading-circle';
+ import { ErrorDisplay } from '$lib/components/ui/error-display';
+ import { TracewayTableHeader } from '$lib/components/ui/traceway-table-header';
+ import { TableEmptyState } from '$lib/components/ui/table-empty-state';
+ import { SearchBar } from '$lib/components/ui/search-bar';
+ import InfoCallout from '$lib/components/traceway/info-callout.svelte';
+ import EmptyState from '$lib/components/traceway/empty-state.svelte';
+ import TableContainer from '$lib/components/traceway/table-container.svelte';
+ import PageHeader from '$lib/components/traceway/page-header.svelte';
+ import SloKindBadge from '$lib/components/slos/slo-kind-badge.svelte';
+ import SloStatusBadge from '$lib/components/slos/slo-status-badge.svelte';
+ import SloSheet from './slo-sheet.svelte';
+ import { browser } from '$app/environment';
+ import { projectsState, isFrontendFramework } from '$lib/state/projects.svelte';
+ import {
+ slosState,
+ SLI_KIND_LABELS,
+ BACKEND_SLI_KINDS,
+ FRONTEND_SLI_KINDS,
+ type SLO
+ } from '$lib/state/slos.svelte';
+ import { createRowClickHandler } from '$lib/utils/navigation';
+ import { resolve } from '$app/paths';
+ import { updateUrl } from '$lib/utils/url-params';
+ import {
+ budgetTone,
+ burnTone,
+ formatBudgetRemaining,
+ formatBurnRate,
+ formatSloPercent,
+ sliSummary,
+ toneBarClass,
+ toneTextClass
+ } from '$lib/utils/slo-math';
+ import {
+ getSortState,
+ setSortState,
+ handleSortClick,
+ type SortDirection
+ } from '$lib/utils/sort-storage';
+ import { Plus } from '@lucide/svelte';
+
+ type SortField =
+ | 'name'
+ | 'kind'
+ | 'target'
+ | 'window'
+ | 'current_sli'
+ | 'budget'
+ | 'burn_1h'
+ | 'status';
+
+ let slos = $state<SLO[]>([]);
+ let loading = $state(true);
+ let error = $state('');
+ let sheetOpen = $state(false);
+
+ const isFrontendProject = $derived(
+ !!projectsState.currentProject && isFrontendFramework(projectsState.currentProject.framework)
+ );
+ const kindOptions = $derived([
+ { value: '', label: 'All kinds' },
+ ...(isFrontendProject ? FRONTEND_SLI_KINDS : BACKEND_SLI_KINDS).map((kind) => ({
+ value: kind,
+ label: SLI_KIND_LABELS[kind]
+ }))
+ ]);
+ let kindFilter = $state('');
+
+ function parseSearchFromUrl(): string {
+ if (!browser) return '';
+ return new URLSearchParams(window.location.search).get('search') || '';
+ }
+
+ let searchQuery = $state(parseSearchFromUrl());
+
+ function updateSearchUrl(pushToHistory = true) {
+ updateUrl({ search: searchQuery.trim() || null }, { pushToHistory });
+ }
+
+ function handlePopState() {
+ searchQuery = parseSearchFromUrl();
+ loadData(false);
+ }
+
+ const SORT_STORAGE_KEY = 'slos';
+ const initialSort = getSortState(SORT_STORAGE_KEY, { field: 'status', direction: 'asc' });
+ let orderBy = $state<SortField>(initialSort.field as SortField);
+ let sortDirection = $state<SortDirection>(initialSort.direction);
+
+ async function loadData(pushToHistory = true) {
+ loading = true;
+ error = '';
+ updateSearchUrl(pushToHistory);
+ try {
+ const response = await api.get('/slos', {
+ projectId: projectsState.currentProjectId ?? undefined
+ });
+ slos = response.slos || [];
+ slosState.refreshBreachedCount();
+ } catch (e) {
+ console.error(e);
+ error = getErrorMessage(e) || 'Failed to load SLOs';
+ } finally {
+ loading = false;
+ }
+ }
+
+ function handleSort(field: SortField) {
+ const newSort = handleSortClick(field, orderBy, sortDirection);
+ orderBy = newSort.field as SortField;
+ sortDirection = newSort.direction;
+ setSortState(SORT_STORAGE_KEY, newSort);
+ }
+
+ function handleSearch() {
+ updateSearchUrl(true);
+ }
+
+ const statusRank: Record<string, number> = { breached: 0, no_data: 1, ok: 2 };
+
+ function numeric(value: number | null): number {
+ return value === null ? Number.NEGATIVE_INFINITY : value;
+ }
+
+ const visibleSlos = $derived.by(() => {
+ const query = searchQuery.trim().toLowerCase();
+ let filtered = kindFilter ? slos.filter((s) => s.sliKind === kindFilter) : slos;
+ if (query) {
+ filtered = filtered.filter(
+ (s) =>
+ s.name.toLowerCase().includes(query) ||
+ s.description.toLowerCase().includes(query) ||
+ sliSummary(s).toLowerCase().includes(query)
+ );
+ }
+ const direction = sortDirection === 'asc' ? 1 : -1;
+ return [...filtered].sort((a, b) => {
+ switch (orderBy) {
+ case 'kind':
+ return direction * a.sliKind.localeCompare(b.sliKind);
+ case 'target':
+ return direction * (a.targetPercent - b.targetPercent);
+ case 'window':
+ return direction * (a.windowDays - b.windowDays);
+ case 'current_sli':
+ return direction * (numeric(a.currentSli) - numeric(b.currentSli));
+ case 'budget':
+ return direction * (numeric(a.budgetRemaining) - numeric(b.budgetRemaining));
+ case 'burn_1h':
+ return direction * (numeric(a.burnRate1h) - numeric(b.burnRate1h));
+ case 'status': {
+ const rankA = a.enabled ? statusRank[a.status] : 3;
+ const rankB = b.enabled ? statusRank[b.status] : 3;
+ return direction * (rankA - rankB) || a.name.localeCompare(b.name);
+ }
+ default:
+ return direction * a.name.localeCompare(b.name);
+ }
+ });
+ });
+
+ onMount(() => {
+ window.addEventListener('popstate', handlePopState);
+ loadData(false);
+ });
+
+ onDestroy(() => {
+ if (typeof window !== 'undefined') {
+ window.removeEventListener('popstate', handlePopState);
+ }
+ });
+</script>
+
+<div class="space-y-4">
+ <PageHeader title="SLOs">
+ {#snippet actions()}
+ {#if projectsState.canWriteCurrentProject}
+ <Button variant="success" size="sm" onclick={() => (sheetOpen = true)}>
+ <Plus class="mr-2 h-4 w-4" />
+ New SLO
+ </Button>
+ {/if}
+ {/snippet}
+ </PageHeader>
+
+ <InfoCallout>
+ A service level objective holds an indicator, the share of good requests, spans, task runs,
+ probes, metric slices or crash-free sessions, to a target over a rolling window. The error
+ budget is what the target leaves over, and the burn rate says how fast it is being spent.
+ </InfoCallout>
+
+ <SearchBar
+ placeholder="Search SLOs..."
+ bind:value={searchQuery}
+ bind:typeValue={kindFilter}
+ typeOptions={kindOptions}
+ onSearch={handleSearch}
+ disabled={loading}
+ />
+
+ {#if loading}
+ <div class="flex h-48 items-center justify-center">
+ <LoadingCircle size="xlg" />
+ </div>
+ {:else if error}
+ <ErrorDisplay status={400} title="Error" description={error} onRetry={() => loadData(false)} />
+ {:else if slos.length === 0}
+ {#if projectsState.canWriteCurrentProject}
+ <EmptyState
+ message={isFrontendProject
+ ? 'No SLOs yet. Define a crash-free sessions target to start tracking its error budget.'
+ : 'No SLOs yet. Define a target for an endpoint, span, task, monitor or metric to start tracking its error budget.'}
+ actionLabel="Create your first SLO"
+ onAction={() => (sheetOpen = true)}
+ />
+ {:else}
+ <EmptyState message="No SLOs yet." />
+ {/if}
+ {:else}
+ <TableContainer minWidth="960px" empty={visibleSlos.length === 0}>
+ <Table.Root>
+ {#if visibleSlos.length === 0}
+ <Table.Body>
+ <TableEmptyState colspan={8} message="No SLOs match your search." />
+ </Table.Body>
+ {:else}
+ <Table.Header>
+ <Table.Row>
+ <TracewayTableHeader
+ label="Name"
+ sortField="name"
+ currentSortField={orderBy}
+ {sortDirection}
+ onSort={(field) => handleSort(field as SortField)}
+ />
+ <TracewayTableHeader
+ label="SLI"
+ sortField="kind"
+ currentSortField={orderBy}
+ {sortDirection}
+ onSort={(field) => handleSort(field as SortField)}
+ class="w-[110px]"
+ />
+ <TracewayTableHeader
+ label="Target"
+ sortField="target"
+ currentSortField={orderBy}
+ {sortDirection}
+ onSort={(field) => handleSort(field as SortField)}
+ class="w-[90px]"
+ />
+ <TracewayTableHeader
+ label="Window"
+ sortField="window"
+ currentSortField={orderBy}
+ {sortDirection}
+ onSort={(field) => handleSort(field as SortField)}
+ class="w-[90px]"
+ />
+ <TracewayTableHeader
+ label="Current SLI"
+ tooltip="Share of good events over the whole window"
+ sortField="current_sli"
+ currentSortField={orderBy}
+ {sortDirection}
+ onSort={(field) => handleSort(field as SortField)}
+ class="w-[120px]"
+ />
+ <TracewayTableHeader
+ label="Budget left"
+ tooltip="Share of the error budget not yet spent; negative once the target is missed"
+ sortField="budget"
+ currentSortField={orderBy}
+ {sortDirection}
+ onSort={(field) => handleSort(field as SortField)}
+ class="w-[170px]"
+ />
+ <TracewayTableHeader
+ label="Burn 1h"
+ tooltip="How fast the budget burned over the last hour; 1x spends it exactly over the window"
+ sortField="burn_1h"
+ currentSortField={orderBy}
+ {sortDirection}
+ onSort={(field) => handleSort(field as SortField)}
+ class="w-[100px]"
+ />
+ <TracewayTableHeader
+ label="Status"
+ sortField="status"
+ currentSortField={orderBy}
+ {sortDirection}
+ onSort={(field) => handleSort(field as SortField)}
+ align="right"
+ class="w-[120px]"
+ />
+ </Table.Row>
+ </Table.Header>
+ <Table.Body>
+ {#each visibleSlos as slo (slo.id)}
+ {@const tone = budgetTone(slo.budgetRemaining)}
+ <Table.Row
+ class="cursor-pointer"
+ onclick={createRowClickHandler(resolve(`/slos/${slo.id}`))}
+ >
+ <Table.Cell class="font-medium whitespace-normal">
+ {slo.name}
+ {#if slo.description}
+ <div
+ class="mt-0.5 max-w-[420px] truncate text-xs font-normal text-muted-foreground"
+ >
+ {slo.description}
+ </div>
+ {/if}
+ <div
+ class="mt-0.5 max-w-[420px] truncate font-mono text-xs font-normal text-muted-foreground"
+ >
+ {sliSummary(slo)}
+ </div>
+ </Table.Cell>
+ <Table.Cell>
+ <SloKindBadge kind={slo.sliKind} />
+ </Table.Cell>
+ <Table.Cell class="font-mono text-sm tabular-nums">{slo.targetPercent}%</Table.Cell>
+ <Table.Cell class="text-sm text-muted-foreground">{slo.windowDays}d</Table.Cell>
+ <Table.Cell class="font-mono text-sm tabular-nums {toneTextClass(tone)}">
+ {formatSloPercent(slo.currentSli, slo.targetPercent)}
+ </Table.Cell>
+ <Table.Cell>
+ {#if slo.budgetRemaining === null}
+ <span class="text-muted-foreground">—</span>
+ {:else}
+ <div class="flex items-center gap-2">
+ <span
+ class="w-16 shrink-0 font-mono text-sm tabular-nums {toneTextClass(tone)}"
+ >
+ {formatBudgetRemaining(slo.budgetRemaining)}
+ </span>
+ <div class="h-1.5 w-14 shrink-0 overflow-hidden rounded-full bg-muted">
+ {#if slo.budgetRemaining > 0}
+ <div
+ class="h-full rounded-full {toneBarClass(tone)}"
+ style="width: {Math.max(Math.min(slo.budgetRemaining, 100), 4)}%"
+ ></div>
+ {/if}
+ </div>
+ </div>
+ {/if}
+ </Table.Cell>
+ <Table.Cell
+ class="font-mono text-sm tabular-nums {toneTextClass(burnTone(slo.burnRate1h))}"
+ >
+ {formatBurnRate(slo.burnRate1h)}
+ </Table.Cell>
+ <Table.Cell class="text-right">
+ <SloStatusBadge status={slo.status} enabled={slo.enabled} />
+ </Table.Cell>
+ </Table.Row>
+ {/each}
+ </Table.Body>
+ {/if}
+ </Table.Root>
+ </TableContainer>
+ {/if}
+</div>
+
+<SloSheet bind:open={sheetOpen} slo={null} onSaved={() => loadData(false)} />
diff --git a/frontend/src/routes/slos/+page.ts b/frontend/src/routes/slos/+page.ts
new file mode 100644
index 00000000..5b66ce85
--- /dev/null
+++ b/frontend/src/routes/slos/+page.ts
@@ -0,0 +1,7 @@
+import { redirect } from '@sveltejs/kit';
+import { authState } from '$lib/state/auth.svelte';
+import type { PageLoad } from './$types';
+
+export const load: PageLoad = () => {
+ if (!authState.isAuthenticated) throw redirect(302, '/login');
+};
diff --git a/frontend/src/routes/slos/[id]/+page.svelte b/frontend/src/routes/slos/[id]/+page.svelte
new file mode 100644
index 00000000..d0c1e034
--- /dev/null
+++ b/frontend/src/routes/slos/[id]/+page.svelte
@@ -0,0 +1,516 @@
+<script lang="ts">
+ import { getErrorMessage, getErrorStatus } from '$lib/utils/errors';
+ import { onMount, onDestroy } from 'svelte';
+ import { api } from '$lib/api';
+ import {
+ formatDateTime,
+ formatRelativeTimeAgo,
+ toUTCISO,
+ calendarDateTimeToLuxon
+ } from '$lib/utils/formatters';
+ import * as Card from '$lib/components/ui/card';
+ import { Badge } from '$lib/components/ui/badge';
+ import { Button } from '$lib/components/ui/button';
+ import { LoadingCircle } from '$lib/components/ui/loading-circle';
+ import { ErrorDisplay } from '$lib/components/ui/error-display';
+ import { TimeRangePicker } from '$lib/components/ui/time-range-picker';
+ import SloKindBadge from '$lib/components/slos/slo-kind-badge.svelte';
+ import SloStatusBadge from '$lib/components/slos/slo-status-badge.svelte';
+ import SloSheet from '../slo-sheet.svelte';
+ import D3LineChart from '$lib/components/dashboard/d3-line-chart.svelte';
+ import PageHeader from '$lib/components/traceway/page-header.svelte';
+ import StatRow from '$lib/components/traceway/stat-row.svelte';
+ import StatTile from '$lib/components/traceway/stat-tile.svelte';
+ import StatusPill from '$lib/components/traceway/status-pill.svelte';
+ import ChartCard from '$lib/components/traceway/chart-card.svelte';
+ import InfoCallout from '$lib/components/traceway/info-callout.svelte';
+ import ConfirmDeleteDialog from '$lib/components/traceway/confirm-delete-dialog.svelte';
+ import { goto } from '$app/navigation';
+ import { resolve } from '$app/paths';
+ import { CalendarDate } from '@internationalized/date';
+ import { projectsState } from '$lib/state/projects.svelte';
+ import { getTimezone } from '$lib/state/timezone.svelte';
+ import {
+ slosState,
+ type SLO,
+ type SloAlertRule,
+ type SloSeriesBucket
+ } from '$lib/state/slos.svelte';
+ import { createSmartBackHandler } from '$lib/utils/back-navigation';
+ import {
+ parseTimeRangeFromUrl,
+ getResolvedTimeRange,
+ dateToCalendarDate,
+ dateToTimeString,
+ updateUrl
+ } from '$lib/utils/url-params';
+ import {
+ budgetTone,
+ burnTone,
+ calculateSloInterval,
+ clampRangeToWindow,
+ defaultPresetForWindow,
+ errorBudgetSeries,
+ formatBudgetRemaining,
+ formatBurnRate,
+ formatSloPercent,
+ SLO_BUCKET_RETENTION_DAYS,
+ sliChartDomain,
+ budgetChartDomain,
+ sliSeries,
+ sliSummary,
+ toneTextClass
+ } from '$lib/utils/slo-math';
+ import { badEventsHref, badEventsLabel } from '$lib/utils/slo-links';
+ import { ruleTypeLabels } from '../../notifications/rule-types';
+ import { Bell, ChartLine, ExternalLink, Pencil, PiggyBank, Trash2 } from '@lucide/svelte';
+ import { toast } from 'svelte-sonner';
+
+ let { data } = $props();
+
+ // svelte-ignore state_referenced_locally
+ const sloId = data.id;
+ // svelte-ignore state_referenced_locally
+ const urlHasRange = !!(data.preset || data.from || data.to);
+ const initialTimezone = getTimezone();
+ const timezone = $derived(getTimezone());
+ const initialUrlParams = parseTimeRangeFromUrl(initialTimezone, '1M');
+ const initialRange = getResolvedTimeRange(initialUrlParams, initialTimezone);
+
+ let slo = $state<SLO | null>(null);
+ let rules = $state<SloAlertRule[]>([]);
+ let loading = $state(true);
+ let error = $state('');
+ let notFound = $state(false);
+
+ let buckets = $state<SloSeriesBucket[]>([]);
+ let seriesLoading = $state(true);
+ let seriesError = $state('');
+ let seriesSequence = 0;
+ let rangeFrom = $state(initialRange.from);
+ let rangeTo = $state(initialRange.to);
+
+ let selectedPreset = $state<string | null>(initialUrlParams.preset);
+ let fromDate = $state<CalendarDate>(dateToCalendarDate(initialRange.from, initialTimezone));
+ let toDate = $state<CalendarDate>(dateToCalendarDate(initialRange.to, initialTimezone));
+ let fromTime = $state(dateToTimeString(initialRange.from, initialTimezone));
+ let toTime = $state(dateToTimeString(initialRange.to, initialTimezone));
+
+ let editOpen = $state(false);
+ let deleteOpen = $state(false);
+ let deleting = $state(false);
+
+ function getFromDateTimeUTC(): string {
+ const [hour, minute] = (fromTime || '00:00').split(':').map(Number);
+ return toUTCISO(
+ calendarDateTimeToLuxon(
+ { year: fromDate.year, month: fromDate.month, day: fromDate.day, hour, minute },
+ timezone
+ )
+ );
+ }
+
+ function getToDateTimeUTC(): string {
+ const [hour, minute] = (toTime || '23:59').split(':').map(Number);
+ return toUTCISO(
+ calendarDateTimeToLuxon(
+ { year: toDate.year, month: toDate.month, day: toDate.day, hour, minute },
+ timezone
+ ).endOf('minute')
+ );
+ }
+
+ function applyRange(from: Date, to: Date) {
+ fromDate = dateToCalendarDate(from, timezone);
+ fromTime = dateToTimeString(from, timezone);
+ toDate = dateToCalendarDate(to, timezone);
+ toTime = dateToTimeString(to, timezone);
+ }
+
+ function resolveRange(): { from: Date; to: Date } {
+ if (selectedPreset) {
+ return getResolvedTimeRange({ preset: selectedPreset, from: null, to: null }, timezone);
+ }
+ return { from: new Date(getFromDateTimeUTC()), to: new Date(getToDateTimeUTC()) };
+ }
+
+ async function loadSlo() {
+ try {
+ const response = await api.get(`/slos/${sloId}`, {
+ projectId: projectsState.currentProjectId ?? undefined
+ });
+ slo = response.slo;
+ rules = response.rules || [];
+ } catch (e) {
+ if (getErrorStatus(e) === 404) {
+ notFound = true;
+ } else {
+ error = getErrorMessage(e) || 'Failed to load the SLO';
+ }
+ }
+ }
+
+ async function loadSeries(pushToHistory: boolean, announceClamp: boolean) {
+ if (!slo) return;
+ const sequence = ++seriesSequence;
+ seriesLoading = true;
+ const picked = resolveRange();
+ let clamped = clampRangeToWindow(picked.from, picked.to, slo.windowDays, new Date());
+ if (clamped.clamped) {
+ if (selectedPreset) {
+ selectedPreset = defaultPresetForWindow(slo.windowDays);
+ clamped = { ...resolveRange(), clamped: true };
+ }
+ applyRange(clamped.from, clamped.to);
+ if (announceClamp) {
+ toast.info(`Range limited to the SLO's ${slo.windowDays}-day window`);
+ }
+ }
+ rangeFrom = clamped.from;
+ rangeTo = clamped.to;
+ updateUrl(
+ selectedPreset
+ ? { preset: selectedPreset, from: null, to: null }
+ : { preset: null, from: rangeFrom.toISOString(), to: rangeTo.toISOString() },
+ { pushToHistory }
+ );
+ try {
+ const response = await api.post(
+ `/slos/${sloId}/series`,
+ {
+ fromDate: rangeFrom.toISOString(),
+ toDate: rangeTo.toISOString(),
+ intervalMinutes: calculateSloInterval(rangeFrom, rangeTo, slo.bucketMinutes)
+ },
+ { projectId: projectsState.currentProjectId ?? undefined }
+ );
+ if (sequence !== seriesSequence) return;
+ buckets = response.buckets || [];
+ seriesError = '';
+ } catch (e) {
+ if (sequence !== seriesSequence) return;
+ buckets = [];
+ seriesError = getErrorMessage(e) || 'Failed to load the history of this SLO';
+ } finally {
+ if (sequence === seriesSequence) seriesLoading = false;
+ }
+ }
+
+ async function reloadAfterSave() {
+ await loadSlo();
+ if (slo) loadSeries(false, false);
+ }
+
+ async function loadAll() {
+ loading = true;
+ error = '';
+ notFound = false;
+ await loadSlo();
+ loading = false;
+ if (slo && !urlHasRange && selectedPreset === initialUrlParams.preset) {
+ selectedPreset = defaultPresetForWindow(slo.windowDays);
+ }
+ if (!notFound && !error) {
+ loadSeries(false, false);
+ }
+ }
+
+ function handleTimeRangeChange(
+ from: { date: CalendarDate; time: string },
+ to: { date: CalendarDate; time: string },
+ preset: string | null
+ ) {
+ fromDate = from.date;
+ fromTime = from.time;
+ toDate = to.date;
+ toTime = to.time;
+ selectedPreset = preset;
+ loadSeries(true, true);
+ }
+
+ function handlePopState() {
+ const urlParams = parseTimeRangeFromUrl(
+ timezone,
+ slo ? defaultPresetForWindow(slo.windowDays) : '1M'
+ );
+ const range = getResolvedTimeRange(urlParams, timezone);
+ selectedPreset = urlParams.preset;
+ applyRange(range.from, range.to);
+ loadSeries(false, false);
+ }
+
+ async function deleteSlo() {
+ if (!slo) return;
+ deleting = true;
+ try {
+ await api.delete(`/slos/${sloId}`, {
+ projectId: projectsState.currentProjectId ?? undefined
+ });
+ toast.success('Successfully deleted the SLO');
+ slosState.refreshBreachedCount();
+ goto(resolve('/slos'));
+ } catch (e) {
+ if (getErrorStatus(e) !== 403) {
+ toast.error(getErrorMessage(e) || 'Failed to delete the SLO');
+ }
+ } finally {
+ deleting = false;
+ deleteOpen = false;
+ }
+ }
+
+ const sliPoints = $derived(sliSeries(buckets));
+ const budgetPoints = $derived(slo ? errorBudgetSeries(buckets, slo.targetPercent) : []);
+ const sliChartSeries = $derived([{ key: 'SLI', color: 'var(--chart-1)', data: sliPoints }]);
+ const budgetChartSeries = $derived([
+ { key: 'Budget consumed', color: 'var(--chart-2)', data: budgetPoints }
+ ]);
+ const coverage = $derived.by(() => {
+ if (!slo?.historyFrom) return null;
+ const historyFrom = new Date(
+ Math.max(
+ new Date(slo.historyFrom).getTime(),
+ Date.now() - SLO_BUCKET_RETENTION_DAYS * 86400000
+ )
+ );
+ if (historyFrom.getTime() <= rangeFrom.getTime()) return null;
+ const coveredDays = Math.max(0, (rangeTo.getTime() - historyFrom.getTime()) / 86400000);
+ const rangeDays = (rangeTo.getTime() - rangeFrom.getTime()) / 86400000;
+ return { historyFrom, coveredDays, rangeDays };
+ });
+
+ function formatDays(days: number): string {
+ return days >= 10 ? days.toFixed(0) : days.toFixed(1);
+ }
+
+ function ruleStateLabel(rule: SloAlertRule): {
+ tone: 'success' | 'warning' | 'neutral';
+ label: string;
+ } {
+ if (!rule.enabled) return { tone: 'neutral', label: 'Off' };
+ if (rule.snoozedUntil && new Date(rule.snoozedUntil).getTime() > Date.now()) {
+ return { tone: 'warning', label: 'Snoozed' };
+ }
+ return { tone: 'success', label: 'On' };
+ }
+
+ onMount(() => {
+ window.addEventListener('popstate', handlePopState);
+ loadAll();
+ });
+
+ onDestroy(() => {
+ if (typeof window !== 'undefined') {
+ window.removeEventListener('popstate', handlePopState);
+ }
+ });
+</script>
+
+{#if loading}
+ <div class="flex h-64 items-center justify-center">
+ <LoadingCircle size="xlg" />
+ </div>
+{:else if notFound}
+ <ErrorDisplay
+ status={404}
+ title="SLO not found"
+ description="This SLO does not exist in the current project."
+ onBack={createSmartBackHandler({ fallbackPath: '/slos' })}
+ backLabel="Back to SLOs"
+ />
+{:else if error || !slo}
+ <ErrorDisplay
+ status={400}
+ title="Error"
+ description={error || 'Failed to load the SLO'}
+ onRetry={() => loadAll()}
+ />
+{:else}
+ {@const budget = budgetTone(slo.budgetRemaining)}
+ <div class="space-y-4">
+ <PageHeader
+ title={slo.name}
+ description={slo.description || undefined}
+ onBack={createSmartBackHandler({ fallbackPath: '/slos' })}
+ >
+ {#snippet meta()}
+ {#if slo}
+ <SloStatusBadge status={slo.status} enabled={slo.enabled} />
+ <SloKindBadge kind={slo.sliKind} />
+ <span class="text-sm text-muted-foreground">Target {slo.targetPercent}%</span>
+ <span class="text-sm text-muted-foreground">{slo.windowDays}-day window</span>
+ <span class="min-w-0 font-mono text-sm break-all text-muted-foreground"
+ >{sliSummary(slo)}</span
+ >
+ {#if slo.computedAt}
+ <span class="text-sm text-muted-foreground">
+ updated {formatRelativeTimeAgo(slo.computedAt)}
+ </span>
+ {/if}
+ {/if}
+ {/snippet}
+ {#snippet actions()}
+ <TimeRangePicker
+ bind:fromDate
+ bind:toDate
+ bind:fromTime
+ bind:toTime
+ bind:preset={selectedPreset}
+ onApply={handleTimeRangeChange}
+ />
+ {#if slo && projectsState.canWriteCurrentProject}
+ <Button variant="outline" size="sm" onclick={() => (editOpen = true)}>
+ <Pencil class="mr-2 h-4 w-4" />
+ Edit
+ </Button>
+ <Button variant="destructiveOutline" size="sm" onclick={() => (deleteOpen = true)}>
+ <Trash2 class="mr-2 h-4 w-4" />
+ Delete
+ </Button>
+ {/if}
+ {/snippet}
+ </PageHeader>
+
+ <StatRow columns={4}>
+ <StatTile
+ label="Current SLI"
+ value={formatSloPercent(slo.currentSli, slo.targetPercent)}
+ valueClass={toneTextClass(budget)}
+ sub="target {slo.targetPercent}% over {slo.windowDays} days"
+ />
+ <StatTile
+ label="Error budget left"
+ value={formatBudgetRemaining(slo.budgetRemaining)}
+ valueClass={toneTextClass(budget)}
+ sub={slo.budgetRemaining !== null && slo.budgetRemaining < 0
+ ? 'overspent: more failures than the target allows'
+ : 'share of the allowed failures not yet used'}
+ />
+ <StatTile
+ label="Burn rate 1h"
+ value={formatBurnRate(slo.burnRate1h)}
+ valueClass={toneTextClass(burnTone(slo.burnRate1h))}
+ sub="1x spends the budget exactly over the window"
+ />
+ <StatTile
+ label="Burn rate 6h"
+ value={formatBurnRate(slo.burnRate6h)}
+ valueClass={toneTextClass(burnTone(slo.burnRate6h))}
+ sub="sustained burn over the last six hours"
+ />
+ </StatRow>
+
+ {#if coverage}
+ <InfoCallout>
+ History covers {formatDays(coverage.coveredDays)} of {formatDays(coverage.rangeDays)} days in
+ this range. Buckets before {formatDateTime(coverage.historyFrom.toISOString(), {
+ format: 'short'
+ })} are still being computed or fall outside telemetry retention.
+ </InfoCallout>
+ {/if}
+
+ <ChartCard
+ title="SLI"
+ loading={seriesLoading}
+ empty={sliPoints.length === 0}
+ emptyMessage={seriesError || 'No events in this range'}
+ >
+ {#snippet icon()}
+ <ChartLine class="h-3.5 w-3.5 text-muted-foreground" />
+ {/snippet}
+ <D3LineChart
+ series={sliChartSeries}
+ referenceLines={[{ value: slo.targetPercent, label: `Target ${slo.targetPercent}%` }]}
+ yDomain={sliChartDomain(sliPoints, slo.targetPercent)}
+ formatValue={(value) => `${value.toFixed(2)}%`}
+ />
+ </ChartCard>
+
+ <ChartCard
+ title="Error budget consumed"
+ loading={seriesLoading}
+ empty={budgetPoints.length === 0}
+ emptyMessage={seriesError || 'No events in this range'}
+ >
+ {#snippet icon()}
+ <PiggyBank class="h-3.5 w-3.5 text-muted-foreground" />
+ {/snippet}
+ <D3LineChart
+ series={budgetChartSeries}
+ referenceLines={[{ value: 100, label: 'Budget exhausted' }]}
+ yDomain={budgetChartDomain(budgetPoints)}
+ formatValue={(value) => `${value.toFixed(1)}%`}
+ areaFill
+ />
+ <p class="mt-2 text-xs text-muted-foreground">
+ Cumulative share of the error budget the traffic in this range has used. On the SLO's own
+ window it ends at 100% minus the budget left.
+ </p>
+ </ChartCard>
+
+ <Card.Root class="gap-3 pt-2">
+ <Card.Header
+ class="flex flex-row flex-wrap items-center justify-between gap-3 space-y-0 border-b pl-3 [.border-b]:pb-2"
+ >
+ <Card.Title class="flex items-center gap-2 text-base font-medium">
+ <Bell class="h-3.5 w-3.5 text-muted-foreground" />
+ <span class="text-sm font-[400] text-muted-foreground">Alert rules</span>
+ </Card.Title>
+ <div class="flex items-center gap-2">
+ <Button variant="outline" size="sm" href="/notifications?tab=rules">
+ Manage alert rules
+ </Button>
+ <Button variant="outline" size="sm" href={badEventsHref(slo)}>
+ <ExternalLink class="mr-2 h-3.5 w-3.5" />
+ {badEventsLabel(slo)}
+ </Button>
+ </div>
+ </Card.Header>
+ <Card.Content class="pb-4">
+ {#if rules.length === 0}
+ <p class="text-sm text-muted-foreground">
+ No alert rules reference this SLO yet. Add an SLO Burn Rate or SLO Budget Consumed rule
+ on the Alerts page to get paged when the budget burns too fast.
+ </p>
+ {:else}
+ <div class="space-y-2">
+ {#each rules as rule (rule.id)}
+ {@const state = ruleStateLabel(rule)}
+ <div
+ class="flex flex-col gap-2 rounded-md border p-3 text-sm sm:flex-row sm:items-center sm:justify-between sm:gap-4"
+ >
+ <div class="min-w-0">
+ <div class="font-medium">{rule.name}</div>
+ <div class="text-xs text-muted-foreground">
+ {ruleTypeLabels[rule.ruleType] ?? rule.ruleType}
+ {#if rule.ruleType === 'slo_burn_rate'}
+ · {rule.config.burnRateThreshold ?? 14.4}x over {rule.config
+ .longWindowMinutes ?? 60}m /
+ {rule.config.shortWindowMinutes ?? 5}m
+ {:else if rule.ruleType === 'slo_budget_consumed'}
+ · {rule.config.consumedPercent ?? 80}% consumed
+ {/if}
+ </div>
+ </div>
+ <div class="flex shrink-0 items-center gap-2">
+ <Badge variant="outline">{rule.channelName}</Badge>
+ <StatusPill tone={state.tone} label={state.label} />
+ </div>
+ </div>
+ {/each}
+ </div>
+ {/if}
+ </Card.Content>
+ </Card.Root>
+ </div>
+
+ <SloSheet bind:open={editOpen} {slo} onSaved={reloadAfterSave} />
+
+ <ConfirmDeleteDialog
+ bind:open={deleteOpen}
+ entity="SLO"
+ description={`This permanently deletes "${slo.name}" and its computed history. Alert rules that reference it are removed too.`}
+ loading={deleting}
+ onConfirm={deleteSlo}
+ />
+{/if}
diff --git a/frontend/src/routes/slos/[id]/+page.ts b/frontend/src/routes/slos/[id]/+page.ts
new file mode 100644
index 00000000..6963916f
--- /dev/null
+++ b/frontend/src/routes/slos/[id]/+page.ts
@@ -0,0 +1,15 @@
+import { redirect } from '@sveltejs/kit';
+import { authState } from '$lib/state/auth.svelte';
+import type { PageLoad } from './$types';
+
+export const prerender = false;
+
+export const load: PageLoad = ({ params, url }) => {
+ if (!authState.isAuthenticated) throw redirect(302, '/login');
+ return {
+ id: params.id,
+ preset: url.searchParams.get('preset'),
+ from: url.searchParams.get('from'),
+ to: url.searchParams.get('to')
+ };
+};
diff --git a/frontend/src/routes/slos/slo-sheet.svelte b/frontend/src/routes/slos/slo-sheet.svelte
new file mode 100644
index 00000000..2475c049
--- /dev/null
+++ b/frontend/src/routes/slos/slo-sheet.svelte
@@ -0,0 +1,898 @@
+<script lang="ts">
+ import { untrack } from 'svelte';
+ import { api } from '$lib/api';
+ import * as Sheet from '$lib/components/ui/sheet';
+ import * as Select from '$lib/components/ui/select';
+ import { Button } from '$lib/components/ui/button';
+ import { Input } from '$lib/components/ui/input';
+ import { Label } from '$lib/components/ui/label';
+ import { Checkbox } from '$lib/components/ui/checkbox';
+ import { ErrorAlert } from '$lib/components/ui/error-alert';
+ import { WarningCallout } from '$lib/components/ui/warning-callout';
+ import KeyValueRows from '$lib/components/synthetics/key-value-rows.svelte';
+ import TagFilter from '$lib/components/dashboard/tag-filter.svelte';
+ import { projectsState, isFrontendFramework } from '$lib/state/projects.svelte';
+ import {
+ BACKEND_SLI_KINDS,
+ FRONTEND_SLI_KINDS,
+ SLI_KIND_LABELS,
+ SLO_WINDOW_OPTIONS,
+ type GoodCondition,
+ type MetricAggregation,
+ type MetricOperator,
+ type SliConfig,
+ type SliKind,
+ type SLO,
+ type SloPreview,
+ type SpanSliAttribute
+ } from '$lib/state/slos.svelte';
+ import type { DiscoveredMetric } from '$lib/types/dashboard';
+ import { SPAN_KIND_OPTIONS } from '$lib/utils/span-fields';
+ import { parseSpanAttributeFilter, type SpanAttributeFilter } from '$lib/utils/span-explorer';
+ import { getErrorMessage } from '$lib/utils/errors';
+ import { formatSloPercent } from '$lib/utils/slo-math';
+ import { Check, FlaskConical, Plus } from '@lucide/svelte';
+ import { toast } from 'svelte-sonner';
+
+ interface Props {
+ open: boolean;
+ slo: SLO | null;
+ onSaved: () => void;
+ }
+
+ let { open = $bindable(), slo, onSaved }: Props = $props();
+
+ const isEditing = $derived(slo !== null);
+ let loading = $state(false);
+ let error = $state('');
+
+ let name = $state('');
+ let description = $state('');
+ let sliKind = $state<SliKind>('endpoint');
+ let targetPercent = $state('99.9');
+ let windowDays = $state('30');
+ let enabled = $state(true);
+
+ let endpointName = $state('');
+ let endpointGood = $state<GoodCondition>('availability');
+ let endpointLatencyMs = $state('500');
+
+ let spanService = $state('');
+ let spanName = $state('');
+ let spanKind = $state('');
+ let spanAttrRows = $state<{ key: string; value: string }[]>([]);
+ let spanGood = $state<GoodCondition>('availability');
+ let spanLatencyMs = $state('500');
+
+ let taskName = $state('');
+
+ let checkIds = $state<string[]>([]);
+
+ let metricName = $state('');
+ let metricTags = $state<Record<string, string>>({});
+ let metricAggregation = $state<MetricAggregation>('avg');
+ let metricOperator = $state<MetricOperator>('lt');
+ let metricThreshold = $state('');
+ let sliceMinutes = $state('5');
+ let missingIsBad = $state(false);
+
+ let sessionAppVersion = $state('');
+ let includeMessages = $state(false);
+
+ let endpointOptions = $state<string[]>([]);
+ let taskOptions = $state<string[]>([]);
+ let serviceOptions = $state<string[]>([]);
+ let checks = $state<{ id: number; name: string; checkType: string }[]>([]);
+ let metrics = $state<DiscoveredMetric[]>([]);
+ let loadedPickers = $state<Record<string, boolean>>({});
+
+ let previewing = $state(false);
+ let preview = $state<SloPreview | null>(null);
+ let previewError = $state('');
+
+ let originalDefinition = $state('');
+
+ const isFrontendProject = $derived(
+ !!projectsState.currentProject && isFrontendFramework(projectsState.currentProject.framework)
+ );
+ const defaultKind = $derived<SliKind>(isFrontendProject ? 'session' : 'endpoint');
+ const kindOptions = $derived.by(() => {
+ const kinds = isFrontendProject ? FRONTEND_SLI_KINDS : BACKEND_SLI_KINDS;
+ const list = slo && !kinds.includes(slo.sliKind) ? [...kinds, slo.sliKind] : kinds;
+ return list.map((kind) => ({ value: kind, label: SLI_KIND_LABELS[kind] }));
+ });
+ const kindHelp: Record<SliKind, string> = {
+ endpoint: 'Requests of one endpoint, or of every endpoint in the project.',
+ span: 'Any span matching the same filters the span explorer offers, so a queue consumer or an outbound call can have its own SLO.',
+ task: 'Background task runs. A run is bad when an exception was recorded on its own span.',
+ monitor: 'Probes of the selected monitors. Good means up; missed probes do not count.',
+ metric: 'Time slices of a metric. A slice is good when its aggregate satisfies the comparison.',
+ session:
+ 'Sessions reported by the browser and mobile SDKs. A session is bad when an exception was recorded in it; captured messages only count if you say so.'
+ };
+ const goodOptions = [
+ { value: 'availability', label: 'Status below 500' },
+ { value: 'latency', label: 'Faster than a threshold' }
+ ];
+ const spanGoodOptions = [
+ { value: 'availability', label: 'Span status is not error' },
+ { value: 'latency', label: 'Faster than a threshold' }
+ ];
+ const spanKindOptions = [{ value: '', label: 'Any kind' }, ...SPAN_KIND_OPTIONS];
+ const sliceOptions = [
+ { value: '1', label: '1 minute' },
+ { value: '5', label: '5 minutes' },
+ { value: '15', label: '15 minutes' },
+ { value: '60', label: '1 hour' }
+ ];
+ const aggregationOptions = [
+ { value: 'avg', label: 'Average' },
+ { value: 'min', label: 'Minimum' },
+ { value: 'max', label: 'Maximum' },
+ { value: 'sum', label: 'Sum' },
+ { value: 'last', label: 'Last value' }
+ ];
+ const operatorOptions = [
+ { value: 'lt', label: 'below' },
+ { value: 'lte', label: 'at most' },
+ { value: 'gt', label: 'above' },
+ { value: 'gte', label: 'at least' }
+ ];
+
+ function labelOf(options: { value: string; label: string }[], value: string, fallback: string) {
+ return options.find((option) => option.value === value)?.label ?? fallback;
+ }
+
+ function resetForm() {
+ name = '';
+ description = '';
+ sliKind = defaultKind;
+ targetPercent = '99.9';
+ windowDays = '30';
+ enabled = true;
+ endpointName = '';
+ endpointGood = 'availability';
+ endpointLatencyMs = '500';
+ spanService = '';
+ spanName = '';
+ spanKind = '';
+ spanAttrRows = [];
+ spanGood = 'availability';
+ spanLatencyMs = '500';
+ taskName = '';
+ checkIds = [];
+ metricName = '';
+ metricTags = {};
+ metricAggregation = 'avg';
+ metricOperator = 'lt';
+ metricThreshold = '';
+ sliceMinutes = '5';
+ missingIsBad = false;
+ sessionAppVersion = '';
+ includeMessages = false;
+ preview = null;
+ previewError = '';
+ error = '';
+ originalDefinition = '';
+ }
+
+ function populateFromSlo(s: SLO) {
+ resetForm();
+ name = s.name;
+ description = s.description;
+ sliKind = s.sliKind;
+ targetPercent = String(s.targetPercent);
+ windowDays = String(s.windowDays);
+ enabled = s.enabled;
+ const cfg = s.sliConfig as unknown as Record<string, unknown>;
+ switch (s.sliKind) {
+ case 'endpoint':
+ endpointName = (cfg.endpoint as string) ?? '';
+ endpointGood = (cfg.good as GoodCondition) ?? 'availability';
+ endpointLatencyMs = String((cfg.latencyThresholdMs as number) || 500);
+ break;
+ case 'span':
+ spanService = (cfg.serviceName as string) ?? '';
+ spanName = (cfg.name as string) ?? '';
+ spanKind = cfg.kind === null || cfg.kind === undefined ? '' : String(cfg.kind);
+ spanAttrRows = ((cfg.attributes as SpanSliAttribute[]) ?? []).map((a) => ({ ...a }));
+ spanGood = (cfg.good as GoodCondition) ?? 'availability';
+ spanLatencyMs = String((cfg.latencyThresholdMs as number) || 500);
+ break;
+ case 'task':
+ taskName = (cfg.taskName as string) ?? '';
+ break;
+ case 'monitor':
+ checkIds = ((cfg.checkIds as number[]) ?? []).map(String);
+ break;
+ case 'metric':
+ metricName = (cfg.metricName as string) ?? '';
+ metricTags = { ...((cfg.tags as Record<string, string>) ?? {}) };
+ metricAggregation = (cfg.aggregation as MetricAggregation) ?? 'avg';
+ metricOperator = (cfg.operator as MetricOperator) ?? 'lt';
+ metricThreshold = String(cfg.threshold ?? '');
+ sliceMinutes = String(s.bucketMinutes || 5);
+ missingIsBad = !!cfg.missingIsBad;
+ break;
+ case 'session':
+ sessionAppVersion = (cfg.appVersion as string) ?? '';
+ includeMessages = !!cfg.includeMessages;
+ break;
+ }
+ originalDefinition = currentDefinition() ?? '';
+ }
+
+ $effect(() => {
+ const current = slo;
+ if (open && current) {
+ untrack(() => populateFromSlo(current));
+ } else if (open && !current) {
+ untrack(resetForm);
+ }
+ if (!open) {
+ loadedPickers = {};
+ }
+ });
+
+ $effect(() => {
+ if (!open) return;
+ loadPickers(sliKind);
+ });
+
+ $effect(() => {
+ void currentDefinition();
+ preview = null;
+ previewError = '';
+ });
+
+ function handleOpenChange(value: boolean) {
+ open = value;
+ if (!value) {
+ error = '';
+ }
+ }
+
+ function rangeParams(days: number): { fromDate: string; toDate: string } {
+ const now = new Date();
+ return {
+ fromDate: new Date(now.getTime() - days * 24 * 60 * 60 * 1000).toISOString(),
+ toDate: now.toISOString()
+ };
+ }
+
+ const projectQuery = () => ({ projectId: projectsState.currentProjectId ?? undefined });
+
+ async function loadPickers(kind: SliKind) {
+ if (loadedPickers[kind]) return;
+ loadedPickers = { ...loadedPickers, [kind]: true };
+ try {
+ switch (kind) {
+ case 'endpoint': {
+ const response = await api.post(
+ '/endpoints/grouped',
+ {
+ ...rangeParams(7),
+ orderBy: 'count',
+ sortDirection: 'desc',
+ search: '',
+ rootFilter: '',
+ pagination: { page: 1, pageSize: 100 }
+ },
+ projectQuery()
+ );
+ endpointOptions = (response.data || []).map((row: { endpoint: string }) => row.endpoint);
+ break;
+ }
+ case 'task': {
+ const response = await api.post(
+ '/tasks/grouped',
+ {
+ ...rangeParams(7),
+ orderBy: 'count',
+ sortDirection: 'desc',
+ search: '',
+ rootFilter: '',
+ pagination: { page: 1, pageSize: 100 }
+ },
+ projectQuery()
+ );
+ taskOptions = (response.data || []).map((row: { taskName: string }) => row.taskName);
+ break;
+ }
+ case 'span': {
+ const response = await api.post(
+ '/spans/search',
+ { ...rangeParams(1), orderBy: 'start_time desc', pagination: { page: 1, pageSize: 1 } },
+ projectQuery()
+ );
+ serviceOptions = response.services || [];
+ break;
+ }
+ case 'monitor': {
+ const response = await api.get('/synthetics/checks', projectQuery());
+ checks = response.checks || [];
+ break;
+ }
+ case 'metric': {
+ const { fromDate, toDate } = rangeParams(7);
+ const response = await api.get(
+ `/metrics/discover?from=${fromDate}&to=${toDate}`,
+ projectQuery()
+ );
+ metrics = response.metrics || [];
+ break;
+ }
+ }
+ } catch (e) {
+ console.error('Failed to load SLO picker options:', e);
+ }
+ }
+
+ async function loadTagValues(key: string): Promise<string[]> {
+ try {
+ const response = await api.get(
+ `/metrics/discover/tags?name=${encodeURIComponent(metricName)}&key=${encodeURIComponent(key)}`,
+ projectQuery()
+ );
+ return response.values || [];
+ } catch {
+ return [];
+ }
+ }
+
+ const metricTagKeys = $derived(metrics.find((m) => m.name === metricName)?.tagKeys ?? []);
+
+ const serviceSelectOptions = $derived([
+ { value: '', label: 'Any service' },
+ ...[...new Set([...serviceOptions, spanService].filter((s) => !!s))].map((s) => ({
+ value: s,
+ label: s
+ }))
+ ]);
+
+ const checksLabel = $derived.by(() => {
+ if (checkIds.length === 0) return 'Select monitors';
+ const names = checks.filter((c) => checkIds.includes(String(c.id))).map((c) => c.name);
+ if (names.length === 0) return `${checkIds.length} selected`;
+ return names.join(', ');
+ });
+
+ function buildSpanAttributes(): SpanAttributeFilter[] {
+ let attributes: SpanAttributeFilter[] = [];
+ for (const row of spanAttrRows) {
+ if (!row.key.trim() && !row.value.trim()) continue;
+ const parsed = parseSpanAttributeFilter(row.key, row.value, attributes);
+ if (typeof parsed === 'string') {
+ throw new Error(parsed);
+ }
+ attributes = [...attributes, parsed];
+ }
+ return attributes;
+ }
+
+ function buildSliConfig(): SliConfig {
+ switch (sliKind) {
+ case 'endpoint':
+ return {
+ endpoint: endpointName.trim(),
+ good: endpointGood,
+ latencyThresholdMs: Number(endpointLatencyMs) || 0
+ };
+ case 'span':
+ return {
+ serviceName: spanService,
+ name: spanName.trim(),
+ kind: spanKind === '' ? null : Number(spanKind),
+ attributes: buildSpanAttributes(),
+ good: spanGood,
+ latencyThresholdMs: Number(spanLatencyMs) || 0
+ };
+ case 'task':
+ return { taskName: taskName.trim() };
+ case 'monitor':
+ return { checkIds: checkIds.map(Number) };
+ case 'metric':
+ return {
+ metricName: metricName.trim(),
+ tags: metricTags,
+ aggregation: metricAggregation,
+ operator: metricOperator,
+ threshold: String(metricThreshold ?? '').trim() === '' ? null : Number(metricThreshold),
+ missingIsBad
+ };
+ case 'session':
+ return { appVersion: sessionAppVersion.trim(), includeMessages };
+ }
+ }
+
+ function currentDefinition(): string | null {
+ try {
+ return JSON.stringify({
+ sliKind,
+ sliConfig: buildSliConfig(),
+ bucketMinutes: sliKind === 'metric' ? Number(sliceMinutes) : 1
+ });
+ } catch {
+ return null;
+ }
+ }
+
+ const definitionChanged = $derived.by(() => {
+ if (!isEditing || !originalDefinition) return false;
+ const current = currentDefinition();
+ return current !== null && current !== originalDefinition;
+ });
+
+ async function runPreview() {
+ previewing = true;
+ previewError = '';
+ preview = null;
+ const definition = currentDefinition();
+ try {
+ const result = await api.post(
+ '/slos/preview',
+ {
+ sliKind,
+ sliConfig: buildSliConfig(),
+ bucketMinutes: sliKind === 'metric' ? Number(sliceMinutes) : 1
+ },
+ projectQuery()
+ );
+ if (currentDefinition() === definition) preview = result;
+ } catch (e) {
+ if (currentDefinition() === definition) previewError = getErrorMessage(e) || 'Preview failed';
+ } finally {
+ previewing = false;
+ }
+ }
+
+ async function handleSubmit() {
+ loading = true;
+ error = '';
+ try {
+ const payload = {
+ name: name.trim(),
+ description: description.trim(),
+ sliKind,
+ sliConfig: buildSliConfig(),
+ targetPercent: Number(targetPercent),
+ windowDays: Number(windowDays),
+ bucketMinutes: sliKind === 'metric' ? Number(sliceMinutes) : 1,
+ enabled
+ };
+ if (isEditing && slo) {
+ await api.put(`/slos/${slo.id}`, payload, projectQuery());
+ toast.success('Successfully updated the SLO', { position: 'top-center' });
+ } else {
+ await api.post('/slos', payload, projectQuery());
+ toast.success('Successfully created the SLO', { position: 'top-center' });
+ }
+ open = false;
+ onSaved();
+ } catch (e) {
+ error = getErrorMessage(e) || 'Failed to save the SLO';
+ } finally {
+ loading = false;
+ }
+ }
+</script>
+
+<Sheet.Root {open} onOpenChange={handleOpenChange}>
+ <Sheet.Content
+ side="right"
+ class="flex w-full flex-col gap-0 overflow-y-auto sm:w-[760px] sm:max-w-[min(760px,100vw)]"
+ >
+ <Sheet.Header class="border-b px-6 pb-4">
+ <Sheet.Title>{isEditing ? 'Edit SLO' : 'New SLO'}</Sheet.Title>
+ <Sheet.Description>
+ {isEditing
+ ? 'Update the objective. Changing what the indicator measures recomputes its history.'
+ : 'Pick what counts as a good event, the target share, and the rolling window to hold it over.'}
+ </Sheet.Description>
+ </Sheet.Header>
+
+ <form
+ onsubmit={(e) => {
+ e.preventDefault();
+ handleSubmit();
+ }}
+ class="flex-1 space-y-5 px-6 py-5"
+ >
+ <ErrorAlert {error} />
+
+ <div class="grid gap-4 sm:grid-cols-2">
+ <div class="space-y-2">
+ <Label for="slo-name">Name</Label>
+ <Input
+ id="slo-name"
+ bind:value={name}
+ placeholder="Checkout availability"
+ disabled={loading}
+ />
+ </div>
+ <div class="space-y-2">
+ <Label for="slo-description">Description</Label>
+ <Input
+ id="slo-description"
+ bind:value={description}
+ placeholder="What this objective protects"
+ disabled={loading}
+ />
+ </div>
+ </div>
+
+ <div class="grid gap-4 sm:grid-cols-3">
+ <div class="space-y-2">
+ <Label>Indicator</Label>
+ <Select.Root type="single" bind:value={sliKind} disabled={loading}>
+ <Select.Trigger class="w-full">{SLI_KIND_LABELS[sliKind]}</Select.Trigger>
+ <Select.Content>
+ {#each kindOptions as option (option.value)}
+ <Select.Item value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ </div>
+ <div class="space-y-2">
+ <Label for="slo-target">Target (%)</Label>
+ <Input
+ id="slo-target"
+ type="number"
+ step="0.01"
+ min="0"
+ max="100"
+ bind:value={targetPercent}
+ disabled={loading}
+ />
+ </div>
+ <div class="space-y-2">
+ <Label>Window</Label>
+ <Select.Root type="single" bind:value={windowDays} disabled={loading}>
+ <Select.Trigger class="w-full">
+ {labelOf(SLO_WINDOW_OPTIONS, windowDays, 'Select window')}
+ </Select.Trigger>
+ <Select.Content>
+ {#each SLO_WINDOW_OPTIONS as option (option.value)}
+ <Select.Item value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ </div>
+ </div>
+
+ <div class="space-y-4 rounded-md border p-4">
+ <p class="text-xs text-muted-foreground">{kindHelp[sliKind]}</p>
+
+ {#if sliKind === 'endpoint'}
+ <div class="space-y-2">
+ <Label for="slo-endpoint">Endpoint</Label>
+ <Input
+ id="slo-endpoint"
+ list="slo-endpoint-options"
+ bind:value={endpointName}
+ placeholder="Leave empty for all endpoints, or GET /api/checkout"
+ disabled={loading}
+ />
+ <datalist id="slo-endpoint-options">
+ {#each endpointOptions as option (option)}
+ <option value={option}></option>
+ {/each}
+ </datalist>
+ </div>
+ <div class="grid gap-4 sm:grid-cols-2">
+ <div class="space-y-2">
+ <Label>Good when</Label>
+ <Select.Root type="single" bind:value={endpointGood} disabled={loading}>
+ <Select.Trigger class="w-full"
+ >{labelOf(goodOptions, endpointGood, 'Select')}</Select.Trigger
+ >
+ <Select.Content>
+ {#each goodOptions as option (option.value)}
+ <Select.Item value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ </div>
+ {#if endpointGood === 'latency'}
+ <div class="space-y-2">
+ <Label for="slo-endpoint-latency">Threshold (ms)</Label>
+ <Input
+ id="slo-endpoint-latency"
+ type="number"
+ min="1"
+ bind:value={endpointLatencyMs}
+ disabled={loading}
+ />
+ </div>
+ {/if}
+ </div>
+ {:else if sliKind === 'span'}
+ <div class="grid gap-4 sm:grid-cols-3">
+ <div class="space-y-2">
+ <Label>Service</Label>
+ <Select.Root type="single" bind:value={spanService} disabled={loading}>
+ <Select.Trigger class="w-full">
+ <span class="truncate"
+ >{labelOf(serviceSelectOptions, spanService, 'Any service')}</span
+ >
+ </Select.Trigger>
+ <Select.Content class="max-h-64 overflow-y-auto">
+ {#each serviceSelectOptions as option (option.value)}
+ <Select.Item value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ </div>
+ <div class="space-y-2">
+ <Label for="slo-span-name">Name contains</Label>
+ <Input
+ id="slo-span-name"
+ bind:value={spanName}
+ placeholder="POST /checkout"
+ disabled={loading}
+ />
+ </div>
+ <div class="space-y-2">
+ <Label>Kind</Label>
+ <Select.Root type="single" bind:value={spanKind} disabled={loading}>
+ <Select.Trigger class="w-full"
+ >{labelOf(spanKindOptions, spanKind, 'Any kind')}</Select.Trigger
+ >
+ <Select.Content>
+ {#each spanKindOptions as option (option.value)}
+ <Select.Item value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ </div>
+ </div>
+ <div class="space-y-2">
+ <Label>Attribute filters</Label>
+ <KeyValueRows
+ bind:rows={spanAttrRows}
+ keyPlaceholder="http.route"
+ valuePlaceholder="/checkout"
+ addLabel="Add attribute"
+ disabled={loading}
+ />
+ <p class="text-xs text-muted-foreground">
+ Exact matches on span attributes, the OpenTelemetry names. Up to 10 filters.
+ </p>
+ </div>
+ <div class="grid gap-4 sm:grid-cols-2">
+ <div class="space-y-2">
+ <Label>Good when</Label>
+ <Select.Root type="single" bind:value={spanGood} disabled={loading}>
+ <Select.Trigger class="w-full"
+ >{labelOf(spanGoodOptions, spanGood, 'Select')}</Select.Trigger
+ >
+ <Select.Content>
+ {#each spanGoodOptions as option (option.value)}
+ <Select.Item value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ </div>
+ {#if spanGood === 'latency'}
+ <div class="space-y-2">
+ <Label for="slo-span-latency">Threshold (ms)</Label>
+ <Input
+ id="slo-span-latency"
+ type="number"
+ min="1"
+ bind:value={spanLatencyMs}
+ disabled={loading}
+ />
+ </div>
+ {/if}
+ </div>
+ {:else if sliKind === 'task'}
+ <div class="space-y-2">
+ <Label for="slo-task">Task</Label>
+ <Input
+ id="slo-task"
+ list="slo-task-options"
+ bind:value={taskName}
+ placeholder="Leave empty for all tasks, or sync_users"
+ disabled={loading}
+ />
+ <datalist id="slo-task-options">
+ {#each taskOptions as option (option)}
+ <option value={option}></option>
+ {/each}
+ </datalist>
+ </div>
+ {:else if sliKind === 'monitor'}
+ <div class="space-y-2">
+ <Label>Monitors</Label>
+ <Select.Root type="multiple" bind:value={checkIds} disabled={loading}>
+ <Select.Trigger class="w-full">
+ <span class="truncate">{checksLabel}</span>
+ </Select.Trigger>
+ <Select.Content class="max-h-64 overflow-y-auto">
+ {#each checks as check (check.id)}
+ <Select.Item value={String(check.id)} label={check.name}>
+ {check.name}
+ <span class="ml-2 text-xs text-muted-foreground uppercase"
+ >{check.checkType}</span
+ >
+ </Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ {#if checks.length === 0}
+ <p class="text-xs text-muted-foreground">No monitors in this project yet.</p>
+ {/if}
+ </div>
+ {:else if sliKind === 'metric'}
+ <div class="space-y-2">
+ <Label for="slo-metric">Metric</Label>
+ <Input
+ id="slo-metric"
+ list="slo-metric-options"
+ bind:value={metricName}
+ placeholder="system.cpu.utilization"
+ disabled={loading}
+ />
+ <datalist id="slo-metric-options">
+ {#each metrics as metric (metric.name)}
+ <option value={metric.name}></option>
+ {/each}
+ </datalist>
+ </div>
+ {#if metricTagKeys.length > 0}
+ <div class="space-y-2">
+ <Label>Tag filters</Label>
+ <TagFilter
+ tagKeys={metricTagKeys}
+ activeFilters={metricTags}
+ onFilterChange={(filters) => (metricTags = filters)}
+ onLoadTagValues={loadTagValues}
+ />
+ </div>
+ {/if}
+ <div class="grid gap-4 sm:grid-cols-3">
+ <div class="space-y-2">
+ <Label>Aggregate per slice</Label>
+ <Select.Root type="single" bind:value={metricAggregation} disabled={loading}>
+ <Select.Trigger class="w-full">
+ {labelOf(aggregationOptions, metricAggregation, 'Average')}
+ </Select.Trigger>
+ <Select.Content>
+ {#each aggregationOptions as option (option.value)}
+ <Select.Item value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ </div>
+ <div class="space-y-2">
+ <Label>Good when value is</Label>
+ <Select.Root type="single" bind:value={metricOperator} disabled={loading}>
+ <Select.Trigger class="w-full"
+ >{labelOf(operatorOptions, metricOperator, 'below')}</Select.Trigger
+ >
+ <Select.Content>
+ {#each operatorOptions as option (option.value)}
+ <Select.Item value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ </div>
+ <div class="space-y-2">
+ <Label for="slo-metric-threshold">Threshold</Label>
+ <Input
+ id="slo-metric-threshold"
+ type="number"
+ step="any"
+ bind:value={metricThreshold}
+ placeholder="90"
+ disabled={loading}
+ />
+ </div>
+ </div>
+ <div class="grid gap-4 sm:grid-cols-2">
+ <div class="space-y-2">
+ <Label>Slice length</Label>
+ <Select.Root type="single" bind:value={sliceMinutes} disabled={loading}>
+ <Select.Trigger class="w-full"
+ >{labelOf(sliceOptions, sliceMinutes, '5 minutes')}</Select.Trigger
+ >
+ <Select.Content>
+ {#each sliceOptions as option (option.value)}
+ <Select.Item value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+ </div>
+ <label class="flex cursor-pointer items-center gap-2 pt-7 text-sm">
+ <Checkbox
+ checked={missingIsBad}
+ onCheckedChange={(checked) => (missingIsBad = checked === true)}
+ disabled={loading}
+ />
+ Slices without data count as bad
+ </label>
+ </div>
+ {:else if sliKind === 'session'}
+ <div class="grid gap-4 sm:grid-cols-2">
+ <div class="space-y-2">
+ <Label for="slo-app-version">App version</Label>
+ <Input
+ id="slo-app-version"
+ bind:value={sessionAppVersion}
+ placeholder="Leave empty for every version, or 1.4.2"
+ disabled={loading}
+ />
+ </div>
+ <label class="flex cursor-pointer items-center gap-2 pt-7 text-sm">
+ <Checkbox
+ checked={includeMessages}
+ onCheckedChange={(checked) => (includeMessages = checked === true)}
+ disabled={loading}
+ />
+ Captured messages count as failures
+ </label>
+ </div>
+ {/if}
+
+ <div class="flex flex-wrap items-center gap-3 border-t pt-3">
+ <Button
+ type="button"
+ variant="outline"
+ size="sm"
+ onclick={runPreview}
+ disabled={loading || previewing}
+ >
+ <FlaskConical class="mr-2 h-4 w-4" />
+ {previewing ? 'Computing...' : 'Preview last 24h'}
+ </Button>
+ {#if previewError}
+ <span class="text-sm text-red-600 dark:text-red-400">{previewError}</span>
+ {:else if preview}
+ <span class="font-mono text-sm tabular-nums">
+ {#if preview.sli === null}
+ Last 24h: no events matched
+ {:else}
+ Last 24h: {formatSloPercent(preview.sli, Number(targetPercent) || 99.9)}
+ <span class="text-muted-foreground">
+ ({preview.good.toLocaleString()} / {preview.total.toLocaleString()})
+ </span>
+ {/if}
+ </span>
+ {/if}
+ </div>
+ </div>
+
+ {#if definitionChanged}
+ <WarningCallout title="History will be recomputed">
+ Changing what the indicator measures recalculates every bucket from your retained
+ telemetry. The current SLI and budget reset until the backfill finishes.
+ </WarningCallout>
+ {/if}
+
+ <label class="flex cursor-pointer items-center gap-2 text-sm">
+ <Checkbox
+ checked={enabled}
+ onCheckedChange={(checked) => (enabled = checked === true)}
+ disabled={loading}
+ />
+ Enabled
+ </label>
+ </form>
+
+ <div class="flex justify-end gap-2 border-t px-6 py-4">
+ <Button variant="outline" onclick={() => handleOpenChange(false)} disabled={loading}
+ >Cancel</Button
+ >
+ <Button variant={isEditing ? 'default' : 'success'} onclick={handleSubmit} disabled={loading}>
+ {#if isEditing}
+ <Check class="mr-2 h-4 w-4" />
+ {loading ? 'Updating...' : 'Update SLO'}
+ {:else}
+ <Plus class="mr-2 h-4 w-4" />
+ {loading ? 'Creating...' : 'New SLO'}
+ {/if}
+ </Button>
+ </div>
+ </Sheet.Content>
+</Sheet.Root>
diff --git a/skills/traceway-learn/alerts.md b/skills/traceway-learn/alerts.md
index 9d23329c..0d297b24 100644
--- a/skills/traceway-learn/alerts.md
+++ b/skills/traceway-learn/alerts.md
@@ -9,7 +9,7 @@ Alerts are how Traceway tells people that something happened. They live under **
| **Rules** | When to notify. Each rule watches one condition and sends to exactly one channel. A channel can serve many rules. |
| **History** | Every notification that was sent, with its status |
-Rules and channels belong to a project and only see that project's data. Channels have their own On/Off switch, and a rule only runs while its channel is on. Changing them needs write access on the project. Browser and mobile projects only offer the New Issue rule type.
+Rules and channels belong to a project and only see that project's data. Channels have their own On/Off switch, and a rule only runs while its channel is on. Changing them needs write access on the project. Browser and mobile projects only offer the New Issue rule type and the two SLO rule types.
## Channels
@@ -62,6 +62,8 @@ The numbers in brackets are what the evaluator uses when a setting is missing. T
| **AI Conversation Cost** | One conversation's cost over 24 hours reaches the threshold | $5 | Warning, Critical at 3x |
| **AI Flagged Content** | An AI call matches a flagged term. The terms come from the project's AI settings. | Optional term list (empty means any flagged match) | Warning |
| **Monitor Down** | Any monitor in the project goes down, plus a recovery notice when it comes back | None | Critical, recovery Info |
+| **SLO Burn Rate** | An SLO spends its error budget faster than the threshold over both the long and the short window, plus a recovery notice once either window calms down. Fires on the transition only. | SLO, long window 60 min, short window 5 min, threshold 14.4x; presets Fast 14.4x 1h/5m, Slow 6x 6h/30m, Long 1x 3d/6h; both windows round up to whole slices of a metric SLO | Warning, Critical at 2x the threshold, recovery Info |
+| **SLO Budget Consumed** | An SLO has used at least a share of its window's error budget, plus a recovery notice when the budget recovers. Fires on the transition only. | SLO, consumed 80% | Warning, Critical at 100%, recovery Info |
The aggregation in Metric Threshold spans every series of the metric with no tag filter, so prefer `max` (or `min` for lower-is-bad metrics) where one bad host could hide in an average.
@@ -72,13 +74,14 @@ Apdex uses fixed bands: satisfied up to 750 ms, tolerating up to 1500 ms, anythi
- **On ingest**: New Issue, Error Regression, AI Trace Cost, AI Conversation Cost, AI Flagged Content. These fire within seconds.
- **On state change**: Monitor Down, the moment a probe flips the monitor.
- **Polled**: everything else, every 60 seconds by default (self-hosted: `NOTIFICATION_POLL_SECONDS`).
+- **SLO rules** are polled too, but read the SLO's precomputed buckets (refreshed every `SLO_POLL_SECONDS`, 60 by default) instead of raw telemetry, and they are evaluated during their cooldown so a recovery notice is never delayed by it. A recovery notice does not restart the cooldown.
Cooldowns differ by kind:
- **Polled rules** have one cooldown per rule. After it fires, it is not evaluated again until the cooldown ends. This cooldown survives restarts.
- **Ingest and monitor rules** cool down per item: per exception hash, trace name, conversation or monitor. Two different new errors a minute apart both alert. The same error does not alert again within its cooldown. This memory is in process only, so a restart can let the same item alert once more.
- The cooldown cannot be 0. Empty or 0 becomes 15 minutes.
-- **Editing** a rule resets its cooldown and its per-item memory, so the new settings apply at once. Toggling it does not.
+- **Changing** a rule's type, channel or condition resets its cooldown and its per-item memory, so the new settings apply at once. Renaming it, changing its cooldown or severity, or toggling it does not.
Delivery goes through a durable queue. Failed sends are retried after 1, 5, 15 and 60 minutes, five attempts in all. A queued message keeps the channel settings it was sent with.
diff --git a/website/app/product/slos/page.tsx b/website/app/product/slos/page.tsx
new file mode 100644
index 00000000..539cb78b
--- /dev/null
+++ b/website/app/product/slos/page.tsx
@@ -0,0 +1,256 @@
+import Link from "next/link";
+import type { Metadata } from "next";
+import { ArrowRight, Goal } from "lucide-react";
+
+import { Chip } from "@/components/chip";
+import { SectionHead } from "@/components/section-head";
+import { FeatureRow } from "@/components/feature-row";
+import { FaqList } from "@/components/faq-list";
+import { FinalCTA } from "@/components/final-cta";
+import { AuroraBackground } from "@/components/aurora-background";
+
+export const metadata: Metadata = {
+ title: "SLOs · Traceway",
+ description:
+ "Service level objectives over the telemetry you already send: endpoint availability and latency, any span filter, task success, monitor uptime, a metric held under a limit, or crash-free sessions. Rolling windows, error budgets, burn rates, and multi-window burn-rate alerts that recover on their own.",
+};
+
+export default function SlosPage() {
+ return (
+ <main className="relative">
+ <section className="hero hero-product relative">
+ <AuroraBackground variant="hero" />
+ <div className="wrap relative z-10">
+ <Chip variant="ok">
+ <Goal className="h-3 w-3 inline mr-1" />
+ SLOs, Error Budgets &amp; Burn-Rate Alerts
+ </Chip>
+ <h1 className="mt-6">
+ A threshold says it broke.{" "}
+ <em>An objective says how much you can afford.</em>
+ </h1>
+ <p className="hero-sub">
+ An SLO holds an indicator, the share of good requests, spans, task
+ runs, probes, metric slices or crash-free sessions, to a target over
+ a rolling 7, 30 or 90 day window. What the target leaves over is the
+ error budget, and the burn rate says how fast you are spending it.
+ Traceway computes all of it from the spans, probes, metrics and
+ sessions it already stores, so there is nothing new to instrument.
+ </p>
+ <div className="hero-cta-row">
+ <Link
+ href="https://docs.tracewayapp.com/learn/slos"
+ className="btn btn-accent"
+ >
+ Read the Docs <ArrowRight className="h-4 w-4" />
+ </Link>
+ <Link
+ href="https://cloud.tracewayapp.com/register"
+ className="btn btn-ghost"
+ >
+ Try Traceway Cloud
+ </Link>
+ </div>
+ </div>
+ </section>
+
+ <div className="band-light">
+ <section className="wrap">
+ <FeatureRow
+ eyebrow="Six kinds of indicator"
+ title={
+ <>
+ Endpoints, spans, tasks, monitors, metrics{" "}
+ <em>or crash-free sessions</em>
+ </>
+ }
+ description="Pick what a good event is and Traceway counts it. An endpoint is good when it answers below 500 or faster than a threshold. A task run is good when no exception landed on its span. A monitor probe is good when it came back up. A metric is cut into slices of 1, 5, 15 or 60 minutes. Each slice is good when its average, minimum, maximum, sum or last value stays on the right side of a limit. Slices without data can count as bad, so a missing heartbeat hurts."
+ bullets={[
+ "Availability or latency for endpoints and spans",
+ "Task success from the exception on the run's own span",
+ "Monitor uptime straight from the probe results",
+ "Metric timeslices: over, under, or missing",
+ "Crash-free sessions for browser apps, per app version",
+ ]}
+ image={{
+ src: "/images/slos-list.png",
+ alt: "The Traceway SLOs list with the current SLI, error budget left, burn rate and status of every objective",
+ }}
+ />
+ </section>
+
+ <section className="wrap">
+ <FeatureRow
+ reverse
+ eyebrow="Browser apps"
+ title={
+ <>
+ Crash-free sessions, <em>per release</em>
+ </>
+ }
+ description="Browser projects get the objective front ends live by: the share of sessions that ended without an exception. Create one per app version and each release gets its own budget, so a release that crashes more than the last one shows up as a breach rather than a hunch. Sessions come from always-on session recording in the browser SDK."
+ bullets={[
+ "All versions, or one app version per objective",
+ "Captured messages count only if you say so",
+ "Works for React, Vue, Svelte and jQuery with always-on recording",
+ "Burn-rate and budget alerts like every other objective",
+ ]}
+ image={{
+ src: "/images/slos-sessions.png",
+ alt: "Crash-free session SLOs of a web app in Traceway, per release and across all versions, with the error budget left and the burn rate of each",
+ }}
+ />
+ </section>
+
+ <section className="wrap">
+ <FeatureRow
+ eyebrow="Any span can carry an objective"
+ title={
+ <>
+ Filter spans the way the explorer does,{" "}
+ <em>then hold them to a number</em>
+ </>
+ }
+ description="Most SLO products only know services and HTTP routes. A span SLO takes the same filters as the span explorer: service, name, kind, and exact attribute values. A queue consumer, an outbound call to a payment provider, or a GraphQL resolver gets its own objective as long as its spans carry an attribute that identifies it. The Preview button runs the indicator against the last 24 hours before you save, so you see the share of good events and the counts and never guess."
+ bullets={[
+ "Service, name, kind and up to ten attribute filters",
+ "Good when the span status is not error, or faster than a threshold",
+ "Preview the last 24 hours before saving",
+ "Works on any OpenTelemetry span, entry point or not",
+ ]}
+ image={{
+ src: "/images/slos-sheet.png",
+ alt: "Creating a span SLO in Traceway with a service, name and attribute filter and a 24-hour preview",
+ }}
+ />
+ </section>
+
+ <section className="wrap">
+ <FeatureRow
+ reverse
+ eyebrow="Error budget and burn rate"
+ title={
+ <>
+ Stop arguing about <em>whether 99.5% is bad</em>
+ </>
+ }
+ description="With a 99.9% target over 30 days and a million requests, a thousand may fail. The detail page shows the current SLI against the target, how much of that budget is left, and the burn rate over the last hour and the last six hours: 1x spends the budget exactly at the end of the window, 14.4x empties a 30-day budget in two days. Charts plot the SLI against the target line and the cumulative share of the budget the range has used. One click opens the endpoint, task, monitor or sessions page, or the failing spans in the explorer."
+ bullets={[
+ "SLI and error budget charts with the target drawn in",
+ "Burn rates over 1 hour and 6 hours, color coded",
+ "One click to the endpoint, task, monitor or sessions page, or the failing spans",
+ "Rolling 7, 30 or 90 day windows, changed without losing history",
+ ]}
+ image={{
+ src: "/images/slos-detail.png",
+ alt: "A Traceway SLO detail page with the SLI chart against its target, the error budget chart, and the burn rate tiles",
+ }}
+ />
+ </section>
+
+ <section className="wrap">
+ <FeatureRow
+ eyebrow="Alert on the burn, not the blip"
+ title={
+ <>
+ Page on fast burns, chat about slow ones,{" "}
+ <em>recover on your own</em>
+ </>
+ }
+ description="Two alert rules read the SLO: Burn Rate fires when the budget burns faster than a threshold over both a long and a short window, so a one-minute spike never pages and a burn that already ended never pages either. Budget Consumed fires when more than a share of the window's budget is gone. Both fire once on the transition, keep their state across restarts, and send a recovery notice when the condition clears. On an escalation channel the recovery resolves the on-call page itself."
+ bullets={[
+ "Google SRE presets: Fast 14.4x over 1h/5m, Slow 6x over 6h/30m, Long 1x over 3d/6h",
+ "Fires on the transition, never re-pages every poll",
+ "Recovery notices, and auto-resolved on-call pages",
+ "Evaluated from precomputed buckets, no raw scans per poll",
+ ]}
+ image={{
+ src: "/images/slos-rule.png",
+ alt: "Configuring an SLO burn-rate alert rule in Traceway with the Fast, Slow and Long presets",
+ }}
+ />
+ </section>
+
+ <section className="wrap">
+ <FeatureRow
+ reverse
+ eyebrow="Computed once, read a thousand times"
+ title={
+ <>
+ Buckets, not scans, <em>on SQLite as well as ClickHouse</em>
+ </>
+ }
+ description="A background worker folds the raw rows into good/total buckets, one per minute or per metric slice. It refreshes the recent buckets as late data arrives, and backfills the whole window in chunks after you create or change an objective. The SLI, the budget, the burn rates and the alert rules are sums over those buckets. That is what makes a 90-day objective cheap on an embedded SQLite or DuckDB instance, and it means the history survives the pruning of the raw telemetry behind it."
+ bullets={[
+ "Per-minute buckets (per slice for metrics) on ClickHouse, SQLite and DuckDB",
+ "Late data within the last hour and a quarter is absorbed, longer for sessions and tasks",
+ "History kept for 120 days whatever the raw retention is",
+ "A breached objective still shows where the budget crossed zero",
+ ]}
+ image={{
+ src: "/images/slos-detail-breached.png",
+ alt: "A breached Traceway task SLO with an overspent error budget and the budget chart crossing the exhausted line",
+ }}
+ />
+ </section>
+ </div>
+
+ <FinalCTA
+ title={
+ <>
+ Objectives in the same tool <em>as the traces that explain them</em>
+ </>
+ }
+ description="SLOs, error budgets, burn-rate alerts, on-call and the spans behind every bad event, in one place. Open source if you self-host."
+ primary={{
+ label: "Read the SLO docs",
+ href: "https://docs.tracewayapp.com/learn/slos",
+ }}
+ secondary={{
+ label: "Start Free",
+ href: "https://cloud.tracewayapp.com/register",
+ }}
+ />
+
+ <section className="wrap pt-10 pb-24">
+ <div className="max-w-3xl mx-auto">
+ <SectionHead
+ align="center"
+ eyebrow="FAQ"
+ title="Questions about SLOs"
+ />
+ <div className="mt-4">
+ <FaqList
+ items={[
+ {
+ q: "How is an SLO different from an alert rule?",
+ a: "A threshold rule looks at the last few minutes and fires when a number crosses a line. An SLO holds a share of good events to a target over weeks, keeps the history, and turns the gap into a budget. The burn-rate rules then alert on how fast that budget is going rather than on the raw rate, which is what keeps a short spike from paging and a slow leak from going unnoticed.",
+ },
+ {
+ q: "What is a burn rate?",
+ a: "The observed bad rate divided by the rate the target allows. A burn rate of 1 spends the error budget exactly at the end of the window. 14.4 spends a 30-day budget in about two days, which is why the Fast preset pages at 14.4x over the last hour. A window with no traffic burns nothing.",
+ },
+ {
+ q: "Which indicators can I build an SLO on?",
+ a: "Endpoint availability or latency, any span filter (service, name, kind, exact attribute values) for availability or latency, task success, monitor uptime, and metric timeslices where each slice is good when its aggregate is over or under a limit. Endpoints and tasks are promoted from spans, so the span kind covers them too. Browser projects get crash-free sessions instead, since their SDK reports sessions and exceptions rather than spans.",
+ },
+ {
+ q: "What happens to the history when I change an SLO?",
+ a: "Changing the target or the window keeps every bucket, since both are applied on read. Changing what the indicator measures, or the slice length of a metric SLO, recomputes the history from your retained telemetry; the form warns before you save and the detail page says how much of the window is covered while the backfill runs.",
+ },
+ {
+ q: "Does it work on the SQLite build?",
+ a: "Yes. The worker computes per-minute (or per-slice) buckets on ClickHouse, SQLite and DuckDB alike, and reads sum over buckets rather than scanning raw rows. Buckets are kept for 120 days, so a 90-day objective keeps its history even where the raw telemetry retention is 30 days; what cannot be backfilled is the part of the window older than what is still retained when the SLO is created.",
+ },
+ {
+ q: "Can an AI agent read my SLOs?",
+ a: "The list, detail, series and breached-count routes are on the agent allowlist, so a read token can call them directly. The CLI and the MCP server do not cover SLOs yet.",
+ },
+ ]}
+ />
+ </div>
+ </div>
+ </section>
+ </main>
+ );
+}
diff --git a/website/components/site-footer.tsx b/website/components/site-footer.tsx
index b9139981..4459b79c 100644
--- a/website/components/site-footer.tsx
+++ b/website/components/site-footer.tsx
@@ -20,6 +20,7 @@ const COLUMNS: Column[] = [
{ label: "Session Replay", href: "/product/session-replay" },
{ label: "Exceptions / Stack Traces", href: "/product/stack-traces" },
{ label: "Monitors", href: "/product/monitors" },
+ { label: "SLOs", href: "/product/slos" },
{ label: "On-Call", href: "/product/on-call" },
{ label: "Fleet Overview", href: "/product/fleet" },
],
@@ -32,7 +33,10 @@ const COLUMNS: Column[] = [
{ label: "AI Tracing", href: "/product/ai-tracing" },
{ label: "Dashboards as Code", href: "/product/dashboards" },
{ label: "Performance", href: "/product/performance" },
- { label: "Flutter Session Replay", href: "/product/flutter-session-replay" },
+ {
+ label: "Flutter Session Replay",
+ href: "/product/flutter-session-replay",
+ },
{ label: "Symbolicator", href: "/product/symbolication" },
],
},
@@ -51,7 +55,11 @@ const COLUMNS: Column[] = [
heading: "Hosting",
links: [
{ label: "Cloud", href: "/cloud" },
- { label: "Self-host", href: "https://docs.tracewayapp.com", external: true },
+ {
+ label: "Self-host",
+ href: "https://docs.tracewayapp.com",
+ external: true,
+ },
],
},
{
@@ -68,7 +76,9 @@ const COLUMNS: Column[] = [
export function SiteFooter() {
const year = new Date().getFullYear();
return (
- <footer style={{ borderTop: "1px solid var(--hair)", padding: "50px 0 40px" }}>
+ <footer
+ style={{ borderTop: "1px solid var(--hair)", padding: "50px 0 40px" }}
+ >
<div className="wrap">
<div className="footer-grid grid gap-10 md:gap-9 grid-cols-2 md:grid-cols-[1.2fr_repeat(5,1fr)]">
<div className="col-span-2 md:col-span-1">
@@ -85,7 +95,8 @@ export function SiteFooter() {
className="mt-4 text-[13px] leading-relaxed max-w-xs"
style={{ color: "var(--fg-2)", fontFamily: "var(--font-mono)" }}
>
- Observability that tells you what to fix first. Open source. Self-host or cloud.
+ Observability that tells you what to fix first. Open source.
+ Self-host or cloud.
</p>
<div className="mt-5 flex items-center gap-2">
<Link
@@ -122,7 +133,9 @@ export function SiteFooter() {
<li key={l.href}>
<Link
href={l.href}
- {...(l.external ? { target: "_blank", rel: "noopener noreferrer" } : {})}
+ {...(l.external
+ ? { target: "_blank", rel: "noopener noreferrer" }
+ : {})}
className="text-[13px] inline-block py-1 transition-colors hover:text-[color:var(--fg-0)]"
style={{ color: "var(--fg-1)" }}
>
diff --git a/website/components/site-header.tsx b/website/components/site-header.tsx
index accb980f..dc43fbc4 100644
--- a/website/components/site-header.tsx
+++ b/website/components/site-header.tsx
@@ -21,6 +21,7 @@ import {
PhoneCall,
Radar,
Boxes,
+ Goal,
} from "lucide-react";
import { MobileNav } from "@/components/mobile-nav";
import { DiscordIcon } from "@/components/discord-icon";
@@ -72,6 +73,12 @@ const PILLARS: NavItem[] = [
href: "/product/monitors",
icon: Radar,
},
+ {
+ title: "SLOs",
+ description: "Error budgets and burn-rate alerts over any indicator.",
+ href: "/product/slos",
+ icon: Goal,
+ },
{
title: "On-Call",
description: "Rotating schedules, escalation policies, real paging.",
diff --git a/website/public/images/slos-detail-breached.png b/website/public/images/slos-detail-breached.png
new file mode 100644
index 00000000..5f72a47e
Binary files /dev/null and b/website/public/images/slos-detail-breached.png differ
diff --git a/website/public/images/slos-detail.png b/website/public/images/slos-detail.png
new file mode 100644
index 00000000..5dea38ff
Binary files /dev/null and b/website/public/images/slos-detail.png differ
diff --git a/website/public/images/slos-list.png b/website/public/images/slos-list.png
new file mode 100644
index 00000000..463f49b5
Binary files /dev/null and b/website/public/images/slos-list.png differ
diff --git a/website/public/images/slos-rule.png b/website/public/images/slos-rule.png
new file mode 100644
index 00000000..600f125f
Binary files /dev/null and b/website/public/images/slos-rule.png differ
diff --git a/website/public/images/slos-sessions.png b/website/public/images/slos-sessions.png
new file mode 100644
index 00000000..d4d4f1bf
Binary files /dev/null and b/website/public/images/slos-sessions.png differ
diff --git a/website/public/images/slos-sheet.png b/website/public/images/slos-sheet.png
new file mode 100644
index 00000000..d2b61f27
Binary files /dev/null and b/website/public/images/slos-sheet.png differ