The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
fix: send monitor recovery notices as info and colour them green in Slack (#385)
FrameAutomata2cfb5b7915f6
diff --git a/CLAUDE.md b/CLAUDE.mdindex c2981aca..7ed04bc4 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -832,7 +832,7 @@ Org-scoped entities in the main DB. Teams (`teams`/`team_members`) own projects**Monitors** (user-facing name for synthetic uptime checks; engine in `backend/app/synthetics`, frontend under `/monitors`, see the SYNTHETICS_* env block)-Checks (`synthetic_checks`, main DB) are http/tcp/browser probes with per-check interval, timeout, and a consecutive-failure threshold (flap damping). Runs flow through the `check_runs` queue (outbox-style guarded claims, terminal rows deleted, expired queued runs recorded as `missed`); results are telemetry (`check_results`, all three backends, pruned by the SQLite retention worker / 90d CH TTL). State transitions open/resolve `check_incidents` and feed the event-driven notification rule type `check_down` (recovery auto-resolves the page a rule-scoped dedup key opened via `oncall.AutoResolveByDedupKey`; recovery never dispatches to escalation channels). The notify hook runs post-commit with no ambient tx (SQLite single-connection). Remember: `notification_rule.repository.go` (both copies) lists event rule types explicitly, and `check_down` is one of them.+Checks (`synthetic_checks`, main DB) are http/tcp/browser probes with per-check interval, timeout, and a consecutive-failure threshold (flap damping). Runs flow through the `check_runs` queue (outbox-style guarded claims, terminal rows deleted, expired queued runs recorded as `missed`); results are telemetry (`check_results`, all three backends, pruned by the SQLite retention worker / 90d CH TTL). State transitions open/resolve `check_incidents` and feed the event-driven notification rule type `check_down` (recovery auto-resolves the page a rule-scoped dedup key opened via `oncall.AutoResolveByDedupKey`; recovery never dispatches to escalation channels). The recovery notice carries `Recovered` on `NotificationMessage`: `dispatch` leaves its `info` severity alone instead of applying the rule's, and the Slack adapter colours it green. The notify hook runs post-commit with no ambient tx (SQLite single-connection). Remember: `notification_rule.repository.go` (both copies) lists event rule types explicitly, and `check_down` is one of them.**Incidents, incident updates & post-mortems.** `check_incidents` covers both auto and manual incidents: auto rows carry `check_id`+`project_id` (nullable now), hand-recorded ones carry only `status_page_id` (org admins record them from a status page for outages monitors missed; they never affect uptime numbers). All incidents can be given a public `title` and statuspage.io-style timeline updates (`incident_updates`: status investigating/identified/monitoring/update/resolved + message; auto open/resolve seeds empty-message investigating/resolved updates in the same tx in `synthetics/result.go`, and the probe `error_message` stays internal, never in the public payload). A `resolved` update closes a manual incident; auto lifecycle stays owned by `ProcessOutcome` (manual resolve/time-edit/delete of auto incidents answer 422). The public `/api/status/:slug` payload gained `pastIncidents` (90d union of the page's checks' auto incidents + its manual ones, titles defaulting to "<check name> is down", nested updates, cap 30, no ids/error messages); incident mutations invalidate every status-page cache slug of the org. Post-mortems (`post_mortems`, **project-scoped**: `project_id` NOT-NULL-in-practice with ON DELETE CASCADE, backfilled by migration from the linked incident's project else the org's oldest project) are internal markdown documents, never public: optional 1:1 incident link (`incident_id` unique where set, `ON DELETE SET NULL` so documents survive monitor/incident deletion; linkable incidents are the project's own autos plus the org's manual status-page incidents — cross-project autos answer 422), JSON-array `tags` (models.StringSlice), LIKE search over title/content/tags, and an append-only edit log (`post_mortem_events`: action created/updated + JSON `changes` field list, recorded only when something actually changed; served as the Activity panel). Routes are project-scoped (`/api/post-mortems*`, standard RequireProjectAccess/RequireWriteAccess chains, org resolved from the project for incident validation); incident mutations are org-admin since they are public content. Frontend: Post-Mortems tab on `/monitors` (all members; creation is a title-only dialog that creates the document immediately and opens the editor page), full-page WYSIWYG editor at `/monitors/post-mortems/[id]` (`@milkdown/crepe`, dynamically imported, app font stack overrides the theme's serif), paginated incident management page at `/monitors/status-pages/[pageId]/incidents` (which owns the Record Incident dialog).diff --git a/backend/app/models/notification_message.model.go b/backend/app/models/notification_message.model.goindex 77c59c4c..fd5616ae 100644--- a/backend/app/models/notification_message.model.go+++ b/backend/app/models/notification_message.model.go@@ -23,6 +23,10 @@ type NotificationMessage struct {Email *NotificationEmail `json:",omitempty"`+ // Recovered marks the notice that closes an alert. Severity cannot carry+ // this: info is shared with test messages and info-severity rules.+ Recovered bool `json:",omitempty"`+// DedupToken is the stable identity of what fired within the rule// (exception hash, endpoint, task or metric name; empty for rule-level// conditions). Page dedup keys are built from it — never from URL, whichdiff --git a/backend/app/notifications/adapter_slack.go b/backend/app/notifications/adapter_slack.goindex de9b755a..18c41d0b 100644--- a/backend/app/notifications/adapter_slack.go+++ b/backend/app/notifications/adapter_slack.go@@ -33,6 +33,9 @@ func (a *SlackAdapter) Send(ctx context.Context, msg Message) error {case SeverityCritical:color = "#F44336"}+ if msg.Recovered {+ color = "#4CAF50"+ }username := a.Usernameif username == "" {diff --git a/backend/app/notifications/adapter_slack_test.go b/backend/app/notifications/adapter_slack_test.gonew file mode 100644index 00000000..97d333ae--- /dev/null+++ b/backend/app/notifications/adapter_slack_test.go@@ -0,0 +1,61 @@+package notifications++import (+ "context"+ "encoding/json"+ "net/http"+ "net/http/httptest"+ "testing"++ "github.com/tracewayapp/traceway/backend/app/models"+)++func sentSlackColor(t *testing.T, msg Message) string {+ t.Helper()+ type attachment struct {+ Color string `json:"color"`+ }+ attachments := make(chan []attachment, 1)+ receiver := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ var payload struct {+ Attachments []attachment `json:"attachments"`+ }+ _ = json.NewDecoder(r.Body).Decode(&payload)+ attachments <- payload.Attachments+ }))+ defer receiver.Close()++ if err := (&SlackAdapter{WebhookURL: receiver.URL}).Send(context.Background(), msg); err != nil {+ t.Fatalf("send to Slack: %v", err)+ }+ got := <-attachments+ if len(got) != 1 {+ t.Fatalf("expected 1 attachment, got %d", len(got))+ }+ return got[0].Color+}++// The recovery rows run on every build tag; the dispatch test that covers the+// same color end to end only builds on the default tags.+func TestSlackColor(t *testing.T) {+ recovered := buildCheckRecoveredMessage(&models.SyntheticCheck{Id: 7, Name: "api health"}, "api")+ relabelled := recovered+ relabelled.Severity = SeverityCritical+ cases := []struct {+ name string+ msg Message+ want string+ }{+ {"critical", Message{Severity: SeverityCritical}, "#F44336"},+ {"warning", Message{Severity: SeverityWarning}, "#FF9800"},+ {"info", Message{Severity: SeverityInfo}, "#2196F3"},+ {"no severity", Message{}, "#2196F3"},+ {"recovery", recovered, "#4CAF50"},+ {"recovery with a critical severity", relabelled, "#4CAF50"},+ }+ for _, tc := range cases {+ if got := sentSlackColor(t, tc.msg); got != tc.want {+ t.Errorf("%s: color = %s, want %s", tc.name, got, tc.want)+ }+ }+}diff --git a/backend/app/notifications/dispatch.go b/backend/app/notifications/dispatch.goindex cc875def..893f905e 100644--- a/backend/app/notifications/dispatch.go+++ b/backend/app/notifications/dispatch.go@@ -42,7 +42,8 @@ func dispatch(rule *models.NotificationRuleWithChannel, msg Message) bool {msg.RuleType = rule.RuleTypemsg.RuleName = rule.Name- if rule.Severity != "" {+ // The rule's severity grades the alert, not the notice that closes it.+ if rule.Severity != "" && !msg.Recovered {msg.Severity = Severity(rule.Severity)}msg.URL = dashboardURL(msg.URL)diff --git a/backend/app/notifications/dispatch_outbox_test.go b/backend/app/notifications/dispatch_outbox_test.goindex 37954f03..7a0bf181 100644--- a/backend/app/notifications/dispatch_outbox_test.go+++ b/backend/app/notifications/dispatch_outbox_test.go@@ -205,3 +205,38 @@ func TestDispatchDashboardLinks(t *testing.T) {})}}++// Issue #383: a recovery took the rule's severity, so it reached Slack in the+// alert's color; under Auto it was plain info blue.+func TestDispatchedCheckRecoveryIsInfoAndGreenInSlack(t *testing.T) {+ check := &models.SyntheticCheck{Id: 7, Name: "api health"}+ recovered := buildCheckRecoveredMessage(check, "api")+ cases := []struct {+ name string+ ruleSeverity string+ msg Message+ wantSeverity Severity+ wantColor string+ }{+ {"recovery under a critical rule", "critical", recovered, SeverityInfo, "#4CAF50"},+ {"recovery under an Auto rule", "", recovered, SeverityInfo, "#4CAF50"},+ {"alert under a warning rule", "warning", buildCheckDownMessage(check, "connection refused", "api"), SeverityWarning, "#FF9800"},+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ fixture := setupDispatchDB(t)+ fixture.Rule.RuleType, fixture.Rule.Severity = "check_down", tc.ruleSeverity++ if !dispatch(fixture.Rule, tc.msg) {+ t.Fatal("dispatch failed")+ }+ msg := queuedMessage(t)+ if msg.Severity != tc.wantSeverity {+ t.Errorf("queued severity = %q, want %q", msg.Severity, tc.wantSeverity)+ }+ if color := sentSlackColor(t, msg); color != tc.wantColor {+ t.Errorf("Slack color = %s, want %s", color, tc.wantColor)+ }+ })+ }+}diff --git a/backend/app/notifications/messages.go b/backend/app/notifications/messages.goindex e703398a..0cc6a992 100644--- a/backend/app/notifications/messages.go+++ b/backend/app/notifications/messages.go@@ -523,6 +523,7 @@ func buildCheckRecoveredMessage(check *models.SyntheticCheck, projectName stringSubject: fmt.Sprintf("[%s] Check %q recovered", projectName, check.Name),Body: fmt.Sprintf("The synthetic check %q is passing again.", check.Name),Severity: SeverityInfo,+ Recovered: true,URL: fmt.Sprintf("/monitors/%d", check.Id),DedupToken: fmt.Sprintf("check:%d", check.Id),Email: &models.NotificationEmail{diff --git a/cli/test/contract/go.mod b/cli/test/contract/go.modindex 7d6adda2..c850b309 100644--- a/cli/test/contract/go.mod+++ b/cli/test/contract/go.mod@@ -59,7 +59,7 @@ require (github.com/go-chi/chi/v5 v5.2.2 // indirectgithub.com/go-faster/city v1.0.1 // indirectgithub.com/go-faster/errors v0.7.1 // indirect- github.com/go-logr/logr v1.4.3 // indirect+ github.com/go-logr/logr v1.4.4 // indirectgithub.com/go-logr/stdr v1.2.2 // indirectgithub.com/go-ole/go-ole v1.3.0 // indirectgithub.com/go-playground/locales v0.14.1 // indirect@@ -112,11 +112,11 @@ require (github.com/zeebo/xxh3 v1.1.0 // indirectgo.opentelemetry.io/auto/sdk v1.2.1 // indirectgo.opentelemetry.io/contrib/instrumentation/host v0.69.0 // indirect- go.opentelemetry.io/otel v1.44.0 // indirect- go.opentelemetry.io/otel/metric v1.44.0 // indirect- go.opentelemetry.io/otel/sdk v1.44.0 // indirect- go.opentelemetry.io/otel/sdk/metric v1.44.0 // indirect- go.opentelemetry.io/otel/trace v1.44.0 // indirect+ go.opentelemetry.io/otel v1.45.0 // indirect+ go.opentelemetry.io/otel/metric v1.45.0 // indirect+ go.opentelemetry.io/otel/sdk v1.45.0 // indirect+ go.opentelemetry.io/otel/sdk/metric v1.45.0 // indirect+ go.opentelemetry.io/otel/trace v1.45.0 // indirectgo.opentelemetry.io/proto/otlp/collector/profiles/v1development v0.3.0 // indirectgo.opentelemetry.io/proto/otlp/profiles/v1development v0.3.0 // indirectgo.tracewayapp.com v1.0.4 // indirect@@ -129,7 +129,7 @@ require (golang.org/x/net v0.56.0 // indirectgolang.org/x/oauth2 v0.36.0 // indirectgolang.org/x/sync v0.21.0 // indirect- golang.org/x/sys v0.46.0 // indirect+ golang.org/x/sys v0.47.0 // indirectgolang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 // indirectgolang.org/x/text v0.39.0 // indirectgolang.org/x/tools v0.47.0 // indirectdiff --git a/cli/test/contract/go.sum b/cli/test/contract/go.sumindex 99535fa8..b99ceaeb 100644--- a/cli/test/contract/go.sum+++ b/cli/test/contract/go.sum@@ -121,8 +121,8 @@ github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vgithub.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=-github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=-github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=+github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=+github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=@@ -311,18 +311,18 @@ go.opentelemetry.io/contrib/instrumentation/host v0.69.0 h1:N/wKAdIOKd9U4bwiwY3jgo.opentelemetry.io/contrib/instrumentation/host v0.69.0/go.mod h1:rE1Bxmu3mwcmyTJV4EQdJ/C6zCUnuT9jCiinhRXLAXc=go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 h1:F7Jx+6hwnZ41NSFTO5q4LYDtJRXBf2PD0rNBkeB/lus=go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0/go.mod h1:UHB22Z8QsdRDrnAtX4PntOl36ajSxcdUMt1sF7Y6E7Q=-go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=-go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=-go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=-go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=-go.opentelemetry.io/otel/metric/x v0.66.0 h1:YkCrx1zLOChi9ZcZ6euupOcsgzbVlec7D/xoEU1+cTA=-go.opentelemetry.io/otel/metric/x v0.66.0/go.mod h1:d1+BDj9t96do0/1LoU1ayfCv79ZgNE41qbhBvnMOBZk=-go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=-go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=-go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=-go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=-go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=-go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=+go.opentelemetry.io/otel v1.45.0 h1:pdrWmLHofpubmArBv1LgFSv1Z0Ie/ppdZzu+kUN5EeU=+go.opentelemetry.io/otel v1.45.0/go.mod h1:XZxIqPapzEYnhNSScF5DIqXhm/rYi0FzCe2XddAwZfQ=+go.opentelemetry.io/otel/metric v1.45.0 h1:7Eg1uH7CJ5cXv9is6tnBe1FI6rj1nwUdbFypRm3br/M=+go.opentelemetry.io/otel/metric v1.45.0/go.mod h1:HAPbm1nd3p1PmFH7v2dR+6BjXxw+Lq4a2+pndMAm08s=+go.opentelemetry.io/otel/metric/x v0.67.0 h1:PcicCNZFkZ4bXfSooXdo3WN7RBOVOtjVdo1wD358Uns=+go.opentelemetry.io/otel/metric/x v0.67.0/go.mod h1:FBjCWZe6wgcqxcMtjdGiClDKXb2YxxXii0CXftE4QtI=+go.opentelemetry.io/otel/sdk v1.45.0 h1:4VVSMgQ83dUgW2aoX5f6JgLvHwIvzcuLnF9lUdCSpCw=+go.opentelemetry.io/otel/sdk v1.45.0/go.mod h1:Sr40LgXV7DsKMMJMKOhUWOgMWTfAaqvm2kF0g7ilwuA=+go.opentelemetry.io/otel/sdk/metric v1.45.0 h1:oVFszMfyj1Am6s24Vtc7wBb8BKLcwepJjNEYILuiE3o=+go.opentelemetry.io/otel/sdk/metric v1.45.0/go.mod h1:vUWUxDZvu1WVRj8JA8S0AdhsPrZoDpA2DdZauIh4mDA=+go.opentelemetry.io/otel/trace v1.45.0 h1:l/mP6Uv7oNO7/TblbhpbgMidxhq1uO/rPsikOyVhxag=+go.opentelemetry.io/otel/trace v1.45.0/go.mod h1:qoJJA2xNMnxRrdISU/kLtfUH2wNeQbiv+jhs/CxI8bc=go.opentelemetry.io/proto/otlp v1.10.0 h1:IQRWgT5srOCYfiWnpqUYz9CVmbO8bFmKcwYxpuCSL2g=go.opentelemetry.io/proto/otlp v1.10.0/go.mod h1:/CV4QoCR/S9yaPj8utp3lvQPoqMtxXdzn7ozvvozVqk=go.opentelemetry.io/proto/otlp/collector/profiles/v1development v0.3.0 h1:AWBLtJn3ajH399gyOuJdqPsC9w152x3xGVVO/UK3vyM=@@ -376,8 +376,8 @@ golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wgolang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=-golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=-golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=+golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=+golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 h1:nwGZBCt+FnXUrGsj5vjzAsEmkcaFvd82BbOjECiFYZc=golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57/go.mod h1:3AWMyWHS+caVoiEXpiq6+tzKA40J4vQT3MYr80ZtQpc=golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=diff --git a/docs/pages/learn/alerts.mdx b/docs/pages/learn/alerts.mdxindex 7ecb4345..58b5e1ba 100644--- a/docs/pages/learn/alerts.mdx+++ b/docs/pages/learn/alerts.mdx@@ -312,7 +312,7 @@ Fires when an [AI conversation](/learn/ai-tracing#content-flagging) matches a fl### Monitor Down-Fires when a [monitor](/learn/monitors) transitions to down, which happens only after its configured number of consecutive failures, and sends a recovery notice when it comes back up. One rule covers every monitor in the project; the alert names the monitor and carries the last probe error.+Fires when a [monitor](/learn/monitors) transitions to down, which happens only after its configured number of consecutive failures, and sends a recovery notice when it comes back up. One rule covers every monitor in the project; the alert names the monitor and carries the last probe error. The recovery notice is always Info, whatever severity the rule sets, and is green in Slack.There are no rule-specific options. Pointing the rule at an escalation channel opens an [on-call page](/learn/on-call) instead of sending a message: the page deduplicates per monitor, re-fires never restart its escalation clock, and recovery resolves it automatically.@@ -387,6 +387,7 @@ Each rule has a **Severity** setting that controls the severity attached to its| Task Failure Rate | Warning; Critical at twice the threshold || Metric Threshold | Warning; Critical when the value deviates from the threshold by more than 20% || AI Trace Cost | Warning; Critical at three times the threshold |+| Monitor Down | Critical; the recovery notice is Info |You can override the automatic severity by setting it explicitly on a rule to **Critical**, **Warning**, or **Info**. This is useful when you want a normally-warning condition to page as critical, or to downgrade noisy rules to info.