The only tool you need to know what is happening and how to fix it.
Alerts
Alerts are how Traceway tells people that something happened. They live under Alerts in the project sidebar (/notifications), in three tabs kept in ?tab=:
| Tab | What it holds |
|---|---|
| Channels (default) | Where notifications go. Create a channel first, then attach rules to it. |
| Rules | When to notify. Each rule watches one condition and sends to exactly one channel. A channel can serve many rules. |
| History | Every notification that was sent, with its status |
Rules and channels belong to a project and only see that project's data. Channels have their own On/Off switch, and a rule only runs while its channel is on. Changing them needs write access on the project. Browser and mobile projects only offer the New Issue rule type and the two SLO rule types.
Channels
| Type | Configuration | What it sends |
|---|---|---|
| 1 to 10 recipient addresses | An HTML email per alert | |
| Slack | Incoming webhook URL, optional channel override and username | An attachment coloured by severity: blue info, orange warning, red critical, green recovery |
| Webhook | URL, method (POST or PUT), optional headers, optional HMAC secret | JSON with subject, body, severity, ruleType, ruleName, url, timestamp. With a secret, X-Traceway-Signature: sha256=<hex> |
| GitHub | Fine-grained token with Issues read and write, owner, repo, optional labels | A GitHub issue. These issues can start the Auto-Fix agent in the repository. |
| Pushover | User key and app token, optional device, priority (normal, high, or emergency with a retry of 30 s or more, an expiry up to 10800 s and an optional callback URL), sound, HTML formatting, TTL | A push notification with the dashboard link |
| Telegram | Bot token and chat id | A message, cut at 4096 characters with the link kept |
| Escalation policy | An escalation policy of the same organization | Nothing by itself: it opens an on-call page that escalates until someone acknowledges (on-call.md) |
SMS is not a channel type. It is a personal on-call contact method.
Every channel has a Test button that sends "Traceway Test Notification". On an escalation channel, Test opens a real page and notifies whoever is on call. Resolve that page when done. Testing again while it is still open is refused.
Alert subjects look like [<Project>] New error: TypeError or [<Project>] Check "api" is down. Exception alerts carry the exception id, hash, occurred-at time, app version, server, the endpoint or task, the stack trace, the attributes and a link to /issues/<hash>. Those three fields (hash, exception id, occurred at) are what an agent needs to open the occurrence (see the notifications knowledge in the read skill).
Rule settings shared by every type
| Setting | Meaning |
|---|---|
| Name | Up to 200 characters, shown in every notification |
| Channel | Exactly one |
| Cooldown | Minimum time between notifications for the same condition. Default 15 minutes. |
| Severity | Auto (derived from how bad the condition is) or a fixed Critical, Warning or Info. Recovery notices are always Info. |
| Enabled | New rules start enabled. A rule runs only while both it and its channel are enabled. |
Rule types
The numbers in brackets are what the evaluator uses when a setting is missing. The New Rule form starts every type at the same values (5%, 5 minutes, 10 requests, 500 ms, *), so set each field on purpose.
| Rule | Fires when | Settings | Auto severity |
|---|---|---|---|
| New Issue | An exception hash is seen for the first time, or again for the first time after it was archived | Ignore patterns: comma-separated, case-insensitive substrings matched against the error type (the first stack line up to its first :). * is stripped, not a wildcard. |
Critical |
| Error Regression | An archived (resolved) issue occurs again | None | Critical |
| Error Rate | The 5xx share across all endpoints exceeds a percentage | Threshold 5%, lookback 5 min, min requests 10 | Warning, Critical at 2x |
| Error Count | Errors in the window reach a count (captured messages are not counted) | Threshold 100, lookback 60 min | Warning, Critical at 5x |
| Endpoint Error Rate | One endpoint's 5xx share exceeds a percentage | A specific endpoint (* matches nothing here), threshold 2%, lookback 10 min, min requests 20 |
Warning, Critical at 2x |
| Endpoint P95 / Endpoint P99 | The percentile exceeds a threshold | Endpoint or * for all, threshold 500 ms, lookback 5 min |
Warning |
| Apdex Drop | Apdex falls below a threshold | Threshold 0.85, lookback 15 min, min requests | Warning, Critical below 0.5 |
| Throughput Drop | Requests drop versus the baseline window just before, scaled to the same length. No baseline traffic, no alert. | Drop 50%, lookback 15 min, baseline 60 min | Warning, Critical at 80% or more |
| Impact Score Critical / High / Medium | An endpoint newly enters the level (0.75, 0.50, 0.25) over the trailing 24 hours. Fires once on entry, again only after it leaves and comes back. The first run after a restart or edit only records the baseline. | Min requests 50 | Critical / Warning / Info |
| Task Duration | A task's P95 duration exceeds a threshold | Task or *, threshold 30000 ms, lookback 30 min |
Warning |
| Task Failure Rate | The share of task runs with an exception on their own span exceeds a percentage | Task or *, threshold 10%, lookback 60 min, min executions 5 |
Warning, Critical at 2x |
| Metric Threshold | An aggregated metric violates a comparison. No data in the window, no alert. | Metric, aggregation (avg, max, min, sum, p95, p99, last), operator (gt, gte, lt, lte, eq), value 90, lookback 5 min | Warning, Critical when 20% past the value |
| No Data | No telemetry of a kind for N minutes. any fires only when all four kinds are silent. |
Data type (any, endpoints, exceptions, metrics, tasks), 10 min | Critical |
| AI Trace Cost | One AI call costs at least the threshold | Trace name or *, $0.50 |
Warning, Critical at 3x |
| AI Conversation Cost | One conversation's cost over 24 hours reaches the threshold | $5 | Warning, Critical at 3x |
| AI Flagged Content | An AI call matches a flagged term. The terms come from the project's AI settings. | Optional term list (empty means any flagged match) | Warning |
| Monitor Down | Any monitor in the project goes down, plus a recovery notice when it comes back | None | Critical, recovery Info |
| SLO Burn Rate | An SLO spends its error budget faster than the threshold over both the long and the short window, plus a recovery notice once either window calms down. Fires on the transition only. | SLO, long window 60 min, short window 5 min, threshold 14.4x; presets Fast 14.4x 1h/5m, Slow 6x 6h/30m, Long 1x 3d/6h; both windows round up to whole slices of a metric SLO | Warning, Critical at 2x the threshold, recovery Info |
| SLO Budget Consumed | An SLO has used at least a share of its window's error budget, plus a recovery notice when the budget recovers. Fires on the transition only. | SLO, consumed 80% | Warning, Critical at 100%, recovery Info |
The aggregation in Metric Threshold spans every series of the metric with no tag filter, so prefer max (or min for lower-is-bad metrics) where one bad host could hide in an average.
Apdex uses fixed bands: satisfied up to 750 ms, tolerating up to 1500 ms, anything slower or a 5xx is frustrated. Apdex = (satisfied + tolerating / 2) / total.
When rules run
- On ingest: New Issue, Error Regression, AI Trace Cost, AI Conversation Cost, AI Flagged Content. These fire within seconds.
- On state change: Monitor Down, the moment a probe flips the monitor.
- Polled: everything else, every 60 seconds by default (self-hosted:
NOTIFICATION_POLL_SECONDS). - SLO rules are polled too, but read the SLO's precomputed buckets (refreshed every
SLO_POLL_SECONDS, 60 by default) instead of raw telemetry, and they are evaluated during their cooldown so a recovery notice is never delayed by it. A recovery notice does not restart the cooldown.
Cooldowns differ by kind:
- Polled rules have one cooldown per rule. After it fires, it is not evaluated again until the cooldown ends. This cooldown survives restarts.
- Ingest and monitor rules cool down per item: per exception hash, trace name, conversation or monitor. Two different new errors a minute apart both alert. The same error does not alert again within its cooldown. This memory is in process only, so a restart can let the same item alert once more.
- The cooldown cannot be 0. Empty or 0 becomes 15 minutes.
- Changing a rule's type, channel or condition resets its cooldown and its per-item memory, so the new settings apply at once. Renaming it, changing its cooldown or severity, or toggling it does not.
Delivery goes through a durable queue. Failed sends are retried after 1, 5, 15 and 60 minutes, five attempts in all. A queued message keeps the channel settings it was sent with.
Disable versus snooze
- Disable turns a rule off until someone turns it back on. The dashboard asks for confirmation.
- Snooze mutes a rule for a set time and it comes back on its own. The dashboard offers 30 minutes and 1, 4, 8 or 24 hours. Clearing the snooze turns it back on early.
- A snoozed or disabled rule is not evaluated at all, so nothing is queued while it is muted and it does not catch up afterwards.
During a known incident or a deploy, snooze the noisy rule rather than disabling it: a snooze cannot be forgotten in the off position.
The Rules table shows each rule's status as On, Off or Snoozed.
History
The History tab lists every notification with severity, rule, subject (linking to the page it is about), channel, status and time. It has search and a time range. Statuses:
| Status | Meaning |
|---|---|
| Sent | Delivered. On an escalation channel, a page was opened. |
| Failed | All retries failed. The error is kept. |
| Deduped | An escalation alert folded into a page that was already open |
History opens on the last 7 days.
Reading and changing it from an agent
| Task | CLI | MCP tool |
|---|---|---|
| List rules with enabled and snoozed state | traceway alerts list |
list_notification_rules |
| Turn a rule on or off (write) | traceway-write alerts enable <id> / disable <id> |
set_notification_rule_enabled |
| Snooze a rule (write) | traceway-write alerts snooze <id> --minutes N (0 clears) |
snooze_notification_rule |
Creating and editing rules and channels, the History tab and the Test buttons stay in the dashboard. Channels hold credentials and webhook URLs.
To resolve an alert someone pasted, the read skill's notifications section explains how to go from an exception or performance alert to the data behind it.