Skip to content

Enterprise agent platform for building, deploying, and operating AI teammates.

22.3 KB
Raw Blame

Execution events: observability runbook

This runbook is for on-call staff running tasks on the V2 execution-event log (task_execution_events, enabled per task by Task.conversation_storage_version = 2). It lists the signals that exist today, which problem class each one covers, and how to triage a single task with the admin diagnostic endpoint.

Read-budget signals do not exist yet. Further signals will be added; this document lists only what the code emits now.

1. Turning signals on

Runtime metrics (OTLP/HTTP)

Metrics are exported only when export is enabled and an endpoint is resolved.

Variable Meaning
XAGENT_RUNTIME_TELEMETRY_ENABLED Explicit on/off. When unset, export is enabled automatically if any OTLP metrics endpoint below is configured. false is a kill switch.
XAGENT_OTEL_EXPORTER_OTLP_METRICS_ENDPOINT Full OTLP/HTTP metrics URL, including /v1/metrics.
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_ENDPOINT Standard fallbacks, in that order. The base endpoint gets /v1/metrics appended.
XAGENT_OTEL_EXPORT_INTERVAL_MILLISECONDS (fallback OTEL_METRIC_EXPORT_INTERVAL) Export interval, default 10000.
XAGENT_OTEL_SERVICE_NAME (fallback OTEL_SERVICE_NAME) service.name resource attribute, default xagent.

Set the same variables for every process that should report: the web process and the worker processes (XAGENT_TASK_EXECUTION_ROLE=worker, or the workers started by the combined launcher, which inherit the environment).

Both the web application and the worker entrypoint (xagent.web.worker) start the exporter. In a split deployment the workers record the execution-side counters (trace writes, write retries), so configure export on them, not only on web processes. Check the startup log of each process for Runtime performance telemetry OTLP export is enabled; a process that does not print it is not exporting.

Per-minute runtime diagnostics log line

Every app process (combined or web role) and every worker logs one INFO line per 60 s window, with no switch:

runtime diagnostics role=worker pid=4242 window_s=60.0 inflight=1 resume_inflight=0 started_1m=2 finished_1m=1 ok=1 failed=0 cancelled=0 gate_held_s=0.0 slot_tenants=1 slot_top=1 loop_lag_max_ms=12.3 loop_lag_samples=0 degradations=-

A pid that stops appearing for two windows is gone or has a blocked event loop. failed counts runs that raised or whose runner handled a setup or run exception; it describes how the execution ended in that process, not the task's stored status. Field definitions are in xagent/web/services/process_diagnostics.py.

/health degradations

GET /health stays 200 {"status": "ok"} while degraded and adds a sorted degradations list of signal names. It is unauthenticated, so names only; the detail strings (which can name a task) are in the logs and in the admin endpoint below. The registry is per process: /health on the web process says nothing about the workers. A worker's active degradations appear, names only, at the end of its per-minute runtime diagnostics line (degradations=, - when none).

execution_events_unobserved is set at startup, in the web process and in each worker, when XAGENT_EXECUTION_EVENT_STORAGE_ENTRIES enables at least one entry (independent of XAGENT_EXECUTION_EVENT_STORAGE_USER_IDS) but that process is not exporting runtime telemetry; the process also logs a WARNING naming the entries. V2 tasks would then run without the metrics in this document. It is never cleared at runtime. To resolve it, enable runtime telemetry export on that process, or remove the entries and restart. Limitation: after the entries are removed the signal disappears, although existing V2 tasks may still be continued, so keep telemetry enabled for as long as V2 tasks can be resumed.

2. Existing signals by problem class

Metric names below are OpenTelemetry instrument names. Log messages are the literal format strings.

(1) V2 event commit failures

  • xagent.execution_event.commit.failures, attributes site (trace_fact, outbound, input_accept, terminal_failure), error.type (the first cause that is not the persistence wrapper, e.g. OperationalError, ExecutionEventConflict, ToolAttemptAlreadyStartedError, TaskLeaseLostError) and storage="v2". One count per failed write, after retries are used up or for a failure that is not retryable.
  • With it, one ERROR line per failed write: execution_event_commit_failed task_id=... run_id=... storage=v2 component=execution-events site=... error_type=... sqlstate=... retried=..., plus tool_attempt_id=... for tool events. It carries no exception text; this is the only ERROR a failed V2 write logs.
  • xagent.execution_event.write.retries, attributes site and outcome: retried (an attempt was rerun), recovered (a rerun succeeded) or failed (a write that was retried at least once ended in failure). Each retryable trace-fact attempt also logs WARNING Trace event commit attempt failed for task %s: %s. The settlement sites (settlement, managed_settlement) read a failed COMMIT back by its result fact first and rerun only when it did not apply; a confirmed one is settlement_ack_reconciled in (2).
  • xagent.trace.database.writes, attributes outcome (succeeded, retryable_failed, failed) and storage (v1/v2).
  • After a failed write the run is aborted and settled by the existing failure path.

(2) Unknown outcomes

  • xagent.execution_event.unknown_outcomes, attributes operation (the kind) and storage (v1, v2, or unknown where the storage version is not in scope). One count per occurrence, with one line execution_event_outcome_unknown task_id=... run_id=... storage=... component=execution-events kind=... ref=.... Kinds:
    • tool_effect: a run was settled as an unknown tool effect; ref is the tool attempt id. Reported once, after the settlement committed (a settlement that rolls back reports nothing; lease recovery reports it later).
    • turn_commit, command_commit: a turn or command acceptance whose COMMIT outcome could not be reconciled; ref is the turn id.
    • finalize_ack_unresolved: a finalize COMMIT (a new or a resumed run's result) lost its acknowledgement (dropped connection) and the read-back could not decide, or there was no inserted row to read back (a legacy wait or pause with no answer, a legacy completion with an empty answer, an ownerless legacy task, a replayed result fact).
    • settlement_ack_unresolved: the same for a V2 settlement (a run that raised, an interruption pause, or a managed inline-transport result). The lease is retained for TTL recovery.
    • settlement_deferred: settlement could not decide from the checkpoint and left the lease to TTL recovery.
    • finalize_ack_reconciled, settlement_ack_reconciled (INFO): a lost acknowledgement the read-back proved applied. A confirmed success counted for visibility; exclude both from alerts on this counter.
  • WARNING Settlement unknown-effect classification unavailable: task_id=... and WARNING Event recovery unavailable: task_id=...: the classification read itself failed; the degradation checkpoint_recovery_unavailable is set.
  • When lease recovery fails a task for this reason, the task's error_message reads Task execution lease expired with an unknown tool effect; automatic replay is unsafe. (or ... stopped with an unknown tool effect ... when the run settled itself). The admin endpoint reports only has_error_message.
  • A start fact exists for an attempt id that is started again: ToolAttemptAlreadyStartedError (Tool attempt already started; result reconciliation required) is raised rather than re-executing the tool.
  • Use the open_tool_attempts section of the admin endpoint to see every attempt in a task's log that started without a committed outcome.

(3) Reconnect duplicates and gaps

  • xagent.websocket.trace.events, attribute outcome: broadcast, dropped (no frame produced for the event), no_audience (no client attached, nothing sent), integrity_gap.
  • integrity_gap means the live display could not be built from the stored fact. It comes with WARNING Live display integrity gap task_id=%s event_id=%s (with traceback) and a client-safe error event to the task's clients.
  • A client attaching later replays history from storage, up to a committed sequence horizon. The connection is registered before the replay, so live frames and the replay can overlap; the client de-duplicates by execution_sequence.

Replay and delivery consistency

  • INFO execution_event_replay task_id=%s storage=v2 component=execution-events horizon=%s events=%s cache_hit=%s duration_ms=%.1f, once per V2 historical replay (initial attach and each status_request), logged before the replay is sent. horizon is the last committed sequence the replay covers, events the number of display events the replay carries, cache_hit whether it came from the history cache, duration_ms the load time.
  • xagent.websocket.replay.consistency, attributes outcome and storage=v2, once per closed V2 connection that received a replay. From the start of its first replay (before the horizon is read) each socket records the display frames it was actually sent after the horizon: event id, sequence and local send time, no payload, at most 5000 per socket. Frames at or below the horizon are not recorded; the replay covers them. On disconnect, a background check reads the committed root-scope display events above the horizon and up to the end of the checked range, and compares their event ids with the sent ones. That read is measured as a display read (see (5)).
    • The checked range ends at the highest sequence sent at least 5 seconds before the disconnect, on this process's monotonic clock. Producers can send frames out of sequence order (the trace handler hops to a worker thread after its fact commits), so a frame sent in the last 5 seconds may still have earlier-sequenced frames in flight. occurred_at is not used: it is producer time, possibly another host's clock.
    • Only kinds whose live frame is always sent are expected. Accepted inputs (input_accepted, the user bubble) get a live frame only when the input is injected; a rejected or deferred input never does. Assistant chat messages (assistant_message) get one only at settlement through the terminal result; a cancellation settlement sends none. The next replay carries both.
    • Only complete, missing and duplicate come from a comparison:
      • complete: every expected event in the range was sent at least once (live or by a later replay). It is not evidence that frames committed after the end of the range reached the socket (tail blind spot); the next attach replays those.
      • missing: at least one was never sent. Comes with WARNING execution_event_delivery_gap task_id=%s storage=v2 component=execution-events horizon=%s missing=%s first_missing_sequence=%s.
      • duplicate: nothing missing, but the same frame after the horizon was sent live twice. Informational: the client drops it. Frames resent by a later replay or by the shared-mode stream reconciliation (settlement frames every 5 seconds), and live frames overlapping the first replay, are not counted.
    • idle: nothing after the horizon was sent before the last 5 seconds, so there was nothing to compare (reported as duplicate instead if a live duplicate was seen).
    • skipped: more than 256 checks were pending (a mass disconnect); this connection was not checked.
    • overflow: the socket was sent more than 5000 display frames after the horizon; recording stopped and the connection is not checked. Sequence gaps are expected and never counted as missing: internal kinds, child scopes, audit-only facts and facts whose display could not be built (on the live path a ValueError there is reported as integrity_gap above) produce no frame. A connection moved to another task drops its record. If the check itself fails, for example because the database cannot be read, it logs WARNING execution_event_delivery_check_failed task_id=%s storage=v2 component=execution-events horizon=%s error_type=%s and emits no outcome. Checks stop at shutdown.
  • To read them: a rising missing rate is the signal to act on. Take the task id and first_missing_sequence from the WARNING and compare with the admin endpoint below (the event at that sequence should be a display kind in scope root). In shared mode, look for task_event_bridge_gap or publish_failed around the same time first.
  • xagent.task_event.bridge.frames (shared mode, cross-process Redis bridge), attribute outcome:
    • duplicate_dropped: the receiver got a frame it had already delivered and dropped it.
    • gap_resync: the receiver saw a jump in a publishing host's sequence and asked its clients to resync. Comes with WARNING task_event_bridge_gap task_id=%s host=%s previous_sequence=%s sequence=%s. Bridge sequences are per publishing host, so the frames in the gap may belong to other tasks. Each receiving host counts the gap it sees, so one lost frame counts once per receiver, and a burst right after a Redis resubscribe is expected.
    • publish_failed: a frame could not be published (the existing WARNING Task event publication failed task_id=%s; stream requires resync names the task).

(4) Recovery refusals

  • xagent.execution_event.recovery.refusals, attributes operation (the reader: owner_check, checkpoint, tool_outcome), outcome and storage="v2", with one line execution_event_recovery_refused task_id=... run_id=... storage=v2 component=execution-events reader=... outcome=... (plus tool_attempt_id=... for tool outcomes). Levels by outcome:
    • INFO: lease_mismatch, task_ended, task_missing. Normal fencing: another owner took over or the task already finished.
    • WARNING: unknown_tool_effect, unavailable, page_limit.
    • ERROR: corrupt.
  • It counts refused reads, including the classification probes made by the lease-recovery sweep, settlement and auto-recovery decisions. A stuck task can count once per sweep; it is neither a per-task nor a per-resume count.
  • Not reported yet: refusals raised while choosing the checkpoint partition on the root read path (lease or run mismatch, active_run, superseded_legacy, a missing task without a lease, stale widening, a failed run-tagged probe, session checkout), and a corrupt checkpoint found by the setup-time history read. The degradations and resume logs below still cover most of them.
  • Degradations (names on /health, details in the admin endpoint on the process that set them): checkpoint_load_unavailable (an execution-event checkpoint read failed; cleared by the next successful read), checkpoint_recovery_unavailable (set when a lease-recovery or settlement classification read fails; cleared once per recovery sweep and set again by any candidate that is still unavailable). Related, less common: checkpoint_decode_fallback, checkpoint_legacy_pointer_ambiguous, checkpoint_pk_anchor_dangling, checkpoint_prune_failed.
  • Resume refusals: ERROR task_id=%s component=resume checkpoint could not be read; restoring prior status %s: %s (the task goes back to PAUSED or WAITING_FOR_USER instead of FAILED). The v1 reply endpoint answers 409 (TASK_BUSY, INTERACTION_NOT_RESUMABLE) or 503 (TEMPORARILY_UNAVAILABLE); the A2A endpoint returns a matching error.
  • Automatic recovery decisions are stored per task: the auto_recovery section of the admin endpoint shows the task_auto_recovery row (state, reason, state_detail such as ineligible:...) and the latest task_recovery_events.
  • task_auto_resume_unavailable is set after three consecutive failed ticks of the auto-resume sweeper in that process.

(5) Read budget

  • The main V2 event-log readers listed below are measured as xagent.execution_event.read.duration (ms) and xagent.execution_event.read.rows, attributes operation, storage="v2" and, for reconnect, outcome (cache_hit or loaded). Other event-log queries (point lookups, the admin diagnostics endpoint and smaller readers) are not measured. Operations and what rows counts:
    • model: model-context events loaded;
    • display: display facts loaded, and for a delegated agent's detail the root-scope summary rows of that agent (a read of its own). The background delivery check run when a V2 socket closes (see (3)) is also a display read, counting every root display fact in its range, so a display budget can be exceeded by that check rather than by a user-facing read;
    • reconnect: frames replayed to a client (its duration includes cache I/O and projection work);
    • checkpoint: recovery-state rows scanned (nested tool-outcome reads add to the duration, not the rows);
    • tool_outcome: rows read for one tool attempt. A measured read made inside another one is counted once, as the outer one.
  • Optional budgets, observe-only: XAGENT_EXECUTION_EVENT_READ_ROW_BUDGETS and XAGENT_EXECUTION_EVENT_READ_DURATION_BUDGETS_MS, e.g. model=2000,reconnect=20000,*=50000 (see example.env). A read over its budget increments xagent.execution_event.read.budget_exceeded (outcome is rows or duration) and logs one WARNING execution_event_read_budget_exceeded task_id=... operation=... rows=... duration_ms=... budget_rows=... budget_ms=.... Budgets never refuse or truncate a read, and a cache-hit reconnect is never compared with them. An invalid value logs execution_event_read_budget_config_invalid once and is treated as no budget.
  • Before budgets are set, watch the p95 and maximum of the two histograms per operation.

3. Triaging one task with the admin endpoint

GET /api/admin/execution-events/tasks/{task_id}

Admin users only: 401 without a valid token, 403 Admin access required for a non-admin, 404 for an unknown task, 503 when the database cannot be read (the response never substitutes zeros). It is read-only and never returns event payloads, the task's error_message, the auto-recovery last_error or the recovery events' detail; for the two error columns it reports only whether they are set. The one free-text field left is degradations, whose details are operator messages written by the platform and can name a task id. All queries are filtered by task id and use aggregates over columns; every list is bounded and has a truncated flag.

The event sections are a snapshot at task.conversation_event_sequence, read first: an append advances it in the same transaction, so every event at or below it is committed, and events appended while the request runs are left out. auto_recovery is read from its own tables afterwards and can be a little newer. A busy task therefore looks consistent within one response but moves between two requests.

Section What it tells you
storage v2 or v1. A V1 task returns the same shape with empty event sections.
task status, control_state, run_id, state_version; conversation_event_sequence (the committed sequence horizon: readers ignore events above it); lease fields runner_id, lease_attempt_id, lease_expires_at, last_heartbeat_at. A non-null lease_attempt_id does not prove the run is alive; compare lease_expires_at and last_heartbeat_at with the current time. has_error_message says only whether the task carries an error text.
events Total, min/max sequence, latest occurred_at, and counts per (scope_id, kind), at most 200 groups, all at or below the snapshot horizon. Events are never deleted and sequences are allocated without gaps, so max_sequence equals task.conversation_event_sequence and total equals max_sequence; a difference within one response means the log itself is damaged and is worth escalating.
runs Per run_id (including null): event count, sequence range, and settled, true when an execution_settled fact exists for that run. Latest 20 by max sequence. An earlier run with settled false never wrote a settlement fact.
recovery_states Per scope, the highest sequence of a recovery_state fact (the checkpoint). Compare with events.max_sequence to see how much was written after the last checkpoint.
open_tool_attempts Attempts (per scope) with a tool_execution_start fact and no tool_execution_end or tool_execution_failed fact, the same test the recovery reader uses to decide that no outcome was committed. Shows attempt id, scope, run, start sequence and the kinds seen, newest first, up to 100, plus open_count.
auto_recovery The task's automatic-recovery row (has_last_error only; the text is the run's own error, the same content as error_message) and the latest 20 recovery events (type, reason, attempt, timestamps; the free-form detail is omitted).
degradations active_degradations() of the process that served the request only.

What an open tool attempt means

An open attempt means the tool was started and nothing durable says how it ended. The tool's external effect (a sent message, a payment, a written file, an API call) is unknown: it may have completed, partly completed or never run. Do not rerun the tool or resume the task to "retry" it on the strength of the log alone. The platform itself refuses to replay such an attempt automatically; a run that ends this way settles as an unknown tool effect. Check the effect in the external system first, then decide with the task owner whether to start a new turn.

Notes on reading open_tool_attempts:

  • It lists every attempt in the log with this shape, not only those named by the latest checkpoint. The recovery verdict is decided from the attempts the latest checkpoint still lists as pending, so an old open attempt does not by itself block recovery.
  • A terminal fact that records an interrupted attempt (interrupted: true in its payload) is still refused by the recovery reader, but the endpoint does not read payloads and cannot show it; such an attempt is not listed as open.
  • A later run's settlement pair (key tool:{attempt}:settlement:...) uses the same kinds, and a settlement end fact closes the attempt like any other end fact.