:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Durable streaming: what we have and what is missing
This is a concept-level assessment of the agent-proxy live event plane against the high-level ideas of durable streaming: resumability, ordering, idempotent append, durable closure and gap awareness. It is not a wire-conformance check against the durable-streams.com protocol. agent-proxy stays byte-identical to the Python proxy during the cutover, so it cannot share that protocol's wire format, and that is by design. What follows is purely about the guarantees the streaming layer provides and where it falls short, regardless of how they are framed on the wire.
Reference concepts:
- https://durablestreams.com/building-a-server
- https://durablestreams.com/building-a-client
- https://github.com/durable-streams/durable-streams/blob/main/PROTOCOL.md
What we have
- Resumability. A client reconnects from its last position (
Last-Event-ID) and the server replays forward from there. This is the core durable-streaming property and it is solid. See the read leg insrc/hono/app.tsandstreamTaskRunEventsinsrc/hono/sse-handler.ts. - Stable, ordered positions. Every event has a monotonic id (the Redis stream id);
events are delivered in a strict, stable order. See
src/lib/redis-stream.ts. - Catch-up then live. A reader gets history from the start, or
?start=latestto skip to the tail, then seamlessly tails live from the same connection. The two modes are fused rather than separate endpoints. - No-skip and no-duplicate delivery within the retention window. Resume is exclusive of the last id, so inside the live window a client neither re-receives nor skips events.
- Idempotent, ordered append. The ingest
seqenforces contiguous ordering (a gap is rejected) and a replayedseqis deduplicated rather than re-appended, so the producer can retry safely. This is effectively exactly-once on the write side for a single producer. SeewriteEventWithSequenceinsrc/lib/redis-stream.ts. - Completion consistency. Closing the stream verifies the final sequence matches what
was accepted, so the stream cannot be marked complete past a hole. See
markCompleteAfterSequence. - Durable, observable EOF. The
stream-endsentinel plus the persistedcompletedmarker mean a late reader still learns the stream ended, and the client honors it to stop reconnecting. - Liveness and reconnect robustness. Keepalives, reconnect backoff, a cumulative
retry budget and a "keepalive proves recovery" reset. The client distinguishes
transport cuts from backend errors (in the
cloud-taskwatcher on the client side). - Backpressure to slow readers. The SSE writer awaits each chunk
(
responseStream.write), so a slow consumer naturally throttles the read loop. - Multi-reader fan-out. Several clients can independently tail the same run, each with its own cursor and its own dedicated Redis connection.
What is missing or partial
Ranked by how much it matters for the current use case (watching a live agent run).
- Gap awareness. The server detects when a reader's resume point has been trimmed
out of the window (
resumePointTrimmed/detectResumeGapinsrc/lib/redis-stream.ts, surfaced as thestream:resume_gapmetric), on attach and again once a reader has waitedSTREAM_CURSOR_RECHECK_STALL_MSin total since the last check, whether in one pause or in many short ones. A reader that opened with?resync=1is told: the connection closes with anendframe carrying{"type":"resync","reason":"trimmed"}(also when the stream of a finished run has expired), and a failed trim check closes it with a stream error. The reader drops its cursor and reopens withoutLast-Event-ID, cursorless so the surviving window replays: the run log trails the live stream and holds a streaming message only once the turn coalesces it into one entry, so a reader that reopens at?start=latestcan miss the message it was receiving. It buffers live frames, reads the durable run log and deduplicates the replay against it by agent event id, covering both thefirst_event_id..event_idrange of a coalesced entry and every id in itscovered_event_ids. A reader without the flag still silently continues from the oldest surviving event, so it can miss a span of events and never know. Server-side backfill from durable storage (S3 hydration) stays out of scope (see theResumeGapcomment insrc/lib/types.tsandDESIGN.md). - Bounded retention. "Durable" here means a sliding 6h TTL while the run is live,
a 30 minute drain window after the terminal sentinel, and a 5,000 event
MAXLEN(STREAM_TTL_SECONDS,STREAM_COMPLETED_TTL_SECONDS,STREAM_MAX_LENGTHinsrc/lib/constants.ts), after which data is gone. A reader arriving after expiry getsStream not available; the durable transcript lives in the run's session logs (session_logson the task-run API). That bound is the cause of gap awareness above. It is fine for watching a run live, but not for replaying a run's stream much later. If long replay becomes a goal, this is the lever. - No "caught up to the tail" signal. A reader cannot tell when it transitions from replaying history to being live at the tail. This is minor for a live UI, which just keeps tailing, but it is a real durable-streaming concept that is not exposed.
- No producer fencing. The
seqprotocol gives ordering and deduplication for one writer, but nothing fences out a second or zombie writer to the same run (for example a retried or duplicated sandbox). We rely on "one sandbox per run" by construction. If that assumption can break, two producers could interleave and we would only catch it as sequence gaps, not as "you are not the active writer." - Not needed for this use case, noted for completeness. Stream forking, durable multi-consumer subscriptions (webhooks or pull-wake) and consumer-group coordination. These are durable-streaming features we do not have and almost certainly do not want here.
Summary
| Concept | Status |
|---|---|
| Resume from last position | Have |
| Stable ordered positions | Have |
| Catch-up then live tail | Have |
| No-skip / no-duplicate within window | Have |
| Idempotent, ordered append (single producer) | Have |
| Completion consistency | Have |
| Durable, observable EOF | Have |
| Keepalive, reconnect robustness | Have |
| Backpressure to slow readers | Have |
| Multi-reader fan-out | Have |
| Gap awareness on trimmed resume | Partial (signalled to ?resync=1 readers) |
| Retention beyond 6h live / 30m terminal | Partial (bounded, then dropped) |
| Caught-up-to-tail signal | Missing |
| Producer fencing across writers | Partial (single-producer by construction) |
| Forking / subscriptions / consumer groups | Missing (not needed) |
We have resumability, ordering, idempotent append and durable closure, which is the meat of durable streaming. The two gaps that actually matter are gap awareness (the silent skip on trimmed resume) and the bounded retention behind it. Everything else missing is either cosmetic (the caught-up signal) or out of scope (forking, subscriptions). The important one is already on the roadmap as deferred S3 hydration.