Skip to content

DeepTutor: Lifelong Personalized Tutoring. https://deeptutor.info/.

4.4 KB
Raw Blame

Events, durable replay and indexing progress

Handbook · Architecture · Indexing recovery

Different event systems have different persistence guarantees. Start with the operation being diagnosed; a process-local notification is not a durable turn journal.

Flow Main owners What to inspect
Model turn Application service, turn runtime, StreamBus Submission identity, persisted sequence, replay cursor, terminal event
Multi-worker coordination Coordinator Lease owner, fencing, cancellation notification, recovery
Knowledge ingestion Progress tracker, indexing journal Task identity, source outcomes, publication state, last activity
Knowledge UI transport Progress ports, broadcaster, task stream Snapshot, WebSocket progress, SSE logs/terminal state
Process notifications EventBus Current publishers/subscribers; process lifetime

Turn lifecycle

The application service coordinates admission and execution through the shared session store and runtime. Each execution has a per-turn stream bus; events must enter the persistence/replay path before adapters deliver them to consumers. A browser disconnect does not by itself prove the turn failed or authorize a second execution. Recovery must distinguish an active lease from a lost worker and preserve terminal state.

When changing streaming, test duplicate submission, reconnect after partial output, replay after a terminal event, cancellation and worker loss. Preserve event IDs/sequences and existing projection rules instead of treating arrival order or rendered text as the durable source of truth. Protocol types live in core/stream.py; the frontend consumes the contracts under web/contracts/.

One worker can use the memory coordinator. More than one backend worker requires Redis coordination, configured through system.backend_workers and integrations.turn_coordination. The settings validator rejects multiple workers with a memory coordinator and rejects Redis without a URL. Redis is the coordination/notification layer; it does not replace the session store. deeptutor doctor runtime --format json reports runtime diagnostics.

Knowledge-base progress

ProgressTracker persists snapshots and emits through injected progress ports. The API installs the WebSocket/SSE ports; domain code must not import FastAPI to report progress. CLI/SDK execution can still persist progress without a browser broadcaster.

The knowledge router exposes a progress WebSocket and task-log SSE stream. useKnowledgeProgress combines them, tracks the expected task, and reconciles terminal state. Preserve workspace scope and task identity when reconnecting or applying a snapshot. Logs and percentage estimates are observations, not proof that an index was published.

The indexing journal records source-level success, failure and interrupted work. Completion must agree with persisted publication; a parser reaching its last batch is insufficient. Rebuilds must not replace a queryable version with a partial candidate. For parsing checkpoints and exact source locators, also read source visuals.

Debugging sequence

  1. Identify the account, workspace, session/turn or KB/task ID and time window.
  2. Check durable status first, then the coordinator/worker and transport.
  3. Compare the browser's cursor/task identity to the server's persisted state.
  4. Reproduce dropped connections, old snapshots and duplicate frames with fixtures. Do not repair progress by overwriting success fields manually.
  5. Verify terminal state after reopening, not only while the original browser connection remains alive.

Relevant suites include application tests, coordination tests, knowledge tests, and knowledge progress WebSocket tests.