Skip to content

DeepTutor: Lifelong Personalized Tutoring. https://deeptutor.info/.

2.7 KB
Raw Blame

Long-document indexing and recovery

Handbook · Related guide

Each knowledge base records its latest indexing run in .indexing-run.json. The Files/detail view exposes Indexing recovery details, including individual source outcomes, current phase and phase-local counts, elapsed runtime, time since output/activity, and time since measurable progress. A phase reaching its last batch is not proof that the whole index is ready. Unknown work remains unknown after interruption; it is never counted as a successful document.

GET /api/knowledge-bases/{id}/indexing-readiness performs a lightweight parser, format and embedding-configuration check without downloading models or sending a provider request. Readiness does not guarantee later downloads, available RAM or GPU memory, nor a provider's runtime response. Runtime failures retain their bounded, credential-redacted reasons in the durable run.

Retry uses the existing content-addressed parser cache: source bytes and parser signature must match, and usable text or blocks must still exist. MinerU cloud PDF slices reuse only validated checkpoints. Changing embeddings can reuse parsing, but builds a separate index version for the new embedding signature. Interrupted stages without a usable checkpoint restart; the journal itself is not a second cache and does not fabricate partial embeddings.

Cancellation is task-specific and takes effect at a safe boundary. Managed local MinerU processes also observe cancellation and fail after 600 seconds without output or 7200 seconds total. Other synchronous engines cannot be interrupted arbitrarily; cancellation remains pending until they return. A stalled LlamaIndex executor is barred from overlapping same-process retries until it actually exits, and cannot publish its candidate after losing ownership.

LlamaIndex rebuilds and incremental additions use fresh version directories. A candidate stays unpublished until its persisted index reloads, its vectors validate and a retrieval probe using a stored vector returns a node. Publication metadata is written atomically. Source changes during indexing reject publication. A failed or partial candidate does not overwrite a previous queryable version. Completed and failed sources are reported separately, including partial success. The durable journal is scoped to the active user/workspace knowledge base; linked/read-only knowledge bases cannot be cancelled through mutation routes.

Implementation and verification