Skip to content

Project NOMAD is an offline-first knowledge and education server. Wikipedia, thousands of books, courses, maps, and optional local AI, all running on hardware you own with no internet required.

TypeScript
1.9 KB
Raw Blame
/**
 * Decision for whether a batched ZIM ingestion should dispatch a continuation.
 *
 * This is the pure, I/O-free core of the batch loop in
 * `RagService.processZIMFile` (mirrors `decideScanAction` in
 * `kb_ingest_decision.ts`).
 *
 * The signal MUST be the number of articles the extractor *consumed*, never the
 * number that produced chunks. An article yields zero chunks whenever its text
 * is empty after HTML cleaning — redirect stubs, category listings, image/video
 * wrappers, PDF containers. Those are perfectly normal ZIM entries, but they are
 * invisible to any count derived from the returned chunks.
 *
 * Gating on chunk-derived counts silently truncates ingestion the first time a
 * window of `batchSize` articles happens to contain mostly empty ones: the
 * continuation is never dispatched and every remaining article in the archive is
 * skipped. That is not a rare edge case — it is the norm for scraped-site and
 * media-heavy archives, where the opening entries are usually navigation pages.
 *
 * `articlesProcessed < batchSize` means the extractor's iterator ran dry, which
 * is the only reliable end-of-archive signal available: `archive.articleCount`
 * cannot serve as an upper bound, because `iterByPath()` legitimately yields
 * more article entries than that figure for some archives.
 *
 * A full final batch costs one extra dispatch that extracts nothing and stops.
 * That is deliberate: over-running by a single empty batch is cheap, while
 * stopping one batch early loses the remainder of the archive.
 */
export interface ZimBatchProgress {
  /** Articles the extractor consumed in this batch (not articles that produced chunks). */
  articlesProcessed: number
  /** The article ceiling requested for this batch. */
  batchSize: number
}

export function hasMoreArticleBatches({
  articlesProcessed,
  batchSize,
}: ZimBatchProgress): boolean {
  return articlesProcessed >= batchSize
}