Skip to content

OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.

Python

bulk loader script (#2128)

Sebastián Estévezca8b6ebcb55f
diff --git a/scripts/README.md b/scripts/README.md
index c3e0dd321..ea289eefc 100644
--- a/scripts/README.md
+++ b/scripts/README.md
@@ -1,5 +1,138 @@
# OpenRAG scripts
+## Bulk ingest (`openrag_bulk.py`)
+
+`openrag_bulk.py` is a standalone client for loading a directory or a set of
+files through OpenRAG's public ingestion API. It groups files into server tasks,
+keeps a bounded number of tasks in flight, polls them to completion, and writes
+an atomic `summary.json` record as the run progresses.
+
+The command families are deliberately separate:
+
+- `bulk upload` loads user-supplied files or directories.
+- `bench arxiv` acquires a repeatable arXiv dataset, then delegates its upload
+ phase to the same bulk loader.
+
+The script has inline `uv` dependency metadata and does not import the OpenRAG
+application or SDK, so it can also be copied and run outside this repository.
+
+Set the URL of the OpenRAG frontend and an API key with `knowledge:upload` and
+`knowledge:read:own` permissions:
+
+```bash
+export OPENRAG_URL=http://localhost:3000
+export OPENRAG_API_KEY=orag_your_api_key
+
+uv run scripts/openrag_bulk.py bulk upload ./documents \
+ --include '*.pdf' \
+ --exclude 'archive/*' \
+ --batch-size 10 \
+ --max-inflight 4 \
+ --max-submit 2
+```
+
+`--batch-size` controls how many files are sent in each multipart request.
+`--max-inflight` bounds the number of submitted OpenRAG tasks, including tasks
+being uploaded or polled. `--max-submit` can impose a lower limit on concurrent
+multipart requests, which is useful when files are large.
+
+Useful upload options include:
+
+| Option | Purpose |
+| --- | --- |
+| `--sort path\|size-asc\|size-desc` | Choose the order in which files are batched. |
+| `--include GLOB`, `--exclude GLOB` | Filter relative paths or filenames; repeat either option. |
+| `--no-recursive` | Only read files directly inside supplied directories. |
+| `--settings-json JSON_OR_@FILE` | Forward per-run knowledge ingest settings. |
+| `--tweaks-json JSON_OR_@FILE` | Forward Langflow ingest tweaks. |
+| `--no-replace-duplicates` | Preserve existing documents with duplicate filenames. |
+| `--task-timeout SECONDS` | Bound the wait for each submitted server task. |
+| `--runs-dir DIR`, `--output-dir DIR`, `--run-id ID` | Control local run records. |
+
+When connecting directly to the backend rather than through the frontend proxy,
+select its route prefix explicitly:
+
+```bash
+uv run scripts/openrag_bulk.py bulk upload ./documents \
+ --base-url http://localhost:8000 \
+ --api-prefix /v1
+```
+
+Runs are stored under `~/.openrag/bulk/runs` by default. They can be inspected
+without a running server or an API key:
+
+```bash
+uv run scripts/openrag_bulk.py bulk list
+uv run scripts/openrag_bulk.py bulk summary
+uv run scripts/openrag_bulk.py bulk summary --detail latest
+```
+
+The upload command exits with status `0` when every batch completes, `1` when
+one or more batches/files fail, and `2` for invalid input or configuration.
+
+### arXiv benchmark and PDF downloader
+
+The same standalone client includes the arXiv benchmark from `openrag-lite`.
+It can stage PDFs without a server, or feed the staged PDFs directly into the
+bulk uploader above.
+
+By default, the benchmark copies a requester-pays arXiv PDF tarball from S3,
+caches the tarball, extracts the selected PDFs, and uploads them to OpenRAG:
+
+```bash
+export OPENRAG_URL=http://localhost:3000
+export OPENRAG_API_KEY=orag_your_api_key
+
+uv run scripts/openrag_bulk.py bench arxiv \
+ --max-results 100 \
+ --batch-size 10 \
+ --max-inflight 4
+```
+
+The S3 path requires the AWS CLI and credentials that can read the
+requester-pays `s3://arxiv` bucket. Override `--s3-uri` to use another tarball;
+plain local paths and `file://` URLs are also supported.
+
+To query arXiv's Atom API and download PDFs individually instead, select the
+Atom source. The client applies a three-second courtesy delay between arXiv
+requests by default:
+
+```bash
+uv run scripts/openrag_bulk.py bench arxiv \
+ --source atom \
+ --category cs.AI \
+ --date-from 2025-01-01 \
+ --date-to 2025-01-31 \
+ --max-results 25
+```
+
+`--query` accepts a raw arXiv search query and overrides the category/date
+query. `--start`, `--sort-by`, and `--sort-order` control source selection.
+
+Download and cache PDFs without connecting to OpenRAG by passing
+`--download-only`; no API key is required:
+
+```bash
+uv run scripts/openrag_bulk.py bench arxiv \
+ --source atom \
+ --query 'cat:cs.CL' \
+ --max-results 10 \
+ --download-only
+```
+
+The PDF cache defaults to `~/.openrag/benchmarks/arxiv/pdfs`. Existing PDFs and
+S3 tarballs are reused by default. Atom download failures are remembered in
+`_failed_downloads.json`; pass `--retry-failed-downloads` to try them again, or
+`--no-skip-existing` to create fresh PDF files.
+
+Benchmark run records default to `~/.openrag/benchmarks/arxiv/runs`:
+
+```bash
+uv run scripts/openrag_bulk.py bench arxiv list
+uv run scripts/openrag_bulk.py bench arxiv summary
+uv run scripts/openrag_bulk.py bench arxiv summary --detail latest
+```
+
## Sync default user roles (`sync_default_user_roles.py`)
Dev-only helper for RBAC in **OSS run mode** (`OPENRAG_RUN_MODE=oss`): updates
diff --git a/scripts/openrag_bulk.py b/scripts/openrag_bulk.py
new file mode 100755
index 000000000..129dcb049
--- /dev/null
+++ b/scripts/openrag_bulk.py
@@ -0,0 +1,1947 @@
+#!/usr/bin/env -S uv run --script
+# /// script
+# requires-python = ">=3.11"
+# dependencies = ["httpx>=0.25.0,<1"]
+# ///
+"""Standalone bulk ingestion and arXiv benchmark client for OpenRAG.
+
+The script intentionally imports no OpenRAG application or SDK modules. This keeps it
+usable from a checkout, a Kubernetes job, or a machine that only has ``uv`` installed.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import fnmatch
+import hashlib
+import json
+import mimetypes
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tarfile
+import time
+import xml.etree.ElementTree as ET
+from collections.abc import Callable, Sequence
+from contextlib import ExitStack
+from dataclasses import dataclass
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any
+from urllib.error import HTTPError
+from urllib.parse import unquote, urlencode, urlparse, urlunparse
+from urllib.request import Request, urlopen
+from uuid import uuid4
+
+import httpx
+
+TERMINAL_TASK_STATUSES = {"completed", "failed", "skipped"}
+TRANSIENT_HTTP_STATUSES = {502, 503, 504}
+DEFAULT_RUNS_DIR = Path("~/.openrag/bulk/runs")
+DEFAULT_ARXIV_DIR = Path("~/.openrag/benchmarks/arxiv")
+ARXIV_API_URL = "https://export.arxiv.org/api/query"
+ATOM = "{http://www.w3.org/2005/Atom}"
+DEFAULT_ARXIV_CATEGORY = "cs.CL"
+DEFAULT_ARXIV_DATE_FROM = "2025-01-01"
+DEFAULT_ARXIV_DATE_TO = "2025-12-31"
+DEFAULT_ARXIV_SOURCE = "s3"
+DEFAULT_ARXIV_S3_URI = "s3://arxiv/pdf/arXiv_pdf_1001_001.tar"
+
+
+class ApiError(RuntimeError):
+ """An OpenRAG API request failed."""
+
+ def __init__(self, message: str, status_code: int):
+ super().__init__(message)
+ self.status_code = status_code
+
+
+class ArxivBenchmarkError(RuntimeError):
+ """The arXiv source could not be staged safely."""
+
+
+@dataclass(frozen=True)
+class BulkUploadItem:
+ path: Path
+ filename: str | None = None
+ content_type: str | None = None
+
+
+@dataclass(frozen=True)
+class BulkUploadOptions:
+ batch_size: int = 10
+ max_inflight: int = 1
+ max_submit: int | None = None
+ request_timeout: float = 300.0
+ poll_request_timeout: float = 10.0
+ task_timeout: float = 1800.0
+ poll_interval: float = 1.0
+ progress_interval: float = 5.0
+ summary_interval: float = 60.0
+ input_sort: str = "path"
+ settings_json: str | None = None
+ tweaks_json: str | None = None
+ replace_duplicates: bool = True
+
+ def validate(self) -> None:
+ positive = {
+ "batch_size": self.batch_size,
+ "max_inflight": self.max_inflight,
+ "request_timeout": self.request_timeout,
+ "poll_request_timeout": self.poll_request_timeout,
+ "task_timeout": self.task_timeout,
+ }
+ for name, value in positive.items():
+ if value <= 0:
+ raise ValueError(f"{name} must be positive")
+ if self.max_submit is not None and self.max_submit <= 0:
+ raise ValueError("max_submit must be positive")
+ for name, value in {
+ "poll_interval": self.poll_interval,
+ "progress_interval": self.progress_interval,
+ "summary_interval": self.summary_interval,
+ }.items():
+ if value < 0:
+ raise ValueError(f"{name} must be non-negative")
+
+
+@dataclass(frozen=True)
+class ArxivPaper:
+ arxiv_id: str
+ title: str
+ summary: str
+ authors: tuple[str, ...]
+ published: str | None
+ updated: str | None
+ abs_url: str
+ pdf_url: str
+
+
+@dataclass(frozen=True)
+class DownloadedPaper:
+ paper: ArxivPaper
+ path: Path
+ size: int
+ sha256: str
+ cached: bool = False
+
+
+class RateLimiter:
+ def __init__(self, delay_seconds: float):
+ self.delay_seconds = max(delay_seconds, 0.0)
+ self.last_request: float | None = None
+
+ def wait(self) -> None:
+ if self.last_request is not None:
+ remaining = self.delay_seconds - (time.monotonic() - self.last_request)
+ if remaining > 0:
+ time.sleep(remaining)
+ self.last_request = time.monotonic()
+
+
+class ProgressReporter:
+ def __init__(
+ self,
+ *,
+ enabled: bool = True,
+ started: float | None = None,
+ prefix: str = "openrag bulk",
+ stream=None,
+ ) -> None:
+ self.enabled = enabled
+ self.started = time.monotonic() if started is None else started
+ self.prefix = prefix
+ self.stream = stream or sys.stderr
+
+ def log(self, message: str) -> None:
+ if not self.enabled:
+ return
+ elapsed = time.monotonic() - self.started
+ print(f"{self.prefix} [{elapsed:8.1f}s] {message}", file=self.stream, flush=True)
+
+
+class OpenRAGApiClient:
+ """Small API client covering only the two calls bulk ingestion needs."""
+
+ def __init__(
+ self,
+ *,
+ base_url: str,
+ api_key: str,
+ api_prefix: str = "/api/v1",
+ transport: httpx.AsyncBaseTransport | None = None,
+ ) -> None:
+ root = base_url.rstrip("/")
+ prefix = "/" + api_prefix.strip("/") if api_prefix.strip("/") else ""
+ self._base_url = root + prefix
+ self._http = httpx.AsyncClient(
+ headers={"X-API-Key": api_key},
+ transport=transport,
+ )
+
+ async def close(self) -> None:
+ await self._http.aclose()
+
+ async def __aenter__(self) -> OpenRAGApiClient:
+ return self
+
+ async def __aexit__(self, *_args: Any) -> None:
+ await self.close()
+
+ async def submit_batch(
+ self,
+ items: Sequence[BulkUploadItem],
+ *,
+ settings_json: str | None,
+ tweaks_json: str | None,
+ replace_duplicates: bool,
+ timeout: float,
+ ) -> dict[str, Any]:
+ data: dict[str, str] = {
+ "replace_duplicates": str(replace_duplicates).lower(),
+ }
+ if settings_json is not None:
+ data["settings"] = settings_json
+ if tweaks_json is not None:
+ data["tweaks"] = tweaks_json
+
+ with ExitStack() as stack:
+ files = [
+ (
+ "file",
+ (
+ item.filename or item.path.name,
+ stack.enter_context(item.path.open("rb")),
+ item.content_type
+ or mimetypes.guess_type(item.filename or item.path.name)[0]
+ or "application/octet-stream",
+ ),
+ )
+ for item in items
+ ]
+ response = await self._http.post(
+ f"{self._base_url}/documents/ingest",
+ data=data,
+ files=files,
+ timeout=timeout,
+ )
+ return _response_json(response)
+
+ async def get_task_status(self, task_id: str, *, timeout: float) -> dict[str, Any]:
+ response = await self._http.get(
+ f"{self._base_url}/tasks/{task_id}",
+ timeout=timeout,
+ )
+ return _response_json(response)
+
+
+SummaryCallback = Callable[[dict[str, Any]], Any]
+
+
+def collect_upload_items(
+ paths: Sequence[str],
+ *,
+ include: Sequence[str] | None = None,
+ exclude: Sequence[str] | None = None,
+ recursive: bool = True,
+) -> list[BulkUploadItem]:
+ include_patterns = list(include or ["*"])
+ exclude_patterns = list(exclude or [])
+ items: list[BulkUploadItem] = []
+ seen: set[Path] = set()
+
+ for raw_path in paths:
+ path = Path(raw_path).expanduser()
+ if path.is_file():
+ root = path.parent
+ candidates = [path]
+ elif path.is_dir():
+ root = path
+ iterator = path.rglob("*") if recursive else path.glob("*")
+ candidates = [candidate for candidate in iterator if candidate.is_file()]
+ else:
+ raise FileNotFoundError(f"bulk upload path not found: {raw_path}")
+
+ for candidate in sorted(candidates):
+ resolved = candidate.resolve()
+ if resolved in seen:
+ continue
+ relative = candidate.name if path.is_file() else str(candidate.relative_to(root))
+ if not _matches(relative, candidate.name, include_patterns):
+ continue
+ if _matches(relative, candidate.name, exclude_patterns):
+ continue
+ seen.add(resolved)
+ items.append(BulkUploadItem(path=resolved))
+
+ return items
+
+
+def sort_upload_items(items: Sequence[BulkUploadItem], order: str) -> list[BulkUploadItem]:
+ if order == "path":
+ return sorted(items, key=lambda item: str(item.path))
+ if order == "size-asc":
+ return sorted(items, key=lambda item: (item.path.stat().st_size, str(item.path)))
+ if order == "size-desc":
+ return sorted(items, key=lambda item: (-item.path.stat().st_size, str(item.path)))
+ raise ValueError(f"unsupported bulk upload sort: {order}")
+
+
+async def upload_bulk_items(
+ *,
+ client: Any,
+ items: Sequence[BulkUploadItem],
+ options: BulkUploadOptions,
+ progress: ProgressReporter,
+ summary_callback: SummaryCallback | None = None,
+) -> dict[str, Any]:
+ """Submit batches concurrently, poll each task, and return a run summary."""
+ options.validate()
+ batches = _chunks(list(items), options.batch_size)
+ max_submit = min(options.max_inflight, options.max_submit or options.max_inflight)
+ total_bytes = sum(item.path.stat().st_size for item in items)
+ summary: dict[str, Any] = {
+ "total_files": len(items),
+ "total_bytes": total_bytes,
+ "total_mb": round(_mb(total_bytes), 3),
+ "batch_size": options.batch_size,
+ "max_inflight": options.max_inflight,
+ "max_submit": max_submit,
+ "input_sort": options.input_sort,
+ "replace_duplicates": options.replace_duplicates,
+ "client_concurrency": {
+ "batch_size": options.batch_size,
+ "max_inflight_upload_tasks": options.max_inflight,
+ "max_inflight_submit_requests": max_submit,
+ "request_timeout_seconds": options.request_timeout,
+ "poll_request_timeout_seconds": options.poll_request_timeout,
+ "task_timeout_seconds": options.task_timeout,
+ "poll_interval_seconds": options.poll_interval,
+ },
+ "batches": [
+ _new_batch_result(batch, batch_index=index)
+ for index, batch in enumerate(batches, start=1)
+ ],
+ }
+ _notify(summary_callback, summary)
+ if not batches:
+ return summary
+
+ progress.log(
+ f"starting files={len(items)} batches={len(batches)} "
+ f"batch_size={options.batch_size} max_inflight={options.max_inflight} "
+ f"max_submit={max_submit} input_mb={_mb(total_bytes):.2f}"
+ )
+ inflight_semaphore = asyncio.Semaphore(options.max_inflight)
+ submit_semaphore = asyncio.Semaphore(max_submit)
+ finished = asyncio.Event()
+
+ async def process_batch(index: int, batch: list[BulkUploadItem]) -> None:
+ result = summary["batches"][index - 1]
+ async with inflight_semaphore:
+ result.update(status="submitting", phase="multipart_upload")
+ _notify(summary_callback, summary)
+ submitted_at = time.monotonic()
+ try:
+ async with submit_semaphore:
+ response = await client.submit_batch(
+ batch,
+ settings_json=options.settings_json,
+ tweaks_json=options.tweaks_json,
+ replace_duplicates=options.replace_duplicates,
+ timeout=options.request_timeout,
+ )
+ task_id = str(response["task_id"])
+ submit_seconds = round(time.monotonic() - submitted_at, 3)
+ result.update(
+ task_id=task_id,
+ status=str(response.get("status") or "accepted"),
+ phase="task_poll",
+ submit_seconds=submit_seconds,
+ )
+ progress.log(
+ f"batch {index}/{len(batches)} submitted task_id={task_id} "
+ f"files={len(batch)} submit_seconds={submit_seconds:.3f}"
+ )
+ _notify(summary_callback, summary)
+ except Exception as exc:
+ result.update(
+ status="failed",
+ phase="multipart_upload",
+ duration_seconds=round(time.monotonic() - submitted_at, 3),
+ error=_error_text(exc),
+ )
+ progress.log(f"batch {index}/{len(batches)} submit failed: {_error_text(exc)}")
+ _notify(summary_callback, summary)
+ return
+
+ await _poll_batch(
+ client=client,
+ result=result,
+ batch_count=len(batches),
+ options=options,
+ progress=progress,
+ summary=summary,
+ summary_callback=summary_callback,
+ )
+
+ async def report_summary() -> None:
+ if options.summary_interval == 0:
+ return
+ while not finished.is_set():
+ try:
+ await asyncio.wait_for(finished.wait(), timeout=options.summary_interval)
+ except TimeoutError:
+ progress.log(format_summary_progress(summary))
+
+ reporter = asyncio.create_task(report_summary())
+ try:
+ await asyncio.gather(
+ *(process_batch(index, batch) for index, batch in enumerate(batches, start=1))
+ )
+ finally:
+ finished.set()
+ await reporter
+
+ _update_summary_totals(summary)
+ _notify(summary_callback, summary)
+ return summary
+
+
+async def _poll_batch(
+ *,
+ client: Any,
+ result: dict[str, Any],
+ batch_count: int,
+ options: BulkUploadOptions,
+ progress: ProgressReporter,
+ summary: dict[str, Any],
+ summary_callback: SummaryCallback | None,
+) -> None:
+ task_id = str(result["task_id"])
+ started = time.monotonic()
+ last_log = 0.0
+ last_snapshot: tuple[Any, ...] | None = None
+ poll_errors = 0
+
+ while True:
+ if time.monotonic() - started >= options.task_timeout:
+ result.update(
+ status="failed",
+ phase="task_poll",
+ duration_seconds=round(time.monotonic() - started, 3),
+ error=f"task did not finish within {options.task_timeout:g}s",
+ )
+ progress.log(f"batch {result['batch_index']}/{batch_count} task {task_id} timed out")
+ _notify(summary_callback, summary)
+ return
+
+ try:
+ status = await client.get_task_status(
+ task_id,
+ timeout=options.poll_request_timeout,
+ )
+ poll_errors = 0
+ except Exception as exc:
+ if not _is_transient_error(exc):
+ result.update(
+ status="failed",
+ phase="task_poll",
+ duration_seconds=round(time.monotonic() - started, 3),
+ error=_error_text(exc),
+ )
+ progress.log(
+ f"batch {result['batch_index']}/{batch_count} task {task_id} "
+ f"poll failed: {_error_text(exc)}"
+ )
+ _notify(summary_callback, summary)
+ return
+ poll_errors += 1
+ now = time.monotonic()
+ if _should_log(now, last_log, options.progress_interval):
+ progress.log(
+ f"batch {result['batch_index']}/{batch_count} task {task_id} "
+ f"poll_error_count={poll_errors} error={_error_text(exc)}"
+ )
+ last_log = now
+ await asyncio.sleep(options.poll_interval)
+ continue
+
+ _update_batch_from_status(result, status)
+ _notify(summary_callback, summary)
+ snapshot = _status_snapshot(status)
+ now = time.monotonic()
+ if (
+ str(status.get("status")) in TERMINAL_TASK_STATUSES
+ or snapshot != last_snapshot
+ or _should_log(now, last_log, options.progress_interval)
+ ):
+ progress.log(
+ f"batch {result['batch_index']}/{batch_count} task {task_id} "
+ f"{format_task_progress(status)}"
+ )
+ last_log = now
+ last_snapshot = snapshot
+
+ if str(status.get("status")) in TERMINAL_TASK_STATUSES:
+ return
+ await asyncio.sleep(options.poll_interval)
+
+
+def bulk_failed(summary: dict[str, Any]) -> bool:
+ return any(
+ batch.get("status") != "completed" or _int_value(batch.get("failed_files")) > 0
+ for batch in summary.get("batches") or []
+ )
+
+
+def format_task_progress(status: dict[str, Any]) -> str:
+ values = [f"status={status.get('status')}", f"phase={_task_phase(status)}"]
+ for label in (
+ "processed_files",
+ "total_files",
+ "successful_files",
+ "failed_files",
+ "running_files",
+ "pending_files",
+ ):
+ if status.get(label) is not None:
+ values.append(f"{label.removesuffix('_files')}={status[label]}")
+ error = _task_error(status)
+ if error:
+ values.append(f"error={error}")
+ return " ".join(values)
+
+
+def format_summary_progress(summary: dict[str, Any]) -> str:
+ statuses: dict[str, int] = {}
+ processed = successful = failed = 0
+ for batch in summary.get("batches") or []:
+ status = str(batch.get("status") or "unknown")
+ statuses[status] = statuses.get(status, 0) + 1
+ processed += _int_value(batch.get("processed_files"))
+ successful += _int_value(batch.get("successful_files"))
+ failed += _int_value(batch.get("failed_files"))
+ counts = ",".join(f"{key}:{value}" for key, value in sorted(statuses.items()))
+ return (
+ f"summary batches={counts} files processed={processed}/{summary.get('total_files', 0)} "
+ f"successful={successful} failed={failed}"
+ )
+
+
+def persist_summary(
+ summary: dict[str, Any],
+ *,
+ output_dir: Path,
+ run_id: str,
+ started: float,
+) -> dict[str, Any]:
+ output_dir.mkdir(parents=True, exist_ok=True)
+ summary_path = output_dir / "summary.json"
+ summary.update(
+ run_id=run_id,
+ output_dir=str(output_dir),
+ summary_path=str(summary_path),
+ elapsed_seconds=round(time.monotonic() - started, 3),
+ )
+ ordered = _ordered_summary(summary)
+ temporary = output_dir / ".summary.json.tmp"
+ temporary.write_text(json.dumps(ordered, indent=2) + "\n", encoding="utf-8")
+ temporary.replace(summary_path)
+ return ordered
+
+
+def list_runs(runs_dir: Path, *, limit: int = 20) -> dict[str, Any]:
+ records: list[dict[str, Any]] = []
+ if runs_dir.is_dir():
+ paths = sorted(
+ (path for path in runs_dir.glob("*/summary.json") if path.is_file()),
+ key=lambda path: path.stat().st_mtime,
+ reverse=True,
+ )
+ if limit > 0:
+ paths = paths[:limit]
+ for path in paths:
+ try:
+ summary = _load_summary(path)
+ except (OSError, ValueError, json.JSONDecodeError):
+ continue
+ records.append(_run_overview(summary, path))
+ return {"runs_dir": str(runs_dir), "count": len(records), "runs": records}
+
+
+def load_summary_reference(
+ runs_dir: Path,
+ reference: str | None,
+ *,
+ detail: bool,
+) -> dict[str, Any]:
+ value = (reference or "latest").strip()
+ if value in {"latest", "current", "recent"}:
+ listed = list_runs(runs_dir, limit=1)
+ if not listed["runs"]:
+ raise FileNotFoundError(f"no runs found under {runs_dir}")
+ path = Path(listed["runs"][0]["summary_path"])
+ else:
+ candidate = Path(value).expanduser()
+ if candidate.is_file():
+ path = candidate
+ elif candidate.is_dir():
+ path = candidate / "summary.json"
+ else:
+ path = runs_dir / value / "summary.json"
+ summary = _load_summary(path.resolve())
+ if detail:
+ return summary
+ response = dict(summary)
+ response.pop("batches", None)
+ response["summary"] = _run_overview(summary, path.resolve())
+ return response
+
+
+def build_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(
+ description="Bulk-load local files or benchmark OpenRAG ingestion.",
+ )
+ subparsers = parser.add_subparsers(dest="command", required=True)
+
+ bulk = subparsers.add_parser("bulk", help="Bulk-load local files and directories.")
+ bulk_subparsers = bulk.add_subparsers(dest="bulk_action", required=True)
+
+ upload = bulk_subparsers.add_parser("upload", help="Upload paths in concurrent batches.")
+ upload.add_argument("paths", nargs="+")
+ upload.add_argument("--base-url", default=os.getenv("OPENRAG_URL", "http://localhost:3000"))
+ upload.add_argument("--api-key", default=os.getenv("OPENRAG_API_KEY"))
+ upload.add_argument(
+ "--api-prefix",
+ default="/api/v1",
+ help="Use /v1 when calling the backend directly (default: /api/v1).",
+ )
+ upload.add_argument("--batch-size", type=int, default=10)
+ upload.add_argument("--max-inflight", type=int, default=1)
+ upload.add_argument("--max-submit", type=int, default=None)
+ upload.add_argument("--sort", choices=("path", "size-asc", "size-desc"), default="path")
+ upload.add_argument("--include", action="append", default=None)
+ upload.add_argument("--exclude", action="append", default=None)
+ upload.add_argument("--recursive", action=argparse.BooleanOptionalAction, default=True)
+ upload.add_argument("--settings-json", default=None, help="JSON object or @path to JSON.")
+ upload.add_argument("--tweaks-json", default=None, help="JSON object or @path to JSON.")
+ upload.add_argument(
+ "--replace-duplicates",
+ action=argparse.BooleanOptionalAction,
+ default=True,
+ )
+ upload.add_argument("--task-timeout", type=float, default=1800.0)
+ upload.add_argument("--request-timeout", type=float, default=300.0)
+ upload.add_argument("--poll-request-timeout", type=float, default=10.0)
+ upload.add_argument("--poll-interval", type=float, default=1.0)
+ upload.add_argument("--progress", action=argparse.BooleanOptionalAction, default=True)
+ upload.add_argument("--progress-interval", type=float, default=5.0)
+ upload.add_argument("--summary-interval", type=float, default=60.0)
+ upload.add_argument("--runs-dir", default=str(DEFAULT_RUNS_DIR))
+ upload.add_argument("--output-dir", default=None)
+ upload.add_argument("--run-id", default=None)
+
+ list_parser = bulk_subparsers.add_parser("list", help="List locally recorded bulk runs.")
+ list_parser.add_argument("--runs-dir", default=str(DEFAULT_RUNS_DIR))
+ list_parser.add_argument("--limit", type=int, default=20)
+
+ summary = bulk_subparsers.add_parser("summary", help="Show a locally recorded run.")
+ summary.add_argument("run", nargs="?", default=None)
+ summary.add_argument("--runs-dir", default=str(DEFAULT_RUNS_DIR))
+ summary.add_argument("--detail", action="store_true")
+
+ bench = subparsers.add_parser("bench", help="Run repeatable ingestion benchmarks.")
+ bench_subparsers = bench.add_subparsers(dest="benchmark", required=True)
+ arxiv = bench_subparsers.add_parser(
+ "arxiv",
+ help="Download arXiv PDFs and optionally bulk-ingest them.",
+ )
+ _add_arxiv_args(arxiv)
+ arxiv_actions = arxiv.add_subparsers(dest="benchmark_action")
+ arxiv_list = arxiv_actions.add_parser("list", help="List arXiv benchmark runs.")
+ arxiv_list.add_argument("--runs-dir", default=str(DEFAULT_ARXIV_DIR / "runs"))
+ arxiv_list.add_argument("--limit", type=int, default=20)
+ arxiv_summary = arxiv_actions.add_parser("summary", help="Show an arXiv benchmark run.")
+ arxiv_summary.add_argument("run", nargs="?", default=None)
+ arxiv_summary.add_argument("--runs-dir", default=str(DEFAULT_ARXIV_DIR / "runs"))
+ arxiv_summary.add_argument("--detail", action="store_true")
+ return parser
+
+
+def _add_arxiv_args(parser: argparse.ArgumentParser) -> None:
+ parser.add_argument(
+ "--source",
+ choices=("s3", "atom"),
+ default=DEFAULT_ARXIV_SOURCE,
+ help="Use the S3 bulk PDF tarball (default) or the arXiv Atom API.",
+ )
+ parser.add_argument(
+ "--query", default=None, help="Raw Atom API query; overrides category/dates."
+ )
+ parser.add_argument("--category", default=DEFAULT_ARXIV_CATEGORY)
+ parser.add_argument("--date-from", default=DEFAULT_ARXIV_DATE_FROM)
+ parser.add_argument("--date-to", default=DEFAULT_ARXIV_DATE_TO)
+ parser.add_argument("--max-results", type=int, default=10)
+ parser.add_argument("--start", type=int, default=0)
+ parser.add_argument(
+ "--sort-by",
+ choices=("relevance", "lastUpdatedDate", "submittedDate"),
+ default="submittedDate",
+ )
+ parser.add_argument(
+ "--sort-order",
+ choices=("ascending", "descending"),
+ default="descending",
+ )
+ parser.add_argument("--s3-uri", default=DEFAULT_ARXIV_S3_URI)
+ parser.add_argument("--s3-cache-dir", default=None)
+ parser.add_argument("--aws-cli", default="aws")
+ parser.add_argument("--s3-request-payer", default="requester")
+ parser.add_argument("--pdf-cache-dir", default=None)
+ parser.add_argument("--failed-download-cache", default=None)
+ parser.add_argument("--download-only", action="store_true")
+ parser.add_argument("--skip-existing", action=argparse.BooleanOptionalAction, default=True)
+ parser.add_argument("--retry-failed-downloads", action="store_true")
+ parser.add_argument("--delay-seconds", type=float, default=3.0)
+ parser.add_argument("--max-pdf-mb", type=int, default=50)
+ parser.add_argument("--user-agent", default="openrag-arxiv-benchmark/0.1")
+ parser.add_argument("--base-url", default=os.getenv("OPENRAG_URL", "http://localhost:3000"))
+ parser.add_argument("--api-key", default=os.getenv("OPENRAG_API_KEY"))
+ parser.add_argument("--api-prefix", default="/api/v1")
+ parser.add_argument("--batch-size", type=int, default=5)
+ parser.add_argument(
+ "--upload-sort",
+ choices=("path", "size-asc", "size-desc"),
+ default="path",
+ )
+ parser.add_argument("--max-inflight", type=int, default=1)
+ parser.add_argument("--max-submit", type=int, default=None)
+ parser.add_argument("--task-timeout", type=float, default=1800.0)
+ parser.add_argument("--request-timeout", type=float, default=60.0)
+ parser.add_argument("--poll-request-timeout", type=float, default=10.0)
+ parser.add_argument("--poll-interval", type=float, default=1.0)
+ parser.add_argument("--progress", action=argparse.BooleanOptionalAction, default=True)
+ parser.add_argument("--progress-interval", type=float, default=5.0)
+ parser.add_argument("--summary-interval", type=float, default=60.0)
+ parser.add_argument("--settings-json", default=None, help="JSON object or @path to JSON.")
+ parser.add_argument("--tweaks-json", default=None, help="JSON object or @path to JSON.")
+ parser.add_argument(
+ "--replace-duplicates",
+ action=argparse.BooleanOptionalAction,
+ default=True,
+ )
+ parser.add_argument("--runs-dir", default=str(DEFAULT_ARXIV_DIR / "runs"))
+ parser.add_argument("--output-dir", default=None)
+ parser.add_argument("--run-id", default=None)
+
+
+async def _upload_main(args: argparse.Namespace) -> int:
+ if not args.api_key:
+ raise ValueError("an API key is required; set OPENRAG_API_KEY or pass --api-key")
+ settings_json = _json_argument(args.settings_json, "settings")
+ tweaks_json = _json_argument(args.tweaks_json, "tweaks")
+ items = collect_upload_items(
+ args.paths,
+ include=args.include,
+ exclude=args.exclude,
+ recursive=args.recursive,
+ )
+ items = sort_upload_items(items, args.sort)
+ if not items:
+ raise ValueError("no files matched")
+
+ started = time.monotonic()
+ run_id = args.run_id or _new_run_id()
+ runs_dir = Path(args.runs_dir).expanduser().resolve()
+ output_dir = (
+ Path(args.output_dir).expanduser().resolve() if args.output_dir else runs_dir / run_id
+ )
+ progress = ProgressReporter(enabled=args.progress, started=started)
+
+ def save(summary: dict[str, Any]) -> dict[str, Any]:
+ return persist_summary(
+ summary,
+ output_dir=output_dir,
+ run_id=run_id,
+ started=started,
+ )
+
+ options = BulkUploadOptions(
+ batch_size=args.batch_size,
+ max_inflight=args.max_inflight,
+ max_submit=args.max_submit,
+ request_timeout=args.request_timeout,
+ poll_request_timeout=args.poll_request_timeout,
+ task_timeout=args.task_timeout,
+ poll_interval=args.poll_interval,
+ progress_interval=args.progress_interval,
+ summary_interval=args.summary_interval,
+ input_sort=args.sort,
+ settings_json=settings_json,
+ tweaks_json=tweaks_json,
+ replace_duplicates=args.replace_duplicates,
+ )
+ async with OpenRAGApiClient(
+ base_url=args.base_url,
+ api_key=args.api_key,
+ api_prefix=args.api_prefix,
+ ) as client:
+ result = await upload_bulk_items(
+ client=client,
+ items=items,
+ options=options,
+ progress=progress,
+ summary_callback=save,
+ )
+ result = save(result)
+ progress.log(f"complete summary={result['summary_path']}")
+ _print_json(result)
+ return 1 if bulk_failed(result) else 0
+
+
+def _validate_arxiv_args(parser: argparse.ArgumentParser, args: argparse.Namespace) -> None:
+ positive = {
+ "--max-results": args.max_results,
+ "--batch-size": args.batch_size,
+ "--max-inflight": args.max_inflight,
+ "--request-timeout": args.request_timeout,
+ "--poll-request-timeout": args.poll_request_timeout,
+ "--task-timeout": args.task_timeout,
+ "--max-pdf-mb": args.max_pdf_mb,
+ }
+ for name, value in positive.items():
+ if value <= 0:
+ parser.error(f"{name} must be positive")
+ if args.max_submit is not None and args.max_submit <= 0:
+ parser.error("--max-submit must be positive")
+ if args.start < 0:
+ parser.error("--start must be non-negative")
+ for name in ("poll_interval", "progress_interval", "summary_interval", "delay_seconds"):
+ if getattr(args, name) < 0:
+ parser.error(f"--{name.replace('_', '-')} must be non-negative")
+ try:
+ if args.query is None:
+ args.query = build_arxiv_query(
+ category=args.category,
+ date_from=args.date_from,
+ date_to=args.date_to,
+ )
+ except ValueError as exc:
+ parser.error(str(exc))
+ if args.source == "s3" and not str(args.s3_uri or "").strip():
+ parser.error("--s3-uri is required when --source=s3")
+
+
+async def _arxiv_main(args: argparse.Namespace) -> int:
+ if not args.download_only and not args.api_key:
+ raise ValueError(
+ "an API key is required unless --download-only is used; "
+ "set OPENRAG_API_KEY or pass --api-key"
+ )
+
+ started = time.monotonic()
+ run_id = args.run_id or _new_run_id()
+ runs_dir = Path(args.runs_dir).expanduser().resolve()
+ output_dir = (
+ Path(args.output_dir).expanduser().resolve() if args.output_dir else runs_dir / run_id
+ )
+ base_dir = runs_dir.parent
+ pdf_dir = (
+ Path(args.pdf_cache_dir).expanduser().resolve() if args.pdf_cache_dir else base_dir / "pdfs"
+ )
+ failure_cache_path = (
+ Path(args.failed_download_cache).expanduser().resolve()
+ if args.failed_download_cache
+ else pdf_dir / "_failed_downloads.json"
+ )
+ output_dir.mkdir(parents=True, exist_ok=True)
+ pdf_dir.mkdir(parents=True, exist_ok=True)
+ progress = ProgressReporter(
+ enabled=args.progress,
+ started=started,
+ prefix="openrag bench arxiv",
+ )
+ progress.log(f"run_id={run_id} output_dir={output_dir} pdf_cache_dir={pdf_dir}")
+
+ rate = RateLimiter(args.delay_seconds)
+ source_ref = str(args.query)
+ source_details: dict[str, Any] = {}
+ if args.source == "s3":
+ s3_cache_dir = (
+ Path(args.s3_cache_dir).expanduser().resolve() if args.s3_cache_dir else base_dir / "s3"
+ )
+ source_ref = str(args.s3_uri)
+ progress.log(
+ f"staging S3 tarball uri={args.s3_uri} start={args.start} "
+ f"max_results={args.max_results}"
+ )
+ downloads, download_errors, skipped_failures, source_details = stage_s3_papers(
+ s3_uri=args.s3_uri,
+ s3_cache_dir=s3_cache_dir,
+ pdf_dir=pdf_dir,
+ aws_cli=args.aws_cli,
+ request_payer=args.s3_request_payer,
+ start=args.start,
+ max_results=args.max_results,
+ max_pdf_bytes=args.max_pdf_mb * 1024 * 1024,
+ skip_existing=args.skip_existing,
+ progress=progress,
+ )
+ matched_papers = int(source_details.get("selected_pdf_members") or len(downloads))
+ else:
+ progress.log(
+ f"fetching Atom metadata query={args.query!r} start={args.start} "
+ f"max_results={args.max_results} sort={args.sort_by}:{args.sort_order}"
+ )
+ papers = fetch_arxiv_papers(
+ query=args.query,
+ start=args.start,
+ max_results=args.max_results,
+ sort_by=args.sort_by,
+ sort_order=args.sort_order,
+ rate=rate,
+ user_agent=args.user_agent,
+ timeout=args.request_timeout,
+ )
+ progress.log(f"metadata fetched matched_papers={len(papers)}")
+ failure_cache = load_failed_download_cache(failure_cache_path)
+ seeded = seed_failed_download_cache_from_summaries(runs_dir, source_ref=source_ref)
+ for key, record in seeded.items():
+ failure_cache.setdefault(key, record)
+ if seeded:
+ save_failed_download_cache(failure_cache_path, failure_cache)
+ downloads, download_errors, skipped_failures = download_papers(
+ papers,
+ pdf_dir=pdf_dir,
+ rate=rate,
+ user_agent=args.user_agent,
+ timeout=args.request_timeout,
+ max_pdf_bytes=args.max_pdf_mb * 1024 * 1024,
+ skip_existing=args.skip_existing,
+ failure_cache=failure_cache,
+ failure_cache_path=failure_cache_path,
+ retry_failed=args.retry_failed_downloads,
+ progress=progress,
+ )
+ matched_papers = len(papers)
+
+ cached_papers = sum(item.cached for item in downloads)
+ downloaded_papers = len(downloads) - cached_papers
+ progress.log(
+ f"PDF stage complete ready={len(downloads)} cached={cached_papers} "
+ f"downloaded={downloaded_papers} skipped_failed={len(skipped_failures)} "
+ f"errors={len(download_errors)}"
+ )
+ summary: dict[str, Any] = {
+ "run_id": run_id,
+ "source": args.source,
+ "source_ref": source_ref,
+ "output_dir": str(output_dir),
+ "pdf_cache_dir": str(pdf_dir),
+ "failed_download_cache_path": str(failure_cache_path),
+ "download_only": args.download_only,
+ "requested_papers": args.max_results,
+ "matched_papers": matched_papers,
+ "ready_papers": len(downloads),
+ "downloaded_papers": downloaded_papers,
+ "network_downloaded_papers": downloaded_papers,
+ "cached_papers": cached_papers,
+ "download_errors": download_errors,
+ "skipped_failed_papers": len(skipped_failures),
+ "skipped_failed_downloads": skipped_failures,
+ "concurrency": {"client": _arxiv_concurrency_settings(args)},
+ "batches": [],
+ "state": "downloaded",
+ }
+ if source_details:
+ summary["source_details"] = source_details
+ persist_arxiv_summary(summary, output_dir=output_dir, started=started)
+
+ if not args.download_only and downloads:
+ settings_json = _json_argument(args.settings_json, "settings")
+ tweaks_json = _json_argument(args.tweaks_json, "tweaks")
+ items = sort_upload_items(
+ [
+ BulkUploadItem(
+ path=item.path,
+ content_type="application/pdf",
+ )
+ for item in downloads
+ ],
+ args.upload_sort,
+ )
+ summary["state"] = "uploading"
+
+ def save_bulk(bulk_summary: dict[str, Any]) -> None:
+ summary["bulk"] = {
+ key: value for key, value in bulk_summary.items() if key != "batches"
+ }
+ summary["batches"] = bulk_summary.get("batches", [])
+ persist_arxiv_summary(summary, output_dir=output_dir, started=started)
+
+ async with OpenRAGApiClient(
+ base_url=args.base_url,
+ api_key=args.api_key,
+ api_prefix=args.api_prefix,
+ ) as client:
+ bulk_summary = await upload_bulk_items(
+ client=client,
+ items=items,
+ options=BulkUploadOptions(
+ batch_size=args.batch_size,
+ max_inflight=args.max_inflight,
+ max_submit=args.max_submit,
+ request_timeout=args.request_timeout,
+ poll_request_timeout=args.poll_request_timeout,
+ task_timeout=args.task_timeout,
+ poll_interval=args.poll_interval,
+ progress_interval=args.progress_interval,
+ summary_interval=args.summary_interval,
+ input_sort=args.upload_sort,
+ settings_json=settings_json,
+ tweaks_json=tweaks_json,
+ replace_duplicates=args.replace_duplicates,
+ ),
+ progress=progress,
+ summary_callback=save_bulk,
+ )
+ save_bulk(bulk_summary)
+
+ failed = (
+ not downloads
+ or bool(download_errors)
+ or (not args.download_only and bool(downloads) and bulk_failed(summary))
+ )
+ summary["state"] = "failed" if failed else "completed"
+ result = persist_arxiv_summary(summary, output_dir=output_dir, started=started)
+ progress.log(f"complete summary={result['summary_path']}")
+ _print_json(result)
+ return 1 if failed else 0
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ parser = build_parser()
+ args = parser.parse_args(argv)
+ try:
+ if args.command == "bulk":
+ if args.bulk_action == "list":
+ _print_json(
+ list_runs(
+ Path(args.runs_dir).expanduser().resolve(),
+ limit=args.limit,
+ )
+ )
+ return 0
+ if args.bulk_action == "summary":
+ _print_json(
+ load_summary_reference(
+ Path(args.runs_dir).expanduser().resolve(),
+ args.run,
+ detail=args.detail,
+ )
+ )
+ return 0
+ return asyncio.run(_upload_main(args))
+ if args.command == "bench" and args.benchmark == "arxiv":
+ runs_dir = Path(args.runs_dir).expanduser().resolve()
+ if args.benchmark_action == "list":
+ _print_json(list_runs(runs_dir, limit=args.limit))
+ return 0
+ if args.benchmark_action == "summary":
+ _print_json(
+ load_summary_reference(
+ runs_dir,
+ args.run,
+ detail=args.detail,
+ )
+ )
+ return 0
+ _validate_arxiv_args(parser, args)
+ return asyncio.run(_arxiv_main(args))
+ parser.error(f"unsupported command: {args.command}")
+ except (
+ ApiError,
+ ArxivBenchmarkError,
+ FileNotFoundError,
+ ValueError,
+ json.JSONDecodeError,
+ tarfile.TarError,
+ ) as exc:
+ print(f"openrag bulk: {exc}", file=sys.stderr)
+ return 2
+ except KeyboardInterrupt:
+ print("openrag bulk: interrupted", file=sys.stderr)
+ return 130
+
+
+def build_arxiv_query(*, category: str, date_from: str, date_to: str) -> str:
+ category = category.strip()
+ if not category:
+ raise ValueError("--category must not be empty")
+ start = arxiv_date_bound(date_from, end=False)
+ stop = arxiv_date_bound(date_to, end=True)
+ if start > stop:
+ raise ValueError("--date-from must be before --date-to")
+ return f"cat:{category} AND submittedDate:[{start} TO {stop}]"
+
+
+def arxiv_date_bound(value: str, *, end: bool) -> str:
+ value = value.strip()
+ if re.fullmatch(r"\d{12}", value):
+ return value
+ if re.fullmatch(r"\d{8}", value):
+ return value + ("2359" if end else "0000")
+ match = re.fullmatch(r"(\d{4})-(\d{2})-(\d{2})", value)
+ if match:
+ return "".join(match.groups()) + ("2359" if end else "0000")
+ raise ValueError("arXiv dates must use YYYY-MM-DD, YYYYMMDD, or YYYYMMDDHHMM")
+
+
+def stage_s3_papers(
+ *,
+ s3_uri: str,
+ s3_cache_dir: Path,
+ pdf_dir: Path,
+ aws_cli: str,
+ request_payer: str,
+ start: int,
+ max_results: int,
+ max_pdf_bytes: int,
+ skip_existing: bool,
+ progress: ProgressReporter,
+) -> tuple[list[DownloadedPaper], list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]:
+ tarball, tarball_downloaded = ensure_s3_tarball(
+ s3_uri=s3_uri,
+ cache_dir=s3_cache_dir,
+ aws_cli=aws_cli,
+ request_payer=request_payer,
+ skip_existing=skip_existing,
+ progress=progress,
+ )
+ downloads, errors = extract_s3_tarball_papers(
+ tarball,
+ pdf_dir=pdf_dir,
+ start=start,
+ max_results=max_results,
+ max_pdf_bytes=max_pdf_bytes,
+ skip_existing=skip_existing,
+ progress=progress,
+ )
+ details = {
+ "s3_uri": s3_uri,
+ "tarball_path": str(tarball),
+ "tarball_bytes": tarball.stat().st_size,
+ "tarball_downloaded": tarball_downloaded,
+ "s3_cache_dir": str(s3_cache_dir),
+ "selected_pdf_members": len(downloads) + len(errors),
+ "ready_pdf_members": len(downloads),
+ "failed_pdf_members": len(errors),
+ }
+ return downloads, errors, [], details
+
+
+def ensure_s3_tarball(
+ *,
+ s3_uri: str,
+ cache_dir: Path,
+ aws_cli: str,
+ request_payer: str,
+ skip_existing: bool,
+ progress: ProgressReporter,
+) -> tuple[Path, bool]:
+ cache_dir.mkdir(parents=True, exist_ok=True)
+ target = cache_dir / tarball_cache_filename(s3_uri)
+ if skip_existing and target.is_file() and target.stat().st_size > 0:
+ progress.log(f"S3 tarball cached path={target} bytes={target.stat().st_size}")
+ return target, False
+
+ temporary = target.with_name(target.name + ".tmp")
+ if temporary.exists():
+ temporary.unlink()
+ parsed = urlparse(s3_uri)
+ progress.log(f"S3 tarball downloading source={s3_uri} target={target}")
+ if parsed.scheme == "s3":
+ command = [aws_cli, "s3", "cp", s3_uri, str(temporary)]
+ if request_payer:
+ command.extend(["--request-payer", request_payer])
+ result = subprocess.run(command, text=True, capture_output=True, check=False)
+ if result.returncode != 0:
+ detail = (result.stderr or result.stdout or "").strip()
+ raise ArxivBenchmarkError(
+ f"failed to download arXiv S3 tarball {s3_uri}: {detail or command!r}"
+ )
+ elif parsed.scheme == "file":
+ shutil.copyfile(Path(unquote(parsed.path)).expanduser(), temporary)
+ elif parsed.scheme:
+ raise ArxivBenchmarkError(f"unsupported arXiv S3 URI scheme: {parsed.scheme}")
+ else:
+ shutil.copyfile(Path(s3_uri).expanduser(), temporary)
+ temporary.replace(target)
+ progress.log(f"S3 tarball downloaded path={target} bytes={target.stat().st_size}")
+ return target, True
+
+
+def extract_s3_tarball_papers(
+ tarball: Path,
+ *,
+ pdf_dir: Path,
+ start: int,
+ max_results: int,
+ max_pdf_bytes: int,
+ skip_existing: bool,
+ progress: ProgressReporter,
+) -> tuple[list[DownloadedPaper], list[dict[str, Any]]]:
+ pdf_dir.mkdir(parents=True, exist_ok=True)
+ downloads: list[DownloadedPaper] = []
+ errors: list[dict[str, Any]] = []
+ used_filenames: set[str] = set()
+ seen_pdf_members = 0
+ selected_members = 0
+ with tarfile.open(tarball, mode="r:*") as archive:
+ for member in archive:
+ if not member.isfile() or not member.name.lower().endswith(".pdf"):
+ continue
+ seen_pdf_members += 1
+ if seen_pdf_members <= start:
+ continue
+ if selected_members >= max_results:
+ break
+ selected_members += 1
+ paper = s3_member_paper(member.name)
+ filename = unique_filename(
+ pdf_dir,
+ pdf_filename(paper),
+ used_filenames,
+ allow_existing=skip_existing,
+ )
+ path = pdf_dir / filename
+ cached = skip_existing and path.is_file()
+ try:
+ if not cached:
+ if member.size > max_pdf_bytes:
+ raise ValueError(f"PDF is larger than limit: {member.size} bytes")
+ source = archive.extractfile(member)
+ if source is None:
+ raise ValueError("tar member has no file content")
+ with source, path.open("wb") as output:
+ shutil.copyfileobj(source, output, length=1024 * 1024)
+ if path.stat().st_size > max_pdf_bytes:
+ raise ValueError(f"PDF exceeded limit: {max_pdf_bytes} bytes")
+ progress.log(
+ f"S3 PDF {selected_members}/{max_results}: extracted "
+ f"arxiv_id={paper.arxiv_id} bytes={path.stat().st_size}"
+ )
+ else:
+ progress.log(
+ f"S3 PDF {selected_members}/{max_results}: cached "
+ f"arxiv_id={paper.arxiv_id} path={path}"
+ )
+ downloads.append(
+ DownloadedPaper(
+ paper=paper,
+ path=path,
+ size=path.stat().st_size,
+ sha256=sha256_file(path),
+ cached=cached,
+ )
+ )
+ except Exception as exc:
+ if not cached and path.exists():
+ path.unlink()
+ progress.log(
+ f"S3 PDF {selected_members}/{max_results}: failed member={member.name} "
+ f"error={_error_text(exc)}"
+ )
+ errors.append(
+ {
+ "arxiv_id": paper.arxiv_id,
+ "pdf_url": paper.pdf_url,
+ "error": _error_text(exc),
+ "tar_member": member.name,
+ }
+ )
+ progress.log(
+ f"S3 tarball scan complete pdf_members_seen={seen_pdf_members} "
+ f"selected={selected_members} ready={len(downloads)} errors={len(errors)}"
+ )
+ return downloads, errors
+
+
+def s3_member_paper(member_name: str) -> ArxivPaper:
+ arxiv_id = arxiv_id_from_pdf_member(member_name)
+ return ArxivPaper(
+ arxiv_id=arxiv_id,
+ title=arxiv_id,
+ summary="",
+ authors=(),
+ published=None,
+ updated=None,
+ abs_url=f"https://arxiv.org/abs/{arxiv_id}",
+ pdf_url=f"https://arxiv.org/pdf/{arxiv_id}",
+ )
+
+
+def arxiv_id_from_pdf_member(member_name: str) -> str:
+ filename = Path(member_name).name
+ stem = filename[:-4] if filename.lower().endswith(".pdf") else Path(filename).stem
+ stem = stem.strip()
+ return stem or hashlib.sha256(member_name.encode()).hexdigest()[:16]
+
+
+def tarball_cache_filename(uri: str) -> str:
+ parsed = urlparse(uri)
+ name = Path(unquote(parsed.path)).name if parsed.path else Path(uri).name
+ name = safe_slug(name, 160)
+ digest = hashlib.sha256(uri.encode()).hexdigest()[:12]
+ suffix = "".join(Path(name).suffixes)
+ stem = name[: -len(suffix)] if suffix else name
+ return f"{stem}-{digest}{suffix or '.tar'}"
+
+
+def fetch_arxiv_papers(
+ *,
+ query: str,
+ start: int,
+ max_results: int,
+ sort_by: str,
+ sort_order: str,
+ rate: RateLimiter,
+ user_agent: str,
+ timeout: float,
+) -> list[ArxivPaper]:
+ url = f"{ARXIV_API_URL}?" + urlencode(
+ {
+ "search_query": query,
+ "start": str(start),
+ "max_results": str(max_results),
+ "sortBy": sort_by,
+ "sortOrder": sort_order,
+ }
+ )
+ rate.wait()
+ try:
+ payload = read_url(
+ url,
+ user_agent=user_agent,
+ timeout=timeout,
+ accept="application/atom+xml",
+ )
+ except HTTPError as exc:
+ if exc.code == 429:
+ raise ArxivBenchmarkError(
+ "arXiv returned HTTP 429 for metadata; wait before retrying, or ingest "
+ "already-cached PDFs with "
+ "`openrag_bulk.py bulk upload <pdf-cache> --include '*.pdf'`"
+ ) from exc
+ raise
+ return parse_arxiv_feed(payload)
+
+
+def parse_arxiv_feed(payload: bytes) -> list[ArxivPaper]:
+ root = ET.fromstring(payload)
+ papers: list[ArxivPaper] = []
+ for entry in root.findall(f"{ATOM}entry"):
+ abs_url = _atom_text(entry, "id")
+ title = collapse_ws(_atom_text(entry, "title"))
+ summary = collapse_ws(_atom_text(entry, "summary"))
+ authors = tuple(
+ collapse_ws(_atom_text(author, "name"))
+ for author in entry.findall(f"{ATOM}author")
+ if collapse_ws(_atom_text(author, "name"))
+ )
+ pdf_url = ""
+ for link in entry.findall(f"{ATOM}link"):
+ if link.attrib.get("title") == "pdf" or link.attrib.get("type") == "application/pdf":
+ pdf_url = link.attrib.get("href", "")
+ break
+ if not abs_url or not pdf_url:
+ continue
+ papers.append(
+ ArxivPaper(
+ arxiv_id=arxiv_id_from_abs_url(abs_url),
+ title=title,
+ summary=summary,
+ authors=authors,
+ published=_atom_text(entry, "published") or None,
+ updated=_atom_text(entry, "updated") or None,
+ abs_url=https_url(abs_url),
+ pdf_url=https_url(pdf_url),
+ )
+ )
+ return papers
+
+
+def download_papers(
+ papers: list[ArxivPaper],
+ *,
+ pdf_dir: Path,
+ rate: RateLimiter,
+ user_agent: str,
+ timeout: float,
+ max_pdf_bytes: int,
+ skip_existing: bool,
+ failure_cache: dict[str, dict[str, Any]] | None = None,
+ failure_cache_path: Path | None = None,
+ retry_failed: bool = False,
+ progress: ProgressReporter,
+) -> tuple[list[DownloadedPaper], list[dict[str, Any]], list[dict[str, Any]]]:
+ downloads: list[DownloadedPaper] = []
+ errors: list[dict[str, Any]] = []
+ skipped_failures: list[dict[str, Any]] = []
+ failure_cache = failure_cache if failure_cache is not None else {}
+ used_filenames: set[str] = set()
+ pdf_dir.mkdir(parents=True, exist_ok=True)
+ for index, paper in enumerate(papers, start=1):
+ failure_key = failed_download_cache_key(paper)
+ filename = unique_filename(
+ pdf_dir,
+ pdf_filename(paper),
+ used_filenames,
+ allow_existing=skip_existing,
+ )
+ path = pdf_dir / filename
+ cached = skip_existing and path.is_file()
+ cached_failure = failure_cache.get(failure_key)
+ if cached_failure and not retry_failed and not cached:
+ skipped = public_failed_download_record(cached_failure)
+ progress.log(
+ f"PDF {index}/{len(papers)}: skipped known failed "
+ f"arxiv_id={paper.arxiv_id} error={skipped['error']}"
+ )
+ skipped_failures.append(skipped)
+ continue
+ try:
+ if not cached:
+ progress.log(
+ f"PDF {index}/{len(papers)}: downloading "
+ f"arxiv_id={paper.arxiv_id} url={paper.pdf_url}"
+ )
+ rate.wait()
+ download_pdf(
+ paper.pdf_url,
+ path=path,
+ user_agent=user_agent,
+ timeout=timeout,
+ max_pdf_bytes=max_pdf_bytes,
+ )
+ else:
+ progress.log(
+ f"PDF {index}/{len(papers)}: cached arxiv_id={paper.arxiv_id} path={path}"
+ )
+ downloads.append(
+ DownloadedPaper(
+ paper=paper,
+ path=path,
+ size=path.stat().st_size,
+ sha256=sha256_file(path),
+ cached=cached,
+ )
+ )
+ if failure_key in failure_cache:
+ failure_cache.pop(failure_key, None)
+ save_failed_download_cache(failure_cache_path, failure_cache)
+ except Exception as exc:
+ progress.log(
+ f"PDF {index}/{len(papers)}: failed arxiv_id={paper.arxiv_id} "
+ f"error={_error_text(exc)}"
+ )
+ record = failed_download_record(paper, exc, previous=failure_cache.get(failure_key))
+ failure_cache[failure_key] = record
+ save_failed_download_cache(failure_cache_path, failure_cache)
+ errors.append(public_failed_download_record(record))
+ return downloads, errors, skipped_failures
+
+
+def download_pdf(
+ url: str,
+ *,
+ path: Path,
+ user_agent: str,
+ timeout: float,
+ max_pdf_bytes: int,
+) -> None:
+ request = Request(url, headers={"User-Agent": user_agent, "Accept": "application/pdf"})
+ total = 0
+ try:
+ with urlopen(request, timeout=timeout) as response, path.open("wb") as output:
+ length = response.headers.get("Content-Length")
+ if length and int(length) > max_pdf_bytes:
+ raise ValueError(f"PDF is larger than limit: {int(length)} bytes")
+ while chunk := response.read(128 * 1024):
+ total += len(chunk)
+ if total > max_pdf_bytes:
+ raise ValueError(f"PDF exceeded limit: {max_pdf_bytes} bytes")
+ output.write(chunk)
+ except Exception:
+ if path.exists():
+ path.unlink()
+ raise
+
+
+def read_url(url: str, *, user_agent: str, timeout: float, accept: str) -> bytes:
+ request = Request(url, headers={"User-Agent": user_agent, "Accept": accept})
+ with urlopen(request, timeout=timeout) as response:
+ return response.read()
+
+
+def load_failed_download_cache(path: Path) -> dict[str, dict[str, Any]]:
+ if not path.is_file():
+ return {}
+ parsed = json.loads(path.read_text(encoding="utf-8"))
+ failures = parsed.get("failures") if isinstance(parsed, dict) else None
+ if not isinstance(failures, dict):
+ return {}
+ return {str(key): value for key, value in failures.items() if isinstance(value, dict)}
+
+
+def seed_failed_download_cache_from_summaries(
+ runs_dir: Path,
+ *,
+ source_ref: str,
+) -> dict[str, dict[str, Any]]:
+ if not runs_dir.is_dir():
+ return {}
+ seeded: dict[str, dict[str, Any]] = {}
+ for summary_path in sorted(runs_dir.glob("*/summary.json")):
+ try:
+ summary = json.loads(summary_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError):
+ continue
+ if not isinstance(summary, dict) or summary.get("source_ref") != source_ref:
+ continue
+ for record in summary_download_failure_records(summary):
+ arxiv_id = str(record.get("arxiv_id") or "")
+ if arxiv_id:
+ seeded[arxiv_id] = {
+ "arxiv_id": arxiv_id,
+ "pdf_url": str(record.get("pdf_url") or ""),
+ "error": str(record.get("error") or "previous benchmark download failure"),
+ "first_failed_at": str(record.get("first_failed_at") or ""),
+ "last_failed_at": str(record.get("last_failed_at") or ""),
+ "attempts": record.get("attempts") or 1,
+ "seeded_from_summary": str(summary_path),
+ }
+ return seeded
+
+
+def summary_download_failure_records(summary: dict[str, Any]) -> list[dict[str, Any]]:
+ records: list[dict[str, Any]] = []
+ for key in ("download_errors", "skipped_failed_downloads"):
+ value = summary.get(key)
+ if isinstance(value, list):
+ records.extend(item for item in value if isinstance(item, dict))
+ return records
+
+
+def save_failed_download_cache(
+ path: Path | None,
+ failures: dict[str, dict[str, Any]],
+) -> None:
+ if path is None:
+ return
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_name(path.name + ".tmp")
+ temporary.write_text(
+ json.dumps({"version": 1, "failures": failures}, indent=2, sort_keys=True) + "\n",
+ encoding="utf-8",
+ )
+ temporary.replace(path)
+
+
+def failed_download_cache_key(paper: ArxivPaper) -> str:
+ return paper.arxiv_id or hashlib.sha256(paper.pdf_url.encode()).hexdigest()
+
+
+def failed_download_record(
+ paper: ArxivPaper,
+ exc: Exception,
+ *,
+ previous: dict[str, Any] | None,
+) -> dict[str, Any]:
+ now = datetime.now(UTC).isoformat()
+ first_failed_at = str((previous or {}).get("first_failed_at") or now)
+ try:
+ attempts = int((previous or {}).get("attempts") or 0) + 1
+ except (TypeError, ValueError):
+ attempts = 1
+ return {
+ "arxiv_id": paper.arxiv_id,
+ "abs_url": paper.abs_url,
+ "pdf_url": paper.pdf_url,
+ "title": paper.title,
+ "error_type": type(exc).__name__,
+ "error": _error_text(exc),
+ "first_failed_at": first_failed_at,
+ "last_failed_at": now,
+ "attempts": attempts,
+ }
+
+
+def public_failed_download_record(record: dict[str, Any]) -> dict[str, Any]:
+ return {
+ "arxiv_id": str(record.get("arxiv_id") or ""),
+ "pdf_url": str(record.get("pdf_url") or ""),
+ "error": str(record.get("error") or "unknown download error"),
+ "first_failed_at": str(record.get("first_failed_at") or ""),
+ "last_failed_at": str(record.get("last_failed_at") or ""),
+ "attempts": record.get("attempts") or None,
+ }
+
+
+def persist_arxiv_summary(
+ summary: dict[str, Any],
+ *,
+ output_dir: Path,
+ started: float,
+) -> dict[str, Any]:
+ summary_path = output_dir / "summary.json"
+ summary["summary_path"] = str(summary_path)
+ summary["elapsed_seconds"] = round(time.monotonic() - started, 3)
+ ordered = _ordered_arxiv_summary(summary)
+ temporary = output_dir / ".summary.json.tmp"
+ temporary.write_text(json.dumps(ordered, indent=2) + "\n", encoding="utf-8")
+ temporary.replace(summary_path)
+ return ordered
+
+
+def _ordered_arxiv_summary(summary: dict[str, Any]) -> dict[str, Any]:
+ preferred = (
+ "run_id",
+ "summary_path",
+ "output_dir",
+ "pdf_cache_dir",
+ "failed_download_cache_path",
+ "source",
+ "source_ref",
+ "download_only",
+ "requested_papers",
+ "matched_papers",
+ "ready_papers",
+ "downloaded_papers",
+ "network_downloaded_papers",
+ "cached_papers",
+ "download_errors",
+ "skipped_failed_papers",
+ "skipped_failed_downloads",
+ "elapsed_seconds",
+ "state",
+ "concurrency",
+ "bulk",
+ )
+ ordered = {key: summary[key] for key in preferred if key in summary}
+ ordered.update(
+ {key: value for key, value in summary.items() if key not in ordered and key != "batches"}
+ )
+ if "batches" in summary:
+ ordered["batches"] = summary["batches"]
+ return ordered
+
+
+def _arxiv_concurrency_settings(args: argparse.Namespace) -> dict[str, Any]:
+ source: dict[str, Any] = {
+ "source": args.source,
+ "max_inflight_requests": 1,
+ "request_timeout_seconds": args.request_timeout,
+ "skip_existing": args.skip_existing,
+ }
+ if args.source == "s3":
+ source.update(s3_uri=args.s3_uri, s3_request_payer=args.s3_request_payer or None)
+ else:
+ source.update(
+ delay_seconds=args.delay_seconds,
+ retry_failed_downloads=args.retry_failed_downloads,
+ )
+ return {
+ "pdf_source": source,
+ "upload": {
+ "batch_size": args.batch_size,
+ "max_inflight_upload_tasks": args.max_inflight,
+ "max_inflight_submit_requests": args.max_submit or args.max_inflight,
+ "input_sort": args.upload_sort,
+ "request_timeout_seconds": args.request_timeout,
+ "poll_request_timeout_seconds": args.poll_request_timeout,
+ "task_timeout_seconds": args.task_timeout,
+ "poll_interval_seconds": args.poll_interval,
+ },
+ }
+
+
+def _atom_text(entry: ET.Element, name: str) -> str:
+ child = entry.find(f"{ATOM}{name}")
+ return child.text.strip() if child is not None and child.text else ""
+
+
+def collapse_ws(value: str) -> str:
+ return re.sub(r"\s+", " ", value).strip()
+
+
+def arxiv_id_from_abs_url(abs_url: str) -> str:
+ return urlparse(abs_url).path.rstrip("/").rsplit("/", 1)[-1]
+
+
+def https_url(value: str) -> str:
+ parsed = urlparse(value)
+ return urlunparse(parsed._replace(scheme="https")) if parsed.scheme == "http" else value
+
+
+def pdf_filename(paper: ArxivPaper) -> str:
+ return f"{safe_slug(paper.arxiv_id, 120)}.pdf"
+
+
+def safe_slug(value: str, max_length: int) -> str:
+ slug = re.sub(r"[^A-Za-z0-9._-]+", "-", value).strip(".-_")
+ slug = re.sub(r"-{2,}", "-", slug)
+ return slug[:max_length].strip(".-_") or "paper"
+
+
+def unique_filename(
+ directory: Path,
+ filename: str,
+ used: set[str],
+ *,
+ allow_existing: bool = False,
+) -> str:
+ stem = Path(filename).stem
+ suffix = Path(filename).suffix
+ candidate = filename
+ index = 2
+ while candidate in used or ((directory / candidate).exists() and not allow_existing):
+ candidate = f"{stem}-{index}{suffix}"
+ index += 1
+ used.add(candidate)
+ return candidate
+
+
+def sha256_file(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _response_json(response: httpx.Response) -> dict[str, Any]:
+ if response.is_error:
+ try:
+ payload = response.json()
+ message = payload.get("error") or payload.get("detail") or response.text
+ except (ValueError, TypeError):
+ message = response.text
+ raise ApiError(str(message or f"HTTP {response.status_code}"), response.status_code)
+ try:
+ payload = response.json()
+ except ValueError as exc:
+ raise ApiError("OpenRAG returned a non-JSON response", response.status_code) from exc
+ if not isinstance(payload, dict):
+ raise ApiError("OpenRAG returned a non-object JSON response", response.status_code)
+ return payload
+
+
+def _new_batch_result(batch: Sequence[BulkUploadItem], *, batch_index: int) -> dict[str, Any]:
+ byte_count = sum(item.path.stat().st_size for item in batch)
+ return {
+ "batch_index": batch_index,
+ "file_count": len(batch),
+ "files": [str(item.path) for item in batch],
+ "input_bytes": byte_count,
+ "input_mb": round(_mb(byte_count), 3),
+ "status": "queued",
+ "phase": "queued",
+ }
+
+
+def _update_batch_from_status(result: dict[str, Any], status: dict[str, Any]) -> None:
+ result.update(
+ status=str(status.get("status") or "unknown"),
+ phase=_task_phase(status),
+ duration_seconds=status.get("duration_seconds"),
+ processed_files=status.get("processed_files"),
+ successful_files=status.get("successful_files"),
+ failed_files=status.get("failed_files"),
+ running_files=status.get("running_files"),
+ pending_files=status.get("pending_files"),
+ error=_task_error(status) or None,
+ )
+
+
+def _task_phase(status: dict[str, Any]) -> str:
+ if status.get("phase"):
+ return str(status["phase"])
+ phases = {
+ str(file_status.get("phase"))
+ for file_status in (status.get("files") or {}).values()
+ if isinstance(file_status, dict) and file_status.get("phase")
+ }
+ if len(phases) == 1:
+ return phases.pop()
+ if phases:
+ return "mixed"
+ return "complete" if status.get("status") in TERMINAL_TASK_STATUSES else "task_poll"
+
+
+def _task_error(status: dict[str, Any]) -> str:
+ if status.get("error"):
+ return _summarize_error(status["error"])
+ errors = [
+ file_status.get("error")
+ for file_status in (status.get("files") or {}).values()
+ if isinstance(file_status, dict) and file_status.get("error")
+ ]
+ if not errors:
+ return ""
+ first = _summarize_error(errors[0])
+ return first if len(errors) == 1 else f"{first} (+{len(errors) - 1} more)"
+
+
+def _status_snapshot(status: dict[str, Any]) -> tuple[Any, ...]:
+ return (
+ status.get("status"),
+ _task_phase(status),
+ status.get("processed_files"),
+ status.get("successful_files"),
+ status.get("failed_files"),
+ status.get("running_files"),
+ status.get("pending_files"),
+ _task_error(status),
+ )
+
+
+def _update_summary_totals(summary: dict[str, Any]) -> None:
+ batches = summary.get("batches") or []
+ summary["successful_files"] = sum(_int_value(item.get("successful_files")) for item in batches)
+ summary["failed_files"] = sum(_int_value(item.get("failed_files")) for item in batches)
+ summary["completed_batches"] = sum(item.get("status") == "completed" for item in batches)
+ summary["failed_batches"] = sum(item.get("status") != "completed" for item in batches)
+
+
+def _run_overview(summary: dict[str, Any], path: Path) -> dict[str, Any]:
+ statuses: dict[str, int] = {}
+ for batch in summary.get("batches") or []:
+ status = str(batch.get("status") or "unknown")
+ statuses[status] = statuses.get(status, 0) + 1
+ if summary.get("state"):
+ state = str(summary["state"])
+ elif not statuses:
+ state = "unknown"
+ elif any(status not in TERMINAL_TASK_STATUSES for status in statuses):
+ state = "running"
+ elif any(status != "completed" for status in statuses):
+ state = "failed"
+ else:
+ state = "completed"
+ return {
+ "run_id": str(summary.get("run_id") or path.parent.name),
+ "state": state,
+ "updated_at": datetime.fromtimestamp(path.stat().st_mtime, tz=UTC).isoformat(),
+ "summary_path": str(path),
+ "elapsed_seconds": summary.get("elapsed_seconds"),
+ "total_files": summary.get("total_files", summary.get("ready_papers")),
+ "batch_count": len(summary.get("batches") or []),
+ "batch_status_counts": statuses,
+ }
+
+
+def _load_summary(path: Path) -> dict[str, Any]:
+ summary = json.loads(path.read_text(encoding="utf-8"))
+ if not isinstance(summary, dict):
+ raise ValueError(f"summary is not a JSON object: {path}")
+ return summary
+
+
+def _json_argument(value: str | None, label: str) -> str | None:
+ if value is None:
+ return None
+ raw = (
+ Path(value[1:]).expanduser().read_text(encoding="utf-8") if value.startswith("@") else value
+ )
+ parsed = json.loads(raw)
+ if not isinstance(parsed, dict):
+ raise ValueError(f"{label} JSON must be an object")
+ return json.dumps(parsed, separators=(",", ":"))
+
+
+def _ordered_summary(summary: dict[str, Any]) -> dict[str, Any]:
+ preferred = (
+ "run_id",
+ "summary_path",
+ "output_dir",
+ "elapsed_seconds",
+ "total_files",
+ "successful_files",
+ "failed_files",
+ "completed_batches",
+ "failed_batches",
+ "batch_size",
+ "max_inflight",
+ "max_submit",
+ "client_concurrency",
+ )
+ ordered = {key: summary[key] for key in preferred if key in summary}
+ ordered.update(
+ {key: value for key, value in summary.items() if key not in ordered and key != "batches"}
+ )
+ if "batches" in summary:
+ ordered["batches"] = summary["batches"]
+ return ordered
+
+
+def _notify(callback: SummaryCallback | None, summary: dict[str, Any]) -> None:
+ if callback:
+ callback(summary)
+
+
+def _matches(relative: str, name: str, patterns: Sequence[str]) -> bool:
+ return any(
+ fnmatch.fnmatch(relative, pattern) or fnmatch.fnmatch(name, pattern) for pattern in patterns
+ )
+
+
+def _chunks(items: list[BulkUploadItem], size: int) -> list[list[BulkUploadItem]]:
+ return [items[index : index + size] for index in range(0, len(items), size)]
+
+
+def _should_log(now: float, last_log: float, interval: float) -> bool:
+ return interval == 0 or now - last_log >= interval
+
+
+def _is_transient_error(exc: BaseException) -> bool:
+ return isinstance(exc, (httpx.TransportError, TimeoutError)) or (
+ isinstance(exc, ApiError) and exc.status_code in TRANSIENT_HTTP_STATUSES
+ )
+
+
+def _error_text(exc: BaseException) -> str:
+ return f"{type(exc).__name__}: {_summarize_error(exc)}"
+
+
+def _summarize_error(value: Any) -> str:
+ lines = [line.strip() for line in str(value).splitlines() if line.strip()]
+ text = lines[-1] if lines else str(value).strip()
+ return text if len(text) <= 320 else text[:317] + "..."
+
+
+def _int_value(value: Any) -> int:
+ try:
+ return int(value or 0)
+ except (TypeError, ValueError):
+ return 0
+
+
+def _mb(value: int | float) -> float:
+ return float(value) / 1_000_000
+
+
+def _new_run_id() -> str:
+ return datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ") + "-" + uuid4().hex[:8]
+
+
+def _print_json(value: Any) -> None:
+ print(json.dumps(value, indent=2))
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/tests/unit/test_openrag_arxiv.py b/tests/unit/test_openrag_arxiv.py
new file mode 100644
index 000000000..31a4bd204
--- /dev/null
+++ b/tests/unit/test_openrag_arxiv.py
@@ -0,0 +1,350 @@
+from __future__ import annotations
+
+import json
+import tarfile
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from scripts import openrag_bulk as bulk
+
+
+def _args(*values: str):
+ parser = bulk.build_parser()
+ args = parser.parse_args(["bench", "arxiv", *values])
+ bulk._validate_arxiv_args(parser, args)
+ return args
+
+
+def _paper(arxiv_id: str = "2501.01234v1") -> bulk.ArxivPaper:
+ return bulk.ArxivPaper(
+ arxiv_id=arxiv_id,
+ title="Benchmark Paper",
+ summary="Summary",
+ authors=("Ada Lovelace",),
+ published="2025-01-01T00:00:00Z",
+ updated="2025-01-02T00:00:00Z",
+ abs_url=f"https://arxiv.org/abs/{arxiv_id}",
+ pdf_url=f"https://arxiv.org/pdf/{arxiv_id}",
+ )
+
+
+def test_arxiv_defaults_and_category_date_query() -> None:
+ defaults = _args("--download-only")
+ atom = _args(
+ "--download-only",
+ "--source",
+ "atom",
+ "--category",
+ "cs.AI",
+ "--date-from",
+ "2024-03-01",
+ "--date-to",
+ "2024-03-31",
+ )
+
+ assert defaults.source == "s3"
+ assert defaults.s3_uri == bulk.DEFAULT_ARXIV_S3_URI
+ assert defaults.skip_existing is True
+ assert defaults.query == "cat:cs.CL AND submittedDate:[202501010000 TO 202512312359]"
+ assert atom.query == "cat:cs.AI AND submittedDate:[202403010000 TO 202403312359]"
+
+
+def test_parse_arxiv_feed_extracts_metadata_and_https_urls() -> None:
+ feed = b"""<?xml version="1.0" encoding="UTF-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom">
+ <entry>
+ <id>http://arxiv.org/abs/2501.01234v2</id>
+ <updated>2025-01-03T00:00:00Z</updated>
+ <published>2025-01-01T00:00:00Z</published>
+ <title> Retrieval
+ Benchmarking for Agents </title>
+ <summary>Short summary.</summary>
+ <author><name>Ada Lovelace</name></author>
+ <author><name>Alan Turing</name></author>
+ <link href="http://arxiv.org/pdf/2501.01234v2" type="application/pdf" title="pdf"/>
+ </entry>
+</feed>
+"""
+
+ [paper] = bulk.parse_arxiv_feed(feed)
+
+ assert paper.arxiv_id == "2501.01234v2"
+ assert paper.title == "Retrieval Benchmarking for Agents"
+ assert paper.authors == ("Ada Lovelace", "Alan Turing")
+ assert paper.abs_url == "https://arxiv.org/abs/2501.01234v2"
+ assert paper.pdf_url == "https://arxiv.org/pdf/2501.01234v2"
+
+
+def test_fetch_arxiv_papers_builds_atom_request(monkeypatch: pytest.MonkeyPatch) -> None:
+ captured: dict[str, Any] = {}
+ feed = b"""<feed xmlns="http://www.w3.org/2005/Atom">
+ <entry>
+ <id>https://arxiv.org/abs/2501.00001v1</id>
+ <title>Paper</title><summary>Summary</summary>
+ <link href="https://arxiv.org/pdf/2501.00001v1" type="application/pdf"/>
+ </entry>
+ </feed>"""
+
+ def fake_read_url(url: str, **kwargs):
+ captured.update(url=url, **kwargs)
+ return feed
+
+ monkeypatch.setattr(bulk, "read_url", fake_read_url)
+ papers = bulk.fetch_arxiv_papers(
+ query="cat:cs.AI",
+ start=5,
+ max_results=2,
+ sort_by="submittedDate",
+ sort_order="descending",
+ rate=bulk.RateLimiter(0),
+ user_agent="openrag-test",
+ timeout=4,
+ )
+
+ assert papers[0].arxiv_id == "2501.00001v1"
+ assert "search_query=cat%3Acs.AI" in captured["url"]
+ assert "start=5" in captured["url"]
+ assert "max_results=2" in captured["url"]
+ assert captured["user_agent"] == "openrag-test"
+ assert captured["accept"] == "application/atom+xml"
+
+
+def test_stage_s3_papers_caches_tarball_and_extracted_pdfs(tmp_path: Path) -> None:
+ source_dir = tmp_path / "source"
+ source_dir.mkdir()
+ first = source_dir / "2501.00001v1.pdf"
+ second = source_dir / "2501.00002v1.pdf"
+ first.write_bytes(b"%PDF first")
+ second.write_bytes(b"%PDF second")
+ tarball = tmp_path / "arxiv.tar"
+ with tarfile.open(tarball, "w") as archive:
+ archive.add(first, arcname="pdf/2501/2501.00001v1.pdf")
+ archive.add(second, arcname="pdf/2501/2501.00002v1.pdf")
+
+ downloads, errors, skipped, details = bulk.stage_s3_papers(
+ s3_uri=str(tarball),
+ s3_cache_dir=tmp_path / "s3-cache",
+ pdf_dir=tmp_path / "pdf-cache",
+ aws_cli="aws",
+ request_payer="",
+ start=0,
+ max_results=2,
+ max_pdf_bytes=1024,
+ skip_existing=True,
+ progress=bulk.ProgressReporter(enabled=False),
+ )
+
+ assert errors == []
+ assert skipped == []
+ assert details["tarball_downloaded"] is True
+ assert [item.paper.arxiv_id for item in downloads] == [
+ "2501.00001v1",
+ "2501.00002v1",
+ ]
+ assert [item.path.read_bytes() for item in downloads] == [b"%PDF first", b"%PDF second"]
+
+ cached, errors, _, details = bulk.stage_s3_papers(
+ s3_uri=str(tarball),
+ s3_cache_dir=tmp_path / "s3-cache",
+ pdf_dir=tmp_path / "pdf-cache",
+ aws_cli="aws",
+ request_payer="",
+ start=1,
+ max_results=1,
+ max_pdf_bytes=1024,
+ skip_existing=True,
+ progress=bulk.ProgressReporter(enabled=False),
+ )
+
+ assert errors == []
+ assert details["tarball_downloaded"] is False
+ assert cached[0].paper.arxiv_id == "2501.00002v1"
+ assert cached[0].cached is True
+
+
+def test_download_papers_reuses_pdf_and_remembers_failures(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ pdf_dir = tmp_path / "pdfs"
+ pdf_dir.mkdir()
+ paper = _paper()
+ cached_path = pdf_dir / bulk.pdf_filename(paper)
+ cached_path.write_bytes(b"%PDF cached")
+
+ def fail_if_called(*_args, **_kwargs):
+ raise AssertionError("cached PDF should not be downloaded")
+
+ monkeypatch.setattr(bulk, "download_pdf", fail_if_called)
+ downloads, errors, skipped = bulk.download_papers(
+ [paper],
+ pdf_dir=pdf_dir,
+ rate=bulk.RateLimiter(0),
+ user_agent="test",
+ timeout=1,
+ max_pdf_bytes=1024,
+ skip_existing=True,
+ progress=bulk.ProgressReporter(enabled=False),
+ )
+
+ assert errors == []
+ assert skipped == []
+ assert downloads[0].cached is True
+
+ broken = _paper("2501.09999v1")
+ failure_path = pdf_dir / "_failed_downloads.json"
+
+ def fail_download(*_args, **_kwargs):
+ raise RuntimeError("missing PDF")
+
+ monkeypatch.setattr(bulk, "download_pdf", fail_download)
+ _, errors, _ = bulk.download_papers(
+ [broken],
+ pdf_dir=pdf_dir,
+ rate=bulk.RateLimiter(0),
+ user_agent="test",
+ timeout=1,
+ max_pdf_bytes=1024,
+ skip_existing=True,
+ failure_cache={},
+ failure_cache_path=failure_path,
+ progress=bulk.ProgressReporter(enabled=False),
+ )
+ failure_cache = bulk.load_failed_download_cache(failure_path)
+ _, second_errors, skipped = bulk.download_papers(
+ [broken],
+ pdf_dir=pdf_dir,
+ rate=bulk.RateLimiter(0),
+ user_agent="test",
+ timeout=1,
+ max_pdf_bytes=1024,
+ skip_existing=True,
+ failure_cache=failure_cache,
+ failure_cache_path=failure_path,
+ progress=bulk.ProgressReporter(enabled=False),
+ )
+
+ assert errors[0]["arxiv_id"] == broken.arxiv_id
+ assert second_errors == []
+ assert skipped[0]["arxiv_id"] == broken.arxiv_id
+
+
+def test_bench_arxiv_download_only_runs_from_local_tarball(
+ tmp_path: Path,
+ capsys: pytest.CaptureFixture[str],
+) -> None:
+ source = tmp_path / "paper.pdf"
+ source.write_bytes(b"%PDF local")
+ tarball = tmp_path / "arxiv.tar"
+ with tarfile.open(tarball, "w") as archive:
+ archive.add(source, arcname="pdf/2501.01234v1.pdf")
+ runs_dir = tmp_path / "runs"
+ pdf_dir = tmp_path / "pdfs"
+
+ exit_code = bulk.main(
+ [
+ "bench",
+ "arxiv",
+ "--download-only",
+ "--no-progress",
+ "--s3-uri",
+ str(tarball),
+ "--s3-request-payer",
+ "",
+ "--runs-dir",
+ str(runs_dir),
+ "--pdf-cache-dir",
+ str(pdf_dir),
+ "--run-id",
+ "local-run",
+ "--max-results",
+ "1",
+ ]
+ )
+ summary = json.loads(capsys.readouterr().out)
+
+ assert exit_code == 0
+ assert summary["state"] == "completed"
+ assert summary["ready_papers"] == 1
+ assert summary["downloaded_papers"] == 1
+ assert (pdf_dir / "2501.01234v1.pdf").read_bytes() == b"%PDF local"
+ assert json.loads((runs_dir / "local-run" / "summary.json").read_text()) == summary
+
+ assert bulk.main(["bench", "arxiv", "list", "--runs-dir", str(runs_dir)]) == 0
+ listed = json.loads(capsys.readouterr().out)
+ assert listed["runs"][0]["run_id"] == "local-run"
+ assert listed["runs"][0]["state"] == "completed"
+
+
+async def test_arxiv_benchmark_hands_downloads_to_bulk_uploader(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+ capsys: pytest.CaptureFixture[str],
+) -> None:
+ pdf = tmp_path / "paper.pdf"
+ pdf.write_bytes(b"%PDF benchmark")
+ downloaded = bulk.DownloadedPaper(
+ paper=_paper(),
+ path=pdf,
+ size=pdf.stat().st_size,
+ sha256=bulk.sha256_file(pdf),
+ )
+
+ def fake_stage(**_kwargs):
+ return [downloaded], [], [], {"selected_pdf_members": 1}
+
+ class FakeClient:
+ submissions: list[list[str]] = []
+
+ def __init__(self, **_kwargs):
+ pass
+
+ async def __aenter__(self):
+ return self
+
+ async def __aexit__(self, *_args):
+ return None
+
+ async def submit_batch(self, items, **_kwargs):
+ self.submissions.append([item.path.name for item in items])
+ return {"task_id": "task-1", "status": "accepted"}
+
+ async def get_task_status(self, _task_id: str, *, timeout: float) -> dict[str, Any]:
+ return {
+ "status": "completed",
+ "total_files": 1,
+ "processed_files": 1,
+ "successful_files": 1,
+ "failed_files": 0,
+ "files": {},
+ }
+
+ monkeypatch.setattr(bulk, "stage_s3_papers", fake_stage)
+ monkeypatch.setattr(bulk, "OpenRAGApiClient", FakeClient)
+ args = _args(
+ "--api-key",
+ "test-key",
+ "--no-progress",
+ "--runs-dir",
+ str(tmp_path / "runs"),
+ "--run-id",
+ "upload-run",
+ "--poll-interval",
+ "0",
+ "--summary-interval",
+ "0",
+ "--max-results",
+ "1",
+ )
+
+ exit_code = await bulk._arxiv_main(args)
+ summary = json.loads(capsys.readouterr().out)
+
+ assert exit_code == 0
+ assert FakeClient.submissions == [["paper.pdf"]]
+ assert summary["state"] == "completed"
+ assert summary["bulk"]["successful_files"] == 1
+ assert summary["batches"][0]["status"] == "completed"
diff --git a/tests/unit/test_openrag_bulk.py b/tests/unit/test_openrag_bulk.py
new file mode 100644
index 000000000..de4d5fcb6
--- /dev/null
+++ b/tests/unit/test_openrag_bulk.py
@@ -0,0 +1,295 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import httpx
+import pytest
+
+from scripts import openrag_bulk as bulk
+
+
+def test_collect_upload_items_filters_deduplicates_and_sorts(tmp_path: Path) -> None:
+ docs = tmp_path / "docs"
+ nested = docs / "nested"
+ nested.mkdir(parents=True)
+ small = docs / "small.pdf"
+ large = nested / "large.pdf"
+ ignored = nested / "notes.txt"
+ small.write_bytes(b"1")
+ large.write_bytes(b"12345")
+ ignored.write_text("ignore", encoding="utf-8")
+
+ items = bulk.collect_upload_items(
+ [str(docs), str(small)],
+ include=["*.pdf"],
+ exclude=["nested/*"],
+ )
+
+ assert [item.path for item in items] == [small.resolve()]
+
+ all_pdfs = bulk.collect_upload_items([str(docs)], include=["*.pdf"])
+ assert [item.path.name for item in bulk.sort_upload_items(all_pdfs, "size-desc")] == [
+ "large.pdf",
+ "small.pdf",
+ ]
+
+
+async def test_api_client_posts_repeated_file_parts_to_openrag_v1(tmp_path: Path) -> None:
+ first = tmp_path / "first.txt"
+ second = tmp_path / "second.txt"
+ first.write_text("first", encoding="utf-8")
+ second.write_text("second", encoding="utf-8")
+ captured: dict[str, Any] = {}
+
+ async def handler(request: httpx.Request) -> httpx.Response:
+ body = await request.aread()
+ captured.update(
+ url=str(request.url),
+ api_key=request.headers.get("X-API-Key"),
+ body=body,
+ )
+ return httpx.Response(202, json={"task_id": "task-1", "file_count": 2})
+
+ client = bulk.OpenRAGApiClient(
+ base_url="http://openrag.test",
+ api_key="secret-key",
+ transport=httpx.MockTransport(handler),
+ )
+ try:
+ response = await client.submit_batch(
+ [bulk.BulkUploadItem(first), bulk.BulkUploadItem(second)],
+ settings_json='{"chunk_size":1000}',
+ tweaks_json=None,
+ replace_duplicates=True,
+ timeout=5,
+ )
+ finally:
+ await client.close()
+
+ assert response["task_id"] == "task-1"
+ assert captured["url"] == "http://openrag.test/api/v1/documents/ingest"
+ assert captured["api_key"] == "secret-key"
+ assert captured["body"].count(b'name="file"') == 2
+ assert b'name="settings"' in captured["body"]
+ assert b'{"chunk_size":1000}' in captured["body"]
+ assert b'name="replace_duplicates"' in captured["body"]
+
+
+async def test_upload_bulk_items_batches_polls_and_aggregates(tmp_path: Path) -> None:
+ paths = []
+ for index in range(5):
+ path = tmp_path / f"doc-{index}.txt"
+ path.write_text(f"document {index}", encoding="utf-8")
+ paths.append(path)
+
+ class FakeClient:
+ def __init__(self) -> None:
+ self.submissions: list[dict[str, Any]] = []
+ self.poll_counts: dict[str, int] = {}
+
+ async def submit_batch(self, items, **kwargs):
+ task_id = f"task-{len(self.submissions) + 1}"
+ self.submissions.append(
+ {"task_id": task_id, "files": [item.path.name for item in items], **kwargs}
+ )
+ return {"task_id": task_id, "status": "accepted"}
+
+ async def get_task_status(self, task_id: str, *, timeout: float):
+ count = self.poll_counts.get(task_id, 0) + 1
+ self.poll_counts[task_id] = count
+ file_count = len(self.submissions[int(task_id.removeprefix("task-")) - 1]["files"])
+ if count == 1:
+ return {
+ "task_id": task_id,
+ "status": "running",
+ "total_files": file_count,
+ "processed_files": 0,
+ "successful_files": 0,
+ "failed_files": 0,
+ "files": {},
+ }
+ return {
+ "task_id": task_id,
+ "status": "completed",
+ "total_files": file_count,
+ "processed_files": file_count,
+ "successful_files": file_count,
+ "failed_files": 0,
+ "duration_seconds": 0.01,
+ "files": {},
+ }
+
+ fake = FakeClient()
+ persisted: list[dict[str, Any]] = []
+ summary = await bulk.upload_bulk_items(
+ client=fake,
+ items=[bulk.BulkUploadItem(path) for path in paths],
+ options=bulk.BulkUploadOptions(
+ batch_size=2,
+ max_inflight=2,
+ max_submit=1,
+ poll_interval=0,
+ progress_interval=60,
+ summary_interval=0,
+ settings_json='{"chunk_size":1000}',
+ ),
+ progress=bulk.ProgressReporter(enabled=False),
+ summary_callback=lambda value: persisted.append(json.loads(json.dumps(value))),
+ )
+
+ assert [submission["files"] for submission in fake.submissions] == [
+ ["doc-0.txt", "doc-1.txt"],
+ ["doc-2.txt", "doc-3.txt"],
+ ["doc-4.txt"],
+ ]
+ assert all(
+ submission["settings_json"] == '{"chunk_size":1000}' for submission in fake.submissions
+ )
+ assert [batch["status"] for batch in summary["batches"]] == [
+ "completed",
+ "completed",
+ "completed",
+ ]
+ assert summary["successful_files"] == 5
+ assert summary["failed_files"] == 0
+ assert summary["completed_batches"] == 3
+ assert summary["failed_batches"] == 0
+ assert not bulk.bulk_failed(summary)
+ assert persisted[0]["batches"][0]["status"] == "queued"
+
+
+async def test_upload_bulk_items_records_batch_failure_and_continues(tmp_path: Path) -> None:
+ first = tmp_path / "first.txt"
+ second = tmp_path / "second.txt"
+ first.write_text("first", encoding="utf-8")
+ second.write_text("second", encoding="utf-8")
+
+ class FakeClient:
+ calls = 0
+
+ async def submit_batch(self, _items, **_kwargs):
+ self.calls += 1
+ if self.calls == 1:
+ raise bulk.ApiError("payload too large", 413)
+ return {"task_id": "task-ok", "status": "accepted"}
+
+ async def get_task_status(self, _task_id: str, *, timeout: float):
+ return {
+ "status": "completed",
+ "total_files": 1,
+ "processed_files": 1,
+ "successful_files": 1,
+ "failed_files": 0,
+ "files": {},
+ }
+
+ summary = await bulk.upload_bulk_items(
+ client=FakeClient(),
+ items=[bulk.BulkUploadItem(first), bulk.BulkUploadItem(second)],
+ options=bulk.BulkUploadOptions(
+ batch_size=1,
+ max_inflight=1,
+ poll_interval=0,
+ summary_interval=0,
+ ),
+ progress=bulk.ProgressReporter(enabled=False),
+ )
+
+ assert [batch["status"] for batch in summary["batches"]] == ["failed", "completed"]
+ assert "payload too large" in summary["batches"][0]["error"]
+ assert summary["failed_batches"] == 1
+ assert bulk.bulk_failed(summary)
+
+
+def test_persist_list_and_load_summary(tmp_path: Path) -> None:
+ runs_dir = tmp_path / "runs"
+ output_dir = runs_dir / "run-1"
+ summary = {
+ "total_files": 1,
+ "batches": [{"batch_index": 1, "status": "completed", "successful_files": 1}],
+ }
+
+ saved = bulk.persist_summary(
+ summary,
+ output_dir=output_dir,
+ run_id="run-1",
+ started=0,
+ )
+ listed = bulk.list_runs(runs_dir)
+ compact = bulk.load_summary_reference(runs_dir, "latest", detail=False)
+ detailed = bulk.load_summary_reference(runs_dir, "run-1", detail=True)
+
+ assert json.loads((output_dir / "summary.json").read_text(encoding="utf-8")) == saved
+ assert listed["runs"][0]["run_id"] == "run-1"
+ assert listed["runs"][0]["state"] == "completed"
+ assert "batches" not in compact
+ assert compact["summary"]["batch_status_counts"] == {"completed": 1}
+ assert detailed["batches"][0]["successful_files"] == 1
+
+
+def test_bulk_upload_command_loads_a_directory(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+ capsys: pytest.CaptureFixture[str],
+) -> None:
+ docs = tmp_path / "docs"
+ docs.mkdir()
+ (docs / "one.txt").write_text("one", encoding="utf-8")
+ (docs / "two.txt").write_text("two", encoding="utf-8")
+
+ class FakeClient:
+ submissions: list[list[str]] = []
+
+ def __init__(self, **_kwargs):
+ pass
+
+ async def __aenter__(self):
+ return self
+
+ async def __aexit__(self, *_args):
+ return None
+
+ async def submit_batch(self, items, **_kwargs):
+ self.submissions.append([item.path.name for item in items])
+ return {"task_id": "task-1", "status": "accepted"}
+
+ async def get_task_status(self, _task_id: str, *, timeout: float):
+ return {
+ "status": "completed",
+ "total_files": 2,
+ "processed_files": 2,
+ "successful_files": 2,
+ "failed_files": 0,
+ "files": {},
+ }
+
+ monkeypatch.setattr(bulk, "OpenRAGApiClient", FakeClient)
+ exit_code = bulk.main(
+ [
+ "bulk",
+ "upload",
+ "--api-key",
+ "test-key",
+ "--no-progress",
+ "--runs-dir",
+ str(tmp_path / "runs"),
+ "--run-id",
+ "directory-run",
+ "--poll-interval",
+ "0",
+ "--summary-interval",
+ "0",
+ str(docs),
+ ]
+ )
+ summary = json.loads(capsys.readouterr().out)
+
+ assert exit_code == 0
+ assert FakeClient.submissions == [["one.txt", "two.txt"]]
+ assert summary["total_files"] == 2
+ assert summary["successful_files"] == 2