Skip to content

https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/

feat(train): synthetic data generation pipeline (#28)

Yichuan Wang28d53dd40a35
diff --git a/assets/29058_1780726525.mp4 b/assets/29058_1780726525.mp4
new file mode 100644
index 0000000..7dd56ad
Binary files /dev/null and b/assets/29058_1780726525.mp4 differ
diff --git a/train/docs/synthetic_data_pipeline.md b/train/docs/synthetic_data_pipeline.md
new file mode 100644
index 0000000..0b44258
--- /dev/null
+++ b/train/docs/synthetic_data_pipeline.md
@@ -0,0 +1,503 @@
+# Synthetic Data Generation Pipeline
+
+End-to-end pipeline for generating the `screenshot-training-natural-filtered-v2`
+training dataset used to fine-tune `Qwen3-VL-Embedding-2B` for visual document
+retrieval.
+
+The pipeline produces ~115K high-quality query→screenshot-chunk pairs with hard
+negatives, starting from raw Wikipedia screenshot tiles.
+
+---
+
+## Pipeline Overview
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│ VISUAL QUERY PIPELINE │
+│ │
+│ Wikipedia pages (kiwix_tiles/) │
+│ │ │
+│ ▼ │
+│ ① generate_query_pairs.py Gemini reads screenshot chunks, │
+│ │ generates Q/A pairs │
+│ ▼ │
+│ ② filter_self_contained.py GPT-4o removes non-self- │
+│ │ contained queries │
+│ ▼ │
+│ ③ mine_hard_negatives.py Search API retrieves confusable │
+│ │ chunks as hard negatives │
+│ ▼ │
+│ ④ filter_hard_negatives_vqa.py VLM removes false negatives │
+│ │ (chunks that actually answer │
+│ │ the query) │
+│ ▼ │
+│ ⑤ clean_queries_simpleqa_style.py Score naturalness + SimpleQA │
+│ │ style fit via Gemini │
+│ ▼ │
+│ ⑥ export_natural_filtered_v2.py Keep only naturalness≥4 & │
+│ │ style_fit≥4 │
+│ ▼ │
+│ ⑦ split_first5_chunks.py 90/5/5 train/eval/test split │
+│ │ │
+│ ▼ │
+│ ⑧ prepare_hf_dataset.py Package for Hugging Face │
+│ package_hf_image_shards.py │
+│ upload_hf_dataset.py │
+└─────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────┐
+│ TEXT WARMUP PIPELINE │
+│ │
+│ Text passage DB (text_baseline.db) │
+│ │ │
+│ ▼ │
+│ ① generate_text_query_pairs.py Gemini reads text passages, │
+│ │ generates Q/A pairs │
+│ ▼ │
+│ ② filter_self_contained.py Same filter as visual pipeline │
+│ │ │
+│ ▼ │
+│ ③ mine_text_hard_negatives.py Text search API retrieves │
+│ │ confusable passages │
+│ ▼ │
+│ ④ filter_text_hard_negatives_llm.py LLM removes false negatives │
+│ │
+│ Output: text-qa-pair dataset (used for --text-warmup-steps) │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Prerequisites
+
+### Data sources
+
+| Resource | Purpose | Approx. size |
+|----------|---------|-------------|
+| `kiwix_tiles/` directory | Wikipedia screenshot tiles with `index.jsonl` | ~500 GB |
+| `text_baseline.db` (SQLite) | Wikipedia text passages for text warmup | ~20 GB |
+| Search API (`:30888`) | Hard negative mining (retrieval over the tile index) | running service |
+
+### API keys
+
+| Key | Used by | Purpose |
+|-----|---------|---------|
+| Google Cloud ADC | `generate_query_pairs.py`, `generate_text_query_pairs.py`, `clean_queries_simpleqa_style.py` | Gemini models via Vertex AI |
+| `OPENAI_API_KEY` | `filter_self_contained.py`, `filter_hard_negatives_vqa.py` | GPT-4o for query filtering and VQA false-negative removal |
+
+```bash
+# Google Cloud (one-time setup)
+gcloud auth application-default login
+
+# OpenAI
+export OPENAI_API_KEY="sk-proj-..."
+export OPENAI_BASE_URL="https://us.api.openai.com/v1" # if your key requires this
+```
+
+---
+
+## Step-by-Step Reproduction
+
+### Step 1: Generate Visual Query Pairs
+
+Send screenshot chunks to Gemini, which generates factual Q/A pairs.
+
+**Script:** `generate_query_pairs.py`
+
+**What it does:**
+1. Loads `index.jsonl` from the kiwix_tiles directory
+2. Filters out infrastructure pages (disambiguation, elections, lists, etc.)
+3. For each eligible page, picks a random chunk from the first 70% of chunks
+4. Sends the chunk image to Gemini with a detailed prompt
+5. Gemini generates a Q/A pair with source_type and subject labels
+6. Post-generation filters reject layout-bound or unnatural questions
+
+**Single batch:**
+```bash
+uv run python generate_query_pairs.py \
+ --tiles-dir /opt/dlami/nvme/kiwix_tiles \
+ --num-pages 2000 \
+ --model gemini-3.1-flash-lite-preview \
+ --output batches/batch_000.jsonl \
+ --seed 0
+```
+
+**Full-scale generation (120 non-overlapping batches):**
+```bash
+bash run_generate_batches.sh /opt/dlami/nvme/kiwix_tiles 0 119 gemini-3.1-flash-lite-preview
+```
+
+This produces ~195K raw pairs across 120 batch files.
+
+**Output format:**
+```json
+{
+ "query": "In what year did Henryk Skarżyński perform the first cochlear implantation in Poland?",
+ "answer": "1992",
+ "source_sentence": "He performed the first operation of cochlear implantation in Poland...",
+ "source_type": "prose",
+ "subject": "medicine",
+ "chunk_path": "shard_400/shard_00010/3316848.png.tiles/chunk_0000_00.png",
+ "url": "https://en.wikipedia.org/wiki/Henryk_Skarżyński",
+ "title": "Henryk Skarżyński",
+ "chunk_index": 0,
+ "tiles_dir": "shard_400/shard_00010/3316848.png.tiles"
+}
+```
+
+**Cost estimate:** ~$100–200 for 195K pairs with Flash Lite, ~$2000 with Pro.
+
+**Merge all batches:**
+```bash
+cat batches/batch_*.jsonl > raw_query_pairs.jsonl
+wc -l raw_query_pairs.jsonl # expect ~195K
+```
+
+### Step 2: Filter Self-Contained Queries
+
+Remove queries that reference the page layout ("in the table") or use vague
+references ("the team", "the film") without naming the entity.
+
+**Script:** `filter_self_contained.py`
+
+```bash
+OPENAI_API_KEY=sk-... uv run python filter_self_contained.py \
+ --input raw_query_pairs.jsonl \
+ --output filtered_query_pairs.jsonl \
+ --model gpt-4o
+```
+
+**Expected results:**
+- Drop rate: ~15% (29K dropped from 195K)
+- Cost: ~$23 with GPT-4o
+- Runtime: ~8 minutes
+
+**Test run first:**
+```bash
+OPENAI_API_KEY=sk-... uv run python filter_self_contained.py \
+ --input raw_query_pairs.jsonl \
+ --output /dev/null \
+ --model gpt-4o \
+ --test-first 100
+```
+
+### Step 3: Mine Hard Negatives
+
+For each (query, positive_chunk) pair, retrieve top-K candidates from the search
+API. Non-positive results become hard negatives.
+
+**Script:** `mine_hard_negatives.py` (already in this repo)
+
+**Prerequisite:** Search API running at `localhost:30888` (see `serve/` and `deploy/`).
+
+```bash
+uv run python mine_hard_negatives.py \
+ --input filtered_query_pairs.jsonl \
+ --output filtered_query_pairs_hn.jsonl \
+ --num-negatives 7 \
+ --n-docs 50 \
+ --filter-mode margin \
+ --margin 0.95
+```
+
+**Output adds:**
+```json
+{
+ "query": "...",
+ "chunk_path": "...",
+ "neg_chunk_paths": ["/path/to/neg1.png", "/path/to/neg2.png", "..."],
+ "retrieve_top20": [{"rank": 1, "path": "...", "score": 0.61}, "..."],
+ "positive_score": 0.65,
+ "positive_rank": 1
+}
+```
+
+### Step 4: Filter Hard Negatives (VQA-based)
+
+Remove false negatives — mined "negatives" that actually do answer the query.
+A VLM answers the query from each candidate image; if it gets the answer right,
+that candidate is NOT a true hard negative and is removed.
+
+**Script:** `filter_hard_negatives_vqa.py` / `run_filter_hard_negatives_chunks.py`
+
+For large files, use the chunked runner for resumability:
+```bash
+uv run python run_filter_hard_negatives_chunks.py \
+ --input-jsonl filtered_query_pairs_hn.jsonl \
+ --output-dir training/data/filtered_hn_chunks \
+ --chunk-size 10000 \
+ --skip-existing
+```
+
+Each chunk produces:
+- `filtered_hn.jsonl` — the cleaned hard negatives
+- `candidate_reviews.jsonl` — per-candidate VLM judgments
+- `summary.json` — statistics
+
+### Step 5: Clean Queries (Naturalness Scoring)
+
+Score each query on naturalness (1–5) and SimpleQA style fit (1–5) using Gemini.
+This step does NOT look at images — only the query text is judged.
+
+**Script:** `clean_queries_simpleqa_style.py`
+
+```bash
+uv run python clean_queries_simpleqa_style.py \
+ --model gemini-2.0-flash-001 \
+ --target-count 50000 \
+ --batch-size 20 \
+ --concurrency 8 \
+ --dedupe-query \
+ --output training/data/cleaned/simpleqa_style_cleaned.jsonl \
+ --reviews-output training/data/cleaned/simpleqa_style_cleaned.reviews.jsonl \
+ --summary-output training/data/cleaned/simpleqa_style_cleaned.summary.json
+```
+
+### Step 6: Export Strict Retained Rows
+
+Keep only rows with `naturalness >= 4` and `simpleqa_style_fit >= 4`.
+
+**Script:** `export_natural_filtered_v2.py`
+
+```bash
+uv run python export_natural_filtered_v2.py \
+ --reviews training/data/cleaned/simpleqa_style_cleaned.reviews.jsonl \
+ --output-dir training/data/natural_filtered_v2 \
+ --output-name filtered_hn.jsonl \
+ --min-naturalness 4 \
+ --min-style-fit 4
+```
+
+**Expected:** ~77% keep rate → ~115K rows from 150K input.
+
+### Step 7: Split into Train/Eval/Test
+
+**Script:** `split_first5_chunks.py`
+
+```bash
+uv run python split_first5_chunks.py \
+ --inputs training/data/natural_filtered_v2/filtered_hn.jsonl \
+ --output-dir training/data/natural_filtered_v2/split \
+ --seed 42 \
+ --train-ratio 0.9 \
+ --eval-ratio 0.05 \
+ --test-ratio 0.05
+```
+
+Output:
+- `split/train_hn.jsonl` (~104K)
+- `split/eval_hn.jsonl` (~5.8K)
+- `split/test_hn.jsonl` (~5.8K)
+
+### Step 8: Package and Upload to Hugging Face
+
+```bash
+# Materialize images (hardlinks from kiwix_tiles)
+uv run python prepare_hf_dataset.py \
+ --split-dir training/data/natural_filtered_v2/split \
+ --image-root /opt/dlami/nvme/kiwix_tiles \
+ --output-dir /opt/dlami/nvme/hf_export/screenshot-training-natural-filtered-v2 \
+ --repo-id Chrisyichuan/screenshot-training-natural-filtered-v2
+
+# Create tar shards (1000 shards, ~93.5 GB total)
+uv run python package_hf_image_shards.py \
+ --source-dir /opt/dlami/nvme/hf_export/screenshot-training-natural-filtered-v2 \
+ --output-dir /opt/dlami/nvme/hf_export_sharded/screenshot-training-natural-filtered-v2 \
+ --overwrite
+
+# Upload
+uv run python upload_hf_dataset.py \
+ --repo-id Chrisyichuan/screenshot-training-natural-filtered-v2 \
+ --local-dir /opt/dlami/nvme/hf_export_sharded/screenshot-training-natural-filtered-v2 \
+ --repo-type dataset \
+ --skip-create
+```
+
+---
+
+## Text Warmup Data Pipeline
+
+The text warmup pipeline generates `text-qa-pair` data used for the
+`--text-warmup-steps` phase of training. It follows a parallel track.
+
+### Generate Text Query Pairs
+
+**Script:** `generate_text_query_pairs.py`
+
+```bash
+uv run python generate_text_query_pairs.py \
+ --db-path /opt/dlami/nvme/text_embeddings_1024/text_baseline.db \
+ --num-articles 52000 \
+ --model gemini-3.1-flash-lite-preview \
+ --output text_query_pairs.jsonl \
+ --min-article-chunks 6 \
+ --min-paragraph-words 60
+```
+
+The script:
+1. Queries the SQLite DB for articles with ≥6 chunks
+2. For each article, selects a chunk with a long natural prose paragraph
+3. Sends the passage to Gemini with 5-shot examples
+4. Filters for natural, self-contained prose questions
+5. Verifies the supporting span exists verbatim in the passage
+
+**Expected yield:** ~40% (21K pairs from 52K articles).
+**Cost:** ~$110 with Flash Lite.
+
+### Filter + Mine + Package
+
+```bash
+# Filter self-contained (same script as visual pipeline)
+OPENAI_API_KEY=sk-... uv run python filter_self_contained.py \
+ --input text_query_pairs.jsonl \
+ --output text_query_pairs_filtered.jsonl \
+ --model gpt-4o
+
+# Mine text hard negatives (requires text search API on :30889)
+uv run python mine_text_hard_negatives.py \
+ --input text_query_pairs_filtered.jsonl \
+ --output text_query_pairs_hn.jsonl
+
+# Filter text hard negatives
+uv run python run_filter_text_hard_negatives_chunks.py \
+ --input-jsonl text_query_pairs_hn.jsonl \
+ --output-dir training/data/text_filtered_hn_chunks
+```
+
+---
+
+## Script Inventory
+
+### Query Generation (Steps 1–2)
+
+| Script | Input | Output | API |
+|--------|-------|--------|-----|
+| `generate_query_pairs.py` | kiwix_tiles/ + index.jsonl | raw visual Q/A pairs (JSONL) | Gemini (Vertex AI) |
+| `generate_text_query_pairs.py` | text_baseline.db (SQLite) | raw text Q/A pairs (JSONL) | Gemini + OpenAI fallback |
+| `filter_self_contained.py` | any Q/A JSONL | filtered JSONL (non-self-contained removed) | OpenAI (GPT-4o) |
+| `run_generate_batches.sh` | kiwix_tiles/ | batched generation wrapper | — |
+
+### Hard Negative Mining (Step 3)
+
+| Script | Input | Output | API |
+|--------|-------|--------|-----|
+| `mine_hard_negatives.py` | filtered Q/A JSONL | JSONL with `neg_chunk_paths` | Search API (:30888) |
+| `mine_text_hard_negatives.py` | filtered text Q/A JSONL | JSONL with `neg_passages` | Text search API (:30889) |
+
+### Hard Negative Filtering (Step 4)
+
+| Script | Input | Output | API |
+|--------|-------|--------|-----|
+| `filter_hard_negatives_vqa.py` | HN JSONL | cleaned HN JSONL | OpenAI/Gemini (VLM) |
+| `run_filter_hard_negatives_chunks.py` | large HN JSONL | chunked output dir | — |
+| `filter_text_hard_negatives_llm.py` | text HN JSONL | cleaned text HN JSONL | OpenAI/Gemini (LLM) |
+| `run_filter_text_hard_negatives_chunks.py` | large text HN JSONL | chunked output dir | — |
+
+### Query Cleaning & Export (Steps 5–7)
+
+| Script | Input | Output | API |
+|--------|-------|--------|-----|
+| `clean_queries_simpleqa_style.py` | filtered HN JSONL | reviews JSONL with scores | Gemini |
+| `export_natural_filtered_v2.py` | reviews JSONL | strict-filtered JSONL | — |
+| `split_first5_chunks.py` | filtered JSONL | train/eval/test split | — |
+
+### Dataset Packaging (Step 8)
+
+| Script | Input | Output | API |
+|--------|-------|--------|-----|
+| `prepare_hf_dataset.py` | split dir + image root | HF dataset folder | — |
+| `package_hf_image_shards.py` | HF dataset folder | tar-sharded images | — |
+| `upload_hf_dataset.py` | sharded dataset | HF Hub upload | HF API |
+| `extract_hf_image_shards.py` | downloaded shards | extracted images/ | — |
+
+---
+
+## Cost Estimates (Full Pipeline)
+
+| Step | Model | Volume | Est. Cost |
+|------|-------|--------|-----------|
+| Generate visual queries | gemini-3.1-flash-lite | 195K pairs | ~$150 |
+| Filter self-contained | gpt-4o | 195K queries | ~$23 |
+| Generate text queries | gemini-3.1-flash-lite + gpt-4o-mini fallback | 52K articles | ~$110 |
+| Filter text self-contained | gpt-4o | 21K queries | ~$3 |
+| Filter HN (VQA) | gpt-4o / gemini | 150K × 7 candidates | ~$200 |
+| Clean queries (naturalness) | gemini-2.0-flash | 150K queries | ~$15 |
+| **Total** | | | **~$500** |
+
+---
+
+## Data Formats
+
+### Raw Q/A pair (after Step 1)
+```json
+{
+ "query": "What is the population of Tokyo?",
+ "answer": "13.96 million",
+ "source_sentence": "Tokyo has a population of 13.96 million...",
+ "source_type": "prose",
+ "subject": "geography",
+ "chunk_path": "shard_400/shard_00010/350170.png.tiles/chunk_0000_00.png",
+ "url": "https://en.wikipedia.org/wiki/Tokyo",
+ "title": "Tokyo",
+ "chunk_index": 0,
+ "tiles_dir": "shard_400/shard_00010/350170.png.tiles"
+}
+```
+
+### With hard negatives (after Step 3)
+```json
+{
+ "query": "...",
+ "chunk_path": "...",
+ "neg_chunk_paths": ["/path/to/neg1.png", "/path/to/neg2.png"],
+ "retrieve_top20": [{"rank": 1, "path": "...", "score": 0.61}],
+ "positive_score": 0.65,
+ "positive_rank": 1
+}
+```
+
+### Published HF format (after Step 8)
+```json
+{
+ "query": "...",
+ "chunk_path": "images/shard_000/...",
+ "neg_chunk_paths": ["images/shard_001/...", "images/shard_002/..."],
+ "source_positive_rank": 1,
+ "source_positive_score": 0.63
+}
+```
+
+---
+
+## Quality Control
+
+### Page-level filtering (Step 1)
+- Skip infrastructure pages (disambiguation, category, template, portal)
+- Skip low-quality content (elections, census, track listings, episode lists)
+- Require page_height ≥ 3000px, ≥1 tile, complete capture
+
+### Query-level filtering
+
+| Filter | What it catches | Drop rate |
+|--------|----------------|-----------|
+| `is_natural_question()` (Step 1) | Layout references, dangling entities, truncated answers | ~30% of raw Gemini output |
+| `filter_self_contained.py` (Step 2) | Unnamed subjects, document-structure references | ~15% |
+| `clean_queries_simpleqa_style.py` (Step 5) | Templatic phrasing, poor naturalness | ~23% |
+
+### Hard negative quality (Step 4)
+The VQA filter ensures hard negatives are truly *confusable but wrong*:
+1. VLM tries to answer the query from the candidate image
+2. A judge grades the answer as CORRECT / WRONG / CANNOT_ANSWER
+3. Only WRONG or CANNOT_ANSWER candidates are kept as hard negatives
+4. If the positive image itself fails the VQA check, the entire row is skipped
+
+---
+
+## Differences from Published Dataset
+
+The published `screenshot-training-natural-filtered-v2` was generated with:
+- **Visual queries:** `gemini-3.1-flash-lite-preview` (120 batches, 195K raw → 165K filtered)
+- **Self-contained filter:** `gpt-4o` (15% drop rate, zero false keeps)
+- **Naturalness cleaning:** `gemini-2.0-flash-001`
+- **Naturalness thresholds:** `naturalness ≥ 4`, `simpleqa_style_fit ≥ 4`
+- **Final size:** 115,593 rows → 104K train / 5.8K eval / 5.8K test
diff --git a/train/filter_self_contained.py b/train/filter_self_contained.py
new file mode 100644
index 0000000..fb6b6b6
--- /dev/null
+++ b/train/filter_self_contained.py
@@ -0,0 +1,229 @@
+#!/usr/bin/env python3
+"""
+Post-filter: use GPT-4o (or GPT-4o-mini) to classify whether generated queries
+are self-contained — BATCH mode (50 queries per API call).
+
+A query is self-contained if it can be understood without knowing which Wikipedia
+page or screenshot it came from. Drops queries with vague references ("the team"),
+document-layout references ("in the table"), or missing context (no year for a
+role that has had many holders).
+
+Usage:
+ OPENAI_API_KEY=sk-... python filter_self_contained.py \
+ --input raw_query_pairs.jsonl \
+ --output filtered_query_pairs.jsonl
+
+ # Test first 100:
+ OPENAI_API_KEY=sk-... python filter_self_contained.py \
+ --input raw_query_pairs.jsonl \
+ --output out.jsonl \
+ --test-first 100
+
+Cost estimate (gpt-4o, batch=50):
+ 72K records → ~1.4K API calls → ~$8 total, ~8 min
+
+Ported from Vis-RAG/agent/scripts/contrastive/filter_self_contained.py
+"""
+
+import json
+import os
+import re
+import asyncio
+import argparse
+import time
+
+import openai
+
+MODEL_PRICING = {
+ "gpt-4o-mini": (0.15 / 1e6, 0.60 / 1e6),
+ "gpt-4o": (2.50 / 1e6, 10.0 / 1e6),
+ "gpt-4.1": (2.00 / 1e6, 8.00 / 1e6),
+}
+
+BATCH_SIZE = 50
+MAX_CONCURRENT = 50
+
+FILTER_PROMPT = """\
+For each numbered question, answer YES (self-contained) or NO (not self-contained).
+
+A question is NOT self-contained (NO) if it requires knowing a specific Wikipedia page, table, or screenshot to understand WHAT is being asked. Specifically answer NO when:
+
+1. The subject is a vague pronoun or generic noun without a proper name:
+ NO: "What was the final score of the basketball game between THE TEAM and Marquette?" ("the team" unnamed)
+ NO: "Who directed the episode of THE TELEVISION SERIES titled 'X'?" ("the television series" unnamed)
+ NO: "In what year did THE SUBJECT OF THE ARTICLE move to Tokyo?" ("the subject" unnamed)
+ NO: "What is the running time of THE FILM DESCRIBED IN THE TEXT?" (layout reference)
+
+2. The question explicitly references document structure:
+ NO: "Which item IS LISTED IN THE TABLE as X?"
+ NO: "What is shown IN THE INFOBOX?"
+ NO: "According to THE PROVIDED TABLE, which..."
+
+3. A role/position question where no year or identifying event is given and the role has had many holders:
+ NO: "Who was THE CAPTAIN of HMS Defence?" (no year, hundreds of captains over centuries)
+
+4. A geographic entity refers only to a category without naming which one:
+ NO: "On what date did THE GOODS YARD at the London and North Eastern Railway station close?" (LNER had hundreds of stations — which one?)
+
+Answer YES if all the key entities (people, places, works, teams, events) are explicitly named, even if the names are obscure. Proper names are always fine.
+ YES: "Who did Sandefjord Fotball hire as manager after firing Arne Sandstø?"
+ YES: "How many consonants does the Pesisir language have?"
+ YES: "What 'fresh' rating did the film Our Man in Havana receive on Rotten Tomatoes?"
+ YES: "In what city were the 2025 Special Olympics World Winter Games held?"
+ YES: "Who did Émile Derlin Zinsou serve as assistant to in 1945?"
+
+Output exactly one line per question, using the question number: "1: YES" or "1: NO"
+
+Questions:
+{questions}"""
+
+
+def parse_batch_response(text: str, n: int) -> list[bool]:
+ results = [True] * n
+ if not text:
+ return results
+ for line in text.strip().splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ m = re.match(r"^(\d+)[:.)\s]+(?:N:\s*)?(YES|NO)", line, re.IGNORECASE)
+ if m:
+ idx = int(m.group(1)) - 1
+ verdict = m.group(2).upper()
+ if 0 <= idx < n:
+ results[idx] = verdict == "YES"
+ return results
+
+
+async def classify_batch(
+ client: openai.AsyncOpenAI,
+ records: list[dict],
+ semaphore: asyncio.Semaphore,
+ token_counter: dict,
+ model: str,
+) -> list[tuple[dict, bool]]:
+ queries = [r["query"] for r in records]
+ numbered = "\n".join(f"{i + 1}. {q}" for i, q in enumerate(queries))
+ prompt = FILTER_PROMPT.format(questions=numbered)
+
+ async with semaphore:
+ for attempt in range(6):
+ try:
+ resp = await client.chat.completions.create(
+ model=model,
+ messages=[{"role": "user", "content": prompt}],
+ temperature=0.0,
+ max_tokens=BATCH_SIZE * 8,
+ )
+ text = resp.choices[0].message.content or ""
+ token_counter["input"] += resp.usage.prompt_tokens
+ token_counter["output"] += resp.usage.completion_tokens
+ token_counter["calls"] += 1
+
+ verdicts = parse_batch_response(text, len(records))
+ return list(zip(records, verdicts))
+
+ except openai.RateLimitError:
+ wait = 20 + attempt * 10
+ print(f" Rate limited, waiting {wait}s...")
+ await asyncio.sleep(wait)
+ except Exception as e:
+ print(f" Error (attempt {attempt + 1}): {str(e)[:100]}")
+ await asyncio.sleep(3)
+
+ return [(r, True) for r in records]
+
+
+async def main():
+ parser = argparse.ArgumentParser(
+ description="Filter non-self-contained queries using LLM classification"
+ )
+ parser.add_argument(
+ "--input", required=True, help="Input JSONL with generated query pairs"
+ )
+ parser.add_argument(
+ "--output", required=True, help="Output JSONL with only self-contained queries"
+ )
+ parser.add_argument(
+ "--test-first",
+ type=int,
+ default=None,
+ help="Only process first N records (dry run)",
+ )
+ parser.add_argument(
+ "--model", default="gpt-4o", help="OpenAI model for classification"
+ )
+ parser.add_argument("--batch-size", type=int, default=BATCH_SIZE)
+ parser.add_argument("--concurrency", type=int, default=MAX_CONCURRENT)
+ args = parser.parse_args()
+
+ data = []
+ with open(args.input) as f:
+ for line in f:
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ data.append(json.loads(line))
+ except Exception:
+ pass
+
+ if args.test_first:
+ data = data[: args.test_first]
+
+ batches = [
+ data[i : i + args.batch_size] for i in range(0, len(data), args.batch_size)
+ ]
+
+ print(f"Loaded {len(data)} records → {len(batches)} batches of {args.batch_size}")
+ print(f"Model: {args.model} | Concurrency: {args.concurrency}")
+
+ api_key = os.environ.get("OPENAI_API_KEY")
+ if not api_key:
+ raise ValueError("Set OPENAI_API_KEY environment variable")
+
+ base_url = os.environ.get("OPENAI_BASE_URL", "https://api.openai.com/v1")
+ client = openai.AsyncOpenAI(api_key=api_key, base_url=base_url)
+
+ semaphore = asyncio.Semaphore(args.concurrency)
+ price_in, price_out = MODEL_PRICING.get(args.model, (2.50 / 1e6, 10.0 / 1e6))
+ token_counter = {"input": 0, "output": 0, "calls": 0}
+
+ t0 = time.time()
+ tasks = [
+ classify_batch(client, b, semaphore, token_counter, args.model) for b in batches
+ ]
+ batch_results = await asyncio.gather(*tasks)
+
+ all_results = [item for batch in batch_results for item in batch]
+ kept = [r for r, ok in all_results if ok]
+ dropped = [r for r, ok in all_results if not ok]
+
+ elapsed = time.time() - t0
+ cost = token_counter["input"] * price_in + token_counter["output"] * price_out
+
+ print(f"\n{'=' * 60}")
+ print(f"Total: {len(data)}")
+ print(f"Kept: {len(kept)} ({len(kept) / len(data) * 100:.1f}%)")
+ print(f"Dropped: {len(dropped)} ({len(dropped) / len(data) * 100:.1f}%)")
+ print(
+ f"Calls: {token_counter['calls']} | Time: {elapsed:.1f}s | Cost: ${cost:.4f}"
+ )
+ print(f"{'=' * 60}")
+
+ print(f"\n--- Sample dropped queries ({min(30, len(dropped))}) ---")
+ for r in dropped[:30]:
+ print(f" [{r.get('source_type', '?'):7s}] {r['query']}")
+
+ if args.test_first:
+ print("\n[TEST MODE] Not writing output.")
+ return
+
+ with open(args.output, "w") as f:
+ for r in kept:
+ f.write(json.dumps(r, ensure_ascii=False) + "\n")
+ print(f"\nWrote {len(kept)} records to {args.output}")
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/train/generate_query_pairs.py b/train/generate_query_pairs.py
new file mode 100644
index 0000000..3147da4
--- /dev/null
+++ b/train/generate_query_pairs.py
@@ -0,0 +1,612 @@
+#!/usr/bin/env python3
+"""
+Generate synthetic query-chunk pairs for contrastive learning.
+
+Samples informative Wikipedia pages from kiwix tiles, sends screenshot chunks
+to Gemini to generate factual Q&A pairs with source_type labels.
+
+Output: JSONL with {query, answer, source_type, subject, source_sentence,
+ chunk_path, url, title, chunk_index, tiles_dir}
+
+Prerequisites:
+ - kiwix_tiles directory with Wikipedia screenshot tiles and index.jsonl
+ - Google Cloud ADC (gcloud auth application-default login) for Vertex AI,
+ OR set GOOGLE_API_KEY for direct Gemini API access
+
+Usage:
+ python generate_query_pairs.py \
+ --tiles-dir /path/to/kiwix_tiles \
+ --num-pages 1000 \
+ --output batches/batch_000.jsonl
+
+ # Batched generation (non-overlapping slices):
+ python generate_query_pairs.py \
+ --tiles-dir /path/to/kiwix_tiles \
+ --batch-index 0 --total-batches 100 \
+ --num-pages 2000 \
+ --output batches/batch_000.jsonl
+
+Ported from Vis-RAG/agent/scripts/contrastive/generate_query_pairs.py
+"""
+
+import json
+import os
+import re
+import random
+import base64
+import asyncio
+import argparse
+import time
+from pathlib import Path
+from io import BytesIO
+from collections import Counter
+
+from PIL import Image
+from google import genai
+from google.genai.types import HttpOptions, GenerateContentConfig
+
+# Vertex AI config — requires gcloud ADC (gcloud auth application-default login)
+os.environ.setdefault("GOOGLE_CLOUD_PROJECT", "wise-coyote-478119-h0")
+os.environ.setdefault("GOOGLE_GENAI_USE_VERTEXAI", "true")
+os.environ.setdefault("GOOGLE_CLOUD_LOCATION", "global")
+
+MODEL_PRICING = {
+ "gemini-3.1-pro-preview": (1.25, 10.00),
+ "gemini-2.5-pro-preview-03-25": (1.25, 10.00),
+ "gemini-2.5-pro": (1.25, 10.00),
+ "gemini-3.1-flash-lite-preview": (1.00, 4.00),
+ "gemini-2.0-flash-001": (0.10, 0.40),
+ "gemini-2.0-flash": (0.10, 0.40),
+}
+DEFAULT_MODEL = "gemini-3.1-pro-preview"
+MAX_CONCURRENT = 40
+IMG_JPEG_QUALITY = 85
+
+# ── Page filtering ───────────────────────────────────────────────────
+SKIP_PATTERNS = [
+ r"disambiguation",
+ r"category:",
+ r"template:",
+ r"wikipedia:",
+ r"portal:",
+ r"file:",
+ r"help:",
+ r"talk:",
+ r"module:",
+ r"draft:",
+ r"_deaths$",
+ r"_births$",
+]
+
+SKIP_CONTENT_PATTERNS = [
+ r"\belection\b",
+ r"\belections\b",
+ r"\breferendum\b",
+ r"\bprimary\b",
+ r"\bby-election\b",
+ r"\bcouncil election\b",
+ r"^list of ",
+ r"^lists of ",
+ r"\bdiscography\b",
+ r"\bfilmography\b",
+ r"\btrack listing\b",
+ r"_discography$",
+ r"_filmography$",
+ r"\bseason\b.*\bleague\b",
+ r"\bleague season\b",
+ r"\bfootball league\b",
+ r"\bnba season\b",
+ r"\bnfl season\b",
+ r"\bcensus\b",
+ r"\bdemographic\b",
+ r"^list of .* episodes",
+ r"\bepisodes of\b",
+ r"\bgovernors? of\b",
+ r"\bmayors? of\b",
+ r"\bprime ministers? of\b",
+ r"\bcareer statistics\b",
+ r"\bplayer statistics\b",
+]
+
+SKIP_RE = [re.compile(p, re.IGNORECASE) for p in SKIP_PATTERNS]
+SKIP_CONTENT_RE = [re.compile(p, re.IGNORECASE) for p in SKIP_CONTENT_PATTERNS]
+
+BAD_QUESTION_PATTERNS = [
+ r"\baccording to the\b",
+ r"\baccording to this\b",
+ r"\bvisible\b",
+ r"\btrack listing\b",
+ r"\blisted (here|above|below|in the table)\b",
+ r"\bthe table (shows|lists|above|below)\b",
+ r"\bin the (following|above) table\b",
+ r"\bshown in\b",
+ r"^what is (the )?listed",
+ r"^what (are|is) listed",
+ r"\bpage (shows|lists|includes)\b",
+ r"\bthe film\b(?!\s+[A-Z\"])",
+ r"\bthe song\b(?!\s+[A-Z\"])",
+ r"\bthe album\b(?!\s+[A-Z\"])",
+ r"\bthe book\b(?!\s+[A-Z\"])",
+ r"\bthe team\b(?!\s+[A-Z\"])",
+ r"\bthe show\b(?!\s+[A-Z\"])",
+ r"\bthe series\b(?!\s+[A-Z\"])",
+ r"\bthe station\b(?!\s+[A-Z\"])",
+ r"\bthe school\b(?!\s+[A-Z\"])",
+ r"\bthe match\b(?!\s+[A-Z\"])",
+ r"\bthe game\b(?!\s+[A-Z\"])",
+ r"\bthe competition\b(?!\s+[A-Z\"])",
+ r"\bthe episode\b(?!\s+[A-Z\"\d])",
+ r"\bthe production\b(?!\s+[A-Z\"])",
+ r"\bthe tournament\b(?!\s+[A-Z\d\"])",
+ r"^(when|where|who|what|how|why) (was|is|were|did|does|has|have) (it|they|this|that|he|she)\b",
+]
+BAD_Q_RE = [re.compile(p, re.IGNORECASE) for p in BAD_QUESTION_PATTERNS]
+
+
+def get_page_chunk_count(entry: dict, tiles_root: Path) -> int:
+ cached = entry.get("_chunk_count")
+ if cached is not None:
+ return cached
+
+ tiles_dir = tiles_root / entry["tiles_dir"]
+ chunks_json = tiles_dir / "chunks.json"
+ if not chunks_json.exists():
+ entry["_chunk_count"] = 0
+ return 0
+
+ with open(chunks_json) as f:
+ meta = json.load(f)
+
+ chunk_count = len(meta.get("chunks", []))
+ entry["_chunk_count"] = chunk_count
+ return chunk_count
+
+
+def is_informative_page(entry: dict) -> bool:
+ if entry.get("page_height", 0) < 3000:
+ return False
+ if entry.get("num_tiles", 0) < 1:
+ return False
+ if not entry.get("complete", False):
+ return False
+
+ title_lower = entry["title"].lower()
+ url_lower = entry.get("url", "").lower()
+ check = title_lower + " " + url_lower
+
+ for pat in SKIP_RE:
+ if pat.search(check):
+ return False
+ for pat in SKIP_CONTENT_RE:
+ if pat.search(check):
+ return False
+
+ return True
+
+
+def is_natural_question(qa: dict) -> bool:
+ q = qa.get("query", "")
+ for pat in BAD_Q_RE:
+ if pat.search(q):
+ return False
+ a = qa.get("answer", "")
+ if a and a[-1] in ("→", "…", "–", "/", "(", ","):
+ return False
+ s = qa.get("source_sentence", "") or ""
+ src_type = qa.get("source_type", "prose")
+ if not s:
+ return False
+ if s.rstrip()[-1:] in ("(", ",", "–", "/", "→", "…"):
+ return False
+ if src_type == "prose" and len(s.split()) < 10:
+ return False
+ if src_type in ("infobox", "table") and len(s.split()) < 3:
+ return False
+ return True
+
+
+def load_and_sample_pages(
+ index_path: Path,
+ n: int,
+ batch_index: int = 0,
+ total_batches: int = 1,
+) -> list:
+ MASTER_SEED = 0
+ print(f"Loading index from {index_path}...")
+ candidates: list = []
+
+ with open(index_path) as f:
+ for line in f:
+ entry = json.loads(line)
+ if is_informative_page(entry):
+ candidates.append(entry)
+
+ print(f"Total eligible: {len(candidates):,} pages")
+
+ rng = random.Random(MASTER_SEED)
+ rng.shuffle(candidates)
+
+ slice_size = len(candidates) // total_batches
+ start = batch_index * slice_size
+ end = start + slice_size if batch_index < total_batches - 1 else len(candidates)
+ pool = candidates[start:end]
+
+ print(
+ f"Batch {batch_index}/{total_batches}: pages [{start}:{end}] ({len(pool):,} in pool)"
+ )
+ selected = pool[:n] if n <= len(pool) else pool
+ return selected
+
+
+def filter_selected_pages_by_chunk_count(
+ pages: list[dict],
+ tiles_root: Path,
+ min_page_chunks: int | None = None,
+ max_page_chunks: int | None = None,
+) -> list[dict]:
+ if min_page_chunks is None and max_page_chunks is None:
+ return pages
+
+ kept = []
+ for entry in pages:
+ chunk_count = get_page_chunk_count(entry, tiles_root)
+ if min_page_chunks is not None and chunk_count < min_page_chunks:
+ continue
+ if max_page_chunks is not None and chunk_count > max_page_chunks:
+ continue
+ kept.append(entry)
+ return kept
+
+
+def pick_random_chunk(entry: dict, tiles_root: Path) -> tuple:
+ tiles_dir = tiles_root / entry["tiles_dir"]
+ chunks_json = tiles_dir / "chunks.json"
+
+ if not chunks_json.exists():
+ return None, None
+
+ with open(chunks_json) as f:
+ meta = json.load(f)
+
+ chunks = meta.get("chunks", [])
+ if not chunks:
+ return None, None
+
+ usable = chunks[: max(1, int(len(chunks) * 0.7))]
+ chunk = random.choice(usable)
+ chunk_path = tiles_dir / chunk["file"]
+
+ if not chunk_path.exists():
+ return None, None
+
+ return str(chunk_path), chunk["chunk_index"]
+
+
+def encode_image(path: str) -> str:
+ img = Image.open(path)
+ buf = BytesIO()
+ img.convert("RGB").save(buf, format="JPEG", quality=IMG_JPEG_QUALITY)
+ return base64.b64encode(buf.getvalue()).decode()
+
+
+QUERY_PROMPT = """\
+You are generating a query–evidence pair for training a visual retrieval model over Wikipedia screenshot chunks.
+
+TASK: Given this screenshot chunk, generate ONE factual question whose answer is explicitly and completely visible in this chunk.
+
+━━━ STYLE — write questions like real search queries, not templates ━━━
+Good questions sound like something a curious person would actually search for online.
+Vary the phrasing — use "how much", "in what year", "which", "who", "where", "what caused", "how long", etc.
+
+Here are examples of the STYLE we want (from SimpleQA benchmark):
+ ✓ "How much money, in euros, was the surgeon held responsible for paying in the Olivia Puls case?"
+ ✓ "In what city was the 2010 FIFA World Cup opening ceremony held?"
+ ✓ "How many days did the 1906 San Francisco earthquake fire burn?"
+ ✓ "Which award did Fullmetal Alchemist win at the American Anime Awards in 2007?"
+ ✓ "Who was the first Black female judge appointed to the Cook County Circuit Court?"
+ ✓ "What was the name of the ship that sank during the 1994 Estonia ferry disaster?"
+
+━━━ EVIDENCE SOURCE — be diverse ━━━
+You may draw from ANY visible content: prose text, infobox fields, table cells, image captions, diagrams, or photographs.
+Do not always default to infobox — choose whichever source produces the most natural, interesting question.
+
+━━━ HARD RULES ━━━
+1. SELF-CONTAINED: The question must be fully understandable on its own — no page title, no external context needed.
+ Every entity in the question must be named explicitly.
+
+ ✗ "Who composed the music for the film?" → missing film name
+ ✗ "What is Rideaux's occupation?" → surname only, who is Rideaux?
+ ✗ "On what date was Lerew awarded the DFC?" → surname only + unexplained acronym
+ ✗ "Which medication is listed as a SARM in the provided table?" → depends on "the provided table"
+ ✗ "Who played the actress in the 2013 film Horns?" → "the actress" not identified
+ ✗ "When was the SMN founded?" → unexplained acronym
+ ✗ "Who was the spouse of John Houston?" → too ambiguous, which John Houston?
+ ✗ "Which cyclist placed second in the Tempo race?" → missing event/year context
+ ✗ "What is listed in the infobox?" → references page layout
+ ✗ "Which mission is shown in the screenshot?" → depends on visual layout
+ ✗ "Which former Cleveland Indians player did the Seattle Mariners sign on December 20?" → missing year
+ ✗ "Who was Kesha Rogers' opponent in the general election?" → missing year and race context
+
+ ✓ "Who composed the music for Once Upon a Time in Hong Kong?"
+ ✓ "What is the occupation of Rabbi Shmuel Kamenetsky?"
+ ✓ "In what year did photographer Clarence Rideaux found the agency PicturePerfect?"
+ ✓ "On what date was RAF pilot Arthur Lerew awarded the Distinguished Flying Cross in World War II?"
+ ✓ "Which former Cleveland Indians pitcher did the Seattle Mariners sign on December 20, 2004?"
+
+2. EVIDENCE COMPLETE: The answer must be fully visible in this chunk — not guessed or inferred.
+ The source sentence (S:) must be a complete, untruncated span.
+
+3. DISTINCTIVE: Include enough specifics (names, dates, locations, titles) to distinguish this chunk from similar pages.
+
+━━━ ANSWER ━━━
+Prefer a single concise entity: name, date, place, number, title, or short phrase.
+
+━━━ SKIP if any is true ━━━
+- Raw vote counts, track listings, census tables, or episode lists
+- Answer not fully visible or requires external context
+- Cannot write a self-contained question naming all entities
+- Source sentence is truncated or a fragment
+
+Write exactly: SKIP
+
+source_type: image | table | infobox | prose
+subject: science | medicine | history | geography | technology | education | culture | politics | economics | biology | sports | entertainment | other
+
+Output format (5 lines only):
+Q: <natural, self-contained question>
+A: <concise answer>
+S: <verbatim complete span from the chunk>
+T: <source_type>
+C: <subject>"""
+
+
+async def generate_qa(
+ client: genai.Client,
+ model: str,
+ chunk_path: str,
+ semaphore: asyncio.Semaphore,
+ token_counter: dict,
+) -> dict | None:
+ async with semaphore:
+ b64 = encode_image(chunk_path)
+
+ contents = [
+ {
+ "role": "user",
+ "parts": [
+ {"text": QUERY_PROMPT},
+ {"inline_data": {"mime_type": "image/jpeg", "data": b64}},
+ ],
+ }
+ ]
+ config = GenerateContentConfig(temperature=0.7, max_output_tokens=1024)
+
+ for attempt in range(5):
+ try:
+ t0 = time.time()
+ resp = await asyncio.get_event_loop().run_in_executor(
+ None,
+ lambda: client.models.generate_content(
+ model=model,
+ contents=contents,
+ config=config,
+ ),
+ )
+ elapsed = time.time() - t0
+
+ usage = resp.usage_metadata
+ if usage:
+ token_counter["input"] += getattr(usage, "prompt_token_count", 0)
+ token_counter["output"] += getattr(
+ usage, "candidates_token_count", 0
+ )
+ token_counter["calls"] += 1
+ token_counter["total_time"] += elapsed
+
+ text = ""
+ for p in resp.candidates[0].content.parts:
+ raw = getattr(p, "text", None)
+ if raw and not getattr(p, "thought", False):
+ text = raw.strip()
+
+ if not text or text.strip() == "SKIP":
+ return None
+
+ fields = {}
+ for line in text.split("\n"):
+ line = line.strip()
+ for prefix, key in [
+ ("Q:", "query"),
+ ("A:", "answer"),
+ ("S:", "source_sentence"),
+ ("T:", "source_type"),
+ ("C:", "subject"),
+ ]:
+ if line.startswith(prefix):
+ fields[key] = line[len(prefix) :].strip()
+ break
+
+ q = fields.get("query")
+ a = fields.get("answer")
+ if not q or not a or len(a) < 2:
+ return None
+
+ qa = {
+ "query": q,
+ "answer": a,
+ "source_sentence": fields.get("source_sentence"),
+ "source_type": fields.get("source_type", "prose"),
+ "subject": fields.get("subject", "other"),
+ }
+
+ if not is_natural_question(qa):
+ return None
+
+ return qa
+
+ except Exception as e:
+ err = str(e)
+ if "429" in err or "RESOURCE_EXHAUSTED" in err:
+ wait = 2**attempt * 15 + random.uniform(1, 5)
+ print(f" Rate limited, waiting {wait:.0f}s...")
+ await asyncio.sleep(wait)
+ elif attempt < 4:
+ await asyncio.sleep(2)
+ else:
+ print(f" Failed after 5 attempts: {e}")
+ return None
+
+ return None
+
+
+async def main():
+ parser = argparse.ArgumentParser(
+ description="Generate synthetic query-chunk pairs from Wikipedia screenshot tiles"
+ )
+ parser.add_argument(
+ "--tiles-dir",
+ type=Path,
+ required=True,
+ help="Root directory of kiwix_tiles (containing index.jsonl)",
+ )
+ parser.add_argument("--num-pages", type=int, default=10)
+ parser.add_argument("--output", type=str, default="query_pairs.jsonl")
+ parser.add_argument("--seed", type=int, default=42)
+ parser.add_argument(
+ "--model",
+ type=str,
+ default=DEFAULT_MODEL,
+ help="Gemini model (e.g. gemini-3.1-pro-preview, gemini-2.0-flash-001)",
+ )
+ parser.add_argument(
+ "--batch-index",
+ type=int,
+ default=0,
+ help="Which non-overlapping batch to process (0-based)",
+ )
+ parser.add_argument(
+ "--total-batches",
+ type=int,
+ default=1,
+ help="Total number of batches the candidate pool is divided into",
+ )
+ parser.add_argument("--max-concurrent", type=int, default=MAX_CONCURRENT)
+ parser.add_argument("--postfilter-min-page-chunks", type=int, default=None)
+ parser.add_argument("--postfilter-max-page-chunks", type=int, default=None)
+ args = parser.parse_args()
+
+ random.seed(args.seed)
+ os.makedirs(os.path.dirname(args.output) or ".", exist_ok=True)
+
+ tiles_root = args.tiles_dir
+ index_path = tiles_root / "index.jsonl"
+ if not index_path.exists():
+ raise FileNotFoundError(f"index.jsonl not found at {index_path}")
+
+ model = args.model
+ price_input, price_output = MODEL_PRICING.get(model, (1.25, 10.00))
+
+ print(
+ f"Model: {model} via Vertex AI (project={os.environ.get('GOOGLE_CLOUD_PROJECT', 'N/A')})"
+ )
+
+ pages = load_and_sample_pages(
+ index_path, args.num_pages, args.batch_index, args.total_batches
+ )
+ pages_before_postfilter = len(pages)
+ pages = filter_selected_pages_by_chunk_count(
+ pages,
+ tiles_root,
+ min_page_chunks=args.postfilter_min_page_chunks,
+ max_page_chunks=args.postfilter_max_page_chunks,
+ )
+ if (
+ args.postfilter_min_page_chunks is not None
+ or args.postfilter_max_page_chunks is not None
+ ):
+ chunk_filter = []
+ if args.postfilter_min_page_chunks is not None:
+ chunk_filter.append(f"chunks>={args.postfilter_min_page_chunks}")
+ if args.postfilter_max_page_chunks is not None:
+ chunk_filter.append(f"chunks<={args.postfilter_max_page_chunks}")
+ print(
+ "Post-slice page filter: "
+ + ", ".join(chunk_filter)
+ + f" -> kept {len(pages)}/{pages_before_postfilter} pages"
+ )
+ print(f"Selected {len(pages)} pages\n")
+
+ semaphore = asyncio.Semaphore(args.max_concurrent)
+ token_counter = {"input": 0, "output": 0, "calls": 0, "total_time": 0.0}
+ results = []
+
+ client = genai.Client(http_options=HttpOptions(api_version="v1"))
+
+ async def process_one(page, chunk_path, chunk_idx):
+ qa = await generate_qa(client, model, chunk_path, semaphore, token_counter)
+ if not qa:
+ return None
+ rel_path = str(Path(chunk_path).relative_to(tiles_root))
+ return {
+ **qa,
+ "chunk_path": rel_path,
+ "url": page["url"],
+ "title": page["title"],
+ "chunk_index": chunk_idx,
+ "tiles_dir": page["tiles_dir"],
+ }
+
+ work_items = []
+ for page in pages:
+ chunk_path, chunk_idx = pick_random_chunk(page, tiles_root)
+ if chunk_path is None:
+ continue
+ work_items.append((page, chunk_path, chunk_idx))
+
+ print(f"Generating Q&A for {len(work_items)} chunks...\n")
+ t_start = time.time()
+
+ tasks = [asyncio.ensure_future(process_one(p, cp, ci)) for p, cp, ci in work_items]
+
+ with open(args.output, "w") as outf:
+ for coro in asyncio.as_completed(tasks):
+ result = await coro
+ if result:
+ results.append(result)
+ outf.write(json.dumps(result, ensure_ascii=False) + "\n")
+ outf.flush()
+ st = result.get("source_type", "?")
+ subj = result.get("subject", "?")
+ print(f" [{st:8s}|{subj:14s}] {result['title']}")
+ print(f" Q: {result['query'][:90]}")
+ print(f" A: {result['answer'][:90]}")
+ print()
+
+ wall_time = time.time() - t_start
+
+ type_dist = Counter(r["source_type"] for r in results)
+ subj_dist = Counter(r["subject"] for r in results)
+
+ in_tok = token_counter["input"]
+ out_tok = token_counter["output"]
+ calls = token_counter["calls"]
+ cost = (in_tok / 1e6 * price_input) + (out_tok / 1e6 * price_output)
+
+ print(f"\n{'=' * 60}")
+ print(f"Wrote {len(results)} Q&A pairs to {args.output}")
+ print(f"Source types: {dict(type_dist)}")
+ print(f"Subjects: {dict(subj_dist)}")
+ print(f"\n--- Throughput & Cost ({model}) ---")
+ print(f" Calls: {calls}")
+ print(f" Input tokens: {in_tok:,} ({in_tok / max(calls, 1):.0f} avg/call)")
+ print(f" Output tokens:{out_tok:,} ({out_tok / max(calls, 1):.0f} avg/call)")
+ print(f" Wall time: {wall_time:.1f}s ({wall_time / max(calls, 1):.1f}s/call)")
+ print(f" Est. cost: ${cost:.4f} for {calls} calls")
+ print(f" Per 10 pairs: ${cost / max(calls, 1) * 10:.4f}")
+ print(f" Per 1K pairs: ${cost / max(calls, 1) * 1000:.2f}")
+ print(f" Per 50K pairs:${cost / max(calls, 1) * 50000:.0f}")
+ print(f"{'=' * 60}")
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/train/generate_text_query_pairs.py b/train/generate_text_query_pairs.py
new file mode 100644
index 0000000..e44c0ed
--- /dev/null
+++ b/train/generate_text_query_pairs.py
@@ -0,0 +1,771 @@
+#!/usr/bin/env python3
+"""
+Generate synthetic query-passage pairs for text-only contrastive learning.
+
+Samples chunks from a local text passage SQLite database, prompts Gemini with
+few-shot examples, and writes JSONL records containing the generated query,
+answer, supporting span, and source passage metadata.
+
+The output is used as text-warmup data for the visual embedding model.
+
+Prerequisites:
+ - SQLite database with text passages (schema: articles + chunks tables)
+ - Google Cloud ADC for Vertex AI Gemini access
+ - Optionally: OPENAI_API_KEY for fallback when Gemini is rate-limited
+ - Optionally: --fewshot-file with seed examples (JSONL with question/answer/text/supporting_span)
+
+Usage:
+ python generate_text_query_pairs.py \
+ --db-path /path/to/text_baseline.db \
+ --num-articles 1000 \
+ --output text_query_pairs.jsonl
+
+Ported from Vis-RAG/agent/scripts/contrastive/generate_text_query_pairs.py
+"""
+
+import argparse
+import asyncio
+import json
+import os
+import random
+import re
+import sqlite3
+import time
+from collections import Counter
+from pathlib import Path
+
+from google import genai
+from google.genai.types import GenerateContentConfig, HttpOptions
+import openai
+
+
+os.environ.setdefault("GOOGLE_CLOUD_PROJECT", "wise-coyote-478119-h0")
+os.environ.setdefault("GOOGLE_GENAI_USE_VERTEXAI", "true")
+os.environ.setdefault("GOOGLE_CLOUD_LOCATION", "global")
+
+MODEL_PRICING = {
+ "gemini-3.1-flash-lite-preview": (1.00, 4.00),
+ "gemini-2.0-flash-001": (0.10, 0.40),
+ "gemini-2.0-flash": (0.10, 0.40),
+ "gemini-3.1-pro-preview": (1.25, 10.00),
+ "gpt-4o-mini": (0.15, 0.60),
+ "gpt-4o": (2.50, 10.00),
+}
+DEFAULT_MODEL = "gemini-3.1-flash-lite-preview"
+MASTER_SEED = 0
+
+SKIP_PATTERNS = [
+ r"disambiguation",
+ r"^portal:",
+ r"^category:",
+ r"^template:",
+ r"^wikipedia:",
+ r"^file:",
+ r"^help:",
+ r"^talk:",
+ r"^module:",
+ r"^draft:",
+ r"^list of ",
+ r"^lists of ",
+]
+SKIP_CONTENT_PATTERNS = [
+ r"\belection\b",
+ r"\belections\b",
+ r"\breferendum\b",
+ r"\bdiscography\b",
+ r"\bfilmography\b",
+ r"\btrack listing\b",
+ r"\bcensus\b",
+ r"\bdemographic\b",
+ r"^list of .* episodes",
+ r"\bepisodes of\b",
+]
+SKIP_RE = [re.compile(p, re.IGNORECASE) for p in SKIP_PATTERNS]
+SKIP_CONTENT_RE = [re.compile(p, re.IGNORECASE) for p in SKIP_CONTENT_PATTERNS]
+
+BAD_QUESTION_PATTERNS = [
+ r"\baccording to the\b",
+ r"\baccording to this\b",
+ r"\bvisible\b",
+ r"\blisted (here|above|below|in the table)\b",
+ r"\bthe table (shows|lists|above|below)\b",
+ r"\bin the (following|above) table\b",
+ r"\bshown in\b",
+ r"\bthis passage\b",
+ r"\bthe passage\b",
+ r"\bthe article\b",
+ r"^what is (the )?listed",
+ r"^what (are|is) listed",
+ r"\bthe film\b(?!\s+[A-Z\"])",
+ r"\bthe song\b(?!\s+[A-Z\"])",
+ r"\bthe album\b(?!\s+[A-Z\"])",
+ r"\bthe book\b(?!\s+[A-Z\"])",
+ r"\bthe team\b(?!\s+[A-Z\"])",
+ r"\bthe show\b(?!\s+[A-Z\"])",
+ r"\bthe series\b(?!\s+[A-Z\"])",
+ r"\bthe station\b(?!\s+[A-Z\"])",
+ r"\bthe school\b(?!\s+[A-Z\"])",
+ r"\bthe match\b(?!\s+[A-Z\"])",
+ r"\bthe game\b(?!\s+[A-Z\"])",
+ r"\bthe competition\b(?!\s+[A-Z\"])",
+ r"\bthe episode\b(?!\s+[A-Z\"\d])",
+ r"\bthe production\b(?!\s+[A-Z\"])",
+ r"\bthe tournament\b(?!\s+[A-Z\d\"])",
+ r"^(when|where|who|what|how|why) (was|is|were|did|does|has|have) (it|they|this|that|he|she)\b",
+]
+BAD_Q_RE = [re.compile(p, re.IGNORECASE) for p in BAD_QUESTION_PATTERNS]
+
+# Built-in few-shot examples (used when --fewshot-file is not provided)
+BUILTIN_FEWSHOT = [
+ {
+ "question": "In what year was the National Register of Historic Places listing for the Alamo Mission established?",
+ "answer": "1966",
+ "text": "The Alamo Mission\n\nThe Alamo Mission (Spanish: Misión de Álamo), commonly called the Alamo and originally known as the Misión San Antonio de Valero, is a historic Spanish mission and fortress compound founded in the 18th century by Roman Catholic missionaries in what is now San Antonio, Texas, United States.\n\nIt was the site of the Battle of the Alamo in 1836. The Alamo complex was designated a UNESCO World Heritage Site in 2015. The compound was listed on the National Register of Historic Places in 1966 and designated a National Historic Landmark in 1960.",
+ "supporting_span": "The compound was listed on the National Register of Historic Places in 1966 and designated a National Historic Landmark in 1960.",
+ },
+ {
+ "question": "How many species of birds have been recorded in Colombia as of the latest count?",
+ "answer": "1,878",
+ "text": "Birds of Colombia\n\nThis is a list of the bird species recorded in Colombia. The avifauna of Colombia included a total of 1,878 species as of July 2006, according to Bird Checklists of the World maintained by Avibase. Of these, 67 are endemic, 3 have been introduced by humans, and 80 are rare or accidental.\n\nColombia has more bird species than any other country in the world except for its neighbor, Brazil, and roughly one-fifth of the world's bird species can be found in Colombia.",
+ "supporting_span": "The avifauna of Colombia included a total of 1,878 species as of July 2006, according to Bird Checklists of the World maintained by Avibase.",
+ },
+ {
+ "question": "What was the top speed of the Concorde supersonic airliner in Mach number?",
+ "answer": "Mach 2.04",
+ "text": "Concorde\n\nConcorde is a retired Franco-British supersonic airliner jointly developed and manufactured by Sud Aviation (later Aérospatiale) and the British Aircraft Corporation (BAC).\n\nConcorde had a maximum speed of Mach 2.04 (2,180 km/h at cruise altitude), over twice the speed of sound. It could seat 92 to 128 passengers. First flown in 1969, Concorde entered service in 1976 and operated for 27 years.\n\nConcorde was retired in 2003 following the crash of Air France Flight 4590 and a general downturn in commercial aviation after the September 11 attacks.",
+ "supporting_span": "Concorde had a maximum speed of Mach 2.04 (2,180 km/h at cruise altitude), over twice the speed of sound.",
+ },
+ {
+ "question": "Who was the first person to reach the summit of Mount Everest?",
+ "answer": "Sir Edmund Hillary and Tenzing Norgay",
+ "text": "Mount Everest\n\nMount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The China–Nepal border runs across its summit point. Its elevation of 8,848.86 m (29,031 ft 8+1⁄2 in) was most recently established in 2020 by the Chinese and Nepali authorities.\n\nThe first recorded efforts to reach Everest's summit were made by British mountaineers. On 29 May 1953, New Zealander Sir Edmund Hillary and Sherpa Tenzing Norgay became the first climbers confirmed to have reached the summit of Mount Everest.",
+ "supporting_span": "On 29 May 1953, New Zealander Sir Edmund Hillary and Sherpa Tenzing Norgay became the first climbers confirmed to have reached the summit of Mount Everest.",
+ },
+ {
+ "question": "In what year did Marie Curie win her second Nobel Prize?",
+ "answer": "1911",
+ "text": "Marie Curie\n\nMarie Salomea Skłodowska–Curie (7 November 1867 – 4 July 1934) was a Polish and naturalized-French physicist and chemist who conducted pioneering research on radioactivity.\n\nShe was the first woman to win a Nobel Prize, the first person to win a Nobel Prize twice, and the only person to win a Nobel Prize in two scientific fields. She was awarded the Nobel Prize in Physics in 1903 together with her husband Pierre Curie and physicist Henri Becquerel. In 1911, she won the Nobel Prize in Chemistry for her discovery of the elements polonium and radium.",
+ "supporting_span": "In 1911, she won the Nobel Prize in Chemistry for her discovery of the elements polonium and radium.",
+ },
+]
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(
+ description="Generate synthetic text query-passage pairs from a text chunk database"
+ )
+ parser.add_argument(
+ "--db-path",
+ type=Path,
+ required=True,
+ help="Path to SQLite database with articles/chunks tables",
+ )
+ parser.add_argument(
+ "--fewshot-file",
+ type=Path,
+ default=None,
+ help="JSONL with few-shot seed examples (question/answer/text/supporting_span). "
+ "If not provided, built-in examples are used.",
+ )
+ parser.add_argument("--output", type=Path, default=Path("text_query_pairs.jsonl"))
+ parser.add_argument("--num-articles", type=int, default=100)
+ parser.add_argument("--seed", type=int, default=42)
+ parser.add_argument("--model", type=str, default=DEFAULT_MODEL)
+ parser.add_argument("--batch-index", type=int, default=0)
+ parser.add_argument("--total-batches", type=int, default=1)
+ parser.add_argument("--max-concurrent", type=int, default=32)
+ parser.add_argument("--min-article-chunks", type=int, default=6)
+ parser.add_argument("--max-article-chunks", type=int, default=None)
+ parser.add_argument("--min-paragraph-words", type=int, default=60)
+ parser.add_argument("--openai-fallback-model", type=str, default="gpt-4o-mini")
+ return parser.parse_args()
+
+
+def normalize_text(text: str) -> str:
+ return re.sub(r"\s+", " ", (text or "").strip()).lower()
+
+
+def infer_title(text: str) -> str:
+ for line in (text or "").splitlines():
+ line = line.strip()
+ if line:
+ return line[:200]
+ return ""
+
+
+def is_bad_title(title: str) -> bool:
+ check = (title or "").strip().lower()
+ if not check:
+ return True
+ for pat in SKIP_RE:
+ if pat.search(check):
+ return True
+ for pat in SKIP_CONTENT_RE:
+ if pat.search(check):
+ return True
+ return False
+
+
+def is_candidate_passage(text: str, n_tokens: int) -> bool:
+ if not text or len(text.strip()) < 300:
+ return False
+ if n_tokens < 80:
+ return False
+ title = infer_title(text)
+ if is_bad_title(title):
+ return False
+ prefix = normalize_text(text[:400])
+ if "may refer to:" in prefix:
+ return False
+ return True
+
+
+def split_paragraphs(text: str) -> list[str]:
+ return [part.strip() for part in re.split(r"\n\s*\n", text or "") if part.strip()]
+
+
+def is_list_like_paragraph(paragraph: str) -> bool:
+ lines = [line.strip() for line in paragraph.splitlines() if line.strip()]
+ if not lines:
+ return True
+ bulletish = 0
+ pipe_lines = 0
+ short_lines = 0
+ for line in lines:
+ if line.startswith(("-", "*", "|")):
+ bulletish += 1
+ if "|" in line:
+ pipe_lines += 1
+ if len(line.split()) <= 6:
+ short_lines += 1
+ if bulletish >= max(2, len(lines) // 2):
+ return True
+ if pipe_lines >= max(2, len(lines) // 2):
+ return True
+ if short_lines >= max(3, len(lines) // 2 + 1):
+ return True
+ return False
+
+
+def extract_best_long_paragraph(text: str, min_paragraph_words: int) -> str | None:
+ best = None
+ best_score = None
+ for paragraph in split_paragraphs(text):
+ normalized = normalize_text(paragraph)
+ word_count = len(paragraph.split())
+ sentence_count = len(re.findall(r"[.!?]", paragraph))
+ if is_list_like_paragraph(paragraph):
+ continue
+ if word_count < min_paragraph_words:
+ continue
+ if sentence_count < 2:
+ continue
+ if "|" in paragraph:
+ continue
+ if normalized.startswith(
+ ("references", "external links", "see also", "bibliography", "notes")
+ ):
+ continue
+ score = (word_count, sentence_count, len(paragraph))
+ if best_score is None or score > best_score:
+ best = paragraph
+ best_score = score
+ return best
+
+
+def load_fewshot_examples(path: Path | None) -> list[dict]:
+ if path is not None:
+ with open(path) as f:
+ return [json.loads(line) for line in f if line.strip()]
+ return BUILTIN_FEWSHOT
+
+
+def format_fewshot_block(examples: list[dict]) -> str:
+ blocks = []
+ for idx, ex in enumerate(examples, 1):
+ blocks.append(
+ f"""Example {idx}
+Passage:
+\"\"\"
+{ex["text"]}
+\"\"\"
+Good output:
+Q: {ex["question"]}
+A: {ex["answer"]}
+S: {ex["supporting_span"]}
+T: prose"""
+ )
+ return "\n\n".join(blocks)
+
+
+def build_prompt(fewshot_block: str, passage: str, focus_paragraph: str) -> str:
+ return f"""You are generating a query-evidence pair for training a text retrieval model over Wikipedia passages.
+
+TASK: Given this passage, generate ONE factual question whose answer is explicitly and completely supported by this passage.
+
+STYLE:
+- Write natural search-style questions like SimpleQA, not templates.
+- Prefer realistic factual questions a user would actually type into a search engine.
+- Use diverse question families such as "what is", "who was", "how many", "in which year", "what did", and similar natural forms.
+- Prefer facts stated in a long natural prose paragraph, not list items or table cells.
+
+HARD RULES:
+1. SELF-CONTAINED: The question must make sense on its own. Name the relevant entities explicitly.
+2. EVIDENCE COMPLETE: The answer must be fully supported by the passage alone.
+3. DISTINCTIVE: Include enough detail to identify the fact cleanly.
+4. VERBATIM SPAN: `S:` must be copied directly from the passage.
+5. PROSE-FIRST: Use the long prose paragraph below as the evidence anchor. If it does not support a good question, write `SKIP`.
+
+SKIP if any is true:
+- The passage is mostly raw list junk, track listings, or vote counts.
+- The long prose paragraph below does not support a clean question-answer pair.
+- You cannot write a self-contained question.
+- The answer is not fully supported by this passage.
+- The supporting span would be truncated or fragmentary.
+
+Write exactly `SKIP` if the passage should be skipped.
+
+source_type: prose | infobox | table | list | other
+
+Few-shot examples:
+
+{fewshot_block}
+
+Now generate for this new passage.
+
+Preferred long prose paragraph:
+\"\"\"
+{focus_paragraph}
+\"\"\"
+
+Passage:
+\"\"\"
+{passage}
+\"\"\"
+
+Output format (4 lines only):
+Q: <natural self-contained question>
+A: <concise answer>
+S: <verbatim supporting span from the passage>
+T: <source_type>"""
+
+
+def parse_model_output(text: str) -> dict | None:
+ if not text or text.strip() == "SKIP":
+ return None
+ fields = {}
+ for line in text.splitlines():
+ line = line.strip()
+ for prefix, key in [
+ ("Q:", "query"),
+ ("A:", "answer"),
+ ("S:", "source_sentence"),
+ ("T:", "source_type"),
+ ]:
+ if line.startswith(prefix):
+ fields[key] = line[len(prefix) :].strip()
+ break
+ if (
+ not fields.get("query")
+ or not fields.get("answer")
+ or not fields.get("source_sentence")
+ ):
+ return None
+ return {
+ "query": fields["query"],
+ "answer": fields["answer"],
+ "source_sentence": fields["source_sentence"],
+ "source_type": fields.get("source_type", "prose"),
+ }
+
+
+async def call_gemini(
+ client: genai.Client,
+ model: str,
+ prompt: str,
+ token_counter: dict,
+) -> str:
+ config = GenerateContentConfig(temperature=0.7, max_output_tokens=512)
+ t0 = time.time()
+ resp = await asyncio.get_event_loop().run_in_executor(
+ None,
+ lambda: client.models.generate_content(
+ model=model,
+ contents=prompt,
+ config=config,
+ ),
+ )
+ elapsed = time.time() - t0
+ usage = resp.usage_metadata
+ if usage:
+ token_counter["gemini_input"] += getattr(usage, "prompt_token_count", 0)
+ token_counter["gemini_output"] += getattr(usage, "candidates_token_count", 0)
+ token_counter["gemini_calls"] += 1
+ token_counter["gemini_total_time"] += elapsed
+
+ text = ""
+ for part in resp.candidates[0].content.parts:
+ raw = getattr(part, "text", None)
+ if raw and not getattr(part, "thought", False):
+ text = raw.strip()
+ return text
+
+
+async def call_openai_fallback(
+ client: openai.AsyncOpenAI,
+ model: str,
+ prompt: str,
+ token_counter: dict,
+) -> str:
+ t0 = time.time()
+ resp = await client.chat.completions.create(
+ model=model,
+ messages=[{"role": "user", "content": prompt}],
+ temperature=0.7,
+ max_tokens=512,
+ )
+ elapsed = time.time() - t0
+ token_counter["openai_input"] += resp.usage.prompt_tokens
+ token_counter["openai_output"] += resp.usage.completion_tokens
+ token_counter["openai_calls"] += 1
+ token_counter["openai_total_time"] += elapsed
+ return (resp.choices[0].message.content or "").strip()
+
+
+def is_natural_question(qa: dict, passage: str) -> bool:
+ q = qa.get("query", "")
+ a = qa.get("answer", "")
+ s = qa.get("source_sentence", "") or ""
+ src_type = qa.get("source_type", "prose")
+
+ if src_type != "prose":
+ return False
+ for pat in BAD_Q_RE:
+ if pat.search(q):
+ return False
+ if a and a[-1] in ("→", "…", "–", "/", "(", ","):
+ return False
+ if not s:
+ return False
+ if s.rstrip()[-1:] in ("(", ",", "–", "/", "→", "…"):
+ return False
+ if len(s.split()) < 10:
+ return False
+ if normalize_text(s) not in normalize_text(passage):
+ return False
+ return True
+
+
+def load_candidate_articles(
+ db_path: Path,
+ num_articles: int,
+ batch_index: int,
+ total_batches: int,
+ min_article_chunks: int,
+ max_article_chunks: int | None,
+) -> list[dict]:
+ conn = sqlite3.connect(str(db_path))
+ cur = conn.cursor()
+ query = """
+ SELECT article_id, n_chunks, text_length
+ FROM articles
+ WHERE status = 'extracted'
+ AND n_chunks >= ?
+ """
+ params: list[int] = [min_article_chunks]
+ if max_article_chunks is not None:
+ query += "\n AND n_chunks <= ?"
+ params.append(max_article_chunks)
+ rows = list(cur.execute(query, params))
+ conn.close()
+
+ candidates = [
+ {"article_id": article_id, "n_chunks": n_chunks, "text_length": text_length}
+ for article_id, n_chunks, text_length in rows
+ ]
+ print(f"Total eligible articles: {len(candidates):,}")
+
+ rng = random.Random(MASTER_SEED)
+ rng.shuffle(candidates)
+
+ slice_size = len(candidates) // total_batches
+ start = batch_index * slice_size
+ end = start + slice_size if batch_index < total_batches - 1 else len(candidates)
+ pool = candidates[start:end]
+ print(
+ f"Batch {batch_index}/{total_batches}: articles [{start}:{end}] ({len(pool):,} in pool)"
+ )
+
+ selected = pool[:num_articles] if num_articles <= len(pool) else pool
+ return selected
+
+
+def select_chunk_rows(
+ db_path: Path,
+ articles: list[dict],
+ seed: int,
+ min_paragraph_words: int,
+) -> list[dict]:
+ rng = random.Random(seed)
+ conn = sqlite3.connect(str(db_path))
+ cur = conn.cursor()
+ work_items = []
+
+ for article in articles:
+ usable_chunk_count = max(1, int(article["n_chunks"] * 0.7))
+ candidate_indices = list(range(usable_chunk_count))
+ rng.shuffle(candidate_indices)
+
+ selected_row = None
+ for chunk_index in candidate_indices[: min(len(candidate_indices), 5)]:
+ row = cur.execute(
+ """
+ SELECT text, char_offset, n_tokens
+ FROM chunks
+ WHERE article_id = ? AND chunk_index = ?
+ """,
+ (article["article_id"], chunk_index),
+ ).fetchone()
+ if not row:
+ continue
+ text, char_offset, n_tokens = row
+ if not is_candidate_passage(text, n_tokens):
+ continue
+ focus_paragraph = extract_best_long_paragraph(
+ text, min_paragraph_words=min_paragraph_words
+ )
+ if not focus_paragraph:
+ continue
+ title = infer_title(text)
+ selected_row = {
+ "article_id": article["article_id"],
+ "article_n_chunks": article["n_chunks"],
+ "article_text_length": article["text_length"],
+ "chunk_index": chunk_index,
+ "char_offset": char_offset,
+ "n_tokens": n_tokens,
+ "title_guess": title,
+ "focus_paragraph": focus_paragraph,
+ "passage": text,
+ }
+ break
+
+ if selected_row:
+ work_items.append(selected_row)
+
+ conn.close()
+ return work_items
+
+
+async def generate_qa(
+ client: genai.Client,
+ openai_client: openai.AsyncOpenAI | None,
+ semaphore: asyncio.Semaphore,
+ token_counter: dict,
+ model: str,
+ openai_fallback_model: str,
+ fewshot_block: str,
+ work_item: dict,
+) -> dict | None:
+ prompt = build_prompt(
+ fewshot_block, work_item["passage"], work_item["focus_paragraph"]
+ )
+
+ async with semaphore:
+ for attempt in range(5):
+ try:
+ text = await call_gemini(client, model, prompt, token_counter)
+
+ qa = parse_model_output(text)
+ if not qa:
+ return None
+ if not is_natural_question(qa, work_item["passage"]):
+ return None
+ return {
+ **qa,
+ "article_id": work_item["article_id"],
+ "article_n_chunks": work_item["article_n_chunks"],
+ "article_text_length": work_item["article_text_length"],
+ "chunk_index": work_item["chunk_index"],
+ "char_offset": work_item["char_offset"],
+ "n_tokens": work_item["n_tokens"],
+ "title_guess": work_item["title_guess"],
+ "focus_paragraph": work_item["focus_paragraph"],
+ "passage": work_item["passage"],
+ "generator_model": model,
+ }
+ except Exception as e:
+ err = str(e)
+ if "429" in err or "RESOURCE_EXHAUSTED" in err:
+ if openai_client is not None:
+ try:
+ print(
+ f" Gemini rate limited for article {work_item['article_id']}; "
+ f"falling back to {openai_fallback_model}"
+ )
+ text = await call_openai_fallback(
+ openai_client,
+ openai_fallback_model,
+ prompt,
+ token_counter,
+ )
+ qa = parse_model_output(text)
+ if not qa:
+ return None
+ if not is_natural_question(qa, work_item["passage"]):
+ return None
+ return {
+ **qa,
+ "article_id": work_item["article_id"],
+ "article_n_chunks": work_item["article_n_chunks"],
+ "article_text_length": work_item["article_text_length"],
+ "chunk_index": work_item["chunk_index"],
+ "char_offset": work_item["char_offset"],
+ "n_tokens": work_item["n_tokens"],
+ "title_guess": work_item["title_guess"],
+ "focus_paragraph": work_item["focus_paragraph"],
+ "passage": work_item["passage"],
+ "generator_model": openai_fallback_model,
+ }
+ except Exception as openai_error:
+ print(
+ f" OpenAI fallback failed for article {work_item['article_id']}: "
+ f"{openai_error}"
+ )
+ wait = 2**attempt * 10 + random.uniform(1, 3)
+ print(f" Rate limited, waiting {wait:.0f}s...")
+ await asyncio.sleep(wait)
+ elif attempt < 4:
+ await asyncio.sleep(2)
+ else:
+ print(
+ f" Failed after 5 attempts for article {work_item['article_id']}: {e}"
+ )
+ return None
+ return None
+
+
+async def main() -> None:
+ args = parse_args()
+ random.seed(args.seed)
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+
+ fewshot_examples = load_fewshot_examples(args.fewshot_file)
+ fewshot_block = format_fewshot_block(fewshot_examples)
+
+ print(
+ f"Model: {args.model} via Vertex AI (project={os.environ.get('GOOGLE_CLOUD_PROJECT', 'N/A')})"
+ )
+ print(
+ f"Few-shot examples: {len(fewshot_examples)} ({'from file' if args.fewshot_file else 'built-in'})"
+ )
+ has_openai_fallback = bool(os.environ.get("OPENAI_API_KEY"))
+ print(
+ "OpenAI fallback: "
+ + (
+ f"enabled ({args.openai_fallback_model})"
+ if has_openai_fallback
+ else "disabled"
+ )
+ )
+
+ articles = load_candidate_articles(
+ db_path=args.db_path,
+ num_articles=args.num_articles,
+ batch_index=args.batch_index,
+ total_batches=args.total_batches,
+ min_article_chunks=args.min_article_chunks,
+ max_article_chunks=args.max_article_chunks,
+ )
+ work_items = select_chunk_rows(
+ args.db_path, articles, args.seed, args.min_paragraph_words
+ )
+ print(f"Selected {len(work_items)} candidate chunks from {len(articles)} articles")
+
+ semaphore = asyncio.Semaphore(args.max_concurrent)
+ token_counter = {
+ "gemini_input": 0,
+ "gemini_output": 0,
+ "gemini_calls": 0,
+ "gemini_total_time": 0.0,
+ "openai_input": 0,
+ "openai_output": 0,
+ "openai_calls": 0,
+ "openai_total_time": 0.0,
+ }
+ client = genai.Client(http_options=HttpOptions(api_version="v1"))
+ openai_client = (
+ openai.AsyncOpenAI(
+ api_key=os.environ.get("OPENAI_API_KEY"),
+ base_url=os.environ.get("OPENAI_BASE_URL", "https://api.openai.com/v1"),
+ )
+ if has_openai_fallback
+ else None
+ )
+ tasks = [
+ asyncio.ensure_future(
+ generate_qa(
+ client=client,
+ openai_client=openai_client,
+ semaphore=semaphore,
+ token_counter=token_counter,
+ model=args.model,
+ openai_fallback_model=args.openai_fallback_model,
+ fewshot_block=fewshot_block,
+ work_item=item,
+ )
+ )
+ for item in work_items
+ ]
+
+ results = []
+ t_start = time.time()
+ with open(args.output, "w") as outf:
+ for coro in asyncio.as_completed(tasks):
+ result = await coro
+ if result:
+ results.append(result)
+ outf.write(json.dumps(result, ensure_ascii=False) + "\n")
+ outf.flush()
+ print(
+ f" [{result['source_type']:7s}] article={result['article_id']} chunk={result['chunk_index']}"
+ )
+ print(f" Q: {result['query'][:100]}")
+ print(f" A: {result['answer'][:80]}")
+
+ wall_time = time.time() - t_start
+ type_dist = Counter(r["source_type"] for r in results)
+ model_dist = Counter(r["generator_model"] for r in results)
+ gemini_in_price, gemini_out_price = MODEL_PRICING.get(args.model, (1.0, 4.0))
+ openai_in_price, openai_out_price = MODEL_PRICING.get(
+ args.openai_fallback_model, (0.15, 0.60)
+ )
+ gemini_cost = (
+ token_counter["gemini_input"] / 1e6 * gemini_in_price
+ + token_counter["gemini_output"] / 1e6 * gemini_out_price
+ )
+ openai_cost = (
+ token_counter["openai_input"] / 1e6 * openai_in_price
+ + token_counter["openai_output"] / 1e6 * openai_out_price
+ )
+ cost = gemini_cost + openai_cost
+
+ print(f"\n{'=' * 60}")
+ print(f"Wrote {len(results)} text query pairs to {args.output}")
+ print(f"Source types: {dict(type_dist)}")
+ print(f"Generator models: {dict(model_dist)}")
+ print(
+ f"Gemini calls: {token_counter['gemini_calls']} | "
+ f"in={token_counter['gemini_input']:,} out={token_counter['gemini_output']:,}"
+ )
+ print(
+ f"OpenAI calls: {token_counter['openai_calls']} | "
+ f"in={token_counter['openai_input']:,} out={token_counter['openai_output']:,}"
+ )
+ print(f"Wall time: {wall_time:.1f}s")
+ print(f"Est. cost: ${cost:.4f}")
+ print(f"{'=' * 60}")
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/train/run_generate_batches.sh b/train/run_generate_batches.sh
new file mode 100755
index 0000000..6ece5b9
--- /dev/null
+++ b/train/run_generate_batches.sh
@@ -0,0 +1,80 @@
+#!/bin/bash
+# Batch runner for generate_query_pairs.py
+#
+# Divides the page pool into non-overlapping batches and runs one at a time,
+# skipping any batch whose output file already exists.
+#
+# Usage:
+# bash run_generate_batches.sh <tiles_dir> <start> <end> [model] [extra args...]
+#
+# Examples:
+# # Generate batches 0–99 with default Gemini Pro:
+# bash run_generate_batches.sh /opt/dlami/nvme/kiwix_tiles 0 99
+#
+# # Use Flash Lite for cheaper/faster generation:
+# bash run_generate_batches.sh /opt/dlami/nvme/kiwix_tiles 0 99 gemini-3.1-flash-lite-preview
+#
+# # Filter by page chunk count:
+# bash run_generate_batches.sh /opt/dlami/nvme/kiwix_tiles 0 50 gemini-3.1-flash-lite-preview \
+# --postfilter-min-page-chunks 21 --postfilter-max-page-chunks 50
+
+set -euo pipefail
+
+TILES_DIR=${1:?Usage: run_generate_batches.sh <tiles_dir> <start> <end> [model] [extra args...]}
+START=${2:-0}
+END=${3:-9}
+MODEL=${4:-gemini-3.1-pro-preview}
+if [ "$#" -ge 4 ]; then
+ shift 4
+else
+ shift "$#"
+fi
+EXTRA_ARGS=("$@")
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+OUTPUT_DIR="$SCRIPT_DIR/batches"
+mkdir -p "$OUTPUT_DIR"
+
+TOTAL_BATCHES=1000
+PAGES_PER_BATCH=2000
+
+echo "Tiles dir: $TILES_DIR"
+echo "Batches: $START to $END (of $TOTAL_BATCHES total)"
+echo "Model: $MODEL"
+echo "Extra args: ${EXTRA_ARGS[*]:-none}"
+echo "Output dir: $OUTPUT_DIR"
+echo "Started: $(date)"
+echo ""
+
+for i in $(seq "$START" "$END"); do
+ BATCH=$(printf "%03d" "$i")
+ OUT="$OUTPUT_DIR/batch_${BATCH}.jsonl"
+ LOG="$OUTPUT_DIR/batch_${BATCH}.log"
+
+ if [ -f "$OUT" ]; then
+ echo "[$BATCH] exists ($(wc -l < "$OUT") pairs), skipping."
+ continue
+ fi
+
+ echo "[$BATCH] start $(date)"
+ uv run python "$SCRIPT_DIR/generate_query_pairs.py" \
+ --tiles-dir "$TILES_DIR" \
+ --batch-index "$i" \
+ --total-batches "$TOTAL_BATCHES" \
+ --num-pages "$PAGES_PER_BATCH" \
+ --output "$OUT" \
+ --seed "$i" \
+ --model "$MODEL" \
+ "${EXTRA_ARGS[@]}" \
+ 2>&1 | tee "$LOG"
+
+ echo "[$BATCH] done $(wc -l < "$OUT" 2>/dev/null || echo 0) pairs | $(date)"
+ echo ""
+done
+
+echo "All done | $(date)"
+
+# Merge all batch files
+MERGED="$OUTPUT_DIR/all_pairs.jsonl"
+cat "$OUTPUT_DIR"/batch_*.jsonl > "$MERGED" 2>/dev/null || true
+echo "Merged: $(wc -l < "$MERGED" 2>/dev/null || echo 0) pairs total → $MERGED"