BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.
Python ai-agentsai-memoryknowledge-graphknowledge-managementknowledge-representationreccomendation-system
Add hybrid search benchmark suite
Christian6006d27336a9
diff --git a/benchmarks/.env.example b/benchmarks/.env.exampleindex 5cbb772..313d245 100644--- a/benchmarks/.env.example+++ b/benchmarks/.env.example@@ -1,6 +1,9 @@BRAINAPI_URL=http://localhost:8000BRAINPAT_TOKEN=+# Search eval (default searchbenchsmoke). Must start with searchbench.+# SEARCH_BRAIN_ID=searchbenchsmoke+# Dual profile: product (greedy) | sota (deepseek-v4-flash + SC + gap-fill)# BENCH_PROFILE=product# BENCH_PROFILE=sotadiff --git a/benchmarks/AGENTS.md b/benchmarks/AGENTS.mdindex a742471..a07584a 100644--- a/benchmarks/AGENTS.md+++ b/benchmarks/AGENTS.md@@ -8,8 +8,9 @@| **LongMemEval** | `benchmarks/longmemeval/` | `./longmemeval.sh` | `benchmarks.longmemeval` || **BEAM** | `benchmarks/beam/` | `./beam.sh` | `benchmarks.beam` || **RecSys** | `benchmarks/recsys/` | `./recsys.sh` | `benchmarks.recsys` |+| **Search** | `benchmarks/search/` | `./search.sh` | `benchmarks.search` |-All harnesses talk to a running BrainAPI over HTTP only (no `src/` imports). RecSys uses `POST /ingest/structured` (KB write) + **train-free** `GET /retrieve/recommend` on brain **`demorecsys` only** — never LoCoMo/BEAM brains. Optional: `plugins/features-rec` for attribute prefs; `plugins/recsys-gnn` LightGCN via `--backend lightgcn`.+All harnesses talk to a running BrainAPI over HTTP only (no `src/` imports, except Search `mapping.py` re-exporting the product catalog mapper). RecSys uses `POST /ingest/structured` (KB write) + **train-free** `GET /retrieve/recommend` on brain **`demorecsys` only** — never LoCoMo/BEAM brains. Optional: `plugins/features-rec` for attribute prefs; `plugins/recsys-gnn` LightGCN via `--backend lightgcn`. Search uses `POST /ingest/` + `POST /retrieve/search` on **`searchbench*` only** — never LoCoMo/BEAM/recsys brains, never `/retrieve/context` as the metric path.### LoCoMo@@ -35,6 +36,28 @@ Held-out next-item HitRate/Recall@K via **train-free graph recommend** (default)Protocol: [`docs/research/16-recsys-eval-protocol.md`](../docs/research/16-recsys-eval-protocol.md). Upserts only `benchmarks.recsys` — never wipe or write to `beam1m1clean` / `locomoconv26*`.+### Search++Hybrid BM25 + dense ranking via `POST /retrieve/search`, plus optional graph channels (`entities` / `events` / `communities`, `expand=neighbors`). Flow: ingest docs (chunk+embed; LLM enrichment skipped by default) → optional structured catalog triples (`--ingest-graph`) → map `DOCID` markers to chunk ids → search with `profile_stages`. Gold matches chunk ids or `hit.id == doc_id`. Brain: `searchbenchsmoke` (any id must start with `searchbench`). Server must have `SEARCH_ENABLED=true`, `DATA_DB=postgresql`, `BRAIN_CREATION_ALLOWED=true`.++```bash+./search.sh smoke+./search.sh evaluate --fusion rrf+./search.sh download --name esci+./search.sh download --name wands+./search.sh evaluate --dataset data/search_esci.jsonl --brain searchbenchesci --run search-esci+./search.sh --brain searchbenchwands evaluate --dataset data/search_wands.jsonl --run search-wands-passages-k50 --channels passages --k 50 --ks 5,10,20,50+./search.sh --brain searchbenchwands evaluate --dataset data/search_wands.jsonl --run search-wands-passages-k50 --channels passages --k 50 --ks 5,10,20,50 --skip-ingest+./search.sh --brain searchbenchwandsgraph evaluate --dataset data/search_wands.jsonl --run search-wandsgraph-communities-k50 --channels communities --k 50 --ks 5,10,20,50 --ingest-graph+./search.sh evaluate --dataset data/search_esci.jsonl --brain searchbenchesci --ingest-graph --channels passages,entities,communities+./search.sh evaluate --rerank plugin:cross-encoder+./search.sh report --run <run_id>+```++Catalog downloads are a **subset** of Amazon ESCI (KDD Cup 2022, US Task 1 / small_version test) and WANDS (Wayfair). Default caps: 80 queries, 2000 products, 40 candidates/query. Frozen WANDS quality slice: `data/search_wands.jsonl` (66 queries / 2000 docs); download refuses overwrite. Gold is graded (ESCI E/S/C/I; WANDS Exact/Partial/Irrelevant). WANDS control is passages k=50 `rerank=none` on `searchbenchwands`, then `--skip-ingest`. Opt-in catalog graph is brain `searchbenchwandsgraph` only (`--ingest-graph`, `channels=communities`); ledger rows are an **architecture demo**, not a claim vs 0.823 / ESCI 0.500. `--ingest-graph` is refused on frozen `searchbenchwands` / `searchbenchesci74` / `searchbenchescies` / `searchbenchesciltr2`. Pass `--enrich` only if you want Scout/Architect on ingest. Use dedicated `searchbench*` brains — never `demorecsys`. Graph mapping is product `src/core/search/catalog_graph.py` (generic HAS events + CLASS/TYPE/ATTR hubs). Do not score recs HitRate as search.++Metrics: Recall@{5,10,20}, nDCG@10, MRR; p50/p95 retrieve **excluding** embed RTT. No LLM judge. Protocol: [`docs/research/18-search-eval-protocol.md`](../docs/research/18-search-eval-protocol.md). Upserts only `benchmarks.search` — never wipe or write to `locomoconv*`, `beam*`, `demorecsys`.+### BEAMTypical flow: `download` → `ingest` → `evaluate` → `report` via `./beam.sh` or `.venv/bin/python -m beam`.@@ -55,6 +78,7 @@ Typical flow: `download` → `ingest` → `evaluate` → `report` via `./beam.sh- LoCoMo harness upserts `benchmarks.locomo` via `write_report` → `update_reports_json` when report `status` is `ok`.- BEAM harness upserts `benchmarks.beam` the same way (field: `headline_score`, continuous `[0,1]`).- RecSys harness upserts `benchmarks.recsys` only (`hit_rate@K` / `recall@K`); it must not touch other suite keys.+- Search harness upserts `benchmarks.search` only (`ndcg@10` / `recall@10` / `p50_retrieve_ms`); it must not touch other suite keys.- After any completed evaluate/report that produces a successful scored run, ensure the suite entry in `REPORTS.json` reflects it.- If you write or patch `runs/<id>/report.json` manually, update `REPORTS.json` yourself (same schema; upsert by `run_id` under the suite).- When adding a new suite, add a `benchmarks.<suite_id>` key (`name` + `leaderboard`) and upsert the same way.diff --git a/benchmarks/README.md b/benchmarks/README.mdindex 69ea41e..de7b60f 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -6,6 +6,8 @@ Standalone HTTP-only harnesses (no `src/` imports):| --- | --- | --- || [LoCoMo](https://github.com/snap-research/LoCoMo) | `locomo/` | `./locomo.sh` || [LongMemEval](https://github.com/xiaowu0162/LongMemEval) | `longmemeval/` | `./longmemeval.sh` |+| RecSys | `recsys/` | `./recsys.sh` |+| Search | `search/` | `./search.sh` |Shared setup: `requirements.txt`, `.env` (`BRAINPAT_TOKEN`, LLM keys). Results ledger: [`REPORTS.json`](REPORTS.json). Agent notes: [`AGENTS.md`](AGENTS.md).@@ -327,3 +329,56 @@ Do **not** ingest probing questions, rubrics, or ideal answers.**Cost warning:** even one 100K chat is ~100+ turns; start with `--limit-turns` / `--limit`.**1M wall time:** ~40h at concurrency 1; ~20–25h at 2 with healthy API (see `docs/research/10-beam-protocol.md`).++---++# Search benchmark for BrainAPI++Standalone HTTP-only harness for `POST /retrieve/search` (hybrid BM25 + dense). It does **not** import `src/` and does **not** call `/retrieve/context`.++Protocol: [`docs/research/18-search-eval-protocol.md`](../docs/research/18-search-eval-protocol.md).++## Prerequisites++- BrainAPI running with `SEARCH_ENABLED=true`, `DATA_DB=postgresql`, `BRAIN_CREATION_ALLOWED=true`+- `BRAINPAT_TOKEN` in `benchmarks/.env`+- Brain id starting with `searchbench` (default `searchbenchsmoke`)++A 404 on `/retrieve/search` means search is disabled — fix the server env; it is not a ranking miss.++## Quickstart++```bash+cd benchmarks+./search.sh dataset-stats+./search.sh smoke+./search.sh evaluate --fusion rrf+./search.sh evaluate --rerank plugin:cross-encoder+./search.sh report --run <run_id>+```++Toy corpus: [`data/search_toy.jsonl`](data/search_toy.jsonl) (keyword + paraphrase queries; gold via `DOCID` markers). Search ingest defaults to **chunk + embed only** (`skip_enrichment`); pass `--enrich` for Scout/Architect.++Product-search corpora (LLM-free ingest; dedicated `searchbench*` brains, not `demorecsys`):++```bash+./search.sh download --name esci+./search.sh download --name wands+./search.sh dataset-stats --dataset data/search_esci.jsonl+./search.sh evaluate --dataset data/search_esci.jsonl --brain searchbenchesci --run search-esci+./search.sh --brain searchbenchwands evaluate --dataset data/search_wands.jsonl --run search-wands-passages-k50 --channels passages --k 50 --ks 5,10,20,50+./search.sh --brain searchbenchwands evaluate --dataset data/search_wands.jsonl --run search-wands-passages-k50 --channels passages --k 50 --ks 5,10,20,50 --skip-ingest+./search.sh --brain searchbenchwandsgraph evaluate --dataset data/search_wands.jsonl --run search-wandsgraph-communities-k50 --channels communities --k 50 --ks 5,10,20,50 --ingest-graph+```++`download` writes a **subset** under `data/` (gitignored): Amazon ESCI / Shopping Queries (KDD Cup 2022, US Task 1 test) and WANDS (Wayfair). Defaults: 80 queries, 2000 products, 40 candidates per query. Frozen WANDS quality slice is `data/search_wands.jsonl` (**66** queries / **2000** docs); `--force` cannot clobber it. Raw caches live in `data/esci/` and `data/wands/`. nDCG@10 is graded (ESCI E=1 / S=0.1 / C=0.01 / I=0; WANDS Exact=1 / Partial=0.5 / Irrelevant=0). WANDS first-stage control: passages, k=50, `rerank=none`, brain `searchbenchwands`, then `--skip-ingest`. Opt-in catalog graph: new brain `searchbenchwandsgraph` only; architecture demo, do not mix with ESCI nDCG or frozen WANDS 0.823.++Optional plugins (core hybrid works if they are absent; unknown `plugin:<name>` is 400):++- `plugins/search-rerank` — `--rerank plugin:cross-encoder` (K≤10)+- `plugins/search-splade` — `--channels plugin:splade` after `POST /search-splade/index`+- `plugins/search-colbert` — `--channels plugin:colbert` after `POST /search-colbert/index`++Metrics: Recall@{5,10,20}, nDCG@10, MRR; p50/p95 of retrieve **excluding** embed RTT. Successful `evaluate` upserts `benchmarks.search` in [`REPORTS.json`](REPORTS.json). Smoke does not publish scores.++Never ingest into or wipe `locomoconv*`, `beam*`, or `demorecsys`.diff --git a/benchmarks/requirements.txt b/benchmarks/requirements.txtindex 40855cb..54e2c03 100644--- a/benchmarks/requirements.txt+++ b/benchmarks/requirements.txt@@ -3,5 +3,8 @@ openai>=1.40.0,<2.0.0python-dotenv>=1.0.0,<2.0.0rich>=13.0.0,<15.0.0datasets>=2.19.0,<4.0.0+huggingface_hub>=0.23.0,<1.0.0+pyarrow>=15.0.0,<20.0.0scipy>=1.11.0,<2.0.0json-repair>=0.30.0,<1.0.0+lightgbm>=4.3.0,<5.0.0diff --git a/benchmarks/search.sh b/benchmarks/search.shnew file mode 100755index 0000000..1d53022--- /dev/null+++ b/benchmarks/search.sh@@ -0,0 +1,31 @@+#!/usr/bin/env bash+set -euo pipefail+ROOT="$(cd "$(dirname "$0")" && pwd)"+PYTHON="${ROOT}/.venv/bin/python"+PARENT="${ROOT}/../.venv/bin/python"+if [[ ! -x "$PYTHON" ]]; then+ echo "Missing ${PYTHON}. Create it with:"+ echo " cd benchmarks && python3 -m venv .venv && .venv/bin/pip install -r requirements.txt"+ exit 1+fi+for arg in "$@"; do+ if [[ "$arg" == "rank-pool-ce" || "$arg" == "finetune-ce" || "$arg" == "pool-first-stage" || "$arg" == "rerank-retrieved" || "$arg" == "finetune-dense" || "$arg" == "local-dense" || "$arg" == "mine-retrieved-lists" || "$arg" == "finetune-4class" || "$arg" == "colbert-local" || "$arg" == "rank-corpus" || "$arg" == "ltr-head" || "$arg" == "backfill-entity-text" ]]; then+ if [[ -x "$PARENT" ]]; then+ PYTHON="$PARENT"+ fi+ if [[ "$arg" == "backfill-entity-text" && -f "${HOME}/.brainapi/source/.env" ]]; then+ set -a+ # shellcheck disable=SC1091+ source "${HOME}/.brainapi/source/.env"+ set +a+ export SEARCH_ENABLED="${SEARCH_ENABLED:-true}"+ export DATA_DB="${DATA_DB:-postgresql}"+ export VECTOR_DB="${VECTOR_DB:-postgresql}"+ export GRAPH_DB="${GRAPH_DB:-networkx}"+ export ENV="${ENV:-development}"+ fi+ break+ fi+done+export PYTHONPATH="${ROOT}${PYTHONPATH:+:$PYTHONPATH}"+exec "$PYTHON" -m search "$@"diff --git a/benchmarks/search/__init__.py b/benchmarks/search/__init__.pynew file mode 100644index 0000000..8d8cd25--- /dev/null+++ b/benchmarks/search/__init__.py@@ -0,0 +1 @@+"""Search eval harness against searchbench* brains."""diff --git a/benchmarks/search/__main__.py b/benchmarks/search/__main__.pynew file mode 100644index 0000000..5f6e430--- /dev/null+++ b/benchmarks/search/__main__.py@@ -0,0 +1,4 @@+from search.cli import main++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/search/backfill_entities.py b/benchmarks/search/backfill_entities.pynew file mode 100644index 0000000..3c159d5--- /dev/null+++ b/benchmarks/search/backfill_entities.py@@ -0,0 +1,96 @@+from __future__ import annotations++from typing import Any, Callable++from search.config import FROZEN_STRUCTURED_BRAINS, validate_brain_id+from search.mapping import catalog_entity_backfill_rows, entity_uuid+++def refuse_entity_backfill(brain_id: str) -> str:+ bid = validate_brain_id(brain_id)+ if bid in FROZEN_STRUCTURED_BRAINS:+ raise SystemExit(+ f"Refusing ENTITY text backfill on frozen brain {bid}. "+ "Use searchbenchwandsgraph only (never wipe frozen structured brains)."+ )+ return bid+++def backfill_rows_from_docs(docs: list[dict[str, Any]]) -> list[dict[str, str]]:+ return catalog_entity_backfill_rows(docs)+++def apply_entity_text_backfill(+ *,+ brain_id: str,+ rows: list[dict[str, str]],+ graph: Any,+ embeddings: Any,+ vector_store: Any,+ is_item: Callable[[str, list[str] | None], bool] | None = None,+) -> dict[str, Any]:+ bid = refuse_entity_backfill(brain_id)+ if is_item is None:+ from src.core.search.graph_channels import is_item_entity as is_item+ from src.constants.embeddings import Vector+ from src.core.agents.scout_agent import ScoutEntity+ from src.core.search.catalog_graph import node_embed_text++ updated = 0+ missing = 0+ skipped = 0+ for row in rows:+ uuid = str(row.get("uuid") or entity_uuid(str(row.get("doc_id") or ""))).strip()+ blob = str(row.get("search_text") or "").strip()+ if not uuid or not blob:+ skipped += 1+ continue+ node = graph.get_by_uuid(uuid, bid)+ if node is None:+ missing += 1+ continue+ labels = list(getattr(node, "labels", None) or [])+ if not is_item(uuid, labels):+ skipped += 1+ continue+ props = dict(getattr(node, "properties", None) or {})+ props["search_text"] = blob+ graph.update_node(+ uuid,+ bid,+ new_properties={"search_text": blob},+ )+ scout = ScoutEntity(+ uuid=uuid,+ name=str(getattr(node, "name", "") or row.get("name") or uuid),+ type="ENTITY",+ properties=props,+ )+ text, _ = node_embed_text(scout)+ vector = embeddings.embed_text(text)+ if not isinstance(vector, Vector):+ skipped += 1+ continue+ existing = props.get("v_id")+ if existing:+ vector.id = str(existing)+ vector.metadata = {+ "labels": labels or ["ENTITY"],+ "name": scout.name,+ "uuid": uuid,+ }+ v_ids = vector_store.add_vectors([vector], store="nodes", brain_id=bid)+ if v_ids:+ graph.update_node(+ uuid,+ bid,+ new_properties={"search_text": blob, "v_id": v_ids[0]},+ )+ updated += 1+ return {+ "brain_id": bid,+ "n_rows": len(rows),+ "updated": updated,+ "missing": missing,+ "skipped": skipped,+ }diff --git a/benchmarks/search/catalog.py b/benchmarks/search/catalog.pynew file mode 100644index 0000000..43340ba--- /dev/null+++ b/benchmarks/search/catalog.py@@ -0,0 +1,743 @@+from __future__ import annotations++import csv+from collections import defaultdict+from pathlib import Path+from typing import Any, Iterable, Iterator+from urllib.request import urlretrieve++from search.config import DATA_DIR+from search.dataset import DOC_MARKER_PREFIX, write_records++ESCI_NAME = "esci"+WANDS_NAME = "wands"+JDSEARCH_NAME = "jdsearch"+SUPPORTED_CATALOGS = (ESCI_NAME, WANDS_NAME, JDSEARCH_NAME)+DOWNLOAD_ALL_CATALOGS = (ESCI_NAME, WANDS_NAME)+ESCI_LOCALES = ("us", "es", "jp")++ESCI_JSONL = DATA_DIR / "search_esci.jsonl"+WANDS_JSONL = DATA_DIR / "search_wands.jsonl"+JDSEARCH_JSONL = DATA_DIR / "search_jdsearch.jsonl"+ESCI_CACHE_DIR = DATA_DIR / "esci"+WANDS_CACHE_DIR = DATA_DIR / "wands"++ESCI_GITHUB_BASE = (+ "https://github.com/amazon-science/esci-data/raw/main/"+ "shopping_queries_dataset"+)+ESCI_HF_REPO = "khanmu2003/amazon-shopping-queries-dataset"+ESCI_EXAMPLES_FILE = "shopping_queries_dataset_examples.parquet"+ESCI_PRODUCTS_FILE = "shopping_queries_dataset_products.parquet"++WANDS_GITHUB_BASE = "https://raw.githubusercontent.com/wayfair/WANDS/main/dataset"++ESCI_GAINS = {"E": 1.0, "S": 0.1, "C": 0.01, "I": 0.0}+WANDS_GAINS = {"Exact": 1.0, "Partial": 0.5, "Irrelevant": 0.0}++DEFAULT_MAX_QUERIES = 80+DEFAULT_MAX_DOCS = 2000+DEFAULT_CANDIDATES_PER_QUERY = 40+PRODUCT_TEXT_MAX_CHARS = 4000+FROZEN_JSONL_ALWAYS = ("search_esci_74.jsonl",)+FROZEN_JSONL_IF_EXISTS = ("search_wands.jsonl", "search_jdsearch.jsonl")+++def normalize_esci_locale(locale: str) -> str:+ loc = (locale or "us").strip().lower()+ if loc not in ESCI_LOCALES:+ raise ValueError(+ f"Unknown ESCI locale {locale!r}. Supported: {ESCI_LOCALES}. "+ "ESCI has no Italian (it) split."+ )+ return loc+++def catalog_overwrite_blocked(path: Path) -> bool:+ resolved = path.resolve()+ for name in FROZEN_JSONL_ALWAYS:+ if resolved == (DATA_DIR / name).resolve():+ return True+ for name in FROZEN_JSONL_IF_EXISTS:+ frozen = (DATA_DIR / name).resolve()+ if resolved == frozen and frozen.exists() and frozen.stat().st_size > 0:+ return True+ return False+++def catalog_jsonl_path(name: str, *, locale: str = "us") -> Path:+ key = name.strip().lower()+ if key == ESCI_NAME:+ loc = normalize_esci_locale(locale)+ if loc == "us":+ return ESCI_JSONL+ return DATA_DIR / f"search_esci_{loc}.jsonl"+ if key == WANDS_NAME:+ return WANDS_JSONL+ if key == JDSEARCH_NAME:+ return JDSEARCH_JSONL+ raise ValueError(f"Unknown catalog {name!r}. Supported: {SUPPORTED_CATALOGS}")+++def prepare_catalog(+ name: str,+ *,+ out_path: Path | None = None,+ force: bool = False,+ max_queries: int = DEFAULT_MAX_QUERIES,+ max_docs: int = DEFAULT_MAX_DOCS,+ candidates_per_query: int = DEFAULT_CANDIDATES_PER_QUERY,+ locale: str = "us",+ split: str = "test",+ holdout_qids: set[str] | None = None,+) -> Path:+ key = name.strip().lower()+ dest = out_path or catalog_jsonl_path(key, locale=locale)+ if dest.exists() and dest.stat().st_size > 0 and not force:+ return dest+ if key == ESCI_NAME:+ rows = prepare_esci_rows(+ max_queries=max_queries,+ max_docs=max_docs,+ candidates_per_query=candidates_per_query,+ locale=locale,+ split=split,+ force=force,+ holdout_qids=holdout_qids,+ )+ elif key == WANDS_NAME:+ rows = prepare_wands_rows(+ max_queries=max_queries,+ max_docs=max_docs,+ candidates_per_query=candidates_per_query,+ force=force,+ )+ elif key == JDSEARCH_NAME:+ from search.jdsearch import prepare_jdsearch_bundle++ return prepare_jdsearch_bundle(+ dest,+ max_queries=max_queries,+ max_docs=max_docs,+ candidates_per_query=candidates_per_query,+ force=force,+ )+ else:+ raise ValueError(f"Unknown catalog {name!r}. Supported: {SUPPORTED_CATALOGS}")+ return write_records(rows, dest)+++def prepare_esci_rows(+ *,+ max_queries: int = DEFAULT_MAX_QUERIES,+ max_docs: int = DEFAULT_MAX_DOCS,+ candidates_per_query: int = DEFAULT_CANDIDATES_PER_QUERY,+ locale: str = "us",+ split: str = "test",+ force: bool = False,+ holdout_qids: set[str] | None = None,+) -> list[dict[str, Any]]:+ locale = normalize_esci_locale(locale)+ examples_path, products_path = download_esci(force=force)+ selected, needed_ids = _select_catalog(+ _esci_judgments(examples_path, locale=locale, split=split),+ max_queries=max_queries,+ max_docs=max_docs,+ candidates_per_query=candidates_per_query,+ slice_name=f"esci-{locale}",+ dataset=ESCI_NAME,+ gains=ESCI_GAINS,+ label_key="esci_label",+ query_text_key="query",+ holdout_qids=holdout_qids,+ )+ products = _esci_products(products_path, needed_ids, locale=locale)+ return _docs_and_queries(+ selected,+ products=products,+ doc_ids=needed_ids,+ dataset=ESCI_NAME,+ title_key="product_title",+ description_key="product_description",+ extra_fields=extra_fields_from_catalog(+ products,+ title_key="product_title",+ description_key="product_description",+ ),+ )+++def prepare_wands_rows(+ *,+ max_queries: int = DEFAULT_MAX_QUERIES,+ max_docs: int = DEFAULT_MAX_DOCS,+ candidates_per_query: int = DEFAULT_CANDIDATES_PER_QUERY,+ force: bool = False,+) -> list[dict[str, Any]]:+ product_path, query_path, label_path = download_wands(force=force)+ queries = _wands_queries(query_path)+ selected, needed_ids = _select_catalog(+ _wands_judgments(label_path, queries),+ max_queries=max_queries,+ max_docs=max_docs,+ candidates_per_query=candidates_per_query,+ slice_name="wands",+ dataset=WANDS_NAME,+ gains=WANDS_GAINS,+ label_key="label",+ query_text_key="query",+ query_slice_key="query_class",+ )+ products = _wands_products(product_path, needed_ids)+ return _docs_and_queries(+ selected,+ products=products,+ doc_ids=needed_ids,+ dataset=WANDS_NAME,+ title_key="product_name",+ description_key="product_description",+ extra_fields=(+ ("Class", "product_class"),+ ("Category", "category_hierarchy"),+ ("Hierarchy", "category_hierarchy"),+ ("Features", "product_features"),+ ),+ )+++def download_esci(*, force: bool = False) -> tuple[Path, Path]:+ ESCI_CACHE_DIR.mkdir(parents=True, exist_ok=True)+ examples = ESCI_CACHE_DIR / ESCI_EXAMPLES_FILE+ products = ESCI_CACHE_DIR / ESCI_PRODUCTS_FILE+ _ensure_file(+ examples,+ url=f"{ESCI_GITHUB_BASE}/{ESCI_EXAMPLES_FILE}",+ hf_filename=ESCI_EXAMPLES_FILE,+ force=force,+ )+ _ensure_file(+ products,+ url=f"{ESCI_GITHUB_BASE}/{ESCI_PRODUCTS_FILE}",+ hf_filename=ESCI_PRODUCTS_FILE,+ force=force,+ )+ return examples, products+++def download_wands(*, force: bool = False) -> tuple[Path, Path, Path]:+ WANDS_CACHE_DIR.mkdir(parents=True, exist_ok=True)+ product = WANDS_CACHE_DIR / "product.csv"+ query = WANDS_CACHE_DIR / "query.csv"+ label = WANDS_CACHE_DIR / "label.csv"+ for path, name in (+ (product, "product.csv"),+ (query, "query.csv"),+ (label, "label.csv"),+ ):+ _download_url(+ f"{WANDS_GITHUB_BASE}/{name}",+ path,+ force=force,+ )+ return product, query, label+++def _stringify_catalog_value(value: Any) -> str:+ if value is None:+ return ""+ if isinstance(value, str):+ return value.strip()+ if isinstance(value, (list, tuple)):+ parts = [_stringify_catalog_value(item) for item in value]+ return " ".join(part for part in parts if part)+ if isinstance(value, dict):+ parts = []+ for key, item in value.items():+ text = _stringify_catalog_value(item)+ if text:+ parts.append(f"{key}: {text}")+ return "; ".join(parts)+ return str(value).strip()+++def extra_fields_from_catalog(+ products: dict[str, dict[str, Any]],+ *,+ title_key: str,+ description_key: str,+ skip: tuple[str, ...] = ("product_id",),+) -> tuple[tuple[str, str], ...]:+ keys: set[str] = set()+ for row in products.values():+ keys.update(str(key) for key in row.keys())+ extras: list[tuple[str, str]] = []+ skipped = {title_key, description_key, *skip}+ for key in sorted(keys):+ if key in skipped:+ continue+ label = key.replace("product_", "").replace("_", " ").title()+ extras.append((label, key))+ return tuple(extras)+++def format_product_text(+ doc_id: str,+ *,+ title: str = "",+ description: str = "",+ extras: Iterable[tuple[str, str]] = (),+ max_chars: int = PRODUCT_TEXT_MAX_CHARS,+) -> str:+ lines = [f"{DOC_MARKER_PREFIX}{doc_id}."]+ title = (title or "").strip()+ if title:+ lines.append(f"Title: {title}")+ for label, value in extras:+ text = (value or "").strip()+ if text:+ lines.append(f"{label}: {text}")+ description = (description or "").strip()+ if description:+ lines.append(f"Description: {description}")+ body = "\n".join(lines).strip()+ if len(body) > max_chars:+ return body[: max_chars - 1].rstrip() + "…"+ return body+++def _holdout_ids(holdout_qids: Iterable[str] | None) -> set[str]:+ out: set[str] = set()+ for raw in holdout_qids or []:+ qid = str(raw or "").strip()+ if not qid:+ continue+ out.add(qid)+ lower = qid.lower()+ if lower.startswith("esci-"):+ out.add(qid[5:])+ else:+ out.add(f"esci-{qid}")+ return out+++def _select_catalog(+ judgments: list[dict[str, Any]],+ *,+ max_queries: int,+ max_docs: int,+ candidates_per_query: int,+ slice_name: str,+ dataset: str,+ gains: dict[str, float],+ label_key: str,+ query_text_key: str,+ query_slice_key: str | None = None,+ holdout_qids: Iterable[str] | None = None,+) -> tuple[list[dict[str, Any]], set[str]]:+ by_query: dict[str, list[dict[str, Any]]] = defaultdict(list)+ query_meta: dict[str, dict[str, Any]] = {}+ for row in judgments:+ qid = str(row.get("query_id") or "").strip()+ pid = str(row.get("product_id") or "").strip()+ if not qid or not pid:+ continue+ by_query[qid].append(row)+ if qid not in query_meta:+ query_meta[qid] = row++ blocked = _holdout_ids(holdout_qids)+ eligible = [+ qid+ for qid in sorted(by_query, key=_sort_key)+ if qid not in blocked and f"esci-{qid}" not in blocked+ ]+ selected_qids = eligible[: max(1, max_queries)]+ needed_ids: set[str] = set()+ queries: list[dict[str, Any]] = []++ for qid in selected_qids:+ ranked = _rank_candidates(by_query[qid], gains, label_key)+ if candidates_per_query > 0:+ ranked = ranked[:candidates_per_query]+ gold_grades: dict[str, float] = {}+ gold_doc_ids: list[str] = []+ candidate_doc_ids: list[str] = []+ candidate_grades: dict[str, float] = {}+ for item in ranked:+ pid = str(item["product_id"])+ label = str(item.get(label_key) or "")+ gain = float(gains.get(label, 0.0))+ candidate_doc_ids.append(pid)+ candidate_grades[pid] = gain+ if pid not in needed_ids and len(needed_ids) >= max_docs:+ continue+ needed_ids.add(pid)+ if gain > 0:+ gold_grades[pid] = gain+ gold_doc_ids.append(pid)+ if not gold_doc_ids:+ continue+ meta = query_meta[qid]+ slice_value = slice_name+ if query_slice_key:+ slice_value = str(meta.get(query_slice_key) or slice_name)+ queries.append(+ {+ "type": "query",+ "qid": f"{dataset}-{qid}",+ "query": str(meta.get(query_text_key) or "").strip(),+ "gold_doc_ids": gold_doc_ids,+ "gold_grades": gold_grades,+ "candidate_doc_ids": candidate_doc_ids,+ "candidate_grades": candidate_grades,+ "slice": slice_value,+ }+ )+ return queries, needed_ids+++def _qid_to_source_id(qid: str, dataset: str) -> str:+ text = str(qid or "").strip()+ prefix = f"{dataset}-"+ if text.lower().startswith(prefix.lower()):+ return text[len(prefix) :]+ return text+++def attach_esci_candidate_pools(+ rows: list[dict[str, Any]],+ *,+ locale: str = "us",+ split: str = "test",+ candidates_per_query: int = DEFAULT_CANDIDATES_PER_QUERY,+) -> list[dict[str, Any]]:+ from search.dataset import split_corpus++ locale = normalize_esci_locale(locale)+ docs, queries = split_corpus(rows)+ if not queries:+ return rows+ wanted = {+ _qid_to_source_id(str(query.get("qid") or ""), ESCI_NAME)+ for query in queries+ }+ wanted.discard("")+ examples_path, products_path = download_esci(force=False)+ judgments = [+ row+ for row in _esci_judgments(examples_path, locale=locale, split=split)+ if str(row.get("query_id") or "").strip() in wanted+ ]+ selected, needed_ids = _select_catalog(+ judgments,+ max_queries=max(1, len(wanted)),+ max_docs=max(len(needed_ids_from_docs(docs)) + 4000, 1),+ candidates_per_query=candidates_per_query,+ slice_name=f"esci-{locale}",+ dataset=ESCI_NAME,+ gains=ESCI_GAINS,+ label_key="esci_label",+ query_text_key="query",+ )+ by_source = {+ _qid_to_source_id(str(row.get("qid") or ""), ESCI_NAME): row+ for row in selected+ }+ existing_ids = {str(doc.get("doc_id") or "") for doc in docs}+ missing = needed_ids - existing_ids+ for query in queries:+ source = _qid_to_source_id(str(query.get("qid") or ""), ESCI_NAME)+ attached = by_source.get(source)+ if not attached:+ query.setdefault("candidate_doc_ids", list(query.get("gold_doc_ids") or []))+ query.setdefault(+ "candidate_grades",+ dict(query.get("gold_grades") or {}),+ )+ continue+ query["candidate_doc_ids"] = list(attached.get("candidate_doc_ids") or [])+ query["candidate_grades"] = dict(attached.get("candidate_grades") or {})+ missing.update(set(query["candidate_doc_ids"]) - existing_ids)+ if not missing:+ return docs + queries+ products = _esci_products(products_path, missing, locale=locale)+ extra = _docs_and_queries(+ [],+ products=products,+ doc_ids=missing,+ dataset=ESCI_NAME,+ title_key="product_title",+ description_key="product_description",+ extra_fields=extra_fields_from_catalog(+ products,+ title_key="product_title",+ description_key="product_description",+ ),+ )+ extra_docs = [row for row in extra if row.get("type") == "doc"]+ return docs + extra_docs + queries+++def needed_ids_from_docs(docs: list[dict[str, Any]]) -> set[str]:+ return {str(doc.get("doc_id") or "") for doc in docs if doc.get("doc_id")}+++def _docs_and_queries(+ queries: list[dict[str, Any]],+ *,+ products: dict[str, dict[str, Any]],+ doc_ids: set[str],+ dataset: str,+ title_key: str,+ description_key: str,+ extra_fields: tuple[tuple[str, str], ...] = (),+) -> list[dict[str, Any]]:+ docs: list[dict[str, Any]] = []+ field_aliases = {+ "brand": "brand",+ "color": "color",+ "class": "class",+ "category": "class",+ "hierarchy": "hierarchy",+ "category hierarchy": "hierarchy",+ "features": "features",+ "locale": "locale",+ "description": "description",+ "price": "price",+ "rating": "rating",+ }+ for pid in sorted(doc_ids, key=_sort_key):+ catalog = products.get(pid) or {}+ extras = [+ (label, _stringify_catalog_value(catalog.get(field)))+ for label, field in extra_fields+ ]+ title = _stringify_catalog_value(catalog.get(title_key)) or pid+ description = _stringify_catalog_value(catalog.get(description_key))+ row: dict[str, Any] = {+ "type": "doc",+ "doc_id": pid,+ "text": format_product_text(+ pid,+ title=title,+ description=description,+ extras=extras,+ ),+ "title": title,+ "description": description,+ "ingest": "chunks",+ "dataset": dataset,+ }+ for label, field in extra_fields:+ mapped = field_aliases.get(label.strip().lower())+ if not mapped:+ continue+ value = _stringify_catalog_value(catalog.get(field))+ if value and mapped not in row:+ row[mapped] = value+ docs.append(row)+ queries = [q for q in queries if q.get("query") and q.get("gold_doc_ids")]+ queries.sort(key=lambda row: str(row["qid"]))+ return docs + queries+++def _rank_candidates(+ rows: list[dict[str, Any]],+ gains: dict[str, float],+ label_key: str,+) -> list[dict[str, Any]]:+ def key(row: dict[str, Any]) -> tuple[float, str]:+ label = str(row.get(label_key) or "")+ return (-float(gains.get(label, 0.0)), str(row.get("product_id") or ""))++ return sorted(rows, key=key)+++def _esci_judgments(+ path: Path,+ *,+ locale: str,+ split: str,+) -> list[dict[str, Any]]:+ wanted_locale = locale.strip().lower()+ wanted_split = split.strip().lower()+ rows: list[dict[str, Any]] = []+ for item in _iter_parquet(path):+ if int(item.get("small_version") or 0) != 1:+ continue+ if str(item.get("product_locale") or "").strip().lower() != wanted_locale:+ continue+ if str(item.get("split") or "").strip().lower() != wanted_split:+ continue+ rows.append(+ {+ "query_id": item.get("query_id"),+ "query": item.get("query"),+ "product_id": item.get("product_id"),+ "esci_label": item.get("esci_label"),+ }+ )+ return rows+++def _esci_products(+ path: Path,+ needed_ids: set[str],+ *,+ locale: str = "us",+) -> dict[str, dict[str, Any]]:+ catalog: dict[str, dict[str, Any]] = {}+ if not needed_ids:+ return catalog+ wanted_locale = locale.strip().lower()+ for item in _iter_parquet(path):+ pid = str(item.get("product_id") or "").strip()+ if pid not in needed_ids or pid in catalog:+ continue+ item_locale = str(item.get("product_locale") or "").strip().lower()+ if item_locale and item_locale != wanted_locale:+ continue+ catalog[pid] = item+ if len(catalog) >= len(needed_ids):+ break+ return catalog+++def _wands_queries(path: Path) -> dict[str, dict[str, str]]:+ out: dict[str, dict[str, str]] = {}+ for row in _iter_tsv(path):+ qid = str(row.get("query_id") or "").strip()+ if not qid:+ continue+ out[qid] = {+ "query_id": qid,+ "query": str(row.get("query") or "").strip(),+ "query_class": str(row.get("query_class") or "wands").strip()+ or "wands",+ }+ return out+++def _wands_products(path: Path, needed_ids: set[str]) -> dict[str, dict[str, Any]]:+ catalog: dict[str, dict[str, Any]] = {}+ if not needed_ids:+ return catalog+ for row in _iter_tsv(path):+ pid = str(row.get("product_id") or "").strip()+ if pid not in needed_ids:+ continue+ catalog[pid] = row+ if len(catalog) >= len(needed_ids):+ break+ return catalog+++def _wands_judgments(+ path: Path,+ queries: dict[str, dict[str, str]],+) -> list[dict[str, Any]]:+ rows: list[dict[str, Any]] = []+ for row in _iter_tsv(path):+ qid = str(row.get("query_id") or "").strip()+ pid = str(row.get("product_id") or "").strip()+ if not qid or not pid:+ continue+ meta = queries.get(qid) or {}+ rows.append(+ {+ "query_id": qid,+ "query": meta.get("query") or "",+ "query_class": meta.get("query_class") or "wands",+ "product_id": pid,+ "label": str(row.get("label") or "").strip(),+ }+ )+ return rows+++def _iter_tsv(path: Path) -> Iterator[dict[str, str]]:+ with path.open("r", encoding="utf-8", newline="") as fh:+ reader = csv.DictReader(fh, delimiter="\t")+ for row in reader:+ if not row:+ continue+ yield {str(k): ("" if v is None else str(v)) for k, v in row.items()}+++def _iter_parquet(path: Path) -> Iterator[dict[str, Any]]:+ try:+ import pyarrow.parquet as pq+ except ImportError as exc:+ raise SystemExit(+ "pyarrow is required to read ESCI parquet files. "+ "Install benchmarks/requirements.txt."+ ) from exc+ parquet = pq.ParquetFile(path)+ for batch in parquet.iter_batches(batch_size=8192):+ for row in batch.to_pylist():+ yield row+++def _ensure_file(+ dest: Path,+ *,+ url: str,+ hf_filename: str,+ force: bool,+) -> Path:+ if dest.exists() and dest.stat().st_size > 64 and not force:+ if not _is_git_lfs_pointer(dest):+ return dest+ try:+ _download_url(url, dest, force=True)+ if dest.exists() and dest.stat().st_size > 64 and not _is_git_lfs_pointer(dest):+ return dest+ except Exception:+ pass+ return _download_hf(hf_filename, dest, force=True)+++def _download_hf(filename: str, dest: Path, *, force: bool) -> Path:+ try:+ from huggingface_hub import hf_hub_download+ except ImportError as exc:+ raise SystemExit(+ "huggingface_hub is required to download ESCI. "+ "Install benchmarks/requirements.txt."+ ) from exc+ cached = hf_hub_download(+ repo_id=ESCI_HF_REPO,+ filename=filename,+ repo_type="dataset",+ force_download=force,+ )+ src = Path(cached)+ dest.parent.mkdir(parents=True, exist_ok=True)+ if src.resolve() != dest.resolve():+ dest.write_bytes(src.read_bytes())+ return dest+++def _download_url(url: str, dest: Path, *, force: bool) -> Path:+ dest.parent.mkdir(parents=True, exist_ok=True)+ if dest.exists() and dest.stat().st_size > 0 and not force:+ return dest+ tmp = dest.with_suffix(dest.suffix + ".tmp")+ urlretrieve(url, tmp)+ tmp.replace(dest)+ return dest+++def _is_git_lfs_pointer(path: Path) -> bool:+ try:+ head = path.read_bytes()[:80]+ except OSError:+ return False+ return head.startswith(b"version https://git-lfs.github.com/spec/v1")+++def _sort_key(value: str) -> tuple[int, str | int]:+ text = str(value)+ if text.isdigit():+ return (0, int(text))+ return (1, text)diff --git a/benchmarks/search/cli.py b/benchmarks/search/cli.pynew file mode 100644index 0000000..8ae12d1--- /dev/null+++ b/benchmarks/search/cli.py@@ -0,0 +1,1563 @@+from __future__ import annotations++import argparse+import json+from pathlib import Path++from rich.console import Console+from rich.table import Table++from search.catalog import (+ DOWNLOAD_ALL_CATALOGS,+ SUPPORTED_CATALOGS,+ catalog_jsonl_path,+ catalog_overwrite_blocked,+ prepare_catalog,+)+from search.client import BrainAPIClient, SearchDisabledError+from search.config import BENCHMARKS_ROOT, DATA_DIR, DEFAULT_BRAIN_ID, Settings, validate_brain_id+from search.dataset import dataset_stats, load_records, split_corpus+from search.evaluate import assert_wandsgraph_node_join, ensure_run_dir, evaluate_search+from search.mapping import load_interaction_rows+from search.report import print_report_table, write_report++console = Console()+++def _parse_channels(args: argparse.Namespace) -> list[str] | None:+ raw = getattr(args, "channels", None)+ if not raw:+ return None+ items = [item.strip() for item in str(raw).split(",") if item.strip()]+ return items or None+++def _parse_label_list(raw: str | None) -> list[str] | None:+ if not raw:+ return None+ items = [item.strip() for item in str(raw).split(",") if item.strip()]+ return items or None+++def _parse_extras(raw: str | None) -> dict[str, str] | None:+ if not raw:+ return None+ try:+ data = json.loads(raw)+ except json.JSONDecodeError as exc:+ raise SystemExit(f"--extras must be a JSON object: {exc}") from exc+ if not isinstance(data, dict):+ raise SystemExit("--extras must be a JSON object")+ return {str(key): str(value) for key, value in data.items() if value is not None} or None+++def _resolve_dataset_path(args: argparse.Namespace, settings: Settings) -> Path:+ if getattr(args, "dataset", None):+ return Path(args.dataset)+ return settings.dataset_path+++def _skip_enrichment(args: argparse.Namespace) -> bool:+ return not bool(getattr(args, "enrich", False))+++def _eval_kwargs(args: argparse.Namespace) -> dict:+ interactions = None+ path = getattr(args, "interactions", None)+ if path:+ interactions = load_interaction_rows(Path(path))+ return {+ "fusion": args.fusion,+ "fusion_alpha": getattr(args, "fusion_alpha", None),+ "rerank": args.rerank,+ "mode": getattr(args, "mode", None) or "default",+ "channels": _parse_channels(args),+ "node_labels": _parse_label_list(getattr(args, "node_labels", None)),+ "community_labels": _parse_label_list(getattr(args, "community_labels", None)),+ "expand": getattr(args, "expand", None) or "none",+ "skip_enrichment": _skip_enrichment(args),+ "ingest_graph": bool(getattr(args, "ingest_graph", False)),+ "skip_ingest": bool(getattr(args, "skip_ingest", False)),+ "rank_pool": bool(getattr(args, "rank_pool", False)),+ "personalize": bool(getattr(args, "personalize", False)),+ "interactions": interactions,+ "extras": _parse_extras(getattr(args, "extras", None)),+ }+++def cmd_download(args: argparse.Namespace, settings: Settings) -> int:+ from search.config import DATA_DIR+ from search.finetune_esci_4class import held_out_query_ids++ names = [item.strip().lower() for item in str(args.name).split(",") if item.strip()]+ if len(names) == 1 and names[0] in {"all", "*"}:+ names = list(DOWNLOAD_ALL_CATALOGS)+ unknown = [name for name in names if name not in SUPPORTED_CATALOGS]+ if unknown:+ console.print(+ f"[red]Unknown dataset[/red] {unknown!r}. "+ f"Supported: {', '.join(SUPPORTED_CATALOGS)}"+ )+ return 1+ holdout: set[str] = set()+ holdout_raw = str(getattr(args, "holdout_dataset", None) or "").strip()+ if holdout_raw:+ holdout_path = Path(holdout_raw)+ if not holdout_path.is_absolute():+ holdout_path = (BENCHMARKS_ROOT / holdout_path).resolve()+ if not holdout_path.exists():+ console.print(f"[red]Missing holdout dataset[/red] {holdout_path}")+ return 1+ holdout = held_out_query_ids(holdout_path)+ console.print(f"[cyan]Holdout[/cyan] {holdout_path.name} n={len(holdout)}")+ protected = {+ (DATA_DIR / "search_esci.jsonl").resolve(),+ (DATA_DIR / "search_esci_74.jsonl").resolve(),+ }+ timeout = 0+ for name in names:+ try:+ default_out = catalog_jsonl_path(name, locale=args.locale)+ except ValueError as exc:+ console.print(f"[red]{exc}[/red]")+ return 1+ out = Path(args.out) if args.out and len(names) == 1 else default_out+ if not out.is_absolute():+ out = (BENCHMARKS_ROOT / out).resolve()+ split = str(args.split or "test")+ if getattr(args, "dry_stats", False):+ if name != "jdsearch":+ console.print("[red]--dry-stats is only implemented for --name jdsearch[/red]")+ return 1+ from search.jdsearch import collect_jdsearch_stats, print_jdsearch_stats++ console.print(f"[cyan]JDsearch stats[/cyan] (no JSONL write)")+ stats = collect_jdsearch_stats()+ print_jdsearch_stats(stats)+ timeout += 1+ continue+ if catalog_overwrite_blocked(out):+ console.print(f"[red]Refusing to overwrite[/red] {out}")+ return 1+ if split == "train" and out.resolve() in protected:+ console.print(+ f"[red]Refusing to overwrite[/red] {out} with split=train. "+ "Pass --out data/search_esci_ltr200.jsonl"+ )+ return 1+ console.print(f"[cyan]Downloading[/cyan] {name} split={split} → {out}")+ path = prepare_catalog(+ name,+ out_path=out,+ force=args.force,+ max_queries=args.max_queries,+ max_docs=args.max_docs,+ candidates_per_query=args.candidates_per_query,+ locale=args.locale,+ split=split,+ holdout_qids=holdout or None,+ )+ rows = load_records(path)+ stats = dataset_stats(rows)+ console.print(f"[green]Ready[/green] {path}")+ console.print({key: value for key, value in stats.items() if key != "doc_ids"})+ timeout += 1+ return 0 if timeout else 1+++def cmd_dataset_stats(args: argparse.Namespace, settings: Settings) -> int:+ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ rows = load_records(path)+ stats = dataset_stats(rows)+ table = Table(title=f"Search dataset — {path.name}")+ table.add_column("field")+ table.add_column("value")+ table.add_row("n_docs", str(stats["n_docs"]))+ table.add_row("n_queries", str(stats["n_queries"]))+ table.add_row("slices", json.dumps(stats["slices"]))+ table.add_row("graded", str(stats.get("graded")))+ console.print(table)+ return 0+++def cmd_smoke(args: argparse.Namespace, settings: Settings) -> int:+ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ rows = load_records(path)+ docs, queries = split_corpus(rows)+ skip_ingest = bool(getattr(args, "skip_ingest", False))+ if skip_ingest and settings.brain_id == "searchbenchwandsgraph":+ queries = queries[: max(1, min(args.limit_queries, len(queries)))]+ else:+ docs = docs[: args.limit]+ queries = queries[: max(1, min(args.limit_queries, len(queries)))]+ limited = [{"type": "doc", **doc} for doc in docs] + [+ {"type": "query", **query} for query in queries+ ]+ console.print(+ f"[cyan]Smoke[/cyan] brain={settings.brain_id} "+ f"docs={len(docs)} queries={len(queries)}"+ )+ with BrainAPIClient(settings) as client:+ try:+ result = evaluate_search(+ client,+ limited,+ ks=(5, 10, 20),+ timeout_s=args.timeout,+ k=args.k,+ dataset_name=path.name,+ **_eval_kwargs(args),+ )+ except SearchDisabledError as exc:+ console.print(f"[red]{exc}[/red]")+ return 1+ if result.get("search_error"):+ console.print(f"[red]{result['search_error']}[/red]")+ return 1+ if result.get("ingest", {}).get("status") not in {"completed", "partial_failed"}:+ console.print("[red]Smoke ingest failed[/red]")+ console.print(result.get("ingest"))+ return 1+ if int(result.get("n_docs_mapped") or 0) < 1:+ console.print("[red]Smoke could not map DOCID markers to chunks[/red]")+ return 1+ if int(result.get("n_queries") or 0) < 1:+ console.print("[red]Smoke search returned no scored queries[/red]")+ return 1+ try:+ join = assert_wandsgraph_node_join(client, result)+ except SystemExit as exc:+ console.print(f"[red]{exc}[/red]")+ return 1+ if not join.get("skipped"):+ console.print(+ f"node_id={join.get('node_id')} neighbors={join.get('status_code')} "+ f"count={join.get('neighbor_count')}"+ )+ first = (result.get("queries") or [{}])[0]+ console.print(+ f"retrieve/search({first.get('qid')}) -> {first.get('n_hits')} hits "+ f"recall@10={((first.get('metrics') or {}).get('recall@10'))}"+ )+ console.print("[green]Smoke passed[/green]")+ return 0+++def cmd_backfill_entity_text(args: argparse.Namespace, settings: Settings) -> int:+ from search.backfill_entities import apply_entity_text_backfill, refuse_entity_backfill+ from search.mapping import catalog_entity_backfill_rows++ try:+ brain_id = refuse_entity_backfill(settings.brain_id)+ except SystemExit as exc:+ console.print(f"[red]{exc}[/red]")+ return 1+ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ docs, _ = split_corpus(load_records(path))+ limit = getattr(args, "limit", None)+ if limit:+ docs = docs[: int(limit)]+ rows = catalog_entity_backfill_rows(docs)+ console.print(+ f"[cyan]ENTITY backfill[/cyan] brain={brain_id} docs={len(rows)}"+ )+ import os+ from dotenv import load_dotenv++ for env_path in (+ Path.home() / ".brainapi" / "source" / ".env",+ Path(__file__).resolve().parents[2] / ".env",+ ):+ if env_path.exists():+ load_dotenv(env_path, override=False)+ os.environ.setdefault("DATA_DB", "postgresql")+ os.environ.setdefault("VECTOR_DB", "postgresql")+ os.environ.setdefault("GRAPH_DB", "networkx")+ os.environ.setdefault("SEARCH_ENABLED", "true")+ os.environ.setdefault("ENV", "development")+ from src.core.instances import (+ embeddings_adapter,+ graph_adapter,+ vector_store_adapter,+ )++ summary = apply_entity_text_backfill(+ brain_id=brain_id,+ rows=rows,+ graph=graph_adapter,+ embeddings=embeddings_adapter,+ vector_store=vector_store_adapter,+ )+ console.print(summary)+ if int(summary.get("updated") or 0) < 1:+ console.print("[red]No ENTITY nodes updated[/red]")+ return 1+ console.print("[green]ENTITY backfill done[/green]")+ return 0+++def cmd_evaluate(args: argparse.Namespace, settings: Settings) -> int:+ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ rows = load_records(path)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ ks = tuple(int(x) for x in args.ks.split(",") if x.strip()) or (5, 10, 20)+ console.print(+ f"[cyan]Evaluating[/cyan] brain={settings.brain_id} "+ f"dataset={path.name} run={run_id} fusion={args.fusion or 'rrf'}"+ )+ with BrainAPIClient(settings) as client:+ try:+ result = evaluate_search(+ client,+ rows,+ ks=ks,+ timeout_s=args.timeout,+ k=args.k,+ dataset_name=path.name,+ limit_docs=getattr(args, "limit_docs", None),+ limit_queries=getattr(args, "limit_queries", None),+ **_eval_kwargs(args),+ )+ except SearchDisabledError as exc:+ console.print(f"[red]{exc}[/red]")+ return 1+ report = write_report(run_dir, result)+ print_report_table(report)+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")+ if result.get("search_error"):+ console.print(f"[red]{result['search_error']}[/red]")+ return 0 if report.get("status") == "ok" else 1+++def cmd_report(args: argparse.Namespace, settings: Settings) -> int:+ run_dir = settings.runs_dir / args.run+ eval_path = run_dir / "eval.json"+ if not eval_path.exists():+ console.print(f"[red]Missing[/red] {eval_path}")+ return 1+ eval_result = json.loads(eval_path.read_text(encoding="utf-8"))+ report = write_report(run_dir, eval_result)+ print_report_table(report)+ return 0 if report.get("status") == "ok" else 1+++def cmd_finetune_ce(args: argparse.Namespace, settings: Settings) -> int:+ from search.finetune_esci_ce import finetune++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ out = Path(args.out)+ console.print(+ f"[cyan]Fine-tune CE[/cyan] dataset={path.name} out={out} "+ f"label_mode={args.label_mode} fields={args.fields}"+ )+ meta = finetune(+ jsonl_path=path,+ out_dir=out,+ max_pairs=int(args.max_pairs),+ epochs=int(args.epochs),+ batch_size=int(args.batch_size),+ label_mode=str(args.label_mode),+ fields=str(args.fields),+ base=str(args.base),+ lr=float(args.lr),+ max_length=int(getattr(args, "max_length", 256)),+ )+ console.print(json.dumps(meta, indent=2))+ return 0+++def cmd_miss_strata(args: argparse.Namespace, settings: Settings) -> int:+ from search.miss_strata import main_classify++ path = _resolve_dataset_path(args, settings)+ eval_path = settings.runs_dir / args.from_run / "eval.json"+ if not eval_path.exists():+ console.print(f"[red]Missing[/red] {eval_path}")+ return 1+ out = Path(args.out) if args.out else settings.runs_dir / args.from_run / "miss_strata.json"+ taxonomy = main_classify(eval_path=eval_path, dataset_path=path, out_path=out)+ table = Table(title=f"Miss strata {args.from_run}")+ table.add_column("field")+ table.add_column("value")+ for key in (+ "n_queries",+ "n_gold",+ "n_in_top10",+ "n_in_11_to_k",+ "n_missed",+ "stratum_counts",+ "n_total_miss",+ "n_rewritable_total_miss",+ "run_query_side",+ "rewritable_qids",+ "total_miss_qids",+ ):+ table.add_row(key, str(taxonomy.get(key)))+ console.print(table)+ console.print(f"[green]Wrote[/green] {out}")+ return 0+++def cmd_query_rewrite(args: argparse.Namespace, settings: Settings) -> int:+ from search.miss_strata import main_classify, write_rewritten_jsonl++ path = _resolve_dataset_path(args, settings)+ eval_path = settings.runs_dir / args.from_run / "eval.json"+ if not eval_path.exists():+ console.print(f"[red]Missing[/red] {eval_path}")+ return 1+ taxonomy = main_classify(eval_path=eval_path, dataset_path=path)+ if not taxonomy.get("run_query_side"):+ console.print(+ "[yellow]No rewritable total-miss qids (skip Task 2).[/yellow] "+ f"total_miss={taxonomy.get('total_miss_qids')}"+ )+ return 0+ dest = Path(args.out) if args.out else path.with_name("search_esci_74_qrewrite.jsonl")+ rows = load_records(path)+ write_rewritten_jsonl(rows, taxonomy, dest)+ console.print(+ f"[green]Wrote[/green] {dest} qids={taxonomy.get('rewritable_qids')}"+ )+ return 0+++def cmd_spell_normalize(args: argparse.Namespace, settings: Settings) -> int:+ from search.miss_strata import write_spell_jsonl++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ dest = Path(args.out) if args.out else path.with_name("search_esci_es_spell.jsonl")+ rows = load_records(path)+ summary = write_spell_jsonl(rows, dest)+ console.print(+ f"[green]Wrote[/green] {dest} changed_queries={summary.get('n_queries_changed')}"+ )+ return 0+++def cmd_finetune_dense(args: argparse.Namespace, settings: Settings) -> int:+ from search.finetune_esci_dense import finetune_dense++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ eval_path = Path(args.from_eval) if args.from_eval else None+ out = Path(args.out)+ console.print(+ f"[cyan]Fine-tune dense ANCE[/cyan] dataset={path.name} out={out} "+ f"(not Reddy 0.857)"+ )+ meta = finetune_dense(+ jsonl_path=path,+ out_dir=out,+ eval_path=eval_path,+ max_pairs=int(args.max_pairs),+ epochs=int(args.epochs),+ batch_size=int(args.batch_size),+ fields=str(args.fields),+ base=str(args.base),+ lr=float(args.lr),+ max_length=int(args.max_length),+ )+ console.print(json.dumps(meta, indent=2))+ return 0+++def cmd_local_dense(args: argparse.Namespace, settings: Settings) -> int:+ from search.local_dense import run_local_dense++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ rows = load_records(path)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)+ model_name = str(args.model)+ protocol = None+ if "bge" in model_name.lower():+ protocol = (+ "local BGE-base dual-encoder over search_esci_74.jsonl; "+ "not MiniLM ANCE; no query: prefix; not Reddy 0.857"+ )+ if "paraphrase-multilingual-minilm" in model_name.lower() or "italian_smoke" in path.name:+ protocol = (+ "local paraphrase-multilingual-MiniLM-L12-v2 over "+ "search_italian_smoke.jsonl; pipeline only (hits returned); "+ "not text-embedding-3-large quality; not Reddy; not ES n=62"+ )+ console.print(+ f"[cyan]Local dense retrieve[/cyan] dataset={path.name} run={run_id} "+ f"model={model_name} brain={settings.brain_id}"+ )+ result = run_local_dense(+ rows,+ model_name=str(args.model),+ dataset_name=path.name,+ k=int(args.k),+ ks=ks,+ brain_id=settings.brain_id,+ protocol=protocol,+ )+ report = write_report(run_dir, result)+ print_report_table(report)+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")+ return 0 if report.get("status") == "ok" else 1+++def cmd_mine_retrieved_lists(args: argparse.Namespace, settings: Settings) -> int:+ from search.mine_retrieved_lists import mine++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ out = Path(args.out)+ console.print(+ f"[cyan]Mine retrieved lists[/cyan] dataset={path.name} out={out} "+ f"(unlabeled=I; hold out test qids)"+ )+ meta = mine(+ jsonl_path=path,+ out_path=out,+ max_queries=int(args.max_queries),+ k=int(args.k),+ seed=int(args.seed),+ )+ console.print(json.dumps(meta, indent=2))+ return 0+++def cmd_export_hybrid_lists(args: argparse.Namespace, settings: Settings) -> int:+ from search.export_hybrid_lists import PROTECTED_OUT_NAMES, export_hybrid_lists+ from search.list_overlap import load_eval_run++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ holdout_raw = str(getattr(args, "holdout_dataset", None) or "").strip()+ holdout = Path(holdout_raw) if holdout_raw else DATA_DIR / "search_esci_74.jsonl"+ if not holdout.is_absolute():+ holdout = (BENCHMARKS_ROOT / holdout).resolve()+ if not holdout.exists():+ console.print(f"[red]Missing holdout[/red] {holdout}")+ return 1+ out = Path(args.out)+ if not out.is_absolute():+ out = (BENCHMARKS_ROOT / out).resolve()+ if out.name in PROTECTED_OUT_NAMES:+ console.print(f"[red]Refusing to overwrite[/red] {out}")+ return 1+ run_dir = settings.runs_dir / str(args.from_run)+ if not (run_dir / "eval.json").exists():+ console.print(f"[red]Missing eval[/red] {run_dir / 'eval.json'}")+ return 1+ eval_result = load_eval_run(run_dir)+ console.print(+ f"[cyan]Export hybrid lists[/cyan] from={args.from_run} "+ f"dataset={path.name} holdout={holdout.name} out={out} "+ f"(unlabeled=I; source=hybrid-k50)"+ )+ try:+ meta = export_hybrid_lists(+ eval_result=eval_result,+ dataset_path=path,+ holdout_path=holdout,+ out_path=out,+ k=int(args.k),+ )+ except ValueError as exc:+ console.print(f"[red]{exc}[/red]")+ return 1+ console.print(json.dumps(meta, indent=2))+ return 0+++def cmd_finetune_4class(args: argparse.Namespace, settings: Settings) -> int:+ from search.finetune_esci_4class import finetune++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ lists_path = Path(args.from_lists) if args.from_lists else None+ if lists_path is not None and not lists_path.exists():+ console.print(f"[red]Missing[/red] {lists_path}")+ return 1+ out = Path(args.out)+ if not out.is_absolute():+ out = (BENCHMARKS_ROOT / out).resolve()+ base_model = str(args.base)+ base_path = Path(base_model)+ if not base_path.is_absolute():+ maybe_base = (BENCHMARKS_ROOT / base_path).resolve()+ if maybe_base.exists():+ base_path = maybe_base+ base_model = str(base_path)+ if out.exists() and base_path.exists() and out.resolve() == base_path.resolve():+ console.print(f"[red]Refusing to overwrite base checkpoint[/red] {out}")+ return 1+ lists_source = str(getattr(args, "lists_source", None) or "").strip()+ if lists_path is not None:+ source = lists_source or "retrieved-bm25"+ else:+ source = "pool"+ console.print(+ f"[cyan]Fine-tune 4-class CE[/cyan] dataset={path.name} out={out} "+ f"source={source}"+ )+ meta = finetune(+ jsonl_path=path,+ out_dir=out,+ max_pairs=int(args.max_pairs),+ epochs=int(args.epochs),+ batch_size=int(args.batch_size),+ max_length=int(args.max_length),+ base_model=base_model,+ seed=int(args.seed),+ ckpt_every=int(args.ckpt_every),+ use_class_weights=bool(args.class_weights),+ lists_path=lists_path,+ lists_source=source if lists_path is not None else None,+ )+ console.print(json.dumps(meta, indent=2))+ return 0+++def cmd_colbert_local(args: argparse.Namespace, settings: Settings) -> int:+ from search.local_colbert import run_local_colbert++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ rows = load_records(path)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)+ console.print(+ f"[cyan]Local ColBERT MaxSim[/cyan] dataset={path.name} run={run_id} "+ f"(not fused with passages; not Reddy 0.857)"+ )+ result = run_local_colbert(+ rows,+ dataset_name=path.name,+ k=int(args.k),+ ks=ks,+ brain_id="harness-local-colbert",+ )+ report = write_report(run_dir, result)+ print_report_table(report)+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")+ return 0 if report.get("status") == "ok" else 1+++def _existing_path(path: Path) -> Path:+ if path.exists():+ return path+ from search.config import BENCHMARKS_ROOT++ alt = BENCHMARKS_ROOT / path+ return alt if alt.exists() else path+++def cmd_rank_corpus(args: argparse.Namespace, settings: Settings) -> int:+ from search.rank_corpus import DEFAULT_MODEL, load_predict, run_exhaustive_ce++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ model_dir = _existing_path(Path(args.model) if args.model else DEFAULT_MODEL)+ if not model_dir.exists():+ console.print(f"[red]Missing model[/red] {model_dir}")+ return 1+ rows = load_records(path)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)+ console.print(+ f"[cyan]Exhaustive 4-class CE[/cyan] dataset={path.name} run={run_id} "+ f"model={model_dir} (protocol exhaustive-catalog; not Reddy 0.857)"+ )+ predict, model_name = load_predict(model_dir, max_length=int(args.max_length))+ result = run_exhaustive_ce(+ rows,+ predict=predict,+ model_name=model_name,+ dataset_name=path.name,+ k=int(args.k),+ ks=ks,+ brain_id="harness-local-exhaustive",+ )+ report = write_report(run_dir, result)+ print_report_table(report)+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")+ return 0 if report.get("status") == "ok" else 1+++def cmd_list_overlap(args: argparse.Namespace, settings: Settings) -> int:+ from search.list_overlap import load_eval_run, summarize_overlap++ names = [item.strip() for item in str(args.against_runs).split(",") if item.strip()]+ passages = load_eval_run(settings.runs_dir / args.passages_run)+ passages["run_id"] = args.passages_run+ sidecars: dict = {}+ for name in names:+ eval_result = load_eval_run(settings.runs_dir / name)+ eval_result["run_id"] = name+ sidecars[name] = eval_result+ queries = None+ dataset = getattr(args, "dataset", None)+ if dataset:+ path = Path(dataset)+ if not path.exists():+ path = _existing_path(path)+ if path.exists():+ _, queries = split_corpus(load_records(path))+ summary = summarize_overlap(+ passages,+ sidecars,+ k=int(args.k),+ queries=queries,+ )+ run_id, run_dir = ensure_run_dir(settings, args.run)+ out = run_dir / "overlap.json"+ out.write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")+ headline = {+ name: {+ "unique_gold_hits": row["unique_gold_hits"],+ "queries_with_unique": row["queries_with_unique"],+ "n_queries": row["n_queries"],+ }+ for name, row in summary["runs"].items()+ }+ console.print(json.dumps(headline, indent=2))+ console.print(f"[green]Wrote[/green] {out}")+ return 0+++def cmd_union_lists(args: argparse.Namespace, settings: Settings) -> int:+ from search.list_overlap import load_eval_run, run_union++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ names = [item.strip() for item in str(args.from_runs).split(",") if item.strip()]+ evals = []+ for name in names:+ eval_result = load_eval_run(settings.runs_dir / name)+ eval_result["run_id"] = name+ evals.append(eval_result)+ rows = load_records(path)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)+ console.print(+ f"[cyan]Harness RRF union[/cyan] from={names} dataset={path.name} "+ f"run={run_id} (not live graph; not Reddy 0.857)"+ )+ result = run_union(+ evals,+ rows,+ dataset_name=path.name,+ k=int(args.k),+ ks=ks,+ brain_id="harness-local-union",+ run_names=names,+ )+ report = write_report(run_dir, result)+ print_report_table(report)+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")+ return 0 if report.get("status") == "ok" else 1+++def cmd_cascade_lists(args: argparse.Namespace, settings: Settings) -> int:+ from search.list_overlap import load_eval_run, run_cascade++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ passages = load_eval_run(settings.runs_dir / args.passages_run)+ passages["run_id"] = args.passages_run+ names = [item.strip() for item in str(args.from_runs).split(",") if item.strip()]+ sidecars = []+ for name in names:+ eval_result = load_eval_run(settings.runs_dir / name)+ eval_result["run_id"] = name+ sidecars.append(eval_result)+ rows = load_records(path)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)+ console.print(+ f"[cyan]Frozen-head cascade[/cyan] passages={args.passages_run} "+ f"sidecars={names} head_k={args.head_k} dataset={path.name} "+ f"run={run_id} (not live graph; not Reddy 0.857)"+ )+ result = run_cascade(+ passages,+ sidecars,+ rows,+ dataset_name=path.name,+ k=int(args.k),+ head_k=int(args.head_k),+ ks=ks,+ brain_id="harness-local-cascade",+ run_names=[args.passages_run, *names],+ )+ report = write_report(run_dir, result)+ print_report_table(report)+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")+ return 0 if report.get("status") == "ok" else 1+++def cmd_ltr_head(args: argparse.Namespace, settings: Settings) -> int:+ from search.list_overlap import load_eval_run+ from search.ltr_head import HEAD_RANKNET, PAIR_UNLABELED_ZERO, ce_cache_name, run_ltr_head++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ if not path.is_absolute():+ alt = (BENCHMARKS_ROOT / path).resolve()+ if alt.exists():+ path = alt+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ passages = load_eval_run(settings.runs_dir / args.from_run)+ passages["run_id"] = args.from_run+ rows = load_records(path)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)+ policy = str(getattr(args, "pair_policy", None) or PAIR_UNLABELED_ZERO)+ head = str(getattr(args, "ltr_model", None) or HEAD_RANKNET)+ ce_raw = str(getattr(args, "ce_model", None) or "").strip()+ ce_model = Path(ce_raw) if ce_raw else None+ if ce_model is not None and not ce_model.is_absolute():+ ce_model = (BENCHMARKS_ROOT / ce_model).resolve()+ if ce_model is not None and not ce_model.exists():+ console.print(f"[red]Missing CE model[/red] {ce_model}")+ return 1+ ce_cache = None+ if ce_model is not None:+ ce_cache = settings.runs_dir / args.from_run / ce_cache_name(ce_model)+ train_run = str(getattr(args, "train_from_run", None) or "").strip()+ train_eval = None+ train_rows = None+ train_ce_cache = None+ train_dataset_name = None+ if train_run:+ train_ds = str(getattr(args, "train_dataset", None) or "").strip()+ if not train_ds:+ console.print("[red]--train-dataset is required with --train-from-run[/red]")+ return 1+ train_path = Path(train_ds)+ if not train_path.is_absolute():+ train_path = (BENCHMARKS_ROOT / train_path).resolve()+ if not train_path.exists():+ console.print(f"[red]Missing train dataset[/red] {train_path}")+ return 1+ train_eval = load_eval_run(settings.runs_dir / train_run)+ train_eval["run_id"] = train_run+ train_rows = load_records(train_path)+ train_dataset_name = train_path.name+ if ce_model is not None:+ train_ce_cache = settings.runs_dir / train_run / ce_cache_name(ce_model)+ mode = "apply" if train_run else "CV"+ console.print(+ f"[cyan]LTR head {mode}[/cyan] from={args.from_run} dataset={path.name} "+ f"train={train_run or 'folds'} {train_dataset_name or ''} "+ f"run={run_id} folds={args.folds} pair={policy} head={head} "+ f"ce={ce_model or 'none'} (not live; not Reddy 0.857)"+ )+ result = run_ltr_head(+ passages,+ rows,+ dataset_name=path.name,+ k=int(args.k),+ ks=ks,+ n_folds=int(args.folds),+ brain_id="harness-local-ltr",+ source_run=args.from_run,+ pair_policy=policy,+ ce_model=ce_model,+ ce_cache_path=ce_cache,+ ltr_head=head,+ train_eval_result=train_eval,+ train_rows=train_rows,+ train_source_run=train_run or None,+ train_ce_cache_path=train_ce_cache,+ )+ report = write_report(run_dir, result)+ print_report_table(report)+ overlap = result.get("overlap_only_metrics") or {}+ console.print(+ f"overlap-only ndcg@10={overlap.get('ndcg@10')} "+ f"recall@10={overlap.get('recall@10')}"+ )+ weights = result.get("ltr_mean_weights") or {}+ if weights:+ kind = "gain importances" if head == "lightgbm" else "weights"+ console.print(f"mean CV {kind} {weights}")+ if result.get("ltr_train_run"):+ console.print(+ f"trained on {result.get('ltr_n_train_queries')} queries "+ f"from {result.get('ltr_train_run')}"+ )+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")+ return 0 if report.get("status") == "ok" else 1+++def cmd_pool_first_stage(args: argparse.Namespace, settings: Settings) -> int:+ from search.pool_first_stage import main as pool_main++ argv = ["--dataset", str(_resolve_dataset_path(args, settings))]+ if args.run:+ argv.extend(["--run", str(args.run)])+ if args.variant:+ argv.extend(["--variant", str(args.variant)])+ if args.expand_out:+ argv.extend(["--expand-out", str(args.expand_out)])+ return int(pool_main(argv))+++def cmd_rerank_retrieved(args: argparse.Namespace, settings: Settings) -> int:+ from search.rerank_retrieved import run_ce_on_retrieved++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ source_dir = settings.runs_dir / args.from_run+ eval_path = source_dir / "eval.json"+ if not eval_path.exists():+ console.print(f"[red]Missing[/red] {eval_path}")+ return 1+ eval_result = json.loads(eval_path.read_text(encoding="utf-8"))+ eval_result["run_id"] = args.from_run+ rows = load_records(path)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ ks = None+ if getattr(args, "ks", None):+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip())+ console.print(+ f"[cyan]Harness CE on retrieved hits[/cyan] from={args.from_run} "+ f"dataset={path.name} run={run_id} (not Reddy 0.857)"+ )+ result = run_ce_on_retrieved(+ eval_result,+ rows,+ dataset_name=path.name,+ ks=ks,+ brain_id=settings.brain_id,+ )+ report = write_report(run_dir, result)+ print_report_table(report)+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")+ return 0 if report.get("status") == "ok" else 1+++def cmd_rank_pool_ce(args: argparse.Namespace, settings: Settings) -> int:+ from search.rank_pool import run_ce_on_pool++ path = _resolve_dataset_path(args, settings)+ if not path.exists():+ console.print(f"[red]Missing[/red] {path}")+ return 1+ rows = load_records(path)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ console.print(+ f"[cyan]Ranking-in-pool CE[/cyan] dataset={path.name} run={run_id} "+ f"(cite Reddy 0.857 only if protocol is CE-on-pool; n is not ~4477)"+ )+ result = run_ce_on_pool(+ rows,+ dataset_name=path.name,+ brain_id=settings.brain_id,+ )+ report = write_report(run_dir, result)+ print_report_table(report)+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")+ if result.get("ce_model"):+ console.print(+ f"ce_model={result.get('ce_model')} "+ f"missing_text={result.get('ce_missing_text')}"+ )+ return 0 if report.get("status") == "ok" else 1+++def build_parser() -> argparse.ArgumentParser:+ parser = argparse.ArgumentParser(+ prog="search",+ description=(+ "Labeled search eval on searchbench*: ingest docs → "+ "POST /retrieve/search (Recall/nDCG/MRR + retrieve latency)"+ ),+ )+ parser.add_argument(+ "--env-file",+ type=Path,+ default=None,+ help="Optional dotenv path (default benchmarks/.env)",+ )+ parser.add_argument(+ "--brain",+ default=None,+ help=f"Brain id (default {DEFAULT_BRAIN_ID}; must start with searchbench)",+ )+ sub = parser.add_subparsers(dest="command", required=True)++ p_download = sub.add_parser(+ "download",+ help="Download Amazon ESCI and/or WANDS into search JSONL",+ )+ p_download.add_argument(+ "--name",+ default="all",+ help="esci, wands, jdsearch, or all (comma-separated; all = esci,wands)",+ )+ p_download.add_argument("--out", type=str, default=None)+ p_download.add_argument("--force", action="store_true")+ p_download.add_argument(+ "--dry-stats",+ action="store_true",+ dest="dry_stats",+ help="JDsearch only: print label histogram from the tar/cache without writing JSONL",+ )+ p_download.add_argument("--max-queries", type=int, default=80)+ p_download.add_argument("--max-docs", type=int, default=2000)+ p_download.add_argument("--candidates-per-query", type=int, default=40)+ p_download.add_argument(+ "--locale",+ default="us",+ help="ESCI locale: us, es, or jp (default us). Not Italian.",+ )+ p_download.add_argument(+ "--split",+ default="test",+ help="ESCI split (default test; small_version / Task 1 only)",+ )+ p_download.add_argument(+ "--holdout-dataset",+ dest="holdout_dataset",+ default="",+ help="JSONL whose qids are skipped (e.g. data/search_esci_74.jsonl)",+ )+ p_download.set_defaults(func=cmd_download)++ p_stats = sub.add_parser("dataset-stats", help="Summarize search JSONL")+ p_stats.add_argument("--dataset", type=str, default=None)+ p_stats.set_defaults(func=cmd_dataset_stats)++ p_smoke = sub.add_parser(+ "smoke", help="Ingest a few docs then POST /retrieve/search"+ )+ p_smoke.add_argument("--dataset", type=str, default=None)+ p_smoke.add_argument("--limit", type=int, default=2)+ p_smoke.add_argument("--limit-queries", type=int, default=2)+ p_smoke.add_argument("--k", type=int, default=10)+ p_smoke.add_argument("--fusion", choices=["rrf", "cc"], default=None)+ p_smoke.add_argument("--fusion-alpha", dest="fusion_alpha", type=float, default=None)+ p_smoke.add_argument(+ "--rerank",+ default=None,+ help="none or plugin:<name> (e.g. plugin:cross-encoder). Unknown name is 400.",+ )+ p_smoke.add_argument(+ "--channels",+ default=None,+ help=(+ "Comma-separated channels (default passages): "+ "passages,entities,events,communities and/or plugin:<name>."+ ),+ )+ p_smoke.add_argument("--node-labels", dest="node_labels", default=None)+ p_smoke.add_argument("--community-labels", dest="community_labels", default=None)+ p_smoke.add_argument(+ "--expand",+ choices=["none", "neighbors"],+ default="none",+ )+ p_smoke.add_argument(+ "--ingest-graph",+ action="store_true",+ help="Also write deterministic catalog triples (entity uuid = doc_id).",+ )+ p_smoke.add_argument(+ "--interactions",+ type=str,+ default=None,+ help="Optional interaction JSONL (EVENT+happened_at) on this searchbench brain.",+ )+ p_smoke.add_argument("--timeout", type=float, default=600.0)+ p_smoke.add_argument(+ "--enrich",+ action="store_true",+ help="Run Scout/Architect LLM ingest. Default skips enrichment (chunk+embed only).",+ )+ p_smoke.add_argument(+ "--skip-ingest",+ action="store_true",+ help="Reuse chunks already on this searchbench brain (no POST /ingest/).",+ )+ p_smoke.add_argument(+ "--personalize",+ action="store_true",+ dest="personalize",+ help="Send each query's target field on POST /retrieve/search. Default omit.",+ )+ p_smoke.add_argument(+ "--target-from-query",+ action="store_true",+ dest="personalize",+ help="Alias for --personalize.",+ )+ p_smoke.set_defaults(func=cmd_smoke)++ p_bf = sub.add_parser(+ "backfill-entity-text",+ help="ENTITY-only search_text + nodes vector refresh (not architect, not frozen brains)",+ )+ p_bf.add_argument("--dataset", type=str, default="data/search_wands.jsonl")+ p_bf.add_argument("--limit", type=int, default=None)+ p_bf.set_defaults(func=cmd_backfill_entity_text)++ p_eval = sub.add_parser(+ "evaluate",+ help="Ingest corpus, search all queries, write report + ledger",+ )+ p_eval.add_argument("--dataset", type=str, default=None)+ p_eval.add_argument("--run", type=str, default=None)+ p_eval.add_argument("--ks", type=str, default="5,10,20")+ p_eval.add_argument("--k", type=int, default=20)+ p_eval.add_argument("--fusion", choices=["rrf", "cc"], default=None)+ p_eval.add_argument("--fusion-alpha", dest="fusion_alpha", type=float, default=None)+ p_eval.add_argument(+ "--rerank",+ default=None,+ help="none or plugin:<name> (e.g. plugin:cross-encoder). Unknown name is 400.",+ )+ p_eval.add_argument(+ "--mode",+ choices=["default", "catalog"],+ default="default",+ help="default keeps RERANK_MAX_K=10. catalog retrieves deeper and reranks up to 50.",+ )+ p_eval.add_argument(+ "--channels",+ default=None,+ help=(+ "Comma-separated channels (default passages): "+ "passages,entities,events,communities and/or plugin:<name>."+ ),+ )+ p_eval.add_argument("--node-labels", dest="node_labels", default=None)+ p_eval.add_argument("--community-labels", dest="community_labels", default=None)+ p_eval.add_argument(+ "--expand",+ choices=["none", "neighbors"],+ default="none",+ )+ p_eval.add_argument(+ "--ingest-graph",+ action="store_true",+ help="Also write deterministic catalog triples (entity uuid = doc_id).",+ )+ p_eval.add_argument(+ "--interactions",+ type=str,+ default=None,+ help="Optional interaction JSONL (EVENT+happened_at) on this searchbench brain.",+ )+ p_eval.add_argument("--timeout", type=float, default=600.0)+ p_eval.add_argument(+ "--enrich",+ action="store_true",+ help="Run Scout/Architect LLM ingest. Default skips enrichment (chunk+embed only).",+ )+ p_eval.add_argument("--limit-docs", dest="limit_docs", type=int, default=None)+ p_eval.add_argument("--limit-queries", dest="limit_queries", type=int, default=None)+ p_eval.add_argument(+ "--skip-ingest",+ action="store_true",+ help="Reuse chunks already on this searchbench brain (no POST /ingest/).",+ )+ p_eval.add_argument(+ "--rank-pool",+ action="store_true",+ dest="rank_pool",+ help=(+ "Restrict hits to each query's candidate_doc_ids (including I) "+ "and score pool nDCG@20. Do not average with shared-corpus nDCG@10."+ ),+ )+ p_eval.add_argument(+ "--personalize",+ action="store_true",+ dest="personalize",+ help="Send each query's target field on POST /retrieve/search. Default omit.",+ )+ p_eval.add_argument(+ "--target-from-query",+ action="store_true",+ dest="personalize",+ help="Alias for --personalize.",+ )+ p_eval.add_argument(+ "--extras",+ default=None,+ help='Optional equality filter JSON object, e.g. {"locale":"it"}',+ )+ p_eval.set_defaults(func=cmd_evaluate)++ p_report = sub.add_parser("report", help="Rebuild report from eval.json")+ p_report.add_argument("--run", type=str, required=True)+ p_report.set_defaults(func=cmd_report)++ p_pool = sub.add_parser(+ "rank-pool-ce",+ help="Score labeled candidate pools with MiniLM CE (no search API)",+ )+ p_pool.add_argument("--dataset", type=str, default=None)+ p_pool.add_argument("--run", type=str, default=None)+ p_pool.set_defaults(func=cmd_rank_pool_ce)++ p_rr = sub.add_parser(+ "rerank-retrieved",+ help="Harness CE over stored first-stage hits (not production RERANK_MAX_K)",+ )+ p_rr.add_argument("--from-run", dest="from_run", type=str, required=True)+ p_rr.add_argument("--dataset", type=str, default=None)+ p_rr.add_argument("--run", type=str, default=None)+ p_rr.add_argument("--ks", type=str, default=None)+ p_rr.set_defaults(func=cmd_rerank_retrieved)++ p_ft = sub.add_parser(+ "finetune-ce",+ help="Fine-tune a pool CE on ESCI US train (test qids held out)",+ )+ p_ft.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_ft.add_argument(+ "--out",+ type=str,+ default="data/models/esci-minilm-graded",+ )+ p_ft.add_argument("--max-pairs", type=int, default=80000)+ p_ft.add_argument("--epochs", type=int, default=1)+ p_ft.add_argument("--batch-size", type=int, default=32)+ p_ft.add_argument(+ "--label-mode",+ choices=["binary", "graded", "multiclass"],+ default="graded",+ )+ p_ft.add_argument("--fields", choices=["title", "catalog"], default="catalog")+ p_ft.add_argument(+ "--base",+ default="cross-encoder/ms-marco-MiniLM-L-6-v2",+ )+ p_ft.add_argument("--lr", type=float, default=7e-6)+ p_ft.add_argument("--max-length", type=int, default=256)+ p_ft.set_defaults(func=cmd_finetune_ce)++ p_ms = sub.add_parser(+ "miss-strata",+ help="Classify k=50 hits: head-ok / rank-too-low / total-miss",+ )+ p_ms.add_argument("--from-run", dest="from_run", type=str, required=True)+ p_ms.add_argument("--dataset", type=str, default=None)+ p_ms.add_argument("--out", type=str, default=None)+ p_ms.set_defaults(func=cmd_miss_strata)++ p_qw = sub.add_parser(+ "query-rewrite",+ help="Rewrite pathological total-miss queries only (skip esci-72)",+ )+ p_qw.add_argument("--from-run", dest="from_run", type=str, required=True)+ p_qw.add_argument("--dataset", type=str, default=None)+ p_qw.add_argument("--out", type=str, default=None)+ p_qw.set_defaults(func=cmd_query_rewrite)++ p_sp = sub.add_parser(+ "spell-normalize",+ help="Harness-only NFKC/punctuation/accent normalize of query text",+ )+ p_sp.add_argument("--dataset", type=str, default="data/search_esci_es.jsonl")+ p_sp.add_argument("--out", type=str, default="data/search_esci_es_spell.jsonl")+ p_sp.set_defaults(func=cmd_spell_normalize)++ p_ftd = sub.add_parser(+ "finetune-dense",+ help="Fine-tune a dual encoder with k=50 hard-negative bank (test qids held out)",+ )+ p_ftd.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_ftd.add_argument(+ "--out",+ type=str,+ default="data/models/esci-minilm-dense-ance",+ )+ p_ftd.add_argument(+ "--from-eval",+ type=str,+ default="runs/search-esci-74-passages-k50/eval.json",+ )+ p_ftd.add_argument("--max-pairs", type=int, default=20000)+ p_ftd.add_argument("--epochs", type=int, default=1)+ p_ftd.add_argument("--batch-size", type=int, default=32)+ p_ftd.add_argument("--fields", choices=["title", "catalog"], default="catalog")+ p_ftd.add_argument(+ "--base",+ default="sentence-transformers/all-MiniLM-L6-v2",+ )+ p_ftd.add_argument("--lr", type=float, default=2e-5)+ p_ftd.add_argument("--max-length", type=int, default=256)+ p_ftd.set_defaults(func=cmd_finetune_dense)++ p_ld = sub.add_parser(+ "local-dense",+ help="Retrieve the JSONL corpus with a local dual encoder (not BrainAPI)",+ )+ p_ld.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_ld.add_argument("--run", type=str, default="search-esci-74-dense-ance-k50")+ p_ld.add_argument("--model", type=str, required=True)+ p_ld.add_argument("--k", type=int, default=50)+ p_ld.add_argument("--ks", type=str, default="5,10,20,50")+ p_ld.set_defaults(func=cmd_local_dense)++ p_mine = sub.add_parser(+ "mine-retrieved-lists",+ help="BM25 top-k train lists with ESCI labels; unlabeled=I",+ )+ p_mine.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_mine.add_argument("--out", type=str, default="data/esci_retrieved_lists.jsonl")+ p_mine.add_argument("--max-queries", type=int, default=6000)+ p_mine.add_argument("--k", type=int, default=50)+ p_mine.add_argument("--seed", type=int, default=11)+ p_mine.set_defaults(func=cmd_mine_retrieved_lists)++ p_exh = sub.add_parser(+ "export-hybrid-lists",+ help="Export stored hybrid k=50 hits as 4-class rows; unlabeled=I",+ )+ p_exh.add_argument(+ "--from-run",+ dest="from_run",+ default="search-esci-ltr200-passages-k50",+ )+ p_exh.add_argument("--dataset", type=str, default="data/search_esci_ltr200.jsonl")+ p_exh.add_argument(+ "--holdout-dataset",+ dest="holdout_dataset",+ default="data/search_esci_74.jsonl",+ )+ p_exh.add_argument("--out", type=str, default="data/esci_hybrid_lists_ltr200.jsonl")+ p_exh.add_argument("--k", type=int, default=50)+ p_exh.set_defaults(func=cmd_export_hybrid_lists)++ p_ft4 = sub.add_parser(+ "finetune-4class",+ help="Fine-tune 4-class L-12 CE (pool or retrieved-bm25 lists)",+ )+ p_ft4.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_ft4.add_argument(+ "--out",+ type=str,+ default="data/models/esci-minilm-l12-retrieved",+ )+ p_ft4.add_argument("--from-lists", type=str, default=None)+ p_ft4.add_argument(+ "--lists-source",+ dest="lists_source",+ default="",+ help="Label for --from-lists (hybrid-k50 or retrieved-bm25)",+ )+ p_ft4.add_argument("--max-pairs", type=int, default=80000)+ p_ft4.add_argument("--epochs", type=int, default=1)+ p_ft4.add_argument("--batch-size", type=int, default=32)+ p_ft4.add_argument("--max-length", type=int, default=192)+ p_ft4.add_argument("--ckpt-every", type=int, default=200)+ p_ft4.add_argument("--class-weights", action="store_true")+ p_ft4.add_argument(+ "--base",+ default="cross-encoder/ms-marco-MiniLM-L-12-v2",+ )+ p_ft4.add_argument("--seed", type=int, default=11)+ p_ft4.set_defaults(func=cmd_finetune_4class)++ p_cb = sub.add_parser(+ "colbert-local",+ help="ColBERT MaxSim sidecar over the JSONL corpus (not BrainAPI)",+ )+ p_cb.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_cb.add_argument("--run", type=str, default="search-esci-74-colbert-k50")+ p_cb.add_argument("--k", type=int, default=50)+ p_cb.add_argument("--ks", type=str, default="5,10,20,50")+ p_cb.set_defaults(func=cmd_colbert_local)++ p_rc = sub.add_parser(+ "rank-corpus",+ help="Score every JSONL doc with 4-class CE (exhaustive-catalog, not production)",+ )+ p_rc.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_rc.add_argument("--run", type=str, default="search-esci-74-exhaustive-ce")+ p_rc.add_argument(+ "--model",+ type=str,+ default="data/models/esci-minilm-l12-4class-nowt-e2",+ )+ p_rc.add_argument("--k", type=int, default=50)+ p_rc.add_argument("--ks", type=str, default="5,10,20,50")+ p_rc.add_argument("--max-length", dest="max_length", type=int, default=192)+ p_rc.set_defaults(func=cmd_rank_corpus)++ p_ov = sub.add_parser(+ "list-overlap",+ help="Count gold ASINs in sidecar top-k missing from passages",+ )+ p_ov.add_argument(+ "--passages-run",+ dest="passages_run",+ default="search-esci-74-passages-k50",+ )+ p_ov.add_argument(+ "--against-runs",+ dest="against_runs",+ default="search-esci-74-bge-base-k50,search-esci-74-colbert-k50",+ )+ p_ov.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_ov.add_argument("--run", type=str, default="search-esci-74-list-overlap")+ p_ov.add_argument("--k", type=int, default=50)+ p_ov.set_defaults(func=cmd_list_overlap)++ p_un = sub.add_parser(+ "union-lists",+ help="RRF-union stored first-stage lists (harness only, not live graph)",+ )+ p_un.add_argument(+ "--from-runs",+ dest="from_runs",+ default="search-esci-74-passages-k50,search-esci-74-bge-base-k50",+ )+ p_un.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_un.add_argument("--run", type=str, default="search-esci-74-union-bge-k50")+ p_un.add_argument("--k", type=int, default=50)+ p_un.add_argument("--ks", type=str, default="5,10,20,50")+ p_un.set_defaults(func=cmd_union_lists)++ p_cs = sub.add_parser(+ "cascade-lists",+ help="Keep passages top-10; inject sidecar unique golds into ranks 11-50",+ )+ p_cs.add_argument(+ "--passages-run",+ dest="passages_run",+ default="search-esci-74-passages-k50",+ )+ p_cs.add_argument(+ "--from-runs",+ dest="from_runs",+ default="search-esci-74-bge-base-k50,search-esci-74-colbert-k50",+ )+ p_cs.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_cs.add_argument("--run", type=str, default="search-esci-74-cascade-tail-k50")+ p_cs.add_argument("--k", type=int, default=50)+ p_cs.add_argument("--head-k", dest="head_k", type=int, default=10)+ p_cs.add_argument("--ks", type=str, default="5,10,20,50")+ p_cs.set_defaults(func=cmd_cascade_lists)++ p_ltr = sub.add_parser(+ "ltr-head",+ help="Query-grouped CV LTR on stored hybrid k=50 (BM25/dense rank + title overlap)",+ )+ p_ltr.add_argument(+ "--from-run",+ dest="from_run",+ default="search-esci-74-passages-k50",+ )+ p_ltr.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_ltr.add_argument("--run", type=str, default="search-esci-74-ltr-head-k50")+ p_ltr.add_argument("--k", type=int, default=50)+ p_ltr.add_argument("--ks", type=str, default="5,10,20,50")+ p_ltr.add_argument("--folds", type=int, default=5)+ p_ltr.add_argument(+ "--pair-policy",+ dest="pair_policy",+ default="unlabeled_zero",+ choices=["unlabeled_zero", "other_query_neg"],+ )+ p_ltr.add_argument(+ "--ce-model",+ dest="ce_model",+ default="",+ help="Optional 4-class CE dir; ce_gain is a feature, not the ranker",+ )+ p_ltr.add_argument(+ "--ltr-model",+ dest="ltr_model",+ default="ranknet",+ choices=["ranknet", "lightgbm"],+ help="Linear RankNet (default) or gated LightGBM lambdarank",+ )+ p_ltr.add_argument(+ "--train-from-run",+ dest="train_from_run",+ default="",+ help="Fit on this run's lists; apply to --from-run (no CV on eval qids)",+ )+ p_ltr.add_argument(+ "--train-dataset",+ dest="train_dataset",+ default="",+ help="JSONL for --train-from-run",+ )+ p_ltr.set_defaults(func=cmd_ltr_head)++ p_fs = sub.add_parser(+ "pool-first-stage",+ help="Local fielded BM25 over the JSONL corpus (not Reddy 0.857)",+ )+ p_fs.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")+ p_fs.add_argument("--run", type=str, default="search-esci-74-fielded-bm25")+ p_fs.add_argument(+ "--variant",+ default="all,title,title-boost,rm3,rm3-title-boost",+ )+ p_fs.add_argument(+ "--expand-out",+ type=str,+ default="data/search_esci_74_rm3.jsonl",+ )+ p_fs.set_defaults(func=cmd_pool_first_stage)++ return parser+++def main(argv: list[str] | None = None) -> int:+ parser = build_parser()+ args = parser.parse_args(argv)+ settings = Settings.load(args.env_file)+ if args.brain:+ settings.brain_id = validate_brain_id(args.brain)+ return int(args.func(args, settings))+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/search/client.py b/benchmarks/search/client.pynew file mode 100644index 0000000..5f693b3--- /dev/null+++ b/benchmarks/search/client.py@@ -0,0 +1,268 @@+from __future__ import annotations++import time+from dataclasses import dataclass, field+from typing import Any++import httpx++from search.config import TERMINAL_TASK_STATUSES, Settings+++class SearchDisabledError(RuntimeError):+ pass+++@dataclass+class TimedResult:+ data: Any+ latency_ms: float+ status_code: int = 200+++@dataclass+class BrainAPIClient:+ settings: Settings+ timeout: float = 120.0+ max_retries: int = 5+ _client: httpx.Client = field(init=False, repr=False)++ def __post_init__(self) -> None:+ self.settings.require_brainapi()+ self._client = httpx.Client(+ base_url=self.settings.brainapi_url,+ headers={+ "Authorization": f"Bearer {self.settings.brainpat_token}",+ "Content-Type": "application/json",+ "X-Brain-ID": self.settings.brain_id,+ },+ timeout=self.timeout,+ )++ def close(self) -> None:+ self._client.close()++ def __enter__(self) -> "BrainAPIClient":+ return self++ def __exit__(self, *args: object) -> None:+ self.close()++ def _request(+ self,+ method: str,+ path: str,+ *,+ json: dict | None = None,+ params: dict | None = None,+ ) -> TimedResult:+ last_error: Exception | None = None+ for attempt in range(self.max_retries):+ started = time.perf_counter()+ try:+ response = self._client.request(+ method, path, json=json, params=params+ )+ latency_ms = (time.perf_counter() - started) * 1000+ if (+ response.status_code == 404+ and path.rstrip("/").endswith("/retrieve/search")+ ):+ detail = ""+ try:+ detail = response.text[:500]+ except Exception:+ detail = ""+ raise SearchDisabledError(+ "POST /retrieve/search returned 404. Set SEARCH_ENABLED=true "+ "(and DATA_DB=postgresql) on the API under test. "+ f"body={detail!r}"+ ) from None+ if response.status_code in (429, 500, 502, 503, 504):+ wait = min(2**attempt * 0.5, 30)+ time.sleep(wait)+ last_error = httpx.HTTPStatusError(+ f"HTTP {response.status_code}",+ request=response.request,+ response=response,+ )+ continue+ if response.status_code >= 400:+ detail = ""+ try:+ detail = response.text[:500]+ except Exception:+ detail = ""+ raise RuntimeError(+ f"{method} {path} returned {response.status_code}. "+ f"body={detail!r}"+ ) from None+ if response.status_code == 204 or not response.content:+ return TimedResult(+ data=None,+ latency_ms=latency_ms,+ status_code=response.status_code,+ )+ return TimedResult(+ data=response.json(),+ latency_ms=latency_ms,+ status_code=response.status_code,+ )+ except SearchDisabledError:+ raise+ except (httpx.TransportError, httpx.TimeoutException) as exc:+ last_error = exc+ wait = min(2**attempt * 0.5, 30)+ time.sleep(wait)+ raise RuntimeError(+ f"Request failed after retries: {method} {path}: {last_error}"+ )++ def ingest_structured(+ self,+ triples: list[dict[str, Any]],+ brain_id: str | None = None,+ *,+ mode: str = "deterministic",+ ) -> TimedResult:+ bid = brain_id or self.settings.brain_id+ body = {+ "mode": mode,+ "brain_id": bid,+ "data": triples,+ }+ return self._request("POST", "/ingest/structured", json=body)++ def ingest_text(+ self,+ text: str,+ brain_id: str | None = None,+ *,+ skip_enrichment: bool = False,+ meta_keys: dict[str, Any] | None = None,+ ) -> TimedResult:+ bid = brain_id or self.settings.brain_id+ body: dict[str, Any] = {+ "data": {"data_type": "text", "text_data": text},+ "brain_id": bid,+ "observate_for": [],+ "preferred_extraction_entities": [],+ "skip_enrichment": skip_enrichment,+ }+ if meta_keys:+ body["meta_keys"] = meta_keys+ return self._request("POST", "/ingest/", json=body)++ def get_task(self, task_id: str, brain_id: str | None = None) -> TimedResult:+ bid = brain_id or self.settings.brain_id+ return self._request(+ "GET", f"/tasks/{task_id}", params={"brain_id": bid}+ )++ def wait_for_task(+ self,+ task_id: str,+ brain_id: str | None = None,+ *,+ timeout_s: float = 600.0,+ poll_interval_s: float = 2.0,+ ) -> TimedResult:+ bid = brain_id or self.settings.brain_id+ started = time.perf_counter()+ while True:+ result = self.get_task(task_id, bid)+ status = (result.data or {}).get("status", "unknown")+ if status in TERMINAL_TASK_STATUSES:+ return TimedResult(+ data=result.data,+ latency_ms=(time.perf_counter() - started) * 1000,+ status_code=result.status_code,+ )+ if time.perf_counter() - started > timeout_s:+ raise TimeoutError(+ f"Task {task_id} did not finish within {timeout_s}s "+ f"(last status={status})"+ )+ time.sleep(poll_interval_s)++ def search(+ self,+ query: str,+ brain_id: str | None = None,+ *,+ k: int = 20,+ fusion: str | None = None,+ fusion_alpha: float | None = None,+ rerank: str | None = None,+ mode: str | None = None,+ profile_stages: bool = True,+ channels: list[str] | None = None,+ node_labels: list[str] | None = None,+ community_labels: list[str] | None = None,+ expand: str | None = None,+ extras: dict[str, str] | None = None,+ target: str | None = None,+ ) -> TimedResult:+ bid = brain_id or self.settings.brain_id+ body: dict[str, Any] = {+ "query": query,+ "brain_id": bid,+ "k": k,+ "channels": channels or ["passages"],+ "profile_stages": profile_stages,+ }+ if fusion:+ body["fusion"] = fusion+ if fusion_alpha is not None:+ body["fusion_alpha"] = fusion_alpha+ if rerank:+ body["rerank"] = rerank+ if mode and mode != "default":+ body["mode"] = mode+ if node_labels:+ body["node_labels"] = node_labels+ if community_labels:+ body["community_labels"] = community_labels+ if expand and expand != "none":+ body["expand"] = expand+ if extras:+ body["extras"] = extras+ if target:+ body["target"] = target+ return self._request("POST", "/retrieve/search", json=body)++ def get_neighbors(+ self,+ uuid: str,+ brain_id: str | None = None,+ *,+ limit: int = 10,+ look_for: str | None = None,+ ) -> TimedResult:+ bid = brain_id or self.settings.brain_id+ params: dict[str, Any] = {+ "uuid": uuid,+ "brain_id": bid,+ "limit": limit,+ }+ if look_for:+ params["look_for"] = look_for+ return self._request("GET", "/retrieve/entities/neighbors", params=params)++ def list_text_chunks(+ self,+ brain_id: str | None = None,+ *,+ limit: int = 100,+ skip: int = 0,+ query_text: str | None = None,+ ) -> TimedResult:+ bid = brain_id or self.settings.brain_id+ params: dict[str, Any] = {+ "brain_id": bid,+ "limit": limit,+ "skip": skip,+ }+ if query_text:+ params["query_text"] = query_text+ return self._request("GET", "/retrieve/text-chunks", params=params)diff --git a/benchmarks/search/config.py b/benchmarks/search/config.pynew file mode 100644index 0000000..20484e0--- /dev/null+++ b/benchmarks/search/config.py@@ -0,0 +1,96 @@+from __future__ import annotations++import os+from dataclasses import dataclass+from pathlib import Path++from dotenv import load_dotenv++BENCHMARKS_ROOT = Path(__file__).resolve().parent.parent+DATA_DIR = BENCHMARKS_ROOT / "data"+RUNS_DIR = BENCHMARKS_ROOT / "runs"+DEFAULT_DATASET_PATH = DATA_DIR / "search_toy.jsonl"+DEFAULT_BRAIN_ID = "searchbenchsmoke"+REQUIRED_PREFIX = "searchbench"+FORBIDDEN_BRAINS = frozenset(+ {+ "beam1m1clean",+ "demorecsys",+ "locomoconv26",+ "locomoconv26clean",+ "locomoconv26nostorm",+ }+)+FORBIDDEN_PREFIXES = (+ "beam",+ "demorecsys",+ "lme",+ "locomoconv",+ "longmemeval",+)+FROZEN_STRUCTURED_BRAINS = frozenset(+ {+ "searchbenchwands",+ "searchbenchesci74",+ "searchbenchescies",+ "searchbenchesciltr2",+ }+)+TERMINAL_TASK_STATUSES = frozenset({"completed", "failed", "partial_failed"})+++def validate_brain_id(brain_id: str) -> str:+ bid = (brain_id or "").strip()+ if not bid:+ raise SystemExit("brain_id is required.")+ lowered = bid.lower()+ forbidden = {item.lower() for item in FORBIDDEN_BRAINS}+ if lowered in forbidden or lowered.startswith(FORBIDDEN_PREFIXES):+ raise SystemExit(+ f"Refusing brain_id={bid!r}. Use a dedicated searchbench* brain "+ f"(default {DEFAULT_BRAIN_ID}). Never wipe eval memory or recsys brains."+ )+ if not lowered.startswith(REQUIRED_PREFIX):+ raise SystemExit(+ f"Refusing brain_id={bid!r}. Search eval brains must start with "+ f"{REQUIRED_PREFIX!r} (default {DEFAULT_BRAIN_ID})."+ )+ return bid+++@dataclass+class Settings:+ brainapi_url: str+ brainpat_token: str+ dataset_path: Path+ runs_dir: Path+ brain_id: str++ @classmethod+ def load(cls, env_file: Path | None = None) -> "Settings":+ if env_file is None:+ env_file = BENCHMARKS_ROOT / ".env"+ if env_file.exists():+ load_dotenv(env_file, override=False)+ else:+ load_dotenv(override=False)++ brain_id = validate_brain_id(+ os.getenv("SEARCH_BRAIN_ID", DEFAULT_BRAIN_ID) or DEFAULT_BRAIN_ID+ )+ return cls(+ brainapi_url=os.getenv("BRAINAPI_URL", "http://localhost:8000").rstrip("/"),+ brainpat_token=os.getenv("BRAINPAT_TOKEN", ""),+ dataset_path=Path(+ os.getenv("SEARCH_DATASET_PATH", str(DEFAULT_DATASET_PATH))+ ),+ runs_dir=Path(os.getenv("BENCH_RUNS_DIR", str(RUNS_DIR))),+ brain_id=brain_id,+ )++ def require_brainapi(self) -> None:+ if not self.brainpat_token:+ raise SystemExit(+ "BRAINPAT_TOKEN is required. Copy benchmarks/.env.example to "+ "benchmarks/.env and set it."+ )diff --git a/benchmarks/search/dataset.py b/benchmarks/search/dataset.pynew file mode 100644index 0000000..3a75241--- /dev/null+++ b/benchmarks/search/dataset.py@@ -0,0 +1,76 @@+from __future__ import annotations++import json+from collections import Counter+from pathlib import Path+from typing import Any++DOC_MARKER_PREFIX = "DOCID "+++def doc_marker(doc_id: str) -> str:+ return f"{DOC_MARKER_PREFIX}{doc_id}"+++def write_records(rows: list[dict[str, Any]], path: Path) -> Path:+ path.parent.mkdir(parents=True, exist_ok=True)+ with path.open("w", encoding="utf-8") as fh:+ for row in rows:+ fh.write(json.dumps(row, ensure_ascii=False) + "\n")+ return path+++def load_records(path: Path) -> list[dict[str, Any]]:+ rows: list[dict[str, Any]] = []+ for line in path.read_text(encoding="utf-8").splitlines():+ line = line.strip()+ if not line:+ continue+ rows.append(json.loads(line))+ return rows+++def split_corpus(+ rows: list[dict[str, Any]],+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:+ docs: list[dict[str, Any]] = []+ queries: list[dict[str, Any]] = []+ for row in rows:+ kind = str(row.get("type") or "").strip().lower()+ if kind == "doc":+ docs.append(row)+ elif kind == "query":+ queries.append(row)+ else:+ raise ValueError(f"Unknown record type {kind!r} in {row!r}")+ return docs, queries+++def dataset_stats(rows: list[dict[str, Any]]) -> dict[str, Any]:+ docs, queries = split_corpus(rows)+ slices = Counter(str(q.get("slice") or "unspecified") for q in queries)+ return {+ "n_docs": len(docs),+ "n_queries": len(queries),+ "slices": dict(slices),+ "doc_ids": [str(d.get("doc_id") or "") for d in docs],+ "graded": any(bool(q.get("gold_grades")) for q in queries),+ }+++def map_doc_ids_to_chunks(+ docs: list[dict[str, Any]],+ chunks: list[dict[str, Any]],+) -> dict[str, set[str]]:+ mapping: dict[str, set[str]] = {str(doc["doc_id"]): set() for doc in docs}+ for chunk in chunks:+ text = str(chunk.get("text") or "")+ chunk_id = str(chunk.get("id") or "")+ if not chunk_id:+ continue+ for doc in docs:+ doc_id = str(doc.get("doc_id") or "")+ marker = str(doc.get("marker") or doc_marker(doc_id))+ if marker and marker in text:+ mapping.setdefault(doc_id, set()).add(chunk_id)+ return mappingdiff --git a/benchmarks/search/evaluate.py b/benchmarks/search/evaluate.pynew file mode 100644index 0000000..e194bf2--- /dev/null+++ b/benchmarks/search/evaluate.py@@ -0,0 +1,747 @@+from __future__ import annotations++from datetime import datetime, timezone+from pathlib import Path+from typing import Any++from search.client import BrainAPIClient, SearchDisabledError, TimedResult+from search.config import FROZEN_STRUCTURED_BRAINS, Settings+from search.dataset import map_doc_ids_to_chunks, split_corpus+from search.mapping import docs_to_triples, interactions_to_triples+from search.metrics import (+ aggregate_query_metrics,+ mrr,+ ndcg_at_k,+ recall_at_k,+ retrieve_latency_ms,+)+++def ensure_run_dir(settings: Settings, run_id: str | None = None) -> tuple[str, Path]:+ stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")+ rid = run_id or f"search-{stamp}"+ run_dir = settings.runs_dir / rid+ run_dir.mkdir(parents=True, exist_ok=True)+ return rid, run_dir+++def _hit_ids(payload: dict[str, Any] | None) -> list[str]:+ hits = (payload or {}).get("hits") or []+ ids: list[str] = []+ for hit in hits:+ if not isinstance(hit, dict):+ continue+ hid = hit.get("id")+ if hid:+ ids.append(str(hid))+ return ids+++def _hit_dump(+ payload: dict[str, Any] | None,+ chunk_to_doc: dict[str, str] | None = None,+) -> list[dict[str, Any]]:+ mapping = chunk_to_doc or {}+ rows: list[dict[str, Any]] = []+ for hit in (payload or {}).get("hits") or []:+ if not isinstance(hit, dict):+ continue+ hid = hit.get("id")+ if not hid:+ continue+ raw_id = str(hid)+ node_id = str(hit.get("node_id") or "").strip() or None+ rows.append(+ {+ "id": raw_id,+ "channel": str(hit.get("channel") or ""),+ "doc_id": mapping.get(raw_id, raw_id),+ "node_id": node_id,+ }+ )+ return rows+++_DOC_EXTRAS_KEYS = ("brand", "color", "locale")+++def doc_meta_keys(doc: dict[str, Any]) -> dict[str, str] | None:+ meta: dict[str, str] = {}+ extras = doc.get("extras")+ if isinstance(extras, dict):+ for key, value in extras.items():+ if value is None:+ continue+ meta[str(key)] = str(value)+ for key in _DOC_EXTRAS_KEYS:+ value = doc.get(key)+ if value is None:+ continue+ meta[str(key)] = str(value)+ return meta or None+++def ingest_docs(+ client: BrainAPIClient,+ docs: list[dict[str, Any]],+ *,+ timeout_s: float,+ skip_enrichment: bool = True,+) -> dict[str, Any]:+ tasks: list[dict[str, Any]] = []+ for doc in docs:+ text = str(doc.get("text") or "")+ accepted = client.ingest_text(+ text,+ skip_enrichment=skip_enrichment,+ meta_keys=doc_meta_keys(doc),+ )+ task_id = str((accepted.data or {}).get("task_id") or "")+ if not task_id:+ tasks.append(+ {+ "doc_id": doc.get("doc_id"),+ "status": "failed",+ "error": "ingest response missing task_id",+ }+ )+ continue+ waited = client.wait_for_task(+ task_id,+ timeout_s=timeout_s,+ poll_interval_s=0.25 if skip_enrichment else 2.0,+ )+ status = str((waited.data or {}).get("status") or "unknown")+ tasks.append(+ {+ "doc_id": doc.get("doc_id"),+ "task_id": task_id,+ "status": status,+ }+ )+ statuses = [str(item.get("status")) for item in tasks]+ if not tasks:+ overall = "failed"+ elif all(status == "completed" for status in statuses):+ overall = "completed"+ elif any(status == "completed" for status in statuses):+ overall = "partial_failed"+ else:+ overall = "failed"+ return {"status": overall, "n_docs": len(docs), "tasks": tasks}+++def ingest_triples(+ client: BrainAPIClient,+ triples: list[dict[str, Any]],+ *,+ timeout_s: float,+ chunk_size: int = 20000,+) -> dict[str, Any]:+ if not triples:+ return {"status": "skipped", "n_triples": 0, "tasks": []}+ tasks: list[dict[str, Any]] = []+ chunk = max(1, int(chunk_size))+ for start in range(0, len(triples), chunk):+ batch = triples[start : start + chunk]+ submitted = client.ingest_structured(batch)+ task_id = str((submitted.data or {}).get("task_id") or "")+ if not task_id:+ tasks.append(+ {+ "status": "failed",+ "error": "structured ingest response missing task_id",+ }+ )+ continue+ waited = client.wait_for_task(task_id, timeout_s=timeout_s)+ status = str((waited.data or {}).get("status") or "unknown")+ tasks.append({"task_id": task_id, "status": status})+ statuses = [str(item.get("status")) for item in tasks]+ if not tasks:+ overall = "failed"+ elif all(status == "completed" for status in statuses):+ overall = "completed"+ elif any(status == "completed" for status in statuses):+ overall = "partial_failed"+ else:+ overall = "failed"+ return {"status": overall, "n_triples": len(triples), "tasks": tasks}+++def list_all_text_chunks(client: BrainAPIClient) -> list[dict[str, Any]]:+ chunks: list[dict[str, Any]] = []+ skip = 0+ page = 100+ total = None+ while True:+ result = client.list_text_chunks(limit=page, skip=skip)+ payload = result.data or {}+ batch = payload.get("data") or []+ if not isinstance(batch, list):+ break+ chunks.extend([item for item in batch if isinstance(item, dict)])+ total = payload.get("total")+ skip += len(batch)+ if not batch:+ break+ if total is not None and skip >= int(total):+ break+ return chunks+++def invert_doc_chunks(doc_to_chunks: dict[str, set[str]]) -> dict[str, str]:+ chunk_to_doc: dict[str, str] = {}+ for doc_id, chunks in (doc_to_chunks or {}).items():+ for chunk_id in chunks or set():+ chunk_to_doc[str(chunk_id)] = str(doc_id)+ return chunk_to_doc+++def canonicalize_hit_ids(+ ranked: list[str],+ chunk_to_doc: dict[str, str],+) -> list[str]:+ out: list[str] = []+ seen: set[str] = set()+ for hid in ranked:+ key = chunk_to_doc.get(str(hid), str(hid))+ if not key or key in seen:+ continue+ seen.add(key)+ out.append(key)+ return out+++def unique_doc_counts(+ raw_ids: list[str],+ chunk_to_doc: dict[str, str] | None = None,+ *,+ k: int = 20,+) -> tuple[int, int]:+ head = [str(item) for item in raw_ids[:k] if item]+ mapping = chunk_to_doc or {}+ canonical = [mapping.get(item, item) for item in head]+ return len(set(head)), len({item for item in canonical if item})+++def channel_id_lists(+ payload: dict[str, Any] | None,+ hits: list[dict[str, str]] | None = None,+) -> dict[str, list[str]]:+ raw = (payload or {}).get("channel_lists") or {}+ if not isinstance(raw, dict):+ raw = {}+ out: dict[str, list[str]] = {+ "dense_ids": [str(item) for item in (raw.get("dense") or []) if item],+ "bm25_ids": [str(item) for item in (raw.get("bm25") or []) if item],+ "entity_ids": [str(item) for item in (raw.get("entities") or []) if item],+ "community_ids": [str(item) for item in (raw.get("communities") or []) if item],+ "event_ids": [str(item) for item in (raw.get("events") or []) if item],+ "neighbor_ids": [str(item) for item in (raw.get("neighbors") or []) if item],+ "passage_ids": [],+ }+ by_channel: dict[str, list[str]] = {}+ for hit in hits or []:+ channel = str(hit.get("channel") or "")+ hid = str(hit.get("id") or "")+ if not channel or not hid:+ continue+ by_channel.setdefault(channel, []).append(hid)+ out["passage_ids"] = list(by_channel.get("passages") or [])+ if not out["entity_ids"]:+ out["entity_ids"] = list(by_channel.get("entities") or [])+ if not out["community_ids"]:+ out["community_ids"] = list(by_channel.get("communities") or [])+ if not out["event_ids"]:+ out["event_ids"] = list(by_channel.get("events") or [])+ if not out["neighbor_ids"]:+ out["neighbor_ids"] = list(by_channel.get("neighbors") or [])+ return out+++def gold_chunk_ids(+ query: dict[str, Any],+ doc_to_chunks: dict[str, set[str]],+) -> set[str]:+ gold: set[str] = set()+ for doc_id in query.get("gold_doc_ids") or []:+ gold.update(doc_to_chunks.get(str(doc_id)) or set())+ for chunk_id in query.get("gold_chunk_ids") or []:+ gold.add(str(chunk_id))+ grades = query.get("gold_grades") or {}+ if isinstance(grades, dict):+ for doc_id, gain in grades.items():+ if float(gain or 0) <= 0:+ continue+ gold.update(doc_to_chunks.get(str(doc_id)) or set())+ return gold+++def gold_chunk_grades(+ query: dict[str, Any],+ doc_to_chunks: dict[str, set[str]],+) -> dict[str, float]:+ grades: dict[str, float] = {}+ raw = query.get("gold_grades") or {}+ if isinstance(raw, dict) and raw:+ for doc_id, gain in raw.items():+ value = float(gain or 0)+ if value <= 0:+ continue+ for chunk_id in doc_to_chunks.get(str(doc_id)) or set():+ grades[chunk_id] = max(grades.get(chunk_id, 0.0), value)+ return grades+ for chunk_id in gold_chunk_ids(query, doc_to_chunks):+ grades[chunk_id] = 1.0+ return grades+++def gold_hit_ids(+ query: dict[str, Any],+ doc_to_chunks: dict[str, set[str]],+) -> set[str]:+ gold: set[str] = set()+ for doc_id in query.get("gold_doc_ids") or []:+ gold.add(str(doc_id))+ grades = query.get("gold_grades") or {}+ if isinstance(grades, dict):+ for doc_id, gain in grades.items():+ if float(gain or 0) <= 0:+ continue+ gold.add(str(doc_id))+ chunk_to_doc = invert_doc_chunks(doc_to_chunks)+ for chunk_id in query.get("gold_chunk_ids") or []:+ gold.add(chunk_to_doc.get(str(chunk_id), str(chunk_id)))+ if gold:+ return gold+ return {+ chunk_to_doc.get(chunk_id, chunk_id)+ for chunk_id in gold_chunk_ids(query, doc_to_chunks)+ }+++def gold_hit_grades(+ query: dict[str, Any],+ doc_to_chunks: dict[str, set[str]],+) -> dict[str, float]:+ grades: dict[str, float] = {}+ raw = query.get("gold_grades") or {}+ if isinstance(raw, dict) and raw:+ for doc_id, gain in raw.items():+ value = float(gain or 0)+ if value <= 0:+ continue+ key = str(doc_id)+ grades[key] = max(grades.get(key, 0.0), value)+ return grades+ for doc_id in gold_hit_ids(query, doc_to_chunks):+ grades[str(doc_id)] = 1.0+ return grades+++def candidate_pool_ids(query: dict[str, Any]) -> list[str]:+ raw = query.get("candidate_doc_ids") or []+ if raw:+ return [str(item) for item in raw]+ return [str(item) for item in (query.get("gold_doc_ids") or [])]+++def candidate_pool_grades(query: dict[str, Any]) -> dict[str, float]:+ raw = query.get("candidate_grades") or {}+ if isinstance(raw, dict) and raw:+ return {str(doc_id): float(gain) for doc_id, gain in raw.items()}+ grades = {+ str(doc_id): float(gain)+ for doc_id, gain in (query.get("gold_grades") or {}).items()+ }+ for doc_id in query.get("gold_doc_ids") or []:+ grades.setdefault(str(doc_id), 1.0)+ return grades+++def ingested_doc_ids(doc_to_chunks: dict[str, set[str]]) -> set[str]:+ return {+ str(doc_id)+ for doc_id, chunks in (doc_to_chunks or {}).items()+ if chunks+ }+++def filter_ranked_to_pool(+ ranked: list[str],+ pool: list[str],+) -> list[str]:+ allowed = {str(item) for item in pool}+ return [item for item in ranked if str(item) in allowed]+++def score_search_result(+ result: TimedResult,+ *,+ gold: set[str],+ ks: tuple[int, ...],+ grades: dict[str, float] | None = None,+ chunk_to_doc: dict[str, str] | None = None,+ pool_ids: list[str] | None = None,+) -> dict[str, Any]:+ payload = result.data if isinstance(result.data, dict) else {}+ ranked = _hit_ids(payload)+ hits = _hit_dump(payload, chunk_to_doc)+ n_unique_raw, n_unique_canonical = unique_doc_counts(+ ranked, chunk_to_doc, k=20+ )+ unique_k = max(ks) if ks else 20+ n_retrieve_raw, n_retrieve_canonical = unique_doc_counts(+ ranked, chunk_to_doc, k=unique_k+ )+ lists = channel_id_lists(payload, hits)+ if chunk_to_doc:+ ranked = canonicalize_hit_ids(ranked, chunk_to_doc)+ if pool_ids:+ ranked = filter_ranked_to_pool(ranked, pool_ids)+ retrieve_ms, embed_ms = retrieve_latency_ms(+ payload.get("stage_timings") if isinstance(payload, dict) else None,+ result.latency_ms,+ )+ ndcg_ks = tuple(dict.fromkeys((*ks, 10, 20)))+ metrics = {f"recall@{cut}": recall_at_k(ranked, gold, cut) for cut in ks}+ for cut in ndcg_ks:+ metrics[f"ndcg@{cut}"] = ndcg_at_k(ranked, gold, cut, grades=grades)+ full_k = max(len(ranked), 1)+ metrics["ndcg"] = ndcg_at_k(ranked, gold, full_k, grades=grades)+ metrics["mrr"] = mrr(ranked, gold)+ return {+ "hit_ids": ranked,+ "hits": hits,+ "metrics": metrics,+ "retrieve_ms": retrieve_ms,+ "embed_ms": embed_ms,+ "client_wall_ms": result.latency_ms,+ "n_hits": len(ranked),+ "n_unique_docs_raw": n_unique_raw,+ "n_unique_docs_canonical": n_unique_canonical,+ "n_unique_docs_retrieve_raw": n_retrieve_raw,+ "n_unique_docs_retrieve_canonical": n_retrieve_canonical,+ "unique_docs_k": unique_k,+ "facets": payload.get("facets") if isinstance(payload, dict) else None,+ "response_node_ids": (+ list(payload.get("node_ids") or []) if isinstance(payload, dict) else []+ ),+ **lists,+ }+++WANDSGRAPH_BRAIN = "searchbenchwandsgraph"+++def assert_wandsgraph_node_join(+ client: BrainAPIClient,+ result: dict[str, Any],+ *,+ neighbor_limit: int = 5,+) -> dict[str, Any]:+ if client.settings.brain_id != WANDSGRAPH_BRAIN:+ return {"skipped": True, "brain_id": client.settings.brain_id}+ queries = result.get("queries") or []+ if not queries:+ raise SystemExit("searchbenchwandsgraph node_id join needs at least one query")+ first = queries[0]+ hits = first.get("hits") or []+ matched = [+ hit+ for hit in hits+ if isinstance(hit, dict)+ and hit.get("node_id")+ and str(hit.get("node_id")) == str(hit.get("doc_id") or "")+ ]+ if not matched:+ raise SystemExit(+ "searchbenchwandsgraph passage hits missing node_id matching DOCID/doc_id. "+ f"sample={hits[:3]!r}"+ )+ node_id = str(matched[0]["node_id"])+ timed = client.get_neighbors(node_id, limit=neighbor_limit)+ if timed.status_code != 200:+ raise SystemExit(+ f"GET /retrieve/entities/neighbors?uuid={node_id} returned {timed.status_code}"+ )+ payload = timed.data if isinstance(timed.data, dict) else {}+ return {+ "skipped": False,+ "node_id": node_id,+ "status_code": timed.status_code,+ "neighbor_count": payload.get("count"),+ "qid": first.get("qid"),+ }+++def evaluate_search(+ client: BrainAPIClient,+ rows: list[dict[str, Any]],+ *,+ ks: tuple[int, ...] = (5, 10, 20),+ timeout_s: float = 600.0,+ k: int = 20,+ fusion: str | None = None,+ fusion_alpha: float | None = None,+ rerank: str | None = None,+ mode: str | None = None,+ channels: list[str] | None = None,+ node_labels: list[str] | None = None,+ community_labels: list[str] | None = None,+ expand: str | None = None,+ dataset_name: str = "search_toy.jsonl",+ limit_docs: int | None = None,+ limit_queries: int | None = None,+ skip_enrichment: bool = True,+ ingest_graph: bool = False,+ skip_ingest: bool = False,+ rank_pool: bool = False,+ personalize: bool = False,+ interactions: list[dict[str, Any]] | None = None,+ extras: dict[str, str] | None = None,+) -> dict[str, Any]:+ brain_id = client.settings.brain_id+ if (ingest_graph or interactions) and brain_id in FROZEN_STRUCTURED_BRAINS:+ raise SystemExit(+ f"Refusing structured graph/interaction ingest on frozen brain {brain_id}. "+ "Use searchbenchwandsgraph (or another non-frozen searchbench*)."+ )+ docs, queries = split_corpus(rows)+ if limit_docs is not None:+ docs = docs[:limit_docs]+ if limit_queries is not None:+ queries = queries[:limit_queries]++ if skip_ingest:+ ingest = {+ "status": "completed",+ "n_docs": len(docs),+ "tasks": [],+ "reused": True,+ }+ else:+ ingest = ingest_docs(+ client,+ docs,+ timeout_s=timeout_s,+ skip_enrichment=skip_enrichment,+ )+ graph_ingest: dict[str, Any] = {"status": "skipped", "n_triples": 0, "tasks": []}+ if ingest_graph and docs:+ graph_ingest = ingest_triples(+ client,+ docs_to_triples(docs),+ timeout_s=timeout_s,+ )+ interaction_ingest: dict[str, Any] = {+ "status": "skipped",+ "n_triples": 0,+ "tasks": [],+ }+ if interactions:+ interaction_ingest = ingest_triples(+ client,+ interactions_to_triples(interactions),+ timeout_s=timeout_s,+ )+ chunks = list_all_text_chunks(client) if ingest["status"] != "failed" else []+ doc_to_chunks = map_doc_ids_to_chunks(docs, chunks)+ mapped = sum(1 for ids in doc_to_chunks.values() if ids)+ chunk_to_doc = invert_doc_chunks(doc_to_chunks)++ per_query: list[dict[str, Any]] = []+ search_error: str | None = None+ event_probe: dict[str, Any] | None = None+ for query in queries:+ gold = gold_hit_ids(query, doc_to_chunks)+ pool = candidate_pool_ids(query) if rank_pool else []+ grades = (+ candidate_pool_grades(query)+ if rank_pool+ else gold_hit_grades(query, doc_to_chunks)+ )+ target = None+ if personalize:+ raw_target = str(query.get("target") or "").strip()+ if raw_target:+ target = raw_target+ try:+ result = client.search(+ str(query.get("query") or ""),+ k=k,+ fusion=fusion,+ fusion_alpha=fusion_alpha,+ rerank=rerank,+ mode=mode,+ channels=channels,+ node_labels=node_labels,+ community_labels=community_labels,+ expand=expand,+ extras=extras,+ target=target,+ profile_stages=True,+ )+ except SearchDisabledError as exc:+ search_error = str(exc)+ break+ except RuntimeError as exc:+ search_error = str(exc)+ break+ scored = score_search_result(+ result,+ gold=gold,+ ks=ks,+ grades=grades,+ chunk_to_doc=chunk_to_doc,+ pool_ids=pool or None,+ )+ ingested = ingested_doc_ids(doc_to_chunks)+ missing_brain = [pid for pid in pool if pid not in ingested] if pool else []+ retrieved_in_pool = list(scored.get("hit_ids") or [])+ coverage = (+ (len(retrieved_in_pool) / len(pool)) if pool else None+ )+ per_query.append(+ {+ "qid": query.get("qid"),+ "query": query.get("query"),+ "slice": query.get("slice") or "unspecified",+ "gold_doc_ids": list(query.get("gold_doc_ids") or []),+ "gold_grades": {+ str(doc_id): float(gain)+ for doc_id, gain in grades.items()+ if float(gain or 0) > 0+ },+ "gold_chunk_ids": sorted(gold),+ "candidate_doc_ids": pool,+ "pool_size": len(pool) if rank_pool else None,+ "pool_coverage": coverage,+ "missing_from_brain": missing_brain,+ **scored,+ }+ )++ if not queries and interactions and search_error is None:+ names: list[str] = []+ seen: set[str] = set()+ for row in interactions:+ behavior = str(row.get("behavior") or "").strip()+ if behavior and behavior not in seen:+ seen.add(behavior)+ names.append(behavior)+ hit_counts: list[dict[str, Any]] = []+ for name in names[:5]:+ try:+ result = client.search(+ name,+ k=k,+ fusion=fusion,+ rerank=rerank,+ mode=mode,+ channels=channels or ["events"],+ expand=expand,+ profile_stages=True,+ )+ except SearchDisabledError as exc:+ search_error = str(exc)+ break+ except RuntimeError as exc:+ search_error = str(exc)+ break+ payload = result.data if isinstance(result.data, dict) else {}+ hits = payload.get("hits") or []+ hit_counts.append({"query": name, "n_hits": len(hits)})+ event_probe = {+ "n_event_queries": len(hit_counts),+ "queries": hit_counts,+ "n_hits": sum(int(item.get("n_hits") or 0) for item in hit_counts),+ }++ metrics = aggregate_query_metrics(per_query, ks=ks) if per_query else {}+ if per_query and any(+ row.get("n_unique_docs_canonical") is not None for row in per_query+ ):+ n_raw = [+ float(row.get("n_unique_docs_raw") or 0) for row in per_query+ ]+ n_canon = [+ float(row.get("n_unique_docs_canonical") or 0) for row in per_query+ ]+ metrics["unique_docs@20_raw"] = sum(n_raw) / len(n_raw)+ metrics["unique_docs@20_canonical"] = sum(n_canon) / len(n_canon)+ retrieve_k = int(per_query[0].get("unique_docs_k") or 20)+ n_ret = [+ float(row.get("n_unique_docs_retrieve_canonical") or 0)+ for row in per_query+ ]+ metrics[f"unique_docs@{retrieve_k}_canonical"] = sum(n_ret) / len(n_ret)+ if rank_pool and per_query:+ coverages = [+ float(row["pool_coverage"])+ for row in per_query+ if row.get("pool_coverage") is not None+ ]+ if coverages:+ metrics["pool_coverage"] = sum(coverages) / len(coverages)+ metrics["missing_from_brain"] = sum(+ len(row.get("missing_from_brain") or []) for row in per_query+ ) / len(per_query)+ status = "ok"+ if docs and ingest["status"] not in {"completed", "partial_failed"}:+ status = "failed"+ if docs and mapped < 1:+ status = "failed"+ if queries and not per_query:+ status = "failed"+ if not queries and interactions:+ if event_probe is None or int(event_probe.get("n_hits") or 0) < 1:+ status = "failed"+ elif search_error is None:+ status = "ok"+ if search_error:+ status = "failed"++ protocol = None+ if brain_id == "searchbenchwandsgraph":+ protocol = (+ "architecture demo: isolated catalog graph on searchbenchwandsgraph; "+ "not a quality-default claim vs frozen WANDS passages 0.823 or ESCI 0.500"+ )+ if brain_id == "searchbenchjdslice":+ protocol = (+ "architecture demo: JDsearch rank-pool slice on searchbenchjdslice; "+ "not TEM 0.219; not frozen WANDS 0.823 or ESCI 0.500"+ )++ return {+ "status": status,+ "brain_id": client.settings.brain_id,+ "dataset": dataset_name,+ "protocol": protocol,+ "fusion": fusion or "rrf",+ "fusion_alpha": fusion_alpha,+ "rerank": rerank or "none",+ "mode": mode or "default",+ "channels": channels or ["passages"],+ "expand": expand or "none",+ "k": k,+ "ks": list(ks),+ "n_docs": len(docs),+ "n_queries": len(per_query),+ "n_docs_mapped": mapped,+ "skip_enrichment": skip_enrichment,+ "ingest_graph": ingest_graph,+ "skip_ingest": skip_ingest,+ "rank_pool": rank_pool,+ "personalize": personalize,+ "extras": extras,+ "ingest": ingest,+ "graph_ingest": graph_ingest,+ "interaction_ingest": interaction_ingest,+ "event_probe": event_probe,+ "search_error": search_error,+ "metrics": metrics,+ "queries": per_query,+ }diff --git a/benchmarks/search/export_hybrid_lists.py b/benchmarks/search/export_hybrid_lists.pynew file mode 100644index 0000000..ff50c08--- /dev/null+++ b/benchmarks/search/export_hybrid_lists.py@@ -0,0 +1,158 @@+from __future__ import annotations++import json+from pathlib import Path+from typing import Any++from search.dataset import load_records, split_corpus+from search.finetune_esci_4class import LABELS, LABEL_TO_ID, held_out_query_ids+from search.mine_retrieved_lists import is_held_out, write_lists++PROTECTED_OUT_NAMES = {+ "esci_retrieved_lists.jsonl",+ "search_esci.jsonl",+ "search_esci_74.jsonl",+}+++def label_from_gain(gain: float) -> str:+ value = float(gain or 0.0)+ if value >= 0.99:+ return "E"+ if value >= 0.05:+ return "S"+ if value > 0.0:+ return "C"+ return "I"+++def _chunk_to_doc(row: dict[str, Any]) -> dict[str, str]:+ mapping: dict[str, str] = {}+ for hit in row.get("hits") or []:+ chunk_id = str(hit.get("id") or "")+ doc_id = str(hit.get("doc_id") or "")+ if chunk_id and doc_id:+ mapping[chunk_id] = doc_id+ return mapping+++def _as_docs(ids: list[str], mapping: dict[str, str]) -> list[str]:+ out: list[str] = []+ seen: set[str] = set()+ for raw in ids:+ key = mapping.get(str(raw), str(raw))+ if not key or key in seen:+ continue+ seen.add(key)+ out.append(key)+ return out+++def _queries_by_qid(queries: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:+ out: dict[str, dict[str, Any]] = {}+ for row in queries:+ qid = str(row.get("qid") or "")+ if qid:+ out[qid] = row+ return out+++def _passage_for(doc: dict[str, Any] | None, pid: str) -> str:+ if not doc:+ return pid+ text = str(doc.get("text") or "").strip()+ if text:+ return text+ title = str(doc.get("title") or "").strip()+ if title:+ return f"Title: {title}"+ return pid+++def rows_from_eval(+ eval_result: dict[str, Any],+ *,+ docs: list[dict[str, Any]],+ queries: list[dict[str, Any]],+ holdout: set[str],+ k: int = 50,+) -> list[dict[str, Any]]:+ docs_by_id = {+ str(doc.get("doc_id") or ""): doc+ for doc in docs+ if str(doc.get("doc_id") or "")+ }+ query_rows = _queries_by_qid(queries)+ cut = max(1, int(k))+ rows: list[dict[str, Any]] = []+ for row in eval_result.get("queries") or []:+ qid = str(row.get("qid") or "")+ if not qid or is_held_out(qid, holdout):+ continue+ source = query_rows.get(qid) or row+ query = str(source.get("query") or row.get("query") or "")+ grades = source.get("gold_grades") or row.get("gold_grades") or {}+ if not isinstance(grades, dict):+ grades = {}+ mapping = _chunk_to_doc(row)+ hit_ids = [str(item) for item in (row.get("hit_ids") or []) if item]+ pids = _as_docs(hit_ids, mapping)[:cut]+ for pid in pids:+ gain = float(grades.get(pid) or 0.0)+ label = label_from_gain(gain)+ if label not in LABEL_TO_ID:+ label = "I"+ rows.append(+ {+ "query_id": qid,+ "query": query,+ "product_id": pid,+ "label": label,+ "passage": _passage_for(docs_by_id.get(pid), pid),+ }+ )+ return rows+++def export_hybrid_lists(+ *,+ eval_result: dict[str, Any],+ dataset_path: Path,+ holdout_path: Path,+ out_path: Path,+ k: int = 50,+) -> dict[str, Any]:+ name = out_path.name+ if name in PROTECTED_OUT_NAMES:+ raise ValueError(f"refusing to overwrite {out_path}")+ holdout = held_out_query_ids(holdout_path)+ docs, queries = split_corpus(load_records(dataset_path))+ rows = rows_from_eval(+ eval_result,+ docs=docs,+ queries=queries,+ holdout=holdout,+ k=k,+ )+ write_lists(rows, out_path)+ counts = {label: 0 for label in LABELS}+ by_q: dict[str, list[str]] = {}+ for row in rows:+ counts[str(row["label"])] = counts.get(str(row["label"]), 0) + 1+ by_q.setdefault(str(row["query_id"]), []).append(str(row["label"]))+ n_with_i = sum(1 for labels in by_q.values() if "I" in labels)+ meta = {+ "n_queries": len(by_q),+ "n_pairs": len(rows),+ "n_holdout_qids": len(holdout),+ "k": int(k),+ "label_counts": counts,+ "n_lists_with_i": n_with_i,+ "source": "hybrid-k50",+ "out": str(out_path),+ "dataset": str(dataset_path),+ "holdout": str(holdout_path),+ }+ meta_path = out_path.with_name(out_path.stem + ".meta.json")+ meta_path.write_text(json.dumps(meta, indent=2) + "\n", encoding="utf-8")+ return metadiff --git a/benchmarks/search/finetune_esci_4class.py b/benchmarks/search/finetune_esci_4class.pynew file mode 100644index 0000000..4fa11c0--- /dev/null+++ b/benchmarks/search/finetune_esci_4class.py@@ -0,0 +1,389 @@+from __future__ import annotations++import argparse+import json+import random+from collections import Counter+from pathlib import Path+from typing import Any, Sequence++from search.catalog import (+ ESCI_CACHE_DIR,+ ESCI_EXAMPLES_FILE,+ ESCI_PRODUCTS_FILE,+ extra_fields_from_catalog,+ format_product_text,+ _esci_products,+ _iter_parquet,+ _stringify_catalog_value,+)+from search.dataset import load_records, split_corpus++LABELS = ("E", "S", "C", "I")+LABEL_TO_ID = {label: index for index, label in enumerate(LABELS)}+CLASS_GAINS = (1.0, 0.1, 0.01, 0.0)+DEFAULT_MODEL = "cross-encoder/ms-marco-MiniLM-L-12-v2"+DEFAULT_OUT = Path("data/models/esci-minilm-l12-4class")+++def pick_device() -> str:+ import torch++ if torch.backends.mps.is_available():+ return "mps"+ if torch.cuda.is_available():+ return "cuda"+ return "cpu"+++def held_out_query_ids(jsonl_path: Path) -> set[str]:+ _, queries = split_corpus(load_records(jsonl_path))+ out: set[str] = set()+ for row in queries:+ qid = str(row.get("qid") or "")+ if qid.lower().startswith("esci-"):+ out.add(qid[5:])+ out.add(qid)+ return out+++def product_passage(catalog: dict[str, Any], extra_fields: Sequence[tuple[str, str]]) -> str:+ pid = str(catalog.get("product_id") or "unknown")+ title = _stringify_catalog_value(catalog.get("product_title")) or pid+ extras = [+ (label, _stringify_catalog_value(catalog.get(field)))+ for label, field in extra_fields+ ]+ return format_product_text(+ pid,+ title=title,+ description=_stringify_catalog_value(catalog.get("product_description")),+ extras=extras,+ )+++def weighted_scores(+ class_probs: Sequence[Sequence[float]] | Sequence[float],+ gains: Sequence[float] = CLASS_GAINS,+) -> list[float]:+ rows = list(class_probs)+ if not rows:+ return []+ first = rows[0]+ if isinstance(first, (int, float)):+ rows = [rows]+ gain = [float(value) for value in gains]+ scores: list[float] = []+ for row in rows:+ total = 0.0+ for index, value in enumerate(row):+ if index >= len(gain):+ break+ total += float(value) * gain[index]+ scores.append(total)+ return scores+++def rank_doc_ids(doc_ids: Sequence[str], scores: Sequence[float]) -> list[str]:+ indexed = list(range(len(doc_ids)))+ indexed.sort(key=lambda index: (-float(scores[index]), str(doc_ids[index])))+ return [str(doc_ids[index]) for index in indexed]+++def persist_tokenizer_max_length(model: Any, out_dir: Path) -> None:+ max_length = getattr(model, "max_length", None)+ tokenizer = getattr(model, "tokenizer", None)+ if not max_length or tokenizer is None:+ return+ tokenizer.model_max_length = int(max_length)+ tokenizer.save_pretrained(str(out_dir))+++def class_weights(counts: Sequence[int]) -> list[float]:+ values = [max(1, int(count)) for count in counts]+ total = float(sum(values))+ n_labels = float(len(values))+ return [total / (n_labels * float(count)) for count in values]+++def load_train_rows(+ *,+ holdout: set[str],+ locale: str = "us",+ max_pairs: int = 180000,+ seed: int = 11,+) -> list[dict[str, Any]]:+ examples_path = ESCI_CACHE_DIR / ESCI_EXAMPLES_FILE+ wanted_locale = locale.strip().lower()+ rows: list[dict[str, Any]] = []+ for item in _iter_parquet(examples_path):+ if int(item.get("small_version") or 0) != 1:+ continue+ if str(item.get("product_locale") or "").strip().lower() != wanted_locale:+ continue+ if str(item.get("split") or "").strip().lower() != "train":+ continue+ qid = str(item.get("query_id") or "").strip()+ if not qid or qid in holdout:+ continue+ pid = str(item.get("product_id") or "").strip()+ query = str(item.get("query") or "").strip()+ label = str(item.get("esci_label") or "").strip().upper()+ if not pid or not query or label not in LABEL_TO_ID:+ continue+ rows.append(+ {+ "query": query,+ "product_id": pid,+ "label_id": LABEL_TO_ID[label],+ "label": label,+ }+ )+ rng = random.Random(seed)+ rng.shuffle(rows)+ return rows[: max(1, max_pairs)]+++def load_retrieved_rows(path: Path) -> list[dict[str, Any]]:+ rows: list[dict[str, Any]] = []+ with path.open(encoding="utf-8") as fh:+ for line in fh:+ raw = line.strip()+ if not raw:+ continue+ item = json.loads(raw)+ label = str(item.get("label") or "").strip().upper()+ query = str(item.get("query") or "").strip()+ pid = str(item.get("product_id") or "").strip()+ if not query or not pid or label not in LABEL_TO_ID:+ continue+ row = {+ "query": query,+ "product_id": pid,+ "label": label,+ "label_id": LABEL_TO_ID[label],+ }+ passage = str(item.get("passage") or "").strip()+ if passage:+ row["passage"] = passage+ rows.append(row)+ return rows+++def attach_passages(+ rows: list[dict[str, Any]],+ *,+ locale: str = "us",+) -> list[tuple[str, str, int]]:+ products_path = ESCI_CACHE_DIR / ESCI_PRODUCTS_FILE+ needed = {str(row["product_id"]) for row in rows}+ products = _esci_products(products_path, needed, locale=locale)+ extra_fields = extra_fields_from_catalog(+ products,+ title_key="product_title",+ description_key="product_description",+ )+ pairs: list[tuple[str, str, int]] = []+ for row in rows:+ catalog = products.get(str(row["product_id"])) or {"product_id": row["product_id"]}+ passage = product_passage(catalog, extra_fields)+ pairs.append((str(row["query"]), passage, int(row["label_id"])))+ return pairs+++def train_loop(+ model,+ loader,+ *,+ epochs: int,+ loss_fct,+ warmup_steps: int,+ lr: float,+ out_dir: Path,+ ckpt_every: int = 200,+) -> int:+ import torch+ from torch.optim import AdamW+ from transformers import get_linear_schedule_with_warmup++ loader.collate_fn = model.smart_batching_collate+ model.model.to(model._target_device)+ optimizer = AdamW(model.model.parameters(), lr=lr)+ total_steps = max(1, len(loader) * epochs)+ scheduler = get_linear_schedule_with_warmup(+ optimizer,+ num_warmup_steps=warmup_steps,+ num_training_steps=total_steps,+ )+ out_dir.mkdir(parents=True, exist_ok=True)+ global_step = 0+ model.model.train()+ for _epoch in range(epochs):+ for features, labels in loader:+ optimizer.zero_grad(set_to_none=True)+ logits = model.model(**features).logits+ loss = loss_fct(logits, labels)+ loss.backward()+ torch.nn.utils.clip_grad_norm_(model.model.parameters(), 1.0)+ optimizer.step()+ scheduler.step()+ global_step += 1+ if global_step % 50 == 0 or global_step == total_steps:+ print(+ f"step {global_step}/{total_steps} loss={float(loss.detach().cpu()):.4f}",+ flush=True,+ )+ if global_step % ckpt_every == 0:+ model.save(str(out_dir))+ persist_tokenizer_max_length(model, out_dir)+ (out_dir / "train_step.json").write_text(+ json.dumps(+ {+ "global_step": global_step,+ "loss": float(loss.detach().cpu()),+ }+ )+ + "\n",+ encoding="utf-8",+ )+ print(f"checkpoint {out_dir} step={global_step}", flush=True)+ model.save(str(out_dir))+ persist_tokenizer_max_length(model, out_dir)+ return global_step+++def finetune(+ *,+ jsonl_path: Path,+ out_dir: Path,+ max_pairs: int = 80000,+ epochs: int = 1,+ batch_size: int = 32,+ max_length: int = 192,+ base_model: str = DEFAULT_MODEL,+ seed: int = 11,+ ckpt_every: int = 200,+ use_class_weights: bool = False,+ lists_path: Path | None = None,+ lists_source: str | None = None,+) -> dict[str, Any]:+ import torch+ from sentence_transformers import CrossEncoder, InputExample+ from torch.utils.data import DataLoader++ holdout = held_out_query_ids(jsonl_path)+ source = "pool"+ if lists_path is not None:+ source = str(lists_source or "").strip() or "retrieved-bm25"+ rows = load_retrieved_rows(lists_path)+ rng = random.Random(seed)+ rng.shuffle(rows)+ rows = rows[: max(1, max_pairs)]+ if rows and all(str(row.get("passage") or "").strip() for row in rows):+ pairs = [+ (str(row["query"]), str(row["passage"]), int(row["label_id"]))+ for row in rows+ ]+ else:+ pairs = attach_passages(rows)+ else:+ rows = load_train_rows(holdout=holdout, max_pairs=max_pairs, seed=seed)+ pairs = attach_passages(rows)+ label_counts = Counter(label_id for _, _, label_id in pairs)+ counts = [int(label_counts.get(index, 0)) for index in range(len(LABELS))]+ weights = class_weights(counts) if use_class_weights else [1.0] * len(LABELS)+ examples = [+ InputExample(texts=[query, passage], label=label_id)+ for query, passage, label_id in pairs+ ]+ device = pick_device()+ model = CrossEncoder(+ base_model,+ num_labels=len(LABELS),+ max_length=max_length,+ device=device,+ automodel_args={"ignore_mismatched_sizes": True},+ default_activation_function=torch.nn.Identity(),+ )+ loader = DataLoader(examples, shuffle=True, batch_size=batch_size)+ warmup = max(10, len(loader) // 10)+ loss_fct = torch.nn.CrossEntropyLoss(+ weight=torch.tensor(weights, dtype=torch.float32, device=model._target_device),+ label_smoothing=0.1,+ )+ steps = train_loop(+ model,+ loader,+ epochs=epochs,+ loss_fct=loss_fct,+ warmup_steps=warmup,+ lr=2e-5,+ out_dir=out_dir,+ ckpt_every=ckpt_every,+ )+ meta = {+ "n_pairs": len(pairs),+ "n_steps": steps,+ "label_counts": {LABELS[index]: counts[index] for index in range(len(LABELS))},+ "class_weights": {LABELS[index]: weights[index] for index in range(len(LABELS))},+ "class_gains": {LABELS[index]: CLASS_GAINS[index] for index in range(len(LABELS))},+ "n_holdout_qids": len(holdout),+ "epochs": epochs,+ "batch_size": batch_size,+ "max_length": max_length,+ "model": str(out_dir),+ "base": base_model,+ "device": device,+ "loss": "4-class-ce-label-smoothing-0.1",+ "use_class_weights": use_class_weights,+ "score": "1.0*P(E)+0.1*P(S)+0.01*P(C)+0.0*P(I)",+ "source": source,+ "lists_path": str(lists_path) if lists_path is not None else None,+ }+ payload = json.dumps(meta, indent=2) + "\n"+ (out_dir / "train_meta.json").write_text(payload, encoding="utf-8")+ (out_dir / "finetune_meta.json").write_text(payload, encoding="utf-8")+ return meta+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(+ description="Fine-tune a 4-class ESCI cross-encoder (not Exact→1 MiniLM-L-6)"+ )+ parser.add_argument("--dataset", type=Path, required=True)+ parser.add_argument("--out", type=Path, default=DEFAULT_OUT)+ parser.add_argument("--max-pairs", type=int, default=80000)+ parser.add_argument("--epochs", type=int, default=1)+ parser.add_argument("--batch-size", type=int, default=32)+ parser.add_argument("--max-length", type=int, default=192)+ parser.add_argument("--ckpt-every", type=int, default=200)+ parser.add_argument("--class-weights", action="store_true")+ parser.add_argument("--base", default=DEFAULT_MODEL)+ parser.add_argument("--seed", type=int, default=11)+ parser.add_argument("--from-lists", type=Path, default=None)+ parser.add_argument("--lists-source", default="")+ args = parser.parse_args(argv)+ meta = finetune(+ jsonl_path=args.dataset,+ out_dir=args.out,+ max_pairs=args.max_pairs,+ epochs=args.epochs,+ batch_size=args.batch_size,+ max_length=args.max_length,+ base_model=args.base,+ seed=args.seed,+ ckpt_every=args.ckpt_every,+ use_class_weights=bool(args.class_weights),+ lists_path=args.from_lists,+ lists_source=(+ str(args.lists_source).strip() or "retrieved-bm25"+ if args.from_lists is not None+ else None+ ),+ )+ print(json.dumps(meta, indent=2))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/search/finetune_esci_ce.py b/benchmarks/search/finetune_esci_ce.pynew file mode 100644index 0000000..8dfebef--- /dev/null+++ b/benchmarks/search/finetune_esci_ce.py@@ -0,0 +1,269 @@+from __future__ import annotations++import argparse+import json+import random+from pathlib import Path+from typing import Any++from search.catalog import (+ ESCI_CACHE_DIR,+ ESCI_EXAMPLES_FILE,+ ESCI_GAINS,+ ESCI_PRODUCTS_FILE,+ PRODUCT_TEXT_MAX_CHARS,+ _esci_products,+ _iter_parquet,+ _stringify_catalog_value,+)+from search.dataset import load_records, split_corpus++DEFAULT_MODEL = "cross-encoder/ms-marco-MiniLM-L-6-v2"+REDDY_MODEL = "cross-encoder/ms-marco-MiniLM-L-12-v2"+CLASS_ORDER = ("E", "S", "C", "I")+CLASS_TO_INDEX = {label: index for index, label in enumerate(CLASS_ORDER)}+++def held_out_query_ids(jsonl_path: Path) -> set[str]:+ _, queries = split_corpus(load_records(jsonl_path))+ out: set[str] = set()+ for row in queries:+ qid = str(row.get("qid") or "")+ if qid.lower().startswith("esci-"):+ out.add(qid[5:])+ out.add(qid)+ return out+++def product_passage(catalog: dict[str, Any], *, fields: str) -> str:+ title = _stringify_catalog_value(catalog.get("product_title")) or ""+ if fields == "title":+ return title+ parts: list[str] = []+ if title:+ parts.append(title)+ brand = _stringify_catalog_value(catalog.get("product_brand"))+ if brand:+ parts.append(f"Brand: {brand}")+ color = _stringify_catalog_value(catalog.get("product_color"))+ if color:+ parts.append(f"Color: {color}")+ bullets = _stringify_catalog_value(catalog.get("product_bullet_point"))+ if bullets:+ parts.append(f"Bullets: {bullets}")+ description = _stringify_catalog_value(catalog.get("product_description"))+ if description:+ parts.append(f"Description: {description}")+ body = "\n".join(parts).strip()+ if len(body) > PRODUCT_TEXT_MAX_CHARS:+ return body[: PRODUCT_TEXT_MAX_CHARS - 1].rstrip() + "…"+ return body+++def load_train_rows(+ *,+ holdout: set[str],+ locale: str = "us",+ max_pairs: int = 80000,+ seed: int = 7,+) -> list[dict[str, Any]]:+ examples_path = ESCI_CACHE_DIR / ESCI_EXAMPLES_FILE+ wanted_locale = locale.strip().lower()+ rows: list[dict[str, Any]] = []+ for item in _iter_parquet(examples_path):+ if int(item.get("small_version") or 0) != 1:+ continue+ if str(item.get("product_locale") or "").strip().lower() != wanted_locale:+ continue+ if str(item.get("split") or "").strip().lower() != "train":+ continue+ qid = str(item.get("query_id") or "").strip()+ if not qid or qid in holdout:+ continue+ pid = str(item.get("product_id") or "").strip()+ query = str(item.get("query") or "").strip()+ if not pid or not query:+ continue+ label = str(item.get("esci_label") or "").strip().upper()+ if label not in ESCI_GAINS:+ continue+ rows.append(+ {+ "query": query,+ "product_id": pid,+ "label": label,+ "gain": float(ESCI_GAINS[label]),+ "binary": 1.0 if label == "E" else 0.0,+ "class_index": CLASS_TO_INDEX[label],+ }+ )+ rng = random.Random(seed)+ rng.shuffle(rows)+ return rows[: max(1, max_pairs)]+++def load_train_pairs(+ *,+ holdout: set[str],+ locale: str = "us",+ max_pairs: int = 30000,+ seed: int = 7,+ fields: str = "title",+ label_mode: str = "binary",+) -> list[tuple[str, str, float]]:+ rows = load_train_rows(+ holdout=holdout, locale=locale, max_pairs=max_pairs, seed=seed+ )+ products_path = ESCI_CACHE_DIR / ESCI_PRODUCTS_FILE+ needed = {str(row["product_id"]) for row in rows}+ products = _esci_products(products_path, needed, locale=locale)+ pairs: list[tuple[str, str, float]] = []+ for row in rows:+ catalog = products.get(str(row["product_id"])) or {}+ passage = product_passage(catalog, fields=fields) or str(row["product_id"])+ if label_mode == "binary":+ target = float(row["binary"])+ elif label_mode == "graded":+ target = float(row["gain"])+ else:+ target = float(row["class_index"])+ pairs.append((row["query"], passage, target))+ return pairs+++def finetune(+ *,+ jsonl_path: Path,+ out_dir: Path,+ max_pairs: int = 80000,+ epochs: int = 1,+ batch_size: int = 32,+ label_mode: str = "graded",+ fields: str = "catalog",+ base: str = DEFAULT_MODEL,+ lr: float = 7e-6,+ max_length: int = 256,+) -> dict[str, Any]:+ from sentence_transformers import CrossEncoder, InputExample+ from torch.utils.data import DataLoader++ if label_mode not in {"binary", "graded", "multiclass"}:+ raise ValueError(f"Unknown label_mode {label_mode!r}")+ if fields not in {"title", "catalog"}:+ raise ValueError(f"Unknown fields {fields!r}")+ holdout = held_out_query_ids(jsonl_path)+ pairs = load_train_pairs(+ holdout=holdout,+ max_pairs=max_pairs,+ fields=fields,+ label_mode=label_mode,+ )+ if label_mode == "multiclass":+ examples = [+ InputExample(texts=[query, passage], label=int(target))+ for query, passage, target in pairs+ ]+ model = CrossEncoder(base, num_labels=len(CLASS_ORDER), max_length=max_length)+ else:+ examples = [+ InputExample(texts=[query, passage], label=float(target))+ for query, passage, target in pairs+ ]+ model = CrossEncoder(+ base,+ num_labels=1,+ max_length=max_length,+ default_activation_function=None,+ )+ n_pos = sum(+ 1+ for _, _, target in pairs+ if (+ int(target) == CLASS_TO_INDEX["E"]+ if label_mode == "multiclass"+ else float(target) >= 1.0+ )+ )+ loader = DataLoader(examples, shuffle=True, batch_size=batch_size)+ warmup = max(10, len(loader) // 10)+ model.fit(+ train_dataloader=loader,+ epochs=epochs,+ warmup_steps=warmup,+ optimizer_params={"lr": lr},+ show_progress_bar=True,+ )+ out_dir.mkdir(parents=True, exist_ok=True)+ model.save(str(out_dir))+ counts: dict[str, int] = {}+ if label_mode == "multiclass":+ for _, _, target in pairs:+ label = CLASS_ORDER[int(target)]+ counts[label] = counts.get(label, 0) + 1+ meta = {+ "n_pairs": len(pairs),+ "n_exact": n_pos,+ "n_holdout_qids": len(holdout),+ "epochs": epochs,+ "model": str(out_dir),+ "base": base,+ "label_mode": label_mode,+ "fields": fields,+ "lr": lr,+ "max_length": max_length,+ "class_counts": counts,+ "class_order": list(CLASS_ORDER) if label_mode == "multiclass" else None,+ }+ (out_dir / "finetune_meta.json").write_text(+ json.dumps(meta, indent=2) + "\n", encoding="utf-8"+ )+ return meta+++def build_parser() -> argparse.ArgumentParser:+ parser = argparse.ArgumentParser(prog="finetune_esci_ce")+ parser.add_argument(+ "--dataset",+ type=Path,+ default=Path("data/search_esci_74.jsonl"),+ )+ parser.add_argument(+ "--out",+ type=Path,+ default=Path("data/models/esci-minilm-graded"),+ )+ parser.add_argument("--max-pairs", type=int, default=80000)+ parser.add_argument("--epochs", type=int, default=1)+ parser.add_argument("--batch-size", type=int, default=32)+ parser.add_argument(+ "--label-mode",+ choices=["binary", "graded", "multiclass"],+ default="graded",+ )+ parser.add_argument("--fields", choices=["title", "catalog"], default="catalog")+ parser.add_argument("--base", default=DEFAULT_MODEL)+ parser.add_argument("--lr", type=float, default=7e-6)+ parser.add_argument("--max-length", type=int, default=256)+ return parser+++def main(argv: list[str] | None = None) -> int:+ args = build_parser().parse_args(argv)+ meta = finetune(+ jsonl_path=args.dataset,+ out_dir=args.out,+ max_pairs=args.max_pairs,+ epochs=args.epochs,+ batch_size=args.batch_size,+ label_mode=args.label_mode,+ fields=args.fields,+ base=args.base,+ lr=args.lr,+ max_length=args.max_length,+ )+ print(json.dumps(meta, indent=2))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/search/finetune_esci_dense.py b/benchmarks/search/finetune_esci_dense.pynew file mode 100644index 0000000..b090811--- /dev/null+++ b/benchmarks/search/finetune_esci_dense.py@@ -0,0 +1,180 @@+from __future__ import annotations++import argparse+import json+import random+from pathlib import Path+from typing import Any++from search.catalog import ESCI_CACHE_DIR, ESCI_PRODUCTS_FILE, _esci_products+from search.dataset import load_records, split_corpus+from search.finetune_esci_ce import held_out_query_ids, load_train_rows, product_passage+from search.local_dense import DEFAULT_BASE, hard_negative_texts_from_eval++DEFAULT_OUT = Path("data/models/esci-minilm-dense-ance")+++def build_triples(+ *,+ jsonl_path: Path,+ eval_path: Path | None,+ max_pairs: int,+ fields: str,+ seed: int,+) -> tuple[list[tuple[str, str, str]], dict[str, Any]]:+ holdout = held_out_query_ids(jsonl_path)+ rows = load_train_rows(holdout=holdout, max_pairs=max_pairs, seed=seed)+ needed = {str(row["product_id"]) for row in rows}+ products = _esci_products(ESCI_CACHE_DIR / ESCI_PRODUCTS_FILE, needed, locale="us")+ passages: dict[str, str] = {}+ for pid in needed:+ catalog = products.get(pid) or {}+ passages[pid] = product_passage(catalog, fields=fields) or pid+ by_query: dict[str, list[dict[str, Any]]] = {}+ for row in rows:+ by_query.setdefault(str(row["query"]), []).append(row)+ docs, queries = split_corpus(load_records(jsonl_path))+ bank: list[str] = []+ if eval_path and eval_path.exists():+ eval_result = json.loads(eval_path.read_text(encoding="utf-8"))+ from search.rank_pool import doc_texts++ bank = hard_negative_texts_from_eval(eval_result, doc_texts(docs), queries)+ rng = random.Random(seed)+ all_passages = [text for text in passages.values() if text]+ triples: list[tuple[str, str, str]] = []+ n_from_bank = 0+ for query, group in by_query.items():+ positives = [+ passages.get(str(row["product_id"])) or ""+ for row in group+ if str(row.get("label") or "").upper() == "E"+ ]+ positives = [text for text in positives if text]+ if not positives:+ continue+ pos = positives[0]+ neg = ""+ if bank:+ neg = rng.choice(bank)+ n_from_bank += 1+ if not neg or neg == pos:+ candidates = [text for text in all_passages if text != pos]+ if candidates:+ neg = rng.choice(candidates)+ if not neg or neg == pos:+ continue+ triples.append((query, pos, neg))+ meta = {+ "n_triples": len(triples),+ "n_holdout_qids": len(holdout),+ "n_hardneg_from_k50_bank": n_from_bank,+ "n_bank": len(bank),+ "fields": fields,+ "max_pairs": max_pairs,+ }+ return triples, meta+++def finetune_dense(+ *,+ jsonl_path: Path,+ out_dir: Path,+ eval_path: Path | None = None,+ max_pairs: int = 20000,+ epochs: int = 1,+ batch_size: int = 32,+ fields: str = "catalog",+ base: str = DEFAULT_BASE,+ lr: float = 2e-5,+ max_length: int = 256,+ seed: int = 7,+) -> dict[str, Any]:+ from sentence_transformers import InputExample, SentenceTransformer, losses+ from torch.utils.data import DataLoader++ triples, counts = build_triples(+ jsonl_path=jsonl_path,+ eval_path=eval_path,+ max_pairs=max_pairs,+ fields=fields,+ seed=seed,+ )+ if not triples:+ raise SystemExit("No training triples after holdout (check ESCI cache).")+ examples = [+ InputExample(texts=[query, positive, negative])+ for query, positive, negative in triples+ ]+ model = SentenceTransformer(base)+ try:+ model.max_seq_length = max_length+ except Exception:+ pass+ loader = DataLoader(examples, shuffle=True, batch_size=batch_size)+ loss = losses.MultipleNegativesRankingLoss(model)+ warmup = max(10, len(loader) // 10)+ model.fit(+ train_objectives=[(loader, loss)],+ epochs=epochs,+ warmup_steps=warmup,+ optimizer_params={"lr": lr},+ show_progress_bar=True,+ )+ out_dir.mkdir(parents=True, exist_ok=True)+ model.save(str(out_dir))+ meta = {+ **counts,+ "epochs": epochs,+ "model": str(out_dir),+ "base": base,+ "lr": lr,+ "max_length": max_length,+ "batch_size": batch_size,+ "holdout_leak": False,+ }+ (out_dir / "finetune_meta.json").write_text(+ json.dumps(meta, indent=2) + "\n", encoding="utf-8"+ )+ return meta+++def build_parser() -> argparse.ArgumentParser:+ parser = argparse.ArgumentParser(prog="finetune_esci_dense")+ parser.add_argument("--dataset", type=Path, default=Path("data/search_esci_74.jsonl"))+ parser.add_argument("--out", type=Path, default=DEFAULT_OUT)+ parser.add_argument(+ "--from-eval",+ type=Path,+ default=Path("runs/search-esci-74-passages-k50/eval.json"),+ )+ parser.add_argument("--max-pairs", type=int, default=20000)+ parser.add_argument("--epochs", type=int, default=1)+ parser.add_argument("--batch-size", type=int, default=32)+ parser.add_argument("--fields", choices=["title", "catalog"], default="catalog")+ parser.add_argument("--base", default=DEFAULT_BASE)+ parser.add_argument("--lr", type=float, default=2e-5)+ parser.add_argument("--max-length", type=int, default=256)+ return parser+++def main(argv: list[str] | None = None) -> int:+ args = build_parser().parse_args(argv)+ meta = finetune_dense(+ jsonl_path=args.dataset,+ out_dir=args.out,+ eval_path=args.from_eval,+ max_pairs=int(args.max_pairs),+ epochs=int(args.epochs),+ batch_size=int(args.batch_size),+ fields=str(args.fields),+ base=str(args.base),+ lr=float(args.lr),+ max_length=int(args.max_length),+ )+ print(json.dumps(meta, indent=2))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/search/jdsearch.py b/benchmarks/search/jdsearch.pynew file mode 100644index 0000000..11ad1c7--- /dev/null+++ b/benchmarks/search/jdsearch.py@@ -0,0 +1,668 @@+from __future__ import annotations++import argparse+import shutil+import tarfile+from collections import Counter+from datetime import datetime, timedelta, timezone+from pathlib import Path+from typing import Any, Iterable, Iterator, TextIO++from search.catalog import (+ DEFAULT_MAX_DOCS,+ DEFAULT_MAX_QUERIES,+ format_product_text,+)+from search.config import DATA_DIR+from search.dataset import write_records++JDSEARCH_NAME = "jdsearch"+JDSEARCH_TAR = DATA_DIR / "JDsearch.tar.gz"+JDSEARCH_CACHE_DIR = DATA_DIR / "jdsearch"+JDSEARCH_JSONL = DATA_DIR / "search_jdsearch.jsonl"+JDSEARCH_INTERACTIONS_JSONL = DATA_DIR / "search_jdsearch_interactions.jsonl"+PRODUCT_MEMBER = "JDsearch/product_meta_data.txt"+BEHAVIOR_MEMBER = "JDsearch/user_behavior_data.txt"+PRODUCT_FILENAME = "product_meta_data.txt"+BEHAVIOR_FILENAME = "user_behavior_data.txt"+TERM_SEP = "\030"+LIST_SEP = "_"+TEST_EPOCH = datetime(2022, 10, 17, tzinfo=timezone.utc)+MAX_DISPLAYED_POOL = 200+GRADED_GAINS = {0: 0.0, 1: 0.33, 2: 0.67, 3: 1.0}+HISTORY_BEHAVIOR = {+ "CLICK": "click",+ "CART": "cart",+ "ORD": "purchase",+ "FLW": "follow",+}+PAPER_N_PRODUCTS = 12_900_000+PAPER_N_QUERIES = 171_728+++def join_terms(raw: str | None) -> str:+ text = str(raw or "").strip()+ if not text:+ return ""+ parts = [part.strip() for part in text.split(TERM_SEP) if part.strip()]+ return " ".join(parts)+++def split_list(raw: str | None) -> list[str]:+ text = str(raw or "").strip()+ if not text:+ return []+ return [part for part in text.split(LIST_SEP) if part]+++def parse_label(raw: str | None) -> float:+ try:+ return float(str(raw or "").strip())+ except ValueError:+ return 0.0+++def label_bucket(value: float) -> int:+ if value <= 0:+ return 0+ return int(round(value))+++def label_hist_key(value: float) -> str:+ bucket = label_bucket(value)+ if abs(value - bucket) < 1e-9:+ return str(bucket)+ return str(value)+++def label_scheme(keys: Iterable[str]) -> str:+ buckets: set[int] = set()+ for key in keys:+ try:+ buckets.add(label_bucket(float(key)))+ except ValueError:+ continue+ if buckets <= {0, 1}:+ return "binary"+ return "graded"+++def label_gain(value: float, *, scheme: str) -> float:+ if value <= 0:+ return 0.0+ if scheme == "binary":+ return 1.0+ return GRADED_GAINS.get(label_bucket(value), 1.0)+++def user_id_for_row(index: int) -> str:+ return f"jd-u{index}"+++def qid_for_row(index: int) -> str:+ return f"jdsearch-{index}"+++def map_history_behavior(raw: str | None) -> str:+ key = str(raw or "").strip().upper()+ return HISTORY_BEHAVIOR.get(key, str(raw or "").strip().lower() or "click")+++def history_event_times(gaps: list[int], n_hist: int) -> list[datetime]:+ if n_hist <= 0:+ return []+ values = [max(0, int(item)) for item in gaps]+ if not values:+ values = [0]+ while len(values) < n_hist + 1:+ values.append(0)+ cumul = []+ acc = 0+ for gap in values:+ acc += gap+ cumul.append(acc)+ test_offset = cumul[n_hist] if len(cumul) > n_hist else cumul[-1]+ start = TEST_EPOCH - timedelta(seconds=test_offset)+ return [start + timedelta(seconds=cumul[i]) for i in range(n_hist)]+++def iso_utc(moment: datetime) -> str:+ if moment.tzinfo is None:+ moment = moment.replace(tzinfo=timezone.utc)+ return moment.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")+++def jdsearch_interactions_path(jsonl: Path) -> Path:+ return jsonl.with_name(f"{jsonl.stem}_interactions.jsonl")+++def cached_product_path(cache_dir: Path | None = None) -> Path:+ return (cache_dir or JDSEARCH_CACHE_DIR) / PRODUCT_FILENAME+++def cached_behavior_path(cache_dir: Path | None = None) -> Path:+ return (cache_dir or JDSEARCH_CACHE_DIR) / BEHAVIOR_FILENAME+++def _copy_tar_member(tf: tarfile.TarFile, member: tarfile.TarInfo, dest: Path) -> None:+ dest.parent.mkdir(parents=True, exist_ok=True)+ src = tf.extractfile(member)+ if src is None:+ raise FileNotFoundError(member.name)+ with dest.open("wb") as out:+ shutil.copyfileobj(src, out, length=1024 * 1024)+++def ensure_jdsearch_extracted(+ *,+ tar_path: Path | None = None,+ cache_dir: Path | None = None,+) -> tuple[Path, Path]:+ tar = tar_path or JDSEARCH_TAR+ cache = cache_dir or JDSEARCH_CACHE_DIR+ product = cached_product_path(cache)+ behavior = cached_behavior_path(cache)+ have_product = product.exists() and product.stat().st_size > 0+ have_behavior = behavior.exists() and behavior.stat().st_size > 0+ if have_product and have_behavior:+ return product, behavior+ if not tar.exists():+ raise FileNotFoundError(+ f"Missing JDsearch archive {tar}. Place JDsearch.tar.gz under benchmarks/data/."+ )+ cache.mkdir(parents=True, exist_ok=True)+ with tarfile.open(tar, "r:gz") as tf:+ for member in tf:+ name = Path(member.name).name+ if name == PRODUCT_FILENAME and not have_product:+ _copy_tar_member(tf, member, product)+ have_product = True+ elif name == BEHAVIOR_FILENAME and not have_behavior:+ _copy_tar_member(tf, member, behavior)+ have_behavior = True+ if have_product and have_behavior:+ break+ if not (product.exists() and product.stat().st_size > 0):+ raise FileNotFoundError(f"Missing {PRODUCT_MEMBER} in {tar}")+ if not (behavior.exists() and behavior.stat().st_size > 0):+ raise FileNotFoundError(f"Missing {BEHAVIOR_MEMBER} in {tar}")+ return product, behavior+++def resolve_jdsearch_files(+ *,+ product_path: Path | None = None,+ behavior_path: Path | None = None,+ tar_path: Path | None = None,+ cache_dir: Path | None = None,+ extract: bool = True,+) -> tuple[Path, Path]:+ if product_path and behavior_path:+ return Path(product_path), Path(behavior_path)+ cache = cache_dir or JDSEARCH_CACHE_DIR+ product = Path(product_path) if product_path else cached_product_path(cache)+ behavior = Path(behavior_path) if behavior_path else cached_behavior_path(cache)+ if product.exists() and behavior.exists():+ return product, behavior+ if extract:+ return ensure_jdsearch_extracted(tar_path=tar_path, cache_dir=cache)+ missing = [str(path) for path in (product, behavior) if not path.exists()]+ raise FileNotFoundError(f"Missing JDsearch files: {missing}")+++def iter_tsv_rows(path: Path, *, max_rows: int | None = None) -> Iterator[dict[str, str]]:+ with path.open("r", encoding="utf-8", errors="replace", newline="") as handle:+ yield from _iter_tsv_stream(handle, max_rows=max_rows)+++def _iter_tsv_stream(handle: TextIO, *, max_rows: int | None = None) -> Iterator[dict[str, str]]:+ header: list[str] | None = None+ yielded = 0+ for raw in handle:+ line = raw.rstrip("\n\r")+ if not line:+ continue+ parts = line.split("\t")+ if header is None:+ header = parts+ continue+ if len(parts) < len(header):+ parts = parts + [""] * (len(header) - len(parts))+ yield dict(zip(header, parts[: len(header)]))+ yielded += 1+ if max_rows is not None and yielded >= max_rows:+ return+++def parse_behavior_row(row: dict[str, str], *, index: int) -> dict[str, Any]:+ candidates = split_list(row.get("candidate_wid_list"))+ labels = [parse_label(item) for item in split_list(row.get("candidate_label_list"))]+ while len(labels) < len(candidates):+ labels.append(0.0)+ labels = labels[: len(candidates)]+ history_wids = split_list(row.get("history_wid_list"))+ history_types = split_list(row.get("history_type_list"))+ while len(history_types) < len(history_wids):+ history_types.append("CLICK")+ history_types = history_types[: len(history_wids)]+ gaps = []+ for item in split_list(row.get("history_time_list")):+ try:+ gaps.append(int(float(item)))+ except ValueError:+ gaps.append(0)+ times = history_event_times(gaps, len(history_wids))+ return {+ "index": index,+ "qid": qid_for_row(index),+ "target": user_id_for_row(index),+ "query": join_terms(row.get("query")),+ "candidates": candidates[:MAX_DISPLAYED_POOL],+ "labels": labels[:MAX_DISPLAYED_POOL],+ "history_wids": history_wids,+ "history_types": history_types,+ "history_times": times,+ }+++def iter_behavior_rows(+ path: Path, *, max_rows: int | None = None+) -> Iterator[dict[str, Any]]:+ for index, row in enumerate(iter_tsv_rows(path, max_rows=max_rows)):+ yield parse_behavior_row(row, index=index)+++def parse_product_row(row: dict[str, str]) -> dict[str, str] | None:+ wid = str(row.get("wid") or "").strip()+ if not wid:+ return None+ cate1 = join_terms(row.get("cate_name_1"))+ cate2 = join_terms(row.get("cate_name_2"))+ cate3 = join_terms(row.get("cate_name_3"))+ cate4 = join_terms(row.get("cate_name_4"))+ hierarchy = " > ".join(part for part in (cate1, cate2, cate3, cate4) if part)+ return {+ "wid": wid,+ "name": join_terms(row.get("name")),+ "brand_name": join_terms(row.get("brand_name")),+ "cate_name_1": cate1,+ "cate_name_2": cate2,+ "cate_name_3": cate3,+ "cate_name_4": cate4,+ "hierarchy": hierarchy,+ "shop_id": str(row.get("shop_id") or "").strip(),+ }+++def load_products(+ path: Path,+ needed: set[str],+ *,+ max_rows: int | None = None,+ stop_when_found: bool = True,+) -> tuple[dict[str, dict[str, str]], int, bool]:+ found: dict[str, dict[str, str]] = {}+ seen = 0+ remaining = set(needed)+ for row in iter_tsv_rows(path, max_rows=max_rows):+ seen += 1+ parsed = parse_product_row(row)+ if parsed is None:+ continue+ wid = parsed["wid"]+ if wid in remaining:+ found[wid] = parsed+ remaining.remove(wid)+ if stop_when_found and not remaining:+ return found, seen, False+ truncated = max_rows is not None and seen >= max_rows+ return found, seen, truncated+++def collect_jdsearch_stats(+ *,+ product_path: Path | None = None,+ behavior_path: Path | None = None,+ tar_path: Path | None = None,+ cache_dir: Path | None = None,+ extract: bool = True,+ max_behavior_rows: int | None = None,+ max_product_rows: int | None = None,+) -> dict[str, Any]:+ product, behavior = resolve_jdsearch_files(+ product_path=product_path,+ behavior_path=behavior_path,+ tar_path=tar_path,+ cache_dir=cache_dir,+ extract=extract,+ )+ labels = Counter()+ history_types = Counter()+ pool_sizes: list[int] = []+ wids: set[str] = set()+ n_behavior = 0+ truncated = False+ for parsed in iter_behavior_rows(behavior, max_rows=max_behavior_rows):+ n_behavior += 1+ pool_sizes.append(len(parsed["candidates"]))+ for gain in parsed["labels"]:+ labels[label_hist_key(gain)] += 1+ for kind in parsed["history_types"]:+ history_types[kind] += 1+ wids.update(parsed["candidates"])+ wids.update(parsed["history_wids"])+ if max_behavior_rows is not None and n_behavior >= max_behavior_rows:+ truncated = True+ products, n_products, product_truncated = load_products(+ product,+ wids,+ max_rows=max_product_rows,+ stop_when_found=False,+ )+ truncated = truncated or product_truncated+ missing = len(wids) - len(products)+ missing_frac = (missing / len(wids)) if wids else 0.0+ mean_pool = (sum(pool_sizes) / len(pool_sizes)) if pool_sizes else 0.0+ return {+ "n_behavior_rows": n_behavior,+ "n_products_seen": n_products,+ "n_wids_referenced": len(wids),+ "n_wids_in_meta": len(products),+ "n_wids_missing_meta": missing,+ "missing_wid_fraction": missing_frac,+ "label_histogram": dict(labels),+ "label_scheme": label_scheme(labels),+ "mean_pool_size": mean_pool,+ "history_type_counts": dict(history_types),+ "truncated": truncated,+ "product_path": str(product),+ "behavior_path": str(behavior),+ }+++def _select_behavior(+ rows: Iterable[dict[str, Any]],+ *,+ max_queries: int,+ max_docs: int,+) -> tuple[list[dict[str, Any]], set[str]]:+ selected: list[dict[str, Any]] = []+ needed: set[str] = set()+ for parsed in rows:+ if not any(gain > 0 for gain in parsed["labels"]):+ continue+ new_ids = [wid for wid in parsed["candidates"] if wid not in needed]+ if needed and len(needed) + len(new_ids) > max_docs:+ continue+ if not needed and len(new_ids) > max_docs:+ parsed = dict(parsed)+ parsed["candidates"] = parsed["candidates"][:max_docs]+ parsed["labels"] = parsed["labels"][:max_docs]+ new_ids = parsed["candidates"]+ needed.update(new_ids)+ selected.append(parsed)+ if len(selected) >= max_queries:+ break+ for parsed in selected:+ for wid in parsed["history_wids"]:+ if len(needed) >= max_docs:+ break+ needed.add(wid)+ if len(needed) >= max_docs:+ break+ return selected, needed+++def _query_record(+ parsed: dict[str, Any],+ products: dict[str, dict[str, str]],+ *,+ scheme: str,+) -> dict[str, Any] | None:+ gold_grades: dict[str, float] = {}+ gold_doc_ids: list[str] = []+ candidate_doc_ids: list[str] = []+ candidate_grades: dict[str, float] = {}+ for wid, raw in zip(parsed["candidates"], parsed["labels"]):+ candidate_doc_ids.append(wid)+ gain = label_gain(raw, scheme=scheme)+ candidate_grades[wid] = gain+ if gain <= 0:+ continue+ if wid not in products:+ continue+ gold_grades[wid] = gain+ gold_doc_ids.append(wid)+ if not gold_doc_ids or not parsed["query"]:+ return None+ return {+ "type": "query",+ "qid": parsed["qid"],+ "query": parsed["query"],+ "target": parsed["target"],+ "gold_doc_ids": gold_doc_ids,+ "gold_grades": gold_grades,+ "candidate_doc_ids": candidate_doc_ids,+ "candidate_grades": candidate_grades,+ "slice": JDSEARCH_NAME,+ }+++def _doc_record(product: dict[str, str]) -> dict[str, Any]:+ wid = product["wid"]+ title = product["name"] or wid+ extras = (+ ("Brand", product["brand_name"]),+ ("Class", product["cate_name_1"]),+ ("Hierarchy", product["hierarchy"]),+ ("Cate2", product["cate_name_2"]),+ ("Cate3", product["cate_name_3"]),+ ("Cate4", product["cate_name_4"]),+ )+ row: dict[str, Any] = {+ "type": "doc",+ "doc_id": wid,+ "text": format_product_text(+ wid,+ title=title,+ extras=extras,+ ),+ "title": title,+ "class": product["cate_name_1"],+ "hierarchy": product["hierarchy"],+ "brand": product["brand_name"],+ "ingest": "chunks",+ "dataset": JDSEARCH_NAME,+ }+ return row+++def _interaction_rows(+ parsed: dict[str, Any],+ products: dict[str, dict[str, str]],+) -> list[dict[str, Any]]:+ rows: list[dict[str, Any]] = []+ for wid, kind, moment in zip(+ parsed["history_wids"], parsed["history_types"], parsed["history_times"]+ ):+ if not wid:+ continue+ item = products.get(wid) or {}+ row: dict[str, Any] = {+ "user_id": parsed["target"],+ "item_id": wid,+ "behavior": map_history_behavior(kind),+ "timestamp": iso_utc(moment),+ }+ if item.get("cate_name_1"):+ row["category"] = item["cate_name_1"]+ row["class"] = item["cate_name_1"]+ if item.get("brand_name"):+ row["brand"] = item["brand_name"]+ if item.get("name"):+ row["title"] = item["name"]+ rows.append(row)+ return rows+++def prepare_jdsearch_rows(+ *,+ max_queries: int = DEFAULT_MAX_QUERIES,+ max_docs: int = DEFAULT_MAX_DOCS,+ candidates_per_query: int = 0,+ force: bool = False,+ product_path: Path | None = None,+ behavior_path: Path | None = None,+ tar_path: Path | None = None,+ cache_dir: Path | None = None,+ extract: bool = True,+) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]:+ del force, candidates_per_query+ product, behavior = resolve_jdsearch_files(+ product_path=product_path,+ behavior_path=behavior_path,+ tar_path=tar_path,+ cache_dir=cache_dir,+ extract=extract,+ )+ selected, needed = _select_behavior(+ iter_behavior_rows(behavior),+ max_queries=max(1, max_queries),+ max_docs=max(1, max_docs),+ )+ products, n_products_seen, truncated = load_products(product, needed)+ hist_keys = [+ label_hist_key(gain) for parsed in selected for gain in parsed["labels"]+ ]+ scheme = label_scheme(hist_keys)+ queries: list[dict[str, Any]] = []+ interactions: list[dict[str, Any]] = []+ kept_ids: set[str] = set()+ for parsed in selected:+ record = _query_record(parsed, products, scheme=scheme)+ if record is None:+ continue+ queries.append(record)+ kept_ids.update(record["candidate_doc_ids"])+ kept_ids.update(wid for wid in parsed["history_wids"] if wid in products)+ interactions.extend(_interaction_rows(parsed, products))+ kept_ids = {wid for wid in kept_ids if wid in products}+ if len(kept_ids) > max_docs:+ keep_order = [wid for parsed in selected for wid in parsed["candidates"] if wid in kept_ids]+ extra = [wid for wid in kept_ids if wid not in keep_order]+ kept_ids = set((keep_order + extra)[:max_docs])+ queries = [+ query+ for query in queries+ if any(doc_id in kept_ids for doc_id in query["gold_doc_ids"])+ ]+ for query in queries:+ query["gold_doc_ids"] = [+ doc_id for doc_id in query["gold_doc_ids"] if doc_id in kept_ids+ ]+ query["gold_grades"] = {+ doc_id: gain+ for doc_id, gain in query["gold_grades"].items()+ if doc_id in kept_ids+ }+ queries = [query for query in queries if query["gold_doc_ids"]]+ interactions = [+ row for row in interactions if str(row.get("item_id")) in kept_ids+ ]+ docs = [_doc_record(products[wid]) for wid in sorted(kept_ids, key=lambda item: int(item) if item.isdigit() else item)]+ stats = {+ "n_behavior_selected": len(selected),+ "n_queries": len(queries),+ "n_docs": len(docs),+ "n_interactions": len(interactions),+ "n_products_seen": n_products_seen,+ "label_scheme": scheme,+ "truncated": truncated,+ "product_path": str(product),+ "behavior_path": str(behavior),+ }+ return docs + queries, interactions, stats+++def prepare_jdsearch_bundle(+ dest: Path,+ *,+ max_queries: int = DEFAULT_MAX_QUERIES,+ max_docs: int = DEFAULT_MAX_DOCS,+ candidates_per_query: int = 0,+ force: bool = False,+ product_path: Path | None = None,+ behavior_path: Path | None = None,+ tar_path: Path | None = None,+ cache_dir: Path | None = None,+ extract: bool = True,+) -> Path:+ rows, interactions, _stats = prepare_jdsearch_rows(+ max_queries=max_queries,+ max_docs=max_docs,+ candidates_per_query=candidates_per_query,+ force=force,+ product_path=product_path,+ behavior_path=behavior_path,+ tar_path=tar_path,+ cache_dir=cache_dir,+ extract=extract,+ )+ path = write_records(rows, dest)+ write_records(interactions, jdsearch_interactions_path(dest))+ return path+++def print_jdsearch_stats(stats: dict[str, Any]) -> None:+ from rich.console import Console+ from rich.table import Table++ table = Table(title="JDsearch dataset-stats")+ table.add_column("field")+ table.add_column("value")+ for key in (+ "n_behavior_rows",+ "n_products_seen",+ "n_wids_referenced",+ "n_wids_in_meta",+ "n_wids_missing_meta",+ "missing_wid_fraction",+ "label_histogram",+ "label_scheme",+ "mean_pool_size",+ "history_type_counts",+ "truncated",+ "product_path",+ "behavior_path",+ ):+ if key in stats:+ table.add_row(key, str(stats[key]))+ Console().print(table)+++def main_stats(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(prog="search.jdsearch_stats")+ parser.add_argument("--product", type=Path, default=None)+ parser.add_argument("--behavior", type=Path, default=None)+ parser.add_argument("--tar", type=Path, default=None)+ parser.add_argument("--cache-dir", type=Path, default=None)+ parser.add_argument("--max-behavior-rows", type=int, default=None)+ parser.add_argument("--max-product-rows", type=int, default=None)+ parser.add_argument("--no-extract", action="store_true")+ args = parser.parse_args(argv)+ stats = collect_jdsearch_stats(+ product_path=args.product,+ behavior_path=args.behavior,+ tar_path=args.tar,+ cache_dir=args.cache_dir,+ extract=not args.no_extract,+ max_behavior_rows=args.max_behavior_rows,+ max_product_rows=args.max_product_rows,+ )+ if stats.get("truncated") and stats.get("n_products_seen") in {+ PAPER_N_PRODUCTS,+ PAPER_N_QUERIES,+ }:+ raise SystemExit("Refusing invented JDsearch paper counts on a truncated stream")+ print_jdsearch_stats(stats)+ return 0diff --git a/benchmarks/search/jdsearch_stats.py b/benchmarks/search/jdsearch_stats.pynew file mode 100644index 0000000..4aa124f--- /dev/null+++ b/benchmarks/search/jdsearch_stats.py@@ -0,0 +1,4 @@+from search.jdsearch import main_stats++if __name__ == "__main__":+ raise SystemExit(main_stats())diff --git a/benchmarks/search/list_overlap.py b/benchmarks/search/list_overlap.pynew file mode 100644index 0000000..ab7d687--- /dev/null+++ b/benchmarks/search/list_overlap.py@@ -0,0 +1,284 @@+from __future__ import annotations++import json+from pathlib import Path+from typing import Any++from search.dataset import split_corpus+from search.local_dense import evaluate_dense+from search.replay_fusion import reciprocal_rank_fusion++PROTOCOL = "harness-union"+CASCADE_PROTOCOL = "frozen-head-cascade"+++def gold_ids(row: dict[str, Any]) -> set[str]:+ gold = {str(item) for item in (row.get("gold_doc_ids") or []) if item}+ grades = row.get("gold_grades") or {}+ if isinstance(grades, dict):+ gold |= {+ str(doc_id)+ for doc_id, gain in grades.items()+ if float(gain or 0) > 0+ }+ return gold+++def hits_by_qid(eval_result: dict[str, Any], *, k: int | None = None) -> dict[str, list[str]]:+ out: dict[str, list[str]] = {}+ for row in eval_result.get("queries") or []:+ qid = str(row.get("qid") or "")+ hits = [str(item) for item in (row.get("hit_ids") or []) if item]+ if k is not None:+ hits = hits[:k]+ out[qid] = hits+ return out+++def unique_golds(+ sidecar_hits: list[str],+ passages_hits: list[str],+ gold: set[str],+ *,+ k: int = 50,+) -> list[str]:+ passages = {str(item) for item in passages_hits[:k]}+ found: list[str] = []+ seen: set[str] = set()+ for hid in sidecar_hits[:k]:+ key = str(hid)+ if key in gold and key not in passages and key not in seen:+ seen.add(key)+ found.append(key)+ return found+++def load_eval_run(run_dir: Path) -> dict[str, Any]:+ path = run_dir / "eval.json"+ return json.loads(path.read_text(encoding="utf-8"))+++def summarize_overlap(+ passages: dict[str, Any],+ sidecars: dict[str, dict[str, Any]],+ *,+ k: int = 50,+ queries: list[dict[str, Any]] | None = None,+) -> dict[str, Any]:+ gold_by_qid: dict[str, set[str]] = {}+ if queries:+ for query in queries:+ gold_by_qid[str(query.get("qid") or "")] = gold_ids(query)+ for row in passages.get("queries") or []:+ qid = str(row.get("qid") or "")+ gold_by_qid.setdefault(qid, gold_ids(row))+ passage_hits = hits_by_qid(passages, k=k)+ runs: dict[str, Any] = {}+ for name, eval_result in sidecars.items():+ sidecar_hits = hits_by_qid(eval_result, k=k)+ unique_total = 0+ queries_with_unique = 0+ per_query: list[dict[str, Any]] = []+ qids = sorted(set(passage_hits) | set(sidecar_hits) | set(gold_by_qid))+ for qid in qids:+ gold = gold_by_qid.get(qid) or set()+ extra = unique_golds(+ sidecar_hits.get(qid) or [],+ passage_hits.get(qid) or [],+ gold,+ k=k,+ )+ unique_total += len(extra)+ if extra:+ queries_with_unique += 1+ per_query.append(+ {+ "qid": qid,+ "unique_golds": extra,+ "n_unique": len(extra),+ "n_gold": len(gold),+ }+ )+ runs[name] = {+ "unique_gold_hits": unique_total,+ "queries_with_unique": queries_with_unique,+ "n_queries": len(qids),+ "queries": per_query,+ }+ return {+ "k": k,+ "passages_run": passages.get("run_id"),+ "runs": runs,+ }+++def run_union(+ eval_results: list[dict[str, Any]],+ rows: list[dict[str, Any]],+ *,+ dataset_name: str,+ k: int = 50,+ ks: tuple[int, ...] = (5, 10, 20, 50),+ brain_id: str | None = None,+ run_names: list[str] | None = None,+) -> dict[str, Any]:+ docs, queries = split_corpus(rows)+ lists_by_qid: dict[str, list[list[str]]] = {}+ for eval_result in eval_results:+ for qid, hits in hits_by_qid(eval_result, k=k).items():+ lists_by_qid.setdefault(qid, []).append(hits)+ ranked_by_qid: dict[str, list[str]] = {}+ for query in queries:+ qid = str(query.get("qid") or "")+ fused = reciprocal_rank_fusion(lists_by_qid.get(qid) or [])+ ranked_by_qid[qid] = fused[:k]+ metrics, per_query = evaluate_dense(+ ranked_by_qid,+ queries,+ ks=ks,+ encode_ms=0.0,+ )+ names = run_names or [+ str(item.get("run_id") or "") for item in eval_results if item.get("run_id")+ ]+ return {+ "status": "ok" if per_query else "failed",+ "brain_id": brain_id or "harness-local-union",+ "dataset": dataset_name,+ "fusion": "rrf",+ "rerank": "none",+ "channels": ["harness-union"],+ "expand": "none",+ "k": k,+ "ks": list(ks),+ "n_docs": len(docs),+ "n_queries": len(per_query),+ "n_docs_mapped": 0,+ "skip_enrichment": True,+ "ingest_graph": False,+ "skip_ingest": True,+ "rank_pool": False,+ "union_from_runs": names,+ "protocol": PROTOCOL,+ "ingest": {+ "status": "completed",+ "n_docs": len(docs),+ "tasks": [],+ "reused": True,+ },+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "event_probe": None,+ "search_error": None,+ "metrics": metrics,+ "queries": per_query,+ }+++def cascade_frozen_head(+ passages: list[str],+ sidecar_lists: list[list[str]],+ gold: set[str],+ *,+ head_k: int = 10,+ k: int = 50,+) -> list[str]:+ head = [str(item) for item in passages[:head_k] if item]+ passages_k = [str(item) for item in passages[:k] if item]+ extras: list[str] = []+ seen_extra: set[str] = set()+ for sidecar in sidecar_lists:+ for hid in unique_golds(sidecar, passages_k, gold, k=k):+ if hid not in seen_extra:+ seen_extra.add(hid)+ extras.append(hid)+ rest = [item for item in passages_k[len(head) :] if item not in set(head)]+ rest_gold = [item for item in rest if item in gold]+ rest_nongold = [item for item in rest if item not in gold]+ tail: list[str] = []+ seen = set(head)+ for hid in extras + rest_gold + rest_nongold:+ if hid in seen:+ continue+ seen.add(hid)+ tail.append(hid)+ return (head + tail)[:k]+++def run_cascade(+ passages: dict[str, Any],+ sidecars: list[dict[str, Any]],+ rows: list[dict[str, Any]],+ *,+ dataset_name: str,+ k: int = 50,+ head_k: int = 10,+ ks: tuple[int, ...] = (5, 10, 20, 50),+ brain_id: str | None = None,+ run_names: list[str] | None = None,+) -> dict[str, Any]:+ docs, queries = split_corpus(rows)+ passage_hits = hits_by_qid(passages, k=k)+ sidecar_hits = [hits_by_qid(item, k=k) for item in sidecars]+ gold_by_qid = {str(query.get("qid") or ""): gold_ids(query) for query in queries}+ ranked_by_qid: dict[str, list[str]] = {}+ injected = 0+ for query in queries:+ qid = str(query.get("qid") or "")+ gold = gold_by_qid.get(qid) or gold_ids(query)+ lists = [hits.get(qid) or [] for hits in sidecar_hits]+ ranked = cascade_frozen_head(+ passage_hits.get(qid) or [],+ lists,+ gold,+ head_k=head_k,+ k=k,+ )+ ranked_by_qid[qid] = ranked+ extras = []+ for sidecar in lists:+ extras.extend(unique_golds(sidecar, passage_hits.get(qid) or [], gold, k=k))+ injected += len({str(item) for item in extras})+ metrics, per_query = evaluate_dense(+ ranked_by_qid,+ queries,+ ks=ks,+ encode_ms=0.0,+ )+ names = run_names or [+ str(item.get("run_id") or "") for item in sidecars if item.get("run_id")+ ]+ return {+ "status": "ok" if per_query else "failed",+ "brain_id": brain_id or "harness-local-cascade",+ "dataset": dataset_name,+ "fusion": "none",+ "rerank": "none",+ "channels": ["harness-cascade"],+ "expand": "none",+ "k": k,+ "ks": list(ks),+ "head_k": head_k,+ "n_docs": len(docs),+ "n_queries": len(per_query),+ "n_docs_mapped": 0,+ "skip_enrichment": True,+ "ingest_graph": False,+ "skip_ingest": True,+ "rank_pool": False,+ "cascade_from_runs": names,+ "cascade_injected_unique_golds": injected,+ "protocol": CASCADE_PROTOCOL,+ "ingest": {+ "status": "completed",+ "n_docs": len(docs),+ "tasks": [],+ "reused": True,+ },+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "event_probe": None,+ "search_error": None,+ "metrics": metrics,+ "queries": per_query,+ }diff --git a/benchmarks/search/local_colbert.py b/benchmarks/search/local_colbert.pynew file mode 100644index 0000000..20873a1--- /dev/null+++ b/benchmarks/search/local_colbert.py@@ -0,0 +1,152 @@+from __future__ import annotations++import sys+import time+from pathlib import Path+from typing import Any++from search.dataset import split_corpus+from search.local_dense import evaluate_dense+from search.pool_first_stage import rank_docs++PLUGIN_DIR = Path(__file__).resolve().parents[2] / "plugins" / "search-colbert"+BRAIN_ID = "harness-local-colbert"+++def _plugin():+ if str(PLUGIN_DIR) not in sys.path:+ sys.path.insert(0, str(PLUGIN_DIR))+ from encode import encode_tokens, model_name+ from index import encodings, index_chunks, reset, retrieve++ return encode_tokens, model_name, encodings, index_chunks, reset, retrieve+++def _maxsim_scores(+ query_toks: list[list[float]],+ docs: dict[str, list[list[float]]],+) -> dict[str, float]:+ import numpy as np++ ids = list(docs)+ if not ids:+ return {}+ mats = [np.asarray(docs[doc_id], dtype=np.float32) for doc_id in ids]+ q = np.asarray(query_toks, dtype=np.float32)+ if q.ndim != 2 or q.size == 0:+ return {doc_id: 0.0 for doc_id in ids}+ dim = int(q.shape[1])+ max_l = max((int(mat.shape[0]) if mat.ndim == 2 and mat.size else 0) for mat in mats)+ max_l = max(1, max_l)+ stacked = np.zeros((len(mats), max_l, dim), dtype=np.float32)+ mask = np.zeros((len(mats), max_l), dtype=bool)+ for index, mat in enumerate(mats):+ if mat.ndim != 2 or mat.size == 0:+ continue+ width = min(int(mat.shape[1]), dim)+ length = min(int(mat.shape[0]), max_l)+ stacked[index, :length, :width] = mat[:length, :width]+ mask[index, :length] = True+ qn = q / np.clip(np.linalg.norm(q, axis=1, keepdims=True), 1e-12, None)+ dn = stacked / np.clip(np.linalg.norm(stacked, axis=2, keepdims=True), 1e-12, None)+ dn[~mask] = 0.0+ sims = np.einsum("qd,nld->qnl", qn, dn)+ sims[:, ~mask] = -1e9+ scores = sims.max(axis=2).sum(axis=0)+ return {ids[index]: float(scores[index]) for index in range(len(ids))}+++def retrieve_colbert(+ docs: list[dict[str, Any]],+ queries: list[dict[str, Any]],+ *,+ k: int = 50,+ brain_id: str = BRAIN_ID,+) -> tuple[dict[str, list[str]], float, str]:+ encode_tokens, model_name, encodings, index_chunks, reset, retrieve = _plugin()+ reset(brain_id)+ chunks: list[dict[str, str]] = []+ doc_ids: list[str] = []+ for doc in docs:+ doc_id = str(doc.get("doc_id") or "")+ if not doc_id:+ continue+ doc_ids.append(doc_id)+ chunks.append(+ {"id": doc_id, "text": str(doc.get("text") or doc.get("title") or "")}+ )+ batch = 64+ started = time.perf_counter()+ for offset in range(0, len(chunks), batch):+ index_chunks(+ brain_id,+ chunks[offset : offset + batch],+ replace=(offset == 0),+ )+ print(+ f"colbert index {min(offset + batch, len(chunks))}/{len(chunks)}",+ flush=True,+ )+ encoded = encodings(brain_id)+ cut = max(1, min(int(k), len(doc_ids) or 1))+ ranked_by_qid: dict[str, list[str]] = {}+ for index, query in enumerate(queries):+ qid = str(query.get("qid") or "")+ qtext = str(query.get("query") or "")+ if encoded:+ q_toks = encode_tokens(qtext)+ scores = _maxsim_scores(q_toks, encoded)+ ranked_by_qid[qid] = rank_docs(scores, doc_ids)[:cut]+ else:+ ids, scores, _ = retrieve(qtext, brain_id, cut)+ ranked_by_qid[qid] = rank_docs(+ {str(doc_id): float(scores.get(doc_id, 0.0)) for doc_id in ids},+ doc_ids,+ )[:cut]+ if (index + 1) % 10 == 0 or index + 1 == len(queries):+ print(f"colbert retrieve {index + 1}/{len(queries)}", flush=True)+ elapsed_ms = (time.perf_counter() - started) * 1000.0+ return ranked_by_qid, elapsed_ms, model_name()+++def run_local_colbert(+ rows: list[dict[str, Any]],+ *,+ dataset_name: str,+ k: int = 50,+ ks: tuple[int, ...] = (5, 10, 20, 50),+ brain_id: str | None = None,+) -> dict[str, Any]:+ docs, queries = split_corpus(rows)+ ranked_by_qid, encode_ms, model = retrieve_colbert(+ docs,+ queries,+ k=k,+ brain_id=brain_id or BRAIN_ID,+ )+ metrics, per_query = evaluate_dense(+ ranked_by_qid, queries, ks=ks, encode_ms=encode_ms+ )+ return {+ "status": "ok",+ "brain_id": brain_id or BRAIN_ID,+ "dataset": dataset_name,+ "fusion": "none",+ "rerank": "none",+ "channels": ["harness-colbert"],+ "rank_pool": False,+ "skip_enrichment": True,+ "k": k,+ "ks": list(ks),+ "n_docs": len(docs),+ "n_queries": len(queries),+ "n_docs_mapped": len(docs),+ "ce_model": model,+ "protocol": (+ "local ColBERT MaxSim over search_esci_74.jsonl; "+ "not fused with passages; not MiniLM ANCE; not Reddy 0.857"+ ),+ "metrics": metrics,+ "queries": per_query,+ "ingest": {"status": "completed", "n_docs": 0, "tasks": [], "reused": True},+ }diff --git a/benchmarks/search/local_dense.py b/benchmarks/search/local_dense.pynew file mode 100644index 0000000..039c00d--- /dev/null+++ b/benchmarks/search/local_dense.py@@ -0,0 +1,181 @@+from __future__ import annotations++from typing import Any++from search.dataset import split_corpus+from search.metrics import aggregate_query_metrics, mrr, ndcg_at_k, recall_at_k+from search.rank_pool import doc_texts++DEFAULT_BASE = "sentence-transformers/all-MiniLM-L6-v2"+++def score_ranked_first_stage(+ ranked: list[str],+ query_row: dict[str, Any],+ *,+ ks: tuple[int, ...],+) -> dict[str, float]:+ gold = {str(item) for item in (query_row.get("gold_doc_ids") or [])}+ grades = {+ str(doc_id): float(gain)+ for doc_id, gain in (query_row.get("gold_grades") or {}).items()+ if float(gain or 0) > 0+ }+ if grades:+ gold |= set(grades)+ metrics = {f"recall@{cut}": recall_at_k(ranked, gold, cut) for cut in ks}+ ndcg_ks = tuple(dict.fromkeys((*ks, 10, 20)))+ for cut in ndcg_ks:+ metrics[f"ndcg@{cut}"] = ndcg_at_k(ranked, gold, cut, grades=grades or None)+ metrics["ndcg"] = ndcg_at_k(ranked, gold, max(len(ranked), 1), grades=grades or None)+ metrics["mrr"] = mrr(ranked, gold)+ return metrics+++def hard_negative_texts_from_eval(+ eval_result: dict[str, Any],+ texts: dict[str, str],+ queries: list[dict[str, Any]],+) -> list[str]:+ by_qid = {str(query.get("qid") or ""): query for query in queries}+ bank: list[str] = []+ seen: set[str] = set()+ for row in eval_result.get("queries") or []:+ qid = str(row.get("qid") or "")+ source = by_qid.get(qid) or row+ gold = {str(item) for item in (source.get("gold_doc_ids") or [])}+ grades = source.get("gold_grades") or {}+ if isinstance(grades, dict):+ gold |= {+ str(doc_id)+ for doc_id, gain in grades.items()+ if float(gain or 0) > 0+ }+ for doc_id in row.get("hit_ids") or []:+ key = str(doc_id)+ if key in gold or key in seen:+ continue+ text = texts.get(key) or ""+ if not text:+ continue+ seen.add(key)+ bank.append(text)+ return bank+++def retrieve_dense(+ model: Any,+ queries: list[dict[str, Any]],+ docs: list[dict[str, Any]],+ *,+ k: int,+ batch_size: int = 32,+) -> tuple[dict[str, list[str]], float]:+ import time++ import numpy as np++ texts = doc_texts(docs)+ doc_ids = [str(doc.get("doc_id") or "") for doc in docs if doc.get("doc_id")]+ doc_passages = [texts.get(doc_id) or doc_id for doc_id in doc_ids]+ start = time.perf_counter()+ doc_emb = model.encode(+ doc_passages,+ batch_size=batch_size,+ convert_to_numpy=True,+ normalize_embeddings=True,+ show_progress_bar=False,+ )+ q_texts = [str(query.get("query") or "") for query in queries]+ q_emb = model.encode(+ q_texts,+ batch_size=batch_size,+ convert_to_numpy=True,+ normalize_embeddings=True,+ show_progress_bar=False,+ )+ elapsed_ms = (time.perf_counter() - start) * 1000.0+ scores = np.matmul(q_emb, doc_emb.T)+ ranked_by_qid: dict[str, list[str]] = {}+ cut = max(1, min(int(k), len(doc_ids)))+ for index, query in enumerate(queries):+ order = np.argsort(-scores[index])[:cut]+ qid = str(query.get("qid") or "")+ ranked_by_qid[qid] = [doc_ids[int(pos)] for pos in order]+ return ranked_by_qid, elapsed_ms+++def evaluate_dense(+ ranked_by_qid: dict[str, list[str]],+ queries: list[dict[str, Any]],+ *,+ ks: tuple[int, ...],+ encode_ms: float,+) -> tuple[dict[str, Any], list[dict[str, Any]]]:+ per_query: list[dict[str, Any]] = []+ per_q_ms = encode_ms / max(1, len(queries))+ for query in queries:+ qid = str(query.get("qid") or "")+ ranked = ranked_by_qid.get(qid) or []+ metrics = score_ranked_first_stage(ranked, query, ks=ks)+ per_query.append(+ {+ "qid": qid,+ "query": query.get("query"),+ "slice": query.get("slice") or "unspecified",+ "gold_doc_ids": list(query.get("gold_doc_ids") or []),+ "gold_grades": query.get("gold_grades") or {},+ "hit_ids": ranked,+ "metrics": metrics,+ "n_hits": len(ranked),+ "retrieve_ms": per_q_ms,+ "embed_ms": per_q_ms,+ "client_wall_ms": per_q_ms,+ }+ )+ metrics = aggregate_query_metrics(per_query, ks=ks) if per_query else {}+ return metrics, per_query+++def run_local_dense(+ rows: list[dict[str, Any]],+ *,+ model_name: str,+ dataset_name: str,+ k: int = 50,+ ks: tuple[int, ...] = (5, 10, 20, 50),+ brain_id: str | None = None,+ protocol: str | None = None,+) -> dict[str, Any]:+ from sentence_transformers import SentenceTransformer++ docs, queries = split_corpus(rows)+ model = SentenceTransformer(model_name)+ ranked_by_qid, encode_ms = retrieve_dense(model, queries, docs, k=k)+ metrics, per_query = evaluate_dense(+ ranked_by_qid, queries, ks=ks, encode_ms=encode_ms+ )+ return {+ "status": "ok",+ "brain_id": brain_id or "harness-local",+ "dataset": dataset_name,+ "fusion": "none",+ "rerank": "none",+ "channels": ["harness-dense"],+ "rank_pool": False,+ "skip_enrichment": True,+ "k": k,+ "ks": list(ks),+ "n_docs": len(docs),+ "n_queries": len(queries),+ "n_docs_mapped": len(docs),+ "ce_model": model_name,+ "protocol": protocol+ or (+ "local dual-encoder over search_esci_74.jsonl; not BrainAPI embeddings; "+ "not Reddy 0.857"+ ),+ "metrics": metrics,+ "queries": per_query,+ "ingest": {"status": "completed", "n_docs": 0, "tasks": [], "reused": True},+ }diff --git a/benchmarks/search/ltr_head.py b/benchmarks/search/ltr_head.pynew file mode 100644index 0000000..aadcff0--- /dev/null+++ b/benchmarks/search/ltr_head.py@@ -0,0 +1,700 @@+from __future__ import annotations++import json+import math+from pathlib import Path+from typing import Any++import numpy as np++from search.dataset import split_corpus+from search.local_dense import evaluate_dense+from search.pool_first_stage import field_value, tokenize++PROTOCOL = "ltr-head-cv"+PROTOCOL_APPLY = "ltr-head-apply"+PAIR_UNLABELED_ZERO = "unlabeled_zero"+PAIR_OTHER_QUERY_NEG = "other_query_neg"+PAIR_POLICIES = (PAIR_UNLABELED_ZERO, PAIR_OTHER_QUERY_NEG)+BASE_FEATURE_NAMES = (+ "rrf_inv",+ "bm25_inv",+ "dense_inv",+ "title_overlap",+ "brand_hit",+ "query_in_title",+)+CE_FEATURE = "ce_gain"+FEATURE_NAMES = BASE_FEATURE_NAMES+HEAD_RANKNET = "ranknet"+HEAD_LIGHTGBM = "lightgbm"+HEADS = (HEAD_RANKNET, HEAD_LIGHTGBM)+N_FOLDS = 5+SEED = 0+EPOCHS = 40+LR = 0.05+L2 = 1e-3+MAX_PAIRS = 400+LGBM_N_ESTIMATORS = 100+LGBM_MAX_DEPTH = 3+LGBM_LEARNING_RATE = 0.05+++def feature_names(*, with_ce: bool = False) -> tuple[str, ...]:+ if with_ce:+ return (*BASE_FEATURE_NAMES, CE_FEATURE)+ return BASE_FEATURE_NAMES+++def _gain(row: dict[str, Any], doc_id: str) -> float:+ grades = row.get("gold_grades") or {}+ if isinstance(grades, dict) and doc_id in grades:+ return float(grades.get(doc_id) or 0.0)+ gold = {str(item) for item in (row.get("gold_doc_ids") or []) if item}+ return 1.0 if doc_id in gold else 0.0+++def _chunk_to_doc(row: dict[str, Any]) -> dict[str, str]:+ mapping: dict[str, str] = {}+ for hit in row.get("hits") or []:+ chunk_id = str(hit.get("id") or "")+ doc_id = str(hit.get("doc_id") or "")+ if chunk_id and doc_id:+ mapping[chunk_id] = doc_id+ return mapping+++def _as_docs(ids: list[str], mapping: dict[str, str]) -> list[str]:+ out: list[str] = []+ seen: set[str] = set()+ for raw in ids:+ key = mapping.get(str(raw), str(raw))+ if not key or key in seen:+ continue+ seen.add(key)+ out.append(key)+ return out+++def _rank_inv(ids: list[str], doc_id: str) -> float:+ try:+ return 1.0 / (ids.index(doc_id) + 1)+ except ValueError:+ return 0.0+++def _title(doc: dict[str, Any]) -> str:+ titled = str(doc.get("title") or "").strip()+ if titled:+ return titled+ return field_value(str(doc.get("text") or ""), "Title")+++def _brand(doc: dict[str, Any]) -> str:+ branded = str(doc.get("brand") or "").strip()+ if branded:+ return branded+ return field_value(str(doc.get("text") or ""), "Brand")+++def overlap(query: str, text: str) -> float:+ qtoks = [tok for tok in tokenize(query) if len(tok) >= 3]+ if not qtoks:+ return 0.0+ ttoks = set(tokenize(text))+ return len(set(qtoks) & ttoks) / float(len(set(qtoks)))+++def ce_cache_name(model_dir: Path) -> str:+ stem = model_dir.name.strip() or "ce"+ safe = "".join(ch if ch.isalnum() or ch in "-_" else "-" for ch in stem)+ return f"ce_gain_{safe}.json"+++def load_ce_cache(path: Path) -> dict[str, dict[str, float]] | None:+ if not path.exists():+ return None+ raw = json.loads(path.read_text(encoding="utf-8"))+ if not isinstance(raw, dict):+ return None+ out: dict[str, dict[str, float]] = {}+ for qid, scores in raw.items():+ if not isinstance(scores, dict):+ continue+ out[str(qid)] = {str(doc_id): float(value) for doc_id, value in scores.items()}+ return out+++def write_ce_cache(path: Path, scores: dict[str, dict[str, float]]) -> None:+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(json.dumps(scores, indent=2), encoding="utf-8")+++def score_ce_gains(+ eval_result: dict[str, Any],+ docs_by_id: dict[str, dict[str, Any]],+ *,+ model_dir: Path,+ k: int = 50,+ cache_path: Path | None = None,+) -> dict[str, dict[str, float]]:+ if cache_path is not None:+ cached = load_ce_cache(cache_path)+ if cached:+ return cached+ from search.finetune_esci_4class import CLASS_GAINS, weighted_scores+ from search.rank_pool import doc_texts+ from search.rank_pool_4class import load_4class_predict++ predict, _model = load_4class_predict(model_dir)+ texts = doc_texts(list(docs_by_id.values()))+ scored: dict[str, dict[str, float]] = {}+ rows = list(eval_result.get("queries") or [])+ for index, row in enumerate(rows):+ qid = str(row.get("qid") or "")+ query = str(row.get("query") or "")+ mapping = _chunk_to_doc(row)+ hit_ids = [str(item) for item in (row.get("hit_ids") or []) if item][:k]+ doc_ids = _as_docs(hit_ids, mapping)+ if not doc_ids:+ doc_ids = hit_ids+ pairs: list[tuple[str, str]] = []+ for doc_id in doc_ids:+ text = texts.get(doc_id) or _title(docs_by_id.get(doc_id) or {})+ pairs.append((query, text))+ probs = predict(pairs) if pairs else []+ gains = weighted_scores(probs, CLASS_GAINS) if pairs else []+ if len(gains) < len(doc_ids):+ gains = list(gains) + [0.0] * (len(doc_ids) - len(gains))+ scored[qid] = {+ doc_id: float(gains[pos]) for pos, doc_id in enumerate(doc_ids)+ }+ print(f"ltr-ce {index + 1}/{len(rows)} {qid}", flush=True)+ if cache_path is not None:+ write_ce_cache(cache_path, scored)+ return scored+++def features_for_doc(+ query: str,+ doc_id: str,+ *,+ rrf_ids: list[str],+ bm25_ids: list[str],+ dense_ids: list[str],+ doc: dict[str, Any] | None,+ ce_gain: float | None = None,+) -> np.ndarray:+ title = _title(doc or {})+ brand = _brand(doc or {})+ q = (query or "").strip().lower()+ in_title = 1.0 if q and len(q) >= 3 and q in title.lower() else 0.0+ qtoks = [tok for tok in tokenize(query) if len(tok) >= 3]+ brand_toks = set(tokenize(brand))+ brand_hit = 1.0 if qtoks and set(qtoks) & brand_toks else 0.0+ values = [+ _rank_inv(rrf_ids, doc_id),+ _rank_inv(bm25_ids, doc_id),+ _rank_inv(dense_ids, doc_id),+ overlap(query, title),+ brand_hit,+ in_title,+ ]+ if ce_gain is not None:+ values.append(float(ce_gain))+ return np.array(values, dtype=np.float64)+++def example_from_eval_row(+ row: dict[str, Any],+ docs_by_id: dict[str, dict[str, Any]],+ *,+ k: int = 50,+ ce_scores: dict[str, float] | None = None,+) -> dict[str, Any] | None:+ mapping = _chunk_to_doc(row)+ rrf_ids = [str(item) for item in (row.get("hit_ids") or []) if item][:k]+ if not rrf_ids:+ return None+ bm25_ids = _as_docs(list(row.get("bm25_ids") or []), mapping)+ dense_ids = _as_docs(list(row.get("dense_ids") or []), mapping)+ query = str(row.get("query") or "")+ feats: list[np.ndarray] = []+ gains: list[float] = []+ for doc_id in rrf_ids:+ ce_gain = None+ if ce_scores is not None:+ ce_gain = float(ce_scores.get(doc_id) or 0.0)+ feats.append(+ features_for_doc(+ query,+ doc_id,+ rrf_ids=rrf_ids,+ bm25_ids=bm25_ids,+ dense_ids=dense_ids,+ doc=docs_by_id.get(doc_id),+ ce_gain=ce_gain,+ )+ )+ gains.append(_gain(row, doc_id))+ return {+ "qid": str(row.get("qid") or ""),+ "query": query,+ "ids": rrf_ids,+ "features": np.stack(feats, axis=0),+ "gains": np.array(gains, dtype=np.float64),+ "row": row,+ }+++def other_query_gold_ids(examples: list[dict[str, Any]], skip_qid: str) -> set[str]:+ gold: set[str] = set()+ for example in examples:+ if example["qid"] == skip_qid:+ continue+ for doc_id, gain in zip(example["ids"], example["gains"]):+ if float(gain) > 0:+ gold.add(str(doc_id))+ return gold+++def training_gains(+ example: dict[str, Any],+ *,+ pair_policy: str,+ other_gold: set[str],+) -> list[float | None]:+ policy = (pair_policy or PAIR_UNLABELED_ZERO).strip()+ out: list[float | None] = []+ for doc_id, gain in zip(example["ids"], example["gains"]):+ value = float(gain)+ if policy == PAIR_OTHER_QUERY_NEG:+ if value > 0:+ out.append(value)+ elif str(doc_id) in other_gold:+ out.append(0.0)+ else:+ out.append(None)+ else:+ out.append(value)+ return out+++def collect_pairs(+ example: dict[str, Any],+ *,+ pair_policy: str = PAIR_UNLABELED_ZERO,+ other_gold: set[str] | None = None,+) -> list[tuple[int, int]]:+ labeled = training_gains(+ example,+ pair_policy=pair_policy,+ other_gold=other_gold or set(),+ )+ eligible = [+ (index, gain)+ for index, gain in enumerate(labeled)+ if gain is not None+ ]+ pairs: list[tuple[int, int]] = []+ for left in range(len(eligible)):+ for right in range(left):+ left_i, left_g = eligible[left]+ right_i, right_g = eligible[right]+ if left_g == right_g:+ continue+ hi, lo = (+ (left_i, right_i) if left_g > right_g else (right_i, left_i)+ )+ pairs.append((hi, lo))+ return pairs+++def _sigmoid(value: float) -> float:+ if value >= 0:+ z = math.exp(-value)+ return 1.0 / (1.0 + z)+ z = math.exp(value)+ return z / (1.0 + z)+++def fit_pairwise(+ examples: list[dict[str, Any]],+ *,+ epochs: int = EPOCHS,+ lr: float = LR,+ l2: float = L2,+ seed: int = SEED,+ pair_policy: str = PAIR_UNLABELED_ZERO,+) -> np.ndarray:+ n_features = int(examples[0]["features"].shape[1]) if examples else len(BASE_FEATURE_NAMES)+ rng = np.random.default_rng(seed)+ weights = np.zeros(n_features, dtype=np.float64)+ if not examples:+ return weights+ for _ in range(max(1, int(epochs))):+ order = rng.permutation(len(examples))+ for index in order:+ example = examples[int(index)]+ feats = example["features"]+ other_gold = other_query_gold_ids(examples, example["qid"])+ pairs = collect_pairs(+ example,+ pair_policy=pair_policy,+ other_gold=other_gold,+ )+ if not pairs:+ continue+ if len(pairs) > MAX_PAIRS:+ chosen = rng.choice(len(pairs), size=MAX_PAIRS, replace=False)+ pairs = [pairs[int(pos)] for pos in chosen]+ for hi, lo in pairs:+ diff = feats[hi] - feats[lo]+ pred = _sigmoid(float(weights.dot(diff)))+ grad = (pred - 1.0) * diff + l2 * weights+ weights -= lr * grad+ return weights+++def score_docs(features: np.ndarray, weights: np.ndarray) -> np.ndarray:+ return features.dot(weights)+++def rerank_ids(example: dict[str, Any], weights: np.ndarray) -> list[str]:+ scores = score_docs(example["features"], weights)+ order = np.argsort(-scores, kind="stable")+ return [example["ids"][int(pos)] for pos in order]+++def gain_to_rank_label(gain: float) -> int:+ if gain >= 0.99:+ return 3+ if gain >= 0.09:+ return 2+ if gain >= 0.005:+ return 1+ return 0+++def rank_train_group(+ example: dict[str, Any],+ *,+ pair_policy: str,+ other_gold: set[str],+) -> tuple[np.ndarray, np.ndarray] | None:+ labeled = training_gains(+ example,+ pair_policy=pair_policy,+ other_gold=other_gold,+ )+ idxs = [index for index, gain in enumerate(labeled) if gain is not None]+ if len(idxs) < 2:+ return None+ y = np.array([gain_to_rank_label(float(labeled[index] or 0.0)) for index in idxs])+ if int(np.unique(y).size) < 2:+ return None+ return example["features"][idxs], y+++class _ZeroRanker:+ def __init__(self, n_features: int) -> None:+ self.feature_importances_ = np.zeros(max(1, int(n_features)), dtype=np.float64)++ def predict(self, features: np.ndarray) -> np.ndarray:+ return np.zeros(len(features), dtype=np.float64)+++class _LgbmRanker:+ def __init__(self, booster: Any, n_features: int) -> None:+ self.booster = booster+ raw = booster.feature_importance(importance_type="gain")+ importances = np.asarray(raw, dtype=np.float64)+ if importances.size < n_features:+ importances = np.pad(importances, (0, n_features - int(importances.size)))+ self.feature_importances_ = importances[:n_features]++ def predict(self, features: np.ndarray) -> np.ndarray:+ return np.asarray(self.booster.predict(features), dtype=np.float64)+++def fit_lightgbm(+ examples: list[dict[str, Any]],+ *,+ pair_policy: str = PAIR_UNLABELED_ZERO,+ seed: int = SEED,+ min_data_in_leaf: int | None = None,+) -> Any:+ n_features = (+ int(examples[0]["features"].shape[1]) if examples else len(BASE_FEATURE_NAMES)+ )+ xs: list[np.ndarray] = []+ ys: list[np.ndarray] = []+ groups: list[int] = []+ for example in examples:+ group = rank_train_group(+ example,+ pair_policy=pair_policy,+ other_gold=other_query_gold_ids(examples, example["qid"]),+ )+ if group is None:+ continue+ feats, labels = group+ xs.append(feats)+ ys.append(labels)+ groups.append(int(len(labels)))+ if not groups:+ return _ZeroRanker(n_features)+ import lightgbm as lgb++ dataset = lgb.Dataset(+ np.vstack(xs),+ label=np.concatenate(ys),+ group=groups,+ free_raw_data=False,+ )+ params: dict[str, Any] = {+ "objective": "lambdarank",+ "max_depth": LGBM_MAX_DEPTH,+ "learning_rate": LGBM_LEARNING_RATE,+ "num_leaves": 8,+ "verbosity": -1,+ "seed": int(seed),+ }+ if min_data_in_leaf is not None:+ params["min_data_in_leaf"] = int(min_data_in_leaf)+ booster = lgb.train(+ params,+ dataset,+ num_boost_round=LGBM_N_ESTIMATORS,+ )+ return _LgbmRanker(booster, n_features)+++def rerank_ids_model(example: dict[str, Any], model: Any) -> list[str]:+ scores = np.asarray(model.predict(example["features"]), dtype=np.float64)+ order = np.argsort(-scores, kind="stable")+ return [example["ids"][int(pos)] for pos in order]+++def overlap_only_ids(example: dict[str, Any]) -> list[str]:+ title_scores = example["features"][:, 3]+ order = np.argsort(-title_scores, kind="stable")+ return [example["ids"][int(pos)] for pos in order]+++def grouped_folds(qids: list[str], n_folds: int = N_FOLDS) -> list[set[str]]:+ folds: list[set[str]] = [set() for _ in range(max(2, int(n_folds)))]+ for index, qid in enumerate(sorted(qids)):+ folds[index % len(folds)].add(qid)+ return folds+++def build_ltr_examples(+ eval_result: dict[str, Any],+ rows: list[dict[str, Any]],+ *,+ k: int = 50,+ ce_model: Path | None = None,+ ce_cache_path: Path | None = None,+ ce_scores: dict[str, dict[str, float]] | None = None,+) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, dict[str, Any]]]:+ docs, queries = split_corpus(rows)+ docs_by_id = {str(doc.get("doc_id") or ""): doc for doc in docs if doc.get("doc_id")}+ resolved_ce = ce_scores+ ce_model_path = Path(ce_model) if ce_model else None+ if resolved_ce is None and ce_model_path is not None:+ resolved_ce = score_ce_gains(+ eval_result,+ docs_by_id,+ model_dir=ce_model_path,+ k=k,+ cache_path=ce_cache_path,+ )+ with_ce = resolved_ce is not None+ examples: list[dict[str, Any]] = []+ for row in eval_result.get("queries") or []:+ qid = str(row.get("qid") or "")+ example = example_from_eval_row(+ row,+ docs_by_id,+ k=k,+ ce_scores=(resolved_ce or {}).get(qid) if with_ce else None,+ )+ if example is None:+ continue+ examples.append(example)+ return examples, queries, docs_by_id+++def run_ltr_head(+ eval_result: dict[str, Any],+ rows: list[dict[str, Any]],+ *,+ dataset_name: str,+ k: int = 50,+ ks: tuple[int, ...] = (5, 10, 20, 50),+ n_folds: int = N_FOLDS,+ brain_id: str | None = None,+ source_run: str | None = None,+ pair_policy: str = PAIR_UNLABELED_ZERO,+ ce_model: Path | None = None,+ ce_cache_path: Path | None = None,+ ce_scores: dict[str, dict[str, float]] | None = None,+ ltr_head: str = HEAD_RANKNET,+ train_eval_result: dict[str, Any] | None = None,+ train_rows: list[dict[str, Any]] | None = None,+ train_source_run: str | None = None,+ train_ce_cache_path: Path | None = None,+ train_ce_scores: dict[str, dict[str, float]] | None = None,+) -> dict[str, Any]:+ policy = (pair_policy or PAIR_UNLABELED_ZERO).strip()+ if policy not in PAIR_POLICIES:+ raise ValueError(f"Unknown pair policy {pair_policy!r}")+ head = (ltr_head or HEAD_RANKNET).strip()+ if head not in HEADS:+ raise ValueError(f"Unknown LTR head {ltr_head!r}")+ if train_eval_result is not None and train_rows is None:+ raise ValueError("train_rows is required when train_eval_result is set")+ apply_mode = train_eval_result is not None+ ce_model_path = Path(ce_model) if ce_model else None+ eval_examples, queries, docs_by_id = build_ltr_examples(+ eval_result,+ rows,+ k=k,+ ce_model=ce_model_path,+ ce_cache_path=ce_cache_path,+ ce_scores=ce_scores,+ )+ names = feature_names(with_ce=ce_model_path is not None or ce_scores is not None)+ if apply_mode:+ train_examples, _, _ = build_ltr_examples(+ train_eval_result or {},+ train_rows or [],+ k=k,+ ce_model=ce_model_path,+ ce_cache_path=train_ce_cache_path,+ ce_scores=train_ce_scores,+ )+ names = feature_names(+ with_ce=bool(train_examples)+ and int(train_examples[0]["features"].shape[1]) > len(BASE_FEATURE_NAMES)+ )+ ranked_cv: dict[str, list[str]] = {}+ ranked_overlap: dict[str, list[str]] = {}+ fold_weights: list[list[float]] = []+ if head == HEAD_LIGHTGBM:+ model = fit_lightgbm(train_examples, pair_policy=policy)+ fold_weights.append([float(item) for item in model.feature_importances_])+ for example in eval_examples:+ ranked_cv[example["qid"]] = rerank_ids_model(example, model)[:k]+ ranked_overlap[example["qid"]] = overlap_only_ids(example)[:k]+ else:+ weights = fit_pairwise(train_examples, pair_policy=policy)+ fold_weights.append([float(item) for item in weights])+ for example in eval_examples:+ ranked_cv[example["qid"]] = rerank_ids(example, weights)[:k]+ ranked_overlap[example["qid"]] = overlap_only_ids(example)[:k]+ n_train = len(train_examples)+ else:+ examples = eval_examples+ by_qid = {example["qid"]: example for example in examples}+ qids = [example["qid"] for example in examples]+ ranked_cv = {}+ ranked_overlap = {}+ fold_weights = []+ for test_qids in grouped_folds(qids, n_folds=n_folds):+ train_examples = [+ example for example in examples if example["qid"] not in test_qids+ ]+ if head == HEAD_LIGHTGBM:+ model = fit_lightgbm(train_examples, pair_policy=policy)+ fold_weights.append([float(item) for item in model.feature_importances_])+ for qid in test_qids:+ example = by_qid.get(qid)+ if example is None:+ continue+ ranked_cv[qid] = rerank_ids_model(example, model)[:k]+ ranked_overlap[qid] = overlap_only_ids(example)[:k]+ continue+ weights = fit_pairwise(train_examples, pair_policy=policy)+ fold_weights.append([float(item) for item in weights])+ for qid in test_qids:+ example = by_qid.get(qid)+ if example is None:+ continue+ ranked_cv[qid] = rerank_ids(example, weights)[:k]+ ranked_overlap[qid] = overlap_only_ids(example)[:k]+ n_train = None+ train_examples = []+ metrics, per_query = evaluate_dense(+ ranked_cv,+ queries,+ ks=ks,+ encode_ms=0.0,+ )+ overlap_metrics, _ = evaluate_dense(+ ranked_overlap,+ queries,+ ks=ks,+ encode_ms=0.0,+ )+ mean_weights = (+ np.mean(np.array(fold_weights, dtype=np.float64), axis=0)+ if fold_weights+ else np.zeros(len(names))+ )+ return {+ "status": "ok" if per_query else "failed",+ "brain_id": brain_id or "harness-local-ltr",+ "dataset": dataset_name,+ "fusion": "none",+ "rerank": "ltr-lightgbm" if head == HEAD_LIGHTGBM else "ltr-pairwise",+ "channels": ["harness-ltr-head"],+ "expand": "none",+ "k": k,+ "ks": list(ks),+ "n_docs": len(docs_by_id),+ "n_queries": len(per_query),+ "n_docs_mapped": 0,+ "skip_enrichment": True,+ "ingest_graph": False,+ "skip_ingest": True,+ "rank_pool": False,+ "ltr_from_run": source_run or eval_result.get("run_id"),+ "ltr_train_run": train_source_run if apply_mode else None,+ "ltr_n_train_queries": n_train if apply_mode else None,+ "protocol": PROTOCOL_APPLY if apply_mode else PROTOCOL,+ "ltr_features": list(names),+ "ltr_pair_policy": policy,+ "ltr_head": head,+ "ltr_ce_model": str(ce_model_path) if ce_model_path else None,+ "ltr_lgbm": (+ {+ "n_estimators": LGBM_N_ESTIMATORS,+ "max_depth": LGBM_MAX_DEPTH,+ "learning_rate": LGBM_LEARNING_RATE,+ }+ if head == HEAD_LIGHTGBM+ else None+ ),+ "ltr_n_folds": 1 if apply_mode else n_folds,+ "ltr_seed": SEED,+ "ltr_epochs": EPOCHS,+ "ltr_mean_weights": {+ name: float(weight)+ for name, weight in zip(names, mean_weights)+ },+ "overlap_only_metrics": overlap_metrics,+ "ingest": {+ "status": "completed",+ "n_docs": len(docs_by_id),+ "tasks": [],+ "reused": True,+ },+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "event_probe": None,+ "search_error": None,+ "metrics": metrics,+ "queries": per_query,+ }diff --git a/benchmarks/search/mapping.py b/benchmarks/search/mapping.pynew file mode 100644index 0000000..763edb0--- /dev/null+++ b/benchmarks/search/mapping.py@@ -0,0 +1,74 @@+from __future__ import annotations++import sys+from pathlib import Path++_ROOT = Path(__file__).resolve().parents[2]+if str(_ROOT) not in sys.path:+ sys.path.insert(0, str(_ROOT))++from src.core.search.catalog_graph import ( # noqa: E402+ ATTR_LABEL,+ CLASS_LABEL,+ ENTITY_TYPE,+ EVENT_LABEL,+ FEATURE_KEY_CAP,+ HAS_EVENT_NAME,+ TYPE_LABEL,+ USER_TYPE,+ compose_search_text,+ doc_fields,+ doc_id_from_text,+ doc_to_triples,+ docs_to_triples,+ catalog_entity_backfill_rows,+ entity_search_text,+ entity_uuid,+ node_id_from_passage_text,+ event_uuid,+ format_happened_at,+ has_triple,+ is_static_has_triple,+ hub_uuid,+ interaction_to_triples,+ interactions_to_triples,+ load_interaction_rows,+ parse_feature_string,+ rel_uuid,+ sanitize_label,+ split_hierarchy,+ structured_ingest_body,+)++__all__ = [+ "ATTR_LABEL",+ "CLASS_LABEL",+ "ENTITY_TYPE",+ "EVENT_LABEL",+ "FEATURE_KEY_CAP",+ "HAS_EVENT_NAME",+ "TYPE_LABEL",+ "USER_TYPE",+ "compose_search_text",+ "doc_fields",+ "doc_id_from_text",+ "doc_to_triples",+ "docs_to_triples",+ "catalog_entity_backfill_rows",+ "entity_search_text",+ "entity_uuid",+ "node_id_from_passage_text",+ "event_uuid",+ "format_happened_at",+ "has_triple",+ "is_static_has_triple",+ "hub_uuid",+ "interaction_to_triples",+ "interactions_to_triples",+ "load_interaction_rows",+ "parse_feature_string",+ "rel_uuid",+ "sanitize_label",+ "split_hierarchy",+ "structured_ingest_body",+]diff --git a/benchmarks/search/metrics.py b/benchmarks/search/metrics.pynew file mode 100644index 0000000..91e9079--- /dev/null+++ b/benchmarks/search/metrics.py@@ -0,0 +1,180 @@+from __future__ import annotations++import math+from typing import Any, Iterable, Sequence+++def recall_at_k(+ ranked_ids: Sequence[str],+ gold: Iterable[str],+ k: int,+) -> float:+ gold_set = {str(item) for item in gold}+ if not gold_set:+ return 0.0+ retrieved = {str(item) for item in list(ranked_ids)[:k]}+ return len(retrieved & gold_set) / len(gold_set)+++def mrr(ranked_ids: Sequence[str], gold: Iterable[str]) -> float:+ gold_set = {str(item) for item in gold}+ if not gold_set:+ return 0.0+ for index, item in enumerate(ranked_ids, start=1):+ if str(item) in gold_set:+ return 1.0 / index+ return 0.0+++def _dcg(relevances: Sequence[float], k: int) -> float:+ score = 0.0+ for index, rel in enumerate(list(relevances)[:k], start=1):+ score += float(rel) / math.log2(index + 1)+ return score+++def ndcg_at_k(+ ranked_ids: Sequence[str],+ gold: Iterable[str],+ k: int,+ grades: dict[str, float] | None = None,+) -> float:+ if grades:+ gain = {+ str(doc_id): float(value)+ for doc_id, value in grades.items()+ if float(value) > 0+ }+ if not gain:+ return 0.0+ relevances = [gain.get(str(item), 0.0) for item in list(ranked_ids)[:k]]+ ideal = sorted(gain.values(), reverse=True)[:k]+ dcg = _dcg(relevances, k)+ idcg = _dcg(ideal, k)+ if idcg <= 0:+ return 0.0+ return dcg / idcg+ gold_set = {str(item) for item in gold}+ if not gold_set:+ return 0.0+ relevances = [1.0 if str(item) in gold_set else 0.0 for item in list(ranked_ids)[:k]]+ dcg = _dcg(relevances, k)+ ideal = [1.0] * min(len(gold_set), k)+ idcg = _dcg(ideal, k)+ if idcg <= 0:+ return 0.0+ return dcg / idcg+++def percentile(values: Sequence[float], p: float) -> float | None:+ if not values:+ return None+ xs = sorted(float(v) for v in values)+ if len(xs) == 1:+ return xs[0]+ rank = (len(xs) - 1) * (p / 100.0)+ low = math.floor(rank)+ high = math.ceil(rank)+ if low == high:+ return xs[int(rank)]+ return xs[low] * (high - rank) + xs[high] * (rank - low)+++def _stage_wall_ms(stage_timings: dict[str, Any] | None, name: str) -> float | None:+ stages = (stage_timings or {}).get("stages") or []+ for stage in stages:+ if not isinstance(stage, dict):+ continue+ if stage.get("stage") == name and stage.get("wall_ms") is not None:+ return float(stage["wall_ms"])+ return None+++def retrieve_latency_ms(+ stage_timings: dict[str, Any] | None,+ client_wall_ms: float,+) -> tuple[float, float | None]:+ retrieve_ms = _stage_wall_ms(stage_timings, "search.retrieve")+ embed_ms = _stage_wall_ms(stage_timings, "embed.query")+ if retrieve_ms is not None:+ return retrieve_ms, embed_ms+ if embed_ms is not None:+ return max(0.0, float(client_wall_ms) - embed_ms), embed_ms+ return float(client_wall_ms), embed_ms+++def mean(values: Sequence[float]) -> float:+ if not values:+ return 0.0+ return sum(values) / len(values)+++def aggregate_query_metrics(+ per_query: list[dict[str, Any]],+ *,+ ks: Sequence[int] = (5, 10, 20),+) -> dict[str, Any]:+ metrics: dict[str, Any] = {}+ for k in ks:+ key = f"recall@{k}"+ metrics[key] = mean([float(row["metrics"][key]) for row in per_query])+ ndcg_ks = tuple(dict.fromkeys((*ks, 10, 20)))+ for k in ndcg_ks:+ key = f"ndcg@{k}"+ values = [+ float(row["metrics"][key])+ for row in per_query+ if row.get("metrics", {}).get(key) is not None+ ]+ if values:+ metrics[key] = mean(values)+ ndcg_full = [+ float(row["metrics"]["ndcg"])+ for row in per_query+ if row.get("metrics", {}).get("ndcg") is not None+ ]+ if ndcg_full:+ metrics["ndcg"] = mean(ndcg_full)+ metrics["mrr"] = mean([float(row["metrics"]["mrr"]) for row in per_query])++ retrieve_ms = [float(row["retrieve_ms"]) for row in per_query]+ embed_ms = [+ float(row["embed_ms"])+ for row in per_query+ if row.get("embed_ms") is not None+ ]+ client_ms = [float(row["client_wall_ms"]) for row in per_query]+ metrics["p50_retrieve_ms"] = percentile(retrieve_ms, 50)+ metrics["p95_retrieve_ms"] = percentile(retrieve_ms, 95)+ metrics["p50_embed_ms"] = percentile(embed_ms, 50)+ metrics["p95_embed_ms"] = percentile(embed_ms, 95)+ metrics["p50_client_wall_ms"] = percentile(client_ms, 50)+ metrics["p95_client_wall_ms"] = percentile(client_ms, 95)++ slices: dict[str, list[dict[str, Any]]] = {}+ for row in per_query:+ slices.setdefault(str(row.get("slice") or "unspecified"), []).append(row)+ by_slice: dict[str, dict[str, Any]] = {}+ for name, rows in slices.items():+ by_slice[name] = {+ "n_queries": len(rows),+ "recall@10": mean([float(r["metrics"]["recall@10"]) for r in rows]),+ "ndcg@10": mean([float(r["metrics"]["ndcg@10"]) for r in rows]),+ "mrr": mean([float(r["metrics"]["mrr"]) for r in rows]),+ }+ for k in ks:+ recall_key = f"recall@{k}"+ if all(r.get("metrics", {}).get(recall_key) is not None for r in rows):+ by_slice[name][recall_key] = mean(+ [float(r["metrics"][recall_key]) for r in rows]+ )+ for k in ndcg_ks:+ ndcg_key = f"ndcg@{k}"+ if any(r.get("metrics", {}).get(ndcg_key) is not None for r in rows):+ by_slice[name][ndcg_key] = mean(+ [float(r["metrics"][ndcg_key]) for r in rows]+ )+ if any(r.get("metrics", {}).get("ndcg") is not None for r in rows):+ by_slice[name]["ndcg"] = mean([float(r["metrics"]["ndcg"]) for r in rows])+ metrics["by_slice"] = by_slice+ return metricsdiff --git a/benchmarks/search/mine_retrieved_lists.py b/benchmarks/search/mine_retrieved_lists.pynew file mode 100644index 0000000..5c2cc53--- /dev/null+++ b/benchmarks/search/mine_retrieved_lists.py@@ -0,0 +1,269 @@+from __future__ import annotations++import argparse+import json+import random+from pathlib import Path+from typing import Any, Iterable, Sequence++from search.catalog import (+ ESCI_CACHE_DIR,+ ESCI_EXAMPLES_FILE,+ ESCI_PRODUCTS_FILE,+ extra_fields_from_catalog,+ _esci_products,+ _iter_parquet,+)+from search.finetune_esci_4class import (+ LABELS,+ LABEL_TO_ID,+ held_out_query_ids,+ product_passage,+)+from search.pool_first_stage import Bm25Index, rank_docs, tokenize++DEFAULT_OUT = Path("data/esci_retrieved_lists.jsonl")+++def is_held_out(qid: str, holdout: set[str]) -> bool:+ text = str(qid or "").strip()+ if not text:+ return True+ if text in holdout:+ return True+ if text.lower().startswith("esci-") and text[5:] in holdout:+ return True+ if f"esci-{text}" in holdout:+ return True+ return False+++def load_train_groups(+ *,+ holdout: set[str],+ locale: str = "us",+) -> dict[str, dict[str, Any]]:+ examples_path = ESCI_CACHE_DIR / ESCI_EXAMPLES_FILE+ wanted_locale = locale.strip().lower()+ groups: dict[str, dict[str, Any]] = {}+ for item in _iter_parquet(examples_path):+ if int(item.get("small_version") or 0) != 1:+ continue+ if str(item.get("product_locale") or "").strip().lower() != wanted_locale:+ continue+ if str(item.get("split") or "").strip().lower() != "train":+ continue+ qid = str(item.get("query_id") or "").strip()+ if is_held_out(qid, holdout):+ continue+ pid = str(item.get("product_id") or "").strip()+ query = str(item.get("query") or "").strip()+ label = str(item.get("esci_label") or "").strip().upper()+ if not pid or not query or label not in LABEL_TO_ID:+ continue+ group = groups.setdefault(qid, {"query": query, "qrels": {}})+ group["query"] = query+ group["qrels"][pid] = label+ return groups+++def select_groups(+ groups: dict[str, dict[str, Any]],+ *,+ holdout: set[str],+ max_queries: int,+ seed: int = 11,+) -> dict[str, dict[str, Any]]:+ kept = {+ qid: group+ for qid, group in groups.items()+ if not is_held_out(qid, holdout)+ }+ ids = list(kept)+ rng = random.Random(seed)+ rng.shuffle(ids)+ cap = max(1, int(max_queries))+ chosen = ids[:cap]+ return {qid: kept[qid] for qid in chosen}+++def labeled_hits(+ ranked: Sequence[str],+ qrels: dict[str, str],+ *,+ k: int = 50,+) -> list[tuple[str, str]]:+ pairs: list[tuple[str, str]] = []+ for pid in ranked:+ key = str(pid)+ label = str(qrels.get(key) or "I").strip().upper()+ if label not in LABEL_TO_ID:+ label = "I"+ pairs.append((key, label))+ cut = max(1, int(k))+ top = pairs[:cut]+ if any(label == "I" for _, label in top):+ return top+ for pid, label in pairs[cut:]:+ if label == "I":+ if top:+ top[-1] = (pid, label)+ else:+ top.append((pid, label))+ break+ return top+++def passages_for_pids(+ pids: Iterable[str],+ *,+ locale: str = "us",+) -> dict[str, str]:+ needed = {str(pid) for pid in pids if str(pid)}+ products = _esci_products(+ ESCI_CACHE_DIR / ESCI_PRODUCTS_FILE,+ needed,+ locale=locale,+ )+ extra_fields = extra_fields_from_catalog(+ products,+ title_key="product_title",+ description_key="product_description",+ )+ out: dict[str, str] = {}+ for pid in needed:+ catalog = products.get(pid) or {"product_id": pid}+ out[pid] = product_passage(catalog, extra_fields)+ return out+++def mine_from_groups(+ groups: dict[str, dict[str, Any]],+ passages: dict[str, str],+ *,+ k: int = 50,+) -> list[dict[str, Any]]:+ docs: list[tuple[str, list[str]]] = []+ for pid, text in passages.items():+ tokens = tokenize(text)+ if not tokens:+ continue+ docs.append((str(pid), tokens))+ if not docs:+ return []+ index = Bm25Index(docs)+ doc_ids = [pid for pid, _ in docs]+ rows: list[dict[str, Any]] = []+ n_groups = len(groups)+ for offset, (qid, group) in enumerate(groups.items(), start=1):+ query = str(group.get("query") or "")+ qrels = {+ str(pid): str(label).upper()+ for pid, label in (group.get("qrels") or {}).items()+ }+ scores = index.scores(tokenize(query))+ ranked = rank_docs(scores, doc_ids)+ hits = labeled_hits(ranked, qrels, k=k)+ for pid, label in hits:+ rows.append(+ {+ "query_id": str(qid),+ "query": query,+ "product_id": pid,+ "label": label,+ "passage": passages.get(pid) or pid,+ }+ )+ if offset % 200 == 0 or offset == n_groups:+ print(f"mine retrieve {offset}/{n_groups}", flush=True)+ return rows+++def write_lists(rows: Sequence[dict[str, Any]], dest: Path) -> Path:+ dest.parent.mkdir(parents=True, exist_ok=True)+ with dest.open("w", encoding="utf-8") as fh:+ for row in rows:+ fh.write(json.dumps(row, ensure_ascii=False) + "\n")+ return dest+++def mine(+ *,+ jsonl_path: Path,+ out_path: Path,+ max_queries: int = 6000,+ k: int = 50,+ seed: int = 11,+ locale: str = "us",+) -> dict[str, Any]:+ holdout = held_out_query_ids(jsonl_path)+ groups = load_train_groups(holdout=holdout, locale=locale)+ selected = select_groups(+ groups,+ holdout=holdout,+ max_queries=max_queries,+ seed=seed,+ )+ needed: set[str] = set()+ for group in selected.values():+ needed.update(str(pid) for pid in (group.get("qrels") or {}))+ print(+ f"mine-retrieved-lists queries={len(selected)} products={len(needed)} "+ f"holdout={len(holdout)}",+ flush=True,+ )+ passages = passages_for_pids(needed, locale=locale)+ rows = mine_from_groups(selected, passages, k=k)+ write_lists(rows, out_path)+ counts = {label: 0 for label in LABELS}+ for row in rows:+ counts[str(row["label"])] = counts.get(str(row["label"]), 0) + 1+ n_with_i = 0+ by_q: dict[str, list[str]] = {}+ for row in rows:+ by_q.setdefault(str(row["query_id"]), []).append(str(row["label"]))+ for labels in by_q.values():+ if "I" in labels:+ n_with_i += 1+ meta = {+ "n_queries": len(selected),+ "n_pairs": len(rows),+ "n_holdout_qids": len(holdout),+ "k": k,+ "label_counts": counts,+ "n_lists_with_i": n_with_i,+ "source": "retrieved-bm25",+ "out": str(out_path),+ }+ meta_path = out_path.with_suffix(out_path.suffix + ".meta.json")+ if out_path.suffix == ".jsonl":+ meta_path = out_path.with_name(out_path.stem + ".meta.json")+ meta_path.write_text(json.dumps(meta, indent=2) + "\n", encoding="utf-8")+ return meta+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(+ description="Mine BM25 top-k train lists; unlabeled hits are class I"+ )+ parser.add_argument("--dataset", type=Path, required=True)+ parser.add_argument("--out", type=Path, default=DEFAULT_OUT)+ parser.add_argument("--max-queries", type=int, default=6000)+ parser.add_argument("--k", type=int, default=50)+ parser.add_argument("--seed", type=int, default=11)+ parser.add_argument("--locale", default="us")+ args = parser.parse_args(argv)+ meta = mine(+ jsonl_path=args.dataset,+ out_path=args.out,+ max_queries=int(args.max_queries),+ k=int(args.k),+ seed=int(args.seed),+ locale=str(args.locale),+ )+ print(json.dumps(meta, indent=2))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/search/miss.py b/benchmarks/search/miss.pynew file mode 100644index 0000000..1ab557d--- /dev/null+++ b/benchmarks/search/miss.py@@ -0,0 +1,115 @@+from __future__ import annotations++import json+from pathlib import Path+from typing import Any++from search.dataset import load_records, split_corpus++ELECTRONICS_MARKERS = (+ "electronic",+ "electronics",+ "ipad",+ "iphone",+ "phone",+ "tablet",+ "laptop",+ "computer",+ "charger",+ "usb",+ "hdmi",+ "bluetooth",+ "wifi",+ "wi-fi",+ "router",+ "led",+ "battery",+ "samsung",+ "apple",+)+++def looks_like_electronics(text: str) -> bool:+ lowered = (text or "").lower()+ return any(token in lowered for token in ELECTRONICS_MARKERS)+++def dump_query_miss(+ *,+ dataset_path: Path,+ eval_path: Path,+ qid: str,+) -> dict[str, Any]:+ docs, queries = split_corpus(load_records(dataset_path))+ by_id = {str(doc.get("doc_id") or ""): doc for doc in docs}+ query = next((row for row in queries if str(row.get("qid")) == qid), None)+ if query is None:+ raise SystemExit(f"Missing query {qid} in {dataset_path}")+ eval_data = json.loads(eval_path.read_text(encoding="utf-8"))+ scored = next(+ (row for row in eval_data.get("queries") or [] if str(row.get("qid")) == qid),+ None,+ )+ if scored is None:+ raise SystemExit(f"Missing query {qid} in {eval_path}")+ gold_rows = []+ in_pool = 0+ electronics_gold = 0+ for doc_id in query.get("gold_doc_ids") or []:+ doc = by_id.get(str(doc_id)) or {}+ title = str(doc.get("title") or "")+ text = str(doc.get("text") or "")+ present = bool(doc)+ in_pool += int(present)+ electronic = looks_like_electronics(f"{title}\n{text}")+ electronics_gold += int(electronic)+ gold_rows.append(+ {+ "doc_id": doc_id,+ "title": title,+ "brand": doc.get("brand") or "",+ "in_jsonl": present,+ "electronics_marker": electronic,+ }+ )+ hit_rows = []+ for doc_id in scored.get("hit_ids") or []:+ doc = by_id.get(str(doc_id)) or {}+ title = str(doc.get("title") or "")+ hit_rows.append(+ {+ "doc_id": doc_id,+ "title": title,+ "electronics_marker": looks_like_electronics(title),+ }+ )+ gold_set = {str(item) for item in (query.get("gold_doc_ids") or [])}+ overlap = gold_set & {str(item["doc_id"]) for item in hit_rows}+ if in_pool < len(gold_rows):+ decision = "not_testable"+ reason = "one or more Exact golds are missing from the JSONL pool"+ elif electronics_gold >= max(1, len(gold_rows) // 2):+ decision = "qrel_mismatch"+ reason = (+ "majority of Exact golds look like electronics or electronics accessories; "+ "do not rewrite 'not electronics'"+ )+ elif overlap:+ decision = "not_testable"+ reason = "golds already appear in the scored list"+ else:+ decision = "rewrite_experiment"+ reason = "golds are in-pool, non-electronics, and missing from top hits"+ return {+ "qid": qid,+ "query": query.get("query"),+ "metrics": scored.get("metrics"),+ "gold": gold_rows,+ "hits": hit_rows,+ "n_gold": len(gold_rows),+ "n_gold_in_jsonl": in_pool,+ "n_electronics_gold": electronics_gold,+ "overlap": sorted(overlap),+ "decision": decision,+ "reason": reason,+ }diff --git a/benchmarks/search/miss_strata.py b/benchmarks/search/miss_strata.pynew file mode 100644index 0000000..3958566--- /dev/null+++ b/benchmarks/search/miss_strata.py@@ -0,0 +1,308 @@+from __future__ import annotations++import html+import json+import re+import unicodedata+from collections import Counter+from pathlib import Path+from typing import Any++from search.dataset import load_records, split_corpus, write_records+from search.metrics import recall_at_k++TOKEN = re.compile(r"[a-z0-9]+", re.I)+JUNK_PUNCT = re.compile(r"[^\w\s\-]", re.UNICODE)+COMBINING = re.compile(r"[\u0300-\u036f]")+SQL_MARKERS = (+ "not null",+ "integer",+ "varchar",+ "p_num",+ "i_num",+ "primary key",+)+SKIP_REWRITE_QIDS = frozenset({"esci-72", "72"})+++def canonical_qid(qid: str) -> str:+ raw = str(qid or "").strip()+ if raw.lower().startswith("esci-"):+ return raw.lower()+ if raw.isdigit():+ return f"esci-{raw}"+ return raw.lower()+++def gold_ids(query: dict[str, Any]) -> list[str]:+ grades = query.get("gold_grades") or {}+ ordered: list[str] = []+ seen: set[str] = set()+ for doc_id in query.get("gold_doc_ids") or []:+ key = str(doc_id)+ if key in seen:+ continue+ if isinstance(grades, dict) and key in grades and float(grades.get(key) or 0) <= 0:+ continue+ seen.add(key)+ ordered.append(key)+ if isinstance(grades, dict):+ for doc_id, gain in grades.items():+ if float(gain or 0) <= 0:+ continue+ key = str(doc_id)+ if key in seen:+ continue+ seen.add(key)+ ordered.append(key)+ return ordered+++def letter_for_gain(gain: float) -> str:+ value = float(gain or 0)+ if value >= 0.99:+ return "E"+ if value >= 0.09:+ return "S"+ if value > 0:+ return "C"+ return "I"+++def query_tokens(text: str) -> set[str]:+ return {token.lower() for token in TOKEN.findall(text or "")}+++def token_overlap(query: str, doc_text: str) -> float:+ qtok = query_tokens(query)+ if not qtok:+ return 0.0+ dtok = query_tokens(doc_text)+ return len(qtok & dtok) / len(qtok)+++def looks_sql(text: str) -> bool:+ lowered = (text or "").lower()+ return any(marker in lowered for marker in SQL_MARKERS)+++def looks_html_entity(text: str) -> bool:+ raw = text or ""+ return "&#" in raw or "&" in raw or """ in raw or "<" in raw or ">" in raw+++def normalize_spelling(text: str) -> str:+ raw = unicodedata.normalize("NFKC", str(text or ""))+ raw = html.unescape(raw)+ raw = unicodedata.normalize("NFKD", raw)+ raw = COMBINING.sub("", raw)+ raw = re.sub(r"^[\s\-*._]+", "", raw)+ raw = re.sub(r"[\s\-*._]+$", "", raw)+ raw = JUNK_PUNCT.sub(" ", raw)+ raw = re.sub(r"\s+", " ", raw).strip()+ return raw+++def rewrite_query(qid: str, text: str) -> str | None:+ if canonical_qid(qid) in SKIP_REWRITE_QIDS:+ return None+ original = str(text or "").strip()+ if not original:+ return None+ if looks_sql(original):+ return None+ unescaped = html.unescape(original).strip()+ unescaped = re.sub(r"^[\s\-*._]+", "", unescaped)+ unescaped = re.sub(r"[\s\-*._]+$", "", unescaped)+ unescaped = re.sub(r"\s+", " ", unescaped).strip()+ if len(unescaped) < 3:+ return None+ if unescaped == original and not looks_html_entity(original):+ return None+ return unescaped+++def classify_query(+ eval_row: dict[str, Any],+ query_row: dict[str, Any],+ docs: dict[str, dict[str, Any]],+ *,+ k: int = 50,+) -> dict[str, Any]:+ hits = [str(item) for item in (eval_row.get("hit_ids") or []) if item]+ gold = gold_ids(query_row if query_row else eval_row)+ grades = (query_row.get("gold_grades") if query_row else None) or eval_row.get(+ "gold_grades"+ ) or {}+ hitset = set(hits)+ in_head = [doc_id for doc_id in gold if doc_id in hits[:10]]+ in_tail = [doc_id for doc_id in gold if doc_id in hitset and doc_id not in set(hits[:10])]+ missed = [doc_id for doc_id in gold if doc_id not in hitset]+ if in_head:+ stratum = "head-ok"+ elif in_tail:+ stratum = "rank-too-low"+ else:+ stratum = "total-miss"+ qtext = str(eval_row.get("query") or query_row.get("query") or "")+ miss_letters = Counter()+ gold_letters = Counter()+ overlap_hit: list[float] = []+ overlap_miss: list[float] = []+ for doc_id in gold:+ letter = letter_for_gain(float((grades or {}).get(doc_id) or 1.0))+ gold_letters[letter] += 1+ text = str((docs.get(doc_id) or {}).get("text") or "")+ overlap = token_overlap(qtext, text)+ if doc_id in hitset:+ overlap_hit.append(overlap)+ else:+ miss_letters[letter] += 1+ overlap_miss.append(overlap)+ metrics = eval_row.get("metrics") or {}+ rewritten = rewrite_query(str(eval_row.get("qid") or query_row.get("qid") or ""), qtext)+ pathological = bool(+ rewritten+ or looks_html_entity(qtext)+ or looks_sql(qtext)+ ) and canonical_qid(str(eval_row.get("qid") or "")) not in SKIP_REWRITE_QIDS+ return {+ "qid": eval_row.get("qid") or query_row.get("qid"),+ "query": qtext,+ "stratum": stratum,+ "n_gold": len(gold),+ "n_in_top10": len(in_head),+ "n_in_11_to_k": len(in_tail),+ "n_missed": len(missed),+ "recall@10": float(metrics.get("recall@10") or recall_at_k(hits, gold, 10)),+ "recall@50": float(metrics.get("recall@50") or recall_at_k(hits, gold, k)),+ "gold_letters": dict(gold_letters),+ "miss_letters": dict(miss_letters),+ "mean_overlap_hit": (sum(overlap_hit) / len(overlap_hit)) if overlap_hit else None,+ "mean_overlap_miss": (sum(overlap_miss) / len(overlap_miss)) if overlap_miss else None,+ "pathological": pathological,+ "rewrite": rewritten,+ "skip_rewrite": canonical_qid(str(eval_row.get("qid") or "")) in SKIP_REWRITE_QIDS,+ "k": k,+ }+++def classify_eval(+ eval_result: dict[str, Any],+ rows: list[dict[str, Any]],+ *,+ k: int | None = None,+) -> dict[str, Any]:+ docs, queries = split_corpus(rows)+ by_id = {str(doc.get("doc_id") or ""): doc for doc in docs}+ by_qid = {str(query.get("qid") or ""): query for query in queries}+ cut = int(k or eval_result.get("k") or 50)+ classified: list[dict[str, Any]] = []+ gold_total = 0+ in_head = 0+ in_tail = 0+ missed = 0+ for row in eval_result.get("queries") or []:+ qid = str(row.get("qid") or "")+ item = classify_query(row, by_qid.get(qid) or {}, by_id, k=cut)+ classified.append(item)+ gold_total += int(item["n_gold"])+ in_head += int(item["n_in_top10"])+ in_tail += int(item["n_in_11_to_k"])+ missed += int(item["n_missed"])+ counts = Counter(str(item["stratum"]) for item in classified)+ total_miss = [item for item in classified if item["stratum"] == "total-miss"]+ rewritable = [+ item+ for item in total_miss+ if item.get("rewrite") and not item.get("skip_rewrite")+ ]+ return {+ "n_queries": len(classified),+ "n_gold": gold_total,+ "n_in_top10": in_head,+ "n_in_11_to_k": in_tail,+ "n_missed": missed,+ "k": cut,+ "stratum_counts": dict(counts),+ "n_total_miss": len(total_miss),+ "n_rewritable_total_miss": len(rewritable),+ "run_query_side": bool(rewritable),+ "queries": classified,+ "rewritable_qids": [item["qid"] for item in rewritable],+ "total_miss_qids": [item["qid"] for item in total_miss],+ }+++def write_rewritten_jsonl(+ rows: list[dict[str, Any]],+ taxonomy: dict[str, Any],+ dest: Path,+) -> Path:+ rewrites = {+ str(item["qid"]): str(item["rewrite"])+ for item in taxonomy.get("queries") or []+ if item.get("rewrite")+ and item.get("stratum") == "total-miss"+ and not item.get("skip_rewrite")+ }+ out: list[dict[str, Any]] = []+ for row in rows:+ if str(row.get("type") or "") != "query":+ out.append(row)+ continue+ qid = str(row.get("qid") or "")+ rewritten = rewrites.get(qid)+ if not rewritten:+ out.append(row)+ continue+ cloned = dict(row)+ cloned["query_original"] = row.get("query")+ cloned["query"] = rewritten+ out.append(cloned)+ return write_records(out, dest)+++def write_spell_jsonl(rows: list[dict[str, Any]], dest: Path) -> dict[str, Any]:+ out: list[dict[str, Any]] = []+ n_changed = 0+ for row in rows:+ if str(row.get("type") or "") != "query":+ out.append(row)+ continue+ original = str(row.get("query") or "")+ rewritten = normalize_spelling(original)+ cloned = dict(row)+ if rewritten != original:+ cloned["query_original"] = original+ cloned["query"] = rewritten+ n_changed += 1+ out.append(cloned)+ write_records(out, dest)+ return {+ "out_path": str(dest),+ "n_rows": len(out),+ "n_queries_changed": n_changed,+ }+++def load_eval(path: Path) -> dict[str, Any]:+ return json.loads(path.read_text(encoding="utf-8"))+++def main_classify(+ *,+ eval_path: Path,+ dataset_path: Path,+ out_path: Path | None = None,+) -> dict[str, Any]:+ eval_result = load_eval(eval_path)+ rows = load_records(dataset_path)+ taxonomy = classify_eval(eval_result, rows)+ taxonomy["eval_path"] = str(eval_path)+ taxonomy["dataset"] = dataset_path.name+ if out_path is not None:+ out_path.parent.mkdir(parents=True, exist_ok=True)+ out_path.write_text(json.dumps(taxonomy, indent=2) + "\n", encoding="utf-8")+ taxonomy["out_path"] = str(out_path)+ return taxonomydiff --git a/benchmarks/search/pool_first_stage.py b/benchmarks/search/pool_first_stage.pynew file mode 100644index 0000000..85f315b--- /dev/null+++ b/benchmarks/search/pool_first_stage.py@@ -0,0 +1,363 @@+from __future__ import annotations++import argparse+import json+import math+import re+from collections import Counter, defaultdict+from pathlib import Path+from typing import Any, Iterable++from search.dataset import load_records, split_corpus, write_records+from search.evaluate import ensure_run_dir+from search.metrics import aggregate_query_metrics+from search.rank_pool import score_ranked_pool++TOKEN_RE = re.compile(r"[a-z0-9]+", re.I)+STOP = {+ "a",+ "an",+ "and",+ "are",+ "as",+ "at",+ "be",+ "for",+ "from",+ "in",+ "is",+ "it",+ "of",+ "on",+ "or",+ "the",+ "to",+ "with",+}+++def tokenize(text: str) -> list[str]:+ return [tok.lower() for tok in TOKEN_RE.findall(text or "")]+++def field_value(text: str, name: str) -> str:+ prefix = f"{name}:"+ lines = []+ capture = False+ for raw in (text or "").splitlines():+ stripped = raw.strip()+ if ":" in stripped and not stripped.lower().startswith("docid "):+ label = stripped.split(":", 1)[0].strip().lower()+ capture = label == name.lower()+ if capture:+ lines.append(stripped.split(":", 1)[1].strip())+ continue+ if capture:+ lines.append(stripped)+ return " ".join(part for part in lines if part)+++def title_text(text: str) -> str:+ title = field_value(text, "Title")+ return title or (text or "")+++def bm25_idf(n_docs: int, df: int) -> float:+ return math.log(1.0 + (n_docs - df + 0.5) / (df + 0.5))+++class Bm25Index:+ def __init__(self, docs: list[tuple[str, list[str]]], *, k1: float = 1.2, b: float = 0.75):+ self.k1 = k1+ self.b = b+ self.doc_ids = [doc_id for doc_id, _ in docs]+ self.tfs: list[Counter[str]] = [Counter(tokens) for _, tokens in docs]+ self.dl = [max(1, len(tokens)) for _, tokens in docs]+ self.avgdl = (sum(self.dl) / len(self.dl)) if self.dl else 1.0+ df: Counter[str] = Counter()+ postings: dict[str, list[int]] = defaultdict(list)+ for index, tokens in enumerate(self.tfs):+ df.update(tokens.keys())+ for term in tokens:+ postings[term].append(index)+ n = max(1, len(self.doc_ids))+ self.idf = {term: bm25_idf(n, count) for term, count in df.items()}+ self.postings = postings++ def scores(self, query_tokens: list[str]) -> dict[str, float]:+ qtf = Counter(query_tokens)+ out: dict[str, float] = defaultdict(float)+ for term, qf in qtf.items():+ idf = self.idf.get(term, 0.0)+ if not idf:+ continue+ for index in self.postings.get(term, ()):+ tf = self.tfs[index].get(term, 0)+ if tf <= 0:+ continue+ dl = self.dl[index]+ denom = tf + self.k1 * (1.0 - self.b + self.b * dl / self.avgdl)+ out[self.doc_ids[index]] += float(qf) * idf * (tf * (self.k1 + 1.0)) / denom+ return out+++def rank_docs(scores: dict[str, float], doc_ids: Iterable[str]) -> list[str]:+ ranked = sorted(scores.items(), key=lambda pair: pair[1], reverse=True)+ seen = {doc_id for doc_id, _ in ranked}+ ordered = [doc_id for doc_id, _ in ranked]+ for doc_id in doc_ids:+ if doc_id not in seen:+ ordered.append(doc_id)+ return ordered+++def rm3_terms(+ query_tokens: list[str],+ ranked: list[str],+ token_map: dict[str, list[str]],+ *,+ top_docs: int = 8,+ extra_terms: int = 8,+) -> list[str]:+ counts: Counter[str] = Counter()+ for doc_id in ranked[:top_docs]:+ counts.update(token_map.get(doc_id) or [])+ query_set = set(query_tokens)+ added: list[str] = []+ for term, _ in counts.most_common():+ if term in STOP or term in query_set or term.isdigit():+ continue+ added.append(term)+ if len(added) >= extra_terms:+ break+ return added+++def expand_query(query: str, extra: list[str]) -> str:+ if not extra:+ return query+ return f"{query} {' '.join(extra)}".strip()+++def evaluate_variant(+ queries: list[dict[str, Any]],+ ranked_by_qid: dict[str, list[str]],+) -> tuple[dict[str, float], list[dict[str, Any]]]:+ per_query: list[dict[str, Any]] = []+ for query in queries:+ qid = str(query.get("qid") or "")+ ranked = ranked_by_qid.get(qid) or []+ metrics = score_ranked_pool(ranked, query)+ per_query.append(+ {+ "qid": qid,+ "query": query.get("query"),+ "hit_ids": ranked[:20],+ "metrics": metrics,+ "n_hits": len(ranked),+ "retrieve_ms": 0.0,+ "embed_ms": None,+ "client_wall_ms": 0.0,+ }+ )+ metrics = aggregate_query_metrics(per_query, ks=(5, 10, 20)) if per_query else {}+ return metrics, per_query+++def _splade_rank(+ docs: list[dict[str, Any]],+ queries: list[dict[str, Any]],+ *,+ k: int = 20,+) -> dict[str, list[str]]:+ plugin_dir = Path(__file__).resolve().parents[2] / "plugins" / "search-splade"+ import sys++ if str(plugin_dir) not in sys.path:+ sys.path.insert(0, str(plugin_dir))+ from index import index_chunks, reset, retrieve++ brain = "harness-local-splade"+ reset(brain)+ chunks = []+ doc_ids: list[str] = []+ for doc in docs:+ doc_id = str(doc.get("doc_id") or "")+ if not doc_id:+ continue+ doc_ids.append(doc_id)+ chunks.append(+ {"id": doc_id, "text": str(doc.get("text") or doc.get("title") or "")}+ )+ index_chunks(brain, chunks)+ ranked_by_qid: dict[str, list[str]] = {}+ for query in queries:+ qid = str(query.get("qid") or "")+ ids, scores, _ = retrieve(str(query.get("query") or ""), brain, max(k, len(doc_ids)))+ ranked_by_qid[qid] = rank_docs(+ {str(doc_id): float(scores.get(doc_id, 0.0)) for doc_id in ids},+ doc_ids,+ )+ return ranked_by_qid+++def run_first_stage(+ rows: list[dict[str, Any]],+ *,+ variant: str,+) -> dict[str, Any]:+ docs, queries = split_corpus(rows)+ if variant == "splade":+ ranked_by_qid = _splade_rank(docs, queries)+ metrics, per_query = evaluate_variant(queries, ranked_by_qid)+ return {+ "variant": variant,+ "metrics": metrics,+ "queries": per_query,+ "expansions": {},+ "n_docs": len(docs),+ "n_queries": len(queries),+ }+ all_tokens: dict[str, list[str]] = {}+ title_tokens: dict[str, list[str]] = {}+ doc_ids: list[str] = []+ for doc in docs:+ doc_id = str(doc.get("doc_id") or "")+ if not doc_id:+ continue+ text = str(doc.get("text") or doc.get("title") or "")+ doc_ids.append(doc_id)+ all_tokens[doc_id] = tokenize(text)+ title_tokens[doc_id] = tokenize(title_text(text))+ all_index = Bm25Index([(doc_id, all_tokens[doc_id]) for doc_id in doc_ids])+ title_index = Bm25Index([(doc_id, title_tokens[doc_id]) for doc_id in doc_ids])+ ranked_by_qid: dict[str, list[str]] = {}+ expansions: dict[str, str] = {}+ for query in queries:+ qid = str(query.get("qid") or "")+ qtext = str(query.get("query") or "")+ q_tokens = tokenize(qtext)+ all_scores = all_index.scores(q_tokens)+ title_scores = title_index.scores(q_tokens)+ if variant == "title":+ scores = title_scores+ elif variant == "title-boost":+ scores = dict(all_scores)+ for doc_id, value in title_scores.items():+ scores[doc_id] = scores.get(doc_id, 0.0) + 2.0 * value+ elif variant in {"rm3", "rm3-title-boost"}:+ seed_scores = dict(all_scores)+ if variant == "rm3-title-boost":+ for doc_id, value in title_scores.items():+ seed_scores[doc_id] = seed_scores.get(doc_id, 0.0) + 2.0 * value+ seed_ranked = rank_docs(seed_scores, doc_ids)+ extra = rm3_terms(q_tokens, seed_ranked, title_tokens)+ expansions[qid] = expand_query(qtext, extra)+ expanded_tokens = tokenize(expansions[qid])+ scores = all_index.scores(expanded_tokens)+ if variant == "rm3-title-boost":+ title_exp = title_index.scores(expanded_tokens)+ for doc_id, value in title_exp.items():+ scores[doc_id] = scores.get(doc_id, 0.0) + 2.0 * value+ else:+ scores = all_scores+ ranked_by_qid[qid] = rank_docs(scores, doc_ids)+ metrics, per_query = evaluate_variant(queries, ranked_by_qid)+ return {+ "variant": variant,+ "metrics": metrics,+ "queries": per_query,+ "expansions": expansions,+ "n_docs": len(doc_ids),+ "n_queries": len(queries),+ }+++def write_expanded_jsonl(+ rows: list[dict[str, Any]],+ expansions: dict[str, str],+ dest: Path,+) -> Path:+ out: list[dict[str, Any]] = []+ for row in rows:+ if str(row.get("type") or "") != "query":+ out.append(row)+ continue+ cloned = dict(row)+ qid = str(cloned.get("qid") or "")+ expanded = expansions.get(qid)+ if expanded:+ cloned["query"] = expanded+ cloned["query_original"] = row.get("query")+ out.append(cloned)+ return write_records(out, dest)+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(prog="pool_first_stage")+ parser.add_argument("--dataset", type=Path, default=Path("data/search_esci_74.jsonl"))+ parser.add_argument("--run", default="search-esci-74-fielded-bm25")+ parser.add_argument(+ "--variant",+ default="all",+ help="Comma-separated: all,title,title-boost,rm3,rm3-title-boost",+ )+ parser.add_argument(+ "--expand-out",+ type=Path,+ default=Path("data/search_esci_74_rm3.jsonl"),+ )+ args = parser.parse_args(argv)+ from search.config import Settings++ settings = Settings.load()+ rows = load_records(args.dataset)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ variants = [item.strip() for item in str(args.variant).split(",") if item.strip()]+ summary: dict[str, Any] = {"run_id": run_id, "dataset": args.dataset.name, "variants": {}}+ expand_source = None+ for variant in variants:+ result = run_first_stage(rows, variant=variant)+ metrics = result["metrics"]+ summary["variants"][variant] = {+ "ndcg@10": metrics.get("ndcg@10"),+ "ndcg@20": metrics.get("ndcg@20"),+ "recall@10": metrics.get("recall@10"),+ "recall@20": metrics.get("recall@20"),+ "mrr": metrics.get("mrr"),+ }+ if variant.startswith("rm3"):+ expand_source = result+ if expand_source and expand_source.get("expansions"):+ write_expanded_jsonl(rows, expand_source["expansions"], args.expand_out)+ summary["expand_out"] = str(args.expand_out)+ docs, queries = split_corpus(rows)+ payload = {+ "status": "ok",+ "brain_id": "harness-local",+ "dataset": args.dataset.name,+ "fusion": "none",+ "rerank": "none",+ "channels": ["harness-bm25"],+ "protocol": (+ "local fielded BM25 over search_esci_74.jsonl 2043-doc corpus; "+ "not BrainAPI passages; not Reddy 0.857"+ ),+ "n_docs": len(docs),+ "n_queries": len(queries),+ "metrics": (summary["variants"].get(variants[0]) or {}),+ "summary": summary,+ "ingest": {"status": "completed", "n_docs": 0, "tasks": [], "reused": True},+ }+ (run_dir / "eval.json").write_text(+ json.dumps(payload, indent=2) + "\n", encoding="utf-8"+ )+ (run_dir / "report.json").write_text(+ json.dumps(summary, indent=2) + "\n", encoding="utf-8"+ )+ print(json.dumps(summary, indent=2))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/search/rank_corpus.py b/benchmarks/search/rank_corpus.pynew file mode 100644index 0000000..68e3942--- /dev/null+++ b/benchmarks/search/rank_corpus.py@@ -0,0 +1,100 @@+from __future__ import annotations++import time+from pathlib import Path+from typing import Any++from search.dataset import split_corpus+from search.finetune_esci_4class import CLASS_GAINS, rank_doc_ids, weighted_scores+from search.local_dense import evaluate_dense+from search.metrics import aggregate_query_metrics+from search.rank_pool import doc_texts+from search.rank_pool_4class import PredictFn, load_4class_predict++DEFAULT_MODEL = Path("data/models/esci-minilm-l12-4class-nowt-e2")+PROTOCOL = "exhaustive-catalog"+++def run_exhaustive_ce(+ rows: list[dict[str, Any]],+ *,+ predict: PredictFn,+ model_name: str,+ dataset_name: str,+ k: int = 50,+ ks: tuple[int, ...] = (5, 10, 20, 50),+ brain_id: str | None = None,+) -> dict[str, Any]:+ docs, queries = split_corpus(rows)+ texts = doc_texts(docs)+ doc_ids = [str(doc.get("doc_id") or "") for doc in docs if doc.get("doc_id")]+ cut = max(1, min(int(k), len(doc_ids) or 1))+ ranked_by_qid: dict[str, list[str]] = {}+ per_q_ms: list[float] = []+ missing_text = 0+ for index, query in enumerate(queries):+ q_text = str(query.get("query") or "")+ pairs: list[tuple[str, str]] = []+ for doc_id in doc_ids:+ text = texts.get(doc_id) or ""+ if not text:+ missing_text += 1+ pairs.append((q_text, text))+ started = time.perf_counter()+ probs = predict(pairs) if pairs else []+ elapsed_ms = (time.perf_counter() - started) * 1000.0+ per_q_ms.append(elapsed_ms)+ scores = weighted_scores(probs, CLASS_GAINS) if pairs else []+ if len(scores) < len(doc_ids):+ scores = list(scores) + [0.0] * (len(doc_ids) - len(scores))+ ranked = rank_doc_ids(doc_ids, scores)[:cut]+ qid = str(query.get("qid") or "")+ ranked_by_qid[qid] = ranked+ print(f"exhaustive {index + 1}/{len(queries)} {qid} {elapsed_ms:.0f}ms", flush=True)+ encode_ms = sum(per_q_ms)+ metrics, per_query = evaluate_dense(+ ranked_by_qid,+ queries,+ ks=ks,+ encode_ms=encode_ms,+ )+ if not metrics and per_query:+ metrics = aggregate_query_metrics(per_query, ks=ks)+ return {+ "status": "ok" if per_query else "failed",+ "brain_id": brain_id or "harness-local-exhaustive",+ "dataset": dataset_name,+ "fusion": "none",+ "rerank": "4class-weighted-ce",+ "channels": ["exhaustive-4class"],+ "expand": "none",+ "k": cut,+ "ks": list(ks),+ "n_docs": len(docs),+ "n_queries": len(per_query),+ "n_docs_mapped": len(texts),+ "skip_enrichment": True,+ "ingest_graph": False,+ "skip_ingest": True,+ "rank_pool": False,+ "rank_pool_ce": False,+ "ce_model": model_name,+ "ce_missing_text": missing_text,+ "protocol": PROTOCOL,+ "ingest": {+ "status": "completed",+ "n_docs": len(docs),+ "tasks": [],+ "reused": True,+ },+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "event_probe": None,+ "search_error": None,+ "metrics": metrics,+ "queries": per_query,+ }+++def load_predict(model_dir: Path, *, max_length: int = 192) -> tuple[PredictFn, str]:+ return load_4class_predict(model_dir, max_length=max_length)diff --git a/benchmarks/search/rank_pool.py b/benchmarks/search/rank_pool.pynew file mode 100644index 0000000..2d50e74--- /dev/null+++ b/benchmarks/search/rank_pool.py@@ -0,0 +1,137 @@+from __future__ import annotations++import sys+from pathlib import Path+from typing import Any++from search.dataset import split_corpus+from search.evaluate import candidate_pool_grades, candidate_pool_ids+from search.metrics import aggregate_query_metrics, mrr, ndcg_at_k, recall_at_k++PLUGIN_DIR = Path(__file__).resolve().parents[2] / "plugins" / "search-rerank"+++def _plugin_rerank():+ if str(PLUGIN_DIR) not in sys.path:+ sys.path.insert(0, str(PLUGIN_DIR))+ from rerank import model_name, rerank++ return rerank, model_name()+++def doc_texts(docs: list[dict[str, Any]]) -> dict[str, str]:+ texts: dict[str, str] = {}+ for doc in docs:+ doc_id = str(doc.get("doc_id") or "")+ if not doc_id:+ continue+ texts[doc_id] = str(doc.get("text") or doc.get("title") or "")+ return texts+++def score_ranked_pool(+ ranked: list[str],+ query: dict[str, Any],+ *,+ ks: tuple[int, ...] = (5, 10, 20),+) -> dict[str, float]:+ gold = {+ str(doc_id)+ for doc_id, gain in candidate_pool_grades(query).items()+ if float(gain) > 0+ }+ if not gold:+ gold = {str(item) for item in (query.get("gold_doc_ids") or [])}+ grades = candidate_pool_grades(query)+ metrics = {f"recall@{k}": recall_at_k(ranked, gold, k) for k in ks}+ ndcg_ks = tuple(dict.fromkeys((*ks, 10, 20)))+ for cut in ndcg_ks:+ metrics[f"ndcg@{cut}"] = ndcg_at_k(ranked, gold, cut, grades=grades)+ metrics["ndcg"] = ndcg_at_k(ranked, gold, max(len(ranked), 1), grades=grades)+ metrics["mrr"] = mrr(ranked, gold)+ return metrics+++def run_ce_on_pool(+ rows: list[dict[str, Any]],+ *,+ dataset_name: str,+ ks: tuple[int, ...] = (5, 10, 20),+ brain_id: str | None = None,+) -> dict[str, Any]:+ docs, queries = split_corpus(rows)+ texts = doc_texts(docs)+ rerank, model = _plugin_rerank()+ per_query: list[dict[str, Any]] = []+ missing_text = 0+ for query in queries:+ pool = candidate_pool_ids(query)+ candidates = []+ for doc_id in pool:+ text = texts.get(doc_id) or ""+ if not text:+ missing_text += 1+ candidates.append({"id": doc_id, "text": text, "score": 0.0})+ ranked_rows = rerank(+ str(query.get("query") or ""),+ candidates,+ len(candidates) or 1,+ )+ ranked = [str(item.get("id") or "") for item in ranked_rows if item.get("id")]+ seen = set(ranked)+ for doc_id in pool:+ if doc_id not in seen:+ ranked.append(doc_id)+ metrics = score_ranked_pool(ranked, query, ks=ks)+ per_query.append(+ {+ "qid": query.get("qid"),+ "query": query.get("query"),+ "slice": query.get("slice") or "unspecified",+ "gold_doc_ids": list(query.get("gold_doc_ids") or []),+ "candidate_doc_ids": pool,+ "pool_size": len(pool),+ "pool_coverage": 1.0 if pool else None,+ "missing_from_brain": [],+ "hit_ids": ranked,+ "metrics": metrics,+ "retrieve_ms": 0.0,+ "embed_ms": None,+ "client_wall_ms": 0.0,+ "n_hits": len(ranked),+ }+ )+ metrics = aggregate_query_metrics(per_query, ks=ks) if per_query else {}+ return {+ "status": "ok" if per_query else "failed",+ "brain_id": brain_id,+ "dataset": dataset_name,+ "fusion": "none",+ "rerank": "plugin:cross-encoder",+ "channels": ["rank-pool-ce"],+ "expand": "none",+ "k": max((len(row.get("hit_ids") or []) for row in per_query), default=0),+ "ks": list(ks),+ "n_docs": len(docs),+ "n_queries": len(per_query),+ "n_docs_mapped": len(texts),+ "skip_enrichment": True,+ "ingest_graph": False,+ "skip_ingest": True,+ "rank_pool": True,+ "rank_pool_ce": True,+ "ce_model": model,+ "ce_missing_text": missing_text,+ "ingest": {"status": "completed", "n_docs": len(docs), "tasks": [], "reused": True},+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "event_probe": None,+ "search_error": None,+ "protocol": (+ "ranking-in-pool CE over labeled candidates; "+ f"model={model}; n is not Reddy ~4477 unless stated; "+ "cite 0.857 only against this CE-on-pool protocol"+ ),+ "metrics": metrics,+ "queries": per_query,+ }diff --git a/benchmarks/search/rank_pool_4class.py b/benchmarks/search/rank_pool_4class.pynew file mode 100644index 0000000..0f1bcaa--- /dev/null+++ b/benchmarks/search/rank_pool_4class.py@@ -0,0 +1,218 @@+from __future__ import annotations++import argparse+import json+from pathlib import Path+from typing import Any, Callable++from search.config import Settings, validate_brain_id+from search.dataset import load_records, split_corpus+from search.evaluate import candidate_pool_grades, candidate_pool_ids, ensure_run_dir+from search.finetune_esci_4class import (+ CLASS_GAINS,+ DEFAULT_MODEL,+ DEFAULT_OUT,+ pick_device,+ rank_doc_ids,+ weighted_scores,+)+from search.metrics import aggregate_query_metrics, mrr, ndcg_at_k, recall_at_k+from search.report import print_report_table, write_report++PredictFn = Callable[[list[tuple[str, str]]], Any]+++def doc_texts(docs: list[dict[str, Any]]) -> dict[str, str]:+ texts: dict[str, str] = {}+ for doc in docs:+ doc_id = str(doc.get("doc_id") or "")+ if not doc_id:+ continue+ texts[doc_id] = str(doc.get("text") or doc.get("title") or "")+ return texts+++def score_ranked_pool(+ ranked: list[str],+ query: dict[str, Any],+ *,+ ks: tuple[int, ...] = (5, 10, 20),+) -> dict[str, float]:+ gold = {+ str(doc_id)+ for doc_id, gain in candidate_pool_grades(query).items()+ if float(gain) > 0+ }+ if not gold:+ gold = {str(item) for item in (query.get("gold_doc_ids") or [])}+ grades = candidate_pool_grades(query)+ metrics = {f"recall@{k}": recall_at_k(ranked, gold, k) for k in ks}+ metrics["ndcg@10"] = ndcg_at_k(ranked, gold, 10, grades=grades)+ metrics["ndcg@20"] = ndcg_at_k(ranked, gold, 20, grades=grades)+ metrics["ndcg"] = ndcg_at_k(ranked, gold, max(len(ranked), 1), grades=grades)+ metrics["mrr"] = mrr(ranked, gold)+ return metrics+++def load_4class_predict(model_dir: Path, *, max_length: int = 192) -> tuple[PredictFn, str]:+ from sentence_transformers import CrossEncoder++ device = pick_device()+ model = CrossEncoder(+ str(model_dir),+ num_labels=4,+ max_length=max_length,+ device=device,+ default_activation_function=None,+ )++ def predict(pairs: list[tuple[str, str]]):+ if not pairs:+ return []+ return model.predict(+ [list(pair) for pair in pairs],+ batch_size=32,+ apply_softmax=True,+ show_progress_bar=False,+ )++ return predict, str(model_dir)+++def run_4class_on_pool(+ rows: list[dict[str, Any]],+ *,+ dataset_name: str,+ predict: PredictFn,+ model_name: str,+ ks: tuple[int, ...] = (5, 10, 20),+ brain_id: str | None = None,+) -> dict[str, Any]:+ docs, queries = split_corpus(rows)+ texts = doc_texts(docs)+ per_query: list[dict[str, Any]] = []+ missing_text = 0+ for query in queries:+ pool = candidate_pool_ids(query)+ pairs: list[tuple[str, str]] = []+ for doc_id in pool:+ text = texts.get(doc_id) or ""+ if not text:+ missing_text += 1+ pairs.append((str(query.get("query") or ""), text))+ probs = predict(pairs) if pairs else []+ scores = weighted_scores(probs, CLASS_GAINS) if len(pairs) else []+ if len(scores) < len(pool):+ scores = list(scores) + [0.0] * (len(pool) - len(scores))+ ranked = rank_doc_ids(pool, scores)+ metrics = score_ranked_pool(ranked, query, ks=ks)+ per_query.append(+ {+ "qid": query.get("qid"),+ "query": query.get("query"),+ "slice": query.get("slice") or "unspecified",+ "gold_doc_ids": list(query.get("gold_doc_ids") or []),+ "candidate_doc_ids": pool,+ "pool_size": len(pool),+ "pool_coverage": 1.0 if pool else None,+ "missing_from_brain": [],+ "hit_ids": ranked,+ "metrics": metrics,+ "retrieve_ms": 0.0,+ "embed_ms": None,+ "client_wall_ms": 0.0,+ "n_hits": len(ranked),+ }+ )+ metrics = aggregate_query_metrics(per_query, ks=ks) if per_query else {}+ return {+ "status": "ok" if per_query else "failed",+ "brain_id": brain_id,+ "dataset": dataset_name,+ "fusion": "none",+ "rerank": "4class-weighted-ce",+ "channels": ["rank-pool-4class"],+ "expand": "none",+ "k": max((len(row.get("hit_ids") or []) for row in per_query), default=0),+ "ks": list(ks),+ "n_docs": len(docs),+ "n_queries": len(per_query),+ "n_docs_mapped": len(texts),+ "skip_enrichment": True,+ "ingest_graph": False,+ "skip_ingest": True,+ "rank_pool": True,+ "rank_pool_ce": True,+ "ce_model": model_name,+ "ce_missing_text": missing_text,+ "ingest": {+ "status": "completed",+ "n_docs": len(docs),+ "tasks": [],+ "reused": True,+ },+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "event_probe": None,+ "search_error": None,+ "protocol": (+ "ranking-in-pool 4-class CE over labeled candidates including I=0; "+ "score=1.0*P(E)+0.1*P(S)+0.01*P(C); not MiniLM Exact→1; "+ "n=74 is not Reddy public test n≈4477"+ ),+ "metrics": metrics,+ "queries": per_query,+ }+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(+ description="Ranking-in-pool eval with a 4-class weighted cross-encoder"+ )+ parser.add_argument("--dataset", type=Path, required=True)+ parser.add_argument("--model", type=Path, default=DEFAULT_OUT)+ parser.add_argument("--run", default="search-esci-74-ce-pool-4class")+ parser.add_argument("--brain", default="searchbenchesci74")+ parser.add_argument("--env-file", type=Path, default=None)+ parser.add_argument("--max-length", type=int, default=192)+ args = parser.parse_args(argv)+ if not args.dataset.exists():+ print(f"Missing {args.dataset}")+ return 1+ if not args.model.exists():+ print(f"Missing model {args.model}")+ return 1+ settings = Settings.load(args.env_file)+ settings.brain_id = validate_brain_id(args.brain)+ rows = load_records(args.dataset)+ run_id, run_dir = ensure_run_dir(settings, args.run)+ predict, model_name = load_4class_predict(args.model, max_length=args.max_length)+ result = run_4class_on_pool(+ rows,+ dataset_name=args.dataset.name,+ predict=predict,+ model_name=model_name,+ brain_id=settings.brain_id,+ )+ report = write_report(run_dir, result)+ print_report_table(report)+ print(f"Wrote {run_dir / 'report.json'}")+ print(+ json.dumps(+ {+ "run_id": run_id,+ "ce_model": result.get("ce_model"),+ "ndcg@20": report.get("ndcg@20"),+ "ndcg": report.get("ndcg"),+ "n_queries": report.get("n_queries"),+ "missing_text": result.get("ce_missing_text"),+ "base_default": DEFAULT_MODEL,+ },+ indent=2,+ )+ )+ return 0 if report.get("status") == "ok" else 1+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/search/replay_fusion.py b/benchmarks/search/replay_fusion.pynew file mode 100644index 0000000..72843a2--- /dev/null+++ b/benchmarks/search/replay_fusion.py@@ -0,0 +1,611 @@+from __future__ import annotations++import argparse+import json+from pathlib import Path+from typing import Any++from search.metrics import mrr, ndcg_at_k, recall_at_k++RRF_K = 60+EXPANSION_N = 10+GRAPH_WEIGHTS = (0.1, 0.25, 0.5)+FUSED_NDCG = 0.653+RECALL20_TARGET = 0.80+_SLICE_JSONL = Path(__file__).resolve().parent.parent / "data" / "search_esci_slice.jsonl"+++def _load_eval(path: Path) -> dict[str, Any]:+ return json.loads(path.read_text(encoding="utf-8"))+++def _gold(query: dict[str, Any]) -> set[str]:+ gold = {str(item) for item in (query.get("gold_doc_ids") or []) if item}+ grades = query.get("gold_grades") or {}+ if isinstance(grades, dict):+ for doc_id, gain in grades.items():+ if float(gain or 0) <= 0:+ continue+ gold.add(str(doc_id))+ if gold:+ return gold+ return {str(item) for item in (query.get("gold_chunk_ids") or []) if item}+++def dataset_gold_grades(path: Path | None = None) -> dict[str, dict[str, float]]:+ jsonl = path or _SLICE_JSONL+ if not jsonl.exists():+ return {}+ grades: dict[str, dict[str, float]] = {}+ for line in jsonl.read_text(encoding="utf-8").splitlines():+ if not line.strip():+ continue+ row = json.loads(line)+ if row.get("type") != "query":+ continue+ qid = str(row.get("qid") or "")+ raw = row.get("gold_grades") or {}+ if not qid or not isinstance(raw, dict):+ continue+ grades[qid] = {+ str(doc_id): float(gain)+ for doc_id, gain in raw.items()+ if float(gain or 0) > 0+ }+ return grades+++def _grades(+ query: dict[str, Any],+ gold: set[str],+ dataset_grades: dict[str, dict[str, float]] | None = None,+) -> dict[str, float]:+ raw = query.get("gold_grades") or {}+ if isinstance(raw, dict) and raw:+ return {+ str(doc_id): float(gain)+ for doc_id, gain in raw.items()+ if float(gain or 0) > 0+ }+ qid = str(query.get("qid") or "")+ from_dataset = (dataset_grades or {}).get(qid) or {}+ if from_dataset:+ return from_dataset+ return {doc_id: 1.0 for doc_id in gold}+++def _ranked(query: dict[str, Any]) -> list[str]:+ return [str(item) for item in (query.get("hit_ids") or []) if item]+++def drop_hubs(ranked: list[str]) -> list[str]:+ return [item for item in ranked if not item.startswith("hub:")]+++def keep_product_asins(ranked: list[str]) -> list[str]:+ return [+ item+ for item in ranked+ if item and not item.startswith("hub:") and not item.startswith("evt:")+ ]+++def score_ranked(+ ranked: list[str],+ gold: set[str],+ grades: dict[str, float],+) -> dict[str, float]:+ return {+ "ndcg@10": ndcg_at_k(ranked, gold, 10, grades=grades),+ "recall@10": recall_at_k(ranked, gold, 10),+ "recall@20": recall_at_k(ranked, gold, 20),+ "mrr": mrr(ranked, gold),+ "unique_docs@20": float(len({str(item) for item in ranked[:20] if item})),+ }+++def mean_metrics(rows: list[dict[str, float]]) -> dict[str, float]:+ keys = ("ndcg@10", "recall@10", "recall@20", "mrr", "unique_docs@20")+ if not rows:+ return {key: 0.0 for key in keys}+ present = [key for key in keys if all(key in row for row in rows)]+ return {key: sum(row[key] for row in rows) / len(rows) for key in present}+++def reciprocal_rank_fusion(+ ranked_lists: list[list[str]],+ *,+ k: int = RRF_K,+ weights: list[float] | None = None,+) -> list[str]:+ scores: dict[str, float] = {}+ for index, ranked in enumerate(ranked_lists):+ weight = 1.0 if not weights else float(weights[index])+ for rank, item in enumerate(ranked):+ if not item:+ continue+ scores[item] = scores.get(item, 0.0) + weight / (k + rank + 1)+ return [+ item+ for item, _ in sorted(scores.items(), key=lambda pair: pair[1], reverse=True)+ ]+++def chunk_to_doc_map(*queries: dict[str, Any]) -> dict[str, str]:+ mapping: dict[str, str] = {}+ for query in queries:+ for hit in query.get("hits") or []:+ if not isinstance(hit, dict):+ continue+ hid = str(hit.get("id") or "")+ doc_id = str(hit.get("doc_id") or hid)+ if hid:+ mapping[hid] = doc_id+ return mapping+++def collapse_to_doc(+ ids: list[str],+ chunk_to_doc: dict[str, str],+) -> list[str]:+ out: list[str] = []+ seen: set[str] = set()+ for item in ids:+ key = chunk_to_doc.get(str(item), str(item))+ if (+ not key+ or key.startswith("hub:")+ or key.startswith("evt:")+ or key in seen+ ):+ continue+ seen.add(key)+ out.append(key)+ return out+++def _id_list(query: dict[str, Any], key: str, channel: str | None = None) -> list[str]:+ dumped = [str(item) for item in (query.get(key) or []) if item]+ if dumped:+ return dumped+ if not channel:+ return []+ out: list[str] = []+ for hit in query.get("hits") or []:+ if not isinstance(hit, dict):+ continue+ hid = str(hit.get("id") or "")+ if hid and str(hit.get("channel") or "") == channel:+ out.append(hid)+ return out+++def query_channel_lists(query: dict[str, Any]) -> dict[str, list[str]]:+ return {+ "dense": _id_list(query, "dense_ids"),+ "bm25": _id_list(query, "bm25_ids"),+ "passages": _id_list(query, "passage_ids", "passages"),+ "entities": _id_list(query, "entity_ids", "entities"),+ "communities": _id_list(query, "community_ids", "communities"),+ }+++def merge_query_lists(+ graph_query: dict[str, Any],+ passages_query: dict[str, Any] | None = None,+ entities_query: dict[str, Any] | None = None,+ communities_query: dict[str, Any] | None = None,+) -> dict[str, list[str]]:+ graph = query_channel_lists(graph_query)+ passages = query_channel_lists(passages_query or {})+ entities = query_channel_lists(entities_query or {})+ communities = query_channel_lists(communities_query or {})+ dense = graph["dense"] or passages["dense"]+ bm25 = graph["bm25"] or passages["bm25"]+ passage_ids = passages["passages"] or graph["passages"]+ if not passage_ids and passages_query:+ passage_ids = [+ str(hit.get("id") or "")+ for hit in (passages_query.get("hits") or [])+ if hit.get("id")+ ]+ if not passage_ids:+ passage_ids = _ranked(passages_query)+ entity_ids = graph["entities"] or entities["entities"]+ if not entity_ids and entities_query:+ entity_ids = [+ str(hit.get("id") or "")+ for hit in (entities_query.get("hits") or [])+ if hit.get("id")+ ]+ if not entity_ids:+ entity_ids = _ranked(entities_query)+ community_ids = graph["communities"] or communities["communities"]+ if not community_ids and communities_query:+ community_ids = [+ str(hit.get("id") or "")+ for hit in (communities_query.get("hits") or [])+ if hit.get("id")+ ]+ if not community_ids:+ community_ids = _ranked(communities_query)+ return {+ "dense": dense,+ "bm25": bm25,+ "passages": passage_ids,+ "entities": entity_ids,+ "communities": community_ids,+ }+++def collapsed_lists(+ lists: dict[str, list[str]],+ chunk_to_doc: dict[str, str],+) -> dict[str, list[str]]:+ collapsed = {+ name: collapse_to_doc(ids, chunk_to_doc) for name, ids in lists.items()+ }+ if not collapsed["dense"] and not collapsed["bm25"] and collapsed["passages"]:+ collapsed["passages_core"] = collapsed["passages"]+ else:+ collapsed["passages_core"] = reciprocal_rank_fusion(+ [ids for ids in (collapsed["dense"], collapsed["bm25"]) if ids]+ ) or collapsed["passages"]+ return collapsed+++def _by_qid(eval_result: dict[str, Any] | None) -> dict[str, dict[str, Any]]:+ if not eval_result:+ return {}+ return {+ str(row.get("qid")): row for row in (eval_result.get("queries") or [])+ }+++def _pair_queries(+ graph_eval: dict[str, Any],+ passages_eval: dict[str, Any],+ entities_eval: dict[str, Any] | None = None,+ communities_eval: dict[str, Any] | None = None,+) -> list[dict[str, Any]]:+ passages_by = _by_qid(passages_eval)+ entities_by = _by_qid(entities_eval)+ communities_by = _by_qid(communities_eval)+ slice_grades = dataset_gold_grades()+ rows: list[dict[str, Any]] = []+ for graph_query in graph_eval.get("queries") or []:+ qid = str(graph_query.get("qid") or "")+ passages_query = passages_by.get(qid) or {}+ entities_query = entities_by.get(qid)+ communities_query = communities_by.get(qid)+ gold = _gold(graph_query) or _gold(passages_query)+ grades = _grades(graph_query, gold, slice_grades)+ if not (graph_query.get("gold_grades") or {}):+ grades = _grades(passages_query, gold, slice_grades)+ mapping = chunk_to_doc_map(+ graph_query, passages_query, entities_query or {}, communities_query or {}+ )+ lists = merge_query_lists(+ graph_query, passages_query, entities_query, communities_query+ )+ collapsed = collapsed_lists(lists, mapping)+ rows.append(+ {+ "qid": qid,+ "gold": gold,+ "grades": grades,+ "lists": lists,+ "collapsed": collapsed,+ "passages_ranked": collapse_to_doc(+ lists["passages"] or _ranked(passages_query), mapping+ ),+ "graph_ranked": _ranked(graph_query),+ "passage_metrics": passages_query.get("metrics") or {},+ }+ )+ return rows+++def replay(graph_eval: dict[str, Any], passages_eval: dict[str, Any]) -> dict[str, Any]:+ passages_by_qid = _by_qid(passages_eval)+ hub_rows: list[dict[str, float]] = []+ asin_rows: list[dict[str, float]] = []+ per_query: list[dict[str, Any]] = []+ for query in graph_eval.get("queries") or []:+ qid = str(query.get("qid") or "")+ gold = _gold(query)+ grades = _grades(query, gold)+ ranked = _ranked(query)+ hub = score_ranked(drop_hubs(ranked), gold, grades)+ asin = score_ranked(keep_product_asins(ranked), gold, grades)+ hub_rows.append(hub)+ asin_rows.append(asin)+ passage_row = passages_by_qid.get(qid) or {}+ passage_metrics = passage_row.get("metrics") or {}+ per_query.append(+ {+ "qid": qid,+ "n_hits": len(ranked),+ "n_hits_no_hub": len(drop_hubs(ranked)),+ "n_product_ids": len(keep_product_asins(ranked)),+ "hub_drop": hub,+ "product_ids": asin,+ "passages": {+ "ndcg@10": float(passage_metrics.get("ndcg@10") or 0.0),+ "recall@10": float(passage_metrics.get("recall@10") or 0.0),+ "recall@20": float(passage_metrics.get("recall@20") or 0.0),+ "mrr": float(passage_metrics.get("mrr") or 0.0),+ },+ }+ )+ return {+ "n_queries": len(per_query),+ "hub_drop": mean_metrics(hub_rows),+ "product_ids": mean_metrics(asin_rows),+ "queries": per_query,+ }+++def _core_and_graph(collapsed: dict[str, list[str]]) -> tuple[list[str], list[str], list[str]]:+ core = [ids for ids in (collapsed["dense"], collapsed["bm25"]) if ids]+ if not core and collapsed.get("passages_core"):+ core = [collapsed["passages_core"]]+ entities = collapsed.get("entities") or []+ communities = collapsed.get("communities") or []+ return core, entities, communities+++def rank_collapse_rrf(collapsed: dict[str, list[str]]) -> list[str]:+ core, entities, communities = _core_and_graph(collapsed)+ lists = [*core]+ if entities:+ lists.append(entities)+ if communities:+ lists.append(communities)+ if not lists:+ return []+ if len(lists) == 1:+ return list(lists[0])+ return reciprocal_rank_fusion(lists)+++def rank_weighted_rrf(collapsed: dict[str, list[str]], graph_weight: float) -> list[str]:+ core, entities, communities = _core_and_graph(collapsed)+ lists: list[list[str]] = [*core]+ weights: list[float] = [1.0] * len(core)+ if entities:+ lists.append(entities)+ weights.append(graph_weight)+ if communities:+ lists.append(communities)+ weights.append(graph_weight)+ if not lists:+ return []+ if len(lists) == 1:+ return list(lists[0])+ return reciprocal_rank_fusion(lists, weights=weights)+++def rank_expansion(+ passage_docs: list[str],+ collapsed: dict[str, list[str]],+ *,+ n: int = EXPANSION_N,+) -> list[str]:+ head = list(passage_docs[:10])+ passage_set = set(passage_docs)+ graph_docs: list[str] = []+ seen: set[str] = set()+ for item in (collapsed.get("entities") or []) + (collapsed.get("communities") or []):+ if item in seen:+ continue+ seen.add(item)+ graph_docs.append(item)+ novel = [item for item in graph_docs if item not in passage_set][:n]+ novel_set = set(novel)+ rest = [item for item in passage_docs[10:] if item not in novel_set]+ return head + novel + rest+++def rank_confirmation(+ passage_docs: list[str],+ collapsed: dict[str, list[str]],+) -> list[str]:+ passage_set = set(passage_docs)+ confirmed: list[str] = []+ seen: set[str] = set()+ for item in (collapsed.get("entities") or []) + (collapsed.get("communities") or []):+ if item not in passage_set or item in seen:+ continue+ seen.add(item)+ confirmed.append(item)+ if not confirmed:+ return list(passage_docs)+ return reciprocal_rank_fusion([passage_docs, confirmed])+++def replay_offline(+ graph_eval: dict[str, Any],+ passages_eval: dict[str, Any],+ *,+ entities_eval: dict[str, Any] | None = None,+ communities_eval: dict[str, Any] | None = None,+) -> dict[str, Any]:+ paired = _pair_queries(+ graph_eval, passages_eval, entities_eval, communities_eval+ )+ arms: dict[str, list[dict[str, float]]] = {+ "passages": [],+ "collapse-rrf": [],+ "weighted-0.1": [],+ "weighted-0.25": [],+ "weighted-0.5": [],+ "expansion-n10": [],+ "confirmation": [],+ }+ per_query: list[dict[str, Any]] = []+ for row in paired:+ gold = row["gold"]+ grades = row["grades"]+ collapsed = row["collapsed"]+ passage_docs = row["passages_ranked"]+ scored = {+ "passages": score_ranked(passage_docs, gold, grades),+ "collapse-rrf": score_ranked(+ rank_collapse_rrf(collapsed), gold, grades+ ),+ "expansion-n10": score_ranked(+ rank_expansion(passage_docs, collapsed, n=EXPANSION_N),+ gold,+ grades,+ ),+ "confirmation": score_ranked(+ rank_confirmation(passage_docs, collapsed), gold, grades+ ),+ }+ for weight in GRAPH_WEIGHTS:+ name = f"weighted-{weight}"+ scored[name] = score_ranked(+ rank_weighted_rrf(collapsed, weight), gold, grades+ )+ for name, metrics in scored.items():+ arms[name].append(metrics)+ per_query.append({"qid": row["qid"], **scored})+ summary = {name: mean_metrics(rows) for name, rows in arms.items()}+ return {+ "n_queries": len(per_query),+ "expansion_n": EXPANSION_N,+ "graph_weights": list(GRAPH_WEIGHTS),+ "primary": ["recall@20", "ndcg@10"],+ "arms": summary,+ "queries": per_query,+ }+++def pick_offline_winner(summary: dict[str, dict[str, float]]) -> str:+ passages = summary.get("passages") or {}+ passages_ndcg = float(passages.get("ndcg@10") or 0.0)+ passages_r20 = float(passages.get("recall@20") or 0.0)+ candidates: list[tuple[str, dict[str, float]]] = []+ for name, metrics in summary.items():+ if name == "passages":+ continue+ recall20 = float(metrics.get("recall@20") or 0.0)+ ndcg = float(metrics.get("ndcg@10") or 0.0)+ if recall20 + 1e-9 < max(RECALL20_TARGET, passages_r20):+ continue+ if ndcg + 1e-9 < max(FUSED_NDCG, passages_ndcg):+ continue+ candidates.append((name, metrics))+ if not candidates:+ return "G08"++ def _key(item: tuple[str, dict[str, float]]) -> tuple[float, float]:+ metrics = item[1]+ return (+ float(metrics.get("recall@20") or 0.0),+ float(metrics.get("ndcg@10") or 0.0),+ )++ name, _metrics = max(candidates, key=_key)+ return name+++def _print_metrics(label: str, metrics: dict[str, float]) -> None:+ unique = metrics.get("unique_docs@20")+ unique_s = f" unique@20={unique:.2f}" if unique is not None else ""+ print(+ f"{label:18} "+ f"nDCG@10={metrics.get('ndcg@10', 0.0):.3f} "+ f"R@10={metrics.get('recall@10', 0.0):.3f} "+ f"R@20={metrics.get('recall@20', 0.0):.3f} "+ f"MRR={metrics.get('mrr', 0.0):.3f}"+ f"{unique_s}"+ )+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(+ description="Offline rescore of fused search eval.json lists."+ )+ parser.add_argument("--graph", required=True, help="Graph-arm eval.json")+ parser.add_argument("--passages", required=True, help="Passages-arm eval.json")+ parser.add_argument(+ "--entities",+ default=None,+ help="Optional isolated entities eval.json",+ )+ parser.add_argument(+ "--communities",+ default=None,+ help="Optional isolated communities eval.json",+ )+ parser.add_argument(+ "--mode",+ default="hub-drop",+ choices=("hub-drop", "collapse-rrf", "gated", "all"),+ )+ args = parser.parse_args(argv)+ graph_eval = _load_eval(Path(args.graph))+ passages_eval = _load_eval(Path(args.passages))+ entities_eval = _load_eval(Path(args.entities)) if args.entities else None+ communities_eval = (+ _load_eval(Path(args.communities)) if args.communities else None+ )+ if args.mode in {"hub-drop", "all"}:+ result = replay(graph_eval, passages_eval)+ hub = result["hub_drop"]+ print(+ "hub-drop "+ f"nDCG@10={hub['ndcg@10']:.3f} "+ f"R@10={hub['recall@10']:.3f} "+ f"R@20={hub['recall@20']:.3f} "+ f"MRR={hub['mrr']:.3f}"+ )+ asin = result["product_ids"]+ print(+ "product-ids "+ f"nDCG@10={asin['ndcg@10']:.3f} "+ f"R@10={asin['recall@10']:.3f} "+ f"R@20={asin['recall@20']:.3f} "+ f"MRR={asin['mrr']:.3f}"+ )+ for row in result["queries"]:+ delta = row["hub_drop"]["ndcg@10"] - row["passages"]["ndcg@10"]+ print(+ f"{row['qid']}: hub-drop nDCG {row['hub_drop']['ndcg@10']:.3f} "+ f"(passages {row['passages']['ndcg@10']:.3f}, delta {delta:+.3f})"+ )+ if args.mode in {"collapse-rrf", "gated", "all"}:+ offline = replay_offline(+ graph_eval,+ passages_eval,+ entities_eval=entities_eval,+ communities_eval=communities_eval,+ )+ names = ["passages", "collapse-rrf"]+ if args.mode in {"gated", "all"}:+ names.extend(+ [+ "weighted-0.1",+ "weighted-0.25",+ "weighted-0.5",+ "expansion-n10",+ "confirmation",+ ]+ )+ print(+ "primary recall@20 then nDCG@10; "+ f"expansion_n={offline['expansion_n']}; "+ f"graph_weights={offline['graph_weights']}"+ )+ for name in names:+ _print_metrics(name, offline["arms"][name])+ if args.mode in {"gated", "all"}:+ winner = pick_offline_winner(offline["arms"])+ print(f"winner {winner}")+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/search/report.py b/benchmarks/search/report.pynew file mode 100644index 0000000..1bf5cc4--- /dev/null+++ b/benchmarks/search/report.py@@ -0,0 +1,269 @@+from __future__ import annotations++import json+import subprocess+import tempfile+from datetime import datetime, timezone+from pathlib import Path+from typing import Any++from rich.console import Console+from rich.table import Table++from search.config import BENCHMARKS_ROOT++console = Console()++REPORTS_PATH = BENCHMARKS_ROOT / "REPORTS.json"+_BENCHMARK_ID = "search"+_BENCHMARK_NAME = "Search (hybrid BM25 + dense)"+_REPORTS_DESCRIPTION = (+ "BrainAPI benchmark results. Top published scores across suites. "+ "Updated when a suite evaluate/report completes successfully."+)+++def _git_sha() -> str | None:+ try:+ out = subprocess.check_output(+ ["git", "rev-parse", "HEAD"],+ cwd=BENCHMARKS_ROOT.parent,+ stderr=subprocess.DEVNULL,+ text=True,+ )+ return out.strip() or None+ except Exception:+ return None+++def build_report(run_dir: Path, eval_result: dict[str, Any]) -> dict[str, Any]:+ metrics = eval_result.get("metrics") or {}+ status = str(eval_result.get("status") or "failed")+ n_queries = int(eval_result.get("n_queries") or 0)+ if status != "ok":+ status = "failed"+ if n_queries < 1:+ status = "failed"+ if (eval_result.get("ingest") or {}).get("status") not in {+ "completed",+ "partial_failed",+ }:+ status = "failed"++ report = {+ "suite": _BENCHMARK_ID,+ "status": status,+ "run_id": run_dir.name,+ "brain_id": eval_result.get("brain_id"),+ "dataset": eval_result.get("dataset"),+ "fusion": eval_result.get("fusion"),+ "rerank": eval_result.get("rerank") or "none",+ "channels": eval_result.get("channels") or ["passages"],+ "rank_pool": bool(eval_result.get("rank_pool")),+ "rank_pool_ce": bool(eval_result.get("rank_pool_ce")),+ "ce_model": eval_result.get("ce_model"),+ "skip_enrichment": eval_result.get("skip_enrichment"),+ "n_queries": n_queries,+ "n_docs": eval_result.get("n_docs"),+ "k": eval_result.get("k"),+ "ndcg@10": metrics.get("ndcg@10"),+ "ndcg@20": metrics.get("ndcg@20"),+ "ndcg@50": metrics.get("ndcg@50"),+ "ndcg@100": metrics.get("ndcg@100"),+ "ndcg": metrics.get("ndcg"),+ "recall@10": metrics.get("recall@10"),+ "recall@20": metrics.get("recall@20"),+ "recall@50": metrics.get("recall@50"),+ "recall@100": metrics.get("recall@100"),+ "mrr": metrics.get("mrr"),+ "pool_coverage": metrics.get("pool_coverage"),+ "p50_retrieve_ms": metrics.get("p50_retrieve_ms"),+ "p95_retrieve_ms": metrics.get("p95_retrieve_ms"),+ "metrics": metrics,+ "ingest": eval_result.get("ingest"),+ "git_sha": _git_sha(),+ "recorded_at": datetime.now(timezone.utc).isoformat(),+ "protocol": eval_result.get("protocol")+ or (+ "labeled toy search: POST /ingest/ → POST /retrieve/search "+ f"(fusion={eval_result.get('fusion') or 'rrf'}, "+ f"rerank={eval_result.get('rerank') or 'none'}, "+ f"rank_pool={bool(eval_result.get('rank_pool'))})"+ ),+ }+ return report+++def write_report(run_dir: Path, eval_result: dict[str, Any]) -> dict[str, Any]:+ report = build_report(run_dir, eval_result)+ (run_dir / "eval.json").write_text(+ json.dumps(eval_result, indent=2) + "\n", encoding="utf-8"+ )+ (run_dir / "report.json").write_text(+ json.dumps(report, indent=2) + "\n", encoding="utf-8"+ )+ md = render_markdown(report)+ (run_dir / "report.md").write_text(md, encoding="utf-8")+ if report.get("status") == "ok":+ update_reports_json(report)+ return report+++def render_markdown(report: dict[str, Any]) -> str:+ lines = [+ f"# Search report — `{report.get('run_id')}`",+ "",+ f"- status: **{report.get('status')}**",+ f"- dataset: `{report.get('dataset')}`",+ f"- brain_id: `{report.get('brain_id')}`",+ f"- fusion: `{report.get('fusion')}`",+ f"- rerank: `{report.get('rerank')}`",+ f"- channels: `{','.join(report.get('channels') or ['passages'])}`",+ f"- rank_pool: `{report.get('rank_pool')}`",+ f"- skip_enrichment: `{report.get('skip_enrichment')}`",+ f"- k: {report.get('k')}",+ f"- n_queries: {report.get('n_queries')}",+ f"- nDCG@10: {report.get('ndcg@10')}",+ f"- nDCG@20: {report.get('ndcg@20')}",+ f"- nDCG@50: {report.get('ndcg@50')}",+ f"- nDCG (full list): {report.get('ndcg')}",+ f"- Recall@10: {report.get('recall@10')}",+ f"- Recall@20: {report.get('recall@20')}",+ f"- Recall@50: {report.get('recall@50')}",+ f"- pool_coverage: {report.get('pool_coverage')}",+ f"- MRR: {report.get('mrr')}",+ f"- p50 retrieve ms (ex-embed): {report.get('p50_retrieve_ms')}",+ f"- p95 retrieve ms (ex-embed): {report.get('p95_retrieve_ms')}",+ "",+ "Never mutate LoCoMo / BEAM / LongMemEval / RecSys brains or their ledger rows.",+ "",+ ]+ return "\n".join(lines)+++def print_report_table(report: dict[str, Any]) -> None:+ table = Table(title=f"Search {report.get('run_id')}")+ table.add_column("metric")+ table.add_column("value")+ for key in (+ "status",+ "dataset",+ "brain_id",+ "fusion",+ "rerank",+ "channels",+ "rank_pool",+ "skip_enrichment",+ "n_queries",+ "k",+ "ndcg@10",+ "ndcg@20",+ "ndcg@50",+ "ndcg",+ "recall@10",+ "recall@20",+ "recall@50",+ "pool_coverage",+ "mrr",+ "p50_retrieve_ms",+ "p95_retrieve_ms",+ ):+ table.add_row(key, str(report.get(key)))+ console.print(table)+++def entry_from_report(report: dict[str, Any]) -> dict[str, Any] | None:+ if report.get("status") != "ok":+ return None+ n_queries = int(report.get("n_queries") or 0)+ if n_queries < 1:+ return None+ entry = {+ "run_id": report.get("run_id"),+ "brain": report.get("brain_id"),+ "dataset": report.get("dataset"),+ "fusion": report.get("fusion"),+ "rerank": report.get("rerank") or "none",+ "channels": report.get("channels") or ["passages"],+ "rank_pool": bool(report.get("rank_pool")),+ "rank_pool_ce": bool(report.get("rank_pool_ce")),+ "n_queries": n_queries,+ "k": report.get("k"),+ "ndcg@10": report.get("ndcg@10"),+ "ndcg@20": report.get("ndcg@20"),+ "ndcg@50": report.get("ndcg@50"),+ "ndcg": report.get("ndcg"),+ "recall@10": report.get("recall@10"),+ "recall@20": report.get("recall@20"),+ "recall@50": report.get("recall@50"),+ "mrr": report.get("mrr"),+ "pool_coverage": report.get("pool_coverage"),+ "p50_retrieve_ms": report.get("p50_retrieve_ms"),+ "p95_retrieve_ms": report.get("p95_retrieve_ms"),+ "git_sha": report.get("git_sha"),+ "report_path": f"runs/{report.get('run_id')}/report.json",+ "recorded_at": report.get("recorded_at"),+ "protocol": report.get("protocol"),+ }+ if report.get("brain_id") == "searchbenchwandsgraph":+ entry["claim"] = "architecture-demo"+ if report.get("brain_id") == "searchbenchjdslice":+ return None+ return entry+++def _empty_reports_ledger() -> dict[str, Any]:+ return {+ "schema_version": 2,+ "updated_at": datetime.now(timezone.utc).isoformat(),+ "description": _REPORTS_DESCRIPTION,+ "benchmarks": {},+ }+++def update_reports_json(report: dict[str, Any]) -> None:+ entry = entry_from_report(report)+ if entry is None:+ return++ if REPORTS_PATH.exists():+ data = json.loads(REPORTS_PATH.read_text(encoding="utf-8"))+ else:+ data = _empty_reports_ledger()++ suite = data.setdefault("benchmarks", {}).setdefault(+ _BENCHMARK_ID,+ {"name": _BENCHMARK_NAME, "leaderboard": []},+ )+ leaderboard = list(suite.get("leaderboard") or [])+ run_id = entry["run_id"]+ lb_idx = next(+ (i for i, row in enumerate(leaderboard) if row.get("run_id") == run_id),+ None,+ )+ if lb_idx is not None:+ leaderboard[lb_idx] = entry+ else:+ leaderboard.append(entry)++ suite["name"] = _BENCHMARK_NAME+ suite["leaderboard"] = leaderboard+ data["benchmarks"][_BENCHMARK_ID] = suite+ data["updated_at"] = datetime.now(timezone.utc).isoformat()+ data["description"] = data.get("description") or _REPORTS_DESCRIPTION++ REPORTS_PATH.parent.mkdir(parents=True, exist_ok=True)+ fd, tmp_name = tempfile.mkstemp(+ prefix="REPORTS.",+ suffix=".json.tmp",+ dir=str(REPORTS_PATH.parent),+ )+ try:+ with open(fd, "w", encoding="utf-8") as fh:+ json.dump(data, fh, indent=2)+ fh.write("\n")+ Path(tmp_name).replace(REPORTS_PATH)+ finally:+ tmp_path = Path(tmp_name)+ if tmp_path.exists():+ tmp_path.unlink(missing_ok=True)diff --git a/benchmarks/search/rerank_retrieved.py b/benchmarks/search/rerank_retrieved.pynew file mode 100644index 0000000..a5ad576--- /dev/null+++ b/benchmarks/search/rerank_retrieved.py@@ -0,0 +1,143 @@+from __future__ import annotations++from typing import Any++from search.evaluate import candidate_pool_grades+from search.metrics import aggregate_query_metrics, mrr, ndcg_at_k, recall_at_k+from search.rank_pool import _plugin_rerank, doc_texts, score_ranked_pool+from search.dataset import split_corpus+++def _first_stage_metrics(+ ranked: list[str],+ query_row: dict[str, Any],+ *,+ ks: tuple[int, ...],+) -> dict[str, float]:+ gold = {str(item) for item in (query_row.get("gold_doc_ids") or [])}+ grades = {+ str(doc_id): float(gain)+ for doc_id, gain in (query_row.get("gold_grades") or {}).items()+ if float(gain or 0) > 0+ }+ metrics = {f"recall@{cut}": recall_at_k(ranked, gold, cut) for cut in ks}+ ndcg_ks = tuple(dict.fromkeys((*ks, 10, 20)))+ for cut in ndcg_ks:+ metrics[f"ndcg@{cut}"] = ndcg_at_k(ranked, gold, cut, grades=grades or None)+ metrics["ndcg"] = ndcg_at_k(ranked, gold, max(len(ranked), 1), grades=grades or None)+ metrics["mrr"] = mrr(ranked, gold)+ return metrics+++def run_ce_on_retrieved(+ eval_result: dict[str, Any],+ rows: list[dict[str, Any]],+ *,+ dataset_name: str,+ ks: tuple[int, ...] | None = None,+ brain_id: str | None = None,+) -> dict[str, Any]:+ docs, queries = split_corpus(rows)+ texts = doc_texts(docs)+ by_qid = {str(query.get("qid") or ""): query for query in queries}+ rerank, model = _plugin_rerank()+ source_ks = tuple(int(item) for item in (eval_result.get("ks") or (5, 10, 20)))+ ks = ks or source_ks+ rank_pool = bool(eval_result.get("rank_pool"))+ per_query: list[dict[str, Any]] = []+ missing_text = 0+ for row in eval_result.get("queries") or []:+ hit_ids = [str(item) for item in (row.get("hit_ids") or []) if item]+ qid = str(row.get("qid") or "")+ source = by_qid.get(qid) or {}+ candidates = []+ for doc_id in hit_ids:+ text = texts.get(doc_id) or ""+ if not text:+ missing_text += 1+ candidates.append({"id": doc_id, "text": text, "score": 0.0})+ ranked_rows = rerank(+ str(row.get("query") or source.get("query") or ""),+ candidates,+ len(candidates) or 1,+ )+ ranked = [str(item.get("id") or "") for item in ranked_rows if item.get("id")]+ seen = set(ranked)+ for doc_id in hit_ids:+ if doc_id not in seen:+ ranked.append(doc_id)+ if rank_pool and source:+ metrics = score_ranked_pool(ranked, source, ks=ks)+ else:+ metrics = _first_stage_metrics(ranked, row, ks=ks)+ per_query.append(+ {+ "qid": row.get("qid"),+ "query": row.get("query"),+ "slice": row.get("slice") or "unspecified",+ "gold_doc_ids": list(row.get("gold_doc_ids") or []),+ "gold_grades": row.get("gold_grades") or {},+ "candidate_doc_ids": row.get("candidate_doc_ids") or [],+ "pool_size": row.get("pool_size"),+ "pool_coverage": row.get("pool_coverage"),+ "missing_from_brain": row.get("missing_from_brain") or [],+ "hit_ids": ranked,+ "metrics": metrics,+ "retrieve_ms": float(row.get("retrieve_ms") or 0.0),+ "embed_ms": row.get("embed_ms"),+ "client_wall_ms": float(row.get("client_wall_ms") or 0.0),+ "n_hits": len(ranked),+ }+ )+ metrics = aggregate_query_metrics(per_query, ks=ks) if per_query else {}+ if rank_pool and per_query:+ coverages = [+ float(row["pool_coverage"])+ for row in per_query+ if row.get("pool_coverage") is not None+ ]+ if coverages:+ metrics["pool_coverage"] = sum(coverages) / len(coverages)+ metrics["missing_from_brain"] = sum(+ len(row.get("missing_from_brain") or []) for row in per_query+ ) / len(per_query)+ retrieve_k = int(eval_result.get("k") or 0)+ return {+ "status": "ok" if per_query else "failed",+ "brain_id": brain_id or eval_result.get("brain_id"),+ "dataset": dataset_name,+ "fusion": eval_result.get("fusion") or "rrf",+ "rerank": "harness:cross-encoder",+ "channels": eval_result.get("channels") or ["passages"],+ "expand": eval_result.get("expand") or "none",+ "k": retrieve_k,+ "ks": list(ks),+ "n_docs": eval_result.get("n_docs") or len(docs),+ "n_queries": len(per_query),+ "n_docs_mapped": len(texts),+ "skip_enrichment": True,+ "ingest_graph": False,+ "skip_ingest": True,+ "rank_pool": rank_pool,+ "rank_pool_ce": False,+ "ce_model": model,+ "ce_missing_text": missing_text,+ "source_run": eval_result.get("run_id"),+ "ingest": {+ "status": "completed",+ "n_docs": eval_result.get("n_docs") or len(docs),+ "tasks": [],+ "reused": True,+ },+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},+ "event_probe": None,+ "search_error": None,+ "protocol": (+ "harness CE over retrieved first-stage hits "+ f"(k={retrieve_k}, model={model}); not CE-on-pool; "+ "do not cite Reddy 0.857"+ ),+ "metrics": metrics,+ "queries": per_query,+ }diff --git a/tests/fixtures/jdsearch/product_meta_data.txt b/tests/fixtures/jdsearch/product_meta_data.txtnew file mode 100644index 0000000..f47d95e--- /dev/null+++ b/tests/fixtures/jdsearch/product_meta_data.txt@@ -0,0 +1,3 @@+wid name brand_id brand_name cate_id_1 cate_name_1 cate_id_2 cate_name_2 cate_id_3 cate_name_3 cate_id_4 cate_name_4 shop_id+100 111222 b1 brandX c1 cat1 c2 cat2 c3 cat3 c4 cat4 s1+200 333 b2 brandY c1 cat1 c2 cat2 c3 cat3 c4 cat4 s2diff --git a/tests/fixtures/jdsearch/user_behavior_data.txt b/tests/fixtures/jdsearch/user_behavior_data.txtnew file mode 100644index 0000000..d8d53b7--- /dev/null+++ b/tests/fixtures/jdsearch/user_behavior_data.txt@@ -0,0 +1,3 @@+query candidate_wid_list candidate_label_list history_qry_list history_wid_list history_type_list history_time_list+111222 100_200_999 1.0_0.0_0.0 -1 200 CLICK 0_120+333 200_100 2.0_1.0 111 100_200 CART_FLW 0_50_10diff --git a/tests/test_benchmark_contract_guardrails.py b/tests/test_benchmark_contract_guardrails.pyindex 69cd877..2aecc31 100644--- a/tests/test_benchmark_contract_guardrails.py+++ b/tests/test_benchmark_contract_guardrails.py@@ -70,6 +70,36 @@ class BenchmarkContractGuardrailsTests(unittest.TestCase):msg=f"{path} should still target core ingest",)+ def test_search_client_scores_via_search_not_context(self):+ client = ROOT / "benchmarks" / "search" / "client.py"+ self.assertTrue(client.exists(), msg="benchmarks/search/client.py is required")+ text = client.read_text(encoding="utf-8")+ self.assertIn("/retrieve/search", text)+ self.assertIn("/ingest/", text)+ self.assertIn("skip_enrichment", text)+ self.assertNotIn(+ "/retrieve/context",+ text,+ msg="search harness must not score via /retrieve/context",+ )+ self.assertNotIn(+ "/retrieve/recommend",+ text,+ msg="search harness must not score via /retrieve/recommend",+ )+ for needle in ("locomoconv", "demorecsys"):+ self.assertNotIn(+ needle,+ text,+ msg=f"search client must not default to {needle}",+ )++ def test_search_config_defaults_to_searchbench(self):+ config = ROOT / "benchmarks" / "search" / "config.py"+ text = config.read_text(encoding="utf-8")+ self.assertIn('DEFAULT_BRAIN_ID = "searchbenchsmoke"', text)+ self.assertIn('REQUIRED_PREFIX = "searchbench"', text)+if __name__ == "__main__":unittest.main()diff --git a/tests/test_search_bench.py b/tests/test_search_bench.pynew file mode 100644index 0000000..62d0695--- /dev/null+++ b/tests/test_search_bench.py@@ -0,0 +1,1678 @@+import json+import tempfile+import unittest+from pathlib import Path+from unittest.mock import patch+++class SearchBenchTests(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ import sys++ root = Path(__file__).resolve().parents[1]+ bench = root / "benchmarks"+ if str(bench) not in sys.path:+ sys.path.insert(0, str(bench))++ def test_brain_id_guard(self):+ from search.config import DEFAULT_BRAIN_ID, validate_brain_id++ self.assertEqual(validate_brain_id("searchbenchsmoke"), "searchbenchsmoke")+ self.assertEqual(validate_brain_id(DEFAULT_BRAIN_ID), DEFAULT_BRAIN_ID)+ self.assertEqual(validate_brain_id("searchbench1"), "searchbench1")+ self.assertEqual(validate_brain_id("searchbenchjdslice"), "searchbenchjdslice")+ for bad in (+ "locomoconv26",+ "locomoconv26clean",+ "beam1m1clean",+ "beam100k1",+ "demorecsys",+ "lme-s-smoke",+ "default",+ "search",+ ):+ with self.assertRaises(SystemExit):+ validate_brain_id(bad)++ def test_metrics_ranking_and_latency(self):+ from search.metrics import (+ aggregate_query_metrics,+ mrr,+ ndcg_at_k,+ percentile,+ recall_at_k,+ retrieve_latency_ms,+ )++ ranked = ["a", "b", "gold", "d"]+ gold = {"gold"}+ self.assertEqual(recall_at_k(ranked, gold, 2), 0.0)+ self.assertEqual(recall_at_k(ranked, gold, 3), 1.0)+ self.assertEqual(mrr(ranked, gold), 1.0 / 3)+ self.assertGreater(ndcg_at_k(ranked, gold, 10), 0.0)+ self.assertLess(ndcg_at_k(ranked, gold, 10), 1.0)+ self.assertEqual(ndcg_at_k(["gold"], gold, 10), 1.0)+ self.assertGreater(+ ndcg_at_k(+ ["exact", "sub"],+ {"exact", "sub"},+ 10,+ grades={"exact": 1.0, "sub": 0.1},+ ),+ ndcg_at_k(+ ["sub", "exact"],+ {"exact", "sub"},+ 10,+ grades={"exact": 1.0, "sub": 0.1},+ ),+ )+ wands_gold = {"exact", "partial"}+ wands_grades = {"exact": 1.0, "partial": 0.5}+ self.assertGreater(+ ndcg_at_k(["exact", "partial"], wands_gold, 10, grades=wands_grades),+ ndcg_at_k(["partial", "exact"], wands_gold, 10, grades=wands_grades),+ )+ self.assertAlmostEqual(+ ndcg_at_k(["exact", "partial"], wands_gold, 10, grades=wands_grades),+ ndcg_at_k(+ ["exact", "partial"],+ wands_gold,+ 10,+ grades={"exact": 2.0, "partial": 1.0},+ ),+ )+ import math++ from search.metrics import _dcg++ self.assertAlmostEqual(+ _dcg([1.0, 0.5], 2),+ 1.0 / math.log2(2) + 0.5 / math.log2(3),+ )+ self.assertEqual(recall_at_k(["exact", "other"], wands_gold, 10), 0.5)+ self.assertEqual(percentile([10.0, 20.0, 30.0], 50), 20.0)++ retrieve, embed = retrieve_latency_ms(+ {+ "stages": [+ {"stage": "embed.query", "wall_ms": 80.0},+ {"stage": "search.retrieve", "wall_ms": 12.5},+ ]+ },+ 200.0,+ )+ self.assertEqual(retrieve, 12.5)+ self.assertEqual(embed, 80.0)+ retrieve_fallback, embed_fallback = retrieve_latency_ms(+ {"stages": [{"stage": "embed.query", "wall_ms": 40.0}]},+ 90.0,+ )+ self.assertEqual(retrieve_fallback, 50.0)+ self.assertEqual(embed_fallback, 40.0)++ rows = [+ {+ "slice": "keyword",+ "metrics": {+ "recall@5": 1.0,+ "recall@10": 1.0,+ "recall@20": 1.0,+ "ndcg@10": 1.0,+ "ndcg@20": 1.0,+ "ndcg": 1.0,+ "mrr": 1.0,+ },+ "retrieve_ms": 10.0,+ "embed_ms": 50.0,+ "client_wall_ms": 70.0,+ },+ {+ "slice": "paraphrase",+ "metrics": {+ "recall@5": 0.0,+ "recall@10": 0.0,+ "recall@20": 1.0,+ "ndcg@10": 0.0,+ "ndcg@20": 0.5,+ "ndcg": 0.5,+ "mrr": 0.0,+ },+ "retrieve_ms": 30.0,+ "embed_ms": 50.0,+ "client_wall_ms": 90.0,+ },+ ]+ metrics = aggregate_query_metrics(rows)+ self.assertEqual(metrics["recall@10"], 0.5)+ self.assertEqual(metrics["ndcg@10"], 0.5)+ self.assertEqual(metrics["ndcg@20"], 0.75)+ self.assertEqual(metrics["by_slice"]["keyword"]["recall@10"], 1.0)+ self.assertEqual(metrics["p50_retrieve_ms"], 20.0)++ from search.metrics import ndcg_at_k++ ranked_pool = ["exact", "irr", "sub", "other"]+ grades_pool = {"exact": 1.0, "sub": 0.1, "irr": 0.0}+ gold_pool = {"exact", "sub"}+ manual = ndcg_at_k(ranked_pool, gold_pool, 20, grades=grades_pool)+ self.assertGreater(manual, 0.0)+ self.assertEqual(+ ndcg_at_k(ranked_pool, gold_pool, 20, grades=grades_pool),+ ndcg_at_k(ranked_pool, gold_pool, 4, grades=grades_pool),+ )++ def test_doc_marker_mapping(self):+ from search.dataset import dataset_stats, load_records, map_doc_ids_to_chunks++ root = Path(__file__).resolve().parents[1]+ rows = load_records(root / "benchmarks" / "data" / "search_toy.jsonl")+ stats = dataset_stats(rows)+ self.assertGreaterEqual(stats["n_docs"], 6)+ self.assertGreaterEqual(stats["n_queries"], 8)+ self.assertIn("keyword", stats["slices"])+ self.assertIn("paraphrase", stats["slices"])+ mapping = map_doc_ids_to_chunks(+ [{"doc_id": "license-alice"}],+ [+ {"id": "c1", "text": "DOCID license-alice. Alice completed her license."},+ {"id": "c2", "text": "unrelated"},+ ],+ )+ self.assertEqual(mapping["license-alice"], {"c1"})++ def test_gold_matches_chunk_or_node_uuid(self):+ from search.evaluate import gold_hit_grades, gold_hit_ids+ from search.metrics import ndcg_at_k++ query = {"gold_doc_ids": ["sku-1"], "gold_grades": {"sku-1": 1.0}}+ mapped = {"sku-1": {"chunk-1"}}+ gold = gold_hit_ids(query, mapped)+ self.assertEqual(gold, {"sku-1"})+ grades = gold_hit_grades(query, mapped)+ self.assertEqual(ndcg_at_k(["sku-1"], gold, 10, grades=grades), 1.0)+ from search.evaluate import canonicalize_hit_ids, invert_doc_chunks++ ranked_chunk = canonicalize_hit_ids(["chunk-1"], invert_doc_chunks(mapped))+ self.assertEqual(ndcg_at_k(ranked_chunk, gold, 10, grades=grades), 1.0)+ self.assertEqual(ndcg_at_k(["other"], gold, 10, grades=grades), 0.0)++ def test_wands_recall_gold_is_exact_and_partial(self):+ from search.evaluate import gold_hit_ids+ from search.metrics import recall_at_k++ query = {+ "gold_doc_ids": ["exact", "partial"],+ "gold_grades": {"exact": 1.0, "partial": 0.5},+ }+ gold = gold_hit_ids(query, {})+ self.assertEqual(gold, {"exact", "partial"})+ self.assertEqual(recall_at_k(["exact", "irr"], gold, 10), 0.5)+ self.assertEqual(recall_at_k(["exact", "partial"], gold, 10), 1.0)+ unlabeled = gold_hit_ids(+ {+ "gold_doc_ids": ["exact"],+ "gold_grades": {"exact": 1.0, "partial": 0.5},+ },+ {},+ )+ self.assertEqual(unlabeled, {"exact", "partial"})+ irr_excluded = gold_hit_ids(+ {+ "gold_doc_ids": ["exact"],+ "gold_grades": {"exact": 1.0, "irr": 0.0},+ },+ {},+ )+ self.assertEqual(irr_excluded, {"exact"})++ def test_replay_fusion_compact_hub_drop(self):+ from search.replay_fusion import replay++ root = Path(__file__).resolve().parents[1]+ graph_path = (+ root / "benchmarks" / "runs" / "search-esci-slice-allcols" / "eval.json"+ )+ passages_path = (+ root / "benchmarks" / "runs" / "search-esci-slice-passages" / "eval.json"+ )+ if not graph_path.exists() or not passages_path.exists():+ self.skipTest("missing paired ESCI slice eval.json files")+ result = replay(+ json.loads(graph_path.read_text(encoding="utf-8")),+ json.loads(passages_path.read_text(encoding="utf-8")),+ )+ self.assertEqual(result["n_queries"], 11)+ self.assertAlmostEqual(result["hub_drop"]["recall@10"], 0.4758, places=3)+ self.assertAlmostEqual(result["hub_drop"]["mrr"], 0.8485, places=3)++ def test_unique_doc_counts_and_channel_lists(self):+ from search.evaluate import channel_id_lists, unique_doc_counts++ raw = ["chunk-1", "sku-1", "chunk-2", "hub:attr:red"]+ mapping = {"chunk-1": "sku-1", "chunk-2": "sku-2"}+ n_raw, n_canon = unique_doc_counts(raw, mapping, k=20)+ self.assertEqual(n_raw, 4)+ self.assertEqual(n_canon, 3)+ lists = channel_id_lists(+ {+ "channel_lists": {+ "dense": ["chunk-1"],+ "bm25": ["chunk-2"],+ "entities": ["sku-1"],+ "communities": ["sku-3"],+ }+ },+ [{"id": "chunk-1", "channel": "passages", "doc_id": "sku-1"}],+ )+ self.assertEqual(lists["dense_ids"], ["chunk-1"])+ self.assertEqual(lists["entity_ids"], ["sku-1"])+ self.assertEqual(lists["community_ids"], ["sku-3"])+ self.assertEqual(lists["passage_ids"], ["chunk-1"])++ def test_replay_collapse_and_gated_arms(self):+ from search.replay_fusion import pick_offline_winner, replay_offline++ gold = [f"d{i}" for i in range(1, 21)]+ grades = {doc: 1.0 for doc in gold}+ passages_hits = [+ {"id": f"chunk-{item}", "channel": "passages", "doc_id": item}+ for item in gold+ ]+ passages = {+ "queries": [+ {+ "qid": "q1",+ "gold_doc_ids": gold,+ "gold_grades": grades,+ "hit_ids": gold,+ "dense_ids": [f"chunk-{item}" for item in gold],+ "bm25_ids": [f"chunk-{item}" for item in gold],+ "passage_ids": [f"chunk-{item}" for item in gold],+ "hits": passages_hits,+ "metrics": {+ "ndcg@10": 1.0,+ "recall@10": 0.5,+ "recall@20": 1.0,+ "mrr": 1.0,+ },+ }+ ]+ }+ fused_raw = [f"chunk-{item}" for item in gold[:8]] + gold[:8] + gold[8:12]+ graph = {+ "queries": [+ {+ "qid": "q1",+ "gold_doc_ids": gold,+ "gold_grades": grades,+ "hit_ids": gold[:12],+ "entity_ids": gold[:5],+ "community_ids": ["d21"],+ "hits": [+ {+ "id": hid,+ "channel": "passages" if hid.startswith("chunk-") else "entities",+ "doc_id": hid.replace("chunk-", ""),+ }+ for hid in fused_raw+ ],+ "metrics": {+ "ndcg@10": 0.6,+ "recall@10": 0.4,+ "recall@20": 0.6,+ "mrr": 1.0,+ },+ }+ ]+ }+ result = replay_offline(graph, passages)+ self.assertEqual(result["n_queries"], 1)+ self.assertGreaterEqual(+ result["arms"]["collapse-rrf"]["recall@20"],+ result["arms"]["passages"]["recall@20"] - 1e-9,+ )+ self.assertEqual(result["arms"]["passages"]["unique_docs@20"], 20.0)+ self.assertEqual(result["arms"]["confirmation"]["unique_docs@20"], 20.0)+ self.assertEqual(result["arms"]["expansion-n10"]["unique_docs@20"], 20.0)+ winner = pick_offline_winner(+ {+ "passages": {+ "ndcg@10": 0.758,+ "recall@20": 0.847,+ },+ "collapse-rrf": {+ "ndcg@10": 0.640,+ "recall@20": 0.847,+ },+ "expansion-n10": {+ "ndcg@10": 0.758,+ "recall@20": 0.847,+ },+ }+ )+ self.assertEqual(winner, "expansion-n10")+ self.assertEqual(+ pick_offline_winner(+ {+ "passages": {"ndcg@10": 0.758, "recall@20": 0.847},+ "collapse-rrf": {"ndcg@10": 0.640, "recall@20": 0.847},+ "expansion-n10": {"ndcg@10": 0.758, "recall@20": 0.815},+ }+ ),+ "G08",+ )+ self.assertEqual(+ pick_offline_winner(+ {+ "passages": {"ndcg@10": 0.758, "recall@20": 0.847},+ "collapse-rrf": {"ndcg@10": 0.640, "recall@20": 0.847},+ }+ ),+ "G08",+ )++ def test_score_search_dumps_channel_and_doc_id(self):+ from search.client import TimedResult+ from search.evaluate import score_search_result+ from search.metrics import ndcg_at_k, recall_at_k++ result = TimedResult(+ data={+ "hits": [+ {"id": "chunk-1", "channel": "passages", "node_id": "sku-1"},+ {"id": "sku-1", "channel": "entities", "node_id": "sku-1"},+ {"id": "hub:attr:red", "channel": "communities"},+ ],+ "stage_timings": {+ "stages": [{"stage": "search.retrieve", "wall_ms": 11.0}]+ },+ },+ latency_ms=40.0,+ )+ gold = {"sku-1"}+ grades = {"sku-1": 1.0}+ scored = score_search_result(+ result,+ gold=gold,+ ks=(5, 10, 20, 50),+ grades=grades,+ chunk_to_doc={"chunk-1": "sku-1"},+ )+ self.assertEqual(scored["hit_ids"], ["sku-1", "hub:attr:red"])+ self.assertEqual(+ scored["hits"],+ [+ {+ "id": "chunk-1",+ "channel": "passages",+ "doc_id": "sku-1",+ "node_id": "sku-1",+ },+ {+ "id": "sku-1",+ "channel": "entities",+ "doc_id": "sku-1",+ "node_id": "sku-1",+ },+ {+ "id": "hub:attr:red",+ "channel": "communities",+ "doc_id": "hub:attr:red",+ "node_id": None,+ },+ ],+ )+ self.assertEqual(+ scored["metrics"]["recall@10"],+ recall_at_k(scored["hit_ids"], gold, 10),+ )+ self.assertEqual(+ scored["metrics"]["ndcg@10"],+ ndcg_at_k(scored["hit_ids"], gold, 10, grades=grades),+ )+ self.assertEqual(+ [row["doc_id"] for row in scored["hits"][:1]],+ scored["hit_ids"][:1],+ )+ self.assertEqual(scored["n_unique_docs_raw"], 3)+ self.assertEqual(scored["n_unique_docs_canonical"], 2)+ self.assertEqual(scored["unique_docs_k"], 50)+ self.assertEqual(scored["n_unique_docs_retrieve_canonical"], 2)+ self.assertIn("ndcg@50", scored["metrics"])+ self.assertEqual(scored["entity_ids"], ["sku-1"])+ self.assertEqual(scored["community_ids"], ["hub:attr:red"])+ self.assertEqual(scored["passage_ids"], ["chunk-1"])++ def test_rank_pool_filters_and_keeps_irrelevant_grades(self):+ from search.client import TimedResult+ from search.evaluate import (+ candidate_pool_grades,+ candidate_pool_ids,+ filter_ranked_to_pool,+ score_search_result,+ )+ from search.metrics import ndcg_at_k++ query = {+ "gold_doc_ids": ["p-e"],+ "gold_grades": {"p-e": 1.0, "p-s": 0.1},+ "candidate_doc_ids": ["p-e", "p-s", "p-i"],+ "candidate_grades": {"p-e": 1.0, "p-s": 0.1, "p-i": 0.0},+ }+ self.assertEqual(candidate_pool_ids(query), ["p-e", "p-s", "p-i"])+ self.assertEqual(candidate_pool_grades(query)["p-i"], 0.0)+ self.assertEqual(+ filter_ranked_to_pool(["p-x", "p-e", "p-i", "p-y"], query["candidate_doc_ids"]),+ ["p-e", "p-i"],+ )+ result = TimedResult(+ data={+ "hits": [+ {"id": "outside", "channel": "passages"},+ {"id": "p-e", "channel": "passages"},+ {"id": "p-i", "channel": "passages"},+ ],+ "stage_timings": {+ "stages": [{"stage": "search.retrieve", "wall_ms": 9.0}]+ },+ },+ latency_ms=12.0,+ )+ scored = score_search_result(+ result,+ gold={"p-e", "p-s"},+ ks=(5, 10, 20),+ grades=candidate_pool_grades(query),+ pool_ids=candidate_pool_ids(query),+ )+ self.assertEqual(scored["hit_ids"], ["p-e", "p-i"])+ expected = ndcg_at_k(+ ["p-e", "p-i"],+ {"p-e", "p-s"},+ 20,+ grades=candidate_pool_grades(query),+ )+ self.assertEqual(scored["metrics"]["ndcg@20"], expected)++ def test_rank_pool_ce_uses_injected_predict(self):+ import sys++ root = Path(__file__).resolve().parents[1]+ plugin = root / "plugins" / "search-rerank"+ if str(plugin) not in sys.path:+ sys.path.insert(0, str(plugin))+ import rerank as rerank_mod++ from search.rank_pool import run_ce_on_pool++ def predict(pairs):+ scores = []+ for _, text in pairs:+ if "gold" in text:+ scores.append(2.0)+ elif "partial" in text:+ scores.append(1.0)+ else:+ scores.append(0.0)+ return scores++ rerank_mod.set_predict(predict)+ try:+ rows = [+ {+ "type": "doc",+ "doc_id": "irr",+ "text": "DOCID irr. Title: other",+ },+ {+ "type": "doc",+ "doc_id": "gold",+ "text": "DOCID gold. Title: gold kettle",+ },+ {+ "type": "doc",+ "doc_id": "sub",+ "text": "DOCID sub. Title: partial kettle",+ },+ {+ "type": "query",+ "qid": "esci-1",+ "query": "kettle",+ "gold_doc_ids": ["gold", "sub"],+ "gold_grades": {"gold": 1.0, "sub": 0.1},+ "candidate_doc_ids": ["irr", "gold", "sub"],+ "candidate_grades": {"irr": 0.0, "gold": 1.0, "sub": 0.1},+ "slice": "esci-us",+ },+ ]+ result = run_ce_on_pool(rows, dataset_name="toy-pool.jsonl")+ finally:+ rerank_mod.set_predict(None)+ self.assertEqual(result["status"], "ok")+ self.assertTrue(result["rank_pool"])+ self.assertTrue(result["rank_pool_ce"])+ self.assertEqual(result["queries"][0]["hit_ids"][0], "gold")+ self.assertGreater(result["metrics"]["ndcg@20"], 0.0)++ def test_ce_on_retrieved_reorders_hits_only(self):+ import sys++ root = Path(__file__).resolve().parents[1]+ plugin = root / "plugins" / "search-rerank"+ if str(plugin) not in sys.path:+ sys.path.insert(0, str(plugin))+ import rerank as rerank_mod++ from search.rerank_retrieved import run_ce_on_retrieved++ def predict(pairs):+ scores = []+ for _, text in pairs:+ if "gold" in text:+ scores.append(2.0)+ elif "partial" in text:+ scores.append(1.0)+ else:+ scores.append(0.0)+ return scores++ rerank_mod.set_predict(predict)+ try:+ rows = [+ {+ "type": "doc",+ "doc_id": "irr",+ "text": "DOCID irr. Title: other",+ },+ {+ "type": "doc",+ "doc_id": "gold",+ "text": "DOCID gold. Title: gold kettle",+ },+ {+ "type": "doc",+ "doc_id": "sub",+ "text": "DOCID sub. Title: partial kettle",+ },+ {+ "type": "query",+ "qid": "esci-1",+ "query": "kettle",+ "gold_doc_ids": ["gold", "sub"],+ "gold_grades": {"gold": 1.0, "sub": 0.1},+ "slice": "esci-us",+ },+ ]+ eval_result = {+ "k": 50,+ "ks": [5, 10, 20, 50],+ "rank_pool": False,+ "fusion": "rrf",+ "channels": ["passages"],+ "n_docs": 3,+ "queries": [+ {+ "qid": "esci-1",+ "query": "kettle",+ "slice": "esci-us",+ "gold_doc_ids": ["gold", "sub"],+ "gold_grades": {"gold": 1.0, "sub": 0.1},+ "hit_ids": ["irr", "sub", "gold"],+ "metrics": {},+ "retrieve_ms": 12.0,+ "pool_coverage": None,+ }+ ],+ }+ result = run_ce_on_retrieved(+ eval_result,+ rows,+ dataset_name="toy-retrieved.jsonl",+ )+ finally:+ rerank_mod.set_predict(None)+ self.assertEqual(result["status"], "ok")+ self.assertFalse(result["rank_pool_ce"])+ self.assertEqual(result["rerank"], "harness:cross-encoder")+ self.assertEqual(result["queries"][0]["hit_ids"][0], "gold")+ self.assertEqual(set(result["queries"][0]["hit_ids"]), {"irr", "sub", "gold"})+ self.assertGreater(result["metrics"]["ndcg@10"], 0.0)++ def test_electronics_marker_for_esci72_style_titles(self):+ from search.miss import looks_like_electronics++ self.assertTrue(looks_like_electronics("Apple iPad Air 2, 16 GB"))+ self.assertFalse(looks_like_electronics("Price Tags Without String"))++ def test_miss_strata_classifies_head_tail_and_total_miss(self):+ from search.miss_strata import classify_eval, rewrite_query++ self.assertIsNone(rewrite_query("esci-72", "$100 things that are not electronics"))+ self.assertIsNone(rewrite_query("esci-177", "p_num: integer not null i_num: integer"))+ rewritten = rewrite_query(+ "esci-113",+ ""tortillas without interesterified soybean oil”",+ )+ self.assertIsNotNone(rewritten)+ self.assertNotIn(""", rewritten)++ eval_result = {+ "k": 50,+ "queries": [+ {+ "qid": "q-head",+ "query": "red mug",+ "hit_ids": ["g1", "x"] + [f"n{i}" for i in range(48)],+ "gold_doc_ids": ["g1", "g2"],+ "gold_grades": {"g1": 1.0, "g2": 0.1},+ "metrics": {"recall@10": 0.5, "recall@50": 0.5},+ },+ {+ "qid": "q-tail",+ "query": "blue mug",+ "hit_ids": ["n0"] * 10 + ["g2"] + [f"n{i}" for i in range(39)],+ "gold_doc_ids": ["g2"],+ "gold_grades": {"g2": 1.0},+ "metrics": {"recall@10": 0.0, "recall@50": 1.0},+ },+ {+ "qid": "esci-113",+ "query": ""tortillas without interesterified soybean oil”",+ "hit_ids": [f"n{i}" for i in range(50)],+ "gold_doc_ids": ["g3"],+ "gold_grades": {"g3": 1.0},+ "metrics": {"recall@10": 0.0, "recall@50": 0.0},+ },+ ],+ }+ rows = [+ {"type": "doc", "doc_id": "g1", "text": "Title: red mug"},+ {"type": "doc", "doc_id": "g2", "text": "Title: blue mug"},+ {"type": "doc", "doc_id": "g3", "text": "Title: tortillas"},+ {+ "type": "query",+ "qid": "q-head",+ "query": "red mug",+ "gold_doc_ids": ["g1", "g2"],+ "gold_grades": {"g1": 1.0, "g2": 0.1},+ },+ {+ "type": "query",+ "qid": "q-tail",+ "query": "blue mug",+ "gold_doc_ids": ["g2"],+ "gold_grades": {"g2": 1.0},+ },+ {+ "type": "query",+ "qid": "esci-113",+ "query": ""tortillas without interesterified soybean oil”",+ "gold_doc_ids": ["g3"],+ "gold_grades": {"g3": 1.0},+ },+ ]+ taxonomy = classify_eval(eval_result, rows, k=50)+ self.assertEqual(taxonomy["n_queries"], 3)+ self.assertEqual(taxonomy["n_gold"], 4)+ self.assertEqual(taxonomy["stratum_counts"]["head-ok"], 1)+ self.assertEqual(taxonomy["stratum_counts"]["rank-too-low"], 1)+ self.assertEqual(taxonomy["stratum_counts"]["total-miss"], 1)+ self.assertTrue(taxonomy["run_query_side"])+ self.assertEqual(taxonomy["rewritable_qids"], ["esci-113"])++ def test_spell_normalize_nfkc_and_accents(self):+ from search.miss_strata import normalize_spelling, write_spell_jsonl++ self.assertEqual(normalize_spelling("niños!!!"), "ninos")+ self.assertEqual(normalize_spelling(" sofa, azul. "), "sofa azul")+ self.assertEqual(normalize_spelling("caf\u00e9"), "cafe")+ with tempfile.TemporaryDirectory() as tmp:+ dest = Path(tmp) / "spell.jsonl"+ summary = write_spell_jsonl(+ [+ {"type": "doc", "doc_id": "d1", "text": "sofa"},+ {"type": "query", "qid": "q1", "query": "sofá azul!!"},+ ],+ dest,+ )+ self.assertEqual(summary["n_queries_changed"], 1)+ rows = dest.read_text(encoding="utf-8").strip().splitlines()+ query = json.loads(rows[1])+ self.assertEqual(query["query"], "sofa azul")+ self.assertEqual(query["query_original"], "sofá azul!!")++ def test_doc_meta_keys_copies_generic_catalog_fields(self):+ from search.evaluate import doc_meta_keys++ self.assertEqual(+ doc_meta_keys(+ {+ "doc_id": "it-bollitore",+ "brand": "CasaLuce",+ "color": "argento",+ "locale": "it",+ "title": "ignored",+ }+ ),+ {"brand": "CasaLuce", "color": "argento", "locale": "it"},+ )+ self.assertIsNone(doc_meta_keys({"doc_id": "x", "text": "only text"}))++ def test_dense_holdout_excludes_eval_qids(self):+ from search.finetune_esci_ce import held_out_query_ids++ holdout = held_out_query_ids(+ Path(__file__).resolve().parents[1]+ / "benchmarks"+ / "data"+ / "search_esci_74.jsonl"+ )+ self.assertIn("72", holdout)+ self.assertIn("esci-72", holdout)+ self.assertGreater(len(holdout), 74)++ def test_local_dense_scores_injected_ranks(self):+ from search.local_dense import evaluate_dense++ queries = [+ {+ "qid": "q1",+ "query": "red mug",+ "slice": "toy",+ "gold_doc_ids": ["gold"],+ "gold_grades": {"gold": 1.0, "sub": 0.1},+ }+ ]+ ranked = {"q1": ["gold", "sub", "irr"]}+ metrics, per_query = evaluate_dense(+ ranked, queries, ks=(5, 10, 20, 50), encode_ms=10.0+ )+ self.assertEqual(per_query[0]["hit_ids"][0], "gold")+ self.assertEqual(metrics["recall@10"], 1.0)+ self.assertGreater(metrics["ndcg@10"], 0.0)++ def test_exhaustive_ce_ranks_exact_above_irrelevant(self):+ from search.rank_corpus import PROTOCOL, run_exhaustive_ce++ rows = [+ {"type": "doc", "doc_id": "irr", "text": "DOCID irr. Title: other"},+ {"type": "doc", "doc_id": "exact", "text": "DOCID exact. Title: gold kettle"},+ {"type": "doc", "doc_id": "maybe", "text": "DOCID maybe. Title: similar kettle"},+ {+ "type": "query",+ "qid": "esci-toy",+ "query": "kettle",+ "gold_doc_ids": ["exact"],+ "gold_grades": {"exact": 1.0, "maybe": 0.1},+ "slice": "toy",+ },+ ]++ def predict(pairs):+ out = []+ for _, text in pairs:+ if "gold kettle" in text:+ out.append([0.95, 0.03, 0.01, 0.01])+ elif "similar" in text:+ out.append([0.05, 0.80, 0.10, 0.05])+ else:+ out.append([0.01, 0.02, 0.07, 0.90])+ return out++ result = run_exhaustive_ce(+ rows,+ predict=predict,+ model_name="toy-4class",+ dataset_name="toy-corpus.jsonl",+ k=3,+ ks=(5, 10, 20, 50),+ )+ self.assertEqual(result["status"], "ok")+ self.assertEqual(result["protocol"], PROTOCOL)+ self.assertFalse(result["rank_pool"])+ self.assertEqual(result["queries"][0]["hit_ids"][0], "exact")+ self.assertLess(result["queries"][0]["hit_ids"].index("exact"), result["queries"][0]["hit_ids"].index("irr"))+ self.assertGreater(result["metrics"]["ndcg@10"], 0.0)++ def test_list_overlap_counts_sidecar_unique_golds(self):+ from search.list_overlap import run_union, summarize_overlap, unique_golds++ gold = {"g1", "g2", "g3"}+ extra = unique_golds(+ ["g2", "noise", "g3"],+ ["g1", "g2", "other"],+ gold,+ k=50,+ )+ self.assertEqual(extra, ["g3"])+ passages = {+ "run_id": "passages",+ "queries": [+ {+ "qid": "q1",+ "hit_ids": ["g1", "g2", "p1"],+ "gold_doc_ids": ["g1", "g2", "g3"],+ "gold_grades": {"g1": 1.0, "g2": 1.0, "g3": 0.1},+ }+ ],+ }+ bge = {+ "run_id": "bge",+ "queries": [{"qid": "q1", "hit_ids": ["g3", "g1", "b1"]}],+ }+ summary = summarize_overlap(passages, {"bge": bge}, k=50)+ self.assertEqual(summary["runs"]["bge"]["unique_gold_hits"], 1)+ rows = [+ {"type": "doc", "doc_id": "g1", "text": "g1"},+ {"type": "doc", "doc_id": "g2", "text": "g2"},+ {"type": "doc", "doc_id": "g3", "text": "g3"},+ {+ "type": "query",+ "qid": "q1",+ "query": "mug",+ "gold_doc_ids": ["g1", "g2", "g3"],+ "gold_grades": {"g1": 1.0, "g2": 1.0, "g3": 0.1},+ },+ ]+ union = run_union(+ [passages, bge],+ rows,+ dataset_name="toy.jsonl",+ k=50,+ ks=(5, 10, 20, 50),+ run_names=["passages", "bge"],+ )+ self.assertEqual(union["channels"], ["harness-union"])+ self.assertIn("g3", union["queries"][0]["hit_ids"][:10])+ self.assertGreaterEqual(union["metrics"]["recall@10"], 1.0)++ def test_cascade_freezes_head_and_injects_tail_golds(self):+ from search.list_overlap import CASCADE_PROTOCOL, cascade_frozen_head, run_cascade++ passages = [f"h{index}" for index in range(10)] + ["tail_gold", "irr"]+ gold = {"h0", "tail_gold", "extra"}+ ranked = cascade_frozen_head(+ passages,+ [["extra", "noise", "h0"]],+ gold,+ head_k=10,+ k=50,+ )+ self.assertEqual(ranked[:10], passages[:10])+ self.assertIn("extra", ranked[10:])+ self.assertNotIn("extra", ranked[:10])+ self.assertIn("tail_gold", ranked)+ rows = [+ {"type": "doc", "doc_id": doc_id, "text": doc_id}+ for doc_id in [*passages, "extra"]+ ]+ rows.append(+ {+ "type": "query",+ "qid": "q1",+ "query": "mug",+ "gold_doc_ids": ["h0", "tail_gold", "extra"],+ "gold_grades": {"h0": 1.0, "tail_gold": 1.0, "extra": 0.1},+ }+ )+ result = run_cascade(+ {"run_id": "passages", "queries": [{"qid": "q1", "hit_ids": passages}]},+ [{"run_id": "bge", "queries": [{"qid": "q1", "hit_ids": ["extra", "noise"]}]}],+ rows,+ dataset_name="toy.jsonl",+ k=50,+ head_k=10,+ ks=(5, 10, 20, 50),+ run_names=["passages", "bge"],+ )+ self.assertEqual(result["protocol"], CASCADE_PROTOCOL)+ self.assertEqual(result["channels"], ["harness-cascade"])+ self.assertEqual(result["queries"][0]["hit_ids"][:10], passages[:10])+ self.assertIn("extra", result["queries"][0]["hit_ids"][10:])+ self.assertEqual(result["metrics"]["recall@10"], 1.0 / 3.0)++ def test_frozen_head_merge_matches_harness_cascade(self):+ from src.core.search.hybrid import frozen_head_merge+ from search.list_overlap import cascade_frozen_head++ passages = [f"h{index}" for index in range(10)] + ["tail_gold", "irr"]+ sidecars = [["extra", "noise", "h0"]]+ gold = {"h0", "tail_gold", "extra"}+ ranked = frozen_head_merge(+ passages,+ sidecars,+ head_k=10,+ k=50,+ prefer_ids=gold,+ )+ self.assertEqual(+ ranked,+ cascade_frozen_head(passages, sidecars, gold, head_k=10, k=50),+ )+ live = frozen_head_merge(passages, sidecars, head_k=10, k=50)+ self.assertEqual(live[:10], passages[:10])+ self.assertIn("extra", live[10:])+ self.assertIn("noise", live[10:])++ def test_cascade_replay_esci74_stored_lists(self):+ from search.dataset import load_records+ from search.list_overlap import load_eval_run, run_cascade++ root = Path(__file__).resolve().parents[1]+ runs = root / "benchmarks" / "runs"+ passages_path = runs / "search-esci-74-passages-k50" / "eval.json"+ bge_path = runs / "search-esci-74-bge-base-k50" / "eval.json"+ colbert_path = runs / "search-esci-74-colbert-k50" / "eval.json"+ dataset = root / "benchmarks" / "data" / "search_esci_74.jsonl"+ if not all(+ path.exists()+ for path in (passages_path, bge_path, colbert_path, dataset)+ ):+ self.skipTest("stored US ESCI n=74 evals not present")+ passages = load_eval_run(passages_path.parent)+ bge = load_eval_run(bge_path.parent)+ colbert = load_eval_run(colbert_path.parent)+ rows = load_records(dataset)+ result = run_cascade(+ passages,+ [bge, colbert],+ rows,+ dataset_name="search_esci_74.jsonl",+ k=50,+ head_k=10,+ ks=(5, 10, 20, 50),+ run_names=["passages", "bge", "colbert"],+ )+ metrics = result["metrics"]+ self.assertAlmostEqual(metrics["ndcg@10"], 0.500, places=3)+ self.assertAlmostEqual(metrics["recall@10"], 0.379, places=3)+ self.assertAlmostEqual(metrics["recall@50"], 0.889, places=3)++ def test_ltr_head_promotes_title_overlap_on_held_out_query(self):+ from search.ltr_head import (+ example_from_eval_row,+ features_for_doc,+ fit_pairwise,+ rerank_ids,+ run_ltr_head,+ )++ feats = features_for_doc(+ "navy velvet sofa",+ "hit",+ rrf_ids=["miss", "hit"],+ bm25_ids=["hit"],+ dense_ids=["miss"],+ doc={"title": "Navy velvet sofa two seats", "brand": "AtelierNord"},+ )+ self.assertGreater(feats[3], 0.5)+ self.assertEqual(feats[4], 0.0)+ train_row = {+ "qid": "q-train",+ "query": "navy velvet sofa",+ "hit_ids": ["irr", "gold"],+ "bm25_ids": ["gold"],+ "dense_ids": ["irr"],+ "hits": [+ {"id": "irr", "doc_id": "irr"},+ {"id": "gold", "doc_id": "gold"},+ ],+ "gold_doc_ids": ["gold"],+ "gold_grades": {"gold": 1.0},+ }+ test_row = {+ "qid": "q-test",+ "query": "navy velvet sofa",+ "hit_ids": ["irr", "gold"],+ "bm25_ids": ["gold"],+ "dense_ids": ["irr"],+ "hits": [+ {"id": "irr", "doc_id": "irr"},+ {"id": "gold", "doc_id": "gold"},+ ],+ "gold_doc_ids": ["gold"],+ "gold_grades": {"gold": 1.0},+ }+ docs = {+ "gold": {"doc_id": "gold", "title": "Navy velvet sofa"},+ "irr": {"doc_id": "irr", "title": "plastic kettle lid"},+ }+ train_ex = example_from_eval_row(train_row, docs, k=10)+ test_ex = example_from_eval_row(test_row, docs, k=10)+ self.assertIsNotNone(train_ex)+ self.assertIsNotNone(test_ex)+ weights = fit_pairwise([train_ex], epochs=20, seed=0)+ ranked = rerank_ids(test_ex, weights)+ self.assertEqual(ranked[0], "gold")+ result = run_ltr_head(+ {"run_id": "toy", "queries": [train_row, test_row]},+ [+ {"type": "doc", "doc_id": "gold", "title": "Navy velvet sofa", "text": "Title: Navy velvet sofa"},+ {"type": "doc", "doc_id": "irr", "title": "plastic kettle lid", "text": "Title: plastic kettle lid"},+ {+ "type": "query",+ "qid": "q-train",+ "query": "navy velvet sofa",+ "gold_doc_ids": ["gold"],+ "gold_grades": {"gold": 1.0},+ },+ {+ "type": "query",+ "qid": "q-test",+ "query": "navy velvet sofa",+ "gold_doc_ids": ["gold"],+ "gold_grades": {"gold": 1.0},+ },+ ],+ dataset_name="toy.jsonl",+ k=10,+ ks=(5, 10),+ n_folds=2,+ )+ self.assertEqual(result["protocol"], "ltr-head-cv")+ self.assertGreaterEqual(result["metrics"]["recall@10"], 1.0)++ def test_ltr_pair_policy_skips_unlabeled_for_other_query_neg(self):+ from search.ltr_head import (+ PAIR_OTHER_QUERY_NEG,+ PAIR_UNLABELED_ZERO,+ collect_pairs,+ example_from_eval_row,+ features_for_doc,+ other_query_gold_ids,+ )++ row_a = {+ "qid": "q-a",+ "query": "navy sofa",+ "hit_ids": ["gold-a", "gold-b", "unlabeled"],+ "hits": [+ {"id": "gold-a", "doc_id": "gold-a"},+ {"id": "gold-b", "doc_id": "gold-b"},+ {"id": "unlabeled", "doc_id": "unlabeled"},+ ],+ "gold_doc_ids": ["gold-a"],+ "gold_grades": {"gold-a": 1.0},+ }+ row_b = {+ "qid": "q-b",+ "query": "steel kettle",+ "hit_ids": ["gold-b", "gold-a", "unlabeled"],+ "hits": [+ {"id": "gold-b", "doc_id": "gold-b"},+ {"id": "gold-a", "doc_id": "gold-a"},+ {"id": "unlabeled", "doc_id": "unlabeled"},+ ],+ "gold_doc_ids": ["gold-b"],+ "gold_grades": {"gold-b": 1.0},+ }+ docs = {+ "gold-a": {"doc_id": "gold-a", "title": "Navy sofa"},+ "gold-b": {"doc_id": "gold-b", "title": "Steel kettle"},+ "unlabeled": {"doc_id": "unlabeled", "title": "random cable"},+ }+ ex_a = example_from_eval_row(row_a, docs, k=10)+ ex_b = example_from_eval_row(row_b, docs, k=10)+ other = other_query_gold_ids([ex_a, ex_b], "q-a")+ self.assertEqual(other, {"gold-b"})+ zero_pairs = collect_pairs(ex_a, pair_policy=PAIR_UNLABELED_ZERO)+ neg_pairs = collect_pairs(+ ex_a, pair_policy=PAIR_OTHER_QUERY_NEG, other_gold=other+ )+ zero_docs = {(ex_a["ids"][hi], ex_a["ids"][lo]) for hi, lo in zero_pairs}+ neg_docs = {(ex_a["ids"][hi], ex_a["ids"][lo]) for hi, lo in neg_pairs}+ self.assertIn(("gold-a", "unlabeled"), zero_docs)+ self.assertNotIn(("gold-a", "unlabeled"), neg_docs)+ self.assertIn(("gold-a", "gold-b"), neg_docs)+ feats = features_for_doc(+ "navy sofa",+ "gold-a",+ rrf_ids=["gold-a"],+ bm25_ids=["gold-a"],+ dense_ids=["gold-a"],+ doc={"title": "Navy sofa"},+ ce_gain=0.42,+ )+ self.assertEqual(len(feats), 7)+ self.assertAlmostEqual(float(feats[-1]), 0.42)+ from search.ltr_head import rank_train_group++ grouped = rank_train_group(+ ex_a, pair_policy=PAIR_OTHER_QUERY_NEG, other_gold=other+ )+ self.assertIsNotNone(grouped)+ _, labels = grouped+ self.assertEqual(len(labels), 2)++ def test_ltr_lightgbm_promotes_high_ce_gain_on_held_out_query(self):+ from search.ltr_head import (+ PAIR_OTHER_QUERY_NEG,+ example_from_eval_row,+ fit_lightgbm,+ rerank_ids_model,+ )++ n_queries = 8+ docs: dict[str, dict[str, str]] = {}+ rows: list[dict] = []+ for index in range(n_queries):+ gold = f"gold-{index}"+ other = f"gold-{(index + 1) % n_queries}"+ miss = f"miss-{index}"+ docs[gold] = {"doc_id": gold, "title": f"Navy sofa {index}"}+ docs[miss] = {"doc_id": miss, "title": f"plastic lid {index}"}+ rows.append(+ {+ "qid": f"q-{index}",+ "query": "navy sofa",+ "hit_ids": [miss, other, gold],+ "bm25_ids": [gold],+ "dense_ids": [miss],+ "hits": [+ {"id": miss, "doc_id": miss},+ {"id": other, "doc_id": other},+ {"id": gold, "doc_id": gold},+ ],+ "gold_doc_ids": [gold],+ "gold_grades": {gold: 1.0},+ "ce": {gold: 0.95, other: 0.05, miss: 0.01},+ }+ )+ examples = [+ example_from_eval_row(+ row, docs, k=10, ce_scores=row["ce"]+ )+ for row in rows+ ]+ self.assertTrue(all(item is not None for item in examples))+ model = fit_lightgbm(+ examples[:-1],+ pair_policy=PAIR_OTHER_QUERY_NEG,+ min_data_in_leaf=1,+ )+ ranked = rerank_ids_model(examples[-1], model)+ self.assertEqual(ranked[0], "gold-7")+ self.assertGreater(float(sum(model.feature_importances_)), 0.0)++ def test_ltr_apply_trains_on_other_run_not_eval_qid(self):+ from search.ltr_head import PROTOCOL_APPLY, run_ltr_head++ train_docs = [+ {"type": "doc", "doc_id": "gold-t", "title": "Navy velvet sofa", "text": "Title: Navy velvet sofa"},+ {"type": "doc", "doc_id": "irr-t", "title": "plastic lid", "text": "Title: plastic lid"},+ ]+ eval_docs = [+ {"type": "doc", "doc_id": "gold-e", "title": "Navy velvet sofa", "text": "Title: Navy velvet sofa"},+ {"type": "doc", "doc_id": "irr-e", "title": "plastic lid", "text": "Title: plastic lid"},+ ]+ train_queries = [+ {+ "type": "query",+ "qid": "q-train",+ "query": "navy velvet sofa",+ "gold_doc_ids": ["gold-t"],+ "gold_grades": {"gold-t": 1.0},+ }+ ]+ eval_queries = [+ {+ "type": "query",+ "qid": "q-eval",+ "query": "navy velvet sofa",+ "gold_doc_ids": ["gold-e"],+ "gold_grades": {"gold-e": 1.0},+ }+ ]+ train_row = {+ "qid": "q-train",+ "query": "navy velvet sofa",+ "hit_ids": ["irr-t", "gold-t"],+ "bm25_ids": ["gold-t"],+ "dense_ids": ["irr-t"],+ "hits": [+ {"id": "irr-t", "doc_id": "irr-t"},+ {"id": "gold-t", "doc_id": "gold-t"},+ ],+ "gold_doc_ids": ["gold-t"],+ "gold_grades": {"gold-t": 1.0},+ }+ eval_row = {+ "qid": "q-eval",+ "query": "navy velvet sofa",+ "hit_ids": ["irr-e", "gold-e"],+ "bm25_ids": ["gold-e"],+ "dense_ids": ["irr-e"],+ "hits": [+ {"id": "irr-e", "doc_id": "irr-e"},+ {"id": "gold-e", "doc_id": "gold-e"},+ ],+ "gold_doc_ids": ["gold-e"],+ "gold_grades": {"gold-e": 1.0},+ }+ result = run_ltr_head(+ {"run_id": "toy-eval", "queries": [eval_row]},+ eval_docs + eval_queries,+ dataset_name="toy-eval.jsonl",+ k=10,+ ks=(5, 10),+ n_folds=2,+ train_eval_result={"run_id": "toy-train", "queries": [train_row]},+ train_rows=train_docs + train_queries,+ train_source_run="toy-train",+ )+ self.assertEqual(result["protocol"], PROTOCOL_APPLY)+ self.assertEqual(result["ltr_train_run"], "toy-train")+ self.assertEqual(result["ltr_n_train_queries"], 1)+ self.assertEqual(result["queries"][0]["qid"], "q-eval")+ self.assertEqual(result["queries"][0]["hit_ids"][0], "gold-e")+ self.assertGreaterEqual(result["metrics"]["recall@10"], 1.0)++ def test_ltr_apply_lightgbm_trains_on_other_run_not_eval_qid(self):+ from search.ltr_head import HEAD_LIGHTGBM, PROTOCOL_APPLY, run_ltr_head++ train_docs: list[dict] = []+ train_queries: list[dict] = []+ train_eval_rows: list[dict] = []+ for index in range(8):+ gold = f"gold-t-{index}"+ miss = f"miss-t-{index}"+ train_docs.extend(+ [+ {+ "type": "doc",+ "doc_id": gold,+ "title": f"Navy velvet sofa {index}",+ "text": f"Title: Navy velvet sofa {index}",+ },+ {+ "type": "doc",+ "doc_id": miss,+ "title": f"plastic lid {index}",+ "text": f"Title: plastic lid {index}",+ },+ ]+ )+ qid = f"q-train-{index}"+ train_queries.append(+ {+ "type": "query",+ "qid": qid,+ "query": "navy velvet sofa",+ "gold_doc_ids": [gold],+ "gold_grades": {gold: 1.0},+ }+ )+ train_eval_rows.append(+ {+ "qid": qid,+ "query": "navy velvet sofa",+ "hit_ids": [miss, gold],+ "bm25_ids": [gold],+ "dense_ids": [miss],+ "hits": [+ {"id": miss, "doc_id": miss},+ {"id": gold, "doc_id": gold},+ ],+ "gold_doc_ids": [gold],+ "gold_grades": {gold: 1.0},+ }+ )+ eval_docs = [+ {+ "type": "doc",+ "doc_id": "gold-e",+ "title": "Navy velvet sofa",+ "text": "Title: Navy velvet sofa",+ },+ {+ "type": "doc",+ "doc_id": "irr-e",+ "title": "plastic lid",+ "text": "Title: plastic lid",+ },+ ]+ eval_queries = [+ {+ "type": "query",+ "qid": "q-eval",+ "query": "navy velvet sofa",+ "gold_doc_ids": ["gold-e"],+ "gold_grades": {"gold-e": 1.0},+ }+ ]+ eval_row = {+ "qid": "q-eval",+ "query": "navy velvet sofa",+ "hit_ids": ["irr-e", "gold-e"],+ "bm25_ids": ["gold-e"],+ "dense_ids": ["irr-e"],+ "hits": [+ {"id": "irr-e", "doc_id": "irr-e"},+ {"id": "gold-e", "doc_id": "gold-e"},+ ],+ "gold_doc_ids": ["gold-e"],+ "gold_grades": {"gold-e": 1.0},+ }+ result = run_ltr_head(+ {"run_id": "toy-eval", "queries": [eval_row]},+ eval_docs + eval_queries,+ dataset_name="toy-eval.jsonl",+ k=10,+ ks=(5, 10),+ n_folds=2,+ ltr_head=HEAD_LIGHTGBM,+ train_eval_result={"run_id": "toy-train", "queries": train_eval_rows},+ train_rows=train_docs + train_queries,+ train_source_run="toy-train",+ )+ self.assertEqual(result["protocol"], PROTOCOL_APPLY)+ self.assertEqual(result["ltr_train_run"], "toy-train")+ self.assertEqual(result["ltr_n_train_queries"], 8)+ self.assertEqual(result["rerank"], "ltr-lightgbm")+ self.assertEqual(result["queries"][0]["qid"], "q-eval")+ self.assertNotEqual(result["queries"][0]["qid"], "q-train-0")++ def test_export_hybrid_lists_holdout_unlabeled_is_i(self):+ from search.export_hybrid_lists import (+ PROTECTED_OUT_NAMES,+ export_hybrid_lists,+ rows_from_eval,+ )++ self.assertIn("esci_retrieved_lists.jsonl", PROTECTED_OUT_NAMES)+ eval_result = {+ "queries": [+ {+ "qid": "esci-hold",+ "query": "navy sofa",+ "hit_ids": ["gold-h", "miss-h"],+ "hits": [+ {"id": "gold-h", "doc_id": "gold-h"},+ {"id": "miss-h", "doc_id": "miss-h"},+ ],+ "gold_grades": {"gold-h": 1.0},+ },+ {+ "qid": "esci-ok",+ "query": "navy sofa",+ "hit_ids": ["gold-ok", "miss-ok"],+ "hits": [+ {"id": "gold-ok", "doc_id": "gold-ok"},+ {"id": "miss-ok", "doc_id": "miss-ok"},+ ],+ "gold_grades": {"gold-ok": 1.0},+ },+ ]+ }+ docs = [+ {+ "type": "doc",+ "doc_id": "gold-ok",+ "title": "Navy sofa",+ "text": "Title: Navy sofa",+ },+ {+ "type": "doc",+ "doc_id": "miss-ok",+ "title": "plastic lid",+ "text": "Title: plastic lid",+ },+ ]+ queries = [+ {+ "type": "query",+ "qid": "esci-ok",+ "query": "navy sofa",+ "gold_doc_ids": ["gold-ok"],+ "gold_grades": {"gold-ok": 1.0},+ },+ {+ "type": "query",+ "qid": "esci-hold",+ "query": "navy sofa",+ "gold_doc_ids": ["gold-h"],+ "gold_grades": {"gold-h": 1.0},+ },+ ]+ rows = rows_from_eval(+ eval_result,+ docs=docs,+ queries=queries,+ holdout={"esci-hold", "hold"},+ k=50,+ )+ qids = {row["query_id"] for row in rows}+ self.assertNotIn("esci-hold", qids)+ self.assertIn("esci-ok", qids)+ by_pid = {row["product_id"]: row["label"] for row in rows}+ self.assertEqual(by_pid["gold-ok"], "E")+ self.assertEqual(by_pid["miss-ok"], "I")+ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ dataset = root / "train.jsonl"+ holdout_path = root / "holdout.jsonl"+ out = root / "hybrid.jsonl"+ dataset.write_text(+ json.dumps(docs[0])+ + "\n"+ + json.dumps(docs[1])+ + "\n"+ + json.dumps(queries[0])+ + "\n",+ encoding="utf-8",+ )+ holdout_path.write_text(+ json.dumps(+ {+ "type": "query",+ "qid": "esci-hold",+ "query": "navy sofa",+ "gold_doc_ids": ["gold-h"],+ }+ )+ + "\n",+ encoding="utf-8",+ )+ meta = export_hybrid_lists(+ eval_result=eval_result,+ dataset_path=dataset,+ holdout_path=holdout_path,+ out_path=out,+ k=50,+ )+ self.assertEqual(meta["source"], "hybrid-k50")+ self.assertEqual(meta["n_queries"], 1)+ with self.assertRaises(ValueError):+ export_hybrid_lists(+ eval_result=eval_result,+ dataset_path=dataset,+ holdout_path=holdout_path,+ out_path=root / "esci_retrieved_lists.jsonl",+ k=50,+ )++ def test_retrieved_lists_unlabeled_is_irrelevant(self):+ from search.finetune_esci_4class import held_out_query_ids+ from search.mine_retrieved_lists import (+ is_held_out,+ labeled_hits,+ mine_from_groups,+ select_groups,+ )+ from search.pool_first_stage import Bm25Index, rank_docs, tokenize++ docs = [+ ("B001", tokenize("red ceramic mug")),+ ("B002", tokenize("blue dinner plate")),+ ("B003", tokenize("red mug gift set")),+ ]+ index = Bm25Index(docs)+ qrels = {"B001": "E", "B003": "S"}+ scores = index.scores(tokenize("red ceramic mug"))+ ranked = rank_docs(scores, ["B001", "B002", "B003"])+ hits = labeled_hits(ranked, qrels, k=3)+ by_id = dict(hits)+ self.assertEqual(by_id["B001"], "E")+ self.assertEqual(by_id["B003"], "S")+ self.assertEqual(by_id["B002"], "I")+ self.assertTrue(any(label == "I" for _, label in hits))++ groups = {+ "q1": {+ "query": "red ceramic mug",+ "qrels": {"B001": "E", "B003": "S"},+ }+ }+ passages = {+ "B001": "Title: red ceramic mug",+ "B002": "Title: blue dinner plate",+ "B003": "Title: red mug gift set",+ }+ rows = mine_from_groups(groups, passages, k=3)+ labels = {row["product_id"]: row["label"] for row in rows}+ self.assertEqual(labels["B001"], "E")+ self.assertEqual(labels["B002"], "I")++ holdout = held_out_query_ids(+ Path(__file__).resolve().parents[1]+ / "benchmarks"+ / "data"+ / "search_esci_74.jsonl"+ )+ self.assertTrue(is_held_out("72", holdout))+ selected = select_groups(+ {+ "72": {"query": "not electronics", "qrels": {"x": "E"}},+ "esci-113": {"query": "html", "qrels": {"y": "E"}},+ "900001": {"query": "ok", "qrels": {"z": "E"}},+ },+ holdout=holdout,+ max_queries=10,+ seed=1,+ )+ self.assertNotIn("72", selected)+ self.assertNotIn("esci-113", selected)+ self.assertIn("900001", selected)++ def test_local_colbert_fake_encoder_ranks_match(self):+ from search.local_colbert import PLUGIN_DIR, run_local_colbert++ import sys++ if str(PLUGIN_DIR) not in sys.path:+ sys.path.insert(0, str(PLUGIN_DIR))+ import encode as colbert_encode++ def fake_tokens(text: str):+ blob = (text or "").lower()+ if "match" in blob or blob == "q":+ return [[1.0, 0.0]]+ return [[0.0, 1.0]]++ colbert_encode.set_encoder(fake_tokens)+ try:+ rows = [+ {"type": "doc", "doc_id": "hit", "text": "match"},+ {"type": "doc", "doc_id": "miss", "text": "other"},+ {+ "type": "query",+ "qid": "q1",+ "query": "q",+ "gold_doc_ids": ["hit"],+ "gold_grades": {"hit": 1.0},+ },+ ]+ result = run_local_colbert(+ rows,+ dataset_name="toy",+ k=2,+ ks=(5, 10, 20, 50),+ brain_id="harness-local-colbert",+ )+ self.assertEqual(result["channels"], ["harness-colbert"])+ self.assertEqual(result["queries"][0]["hit_ids"][0], "hit")+ finally:+ colbert_encode.set_encoder(None)++ def test_failed_report_skips_ledger(self):+ from search.report import entry_from_report, update_reports_json++ failed = {+ "status": "failed",+ "run_id": "search-fail",+ "brain_id": "searchbenchsmoke",+ "dataset": "search_toy.jsonl",+ "n_queries": 8,+ "ndcg@10": 0.9,+ }+ self.assertIsNone(entry_from_report(failed))+ empty_ok = {+ "status": "ok",+ "run_id": "search-empty",+ "n_queries": 0,+ }+ self.assertIsNone(entry_from_report(empty_ok))++ ledger = {+ "schema_version": 2,+ "benchmarks": {+ "locomo": {"name": "LoCoMo", "leaderboard": [{"run_id": "keep-me"}]},+ "recsys": {"name": "RecSys", "leaderboard": [{"run_id": "keep-recsys"}]},+ "search": {"name": "Search (hybrid BM25 + dense)", "leaderboard": []},+ },+ }+ with tempfile.TemporaryDirectory() as tmp:+ path = Path(tmp) / "REPORTS.json"+ path.write_text(json.dumps(ledger), encoding="utf-8")+ with patch("search.report.REPORTS_PATH", path):+ update_reports_json(failed)+ after = json.loads(path.read_text(encoding="utf-8"))+ self.assertEqual(+ after["benchmarks"]["locomo"]["leaderboard"][0]["run_id"],+ "keep-me",+ )+ self.assertEqual(+ after["benchmarks"]["recsys"]["leaderboard"][0]["run_id"],+ "keep-recsys",+ )+ self.assertEqual(after["benchmarks"]["search"]["leaderboard"], [])++ ok = {+ "status": "ok",+ "run_id": "search-ok",+ "brain_id": "searchbenchsmoke",+ "dataset": "search_toy.jsonl",+ "fusion": "rrf",+ "n_queries": 8,+ "ndcg@10": 0.5,+ "recall@10": 0.6,+ "mrr": 0.4,+ "p50_retrieve_ms": 12.0,+ "p95_retrieve_ms": 30.0,+ "git_sha": "abc",+ "recorded_at": "2026-08-18T00:00:00+00:00",+ }+ with patch("search.report.REPORTS_PATH", path):+ update_reports_json(ok)+ after_ok = json.loads(path.read_text(encoding="utf-8"))+ self.assertEqual(+ after_ok["benchmarks"]["locomo"]["leaderboard"][0]["run_id"],+ "keep-me",+ )+ self.assertEqual(len(after_ok["benchmarks"]["search"]["leaderboard"]), 1)+ self.assertEqual(+ after_ok["benchmarks"]["search"]["leaderboard"][0]["run_id"],+ "search-ok",+ )++ def test_cli_refuses_memory_brain(self):+ from search.cli import main++ with self.assertRaises(SystemExit) as ctx:+ main(["--brain", "locomoconv26", "dataset-stats"])+ self.assertIn("Refusing brain_id", str(ctx.exception))+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/test_search_catalog.py b/tests/test_search_catalog.pynew file mode 100644index 0000000..882acc0--- /dev/null+++ b/tests/test_search_catalog.py@@ -0,0 +1,679 @@+import tempfile+import unittest+from pathlib import Path+import os++ENV_DEFAULTS = {+ "BRAINPAT_TOKEN": "test-token",+ "MODELS_MODE": "local",+ "EMBEDDINGS_LOCAL_MODEL": "local-model",+ "EMBEDDINGS_SMALL_MODEL": "small-model",+ "EMBEDDING_NODES_DIMENSION": "3",+ "EMBEDDING_TRIPLETS_DIMENSION": "3",+ "EMBEDDING_OBSERVATIONS_DIMENSION": "3",+ "EMBEDDING_DATA_DIMENSION": "3",+ "EMBEDDING_RELATIONSHIPS_DIMENSION": "3",+ "REDIS_HOST": "localhost",+ "REDIS_PORT": "6379",+ "NEO4J_HOST": "localhost",+ "NEO4J_PORT": "7687",+ "NEO4J_USERNAME": "neo4j",+ "NEO4J_PASSWORD": "password",+ "MILVUS_HOST": "localhost",+ "MILVUS_PORT": "19530",+ "MONGO_CONNECTION_STRING": "mongodb://localhost:27017",+ "CELERY_WORKER_CONCURRENCY": "1",+ "OLLAMA_HOST": "localhost",+ "OLLAMA_PORT": "11434",+ "OLLAMA_LLM_SMALL_MODEL": "small",+ "OLLAMA_LLM_LARGE_MODEL": "large",+}+for key, value in ENV_DEFAULTS.items():+ os.environ.setdefault(key, value)+++class SearchCatalogTests(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ import sys++ root = Path(__file__).resolve().parents[1]+ bench = root / "benchmarks"+ if str(root) not in sys.path:+ sys.path.insert(0, str(root))+ if str(bench) not in sys.path:+ sys.path.insert(0, str(bench))++ def test_format_product_text_includes_marker(self):+ from search.catalog import format_product_text+ from search.dataset import doc_marker++ text = format_product_text(+ "B07ABC",+ title="Blue kettle",+ description="1.5 liter stovetop kettle",+ extras=(("Brand", "Acme"), ("Color", "blue")),+ )+ self.assertIn(doc_marker("B07ABC"), text)+ self.assertIn("Title: Blue kettle", text)+ self.assertIn("Brand: Acme", text)++ def test_select_catalog_keeps_distractors_and_grades(self):+ from search.catalog import ESCI_GAINS, _docs_and_queries, _select_catalog++ queries, needed = _select_catalog(+ [+ {+ "query_id": "10",+ "query": "blue kettle",+ "product_id": "p-exact",+ "esci_label": "E",+ },+ {+ "query_id": "10",+ "query": "blue kettle",+ "product_id": "p-sub",+ "esci_label": "S",+ },+ {+ "query_id": "10",+ "query": "blue kettle",+ "product_id": "p-irr",+ "esci_label": "I",+ },+ ],+ max_queries=8,+ max_docs=20,+ candidates_per_query=40,+ slice_name="esci-us",+ dataset="esci",+ gains=ESCI_GAINS,+ label_key="esci_label",+ query_text_key="query",+ )+ self.assertEqual(needed, {"p-exact", "p-sub", "p-irr"})+ self.assertEqual(len(queries), 1)+ self.assertEqual(queries[0]["gold_grades"]["p-exact"], 1.0)+ self.assertEqual(queries[0]["gold_grades"]["p-sub"], 0.1)+ self.assertNotIn("p-irr", queries[0]["gold_grades"])+ self.assertEqual(queries[0]["candidate_grades"]["p-irr"], 0.0)+ self.assertEqual(+ queries[0]["candidate_doc_ids"],+ ["p-exact", "p-sub", "p-irr"],+ )+ rows = _docs_and_queries(+ queries,+ products={+ "p-exact": {"product_title": "Blue kettle", "product_description": ""},+ "p-sub": {"product_title": "Teapot", "product_description": ""},+ "p-irr": {"product_title": "Lamp", "product_description": ""},+ },+ doc_ids=needed,+ dataset="esci",+ title_key="product_title",+ description_key="product_description",+ )+ docs = [row for row in rows if row["type"] == "doc"]+ self.assertEqual({row["doc_id"] for row in docs}, needed)+ self.assertEqual(docs[0]["title"], "Blue kettle")++ def test_select_catalog_skips_holdout_qids(self):+ from search.catalog import ESCI_GAINS, _select_catalog++ queries, needed = _select_catalog(+ [+ {+ "query_id": "72",+ "query": "held out kettle",+ "product_id": "p-hold",+ "esci_label": "E",+ },+ {+ "query_id": "99",+ "query": "train sofa",+ "product_id": "p-train",+ "esci_label": "E",+ },+ ],+ max_queries=8,+ max_docs=20,+ candidates_per_query=40,+ slice_name="esci-us",+ dataset="esci",+ gains=ESCI_GAINS,+ label_key="esci_label",+ query_text_key="query",+ holdout_qids={"esci-72", "72"},+ )+ self.assertEqual([row["qid"] for row in queries], ["esci-99"])+ self.assertEqual(needed, {"p-train"})++ def test_wands_gains_select_exact_and_partial_not_irrelevant(self):+ from search.catalog import WANDS_GAINS, _select_catalog++ self.assertEqual(+ WANDS_GAINS,+ {"Exact": 1.0, "Partial": 0.5, "Irrelevant": 0.0},+ )+ queries, needed = _select_catalog(+ [+ {+ "query_id": "1",+ "query": "velvet sofa",+ "query_class": "Sofas",+ "product_id": "p-exact",+ "label": "Exact",+ },+ {+ "query_id": "1",+ "query": "velvet sofa",+ "query_class": "Sofas",+ "product_id": "p-partial",+ "label": "Partial",+ },+ {+ "query_id": "1",+ "query": "velvet sofa",+ "query_class": "Sofas",+ "product_id": "p-irr",+ "label": "Irrelevant",+ },+ ],+ max_queries=8,+ max_docs=20,+ candidates_per_query=40,+ slice_name="wands",+ dataset="wands",+ gains=WANDS_GAINS,+ label_key="label",+ query_text_key="query",+ query_slice_key="query_class",+ )+ self.assertEqual(needed, {"p-exact", "p-partial", "p-irr"})+ self.assertEqual(len(queries), 1)+ self.assertEqual(queries[0]["qid"], "wands-1")+ self.assertEqual(queries[0]["gold_grades"]["p-exact"], 1.0)+ self.assertEqual(queries[0]["gold_grades"]["p-partial"], 0.5)+ self.assertNotIn("p-irr", queries[0]["gold_grades"])+ self.assertEqual(queries[0]["gold_doc_ids"], ["p-exact", "p-partial"])+ self.assertEqual(queries[0]["candidate_grades"]["p-irr"], 0.0)++ def test_frozen_wands_jsonl_blocks_overwrite(self):+ from search.catalog import DATA_DIR, catalog_overwrite_blocked++ self.assertTrue(+ catalog_overwrite_blocked(DATA_DIR / "search_esci_74.jsonl")+ )+ wands = DATA_DIR / "search_wands.jsonl"+ self.assertTrue(wands.exists() and wands.stat().st_size > 0)+ self.assertTrue(catalog_overwrite_blocked(wands))+ self.assertFalse(+ catalog_overwrite_blocked(DATA_DIR / "search_esci.jsonl")+ )++ def test_frozen_structured_brains_exclude_wandsgraph(self):+ from search.config import FROZEN_STRUCTURED_BRAINS++ self.assertIn("searchbenchwands", FROZEN_STRUCTURED_BRAINS)+ self.assertIn("searchbenchesci74", FROZEN_STRUCTURED_BRAINS)+ self.assertNotIn("searchbenchwandsgraph", FROZEN_STRUCTURED_BRAINS)+ self.assertNotIn("searchbenchjdslice", FROZEN_STRUCTURED_BRAINS)++ def test_evaluate_refuses_frozen_structured_ingest(self):+ from search.evaluate import evaluate_search++ class _Client:+ class _Settings:+ brain_id = "searchbenchwands"++ settings = _Settings()++ with self.assertRaises(SystemExit):+ evaluate_search(_Client(), [], ingest_graph=True)+ with self.assertRaises(SystemExit):+ evaluate_search(+ _Client(),+ [],+ interactions=[{"user_id": "u", "item_id": "0"}],+ )++ def test_wands_feature_parse_and_catalog_triples(self):+ from search.mapping import (+ docs_to_triples,+ entity_uuid,+ parse_feature_string,+ )++ parsed = parse_feature_string(+ "|Color:Navy|Material:Velvet|InStock:yes|Finish:|"+ )+ self.assertEqual(parsed, [("Color", "Navy"), ("Material", "Velvet")])+ docs = [+ {+ "doc_id": f"d{i}",+ "title": f"Item {i}",+ "class": "sofas" if i == 1 else "",+ "brand": "Acme" if i == 2 else "",+ "features": "|Color:Navy|" if i == 3 else "",+ "dataset": "wands",+ }+ for i in range(1, 6)+ ]+ triples = docs_to_triples(docs)+ uuids = {row["subject"]["uuid"] for row in triples}+ self.assertEqual(uuids, {entity_uuid(f"d{i}") for i in range(1, 6)})+ self.assertTrue(all(row.get("event") is None for row in triples))+ self.assertTrue(all(row["subj_event"]["name"] == "HAS" for row in triples))+ self.assertTrue(all("event_obj" not in row or row.get("event_obj") is None for row in triples))+ objects = {(row["object"]["type"], row["object"]["name"]) for row in triples}+ self.assertIn(("CLASS", "sofas"), objects)+ self.assertIn(("ATTR", "Acme"), objects)+ self.assertIn(("ATTR", "Navy"), objects)+ self.assertNotIn("demorecsys", str(triples))++ def test_catalog_graph_sofas_modern_has_search_text(self):+ from search.mapping import doc_to_triples, parse_feature_string, split_hierarchy++ parsed = parse_feature_string("dsprimaryproductstyle : modern|color : navy")+ self.assertIn(("dsprimaryproductstyle", "modern"), parsed)+ self.assertEqual(split_hierarchy("Furniture > Living Room > Sofas"), ["Furniture", "Living Room", "Sofas"])+ triples = doc_to_triples(+ {+ "doc_id": "sofa-1",+ "title": "Velvet sofa",+ "description": "A low modern sofa for small rooms",+ "class": "sofas",+ "hierarchy": "Furniture > Living Room > Sofas",+ "features": "style:modern|color:navy",+ "price": "899",+ "dataset": "wands",+ }+ )+ objects = {(row["object"]["type"], row["object"]["name"]) for row in triples}+ self.assertIn(("CLASS", "sofas"), objects)+ self.assertIn(("ATTR", "modern"), objects)+ self.assertIn(("TYPE", "Furniture"), objects)+ subject = triples[0]["subject"]+ self.assertIn("modern sofa", subject["properties"]["search_text"].lower())+ self.assertIn("sofas", subject["properties"]["search_text"].lower())+ self.assertIn("modern", subject["properties"]["search_text"].lower())+ self.assertIn("navy", subject["properties"]["search_text"].lower())+ self.assertEqual(subject["properties"]["price"], "899")+ hub_texts = {row["object"]["properties"]["search_text"] for row in triples}+ self.assertTrue(any("modern" in text.lower() for text in hub_texts))+ self.assertNotIn(("ATTR", "899"), objects)+ self.assertTrue(all(row.get("event") is None for row in triples))+ self.assertTrue(all(row["subj_event"]["name"] == "HAS" for row in triples))+ self.assertTrue(+ all("dsprimaryproductstyle" not in str(row.get("subj_event") or {}) for row in triples)+ )++ def test_entity_search_text_prefers_catalog_blob(self):+ from search.mapping import doc_id_from_text, doc_to_triples, node_id_from_passage_text++ blob = (+ "DOCID wands-bed.\n"+ "Title: solid wood platform bed\n"+ "Class: Beds\n"+ "Features: dsprimaryproductstyle : modern|woodspecies : rubberwood"+ )+ triples = doc_to_triples(+ {+ "doc_id": "wands-bed",+ "title": "solid wood platform bed",+ "class": "Beds",+ "features": "dsprimaryproductstyle : modern|woodspecies : rubberwood",+ "text": blob,+ }+ )+ subject = triples[0]["subject"]+ self.assertEqual(subject["properties"]["search_text"], blob)+ self.assertIn("rubberwood", subject["properties"]["search_text"])+ self.assertIsNone(triples[0].get("event"))+ self.assertEqual(triples[0]["subj_event"]["name"], "HAS")+ self.assertNotIn("rubberwood", triples[0]["subj_event"]["name"])+ self.assertEqual(doc_id_from_text(blob), "wands-bed")+ self.assertEqual(node_id_from_passage_text(blob), "wands-bed")+ self.assertIsNone(node_id_from_passage_text("no marker here"))++ def test_catalog_has_is_direct_edge(self):+ from search.mapping import doc_to_triples, is_static_has_triple++ triples = doc_to_triples(+ {+ "doc_id": "0",+ "title": "solid wood platform bed",+ "class": "Beds",+ "features": "woodspecies : rubberwood",+ }+ )+ self.assertTrue(triples)+ self.assertTrue(all(is_static_has_triple(row) for row in triples))+ self.assertTrue(all(row.get("event") is None for row in triples))+ kinds = {(row["object"]["type"], row["object"]["name"]) for row in triples}+ self.assertIn(("CLASS", "Beds"), kinds)+ self.assertIn(("ATTR", "rubberwood"), kinds)++ def test_entity_backfill_refuses_frozen_brains(self):+ from search.backfill_entities import (+ apply_entity_text_backfill,+ backfill_rows_from_docs,+ refuse_entity_backfill,+ )++ self.assertEqual(refuse_entity_backfill("searchbenchwandsgraph"), "searchbenchwandsgraph")+ with self.assertRaises(SystemExit):+ refuse_entity_backfill("searchbenchwands")+ with self.assertRaises(SystemExit):+ refuse_entity_backfill("searchbenchesci74")+ rows = backfill_rows_from_docs(+ [+ {+ "doc_id": "0",+ "title": "solid wood platform bed",+ "class": "Beds",+ "features": "dsprimaryproductstyle : modern|woodspecies : rubberwood",+ "text": "DOCID 0.\nTitle: solid wood platform bed\nFeatures: woodspecies : rubberwood",+ }+ ]+ )+ self.assertEqual(rows[0]["uuid"], "0")+ self.assertIn("rubberwood", rows[0]["search_text"])+ with self.assertRaises(SystemExit):+ apply_entity_text_backfill(+ brain_id="searchbenchwands",+ rows=rows,+ graph=None,+ embeddings=None,+ vector_store=None,+ )++ def test_apply_entity_backfill_updates_entity_not_event(self):+ from types import SimpleNamespace+ from unittest.mock import MagicMock++ from search.backfill_entities import apply_entity_text_backfill+ from src.constants.embeddings import Vector++ node = SimpleNamespace(+ uuid="0",+ name="bed",+ labels=["ENTITY"],+ properties={"search_text": "bed", "v_id": "old-v"},+ )+ graph = MagicMock()+ graph.get_by_uuid.return_value = node+ embeddings = MagicMock()+ embeddings.embed_text.return_value = Vector(+ id="new", embeddings=[0.1], metadata={}+ )+ store = MagicMock()+ store.add_vectors.return_value = ["vid-1"]+ summary = apply_entity_text_backfill(+ brain_id="searchbenchwandsgraph",+ rows=[{"uuid": "0", "name": "bed", "search_text": "rubberwood modern bed"}],+ graph=graph,+ embeddings=embeddings,+ vector_store=store,+ is_item=lambda uuid, labels: "ENTITY" in (labels or []),+ )+ self.assertEqual(summary["updated"], 1)+ self.assertEqual(summary["missing"], 0)+ written = [+ call.kwargs.get("new_properties") or {}+ for call in graph.update_node.call_args_list+ ]+ self.assertTrue(any("rubberwood" in str(props.get("search_text") or "") for props in written))+ event = SimpleNamespace(+ uuid="evt-1",+ name="HAS",+ labels=["EVENT"],+ properties={},+ )+ graph.get_by_uuid.return_value = event+ skipped = apply_entity_text_backfill(+ brain_id="searchbenchwandsgraph",+ rows=[{"uuid": "evt-1", "name": "HAS", "search_text": "should not copy"}],+ graph=graph,+ embeddings=embeddings,+ vector_store=store,+ is_item=lambda uuid, labels: "ENTITY" in (labels or []),+ )+ self.assertEqual(skipped["updated"], 0)+ self.assertEqual(skipped["skipped"], 1)++ def test_wandsgraph_node_join_calls_neighbors(self):+ from search.client import TimedResult+ from search.evaluate import assert_wandsgraph_node_join++ class _Client:+ class _Settings:+ brain_id = "searchbenchwandsgraph"++ settings = _Settings()++ def get_neighbors(self, uuid, limit=5):+ self.seen = uuid+ return TimedResult(data={"count": 4}, latency_ms=1.0, status_code=200)++ client = _Client()+ out = assert_wandsgraph_node_join(+ client,+ {+ "queries": [+ {+ "qid": "q1",+ "hits": [+ {+ "id": "chunk-1",+ "doc_id": "0",+ "node_id": "0",+ "channel": "passages",+ }+ ],+ }+ ]+ },+ )+ self.assertEqual(out["node_id"], "0")+ self.assertEqual(out["status_code"], 200)+ self.assertEqual(client.seen, "0")+ skipped = assert_wandsgraph_node_join(+ type("C", (), {"settings": type("S", (), {"brain_id": "searchbenchsmoke"})()})(),+ {"queries": []},+ )+ self.assertTrue(skipped["skipped"])++ def test_interaction_triples_have_happened_at(self):+ from search.mapping import interactions_to_triples, load_interaction_rows++ rows = [+ {+ "user_id": "u01",+ "item_id": "sku-101",+ "behavior": "view",+ "timestamp": "2024-01-02T10:00:00Z",+ "category": "phones",+ "brand": "Acme",+ },+ {+ "user_id": "u01",+ "item_id": "sku-102",+ "behavior": "purchase",+ "timestamp": "2024-01-05T12:00:00Z",+ },+ {+ "user_id": "u02",+ "item_id": "sku-201",+ "behavior": "view",+ "timestamp": "2024-01-02T09:00:00Z",+ },+ ]+ triples = interactions_to_triples(rows)+ events = [+ row["event"]+ for row in triples+ if row.get("event") and row["event"]["name"] != "HAS"+ ]+ self.assertGreaterEqual(len(events), 3)+ self.assertTrue(all(event.get("happened_at") for event in events))+ self.assertTrue(all(event["type"] == "EVENT" for event in events))+ self.assertTrue(all(row["object"]["uuid"] in {"sku-101", "sku-102", "sku-201"} or row["object"]["type"] != "ENTITY" for row in triples))+ entity_ids = {+ row["object"]["uuid"]+ for row in triples+ if row["object"]["type"] == "ENTITY"+ }+ self.assertEqual(entity_ids, {"sku-101", "sku-102", "sku-201"})+ self.assertNotIn("PRODUCT", str(triples))+ self.assertNotIn("Categorized", str(triples))+ self.assertNotIn("demorecsys", str(triples))+ root = Path(__file__).resolve().parents[1]+ toy = root / "benchmarks" / "data" / "recsys_toy.jsonl"+ loaded = load_interaction_rows(toy)[:3]+ self.assertEqual(len(loaded), 3)+ self.assertEqual(loaded[0]["item_id"], "sku-101")++ def test_interaction_options_emit_prefers_not_catalog_has(self):+ from search.mapping import hub_uuid, interaction_to_triples++ triples = interaction_to_triples(+ {+ "user_id": "u01",+ "item_id": "sku-101",+ "behavior": "view",+ "timestamp": "2024-01-02T10:00:00Z",+ "color": "navy",+ "options": {"style": "70s", "color": "olive"},+ },+ seq=1,+ )+ prefers = [+ row+ for row in triples+ if (row.get("subj_event") or {}).get("name") == "PREFERS"+ ]+ self.assertEqual(len(prefers), 2)+ self.assertTrue(all(row.get("event") is None for row in prefers))+ self.assertTrue(all(row["subject"]["uuid"] == "user:u01" for row in prefers))+ style_hub = hub_uuid("attr", "70s")+ olive_hub = hub_uuid("attr", "olive")+ prefers_tips = {row["object"]["uuid"] for row in prefers}+ self.assertEqual(prefers_tips, {style_hub, olive_hub})+ has_tips = {+ row["object"]["uuid"]+ for row in triples+ if (row.get("subj_event") or {}).get("name") == "HAS"+ }+ self.assertIn(hub_uuid("attr", "navy"), has_tips)+ self.assertNotIn(style_hub, has_tips)+ self.assertNotIn(olive_hub, has_tips)++ def test_personalize_smoke_fixture_emits_prefers(self):+ from search.mapping import hub_uuid, load_interaction_rows, interactions_to_triples++ root = Path(__file__).resolve().parents[1]+ rows = load_interaction_rows(+ root / "benchmarks" / "data" / "search_personalize_smoke.jsonl"+ )+ self.assertEqual(len(rows), 4)+ triples = interactions_to_triples(rows)+ prefers = [+ row+ for row in triples+ if (row.get("subj_event") or {}).get("name") == "PREFERS"+ ]+ self.assertGreaterEqual(len(prefers), 1)+ self.assertIn(hub_uuid("attr", "70s"), {row["object"]["uuid"] for row in prefers})+ self.assertTrue(any(row["event"]["name"] == "Favorite" for row in triples if row.get("event")))++ def test_cli_unknown_catalog(self):+ from search.cli import main++ self.assertEqual(main(["download", "--name", "not-a-dataset"]), 1)++ def test_write_roundtrip(self):+ from search.dataset import dataset_stats, load_records, write_records++ rows = [+ {"type": "doc", "doc_id": "a", "text": "DOCID a. Title: kettle"},+ {+ "type": "query",+ "qid": "q1",+ "query": "kettle",+ "gold_doc_ids": ["a"],+ "gold_grades": {"a": 1.0},+ "slice": "esci-us",+ },+ ]+ with tempfile.TemporaryDirectory() as tmp:+ path = Path(tmp) / "search_esci.jsonl"+ write_records(rows, path)+ loaded = load_records(path)+ stats = dataset_stats(loaded)+ self.assertEqual(stats["n_docs"], 1)+ self.assertEqual(stats["n_queries"], 1)+ self.assertTrue(stats["graded"])++ def test_esci_locale_paths_keep_us_default(self):+ from search.catalog import ESCI_JSONL, catalog_jsonl_path, normalize_esci_locale++ self.assertEqual(catalog_jsonl_path("esci"), ESCI_JSONL)+ self.assertEqual(catalog_jsonl_path("esci", locale="us"), ESCI_JSONL)+ self.assertEqual(catalog_jsonl_path("esci", locale="US"), ESCI_JSONL)+ es_path = catalog_jsonl_path("esci", locale="es")+ jp_path = catalog_jsonl_path("esci", locale="jp")+ self.assertEqual(es_path.name, "search_esci_es.jsonl")+ self.assertEqual(jp_path.name, "search_esci_jp.jsonl")+ self.assertNotEqual(es_path, ESCI_JSONL)+ self.assertNotEqual(jp_path, ESCI_JSONL)+ self.assertEqual(normalize_esci_locale("es"), "es")++ def test_esci_locale_rejects_italian(self):+ from search.catalog import catalog_jsonl_path, normalize_esci_locale+ from search.cli import main++ with self.assertRaises(ValueError) as ctx:+ normalize_esci_locale("it")+ self.assertIn("no Italian", str(ctx.exception))+ with self.assertRaises(ValueError):+ catalog_jsonl_path("esci", locale="it")+ self.assertEqual(main(["download", "--name", "esci", "--locale", "it"]), 1)++ def test_italian_smoke_is_not_esci(self):+ from search.dataset import dataset_stats, load_records++ root = Path(__file__).resolve().parents[1]+ path = root / "benchmarks" / "data" / "search_italian_smoke.jsonl"+ rows = load_records(path)+ stats = dataset_stats(rows)+ self.assertGreaterEqual(stats["n_docs"], 3)+ self.assertGreaterEqual(stats["n_queries"], 3)+ self.assertEqual(stats["slices"], {"italian-smoke": stats["n_queries"]})+ for row in rows:+ self.assertNotEqual(row.get("dataset"), "esci")+ self.assertNotIn("esci-", str(row.get("qid") or ""))+ self.assertNotIn("esci-", str(row.get("slice") or ""))++ def test_italian_inflect_is_not_esci(self):+ from search.dataset import dataset_stats, load_records++ root = Path(__file__).resolve().parents[1]+ path = root / "benchmarks" / "data" / "search_italian_smoke_inflect.jsonl"+ rows = load_records(path)+ stats = dataset_stats(rows)+ self.assertGreaterEqual(stats["n_docs"], 3)+ self.assertGreaterEqual(stats["n_queries"], 3)+ self.assertEqual(stats["slices"], {"italian-smoke-inflect": stats["n_queries"]})+ queries = [row["query"] for row in rows if row.get("type") == "query"]+ self.assertIn("bollitori acciaio", queries)+ self.assertIn("divani velluto", queries)+ self.assertIn("caffettiere alluminio", queries)+ for row in rows:+ self.assertNotEqual(row.get("dataset"), "esci")+ self.assertNotIn("esci-", str(row.get("qid") or ""))+ self.assertNotIn("esci-", str(row.get("slice") or ""))+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/test_search_esci_4class.py b/tests/test_search_esci_4class.pynew file mode 100644index 0000000..ffdbf1f--- /dev/null+++ b/tests/test_search_esci_4class.py@@ -0,0 +1,156 @@+import unittest+from pathlib import Path+++class SearchEsci4ClassTests(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ import sys++ root = Path(__file__).resolve().parents[1]+ bench = root / "benchmarks"+ if str(bench) not in sys.path:+ sys.path.insert(0, str(bench))++ def test_weighted_scores_match_esci_gains(self):+ from search.finetune_esci_4class import weighted_scores++ exact = weighted_scores([[1.0, 0.0, 0.0, 0.0]])+ sub = weighted_scores([[0.0, 1.0, 0.0, 0.0]])+ comp = weighted_scores([[0.0, 0.0, 1.0, 0.0]])+ irr = weighted_scores([[0.0, 0.0, 0.0, 1.0]])+ self.assertEqual(exact, [1.0])+ self.assertEqual(sub, [0.1])+ self.assertEqual(comp, [0.01])+ self.assertEqual(irr, [0.0])+ mixed = weighted_scores([[0.5, 0.5, 0.0, 0.0]])+ self.assertAlmostEqual(mixed[0], 0.55)++ def test_rank_doc_ids_orders_by_weighted_score(self):+ from search.finetune_esci_4class import rank_doc_ids, weighted_scores++ doc_ids = ["irr", "exact", "sub"]+ scores = weighted_scores(+ [+ [0.05, 0.05, 0.1, 0.8],+ [0.9, 0.05, 0.05, 0.0],+ [0.1, 0.8, 0.05, 0.05],+ ]+ )+ self.assertEqual(rank_doc_ids(doc_ids, scores), ["exact", "sub", "irr"])++ def test_class_weights_upweight_rare_labels(self):+ from search.finetune_esci_4class import class_weights++ weights = class_weights([80, 10, 5, 5])+ self.assertGreater(weights[2], weights[0])+ self.assertGreater(weights[3], weights[0])++ def test_held_out_query_ids_strip_esci_prefix(self):+ import json+ import tempfile++ from search.finetune_esci_4class import held_out_query_ids++ with tempfile.TemporaryDirectory() as tmp:+ path = Path(tmp) / "slice.jsonl"+ path.write_text(+ json.dumps(+ {+ "type": "doc",+ "doc_id": "p1",+ "text": "DOCID p1.\nTitle: kettle",+ }+ )+ + "\n"+ + json.dumps(+ {+ "type": "query",+ "qid": "esci-99",+ "query": "kettle",+ "gold_doc_ids": ["p1"],+ }+ )+ + "\n",+ encoding="utf-8",+ )+ holdout = held_out_query_ids(path)+ self.assertIn("esci-99", holdout)+ self.assertIn("99", holdout)++ def test_run_4class_on_pool_keeps_irrelevant_and_ranks_exact_first(self):+ from search.rank_pool_4class import run_4class_on_pool++ def predict(pairs):+ scores = []+ for _, text in pairs:+ if "gold kettle" in text:+ scores.append([0.85, 0.1, 0.05, 0.0])+ elif "partial kettle" in text:+ scores.append([0.1, 0.8, 0.05, 0.05])+ else:+ scores.append([0.05, 0.05, 0.1, 0.8])+ return scores++ rows = [+ {"type": "doc", "doc_id": "irr", "text": "DOCID irr. Title: lamp"},+ {+ "type": "doc",+ "doc_id": "gold",+ "text": "DOCID gold. Title: gold kettle",+ },+ {+ "type": "doc",+ "doc_id": "sub",+ "text": "DOCID sub. Title: partial kettle",+ },+ {+ "type": "query",+ "qid": "esci-1",+ "query": "kettle",+ "gold_doc_ids": ["gold", "sub"],+ "gold_grades": {"gold": 1.0, "sub": 0.1},+ "candidate_doc_ids": ["irr", "gold", "sub"],+ "candidate_grades": {"irr": 0.0, "gold": 1.0, "sub": 0.1},+ "slice": "esci-us",+ },+ ]+ result = run_4class_on_pool(+ rows,+ dataset_name="toy.jsonl",+ predict=predict,+ model_name="toy-4class",+ brain_id="searchbenchesci74",+ )+ self.assertEqual(result["status"], "ok")+ self.assertEqual(result["n_queries"], 1)+ self.assertTrue(result["rank_pool"])+ ranked = result["queries"][0]["hit_ids"]+ self.assertEqual(ranked, ["gold", "sub", "irr"])+ self.assertEqual(ranked[-1], "irr")+ self.assertGreater(result["metrics"]["ndcg@20"], 0.9)++ def test_persist_tokenizer_max_length_writes_train_length(self):+ import tempfile+ from types import SimpleNamespace++ from search.finetune_esci_4class import persist_tokenizer_max_length++ class FakeTokenizer:+ def __init__(self):+ self.model_max_length = 10**18+ self.saved = None++ def save_pretrained(self, path):+ self.saved = path++ tokenizer = FakeTokenizer()+ model = SimpleNamespace(max_length=192, tokenizer=tokenizer)+ with tempfile.TemporaryDirectory() as tmp:+ persist_tokenizer_max_length(model, Path(tmp))+ self.assertEqual(tokenizer.model_max_length, 192)+ self.assertEqual(tokenizer.saved, tmp)+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/test_search_jdsearch.py b/tests/test_search_jdsearch.pynew file mode 100644index 0000000..9d22353--- /dev/null+++ b/tests/test_search_jdsearch.py@@ -0,0 +1,308 @@+import tempfile+import unittest+from pathlib import Path+from unittest.mock import MagicMock++FIXTURE_DIR = (+ Path(__file__).resolve().parents[1] / "tests" / "fixtures" / "jdsearch"+)+++class SearchJdsearchTests(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ import sys++ root = Path(__file__).resolve().parents[1]+ bench = root / "benchmarks"+ if str(root) not in sys.path:+ sys.path.insert(0, str(root))+ if str(bench) not in sys.path:+ sys.path.insert(0, str(bench))++ def _fixture_paths(self) -> tuple[Path, Path]:+ product = FIXTURE_DIR / "product_meta_data.txt"+ behavior = FIXTURE_DIR / "user_behavior_data.txt"+ self.assertTrue(product.exists(), product)+ self.assertTrue(behavior.exists(), behavior)+ return product, behavior++ def test_term_and_list_separators(self):+ from search.jdsearch import TERM_SEP, join_terms, split_list++ self.assertEqual(TERM_SEP, "\030")+ self.assertEqual(join_terms(f"111{TERM_SEP}222"), "111 222")+ self.assertEqual(split_list("100_200_999"), ["100", "200", "999"])+ self.assertEqual(split_list(""), [])++ def test_stats_histogram_and_truncated_count(self):+ from search.jdsearch import PAPER_N_PRODUCTS, collect_jdsearch_stats++ product, behavior = self._fixture_paths()+ stats = collect_jdsearch_stats(+ product_path=product,+ behavior_path=behavior,+ extract=False,+ )+ self.assertEqual(stats["n_behavior_rows"], 2)+ self.assertEqual(stats["n_products_seen"], 2)+ self.assertFalse(stats["truncated"])+ self.assertIn("1", stats["label_histogram"])+ self.assertEqual(stats["label_scheme"], "graded")+ self.assertGreater(stats["history_type_counts"].get("CLICK", 0), 0)+ self.assertGreater(stats["history_type_counts"].get("FLW", 0), 0)+ self.assertIn("999", {"999"})+ self.assertGreater(stats["n_wids_missing_meta"], 0)++ truncated = collect_jdsearch_stats(+ product_path=product,+ behavior_path=behavior,+ extract=False,+ max_behavior_rows=1,+ max_product_rows=1,+ )+ self.assertTrue(truncated["truncated"])+ self.assertEqual(truncated["n_behavior_rows"], 1)+ self.assertEqual(truncated["n_products_seen"], 1)+ self.assertNotEqual(truncated["n_products_seen"], PAPER_N_PRODUCTS)+ self.assertLess(truncated["n_products_seen"], 12_000_000)++ def test_prepare_rows_caps_target_and_gold(self):+ from search.catalog import FROZEN_JSONL_IF_EXISTS, prepare_catalog+ from search.dataset import dataset_stats, load_records, split_corpus+ from search.jdsearch import (+ JDSEARCH_NAME,+ jdsearch_interactions_path,+ prepare_jdsearch_rows,+ )++ product, behavior = self._fixture_paths()+ wands = (+ Path(__file__).resolve().parents[1]+ / "benchmarks"+ / "data"+ / "search_wands.jsonl"+ )+ wands_mtime = wands.stat().st_mtime if wands.exists() else None+ rows, interactions, stats = prepare_jdsearch_rows(+ max_queries=80,+ max_docs=2000,+ product_path=product,+ behavior_path=behavior,+ extract=False,+ )+ docs, queries = split_corpus(rows)+ self.assertLessEqual(len(queries), 80)+ self.assertLessEqual(len(docs), 2000)+ self.assertEqual(stats["label_scheme"], "graded")+ self.assertTrue(queries)+ for query in queries:+ self.assertTrue(str(query.get("target") or "").startswith("jd-u"))+ self.assertTrue(query.get("gold_doc_ids"))+ self.assertTrue(query.get("gold_grades"))+ self.assertTrue(query.get("candidate_doc_ids"))+ self.assertTrue(any(row.get("behavior") == "follow" for row in interactions))+ self.assertTrue(any(row.get("behavior") == "click" for row in interactions))+ qids = {query["qid"] for query in queries}+ self.assertTrue(qids)+ for query in queries:+ self.assertNotIn("999", query["gold_doc_ids"])+ self.assertIn("search_jdsearch.jsonl", FROZEN_JSONL_IF_EXISTS)+ with tempfile.TemporaryDirectory() as tmp:+ dest = Path(tmp) / "search_jdsearch.jsonl"+ written = prepare_catalog(+ JDSEARCH_NAME,+ out_path=dest,+ max_queries=80,+ max_docs=2000,+ )+ self.assertEqual(written, dest)+ loaded = load_records(dest)+ stats = dataset_stats(loaded)+ self.assertLessEqual(stats["n_queries"], 80)+ self.assertLessEqual(stats["n_docs"], 2000)+ _, loaded_queries = split_corpus(loaded)+ self.assertTrue(all(q.get("target") and q.get("gold_doc_ids") for q in loaded_queries))+ inter_path = jdsearch_interactions_path(dest)+ self.assertTrue(inter_path.exists())+ self.assertFalse((Path(tmp) / "search_wands.jsonl").exists())+ self.assertFalse((Path(tmp) / "search_esci_74.jsonl").exists())+ if wands_mtime is not None:+ self.assertEqual(wands.stat().st_mtime, wands_mtime)++ def test_prepare_drops_all_missing_gold(self):+ from search.jdsearch import prepare_jdsearch_rows++ product, behavior = self._fixture_paths()+ with tempfile.TemporaryDirectory() as tmp:+ extra_behavior = Path(tmp) / "user_behavior_data.txt"+ extra_behavior.write_text(+ behavior.read_text(encoding="utf-8")+ + "onlymissing\t999\t1.0\t-1\t\t\t0\n",+ encoding="utf-8",+ )+ rows, _interactions, _stats = prepare_jdsearch_rows(+ product_path=product,+ behavior_path=extra_behavior,+ extract=False,+ )+ queries = [row for row in rows if row.get("type") == "query"]+ self.assertTrue(all("999" not in (q.get("gold_doc_ids") or []) for q in queries))+ self.assertTrue(all(q.get("gold_doc_ids") for q in queries))++ def test_follow_event_not_has(self):+ import importlib.util++ from search.mapping import HAS_EVENT_NAME, interaction_to_triples++ triples = interaction_to_triples(+ [+ {+ "user_id": "jd-u0",+ "item_id": "100",+ "behavior": "follow",+ "timestamp": "2022-10-16T00:00:00Z",+ }+ ]+ )+ events = [+ row["event"]["name"]+ for row in triples+ if isinstance(row.get("event"), dict) and row["event"].get("name")+ ]+ self.assertIn("Follow", events)+ self.assertNotIn(HAS_EVENT_NAME, events)+ mapping_path = (+ Path(__file__).resolve().parents[1]+ / "plugins"+ / "features-rec"+ / "models"+ / "mapping.py"+ )+ spec = importlib.util.spec_from_file_location(+ "features_rec_mapping_jdsearch", mapping_path+ )+ self.assertIsNotNone(spec)+ self.assertIsNotNone(spec.loader)+ mod = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(mod)+ self.assertEqual(mod.normalize_behavior("FLW"), ("Follow", "TARGETED"))+ self.assertEqual(mod.behavior_weight("FLW"), 0.5)+ self.assertEqual(mod.behavior_weight("follow"), 0.5)++ def test_evaluate_sends_target_only_when_personalized(self):+ from search.client import TimedResult+ from search.evaluate import evaluate_search+ from src.services.api.constants.requests import SearchRequestBody++ fields = set(SearchRequestBody.model_fields)+ self.assertIn("target", fields)+ self.assertNotIn("user_id", fields)++ class _Client:+ class _Settings:+ brain_id = "searchbenchjdslice"++ settings = _Settings()++ def __init__(self):+ self.calls = []++ def search(self, query, **kwargs):+ self.calls.append({"query": query, **kwargs})+ return TimedResult(data={"hits": []}, latency_ms=1.0)++ def list_text_chunks(self, **kwargs):+ skip = int(kwargs.get("skip") or 0)+ if skip:+ return TimedResult(data={"data": [], "total": 1}, latency_ms=1.0)+ return TimedResult(+ data={+ "data": [{"id": "c1", "text": "DOCID 100. 111 222"}],+ "total": 1,+ },+ latency_ms=1.0,+ )++ rows = [+ {+ "type": "doc",+ "doc_id": "100",+ "text": "DOCID 100. 111 222",+ },+ {+ "type": "query",+ "qid": "jdsearch-0",+ "query": "111 222",+ "target": "jd-u0",+ "gold_doc_ids": ["100"],+ "gold_grades": {"100": 1.0},+ "candidate_doc_ids": ["100"],+ },+ ]+ client = _Client()+ evaluate_search(client, rows, skip_ingest=True, personalize=True)+ self.assertEqual(client.calls[0].get("target"), "jd-u0")+ client.calls.clear()+ evaluate_search(client, rows, skip_ingest=True, personalize=False)+ self.assertFalse(client.calls[0].get("target"))+ client.calls.clear()+ evaluate_search(+ client,+ [{**rows[1], "target": ""}, rows[0]]+ if False+ else [+ rows[0],+ {**rows[1], "target": ""},+ ],+ skip_ingest=True,+ personalize=True,+ )+ self.assertFalse(client.calls[0].get("target"))++ def test_client_search_body_omits_target_by_default(self):+ from search.client import BrainAPIClient, TimedResult+ from search.config import Settings++ captured = {}++ class _Stub(BrainAPIClient):+ def __post_init__(self):+ self._client = MagicMock()++ def _request(self, method, path, *, json=None, params=None):+ captured["json"] = json+ return TimedResult(data={"hits": []}, latency_ms=1.0)++ settings = MagicMock(spec=Settings)+ settings.brain_id = "searchbenchjdslice"+ settings.brainapi_url = "http://127.0.0.1"+ settings.brainpat_token = "token"+ settings.require_brainapi = lambda: None+ client = _Stub(settings)+ client.search("111 222")+ self.assertNotIn("target", captured["json"])+ client.search("111 222", target="jd-u0")+ self.assertEqual(captured["json"]["target"], "jd-u0")+ self.assertNotIn("user_id", captured["json"])++ def test_bm25_tokens_present_in_doc_text(self):+ from search.jdsearch import prepare_jdsearch_rows++ product, behavior = self._fixture_paths()+ rows, _interactions, _stats = prepare_jdsearch_rows(+ product_path=product,+ behavior_path=behavior,+ extract=False,+ )+ docs = [row for row in rows if row.get("type") == "doc"]+ queries = [row for row in rows if row.get("type") == "query"]+ self.assertTrue(docs)+ blob = " ".join(str(doc.get("text") or "") for doc in docs)+ self.assertIn("DOCID 100.", blob)+ nonempty = 0+ for query in queries:+ tokens = str(query.get("query") or "").split()+ if any(token and token in blob for token in tokens):+ nonempty += 1+ self.assertGreater(nonempty, 0)