Skip to content

BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.

Add hybrid search benchmark suite

Christian6006d27336a9
diff --git a/benchmarks/.env.example b/benchmarks/.env.example
index 5cbb772..313d245 100644
--- a/benchmarks/.env.example
+++ b/benchmarks/.env.example
@@ -1,6 +1,9 @@
BRAINAPI_URL=http://localhost:8000
BRAINPAT_TOKEN=
+# Search eval (default searchbenchsmoke). Must start with searchbench.
+# SEARCH_BRAIN_ID=searchbenchsmoke
+
# Dual profile: product (greedy) | sota (deepseek-v4-flash + SC + gap-fill)
# BENCH_PROFILE=product
# BENCH_PROFILE=sota
diff --git a/benchmarks/AGENTS.md b/benchmarks/AGENTS.md
index a742471..a07584a 100644
--- a/benchmarks/AGENTS.md
+++ b/benchmarks/AGENTS.md
@@ -8,8 +8,9 @@
| **LongMemEval** | `benchmarks/longmemeval/` | `./longmemeval.sh` | `benchmarks.longmemeval` |
| **BEAM** | `benchmarks/beam/` | `./beam.sh` | `benchmarks.beam` |
| **RecSys** | `benchmarks/recsys/` | `./recsys.sh` | `benchmarks.recsys` |
+| **Search** | `benchmarks/search/` | `./search.sh` | `benchmarks.search` |
-All harnesses talk to a running BrainAPI over HTTP only (no `src/` imports). RecSys uses `POST /ingest/structured` (KB write) + **train-free** `GET /retrieve/recommend` on brain **`demorecsys` only** — never LoCoMo/BEAM brains. Optional: `plugins/features-rec` for attribute prefs; `plugins/recsys-gnn` LightGCN via `--backend lightgcn`.
+All harnesses talk to a running BrainAPI over HTTP only (no `src/` imports, except Search `mapping.py` re-exporting the product catalog mapper). RecSys uses `POST /ingest/structured` (KB write) + **train-free** `GET /retrieve/recommend` on brain **`demorecsys` only** — never LoCoMo/BEAM brains. Optional: `plugins/features-rec` for attribute prefs; `plugins/recsys-gnn` LightGCN via `--backend lightgcn`. Search uses `POST /ingest/` + `POST /retrieve/search` on **`searchbench*` only** — never LoCoMo/BEAM/recsys brains, never `/retrieve/context` as the metric path.
### LoCoMo
@@ -35,6 +36,28 @@ Held-out next-item HitRate/Recall@K via **train-free graph recommend** (default)
Protocol: [`docs/research/16-recsys-eval-protocol.md`](../docs/research/16-recsys-eval-protocol.md). Upserts only `benchmarks.recsys` — never wipe or write to `beam1m1clean` / `locomoconv26*`.
+### Search
+
+Hybrid BM25 + dense ranking via `POST /retrieve/search`, plus optional graph channels (`entities` / `events` / `communities`, `expand=neighbors`). Flow: ingest docs (chunk+embed; LLM enrichment skipped by default) → optional structured catalog triples (`--ingest-graph`) → map `DOCID` markers to chunk ids → search with `profile_stages`. Gold matches chunk ids or `hit.id == doc_id`. Brain: `searchbenchsmoke` (any id must start with `searchbench`). Server must have `SEARCH_ENABLED=true`, `DATA_DB=postgresql`, `BRAIN_CREATION_ALLOWED=true`.
+
+```bash
+./search.sh smoke
+./search.sh evaluate --fusion rrf
+./search.sh download --name esci
+./search.sh download --name wands
+./search.sh evaluate --dataset data/search_esci.jsonl --brain searchbenchesci --run search-esci
+./search.sh --brain searchbenchwands evaluate --dataset data/search_wands.jsonl --run search-wands-passages-k50 --channels passages --k 50 --ks 5,10,20,50
+./search.sh --brain searchbenchwands evaluate --dataset data/search_wands.jsonl --run search-wands-passages-k50 --channels passages --k 50 --ks 5,10,20,50 --skip-ingest
+./search.sh --brain searchbenchwandsgraph evaluate --dataset data/search_wands.jsonl --run search-wandsgraph-communities-k50 --channels communities --k 50 --ks 5,10,20,50 --ingest-graph
+./search.sh evaluate --dataset data/search_esci.jsonl --brain searchbenchesci --ingest-graph --channels passages,entities,communities
+./search.sh evaluate --rerank plugin:cross-encoder
+./search.sh report --run <run_id>
+```
+
+Catalog downloads are a **subset** of Amazon ESCI (KDD Cup 2022, US Task 1 / small_version test) and WANDS (Wayfair). Default caps: 80 queries, 2000 products, 40 candidates/query. Frozen WANDS quality slice: `data/search_wands.jsonl` (66 queries / 2000 docs); download refuses overwrite. Gold is graded (ESCI E/S/C/I; WANDS Exact/Partial/Irrelevant). WANDS control is passages k=50 `rerank=none` on `searchbenchwands`, then `--skip-ingest`. Opt-in catalog graph is brain `searchbenchwandsgraph` only (`--ingest-graph`, `channels=communities`); ledger rows are an **architecture demo**, not a claim vs 0.823 / ESCI 0.500. `--ingest-graph` is refused on frozen `searchbenchwands` / `searchbenchesci74` / `searchbenchescies` / `searchbenchesciltr2`. Pass `--enrich` only if you want Scout/Architect on ingest. Use dedicated `searchbench*` brains — never `demorecsys`. Graph mapping is product `src/core/search/catalog_graph.py` (generic HAS events + CLASS/TYPE/ATTR hubs). Do not score recs HitRate as search.
+
+Metrics: Recall@{5,10,20}, nDCG@10, MRR; p50/p95 retrieve **excluding** embed RTT. No LLM judge. Protocol: [`docs/research/18-search-eval-protocol.md`](../docs/research/18-search-eval-protocol.md). Upserts only `benchmarks.search` — never wipe or write to `locomoconv*`, `beam*`, `demorecsys`.
+
### BEAM
Typical flow: `download` → `ingest` → `evaluate` → `report` via `./beam.sh` or `.venv/bin/python -m beam`.
@@ -55,6 +78,7 @@ Typical flow: `download` → `ingest` → `evaluate` → `report` via `./beam.sh
- LoCoMo harness upserts `benchmarks.locomo` via `write_report` → `update_reports_json` when report `status` is `ok`.
- BEAM harness upserts `benchmarks.beam` the same way (field: `headline_score`, continuous `[0,1]`).
- RecSys harness upserts `benchmarks.recsys` only (`hit_rate@K` / `recall@K`); it must not touch other suite keys.
+- Search harness upserts `benchmarks.search` only (`ndcg@10` / `recall@10` / `p50_retrieve_ms`); it must not touch other suite keys.
- After any completed evaluate/report that produces a successful scored run, ensure the suite entry in `REPORTS.json` reflects it.
- If you write or patch `runs/<id>/report.json` manually, update `REPORTS.json` yourself (same schema; upsert by `run_id` under the suite).
- When adding a new suite, add a `benchmarks.<suite_id>` key (`name` + `leaderboard`) and upsert the same way.
diff --git a/benchmarks/README.md b/benchmarks/README.md
index 69ea41e..de7b60f 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -6,6 +6,8 @@ Standalone HTTP-only harnesses (no `src/` imports):
| --- | --- | --- |
| [LoCoMo](https://github.com/snap-research/LoCoMo) | `locomo/` | `./locomo.sh` |
| [LongMemEval](https://github.com/xiaowu0162/LongMemEval) | `longmemeval/` | `./longmemeval.sh` |
+| RecSys | `recsys/` | `./recsys.sh` |
+| Search | `search/` | `./search.sh` |
Shared setup: `requirements.txt`, `.env` (`BRAINPAT_TOKEN`, LLM keys). Results ledger: [`REPORTS.json`](REPORTS.json). Agent notes: [`AGENTS.md`](AGENTS.md).
@@ -327,3 +329,56 @@ Do **not** ingest probing questions, rubrics, or ideal answers.
**Cost warning:** even one 100K chat is ~100+ turns; start with `--limit-turns` / `--limit`.
**1M wall time:** ~40h at concurrency 1; ~20–25h at 2 with healthy API (see `docs/research/10-beam-protocol.md`).
+
+---
+
+# Search benchmark for BrainAPI
+
+Standalone HTTP-only harness for `POST /retrieve/search` (hybrid BM25 + dense). It does **not** import `src/` and does **not** call `/retrieve/context`.
+
+Protocol: [`docs/research/18-search-eval-protocol.md`](../docs/research/18-search-eval-protocol.md).
+
+## Prerequisites
+
+- BrainAPI running with `SEARCH_ENABLED=true`, `DATA_DB=postgresql`, `BRAIN_CREATION_ALLOWED=true`
+- `BRAINPAT_TOKEN` in `benchmarks/.env`
+- Brain id starting with `searchbench` (default `searchbenchsmoke`)
+
+A 404 on `/retrieve/search` means search is disabled — fix the server env; it is not a ranking miss.
+
+## Quickstart
+
+```bash
+cd benchmarks
+./search.sh dataset-stats
+./search.sh smoke
+./search.sh evaluate --fusion rrf
+./search.sh evaluate --rerank plugin:cross-encoder
+./search.sh report --run <run_id>
+```
+
+Toy corpus: [`data/search_toy.jsonl`](data/search_toy.jsonl) (keyword + paraphrase queries; gold via `DOCID` markers). Search ingest defaults to **chunk + embed only** (`skip_enrichment`); pass `--enrich` for Scout/Architect.
+
+Product-search corpora (LLM-free ingest; dedicated `searchbench*` brains, not `demorecsys`):
+
+```bash
+./search.sh download --name esci
+./search.sh download --name wands
+./search.sh dataset-stats --dataset data/search_esci.jsonl
+./search.sh evaluate --dataset data/search_esci.jsonl --brain searchbenchesci --run search-esci
+./search.sh --brain searchbenchwands evaluate --dataset data/search_wands.jsonl --run search-wands-passages-k50 --channels passages --k 50 --ks 5,10,20,50
+./search.sh --brain searchbenchwands evaluate --dataset data/search_wands.jsonl --run search-wands-passages-k50 --channels passages --k 50 --ks 5,10,20,50 --skip-ingest
+./search.sh --brain searchbenchwandsgraph evaluate --dataset data/search_wands.jsonl --run search-wandsgraph-communities-k50 --channels communities --k 50 --ks 5,10,20,50 --ingest-graph
+```
+
+`download` writes a **subset** under `data/` (gitignored): Amazon ESCI / Shopping Queries (KDD Cup 2022, US Task 1 test) and WANDS (Wayfair). Defaults: 80 queries, 2000 products, 40 candidates per query. Frozen WANDS quality slice is `data/search_wands.jsonl` (**66** queries / **2000** docs); `--force` cannot clobber it. Raw caches live in `data/esci/` and `data/wands/`. nDCG@10 is graded (ESCI E=1 / S=0.1 / C=0.01 / I=0; WANDS Exact=1 / Partial=0.5 / Irrelevant=0). WANDS first-stage control: passages, k=50, `rerank=none`, brain `searchbenchwands`, then `--skip-ingest`. Opt-in catalog graph: new brain `searchbenchwandsgraph` only; architecture demo, do not mix with ESCI nDCG or frozen WANDS 0.823.
+
+Optional plugins (core hybrid works if they are absent; unknown `plugin:<name>` is 400):
+
+- `plugins/search-rerank` — `--rerank plugin:cross-encoder` (K≤10)
+- `plugins/search-splade` — `--channels plugin:splade` after `POST /search-splade/index`
+- `plugins/search-colbert` — `--channels plugin:colbert` after `POST /search-colbert/index`
+
+Metrics: Recall@{5,10,20}, nDCG@10, MRR; p50/p95 of retrieve **excluding** embed RTT. Successful `evaluate` upserts `benchmarks.search` in [`REPORTS.json`](REPORTS.json). Smoke does not publish scores.
+
+Never ingest into or wipe `locomoconv*`, `beam*`, or `demorecsys`.
diff --git a/benchmarks/requirements.txt b/benchmarks/requirements.txt
index 40855cb..54e2c03 100644
--- a/benchmarks/requirements.txt
+++ b/benchmarks/requirements.txt
@@ -3,5 +3,8 @@ openai>=1.40.0,<2.0.0
python-dotenv>=1.0.0,<2.0.0
rich>=13.0.0,<15.0.0
datasets>=2.19.0,<4.0.0
+huggingface_hub>=0.23.0,<1.0.0
+pyarrow>=15.0.0,<20.0.0
scipy>=1.11.0,<2.0.0
json-repair>=0.30.0,<1.0.0
+lightgbm>=4.3.0,<5.0.0
diff --git a/benchmarks/search.sh b/benchmarks/search.sh
new file mode 100755
index 0000000..1d53022
--- /dev/null
+++ b/benchmarks/search.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+PYTHON="${ROOT}/.venv/bin/python"
+PARENT="${ROOT}/../.venv/bin/python"
+if [[ ! -x "$PYTHON" ]]; then
+ echo "Missing ${PYTHON}. Create it with:"
+ echo " cd benchmarks && python3 -m venv .venv && .venv/bin/pip install -r requirements.txt"
+ exit 1
+fi
+for arg in "$@"; do
+ if [[ "$arg" == "rank-pool-ce" || "$arg" == "finetune-ce" || "$arg" == "pool-first-stage" || "$arg" == "rerank-retrieved" || "$arg" == "finetune-dense" || "$arg" == "local-dense" || "$arg" == "mine-retrieved-lists" || "$arg" == "finetune-4class" || "$arg" == "colbert-local" || "$arg" == "rank-corpus" || "$arg" == "ltr-head" || "$arg" == "backfill-entity-text" ]]; then
+ if [[ -x "$PARENT" ]]; then
+ PYTHON="$PARENT"
+ fi
+ if [[ "$arg" == "backfill-entity-text" && -f "${HOME}/.brainapi/source/.env" ]]; then
+ set -a
+ # shellcheck disable=SC1091
+ source "${HOME}/.brainapi/source/.env"
+ set +a
+ export SEARCH_ENABLED="${SEARCH_ENABLED:-true}"
+ export DATA_DB="${DATA_DB:-postgresql}"
+ export VECTOR_DB="${VECTOR_DB:-postgresql}"
+ export GRAPH_DB="${GRAPH_DB:-networkx}"
+ export ENV="${ENV:-development}"
+ fi
+ break
+ fi
+done
+export PYTHONPATH="${ROOT}${PYTHONPATH:+:$PYTHONPATH}"
+exec "$PYTHON" -m search "$@"
diff --git a/benchmarks/search/__init__.py b/benchmarks/search/__init__.py
new file mode 100644
index 0000000..8d8cd25
--- /dev/null
+++ b/benchmarks/search/__init__.py
@@ -0,0 +1 @@
+"""Search eval harness against searchbench* brains."""
diff --git a/benchmarks/search/__main__.py b/benchmarks/search/__main__.py
new file mode 100644
index 0000000..5f6e430
--- /dev/null
+++ b/benchmarks/search/__main__.py
@@ -0,0 +1,4 @@
+from search.cli import main
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/search/backfill_entities.py b/benchmarks/search/backfill_entities.py
new file mode 100644
index 0000000..3c159d5
--- /dev/null
+++ b/benchmarks/search/backfill_entities.py
@@ -0,0 +1,96 @@
+from __future__ import annotations
+
+from typing import Any, Callable
+
+from search.config import FROZEN_STRUCTURED_BRAINS, validate_brain_id
+from search.mapping import catalog_entity_backfill_rows, entity_uuid
+
+
+def refuse_entity_backfill(brain_id: str) -> str:
+ bid = validate_brain_id(brain_id)
+ if bid in FROZEN_STRUCTURED_BRAINS:
+ raise SystemExit(
+ f"Refusing ENTITY text backfill on frozen brain {bid}. "
+ "Use searchbenchwandsgraph only (never wipe frozen structured brains)."
+ )
+ return bid
+
+
+def backfill_rows_from_docs(docs: list[dict[str, Any]]) -> list[dict[str, str]]:
+ return catalog_entity_backfill_rows(docs)
+
+
+def apply_entity_text_backfill(
+ *,
+ brain_id: str,
+ rows: list[dict[str, str]],
+ graph: Any,
+ embeddings: Any,
+ vector_store: Any,
+ is_item: Callable[[str, list[str] | None], bool] | None = None,
+) -> dict[str, Any]:
+ bid = refuse_entity_backfill(brain_id)
+ if is_item is None:
+ from src.core.search.graph_channels import is_item_entity as is_item
+ from src.constants.embeddings import Vector
+ from src.core.agents.scout_agent import ScoutEntity
+ from src.core.search.catalog_graph import node_embed_text
+
+ updated = 0
+ missing = 0
+ skipped = 0
+ for row in rows:
+ uuid = str(row.get("uuid") or entity_uuid(str(row.get("doc_id") or ""))).strip()
+ blob = str(row.get("search_text") or "").strip()
+ if not uuid or not blob:
+ skipped += 1
+ continue
+ node = graph.get_by_uuid(uuid, bid)
+ if node is None:
+ missing += 1
+ continue
+ labels = list(getattr(node, "labels", None) or [])
+ if not is_item(uuid, labels):
+ skipped += 1
+ continue
+ props = dict(getattr(node, "properties", None) or {})
+ props["search_text"] = blob
+ graph.update_node(
+ uuid,
+ bid,
+ new_properties={"search_text": blob},
+ )
+ scout = ScoutEntity(
+ uuid=uuid,
+ name=str(getattr(node, "name", "") or row.get("name") or uuid),
+ type="ENTITY",
+ properties=props,
+ )
+ text, _ = node_embed_text(scout)
+ vector = embeddings.embed_text(text)
+ if not isinstance(vector, Vector):
+ skipped += 1
+ continue
+ existing = props.get("v_id")
+ if existing:
+ vector.id = str(existing)
+ vector.metadata = {
+ "labels": labels or ["ENTITY"],
+ "name": scout.name,
+ "uuid": uuid,
+ }
+ v_ids = vector_store.add_vectors([vector], store="nodes", brain_id=bid)
+ if v_ids:
+ graph.update_node(
+ uuid,
+ bid,
+ new_properties={"search_text": blob, "v_id": v_ids[0]},
+ )
+ updated += 1
+ return {
+ "brain_id": bid,
+ "n_rows": len(rows),
+ "updated": updated,
+ "missing": missing,
+ "skipped": skipped,
+ }
diff --git a/benchmarks/search/catalog.py b/benchmarks/search/catalog.py
new file mode 100644
index 0000000..43340ba
--- /dev/null
+++ b/benchmarks/search/catalog.py
@@ -0,0 +1,743 @@
+from __future__ import annotations
+
+import csv
+from collections import defaultdict
+from pathlib import Path
+from typing import Any, Iterable, Iterator
+from urllib.request import urlretrieve
+
+from search.config import DATA_DIR
+from search.dataset import DOC_MARKER_PREFIX, write_records
+
+ESCI_NAME = "esci"
+WANDS_NAME = "wands"
+JDSEARCH_NAME = "jdsearch"
+SUPPORTED_CATALOGS = (ESCI_NAME, WANDS_NAME, JDSEARCH_NAME)
+DOWNLOAD_ALL_CATALOGS = (ESCI_NAME, WANDS_NAME)
+ESCI_LOCALES = ("us", "es", "jp")
+
+ESCI_JSONL = DATA_DIR / "search_esci.jsonl"
+WANDS_JSONL = DATA_DIR / "search_wands.jsonl"
+JDSEARCH_JSONL = DATA_DIR / "search_jdsearch.jsonl"
+ESCI_CACHE_DIR = DATA_DIR / "esci"
+WANDS_CACHE_DIR = DATA_DIR / "wands"
+
+ESCI_GITHUB_BASE = (
+ "https://github.com/amazon-science/esci-data/raw/main/"
+ "shopping_queries_dataset"
+)
+ESCI_HF_REPO = "khanmu2003/amazon-shopping-queries-dataset"
+ESCI_EXAMPLES_FILE = "shopping_queries_dataset_examples.parquet"
+ESCI_PRODUCTS_FILE = "shopping_queries_dataset_products.parquet"
+
+WANDS_GITHUB_BASE = "https://raw.githubusercontent.com/wayfair/WANDS/main/dataset"
+
+ESCI_GAINS = {"E": 1.0, "S": 0.1, "C": 0.01, "I": 0.0}
+WANDS_GAINS = {"Exact": 1.0, "Partial": 0.5, "Irrelevant": 0.0}
+
+DEFAULT_MAX_QUERIES = 80
+DEFAULT_MAX_DOCS = 2000
+DEFAULT_CANDIDATES_PER_QUERY = 40
+PRODUCT_TEXT_MAX_CHARS = 4000
+FROZEN_JSONL_ALWAYS = ("search_esci_74.jsonl",)
+FROZEN_JSONL_IF_EXISTS = ("search_wands.jsonl", "search_jdsearch.jsonl")
+
+
+def normalize_esci_locale(locale: str) -> str:
+ loc = (locale or "us").strip().lower()
+ if loc not in ESCI_LOCALES:
+ raise ValueError(
+ f"Unknown ESCI locale {locale!r}. Supported: {ESCI_LOCALES}. "
+ "ESCI has no Italian (it) split."
+ )
+ return loc
+
+
+def catalog_overwrite_blocked(path: Path) -> bool:
+ resolved = path.resolve()
+ for name in FROZEN_JSONL_ALWAYS:
+ if resolved == (DATA_DIR / name).resolve():
+ return True
+ for name in FROZEN_JSONL_IF_EXISTS:
+ frozen = (DATA_DIR / name).resolve()
+ if resolved == frozen and frozen.exists() and frozen.stat().st_size > 0:
+ return True
+ return False
+
+
+def catalog_jsonl_path(name: str, *, locale: str = "us") -> Path:
+ key = name.strip().lower()
+ if key == ESCI_NAME:
+ loc = normalize_esci_locale(locale)
+ if loc == "us":
+ return ESCI_JSONL
+ return DATA_DIR / f"search_esci_{loc}.jsonl"
+ if key == WANDS_NAME:
+ return WANDS_JSONL
+ if key == JDSEARCH_NAME:
+ return JDSEARCH_JSONL
+ raise ValueError(f"Unknown catalog {name!r}. Supported: {SUPPORTED_CATALOGS}")
+
+
+def prepare_catalog(
+ name: str,
+ *,
+ out_path: Path | None = None,
+ force: bool = False,
+ max_queries: int = DEFAULT_MAX_QUERIES,
+ max_docs: int = DEFAULT_MAX_DOCS,
+ candidates_per_query: int = DEFAULT_CANDIDATES_PER_QUERY,
+ locale: str = "us",
+ split: str = "test",
+ holdout_qids: set[str] | None = None,
+) -> Path:
+ key = name.strip().lower()
+ dest = out_path or catalog_jsonl_path(key, locale=locale)
+ if dest.exists() and dest.stat().st_size > 0 and not force:
+ return dest
+ if key == ESCI_NAME:
+ rows = prepare_esci_rows(
+ max_queries=max_queries,
+ max_docs=max_docs,
+ candidates_per_query=candidates_per_query,
+ locale=locale,
+ split=split,
+ force=force,
+ holdout_qids=holdout_qids,
+ )
+ elif key == WANDS_NAME:
+ rows = prepare_wands_rows(
+ max_queries=max_queries,
+ max_docs=max_docs,
+ candidates_per_query=candidates_per_query,
+ force=force,
+ )
+ elif key == JDSEARCH_NAME:
+ from search.jdsearch import prepare_jdsearch_bundle
+
+ return prepare_jdsearch_bundle(
+ dest,
+ max_queries=max_queries,
+ max_docs=max_docs,
+ candidates_per_query=candidates_per_query,
+ force=force,
+ )
+ else:
+ raise ValueError(f"Unknown catalog {name!r}. Supported: {SUPPORTED_CATALOGS}")
+ return write_records(rows, dest)
+
+
+def prepare_esci_rows(
+ *,
+ max_queries: int = DEFAULT_MAX_QUERIES,
+ max_docs: int = DEFAULT_MAX_DOCS,
+ candidates_per_query: int = DEFAULT_CANDIDATES_PER_QUERY,
+ locale: str = "us",
+ split: str = "test",
+ force: bool = False,
+ holdout_qids: set[str] | None = None,
+) -> list[dict[str, Any]]:
+ locale = normalize_esci_locale(locale)
+ examples_path, products_path = download_esci(force=force)
+ selected, needed_ids = _select_catalog(
+ _esci_judgments(examples_path, locale=locale, split=split),
+ max_queries=max_queries,
+ max_docs=max_docs,
+ candidates_per_query=candidates_per_query,
+ slice_name=f"esci-{locale}",
+ dataset=ESCI_NAME,
+ gains=ESCI_GAINS,
+ label_key="esci_label",
+ query_text_key="query",
+ holdout_qids=holdout_qids,
+ )
+ products = _esci_products(products_path, needed_ids, locale=locale)
+ return _docs_and_queries(
+ selected,
+ products=products,
+ doc_ids=needed_ids,
+ dataset=ESCI_NAME,
+ title_key="product_title",
+ description_key="product_description",
+ extra_fields=extra_fields_from_catalog(
+ products,
+ title_key="product_title",
+ description_key="product_description",
+ ),
+ )
+
+
+def prepare_wands_rows(
+ *,
+ max_queries: int = DEFAULT_MAX_QUERIES,
+ max_docs: int = DEFAULT_MAX_DOCS,
+ candidates_per_query: int = DEFAULT_CANDIDATES_PER_QUERY,
+ force: bool = False,
+) -> list[dict[str, Any]]:
+ product_path, query_path, label_path = download_wands(force=force)
+ queries = _wands_queries(query_path)
+ selected, needed_ids = _select_catalog(
+ _wands_judgments(label_path, queries),
+ max_queries=max_queries,
+ max_docs=max_docs,
+ candidates_per_query=candidates_per_query,
+ slice_name="wands",
+ dataset=WANDS_NAME,
+ gains=WANDS_GAINS,
+ label_key="label",
+ query_text_key="query",
+ query_slice_key="query_class",
+ )
+ products = _wands_products(product_path, needed_ids)
+ return _docs_and_queries(
+ selected,
+ products=products,
+ doc_ids=needed_ids,
+ dataset=WANDS_NAME,
+ title_key="product_name",
+ description_key="product_description",
+ extra_fields=(
+ ("Class", "product_class"),
+ ("Category", "category_hierarchy"),
+ ("Hierarchy", "category_hierarchy"),
+ ("Features", "product_features"),
+ ),
+ )
+
+
+def download_esci(*, force: bool = False) -> tuple[Path, Path]:
+ ESCI_CACHE_DIR.mkdir(parents=True, exist_ok=True)
+ examples = ESCI_CACHE_DIR / ESCI_EXAMPLES_FILE
+ products = ESCI_CACHE_DIR / ESCI_PRODUCTS_FILE
+ _ensure_file(
+ examples,
+ url=f"{ESCI_GITHUB_BASE}/{ESCI_EXAMPLES_FILE}",
+ hf_filename=ESCI_EXAMPLES_FILE,
+ force=force,
+ )
+ _ensure_file(
+ products,
+ url=f"{ESCI_GITHUB_BASE}/{ESCI_PRODUCTS_FILE}",
+ hf_filename=ESCI_PRODUCTS_FILE,
+ force=force,
+ )
+ return examples, products
+
+
+def download_wands(*, force: bool = False) -> tuple[Path, Path, Path]:
+ WANDS_CACHE_DIR.mkdir(parents=True, exist_ok=True)
+ product = WANDS_CACHE_DIR / "product.csv"
+ query = WANDS_CACHE_DIR / "query.csv"
+ label = WANDS_CACHE_DIR / "label.csv"
+ for path, name in (
+ (product, "product.csv"),
+ (query, "query.csv"),
+ (label, "label.csv"),
+ ):
+ _download_url(
+ f"{WANDS_GITHUB_BASE}/{name}",
+ path,
+ force=force,
+ )
+ return product, query, label
+
+
+def _stringify_catalog_value(value: Any) -> str:
+ if value is None:
+ return ""
+ if isinstance(value, str):
+ return value.strip()
+ if isinstance(value, (list, tuple)):
+ parts = [_stringify_catalog_value(item) for item in value]
+ return " ".join(part for part in parts if part)
+ if isinstance(value, dict):
+ parts = []
+ for key, item in value.items():
+ text = _stringify_catalog_value(item)
+ if text:
+ parts.append(f"{key}: {text}")
+ return "; ".join(parts)
+ return str(value).strip()
+
+
+def extra_fields_from_catalog(
+ products: dict[str, dict[str, Any]],
+ *,
+ title_key: str,
+ description_key: str,
+ skip: tuple[str, ...] = ("product_id",),
+) -> tuple[tuple[str, str], ...]:
+ keys: set[str] = set()
+ for row in products.values():
+ keys.update(str(key) for key in row.keys())
+ extras: list[tuple[str, str]] = []
+ skipped = {title_key, description_key, *skip}
+ for key in sorted(keys):
+ if key in skipped:
+ continue
+ label = key.replace("product_", "").replace("_", " ").title()
+ extras.append((label, key))
+ return tuple(extras)
+
+
+def format_product_text(
+ doc_id: str,
+ *,
+ title: str = "",
+ description: str = "",
+ extras: Iterable[tuple[str, str]] = (),
+ max_chars: int = PRODUCT_TEXT_MAX_CHARS,
+) -> str:
+ lines = [f"{DOC_MARKER_PREFIX}{doc_id}."]
+ title = (title or "").strip()
+ if title:
+ lines.append(f"Title: {title}")
+ for label, value in extras:
+ text = (value or "").strip()
+ if text:
+ lines.append(f"{label}: {text}")
+ description = (description or "").strip()
+ if description:
+ lines.append(f"Description: {description}")
+ body = "\n".join(lines).strip()
+ if len(body) > max_chars:
+ return body[: max_chars - 1].rstrip() + "…"
+ return body
+
+
+def _holdout_ids(holdout_qids: Iterable[str] | None) -> set[str]:
+ out: set[str] = set()
+ for raw in holdout_qids or []:
+ qid = str(raw or "").strip()
+ if not qid:
+ continue
+ out.add(qid)
+ lower = qid.lower()
+ if lower.startswith("esci-"):
+ out.add(qid[5:])
+ else:
+ out.add(f"esci-{qid}")
+ return out
+
+
+def _select_catalog(
+ judgments: list[dict[str, Any]],
+ *,
+ max_queries: int,
+ max_docs: int,
+ candidates_per_query: int,
+ slice_name: str,
+ dataset: str,
+ gains: dict[str, float],
+ label_key: str,
+ query_text_key: str,
+ query_slice_key: str | None = None,
+ holdout_qids: Iterable[str] | None = None,
+) -> tuple[list[dict[str, Any]], set[str]]:
+ by_query: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ query_meta: dict[str, dict[str, Any]] = {}
+ for row in judgments:
+ qid = str(row.get("query_id") or "").strip()
+ pid = str(row.get("product_id") or "").strip()
+ if not qid or not pid:
+ continue
+ by_query[qid].append(row)
+ if qid not in query_meta:
+ query_meta[qid] = row
+
+ blocked = _holdout_ids(holdout_qids)
+ eligible = [
+ qid
+ for qid in sorted(by_query, key=_sort_key)
+ if qid not in blocked and f"esci-{qid}" not in blocked
+ ]
+ selected_qids = eligible[: max(1, max_queries)]
+ needed_ids: set[str] = set()
+ queries: list[dict[str, Any]] = []
+
+ for qid in selected_qids:
+ ranked = _rank_candidates(by_query[qid], gains, label_key)
+ if candidates_per_query > 0:
+ ranked = ranked[:candidates_per_query]
+ gold_grades: dict[str, float] = {}
+ gold_doc_ids: list[str] = []
+ candidate_doc_ids: list[str] = []
+ candidate_grades: dict[str, float] = {}
+ for item in ranked:
+ pid = str(item["product_id"])
+ label = str(item.get(label_key) or "")
+ gain = float(gains.get(label, 0.0))
+ candidate_doc_ids.append(pid)
+ candidate_grades[pid] = gain
+ if pid not in needed_ids and len(needed_ids) >= max_docs:
+ continue
+ needed_ids.add(pid)
+ if gain > 0:
+ gold_grades[pid] = gain
+ gold_doc_ids.append(pid)
+ if not gold_doc_ids:
+ continue
+ meta = query_meta[qid]
+ slice_value = slice_name
+ if query_slice_key:
+ slice_value = str(meta.get(query_slice_key) or slice_name)
+ queries.append(
+ {
+ "type": "query",
+ "qid": f"{dataset}-{qid}",
+ "query": str(meta.get(query_text_key) or "").strip(),
+ "gold_doc_ids": gold_doc_ids,
+ "gold_grades": gold_grades,
+ "candidate_doc_ids": candidate_doc_ids,
+ "candidate_grades": candidate_grades,
+ "slice": slice_value,
+ }
+ )
+ return queries, needed_ids
+
+
+def _qid_to_source_id(qid: str, dataset: str) -> str:
+ text = str(qid or "").strip()
+ prefix = f"{dataset}-"
+ if text.lower().startswith(prefix.lower()):
+ return text[len(prefix) :]
+ return text
+
+
+def attach_esci_candidate_pools(
+ rows: list[dict[str, Any]],
+ *,
+ locale: str = "us",
+ split: str = "test",
+ candidates_per_query: int = DEFAULT_CANDIDATES_PER_QUERY,
+) -> list[dict[str, Any]]:
+ from search.dataset import split_corpus
+
+ locale = normalize_esci_locale(locale)
+ docs, queries = split_corpus(rows)
+ if not queries:
+ return rows
+ wanted = {
+ _qid_to_source_id(str(query.get("qid") or ""), ESCI_NAME)
+ for query in queries
+ }
+ wanted.discard("")
+ examples_path, products_path = download_esci(force=False)
+ judgments = [
+ row
+ for row in _esci_judgments(examples_path, locale=locale, split=split)
+ if str(row.get("query_id") or "").strip() in wanted
+ ]
+ selected, needed_ids = _select_catalog(
+ judgments,
+ max_queries=max(1, len(wanted)),
+ max_docs=max(len(needed_ids_from_docs(docs)) + 4000, 1),
+ candidates_per_query=candidates_per_query,
+ slice_name=f"esci-{locale}",
+ dataset=ESCI_NAME,
+ gains=ESCI_GAINS,
+ label_key="esci_label",
+ query_text_key="query",
+ )
+ by_source = {
+ _qid_to_source_id(str(row.get("qid") or ""), ESCI_NAME): row
+ for row in selected
+ }
+ existing_ids = {str(doc.get("doc_id") or "") for doc in docs}
+ missing = needed_ids - existing_ids
+ for query in queries:
+ source = _qid_to_source_id(str(query.get("qid") or ""), ESCI_NAME)
+ attached = by_source.get(source)
+ if not attached:
+ query.setdefault("candidate_doc_ids", list(query.get("gold_doc_ids") or []))
+ query.setdefault(
+ "candidate_grades",
+ dict(query.get("gold_grades") or {}),
+ )
+ continue
+ query["candidate_doc_ids"] = list(attached.get("candidate_doc_ids") or [])
+ query["candidate_grades"] = dict(attached.get("candidate_grades") or {})
+ missing.update(set(query["candidate_doc_ids"]) - existing_ids)
+ if not missing:
+ return docs + queries
+ products = _esci_products(products_path, missing, locale=locale)
+ extra = _docs_and_queries(
+ [],
+ products=products,
+ doc_ids=missing,
+ dataset=ESCI_NAME,
+ title_key="product_title",
+ description_key="product_description",
+ extra_fields=extra_fields_from_catalog(
+ products,
+ title_key="product_title",
+ description_key="product_description",
+ ),
+ )
+ extra_docs = [row for row in extra if row.get("type") == "doc"]
+ return docs + extra_docs + queries
+
+
+def needed_ids_from_docs(docs: list[dict[str, Any]]) -> set[str]:
+ return {str(doc.get("doc_id") or "") for doc in docs if doc.get("doc_id")}
+
+
+def _docs_and_queries(
+ queries: list[dict[str, Any]],
+ *,
+ products: dict[str, dict[str, Any]],
+ doc_ids: set[str],
+ dataset: str,
+ title_key: str,
+ description_key: str,
+ extra_fields: tuple[tuple[str, str], ...] = (),
+) -> list[dict[str, Any]]:
+ docs: list[dict[str, Any]] = []
+ field_aliases = {
+ "brand": "brand",
+ "color": "color",
+ "class": "class",
+ "category": "class",
+ "hierarchy": "hierarchy",
+ "category hierarchy": "hierarchy",
+ "features": "features",
+ "locale": "locale",
+ "description": "description",
+ "price": "price",
+ "rating": "rating",
+ }
+ for pid in sorted(doc_ids, key=_sort_key):
+ catalog = products.get(pid) or {}
+ extras = [
+ (label, _stringify_catalog_value(catalog.get(field)))
+ for label, field in extra_fields
+ ]
+ title = _stringify_catalog_value(catalog.get(title_key)) or pid
+ description = _stringify_catalog_value(catalog.get(description_key))
+ row: dict[str, Any] = {
+ "type": "doc",
+ "doc_id": pid,
+ "text": format_product_text(
+ pid,
+ title=title,
+ description=description,
+ extras=extras,
+ ),
+ "title": title,
+ "description": description,
+ "ingest": "chunks",
+ "dataset": dataset,
+ }
+ for label, field in extra_fields:
+ mapped = field_aliases.get(label.strip().lower())
+ if not mapped:
+ continue
+ value = _stringify_catalog_value(catalog.get(field))
+ if value and mapped not in row:
+ row[mapped] = value
+ docs.append(row)
+ queries = [q for q in queries if q.get("query") and q.get("gold_doc_ids")]
+ queries.sort(key=lambda row: str(row["qid"]))
+ return docs + queries
+
+
+def _rank_candidates(
+ rows: list[dict[str, Any]],
+ gains: dict[str, float],
+ label_key: str,
+) -> list[dict[str, Any]]:
+ def key(row: dict[str, Any]) -> tuple[float, str]:
+ label = str(row.get(label_key) or "")
+ return (-float(gains.get(label, 0.0)), str(row.get("product_id") or ""))
+
+ return sorted(rows, key=key)
+
+
+def _esci_judgments(
+ path: Path,
+ *,
+ locale: str,
+ split: str,
+) -> list[dict[str, Any]]:
+ wanted_locale = locale.strip().lower()
+ wanted_split = split.strip().lower()
+ rows: list[dict[str, Any]] = []
+ for item in _iter_parquet(path):
+ if int(item.get("small_version") or 0) != 1:
+ continue
+ if str(item.get("product_locale") or "").strip().lower() != wanted_locale:
+ continue
+ if str(item.get("split") or "").strip().lower() != wanted_split:
+ continue
+ rows.append(
+ {
+ "query_id": item.get("query_id"),
+ "query": item.get("query"),
+ "product_id": item.get("product_id"),
+ "esci_label": item.get("esci_label"),
+ }
+ )
+ return rows
+
+
+def _esci_products(
+ path: Path,
+ needed_ids: set[str],
+ *,
+ locale: str = "us",
+) -> dict[str, dict[str, Any]]:
+ catalog: dict[str, dict[str, Any]] = {}
+ if not needed_ids:
+ return catalog
+ wanted_locale = locale.strip().lower()
+ for item in _iter_parquet(path):
+ pid = str(item.get("product_id") or "").strip()
+ if pid not in needed_ids or pid in catalog:
+ continue
+ item_locale = str(item.get("product_locale") or "").strip().lower()
+ if item_locale and item_locale != wanted_locale:
+ continue
+ catalog[pid] = item
+ if len(catalog) >= len(needed_ids):
+ break
+ return catalog
+
+
+def _wands_queries(path: Path) -> dict[str, dict[str, str]]:
+ out: dict[str, dict[str, str]] = {}
+ for row in _iter_tsv(path):
+ qid = str(row.get("query_id") or "").strip()
+ if not qid:
+ continue
+ out[qid] = {
+ "query_id": qid,
+ "query": str(row.get("query") or "").strip(),
+ "query_class": str(row.get("query_class") or "wands").strip()
+ or "wands",
+ }
+ return out
+
+
+def _wands_products(path: Path, needed_ids: set[str]) -> dict[str, dict[str, Any]]:
+ catalog: dict[str, dict[str, Any]] = {}
+ if not needed_ids:
+ return catalog
+ for row in _iter_tsv(path):
+ pid = str(row.get("product_id") or "").strip()
+ if pid not in needed_ids:
+ continue
+ catalog[pid] = row
+ if len(catalog) >= len(needed_ids):
+ break
+ return catalog
+
+
+def _wands_judgments(
+ path: Path,
+ queries: dict[str, dict[str, str]],
+) -> list[dict[str, Any]]:
+ rows: list[dict[str, Any]] = []
+ for row in _iter_tsv(path):
+ qid = str(row.get("query_id") or "").strip()
+ pid = str(row.get("product_id") or "").strip()
+ if not qid or not pid:
+ continue
+ meta = queries.get(qid) or {}
+ rows.append(
+ {
+ "query_id": qid,
+ "query": meta.get("query") or "",
+ "query_class": meta.get("query_class") or "wands",
+ "product_id": pid,
+ "label": str(row.get("label") or "").strip(),
+ }
+ )
+ return rows
+
+
+def _iter_tsv(path: Path) -> Iterator[dict[str, str]]:
+ with path.open("r", encoding="utf-8", newline="") as fh:
+ reader = csv.DictReader(fh, delimiter="\t")
+ for row in reader:
+ if not row:
+ continue
+ yield {str(k): ("" if v is None else str(v)) for k, v in row.items()}
+
+
+def _iter_parquet(path: Path) -> Iterator[dict[str, Any]]:
+ try:
+ import pyarrow.parquet as pq
+ except ImportError as exc:
+ raise SystemExit(
+ "pyarrow is required to read ESCI parquet files. "
+ "Install benchmarks/requirements.txt."
+ ) from exc
+ parquet = pq.ParquetFile(path)
+ for batch in parquet.iter_batches(batch_size=8192):
+ for row in batch.to_pylist():
+ yield row
+
+
+def _ensure_file(
+ dest: Path,
+ *,
+ url: str,
+ hf_filename: str,
+ force: bool,
+) -> Path:
+ if dest.exists() and dest.stat().st_size > 64 and not force:
+ if not _is_git_lfs_pointer(dest):
+ return dest
+ try:
+ _download_url(url, dest, force=True)
+ if dest.exists() and dest.stat().st_size > 64 and not _is_git_lfs_pointer(dest):
+ return dest
+ except Exception:
+ pass
+ return _download_hf(hf_filename, dest, force=True)
+
+
+def _download_hf(filename: str, dest: Path, *, force: bool) -> Path:
+ try:
+ from huggingface_hub import hf_hub_download
+ except ImportError as exc:
+ raise SystemExit(
+ "huggingface_hub is required to download ESCI. "
+ "Install benchmarks/requirements.txt."
+ ) from exc
+ cached = hf_hub_download(
+ repo_id=ESCI_HF_REPO,
+ filename=filename,
+ repo_type="dataset",
+ force_download=force,
+ )
+ src = Path(cached)
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ if src.resolve() != dest.resolve():
+ dest.write_bytes(src.read_bytes())
+ return dest
+
+
+def _download_url(url: str, dest: Path, *, force: bool) -> Path:
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ if dest.exists() and dest.stat().st_size > 0 and not force:
+ return dest
+ tmp = dest.with_suffix(dest.suffix + ".tmp")
+ urlretrieve(url, tmp)
+ tmp.replace(dest)
+ return dest
+
+
+def _is_git_lfs_pointer(path: Path) -> bool:
+ try:
+ head = path.read_bytes()[:80]
+ except OSError:
+ return False
+ return head.startswith(b"version https://git-lfs.github.com/spec/v1")
+
+
+def _sort_key(value: str) -> tuple[int, str | int]:
+ text = str(value)
+ if text.isdigit():
+ return (0, int(text))
+ return (1, text)
diff --git a/benchmarks/search/cli.py b/benchmarks/search/cli.py
new file mode 100644
index 0000000..8ae12d1
--- /dev/null
+++ b/benchmarks/search/cli.py
@@ -0,0 +1,1563 @@
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from rich.console import Console
+from rich.table import Table
+
+from search.catalog import (
+ DOWNLOAD_ALL_CATALOGS,
+ SUPPORTED_CATALOGS,
+ catalog_jsonl_path,
+ catalog_overwrite_blocked,
+ prepare_catalog,
+)
+from search.client import BrainAPIClient, SearchDisabledError
+from search.config import BENCHMARKS_ROOT, DATA_DIR, DEFAULT_BRAIN_ID, Settings, validate_brain_id
+from search.dataset import dataset_stats, load_records, split_corpus
+from search.evaluate import assert_wandsgraph_node_join, ensure_run_dir, evaluate_search
+from search.mapping import load_interaction_rows
+from search.report import print_report_table, write_report
+
+console = Console()
+
+
+def _parse_channels(args: argparse.Namespace) -> list[str] | None:
+ raw = getattr(args, "channels", None)
+ if not raw:
+ return None
+ items = [item.strip() for item in str(raw).split(",") if item.strip()]
+ return items or None
+
+
+def _parse_label_list(raw: str | None) -> list[str] | None:
+ if not raw:
+ return None
+ items = [item.strip() for item in str(raw).split(",") if item.strip()]
+ return items or None
+
+
+def _parse_extras(raw: str | None) -> dict[str, str] | None:
+ if not raw:
+ return None
+ try:
+ data = json.loads(raw)
+ except json.JSONDecodeError as exc:
+ raise SystemExit(f"--extras must be a JSON object: {exc}") from exc
+ if not isinstance(data, dict):
+ raise SystemExit("--extras must be a JSON object")
+ return {str(key): str(value) for key, value in data.items() if value is not None} or None
+
+
+def _resolve_dataset_path(args: argparse.Namespace, settings: Settings) -> Path:
+ if getattr(args, "dataset", None):
+ return Path(args.dataset)
+ return settings.dataset_path
+
+
+def _skip_enrichment(args: argparse.Namespace) -> bool:
+ return not bool(getattr(args, "enrich", False))
+
+
+def _eval_kwargs(args: argparse.Namespace) -> dict:
+ interactions = None
+ path = getattr(args, "interactions", None)
+ if path:
+ interactions = load_interaction_rows(Path(path))
+ return {
+ "fusion": args.fusion,
+ "fusion_alpha": getattr(args, "fusion_alpha", None),
+ "rerank": args.rerank,
+ "mode": getattr(args, "mode", None) or "default",
+ "channels": _parse_channels(args),
+ "node_labels": _parse_label_list(getattr(args, "node_labels", None)),
+ "community_labels": _parse_label_list(getattr(args, "community_labels", None)),
+ "expand": getattr(args, "expand", None) or "none",
+ "skip_enrichment": _skip_enrichment(args),
+ "ingest_graph": bool(getattr(args, "ingest_graph", False)),
+ "skip_ingest": bool(getattr(args, "skip_ingest", False)),
+ "rank_pool": bool(getattr(args, "rank_pool", False)),
+ "personalize": bool(getattr(args, "personalize", False)),
+ "interactions": interactions,
+ "extras": _parse_extras(getattr(args, "extras", None)),
+ }
+
+
+def cmd_download(args: argparse.Namespace, settings: Settings) -> int:
+ from search.config import DATA_DIR
+ from search.finetune_esci_4class import held_out_query_ids
+
+ names = [item.strip().lower() for item in str(args.name).split(",") if item.strip()]
+ if len(names) == 1 and names[0] in {"all", "*"}:
+ names = list(DOWNLOAD_ALL_CATALOGS)
+ unknown = [name for name in names if name not in SUPPORTED_CATALOGS]
+ if unknown:
+ console.print(
+ f"[red]Unknown dataset[/red] {unknown!r}. "
+ f"Supported: {', '.join(SUPPORTED_CATALOGS)}"
+ )
+ return 1
+ holdout: set[str] = set()
+ holdout_raw = str(getattr(args, "holdout_dataset", None) or "").strip()
+ if holdout_raw:
+ holdout_path = Path(holdout_raw)
+ if not holdout_path.is_absolute():
+ holdout_path = (BENCHMARKS_ROOT / holdout_path).resolve()
+ if not holdout_path.exists():
+ console.print(f"[red]Missing holdout dataset[/red] {holdout_path}")
+ return 1
+ holdout = held_out_query_ids(holdout_path)
+ console.print(f"[cyan]Holdout[/cyan] {holdout_path.name} n={len(holdout)}")
+ protected = {
+ (DATA_DIR / "search_esci.jsonl").resolve(),
+ (DATA_DIR / "search_esci_74.jsonl").resolve(),
+ }
+ timeout = 0
+ for name in names:
+ try:
+ default_out = catalog_jsonl_path(name, locale=args.locale)
+ except ValueError as exc:
+ console.print(f"[red]{exc}[/red]")
+ return 1
+ out = Path(args.out) if args.out and len(names) == 1 else default_out
+ if not out.is_absolute():
+ out = (BENCHMARKS_ROOT / out).resolve()
+ split = str(args.split or "test")
+ if getattr(args, "dry_stats", False):
+ if name != "jdsearch":
+ console.print("[red]--dry-stats is only implemented for --name jdsearch[/red]")
+ return 1
+ from search.jdsearch import collect_jdsearch_stats, print_jdsearch_stats
+
+ console.print(f"[cyan]JDsearch stats[/cyan] (no JSONL write)")
+ stats = collect_jdsearch_stats()
+ print_jdsearch_stats(stats)
+ timeout += 1
+ continue
+ if catalog_overwrite_blocked(out):
+ console.print(f"[red]Refusing to overwrite[/red] {out}")
+ return 1
+ if split == "train" and out.resolve() in protected:
+ console.print(
+ f"[red]Refusing to overwrite[/red] {out} with split=train. "
+ "Pass --out data/search_esci_ltr200.jsonl"
+ )
+ return 1
+ console.print(f"[cyan]Downloading[/cyan] {name} split={split} → {out}")
+ path = prepare_catalog(
+ name,
+ out_path=out,
+ force=args.force,
+ max_queries=args.max_queries,
+ max_docs=args.max_docs,
+ candidates_per_query=args.candidates_per_query,
+ locale=args.locale,
+ split=split,
+ holdout_qids=holdout or None,
+ )
+ rows = load_records(path)
+ stats = dataset_stats(rows)
+ console.print(f"[green]Ready[/green] {path}")
+ console.print({key: value for key, value in stats.items() if key != "doc_ids"})
+ timeout += 1
+ return 0 if timeout else 1
+
+
+def cmd_dataset_stats(args: argparse.Namespace, settings: Settings) -> int:
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ rows = load_records(path)
+ stats = dataset_stats(rows)
+ table = Table(title=f"Search dataset — {path.name}")
+ table.add_column("field")
+ table.add_column("value")
+ table.add_row("n_docs", str(stats["n_docs"]))
+ table.add_row("n_queries", str(stats["n_queries"]))
+ table.add_row("slices", json.dumps(stats["slices"]))
+ table.add_row("graded", str(stats.get("graded")))
+ console.print(table)
+ return 0
+
+
+def cmd_smoke(args: argparse.Namespace, settings: Settings) -> int:
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ rows = load_records(path)
+ docs, queries = split_corpus(rows)
+ skip_ingest = bool(getattr(args, "skip_ingest", False))
+ if skip_ingest and settings.brain_id == "searchbenchwandsgraph":
+ queries = queries[: max(1, min(args.limit_queries, len(queries)))]
+ else:
+ docs = docs[: args.limit]
+ queries = queries[: max(1, min(args.limit_queries, len(queries)))]
+ limited = [{"type": "doc", **doc} for doc in docs] + [
+ {"type": "query", **query} for query in queries
+ ]
+ console.print(
+ f"[cyan]Smoke[/cyan] brain={settings.brain_id} "
+ f"docs={len(docs)} queries={len(queries)}"
+ )
+ with BrainAPIClient(settings) as client:
+ try:
+ result = evaluate_search(
+ client,
+ limited,
+ ks=(5, 10, 20),
+ timeout_s=args.timeout,
+ k=args.k,
+ dataset_name=path.name,
+ **_eval_kwargs(args),
+ )
+ except SearchDisabledError as exc:
+ console.print(f"[red]{exc}[/red]")
+ return 1
+ if result.get("search_error"):
+ console.print(f"[red]{result['search_error']}[/red]")
+ return 1
+ if result.get("ingest", {}).get("status") not in {"completed", "partial_failed"}:
+ console.print("[red]Smoke ingest failed[/red]")
+ console.print(result.get("ingest"))
+ return 1
+ if int(result.get("n_docs_mapped") or 0) < 1:
+ console.print("[red]Smoke could not map DOCID markers to chunks[/red]")
+ return 1
+ if int(result.get("n_queries") or 0) < 1:
+ console.print("[red]Smoke search returned no scored queries[/red]")
+ return 1
+ try:
+ join = assert_wandsgraph_node_join(client, result)
+ except SystemExit as exc:
+ console.print(f"[red]{exc}[/red]")
+ return 1
+ if not join.get("skipped"):
+ console.print(
+ f"node_id={join.get('node_id')} neighbors={join.get('status_code')} "
+ f"count={join.get('neighbor_count')}"
+ )
+ first = (result.get("queries") or [{}])[0]
+ console.print(
+ f"retrieve/search({first.get('qid')}) -> {first.get('n_hits')} hits "
+ f"recall@10={((first.get('metrics') or {}).get('recall@10'))}"
+ )
+ console.print("[green]Smoke passed[/green]")
+ return 0
+
+
+def cmd_backfill_entity_text(args: argparse.Namespace, settings: Settings) -> int:
+ from search.backfill_entities import apply_entity_text_backfill, refuse_entity_backfill
+ from search.mapping import catalog_entity_backfill_rows
+
+ try:
+ brain_id = refuse_entity_backfill(settings.brain_id)
+ except SystemExit as exc:
+ console.print(f"[red]{exc}[/red]")
+ return 1
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ docs, _ = split_corpus(load_records(path))
+ limit = getattr(args, "limit", None)
+ if limit:
+ docs = docs[: int(limit)]
+ rows = catalog_entity_backfill_rows(docs)
+ console.print(
+ f"[cyan]ENTITY backfill[/cyan] brain={brain_id} docs={len(rows)}"
+ )
+ import os
+ from dotenv import load_dotenv
+
+ for env_path in (
+ Path.home() / ".brainapi" / "source" / ".env",
+ Path(__file__).resolve().parents[2] / ".env",
+ ):
+ if env_path.exists():
+ load_dotenv(env_path, override=False)
+ os.environ.setdefault("DATA_DB", "postgresql")
+ os.environ.setdefault("VECTOR_DB", "postgresql")
+ os.environ.setdefault("GRAPH_DB", "networkx")
+ os.environ.setdefault("SEARCH_ENABLED", "true")
+ os.environ.setdefault("ENV", "development")
+ from src.core.instances import (
+ embeddings_adapter,
+ graph_adapter,
+ vector_store_adapter,
+ )
+
+ summary = apply_entity_text_backfill(
+ brain_id=brain_id,
+ rows=rows,
+ graph=graph_adapter,
+ embeddings=embeddings_adapter,
+ vector_store=vector_store_adapter,
+ )
+ console.print(summary)
+ if int(summary.get("updated") or 0) < 1:
+ console.print("[red]No ENTITY nodes updated[/red]")
+ return 1
+ console.print("[green]ENTITY backfill done[/green]")
+ return 0
+
+
+def cmd_evaluate(args: argparse.Namespace, settings: Settings) -> int:
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ rows = load_records(path)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ ks = tuple(int(x) for x in args.ks.split(",") if x.strip()) or (5, 10, 20)
+ console.print(
+ f"[cyan]Evaluating[/cyan] brain={settings.brain_id} "
+ f"dataset={path.name} run={run_id} fusion={args.fusion or 'rrf'}"
+ )
+ with BrainAPIClient(settings) as client:
+ try:
+ result = evaluate_search(
+ client,
+ rows,
+ ks=ks,
+ timeout_s=args.timeout,
+ k=args.k,
+ dataset_name=path.name,
+ limit_docs=getattr(args, "limit_docs", None),
+ limit_queries=getattr(args, "limit_queries", None),
+ **_eval_kwargs(args),
+ )
+ except SearchDisabledError as exc:
+ console.print(f"[red]{exc}[/red]")
+ return 1
+ report = write_report(run_dir, result)
+ print_report_table(report)
+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")
+ if result.get("search_error"):
+ console.print(f"[red]{result['search_error']}[/red]")
+ return 0 if report.get("status") == "ok" else 1
+
+
+def cmd_report(args: argparse.Namespace, settings: Settings) -> int:
+ run_dir = settings.runs_dir / args.run
+ eval_path = run_dir / "eval.json"
+ if not eval_path.exists():
+ console.print(f"[red]Missing[/red] {eval_path}")
+ return 1
+ eval_result = json.loads(eval_path.read_text(encoding="utf-8"))
+ report = write_report(run_dir, eval_result)
+ print_report_table(report)
+ return 0 if report.get("status") == "ok" else 1
+
+
+def cmd_finetune_ce(args: argparse.Namespace, settings: Settings) -> int:
+ from search.finetune_esci_ce import finetune
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ out = Path(args.out)
+ console.print(
+ f"[cyan]Fine-tune CE[/cyan] dataset={path.name} out={out} "
+ f"label_mode={args.label_mode} fields={args.fields}"
+ )
+ meta = finetune(
+ jsonl_path=path,
+ out_dir=out,
+ max_pairs=int(args.max_pairs),
+ epochs=int(args.epochs),
+ batch_size=int(args.batch_size),
+ label_mode=str(args.label_mode),
+ fields=str(args.fields),
+ base=str(args.base),
+ lr=float(args.lr),
+ max_length=int(getattr(args, "max_length", 256)),
+ )
+ console.print(json.dumps(meta, indent=2))
+ return 0
+
+
+def cmd_miss_strata(args: argparse.Namespace, settings: Settings) -> int:
+ from search.miss_strata import main_classify
+
+ path = _resolve_dataset_path(args, settings)
+ eval_path = settings.runs_dir / args.from_run / "eval.json"
+ if not eval_path.exists():
+ console.print(f"[red]Missing[/red] {eval_path}")
+ return 1
+ out = Path(args.out) if args.out else settings.runs_dir / args.from_run / "miss_strata.json"
+ taxonomy = main_classify(eval_path=eval_path, dataset_path=path, out_path=out)
+ table = Table(title=f"Miss strata {args.from_run}")
+ table.add_column("field")
+ table.add_column("value")
+ for key in (
+ "n_queries",
+ "n_gold",
+ "n_in_top10",
+ "n_in_11_to_k",
+ "n_missed",
+ "stratum_counts",
+ "n_total_miss",
+ "n_rewritable_total_miss",
+ "run_query_side",
+ "rewritable_qids",
+ "total_miss_qids",
+ ):
+ table.add_row(key, str(taxonomy.get(key)))
+ console.print(table)
+ console.print(f"[green]Wrote[/green] {out}")
+ return 0
+
+
+def cmd_query_rewrite(args: argparse.Namespace, settings: Settings) -> int:
+ from search.miss_strata import main_classify, write_rewritten_jsonl
+
+ path = _resolve_dataset_path(args, settings)
+ eval_path = settings.runs_dir / args.from_run / "eval.json"
+ if not eval_path.exists():
+ console.print(f"[red]Missing[/red] {eval_path}")
+ return 1
+ taxonomy = main_classify(eval_path=eval_path, dataset_path=path)
+ if not taxonomy.get("run_query_side"):
+ console.print(
+ "[yellow]No rewritable total-miss qids (skip Task 2).[/yellow] "
+ f"total_miss={taxonomy.get('total_miss_qids')}"
+ )
+ return 0
+ dest = Path(args.out) if args.out else path.with_name("search_esci_74_qrewrite.jsonl")
+ rows = load_records(path)
+ write_rewritten_jsonl(rows, taxonomy, dest)
+ console.print(
+ f"[green]Wrote[/green] {dest} qids={taxonomy.get('rewritable_qids')}"
+ )
+ return 0
+
+
+def cmd_spell_normalize(args: argparse.Namespace, settings: Settings) -> int:
+ from search.miss_strata import write_spell_jsonl
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ dest = Path(args.out) if args.out else path.with_name("search_esci_es_spell.jsonl")
+ rows = load_records(path)
+ summary = write_spell_jsonl(rows, dest)
+ console.print(
+ f"[green]Wrote[/green] {dest} changed_queries={summary.get('n_queries_changed')}"
+ )
+ return 0
+
+
+def cmd_finetune_dense(args: argparse.Namespace, settings: Settings) -> int:
+ from search.finetune_esci_dense import finetune_dense
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ eval_path = Path(args.from_eval) if args.from_eval else None
+ out = Path(args.out)
+ console.print(
+ f"[cyan]Fine-tune dense ANCE[/cyan] dataset={path.name} out={out} "
+ f"(not Reddy 0.857)"
+ )
+ meta = finetune_dense(
+ jsonl_path=path,
+ out_dir=out,
+ eval_path=eval_path,
+ max_pairs=int(args.max_pairs),
+ epochs=int(args.epochs),
+ batch_size=int(args.batch_size),
+ fields=str(args.fields),
+ base=str(args.base),
+ lr=float(args.lr),
+ max_length=int(args.max_length),
+ )
+ console.print(json.dumps(meta, indent=2))
+ return 0
+
+
+def cmd_local_dense(args: argparse.Namespace, settings: Settings) -> int:
+ from search.local_dense import run_local_dense
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ rows = load_records(path)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)
+ model_name = str(args.model)
+ protocol = None
+ if "bge" in model_name.lower():
+ protocol = (
+ "local BGE-base dual-encoder over search_esci_74.jsonl; "
+ "not MiniLM ANCE; no query: prefix; not Reddy 0.857"
+ )
+ if "paraphrase-multilingual-minilm" in model_name.lower() or "italian_smoke" in path.name:
+ protocol = (
+ "local paraphrase-multilingual-MiniLM-L12-v2 over "
+ "search_italian_smoke.jsonl; pipeline only (hits returned); "
+ "not text-embedding-3-large quality; not Reddy; not ES n=62"
+ )
+ console.print(
+ f"[cyan]Local dense retrieve[/cyan] dataset={path.name} run={run_id} "
+ f"model={model_name} brain={settings.brain_id}"
+ )
+ result = run_local_dense(
+ rows,
+ model_name=str(args.model),
+ dataset_name=path.name,
+ k=int(args.k),
+ ks=ks,
+ brain_id=settings.brain_id,
+ protocol=protocol,
+ )
+ report = write_report(run_dir, result)
+ print_report_table(report)
+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")
+ return 0 if report.get("status") == "ok" else 1
+
+
+def cmd_mine_retrieved_lists(args: argparse.Namespace, settings: Settings) -> int:
+ from search.mine_retrieved_lists import mine
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ out = Path(args.out)
+ console.print(
+ f"[cyan]Mine retrieved lists[/cyan] dataset={path.name} out={out} "
+ f"(unlabeled=I; hold out test qids)"
+ )
+ meta = mine(
+ jsonl_path=path,
+ out_path=out,
+ max_queries=int(args.max_queries),
+ k=int(args.k),
+ seed=int(args.seed),
+ )
+ console.print(json.dumps(meta, indent=2))
+ return 0
+
+
+def cmd_export_hybrid_lists(args: argparse.Namespace, settings: Settings) -> int:
+ from search.export_hybrid_lists import PROTECTED_OUT_NAMES, export_hybrid_lists
+ from search.list_overlap import load_eval_run
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ holdout_raw = str(getattr(args, "holdout_dataset", None) or "").strip()
+ holdout = Path(holdout_raw) if holdout_raw else DATA_DIR / "search_esci_74.jsonl"
+ if not holdout.is_absolute():
+ holdout = (BENCHMARKS_ROOT / holdout).resolve()
+ if not holdout.exists():
+ console.print(f"[red]Missing holdout[/red] {holdout}")
+ return 1
+ out = Path(args.out)
+ if not out.is_absolute():
+ out = (BENCHMARKS_ROOT / out).resolve()
+ if out.name in PROTECTED_OUT_NAMES:
+ console.print(f"[red]Refusing to overwrite[/red] {out}")
+ return 1
+ run_dir = settings.runs_dir / str(args.from_run)
+ if not (run_dir / "eval.json").exists():
+ console.print(f"[red]Missing eval[/red] {run_dir / 'eval.json'}")
+ return 1
+ eval_result = load_eval_run(run_dir)
+ console.print(
+ f"[cyan]Export hybrid lists[/cyan] from={args.from_run} "
+ f"dataset={path.name} holdout={holdout.name} out={out} "
+ f"(unlabeled=I; source=hybrid-k50)"
+ )
+ try:
+ meta = export_hybrid_lists(
+ eval_result=eval_result,
+ dataset_path=path,
+ holdout_path=holdout,
+ out_path=out,
+ k=int(args.k),
+ )
+ except ValueError as exc:
+ console.print(f"[red]{exc}[/red]")
+ return 1
+ console.print(json.dumps(meta, indent=2))
+ return 0
+
+
+def cmd_finetune_4class(args: argparse.Namespace, settings: Settings) -> int:
+ from search.finetune_esci_4class import finetune
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ lists_path = Path(args.from_lists) if args.from_lists else None
+ if lists_path is not None and not lists_path.exists():
+ console.print(f"[red]Missing[/red] {lists_path}")
+ return 1
+ out = Path(args.out)
+ if not out.is_absolute():
+ out = (BENCHMARKS_ROOT / out).resolve()
+ base_model = str(args.base)
+ base_path = Path(base_model)
+ if not base_path.is_absolute():
+ maybe_base = (BENCHMARKS_ROOT / base_path).resolve()
+ if maybe_base.exists():
+ base_path = maybe_base
+ base_model = str(base_path)
+ if out.exists() and base_path.exists() and out.resolve() == base_path.resolve():
+ console.print(f"[red]Refusing to overwrite base checkpoint[/red] {out}")
+ return 1
+ lists_source = str(getattr(args, "lists_source", None) or "").strip()
+ if lists_path is not None:
+ source = lists_source or "retrieved-bm25"
+ else:
+ source = "pool"
+ console.print(
+ f"[cyan]Fine-tune 4-class CE[/cyan] dataset={path.name} out={out} "
+ f"source={source}"
+ )
+ meta = finetune(
+ jsonl_path=path,
+ out_dir=out,
+ max_pairs=int(args.max_pairs),
+ epochs=int(args.epochs),
+ batch_size=int(args.batch_size),
+ max_length=int(args.max_length),
+ base_model=base_model,
+ seed=int(args.seed),
+ ckpt_every=int(args.ckpt_every),
+ use_class_weights=bool(args.class_weights),
+ lists_path=lists_path,
+ lists_source=source if lists_path is not None else None,
+ )
+ console.print(json.dumps(meta, indent=2))
+ return 0
+
+
+def cmd_colbert_local(args: argparse.Namespace, settings: Settings) -> int:
+ from search.local_colbert import run_local_colbert
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ rows = load_records(path)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)
+ console.print(
+ f"[cyan]Local ColBERT MaxSim[/cyan] dataset={path.name} run={run_id} "
+ f"(not fused with passages; not Reddy 0.857)"
+ )
+ result = run_local_colbert(
+ rows,
+ dataset_name=path.name,
+ k=int(args.k),
+ ks=ks,
+ brain_id="harness-local-colbert",
+ )
+ report = write_report(run_dir, result)
+ print_report_table(report)
+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")
+ return 0 if report.get("status") == "ok" else 1
+
+
+def _existing_path(path: Path) -> Path:
+ if path.exists():
+ return path
+ from search.config import BENCHMARKS_ROOT
+
+ alt = BENCHMARKS_ROOT / path
+ return alt if alt.exists() else path
+
+
+def cmd_rank_corpus(args: argparse.Namespace, settings: Settings) -> int:
+ from search.rank_corpus import DEFAULT_MODEL, load_predict, run_exhaustive_ce
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ model_dir = _existing_path(Path(args.model) if args.model else DEFAULT_MODEL)
+ if not model_dir.exists():
+ console.print(f"[red]Missing model[/red] {model_dir}")
+ return 1
+ rows = load_records(path)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)
+ console.print(
+ f"[cyan]Exhaustive 4-class CE[/cyan] dataset={path.name} run={run_id} "
+ f"model={model_dir} (protocol exhaustive-catalog; not Reddy 0.857)"
+ )
+ predict, model_name = load_predict(model_dir, max_length=int(args.max_length))
+ result = run_exhaustive_ce(
+ rows,
+ predict=predict,
+ model_name=model_name,
+ dataset_name=path.name,
+ k=int(args.k),
+ ks=ks,
+ brain_id="harness-local-exhaustive",
+ )
+ report = write_report(run_dir, result)
+ print_report_table(report)
+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")
+ return 0 if report.get("status") == "ok" else 1
+
+
+def cmd_list_overlap(args: argparse.Namespace, settings: Settings) -> int:
+ from search.list_overlap import load_eval_run, summarize_overlap
+
+ names = [item.strip() for item in str(args.against_runs).split(",") if item.strip()]
+ passages = load_eval_run(settings.runs_dir / args.passages_run)
+ passages["run_id"] = args.passages_run
+ sidecars: dict = {}
+ for name in names:
+ eval_result = load_eval_run(settings.runs_dir / name)
+ eval_result["run_id"] = name
+ sidecars[name] = eval_result
+ queries = None
+ dataset = getattr(args, "dataset", None)
+ if dataset:
+ path = Path(dataset)
+ if not path.exists():
+ path = _existing_path(path)
+ if path.exists():
+ _, queries = split_corpus(load_records(path))
+ summary = summarize_overlap(
+ passages,
+ sidecars,
+ k=int(args.k),
+ queries=queries,
+ )
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ out = run_dir / "overlap.json"
+ out.write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
+ headline = {
+ name: {
+ "unique_gold_hits": row["unique_gold_hits"],
+ "queries_with_unique": row["queries_with_unique"],
+ "n_queries": row["n_queries"],
+ }
+ for name, row in summary["runs"].items()
+ }
+ console.print(json.dumps(headline, indent=2))
+ console.print(f"[green]Wrote[/green] {out}")
+ return 0
+
+
+def cmd_union_lists(args: argparse.Namespace, settings: Settings) -> int:
+ from search.list_overlap import load_eval_run, run_union
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ names = [item.strip() for item in str(args.from_runs).split(",") if item.strip()]
+ evals = []
+ for name in names:
+ eval_result = load_eval_run(settings.runs_dir / name)
+ eval_result["run_id"] = name
+ evals.append(eval_result)
+ rows = load_records(path)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)
+ console.print(
+ f"[cyan]Harness RRF union[/cyan] from={names} dataset={path.name} "
+ f"run={run_id} (not live graph; not Reddy 0.857)"
+ )
+ result = run_union(
+ evals,
+ rows,
+ dataset_name=path.name,
+ k=int(args.k),
+ ks=ks,
+ brain_id="harness-local-union",
+ run_names=names,
+ )
+ report = write_report(run_dir, result)
+ print_report_table(report)
+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")
+ return 0 if report.get("status") == "ok" else 1
+
+
+def cmd_cascade_lists(args: argparse.Namespace, settings: Settings) -> int:
+ from search.list_overlap import load_eval_run, run_cascade
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ passages = load_eval_run(settings.runs_dir / args.passages_run)
+ passages["run_id"] = args.passages_run
+ names = [item.strip() for item in str(args.from_runs).split(",") if item.strip()]
+ sidecars = []
+ for name in names:
+ eval_result = load_eval_run(settings.runs_dir / name)
+ eval_result["run_id"] = name
+ sidecars.append(eval_result)
+ rows = load_records(path)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)
+ console.print(
+ f"[cyan]Frozen-head cascade[/cyan] passages={args.passages_run} "
+ f"sidecars={names} head_k={args.head_k} dataset={path.name} "
+ f"run={run_id} (not live graph; not Reddy 0.857)"
+ )
+ result = run_cascade(
+ passages,
+ sidecars,
+ rows,
+ dataset_name=path.name,
+ k=int(args.k),
+ head_k=int(args.head_k),
+ ks=ks,
+ brain_id="harness-local-cascade",
+ run_names=[args.passages_run, *names],
+ )
+ report = write_report(run_dir, result)
+ print_report_table(report)
+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")
+ return 0 if report.get("status") == "ok" else 1
+
+
+def cmd_ltr_head(args: argparse.Namespace, settings: Settings) -> int:
+ from search.list_overlap import load_eval_run
+ from search.ltr_head import HEAD_RANKNET, PAIR_UNLABELED_ZERO, ce_cache_name, run_ltr_head
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ if not path.is_absolute():
+ alt = (BENCHMARKS_ROOT / path).resolve()
+ if alt.exists():
+ path = alt
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ passages = load_eval_run(settings.runs_dir / args.from_run)
+ passages["run_id"] = args.from_run
+ rows = load_records(path)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip()) or (5, 10, 20, 50)
+ policy = str(getattr(args, "pair_policy", None) or PAIR_UNLABELED_ZERO)
+ head = str(getattr(args, "ltr_model", None) or HEAD_RANKNET)
+ ce_raw = str(getattr(args, "ce_model", None) or "").strip()
+ ce_model = Path(ce_raw) if ce_raw else None
+ if ce_model is not None and not ce_model.is_absolute():
+ ce_model = (BENCHMARKS_ROOT / ce_model).resolve()
+ if ce_model is not None and not ce_model.exists():
+ console.print(f"[red]Missing CE model[/red] {ce_model}")
+ return 1
+ ce_cache = None
+ if ce_model is not None:
+ ce_cache = settings.runs_dir / args.from_run / ce_cache_name(ce_model)
+ train_run = str(getattr(args, "train_from_run", None) or "").strip()
+ train_eval = None
+ train_rows = None
+ train_ce_cache = None
+ train_dataset_name = None
+ if train_run:
+ train_ds = str(getattr(args, "train_dataset", None) or "").strip()
+ if not train_ds:
+ console.print("[red]--train-dataset is required with --train-from-run[/red]")
+ return 1
+ train_path = Path(train_ds)
+ if not train_path.is_absolute():
+ train_path = (BENCHMARKS_ROOT / train_path).resolve()
+ if not train_path.exists():
+ console.print(f"[red]Missing train dataset[/red] {train_path}")
+ return 1
+ train_eval = load_eval_run(settings.runs_dir / train_run)
+ train_eval["run_id"] = train_run
+ train_rows = load_records(train_path)
+ train_dataset_name = train_path.name
+ if ce_model is not None:
+ train_ce_cache = settings.runs_dir / train_run / ce_cache_name(ce_model)
+ mode = "apply" if train_run else "CV"
+ console.print(
+ f"[cyan]LTR head {mode}[/cyan] from={args.from_run} dataset={path.name} "
+ f"train={train_run or 'folds'} {train_dataset_name or ''} "
+ f"run={run_id} folds={args.folds} pair={policy} head={head} "
+ f"ce={ce_model or 'none'} (not live; not Reddy 0.857)"
+ )
+ result = run_ltr_head(
+ passages,
+ rows,
+ dataset_name=path.name,
+ k=int(args.k),
+ ks=ks,
+ n_folds=int(args.folds),
+ brain_id="harness-local-ltr",
+ source_run=args.from_run,
+ pair_policy=policy,
+ ce_model=ce_model,
+ ce_cache_path=ce_cache,
+ ltr_head=head,
+ train_eval_result=train_eval,
+ train_rows=train_rows,
+ train_source_run=train_run or None,
+ train_ce_cache_path=train_ce_cache,
+ )
+ report = write_report(run_dir, result)
+ print_report_table(report)
+ overlap = result.get("overlap_only_metrics") or {}
+ console.print(
+ f"overlap-only ndcg@10={overlap.get('ndcg@10')} "
+ f"recall@10={overlap.get('recall@10')}"
+ )
+ weights = result.get("ltr_mean_weights") or {}
+ if weights:
+ kind = "gain importances" if head == "lightgbm" else "weights"
+ console.print(f"mean CV {kind} {weights}")
+ if result.get("ltr_train_run"):
+ console.print(
+ f"trained on {result.get('ltr_n_train_queries')} queries "
+ f"from {result.get('ltr_train_run')}"
+ )
+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")
+ return 0 if report.get("status") == "ok" else 1
+
+
+def cmd_pool_first_stage(args: argparse.Namespace, settings: Settings) -> int:
+ from search.pool_first_stage import main as pool_main
+
+ argv = ["--dataset", str(_resolve_dataset_path(args, settings))]
+ if args.run:
+ argv.extend(["--run", str(args.run)])
+ if args.variant:
+ argv.extend(["--variant", str(args.variant)])
+ if args.expand_out:
+ argv.extend(["--expand-out", str(args.expand_out)])
+ return int(pool_main(argv))
+
+
+def cmd_rerank_retrieved(args: argparse.Namespace, settings: Settings) -> int:
+ from search.rerank_retrieved import run_ce_on_retrieved
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ source_dir = settings.runs_dir / args.from_run
+ eval_path = source_dir / "eval.json"
+ if not eval_path.exists():
+ console.print(f"[red]Missing[/red] {eval_path}")
+ return 1
+ eval_result = json.loads(eval_path.read_text(encoding="utf-8"))
+ eval_result["run_id"] = args.from_run
+ rows = load_records(path)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ ks = None
+ if getattr(args, "ks", None):
+ ks = tuple(int(x) for x in str(args.ks).split(",") if x.strip())
+ console.print(
+ f"[cyan]Harness CE on retrieved hits[/cyan] from={args.from_run} "
+ f"dataset={path.name} run={run_id} (not Reddy 0.857)"
+ )
+ result = run_ce_on_retrieved(
+ eval_result,
+ rows,
+ dataset_name=path.name,
+ ks=ks,
+ brain_id=settings.brain_id,
+ )
+ report = write_report(run_dir, result)
+ print_report_table(report)
+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")
+ return 0 if report.get("status") == "ok" else 1
+
+
+def cmd_rank_pool_ce(args: argparse.Namespace, settings: Settings) -> int:
+ from search.rank_pool import run_ce_on_pool
+
+ path = _resolve_dataset_path(args, settings)
+ if not path.exists():
+ console.print(f"[red]Missing[/red] {path}")
+ return 1
+ rows = load_records(path)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ console.print(
+ f"[cyan]Ranking-in-pool CE[/cyan] dataset={path.name} run={run_id} "
+ f"(cite Reddy 0.857 only if protocol is CE-on-pool; n is not ~4477)"
+ )
+ result = run_ce_on_pool(
+ rows,
+ dataset_name=path.name,
+ brain_id=settings.brain_id,
+ )
+ report = write_report(run_dir, result)
+ print_report_table(report)
+ console.print(f"[green]Wrote[/green] {run_dir / 'report.json'}")
+ if result.get("ce_model"):
+ console.print(
+ f"ce_model={result.get('ce_model')} "
+ f"missing_text={result.get('ce_missing_text')}"
+ )
+ return 0 if report.get("status") == "ok" else 1
+
+
+def build_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(
+ prog="search",
+ description=(
+ "Labeled search eval on searchbench*: ingest docs → "
+ "POST /retrieve/search (Recall/nDCG/MRR + retrieve latency)"
+ ),
+ )
+ parser.add_argument(
+ "--env-file",
+ type=Path,
+ default=None,
+ help="Optional dotenv path (default benchmarks/.env)",
+ )
+ parser.add_argument(
+ "--brain",
+ default=None,
+ help=f"Brain id (default {DEFAULT_BRAIN_ID}; must start with searchbench)",
+ )
+ sub = parser.add_subparsers(dest="command", required=True)
+
+ p_download = sub.add_parser(
+ "download",
+ help="Download Amazon ESCI and/or WANDS into search JSONL",
+ )
+ p_download.add_argument(
+ "--name",
+ default="all",
+ help="esci, wands, jdsearch, or all (comma-separated; all = esci,wands)",
+ )
+ p_download.add_argument("--out", type=str, default=None)
+ p_download.add_argument("--force", action="store_true")
+ p_download.add_argument(
+ "--dry-stats",
+ action="store_true",
+ dest="dry_stats",
+ help="JDsearch only: print label histogram from the tar/cache without writing JSONL",
+ )
+ p_download.add_argument("--max-queries", type=int, default=80)
+ p_download.add_argument("--max-docs", type=int, default=2000)
+ p_download.add_argument("--candidates-per-query", type=int, default=40)
+ p_download.add_argument(
+ "--locale",
+ default="us",
+ help="ESCI locale: us, es, or jp (default us). Not Italian.",
+ )
+ p_download.add_argument(
+ "--split",
+ default="test",
+ help="ESCI split (default test; small_version / Task 1 only)",
+ )
+ p_download.add_argument(
+ "--holdout-dataset",
+ dest="holdout_dataset",
+ default="",
+ help="JSONL whose qids are skipped (e.g. data/search_esci_74.jsonl)",
+ )
+ p_download.set_defaults(func=cmd_download)
+
+ p_stats = sub.add_parser("dataset-stats", help="Summarize search JSONL")
+ p_stats.add_argument("--dataset", type=str, default=None)
+ p_stats.set_defaults(func=cmd_dataset_stats)
+
+ p_smoke = sub.add_parser(
+ "smoke", help="Ingest a few docs then POST /retrieve/search"
+ )
+ p_smoke.add_argument("--dataset", type=str, default=None)
+ p_smoke.add_argument("--limit", type=int, default=2)
+ p_smoke.add_argument("--limit-queries", type=int, default=2)
+ p_smoke.add_argument("--k", type=int, default=10)
+ p_smoke.add_argument("--fusion", choices=["rrf", "cc"], default=None)
+ p_smoke.add_argument("--fusion-alpha", dest="fusion_alpha", type=float, default=None)
+ p_smoke.add_argument(
+ "--rerank",
+ default=None,
+ help="none or plugin:<name> (e.g. plugin:cross-encoder). Unknown name is 400.",
+ )
+ p_smoke.add_argument(
+ "--channels",
+ default=None,
+ help=(
+ "Comma-separated channels (default passages): "
+ "passages,entities,events,communities and/or plugin:<name>."
+ ),
+ )
+ p_smoke.add_argument("--node-labels", dest="node_labels", default=None)
+ p_smoke.add_argument("--community-labels", dest="community_labels", default=None)
+ p_smoke.add_argument(
+ "--expand",
+ choices=["none", "neighbors"],
+ default="none",
+ )
+ p_smoke.add_argument(
+ "--ingest-graph",
+ action="store_true",
+ help="Also write deterministic catalog triples (entity uuid = doc_id).",
+ )
+ p_smoke.add_argument(
+ "--interactions",
+ type=str,
+ default=None,
+ help="Optional interaction JSONL (EVENT+happened_at) on this searchbench brain.",
+ )
+ p_smoke.add_argument("--timeout", type=float, default=600.0)
+ p_smoke.add_argument(
+ "--enrich",
+ action="store_true",
+ help="Run Scout/Architect LLM ingest. Default skips enrichment (chunk+embed only).",
+ )
+ p_smoke.add_argument(
+ "--skip-ingest",
+ action="store_true",
+ help="Reuse chunks already on this searchbench brain (no POST /ingest/).",
+ )
+ p_smoke.add_argument(
+ "--personalize",
+ action="store_true",
+ dest="personalize",
+ help="Send each query's target field on POST /retrieve/search. Default omit.",
+ )
+ p_smoke.add_argument(
+ "--target-from-query",
+ action="store_true",
+ dest="personalize",
+ help="Alias for --personalize.",
+ )
+ p_smoke.set_defaults(func=cmd_smoke)
+
+ p_bf = sub.add_parser(
+ "backfill-entity-text",
+ help="ENTITY-only search_text + nodes vector refresh (not architect, not frozen brains)",
+ )
+ p_bf.add_argument("--dataset", type=str, default="data/search_wands.jsonl")
+ p_bf.add_argument("--limit", type=int, default=None)
+ p_bf.set_defaults(func=cmd_backfill_entity_text)
+
+ p_eval = sub.add_parser(
+ "evaluate",
+ help="Ingest corpus, search all queries, write report + ledger",
+ )
+ p_eval.add_argument("--dataset", type=str, default=None)
+ p_eval.add_argument("--run", type=str, default=None)
+ p_eval.add_argument("--ks", type=str, default="5,10,20")
+ p_eval.add_argument("--k", type=int, default=20)
+ p_eval.add_argument("--fusion", choices=["rrf", "cc"], default=None)
+ p_eval.add_argument("--fusion-alpha", dest="fusion_alpha", type=float, default=None)
+ p_eval.add_argument(
+ "--rerank",
+ default=None,
+ help="none or plugin:<name> (e.g. plugin:cross-encoder). Unknown name is 400.",
+ )
+ p_eval.add_argument(
+ "--mode",
+ choices=["default", "catalog"],
+ default="default",
+ help="default keeps RERANK_MAX_K=10. catalog retrieves deeper and reranks up to 50.",
+ )
+ p_eval.add_argument(
+ "--channels",
+ default=None,
+ help=(
+ "Comma-separated channels (default passages): "
+ "passages,entities,events,communities and/or plugin:<name>."
+ ),
+ )
+ p_eval.add_argument("--node-labels", dest="node_labels", default=None)
+ p_eval.add_argument("--community-labels", dest="community_labels", default=None)
+ p_eval.add_argument(
+ "--expand",
+ choices=["none", "neighbors"],
+ default="none",
+ )
+ p_eval.add_argument(
+ "--ingest-graph",
+ action="store_true",
+ help="Also write deterministic catalog triples (entity uuid = doc_id).",
+ )
+ p_eval.add_argument(
+ "--interactions",
+ type=str,
+ default=None,
+ help="Optional interaction JSONL (EVENT+happened_at) on this searchbench brain.",
+ )
+ p_eval.add_argument("--timeout", type=float, default=600.0)
+ p_eval.add_argument(
+ "--enrich",
+ action="store_true",
+ help="Run Scout/Architect LLM ingest. Default skips enrichment (chunk+embed only).",
+ )
+ p_eval.add_argument("--limit-docs", dest="limit_docs", type=int, default=None)
+ p_eval.add_argument("--limit-queries", dest="limit_queries", type=int, default=None)
+ p_eval.add_argument(
+ "--skip-ingest",
+ action="store_true",
+ help="Reuse chunks already on this searchbench brain (no POST /ingest/).",
+ )
+ p_eval.add_argument(
+ "--rank-pool",
+ action="store_true",
+ dest="rank_pool",
+ help=(
+ "Restrict hits to each query's candidate_doc_ids (including I) "
+ "and score pool nDCG@20. Do not average with shared-corpus nDCG@10."
+ ),
+ )
+ p_eval.add_argument(
+ "--personalize",
+ action="store_true",
+ dest="personalize",
+ help="Send each query's target field on POST /retrieve/search. Default omit.",
+ )
+ p_eval.add_argument(
+ "--target-from-query",
+ action="store_true",
+ dest="personalize",
+ help="Alias for --personalize.",
+ )
+ p_eval.add_argument(
+ "--extras",
+ default=None,
+ help='Optional equality filter JSON object, e.g. {"locale":"it"}',
+ )
+ p_eval.set_defaults(func=cmd_evaluate)
+
+ p_report = sub.add_parser("report", help="Rebuild report from eval.json")
+ p_report.add_argument("--run", type=str, required=True)
+ p_report.set_defaults(func=cmd_report)
+
+ p_pool = sub.add_parser(
+ "rank-pool-ce",
+ help="Score labeled candidate pools with MiniLM CE (no search API)",
+ )
+ p_pool.add_argument("--dataset", type=str, default=None)
+ p_pool.add_argument("--run", type=str, default=None)
+ p_pool.set_defaults(func=cmd_rank_pool_ce)
+
+ p_rr = sub.add_parser(
+ "rerank-retrieved",
+ help="Harness CE over stored first-stage hits (not production RERANK_MAX_K)",
+ )
+ p_rr.add_argument("--from-run", dest="from_run", type=str, required=True)
+ p_rr.add_argument("--dataset", type=str, default=None)
+ p_rr.add_argument("--run", type=str, default=None)
+ p_rr.add_argument("--ks", type=str, default=None)
+ p_rr.set_defaults(func=cmd_rerank_retrieved)
+
+ p_ft = sub.add_parser(
+ "finetune-ce",
+ help="Fine-tune a pool CE on ESCI US train (test qids held out)",
+ )
+ p_ft.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_ft.add_argument(
+ "--out",
+ type=str,
+ default="data/models/esci-minilm-graded",
+ )
+ p_ft.add_argument("--max-pairs", type=int, default=80000)
+ p_ft.add_argument("--epochs", type=int, default=1)
+ p_ft.add_argument("--batch-size", type=int, default=32)
+ p_ft.add_argument(
+ "--label-mode",
+ choices=["binary", "graded", "multiclass"],
+ default="graded",
+ )
+ p_ft.add_argument("--fields", choices=["title", "catalog"], default="catalog")
+ p_ft.add_argument(
+ "--base",
+ default="cross-encoder/ms-marco-MiniLM-L-6-v2",
+ )
+ p_ft.add_argument("--lr", type=float, default=7e-6)
+ p_ft.add_argument("--max-length", type=int, default=256)
+ p_ft.set_defaults(func=cmd_finetune_ce)
+
+ p_ms = sub.add_parser(
+ "miss-strata",
+ help="Classify k=50 hits: head-ok / rank-too-low / total-miss",
+ )
+ p_ms.add_argument("--from-run", dest="from_run", type=str, required=True)
+ p_ms.add_argument("--dataset", type=str, default=None)
+ p_ms.add_argument("--out", type=str, default=None)
+ p_ms.set_defaults(func=cmd_miss_strata)
+
+ p_qw = sub.add_parser(
+ "query-rewrite",
+ help="Rewrite pathological total-miss queries only (skip esci-72)",
+ )
+ p_qw.add_argument("--from-run", dest="from_run", type=str, required=True)
+ p_qw.add_argument("--dataset", type=str, default=None)
+ p_qw.add_argument("--out", type=str, default=None)
+ p_qw.set_defaults(func=cmd_query_rewrite)
+
+ p_sp = sub.add_parser(
+ "spell-normalize",
+ help="Harness-only NFKC/punctuation/accent normalize of query text",
+ )
+ p_sp.add_argument("--dataset", type=str, default="data/search_esci_es.jsonl")
+ p_sp.add_argument("--out", type=str, default="data/search_esci_es_spell.jsonl")
+ p_sp.set_defaults(func=cmd_spell_normalize)
+
+ p_ftd = sub.add_parser(
+ "finetune-dense",
+ help="Fine-tune a dual encoder with k=50 hard-negative bank (test qids held out)",
+ )
+ p_ftd.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_ftd.add_argument(
+ "--out",
+ type=str,
+ default="data/models/esci-minilm-dense-ance",
+ )
+ p_ftd.add_argument(
+ "--from-eval",
+ type=str,
+ default="runs/search-esci-74-passages-k50/eval.json",
+ )
+ p_ftd.add_argument("--max-pairs", type=int, default=20000)
+ p_ftd.add_argument("--epochs", type=int, default=1)
+ p_ftd.add_argument("--batch-size", type=int, default=32)
+ p_ftd.add_argument("--fields", choices=["title", "catalog"], default="catalog")
+ p_ftd.add_argument(
+ "--base",
+ default="sentence-transformers/all-MiniLM-L6-v2",
+ )
+ p_ftd.add_argument("--lr", type=float, default=2e-5)
+ p_ftd.add_argument("--max-length", type=int, default=256)
+ p_ftd.set_defaults(func=cmd_finetune_dense)
+
+ p_ld = sub.add_parser(
+ "local-dense",
+ help="Retrieve the JSONL corpus with a local dual encoder (not BrainAPI)",
+ )
+ p_ld.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_ld.add_argument("--run", type=str, default="search-esci-74-dense-ance-k50")
+ p_ld.add_argument("--model", type=str, required=True)
+ p_ld.add_argument("--k", type=int, default=50)
+ p_ld.add_argument("--ks", type=str, default="5,10,20,50")
+ p_ld.set_defaults(func=cmd_local_dense)
+
+ p_mine = sub.add_parser(
+ "mine-retrieved-lists",
+ help="BM25 top-k train lists with ESCI labels; unlabeled=I",
+ )
+ p_mine.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_mine.add_argument("--out", type=str, default="data/esci_retrieved_lists.jsonl")
+ p_mine.add_argument("--max-queries", type=int, default=6000)
+ p_mine.add_argument("--k", type=int, default=50)
+ p_mine.add_argument("--seed", type=int, default=11)
+ p_mine.set_defaults(func=cmd_mine_retrieved_lists)
+
+ p_exh = sub.add_parser(
+ "export-hybrid-lists",
+ help="Export stored hybrid k=50 hits as 4-class rows; unlabeled=I",
+ )
+ p_exh.add_argument(
+ "--from-run",
+ dest="from_run",
+ default="search-esci-ltr200-passages-k50",
+ )
+ p_exh.add_argument("--dataset", type=str, default="data/search_esci_ltr200.jsonl")
+ p_exh.add_argument(
+ "--holdout-dataset",
+ dest="holdout_dataset",
+ default="data/search_esci_74.jsonl",
+ )
+ p_exh.add_argument("--out", type=str, default="data/esci_hybrid_lists_ltr200.jsonl")
+ p_exh.add_argument("--k", type=int, default=50)
+ p_exh.set_defaults(func=cmd_export_hybrid_lists)
+
+ p_ft4 = sub.add_parser(
+ "finetune-4class",
+ help="Fine-tune 4-class L-12 CE (pool or retrieved-bm25 lists)",
+ )
+ p_ft4.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_ft4.add_argument(
+ "--out",
+ type=str,
+ default="data/models/esci-minilm-l12-retrieved",
+ )
+ p_ft4.add_argument("--from-lists", type=str, default=None)
+ p_ft4.add_argument(
+ "--lists-source",
+ dest="lists_source",
+ default="",
+ help="Label for --from-lists (hybrid-k50 or retrieved-bm25)",
+ )
+ p_ft4.add_argument("--max-pairs", type=int, default=80000)
+ p_ft4.add_argument("--epochs", type=int, default=1)
+ p_ft4.add_argument("--batch-size", type=int, default=32)
+ p_ft4.add_argument("--max-length", type=int, default=192)
+ p_ft4.add_argument("--ckpt-every", type=int, default=200)
+ p_ft4.add_argument("--class-weights", action="store_true")
+ p_ft4.add_argument(
+ "--base",
+ default="cross-encoder/ms-marco-MiniLM-L-12-v2",
+ )
+ p_ft4.add_argument("--seed", type=int, default=11)
+ p_ft4.set_defaults(func=cmd_finetune_4class)
+
+ p_cb = sub.add_parser(
+ "colbert-local",
+ help="ColBERT MaxSim sidecar over the JSONL corpus (not BrainAPI)",
+ )
+ p_cb.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_cb.add_argument("--run", type=str, default="search-esci-74-colbert-k50")
+ p_cb.add_argument("--k", type=int, default=50)
+ p_cb.add_argument("--ks", type=str, default="5,10,20,50")
+ p_cb.set_defaults(func=cmd_colbert_local)
+
+ p_rc = sub.add_parser(
+ "rank-corpus",
+ help="Score every JSONL doc with 4-class CE (exhaustive-catalog, not production)",
+ )
+ p_rc.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_rc.add_argument("--run", type=str, default="search-esci-74-exhaustive-ce")
+ p_rc.add_argument(
+ "--model",
+ type=str,
+ default="data/models/esci-minilm-l12-4class-nowt-e2",
+ )
+ p_rc.add_argument("--k", type=int, default=50)
+ p_rc.add_argument("--ks", type=str, default="5,10,20,50")
+ p_rc.add_argument("--max-length", dest="max_length", type=int, default=192)
+ p_rc.set_defaults(func=cmd_rank_corpus)
+
+ p_ov = sub.add_parser(
+ "list-overlap",
+ help="Count gold ASINs in sidecar top-k missing from passages",
+ )
+ p_ov.add_argument(
+ "--passages-run",
+ dest="passages_run",
+ default="search-esci-74-passages-k50",
+ )
+ p_ov.add_argument(
+ "--against-runs",
+ dest="against_runs",
+ default="search-esci-74-bge-base-k50,search-esci-74-colbert-k50",
+ )
+ p_ov.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_ov.add_argument("--run", type=str, default="search-esci-74-list-overlap")
+ p_ov.add_argument("--k", type=int, default=50)
+ p_ov.set_defaults(func=cmd_list_overlap)
+
+ p_un = sub.add_parser(
+ "union-lists",
+ help="RRF-union stored first-stage lists (harness only, not live graph)",
+ )
+ p_un.add_argument(
+ "--from-runs",
+ dest="from_runs",
+ default="search-esci-74-passages-k50,search-esci-74-bge-base-k50",
+ )
+ p_un.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_un.add_argument("--run", type=str, default="search-esci-74-union-bge-k50")
+ p_un.add_argument("--k", type=int, default=50)
+ p_un.add_argument("--ks", type=str, default="5,10,20,50")
+ p_un.set_defaults(func=cmd_union_lists)
+
+ p_cs = sub.add_parser(
+ "cascade-lists",
+ help="Keep passages top-10; inject sidecar unique golds into ranks 11-50",
+ )
+ p_cs.add_argument(
+ "--passages-run",
+ dest="passages_run",
+ default="search-esci-74-passages-k50",
+ )
+ p_cs.add_argument(
+ "--from-runs",
+ dest="from_runs",
+ default="search-esci-74-bge-base-k50,search-esci-74-colbert-k50",
+ )
+ p_cs.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_cs.add_argument("--run", type=str, default="search-esci-74-cascade-tail-k50")
+ p_cs.add_argument("--k", type=int, default=50)
+ p_cs.add_argument("--head-k", dest="head_k", type=int, default=10)
+ p_cs.add_argument("--ks", type=str, default="5,10,20,50")
+ p_cs.set_defaults(func=cmd_cascade_lists)
+
+ p_ltr = sub.add_parser(
+ "ltr-head",
+ help="Query-grouped CV LTR on stored hybrid k=50 (BM25/dense rank + title overlap)",
+ )
+ p_ltr.add_argument(
+ "--from-run",
+ dest="from_run",
+ default="search-esci-74-passages-k50",
+ )
+ p_ltr.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_ltr.add_argument("--run", type=str, default="search-esci-74-ltr-head-k50")
+ p_ltr.add_argument("--k", type=int, default=50)
+ p_ltr.add_argument("--ks", type=str, default="5,10,20,50")
+ p_ltr.add_argument("--folds", type=int, default=5)
+ p_ltr.add_argument(
+ "--pair-policy",
+ dest="pair_policy",
+ default="unlabeled_zero",
+ choices=["unlabeled_zero", "other_query_neg"],
+ )
+ p_ltr.add_argument(
+ "--ce-model",
+ dest="ce_model",
+ default="",
+ help="Optional 4-class CE dir; ce_gain is a feature, not the ranker",
+ )
+ p_ltr.add_argument(
+ "--ltr-model",
+ dest="ltr_model",
+ default="ranknet",
+ choices=["ranknet", "lightgbm"],
+ help="Linear RankNet (default) or gated LightGBM lambdarank",
+ )
+ p_ltr.add_argument(
+ "--train-from-run",
+ dest="train_from_run",
+ default="",
+ help="Fit on this run's lists; apply to --from-run (no CV on eval qids)",
+ )
+ p_ltr.add_argument(
+ "--train-dataset",
+ dest="train_dataset",
+ default="",
+ help="JSONL for --train-from-run",
+ )
+ p_ltr.set_defaults(func=cmd_ltr_head)
+
+ p_fs = sub.add_parser(
+ "pool-first-stage",
+ help="Local fielded BM25 over the JSONL corpus (not Reddy 0.857)",
+ )
+ p_fs.add_argument("--dataset", type=str, default="data/search_esci_74.jsonl")
+ p_fs.add_argument("--run", type=str, default="search-esci-74-fielded-bm25")
+ p_fs.add_argument(
+ "--variant",
+ default="all,title,title-boost,rm3,rm3-title-boost",
+ )
+ p_fs.add_argument(
+ "--expand-out",
+ type=str,
+ default="data/search_esci_74_rm3.jsonl",
+ )
+ p_fs.set_defaults(func=cmd_pool_first_stage)
+
+ return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = build_parser()
+ args = parser.parse_args(argv)
+ settings = Settings.load(args.env_file)
+ if args.brain:
+ settings.brain_id = validate_brain_id(args.brain)
+ return int(args.func(args, settings))
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/search/client.py b/benchmarks/search/client.py
new file mode 100644
index 0000000..5f693b3
--- /dev/null
+++ b/benchmarks/search/client.py
@@ -0,0 +1,268 @@
+from __future__ import annotations
+
+import time
+from dataclasses import dataclass, field
+from typing import Any
+
+import httpx
+
+from search.config import TERMINAL_TASK_STATUSES, Settings
+
+
+class SearchDisabledError(RuntimeError):
+ pass
+
+
+@dataclass
+class TimedResult:
+ data: Any
+ latency_ms: float
+ status_code: int = 200
+
+
+@dataclass
+class BrainAPIClient:
+ settings: Settings
+ timeout: float = 120.0
+ max_retries: int = 5
+ _client: httpx.Client = field(init=False, repr=False)
+
+ def __post_init__(self) -> None:
+ self.settings.require_brainapi()
+ self._client = httpx.Client(
+ base_url=self.settings.brainapi_url,
+ headers={
+ "Authorization": f"Bearer {self.settings.brainpat_token}",
+ "Content-Type": "application/json",
+ "X-Brain-ID": self.settings.brain_id,
+ },
+ timeout=self.timeout,
+ )
+
+ def close(self) -> None:
+ self._client.close()
+
+ def __enter__(self) -> "BrainAPIClient":
+ return self
+
+ def __exit__(self, *args: object) -> None:
+ self.close()
+
+ def _request(
+ self,
+ method: str,
+ path: str,
+ *,
+ json: dict | None = None,
+ params: dict | None = None,
+ ) -> TimedResult:
+ last_error: Exception | None = None
+ for attempt in range(self.max_retries):
+ started = time.perf_counter()
+ try:
+ response = self._client.request(
+ method, path, json=json, params=params
+ )
+ latency_ms = (time.perf_counter() - started) * 1000
+ if (
+ response.status_code == 404
+ and path.rstrip("/").endswith("/retrieve/search")
+ ):
+ detail = ""
+ try:
+ detail = response.text[:500]
+ except Exception:
+ detail = ""
+ raise SearchDisabledError(
+ "POST /retrieve/search returned 404. Set SEARCH_ENABLED=true "
+ "(and DATA_DB=postgresql) on the API under test. "
+ f"body={detail!r}"
+ ) from None
+ if response.status_code in (429, 500, 502, 503, 504):
+ wait = min(2**attempt * 0.5, 30)
+ time.sleep(wait)
+ last_error = httpx.HTTPStatusError(
+ f"HTTP {response.status_code}",
+ request=response.request,
+ response=response,
+ )
+ continue
+ if response.status_code >= 400:
+ detail = ""
+ try:
+ detail = response.text[:500]
+ except Exception:
+ detail = ""
+ raise RuntimeError(
+ f"{method} {path} returned {response.status_code}. "
+ f"body={detail!r}"
+ ) from None
+ if response.status_code == 204 or not response.content:
+ return TimedResult(
+ data=None,
+ latency_ms=latency_ms,
+ status_code=response.status_code,
+ )
+ return TimedResult(
+ data=response.json(),
+ latency_ms=latency_ms,
+ status_code=response.status_code,
+ )
+ except SearchDisabledError:
+ raise
+ except (httpx.TransportError, httpx.TimeoutException) as exc:
+ last_error = exc
+ wait = min(2**attempt * 0.5, 30)
+ time.sleep(wait)
+ raise RuntimeError(
+ f"Request failed after retries: {method} {path}: {last_error}"
+ )
+
+ def ingest_structured(
+ self,
+ triples: list[dict[str, Any]],
+ brain_id: str | None = None,
+ *,
+ mode: str = "deterministic",
+ ) -> TimedResult:
+ bid = brain_id or self.settings.brain_id
+ body = {
+ "mode": mode,
+ "brain_id": bid,
+ "data": triples,
+ }
+ return self._request("POST", "/ingest/structured", json=body)
+
+ def ingest_text(
+ self,
+ text: str,
+ brain_id: str | None = None,
+ *,
+ skip_enrichment: bool = False,
+ meta_keys: dict[str, Any] | None = None,
+ ) -> TimedResult:
+ bid = brain_id or self.settings.brain_id
+ body: dict[str, Any] = {
+ "data": {"data_type": "text", "text_data": text},
+ "brain_id": bid,
+ "observate_for": [],
+ "preferred_extraction_entities": [],
+ "skip_enrichment": skip_enrichment,
+ }
+ if meta_keys:
+ body["meta_keys"] = meta_keys
+ return self._request("POST", "/ingest/", json=body)
+
+ def get_task(self, task_id: str, brain_id: str | None = None) -> TimedResult:
+ bid = brain_id or self.settings.brain_id
+ return self._request(
+ "GET", f"/tasks/{task_id}", params={"brain_id": bid}
+ )
+
+ def wait_for_task(
+ self,
+ task_id: str,
+ brain_id: str | None = None,
+ *,
+ timeout_s: float = 600.0,
+ poll_interval_s: float = 2.0,
+ ) -> TimedResult:
+ bid = brain_id or self.settings.brain_id
+ started = time.perf_counter()
+ while True:
+ result = self.get_task(task_id, bid)
+ status = (result.data or {}).get("status", "unknown")
+ if status in TERMINAL_TASK_STATUSES:
+ return TimedResult(
+ data=result.data,
+ latency_ms=(time.perf_counter() - started) * 1000,
+ status_code=result.status_code,
+ )
+ if time.perf_counter() - started > timeout_s:
+ raise TimeoutError(
+ f"Task {task_id} did not finish within {timeout_s}s "
+ f"(last status={status})"
+ )
+ time.sleep(poll_interval_s)
+
+ def search(
+ self,
+ query: str,
+ brain_id: str | None = None,
+ *,
+ k: int = 20,
+ fusion: str | None = None,
+ fusion_alpha: float | None = None,
+ rerank: str | None = None,
+ mode: str | None = None,
+ profile_stages: bool = True,
+ channels: list[str] | None = None,
+ node_labels: list[str] | None = None,
+ community_labels: list[str] | None = None,
+ expand: str | None = None,
+ extras: dict[str, str] | None = None,
+ target: str | None = None,
+ ) -> TimedResult:
+ bid = brain_id or self.settings.brain_id
+ body: dict[str, Any] = {
+ "query": query,
+ "brain_id": bid,
+ "k": k,
+ "channels": channels or ["passages"],
+ "profile_stages": profile_stages,
+ }
+ if fusion:
+ body["fusion"] = fusion
+ if fusion_alpha is not None:
+ body["fusion_alpha"] = fusion_alpha
+ if rerank:
+ body["rerank"] = rerank
+ if mode and mode != "default":
+ body["mode"] = mode
+ if node_labels:
+ body["node_labels"] = node_labels
+ if community_labels:
+ body["community_labels"] = community_labels
+ if expand and expand != "none":
+ body["expand"] = expand
+ if extras:
+ body["extras"] = extras
+ if target:
+ body["target"] = target
+ return self._request("POST", "/retrieve/search", json=body)
+
+ def get_neighbors(
+ self,
+ uuid: str,
+ brain_id: str | None = None,
+ *,
+ limit: int = 10,
+ look_for: str | None = None,
+ ) -> TimedResult:
+ bid = brain_id or self.settings.brain_id
+ params: dict[str, Any] = {
+ "uuid": uuid,
+ "brain_id": bid,
+ "limit": limit,
+ }
+ if look_for:
+ params["look_for"] = look_for
+ return self._request("GET", "/retrieve/entities/neighbors", params=params)
+
+ def list_text_chunks(
+ self,
+ brain_id: str | None = None,
+ *,
+ limit: int = 100,
+ skip: int = 0,
+ query_text: str | None = None,
+ ) -> TimedResult:
+ bid = brain_id or self.settings.brain_id
+ params: dict[str, Any] = {
+ "brain_id": bid,
+ "limit": limit,
+ "skip": skip,
+ }
+ if query_text:
+ params["query_text"] = query_text
+ return self._request("GET", "/retrieve/text-chunks", params=params)
diff --git a/benchmarks/search/config.py b/benchmarks/search/config.py
new file mode 100644
index 0000000..20484e0
--- /dev/null
+++ b/benchmarks/search/config.py
@@ -0,0 +1,96 @@
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+from pathlib import Path
+
+from dotenv import load_dotenv
+
+BENCHMARKS_ROOT = Path(__file__).resolve().parent.parent
+DATA_DIR = BENCHMARKS_ROOT / "data"
+RUNS_DIR = BENCHMARKS_ROOT / "runs"
+DEFAULT_DATASET_PATH = DATA_DIR / "search_toy.jsonl"
+DEFAULT_BRAIN_ID = "searchbenchsmoke"
+REQUIRED_PREFIX = "searchbench"
+FORBIDDEN_BRAINS = frozenset(
+ {
+ "beam1m1clean",
+ "demorecsys",
+ "locomoconv26",
+ "locomoconv26clean",
+ "locomoconv26nostorm",
+ }
+)
+FORBIDDEN_PREFIXES = (
+ "beam",
+ "demorecsys",
+ "lme",
+ "locomoconv",
+ "longmemeval",
+)
+FROZEN_STRUCTURED_BRAINS = frozenset(
+ {
+ "searchbenchwands",
+ "searchbenchesci74",
+ "searchbenchescies",
+ "searchbenchesciltr2",
+ }
+)
+TERMINAL_TASK_STATUSES = frozenset({"completed", "failed", "partial_failed"})
+
+
+def validate_brain_id(brain_id: str) -> str:
+ bid = (brain_id or "").strip()
+ if not bid:
+ raise SystemExit("brain_id is required.")
+ lowered = bid.lower()
+ forbidden = {item.lower() for item in FORBIDDEN_BRAINS}
+ if lowered in forbidden or lowered.startswith(FORBIDDEN_PREFIXES):
+ raise SystemExit(
+ f"Refusing brain_id={bid!r}. Use a dedicated searchbench* brain "
+ f"(default {DEFAULT_BRAIN_ID}). Never wipe eval memory or recsys brains."
+ )
+ if not lowered.startswith(REQUIRED_PREFIX):
+ raise SystemExit(
+ f"Refusing brain_id={bid!r}. Search eval brains must start with "
+ f"{REQUIRED_PREFIX!r} (default {DEFAULT_BRAIN_ID})."
+ )
+ return bid
+
+
+@dataclass
+class Settings:
+ brainapi_url: str
+ brainpat_token: str
+ dataset_path: Path
+ runs_dir: Path
+ brain_id: str
+
+ @classmethod
+ def load(cls, env_file: Path | None = None) -> "Settings":
+ if env_file is None:
+ env_file = BENCHMARKS_ROOT / ".env"
+ if env_file.exists():
+ load_dotenv(env_file, override=False)
+ else:
+ load_dotenv(override=False)
+
+ brain_id = validate_brain_id(
+ os.getenv("SEARCH_BRAIN_ID", DEFAULT_BRAIN_ID) or DEFAULT_BRAIN_ID
+ )
+ return cls(
+ brainapi_url=os.getenv("BRAINAPI_URL", "http://localhost:8000").rstrip("/"),
+ brainpat_token=os.getenv("BRAINPAT_TOKEN", ""),
+ dataset_path=Path(
+ os.getenv("SEARCH_DATASET_PATH", str(DEFAULT_DATASET_PATH))
+ ),
+ runs_dir=Path(os.getenv("BENCH_RUNS_DIR", str(RUNS_DIR))),
+ brain_id=brain_id,
+ )
+
+ def require_brainapi(self) -> None:
+ if not self.brainpat_token:
+ raise SystemExit(
+ "BRAINPAT_TOKEN is required. Copy benchmarks/.env.example to "
+ "benchmarks/.env and set it."
+ )
diff --git a/benchmarks/search/dataset.py b/benchmarks/search/dataset.py
new file mode 100644
index 0000000..3a75241
--- /dev/null
+++ b/benchmarks/search/dataset.py
@@ -0,0 +1,76 @@
+from __future__ import annotations
+
+import json
+from collections import Counter
+from pathlib import Path
+from typing import Any
+
+DOC_MARKER_PREFIX = "DOCID "
+
+
+def doc_marker(doc_id: str) -> str:
+ return f"{DOC_MARKER_PREFIX}{doc_id}"
+
+
+def write_records(rows: list[dict[str, Any]], path: Path) -> Path:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with path.open("w", encoding="utf-8") as fh:
+ for row in rows:
+ fh.write(json.dumps(row, ensure_ascii=False) + "\n")
+ return path
+
+
+def load_records(path: Path) -> list[dict[str, Any]]:
+ rows: list[dict[str, Any]] = []
+ for line in path.read_text(encoding="utf-8").splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ rows.append(json.loads(line))
+ return rows
+
+
+def split_corpus(
+ rows: list[dict[str, Any]],
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ docs: list[dict[str, Any]] = []
+ queries: list[dict[str, Any]] = []
+ for row in rows:
+ kind = str(row.get("type") or "").strip().lower()
+ if kind == "doc":
+ docs.append(row)
+ elif kind == "query":
+ queries.append(row)
+ else:
+ raise ValueError(f"Unknown record type {kind!r} in {row!r}")
+ return docs, queries
+
+
+def dataset_stats(rows: list[dict[str, Any]]) -> dict[str, Any]:
+ docs, queries = split_corpus(rows)
+ slices = Counter(str(q.get("slice") or "unspecified") for q in queries)
+ return {
+ "n_docs": len(docs),
+ "n_queries": len(queries),
+ "slices": dict(slices),
+ "doc_ids": [str(d.get("doc_id") or "") for d in docs],
+ "graded": any(bool(q.get("gold_grades")) for q in queries),
+ }
+
+
+def map_doc_ids_to_chunks(
+ docs: list[dict[str, Any]],
+ chunks: list[dict[str, Any]],
+) -> dict[str, set[str]]:
+ mapping: dict[str, set[str]] = {str(doc["doc_id"]): set() for doc in docs}
+ for chunk in chunks:
+ text = str(chunk.get("text") or "")
+ chunk_id = str(chunk.get("id") or "")
+ if not chunk_id:
+ continue
+ for doc in docs:
+ doc_id = str(doc.get("doc_id") or "")
+ marker = str(doc.get("marker") or doc_marker(doc_id))
+ if marker and marker in text:
+ mapping.setdefault(doc_id, set()).add(chunk_id)
+ return mapping
diff --git a/benchmarks/search/evaluate.py b/benchmarks/search/evaluate.py
new file mode 100644
index 0000000..e194bf2
--- /dev/null
+++ b/benchmarks/search/evaluate.py
@@ -0,0 +1,747 @@
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from search.client import BrainAPIClient, SearchDisabledError, TimedResult
+from search.config import FROZEN_STRUCTURED_BRAINS, Settings
+from search.dataset import map_doc_ids_to_chunks, split_corpus
+from search.mapping import docs_to_triples, interactions_to_triples
+from search.metrics import (
+ aggregate_query_metrics,
+ mrr,
+ ndcg_at_k,
+ recall_at_k,
+ retrieve_latency_ms,
+)
+
+
+def ensure_run_dir(settings: Settings, run_id: str | None = None) -> tuple[str, Path]:
+ stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+ rid = run_id or f"search-{stamp}"
+ run_dir = settings.runs_dir / rid
+ run_dir.mkdir(parents=True, exist_ok=True)
+ return rid, run_dir
+
+
+def _hit_ids(payload: dict[str, Any] | None) -> list[str]:
+ hits = (payload or {}).get("hits") or []
+ ids: list[str] = []
+ for hit in hits:
+ if not isinstance(hit, dict):
+ continue
+ hid = hit.get("id")
+ if hid:
+ ids.append(str(hid))
+ return ids
+
+
+def _hit_dump(
+ payload: dict[str, Any] | None,
+ chunk_to_doc: dict[str, str] | None = None,
+) -> list[dict[str, Any]]:
+ mapping = chunk_to_doc or {}
+ rows: list[dict[str, Any]] = []
+ for hit in (payload or {}).get("hits") or []:
+ if not isinstance(hit, dict):
+ continue
+ hid = hit.get("id")
+ if not hid:
+ continue
+ raw_id = str(hid)
+ node_id = str(hit.get("node_id") or "").strip() or None
+ rows.append(
+ {
+ "id": raw_id,
+ "channel": str(hit.get("channel") or ""),
+ "doc_id": mapping.get(raw_id, raw_id),
+ "node_id": node_id,
+ }
+ )
+ return rows
+
+
+_DOC_EXTRAS_KEYS = ("brand", "color", "locale")
+
+
+def doc_meta_keys(doc: dict[str, Any]) -> dict[str, str] | None:
+ meta: dict[str, str] = {}
+ extras = doc.get("extras")
+ if isinstance(extras, dict):
+ for key, value in extras.items():
+ if value is None:
+ continue
+ meta[str(key)] = str(value)
+ for key in _DOC_EXTRAS_KEYS:
+ value = doc.get(key)
+ if value is None:
+ continue
+ meta[str(key)] = str(value)
+ return meta or None
+
+
+def ingest_docs(
+ client: BrainAPIClient,
+ docs: list[dict[str, Any]],
+ *,
+ timeout_s: float,
+ skip_enrichment: bool = True,
+) -> dict[str, Any]:
+ tasks: list[dict[str, Any]] = []
+ for doc in docs:
+ text = str(doc.get("text") or "")
+ accepted = client.ingest_text(
+ text,
+ skip_enrichment=skip_enrichment,
+ meta_keys=doc_meta_keys(doc),
+ )
+ task_id = str((accepted.data or {}).get("task_id") or "")
+ if not task_id:
+ tasks.append(
+ {
+ "doc_id": doc.get("doc_id"),
+ "status": "failed",
+ "error": "ingest response missing task_id",
+ }
+ )
+ continue
+ waited = client.wait_for_task(
+ task_id,
+ timeout_s=timeout_s,
+ poll_interval_s=0.25 if skip_enrichment else 2.0,
+ )
+ status = str((waited.data or {}).get("status") or "unknown")
+ tasks.append(
+ {
+ "doc_id": doc.get("doc_id"),
+ "task_id": task_id,
+ "status": status,
+ }
+ )
+ statuses = [str(item.get("status")) for item in tasks]
+ if not tasks:
+ overall = "failed"
+ elif all(status == "completed" for status in statuses):
+ overall = "completed"
+ elif any(status == "completed" for status in statuses):
+ overall = "partial_failed"
+ else:
+ overall = "failed"
+ return {"status": overall, "n_docs": len(docs), "tasks": tasks}
+
+
+def ingest_triples(
+ client: BrainAPIClient,
+ triples: list[dict[str, Any]],
+ *,
+ timeout_s: float,
+ chunk_size: int = 20000,
+) -> dict[str, Any]:
+ if not triples:
+ return {"status": "skipped", "n_triples": 0, "tasks": []}
+ tasks: list[dict[str, Any]] = []
+ chunk = max(1, int(chunk_size))
+ for start in range(0, len(triples), chunk):
+ batch = triples[start : start + chunk]
+ submitted = client.ingest_structured(batch)
+ task_id = str((submitted.data or {}).get("task_id") or "")
+ if not task_id:
+ tasks.append(
+ {
+ "status": "failed",
+ "error": "structured ingest response missing task_id",
+ }
+ )
+ continue
+ waited = client.wait_for_task(task_id, timeout_s=timeout_s)
+ status = str((waited.data or {}).get("status") or "unknown")
+ tasks.append({"task_id": task_id, "status": status})
+ statuses = [str(item.get("status")) for item in tasks]
+ if not tasks:
+ overall = "failed"
+ elif all(status == "completed" for status in statuses):
+ overall = "completed"
+ elif any(status == "completed" for status in statuses):
+ overall = "partial_failed"
+ else:
+ overall = "failed"
+ return {"status": overall, "n_triples": len(triples), "tasks": tasks}
+
+
+def list_all_text_chunks(client: BrainAPIClient) -> list[dict[str, Any]]:
+ chunks: list[dict[str, Any]] = []
+ skip = 0
+ page = 100
+ total = None
+ while True:
+ result = client.list_text_chunks(limit=page, skip=skip)
+ payload = result.data or {}
+ batch = payload.get("data") or []
+ if not isinstance(batch, list):
+ break
+ chunks.extend([item for item in batch if isinstance(item, dict)])
+ total = payload.get("total")
+ skip += len(batch)
+ if not batch:
+ break
+ if total is not None and skip >= int(total):
+ break
+ return chunks
+
+
+def invert_doc_chunks(doc_to_chunks: dict[str, set[str]]) -> dict[str, str]:
+ chunk_to_doc: dict[str, str] = {}
+ for doc_id, chunks in (doc_to_chunks or {}).items():
+ for chunk_id in chunks or set():
+ chunk_to_doc[str(chunk_id)] = str(doc_id)
+ return chunk_to_doc
+
+
+def canonicalize_hit_ids(
+ ranked: list[str],
+ chunk_to_doc: dict[str, str],
+) -> list[str]:
+ out: list[str] = []
+ seen: set[str] = set()
+ for hid in ranked:
+ key = chunk_to_doc.get(str(hid), str(hid))
+ if not key or key in seen:
+ continue
+ seen.add(key)
+ out.append(key)
+ return out
+
+
+def unique_doc_counts(
+ raw_ids: list[str],
+ chunk_to_doc: dict[str, str] | None = None,
+ *,
+ k: int = 20,
+) -> tuple[int, int]:
+ head = [str(item) for item in raw_ids[:k] if item]
+ mapping = chunk_to_doc or {}
+ canonical = [mapping.get(item, item) for item in head]
+ return len(set(head)), len({item for item in canonical if item})
+
+
+def channel_id_lists(
+ payload: dict[str, Any] | None,
+ hits: list[dict[str, str]] | None = None,
+) -> dict[str, list[str]]:
+ raw = (payload or {}).get("channel_lists") or {}
+ if not isinstance(raw, dict):
+ raw = {}
+ out: dict[str, list[str]] = {
+ "dense_ids": [str(item) for item in (raw.get("dense") or []) if item],
+ "bm25_ids": [str(item) for item in (raw.get("bm25") or []) if item],
+ "entity_ids": [str(item) for item in (raw.get("entities") or []) if item],
+ "community_ids": [str(item) for item in (raw.get("communities") or []) if item],
+ "event_ids": [str(item) for item in (raw.get("events") or []) if item],
+ "neighbor_ids": [str(item) for item in (raw.get("neighbors") or []) if item],
+ "passage_ids": [],
+ }
+ by_channel: dict[str, list[str]] = {}
+ for hit in hits or []:
+ channel = str(hit.get("channel") or "")
+ hid = str(hit.get("id") or "")
+ if not channel or not hid:
+ continue
+ by_channel.setdefault(channel, []).append(hid)
+ out["passage_ids"] = list(by_channel.get("passages") or [])
+ if not out["entity_ids"]:
+ out["entity_ids"] = list(by_channel.get("entities") or [])
+ if not out["community_ids"]:
+ out["community_ids"] = list(by_channel.get("communities") or [])
+ if not out["event_ids"]:
+ out["event_ids"] = list(by_channel.get("events") or [])
+ if not out["neighbor_ids"]:
+ out["neighbor_ids"] = list(by_channel.get("neighbors") or [])
+ return out
+
+
+def gold_chunk_ids(
+ query: dict[str, Any],
+ doc_to_chunks: dict[str, set[str]],
+) -> set[str]:
+ gold: set[str] = set()
+ for doc_id in query.get("gold_doc_ids") or []:
+ gold.update(doc_to_chunks.get(str(doc_id)) or set())
+ for chunk_id in query.get("gold_chunk_ids") or []:
+ gold.add(str(chunk_id))
+ grades = query.get("gold_grades") or {}
+ if isinstance(grades, dict):
+ for doc_id, gain in grades.items():
+ if float(gain or 0) <= 0:
+ continue
+ gold.update(doc_to_chunks.get(str(doc_id)) or set())
+ return gold
+
+
+def gold_chunk_grades(
+ query: dict[str, Any],
+ doc_to_chunks: dict[str, set[str]],
+) -> dict[str, float]:
+ grades: dict[str, float] = {}
+ raw = query.get("gold_grades") or {}
+ if isinstance(raw, dict) and raw:
+ for doc_id, gain in raw.items():
+ value = float(gain or 0)
+ if value <= 0:
+ continue
+ for chunk_id in doc_to_chunks.get(str(doc_id)) or set():
+ grades[chunk_id] = max(grades.get(chunk_id, 0.0), value)
+ return grades
+ for chunk_id in gold_chunk_ids(query, doc_to_chunks):
+ grades[chunk_id] = 1.0
+ return grades
+
+
+def gold_hit_ids(
+ query: dict[str, Any],
+ doc_to_chunks: dict[str, set[str]],
+) -> set[str]:
+ gold: set[str] = set()
+ for doc_id in query.get("gold_doc_ids") or []:
+ gold.add(str(doc_id))
+ grades = query.get("gold_grades") or {}
+ if isinstance(grades, dict):
+ for doc_id, gain in grades.items():
+ if float(gain or 0) <= 0:
+ continue
+ gold.add(str(doc_id))
+ chunk_to_doc = invert_doc_chunks(doc_to_chunks)
+ for chunk_id in query.get("gold_chunk_ids") or []:
+ gold.add(chunk_to_doc.get(str(chunk_id), str(chunk_id)))
+ if gold:
+ return gold
+ return {
+ chunk_to_doc.get(chunk_id, chunk_id)
+ for chunk_id in gold_chunk_ids(query, doc_to_chunks)
+ }
+
+
+def gold_hit_grades(
+ query: dict[str, Any],
+ doc_to_chunks: dict[str, set[str]],
+) -> dict[str, float]:
+ grades: dict[str, float] = {}
+ raw = query.get("gold_grades") or {}
+ if isinstance(raw, dict) and raw:
+ for doc_id, gain in raw.items():
+ value = float(gain or 0)
+ if value <= 0:
+ continue
+ key = str(doc_id)
+ grades[key] = max(grades.get(key, 0.0), value)
+ return grades
+ for doc_id in gold_hit_ids(query, doc_to_chunks):
+ grades[str(doc_id)] = 1.0
+ return grades
+
+
+def candidate_pool_ids(query: dict[str, Any]) -> list[str]:
+ raw = query.get("candidate_doc_ids") or []
+ if raw:
+ return [str(item) for item in raw]
+ return [str(item) for item in (query.get("gold_doc_ids") or [])]
+
+
+def candidate_pool_grades(query: dict[str, Any]) -> dict[str, float]:
+ raw = query.get("candidate_grades") or {}
+ if isinstance(raw, dict) and raw:
+ return {str(doc_id): float(gain) for doc_id, gain in raw.items()}
+ grades = {
+ str(doc_id): float(gain)
+ for doc_id, gain in (query.get("gold_grades") or {}).items()
+ }
+ for doc_id in query.get("gold_doc_ids") or []:
+ grades.setdefault(str(doc_id), 1.0)
+ return grades
+
+
+def ingested_doc_ids(doc_to_chunks: dict[str, set[str]]) -> set[str]:
+ return {
+ str(doc_id)
+ for doc_id, chunks in (doc_to_chunks or {}).items()
+ if chunks
+ }
+
+
+def filter_ranked_to_pool(
+ ranked: list[str],
+ pool: list[str],
+) -> list[str]:
+ allowed = {str(item) for item in pool}
+ return [item for item in ranked if str(item) in allowed]
+
+
+def score_search_result(
+ result: TimedResult,
+ *,
+ gold: set[str],
+ ks: tuple[int, ...],
+ grades: dict[str, float] | None = None,
+ chunk_to_doc: dict[str, str] | None = None,
+ pool_ids: list[str] | None = None,
+) -> dict[str, Any]:
+ payload = result.data if isinstance(result.data, dict) else {}
+ ranked = _hit_ids(payload)
+ hits = _hit_dump(payload, chunk_to_doc)
+ n_unique_raw, n_unique_canonical = unique_doc_counts(
+ ranked, chunk_to_doc, k=20
+ )
+ unique_k = max(ks) if ks else 20
+ n_retrieve_raw, n_retrieve_canonical = unique_doc_counts(
+ ranked, chunk_to_doc, k=unique_k
+ )
+ lists = channel_id_lists(payload, hits)
+ if chunk_to_doc:
+ ranked = canonicalize_hit_ids(ranked, chunk_to_doc)
+ if pool_ids:
+ ranked = filter_ranked_to_pool(ranked, pool_ids)
+ retrieve_ms, embed_ms = retrieve_latency_ms(
+ payload.get("stage_timings") if isinstance(payload, dict) else None,
+ result.latency_ms,
+ )
+ ndcg_ks = tuple(dict.fromkeys((*ks, 10, 20)))
+ metrics = {f"recall@{cut}": recall_at_k(ranked, gold, cut) for cut in ks}
+ for cut in ndcg_ks:
+ metrics[f"ndcg@{cut}"] = ndcg_at_k(ranked, gold, cut, grades=grades)
+ full_k = max(len(ranked), 1)
+ metrics["ndcg"] = ndcg_at_k(ranked, gold, full_k, grades=grades)
+ metrics["mrr"] = mrr(ranked, gold)
+ return {
+ "hit_ids": ranked,
+ "hits": hits,
+ "metrics": metrics,
+ "retrieve_ms": retrieve_ms,
+ "embed_ms": embed_ms,
+ "client_wall_ms": result.latency_ms,
+ "n_hits": len(ranked),
+ "n_unique_docs_raw": n_unique_raw,
+ "n_unique_docs_canonical": n_unique_canonical,
+ "n_unique_docs_retrieve_raw": n_retrieve_raw,
+ "n_unique_docs_retrieve_canonical": n_retrieve_canonical,
+ "unique_docs_k": unique_k,
+ "facets": payload.get("facets") if isinstance(payload, dict) else None,
+ "response_node_ids": (
+ list(payload.get("node_ids") or []) if isinstance(payload, dict) else []
+ ),
+ **lists,
+ }
+
+
+WANDSGRAPH_BRAIN = "searchbenchwandsgraph"
+
+
+def assert_wandsgraph_node_join(
+ client: BrainAPIClient,
+ result: dict[str, Any],
+ *,
+ neighbor_limit: int = 5,
+) -> dict[str, Any]:
+ if client.settings.brain_id != WANDSGRAPH_BRAIN:
+ return {"skipped": True, "brain_id": client.settings.brain_id}
+ queries = result.get("queries") or []
+ if not queries:
+ raise SystemExit("searchbenchwandsgraph node_id join needs at least one query")
+ first = queries[0]
+ hits = first.get("hits") or []
+ matched = [
+ hit
+ for hit in hits
+ if isinstance(hit, dict)
+ and hit.get("node_id")
+ and str(hit.get("node_id")) == str(hit.get("doc_id") or "")
+ ]
+ if not matched:
+ raise SystemExit(
+ "searchbenchwandsgraph passage hits missing node_id matching DOCID/doc_id. "
+ f"sample={hits[:3]!r}"
+ )
+ node_id = str(matched[0]["node_id"])
+ timed = client.get_neighbors(node_id, limit=neighbor_limit)
+ if timed.status_code != 200:
+ raise SystemExit(
+ f"GET /retrieve/entities/neighbors?uuid={node_id} returned {timed.status_code}"
+ )
+ payload = timed.data if isinstance(timed.data, dict) else {}
+ return {
+ "skipped": False,
+ "node_id": node_id,
+ "status_code": timed.status_code,
+ "neighbor_count": payload.get("count"),
+ "qid": first.get("qid"),
+ }
+
+
+def evaluate_search(
+ client: BrainAPIClient,
+ rows: list[dict[str, Any]],
+ *,
+ ks: tuple[int, ...] = (5, 10, 20),
+ timeout_s: float = 600.0,
+ k: int = 20,
+ fusion: str | None = None,
+ fusion_alpha: float | None = None,
+ rerank: str | None = None,
+ mode: str | None = None,
+ channels: list[str] | None = None,
+ node_labels: list[str] | None = None,
+ community_labels: list[str] | None = None,
+ expand: str | None = None,
+ dataset_name: str = "search_toy.jsonl",
+ limit_docs: int | None = None,
+ limit_queries: int | None = None,
+ skip_enrichment: bool = True,
+ ingest_graph: bool = False,
+ skip_ingest: bool = False,
+ rank_pool: bool = False,
+ personalize: bool = False,
+ interactions: list[dict[str, Any]] | None = None,
+ extras: dict[str, str] | None = None,
+) -> dict[str, Any]:
+ brain_id = client.settings.brain_id
+ if (ingest_graph or interactions) and brain_id in FROZEN_STRUCTURED_BRAINS:
+ raise SystemExit(
+ f"Refusing structured graph/interaction ingest on frozen brain {brain_id}. "
+ "Use searchbenchwandsgraph (or another non-frozen searchbench*)."
+ )
+ docs, queries = split_corpus(rows)
+ if limit_docs is not None:
+ docs = docs[:limit_docs]
+ if limit_queries is not None:
+ queries = queries[:limit_queries]
+
+ if skip_ingest:
+ ingest = {
+ "status": "completed",
+ "n_docs": len(docs),
+ "tasks": [],
+ "reused": True,
+ }
+ else:
+ ingest = ingest_docs(
+ client,
+ docs,
+ timeout_s=timeout_s,
+ skip_enrichment=skip_enrichment,
+ )
+ graph_ingest: dict[str, Any] = {"status": "skipped", "n_triples": 0, "tasks": []}
+ if ingest_graph and docs:
+ graph_ingest = ingest_triples(
+ client,
+ docs_to_triples(docs),
+ timeout_s=timeout_s,
+ )
+ interaction_ingest: dict[str, Any] = {
+ "status": "skipped",
+ "n_triples": 0,
+ "tasks": [],
+ }
+ if interactions:
+ interaction_ingest = ingest_triples(
+ client,
+ interactions_to_triples(interactions),
+ timeout_s=timeout_s,
+ )
+ chunks = list_all_text_chunks(client) if ingest["status"] != "failed" else []
+ doc_to_chunks = map_doc_ids_to_chunks(docs, chunks)
+ mapped = sum(1 for ids in doc_to_chunks.values() if ids)
+ chunk_to_doc = invert_doc_chunks(doc_to_chunks)
+
+ per_query: list[dict[str, Any]] = []
+ search_error: str | None = None
+ event_probe: dict[str, Any] | None = None
+ for query in queries:
+ gold = gold_hit_ids(query, doc_to_chunks)
+ pool = candidate_pool_ids(query) if rank_pool else []
+ grades = (
+ candidate_pool_grades(query)
+ if rank_pool
+ else gold_hit_grades(query, doc_to_chunks)
+ )
+ target = None
+ if personalize:
+ raw_target = str(query.get("target") or "").strip()
+ if raw_target:
+ target = raw_target
+ try:
+ result = client.search(
+ str(query.get("query") or ""),
+ k=k,
+ fusion=fusion,
+ fusion_alpha=fusion_alpha,
+ rerank=rerank,
+ mode=mode,
+ channels=channels,
+ node_labels=node_labels,
+ community_labels=community_labels,
+ expand=expand,
+ extras=extras,
+ target=target,
+ profile_stages=True,
+ )
+ except SearchDisabledError as exc:
+ search_error = str(exc)
+ break
+ except RuntimeError as exc:
+ search_error = str(exc)
+ break
+ scored = score_search_result(
+ result,
+ gold=gold,
+ ks=ks,
+ grades=grades,
+ chunk_to_doc=chunk_to_doc,
+ pool_ids=pool or None,
+ )
+ ingested = ingested_doc_ids(doc_to_chunks)
+ missing_brain = [pid for pid in pool if pid not in ingested] if pool else []
+ retrieved_in_pool = list(scored.get("hit_ids") or [])
+ coverage = (
+ (len(retrieved_in_pool) / len(pool)) if pool else None
+ )
+ per_query.append(
+ {
+ "qid": query.get("qid"),
+ "query": query.get("query"),
+ "slice": query.get("slice") or "unspecified",
+ "gold_doc_ids": list(query.get("gold_doc_ids") or []),
+ "gold_grades": {
+ str(doc_id): float(gain)
+ for doc_id, gain in grades.items()
+ if float(gain or 0) > 0
+ },
+ "gold_chunk_ids": sorted(gold),
+ "candidate_doc_ids": pool,
+ "pool_size": len(pool) if rank_pool else None,
+ "pool_coverage": coverage,
+ "missing_from_brain": missing_brain,
+ **scored,
+ }
+ )
+
+ if not queries and interactions and search_error is None:
+ names: list[str] = []
+ seen: set[str] = set()
+ for row in interactions:
+ behavior = str(row.get("behavior") or "").strip()
+ if behavior and behavior not in seen:
+ seen.add(behavior)
+ names.append(behavior)
+ hit_counts: list[dict[str, Any]] = []
+ for name in names[:5]:
+ try:
+ result = client.search(
+ name,
+ k=k,
+ fusion=fusion,
+ rerank=rerank,
+ mode=mode,
+ channels=channels or ["events"],
+ expand=expand,
+ profile_stages=True,
+ )
+ except SearchDisabledError as exc:
+ search_error = str(exc)
+ break
+ except RuntimeError as exc:
+ search_error = str(exc)
+ break
+ payload = result.data if isinstance(result.data, dict) else {}
+ hits = payload.get("hits") or []
+ hit_counts.append({"query": name, "n_hits": len(hits)})
+ event_probe = {
+ "n_event_queries": len(hit_counts),
+ "queries": hit_counts,
+ "n_hits": sum(int(item.get("n_hits") or 0) for item in hit_counts),
+ }
+
+ metrics = aggregate_query_metrics(per_query, ks=ks) if per_query else {}
+ if per_query and any(
+ row.get("n_unique_docs_canonical") is not None for row in per_query
+ ):
+ n_raw = [
+ float(row.get("n_unique_docs_raw") or 0) for row in per_query
+ ]
+ n_canon = [
+ float(row.get("n_unique_docs_canonical") or 0) for row in per_query
+ ]
+ metrics["unique_docs@20_raw"] = sum(n_raw) / len(n_raw)
+ metrics["unique_docs@20_canonical"] = sum(n_canon) / len(n_canon)
+ retrieve_k = int(per_query[0].get("unique_docs_k") or 20)
+ n_ret = [
+ float(row.get("n_unique_docs_retrieve_canonical") or 0)
+ for row in per_query
+ ]
+ metrics[f"unique_docs@{retrieve_k}_canonical"] = sum(n_ret) / len(n_ret)
+ if rank_pool and per_query:
+ coverages = [
+ float(row["pool_coverage"])
+ for row in per_query
+ if row.get("pool_coverage") is not None
+ ]
+ if coverages:
+ metrics["pool_coverage"] = sum(coverages) / len(coverages)
+ metrics["missing_from_brain"] = sum(
+ len(row.get("missing_from_brain") or []) for row in per_query
+ ) / len(per_query)
+ status = "ok"
+ if docs and ingest["status"] not in {"completed", "partial_failed"}:
+ status = "failed"
+ if docs and mapped < 1:
+ status = "failed"
+ if queries and not per_query:
+ status = "failed"
+ if not queries and interactions:
+ if event_probe is None or int(event_probe.get("n_hits") or 0) < 1:
+ status = "failed"
+ elif search_error is None:
+ status = "ok"
+ if search_error:
+ status = "failed"
+
+ protocol = None
+ if brain_id == "searchbenchwandsgraph":
+ protocol = (
+ "architecture demo: isolated catalog graph on searchbenchwandsgraph; "
+ "not a quality-default claim vs frozen WANDS passages 0.823 or ESCI 0.500"
+ )
+ if brain_id == "searchbenchjdslice":
+ protocol = (
+ "architecture demo: JDsearch rank-pool slice on searchbenchjdslice; "
+ "not TEM 0.219; not frozen WANDS 0.823 or ESCI 0.500"
+ )
+
+ return {
+ "status": status,
+ "brain_id": client.settings.brain_id,
+ "dataset": dataset_name,
+ "protocol": protocol,
+ "fusion": fusion or "rrf",
+ "fusion_alpha": fusion_alpha,
+ "rerank": rerank or "none",
+ "mode": mode or "default",
+ "channels": channels or ["passages"],
+ "expand": expand or "none",
+ "k": k,
+ "ks": list(ks),
+ "n_docs": len(docs),
+ "n_queries": len(per_query),
+ "n_docs_mapped": mapped,
+ "skip_enrichment": skip_enrichment,
+ "ingest_graph": ingest_graph,
+ "skip_ingest": skip_ingest,
+ "rank_pool": rank_pool,
+ "personalize": personalize,
+ "extras": extras,
+ "ingest": ingest,
+ "graph_ingest": graph_ingest,
+ "interaction_ingest": interaction_ingest,
+ "event_probe": event_probe,
+ "search_error": search_error,
+ "metrics": metrics,
+ "queries": per_query,
+ }
diff --git a/benchmarks/search/export_hybrid_lists.py b/benchmarks/search/export_hybrid_lists.py
new file mode 100644
index 0000000..ff50c08
--- /dev/null
+++ b/benchmarks/search/export_hybrid_lists.py
@@ -0,0 +1,158 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+from search.dataset import load_records, split_corpus
+from search.finetune_esci_4class import LABELS, LABEL_TO_ID, held_out_query_ids
+from search.mine_retrieved_lists import is_held_out, write_lists
+
+PROTECTED_OUT_NAMES = {
+ "esci_retrieved_lists.jsonl",
+ "search_esci.jsonl",
+ "search_esci_74.jsonl",
+}
+
+
+def label_from_gain(gain: float) -> str:
+ value = float(gain or 0.0)
+ if value >= 0.99:
+ return "E"
+ if value >= 0.05:
+ return "S"
+ if value > 0.0:
+ return "C"
+ return "I"
+
+
+def _chunk_to_doc(row: dict[str, Any]) -> dict[str, str]:
+ mapping: dict[str, str] = {}
+ for hit in row.get("hits") or []:
+ chunk_id = str(hit.get("id") or "")
+ doc_id = str(hit.get("doc_id") or "")
+ if chunk_id and doc_id:
+ mapping[chunk_id] = doc_id
+ return mapping
+
+
+def _as_docs(ids: list[str], mapping: dict[str, str]) -> list[str]:
+ out: list[str] = []
+ seen: set[str] = set()
+ for raw in ids:
+ key = mapping.get(str(raw), str(raw))
+ if not key or key in seen:
+ continue
+ seen.add(key)
+ out.append(key)
+ return out
+
+
+def _queries_by_qid(queries: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
+ out: dict[str, dict[str, Any]] = {}
+ for row in queries:
+ qid = str(row.get("qid") or "")
+ if qid:
+ out[qid] = row
+ return out
+
+
+def _passage_for(doc: dict[str, Any] | None, pid: str) -> str:
+ if not doc:
+ return pid
+ text = str(doc.get("text") or "").strip()
+ if text:
+ return text
+ title = str(doc.get("title") or "").strip()
+ if title:
+ return f"Title: {title}"
+ return pid
+
+
+def rows_from_eval(
+ eval_result: dict[str, Any],
+ *,
+ docs: list[dict[str, Any]],
+ queries: list[dict[str, Any]],
+ holdout: set[str],
+ k: int = 50,
+) -> list[dict[str, Any]]:
+ docs_by_id = {
+ str(doc.get("doc_id") or ""): doc
+ for doc in docs
+ if str(doc.get("doc_id") or "")
+ }
+ query_rows = _queries_by_qid(queries)
+ cut = max(1, int(k))
+ rows: list[dict[str, Any]] = []
+ for row in eval_result.get("queries") or []:
+ qid = str(row.get("qid") or "")
+ if not qid or is_held_out(qid, holdout):
+ continue
+ source = query_rows.get(qid) or row
+ query = str(source.get("query") or row.get("query") or "")
+ grades = source.get("gold_grades") or row.get("gold_grades") or {}
+ if not isinstance(grades, dict):
+ grades = {}
+ mapping = _chunk_to_doc(row)
+ hit_ids = [str(item) for item in (row.get("hit_ids") or []) if item]
+ pids = _as_docs(hit_ids, mapping)[:cut]
+ for pid in pids:
+ gain = float(grades.get(pid) or 0.0)
+ label = label_from_gain(gain)
+ if label not in LABEL_TO_ID:
+ label = "I"
+ rows.append(
+ {
+ "query_id": qid,
+ "query": query,
+ "product_id": pid,
+ "label": label,
+ "passage": _passage_for(docs_by_id.get(pid), pid),
+ }
+ )
+ return rows
+
+
+def export_hybrid_lists(
+ *,
+ eval_result: dict[str, Any],
+ dataset_path: Path,
+ holdout_path: Path,
+ out_path: Path,
+ k: int = 50,
+) -> dict[str, Any]:
+ name = out_path.name
+ if name in PROTECTED_OUT_NAMES:
+ raise ValueError(f"refusing to overwrite {out_path}")
+ holdout = held_out_query_ids(holdout_path)
+ docs, queries = split_corpus(load_records(dataset_path))
+ rows = rows_from_eval(
+ eval_result,
+ docs=docs,
+ queries=queries,
+ holdout=holdout,
+ k=k,
+ )
+ write_lists(rows, out_path)
+ counts = {label: 0 for label in LABELS}
+ by_q: dict[str, list[str]] = {}
+ for row in rows:
+ counts[str(row["label"])] = counts.get(str(row["label"]), 0) + 1
+ by_q.setdefault(str(row["query_id"]), []).append(str(row["label"]))
+ n_with_i = sum(1 for labels in by_q.values() if "I" in labels)
+ meta = {
+ "n_queries": len(by_q),
+ "n_pairs": len(rows),
+ "n_holdout_qids": len(holdout),
+ "k": int(k),
+ "label_counts": counts,
+ "n_lists_with_i": n_with_i,
+ "source": "hybrid-k50",
+ "out": str(out_path),
+ "dataset": str(dataset_path),
+ "holdout": str(holdout_path),
+ }
+ meta_path = out_path.with_name(out_path.stem + ".meta.json")
+ meta_path.write_text(json.dumps(meta, indent=2) + "\n", encoding="utf-8")
+ return meta
diff --git a/benchmarks/search/finetune_esci_4class.py b/benchmarks/search/finetune_esci_4class.py
new file mode 100644
index 0000000..4fa11c0
--- /dev/null
+++ b/benchmarks/search/finetune_esci_4class.py
@@ -0,0 +1,389 @@
+from __future__ import annotations
+
+import argparse
+import json
+import random
+from collections import Counter
+from pathlib import Path
+from typing import Any, Sequence
+
+from search.catalog import (
+ ESCI_CACHE_DIR,
+ ESCI_EXAMPLES_FILE,
+ ESCI_PRODUCTS_FILE,
+ extra_fields_from_catalog,
+ format_product_text,
+ _esci_products,
+ _iter_parquet,
+ _stringify_catalog_value,
+)
+from search.dataset import load_records, split_corpus
+
+LABELS = ("E", "S", "C", "I")
+LABEL_TO_ID = {label: index for index, label in enumerate(LABELS)}
+CLASS_GAINS = (1.0, 0.1, 0.01, 0.0)
+DEFAULT_MODEL = "cross-encoder/ms-marco-MiniLM-L-12-v2"
+DEFAULT_OUT = Path("data/models/esci-minilm-l12-4class")
+
+
+def pick_device() -> str:
+ import torch
+
+ if torch.backends.mps.is_available():
+ return "mps"
+ if torch.cuda.is_available():
+ return "cuda"
+ return "cpu"
+
+
+def held_out_query_ids(jsonl_path: Path) -> set[str]:
+ _, queries = split_corpus(load_records(jsonl_path))
+ out: set[str] = set()
+ for row in queries:
+ qid = str(row.get("qid") or "")
+ if qid.lower().startswith("esci-"):
+ out.add(qid[5:])
+ out.add(qid)
+ return out
+
+
+def product_passage(catalog: dict[str, Any], extra_fields: Sequence[tuple[str, str]]) -> str:
+ pid = str(catalog.get("product_id") or "unknown")
+ title = _stringify_catalog_value(catalog.get("product_title")) or pid
+ extras = [
+ (label, _stringify_catalog_value(catalog.get(field)))
+ for label, field in extra_fields
+ ]
+ return format_product_text(
+ pid,
+ title=title,
+ description=_stringify_catalog_value(catalog.get("product_description")),
+ extras=extras,
+ )
+
+
+def weighted_scores(
+ class_probs: Sequence[Sequence[float]] | Sequence[float],
+ gains: Sequence[float] = CLASS_GAINS,
+) -> list[float]:
+ rows = list(class_probs)
+ if not rows:
+ return []
+ first = rows[0]
+ if isinstance(first, (int, float)):
+ rows = [rows]
+ gain = [float(value) for value in gains]
+ scores: list[float] = []
+ for row in rows:
+ total = 0.0
+ for index, value in enumerate(row):
+ if index >= len(gain):
+ break
+ total += float(value) * gain[index]
+ scores.append(total)
+ return scores
+
+
+def rank_doc_ids(doc_ids: Sequence[str], scores: Sequence[float]) -> list[str]:
+ indexed = list(range(len(doc_ids)))
+ indexed.sort(key=lambda index: (-float(scores[index]), str(doc_ids[index])))
+ return [str(doc_ids[index]) for index in indexed]
+
+
+def persist_tokenizer_max_length(model: Any, out_dir: Path) -> None:
+ max_length = getattr(model, "max_length", None)
+ tokenizer = getattr(model, "tokenizer", None)
+ if not max_length or tokenizer is None:
+ return
+ tokenizer.model_max_length = int(max_length)
+ tokenizer.save_pretrained(str(out_dir))
+
+
+def class_weights(counts: Sequence[int]) -> list[float]:
+ values = [max(1, int(count)) for count in counts]
+ total = float(sum(values))
+ n_labels = float(len(values))
+ return [total / (n_labels * float(count)) for count in values]
+
+
+def load_train_rows(
+ *,
+ holdout: set[str],
+ locale: str = "us",
+ max_pairs: int = 180000,
+ seed: int = 11,
+) -> list[dict[str, Any]]:
+ examples_path = ESCI_CACHE_DIR / ESCI_EXAMPLES_FILE
+ wanted_locale = locale.strip().lower()
+ rows: list[dict[str, Any]] = []
+ for item in _iter_parquet(examples_path):
+ if int(item.get("small_version") or 0) != 1:
+ continue
+ if str(item.get("product_locale") or "").strip().lower() != wanted_locale:
+ continue
+ if str(item.get("split") or "").strip().lower() != "train":
+ continue
+ qid = str(item.get("query_id") or "").strip()
+ if not qid or qid in holdout:
+ continue
+ pid = str(item.get("product_id") or "").strip()
+ query = str(item.get("query") or "").strip()
+ label = str(item.get("esci_label") or "").strip().upper()
+ if not pid or not query or label not in LABEL_TO_ID:
+ continue
+ rows.append(
+ {
+ "query": query,
+ "product_id": pid,
+ "label_id": LABEL_TO_ID[label],
+ "label": label,
+ }
+ )
+ rng = random.Random(seed)
+ rng.shuffle(rows)
+ return rows[: max(1, max_pairs)]
+
+
+def load_retrieved_rows(path: Path) -> list[dict[str, Any]]:
+ rows: list[dict[str, Any]] = []
+ with path.open(encoding="utf-8") as fh:
+ for line in fh:
+ raw = line.strip()
+ if not raw:
+ continue
+ item = json.loads(raw)
+ label = str(item.get("label") or "").strip().upper()
+ query = str(item.get("query") or "").strip()
+ pid = str(item.get("product_id") or "").strip()
+ if not query or not pid or label not in LABEL_TO_ID:
+ continue
+ row = {
+ "query": query,
+ "product_id": pid,
+ "label": label,
+ "label_id": LABEL_TO_ID[label],
+ }
+ passage = str(item.get("passage") or "").strip()
+ if passage:
+ row["passage"] = passage
+ rows.append(row)
+ return rows
+
+
+def attach_passages(
+ rows: list[dict[str, Any]],
+ *,
+ locale: str = "us",
+) -> list[tuple[str, str, int]]:
+ products_path = ESCI_CACHE_DIR / ESCI_PRODUCTS_FILE
+ needed = {str(row["product_id"]) for row in rows}
+ products = _esci_products(products_path, needed, locale=locale)
+ extra_fields = extra_fields_from_catalog(
+ products,
+ title_key="product_title",
+ description_key="product_description",
+ )
+ pairs: list[tuple[str, str, int]] = []
+ for row in rows:
+ catalog = products.get(str(row["product_id"])) or {"product_id": row["product_id"]}
+ passage = product_passage(catalog, extra_fields)
+ pairs.append((str(row["query"]), passage, int(row["label_id"])))
+ return pairs
+
+
+def train_loop(
+ model,
+ loader,
+ *,
+ epochs: int,
+ loss_fct,
+ warmup_steps: int,
+ lr: float,
+ out_dir: Path,
+ ckpt_every: int = 200,
+) -> int:
+ import torch
+ from torch.optim import AdamW
+ from transformers import get_linear_schedule_with_warmup
+
+ loader.collate_fn = model.smart_batching_collate
+ model.model.to(model._target_device)
+ optimizer = AdamW(model.model.parameters(), lr=lr)
+ total_steps = max(1, len(loader) * epochs)
+ scheduler = get_linear_schedule_with_warmup(
+ optimizer,
+ num_warmup_steps=warmup_steps,
+ num_training_steps=total_steps,
+ )
+ out_dir.mkdir(parents=True, exist_ok=True)
+ global_step = 0
+ model.model.train()
+ for _epoch in range(epochs):
+ for features, labels in loader:
+ optimizer.zero_grad(set_to_none=True)
+ logits = model.model(**features).logits
+ loss = loss_fct(logits, labels)
+ loss.backward()
+ torch.nn.utils.clip_grad_norm_(model.model.parameters(), 1.0)
+ optimizer.step()
+ scheduler.step()
+ global_step += 1
+ if global_step % 50 == 0 or global_step == total_steps:
+ print(
+ f"step {global_step}/{total_steps} loss={float(loss.detach().cpu()):.4f}",
+ flush=True,
+ )
+ if global_step % ckpt_every == 0:
+ model.save(str(out_dir))
+ persist_tokenizer_max_length(model, out_dir)
+ (out_dir / "train_step.json").write_text(
+ json.dumps(
+ {
+ "global_step": global_step,
+ "loss": float(loss.detach().cpu()),
+ }
+ )
+ + "\n",
+ encoding="utf-8",
+ )
+ print(f"checkpoint {out_dir} step={global_step}", flush=True)
+ model.save(str(out_dir))
+ persist_tokenizer_max_length(model, out_dir)
+ return global_step
+
+
+def finetune(
+ *,
+ jsonl_path: Path,
+ out_dir: Path,
+ max_pairs: int = 80000,
+ epochs: int = 1,
+ batch_size: int = 32,
+ max_length: int = 192,
+ base_model: str = DEFAULT_MODEL,
+ seed: int = 11,
+ ckpt_every: int = 200,
+ use_class_weights: bool = False,
+ lists_path: Path | None = None,
+ lists_source: str | None = None,
+) -> dict[str, Any]:
+ import torch
+ from sentence_transformers import CrossEncoder, InputExample
+ from torch.utils.data import DataLoader
+
+ holdout = held_out_query_ids(jsonl_path)
+ source = "pool"
+ if lists_path is not None:
+ source = str(lists_source or "").strip() or "retrieved-bm25"
+ rows = load_retrieved_rows(lists_path)
+ rng = random.Random(seed)
+ rng.shuffle(rows)
+ rows = rows[: max(1, max_pairs)]
+ if rows and all(str(row.get("passage") or "").strip() for row in rows):
+ pairs = [
+ (str(row["query"]), str(row["passage"]), int(row["label_id"]))
+ for row in rows
+ ]
+ else:
+ pairs = attach_passages(rows)
+ else:
+ rows = load_train_rows(holdout=holdout, max_pairs=max_pairs, seed=seed)
+ pairs = attach_passages(rows)
+ label_counts = Counter(label_id for _, _, label_id in pairs)
+ counts = [int(label_counts.get(index, 0)) for index in range(len(LABELS))]
+ weights = class_weights(counts) if use_class_weights else [1.0] * len(LABELS)
+ examples = [
+ InputExample(texts=[query, passage], label=label_id)
+ for query, passage, label_id in pairs
+ ]
+ device = pick_device()
+ model = CrossEncoder(
+ base_model,
+ num_labels=len(LABELS),
+ max_length=max_length,
+ device=device,
+ automodel_args={"ignore_mismatched_sizes": True},
+ default_activation_function=torch.nn.Identity(),
+ )
+ loader = DataLoader(examples, shuffle=True, batch_size=batch_size)
+ warmup = max(10, len(loader) // 10)
+ loss_fct = torch.nn.CrossEntropyLoss(
+ weight=torch.tensor(weights, dtype=torch.float32, device=model._target_device),
+ label_smoothing=0.1,
+ )
+ steps = train_loop(
+ model,
+ loader,
+ epochs=epochs,
+ loss_fct=loss_fct,
+ warmup_steps=warmup,
+ lr=2e-5,
+ out_dir=out_dir,
+ ckpt_every=ckpt_every,
+ )
+ meta = {
+ "n_pairs": len(pairs),
+ "n_steps": steps,
+ "label_counts": {LABELS[index]: counts[index] for index in range(len(LABELS))},
+ "class_weights": {LABELS[index]: weights[index] for index in range(len(LABELS))},
+ "class_gains": {LABELS[index]: CLASS_GAINS[index] for index in range(len(LABELS))},
+ "n_holdout_qids": len(holdout),
+ "epochs": epochs,
+ "batch_size": batch_size,
+ "max_length": max_length,
+ "model": str(out_dir),
+ "base": base_model,
+ "device": device,
+ "loss": "4-class-ce-label-smoothing-0.1",
+ "use_class_weights": use_class_weights,
+ "score": "1.0*P(E)+0.1*P(S)+0.01*P(C)+0.0*P(I)",
+ "source": source,
+ "lists_path": str(lists_path) if lists_path is not None else None,
+ }
+ payload = json.dumps(meta, indent=2) + "\n"
+ (out_dir / "train_meta.json").write_text(payload, encoding="utf-8")
+ (out_dir / "finetune_meta.json").write_text(payload, encoding="utf-8")
+ return meta
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(
+ description="Fine-tune a 4-class ESCI cross-encoder (not Exact→1 MiniLM-L-6)"
+ )
+ parser.add_argument("--dataset", type=Path, required=True)
+ parser.add_argument("--out", type=Path, default=DEFAULT_OUT)
+ parser.add_argument("--max-pairs", type=int, default=80000)
+ parser.add_argument("--epochs", type=int, default=1)
+ parser.add_argument("--batch-size", type=int, default=32)
+ parser.add_argument("--max-length", type=int, default=192)
+ parser.add_argument("--ckpt-every", type=int, default=200)
+ parser.add_argument("--class-weights", action="store_true")
+ parser.add_argument("--base", default=DEFAULT_MODEL)
+ parser.add_argument("--seed", type=int, default=11)
+ parser.add_argument("--from-lists", type=Path, default=None)
+ parser.add_argument("--lists-source", default="")
+ args = parser.parse_args(argv)
+ meta = finetune(
+ jsonl_path=args.dataset,
+ out_dir=args.out,
+ max_pairs=args.max_pairs,
+ epochs=args.epochs,
+ batch_size=args.batch_size,
+ max_length=args.max_length,
+ base_model=args.base,
+ seed=args.seed,
+ ckpt_every=args.ckpt_every,
+ use_class_weights=bool(args.class_weights),
+ lists_path=args.from_lists,
+ lists_source=(
+ str(args.lists_source).strip() or "retrieved-bm25"
+ if args.from_lists is not None
+ else None
+ ),
+ )
+ print(json.dumps(meta, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/search/finetune_esci_ce.py b/benchmarks/search/finetune_esci_ce.py
new file mode 100644
index 0000000..8dfebef
--- /dev/null
+++ b/benchmarks/search/finetune_esci_ce.py
@@ -0,0 +1,269 @@
+from __future__ import annotations
+
+import argparse
+import json
+import random
+from pathlib import Path
+from typing import Any
+
+from search.catalog import (
+ ESCI_CACHE_DIR,
+ ESCI_EXAMPLES_FILE,
+ ESCI_GAINS,
+ ESCI_PRODUCTS_FILE,
+ PRODUCT_TEXT_MAX_CHARS,
+ _esci_products,
+ _iter_parquet,
+ _stringify_catalog_value,
+)
+from search.dataset import load_records, split_corpus
+
+DEFAULT_MODEL = "cross-encoder/ms-marco-MiniLM-L-6-v2"
+REDDY_MODEL = "cross-encoder/ms-marco-MiniLM-L-12-v2"
+CLASS_ORDER = ("E", "S", "C", "I")
+CLASS_TO_INDEX = {label: index for index, label in enumerate(CLASS_ORDER)}
+
+
+def held_out_query_ids(jsonl_path: Path) -> set[str]:
+ _, queries = split_corpus(load_records(jsonl_path))
+ out: set[str] = set()
+ for row in queries:
+ qid = str(row.get("qid") or "")
+ if qid.lower().startswith("esci-"):
+ out.add(qid[5:])
+ out.add(qid)
+ return out
+
+
+def product_passage(catalog: dict[str, Any], *, fields: str) -> str:
+ title = _stringify_catalog_value(catalog.get("product_title")) or ""
+ if fields == "title":
+ return title
+ parts: list[str] = []
+ if title:
+ parts.append(title)
+ brand = _stringify_catalog_value(catalog.get("product_brand"))
+ if brand:
+ parts.append(f"Brand: {brand}")
+ color = _stringify_catalog_value(catalog.get("product_color"))
+ if color:
+ parts.append(f"Color: {color}")
+ bullets = _stringify_catalog_value(catalog.get("product_bullet_point"))
+ if bullets:
+ parts.append(f"Bullets: {bullets}")
+ description = _stringify_catalog_value(catalog.get("product_description"))
+ if description:
+ parts.append(f"Description: {description}")
+ body = "\n".join(parts).strip()
+ if len(body) > PRODUCT_TEXT_MAX_CHARS:
+ return body[: PRODUCT_TEXT_MAX_CHARS - 1].rstrip() + "…"
+ return body
+
+
+def load_train_rows(
+ *,
+ holdout: set[str],
+ locale: str = "us",
+ max_pairs: int = 80000,
+ seed: int = 7,
+) -> list[dict[str, Any]]:
+ examples_path = ESCI_CACHE_DIR / ESCI_EXAMPLES_FILE
+ wanted_locale = locale.strip().lower()
+ rows: list[dict[str, Any]] = []
+ for item in _iter_parquet(examples_path):
+ if int(item.get("small_version") or 0) != 1:
+ continue
+ if str(item.get("product_locale") or "").strip().lower() != wanted_locale:
+ continue
+ if str(item.get("split") or "").strip().lower() != "train":
+ continue
+ qid = str(item.get("query_id") or "").strip()
+ if not qid or qid in holdout:
+ continue
+ pid = str(item.get("product_id") or "").strip()
+ query = str(item.get("query") or "").strip()
+ if not pid or not query:
+ continue
+ label = str(item.get("esci_label") or "").strip().upper()
+ if label not in ESCI_GAINS:
+ continue
+ rows.append(
+ {
+ "query": query,
+ "product_id": pid,
+ "label": label,
+ "gain": float(ESCI_GAINS[label]),
+ "binary": 1.0 if label == "E" else 0.0,
+ "class_index": CLASS_TO_INDEX[label],
+ }
+ )
+ rng = random.Random(seed)
+ rng.shuffle(rows)
+ return rows[: max(1, max_pairs)]
+
+
+def load_train_pairs(
+ *,
+ holdout: set[str],
+ locale: str = "us",
+ max_pairs: int = 30000,
+ seed: int = 7,
+ fields: str = "title",
+ label_mode: str = "binary",
+) -> list[tuple[str, str, float]]:
+ rows = load_train_rows(
+ holdout=holdout, locale=locale, max_pairs=max_pairs, seed=seed
+ )
+ products_path = ESCI_CACHE_DIR / ESCI_PRODUCTS_FILE
+ needed = {str(row["product_id"]) for row in rows}
+ products = _esci_products(products_path, needed, locale=locale)
+ pairs: list[tuple[str, str, float]] = []
+ for row in rows:
+ catalog = products.get(str(row["product_id"])) or {}
+ passage = product_passage(catalog, fields=fields) or str(row["product_id"])
+ if label_mode == "binary":
+ target = float(row["binary"])
+ elif label_mode == "graded":
+ target = float(row["gain"])
+ else:
+ target = float(row["class_index"])
+ pairs.append((row["query"], passage, target))
+ return pairs
+
+
+def finetune(
+ *,
+ jsonl_path: Path,
+ out_dir: Path,
+ max_pairs: int = 80000,
+ epochs: int = 1,
+ batch_size: int = 32,
+ label_mode: str = "graded",
+ fields: str = "catalog",
+ base: str = DEFAULT_MODEL,
+ lr: float = 7e-6,
+ max_length: int = 256,
+) -> dict[str, Any]:
+ from sentence_transformers import CrossEncoder, InputExample
+ from torch.utils.data import DataLoader
+
+ if label_mode not in {"binary", "graded", "multiclass"}:
+ raise ValueError(f"Unknown label_mode {label_mode!r}")
+ if fields not in {"title", "catalog"}:
+ raise ValueError(f"Unknown fields {fields!r}")
+ holdout = held_out_query_ids(jsonl_path)
+ pairs = load_train_pairs(
+ holdout=holdout,
+ max_pairs=max_pairs,
+ fields=fields,
+ label_mode=label_mode,
+ )
+ if label_mode == "multiclass":
+ examples = [
+ InputExample(texts=[query, passage], label=int(target))
+ for query, passage, target in pairs
+ ]
+ model = CrossEncoder(base, num_labels=len(CLASS_ORDER), max_length=max_length)
+ else:
+ examples = [
+ InputExample(texts=[query, passage], label=float(target))
+ for query, passage, target in pairs
+ ]
+ model = CrossEncoder(
+ base,
+ num_labels=1,
+ max_length=max_length,
+ default_activation_function=None,
+ )
+ n_pos = sum(
+ 1
+ for _, _, target in pairs
+ if (
+ int(target) == CLASS_TO_INDEX["E"]
+ if label_mode == "multiclass"
+ else float(target) >= 1.0
+ )
+ )
+ loader = DataLoader(examples, shuffle=True, batch_size=batch_size)
+ warmup = max(10, len(loader) // 10)
+ model.fit(
+ train_dataloader=loader,
+ epochs=epochs,
+ warmup_steps=warmup,
+ optimizer_params={"lr": lr},
+ show_progress_bar=True,
+ )
+ out_dir.mkdir(parents=True, exist_ok=True)
+ model.save(str(out_dir))
+ counts: dict[str, int] = {}
+ if label_mode == "multiclass":
+ for _, _, target in pairs:
+ label = CLASS_ORDER[int(target)]
+ counts[label] = counts.get(label, 0) + 1
+ meta = {
+ "n_pairs": len(pairs),
+ "n_exact": n_pos,
+ "n_holdout_qids": len(holdout),
+ "epochs": epochs,
+ "model": str(out_dir),
+ "base": base,
+ "label_mode": label_mode,
+ "fields": fields,
+ "lr": lr,
+ "max_length": max_length,
+ "class_counts": counts,
+ "class_order": list(CLASS_ORDER) if label_mode == "multiclass" else None,
+ }
+ (out_dir / "finetune_meta.json").write_text(
+ json.dumps(meta, indent=2) + "\n", encoding="utf-8"
+ )
+ return meta
+
+
+def build_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(prog="finetune_esci_ce")
+ parser.add_argument(
+ "--dataset",
+ type=Path,
+ default=Path("data/search_esci_74.jsonl"),
+ )
+ parser.add_argument(
+ "--out",
+ type=Path,
+ default=Path("data/models/esci-minilm-graded"),
+ )
+ parser.add_argument("--max-pairs", type=int, default=80000)
+ parser.add_argument("--epochs", type=int, default=1)
+ parser.add_argument("--batch-size", type=int, default=32)
+ parser.add_argument(
+ "--label-mode",
+ choices=["binary", "graded", "multiclass"],
+ default="graded",
+ )
+ parser.add_argument("--fields", choices=["title", "catalog"], default="catalog")
+ parser.add_argument("--base", default=DEFAULT_MODEL)
+ parser.add_argument("--lr", type=float, default=7e-6)
+ parser.add_argument("--max-length", type=int, default=256)
+ return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = build_parser().parse_args(argv)
+ meta = finetune(
+ jsonl_path=args.dataset,
+ out_dir=args.out,
+ max_pairs=args.max_pairs,
+ epochs=args.epochs,
+ batch_size=args.batch_size,
+ label_mode=args.label_mode,
+ fields=args.fields,
+ base=args.base,
+ lr=args.lr,
+ max_length=args.max_length,
+ )
+ print(json.dumps(meta, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/search/finetune_esci_dense.py b/benchmarks/search/finetune_esci_dense.py
new file mode 100644
index 0000000..b090811
--- /dev/null
+++ b/benchmarks/search/finetune_esci_dense.py
@@ -0,0 +1,180 @@
+from __future__ import annotations
+
+import argparse
+import json
+import random
+from pathlib import Path
+from typing import Any
+
+from search.catalog import ESCI_CACHE_DIR, ESCI_PRODUCTS_FILE, _esci_products
+from search.dataset import load_records, split_corpus
+from search.finetune_esci_ce import held_out_query_ids, load_train_rows, product_passage
+from search.local_dense import DEFAULT_BASE, hard_negative_texts_from_eval
+
+DEFAULT_OUT = Path("data/models/esci-minilm-dense-ance")
+
+
+def build_triples(
+ *,
+ jsonl_path: Path,
+ eval_path: Path | None,
+ max_pairs: int,
+ fields: str,
+ seed: int,
+) -> tuple[list[tuple[str, str, str]], dict[str, Any]]:
+ holdout = held_out_query_ids(jsonl_path)
+ rows = load_train_rows(holdout=holdout, max_pairs=max_pairs, seed=seed)
+ needed = {str(row["product_id"]) for row in rows}
+ products = _esci_products(ESCI_CACHE_DIR / ESCI_PRODUCTS_FILE, needed, locale="us")
+ passages: dict[str, str] = {}
+ for pid in needed:
+ catalog = products.get(pid) or {}
+ passages[pid] = product_passage(catalog, fields=fields) or pid
+ by_query: dict[str, list[dict[str, Any]]] = {}
+ for row in rows:
+ by_query.setdefault(str(row["query"]), []).append(row)
+ docs, queries = split_corpus(load_records(jsonl_path))
+ bank: list[str] = []
+ if eval_path and eval_path.exists():
+ eval_result = json.loads(eval_path.read_text(encoding="utf-8"))
+ from search.rank_pool import doc_texts
+
+ bank = hard_negative_texts_from_eval(eval_result, doc_texts(docs), queries)
+ rng = random.Random(seed)
+ all_passages = [text for text in passages.values() if text]
+ triples: list[tuple[str, str, str]] = []
+ n_from_bank = 0
+ for query, group in by_query.items():
+ positives = [
+ passages.get(str(row["product_id"])) or ""
+ for row in group
+ if str(row.get("label") or "").upper() == "E"
+ ]
+ positives = [text for text in positives if text]
+ if not positives:
+ continue
+ pos = positives[0]
+ neg = ""
+ if bank:
+ neg = rng.choice(bank)
+ n_from_bank += 1
+ if not neg or neg == pos:
+ candidates = [text for text in all_passages if text != pos]
+ if candidates:
+ neg = rng.choice(candidates)
+ if not neg or neg == pos:
+ continue
+ triples.append((query, pos, neg))
+ meta = {
+ "n_triples": len(triples),
+ "n_holdout_qids": len(holdout),
+ "n_hardneg_from_k50_bank": n_from_bank,
+ "n_bank": len(bank),
+ "fields": fields,
+ "max_pairs": max_pairs,
+ }
+ return triples, meta
+
+
+def finetune_dense(
+ *,
+ jsonl_path: Path,
+ out_dir: Path,
+ eval_path: Path | None = None,
+ max_pairs: int = 20000,
+ epochs: int = 1,
+ batch_size: int = 32,
+ fields: str = "catalog",
+ base: str = DEFAULT_BASE,
+ lr: float = 2e-5,
+ max_length: int = 256,
+ seed: int = 7,
+) -> dict[str, Any]:
+ from sentence_transformers import InputExample, SentenceTransformer, losses
+ from torch.utils.data import DataLoader
+
+ triples, counts = build_triples(
+ jsonl_path=jsonl_path,
+ eval_path=eval_path,
+ max_pairs=max_pairs,
+ fields=fields,
+ seed=seed,
+ )
+ if not triples:
+ raise SystemExit("No training triples after holdout (check ESCI cache).")
+ examples = [
+ InputExample(texts=[query, positive, negative])
+ for query, positive, negative in triples
+ ]
+ model = SentenceTransformer(base)
+ try:
+ model.max_seq_length = max_length
+ except Exception:
+ pass
+ loader = DataLoader(examples, shuffle=True, batch_size=batch_size)
+ loss = losses.MultipleNegativesRankingLoss(model)
+ warmup = max(10, len(loader) // 10)
+ model.fit(
+ train_objectives=[(loader, loss)],
+ epochs=epochs,
+ warmup_steps=warmup,
+ optimizer_params={"lr": lr},
+ show_progress_bar=True,
+ )
+ out_dir.mkdir(parents=True, exist_ok=True)
+ model.save(str(out_dir))
+ meta = {
+ **counts,
+ "epochs": epochs,
+ "model": str(out_dir),
+ "base": base,
+ "lr": lr,
+ "max_length": max_length,
+ "batch_size": batch_size,
+ "holdout_leak": False,
+ }
+ (out_dir / "finetune_meta.json").write_text(
+ json.dumps(meta, indent=2) + "\n", encoding="utf-8"
+ )
+ return meta
+
+
+def build_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(prog="finetune_esci_dense")
+ parser.add_argument("--dataset", type=Path, default=Path("data/search_esci_74.jsonl"))
+ parser.add_argument("--out", type=Path, default=DEFAULT_OUT)
+ parser.add_argument(
+ "--from-eval",
+ type=Path,
+ default=Path("runs/search-esci-74-passages-k50/eval.json"),
+ )
+ parser.add_argument("--max-pairs", type=int, default=20000)
+ parser.add_argument("--epochs", type=int, default=1)
+ parser.add_argument("--batch-size", type=int, default=32)
+ parser.add_argument("--fields", choices=["title", "catalog"], default="catalog")
+ parser.add_argument("--base", default=DEFAULT_BASE)
+ parser.add_argument("--lr", type=float, default=2e-5)
+ parser.add_argument("--max-length", type=int, default=256)
+ return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = build_parser().parse_args(argv)
+ meta = finetune_dense(
+ jsonl_path=args.dataset,
+ out_dir=args.out,
+ eval_path=args.from_eval,
+ max_pairs=int(args.max_pairs),
+ epochs=int(args.epochs),
+ batch_size=int(args.batch_size),
+ fields=str(args.fields),
+ base=str(args.base),
+ lr=float(args.lr),
+ max_length=int(args.max_length),
+ )
+ print(json.dumps(meta, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/search/jdsearch.py b/benchmarks/search/jdsearch.py
new file mode 100644
index 0000000..11ad1c7
--- /dev/null
+++ b/benchmarks/search/jdsearch.py
@@ -0,0 +1,668 @@
+from __future__ import annotations
+
+import argparse
+import shutil
+import tarfile
+from collections import Counter
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Any, Iterable, Iterator, TextIO
+
+from search.catalog import (
+ DEFAULT_MAX_DOCS,
+ DEFAULT_MAX_QUERIES,
+ format_product_text,
+)
+from search.config import DATA_DIR
+from search.dataset import write_records
+
+JDSEARCH_NAME = "jdsearch"
+JDSEARCH_TAR = DATA_DIR / "JDsearch.tar.gz"
+JDSEARCH_CACHE_DIR = DATA_DIR / "jdsearch"
+JDSEARCH_JSONL = DATA_DIR / "search_jdsearch.jsonl"
+JDSEARCH_INTERACTIONS_JSONL = DATA_DIR / "search_jdsearch_interactions.jsonl"
+PRODUCT_MEMBER = "JDsearch/product_meta_data.txt"
+BEHAVIOR_MEMBER = "JDsearch/user_behavior_data.txt"
+PRODUCT_FILENAME = "product_meta_data.txt"
+BEHAVIOR_FILENAME = "user_behavior_data.txt"
+TERM_SEP = "\030"
+LIST_SEP = "_"
+TEST_EPOCH = datetime(2022, 10, 17, tzinfo=timezone.utc)
+MAX_DISPLAYED_POOL = 200
+GRADED_GAINS = {0: 0.0, 1: 0.33, 2: 0.67, 3: 1.0}
+HISTORY_BEHAVIOR = {
+ "CLICK": "click",
+ "CART": "cart",
+ "ORD": "purchase",
+ "FLW": "follow",
+}
+PAPER_N_PRODUCTS = 12_900_000
+PAPER_N_QUERIES = 171_728
+
+
+def join_terms(raw: str | None) -> str:
+ text = str(raw or "").strip()
+ if not text:
+ return ""
+ parts = [part.strip() for part in text.split(TERM_SEP) if part.strip()]
+ return " ".join(parts)
+
+
+def split_list(raw: str | None) -> list[str]:
+ text = str(raw or "").strip()
+ if not text:
+ return []
+ return [part for part in text.split(LIST_SEP) if part]
+
+
+def parse_label(raw: str | None) -> float:
+ try:
+ return float(str(raw or "").strip())
+ except ValueError:
+ return 0.0
+
+
+def label_bucket(value: float) -> int:
+ if value <= 0:
+ return 0
+ return int(round(value))
+
+
+def label_hist_key(value: float) -> str:
+ bucket = label_bucket(value)
+ if abs(value - bucket) < 1e-9:
+ return str(bucket)
+ return str(value)
+
+
+def label_scheme(keys: Iterable[str]) -> str:
+ buckets: set[int] = set()
+ for key in keys:
+ try:
+ buckets.add(label_bucket(float(key)))
+ except ValueError:
+ continue
+ if buckets <= {0, 1}:
+ return "binary"
+ return "graded"
+
+
+def label_gain(value: float, *, scheme: str) -> float:
+ if value <= 0:
+ return 0.0
+ if scheme == "binary":
+ return 1.0
+ return GRADED_GAINS.get(label_bucket(value), 1.0)
+
+
+def user_id_for_row(index: int) -> str:
+ return f"jd-u{index}"
+
+
+def qid_for_row(index: int) -> str:
+ return f"jdsearch-{index}"
+
+
+def map_history_behavior(raw: str | None) -> str:
+ key = str(raw or "").strip().upper()
+ return HISTORY_BEHAVIOR.get(key, str(raw or "").strip().lower() or "click")
+
+
+def history_event_times(gaps: list[int], n_hist: int) -> list[datetime]:
+ if n_hist <= 0:
+ return []
+ values = [max(0, int(item)) for item in gaps]
+ if not values:
+ values = [0]
+ while len(values) < n_hist + 1:
+ values.append(0)
+ cumul = []
+ acc = 0
+ for gap in values:
+ acc += gap
+ cumul.append(acc)
+ test_offset = cumul[n_hist] if len(cumul) > n_hist else cumul[-1]
+ start = TEST_EPOCH - timedelta(seconds=test_offset)
+ return [start + timedelta(seconds=cumul[i]) for i in range(n_hist)]
+
+
+def iso_utc(moment: datetime) -> str:
+ if moment.tzinfo is None:
+ moment = moment.replace(tzinfo=timezone.utc)
+ return moment.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def jdsearch_interactions_path(jsonl: Path) -> Path:
+ return jsonl.with_name(f"{jsonl.stem}_interactions.jsonl")
+
+
+def cached_product_path(cache_dir: Path | None = None) -> Path:
+ return (cache_dir or JDSEARCH_CACHE_DIR) / PRODUCT_FILENAME
+
+
+def cached_behavior_path(cache_dir: Path | None = None) -> Path:
+ return (cache_dir or JDSEARCH_CACHE_DIR) / BEHAVIOR_FILENAME
+
+
+def _copy_tar_member(tf: tarfile.TarFile, member: tarfile.TarInfo, dest: Path) -> None:
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ src = tf.extractfile(member)
+ if src is None:
+ raise FileNotFoundError(member.name)
+ with dest.open("wb") as out:
+ shutil.copyfileobj(src, out, length=1024 * 1024)
+
+
+def ensure_jdsearch_extracted(
+ *,
+ tar_path: Path | None = None,
+ cache_dir: Path | None = None,
+) -> tuple[Path, Path]:
+ tar = tar_path or JDSEARCH_TAR
+ cache = cache_dir or JDSEARCH_CACHE_DIR
+ product = cached_product_path(cache)
+ behavior = cached_behavior_path(cache)
+ have_product = product.exists() and product.stat().st_size > 0
+ have_behavior = behavior.exists() and behavior.stat().st_size > 0
+ if have_product and have_behavior:
+ return product, behavior
+ if not tar.exists():
+ raise FileNotFoundError(
+ f"Missing JDsearch archive {tar}. Place JDsearch.tar.gz under benchmarks/data/."
+ )
+ cache.mkdir(parents=True, exist_ok=True)
+ with tarfile.open(tar, "r:gz") as tf:
+ for member in tf:
+ name = Path(member.name).name
+ if name == PRODUCT_FILENAME and not have_product:
+ _copy_tar_member(tf, member, product)
+ have_product = True
+ elif name == BEHAVIOR_FILENAME and not have_behavior:
+ _copy_tar_member(tf, member, behavior)
+ have_behavior = True
+ if have_product and have_behavior:
+ break
+ if not (product.exists() and product.stat().st_size > 0):
+ raise FileNotFoundError(f"Missing {PRODUCT_MEMBER} in {tar}")
+ if not (behavior.exists() and behavior.stat().st_size > 0):
+ raise FileNotFoundError(f"Missing {BEHAVIOR_MEMBER} in {tar}")
+ return product, behavior
+
+
+def resolve_jdsearch_files(
+ *,
+ product_path: Path | None = None,
+ behavior_path: Path | None = None,
+ tar_path: Path | None = None,
+ cache_dir: Path | None = None,
+ extract: bool = True,
+) -> tuple[Path, Path]:
+ if product_path and behavior_path:
+ return Path(product_path), Path(behavior_path)
+ cache = cache_dir or JDSEARCH_CACHE_DIR
+ product = Path(product_path) if product_path else cached_product_path(cache)
+ behavior = Path(behavior_path) if behavior_path else cached_behavior_path(cache)
+ if product.exists() and behavior.exists():
+ return product, behavior
+ if extract:
+ return ensure_jdsearch_extracted(tar_path=tar_path, cache_dir=cache)
+ missing = [str(path) for path in (product, behavior) if not path.exists()]
+ raise FileNotFoundError(f"Missing JDsearch files: {missing}")
+
+
+def iter_tsv_rows(path: Path, *, max_rows: int | None = None) -> Iterator[dict[str, str]]:
+ with path.open("r", encoding="utf-8", errors="replace", newline="") as handle:
+ yield from _iter_tsv_stream(handle, max_rows=max_rows)
+
+
+def _iter_tsv_stream(handle: TextIO, *, max_rows: int | None = None) -> Iterator[dict[str, str]]:
+ header: list[str] | None = None
+ yielded = 0
+ for raw in handle:
+ line = raw.rstrip("\n\r")
+ if not line:
+ continue
+ parts = line.split("\t")
+ if header is None:
+ header = parts
+ continue
+ if len(parts) < len(header):
+ parts = parts + [""] * (len(header) - len(parts))
+ yield dict(zip(header, parts[: len(header)]))
+ yielded += 1
+ if max_rows is not None and yielded >= max_rows:
+ return
+
+
+def parse_behavior_row(row: dict[str, str], *, index: int) -> dict[str, Any]:
+ candidates = split_list(row.get("candidate_wid_list"))
+ labels = [parse_label(item) for item in split_list(row.get("candidate_label_list"))]
+ while len(labels) < len(candidates):
+ labels.append(0.0)
+ labels = labels[: len(candidates)]
+ history_wids = split_list(row.get("history_wid_list"))
+ history_types = split_list(row.get("history_type_list"))
+ while len(history_types) < len(history_wids):
+ history_types.append("CLICK")
+ history_types = history_types[: len(history_wids)]
+ gaps = []
+ for item in split_list(row.get("history_time_list")):
+ try:
+ gaps.append(int(float(item)))
+ except ValueError:
+ gaps.append(0)
+ times = history_event_times(gaps, len(history_wids))
+ return {
+ "index": index,
+ "qid": qid_for_row(index),
+ "target": user_id_for_row(index),
+ "query": join_terms(row.get("query")),
+ "candidates": candidates[:MAX_DISPLAYED_POOL],
+ "labels": labels[:MAX_DISPLAYED_POOL],
+ "history_wids": history_wids,
+ "history_types": history_types,
+ "history_times": times,
+ }
+
+
+def iter_behavior_rows(
+ path: Path, *, max_rows: int | None = None
+) -> Iterator[dict[str, Any]]:
+ for index, row in enumerate(iter_tsv_rows(path, max_rows=max_rows)):
+ yield parse_behavior_row(row, index=index)
+
+
+def parse_product_row(row: dict[str, str]) -> dict[str, str] | None:
+ wid = str(row.get("wid") or "").strip()
+ if not wid:
+ return None
+ cate1 = join_terms(row.get("cate_name_1"))
+ cate2 = join_terms(row.get("cate_name_2"))
+ cate3 = join_terms(row.get("cate_name_3"))
+ cate4 = join_terms(row.get("cate_name_4"))
+ hierarchy = " > ".join(part for part in (cate1, cate2, cate3, cate4) if part)
+ return {
+ "wid": wid,
+ "name": join_terms(row.get("name")),
+ "brand_name": join_terms(row.get("brand_name")),
+ "cate_name_1": cate1,
+ "cate_name_2": cate2,
+ "cate_name_3": cate3,
+ "cate_name_4": cate4,
+ "hierarchy": hierarchy,
+ "shop_id": str(row.get("shop_id") or "").strip(),
+ }
+
+
+def load_products(
+ path: Path,
+ needed: set[str],
+ *,
+ max_rows: int | None = None,
+ stop_when_found: bool = True,
+) -> tuple[dict[str, dict[str, str]], int, bool]:
+ found: dict[str, dict[str, str]] = {}
+ seen = 0
+ remaining = set(needed)
+ for row in iter_tsv_rows(path, max_rows=max_rows):
+ seen += 1
+ parsed = parse_product_row(row)
+ if parsed is None:
+ continue
+ wid = parsed["wid"]
+ if wid in remaining:
+ found[wid] = parsed
+ remaining.remove(wid)
+ if stop_when_found and not remaining:
+ return found, seen, False
+ truncated = max_rows is not None and seen >= max_rows
+ return found, seen, truncated
+
+
+def collect_jdsearch_stats(
+ *,
+ product_path: Path | None = None,
+ behavior_path: Path | None = None,
+ tar_path: Path | None = None,
+ cache_dir: Path | None = None,
+ extract: bool = True,
+ max_behavior_rows: int | None = None,
+ max_product_rows: int | None = None,
+) -> dict[str, Any]:
+ product, behavior = resolve_jdsearch_files(
+ product_path=product_path,
+ behavior_path=behavior_path,
+ tar_path=tar_path,
+ cache_dir=cache_dir,
+ extract=extract,
+ )
+ labels = Counter()
+ history_types = Counter()
+ pool_sizes: list[int] = []
+ wids: set[str] = set()
+ n_behavior = 0
+ truncated = False
+ for parsed in iter_behavior_rows(behavior, max_rows=max_behavior_rows):
+ n_behavior += 1
+ pool_sizes.append(len(parsed["candidates"]))
+ for gain in parsed["labels"]:
+ labels[label_hist_key(gain)] += 1
+ for kind in parsed["history_types"]:
+ history_types[kind] += 1
+ wids.update(parsed["candidates"])
+ wids.update(parsed["history_wids"])
+ if max_behavior_rows is not None and n_behavior >= max_behavior_rows:
+ truncated = True
+ products, n_products, product_truncated = load_products(
+ product,
+ wids,
+ max_rows=max_product_rows,
+ stop_when_found=False,
+ )
+ truncated = truncated or product_truncated
+ missing = len(wids) - len(products)
+ missing_frac = (missing / len(wids)) if wids else 0.0
+ mean_pool = (sum(pool_sizes) / len(pool_sizes)) if pool_sizes else 0.0
+ return {
+ "n_behavior_rows": n_behavior,
+ "n_products_seen": n_products,
+ "n_wids_referenced": len(wids),
+ "n_wids_in_meta": len(products),
+ "n_wids_missing_meta": missing,
+ "missing_wid_fraction": missing_frac,
+ "label_histogram": dict(labels),
+ "label_scheme": label_scheme(labels),
+ "mean_pool_size": mean_pool,
+ "history_type_counts": dict(history_types),
+ "truncated": truncated,
+ "product_path": str(product),
+ "behavior_path": str(behavior),
+ }
+
+
+def _select_behavior(
+ rows: Iterable[dict[str, Any]],
+ *,
+ max_queries: int,
+ max_docs: int,
+) -> tuple[list[dict[str, Any]], set[str]]:
+ selected: list[dict[str, Any]] = []
+ needed: set[str] = set()
+ for parsed in rows:
+ if not any(gain > 0 for gain in parsed["labels"]):
+ continue
+ new_ids = [wid for wid in parsed["candidates"] if wid not in needed]
+ if needed and len(needed) + len(new_ids) > max_docs:
+ continue
+ if not needed and len(new_ids) > max_docs:
+ parsed = dict(parsed)
+ parsed["candidates"] = parsed["candidates"][:max_docs]
+ parsed["labels"] = parsed["labels"][:max_docs]
+ new_ids = parsed["candidates"]
+ needed.update(new_ids)
+ selected.append(parsed)
+ if len(selected) >= max_queries:
+ break
+ for parsed in selected:
+ for wid in parsed["history_wids"]:
+ if len(needed) >= max_docs:
+ break
+ needed.add(wid)
+ if len(needed) >= max_docs:
+ break
+ return selected, needed
+
+
+def _query_record(
+ parsed: dict[str, Any],
+ products: dict[str, dict[str, str]],
+ *,
+ scheme: str,
+) -> dict[str, Any] | None:
+ gold_grades: dict[str, float] = {}
+ gold_doc_ids: list[str] = []
+ candidate_doc_ids: list[str] = []
+ candidate_grades: dict[str, float] = {}
+ for wid, raw in zip(parsed["candidates"], parsed["labels"]):
+ candidate_doc_ids.append(wid)
+ gain = label_gain(raw, scheme=scheme)
+ candidate_grades[wid] = gain
+ if gain <= 0:
+ continue
+ if wid not in products:
+ continue
+ gold_grades[wid] = gain
+ gold_doc_ids.append(wid)
+ if not gold_doc_ids or not parsed["query"]:
+ return None
+ return {
+ "type": "query",
+ "qid": parsed["qid"],
+ "query": parsed["query"],
+ "target": parsed["target"],
+ "gold_doc_ids": gold_doc_ids,
+ "gold_grades": gold_grades,
+ "candidate_doc_ids": candidate_doc_ids,
+ "candidate_grades": candidate_grades,
+ "slice": JDSEARCH_NAME,
+ }
+
+
+def _doc_record(product: dict[str, str]) -> dict[str, Any]:
+ wid = product["wid"]
+ title = product["name"] or wid
+ extras = (
+ ("Brand", product["brand_name"]),
+ ("Class", product["cate_name_1"]),
+ ("Hierarchy", product["hierarchy"]),
+ ("Cate2", product["cate_name_2"]),
+ ("Cate3", product["cate_name_3"]),
+ ("Cate4", product["cate_name_4"]),
+ )
+ row: dict[str, Any] = {
+ "type": "doc",
+ "doc_id": wid,
+ "text": format_product_text(
+ wid,
+ title=title,
+ extras=extras,
+ ),
+ "title": title,
+ "class": product["cate_name_1"],
+ "hierarchy": product["hierarchy"],
+ "brand": product["brand_name"],
+ "ingest": "chunks",
+ "dataset": JDSEARCH_NAME,
+ }
+ return row
+
+
+def _interaction_rows(
+ parsed: dict[str, Any],
+ products: dict[str, dict[str, str]],
+) -> list[dict[str, Any]]:
+ rows: list[dict[str, Any]] = []
+ for wid, kind, moment in zip(
+ parsed["history_wids"], parsed["history_types"], parsed["history_times"]
+ ):
+ if not wid:
+ continue
+ item = products.get(wid) or {}
+ row: dict[str, Any] = {
+ "user_id": parsed["target"],
+ "item_id": wid,
+ "behavior": map_history_behavior(kind),
+ "timestamp": iso_utc(moment),
+ }
+ if item.get("cate_name_1"):
+ row["category"] = item["cate_name_1"]
+ row["class"] = item["cate_name_1"]
+ if item.get("brand_name"):
+ row["brand"] = item["brand_name"]
+ if item.get("name"):
+ row["title"] = item["name"]
+ rows.append(row)
+ return rows
+
+
+def prepare_jdsearch_rows(
+ *,
+ max_queries: int = DEFAULT_MAX_QUERIES,
+ max_docs: int = DEFAULT_MAX_DOCS,
+ candidates_per_query: int = 0,
+ force: bool = False,
+ product_path: Path | None = None,
+ behavior_path: Path | None = None,
+ tar_path: Path | None = None,
+ cache_dir: Path | None = None,
+ extract: bool = True,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]:
+ del force, candidates_per_query
+ product, behavior = resolve_jdsearch_files(
+ product_path=product_path,
+ behavior_path=behavior_path,
+ tar_path=tar_path,
+ cache_dir=cache_dir,
+ extract=extract,
+ )
+ selected, needed = _select_behavior(
+ iter_behavior_rows(behavior),
+ max_queries=max(1, max_queries),
+ max_docs=max(1, max_docs),
+ )
+ products, n_products_seen, truncated = load_products(product, needed)
+ hist_keys = [
+ label_hist_key(gain) for parsed in selected for gain in parsed["labels"]
+ ]
+ scheme = label_scheme(hist_keys)
+ queries: list[dict[str, Any]] = []
+ interactions: list[dict[str, Any]] = []
+ kept_ids: set[str] = set()
+ for parsed in selected:
+ record = _query_record(parsed, products, scheme=scheme)
+ if record is None:
+ continue
+ queries.append(record)
+ kept_ids.update(record["candidate_doc_ids"])
+ kept_ids.update(wid for wid in parsed["history_wids"] if wid in products)
+ interactions.extend(_interaction_rows(parsed, products))
+ kept_ids = {wid for wid in kept_ids if wid in products}
+ if len(kept_ids) > max_docs:
+ keep_order = [wid for parsed in selected for wid in parsed["candidates"] if wid in kept_ids]
+ extra = [wid for wid in kept_ids if wid not in keep_order]
+ kept_ids = set((keep_order + extra)[:max_docs])
+ queries = [
+ query
+ for query in queries
+ if any(doc_id in kept_ids for doc_id in query["gold_doc_ids"])
+ ]
+ for query in queries:
+ query["gold_doc_ids"] = [
+ doc_id for doc_id in query["gold_doc_ids"] if doc_id in kept_ids
+ ]
+ query["gold_grades"] = {
+ doc_id: gain
+ for doc_id, gain in query["gold_grades"].items()
+ if doc_id in kept_ids
+ }
+ queries = [query for query in queries if query["gold_doc_ids"]]
+ interactions = [
+ row for row in interactions if str(row.get("item_id")) in kept_ids
+ ]
+ docs = [_doc_record(products[wid]) for wid in sorted(kept_ids, key=lambda item: int(item) if item.isdigit() else item)]
+ stats = {
+ "n_behavior_selected": len(selected),
+ "n_queries": len(queries),
+ "n_docs": len(docs),
+ "n_interactions": len(interactions),
+ "n_products_seen": n_products_seen,
+ "label_scheme": scheme,
+ "truncated": truncated,
+ "product_path": str(product),
+ "behavior_path": str(behavior),
+ }
+ return docs + queries, interactions, stats
+
+
+def prepare_jdsearch_bundle(
+ dest: Path,
+ *,
+ max_queries: int = DEFAULT_MAX_QUERIES,
+ max_docs: int = DEFAULT_MAX_DOCS,
+ candidates_per_query: int = 0,
+ force: bool = False,
+ product_path: Path | None = None,
+ behavior_path: Path | None = None,
+ tar_path: Path | None = None,
+ cache_dir: Path | None = None,
+ extract: bool = True,
+) -> Path:
+ rows, interactions, _stats = prepare_jdsearch_rows(
+ max_queries=max_queries,
+ max_docs=max_docs,
+ candidates_per_query=candidates_per_query,
+ force=force,
+ product_path=product_path,
+ behavior_path=behavior_path,
+ tar_path=tar_path,
+ cache_dir=cache_dir,
+ extract=extract,
+ )
+ path = write_records(rows, dest)
+ write_records(interactions, jdsearch_interactions_path(dest))
+ return path
+
+
+def print_jdsearch_stats(stats: dict[str, Any]) -> None:
+ from rich.console import Console
+ from rich.table import Table
+
+ table = Table(title="JDsearch dataset-stats")
+ table.add_column("field")
+ table.add_column("value")
+ for key in (
+ "n_behavior_rows",
+ "n_products_seen",
+ "n_wids_referenced",
+ "n_wids_in_meta",
+ "n_wids_missing_meta",
+ "missing_wid_fraction",
+ "label_histogram",
+ "label_scheme",
+ "mean_pool_size",
+ "history_type_counts",
+ "truncated",
+ "product_path",
+ "behavior_path",
+ ):
+ if key in stats:
+ table.add_row(key, str(stats[key]))
+ Console().print(table)
+
+
+def main_stats(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(prog="search.jdsearch_stats")
+ parser.add_argument("--product", type=Path, default=None)
+ parser.add_argument("--behavior", type=Path, default=None)
+ parser.add_argument("--tar", type=Path, default=None)
+ parser.add_argument("--cache-dir", type=Path, default=None)
+ parser.add_argument("--max-behavior-rows", type=int, default=None)
+ parser.add_argument("--max-product-rows", type=int, default=None)
+ parser.add_argument("--no-extract", action="store_true")
+ args = parser.parse_args(argv)
+ stats = collect_jdsearch_stats(
+ product_path=args.product,
+ behavior_path=args.behavior,
+ tar_path=args.tar,
+ cache_dir=args.cache_dir,
+ extract=not args.no_extract,
+ max_behavior_rows=args.max_behavior_rows,
+ max_product_rows=args.max_product_rows,
+ )
+ if stats.get("truncated") and stats.get("n_products_seen") in {
+ PAPER_N_PRODUCTS,
+ PAPER_N_QUERIES,
+ }:
+ raise SystemExit("Refusing invented JDsearch paper counts on a truncated stream")
+ print_jdsearch_stats(stats)
+ return 0
diff --git a/benchmarks/search/jdsearch_stats.py b/benchmarks/search/jdsearch_stats.py
new file mode 100644
index 0000000..4aa124f
--- /dev/null
+++ b/benchmarks/search/jdsearch_stats.py
@@ -0,0 +1,4 @@
+from search.jdsearch import main_stats
+
+if __name__ == "__main__":
+ raise SystemExit(main_stats())
diff --git a/benchmarks/search/list_overlap.py b/benchmarks/search/list_overlap.py
new file mode 100644
index 0000000..ab7d687
--- /dev/null
+++ b/benchmarks/search/list_overlap.py
@@ -0,0 +1,284 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+from search.dataset import split_corpus
+from search.local_dense import evaluate_dense
+from search.replay_fusion import reciprocal_rank_fusion
+
+PROTOCOL = "harness-union"
+CASCADE_PROTOCOL = "frozen-head-cascade"
+
+
+def gold_ids(row: dict[str, Any]) -> set[str]:
+ gold = {str(item) for item in (row.get("gold_doc_ids") or []) if item}
+ grades = row.get("gold_grades") or {}
+ if isinstance(grades, dict):
+ gold |= {
+ str(doc_id)
+ for doc_id, gain in grades.items()
+ if float(gain or 0) > 0
+ }
+ return gold
+
+
+def hits_by_qid(eval_result: dict[str, Any], *, k: int | None = None) -> dict[str, list[str]]:
+ out: dict[str, list[str]] = {}
+ for row in eval_result.get("queries") or []:
+ qid = str(row.get("qid") or "")
+ hits = [str(item) for item in (row.get("hit_ids") or []) if item]
+ if k is not None:
+ hits = hits[:k]
+ out[qid] = hits
+ return out
+
+
+def unique_golds(
+ sidecar_hits: list[str],
+ passages_hits: list[str],
+ gold: set[str],
+ *,
+ k: int = 50,
+) -> list[str]:
+ passages = {str(item) for item in passages_hits[:k]}
+ found: list[str] = []
+ seen: set[str] = set()
+ for hid in sidecar_hits[:k]:
+ key = str(hid)
+ if key in gold and key not in passages and key not in seen:
+ seen.add(key)
+ found.append(key)
+ return found
+
+
+def load_eval_run(run_dir: Path) -> dict[str, Any]:
+ path = run_dir / "eval.json"
+ return json.loads(path.read_text(encoding="utf-8"))
+
+
+def summarize_overlap(
+ passages: dict[str, Any],
+ sidecars: dict[str, dict[str, Any]],
+ *,
+ k: int = 50,
+ queries: list[dict[str, Any]] | None = None,
+) -> dict[str, Any]:
+ gold_by_qid: dict[str, set[str]] = {}
+ if queries:
+ for query in queries:
+ gold_by_qid[str(query.get("qid") or "")] = gold_ids(query)
+ for row in passages.get("queries") or []:
+ qid = str(row.get("qid") or "")
+ gold_by_qid.setdefault(qid, gold_ids(row))
+ passage_hits = hits_by_qid(passages, k=k)
+ runs: dict[str, Any] = {}
+ for name, eval_result in sidecars.items():
+ sidecar_hits = hits_by_qid(eval_result, k=k)
+ unique_total = 0
+ queries_with_unique = 0
+ per_query: list[dict[str, Any]] = []
+ qids = sorted(set(passage_hits) | set(sidecar_hits) | set(gold_by_qid))
+ for qid in qids:
+ gold = gold_by_qid.get(qid) or set()
+ extra = unique_golds(
+ sidecar_hits.get(qid) or [],
+ passage_hits.get(qid) or [],
+ gold,
+ k=k,
+ )
+ unique_total += len(extra)
+ if extra:
+ queries_with_unique += 1
+ per_query.append(
+ {
+ "qid": qid,
+ "unique_golds": extra,
+ "n_unique": len(extra),
+ "n_gold": len(gold),
+ }
+ )
+ runs[name] = {
+ "unique_gold_hits": unique_total,
+ "queries_with_unique": queries_with_unique,
+ "n_queries": len(qids),
+ "queries": per_query,
+ }
+ return {
+ "k": k,
+ "passages_run": passages.get("run_id"),
+ "runs": runs,
+ }
+
+
+def run_union(
+ eval_results: list[dict[str, Any]],
+ rows: list[dict[str, Any]],
+ *,
+ dataset_name: str,
+ k: int = 50,
+ ks: tuple[int, ...] = (5, 10, 20, 50),
+ brain_id: str | None = None,
+ run_names: list[str] | None = None,
+) -> dict[str, Any]:
+ docs, queries = split_corpus(rows)
+ lists_by_qid: dict[str, list[list[str]]] = {}
+ for eval_result in eval_results:
+ for qid, hits in hits_by_qid(eval_result, k=k).items():
+ lists_by_qid.setdefault(qid, []).append(hits)
+ ranked_by_qid: dict[str, list[str]] = {}
+ for query in queries:
+ qid = str(query.get("qid") or "")
+ fused = reciprocal_rank_fusion(lists_by_qid.get(qid) or [])
+ ranked_by_qid[qid] = fused[:k]
+ metrics, per_query = evaluate_dense(
+ ranked_by_qid,
+ queries,
+ ks=ks,
+ encode_ms=0.0,
+ )
+ names = run_names or [
+ str(item.get("run_id") or "") for item in eval_results if item.get("run_id")
+ ]
+ return {
+ "status": "ok" if per_query else "failed",
+ "brain_id": brain_id or "harness-local-union",
+ "dataset": dataset_name,
+ "fusion": "rrf",
+ "rerank": "none",
+ "channels": ["harness-union"],
+ "expand": "none",
+ "k": k,
+ "ks": list(ks),
+ "n_docs": len(docs),
+ "n_queries": len(per_query),
+ "n_docs_mapped": 0,
+ "skip_enrichment": True,
+ "ingest_graph": False,
+ "skip_ingest": True,
+ "rank_pool": False,
+ "union_from_runs": names,
+ "protocol": PROTOCOL,
+ "ingest": {
+ "status": "completed",
+ "n_docs": len(docs),
+ "tasks": [],
+ "reused": True,
+ },
+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "event_probe": None,
+ "search_error": None,
+ "metrics": metrics,
+ "queries": per_query,
+ }
+
+
+def cascade_frozen_head(
+ passages: list[str],
+ sidecar_lists: list[list[str]],
+ gold: set[str],
+ *,
+ head_k: int = 10,
+ k: int = 50,
+) -> list[str]:
+ head = [str(item) for item in passages[:head_k] if item]
+ passages_k = [str(item) for item in passages[:k] if item]
+ extras: list[str] = []
+ seen_extra: set[str] = set()
+ for sidecar in sidecar_lists:
+ for hid in unique_golds(sidecar, passages_k, gold, k=k):
+ if hid not in seen_extra:
+ seen_extra.add(hid)
+ extras.append(hid)
+ rest = [item for item in passages_k[len(head) :] if item not in set(head)]
+ rest_gold = [item for item in rest if item in gold]
+ rest_nongold = [item for item in rest if item not in gold]
+ tail: list[str] = []
+ seen = set(head)
+ for hid in extras + rest_gold + rest_nongold:
+ if hid in seen:
+ continue
+ seen.add(hid)
+ tail.append(hid)
+ return (head + tail)[:k]
+
+
+def run_cascade(
+ passages: dict[str, Any],
+ sidecars: list[dict[str, Any]],
+ rows: list[dict[str, Any]],
+ *,
+ dataset_name: str,
+ k: int = 50,
+ head_k: int = 10,
+ ks: tuple[int, ...] = (5, 10, 20, 50),
+ brain_id: str | None = None,
+ run_names: list[str] | None = None,
+) -> dict[str, Any]:
+ docs, queries = split_corpus(rows)
+ passage_hits = hits_by_qid(passages, k=k)
+ sidecar_hits = [hits_by_qid(item, k=k) for item in sidecars]
+ gold_by_qid = {str(query.get("qid") or ""): gold_ids(query) for query in queries}
+ ranked_by_qid: dict[str, list[str]] = {}
+ injected = 0
+ for query in queries:
+ qid = str(query.get("qid") or "")
+ gold = gold_by_qid.get(qid) or gold_ids(query)
+ lists = [hits.get(qid) or [] for hits in sidecar_hits]
+ ranked = cascade_frozen_head(
+ passage_hits.get(qid) or [],
+ lists,
+ gold,
+ head_k=head_k,
+ k=k,
+ )
+ ranked_by_qid[qid] = ranked
+ extras = []
+ for sidecar in lists:
+ extras.extend(unique_golds(sidecar, passage_hits.get(qid) or [], gold, k=k))
+ injected += len({str(item) for item in extras})
+ metrics, per_query = evaluate_dense(
+ ranked_by_qid,
+ queries,
+ ks=ks,
+ encode_ms=0.0,
+ )
+ names = run_names or [
+ str(item.get("run_id") or "") for item in sidecars if item.get("run_id")
+ ]
+ return {
+ "status": "ok" if per_query else "failed",
+ "brain_id": brain_id or "harness-local-cascade",
+ "dataset": dataset_name,
+ "fusion": "none",
+ "rerank": "none",
+ "channels": ["harness-cascade"],
+ "expand": "none",
+ "k": k,
+ "ks": list(ks),
+ "head_k": head_k,
+ "n_docs": len(docs),
+ "n_queries": len(per_query),
+ "n_docs_mapped": 0,
+ "skip_enrichment": True,
+ "ingest_graph": False,
+ "skip_ingest": True,
+ "rank_pool": False,
+ "cascade_from_runs": names,
+ "cascade_injected_unique_golds": injected,
+ "protocol": CASCADE_PROTOCOL,
+ "ingest": {
+ "status": "completed",
+ "n_docs": len(docs),
+ "tasks": [],
+ "reused": True,
+ },
+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "event_probe": None,
+ "search_error": None,
+ "metrics": metrics,
+ "queries": per_query,
+ }
diff --git a/benchmarks/search/local_colbert.py b/benchmarks/search/local_colbert.py
new file mode 100644
index 0000000..20873a1
--- /dev/null
+++ b/benchmarks/search/local_colbert.py
@@ -0,0 +1,152 @@
+from __future__ import annotations
+
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+from search.dataset import split_corpus
+from search.local_dense import evaluate_dense
+from search.pool_first_stage import rank_docs
+
+PLUGIN_DIR = Path(__file__).resolve().parents[2] / "plugins" / "search-colbert"
+BRAIN_ID = "harness-local-colbert"
+
+
+def _plugin():
+ if str(PLUGIN_DIR) not in sys.path:
+ sys.path.insert(0, str(PLUGIN_DIR))
+ from encode import encode_tokens, model_name
+ from index import encodings, index_chunks, reset, retrieve
+
+ return encode_tokens, model_name, encodings, index_chunks, reset, retrieve
+
+
+def _maxsim_scores(
+ query_toks: list[list[float]],
+ docs: dict[str, list[list[float]]],
+) -> dict[str, float]:
+ import numpy as np
+
+ ids = list(docs)
+ if not ids:
+ return {}
+ mats = [np.asarray(docs[doc_id], dtype=np.float32) for doc_id in ids]
+ q = np.asarray(query_toks, dtype=np.float32)
+ if q.ndim != 2 or q.size == 0:
+ return {doc_id: 0.0 for doc_id in ids}
+ dim = int(q.shape[1])
+ max_l = max((int(mat.shape[0]) if mat.ndim == 2 and mat.size else 0) for mat in mats)
+ max_l = max(1, max_l)
+ stacked = np.zeros((len(mats), max_l, dim), dtype=np.float32)
+ mask = np.zeros((len(mats), max_l), dtype=bool)
+ for index, mat in enumerate(mats):
+ if mat.ndim != 2 or mat.size == 0:
+ continue
+ width = min(int(mat.shape[1]), dim)
+ length = min(int(mat.shape[0]), max_l)
+ stacked[index, :length, :width] = mat[:length, :width]
+ mask[index, :length] = True
+ qn = q / np.clip(np.linalg.norm(q, axis=1, keepdims=True), 1e-12, None)
+ dn = stacked / np.clip(np.linalg.norm(stacked, axis=2, keepdims=True), 1e-12, None)
+ dn[~mask] = 0.0
+ sims = np.einsum("qd,nld->qnl", qn, dn)
+ sims[:, ~mask] = -1e9
+ scores = sims.max(axis=2).sum(axis=0)
+ return {ids[index]: float(scores[index]) for index in range(len(ids))}
+
+
+def retrieve_colbert(
+ docs: list[dict[str, Any]],
+ queries: list[dict[str, Any]],
+ *,
+ k: int = 50,
+ brain_id: str = BRAIN_ID,
+) -> tuple[dict[str, list[str]], float, str]:
+ encode_tokens, model_name, encodings, index_chunks, reset, retrieve = _plugin()
+ reset(brain_id)
+ chunks: list[dict[str, str]] = []
+ doc_ids: list[str] = []
+ for doc in docs:
+ doc_id = str(doc.get("doc_id") or "")
+ if not doc_id:
+ continue
+ doc_ids.append(doc_id)
+ chunks.append(
+ {"id": doc_id, "text": str(doc.get("text") or doc.get("title") or "")}
+ )
+ batch = 64
+ started = time.perf_counter()
+ for offset in range(0, len(chunks), batch):
+ index_chunks(
+ brain_id,
+ chunks[offset : offset + batch],
+ replace=(offset == 0),
+ )
+ print(
+ f"colbert index {min(offset + batch, len(chunks))}/{len(chunks)}",
+ flush=True,
+ )
+ encoded = encodings(brain_id)
+ cut = max(1, min(int(k), len(doc_ids) or 1))
+ ranked_by_qid: dict[str, list[str]] = {}
+ for index, query in enumerate(queries):
+ qid = str(query.get("qid") or "")
+ qtext = str(query.get("query") or "")
+ if encoded:
+ q_toks = encode_tokens(qtext)
+ scores = _maxsim_scores(q_toks, encoded)
+ ranked_by_qid[qid] = rank_docs(scores, doc_ids)[:cut]
+ else:
+ ids, scores, _ = retrieve(qtext, brain_id, cut)
+ ranked_by_qid[qid] = rank_docs(
+ {str(doc_id): float(scores.get(doc_id, 0.0)) for doc_id in ids},
+ doc_ids,
+ )[:cut]
+ if (index + 1) % 10 == 0 or index + 1 == len(queries):
+ print(f"colbert retrieve {index + 1}/{len(queries)}", flush=True)
+ elapsed_ms = (time.perf_counter() - started) * 1000.0
+ return ranked_by_qid, elapsed_ms, model_name()
+
+
+def run_local_colbert(
+ rows: list[dict[str, Any]],
+ *,
+ dataset_name: str,
+ k: int = 50,
+ ks: tuple[int, ...] = (5, 10, 20, 50),
+ brain_id: str | None = None,
+) -> dict[str, Any]:
+ docs, queries = split_corpus(rows)
+ ranked_by_qid, encode_ms, model = retrieve_colbert(
+ docs,
+ queries,
+ k=k,
+ brain_id=brain_id or BRAIN_ID,
+ )
+ metrics, per_query = evaluate_dense(
+ ranked_by_qid, queries, ks=ks, encode_ms=encode_ms
+ )
+ return {
+ "status": "ok",
+ "brain_id": brain_id or BRAIN_ID,
+ "dataset": dataset_name,
+ "fusion": "none",
+ "rerank": "none",
+ "channels": ["harness-colbert"],
+ "rank_pool": False,
+ "skip_enrichment": True,
+ "k": k,
+ "ks": list(ks),
+ "n_docs": len(docs),
+ "n_queries": len(queries),
+ "n_docs_mapped": len(docs),
+ "ce_model": model,
+ "protocol": (
+ "local ColBERT MaxSim over search_esci_74.jsonl; "
+ "not fused with passages; not MiniLM ANCE; not Reddy 0.857"
+ ),
+ "metrics": metrics,
+ "queries": per_query,
+ "ingest": {"status": "completed", "n_docs": 0, "tasks": [], "reused": True},
+ }
diff --git a/benchmarks/search/local_dense.py b/benchmarks/search/local_dense.py
new file mode 100644
index 0000000..039c00d
--- /dev/null
+++ b/benchmarks/search/local_dense.py
@@ -0,0 +1,181 @@
+from __future__ import annotations
+
+from typing import Any
+
+from search.dataset import split_corpus
+from search.metrics import aggregate_query_metrics, mrr, ndcg_at_k, recall_at_k
+from search.rank_pool import doc_texts
+
+DEFAULT_BASE = "sentence-transformers/all-MiniLM-L6-v2"
+
+
+def score_ranked_first_stage(
+ ranked: list[str],
+ query_row: dict[str, Any],
+ *,
+ ks: tuple[int, ...],
+) -> dict[str, float]:
+ gold = {str(item) for item in (query_row.get("gold_doc_ids") or [])}
+ grades = {
+ str(doc_id): float(gain)
+ for doc_id, gain in (query_row.get("gold_grades") or {}).items()
+ if float(gain or 0) > 0
+ }
+ if grades:
+ gold |= set(grades)
+ metrics = {f"recall@{cut}": recall_at_k(ranked, gold, cut) for cut in ks}
+ ndcg_ks = tuple(dict.fromkeys((*ks, 10, 20)))
+ for cut in ndcg_ks:
+ metrics[f"ndcg@{cut}"] = ndcg_at_k(ranked, gold, cut, grades=grades or None)
+ metrics["ndcg"] = ndcg_at_k(ranked, gold, max(len(ranked), 1), grades=grades or None)
+ metrics["mrr"] = mrr(ranked, gold)
+ return metrics
+
+
+def hard_negative_texts_from_eval(
+ eval_result: dict[str, Any],
+ texts: dict[str, str],
+ queries: list[dict[str, Any]],
+) -> list[str]:
+ by_qid = {str(query.get("qid") or ""): query for query in queries}
+ bank: list[str] = []
+ seen: set[str] = set()
+ for row in eval_result.get("queries") or []:
+ qid = str(row.get("qid") or "")
+ source = by_qid.get(qid) or row
+ gold = {str(item) for item in (source.get("gold_doc_ids") or [])}
+ grades = source.get("gold_grades") or {}
+ if isinstance(grades, dict):
+ gold |= {
+ str(doc_id)
+ for doc_id, gain in grades.items()
+ if float(gain or 0) > 0
+ }
+ for doc_id in row.get("hit_ids") or []:
+ key = str(doc_id)
+ if key in gold or key in seen:
+ continue
+ text = texts.get(key) or ""
+ if not text:
+ continue
+ seen.add(key)
+ bank.append(text)
+ return bank
+
+
+def retrieve_dense(
+ model: Any,
+ queries: list[dict[str, Any]],
+ docs: list[dict[str, Any]],
+ *,
+ k: int,
+ batch_size: int = 32,
+) -> tuple[dict[str, list[str]], float]:
+ import time
+
+ import numpy as np
+
+ texts = doc_texts(docs)
+ doc_ids = [str(doc.get("doc_id") or "") for doc in docs if doc.get("doc_id")]
+ doc_passages = [texts.get(doc_id) or doc_id for doc_id in doc_ids]
+ start = time.perf_counter()
+ doc_emb = model.encode(
+ doc_passages,
+ batch_size=batch_size,
+ convert_to_numpy=True,
+ normalize_embeddings=True,
+ show_progress_bar=False,
+ )
+ q_texts = [str(query.get("query") or "") for query in queries]
+ q_emb = model.encode(
+ q_texts,
+ batch_size=batch_size,
+ convert_to_numpy=True,
+ normalize_embeddings=True,
+ show_progress_bar=False,
+ )
+ elapsed_ms = (time.perf_counter() - start) * 1000.0
+ scores = np.matmul(q_emb, doc_emb.T)
+ ranked_by_qid: dict[str, list[str]] = {}
+ cut = max(1, min(int(k), len(doc_ids)))
+ for index, query in enumerate(queries):
+ order = np.argsort(-scores[index])[:cut]
+ qid = str(query.get("qid") or "")
+ ranked_by_qid[qid] = [doc_ids[int(pos)] for pos in order]
+ return ranked_by_qid, elapsed_ms
+
+
+def evaluate_dense(
+ ranked_by_qid: dict[str, list[str]],
+ queries: list[dict[str, Any]],
+ *,
+ ks: tuple[int, ...],
+ encode_ms: float,
+) -> tuple[dict[str, Any], list[dict[str, Any]]]:
+ per_query: list[dict[str, Any]] = []
+ per_q_ms = encode_ms / max(1, len(queries))
+ for query in queries:
+ qid = str(query.get("qid") or "")
+ ranked = ranked_by_qid.get(qid) or []
+ metrics = score_ranked_first_stage(ranked, query, ks=ks)
+ per_query.append(
+ {
+ "qid": qid,
+ "query": query.get("query"),
+ "slice": query.get("slice") or "unspecified",
+ "gold_doc_ids": list(query.get("gold_doc_ids") or []),
+ "gold_grades": query.get("gold_grades") or {},
+ "hit_ids": ranked,
+ "metrics": metrics,
+ "n_hits": len(ranked),
+ "retrieve_ms": per_q_ms,
+ "embed_ms": per_q_ms,
+ "client_wall_ms": per_q_ms,
+ }
+ )
+ metrics = aggregate_query_metrics(per_query, ks=ks) if per_query else {}
+ return metrics, per_query
+
+
+def run_local_dense(
+ rows: list[dict[str, Any]],
+ *,
+ model_name: str,
+ dataset_name: str,
+ k: int = 50,
+ ks: tuple[int, ...] = (5, 10, 20, 50),
+ brain_id: str | None = None,
+ protocol: str | None = None,
+) -> dict[str, Any]:
+ from sentence_transformers import SentenceTransformer
+
+ docs, queries = split_corpus(rows)
+ model = SentenceTransformer(model_name)
+ ranked_by_qid, encode_ms = retrieve_dense(model, queries, docs, k=k)
+ metrics, per_query = evaluate_dense(
+ ranked_by_qid, queries, ks=ks, encode_ms=encode_ms
+ )
+ return {
+ "status": "ok",
+ "brain_id": brain_id or "harness-local",
+ "dataset": dataset_name,
+ "fusion": "none",
+ "rerank": "none",
+ "channels": ["harness-dense"],
+ "rank_pool": False,
+ "skip_enrichment": True,
+ "k": k,
+ "ks": list(ks),
+ "n_docs": len(docs),
+ "n_queries": len(queries),
+ "n_docs_mapped": len(docs),
+ "ce_model": model_name,
+ "protocol": protocol
+ or (
+ "local dual-encoder over search_esci_74.jsonl; not BrainAPI embeddings; "
+ "not Reddy 0.857"
+ ),
+ "metrics": metrics,
+ "queries": per_query,
+ "ingest": {"status": "completed", "n_docs": 0, "tasks": [], "reused": True},
+ }
diff --git a/benchmarks/search/ltr_head.py b/benchmarks/search/ltr_head.py
new file mode 100644
index 0000000..aadcff0
--- /dev/null
+++ b/benchmarks/search/ltr_head.py
@@ -0,0 +1,700 @@
+from __future__ import annotations
+
+import json
+import math
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from search.dataset import split_corpus
+from search.local_dense import evaluate_dense
+from search.pool_first_stage import field_value, tokenize
+
+PROTOCOL = "ltr-head-cv"
+PROTOCOL_APPLY = "ltr-head-apply"
+PAIR_UNLABELED_ZERO = "unlabeled_zero"
+PAIR_OTHER_QUERY_NEG = "other_query_neg"
+PAIR_POLICIES = (PAIR_UNLABELED_ZERO, PAIR_OTHER_QUERY_NEG)
+BASE_FEATURE_NAMES = (
+ "rrf_inv",
+ "bm25_inv",
+ "dense_inv",
+ "title_overlap",
+ "brand_hit",
+ "query_in_title",
+)
+CE_FEATURE = "ce_gain"
+FEATURE_NAMES = BASE_FEATURE_NAMES
+HEAD_RANKNET = "ranknet"
+HEAD_LIGHTGBM = "lightgbm"
+HEADS = (HEAD_RANKNET, HEAD_LIGHTGBM)
+N_FOLDS = 5
+SEED = 0
+EPOCHS = 40
+LR = 0.05
+L2 = 1e-3
+MAX_PAIRS = 400
+LGBM_N_ESTIMATORS = 100
+LGBM_MAX_DEPTH = 3
+LGBM_LEARNING_RATE = 0.05
+
+
+def feature_names(*, with_ce: bool = False) -> tuple[str, ...]:
+ if with_ce:
+ return (*BASE_FEATURE_NAMES, CE_FEATURE)
+ return BASE_FEATURE_NAMES
+
+
+def _gain(row: dict[str, Any], doc_id: str) -> float:
+ grades = row.get("gold_grades") or {}
+ if isinstance(grades, dict) and doc_id in grades:
+ return float(grades.get(doc_id) or 0.0)
+ gold = {str(item) for item in (row.get("gold_doc_ids") or []) if item}
+ return 1.0 if doc_id in gold else 0.0
+
+
+def _chunk_to_doc(row: dict[str, Any]) -> dict[str, str]:
+ mapping: dict[str, str] = {}
+ for hit in row.get("hits") or []:
+ chunk_id = str(hit.get("id") or "")
+ doc_id = str(hit.get("doc_id") or "")
+ if chunk_id and doc_id:
+ mapping[chunk_id] = doc_id
+ return mapping
+
+
+def _as_docs(ids: list[str], mapping: dict[str, str]) -> list[str]:
+ out: list[str] = []
+ seen: set[str] = set()
+ for raw in ids:
+ key = mapping.get(str(raw), str(raw))
+ if not key or key in seen:
+ continue
+ seen.add(key)
+ out.append(key)
+ return out
+
+
+def _rank_inv(ids: list[str], doc_id: str) -> float:
+ try:
+ return 1.0 / (ids.index(doc_id) + 1)
+ except ValueError:
+ return 0.0
+
+
+def _title(doc: dict[str, Any]) -> str:
+ titled = str(doc.get("title") or "").strip()
+ if titled:
+ return titled
+ return field_value(str(doc.get("text") or ""), "Title")
+
+
+def _brand(doc: dict[str, Any]) -> str:
+ branded = str(doc.get("brand") or "").strip()
+ if branded:
+ return branded
+ return field_value(str(doc.get("text") or ""), "Brand")
+
+
+def overlap(query: str, text: str) -> float:
+ qtoks = [tok for tok in tokenize(query) if len(tok) >= 3]
+ if not qtoks:
+ return 0.0
+ ttoks = set(tokenize(text))
+ return len(set(qtoks) & ttoks) / float(len(set(qtoks)))
+
+
+def ce_cache_name(model_dir: Path) -> str:
+ stem = model_dir.name.strip() or "ce"
+ safe = "".join(ch if ch.isalnum() or ch in "-_" else "-" for ch in stem)
+ return f"ce_gain_{safe}.json"
+
+
+def load_ce_cache(path: Path) -> dict[str, dict[str, float]] | None:
+ if not path.exists():
+ return None
+ raw = json.loads(path.read_text(encoding="utf-8"))
+ if not isinstance(raw, dict):
+ return None
+ out: dict[str, dict[str, float]] = {}
+ for qid, scores in raw.items():
+ if not isinstance(scores, dict):
+ continue
+ out[str(qid)] = {str(doc_id): float(value) for doc_id, value in scores.items()}
+ return out
+
+
+def write_ce_cache(path: Path, scores: dict[str, dict[str, float]]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(json.dumps(scores, indent=2), encoding="utf-8")
+
+
+def score_ce_gains(
+ eval_result: dict[str, Any],
+ docs_by_id: dict[str, dict[str, Any]],
+ *,
+ model_dir: Path,
+ k: int = 50,
+ cache_path: Path | None = None,
+) -> dict[str, dict[str, float]]:
+ if cache_path is not None:
+ cached = load_ce_cache(cache_path)
+ if cached:
+ return cached
+ from search.finetune_esci_4class import CLASS_GAINS, weighted_scores
+ from search.rank_pool import doc_texts
+ from search.rank_pool_4class import load_4class_predict
+
+ predict, _model = load_4class_predict(model_dir)
+ texts = doc_texts(list(docs_by_id.values()))
+ scored: dict[str, dict[str, float]] = {}
+ rows = list(eval_result.get("queries") or [])
+ for index, row in enumerate(rows):
+ qid = str(row.get("qid") or "")
+ query = str(row.get("query") or "")
+ mapping = _chunk_to_doc(row)
+ hit_ids = [str(item) for item in (row.get("hit_ids") or []) if item][:k]
+ doc_ids = _as_docs(hit_ids, mapping)
+ if not doc_ids:
+ doc_ids = hit_ids
+ pairs: list[tuple[str, str]] = []
+ for doc_id in doc_ids:
+ text = texts.get(doc_id) or _title(docs_by_id.get(doc_id) or {})
+ pairs.append((query, text))
+ probs = predict(pairs) if pairs else []
+ gains = weighted_scores(probs, CLASS_GAINS) if pairs else []
+ if len(gains) < len(doc_ids):
+ gains = list(gains) + [0.0] * (len(doc_ids) - len(gains))
+ scored[qid] = {
+ doc_id: float(gains[pos]) for pos, doc_id in enumerate(doc_ids)
+ }
+ print(f"ltr-ce {index + 1}/{len(rows)} {qid}", flush=True)
+ if cache_path is not None:
+ write_ce_cache(cache_path, scored)
+ return scored
+
+
+def features_for_doc(
+ query: str,
+ doc_id: str,
+ *,
+ rrf_ids: list[str],
+ bm25_ids: list[str],
+ dense_ids: list[str],
+ doc: dict[str, Any] | None,
+ ce_gain: float | None = None,
+) -> np.ndarray:
+ title = _title(doc or {})
+ brand = _brand(doc or {})
+ q = (query or "").strip().lower()
+ in_title = 1.0 if q and len(q) >= 3 and q in title.lower() else 0.0
+ qtoks = [tok for tok in tokenize(query) if len(tok) >= 3]
+ brand_toks = set(tokenize(brand))
+ brand_hit = 1.0 if qtoks and set(qtoks) & brand_toks else 0.0
+ values = [
+ _rank_inv(rrf_ids, doc_id),
+ _rank_inv(bm25_ids, doc_id),
+ _rank_inv(dense_ids, doc_id),
+ overlap(query, title),
+ brand_hit,
+ in_title,
+ ]
+ if ce_gain is not None:
+ values.append(float(ce_gain))
+ return np.array(values, dtype=np.float64)
+
+
+def example_from_eval_row(
+ row: dict[str, Any],
+ docs_by_id: dict[str, dict[str, Any]],
+ *,
+ k: int = 50,
+ ce_scores: dict[str, float] | None = None,
+) -> dict[str, Any] | None:
+ mapping = _chunk_to_doc(row)
+ rrf_ids = [str(item) for item in (row.get("hit_ids") or []) if item][:k]
+ if not rrf_ids:
+ return None
+ bm25_ids = _as_docs(list(row.get("bm25_ids") or []), mapping)
+ dense_ids = _as_docs(list(row.get("dense_ids") or []), mapping)
+ query = str(row.get("query") or "")
+ feats: list[np.ndarray] = []
+ gains: list[float] = []
+ for doc_id in rrf_ids:
+ ce_gain = None
+ if ce_scores is not None:
+ ce_gain = float(ce_scores.get(doc_id) or 0.0)
+ feats.append(
+ features_for_doc(
+ query,
+ doc_id,
+ rrf_ids=rrf_ids,
+ bm25_ids=bm25_ids,
+ dense_ids=dense_ids,
+ doc=docs_by_id.get(doc_id),
+ ce_gain=ce_gain,
+ )
+ )
+ gains.append(_gain(row, doc_id))
+ return {
+ "qid": str(row.get("qid") or ""),
+ "query": query,
+ "ids": rrf_ids,
+ "features": np.stack(feats, axis=0),
+ "gains": np.array(gains, dtype=np.float64),
+ "row": row,
+ }
+
+
+def other_query_gold_ids(examples: list[dict[str, Any]], skip_qid: str) -> set[str]:
+ gold: set[str] = set()
+ for example in examples:
+ if example["qid"] == skip_qid:
+ continue
+ for doc_id, gain in zip(example["ids"], example["gains"]):
+ if float(gain) > 0:
+ gold.add(str(doc_id))
+ return gold
+
+
+def training_gains(
+ example: dict[str, Any],
+ *,
+ pair_policy: str,
+ other_gold: set[str],
+) -> list[float | None]:
+ policy = (pair_policy or PAIR_UNLABELED_ZERO).strip()
+ out: list[float | None] = []
+ for doc_id, gain in zip(example["ids"], example["gains"]):
+ value = float(gain)
+ if policy == PAIR_OTHER_QUERY_NEG:
+ if value > 0:
+ out.append(value)
+ elif str(doc_id) in other_gold:
+ out.append(0.0)
+ else:
+ out.append(None)
+ else:
+ out.append(value)
+ return out
+
+
+def collect_pairs(
+ example: dict[str, Any],
+ *,
+ pair_policy: str = PAIR_UNLABELED_ZERO,
+ other_gold: set[str] | None = None,
+) -> list[tuple[int, int]]:
+ labeled = training_gains(
+ example,
+ pair_policy=pair_policy,
+ other_gold=other_gold or set(),
+ )
+ eligible = [
+ (index, gain)
+ for index, gain in enumerate(labeled)
+ if gain is not None
+ ]
+ pairs: list[tuple[int, int]] = []
+ for left in range(len(eligible)):
+ for right in range(left):
+ left_i, left_g = eligible[left]
+ right_i, right_g = eligible[right]
+ if left_g == right_g:
+ continue
+ hi, lo = (
+ (left_i, right_i) if left_g > right_g else (right_i, left_i)
+ )
+ pairs.append((hi, lo))
+ return pairs
+
+
+def _sigmoid(value: float) -> float:
+ if value >= 0:
+ z = math.exp(-value)
+ return 1.0 / (1.0 + z)
+ z = math.exp(value)
+ return z / (1.0 + z)
+
+
+def fit_pairwise(
+ examples: list[dict[str, Any]],
+ *,
+ epochs: int = EPOCHS,
+ lr: float = LR,
+ l2: float = L2,
+ seed: int = SEED,
+ pair_policy: str = PAIR_UNLABELED_ZERO,
+) -> np.ndarray:
+ n_features = int(examples[0]["features"].shape[1]) if examples else len(BASE_FEATURE_NAMES)
+ rng = np.random.default_rng(seed)
+ weights = np.zeros(n_features, dtype=np.float64)
+ if not examples:
+ return weights
+ for _ in range(max(1, int(epochs))):
+ order = rng.permutation(len(examples))
+ for index in order:
+ example = examples[int(index)]
+ feats = example["features"]
+ other_gold = other_query_gold_ids(examples, example["qid"])
+ pairs = collect_pairs(
+ example,
+ pair_policy=pair_policy,
+ other_gold=other_gold,
+ )
+ if not pairs:
+ continue
+ if len(pairs) > MAX_PAIRS:
+ chosen = rng.choice(len(pairs), size=MAX_PAIRS, replace=False)
+ pairs = [pairs[int(pos)] for pos in chosen]
+ for hi, lo in pairs:
+ diff = feats[hi] - feats[lo]
+ pred = _sigmoid(float(weights.dot(diff)))
+ grad = (pred - 1.0) * diff + l2 * weights
+ weights -= lr * grad
+ return weights
+
+
+def score_docs(features: np.ndarray, weights: np.ndarray) -> np.ndarray:
+ return features.dot(weights)
+
+
+def rerank_ids(example: dict[str, Any], weights: np.ndarray) -> list[str]:
+ scores = score_docs(example["features"], weights)
+ order = np.argsort(-scores, kind="stable")
+ return [example["ids"][int(pos)] for pos in order]
+
+
+def gain_to_rank_label(gain: float) -> int:
+ if gain >= 0.99:
+ return 3
+ if gain >= 0.09:
+ return 2
+ if gain >= 0.005:
+ return 1
+ return 0
+
+
+def rank_train_group(
+ example: dict[str, Any],
+ *,
+ pair_policy: str,
+ other_gold: set[str],
+) -> tuple[np.ndarray, np.ndarray] | None:
+ labeled = training_gains(
+ example,
+ pair_policy=pair_policy,
+ other_gold=other_gold,
+ )
+ idxs = [index for index, gain in enumerate(labeled) if gain is not None]
+ if len(idxs) < 2:
+ return None
+ y = np.array([gain_to_rank_label(float(labeled[index] or 0.0)) for index in idxs])
+ if int(np.unique(y).size) < 2:
+ return None
+ return example["features"][idxs], y
+
+
+class _ZeroRanker:
+ def __init__(self, n_features: int) -> None:
+ self.feature_importances_ = np.zeros(max(1, int(n_features)), dtype=np.float64)
+
+ def predict(self, features: np.ndarray) -> np.ndarray:
+ return np.zeros(len(features), dtype=np.float64)
+
+
+class _LgbmRanker:
+ def __init__(self, booster: Any, n_features: int) -> None:
+ self.booster = booster
+ raw = booster.feature_importance(importance_type="gain")
+ importances = np.asarray(raw, dtype=np.float64)
+ if importances.size < n_features:
+ importances = np.pad(importances, (0, n_features - int(importances.size)))
+ self.feature_importances_ = importances[:n_features]
+
+ def predict(self, features: np.ndarray) -> np.ndarray:
+ return np.asarray(self.booster.predict(features), dtype=np.float64)
+
+
+def fit_lightgbm(
+ examples: list[dict[str, Any]],
+ *,
+ pair_policy: str = PAIR_UNLABELED_ZERO,
+ seed: int = SEED,
+ min_data_in_leaf: int | None = None,
+) -> Any:
+ n_features = (
+ int(examples[0]["features"].shape[1]) if examples else len(BASE_FEATURE_NAMES)
+ )
+ xs: list[np.ndarray] = []
+ ys: list[np.ndarray] = []
+ groups: list[int] = []
+ for example in examples:
+ group = rank_train_group(
+ example,
+ pair_policy=pair_policy,
+ other_gold=other_query_gold_ids(examples, example["qid"]),
+ )
+ if group is None:
+ continue
+ feats, labels = group
+ xs.append(feats)
+ ys.append(labels)
+ groups.append(int(len(labels)))
+ if not groups:
+ return _ZeroRanker(n_features)
+ import lightgbm as lgb
+
+ dataset = lgb.Dataset(
+ np.vstack(xs),
+ label=np.concatenate(ys),
+ group=groups,
+ free_raw_data=False,
+ )
+ params: dict[str, Any] = {
+ "objective": "lambdarank",
+ "max_depth": LGBM_MAX_DEPTH,
+ "learning_rate": LGBM_LEARNING_RATE,
+ "num_leaves": 8,
+ "verbosity": -1,
+ "seed": int(seed),
+ }
+ if min_data_in_leaf is not None:
+ params["min_data_in_leaf"] = int(min_data_in_leaf)
+ booster = lgb.train(
+ params,
+ dataset,
+ num_boost_round=LGBM_N_ESTIMATORS,
+ )
+ return _LgbmRanker(booster, n_features)
+
+
+def rerank_ids_model(example: dict[str, Any], model: Any) -> list[str]:
+ scores = np.asarray(model.predict(example["features"]), dtype=np.float64)
+ order = np.argsort(-scores, kind="stable")
+ return [example["ids"][int(pos)] for pos in order]
+
+
+def overlap_only_ids(example: dict[str, Any]) -> list[str]:
+ title_scores = example["features"][:, 3]
+ order = np.argsort(-title_scores, kind="stable")
+ return [example["ids"][int(pos)] for pos in order]
+
+
+def grouped_folds(qids: list[str], n_folds: int = N_FOLDS) -> list[set[str]]:
+ folds: list[set[str]] = [set() for _ in range(max(2, int(n_folds)))]
+ for index, qid in enumerate(sorted(qids)):
+ folds[index % len(folds)].add(qid)
+ return folds
+
+
+def build_ltr_examples(
+ eval_result: dict[str, Any],
+ rows: list[dict[str, Any]],
+ *,
+ k: int = 50,
+ ce_model: Path | None = None,
+ ce_cache_path: Path | None = None,
+ ce_scores: dict[str, dict[str, float]] | None = None,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, dict[str, Any]]]:
+ docs, queries = split_corpus(rows)
+ docs_by_id = {str(doc.get("doc_id") or ""): doc for doc in docs if doc.get("doc_id")}
+ resolved_ce = ce_scores
+ ce_model_path = Path(ce_model) if ce_model else None
+ if resolved_ce is None and ce_model_path is not None:
+ resolved_ce = score_ce_gains(
+ eval_result,
+ docs_by_id,
+ model_dir=ce_model_path,
+ k=k,
+ cache_path=ce_cache_path,
+ )
+ with_ce = resolved_ce is not None
+ examples: list[dict[str, Any]] = []
+ for row in eval_result.get("queries") or []:
+ qid = str(row.get("qid") or "")
+ example = example_from_eval_row(
+ row,
+ docs_by_id,
+ k=k,
+ ce_scores=(resolved_ce or {}).get(qid) if with_ce else None,
+ )
+ if example is None:
+ continue
+ examples.append(example)
+ return examples, queries, docs_by_id
+
+
+def run_ltr_head(
+ eval_result: dict[str, Any],
+ rows: list[dict[str, Any]],
+ *,
+ dataset_name: str,
+ k: int = 50,
+ ks: tuple[int, ...] = (5, 10, 20, 50),
+ n_folds: int = N_FOLDS,
+ brain_id: str | None = None,
+ source_run: str | None = None,
+ pair_policy: str = PAIR_UNLABELED_ZERO,
+ ce_model: Path | None = None,
+ ce_cache_path: Path | None = None,
+ ce_scores: dict[str, dict[str, float]] | None = None,
+ ltr_head: str = HEAD_RANKNET,
+ train_eval_result: dict[str, Any] | None = None,
+ train_rows: list[dict[str, Any]] | None = None,
+ train_source_run: str | None = None,
+ train_ce_cache_path: Path | None = None,
+ train_ce_scores: dict[str, dict[str, float]] | None = None,
+) -> dict[str, Any]:
+ policy = (pair_policy or PAIR_UNLABELED_ZERO).strip()
+ if policy not in PAIR_POLICIES:
+ raise ValueError(f"Unknown pair policy {pair_policy!r}")
+ head = (ltr_head or HEAD_RANKNET).strip()
+ if head not in HEADS:
+ raise ValueError(f"Unknown LTR head {ltr_head!r}")
+ if train_eval_result is not None and train_rows is None:
+ raise ValueError("train_rows is required when train_eval_result is set")
+ apply_mode = train_eval_result is not None
+ ce_model_path = Path(ce_model) if ce_model else None
+ eval_examples, queries, docs_by_id = build_ltr_examples(
+ eval_result,
+ rows,
+ k=k,
+ ce_model=ce_model_path,
+ ce_cache_path=ce_cache_path,
+ ce_scores=ce_scores,
+ )
+ names = feature_names(with_ce=ce_model_path is not None or ce_scores is not None)
+ if apply_mode:
+ train_examples, _, _ = build_ltr_examples(
+ train_eval_result or {},
+ train_rows or [],
+ k=k,
+ ce_model=ce_model_path,
+ ce_cache_path=train_ce_cache_path,
+ ce_scores=train_ce_scores,
+ )
+ names = feature_names(
+ with_ce=bool(train_examples)
+ and int(train_examples[0]["features"].shape[1]) > len(BASE_FEATURE_NAMES)
+ )
+ ranked_cv: dict[str, list[str]] = {}
+ ranked_overlap: dict[str, list[str]] = {}
+ fold_weights: list[list[float]] = []
+ if head == HEAD_LIGHTGBM:
+ model = fit_lightgbm(train_examples, pair_policy=policy)
+ fold_weights.append([float(item) for item in model.feature_importances_])
+ for example in eval_examples:
+ ranked_cv[example["qid"]] = rerank_ids_model(example, model)[:k]
+ ranked_overlap[example["qid"]] = overlap_only_ids(example)[:k]
+ else:
+ weights = fit_pairwise(train_examples, pair_policy=policy)
+ fold_weights.append([float(item) for item in weights])
+ for example in eval_examples:
+ ranked_cv[example["qid"]] = rerank_ids(example, weights)[:k]
+ ranked_overlap[example["qid"]] = overlap_only_ids(example)[:k]
+ n_train = len(train_examples)
+ else:
+ examples = eval_examples
+ by_qid = {example["qid"]: example for example in examples}
+ qids = [example["qid"] for example in examples]
+ ranked_cv = {}
+ ranked_overlap = {}
+ fold_weights = []
+ for test_qids in grouped_folds(qids, n_folds=n_folds):
+ train_examples = [
+ example for example in examples if example["qid"] not in test_qids
+ ]
+ if head == HEAD_LIGHTGBM:
+ model = fit_lightgbm(train_examples, pair_policy=policy)
+ fold_weights.append([float(item) for item in model.feature_importances_])
+ for qid in test_qids:
+ example = by_qid.get(qid)
+ if example is None:
+ continue
+ ranked_cv[qid] = rerank_ids_model(example, model)[:k]
+ ranked_overlap[qid] = overlap_only_ids(example)[:k]
+ continue
+ weights = fit_pairwise(train_examples, pair_policy=policy)
+ fold_weights.append([float(item) for item in weights])
+ for qid in test_qids:
+ example = by_qid.get(qid)
+ if example is None:
+ continue
+ ranked_cv[qid] = rerank_ids(example, weights)[:k]
+ ranked_overlap[qid] = overlap_only_ids(example)[:k]
+ n_train = None
+ train_examples = []
+ metrics, per_query = evaluate_dense(
+ ranked_cv,
+ queries,
+ ks=ks,
+ encode_ms=0.0,
+ )
+ overlap_metrics, _ = evaluate_dense(
+ ranked_overlap,
+ queries,
+ ks=ks,
+ encode_ms=0.0,
+ )
+ mean_weights = (
+ np.mean(np.array(fold_weights, dtype=np.float64), axis=0)
+ if fold_weights
+ else np.zeros(len(names))
+ )
+ return {
+ "status": "ok" if per_query else "failed",
+ "brain_id": brain_id or "harness-local-ltr",
+ "dataset": dataset_name,
+ "fusion": "none",
+ "rerank": "ltr-lightgbm" if head == HEAD_LIGHTGBM else "ltr-pairwise",
+ "channels": ["harness-ltr-head"],
+ "expand": "none",
+ "k": k,
+ "ks": list(ks),
+ "n_docs": len(docs_by_id),
+ "n_queries": len(per_query),
+ "n_docs_mapped": 0,
+ "skip_enrichment": True,
+ "ingest_graph": False,
+ "skip_ingest": True,
+ "rank_pool": False,
+ "ltr_from_run": source_run or eval_result.get("run_id"),
+ "ltr_train_run": train_source_run if apply_mode else None,
+ "ltr_n_train_queries": n_train if apply_mode else None,
+ "protocol": PROTOCOL_APPLY if apply_mode else PROTOCOL,
+ "ltr_features": list(names),
+ "ltr_pair_policy": policy,
+ "ltr_head": head,
+ "ltr_ce_model": str(ce_model_path) if ce_model_path else None,
+ "ltr_lgbm": (
+ {
+ "n_estimators": LGBM_N_ESTIMATORS,
+ "max_depth": LGBM_MAX_DEPTH,
+ "learning_rate": LGBM_LEARNING_RATE,
+ }
+ if head == HEAD_LIGHTGBM
+ else None
+ ),
+ "ltr_n_folds": 1 if apply_mode else n_folds,
+ "ltr_seed": SEED,
+ "ltr_epochs": EPOCHS,
+ "ltr_mean_weights": {
+ name: float(weight)
+ for name, weight in zip(names, mean_weights)
+ },
+ "overlap_only_metrics": overlap_metrics,
+ "ingest": {
+ "status": "completed",
+ "n_docs": len(docs_by_id),
+ "tasks": [],
+ "reused": True,
+ },
+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "event_probe": None,
+ "search_error": None,
+ "metrics": metrics,
+ "queries": per_query,
+ }
diff --git a/benchmarks/search/mapping.py b/benchmarks/search/mapping.py
new file mode 100644
index 0000000..763edb0
--- /dev/null
+++ b/benchmarks/search/mapping.py
@@ -0,0 +1,74 @@
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+_ROOT = Path(__file__).resolve().parents[2]
+if str(_ROOT) not in sys.path:
+ sys.path.insert(0, str(_ROOT))
+
+from src.core.search.catalog_graph import ( # noqa: E402
+ ATTR_LABEL,
+ CLASS_LABEL,
+ ENTITY_TYPE,
+ EVENT_LABEL,
+ FEATURE_KEY_CAP,
+ HAS_EVENT_NAME,
+ TYPE_LABEL,
+ USER_TYPE,
+ compose_search_text,
+ doc_fields,
+ doc_id_from_text,
+ doc_to_triples,
+ docs_to_triples,
+ catalog_entity_backfill_rows,
+ entity_search_text,
+ entity_uuid,
+ node_id_from_passage_text,
+ event_uuid,
+ format_happened_at,
+ has_triple,
+ is_static_has_triple,
+ hub_uuid,
+ interaction_to_triples,
+ interactions_to_triples,
+ load_interaction_rows,
+ parse_feature_string,
+ rel_uuid,
+ sanitize_label,
+ split_hierarchy,
+ structured_ingest_body,
+)
+
+__all__ = [
+ "ATTR_LABEL",
+ "CLASS_LABEL",
+ "ENTITY_TYPE",
+ "EVENT_LABEL",
+ "FEATURE_KEY_CAP",
+ "HAS_EVENT_NAME",
+ "TYPE_LABEL",
+ "USER_TYPE",
+ "compose_search_text",
+ "doc_fields",
+ "doc_id_from_text",
+ "doc_to_triples",
+ "docs_to_triples",
+ "catalog_entity_backfill_rows",
+ "entity_search_text",
+ "entity_uuid",
+ "node_id_from_passage_text",
+ "event_uuid",
+ "format_happened_at",
+ "has_triple",
+ "is_static_has_triple",
+ "hub_uuid",
+ "interaction_to_triples",
+ "interactions_to_triples",
+ "load_interaction_rows",
+ "parse_feature_string",
+ "rel_uuid",
+ "sanitize_label",
+ "split_hierarchy",
+ "structured_ingest_body",
+]
diff --git a/benchmarks/search/metrics.py b/benchmarks/search/metrics.py
new file mode 100644
index 0000000..91e9079
--- /dev/null
+++ b/benchmarks/search/metrics.py
@@ -0,0 +1,180 @@
+from __future__ import annotations
+
+import math
+from typing import Any, Iterable, Sequence
+
+
+def recall_at_k(
+ ranked_ids: Sequence[str],
+ gold: Iterable[str],
+ k: int,
+) -> float:
+ gold_set = {str(item) for item in gold}
+ if not gold_set:
+ return 0.0
+ retrieved = {str(item) for item in list(ranked_ids)[:k]}
+ return len(retrieved & gold_set) / len(gold_set)
+
+
+def mrr(ranked_ids: Sequence[str], gold: Iterable[str]) -> float:
+ gold_set = {str(item) for item in gold}
+ if not gold_set:
+ return 0.0
+ for index, item in enumerate(ranked_ids, start=1):
+ if str(item) in gold_set:
+ return 1.0 / index
+ return 0.0
+
+
+def _dcg(relevances: Sequence[float], k: int) -> float:
+ score = 0.0
+ for index, rel in enumerate(list(relevances)[:k], start=1):
+ score += float(rel) / math.log2(index + 1)
+ return score
+
+
+def ndcg_at_k(
+ ranked_ids: Sequence[str],
+ gold: Iterable[str],
+ k: int,
+ grades: dict[str, float] | None = None,
+) -> float:
+ if grades:
+ gain = {
+ str(doc_id): float(value)
+ for doc_id, value in grades.items()
+ if float(value) > 0
+ }
+ if not gain:
+ return 0.0
+ relevances = [gain.get(str(item), 0.0) for item in list(ranked_ids)[:k]]
+ ideal = sorted(gain.values(), reverse=True)[:k]
+ dcg = _dcg(relevances, k)
+ idcg = _dcg(ideal, k)
+ if idcg <= 0:
+ return 0.0
+ return dcg / idcg
+ gold_set = {str(item) for item in gold}
+ if not gold_set:
+ return 0.0
+ relevances = [1.0 if str(item) in gold_set else 0.0 for item in list(ranked_ids)[:k]]
+ dcg = _dcg(relevances, k)
+ ideal = [1.0] * min(len(gold_set), k)
+ idcg = _dcg(ideal, k)
+ if idcg <= 0:
+ return 0.0
+ return dcg / idcg
+
+
+def percentile(values: Sequence[float], p: float) -> float | None:
+ if not values:
+ return None
+ xs = sorted(float(v) for v in values)
+ if len(xs) == 1:
+ return xs[0]
+ rank = (len(xs) - 1) * (p / 100.0)
+ low = math.floor(rank)
+ high = math.ceil(rank)
+ if low == high:
+ return xs[int(rank)]
+ return xs[low] * (high - rank) + xs[high] * (rank - low)
+
+
+def _stage_wall_ms(stage_timings: dict[str, Any] | None, name: str) -> float | None:
+ stages = (stage_timings or {}).get("stages") or []
+ for stage in stages:
+ if not isinstance(stage, dict):
+ continue
+ if stage.get("stage") == name and stage.get("wall_ms") is not None:
+ return float(stage["wall_ms"])
+ return None
+
+
+def retrieve_latency_ms(
+ stage_timings: dict[str, Any] | None,
+ client_wall_ms: float,
+) -> tuple[float, float | None]:
+ retrieve_ms = _stage_wall_ms(stage_timings, "search.retrieve")
+ embed_ms = _stage_wall_ms(stage_timings, "embed.query")
+ if retrieve_ms is not None:
+ return retrieve_ms, embed_ms
+ if embed_ms is not None:
+ return max(0.0, float(client_wall_ms) - embed_ms), embed_ms
+ return float(client_wall_ms), embed_ms
+
+
+def mean(values: Sequence[float]) -> float:
+ if not values:
+ return 0.0
+ return sum(values) / len(values)
+
+
+def aggregate_query_metrics(
+ per_query: list[dict[str, Any]],
+ *,
+ ks: Sequence[int] = (5, 10, 20),
+) -> dict[str, Any]:
+ metrics: dict[str, Any] = {}
+ for k in ks:
+ key = f"recall@{k}"
+ metrics[key] = mean([float(row["metrics"][key]) for row in per_query])
+ ndcg_ks = tuple(dict.fromkeys((*ks, 10, 20)))
+ for k in ndcg_ks:
+ key = f"ndcg@{k}"
+ values = [
+ float(row["metrics"][key])
+ for row in per_query
+ if row.get("metrics", {}).get(key) is not None
+ ]
+ if values:
+ metrics[key] = mean(values)
+ ndcg_full = [
+ float(row["metrics"]["ndcg"])
+ for row in per_query
+ if row.get("metrics", {}).get("ndcg") is not None
+ ]
+ if ndcg_full:
+ metrics["ndcg"] = mean(ndcg_full)
+ metrics["mrr"] = mean([float(row["metrics"]["mrr"]) for row in per_query])
+
+ retrieve_ms = [float(row["retrieve_ms"]) for row in per_query]
+ embed_ms = [
+ float(row["embed_ms"])
+ for row in per_query
+ if row.get("embed_ms") is not None
+ ]
+ client_ms = [float(row["client_wall_ms"]) for row in per_query]
+ metrics["p50_retrieve_ms"] = percentile(retrieve_ms, 50)
+ metrics["p95_retrieve_ms"] = percentile(retrieve_ms, 95)
+ metrics["p50_embed_ms"] = percentile(embed_ms, 50)
+ metrics["p95_embed_ms"] = percentile(embed_ms, 95)
+ metrics["p50_client_wall_ms"] = percentile(client_ms, 50)
+ metrics["p95_client_wall_ms"] = percentile(client_ms, 95)
+
+ slices: dict[str, list[dict[str, Any]]] = {}
+ for row in per_query:
+ slices.setdefault(str(row.get("slice") or "unspecified"), []).append(row)
+ by_slice: dict[str, dict[str, Any]] = {}
+ for name, rows in slices.items():
+ by_slice[name] = {
+ "n_queries": len(rows),
+ "recall@10": mean([float(r["metrics"]["recall@10"]) for r in rows]),
+ "ndcg@10": mean([float(r["metrics"]["ndcg@10"]) for r in rows]),
+ "mrr": mean([float(r["metrics"]["mrr"]) for r in rows]),
+ }
+ for k in ks:
+ recall_key = f"recall@{k}"
+ if all(r.get("metrics", {}).get(recall_key) is not None for r in rows):
+ by_slice[name][recall_key] = mean(
+ [float(r["metrics"][recall_key]) for r in rows]
+ )
+ for k in ndcg_ks:
+ ndcg_key = f"ndcg@{k}"
+ if any(r.get("metrics", {}).get(ndcg_key) is not None for r in rows):
+ by_slice[name][ndcg_key] = mean(
+ [float(r["metrics"][ndcg_key]) for r in rows]
+ )
+ if any(r.get("metrics", {}).get("ndcg") is not None for r in rows):
+ by_slice[name]["ndcg"] = mean([float(r["metrics"]["ndcg"]) for r in rows])
+ metrics["by_slice"] = by_slice
+ return metrics
diff --git a/benchmarks/search/mine_retrieved_lists.py b/benchmarks/search/mine_retrieved_lists.py
new file mode 100644
index 0000000..5c2cc53
--- /dev/null
+++ b/benchmarks/search/mine_retrieved_lists.py
@@ -0,0 +1,269 @@
+from __future__ import annotations
+
+import argparse
+import json
+import random
+from pathlib import Path
+from typing import Any, Iterable, Sequence
+
+from search.catalog import (
+ ESCI_CACHE_DIR,
+ ESCI_EXAMPLES_FILE,
+ ESCI_PRODUCTS_FILE,
+ extra_fields_from_catalog,
+ _esci_products,
+ _iter_parquet,
+)
+from search.finetune_esci_4class import (
+ LABELS,
+ LABEL_TO_ID,
+ held_out_query_ids,
+ product_passage,
+)
+from search.pool_first_stage import Bm25Index, rank_docs, tokenize
+
+DEFAULT_OUT = Path("data/esci_retrieved_lists.jsonl")
+
+
+def is_held_out(qid: str, holdout: set[str]) -> bool:
+ text = str(qid or "").strip()
+ if not text:
+ return True
+ if text in holdout:
+ return True
+ if text.lower().startswith("esci-") and text[5:] in holdout:
+ return True
+ if f"esci-{text}" in holdout:
+ return True
+ return False
+
+
+def load_train_groups(
+ *,
+ holdout: set[str],
+ locale: str = "us",
+) -> dict[str, dict[str, Any]]:
+ examples_path = ESCI_CACHE_DIR / ESCI_EXAMPLES_FILE
+ wanted_locale = locale.strip().lower()
+ groups: dict[str, dict[str, Any]] = {}
+ for item in _iter_parquet(examples_path):
+ if int(item.get("small_version") or 0) != 1:
+ continue
+ if str(item.get("product_locale") or "").strip().lower() != wanted_locale:
+ continue
+ if str(item.get("split") or "").strip().lower() != "train":
+ continue
+ qid = str(item.get("query_id") or "").strip()
+ if is_held_out(qid, holdout):
+ continue
+ pid = str(item.get("product_id") or "").strip()
+ query = str(item.get("query") or "").strip()
+ label = str(item.get("esci_label") or "").strip().upper()
+ if not pid or not query or label not in LABEL_TO_ID:
+ continue
+ group = groups.setdefault(qid, {"query": query, "qrels": {}})
+ group["query"] = query
+ group["qrels"][pid] = label
+ return groups
+
+
+def select_groups(
+ groups: dict[str, dict[str, Any]],
+ *,
+ holdout: set[str],
+ max_queries: int,
+ seed: int = 11,
+) -> dict[str, dict[str, Any]]:
+ kept = {
+ qid: group
+ for qid, group in groups.items()
+ if not is_held_out(qid, holdout)
+ }
+ ids = list(kept)
+ rng = random.Random(seed)
+ rng.shuffle(ids)
+ cap = max(1, int(max_queries))
+ chosen = ids[:cap]
+ return {qid: kept[qid] for qid in chosen}
+
+
+def labeled_hits(
+ ranked: Sequence[str],
+ qrels: dict[str, str],
+ *,
+ k: int = 50,
+) -> list[tuple[str, str]]:
+ pairs: list[tuple[str, str]] = []
+ for pid in ranked:
+ key = str(pid)
+ label = str(qrels.get(key) or "I").strip().upper()
+ if label not in LABEL_TO_ID:
+ label = "I"
+ pairs.append((key, label))
+ cut = max(1, int(k))
+ top = pairs[:cut]
+ if any(label == "I" for _, label in top):
+ return top
+ for pid, label in pairs[cut:]:
+ if label == "I":
+ if top:
+ top[-1] = (pid, label)
+ else:
+ top.append((pid, label))
+ break
+ return top
+
+
+def passages_for_pids(
+ pids: Iterable[str],
+ *,
+ locale: str = "us",
+) -> dict[str, str]:
+ needed = {str(pid) for pid in pids if str(pid)}
+ products = _esci_products(
+ ESCI_CACHE_DIR / ESCI_PRODUCTS_FILE,
+ needed,
+ locale=locale,
+ )
+ extra_fields = extra_fields_from_catalog(
+ products,
+ title_key="product_title",
+ description_key="product_description",
+ )
+ out: dict[str, str] = {}
+ for pid in needed:
+ catalog = products.get(pid) or {"product_id": pid}
+ out[pid] = product_passage(catalog, extra_fields)
+ return out
+
+
+def mine_from_groups(
+ groups: dict[str, dict[str, Any]],
+ passages: dict[str, str],
+ *,
+ k: int = 50,
+) -> list[dict[str, Any]]:
+ docs: list[tuple[str, list[str]]] = []
+ for pid, text in passages.items():
+ tokens = tokenize(text)
+ if not tokens:
+ continue
+ docs.append((str(pid), tokens))
+ if not docs:
+ return []
+ index = Bm25Index(docs)
+ doc_ids = [pid for pid, _ in docs]
+ rows: list[dict[str, Any]] = []
+ n_groups = len(groups)
+ for offset, (qid, group) in enumerate(groups.items(), start=1):
+ query = str(group.get("query") or "")
+ qrels = {
+ str(pid): str(label).upper()
+ for pid, label in (group.get("qrels") or {}).items()
+ }
+ scores = index.scores(tokenize(query))
+ ranked = rank_docs(scores, doc_ids)
+ hits = labeled_hits(ranked, qrels, k=k)
+ for pid, label in hits:
+ rows.append(
+ {
+ "query_id": str(qid),
+ "query": query,
+ "product_id": pid,
+ "label": label,
+ "passage": passages.get(pid) or pid,
+ }
+ )
+ if offset % 200 == 0 or offset == n_groups:
+ print(f"mine retrieve {offset}/{n_groups}", flush=True)
+ return rows
+
+
+def write_lists(rows: Sequence[dict[str, Any]], dest: Path) -> Path:
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ with dest.open("w", encoding="utf-8") as fh:
+ for row in rows:
+ fh.write(json.dumps(row, ensure_ascii=False) + "\n")
+ return dest
+
+
+def mine(
+ *,
+ jsonl_path: Path,
+ out_path: Path,
+ max_queries: int = 6000,
+ k: int = 50,
+ seed: int = 11,
+ locale: str = "us",
+) -> dict[str, Any]:
+ holdout = held_out_query_ids(jsonl_path)
+ groups = load_train_groups(holdout=holdout, locale=locale)
+ selected = select_groups(
+ groups,
+ holdout=holdout,
+ max_queries=max_queries,
+ seed=seed,
+ )
+ needed: set[str] = set()
+ for group in selected.values():
+ needed.update(str(pid) for pid in (group.get("qrels") or {}))
+ print(
+ f"mine-retrieved-lists queries={len(selected)} products={len(needed)} "
+ f"holdout={len(holdout)}",
+ flush=True,
+ )
+ passages = passages_for_pids(needed, locale=locale)
+ rows = mine_from_groups(selected, passages, k=k)
+ write_lists(rows, out_path)
+ counts = {label: 0 for label in LABELS}
+ for row in rows:
+ counts[str(row["label"])] = counts.get(str(row["label"]), 0) + 1
+ n_with_i = 0
+ by_q: dict[str, list[str]] = {}
+ for row in rows:
+ by_q.setdefault(str(row["query_id"]), []).append(str(row["label"]))
+ for labels in by_q.values():
+ if "I" in labels:
+ n_with_i += 1
+ meta = {
+ "n_queries": len(selected),
+ "n_pairs": len(rows),
+ "n_holdout_qids": len(holdout),
+ "k": k,
+ "label_counts": counts,
+ "n_lists_with_i": n_with_i,
+ "source": "retrieved-bm25",
+ "out": str(out_path),
+ }
+ meta_path = out_path.with_suffix(out_path.suffix + ".meta.json")
+ if out_path.suffix == ".jsonl":
+ meta_path = out_path.with_name(out_path.stem + ".meta.json")
+ meta_path.write_text(json.dumps(meta, indent=2) + "\n", encoding="utf-8")
+ return meta
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(
+ description="Mine BM25 top-k train lists; unlabeled hits are class I"
+ )
+ parser.add_argument("--dataset", type=Path, required=True)
+ parser.add_argument("--out", type=Path, default=DEFAULT_OUT)
+ parser.add_argument("--max-queries", type=int, default=6000)
+ parser.add_argument("--k", type=int, default=50)
+ parser.add_argument("--seed", type=int, default=11)
+ parser.add_argument("--locale", default="us")
+ args = parser.parse_args(argv)
+ meta = mine(
+ jsonl_path=args.dataset,
+ out_path=args.out,
+ max_queries=int(args.max_queries),
+ k=int(args.k),
+ seed=int(args.seed),
+ locale=str(args.locale),
+ )
+ print(json.dumps(meta, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/search/miss.py b/benchmarks/search/miss.py
new file mode 100644
index 0000000..1ab557d
--- /dev/null
+++ b/benchmarks/search/miss.py
@@ -0,0 +1,115 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+from search.dataset import load_records, split_corpus
+
+ELECTRONICS_MARKERS = (
+ "electronic",
+ "electronics",
+ "ipad",
+ "iphone",
+ "phone",
+ "tablet",
+ "laptop",
+ "computer",
+ "charger",
+ "usb",
+ "hdmi",
+ "bluetooth",
+ "wifi",
+ "wi-fi",
+ "router",
+ "led",
+ "battery",
+ "samsung",
+ "apple",
+)
+
+
+def looks_like_electronics(text: str) -> bool:
+ lowered = (text or "").lower()
+ return any(token in lowered for token in ELECTRONICS_MARKERS)
+
+
+def dump_query_miss(
+ *,
+ dataset_path: Path,
+ eval_path: Path,
+ qid: str,
+) -> dict[str, Any]:
+ docs, queries = split_corpus(load_records(dataset_path))
+ by_id = {str(doc.get("doc_id") or ""): doc for doc in docs}
+ query = next((row for row in queries if str(row.get("qid")) == qid), None)
+ if query is None:
+ raise SystemExit(f"Missing query {qid} in {dataset_path}")
+ eval_data = json.loads(eval_path.read_text(encoding="utf-8"))
+ scored = next(
+ (row for row in eval_data.get("queries") or [] if str(row.get("qid")) == qid),
+ None,
+ )
+ if scored is None:
+ raise SystemExit(f"Missing query {qid} in {eval_path}")
+ gold_rows = []
+ in_pool = 0
+ electronics_gold = 0
+ for doc_id in query.get("gold_doc_ids") or []:
+ doc = by_id.get(str(doc_id)) or {}
+ title = str(doc.get("title") or "")
+ text = str(doc.get("text") or "")
+ present = bool(doc)
+ in_pool += int(present)
+ electronic = looks_like_electronics(f"{title}\n{text}")
+ electronics_gold += int(electronic)
+ gold_rows.append(
+ {
+ "doc_id": doc_id,
+ "title": title,
+ "brand": doc.get("brand") or "",
+ "in_jsonl": present,
+ "electronics_marker": electronic,
+ }
+ )
+ hit_rows = []
+ for doc_id in scored.get("hit_ids") or []:
+ doc = by_id.get(str(doc_id)) or {}
+ title = str(doc.get("title") or "")
+ hit_rows.append(
+ {
+ "doc_id": doc_id,
+ "title": title,
+ "electronics_marker": looks_like_electronics(title),
+ }
+ )
+ gold_set = {str(item) for item in (query.get("gold_doc_ids") or [])}
+ overlap = gold_set & {str(item["doc_id"]) for item in hit_rows}
+ if in_pool < len(gold_rows):
+ decision = "not_testable"
+ reason = "one or more Exact golds are missing from the JSONL pool"
+ elif electronics_gold >= max(1, len(gold_rows) // 2):
+ decision = "qrel_mismatch"
+ reason = (
+ "majority of Exact golds look like electronics or electronics accessories; "
+ "do not rewrite 'not electronics'"
+ )
+ elif overlap:
+ decision = "not_testable"
+ reason = "golds already appear in the scored list"
+ else:
+ decision = "rewrite_experiment"
+ reason = "golds are in-pool, non-electronics, and missing from top hits"
+ return {
+ "qid": qid,
+ "query": query.get("query"),
+ "metrics": scored.get("metrics"),
+ "gold": gold_rows,
+ "hits": hit_rows,
+ "n_gold": len(gold_rows),
+ "n_gold_in_jsonl": in_pool,
+ "n_electronics_gold": electronics_gold,
+ "overlap": sorted(overlap),
+ "decision": decision,
+ "reason": reason,
+ }
diff --git a/benchmarks/search/miss_strata.py b/benchmarks/search/miss_strata.py
new file mode 100644
index 0000000..3958566
--- /dev/null
+++ b/benchmarks/search/miss_strata.py
@@ -0,0 +1,308 @@
+from __future__ import annotations
+
+import html
+import json
+import re
+import unicodedata
+from collections import Counter
+from pathlib import Path
+from typing import Any
+
+from search.dataset import load_records, split_corpus, write_records
+from search.metrics import recall_at_k
+
+TOKEN = re.compile(r"[a-z0-9]+", re.I)
+JUNK_PUNCT = re.compile(r"[^\w\s\-]", re.UNICODE)
+COMBINING = re.compile(r"[\u0300-\u036f]")
+SQL_MARKERS = (
+ "not null",
+ "integer",
+ "varchar",
+ "p_num",
+ "i_num",
+ "primary key",
+)
+SKIP_REWRITE_QIDS = frozenset({"esci-72", "72"})
+
+
+def canonical_qid(qid: str) -> str:
+ raw = str(qid or "").strip()
+ if raw.lower().startswith("esci-"):
+ return raw.lower()
+ if raw.isdigit():
+ return f"esci-{raw}"
+ return raw.lower()
+
+
+def gold_ids(query: dict[str, Any]) -> list[str]:
+ grades = query.get("gold_grades") or {}
+ ordered: list[str] = []
+ seen: set[str] = set()
+ for doc_id in query.get("gold_doc_ids") or []:
+ key = str(doc_id)
+ if key in seen:
+ continue
+ if isinstance(grades, dict) and key in grades and float(grades.get(key) or 0) <= 0:
+ continue
+ seen.add(key)
+ ordered.append(key)
+ if isinstance(grades, dict):
+ for doc_id, gain in grades.items():
+ if float(gain or 0) <= 0:
+ continue
+ key = str(doc_id)
+ if key in seen:
+ continue
+ seen.add(key)
+ ordered.append(key)
+ return ordered
+
+
+def letter_for_gain(gain: float) -> str:
+ value = float(gain or 0)
+ if value >= 0.99:
+ return "E"
+ if value >= 0.09:
+ return "S"
+ if value > 0:
+ return "C"
+ return "I"
+
+
+def query_tokens(text: str) -> set[str]:
+ return {token.lower() for token in TOKEN.findall(text or "")}
+
+
+def token_overlap(query: str, doc_text: str) -> float:
+ qtok = query_tokens(query)
+ if not qtok:
+ return 0.0
+ dtok = query_tokens(doc_text)
+ return len(qtok & dtok) / len(qtok)
+
+
+def looks_sql(text: str) -> bool:
+ lowered = (text or "").lower()
+ return any(marker in lowered for marker in SQL_MARKERS)
+
+
+def looks_html_entity(text: str) -> bool:
+ raw = text or ""
+ return "&#" in raw or "&amp;" in raw or "&quot;" in raw or "&lt;" in raw or "&gt;" in raw
+
+
+def normalize_spelling(text: str) -> str:
+ raw = unicodedata.normalize("NFKC", str(text or ""))
+ raw = html.unescape(raw)
+ raw = unicodedata.normalize("NFKD", raw)
+ raw = COMBINING.sub("", raw)
+ raw = re.sub(r"^[\s\-*._]+", "", raw)
+ raw = re.sub(r"[\s\-*._]+$", "", raw)
+ raw = JUNK_PUNCT.sub(" ", raw)
+ raw = re.sub(r"\s+", " ", raw).strip()
+ return raw
+
+
+def rewrite_query(qid: str, text: str) -> str | None:
+ if canonical_qid(qid) in SKIP_REWRITE_QIDS:
+ return None
+ original = str(text or "").strip()
+ if not original:
+ return None
+ if looks_sql(original):
+ return None
+ unescaped = html.unescape(original).strip()
+ unescaped = re.sub(r"^[\s\-*._]+", "", unescaped)
+ unescaped = re.sub(r"[\s\-*._]+$", "", unescaped)
+ unescaped = re.sub(r"\s+", " ", unescaped).strip()
+ if len(unescaped) < 3:
+ return None
+ if unescaped == original and not looks_html_entity(original):
+ return None
+ return unescaped
+
+
+def classify_query(
+ eval_row: dict[str, Any],
+ query_row: dict[str, Any],
+ docs: dict[str, dict[str, Any]],
+ *,
+ k: int = 50,
+) -> dict[str, Any]:
+ hits = [str(item) for item in (eval_row.get("hit_ids") or []) if item]
+ gold = gold_ids(query_row if query_row else eval_row)
+ grades = (query_row.get("gold_grades") if query_row else None) or eval_row.get(
+ "gold_grades"
+ ) or {}
+ hitset = set(hits)
+ in_head = [doc_id for doc_id in gold if doc_id in hits[:10]]
+ in_tail = [doc_id for doc_id in gold if doc_id in hitset and doc_id not in set(hits[:10])]
+ missed = [doc_id for doc_id in gold if doc_id not in hitset]
+ if in_head:
+ stratum = "head-ok"
+ elif in_tail:
+ stratum = "rank-too-low"
+ else:
+ stratum = "total-miss"
+ qtext = str(eval_row.get("query") or query_row.get("query") or "")
+ miss_letters = Counter()
+ gold_letters = Counter()
+ overlap_hit: list[float] = []
+ overlap_miss: list[float] = []
+ for doc_id in gold:
+ letter = letter_for_gain(float((grades or {}).get(doc_id) or 1.0))
+ gold_letters[letter] += 1
+ text = str((docs.get(doc_id) or {}).get("text") or "")
+ overlap = token_overlap(qtext, text)
+ if doc_id in hitset:
+ overlap_hit.append(overlap)
+ else:
+ miss_letters[letter] += 1
+ overlap_miss.append(overlap)
+ metrics = eval_row.get("metrics") or {}
+ rewritten = rewrite_query(str(eval_row.get("qid") or query_row.get("qid") or ""), qtext)
+ pathological = bool(
+ rewritten
+ or looks_html_entity(qtext)
+ or looks_sql(qtext)
+ ) and canonical_qid(str(eval_row.get("qid") or "")) not in SKIP_REWRITE_QIDS
+ return {
+ "qid": eval_row.get("qid") or query_row.get("qid"),
+ "query": qtext,
+ "stratum": stratum,
+ "n_gold": len(gold),
+ "n_in_top10": len(in_head),
+ "n_in_11_to_k": len(in_tail),
+ "n_missed": len(missed),
+ "recall@10": float(metrics.get("recall@10") or recall_at_k(hits, gold, 10)),
+ "recall@50": float(metrics.get("recall@50") or recall_at_k(hits, gold, k)),
+ "gold_letters": dict(gold_letters),
+ "miss_letters": dict(miss_letters),
+ "mean_overlap_hit": (sum(overlap_hit) / len(overlap_hit)) if overlap_hit else None,
+ "mean_overlap_miss": (sum(overlap_miss) / len(overlap_miss)) if overlap_miss else None,
+ "pathological": pathological,
+ "rewrite": rewritten,
+ "skip_rewrite": canonical_qid(str(eval_row.get("qid") or "")) in SKIP_REWRITE_QIDS,
+ "k": k,
+ }
+
+
+def classify_eval(
+ eval_result: dict[str, Any],
+ rows: list[dict[str, Any]],
+ *,
+ k: int | None = None,
+) -> dict[str, Any]:
+ docs, queries = split_corpus(rows)
+ by_id = {str(doc.get("doc_id") or ""): doc for doc in docs}
+ by_qid = {str(query.get("qid") or ""): query for query in queries}
+ cut = int(k or eval_result.get("k") or 50)
+ classified: list[dict[str, Any]] = []
+ gold_total = 0
+ in_head = 0
+ in_tail = 0
+ missed = 0
+ for row in eval_result.get("queries") or []:
+ qid = str(row.get("qid") or "")
+ item = classify_query(row, by_qid.get(qid) or {}, by_id, k=cut)
+ classified.append(item)
+ gold_total += int(item["n_gold"])
+ in_head += int(item["n_in_top10"])
+ in_tail += int(item["n_in_11_to_k"])
+ missed += int(item["n_missed"])
+ counts = Counter(str(item["stratum"]) for item in classified)
+ total_miss = [item for item in classified if item["stratum"] == "total-miss"]
+ rewritable = [
+ item
+ for item in total_miss
+ if item.get("rewrite") and not item.get("skip_rewrite")
+ ]
+ return {
+ "n_queries": len(classified),
+ "n_gold": gold_total,
+ "n_in_top10": in_head,
+ "n_in_11_to_k": in_tail,
+ "n_missed": missed,
+ "k": cut,
+ "stratum_counts": dict(counts),
+ "n_total_miss": len(total_miss),
+ "n_rewritable_total_miss": len(rewritable),
+ "run_query_side": bool(rewritable),
+ "queries": classified,
+ "rewritable_qids": [item["qid"] for item in rewritable],
+ "total_miss_qids": [item["qid"] for item in total_miss],
+ }
+
+
+def write_rewritten_jsonl(
+ rows: list[dict[str, Any]],
+ taxonomy: dict[str, Any],
+ dest: Path,
+) -> Path:
+ rewrites = {
+ str(item["qid"]): str(item["rewrite"])
+ for item in taxonomy.get("queries") or []
+ if item.get("rewrite")
+ and item.get("stratum") == "total-miss"
+ and not item.get("skip_rewrite")
+ }
+ out: list[dict[str, Any]] = []
+ for row in rows:
+ if str(row.get("type") or "") != "query":
+ out.append(row)
+ continue
+ qid = str(row.get("qid") or "")
+ rewritten = rewrites.get(qid)
+ if not rewritten:
+ out.append(row)
+ continue
+ cloned = dict(row)
+ cloned["query_original"] = row.get("query")
+ cloned["query"] = rewritten
+ out.append(cloned)
+ return write_records(out, dest)
+
+
+def write_spell_jsonl(rows: list[dict[str, Any]], dest: Path) -> dict[str, Any]:
+ out: list[dict[str, Any]] = []
+ n_changed = 0
+ for row in rows:
+ if str(row.get("type") or "") != "query":
+ out.append(row)
+ continue
+ original = str(row.get("query") or "")
+ rewritten = normalize_spelling(original)
+ cloned = dict(row)
+ if rewritten != original:
+ cloned["query_original"] = original
+ cloned["query"] = rewritten
+ n_changed += 1
+ out.append(cloned)
+ write_records(out, dest)
+ return {
+ "out_path": str(dest),
+ "n_rows": len(out),
+ "n_queries_changed": n_changed,
+ }
+
+
+def load_eval(path: Path) -> dict[str, Any]:
+ return json.loads(path.read_text(encoding="utf-8"))
+
+
+def main_classify(
+ *,
+ eval_path: Path,
+ dataset_path: Path,
+ out_path: Path | None = None,
+) -> dict[str, Any]:
+ eval_result = load_eval(eval_path)
+ rows = load_records(dataset_path)
+ taxonomy = classify_eval(eval_result, rows)
+ taxonomy["eval_path"] = str(eval_path)
+ taxonomy["dataset"] = dataset_path.name
+ if out_path is not None:
+ out_path.parent.mkdir(parents=True, exist_ok=True)
+ out_path.write_text(json.dumps(taxonomy, indent=2) + "\n", encoding="utf-8")
+ taxonomy["out_path"] = str(out_path)
+ return taxonomy
diff --git a/benchmarks/search/pool_first_stage.py b/benchmarks/search/pool_first_stage.py
new file mode 100644
index 0000000..85f315b
--- /dev/null
+++ b/benchmarks/search/pool_first_stage.py
@@ -0,0 +1,363 @@
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import re
+from collections import Counter, defaultdict
+from pathlib import Path
+from typing import Any, Iterable
+
+from search.dataset import load_records, split_corpus, write_records
+from search.evaluate import ensure_run_dir
+from search.metrics import aggregate_query_metrics
+from search.rank_pool import score_ranked_pool
+
+TOKEN_RE = re.compile(r"[a-z0-9]+", re.I)
+STOP = {
+ "a",
+ "an",
+ "and",
+ "are",
+ "as",
+ "at",
+ "be",
+ "for",
+ "from",
+ "in",
+ "is",
+ "it",
+ "of",
+ "on",
+ "or",
+ "the",
+ "to",
+ "with",
+}
+
+
+def tokenize(text: str) -> list[str]:
+ return [tok.lower() for tok in TOKEN_RE.findall(text or "")]
+
+
+def field_value(text: str, name: str) -> str:
+ prefix = f"{name}:"
+ lines = []
+ capture = False
+ for raw in (text or "").splitlines():
+ stripped = raw.strip()
+ if ":" in stripped and not stripped.lower().startswith("docid "):
+ label = stripped.split(":", 1)[0].strip().lower()
+ capture = label == name.lower()
+ if capture:
+ lines.append(stripped.split(":", 1)[1].strip())
+ continue
+ if capture:
+ lines.append(stripped)
+ return " ".join(part for part in lines if part)
+
+
+def title_text(text: str) -> str:
+ title = field_value(text, "Title")
+ return title or (text or "")
+
+
+def bm25_idf(n_docs: int, df: int) -> float:
+ return math.log(1.0 + (n_docs - df + 0.5) / (df + 0.5))
+
+
+class Bm25Index:
+ def __init__(self, docs: list[tuple[str, list[str]]], *, k1: float = 1.2, b: float = 0.75):
+ self.k1 = k1
+ self.b = b
+ self.doc_ids = [doc_id for doc_id, _ in docs]
+ self.tfs: list[Counter[str]] = [Counter(tokens) for _, tokens in docs]
+ self.dl = [max(1, len(tokens)) for _, tokens in docs]
+ self.avgdl = (sum(self.dl) / len(self.dl)) if self.dl else 1.0
+ df: Counter[str] = Counter()
+ postings: dict[str, list[int]] = defaultdict(list)
+ for index, tokens in enumerate(self.tfs):
+ df.update(tokens.keys())
+ for term in tokens:
+ postings[term].append(index)
+ n = max(1, len(self.doc_ids))
+ self.idf = {term: bm25_idf(n, count) for term, count in df.items()}
+ self.postings = postings
+
+ def scores(self, query_tokens: list[str]) -> dict[str, float]:
+ qtf = Counter(query_tokens)
+ out: dict[str, float] = defaultdict(float)
+ for term, qf in qtf.items():
+ idf = self.idf.get(term, 0.0)
+ if not idf:
+ continue
+ for index in self.postings.get(term, ()):
+ tf = self.tfs[index].get(term, 0)
+ if tf <= 0:
+ continue
+ dl = self.dl[index]
+ denom = tf + self.k1 * (1.0 - self.b + self.b * dl / self.avgdl)
+ out[self.doc_ids[index]] += float(qf) * idf * (tf * (self.k1 + 1.0)) / denom
+ return out
+
+
+def rank_docs(scores: dict[str, float], doc_ids: Iterable[str]) -> list[str]:
+ ranked = sorted(scores.items(), key=lambda pair: pair[1], reverse=True)
+ seen = {doc_id for doc_id, _ in ranked}
+ ordered = [doc_id for doc_id, _ in ranked]
+ for doc_id in doc_ids:
+ if doc_id not in seen:
+ ordered.append(doc_id)
+ return ordered
+
+
+def rm3_terms(
+ query_tokens: list[str],
+ ranked: list[str],
+ token_map: dict[str, list[str]],
+ *,
+ top_docs: int = 8,
+ extra_terms: int = 8,
+) -> list[str]:
+ counts: Counter[str] = Counter()
+ for doc_id in ranked[:top_docs]:
+ counts.update(token_map.get(doc_id) or [])
+ query_set = set(query_tokens)
+ added: list[str] = []
+ for term, _ in counts.most_common():
+ if term in STOP or term in query_set or term.isdigit():
+ continue
+ added.append(term)
+ if len(added) >= extra_terms:
+ break
+ return added
+
+
+def expand_query(query: str, extra: list[str]) -> str:
+ if not extra:
+ return query
+ return f"{query} {' '.join(extra)}".strip()
+
+
+def evaluate_variant(
+ queries: list[dict[str, Any]],
+ ranked_by_qid: dict[str, list[str]],
+) -> tuple[dict[str, float], list[dict[str, Any]]]:
+ per_query: list[dict[str, Any]] = []
+ for query in queries:
+ qid = str(query.get("qid") or "")
+ ranked = ranked_by_qid.get(qid) or []
+ metrics = score_ranked_pool(ranked, query)
+ per_query.append(
+ {
+ "qid": qid,
+ "query": query.get("query"),
+ "hit_ids": ranked[:20],
+ "metrics": metrics,
+ "n_hits": len(ranked),
+ "retrieve_ms": 0.0,
+ "embed_ms": None,
+ "client_wall_ms": 0.0,
+ }
+ )
+ metrics = aggregate_query_metrics(per_query, ks=(5, 10, 20)) if per_query else {}
+ return metrics, per_query
+
+
+def _splade_rank(
+ docs: list[dict[str, Any]],
+ queries: list[dict[str, Any]],
+ *,
+ k: int = 20,
+) -> dict[str, list[str]]:
+ plugin_dir = Path(__file__).resolve().parents[2] / "plugins" / "search-splade"
+ import sys
+
+ if str(plugin_dir) not in sys.path:
+ sys.path.insert(0, str(plugin_dir))
+ from index import index_chunks, reset, retrieve
+
+ brain = "harness-local-splade"
+ reset(brain)
+ chunks = []
+ doc_ids: list[str] = []
+ for doc in docs:
+ doc_id = str(doc.get("doc_id") or "")
+ if not doc_id:
+ continue
+ doc_ids.append(doc_id)
+ chunks.append(
+ {"id": doc_id, "text": str(doc.get("text") or doc.get("title") or "")}
+ )
+ index_chunks(brain, chunks)
+ ranked_by_qid: dict[str, list[str]] = {}
+ for query in queries:
+ qid = str(query.get("qid") or "")
+ ids, scores, _ = retrieve(str(query.get("query") or ""), brain, max(k, len(doc_ids)))
+ ranked_by_qid[qid] = rank_docs(
+ {str(doc_id): float(scores.get(doc_id, 0.0)) for doc_id in ids},
+ doc_ids,
+ )
+ return ranked_by_qid
+
+
+def run_first_stage(
+ rows: list[dict[str, Any]],
+ *,
+ variant: str,
+) -> dict[str, Any]:
+ docs, queries = split_corpus(rows)
+ if variant == "splade":
+ ranked_by_qid = _splade_rank(docs, queries)
+ metrics, per_query = evaluate_variant(queries, ranked_by_qid)
+ return {
+ "variant": variant,
+ "metrics": metrics,
+ "queries": per_query,
+ "expansions": {},
+ "n_docs": len(docs),
+ "n_queries": len(queries),
+ }
+ all_tokens: dict[str, list[str]] = {}
+ title_tokens: dict[str, list[str]] = {}
+ doc_ids: list[str] = []
+ for doc in docs:
+ doc_id = str(doc.get("doc_id") or "")
+ if not doc_id:
+ continue
+ text = str(doc.get("text") or doc.get("title") or "")
+ doc_ids.append(doc_id)
+ all_tokens[doc_id] = tokenize(text)
+ title_tokens[doc_id] = tokenize(title_text(text))
+ all_index = Bm25Index([(doc_id, all_tokens[doc_id]) for doc_id in doc_ids])
+ title_index = Bm25Index([(doc_id, title_tokens[doc_id]) for doc_id in doc_ids])
+ ranked_by_qid: dict[str, list[str]] = {}
+ expansions: dict[str, str] = {}
+ for query in queries:
+ qid = str(query.get("qid") or "")
+ qtext = str(query.get("query") or "")
+ q_tokens = tokenize(qtext)
+ all_scores = all_index.scores(q_tokens)
+ title_scores = title_index.scores(q_tokens)
+ if variant == "title":
+ scores = title_scores
+ elif variant == "title-boost":
+ scores = dict(all_scores)
+ for doc_id, value in title_scores.items():
+ scores[doc_id] = scores.get(doc_id, 0.0) + 2.0 * value
+ elif variant in {"rm3", "rm3-title-boost"}:
+ seed_scores = dict(all_scores)
+ if variant == "rm3-title-boost":
+ for doc_id, value in title_scores.items():
+ seed_scores[doc_id] = seed_scores.get(doc_id, 0.0) + 2.0 * value
+ seed_ranked = rank_docs(seed_scores, doc_ids)
+ extra = rm3_terms(q_tokens, seed_ranked, title_tokens)
+ expansions[qid] = expand_query(qtext, extra)
+ expanded_tokens = tokenize(expansions[qid])
+ scores = all_index.scores(expanded_tokens)
+ if variant == "rm3-title-boost":
+ title_exp = title_index.scores(expanded_tokens)
+ for doc_id, value in title_exp.items():
+ scores[doc_id] = scores.get(doc_id, 0.0) + 2.0 * value
+ else:
+ scores = all_scores
+ ranked_by_qid[qid] = rank_docs(scores, doc_ids)
+ metrics, per_query = evaluate_variant(queries, ranked_by_qid)
+ return {
+ "variant": variant,
+ "metrics": metrics,
+ "queries": per_query,
+ "expansions": expansions,
+ "n_docs": len(doc_ids),
+ "n_queries": len(queries),
+ }
+
+
+def write_expanded_jsonl(
+ rows: list[dict[str, Any]],
+ expansions: dict[str, str],
+ dest: Path,
+) -> Path:
+ out: list[dict[str, Any]] = []
+ for row in rows:
+ if str(row.get("type") or "") != "query":
+ out.append(row)
+ continue
+ cloned = dict(row)
+ qid = str(cloned.get("qid") or "")
+ expanded = expansions.get(qid)
+ if expanded:
+ cloned["query"] = expanded
+ cloned["query_original"] = row.get("query")
+ out.append(cloned)
+ return write_records(out, dest)
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(prog="pool_first_stage")
+ parser.add_argument("--dataset", type=Path, default=Path("data/search_esci_74.jsonl"))
+ parser.add_argument("--run", default="search-esci-74-fielded-bm25")
+ parser.add_argument(
+ "--variant",
+ default="all",
+ help="Comma-separated: all,title,title-boost,rm3,rm3-title-boost",
+ )
+ parser.add_argument(
+ "--expand-out",
+ type=Path,
+ default=Path("data/search_esci_74_rm3.jsonl"),
+ )
+ args = parser.parse_args(argv)
+ from search.config import Settings
+
+ settings = Settings.load()
+ rows = load_records(args.dataset)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ variants = [item.strip() for item in str(args.variant).split(",") if item.strip()]
+ summary: dict[str, Any] = {"run_id": run_id, "dataset": args.dataset.name, "variants": {}}
+ expand_source = None
+ for variant in variants:
+ result = run_first_stage(rows, variant=variant)
+ metrics = result["metrics"]
+ summary["variants"][variant] = {
+ "ndcg@10": metrics.get("ndcg@10"),
+ "ndcg@20": metrics.get("ndcg@20"),
+ "recall@10": metrics.get("recall@10"),
+ "recall@20": metrics.get("recall@20"),
+ "mrr": metrics.get("mrr"),
+ }
+ if variant.startswith("rm3"):
+ expand_source = result
+ if expand_source and expand_source.get("expansions"):
+ write_expanded_jsonl(rows, expand_source["expansions"], args.expand_out)
+ summary["expand_out"] = str(args.expand_out)
+ docs, queries = split_corpus(rows)
+ payload = {
+ "status": "ok",
+ "brain_id": "harness-local",
+ "dataset": args.dataset.name,
+ "fusion": "none",
+ "rerank": "none",
+ "channels": ["harness-bm25"],
+ "protocol": (
+ "local fielded BM25 over search_esci_74.jsonl 2043-doc corpus; "
+ "not BrainAPI passages; not Reddy 0.857"
+ ),
+ "n_docs": len(docs),
+ "n_queries": len(queries),
+ "metrics": (summary["variants"].get(variants[0]) or {}),
+ "summary": summary,
+ "ingest": {"status": "completed", "n_docs": 0, "tasks": [], "reused": True},
+ }
+ (run_dir / "eval.json").write_text(
+ json.dumps(payload, indent=2) + "\n", encoding="utf-8"
+ )
+ (run_dir / "report.json").write_text(
+ json.dumps(summary, indent=2) + "\n", encoding="utf-8"
+ )
+ print(json.dumps(summary, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/search/rank_corpus.py b/benchmarks/search/rank_corpus.py
new file mode 100644
index 0000000..68e3942
--- /dev/null
+++ b/benchmarks/search/rank_corpus.py
@@ -0,0 +1,100 @@
+from __future__ import annotations
+
+import time
+from pathlib import Path
+from typing import Any
+
+from search.dataset import split_corpus
+from search.finetune_esci_4class import CLASS_GAINS, rank_doc_ids, weighted_scores
+from search.local_dense import evaluate_dense
+from search.metrics import aggregate_query_metrics
+from search.rank_pool import doc_texts
+from search.rank_pool_4class import PredictFn, load_4class_predict
+
+DEFAULT_MODEL = Path("data/models/esci-minilm-l12-4class-nowt-e2")
+PROTOCOL = "exhaustive-catalog"
+
+
+def run_exhaustive_ce(
+ rows: list[dict[str, Any]],
+ *,
+ predict: PredictFn,
+ model_name: str,
+ dataset_name: str,
+ k: int = 50,
+ ks: tuple[int, ...] = (5, 10, 20, 50),
+ brain_id: str | None = None,
+) -> dict[str, Any]:
+ docs, queries = split_corpus(rows)
+ texts = doc_texts(docs)
+ doc_ids = [str(doc.get("doc_id") or "") for doc in docs if doc.get("doc_id")]
+ cut = max(1, min(int(k), len(doc_ids) or 1))
+ ranked_by_qid: dict[str, list[str]] = {}
+ per_q_ms: list[float] = []
+ missing_text = 0
+ for index, query in enumerate(queries):
+ q_text = str(query.get("query") or "")
+ pairs: list[tuple[str, str]] = []
+ for doc_id in doc_ids:
+ text = texts.get(doc_id) or ""
+ if not text:
+ missing_text += 1
+ pairs.append((q_text, text))
+ started = time.perf_counter()
+ probs = predict(pairs) if pairs else []
+ elapsed_ms = (time.perf_counter() - started) * 1000.0
+ per_q_ms.append(elapsed_ms)
+ scores = weighted_scores(probs, CLASS_GAINS) if pairs else []
+ if len(scores) < len(doc_ids):
+ scores = list(scores) + [0.0] * (len(doc_ids) - len(scores))
+ ranked = rank_doc_ids(doc_ids, scores)[:cut]
+ qid = str(query.get("qid") or "")
+ ranked_by_qid[qid] = ranked
+ print(f"exhaustive {index + 1}/{len(queries)} {qid} {elapsed_ms:.0f}ms", flush=True)
+ encode_ms = sum(per_q_ms)
+ metrics, per_query = evaluate_dense(
+ ranked_by_qid,
+ queries,
+ ks=ks,
+ encode_ms=encode_ms,
+ )
+ if not metrics and per_query:
+ metrics = aggregate_query_metrics(per_query, ks=ks)
+ return {
+ "status": "ok" if per_query else "failed",
+ "brain_id": brain_id or "harness-local-exhaustive",
+ "dataset": dataset_name,
+ "fusion": "none",
+ "rerank": "4class-weighted-ce",
+ "channels": ["exhaustive-4class"],
+ "expand": "none",
+ "k": cut,
+ "ks": list(ks),
+ "n_docs": len(docs),
+ "n_queries": len(per_query),
+ "n_docs_mapped": len(texts),
+ "skip_enrichment": True,
+ "ingest_graph": False,
+ "skip_ingest": True,
+ "rank_pool": False,
+ "rank_pool_ce": False,
+ "ce_model": model_name,
+ "ce_missing_text": missing_text,
+ "protocol": PROTOCOL,
+ "ingest": {
+ "status": "completed",
+ "n_docs": len(docs),
+ "tasks": [],
+ "reused": True,
+ },
+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "event_probe": None,
+ "search_error": None,
+ "metrics": metrics,
+ "queries": per_query,
+ }
+
+
+def load_predict(model_dir: Path, *, max_length: int = 192) -> tuple[PredictFn, str]:
+ return load_4class_predict(model_dir, max_length=max_length)
diff --git a/benchmarks/search/rank_pool.py b/benchmarks/search/rank_pool.py
new file mode 100644
index 0000000..2d50e74
--- /dev/null
+++ b/benchmarks/search/rank_pool.py
@@ -0,0 +1,137 @@
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+from typing import Any
+
+from search.dataset import split_corpus
+from search.evaluate import candidate_pool_grades, candidate_pool_ids
+from search.metrics import aggregate_query_metrics, mrr, ndcg_at_k, recall_at_k
+
+PLUGIN_DIR = Path(__file__).resolve().parents[2] / "plugins" / "search-rerank"
+
+
+def _plugin_rerank():
+ if str(PLUGIN_DIR) not in sys.path:
+ sys.path.insert(0, str(PLUGIN_DIR))
+ from rerank import model_name, rerank
+
+ return rerank, model_name()
+
+
+def doc_texts(docs: list[dict[str, Any]]) -> dict[str, str]:
+ texts: dict[str, str] = {}
+ for doc in docs:
+ doc_id = str(doc.get("doc_id") or "")
+ if not doc_id:
+ continue
+ texts[doc_id] = str(doc.get("text") or doc.get("title") or "")
+ return texts
+
+
+def score_ranked_pool(
+ ranked: list[str],
+ query: dict[str, Any],
+ *,
+ ks: tuple[int, ...] = (5, 10, 20),
+) -> dict[str, float]:
+ gold = {
+ str(doc_id)
+ for doc_id, gain in candidate_pool_grades(query).items()
+ if float(gain) > 0
+ }
+ if not gold:
+ gold = {str(item) for item in (query.get("gold_doc_ids") or [])}
+ grades = candidate_pool_grades(query)
+ metrics = {f"recall@{k}": recall_at_k(ranked, gold, k) for k in ks}
+ ndcg_ks = tuple(dict.fromkeys((*ks, 10, 20)))
+ for cut in ndcg_ks:
+ metrics[f"ndcg@{cut}"] = ndcg_at_k(ranked, gold, cut, grades=grades)
+ metrics["ndcg"] = ndcg_at_k(ranked, gold, max(len(ranked), 1), grades=grades)
+ metrics["mrr"] = mrr(ranked, gold)
+ return metrics
+
+
+def run_ce_on_pool(
+ rows: list[dict[str, Any]],
+ *,
+ dataset_name: str,
+ ks: tuple[int, ...] = (5, 10, 20),
+ brain_id: str | None = None,
+) -> dict[str, Any]:
+ docs, queries = split_corpus(rows)
+ texts = doc_texts(docs)
+ rerank, model = _plugin_rerank()
+ per_query: list[dict[str, Any]] = []
+ missing_text = 0
+ for query in queries:
+ pool = candidate_pool_ids(query)
+ candidates = []
+ for doc_id in pool:
+ text = texts.get(doc_id) or ""
+ if not text:
+ missing_text += 1
+ candidates.append({"id": doc_id, "text": text, "score": 0.0})
+ ranked_rows = rerank(
+ str(query.get("query") or ""),
+ candidates,
+ len(candidates) or 1,
+ )
+ ranked = [str(item.get("id") or "") for item in ranked_rows if item.get("id")]
+ seen = set(ranked)
+ for doc_id in pool:
+ if doc_id not in seen:
+ ranked.append(doc_id)
+ metrics = score_ranked_pool(ranked, query, ks=ks)
+ per_query.append(
+ {
+ "qid": query.get("qid"),
+ "query": query.get("query"),
+ "slice": query.get("slice") or "unspecified",
+ "gold_doc_ids": list(query.get("gold_doc_ids") or []),
+ "candidate_doc_ids": pool,
+ "pool_size": len(pool),
+ "pool_coverage": 1.0 if pool else None,
+ "missing_from_brain": [],
+ "hit_ids": ranked,
+ "metrics": metrics,
+ "retrieve_ms": 0.0,
+ "embed_ms": None,
+ "client_wall_ms": 0.0,
+ "n_hits": len(ranked),
+ }
+ )
+ metrics = aggregate_query_metrics(per_query, ks=ks) if per_query else {}
+ return {
+ "status": "ok" if per_query else "failed",
+ "brain_id": brain_id,
+ "dataset": dataset_name,
+ "fusion": "none",
+ "rerank": "plugin:cross-encoder",
+ "channels": ["rank-pool-ce"],
+ "expand": "none",
+ "k": max((len(row.get("hit_ids") or []) for row in per_query), default=0),
+ "ks": list(ks),
+ "n_docs": len(docs),
+ "n_queries": len(per_query),
+ "n_docs_mapped": len(texts),
+ "skip_enrichment": True,
+ "ingest_graph": False,
+ "skip_ingest": True,
+ "rank_pool": True,
+ "rank_pool_ce": True,
+ "ce_model": model,
+ "ce_missing_text": missing_text,
+ "ingest": {"status": "completed", "n_docs": len(docs), "tasks": [], "reused": True},
+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "event_probe": None,
+ "search_error": None,
+ "protocol": (
+ "ranking-in-pool CE over labeled candidates; "
+ f"model={model}; n is not Reddy ~4477 unless stated; "
+ "cite 0.857 only against this CE-on-pool protocol"
+ ),
+ "metrics": metrics,
+ "queries": per_query,
+ }
diff --git a/benchmarks/search/rank_pool_4class.py b/benchmarks/search/rank_pool_4class.py
new file mode 100644
index 0000000..0f1bcaa
--- /dev/null
+++ b/benchmarks/search/rank_pool_4class.py
@@ -0,0 +1,218 @@
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any, Callable
+
+from search.config import Settings, validate_brain_id
+from search.dataset import load_records, split_corpus
+from search.evaluate import candidate_pool_grades, candidate_pool_ids, ensure_run_dir
+from search.finetune_esci_4class import (
+ CLASS_GAINS,
+ DEFAULT_MODEL,
+ DEFAULT_OUT,
+ pick_device,
+ rank_doc_ids,
+ weighted_scores,
+)
+from search.metrics import aggregate_query_metrics, mrr, ndcg_at_k, recall_at_k
+from search.report import print_report_table, write_report
+
+PredictFn = Callable[[list[tuple[str, str]]], Any]
+
+
+def doc_texts(docs: list[dict[str, Any]]) -> dict[str, str]:
+ texts: dict[str, str] = {}
+ for doc in docs:
+ doc_id = str(doc.get("doc_id") or "")
+ if not doc_id:
+ continue
+ texts[doc_id] = str(doc.get("text") or doc.get("title") or "")
+ return texts
+
+
+def score_ranked_pool(
+ ranked: list[str],
+ query: dict[str, Any],
+ *,
+ ks: tuple[int, ...] = (5, 10, 20),
+) -> dict[str, float]:
+ gold = {
+ str(doc_id)
+ for doc_id, gain in candidate_pool_grades(query).items()
+ if float(gain) > 0
+ }
+ if not gold:
+ gold = {str(item) for item in (query.get("gold_doc_ids") or [])}
+ grades = candidate_pool_grades(query)
+ metrics = {f"recall@{k}": recall_at_k(ranked, gold, k) for k in ks}
+ metrics["ndcg@10"] = ndcg_at_k(ranked, gold, 10, grades=grades)
+ metrics["ndcg@20"] = ndcg_at_k(ranked, gold, 20, grades=grades)
+ metrics["ndcg"] = ndcg_at_k(ranked, gold, max(len(ranked), 1), grades=grades)
+ metrics["mrr"] = mrr(ranked, gold)
+ return metrics
+
+
+def load_4class_predict(model_dir: Path, *, max_length: int = 192) -> tuple[PredictFn, str]:
+ from sentence_transformers import CrossEncoder
+
+ device = pick_device()
+ model = CrossEncoder(
+ str(model_dir),
+ num_labels=4,
+ max_length=max_length,
+ device=device,
+ default_activation_function=None,
+ )
+
+ def predict(pairs: list[tuple[str, str]]):
+ if not pairs:
+ return []
+ return model.predict(
+ [list(pair) for pair in pairs],
+ batch_size=32,
+ apply_softmax=True,
+ show_progress_bar=False,
+ )
+
+ return predict, str(model_dir)
+
+
+def run_4class_on_pool(
+ rows: list[dict[str, Any]],
+ *,
+ dataset_name: str,
+ predict: PredictFn,
+ model_name: str,
+ ks: tuple[int, ...] = (5, 10, 20),
+ brain_id: str | None = None,
+) -> dict[str, Any]:
+ docs, queries = split_corpus(rows)
+ texts = doc_texts(docs)
+ per_query: list[dict[str, Any]] = []
+ missing_text = 0
+ for query in queries:
+ pool = candidate_pool_ids(query)
+ pairs: list[tuple[str, str]] = []
+ for doc_id in pool:
+ text = texts.get(doc_id) or ""
+ if not text:
+ missing_text += 1
+ pairs.append((str(query.get("query") or ""), text))
+ probs = predict(pairs) if pairs else []
+ scores = weighted_scores(probs, CLASS_GAINS) if len(pairs) else []
+ if len(scores) < len(pool):
+ scores = list(scores) + [0.0] * (len(pool) - len(scores))
+ ranked = rank_doc_ids(pool, scores)
+ metrics = score_ranked_pool(ranked, query, ks=ks)
+ per_query.append(
+ {
+ "qid": query.get("qid"),
+ "query": query.get("query"),
+ "slice": query.get("slice") or "unspecified",
+ "gold_doc_ids": list(query.get("gold_doc_ids") or []),
+ "candidate_doc_ids": pool,
+ "pool_size": len(pool),
+ "pool_coverage": 1.0 if pool else None,
+ "missing_from_brain": [],
+ "hit_ids": ranked,
+ "metrics": metrics,
+ "retrieve_ms": 0.0,
+ "embed_ms": None,
+ "client_wall_ms": 0.0,
+ "n_hits": len(ranked),
+ }
+ )
+ metrics = aggregate_query_metrics(per_query, ks=ks) if per_query else {}
+ return {
+ "status": "ok" if per_query else "failed",
+ "brain_id": brain_id,
+ "dataset": dataset_name,
+ "fusion": "none",
+ "rerank": "4class-weighted-ce",
+ "channels": ["rank-pool-4class"],
+ "expand": "none",
+ "k": max((len(row.get("hit_ids") or []) for row in per_query), default=0),
+ "ks": list(ks),
+ "n_docs": len(docs),
+ "n_queries": len(per_query),
+ "n_docs_mapped": len(texts),
+ "skip_enrichment": True,
+ "ingest_graph": False,
+ "skip_ingest": True,
+ "rank_pool": True,
+ "rank_pool_ce": True,
+ "ce_model": model_name,
+ "ce_missing_text": missing_text,
+ "ingest": {
+ "status": "completed",
+ "n_docs": len(docs),
+ "tasks": [],
+ "reused": True,
+ },
+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "event_probe": None,
+ "search_error": None,
+ "protocol": (
+ "ranking-in-pool 4-class CE over labeled candidates including I=0; "
+ "score=1.0*P(E)+0.1*P(S)+0.01*P(C); not MiniLM Exact→1; "
+ "n=74 is not Reddy public test n≈4477"
+ ),
+ "metrics": metrics,
+ "queries": per_query,
+ }
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(
+ description="Ranking-in-pool eval with a 4-class weighted cross-encoder"
+ )
+ parser.add_argument("--dataset", type=Path, required=True)
+ parser.add_argument("--model", type=Path, default=DEFAULT_OUT)
+ parser.add_argument("--run", default="search-esci-74-ce-pool-4class")
+ parser.add_argument("--brain", default="searchbenchesci74")
+ parser.add_argument("--env-file", type=Path, default=None)
+ parser.add_argument("--max-length", type=int, default=192)
+ args = parser.parse_args(argv)
+ if not args.dataset.exists():
+ print(f"Missing {args.dataset}")
+ return 1
+ if not args.model.exists():
+ print(f"Missing model {args.model}")
+ return 1
+ settings = Settings.load(args.env_file)
+ settings.brain_id = validate_brain_id(args.brain)
+ rows = load_records(args.dataset)
+ run_id, run_dir = ensure_run_dir(settings, args.run)
+ predict, model_name = load_4class_predict(args.model, max_length=args.max_length)
+ result = run_4class_on_pool(
+ rows,
+ dataset_name=args.dataset.name,
+ predict=predict,
+ model_name=model_name,
+ brain_id=settings.brain_id,
+ )
+ report = write_report(run_dir, result)
+ print_report_table(report)
+ print(f"Wrote {run_dir / 'report.json'}")
+ print(
+ json.dumps(
+ {
+ "run_id": run_id,
+ "ce_model": result.get("ce_model"),
+ "ndcg@20": report.get("ndcg@20"),
+ "ndcg": report.get("ndcg"),
+ "n_queries": report.get("n_queries"),
+ "missing_text": result.get("ce_missing_text"),
+ "base_default": DEFAULT_MODEL,
+ },
+ indent=2,
+ )
+ )
+ return 0 if report.get("status") == "ok" else 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/search/replay_fusion.py b/benchmarks/search/replay_fusion.py
new file mode 100644
index 0000000..72843a2
--- /dev/null
+++ b/benchmarks/search/replay_fusion.py
@@ -0,0 +1,611 @@
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+from search.metrics import mrr, ndcg_at_k, recall_at_k
+
+RRF_K = 60
+EXPANSION_N = 10
+GRAPH_WEIGHTS = (0.1, 0.25, 0.5)
+FUSED_NDCG = 0.653
+RECALL20_TARGET = 0.80
+_SLICE_JSONL = Path(__file__).resolve().parent.parent / "data" / "search_esci_slice.jsonl"
+
+
+def _load_eval(path: Path) -> dict[str, Any]:
+ return json.loads(path.read_text(encoding="utf-8"))
+
+
+def _gold(query: dict[str, Any]) -> set[str]:
+ gold = {str(item) for item in (query.get("gold_doc_ids") or []) if item}
+ grades = query.get("gold_grades") or {}
+ if isinstance(grades, dict):
+ for doc_id, gain in grades.items():
+ if float(gain or 0) <= 0:
+ continue
+ gold.add(str(doc_id))
+ if gold:
+ return gold
+ return {str(item) for item in (query.get("gold_chunk_ids") or []) if item}
+
+
+def dataset_gold_grades(path: Path | None = None) -> dict[str, dict[str, float]]:
+ jsonl = path or _SLICE_JSONL
+ if not jsonl.exists():
+ return {}
+ grades: dict[str, dict[str, float]] = {}
+ for line in jsonl.read_text(encoding="utf-8").splitlines():
+ if not line.strip():
+ continue
+ row = json.loads(line)
+ if row.get("type") != "query":
+ continue
+ qid = str(row.get("qid") or "")
+ raw = row.get("gold_grades") or {}
+ if not qid or not isinstance(raw, dict):
+ continue
+ grades[qid] = {
+ str(doc_id): float(gain)
+ for doc_id, gain in raw.items()
+ if float(gain or 0) > 0
+ }
+ return grades
+
+
+def _grades(
+ query: dict[str, Any],
+ gold: set[str],
+ dataset_grades: dict[str, dict[str, float]] | None = None,
+) -> dict[str, float]:
+ raw = query.get("gold_grades") or {}
+ if isinstance(raw, dict) and raw:
+ return {
+ str(doc_id): float(gain)
+ for doc_id, gain in raw.items()
+ if float(gain or 0) > 0
+ }
+ qid = str(query.get("qid") or "")
+ from_dataset = (dataset_grades or {}).get(qid) or {}
+ if from_dataset:
+ return from_dataset
+ return {doc_id: 1.0 for doc_id in gold}
+
+
+def _ranked(query: dict[str, Any]) -> list[str]:
+ return [str(item) for item in (query.get("hit_ids") or []) if item]
+
+
+def drop_hubs(ranked: list[str]) -> list[str]:
+ return [item for item in ranked if not item.startswith("hub:")]
+
+
+def keep_product_asins(ranked: list[str]) -> list[str]:
+ return [
+ item
+ for item in ranked
+ if item and not item.startswith("hub:") and not item.startswith("evt:")
+ ]
+
+
+def score_ranked(
+ ranked: list[str],
+ gold: set[str],
+ grades: dict[str, float],
+) -> dict[str, float]:
+ return {
+ "ndcg@10": ndcg_at_k(ranked, gold, 10, grades=grades),
+ "recall@10": recall_at_k(ranked, gold, 10),
+ "recall@20": recall_at_k(ranked, gold, 20),
+ "mrr": mrr(ranked, gold),
+ "unique_docs@20": float(len({str(item) for item in ranked[:20] if item})),
+ }
+
+
+def mean_metrics(rows: list[dict[str, float]]) -> dict[str, float]:
+ keys = ("ndcg@10", "recall@10", "recall@20", "mrr", "unique_docs@20")
+ if not rows:
+ return {key: 0.0 for key in keys}
+ present = [key for key in keys if all(key in row for row in rows)]
+ return {key: sum(row[key] for row in rows) / len(rows) for key in present}
+
+
+def reciprocal_rank_fusion(
+ ranked_lists: list[list[str]],
+ *,
+ k: int = RRF_K,
+ weights: list[float] | None = None,
+) -> list[str]:
+ scores: dict[str, float] = {}
+ for index, ranked in enumerate(ranked_lists):
+ weight = 1.0 if not weights else float(weights[index])
+ for rank, item in enumerate(ranked):
+ if not item:
+ continue
+ scores[item] = scores.get(item, 0.0) + weight / (k + rank + 1)
+ return [
+ item
+ for item, _ in sorted(scores.items(), key=lambda pair: pair[1], reverse=True)
+ ]
+
+
+def chunk_to_doc_map(*queries: dict[str, Any]) -> dict[str, str]:
+ mapping: dict[str, str] = {}
+ for query in queries:
+ for hit in query.get("hits") or []:
+ if not isinstance(hit, dict):
+ continue
+ hid = str(hit.get("id") or "")
+ doc_id = str(hit.get("doc_id") or hid)
+ if hid:
+ mapping[hid] = doc_id
+ return mapping
+
+
+def collapse_to_doc(
+ ids: list[str],
+ chunk_to_doc: dict[str, str],
+) -> list[str]:
+ out: list[str] = []
+ seen: set[str] = set()
+ for item in ids:
+ key = chunk_to_doc.get(str(item), str(item))
+ if (
+ not key
+ or key.startswith("hub:")
+ or key.startswith("evt:")
+ or key in seen
+ ):
+ continue
+ seen.add(key)
+ out.append(key)
+ return out
+
+
+def _id_list(query: dict[str, Any], key: str, channel: str | None = None) -> list[str]:
+ dumped = [str(item) for item in (query.get(key) or []) if item]
+ if dumped:
+ return dumped
+ if not channel:
+ return []
+ out: list[str] = []
+ for hit in query.get("hits") or []:
+ if not isinstance(hit, dict):
+ continue
+ hid = str(hit.get("id") or "")
+ if hid and str(hit.get("channel") or "") == channel:
+ out.append(hid)
+ return out
+
+
+def query_channel_lists(query: dict[str, Any]) -> dict[str, list[str]]:
+ return {
+ "dense": _id_list(query, "dense_ids"),
+ "bm25": _id_list(query, "bm25_ids"),
+ "passages": _id_list(query, "passage_ids", "passages"),
+ "entities": _id_list(query, "entity_ids", "entities"),
+ "communities": _id_list(query, "community_ids", "communities"),
+ }
+
+
+def merge_query_lists(
+ graph_query: dict[str, Any],
+ passages_query: dict[str, Any] | None = None,
+ entities_query: dict[str, Any] | None = None,
+ communities_query: dict[str, Any] | None = None,
+) -> dict[str, list[str]]:
+ graph = query_channel_lists(graph_query)
+ passages = query_channel_lists(passages_query or {})
+ entities = query_channel_lists(entities_query or {})
+ communities = query_channel_lists(communities_query or {})
+ dense = graph["dense"] or passages["dense"]
+ bm25 = graph["bm25"] or passages["bm25"]
+ passage_ids = passages["passages"] or graph["passages"]
+ if not passage_ids and passages_query:
+ passage_ids = [
+ str(hit.get("id") or "")
+ for hit in (passages_query.get("hits") or [])
+ if hit.get("id")
+ ]
+ if not passage_ids:
+ passage_ids = _ranked(passages_query)
+ entity_ids = graph["entities"] or entities["entities"]
+ if not entity_ids and entities_query:
+ entity_ids = [
+ str(hit.get("id") or "")
+ for hit in (entities_query.get("hits") or [])
+ if hit.get("id")
+ ]
+ if not entity_ids:
+ entity_ids = _ranked(entities_query)
+ community_ids = graph["communities"] or communities["communities"]
+ if not community_ids and communities_query:
+ community_ids = [
+ str(hit.get("id") or "")
+ for hit in (communities_query.get("hits") or [])
+ if hit.get("id")
+ ]
+ if not community_ids:
+ community_ids = _ranked(communities_query)
+ return {
+ "dense": dense,
+ "bm25": bm25,
+ "passages": passage_ids,
+ "entities": entity_ids,
+ "communities": community_ids,
+ }
+
+
+def collapsed_lists(
+ lists: dict[str, list[str]],
+ chunk_to_doc: dict[str, str],
+) -> dict[str, list[str]]:
+ collapsed = {
+ name: collapse_to_doc(ids, chunk_to_doc) for name, ids in lists.items()
+ }
+ if not collapsed["dense"] and not collapsed["bm25"] and collapsed["passages"]:
+ collapsed["passages_core"] = collapsed["passages"]
+ else:
+ collapsed["passages_core"] = reciprocal_rank_fusion(
+ [ids for ids in (collapsed["dense"], collapsed["bm25"]) if ids]
+ ) or collapsed["passages"]
+ return collapsed
+
+
+def _by_qid(eval_result: dict[str, Any] | None) -> dict[str, dict[str, Any]]:
+ if not eval_result:
+ return {}
+ return {
+ str(row.get("qid")): row for row in (eval_result.get("queries") or [])
+ }
+
+
+def _pair_queries(
+ graph_eval: dict[str, Any],
+ passages_eval: dict[str, Any],
+ entities_eval: dict[str, Any] | None = None,
+ communities_eval: dict[str, Any] | None = None,
+) -> list[dict[str, Any]]:
+ passages_by = _by_qid(passages_eval)
+ entities_by = _by_qid(entities_eval)
+ communities_by = _by_qid(communities_eval)
+ slice_grades = dataset_gold_grades()
+ rows: list[dict[str, Any]] = []
+ for graph_query in graph_eval.get("queries") or []:
+ qid = str(graph_query.get("qid") or "")
+ passages_query = passages_by.get(qid) or {}
+ entities_query = entities_by.get(qid)
+ communities_query = communities_by.get(qid)
+ gold = _gold(graph_query) or _gold(passages_query)
+ grades = _grades(graph_query, gold, slice_grades)
+ if not (graph_query.get("gold_grades") or {}):
+ grades = _grades(passages_query, gold, slice_grades)
+ mapping = chunk_to_doc_map(
+ graph_query, passages_query, entities_query or {}, communities_query or {}
+ )
+ lists = merge_query_lists(
+ graph_query, passages_query, entities_query, communities_query
+ )
+ collapsed = collapsed_lists(lists, mapping)
+ rows.append(
+ {
+ "qid": qid,
+ "gold": gold,
+ "grades": grades,
+ "lists": lists,
+ "collapsed": collapsed,
+ "passages_ranked": collapse_to_doc(
+ lists["passages"] or _ranked(passages_query), mapping
+ ),
+ "graph_ranked": _ranked(graph_query),
+ "passage_metrics": passages_query.get("metrics") or {},
+ }
+ )
+ return rows
+
+
+def replay(graph_eval: dict[str, Any], passages_eval: dict[str, Any]) -> dict[str, Any]:
+ passages_by_qid = _by_qid(passages_eval)
+ hub_rows: list[dict[str, float]] = []
+ asin_rows: list[dict[str, float]] = []
+ per_query: list[dict[str, Any]] = []
+ for query in graph_eval.get("queries") or []:
+ qid = str(query.get("qid") or "")
+ gold = _gold(query)
+ grades = _grades(query, gold)
+ ranked = _ranked(query)
+ hub = score_ranked(drop_hubs(ranked), gold, grades)
+ asin = score_ranked(keep_product_asins(ranked), gold, grades)
+ hub_rows.append(hub)
+ asin_rows.append(asin)
+ passage_row = passages_by_qid.get(qid) or {}
+ passage_metrics = passage_row.get("metrics") or {}
+ per_query.append(
+ {
+ "qid": qid,
+ "n_hits": len(ranked),
+ "n_hits_no_hub": len(drop_hubs(ranked)),
+ "n_product_ids": len(keep_product_asins(ranked)),
+ "hub_drop": hub,
+ "product_ids": asin,
+ "passages": {
+ "ndcg@10": float(passage_metrics.get("ndcg@10") or 0.0),
+ "recall@10": float(passage_metrics.get("recall@10") or 0.0),
+ "recall@20": float(passage_metrics.get("recall@20") or 0.0),
+ "mrr": float(passage_metrics.get("mrr") or 0.0),
+ },
+ }
+ )
+ return {
+ "n_queries": len(per_query),
+ "hub_drop": mean_metrics(hub_rows),
+ "product_ids": mean_metrics(asin_rows),
+ "queries": per_query,
+ }
+
+
+def _core_and_graph(collapsed: dict[str, list[str]]) -> tuple[list[str], list[str], list[str]]:
+ core = [ids for ids in (collapsed["dense"], collapsed["bm25"]) if ids]
+ if not core and collapsed.get("passages_core"):
+ core = [collapsed["passages_core"]]
+ entities = collapsed.get("entities") or []
+ communities = collapsed.get("communities") or []
+ return core, entities, communities
+
+
+def rank_collapse_rrf(collapsed: dict[str, list[str]]) -> list[str]:
+ core, entities, communities = _core_and_graph(collapsed)
+ lists = [*core]
+ if entities:
+ lists.append(entities)
+ if communities:
+ lists.append(communities)
+ if not lists:
+ return []
+ if len(lists) == 1:
+ return list(lists[0])
+ return reciprocal_rank_fusion(lists)
+
+
+def rank_weighted_rrf(collapsed: dict[str, list[str]], graph_weight: float) -> list[str]:
+ core, entities, communities = _core_and_graph(collapsed)
+ lists: list[list[str]] = [*core]
+ weights: list[float] = [1.0] * len(core)
+ if entities:
+ lists.append(entities)
+ weights.append(graph_weight)
+ if communities:
+ lists.append(communities)
+ weights.append(graph_weight)
+ if not lists:
+ return []
+ if len(lists) == 1:
+ return list(lists[0])
+ return reciprocal_rank_fusion(lists, weights=weights)
+
+
+def rank_expansion(
+ passage_docs: list[str],
+ collapsed: dict[str, list[str]],
+ *,
+ n: int = EXPANSION_N,
+) -> list[str]:
+ head = list(passage_docs[:10])
+ passage_set = set(passage_docs)
+ graph_docs: list[str] = []
+ seen: set[str] = set()
+ for item in (collapsed.get("entities") or []) + (collapsed.get("communities") or []):
+ if item in seen:
+ continue
+ seen.add(item)
+ graph_docs.append(item)
+ novel = [item for item in graph_docs if item not in passage_set][:n]
+ novel_set = set(novel)
+ rest = [item for item in passage_docs[10:] if item not in novel_set]
+ return head + novel + rest
+
+
+def rank_confirmation(
+ passage_docs: list[str],
+ collapsed: dict[str, list[str]],
+) -> list[str]:
+ passage_set = set(passage_docs)
+ confirmed: list[str] = []
+ seen: set[str] = set()
+ for item in (collapsed.get("entities") or []) + (collapsed.get("communities") or []):
+ if item not in passage_set or item in seen:
+ continue
+ seen.add(item)
+ confirmed.append(item)
+ if not confirmed:
+ return list(passage_docs)
+ return reciprocal_rank_fusion([passage_docs, confirmed])
+
+
+def replay_offline(
+ graph_eval: dict[str, Any],
+ passages_eval: dict[str, Any],
+ *,
+ entities_eval: dict[str, Any] | None = None,
+ communities_eval: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+ paired = _pair_queries(
+ graph_eval, passages_eval, entities_eval, communities_eval
+ )
+ arms: dict[str, list[dict[str, float]]] = {
+ "passages": [],
+ "collapse-rrf": [],
+ "weighted-0.1": [],
+ "weighted-0.25": [],
+ "weighted-0.5": [],
+ "expansion-n10": [],
+ "confirmation": [],
+ }
+ per_query: list[dict[str, Any]] = []
+ for row in paired:
+ gold = row["gold"]
+ grades = row["grades"]
+ collapsed = row["collapsed"]
+ passage_docs = row["passages_ranked"]
+ scored = {
+ "passages": score_ranked(passage_docs, gold, grades),
+ "collapse-rrf": score_ranked(
+ rank_collapse_rrf(collapsed), gold, grades
+ ),
+ "expansion-n10": score_ranked(
+ rank_expansion(passage_docs, collapsed, n=EXPANSION_N),
+ gold,
+ grades,
+ ),
+ "confirmation": score_ranked(
+ rank_confirmation(passage_docs, collapsed), gold, grades
+ ),
+ }
+ for weight in GRAPH_WEIGHTS:
+ name = f"weighted-{weight}"
+ scored[name] = score_ranked(
+ rank_weighted_rrf(collapsed, weight), gold, grades
+ )
+ for name, metrics in scored.items():
+ arms[name].append(metrics)
+ per_query.append({"qid": row["qid"], **scored})
+ summary = {name: mean_metrics(rows) for name, rows in arms.items()}
+ return {
+ "n_queries": len(per_query),
+ "expansion_n": EXPANSION_N,
+ "graph_weights": list(GRAPH_WEIGHTS),
+ "primary": ["recall@20", "ndcg@10"],
+ "arms": summary,
+ "queries": per_query,
+ }
+
+
+def pick_offline_winner(summary: dict[str, dict[str, float]]) -> str:
+ passages = summary.get("passages") or {}
+ passages_ndcg = float(passages.get("ndcg@10") or 0.0)
+ passages_r20 = float(passages.get("recall@20") or 0.0)
+ candidates: list[tuple[str, dict[str, float]]] = []
+ for name, metrics in summary.items():
+ if name == "passages":
+ continue
+ recall20 = float(metrics.get("recall@20") or 0.0)
+ ndcg = float(metrics.get("ndcg@10") or 0.0)
+ if recall20 + 1e-9 < max(RECALL20_TARGET, passages_r20):
+ continue
+ if ndcg + 1e-9 < max(FUSED_NDCG, passages_ndcg):
+ continue
+ candidates.append((name, metrics))
+ if not candidates:
+ return "G08"
+
+ def _key(item: tuple[str, dict[str, float]]) -> tuple[float, float]:
+ metrics = item[1]
+ return (
+ float(metrics.get("recall@20") or 0.0),
+ float(metrics.get("ndcg@10") or 0.0),
+ )
+
+ name, _metrics = max(candidates, key=_key)
+ return name
+
+
+def _print_metrics(label: str, metrics: dict[str, float]) -> None:
+ unique = metrics.get("unique_docs@20")
+ unique_s = f" unique@20={unique:.2f}" if unique is not None else ""
+ print(
+ f"{label:18} "
+ f"nDCG@10={metrics.get('ndcg@10', 0.0):.3f} "
+ f"R@10={metrics.get('recall@10', 0.0):.3f} "
+ f"R@20={metrics.get('recall@20', 0.0):.3f} "
+ f"MRR={metrics.get('mrr', 0.0):.3f}"
+ f"{unique_s}"
+ )
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(
+ description="Offline rescore of fused search eval.json lists."
+ )
+ parser.add_argument("--graph", required=True, help="Graph-arm eval.json")
+ parser.add_argument("--passages", required=True, help="Passages-arm eval.json")
+ parser.add_argument(
+ "--entities",
+ default=None,
+ help="Optional isolated entities eval.json",
+ )
+ parser.add_argument(
+ "--communities",
+ default=None,
+ help="Optional isolated communities eval.json",
+ )
+ parser.add_argument(
+ "--mode",
+ default="hub-drop",
+ choices=("hub-drop", "collapse-rrf", "gated", "all"),
+ )
+ args = parser.parse_args(argv)
+ graph_eval = _load_eval(Path(args.graph))
+ passages_eval = _load_eval(Path(args.passages))
+ entities_eval = _load_eval(Path(args.entities)) if args.entities else None
+ communities_eval = (
+ _load_eval(Path(args.communities)) if args.communities else None
+ )
+ if args.mode in {"hub-drop", "all"}:
+ result = replay(graph_eval, passages_eval)
+ hub = result["hub_drop"]
+ print(
+ "hub-drop "
+ f"nDCG@10={hub['ndcg@10']:.3f} "
+ f"R@10={hub['recall@10']:.3f} "
+ f"R@20={hub['recall@20']:.3f} "
+ f"MRR={hub['mrr']:.3f}"
+ )
+ asin = result["product_ids"]
+ print(
+ "product-ids "
+ f"nDCG@10={asin['ndcg@10']:.3f} "
+ f"R@10={asin['recall@10']:.3f} "
+ f"R@20={asin['recall@20']:.3f} "
+ f"MRR={asin['mrr']:.3f}"
+ )
+ for row in result["queries"]:
+ delta = row["hub_drop"]["ndcg@10"] - row["passages"]["ndcg@10"]
+ print(
+ f"{row['qid']}: hub-drop nDCG {row['hub_drop']['ndcg@10']:.3f} "
+ f"(passages {row['passages']['ndcg@10']:.3f}, delta {delta:+.3f})"
+ )
+ if args.mode in {"collapse-rrf", "gated", "all"}:
+ offline = replay_offline(
+ graph_eval,
+ passages_eval,
+ entities_eval=entities_eval,
+ communities_eval=communities_eval,
+ )
+ names = ["passages", "collapse-rrf"]
+ if args.mode in {"gated", "all"}:
+ names.extend(
+ [
+ "weighted-0.1",
+ "weighted-0.25",
+ "weighted-0.5",
+ "expansion-n10",
+ "confirmation",
+ ]
+ )
+ print(
+ "primary recall@20 then nDCG@10; "
+ f"expansion_n={offline['expansion_n']}; "
+ f"graph_weights={offline['graph_weights']}"
+ )
+ for name in names:
+ _print_metrics(name, offline["arms"][name])
+ if args.mode in {"gated", "all"}:
+ winner = pick_offline_winner(offline["arms"])
+ print(f"winner {winner}")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/search/report.py b/benchmarks/search/report.py
new file mode 100644
index 0000000..1bf5cc4
--- /dev/null
+++ b/benchmarks/search/report.py
@@ -0,0 +1,269 @@
+from __future__ import annotations
+
+import json
+import subprocess
+import tempfile
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from rich.console import Console
+from rich.table import Table
+
+from search.config import BENCHMARKS_ROOT
+
+console = Console()
+
+REPORTS_PATH = BENCHMARKS_ROOT / "REPORTS.json"
+_BENCHMARK_ID = "search"
+_BENCHMARK_NAME = "Search (hybrid BM25 + dense)"
+_REPORTS_DESCRIPTION = (
+ "BrainAPI benchmark results. Top published scores across suites. "
+ "Updated when a suite evaluate/report completes successfully."
+)
+
+
+def _git_sha() -> str | None:
+ try:
+ out = subprocess.check_output(
+ ["git", "rev-parse", "HEAD"],
+ cwd=BENCHMARKS_ROOT.parent,
+ stderr=subprocess.DEVNULL,
+ text=True,
+ )
+ return out.strip() or None
+ except Exception:
+ return None
+
+
+def build_report(run_dir: Path, eval_result: dict[str, Any]) -> dict[str, Any]:
+ metrics = eval_result.get("metrics") or {}
+ status = str(eval_result.get("status") or "failed")
+ n_queries = int(eval_result.get("n_queries") or 0)
+ if status != "ok":
+ status = "failed"
+ if n_queries < 1:
+ status = "failed"
+ if (eval_result.get("ingest") or {}).get("status") not in {
+ "completed",
+ "partial_failed",
+ }:
+ status = "failed"
+
+ report = {
+ "suite": _BENCHMARK_ID,
+ "status": status,
+ "run_id": run_dir.name,
+ "brain_id": eval_result.get("brain_id"),
+ "dataset": eval_result.get("dataset"),
+ "fusion": eval_result.get("fusion"),
+ "rerank": eval_result.get("rerank") or "none",
+ "channels": eval_result.get("channels") or ["passages"],
+ "rank_pool": bool(eval_result.get("rank_pool")),
+ "rank_pool_ce": bool(eval_result.get("rank_pool_ce")),
+ "ce_model": eval_result.get("ce_model"),
+ "skip_enrichment": eval_result.get("skip_enrichment"),
+ "n_queries": n_queries,
+ "n_docs": eval_result.get("n_docs"),
+ "k": eval_result.get("k"),
+ "ndcg@10": metrics.get("ndcg@10"),
+ "ndcg@20": metrics.get("ndcg@20"),
+ "ndcg@50": metrics.get("ndcg@50"),
+ "ndcg@100": metrics.get("ndcg@100"),
+ "ndcg": metrics.get("ndcg"),
+ "recall@10": metrics.get("recall@10"),
+ "recall@20": metrics.get("recall@20"),
+ "recall@50": metrics.get("recall@50"),
+ "recall@100": metrics.get("recall@100"),
+ "mrr": metrics.get("mrr"),
+ "pool_coverage": metrics.get("pool_coverage"),
+ "p50_retrieve_ms": metrics.get("p50_retrieve_ms"),
+ "p95_retrieve_ms": metrics.get("p95_retrieve_ms"),
+ "metrics": metrics,
+ "ingest": eval_result.get("ingest"),
+ "git_sha": _git_sha(),
+ "recorded_at": datetime.now(timezone.utc).isoformat(),
+ "protocol": eval_result.get("protocol")
+ or (
+ "labeled toy search: POST /ingest/ → POST /retrieve/search "
+ f"(fusion={eval_result.get('fusion') or 'rrf'}, "
+ f"rerank={eval_result.get('rerank') or 'none'}, "
+ f"rank_pool={bool(eval_result.get('rank_pool'))})"
+ ),
+ }
+ return report
+
+
+def write_report(run_dir: Path, eval_result: dict[str, Any]) -> dict[str, Any]:
+ report = build_report(run_dir, eval_result)
+ (run_dir / "eval.json").write_text(
+ json.dumps(eval_result, indent=2) + "\n", encoding="utf-8"
+ )
+ (run_dir / "report.json").write_text(
+ json.dumps(report, indent=2) + "\n", encoding="utf-8"
+ )
+ md = render_markdown(report)
+ (run_dir / "report.md").write_text(md, encoding="utf-8")
+ if report.get("status") == "ok":
+ update_reports_json(report)
+ return report
+
+
+def render_markdown(report: dict[str, Any]) -> str:
+ lines = [
+ f"# Search report — `{report.get('run_id')}`",
+ "",
+ f"- status: **{report.get('status')}**",
+ f"- dataset: `{report.get('dataset')}`",
+ f"- brain_id: `{report.get('brain_id')}`",
+ f"- fusion: `{report.get('fusion')}`",
+ f"- rerank: `{report.get('rerank')}`",
+ f"- channels: `{','.join(report.get('channels') or ['passages'])}`",
+ f"- rank_pool: `{report.get('rank_pool')}`",
+ f"- skip_enrichment: `{report.get('skip_enrichment')}`",
+ f"- k: {report.get('k')}",
+ f"- n_queries: {report.get('n_queries')}",
+ f"- nDCG@10: {report.get('ndcg@10')}",
+ f"- nDCG@20: {report.get('ndcg@20')}",
+ f"- nDCG@50: {report.get('ndcg@50')}",
+ f"- nDCG (full list): {report.get('ndcg')}",
+ f"- Recall@10: {report.get('recall@10')}",
+ f"- Recall@20: {report.get('recall@20')}",
+ f"- Recall@50: {report.get('recall@50')}",
+ f"- pool_coverage: {report.get('pool_coverage')}",
+ f"- MRR: {report.get('mrr')}",
+ f"- p50 retrieve ms (ex-embed): {report.get('p50_retrieve_ms')}",
+ f"- p95 retrieve ms (ex-embed): {report.get('p95_retrieve_ms')}",
+ "",
+ "Never mutate LoCoMo / BEAM / LongMemEval / RecSys brains or their ledger rows.",
+ "",
+ ]
+ return "\n".join(lines)
+
+
+def print_report_table(report: dict[str, Any]) -> None:
+ table = Table(title=f"Search {report.get('run_id')}")
+ table.add_column("metric")
+ table.add_column("value")
+ for key in (
+ "status",
+ "dataset",
+ "brain_id",
+ "fusion",
+ "rerank",
+ "channels",
+ "rank_pool",
+ "skip_enrichment",
+ "n_queries",
+ "k",
+ "ndcg@10",
+ "ndcg@20",
+ "ndcg@50",
+ "ndcg",
+ "recall@10",
+ "recall@20",
+ "recall@50",
+ "pool_coverage",
+ "mrr",
+ "p50_retrieve_ms",
+ "p95_retrieve_ms",
+ ):
+ table.add_row(key, str(report.get(key)))
+ console.print(table)
+
+
+def entry_from_report(report: dict[str, Any]) -> dict[str, Any] | None:
+ if report.get("status") != "ok":
+ return None
+ n_queries = int(report.get("n_queries") or 0)
+ if n_queries < 1:
+ return None
+ entry = {
+ "run_id": report.get("run_id"),
+ "brain": report.get("brain_id"),
+ "dataset": report.get("dataset"),
+ "fusion": report.get("fusion"),
+ "rerank": report.get("rerank") or "none",
+ "channels": report.get("channels") or ["passages"],
+ "rank_pool": bool(report.get("rank_pool")),
+ "rank_pool_ce": bool(report.get("rank_pool_ce")),
+ "n_queries": n_queries,
+ "k": report.get("k"),
+ "ndcg@10": report.get("ndcg@10"),
+ "ndcg@20": report.get("ndcg@20"),
+ "ndcg@50": report.get("ndcg@50"),
+ "ndcg": report.get("ndcg"),
+ "recall@10": report.get("recall@10"),
+ "recall@20": report.get("recall@20"),
+ "recall@50": report.get("recall@50"),
+ "mrr": report.get("mrr"),
+ "pool_coverage": report.get("pool_coverage"),
+ "p50_retrieve_ms": report.get("p50_retrieve_ms"),
+ "p95_retrieve_ms": report.get("p95_retrieve_ms"),
+ "git_sha": report.get("git_sha"),
+ "report_path": f"runs/{report.get('run_id')}/report.json",
+ "recorded_at": report.get("recorded_at"),
+ "protocol": report.get("protocol"),
+ }
+ if report.get("brain_id") == "searchbenchwandsgraph":
+ entry["claim"] = "architecture-demo"
+ if report.get("brain_id") == "searchbenchjdslice":
+ return None
+ return entry
+
+
+def _empty_reports_ledger() -> dict[str, Any]:
+ return {
+ "schema_version": 2,
+ "updated_at": datetime.now(timezone.utc).isoformat(),
+ "description": _REPORTS_DESCRIPTION,
+ "benchmarks": {},
+ }
+
+
+def update_reports_json(report: dict[str, Any]) -> None:
+ entry = entry_from_report(report)
+ if entry is None:
+ return
+
+ if REPORTS_PATH.exists():
+ data = json.loads(REPORTS_PATH.read_text(encoding="utf-8"))
+ else:
+ data = _empty_reports_ledger()
+
+ suite = data.setdefault("benchmarks", {}).setdefault(
+ _BENCHMARK_ID,
+ {"name": _BENCHMARK_NAME, "leaderboard": []},
+ )
+ leaderboard = list(suite.get("leaderboard") or [])
+ run_id = entry["run_id"]
+ lb_idx = next(
+ (i for i, row in enumerate(leaderboard) if row.get("run_id") == run_id),
+ None,
+ )
+ if lb_idx is not None:
+ leaderboard[lb_idx] = entry
+ else:
+ leaderboard.append(entry)
+
+ suite["name"] = _BENCHMARK_NAME
+ suite["leaderboard"] = leaderboard
+ data["benchmarks"][_BENCHMARK_ID] = suite
+ data["updated_at"] = datetime.now(timezone.utc).isoformat()
+ data["description"] = data.get("description") or _REPORTS_DESCRIPTION
+
+ REPORTS_PATH.parent.mkdir(parents=True, exist_ok=True)
+ fd, tmp_name = tempfile.mkstemp(
+ prefix="REPORTS.",
+ suffix=".json.tmp",
+ dir=str(REPORTS_PATH.parent),
+ )
+ try:
+ with open(fd, "w", encoding="utf-8") as fh:
+ json.dump(data, fh, indent=2)
+ fh.write("\n")
+ Path(tmp_name).replace(REPORTS_PATH)
+ finally:
+ tmp_path = Path(tmp_name)
+ if tmp_path.exists():
+ tmp_path.unlink(missing_ok=True)
diff --git a/benchmarks/search/rerank_retrieved.py b/benchmarks/search/rerank_retrieved.py
new file mode 100644
index 0000000..a5ad576
--- /dev/null
+++ b/benchmarks/search/rerank_retrieved.py
@@ -0,0 +1,143 @@
+from __future__ import annotations
+
+from typing import Any
+
+from search.evaluate import candidate_pool_grades
+from search.metrics import aggregate_query_metrics, mrr, ndcg_at_k, recall_at_k
+from search.rank_pool import _plugin_rerank, doc_texts, score_ranked_pool
+from search.dataset import split_corpus
+
+
+def _first_stage_metrics(
+ ranked: list[str],
+ query_row: dict[str, Any],
+ *,
+ ks: tuple[int, ...],
+) -> dict[str, float]:
+ gold = {str(item) for item in (query_row.get("gold_doc_ids") or [])}
+ grades = {
+ str(doc_id): float(gain)
+ for doc_id, gain in (query_row.get("gold_grades") or {}).items()
+ if float(gain or 0) > 0
+ }
+ metrics = {f"recall@{cut}": recall_at_k(ranked, gold, cut) for cut in ks}
+ ndcg_ks = tuple(dict.fromkeys((*ks, 10, 20)))
+ for cut in ndcg_ks:
+ metrics[f"ndcg@{cut}"] = ndcg_at_k(ranked, gold, cut, grades=grades or None)
+ metrics["ndcg"] = ndcg_at_k(ranked, gold, max(len(ranked), 1), grades=grades or None)
+ metrics["mrr"] = mrr(ranked, gold)
+ return metrics
+
+
+def run_ce_on_retrieved(
+ eval_result: dict[str, Any],
+ rows: list[dict[str, Any]],
+ *,
+ dataset_name: str,
+ ks: tuple[int, ...] | None = None,
+ brain_id: str | None = None,
+) -> dict[str, Any]:
+ docs, queries = split_corpus(rows)
+ texts = doc_texts(docs)
+ by_qid = {str(query.get("qid") or ""): query for query in queries}
+ rerank, model = _plugin_rerank()
+ source_ks = tuple(int(item) for item in (eval_result.get("ks") or (5, 10, 20)))
+ ks = ks or source_ks
+ rank_pool = bool(eval_result.get("rank_pool"))
+ per_query: list[dict[str, Any]] = []
+ missing_text = 0
+ for row in eval_result.get("queries") or []:
+ hit_ids = [str(item) for item in (row.get("hit_ids") or []) if item]
+ qid = str(row.get("qid") or "")
+ source = by_qid.get(qid) or {}
+ candidates = []
+ for doc_id in hit_ids:
+ text = texts.get(doc_id) or ""
+ if not text:
+ missing_text += 1
+ candidates.append({"id": doc_id, "text": text, "score": 0.0})
+ ranked_rows = rerank(
+ str(row.get("query") or source.get("query") or ""),
+ candidates,
+ len(candidates) or 1,
+ )
+ ranked = [str(item.get("id") or "") for item in ranked_rows if item.get("id")]
+ seen = set(ranked)
+ for doc_id in hit_ids:
+ if doc_id not in seen:
+ ranked.append(doc_id)
+ if rank_pool and source:
+ metrics = score_ranked_pool(ranked, source, ks=ks)
+ else:
+ metrics = _first_stage_metrics(ranked, row, ks=ks)
+ per_query.append(
+ {
+ "qid": row.get("qid"),
+ "query": row.get("query"),
+ "slice": row.get("slice") or "unspecified",
+ "gold_doc_ids": list(row.get("gold_doc_ids") or []),
+ "gold_grades": row.get("gold_grades") or {},
+ "candidate_doc_ids": row.get("candidate_doc_ids") or [],
+ "pool_size": row.get("pool_size"),
+ "pool_coverage": row.get("pool_coverage"),
+ "missing_from_brain": row.get("missing_from_brain") or [],
+ "hit_ids": ranked,
+ "metrics": metrics,
+ "retrieve_ms": float(row.get("retrieve_ms") or 0.0),
+ "embed_ms": row.get("embed_ms"),
+ "client_wall_ms": float(row.get("client_wall_ms") or 0.0),
+ "n_hits": len(ranked),
+ }
+ )
+ metrics = aggregate_query_metrics(per_query, ks=ks) if per_query else {}
+ if rank_pool and per_query:
+ coverages = [
+ float(row["pool_coverage"])
+ for row in per_query
+ if row.get("pool_coverage") is not None
+ ]
+ if coverages:
+ metrics["pool_coverage"] = sum(coverages) / len(coverages)
+ metrics["missing_from_brain"] = sum(
+ len(row.get("missing_from_brain") or []) for row in per_query
+ ) / len(per_query)
+ retrieve_k = int(eval_result.get("k") or 0)
+ return {
+ "status": "ok" if per_query else "failed",
+ "brain_id": brain_id or eval_result.get("brain_id"),
+ "dataset": dataset_name,
+ "fusion": eval_result.get("fusion") or "rrf",
+ "rerank": "harness:cross-encoder",
+ "channels": eval_result.get("channels") or ["passages"],
+ "expand": eval_result.get("expand") or "none",
+ "k": retrieve_k,
+ "ks": list(ks),
+ "n_docs": eval_result.get("n_docs") or len(docs),
+ "n_queries": len(per_query),
+ "n_docs_mapped": len(texts),
+ "skip_enrichment": True,
+ "ingest_graph": False,
+ "skip_ingest": True,
+ "rank_pool": rank_pool,
+ "rank_pool_ce": False,
+ "ce_model": model,
+ "ce_missing_text": missing_text,
+ "source_run": eval_result.get("run_id"),
+ "ingest": {
+ "status": "completed",
+ "n_docs": eval_result.get("n_docs") or len(docs),
+ "tasks": [],
+ "reused": True,
+ },
+ "graph_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "interaction_ingest": {"status": "skipped", "n_triples": 0, "tasks": []},
+ "event_probe": None,
+ "search_error": None,
+ "protocol": (
+ "harness CE over retrieved first-stage hits "
+ f"(k={retrieve_k}, model={model}); not CE-on-pool; "
+ "do not cite Reddy 0.857"
+ ),
+ "metrics": metrics,
+ "queries": per_query,
+ }
diff --git a/tests/fixtures/jdsearch/product_meta_data.txt b/tests/fixtures/jdsearch/product_meta_data.txt
new file mode 100644
index 0000000..f47d95e
--- /dev/null
+++ b/tests/fixtures/jdsearch/product_meta_data.txt
@@ -0,0 +1,3 @@
+wid name brand_id brand_name cate_id_1 cate_name_1 cate_id_2 cate_name_2 cate_id_3 cate_name_3 cate_id_4 cate_name_4 shop_id
+100 111222 b1 brandX c1 cat1 c2 cat2 c3 cat3 c4 cat4 s1
+200 333 b2 brandY c1 cat1 c2 cat2 c3 cat3 c4 cat4 s2
diff --git a/tests/fixtures/jdsearch/user_behavior_data.txt b/tests/fixtures/jdsearch/user_behavior_data.txt
new file mode 100644
index 0000000..d8d53b7
--- /dev/null
+++ b/tests/fixtures/jdsearch/user_behavior_data.txt
@@ -0,0 +1,3 @@
+query candidate_wid_list candidate_label_list history_qry_list history_wid_list history_type_list history_time_list
+111222 100_200_999 1.0_0.0_0.0 -1 200 CLICK 0_120
+333 200_100 2.0_1.0 111 100_200 CART_FLW 0_50_10
diff --git a/tests/test_benchmark_contract_guardrails.py b/tests/test_benchmark_contract_guardrails.py
index 69cd877..2aecc31 100644
--- a/tests/test_benchmark_contract_guardrails.py
+++ b/tests/test_benchmark_contract_guardrails.py
@@ -70,6 +70,36 @@ class BenchmarkContractGuardrailsTests(unittest.TestCase):
msg=f"{path} should still target core ingest",
)
+ def test_search_client_scores_via_search_not_context(self):
+ client = ROOT / "benchmarks" / "search" / "client.py"
+ self.assertTrue(client.exists(), msg="benchmarks/search/client.py is required")
+ text = client.read_text(encoding="utf-8")
+ self.assertIn("/retrieve/search", text)
+ self.assertIn("/ingest/", text)
+ self.assertIn("skip_enrichment", text)
+ self.assertNotIn(
+ "/retrieve/context",
+ text,
+ msg="search harness must not score via /retrieve/context",
+ )
+ self.assertNotIn(
+ "/retrieve/recommend",
+ text,
+ msg="search harness must not score via /retrieve/recommend",
+ )
+ for needle in ("locomoconv", "demorecsys"):
+ self.assertNotIn(
+ needle,
+ text,
+ msg=f"search client must not default to {needle}",
+ )
+
+ def test_search_config_defaults_to_searchbench(self):
+ config = ROOT / "benchmarks" / "search" / "config.py"
+ text = config.read_text(encoding="utf-8")
+ self.assertIn('DEFAULT_BRAIN_ID = "searchbenchsmoke"', text)
+ self.assertIn('REQUIRED_PREFIX = "searchbench"', text)
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_search_bench.py b/tests/test_search_bench.py
new file mode 100644
index 0000000..62d0695
--- /dev/null
+++ b/tests/test_search_bench.py
@@ -0,0 +1,1678 @@
+import json
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+
+class SearchBenchTests(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ import sys
+
+ root = Path(__file__).resolve().parents[1]
+ bench = root / "benchmarks"
+ if str(bench) not in sys.path:
+ sys.path.insert(0, str(bench))
+
+ def test_brain_id_guard(self):
+ from search.config import DEFAULT_BRAIN_ID, validate_brain_id
+
+ self.assertEqual(validate_brain_id("searchbenchsmoke"), "searchbenchsmoke")
+ self.assertEqual(validate_brain_id(DEFAULT_BRAIN_ID), DEFAULT_BRAIN_ID)
+ self.assertEqual(validate_brain_id("searchbench1"), "searchbench1")
+ self.assertEqual(validate_brain_id("searchbenchjdslice"), "searchbenchjdslice")
+ for bad in (
+ "locomoconv26",
+ "locomoconv26clean",
+ "beam1m1clean",
+ "beam100k1",
+ "demorecsys",
+ "lme-s-smoke",
+ "default",
+ "search",
+ ):
+ with self.assertRaises(SystemExit):
+ validate_brain_id(bad)
+
+ def test_metrics_ranking_and_latency(self):
+ from search.metrics import (
+ aggregate_query_metrics,
+ mrr,
+ ndcg_at_k,
+ percentile,
+ recall_at_k,
+ retrieve_latency_ms,
+ )
+
+ ranked = ["a", "b", "gold", "d"]
+ gold = {"gold"}
+ self.assertEqual(recall_at_k(ranked, gold, 2), 0.0)
+ self.assertEqual(recall_at_k(ranked, gold, 3), 1.0)
+ self.assertEqual(mrr(ranked, gold), 1.0 / 3)
+ self.assertGreater(ndcg_at_k(ranked, gold, 10), 0.0)
+ self.assertLess(ndcg_at_k(ranked, gold, 10), 1.0)
+ self.assertEqual(ndcg_at_k(["gold"], gold, 10), 1.0)
+ self.assertGreater(
+ ndcg_at_k(
+ ["exact", "sub"],
+ {"exact", "sub"},
+ 10,
+ grades={"exact": 1.0, "sub": 0.1},
+ ),
+ ndcg_at_k(
+ ["sub", "exact"],
+ {"exact", "sub"},
+ 10,
+ grades={"exact": 1.0, "sub": 0.1},
+ ),
+ )
+ wands_gold = {"exact", "partial"}
+ wands_grades = {"exact": 1.0, "partial": 0.5}
+ self.assertGreater(
+ ndcg_at_k(["exact", "partial"], wands_gold, 10, grades=wands_grades),
+ ndcg_at_k(["partial", "exact"], wands_gold, 10, grades=wands_grades),
+ )
+ self.assertAlmostEqual(
+ ndcg_at_k(["exact", "partial"], wands_gold, 10, grades=wands_grades),
+ ndcg_at_k(
+ ["exact", "partial"],
+ wands_gold,
+ 10,
+ grades={"exact": 2.0, "partial": 1.0},
+ ),
+ )
+ import math
+
+ from search.metrics import _dcg
+
+ self.assertAlmostEqual(
+ _dcg([1.0, 0.5], 2),
+ 1.0 / math.log2(2) + 0.5 / math.log2(3),
+ )
+ self.assertEqual(recall_at_k(["exact", "other"], wands_gold, 10), 0.5)
+ self.assertEqual(percentile([10.0, 20.0, 30.0], 50), 20.0)
+
+ retrieve, embed = retrieve_latency_ms(
+ {
+ "stages": [
+ {"stage": "embed.query", "wall_ms": 80.0},
+ {"stage": "search.retrieve", "wall_ms": 12.5},
+ ]
+ },
+ 200.0,
+ )
+ self.assertEqual(retrieve, 12.5)
+ self.assertEqual(embed, 80.0)
+ retrieve_fallback, embed_fallback = retrieve_latency_ms(
+ {"stages": [{"stage": "embed.query", "wall_ms": 40.0}]},
+ 90.0,
+ )
+ self.assertEqual(retrieve_fallback, 50.0)
+ self.assertEqual(embed_fallback, 40.0)
+
+ rows = [
+ {
+ "slice": "keyword",
+ "metrics": {
+ "recall@5": 1.0,
+ "recall@10": 1.0,
+ "recall@20": 1.0,
+ "ndcg@10": 1.0,
+ "ndcg@20": 1.0,
+ "ndcg": 1.0,
+ "mrr": 1.0,
+ },
+ "retrieve_ms": 10.0,
+ "embed_ms": 50.0,
+ "client_wall_ms": 70.0,
+ },
+ {
+ "slice": "paraphrase",
+ "metrics": {
+ "recall@5": 0.0,
+ "recall@10": 0.0,
+ "recall@20": 1.0,
+ "ndcg@10": 0.0,
+ "ndcg@20": 0.5,
+ "ndcg": 0.5,
+ "mrr": 0.0,
+ },
+ "retrieve_ms": 30.0,
+ "embed_ms": 50.0,
+ "client_wall_ms": 90.0,
+ },
+ ]
+ metrics = aggregate_query_metrics(rows)
+ self.assertEqual(metrics["recall@10"], 0.5)
+ self.assertEqual(metrics["ndcg@10"], 0.5)
+ self.assertEqual(metrics["ndcg@20"], 0.75)
+ self.assertEqual(metrics["by_slice"]["keyword"]["recall@10"], 1.0)
+ self.assertEqual(metrics["p50_retrieve_ms"], 20.0)
+
+ from search.metrics import ndcg_at_k
+
+ ranked_pool = ["exact", "irr", "sub", "other"]
+ grades_pool = {"exact": 1.0, "sub": 0.1, "irr": 0.0}
+ gold_pool = {"exact", "sub"}
+ manual = ndcg_at_k(ranked_pool, gold_pool, 20, grades=grades_pool)
+ self.assertGreater(manual, 0.0)
+ self.assertEqual(
+ ndcg_at_k(ranked_pool, gold_pool, 20, grades=grades_pool),
+ ndcg_at_k(ranked_pool, gold_pool, 4, grades=grades_pool),
+ )
+
+ def test_doc_marker_mapping(self):
+ from search.dataset import dataset_stats, load_records, map_doc_ids_to_chunks
+
+ root = Path(__file__).resolve().parents[1]
+ rows = load_records(root / "benchmarks" / "data" / "search_toy.jsonl")
+ stats = dataset_stats(rows)
+ self.assertGreaterEqual(stats["n_docs"], 6)
+ self.assertGreaterEqual(stats["n_queries"], 8)
+ self.assertIn("keyword", stats["slices"])
+ self.assertIn("paraphrase", stats["slices"])
+ mapping = map_doc_ids_to_chunks(
+ [{"doc_id": "license-alice"}],
+ [
+ {"id": "c1", "text": "DOCID license-alice. Alice completed her license."},
+ {"id": "c2", "text": "unrelated"},
+ ],
+ )
+ self.assertEqual(mapping["license-alice"], {"c1"})
+
+ def test_gold_matches_chunk_or_node_uuid(self):
+ from search.evaluate import gold_hit_grades, gold_hit_ids
+ from search.metrics import ndcg_at_k
+
+ query = {"gold_doc_ids": ["sku-1"], "gold_grades": {"sku-1": 1.0}}
+ mapped = {"sku-1": {"chunk-1"}}
+ gold = gold_hit_ids(query, mapped)
+ self.assertEqual(gold, {"sku-1"})
+ grades = gold_hit_grades(query, mapped)
+ self.assertEqual(ndcg_at_k(["sku-1"], gold, 10, grades=grades), 1.0)
+ from search.evaluate import canonicalize_hit_ids, invert_doc_chunks
+
+ ranked_chunk = canonicalize_hit_ids(["chunk-1"], invert_doc_chunks(mapped))
+ self.assertEqual(ndcg_at_k(ranked_chunk, gold, 10, grades=grades), 1.0)
+ self.assertEqual(ndcg_at_k(["other"], gold, 10, grades=grades), 0.0)
+
+ def test_wands_recall_gold_is_exact_and_partial(self):
+ from search.evaluate import gold_hit_ids
+ from search.metrics import recall_at_k
+
+ query = {
+ "gold_doc_ids": ["exact", "partial"],
+ "gold_grades": {"exact": 1.0, "partial": 0.5},
+ }
+ gold = gold_hit_ids(query, {})
+ self.assertEqual(gold, {"exact", "partial"})
+ self.assertEqual(recall_at_k(["exact", "irr"], gold, 10), 0.5)
+ self.assertEqual(recall_at_k(["exact", "partial"], gold, 10), 1.0)
+ unlabeled = gold_hit_ids(
+ {
+ "gold_doc_ids": ["exact"],
+ "gold_grades": {"exact": 1.0, "partial": 0.5},
+ },
+ {},
+ )
+ self.assertEqual(unlabeled, {"exact", "partial"})
+ irr_excluded = gold_hit_ids(
+ {
+ "gold_doc_ids": ["exact"],
+ "gold_grades": {"exact": 1.0, "irr": 0.0},
+ },
+ {},
+ )
+ self.assertEqual(irr_excluded, {"exact"})
+
+ def test_replay_fusion_compact_hub_drop(self):
+ from search.replay_fusion import replay
+
+ root = Path(__file__).resolve().parents[1]
+ graph_path = (
+ root / "benchmarks" / "runs" / "search-esci-slice-allcols" / "eval.json"
+ )
+ passages_path = (
+ root / "benchmarks" / "runs" / "search-esci-slice-passages" / "eval.json"
+ )
+ if not graph_path.exists() or not passages_path.exists():
+ self.skipTest("missing paired ESCI slice eval.json files")
+ result = replay(
+ json.loads(graph_path.read_text(encoding="utf-8")),
+ json.loads(passages_path.read_text(encoding="utf-8")),
+ )
+ self.assertEqual(result["n_queries"], 11)
+ self.assertAlmostEqual(result["hub_drop"]["recall@10"], 0.4758, places=3)
+ self.assertAlmostEqual(result["hub_drop"]["mrr"], 0.8485, places=3)
+
+ def test_unique_doc_counts_and_channel_lists(self):
+ from search.evaluate import channel_id_lists, unique_doc_counts
+
+ raw = ["chunk-1", "sku-1", "chunk-2", "hub:attr:red"]
+ mapping = {"chunk-1": "sku-1", "chunk-2": "sku-2"}
+ n_raw, n_canon = unique_doc_counts(raw, mapping, k=20)
+ self.assertEqual(n_raw, 4)
+ self.assertEqual(n_canon, 3)
+ lists = channel_id_lists(
+ {
+ "channel_lists": {
+ "dense": ["chunk-1"],
+ "bm25": ["chunk-2"],
+ "entities": ["sku-1"],
+ "communities": ["sku-3"],
+ }
+ },
+ [{"id": "chunk-1", "channel": "passages", "doc_id": "sku-1"}],
+ )
+ self.assertEqual(lists["dense_ids"], ["chunk-1"])
+ self.assertEqual(lists["entity_ids"], ["sku-1"])
+ self.assertEqual(lists["community_ids"], ["sku-3"])
+ self.assertEqual(lists["passage_ids"], ["chunk-1"])
+
+ def test_replay_collapse_and_gated_arms(self):
+ from search.replay_fusion import pick_offline_winner, replay_offline
+
+ gold = [f"d{i}" for i in range(1, 21)]
+ grades = {doc: 1.0 for doc in gold}
+ passages_hits = [
+ {"id": f"chunk-{item}", "channel": "passages", "doc_id": item}
+ for item in gold
+ ]
+ passages = {
+ "queries": [
+ {
+ "qid": "q1",
+ "gold_doc_ids": gold,
+ "gold_grades": grades,
+ "hit_ids": gold,
+ "dense_ids": [f"chunk-{item}" for item in gold],
+ "bm25_ids": [f"chunk-{item}" for item in gold],
+ "passage_ids": [f"chunk-{item}" for item in gold],
+ "hits": passages_hits,
+ "metrics": {
+ "ndcg@10": 1.0,
+ "recall@10": 0.5,
+ "recall@20": 1.0,
+ "mrr": 1.0,
+ },
+ }
+ ]
+ }
+ fused_raw = [f"chunk-{item}" for item in gold[:8]] + gold[:8] + gold[8:12]
+ graph = {
+ "queries": [
+ {
+ "qid": "q1",
+ "gold_doc_ids": gold,
+ "gold_grades": grades,
+ "hit_ids": gold[:12],
+ "entity_ids": gold[:5],
+ "community_ids": ["d21"],
+ "hits": [
+ {
+ "id": hid,
+ "channel": "passages" if hid.startswith("chunk-") else "entities",
+ "doc_id": hid.replace("chunk-", ""),
+ }
+ for hid in fused_raw
+ ],
+ "metrics": {
+ "ndcg@10": 0.6,
+ "recall@10": 0.4,
+ "recall@20": 0.6,
+ "mrr": 1.0,
+ },
+ }
+ ]
+ }
+ result = replay_offline(graph, passages)
+ self.assertEqual(result["n_queries"], 1)
+ self.assertGreaterEqual(
+ result["arms"]["collapse-rrf"]["recall@20"],
+ result["arms"]["passages"]["recall@20"] - 1e-9,
+ )
+ self.assertEqual(result["arms"]["passages"]["unique_docs@20"], 20.0)
+ self.assertEqual(result["arms"]["confirmation"]["unique_docs@20"], 20.0)
+ self.assertEqual(result["arms"]["expansion-n10"]["unique_docs@20"], 20.0)
+ winner = pick_offline_winner(
+ {
+ "passages": {
+ "ndcg@10": 0.758,
+ "recall@20": 0.847,
+ },
+ "collapse-rrf": {
+ "ndcg@10": 0.640,
+ "recall@20": 0.847,
+ },
+ "expansion-n10": {
+ "ndcg@10": 0.758,
+ "recall@20": 0.847,
+ },
+ }
+ )
+ self.assertEqual(winner, "expansion-n10")
+ self.assertEqual(
+ pick_offline_winner(
+ {
+ "passages": {"ndcg@10": 0.758, "recall@20": 0.847},
+ "collapse-rrf": {"ndcg@10": 0.640, "recall@20": 0.847},
+ "expansion-n10": {"ndcg@10": 0.758, "recall@20": 0.815},
+ }
+ ),
+ "G08",
+ )
+ self.assertEqual(
+ pick_offline_winner(
+ {
+ "passages": {"ndcg@10": 0.758, "recall@20": 0.847},
+ "collapse-rrf": {"ndcg@10": 0.640, "recall@20": 0.847},
+ }
+ ),
+ "G08",
+ )
+
+ def test_score_search_dumps_channel_and_doc_id(self):
+ from search.client import TimedResult
+ from search.evaluate import score_search_result
+ from search.metrics import ndcg_at_k, recall_at_k
+
+ result = TimedResult(
+ data={
+ "hits": [
+ {"id": "chunk-1", "channel": "passages", "node_id": "sku-1"},
+ {"id": "sku-1", "channel": "entities", "node_id": "sku-1"},
+ {"id": "hub:attr:red", "channel": "communities"},
+ ],
+ "stage_timings": {
+ "stages": [{"stage": "search.retrieve", "wall_ms": 11.0}]
+ },
+ },
+ latency_ms=40.0,
+ )
+ gold = {"sku-1"}
+ grades = {"sku-1": 1.0}
+ scored = score_search_result(
+ result,
+ gold=gold,
+ ks=(5, 10, 20, 50),
+ grades=grades,
+ chunk_to_doc={"chunk-1": "sku-1"},
+ )
+ self.assertEqual(scored["hit_ids"], ["sku-1", "hub:attr:red"])
+ self.assertEqual(
+ scored["hits"],
+ [
+ {
+ "id": "chunk-1",
+ "channel": "passages",
+ "doc_id": "sku-1",
+ "node_id": "sku-1",
+ },
+ {
+ "id": "sku-1",
+ "channel": "entities",
+ "doc_id": "sku-1",
+ "node_id": "sku-1",
+ },
+ {
+ "id": "hub:attr:red",
+ "channel": "communities",
+ "doc_id": "hub:attr:red",
+ "node_id": None,
+ },
+ ],
+ )
+ self.assertEqual(
+ scored["metrics"]["recall@10"],
+ recall_at_k(scored["hit_ids"], gold, 10),
+ )
+ self.assertEqual(
+ scored["metrics"]["ndcg@10"],
+ ndcg_at_k(scored["hit_ids"], gold, 10, grades=grades),
+ )
+ self.assertEqual(
+ [row["doc_id"] for row in scored["hits"][:1]],
+ scored["hit_ids"][:1],
+ )
+ self.assertEqual(scored["n_unique_docs_raw"], 3)
+ self.assertEqual(scored["n_unique_docs_canonical"], 2)
+ self.assertEqual(scored["unique_docs_k"], 50)
+ self.assertEqual(scored["n_unique_docs_retrieve_canonical"], 2)
+ self.assertIn("ndcg@50", scored["metrics"])
+ self.assertEqual(scored["entity_ids"], ["sku-1"])
+ self.assertEqual(scored["community_ids"], ["hub:attr:red"])
+ self.assertEqual(scored["passage_ids"], ["chunk-1"])
+
+ def test_rank_pool_filters_and_keeps_irrelevant_grades(self):
+ from search.client import TimedResult
+ from search.evaluate import (
+ candidate_pool_grades,
+ candidate_pool_ids,
+ filter_ranked_to_pool,
+ score_search_result,
+ )
+ from search.metrics import ndcg_at_k
+
+ query = {
+ "gold_doc_ids": ["p-e"],
+ "gold_grades": {"p-e": 1.0, "p-s": 0.1},
+ "candidate_doc_ids": ["p-e", "p-s", "p-i"],
+ "candidate_grades": {"p-e": 1.0, "p-s": 0.1, "p-i": 0.0},
+ }
+ self.assertEqual(candidate_pool_ids(query), ["p-e", "p-s", "p-i"])
+ self.assertEqual(candidate_pool_grades(query)["p-i"], 0.0)
+ self.assertEqual(
+ filter_ranked_to_pool(["p-x", "p-e", "p-i", "p-y"], query["candidate_doc_ids"]),
+ ["p-e", "p-i"],
+ )
+ result = TimedResult(
+ data={
+ "hits": [
+ {"id": "outside", "channel": "passages"},
+ {"id": "p-e", "channel": "passages"},
+ {"id": "p-i", "channel": "passages"},
+ ],
+ "stage_timings": {
+ "stages": [{"stage": "search.retrieve", "wall_ms": 9.0}]
+ },
+ },
+ latency_ms=12.0,
+ )
+ scored = score_search_result(
+ result,
+ gold={"p-e", "p-s"},
+ ks=(5, 10, 20),
+ grades=candidate_pool_grades(query),
+ pool_ids=candidate_pool_ids(query),
+ )
+ self.assertEqual(scored["hit_ids"], ["p-e", "p-i"])
+ expected = ndcg_at_k(
+ ["p-e", "p-i"],
+ {"p-e", "p-s"},
+ 20,
+ grades=candidate_pool_grades(query),
+ )
+ self.assertEqual(scored["metrics"]["ndcg@20"], expected)
+
+ def test_rank_pool_ce_uses_injected_predict(self):
+ import sys
+
+ root = Path(__file__).resolve().parents[1]
+ plugin = root / "plugins" / "search-rerank"
+ if str(plugin) not in sys.path:
+ sys.path.insert(0, str(plugin))
+ import rerank as rerank_mod
+
+ from search.rank_pool import run_ce_on_pool
+
+ def predict(pairs):
+ scores = []
+ for _, text in pairs:
+ if "gold" in text:
+ scores.append(2.0)
+ elif "partial" in text:
+ scores.append(1.0)
+ else:
+ scores.append(0.0)
+ return scores
+
+ rerank_mod.set_predict(predict)
+ try:
+ rows = [
+ {
+ "type": "doc",
+ "doc_id": "irr",
+ "text": "DOCID irr. Title: other",
+ },
+ {
+ "type": "doc",
+ "doc_id": "gold",
+ "text": "DOCID gold. Title: gold kettle",
+ },
+ {
+ "type": "doc",
+ "doc_id": "sub",
+ "text": "DOCID sub. Title: partial kettle",
+ },
+ {
+ "type": "query",
+ "qid": "esci-1",
+ "query": "kettle",
+ "gold_doc_ids": ["gold", "sub"],
+ "gold_grades": {"gold": 1.0, "sub": 0.1},
+ "candidate_doc_ids": ["irr", "gold", "sub"],
+ "candidate_grades": {"irr": 0.0, "gold": 1.0, "sub": 0.1},
+ "slice": "esci-us",
+ },
+ ]
+ result = run_ce_on_pool(rows, dataset_name="toy-pool.jsonl")
+ finally:
+ rerank_mod.set_predict(None)
+ self.assertEqual(result["status"], "ok")
+ self.assertTrue(result["rank_pool"])
+ self.assertTrue(result["rank_pool_ce"])
+ self.assertEqual(result["queries"][0]["hit_ids"][0], "gold")
+ self.assertGreater(result["metrics"]["ndcg@20"], 0.0)
+
+ def test_ce_on_retrieved_reorders_hits_only(self):
+ import sys
+
+ root = Path(__file__).resolve().parents[1]
+ plugin = root / "plugins" / "search-rerank"
+ if str(plugin) not in sys.path:
+ sys.path.insert(0, str(plugin))
+ import rerank as rerank_mod
+
+ from search.rerank_retrieved import run_ce_on_retrieved
+
+ def predict(pairs):
+ scores = []
+ for _, text in pairs:
+ if "gold" in text:
+ scores.append(2.0)
+ elif "partial" in text:
+ scores.append(1.0)
+ else:
+ scores.append(0.0)
+ return scores
+
+ rerank_mod.set_predict(predict)
+ try:
+ rows = [
+ {
+ "type": "doc",
+ "doc_id": "irr",
+ "text": "DOCID irr. Title: other",
+ },
+ {
+ "type": "doc",
+ "doc_id": "gold",
+ "text": "DOCID gold. Title: gold kettle",
+ },
+ {
+ "type": "doc",
+ "doc_id": "sub",
+ "text": "DOCID sub. Title: partial kettle",
+ },
+ {
+ "type": "query",
+ "qid": "esci-1",
+ "query": "kettle",
+ "gold_doc_ids": ["gold", "sub"],
+ "gold_grades": {"gold": 1.0, "sub": 0.1},
+ "slice": "esci-us",
+ },
+ ]
+ eval_result = {
+ "k": 50,
+ "ks": [5, 10, 20, 50],
+ "rank_pool": False,
+ "fusion": "rrf",
+ "channels": ["passages"],
+ "n_docs": 3,
+ "queries": [
+ {
+ "qid": "esci-1",
+ "query": "kettle",
+ "slice": "esci-us",
+ "gold_doc_ids": ["gold", "sub"],
+ "gold_grades": {"gold": 1.0, "sub": 0.1},
+ "hit_ids": ["irr", "sub", "gold"],
+ "metrics": {},
+ "retrieve_ms": 12.0,
+ "pool_coverage": None,
+ }
+ ],
+ }
+ result = run_ce_on_retrieved(
+ eval_result,
+ rows,
+ dataset_name="toy-retrieved.jsonl",
+ )
+ finally:
+ rerank_mod.set_predict(None)
+ self.assertEqual(result["status"], "ok")
+ self.assertFalse(result["rank_pool_ce"])
+ self.assertEqual(result["rerank"], "harness:cross-encoder")
+ self.assertEqual(result["queries"][0]["hit_ids"][0], "gold")
+ self.assertEqual(set(result["queries"][0]["hit_ids"]), {"irr", "sub", "gold"})
+ self.assertGreater(result["metrics"]["ndcg@10"], 0.0)
+
+ def test_electronics_marker_for_esci72_style_titles(self):
+ from search.miss import looks_like_electronics
+
+ self.assertTrue(looks_like_electronics("Apple iPad Air 2, 16 GB"))
+ self.assertFalse(looks_like_electronics("Price Tags Without String"))
+
+ def test_miss_strata_classifies_head_tail_and_total_miss(self):
+ from search.miss_strata import classify_eval, rewrite_query
+
+ self.assertIsNone(rewrite_query("esci-72", "$100 things that are not electronics"))
+ self.assertIsNone(rewrite_query("esci-177", "p_num: integer not null i_num: integer"))
+ rewritten = rewrite_query(
+ "esci-113",
+ "&#34;tortillas without interesterified soybean oil”",
+ )
+ self.assertIsNotNone(rewritten)
+ self.assertNotIn("&#34;", rewritten)
+
+ eval_result = {
+ "k": 50,
+ "queries": [
+ {
+ "qid": "q-head",
+ "query": "red mug",
+ "hit_ids": ["g1", "x"] + [f"n{i}" for i in range(48)],
+ "gold_doc_ids": ["g1", "g2"],
+ "gold_grades": {"g1": 1.0, "g2": 0.1},
+ "metrics": {"recall@10": 0.5, "recall@50": 0.5},
+ },
+ {
+ "qid": "q-tail",
+ "query": "blue mug",
+ "hit_ids": ["n0"] * 10 + ["g2"] + [f"n{i}" for i in range(39)],
+ "gold_doc_ids": ["g2"],
+ "gold_grades": {"g2": 1.0},
+ "metrics": {"recall@10": 0.0, "recall@50": 1.0},
+ },
+ {
+ "qid": "esci-113",
+ "query": "&#34;tortillas without interesterified soybean oil”",
+ "hit_ids": [f"n{i}" for i in range(50)],
+ "gold_doc_ids": ["g3"],
+ "gold_grades": {"g3": 1.0},
+ "metrics": {"recall@10": 0.0, "recall@50": 0.0},
+ },
+ ],
+ }
+ rows = [
+ {"type": "doc", "doc_id": "g1", "text": "Title: red mug"},
+ {"type": "doc", "doc_id": "g2", "text": "Title: blue mug"},
+ {"type": "doc", "doc_id": "g3", "text": "Title: tortillas"},
+ {
+ "type": "query",
+ "qid": "q-head",
+ "query": "red mug",
+ "gold_doc_ids": ["g1", "g2"],
+ "gold_grades": {"g1": 1.0, "g2": 0.1},
+ },
+ {
+ "type": "query",
+ "qid": "q-tail",
+ "query": "blue mug",
+ "gold_doc_ids": ["g2"],
+ "gold_grades": {"g2": 1.0},
+ },
+ {
+ "type": "query",
+ "qid": "esci-113",
+ "query": "&#34;tortillas without interesterified soybean oil”",
+ "gold_doc_ids": ["g3"],
+ "gold_grades": {"g3": 1.0},
+ },
+ ]
+ taxonomy = classify_eval(eval_result, rows, k=50)
+ self.assertEqual(taxonomy["n_queries"], 3)
+ self.assertEqual(taxonomy["n_gold"], 4)
+ self.assertEqual(taxonomy["stratum_counts"]["head-ok"], 1)
+ self.assertEqual(taxonomy["stratum_counts"]["rank-too-low"], 1)
+ self.assertEqual(taxonomy["stratum_counts"]["total-miss"], 1)
+ self.assertTrue(taxonomy["run_query_side"])
+ self.assertEqual(taxonomy["rewritable_qids"], ["esci-113"])
+
+ def test_spell_normalize_nfkc_and_accents(self):
+ from search.miss_strata import normalize_spelling, write_spell_jsonl
+
+ self.assertEqual(normalize_spelling("niños!!!"), "ninos")
+ self.assertEqual(normalize_spelling(" sofa, azul. "), "sofa azul")
+ self.assertEqual(normalize_spelling("caf\u00e9"), "cafe")
+ with tempfile.TemporaryDirectory() as tmp:
+ dest = Path(tmp) / "spell.jsonl"
+ summary = write_spell_jsonl(
+ [
+ {"type": "doc", "doc_id": "d1", "text": "sofa"},
+ {"type": "query", "qid": "q1", "query": "sofá azul!!"},
+ ],
+ dest,
+ )
+ self.assertEqual(summary["n_queries_changed"], 1)
+ rows = dest.read_text(encoding="utf-8").strip().splitlines()
+ query = json.loads(rows[1])
+ self.assertEqual(query["query"], "sofa azul")
+ self.assertEqual(query["query_original"], "sofá azul!!")
+
+ def test_doc_meta_keys_copies_generic_catalog_fields(self):
+ from search.evaluate import doc_meta_keys
+
+ self.assertEqual(
+ doc_meta_keys(
+ {
+ "doc_id": "it-bollitore",
+ "brand": "CasaLuce",
+ "color": "argento",
+ "locale": "it",
+ "title": "ignored",
+ }
+ ),
+ {"brand": "CasaLuce", "color": "argento", "locale": "it"},
+ )
+ self.assertIsNone(doc_meta_keys({"doc_id": "x", "text": "only text"}))
+
+ def test_dense_holdout_excludes_eval_qids(self):
+ from search.finetune_esci_ce import held_out_query_ids
+
+ holdout = held_out_query_ids(
+ Path(__file__).resolve().parents[1]
+ / "benchmarks"
+ / "data"
+ / "search_esci_74.jsonl"
+ )
+ self.assertIn("72", holdout)
+ self.assertIn("esci-72", holdout)
+ self.assertGreater(len(holdout), 74)
+
+ def test_local_dense_scores_injected_ranks(self):
+ from search.local_dense import evaluate_dense
+
+ queries = [
+ {
+ "qid": "q1",
+ "query": "red mug",
+ "slice": "toy",
+ "gold_doc_ids": ["gold"],
+ "gold_grades": {"gold": 1.0, "sub": 0.1},
+ }
+ ]
+ ranked = {"q1": ["gold", "sub", "irr"]}
+ metrics, per_query = evaluate_dense(
+ ranked, queries, ks=(5, 10, 20, 50), encode_ms=10.0
+ )
+ self.assertEqual(per_query[0]["hit_ids"][0], "gold")
+ self.assertEqual(metrics["recall@10"], 1.0)
+ self.assertGreater(metrics["ndcg@10"], 0.0)
+
+ def test_exhaustive_ce_ranks_exact_above_irrelevant(self):
+ from search.rank_corpus import PROTOCOL, run_exhaustive_ce
+
+ rows = [
+ {"type": "doc", "doc_id": "irr", "text": "DOCID irr. Title: other"},
+ {"type": "doc", "doc_id": "exact", "text": "DOCID exact. Title: gold kettle"},
+ {"type": "doc", "doc_id": "maybe", "text": "DOCID maybe. Title: similar kettle"},
+ {
+ "type": "query",
+ "qid": "esci-toy",
+ "query": "kettle",
+ "gold_doc_ids": ["exact"],
+ "gold_grades": {"exact": 1.0, "maybe": 0.1},
+ "slice": "toy",
+ },
+ ]
+
+ def predict(pairs):
+ out = []
+ for _, text in pairs:
+ if "gold kettle" in text:
+ out.append([0.95, 0.03, 0.01, 0.01])
+ elif "similar" in text:
+ out.append([0.05, 0.80, 0.10, 0.05])
+ else:
+ out.append([0.01, 0.02, 0.07, 0.90])
+ return out
+
+ result = run_exhaustive_ce(
+ rows,
+ predict=predict,
+ model_name="toy-4class",
+ dataset_name="toy-corpus.jsonl",
+ k=3,
+ ks=(5, 10, 20, 50),
+ )
+ self.assertEqual(result["status"], "ok")
+ self.assertEqual(result["protocol"], PROTOCOL)
+ self.assertFalse(result["rank_pool"])
+ self.assertEqual(result["queries"][0]["hit_ids"][0], "exact")
+ self.assertLess(result["queries"][0]["hit_ids"].index("exact"), result["queries"][0]["hit_ids"].index("irr"))
+ self.assertGreater(result["metrics"]["ndcg@10"], 0.0)
+
+ def test_list_overlap_counts_sidecar_unique_golds(self):
+ from search.list_overlap import run_union, summarize_overlap, unique_golds
+
+ gold = {"g1", "g2", "g3"}
+ extra = unique_golds(
+ ["g2", "noise", "g3"],
+ ["g1", "g2", "other"],
+ gold,
+ k=50,
+ )
+ self.assertEqual(extra, ["g3"])
+ passages = {
+ "run_id": "passages",
+ "queries": [
+ {
+ "qid": "q1",
+ "hit_ids": ["g1", "g2", "p1"],
+ "gold_doc_ids": ["g1", "g2", "g3"],
+ "gold_grades": {"g1": 1.0, "g2": 1.0, "g3": 0.1},
+ }
+ ],
+ }
+ bge = {
+ "run_id": "bge",
+ "queries": [{"qid": "q1", "hit_ids": ["g3", "g1", "b1"]}],
+ }
+ summary = summarize_overlap(passages, {"bge": bge}, k=50)
+ self.assertEqual(summary["runs"]["bge"]["unique_gold_hits"], 1)
+ rows = [
+ {"type": "doc", "doc_id": "g1", "text": "g1"},
+ {"type": "doc", "doc_id": "g2", "text": "g2"},
+ {"type": "doc", "doc_id": "g3", "text": "g3"},
+ {
+ "type": "query",
+ "qid": "q1",
+ "query": "mug",
+ "gold_doc_ids": ["g1", "g2", "g3"],
+ "gold_grades": {"g1": 1.0, "g2": 1.0, "g3": 0.1},
+ },
+ ]
+ union = run_union(
+ [passages, bge],
+ rows,
+ dataset_name="toy.jsonl",
+ k=50,
+ ks=(5, 10, 20, 50),
+ run_names=["passages", "bge"],
+ )
+ self.assertEqual(union["channels"], ["harness-union"])
+ self.assertIn("g3", union["queries"][0]["hit_ids"][:10])
+ self.assertGreaterEqual(union["metrics"]["recall@10"], 1.0)
+
+ def test_cascade_freezes_head_and_injects_tail_golds(self):
+ from search.list_overlap import CASCADE_PROTOCOL, cascade_frozen_head, run_cascade
+
+ passages = [f"h{index}" for index in range(10)] + ["tail_gold", "irr"]
+ gold = {"h0", "tail_gold", "extra"}
+ ranked = cascade_frozen_head(
+ passages,
+ [["extra", "noise", "h0"]],
+ gold,
+ head_k=10,
+ k=50,
+ )
+ self.assertEqual(ranked[:10], passages[:10])
+ self.assertIn("extra", ranked[10:])
+ self.assertNotIn("extra", ranked[:10])
+ self.assertIn("tail_gold", ranked)
+ rows = [
+ {"type": "doc", "doc_id": doc_id, "text": doc_id}
+ for doc_id in [*passages, "extra"]
+ ]
+ rows.append(
+ {
+ "type": "query",
+ "qid": "q1",
+ "query": "mug",
+ "gold_doc_ids": ["h0", "tail_gold", "extra"],
+ "gold_grades": {"h0": 1.0, "tail_gold": 1.0, "extra": 0.1},
+ }
+ )
+ result = run_cascade(
+ {"run_id": "passages", "queries": [{"qid": "q1", "hit_ids": passages}]},
+ [{"run_id": "bge", "queries": [{"qid": "q1", "hit_ids": ["extra", "noise"]}]}],
+ rows,
+ dataset_name="toy.jsonl",
+ k=50,
+ head_k=10,
+ ks=(5, 10, 20, 50),
+ run_names=["passages", "bge"],
+ )
+ self.assertEqual(result["protocol"], CASCADE_PROTOCOL)
+ self.assertEqual(result["channels"], ["harness-cascade"])
+ self.assertEqual(result["queries"][0]["hit_ids"][:10], passages[:10])
+ self.assertIn("extra", result["queries"][0]["hit_ids"][10:])
+ self.assertEqual(result["metrics"]["recall@10"], 1.0 / 3.0)
+
+ def test_frozen_head_merge_matches_harness_cascade(self):
+ from src.core.search.hybrid import frozen_head_merge
+ from search.list_overlap import cascade_frozen_head
+
+ passages = [f"h{index}" for index in range(10)] + ["tail_gold", "irr"]
+ sidecars = [["extra", "noise", "h0"]]
+ gold = {"h0", "tail_gold", "extra"}
+ ranked = frozen_head_merge(
+ passages,
+ sidecars,
+ head_k=10,
+ k=50,
+ prefer_ids=gold,
+ )
+ self.assertEqual(
+ ranked,
+ cascade_frozen_head(passages, sidecars, gold, head_k=10, k=50),
+ )
+ live = frozen_head_merge(passages, sidecars, head_k=10, k=50)
+ self.assertEqual(live[:10], passages[:10])
+ self.assertIn("extra", live[10:])
+ self.assertIn("noise", live[10:])
+
+ def test_cascade_replay_esci74_stored_lists(self):
+ from search.dataset import load_records
+ from search.list_overlap import load_eval_run, run_cascade
+
+ root = Path(__file__).resolve().parents[1]
+ runs = root / "benchmarks" / "runs"
+ passages_path = runs / "search-esci-74-passages-k50" / "eval.json"
+ bge_path = runs / "search-esci-74-bge-base-k50" / "eval.json"
+ colbert_path = runs / "search-esci-74-colbert-k50" / "eval.json"
+ dataset = root / "benchmarks" / "data" / "search_esci_74.jsonl"
+ if not all(
+ path.exists()
+ for path in (passages_path, bge_path, colbert_path, dataset)
+ ):
+ self.skipTest("stored US ESCI n=74 evals not present")
+ passages = load_eval_run(passages_path.parent)
+ bge = load_eval_run(bge_path.parent)
+ colbert = load_eval_run(colbert_path.parent)
+ rows = load_records(dataset)
+ result = run_cascade(
+ passages,
+ [bge, colbert],
+ rows,
+ dataset_name="search_esci_74.jsonl",
+ k=50,
+ head_k=10,
+ ks=(5, 10, 20, 50),
+ run_names=["passages", "bge", "colbert"],
+ )
+ metrics = result["metrics"]
+ self.assertAlmostEqual(metrics["ndcg@10"], 0.500, places=3)
+ self.assertAlmostEqual(metrics["recall@10"], 0.379, places=3)
+ self.assertAlmostEqual(metrics["recall@50"], 0.889, places=3)
+
+ def test_ltr_head_promotes_title_overlap_on_held_out_query(self):
+ from search.ltr_head import (
+ example_from_eval_row,
+ features_for_doc,
+ fit_pairwise,
+ rerank_ids,
+ run_ltr_head,
+ )
+
+ feats = features_for_doc(
+ "navy velvet sofa",
+ "hit",
+ rrf_ids=["miss", "hit"],
+ bm25_ids=["hit"],
+ dense_ids=["miss"],
+ doc={"title": "Navy velvet sofa two seats", "brand": "AtelierNord"},
+ )
+ self.assertGreater(feats[3], 0.5)
+ self.assertEqual(feats[4], 0.0)
+ train_row = {
+ "qid": "q-train",
+ "query": "navy velvet sofa",
+ "hit_ids": ["irr", "gold"],
+ "bm25_ids": ["gold"],
+ "dense_ids": ["irr"],
+ "hits": [
+ {"id": "irr", "doc_id": "irr"},
+ {"id": "gold", "doc_id": "gold"},
+ ],
+ "gold_doc_ids": ["gold"],
+ "gold_grades": {"gold": 1.0},
+ }
+ test_row = {
+ "qid": "q-test",
+ "query": "navy velvet sofa",
+ "hit_ids": ["irr", "gold"],
+ "bm25_ids": ["gold"],
+ "dense_ids": ["irr"],
+ "hits": [
+ {"id": "irr", "doc_id": "irr"},
+ {"id": "gold", "doc_id": "gold"},
+ ],
+ "gold_doc_ids": ["gold"],
+ "gold_grades": {"gold": 1.0},
+ }
+ docs = {
+ "gold": {"doc_id": "gold", "title": "Navy velvet sofa"},
+ "irr": {"doc_id": "irr", "title": "plastic kettle lid"},
+ }
+ train_ex = example_from_eval_row(train_row, docs, k=10)
+ test_ex = example_from_eval_row(test_row, docs, k=10)
+ self.assertIsNotNone(train_ex)
+ self.assertIsNotNone(test_ex)
+ weights = fit_pairwise([train_ex], epochs=20, seed=0)
+ ranked = rerank_ids(test_ex, weights)
+ self.assertEqual(ranked[0], "gold")
+ result = run_ltr_head(
+ {"run_id": "toy", "queries": [train_row, test_row]},
+ [
+ {"type": "doc", "doc_id": "gold", "title": "Navy velvet sofa", "text": "Title: Navy velvet sofa"},
+ {"type": "doc", "doc_id": "irr", "title": "plastic kettle lid", "text": "Title: plastic kettle lid"},
+ {
+ "type": "query",
+ "qid": "q-train",
+ "query": "navy velvet sofa",
+ "gold_doc_ids": ["gold"],
+ "gold_grades": {"gold": 1.0},
+ },
+ {
+ "type": "query",
+ "qid": "q-test",
+ "query": "navy velvet sofa",
+ "gold_doc_ids": ["gold"],
+ "gold_grades": {"gold": 1.0},
+ },
+ ],
+ dataset_name="toy.jsonl",
+ k=10,
+ ks=(5, 10),
+ n_folds=2,
+ )
+ self.assertEqual(result["protocol"], "ltr-head-cv")
+ self.assertGreaterEqual(result["metrics"]["recall@10"], 1.0)
+
+ def test_ltr_pair_policy_skips_unlabeled_for_other_query_neg(self):
+ from search.ltr_head import (
+ PAIR_OTHER_QUERY_NEG,
+ PAIR_UNLABELED_ZERO,
+ collect_pairs,
+ example_from_eval_row,
+ features_for_doc,
+ other_query_gold_ids,
+ )
+
+ row_a = {
+ "qid": "q-a",
+ "query": "navy sofa",
+ "hit_ids": ["gold-a", "gold-b", "unlabeled"],
+ "hits": [
+ {"id": "gold-a", "doc_id": "gold-a"},
+ {"id": "gold-b", "doc_id": "gold-b"},
+ {"id": "unlabeled", "doc_id": "unlabeled"},
+ ],
+ "gold_doc_ids": ["gold-a"],
+ "gold_grades": {"gold-a": 1.0},
+ }
+ row_b = {
+ "qid": "q-b",
+ "query": "steel kettle",
+ "hit_ids": ["gold-b", "gold-a", "unlabeled"],
+ "hits": [
+ {"id": "gold-b", "doc_id": "gold-b"},
+ {"id": "gold-a", "doc_id": "gold-a"},
+ {"id": "unlabeled", "doc_id": "unlabeled"},
+ ],
+ "gold_doc_ids": ["gold-b"],
+ "gold_grades": {"gold-b": 1.0},
+ }
+ docs = {
+ "gold-a": {"doc_id": "gold-a", "title": "Navy sofa"},
+ "gold-b": {"doc_id": "gold-b", "title": "Steel kettle"},
+ "unlabeled": {"doc_id": "unlabeled", "title": "random cable"},
+ }
+ ex_a = example_from_eval_row(row_a, docs, k=10)
+ ex_b = example_from_eval_row(row_b, docs, k=10)
+ other = other_query_gold_ids([ex_a, ex_b], "q-a")
+ self.assertEqual(other, {"gold-b"})
+ zero_pairs = collect_pairs(ex_a, pair_policy=PAIR_UNLABELED_ZERO)
+ neg_pairs = collect_pairs(
+ ex_a, pair_policy=PAIR_OTHER_QUERY_NEG, other_gold=other
+ )
+ zero_docs = {(ex_a["ids"][hi], ex_a["ids"][lo]) for hi, lo in zero_pairs}
+ neg_docs = {(ex_a["ids"][hi], ex_a["ids"][lo]) for hi, lo in neg_pairs}
+ self.assertIn(("gold-a", "unlabeled"), zero_docs)
+ self.assertNotIn(("gold-a", "unlabeled"), neg_docs)
+ self.assertIn(("gold-a", "gold-b"), neg_docs)
+ feats = features_for_doc(
+ "navy sofa",
+ "gold-a",
+ rrf_ids=["gold-a"],
+ bm25_ids=["gold-a"],
+ dense_ids=["gold-a"],
+ doc={"title": "Navy sofa"},
+ ce_gain=0.42,
+ )
+ self.assertEqual(len(feats), 7)
+ self.assertAlmostEqual(float(feats[-1]), 0.42)
+ from search.ltr_head import rank_train_group
+
+ grouped = rank_train_group(
+ ex_a, pair_policy=PAIR_OTHER_QUERY_NEG, other_gold=other
+ )
+ self.assertIsNotNone(grouped)
+ _, labels = grouped
+ self.assertEqual(len(labels), 2)
+
+ def test_ltr_lightgbm_promotes_high_ce_gain_on_held_out_query(self):
+ from search.ltr_head import (
+ PAIR_OTHER_QUERY_NEG,
+ example_from_eval_row,
+ fit_lightgbm,
+ rerank_ids_model,
+ )
+
+ n_queries = 8
+ docs: dict[str, dict[str, str]] = {}
+ rows: list[dict] = []
+ for index in range(n_queries):
+ gold = f"gold-{index}"
+ other = f"gold-{(index + 1) % n_queries}"
+ miss = f"miss-{index}"
+ docs[gold] = {"doc_id": gold, "title": f"Navy sofa {index}"}
+ docs[miss] = {"doc_id": miss, "title": f"plastic lid {index}"}
+ rows.append(
+ {
+ "qid": f"q-{index}",
+ "query": "navy sofa",
+ "hit_ids": [miss, other, gold],
+ "bm25_ids": [gold],
+ "dense_ids": [miss],
+ "hits": [
+ {"id": miss, "doc_id": miss},
+ {"id": other, "doc_id": other},
+ {"id": gold, "doc_id": gold},
+ ],
+ "gold_doc_ids": [gold],
+ "gold_grades": {gold: 1.0},
+ "ce": {gold: 0.95, other: 0.05, miss: 0.01},
+ }
+ )
+ examples = [
+ example_from_eval_row(
+ row, docs, k=10, ce_scores=row["ce"]
+ )
+ for row in rows
+ ]
+ self.assertTrue(all(item is not None for item in examples))
+ model = fit_lightgbm(
+ examples[:-1],
+ pair_policy=PAIR_OTHER_QUERY_NEG,
+ min_data_in_leaf=1,
+ )
+ ranked = rerank_ids_model(examples[-1], model)
+ self.assertEqual(ranked[0], "gold-7")
+ self.assertGreater(float(sum(model.feature_importances_)), 0.0)
+
+ def test_ltr_apply_trains_on_other_run_not_eval_qid(self):
+ from search.ltr_head import PROTOCOL_APPLY, run_ltr_head
+
+ train_docs = [
+ {"type": "doc", "doc_id": "gold-t", "title": "Navy velvet sofa", "text": "Title: Navy velvet sofa"},
+ {"type": "doc", "doc_id": "irr-t", "title": "plastic lid", "text": "Title: plastic lid"},
+ ]
+ eval_docs = [
+ {"type": "doc", "doc_id": "gold-e", "title": "Navy velvet sofa", "text": "Title: Navy velvet sofa"},
+ {"type": "doc", "doc_id": "irr-e", "title": "plastic lid", "text": "Title: plastic lid"},
+ ]
+ train_queries = [
+ {
+ "type": "query",
+ "qid": "q-train",
+ "query": "navy velvet sofa",
+ "gold_doc_ids": ["gold-t"],
+ "gold_grades": {"gold-t": 1.0},
+ }
+ ]
+ eval_queries = [
+ {
+ "type": "query",
+ "qid": "q-eval",
+ "query": "navy velvet sofa",
+ "gold_doc_ids": ["gold-e"],
+ "gold_grades": {"gold-e": 1.0},
+ }
+ ]
+ train_row = {
+ "qid": "q-train",
+ "query": "navy velvet sofa",
+ "hit_ids": ["irr-t", "gold-t"],
+ "bm25_ids": ["gold-t"],
+ "dense_ids": ["irr-t"],
+ "hits": [
+ {"id": "irr-t", "doc_id": "irr-t"},
+ {"id": "gold-t", "doc_id": "gold-t"},
+ ],
+ "gold_doc_ids": ["gold-t"],
+ "gold_grades": {"gold-t": 1.0},
+ }
+ eval_row = {
+ "qid": "q-eval",
+ "query": "navy velvet sofa",
+ "hit_ids": ["irr-e", "gold-e"],
+ "bm25_ids": ["gold-e"],
+ "dense_ids": ["irr-e"],
+ "hits": [
+ {"id": "irr-e", "doc_id": "irr-e"},
+ {"id": "gold-e", "doc_id": "gold-e"},
+ ],
+ "gold_doc_ids": ["gold-e"],
+ "gold_grades": {"gold-e": 1.0},
+ }
+ result = run_ltr_head(
+ {"run_id": "toy-eval", "queries": [eval_row]},
+ eval_docs + eval_queries,
+ dataset_name="toy-eval.jsonl",
+ k=10,
+ ks=(5, 10),
+ n_folds=2,
+ train_eval_result={"run_id": "toy-train", "queries": [train_row]},
+ train_rows=train_docs + train_queries,
+ train_source_run="toy-train",
+ )
+ self.assertEqual(result["protocol"], PROTOCOL_APPLY)
+ self.assertEqual(result["ltr_train_run"], "toy-train")
+ self.assertEqual(result["ltr_n_train_queries"], 1)
+ self.assertEqual(result["queries"][0]["qid"], "q-eval")
+ self.assertEqual(result["queries"][0]["hit_ids"][0], "gold-e")
+ self.assertGreaterEqual(result["metrics"]["recall@10"], 1.0)
+
+ def test_ltr_apply_lightgbm_trains_on_other_run_not_eval_qid(self):
+ from search.ltr_head import HEAD_LIGHTGBM, PROTOCOL_APPLY, run_ltr_head
+
+ train_docs: list[dict] = []
+ train_queries: list[dict] = []
+ train_eval_rows: list[dict] = []
+ for index in range(8):
+ gold = f"gold-t-{index}"
+ miss = f"miss-t-{index}"
+ train_docs.extend(
+ [
+ {
+ "type": "doc",
+ "doc_id": gold,
+ "title": f"Navy velvet sofa {index}",
+ "text": f"Title: Navy velvet sofa {index}",
+ },
+ {
+ "type": "doc",
+ "doc_id": miss,
+ "title": f"plastic lid {index}",
+ "text": f"Title: plastic lid {index}",
+ },
+ ]
+ )
+ qid = f"q-train-{index}"
+ train_queries.append(
+ {
+ "type": "query",
+ "qid": qid,
+ "query": "navy velvet sofa",
+ "gold_doc_ids": [gold],
+ "gold_grades": {gold: 1.0},
+ }
+ )
+ train_eval_rows.append(
+ {
+ "qid": qid,
+ "query": "navy velvet sofa",
+ "hit_ids": [miss, gold],
+ "bm25_ids": [gold],
+ "dense_ids": [miss],
+ "hits": [
+ {"id": miss, "doc_id": miss},
+ {"id": gold, "doc_id": gold},
+ ],
+ "gold_doc_ids": [gold],
+ "gold_grades": {gold: 1.0},
+ }
+ )
+ eval_docs = [
+ {
+ "type": "doc",
+ "doc_id": "gold-e",
+ "title": "Navy velvet sofa",
+ "text": "Title: Navy velvet sofa",
+ },
+ {
+ "type": "doc",
+ "doc_id": "irr-e",
+ "title": "plastic lid",
+ "text": "Title: plastic lid",
+ },
+ ]
+ eval_queries = [
+ {
+ "type": "query",
+ "qid": "q-eval",
+ "query": "navy velvet sofa",
+ "gold_doc_ids": ["gold-e"],
+ "gold_grades": {"gold-e": 1.0},
+ }
+ ]
+ eval_row = {
+ "qid": "q-eval",
+ "query": "navy velvet sofa",
+ "hit_ids": ["irr-e", "gold-e"],
+ "bm25_ids": ["gold-e"],
+ "dense_ids": ["irr-e"],
+ "hits": [
+ {"id": "irr-e", "doc_id": "irr-e"},
+ {"id": "gold-e", "doc_id": "gold-e"},
+ ],
+ "gold_doc_ids": ["gold-e"],
+ "gold_grades": {"gold-e": 1.0},
+ }
+ result = run_ltr_head(
+ {"run_id": "toy-eval", "queries": [eval_row]},
+ eval_docs + eval_queries,
+ dataset_name="toy-eval.jsonl",
+ k=10,
+ ks=(5, 10),
+ n_folds=2,
+ ltr_head=HEAD_LIGHTGBM,
+ train_eval_result={"run_id": "toy-train", "queries": train_eval_rows},
+ train_rows=train_docs + train_queries,
+ train_source_run="toy-train",
+ )
+ self.assertEqual(result["protocol"], PROTOCOL_APPLY)
+ self.assertEqual(result["ltr_train_run"], "toy-train")
+ self.assertEqual(result["ltr_n_train_queries"], 8)
+ self.assertEqual(result["rerank"], "ltr-lightgbm")
+ self.assertEqual(result["queries"][0]["qid"], "q-eval")
+ self.assertNotEqual(result["queries"][0]["qid"], "q-train-0")
+
+ def test_export_hybrid_lists_holdout_unlabeled_is_i(self):
+ from search.export_hybrid_lists import (
+ PROTECTED_OUT_NAMES,
+ export_hybrid_lists,
+ rows_from_eval,
+ )
+
+ self.assertIn("esci_retrieved_lists.jsonl", PROTECTED_OUT_NAMES)
+ eval_result = {
+ "queries": [
+ {
+ "qid": "esci-hold",
+ "query": "navy sofa",
+ "hit_ids": ["gold-h", "miss-h"],
+ "hits": [
+ {"id": "gold-h", "doc_id": "gold-h"},
+ {"id": "miss-h", "doc_id": "miss-h"},
+ ],
+ "gold_grades": {"gold-h": 1.0},
+ },
+ {
+ "qid": "esci-ok",
+ "query": "navy sofa",
+ "hit_ids": ["gold-ok", "miss-ok"],
+ "hits": [
+ {"id": "gold-ok", "doc_id": "gold-ok"},
+ {"id": "miss-ok", "doc_id": "miss-ok"},
+ ],
+ "gold_grades": {"gold-ok": 1.0},
+ },
+ ]
+ }
+ docs = [
+ {
+ "type": "doc",
+ "doc_id": "gold-ok",
+ "title": "Navy sofa",
+ "text": "Title: Navy sofa",
+ },
+ {
+ "type": "doc",
+ "doc_id": "miss-ok",
+ "title": "plastic lid",
+ "text": "Title: plastic lid",
+ },
+ ]
+ queries = [
+ {
+ "type": "query",
+ "qid": "esci-ok",
+ "query": "navy sofa",
+ "gold_doc_ids": ["gold-ok"],
+ "gold_grades": {"gold-ok": 1.0},
+ },
+ {
+ "type": "query",
+ "qid": "esci-hold",
+ "query": "navy sofa",
+ "gold_doc_ids": ["gold-h"],
+ "gold_grades": {"gold-h": 1.0},
+ },
+ ]
+ rows = rows_from_eval(
+ eval_result,
+ docs=docs,
+ queries=queries,
+ holdout={"esci-hold", "hold"},
+ k=50,
+ )
+ qids = {row["query_id"] for row in rows}
+ self.assertNotIn("esci-hold", qids)
+ self.assertIn("esci-ok", qids)
+ by_pid = {row["product_id"]: row["label"] for row in rows}
+ self.assertEqual(by_pid["gold-ok"], "E")
+ self.assertEqual(by_pid["miss-ok"], "I")
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ dataset = root / "train.jsonl"
+ holdout_path = root / "holdout.jsonl"
+ out = root / "hybrid.jsonl"
+ dataset.write_text(
+ json.dumps(docs[0])
+ + "\n"
+ + json.dumps(docs[1])
+ + "\n"
+ + json.dumps(queries[0])
+ + "\n",
+ encoding="utf-8",
+ )
+ holdout_path.write_text(
+ json.dumps(
+ {
+ "type": "query",
+ "qid": "esci-hold",
+ "query": "navy sofa",
+ "gold_doc_ids": ["gold-h"],
+ }
+ )
+ + "\n",
+ encoding="utf-8",
+ )
+ meta = export_hybrid_lists(
+ eval_result=eval_result,
+ dataset_path=dataset,
+ holdout_path=holdout_path,
+ out_path=out,
+ k=50,
+ )
+ self.assertEqual(meta["source"], "hybrid-k50")
+ self.assertEqual(meta["n_queries"], 1)
+ with self.assertRaises(ValueError):
+ export_hybrid_lists(
+ eval_result=eval_result,
+ dataset_path=dataset,
+ holdout_path=holdout_path,
+ out_path=root / "esci_retrieved_lists.jsonl",
+ k=50,
+ )
+
+ def test_retrieved_lists_unlabeled_is_irrelevant(self):
+ from search.finetune_esci_4class import held_out_query_ids
+ from search.mine_retrieved_lists import (
+ is_held_out,
+ labeled_hits,
+ mine_from_groups,
+ select_groups,
+ )
+ from search.pool_first_stage import Bm25Index, rank_docs, tokenize
+
+ docs = [
+ ("B001", tokenize("red ceramic mug")),
+ ("B002", tokenize("blue dinner plate")),
+ ("B003", tokenize("red mug gift set")),
+ ]
+ index = Bm25Index(docs)
+ qrels = {"B001": "E", "B003": "S"}
+ scores = index.scores(tokenize("red ceramic mug"))
+ ranked = rank_docs(scores, ["B001", "B002", "B003"])
+ hits = labeled_hits(ranked, qrels, k=3)
+ by_id = dict(hits)
+ self.assertEqual(by_id["B001"], "E")
+ self.assertEqual(by_id["B003"], "S")
+ self.assertEqual(by_id["B002"], "I")
+ self.assertTrue(any(label == "I" for _, label in hits))
+
+ groups = {
+ "q1": {
+ "query": "red ceramic mug",
+ "qrels": {"B001": "E", "B003": "S"},
+ }
+ }
+ passages = {
+ "B001": "Title: red ceramic mug",
+ "B002": "Title: blue dinner plate",
+ "B003": "Title: red mug gift set",
+ }
+ rows = mine_from_groups(groups, passages, k=3)
+ labels = {row["product_id"]: row["label"] for row in rows}
+ self.assertEqual(labels["B001"], "E")
+ self.assertEqual(labels["B002"], "I")
+
+ holdout = held_out_query_ids(
+ Path(__file__).resolve().parents[1]
+ / "benchmarks"
+ / "data"
+ / "search_esci_74.jsonl"
+ )
+ self.assertTrue(is_held_out("72", holdout))
+ selected = select_groups(
+ {
+ "72": {"query": "not electronics", "qrels": {"x": "E"}},
+ "esci-113": {"query": "html", "qrels": {"y": "E"}},
+ "900001": {"query": "ok", "qrels": {"z": "E"}},
+ },
+ holdout=holdout,
+ max_queries=10,
+ seed=1,
+ )
+ self.assertNotIn("72", selected)
+ self.assertNotIn("esci-113", selected)
+ self.assertIn("900001", selected)
+
+ def test_local_colbert_fake_encoder_ranks_match(self):
+ from search.local_colbert import PLUGIN_DIR, run_local_colbert
+
+ import sys
+
+ if str(PLUGIN_DIR) not in sys.path:
+ sys.path.insert(0, str(PLUGIN_DIR))
+ import encode as colbert_encode
+
+ def fake_tokens(text: str):
+ blob = (text or "").lower()
+ if "match" in blob or blob == "q":
+ return [[1.0, 0.0]]
+ return [[0.0, 1.0]]
+
+ colbert_encode.set_encoder(fake_tokens)
+ try:
+ rows = [
+ {"type": "doc", "doc_id": "hit", "text": "match"},
+ {"type": "doc", "doc_id": "miss", "text": "other"},
+ {
+ "type": "query",
+ "qid": "q1",
+ "query": "q",
+ "gold_doc_ids": ["hit"],
+ "gold_grades": {"hit": 1.0},
+ },
+ ]
+ result = run_local_colbert(
+ rows,
+ dataset_name="toy",
+ k=2,
+ ks=(5, 10, 20, 50),
+ brain_id="harness-local-colbert",
+ )
+ self.assertEqual(result["channels"], ["harness-colbert"])
+ self.assertEqual(result["queries"][0]["hit_ids"][0], "hit")
+ finally:
+ colbert_encode.set_encoder(None)
+
+ def test_failed_report_skips_ledger(self):
+ from search.report import entry_from_report, update_reports_json
+
+ failed = {
+ "status": "failed",
+ "run_id": "search-fail",
+ "brain_id": "searchbenchsmoke",
+ "dataset": "search_toy.jsonl",
+ "n_queries": 8,
+ "ndcg@10": 0.9,
+ }
+ self.assertIsNone(entry_from_report(failed))
+ empty_ok = {
+ "status": "ok",
+ "run_id": "search-empty",
+ "n_queries": 0,
+ }
+ self.assertIsNone(entry_from_report(empty_ok))
+
+ ledger = {
+ "schema_version": 2,
+ "benchmarks": {
+ "locomo": {"name": "LoCoMo", "leaderboard": [{"run_id": "keep-me"}]},
+ "recsys": {"name": "RecSys", "leaderboard": [{"run_id": "keep-recsys"}]},
+ "search": {"name": "Search (hybrid BM25 + dense)", "leaderboard": []},
+ },
+ }
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "REPORTS.json"
+ path.write_text(json.dumps(ledger), encoding="utf-8")
+ with patch("search.report.REPORTS_PATH", path):
+ update_reports_json(failed)
+ after = json.loads(path.read_text(encoding="utf-8"))
+ self.assertEqual(
+ after["benchmarks"]["locomo"]["leaderboard"][0]["run_id"],
+ "keep-me",
+ )
+ self.assertEqual(
+ after["benchmarks"]["recsys"]["leaderboard"][0]["run_id"],
+ "keep-recsys",
+ )
+ self.assertEqual(after["benchmarks"]["search"]["leaderboard"], [])
+
+ ok = {
+ "status": "ok",
+ "run_id": "search-ok",
+ "brain_id": "searchbenchsmoke",
+ "dataset": "search_toy.jsonl",
+ "fusion": "rrf",
+ "n_queries": 8,
+ "ndcg@10": 0.5,
+ "recall@10": 0.6,
+ "mrr": 0.4,
+ "p50_retrieve_ms": 12.0,
+ "p95_retrieve_ms": 30.0,
+ "git_sha": "abc",
+ "recorded_at": "2026-08-18T00:00:00+00:00",
+ }
+ with patch("search.report.REPORTS_PATH", path):
+ update_reports_json(ok)
+ after_ok = json.loads(path.read_text(encoding="utf-8"))
+ self.assertEqual(
+ after_ok["benchmarks"]["locomo"]["leaderboard"][0]["run_id"],
+ "keep-me",
+ )
+ self.assertEqual(len(after_ok["benchmarks"]["search"]["leaderboard"]), 1)
+ self.assertEqual(
+ after_ok["benchmarks"]["search"]["leaderboard"][0]["run_id"],
+ "search-ok",
+ )
+
+ def test_cli_refuses_memory_brain(self):
+ from search.cli import main
+
+ with self.assertRaises(SystemExit) as ctx:
+ main(["--brain", "locomoconv26", "dataset-stats"])
+ self.assertIn("Refusing brain_id", str(ctx.exception))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_search_catalog.py b/tests/test_search_catalog.py
new file mode 100644
index 0000000..882acc0
--- /dev/null
+++ b/tests/test_search_catalog.py
@@ -0,0 +1,679 @@
+import tempfile
+import unittest
+from pathlib import Path
+import os
+
+ENV_DEFAULTS = {
+ "BRAINPAT_TOKEN": "test-token",
+ "MODELS_MODE": "local",
+ "EMBEDDINGS_LOCAL_MODEL": "local-model",
+ "EMBEDDINGS_SMALL_MODEL": "small-model",
+ "EMBEDDING_NODES_DIMENSION": "3",
+ "EMBEDDING_TRIPLETS_DIMENSION": "3",
+ "EMBEDDING_OBSERVATIONS_DIMENSION": "3",
+ "EMBEDDING_DATA_DIMENSION": "3",
+ "EMBEDDING_RELATIONSHIPS_DIMENSION": "3",
+ "REDIS_HOST": "localhost",
+ "REDIS_PORT": "6379",
+ "NEO4J_HOST": "localhost",
+ "NEO4J_PORT": "7687",
+ "NEO4J_USERNAME": "neo4j",
+ "NEO4J_PASSWORD": "password",
+ "MILVUS_HOST": "localhost",
+ "MILVUS_PORT": "19530",
+ "MONGO_CONNECTION_STRING": "mongodb://localhost:27017",
+ "CELERY_WORKER_CONCURRENCY": "1",
+ "OLLAMA_HOST": "localhost",
+ "OLLAMA_PORT": "11434",
+ "OLLAMA_LLM_SMALL_MODEL": "small",
+ "OLLAMA_LLM_LARGE_MODEL": "large",
+}
+for key, value in ENV_DEFAULTS.items():
+ os.environ.setdefault(key, value)
+
+
+class SearchCatalogTests(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ import sys
+
+ root = Path(__file__).resolve().parents[1]
+ bench = root / "benchmarks"
+ if str(root) not in sys.path:
+ sys.path.insert(0, str(root))
+ if str(bench) not in sys.path:
+ sys.path.insert(0, str(bench))
+
+ def test_format_product_text_includes_marker(self):
+ from search.catalog import format_product_text
+ from search.dataset import doc_marker
+
+ text = format_product_text(
+ "B07ABC",
+ title="Blue kettle",
+ description="1.5 liter stovetop kettle",
+ extras=(("Brand", "Acme"), ("Color", "blue")),
+ )
+ self.assertIn(doc_marker("B07ABC"), text)
+ self.assertIn("Title: Blue kettle", text)
+ self.assertIn("Brand: Acme", text)
+
+ def test_select_catalog_keeps_distractors_and_grades(self):
+ from search.catalog import ESCI_GAINS, _docs_and_queries, _select_catalog
+
+ queries, needed = _select_catalog(
+ [
+ {
+ "query_id": "10",
+ "query": "blue kettle",
+ "product_id": "p-exact",
+ "esci_label": "E",
+ },
+ {
+ "query_id": "10",
+ "query": "blue kettle",
+ "product_id": "p-sub",
+ "esci_label": "S",
+ },
+ {
+ "query_id": "10",
+ "query": "blue kettle",
+ "product_id": "p-irr",
+ "esci_label": "I",
+ },
+ ],
+ max_queries=8,
+ max_docs=20,
+ candidates_per_query=40,
+ slice_name="esci-us",
+ dataset="esci",
+ gains=ESCI_GAINS,
+ label_key="esci_label",
+ query_text_key="query",
+ )
+ self.assertEqual(needed, {"p-exact", "p-sub", "p-irr"})
+ self.assertEqual(len(queries), 1)
+ self.assertEqual(queries[0]["gold_grades"]["p-exact"], 1.0)
+ self.assertEqual(queries[0]["gold_grades"]["p-sub"], 0.1)
+ self.assertNotIn("p-irr", queries[0]["gold_grades"])
+ self.assertEqual(queries[0]["candidate_grades"]["p-irr"], 0.0)
+ self.assertEqual(
+ queries[0]["candidate_doc_ids"],
+ ["p-exact", "p-sub", "p-irr"],
+ )
+ rows = _docs_and_queries(
+ queries,
+ products={
+ "p-exact": {"product_title": "Blue kettle", "product_description": ""},
+ "p-sub": {"product_title": "Teapot", "product_description": ""},
+ "p-irr": {"product_title": "Lamp", "product_description": ""},
+ },
+ doc_ids=needed,
+ dataset="esci",
+ title_key="product_title",
+ description_key="product_description",
+ )
+ docs = [row for row in rows if row["type"] == "doc"]
+ self.assertEqual({row["doc_id"] for row in docs}, needed)
+ self.assertEqual(docs[0]["title"], "Blue kettle")
+
+ def test_select_catalog_skips_holdout_qids(self):
+ from search.catalog import ESCI_GAINS, _select_catalog
+
+ queries, needed = _select_catalog(
+ [
+ {
+ "query_id": "72",
+ "query": "held out kettle",
+ "product_id": "p-hold",
+ "esci_label": "E",
+ },
+ {
+ "query_id": "99",
+ "query": "train sofa",
+ "product_id": "p-train",
+ "esci_label": "E",
+ },
+ ],
+ max_queries=8,
+ max_docs=20,
+ candidates_per_query=40,
+ slice_name="esci-us",
+ dataset="esci",
+ gains=ESCI_GAINS,
+ label_key="esci_label",
+ query_text_key="query",
+ holdout_qids={"esci-72", "72"},
+ )
+ self.assertEqual([row["qid"] for row in queries], ["esci-99"])
+ self.assertEqual(needed, {"p-train"})
+
+ def test_wands_gains_select_exact_and_partial_not_irrelevant(self):
+ from search.catalog import WANDS_GAINS, _select_catalog
+
+ self.assertEqual(
+ WANDS_GAINS,
+ {"Exact": 1.0, "Partial": 0.5, "Irrelevant": 0.0},
+ )
+ queries, needed = _select_catalog(
+ [
+ {
+ "query_id": "1",
+ "query": "velvet sofa",
+ "query_class": "Sofas",
+ "product_id": "p-exact",
+ "label": "Exact",
+ },
+ {
+ "query_id": "1",
+ "query": "velvet sofa",
+ "query_class": "Sofas",
+ "product_id": "p-partial",
+ "label": "Partial",
+ },
+ {
+ "query_id": "1",
+ "query": "velvet sofa",
+ "query_class": "Sofas",
+ "product_id": "p-irr",
+ "label": "Irrelevant",
+ },
+ ],
+ max_queries=8,
+ max_docs=20,
+ candidates_per_query=40,
+ slice_name="wands",
+ dataset="wands",
+ gains=WANDS_GAINS,
+ label_key="label",
+ query_text_key="query",
+ query_slice_key="query_class",
+ )
+ self.assertEqual(needed, {"p-exact", "p-partial", "p-irr"})
+ self.assertEqual(len(queries), 1)
+ self.assertEqual(queries[0]["qid"], "wands-1")
+ self.assertEqual(queries[0]["gold_grades"]["p-exact"], 1.0)
+ self.assertEqual(queries[0]["gold_grades"]["p-partial"], 0.5)
+ self.assertNotIn("p-irr", queries[0]["gold_grades"])
+ self.assertEqual(queries[0]["gold_doc_ids"], ["p-exact", "p-partial"])
+ self.assertEqual(queries[0]["candidate_grades"]["p-irr"], 0.0)
+
+ def test_frozen_wands_jsonl_blocks_overwrite(self):
+ from search.catalog import DATA_DIR, catalog_overwrite_blocked
+
+ self.assertTrue(
+ catalog_overwrite_blocked(DATA_DIR / "search_esci_74.jsonl")
+ )
+ wands = DATA_DIR / "search_wands.jsonl"
+ self.assertTrue(wands.exists() and wands.stat().st_size > 0)
+ self.assertTrue(catalog_overwrite_blocked(wands))
+ self.assertFalse(
+ catalog_overwrite_blocked(DATA_DIR / "search_esci.jsonl")
+ )
+
+ def test_frozen_structured_brains_exclude_wandsgraph(self):
+ from search.config import FROZEN_STRUCTURED_BRAINS
+
+ self.assertIn("searchbenchwands", FROZEN_STRUCTURED_BRAINS)
+ self.assertIn("searchbenchesci74", FROZEN_STRUCTURED_BRAINS)
+ self.assertNotIn("searchbenchwandsgraph", FROZEN_STRUCTURED_BRAINS)
+ self.assertNotIn("searchbenchjdslice", FROZEN_STRUCTURED_BRAINS)
+
+ def test_evaluate_refuses_frozen_structured_ingest(self):
+ from search.evaluate import evaluate_search
+
+ class _Client:
+ class _Settings:
+ brain_id = "searchbenchwands"
+
+ settings = _Settings()
+
+ with self.assertRaises(SystemExit):
+ evaluate_search(_Client(), [], ingest_graph=True)
+ with self.assertRaises(SystemExit):
+ evaluate_search(
+ _Client(),
+ [],
+ interactions=[{"user_id": "u", "item_id": "0"}],
+ )
+
+ def test_wands_feature_parse_and_catalog_triples(self):
+ from search.mapping import (
+ docs_to_triples,
+ entity_uuid,
+ parse_feature_string,
+ )
+
+ parsed = parse_feature_string(
+ "|Color:Navy|Material:Velvet|InStock:yes|Finish:|"
+ )
+ self.assertEqual(parsed, [("Color", "Navy"), ("Material", "Velvet")])
+ docs = [
+ {
+ "doc_id": f"d{i}",
+ "title": f"Item {i}",
+ "class": "sofas" if i == 1 else "",
+ "brand": "Acme" if i == 2 else "",
+ "features": "|Color:Navy|" if i == 3 else "",
+ "dataset": "wands",
+ }
+ for i in range(1, 6)
+ ]
+ triples = docs_to_triples(docs)
+ uuids = {row["subject"]["uuid"] for row in triples}
+ self.assertEqual(uuids, {entity_uuid(f"d{i}") for i in range(1, 6)})
+ self.assertTrue(all(row.get("event") is None for row in triples))
+ self.assertTrue(all(row["subj_event"]["name"] == "HAS" for row in triples))
+ self.assertTrue(all("event_obj" not in row or row.get("event_obj") is None for row in triples))
+ objects = {(row["object"]["type"], row["object"]["name"]) for row in triples}
+ self.assertIn(("CLASS", "sofas"), objects)
+ self.assertIn(("ATTR", "Acme"), objects)
+ self.assertIn(("ATTR", "Navy"), objects)
+ self.assertNotIn("demorecsys", str(triples))
+
+ def test_catalog_graph_sofas_modern_has_search_text(self):
+ from search.mapping import doc_to_triples, parse_feature_string, split_hierarchy
+
+ parsed = parse_feature_string("dsprimaryproductstyle : modern|color : navy")
+ self.assertIn(("dsprimaryproductstyle", "modern"), parsed)
+ self.assertEqual(split_hierarchy("Furniture > Living Room > Sofas"), ["Furniture", "Living Room", "Sofas"])
+ triples = doc_to_triples(
+ {
+ "doc_id": "sofa-1",
+ "title": "Velvet sofa",
+ "description": "A low modern sofa for small rooms",
+ "class": "sofas",
+ "hierarchy": "Furniture > Living Room > Sofas",
+ "features": "style:modern|color:navy",
+ "price": "899",
+ "dataset": "wands",
+ }
+ )
+ objects = {(row["object"]["type"], row["object"]["name"]) for row in triples}
+ self.assertIn(("CLASS", "sofas"), objects)
+ self.assertIn(("ATTR", "modern"), objects)
+ self.assertIn(("TYPE", "Furniture"), objects)
+ subject = triples[0]["subject"]
+ self.assertIn("modern sofa", subject["properties"]["search_text"].lower())
+ self.assertIn("sofas", subject["properties"]["search_text"].lower())
+ self.assertIn("modern", subject["properties"]["search_text"].lower())
+ self.assertIn("navy", subject["properties"]["search_text"].lower())
+ self.assertEqual(subject["properties"]["price"], "899")
+ hub_texts = {row["object"]["properties"]["search_text"] for row in triples}
+ self.assertTrue(any("modern" in text.lower() for text in hub_texts))
+ self.assertNotIn(("ATTR", "899"), objects)
+ self.assertTrue(all(row.get("event") is None for row in triples))
+ self.assertTrue(all(row["subj_event"]["name"] == "HAS" for row in triples))
+ self.assertTrue(
+ all("dsprimaryproductstyle" not in str(row.get("subj_event") or {}) for row in triples)
+ )
+
+ def test_entity_search_text_prefers_catalog_blob(self):
+ from search.mapping import doc_id_from_text, doc_to_triples, node_id_from_passage_text
+
+ blob = (
+ "DOCID wands-bed.\n"
+ "Title: solid wood platform bed\n"
+ "Class: Beds\n"
+ "Features: dsprimaryproductstyle : modern|woodspecies : rubberwood"
+ )
+ triples = doc_to_triples(
+ {
+ "doc_id": "wands-bed",
+ "title": "solid wood platform bed",
+ "class": "Beds",
+ "features": "dsprimaryproductstyle : modern|woodspecies : rubberwood",
+ "text": blob,
+ }
+ )
+ subject = triples[0]["subject"]
+ self.assertEqual(subject["properties"]["search_text"], blob)
+ self.assertIn("rubberwood", subject["properties"]["search_text"])
+ self.assertIsNone(triples[0].get("event"))
+ self.assertEqual(triples[0]["subj_event"]["name"], "HAS")
+ self.assertNotIn("rubberwood", triples[0]["subj_event"]["name"])
+ self.assertEqual(doc_id_from_text(blob), "wands-bed")
+ self.assertEqual(node_id_from_passage_text(blob), "wands-bed")
+ self.assertIsNone(node_id_from_passage_text("no marker here"))
+
+ def test_catalog_has_is_direct_edge(self):
+ from search.mapping import doc_to_triples, is_static_has_triple
+
+ triples = doc_to_triples(
+ {
+ "doc_id": "0",
+ "title": "solid wood platform bed",
+ "class": "Beds",
+ "features": "woodspecies : rubberwood",
+ }
+ )
+ self.assertTrue(triples)
+ self.assertTrue(all(is_static_has_triple(row) for row in triples))
+ self.assertTrue(all(row.get("event") is None for row in triples))
+ kinds = {(row["object"]["type"], row["object"]["name"]) for row in triples}
+ self.assertIn(("CLASS", "Beds"), kinds)
+ self.assertIn(("ATTR", "rubberwood"), kinds)
+
+ def test_entity_backfill_refuses_frozen_brains(self):
+ from search.backfill_entities import (
+ apply_entity_text_backfill,
+ backfill_rows_from_docs,
+ refuse_entity_backfill,
+ )
+
+ self.assertEqual(refuse_entity_backfill("searchbenchwandsgraph"), "searchbenchwandsgraph")
+ with self.assertRaises(SystemExit):
+ refuse_entity_backfill("searchbenchwands")
+ with self.assertRaises(SystemExit):
+ refuse_entity_backfill("searchbenchesci74")
+ rows = backfill_rows_from_docs(
+ [
+ {
+ "doc_id": "0",
+ "title": "solid wood platform bed",
+ "class": "Beds",
+ "features": "dsprimaryproductstyle : modern|woodspecies : rubberwood",
+ "text": "DOCID 0.\nTitle: solid wood platform bed\nFeatures: woodspecies : rubberwood",
+ }
+ ]
+ )
+ self.assertEqual(rows[0]["uuid"], "0")
+ self.assertIn("rubberwood", rows[0]["search_text"])
+ with self.assertRaises(SystemExit):
+ apply_entity_text_backfill(
+ brain_id="searchbenchwands",
+ rows=rows,
+ graph=None,
+ embeddings=None,
+ vector_store=None,
+ )
+
+ def test_apply_entity_backfill_updates_entity_not_event(self):
+ from types import SimpleNamespace
+ from unittest.mock import MagicMock
+
+ from search.backfill_entities import apply_entity_text_backfill
+ from src.constants.embeddings import Vector
+
+ node = SimpleNamespace(
+ uuid="0",
+ name="bed",
+ labels=["ENTITY"],
+ properties={"search_text": "bed", "v_id": "old-v"},
+ )
+ graph = MagicMock()
+ graph.get_by_uuid.return_value = node
+ embeddings = MagicMock()
+ embeddings.embed_text.return_value = Vector(
+ id="new", embeddings=[0.1], metadata={}
+ )
+ store = MagicMock()
+ store.add_vectors.return_value = ["vid-1"]
+ summary = apply_entity_text_backfill(
+ brain_id="searchbenchwandsgraph",
+ rows=[{"uuid": "0", "name": "bed", "search_text": "rubberwood modern bed"}],
+ graph=graph,
+ embeddings=embeddings,
+ vector_store=store,
+ is_item=lambda uuid, labels: "ENTITY" in (labels or []),
+ )
+ self.assertEqual(summary["updated"], 1)
+ self.assertEqual(summary["missing"], 0)
+ written = [
+ call.kwargs.get("new_properties") or {}
+ for call in graph.update_node.call_args_list
+ ]
+ self.assertTrue(any("rubberwood" in str(props.get("search_text") or "") for props in written))
+ event = SimpleNamespace(
+ uuid="evt-1",
+ name="HAS",
+ labels=["EVENT"],
+ properties={},
+ )
+ graph.get_by_uuid.return_value = event
+ skipped = apply_entity_text_backfill(
+ brain_id="searchbenchwandsgraph",
+ rows=[{"uuid": "evt-1", "name": "HAS", "search_text": "should not copy"}],
+ graph=graph,
+ embeddings=embeddings,
+ vector_store=store,
+ is_item=lambda uuid, labels: "ENTITY" in (labels or []),
+ )
+ self.assertEqual(skipped["updated"], 0)
+ self.assertEqual(skipped["skipped"], 1)
+
+ def test_wandsgraph_node_join_calls_neighbors(self):
+ from search.client import TimedResult
+ from search.evaluate import assert_wandsgraph_node_join
+
+ class _Client:
+ class _Settings:
+ brain_id = "searchbenchwandsgraph"
+
+ settings = _Settings()
+
+ def get_neighbors(self, uuid, limit=5):
+ self.seen = uuid
+ return TimedResult(data={"count": 4}, latency_ms=1.0, status_code=200)
+
+ client = _Client()
+ out = assert_wandsgraph_node_join(
+ client,
+ {
+ "queries": [
+ {
+ "qid": "q1",
+ "hits": [
+ {
+ "id": "chunk-1",
+ "doc_id": "0",
+ "node_id": "0",
+ "channel": "passages",
+ }
+ ],
+ }
+ ]
+ },
+ )
+ self.assertEqual(out["node_id"], "0")
+ self.assertEqual(out["status_code"], 200)
+ self.assertEqual(client.seen, "0")
+ skipped = assert_wandsgraph_node_join(
+ type("C", (), {"settings": type("S", (), {"brain_id": "searchbenchsmoke"})()})(),
+ {"queries": []},
+ )
+ self.assertTrue(skipped["skipped"])
+
+ def test_interaction_triples_have_happened_at(self):
+ from search.mapping import interactions_to_triples, load_interaction_rows
+
+ rows = [
+ {
+ "user_id": "u01",
+ "item_id": "sku-101",
+ "behavior": "view",
+ "timestamp": "2024-01-02T10:00:00Z",
+ "category": "phones",
+ "brand": "Acme",
+ },
+ {
+ "user_id": "u01",
+ "item_id": "sku-102",
+ "behavior": "purchase",
+ "timestamp": "2024-01-05T12:00:00Z",
+ },
+ {
+ "user_id": "u02",
+ "item_id": "sku-201",
+ "behavior": "view",
+ "timestamp": "2024-01-02T09:00:00Z",
+ },
+ ]
+ triples = interactions_to_triples(rows)
+ events = [
+ row["event"]
+ for row in triples
+ if row.get("event") and row["event"]["name"] != "HAS"
+ ]
+ self.assertGreaterEqual(len(events), 3)
+ self.assertTrue(all(event.get("happened_at") for event in events))
+ self.assertTrue(all(event["type"] == "EVENT" for event in events))
+ self.assertTrue(all(row["object"]["uuid"] in {"sku-101", "sku-102", "sku-201"} or row["object"]["type"] != "ENTITY" for row in triples))
+ entity_ids = {
+ row["object"]["uuid"]
+ for row in triples
+ if row["object"]["type"] == "ENTITY"
+ }
+ self.assertEqual(entity_ids, {"sku-101", "sku-102", "sku-201"})
+ self.assertNotIn("PRODUCT", str(triples))
+ self.assertNotIn("Categorized", str(triples))
+ self.assertNotIn("demorecsys", str(triples))
+ root = Path(__file__).resolve().parents[1]
+ toy = root / "benchmarks" / "data" / "recsys_toy.jsonl"
+ loaded = load_interaction_rows(toy)[:3]
+ self.assertEqual(len(loaded), 3)
+ self.assertEqual(loaded[0]["item_id"], "sku-101")
+
+ def test_interaction_options_emit_prefers_not_catalog_has(self):
+ from search.mapping import hub_uuid, interaction_to_triples
+
+ triples = interaction_to_triples(
+ {
+ "user_id": "u01",
+ "item_id": "sku-101",
+ "behavior": "view",
+ "timestamp": "2024-01-02T10:00:00Z",
+ "color": "navy",
+ "options": {"style": "70s", "color": "olive"},
+ },
+ seq=1,
+ )
+ prefers = [
+ row
+ for row in triples
+ if (row.get("subj_event") or {}).get("name") == "PREFERS"
+ ]
+ self.assertEqual(len(prefers), 2)
+ self.assertTrue(all(row.get("event") is None for row in prefers))
+ self.assertTrue(all(row["subject"]["uuid"] == "user:u01" for row in prefers))
+ style_hub = hub_uuid("attr", "70s")
+ olive_hub = hub_uuid("attr", "olive")
+ prefers_tips = {row["object"]["uuid"] for row in prefers}
+ self.assertEqual(prefers_tips, {style_hub, olive_hub})
+ has_tips = {
+ row["object"]["uuid"]
+ for row in triples
+ if (row.get("subj_event") or {}).get("name") == "HAS"
+ }
+ self.assertIn(hub_uuid("attr", "navy"), has_tips)
+ self.assertNotIn(style_hub, has_tips)
+ self.assertNotIn(olive_hub, has_tips)
+
+ def test_personalize_smoke_fixture_emits_prefers(self):
+ from search.mapping import hub_uuid, load_interaction_rows, interactions_to_triples
+
+ root = Path(__file__).resolve().parents[1]
+ rows = load_interaction_rows(
+ root / "benchmarks" / "data" / "search_personalize_smoke.jsonl"
+ )
+ self.assertEqual(len(rows), 4)
+ triples = interactions_to_triples(rows)
+ prefers = [
+ row
+ for row in triples
+ if (row.get("subj_event") or {}).get("name") == "PREFERS"
+ ]
+ self.assertGreaterEqual(len(prefers), 1)
+ self.assertIn(hub_uuid("attr", "70s"), {row["object"]["uuid"] for row in prefers})
+ self.assertTrue(any(row["event"]["name"] == "Favorite" for row in triples if row.get("event")))
+
+ def test_cli_unknown_catalog(self):
+ from search.cli import main
+
+ self.assertEqual(main(["download", "--name", "not-a-dataset"]), 1)
+
+ def test_write_roundtrip(self):
+ from search.dataset import dataset_stats, load_records, write_records
+
+ rows = [
+ {"type": "doc", "doc_id": "a", "text": "DOCID a. Title: kettle"},
+ {
+ "type": "query",
+ "qid": "q1",
+ "query": "kettle",
+ "gold_doc_ids": ["a"],
+ "gold_grades": {"a": 1.0},
+ "slice": "esci-us",
+ },
+ ]
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "search_esci.jsonl"
+ write_records(rows, path)
+ loaded = load_records(path)
+ stats = dataset_stats(loaded)
+ self.assertEqual(stats["n_docs"], 1)
+ self.assertEqual(stats["n_queries"], 1)
+ self.assertTrue(stats["graded"])
+
+ def test_esci_locale_paths_keep_us_default(self):
+ from search.catalog import ESCI_JSONL, catalog_jsonl_path, normalize_esci_locale
+
+ self.assertEqual(catalog_jsonl_path("esci"), ESCI_JSONL)
+ self.assertEqual(catalog_jsonl_path("esci", locale="us"), ESCI_JSONL)
+ self.assertEqual(catalog_jsonl_path("esci", locale="US"), ESCI_JSONL)
+ es_path = catalog_jsonl_path("esci", locale="es")
+ jp_path = catalog_jsonl_path("esci", locale="jp")
+ self.assertEqual(es_path.name, "search_esci_es.jsonl")
+ self.assertEqual(jp_path.name, "search_esci_jp.jsonl")
+ self.assertNotEqual(es_path, ESCI_JSONL)
+ self.assertNotEqual(jp_path, ESCI_JSONL)
+ self.assertEqual(normalize_esci_locale("es"), "es")
+
+ def test_esci_locale_rejects_italian(self):
+ from search.catalog import catalog_jsonl_path, normalize_esci_locale
+ from search.cli import main
+
+ with self.assertRaises(ValueError) as ctx:
+ normalize_esci_locale("it")
+ self.assertIn("no Italian", str(ctx.exception))
+ with self.assertRaises(ValueError):
+ catalog_jsonl_path("esci", locale="it")
+ self.assertEqual(main(["download", "--name", "esci", "--locale", "it"]), 1)
+
+ def test_italian_smoke_is_not_esci(self):
+ from search.dataset import dataset_stats, load_records
+
+ root = Path(__file__).resolve().parents[1]
+ path = root / "benchmarks" / "data" / "search_italian_smoke.jsonl"
+ rows = load_records(path)
+ stats = dataset_stats(rows)
+ self.assertGreaterEqual(stats["n_docs"], 3)
+ self.assertGreaterEqual(stats["n_queries"], 3)
+ self.assertEqual(stats["slices"], {"italian-smoke": stats["n_queries"]})
+ for row in rows:
+ self.assertNotEqual(row.get("dataset"), "esci")
+ self.assertNotIn("esci-", str(row.get("qid") or ""))
+ self.assertNotIn("esci-", str(row.get("slice") or ""))
+
+ def test_italian_inflect_is_not_esci(self):
+ from search.dataset import dataset_stats, load_records
+
+ root = Path(__file__).resolve().parents[1]
+ path = root / "benchmarks" / "data" / "search_italian_smoke_inflect.jsonl"
+ rows = load_records(path)
+ stats = dataset_stats(rows)
+ self.assertGreaterEqual(stats["n_docs"], 3)
+ self.assertGreaterEqual(stats["n_queries"], 3)
+ self.assertEqual(stats["slices"], {"italian-smoke-inflect": stats["n_queries"]})
+ queries = [row["query"] for row in rows if row.get("type") == "query"]
+ self.assertIn("bollitori acciaio", queries)
+ self.assertIn("divani velluto", queries)
+ self.assertIn("caffettiere alluminio", queries)
+ for row in rows:
+ self.assertNotEqual(row.get("dataset"), "esci")
+ self.assertNotIn("esci-", str(row.get("qid") or ""))
+ self.assertNotIn("esci-", str(row.get("slice") or ""))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_search_esci_4class.py b/tests/test_search_esci_4class.py
new file mode 100644
index 0000000..ffdbf1f
--- /dev/null
+++ b/tests/test_search_esci_4class.py
@@ -0,0 +1,156 @@
+import unittest
+from pathlib import Path
+
+
+class SearchEsci4ClassTests(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ import sys
+
+ root = Path(__file__).resolve().parents[1]
+ bench = root / "benchmarks"
+ if str(bench) not in sys.path:
+ sys.path.insert(0, str(bench))
+
+ def test_weighted_scores_match_esci_gains(self):
+ from search.finetune_esci_4class import weighted_scores
+
+ exact = weighted_scores([[1.0, 0.0, 0.0, 0.0]])
+ sub = weighted_scores([[0.0, 1.0, 0.0, 0.0]])
+ comp = weighted_scores([[0.0, 0.0, 1.0, 0.0]])
+ irr = weighted_scores([[0.0, 0.0, 0.0, 1.0]])
+ self.assertEqual(exact, [1.0])
+ self.assertEqual(sub, [0.1])
+ self.assertEqual(comp, [0.01])
+ self.assertEqual(irr, [0.0])
+ mixed = weighted_scores([[0.5, 0.5, 0.0, 0.0]])
+ self.assertAlmostEqual(mixed[0], 0.55)
+
+ def test_rank_doc_ids_orders_by_weighted_score(self):
+ from search.finetune_esci_4class import rank_doc_ids, weighted_scores
+
+ doc_ids = ["irr", "exact", "sub"]
+ scores = weighted_scores(
+ [
+ [0.05, 0.05, 0.1, 0.8],
+ [0.9, 0.05, 0.05, 0.0],
+ [0.1, 0.8, 0.05, 0.05],
+ ]
+ )
+ self.assertEqual(rank_doc_ids(doc_ids, scores), ["exact", "sub", "irr"])
+
+ def test_class_weights_upweight_rare_labels(self):
+ from search.finetune_esci_4class import class_weights
+
+ weights = class_weights([80, 10, 5, 5])
+ self.assertGreater(weights[2], weights[0])
+ self.assertGreater(weights[3], weights[0])
+
+ def test_held_out_query_ids_strip_esci_prefix(self):
+ import json
+ import tempfile
+
+ from search.finetune_esci_4class import held_out_query_ids
+
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "slice.jsonl"
+ path.write_text(
+ json.dumps(
+ {
+ "type": "doc",
+ "doc_id": "p1",
+ "text": "DOCID p1.\nTitle: kettle",
+ }
+ )
+ + "\n"
+ + json.dumps(
+ {
+ "type": "query",
+ "qid": "esci-99",
+ "query": "kettle",
+ "gold_doc_ids": ["p1"],
+ }
+ )
+ + "\n",
+ encoding="utf-8",
+ )
+ holdout = held_out_query_ids(path)
+ self.assertIn("esci-99", holdout)
+ self.assertIn("99", holdout)
+
+ def test_run_4class_on_pool_keeps_irrelevant_and_ranks_exact_first(self):
+ from search.rank_pool_4class import run_4class_on_pool
+
+ def predict(pairs):
+ scores = []
+ for _, text in pairs:
+ if "gold kettle" in text:
+ scores.append([0.85, 0.1, 0.05, 0.0])
+ elif "partial kettle" in text:
+ scores.append([0.1, 0.8, 0.05, 0.05])
+ else:
+ scores.append([0.05, 0.05, 0.1, 0.8])
+ return scores
+
+ rows = [
+ {"type": "doc", "doc_id": "irr", "text": "DOCID irr. Title: lamp"},
+ {
+ "type": "doc",
+ "doc_id": "gold",
+ "text": "DOCID gold. Title: gold kettle",
+ },
+ {
+ "type": "doc",
+ "doc_id": "sub",
+ "text": "DOCID sub. Title: partial kettle",
+ },
+ {
+ "type": "query",
+ "qid": "esci-1",
+ "query": "kettle",
+ "gold_doc_ids": ["gold", "sub"],
+ "gold_grades": {"gold": 1.0, "sub": 0.1},
+ "candidate_doc_ids": ["irr", "gold", "sub"],
+ "candidate_grades": {"irr": 0.0, "gold": 1.0, "sub": 0.1},
+ "slice": "esci-us",
+ },
+ ]
+ result = run_4class_on_pool(
+ rows,
+ dataset_name="toy.jsonl",
+ predict=predict,
+ model_name="toy-4class",
+ brain_id="searchbenchesci74",
+ )
+ self.assertEqual(result["status"], "ok")
+ self.assertEqual(result["n_queries"], 1)
+ self.assertTrue(result["rank_pool"])
+ ranked = result["queries"][0]["hit_ids"]
+ self.assertEqual(ranked, ["gold", "sub", "irr"])
+ self.assertEqual(ranked[-1], "irr")
+ self.assertGreater(result["metrics"]["ndcg@20"], 0.9)
+
+ def test_persist_tokenizer_max_length_writes_train_length(self):
+ import tempfile
+ from types import SimpleNamespace
+
+ from search.finetune_esci_4class import persist_tokenizer_max_length
+
+ class FakeTokenizer:
+ def __init__(self):
+ self.model_max_length = 10**18
+ self.saved = None
+
+ def save_pretrained(self, path):
+ self.saved = path
+
+ tokenizer = FakeTokenizer()
+ model = SimpleNamespace(max_length=192, tokenizer=tokenizer)
+ with tempfile.TemporaryDirectory() as tmp:
+ persist_tokenizer_max_length(model, Path(tmp))
+ self.assertEqual(tokenizer.model_max_length, 192)
+ self.assertEqual(tokenizer.saved, tmp)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_search_jdsearch.py b/tests/test_search_jdsearch.py
new file mode 100644
index 0000000..9d22353
--- /dev/null
+++ b/tests/test_search_jdsearch.py
@@ -0,0 +1,308 @@
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import MagicMock
+
+FIXTURE_DIR = (
+ Path(__file__).resolve().parents[1] / "tests" / "fixtures" / "jdsearch"
+)
+
+
+class SearchJdsearchTests(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ import sys
+
+ root = Path(__file__).resolve().parents[1]
+ bench = root / "benchmarks"
+ if str(root) not in sys.path:
+ sys.path.insert(0, str(root))
+ if str(bench) not in sys.path:
+ sys.path.insert(0, str(bench))
+
+ def _fixture_paths(self) -> tuple[Path, Path]:
+ product = FIXTURE_DIR / "product_meta_data.txt"
+ behavior = FIXTURE_DIR / "user_behavior_data.txt"
+ self.assertTrue(product.exists(), product)
+ self.assertTrue(behavior.exists(), behavior)
+ return product, behavior
+
+ def test_term_and_list_separators(self):
+ from search.jdsearch import TERM_SEP, join_terms, split_list
+
+ self.assertEqual(TERM_SEP, "\030")
+ self.assertEqual(join_terms(f"111{TERM_SEP}222"), "111 222")
+ self.assertEqual(split_list("100_200_999"), ["100", "200", "999"])
+ self.assertEqual(split_list(""), [])
+
+ def test_stats_histogram_and_truncated_count(self):
+ from search.jdsearch import PAPER_N_PRODUCTS, collect_jdsearch_stats
+
+ product, behavior = self._fixture_paths()
+ stats = collect_jdsearch_stats(
+ product_path=product,
+ behavior_path=behavior,
+ extract=False,
+ )
+ self.assertEqual(stats["n_behavior_rows"], 2)
+ self.assertEqual(stats["n_products_seen"], 2)
+ self.assertFalse(stats["truncated"])
+ self.assertIn("1", stats["label_histogram"])
+ self.assertEqual(stats["label_scheme"], "graded")
+ self.assertGreater(stats["history_type_counts"].get("CLICK", 0), 0)
+ self.assertGreater(stats["history_type_counts"].get("FLW", 0), 0)
+ self.assertIn("999", {"999"})
+ self.assertGreater(stats["n_wids_missing_meta"], 0)
+
+ truncated = collect_jdsearch_stats(
+ product_path=product,
+ behavior_path=behavior,
+ extract=False,
+ max_behavior_rows=1,
+ max_product_rows=1,
+ )
+ self.assertTrue(truncated["truncated"])
+ self.assertEqual(truncated["n_behavior_rows"], 1)
+ self.assertEqual(truncated["n_products_seen"], 1)
+ self.assertNotEqual(truncated["n_products_seen"], PAPER_N_PRODUCTS)
+ self.assertLess(truncated["n_products_seen"], 12_000_000)
+
+ def test_prepare_rows_caps_target_and_gold(self):
+ from search.catalog import FROZEN_JSONL_IF_EXISTS, prepare_catalog
+ from search.dataset import dataset_stats, load_records, split_corpus
+ from search.jdsearch import (
+ JDSEARCH_NAME,
+ jdsearch_interactions_path,
+ prepare_jdsearch_rows,
+ )
+
+ product, behavior = self._fixture_paths()
+ wands = (
+ Path(__file__).resolve().parents[1]
+ / "benchmarks"
+ / "data"
+ / "search_wands.jsonl"
+ )
+ wands_mtime = wands.stat().st_mtime if wands.exists() else None
+ rows, interactions, stats = prepare_jdsearch_rows(
+ max_queries=80,
+ max_docs=2000,
+ product_path=product,
+ behavior_path=behavior,
+ extract=False,
+ )
+ docs, queries = split_corpus(rows)
+ self.assertLessEqual(len(queries), 80)
+ self.assertLessEqual(len(docs), 2000)
+ self.assertEqual(stats["label_scheme"], "graded")
+ self.assertTrue(queries)
+ for query in queries:
+ self.assertTrue(str(query.get("target") or "").startswith("jd-u"))
+ self.assertTrue(query.get("gold_doc_ids"))
+ self.assertTrue(query.get("gold_grades"))
+ self.assertTrue(query.get("candidate_doc_ids"))
+ self.assertTrue(any(row.get("behavior") == "follow" for row in interactions))
+ self.assertTrue(any(row.get("behavior") == "click" for row in interactions))
+ qids = {query["qid"] for query in queries}
+ self.assertTrue(qids)
+ for query in queries:
+ self.assertNotIn("999", query["gold_doc_ids"])
+ self.assertIn("search_jdsearch.jsonl", FROZEN_JSONL_IF_EXISTS)
+ with tempfile.TemporaryDirectory() as tmp:
+ dest = Path(tmp) / "search_jdsearch.jsonl"
+ written = prepare_catalog(
+ JDSEARCH_NAME,
+ out_path=dest,
+ max_queries=80,
+ max_docs=2000,
+ )
+ self.assertEqual(written, dest)
+ loaded = load_records(dest)
+ stats = dataset_stats(loaded)
+ self.assertLessEqual(stats["n_queries"], 80)
+ self.assertLessEqual(stats["n_docs"], 2000)
+ _, loaded_queries = split_corpus(loaded)
+ self.assertTrue(all(q.get("target") and q.get("gold_doc_ids") for q in loaded_queries))
+ inter_path = jdsearch_interactions_path(dest)
+ self.assertTrue(inter_path.exists())
+ self.assertFalse((Path(tmp) / "search_wands.jsonl").exists())
+ self.assertFalse((Path(tmp) / "search_esci_74.jsonl").exists())
+ if wands_mtime is not None:
+ self.assertEqual(wands.stat().st_mtime, wands_mtime)
+
+ def test_prepare_drops_all_missing_gold(self):
+ from search.jdsearch import prepare_jdsearch_rows
+
+ product, behavior = self._fixture_paths()
+ with tempfile.TemporaryDirectory() as tmp:
+ extra_behavior = Path(tmp) / "user_behavior_data.txt"
+ extra_behavior.write_text(
+ behavior.read_text(encoding="utf-8")
+ + "onlymissing\t999\t1.0\t-1\t\t\t0\n",
+ encoding="utf-8",
+ )
+ rows, _interactions, _stats = prepare_jdsearch_rows(
+ product_path=product,
+ behavior_path=extra_behavior,
+ extract=False,
+ )
+ queries = [row for row in rows if row.get("type") == "query"]
+ self.assertTrue(all("999" not in (q.get("gold_doc_ids") or []) for q in queries))
+ self.assertTrue(all(q.get("gold_doc_ids") for q in queries))
+
+ def test_follow_event_not_has(self):
+ import importlib.util
+
+ from search.mapping import HAS_EVENT_NAME, interaction_to_triples
+
+ triples = interaction_to_triples(
+ [
+ {
+ "user_id": "jd-u0",
+ "item_id": "100",
+ "behavior": "follow",
+ "timestamp": "2022-10-16T00:00:00Z",
+ }
+ ]
+ )
+ events = [
+ row["event"]["name"]
+ for row in triples
+ if isinstance(row.get("event"), dict) and row["event"].get("name")
+ ]
+ self.assertIn("Follow", events)
+ self.assertNotIn(HAS_EVENT_NAME, events)
+ mapping_path = (
+ Path(__file__).resolve().parents[1]
+ / "plugins"
+ / "features-rec"
+ / "models"
+ / "mapping.py"
+ )
+ spec = importlib.util.spec_from_file_location(
+ "features_rec_mapping_jdsearch", mapping_path
+ )
+ self.assertIsNotNone(spec)
+ self.assertIsNotNone(spec.loader)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ self.assertEqual(mod.normalize_behavior("FLW"), ("Follow", "TARGETED"))
+ self.assertEqual(mod.behavior_weight("FLW"), 0.5)
+ self.assertEqual(mod.behavior_weight("follow"), 0.5)
+
+ def test_evaluate_sends_target_only_when_personalized(self):
+ from search.client import TimedResult
+ from search.evaluate import evaluate_search
+ from src.services.api.constants.requests import SearchRequestBody
+
+ fields = set(SearchRequestBody.model_fields)
+ self.assertIn("target", fields)
+ self.assertNotIn("user_id", fields)
+
+ class _Client:
+ class _Settings:
+ brain_id = "searchbenchjdslice"
+
+ settings = _Settings()
+
+ def __init__(self):
+ self.calls = []
+
+ def search(self, query, **kwargs):
+ self.calls.append({"query": query, **kwargs})
+ return TimedResult(data={"hits": []}, latency_ms=1.0)
+
+ def list_text_chunks(self, **kwargs):
+ skip = int(kwargs.get("skip") or 0)
+ if skip:
+ return TimedResult(data={"data": [], "total": 1}, latency_ms=1.0)
+ return TimedResult(
+ data={
+ "data": [{"id": "c1", "text": "DOCID 100. 111 222"}],
+ "total": 1,
+ },
+ latency_ms=1.0,
+ )
+
+ rows = [
+ {
+ "type": "doc",
+ "doc_id": "100",
+ "text": "DOCID 100. 111 222",
+ },
+ {
+ "type": "query",
+ "qid": "jdsearch-0",
+ "query": "111 222",
+ "target": "jd-u0",
+ "gold_doc_ids": ["100"],
+ "gold_grades": {"100": 1.0},
+ "candidate_doc_ids": ["100"],
+ },
+ ]
+ client = _Client()
+ evaluate_search(client, rows, skip_ingest=True, personalize=True)
+ self.assertEqual(client.calls[0].get("target"), "jd-u0")
+ client.calls.clear()
+ evaluate_search(client, rows, skip_ingest=True, personalize=False)
+ self.assertFalse(client.calls[0].get("target"))
+ client.calls.clear()
+ evaluate_search(
+ client,
+ [{**rows[1], "target": ""}, rows[0]]
+ if False
+ else [
+ rows[0],
+ {**rows[1], "target": ""},
+ ],
+ skip_ingest=True,
+ personalize=True,
+ )
+ self.assertFalse(client.calls[0].get("target"))
+
+ def test_client_search_body_omits_target_by_default(self):
+ from search.client import BrainAPIClient, TimedResult
+ from search.config import Settings
+
+ captured = {}
+
+ class _Stub(BrainAPIClient):
+ def __post_init__(self):
+ self._client = MagicMock()
+
+ def _request(self, method, path, *, json=None, params=None):
+ captured["json"] = json
+ return TimedResult(data={"hits": []}, latency_ms=1.0)
+
+ settings = MagicMock(spec=Settings)
+ settings.brain_id = "searchbenchjdslice"
+ settings.brainapi_url = "http://127.0.0.1"
+ settings.brainpat_token = "token"
+ settings.require_brainapi = lambda: None
+ client = _Stub(settings)
+ client.search("111 222")
+ self.assertNotIn("target", captured["json"])
+ client.search("111 222", target="jd-u0")
+ self.assertEqual(captured["json"]["target"], "jd-u0")
+ self.assertNotIn("user_id", captured["json"])
+
+ def test_bm25_tokens_present_in_doc_text(self):
+ from search.jdsearch import prepare_jdsearch_rows
+
+ product, behavior = self._fixture_paths()
+ rows, _interactions, _stats = prepare_jdsearch_rows(
+ product_path=product,
+ behavior_path=behavior,
+ extract=False,
+ )
+ docs = [row for row in rows if row.get("type") == "doc"]
+ queries = [row for row in rows if row.get("type") == "query"]
+ self.assertTrue(docs)
+ blob = " ".join(str(doc.get("text") or "") for doc in docs)
+ self.assertIn("DOCID 100.", blob)
+ nonempty = 0
+ for query in queries:
+ tokens = str(query.get("query") or "").split()
+ if any(token and token in blob for token in tokens):
+ nonempty += 1
+ self.assertGreater(nonempty, 0)