An open source, non-profit web search engine
Super Search: gold-grounded source-selection eval (Phase 2)
Daoud Clarke4307a96f7997
diff --git a/.gitignore b/.gitignoreindex dc840b2..6e8bc41 100644--- a/.gitignore+++ b/.gitignore@@ -188,6 +188,14 @@ scripts/downloads/# Super Search candidate classification intermediatescandidate_chunks/+recipe_chunks/++# Super Search data artifacts (inputs/outputs of scripts/, not source)+curated-domains.json+super-search-targets.json+prescreen.json+devdata/ss_eval_matrix*+devdata/ss_gold_matrix*# AI planning documentsplans/diff --git a/mwmbl/rankeval/SUPER_SEARCH_GOLD_SELECTION_FINDINGS.md b/mwmbl/rankeval/SUPER_SEARCH_GOLD_SELECTION_FINDINGS.mdnew file mode 100644index 0000000..6a638c1--- /dev/null+++ b/mwmbl/rankeval/SUPER_SEARCH_GOLD_SELECTION_FINDINGS.md@@ -0,0 +1,129 @@+# Super Search: gold-grounded source selection — findings++_Branch `super-search-intent-feature`, June 2026._++## Why this exists++Source selection (which ~10 of ~134 sources to query per request) has so far been+evaluated against a **circular proxy**: `scripts/super_search_eval.py build-matrix`+rewards a source by the fraction of its results that survive into the LTR model's+*own* top-K (`devdata/ss_eval_matrix.*`). That measures agreement with the ranker,+not real relevance.++This is the Phase-2 step foreshadowed in `scripts/super_search_coverage.py`: build a+**gold-grounded** reward matrix and selector, so we score a source by whether it+actually contains a *gold-relevant* result. The Phase-1 coverage gate already warned+the ceiling is low (~0.36% of gold mass on our source domains); this work measures the+headroom inside that subset with a real selector.++## What was built++- `scripts/super_search_eval.py build-gold-matrix` — fully **offline** (no network).+ Reads the LTR dataset (`devdata/rankeval-2026-04/learning-to-rank.csv.gz`), maps each+ row's URL to its source by registrable domain, and writes a `RewardMatrix`+ (`devdata/ss_gold_matrix.*`) where:+ - `mask[q, s]` = source `s` appears in query `q`'s LTR candidate rows (availability),+ - `R[q, s]` = **1.0 if any of those rows is gold-relevant** (non-null+ `gold_standard_rank`), else 0.0 (binary has-gold).+ - Cosine-feature source profiles are accumulated from the rows' title/extract text.+- Shared domain helpers factored into+ `mwmbl/tinysearchengine/super_search_select/domains.py` (`host_of`, `registrable`,+ `source_domain_map`), reused by `super_search_coverage.py`.+- Evaluation reuses the existing `select` / `simulate` subcommands unchanged.++## The shape of the data++LTR dataset: 252,003 rows / 2,096 queries.++- **In-coverage queries** (≥1 candidate on a source domain): **2,009 (96%)** — most+ queries do touch a source domain (github/stackoverflow/etc. are everywhere).+- **Queries with a gold-bearing source: 56 (2.8%)** — this is the hard ceiling, and it+ matches the gate's ~0.36% gold-mass finding.+- Availability is *not* sparse: mean 6.16 sources/query (median 6, max 23), so top-1/2/3+ selection has genuine room to act.+- Every gold query has **exactly one** gold-bearing source (56 gold cells / 56 queries).++Where the 56 gold queries live (gold-query count / availability):++| source | gold queries | available in |+|---|---:|---:|+| www_gov_uk | 24 | 58 |+| imdb | 15 | 62 |+| github | 5 | 1,772 |+| devblogs_microsoft_com | 5 | 64 |+| genius_com | 3 | 46 |+| duckduckgo_com / nist / space / arxiv | 1 each | — |++The recoverable gold is **concentrated in gov.uk + imdb (39 of 56)** — the two sources+added in `0d10c78` precisely for their gold mass. github is available in 1,772 queries+but is the gold answer in only 5 (very low precision).++## Results++### Held-out selector quality (`select`, XGBoost grouped CV by query, coverage@k vs oracle on the 56 gold queries)++| k | coverage@k | top features (ablation drop) |+|---:|---:|---|+| 1 | 0.464 | estimated_pages +0.232, intent_news +0.089, popularity +0.036 |+| 2 | 0.696 | estimated_pages +0.286, popularity +0.089 |+| 3 | 0.768 | estimated_pages +0.196, popularity +0.054 |++A learned model can place the gold-bearing source in the top-3 ~77% of the time.++### Policy replay (`simulate`, mean captured reward / query over all 2,009; oracle = 0.0279)++| k | oracle | TS (best ν) | cosine | random | popularity |+|---:|---:|---:|---:|---:|---:|+| 1 | 0.0279 | **0.0184** (ν=0.25) | 0.0070 | 0.0050 | 0.0035 |+| 2 | 0.0279 | **0.0249** (ν=0.5) | 0.0124 | 0.0129 | 0.0075 |+| 3 | 0.0279 | **0.0254** (ν=0.5) | 0.0169 | 0.0139 | 0.0090 |++The learned bandit beats the cosine baseline by ~2.6× at k=1 and reaches ~91% of oracle+by k=3.++## Interpretation++1. **The ceiling is tiny — 56 queries (2.8%).** Even a perfect selector moves relevance+ on <3% of queries, with bounded NDCG upside. This corroborates the coverage gate and+ the add-sources track (`SUPER_SEARCH_ADD_SOURCES_FINDINGS.md`).++2. **The production cosine signal is misaligned with gold.** Across every k, the gold+ reward is driven by `estimated_pages` and `popularity` (source size/quality priors),+ while `cos_bow` / `cos_cng` — the features the live `_select_cosine` baseline ranks on+ — have ~zero or **negative** ablation. Query↔source cosine does not predict which+ source holds the gold result.++3. **"Learned beats cosine" is really "always include gov.uk + imdb."** The gold mass is+ carried by gov.uk and imdb, each available in only ~60 queries, and every gold query+ has a single gold source. A learned policy succeeds by learning a near-static per-source+ prior (favour the high-gold sources whenever present) — which is exactly what+ `SUPER_SEARCH_FORCE_INCLUDE` already does for them.++## Recommendation: static, not learned++The data says go static, consistent with the gate's stated default:++- **Force-include the handful of high-gold sources** (gov.uk, imdb already in+ `SUPER_SEARCH_FORCE_INCLUDE`; consider github/devblogs by size prior) rather than+ shipping a per-query learned selector — 56 single-source gold queries do not justify a+ contextual model.+- **If selection is tuned at all, rank by `popularity`/`estimated_pages`, not cosine** —+ the cosine baseline optimizes a signal uncorrelated with gold relevance.+- **Keep investing in adding high-gold-mass domains** (the add-sources track) — that, not+ selection among existing sources, is where the gold mass is.++## Caveats++- 56 gold queries → small CV folds; absolute coverage@k numbers are noisy and+ `estimated_pages` dominance partly reflects that gold-bearing sources happen to be large+ (gov.uk, imdb, github). Directional, not precise.+- Reward is binary has-gold; it ignores *where* in gold the URL ranked. A graded variant+ would refine the within-subset story but not the ceiling.++## Reproduce++```+DJANGO_SETTINGS_MODULE=mwmbl.settings_dev uv run python scripts/super_search_eval.py build-gold-matrix --out devdata/ss_gold_matrix+uv run python scripts/super_search_eval.py select --matrix devdata/ss_gold_matrix --k 2+uv run python scripts/super_search_eval.py simulate --matrix devdata/ss_gold_matrix --k 2+```diff --git a/mwmbl/tinysearchengine/super_search_select/domains.py b/mwmbl/tinysearchengine/super_search_select/domains.pynew file mode 100644index 0000000..bebfe2c--- /dev/null+++ b/mwmbl/tinysearchengine/super_search_select/domains.py@@ -0,0 +1,65 @@+"""URL/host normalization and source-domain matching for Super Search.++Shared by the offline evaluation harnesses (``scripts/super_search_coverage.py``,+``scripts/super_search_eval.py``) that need to decide which Super Search source a+URL belongs to. Matching is by *registrable domain* (``www.``/``m.`` stripped,+last two labels — three for known multi-label suffixes like ``co.uk``); exact-host+string equality is what the NDCG harness scores on, but for "which source contains+this result" the registrable domain is the right granularity.+"""+from __future__ import annotations++from collections import defaultdict+from urllib.parse import urlparse++# Common multi-label public suffixes so registrable-domain folding doesn't collapse+# e.g. bbc.co.uk -> co.uk. Not exhaustive (no full public-suffix list), but covers+# the suffixes that actually appear in the gold set / source shortlist.+_MULTI_SUFFIXES = {+ "co.uk", "org.uk", "ac.uk", "gov.uk", "me.uk", "ltd.uk", "plc.uk",+ "com.au", "net.au", "org.au", "edu.au", "gov.au",+ "co.nz", "co.za", "co.in", "co.jp", "co.kr", "com.br", "com.mx",+ "com.cn", "com.tr", "com.sg", "com.hk", "or.jp", "ne.jp",+}+++def host_of(url: str) -> str:+ """Lowercase host of a URL, sans userinfo and port. ``""`` if unparseable."""+ try:+ netloc = urlparse(str(url)).netloc.lower()+ except ValueError:+ return ""+ return netloc.split("@")[-1].split(":")[0]+++def _strip_www(host: str) -> str:+ for prefix in ("www.", "m."):+ if host.startswith(prefix):+ return host[len(prefix):]+ return host+++def registrable(host: str) -> str:+ """Registrable domain: strip ``www.``/``m.`` then take the last 2 labels (3 for+ known multi-label suffixes like ``co.uk``)."""+ host = _strip_www(host)+ labels = host.split(".")+ if len(labels) <= 2:+ return host+ if ".".join(labels[-2:]) in _MULTI_SUFFIXES:+ return ".".join(labels[-3:])+ return ".".join(labels[-2:])+++def source_domain_map() -> dict[str, list[str]]:+ """Registrable source domain -> source names, built from the source registry.++ A single registrable domain can map to several sources (e.g. sibling+ subdomains of the same registrable domain).+ """+ from mwmbl.tinysearchengine.super_search_select.registry import get_registry++ reg_map: dict[str, list[str]] = defaultdict(list)+ for name, meta in get_registry().items():+ reg_map[registrable(meta.domain.lower())].append(name)+ return dict(reg_map)diff --git a/scripts/super_search_coverage.py b/scripts/super_search_coverage.pyindex 733e12e..14e72ff 100644--- a/scripts/super_search_coverage.py+++ b/scripts/super_search_coverage.py@@ -40,50 +40,13 @@ from mwmbl.rankeval.paths import ( # noqa: E402RANKINGS_DATASET_TEST_PATH,RANKINGS_DATASET_TRAIN_PATH,)+from mwmbl.tinysearchengine.super_search_select.domains import host_of, registrable # noqa: E402from mwmbl.tinysearchengine.super_search_select.features import ( # noqa: E402INTENT_NAMES,classify_intent,)from mwmbl.tinysearchengine.super_search_select.registry import get_registry # noqa: E402-# Common multi-label public suffixes so registrable-domain folding doesn't collapse-# e.g. bbc.co.uk -> co.uk. Not exhaustive (no full public-suffix list), but covers-# the suffixes that actually appear in the gold set / source shortlist.-_MULTI_SUFFIXES = {- "co.uk", "org.uk", "ac.uk", "gov.uk", "me.uk", "ltd.uk", "plc.uk",- "com.au", "net.au", "org.au", "edu.au", "gov.au",- "co.nz", "co.za", "co.in", "co.jp", "co.kr", "com.br", "com.mx",- "com.cn", "com.tr", "com.sg", "com.hk", "or.jp", "ne.jp",-}---def host_of(url: str) -> str:- """Lowercase host of a URL, sans userinfo and port. ``""`` if unparseable."""- try:- netloc = urlparse(str(url)).netloc.lower()- except ValueError:- return ""- return netloc.split("@")[-1].split(":")[0]---def _strip_www(host: str) -> str:- for prefix in ("www.", "m."):- if host.startswith(prefix):- return host[len(prefix):]- return host---def registrable(host: str) -> str:- """Registrable domain: strip ``www.``/``m.`` then take the last 2 labels (3 for- known multi-label suffixes like ``co.uk``)."""- host = _strip_www(host)- labels = host.split(".")- if len(labels) <= 2:- return host- if ".".join(labels[-2:]) in _MULTI_SUFFIXES:- return ".".join(labels[-3:])- return ".".join(labels[-2:])-def load_gold() -> pd.DataFrame:frames = []diff --git a/scripts/super_search_eval.py b/scripts/super_search_eval.pyindex d4c6790..a1ebb99 100644--- a/scripts/super_search_eval.py+++ b/scripts/super_search_eval.py@@ -7,6 +7,14 @@ Subcommands:union with the LTR model, and write a dense (queries x sources)reward + feature matrix. Requires network + Django.+ build-gold-matrix+ - build the same kind of matrix offline from the LTR dataset's+ gold labels (no network): a (query x source) cell is "available"+ (mask) if that source's domain appears in the query's LTR rows,+ and gets reward 1.0 if any of those rows is gold-relevant. Lets+ us learn/evaluate source selection against real gold relevance+ instead of the LTR-top-K-survival proxy.+select - fit the XGBoost reward model on the matrix (grouped CV byquery) and print each feature's ablation drop in coverage@k.@@ -22,8 +30,10 @@ Usage:"""import argparseimport asyncio+import mathimport osimport sys+from collections import defaultdictfrom pathlib import Pathimport django@@ -33,6 +43,12 @@ REPO_ROOT = Path(__file__).resolve().parent.parentsys.path.insert(0, str(REPO_ROOT))os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mwmbl.settings_dev")+# host_of/registrable are pure stdlib helpers (no Django) — safe to import at module+# load, before django.setup(); the registry-backed source_domain_map is loaded lazily.+from mwmbl.tinysearchengine.super_search_select.domains import ( # noqa: E402+ host_of, registrable,+)+def _bootstrap_django():django.setup()@@ -135,6 +151,134 @@ def build_matrix(queries: list[str], out: str):f"{int(mask.sum())} filled cells.")+# ---------------------------------------------------------------------------+# build-gold-matrix (offline: LTR dataset gold labels, no network)+# ---------------------------------------------------------------------------++def _is_gold(rank) -> bool:+ """True if ``rank`` is a real gold rank (not pandas NaN / None / blank)."""+ if rank is None:+ return False+ if isinstance(rank, float) and math.isnan(rank):+ return False+ if isinstance(rank, str) and not rank.strip():+ return False+ return True+++def attribute_rows(rows, reg_map):+ """Attribute LTR rows to Super Search sources by registrable domain (pure).++ ``rows`` is an iterable of ``(query, url, title, extract, gold_standard_rank)``+ and ``reg_map`` maps a registrable domain to the source names on it. Returns+ ``(per_query, prof_text)`` where ``per_query[query][source]`` is True iff that+ source has a gold-relevant row for the query (else False = available-but-not-gold),+ and ``prof_text[source]`` accumulates that source's title/extract text.+ """+ per_query: dict[str, dict[str, bool]] = {} # query -> {source -> has_gold}+ prof_text: dict[str, list[str]] = defaultdict(list)+ for query, url, title, extract, gold in rows:+ names = reg_map.get(registrable(host_of(str(url))))+ if not names:+ continue+ is_gold = _is_gold(gold)+ q_sources = per_query.setdefault(str(query), {})+ text = f"{title or ''} {extract or ''}"+ for name in names:+ q_sources[name] = q_sources.get(name, False) or is_gold+ prof_text[name].append(text)+ return per_query, prof_text+++def build_gold_matrix(out: str):+ """Build a (query x source) reward matrix from the LTR dataset's gold labels.++ Unlike ``build-matrix`` (which queries live sources and rewards a source by how+ many of its results survive the LTR model's *own* top-K), this is fully offline+ and grounds the reward in real gold relevance:++ * mask[q, s] = source ``s``'s registrable domain appears in query ``q``'s LTR+ rows (the offline "this source contributed a candidate" signal);+ * R[q, s] = 1.0 if any such row is gold-relevant (non-null+ ``gold_standard_rank``), else 0.0 (binary has-gold).++ Source content profiles for the cosine features are accumulated from the LTR+ rows' title/extract text per source, so no network access is needed.+ """+ import pandas as pd+ from django.conf import settings+ from mwmbl.rankeval.paths import LEARNING_TO_RANK_DATASET_PATH+ from mwmbl.tinysearchengine.super_search_select import vectors+ from mwmbl.tinysearchengine.super_search_select.domains import source_domain_map+ from mwmbl.tinysearchengine.super_search_select.features import (+ FEATURE_NAMES, QueryContext, feature_vector,+ )+ from mwmbl.tinysearchengine.super_search_select.registry import get_meta, get_registry+ from mwmbl.tinysearchengine.super_search_select.evaluation import RewardMatrix++ dim = settings.SUPER_SEARCH_PROJECTION_DIM+ F = len(FEATURE_NAMES)++ sources = list(get_registry().keys())+ s_index = {name: i for i, name in enumerate(sources)}+ reg_map = source_domain_map() # registrable domain -> [source names]++ df = pd.read_csv(LEARNING_TO_RANK_DATASET_PATH, lineterminator="\n")+ print(f"Loaded LTR dataset: {len(df)} rows, {df['query'].nunique()} queries")++ # Pass 1: attribute every row to its source(s), recording per-(query, source)+ # availability + gold and accumulating per-source content text for the profiles.+ per_query, prof_text = attribute_rows(+ zip(df["query"], df["url"], df["title"], df["extract"], df["gold_standard_rank"]),+ reg_map,+ )++ # In-coverage queries only: those with >=1 in-source candidate row.+ in_cov_queries = list(per_query.keys())+ profile = {+ n: (+ vectors.project_bow(" ".join(prof_text[n]), dim) if prof_text[n] else None,+ vectors.project_char_ngrams(" ".join(prof_text[n]), dim) if prof_text[n] else None,+ )+ for n in sources+ }++ # Pass 2: features + binary has-gold reward.+ Q, S = len(in_cov_queries), len(sources)+ X = np.zeros((Q, S, F))+ R = np.zeros((Q, S))+ mask = np.zeros((Q, S), dtype=bool)+ for qi, query in enumerate(in_cov_queries):+ bow = vectors.project_bow(query, dim)+ cng = vectors.project_char_ngrams(query, dim)+ qctx = QueryContext.build(query, bow, cng)+ for name, has_gold in per_query[query].items():+ si = s_index[name]+ X[qi, si] = feature_vector(qctx, get_meta(name), profile[name])+ mask[qi, si] = True+ if has_gold:+ R[qi, si] = 1.0++ matrix = RewardMatrix(queries=in_cov_queries, sources=sources,+ feature_names=list(FEATURE_NAMES), X=X, R=R, mask=mask)+ matrix.save(out)++ # Diagnostics: does selection even have room to matter? If most queries have+ # few in-source candidates, "query them all" is as good as any learned policy.+ avail_per_q = mask.sum(axis=1)+ gold_per_q = (R > 0).sum(axis=1)+ n_gold_q = int((gold_per_q > 0).sum())+ print(f"\nWrote matrix {out}.npz/.json: {Q} in-coverage queries x {S} sources, "+ f"{int(mask.sum())} filled cells.")+ print(f" queries with >=1 gold source: {n_gold_q} ({100*n_gold_q/max(Q,1):.1f}%)")+ print(f" mean available sources / query: {avail_per_q.mean():.2f} "+ f"(median {int(np.median(avail_per_q))}, max {int(avail_per_q.max())})")+ for k in (1, 2, 3, 5, 10):+ n = int((avail_per_q > k).sum())+ print(f" queries with > {k:>2} available sources: {n:>4} "+ f"({100*n/max(Q,1):.1f}%) <- room for top-{k} selection to matter")++# ---------------------------------------------------------------------------# select / simulate (pure, no network)# ---------------------------------------------------------------------------@@ -175,6 +319,9 @@ def main():p_build.add_argument("--queries", required=True, help="file with one query per line")p_build.add_argument("--out", default="eval_matrix")+ p_gold = sub.add_parser("build-gold-matrix")+ p_gold.add_argument("--out", default="devdata/ss_gold_matrix")+p_select = sub.add_parser("select")p_select.add_argument("--matrix", default="eval_matrix")p_select.add_argument("--k", type=int, default=10)@@ -189,6 +336,8 @@ def main():if args.command == "build-matrix":queries = [ln.strip() for ln in Path(args.queries).read_text().splitlines() if ln.strip()]build_matrix(queries, args.out)+ elif args.command == "build-gold-matrix":+ build_gold_matrix(args.out)elif args.command == "select":cmd_select(args.matrix, args.k)elif args.command == "simulate":diff --git a/test/test_super_search_eval.py b/test/test_super_search_eval.pyindex 3edbb42..52ac0a0 100644--- a/test/test_super_search_eval.py+++ b/test/test_super_search_eval.py@@ -1,12 +1,27 @@"""Tests for the offline evaluation harness on synthetic reward matrices."""+import importlib.util+from pathlib import Path+import numpy as npimport pytestfrom mwmbl.tinysearchengine.super_search_select import evaluation+from mwmbl.tinysearchengine.super_search_select.domains import (+ host_of, registrable, source_domain_map,+)from mwmbl.tinysearchengine.super_search_select.evaluation import RewardMatrixfrom mwmbl.tinysearchengine.super_search_select.features import FEATURE_NAMES+def _load_eval_script():+ """Load scripts/super_search_eval.py by path (it isn't an importable package)."""+ path = Path(__file__).resolve().parents[1] / "scripts" / "super_search_eval.py"+ spec = importlib.util.spec_from_file_location("super_search_eval_script", path)+ mod = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(mod)+ return mod++def _synthetic_matrix(Q=120, S=30, seed=0) -> RewardMatrix:"""Reward driven mostly by cos_bow (idx 1) and popularity (idx 3)."""rng = np.random.default_rng(seed)@@ -82,3 +97,59 @@ def test_feature_selection_flags_cos_bow(tmp_path):drops = result["ablation_drop"]# cos_bow is the dominant driver, so removing it should hurt coverage most.assert drops["cos_bow"] == max(drops.values())+++# ---------------------------------------------------------------------------+# Domain matching (shared helpers behind the gold-grounded matrix)+# ---------------------------------------------------------------------------++def test_registrable_folds_subdomains_and_multi_suffixes():+ assert registrable("www.github.com") == "github.com"+ assert registrable("m.example.org") == "example.org"+ assert registrable("news.ycombinator.com") == "ycombinator.com"+ assert registrable("foo.bbc.co.uk") == "bbc.co.uk" # multi-label suffix kept+ assert registrable("example.com") == "example.com"+++def test_host_of_normalises_and_handles_junk():+ assert host_of("https://[email protected]:443/x") == "news.ycombinator.com"+ assert host_of("not a url") == ""+++def test_source_domain_map_groups_sources_by_registrable_domain():+ m = source_domain_map()+ assert "github" in m.get("github.com", [])+ assert "hn" in m.get("ycombinator.com", []) # news.ycombinator.com -> ycombinator.com+++# ---------------------------------------------------------------------------+# Gold-grounded matrix construction (build-gold-matrix core logic)+# ---------------------------------------------------------------------------++def test_is_gold_treats_nan_none_blank_as_not_gold():+ mod = _load_eval_script()+ assert mod._is_gold(1) and mod._is_gold("3")+ assert not mod._is_gold(None)+ assert not mod._is_gold(float("nan"))+ assert not mod._is_gold("")+ assert not mod._is_gold(" ")+++def test_attribute_rows_binary_gold_and_in_coverage_filter():+ mod = _load_eval_script()+ reg_map = {"github.com": ["github"], "stackoverflow.com": ["stackexchange"]}+ rows = [+ ("q1", "https://github.com/a", "ta", "ea", float("nan")), # available, not gold+ ("q1", "https://github.com/b", "tb", "eb", 3), # gold -> github True+ ("q1", "https://stackoverflow.com/x", "tx", "ex", None), # available, not gold+ ("q2", "https://example.com/none", "t", "e", 1), # off-source -> dropped+ ]+ per_query, prof_text = mod.attribute_rows(rows, reg_map)++ # q2 had no in-source row, so it's filtered out of the in-coverage set.+ assert set(per_query) == {"q1"}+ # binary has-gold: github saw a gold row, stackexchange only non-gold.+ assert per_query["q1"] == {"github": True, "stackexchange": False}+ # both github rows accumulate into the source's content profile text.+ assert len(prof_text["github"]) == 2+ assert len(prof_text["stackexchange"]) == 1