An open source, non-profit web search engine
Super Search: evaluate as a fallback when standard search fails
Daoud Clarkeaef786808655
diff --git a/mwmbl/rankeval/SUPER_SEARCH_ADD_SOURCES_FINDINGS.md b/mwmbl/rankeval/SUPER_SEARCH_ADD_SOURCES_FINDINGS.mdindex 33932b5..6777c9f 100644--- a/mwmbl/rankeval/SUPER_SEARCH_ADD_SOURCES_FINDINGS.md+++ b/mwmbl/rankeval/SUPER_SEARCH_ADD_SOURCES_FINDINGS.md@@ -112,11 +112,56 @@ selection can actually change the ranking.| Δ (new − baseline) | +0.014 | **+0.003** || queries better / worse | 7 / 6 | **16 / 31** |+## Super Search as a fallback (only when standard search fails)++The tables above run Super Search on *every* query. But Super Search is meant to+*rescue* queries where standard search comes up short, not to replace every ranking.+So we gated it: serve standard search, and fall back to Super Search only when+standard returns `<= n` results. Implemented as `FallbackRankingModel` (in+`mwmbl/rankeval/evaluation/evaluate_fallback.py`, unit-tested in+`test/test_fallback_model.py`) and evaluated as extra arms in+`scripts/super_search_new_sources_eval.py` (`--fallback-thresholds`). Standard+search here uses the **remote production index** so the gate fires realistically —+on the tiny local index standard returns `<= 3` for almost every query.++In-coverage test subset, 150 queries, standard = remote index:++| arm | mean NDCG | Δ vs standard | better / worse |+|---|---|---|---|+| standard search (remote) | 0.478 | — | — |+| Super Search baseline (always) | 0.485 | +0.007 | — |+| Super Search + new sources (always) | 0.488 | +0.010 | — |+| fallback@1 → ss-baseline / ss+new | 0.478 | **+0.000** | 0 / 0 |+| fallback@3 → ss-baseline / ss+new | 0.477 / 0.476 | **−0.001 / −0.002** | 0 / 1 |+| fallback@5 → ss-baseline / ss+new | 0.477 / 0.476 | **−0.001 / −0.002** | 0 / 1 |++The fallback fires on 11–19% of queries (n=1→5) and **never once improves over+standard** (better = 0 at every threshold); at n≥3 it is marginally *negative*. On+nearly every fired query standard, ss-baseline and ss+new score *identically*, in two+useless buckets:++- **all score 0.000** (`songs chelmsford`, `mathspad teach`, `ringgo login`, …):+ standard is starved *and* Super Search also fails to surface the gold URL — these+ obscure queries are exactly the ones Super Search can't rescue either.+- **all score 1.000** (`roses film`, `dynamite kiss`, 4 results each): standard+ already nails it with a handful of results — nothing to rescue.++The only fired query where the arms differ is `pudsey leeds` (standard count 3),+where falling back *hurts*: standard 0.631 → ss-baseline 0.431 → ss+new 0.356. That+single regression is the entire n≥3 deficit.++So **result count is the wrong trigger**: it does not correlate with where Super+Search adds value. Super Search's entire aggregate edge (0.488 vs 0.478) comes from+queries where standard returns *many* results but Super Search re-ranks/adds better+ones — precisely the queries the `<= n` gate never fires on. A useful trigger would+have to be a *relevance/score* signal (standard's top result is weak), not a count.+## Conclusion**Adding good sources nets ≈ 0 on NDCG, and Super Search still trails plainstandard search (0.488 vs 0.515) even on the queries where the new domains are-relevant.**+relevant. Gating Super Search behind a standard-search result-count threshold does+not help either — the count does not identify the queries Super Search can rescue.**The mechanism works in isolation — the new sources produce large, real individualwins (gov.uk takes `inheritance tax threshold` and `login universal credit` to@@ -198,6 +243,19 @@ link-follow) per query. Absolute numbers drift run-to-run (live APIs, crawl varithe query *sample* is seeded (`default_rng(0)`) so the same queries are scored each time,and the signal to trust is the paired `ss+new − ss-baseline` delta and the better/worse count.+The same script also evaluates the **fallback** arms (Super Search only when standard+returns `<= n` results). Standard search defaults to the remote production index so the+gate fires realistically; the fallback arms are derived post-hoc from the other arms'+per-query NDCG, so they add no model runs:+```+$E uv run python scripts/super_search_new_sources_eval.py --max-queries 50 \+ --standard-index remote --fallback-thresholds 1 3 5+```+It prints the `fallback@n/...` arms, the fire rate, the paired Δ vs standard-always, and a+per-fired-query dump (standard count + the three arms' NDCG) showing why the gate nets ~0.+`FallbackRankingModel` itself (reusable, unit-tested) lives in+`mwmbl/rankeval/evaluation/evaluate_fallback.py` / `test/test_fallback_model.py`.+**5. Full-sample 3-way comparison (slower, and too diluted to show the effect — for context)**```$E uv run python -m mwmbl.rankeval.evaluation.compare_search_modes --fraction 0.03 --clear-cachediff --git a/mwmbl/rankeval/evaluation/evaluate_fallback.py b/mwmbl/rankeval/evaluation/evaluate_fallback.pynew file mode 100644index 0000000..a34d1aa--- /dev/null+++ b/mwmbl/rankeval/evaluation/evaluate_fallback.py@@ -0,0 +1,208 @@+"""+Evaluate Super Search as a *fallback* for standard search.++Super Search is a heavy pipeline (extra source fan-out, page crawling, outbound-link+following) that, on average, trails plain standard search on the gold set+(see ``SUPER_SEARCH_ADD_SOURCES_FINDINGS.md``). But it was never meant to run on+every query — it is intended to rescue queries where the *standard* Mwmbl search+comes up short. The prior evaluations applied it to all queries and so measured the+wrong thing.++This harness models the intended usage: serve standard search, and only fall back+to Super Search when standard returns ``<= threshold`` results. Standard search is+the full served pipeline (LTR + MMR + up to 3 Wikipedia results) over the **remote+production index** (``api.mwmbl.org``, like ``evaluate_remote.py``) by default — the+tiny local dev index returns ``<= 3`` results for almost every query, which would+make the fallback fire on nearly everything and the experiment meaningless. With the+production index the fallback fires only on genuine standard-search failures, which+is the whole point. Pass ``--local`` to score against the local index instead.++Caveat: when the fallback fires, Super Search's own ``mwmbl_index`` source still+queries the *local* index (that is what the offline pipeline has loaded), so Super+Search's in-index contribution is understated. But the fallback only fires when the+*production* index has already come up short, so on those queries the local index+would almost certainly fail too — the value Super Search adds there is its extra+sources, crawling and link-following, which the offline pipeline does run for real.++When the fallback fires we *replace* the ranking with Super Search's, rather than+merging: Super Search's pool already contains the mwmbl-index source and the same+Wikipedia augmentation, so its ranking subsumes standard's inputs. This matches what+production would actually show (swap the SERP for the Super Search SERP).++For each threshold we report overall NDCG (directly comparable to the standard and+always-Super-Search baselines on the same sampled queries) plus the diagnostic that+matters: how often the fallback fires, and whether Super Search beats standard *on+the subset where it fires* — that subset is the entire case for the feature.++Usage::++ DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \\+ uv run python -m mwmbl.rankeval.evaluation.evaluate_fallback --fraction 0.05+"""+import os+from argparse import ArgumentParser++import django+import numpy as np+import pandas as pd+from scipy.stats import sem+from sklearn.metrics import ndcg_score++os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mwmbl.settings_dev")+django.setup()++import mwmbl.rankeval.evaluation.evaluate_super_search as ss_module # noqa: E402+from mwmbl.rankeval.evaluation.evaluate import ( # noqa: E402+ CLICK_PROPORTIONS, NUM_RESULTS_FOR_EVAL, RankingModel)+from mwmbl.rankeval.evaluation.evaluate_ranker import DummyCompleter, MwmblRankingModel # noqa: E402+from mwmbl.rankeval.evaluation.evaluate_super_search import SuperSearchRankingModel # noqa: E402+from mwmbl.rankeval.evaluation.remote_index import RemoteIndex # noqa: E402+from mwmbl.rankeval.paths import ( # noqa: E402+ RANKINGS_DATASET_TEST_PATH, RANKINGS_DATASET_TRAIN_PATH, RUST_MODEL_PATH)+from mwmbl.tinysearchengine.ltr import RustXGBPipeline # noqa: E402+from mwmbl.tinysearchengine.ltr_rank import LTRRanker # noqa: E402+from mwmbl.tinysearchengine.mmr_rank import MMRRanker # noqa: E402+++def _standard_model(use_local: bool) -> MwmblRankingModel:+ """Standard search: LTR + MMR + 3 wiki results, over remote or local index.++ Remote (default) hits the production index at ``api.mwmbl.org`` so the fallback+ fires at a realistic rate; ``--local`` reuses the dev index from ``search_setup``.+ """+ if use_local:+ from mwmbl.search_setup import ranker # local index + LTR + MMR + wiki+ return MwmblRankingModel(ranker)+ model = RustXGBPipeline.from_model_path(str(RUST_MODEL_PATH))+ ranker = MMRRanker(LTRRanker(RemoteIndex(), DummyCompleter(), model, True, 3))+ return MwmblRankingModel(ranker)+++class FallbackRankingModel(RankingModel):+ """Serve ``primary``; fall back to ``fallback`` only when primary is starved.++ If the primary model returns ``<= threshold`` results for a query, the fallback+ model's ranking replaces it; otherwise the primary ranking is served unchanged.+ Reusable for any pair of ``RankingModel`` s, not just standard vs Super Search.+ """++ def __init__(self, primary: RankingModel, fallback: RankingModel, threshold: int):+ self.primary = primary+ self.fallback = fallback+ self.threshold = threshold++ def predict(self, query: str) -> list[str]:+ primary_results = self.primary.predict(query)+ if len(primary_results) <= self.threshold:+ return self.fallback.predict(query)+ return primary_results+++def query_ndcg(predicted_urls: list[str], gold_scores: dict[str, float]) -> float:+ """NDCG@10 of a predicted ranking against gold, matching ``evaluate.evaluate``.++ ``gold_scores`` maps each gold URL to its click-proportion relevance weight.+ """+ top_urls = predicted_urls[:NUM_RESULTS_FOR_EVAL]+ y_true = [gold_scores.get(url, 0.0) for url in top_urls] + [0.0] * (10 - len(top_urls))+ y_predicted = list(range(NUM_RESULTS_FOR_EVAL, 0, -1))+ return ndcg_score([y_true], [y_predicted])+++def _mean_sem(values: list[float]) -> str:+ if not values:+ return " n/a"+ return f"{np.mean(values):.4f} ± {sem(values):.4f}" if len(values) > 1 else f"{np.mean(values):.4f}"+++def run():+ parser = ArgumentParser()+ parser.add_argument("--fraction", type=float, default=0.05,+ help="Fraction of gold queries to sample (Super Search is slow).")+ parser.add_argument("--thresholds", type=int, nargs="+", default=[1, 3, 5],+ help="Fallback when standard returns <= threshold results.")+ parser.add_argument("--train", action="store_true",+ help="Evaluate on the train split instead of test.")+ parser.add_argument("--local", action="store_true",+ help="Use the local dev index for standard search instead of "+ "the remote production index (fallback then fires on almost "+ "every query — for debugging only).")+ parser.add_argument("--fired-only", action="store_true",+ help="Only run Super Search on queries where the fallback fires "+ "(faster, but omits the always-Super-Search comparison arm).")+ parser.add_argument("--clear-cache", action="store_true",+ help="Clear the Super Search doc-pool cache before evaluating.")+ args = parser.parse_args()++ if args.clear_cache:+ ss_module.memory.clear(warn=False)++ print("Standard search index:", "local dev" if args.local else "remote (api.mwmbl.org)")+ standard = _standard_model(use_local=args.local)+ super_search = SuperSearchRankingModel()++ path = RANKINGS_DATASET_TRAIN_PATH if args.train else RANKINGS_DATASET_TEST_PATH+ print("Evaluating against dataset", path)+ dataset = pd.read_csv(path)++ queries = dataset["query"].unique()+ rng = np.random.default_rng(42) # same seed/sampling as evaluate.evaluate+ if args.fraction < 1.0:+ num_queries = int(args.fraction * len(queries))+ sampled = set(rng.choice(queries, num_queries, replace=False))+ else:+ sampled = set(queries)+ print("Num queries", len(sampled))++ # Per-query: gold scores, standard NDCG + result count, and Super Search NDCG.+ # By default Super Search is run on every sampled query so we can report the+ # always-Super-Search arm (directly comparable to compare_search_modes.py); with+ # --fired-only it is computed just for queries where the fallback fires.+ max_threshold = max(args.thresholds)+ standard_ndcg: dict[str, float] = {}+ standard_count: dict[str, int] = {}+ super_ndcg: dict[str, float] = {}++ for query, rankings in dataset.groupby("query"):+ if query not in sampled:+ continue+ top_ranked = rankings[["url"]].iloc[:NUM_RESULTS_FOR_EVAL].copy()+ top_ranked["score"] = CLICK_PROPORTIONS[:len(top_ranked)]+ gold_scores = top_ranked.set_index("url")["score"].to_dict()++ std_results = standard.predict(query)+ standard_count[query] = len(std_results)+ standard_ndcg[query] = query_ndcg(std_results, gold_scores)++ if not args.fired_only or len(std_results) <= max_threshold:+ super_ndcg[query] = query_ndcg(super_search.predict(query), gold_scores)++ # Headline arms over all sampled queries.+ print(f"\n{'=' * 78}")+ print("Standard search (always): NDCG =", _mean_sem(list(standard_ndcg.values())))+ if not args.fired_only:+ print("Super Search (always): NDCG =", _mean_sem(list(super_ndcg.values())))+ print(f"{'=' * 78}")++ print(f"\n{'thresh':>6} {'fired':>12} {'fallback NDCG (all)':>22} "+ f"{'fired-subset std':>17} {'fired-subset super':>18} {'Δ on fired':>11}")+ for threshold in sorted(args.thresholds):+ fired = [q for q, c in standard_count.items() if c <= threshold]+ fallback_all = [+ super_ndcg[q] if standard_count[q] <= threshold else standard_ndcg[q]+ for q in standard_ndcg+ ]+ fired_std = [standard_ndcg[q] for q in fired]+ fired_super = [super_ndcg[q] for q in fired]+ delta = (np.mean(fired_super) - np.mean(fired_std)) if fired else float("nan")+ n_total = len(standard_ndcg)+ print(f"{threshold:>6} {len(fired):>5}/{n_total:<6} {_mean_sem(fallback_all):>22} "+ f"{_mean_sem(fired_std):>17} {_mean_sem(fired_super):>18} {delta:>+11.4f}")++ print("\nfired-subset Δ = mean(Super Search NDCG − standard NDCG) on queries where the "+ "fallback fires;\npositive means falling back to Super Search helps where standard "+ "is starved.")+++if __name__ == "__main__":+ run()diff --git a/scripts/super_search_new_sources_eval.py b/scripts/super_search_new_sources_eval.pyindex 71fdd93..eaed2d8 100644--- a/scripts/super_search_new_sources_eval.py+++ b/scripts/super_search_new_sources_eval.py@@ -8,6 +8,20 @@ source's domain — where source selection can actually change the ranking. Forsuch query it scores three arms (standard / SS baseline / SS + new sources) withthe same gold→NDCG method as rankeval.evaluation.evaluate.+It also evaluates Super Search as a *fallback*: Super Search is only invoked when+standard search comes up short (returns ``<= --fallback-threshold`` results),+otherwise the standard ranking is kept. This models the intended production usage —+Super Search rescues queries where standard search fails — rather than replacing+every ranking. The fallback arms are derived post-hoc from the per-query results of+the other arms (``fallback@n`` for a query = the Super Search NDCG when standard+returns ``<= n`` results, else the standard NDCG), so they add no extra model runs.++Standard search uses the **remote** production index (``api.mwmbl.org``) by default,+not the tiny local dev index: the gate counts standard results, and the local index+returns ``<= 3`` for almost every in-coverage query (their answers live on external+domains it does not hold), which would make the fallback fire on everything. Pass+``--standard-index local`` to reproduce the original local-index standard arm.+Run:DATABASE_URL="postgres://daoud@" DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \uv run python scripts/super_search_new_sources_eval.py --max-queries 80@@ -31,10 +45,9 @@ from django.conf import settings # noqa: E402from sklearn.metrics import ndcg_score # noqa: E402from mwmbl.rankeval.evaluation.evaluate import CLICK_PROPORTIONS, NUM_RESULTS_FOR_EVAL # noqa: E402-from mwmbl.rankeval.evaluation.evaluate_ranker import MwmblRankingModel # noqa: E402+from mwmbl.rankeval.evaluation.evaluate_fallback import _standard_model # noqa: E402from mwmbl.rankeval.evaluation.evaluate_super_search import SuperSearchRankingModel # noqa: E402from mwmbl.rankeval.paths import RANKINGS_DATASET_TEST_PATH # noqa: E402-from mwmbl.search_setup import ranker # noqa: E402from mwmbl.tinysearchengine.super_search_select.registry import get_meta # noqa: E402NEW_SOURCES = ["www_gov_uk", "imdb"]@@ -65,6 +78,12 @@ def _ndcg(predicted_urls: list[str], gold_scores: dict[str, float]) -> float:def main():parser = argparse.ArgumentParser()parser.add_argument("--max-queries", type=int, default=80)+ parser.add_argument("--standard-index", choices=["remote", "local"], default="remote",+ help="Index for the standard-search arm and the fallback gate "+ "(default remote = api.mwmbl.org production index).")+ parser.add_argument("--fallback-thresholds", type=int, nargs="+", default=[1, 3, 5],+ help="Fall back to Super Search when standard returns <= threshold "+ "results. Evaluated post-hoc for each value.")args = parser.parse_args()df = pd.read_csv(RANKINGS_DATASET_TEST_PATH)@@ -86,12 +105,14 @@ def main():rows["score"] = CLICK_PROPORTIONS[:len(rows)]gold[q] = rows.set_index("url")["score"].to_dict()+ print(f"standard-search index: {args.standard_index}")arms = [- ("standard", lambda: MwmblRankingModel(ranker), None),+ ("standard", lambda: _standard_model(use_local=args.standard_index == "local"), None),("ss-baseline", lambda: SuperSearchRankingModel(selection_key=""), []),("ss+new", lambda: SuperSearchRankingModel(selection_key="+".join(NEW_SOURCES)), NEW_SOURCES),]per_query = {name: {} for name, _, _ in arms}+ standard_count = {} # number of results standard search returned (the fallback gate)for name, make_model, force in arms:if force is not None:settings.SUPER_SEARCH_FORCE_INCLUDE = force@@ -100,15 +121,30 @@ def main():for i, q in enumerate(queries):preds = model.predict(q)per_query[name][q] = _ndcg(preds, gold[q])+ if name == "standard":+ standard_count[q] = len(preds)if (i + 1) % 20 == 0:print(f" {i + 1}/{len(queries)}")settings.SUPER_SEARCH_FORCE_INCLUDE = []+ # Fallback arms, derived post-hoc: keep standard unless it returned <= n results,+ # in which case fall back to the corresponding always-Super-Search arm.+ for n in sorted(args.fallback_thresholds):+ for ss_name in ("ss-baseline", "ss+new"):+ per_query[f"fallback@{n}/{ss_name}"] = {+ q: (per_query[ss_name][q] if standard_count[q] <= n else per_query["standard"][q])+ for q in queries+ }+print("\n" + "=" * 60)- print(f"{'arm':14} {'mean NDCG':>10}")- for name, _, _ in arms:- vals = list(per_query[name].values())- print(f"{name:14} {np.mean(vals):>10.4f}")+ print(f"{'arm':24} {'mean NDCG':>10}")+ for name in per_query:+ print(f"{name:24} {np.mean(list(per_query[name].values())):>10.4f}")++ print(f"\nfallback fire rate (standard <= n results), of {len(queries)} queries:")+ for n in sorted(args.fallback_thresholds):+ fired = sum(c <= n for c in standard_count.values())+ print(f" n={n}: {fired:3d} fired ({fired / len(queries):.0%})")base = per_query["ss-baseline"]new = per_query["ss+new"]@@ -117,11 +153,28 @@ def main():delta = np.mean([new[q] - base[q] for q in queries])print(f"\nss+new vs ss-baseline: mean Δ {delta:+.4f} | better {wins}, worse {losses}, "f"same {len(queries) - wins - losses}")- movers = sorted(queries, key=lambda q: new[q] - base[q], reverse=True)- print("\ntop NDCG gains (ss+new over baseline):")- for q in movers[:8]:- if new[q] - base[q] > 1e-9:- print(f" {new[q] - base[q]:+.3f} {q!r}")++ # The headline fallback question: does gating Super Search behind standard-search+ # failure beat serving standard search alone, on this in-coverage subset?+ std = per_query["standard"]+ print("\nfallback vs standard-always (paired, same queries):")+ for n in sorted(args.fallback_thresholds):+ for ss_name in ("ss-baseline", "ss+new"):+ fb = per_query[f"fallback@{n}/{ss_name}"]+ d = np.mean([fb[q] - std[q] for q in queries])+ w = sum(fb[q] > std[q] + 1e-9 for q in queries)+ l = sum(fb[q] < std[q] - 1e-9 for q in queries)+ print(f" fallback@{n}/{ss_name:11} mean Δ {d:+.4f} | better {w}, worse {l}")++ # Why the fallback nets ~0: show, for the queries it fires on, whether Super+ # Search actually rescues them. If standard and Super Search both score ~0 on the+ # starved queries, the result-count gate is selecting queries SS cannot help.+ nmax = max(args.fallback_thresholds)+ fired = sorted([q for q in queries if standard_count[q] <= nmax],+ key=lambda q: standard_count[q])+ print(f"\nfired queries at n={nmax} (standard count, NDCG: standard / ss-baseline / ss+new):")+ for q in fired:+ print(f" cnt={standard_count[q]:2d} {std[q]:.3f} / {base[q]:.3f} / {new[q]:.3f} {q!r}")if __name__ == "__main__":diff --git a/test/test_fallback_model.py b/test/test_fallback_model.pynew file mode 100644index 0000000..131a01f--- /dev/null+++ b/test/test_fallback_model.py@@ -0,0 +1,52 @@+"""Trigger logic for the Super-Search-as-fallback ranking model (no network)."""+from mwmbl.rankeval.evaluation.evaluate import RankingModel+++class _FixedModel(RankingModel):+ def __init__(self, urls, calls=None):+ self.urls = urls+ self.calls = calls # optional list to record invocations++ def predict(self, query):+ if self.calls is not None:+ self.calls.append(query)+ return list(self.urls)+++def _fallback(primary_urls, fallback_urls, threshold, fb_calls=None):+ # Imported lazily so the module's django.setup()/search_setup import cost is only+ # paid when these tests actually run.+ from mwmbl.rankeval.evaluation.evaluate_fallback import FallbackRankingModel+ return FallbackRankingModel(+ _FixedModel(primary_urls),+ _FixedModel(fallback_urls, calls=fb_calls),+ threshold=threshold,+ )+++def test_serves_primary_when_above_threshold():+ primary = ["a", "b", "c", "d"]+ model = _fallback(primary, ["x", "y"], threshold=3)+ assert model.predict("q") == primary+++def test_falls_back_when_at_or_below_threshold():+ model = _fallback(["a", "b", "c"], ["x", "y"], threshold=3)+ assert model.predict("q") == ["x", "y"]+++def test_falls_back_on_empty_primary():+ model = _fallback([], ["x", "y"], threshold=3)+ assert model.predict("q") == ["x", "y"]+++def test_fallback_not_invoked_when_primary_sufficient():+ fb_calls: list[str] = []+ model = _fallback(["a", "b", "c", "d"], ["x"], threshold=3, fb_calls=fb_calls)+ model.predict("q")+ assert fb_calls == [] # the expensive Super Search pipeline is never run+++def test_threshold_zero_only_fires_on_empty():+ assert _fallback(["a"], ["x"], threshold=0).predict("q") == ["a"]+ assert _fallback([], ["x"], threshold=0).predict("q") == ["x"]