Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Super Search: evaluate as a fallback when standard search fails

Daoud Clarkeaef786808655
diff --git a/mwmbl/rankeval/SUPER_SEARCH_ADD_SOURCES_FINDINGS.md b/mwmbl/rankeval/SUPER_SEARCH_ADD_SOURCES_FINDINGS.md
index 33932b5..6777c9f 100644
--- a/mwmbl/rankeval/SUPER_SEARCH_ADD_SOURCES_FINDINGS.md
+++ b/mwmbl/rankeval/SUPER_SEARCH_ADD_SOURCES_FINDINGS.md
@@ -112,11 +112,56 @@ selection can actually change the ranking.
| Δ (new − baseline) | +0.014 | **+0.003** |
| queries better / worse | 7 / 6 | **16 / 31** |
+## Super Search as a fallback (only when standard search fails)
+
+The tables above run Super Search on *every* query. But Super Search is meant to
+*rescue* queries where standard search comes up short, not to replace every ranking.
+So we gated it: serve standard search, and fall back to Super Search only when
+standard returns `<= n` results. Implemented as `FallbackRankingModel` (in
+`mwmbl/rankeval/evaluation/evaluate_fallback.py`, unit-tested in
+`test/test_fallback_model.py`) and evaluated as extra arms in
+`scripts/super_search_new_sources_eval.py` (`--fallback-thresholds`). Standard
+search here uses the **remote production index** so the gate fires realistically —
+on the tiny local index standard returns `<= 3` for almost every query.
+
+In-coverage test subset, 150 queries, standard = remote index:
+
+| arm | mean NDCG | Δ vs standard | better / worse |
+|---|---|---|---|
+| standard search (remote) | 0.478 | — | — |
+| Super Search baseline (always) | 0.485 | +0.007 | — |
+| Super Search + new sources (always) | 0.488 | +0.010 | — |
+| fallback@1 → ss-baseline / ss+new | 0.478 | **+0.000** | 0 / 0 |
+| fallback@3 → ss-baseline / ss+new | 0.477 / 0.476 | **−0.001 / −0.002** | 0 / 1 |
+| fallback@5 → ss-baseline / ss+new | 0.477 / 0.476 | **−0.001 / −0.002** | 0 / 1 |
+
+The fallback fires on 11–19% of queries (n=1→5) and **never once improves over
+standard** (better = 0 at every threshold); at n≥3 it is marginally *negative*. On
+nearly every fired query standard, ss-baseline and ss+new score *identically*, in two
+useless buckets:
+
+- **all score 0.000** (`songs chelmsford`, `mathspad teach`, `ringgo login`, …):
+ standard is starved *and* Super Search also fails to surface the gold URL — these
+ obscure queries are exactly the ones Super Search can't rescue either.
+- **all score 1.000** (`roses film`, `dynamite kiss`, 4 results each): standard
+ already nails it with a handful of results — nothing to rescue.
+
+The only fired query where the arms differ is `pudsey leeds` (standard count 3),
+where falling back *hurts*: standard 0.631 → ss-baseline 0.431 → ss+new 0.356. That
+single regression is the entire n≥3 deficit.
+
+So **result count is the wrong trigger**: it does not correlate with where Super
+Search adds value. Super Search's entire aggregate edge (0.488 vs 0.478) comes from
+queries where standard returns *many* results but Super Search re-ranks/adds better
+ones — precisely the queries the `<= n` gate never fires on. A useful trigger would
+have to be a *relevance/score* signal (standard's top result is weak), not a count.
+
## Conclusion
**Adding good sources nets ≈ 0 on NDCG, and Super Search still trails plain
standard search (0.488 vs 0.515) even on the queries where the new domains are
-relevant.**
+relevant. Gating Super Search behind a standard-search result-count threshold does
+not help either — the count does not identify the queries Super Search can rescue.**
The mechanism works in isolation — the new sources produce large, real individual
wins (gov.uk takes `inheritance tax threshold` and `login universal credit` to
@@ -198,6 +243,19 @@ link-follow) per query. Absolute numbers drift run-to-run (live APIs, crawl vari
the query *sample* is seeded (`default_rng(0)`) so the same queries are scored each time,
and the signal to trust is the paired `ss+new − ss-baseline` delta and the better/worse count.
+The same script also evaluates the **fallback** arms (Super Search only when standard
+returns `<= n` results). Standard search defaults to the remote production index so the
+gate fires realistically; the fallback arms are derived post-hoc from the other arms'
+per-query NDCG, so they add no model runs:
+```
+$E uv run python scripts/super_search_new_sources_eval.py --max-queries 50 \
+ --standard-index remote --fallback-thresholds 1 3 5
+```
+It prints the `fallback@n/...` arms, the fire rate, the paired Δ vs standard-always, and a
+per-fired-query dump (standard count + the three arms' NDCG) showing why the gate nets ~0.
+`FallbackRankingModel` itself (reusable, unit-tested) lives in
+`mwmbl/rankeval/evaluation/evaluate_fallback.py` / `test/test_fallback_model.py`.
+
**5. Full-sample 3-way comparison (slower, and too diluted to show the effect — for context)**
```
$E uv run python -m mwmbl.rankeval.evaluation.compare_search_modes --fraction 0.03 --clear-cache
diff --git a/mwmbl/rankeval/evaluation/evaluate_fallback.py b/mwmbl/rankeval/evaluation/evaluate_fallback.py
new file mode 100644
index 0000000..a34d1aa
--- /dev/null
+++ b/mwmbl/rankeval/evaluation/evaluate_fallback.py
@@ -0,0 +1,208 @@
+"""
+Evaluate Super Search as a *fallback* for standard search.
+
+Super Search is a heavy pipeline (extra source fan-out, page crawling, outbound-link
+following) that, on average, trails plain standard search on the gold set
+(see ``SUPER_SEARCH_ADD_SOURCES_FINDINGS.md``). But it was never meant to run on
+every query — it is intended to rescue queries where the *standard* Mwmbl search
+comes up short. The prior evaluations applied it to all queries and so measured the
+wrong thing.
+
+This harness models the intended usage: serve standard search, and only fall back
+to Super Search when standard returns ``<= threshold`` results. Standard search is
+the full served pipeline (LTR + MMR + up to 3 Wikipedia results) over the **remote
+production index** (``api.mwmbl.org``, like ``evaluate_remote.py``) by default — the
+tiny local dev index returns ``<= 3`` results for almost every query, which would
+make the fallback fire on nearly everything and the experiment meaningless. With the
+production index the fallback fires only on genuine standard-search failures, which
+is the whole point. Pass ``--local`` to score against the local index instead.
+
+Caveat: when the fallback fires, Super Search's own ``mwmbl_index`` source still
+queries the *local* index (that is what the offline pipeline has loaded), so Super
+Search's in-index contribution is understated. But the fallback only fires when the
+*production* index has already come up short, so on those queries the local index
+would almost certainly fail too — the value Super Search adds there is its extra
+sources, crawling and link-following, which the offline pipeline does run for real.
+
+When the fallback fires we *replace* the ranking with Super Search's, rather than
+merging: Super Search's pool already contains the mwmbl-index source and the same
+Wikipedia augmentation, so its ranking subsumes standard's inputs. This matches what
+production would actually show (swap the SERP for the Super Search SERP).
+
+For each threshold we report overall NDCG (directly comparable to the standard and
+always-Super-Search baselines on the same sampled queries) plus the diagnostic that
+matters: how often the fallback fires, and whether Super Search beats standard *on
+the subset where it fires* — that subset is the entire case for the feature.
+
+Usage::
+
+ DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \\
+ uv run python -m mwmbl.rankeval.evaluation.evaluate_fallback --fraction 0.05
+"""
+import os
+from argparse import ArgumentParser
+
+import django
+import numpy as np
+import pandas as pd
+from scipy.stats import sem
+from sklearn.metrics import ndcg_score
+
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mwmbl.settings_dev")
+django.setup()
+
+import mwmbl.rankeval.evaluation.evaluate_super_search as ss_module # noqa: E402
+from mwmbl.rankeval.evaluation.evaluate import ( # noqa: E402
+ CLICK_PROPORTIONS, NUM_RESULTS_FOR_EVAL, RankingModel)
+from mwmbl.rankeval.evaluation.evaluate_ranker import DummyCompleter, MwmblRankingModel # noqa: E402
+from mwmbl.rankeval.evaluation.evaluate_super_search import SuperSearchRankingModel # noqa: E402
+from mwmbl.rankeval.evaluation.remote_index import RemoteIndex # noqa: E402
+from mwmbl.rankeval.paths import ( # noqa: E402
+ RANKINGS_DATASET_TEST_PATH, RANKINGS_DATASET_TRAIN_PATH, RUST_MODEL_PATH)
+from mwmbl.tinysearchengine.ltr import RustXGBPipeline # noqa: E402
+from mwmbl.tinysearchengine.ltr_rank import LTRRanker # noqa: E402
+from mwmbl.tinysearchengine.mmr_rank import MMRRanker # noqa: E402
+
+
+def _standard_model(use_local: bool) -> MwmblRankingModel:
+ """Standard search: LTR + MMR + 3 wiki results, over remote or local index.
+
+ Remote (default) hits the production index at ``api.mwmbl.org`` so the fallback
+ fires at a realistic rate; ``--local`` reuses the dev index from ``search_setup``.
+ """
+ if use_local:
+ from mwmbl.search_setup import ranker # local index + LTR + MMR + wiki
+ return MwmblRankingModel(ranker)
+ model = RustXGBPipeline.from_model_path(str(RUST_MODEL_PATH))
+ ranker = MMRRanker(LTRRanker(RemoteIndex(), DummyCompleter(), model, True, 3))
+ return MwmblRankingModel(ranker)
+
+
+class FallbackRankingModel(RankingModel):
+ """Serve ``primary``; fall back to ``fallback`` only when primary is starved.
+
+ If the primary model returns ``<= threshold`` results for a query, the fallback
+ model's ranking replaces it; otherwise the primary ranking is served unchanged.
+ Reusable for any pair of ``RankingModel`` s, not just standard vs Super Search.
+ """
+
+ def __init__(self, primary: RankingModel, fallback: RankingModel, threshold: int):
+ self.primary = primary
+ self.fallback = fallback
+ self.threshold = threshold
+
+ def predict(self, query: str) -> list[str]:
+ primary_results = self.primary.predict(query)
+ if len(primary_results) <= self.threshold:
+ return self.fallback.predict(query)
+ return primary_results
+
+
+def query_ndcg(predicted_urls: list[str], gold_scores: dict[str, float]) -> float:
+ """NDCG@10 of a predicted ranking against gold, matching ``evaluate.evaluate``.
+
+ ``gold_scores`` maps each gold URL to its click-proportion relevance weight.
+ """
+ top_urls = predicted_urls[:NUM_RESULTS_FOR_EVAL]
+ y_true = [gold_scores.get(url, 0.0) for url in top_urls] + [0.0] * (10 - len(top_urls))
+ y_predicted = list(range(NUM_RESULTS_FOR_EVAL, 0, -1))
+ return ndcg_score([y_true], [y_predicted])
+
+
+def _mean_sem(values: list[float]) -> str:
+ if not values:
+ return " n/a"
+ return f"{np.mean(values):.4f} ± {sem(values):.4f}" if len(values) > 1 else f"{np.mean(values):.4f}"
+
+
+def run():
+ parser = ArgumentParser()
+ parser.add_argument("--fraction", type=float, default=0.05,
+ help="Fraction of gold queries to sample (Super Search is slow).")
+ parser.add_argument("--thresholds", type=int, nargs="+", default=[1, 3, 5],
+ help="Fallback when standard returns <= threshold results.")
+ parser.add_argument("--train", action="store_true",
+ help="Evaluate on the train split instead of test.")
+ parser.add_argument("--local", action="store_true",
+ help="Use the local dev index for standard search instead of "
+ "the remote production index (fallback then fires on almost "
+ "every query — for debugging only).")
+ parser.add_argument("--fired-only", action="store_true",
+ help="Only run Super Search on queries where the fallback fires "
+ "(faster, but omits the always-Super-Search comparison arm).")
+ parser.add_argument("--clear-cache", action="store_true",
+ help="Clear the Super Search doc-pool cache before evaluating.")
+ args = parser.parse_args()
+
+ if args.clear_cache:
+ ss_module.memory.clear(warn=False)
+
+ print("Standard search index:", "local dev" if args.local else "remote (api.mwmbl.org)")
+ standard = _standard_model(use_local=args.local)
+ super_search = SuperSearchRankingModel()
+
+ path = RANKINGS_DATASET_TRAIN_PATH if args.train else RANKINGS_DATASET_TEST_PATH
+ print("Evaluating against dataset", path)
+ dataset = pd.read_csv(path)
+
+ queries = dataset["query"].unique()
+ rng = np.random.default_rng(42) # same seed/sampling as evaluate.evaluate
+ if args.fraction < 1.0:
+ num_queries = int(args.fraction * len(queries))
+ sampled = set(rng.choice(queries, num_queries, replace=False))
+ else:
+ sampled = set(queries)
+ print("Num queries", len(sampled))
+
+ # Per-query: gold scores, standard NDCG + result count, and Super Search NDCG.
+ # By default Super Search is run on every sampled query so we can report the
+ # always-Super-Search arm (directly comparable to compare_search_modes.py); with
+ # --fired-only it is computed just for queries where the fallback fires.
+ max_threshold = max(args.thresholds)
+ standard_ndcg: dict[str, float] = {}
+ standard_count: dict[str, int] = {}
+ super_ndcg: dict[str, float] = {}
+
+ for query, rankings in dataset.groupby("query"):
+ if query not in sampled:
+ continue
+ top_ranked = rankings[["url"]].iloc[:NUM_RESULTS_FOR_EVAL].copy()
+ top_ranked["score"] = CLICK_PROPORTIONS[:len(top_ranked)]
+ gold_scores = top_ranked.set_index("url")["score"].to_dict()
+
+ std_results = standard.predict(query)
+ standard_count[query] = len(std_results)
+ standard_ndcg[query] = query_ndcg(std_results, gold_scores)
+
+ if not args.fired_only or len(std_results) <= max_threshold:
+ super_ndcg[query] = query_ndcg(super_search.predict(query), gold_scores)
+
+ # Headline arms over all sampled queries.
+ print(f"\n{'=' * 78}")
+ print("Standard search (always): NDCG =", _mean_sem(list(standard_ndcg.values())))
+ if not args.fired_only:
+ print("Super Search (always): NDCG =", _mean_sem(list(super_ndcg.values())))
+ print(f"{'=' * 78}")
+
+ print(f"\n{'thresh':>6} {'fired':>12} {'fallback NDCG (all)':>22} "
+ f"{'fired-subset std':>17} {'fired-subset super':>18} {'Δ on fired':>11}")
+ for threshold in sorted(args.thresholds):
+ fired = [q for q, c in standard_count.items() if c <= threshold]
+ fallback_all = [
+ super_ndcg[q] if standard_count[q] <= threshold else standard_ndcg[q]
+ for q in standard_ndcg
+ ]
+ fired_std = [standard_ndcg[q] for q in fired]
+ fired_super = [super_ndcg[q] for q in fired]
+ delta = (np.mean(fired_super) - np.mean(fired_std)) if fired else float("nan")
+ n_total = len(standard_ndcg)
+ print(f"{threshold:>6} {len(fired):>5}/{n_total:<6} {_mean_sem(fallback_all):>22} "
+ f"{_mean_sem(fired_std):>17} {_mean_sem(fired_super):>18} {delta:>+11.4f}")
+
+ print("\nfired-subset Δ = mean(Super Search NDCG − standard NDCG) on queries where the "
+ "fallback fires;\npositive means falling back to Super Search helps where standard "
+ "is starved.")
+
+
+if __name__ == "__main__":
+ run()
diff --git a/scripts/super_search_new_sources_eval.py b/scripts/super_search_new_sources_eval.py
index 71fdd93..eaed2d8 100644
--- a/scripts/super_search_new_sources_eval.py
+++ b/scripts/super_search_new_sources_eval.py
@@ -8,6 +8,20 @@ source's domain — where source selection can actually change the ranking. For
such query it scores three arms (standard / SS baseline / SS + new sources) with
the same gold→NDCG method as rankeval.evaluation.evaluate.
+It also evaluates Super Search as a *fallback*: Super Search is only invoked when
+standard search comes up short (returns ``<= --fallback-threshold`` results),
+otherwise the standard ranking is kept. This models the intended production usage —
+Super Search rescues queries where standard search fails — rather than replacing
+every ranking. The fallback arms are derived post-hoc from the per-query results of
+the other arms (``fallback@n`` for a query = the Super Search NDCG when standard
+returns ``<= n`` results, else the standard NDCG), so they add no extra model runs.
+
+Standard search uses the **remote** production index (``api.mwmbl.org``) by default,
+not the tiny local dev index: the gate counts standard results, and the local index
+returns ``<= 3`` for almost every in-coverage query (their answers live on external
+domains it does not hold), which would make the fallback fire on everything. Pass
+``--standard-index local`` to reproduce the original local-index standard arm.
+
Run:
DATABASE_URL="postgres://daoud@" DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \
uv run python scripts/super_search_new_sources_eval.py --max-queries 80
@@ -31,10 +45,9 @@ from django.conf import settings # noqa: E402
from sklearn.metrics import ndcg_score # noqa: E402
from mwmbl.rankeval.evaluation.evaluate import CLICK_PROPORTIONS, NUM_RESULTS_FOR_EVAL # noqa: E402
-from mwmbl.rankeval.evaluation.evaluate_ranker import MwmblRankingModel # noqa: E402
+from mwmbl.rankeval.evaluation.evaluate_fallback import _standard_model # noqa: E402
from mwmbl.rankeval.evaluation.evaluate_super_search import SuperSearchRankingModel # noqa: E402
from mwmbl.rankeval.paths import RANKINGS_DATASET_TEST_PATH # noqa: E402
-from mwmbl.search_setup import ranker # noqa: E402
from mwmbl.tinysearchengine.super_search_select.registry import get_meta # noqa: E402
NEW_SOURCES = ["www_gov_uk", "imdb"]
@@ -65,6 +78,12 @@ def _ndcg(predicted_urls: list[str], gold_scores: dict[str, float]) -> float:
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--max-queries", type=int, default=80)
+ parser.add_argument("--standard-index", choices=["remote", "local"], default="remote",
+ help="Index for the standard-search arm and the fallback gate "
+ "(default remote = api.mwmbl.org production index).")
+ parser.add_argument("--fallback-thresholds", type=int, nargs="+", default=[1, 3, 5],
+ help="Fall back to Super Search when standard returns <= threshold "
+ "results. Evaluated post-hoc for each value.")
args = parser.parse_args()
df = pd.read_csv(RANKINGS_DATASET_TEST_PATH)
@@ -86,12 +105,14 @@ def main():
rows["score"] = CLICK_PROPORTIONS[:len(rows)]
gold[q] = rows.set_index("url")["score"].to_dict()
+ print(f"standard-search index: {args.standard_index}")
arms = [
- ("standard", lambda: MwmblRankingModel(ranker), None),
+ ("standard", lambda: _standard_model(use_local=args.standard_index == "local"), None),
("ss-baseline", lambda: SuperSearchRankingModel(selection_key=""), []),
("ss+new", lambda: SuperSearchRankingModel(selection_key="+".join(NEW_SOURCES)), NEW_SOURCES),
]
per_query = {name: {} for name, _, _ in arms}
+ standard_count = {} # number of results standard search returned (the fallback gate)
for name, make_model, force in arms:
if force is not None:
settings.SUPER_SEARCH_FORCE_INCLUDE = force
@@ -100,15 +121,30 @@ def main():
for i, q in enumerate(queries):
preds = model.predict(q)
per_query[name][q] = _ndcg(preds, gold[q])
+ if name == "standard":
+ standard_count[q] = len(preds)
if (i + 1) % 20 == 0:
print(f" {i + 1}/{len(queries)}")
settings.SUPER_SEARCH_FORCE_INCLUDE = []
+ # Fallback arms, derived post-hoc: keep standard unless it returned <= n results,
+ # in which case fall back to the corresponding always-Super-Search arm.
+ for n in sorted(args.fallback_thresholds):
+ for ss_name in ("ss-baseline", "ss+new"):
+ per_query[f"fallback@{n}/{ss_name}"] = {
+ q: (per_query[ss_name][q] if standard_count[q] <= n else per_query["standard"][q])
+ for q in queries
+ }
+
print("\n" + "=" * 60)
- print(f"{'arm':14} {'mean NDCG':>10}")
- for name, _, _ in arms:
- vals = list(per_query[name].values())
- print(f"{name:14} {np.mean(vals):>10.4f}")
+ print(f"{'arm':24} {'mean NDCG':>10}")
+ for name in per_query:
+ print(f"{name:24} {np.mean(list(per_query[name].values())):>10.4f}")
+
+ print(f"\nfallback fire rate (standard <= n results), of {len(queries)} queries:")
+ for n in sorted(args.fallback_thresholds):
+ fired = sum(c <= n for c in standard_count.values())
+ print(f" n={n}: {fired:3d} fired ({fired / len(queries):.0%})")
base = per_query["ss-baseline"]
new = per_query["ss+new"]
@@ -117,11 +153,28 @@ def main():
delta = np.mean([new[q] - base[q] for q in queries])
print(f"\nss+new vs ss-baseline: mean Δ {delta:+.4f} | better {wins}, worse {losses}, "
f"same {len(queries) - wins - losses}")
- movers = sorted(queries, key=lambda q: new[q] - base[q], reverse=True)
- print("\ntop NDCG gains (ss+new over baseline):")
- for q in movers[:8]:
- if new[q] - base[q] > 1e-9:
- print(f" {new[q] - base[q]:+.3f} {q!r}")
+
+ # The headline fallback question: does gating Super Search behind standard-search
+ # failure beat serving standard search alone, on this in-coverage subset?
+ std = per_query["standard"]
+ print("\nfallback vs standard-always (paired, same queries):")
+ for n in sorted(args.fallback_thresholds):
+ for ss_name in ("ss-baseline", "ss+new"):
+ fb = per_query[f"fallback@{n}/{ss_name}"]
+ d = np.mean([fb[q] - std[q] for q in queries])
+ w = sum(fb[q] > std[q] + 1e-9 for q in queries)
+ l = sum(fb[q] < std[q] - 1e-9 for q in queries)
+ print(f" fallback@{n}/{ss_name:11} mean Δ {d:+.4f} | better {w}, worse {l}")
+
+ # Why the fallback nets ~0: show, for the queries it fires on, whether Super
+ # Search actually rescues them. If standard and Super Search both score ~0 on the
+ # starved queries, the result-count gate is selecting queries SS cannot help.
+ nmax = max(args.fallback_thresholds)
+ fired = sorted([q for q in queries if standard_count[q] <= nmax],
+ key=lambda q: standard_count[q])
+ print(f"\nfired queries at n={nmax} (standard count, NDCG: standard / ss-baseline / ss+new):")
+ for q in fired:
+ print(f" cnt={standard_count[q]:2d} {std[q]:.3f} / {base[q]:.3f} / {new[q]:.3f} {q!r}")
if __name__ == "__main__":
diff --git a/test/test_fallback_model.py b/test/test_fallback_model.py
new file mode 100644
index 0000000..131a01f
--- /dev/null
+++ b/test/test_fallback_model.py
@@ -0,0 +1,52 @@
+"""Trigger logic for the Super-Search-as-fallback ranking model (no network)."""
+from mwmbl.rankeval.evaluation.evaluate import RankingModel
+
+
+class _FixedModel(RankingModel):
+ def __init__(self, urls, calls=None):
+ self.urls = urls
+ self.calls = calls # optional list to record invocations
+
+ def predict(self, query):
+ if self.calls is not None:
+ self.calls.append(query)
+ return list(self.urls)
+
+
+def _fallback(primary_urls, fallback_urls, threshold, fb_calls=None):
+ # Imported lazily so the module's django.setup()/search_setup import cost is only
+ # paid when these tests actually run.
+ from mwmbl.rankeval.evaluation.evaluate_fallback import FallbackRankingModel
+ return FallbackRankingModel(
+ _FixedModel(primary_urls),
+ _FixedModel(fallback_urls, calls=fb_calls),
+ threshold=threshold,
+ )
+
+
+def test_serves_primary_when_above_threshold():
+ primary = ["a", "b", "c", "d"]
+ model = _fallback(primary, ["x", "y"], threshold=3)
+ assert model.predict("q") == primary
+
+
+def test_falls_back_when_at_or_below_threshold():
+ model = _fallback(["a", "b", "c"], ["x", "y"], threshold=3)
+ assert model.predict("q") == ["x", "y"]
+
+
+def test_falls_back_on_empty_primary():
+ model = _fallback([], ["x", "y"], threshold=3)
+ assert model.predict("q") == ["x", "y"]
+
+
+def test_fallback_not_invoked_when_primary_sufficient():
+ fb_calls: list[str] = []
+ model = _fallback(["a", "b", "c", "d"], ["x"], threshold=3, fb_calls=fb_calls)
+ model.predict("q")
+ assert fb_calls == [] # the expensive Super Search pipeline is never run
+
+
+def test_threshold_zero_only_fires_on_empty():
+ assert _fallback(["a"], ["x"], threshold=0).predict("q") == ["a"]
+ assert _fallback([], ["x"], threshold=0).predict("q") == ["x"]