Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Ship Combined Search's provider features, in the Rust pipeline

Daoud Clarked8275d7d0843
diff --git a/mwmbl/rankeval/README.md b/mwmbl/rankeval/README.md
index 9771371..22d2a89 100644
--- a/mwmbl/rankeval/README.md
+++ b/mwmbl/rankeval/README.md
@@ -48,10 +48,10 @@ Search v2 against standard search, is added separately.)
## Combined Search
-`GET /api/v2/combined-search/` pools the Mwmbl index, Staan and Wikipedia and
-ranks the union with its own LTR model (`mwmbl/resources/model-combined.xgb`,
-falling back to the deployed `model.xgb` until that artifact exists). It is
-intended to replace Super Search, which repeated evaluation has found does not
+`GET /api/v2/combined-search/` pools the Mwmbl index and Staan and ranks the
+union with its own LTR model (`mwmbl/resources/model-combined.xgb`), which sees
+Staan's own ranking as features, and exempts Staan's results from the
+majority-terms filter (`CombinedLTRRanker`). It is intended to replace Super Search, which repeated evaluation has found does not
beat standard search — see `SUPER_SEARCH_EVAL_FINDINGS.md` and
`SUPER_SEARCH_ADD_SOURCES_FINDINGS.md`, where the conclusion is that the
re-ranker, not the source set, is the bottleneck.
@@ -63,6 +63,11 @@ DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \
DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \
uv run python -m mwmbl.rankeval.evaluation.evaluate_combined_search --fraction 0.05 --no-staan
+# The shipped endpoint against Staan alone, the previous endpoint and Brave, on the
+# gold set and the blind MiniLM judge. Needs .env sourced, or Staan returns nothing.
+DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \
+ uv run --no-sync python -m mwmbl.rankeval.evaluation.compare_combined_providers --fraction 0.05
+
# Standard search, Super Search and both Combined Search arms on the same queries.
DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \
uv run python -m mwmbl.rankeval.evaluation.compare_search_modes --fraction 0.05
@@ -70,12 +75,14 @@ DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \
### Training the combined model
-The combined model is the same 50-feature Rust XGBoost model as standard search,
-retrained on a candidate pool that includes Staan. A provider's own ranking
-reaches it only through the `score` column (the `item_score` feature), exactly as
-Wikipedia's does — there is no source feature — so the constant used to build the
-training data has to be the constant the serving path uses. See `STAAN_TOP_SCORE`
-in `mwmbl/tinysearchengine/staan.py`.
+The combined model is standard search's 50 Rust features plus two provider
+features, `in_staan` and `staan_rank` (`PROVIDER_FEATURE_NAMES` in `mwmbl_rank`),
+retrained on a candidate pool that includes Staan. A model saved with provider
+features records that in its own metadata, so loading it is enough to switch them
+on. Staan's rank also still reaches the model through the `score` column (the
+`item_score` feature), so the constant used to build the training data has to be
+the constant the serving path uses. See `STAAN_TOP_SCORE` in
+`mwmbl/tinysearchengine/staan.py`.
```bash
# 1. Add Staan's results to the existing Pass-2 pool (resumable; merge is idempotent).
@@ -99,12 +106,12 @@ DATABASE_URL="postgres://daoud@" uv run python -m mwmbl.rankeval.ltr.llm_experim
--mode mixed --ext-weight 0.25 --overall-threshold 4 --scale-pos-weight 1.0 \
--add-curation --curation-weight 0.5 --note combined
-# 5. Once a config wins, retrain it on everything and install it.
-DATABASE_URL="postgres://daoud@" uv run python -m mwmbl.rankeval.ltr.llm_experiment \
- --mode mixed --ext-weight 0.25 --overall-threshold 4 --scale-pos-weight 1.0 \
- --add-curation --curation-weight 0.5 --test-size 0 --note combined-final \
- --save-model devdata/rankeval-2026-04/model-combined.xgb
-cp devdata/rankeval-2026-04/model-combined.xgb mwmbl/resources/model-combined.xgb
+# 5. Retrain the winning config on everything, with the provider features, and
+# install it. `uv run` re-syncs and reinstalls the last released build of
+# mwmbl_rank, so after `maturin develop` use `uv run --no-sync`.
+DATABASE_URL="postgres://daoud@" uv run --no-sync python -m mwmbl.rankeval.ltr.provider_features \
+ --save-model devdata/rankeval-2026-04/model-combined-provider.xgb
+cp devdata/rankeval-2026-04/model-combined-provider.xgb mwmbl/resources/model-combined.xgb
```
`llm_experiment` reports two axes and **both** gate a retrain: NDCG on a held-out
@@ -130,8 +137,13 @@ same Staan-containing test pool:
| without Staan (`--exclude-pool staan`) | 0.7583 | 0.803 |
Marginal, and expected: the 50-feature model has no source feature, so Staan's prior
-reaches it only through `score`. **The endpoint's value is recall, not the re-ranker.**
-On the same 2% sample of the gold test set:
+reaches it only through `score`. That is what the provider features fix: with them,
+and with Staan's results kept past the term filter, Combined Search beats calling
+Staan alone on the blind judge (0.774 against 0.756 NDCG@10, +0.018 [+0.008, +0.028]
+over 298 gold test queries). It still trails Staan on gold NDCG (0.682 against 0.734),
+and Brave on both. See `combined-search-handover.md`.
+
+The earlier 50-feature endpoint, on a 2% sample of the gold test set:
| arm | NDCG | ±SEM | gold recall | latency |
|---|---|---|---|---|
diff --git a/mwmbl/rankeval/combined-search-handover.md b/mwmbl/rankeval/combined-search-handover.md
index 0e60e13..04f6cde 100644
--- a/mwmbl/rankeval/combined-search-handover.md
+++ b/mwmbl/rankeval/combined-search-handover.md
@@ -22,6 +22,46 @@ Brave is still ahead on both measures (0.757 gold, 0.859 judge).
The previous handover, with the method and the Brave and latency findings, is at
https://claude.ai/code/artifact/9931df9a-ebd1-4f18-b751-a23b48904fc9.
+## Status: shipped (2026-09-23, branch `combined-search-provider-features`)
+
+The plan below was carried out as **Option B**, with the ranking logic in Rust:
+
+- **`mwmbl_rank`:** `DocumentRecord` carries a `StaanRank` (not asked / not returned /
+ rank) and a `from_staan` flag.
+ - A pipeline with `provider_features` appends `in_staan` and `staan_rank` to the 50
+ shared features. `predict` exempts `from_staan` records from the term filter.
+ - A model saved with provider features records that in a booster attribute, so
+ `load_model` switches them on. Standard search's `model.xgb` has no such attribute and
+ is unchanged.
+- **Serving:** `CombinedLTRRanker` (`mwmbl/tinysearchengine/ltr_rank.py`) supplies Staan's
+ rank from each Staan result's score. The endpoint no longer fetches Wikipedia.
+ - Staan counts as "asked" when any of its results reached the pool. So a Staan outage and
+ an empty Staan answer both score as "not asked", like the index-only training rows.
+- **Model:** retrained in Rust (`mwmbl.rankeval.ltr.provider_features`) and installed as
+ `mwmbl/resources/model-combined.xgb`.
+- **Eval:** `compare_combined_providers` now runs `staan`, `combined` (shipped),
+ `previous` (the old endpoint), `reference` (this handover's Python booster) and
+ `brave`. The experiment arms below are in `f53ec73`. The log is
+ `devdata/combined_providers_eval/verify-shipped.log`.
+
+Re-verified on the same 298 queries:
+
+| Arm | Gold NDCG | Judge NDCG@10 |
+|---|---|---|
+| brave | 0.757 | 0.865 |
+| **combined (shipped, Rust)** | **0.682** | **0.774** |
+| reference (Python `provider-nowiki-keep`) | 0.664 | 0.770 |
+| staan | 0.734 | 0.756 |
+| previous (old endpoint) | 0.596 | 0.698 |
+
+- **Combined − staan:** judge +0.018 [+0.008, +0.028], W/T/L 187/0/111. Gold −0.052
+ [−0.076, −0.028].
+- **Combined − reference:** gold +0.019 [−0.001, +0.038], judge +0.003 [−0.001, +0.008]. The
+ Rust retrain lands on the Python one or slightly above it.
+
+The reference arm reproduces this handover's gold number exactly. The judge numbers shift
+slightly because the judged pool has different arms in it.
+
---
## Results (absolute)
diff --git a/mwmbl/rankeval/evaluation/compare_combined_providers.py b/mwmbl/rankeval/evaluation/compare_combined_providers.py
index 0f3156b..36ac9eb 100644
--- a/mwmbl/rankeval/evaluation/compare_combined_providers.py
+++ b/mwmbl/rankeval/evaluation/compare_combined_providers.py
@@ -8,28 +8,18 @@ Two questions, one run:
The arms, each scored on the same sampled gold test queries:
- ``staan``: Staan's top ten in Staan's own order - what a client gets calling Staan direct.
-- ``staan+wiki``: Staan and Wikipedia ranked by the combined model over an empty index, so
- the step from ``staan`` is the re-ranker and Wikipedia, and nothing from Mwmbl.
-- ``combined``: the endpoint - production index + Staan + Wikipedia, combined model, MMR.
- The step from ``staan+wiki`` is exactly what Mwmbl's own results add.
-- ``mwmbl``: the production index + Wikipedia under the combined model, no Staan.
+- ``combined``: the endpoint as shipped - production index + Staan, ``CombinedLTRRanker``
+ with ``model-combined.xgb`` (Staan's rank as features, Staan's results exempt from the
+ majority-terms filter), MMR.
+- ``previous``: the endpoint before that - index + Staan + Wikipedia under the 50-feature
+ combined model, with the filter applied to everything. Needs the old model at
+ ``devdata/rankeval-2026-04/model-combined.xgb``.
+- ``reference``: the Python XGBoost booster the shipped model was ported from
+ (``model-combined-provider.json``, the ``provider-nowiki-keep`` arm in
+ ``mwmbl/rankeval/combined-search-handover.md``), over the same pool as ``combined``.
+ ``combined`` should land on it: the Rust and Python trainings are not bit-identical.
- ``brave``: the Brave Search API's top ten, when ``BRAVE_SEARCH_API_KEY`` is set.
-And the experiments on the combined arm, named by what they change:
-
-- ``-nowiki``: Staan only, no Wikipedia - Staan already returns Wikipedia pages when they
- are relevant, so the separate Wikipedia results may only be adding noise.
-- ``-keep``: Staan results are exempt from the model's majority-terms filter, which zeroes
- (and so drops) any candidate matching no more than half the query terms. Staan matches
- semantically, so a low term overlap says less about its results than about the index's.
-- ``control-``: a Python retrain of the combined model on the same rows and parameters -
- the baseline for ``provider-``, confirming the retrain alone moves nothing.
-- ``provider-``: that retrain with Staan's own ranking as features (``in_staan`` and
- ``staan_rank``; see ``mwmbl.rankeval.ltr.provider_features``).
-
-The ``-keep`` and model experiments score with the Python booster over the Rust feature
-matrix, which reproduces RustXGBPipeline.predict exactly before its filter.
-
The Mwmbl side uses RemoteIndex (api.mwmbl.org) rather than the local dev index, which is a
10 MB sample and would understate what the index contributes.
@@ -74,18 +64,17 @@ from django.conf import settings # noqa: E402
from mwmbl.rankeval.evaluation.evaluate import NUM_RESULTS_FOR_EVAL, gold_scores_for # noqa: E402
from mwmbl.rankeval.evaluation.evaluate_ranker import DummyCompleter # noqa: E402
from mwmbl.rankeval.evaluation.remote_index import RemoteIndex # noqa: E402
-from mwmbl.rankeval.ltr.provider_features import ( # noqa: E402
- fails_term_filter,
- feature_matrix,
-)
from mwmbl.rankeval.paths import RANKINGS_DATASET_TEST_PATH # noqa: E402
from mwmbl.search_setup import combined_ltr_model # noqa: E402
-from mwmbl.tinysearchengine.indexer import Document, DocumentSource # noqa: E402
-from mwmbl.tinysearchengine.ltr_rank import LTRRanker # noqa: E402
+from mwmbl.tinysearchengine.indexer import Document # noqa: E402
+from mwmbl.tinysearchengine.ltr import RustXGBPipeline # noqa: E402
+from mwmbl.tinysearchengine.ltr_rank import CombinedLTRRanker, LTRRanker # noqa: E402
from mwmbl.tinysearchengine.mmr_rank import MMRRanker # noqa: E402
from mwmbl.tinysearchengine.rank import get_wiki_results # noqa: E402
-from mwmbl.tinysearchengine.staan import STAAN_TOP_SCORE, get_staan_results # noqa: E402
+from mwmbl.tinysearchengine.staan import get_staan_results # noqa: E402
from mwmbl.tinysearchengine.super_search_select.judge import Judge, doc_text # noqa: E402
+from mwmbl_rank import FEATURE_NAMES # noqa: E402
+from mwmbl_rank import RustXGBPipeline as RustPipeline # noqa: E402
BRAVE_SEARCH_URL = "https://api.search.brave.com/res/v1/web/search"
BRAVE_SEARCH_API_KEY = os.environ.get("BRAVE_SEARCH_API_KEY", "")
@@ -93,65 +82,32 @@ BRAVE_SEARCH_API_KEY = os.environ.get("BRAVE_SEARCH_API_KEY", "")
BRAVE_MIN_INTERVAL_SECONDS = 1.1
OUTPUT_DIR = Path("devdata/combined_providers_eval")
-CONTROL_MODEL_PATH = Path("devdata/rankeval-2026-04/model-combined-control.json")
-PROVIDER_MODEL_PATH = Path("devdata/rankeval-2026-04/model-combined-provider.json")
+PREVIOUS_MODEL_PATH = Path("devdata/rankeval-2026-04/model-combined.xgb")
+REFERENCE_MODEL_PATH = Path("devdata/rankeval-2026-04/model-combined-provider.json")
NUM_BOOTSTRAP = 10_000
+NUM_TERMS_INDEX = FEATURE_NAMES.index("num_terms")
+MATCH_TERMS_INDEX = FEATURE_NAMES.index("match_terms")
memory = Memory(location="devdata/cache")
-class EmptyIndex:
- """An index with nothing in it, so the ranker sees only the additional results."""
-
- def retrieve(self, term: str) -> list[Document]:
- return []
-
-
-class BoosterRanker(LTRRanker):
- """LTRRanker scored by a Python XGBoost booster, so the experiments can change what it sees.
+class ReferenceRanker(CombinedLTRRanker):
+ """CombinedLTRRanker scored by the Python booster the shipped model was ported from."""
- ``provider_features`` adds Staan's ranking as features, for a booster trained with them.
- ``keep_staan`` exempts Staan's results from the majority-terms filter.
- """
-
- def __init__(self, tiny_index, booster: xgb.Booster, provider_features: bool, keep_staan: bool):
- super().__init__(tiny_index, DummyCompleter(), model=None, include_wiki=False)
+ def __init__(self, tiny_index, booster: xgb.Booster):
+ super().__init__(tiny_index, DummyCompleter(), model=None)
self.booster = booster
- self.provider_features = provider_features
- self.keep_staan = keep_staan
def order_results(self, terms: list[str], results: list[Document], is_complete: bool) -> list[Document]:
if len(results) == 0:
return []
- query = " ".join(terms)
- records = [
- {
- "query": query,
- "url": page.url,
- "title": page.title if page.title is not None else "",
- "extract": page.extract if page.extract is not None else "",
- "score": page.score if page.score is not None else 0.0,
- }
- for page in results
- ]
- is_staan = np.array([page.source == DocumentSource.STAAN for page in results])
- staan_ranks = None
- if self.provider_features:
- # Each Staan result carries its rank in its score (see staan_score), which
- # survives the blacklist filter that would throw off counting positions.
- ranks = {
- page.url: round(STAAN_TOP_SCORE - page.score) for page in results if page.source == DocumentSource.STAAN
- }
- staan_ranks = [ranks] * len(records)
-
- features = feature_matrix(records, staan_ranks)
+ records = self.records(" ".join(terms), results)
+ features = np.array(RustPipeline.extract_features(records, True), dtype=np.float32)
predictions = self.booster.predict(xgb.DMatrix(features))
- dropped = fails_term_filter(features)
- if self.keep_staan:
- dropped &= ~is_staan
-
- kept = np.flatnonzero(~dropped)
+ fails_term_filter = features[:, MATCH_TERMS_INDEX] <= features[:, NUM_TERMS_INDEX] / 2.0
+ from_staan = np.array([record["from_staan"] for record in records])
+ kept = np.flatnonzero(~(fails_term_filter & ~from_staan))
order = kept[np.argsort(predictions[kept])[::-1]]
return [results[i] for i in order]
@@ -184,52 +140,27 @@ def _as_dicts(documents: list[Document]) -> list[dict]:
def build_arms(include_brave: bool):
remote_index = RemoteIndex()
- remote_ranker = MMRRanker(LTRRanker(remote_index, DummyCompleter(), combined_ltr_model, include_wiki=False))
- empty_ranker = MMRRanker(LTRRanker(EmptyIndex(), DummyCompleter(), combined_ltr_model, include_wiki=False))
-
- combined_booster = xgb.Booster(model_file=str(settings.COMBINED_MODEL_PATH))
- control_booster = xgb.Booster(model_file=str(CONTROL_MODEL_PATH))
- provider_booster = xgb.Booster(model_file=str(PROVIDER_MODEL_PATH))
-
- def booster_arm(booster, provider_features, keep_staan, with_wiki):
- ranker = MMRRanker(BoosterRanker(remote_index, booster, provider_features, keep_staan))
-
- def arm(query, staan_docs, wiki_docs):
- additional = staan_docs + wiki_docs if with_wiki else staan_docs
- return _as_dicts(ranker.search(query, additional, False))
-
- return arm
+ combined_ranker = MMRRanker(CombinedLTRRanker(remote_index, DummyCompleter(), combined_ltr_model))
+ previous_model = RustXGBPipeline.from_model_path(str(PREVIOUS_MODEL_PATH))
+ previous_ranker = MMRRanker(LTRRanker(remote_index, DummyCompleter(), previous_model, include_wiki=False))
+ reference_ranker = MMRRanker(ReferenceRanker(remote_index, xgb.Booster(model_file=str(REFERENCE_MODEL_PATH))))
def staan(query, staan_docs, wiki_docs):
return _as_dicts(staan_docs)
- def staan_wiki(query, staan_docs, wiki_docs):
- return _as_dicts(empty_ranker.search(query, staan_docs + wiki_docs, False))
-
def combined(query, staan_docs, wiki_docs):
- return _as_dicts(remote_ranker.search(query, staan_docs + wiki_docs, False))
+ return _as_dicts(combined_ranker.search(query, staan_docs, False))
- def mwmbl(query, staan_docs, wiki_docs):
- return _as_dicts(remote_ranker.search(query, wiki_docs, False))
+ def previous(query, staan_docs, wiki_docs):
+ return _as_dicts(previous_ranker.search(query, staan_docs + wiki_docs, False))
+
+ def reference(query, staan_docs, wiki_docs):
+ return _as_dicts(reference_ranker.search(query, staan_docs, False))
def brave(query, staan_docs, wiki_docs):
return brave_results(query)
- def combined_nowiki(query, staan_docs, wiki_docs):
- return _as_dicts(remote_ranker.search(query, staan_docs, False))
-
- arms = {
- "staan": staan,
- "staan+wiki": staan_wiki,
- "combined": combined,
- "mwmbl": mwmbl,
- "combined-nowiki": combined_nowiki,
- "combined-keep": booster_arm(combined_booster, False, keep_staan=True, with_wiki=True),
- "combined-nowiki-keep": booster_arm(combined_booster, False, keep_staan=True, with_wiki=False),
- "control-nowiki-keep": booster_arm(control_booster, False, keep_staan=True, with_wiki=False),
- "provider-nowiki": booster_arm(provider_booster, True, keep_staan=False, with_wiki=False),
- "provider-nowiki-keep": booster_arm(provider_booster, True, keep_staan=True, with_wiki=False),
- }
+ arms = {"staan": staan, "combined": combined, "previous": previous, "reference": reference}
if include_brave:
arms["brave"] = brave
return arms
@@ -326,16 +257,15 @@ def per_query_metrics(rows: list[dict]) -> pd.DataFrame:
def mwmbl_contribution(rows: list[dict]) -> dict:
"""What Mwmbl's own results do inside the combined top ten, against what they displaced.
- A combined result is Mwmbl's when neither Staan nor Wikipedia returned it. The displaced
- results are Staan's and Wikipedia's that ranked in ``staan+wiki``'s top ten but not in
- ``combined``'s.
+ A combined result is Mwmbl's when Staan did not return it. The displaced results are
+ Staan's top ten that did not make the combined top ten.
"""
mwmbl_scores, displaced_scores, counts = [], [], []
for row in rows:
- external = set(row["staan_urls"]) | set(row["wiki_urls"])
+ staan_urls = set(row["staan_urls"])
combined = row["lists"]["combined"]
- mwmbl_in_combined = [url for url in combined if url not in external]
- displaced = [url for url in row["lists"]["staan+wiki"] if url not in combined]
+ mwmbl_in_combined = [url for url in combined if url not in staan_urls]
+ displaced = [url for url in row["lists"]["staan"] if url not in combined]
counts.append(len(mwmbl_in_combined))
mwmbl_scores += [row["judged"][url] for url in mwmbl_in_combined]
displaced_scores += [row["judged"][url] for url in displaced]
@@ -358,10 +288,7 @@ def report(rows: list[dict]) -> None:
wide = {name: metrics.pivot(index="query", columns="arm", values=name) for name in metric_names[:3]}
arms = metrics["arm"].unique()
- comparisons = [("combined", "staan"), ("combined", "staan+wiki"), ("staan+wiki", "staan"), ("combined", "mwmbl")]
- experiments = [arm for arm in arms if arm.startswith(("combined-", "control-", "provider-"))]
- comparisons += [(arm, baseline) for arm in experiments for baseline in ("combined", "staan")]
- comparisons += [("provider-nowiki-keep", "control-nowiki-keep"), ("control-nowiki-keep", "combined-nowiki-keep")]
+ comparisons = [("combined", "staan"), ("combined", "previous"), ("combined", "reference"), ("reference", "staan")]
if "brave" in arms:
comparisons += [("combined", "brave"), ("staan", "brave")]
@@ -386,9 +313,8 @@ def report(rows: list[dict]) -> None:
def measure_latency(dataset: pd.DataFrame, num_queries: int, include_brave: bool) -> None:
"""Raw provider latency, uncached, interleaved so both providers share the conditions.
- Staan is Combined Search's critical path: Wikipedia is fetched concurrently with it (and
- is cached and circuit-broken in production), and ranking against the local index is
- in-process. Wikipedia is not timed here because a burst of uncached calls gets rate-limited.
+ Staan is Combined Search's critical path: it is the only network call, and ranking
+ against the local index is in-process.
"""
queries = sorted(sample_queries(dataset, 0.05))[:num_queries]
timings = {"staan": []}
diff --git a/mwmbl/rankeval/evaluation/evaluate_combined_search.py b/mwmbl/rankeval/evaluation/evaluate_combined_search.py
index 71b3bc5..f48cbb9 100644
--- a/mwmbl/rankeval/evaluation/evaluate_combined_search.py
+++ b/mwmbl/rankeval/evaluation/evaluate_combined_search.py
@@ -1,15 +1,14 @@
"""Evaluate the Combined Search ranking core as a RankingModel against the gold dataset.
-Runs what /api/v2/combined-search/ runs, minus the HTTP layer: pool the Mwmbl index, Staan
-and Wikipedia, rank the union with the combined LTR model, diversify. Only auth, quota and
+Runs what /api/v2/combined-search/ runs, minus the HTTP layer: pool the Mwmbl index and
+Staan, rank the union with the combined LTR model, diversify. Only auth, quota and
response formatting are skipped, so the ranking measured here is the ranking served.
Not to be confused with evaluate_combined.py, which combines two ranking *models* (Wikipedia
and Mwmbl) and predates this endpoint.
-Staan and Wikipedia are both network calls, so both are cached: Wikipedia through its own
-external results cache, and Staan through the same. A gold-set run therefore pays for each
-provider once per query and is cheap to repeat while iterating on the ranking. Pass
+Staan is a network call, cached through the external results cache, so a gold-set run pays
+for it once per query and is cheap to repeat while iterating on the ranking. Pass
``--no-staan`` to measure what Staan is actually contributing.
Usage::
@@ -29,12 +28,11 @@ django.setup()
# Imported after django.setup(): search_setup opens the index and loads the models.
from mwmbl.rankeval.evaluation.evaluate import RankingModel, evaluate # noqa: E402
from mwmbl.search_setup import combined_ranker # noqa: E402
-from mwmbl.tinysearchengine.rank import get_wiki_results # noqa: E402
from mwmbl.tinysearchengine.staan import get_staan_results # noqa: E402
class CombinedSearchRankingModel(RankingModel):
- """The endpoint's ranking core: three sources pooled, one model, MMR.
+ """The endpoint's ranking core: the index and Staan pooled, one model, MMR.
include_staan=False is the ablation arm - the same pool minus Staan, so the difference
between the two arms is what Staan contributes and nothing else.
@@ -46,9 +44,6 @@ class CombinedSearchRankingModel(RankingModel):
def predict(self, query: str) -> list[str]:
additional = get_staan_results(query) if self.include_staan else []
- additional = additional + get_wiki_results(query)
- # use_external_search=False: Wikipedia is already in `additional`, exactly as the
- # endpoint arranges it.
results = self.ranker.search(query, additional, False)
return [result.url for result in results]
@@ -57,7 +52,7 @@ def run():
parser = ArgumentParser(description=__doc__)
parser.add_argument("--fraction", type=float, default=0.05, help="Fraction of gold queries to sample.")
parser.add_argument("--train", action="store_true", help="Evaluate on the train split instead of test.")
- parser.add_argument("--no-staan", action="store_true", help="Ablation: pool the index and Wikipedia only.")
+ parser.add_argument("--no-staan", action="store_true", help="Ablation: the index only.")
args = parser.parse_args()
model = CombinedSearchRankingModel(include_staan=not args.no_staan)
diff --git a/mwmbl/rankeval/ltr/provider_features.py b/mwmbl/rankeval/ltr/provider_features.py
index 8571c03..9da5b67 100644
--- a/mwmbl/rankeval/ltr/provider_features.py
+++ b/mwmbl/rankeval/ltr/provider_features.py
@@ -1,42 +1,34 @@
-"""The Combined Search model with Staan's own ranking as a feature.
+"""Train the Combined Search model: the combined config, plus Staan's own ranking as features.
-The deployed combined model has no source feature: Staan's ordering reaches it only through
-`item_score` (``STAAN_TOP_SCORE - rank``), a column it shares with the index's own scores and
-Wikipedia's. This trains the same model with two features added, so it can learn how far to
-trust Staan's order:
+Staan's ordering used to reach the combined model only through `item_score`
+(``STAAN_TOP_SCORE - rank``), a column it shares with the index's own scores. This trains the
+same config with the Rust pipeline's provider features switched on, so the model can learn
+how far to trust Staan's order:
- ``in_staan``: 1 when Staan returned this URL for the query, else 0.
- ``staan_rank``: Staan's 0-based rank for the URL, missing when Staan did not return it.
Both are keyed on the URL, not on a document's source, so a page the index and Staan both
-return carries them whichever copy is scored - at training time the pool has one row per
-URL, and at serving time both copies are candidates.
-
-The extra features have no home in the Rust pipeline yet, so this is a Python XGBoost model
-over the Rust feature matrix (``RustXGBPipeline.extract_features``) plus the extra columns,
-trained with the Rust pipeline's exact parameters. ``--no-provider-features`` trains the
-control on the same rows and code path, so the two models differ by the features alone.
+return carries them whichever copy is scored. The feature definitions live in
+``mwmbl_rank`` (``StaanRank``); this module only supplies each row's Staan rank.
Staan ranks come from the Pass-2 checkpoint, which covers the 849 LLM-labelled queries. Rows
-for any other query (most extension and curation rows) never had Staan asked, so their
-provider features are missing rather than "not in Staan" - XGBoost learns a default
+for any other query (most extension and curation rows) never had Staan asked, so both
+features are missing for them rather than "not in Staan" - XGBoost learns a default
direction for them.
Usage::
DATABASE_URL="postgres://daoud@" uv run --no-sync python -m mwmbl.rankeval.ltr.provider_features \\
- --save-model devdata/rankeval-2026-04/model-combined-provider.json
- DATABASE_URL="postgres://daoud@" uv run --no-sync python -m mwmbl.rankeval.ltr.provider_features \\
- --no-provider-features --save-model devdata/rankeval-2026-04/model-combined-control.json
+ --save-model devdata/rankeval-2026-04/model-combined-provider.xgb
+ cp devdata/rankeval-2026-04/model-combined-provider.xgb mwmbl/resources/model-combined.xgb
"""
import json
from argparse import ArgumentParser
from pathlib import Path
-import numpy as np
import pandas as pd
-import xgboost as xgb
from mwmbl.rankeval.ltr.llm_experiment import (
FEATURE_COLUMNS,
@@ -45,55 +37,20 @@ from mwmbl.rankeval.ltr.llm_experiment import (
load_datasets,
)
from mwmbl.rankeval.paths import ROOT_DIR
-from mwmbl_rank import RustXGBPipeline as RustPipeline
+from mwmbl.tinysearchengine.ltr import RustXGBPipeline
STAAN_CHECKPOINT = ROOT_DIR / "devdata" / "llm_relabel" / "pass2_staan.jsonl"
-PROVIDER_FEATURE_NAMES = ["in_staan", "staan_rank"]
-# The config model-combined.xgb was trained with (see mwmbl/rankeval/README.md), so the
-# control reproduces it and the provider model differs from it only by the features.
+# The config model-combined.xgb was first trained with (see mwmbl/rankeval/README.md), so
+# the provider model differs from it only by the features.
OVERALL_THRESHOLD = 4.0
EXT_WEIGHT = 0.25
CURATION_WEIGHT = 0.5
WEAK_NEG_WEIGHT = 0.25
-XGB_PARAMS = {
- "objective": "binary:logistic",
- "tree_method": "exact",
- "reg_lambda": 2.0,
- "scale_pos_weight": 1.0,
-}
+SCALE_POS_WEIGHT = 1.0
+REG_LAMBDA = 2.0
NUM_ROUNDS = 100
-# Indices into the Rust feature row, for the majority-terms filter RustXGBPipeline.predict
-# applies. Named by the Rust side's FEATURE_NAMES; checked on import below.
-NUM_TERMS_INDEX = 42
-MATCH_TERMS_INDEX = 49
-assert RustPipeline.feature_names()[NUM_TERMS_INDEX] == "num_terms"
-assert RustPipeline.feature_names()[MATCH_TERMS_INDEX] == "match_terms"
-
-
-def provider_features(urls: list[str], staan_ranks: dict[str, int] | None) -> np.ndarray:
- """The ``in_staan`` and ``staan_rank`` columns. ``staan_ranks`` None means Staan was never asked."""
- if staan_ranks is None:
- return np.full((len(urls), len(PROVIDER_FEATURE_NAMES)), np.nan, dtype=np.float32)
- in_staan = [1.0 if url in staan_ranks else 0.0 for url in urls]
- rank = [staan_ranks.get(url, np.nan) for url in urls]
- return np.column_stack([in_staan, rank]).astype(np.float32)
-
-
-def feature_matrix(records: list[dict], staan_ranks: list[dict[str, int] | None] | None) -> np.ndarray:
- """The Rust features, plus the provider features when ``staan_ranks`` (one per record) is given."""
- base = np.array(RustPipeline.extract_features(records), dtype=np.float32)
- if staan_ranks is None:
- return base
- extra = np.vstack([provider_features([r["url"]], ranks) for r, ranks in zip(records, staan_ranks)])
- return np.hstack([base, extra])
-
-
-def fails_term_filter(features: np.ndarray) -> np.ndarray:
- """Rows RustXGBPipeline.predict zeroes: matching no more than half the query terms."""
- return features[:, MATCH_TERMS_INDEX] <= features[:, NUM_TERMS_INDEX] / 2.0
-
def load_staan_ranks() -> dict[str, dict[str, int]]:
"""qnorm -> {url: Staan's 0-based rank}, for the 849 LLM-labelled queries."""
@@ -106,10 +63,11 @@ def load_staan_ranks() -> dict[str, dict[str, int]]:
def training_frame() -> pd.DataFrame:
- """The combined model's training rows: LLM + extension + curation, no held-out split."""
+ """The combined model's training rows: LLM + extension + curation, no held-out split,
+ with each row's Staan rank."""
ext, llm = load_datasets()
curation = load_curation_frame(WEAK_NEG_WEIGHT)
- return build_training_frame(
+ frame = build_training_frame(
"mixed",
ext,
llm,
@@ -123,33 +81,37 @@ def training_frame() -> pd.DataFrame:
curation_weight=CURATION_WEIGHT,
)
+ by_query = load_staan_ranks()
+ query_ranks = [by_query.get(query.lower().strip()) for query in frame["query"]]
+ frame["staan_asked"] = [ranks is not None for ranks in query_ranks]
+ frame["staan_rank"] = [None if ranks is None else ranks.get(url) for url, ranks in zip(frame["url"], query_ranks)]
+ in_staan = frame["staan_rank"].notna()
+ print(f"Rows with Staan asked: {frame['staan_asked'].sum()} / {len(frame)}; in Staan's results: {in_staan.sum()}")
+ return frame
-def train(with_provider_features: bool) -> xgb.Booster:
- frame = training_frame()
- records = frame[FEATURE_COLUMNS].to_dict("records")
- staan_ranks = None
- if with_provider_features:
- by_query = load_staan_ranks()
- staan_ranks = [by_query.get(r["query"].lower().strip()) for r in records]
- covered = [ranks is not None for ranks in staan_ranks]
- in_staan = [ranks is not None and r["url"] in ranks for r, ranks in zip(records, staan_ranks)]
- print(f"Rows with Staan asked: {sum(covered)} / {len(records)}; in Staan's results: {sum(in_staan)}")
- features = feature_matrix(records, staan_ranks)
- matrix = xgb.DMatrix(features, label=frame["label"].to_numpy(), weight=frame["weight"].to_numpy())
- print(f"Training on {features.shape[0]} rows x {features.shape[1]} features")
- return xgb.train(XGB_PARAMS, matrix, num_boost_round=NUM_ROUNDS)
+def train() -> RustXGBPipeline:
+ frame = training_frame()
+ # Labels are already binary; threshold=0.5 makes the Rust-side binarisation a no-op.
+ model = RustXGBPipeline(
+ threshold=0.5,
+ scale_pos_weight=SCALE_POS_WEIGHT,
+ reg_lambda=REG_LAMBDA,
+ num_rounds=NUM_ROUNDS,
+ provider_features=True,
+ )
+ model.fit(frame[FEATURE_COLUMNS + ["staan_asked", "staan_rank"]], frame["label"], sample_weight=frame["weight"])
+ return model
def run():
parser = ArgumentParser(description=__doc__)
- parser.add_argument("--save-model", required=True, help="where to write the booster (.json)")
- parser.add_argument("--no-provider-features", action="store_true", help="train the control: the Rust features only")
+ parser.add_argument("--save-model", required=True, help="where to write the model (.xgb)")
args = parser.parse_args()
- booster = train(with_provider_features=not args.no_provider_features)
+ model = train()
Path(args.save_model).parent.mkdir(parents=True, exist_ok=True)
- booster.save_model(args.save_model)
+ model.save_model(args.save_model)
print(f"Saved to {args.save_model}")
diff --git a/mwmbl/resources/model-combined.xgb b/mwmbl/resources/model-combined.xgb
index 5a93b67..aaf256f 100644
Binary files a/mwmbl/resources/model-combined.xgb and b/mwmbl/resources/model-combined.xgb differ
diff --git a/mwmbl/search_setup.py b/mwmbl/search_setup.py
index 0cdbe23..510cfbc 100644
--- a/mwmbl/search_setup.py
+++ b/mwmbl/search_setup.py
@@ -6,7 +6,7 @@ from mwmbl.indexer.blacklist_snapshot import get_snapshot_blacklist
from mwmbl.tinysearchengine.completer import Completer
from mwmbl.tinysearchengine.indexer import Document, TinyIndex
from mwmbl.tinysearchengine.ltr import RustXGBPipeline
-from mwmbl.tinysearchengine.ltr_rank import LTRRanker
+from mwmbl.tinysearchengine.ltr_rank import CombinedLTRRanker, LTRRanker
from mwmbl.tinysearchengine.mmr_rank import MMRRanker
# Pull the blacklist snapshot in at worker startup rather than letting the first query
@@ -27,15 +27,7 @@ ltr_model = RustXGBPipeline.from_model_path(str(settings.RUST_MODEL_PATH))
ranker = MMRRanker(LTRRanker(tiny_index, completer, ltr_model, include_wiki=True))
# Combined Search gets its own model and its own ranker, so retraining on the pooled
-# Mwmbl + Staan + Wikipedia candidate set never moves standard search's ranking. Until
-# model-combined.xgb has been trained the endpoint serves the deployed model, which is a
-# working answer rather than a missing endpoint - the artifact ships in Stage 4.
-combined_model_path = Path(settings.COMBINED_MODEL_PATH)
-if not combined_model_path.exists():
- combined_model_path = Path(settings.RUST_MODEL_PATH)
-combined_ltr_model = RustXGBPipeline.from_model_path(str(combined_model_path))
-
-# include_wiki=False on purpose: the endpoint fetches Wikipedia itself so that call can run
-# concurrently with Staan's, and passes both in as additional_results. Leaving it True would
-# fetch Wikipedia a second time, serially, inside get_results.
-combined_ranker = MMRRanker(LTRRanker(tiny_index, completer, combined_ltr_model, include_wiki=False))
+# Mwmbl + Staan candidate set never moves standard search's ranking. The model uses
+# Staan's ranking as features, which only CombinedLTRRanker supplies.
+combined_ltr_model = RustXGBPipeline.from_model_path(str(settings.COMBINED_MODEL_PATH))
+combined_ranker = MMRRanker(CombinedLTRRanker(tiny_index, completer, combined_ltr_model))
diff --git a/mwmbl/tinysearchengine/combined_search.py b/mwmbl/tinysearchengine/combined_search.py
index 00aac26..b00d46d 100644
--- a/mwmbl/tinysearchengine/combined_search.py
+++ b/mwmbl/tinysearchengine/combined_search.py
@@ -1,18 +1,21 @@
-"""Combined Search - one request over the Mwmbl index, Staan and Wikipedia.
+"""Combined Search - one request over the Mwmbl index and Staan.
-Pools three sources, ranks the union with a model of its own, and answers with the same
+Pools both sources, ranks the union with a model of its own, and answers with the same
SearXNG-shaped JSON as /api/v2/search/. Authentication is required and a flat monthly quota
applies, exactly as for Super Search, which this endpoint is meant to replace.
There is deliberately no streaming and no crawling here. Super Search streams because it
-crawls promoted pages and follows their outbound links, which takes seconds; all three
-sources here answer in well under a second, and two of the three are cached, so there is
-nothing to stream and a plain response is what a client actually wants.
+crawls promoted pages and follows their outbound links, which takes seconds; Staan is the
+only network call here, and it is cached, so there is nothing to stream and a plain response
+is what a client actually wants.
-The ranking core is three calls, because Ranker.get_results already pools
-``pages + additional_results + external_search_items`` and blacklist-filters the lot. Staan
-and Wikipedia go in as additional_results, fetched concurrently; the index side and the
-ranking are the ranker's own job.
+The ranking core is two calls, because Ranker.get_results already pools
+``pages + additional_results`` and blacklist-filters the lot. Staan goes in as
+additional_results; the index side and the ranking are the ranker's own job.
+
+There is no separate Wikipedia fetch: Staan already returns Wikipedia pages when they are
+relevant, and evaluation found the extra fetch roughly neutral on quality (see
+mwmbl/rankeval/combined-search-handover.md).
Nothing found here is written back to the search index. Super Search indexes what it finds;
whether third-party SERP results belong in our index is a separate question with its own
@@ -34,7 +37,6 @@ from mwmbl.quota import (
increment_monthly_combined_search,
)
from mwmbl.search_auth import authenticate_user
-from mwmbl.tinysearchengine.rank import get_wiki_results
from mwmbl.tinysearchengine.search import SearchResponse
from mwmbl.tinysearchengine.staan import get_staan_results
@@ -44,22 +46,21 @@ router = Router(tags=["Combined Search"])
DESCRIPTION = (
- "Search the Mwmbl index, Staan and Wikipedia in one request and return the ranked "
- "union.\n\n"
- "Every candidate - crawled by Mwmbl, returned by Staan, or fetched from Wikipedia - is "
- "scored by one learning-to-rank model trained on the pooled candidate set, then "
+ "Search the Mwmbl index and Staan in one request and return the ranked union.\n\n"
+ "Every candidate - crawled by Mwmbl or returned by Staan - is scored by one "
+ "learning-to-rank model trained on the pooled candidate set, then "
"diversified so a single domain cannot take the whole page. The response is the same "
"SearXNG-compatible shape as `/api/v2/search/`.\n\n"
"The `engine` field names the provider a result came from:\n"
"- `mwmbl` - organically crawled by the Mwmbl crawler\n"
"- `staan` - returned by the Staan web-search API\n"
- "- `wikipedia` - fetched from Wikipedia\n"
+ "- `wikipedia` - a Wikipedia page from the Mwmbl index\n"
"- `google`, `user` - originally suggested via Google, or submitted by a user\n\n"
"Authentication is required: a search-scoped API key in `X-API-Key`, or a JWT bearer "
"token. Obtain a key via `POST /api/v1/platform/api-keys/`. A per-user monthly quota "
"applies; `monthly_usage` and `monthly_limit` report it on every response.\n\n"
- "A source that is down or unconfigured costs the request its extra recall, not its "
- "results: the remaining sources are ranked and returned as usual.\n\n"
+ "If Staan is down or unconfigured, the request loses its extra recall, not its "
+ "results: the index's results are ranked and returned as usual.\n\n"
"**Query parameter:** `q` - the search query string (required)."
)
@@ -75,18 +76,6 @@ OPENAPI_EXTRA = {
}
-async def _gather_external(query: str):
- """Staan's and Wikipedia's results, fetched concurrently.
-
- Both are synchronous, network-bound and independent, so they run on threads and the
- request waits once rather than twice. Each returns [] on failure by itself.
- """
- return await asyncio.gather(
- asyncio.to_thread(get_staan_results, query),
- asyncio.to_thread(get_wiki_results, query),
- )
-
-
def init_router(ranker) -> None:
@router.get(
"",
@@ -94,7 +83,7 @@ def init_router(ranker) -> None:
# Handled manually in the view so both an API key and a JWT work under an async
# view - the same reason Super Search does it this way.
auth=None,
- summary="Combined Search (Mwmbl + Staan + Wikipedia)",
+ summary="Combined Search (Mwmbl + Staan)",
description=DESCRIPTION,
openapi_extra=OPENAPI_EXTRA,
)
@@ -117,11 +106,12 @@ def init_router(ranker) -> None:
f"and you have used {monthly_usage - 1}.",
)
- staan_results, wiki_results = await _gather_external(q)
+ # get_staan_results returns [] on failure by itself.
+ staan_results = await asyncio.to_thread(get_staan_results, q)
results = await asyncio.to_thread(
ranker.search,
q,
- staan_results + wiki_results,
+ staan_results,
False, # use_external_search=False
)
diff --git a/mwmbl/tinysearchengine/ltr.py b/mwmbl/tinysearchengine/ltr.py
index 8d8b792..e6701e5 100644
--- a/mwmbl/tinysearchengine/ltr.py
+++ b/mwmbl/tinysearchengine/ltr.py
@@ -18,6 +18,9 @@ from sklearn.base import BaseEstimator, RegressorMixin, TransformerMixin
import mwmbl_rank
from mwmbl.tinysearchengine.rank import get_features
+# Optional record keys the Rust pipeline reads for Combined Search - see its docstring.
+PROVIDER_COLUMNS = ["staan_asked", "staan_rank", "from_staan"]
+
class ThresholdPredictor(BaseEstimator, RegressorMixin):
def __init__(self, threshold: float, classifier: BaseEstimator):
@@ -102,6 +105,9 @@ class RustXGBPipeline(BaseEstimator, RegressorMixin):
XGBoost gamma (min_split_loss) hyperparameter (default None, uses XGBoost default of 0.0).
subsample : float or None
XGBoost subsample hyperparameter (default None, uses XGBoost default of 1.0).
+ provider_features : bool
+ Append Staan's ranking to the features (default False). Combined Search only; a
+ loaded model sets this from its own metadata.
"""
def __init__(
@@ -114,6 +120,7 @@ class RustXGBPipeline(BaseEstimator, RegressorMixin):
min_child_weight: float | None = None,
gamma: float | None = None,
subsample: float | None = None,
+ provider_features: bool = False,
):
self.threshold = threshold
self.scale_pos_weight = scale_pos_weight
@@ -123,6 +130,7 @@ class RustXGBPipeline(BaseEstimator, RegressorMixin):
self.min_child_weight = min_child_weight
self.gamma = gamma
self.subsample = subsample
+ self.provider_features = provider_features
self._inner = mwmbl_rank.RustXGBPipeline(
threshold=self.threshold,
scale_pos_weight=self.scale_pos_weight,
@@ -132,12 +140,14 @@ class RustXGBPipeline(BaseEstimator, RegressorMixin):
min_child_weight=self.min_child_weight,
gamma=self.gamma,
subsample=self.subsample,
+ provider_features=self.provider_features,
)
@staticmethod
def _df_to_records(X: DataFrame) -> list:
"""Convert a DataFrame to a list of dicts for the Rust boundary."""
cols = ["query", "url", "title", "extract", "score"]
+ cols += [col for col in PROVIDER_COLUMNS if col in X.columns]
subset = X[cols].copy()
subset["title"] = subset["title"].fillna("")
subset["extract"] = subset["extract"].fillna("")
@@ -201,6 +211,7 @@ class RustXGBPipeline(BaseEstimator, RegressorMixin):
def load_model(self, path: str) -> "RustXGBPipeline":
"""Load a model from disk (XGBoost binary format)."""
self._inner.load_model(path)
+ self.provider_features = self._inner.provider_features
return self
@classmethod
@@ -239,5 +250,6 @@ class RustXGBPipeline(BaseEstimator, RegressorMixin):
f"max_depth={self.max_depth}, "
f"min_child_weight={self.min_child_weight}, "
f"gamma={self.gamma}, "
- f"subsample={self.subsample})"
+ f"subsample={self.subsample}, "
+ f"provider_features={self.provider_features})"
)
diff --git a/mwmbl/tinysearchengine/ltr_rank.py b/mwmbl/tinysearchengine/ltr_rank.py
index a6f9cf8..edd3e69 100644
--- a/mwmbl/tinysearchengine/ltr_rank.py
+++ b/mwmbl/tinysearchengine/ltr_rank.py
@@ -8,8 +8,9 @@ including both the Python sklearn pipeline and the Rust RustXGBPipeline.
import numpy as np
from mwmbl.tinysearchengine.completer import Completer
-from mwmbl.tinysearchengine.indexer import Document, TinyIndex
+from mwmbl.tinysearchengine.indexer import Document, DocumentSource, TinyIndex
from mwmbl.tinysearchengine.rank import NUM_WIKI_RESULTS, Ranker, get_wiki_results
+from mwmbl.tinysearchengine.staan import STAAN_TOP_SCORE
class LTRRanker(Ranker):
@@ -54,13 +55,9 @@ class LTRRanker(Ranker):
self.include_wiki = include_wiki
self.num_wiki_results = num_wiki_results
- def order_results(self, terms: list[str], results: list[Document], is_complete: bool) -> list[Document]:
- if len(results) == 0:
- return []
-
- query = " ".join(terms)
-
- data = [
+ def records(self, query: str, results: list[Document]) -> list[dict]:
+ """The records the model scores, one per result."""
+ return [
{
"query": query,
"url": page.url,
@@ -71,7 +68,12 @@ class LTRRanker(Ranker):
for page in results
]
- predictions = self.model.predict(data)
+ def order_results(self, terms: list[str], results: list[Document], is_complete: bool) -> list[Document]:
+ if len(results) == 0:
+ return []
+
+ query = " ".join(terms)
+ predictions = self.model.predict(self.records(query, results))
mask = predictions > 0.0
filtered_predictions = predictions[mask]
filtered_pages = np.array(results)[mask]
@@ -88,6 +90,37 @@ class LTRRanker(Ranker):
return []
+class CombinedLTRRanker(LTRRanker):
+ """Combined Search's ranker: tells the model what Staan made of each candidate.
+
+ Each record carries Staan's rank for its URL, for the model's provider features, and
+ whether it is Staan's own result, which exempts it from the majority-terms filter. Both
+ are applied in mwmbl_rank; see DocumentRecord there.
+
+ Staan counts as asked when any of its results reached the pool. A Staan that is down
+ returns nothing, which is then indistinguishable from Staan having no results: both
+ score as "not asked", the same as the index-only queries in the training data.
+ """
+
+ def __init__(self, tiny_index: TinyIndex, completer: Completer, model):
+ # The endpoint fetches Staan itself and passes it in as additional results.
+ super().__init__(tiny_index, completer, model, include_wiki=False)
+
+ def records(self, query: str, results: list[Document]) -> list[dict]:
+ # Each Staan result carries its rank in its score (see staan_score), which survives
+ # the blacklist filter that would throw off counting positions.
+ staan_ranks = {
+ page.url: round(STAAN_TOP_SCORE - page.score) for page in results if page.source == DocumentSource.STAAN
+ }
+ staan_asked = len(staan_ranks) > 0
+ records = super().records(query, results)
+ for record, page in zip(records, results):
+ record["staan_asked"] = staan_asked
+ record["staan_rank"] = staan_ranks.get(page.url)
+ record["from_staan"] = page.source == DocumentSource.STAAN
+ return records
+
+
def score_documents(model, query: str, documents: list[Document]) -> list[float]:
"""Run the LTR model over the given documents and return raw per-doc scores.
diff --git a/mwmbl_rank/src/features.rs b/mwmbl_rank/src/features.rs
index 070e4d7..e8002d1 100644
--- a/mwmbl_rank/src/features.rs
+++ b/mwmbl_rank/src/features.rs
@@ -84,6 +84,10 @@ pub const NUM_TERMS_INDEX: usize = 42;
/// domain_tokenized/path) in the feature vector.
pub const MATCH_TERMS_INDEX: usize = NUM_FEATURES - 1;
+/// Staan's own ranking, appended after FEATURE_NAMES by a pipeline that uses provider
+/// features (Combined Search's). Standard search never sees these columns.
+pub const PROVIDER_FEATURE_NAMES: &[&str] = &["in_staan", "staan_rank"];
+
/// Result of matching query terms against a text part.
#[derive(Debug, Default)]
pub struct MatchFeatures {
diff --git a/mwmbl_rank/src/lib.rs b/mwmbl_rank/src/lib.rs
index 65ad5ca..1203eb5 100644
--- a/mwmbl_rank/src/lib.rs
+++ b/mwmbl_rank/src/lib.rs
@@ -14,7 +14,7 @@ mod wiki;
use pyo3::prelude::*;
use pyo3::exceptions::PyValueError;
-use pipeline::{extract_features_batch, DocumentRecord, XGBPipeline};
+use pipeline::{extract_features_batch, num_features, DocumentRecord, StaanRank, XGBPipeline};
/// Convert a Python dict (passed as a Bound<PyAny>) to a DocumentRecord.
fn py_dict_to_record(obj: &Bound<'_, PyAny>) -> PyResult<DocumentRecord> {
@@ -34,7 +34,25 @@ fn py_dict_to_record(obj: &Bound<'_, PyAny>) -> PyResult<DocumentRecord> {
.map(|v| v.extract::<f64>().unwrap_or(0.0) as f32)
.unwrap_or(0.0);
- Ok(DocumentRecord { query, url, title, extract, score })
+ // Optional, and only read by a pipeline with provider features. A missing or NaN rank
+ // (as pandas writes None) means Staan did not return the URL.
+ let staan_asked: bool = obj.get_item("staan_asked")
+ .map(|v| v.extract::<bool>().unwrap_or(false))
+ .unwrap_or(false);
+ let rank: Option<f64> = obj.get_item("staan_rank")
+ .map(|v| v.extract::<Option<f64>>().unwrap_or(None))
+ .unwrap_or(None)
+ .filter(|rank| !rank.is_nan());
+ let staan_rank = match (staan_asked, rank) {
+ (false, _) => StaanRank::NotAsked,
+ (true, None) => StaanRank::NotReturned,
+ (true, Some(rank)) => StaanRank::Rank(rank as u32),
+ };
+ let from_staan: bool = obj.get_item("from_staan")
+ .map(|v| v.extract::<bool>().unwrap_or(false))
+ .unwrap_or(false);
+
+ Ok(DocumentRecord { query, url, title, extract, score, staan_rank, from_staan })
}
/// Python-visible XGBoost pipeline class.
@@ -45,7 +63,14 @@ fn py_dict_to_record(obj: &Bound<'_, PyAny>) -> PyResult<DocumentRecord> {
/// - save_model(path: str) -> None
/// - load_model(path: str) -> None
///
-/// Each record dict must have keys: query, url, title, extract, score.
+/// Each record dict must have keys: query, url, title, extract, score. Optional keys, for
+/// Combined Search:
+/// - staan_asked (bool): Staan was asked about this query
+/// - staan_rank (int or None): Staan's 0-based rank for this URL, None if not returned
+/// - from_staan (bool): the record is Staan's own result, exempt from the term filter
+///
+/// The first two are features only for a pipeline with provider_features, which a loaded
+/// model sets from its own metadata.
///
/// XGBPipeline implements Send (see pipeline.rs), so this class is safe to use
/// from multiple Python threads (e.g. Django worker threads).
@@ -67,8 +92,9 @@ impl PyXGBPipeline {
/// min_child_weight: XGBoost min_child_weight (default None → XGBoost default of 1.0)
/// gamma: XGBoost gamma / min_split_loss (default None → XGBoost default of 0.0)
/// subsample: XGBoost subsample (default None → XGBoost default of 1.0)
+ /// provider_features: append Staan's ranking to the features (default False)
#[new]
- #[pyo3(signature = (threshold=0.0, scale_pos_weight=0.1, reg_lambda=2.0, num_rounds=100, max_depth=None, min_child_weight=None, gamma=None, subsample=None))]
+ #[pyo3(signature = (threshold=0.0, scale_pos_weight=0.1, reg_lambda=2.0, num_rounds=100, max_depth=None, min_child_weight=None, gamma=None, subsample=None, provider_features=false))]
fn new(
threshold: f32,
scale_pos_weight: f32,
@@ -78,11 +104,12 @@ impl PyXGBPipeline {
min_child_weight: Option<f32>,
gamma: Option<f32>,
subsample: Option<f32>,
+ provider_features: bool,
) -> Self {
PyXGBPipeline {
inner: XGBPipeline::with_params(
threshold, scale_pos_weight, reg_lambda, num_rounds,
- max_depth, min_child_weight, gamma, subsample,
+ max_depth, min_child_weight, gamma, subsample, provider_features,
),
}
}
@@ -128,24 +155,32 @@ impl PyXGBPipeline {
.map_err(|e| PyValueError::new_err(e))
}
- /// Load a model from disk (XGBoost binary format).
+ /// Load a model from disk (XGBoost binary format). Sets provider_features from the model.
fn load_model(&mut self, path: &str) -> PyResult<()> {
self.inner.load_model(path)
.map_err(|e| PyValueError::new_err(e))
}
- /// The feature matrix `predict` scores, one row of NUM_FEATURES floats per record.
+ /// Whether this pipeline appends Staan's ranking to the features.
+ #[getter]
+ fn provider_features(&self) -> bool {
+ self.inner.provider_features
+ }
+
+ /// The feature matrix `predict` scores, one row per record: NUM_FEATURES floats, plus
+ /// the provider features when `provider_features`.
///
/// Exposed so experiments can add features of their own alongside these without
/// reimplementing the extraction in Python.
#[staticmethod]
- fn extract_features(records: &Bound<'_, PyAny>) -> PyResult<Vec<Vec<f32>>> {
+ #[pyo3(signature = (records, provider_features=false))]
+ fn extract_features(records: &Bound<'_, PyAny>, provider_features: bool) -> PyResult<Vec<Vec<f32>>> {
let list: Vec<Bound<'_, PyAny>> = records.extract()?;
let doc_records = list.iter()
.map(|item| py_dict_to_record(item))
.collect::<PyResult<Vec<DocumentRecord>>>()?;
- let flat = extract_features_batch(&doc_records);
- Ok(flat.chunks(features::NUM_FEATURES).map(|row| row.to_vec()).collect())
+ let flat = extract_features_batch(&doc_records, provider_features);
+ Ok(flat.chunks(num_features(provider_features)).map(|row| row.to_vec()).collect())
}
/// Return the feature names in the canonical order.
@@ -162,7 +197,7 @@ impl PyXGBPipeline {
fn __repr__(&self) -> String {
format!(
- "RustXGBPipeline(threshold={}, scale_pos_weight={}, reg_lambda={}, num_rounds={}, max_depth={:?}, min_child_weight={:?}, gamma={:?}, subsample={:?})",
+ "RustXGBPipeline(threshold={}, scale_pos_weight={}, reg_lambda={}, num_rounds={}, max_depth={:?}, min_child_weight={:?}, gamma={:?}, subsample={:?}, provider_features={})",
self.inner.threshold,
self.inner.scale_pos_weight,
self.inner.reg_lambda,
@@ -171,6 +206,7 @@ impl PyXGBPipeline {
self.inner.min_child_weight,
self.inner.gamma,
self.inner.subsample,
+ self.inner.provider_features,
)
}
}
@@ -220,5 +256,6 @@ fn mwmbl_rank(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add("PageError", m.py().get_type_bound::<index::PageError>())?;
m.add("NUM_FEATURES", features::NUM_FEATURES)?;
m.add("FEATURE_NAMES", features::FEATURE_NAMES.to_vec())?;
+ m.add("PROVIDER_FEATURE_NAMES", features::PROVIDER_FEATURE_NAMES.to_vec())?;
Ok(())
}
diff --git a/mwmbl_rank/src/pipeline.rs b/mwmbl_rank/src/pipeline.rs
index ae0d2f3..470bd3e 100644
--- a/mwmbl_rank/src/pipeline.rs
+++ b/mwmbl_rank/src/pipeline.rs
@@ -6,9 +6,36 @@
use xgb::{parameters, Booster, DMatrix};
-use crate::features::{get_features_with_regex, MATCH_TERMS_INDEX, NUM_FEATURES, NUM_TERMS_INDEX};
+use crate::features::{
+ get_features_with_regex, MATCH_TERMS_INDEX, NUM_FEATURES, NUM_TERMS_INDEX, PROVIDER_FEATURE_NAMES,
+};
use crate::text::{tokenize, build_query_regex};
+/// What Staan said about a record's URL for its query.
+///
+/// Keyed on the URL, not on where the record came from, so an index copy of a page Staan
+/// also returned carries Staan's rank too.
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum StaanRank {
+ /// Staan was never asked about this query, as for most extension and curation rows.
+ NotAsked,
+ /// Staan was asked and did not return this URL.
+ NotReturned,
+ /// Staan returned this URL at this 0-based rank.
+ Rank(u32),
+}
+
+impl StaanRank {
+ /// The `in_staan` and `staan_rank` columns, NaN where XGBoost should treat them as missing.
+ fn features(self) -> [f32; 2] {
+ match self {
+ StaanRank::NotAsked => [f32::NAN, f32::NAN],
+ StaanRank::NotReturned => [0.0, f32::NAN],
+ StaanRank::Rank(rank) => [1.0, rank as f32],
+ }
+ }
+}
+
/// A single document record passed from Python.
#[derive(Debug, Clone)]
pub struct DocumentRecord {
@@ -17,10 +44,29 @@ pub struct DocumentRecord {
pub title: String,
pub extract: String,
pub score: f32,
+ /// Only read by a pipeline with provider features.
+ pub staan_rank: StaanRank,
+ /// This record is Staan's own result, so it is exempt from the majority-terms filter:
+ /// Staan matches semantically, and a low term overlap says less about its results than
+ /// about the index's. An index copy of the same URL is not exempt.
+ pub from_staan: bool,
+}
+
+/// Width of a feature row: the shared features, plus Staan's when `provider_features`.
+pub fn num_features(provider_features: bool) -> usize {
+ if provider_features {
+ NUM_FEATURES + PROVIDER_FEATURE_NAMES.len()
+ } else {
+ NUM_FEATURES
+ }
}
+/// The booster attribute recording whether a model was trained with provider features, so a
+/// loaded model knows the row width it scores.
+const PROVIDER_FEATURES_ATTRIBUTE: &str = "mwmbl_provider_features";
+
/// Extract features for a slice of document records.
-/// Returns a flat row-major Vec<f32> of shape (n_rows × NUM_FEATURES).
+/// Returns a flat row-major Vec<f32> of shape (n_rows × num_features(provider_features)).
///
/// Optimisation: tokenizes each unique query only once, so a batch where many
/// records share the same query (typical in LTR datasets) avoids redundant work.
@@ -31,7 +77,7 @@ struct QueryCache {
re_url: Option<regex::Regex>,
}
-pub fn extract_features_batch(records: &[DocumentRecord]) -> Vec<f32> {
+pub fn extract_features_batch(records: &[DocumentRecord], provider_features: bool) -> Vec<f32> {
use std::collections::HashMap;
// Build a per-query cache of tokenized terms and compiled regexes.
@@ -39,7 +85,7 @@ pub fn extract_features_batch(records: &[DocumentRecord]) -> Vec<f32> {
// regardless of how many records share that query.
let mut query_cache: HashMap<&str, QueryCache> = HashMap::new();
- let mut flat: Vec<f32> = Vec::with_capacity(records.len() * NUM_FEATURES);
+ let mut flat: Vec<f32> = Vec::with_capacity(records.len() * num_features(provider_features));
for rec in records {
let entry = query_cache.entry(rec.query.as_str()).or_insert_with(|| {
let terms = tokenize(&rec.query.to_lowercase());
@@ -60,6 +106,9 @@ pub fn extract_features_batch(records: &[DocumentRecord]) -> Vec<f32> {
entry.re_url.as_ref(),
);
flat.extend_from_slice(&row);
+ if provider_features {
+ flat.extend_from_slice(&rec.staan_rank.features());
+ }
}
flat
}
@@ -81,6 +130,9 @@ pub struct XGBPipeline {
pub min_child_weight: Option<f32>,
pub gamma: Option<f32>,
pub subsample: Option<f32>,
+ /// Append Staan's ranking (PROVIDER_FEATURE_NAMES) to every row. Set at construction for
+ /// training, and read back from the model by load_model.
+ pub provider_features: bool,
booster: Option<Booster>,
}
@@ -98,6 +150,7 @@ impl XGBPipeline {
min_child_weight: None,
gamma: None,
subsample: None,
+ provider_features: false,
booster: None,
}
}
@@ -111,6 +164,7 @@ impl XGBPipeline {
min_child_weight: Option<f32>,
gamma: Option<f32>,
subsample: Option<f32>,
+ provider_features: bool,
) -> Self {
XGBPipeline {
threshold,
@@ -121,6 +175,7 @@ impl XGBPipeline {
min_child_weight,
gamma,
subsample,
+ provider_features,
booster: None,
}
}
@@ -172,7 +227,7 @@ impl XGBPipeline {
// Extract features
let t1 = Instant::now();
- let flat_features = extract_features_batch(records);
+ let flat_features = extract_features_batch(records, self.provider_features);
let n_rows = records.len();
eprintln!("[XGBPipeline::fit] Feature extraction done in {:.2?}: {} rows × {} features = {} values",
t1.elapsed(), n_rows, flat_features.len() / n_rows.max(1), flat_features.len());
@@ -250,6 +305,8 @@ impl XGBPipeline {
}
eprintln!("[XGBPipeline::fit] Training loop done in {:.2?}", t6.elapsed());
+ booster.set_attribute(PROVIDER_FEATURES_ATTRIBUTE, &self.provider_features.to_string())
+ .map_err(|e| format!("Failed to set provider features attribute: {}", e))?;
self.booster = Some(booster);
eprintln!("[XGBPipeline::fit] fit() complete");
Ok(())
@@ -265,8 +322,9 @@ impl XGBPipeline {
return Ok(vec![]);
}
- let flat_features = extract_features_batch(records);
+ let flat_features = extract_features_batch(records, self.provider_features);
let n_rows = records.len();
+ let row_width = num_features(self.provider_features);
let dmat = DMatrix::from_dense(&flat_features, n_rows)
.map_err(|e| format!("Failed to create DMatrix: {}", e))?;
@@ -277,10 +335,11 @@ impl XGBPipeline {
// Replicate the heuristic ranker's majority-terms filter: drop any result that
// matches no more than half the query terms. match_terms (the max matched terms
// over fields) and num_terms are already in the extracted feature matrix, so this
- // reuses the computed features rather than recomputing them.
+ // reuses the computed features rather than recomputing them. Staan's own results are
+ // exempt (see DocumentRecord::from_staan).
for (i, pred) in predictions.iter_mut().enumerate() {
- let row = &flat_features[i * NUM_FEATURES..(i + 1) * NUM_FEATURES];
- if row[MATCH_TERMS_INDEX] <= row[NUM_TERMS_INDEX] / 2.0 {
+ let row = &flat_features[i * row_width..(i + 1) * row_width];
+ if !records[i].from_staan && row[MATCH_TERMS_INDEX] <= row[NUM_TERMS_INDEX] / 2.0 {
*pred = 0.0;
}
}
@@ -297,9 +356,15 @@ impl XGBPipeline {
}
/// Load a model from a file path (XGBoost binary format).
+ ///
+ /// The model's own attribute sets `provider_features`; a model trained before the
+ /// attribute existed has none and uses the shared features only.
pub fn load_model(&mut self, path: &str) -> Result<(), String> {
let booster = Booster::load(path)
.map_err(|e| format!("Failed to load model: {}", e))?;
+ let attribute = booster.get_attribute(PROVIDER_FEATURES_ATTRIBUTE)
+ .map_err(|e| format!("Failed to read provider features attribute: {}", e))?;
+ self.provider_features = attribute.as_deref() == Some("true");
self.booster = Some(booster);
Ok(())
}
@@ -318,7 +383,7 @@ impl XGBPipeline {
) -> Result<Self, String> {
let mut pipeline = XGBPipeline::with_params(
threshold, scale_pos_weight, reg_lambda, num_rounds,
- max_depth, min_child_weight, gamma, subsample,
+ max_depth, min_child_weight, gamma, subsample, false,
);
pipeline.load_model(path)?;
Ok(pipeline)
@@ -336,16 +401,109 @@ mod tests {
title: format!("Rust Programming Example {}", i),
extract: "A great systems language".to_string(),
score: i as f32 * 0.1,
+ staan_rank: StaanRank::NotAsked,
+ from_staan: false,
}).collect()
}
+ fn record(url: &str, title: &str, staan_rank: StaanRank, from_staan: bool) -> DocumentRecord {
+ DocumentRecord {
+ query: "rust programming".to_string(),
+ url: url.to_string(),
+ title: title.to_string(),
+ extract: String::new(),
+ score: 1.0,
+ staan_rank,
+ from_staan,
+ }
+ }
+
#[test]
fn test_extract_features_batch_shape() {
let records = make_records(5);
- let flat = extract_features_batch(&records);
+ let flat = extract_features_batch(&records, false);
assert_eq!(flat.len(), 5 * NUM_FEATURES);
}
+ #[test]
+ fn test_provider_features_are_appended() {
+ let records = vec![
+ record("https://a.com/", "Rust", StaanRank::NotAsked, false),
+ record("https://b.com/", "Rust", StaanRank::NotReturned, false),
+ record("https://c.com/", "Rust", StaanRank::Rank(3), true),
+ ];
+ let shared = extract_features_batch(&records, false);
+ let flat = extract_features_batch(&records, true);
+ let width = num_features(true);
+ assert_eq!(width, NUM_FEATURES + 2);
+ assert_eq!(flat.len(), 3 * width);
+
+ let rows: Vec<&[f32]> = flat.chunks(width).collect();
+ for (i, row) in rows.iter().enumerate() {
+ assert_eq!(&row[..NUM_FEATURES], &shared[i * NUM_FEATURES..(i + 1) * NUM_FEATURES]);
+ }
+ assert!(rows[0][NUM_FEATURES].is_nan() && rows[0][NUM_FEATURES + 1].is_nan());
+ assert_eq!(rows[1][NUM_FEATURES], 0.0);
+ assert!(rows[1][NUM_FEATURES + 1].is_nan());
+ assert_eq!(&rows[2][NUM_FEATURES..], &[1.0, 3.0]);
+ }
+
+ #[test]
+ fn test_staan_results_are_exempt_from_the_term_filter() {
+ let records = make_records(20);
+ let mut pipeline = XGBPipeline::new(0.0, 0.1, 2.0, 10);
+ pipeline.fit(&records, &vec![1.0; 20], None).expect("fit should succeed");
+
+ // Neither matches a query term; only Staan's own result survives the filter, not
+ // an index copy that Staan also returned.
+ let test = vec![
+ record("https://example.com/cooking", "Cooking Recipes", StaanRank::Rank(0), true),
+ record("https://example.com/cooking", "Cooking Recipes", StaanRank::Rank(0), false),
+ ];
+ let preds = pipeline.predict(&test).expect("predict should succeed");
+ assert!(preds[0] > 0.0, "Staan's result should keep its prediction");
+ assert_eq!(preds[1], 0.0, "the index copy should still be filtered");
+ }
+
+ #[test]
+ fn test_provider_features_survive_save_and_load() {
+ let mut records = make_records(20);
+ for (i, rec) in records.iter_mut().enumerate() {
+ rec.staan_rank = if i % 2 == 0 { StaanRank::Rank(i as u32 / 2) } else { StaanRank::NotReturned };
+ }
+ let labels: Vec<f32> = (0..20).map(|i| if i % 2 == 0 { 1.0 } else { 0.0 }).collect();
+ let mut trained = XGBPipeline::with_params(0.0, 0.1, 2.0, 10, None, None, None, None, true);
+ trained.fit(&records, &labels, None).expect("fit should succeed");
+
+ let path = std::env::temp_dir().join(format!("mwmbl-rank-provider-{}.xgb", std::process::id()));
+ let path = path.to_str().unwrap();
+ trained.save_model(path).expect("save should succeed");
+
+ let mut loaded = XGBPipeline::new(0.0, 0.1, 2.0, 10);
+ loaded.load_model(path).expect("load should succeed");
+ std::fs::remove_file(path).unwrap();
+
+ assert!(loaded.provider_features);
+ assert_eq!(loaded.predict(&records).unwrap(), trained.predict(&records).unwrap());
+ }
+
+ #[test]
+ fn test_a_model_without_provider_features_loads_without_them() {
+ let records = make_records(20);
+ let labels: Vec<f32> = (0..20).map(|i| if i % 2 == 0 { 1.0 } else { 0.0 }).collect();
+ let mut trained = XGBPipeline::new(0.0, 0.1, 2.0, 10);
+ trained.fit(&records, &labels, None).expect("fit should succeed");
+ let path = std::env::temp_dir().join(format!("mwmbl-rank-shared-{}.xgb", std::process::id()));
+ let path = path.to_str().unwrap();
+ trained.save_model(path).expect("save should succeed");
+
+ let mut loaded = XGBPipeline::with_params(0.0, 0.1, 2.0, 10, None, None, None, None, true);
+ loaded.load_model(path).expect("load should succeed");
+ std::fs::remove_file(path).unwrap();
+
+ assert!(!loaded.provider_features);
+ }
+
#[test]
fn test_pipeline_fit_predict() {
let records = make_records(20);
@@ -379,6 +537,8 @@ mod tests {
title: "Rust Programming".to_string(),
extract: "rust programming language".to_string(),
score: 1.0,
+ staan_rank: StaanRank::NotAsked,
+ from_staan: false,
},
// Matches no query terms -> filtered to 0.0.
DocumentRecord {
@@ -387,6 +547,8 @@ mod tests {
title: "Cooking Recipes".to_string(),
extract: "how to bake bread".to_string(),
score: 1.0,
+ staan_rank: StaanRank::NotAsked,
+ from_staan: false,
},
];
diff --git a/test/test_combined_ranker.py b/test/test_combined_ranker.py
new file mode 100644
index 0000000..e944190
--- /dev/null
+++ b/test/test_combined_ranker.py
@@ -0,0 +1,61 @@
+"""CombinedLTRRanker: Combined Search's ranker tells the model what Staan made of each candidate."""
+
+from pandas import DataFrame
+
+from mwmbl.tinysearchengine.indexer import Document, DocumentSource
+from mwmbl.tinysearchengine.ltr import RustXGBPipeline
+from mwmbl.tinysearchengine.ltr_rank import CombinedLTRRanker
+from mwmbl.tinysearchengine.staan import staan_score
+
+QUERY = "rust programming"
+
+
+def staan_result(url: str, title: str, rank: int) -> Document:
+ return Document(title, url, "", staan_score(rank), source=DocumentSource.STAAN)
+
+
+def index_result(url: str, title: str) -> Document:
+ return Document(title, url, "", 1.0)
+
+
+def ranker(model=None) -> CombinedLTRRanker:
+ return CombinedLTRRanker(None, None, model)
+
+
+def test_staan_rank_is_keyed_on_the_url_and_read_from_the_score():
+ results = [
+ staan_result("https://a.com/", "Rust", 0),
+ staan_result("https://b.com/", "Rust", 3),
+ index_result("https://b.com/", "Rust"),
+ index_result("https://c.com/", "Rust"),
+ ]
+
+ records = ranker().records(QUERY, results)
+
+ assert [r["staan_rank"] for r in records] == [0, 3, 3, None]
+ assert [r["from_staan"] for r in records] == [True, True, False, False]
+ assert all(r["staan_asked"] for r in records)
+
+
+def test_an_index_only_pool_counts_as_staan_not_asked():
+ records = ranker().records(QUERY, [index_result("https://c.com/", "Rust")])
+
+ assert records[0]["staan_asked"] is False
+ assert records[0]["staan_rank"] is None
+
+
+def test_only_staan_results_escape_the_term_filter():
+ training = [
+ {"query": QUERY, "url": f"https://{i}.com/", "title": "Rust Programming", "extract": "", "score": 1.0}
+ for i in range(20)
+ ]
+ model = RustXGBPipeline(num_rounds=5, provider_features=True)
+ model.fit(DataFrame(training), [1.0] * 20)
+
+ staan_off_topic = staan_result("https://bread.com/", "Baking bread", 0)
+ index_off_topic = index_result("https://cakes.com/", "Baking cakes")
+ index_on_topic = index_result("https://rust-lang.org/", "Rust Programming")
+
+ ordered = ranker(model).order_results(QUERY.split(), [staan_off_topic, index_off_topic, index_on_topic], True)
+
+ assert {page.url for page in ordered} == {staan_off_topic.url, index_on_topic.url}
diff --git a/test/test_combined_search.py b/test/test_combined_search.py
index 11df5fa..ff139fb 100644
--- a/test/test_combined_search.py
+++ b/test/test_combined_search.py
@@ -1,8 +1,8 @@
"""Integration tests for the Combined Search endpoint (/api/v2/combined-search/).
-The endpoint's own job is auth, quota, fanning out to three sources concurrently and
-handing the union to one ranker. The ranking itself belongs to LTRRanker and is tested
-there, so these tests stub the ranker and the two external providers: what is checked here
+The endpoint's own job is auth, quota, fetching Staan and handing it to one ranker alongside
+the index. The ranking itself belongs to CombinedLTRRanker and is tested there, so these
+tests stub the ranker and Staan: what is checked here
is that each source reaches the pool, that provenance survives as far as the wire, that a
provider being down costs recall rather than the request, and that the quota is enforced
atomically.
@@ -28,13 +28,12 @@ INDEX_RESULT = Document("Tokio internals", "https://blog.example.com/tokio", "A
STAAN_RESULT = Document(
"Tokio", "https://tokio.rs/", "An asynchronous runtime for Rust.", 6.0, source=DocumentSource.STAAN
)
-WIKI_RESULT = Document(
+# There is no Wikipedia fetch, but the index holds Wikipedia pages of its own.
+WIKI_INDEX_RESULT = Document(
"Tokio (software)",
"https://en.wikipedia.org/wiki/Tokio",
"A Rust runtime.",
- 6.0,
state=DocumentState.FROM_WIKI,
- source=DocumentSource.WIKIPEDIA,
)
@@ -77,31 +76,26 @@ def fresh_quota(api_key):
@pytest.fixture
def stub_sources(monkeypatch):
- """Stub the two external providers and the ranker, recording what the ranker was given.
+ """Stub Staan and the ranker, recording what the ranker was given.
The stub ranker returns the pool unchanged, so the response is the pool: that is what
lets these tests assert on which sources reached it.
"""
calls = {}
- def configure(staan=(STAAN_RESULT,), wiki=(WIKI_RESULT,), index=(INDEX_RESULT,)):
+ def configure(staan=(STAAN_RESULT,), index=(INDEX_RESULT,)):
def fake_staan(query, *args, **kwargs):
calls["staan_query"] = query
if isinstance(staan, Exception):
raise staan
return list(staan)
- def fake_wiki(query, *args, **kwargs):
- calls["wiki_query"] = query
- return list(wiki)
-
def fake_search(query, additional_results, use_external_search=True):
calls["additional_results"] = additional_results
calls["use_external_search"] = use_external_search
return list(index) + list(additional_results)
monkeypatch.setattr(combined_search, "get_staan_results", fake_staan)
- monkeypatch.setattr(combined_search, "get_wiki_results", fake_wiki)
# The router closed over the ranker at registration time, so the ranker instance
# itself is what has to be patched, not the name in search_setup.
from mwmbl.search_setup import combined_ranker
@@ -199,42 +193,38 @@ def test_the_quota_counter_is_its_own(client, api_key, fresh_quota, stub_sources
@pytest.mark.django_db
-def test_all_three_sources_reach_the_response(client, api_key, fresh_quota, stub_sources):
+def test_both_sources_reach_the_response(client, api_key, fresh_quota, stub_sources):
stub_sources()
results = _get(client, api_key).json()["results"]
- assert [result["url"] for result in results] == [
- INDEX_RESULT.url,
- STAAN_RESULT.url,
- WIKI_RESULT.url,
- ]
+ assert [result["url"] for result in results] == [INDEX_RESULT.url, STAAN_RESULT.url]
@pytest.mark.django_db
def test_each_result_names_the_provider_it_came_from(client, api_key, fresh_quota, stub_sources):
- stub_sources()
+ stub_sources(index=(INDEX_RESULT, WIKI_INDEX_RESULT))
results = _get(client, api_key).json()["results"]
- assert [result["engine"] for result in results] == ["mwmbl", "staan", "wikipedia"]
+ assert [result["engine"] for result in results] == ["mwmbl", "wikipedia", "staan"]
@pytest.mark.django_db
def test_the_external_results_are_passed_in_as_additional_results(client, api_key, fresh_quota, stub_sources):
- """Staan and Wikipedia go in through Ranker.get_results' additional_results hook, which
- is what gets them blacklist-filtered and ranked alongside the index candidates."""
+ """Staan goes in through Ranker.get_results' additional_results hook, which is what gets
+ it blacklist-filtered and ranked alongside the index candidates."""
calls = stub_sources()
_get(client, api_key)
- assert [document.url for document in calls["additional_results"]] == [STAAN_RESULT.url, WIKI_RESULT.url]
+ assert [document.url for document in calls["additional_results"]] == [STAAN_RESULT.url]
@pytest.mark.django_db
-def test_the_ranker_is_told_not_to_fetch_wikipedia_again(client, api_key, fresh_quota, stub_sources):
- """The endpoint fetches Wikipedia itself so that call runs concurrently with Staan's.
- Leaving external search on would fetch it a second time, serially."""
+def test_the_ranker_is_told_not_to_fetch_wikipedia(client, api_key, fresh_quota, stub_sources):
+ """External search is standard search's Wikipedia fetch, which Combined Search drops:
+ Staan already returns Wikipedia pages when they are relevant."""
calls = stub_sources()
_get(client, api_key)
@@ -243,13 +233,12 @@ def test_the_ranker_is_told_not_to_fetch_wikipedia_again(client, api_key, fresh_
@pytest.mark.django_db
-def test_the_query_reaches_every_source(client, api_key, fresh_quota, stub_sources):
+def test_the_query_reaches_staan(client, api_key, fresh_quota, stub_sources):
calls = stub_sources()
_get(client, api_key, query="rust")
assert calls["staan_query"] == "rust"
- assert calls["wiki_query"] == "rust"
@pytest.mark.django_db
@@ -267,26 +256,17 @@ def test_the_result_count_matches_the_results(client, api_key, fresh_quota, stub
@pytest.mark.django_db
-def test_staan_returning_nothing_still_serves_the_other_sources(client, api_key, fresh_quota, stub_sources):
+def test_staan_returning_nothing_still_serves_the_index(client, api_key, fresh_quota, stub_sources):
stub_sources(staan=())
results = _get(client, api_key).json()["results"]
- assert [result["url"] for result in results] == [INDEX_RESULT.url, WIKI_RESULT.url]
-
-
-def test_wikipedia_returning_nothing_still_serves_the_other_sources(client, api_key, fresh_quota, stub_sources):
- stub_sources(wiki=())
-
- results = _get(client, api_key).json()["results"]
-
- assert [result["url"] for result in results] == [INDEX_RESULT.url, STAAN_RESULT.url]
+ assert [result["url"] for result in results] == [INDEX_RESULT.url]
@pytest.mark.django_db
def test_an_empty_pool_is_an_empty_response_not_an_error(client, api_key, fresh_quota, stub_sources):
- stub_sources(staan=(), wiki=(), index=())
+ stub_sources(staan=(), index=())
body = _get(client, api_key).json()
diff --git a/test/test_rust_pipeline.py b/test/test_rust_pipeline.py
index e3f78b2..b8c0bb8 100644
--- a/test/test_rust_pipeline.py
+++ b/test/test_rust_pipeline.py
@@ -19,6 +19,7 @@ import pytest
pytest.importorskip("mwmbl_rank", reason="mwmbl_rank Rust extension not built")
from mwmbl.tinysearchengine.ltr import RustXGBPipeline
+from mwmbl_rank import NUM_FEATURES, PROVIDER_FEATURE_NAMES
# ---------------------------------------------------------------------------
# Fixtures
@@ -266,3 +267,52 @@ class TestRustXGBPipelineNDCG:
# Compute NDCG over the whole dataset (single group)
score = ndcg_score([y.tolist()], [preds.tolist()])
assert score > 0.5, f"NDCG {score:.3f} is not above 0.5"
+
+
+class TestRustXGBPipelineProviderFeatures:
+ """Combined Search's pipeline appends Staan's ranking to the shared features."""
+
+ def test_provider_features_are_appended(self):
+ records = make_dataframe(3).to_dict("records")
+ records[0] |= {"staan_asked": True, "staan_rank": 2}
+ records[1] |= {"staan_asked": True, "staan_rank": None}
+
+ rows = np.array(RustXGBPipeline()._inner.extract_features(records, True))
+
+ assert rows.shape == (3, NUM_FEATURES + len(PROVIDER_FEATURE_NAMES))
+ np.testing.assert_array_equal(rows[:, :NUM_FEATURES], RustXGBPipeline()._inner.extract_features(records))
+ np.testing.assert_array_equal(rows[0, NUM_FEATURES:], [1.0, 2.0])
+ assert rows[1, NUM_FEATURES] == 0.0 and np.isnan(rows[1, NUM_FEATURES + 1])
+ assert np.isnan(rows[2, NUM_FEATURES:]).all()
+
+ def test_a_loaded_model_knows_it_uses_provider_features(self):
+ X = make_dataframe(30)
+ X["staan_asked"] = True
+ X["staan_rank"] = [float(i % 10) if i % 3 else np.nan for i in range(30)]
+ pipeline = RustXGBPipeline(num_rounds=10, provider_features=True)
+ pipeline.fit(X, make_labels(30))
+
+ with tempfile.NamedTemporaryFile(suffix=".xgb", delete=False) as f:
+ model_path = f.name
+ try:
+ pipeline.save_model(model_path)
+ loaded = RustXGBPipeline.from_model_path(model_path)
+ finally:
+ os.unlink(model_path)
+
+ assert loaded.provider_features
+ np.testing.assert_allclose(loaded.predict(X), pipeline.predict(X), atol=1e-5)
+
+ def test_a_standard_model_loads_without_provider_features(self):
+ pipeline = RustXGBPipeline(num_rounds=10)
+ pipeline.fit(make_dataframe(20), make_labels(20))
+
+ with tempfile.NamedTemporaryFile(suffix=".xgb", delete=False) as f:
+ model_path = f.name
+ try:
+ pipeline.save_model(model_path)
+ loaded = RustXGBPipeline(provider_features=True).load_model(model_path)
+ finally:
+ os.unlink(model_path)
+
+ assert not loaded.provider_features