An open source, non-profit web search engine
Document Combined Search and how its model is trained
Daoud Clarkeef507c2cc248
diff --git a/mwmbl/rankeval/README.md b/mwmbl/rankeval/README.mdindex ac93ecf..5611e8f 100644--- a/mwmbl/rankeval/README.md+++ b/mwmbl/rankeval/README.md@@ -45,3 +45,63 @@ uv run python -m mwmbl.rankeval.evaluation.evaluate_remote(A `RankingModel` wrapper around the Super Search pipeline, for comparing SuperSearch v2 against standard search, is added separately.)++## Combined Search++`GET /api/v2/combined-search/` pools the Mwmbl index, Staan and Wikipedia and+ranks the union with its own LTR model (`mwmbl/resources/model-combined.xgb`,+falling back to the deployed `model.xgb` until that artifact exists). It is+intended to replace Super Search, which repeated evaluation has found does not+beat standard search — see `SUPER_SEARCH_EVAL_FINDINGS.md` and+`SUPER_SEARCH_ADD_SOURCES_FINDINGS.md`, where the conclusion is that the+re-ranker, not the source set, is the bottleneck.++```bash+# The endpoint's ranking core against the gold set, and the no-Staan ablation.+DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \+ uv run python -m mwmbl.rankeval.evaluation.evaluate_combined_search --fraction 0.05+DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \+ uv run python -m mwmbl.rankeval.evaluation.evaluate_combined_search --fraction 0.05 --no-staan++# Standard search, Super Search and both Combined Search arms on the same queries.+DJANGO_SETTINGS_MODULE=mwmbl.settings_dev \+ uv run python -m mwmbl.rankeval.evaluation.compare_search_modes --fraction 0.05+```++### Training the combined model++The combined model is the same 50-feature Rust XGBoost model as standard search,+retrained on a candidate pool that includes Staan. A provider's own ranking+reaches it only through the `score` column (the `item_score` feature), exactly as+Wikipedia's does — there is no source feature — so the constant used to build the+training data has to be the constant the serving path uses. See `STAAN_TOP_SCORE`+in `mwmbl/tinysearchengine/staan.py`.++```bash+# 1. Add Staan's results to the existing Pass-2 pool (resumable; merge is idempotent).+DATABASE_URL="postgres://daoud@" uv run python scripts/llm_relabel_pass2_augment_staan.py+DATABASE_URL="postgres://daoud@" uv run python scripts/llm_relabel_pass2_augment_staan.py --merge++# 2. Judge only the pairs Staan added — Pass 3 is append-only and keyed by (query, url).+uv run python scripts/llm_relabel_pass3_judge.py --status+uv run python scripts/llm_relabel_pass3_judge.py --dump-batch 200 --tag staan1 > batch.txt+uv run python scripts/llm_relabel_pass3_judge.py --merge judge_out.txt --tag staan1+uv run python scripts/llm_relabel_pass3_judge.py --calibration # what Staan is worth, blind++# 3. Rebuild learning-to-rank-llm.csv.gz with the `staan` pool in it.+uv run python scripts/llm_relabel_build_dataset.py++# 4. Train and gate. Keep --split-seed fixed across arms.+DATABASE_URL="postgres://daoud@" uv run python -m mwmbl.rankeval.ltr.llm_experiment \+ --mode baseline --model-path mwmbl/resources/model.xgb --note deployed+DATABASE_URL="postgres://daoud@" uv run python -m mwmbl.rankeval.ltr.llm_experiment \+ --mode mixed --ext-weight 0.25 --overall-threshold 4 --scale-pos-weight 1.0 \+ --add-curation --curation-weight 0.5 --note combined \+ --save-model devdata/rankeval-2026-04/model-combined.xgb+```++`llm_experiment` reports two axes and **both** gate a retrain: NDCG on a held-out+split of the Haiku grades, and pair accuracy on a held-out split of the human+curation pairs. The deployed model set the precedent at NDCG@10 0.744 / human+pair-accuracy 0.818; a regression on the human axis is a blocker even when the+Haiku axis improves.