Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Super Search: gold-grounded source-selection eval (Phase 2)

Daoud Clarke4307a96f7997
diff --git a/.gitignore b/.gitignore
index dc840b2..6e8bc41 100644
--- a/.gitignore
+++ b/.gitignore
@@ -188,6 +188,14 @@ scripts/downloads/
# Super Search candidate classification intermediates
candidate_chunks/
+recipe_chunks/
+
+# Super Search data artifacts (inputs/outputs of scripts/, not source)
+curated-domains.json
+super-search-targets.json
+prescreen.json
+devdata/ss_eval_matrix*
+devdata/ss_gold_matrix*
# AI planning documents
plans/
diff --git a/mwmbl/rankeval/SUPER_SEARCH_GOLD_SELECTION_FINDINGS.md b/mwmbl/rankeval/SUPER_SEARCH_GOLD_SELECTION_FINDINGS.md
new file mode 100644
index 0000000..6a638c1
--- /dev/null
+++ b/mwmbl/rankeval/SUPER_SEARCH_GOLD_SELECTION_FINDINGS.md
@@ -0,0 +1,129 @@
+# Super Search: gold-grounded source selection — findings
+
+_Branch `super-search-intent-feature`, June 2026._
+
+## Why this exists
+
+Source selection (which ~10 of ~134 sources to query per request) has so far been
+evaluated against a **circular proxy**: `scripts/super_search_eval.py build-matrix`
+rewards a source by the fraction of its results that survive into the LTR model's
+*own* top-K (`devdata/ss_eval_matrix.*`). That measures agreement with the ranker,
+not real relevance.
+
+This is the Phase-2 step foreshadowed in `scripts/super_search_coverage.py`: build a
+**gold-grounded** reward matrix and selector, so we score a source by whether it
+actually contains a *gold-relevant* result. The Phase-1 coverage gate already warned
+the ceiling is low (~0.36% of gold mass on our source domains); this work measures the
+headroom inside that subset with a real selector.
+
+## What was built
+
+- `scripts/super_search_eval.py build-gold-matrix` — fully **offline** (no network).
+ Reads the LTR dataset (`devdata/rankeval-2026-04/learning-to-rank.csv.gz`), maps each
+ row's URL to its source by registrable domain, and writes a `RewardMatrix`
+ (`devdata/ss_gold_matrix.*`) where:
+ - `mask[q, s]` = source `s` appears in query `q`'s LTR candidate rows (availability),
+ - `R[q, s]` = **1.0 if any of those rows is gold-relevant** (non-null
+ `gold_standard_rank`), else 0.0 (binary has-gold).
+ - Cosine-feature source profiles are accumulated from the rows' title/extract text.
+- Shared domain helpers factored into
+ `mwmbl/tinysearchengine/super_search_select/domains.py` (`host_of`, `registrable`,
+ `source_domain_map`), reused by `super_search_coverage.py`.
+- Evaluation reuses the existing `select` / `simulate` subcommands unchanged.
+
+## The shape of the data
+
+LTR dataset: 252,003 rows / 2,096 queries.
+
+- **In-coverage queries** (≥1 candidate on a source domain): **2,009 (96%)** — most
+ queries do touch a source domain (github/stackoverflow/etc. are everywhere).
+- **Queries with a gold-bearing source: 56 (2.8%)** — this is the hard ceiling, and it
+ matches the gate's ~0.36% gold-mass finding.
+- Availability is *not* sparse: mean 6.16 sources/query (median 6, max 23), so top-1/2/3
+ selection has genuine room to act.
+- Every gold query has **exactly one** gold-bearing source (56 gold cells / 56 queries).
+
+Where the 56 gold queries live (gold-query count / availability):
+
+| source | gold queries | available in |
+|---|---:|---:|
+| www_gov_uk | 24 | 58 |
+| imdb | 15 | 62 |
+| github | 5 | 1,772 |
+| devblogs_microsoft_com | 5 | 64 |
+| genius_com | 3 | 46 |
+| duckduckgo_com / nist / space / arxiv | 1 each | — |
+
+The recoverable gold is **concentrated in gov.uk + imdb (39 of 56)** — the two sources
+added in `0d10c78` precisely for their gold mass. github is available in 1,772 queries
+but is the gold answer in only 5 (very low precision).
+
+## Results
+
+### Held-out selector quality (`select`, XGBoost grouped CV by query, coverage@k vs oracle on the 56 gold queries)
+
+| k | coverage@k | top features (ablation drop) |
+|---:|---:|---|
+| 1 | 0.464 | estimated_pages +0.232, intent_news +0.089, popularity +0.036 |
+| 2 | 0.696 | estimated_pages +0.286, popularity +0.089 |
+| 3 | 0.768 | estimated_pages +0.196, popularity +0.054 |
+
+A learned model can place the gold-bearing source in the top-3 ~77% of the time.
+
+### Policy replay (`simulate`, mean captured reward / query over all 2,009; oracle = 0.0279)
+
+| k | oracle | TS (best ν) | cosine | random | popularity |
+|---:|---:|---:|---:|---:|---:|
+| 1 | 0.0279 | **0.0184** (ν=0.25) | 0.0070 | 0.0050 | 0.0035 |
+| 2 | 0.0279 | **0.0249** (ν=0.5) | 0.0124 | 0.0129 | 0.0075 |
+| 3 | 0.0279 | **0.0254** (ν=0.5) | 0.0169 | 0.0139 | 0.0090 |
+
+The learned bandit beats the cosine baseline by ~2.6× at k=1 and reaches ~91% of oracle
+by k=3.
+
+## Interpretation
+
+1. **The ceiling is tiny — 56 queries (2.8%).** Even a perfect selector moves relevance
+ on <3% of queries, with bounded NDCG upside. This corroborates the coverage gate and
+ the add-sources track (`SUPER_SEARCH_ADD_SOURCES_FINDINGS.md`).
+
+2. **The production cosine signal is misaligned with gold.** Across every k, the gold
+ reward is driven by `estimated_pages` and `popularity` (source size/quality priors),
+ while `cos_bow` / `cos_cng` — the features the live `_select_cosine` baseline ranks on
+ — have ~zero or **negative** ablation. Query↔source cosine does not predict which
+ source holds the gold result.
+
+3. **"Learned beats cosine" is really "always include gov.uk + imdb."** The gold mass is
+ carried by gov.uk and imdb, each available in only ~60 queries, and every gold query
+ has a single gold source. A learned policy succeeds by learning a near-static per-source
+ prior (favour the high-gold sources whenever present) — which is exactly what
+ `SUPER_SEARCH_FORCE_INCLUDE` already does for them.
+
+## Recommendation: static, not learned
+
+The data says go static, consistent with the gate's stated default:
+
+- **Force-include the handful of high-gold sources** (gov.uk, imdb already in
+ `SUPER_SEARCH_FORCE_INCLUDE`; consider github/devblogs by size prior) rather than
+ shipping a per-query learned selector — 56 single-source gold queries do not justify a
+ contextual model.
+- **If selection is tuned at all, rank by `popularity`/`estimated_pages`, not cosine** —
+ the cosine baseline optimizes a signal uncorrelated with gold relevance.
+- **Keep investing in adding high-gold-mass domains** (the add-sources track) — that, not
+ selection among existing sources, is where the gold mass is.
+
+## Caveats
+
+- 56 gold queries → small CV folds; absolute coverage@k numbers are noisy and
+ `estimated_pages` dominance partly reflects that gold-bearing sources happen to be large
+ (gov.uk, imdb, github). Directional, not precise.
+- Reward is binary has-gold; it ignores *where* in gold the URL ranked. A graded variant
+ would refine the within-subset story but not the ceiling.
+
+## Reproduce
+
+```
+DJANGO_SETTINGS_MODULE=mwmbl.settings_dev uv run python scripts/super_search_eval.py build-gold-matrix --out devdata/ss_gold_matrix
+uv run python scripts/super_search_eval.py select --matrix devdata/ss_gold_matrix --k 2
+uv run python scripts/super_search_eval.py simulate --matrix devdata/ss_gold_matrix --k 2
+```
diff --git a/mwmbl/tinysearchengine/super_search_select/domains.py b/mwmbl/tinysearchengine/super_search_select/domains.py
new file mode 100644
index 0000000..bebfe2c
--- /dev/null
+++ b/mwmbl/tinysearchengine/super_search_select/domains.py
@@ -0,0 +1,65 @@
+"""URL/host normalization and source-domain matching for Super Search.
+
+Shared by the offline evaluation harnesses (``scripts/super_search_coverage.py``,
+``scripts/super_search_eval.py``) that need to decide which Super Search source a
+URL belongs to. Matching is by *registrable domain* (``www.``/``m.`` stripped,
+last two labels — three for known multi-label suffixes like ``co.uk``); exact-host
+string equality is what the NDCG harness scores on, but for "which source contains
+this result" the registrable domain is the right granularity.
+"""
+from __future__ import annotations
+
+from collections import defaultdict
+from urllib.parse import urlparse
+
+# Common multi-label public suffixes so registrable-domain folding doesn't collapse
+# e.g. bbc.co.uk -> co.uk. Not exhaustive (no full public-suffix list), but covers
+# the suffixes that actually appear in the gold set / source shortlist.
+_MULTI_SUFFIXES = {
+ "co.uk", "org.uk", "ac.uk", "gov.uk", "me.uk", "ltd.uk", "plc.uk",
+ "com.au", "net.au", "org.au", "edu.au", "gov.au",
+ "co.nz", "co.za", "co.in", "co.jp", "co.kr", "com.br", "com.mx",
+ "com.cn", "com.tr", "com.sg", "com.hk", "or.jp", "ne.jp",
+}
+
+
+def host_of(url: str) -> str:
+ """Lowercase host of a URL, sans userinfo and port. ``""`` if unparseable."""
+ try:
+ netloc = urlparse(str(url)).netloc.lower()
+ except ValueError:
+ return ""
+ return netloc.split("@")[-1].split(":")[0]
+
+
+def _strip_www(host: str) -> str:
+ for prefix in ("www.", "m."):
+ if host.startswith(prefix):
+ return host[len(prefix):]
+ return host
+
+
+def registrable(host: str) -> str:
+ """Registrable domain: strip ``www.``/``m.`` then take the last 2 labels (3 for
+ known multi-label suffixes like ``co.uk``)."""
+ host = _strip_www(host)
+ labels = host.split(".")
+ if len(labels) <= 2:
+ return host
+ if ".".join(labels[-2:]) in _MULTI_SUFFIXES:
+ return ".".join(labels[-3:])
+ return ".".join(labels[-2:])
+
+
+def source_domain_map() -> dict[str, list[str]]:
+ """Registrable source domain -> source names, built from the source registry.
+
+ A single registrable domain can map to several sources (e.g. sibling
+ subdomains of the same registrable domain).
+ """
+ from mwmbl.tinysearchengine.super_search_select.registry import get_registry
+
+ reg_map: dict[str, list[str]] = defaultdict(list)
+ for name, meta in get_registry().items():
+ reg_map[registrable(meta.domain.lower())].append(name)
+ return dict(reg_map)
diff --git a/scripts/super_search_coverage.py b/scripts/super_search_coverage.py
index 733e12e..14e72ff 100644
--- a/scripts/super_search_coverage.py
+++ b/scripts/super_search_coverage.py
@@ -40,50 +40,13 @@ from mwmbl.rankeval.paths import ( # noqa: E402
RANKINGS_DATASET_TEST_PATH,
RANKINGS_DATASET_TRAIN_PATH,
)
+from mwmbl.tinysearchengine.super_search_select.domains import host_of, registrable # noqa: E402
from mwmbl.tinysearchengine.super_search_select.features import ( # noqa: E402
INTENT_NAMES,
classify_intent,
)
from mwmbl.tinysearchengine.super_search_select.registry import get_registry # noqa: E402
-# Common multi-label public suffixes so registrable-domain folding doesn't collapse
-# e.g. bbc.co.uk -> co.uk. Not exhaustive (no full public-suffix list), but covers
-# the suffixes that actually appear in the gold set / source shortlist.
-_MULTI_SUFFIXES = {
- "co.uk", "org.uk", "ac.uk", "gov.uk", "me.uk", "ltd.uk", "plc.uk",
- "com.au", "net.au", "org.au", "edu.au", "gov.au",
- "co.nz", "co.za", "co.in", "co.jp", "co.kr", "com.br", "com.mx",
- "com.cn", "com.tr", "com.sg", "com.hk", "or.jp", "ne.jp",
-}
-
-
-def host_of(url: str) -> str:
- """Lowercase host of a URL, sans userinfo and port. ``""`` if unparseable."""
- try:
- netloc = urlparse(str(url)).netloc.lower()
- except ValueError:
- return ""
- return netloc.split("@")[-1].split(":")[0]
-
-
-def _strip_www(host: str) -> str:
- for prefix in ("www.", "m."):
- if host.startswith(prefix):
- return host[len(prefix):]
- return host
-
-
-def registrable(host: str) -> str:
- """Registrable domain: strip ``www.``/``m.`` then take the last 2 labels (3 for
- known multi-label suffixes like ``co.uk``)."""
- host = _strip_www(host)
- labels = host.split(".")
- if len(labels) <= 2:
- return host
- if ".".join(labels[-2:]) in _MULTI_SUFFIXES:
- return ".".join(labels[-3:])
- return ".".join(labels[-2:])
-
def load_gold() -> pd.DataFrame:
frames = []
diff --git a/scripts/super_search_eval.py b/scripts/super_search_eval.py
index d4c6790..a1ebb99 100644
--- a/scripts/super_search_eval.py
+++ b/scripts/super_search_eval.py
@@ -7,6 +7,14 @@ Subcommands:
union with the LTR model, and write a dense (queries x sources)
reward + feature matrix. Requires network + Django.
+ build-gold-matrix
+ - build the same kind of matrix offline from the LTR dataset's
+ gold labels (no network): a (query x source) cell is "available"
+ (mask) if that source's domain appears in the query's LTR rows,
+ and gets reward 1.0 if any of those rows is gold-relevant. Lets
+ us learn/evaluate source selection against real gold relevance
+ instead of the LTR-top-K-survival proxy.
+
select - fit the XGBoost reward model on the matrix (grouped CV by
query) and print each feature's ablation drop in coverage@k.
@@ -22,8 +30,10 @@ Usage:
"""
import argparse
import asyncio
+import math
import os
import sys
+from collections import defaultdict
from pathlib import Path
import django
@@ -33,6 +43,12 @@ REPO_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(REPO_ROOT))
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mwmbl.settings_dev")
+# host_of/registrable are pure stdlib helpers (no Django) — safe to import at module
+# load, before django.setup(); the registry-backed source_domain_map is loaded lazily.
+from mwmbl.tinysearchengine.super_search_select.domains import ( # noqa: E402
+ host_of, registrable,
+)
+
def _bootstrap_django():
django.setup()
@@ -135,6 +151,134 @@ def build_matrix(queries: list[str], out: str):
f"{int(mask.sum())} filled cells.")
+# ---------------------------------------------------------------------------
+# build-gold-matrix (offline: LTR dataset gold labels, no network)
+# ---------------------------------------------------------------------------
+
+def _is_gold(rank) -> bool:
+ """True if ``rank`` is a real gold rank (not pandas NaN / None / blank)."""
+ if rank is None:
+ return False
+ if isinstance(rank, float) and math.isnan(rank):
+ return False
+ if isinstance(rank, str) and not rank.strip():
+ return False
+ return True
+
+
+def attribute_rows(rows, reg_map):
+ """Attribute LTR rows to Super Search sources by registrable domain (pure).
+
+ ``rows`` is an iterable of ``(query, url, title, extract, gold_standard_rank)``
+ and ``reg_map`` maps a registrable domain to the source names on it. Returns
+ ``(per_query, prof_text)`` where ``per_query[query][source]`` is True iff that
+ source has a gold-relevant row for the query (else False = available-but-not-gold),
+ and ``prof_text[source]`` accumulates that source's title/extract text.
+ """
+ per_query: dict[str, dict[str, bool]] = {} # query -> {source -> has_gold}
+ prof_text: dict[str, list[str]] = defaultdict(list)
+ for query, url, title, extract, gold in rows:
+ names = reg_map.get(registrable(host_of(str(url))))
+ if not names:
+ continue
+ is_gold = _is_gold(gold)
+ q_sources = per_query.setdefault(str(query), {})
+ text = f"{title or ''} {extract or ''}"
+ for name in names:
+ q_sources[name] = q_sources.get(name, False) or is_gold
+ prof_text[name].append(text)
+ return per_query, prof_text
+
+
+def build_gold_matrix(out: str):
+ """Build a (query x source) reward matrix from the LTR dataset's gold labels.
+
+ Unlike ``build-matrix`` (which queries live sources and rewards a source by how
+ many of its results survive the LTR model's *own* top-K), this is fully offline
+ and grounds the reward in real gold relevance:
+
+ * mask[q, s] = source ``s``'s registrable domain appears in query ``q``'s LTR
+ rows (the offline "this source contributed a candidate" signal);
+ * R[q, s] = 1.0 if any such row is gold-relevant (non-null
+ ``gold_standard_rank``), else 0.0 (binary has-gold).
+
+ Source content profiles for the cosine features are accumulated from the LTR
+ rows' title/extract text per source, so no network access is needed.
+ """
+ import pandas as pd
+ from django.conf import settings
+ from mwmbl.rankeval.paths import LEARNING_TO_RANK_DATASET_PATH
+ from mwmbl.tinysearchengine.super_search_select import vectors
+ from mwmbl.tinysearchengine.super_search_select.domains import source_domain_map
+ from mwmbl.tinysearchengine.super_search_select.features import (
+ FEATURE_NAMES, QueryContext, feature_vector,
+ )
+ from mwmbl.tinysearchengine.super_search_select.registry import get_meta, get_registry
+ from mwmbl.tinysearchengine.super_search_select.evaluation import RewardMatrix
+
+ dim = settings.SUPER_SEARCH_PROJECTION_DIM
+ F = len(FEATURE_NAMES)
+
+ sources = list(get_registry().keys())
+ s_index = {name: i for i, name in enumerate(sources)}
+ reg_map = source_domain_map() # registrable domain -> [source names]
+
+ df = pd.read_csv(LEARNING_TO_RANK_DATASET_PATH, lineterminator="\n")
+ print(f"Loaded LTR dataset: {len(df)} rows, {df['query'].nunique()} queries")
+
+ # Pass 1: attribute every row to its source(s), recording per-(query, source)
+ # availability + gold and accumulating per-source content text for the profiles.
+ per_query, prof_text = attribute_rows(
+ zip(df["query"], df["url"], df["title"], df["extract"], df["gold_standard_rank"]),
+ reg_map,
+ )
+
+ # In-coverage queries only: those with >=1 in-source candidate row.
+ in_cov_queries = list(per_query.keys())
+ profile = {
+ n: (
+ vectors.project_bow(" ".join(prof_text[n]), dim) if prof_text[n] else None,
+ vectors.project_char_ngrams(" ".join(prof_text[n]), dim) if prof_text[n] else None,
+ )
+ for n in sources
+ }
+
+ # Pass 2: features + binary has-gold reward.
+ Q, S = len(in_cov_queries), len(sources)
+ X = np.zeros((Q, S, F))
+ R = np.zeros((Q, S))
+ mask = np.zeros((Q, S), dtype=bool)
+ for qi, query in enumerate(in_cov_queries):
+ bow = vectors.project_bow(query, dim)
+ cng = vectors.project_char_ngrams(query, dim)
+ qctx = QueryContext.build(query, bow, cng)
+ for name, has_gold in per_query[query].items():
+ si = s_index[name]
+ X[qi, si] = feature_vector(qctx, get_meta(name), profile[name])
+ mask[qi, si] = True
+ if has_gold:
+ R[qi, si] = 1.0
+
+ matrix = RewardMatrix(queries=in_cov_queries, sources=sources,
+ feature_names=list(FEATURE_NAMES), X=X, R=R, mask=mask)
+ matrix.save(out)
+
+ # Diagnostics: does selection even have room to matter? If most queries have
+ # few in-source candidates, "query them all" is as good as any learned policy.
+ avail_per_q = mask.sum(axis=1)
+ gold_per_q = (R > 0).sum(axis=1)
+ n_gold_q = int((gold_per_q > 0).sum())
+ print(f"\nWrote matrix {out}.npz/.json: {Q} in-coverage queries x {S} sources, "
+ f"{int(mask.sum())} filled cells.")
+ print(f" queries with >=1 gold source: {n_gold_q} ({100*n_gold_q/max(Q,1):.1f}%)")
+ print(f" mean available sources / query: {avail_per_q.mean():.2f} "
+ f"(median {int(np.median(avail_per_q))}, max {int(avail_per_q.max())})")
+ for k in (1, 2, 3, 5, 10):
+ n = int((avail_per_q > k).sum())
+ print(f" queries with > {k:>2} available sources: {n:>4} "
+ f"({100*n/max(Q,1):.1f}%) <- room for top-{k} selection to matter")
+
+
# ---------------------------------------------------------------------------
# select / simulate (pure, no network)
# ---------------------------------------------------------------------------
@@ -175,6 +319,9 @@ def main():
p_build.add_argument("--queries", required=True, help="file with one query per line")
p_build.add_argument("--out", default="eval_matrix")
+ p_gold = sub.add_parser("build-gold-matrix")
+ p_gold.add_argument("--out", default="devdata/ss_gold_matrix")
+
p_select = sub.add_parser("select")
p_select.add_argument("--matrix", default="eval_matrix")
p_select.add_argument("--k", type=int, default=10)
@@ -189,6 +336,8 @@ def main():
if args.command == "build-matrix":
queries = [ln.strip() for ln in Path(args.queries).read_text().splitlines() if ln.strip()]
build_matrix(queries, args.out)
+ elif args.command == "build-gold-matrix":
+ build_gold_matrix(args.out)
elif args.command == "select":
cmd_select(args.matrix, args.k)
elif args.command == "simulate":
diff --git a/test/test_super_search_eval.py b/test/test_super_search_eval.py
index 3edbb42..52ac0a0 100644
--- a/test/test_super_search_eval.py
+++ b/test/test_super_search_eval.py
@@ -1,12 +1,27 @@
"""Tests for the offline evaluation harness on synthetic reward matrices."""
+import importlib.util
+from pathlib import Path
+
import numpy as np
import pytest
from mwmbl.tinysearchengine.super_search_select import evaluation
+from mwmbl.tinysearchengine.super_search_select.domains import (
+ host_of, registrable, source_domain_map,
+)
from mwmbl.tinysearchengine.super_search_select.evaluation import RewardMatrix
from mwmbl.tinysearchengine.super_search_select.features import FEATURE_NAMES
+def _load_eval_script():
+ """Load scripts/super_search_eval.py by path (it isn't an importable package)."""
+ path = Path(__file__).resolve().parents[1] / "scripts" / "super_search_eval.py"
+ spec = importlib.util.spec_from_file_location("super_search_eval_script", path)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
def _synthetic_matrix(Q=120, S=30, seed=0) -> RewardMatrix:
"""Reward driven mostly by cos_bow (idx 1) and popularity (idx 3)."""
rng = np.random.default_rng(seed)
@@ -82,3 +97,59 @@ def test_feature_selection_flags_cos_bow(tmp_path):
drops = result["ablation_drop"]
# cos_bow is the dominant driver, so removing it should hurt coverage most.
assert drops["cos_bow"] == max(drops.values())
+
+
+# ---------------------------------------------------------------------------
+# Domain matching (shared helpers behind the gold-grounded matrix)
+# ---------------------------------------------------------------------------
+
+def test_registrable_folds_subdomains_and_multi_suffixes():
+ assert registrable("www.github.com") == "github.com"
+ assert registrable("m.example.org") == "example.org"
+ assert registrable("news.ycombinator.com") == "ycombinator.com"
+ assert registrable("foo.bbc.co.uk") == "bbc.co.uk" # multi-label suffix kept
+ assert registrable("example.com") == "example.com"
+
+
+def test_host_of_normalises_and_handles_junk():
+ assert host_of("https://[email protected]:443/x") == "news.ycombinator.com"
+ assert host_of("not a url") == ""
+
+
+def test_source_domain_map_groups_sources_by_registrable_domain():
+ m = source_domain_map()
+ assert "github" in m.get("github.com", [])
+ assert "hn" in m.get("ycombinator.com", []) # news.ycombinator.com -> ycombinator.com
+
+
+# ---------------------------------------------------------------------------
+# Gold-grounded matrix construction (build-gold-matrix core logic)
+# ---------------------------------------------------------------------------
+
+def test_is_gold_treats_nan_none_blank_as_not_gold():
+ mod = _load_eval_script()
+ assert mod._is_gold(1) and mod._is_gold("3")
+ assert not mod._is_gold(None)
+ assert not mod._is_gold(float("nan"))
+ assert not mod._is_gold("")
+ assert not mod._is_gold(" ")
+
+
+def test_attribute_rows_binary_gold_and_in_coverage_filter():
+ mod = _load_eval_script()
+ reg_map = {"github.com": ["github"], "stackoverflow.com": ["stackexchange"]}
+ rows = [
+ ("q1", "https://github.com/a", "ta", "ea", float("nan")), # available, not gold
+ ("q1", "https://github.com/b", "tb", "eb", 3), # gold -> github True
+ ("q1", "https://stackoverflow.com/x", "tx", "ex", None), # available, not gold
+ ("q2", "https://example.com/none", "t", "e", 1), # off-source -> dropped
+ ]
+ per_query, prof_text = mod.attribute_rows(rows, reg_map)
+
+ # q2 had no in-source row, so it's filtered out of the in-coverage set.
+ assert set(per_query) == {"q1"}
+ # binary has-gold: github saw a gold row, stackexchange only non-gold.
+ assert per_query["q1"] == {"github": True, "stackexchange": False}
+ # both github rows accumulate into the source's content profile text.
+ assert len(prof_text["github"]) == 2
+ assert len(prof_text["stackexchange"]) == 1