An open source, non-profit web search engine
has_text lost again at even coverage, so record that and stop asking for a backfill
Daoud Clarke2e18879fa520
diff --git a/mwmbl/management/commands/train_domain_moderation_model.py b/mwmbl/management/commands/train_domain_moderation_model.pyindex 43cb88f..e772276 100644--- a/mwmbl/management/commands/train_domain_moderation_model.py+++ b/mwmbl/management/commands/train_domain_moderation_model.py@@ -34,6 +34,12 @@ from mwmbl.moderation.train import passes_gate, trainMIN_REAL_ROWS_PER_REASON = 30DERIVED_ROWS_PER_REASON = 300+# How far the train and test text coverage may differ before the ablation stops comparing like+# with like. Evidence is crawled newest-first, so before a backfill the two sides can sit at 46%+# and 92%, at which point has_text measures recency and the text vocabulary is fitted mostly on+# a different period than it is tested on.+COVERAGE_GAP_TOLERANCE = 0.15+class Command(BaseCommand):help = "Train the domain moderation approve/reject suggester"@@ -140,14 +146,32 @@ class Command(BaseCommand):f"{self._with_interval(at['reject_recall'], at['reject_recall_ci'])} "f"{at['unsure_share']:>7.3f}")- cold = full.get("cold_start", {})- self.stdout.write(- f"\n {full.get('train_rows_with_text', 0)} of "- f"{sum(full.get('train_rows_by_source', {}).values())} training rows and "- f"{cold.get('rows_with_text', 0)} of {cold.get('rows', 0)} cold-start test rows "- f"have text.\n A large gap between those two makes has_text a proxy for recency "- f"rather than for evidence, which is why it is off by default. Backfill evidence "- f"for the older\n submissions to close the gap, then re-read this table.\n")+ self.stdout.write("\n " + self._coverage_note(full) + "\n")++ @staticmethod+ def _coverage_note(metrics: dict) -> str:+ """How evenly the page text is spread across the split, and what that means to read.++ Conditional, because the advice inverts. While evidence is crawled newest-first the two+ sides disagree badly and has_text is a proxy for recency rather than for evidence, so+ the table cannot be read at face value and the fix is a backfill. Once the backfill has+ run they agree, the table means what it says - and telling someone to close a gap they+ have already closed is worse than saying nothing.+ """+ cold = metrics.get("cold_start", {})+ train_rows = sum(metrics.get("train_rows_by_source", {}).values())+ train_share = metrics.get("train_rows_with_text", 0) / train_rows if train_rows else 0+ test_share = (cold.get("rows_with_text", 0) / cold["rows"]) if cold.get("rows") else 0++ coverage = (f"{metrics.get('train_rows_with_text', 0)} of {train_rows} training rows "+ f"({train_share:.0%}) and {cold.get('rows_with_text', 0)} of "+ f"{cold.get('rows', 0)} cold-start test rows ({test_share:.0%}) have text.")+ if abs(train_share - test_share) > COVERAGE_GAP_TOLERANCE:+ return (coverage + "\n That gap makes has_text a proxy for recency rather than "+ "for evidence, and skews the text block's\n vocabulary towards the test "+ "period. Run backfill_domain_evidence over the older submissions\n and "+ "re-read this table before drawing conclusions from it.")+ return coverage + " Coverage is even, so these rows compare like for like."@staticmethoddef _with_interval(value, interval) -> str:diff --git a/mwmbl/moderation/artifacts/README.md b/mwmbl/moderation/artifacts/README.mdindex f097f80..ea8d893 100644--- a/mwmbl/moderation/artifacts/README.md+++ b/mwmbl/moderation/artifacts/README.md@@ -33,13 +33,17 @@ its score distribution drifts through the two the server serves — read `rejectmodel, the same model plus the `has_text` indicator, and the domain-name-only model on the samesplit; `--no-text` and `--has-text` fit those variants directly.-Measured on production data, the page-text vocabulary is worth **+0.143 normalised AP** on the-cold-start slice (0.535 against 0.392) and halves the share of submissions the tool declines to-judge (`unsure` 0.118 against 0.249). The `has_text` indicator is **off**, because it made every-metric worse — normalised AP 0.535 → 0.521, reject precision 0.657 → 0.636, reject recall 0.121-→ 0.111. Evidence is crawled newest-first, so with 46% of training rows and 92% of cold-start-test rows carrying text the indicator is a proxy for recency, and recency is when the .ai spam-wave happened. Re-measure it with `--ablate` once `backfill_domain_evidence` has closed that gap.+Measured on production data with evidence backfilled to even coverage (94% of training rows, 92%+of cold-start test rows), the page-text vocabulary is worth **+0.230 normalised AP** on the+cold-start slice — 0.622 against 0.392 for domain-name-only — and lifts reject recall from 0.111+to 0.153. It is the largest single lever measured on this model.++The `has_text` indicator is **off**. It was measured twice and lost twice: at skewed coverage+(0.535 against 0.521) and again at even coverage, by more (0.622 against 0.598, reject precision+0.707 against 0.682). Once nearly everything has been crawled it is close to a constant, and the+rows it does fire on — the ones nothing could fetch — are already settled decisively by the+liveness check in `rules.py`. `--ablate` still reports the row, so a crawl that silently degrades+would show up as the indicator regaining value.To reproduce the numbers without a production database, train from the sanitized judgments exportinstead:diff --git a/mwmbl/moderation/artifacts/model.joblib b/mwmbl/moderation/artifacts/model.joblibindex e2f7b99..228dda0 100644Binary files a/mwmbl/moderation/artifacts/model.joblib and b/mwmbl/moderation/artifacts/model.joblib differdiff --git a/mwmbl/moderation/features.py b/mwmbl/moderation/features.pyindex 13c45c6..aef8d00 100644--- a/mwmbl/moderation/features.py+++ b/mwmbl/moderation/features.py@@ -105,17 +105,26 @@ class Featuriser:cold-start slice (0.535 -> 0.392) and doubles the share of submissions the tool declinesto have an opinion about (0.118 -> 0.249).- ``has_text`` is **off by default, because it was measured and did not help.** The idea- was sound - an uncrawled domain and a page whose words are all out of vocabulary produce- the same all-zero TF-IDF row, and they are not the same thing - but evidence is crawled- newest-first, so under a chronological split the indicator is also a proxy for *recency*- (46% of training rows against 92% of cold-start test rows), and recency is when the .ai- spam wave happened. Holding it out improved every metric measured: normalised AP 0.521- -> 0.535, reject precision 0.636 -> 0.657, reject recall 0.111 -> 0.121. Small against- their intervals, individually; consistent across all of them.-- The flag stays so it can be re-measured once ``backfill_domain_evidence`` has evened out- the coverage, which is the condition under which it would start meaning what it says.+ ``has_text`` is **off, because it was measured twice and lost twice.** The idea was+ sound - an uncrawled domain and a page whose words are all out of vocabulary produce the+ same all-zero TF-IDF row, and they are not the same thing - and the first result had an+ obvious excuse: evidence was crawled newest-first, so at 46% training coverage against+ 92% test coverage the indicator was a proxy for *recency*, and recency is when the .ai+ spam wave happened. Holding it out gave normalised AP 0.521 -> 0.535, reject precision+ 0.636 -> 0.657.++ Backfilling evidence over the older submissions closed that gap - 94% against 92% - and+ it lost by *more*: 0.598 -> 0.622 normalised AP, 0.682 -> 0.707 reject precision. Which+ is the answer, and in hindsight the obvious one. When almost everything has been crawled+ the indicator is very nearly a constant, so it carries no information and only spends a+ parameter. The rows it does fire on are the ones nothing could fetch, and those are+ already settled decisively and exactly by the liveness check in mwmbl.moderation.rules,+ for the reason this module's own docstring gives: the crawl happens now and the labels+ are up to two years old, so "could not be fetched" is a fact about today, not about the+ decision. It was redundant with a deterministic check the whole time.++ The flag stays because ``--ablate`` still reports the row, and a crawl that silently+ degrades would show up there as the indicator regaining value."""self.use_text = use_textself.use_has_text = use_has_textdiff --git a/scripts/moderation_eval.py b/scripts/moderation_eval.pyindex 93244b1..eff45e4 100644--- a/scripts/moderation_eval.py+++ b/scripts/moderation_eval.py@@ -90,8 +90,11 @@ def main() -> int:exclude={row.domain for row in rows})# Dated like a retrain's, because this artifact is served in production until the first- # retrain publishes and "offline-eval" in a worker log says nothing about how old it is.- version = (f"domain-mod-{date.today():%Y-%m-%d}" if options.write_artifact+ # retrain publishes and "offline-eval" in a worker log says nothing about how old it is -+ # but suffixed, because it is *not* one. DomainEvidence.model_version records which model+ # scored a row, and a warm start sharing a retrain's version string makes "has the rescore+ # run yet?" unanswerable from the data.+ version = (f"domain-mod-{date.today():%Y-%m-%d}-warmstart" if options.write_artifactelse "offline-eval")model, metrics = train(rows, version)print(json.dumps(metrics, indent=2))