Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

has_text lost again at even coverage, so record that and stop asking for a backfill

Daoud Clarke2e18879fa520
diff --git a/mwmbl/management/commands/train_domain_moderation_model.py b/mwmbl/management/commands/train_domain_moderation_model.py
index 43cb88f..e772276 100644
--- a/mwmbl/management/commands/train_domain_moderation_model.py
+++ b/mwmbl/management/commands/train_domain_moderation_model.py
@@ -34,6 +34,12 @@ from mwmbl.moderation.train import passes_gate, train
MIN_REAL_ROWS_PER_REASON = 30
DERIVED_ROWS_PER_REASON = 300
+# How far the train and test text coverage may differ before the ablation stops comparing like
+# with like. Evidence is crawled newest-first, so before a backfill the two sides can sit at 46%
+# and 92%, at which point has_text measures recency and the text vocabulary is fitted mostly on
+# a different period than it is tested on.
+COVERAGE_GAP_TOLERANCE = 0.15
+
class Command(BaseCommand):
help = "Train the domain moderation approve/reject suggester"
@@ -140,14 +146,32 @@ class Command(BaseCommand):
f"{self._with_interval(at['reject_recall'], at['reject_recall_ci'])} "
f"{at['unsure_share']:>7.3f}")
- cold = full.get("cold_start", {})
- self.stdout.write(
- f"\n {full.get('train_rows_with_text', 0)} of "
- f"{sum(full.get('train_rows_by_source', {}).values())} training rows and "
- f"{cold.get('rows_with_text', 0)} of {cold.get('rows', 0)} cold-start test rows "
- f"have text.\n A large gap between those two makes has_text a proxy for recency "
- f"rather than for evidence, which is why it is off by default. Backfill evidence "
- f"for the older\n submissions to close the gap, then re-read this table.\n")
+ self.stdout.write("\n " + self._coverage_note(full) + "\n")
+
+ @staticmethod
+ def _coverage_note(metrics: dict) -> str:
+ """How evenly the page text is spread across the split, and what that means to read.
+
+ Conditional, because the advice inverts. While evidence is crawled newest-first the two
+ sides disagree badly and has_text is a proxy for recency rather than for evidence, so
+ the table cannot be read at face value and the fix is a backfill. Once the backfill has
+ run they agree, the table means what it says - and telling someone to close a gap they
+ have already closed is worse than saying nothing.
+ """
+ cold = metrics.get("cold_start", {})
+ train_rows = sum(metrics.get("train_rows_by_source", {}).values())
+ train_share = metrics.get("train_rows_with_text", 0) / train_rows if train_rows else 0
+ test_share = (cold.get("rows_with_text", 0) / cold["rows"]) if cold.get("rows") else 0
+
+ coverage = (f"{metrics.get('train_rows_with_text', 0)} of {train_rows} training rows "
+ f"({train_share:.0%}) and {cold.get('rows_with_text', 0)} of "
+ f"{cold.get('rows', 0)} cold-start test rows ({test_share:.0%}) have text.")
+ if abs(train_share - test_share) > COVERAGE_GAP_TOLERANCE:
+ return (coverage + "\n That gap makes has_text a proxy for recency rather than "
+ "for evidence, and skews the text block's\n vocabulary towards the test "
+ "period. Run backfill_domain_evidence over the older submissions\n and "
+ "re-read this table before drawing conclusions from it.")
+ return coverage + " Coverage is even, so these rows compare like for like."
@staticmethod
def _with_interval(value, interval) -> str:
diff --git a/mwmbl/moderation/artifacts/README.md b/mwmbl/moderation/artifacts/README.md
index f097f80..ea8d893 100644
--- a/mwmbl/moderation/artifacts/README.md
+++ b/mwmbl/moderation/artifacts/README.md
@@ -33,13 +33,17 @@ its score distribution drifts through the two the server serves — read `reject
model, the same model plus the `has_text` indicator, and the domain-name-only model on the same
split; `--no-text` and `--has-text` fit those variants directly.
-Measured on production data, the page-text vocabulary is worth **+0.143 normalised AP** on the
-cold-start slice (0.535 against 0.392) and halves the share of submissions the tool declines to
-judge (`unsure` 0.118 against 0.249). The `has_text` indicator is **off**, because it made every
-metric worse — normalised AP 0.535 → 0.521, reject precision 0.657 → 0.636, reject recall 0.121
-→ 0.111. Evidence is crawled newest-first, so with 46% of training rows and 92% of cold-start
-test rows carrying text the indicator is a proxy for recency, and recency is when the .ai spam
-wave happened. Re-measure it with `--ablate` once `backfill_domain_evidence` has closed that gap.
+Measured on production data with evidence backfilled to even coverage (94% of training rows, 92%
+of cold-start test rows), the page-text vocabulary is worth **+0.230 normalised AP** on the
+cold-start slice — 0.622 against 0.392 for domain-name-only — and lifts reject recall from 0.111
+to 0.153. It is the largest single lever measured on this model.
+
+The `has_text` indicator is **off**. It was measured twice and lost twice: at skewed coverage
+(0.535 against 0.521) and again at even coverage, by more (0.622 against 0.598, reject precision
+0.707 against 0.682). Once nearly everything has been crawled it is close to a constant, and the
+rows it does fire on — the ones nothing could fetch — are already settled decisively by the
+liveness check in `rules.py`. `--ablate` still reports the row, so a crawl that silently degrades
+would show up as the indicator regaining value.
To reproduce the numbers without a production database, train from the sanitized judgments export
instead:
diff --git a/mwmbl/moderation/artifacts/model.joblib b/mwmbl/moderation/artifacts/model.joblib
index e2f7b99..228dda0 100644
Binary files a/mwmbl/moderation/artifacts/model.joblib and b/mwmbl/moderation/artifacts/model.joblib differ
diff --git a/mwmbl/moderation/features.py b/mwmbl/moderation/features.py
index 13c45c6..aef8d00 100644
--- a/mwmbl/moderation/features.py
+++ b/mwmbl/moderation/features.py
@@ -105,17 +105,26 @@ class Featuriser:
cold-start slice (0.535 -> 0.392) and doubles the share of submissions the tool declines
to have an opinion about (0.118 -> 0.249).
- ``has_text`` is **off by default, because it was measured and did not help.** The idea
- was sound - an uncrawled domain and a page whose words are all out of vocabulary produce
- the same all-zero TF-IDF row, and they are not the same thing - but evidence is crawled
- newest-first, so under a chronological split the indicator is also a proxy for *recency*
- (46% of training rows against 92% of cold-start test rows), and recency is when the .ai
- spam wave happened. Holding it out improved every metric measured: normalised AP 0.521
- -> 0.535, reject precision 0.636 -> 0.657, reject recall 0.111 -> 0.121. Small against
- their intervals, individually; consistent across all of them.
-
- The flag stays so it can be re-measured once ``backfill_domain_evidence`` has evened out
- the coverage, which is the condition under which it would start meaning what it says.
+ ``has_text`` is **off, because it was measured twice and lost twice.** The idea was
+ sound - an uncrawled domain and a page whose words are all out of vocabulary produce the
+ same all-zero TF-IDF row, and they are not the same thing - and the first result had an
+ obvious excuse: evidence was crawled newest-first, so at 46% training coverage against
+ 92% test coverage the indicator was a proxy for *recency*, and recency is when the .ai
+ spam wave happened. Holding it out gave normalised AP 0.521 -> 0.535, reject precision
+ 0.636 -> 0.657.
+
+ Backfilling evidence over the older submissions closed that gap - 94% against 92% - and
+ it lost by *more*: 0.598 -> 0.622 normalised AP, 0.682 -> 0.707 reject precision. Which
+ is the answer, and in hindsight the obvious one. When almost everything has been crawled
+ the indicator is very nearly a constant, so it carries no information and only spends a
+ parameter. The rows it does fire on are the ones nothing could fetch, and those are
+ already settled decisively and exactly by the liveness check in mwmbl.moderation.rules,
+ for the reason this module's own docstring gives: the crawl happens now and the labels
+ are up to two years old, so "could not be fetched" is a fact about today, not about the
+ decision. It was redundant with a deterministic check the whole time.
+
+ The flag stays because ``--ablate`` still reports the row, and a crawl that silently
+ degrades would show up there as the indicator regaining value.
"""
self.use_text = use_text
self.use_has_text = use_has_text
diff --git a/scripts/moderation_eval.py b/scripts/moderation_eval.py
index 93244b1..eff45e4 100644
--- a/scripts/moderation_eval.py
+++ b/scripts/moderation_eval.py
@@ -90,8 +90,11 @@ def main() -> int:
exclude={row.domain for row in rows})
# Dated like a retrain's, because this artifact is served in production until the first
- # retrain publishes and "offline-eval" in a worker log says nothing about how old it is.
- version = (f"domain-mod-{date.today():%Y-%m-%d}" if options.write_artifact
+ # retrain publishes and "offline-eval" in a worker log says nothing about how old it is -
+ # but suffixed, because it is *not* one. DomainEvidence.model_version records which model
+ # scored a row, and a warm start sharing a retrain's version string makes "has the rescore
+ # run yet?" unanswerable from the data.
+ version = (f"domain-mod-{date.today():%Y-%m-%d}-warmstart" if options.write_artifact
else "offline-eval")
model, metrics = train(rows, version)
print(json.dumps(metrics, indent=2))