An open source, non-profit web search engine
Moderate a domain, not a submission
Daoud Clarkeb7abe8fcfae4
diff --git a/docs/domain-moderation-gaps.md b/docs/domain-moderation-gaps.mdnew file mode 100644index 0000000..7ae2049--- /dev/null+++ b/docs/domain-moderation-gaps.md@@ -0,0 +1,147 @@+# Domain moderation: design vs. implementation++Design: <https://claude.ai/design/p/bca5ef6e-47c5-44e1-a45b-9e91d3483c53?file=Domain+Moderation+Alternatives.dc.html>++Turn **2a** ("Focus slot") is the built-out target. Turns 1a/1b/1c are the earlier alternatives it+was built from; 1a's status filter is the only thing from them still wanted (gap 8).++All nine gaps below are closed. What follows is what each one turned into, so the reasoning+behind the shape of the API stays attached to the design it came from.++## What the API returns now++`GET /domain-submissions/queue` is one row per **domain**, carrying everything one focus card and+its UP NEXT rows draw — so moving to the next card costs no request:++```+name, submission_count, first_submitted_on, first_submitted_by(+_username),+upvotes, downvotes, https, evidence_state, pages[], suggestion+```++`count` is distinct pending domains — the header's "24 pending". Default ordering is+`submissions` (`-submission_count, -upvotes, submitted_on`), which is the sort the screen labels;+`needs_review`, `confidence` and `oldest` are still accepted.++Decisions, undo and history are all addressed to a domain:++- `POST /domain-submissions/decisions` — `{"decisions": [{"domain": …, "status": …}]}`; each entry+ settles every submission of that name.+- `POST /domain-submissions/domains/{domain}/undo` — back to PENDING, rejection fields cleared.+- `GET /domain-submissions/moderated` — past moderations, one row per domain, filterable by+ `status` / `moderator` / `name`.++## Gap by gap++### 1. The queue is per domain++`one_row_per_domain` (`mwmbl/moderation/suggest.py`) keeps the **earliest** submission of each+name and annotates `submission_count` onto it.++Deliberately not a `GROUP BY`. The suggestion depends on the submitter's own track record, and a+domain submitted by two accounts has no single one to read — grouping would have meant rewriting+the withheld-approval rule and breaking the `suggestion_for` / `annotate_queue` parity that+`test_queue_display_matches_suggestion_for` holds together. Collapsing to a representative row+instead leaves both of them working on a submission. The earliest row is also the one the card+draws ("first submitted 6 days ago by anon_4417"), so the row we deduplicate to and the row we+show are the same row.++### 2. `submissions` ordering++`QUEUE_ORDERINGS["submissions"]` = `-submission_count, -upvotes, submitted_on`, and it is the+default. How many people asked for a domain is the one signal in the sort that is not the tool's+own opinion, which is why it leads rather than the suggestion's confidence.++### 3. Vote counts aggregated to the domain++`SearchResultVote` gained an indexed `domain` column, derived on `save()` by `utils.bare_host`+(normalise, drop a leading `www.`), with a batched backfill in migration 0036. `annotate_votes`+counts it in SQL against the submission name reduced the same way, so a vote on+`www.example.com/x` counts towards a submission of `example.com`. Coalesced to zero, because a+domain with no votes has to sort with the rest rather than scatter through it as NULL.++A rollup table was the alternative and was rejected: the live aggregate is one indexed subquery+and never stale.++### 4. Sample pages on queue rows++`pages` on `QueueItemSchema`, read off the `DomainEvidence` rows `_queue_items` already fetches+for the suggestion. No extra query.++### 5. Usernames++`first_submitted_by_username` on the queue row and `submitted_by_username` on the detail, both+via `select_related("submitted_by")` — a join, not a query. The numeric ids stay, for filtering.++### 6. The https padlock++`crawl_domain` now retries `http://` once when the https fetch **errors** (not when it answers+with a non-2xx), and records `signals["https"]`. A 404 over https is an answer; asking the same+server the same question without TLS gets the same one.++This changes suggestions: a site serving only over plain HTTP used to be `unreachable` — a+decisive REJECT at 0.9 — and is now reachable with a neutral `no_tls` evidence line. Neutral for+the same reason `robots` is: plenty of the small personal sites this index exists for have no+certificate, and no rejection detail a moderator has written mentions TLS.++`https` is `Optional[bool]`: `null` until the domain is crawled, because an uncrawled domain must+not draw an open padlock. Existing evidence rows report `null` until they go stale+(`MODERATION_EVIDENCE_MAX_AGE_DAYS`) or `backfill_domain_evidence` is run.++### 7. Undo++`POST /domain-submissions/domains/{domain}/undo` restores PENDING and clears+`rejection_reason` / `rejection_detail`, and **leaves `suggested_status`, `suggested_reason`,+`suggestion_confidence` and `suggestion_model_version` alone** — the record of what was on screen+when the decision being undone was made, which is the one thing those columns exist for and+exactly what re-posting a PENDING status destroyed.++`status_changed_by` / `status_changed_on` do move to the undoer: the field means "who last+changed the status", and that is now them.++`.update()` fires no `post_save`, so undoing an *approval* schedules a blacklist snapshot rebuild+explicitly, with the same debounce as `mwmbl/signals.py`.++The "Reviewed this session" tray is client-side session state.++### 8. Viewing and changing past moderations++`GET /domain-submissions/moderated`, moderator-only, paginated, one row per domain represented by+its **most recently touched** submission — a history row shows the decision that currently+stands. Rows carry the `suggested_*` audit columns, so a moderator revisiting a call sees what+was on screen then rather than what the model would say today.++Changing a past decision is a normal POST to `/decisions`: a decision applies to every submission+of the name, decided or not, so re-deciding needs no separate endpoint and a domain can never sit+half approved and half rejected.++The trade-off: re-deciding overwrites `status_changed_*` and the `suggested_*` columns on the+already-decided rows of that name. An append-only `ModerationDecision` log would keep both, and+is the change to make if decision history is ever read.++### 9. Rejection reasons are validated++`RejectionFieldsMixin` in `mwmbl/platform/schemas.py`, shared by `DomainDecision` and+`UpdateDomainSubmission` so the two cannot drift:++- `rejection_reason` must be one of the four choices — Django does not enforce `choices` on+ `save()`, so any string of ≤20 characters used to land in the column;+- `rejection_detail` is required when the reason is `OTHER`;+- `rejection_reason` is only accepted alongside `status=REJECTED`.++A schema validator rather than a view check, so these are 422s naming the field.++**No notification is sent.** The design's "The submitter is told the reason" line is being removed+from the dialog instead; the reason is already visible on the submitter's own submission list.++## Dropped++- **A one-line "why" for the suggestion** (`suggested by the index — 9 submissions from 2+ accounts, heavy downvotes, 4 crawled pages`). The API returns per-check `EvidenceItem` labels+ instead, which a client can join. "from 2 accounts" — the count of distinct submitting accounts+ — is still not computed anywhere; it would be one more correlated subquery if it comes back.++## Known loose end++`views.py`'s `DomainSubmissionApprovalForm` decides submissions without going through+`apply_decision`: it sets neither `status_changed_by` nor the `suggested_*` audit columns. Not one+of these nine gaps, but a second decision path that quietly writes worse records than the API's.diff --git a/mwmbl/background.py b/mwmbl/background.pyindex d7b8dfc..eaf8687 100644--- a/mwmbl/background.py+++ b/mwmbl/background.py@@ -18,6 +18,7 @@ from background_task import backgroundfrom django.conf import settingsfrom django.core.cache import cachefrom django.core.management import call_command+from django.db import transactionfrom redis import Redisfrom mwmbl import pricing@@ -28,7 +29,7 @@ from mwmbl.indexer.blacklist_snapshot import get_snapshot_blacklist, refresh_snafrom mwmbl.indexer.purge_blacklisted import purge_documentsfrom mwmbl.indexer.purge_queue import drain_purge_queue, queue_sizefrom mwmbl.models import DomainEvidence, DomainSubmission, OldIndex, UsageBucket-from mwmbl.moderation.evidence import crawl_domain, store_evidence+from mwmbl.moderation.evidence import crawl_domain, store_evidence, store_failurefrom mwmbl.moderation.model import reset_model_cachefrom mwmbl.moderation.suggest import refresh_suggestionfrom mwmbl.quota import MONTHLY_TTL, _monthly_key, get_all_monthly_keys@@ -262,14 +263,21 @@ def enrich_domain_submission(domain: str):redis = Redis.from_url(settings.REDIS_URL)try:- evidence = store_evidence(domain, crawl_domain(domain, redis))- except Exception:+ crawl = crawl_domain(domain, redis)+ except Exception as exception:logger.exception("Failed to crawl %s for moderation", domain)- DomainEvidence.objects.update_or_create(- domain=domain, defaults={"state": DomainEvidence.State.FAILED})+ store_failure(domain, type(exception).__name__)return- refresh_suggestion(evidence)+ # One transaction, because the row is only useful once both halves are in it. Committing+ # READY with no suggestion on it - a deploy restarting the worker in between would do it -+ # leaves a row the freshness check above then skips for a month, so the queue would show+ # "not assessed yet" for a domain that has in fact been crawled. Scoring failures are left+ # to propagate: the transaction rolls back and the task runner retries, which is a better+ # answer than recording a crawl that did happen as a crawl failure.+ with transaction.atomic():+ evidence = store_evidence(domain, crawl)+ refresh_suggestion(evidence)logger.info("Enriched %s: %s (%.2f)", domain, evidence.suggested_action,evidence.confidence or 0.0)diff --git a/mwmbl/crawler/retrieve.py b/mwmbl/crawler/retrieve.pyindex 553c15e..7083d17 100644--- a/mwmbl/crawler/retrieve.py+++ b/mwmbl/crawler/retrieve.py@@ -60,6 +60,10 @@ def fetch(url):Redirects are followed manually so each hop can be re-validated against theSSRF guard: a public URL must not be able to 3xx us into an internal address.+ Returns (status code, content, resolved URL). The resolved URL is the last hop, which is+ the only place the redirect chain is visible to a caller: it says which host actually+ served the page, rather than which one we asked.+https://stackoverflow.com/a/22347526"""@@ -89,7 +93,7 @@ def fetch(url):logger.debug(f"Maximum size reached for URL {url}")break- return r.status_code, content+ return r.status_code, content, urlraise ValueError(f"Too many redirects for URL {url}")@@ -114,7 +118,7 @@ def robots_allowed(url: str, redis: Redis) -> bool:return allowedtry:- status_code, content = fetch(robots_url)+ status_code, content, _ = fetch(robots_url)except ALLOWED_EXCEPTIONS as e:logger.debug(f"Robots error: {robots_url}, {e}")_cache_robots_content(redis, domain, [], error=True)@@ -320,7 +324,7 @@ def crawl_url(url, redis: Redis):}try:- status_code, content = fetch(url)+ status_code, content, resolved_url = fetch(url)except ALLOWED_EXCEPTIONS as e:logger.debug(f"Exception crawling URl {url}: {e}")return {@@ -337,6 +341,7 @@ def crawl_url(url, redis: Redis):if len(content) == 0:return {'url': url,+ 'resolved_url': resolved_url,'status': status_code,'timestamp': js_timestamp,'content': None,@@ -354,6 +359,7 @@ def crawl_url(url, redis: Redis):logger.exception(f"Error parsing dom: {url}")return {'url': url,+ 'resolved_url': resolved_url,'status': status_code,'timestamp': js_timestamp,'content': None,@@ -380,6 +386,7 @@ def crawl_url(url, redis: Redis):logger.exception("Error parsing paragraphs - offending control bytes: %s", bad_bytes)return {'url': url,+ 'resolved_url': resolved_url,'status': status_code,'timestamp': js_timestamp,'content': None,@@ -424,6 +431,7 @@ def crawl_url(url, redis: Redis):return {'url': url,+ 'resolved_url': resolved_url,'status': status_code,'timestamp': js_timestamp,'content': {diff --git a/mwmbl/management/commands/backfill_domain_evidence.py b/mwmbl/management/commands/backfill_domain_evidence.pyindex ef18823..028c0a8 100644--- a/mwmbl/management/commands/backfill_domain_evidence.py+++ b/mwmbl/management/commands/backfill_domain_evidence.py@@ -20,10 +20,12 @@ import timefrom django.core.management.base import BaseCommandfrom django.utils.dateparse import parse_date+from django.db import transaction+from mwmbl.crawler.env_vars import CRAWL_DELAY_SECONDSfrom mwmbl.indexer.blacklist_snapshot import get_redisfrom mwmbl.models import DomainEvidence, DomainSubmission-from mwmbl.moderation.evidence import crawl_domain, store_evidence+from mwmbl.moderation.evidence import crawl_domain, store_evidence, store_failurefrom mwmbl.moderation.suggest import refresh_suggestionfrom mwmbl.moderation.training_data import is_trainable_domain@@ -46,17 +48,17 @@ class Command(BaseCommand):redis = get_redis()for number, domain in enumerate(domains, start=1):try:- evidence = store_evidence(domain, crawl_domain(domain, redis))- refresh_suggestion(evidence)+ # Both halves in one transaction, so an interrupted run never leaves a READY+ # row with no suggestion on it - which the freshness check would then skip.+ with transaction.atomic():+ evidence = store_evidence(domain, crawl_domain(domain, redis))+ refresh_suggestion(evidence)self.stdout.write(f"[{number}/{len(domains)}] {domain}: {evidence.suggested_action or '?'} "f"({evidence.state})")except Exception as exception:# One unfetchable domain must not end a backfill of thousands.- DomainEvidence.objects.update_or_create(- domain=domain,- defaults={"state": DomainEvidence.State.FAILED,- "error": type(exception).__name__})+ store_failure(domain, type(exception).__name__)self.stderr.write(f"[{number}/{len(domains)}] {domain}: FAILED ({exception})")if options["delay"]:time.sleep(options["delay"])diff --git a/mwmbl/management/commands/train_domain_moderation_model.py b/mwmbl/management/commands/train_domain_moderation_model.pyindex 8fcd0b8..92a2c94 100644--- a/mwmbl/management/commands/train_domain_moderation_model.py+++ b/mwmbl/management/commands/train_domain_moderation_model.py@@ -5,19 +5,20 @@The gate compares the new model's cold-start PR-AUC against the lower bound of the incumbent'sbootstrap interval, so a change that only looks like an improvement does not ship.++Publishing writes a ModerationModelArtifact row rather than a file: the incumbent the gate+reads and the model the workers serve are then the same object, and neither is lost to a+deploy. See mwmbl.moderation.model."""import jsonfrom datetime import date-from pathlib import Path-from django.conf import settingsfrom django.core.management.base import BaseCommandfrom mwmbl.background import rescore_pending_submissionsfrom mwmbl.models import DomainSubmissionfrom mwmbl.moderation import training_data-from mwmbl.moderation.model import (- METRICS_FILENAME, MODEL_FILENAME, load_metrics, reset_model_cache)+from mwmbl.moderation.model import load_metrics, publishfrom mwmbl.moderation.train import passes_gate, train# A reason class with fewer real examples than this is a candidate for blocklist-derived@@ -36,7 +37,6 @@ class Command(BaseCommand):help="Train and report metrics without writing the artifact")parser.add_argument("--force", action="store_true",help="Publish even if the gate fails")- parser.add_argument("--output-dir", default=settings.DOMAIN_MODERATION_MODEL_DIR)parser.add_argument("--derived", action="store_true",help=("Add blocklist-derived rows for reason classes short of real data. Off by "@@ -74,7 +74,11 @@ class Command(BaseCommand):self.stderr.write("Gate failed; not publishing. Re-run with --force to override.")return- self._publish(model, metrics, Path(options["output_dir"]))+ publish(model, metrics)+ self.stdout.write(self.style.SUCCESS(+ f"Published {model.version}; every worker picks it up within a minute."))+ rescore_pending_submissions(schedule=0)+ self.stdout.write("Scheduled a rescore of the pending queue with the new model.")@staticmethoddef _reasons_needing_data(rows) -> set[str]:@@ -84,16 +88,3 @@ class Command(BaseCommand):counts[row.reason] = counts.get(row.reason, 0) + 1return {reason for reason in DomainSubmission.DOMAIN_REJECTION_REASONif counts.get(reason, 0) < MIN_REAL_ROWS_PER_REASON}-- def _publish(self, model, metrics, output_dir: Path):- import joblib-- output_dir.mkdir(parents=True, exist_ok=True)- joblib.dump(model, output_dir / MODEL_FILENAME)- (output_dir / METRICS_FILENAME).write_text(json.dumps(metrics, indent=2))- self.stdout.write(self.style.SUCCESS(- f"Wrote {model.version} to {output_dir / MODEL_FILENAME}"))-- reset_model_cache()- rescore_pending_submissions(schedule=0)- self.stdout.write("Scheduled a rescore of the pending queue with the new model.")diff --git a/mwmbl/migrations/0035_moderationmodelartifact_and_more.py b/mwmbl/migrations/0035_moderationmodelartifact_and_more.pynew file mode 100644index 0000000..837669b--- /dev/null+++ b/mwmbl/migrations/0035_moderationmodelartifact_and_more.py@@ -0,0 +1,57 @@+# Generated by Django 5.2.13 on 2026-08-28 18:12++import django.utils.timezone+from django.db import migrations, models+++class Migration(migrations.Migration):++ dependencies = [+ ('mwmbl', '0034_domainsubmission_suggested_reason_and_more'),+ ]++ operations = [+ migrations.CreateModel(+ name='ModerationModelArtifact',+ fields=[+ ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),+ ('version', models.CharField(max_length=50, unique=True)),+ ('model', models.BinaryField()),+ ('metrics', models.JSONField(default=dict)),+ ('created_on', models.DateTimeField(default=django.utils.timezone.now)),+ ],+ options={+ 'get_latest_by': 'created_on',+ },+ ),+ migrations.RemoveIndex(+ model_name='domainevidence',+ name='mwmbl_domai_state_1c3406_idx',+ ),+ migrations.RemoveField(+ model_name='domainevidence',+ name='review_priority',+ ),+ migrations.AddField(+ model_name='domainevidence',+ name='reason_confidence',+ field=models.FloatField(blank=True, null=True),+ ),+ migrations.AddField(+ model_name='domainevidence',+ name='reason_source',+ field=models.CharField(blank=True, max_length=10),+ ),+ migrations.AddIndex(+ model_name='domainsubmission',+ index=models.Index(fields=['name', 'status'], name='mwmbl_domai_name_0a9d6d_idx'),+ ),+ migrations.AddIndex(+ model_name='domainsubmission',+ index=models.Index(fields=['submitted_by', 'status'], name='mwmbl_domai_submitt_2d8fb1_idx'),+ ),+ migrations.AddIndex(+ model_name='moderationmodelartifact',+ index=models.Index(fields=['-created_on'], name='mwmbl_moder_created_1c74be_idx'),+ ),+ ]diff --git a/mwmbl/migrations/0036_search_result_vote_domain.py b/mwmbl/migrations/0036_search_result_vote_domain.pynew file mode 100644index 0000000..bfc8983--- /dev/null+++ b/mwmbl/migrations/0036_search_result_vote_domain.py@@ -0,0 +1,57 @@+import re++from django.db import migrations, models++# Frozen copies of mwmbl.utils.parse_url/normalize_domain/bare_host: a migration has to keep+# producing the same data forever, so it must not depend on application code that is free to+# change. Matches migration 0032, which normalised the domain submission names.+URL_REGEX = re.compile("^(([^:/?#]+):)?(//([^/?#]*)|///)?([^?#]*)(\\?[^#]*)?(#.*)?")+DOMAIN_END_REGEX = re.compile(r"[/?#]")++BATCH_SIZE = 1000+++def bare_host(domain_or_url: str) -> str:+ netloc = URL_REGEX.match(domain_or_url).group(4)+ if not netloc:+ netloc = DOMAIN_END_REGEX.split(domain_or_url, maxsplit=1)[0]+ host = netloc.lower()+ return host[4:] if host.startswith("www.") else host+++def backfill_domains(apps, schema_editor):+ """Derive the host for every existing vote, in batches.++ Written before the index is created, so the whole table is scanned once rather than+ maintaining an index through the rewrite.+ """+ SearchResultVote = apps.get_model("mwmbl", "SearchResultVote")+ batch = []+ for vote in SearchResultVote.objects.all().only("id", "url").iterator(chunk_size=BATCH_SIZE):+ vote.domain = bare_host(vote.url)+ batch.append(vote)+ if len(batch) >= BATCH_SIZE:+ SearchResultVote.objects.bulk_update(batch, ["domain"])+ batch = []+ if batch:+ SearchResultVote.objects.bulk_update(batch, ["domain"])+++class Migration(migrations.Migration):++ dependencies = [+ ('mwmbl', '0035_moderationmodelartifact_and_more'),+ ]++ operations = [+ migrations.AddField(+ model_name='searchresultvote',+ name='domain',+ field=models.CharField(blank=True, max_length=300),+ ),+ migrations.RunPython(backfill_domains, migrations.RunPython.noop),+ migrations.AddIndex(+ model_name='searchresultvote',+ index=models.Index(fields=['domain', 'vote_type'], name='mwmbl_searc_domain_7f4ebd_idx'),+ ),+ ]diff --git a/mwmbl/models.py b/mwmbl/models.pyindex be37e9b..54bfe32 100644--- a/mwmbl/models.py+++ b/mwmbl/models.py@@ -9,6 +9,7 @@ from ninja import ModelSchemafrom ninja.orm import create_schemafrom mwmbl.usernames import generate_username+from mwmbl.utils import bare_hostclass MwmblUser(AbstractUser):@@ -82,6 +83,11 @@ class DomainSubmission(models.Model):]indexes = [models.Index(fields=['submitted_on']),+ # The moderation queue asks "has this domain been decided before?" and "has this+ # submitter?" once per row it renders, as correlated subqueries. Both are lookups+ # into this table by a decided status. See mwmbl.moderation.suggest.+ models.Index(fields=['name', 'status']),+ models.Index(fields=['submitted_by', 'status']),]DOMAIN_SUBMISSION_STATUS = {@@ -109,8 +115,10 @@ class DomainSubmission(models.Model):# What the suggestion model was showing when the moderator made this decision. Deliberately# separate storage from the live suggestion on DomainEvidence: a retrain rewrites that row,# and sharing the columns would silently rewrite the record of what was actually on screen.- # These are what let us measure whether the suggestions help, and de-bias the next retrain- # against decisions that merely confirmed a suggestion.+ # These are what make the suggestions' influence measurable: a retrain trains on decisions+ # that were themselves made with a suggestion on screen, and without a record of what was+ # shown there is no way to see that happening. The retrain does not yet weight rows by it -+ # see mwmbl.moderation.training_data - it reports it.suggested_status = models.CharField(max_length=20, blank=True)suggested_reason = models.CharField(max_length=20, blank=True)suggestion_confidence = models.FloatField(null=True, blank=True)@@ -248,12 +256,23 @@ class SearchResultVote(models.Model):query = models.CharField(max_length=300) # The search query contextvote_type = models.CharField(max_length=10, choices=[(k, v) for k, v in VOTE_TYPES.items()])timestamp = models.DateTimeField(auto_now_add=True)-+ # The host the URL belongs to, derived on save. Denormalised because the moderation queue+ # rolls votes up per domain, and it orders and paginates thousands of pending domains on+ # the result - which has to happen in the database, and cannot parse a URLField there.+ domain = models.CharField(max_length=300, blank=True)++ def save(self, *args, **kwargs):+ self.domain = bare_host(self.url)+ super().save(*args, **kwargs)+class Meta:unique_together = ['user', 'url', 'query'] # One vote per user per result per queryindexes = [models.Index(fields=['url', 'query']),models.Index(fields=['timestamp']),+ # The per-domain rollup the moderation queue reads: count this domain's upvotes,+ # count its downvotes. See mwmbl.moderation.suggest.annotate_votes.+ models.Index(fields=['domain', 'vote_type']),]@@ -309,6 +328,12 @@ class DomainEvidence(models.Model):Keyed on the domain rather than the submission because domains get resubmitted - 615 of6,949 distinct names in the last judgments export had more than one submission - and thecrawl is about the domain, not about who asked for it.++ What the queue orders on is *not* stored here. The suggestion a moderator sees depends on+ rows this one knows nothing about - the submitter's record, and whether the domain has+ since been decided - so the sort key is computed per query in SQL by+ mwmbl.moderation.suggest.annotate_queue. Storing one would only ever be the answer to a+ question nobody asked in that form."""class State(models.TextChoices):@@ -331,18 +356,52 @@ class DomainEvidence(models.Model):suggested_action = models.CharField(max_length=10, blank=True) # APPROVE | REJECT | UNSUREsuggested_reason = models.CharField(max_length=20, blank=True)confidence = models.FloatField(null=True, blank=True)- # One number combining direction and confidence so "confident rejects and genuinely- # uncertain rows first, obvious approvals last" is a single indexed ORDER BY. Sorting on- # confidence alone cannot express that, because it is highest at both ends.- review_priority = models.FloatField(null=True, blank=True)+ # How sure we are of the *reason*, and where it came from - both separate from the numbers+ # above, because a confident rejection can carry a barely-held guess at why. Storing them+ # is what lets the API show the reason with its own confidence instead of the rejection's,+ # and keeps `derived` (a reason class learned from blocklists rather than from moderator+ # decisions) visible to the moderator it is a caveat for.+ reason_confidence = models.FloatField(null=True, blank=True)+ reason_source = models.CharField(max_length=10, blank=True) # rule | model | derivedevidence = models.JSONField(default=list) # cached rule + model evidence itemsmodel_version = models.CharField(max_length=50, blank=True)class Meta:indexes = [- models.Index(fields=["state", "-review_priority"]),models.Index(fields=["suggested_action", "suggested_reason"]),]def __str__(self):return f"{self.domain} ({self.state})"+++class ModerationModelArtifact(models.Model):+ """A trained moderation suggester, stored where a deploy cannot overwrite it.++ The artifact used to live in the source tree, which meant every deploy silently reverted+ the monthly retrain to whatever was committed, and each worker replica kept its own+ divergent copy. The database is the one place all the workers already agree on.++ Rows accumulate: the newest is served, and the ones behind it are the record of what was+ serving when a given decision was made. Versions are dated, so a second retrain on the+ same day replaces its own row rather than adding one. ``metrics`` travels in the same row+ as the model it describes, so the retrain gate cannot compare a candidate against numbers+ that belong to a different artifact. At a monthly retrain that is ~12 MB a year.++ The model bundled in mwmbl/moderation/artifacts is the warm start for a database with no+ rows yet; nothing ever writes to it. See mwmbl.moderation.model.+ """++ version = models.CharField(max_length=50, unique=True)+ model = models.BinaryField() # joblib pickle of a ModerationModel, ~1 MB+ metrics = models.JSONField(default=dict) # what train.evaluate measured for this model+ created_on = models.DateTimeField(default=timezone.now)++ class Meta:+ get_latest_by = "created_on"+ indexes = [+ models.Index(fields=["-created_on"]),+ ]++ def __str__(self):+ return f"{self.version} ({self.created_on:%Y-%m-%d})"diff --git a/mwmbl/moderation/artifacts/README.md b/mwmbl/moderation/artifacts/README.mdindex 9530121..22df4fc 100644--- a/mwmbl/moderation/artifacts/README.md+++ b/mwmbl/moderation/artifacts/README.md@@ -6,15 +6,22 @@ held-out real decisions when it was built. Both are committed deliberately, fordeterministic checks alone, so shipping one means a fresh deploy is useful on day one instead ofafter the first retrain.+**This directory is the warm start, and nothing writes to it.** A retrain publishes a+`ModerationModelArtifact` row in Postgres, and that row is what every worker serves and what the+gate compares against; these files are read only when the table is empty. Writing the artifact+here would mean each deploy reverting the retrain to whatever was last committed, and each+worker replica keeping its own copy.+## Regeneratinguv run manage.py train_domain_moderation_model --dry-run # report metrics onlyuv run manage.py train_domain_moderation_model # publish if it passes the gateThe gate compares the new model's cold-start PR-AUC against the *lower bound* of the incumbent's-bootstrap interval in `metrics.json`, so a change that only looks like an improvement does not-ship. Publishing also schedules `rescore_pending_submissions`, which re-scores the existing queue-with the new model.+bootstrap interval — the metrics stored beside the published model, falling back to `metrics.json`+until something has been published — so a change that only looks like an improvement does not+ship. Publishing writes a new artifact row, which every worker picks up within a minute, and+schedules `rescore_pending_submissions` to re-score the existing queue with the new model.To reproduce the numbers without a production database, train from the sanitized judgments exportinstead:@@ -35,8 +42,10 @@ retrain to change the artifact substantially.`LogisticRegression` heads, and ONNX export of a TF-IDF pipeline is far more trouble than it isworth at this size. Two consequences worth knowing:-- Unpickling executes code, so this file is trusted exactly as much as the rest of the repo. Do- not point `DOMAIN_MODERATION_MODEL_DIR` at a directory anyone else can write to.+- Unpickling executes code, so this file — and the artifact rows, which anything that can write+ to the database could replace — is trusted exactly as much as the rest of the repo. Do not+ point `DOMAIN_MODERATION_MODEL_DIR` at a directory anyone else can write to.- A scikit-learn upgrade can make an old pickle unloadable. `mwmbl.moderation.model` catches that- and logs it rather than failing, falling back to the deterministic checks — so the symptom is- "every suggestion says UNSURE", and the fix is a retrain.+ and logs it, falling back to this committed artifact and then to the deterministic checks — so+ the symptom is suggestions reverting to an older model version, or saying UNSURE, and the fix+ is a retrain.diff --git a/mwmbl/moderation/evidence.py b/mwmbl/moderation/evidence.pyindex 4ea0e07..9cab277 100644--- a/mwmbl/moderation/evidence.py+++ b/mwmbl/moderation/evidence.py@@ -6,6 +6,10 @@ detection") are judgements about a site, not a page. The same three pages are shmoderator's detail panel and fed to the model, so what a moderator sees and what the modeljudged can never disagree.+The homepage is fetched over https, falling back to http once if that errors, so that a+site without a certificate is reported as a site without a certificate rather than as a site+that is down.+This reuses mwmbl.crawler.retrieve.crawl_url, which already does robots.txt with Rediscaching, SSRF validation, a 3 second timeout, a 1 MB cap and justext extraction with OpenGraph fallbacks. Super Search already calls it from inside the server process, so nothing@@ -50,7 +54,7 @@ def crawl_domain(domain: str, redis, max_pages: int = None) -> dict:if max_pages is None:max_pages = settings.MODERATION_PAGES_PER_DOMAIN- homepage = crawl_url(f"https://{domain}/", redis)+ homepage, https = _fetch_homepage(domain, redis)pages = [_page(homepage)]error = (homepage.get("error") or {}).get("name", "") if homepage.get("error") else ""@@ -64,10 +68,35 @@ def crawl_domain(domain: str, redis, max_pages: int = None) -> dict:"final_domain": _final_domain(homepage, domain),"error": error,"pages": pages,- "signals": _signals(domain, pages),+ "signals": _signals(domain, pages, https),}+def _fetch_homepage(domain: str, redis) -> tuple[dict, bool]:+ """The homepage, and whether it came back over TLS.++ Only https used to be tried, which made "this site has no certificate" and "this site is+ down" the same result - a decisive REJECT for being unreachable. They are not the same+ thing: a site served over plain http is a site, and whether it has TLS is something to+ show a moderator rather than to decide for them. So a failed https fetch is retried once+ over http, and which one answered is recorded.++ Only a fetch *error* is retried. A 404 over https is an answer, and asking the same+ server the same question again without TLS will get the same one.+ """+ over_https = crawl_url(f"https://{domain}/", redis)+ if not over_https.get("error"):+ return over_https, True++ over_http = crawl_url(f"http://{domain}/", redis)+ if over_http.get("error"):+ # Neither worked. Report the https attempt: it is the one whose error describes what+ # we would do in production, and reporting the http failure would say "no TLS" about+ # a domain that simply is not there.+ return over_https, False+ return over_http, False++def store_evidence(domain: str, crawl: dict) -> DomainEvidence:"""Persist a crawl and the checks derived from it, ready for the queue to read."""evidence, _ = DomainEvidence.objects.update_or_create(@@ -86,6 +115,36 @@ def store_evidence(domain: str, crawl: dict) -> DomainEvidence:return evidence+def store_failure(domain: str, error: str) -> DomainEvidence:+ """Record that a domain could not be crawled, clearing what an earlier crawl left behind.++ Everything the crawl produced goes with it, rather than only the state. The detail panel+ renders whatever is on the row, so a month-old page list and its suggestion sitting next+ to a FAILED state reads to a moderator as current evidence about a site that is now+ unreachable - which is precisely the fact the state is trying to report.+ """+ evidence, _ = DomainEvidence.objects.update_or_create(+ domain=domain,+ defaults={+ "state": DomainEvidence.State.FAILED,+ "fetched_at": timezone.now(), # when we last tried, not when we last succeeded+ "error": error,+ "http_status": None,+ "final_domain": "",+ "pages": [],+ "signals": {},+ "evidence": [],+ "suggested_action": "",+ "suggested_reason": "",+ "confidence": None,+ "reason_confidence": None,+ "reason_source": "",+ "model_version": "",+ },+ )+ return evidence++def _page(result: dict) -> dict:content = result.get("content") or {}error = result.get("error") or {}@@ -125,12 +184,17 @@ def _follow_up_urls(domain: str, homepage: dict, limit: int) -> list[str]:def _final_domain(homepage: dict, domain: str) -> str:- """The host we ended up on. crawl_url follows redirects and reports the URL it fetched."""- final = urlparse(homepage.get("url") or "").netloc+ """The host we ended up on, empty when that is the domain we asked for.++ ``resolved_url`` is the last hop of the redirect chain, which crawl_url reports separately+ from ``url`` - the URL we asked for, and the one the rest of the crawler keys on. Reading+ ``url`` here would compare the domain against itself and silently never find a redirect.+ """+ final = urlparse(homepage.get("resolved_url") or "").netlocreturn final if final and final != domain else ""-def _signals(domain: str, pages: list[dict]) -> dict:+def _signals(domain: str, pages: list[dict], https: bool) -> dict:text = " ".join(f"{page['title']} {page['extract']}" for page in pages).lower()try:blacklisted = get_snapshot_blacklist().is_domain_blacklisted(domain)@@ -146,6 +210,7 @@ def _signals(domain: str, pages: list[dict]) -> dict:"ad_markers": sum(marker in text for marker in AD_MARKERS),"paywall_markers": sum(marker in text for marker in PAYWALL_MARKERS),"blacklisted": blacklisted,+ "https": https,}diff --git a/mwmbl/moderation/model.py b/mwmbl/moderation/model.pyindex 199574b..4243c64 100644--- a/mwmbl/moderation/model.py+++ b/mwmbl/moderation/model.py@@ -9,22 +9,31 @@ DomainEvidence, so the moderation queue is a plain indexed database read and noever loads this module. A missing or slow artifact therefore cannot touch a moderator'srequest; the worst case is a queue row that honestly says it has not been assessed yet.+The fitted artifact lives in Postgres (mwmbl.models.ModerationModelArtifact), not on disk. It+is written monthly by a retrain and read by every worker, and a container filesystem is+neither shared between workers nor kept across a deploy. The copy in ``artifacts/`` is the+warm start for a database that has none yet, and nothing writes to it.+Loading mirrors mwmbl.tinysearchengine.super_search_select.judge: a lazy singleton behind a-lock, memoized failure, and graceful degradation to rules-only when the artifact is absent so-a deploy without it keeps working.+lock, memoized failure, and graceful degradation to rules-only when no artifact can be loaded+so a deploy without one keeps working."""from __future__ import annotations+import ioimport jsonimport threading+import timefrom dataclasses import dataclass, fieldfrom logging import getLoggerfrom pathlib import Pathfrom typing import Optional+import joblibimport numpy as npfrom django.conf import settings+from mwmbl.models import ModerationModelArtifactfrom mwmbl.moderation.features import Featuriser, ModerationExamplefrom mwmbl.moderation.rules import APPROVE, EvidenceItem, REJECT, decisive@@ -33,6 +42,12 @@ logger = getLogger(__name__)MODEL_FILENAME = "model.joblib"METRICS_FILENAME = "metrics.json"+# How long a loaded model is served before the database is asked whether a retrain has+# published a newer one. A retrain happens in one worker process and every worker serves+# suggestions, so without this they would disagree until the next restart. The check is a+# single indexed row read, and a minute of staleness after a monthly retrain costs nothing.+MODEL_REFRESH_SECONDS = 60+# A reason class with no real decisions behind it must not present as though it had. The# reason head learns whatever classes the training data contains (reason_head.classes_ is the# authority); this caps the ones that are not grounded in moderator decisions.@@ -152,40 +167,83 @@ def suggest(domain: str, page_texts: list[str], evidence_items: list[EvidenceIte_model: Optional[ModerationModel] = None-_load_attempted = False+_loaded_version: Optional[str] = None+_checked_at: Optional[float] = None_lock = threading.Lock()def get_model() -> Optional[ModerationModel]:- """Lazily load the shared model; None (memoized) if the artifact is unavailable."""- global _model, _load_attempted- if _load_attempted:- return _model+ """The published model, reloaded when a retrain has published a newer one.++ None (until the published version changes) when nothing can be loaded, so a suggestion+ falls back to the deterministic checks rather than failing.+ """+ global _model, _loaded_version, _checked_atwith _lock:- if _load_attempted:+ if _checked_at is not None and time.monotonic() - _checked_at < MODEL_REFRESH_SECONDS:return _model- _model = _load(Path(settings.DOMAIN_MODERATION_MODEL_DIR))- _load_attempted = True+ published = _published_version()+ if _checked_at is None or published != _loaded_version:+ _model = _load(published)+ _loaded_version = published+ _checked_at = time.monotonic()return _model+def publish(model: ModerationModel, metrics: dict) -> ModerationModelArtifact:+ """Store a trained model and its metrics as the artifact every worker will serve."""+ buffer = io.BytesIO()+ joblib.dump(model, buffer)+ # Versions are dated, so a second retrain on the same day replaces the first rather than+ # colliding on the unique version.+ artifact, _ = ModerationModelArtifact.objects.update_or_create(+ version=model.version,+ defaults={"model": buffer.getvalue(), "metrics": metrics})+ reset_model_cache()+ return artifact++def reset_model_cache() -> None:"""Drop the cached model so the next call reloads. Used after a retrain, and by tests."""- global _model, _load_attempted+ global _model, _loaded_version, _checked_atwith _lock:_model = None- _load_attempted = False+ _loaded_version = None+ _checked_at = None+++def _published_version() -> Optional[str]:+ """The version of the newest stored artifact, or None to fall back to the bundled one."""+ row = ModerationModelArtifact.objects.order_by("-created_on").values("version").first()+ return row["version"] if row else None-def _load(model_dir: Path) -> Optional[ModerationModel]:- model_path = model_dir / MODEL_FILENAME+def _load(version: Optional[str]) -> Optional[ModerationModel]:+ if version is None:+ return _load_bundled()+ try:+ artifact = ModerationModelArtifact.objects.get(version=version)+ model = joblib.load(io.BytesIO(bytes(artifact.model)))+ except Exception:+ # An artifact pickled by a different scikit-learn version is the realistic case, and+ # it must not take the enrichment task down with it: fall back to the model shipped+ # with this code, which was pickled by the dependencies this code is running.+ logger.exception("Failed to load moderation model %s; falling back to the bundled "+ "artifact", version)+ return _load_bundled()+ logger.info("Domain moderation model %s loaded from the database", model.version)+ return model+++def _load_bundled() -> Optional[ModerationModel]:+ """The artifact shipped in the source tree, for a database with no retrain in it yet."""+ model_path = Path(settings.DOMAIN_MODERATION_MODEL_DIR) / MODEL_FILENAMEif not model_path.exists():logger.warning(- "Domain moderation model not found at %s; suggestions will use the deterministic "- "checks only", model_path)+ "Domain moderation model not found at %s and none published; suggestions will "+ "use the deterministic checks only", model_path)return Nonetry:- import joblibmodel = joblib.load(model_path)except Exception:logger.exception("Failed to load the domain moderation model from %s", model_path)@@ -194,9 +252,15 @@ def _load(model_dir: Path) -> Optional[ModerationModel]:return model-def load_metrics(model_dir: Optional[Path] = None) -> dict:- """The metrics written alongside the artifact, so the retrain gate can compare against it."""- path = Path(model_dir or settings.DOMAIN_MODERATION_MODEL_DIR) / METRICS_FILENAME- if not path.exists():- return {}- return json.loads(path.read_text())+def load_metrics() -> dict:+ """The metrics of the published model, so the retrain gate can compare against them.++ Read from the same row as the model they describe: the gate's guarantee is that a+ candidate beats the artifact currently being served, and metrics stored anywhere else+ could be describing a different one.+ """+ row = ModerationModelArtifact.objects.order_by("-created_on").values("metrics").first()+ if row is not None:+ return row["metrics"]+ bundled = Path(settings.DOMAIN_MODERATION_MODEL_DIR) / METRICS_FILENAME+ return json.loads(bundled.read_text()) if bundled.exists() else {}diff --git a/mwmbl/moderation/rules.py b/mwmbl/moderation/rules.pyindex cc6dfc3..81f953a 100644--- a/mwmbl/moderation/rules.py+++ b/mwmbl/moderation/rules.py@@ -44,6 +44,12 @@ NON_ENGLISH_TLDS = frozenset({"pt", "es", "it", "fr", "de", "nl", "se", "no", "fi", "dk", "vn", "th", "id", "il", "ir",})+# How much an earlier decision on the same domain is worth. Named because the queue's SQL+# reproduces this check to filter and order on it (mwmbl.moderation.suggest), and the two+# drifting apart would show a moderator one suggestion and filter on another.+PRIOR_DECISION_CONFIDENCE = 0.9+PRIOR_DECISION_REASON = "OTHER"+@dataclass(frozen=True)class EvidenceItem:@@ -138,6 +144,16 @@ def crawl_evidence(domain: str, crawl: dict) -> list[EvidenceItem]:"On a public malware/adult blocklist - approving here also unblocks it. ""These lists are built for ad-blocking and do have false positives."))+ # Neutral for the same reason robots.txt is: a fact the moderator should have, not a+ # decision to make for them. Plenty of small personal sites - the ones this index exists+ # for - still have no certificate, and no rejection detail a moderator has written+ # mentions TLS. It only means anything when we actually reached the site: after a failed+ # fetch "https: False" says the https attempt failed, which the unreachable line already+ # said better.+ if fetched and signals.get("https") is False:+ items.append(EvidenceItem(+ "no_tls", NEUTRAL, "Served over plain HTTP - no TLS certificate"))+if fetched:items.append(EvidenceItem("reachable", APPROVE,@@ -171,11 +187,12 @@ def live_evidence(submitter_record: dict, prior_decisions: dict) -> list[Evidencif prior_decisions.get("approved"):items.append(EvidenceItem("prior_decision", APPROVE, "This domain has already been approved before",- implies_action="APPROVE", implies_confidence=0.9))+ implies_action="APPROVE", implies_confidence=PRIOR_DECISION_CONFIDENCE))elif prior_decisions.get("rejected"):items.append(EvidenceItem("prior_decision", REJECT, "This domain has already been rejected before",- implies_action="REJECT", implies_reason="OTHER", implies_confidence=0.9))+ implies_action="REJECT", implies_reason=PRIOR_DECISION_REASON,+ implies_confidence=PRIOR_DECISION_CONFIDENCE))return itemsdiff --git a/mwmbl/moderation/suggest.py b/mwmbl/moderation/suggest.pyindex fabe619..fdd9d34 100644--- a/mwmbl/moderation/suggest.py+++ b/mwmbl/moderation/suggest.py@@ -1,6 +1,6 @@"""Tie the checks and the model together, and put the answer where the queue can read it.-Two entry points, split by when they can be run:+Three entry points, split by when they can be run::func:`refresh_suggestion` writes the cached suggestion onto a DomainEvidence row. It iscalled by the enrichment task, the backfill command and the post-retrain rescore - never by a@@ -9,15 +9,26 @@ request.:func:`suggestion_for` reads that cached suggestion back and adds the evidence that would gostale if it were cached: the submitter's record, and whether this domain has been decidedsince. Both are indexed lookups.++:func:`annotate_queue` says the same thing in SQL, because the queue filters, orders and+paginates thousands of pending rows in the database and cannot call :func:`suggestion_for` to+decide which ones to return.++:func:`one_row_per_domain` and :func:`annotate_votes` are what turn that row-per-submission+queryset into the row-per-domain the moderator actually reviews. Both are SQL for the same+reason."""from __future__ import annotationsfrom logging import getLoggerfrom typing import Iterable-from django.db.models import Count, Q+from django.db.models import (+ Case, CharField, Count, Exists, F, FloatField, IntegerField, OuterRef, Q, Subquery, Value,+ When)+from django.db.models.functions import Coalesce, Substr-from mwmbl.models import DomainEvidence, DomainSubmission+from mwmbl.models import DomainEvidence, DomainSubmission, SearchResultVotefrom mwmbl.moderation import rulesfrom mwmbl.moderation.evidence import page_textsfrom mwmbl.moderation.model import Suggestion, suggest@@ -33,12 +44,13 @@ def refresh_suggestion(evidence: DomainEvidence) -> DomainEvidence:evidence.suggested_action = suggestion.actionevidence.suggested_reason = suggestion.reasonevidence.confidence = suggestion.confidence- evidence.review_priority = suggestion.review_priority+ evidence.reason_confidence = suggestion.reason_confidence+ evidence.reason_source = suggestion.reason_sourceevidence.model_version = suggestion.model_versionevidence.evidence = suggestion.evidenceevidence.save(update_fields=[- "suggested_action", "suggested_reason", "confidence", "review_priority",- "model_version", "evidence",+ "suggested_action", "suggested_reason", "confidence", "reason_confidence",+ "reason_source", "model_version", "evidence",])return evidence@@ -97,13 +109,190 @@ def suggestion_for(submission: DomainSubmission,action=action,confidence=confidence,reason=evidence.suggested_reason if action == "REJECT" else "",- reason_confidence=confidence if action == "REJECT" else 0.0,- reason_source="rule" if cached_decisive is not None else "model",+ # The reason's own confidence, not the rejection's. A model can be sure a domain+ # should go and barely have a view on why, and showing "SPAM (0.92)" when the reason+ # head scored 0.31 would present a guess as a finding.+ reason_confidence=(evidence.reason_confidence or 0.0) if action == "REJECT" else 0.0,+ reason_source=evidence.reason_source or "model",model_version=evidence.model_version,evidence=list(evidence.evidence or []) + [item.to_dict() for item in live],)+# --------------------------------------------------------------------------------------+# The same adjustment, in SQL.+#+# suggestion_for adjusts the stored suggestion in two ways that depend on other rows - an+# earlier decision on the same domain overrides it, and an approval is withheld from a+# submitter with no track record - so neither can be precomputed onto DomainEvidence. The+# queue still has to filter, order and paginate on the result across every pending submission,+# which has to happen in the database.+#+# So it is written twice, and test_queue_display_matches_suggestion_for holds the two+# together. Filtering on UNSURE must return exactly the rows drawn as UNSURE, and the "needs a+# human first" ordering must sort on the confidence the moderator is actually shown.++DECIDED = Q(status__in=("APPROVED", "REJECTED"))+++def annotate_queue(submissions):+ """Annotate submissions with the suggestion each one will be rendered with.++ Adds ``displayed_action``, ``displayed_reason``, ``displayed_confidence`` and+ ``displayed_priority``, all NULL while the domain has no READY evidence - which is how+ "not assessed yet" sorts last and matches no suggestion filter.+ """+ # DomainEvidence is keyed on the domain rather than by a foreign key, because domains get+ # resubmitted and the crawl belongs to the domain. A correlated subquery joins them+ # without one, and keeps the filtering and ordering in SQL.+ evidence = DomainEvidence.objects.filter(domain=OuterRef("name"))+ submissions = submissions.annotate(+ # Prefixed because DomainSubmission already has suggested_status and suggested_reason+ # columns - the audit of what a moderator was shown - and an annotation may not+ # shadow a real field.+ evidence_state=Subquery(evidence.values("state")[:1]),+ evidence_action=Subquery(evidence.values("suggested_action")[:1]),+ evidence_reason=Subquery(evidence.values("suggested_reason")[:1]),+ evidence_confidence=Subquery(evidence.values("confidence")[:1]),+ evidence_source=Subquery(evidence.values("reason_source")[:1]),+ # The live checks, as one boolean each. Only PENDING submissions are ever queued, so+ # a submission never counts itself in either of these.+ prior_approved=Exists(DomainSubmission.objects.filter(+ name=OuterRef("name"), status="APPROVED")),+ prior_rejected=Exists(DomainSubmission.objects.filter(+ name=OuterRef("name"), status="REJECTED")),+ submitter_decided=Exists(DomainSubmission.objects.filter(+ DECIDED, submitted_by_id=OuterRef("submitted_by_id"))),+ )++ scored = Q(evidence_state=DomainEvidence.State.READY)+ # A prior decision only overrides the stored suggestion when it is stronger than the check+ # that produced it. reason_source is how we know a check produced it at all: when it did,+ # the stored confidence *is* that check's implied confidence (see model.suggest).+ stored_check_holds = (Q(evidence_source="rule")+ & Q(evidence_confidence__gte=rules.PRIOR_DECISION_CONFIDENCE))+ prior_approval = scored & Q(prior_approved=True) & ~stored_check_holds+ prior_rejection = (scored & Q(prior_rejected=True) & Q(prior_approved=False)+ & ~stored_check_holds)+ # No cached check implies APPROVE - they are all reasons to reject - so a stored APPROVE+ # is always the model's, which is what the withheld-approval rule is about.+ withheld = scored & Q(evidence_action="APPROVE") & Q(submitter_decided=False)++ submissions = submissions.annotate(+ displayed_action=Case(+ When(prior_approval, then=Value("APPROVE")),+ When(prior_rejection, then=Value("REJECT")),+ When(withheld, then=Value("UNSURE")),+ When(scored, then=F("evidence_action")),+ default=Value(None),+ output_field=CharField(),+ ),+ displayed_confidence=Case(+ When(prior_approval | prior_rejection,+ then=Value(rules.PRIOR_DECISION_CONFIDENCE)),+ When(withheld, then=Value(0.0)),+ When(scored, then=F("evidence_confidence")),+ default=Value(None),+ output_field=FloatField(),+ ),+ displayed_reason=Case(+ When(prior_rejection, then=Value(rules.PRIOR_DECISION_REASON)),+ When(scored & Q(evidence_action="REJECT") & ~prior_approval,+ then=F("evidence_reason")),+ When(scored, then=Value("")),+ default=Value(None),+ output_field=CharField(),+ ),+ )++ # Suggestion.review_priority, over the adjusted values. This is why the sort key is not a+ # stored column: the rows whose approval was withheld are the ones a human most needs to+ # look at, and a precomputed priority would sort them by the confidence of the approval we+ # decided not to show - straight to the bottom of "what needs a human".+ return submissions.annotate(+ displayed_priority=Case(+ When(Q(displayed_action="REJECT"), then=Value(1.0) + F("displayed_confidence")),+ When(Q(displayed_action="APPROVE"), then=Value(1.0) - F("displayed_confidence")),+ When(Q(displayed_action="UNSURE"), then=Value(1.0)),+ default=Value(None),+ output_field=FloatField(),+ ),+ )+++# --------------------------------------------------------------------------------------+# From a submission per row to a domain per row.+++def one_row_per_domain(submissions, status: str | None = "PENDING", pick: str = "first"):+ """Keep one submission per name, and count the rest onto it as ``submission_count``.++ A moderator reviews a *domain*: nine submissions of cheap-rolex-outlet.biz are one card+ and one decision, not nine. Deliberately not a GROUP BY, though: the suggestion depends on+ the submitter's own track record (see :func:`suggestion_for`), and a domain submitted by+ two accounts has no single one to read. Collapsing to a representative row instead leaves+ :func:`annotate_queue` and :func:`suggestion_for` working on a submission, which is what+ holds those two together.++ ``pick`` chooses which row represents the domain, and in both cases it is the row whose+ own fields the client is going to draw:++ * ``"first"`` - the earliest submission, because the queue card says "first submitted 6+ days ago by anon_4417";+ * ``"last"`` - the most recently touched one, because a history listing shows the decision+ that currently stands.++ Ties break on the primary key, so the representative is deterministic either way.+ """+ touched = Coalesce("status_changed_on", "submitted_on")+ submissions = submissions.annotate(touched=touched)++ same_domain = DomainSubmission.objects.filter(name=OuterRef("name"))+ if status is not None:+ same_domain = same_domain.filter(status=status)++ if pick == "first":+ beats = same_domain.filter(+ Q(submitted_on__lt=OuterRef("submitted_on"))+ | Q(submitted_on=OuterRef("submitted_on"), pk__lt=OuterRef("pk")))+ else:+ beats = same_domain.annotate(touched=touched).filter(+ Q(touched__gt=OuterRef("touched"))+ | Q(touched=OuterRef("touched"), pk__gt=OuterRef("pk")))++ return submissions.annotate(+ submission_count=Coalesce(+ Subquery(same_domain.values("name").annotate(n=Count("pk")).values("n"),+ output_field=IntegerField()),+ Value(0)),+ ).filter(~Exists(beats))+++def annotate_votes(submissions):+ """Add ``upvotes`` and ``downvotes``: the votes cast on URLs belonging to the domain.++ In SQL because the queue orders on these across the whole backlog. SearchResultVote+ carries the host in its own indexed column for exactly this - Postgres cannot pick a+ domain out of a URLField - and both sides drop a leading ``www.`` so that a vote on+ www.example.com/x counts towards a submission of example.com.++ Coalesced to zero rather than left NULL: a domain nobody has voted on has no votes, and+ ordering on NULL would scatter those rows through the sort instead of ending it.+ """+ def counted(vote_type):+ return Coalesce(+ Subquery(SearchResultVote.objects+ .filter(domain=OuterRef("bare_name"), vote_type=vote_type)+ .values("domain").annotate(n=Count("pk")).values("n"),+ output_field=IntegerField()),+ Value(0))++ return submissions.annotate(+ bare_name=Case(When(name__startswith="www.", then=Substr("name", 5)),+ default=F("name"), output_field=CharField()),+ ).annotate(upvotes=counted("upvote"), downvotes=counted("downvote"))++def submitter_record(user_id: int) -> dict:"""How this submitter's previous submissions were decided.diff --git a/mwmbl/moderation/train.py b/mwmbl/moderation/train.pyindex 9bdad1d..3ac58bc 100644--- a/mwmbl/moderation/train.py+++ b/mwmbl/moderation/train.py@@ -117,6 +117,7 @@ def evaluate(model: ModerationModel, train_rows: list[TrainingRow],}metrics["reason_head"] = _evaluate_reason_head(predictions, test_rows)metrics["train_rows_by_source"] = _count_sources(train_rows)+ metrics["suggestion_influence"] = _suggestion_influence(train_rows + test_rows)return metrics@@ -169,6 +170,28 @@ def _evaluate_reason_head(predictions, test_rows: list[TrainingRow]) -> dict:}+def _suggestion_influence(rows: list[TrainingRow]) -> dict:+ """How much of the training data was decided with a suggestion on screen, and how often+ the moderator agreed with it.++ A retrain learns from decisions the previous model influenced, and the gate cannot see+ that: the held-out labels come from the same influenced population, so a model that has+ taught moderators its own mistakes scores well on them. Nothing here corrects for it -+ reweighting on an unmeasured hunch is exactly what this module refuses to do elsewhere -+ but a rising ``agreed`` against a rising ``shown`` is the shape to watch for, and it is+ the argument for down-weighting confirmations when there is enough data to test it on.+ """+ real = [row for row in rows if row.source == REAL]+ shown = [row for row in real if row.suggested_status]+ agreed = [row for row in shown+ if (row.suggested_status == "REJECTED") == row.rejected]+ return {+ "real_rows": len(real),+ "shown_a_suggestion": len(shown),+ "agreed_with_it": len(agreed),+ }++def _has_both_classes(truth: np.ndarray) -> bool:return truth.sum() >= MIN_CLASS_ROWS and (1 - truth).sum() >= MIN_CLASS_ROWSdiff --git a/mwmbl/moderation/training_data.py b/mwmbl/moderation/training_data.pyindex 5d29ba9..870a98d 100644--- a/mwmbl/moderation/training_data.py+++ b/mwmbl/moderation/training_data.py@@ -33,6 +33,13 @@ gate needs to vary them independently:that is not adult content, the do-not-crawl cases, promotion of a paid product. It existsto cover concepts, not to move aggregate numbers.+Real rows carry the suggestion that was on screen when the decision was made, because a+retrain trains on decisions the previous model had a hand in. That is reported by+mwmbl.moderation.train as ``suggestion_influence`` rather than corrected for: with no decided+submission yet made with a suggestion shown, any weighting would be an unmeasured guess, and+this module's whole argument is that unmeasured changes to the training mix do not ship. The+number is what will say when it needs to be.+Historic malformed submissions are excluded. 70 rows are ``null``, an IP address orcapitalised; migrations 0032/0033 fixed that at the API layer, so training on them would teachthe model to detect a problem that can no longer occur.@@ -84,6 +91,10 @@ class TrainingRow:# all, so that is the slice the retrain gate reads. None for derived and seed rows, which# are training-only.submitter: Optional[str] = None+ # What the tool suggested when this decision was made, empty when nothing was shown. Not+ # a feature either - it is how train.evaluate reports how much of the training data the+ # previous model had a hand in.+ suggested_status: str = ""def to_example(self) -> ModerationExample:return ModerationExample(self.domain, self.page_texts)@@ -119,6 +130,7 @@ def real_rows() -> list[TrainingRow]:page_texts=page_texts(evidence_by_domain.get(submission.name)),timestamp=submission.submitted_on.isoformat() if submission.submitted_on else None,submitter=str(submission.submitted_by_id),+ suggested_status=submission.suggested_status,))return rowsdiff --git a/mwmbl/platform/api.py b/mwmbl/platform/api.pyindex 059fa09..293216f 100644--- a/mwmbl/platform/api.py+++ b/mwmbl/platform/api.py@@ -6,9 +6,9 @@ from allauth.account.utils import setup_user_email, send_email_confirmationfrom django.conf import settingsfrom django.core import signingfrom django.core.exceptions import ValidationError-from django.db.models import F, OuterRef, Subquery+from django.db import transaction+from django.db.models import Ffrom django.utils import timezone-from redis import Redisfrom ninja import Routerfrom ninja.pagination import paginatefrom ninja_jwt.authentication import JWTAuth@@ -21,13 +21,13 @@ from mwmbl.exceptions import InvalidRequestfrom mwmbl.search_auth import invalidate_api_key_cache, invalidate_user_api_key_cachefrom mwmbl.utils import normalize_domain, validate_domainfrom mwmbl import pricing-from mwmbl.background import enrich_domain_submission-from mwmbl.crawler.stats import StatsManager+from mwmbl.background import enrich_domain_submission, stats_managerfrom mwmbl.models import AgreementType, MwmblUser, DomainEvidence, DomainSubmission, SearchResultVote, ApiKey, UsageBucket, UserBilling, UserAgreement, MarketingConsent, MarketingSource, generate_usernamefrom mwmbl.moderation.suggest import (- prior_decision_counts, prior_decisions, submitter_record, submitter_records,- suggestion_for,+ annotate_queue, annotate_votes, one_row_per_domain, prior_decision_counts, prior_decisions,+ submitter_record, submitter_records, suggestion_for,)+from mwmbl.signals import schedule_blacklist_rebuildfrom mwmbl.platform.schemas import (Registration, ConfirmEmail, DomainSubmissionSchema, UpdateDomainSubmission,VoteRequest, VoteRemoveRequest, VoteStatsRequest, VoteResponse, VoteStats, UserVoteHistory,@@ -36,7 +36,8 @@ from mwmbl.platform.schemas import (ForgotPasswordRequest, ResetPasswordRequest,AgreementAcceptRequest, AgreementResponse,MarketingConsentRequest, MarketingConsentResponse, MarketingConsentListResponse,- BulkDecisionRequest, ModerationQueue, QueueItemSchema, SubmissionDetailSchema,+ BulkDecisionRequest, ModeratedDomainSchema, ModerationHistory, ModerationQueue,+ QueueItemSchema, SubmissionDetailSchema,)logger = logging.getLogger(__name__)@@ -283,22 +284,26 @@ def check_moderator(request):response=ModerationQueue,summary="The moderation queue",description=(- "Domain submissions awaiting review, with the suggestion for each one. "- "Requires the `change_domain_submission_status` permission.\n\n"+ "Domains awaiting review, with the suggestion, the sample pages and the vote counts "+ "for each one. Requires the `change_domain_submission_status` permission.\n\n"+ "One row per *domain*, not per submission: a domain submitted nine times is one card "+ "carrying `submission_count` of 9, and one decision settles all nine. `count` is "+ "therefore distinct pending domains.\n\n""Suggestions are precomputed when a domain is submitted, so this endpoint runs no "- "model: filtering, ordering and pagination all happen in the database. A submission "- "whose domain is still being crawled comes back with `evidence_state` of `PENDING` "- "and no suggestion, rather than a placeholder.\n\n"- "`order_by=needs_review` (the default) puts confident rejections first, then rows the "- "tool is unsure about, then confident approvals, and finally submissions that have "- "not been crawled yet. `oldest` and `confidence` are also accepted."+ "model: filtering, ordering and pagination all happen in the database. A domain "+ "still being crawled comes back with `evidence_state` of `PENDING` and no suggestion, "+ "rather than a placeholder.\n\n"+ "`order_by=submissions` (the default) is the order the review screen shows - most "+ "asked for first, then most upvoted. `needs_review` puts confident rejections first, "+ "then rows the tool is unsure about, then confident approvals, and finally domains "+ "that have not been crawled yet. `oldest` and `confidence` are also accepted."),)def get_moderation_queue(request,limit: int = 50,offset: int = 0,- order_by: str = "needs_review",+ order_by: str = "submissions",suggested_action: str = None,suggested_reason: str = None,submitted_by: int = None,@@ -306,33 +311,24 @@ def get_moderation_queue() -> dict:check_moderator(request)- # DomainEvidence is keyed on the domain name rather than by a foreign key, because domains- # get resubmitted and the crawl belongs to the domain. A correlated subquery joins them- # without one, and keeps the filtering, ordering and slicing in SQL - which is the whole- # reason suggestions are precomputed. Building the list in Python would mean materialising- # every pending submission (there are ~4,000) on every request.- evidence = DomainEvidence.objects.filter(domain=OuterRef("name"))- submissions = (DomainSubmission.objects- .filter(status="PENDING")- # Prefixed because DomainSubmission already has suggested_status and- # suggested_reason columns - the audit of what a moderator was shown -- # and an annotation may not shadow a real field.- .annotate(- evidence_state=Subquery(evidence.values("state")[:1]),- evidence_action=Subquery(evidence.values("suggested_action")[:1]),- evidence_reason=Subquery(evidence.values("suggested_reason")[:1]),- evidence_confidence=Subquery(evidence.values("confidence")[:1]),- evidence_priority=Subquery(evidence.values("review_priority")[:1]),- ))+ # Filtering and ordering happen on the suggestion as displayed, not as stored: the two+ # differ wherever a live check adjusts the cached row, and a filter that disagrees with+ # the screen is worse than no filter. annotate_queue keeps all of it in SQL - building the+ # list in Python would mean materialising every pending submission (there are ~4,000) on+ # every request, which is the whole reason suggestions are precomputed. one_row_per_domain+ # and annotate_votes are in SQL for the same reason: the ordering sorts on both.+ submissions = annotate_votes(one_row_per_domain(+ annotate_queue(DomainSubmission.objects.filter(status="PENDING"))+ )).select_related("submitted_by")if submitted_by is not None:submissions = submissions.filter(submitted_by_id=submitted_by)if suggested_action:- submissions = submissions.filter(evidence_action=suggested_action)+ submissions = submissions.filter(displayed_action=suggested_action)if suggested_reason:- submissions = submissions.filter(evidence_reason=suggested_reason)+ submissions = submissions.filter(displayed_reason=suggested_reason)if min_confidence is not None:- submissions = submissions.filter(evidence_confidence__gte=min_confidence)+ submissions = submissions.filter(displayed_confidence__gte=min_confidence)submissions = submissions.order_by(*_queue_ordering(order_by))@@ -345,17 +341,27 @@ def get_moderation_queue(# rejection, and a moderator can do nothing with one until the crawl lands.QUEUE_ORDERINGS = {"oldest": ("submitted_on",),- "confidence": (F("evidence_confidence").desc(nulls_last=True), "submitted_on"),- "needs_review": (F("evidence_priority").desc(nulls_last=True), "submitted_on"),+ "confidence": (F("displayed_confidence").desc(nulls_last=True), "submitted_on"),+ "needs_review": (F("displayed_priority").desc(nulls_last=True), "submitted_on"),+ # What the review screen says on it: "sorted by submissions, then upvotes". The number of+ # people who asked for a domain is the one signal here that is not the tool's own opinion,+ # which is why it leads rather than the suggestion's confidence.+ "submissions": (F("submission_count").desc(), F("upvotes").desc(), "submitted_on"),}def _queue_ordering(order_by: str):- return QUEUE_ORDERINGS.get(order_by, QUEUE_ORDERINGS["needs_review"])+ return QUEUE_ORDERINGS.get(order_by, QUEUE_ORDERINGS["submissions"])def _queue_items(submissions: list[DomainSubmission]) -> list[QueueItemSchema]:- """Build the page's rows, aggregating the live checks across the page rather than per row."""+ """Build the page's rows, aggregating the live checks across the page rather than per row.++ Each ``submission`` is the earliest pending submission of its domain, carrying the+ per-domain counts one_row_per_domain and annotate_votes put on it. The pages and the+ padlock come off the DomainEvidence rows already fetched here for the suggestion, so+ showing them costs nothing.+ """evidence_by_domain = {row.domain: row for row in DomainEvidence.objects.filter(domain__in=[submission.name for submission in submissions])}submitters = submitter_records({submission.submitted_by_id for submission in submissions})@@ -369,13 +375,18 @@ def _queue_items(submissions: list[DomainSubmission]) -> list[QueueItemSchema]:submission, evidence,submitter=submitters.get(submission.submitted_by_id, empty),prior=priors.get(submission.name, empty))+ signals = evidence.signals if evidence else {}items.append(QueueItemSchema(- id=submission.id,name=submission.name,- submitted_by=submission.submitted_by_id,- submitted_on=submission.submitted_on,- status=submission.status,+ submission_count=submission.submission_count,+ first_submitted_on=submission.submitted_on,+ first_submitted_by=submission.submitted_by_id,+ first_submitted_by_username=submission.submitted_by.username,+ upvotes=submission.upvotes,+ downvotes=submission.downvotes,+ https=signals.get("https"),evidence_state=evidence.state if evidence else "PENDING",+ pages=evidence.pages if evidence else [],suggestion=suggestion.__dict__ if suggestion else None,))return items@@ -395,7 +406,8 @@ def _queue_items(submissions: list[DomainSubmission]) -> list[QueueItemSchema]:def get_submission_detail(request, submission_id: int) -> SubmissionDetailSchema:check_moderator(request)- submission = DomainSubmission.objects.filter(id=submission_id).first()+ submission = (DomainSubmission.objects.filter(id=submission_id)+ .select_related("submitted_by").first())if submission is None:raise InvalidRequest("Submission not found.", status=404)@@ -405,6 +417,7 @@ def get_submission_detail(request, submission_id: int) -> SubmissionDetailSchemaid=submission.id,name=submission.name,submitted_by=submission.submitted_by_id,+ submitted_by_username=submission.submitted_by.username,submitted_on=submission.submitted_on,status=submission.status,rejection_reason=submission.rejection_reason,@@ -426,7 +439,9 @@ def get_submission_detail(request, submission_id: int) -> SubmissionDetailSchemadescription=("Submit a screenful of individually-made decisions in one request. Each entry carries ""its own status, so this cannot be used to accept every suggestion in bulk - a "- "moderator still decides each submission. "+ "moderator still decides each domain.\n\n"+ "A decision is addressed to a domain and settles **every** submission of that name, "+ "including ones already decided - which is also how a past decision is changed. ""Requires the `change_domain_submission_status` permission."),)@@ -434,19 +449,150 @@ def submit_decisions(request, decisions: BulkDecisionRequest):check_email_verified(request)check_moderator(request)- submissions = {submission.id: submission for submission in DomainSubmission.objects.filter(- id__in=[decision.submission_id for decision in decisions.decisions])}+ names = [normalize_domain(decision.domain) for decision in decisions.decisions]+ by_name = {}+ for submission in DomainSubmission.objects.filter(name__in=names):+ by_name.setdefault(submission.name, []).append(submission)++ domains, updated, missing = 0, 0, []+ with transaction.atomic():+ for decision, name in zip(decisions.decisions, names):+ submissions = by_name.get(name)+ if not submissions:+ missing.append(decision.domain)+ continue+ # Every submission of the name, not only the pending ones. A domain is one thing+ # to a moderator, and leaving the already-decided rows behind would let a domain+ # sit half approved and half rejected - and would mean re-deciding one needed a+ # different endpoint from deciding it.+ for submission in submissions:+ apply_decision(submission, decision, request.user)+ updated += 1+ domains += 1++ return {"status": "ok", "domains": domains, "updated": updated, "not_found": missing}+++ "/domain-submissions/domains/{domain}/undo",+ auth=JWTAuth(),+ summary="Undo the decision on a domain",+ description=(+ "Put every submission of a domain back to PENDING and clear the rejection reason and "+ "detail, so it returns to the queue. "+ "Requires the `change_domain_submission_status` permission."+ ),+)+def undo_decision(request, domain: str):+ check_email_verified(request)+ check_moderator(request)++ name = normalize_domain(domain)+ submissions = DomainSubmission.objects.filter(name=name)+ was_approved = submissions.filter(status="APPROVED").exists()+ # Deliberately leaves suggested_status, suggested_reason, suggestion_confidence and+ # suggestion_model_version alone. Those record what was on screen when the decision being+ # undone was made, which is the whole reason they are stored separately from the live+ # suggestion - and re-posting a PENDING status through apply_decision, the nearest thing+ # to an undo before this endpoint, overwrote them from the request.+ undone = submissions.update(+ status="PENDING", rejection_reason="", rejection_detail="",+ status_changed_by=request.user, status_changed_on=timezone.now())+ if not undone:+ raise InvalidRequest(f"No submissions found for {name}.", status=404)++ if was_approved:+ # .update() does not fire post_save, so the approval receiver never runs. An undone+ # approval has to reach the snapshot too: the domain was subtracted from the remote+ # blocklists when it was approved, and until the snapshot is rebuilt it stays+ # subtracted. Same debounce as mwmbl.signals.+ schedule_blacklist_rebuild()++ return {"status": "ok", "message": f"{name} is pending again.", "updated": undone}+++HISTORY_ORDERINGS = {+ "recent": (F("touched").desc(), "-pk"),+ "oldest": (F("touched").asc(), "pk"),+}+++ "/domain-submissions/moderated",+ auth=JWTAuth(),+ response=ModerationHistory,+ summary="Domains that have been moderated",+ description=(+ "Past moderations, one row per domain, newest decision first. Filter by `status` "+ "(`PENDING`, `APPROVED` or `REJECTED`; omit for any), by the `moderator` who made the "+ "call, or by exact domain `name`.\n\n"+ "Each row carries the `suggested_*` audit columns, so a moderator revisiting a "+ "decision sees what was on screen when it was made rather than what the model would "+ "say about the domain today. Changing a decision is a normal POST to "+ "`/domain-submissions/decisions`. "+ "Requires the `change_domain_submission_status` permission."+ ),+)+def get_moderation_history(+ request,+ limit: int = 50,+ offset: int = 0,+ status: str = None,+ moderator: int = None,+ name: str = None,+ order_by: str = "recent",+) -> dict:+ check_moderator(request)++ submissions = DomainSubmission.objects.all()+ if status:+ submissions = submissions.filter(status=status)+ if moderator is not None:+ submissions = submissions.filter(status_changed_by_id=moderator)+ if name:+ submissions = submissions.filter(name=normalize_domain(name))- updated, missing = 0, []- for decision in decisions.decisions:- submission = submissions.get(decision.submission_id)- if submission is None:- missing.append(decision.submission_id)- continue- apply_decision(submission, decision, request.user)- updated += 1+ # "last" rather than "first": a history row shows the decision that currently stands, so+ # the domain is represented by its most recently touched submission, not its oldest.+ submissions = one_row_per_domain(submissions, status=status, pick="last")+ submissions = (submissions.select_related("submitted_by", "status_changed_by")+ .order_by(*HISTORY_ORDERINGS.get(order_by, HISTORY_ORDERINGS["recent"])))- return {"status": "ok", "updated": updated, "not_found": missing}+ count = submissions.count()+ return {"items": _moderated_items(list(submissions[offset:offset + limit])), "count": count}+++def _moderated_items(submissions: list[DomainSubmission]) -> list[ModeratedDomainSchema]:+ """Build the page's rows. The representative row here is the last-touched submission, so+ "who first asked for this domain" needs one more lookup - once for the page, not per row."""+ first_by_name = {}+ for submission in (DomainSubmission.objects+ .filter(name__in=[row.name for row in submissions])+ .select_related("submitted_by").order_by("submitted_on", "pk")):+ first_by_name.setdefault(submission.name, submission)++ items = []+ for submission in submissions:+ first = first_by_name[submission.name]+ items.append(ModeratedDomainSchema(+ name=submission.name,+ submission_count=submission.submission_count,+ status=submission.status,+ rejection_reason=submission.rejection_reason,+ rejection_detail=submission.rejection_detail,+ first_submitted_on=first.submitted_on,+ first_submitted_by=first.submitted_by_id,+ first_submitted_by_username=first.submitted_by.username,+ status_changed_on=submission.status_changed_on,+ status_changed_by=submission.status_changed_by_id,+ status_changed_by_username=(submission.status_changed_by.username+ if submission.status_changed_by else ""),+ suggested_status=submission.suggested_status,+ suggested_reason=submission.suggested_reason,+ suggestion_confidence=submission.suggestion_confidence,+ suggestion_model_version=submission.suggestion_model_version,+ ))+ return items@router.post(@@ -476,8 +622,7 @@ def refetch_submission_evidence(request, submission_id: int):def _index_stats(domain: str) -> dict:"""What the crawler already knows about this domain, when Redis is reachable."""try:- manager = StatsManager(Redis.from_url(settings.REDIS_URL, decode_responses=True))- return manager.get_stats_for_domain(domain).__dict__+ return stats_manager.get_stats_for_domain(domain).__dict__except Exception:logger.warning("Could not read index stats for %s", domain, exc_info=True)return {}diff --git a/mwmbl/platform/schemas.py b/mwmbl/platform/schemas.pyindex 6db74d2..a3d510c 100644--- a/mwmbl/platform/schemas.py+++ b/mwmbl/platform/schemas.py@@ -1,6 +1,7 @@from datetime import datetimefrom typing import Optional, Literalfrom ninja import Schema, ModelSchema, Field+from pydantic import model_validatorfrom mwmbl.models import AgreementType, DomainSubmission, MarketingSource@@ -165,13 +166,44 @@ class ConfirmEmail(Schema):)+class RejectionFieldsMixin:+ """Validate the rejection reason and its detail, wherever a decision arrives.++ Django enforces neither of these. ``choices`` is not checked on ``save()``, so an+ arbitrary string of twenty characters or fewer lands in the column and comes back out at+ a moderator as a reason nothing knows how to render; and nothing has ever required the+ detail that "Other - needs detail" promises, which is the only place a rejected submitter+ finds out what was actually wrong.++ A schema validator rather than a check in the view, so these come back as a 422 naming+ the field, the same as an over-long audit value.+ """++ @model_validator(mode="after")+ def _check_rejection(self):+ # Both columns are blank=True, so ninja types them Optional and an omitted field+ # arrives as None rather than "".+ reason = self.rejection_reason or ""+ detail = self.rejection_detail or ""+ if reason and reason not in DomainSubmission.DOMAIN_REJECTION_REASON:+ allowed = ", ".join(DomainSubmission.DOMAIN_REJECTION_REASON)+ raise ValueError(f"rejection_reason must be one of: {allowed}")+ if reason and self.status != "REJECTED":+ raise ValueError("rejection_reason is only meaningful when status is REJECTED")+ if reason == "OTHER" and not detail.strip():+ raise ValueError(+ "rejection_detail is required when rejection_reason is OTHER - it is what "+ "the submitter is shown")+ return self++class DomainSubmissionSchema(ModelSchema):class Meta:model = DomainSubmissionfields = ["id", "name", "submitted_by", "submitted_on", "status", "rejection_reason", "rejection_detail"]-class UpdateDomainSubmission(ModelSchema):+class UpdateDomainSubmission(RejectionFieldsMixin, ModelSchema):class Meta:model = DomainSubmissionfields = ["status", "rejection_reason", "rejection_detail"]@@ -179,14 +211,21 @@ class UpdateDomainSubmission(ModelSchema):# Echoed back by the client so we record what the moderator was actually looking at.# Optional: a client that shows no suggestions simply omits them, and a decision made# without a suggestion is stored as exactly that rather than as a suggestion of "".+ #+ # The lengths match the columns these land in. Django does not enforce max_length on+ # save(), so without them an over-long value is a 500 from Postgres where it should be a+ # 400 naming the field.suggested_status: Optional[str] = Field(- default=None, description="The action the tool suggested, as shown to the moderator.")+ default=None, max_length=20,+ description="The action the tool suggested, as shown to the moderator.")suggested_reason: Optional[str] = Field(- default=None, description="The rejection reason the tool suggested, if any.")+ default=None, max_length=20,+ description="The rejection reason the tool suggested, if any.")suggestion_confidence: Optional[float] = Field(default=None, description="Confidence of the suggestion that was shown.")suggestion_model_version: Optional[str] = Field(- default=None, description="Version of the model that produced the shown suggestion.")+ default=None, max_length=50,+ description="Version of the model that produced the shown suggestion.")class EvidenceItemSchema(Schema):@@ -223,14 +262,35 @@ class CrawledPageSchema(Schema):class QueueItemSchema(Schema):- """A row of the moderation queue: the submission plus its precomputed suggestion."""- id: int+ """One pending domain: everything the review card and its up-next row draw.++ Keyed on the domain, not on a submission. A domain submitted nine times is one thing to+ look at and one decision to make, so there is no single submission id behind a row -+ decisions are addressed to `name` (see `DomainDecision`).++ Everything a card shows is here so that moving to the next one costs no request: the+ sample pages, the vote counts, the padlock and the suggestion all travel with the row.+ """name: str- submitted_by: int- submitted_on: datetime- status: str+ submission_count: int = Field(+ description="How many submissions of this domain are still pending.")+ first_submitted_on: datetime = Field(+ description="When this domain was first submitted, by whoever asked first.")+ first_submitted_by: int+ first_submitted_by_username: str+ upvotes: int = Field(+ description="Votes cast on results from this domain, ignoring a leading `www.`.")+ downvotes: int+ https: Optional[bool] = Field(+ default=None,+ description=(+ "Whether the homepage answered over TLS. `null` until the domain has been "+ "crawled - an uncrawled domain must not draw an open padlock."+ ))evidence_state: str = Field(description="`PENDING` while the domain is still being crawled, then `READY` or `FAILED`.")+ pages: list[CrawledPageSchema] = Field(+ default=[], description="The pages crawled from the domain, as shown on the card.")suggestion: Optional[SuggestionSchema] = Field(default=None,description="Absent until the domain has been crawled. Never a placeholder.")@@ -241,6 +301,7 @@ class SubmissionDetailSchema(Schema):id: intname: strsubmitted_by: int+ submitted_by_username: str = ""submitted_on: datetimestatus: strrejection_reason: str = ""@@ -257,15 +318,21 @@ class SubmissionDetailSchema(Schema):default={}, description="What the crawler already knows about this domain.")-class BulkDecision(Schema):- submission_id: int- status: str- rejection_reason: str = ""- rejection_detail: str = ""- suggested_status: Optional[str] = None- suggested_reason: Optional[str] = None+class DomainDecision(RejectionFieldsMixin, Schema):+ """One decision, about one domain. Lengths match the columns these are stored in.++ Addressed to the domain rather than to a submission id, because that is what a moderator+ decided about: a decision settles every submission of the name at once, so nine asks for+ the same site cannot end up half approved and half rejected.+ """+ domain: str = Field(max_length=300)+ status: str = Field(max_length=20)+ rejection_reason: str = Field(default="", max_length=20)+ rejection_detail: str = Field(default="", max_length=300)+ suggested_status: Optional[str] = Field(default=None, max_length=20)+ suggested_reason: Optional[str] = Field(default=None, max_length=20)suggestion_confidence: Optional[float] = None- suggestion_model_version: Optional[str] = None+ suggestion_model_version: Optional[str] = Field(default=None, max_length=50)class BulkDecisionRequest(Schema):@@ -274,7 +341,37 @@ class BulkDecisionRequest(Schema):Deliberately not "accept all suggestions": each entry carries its own status, so theclient can only send choices a moderator actually made."""- decisions: list[BulkDecision]+ decisions: list[DomainDecision]+++class ModeratedDomainSchema(Schema):+ """One domain in the moderation history: the decision that currently stands on it.++ One row per domain, like the queue, and for the same reason - and it carries the+ `suggested_*` audit columns so a moderator revisiting a call can see what was on screen+ when it was made, rather than what the model would say about the domain today.+ """+ name: str+ submission_count: int+ status: str+ rejection_reason: str = ""+ rejection_detail: str = ""+ first_submitted_on: datetime+ first_submitted_by: int+ first_submitted_by_username: str+ status_changed_on: Optional[datetime] = None+ status_changed_by: Optional[int] = None+ status_changed_by_username: str = ""+ suggested_status: str = ""+ suggested_reason: str = ""+ suggestion_confidence: Optional[float] = None+ suggestion_model_version: str = ""+++class ModerationHistory(Schema):+ """A page of past moderations. Same `items`/`count` shape as the queue."""+ items: list[ModeratedDomainSchema]+ count: intclass VoteRequest(Schema):@@ -381,6 +478,10 @@ class UserVoteHistory(Schema):class ModerationQueue(Schema):"""A page of the moderation queue. Shape matches the paginated endpoints the client- already consumes, so `items`/`count` need no special handling."""+ already consumes, so `items`/`count` need no special handling.++ `count` is distinct pending *domains* - the "24 pending" in the header - not pending+ submissions.+ """items: list[QueueItemSchema]count: intdiff --git a/mwmbl/settings_common.py b/mwmbl/settings_common.pyindex b04cd28..04cbd5d 100644--- a/mwmbl/settings_common.py+++ b/mwmbl/settings_common.py@@ -355,6 +355,10 @@ EXTERNAL_CACHE_NEGATIVE_TTL_SECONDS = 7 * 24 * 60 * 60# Domain moderation suggestions. The model is loaded only by the background worker that# enriches submissions - the queue reads the stored suggestion - so a missing artifact# degrades to the deterministic checks rather than affecting any moderator request.+#+# Retrained models are stored in Postgres (mwmbl.models.ModerationModelArtifact), because a+# container filesystem is neither shared between workers nor kept across a deploy. This+# directory holds the read-only warm start used until the first retrain publishes one.DOMAIN_MODERATION_MODEL_DIR = os.environ.get("DOMAIN_MODERATION_MODEL_DIR",str(Path(__file__).parent / "moderation" / "artifacts"))diff --git a/mwmbl/signals.py b/mwmbl/signals.pyindex 16a4791..0b58ce0 100644--- a/mwmbl/signals.py+++ b/mwmbl/signals.py@@ -62,13 +62,23 @@ def rebuild_blacklist_snapshot_on_approval(sender, instance: DomainSubmission, *if instance.status != "APPROVED":return+ if schedule_blacklist_rebuild():+ logger.info("Scheduled a blacklist snapshot rebuild after approving %s", instance.name)+++def schedule_blacklist_rebuild() -> bool:+ """Ask for a snapshot rebuild, unless one is already due. True if this call scheduled it.++ Shared with the undo endpoint, which changes statuses with .update() and so fires no+ post_save at all: an approval that is undone has to reach the snapshot for the same reason+ the approval did, in the other direction.+ """from mwmbl.background import refresh_blacklist_snapshotdelay = settings.BLACKLIST_SNAPSHOT_APPROVAL_DELAY_SECONDScutoff = timezone.now() + timedelta(seconds=delay)if Task.objects.filter(task_name=BLACKLIST_SNAPSHOT_TASK, run_at__lte=cutoff).exists():- return+ return Falserefresh_blacklist_snapshot(schedule=delay)- logger.info("Scheduled a blacklist snapshot rebuild in %ds after approving %s",- delay, instance.name)+ return Truediff --git a/mwmbl/utils.py b/mwmbl/utils.pyindex 3c5645b..801b843 100644--- a/mwmbl/utils.py+++ b/mwmbl/utils.py@@ -93,6 +93,20 @@ def normalize_domain(domain_or_url: str) -> str:return netloc.lower()+def bare_host(domain_or_url: str) -> str:+ """:func:`normalize_domain`, minus a leading ``www.``.++ >>> bare_host("https://www.example.com/a/b?c=d"), bare_host("example.com")+ ('example.com', 'example.com')++ Votes are cast against result URLs and domains are submitted by hand, so the two disagree+ about ``www.`` constantly. Reducing both sides to the same host is what makes a vote on+ www.example.com/x count towards a submission of example.com.+ """+ host = normalize_domain(domain_or_url)+ return host[4:] if host.startswith("www.") else host++def validate_domain(domain_or_url: str):domain = normalize_domain(domain_or_url)if VALID_DOMAIN_REGEX.fullmatch(domain) is None:diff --git a/test/test_crawl_functional.py b/test/test_crawl_functional.pyindex 0a60f28..b16a8cb 100644--- a/test/test_crawl_functional.py+++ b/test/test_crawl_functional.py@@ -242,11 +242,12 @@ class TestCrawlFunctional:with patch('mwmbl.crawler.retrieve.fetch') as mock_fetch:with patch('mwmbl.crawler.retrieve.robots_allowed', return_value=True):- mock_fetch.return_value = (200, mock_html_content)+ mock_fetch.return_value = (200, mock_html_content, 'https://example.com')result = crawl_url('https://example.com', redis=None)assert result['url'] == 'https://example.com'+ assert result['resolved_url'] == 'https://example.com'assert result['status'] == 200assert result['content'] is not Noneassert result['content']['title'] == 'Test Page'diff --git a/test/test_domain_moderation.py b/test/test_domain_moderation.pyindex 176693d..b941ad0 100644--- a/test/test_domain_moderation.py+++ b/test/test_domain_moderation.py@@ -5,20 +5,28 @@ the training gate - but that a moderator is never shown something misleading: aeither backed by evidence or absent, a deterministic check always beats a probability, and thequeue never runs a model on the request path."""+import io+import jsonfrom datetime import timedeltafrom unittest import mock+import joblibimport pytestfrom allauth.account.models import EmailAddress+from background_task.models import Taskfrom django.contrib.auth.models import Permissionfrom django.utils import timezonefrom ninja_jwt.tokens import RefreshToken-from mwmbl.models import DomainEvidence, DomainSubmission, MwmblUser+from mwmbl.models import (+ DomainEvidence, DomainSubmission, ModerationModelArtifact, MwmblUser, SearchResultVote)+from mwmbl.moderation import model as model_modulefrom mwmbl.moderation import rules+from mwmbl.moderation.evidence import crawl_domainfrom mwmbl.moderation.features import Featuriser, ModerationExample-from mwmbl.moderation.model import Suggestion, suggest-from mwmbl.moderation.suggest import refresh_suggestion, suggestion_for+from mwmbl.moderation.model import (+ Suggestion, get_model, load_metrics, publish, reset_model_cache, suggest)+from mwmbl.moderation.suggest import annotate_queue, refresh_suggestion, suggestion_forfrom mwmbl.moderation.train import passes_gatefrom mwmbl.moderation.training_data import TrainingRow, is_trainable_domain@@ -31,6 +39,25 @@ def submitter(db):username="submitter", email="[email protected]", password="password")[email protected](autouse=True)+def clear_the_model_cache():+ """The loaded model is a process-wide singleton, and several tests here publish one."""+ reset_model_cache()+ yield+ reset_model_cache()+++def established(db):+ """A submitter with a track record, whose approvals are not withheld."""+ user = MwmblUser.objects.create_user(+ username="established", email="[email protected]", password="password")+ for index in range(3):+ DomainSubmission.objects.create(name=f"past{index}.example", submitted_by=user,+ status="APPROVED")+ return user++@pytest.fixturedef moderator(db):user = MwmblUser.objects.create_user(@@ -55,7 +82,7 @@ def ready_evidence(domain, **overrides):"signals": {"has_links": True, "num_pages_fetched": 1, "blacklisted": False},"suggested_action": "APPROVE","confidence": 0.8,- "review_priority": 0.2,+ "reason_source": "model","evidence": [],"model_version": "test-model",}@@ -209,20 +236,21 @@ def test_uncrawled_submission_reports_that_rather_than_guessing(client, moderato@pytest.mark.django_db-def test_queue_orders_rows_that_need_a_human_first(client, moderator, submitter):- for name, action, priority in [("approve-me.com", "APPROVE", 0.1),- ("reject-me.com", "REJECT", 1.9),- ("unsure.com", "UNSURE", 1.0)]:- DomainSubmission.objects.create(name=name, submitted_by=submitter)- ready_evidence(name, suggested_action=action, review_priority=priority)-- response = client.get(QUEUE_URL, headers={"Authorization": token(moderator)})+def test_queue_orders_rows_that_need_a_human_first(client, moderator, established):+ for name, action in [("approve-me.com", "APPROVE"),+ ("reject-me.com", "REJECT"),+ ("unsure.com", "UNSURE")]:+ DomainSubmission.objects.create(name=name, submitted_by=established)+ ready_evidence(name, suggested_action=action, confidence=0.8)++ response = client.get(f"{QUEUE_URL}?order_by=needs_review",+ headers={"Authorization": token(moderator)})assert [item["name"] for item in response.json()["items"]] == ["reject-me.com", "unsure.com", "approve-me.com"]@pytest.mark.django_db-def test_queue_filters_on_the_stored_suggestion(client, moderator, submitter):+def test_queue_filters_on_the_suggestion(client, moderator, submitter):for name, action in [("a.com", "APPROVE"), ("b.com", "REJECT")]:DomainSubmission.objects.create(name=name, submitted_by=submitter)ready_evidence(name, suggested_action=action)@@ -261,8 +289,8 @@ def test_bulk_decisions_apply_each_choice_separately(client, moderator, submitteresponse = client.post("/api/v1/platform/domain-submissions/decisions",data={"decisions": [- {"submission_id": first.id, "status": "APPROVED"},- {"submission_id": second.id, "status": "REJECTED", "rejection_reason": "SPAM"},+ {"domain": "a.com", "status": "APPROVED"},+ {"domain": "b.com", "status": "REJECTED", "rejection_reason": "SPAM"},]},content_type="application/json", headers={"Authorization": token(moderator)})@@ -287,6 +315,296 @@ def test_refetch_clears_the_evidence_so_the_crawl_actually_reruns(client, moderaenrich.assert_called_once_with("was-down.com")+# ------------------------------------------------------------ the queue is per domain++DECISIONS_URL = "/api/v1/platform/domain-submissions/decisions"+HISTORY_URL = "/api/v1/platform/domain-submissions/moderated"+++def upvote(url, username):+ SearchResultVote.objects.create(+ user=MwmblUser.objects.create_user(username=username), url=url, query="q",+ vote_type="upvote")+++def downvote(url, username):+ SearchResultVote.objects.create(+ user=MwmblUser.objects.create_user(username=username), url=url, query="q",+ vote_type="downvote")+++def test_a_domain_submitted_nine_times_is_one_row(client, moderator, established):+ """A moderator reviews a domain, not a submission. Nine asks for the same site are one+ card and one decision - the old queue made them nine of each."""+ for _ in range(9):+ DomainSubmission.objects.create(name="cheap-rolex.biz", submitted_by=established)+ DomainSubmission.objects.create(name="solarpunk.zone", submitted_by=established)+ ready_evidence("cheap-rolex.biz")+ ready_evidence("solarpunk.zone")++ body = client.get(QUEUE_URL, headers={"Authorization": token(moderator)}).json()++ assert body["count"] == 2+ counts = {item["name"]: item["submission_count"] for item in body["items"]}+ assert counts == {"cheap-rolex.biz": 9, "solarpunk.zone": 1}+++def test_the_row_is_the_first_submission_of_the_domain(client, moderator):+ """The card says "first submitted 6 days ago by anon_4417", so the row we deduplicate to+ has to be the row those two fields come from."""+ first = MwmblUser.objects.create_user(username="anon_4417")+ later = MwmblUser.objects.create_user(username="someone_else")+ DomainSubmission.objects.create(name="a.com", submitted_by=first,+ submitted_on=timezone.now() - timedelta(days=6))+ DomainSubmission.objects.create(name="a.com", submitted_by=later,+ submitted_on=timezone.now() - timedelta(days=1))+ ready_evidence("a.com")++ item = client.get(QUEUE_URL, headers={"Authorization": token(moderator)}).json()["items"][0]++ assert item["first_submitted_by_username"] == "anon_4417"+ assert item["first_submitted_by"] == first.id+++def test_votes_are_rolled_up_to_the_domain(client, moderator, established):+ """The card shows the domain's score, so votes on any of its URLs count - including ones+ cast against www., which is how half the index refers to the same site."""+ DomainSubmission.objects.create(name="solarpunk.zone", submitted_by=established)+ ready_evidence("solarpunk.zone")+ upvote("https://solarpunk.zone/notes/co-op", "voter1")+ upvote("https://www.solarpunk.zone/repair-cafes", "voter2")+ downvote("https://solarpunk.zone/about", "voter3")+ upvote("https://somewhere-else.com/", "voter4")++ DomainSubmission.objects.create(name="silent.example", submitted_by=established)+ ready_evidence("silent.example")++ items = {item["name"]: item+ for item in client.get(QUEUE_URL,+ headers={"Authorization": token(moderator)}).json()["items"]}++ assert (items["solarpunk.zone"]["upvotes"], items["solarpunk.zone"]["downvotes"]) == (2, 1)+ # Zero, not null: a domain nobody has voted on has to sort with the rest.+ assert (items["silent.example"]["upvotes"], items["silent.example"]["downvotes"]) == (0, 0)+++def test_a_www_submission_still_finds_its_votes(client, moderator, established):+ DomainSubmission.objects.create(name="www.example.com", submitted_by=established)+ ready_evidence("www.example.com")+ upvote("https://example.com/a", "voter1")++ item = client.get(QUEUE_URL, headers={"Authorization": token(moderator)}).json()["items"][0]+ assert item["upvotes"] == 1+++def test_the_default_order_is_most_submitted_then_most_upvoted(client, moderator, established):+ for name, submissions, upvotes in [("popular.com", 3, 1), ("tied-low.com", 2, 1),+ ("tied-high.com", 2, 40), ("lonely.com", 1, 99)]:+ for _ in range(submissions):+ DomainSubmission.objects.create(name=name, submitted_by=established)+ for index in range(upvotes):+ upvote(f"https://{name}/{index}", f"voter-{name}-{index}")+ ready_evidence(name)++ body = client.get(QUEUE_URL, headers={"Authorization": token(moderator)}).json()+ assert [item["name"] for item in body["items"]] == [+ "popular.com", "tied-high.com", "tied-low.com", "lonely.com"]+++def test_the_card_carries_its_sample_pages_and_padlock(client, moderator, established):+ """Everything a card draws travels with the row, so advancing to the next one costs no+ request."""+ DomainSubmission.objects.create(name="crawled.com", submitted_by=established)+ ready_evidence("crawled.com", signals={"has_links": True, "https": False})+ DomainSubmission.objects.create(name="uncrawled.com", submitted_by=established)++ items = {item["name"]: item+ for item in client.get(QUEUE_URL,+ headers={"Authorization": token(moderator)}).json()["items"]}++ assert items["crawled.com"]["pages"][0]["title"] == "A title"+ assert items["crawled.com"]["https"] is False+ # Not False: an uncrawled domain must not draw an open padlock.+ assert items["uncrawled.com"]["https"] is None+ assert items["uncrawled.com"]["pages"] == []+++def test_one_decision_settles_every_submission_of_the_domain(client, moderator, submitter):+ for _ in range(9):+ DomainSubmission.objects.create(name="cheap-rolex.biz", submitted_by=submitter)++ response = client.post(+ DECISIONS_URL,+ data={"decisions": [{"domain": "cheap-rolex.biz", "status": "REJECTED",+ "rejection_reason": "SPAM"}]},+ content_type="application/json", headers={"Authorization": token(moderator)})++ assert response.json() == {"status": "ok", "domains": 1, "updated": 9, "not_found": []}+ assert not DomainSubmission.objects.filter(name="cheap-rolex.biz",+ status="PENDING").exists()+++def test_a_decision_can_be_changed_after_the_fact(client, moderator, submitter):+ """Re-deciding is the same request as deciding: a domain already rejected can be approved+ without a second endpoint, and does not end up half one and half the other."""+ DomainSubmission.objects.create(name="a.com", submitted_by=submitter, status="REJECTED",+ rejection_reason="SPAM")+ DomainSubmission.objects.create(name="a.com", submitted_by=submitter)++ client.post(DECISIONS_URL,+ data={"decisions": [{"domain": "a.com", "status": "APPROVED"}]},+ content_type="application/json",+ headers={"Authorization": token(moderator)})++ statuses = set(DomainSubmission.objects.filter(name="a.com")+ .values_list("status", "rejection_reason"))+ assert statuses == {("APPROVED", "")}+++def test_a_decision_on_an_unknown_domain_is_reported_not_invented(client, moderator):+ response = client.post(+ DECISIONS_URL,+ data={"decisions": [{"domain": "never-submitted.com", "status": "APPROVED"}]},+ content_type="application/json", headers={"Authorization": token(moderator)})++ assert response.json()["not_found"] == ["never-submitted.com"]+++# ------------------------------------------------------------------- undo and history++UNDO_URL = "/api/v1/platform/domain-submissions/domains/{}/undo"+++def test_undo_returns_a_domain_to_the_queue_without_rewriting_the_audit(client, moderator,+ submitter):+ """The suggested_* columns record what was on screen when the decision was made. Posting+ a PENDING status back through the decision endpoint - the nearest thing to an undo before+ this - overwrote them from the request, destroying the one thing they exist for."""+ submission = DomainSubmission.objects.create(name="a.com", submitted_by=submitter)+ ready_evidence("a.com")+ client.post(DECISIONS_URL,+ data={"decisions": [{"domain": "a.com", "status": "REJECTED",+ "rejection_reason": "SPAM",+ "suggested_status": "REJECT",+ "suggested_reason": "SPAM",+ "suggestion_confidence": 0.91,+ "suggestion_model_version": "2026-08-01"}]},+ content_type="application/json", headers={"Authorization": token(moderator)})++ response = client.post(UNDO_URL.format("a.com"),+ headers={"Authorization": token(moderator)})++ assert response.status_code == 200+ submission.refresh_from_db()+ assert (submission.status, submission.rejection_reason) == ("PENDING", "")+ assert (submission.suggested_status, submission.suggested_reason) == ("REJECT", "SPAM")+ assert submission.suggestion_confidence == 0.91+ assert submission.suggestion_model_version == "2026-08-01"++ body = client.get(QUEUE_URL, headers={"Authorization": token(moderator)}).json()+ assert [item["name"] for item in body["items"]] == ["a.com"]+++def test_undoing_an_approval_rebuilds_the_blacklist_snapshot(client, moderator, submitter):+ """An approved domain is subtracted from the remote blocklists when the snapshot is built,+ so it stays subtracted until the snapshot is rebuilt. .update() fires no post_save, so the+ approval receiver never runs and the undo has to ask for the rebuild itself."""+ DomainSubmission.objects.create(name="a.com", submitted_by=submitter, status="APPROVED")+ # The approval itself scheduled one, and the debounce would rightly decline a second.+ # Clearing it is the run having already happened, which is the case the undo is about.+ Task.objects.all().delete()++ with mock.patch("mwmbl.background.refresh_blacklist_snapshot") as refresh:+ client.post(UNDO_URL.format("a.com"), headers={"Authorization": token(moderator)})++ refresh.assert_called_once()+++def test_undoing_a_rejection_does_not_rebuild_the_snapshot(client, moderator, submitter):+ """Only an approval ever changed the snapshot, so only undoing one has to change it back -+ a rebuild downloads and parses tens of megabytes."""+ DomainSubmission.objects.create(name="a.com", submitted_by=submitter, status="REJECTED",+ rejection_reason="SPAM")+ Task.objects.all().delete()++ with mock.patch("mwmbl.background.refresh_blacklist_snapshot") as refresh:+ client.post(UNDO_URL.format("a.com"), headers={"Authorization": token(moderator)})++ refresh.assert_not_called()+++def test_undoing_a_domain_nobody_submitted_is_a_404(client, moderator):+ response = client.post(UNDO_URL.format("never-submitted.com"),+ headers={"Authorization": token(moderator)})+ assert response.status_code == 404+++def test_history_lists_past_decisions_by_status_newest_first(client, moderator, submitter):+ for name, status, when in [("old-reject.com", "REJECTED", 5),+ ("new-reject.com", "REJECTED", 1),+ ("approved.com", "APPROVED", 2)]:+ DomainSubmission.objects.create(+ name=name, submitted_by=submitter, status=status,+ status_changed_by=moderator, status_changed_on=timezone.now() - timedelta(days=when))++ body = client.get(f"{HISTORY_URL}?status=REJECTED",+ headers={"Authorization": token(moderator)}).json()++ assert [item["name"] for item in body["items"]] == ["new-reject.com", "old-reject.com"]+ assert body["count"] == 2+ assert body["items"][0]["status_changed_by_username"] == "moderator"+++def test_history_shows_one_row_per_domain(client, moderator, submitter):+ for index in range(4):+ DomainSubmission.objects.create(+ name="a.com", submitted_by=submitter, status="REJECTED", rejection_reason="SPAM",+ status_changed_by=moderator,+ status_changed_on=timezone.now() - timedelta(days=index))++ body = client.get(HISTORY_URL, headers={"Authorization": token(moderator)}).json()++ assert body["count"] == 1+ assert body["items"][0]["submission_count"] == 4+++def test_history_carries_what_the_moderator_was_shown(client, moderator, submitter):+ DomainSubmission.objects.create(+ name="a.com", submitted_by=submitter, status="REJECTED", rejection_reason="SPAM",+ status_changed_by=moderator, status_changed_on=timezone.now(),+ suggested_status="REJECT", suggested_reason="SPAM", suggestion_confidence=0.77,+ suggestion_model_version="2026-07-01")++ item = client.get(HISTORY_URL,+ headers={"Authorization": token(moderator)}).json()["items"][0]++ assert item["suggested_status"] == "REJECT"+ assert item["suggestion_confidence"] == 0.77+ assert item["suggestion_model_version"] == "2026-07-01"+++def test_history_requires_the_moderator_permission(client, submitter):+ response = client.get(HISTORY_URL, headers={"Authorization": token(submitter)})+ assert response.status_code == 403++# --------------------------------------------------------------------- rescoring@pytest.mark.django_db@@ -333,7 +651,7 @@ def test_enrichment_recrawls_stale_evidence(submitter):@pytest.mark.django_db-def test_queue_cost_does_not_grow_with_the_backlog(client, moderator, submitter,+def test_queue_cost_does_not_grow_with_the_backlog(client, moderator, established,django_assert_max_num_queries):"""Ordering, filtering and slicing must happen in SQL.@@ -344,30 +662,40 @@ def test_queue_cost_does_not_grow_with_the_backlog(client, moderator, submitter,"""for index in range(30):name = f"site{index}.example"- DomainSubmission.objects.create(name=name, submitted_by=submitter)- ready_evidence(name, review_priority=index / 30)+ # Several submissions and several votes per domain, so that grouping to one row per+ # domain and rolling the votes up cannot quietly become a query per row either.+ for _ in range(3):+ DomainSubmission.objects.create(name=name, submitted_by=established)+ for voter in range(index % 4):+ SearchResultVote.objects.create(+ user=MwmblUser.objects.create_user(username=f"voter{index}-{voter}"),+ url=f"https://{name}/page{voter}", query="q", vote_type="upvote")+ ready_evidence(name, suggested_action="REJECT", confidence=index / 30)# Auth, permissions, count, page, evidence, submitter counts, prior-decision counts.with django_assert_max_num_queries(12):- response = client.get(f"{QUEUE_URL}?limit=5",+ response = client.get(f"{QUEUE_URL}?limit=5&order_by=needs_review",headers={"Authorization": token(moderator)})body = response.json()+ # Distinct pending domains, not the 90 pending submissions behind them.assert body["count"] == 30assert len(body["items"]) == 5# ...and the page is the top of the global ordering, not the top of an arbitrary slice.assert body["items"][0]["name"] == "site29.example"+ assert body["items"][0]["submission_count"] == 3+ assert body["items"][0]["upvotes"] == 1@pytest.mark.django_db-def test_queue_paginates_over_the_whole_ordering(client, moderator, submitter):+def test_queue_paginates_over_the_whole_ordering(client, moderator, established):for index in range(10):name = f"site{index}.example"- DomainSubmission.objects.create(name=name, submitted_by=submitter)- ready_evidence(name, review_priority=index / 10)+ DomainSubmission.objects.create(name=name, submitted_by=established)+ ready_evidence(name, suggested_action="REJECT", confidence=index / 10)def page(offset):- response = client.get(f"{QUEUE_URL}?limit=4&offset={offset}",+ response = client.get(f"{QUEUE_URL}?limit=4&offset={offset}&order_by=needs_review",headers={"Authorization": token(moderator)})return [item["name"] for item in response.json()["items"]]@@ -508,3 +836,410 @@ def test_rejections_are_not_withheld_from_first_time_submitters(submitter):assert suggestion.action == "REJECT"assert suggestion.reason == "SPAM"+++# --------------------------------------------------------------------- crawl evidence++class _FakeResponse:+ """Enough of a requests Response for retrieve.fetch."""++ def __init__(self, status_code, headers=None, body=b""):+ self.status_code = status_code+ self.headers = headers or {}+ self.is_redirect = 300 <= status_code < 400+ self.next = object() if self.is_redirect else None+ self._body = body++ def iter_content(self, size):+ yield self._body++ def close(self):+ pass+++A_PAGE = (b"<html><head><title>Buy this domain</title></head><body>"+ b"<p>This premium domain name is for sale. Make an offer today and we will be in "+ b"touch with you shortly to complete the transfer.</p></body></html>")+++def crawl_with(responses: dict, domain: str) -> dict:+ """Crawl ``domain`` against canned HTTP responses, through the real crawler.++ Driven from requests.get rather than by stubbing crawl_url, because everything this+ covers lives in the plumbing between the fetch and the evidence.+ """+ def get(url, **kwargs):+ response = responses[url]+ if isinstance(response, Exception):+ raise response+ return response++ blacklist = mock.Mock(is_domain_blacklisted=mock.Mock(return_value=False))+ with mock.patch("mwmbl.crawler.retrieve.requests.get", get), \+ mock.patch("mwmbl.crawler.retrieve.validate_url"), \+ mock.patch("mwmbl.crawler.retrieve.robots_allowed", return_value=True), \+ mock.patch("mwmbl.moderation.evidence.get_snapshot_blacklist",+ return_value=blacklist):+ return crawl_domain(domain, redis=None)+++def test_a_domain_that_redirects_elsewhere_is_reported_as_a_redirect():+ """Roughly a third of the rejection details moderators write are about dead or squatted+ domains, and "redirects to somewhere else" is the fetch that proves it. The check is only+ as good as the URL the crawler reports: reporting the requested one compares the domain+ against itself and the rule can never fire."""+ crawl = crawl_with({+ "https://squatted.example/": _FakeResponse(+ 301, headers={"Location": "https://parked.test/lander"}),+ "https://parked.test/lander": _FakeResponse(200, body=A_PAGE),+ }, "squatted.example")++ assert crawl["final_domain"] == "parked.test"+ decisive = rules.decisive(rules.crawl_evidence("squatted.example", crawl))+ assert decisive.kind == "redirect"+ assert decisive.implies_action == "REJECT"+++def test_a_domain_that_serves_its_own_pages_reports_no_redirect():+ """The other direction, because the check is decisive: a site that answers for itself must+ never be rejected for redirecting to itself."""+ crawl = crawl_with(+ {"https://honest.example/": _FakeResponse(200, body=A_PAGE)}, "honest.example")++ assert crawl["final_domain"] == ""+ kinds = {item.kind for item in rules.crawl_evidence("honest.example", crawl)}+ assert "redirect" not in kinds+++def test_a_site_without_a_certificate_is_a_site_not_a_dead_domain():+ """Only https used to be tried, so "no TLS" and "not there" were the same result - and+ that result is a decisive REJECT for being unreachable. A site served over plain http is+ a site; whether it has a certificate is a line for the moderator, not a verdict."""+ crawl = crawl_with({+ "https://no-cert.example/": ConnectionError("certificate verify failed"),+ "http://no-cert.example/": _FakeResponse(200, body=A_PAGE),+ }, "no-cert.example")++ assert crawl["signals"]["https"] is False+ assert crawl["error"] == ""+ items = {item.kind: item for item in rules.crawl_evidence("no-cert.example", crawl)}+ assert items["no_tls"].direction == rules.NEUTRAL+ assert rules.decisive(items.values()) is None+++def test_a_domain_that_is_simply_down_is_still_reported_as_unreachable():+ """The other direction: falling back to http must not turn a dead domain into a live one,+ and must not label it "no TLS" - the https attempt is the one that describes production."""+ crawl = crawl_with({+ "https://gone.example/": ConnectionError("no route to host"),+ "http://gone.example/": ConnectionError("no route to host"),+ }, "gone.example")++ assert crawl["signals"]["https"] is False+ assert crawl["error"] == "AbortError"+ kinds = {item.kind for item in rules.crawl_evidence("gone.example", crawl)}+ assert "no_tls" not in kinds+ assert rules.decisive(rules.crawl_evidence("gone.example", crawl)).kind == "unreachable"+++def test_a_normal_https_site_is_not_asked_for_over_http():+ crawl = crawl_with(+ {"https://secure.example/": _FakeResponse(200, body=A_PAGE)}, "secure.example")++ assert crawl["signals"]["https"] is True+ assert "no_tls" not in {item.kind+ for item in rules.crawl_evidence("secure.example", crawl)}+++# --------------------------------------------------------------------- enrichment++def test_enrichment_never_commits_a_scored_row_without_the_score():+ """READY with a fresh timestamp is what makes the task skip a domain for a month. A row+ that reached it without a suggestion - a worker restarted between the two writes - would+ tell moderators "not assessed yet" until the evidence expired."""+ from mwmbl.background import enrich_domain_submission++ crawl = {"http_status": 200, "final_domain": "", "error": "", "pages": [], "signals": {}}+ with mock.patch("mwmbl.background.crawl_domain", return_value=crawl), \+ mock.patch("mwmbl.background.refresh_suggestion",+ side_effect=RuntimeError("scoring blew up")):+ with pytest.raises(RuntimeError):+ enrich_domain_submission.now("half-written.example")++ assert not DomainEvidence.objects.filter(domain="half-written.example").exists()+++def test_a_failed_recrawl_does_not_leave_the_old_crawl_on_screen():+ """The detail panel renders whatever is on the row, so a month-old crawl next to a FAILED+ state reads as current evidence about a site that is in fact unreachable."""+ from mwmbl.background import enrich_domain_submission++ ready_evidence("gone.example", fetched_at=timezone.now() - timedelta(days=365))+ with mock.patch("mwmbl.background.crawl_domain",+ side_effect=ConnectionError("no route to host")):+ enrich_domain_submission.now("gone.example")++ evidence = DomainEvidence.objects.get(domain="gone.example")+ assert evidence.state == DomainEvidence.State.FAILED+ assert evidence.error == "ConnectionError"+ assert evidence.pages == []+ assert evidence.suggested_action == ""+ assert evidence.confidence is None+++def test_the_reason_is_shown_with_its_own_confidence(submitter):+ """A model can be sure a domain should go and barely have a view on why. Showing the+ rejection's confidence next to the reason presents a guess as a finding."""+ submission = DomainSubmission.objects.create(name="spam.example", submitted_by=submitter)+ model = mock.Mock(version="test-model")+ model.predict.return_value = [(0.92, "SPAM", 0.31)]++ with mock.patch("mwmbl.moderation.model.get_model", return_value=model):+ refresh_suggestion(ready_evidence("spam.example"))++ evidence = DomainEvidence.objects.get(domain="spam.example")+ assert evidence.confidence == pytest.approx(0.92)+ assert evidence.reason_confidence == pytest.approx(0.31)++ suggestion = suggestion_for(submission, evidence)+ assert suggestion.reason == "SPAM"+ assert suggestion.reason_confidence == pytest.approx(0.31)+ assert suggestion.reason_source == "model"+++# --------------------------------------------------------------------- queue parity++def test_queue_display_matches_suggestion_for(submitter, established):+ """The queue filters and orders in SQL; the rows are rendered by suggestion_for. Where the+ two disagree, a filter hides rows that are on screen and the ordering sorts on a number+ nobody is shown - so every branch of the adjustment is checked against the other."""+ DomainSubmission.objects.create(name="approved-before.example",+ submitted_by=established, status="APPROVED")+ DomainSubmission.objects.create(name="rejected-before.example",+ submitted_by=established, status="REJECTED")+ DomainSubmission.objects.create(name="dead-and-approved-before.example",+ submitted_by=established, status="APPROVED")++ a_404 = rules.EvidenceItem(+ "http_status", rules.REJECT, "Homepage returns HTTP 404", implies_action="REJECT",+ implies_reason="OTHER", implies_confidence=0.9).to_dict()++ cases = [+ # (domain, submitter, evidence overrides) - one per branch of the adjustment.+ ("first-timer-approve.example", submitter, {}),+ ("known-approve.example", established, {}),+ ("reject.example", submitter, {"suggested_action": "REJECT",+ "suggested_reason": "SPAM", "confidence": 0.91}),+ ("unsure.example", submitter, {"suggested_action": "UNSURE", "confidence": 0.1}),+ ("approved-before.example", submitter, {"suggested_action": "REJECT",+ "suggested_reason": "SPAM"}),+ ("rejected-before.example", submitter, {}),+ # A check that already decided it, at least as strongly as the prior decision does.+ ("dead-and-approved-before.example", submitter,+ {"suggested_action": "REJECT", "suggested_reason": "OTHER", "confidence": 0.9,+ "reason_source": "rule", "evidence": [a_404]}),+ ("still-crawling.example", submitter, None),+ ]+ for domain, user, overrides in cases:+ DomainSubmission.objects.create(name=domain, submitted_by=user)+ if overrides is None:+ DomainEvidence.objects.create(domain=domain, state=DomainEvidence.State.PENDING)+ else:+ ready_evidence(domain, **overrides)++ for submission in DomainSubmission.objects.filter(status="PENDING"):+ evidence = DomainEvidence.objects.filter(domain=submission.name).first()+ shown = suggestion_for(submission, evidence)+ row = annotate_queue(DomainSubmission.objects.filter(pk=submission.pk)).get()++ if shown is None:+ assert row.displayed_action is None, submission.name+ assert row.displayed_priority is None, submission.name+ continue+ assert row.displayed_action == shown.action, submission.name+ assert row.displayed_reason == shown.reason, submission.name+ assert row.displayed_confidence == pytest.approx(shown.confidence), submission.name+ assert row.displayed_priority == pytest.approx(shown.review_priority), submission.name+++def test_queue_filters_find_a_withheld_approval_where_it_is_shown(client, moderator, submitter):+ """A moderator filtering for UNSURE is asking for the rows only a human can settle, which+ is exactly what a withheld approval is."""+ DomainSubmission.objects.create(name="withheld.example", submitted_by=submitter)+ ready_evidence("withheld.example", suggested_action="APPROVE", confidence=0.9)++ response = client.get(f"{QUEUE_URL}?suggested_action=UNSURE",+ headers={"Authorization": token(moderator)})+ items = response.json()["items"]++ assert [item["name"] for item in items] == ["withheld.example"]+ assert items[0]["suggestion"]["action"] == "UNSURE"+++def test_a_withheld_approval_outranks_a_confident_one(client, moderator, submitter,+ established):+ """It is the 44%-wrong slice: it needs a human more than an approval we trust does."""+ DomainSubmission.objects.create(name="withheld.example", submitted_by=submitter)+ ready_evidence("withheld.example", suggested_action="APPROVE", confidence=0.95)+ DomainSubmission.objects.create(name="trusted.example", submitted_by=established)+ ready_evidence("trusted.example", suggested_action="APPROVE", confidence=0.95)++ response = client.get(QUEUE_URL, headers={"Authorization": token(moderator)})+ assert [item["name"] for item in response.json()["items"]] == [+ "withheld.example", "trusted.example"]+++# --------------------------------------------------------------------- the artifact++class _StubModel:+ """A stand-in for ModerationModel. joblib only needs it to pickle and carry a version."""++ def __init__(self, version):+ self.version = version++ def predict(self, examples):+ return [(0.5, "SPAM", 0.5) for _ in examples]+++def artifact_bytes(model) -> bytes:+ buffer = io.BytesIO()+ joblib.dump(model, buffer)+ return buffer.getvalue()+++def test_a_published_model_outlives_the_process_that_trained_it():+ """It used to be written into the source tree, so a deploy reverted every retrain and each+ worker replica kept its own copy."""+ publish(_StubModel("domain-mod-2026-09-01"), {"cold_start": {"pr_auc": 0.81}})++ reset_model_cache() # a different worker, loading for the first time+ assert get_model().version == "domain-mod-2026-09-01"+ assert load_metrics()["cold_start"]["pr_auc"] == 0.81+++def test_a_worker_picks_up_a_retrain_it_did_not_run(monkeypatch):+ """The retrain runs in one process and every process serves suggestions."""+ publish(_StubModel("domain-mod-2026-09-01"), {})+ assert get_model().version == "domain-mod-2026-09-01"++ monkeypatch.setattr(model_module, "MODEL_REFRESH_SECONDS", 0)+ ModerationModelArtifact.objects.create(+ version="domain-mod-2026-10-01", metrics={},+ model=artifact_bytes(_StubModel("domain-mod-2026-10-01")))++ assert get_model().version == "domain-mod-2026-10-01"+++def test_the_bundled_artifact_is_the_warm_start_until_a_retrain_publishes(tmp_path, settings):+ """A deploy against a database with no artifact in it still suggests something."""+ joblib.dump(_StubModel("bundled"), tmp_path / "model.joblib")+ (tmp_path / "metrics.json").write_text(json.dumps({"cold_start": {"pr_auc": 0.7}}))+ settings.DOMAIN_MODERATION_MODEL_DIR = str(tmp_path)++ assert get_model().version == "bundled"+ assert load_metrics()["cold_start"]["pr_auc"] == 0.7++ publish(_StubModel("domain-mod-2026-09-01"), {"cold_start": {"pr_auc": 0.81}})+ # Once something is published the gate compares against that, not against the shipped+ # numbers, which describe a model nobody is serving any more.+ assert load_metrics()["cold_start"]["pr_auc"] == 0.81+++def test_an_unloadable_artifact_falls_back_rather_than_failing(tmp_path, settings):+ """A scikit-learn upgrade can make an old pickle unloadable, and the enrichment task must+ not go down with it."""+ joblib.dump(_StubModel("bundled"), tmp_path / "model.joblib")+ settings.DOMAIN_MODERATION_MODEL_DIR = str(tmp_path)+ ModerationModelArtifact.objects.create(+ version="corrupt", model=b"not a pickle", metrics={})++ assert get_model().version == "bundled"+++# --------------------------------------------------------------------- request validation++def test_an_over_long_audit_field_is_a_validation_error_not_a_database_one(+ client, moderator, submitter):+ """Django does not enforce max_length on save(), so without the schema constraint this is+ a Postgres DataError and a 500 rather than a 400 naming the field."""+ submission = DomainSubmission.objects.create(name="example.com", submitted_by=submitter)++ response = client.post(+ f"/api/v1/platform/domain-submissions/ids/{submission.id}",+ data={"status": "APPROVED", "rejection_reason": "", "rejection_detail": "",+ "suggestion_model_version": "v" * 100},+ content_type="application/json", headers={"Authorization": token(moderator)})++ assert response.status_code == 422+ submission.refresh_from_db()+ assert submission.status == "PENDING"+++def test_other_without_a_detail_is_refused(client, moderator, submitter):+ """The dialog labels the fourth reason "Other - needs detail", and the detail is the only+ place a rejected submitter finds out what was actually wrong."""+ DomainSubmission.objects.create(name="example.com", submitted_by=submitter)++ response = client.post(+ DECISIONS_URL,+ data={"decisions": [{"domain": "example.com", "status": "REJECTED",+ "rejection_reason": "OTHER", "rejection_detail": " "}]},+ content_type="application/json", headers={"Authorization": token(moderator)})++ assert response.status_code == 422+ assert DomainSubmission.objects.get(name="example.com").status == "PENDING"+++def test_a_rejection_reason_outside_the_four_choices_is_refused(client, moderator, submitter):+ """Django does not enforce choices on save(), so any string of twenty characters or fewer+ lands in the column and comes back out at a moderator as a reason nothing can render."""+ DomainSubmission.objects.create(name="example.com", submitted_by=submitter)++ response = client.post(+ DECISIONS_URL,+ data={"decisions": [{"domain": "example.com", "status": "REJECTED",+ "rejection_reason": "BECAUSE"}]},+ content_type="application/json", headers={"Authorization": token(moderator)})++ assert response.status_code == 422+++def test_an_approval_cannot_carry_a_rejection_reason(client, moderator, submitter):+ DomainSubmission.objects.create(name="example.com", submitted_by=submitter)++ response = client.post(+ DECISIONS_URL,+ data={"decisions": [{"domain": "example.com", "status": "APPROVED",+ "rejection_reason": "SPAM"}]},+ content_type="application/json", headers={"Authorization": token(moderator)})++ assert response.status_code == 422+++def test_other_with_a_detail_is_accepted(client, moderator, submitter):+ DomainSubmission.objects.create(name="example.com", submitted_by=submitter)++ response = client.post(+ DECISIONS_URL,+ data={"decisions": [{"domain": "example.com", "status": "REJECTED",+ "rejection_reason": "OTHER",+ "rejection_detail": "Mirrors content we already index."}]},+ content_type="application/json", headers={"Authorization": token(moderator)})++ assert response.status_code == 200+ assert DomainSubmission.objects.get(name="example.com").status == "REJECTED"diff --git a/test/test_retrieve_extract.py b/test/test_retrieve_extract.pyindex f0ad9d7..d075a67 100644--- a/test/test_retrieve_extract.py+++ b/test/test_retrieve_extract.py@@ -61,7 +61,7 @@ def test_get_first_paragraph_none_when_no_body_prose():def _crawl_html(monkeypatch, html: str) -> dict:monkeypatch.setattr(retrieve, "robots_allowed", lambda url, redis: True)- monkeypatch.setattr(retrieve, "fetch", lambda url: (200, html.encode("utf8")))+ monkeypatch.setattr(retrieve, "fetch", lambda url: (200, html.encode("utf8"), url))return retrieve.crawl_url("https://example.test/page", fakeredis.FakeStrictRedis())diff --git a/test/test_ssrf.py b/test/test_ssrf.pyindex d353db0..ac4b31e 100644--- a/test/test_ssrf.py+++ b/test/test_ssrf.py@@ -124,9 +124,25 @@ def test_fetch_blocks_redirect_to_internal(monkeypatch):def test_fetch_returns_content_for_public_url(monkeypatch):monkeypatch.setattr(retrieve.requests, "get",lambda *a, **k: _FakeResponse(200, body=b"page body"))- status, content = retrieve.fetch("http://93.184.216.34/page")+ status, content, resolved_url = retrieve.fetch("http://93.184.216.34/page")assert status == 200assert content == b"page body"+ assert resolved_url == "http://93.184.216.34/page"+++def test_fetch_reports_the_host_that_answered_not_the_one_we_asked(monkeypatch):+ """The last hop is the only place a redirect is visible to a caller. Moderation rejects+ domains that redirect elsewhere, and cannot see one if fetch reports the requested URL."""+ def fake_get(url, **kwargs):+ if url == "http://93.184.216.34/start":+ return _FakeResponse(301, headers={"Location": "http://93.184.216.35/parked"},+ is_redirect=True)+ return _FakeResponse(200, body=b"parking page")++ monkeypatch.setattr(retrieve.requests, "get", fake_get)+ status, _, resolved_url = retrieve.fetch("http://93.184.216.34/start")+ assert status == 200+ assert resolved_url == "http://93.184.216.35/parked"# ---------------------------------------------------------------------------