Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Moderate a domain, not a submission

Daoud Clarkeb7abe8fcfae4
diff --git a/docs/domain-moderation-gaps.md b/docs/domain-moderation-gaps.md
new file mode 100644
index 0000000..7ae2049
--- /dev/null
+++ b/docs/domain-moderation-gaps.md
@@ -0,0 +1,147 @@
+# Domain moderation: design vs. implementation
+
+Design: <https://claude.ai/design/p/bca5ef6e-47c5-44e1-a45b-9e91d3483c53?file=Domain+Moderation+Alternatives.dc.html>
+
+Turn **2a** ("Focus slot") is the built-out target. Turns 1a/1b/1c are the earlier alternatives it
+was built from; 1a's status filter is the only thing from them still wanted (gap 8).
+
+All nine gaps below are closed. What follows is what each one turned into, so the reasoning
+behind the shape of the API stays attached to the design it came from.
+
+## What the API returns now
+
+`GET /domain-submissions/queue` is one row per **domain**, carrying everything one focus card and
+its UP NEXT rows draw — so moving to the next card costs no request:
+
+```
+name, submission_count, first_submitted_on, first_submitted_by(+_username),
+upvotes, downvotes, https, evidence_state, pages[], suggestion
+```
+
+`count` is distinct pending domains — the header's "24 pending". Default ordering is
+`submissions` (`-submission_count, -upvotes, submitted_on`), which is the sort the screen labels;
+`needs_review`, `confidence` and `oldest` are still accepted.
+
+Decisions, undo and history are all addressed to a domain:
+
+- `POST /domain-submissions/decisions` — `{"decisions": [{"domain": …, "status": …}]}`; each entry
+ settles every submission of that name.
+- `POST /domain-submissions/domains/{domain}/undo` — back to PENDING, rejection fields cleared.
+- `GET /domain-submissions/moderated` — past moderations, one row per domain, filterable by
+ `status` / `moderator` / `name`.
+
+## Gap by gap
+
+### 1. The queue is per domain
+
+`one_row_per_domain` (`mwmbl/moderation/suggest.py`) keeps the **earliest** submission of each
+name and annotates `submission_count` onto it.
+
+Deliberately not a `GROUP BY`. The suggestion depends on the submitter's own track record, and a
+domain submitted by two accounts has no single one to read — grouping would have meant rewriting
+the withheld-approval rule and breaking the `suggestion_for` / `annotate_queue` parity that
+`test_queue_display_matches_suggestion_for` holds together. Collapsing to a representative row
+instead leaves both of them working on a submission. The earliest row is also the one the card
+draws ("first submitted 6 days ago by anon_4417"), so the row we deduplicate to and the row we
+show are the same row.
+
+### 2. `submissions` ordering
+
+`QUEUE_ORDERINGS["submissions"]` = `-submission_count, -upvotes, submitted_on`, and it is the
+default. How many people asked for a domain is the one signal in the sort that is not the tool's
+own opinion, which is why it leads rather than the suggestion's confidence.
+
+### 3. Vote counts aggregated to the domain
+
+`SearchResultVote` gained an indexed `domain` column, derived on `save()` by `utils.bare_host`
+(normalise, drop a leading `www.`), with a batched backfill in migration 0036. `annotate_votes`
+counts it in SQL against the submission name reduced the same way, so a vote on
+`www.example.com/x` counts towards a submission of `example.com`. Coalesced to zero, because a
+domain with no votes has to sort with the rest rather than scatter through it as NULL.
+
+A rollup table was the alternative and was rejected: the live aggregate is one indexed subquery
+and never stale.
+
+### 4. Sample pages on queue rows
+
+`pages` on `QueueItemSchema`, read off the `DomainEvidence` rows `_queue_items` already fetches
+for the suggestion. No extra query.
+
+### 5. Usernames
+
+`first_submitted_by_username` on the queue row and `submitted_by_username` on the detail, both
+via `select_related("submitted_by")` — a join, not a query. The numeric ids stay, for filtering.
+
+### 6. The https padlock
+
+`crawl_domain` now retries `http://` once when the https fetch **errors** (not when it answers
+with a non-2xx), and records `signals["https"]`. A 404 over https is an answer; asking the same
+server the same question without TLS gets the same one.
+
+This changes suggestions: a site serving only over plain HTTP used to be `unreachable` — a
+decisive REJECT at 0.9 — and is now reachable with a neutral `no_tls` evidence line. Neutral for
+the same reason `robots` is: plenty of the small personal sites this index exists for have no
+certificate, and no rejection detail a moderator has written mentions TLS.
+
+`https` is `Optional[bool]`: `null` until the domain is crawled, because an uncrawled domain must
+not draw an open padlock. Existing evidence rows report `null` until they go stale
+(`MODERATION_EVIDENCE_MAX_AGE_DAYS`) or `backfill_domain_evidence` is run.
+
+### 7. Undo
+
+`POST /domain-submissions/domains/{domain}/undo` restores PENDING and clears
+`rejection_reason` / `rejection_detail`, and **leaves `suggested_status`, `suggested_reason`,
+`suggestion_confidence` and `suggestion_model_version` alone** — the record of what was on screen
+when the decision being undone was made, which is the one thing those columns exist for and
+exactly what re-posting a PENDING status destroyed.
+
+`status_changed_by` / `status_changed_on` do move to the undoer: the field means "who last
+changed the status", and that is now them.
+
+`.update()` fires no `post_save`, so undoing an *approval* schedules a blacklist snapshot rebuild
+explicitly, with the same debounce as `mwmbl/signals.py`.
+
+The "Reviewed this session" tray is client-side session state.
+
+### 8. Viewing and changing past moderations
+
+`GET /domain-submissions/moderated`, moderator-only, paginated, one row per domain represented by
+its **most recently touched** submission — a history row shows the decision that currently
+stands. Rows carry the `suggested_*` audit columns, so a moderator revisiting a call sees what
+was on screen then rather than what the model would say today.
+
+Changing a past decision is a normal POST to `/decisions`: a decision applies to every submission
+of the name, decided or not, so re-deciding needs no separate endpoint and a domain can never sit
+half approved and half rejected.
+
+The trade-off: re-deciding overwrites `status_changed_*` and the `suggested_*` columns on the
+already-decided rows of that name. An append-only `ModerationDecision` log would keep both, and
+is the change to make if decision history is ever read.
+
+### 9. Rejection reasons are validated
+
+`RejectionFieldsMixin` in `mwmbl/platform/schemas.py`, shared by `DomainDecision` and
+`UpdateDomainSubmission` so the two cannot drift:
+
+- `rejection_reason` must be one of the four choices — Django does not enforce `choices` on
+ `save()`, so any string of ≤20 characters used to land in the column;
+- `rejection_detail` is required when the reason is `OTHER`;
+- `rejection_reason` is only accepted alongside `status=REJECTED`.
+
+A schema validator rather than a view check, so these are 422s naming the field.
+
+**No notification is sent.** The design's "The submitter is told the reason" line is being removed
+from the dialog instead; the reason is already visible on the submitter's own submission list.
+
+## Dropped
+
+- **A one-line "why" for the suggestion** (`suggested by the index — 9 submissions from 2
+ accounts, heavy downvotes, 4 crawled pages`). The API returns per-check `EvidenceItem` labels
+ instead, which a client can join. "from 2 accounts" — the count of distinct submitting accounts
+ — is still not computed anywhere; it would be one more correlated subquery if it comes back.
+
+## Known loose end
+
+`views.py`'s `DomainSubmissionApprovalForm` decides submissions without going through
+`apply_decision`: it sets neither `status_changed_by` nor the `suggested_*` audit columns. Not one
+of these nine gaps, but a second decision path that quietly writes worse records than the API's.
diff --git a/mwmbl/background.py b/mwmbl/background.py
index d7b8dfc..eaf8687 100644
--- a/mwmbl/background.py
+++ b/mwmbl/background.py
@@ -18,6 +18,7 @@ from background_task import background
from django.conf import settings
from django.core.cache import cache
from django.core.management import call_command
+from django.db import transaction
from redis import Redis
from mwmbl import pricing
@@ -28,7 +29,7 @@ from mwmbl.indexer.blacklist_snapshot import get_snapshot_blacklist, refresh_sna
from mwmbl.indexer.purge_blacklisted import purge_documents
from mwmbl.indexer.purge_queue import drain_purge_queue, queue_size
from mwmbl.models import DomainEvidence, DomainSubmission, OldIndex, UsageBucket
-from mwmbl.moderation.evidence import crawl_domain, store_evidence
+from mwmbl.moderation.evidence import crawl_domain, store_evidence, store_failure
from mwmbl.moderation.model import reset_model_cache
from mwmbl.moderation.suggest import refresh_suggestion
from mwmbl.quota import MONTHLY_TTL, _monthly_key, get_all_monthly_keys
@@ -262,14 +263,21 @@ def enrich_domain_submission(domain: str):
redis = Redis.from_url(settings.REDIS_URL)
try:
- evidence = store_evidence(domain, crawl_domain(domain, redis))
- except Exception:
+ crawl = crawl_domain(domain, redis)
+ except Exception as exception:
logger.exception("Failed to crawl %s for moderation", domain)
- DomainEvidence.objects.update_or_create(
- domain=domain, defaults={"state": DomainEvidence.State.FAILED})
+ store_failure(domain, type(exception).__name__)
return
- refresh_suggestion(evidence)
+ # One transaction, because the row is only useful once both halves are in it. Committing
+ # READY with no suggestion on it - a deploy restarting the worker in between would do it -
+ # leaves a row the freshness check above then skips for a month, so the queue would show
+ # "not assessed yet" for a domain that has in fact been crawled. Scoring failures are left
+ # to propagate: the transaction rolls back and the task runner retries, which is a better
+ # answer than recording a crawl that did happen as a crawl failure.
+ with transaction.atomic():
+ evidence = store_evidence(domain, crawl)
+ refresh_suggestion(evidence)
logger.info("Enriched %s: %s (%.2f)", domain, evidence.suggested_action,
evidence.confidence or 0.0)
diff --git a/mwmbl/crawler/retrieve.py b/mwmbl/crawler/retrieve.py
index 553c15e..7083d17 100644
--- a/mwmbl/crawler/retrieve.py
+++ b/mwmbl/crawler/retrieve.py
@@ -60,6 +60,10 @@ def fetch(url):
Redirects are followed manually so each hop can be re-validated against the
SSRF guard: a public URL must not be able to 3xx us into an internal address.
+ Returns (status code, content, resolved URL). The resolved URL is the last hop, which is
+ the only place the redirect chain is visible to a caller: it says which host actually
+ served the page, rather than which one we asked.
+
https://stackoverflow.com/a/22347526
"""
@@ -89,7 +93,7 @@ def fetch(url):
logger.debug(f"Maximum size reached for URL {url}")
break
- return r.status_code, content
+ return r.status_code, content, url
raise ValueError(f"Too many redirects for URL {url}")
@@ -114,7 +118,7 @@ def robots_allowed(url: str, redis: Redis) -> bool:
return allowed
try:
- status_code, content = fetch(robots_url)
+ status_code, content, _ = fetch(robots_url)
except ALLOWED_EXCEPTIONS as e:
logger.debug(f"Robots error: {robots_url}, {e}")
_cache_robots_content(redis, domain, [], error=True)
@@ -320,7 +324,7 @@ def crawl_url(url, redis: Redis):
}
try:
- status_code, content = fetch(url)
+ status_code, content, resolved_url = fetch(url)
except ALLOWED_EXCEPTIONS as e:
logger.debug(f"Exception crawling URl {url}: {e}")
return {
@@ -337,6 +341,7 @@ def crawl_url(url, redis: Redis):
if len(content) == 0:
return {
'url': url,
+ 'resolved_url': resolved_url,
'status': status_code,
'timestamp': js_timestamp,
'content': None,
@@ -354,6 +359,7 @@ def crawl_url(url, redis: Redis):
logger.exception(f"Error parsing dom: {url}")
return {
'url': url,
+ 'resolved_url': resolved_url,
'status': status_code,
'timestamp': js_timestamp,
'content': None,
@@ -380,6 +386,7 @@ def crawl_url(url, redis: Redis):
logger.exception("Error parsing paragraphs - offending control bytes: %s", bad_bytes)
return {
'url': url,
+ 'resolved_url': resolved_url,
'status': status_code,
'timestamp': js_timestamp,
'content': None,
@@ -424,6 +431,7 @@ def crawl_url(url, redis: Redis):
return {
'url': url,
+ 'resolved_url': resolved_url,
'status': status_code,
'timestamp': js_timestamp,
'content': {
diff --git a/mwmbl/management/commands/backfill_domain_evidence.py b/mwmbl/management/commands/backfill_domain_evidence.py
index ef18823..028c0a8 100644
--- a/mwmbl/management/commands/backfill_domain_evidence.py
+++ b/mwmbl/management/commands/backfill_domain_evidence.py
@@ -20,10 +20,12 @@ import time
from django.core.management.base import BaseCommand
from django.utils.dateparse import parse_date
+from django.db import transaction
+
from mwmbl.crawler.env_vars import CRAWL_DELAY_SECONDS
from mwmbl.indexer.blacklist_snapshot import get_redis
from mwmbl.models import DomainEvidence, DomainSubmission
-from mwmbl.moderation.evidence import crawl_domain, store_evidence
+from mwmbl.moderation.evidence import crawl_domain, store_evidence, store_failure
from mwmbl.moderation.suggest import refresh_suggestion
from mwmbl.moderation.training_data import is_trainable_domain
@@ -46,17 +48,17 @@ class Command(BaseCommand):
redis = get_redis()
for number, domain in enumerate(domains, start=1):
try:
- evidence = store_evidence(domain, crawl_domain(domain, redis))
- refresh_suggestion(evidence)
+ # Both halves in one transaction, so an interrupted run never leaves a READY
+ # row with no suggestion on it - which the freshness check would then skip.
+ with transaction.atomic():
+ evidence = store_evidence(domain, crawl_domain(domain, redis))
+ refresh_suggestion(evidence)
self.stdout.write(
f"[{number}/{len(domains)}] {domain}: {evidence.suggested_action or '?'} "
f"({evidence.state})")
except Exception as exception:
# One unfetchable domain must not end a backfill of thousands.
- DomainEvidence.objects.update_or_create(
- domain=domain,
- defaults={"state": DomainEvidence.State.FAILED,
- "error": type(exception).__name__})
+ store_failure(domain, type(exception).__name__)
self.stderr.write(f"[{number}/{len(domains)}] {domain}: FAILED ({exception})")
if options["delay"]:
time.sleep(options["delay"])
diff --git a/mwmbl/management/commands/train_domain_moderation_model.py b/mwmbl/management/commands/train_domain_moderation_model.py
index 8fcd0b8..92a2c94 100644
--- a/mwmbl/management/commands/train_domain_moderation_model.py
+++ b/mwmbl/management/commands/train_domain_moderation_model.py
@@ -5,19 +5,20 @@
The gate compares the new model's cold-start PR-AUC against the lower bound of the incumbent's
bootstrap interval, so a change that only looks like an improvement does not ship.
+
+Publishing writes a ModerationModelArtifact row rather than a file: the incumbent the gate
+reads and the model the workers serve are then the same object, and neither is lost to a
+deploy. See mwmbl.moderation.model.
"""
import json
from datetime import date
-from pathlib import Path
-from django.conf import settings
from django.core.management.base import BaseCommand
from mwmbl.background import rescore_pending_submissions
from mwmbl.models import DomainSubmission
from mwmbl.moderation import training_data
-from mwmbl.moderation.model import (
- METRICS_FILENAME, MODEL_FILENAME, load_metrics, reset_model_cache)
+from mwmbl.moderation.model import load_metrics, publish
from mwmbl.moderation.train import passes_gate, train
# A reason class with fewer real examples than this is a candidate for blocklist-derived
@@ -36,7 +37,6 @@ class Command(BaseCommand):
help="Train and report metrics without writing the artifact")
parser.add_argument("--force", action="store_true",
help="Publish even if the gate fails")
- parser.add_argument("--output-dir", default=settings.DOMAIN_MODERATION_MODEL_DIR)
parser.add_argument(
"--derived", action="store_true",
help=("Add blocklist-derived rows for reason classes short of real data. Off by "
@@ -74,7 +74,11 @@ class Command(BaseCommand):
self.stderr.write("Gate failed; not publishing. Re-run with --force to override.")
return
- self._publish(model, metrics, Path(options["output_dir"]))
+ publish(model, metrics)
+ self.stdout.write(self.style.SUCCESS(
+ f"Published {model.version}; every worker picks it up within a minute."))
+ rescore_pending_submissions(schedule=0)
+ self.stdout.write("Scheduled a rescore of the pending queue with the new model.")
@staticmethod
def _reasons_needing_data(rows) -> set[str]:
@@ -84,16 +88,3 @@ class Command(BaseCommand):
counts[row.reason] = counts.get(row.reason, 0) + 1
return {reason for reason in DomainSubmission.DOMAIN_REJECTION_REASON
if counts.get(reason, 0) < MIN_REAL_ROWS_PER_REASON}
-
- def _publish(self, model, metrics, output_dir: Path):
- import joblib
-
- output_dir.mkdir(parents=True, exist_ok=True)
- joblib.dump(model, output_dir / MODEL_FILENAME)
- (output_dir / METRICS_FILENAME).write_text(json.dumps(metrics, indent=2))
- self.stdout.write(self.style.SUCCESS(
- f"Wrote {model.version} to {output_dir / MODEL_FILENAME}"))
-
- reset_model_cache()
- rescore_pending_submissions(schedule=0)
- self.stdout.write("Scheduled a rescore of the pending queue with the new model.")
diff --git a/mwmbl/migrations/0035_moderationmodelartifact_and_more.py b/mwmbl/migrations/0035_moderationmodelartifact_and_more.py
new file mode 100644
index 0000000..837669b
--- /dev/null
+++ b/mwmbl/migrations/0035_moderationmodelartifact_and_more.py
@@ -0,0 +1,57 @@
+# Generated by Django 5.2.13 on 2026-08-28 18:12
+
+import django.utils.timezone
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('mwmbl', '0034_domainsubmission_suggested_reason_and_more'),
+ ]
+
+ operations = [
+ migrations.CreateModel(
+ name='ModerationModelArtifact',
+ fields=[
+ ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('version', models.CharField(max_length=50, unique=True)),
+ ('model', models.BinaryField()),
+ ('metrics', models.JSONField(default=dict)),
+ ('created_on', models.DateTimeField(default=django.utils.timezone.now)),
+ ],
+ options={
+ 'get_latest_by': 'created_on',
+ },
+ ),
+ migrations.RemoveIndex(
+ model_name='domainevidence',
+ name='mwmbl_domai_state_1c3406_idx',
+ ),
+ migrations.RemoveField(
+ model_name='domainevidence',
+ name='review_priority',
+ ),
+ migrations.AddField(
+ model_name='domainevidence',
+ name='reason_confidence',
+ field=models.FloatField(blank=True, null=True),
+ ),
+ migrations.AddField(
+ model_name='domainevidence',
+ name='reason_source',
+ field=models.CharField(blank=True, max_length=10),
+ ),
+ migrations.AddIndex(
+ model_name='domainsubmission',
+ index=models.Index(fields=['name', 'status'], name='mwmbl_domai_name_0a9d6d_idx'),
+ ),
+ migrations.AddIndex(
+ model_name='domainsubmission',
+ index=models.Index(fields=['submitted_by', 'status'], name='mwmbl_domai_submitt_2d8fb1_idx'),
+ ),
+ migrations.AddIndex(
+ model_name='moderationmodelartifact',
+ index=models.Index(fields=['-created_on'], name='mwmbl_moder_created_1c74be_idx'),
+ ),
+ ]
diff --git a/mwmbl/migrations/0036_search_result_vote_domain.py b/mwmbl/migrations/0036_search_result_vote_domain.py
new file mode 100644
index 0000000..bfc8983
--- /dev/null
+++ b/mwmbl/migrations/0036_search_result_vote_domain.py
@@ -0,0 +1,57 @@
+import re
+
+from django.db import migrations, models
+
+# Frozen copies of mwmbl.utils.parse_url/normalize_domain/bare_host: a migration has to keep
+# producing the same data forever, so it must not depend on application code that is free to
+# change. Matches migration 0032, which normalised the domain submission names.
+URL_REGEX = re.compile("^(([^:/?#]+):)?(//([^/?#]*)|///)?([^?#]*)(\\?[^#]*)?(#.*)?")
+DOMAIN_END_REGEX = re.compile(r"[/?#]")
+
+BATCH_SIZE = 1000
+
+
+def bare_host(domain_or_url: str) -> str:
+ netloc = URL_REGEX.match(domain_or_url).group(4)
+ if not netloc:
+ netloc = DOMAIN_END_REGEX.split(domain_or_url, maxsplit=1)[0]
+ host = netloc.lower()
+ return host[4:] if host.startswith("www.") else host
+
+
+def backfill_domains(apps, schema_editor):
+ """Derive the host for every existing vote, in batches.
+
+ Written before the index is created, so the whole table is scanned once rather than
+ maintaining an index through the rewrite.
+ """
+ SearchResultVote = apps.get_model("mwmbl", "SearchResultVote")
+ batch = []
+ for vote in SearchResultVote.objects.all().only("id", "url").iterator(chunk_size=BATCH_SIZE):
+ vote.domain = bare_host(vote.url)
+ batch.append(vote)
+ if len(batch) >= BATCH_SIZE:
+ SearchResultVote.objects.bulk_update(batch, ["domain"])
+ batch = []
+ if batch:
+ SearchResultVote.objects.bulk_update(batch, ["domain"])
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('mwmbl', '0035_moderationmodelartifact_and_more'),
+ ]
+
+ operations = [
+ migrations.AddField(
+ model_name='searchresultvote',
+ name='domain',
+ field=models.CharField(blank=True, max_length=300),
+ ),
+ migrations.RunPython(backfill_domains, migrations.RunPython.noop),
+ migrations.AddIndex(
+ model_name='searchresultvote',
+ index=models.Index(fields=['domain', 'vote_type'], name='mwmbl_searc_domain_7f4ebd_idx'),
+ ),
+ ]
diff --git a/mwmbl/models.py b/mwmbl/models.py
index be37e9b..54bfe32 100644
--- a/mwmbl/models.py
+++ b/mwmbl/models.py
@@ -9,6 +9,7 @@ from ninja import ModelSchema
from ninja.orm import create_schema
from mwmbl.usernames import generate_username
+from mwmbl.utils import bare_host
class MwmblUser(AbstractUser):
@@ -82,6 +83,11 @@ class DomainSubmission(models.Model):
]
indexes = [
models.Index(fields=['submitted_on']),
+ # The moderation queue asks "has this domain been decided before?" and "has this
+ # submitter?" once per row it renders, as correlated subqueries. Both are lookups
+ # into this table by a decided status. See mwmbl.moderation.suggest.
+ models.Index(fields=['name', 'status']),
+ models.Index(fields=['submitted_by', 'status']),
]
DOMAIN_SUBMISSION_STATUS = {
@@ -109,8 +115,10 @@ class DomainSubmission(models.Model):
# What the suggestion model was showing when the moderator made this decision. Deliberately
# separate storage from the live suggestion on DomainEvidence: a retrain rewrites that row,
# and sharing the columns would silently rewrite the record of what was actually on screen.
- # These are what let us measure whether the suggestions help, and de-bias the next retrain
- # against decisions that merely confirmed a suggestion.
+ # These are what make the suggestions' influence measurable: a retrain trains on decisions
+ # that were themselves made with a suggestion on screen, and without a record of what was
+ # shown there is no way to see that happening. The retrain does not yet weight rows by it -
+ # see mwmbl.moderation.training_data - it reports it.
suggested_status = models.CharField(max_length=20, blank=True)
suggested_reason = models.CharField(max_length=20, blank=True)
suggestion_confidence = models.FloatField(null=True, blank=True)
@@ -248,12 +256,23 @@ class SearchResultVote(models.Model):
query = models.CharField(max_length=300) # The search query context
vote_type = models.CharField(max_length=10, choices=[(k, v) for k, v in VOTE_TYPES.items()])
timestamp = models.DateTimeField(auto_now_add=True)
-
+ # The host the URL belongs to, derived on save. Denormalised because the moderation queue
+ # rolls votes up per domain, and it orders and paginates thousands of pending domains on
+ # the result - which has to happen in the database, and cannot parse a URLField there.
+ domain = models.CharField(max_length=300, blank=True)
+
+ def save(self, *args, **kwargs):
+ self.domain = bare_host(self.url)
+ super().save(*args, **kwargs)
+
class Meta:
unique_together = ['user', 'url', 'query'] # One vote per user per result per query
indexes = [
models.Index(fields=['url', 'query']),
models.Index(fields=['timestamp']),
+ # The per-domain rollup the moderation queue reads: count this domain's upvotes,
+ # count its downvotes. See mwmbl.moderation.suggest.annotate_votes.
+ models.Index(fields=['domain', 'vote_type']),
]
@@ -309,6 +328,12 @@ class DomainEvidence(models.Model):
Keyed on the domain rather than the submission because domains get resubmitted - 615 of
6,949 distinct names in the last judgments export had more than one submission - and the
crawl is about the domain, not about who asked for it.
+
+ What the queue orders on is *not* stored here. The suggestion a moderator sees depends on
+ rows this one knows nothing about - the submitter's record, and whether the domain has
+ since been decided - so the sort key is computed per query in SQL by
+ mwmbl.moderation.suggest.annotate_queue. Storing one would only ever be the answer to a
+ question nobody asked in that form.
"""
class State(models.TextChoices):
@@ -331,18 +356,52 @@ class DomainEvidence(models.Model):
suggested_action = models.CharField(max_length=10, blank=True) # APPROVE | REJECT | UNSURE
suggested_reason = models.CharField(max_length=20, blank=True)
confidence = models.FloatField(null=True, blank=True)
- # One number combining direction and confidence so "confident rejects and genuinely
- # uncertain rows first, obvious approvals last" is a single indexed ORDER BY. Sorting on
- # confidence alone cannot express that, because it is highest at both ends.
- review_priority = models.FloatField(null=True, blank=True)
+ # How sure we are of the *reason*, and where it came from - both separate from the numbers
+ # above, because a confident rejection can carry a barely-held guess at why. Storing them
+ # is what lets the API show the reason with its own confidence instead of the rejection's,
+ # and keeps `derived` (a reason class learned from blocklists rather than from moderator
+ # decisions) visible to the moderator it is a caveat for.
+ reason_confidence = models.FloatField(null=True, blank=True)
+ reason_source = models.CharField(max_length=10, blank=True) # rule | model | derived
evidence = models.JSONField(default=list) # cached rule + model evidence items
model_version = models.CharField(max_length=50, blank=True)
class Meta:
indexes = [
- models.Index(fields=["state", "-review_priority"]),
models.Index(fields=["suggested_action", "suggested_reason"]),
]
def __str__(self):
return f"{self.domain} ({self.state})"
+
+
+class ModerationModelArtifact(models.Model):
+ """A trained moderation suggester, stored where a deploy cannot overwrite it.
+
+ The artifact used to live in the source tree, which meant every deploy silently reverted
+ the monthly retrain to whatever was committed, and each worker replica kept its own
+ divergent copy. The database is the one place all the workers already agree on.
+
+ Rows accumulate: the newest is served, and the ones behind it are the record of what was
+ serving when a given decision was made. Versions are dated, so a second retrain on the
+ same day replaces its own row rather than adding one. ``metrics`` travels in the same row
+ as the model it describes, so the retrain gate cannot compare a candidate against numbers
+ that belong to a different artifact. At a monthly retrain that is ~12 MB a year.
+
+ The model bundled in mwmbl/moderation/artifacts is the warm start for a database with no
+ rows yet; nothing ever writes to it. See mwmbl.moderation.model.
+ """
+
+ version = models.CharField(max_length=50, unique=True)
+ model = models.BinaryField() # joblib pickle of a ModerationModel, ~1 MB
+ metrics = models.JSONField(default=dict) # what train.evaluate measured for this model
+ created_on = models.DateTimeField(default=timezone.now)
+
+ class Meta:
+ get_latest_by = "created_on"
+ indexes = [
+ models.Index(fields=["-created_on"]),
+ ]
+
+ def __str__(self):
+ return f"{self.version} ({self.created_on:%Y-%m-%d})"
diff --git a/mwmbl/moderation/artifacts/README.md b/mwmbl/moderation/artifacts/README.md
index 9530121..22df4fc 100644
--- a/mwmbl/moderation/artifacts/README.md
+++ b/mwmbl/moderation/artifacts/README.md
@@ -6,15 +6,22 @@ held-out real decisions when it was built. Both are committed deliberately, for
deterministic checks alone, so shipping one means a fresh deploy is useful on day one instead of
after the first retrain.
+**This directory is the warm start, and nothing writes to it.** A retrain publishes a
+`ModerationModelArtifact` row in Postgres, and that row is what every worker serves and what the
+gate compares against; these files are read only when the table is empty. Writing the artifact
+here would mean each deploy reverting the retrain to whatever was last committed, and each
+worker replica keeping its own copy.
+
## Regenerating
uv run manage.py train_domain_moderation_model --dry-run # report metrics only
uv run manage.py train_domain_moderation_model # publish if it passes the gate
The gate compares the new model's cold-start PR-AUC against the *lower bound* of the incumbent's
-bootstrap interval in `metrics.json`, so a change that only looks like an improvement does not
-ship. Publishing also schedules `rescore_pending_submissions`, which re-scores the existing queue
-with the new model.
+bootstrap interval — the metrics stored beside the published model, falling back to `metrics.json`
+until something has been published — so a change that only looks like an improvement does not
+ship. Publishing writes a new artifact row, which every worker picks up within a minute, and
+schedules `rescore_pending_submissions` to re-score the existing queue with the new model.
To reproduce the numbers without a production database, train from the sanitized judgments export
instead:
@@ -35,8 +42,10 @@ retrain to change the artifact substantially.
`LogisticRegression` heads, and ONNX export of a TF-IDF pipeline is far more trouble than it is
worth at this size. Two consequences worth knowing:
-- Unpickling executes code, so this file is trusted exactly as much as the rest of the repo. Do
- not point `DOMAIN_MODERATION_MODEL_DIR` at a directory anyone else can write to.
+- Unpickling executes code, so this file — and the artifact rows, which anything that can write
+ to the database could replace — is trusted exactly as much as the rest of the repo. Do not
+ point `DOMAIN_MODERATION_MODEL_DIR` at a directory anyone else can write to.
- A scikit-learn upgrade can make an old pickle unloadable. `mwmbl.moderation.model` catches that
- and logs it rather than failing, falling back to the deterministic checks — so the symptom is
- "every suggestion says UNSURE", and the fix is a retrain.
+ and logs it, falling back to this committed artifact and then to the deterministic checks — so
+ the symptom is suggestions reverting to an older model version, or saying UNSURE, and the fix
+ is a retrain.
diff --git a/mwmbl/moderation/evidence.py b/mwmbl/moderation/evidence.py
index 4ea0e07..9cab277 100644
--- a/mwmbl/moderation/evidence.py
+++ b/mwmbl/moderation/evidence.py
@@ -6,6 +6,10 @@ detection") are judgements about a site, not a page. The same three pages are sh
moderator's detail panel and fed to the model, so what a moderator sees and what the model
judged can never disagree.
+The homepage is fetched over https, falling back to http once if that errors, so that a
+site without a certificate is reported as a site without a certificate rather than as a site
+that is down.
+
This reuses mwmbl.crawler.retrieve.crawl_url, which already does robots.txt with Redis
caching, SSRF validation, a 3 second timeout, a 1 MB cap and justext extraction with Open
Graph fallbacks. Super Search already calls it from inside the server process, so nothing
@@ -50,7 +54,7 @@ def crawl_domain(domain: str, redis, max_pages: int = None) -> dict:
if max_pages is None:
max_pages = settings.MODERATION_PAGES_PER_DOMAIN
- homepage = crawl_url(f"https://{domain}/", redis)
+ homepage, https = _fetch_homepage(domain, redis)
pages = [_page(homepage)]
error = (homepage.get("error") or {}).get("name", "") if homepage.get("error") else ""
@@ -64,10 +68,35 @@ def crawl_domain(domain: str, redis, max_pages: int = None) -> dict:
"final_domain": _final_domain(homepage, domain),
"error": error,
"pages": pages,
- "signals": _signals(domain, pages),
+ "signals": _signals(domain, pages, https),
}
+def _fetch_homepage(domain: str, redis) -> tuple[dict, bool]:
+ """The homepage, and whether it came back over TLS.
+
+ Only https used to be tried, which made "this site has no certificate" and "this site is
+ down" the same result - a decisive REJECT for being unreachable. They are not the same
+ thing: a site served over plain http is a site, and whether it has TLS is something to
+ show a moderator rather than to decide for them. So a failed https fetch is retried once
+ over http, and which one answered is recorded.
+
+ Only a fetch *error* is retried. A 404 over https is an answer, and asking the same
+ server the same question again without TLS will get the same one.
+ """
+ over_https = crawl_url(f"https://{domain}/", redis)
+ if not over_https.get("error"):
+ return over_https, True
+
+ over_http = crawl_url(f"http://{domain}/", redis)
+ if over_http.get("error"):
+ # Neither worked. Report the https attempt: it is the one whose error describes what
+ # we would do in production, and reporting the http failure would say "no TLS" about
+ # a domain that simply is not there.
+ return over_https, False
+ return over_http, False
+
+
def store_evidence(domain: str, crawl: dict) -> DomainEvidence:
"""Persist a crawl and the checks derived from it, ready for the queue to read."""
evidence, _ = DomainEvidence.objects.update_or_create(
@@ -86,6 +115,36 @@ def store_evidence(domain: str, crawl: dict) -> DomainEvidence:
return evidence
+def store_failure(domain: str, error: str) -> DomainEvidence:
+ """Record that a domain could not be crawled, clearing what an earlier crawl left behind.
+
+ Everything the crawl produced goes with it, rather than only the state. The detail panel
+ renders whatever is on the row, so a month-old page list and its suggestion sitting next
+ to a FAILED state reads to a moderator as current evidence about a site that is now
+ unreachable - which is precisely the fact the state is trying to report.
+ """
+ evidence, _ = DomainEvidence.objects.update_or_create(
+ domain=domain,
+ defaults={
+ "state": DomainEvidence.State.FAILED,
+ "fetched_at": timezone.now(), # when we last tried, not when we last succeeded
+ "error": error,
+ "http_status": None,
+ "final_domain": "",
+ "pages": [],
+ "signals": {},
+ "evidence": [],
+ "suggested_action": "",
+ "suggested_reason": "",
+ "confidence": None,
+ "reason_confidence": None,
+ "reason_source": "",
+ "model_version": "",
+ },
+ )
+ return evidence
+
+
def _page(result: dict) -> dict:
content = result.get("content") or {}
error = result.get("error") or {}
@@ -125,12 +184,17 @@ def _follow_up_urls(domain: str, homepage: dict, limit: int) -> list[str]:
def _final_domain(homepage: dict, domain: str) -> str:
- """The host we ended up on. crawl_url follows redirects and reports the URL it fetched."""
- final = urlparse(homepage.get("url") or "").netloc
+ """The host we ended up on, empty when that is the domain we asked for.
+
+ ``resolved_url`` is the last hop of the redirect chain, which crawl_url reports separately
+ from ``url`` - the URL we asked for, and the one the rest of the crawler keys on. Reading
+ ``url`` here would compare the domain against itself and silently never find a redirect.
+ """
+ final = urlparse(homepage.get("resolved_url") or "").netloc
return final if final and final != domain else ""
-def _signals(domain: str, pages: list[dict]) -> dict:
+def _signals(domain: str, pages: list[dict], https: bool) -> dict:
text = " ".join(f"{page['title']} {page['extract']}" for page in pages).lower()
try:
blacklisted = get_snapshot_blacklist().is_domain_blacklisted(domain)
@@ -146,6 +210,7 @@ def _signals(domain: str, pages: list[dict]) -> dict:
"ad_markers": sum(marker in text for marker in AD_MARKERS),
"paywall_markers": sum(marker in text for marker in PAYWALL_MARKERS),
"blacklisted": blacklisted,
+ "https": https,
}
diff --git a/mwmbl/moderation/model.py b/mwmbl/moderation/model.py
index 199574b..4243c64 100644
--- a/mwmbl/moderation/model.py
+++ b/mwmbl/moderation/model.py
@@ -9,22 +9,31 @@ DomainEvidence, so the moderation queue is a plain indexed database read and no
ever loads this module. A missing or slow artifact therefore cannot touch a moderator's
request; the worst case is a queue row that honestly says it has not been assessed yet.
+The fitted artifact lives in Postgres (mwmbl.models.ModerationModelArtifact), not on disk. It
+is written monthly by a retrain and read by every worker, and a container filesystem is
+neither shared between workers nor kept across a deploy. The copy in ``artifacts/`` is the
+warm start for a database that has none yet, and nothing writes to it.
+
Loading mirrors mwmbl.tinysearchengine.super_search_select.judge: a lazy singleton behind a
-lock, memoized failure, and graceful degradation to rules-only when the artifact is absent so
-a deploy without it keeps working.
+lock, memoized failure, and graceful degradation to rules-only when no artifact can be loaded
+so a deploy without one keeps working.
"""
from __future__ import annotations
+import io
import json
import threading
+import time
from dataclasses import dataclass, field
from logging import getLogger
from pathlib import Path
from typing import Optional
+import joblib
import numpy as np
from django.conf import settings
+from mwmbl.models import ModerationModelArtifact
from mwmbl.moderation.features import Featuriser, ModerationExample
from mwmbl.moderation.rules import APPROVE, EvidenceItem, REJECT, decisive
@@ -33,6 +42,12 @@ logger = getLogger(__name__)
MODEL_FILENAME = "model.joblib"
METRICS_FILENAME = "metrics.json"
+# How long a loaded model is served before the database is asked whether a retrain has
+# published a newer one. A retrain happens in one worker process and every worker serves
+# suggestions, so without this they would disagree until the next restart. The check is a
+# single indexed row read, and a minute of staleness after a monthly retrain costs nothing.
+MODEL_REFRESH_SECONDS = 60
+
# A reason class with no real decisions behind it must not present as though it had. The
# reason head learns whatever classes the training data contains (reason_head.classes_ is the
# authority); this caps the ones that are not grounded in moderator decisions.
@@ -152,40 +167,83 @@ def suggest(domain: str, page_texts: list[str], evidence_items: list[EvidenceIte
_model: Optional[ModerationModel] = None
-_load_attempted = False
+_loaded_version: Optional[str] = None
+_checked_at: Optional[float] = None
_lock = threading.Lock()
def get_model() -> Optional[ModerationModel]:
- """Lazily load the shared model; None (memoized) if the artifact is unavailable."""
- global _model, _load_attempted
- if _load_attempted:
- return _model
+ """The published model, reloaded when a retrain has published a newer one.
+
+ None (until the published version changes) when nothing can be loaded, so a suggestion
+ falls back to the deterministic checks rather than failing.
+ """
+ global _model, _loaded_version, _checked_at
with _lock:
- if _load_attempted:
+ if _checked_at is not None and time.monotonic() - _checked_at < MODEL_REFRESH_SECONDS:
return _model
- _model = _load(Path(settings.DOMAIN_MODERATION_MODEL_DIR))
- _load_attempted = True
+ published = _published_version()
+ if _checked_at is None or published != _loaded_version:
+ _model = _load(published)
+ _loaded_version = published
+ _checked_at = time.monotonic()
return _model
+def publish(model: ModerationModel, metrics: dict) -> ModerationModelArtifact:
+ """Store a trained model and its metrics as the artifact every worker will serve."""
+ buffer = io.BytesIO()
+ joblib.dump(model, buffer)
+ # Versions are dated, so a second retrain on the same day replaces the first rather than
+ # colliding on the unique version.
+ artifact, _ = ModerationModelArtifact.objects.update_or_create(
+ version=model.version,
+ defaults={"model": buffer.getvalue(), "metrics": metrics})
+ reset_model_cache()
+ return artifact
+
+
def reset_model_cache() -> None:
"""Drop the cached model so the next call reloads. Used after a retrain, and by tests."""
- global _model, _load_attempted
+ global _model, _loaded_version, _checked_at
with _lock:
_model = None
- _load_attempted = False
+ _loaded_version = None
+ _checked_at = None
+
+
+def _published_version() -> Optional[str]:
+ """The version of the newest stored artifact, or None to fall back to the bundled one."""
+ row = ModerationModelArtifact.objects.order_by("-created_on").values("version").first()
+ return row["version"] if row else None
-def _load(model_dir: Path) -> Optional[ModerationModel]:
- model_path = model_dir / MODEL_FILENAME
+def _load(version: Optional[str]) -> Optional[ModerationModel]:
+ if version is None:
+ return _load_bundled()
+ try:
+ artifact = ModerationModelArtifact.objects.get(version=version)
+ model = joblib.load(io.BytesIO(bytes(artifact.model)))
+ except Exception:
+ # An artifact pickled by a different scikit-learn version is the realistic case, and
+ # it must not take the enrichment task down with it: fall back to the model shipped
+ # with this code, which was pickled by the dependencies this code is running.
+ logger.exception("Failed to load moderation model %s; falling back to the bundled "
+ "artifact", version)
+ return _load_bundled()
+ logger.info("Domain moderation model %s loaded from the database", model.version)
+ return model
+
+
+def _load_bundled() -> Optional[ModerationModel]:
+ """The artifact shipped in the source tree, for a database with no retrain in it yet."""
+ model_path = Path(settings.DOMAIN_MODERATION_MODEL_DIR) / MODEL_FILENAME
if not model_path.exists():
logger.warning(
- "Domain moderation model not found at %s; suggestions will use the deterministic "
- "checks only", model_path)
+ "Domain moderation model not found at %s and none published; suggestions will "
+ "use the deterministic checks only", model_path)
return None
try:
- import joblib
model = joblib.load(model_path)
except Exception:
logger.exception("Failed to load the domain moderation model from %s", model_path)
@@ -194,9 +252,15 @@ def _load(model_dir: Path) -> Optional[ModerationModel]:
return model
-def load_metrics(model_dir: Optional[Path] = None) -> dict:
- """The metrics written alongside the artifact, so the retrain gate can compare against it."""
- path = Path(model_dir or settings.DOMAIN_MODERATION_MODEL_DIR) / METRICS_FILENAME
- if not path.exists():
- return {}
- return json.loads(path.read_text())
+def load_metrics() -> dict:
+ """The metrics of the published model, so the retrain gate can compare against them.
+
+ Read from the same row as the model they describe: the gate's guarantee is that a
+ candidate beats the artifact currently being served, and metrics stored anywhere else
+ could be describing a different one.
+ """
+ row = ModerationModelArtifact.objects.order_by("-created_on").values("metrics").first()
+ if row is not None:
+ return row["metrics"]
+ bundled = Path(settings.DOMAIN_MODERATION_MODEL_DIR) / METRICS_FILENAME
+ return json.loads(bundled.read_text()) if bundled.exists() else {}
diff --git a/mwmbl/moderation/rules.py b/mwmbl/moderation/rules.py
index cc6dfc3..81f953a 100644
--- a/mwmbl/moderation/rules.py
+++ b/mwmbl/moderation/rules.py
@@ -44,6 +44,12 @@ NON_ENGLISH_TLDS = frozenset({
"pt", "es", "it", "fr", "de", "nl", "se", "no", "fi", "dk", "vn", "th", "id", "il", "ir",
})
+# How much an earlier decision on the same domain is worth. Named because the queue's SQL
+# reproduces this check to filter and order on it (mwmbl.moderation.suggest), and the two
+# drifting apart would show a moderator one suggestion and filter on another.
+PRIOR_DECISION_CONFIDENCE = 0.9
+PRIOR_DECISION_REASON = "OTHER"
+
@dataclass(frozen=True)
class EvidenceItem:
@@ -138,6 +144,16 @@ def crawl_evidence(domain: str, crawl: dict) -> list[EvidenceItem]:
"On a public malware/adult blocklist - approving here also unblocks it. "
"These lists are built for ad-blocking and do have false positives."))
+ # Neutral for the same reason robots.txt is: a fact the moderator should have, not a
+ # decision to make for them. Plenty of small personal sites - the ones this index exists
+ # for - still have no certificate, and no rejection detail a moderator has written
+ # mentions TLS. It only means anything when we actually reached the site: after a failed
+ # fetch "https: False" says the https attempt failed, which the unreachable line already
+ # said better.
+ if fetched and signals.get("https") is False:
+ items.append(EvidenceItem(
+ "no_tls", NEUTRAL, "Served over plain HTTP - no TLS certificate"))
+
if fetched:
items.append(EvidenceItem(
"reachable", APPROVE,
@@ -171,11 +187,12 @@ def live_evidence(submitter_record: dict, prior_decisions: dict) -> list[Evidenc
if prior_decisions.get("approved"):
items.append(EvidenceItem(
"prior_decision", APPROVE, "This domain has already been approved before",
- implies_action="APPROVE", implies_confidence=0.9))
+ implies_action="APPROVE", implies_confidence=PRIOR_DECISION_CONFIDENCE))
elif prior_decisions.get("rejected"):
items.append(EvidenceItem(
"prior_decision", REJECT, "This domain has already been rejected before",
- implies_action="REJECT", implies_reason="OTHER", implies_confidence=0.9))
+ implies_action="REJECT", implies_reason=PRIOR_DECISION_REASON,
+ implies_confidence=PRIOR_DECISION_CONFIDENCE))
return items
diff --git a/mwmbl/moderation/suggest.py b/mwmbl/moderation/suggest.py
index fabe619..fdd9d34 100644
--- a/mwmbl/moderation/suggest.py
+++ b/mwmbl/moderation/suggest.py
@@ -1,6 +1,6 @@
"""Tie the checks and the model together, and put the answer where the queue can read it.
-Two entry points, split by when they can be run:
+Three entry points, split by when they can be run:
:func:`refresh_suggestion` writes the cached suggestion onto a DomainEvidence row. It is
called by the enrichment task, the backfill command and the post-retrain rescore - never by a
@@ -9,15 +9,26 @@ request.
:func:`suggestion_for` reads that cached suggestion back and adds the evidence that would go
stale if it were cached: the submitter's record, and whether this domain has been decided
since. Both are indexed lookups.
+
+:func:`annotate_queue` says the same thing in SQL, because the queue filters, orders and
+paginates thousands of pending rows in the database and cannot call :func:`suggestion_for` to
+decide which ones to return.
+
+:func:`one_row_per_domain` and :func:`annotate_votes` are what turn that row-per-submission
+queryset into the row-per-domain the moderator actually reviews. Both are SQL for the same
+reason.
"""
from __future__ import annotations
from logging import getLogger
from typing import Iterable
-from django.db.models import Count, Q
+from django.db.models import (
+ Case, CharField, Count, Exists, F, FloatField, IntegerField, OuterRef, Q, Subquery, Value,
+ When)
+from django.db.models.functions import Coalesce, Substr
-from mwmbl.models import DomainEvidence, DomainSubmission
+from mwmbl.models import DomainEvidence, DomainSubmission, SearchResultVote
from mwmbl.moderation import rules
from mwmbl.moderation.evidence import page_texts
from mwmbl.moderation.model import Suggestion, suggest
@@ -33,12 +44,13 @@ def refresh_suggestion(evidence: DomainEvidence) -> DomainEvidence:
evidence.suggested_action = suggestion.action
evidence.suggested_reason = suggestion.reason
evidence.confidence = suggestion.confidence
- evidence.review_priority = suggestion.review_priority
+ evidence.reason_confidence = suggestion.reason_confidence
+ evidence.reason_source = suggestion.reason_source
evidence.model_version = suggestion.model_version
evidence.evidence = suggestion.evidence
evidence.save(update_fields=[
- "suggested_action", "suggested_reason", "confidence", "review_priority",
- "model_version", "evidence",
+ "suggested_action", "suggested_reason", "confidence", "reason_confidence",
+ "reason_source", "model_version", "evidence",
])
return evidence
@@ -97,13 +109,190 @@ def suggestion_for(submission: DomainSubmission,
action=action,
confidence=confidence,
reason=evidence.suggested_reason if action == "REJECT" else "",
- reason_confidence=confidence if action == "REJECT" else 0.0,
- reason_source="rule" if cached_decisive is not None else "model",
+ # The reason's own confidence, not the rejection's. A model can be sure a domain
+ # should go and barely have a view on why, and showing "SPAM (0.92)" when the reason
+ # head scored 0.31 would present a guess as a finding.
+ reason_confidence=(evidence.reason_confidence or 0.0) if action == "REJECT" else 0.0,
+ reason_source=evidence.reason_source or "model",
model_version=evidence.model_version,
evidence=list(evidence.evidence or []) + [item.to_dict() for item in live],
)
+# --------------------------------------------------------------------------------------
+# The same adjustment, in SQL.
+#
+# suggestion_for adjusts the stored suggestion in two ways that depend on other rows - an
+# earlier decision on the same domain overrides it, and an approval is withheld from a
+# submitter with no track record - so neither can be precomputed onto DomainEvidence. The
+# queue still has to filter, order and paginate on the result across every pending submission,
+# which has to happen in the database.
+#
+# So it is written twice, and test_queue_display_matches_suggestion_for holds the two
+# together. Filtering on UNSURE must return exactly the rows drawn as UNSURE, and the "needs a
+# human first" ordering must sort on the confidence the moderator is actually shown.
+
+DECIDED = Q(status__in=("APPROVED", "REJECTED"))
+
+
+def annotate_queue(submissions):
+ """Annotate submissions with the suggestion each one will be rendered with.
+
+ Adds ``displayed_action``, ``displayed_reason``, ``displayed_confidence`` and
+ ``displayed_priority``, all NULL while the domain has no READY evidence - which is how
+ "not assessed yet" sorts last and matches no suggestion filter.
+ """
+ # DomainEvidence is keyed on the domain rather than by a foreign key, because domains get
+ # resubmitted and the crawl belongs to the domain. A correlated subquery joins them
+ # without one, and keeps the filtering and ordering in SQL.
+ evidence = DomainEvidence.objects.filter(domain=OuterRef("name"))
+ submissions = submissions.annotate(
+ # Prefixed because DomainSubmission already has suggested_status and suggested_reason
+ # columns - the audit of what a moderator was shown - and an annotation may not
+ # shadow a real field.
+ evidence_state=Subquery(evidence.values("state")[:1]),
+ evidence_action=Subquery(evidence.values("suggested_action")[:1]),
+ evidence_reason=Subquery(evidence.values("suggested_reason")[:1]),
+ evidence_confidence=Subquery(evidence.values("confidence")[:1]),
+ evidence_source=Subquery(evidence.values("reason_source")[:1]),
+ # The live checks, as one boolean each. Only PENDING submissions are ever queued, so
+ # a submission never counts itself in either of these.
+ prior_approved=Exists(DomainSubmission.objects.filter(
+ name=OuterRef("name"), status="APPROVED")),
+ prior_rejected=Exists(DomainSubmission.objects.filter(
+ name=OuterRef("name"), status="REJECTED")),
+ submitter_decided=Exists(DomainSubmission.objects.filter(
+ DECIDED, submitted_by_id=OuterRef("submitted_by_id"))),
+ )
+
+ scored = Q(evidence_state=DomainEvidence.State.READY)
+ # A prior decision only overrides the stored suggestion when it is stronger than the check
+ # that produced it. reason_source is how we know a check produced it at all: when it did,
+ # the stored confidence *is* that check's implied confidence (see model.suggest).
+ stored_check_holds = (Q(evidence_source="rule")
+ & Q(evidence_confidence__gte=rules.PRIOR_DECISION_CONFIDENCE))
+ prior_approval = scored & Q(prior_approved=True) & ~stored_check_holds
+ prior_rejection = (scored & Q(prior_rejected=True) & Q(prior_approved=False)
+ & ~stored_check_holds)
+ # No cached check implies APPROVE - they are all reasons to reject - so a stored APPROVE
+ # is always the model's, which is what the withheld-approval rule is about.
+ withheld = scored & Q(evidence_action="APPROVE") & Q(submitter_decided=False)
+
+ submissions = submissions.annotate(
+ displayed_action=Case(
+ When(prior_approval, then=Value("APPROVE")),
+ When(prior_rejection, then=Value("REJECT")),
+ When(withheld, then=Value("UNSURE")),
+ When(scored, then=F("evidence_action")),
+ default=Value(None),
+ output_field=CharField(),
+ ),
+ displayed_confidence=Case(
+ When(prior_approval | prior_rejection,
+ then=Value(rules.PRIOR_DECISION_CONFIDENCE)),
+ When(withheld, then=Value(0.0)),
+ When(scored, then=F("evidence_confidence")),
+ default=Value(None),
+ output_field=FloatField(),
+ ),
+ displayed_reason=Case(
+ When(prior_rejection, then=Value(rules.PRIOR_DECISION_REASON)),
+ When(scored & Q(evidence_action="REJECT") & ~prior_approval,
+ then=F("evidence_reason")),
+ When(scored, then=Value("")),
+ default=Value(None),
+ output_field=CharField(),
+ ),
+ )
+
+ # Suggestion.review_priority, over the adjusted values. This is why the sort key is not a
+ # stored column: the rows whose approval was withheld are the ones a human most needs to
+ # look at, and a precomputed priority would sort them by the confidence of the approval we
+ # decided not to show - straight to the bottom of "what needs a human".
+ return submissions.annotate(
+ displayed_priority=Case(
+ When(Q(displayed_action="REJECT"), then=Value(1.0) + F("displayed_confidence")),
+ When(Q(displayed_action="APPROVE"), then=Value(1.0) - F("displayed_confidence")),
+ When(Q(displayed_action="UNSURE"), then=Value(1.0)),
+ default=Value(None),
+ output_field=FloatField(),
+ ),
+ )
+
+
+# --------------------------------------------------------------------------------------
+# From a submission per row to a domain per row.
+
+
+def one_row_per_domain(submissions, status: str | None = "PENDING", pick: str = "first"):
+ """Keep one submission per name, and count the rest onto it as ``submission_count``.
+
+ A moderator reviews a *domain*: nine submissions of cheap-rolex-outlet.biz are one card
+ and one decision, not nine. Deliberately not a GROUP BY, though: the suggestion depends on
+ the submitter's own track record (see :func:`suggestion_for`), and a domain submitted by
+ two accounts has no single one to read. Collapsing to a representative row instead leaves
+ :func:`annotate_queue` and :func:`suggestion_for` working on a submission, which is what
+ holds those two together.
+
+ ``pick`` chooses which row represents the domain, and in both cases it is the row whose
+ own fields the client is going to draw:
+
+ * ``"first"`` - the earliest submission, because the queue card says "first submitted 6
+ days ago by anon_4417";
+ * ``"last"`` - the most recently touched one, because a history listing shows the decision
+ that currently stands.
+
+ Ties break on the primary key, so the representative is deterministic either way.
+ """
+ touched = Coalesce("status_changed_on", "submitted_on")
+ submissions = submissions.annotate(touched=touched)
+
+ same_domain = DomainSubmission.objects.filter(name=OuterRef("name"))
+ if status is not None:
+ same_domain = same_domain.filter(status=status)
+
+ if pick == "first":
+ beats = same_domain.filter(
+ Q(submitted_on__lt=OuterRef("submitted_on"))
+ | Q(submitted_on=OuterRef("submitted_on"), pk__lt=OuterRef("pk")))
+ else:
+ beats = same_domain.annotate(touched=touched).filter(
+ Q(touched__gt=OuterRef("touched"))
+ | Q(touched=OuterRef("touched"), pk__gt=OuterRef("pk")))
+
+ return submissions.annotate(
+ submission_count=Coalesce(
+ Subquery(same_domain.values("name").annotate(n=Count("pk")).values("n"),
+ output_field=IntegerField()),
+ Value(0)),
+ ).filter(~Exists(beats))
+
+
+def annotate_votes(submissions):
+ """Add ``upvotes`` and ``downvotes``: the votes cast on URLs belonging to the domain.
+
+ In SQL because the queue orders on these across the whole backlog. SearchResultVote
+ carries the host in its own indexed column for exactly this - Postgres cannot pick a
+ domain out of a URLField - and both sides drop a leading ``www.`` so that a vote on
+ www.example.com/x counts towards a submission of example.com.
+
+ Coalesced to zero rather than left NULL: a domain nobody has voted on has no votes, and
+ ordering on NULL would scatter those rows through the sort instead of ending it.
+ """
+ def counted(vote_type):
+ return Coalesce(
+ Subquery(SearchResultVote.objects
+ .filter(domain=OuterRef("bare_name"), vote_type=vote_type)
+ .values("domain").annotate(n=Count("pk")).values("n"),
+ output_field=IntegerField()),
+ Value(0))
+
+ return submissions.annotate(
+ bare_name=Case(When(name__startswith="www.", then=Substr("name", 5)),
+ default=F("name"), output_field=CharField()),
+ ).annotate(upvotes=counted("upvote"), downvotes=counted("downvote"))
+
+
def submitter_record(user_id: int) -> dict:
"""How this submitter's previous submissions were decided.
diff --git a/mwmbl/moderation/train.py b/mwmbl/moderation/train.py
index 9bdad1d..3ac58bc 100644
--- a/mwmbl/moderation/train.py
+++ b/mwmbl/moderation/train.py
@@ -117,6 +117,7 @@ def evaluate(model: ModerationModel, train_rows: list[TrainingRow],
}
metrics["reason_head"] = _evaluate_reason_head(predictions, test_rows)
metrics["train_rows_by_source"] = _count_sources(train_rows)
+ metrics["suggestion_influence"] = _suggestion_influence(train_rows + test_rows)
return metrics
@@ -169,6 +170,28 @@ def _evaluate_reason_head(predictions, test_rows: list[TrainingRow]) -> dict:
}
+def _suggestion_influence(rows: list[TrainingRow]) -> dict:
+ """How much of the training data was decided with a suggestion on screen, and how often
+ the moderator agreed with it.
+
+ A retrain learns from decisions the previous model influenced, and the gate cannot see
+ that: the held-out labels come from the same influenced population, so a model that has
+ taught moderators its own mistakes scores well on them. Nothing here corrects for it -
+ reweighting on an unmeasured hunch is exactly what this module refuses to do elsewhere -
+ but a rising ``agreed`` against a rising ``shown`` is the shape to watch for, and it is
+ the argument for down-weighting confirmations when there is enough data to test it on.
+ """
+ real = [row for row in rows if row.source == REAL]
+ shown = [row for row in real if row.suggested_status]
+ agreed = [row for row in shown
+ if (row.suggested_status == "REJECTED") == row.rejected]
+ return {
+ "real_rows": len(real),
+ "shown_a_suggestion": len(shown),
+ "agreed_with_it": len(agreed),
+ }
+
+
def _has_both_classes(truth: np.ndarray) -> bool:
return truth.sum() >= MIN_CLASS_ROWS and (1 - truth).sum() >= MIN_CLASS_ROWS
diff --git a/mwmbl/moderation/training_data.py b/mwmbl/moderation/training_data.py
index 5d29ba9..870a98d 100644
--- a/mwmbl/moderation/training_data.py
+++ b/mwmbl/moderation/training_data.py
@@ -33,6 +33,13 @@ gate needs to vary them independently:
that is not adult content, the do-not-crawl cases, promotion of a paid product. It exists
to cover concepts, not to move aggregate numbers.
+Real rows carry the suggestion that was on screen when the decision was made, because a
+retrain trains on decisions the previous model had a hand in. That is reported by
+mwmbl.moderation.train as ``suggestion_influence`` rather than corrected for: with no decided
+submission yet made with a suggestion shown, any weighting would be an unmeasured guess, and
+this module's whole argument is that unmeasured changes to the training mix do not ship. The
+number is what will say when it needs to be.
+
Historic malformed submissions are excluded. 70 rows are ``null``, an IP address or
capitalised; migrations 0032/0033 fixed that at the API layer, so training on them would teach
the model to detect a problem that can no longer occur.
@@ -84,6 +91,10 @@ class TrainingRow:
# all, so that is the slice the retrain gate reads. None for derived and seed rows, which
# are training-only.
submitter: Optional[str] = None
+ # What the tool suggested when this decision was made, empty when nothing was shown. Not
+ # a feature either - it is how train.evaluate reports how much of the training data the
+ # previous model had a hand in.
+ suggested_status: str = ""
def to_example(self) -> ModerationExample:
return ModerationExample(self.domain, self.page_texts)
@@ -119,6 +130,7 @@ def real_rows() -> list[TrainingRow]:
page_texts=page_texts(evidence_by_domain.get(submission.name)),
timestamp=submission.submitted_on.isoformat() if submission.submitted_on else None,
submitter=str(submission.submitted_by_id),
+ suggested_status=submission.suggested_status,
))
return rows
diff --git a/mwmbl/platform/api.py b/mwmbl/platform/api.py
index 059fa09..293216f 100644
--- a/mwmbl/platform/api.py
+++ b/mwmbl/platform/api.py
@@ -6,9 +6,9 @@ from allauth.account.utils import setup_user_email, send_email_confirmation
from django.conf import settings
from django.core import signing
from django.core.exceptions import ValidationError
-from django.db.models import F, OuterRef, Subquery
+from django.db import transaction
+from django.db.models import F
from django.utils import timezone
-from redis import Redis
from ninja import Router
from ninja.pagination import paginate
from ninja_jwt.authentication import JWTAuth
@@ -21,13 +21,13 @@ from mwmbl.exceptions import InvalidRequest
from mwmbl.search_auth import invalidate_api_key_cache, invalidate_user_api_key_cache
from mwmbl.utils import normalize_domain, validate_domain
from mwmbl import pricing
-from mwmbl.background import enrich_domain_submission
-from mwmbl.crawler.stats import StatsManager
+from mwmbl.background import enrich_domain_submission, stats_manager
from mwmbl.models import AgreementType, MwmblUser, DomainEvidence, DomainSubmission, SearchResultVote, ApiKey, UsageBucket, UserBilling, UserAgreement, MarketingConsent, MarketingSource, generate_username
from mwmbl.moderation.suggest import (
- prior_decision_counts, prior_decisions, submitter_record, submitter_records,
- suggestion_for,
+ annotate_queue, annotate_votes, one_row_per_domain, prior_decision_counts, prior_decisions,
+ submitter_record, submitter_records, suggestion_for,
)
+from mwmbl.signals import schedule_blacklist_rebuild
from mwmbl.platform.schemas import (
Registration, ConfirmEmail, DomainSubmissionSchema, UpdateDomainSubmission,
VoteRequest, VoteRemoveRequest, VoteStatsRequest, VoteResponse, VoteStats, UserVoteHistory,
@@ -36,7 +36,8 @@ from mwmbl.platform.schemas import (
ForgotPasswordRequest, ResetPasswordRequest,
AgreementAcceptRequest, AgreementResponse,
MarketingConsentRequest, MarketingConsentResponse, MarketingConsentListResponse,
- BulkDecisionRequest, ModerationQueue, QueueItemSchema, SubmissionDetailSchema,
+ BulkDecisionRequest, ModeratedDomainSchema, ModerationHistory, ModerationQueue,
+ QueueItemSchema, SubmissionDetailSchema,
)
logger = logging.getLogger(__name__)
@@ -283,22 +284,26 @@ def check_moderator(request):
response=ModerationQueue,
summary="The moderation queue",
description=(
- "Domain submissions awaiting review, with the suggestion for each one. "
- "Requires the `change_domain_submission_status` permission.\n\n"
+ "Domains awaiting review, with the suggestion, the sample pages and the vote counts "
+ "for each one. Requires the `change_domain_submission_status` permission.\n\n"
+ "One row per *domain*, not per submission: a domain submitted nine times is one card "
+ "carrying `submission_count` of 9, and one decision settles all nine. `count` is "
+ "therefore distinct pending domains.\n\n"
"Suggestions are precomputed when a domain is submitted, so this endpoint runs no "
- "model: filtering, ordering and pagination all happen in the database. A submission "
- "whose domain is still being crawled comes back with `evidence_state` of `PENDING` "
- "and no suggestion, rather than a placeholder.\n\n"
- "`order_by=needs_review` (the default) puts confident rejections first, then rows the "
- "tool is unsure about, then confident approvals, and finally submissions that have "
- "not been crawled yet. `oldest` and `confidence` are also accepted."
+ "model: filtering, ordering and pagination all happen in the database. A domain "
+ "still being crawled comes back with `evidence_state` of `PENDING` and no suggestion, "
+ "rather than a placeholder.\n\n"
+ "`order_by=submissions` (the default) is the order the review screen shows - most "
+ "asked for first, then most upvoted. `needs_review` puts confident rejections first, "
+ "then rows the tool is unsure about, then confident approvals, and finally domains "
+ "that have not been crawled yet. `oldest` and `confidence` are also accepted."
),
)
def get_moderation_queue(
request,
limit: int = 50,
offset: int = 0,
- order_by: str = "needs_review",
+ order_by: str = "submissions",
suggested_action: str = None,
suggested_reason: str = None,
submitted_by: int = None,
@@ -306,33 +311,24 @@ def get_moderation_queue(
) -> dict:
check_moderator(request)
- # DomainEvidence is keyed on the domain name rather than by a foreign key, because domains
- # get resubmitted and the crawl belongs to the domain. A correlated subquery joins them
- # without one, and keeps the filtering, ordering and slicing in SQL - which is the whole
- # reason suggestions are precomputed. Building the list in Python would mean materialising
- # every pending submission (there are ~4,000) on every request.
- evidence = DomainEvidence.objects.filter(domain=OuterRef("name"))
- submissions = (DomainSubmission.objects
- .filter(status="PENDING")
- # Prefixed because DomainSubmission already has suggested_status and
- # suggested_reason columns - the audit of what a moderator was shown -
- # and an annotation may not shadow a real field.
- .annotate(
- evidence_state=Subquery(evidence.values("state")[:1]),
- evidence_action=Subquery(evidence.values("suggested_action")[:1]),
- evidence_reason=Subquery(evidence.values("suggested_reason")[:1]),
- evidence_confidence=Subquery(evidence.values("confidence")[:1]),
- evidence_priority=Subquery(evidence.values("review_priority")[:1]),
- ))
+ # Filtering and ordering happen on the suggestion as displayed, not as stored: the two
+ # differ wherever a live check adjusts the cached row, and a filter that disagrees with
+ # the screen is worse than no filter. annotate_queue keeps all of it in SQL - building the
+ # list in Python would mean materialising every pending submission (there are ~4,000) on
+ # every request, which is the whole reason suggestions are precomputed. one_row_per_domain
+ # and annotate_votes are in SQL for the same reason: the ordering sorts on both.
+ submissions = annotate_votes(one_row_per_domain(
+ annotate_queue(DomainSubmission.objects.filter(status="PENDING"))
+ )).select_related("submitted_by")
if submitted_by is not None:
submissions = submissions.filter(submitted_by_id=submitted_by)
if suggested_action:
- submissions = submissions.filter(evidence_action=suggested_action)
+ submissions = submissions.filter(displayed_action=suggested_action)
if suggested_reason:
- submissions = submissions.filter(evidence_reason=suggested_reason)
+ submissions = submissions.filter(displayed_reason=suggested_reason)
if min_confidence is not None:
- submissions = submissions.filter(evidence_confidence__gte=min_confidence)
+ submissions = submissions.filter(displayed_confidence__gte=min_confidence)
submissions = submissions.order_by(*_queue_ordering(order_by))
@@ -345,17 +341,27 @@ def get_moderation_queue(
# rejection, and a moderator can do nothing with one until the crawl lands.
QUEUE_ORDERINGS = {
"oldest": ("submitted_on",),
- "confidence": (F("evidence_confidence").desc(nulls_last=True), "submitted_on"),
- "needs_review": (F("evidence_priority").desc(nulls_last=True), "submitted_on"),
+ "confidence": (F("displayed_confidence").desc(nulls_last=True), "submitted_on"),
+ "needs_review": (F("displayed_priority").desc(nulls_last=True), "submitted_on"),
+ # What the review screen says on it: "sorted by submissions, then upvotes". The number of
+ # people who asked for a domain is the one signal here that is not the tool's own opinion,
+ # which is why it leads rather than the suggestion's confidence.
+ "submissions": (F("submission_count").desc(), F("upvotes").desc(), "submitted_on"),
}
def _queue_ordering(order_by: str):
- return QUEUE_ORDERINGS.get(order_by, QUEUE_ORDERINGS["needs_review"])
+ return QUEUE_ORDERINGS.get(order_by, QUEUE_ORDERINGS["submissions"])
def _queue_items(submissions: list[DomainSubmission]) -> list[QueueItemSchema]:
- """Build the page's rows, aggregating the live checks across the page rather than per row."""
+ """Build the page's rows, aggregating the live checks across the page rather than per row.
+
+ Each ``submission`` is the earliest pending submission of its domain, carrying the
+ per-domain counts one_row_per_domain and annotate_votes put on it. The pages and the
+ padlock come off the DomainEvidence rows already fetched here for the suggestion, so
+ showing them costs nothing.
+ """
evidence_by_domain = {row.domain: row for row in DomainEvidence.objects.filter(
domain__in=[submission.name for submission in submissions])}
submitters = submitter_records({submission.submitted_by_id for submission in submissions})
@@ -369,13 +375,18 @@ def _queue_items(submissions: list[DomainSubmission]) -> list[QueueItemSchema]:
submission, evidence,
submitter=submitters.get(submission.submitted_by_id, empty),
prior=priors.get(submission.name, empty))
+ signals = evidence.signals if evidence else {}
items.append(QueueItemSchema(
- id=submission.id,
name=submission.name,
- submitted_by=submission.submitted_by_id,
- submitted_on=submission.submitted_on,
- status=submission.status,
+ submission_count=submission.submission_count,
+ first_submitted_on=submission.submitted_on,
+ first_submitted_by=submission.submitted_by_id,
+ first_submitted_by_username=submission.submitted_by.username,
+ upvotes=submission.upvotes,
+ downvotes=submission.downvotes,
+ https=signals.get("https"),
evidence_state=evidence.state if evidence else "PENDING",
+ pages=evidence.pages if evidence else [],
suggestion=suggestion.__dict__ if suggestion else None,
))
return items
@@ -395,7 +406,8 @@ def _queue_items(submissions: list[DomainSubmission]) -> list[QueueItemSchema]:
def get_submission_detail(request, submission_id: int) -> SubmissionDetailSchema:
check_moderator(request)
- submission = DomainSubmission.objects.filter(id=submission_id).first()
+ submission = (DomainSubmission.objects.filter(id=submission_id)
+ .select_related("submitted_by").first())
if submission is None:
raise InvalidRequest("Submission not found.", status=404)
@@ -405,6 +417,7 @@ def get_submission_detail(request, submission_id: int) -> SubmissionDetailSchema
id=submission.id,
name=submission.name,
submitted_by=submission.submitted_by_id,
+ submitted_by_username=submission.submitted_by.username,
submitted_on=submission.submitted_on,
status=submission.status,
rejection_reason=submission.rejection_reason,
@@ -426,7 +439,9 @@ def get_submission_detail(request, submission_id: int) -> SubmissionDetailSchema
description=(
"Submit a screenful of individually-made decisions in one request. Each entry carries "
"its own status, so this cannot be used to accept every suggestion in bulk - a "
- "moderator still decides each submission. "
+ "moderator still decides each domain.\n\n"
+ "A decision is addressed to a domain and settles **every** submission of that name, "
+ "including ones already decided - which is also how a past decision is changed. "
"Requires the `change_domain_submission_status` permission."
),
)
@@ -434,19 +449,150 @@ def submit_decisions(request, decisions: BulkDecisionRequest):
check_email_verified(request)
check_moderator(request)
- submissions = {submission.id: submission for submission in DomainSubmission.objects.filter(
- id__in=[decision.submission_id for decision in decisions.decisions])}
+ names = [normalize_domain(decision.domain) for decision in decisions.decisions]
+ by_name = {}
+ for submission in DomainSubmission.objects.filter(name__in=names):
+ by_name.setdefault(submission.name, []).append(submission)
+
+ domains, updated, missing = 0, 0, []
+ with transaction.atomic():
+ for decision, name in zip(decisions.decisions, names):
+ submissions = by_name.get(name)
+ if not submissions:
+ missing.append(decision.domain)
+ continue
+ # Every submission of the name, not only the pending ones. A domain is one thing
+ # to a moderator, and leaving the already-decided rows behind would let a domain
+ # sit half approved and half rejected - and would mean re-deciding one needed a
+ # different endpoint from deciding it.
+ for submission in submissions:
+ apply_decision(submission, decision, request.user)
+ updated += 1
+ domains += 1
+
+ return {"status": "ok", "domains": domains, "updated": updated, "not_found": missing}
+
+
+ "/domain-submissions/domains/{domain}/undo",
+ auth=JWTAuth(),
+ summary="Undo the decision on a domain",
+ description=(
+ "Put every submission of a domain back to PENDING and clear the rejection reason and "
+ "detail, so it returns to the queue. "
+ "Requires the `change_domain_submission_status` permission."
+ ),
+)
+def undo_decision(request, domain: str):
+ check_email_verified(request)
+ check_moderator(request)
+
+ name = normalize_domain(domain)
+ submissions = DomainSubmission.objects.filter(name=name)
+ was_approved = submissions.filter(status="APPROVED").exists()
+ # Deliberately leaves suggested_status, suggested_reason, suggestion_confidence and
+ # suggestion_model_version alone. Those record what was on screen when the decision being
+ # undone was made, which is the whole reason they are stored separately from the live
+ # suggestion - and re-posting a PENDING status through apply_decision, the nearest thing
+ # to an undo before this endpoint, overwrote them from the request.
+ undone = submissions.update(
+ status="PENDING", rejection_reason="", rejection_detail="",
+ status_changed_by=request.user, status_changed_on=timezone.now())
+ if not undone:
+ raise InvalidRequest(f"No submissions found for {name}.", status=404)
+
+ if was_approved:
+ # .update() does not fire post_save, so the approval receiver never runs. An undone
+ # approval has to reach the snapshot too: the domain was subtracted from the remote
+ # blocklists when it was approved, and until the snapshot is rebuilt it stays
+ # subtracted. Same debounce as mwmbl.signals.
+ schedule_blacklist_rebuild()
+
+ return {"status": "ok", "message": f"{name} is pending again.", "updated": undone}
+
+
+HISTORY_ORDERINGS = {
+ "recent": (F("touched").desc(), "-pk"),
+ "oldest": (F("touched").asc(), "pk"),
+}
+
+
+ "/domain-submissions/moderated",
+ auth=JWTAuth(),
+ response=ModerationHistory,
+ summary="Domains that have been moderated",
+ description=(
+ "Past moderations, one row per domain, newest decision first. Filter by `status` "
+ "(`PENDING`, `APPROVED` or `REJECTED`; omit for any), by the `moderator` who made the "
+ "call, or by exact domain `name`.\n\n"
+ "Each row carries the `suggested_*` audit columns, so a moderator revisiting a "
+ "decision sees what was on screen when it was made rather than what the model would "
+ "say about the domain today. Changing a decision is a normal POST to "
+ "`/domain-submissions/decisions`. "
+ "Requires the `change_domain_submission_status` permission."
+ ),
+)
+def get_moderation_history(
+ request,
+ limit: int = 50,
+ offset: int = 0,
+ status: str = None,
+ moderator: int = None,
+ name: str = None,
+ order_by: str = "recent",
+) -> dict:
+ check_moderator(request)
+
+ submissions = DomainSubmission.objects.all()
+ if status:
+ submissions = submissions.filter(status=status)
+ if moderator is not None:
+ submissions = submissions.filter(status_changed_by_id=moderator)
+ if name:
+ submissions = submissions.filter(name=normalize_domain(name))
- updated, missing = 0, []
- for decision in decisions.decisions:
- submission = submissions.get(decision.submission_id)
- if submission is None:
- missing.append(decision.submission_id)
- continue
- apply_decision(submission, decision, request.user)
- updated += 1
+ # "last" rather than "first": a history row shows the decision that currently stands, so
+ # the domain is represented by its most recently touched submission, not its oldest.
+ submissions = one_row_per_domain(submissions, status=status, pick="last")
+ submissions = (submissions.select_related("submitted_by", "status_changed_by")
+ .order_by(*HISTORY_ORDERINGS.get(order_by, HISTORY_ORDERINGS["recent"])))
- return {"status": "ok", "updated": updated, "not_found": missing}
+ count = submissions.count()
+ return {"items": _moderated_items(list(submissions[offset:offset + limit])), "count": count}
+
+
+def _moderated_items(submissions: list[DomainSubmission]) -> list[ModeratedDomainSchema]:
+ """Build the page's rows. The representative row here is the last-touched submission, so
+ "who first asked for this domain" needs one more lookup - once for the page, not per row."""
+ first_by_name = {}
+ for submission in (DomainSubmission.objects
+ .filter(name__in=[row.name for row in submissions])
+ .select_related("submitted_by").order_by("submitted_on", "pk")):
+ first_by_name.setdefault(submission.name, submission)
+
+ items = []
+ for submission in submissions:
+ first = first_by_name[submission.name]
+ items.append(ModeratedDomainSchema(
+ name=submission.name,
+ submission_count=submission.submission_count,
+ status=submission.status,
+ rejection_reason=submission.rejection_reason,
+ rejection_detail=submission.rejection_detail,
+ first_submitted_on=first.submitted_on,
+ first_submitted_by=first.submitted_by_id,
+ first_submitted_by_username=first.submitted_by.username,
+ status_changed_on=submission.status_changed_on,
+ status_changed_by=submission.status_changed_by_id,
+ status_changed_by_username=(submission.status_changed_by.username
+ if submission.status_changed_by else ""),
+ suggested_status=submission.suggested_status,
+ suggested_reason=submission.suggested_reason,
+ suggestion_confidence=submission.suggestion_confidence,
+ suggestion_model_version=submission.suggestion_model_version,
+ ))
+ return items
@router.post(
@@ -476,8 +622,7 @@ def refetch_submission_evidence(request, submission_id: int):
def _index_stats(domain: str) -> dict:
"""What the crawler already knows about this domain, when Redis is reachable."""
try:
- manager = StatsManager(Redis.from_url(settings.REDIS_URL, decode_responses=True))
- return manager.get_stats_for_domain(domain).__dict__
+ return stats_manager.get_stats_for_domain(domain).__dict__
except Exception:
logger.warning("Could not read index stats for %s", domain, exc_info=True)
return {}
diff --git a/mwmbl/platform/schemas.py b/mwmbl/platform/schemas.py
index 6db74d2..a3d510c 100644
--- a/mwmbl/platform/schemas.py
+++ b/mwmbl/platform/schemas.py
@@ -1,6 +1,7 @@
from datetime import datetime
from typing import Optional, Literal
from ninja import Schema, ModelSchema, Field
+from pydantic import model_validator
from mwmbl.models import AgreementType, DomainSubmission, MarketingSource
@@ -165,13 +166,44 @@ class ConfirmEmail(Schema):
)
+class RejectionFieldsMixin:
+ """Validate the rejection reason and its detail, wherever a decision arrives.
+
+ Django enforces neither of these. ``choices`` is not checked on ``save()``, so an
+ arbitrary string of twenty characters or fewer lands in the column and comes back out at
+ a moderator as a reason nothing knows how to render; and nothing has ever required the
+ detail that "Other - needs detail" promises, which is the only place a rejected submitter
+ finds out what was actually wrong.
+
+ A schema validator rather than a check in the view, so these come back as a 422 naming
+ the field, the same as an over-long audit value.
+ """
+
+ @model_validator(mode="after")
+ def _check_rejection(self):
+ # Both columns are blank=True, so ninja types them Optional and an omitted field
+ # arrives as None rather than "".
+ reason = self.rejection_reason or ""
+ detail = self.rejection_detail or ""
+ if reason and reason not in DomainSubmission.DOMAIN_REJECTION_REASON:
+ allowed = ", ".join(DomainSubmission.DOMAIN_REJECTION_REASON)
+ raise ValueError(f"rejection_reason must be one of: {allowed}")
+ if reason and self.status != "REJECTED":
+ raise ValueError("rejection_reason is only meaningful when status is REJECTED")
+ if reason == "OTHER" and not detail.strip():
+ raise ValueError(
+ "rejection_detail is required when rejection_reason is OTHER - it is what "
+ "the submitter is shown")
+ return self
+
+
class DomainSubmissionSchema(ModelSchema):
class Meta:
model = DomainSubmission
fields = ["id", "name", "submitted_by", "submitted_on", "status", "rejection_reason", "rejection_detail"]
-class UpdateDomainSubmission(ModelSchema):
+class UpdateDomainSubmission(RejectionFieldsMixin, ModelSchema):
class Meta:
model = DomainSubmission
fields = ["status", "rejection_reason", "rejection_detail"]
@@ -179,14 +211,21 @@ class UpdateDomainSubmission(ModelSchema):
# Echoed back by the client so we record what the moderator was actually looking at.
# Optional: a client that shows no suggestions simply omits them, and a decision made
# without a suggestion is stored as exactly that rather than as a suggestion of "".
+ #
+ # The lengths match the columns these land in. Django does not enforce max_length on
+ # save(), so without them an over-long value is a 500 from Postgres where it should be a
+ # 400 naming the field.
suggested_status: Optional[str] = Field(
- default=None, description="The action the tool suggested, as shown to the moderator.")
+ default=None, max_length=20,
+ description="The action the tool suggested, as shown to the moderator.")
suggested_reason: Optional[str] = Field(
- default=None, description="The rejection reason the tool suggested, if any.")
+ default=None, max_length=20,
+ description="The rejection reason the tool suggested, if any.")
suggestion_confidence: Optional[float] = Field(
default=None, description="Confidence of the suggestion that was shown.")
suggestion_model_version: Optional[str] = Field(
- default=None, description="Version of the model that produced the shown suggestion.")
+ default=None, max_length=50,
+ description="Version of the model that produced the shown suggestion.")
class EvidenceItemSchema(Schema):
@@ -223,14 +262,35 @@ class CrawledPageSchema(Schema):
class QueueItemSchema(Schema):
- """A row of the moderation queue: the submission plus its precomputed suggestion."""
- id: int
+ """One pending domain: everything the review card and its up-next row draw.
+
+ Keyed on the domain, not on a submission. A domain submitted nine times is one thing to
+ look at and one decision to make, so there is no single submission id behind a row -
+ decisions are addressed to `name` (see `DomainDecision`).
+
+ Everything a card shows is here so that moving to the next one costs no request: the
+ sample pages, the vote counts, the padlock and the suggestion all travel with the row.
+ """
name: str
- submitted_by: int
- submitted_on: datetime
- status: str
+ submission_count: int = Field(
+ description="How many submissions of this domain are still pending.")
+ first_submitted_on: datetime = Field(
+ description="When this domain was first submitted, by whoever asked first.")
+ first_submitted_by: int
+ first_submitted_by_username: str
+ upvotes: int = Field(
+ description="Votes cast on results from this domain, ignoring a leading `www.`.")
+ downvotes: int
+ https: Optional[bool] = Field(
+ default=None,
+ description=(
+ "Whether the homepage answered over TLS. `null` until the domain has been "
+ "crawled - an uncrawled domain must not draw an open padlock."
+ ))
evidence_state: str = Field(
description="`PENDING` while the domain is still being crawled, then `READY` or `FAILED`.")
+ pages: list[CrawledPageSchema] = Field(
+ default=[], description="The pages crawled from the domain, as shown on the card.")
suggestion: Optional[SuggestionSchema] = Field(
default=None,
description="Absent until the domain has been crawled. Never a placeholder.")
@@ -241,6 +301,7 @@ class SubmissionDetailSchema(Schema):
id: int
name: str
submitted_by: int
+ submitted_by_username: str = ""
submitted_on: datetime
status: str
rejection_reason: str = ""
@@ -257,15 +318,21 @@ class SubmissionDetailSchema(Schema):
default={}, description="What the crawler already knows about this domain.")
-class BulkDecision(Schema):
- submission_id: int
- status: str
- rejection_reason: str = ""
- rejection_detail: str = ""
- suggested_status: Optional[str] = None
- suggested_reason: Optional[str] = None
+class DomainDecision(RejectionFieldsMixin, Schema):
+ """One decision, about one domain. Lengths match the columns these are stored in.
+
+ Addressed to the domain rather than to a submission id, because that is what a moderator
+ decided about: a decision settles every submission of the name at once, so nine asks for
+ the same site cannot end up half approved and half rejected.
+ """
+ domain: str = Field(max_length=300)
+ status: str = Field(max_length=20)
+ rejection_reason: str = Field(default="", max_length=20)
+ rejection_detail: str = Field(default="", max_length=300)
+ suggested_status: Optional[str] = Field(default=None, max_length=20)
+ suggested_reason: Optional[str] = Field(default=None, max_length=20)
suggestion_confidence: Optional[float] = None
- suggestion_model_version: Optional[str] = None
+ suggestion_model_version: Optional[str] = Field(default=None, max_length=50)
class BulkDecisionRequest(Schema):
@@ -274,7 +341,37 @@ class BulkDecisionRequest(Schema):
Deliberately not "accept all suggestions": each entry carries its own status, so the
client can only send choices a moderator actually made.
"""
- decisions: list[BulkDecision]
+ decisions: list[DomainDecision]
+
+
+class ModeratedDomainSchema(Schema):
+ """One domain in the moderation history: the decision that currently stands on it.
+
+ One row per domain, like the queue, and for the same reason - and it carries the
+ `suggested_*` audit columns so a moderator revisiting a call can see what was on screen
+ when it was made, rather than what the model would say about the domain today.
+ """
+ name: str
+ submission_count: int
+ status: str
+ rejection_reason: str = ""
+ rejection_detail: str = ""
+ first_submitted_on: datetime
+ first_submitted_by: int
+ first_submitted_by_username: str
+ status_changed_on: Optional[datetime] = None
+ status_changed_by: Optional[int] = None
+ status_changed_by_username: str = ""
+ suggested_status: str = ""
+ suggested_reason: str = ""
+ suggestion_confidence: Optional[float] = None
+ suggestion_model_version: str = ""
+
+
+class ModerationHistory(Schema):
+ """A page of past moderations. Same `items`/`count` shape as the queue."""
+ items: list[ModeratedDomainSchema]
+ count: int
class VoteRequest(Schema):
@@ -381,6 +478,10 @@ class UserVoteHistory(Schema):
class ModerationQueue(Schema):
"""A page of the moderation queue. Shape matches the paginated endpoints the client
- already consumes, so `items`/`count` need no special handling."""
+ already consumes, so `items`/`count` need no special handling.
+
+ `count` is distinct pending *domains* - the "24 pending" in the header - not pending
+ submissions.
+ """
items: list[QueueItemSchema]
count: int
diff --git a/mwmbl/settings_common.py b/mwmbl/settings_common.py
index b04cd28..04cbd5d 100644
--- a/mwmbl/settings_common.py
+++ b/mwmbl/settings_common.py
@@ -355,6 +355,10 @@ EXTERNAL_CACHE_NEGATIVE_TTL_SECONDS = 7 * 24 * 60 * 60
# Domain moderation suggestions. The model is loaded only by the background worker that
# enriches submissions - the queue reads the stored suggestion - so a missing artifact
# degrades to the deterministic checks rather than affecting any moderator request.
+#
+# Retrained models are stored in Postgres (mwmbl.models.ModerationModelArtifact), because a
+# container filesystem is neither shared between workers nor kept across a deploy. This
+# directory holds the read-only warm start used until the first retrain publishes one.
DOMAIN_MODERATION_MODEL_DIR = os.environ.get(
"DOMAIN_MODERATION_MODEL_DIR",
str(Path(__file__).parent / "moderation" / "artifacts"))
diff --git a/mwmbl/signals.py b/mwmbl/signals.py
index 16a4791..0b58ce0 100644
--- a/mwmbl/signals.py
+++ b/mwmbl/signals.py
@@ -62,13 +62,23 @@ def rebuild_blacklist_snapshot_on_approval(sender, instance: DomainSubmission, *
if instance.status != "APPROVED":
return
+ if schedule_blacklist_rebuild():
+ logger.info("Scheduled a blacklist snapshot rebuild after approving %s", instance.name)
+
+
+def schedule_blacklist_rebuild() -> bool:
+ """Ask for a snapshot rebuild, unless one is already due. True if this call scheduled it.
+
+ Shared with the undo endpoint, which changes statuses with .update() and so fires no
+ post_save at all: an approval that is undone has to reach the snapshot for the same reason
+ the approval did, in the other direction.
+ """
from mwmbl.background import refresh_blacklist_snapshot
delay = settings.BLACKLIST_SNAPSHOT_APPROVAL_DELAY_SECONDS
cutoff = timezone.now() + timedelta(seconds=delay)
if Task.objects.filter(task_name=BLACKLIST_SNAPSHOT_TASK, run_at__lte=cutoff).exists():
- return
+ return False
refresh_blacklist_snapshot(schedule=delay)
- logger.info("Scheduled a blacklist snapshot rebuild in %ds after approving %s",
- delay, instance.name)
+ return True
diff --git a/mwmbl/utils.py b/mwmbl/utils.py
index 3c5645b..801b843 100644
--- a/mwmbl/utils.py
+++ b/mwmbl/utils.py
@@ -93,6 +93,20 @@ def normalize_domain(domain_or_url: str) -> str:
return netloc.lower()
+def bare_host(domain_or_url: str) -> str:
+ """:func:`normalize_domain`, minus a leading ``www.``.
+
+ >>> bare_host("https://www.example.com/a/b?c=d"), bare_host("example.com")
+ ('example.com', 'example.com')
+
+ Votes are cast against result URLs and domains are submitted by hand, so the two disagree
+ about ``www.`` constantly. Reducing both sides to the same host is what makes a vote on
+ www.example.com/x count towards a submission of example.com.
+ """
+ host = normalize_domain(domain_or_url)
+ return host[4:] if host.startswith("www.") else host
+
+
def validate_domain(domain_or_url: str):
domain = normalize_domain(domain_or_url)
if VALID_DOMAIN_REGEX.fullmatch(domain) is None:
diff --git a/test/test_crawl_functional.py b/test/test_crawl_functional.py
index 0a60f28..b16a8cb 100644
--- a/test/test_crawl_functional.py
+++ b/test/test_crawl_functional.py
@@ -242,11 +242,12 @@ class TestCrawlFunctional:
with patch('mwmbl.crawler.retrieve.fetch') as mock_fetch:
with patch('mwmbl.crawler.retrieve.robots_allowed', return_value=True):
- mock_fetch.return_value = (200, mock_html_content)
+ mock_fetch.return_value = (200, mock_html_content, 'https://example.com')
result = crawl_url('https://example.com', redis=None)
assert result['url'] == 'https://example.com'
+ assert result['resolved_url'] == 'https://example.com'
assert result['status'] == 200
assert result['content'] is not None
assert result['content']['title'] == 'Test Page'
diff --git a/test/test_domain_moderation.py b/test/test_domain_moderation.py
index 176693d..b941ad0 100644
--- a/test/test_domain_moderation.py
+++ b/test/test_domain_moderation.py
@@ -5,20 +5,28 @@ the training gate - but that a moderator is never shown something misleading: a
either backed by evidence or absent, a deterministic check always beats a probability, and the
queue never runs a model on the request path.
"""
+import io
+import json
from datetime import timedelta
from unittest import mock
+import joblib
import pytest
from allauth.account.models import EmailAddress
+from background_task.models import Task
from django.contrib.auth.models import Permission
from django.utils import timezone
from ninja_jwt.tokens import RefreshToken
-from mwmbl.models import DomainEvidence, DomainSubmission, MwmblUser
+from mwmbl.models import (
+ DomainEvidence, DomainSubmission, ModerationModelArtifact, MwmblUser, SearchResultVote)
+from mwmbl.moderation import model as model_module
from mwmbl.moderation import rules
+from mwmbl.moderation.evidence import crawl_domain
from mwmbl.moderation.features import Featuriser, ModerationExample
-from mwmbl.moderation.model import Suggestion, suggest
-from mwmbl.moderation.suggest import refresh_suggestion, suggestion_for
+from mwmbl.moderation.model import (
+ Suggestion, get_model, load_metrics, publish, reset_model_cache, suggest)
+from mwmbl.moderation.suggest import annotate_queue, refresh_suggestion, suggestion_for
from mwmbl.moderation.train import passes_gate
from mwmbl.moderation.training_data import TrainingRow, is_trainable_domain
@@ -31,6 +39,25 @@ def submitter(db):
username="submitter", email="[email protected]", password="password")
[email protected](autouse=True)
+def clear_the_model_cache():
+ """The loaded model is a process-wide singleton, and several tests here publish one."""
+ reset_model_cache()
+ yield
+ reset_model_cache()
+
+
+def established(db):
+ """A submitter with a track record, whose approvals are not withheld."""
+ user = MwmblUser.objects.create_user(
+ username="established", email="[email protected]", password="password")
+ for index in range(3):
+ DomainSubmission.objects.create(name=f"past{index}.example", submitted_by=user,
+ status="APPROVED")
+ return user
+
+
@pytest.fixture
def moderator(db):
user = MwmblUser.objects.create_user(
@@ -55,7 +82,7 @@ def ready_evidence(domain, **overrides):
"signals": {"has_links": True, "num_pages_fetched": 1, "blacklisted": False},
"suggested_action": "APPROVE",
"confidence": 0.8,
- "review_priority": 0.2,
+ "reason_source": "model",
"evidence": [],
"model_version": "test-model",
}
@@ -209,20 +236,21 @@ def test_uncrawled_submission_reports_that_rather_than_guessing(client, moderato
@pytest.mark.django_db
-def test_queue_orders_rows_that_need_a_human_first(client, moderator, submitter):
- for name, action, priority in [("approve-me.com", "APPROVE", 0.1),
- ("reject-me.com", "REJECT", 1.9),
- ("unsure.com", "UNSURE", 1.0)]:
- DomainSubmission.objects.create(name=name, submitted_by=submitter)
- ready_evidence(name, suggested_action=action, review_priority=priority)
-
- response = client.get(QUEUE_URL, headers={"Authorization": token(moderator)})
+def test_queue_orders_rows_that_need_a_human_first(client, moderator, established):
+ for name, action in [("approve-me.com", "APPROVE"),
+ ("reject-me.com", "REJECT"),
+ ("unsure.com", "UNSURE")]:
+ DomainSubmission.objects.create(name=name, submitted_by=established)
+ ready_evidence(name, suggested_action=action, confidence=0.8)
+
+ response = client.get(f"{QUEUE_URL}?order_by=needs_review",
+ headers={"Authorization": token(moderator)})
assert [item["name"] for item in response.json()["items"]] == [
"reject-me.com", "unsure.com", "approve-me.com"]
@pytest.mark.django_db
-def test_queue_filters_on_the_stored_suggestion(client, moderator, submitter):
+def test_queue_filters_on_the_suggestion(client, moderator, submitter):
for name, action in [("a.com", "APPROVE"), ("b.com", "REJECT")]:
DomainSubmission.objects.create(name=name, submitted_by=submitter)
ready_evidence(name, suggested_action=action)
@@ -261,8 +289,8 @@ def test_bulk_decisions_apply_each_choice_separately(client, moderator, submitte
response = client.post(
"/api/v1/platform/domain-submissions/decisions",
data={"decisions": [
- {"submission_id": first.id, "status": "APPROVED"},
- {"submission_id": second.id, "status": "REJECTED", "rejection_reason": "SPAM"},
+ {"domain": "a.com", "status": "APPROVED"},
+ {"domain": "b.com", "status": "REJECTED", "rejection_reason": "SPAM"},
]},
content_type="application/json", headers={"Authorization": token(moderator)})
@@ -287,6 +315,296 @@ def test_refetch_clears_the_evidence_so_the_crawl_actually_reruns(client, modera
enrich.assert_called_once_with("was-down.com")
+# ------------------------------------------------------------ the queue is per domain
+
+DECISIONS_URL = "/api/v1/platform/domain-submissions/decisions"
+HISTORY_URL = "/api/v1/platform/domain-submissions/moderated"
+
+
+def upvote(url, username):
+ SearchResultVote.objects.create(
+ user=MwmblUser.objects.create_user(username=username), url=url, query="q",
+ vote_type="upvote")
+
+
+def downvote(url, username):
+ SearchResultVote.objects.create(
+ user=MwmblUser.objects.create_user(username=username), url=url, query="q",
+ vote_type="downvote")
+
+
+def test_a_domain_submitted_nine_times_is_one_row(client, moderator, established):
+ """A moderator reviews a domain, not a submission. Nine asks for the same site are one
+ card and one decision - the old queue made them nine of each."""
+ for _ in range(9):
+ DomainSubmission.objects.create(name="cheap-rolex.biz", submitted_by=established)
+ DomainSubmission.objects.create(name="solarpunk.zone", submitted_by=established)
+ ready_evidence("cheap-rolex.biz")
+ ready_evidence("solarpunk.zone")
+
+ body = client.get(QUEUE_URL, headers={"Authorization": token(moderator)}).json()
+
+ assert body["count"] == 2
+ counts = {item["name"]: item["submission_count"] for item in body["items"]}
+ assert counts == {"cheap-rolex.biz": 9, "solarpunk.zone": 1}
+
+
+def test_the_row_is_the_first_submission_of_the_domain(client, moderator):
+ """The card says "first submitted 6 days ago by anon_4417", so the row we deduplicate to
+ has to be the row those two fields come from."""
+ first = MwmblUser.objects.create_user(username="anon_4417")
+ later = MwmblUser.objects.create_user(username="someone_else")
+ DomainSubmission.objects.create(name="a.com", submitted_by=first,
+ submitted_on=timezone.now() - timedelta(days=6))
+ DomainSubmission.objects.create(name="a.com", submitted_by=later,
+ submitted_on=timezone.now() - timedelta(days=1))
+ ready_evidence("a.com")
+
+ item = client.get(QUEUE_URL, headers={"Authorization": token(moderator)}).json()["items"][0]
+
+ assert item["first_submitted_by_username"] == "anon_4417"
+ assert item["first_submitted_by"] == first.id
+
+
+def test_votes_are_rolled_up_to_the_domain(client, moderator, established):
+ """The card shows the domain's score, so votes on any of its URLs count - including ones
+ cast against www., which is how half the index refers to the same site."""
+ DomainSubmission.objects.create(name="solarpunk.zone", submitted_by=established)
+ ready_evidence("solarpunk.zone")
+ upvote("https://solarpunk.zone/notes/co-op", "voter1")
+ upvote("https://www.solarpunk.zone/repair-cafes", "voter2")
+ downvote("https://solarpunk.zone/about", "voter3")
+ upvote("https://somewhere-else.com/", "voter4")
+
+ DomainSubmission.objects.create(name="silent.example", submitted_by=established)
+ ready_evidence("silent.example")
+
+ items = {item["name"]: item
+ for item in client.get(QUEUE_URL,
+ headers={"Authorization": token(moderator)}).json()["items"]}
+
+ assert (items["solarpunk.zone"]["upvotes"], items["solarpunk.zone"]["downvotes"]) == (2, 1)
+ # Zero, not null: a domain nobody has voted on has to sort with the rest.
+ assert (items["silent.example"]["upvotes"], items["silent.example"]["downvotes"]) == (0, 0)
+
+
+def test_a_www_submission_still_finds_its_votes(client, moderator, established):
+ DomainSubmission.objects.create(name="www.example.com", submitted_by=established)
+ ready_evidence("www.example.com")
+ upvote("https://example.com/a", "voter1")
+
+ item = client.get(QUEUE_URL, headers={"Authorization": token(moderator)}).json()["items"][0]
+ assert item["upvotes"] == 1
+
+
+def test_the_default_order_is_most_submitted_then_most_upvoted(client, moderator, established):
+ for name, submissions, upvotes in [("popular.com", 3, 1), ("tied-low.com", 2, 1),
+ ("tied-high.com", 2, 40), ("lonely.com", 1, 99)]:
+ for _ in range(submissions):
+ DomainSubmission.objects.create(name=name, submitted_by=established)
+ for index in range(upvotes):
+ upvote(f"https://{name}/{index}", f"voter-{name}-{index}")
+ ready_evidence(name)
+
+ body = client.get(QUEUE_URL, headers={"Authorization": token(moderator)}).json()
+ assert [item["name"] for item in body["items"]] == [
+ "popular.com", "tied-high.com", "tied-low.com", "lonely.com"]
+
+
+def test_the_card_carries_its_sample_pages_and_padlock(client, moderator, established):
+ """Everything a card draws travels with the row, so advancing to the next one costs no
+ request."""
+ DomainSubmission.objects.create(name="crawled.com", submitted_by=established)
+ ready_evidence("crawled.com", signals={"has_links": True, "https": False})
+ DomainSubmission.objects.create(name="uncrawled.com", submitted_by=established)
+
+ items = {item["name"]: item
+ for item in client.get(QUEUE_URL,
+ headers={"Authorization": token(moderator)}).json()["items"]}
+
+ assert items["crawled.com"]["pages"][0]["title"] == "A title"
+ assert items["crawled.com"]["https"] is False
+ # Not False: an uncrawled domain must not draw an open padlock.
+ assert items["uncrawled.com"]["https"] is None
+ assert items["uncrawled.com"]["pages"] == []
+
+
+def test_one_decision_settles_every_submission_of_the_domain(client, moderator, submitter):
+ for _ in range(9):
+ DomainSubmission.objects.create(name="cheap-rolex.biz", submitted_by=submitter)
+
+ response = client.post(
+ DECISIONS_URL,
+ data={"decisions": [{"domain": "cheap-rolex.biz", "status": "REJECTED",
+ "rejection_reason": "SPAM"}]},
+ content_type="application/json", headers={"Authorization": token(moderator)})
+
+ assert response.json() == {"status": "ok", "domains": 1, "updated": 9, "not_found": []}
+ assert not DomainSubmission.objects.filter(name="cheap-rolex.biz",
+ status="PENDING").exists()
+
+
+def test_a_decision_can_be_changed_after_the_fact(client, moderator, submitter):
+ """Re-deciding is the same request as deciding: a domain already rejected can be approved
+ without a second endpoint, and does not end up half one and half the other."""
+ DomainSubmission.objects.create(name="a.com", submitted_by=submitter, status="REJECTED",
+ rejection_reason="SPAM")
+ DomainSubmission.objects.create(name="a.com", submitted_by=submitter)
+
+ client.post(DECISIONS_URL,
+ data={"decisions": [{"domain": "a.com", "status": "APPROVED"}]},
+ content_type="application/json",
+ headers={"Authorization": token(moderator)})
+
+ statuses = set(DomainSubmission.objects.filter(name="a.com")
+ .values_list("status", "rejection_reason"))
+ assert statuses == {("APPROVED", "")}
+
+
+def test_a_decision_on_an_unknown_domain_is_reported_not_invented(client, moderator):
+ response = client.post(
+ DECISIONS_URL,
+ data={"decisions": [{"domain": "never-submitted.com", "status": "APPROVED"}]},
+ content_type="application/json", headers={"Authorization": token(moderator)})
+
+ assert response.json()["not_found"] == ["never-submitted.com"]
+
+
+# ------------------------------------------------------------------- undo and history
+
+UNDO_URL = "/api/v1/platform/domain-submissions/domains/{}/undo"
+
+
+def test_undo_returns_a_domain_to_the_queue_without_rewriting_the_audit(client, moderator,
+ submitter):
+ """The suggested_* columns record what was on screen when the decision was made. Posting
+ a PENDING status back through the decision endpoint - the nearest thing to an undo before
+ this - overwrote them from the request, destroying the one thing they exist for."""
+ submission = DomainSubmission.objects.create(name="a.com", submitted_by=submitter)
+ ready_evidence("a.com")
+ client.post(DECISIONS_URL,
+ data={"decisions": [{"domain": "a.com", "status": "REJECTED",
+ "rejection_reason": "SPAM",
+ "suggested_status": "REJECT",
+ "suggested_reason": "SPAM",
+ "suggestion_confidence": 0.91,
+ "suggestion_model_version": "2026-08-01"}]},
+ content_type="application/json", headers={"Authorization": token(moderator)})
+
+ response = client.post(UNDO_URL.format("a.com"),
+ headers={"Authorization": token(moderator)})
+
+ assert response.status_code == 200
+ submission.refresh_from_db()
+ assert (submission.status, submission.rejection_reason) == ("PENDING", "")
+ assert (submission.suggested_status, submission.suggested_reason) == ("REJECT", "SPAM")
+ assert submission.suggestion_confidence == 0.91
+ assert submission.suggestion_model_version == "2026-08-01"
+
+ body = client.get(QUEUE_URL, headers={"Authorization": token(moderator)}).json()
+ assert [item["name"] for item in body["items"]] == ["a.com"]
+
+
+def test_undoing_an_approval_rebuilds_the_blacklist_snapshot(client, moderator, submitter):
+ """An approved domain is subtracted from the remote blocklists when the snapshot is built,
+ so it stays subtracted until the snapshot is rebuilt. .update() fires no post_save, so the
+ approval receiver never runs and the undo has to ask for the rebuild itself."""
+ DomainSubmission.objects.create(name="a.com", submitted_by=submitter, status="APPROVED")
+ # The approval itself scheduled one, and the debounce would rightly decline a second.
+ # Clearing it is the run having already happened, which is the case the undo is about.
+ Task.objects.all().delete()
+
+ with mock.patch("mwmbl.background.refresh_blacklist_snapshot") as refresh:
+ client.post(UNDO_URL.format("a.com"), headers={"Authorization": token(moderator)})
+
+ refresh.assert_called_once()
+
+
+def test_undoing_a_rejection_does_not_rebuild_the_snapshot(client, moderator, submitter):
+ """Only an approval ever changed the snapshot, so only undoing one has to change it back -
+ a rebuild downloads and parses tens of megabytes."""
+ DomainSubmission.objects.create(name="a.com", submitted_by=submitter, status="REJECTED",
+ rejection_reason="SPAM")
+ Task.objects.all().delete()
+
+ with mock.patch("mwmbl.background.refresh_blacklist_snapshot") as refresh:
+ client.post(UNDO_URL.format("a.com"), headers={"Authorization": token(moderator)})
+
+ refresh.assert_not_called()
+
+
+def test_undoing_a_domain_nobody_submitted_is_a_404(client, moderator):
+ response = client.post(UNDO_URL.format("never-submitted.com"),
+ headers={"Authorization": token(moderator)})
+ assert response.status_code == 404
+
+
+def test_history_lists_past_decisions_by_status_newest_first(client, moderator, submitter):
+ for name, status, when in [("old-reject.com", "REJECTED", 5),
+ ("new-reject.com", "REJECTED", 1),
+ ("approved.com", "APPROVED", 2)]:
+ DomainSubmission.objects.create(
+ name=name, submitted_by=submitter, status=status,
+ status_changed_by=moderator, status_changed_on=timezone.now() - timedelta(days=when))
+
+ body = client.get(f"{HISTORY_URL}?status=REJECTED",
+ headers={"Authorization": token(moderator)}).json()
+
+ assert [item["name"] for item in body["items"]] == ["new-reject.com", "old-reject.com"]
+ assert body["count"] == 2
+ assert body["items"][0]["status_changed_by_username"] == "moderator"
+
+
+def test_history_shows_one_row_per_domain(client, moderator, submitter):
+ for index in range(4):
+ DomainSubmission.objects.create(
+ name="a.com", submitted_by=submitter, status="REJECTED", rejection_reason="SPAM",
+ status_changed_by=moderator,
+ status_changed_on=timezone.now() - timedelta(days=index))
+
+ body = client.get(HISTORY_URL, headers={"Authorization": token(moderator)}).json()
+
+ assert body["count"] == 1
+ assert body["items"][0]["submission_count"] == 4
+
+
+def test_history_carries_what_the_moderator_was_shown(client, moderator, submitter):
+ DomainSubmission.objects.create(
+ name="a.com", submitted_by=submitter, status="REJECTED", rejection_reason="SPAM",
+ status_changed_by=moderator, status_changed_on=timezone.now(),
+ suggested_status="REJECT", suggested_reason="SPAM", suggestion_confidence=0.77,
+ suggestion_model_version="2026-07-01")
+
+ item = client.get(HISTORY_URL,
+ headers={"Authorization": token(moderator)}).json()["items"][0]
+
+ assert item["suggested_status"] == "REJECT"
+ assert item["suggestion_confidence"] == 0.77
+ assert item["suggestion_model_version"] == "2026-07-01"
+
+
+def test_history_requires_the_moderator_permission(client, submitter):
+ response = client.get(HISTORY_URL, headers={"Authorization": token(submitter)})
+ assert response.status_code == 403
+
+
# --------------------------------------------------------------------- rescoring
@pytest.mark.django_db
@@ -333,7 +651,7 @@ def test_enrichment_recrawls_stale_evidence(submitter):
@pytest.mark.django_db
-def test_queue_cost_does_not_grow_with_the_backlog(client, moderator, submitter,
+def test_queue_cost_does_not_grow_with_the_backlog(client, moderator, established,
django_assert_max_num_queries):
"""Ordering, filtering and slicing must happen in SQL.
@@ -344,30 +662,40 @@ def test_queue_cost_does_not_grow_with_the_backlog(client, moderator, submitter,
"""
for index in range(30):
name = f"site{index}.example"
- DomainSubmission.objects.create(name=name, submitted_by=submitter)
- ready_evidence(name, review_priority=index / 30)
+ # Several submissions and several votes per domain, so that grouping to one row per
+ # domain and rolling the votes up cannot quietly become a query per row either.
+ for _ in range(3):
+ DomainSubmission.objects.create(name=name, submitted_by=established)
+ for voter in range(index % 4):
+ SearchResultVote.objects.create(
+ user=MwmblUser.objects.create_user(username=f"voter{index}-{voter}"),
+ url=f"https://{name}/page{voter}", query="q", vote_type="upvote")
+ ready_evidence(name, suggested_action="REJECT", confidence=index / 30)
# Auth, permissions, count, page, evidence, submitter counts, prior-decision counts.
with django_assert_max_num_queries(12):
- response = client.get(f"{QUEUE_URL}?limit=5",
+ response = client.get(f"{QUEUE_URL}?limit=5&order_by=needs_review",
headers={"Authorization": token(moderator)})
body = response.json()
+ # Distinct pending domains, not the 90 pending submissions behind them.
assert body["count"] == 30
assert len(body["items"]) == 5
# ...and the page is the top of the global ordering, not the top of an arbitrary slice.
assert body["items"][0]["name"] == "site29.example"
+ assert body["items"][0]["submission_count"] == 3
+ assert body["items"][0]["upvotes"] == 1
@pytest.mark.django_db
-def test_queue_paginates_over_the_whole_ordering(client, moderator, submitter):
+def test_queue_paginates_over_the_whole_ordering(client, moderator, established):
for index in range(10):
name = f"site{index}.example"
- DomainSubmission.objects.create(name=name, submitted_by=submitter)
- ready_evidence(name, review_priority=index / 10)
+ DomainSubmission.objects.create(name=name, submitted_by=established)
+ ready_evidence(name, suggested_action="REJECT", confidence=index / 10)
def page(offset):
- response = client.get(f"{QUEUE_URL}?limit=4&offset={offset}",
+ response = client.get(f"{QUEUE_URL}?limit=4&offset={offset}&order_by=needs_review",
headers={"Authorization": token(moderator)})
return [item["name"] for item in response.json()["items"]]
@@ -508,3 +836,410 @@ def test_rejections_are_not_withheld_from_first_time_submitters(submitter):
assert suggestion.action == "REJECT"
assert suggestion.reason == "SPAM"
+
+
+# --------------------------------------------------------------------- crawl evidence
+
+class _FakeResponse:
+ """Enough of a requests Response for retrieve.fetch."""
+
+ def __init__(self, status_code, headers=None, body=b""):
+ self.status_code = status_code
+ self.headers = headers or {}
+ self.is_redirect = 300 <= status_code < 400
+ self.next = object() if self.is_redirect else None
+ self._body = body
+
+ def iter_content(self, size):
+ yield self._body
+
+ def close(self):
+ pass
+
+
+A_PAGE = (b"<html><head><title>Buy this domain</title></head><body>"
+ b"<p>This premium domain name is for sale. Make an offer today and we will be in "
+ b"touch with you shortly to complete the transfer.</p></body></html>")
+
+
+def crawl_with(responses: dict, domain: str) -> dict:
+ """Crawl ``domain`` against canned HTTP responses, through the real crawler.
+
+ Driven from requests.get rather than by stubbing crawl_url, because everything this
+ covers lives in the plumbing between the fetch and the evidence.
+ """
+ def get(url, **kwargs):
+ response = responses[url]
+ if isinstance(response, Exception):
+ raise response
+ return response
+
+ blacklist = mock.Mock(is_domain_blacklisted=mock.Mock(return_value=False))
+ with mock.patch("mwmbl.crawler.retrieve.requests.get", get), \
+ mock.patch("mwmbl.crawler.retrieve.validate_url"), \
+ mock.patch("mwmbl.crawler.retrieve.robots_allowed", return_value=True), \
+ mock.patch("mwmbl.moderation.evidence.get_snapshot_blacklist",
+ return_value=blacklist):
+ return crawl_domain(domain, redis=None)
+
+
+def test_a_domain_that_redirects_elsewhere_is_reported_as_a_redirect():
+ """Roughly a third of the rejection details moderators write are about dead or squatted
+ domains, and "redirects to somewhere else" is the fetch that proves it. The check is only
+ as good as the URL the crawler reports: reporting the requested one compares the domain
+ against itself and the rule can never fire."""
+ crawl = crawl_with({
+ "https://squatted.example/": _FakeResponse(
+ 301, headers={"Location": "https://parked.test/lander"}),
+ "https://parked.test/lander": _FakeResponse(200, body=A_PAGE),
+ }, "squatted.example")
+
+ assert crawl["final_domain"] == "parked.test"
+ decisive = rules.decisive(rules.crawl_evidence("squatted.example", crawl))
+ assert decisive.kind == "redirect"
+ assert decisive.implies_action == "REJECT"
+
+
+def test_a_domain_that_serves_its_own_pages_reports_no_redirect():
+ """The other direction, because the check is decisive: a site that answers for itself must
+ never be rejected for redirecting to itself."""
+ crawl = crawl_with(
+ {"https://honest.example/": _FakeResponse(200, body=A_PAGE)}, "honest.example")
+
+ assert crawl["final_domain"] == ""
+ kinds = {item.kind for item in rules.crawl_evidence("honest.example", crawl)}
+ assert "redirect" not in kinds
+
+
+def test_a_site_without_a_certificate_is_a_site_not_a_dead_domain():
+ """Only https used to be tried, so "no TLS" and "not there" were the same result - and
+ that result is a decisive REJECT for being unreachable. A site served over plain http is
+ a site; whether it has a certificate is a line for the moderator, not a verdict."""
+ crawl = crawl_with({
+ "https://no-cert.example/": ConnectionError("certificate verify failed"),
+ "http://no-cert.example/": _FakeResponse(200, body=A_PAGE),
+ }, "no-cert.example")
+
+ assert crawl["signals"]["https"] is False
+ assert crawl["error"] == ""
+ items = {item.kind: item for item in rules.crawl_evidence("no-cert.example", crawl)}
+ assert items["no_tls"].direction == rules.NEUTRAL
+ assert rules.decisive(items.values()) is None
+
+
+def test_a_domain_that_is_simply_down_is_still_reported_as_unreachable():
+ """The other direction: falling back to http must not turn a dead domain into a live one,
+ and must not label it "no TLS" - the https attempt is the one that describes production."""
+ crawl = crawl_with({
+ "https://gone.example/": ConnectionError("no route to host"),
+ "http://gone.example/": ConnectionError("no route to host"),
+ }, "gone.example")
+
+ assert crawl["signals"]["https"] is False
+ assert crawl["error"] == "AbortError"
+ kinds = {item.kind for item in rules.crawl_evidence("gone.example", crawl)}
+ assert "no_tls" not in kinds
+ assert rules.decisive(rules.crawl_evidence("gone.example", crawl)).kind == "unreachable"
+
+
+def test_a_normal_https_site_is_not_asked_for_over_http():
+ crawl = crawl_with(
+ {"https://secure.example/": _FakeResponse(200, body=A_PAGE)}, "secure.example")
+
+ assert crawl["signals"]["https"] is True
+ assert "no_tls" not in {item.kind
+ for item in rules.crawl_evidence("secure.example", crawl)}
+
+
+# --------------------------------------------------------------------- enrichment
+
+def test_enrichment_never_commits_a_scored_row_without_the_score():
+ """READY with a fresh timestamp is what makes the task skip a domain for a month. A row
+ that reached it without a suggestion - a worker restarted between the two writes - would
+ tell moderators "not assessed yet" until the evidence expired."""
+ from mwmbl.background import enrich_domain_submission
+
+ crawl = {"http_status": 200, "final_domain": "", "error": "", "pages": [], "signals": {}}
+ with mock.patch("mwmbl.background.crawl_domain", return_value=crawl), \
+ mock.patch("mwmbl.background.refresh_suggestion",
+ side_effect=RuntimeError("scoring blew up")):
+ with pytest.raises(RuntimeError):
+ enrich_domain_submission.now("half-written.example")
+
+ assert not DomainEvidence.objects.filter(domain="half-written.example").exists()
+
+
+def test_a_failed_recrawl_does_not_leave_the_old_crawl_on_screen():
+ """The detail panel renders whatever is on the row, so a month-old crawl next to a FAILED
+ state reads as current evidence about a site that is in fact unreachable."""
+ from mwmbl.background import enrich_domain_submission
+
+ ready_evidence("gone.example", fetched_at=timezone.now() - timedelta(days=365))
+ with mock.patch("mwmbl.background.crawl_domain",
+ side_effect=ConnectionError("no route to host")):
+ enrich_domain_submission.now("gone.example")
+
+ evidence = DomainEvidence.objects.get(domain="gone.example")
+ assert evidence.state == DomainEvidence.State.FAILED
+ assert evidence.error == "ConnectionError"
+ assert evidence.pages == []
+ assert evidence.suggested_action == ""
+ assert evidence.confidence is None
+
+
+def test_the_reason_is_shown_with_its_own_confidence(submitter):
+ """A model can be sure a domain should go and barely have a view on why. Showing the
+ rejection's confidence next to the reason presents a guess as a finding."""
+ submission = DomainSubmission.objects.create(name="spam.example", submitted_by=submitter)
+ model = mock.Mock(version="test-model")
+ model.predict.return_value = [(0.92, "SPAM", 0.31)]
+
+ with mock.patch("mwmbl.moderation.model.get_model", return_value=model):
+ refresh_suggestion(ready_evidence("spam.example"))
+
+ evidence = DomainEvidence.objects.get(domain="spam.example")
+ assert evidence.confidence == pytest.approx(0.92)
+ assert evidence.reason_confidence == pytest.approx(0.31)
+
+ suggestion = suggestion_for(submission, evidence)
+ assert suggestion.reason == "SPAM"
+ assert suggestion.reason_confidence == pytest.approx(0.31)
+ assert suggestion.reason_source == "model"
+
+
+# --------------------------------------------------------------------- queue parity
+
+def test_queue_display_matches_suggestion_for(submitter, established):
+ """The queue filters and orders in SQL; the rows are rendered by suggestion_for. Where the
+ two disagree, a filter hides rows that are on screen and the ordering sorts on a number
+ nobody is shown - so every branch of the adjustment is checked against the other."""
+ DomainSubmission.objects.create(name="approved-before.example",
+ submitted_by=established, status="APPROVED")
+ DomainSubmission.objects.create(name="rejected-before.example",
+ submitted_by=established, status="REJECTED")
+ DomainSubmission.objects.create(name="dead-and-approved-before.example",
+ submitted_by=established, status="APPROVED")
+
+ a_404 = rules.EvidenceItem(
+ "http_status", rules.REJECT, "Homepage returns HTTP 404", implies_action="REJECT",
+ implies_reason="OTHER", implies_confidence=0.9).to_dict()
+
+ cases = [
+ # (domain, submitter, evidence overrides) - one per branch of the adjustment.
+ ("first-timer-approve.example", submitter, {}),
+ ("known-approve.example", established, {}),
+ ("reject.example", submitter, {"suggested_action": "REJECT",
+ "suggested_reason": "SPAM", "confidence": 0.91}),
+ ("unsure.example", submitter, {"suggested_action": "UNSURE", "confidence": 0.1}),
+ ("approved-before.example", submitter, {"suggested_action": "REJECT",
+ "suggested_reason": "SPAM"}),
+ ("rejected-before.example", submitter, {}),
+ # A check that already decided it, at least as strongly as the prior decision does.
+ ("dead-and-approved-before.example", submitter,
+ {"suggested_action": "REJECT", "suggested_reason": "OTHER", "confidence": 0.9,
+ "reason_source": "rule", "evidence": [a_404]}),
+ ("still-crawling.example", submitter, None),
+ ]
+ for domain, user, overrides in cases:
+ DomainSubmission.objects.create(name=domain, submitted_by=user)
+ if overrides is None:
+ DomainEvidence.objects.create(domain=domain, state=DomainEvidence.State.PENDING)
+ else:
+ ready_evidence(domain, **overrides)
+
+ for submission in DomainSubmission.objects.filter(status="PENDING"):
+ evidence = DomainEvidence.objects.filter(domain=submission.name).first()
+ shown = suggestion_for(submission, evidence)
+ row = annotate_queue(DomainSubmission.objects.filter(pk=submission.pk)).get()
+
+ if shown is None:
+ assert row.displayed_action is None, submission.name
+ assert row.displayed_priority is None, submission.name
+ continue
+ assert row.displayed_action == shown.action, submission.name
+ assert row.displayed_reason == shown.reason, submission.name
+ assert row.displayed_confidence == pytest.approx(shown.confidence), submission.name
+ assert row.displayed_priority == pytest.approx(shown.review_priority), submission.name
+
+
+def test_queue_filters_find_a_withheld_approval_where_it_is_shown(client, moderator, submitter):
+ """A moderator filtering for UNSURE is asking for the rows only a human can settle, which
+ is exactly what a withheld approval is."""
+ DomainSubmission.objects.create(name="withheld.example", submitted_by=submitter)
+ ready_evidence("withheld.example", suggested_action="APPROVE", confidence=0.9)
+
+ response = client.get(f"{QUEUE_URL}?suggested_action=UNSURE",
+ headers={"Authorization": token(moderator)})
+ items = response.json()["items"]
+
+ assert [item["name"] for item in items] == ["withheld.example"]
+ assert items[0]["suggestion"]["action"] == "UNSURE"
+
+
+def test_a_withheld_approval_outranks_a_confident_one(client, moderator, submitter,
+ established):
+ """It is the 44%-wrong slice: it needs a human more than an approval we trust does."""
+ DomainSubmission.objects.create(name="withheld.example", submitted_by=submitter)
+ ready_evidence("withheld.example", suggested_action="APPROVE", confidence=0.95)
+ DomainSubmission.objects.create(name="trusted.example", submitted_by=established)
+ ready_evidence("trusted.example", suggested_action="APPROVE", confidence=0.95)
+
+ response = client.get(QUEUE_URL, headers={"Authorization": token(moderator)})
+ assert [item["name"] for item in response.json()["items"]] == [
+ "withheld.example", "trusted.example"]
+
+
+# --------------------------------------------------------------------- the artifact
+
+class _StubModel:
+ """A stand-in for ModerationModel. joblib only needs it to pickle and carry a version."""
+
+ def __init__(self, version):
+ self.version = version
+
+ def predict(self, examples):
+ return [(0.5, "SPAM", 0.5) for _ in examples]
+
+
+def artifact_bytes(model) -> bytes:
+ buffer = io.BytesIO()
+ joblib.dump(model, buffer)
+ return buffer.getvalue()
+
+
+def test_a_published_model_outlives_the_process_that_trained_it():
+ """It used to be written into the source tree, so a deploy reverted every retrain and each
+ worker replica kept its own copy."""
+ publish(_StubModel("domain-mod-2026-09-01"), {"cold_start": {"pr_auc": 0.81}})
+
+ reset_model_cache() # a different worker, loading for the first time
+ assert get_model().version == "domain-mod-2026-09-01"
+ assert load_metrics()["cold_start"]["pr_auc"] == 0.81
+
+
+def test_a_worker_picks_up_a_retrain_it_did_not_run(monkeypatch):
+ """The retrain runs in one process and every process serves suggestions."""
+ publish(_StubModel("domain-mod-2026-09-01"), {})
+ assert get_model().version == "domain-mod-2026-09-01"
+
+ monkeypatch.setattr(model_module, "MODEL_REFRESH_SECONDS", 0)
+ ModerationModelArtifact.objects.create(
+ version="domain-mod-2026-10-01", metrics={},
+ model=artifact_bytes(_StubModel("domain-mod-2026-10-01")))
+
+ assert get_model().version == "domain-mod-2026-10-01"
+
+
+def test_the_bundled_artifact_is_the_warm_start_until_a_retrain_publishes(tmp_path, settings):
+ """A deploy against a database with no artifact in it still suggests something."""
+ joblib.dump(_StubModel("bundled"), tmp_path / "model.joblib")
+ (tmp_path / "metrics.json").write_text(json.dumps({"cold_start": {"pr_auc": 0.7}}))
+ settings.DOMAIN_MODERATION_MODEL_DIR = str(tmp_path)
+
+ assert get_model().version == "bundled"
+ assert load_metrics()["cold_start"]["pr_auc"] == 0.7
+
+ publish(_StubModel("domain-mod-2026-09-01"), {"cold_start": {"pr_auc": 0.81}})
+ # Once something is published the gate compares against that, not against the shipped
+ # numbers, which describe a model nobody is serving any more.
+ assert load_metrics()["cold_start"]["pr_auc"] == 0.81
+
+
+def test_an_unloadable_artifact_falls_back_rather_than_failing(tmp_path, settings):
+ """A scikit-learn upgrade can make an old pickle unloadable, and the enrichment task must
+ not go down with it."""
+ joblib.dump(_StubModel("bundled"), tmp_path / "model.joblib")
+ settings.DOMAIN_MODERATION_MODEL_DIR = str(tmp_path)
+ ModerationModelArtifact.objects.create(
+ version="corrupt", model=b"not a pickle", metrics={})
+
+ assert get_model().version == "bundled"
+
+
+# --------------------------------------------------------------------- request validation
+
+def test_an_over_long_audit_field_is_a_validation_error_not_a_database_one(
+ client, moderator, submitter):
+ """Django does not enforce max_length on save(), so without the schema constraint this is
+ a Postgres DataError and a 500 rather than a 400 naming the field."""
+ submission = DomainSubmission.objects.create(name="example.com", submitted_by=submitter)
+
+ response = client.post(
+ f"/api/v1/platform/domain-submissions/ids/{submission.id}",
+ data={"status": "APPROVED", "rejection_reason": "", "rejection_detail": "",
+ "suggestion_model_version": "v" * 100},
+ content_type="application/json", headers={"Authorization": token(moderator)})
+
+ assert response.status_code == 422
+ submission.refresh_from_db()
+ assert submission.status == "PENDING"
+
+
+def test_other_without_a_detail_is_refused(client, moderator, submitter):
+ """The dialog labels the fourth reason "Other - needs detail", and the detail is the only
+ place a rejected submitter finds out what was actually wrong."""
+ DomainSubmission.objects.create(name="example.com", submitted_by=submitter)
+
+ response = client.post(
+ DECISIONS_URL,
+ data={"decisions": [{"domain": "example.com", "status": "REJECTED",
+ "rejection_reason": "OTHER", "rejection_detail": " "}]},
+ content_type="application/json", headers={"Authorization": token(moderator)})
+
+ assert response.status_code == 422
+ assert DomainSubmission.objects.get(name="example.com").status == "PENDING"
+
+
+def test_a_rejection_reason_outside_the_four_choices_is_refused(client, moderator, submitter):
+ """Django does not enforce choices on save(), so any string of twenty characters or fewer
+ lands in the column and comes back out at a moderator as a reason nothing can render."""
+ DomainSubmission.objects.create(name="example.com", submitted_by=submitter)
+
+ response = client.post(
+ DECISIONS_URL,
+ data={"decisions": [{"domain": "example.com", "status": "REJECTED",
+ "rejection_reason": "BECAUSE"}]},
+ content_type="application/json", headers={"Authorization": token(moderator)})
+
+ assert response.status_code == 422
+
+
+def test_an_approval_cannot_carry_a_rejection_reason(client, moderator, submitter):
+ DomainSubmission.objects.create(name="example.com", submitted_by=submitter)
+
+ response = client.post(
+ DECISIONS_URL,
+ data={"decisions": [{"domain": "example.com", "status": "APPROVED",
+ "rejection_reason": "SPAM"}]},
+ content_type="application/json", headers={"Authorization": token(moderator)})
+
+ assert response.status_code == 422
+
+
+def test_other_with_a_detail_is_accepted(client, moderator, submitter):
+ DomainSubmission.objects.create(name="example.com", submitted_by=submitter)
+
+ response = client.post(
+ DECISIONS_URL,
+ data={"decisions": [{"domain": "example.com", "status": "REJECTED",
+ "rejection_reason": "OTHER",
+ "rejection_detail": "Mirrors content we already index."}]},
+ content_type="application/json", headers={"Authorization": token(moderator)})
+
+ assert response.status_code == 200
+ assert DomainSubmission.objects.get(name="example.com").status == "REJECTED"
diff --git a/test/test_retrieve_extract.py b/test/test_retrieve_extract.py
index f0ad9d7..d075a67 100644
--- a/test/test_retrieve_extract.py
+++ b/test/test_retrieve_extract.py
@@ -61,7 +61,7 @@ def test_get_first_paragraph_none_when_no_body_prose():
def _crawl_html(monkeypatch, html: str) -> dict:
monkeypatch.setattr(retrieve, "robots_allowed", lambda url, redis: True)
- monkeypatch.setattr(retrieve, "fetch", lambda url: (200, html.encode("utf8")))
+ monkeypatch.setattr(retrieve, "fetch", lambda url: (200, html.encode("utf8"), url))
return retrieve.crawl_url("https://example.test/page", fakeredis.FakeStrictRedis())
diff --git a/test/test_ssrf.py b/test/test_ssrf.py
index d353db0..ac4b31e 100644
--- a/test/test_ssrf.py
+++ b/test/test_ssrf.py
@@ -124,9 +124,25 @@ def test_fetch_blocks_redirect_to_internal(monkeypatch):
def test_fetch_returns_content_for_public_url(monkeypatch):
monkeypatch.setattr(retrieve.requests, "get",
lambda *a, **k: _FakeResponse(200, body=b"page body"))
- status, content = retrieve.fetch("http://93.184.216.34/page")
+ status, content, resolved_url = retrieve.fetch("http://93.184.216.34/page")
assert status == 200
assert content == b"page body"
+ assert resolved_url == "http://93.184.216.34/page"
+
+
+def test_fetch_reports_the_host_that_answered_not_the_one_we_asked(monkeypatch):
+ """The last hop is the only place a redirect is visible to a caller. Moderation rejects
+ domains that redirect elsewhere, and cannot see one if fetch reports the requested URL."""
+ def fake_get(url, **kwargs):
+ if url == "http://93.184.216.34/start":
+ return _FakeResponse(301, headers={"Location": "http://93.184.216.35/parked"},
+ is_redirect=True)
+ return _FakeResponse(200, body=b"parking page")
+
+ monkeypatch.setattr(retrieve.requests, "get", fake_get)
+ status, _, resolved_url = retrieve.fetch("http://93.184.216.34/start")
+ assert status == 200
+ assert resolved_url == "http://93.184.216.35/parked"
# ---------------------------------------------------------------------------