Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Correct the notes the first day of data contradicted

Daoud Clarke8d4ad91785bf
diff --git a/mwmbl/templates/admin/search_traffic.html b/mwmbl/templates/admin/search_traffic.html
index c71026a..4259aca 100644
--- a/mwmbl/templates/admin/search_traffic.html
+++ b/mwmbl/templates/admin/search_traffic.html
@@ -65,9 +65,14 @@
{% include "admin/_traffic_table.html" with table=by_headers column_header="Endpoint" %}
<p class="note">
Which of <code>User-Agent</code> and <code>Accept-Language</code> arrived, and nothing
- inferred from it. <code>neither</code> is what our own server-side render sends, so on
- <code>v1_search</code> it is mostly mwmbl.org rather than anything exotic; a missing
- <code>Accept-Language</code> is a hint that a client is not a browser, and only a hint.
+ inferred from it. A missing <code>Accept-Language</code> is a hint that a client is not
+ a browser, and only a hint. <code>neither</code> has stayed empty: the SvelteKit front
+ end's server-side render calls the API through Node's <code>fetch</code>, which sends
+ <code>User-Agent: node</code> and <code>Accept-Language: *</code>, so it lands in
+ <code>user_agent+language</code> alongside every real browser. The front end's other
+ path does not even do that - <code>handleApiProxy</code> forwards the visitor's own
+ headers verbatim - so our own traffic arrives under two different identities depending
+ on which code path made the call, and neither of them is distinguishable here.
</p>
</div>
@@ -75,8 +80,13 @@
<h2>Requests by status class</h2>
{% include "admin/_traffic_table.html" with table=by_status column_header="Endpoint" %}
<p class="note">
- What it cost us, roughly: a 4xx bought a URL match and nothing else, a 2xx bought a
- ranker run. A client sending queries it never gets an answer to lives in the 4xx column.
+ What it cost us, roughly: a 4xx bought a URL match and nothing else. A 2xx on
+ <code>*_search</code> bought a ranker run - retrieval for every term, completion and
+ bigram, then the model. A 2xx on <code>*_raw</code> did not: <code>get_raw_results</code>
+ is one <code>retrieve</code> of the whole query as a single term plus the blacklist
+ filter, so those are the cheapest requests in the table by a wide margin, and reading
+ this column as uniform cost will overstate them. A client sending queries it never gets
+ an answer to lives in the 4xx column.
</p>
</div>
@@ -91,9 +101,13 @@
<span class="{% if user_agents.gap %}bad{% endif %}">{{ user_agents.gap|intcomma }}</span>.
</p>
<p class="note">
- The gap is the tail the trim dropped, and the one number here that reports on
- something otherwise invisible: a client randomising its user agent shows up as a
- growing gap and in no other way.
+ The gap is the tail the trim dropped. It catches a client that randomises its user
+ agent widely enough to overflow the trim, and nothing narrower: a rotation over a
+ small fixed pool never reaches the limit, so the gap stays at zero while the rotation
+ runs. SearxNG is the worked example - <code>gen_useragent()</code> builds every
+ request's <code>User-Agent</code> from two OS strings and a handful of Firefox
+ versions, which is a pool of a few dozen strings and no gap at all. A gap of zero
+ means nobody overflowed the trim; it does not mean nobody is rotating.
</p>
<h3>Top {{ top_user_agents }} by requests</h3>
@@ -117,8 +131,9 @@
</table>
<p class="note">
Summed across the window, so a client that comes back every day outranks one that
- came once. <code>(none)</code> sent no <code>User-Agent</code> at all, which is
- what our own front end does. Strings are truncated at 512 characters on the way in.
+ came once. <code>(none)</code> sent no <code>User-Agent</code> at all - nothing we run
+ does that, so a row here is worth looking at rather than explaining away. Strings are
+ truncated at 512 characters on the way in.
</p>
{% else %}
<p class="note">No user agents recorded in this window.</p>
diff --git a/mwmbl/traffic.py b/mwmbl/traffic.py
index 1f4cc7f..23df148 100644
--- a/mwmbl/traffic.py
+++ b/mwmbl/traffic.py
@@ -87,9 +87,12 @@ def endpoint_label(request: HttpRequest) -> Optional[str]:
# The one split on the request counter, and a description rather than a judgement: which of
-# the two headers arrived, not what that means. A request with neither is what our own
-# server-side render sends today, and a missing Accept-Language is the cheapest hint that a
-# client is not a browser, since browsers send one and HTTP clients do not.
+# the two headers arrived, not what that means. A missing Accept-Language is the cheapest hint
+# that a client is not a browser, since browsers send one and HTTP clients do not. Note what it
+# cannot see: SearxNG sets Accept-Language (send_accept_language_header defaults to True) and a
+# browser User-Agent, and our own SvelteKit render reaches us through Node's fetch as
+# User-Agent: node with Accept-Language: *. Both land in user_agent+language, which is where
+# real browsers land too. The split separates clients that do not try from everything else.
HEADER_LABELS = ["user_agent+language", "user_agent", "language", "neither"]
@@ -102,11 +105,16 @@ def header_label(request: HttpRequest) -> str:
# The status class the client got, and like the header split a description rather than a
-# judgement. A request that 4xx'd cost us a URL match and nothing else; one that 2xx'd cost
-# us a ranker run. Without this dimension a bot sending malformed queries that never reach
-# the index is indistinguishable from a visitor being served, which is the whole question
-# #410 is asking. The class rather than the code, because the codes are the view's business
-# and this counter should not have an opinion about how many of them there are.
+# judgement. A request that 4xx'd cost us a URL match and nothing else. What a 2xx cost
+# depends on the endpoint and this dimension cannot tell them apart: /search retrieves for
+# every term, completion and bigram and then runs the model, while /raw is a single retrieve
+# of the whole query as one term (rank.get_raw_results). Reading a 2xx as one fixed price
+# overstates /raw, which is most of what the crawler's index sync does.
+#
+# Without this dimension a bot sending malformed queries that never reach the index is
+# indistinguishable from a visitor being served, which is the whole question #410 is asking.
+# The class rather than the code, because the codes are the view's business and this counter
+# should not have an opinion about how many of them there are.
STATUS_LABELS = ["1xx", "2xx", "3xx", "4xx", "5xx"]