An open source, non-profit web search engine
Correct the notes the first day of data contradicted
Daoud Clarke8d4ad91785bf
diff --git a/mwmbl/templates/admin/search_traffic.html b/mwmbl/templates/admin/search_traffic.htmlindex c71026a..4259aca 100644--- a/mwmbl/templates/admin/search_traffic.html+++ b/mwmbl/templates/admin/search_traffic.html@@ -65,9 +65,14 @@{% include "admin/_traffic_table.html" with table=by_headers column_header="Endpoint" %}<p class="note">Which of <code>User-Agent</code> and <code>Accept-Language</code> arrived, and nothing- inferred from it. <code>neither</code> is what our own server-side render sends, so on- <code>v1_search</code> it is mostly mwmbl.org rather than anything exotic; a missing- <code>Accept-Language</code> is a hint that a client is not a browser, and only a hint.+ inferred from it. A missing <code>Accept-Language</code> is a hint that a client is not+ a browser, and only a hint. <code>neither</code> has stayed empty: the SvelteKit front+ end's server-side render calls the API through Node's <code>fetch</code>, which sends+ <code>User-Agent: node</code> and <code>Accept-Language: *</code>, so it lands in+ <code>user_agent+language</code> alongside every real browser. The front end's other+ path does not even do that - <code>handleApiProxy</code> forwards the visitor's own+ headers verbatim - so our own traffic arrives under two different identities depending+ on which code path made the call, and neither of them is distinguishable here.</p></div>@@ -75,8 +80,13 @@<h2>Requests by status class</h2>{% include "admin/_traffic_table.html" with table=by_status column_header="Endpoint" %}<p class="note">- What it cost us, roughly: a 4xx bought a URL match and nothing else, a 2xx bought a- ranker run. A client sending queries it never gets an answer to lives in the 4xx column.+ What it cost us, roughly: a 4xx bought a URL match and nothing else. A 2xx on+ <code>*_search</code> bought a ranker run - retrieval for every term, completion and+ bigram, then the model. A 2xx on <code>*_raw</code> did not: <code>get_raw_results</code>+ is one <code>retrieve</code> of the whole query as a single term plus the blacklist+ filter, so those are the cheapest requests in the table by a wide margin, and reading+ this column as uniform cost will overstate them. A client sending queries it never gets+ an answer to lives in the 4xx column.</p></div>@@ -91,9 +101,13 @@<span class="{% if user_agents.gap %}bad{% endif %}">{{ user_agents.gap|intcomma }}</span>.</p><p class="note">- The gap is the tail the trim dropped, and the one number here that reports on- something otherwise invisible: a client randomising its user agent shows up as a- growing gap and in no other way.+ The gap is the tail the trim dropped. It catches a client that randomises its user+ agent widely enough to overflow the trim, and nothing narrower: a rotation over a+ small fixed pool never reaches the limit, so the gap stays at zero while the rotation+ runs. SearxNG is the worked example - <code>gen_useragent()</code> builds every+ request's <code>User-Agent</code> from two OS strings and a handful of Firefox+ versions, which is a pool of a few dozen strings and no gap at all. A gap of zero+ means nobody overflowed the trim; it does not mean nobody is rotating.</p><h3>Top {{ top_user_agents }} by requests</h3>@@ -117,8 +131,9 @@</table><p class="note">Summed across the window, so a client that comes back every day outranks one that- came once. <code>(none)</code> sent no <code>User-Agent</code> at all, which is- what our own front end does. Strings are truncated at 512 characters on the way in.+ came once. <code>(none)</code> sent no <code>User-Agent</code> at all - nothing we run+ does that, so a row here is worth looking at rather than explaining away. Strings are+ truncated at 512 characters on the way in.</p>{% else %}<p class="note">No user agents recorded in this window.</p>diff --git a/mwmbl/traffic.py b/mwmbl/traffic.pyindex 1f4cc7f..23df148 100644--- a/mwmbl/traffic.py+++ b/mwmbl/traffic.py@@ -87,9 +87,12 @@ def endpoint_label(request: HttpRequest) -> Optional[str]:# The one split on the request counter, and a description rather than a judgement: which of-# the two headers arrived, not what that means. A request with neither is what our own-# server-side render sends today, and a missing Accept-Language is the cheapest hint that a-# client is not a browser, since browsers send one and HTTP clients do not.+# the two headers arrived, not what that means. A missing Accept-Language is the cheapest hint+# that a client is not a browser, since browsers send one and HTTP clients do not. Note what it+# cannot see: SearxNG sets Accept-Language (send_accept_language_header defaults to True) and a+# browser User-Agent, and our own SvelteKit render reaches us through Node's fetch as+# User-Agent: node with Accept-Language: *. Both land in user_agent+language, which is where+# real browsers land too. The split separates clients that do not try from everything else.HEADER_LABELS = ["user_agent+language", "user_agent", "language", "neither"]@@ -102,11 +105,16 @@ def header_label(request: HttpRequest) -> str:# The status class the client got, and like the header split a description rather than a-# judgement. A request that 4xx'd cost us a URL match and nothing else; one that 2xx'd cost-# us a ranker run. Without this dimension a bot sending malformed queries that never reach-# the index is indistinguishable from a visitor being served, which is the whole question-# #410 is asking. The class rather than the code, because the codes are the view's business-# and this counter should not have an opinion about how many of them there are.+# judgement. A request that 4xx'd cost us a URL match and nothing else. What a 2xx cost+# depends on the endpoint and this dimension cannot tell them apart: /search retrieves for+# every term, completion and bigram and then runs the model, while /raw is a single retrieve+# of the whole query as one term (rank.get_raw_results). Reading a 2xx as one fixed price+# overstates /raw, which is most of what the crawler's index sync does.+#+# Without this dimension a bot sending malformed queries that never reach the index is+# indistinguishable from a visitor being served, which is the whole question #410 is asking.+# The class rather than the code, because the codes are the view's business and this counter+# should not have an opinion about how many of them there are.STATUS_LABELS = ["1xx", "2xx", "3xx", "4xx", "5xx"]