Skip to content

Open-source agentic AI Search Execution Platform. Track AI visibility, find opportunities, and let agents turn signals into prioritized actions and drafted fixes across ChatGPT, Claude, Gemini, Google AI Overviews, Google AI Mode, Perplexity, Grok, and Microsoft Copilot.

fix(traffic): use a bot's range list while it is read again (#1254)

Cihan27e70550cb0b
diff --git a/server/src/lib/ai-bots.js b/server/src/lib/ai-bots.js
index 72f387ca7..3fb1887eb 100644
--- a/server/src/lib/ai-bots.js
+++ b/server/src/lib/ai-bots.js
@@ -12,8 +12,8 @@
* The ranges are fetched on first use and again once a day, and kept in
* memory. A failed refresh keeps the last list; with none ever read, the
* bot's requests can't be verified and aren't counted, and the list isn't
- * tried again for a few minutes. Requests that need a list being read wait
- * for that one read.
+ * tried again for a few minutes. A list past its day is still used while it
+ * is read again; only requests with no list to use wait, for that one read.
*
* The list is versioned like ai-sources.json: each change adds an entry to
* `changes` and bumps `version`.
@@ -132,7 +132,9 @@ async function readRanges(url, hit, fetchImpl, now) {
/**
* A range file's prefixes: from memory while fresh, read again after a day.
* A failed read keeps what was read before; null when nothing ever was, and
- * then until RANGES_RETRY_MS has passed. One read at a time per file.
+ * then until RANGES_RETRY_MS has passed. One read at a time per file, and a
+ * request with a list past its day uses it rather than waiting for the read
+ * (gkhngyk on #1251).
*/
export async function rangesFor(url, { fetchImpl = fetch, now = Date.now() } = {}) {
const hit = cache.get(url);
@@ -142,7 +144,7 @@ export async function rangesFor(url, { fetchImpl = fetch, now = Date.now() } = {
read = readRanges(url, hit, fetchImpl, now).finally(() => pending.delete(url));
pending.set(url, read);
}
- return read;
+ return hit?.prefixes ?? read;
}
/** Forgets every range list (tests). */
diff --git a/server/src/lib/ai-bots.test.js b/server/src/lib/ai-bots.test.js
index 0a55ddd85..4fdcca6fe 100644
--- a/server/src/lib/ai-bots.test.js
+++ b/server/src/lib/ai-bots.test.js
@@ -101,6 +101,33 @@ describe('rangesFor', () => {
expect(fetchImpl).toHaveBeenCalledTimes(2);
});
+ // gkhngyk on #1251: once a day, requests waited for the refresh although
+ // a usable list was still in memory.
+ it('uses the list it has while it reads it again', async () => {
+ const old = json({ prefixes: [{ ipv4Prefix: '1.2.3.0/24' }] });
+ let answer;
+ const fetchImpl = vi
+ .fn()
+ .mockResolvedValueOnce(old)
+ .mockImplementationOnce(() => new Promise((resolve) => (answer = resolve)));
+ const first = await rangesFor('https://x/e.json', { fetchImpl, now: 0 });
+
+ // Past its day: the old list at once, with one read under way.
+ const during = await rangesFor('https://x/e.json', { fetchImpl, now: RANGES_TTL_MS + 1 });
+ const again = await rangesFor('https://x/e.json', { fetchImpl, now: RANGES_TTL_MS + 2 });
+ expect(during).toBe(first);
+ expect(again).toBe(first);
+ expect(fetchImpl).toHaveBeenCalledTimes(2);
+
+ // Once read, the new list.
+ answer(json({ prefixes: [{ ipv4Prefix: '5.6.7.0/24' }, { ipv4Prefix: '8.8.8.0/24' }] }));
+ await vi.waitFor(async () => {
+ const fresh = await rangesFor('https://x/e.json', { fetchImpl, now: RANGES_TTL_MS + 3 });
+ expect(fresh).toHaveLength(2);
+ });
+ expect(fetchImpl).toHaveBeenCalledTimes(2);
+ });
+
it('shares one read among the requests waiting for it', async () => {
let answer;
const fetchImpl = vi.fn(() => new Promise((resolve) => (answer = resolve)));