Skip to content

Open-source agentic AI Search Execution Platform. Track AI visibility, find opportunities, and let agents turn signals into prioritized actions and drafted fixes across ChatGPT, Claude, Gemini, Google AI Overviews, Google AI Mode, Perplexity, Grok, and Microsoft Copilot.

fix(citations): resolve URLs through the request body instead of the query string (#741)

Gökhan Geyik8ccf5a288249
diff --git a/server/src/lib/citation-rows.js b/server/src/lib/citation-rows.js
index 1cf161426..cdceb229c 100644
--- a/server/src/lib/citation-rows.js
+++ b/server/src/lib/citation-rows.js
@@ -73,82 +73,34 @@ export function normalizeCitations(rawCitations) {
return out;
}
-/**
- * URLs looked up per request.
- *
- * A PostgREST `.in()` filter is spelled out in the query string, so the whole
- * list travels in the request line — which has a length limit that a plain
- * insert does not. The first version sent every URL of an answer at once and
- * silently lost the answers that most needed storing: 228 of 174,466 in the
- * first production backfill, averaging 45 citations and 16 KB of URL text
- * each, against 12 citations and 1.2 KB for the ones that succeeded. The
- * failure surfaced as a logged error and a skipped answer, not as a crash.
- *
- * Sized so that even pathological URLs stay well inside the limit: 100 URLs
- * at the observed 2,048-character ceiling is roughly 200 KB, and the longest
- * single answer seen carries 191 citations.
- */
-const URL_LOOKUP_CHUNK = 100;
-
-function chunk(items, size) {
- const out = [];
- for (let i = 0; i < items.length; i += size) out.push(items.slice(i, i + size));
- return out;
-}
-
-/** `select id, url where url in (...)`, split across as many requests as it takes. */
-async function selectUrlIds(urls, label) {
- const ids = new Map();
- for (const batch of chunk(urls, URL_LOOKUP_CHUNK)) {
- const { data, error } = await supabaseAdmin
- .from('citation_urls')
- .select('id, url')
- .in('url', batch);
- if (error) throw new Error(`citation url ${label}: ${error.message}`);
- for (const row of data ?? []) ids.set(row.url, row.id);
- }
- return ids;
-}
-
/**
* Resolve URLs to dictionary ids, inserting the ones not seen before.
*
- * Two steps rather than one upsert-returning: `ignoreDuplicates` does not
- * return the rows it skipped, so an upsert alone would leave the already-known
- * URLs — the overwhelming majority — without ids. Insert only what is missing,
- * then read back the whole set.
+ * One call to `citation_url_ids`, which takes the list as a jsonb argument in
+ * the request body. The previous version filtered with `.in('url', ...)`,
+ * which PostgREST spells out in the query string: URLs are percent-encoded on
+ * the way into a URI, so even a hundred of them overflowed the request-line
+ * limits in front of the API and failed with `Bad Request`, or reset the
+ * connection outright. It failed on exactly the answers carrying the most
+ * citations — 182 of 174,466 survived a chunked version of the same mistake.
*
- * The re-read after insert is not redundant. Two runs can insert the same new
- * URL concurrently; the loser's insert is ignored rather than failing, and the
- * final select is what gives it the winner's id.
+ * Nothing here needs chunking any more, and there is no lookup-then-insert
+ * race to re-read around: the function inserts what is new and returns ids for
+ * everything asked about, including rows a concurrent writer won.
*/
async function resolveUrlIds(entries) {
const byUrl = new Map();
for (const entry of entries) {
if (!byUrl.has(entry.url)) byUrl.set(entry.url, entry);
}
- const urls = [...byUrl.keys()];
- if (urls.length === 0) return new Map();
-
- const ids = await selectUrlIds(urls, 'lookup');
- const missing = urls.filter((url) => !ids.has(url));
+ if (byUrl.size === 0) return new Map();
- if (missing.length > 0) {
- // The insert itself carries its payload in the body, not the query string,
- // so it needs no chunking — only the `.in()` reads around it do.
- const { error: insertErr } = await supabaseAdmin.from('citation_urls').upsert(
- missing.map((url) => {
- const entry = byUrl.get(url);
- return { url, domain: entry.domain, title: entry.title };
- }),
- { onConflict: 'url', ignoreDuplicates: true },
- );
- if (insertErr) throw new Error(`citation url insert: ${insertErr.message}`);
-
- for (const [url, id] of await selectUrlIds(missing, 're-read')) ids.set(url, id);
- }
+ const { data, error } = await supabaseAdmin.rpc('citation_url_ids', {
+ p_urls: [...byUrl.values()].map(({ url, domain, title }) => ({ url, domain, title })),
+ });
+ if (error) throw new Error(`citation url resolve: ${error.message}`);
- return ids;
+ return new Map((data ?? []).map((row) => [row.url, row.id]));
}
/**
diff --git a/server/src/lib/citation-rows.test.js b/server/src/lib/citation-rows.test.js
index 5b192c1d0..be5897f10 100644
--- a/server/src/lib/citation-rows.test.js
+++ b/server/src/lib/citation-rows.test.js
@@ -110,57 +110,38 @@ describe('normalizeCitations', () => {
});
/**
- * URL lookups are chunked because a PostgREST `.in()` filter travels in the
- * query string (#732).
+ * URLs are resolved through the request body, not the query string (#732).
*
- * The first production backfill lost 228 of 174,466 answers to this — the ones
- * carrying the most citations, averaging 16 KB of URL text against 1.2 KB for
- * the answers that succeeded. The failure was logged and the answer skipped,
- * so the totals looked healthy. These tests pin the batching rather than the
- * limit, since the limit is the server's to enforce.
+ * Two versions of this failed before: an unbounded `.in()` filter, then a
+ * chunked one. Both put the URLs in the URI, and both lost exactly the answers
+ * carrying the most citations — the last of them 182 of 174,466. These tests
+ * pin the shape that removed the class rather than a chunk size, which was
+ * only ever a guess at someone else's limit.
*/
-describe('persistCitationRows url lookups', () => {
- /** Records every `.in()` batch so the test can assert on their sizes. */
- function makeClient({ existing = new Map() } = {}) {
- const batches = [];
- let nextId = existing.size + 1;
-
+describe('persistCitationRows url resolution', () => {
+ /** Records the arguments every rpc call was made with. */
+ function makeClient({ ids = null } = {}) {
+ const calls = [];
const client = {
- from(table) {
- if (table === 'citation_urls') {
- return {
- select: () => ({
- in: (_col, urls) => {
- batches.push(urls.length);
- return Promise.resolve({
- data: urls
- .filter((u) => existing.has(u))
- .map((u) => ({ id: existing.get(u), url: u })),
- error: null,
- });
- },
- }),
- upsert: (rows) => {
- for (const row of rows) if (!existing.has(row.url)) existing.set(row.url, nextId++);
- return Promise.resolve({ error: null });
- },
- };
- }
- // prompt_result_citations: `.upsert().select()` returns the rows that
- // were actually inserted, which is what persistCitationRows counts.
- return {
- upsert: (rows) => ({
- select: () =>
- Promise.resolve({ data: rows.map((r) => ({ position: r.position })), error: null }),
- }),
- };
+ rpc: (fn, args) => {
+ calls.push({ fn, args });
+ const rows = args.p_urls.map((u, i) => ({
+ url: u.url,
+ id: ids ? ids.get(u.url) : i + 1,
+ }));
+ return Promise.resolve({ data: rows, error: null });
},
+ from: () => ({
+ upsert: (rows) => ({
+ select: () => Promise.resolve({ data: rows, error: null }),
+ }),
+ }),
};
- return { client, batches };
+ return { client, calls };
}
async function run(citations, opts) {
- const { client, batches } = makeClient(opts);
+ const { client, calls } = makeClient(opts);
vi.resetModules();
vi.doMock('../config/supabase.js', () => ({ default: client }));
const { persistCitationRows } = await import('./citation-rows.js');
@@ -170,33 +151,49 @@ describe('persistCitationRows url lookups', () => {
createdAt: '2026-08-19T00:00:00Z',
citations,
});
- return { written, batches };
+ return { written, calls };
}
- it('splits a large URL set into bounded batches', async () => {
+ it('resolves every URL in a single call, however many there are', async () => {
const citations = Array.from({ length: 250 }, (_, i) => ({ url: `https://a.com/${i}` }));
- const { written, batches } = await run(citations);
+ const { written, calls } = await run(citations);
expect(written).toBe(250);
- // 250 unknown URLs: three batches to look them up, three to read them back.
- expect(batches).toEqual([100, 100, 50, 100, 100, 50]);
- expect(Math.max(...batches)).toBeLessThanOrEqual(100);
+ expect(calls).toHaveLength(1);
+ expect(calls[0].fn).toBe('citation_url_ids');
+ expect(calls[0].args.p_urls).toHaveLength(250);
});
- it('stores every citation of an answer far larger than one batch', async () => {
- // The largest answer in production carries 191 citations.
+ it('stores every citation of an answer far larger than typical', async () => {
+ // The largest answer in production carries 191 citations; the answers that
+ // used to fail carried 22 to 51.
const citations = Array.from({ length: 191 }, (_, i) => ({ url: `https://b.com/${i}` }));
const { written } = await run(citations);
expect(written).toBe(191);
});
- it('makes no second round trip when every URL is already known', async () => {
- const existing = new Map(Array.from({ length: 120 }, (_, i) => [`https://c.com/${i}`, i + 1]));
- const citations = Array.from({ length: 120 }, (_, i) => ({ url: `https://c.com/${i}` }));
- const { written, batches } = await run(citations, { existing });
+ it('sends each distinct URL once, with the fields the dictionary stores', async () => {
+ const { calls } = await run([
+ { url: 'https://a.com/x', title: 'X' },
+ { url: 'https://a.com/x', title: 'X again' },
+ { url: 'https://b.com/y' },
+ ]);
+
+ expect(calls[0].args.p_urls).toEqual([
+ { url: 'https://a.com/x', domain: 'a.com', title: 'X' },
+ { url: 'https://b.com/y', domain: 'b.com', title: null },
+ ]);
+ });
- expect(written).toBe(120);
- expect(batches).toEqual([100, 20]); // lookup only — nothing to insert
+ // A citation whose URL the function did not return an id for is dropped
+ // rather than sent with url_id undefined, which the insert would reject and
+ // take the answer's other citations down with it.
+ it('keeps the citations it can resolve when one id is missing', async () => {
+ const ids = new Map([['https://a.com/1', 7]]);
+ const { written } = await run([{ url: 'https://a.com/1' }, { url: 'https://a.com/2' }], {
+ ids,
+ });
+ expect(written).toBe(1);
});
});
@@ -221,21 +218,19 @@ describe('persistCitationRows return contract', () => {
});
}
- const urlTable = (inserted) => ({
- select: () => ({
- in: (_c, urls) =>
- Promise.resolve({ data: urls.map((u, i) => ({ id: i + 1, url: u })), error: null }),
- }),
- upsert: () => Promise.resolve({ error: null }),
- _inserted: inserted,
- });
+ /** Every URL resolves to an id, so the outcome depends only on the insert. */
+ const resolvesAll = (_fn, args) =>
+ Promise.resolve({
+ data: args.p_urls.map((u, i) => ({ url: u.url, id: i + 1 })),
+ error: null,
+ });
it('reports how many rows were actually inserted', async () => {
const client = {
- from: (t) =>
- t === 'citation_urls'
- ? urlTable()
- : { upsert: (rows) => ({ select: () => Promise.resolve({ data: rows, error: null }) }) },
+ rpc: resolvesAll,
+ from: () => ({
+ upsert: (rows) => ({ select: () => Promise.resolve({ data: rows, error: null }) }),
+ }),
};
expect(await withClient(client, [{ url: 'https://a.com/1' }, { url: 'https://a.com/2' }])).toBe(
2,
@@ -244,39 +239,38 @@ describe('persistCitationRows return contract', () => {
it('returns 0 — not a failure — when every row was already present', async () => {
const client = {
- from: (t) =>
- t === 'citation_urls'
- ? urlTable()
- : { upsert: () => ({ select: () => Promise.resolve({ data: [], error: null }) }) },
+ rpc: resolvesAll,
+ from: () => ({
+ upsert: () => ({ select: () => Promise.resolve({ data: [], error: null }) }),
+ }),
};
expect(await withClient(client, [{ url: 'https://a.com/1' }])).toBe(0);
});
it('returns null when the write fails', async () => {
const client = {
- from: (t) =>
- t === 'citation_urls'
- ? urlTable()
- : {
- upsert: () => ({
- select: () => Promise.resolve({ data: null, error: { message: 'boom' } }),
- }),
- },
+ rpc: resolvesAll,
+ from: () => ({
+ upsert: () => ({
+ select: () => Promise.resolve({ data: null, error: { message: 'boom' } }),
+ }),
+ }),
};
expect(await withClient(client, [{ url: 'https://a.com/1' }])).toBeNull();
});
- it('returns null when the URL lookup fails', async () => {
+ it('returns null when URL resolution fails', async () => {
const client = {
+ rpc: () => Promise.resolve({ data: null, error: { message: 'boom' } }),
from: () => ({
- select: () => ({ in: () => Promise.resolve({ data: null, error: { message: 'boom' } }) }),
+ upsert: () => ({ select: () => Promise.resolve({ data: [], error: null }) }),
}),
};
expect(await withClient(client, [{ url: 'https://a.com/1' }])).toBeNull();
});
it('returns 0 for an answer with no storable citations', async () => {
- const client = { from: () => ({}) };
+ const client = { rpc: resolvesAll, from: () => ({}) };
expect(await withClient(client, [{ url: '/relative' }])).toBe(0);
});
});
diff --git a/supabase/migrations/00059_citation_url_ids.sql b/supabase/migrations/00059_citation_url_ids.sql
new file mode 100644
index 000000000..b65379906
--- /dev/null
+++ b/supabase/migrations/00059_citation_url_ids.sql
@@ -0,0 +1,66 @@
+-- Resolve citation URLs to dictionary ids through the request body (#732).
+--
+-- The write path looked URLs up with a PostgREST `.in()` filter, which is
+-- spelled out in the query string. That was wrong three times over: first
+-- unbounded, then chunked at 100, and it still failed — URLs are
+-- percent-encoded on the way into a URI, so a hundred of them at an average
+-- 92 characters (and up to 1,333) exceeds the request-line and header limits
+-- in front of PostgREST. The symptom was `Bad Request` from the proxy and
+-- `TypeError: fetch failed` when the connection was reset, on precisely the
+-- answers carrying the most citations: 182 of 174,466 after the second fix.
+--
+-- Chunking smaller would have been another guess at someone else's limit.
+-- This removes the class instead: the URLs travel as a jsonb argument in the
+-- POST body, which has no such bound, and one call both inserts what is new
+-- and returns ids for everything asked about.
+
+create or replace function public.citation_url_ids(p_urls jsonb)
+returns table (id bigint, url text)
+language plpgsql
+volatile
+security definer
+set search_path = public
+as $$
+begin
+ -- p_urls: [{"url": "...", "domain": "...", "title": "..."}, ...]
+ --
+ -- `distinct on` because one answer can cite the same page twice, and the
+ -- unique index would reject the second copy inside the same statement
+ -- rather than treating it as a conflict.
+ insert into public.citation_urls (url, domain, title)
+ select distinct on (e.value ->> 'url')
+ e.value ->> 'url',
+ e.value ->> 'domain',
+ nullif(e.value ->> 'title', '')
+ from jsonb_array_elements(p_urls) e
+ where e.value ->> 'url' is not null
+ and e.value ->> 'domain' is not null
+ order by e.value ->> 'url'
+ on conflict (url) do nothing;
+
+ -- Returned for every URL asked about, not only the ones just inserted:
+ -- the caller needs an id per citation, and the overwhelming majority were
+ -- already in the dictionary. A concurrent writer that won the insert race
+ -- is picked up here too, which is what makes the call safe to run twice.
+ return query
+ select cu.id, cu.url
+ from public.citation_urls cu
+ join (
+ select distinct e.value ->> 'url' as u
+ from jsonb_array_elements(p_urls) e
+ ) asked on asked.u = cu.url;
+end;
+$$;
+
+-- The dictionary is cross-tenant and has no select policy of its own, so this
+-- function is the only way in. It is SECURITY DEFINER to reach past that, and
+-- therefore must not be callable by anyone who should not be writing to the
+-- table: the tracking worker and the backfill both go through the service
+-- role, and nothing else has any reason to call it.
+revoke all on function public.citation_url_ids(jsonb) from public;
+revoke all on function public.citation_url_ids(jsonb) from anon;
+revoke all on function public.citation_url_ids(jsonb) from authenticated;
+grant execute on function public.citation_url_ids(jsonb) to service_role;
+
+comment on function public.citation_url_ids(jsonb) is
+ 'Insert any unseen citation URLs and return the dictionary id for every URL asked about. Takes its argument in the request body so a long list cannot overflow the query string, which is what broke the .in() lookup it replaces (#732).';
diff --git a/supabase/schema.sql b/supabase/schema.sql
index e41134525..802c77219 100644
--- a/supabase/schema.sql
+++ b/supabase/schema.sql
@@ -5935,3 +5935,73 @@ comment on table public.citation_urls is
comment on column public.prompt_result_citations.position is
'Index within prompt_results.citations. With prompt_result_id it is the natural key that makes re-running the write path or the backfill idempotent.';
+-- ─────────────────────────────────────────────────────────────────────────
+-- migrations/00059_citation_url_ids.sql
+-- ─────────────────────────────────────────────────────────────────────────
+-- Resolve citation URLs to dictionary ids through the request body (#732).
+--
+-- The write path looked URLs up with a PostgREST `.in()` filter, which is
+-- spelled out in the query string. That was wrong three times over: first
+-- unbounded, then chunked at 100, and it still failed — URLs are
+-- percent-encoded on the way into a URI, so a hundred of them at an average
+-- 92 characters (and up to 1,333) exceeds the request-line and header limits
+-- in front of PostgREST. The symptom was `Bad Request` from the proxy and
+-- `TypeError: fetch failed` when the connection was reset, on precisely the
+-- answers carrying the most citations: 182 of 174,466 after the second fix.
+--
+-- Chunking smaller would have been another guess at someone else's limit.
+-- This removes the class instead: the URLs travel as a jsonb argument in the
+-- POST body, which has no such bound, and one call both inserts what is new
+-- and returns ids for everything asked about.
+
+create or replace function public.citation_url_ids(p_urls jsonb)
+returns table (id bigint, url text)
+language plpgsql
+volatile
+security definer
+set search_path = public
+as $$
+begin
+ -- p_urls: [{"url": "...", "domain": "...", "title": "..."}, ...]
+ --
+ -- `distinct on` because one answer can cite the same page twice, and the
+ -- unique index would reject the second copy inside the same statement
+ -- rather than treating it as a conflict.
+ insert into public.citation_urls (url, domain, title)
+ select distinct on (e.value ->> 'url')
+ e.value ->> 'url',
+ e.value ->> 'domain',
+ nullif(e.value ->> 'title', '')
+ from jsonb_array_elements(p_urls) e
+ where e.value ->> 'url' is not null
+ and e.value ->> 'domain' is not null
+ order by e.value ->> 'url'
+ on conflict (url) do nothing;
+
+ -- Returned for every URL asked about, not only the ones just inserted:
+ -- the caller needs an id per citation, and the overwhelming majority were
+ -- already in the dictionary. A concurrent writer that won the insert race
+ -- is picked up here too, which is what makes the call safe to run twice.
+ return query
+ select cu.id, cu.url
+ from public.citation_urls cu
+ join (
+ select distinct e.value ->> 'url' as u
+ from jsonb_array_elements(p_urls) e
+ ) asked on asked.u = cu.url;
+end;
+$$;
+
+-- The dictionary is cross-tenant and has no select policy of its own, so this
+-- function is the only way in. It is SECURITY DEFINER to reach past that, and
+-- therefore must not be callable by anyone who should not be writing to the
+-- table: the tracking worker and the backfill both go through the service
+-- role, and nothing else has any reason to call it.
+revoke all on function public.citation_url_ids(jsonb) from public;
+revoke all on function public.citation_url_ids(jsonb) from anon;
+revoke all on function public.citation_url_ids(jsonb) from authenticated;
+grant execute on function public.citation_url_ids(jsonb) to service_role;
+
+comment on function public.citation_url_ids(jsonb) is
+ 'Insert any unseen citation URLs and return the dictionary id for every URL asked about. Takes its argument in the request body so a long list cannot overflow the query string, which is what broke the .in() lookup it replaces (#732).';
+