Skip to content

Open-source agentic AI Search Execution Platform. Track AI visibility, find opportunities, and let agents turn signals into prioritized actions and drafted fixes across ChatGPT, Claude, Gemini, Google AI Overviews, Google AI Mode, Perplexity, Grok, and Microsoft Copilot.

feat(tracking): keep the searches the OpenAI API model ran as its query fan-out (#1756)

Cihan24207bc8e669
diff --git a/docs/guides/prompts.mdx b/docs/guides/prompts.mdx
index 69a6971cf..744e28d59 100644
--- a/docs/guides/prompts.mdx
+++ b/docs/guides/prompts.mdx
@@ -34,6 +34,10 @@ Over 7 days or more, every prompt tracked daily has had its day. A 24-hour range
Self-hosted instances turn this on with `CLORO_CHATGPT_FANOUT` (off unless set; see [Environment variables](/self-host/env-variables)). The extra cost is per ChatGPT task, and a prompt has one task per location: a prompt tracked in 3 locations pays it 3 times on its day. The day is the UTC day the nightly run starts, so with a schedule close to midnight UTC a prompt can be asked 0 or 2 times in a week.
+### API models' fan-out
+
+A prompt tracked with an OpenAI model (an AI model, not the ChatGPT platform) shares the searches that model ran, from the same answer: they show as **ChatGPT API**, at no extra cost. The model doesn't search for every prompt, so some answers have none. We don't capture Claude and Gemini models' searches yet.
+
## Adding prompts
Three ways:
diff --git a/server/src/lib/health/report.js b/server/src/lib/health/report.js
index 9d6c07e3f..4688b2c18 100644
--- a/server/src/lib/health/report.js
+++ b/server/src/lib/health/report.js
@@ -175,6 +175,10 @@ async function checkCloro(now) {
* When those days can't be read, ChatGPT alone goes unjudged that day.
* - `off` (the self-host default): nothing asks, so it isn't judged.
*/
+// `chatgpt`, the OpenAI API model's answers (the tracking worker's
+// resolveModelPlatform), isn't expected: its web_search calls carry their
+// queries, but gpt-5-chat-latest may search rarely or not at all, so a silent
+// day is no news. It's judged on a drop only, like any other platform.
export const FANOUT_EXPECTED = ['copilot-web', 'perplexity-web'];
export const FANOUT_DROP_RATIO = 0.25;
/** Fewer answers than this, on either side, are too few to judge. */
@@ -399,12 +403,14 @@ export function findIssues(report) {
);
continue;
}
+ // The API model's answers come from OpenAI, not from Cloro.
+ const source = p.platform === 'chatgpt' ? "OpenAI's web_search returns" : 'Cloro returns';
issues.push(
p.withFanout === 0
? `${p.platform} returned no query fan-out on any of its ${p.answers} answers ` +
- `in the last 24h (${percent(p.baselineRate)} the days before) — check what Cloro returns`
+ `in the last 24h (${percent(p.baselineRate)} the days before) — check what ${source}`
: `${p.platform} query fan-out fell to ${percent(p.rate)} of answers ` +
- `(${percent(p.baselineRate)} the days before) — check what Cloro returns`,
+ `(${percent(p.baselineRate)} the days before) — check what ${source}`,
);
}
diff --git a/server/src/lib/health/report.test.js b/server/src/lib/health/report.test.js
index 8cdc0899c..1728a4837 100644
--- a/server/src/lib/health/report.test.js
+++ b/server/src/lib/health/report.test.js
@@ -264,6 +264,22 @@ describe('query fan-out', () => {
expect(fanoutDropped({ ...silent, platform: 'chatgpt-web', chatgptFanout: 'off' })).toBe(false);
// When every ChatGPT task asks, it should fan out like Copilot.
expect(fanoutDropped({ ...silent, platform: 'chatgpt-web', chatgptFanout: 'all' })).toBe(true);
+ // The OpenAI API model may search rarely or not at all, so a silent day
+ // is no news for it either...
+ expect(fanoutDropped({ ...silent, platform: 'chatgpt' })).toBe(false);
+ // ...but a fall from what it shared the days before is, and it's OpenAI's.
+ const fell = { ...silent, platform: 'chatgpt', withFanout: 4, rate: 0.01, baselineRate: 0.3 };
+ expect(fanoutDropped(fell)).toBe(true);
+ expect(
+ findIssues({
+ probes: {},
+ nightly: {},
+ cloro: {},
+ fanout: { platforms: [fell] },
+ }).filter((i) => i.includes('fan-out')),
+ ).toEqual([
+ "chatgpt query fan-out fell to 1% of answers (30% the days before) — check what OpenAI's web_search returns",
+ ]);
});
describe("ChatGPT's sample, one day a week per prompt", () => {
diff --git a/server/src/lib/openai-tracker.js b/server/src/lib/openai-tracker.js
index b89570180..7c7d97850 100644
--- a/server/src/lib/openai-tracker.js
+++ b/server/src/lib/openai-tracker.js
@@ -1,6 +1,8 @@
/**
* OpenAI Responses API client with web_search tool for prompt tracking.
- * Sends prompts to OpenAI and returns the response with citations.
+ * Sends prompts to OpenAI and returns the response with citations, and the
+ * searches the model ran (its query fan-out, #332), read from the same
+ * response: no other call.
*/
import OpenAI from 'openai';
@@ -21,6 +23,39 @@ function getClient() {
return client;
}
+/** The platform API-model answers are stored under (the tracking worker's resolveModelPlatform). */
+const SOURCE_PLATFORM = 'chatgpt';
+
+/**
+ * The searches a response's web_search calls ran, as the query fan-out is
+ * kept (#332, as Cloro's engines give it): `[{ query, source_platform }]`,
+ * each once, trimmed. A search action lists them in `queries`; the single
+ * `query` it carried before is deprecated, and still read while responses
+ * may hold it. Opening or searching within a page is no search, and a call
+ * whose `status` is `failed` ran none.
+ *
+ * @param {Array<{ type?: string, status?: string, action?: { type?: string, queries?: unknown, query?: unknown } }>} output
+ */
+export function searchQueriesOf(output) {
+ const seen = new Set();
+ const queries = [];
+ for (const item of output || []) {
+ if (item?.type !== 'web_search_call' || item.action?.type !== 'search') continue;
+ if (item.status === 'failed') continue;
+ const asked = [
+ ...(Array.isArray(item.action.queries) ? item.action.queries : []),
+ item.action.query,
+ ];
+ for (const value of asked) {
+ const query = typeof value === 'string' ? value.trim() : '';
+ if (!query || seen.has(query)) continue;
+ seen.add(query);
+ queries.push({ query, source_platform: SOURCE_PLATFORM });
+ }
+ }
+ return queries;
+}
+
/**
* Run a single prompt through OpenAI Responses API with web_search enabled.
* @param {string} promptText - The prompt to send
@@ -28,7 +63,7 @@ function getClient() {
* @param {string} [region] - Location code for web_search user_location: an
* ISO country (`DE`) or a US state (`US-CA`, #691). Country-only codes
* produce exactly the payload they did before states existed.
- * @returns {Promise<{ text: string, citations: Array<{ url: string, title: string, startIndex: number, endIndex: number }>, model: string }>}
+ * @returns {Promise<{ text: string, citations: Array<{ url: string, title: string, startIndex: number, endIndex: number }>, model: string, search_queries: Array<{ query: string, source_platform: string }> }>}
*/
export async function runPrompt(promptText, model, region) {
const openai = getClient();
@@ -68,7 +103,7 @@ export async function runPrompt(promptText, model, region) {
}
}
- return { text, citations, model: modelName };
+ return { text, citations, model: modelName, search_queries: searchQueriesOf(response.output) };
}
const SENTIMENT_MODEL = 'gpt-5-mini';
diff --git a/server/src/lib/openai-tracker.test.js b/server/src/lib/openai-tracker.test.js
new file mode 100644
index 000000000..76153f910
--- /dev/null
+++ b/server/src/lib/openai-tracker.test.js
@@ -0,0 +1,106 @@
+import { beforeEach, describe, expect, it, vi } from 'vitest';
+
+/**
+ * The OpenAI API model's searches (#332): read from the same Responses
+ * answer the tracking already asks for, its web_search calls' queries kept
+ * as the query fan-out. No other call.
+ */
+
+const create = vi.hoisted(() => vi.fn());
+vi.mock('openai', () => ({
+ default: class {
+ responses = { create };
+ },
+}));
+vi.mock('./logger.js', () => ({ logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() } }));
+
+process.env.OPENAI_API_KEY = 'sk-test';
+const { runPrompt, searchQueriesOf } = await import('./openai-tracker.js');
+
+const search = (action, status = 'completed') => ({
+ type: 'web_search_call',
+ id: 'ws_1',
+ status,
+ action,
+});
+
+beforeEach(() => {
+ create.mockReset();
+});
+
+describe('searchQueriesOf', () => {
+ it("keeps each search's queries (and the deprecated query) once, trimmed, and nothing else", () => {
+ expect(
+ searchQueriesOf([
+ search({ type: 'search', queries: [' best electric kettle 2026 ', 'kettle reviews', ' '] }),
+ search({ type: 'search', query: 'best electric kettle 2026' }),
+ search({ type: 'search', query: 'kettle descaling' }),
+ search({ type: 'open_page', url: 'https://example.com' }),
+ search({ type: 'find_in_page', pattern: 'price' }),
+ { type: 'message', content: [] },
+ search({ type: 'search', query: 42 }),
+ ]),
+ ).toEqual([
+ { query: 'best electric kettle 2026', source_platform: 'chatgpt' },
+ { query: 'kettle reviews', source_platform: 'chatgpt' },
+ { query: 'kettle descaling', source_platform: 'chatgpt' },
+ ]);
+ expect(searchQueriesOf(undefined)).toEqual([]);
+ });
+
+ it('counts no search a failed call ran', () => {
+ expect(
+ searchQueriesOf([
+ search({ type: 'search', queries: ['kettle reviews'] }, 'failed'),
+ search({ type: 'search', queries: ['kettle prices'] }, 'in_progress'),
+ ]),
+ ).toEqual([{ query: 'kettle prices', source_platform: 'chatgpt' }]);
+ });
+});
+
+describe('runPrompt', () => {
+ it('returns the searches beside the answer and its citations, from one call', async () => {
+ create.mockResolvedValue({
+ output_text: 'The Acme Kettle is a good pick.',
+ output: [
+ search({ type: 'search', query: 'best electric kettle' }),
+ {
+ type: 'message',
+ content: [
+ {
+ type: 'output_text',
+ annotations: [
+ {
+ type: 'url_citation',
+ url: 'https://acme.com/kettle',
+ title: 'Acme Kettle',
+ start_index: 4,
+ end_index: 15,
+ },
+ ],
+ },
+ ],
+ },
+ ],
+ });
+ const result = await runPrompt('best electric kettle?', undefined, 'US');
+ expect(create).toHaveBeenCalledTimes(1);
+ expect(create.mock.calls[0][0]).toMatchObject({
+ model: 'gpt-5-chat-latest',
+ tools: [{ type: 'web_search' }],
+ });
+ expect(result).toEqual({
+ text: 'The Acme Kettle is a good pick.',
+ citations: [
+ { url: 'https://acme.com/kettle', title: 'Acme Kettle', startIndex: 4, endIndex: 15 },
+ ],
+ model: 'gpt-5-chat-latest',
+ search_queries: [{ query: 'best electric kettle', source_platform: 'chatgpt' }],
+ });
+ });
+
+ it('gives no searches for an answer that ran none', async () => {
+ create.mockResolvedValue({ output_text: 'Hello.', output: [] });
+ expect((await runPrompt('hi', 'gpt-4.1')).search_queries).toEqual([]);
+ });
+});
diff --git a/server/src/workers/tracking-worker.js b/server/src/workers/tracking-worker.js
index 112e990b6..d0c1553b4 100644
--- a/server/src/workers/tracking-worker.js
+++ b/server/src/workers/tracking-worker.js
@@ -945,6 +945,10 @@ export async function processTrackingJob({ brandId, promptId, promptIds, source,
competitor_mentions: metrics.competitorMentions,
mention_position: metrics.mentionPosition,
mentioned_entity_count: metrics.mentionedEntityCount,
+ // The searches the model ran, where its API gives them (OpenAI's web_search).
+ search_queries: Array.isArray(aiResponse.search_queries)
+ ? aiResponse.search_queries
+ : [],
});
} catch (err) {
logger.error({ err, model: modelName, region }, 'ai model task failed');
diff --git a/web/messages/en.json b/web/messages/en.json
index a47976809..820740c73 100644
--- a/web/messages/en.json
+++ b/web/messages/en.json
@@ -8191,8 +8191,8 @@
"engineEmptyTitle": "No {engine} searches in the {range}",
"engineEmptyBody": "{engine} usually shares the searches behind its answers. Try a longer date range.",
"emptyTitle": "No fan-out captured yet",
- "emptyBody": "Copilot and Perplexity share the searches behind their answers, and ChatGPT's are sampled on some days; Google AI Overviews, AI Mode and Gemini don't share them. Once those engines run for your prompts, the searches will appear here.",
- "coverageTooltip": "Of {total} tracked answers in this window, {withFanout} came with the searches the engine ran. Copilot and Perplexity share them, and ChatGPT's are sampled on some days."
+ "emptyBody": "Copilot and Perplexity share the searches behind their answers; ChatGPT's are sampled on some days, and the ChatGPT API shares them when it searches. Google AI Overviews, AI Mode and Gemini don't share them. Once those engines run for your prompts, the searches will appear here.",
+ "coverageTooltip": "Of {total} tracked answers in this window, {withFanout} came with the searches the engine ran. Copilot and Perplexity share them, ChatGPT's are sampled on some days, and the ChatGPT API shares them when it searches."
},
"bulkDelete": {
"progress": "Deleting {done} of {total} prompts…",
diff --git a/web/src/config/fanout-platforms.test.ts b/web/src/config/fanout-platforms.test.ts
new file mode 100644
index 000000000..287583b53
--- /dev/null
+++ b/web/src/config/fanout-platforms.test.ts
@@ -0,0 +1,23 @@
+import { describe, expect, it } from 'vitest';
+import { fanoutSharing } from './fanout-platforms';
+
+/** How the Query Fan-out page reads each engine's searches (#332). */
+describe('fanoutSharing', () => {
+ it('expects searches from the engines that send them with their answers', () => {
+ expect(fanoutSharing('copilot-web')).toBe('shares');
+ expect(fanoutSharing('perplexity-web')).toBe('shares');
+ });
+
+ it('reads the engines that never send them, or whose searches we miss, as not shared', () => {
+ for (const platform of ['google-aio', 'google-aimode', 'gemini-web', 'claude', 'gemini']) {
+ expect(fanoutSharing(platform)).toBe('not-shared');
+ }
+ });
+
+ it('expects nothing either way from ChatGPT and the ChatGPT API, which send them some days', () => {
+ // ChatGPT: sampled one day a week per prompt. The API model: when it searches.
+ expect(fanoutSharing('chatgpt-web')).toBe('unknown');
+ expect(fanoutSharing('chatgpt')).toBe('unknown');
+ expect(fanoutSharing('something-new')).toBe('unknown');
+ });
+});
diff --git a/web/src/config/fanout-platforms.ts b/web/src/config/fanout-platforms.ts
index 0c4de9903..ea9910d6d 100644
--- a/web/src/config/fanout-platforms.ts
+++ b/web/src/config/fanout-platforms.ts
@@ -12,15 +12,17 @@ export const FANOUT_SHARING_PLATFORMS: readonly string[] = ['copilot-web', 'perp
/**
* Platforms whose answers never come with their searches: those our provider
- * documents no fan-out for, and the API models (`claude`, `gemini`,
- * `chatgpt`, see resolveModelPlatform in the tracking worker), whose searches
- * we don't capture.
+ * documents no fan-out for, and the API models whose searches we don't
+ * capture (`claude`, `gemini`, see resolveModelPlatform in the tracking
+ * worker).
*
- * ChatGPT (`chatgpt-web`) is in neither list. Its default interface carries no
- * query text since 2026-09-30 (#1119); the server asks its legacy interface on
- * one day a week per prompt (CLORO_CHATGPT_FANOUT, off unless set on
- * self-host). So a window may hold some of its searches or none, and neither
- * is news: it reads as an ordinary engine, never flagged.
+ * Two are in neither list, so a window may hold some of their searches or
+ * none and neither is news: they read as ordinary engines, never flagged.
+ * - ChatGPT (`chatgpt-web`): its default interface carries no query text
+ * since 2026-09-30 (#1119); the server asks its legacy interface on one day
+ * a week per prompt (CLORO_CHATGPT_FANOUT, off unless set on self-host).
+ * - The OpenAI API model (`chatgpt`): its web_search calls carry their
+ * queries, but gpt-5-chat-latest may search rarely or not at all.
*/
export const FANOUT_NOT_SHARED_PLATFORMS: readonly string[] = [
'google-aio',
@@ -28,7 +30,6 @@ export const FANOUT_NOT_SHARED_PLATFORMS: readonly string[] = [
'gemini-web',
'claude',
'gemini',
- 'chatgpt',
];
/**