Open-source agentic AI Search Execution Platform. Track AI visibility, find opportunities, and let agents turn signals into prioritized actions and drafted fixes across ChatGPT, Claude, Gemini, Google AI Overviews, Google AI Mode, Perplexity, Grok, and Microsoft Copilot.
TypeScript aeoai-agentai-searchanswer-engine-optimizationbrand-monitoringchatgptclaudecopilotdockergeminigeogoogle-ai-overviewsgrokmcpnextjsopen-sourceperplexityseosupabasetypescript
feat(traffic): count AI traffic from one versioned source list in the snippet and Google Analytics (#1166)
Cihan6b1b74360f06
diff --git a/docs/guides/ai-traffic-analytics.mdx b/docs/guides/ai-traffic-analytics.mdxindex 7fa05eb9e..c46216786 100644--- a/docs/guides/ai-traffic-analytics.mdx+++ b/docs/guides/ai-traffic-analytics.mdx@@ -46,16 +46,18 @@ Most analytics platforms (Google Analytics, Plausible, etc.) lump AI traffic int## Supported AI sources-Pixel detects and classifies referrers from:+The pixel and the Google Analytics sync count the same AI assistants, from one list (`server/src/lib/ai-sources.json`):-- ChatGPT (chatgpt.com, chat.openai.com, search.chatgpt.com)+- ChatGPT (chatgpt.com, chat.openai.com)- Perplexity (perplexity.ai)- Claude (claude.ai)- Gemini (gemini.google.com)-- Copilot (copilot.microsoft.com)-- You.com, Phind, Meta AI, Poe+- Copilot (copilot.microsoft.com, copilot.com, and Bing links from Copilot answers)+- Grok, DeepSeek, Meta AI, You.com, Phind, Poe, Google Labs-Anything else falls into `unknown` — likely a future AI source we haven't catalogued yet. The list is updated regularly.+A visit counts when the referrer is one of these hosts or a subdomain of it (m.perplexity.ai), or, without one, when the link's `utm_source` names it (ChatGPT adds `utm_source=chatgpt.com`). A name inside another host doesn't count.++Google Analytics also counts its own **AI Assistant** channel, so an assistant that isn't on the list yet still shows, under its source's name. The list is versioned: each change is dated in the file, and every stored visit records the version that counted it.## Privacydiff --git a/server/src/lib/ai-sources.js b/server/src/lib/ai-sources.jsnew file mode 100644index 000000000..1894a7fc4--- /dev/null+++ b/server/src/lib/ai-sources.js@@ -0,0 +1,86 @@+/**+ * The AI assistants Ansvisor counts traffic from (AT-004, spec §5.1): one+ * list, in ai-sources.json, for the tracking snippet and the Google Analytics+ * sync, so the two name the same engine the same way and never disagree on+ * what counts.+ *+ * Each source has a key (the host-style name stored in+ * ai_traffic_logs.source_platform and ga_ai_traffic_stats.platform), the+ * hosts it is reached from, and aliases: exact names some tools use instead+ * of a host, like GA4's "chatgpt" or "openai".+ *+ * A value matches a host when it is the host or one of its subdomains+ * (m.perplexity.ai, search.chatgpt.com) — never when it merely contains an+ * engine's name. A *.workers.dev proxy with "claude" in its name once showed+ * up in a real property; substring matching would have booked it as AI+ * traffic.+ *+ * The list is versioned: each change adds an entry to `changes` and bumps+ * `version`, and every stored row records the version that counted it, so a+ * step in the history can be traced to the list rather than to the traffic.+ */++import { createRequire } from 'module';++const require = createRequire(import.meta.url);+const config = require('./ai-sources.json');++export const AI_SOURCES_VERSION = config.version;+export const AI_SOURCE_CHANGES = config.changes;+export const AI_SOURCES = config.sources;++/**+ * Where a value comes from. A referrer is the snippet's main signal; a+ * utm_source tag and a GA4 session source are names chosen by whoever built+ * the link or by GA4, so a source can opt out of either (Bing's links say+ * nothing about Copilot without the /ck/a path only a referrer carries).+ */+const ORIGINS = ['referrer', 'utm', 'analytics'];++/** Lowercased, trimmed, without a leading "www.". */+export function normalizeSource(raw) {+ if (typeof raw !== 'string') return '';+ return raw+ .trim()+ .toLowerCase()+ .replace(/^www\./, '');+}++function counts(source, origin) {+ if (origin === 'utm') return source.utm !== false;+ if (origin === 'analytics') return source.analytics !== false;+ return true;+}++/**+ * The source key behind a referrer host, a utm_source value or a GA4 session+ * source, or null when it isn't on the list. Unrecognised is a normal+ * outcome: most traffic isn't from an AI assistant.+ */+export function classifyAiSource(raw, { origin = 'referrer' } = {}) {+ if (!ORIGINS.includes(origin)) throw new Error(`Unknown AI source origin: ${origin}`);+ const value = normalizeSource(raw);+ if (!value) return null;+ for (const source of AI_SOURCES) {+ if (!counts(source, origin)) continue;+ if (source.aliases.includes(value)) return source.key;+ if (source.hosts.some((host) => value === host || value.endsWith(`.${host}`))) {+ return source.key;+ }+ }+ return null;+}++const escapeRegex = (value) => value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');++/**+ * The same match as one regular expression, for asking GA4 for these+ * sessions (FULL_REGEXP, RE2 syntax): a listed host or any subdomain of it,+ * or an alias exactly.+ */+export function aiSourcePattern({ origin = 'analytics' } = {}) {+ const sources = AI_SOURCES.filter((source) => counts(source, origin));+ const hosts = sources.flatMap((source) => source.hosts).map(escapeRegex);+ const aliases = sources.flatMap((source) => source.aliases).map(escapeRegex);+ return `^((.+\\.)?(${hosts.join('|')})|(www\\.)?(${aliases.join('|')}))$`;+}diff --git a/server/src/lib/ai-sources.json b/server/src/lib/ai-sources.jsonnew file mode 100644index 000000000..9825e3bdf--- /dev/null+++ b/server/src/lib/ai-sources.json@@ -0,0 +1,48 @@+{+ "version": 2,+ "changes": [+ {+ "version": 1,+ "summary": "Two separate lists: one in the tracking snippet, one in the Google Analytics sync."+ },+ {+ "version": 2,+ "date": "2026-10-18",+ "summary": "One list for both. The snippet also counts Grok, DeepSeek, OpenAI and Bard links and stores chat.openai.com as ChatGPT; Google Analytics also counts Google Labs, subdomains such as m.perplexity.ai, and its own AI Assistant channel."+ }+ ],+ "sources": [+ {+ "key": "chatgpt.com",+ "hosts": ["chatgpt.com", "chat.openai.com"],+ "aliases": ["chatgpt", "openai", "openai.com"]+ },+ { "key": "perplexity.ai", "hosts": ["perplexity.ai"], "aliases": ["perplexity"] },+ { "key": "claude.ai", "hosts": ["claude.ai"], "aliases": ["claude"] },+ {+ "key": "gemini.google.com",+ "hosts": ["gemini.google.com", "bard.google.com"],+ "aliases": ["gemini", "bard"]+ },+ {+ "key": "copilot.microsoft.com",+ "hosts": ["copilot.microsoft.com", "copilot.com"],+ "aliases": ["copilot"]+ },+ { "key": "grok.com", "hosts": ["grok.com"], "aliases": ["grok"] },+ { "key": "deepseek.com", "hosts": ["deepseek.com"], "aliases": ["deepseek"] },+ { "key": "meta.ai", "hosts": ["meta.ai"], "aliases": [] },+ { "key": "you.com", "hosts": ["you.com"], "aliases": [] },+ { "key": "phind.com", "hosts": ["phind.com"], "aliases": [] },+ { "key": "poe.com", "hosts": ["poe.com"], "aliases": [] },+ { "key": "labs.google", "hosts": ["labs.google"], "aliases": [] },+ {+ "key": "bing.com",+ "hosts": ["bing.com"],+ "aliases": [],+ "referrerPath": "/ck/a",+ "analytics": false,+ "utm": false+ }+ ]+}diff --git a/server/src/lib/ai-sources.test.js b/server/src/lib/ai-sources.test.jsnew file mode 100644index 000000000..34bfff4c3--- /dev/null+++ b/server/src/lib/ai-sources.test.js@@ -0,0 +1,104 @@+import { readFileSync } from 'fs';+import { describe, expect, it } from 'vitest';+import {+ AI_SOURCES,+ AI_SOURCES_VERSION,+ AI_SOURCE_CHANGES,+ aiSourcePattern,+ classifyAiSource,+} from './ai-sources.js';++/**+ * The AI source list (AT-004, spec §5.1): one list for the tracking snippet+ * and the Google Analytics sync, matched by host or subdomain, never by a+ * name inside another host, and versioned.+ */++describe('classifyAiSource', () => {+ it('matches a listed host, a subdomain of it, or an alias', () => {+ expect(classifyAiSource('chatgpt.com')).toBe('chatgpt.com');+ expect(classifyAiSource('search.chatgpt.com')).toBe('chatgpt.com');+ expect(classifyAiSource('chat.openai.com')).toBe('chatgpt.com');+ expect(classifyAiSource('openai')).toBe('chatgpt.com');+ expect(classifyAiSource('www.perplexity.ai')).toBe('perplexity.ai');+ expect(classifyAiSource(' Gemini ')).toBe('gemini.google.com');+ expect(classifyAiSource('bard.google.com')).toBe('gemini.google.com');+ expect(classifyAiSource('grok.com')).toBe('grok.com');+ // Copilot tags its links utm_source=copilot.com, often with no referrer.+ expect(classifyAiSource('copilot.com', { origin: 'utm' })).toBe('copilot.microsoft.com');+ });++ it("doesn't match an engine's name inside another host", () => {+ expect(classifyAiSource('claude-proxy.example.workers.dev')).toBeNull();+ expect(classifyAiSource('notchatgpt.com')).toBeNull();+ expect(classifyAiSource('chatgpt.com.evil.example')).toBeNull();+ // An alias is exact: other OpenAI sites aren't the assistant.+ expect(classifyAiSource('help.openai.com')).toBeNull();+ expect(classifyAiSource('google.com')).toBeNull();+ });++ it('counts Bing only from a referrer, never from a tag or GA4', () => {+ expect(classifyAiSource('bing.com', { origin: 'referrer' })).toBe('bing.com');+ expect(classifyAiSource('bing.com', { origin: 'utm' })).toBeNull();+ expect(classifyAiSource('bing.com', { origin: 'analytics' })).toBeNull();+ });++ it('returns null for nothing, and refuses an unknown origin', () => {+ expect(classifyAiSource('')).toBeNull();+ expect(classifyAiSource(null)).toBeNull();+ expect(() => classifyAiSource('chatgpt.com', { origin: 'email' })).toThrow();+ });+});++describe('aiSourcePattern', () => {+ // The pattern asks GA4 for sessions; the classifier names them. A source+ // the pattern lets in but the classifier can't name would be stored+ // unnamed, and one the classifier knows but the pattern misses never+ // arrives.+ const samples = [+ ...AI_SOURCES.flatMap((s) => [...s.hosts, ...s.aliases]),+ ...AI_SOURCES.flatMap((s) => s.hosts.map((h) => `m.${h}`)),+ ...AI_SOURCES.flatMap((s) => s.aliases.map((a) => `www.${a}`)),+ 'CHATGPT.COM',+ 'notchatgpt.com',+ 'help.openai.com',+ 'claude-proxy.example.workers.dev',+ 'google',+ 'bing.com',+ '(direct)',+ ];++ it('lets in exactly what the classifier names', () => {+ const pattern = new RegExp(aiSourcePattern({ origin: 'analytics' }), 'i');+ for (const value of samples) {+ expect(pattern.test(value), value).toBe(+ classifyAiSource(value, { origin: 'analytics' }) !== null,+ );+ }+ });+});++describe('the list', () => {+ it('has one entry per key, each naming itself', () => {+ const keys = AI_SOURCES.map((s) => s.key);+ expect(new Set(keys).size).toBe(keys.length);+ for (const key of keys) expect(classifyAiSource(key)).toBe(key);+ });++ it('records each version in its changes, the current one last', () => {+ expect(AI_SOURCE_CHANGES.at(-1).version).toBe(AI_SOURCES_VERSION);+ expect(AI_SOURCE_CHANGES.map((c) => c.version)).toEqual(+ Array.from({ length: AI_SOURCES_VERSION }, (_, i) => i + 1),+ );+ });++ it("agrees with 00175's renaming of the snippet's older rows", () => {+ const migration = readFileSync(+ new URL('../../../supabase/migrations/00175_ai_source_list.sql', import.meta.url),+ 'utf8',+ );+ const pairs = [...migration.matchAll(/\('([^']+)', '([^']+)'\)/g)];+ expect(pairs.length).toBeGreaterThan(10);+ for (const [, raw, key] of pairs) expect(classifyAiSource(raw), raw).toBe(key);+ });+});diff --git a/server/src/lib/ga-sources.js b/server/src/lib/ga-sources.jsindex 0e7260719..acf2550a0 100644--- a/server/src/lib/ga-sources.js+++ b/server/src/lib/ga-sources.js@@ -1,72 +1,53 @@/**- * GA4 session-source classification (#704).+ * GA4 session-source classification (#704, AT-004).*- * Deliberately an explicit list rather than a pattern. Real data from a- * connected property shows why both halves of that matter:+ * Two signals, as spec §5.1 asks:*- * - One engine arrives under several source strings. `chatgpt.com`,- * `chatgpt` and `openai` are the same product; a list keyed on the exact- * string collapses them without guessing.- * - Substring matching is actively wrong. A `*.workers.dev` subdomain with- * "claude" in its name showed up as an ordinary referrer — a `includes()`- * check would have booked it as AI traffic and inflated the number nobody- * can then verify.+ * - The AI source list (ai-sources.js), shared with the tracking snippet: a+ * listed host or one of its subdomains, or an exact alias. One engine+ * arrives under several source strings — `chatgpt.com`, `chatgpt` and+ * `openai` are the same product — and the list collapses them without+ * guessing. It also covers history from before GA4 had a channel for AI.+ * - GA4's own "AI Assistant" channel (May 2026; Default Channel Group+ * definitions: medium exactly "ai-assistant"). It needs no setup in the+ * customer's property and catches assistants the list doesn't know yet.*- * Values are the same host-style identifiers the tracking snippet records in- * ai_traffic_logs.source_platform, so the two origins name the same engine the- * same way. They are still counted separately everywhere — matching names is- * for comparison, never for summing.+ * Values are the same host-style keys the snippet records in+ * ai_traffic_logs.source_platform, so the two origins name the same engine+ * the same way. They are still never summed — matching names is for+ * comparison and for choosing one origin per day, never for adding.*/-/**- * Raw GA4 `sessionSource` → platform. Keys are matched exactly after- * lowercasing and trimming; add observed variants here rather than loosening- * the match.- */-export const GA_SOURCE_PLATFORMS = {- // OpenAI- 'chatgpt.com': 'chatgpt.com',- 'chat.openai.com': 'chatgpt.com',- 'search.chatgpt.com': 'chatgpt.com',- chatgpt: 'chatgpt.com',- openai: 'chatgpt.com',- 'openai.com': 'chatgpt.com',- // Perplexity- 'perplexity.ai': 'perplexity.ai',- 'www.perplexity.ai': 'perplexity.ai',- perplexity: 'perplexity.ai',- // Anthropic- 'claude.ai': 'claude.ai',- claude: 'claude.ai',- 'gemini.google.com': 'gemini.google.com',- gemini: 'gemini.google.com',- 'bard.google.com': 'gemini.google.com',- // Microsoft- 'copilot.microsoft.com': 'copilot.microsoft.com',- copilot: 'copilot.microsoft.com',- // Others the snippet already knows about- 'you.com': 'you.com',- 'phind.com': 'phind.com',- 'meta.ai': 'meta.ai',- 'poe.com': 'poe.com',- // Newer assistants, listed before they show up rather than after- 'grok.com': 'grok.com',- grok: 'grok.com',- 'deepseek.com': 'deepseek.com',- 'chat.deepseek.com': 'deepseek.com',- deepseek: 'deepseek.com',-};+import { aiSourcePattern, classifyAiSource, normalizeSource } from './ai-sources.js';++/** GA4's channel for sessions from AI assistants, in sessionDefaultChannelGroup. */+export const GA_AI_ASSISTANT_CHANNEL = 'AI Assistant';++/** Every session source worth asking GA4 for, as one RE2 pattern. */+export const GA_AI_SOURCE_PATTERN = aiSourcePattern({ origin: 'analytics' });-/** Every raw source string worth asking GA4 for. */-export const GA_AI_SOURCE_VALUES = Object.keys(GA_SOURCE_PLATFORMS);+/** GA4's own placeholders for a session without a source. */+const NO_SOURCE = new Set(['', '(not set)', '(direct)', '(none)']);/*** The platform behind a raw GA4 source string, or null when unrecognised.* Unrecognised is a normal outcome — the caller stores the raw value so the- * list above can grow from what customers actually receive.+ * list can grow from what customers actually receive.*/export function classifyGaSource(raw) {- if (typeof raw !== 'string') return null;- return GA_SOURCE_PLATFORMS[raw.trim().toLowerCase()] ?? null;+ return classifyAiSource(raw, { origin: 'analytics' });+}++/**+ * The platform a GA4 AI-traffic row is stored under: the list's key when the+ * source is on it; for a session GA4 put in its AI Assistant channel but the+ * list doesn't know yet, the source itself (normalized), so a new assistant+ * shows under its own name instead of vanishing. Null otherwise.+ */+export function gaAiPlatform(source, channel) {+ const known = classifyGaSource(source);+ if (known) return known;+ if (channel !== GA_AI_ASSISTANT_CHANNEL) return null;+ const value = normalizeSource(source);+ return NO_SOURCE.has(value) ? null : value;}diff --git a/server/src/lib/ga-sync.js b/server/src/lib/ga-sync.jsindex b3aa01d3b..1191aaedc 100644--- a/server/src/lib/ga-sync.js+++ b/server/src/lib/ga-sync.js@@ -18,7 +18,13 @@import supabaseAdmin from '../config/supabase.js';import { isComposioConfigured, runGaReport } from './composio.js';-import { GA_AI_SOURCE_VALUES, classifyGaSource } from './ga-sources.js';+import { AI_SOURCES_VERSION } from './ai-sources.js';+import {+ GA_AI_ASSISTANT_CHANNEL,+ GA_AI_SOURCE_PATTERN,+ classifyGaSource,+ gaAiPlatform,+} from './ga-sources.js';import { logger } from './logger.js';/** First sync pulls this much history; later runs only refresh the tail. */@@ -133,25 +139,55 @@ function trafficMetricColumns(row) {};}-/** AI-sourced rows → ga_ai_traffic_stats. Rows without a date are dropped. */+const SUMMED_COLUMNS = [+ 'sessions',+ 'engaged_sessions',+ 'key_events',+ 'total_users',+ 'transactions',+ 'purchase_revenue',+ 'engagement_duration_seconds',+];++/**+ * AI-sourced rows → ga_ai_traffic_stats. Rows without a date are dropped.+ *+ * The report is split by channel too, to know which sessions GA4 itself+ * calls AI (AT-004), so one source can come back twice for a page and day —+ * as a referral and as an AI Assistant session. Storage keeps one row per+ * (date, source, entry URL), so those are summed into it: the same visits+ * GA4 would report without the channel, never counted twice.+ */export function mapAiTrafficRows(brandId, report) {- const mapped = [];+ const byKey = new Map();for (const row of readReportRows(report)) {const date = gaDateToIso(row.date);if (!date) continue;const source = row.sessionSource ?? '';- mapped.push({+ const landingPageQuery = row.landingPagePlusQueryString ?? '';+ const metrics = trafficMetricColumns(row);+ const key = `${date}|${source}|${landingPageQuery}`;+ const seen = byKey.get(key);+ if (seen) {+ for (const column of SUMMED_COLUMNS) seen[column] += metrics[column];+ // Known to GA4's channel on either row is enough to name it.+ seen.platform ??= gaAiPlatform(source, row.sessionDefaultChannelGroup);+ continue;+ }+ byKey.set(key, {brand_id: brandId,date,source,- platform: classifyGaSource(source),+ platform: gaAiPlatform(source, row.sessionDefaultChannelGroup),// Path for joining the page totals, full URL for attribution.landing_page: row.landingPage ?? '',- landing_page_query: row.landingPagePlusQueryString ?? '',- ...trafficMetricColumns(row),+ landing_page_query: landingPageQuery,+ ...metrics,+ // The AI source list that counted this row (ai-sources.json).+ source_list_version: AI_SOURCES_VERSION,});}- return mapped;+ return [...byKey.values()];}/** Landing-page totals → ga_page_stats. */@@ -194,7 +230,7 @@ export function mapItemRows(brandId, report) {/*** Sources GA4 reported that the classification list does not know, ordered by* sessions. Logged rather than stored: the point is to notice a new assistant- * showing up in real traffic and extend ga-sources.js deliberately, not to+ * showing up in real traffic and extend ai-sources.json deliberately, not to* accumulate every referrer a site ever sees.*/export function unclassifiedSources(report, { minSessions = 1, limit = 10 } = {}) {@@ -299,32 +335,65 @@ async function syncDays({ brand, entityId, days, label, ...report }) {return stored;}-async function syncBrand(brand) {+/**+ * The days the AI report covers. A first sync, or a history backfill, takes+ * as far back as storage keeps (RETENTION_DAYS, about 13 months): GA4 fills+ * the AI traffic history from before the snippet was installed (spec §5.2),+ * and the AI slice is small. Page totals stay at BACKFILL_DAYS — up to 5,000+ * rows a day each.+ */+export function aiReportDays({ first, aiHistory = false }) {+ return dayRange(first || aiHistory ? RETENTION_DAYS : REFRESH_DAYS);+}++/** The AI report's filter: a session source on the list, or GA4's own AI channel. */+export const GA_AI_FILTER = {+ orGroup: {+ expressions: [+ {+ filter: {+ fieldName: 'sessionSource',+ stringFilter: {+ matchType: 'FULL_REGEXP',+ value: GA_AI_SOURCE_PATTERN,+ caseSensitive: false,+ },+ },+ },+ {+ filter: {+ fieldName: 'sessionDefaultChannelGroup',+ stringFilter: { matchType: 'EXACT', value: GA_AI_ASSISTANT_CHANNEL },+ },+ },+ ],+ },+};++async function syncBrand(brand, { aiHistory = false } = {}) {const entityId = `org_${brand.organization_id}`;const first = await isFirstSync(brand.id);const days = dayRange(first ? BACKFILL_DAYS : REFRESH_DAYS);const dateRanges = [{ startDate: days[days.length - 1], endDate: days[0] }];+ const aiDays = aiReportDays({ first, aiHistory });const metrics = TRAFFIC_METRICS.map((name) => ({ name }));// The AI slice is small enough to take in one call — a live property// produced 96 rows over 90 days — so it keeps the full entry URL and needs// no ceiling.const aiReport = await runGaReport(entityId, brand.ga_property_id, {- dateRanges,+ dateRanges: [{ startDate: aiDays[aiDays.length - 1], endDate: aiDays[0] }],dimensions: [{ name: 'date' },{ name: 'sessionSource' },+ { name: 'sessionDefaultChannelGroup' },{ name: 'landingPage' },{ name: 'landingPagePlusQueryString' },],metrics,- // An explicit value list, not a pattern — see ga-sources.js for why.- dimensionFilter: {- filter: {- fieldName: 'sessionSource',- inListFilter: { values: GA_AI_SOURCE_VALUES, caseSensitive: false },- },- },+ // A host on the list or one of its subdomains, never a substring match —+ // see ai-sources.js for why — or GA4's AI Assistant channel.+ dimensionFilter: GA_AI_FILTER,orderBys: PAGE_ORDER,limit: 100_000,});@@ -380,7 +449,7 @@ async function syncBrand(brand) {// Source discovery: one cheap call (no page dimension) over the same window// so a newly popular assistant shows up in the logs instead of silently- // failing the inList filter above forever.+ // failing the filter above forever.try {const report = await runGaReport(entityId, brand.ga_property_id, {dateRanges,@@ -411,9 +480,11 @@ async function prune(table) {/*** Sync every eligible brand; optionally scoped to one organization (the* manual trigger) or to one brand (right after its property is mapped).+ * `aiHistory` re-reads the AI report as far back as storage keeps, for brands+ * synced before the AI history went past BACKFILL_DAYS (backfill-ga-ai-history.js).* Returns per-brand counts.*/-export async function runGaSync({ organizationId, brandId } = {}) {+export async function runGaSync({ organizationId, brandId, aiHistory = false } = {}) {if (!isComposioConfigured('google-analytics')) return { synced: 0, skipped: 0, results: [] };// Orgs with a live connection…@@ -444,7 +515,7 @@ export async function runGaSync({ organizationId, brandId } = {}) {let skipped = 0;for (const brand of brands || []) {try {- const counts = await syncBrand(brand);+ const counts = await syncBrand(brand, { aiHistory });results.push({ brandId: brand.id, ...counts });synced++;} catch (err) {diff --git a/server/src/lib/ga-sync.test.js b/server/src/lib/ga-sync.test.jsindex d8d0f2e34..2a3e6aeff 100644--- a/server/src/lib/ga-sync.test.js+++ b/server/src/lib/ga-sync.test.js@@ -4,8 +4,11 @@ import { describe, expect, it, vi } from 'vitest';// SUPABASE_* env is absent (as in CI) — stub it before the import chain.vi.mock('../config/supabase.js', () => ({ default: {} }));-import { classifyGaSource } from './ga-sources.js';+import { AI_SOURCES_VERSION } from './ai-sources.js';+import { GA_AI_SOURCE_PATTERN, classifyGaSource, gaAiPlatform } from './ga-sources.js';import {+ GA_AI_FILTER,+ aiReportDays,dayRange,gaDateToIso,mapAiTrafficRows,@@ -67,6 +70,59 @@ describe('classifyGaSource', () => {expect(classifyGaSource('(not set)')).toBeNull();expect(classifyGaSource(undefined)).toBeNull();});++ it('matches a subdomain of a listed host (AT-004)', () => {+ expect(classifyGaSource('m.perplexity.ai')).toBe('perplexity.ai');+ expect(classifyGaSource('search.chatgpt.com')).toBe('chatgpt.com');+ expect(classifyGaSource('chat.deepseek.com')).toBe('deepseek.com');+ });++ it("doesn't count Bing: without the snippet's /ck/a path, a GA4 source can't tell Copilot from search", () => {+ expect(classifyGaSource('bing.com')).toBeNull();+ expect(classifyGaSource('bing')).toBeNull();+ });+});++describe('gaAiPlatform', () => {+ it("names a session GA4's AI Assistant channel holds but the list doesn't know by its source", () => {+ expect(gaAiPlatform('chat.mistral.ai', 'AI Assistant')).toBe('chat.mistral.ai');+ expect(gaAiPlatform('www.Mistral.ai', 'AI Assistant')).toBe('mistral.ai');+ expect(gaAiPlatform('(not set)', 'AI Assistant')).toBeNull();+ });++ it('prefers the list, and leaves an unknown source outside the channel unnamed', () => {+ expect(gaAiPlatform('chatgpt.com', 'Referral')).toBe('chatgpt.com');+ expect(gaAiPlatform('example.com', 'Referral')).toBeNull();+ });+});++describe('the AI report request', () => {+ it("asks for a source on the list or GA4's AI Assistant channel", () => {+ expect(GA_AI_FILTER.orGroup.expressions).toEqual([+ {+ filter: {+ fieldName: 'sessionSource',+ stringFilter: {+ matchType: 'FULL_REGEXP',+ value: GA_AI_SOURCE_PATTERN,+ caseSensitive: false,+ },+ },+ },+ {+ filter: {+ fieldName: 'sessionDefaultChannelGroup',+ stringFilter: { matchType: 'EXACT', value: 'AI Assistant' },+ },+ },+ ]);+ });++ it('reads as far back as storage keeps on a first sync or a history backfill, else the last days', () => {+ expect(aiReportDays({ first: true })).toHaveLength(401);+ expect(aiReportDays({ first: false, aiHistory: true })).toHaveLength(401);+ expect(aiReportDays({ first: false })).toHaveLength(4);+ });});describe('gaDateToIso', () => {@@ -143,10 +199,71 @@ describe('mapAiTrafficRows', () => {transactions: 2,purchase_revenue: 149.9,engagement_duration_seconds: 106,+ source_list_version: AI_SOURCES_VERSION,},]);});+ it('sums one source split across channels back into one row (AT-004)', () => {+ const split = report({+ dims: [+ 'date',+ 'sessionSource',+ 'sessionDefaultChannelGroup',+ 'landingPage',+ 'landingPagePlusQueryString',+ ],+ metrics: TRAFFIC_METRICS,+ rows: [+ ['20260721', 'chatgpt.com', 'Referral', '/a', '/a', '4', '2', '1', '4', '0', '10', '30'],+ [+ '20260721',+ 'chatgpt.com',+ 'AI Assistant',+ '/a',+ '/a',+ '6',+ '3',+ '0',+ '6',+ '1',+ '5.5',+ '60',+ ],+ [+ '20260721',+ 'chat.mistral.ai',+ 'AI Assistant',+ '/a',+ '/a',+ '2',+ '1',+ '0',+ '2',+ '0',+ '0',+ '8',+ ],+ ],+ });++ const rows = mapAiTrafficRows('brand-1', split);+ expect(rows).toHaveLength(2);+ expect(rows[0]).toMatchObject({+ source: 'chatgpt.com',+ platform: 'chatgpt.com',+ sessions: 10,+ engaged_sessions: 5,+ key_events: 1,+ total_users: 10,+ transactions: 1,+ purchase_revenue: 15.5,+ engagement_duration_seconds: 90,+ });+ // An assistant only GA4's channel knows, under its own name.+ expect(rows[1]).toMatchObject({ source: 'chat.mistral.ai', platform: 'chat.mistral.ai' });+ });+it('keeps the path separately from the full entry URL', () => {// The path is what joins ga_page_stats, which drops query strings to stay// a manageable size; the full URL is what attribution needs.diff --git a/server/src/lib/tracking-script.js b/server/src/lib/tracking-script.jsnew file mode 100644index 000000000..505bf3599--- /dev/null+++ b/server/src/lib/tracking-script.js@@ -0,0 +1,58 @@+/**+ * The tracking snippet (t.js), built from the AI source list (AT-004) so the+ * snippet counts exactly what the Google Analytics sync counts, the same+ * way: a referrer host that is a listed host or one of its subdomains, else+ * a utm_source tag matched the same way (ChatGPT adds utm_source=chatgpt.com+ * to its links). A source with a referrerPath (Bing's /ck/a, its Copilot+ * answers) counts only when the referrer carries that path.+ *+ * It sends the source's key, not the host it saw, so chat.openai.com and+ * chatgpt.com are stored as one engine, and the list's version, stored with+ * the visit.+ *+ * Plain ES5 apart from URL / URLSearchParams, as before: it runs on the+ * customer's pages, in whatever browser their visitors have.+ */++import { AI_SOURCES, AI_SOURCES_VERSION } from './ai-sources.js';++/** The list in the snippet's compact shape: k key, h hosts, a aliases, p referrer path, u 0 = not by utm. */+export function snippetSources() {+ return AI_SOURCES.map((source) => ({+ k: source.key,+ h: source.hosts,+ a: source.aliases,+ ...(source.referrerPath ? { p: source.referrerPath } : {}),+ ...(source.utm === false ? { u: 0 } : {}),+ }));+}++export function buildTrackingScript() {+ return `(function(){+if(window.__at)return;+var S=${JSON.stringify(snippetSources())},V=${JSON.stringify(AI_SOURCES_VERSION)};+var r=document.referrer,h="";+try{h=new URL(r).hostname}catch(e){}+function m(x,utm){+x=String(x||"").replace(/^\\s+|\\s+$/g,"").toLowerCase().replace(/^www\\./,"");+if(!x)return"";+for(var i=0;i<S.length;i++){+var e=S[i];+if(utm?e.u===0:e.p&&r.indexOf(e.p)===-1)continue;+if(e.a.indexOf(x)!==-1)return e.k;+for(var j=0;j<e.h.length;j++){var d=e.h[j];if(x===d||x.slice(-d.length-1)==="."+d)return e.k}+}+return"";+}+var pf=m(h,false);+if(!pf){try{pf=m(new URLSearchParams(location.search).get("utm_source"),true)}catch(e){}}+if(!pf)return;+window.__at=1;+var c=document.currentScript,k=c.getAttribute("data-t"),o=c.src.replace(/\\/t\\.js(\\?.*)?$/,"");+var b=JSON.stringify({u:location.href,r:r,s:pf,v:V,a:navigator.userAgent,t:Date.now(),l:navigator.language,d:screen.width+"x"+screen.height});+var x=o+"/track/"+k;+if(navigator.sendBeacon)navigator.sendBeacon(x,new Blob([b],{type:"text/plain;charset=UTF-8"}));+else fetch(x,{method:"POST",body:b,headers:{"Content-Type":"text/plain;charset=UTF-8"},keepalive:!0}).catch(function(){});+})();+`;+}diff --git a/server/src/lib/tracking-script.test.js b/server/src/lib/tracking-script.test.jsnew file mode 100644index 000000000..a20a29620--- /dev/null+++ b/server/src/lib/tracking-script.test.js@@ -0,0 +1,101 @@+import { runInNewContext } from 'vm';+import { describe, expect, it } from 'vitest';+import { AI_SOURCES, AI_SOURCES_VERSION, classifyAiSource } from './ai-sources.js';+import { buildTrackingScript } from './tracking-script.js';++/**+ * The tracking snippet (AT-004) counts what the AI source list counts, the+ * same way the Google Analytics sync does, and sends the source's key and the+ * list version. Run here in a stand-in browser.+ */++const SCRIPT = buildTrackingScript();++/** Runs t.js on a page; returns the beacon it sent, or null. */+function visit({ referrer = '', search = '', alreadyTracked = false } = {}) {+ const sent = [];+ const window = alreadyTracked ? { __at: 1 } : {};+ runInNewContext(SCRIPT, {+ window,+ document: {+ referrer,+ currentScript: {+ src: 'https://api.ansvisor.test/t.js',+ getAttribute: (name) => (name === 'data-t' ? 'abcdef123' : null),+ },+ },+ location: { href: `https://acme.example/menu${search}`, search },+ navigator: {+ userAgent: 'Mozilla/5.0',+ language: 'en-US',+ sendBeacon: (url, blob) => sent.push({ url, blob }),+ },+ screen: { width: 1440, height: 900 },+ URL,+ URLSearchParams,+ Blob: class {+ constructor(parts) {+ this.text = parts.join('');+ }+ },+ JSON,+ Date,+ String,+ });+ if (sent.length === 0) return null;+ return { url: sent[0].url, ...JSON.parse(sent[0].blob.text) };+}++describe('the tracking snippet', () => {+ it("sends the source's key and the list version to the brand's beacon", () => {+ const beacon = visit({ referrer: 'https://chat.openai.com/c/123' });+ expect(beacon).toMatchObject({+ url: 'https://api.ansvisor.test/track/abcdef123',+ s: 'chatgpt.com',+ v: AI_SOURCES_VERSION,+ r: 'https://chat.openai.com/c/123',+ u: 'https://acme.example/menu',+ });+ });++ it('counts a subdomain, and engines the old snippet missed', () => {+ expect(visit({ referrer: 'https://m.perplexity.ai/' }).s).toBe('perplexity.ai');+ expect(visit({ referrer: 'https://grok.com/' }).s).toBe('grok.com');+ expect(visit({ referrer: 'https://chat.deepseek.com/' }).s).toBe('deepseek.com');+ });++ it('falls back to utm_source, matched the same way', () => {+ expect(+ visit({ referrer: 'https://www.google.com/', search: '?utm_source=chatgpt.com' }).s,+ ).toBe('chatgpt.com');+ expect(visit({ search: '?utm_source=Perplexity' }).s).toBe('perplexity.ai');+ // A name inside another tag isn't an engine.+ expect(visit({ search: '?utm_source=newsletter-chatgpt-tips' })).toBeNull();+ });++ it('counts Bing only from a Copilot answer link', () => {+ expect(visit({ referrer: 'https://www.bing.com/ck/a?!&&p=abc' }).s).toBe('bing.com');+ expect(visit({ referrer: 'https://www.bing.com/search?q=coffee' })).toBeNull();+ expect(visit({ search: '?utm_source=bing' })).toBeNull();+ });++ it('sends nothing for other traffic, or twice on one page', () => {+ expect(visit({ referrer: 'https://notchatgpt.com/' })).toBeNull();+ expect(visit({ referrer: 'https://example.com/' })).toBeNull();+ expect(visit({ referrer: 'https://chatgpt.com/', alreadyTracked: true })).toBeNull();+ });++ it('names every listed host and alias as the server does', () => {+ for (const source of AI_SOURCES) {+ for (const host of source.hosts) {+ const path = source.referrerPath ?? '/';+ expect(visit({ referrer: `https://${host}${path}` })?.s, host).toBe(classifyAiSource(host));+ }+ for (const alias of source.aliases) {+ expect(visit({ search: `?utm_source=${alias}` })?.s ?? null, alias).toBe(+ classifyAiSource(alias, { origin: 'utm' }),+ );+ }+ }+ });+});diff --git a/server/src/public/t.js b/server/src/public/t.jsdeleted file mode 100644index 848b99d7c..000000000--- a/server/src/public/t.js+++ /dev/null@@ -1 +0,0 @@-(function(){if(window.__at)return;var r=document.referrer,h="";try{h=new URL(r).hostname.replace(/^www\./,"")}catch(e){}var d=["chatgpt.com","chat.openai.com","search.chatgpt.com","perplexity.ai","claude.ai","gemini.google.com","copilot.microsoft.com","you.com","phind.com","meta.ai","poe.com","labs.google"],pf="";for(var i=0;i<d.length;i++){var a=d[i];if(h===a||h.endsWith("."+a)){pf=a;break}}if(!pf){var s=(new URLSearchParams(location.search).get("utm_source")||"").toLowerCase();if(s.indexOf("chatgpt")!==-1||s.indexOf("openai")!==-1)pf="chatgpt.com";else if(s.indexOf("perplexity")!==-1)pf="perplexity.ai";else if(s.indexOf("claude")!==-1)pf="claude.ai";else if(s.indexOf("gemini")!==-1)pf="gemini.google.com";else if(s.indexOf("copilot")!==-1)pf="copilot.microsoft.com"}if(!pf&&h.indexOf("bing.com")!==-1&&r.indexOf("/ck/a")!==-1)pf="bing.com";if(!pf)return;window.__at=1;var c=document.currentScript,k=c.getAttribute("data-t"),o=c.src.replace(/\/t\.js(\?.*)?$/,"");var b=JSON.stringify({u:location.href,r:r,s:pf,a:navigator.userAgent,t:Date.now(),l:navigator.language,d:screen.width+"x"+screen.height});var e=o+"/track/"+k;if(navigator.sendBeacon)navigator.sendBeacon(e,new Blob([b],{type:"text/plain;charset=UTF-8"}));else fetch(e,{method:"POST",body:b,headers:{"Content-Type":"text/plain;charset=UTF-8"},keepalive:!0}).catch(function(){})})();diff --git a/server/src/routes/traffic-track.test.js b/server/src/routes/traffic-track.test.jsindex 27a916616..5fdd2f175 100644--- a/server/src/routes/traffic-track.test.js+++ b/server/src/routes/traffic-track.test.js@@ -1,5 +1,6 @@import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';import express from 'express';+import { AI_SOURCES_VERSION } from '../lib/ai-sources.js';/*** POST /track/:trackingCode (AT-003): each pageview is stored with the visit@@ -8,6 +9,7 @@ import express from 'express';const inserted = [];const findVisitOf = vi.fn();+let insertError = null;vi.mock('../config/supabase.js', () => ({default: {@@ -17,7 +19,7 @@ vi.mock('../config/supabase.js', () => ({}),insert: async (row) => {inserted.push({ table, row });- return { error: null };+ return { error: insertError };},}),},@@ -46,6 +48,7 @@ afterAll(() => new Promise((resolve) => server.close(resolve)));beforeEach(() => {inserted.length = 0;+ insertError = null;findVisitOf.mockReset();});@@ -102,4 +105,43 @@ describe('POST /track/:trackingCode', () => {expect(findVisitOf).not.toHaveBeenCalled();expect(inserted).toEqual([]);});++ it("stores an older snippet's host under the AI source list's key (AT-004)", async () => {+ findVisitOf.mockResolvedValue(null);+ await beacon({ s: 'chat.openai.com' });+ expect(findVisitOf).toHaveBeenCalledWith(+ expect.objectContaining({ sourcePlatform: 'chatgpt.com' }),+ );+ expect(inserted[0].row.source_platform).toBe('chatgpt.com');+ // No list version from a snippet that predates it.+ expect(inserted[0].row.source_list_version).toBeNull();+ });++ it('stores the list version the snippet counted the visit with', async () => {+ findVisitOf.mockResolvedValue(null);+ await beacon({ v: AI_SOURCES_VERSION });+ expect(inserted[0].row.source_list_version).toBe(AI_SOURCES_VERSION);+ });++ it('ignores a version the list has never had', async () => {+ findVisitOf.mockResolvedValue(null);+ await beacon({ v: AI_SOURCES_VERSION + 1 });+ await beacon({ v: '2' });+ expect(inserted.map((i) => i.row.source_list_version)).toEqual([null, null]);+ });++ it('keeps a platform the list does not know as it was sent', async () => {+ findVisitOf.mockResolvedValue(null);+ await beacon({ s: 'some-new-assistant.example' });+ expect(inserted[0].row.source_platform).toBe('some-new-assistant.example');+ });++ it('logs a write the database refuses', async () => {+ const { default: logger } = await import('../lib/logger.js');+ findVisitOf.mockResolvedValue(null);+ insertError = { message: 'column "source_list_version" does not exist' };+ const res = await beacon();+ expect(res.status).toBe(204);+ expect(logger.error).toHaveBeenCalledWith({ err: insertError }, 'traffic track insert failed');+ });});diff --git a/server/src/routes/traffic.js b/server/src/routes/traffic.jsindex e7fbf815d..ad7918eec 100644--- a/server/src/routes/traffic.js+++ b/server/src/routes/traffic.js@@ -1,11 +1,10 @@import express, { Router } from 'express';-import { readFileSync } from 'fs';-import { join, dirname } from 'path';-import { fileURLToPath } from 'url';import supabaseAdmin from '../config/supabase.js';import { trafficLimiter } from '../middleware/rate-limiter.js';import logger from '../lib/logger.js';import { findVisitOf } from '../lib/traffic-visits.js';+import { AI_SOURCES_VERSION, classifyAiSource } from '../lib/ai-sources.js';+import { buildTrackingScript } from '../lib/tracking-script.js';const router = Router();@@ -17,15 +16,14 @@ const router = Router();// once a body has already been parsed.const beaconBody = express.text({ type: ['text/plain', 'application/json'], limit: '64kb' });-const __dirname = dirname(fileURLToPath(import.meta.url));-const scriptPath = join(__dirname, '..', 'public', 't.js');-let cachedScript = null;let cachedEtag = null;+// Built once from the AI source list (AT-004); a list change ships with a+// deploy, which restarts the server and changes the ETag.function getScript() {if (!cachedScript) {- cachedScript = readFileSync(scriptPath, 'utf-8');+ cachedScript = buildTrackingScript();cachedEtag = `"${Buffer.from(cachedScript).length.toString(36)}-${Date.now().toString(36)}"`;}return { body: cachedScript, etag: cachedEtag };@@ -116,10 +114,16 @@ router.post('/track/:trackingCode', trafficLimiter, beaconBody, async (req, res)// Drop those defensively so the data has no unattributable rows. The body is// untrusted: only accept a string, and trim so whitespace-only values can't// slip past the emptiness check (a non-string can't reach .slice and crash).- const sourcePlatform = (typeof body.s === 'string' ? body.s : '').trim().slice(0, 255);- if (!sourcePlatform) {+ const rawPlatform = (typeof body.s === 'string' ? body.s : '').trim().slice(0, 255);+ if (!rawPlatform) {return res.status(204).end();}+ // Stored under the AI source list's key (AT-004): an older cached t.js+ // still sends the host it matched, like chat.openai.com for ChatGPT.+ const sourcePlatform = classifyAiSource(rawPlatform) ?? rawPlatform;+ // The list version that counted the visit; older snippets don't send one.+ const sourceListVersion =+ Number.isInteger(body.v) && body.v >= 1 && body.v <= AI_SOURCES_VERSION ? body.v : null;const userAgent = (body.a || '').slice(0, 512) || null;const ipAddress = ip || null;@@ -134,7 +138,7 @@ router.post('/track/:trackingCode', trafficLimiter, beaconBody, async (req, res)logger.warn({ err }, 'traffic visit lookup failed; stored as a new visit');}- await supabaseAdmin.from('ai_traffic_logs').insert({+ const { error: insertError } = await supabaseAdmin.from('ai_traffic_logs').insert({brand_id: brand.id,url: (body.u || '').slice(0, 2048),referrer: (body.r || '').slice(0, 2048) || null,@@ -145,7 +149,10 @@ router.post('/track/:trackingCode', trafficLimiter, beaconBody, async (req, res)language: (body.l || '').slice(0, 16) || null,screen: (body.d || '').slice(0, 16) || null,visit_of: visitOf,+ source_list_version: sourceListVersion,});+ // The beacon can't be retried, so a failed write is at least visible.+ if (insertError) logger.error({ err: insertError }, 'traffic track insert failed');return res.status(204).end();} catch (error) {diff --git a/server/src/scripts/backfill-ga-ai-history.js b/server/src/scripts/backfill-ga-ai-history.jsnew file mode 100644index 000000000..d0a17161b--- /dev/null+++ b/server/src/scripts/backfill-ga-ai-history.js@@ -0,0 +1,40 @@+/**+ * Re-read every mapped brand's Google Analytics AI traffic as far back as+ * storage keeps (AT-004), so the AI Traffic page can fill the days before the+ * tracking snippet was installed from it (spec §5.2).+ *+ * Run: `node src/scripts/backfill-ga-ai-history.js`+ *+ * Options via env:+ * - GA_HISTORY_BRAND limit the run to one brand id+ *+ * A brand mapped from now on gets this history on its first sync. Brands+ * synced before have 90 days, counted with the old source list; this run+ * reads the rest and re-counts every row with the current list (version+ * stamped on each row). Idempotent: rows are upserted on (brand, date,+ * source, entry URL), so it can be run again. It also refreshes the usual+ * last few days of page totals and products, as the daily sync does.+ */++import 'dotenv/config';+import { runGaSync } from '../lib/ga-sync.js';++const ONLY_BRAND = process.env.GA_HISTORY_BRAND || undefined;++async function main() {+ const { synced, skipped, results } = await runGaSync({ brandId: ONLY_BRAND, aiHistory: true });+ for (const result of results) {+ console.log(+ result.error+ ? `FAILED ${result.brandId}: ${result.error}`+ : `${result.brandId} — ${result.aiRows} AI traffic row(s)`,+ );+ }+ console.log(`Backfill complete: ${synced} brand(s), ${skipped} failure(s).`);+ if (skipped > 0) process.exitCode = 1;+}++main().catch((err) => {+ console.error(err);+ process.exit(1);+});diff --git a/supabase/migrations/00175_ai_source_list.sql b/supabase/migrations/00175_ai_source_list.sqlnew file mode 100644index 000000000..a0b38b57b--- /dev/null+++ b/supabase/migrations/00175_ai_source_list.sql@@ -0,0 +1,54 @@+-- ============================================================================+-- 00175: one versioned AI source list for the snippet and Google Analytics (AT-004)+-- ============================================================================+-- The tracking snippet and the Google Analytics sync each had their own list+-- of AI assistants, and they disagreed: the snippet stored chat.openai.com+-- as an engine of its own and didn't know Grok or DeepSeek; the sync didn't+-- know Google Labs or subdomains. Both now read server/src/lib/ai-sources.json+-- (spec §5.1), which names each engine by one key.+--+-- * source_list_version records the list version that counted a row, so a+-- step in the history can be traced to a change of the list rather than+-- of the traffic. Null on rows stored before the list was versioned+-- (version 1: the two separate lists) and on visits from a cached copy of+-- the old snippet.+-- * The snippet's rows stored before it are moved onto the list's keys:+-- chat.openai.com and the bare names some old snippets sent ("chatgpt",+-- "gemini") become chatgpt.com and gemini.google.com, so one engine is+-- one row in every breakdown. Every pageview of a visit carries the same+-- value, so visits stay grouped (00172). Google Analytics rows already+-- use the keys; a history backfill re-reads them with the new list.++alter table public.ai_traffic_logs+ add column if not exists source_list_version smallint;++alter table public.ga_ai_traffic_stats+ add column if not exists source_list_version smallint;++comment on column public.ai_traffic_logs.source_list_version is+ 'The AI source list version (server/src/lib/ai-sources.json) that counted this visit; null before the list was versioned or from a cached older snippet (AT-004).';++comment on column public.ga_ai_traffic_stats.source_list_version is+ 'The AI source list version (server/src/lib/ai-sources.json) that classified this row; null before the list was versioned (AT-004).';++update public.ai_traffic_logs t+set source_platform = m.key+from (+ values+ ('chat.openai.com', 'chatgpt.com'),+ ('search.chatgpt.com', 'chatgpt.com'),+ ('chatgpt', 'chatgpt.com'),+ ('openai', 'chatgpt.com'),+ ('openai.com', 'chatgpt.com'),+ ('perplexity', 'perplexity.ai'),+ ('www.perplexity.ai', 'perplexity.ai'),+ ('claude', 'claude.ai'),+ ('gemini', 'gemini.google.com'),+ ('bard', 'gemini.google.com'),+ ('bard.google.com', 'gemini.google.com'),+ ('copilot', 'copilot.microsoft.com'),+ ('grok', 'grok.com'),+ ('deepseek', 'deepseek.com'),+ ('chat.deepseek.com', 'deepseek.com')+) as m (raw, key)+where lower(t.source_platform) = m.raw;diff --git a/supabase/schema.sql b/supabase/schema.sqlindex be07de880..3476a2d0a 100644--- a/supabase/schema.sql+++ b/supabase/schema.sql@@ -23602,3 +23602,61 @@ comment on function public.gsc_click_totals(uuid, date, date, boolean) isnotify pgrst, 'reload schema';+-- ─────────────────────────────────────────────────────────────────────────+-- migrations/00175_ai_source_list.sql+-- ─────────────────────────────────────────────────────────────────────────+-- ============================================================================+-- 00175: one versioned AI source list for the snippet and Google Analytics (AT-004)+-- ============================================================================+-- The tracking snippet and the Google Analytics sync each had their own list+-- of AI assistants, and they disagreed: the snippet stored chat.openai.com+-- as an engine of its own and didn't know Grok or DeepSeek; the sync didn't+-- know Google Labs or subdomains. Both now read server/src/lib/ai-sources.json+-- (spec §5.1), which names each engine by one key.+--+-- * source_list_version records the list version that counted a row, so a+-- step in the history can be traced to a change of the list rather than+-- of the traffic. Null on rows stored before the list was versioned+-- (version 1: the two separate lists) and on visits from a cached copy of+-- the old snippet.+-- * The snippet's rows stored before it are moved onto the list's keys:+-- chat.openai.com and the bare names some old snippets sent ("chatgpt",+-- "gemini") become chatgpt.com and gemini.google.com, so one engine is+-- one row in every breakdown. Every pageview of a visit carries the same+-- value, so visits stay grouped (00172). Google Analytics rows already+-- use the keys; a history backfill re-reads them with the new list.++alter table public.ai_traffic_logs+ add column if not exists source_list_version smallint;++alter table public.ga_ai_traffic_stats+ add column if not exists source_list_version smallint;++comment on column public.ai_traffic_logs.source_list_version is+ 'The AI source list version (server/src/lib/ai-sources.json) that counted this visit; null before the list was versioned or from a cached older snippet (AT-004).';++comment on column public.ga_ai_traffic_stats.source_list_version is+ 'The AI source list version (server/src/lib/ai-sources.json) that classified this row; null before the list was versioned (AT-004).';++update public.ai_traffic_logs t+set source_platform = m.key+from (+ values+ ('chat.openai.com', 'chatgpt.com'),+ ('search.chatgpt.com', 'chatgpt.com'),+ ('chatgpt', 'chatgpt.com'),+ ('openai', 'chatgpt.com'),+ ('openai.com', 'chatgpt.com'),+ ('perplexity', 'perplexity.ai'),+ ('www.perplexity.ai', 'perplexity.ai'),+ ('claude', 'claude.ai'),+ ('gemini', 'gemini.google.com'),+ ('bard', 'gemini.google.com'),+ ('bard.google.com', 'gemini.google.com'),+ ('copilot', 'copilot.microsoft.com'),+ ('grok', 'grok.com'),+ ('deepseek', 'deepseek.com'),+ ('chat.deepseek.com', 'deepseek.com')+) as m (raw, key)+where lower(t.source_platform) = m.raw;+diff --git a/web/src/app/[locale]/(dashboard)/dashboard/traffic/_charts.tsx b/web/src/app/[locale]/(dashboard)/dashboard/traffic/_charts.tsxindex 8465ae93b..93c94571b 100644--- a/web/src/app/[locale]/(dashboard)/dashboard/traffic/_charts.tsx+++ b/web/src/app/[locale]/(dashboard)/dashboard/traffic/_charts.tsx@@ -50,12 +50,12 @@ function ChartContainer({// ─── Platform colors ─────────────────────────────────────────────────────────//-// Keys must stay in sync with the detection host list in the tracking snippet-// (`server/src/public/t.js`): every platform t.js can resolve and store needs a+// Keys are the AI source list's (`server/src/lib/ai-sources.json`), which the+// tracking snippet and the Analytics sync both store: every key there needs a// color here and a name in `@/lib/traffic-platforms`, otherwise it renders as-// a raw hostname. `unknown` is kept-// only for historical rows that predate #288 (null source_platform); new visits-// always carry a concrete platform.+// a grey raw hostname (traffic-platforms.test.ts checks). The other ChatGPT+// hosts are kept for rows from before the list (AT-004), and `unknown` for+// rows that predate #288 (null source_platform).const PLATFORM_COLORS: Record<string, string> = {'chatgpt.com': '#6366f1',@@ -71,10 +71,12 @@ const PLATFORM_COLORS: Record<string, string> = {'poe.com': '#e9d5ff','labs.google': '#b794f6','bing.com': '#67c5d6',+ 'grok.com': '#4f46e5',+ 'deepseek.com': '#9333ea',unknown: '#94a3b8',};-function getColor(platform: string): string {+export function getColor(platform: string): string {return PLATFORM_COLORS[platform] ?? '#94a3b8';}diff --git a/web/src/lib/traffic-platforms.test.ts b/web/src/lib/traffic-platforms.test.tsnew file mode 100644index 000000000..429c1e90a--- /dev/null+++ b/web/src/lib/traffic-platforms.test.ts@@ -0,0 +1,30 @@+import { readFileSync } from 'node:fs';+import { join } from 'node:path';+import { describe, expect, it } from 'vitest';+import { getPlatformName } from './traffic-platforms';+import { getColor } from '@/app/[locale]/(dashboard)/dashboard/traffic/_charts';++/**+ * Every engine on the AI source list (server/src/lib/ai-sources.json, AT-004)+ * has a name and a color on the AI Traffic page; otherwise a new engine would+ * show as a grey raw hostname.+ */++const list = JSON.parse(+ readFileSync(join(__dirname, '../../../server/src/lib/ai-sources.json'), 'utf8'),+) as { sources: { key: string }[] };++describe('the AI source list on the page', () => {+ it('names every engine on the list', () => {+ for (const { key } of list.sources) {+ expect(getPlatformName(key), key).not.toBe(key);+ }+ });++ it('colors every engine on the list', () => {+ const fallback = getColor('not-an-engine.example');+ for (const { key } of list.sources) {+ expect(getColor(key), key).not.toBe(fallback);+ }+ });+});diff --git a/web/src/lib/traffic-platforms.ts b/web/src/lib/traffic-platforms.tsindex a0513009a..3b50acccf 100644--- a/web/src/lib/traffic-platforms.ts+++ b/web/src/lib/traffic-platforms.ts@@ -1,8 +1,9 @@/**- * AI referrers by the name their platform goes by. Keys are what the- * tracking snippet (`server/src/public/t.js`) and the Analytics sync- * (`server/src/lib/ga-sources.js`) store; `unknown` is kept only for- * historical rows that predate #288 (null source_platform).+ * AI referrers by the name their platform goes by. Keys are the AI source+ * list's (`server/src/lib/ai-sources.json`), which the tracking snippet and+ * the Analytics sync both store (AT-004). The other ChatGPT hosts are kept+ * for rows from before the list, and `unknown` for historical rows that+ * predate #288 (null source_platform).*/const PLATFORM_NAMES: Record<string, string> = {'chatgpt.com': 'ChatGPT',