Comp AI CRM is an open source, CRM designed for AI agents. Agentic-first CRM.
Website tracking
A first-party script on the customer's own marketing site, a collector in the API, and one rule about what it is for: a form submission becomes a contact. Page views exist to give that contact a story, not to be a web-analytics product.
Everything here is one install's own website. There is no second tenant, no shared
pixel, and no vendor: the script is served from the same origin as the app, the
cookie is first-party, and the only thing that ever leaves the browser is a POST to
/api/t/e on the install's own API.
Two scripts, and why
apps/app/lib/tracking/loader.ts → /t/crm.js |
The tag a rep pastes. Reads the site id from data-site or from ?site=, checks the shape, injects the second script. Immutable and cached for a year at the edge |
apps/app/lib/tracking/tracker.ts → /t/<siteId>.js |
The tracker itself, with the config baked into the source rather than fetched. Cached for five minutes |
The split is the whole cache design. The tag never changes, so it is immutable
and free forever; the config does change, so the file that carries it is the one
with the short life. Baking the config in also means a page view costs one request,
not a request and then a config fetch before anything can be recorded.
- The site id has two carriers, and a tag manager is the reason. Google Tag
Manager's Custom HTML injector rebuilds the script element and keeps only the
URL —
data-site,asyncanddeferare all dropped on the way in. An attribute-only tag therefore loads/t/crm.js, finds no site, and returns: the loader is in the network tab, the tracker never is, and the install looks installed while recording nothing.?site=rides in thesrc, which no injector can strip.data-sitestill wins when both are present, so a pasted tag beats a stale URL. Either way the loader stays config-free — baking the install's own id into it would put a year-immutable edge cache in front of the rotate kill switch. - Five minutes is a promise. Pause tracking and every browser stops within
CONFIG_MAX_AGE_SECONDS. That is why/t/[site]carries nostale-while-revalidate— a revalidation window is exactly a licence to keep executing the old config after the pause, and 24 hours of it once cost this guarantee entirely. - Both routes are anonymous, listed in
proxy.tsasANONYMOUS = ["/t"]. A stranger's browser on the customer's marketing site has no session and must not be redirected to/sign-in. - The tracker has a size budget, asserted in
apps/app/test/tracking-bundle.spec.ts: 1 KB brotli for the loader, 4 KB for the tracker, because the settings page promises a number. A change that busts it fails the test rather than the promise.
Writing tracker source
It is a string of ES5 in a template literal, minified by hand, and it runs on somebody else's page. That imposes rules nothing else in this repo has:
- Never throw. An uncaught error on line 40 means no listeners are installed and
the page records nothing at all. Every
decodeURIComponent,JSON.parse,new URLandhistorycall is already wrapped; a malformed UTM parameter must read as absent, never as fatal. - Constants come from
@crm/db/tracking, interpolated in —MAX_BODY_BYTES,MAX_EVENTS_PER_BATCH,LINKER_MAX_AGE_SECONDS,COOKIE_NAME. A literal at the call site is a client and a server that disagree, and the disagreement is silent. - The client stays inside the body limit the collector enforces.
flush()measures the packed batch and splits it, then trims a single oversized form's fields, rather than posting a body the collector will destroy — a rejected POST is not retried, so the whole batch would simply be lost.
The collector
POST /api/t/e, anonymous, 204, in TrackingController. It answers nothing: a
tracker that could read a response is a tracker whose failures a stranger can probe.
It is the one route that sets Cross-Origin-Resource-Policy: cross-origin, and
it must. helmet() puts same-origin on every response, which is right for an API
only its own app calls — but this one is called by a no-cors beacon on somebody
else's marketing site, so Chrome blocks the reply with
ERR_BLOCKED_BY_RESPONSE.NotSameOrigin and logs a failure under every page view.
The header is set on the response, not switched off in helmet, so the exception
stays with the route that needs it.
The gauntlet, in order, in TrackingIngestService.accept:
- User agent — the
BOTpattern. - Site id — must be a live
cmp_id, andforSiterefuses a rotated one. - Origin —
originAllowed. A missingOriginheader is refused even with the allow list off, because the header is the only thing tying the POST to a browser on a page. - Visitor id — 8–64 of
[a-zA-Z0-9_-], or the batch is dropped. scripted()— three or more events sharing one timestamp is a replay. It only fires when every event carries a numericat; missing timestamps are no signal, and treating them as proof discarded honest traffic.- Host — each event's own host against the allow list. The batch is filtered, not refused, because one bad host in twenty is a bug in somebody's SPA.
- Rate —
EVENTS_PER_MINUTEcharged per accepted event, atomically, throughTrackingCounter.
The rate limit counts events, not requests, and it counts them in the database. A per-request counter with a full batch behind it admits twenty times the number on the constant, and a read-then-write in
cache-manageradmits however many requests are in flight.TrackingCounterService.take(key, limit, amount)is one statement — anINSERT … ON CONFLICT DO UPDATEwhoseWHEREcarries the limit — so the check and the charge cannot be separated, and a refused batch spends nothing. Increment-then-compare looks equivalent and is not: the refused batch still burns its own size out of the window, so one oversized batch locks out the honest small ones behind it for the rest of the minute. It fails closed: a counter it cannot reach refuses the write.
MAX_BODY_BYTES is enforced by destroying the request as it streams, before any
parse. Keep it in step with what the tracker can produce — a form is forty fields
of five hundred characters, and a limit under that silently drops the submission
this whole feature exists to catch.
What is never stored
- No query strings.
normalizePathstrips?and#from every path, andstripQuerydoes the same to every referrer, on the event row and inside thefirstTouch/lastTouchblobs. A password-reset link in a referrer is a password-reset link in the database. UTM parameters are captured as their own fields, so nothing is lost by it. - No sensitive fields.
clean()drops any field whose name matchesSENSITIVE, anything shaped like a card number, and everypassword,hiddenandfileinput — the last three never leave the browser at all. - No IP address, anywhere, ever.
Attribution
packages/db/src/attribution.ts, and it is pure: no database, no config, one
function. classifyTouch turns UTM parameters and a referrer into a Touch.
- An explicit
utm_sourcebeats the referrer, because the marketer said so. - The referrer host is matched on DNS labels, never on a substring, and the
provider's labels must sit at the registrable name — one tail label, or two when
the first is a second-level suffix and the last is a two-letter country code
(
co.uk,com.au,co.jp). Soimages.google.deandscholar.google.co.jpare Google, whilenotgoogle.com,google.evil.com,google.com.phish.exampleandgoogle.com.exampleare all plain referrals. A substring match hands a phisher a trusted source name in the rep's report, and so doescom.<anything>: the country code is what stopsSECOND_LEVELfrom waving through everycom.lookalike. It is still a heuristic, not the public suffix list — a provider under an unusual two-label suffix reads as a referral, which is the safe way to be wrong. - Webmail is checked before search.
mail.google.comcontainsgoogle., so order alone decides whether a campaign's own click-throughs read as email or as organic search — and reading your newsletter as Google organic is how a marketing report lies. Directis a source, not a null. Everything lands in one of the sevenMEDIUMS; an unrecognised medium isother.
Filing: from a submission to a contact
TrackingFilingService.file is the only path from a form to a Contact, and it
reuses the mailbox pipeline's judgement rather than inventing a second one —
isMachineAddress, isAutomatedAddress, isMachineDomain, workspaceDomains,
SuppressedContact, SuppressedDomain, companyForEmail, splitName. A rule
that holds for the inbox holds here. A second copy is how deleted contacts stay
deleted comes to be true of email and not of forms.
- Every submission is stored; filing is separate and may decline.
skipReasonsays why, and the row stays for a rep to look at. - A refusal is a
skipReason, never a lost row, andCONTACT_CAP_REASONis a shared constant becauserollup.service.tsmatches on it. Telemetry that substring-matches prose breaks the first time somebody improves the wording. CONTACTS_PER_HOURbounds the blast radius of a scripted form. It is charged only when a new contact would be created — an existing contact costs nothing, because attaching to somebody already in the CRM cannot flood it — and a create that loses the race hands its slot back, so duplicate delivery of one form cannot eat the hour's quota for a contact that was only ever made once.- The email race is handled, not hoped away. Two submissions for one address
land at once, one
createloses on the unique index, and the loser attaches to the contact that won. Left as a rawP2002it becomes a stored submission that is never filed and never retried. attachclaims the row before it writes anything. It is anupdateManyonfiledAt: null, and a caller that loses the claim returns without writing — otherwise two concurrent deliveries of one submission each leave the contact aSubmitted a form on …note.- Duplicate delivery retries an unfiled row.
dedupeKeycollapses a resend inside one minute, but if the first attempt died before it filed — nofiledAt, noskipReason— the resend is the retry. - The agent is told, never asked.
agent.contactCreatedwrites anAgentTask. No enrichment, no scoring and no identity matching happens here; seedocs/api.md.
Retention
POST /internal/tracking/retention, nightly at 04:00 via apps/api/vercel.json,
CRON_SECRET or nothing.
- The cutoff is a whole UTC day,
EVENT_RETENTION_DAYSback and then truncated. A mid-day cutoff splits one calendar day across two nightly runs, andtrackedPageDailykeeps the larger half of a day it saw twice — so the boundary day is quietly under-reported forever. Roll whole days or do not roll. - Roll before you delete.
TrackingRollupServiceaggregatespage_viewrows only; a click is not a page view, and counting visitors over both produces a row readingviews = 0, visitors = 3. - Deleting is batched and bounded —
SWEEP_BATCH×MAX_SWEEP_PASSES. A sweep that hits the ceiling says so, in the log and in the response, because silently leaving events behind reads exactly like having deleted them. - A visitor outlives their events only if a contact points at them. An anonymous visitor with nothing left is removed.
Settings, and who may change them
Settings → Tracking & Analytics. canManageTracking (@crm/auth) gates every
mutation in the service, and the same flag disables the control — the button and
the 403 cannot disagree, as everywhere else.
tracking.sources is manager-only too, so the page must not prefetch it for
everybody: a member's render would fire a request that can only be refused.
- Verify reports on the page it actually fetched.
safeFetchfollows redirects, so the host in the result comes fromfetched.url, not from what the rep typed — otherwiseacme.comreports the allow-list status of a page that lives onwww.acme.com. - Verify reads Tag Manager containers, not only the HTML. A tag a container
injects is not in the response, so an HTML-only check calls a working install
broken — and it did. When the HTML carries no tag,
verifytakes theGTM-…ids out of it and readsgtmContainerUrl(id), at mostMAX_CONTAINERSof them. The URL is built from the id we matched, never from anything a rep typed, so the one host this can ever reach is Google's. A container holding the attribute form is reported as found and broken, with the fix, because that is precisely what it is — andmissingnames the containers it read, so a rep can tell not installed from not looked at. MAX_VERIFY_BYTESis measured against real marketing pages. The old 512 KB cut was smaller than one homepage this repo's own company ships, and a tag past the cut reads exactly like a tag that is not there.- Rotating the site id is the kill switch for a stolen snippet. The old id stops
resolving at
forSitewithin the cache TTL. - The compiled config is cached for five minutes and invalidated on every write.
Two things guard the cache, because one is not enough.
invalidate()bumps a per-process generation before it deletes, which stops this replica's own in-flight read putting the pre-pause config back. That counter means nothing to a second replica, so nothing is written to the cache unless its hash still equalsAppSetting.trackingConfigHash— the shared version, in the database, that every replica can see. A read that raced a change computes the old hash, finds it no longer current, and declines to cache it. Pausing sets the column tonull, so while tracking is paused no config can validate and none is cached at all.
Where it lives
packages/db/src/tracking.ts |
Every constant and every pure helper both sides share. The single source of truth — a literal copied out of here is a future divergence |
packages/db/src/attribution.ts |
classifyTouch and the source tables. No imports, no database |
apps/app/lib/tracking/ |
The loader and the tracker source |
apps/app/app/t/ |
The two public routes |
apps/api/src/tracking/ |
Collector, config cache, ingest, filing, counters, rollup, retention, tRPC router |
apps/app/app/(app)/[slug]/settings/tracking/ |
The settings page |
apps/app/components/crm/website-activity.tsx |
The record-sheet section, which renders its own heading so it can render nothing at all |