Skip to content

armgit / crm

Public Imported from GitHub
0

Comp AI CRM is an open source, CRM designed for AI agents. Agentic-first CRM.

TypeScript

Report installs without a cron, and stop double counting them

Lewis Carhart2975c4f34d79
diff --git a/apps/api/src/telemetry/rollup.service.ts b/apps/api/src/telemetry/rollup.service.ts
index bcd129cd..91df3f36 100644
--- a/apps/api/src/telemetry/rollup.service.ts
+++ b/apps/api/src/telemetry/rollup.service.ts
@@ -81,7 +81,7 @@ export class RollupService {
const gathered = await this.gather(since);
counters = gathered.counters;
- if (!(await installDaily(gathered.properties))) {
+ if (!(await installDaily(gathered.properties, now))) {
await this.giveBack(claim.previous, counters);
return { sent: false, reason: "not delivered", milestones };
diff --git a/docs/telemetry.md b/docs/telemetry.md
index 322b14d6..8c70272d 100644
--- a/docs/telemetry.md
+++ b/docs/telemetry.md
@@ -62,6 +62,20 @@ fire at once produce one event rather than two — and the second does not find
already drained. A rollup that fails to send hands the day and the counters back, so tomorrow's
run is not suppressed by a send that never happened.
+**The event carries a deterministic `uuid`**, `stableUuid(install.uuid, "install_daily", day)` — a
+v5-shaped digest — and PostHog ingests a repeat of one it has already seen only once. The lock
+stops two processes racing; the id stops the *same* process sending twice, which the release path
+can otherwise cause. Counting installs survives a duplicate either way, because PostHog's unique
+math is per install per day, but `tool_calls_total` and the other summed properties would be
+counted twice, and now are not.
+
+**A send reports failure conservatively.** `posthog-node` does not reject on a failed send — its
+`sendImmediate` catches the error and emits it on the client instead — so `client.ts` enqueues and
+awaits `flush()`, which does throw, and additionally checks the client's error counter, which any
+capture in flight can move. Either signal is treated as a failure. That errs toward re-sending a
+rollup that actually landed, which the deterministic id makes free, rather than consuming a day
+whose event never arrived, which cannot be recovered.
+
#### The install
| Property | What it is |
@@ -166,11 +180,20 @@ the cron, so an install that arrives configured is stamped on its first boot.
applied and later superseded still count — being replaced later does not make the first
application not have happened.
-A step is only recorded as reached once the event has actually left, so a failed send is tried
-again on the next sweep rather than being consumed and lost forever. The cost of that ordering
-is that a crash between the send and the write — or two processes sweeping at the same instant
-— can repeat a step. A one-shot event that arrives twice is a rounding error in a funnel; one
-that never arrives cannot be recovered. Nothing is recorded at all while telemetry is off.
+**The step is claimed before it is sent, and handed back if the send fails.** The claim is the
+insert itself — `telemetryMilestone.step` is the primary key, so of two processes sweeping at the
+same instant exactly one is told it landed the row, and only that one sends. A failed send deletes
+the row, so the next sweep tries again rather than losing the step forever.
+
+This is the opposite of the ordering the funnel shipped with, which sent first and recorded after.
+That was chosen on the grounds that a duplicate is a rounding error while a lost step cannot be
+recovered — but it made a duplicate the *likely* outcome rather than a rare one, because both the
+boot sweep and the rollup call `sweep()`. One install sent `first_fact_applied` six times.
+Claiming first costs the case the old ordering was protecting against: a crash between the claim
+and the send drops the step. Each event also carries a deterministic `uuid`,
+`stableUuid(install.uuid, step)`, so a step that is genuinely sent twice is still ingested once.
+
+Nothing is recorded at all while telemetry is off.
### Errors
diff --git a/packages/telemetry/src/client.ts b/packages/telemetry/src/client.ts
index c9f7a014..ad0e47c4 100644
--- a/packages/telemetry/src/client.ts
+++ b/packages/telemetry/src/client.ts
@@ -104,8 +104,9 @@ export async function captureNow(
event: string,
properties: Properties = {},
at?: Date,
+ uuid?: string,
): Promise<boolean> {
- return send(event, properties, true, at);
+ return send(event, properties, true, at, uuid);
}
async function send(
@@ -113,6 +114,7 @@ async function send(
properties: Properties,
immediate = false,
at?: Date,
+ uuid?: string,
): Promise<boolean> {
try {
if (telemetryDisabled()) return false;
@@ -127,11 +129,18 @@ async function send(
...message,
event,
...(at ? { timestamp: at } : {}),
+ ...(uuid ? { uuid } : {}),
} as Parameters<PostHog["capture"]>[0];
if (immediate) {
const before = failures;
- await posted.captureImmediate(full);
+ posted.capture(full);
+
+ try {
+ await posted.flush();
+ } catch {
+ return false;
+ }
return failures === before;
}
diff --git a/packages/telemetry/src/events.ts b/packages/telemetry/src/events.ts
index e64daa14..390193d1 100644
--- a/packages/telemetry/src/events.ts
+++ b/packages/telemetry/src/events.ts
@@ -9,7 +9,14 @@ import {
} from "./allowlist";
import { capture, captureNow } from "./client";
import { telemetryDisabled } from "./disabled";
-import { type Milestone, reachMilestone } from "./install";
+import {
+ forgetMilestone,
+ type Milestone,
+ reachMilestone,
+ readInstall,
+ stableUuid,
+ utcDay,
+} from "./install";
export const INSTALL_DAILY = "install_daily";
@@ -21,17 +28,37 @@ export const API_ERROR = "api_error";
export const MODEL_ERROR = "model_error";
-export async function installDaily(properties: Properties): Promise<boolean> {
- return captureNow(INSTALL_DAILY, properties);
+export async function installDaily(
+ properties: Properties,
+ at = new Date(),
+): Promise<boolean> {
+ const install = await readInstall();
+
+ return captureNow(
+ INSTALL_DAILY,
+ properties,
+ undefined,
+ install ? stableUuid(install.uuid, INSTALL_DAILY, utcDay(at)) : undefined,
+ );
}
export async function milestone(step: Milestone, at?: Date): Promise<boolean> {
if (telemetryDisabled()) return false;
+ if (!(await reachMilestone(step))) return false;
+
+ const install = await readInstall();
- const sent = await captureNow(step, {}, at);
- if (!sent) return false;
+ const sent = await captureNow(
+ step,
+ {},
+ at,
+ install ? stableUuid(install.uuid, step) : undefined,
+ );
- await reachMilestone(step);
+ if (!sent) {
+ await forgetMilestone(step);
+ return false;
+ }
return true;
}
diff --git a/packages/telemetry/src/index.ts b/packages/telemetry/src/index.ts
index 3f412ba2..96be3713 100644
--- a/packages/telemetry/src/index.ts
+++ b/packages/telemetry/src/index.ts
@@ -49,6 +49,7 @@ export {
daysSince,
drainCounters,
forgetInstall,
+ forgetMilestone,
INSTALL_ID,
type Install,
MILESTONES,
@@ -60,7 +61,9 @@ export {
releaseRollup,
restoreCounters,
sameUtcDay,
+ stableUuid,
syncVersion,
+ utcDay,
} from "./install";
export { POSTHOG_HOST, POSTHOG_KEY, POSTHOG_UI_HOST } from "./project";
export { commitSha, crmVersion } from "./version";
diff --git a/packages/telemetry/src/install.ts b/packages/telemetry/src/install.ts
index 0c8757b4..decab68f 100644
--- a/packages/telemetry/src/install.ts
+++ b/packages/telemetry/src/install.ts
@@ -1,5 +1,6 @@
import "@crm/env/load";
+import { createHash } from "node:crypto";
import { db } from "@crm/db";
import { crmVersion } from "./version";
@@ -97,6 +98,12 @@ export async function reachMilestone(step: Milestone): Promise<boolean> {
}
}
+export async function forgetMilestone(step: Milestone): Promise<void> {
+ try {
+ await db.telemetryMilestone.delete({ where: { step } });
+ } catch {}
+}
+
export async function reachedMilestones(): Promise<Milestone[]> {
try {
const rows = await db.telemetryMilestone.findMany({
@@ -210,6 +217,28 @@ export function daysSince(from: Date, now = new Date()): number {
return Math.max(0, Math.floor((now.getTime() - from.getTime()) / DAY_MS));
}
+export function utcDay(at: Date): string {
+ return at.toISOString().slice(0, 10);
+}
+
+export function stableUuid(...parts: string[]): string {
+ const digest = createHash("sha1").update(parts.join(":")).digest();
+ const bytes = Uint8Array.from(digest.subarray(0, 16));
+
+ bytes[6] = ((bytes[6] ?? 0) & 0x0f) | 0x50;
+ bytes[8] = ((bytes[8] ?? 0) & 0x3f) | 0x80;
+
+ const hex = Buffer.from(bytes).toString("hex");
+
+ return [
+ hex.slice(0, 8),
+ hex.slice(8, 12),
+ hex.slice(12, 16),
+ hex.slice(16, 20),
+ hex.slice(20, 32),
+ ].join("-");
+}
+
export function sameUtcDay(a: Date | null, b: Date): boolean {
if (!a) return false;
diff --git a/packages/telemetry/test/client.integration.spec.ts b/packages/telemetry/test/client.integration.spec.ts
index 8181a53e..acbea24a 100644
--- a/packages/telemetry/test/client.integration.spec.ts
+++ b/packages/telemetry/test/client.integration.spec.ts
@@ -2,7 +2,7 @@ import { afterEach, beforeEach, describe, expect, it } from "bun:test";
import { db } from "@crm/db";
import { captureNow, resetTelemetryClient } from "../src/client";
import { milestone } from "../src/events";
-import { forgetInstall, readInstall } from "../src/install";
+import { forgetInstall, readInstall, stableUuid } from "../src/install";
import { POSTHOG_HOST } from "../src/project";
const real = {
@@ -201,6 +201,33 @@ describe("milestone", () => {
expect(await db.telemetryMilestone.count({ where: { step } })).toBe(1);
});
+ it("sends once when two processes sweep at the same instant", async () => {
+ const [first, second] = await Promise.all([
+ milestone(step),
+ milestone(step),
+ ]);
+
+ expect([first, second].filter(Boolean).length).toBe(1);
+ expect(sentSteps()).toEqual([step]);
+ expect(await db.telemetryMilestone.count({ where: { step } })).toBe(1);
+ });
+
+ it("carries an id that stays the same across a resent step", async () => {
+ globalThis.fetch = (async () => {
+ throw new Error("network is down");
+ }) as typeof fetch;
+
+ await milestone(step);
+
+ stubFetch();
+ resetTelemetryClient();
+ await milestone(step);
+
+ const resent = eventOf(calls[0]?.body);
+ expect(resent.event).toBe(step);
+ expect(resent.uuid).toBe(stableUuid(await installUuid(), step));
+ });
+
it("leaves the step unreached when telemetry is off", async () => {
process.env.CRM_TELEMETRY_DISABLED = "1";
resetTelemetryClient();
@@ -214,6 +241,7 @@ describe("milestone", () => {
type Captured = {
event: string;
distinct_id: string;
+ uuid?: string;
properties: Record<string, unknown>;
};
@@ -224,6 +252,18 @@ function eventOf(body: unknown): Captured {
return {
event: event?.event,
distinct_id: event?.distinct_id,
+ uuid: event?.uuid,
properties: event?.properties ?? {},
};
}
+
+function sentSteps(): string[] {
+ return calls.map((call) => eventOf(call.body).event).filter(Boolean);
+}
+
+async function installUuid(): Promise<string> {
+ const install = await readInstall();
+ if (!install) throw new Error("no install row");
+
+ return install.uuid;
+}
diff --git a/packages/telemetry/test/stable-uuid.spec.ts b/packages/telemetry/test/stable-uuid.spec.ts
new file mode 100644
index 00000000..59c863b9
--- /dev/null
+++ b/packages/telemetry/test/stable-uuid.spec.ts
@@ -0,0 +1,59 @@
+import { describe, expect, it } from "bun:test";
+import { stableUuid, utcDay } from "../src/install";
+
+const INSTALL = "cdf04e64-7722-4921-8b28-be936ce36353";
+
+const OTHER = "c4b8d582-ec43-44f1-9a73-7b331eec90c7";
+
+const UUID_SHAPE =
+ /^[0-9a-f]{8}-[0-9a-f]{4}-5[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
+
+describe("utcDay", () => {
+ it("is the same string across a whole utc day", () => {
+ expect(utcDay(new Date("2026-08-05T00:00:00.000Z"))).toBe("2026-08-05");
+ expect(utcDay(new Date("2026-08-05T23:59:59.999Z"))).toBe("2026-08-05");
+ });
+
+ it("turns over at midnight utc", () => {
+ expect(utcDay(new Date("2026-08-06T00:00:00.000Z"))).toBe("2026-08-06");
+ });
+});
+
+describe("stableUuid", () => {
+ it("is a well formed v5 uuid", () => {
+ expect(stableUuid(INSTALL, "install_daily", "2026-08-05")).toMatch(
+ UUID_SHAPE,
+ );
+ });
+
+ it("repeats for the same parts, so a retry dedupes", () => {
+ const first = stableUuid(INSTALL, "install_daily", "2026-08-05");
+ const retry = stableUuid(INSTALL, "install_daily", "2026-08-05");
+
+ expect(first).toBe(retry);
+ });
+
+ it("changes with the day", () => {
+ expect(stableUuid(INSTALL, "install_daily", "2026-08-05")).not.toBe(
+ stableUuid(INSTALL, "install_daily", "2026-08-06"),
+ );
+ });
+
+ it("changes with the install", () => {
+ expect(stableUuid(INSTALL, "install_daily", "2026-08-05")).not.toBe(
+ stableUuid(OTHER, "install_daily", "2026-08-05"),
+ );
+ });
+
+ it("separates a milestone from a rollup on the same install", () => {
+ expect(stableUuid(INSTALL, "first_fact_applied")).not.toBe(
+ stableUuid(INSTALL, "install_daily", "2026-08-05"),
+ );
+ });
+
+ it("gives a milestone one id for the life of the install", () => {
+ expect(stableUuid(INSTALL, "first_fact_applied")).toBe(
+ stableUuid(INSTALL, "first_fact_applied"),
+ );
+ });
+});