Skip to content

A personal agent with a browser, terminal, files, and work that keeps going built with CopilotKit and AG-UI.

TypeScript

Enable OpenMuse SDK telemetry for Intelligence signup reporting (#144)

Jerel Velarde73a714963b57
diff --git a/.env.example b/.env.example
index 86655dd..d5ba85e 100644
--- a/.env.example
+++ b/.env.example
@@ -50,6 +50,17 @@ WORKER_HOST=127.0.0.1
# npx copilotkit@latest project select
CPK_INTELLIGENCE_API_KEY=
+# CopilotKit SDK telemetry goes through the shared sink to PostHog, tagged OpenMuse.
+# project select also writes CPK_TELEMETRY_ID: preserve that generated value and
+# deploy it alongside CPK_INTELLIGENCE_API_KEY to link runtime usage to the account.
+# Do not invent an identity or copy one from another project.
+# CPK_TELEMETRY_ID= # generated by the CLI; keep server-only
+# Full funnel capture is the default; an explicit sample rate overrides it.
+# COPILOTKIT_TELEMETRY_SAMPLE_RATE=1
+# Opt out of SDK telemetry with either flag (true or 1), then restart the API:
+# COPILOTKIT_TELEMETRY_DISABLED=true
+# DO_NOT_TRACK=1
+
# Optional private Docker Linux computer (separate from the browser).
# Build first: docker build -t openmuse-computer:local apps/computer
# Docker must be reachable from the API and any separate task worker.
diff --git a/README.md b/README.md
index dc62ea9..c1c6982 100644
--- a/README.md
+++ b/README.md
@@ -73,6 +73,7 @@ cp .env.example .env
npx copilotkit@latest login
npx copilotkit@latest project select
# Set CPK_INTELLIGENCE_API_KEY in .env to the generated server-only project key.
+# Keep the CLI-generated CPK_TELEMETRY_ID too, including in deployed server environments.
pnpm dev
```
@@ -106,11 +107,12 @@ For iOS or Android, use `pnpm --dir apps/mobile ios` or `pnpm --dir apps/mobile
3. Open the `openmuse-web` URL and sign in with that key.
4. Send a message.
-The deploy form asks for two values you provide. Render generates the other two.
+The deploy form asks for three values you provide. Render generates the other two.
| Variable | Set by | If it is missing |
|---|---|---|
| `CPK_INTELLIGENCE_API_KEY` | You. Run `npx copilotkit@latest login`, then `npx copilotkit@latest project select`. Keep it on the server. | Chat cannot open a thread. |
+| `CPK_TELEMETRY_ID` | You. Keep the identity generated by the same project selection. | Runtime usage cannot be attributed to the Intelligence signup account. |
| `OPENAI_API_KEY` | You. Used by the default `openai/gpt-5`. Change `MODEL` and supply the matching provider key for Anthropic or Google. | The model call fails. |
| `OPENMUSE_ACCESS_KEY` | Render | You cannot sign in. |
| `TOKEN_ENCRYPTION_KEY` | Render | The API refuses to start in live mode. |
@@ -191,6 +193,10 @@ Every deployment requires `CPK_INTELLIGENCE_API_KEY` on the API server for Copil
Intelligence is a separate service and is not included in this repository's MIT license. No project key is shipped. [Configuration and validation boundaries](docs/RICH-THREADS.md).
+## Signup and runtime tracking
+
+The CopilotKit SDK sends runtime telemetry through the shared CopilotKit sink to PostHog with `accessibility_title: "OpenMuse"`. Telemetry defaults to full capture; explicit SDK opt-outs and sampling settings take precedence. Keep the CLI-generated `CPK_TELEMETRY_ID` alongside the server-only project key when deploying. Intelligence records completed account signups separately; OpenMuse runtime usage can be joined to those accounts after the sink resolves the identity. [Setup, opt-out and PostHog validation](docs/TELEMETRY.md).
+
## Architecture
```mermaid
diff --git a/apps/server/src/agent.ts b/apps/server/src/agent.ts
index de593bf..4a5a758 100644
--- a/apps/server/src/agent.ts
+++ b/apps/server/src/agent.ts
@@ -58,6 +58,8 @@ export function makeRuntime(
const runtime = new CopilotRuntime({
agents,
intelligence,
+ // The shared CopilotKit sink carries this tag onto existing PostHog events.
+ telemetryProperties: { accessibility_title: "OpenMuse" },
identifyUser: async (request) => ({
id: await auth.owner(request.headers.get("authorization") ?? undefined),
name: "OpenMuse user",
diff --git a/apps/server/src/app.ts b/apps/server/src/app.ts
index 2ffca91..e625bcc 100644
--- a/apps/server/src/app.ts
+++ b/apps/server/src/app.ts
@@ -1,3 +1,4 @@
+import "./config.ts";
import { randomUUID } from "node:crypto";
import { MessageSchema } from "@ag-ui/core";
import { CopilotKitIntelligence } from "@copilotkit/runtime/v2";
diff --git a/apps/server/src/config.ts b/apps/server/src/config.ts
index 75918fa..55a70bd 100644
--- a/apps/server/src/config.ts
+++ b/apps/server/src/config.ts
@@ -23,8 +23,9 @@ if (existsSync(".env")) {
: "Unset them to use the .env values."),
);
}
-process.env.DO_NOT_TRACK ??= "1";
-process.env.COPILOTKIT_TELEMETRY_DISABLED ??= "true";
+// Capture the full setup/activation funnel while preserving explicit SDK opt-outs
+// and any deployment-specific sampling rate. Config loads before runtime imports.
+process.env.COPILOTKIT_TELEMETRY_SAMPLE_RATE ??= "1";
export interface Config {
mode: "sample" | "live";
diff --git a/docs/TELEMETRY.md b/docs/TELEMETRY.md
new file mode 100644
index 0000000..3d04fbc
--- /dev/null
+++ b/docs/TELEMETRY.md
@@ -0,0 +1,116 @@
+# Intelligence signup and OpenMuse usage tracking
+
+OpenMuse uses the CopilotKit SDK's existing telemetry transport. There is no
+separate PostHog SDK or PostHog key to configure in this application.
+
+## Configure the server
+
+Run these commands from the OpenMuse root:
+
+```sh
+npx copilotkit@latest login
+npx copilotkit@latest project select
+```
+
+Keep both generated values in the server environment:
+
+| Variable | Purpose |
+| --- | --- |
+| `CPK_INTELLIGENCE_API_KEY` | Server-only project key for Rich Threads. |
+| `CPK_TELEMETRY_ID` | CLI-issued project binding that the telemetry sink can resolve to an Intelligence account. |
+
+Deploy the values for the selected project together. Do not generate your own
+telemetry ID, reuse another project's ID, or put either value in public Expo
+variables. An API key alone does not give runtime telemetry a resolvable identity.
+If the CLI reports an identity provisioning failure, resolve that failure and
+run project selection again before claiming account attribution works.
+
+The Render Blueprint prompts for both values. For an existing Render service,
+add `CPK_TELEMETRY_ID` in the API service's Environment settings and redeploy the
+tracking commit; adding it to the Blueprint does not populate an existing service.
+
+The SDK sends existing runtime events to
+`https://telemetry.copilotkit.ai/ingest`. OpenMuse adds
+`accessibility_title: "OpenMuse"` and defaults
+`COPILOTKIT_TELEMETRY_SAMPLE_RATE` to `1` so setup and usage events are not
+randomly sampled away. It respects an explicit sampling override.
+
+Remove old deployment settings that disable telemetry if tracking is desired.
+To opt out, set either `COPILOTKIT_TELEMETRY_DISABLED=true` or `DO_NOT_TRACK=1`
+and restart the API. Both variables accept `true` or `1`. The isolated demo
+and automated test fixtures disable telemetry.
+
+## What the events measure
+
+- `intelligence_signup`: an Intelligence account created in Clerk, emitted by
+ the existing backend sync with the Clerk user ID as its PostHog `distinct_id`.
+- `oss.runtime.instance_created`: the OpenMuse runtime handler was created.
+- `oss.runtime.copilot_request_created`: a request reached the runtime; this is
+ usage evidence, not proof of a successful model answer.
+
+The sink enriches runtime events with `telemetry_id`, and with `clerk_user_id`
+when that identity resolves. Runtime and signup events can have different
+PostHog `distinct_id` values: join the runtime `clerk_user_id` property to the
+signup `distinct_id` rather than assuming a standard person funnel joins them.
+
+This connects completed signups to OpenMuse usage. It does not prove OpenMuse
+was the acquisition source: the account might predate OpenMuse. Measuring
+acquisition requires carrying a source through the Intelligence signup flow.
+OpenMuse does not emit a duplicate signup event on startup, login or key entry.
+
+## Validate in PostHog
+
+After deploying, restart the API and make one chat request. In
+[PostHog project 26816](https://eu.posthog.com/project/26816), run:
+
+```sql
+SELECT event, timestamp, distinct_id,
+ properties.accessibility_title AS app,
+ properties.telemetry_id AS telemetry_id,
+ properties.clerk_user_id AS clerk_user_id
+FROM events
+WHERE event IN (
+ 'oss.runtime.instance_created',
+ 'oss.runtime.copilot_request_created'
+ )
+ AND properties.accessibility_title = 'OpenMuse'
+ AND timestamp >= now() - INTERVAL 1 DAY
+ORDER BY timestamp DESC
+LIMIT 20
+```
+
+Confirm the tag, the expected telemetry identity and the resolved Clerk subject.
+Then find that subject's `intelligence_signup` event. To count accounts signed
+up in the last 30 days that also used OpenMuse in that period:
+
+```sql
+SELECT count(DISTINCT signup.distinct_id) AS signups_with_openmuse_usage
+FROM events AS signup
+INNER JOIN (
+ SELECT DISTINCT properties.clerk_user_id AS clerk_user_id
+ FROM events
+ WHERE event = 'oss.runtime.copilot_request_created'
+ AND properties.accessibility_title = 'OpenMuse'
+ AND timestamp >= now() - INTERVAL 30 DAY
+ AND notEmpty(toString(properties.clerk_user_id))
+) AS usage ON signup.distinct_id = usage.clerk_user_id
+WHERE signup.event = 'intelligence_signup'
+ AND signup.timestamp >= now() - INTERVAL 30 DAY
+```
+
+For OpenDots, use the same runtime configuration with the app tag `OpenDots`
+and its own CLI-issued project identity, then change the tag filter.
+
+An HTTP `202` from ingest alone does not prove PostHog delivery. If the event
+or identity enrichment is missing, check the deployed server environment and
+the `TelemetrySinkIngest` / `TelemetrySinkFanout` CloudWatch logs. The canonical
+reporting surface is [dashboard 664553](https://eu.posthog.com/project/26816/dashboard/664553).
+
+## Local verification
+
+`pnpm test` disables production telemetry for ordinary fixtures. The telemetry
+tests start fresh processes, intercept SDK fetch calls before importing it,
+and check the OpenMuse tag, CLI-issued identity header, full sampling, explicit
+sampling override, both opt-outs and absence of the project key in the payload.
+These checks prove the local SDK contract; production delivery and identity
+resolution require the PostHog checks above.
diff --git a/package.json b/package.json
index 00e99ce..3110886 100644
--- a/package.json
+++ b/package.json
@@ -13,7 +13,7 @@
"dev:demo": "tsx apps/server/src/demo/entry.ts",
"dev:mobile": "pnpm --dir apps/mobile start",
"dev:web": "pnpm --dir apps/mobile web",
- "test": "tsx --test tests/*.test.ts apps/mobile/test/*.test.ts",
+ "test": "tsx --import ./tests/setup.ts --test tests/*.test.ts apps/mobile/test/*.test.ts",
"test:browser": "node --experimental-strip-types --test apps/worker/tests/lifecycle.test.ts",
"typecheck": "tsc --noEmit && pnpm --dir apps/mobile typecheck",
"lint": "biome check .",
diff --git a/render.yaml b/render.yaml
index 88ea052..6299949 100644
--- a/render.yaml
+++ b/render.yaml
@@ -46,6 +46,9 @@ services:
sync: false
- key: CPK_INTELLIGENCE_API_KEY
sync: false
+ # Use the CLI-issued identity for the same Intelligence project as the key.
+ - key: CPK_TELEMETRY_ID
+ sync: false
# Delete these two entries, and the openmuse-browser service below, to
# deploy without browsing. The API boots either way.
- key: BROWSER_WORKER_URL
diff --git a/tests/setup.ts b/tests/setup.ts
new file mode 100644
index 0000000..c86be47
--- /dev/null
+++ b/tests/setup.ts
@@ -0,0 +1,4 @@
+// Unit/API fixtures must never emit production SDK telemetry.
+// Telemetry tests exercise enabled behavior in isolated child processes.
+process.env.DO_NOT_TRACK = "1";
+process.env.COPILOTKIT_TELEMETRY_DISABLED = "true";
diff --git a/tests/telemetry.test.ts b/tests/telemetry.test.ts
new file mode 100644
index 0000000..10d082d
--- /dev/null
+++ b/tests/telemetry.test.ts
@@ -0,0 +1,83 @@
+import assert from "node:assert/strict";
+import { execFile } from "node:child_process";
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { test } from "node:test";
+import { promisify } from "node:util";
+
+const run = promisify(execFile);
+const agentUrl = new URL("../apps/server/src/agent.ts", import.meta.url).href;
+const appUrl = new URL("../apps/server/src/app.ts", import.meta.url).href;
+const runtimeUrl = import.meta.resolve("@copilotkit/runtime/v2");
+const tsxUrl = import.meta.resolve("tsx");
+
+// A fresh process exercises config loading before the SDK creates its telemetry singleton.
+// Replace fetch before importing the runtime so no test can send production telemetry.
+const probe = `
+const requests = [];
+globalThis.fetch = async (url, options) => {
+ requests.push({ url: String(url), headers: options.headers, body: JSON.parse(options.body) });
+ return new Response('{"ok":true}', { status: 202 });
+};
+await import(${JSON.stringify(appUrl)});
+const { makeRuntime } = await import(${JSON.stringify(agentUrl)});
+const { CopilotKitIntelligence } = await import(${JSON.stringify(runtimeUrl)});
+makeRuntime(
+ { agentBackend: "sample" },
+ {},
+ { owner: async () => "local-user" },
+ new CopilotKitIntelligence({ apiKey: "test-project-key-never-sent" }),
+);
+await new Promise(resolve => setImmediate(resolve));
+console.log(JSON.stringify(requests));
+`;
+
+async function captureRuntime(envOverrides: Record<string, string> = {}) {
+ const directory = await mkdtemp(join(tmpdir(), "openmuse-telemetry-test-"));
+ const env = { ...process.env };
+ for (const key of [
+ "DO_NOT_TRACK",
+ "COPILOTKIT_TELEMETRY_DISABLED",
+ "COPILOTKIT_TELEMETRY_SAMPLE_RATE",
+ "COPILOTKIT_LICENSE_TOKEN",
+ "COPILOTKIT_TELEMETRY_URL",
+ ])
+ delete env[key];
+ try {
+ const { stdout } = await run(
+ process.execPath,
+ ["--import", tsxUrl, "--input-type=module", "--eval", probe],
+ {
+ cwd: directory,
+ env: { ...env, CPK_TELEMETRY_ID: "test-project-identity", ...envOverrides },
+ timeout: 15000,
+ },
+ );
+ return JSON.parse(stdout.trim().split("\n").at(-1) ?? "[]");
+ } finally {
+ await rm(directory, { recursive: true, force: true });
+ }
+}
+
+test("runtime telemetry carries OpenMuse attribution and the CLI-issued project identity", async () => {
+ const events = await captureRuntime();
+ assert.equal(events.length, 1);
+ const event = events[0];
+ assert.equal(event.url, "https://telemetry.copilotkit.ai/ingest");
+ assert.equal(event.headers["X-CopilotKit-Telemetry-Id"], "test-project-identity");
+ assert.equal(event.body.event, "oss.runtime.instance_created");
+ assert.equal(event.body.global_properties.accessibility_title, "OpenMuse");
+ assert.equal(event.body.global_properties.sampleRate, 1);
+ assert.equal(JSON.stringify(event).includes("test-project-key-never-sent"), false);
+});
+
+test("each SDK opt-out prevents telemetry with a configured project identity", async () => {
+ for (const key of ["DO_NOT_TRACK", "COPILOTKIT_TELEMETRY_DISABLED"])
+ for (const value of ["true", "1"])
+ assert.deepEqual(await captureRuntime({ [key]: value }), [], `${key}=${value}`);
+});
+
+test("an explicit zero sample rate prevents telemetry", async () => {
+ assert.deepEqual(await captureRuntime({ COPILOTKIT_TELEMETRY_SAMPLE_RATE: "0" }), []);
+});