Skip to content

Open source file-system and knowledge based agent template. Build AI agents that stay up to date with your knowledge base

feat: sdk usage tracking

Hugo Richard7cea2a240a51
diff --git a/apps/bot/README.md b/apps/bot/README.md
new file mode 100644
index 0000000..f26274b
--- /dev/null
+++ b/apps/bot/README.md
@@ -0,0 +1,138 @@
+# @savoir/bot
+
+Multi-platform documentation bot powered by [chat-sdk](https://github.com/vercel-labs/chat).
+
+## Setup
+
+```bash
+cp .env.example .env
+# Fill in the env vars
+bun run dev:bot
+```
+
+Expose port 3001 with a tunnel (ngrok, cloudflared) and configure the webhook URL as `https://<tunnel>/api/webhooks/<platform>`.
+
+## Architecture
+
+```
+server/
+ api/webhooks/[platform].post.ts # Dynamic webhook router
+ utils/
+ bot.ts # Chat instance + mention handler
+ ai.ts # AI pipeline (platform-agnostic)
+ types.ts # ThreadContext, ContextProvider
+ router-schema.ts # Complexity classification schema
+ adapters/
+ github.ts # GitHub adapter (issues + PRs)
+```
+
+The mention handler in `bot.ts` is platform-agnostic. It uses `thread.adapter` to interact with whatever platform the message came from, and the `ContextProvider` interface for optional context enrichment.
+
+## chat-sdk resources
+
+- [GitHub repo](https://github.com/vercel-labs/chat) — source, examples, and README
+- [`chat` on npm](https://www.npmjs.com/package/chat) — core SDK (`Chat`, `Adapter`, `Message`, `Thread`)
+- [`@chat-adapter/state-memory`](https://www.npmjs.com/package/@chat-adapter/state-memory) — in-memory state (dev/single-instance)
+- [`@chat-adapter/state-redis`](https://www.npmjs.com/package/@chat-adapter/state-redis) — Redis state (production)
+- [`@chat-adapter/github`](https://www.npmjs.com/package/@chat-adapter/github) — official GitHub adapter (PR-only, we use a custom one for issues)
+- [`@chat-adapter/slack`](https://www.npmjs.com/package/@chat-adapter/slack) — official Slack adapter
+- [`@chat-adapter/discord`](https://www.npmjs.com/package/@chat-adapter/discord) — official Discord adapter
+
+## Adding a new adapter
+
+### 1. Create the adapter file
+
+```
+server/utils/adapters/slack.ts
+```
+
+Implement the `Adapter` interface from `chat` and optionally the `ContextProvider` interface from `../types`:
+
+```typescript
+import type { Adapter, ChatInstance, /* ... */ } from 'chat'
+import type { ThreadContext, ContextProvider } from '../types'
+
+export class SlackAdapter implements Adapter<SlackThreadId, SlackRawMessage>, ContextProvider {
+ readonly name = 'slack'
+ readonly userName: string
+
+ // ... implement all Adapter methods:
+ // initialize, handleWebhook, postMessage, editMessage, deleteMessage,
+ // addReaction, removeReaction, startTyping, fetchMessages, fetchThread,
+ // encodeThreadId, decodeThreadId, parseMessage, renderFormatted
+
+ // Optional: implement ContextProvider for AI context enrichment
+ async fetchThreadContext(threadId: string): Promise<ThreadContext> {
+ return {
+ platform: 'slack',
+ title: '...',
+ body: '...',
+ labels: [],
+ state: 'open',
+ source: 'workspace/channel',
+ }
+ }
+}
+```
+
+If `fetchThreadContext` is implemented, the AI pipeline automatically uses it to enrich the prompt. If not, the bot still works — it just won't have extra context.
+
+### 2. Register the adapter in `bot.ts`
+
+```typescript
+import { SlackAdapter } from './adapters/slack'
+
+const slack = new SlackAdapter({
+ botToken: config.slack.botToken,
+ signingSecret: config.slack.signingSecret,
+ userName: botUserName,
+})
+
+const bot = new Chat({
+ userName: botUserName,
+ adapters: { github, slack },
+ state: createMemoryState(),
+ logger: 'info',
+})
+```
+
+### 3. Add config to `nuxt.config.ts`
+
+```typescript
+runtimeConfig: {
+ github: { /* ... */ },
+ slack: {
+ botToken: '',
+ signingSecret: '',
+ },
+},
+```
+
+### 4. Done
+
+The webhook endpoint `/api/webhooks/slack` is automatically available. The `onNewMention` handler works for all adapters — no changes needed.
+
+## Stats tracking
+
+The bot automatically reports usage stats (tokens, duration, model) to the main Savoir app after each AI response via `savoir.reportUsage()` from `@savoir/sdk`. Stats appear in the admin dashboard under the source name `<platform>-bot` (e.g. `github-bot`, `slack-bot`).
+
+The `source` and `sourceId` are passed to `createSavoir()` at initialization, so usage tracking is transparent — no manual API calls needed. If your adapter implements `ContextProvider`, stats tracking works automatically with the correct platform name and issue number.
+
+## ContextProvider
+
+The `ContextProvider` interface is optional. It lets an adapter provide platform-specific context to the AI pipeline:
+
+```typescript
+interface ThreadContext {
+ platform: string // "github", "slack", "discord"
+ title: string // issue title, channel topic, thread subject
+ body: string // issue body, first message
+ labels: string[] // labels, tags
+ state: string // "open", "closed", "active"
+ source: string // "owner/repo", "workspace/channel"
+ number?: number // issue/ticket number
+ previousComments?: Array<{ author: string, body: string, isBot: boolean }>
+}
+```
+
+The `hasContextProvider()` type guard in `types.ts` checks at runtime if an adapter implements it.
diff --git a/apps/bot/nuxt.config.ts b/apps/bot/nuxt.config.ts
index fa98082..867a4c6 100644
--- a/apps/bot/nuxt.config.ts
+++ b/apps/bot/nuxt.config.ts
@@ -33,6 +33,7 @@ export default defineNuxtConfig({
webhookSecret: '',
appId: '',
appPrivateKey: '',
+ replyToNewIssues: false,
},
savoir: {
apiUrl: 'http://localhost:3000',
diff --git a/apps/bot/server/utils/adapters/github.ts b/apps/bot/server/utils/adapters/github.ts
index 644944f..482fd9a 100644
--- a/apps/bot/server/utils/adapters/github.ts
+++ b/apps/bot/server/utils/adapters/github.ts
@@ -15,6 +15,8 @@ export interface GitHubAdapterConfig {
privateKey: string
webhookSecret: string
userName: string
+ /** If true, the bot replies to all new issues even without a mention. Default: false. */
+ replyToNewIssues?: boolean
}
interface GitHubComment {
@@ -72,6 +74,7 @@ export class SavoirGitHubAdapter implements Adapter<GitHubThreadId, GitHubRawMes
private webhookSecret: string
private appId: string
private privateKey: string
+ private replyToNewIssues: boolean
private chat: ChatInstance | null = null
private octokitCache = new Map<string, { octokit: Octokit, expiresAt: number }>()
@@ -80,6 +83,7 @@ export class SavoirGitHubAdapter implements Adapter<GitHubThreadId, GitHubRawMes
this.webhookSecret = config.webhookSecret
this.appId = config.appId
this.privateKey = config.privateKey
+ this.replyToNewIssues = config.replyToNewIssues ?? false
}
async initialize(chat: ChatInstance): Promise<void> {
@@ -179,13 +183,19 @@ export class SavoirGitHubAdapter implements Adapter<GitHubThreadId, GitHubRawMes
return new Response(JSON.stringify({ ok: true }), { status: 200 })
}
+ const issueText = `${payload.issue.title}\n\n${payload.issue.body || ''}`
+ const hasMention = issueText.includes(`@${this.userName}`)
+
+ if (!this.replyToNewIssues && !hasMention) {
+ return new Response(JSON.stringify({ ok: true }), { status: 200 })
+ }
+
const threadId = this.encodeThreadId({
owner: payload.repository.owner.login,
repo: payload.repository.name,
issueNumber: payload.issue.number,
})
- const issueText = `${payload.issue.title}\n\n${payload.issue.body || ''}`
const message = this.parseMessage({
type: 'issue_comment',
id: payload.issue.id,
@@ -194,7 +204,6 @@ export class SavoirGitHubAdapter implements Adapter<GitHubThreadId, GitHubRawMes
created_at: payload.issue.created_at,
})
- // Mark as mention since new issues should always be processed
message.isMention = true
this.chat!.processMessage(this, threadId, message, options)
diff --git a/apps/bot/server/utils/ai.ts b/apps/bot/server/utils/ai.ts
index f1cabf9..ef47395 100644
--- a/apps/bot/server/utils/ai.ts
+++ b/apps/bot/server/utils/ai.ts
@@ -96,15 +96,13 @@ Once configured, I'll be able to search the documentation and help answer your q
}
const startTime = Date.now()
- let stepCount = 0
- let totalInputTokens = 0
- let totalOutputTokens = 0
- let toolCallCount = 0
try {
const savoir = createSavoir({
apiUrl: config.savoir.apiUrl,
apiKey: config.savoir.apiKey || undefined,
+ source: context?.platform ? `${context.platform}-bot` : 'bot',
+ sourceId: context?.number ? `issue-${context.number}` : undefined,
onToolCall: (info) => {
if (info.state === 'loading') {
log.info('bot', `bash: ${JSON.stringify(info.args).slice(0, 150)}`)
@@ -132,12 +130,6 @@ Once configured, I'll be able to search the documentation and help answer your q
instructions: buildSystemPrompt(context, routerConfig, savoirConfig),
tools: savoir.tools,
stopWhen: stepCountIs(effectiveMaxSteps),
- onStepFinish: ({ usage, toolCalls }) => {
- stepCount++
- totalInputTokens += usage.inputTokens || 0
- totalOutputTokens += usage.outputTokens || 0
- toolCallCount += toolCalls?.length || 0
- },
})
const result = await agent.generate({
@@ -145,8 +137,14 @@ Once configured, I'll be able to search the documentation and help answer your q
})
const durationMs = Date.now() - startTime
- log.info('bot', `Response generated (${durationMs}ms, ${stepCount} steps, ${toolCallCount} commands)`)
- log.info('bot', `Tokens: ${totalInputTokens} in / ${totalOutputTokens} out`)
+ const { totalUsage } = result
+ log.info('bot', `Response generated (${durationMs}ms, ${result.steps.length} steps)`)
+ log.info('bot', `Tokens: ${totalUsage.inputTokens ?? 0} in / ${totalUsage.outputTokens ?? 0} out`)
+
+ savoir.reportUsage(result, {
+ startTime,
+ metadata: context ? { source: context.source } : undefined,
+ }).catch(() => {})
if (!result.text) {
return `I searched the documentation but couldn't generate a helpful response for:
diff --git a/apps/bot/server/utils/bot.ts b/apps/bot/server/utils/bot.ts
index 84965d3..cf3fa09 100644
--- a/apps/bot/server/utils/bot.ts
+++ b/apps/bot/server/utils/bot.ts
@@ -17,6 +17,7 @@ function createBot(): Chat {
privateKey: config.github.appPrivateKey,
webhookSecret: config.github.webhookSecret,
userName: botUserName,
+ replyToNewIssues: config.github.replyToNewIssues as boolean,
})
const bot = new Chat({
diff --git a/packages/sdk/package.json b/packages/sdk/package.json
index e0e1ac8..fe29ebb 100644
--- a/packages/sdk/package.json
+++ b/packages/sdk/package.json
@@ -1,6 +1,6 @@
{
"name": "@savoir/sdk",
- "version": "0.1.0-alpha.1",
+ "version": "0.1.0-alpha.2",
"description": "SDK for Savoir documentation AI with AI SDK compatible tools",
"license": "MIT",
"author": "Hugo Richard",
diff --git a/packages/sdk/src/client.ts b/packages/sdk/src/client.ts
index ca314e0..1abceaa 100644
--- a/packages/sdk/src/client.ts
+++ b/packages/sdk/src/client.ts
@@ -1,5 +1,7 @@
import type {
AgentConfig,
+ GenerateResult,
+ ReportUsageOptions,
SavoirConfig,
ShellBatchResponse,
ShellResponse,
@@ -16,6 +18,8 @@ export class SavoirClient {
private readonly apiUrl: string
private readonly apiKey?: string
+ private readonly source?: string
+ private readonly sourceId?: string
private sessionId?: string
constructor(config: SavoirConfig) {
@@ -29,6 +33,8 @@ export class SavoirClient {
this.apiUrl = config.apiUrl.replace(/\/$/, '')
this.apiKey = config.apiKey
this.sessionId = config.sessionId
+ this.source = config.source
+ this.sourceId = config.sourceId
}
getSessionId(): string | undefined {
@@ -166,4 +172,18 @@ export class SavoirClient {
return await this.get<AgentConfig>('/api/agent-config/public')
}
+ async reportUsage(result: GenerateResult, options?: ReportUsageOptions): Promise<void> {
+ const durationMs = options?.durationMs ?? (options?.startTime ? Date.now() - options.startTime : undefined)
+
+ await this.post('/api/stats/usage', {
+ source: this.source || 'sdk',
+ sourceId: options?.sourceId ?? this.sourceId,
+ model: result.response.modelId,
+ inputTokens: result.totalUsage.inputTokens,
+ outputTokens: result.totalUsage.outputTokens,
+ durationMs,
+ metadata: options?.metadata,
+ })
+ }
+
}
diff --git a/packages/sdk/src/index.ts b/packages/sdk/src/index.ts
index f2c7832..e8d5979 100644
--- a/packages/sdk/src/index.ts
+++ b/packages/sdk/src/index.ts
@@ -1,8 +1,8 @@
import { SavoirClient } from './client'
import { createBashBatchTool, createBashTool } from './tools'
-import type { AgentConfig, SavoirConfig } from './types'
+import type { AgentConfig, GenerateResult, ReportUsageOptions, SavoirConfig } from './types'
-export type { SavoirConfig, ShellResponse, ShellBatchResponse, ShellCommandResult, SyncOptions, SyncResponse, SnapshotResponse, GitHubSource, YouTubeSource, SourcesResponse, SyncSourceResponse, ToolCallInfo, ToolCallCallback, ToolCallState, ToolExecutionResult, CommandResult, AgentConfig } from './types'
+export type { SavoirConfig, ShellResponse, ShellBatchResponse, ShellCommandResult, SyncOptions, SyncResponse, SnapshotResponse, GitHubSource, YouTubeSource, SourcesResponse, SyncSourceResponse, ToolCallInfo, ToolCallCallback, ToolCallState, ToolExecutionResult, CommandResult, AgentConfig, GenerateResult, ReportUsageOptions } from './types'
export { SavoirError, NetworkError } from './errors'
export { SavoirClient } from './client'
@@ -19,6 +19,8 @@ export interface Savoir {
setSessionId: (sessionId: string) => void
/** Fetch admin-defined agent customization settings */
getAgentConfig: () => Promise<AgentConfig>
+ /** Report usage from an AI SDK generate result. Extracts totalUsage and modelId automatically. */
+ reportUsage: (result: GenerateResult, options?: ReportUsageOptions) => Promise<void>
}
/**
@@ -58,5 +60,6 @@ export function createSavoir(config: SavoirConfig): Savoir {
getSessionId: () => client.getSessionId(),
setSessionId: (sessionId: string) => client.setSessionId(sessionId),
getAgentConfig: () => client.getAgentConfig(),
+ reportUsage: (result, options) => client.reportUsage(result, options),
}
}
diff --git a/packages/sdk/src/types.ts b/packages/sdk/src/types.ts
index cf2433b..e50cac8 100644
--- a/packages/sdk/src/types.ts
+++ b/packages/sdk/src/types.ts
@@ -5,6 +5,10 @@ export interface SavoirConfig {
apiKey?: string
/** Reuse an existing sandbox session instead of creating a new one. */
sessionId?: string
+ /** Usage source identifier (e.g. 'github-bot'). Defaults to 'sdk'. */
+ source?: string
+ /** Optional tracking ID (e.g. 'issue-123'). Can be overridden per reportUsage() call. */
+ sourceId?: string
/** Callback fired on tool execution (loading/done/error states). */
onToolCall?: ToolCallCallback
}
@@ -165,6 +169,23 @@ export interface SyncSourceResponse {
}
}
+/** Duck-typed AI SDK generate result — works with GenerateTextResult/StreamTextResult without importing ai. */
+export interface GenerateResult {
+ totalUsage: { inputTokens?: number, outputTokens?: number }
+ response: { modelId?: string }
+}
+
+export interface ReportUsageOptions {
+ /** Override the sourceId set in config. */
+ sourceId?: string
+ /** Start time (Date.now()) to compute durationMs automatically. */
+ startTime?: number
+ /** Explicit duration in ms (takes precedence over startTime). */
+ durationMs?: number
+ /** Extra metadata to attach to the usage record. */
+ metadata?: Record<string, unknown>
+}
+
/** Admin-defined settings that customize agent behavior (fetched from /api/agent-config/public). */
export interface AgentConfig {
id: string