Open source file-system and knowledge based agent template. Build AI agents that stay up to date with your knowledge base
improve documentation
Hugo Richard5650717516e9
diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.ymlindex 226ec6b..a0eb66f 100644--- a/.github/ISSUE_TEMPLATE/bug-report.yml+++ b/.github/ISSUE_TEMPLATE/bug-report.yml@@ -5,7 +5,7 @@ body:- type: markdownattributes:value: |- Before reporting a bug, please make sure that you have read through our documentation and existing [issues](https://github.com/HugoRCD/savoir/issues) to see if the bug has already been reported or fixed. If it has, please add your reaction to the existing issue instead of creating a new one.+ Before reporting a bug, please make sure that you have read through our documentation and existing [issues](https://github.com/vercel-labs/savoir/issues) to see if the bug has already been reported or fixed. If it has, please add your reaction to the existing issue instead of creating a new one.- type: textareaid: envattributes:diff --git a/.github/ISSUE_TEMPLATE/feature-request.yml b/.github/ISSUE_TEMPLATE/feature-request.ymlindex ceb3c2f..4649f83 100644--- a/.github/ISSUE_TEMPLATE/feature-request.yml+++ b/.github/ISSUE_TEMPLATE/feature-request.yml@@ -5,7 +5,7 @@ body:- type: markdownattributes:value: |- Before requesting a feature, please make sure that you have read through our documentation and existing [issues](https://github.com/HugoRCD/savoir/issues) to see if the feature has already been requested or implemented. If it has, please add your reaction to the existing issue instead of creating a new one.+ Before requesting a feature, please make sure that you have read through our documentation and existing [issues](https://github.com/vercel-labs/savoir/issues) to see if the feature has already been requested or implemented. If it has, please add your reaction to the existing issue instead of creating a new one.- type: textareaid: descriptionattributes:diff --git a/.github/ISSUE_TEMPLATE/question.yml b/.github/ISSUE_TEMPLATE/question.ymlindex d03054a..bdd49c6 100644--- a/.github/ISSUE_TEMPLATE/question.yml+++ b/.github/ISSUE_TEMPLATE/question.yml@@ -5,7 +5,7 @@ body:- type: markdownattributes:value: |- Before requesting a question, please make sure that you have read through our documentation and existing [issues](https://github.com/HugoRCD/savoir/issues) to see if the feature has already been requested or implemented. If it has, please add your reaction to the existing issue instead of creating a new one.+ Before requesting a question, please make sure that you have read through our documentation and existing [issues](https://github.com/vercel-labs/savoir/issues) to see if the feature has already been requested or implemented. If it has, please add your reaction to the existing issue instead of creating a new one.- type: textareaid: descriptionattributes:diff --git a/.github/funding.yml b/.github/funding.ymlindex 7698730..2ce4c1e 100644--- a/.github/funding.yml+++ b/.github/funding.yml@@ -1 +1 @@-github: [HugoRCD]+github: [vercel-labs]diff --git a/.github/pull_request_template.md b/.github/pull_request_template.mdindex 052381a..586ee82 100644--- a/.github/pull_request_template.md+++ b/.github/pull_request_template.md@@ -19,9 +19,7 @@ Here are the available types and scopes:### Scopes- chat (apps/chat - Main application)-- bot (apps/bot - Bot)- sdk (packages/sdk - @savoir/sdk)-- logger (packages/logger - @savoir/logger)- deps (dependencies updates)- docs (documentation)-->diff --git a/AGENTS.md b/AGENTS.mdindex 175968c..9d98538 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -17,13 +17,11 @@ AI agents with real-time knowledge base access.```savoir/-├── apps/chat/ # Nuxt app (chat UI + API)+├── apps/chat/ # Nuxt app (chat UI + API + bots)│ ├── app/ # Vue components, pages-│ └── server/ # API, workflows, sandbox-├── packages/-│ ├── sdk/ # @savoir/sdk - AI SDK tools-│ └── logger/ # @savoir/logger-└── savoir.config.ts # Source definitions+│ └── server/ # API, workflows, sandbox, bot adapters+└── packages/+ └── sdk/ # @savoir/sdk - AI SDK tools (bash, bash_batch)```## Documentation@@ -31,5 +29,4 @@ savoir/- [Architecture](docs/ARCHITECTURE.md) - System design, API specs, components- [Sources](docs/SOURCES.md) - Content sources configuration- [Coding Guidelines](docs/CODING-GUIDELINES.md) - Code style, patterns-- [Logging](docs/LOGGING.md) - @savoir/logger usage- [Environment](docs/ENVIRONMENT.md) - Environment variablesdiff --git a/README.md b/README.mdindex 931eb62..2438c5d 100644--- a/README.md+++ b/README.md@@ -2,7 +2,7 @@Build AI agents with real-time knowledge access.-Savoir provides the infrastructure to create file-based AI agents (chatbots, Discord bots, GitHub bots, etc.) that can search and read from frequently updated knowledge bases. It combines a unified Nuxt application for the chat interface and API with an SDK that provides AI SDK-compatible tools.+Savoir provides the infrastructure to create file-based AI agents (chatbots, Discord bots, GitHub bots, etc.) that can search and read from frequently updated knowledge bases. It combines a unified [Nuxt](https://nuxt.com) application for the chat interface and API with an SDK that provides [AI SDK](https://ai-sdk.dev)-compatible tools.## Architecture@@ -15,7 +15,7 @@ Savoir provides the infrastructure to create file-based AI agents (chatbots, Dis▼┌─────────────────────────────────────────────────────────────────┐│ @savoir/sdk │-│ AI SDK compatible tools (searchAndRead, etc.) │+│ AI SDK compatible tools (bash, bash_batch) │└─────────────────────────────────┬───────────────────────────────┘│ API calls▼@@ -40,9 +40,8 @@ Savoir provides the infrastructure to create file-based AI agents (chatbots, Dis| Package | Description ||---------|-------------|-| [`@savoir/sdk`](./packages/sdk) | AI SDK compatible tools for agents |-| [`@savoir/logger`](./packages/logger) | Logging utilities |-| [`apps/chat`](./apps/chat) | Unified Nuxt app (chat UI + API) |+| [`@savoir/sdk`](./packages/sdk) | [AI SDK](https://ai-sdk.dev) compatible tools for agents |+| [`apps/chat`](./apps/chat) | Unified [Nuxt](https://nuxt.com) app (chat UI + API + bots) |## Quick Start@@ -52,19 +51,16 @@ Savoir provides the infrastructure to create file-based AI agents (chatbots, Disimport { generateText } from 'ai'import { createSavoir } from '@savoir/sdk'-// Initialize Savoir clientconst savoir = createSavoir({+ apiUrl: process.env.SAVOIR_API_URL!,apiKey: process.env.SAVOIR_API_KEY,- apiUrl: process.env.SAVOIR_API_URL || 'https://savoir.example.com'})-// Use with AI SDKconst { text } = await generateText({- model: 'google/gemini-3-flash',- prompt: 'Tell me the latest developments in Nuxt',- tools: {- ...savoir.tools,- }+ model: yourModel, // any AI SDK compatible model+ tools: savoir.tools, // bash and bash_batch tools+ maxSteps: 10,+ prompt: 'How do I configure authentication in Nuxt?',})console.log(text)@@ -74,7 +70,7 @@ console.log(text)```bash# Clone the repository-git clone https://github.com/HugoRCD/savoir.git+git clone https://github.com/vercel-labs/savoir.gitcd savoir# Install dependencies@@ -84,9 +80,6 @@ bun installcp apps/chat/.env.example apps/chat/.env# Edit .env with your configuration-# Run database migrations-turbo run db:migrate --filter=@savoir/chat-# Start the appbun run dev```@@ -94,46 +87,51 @@ bun run dev**Required environment variables:**```bash-# GitHub token for cloning snapshot repository-NUXT_GITHUB_TOKEN=ghp_...+# Authentication+BETTER_AUTH_SECRET=your-secret # Secret for signing sessions/tokens+GITHUB_CLIENT_ID=... # GitHub OAuth app client ID+GITHUB_CLIENT_SECRET=... # GitHub OAuth app client secret++# AI+AI_GATEWAY_API_KEY=... # Vercel AI Gateway API key++# Admin+NUXT_ADMIN_USERS=user1,user2 # Comma-separated admin emails/usernames-# Repository containing the content snapshot-NUXT_GITHUB_SNAPSHOT_REPO=your-org/your-content-repo+# Sandbox+NUXT_GITHUB_TOKEN=ghp_... # GitHub token for repo access+NUXT_GITHUB_SNAPSHOT_REPO=org/repo # Snapshot repository (owner/repo)-# Optional: API key for securing endpoints-NUXT_SAVOIR_SECRET_KEY=your-secret-key+# Site+NUXT_PUBLIC_SITE_URL=https://... # Public URL of your instance```+See [ENVIRONMENT.md](./docs/ENVIRONMENT.md) for the full list of environment variables.+## Configuration-Sources are stored in the database and can be managed via the API (`/api/sources`). See `savoir.config.ts` at the project root for example source definitions:+Sources are managed through the **admin interface** at `/admin`. You can add GitHub repositories and YouTube channels as knowledge sources, then trigger a sync from the UI.-The config file format:--```typescript-// savoir.config.ts-export default {- sources: {- github: [- { id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' },- { id: 'nitro', repo: 'nitrojs/nitro', branch: 'main' },- ],- youtube: [- { id: 'alex-lichter', channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA' },- ],- },-}-```+Sources can also be listed programmatically via the SDK (`savoir.client.getSources()`).See [SOURCES.md](./docs/SOURCES.md) for detailed source configuration options.## How It Works-1. **Sources in Database**: Sources are stored in SQLite via NuxtHub, seeded from `savoir.config.ts`-2. **Content Aggregation**: Sources (GitHub docs, YouTube transcripts, etc.) are synced to a snapshot repository via Vercel Workflow-3. **Sandbox Creation**: When an agent needs to search, the API creates/recovers a Vercel Sandbox with the snapshot repo cloned-4. **File-based Search**: The SDK tools execute grep/find commands in the sandbox to search and read content-5. **AI Integration**: Tools are compatible with the Vercel AI SDK for seamless integration with any LLM+1. **Sources in Database**: Sources are stored in SQLite via [NuxtHub](https://hub.nuxt.com), managed through the admin interface+2. **Content Aggregation**: Sources (GitHub docs, YouTube transcripts, etc.) are synced to a snapshot repository via [Vercel Workflow](https://useworkflow.dev)+3. **Sandbox Creation**: When an agent needs to search, the API creates/recovers a [Vercel Sandbox](https://vercel.com/docs/vercel-sandbox) with the snapshot repo cloned+4. **File-based Search**: The SDK `bash` and `bash_batch` tools execute grep/find/cat commands in the sandbox to search and read content+5. **AI Integration**: Tools are compatible with the [Vercel AI SDK](https://ai-sdk.dev) for seamless integration with any LLM++## Bots++Savoir includes built-in bot integrations powered by the [Vercel Chat SDK](https://github.com/vercel-labs/chat):++- **GitHub Bot**: Responds to mentions in GitHub issues and PRs. Uses a [GitHub App](https://docs.github.com/en/apps) for authentication and webhooks.+- **Discord Bot**: Responds to mentions and continues conversations in threads. Uses the [Discord API](https://discord.com/developers/docs).++Both bots use the same AI agent and knowledge base as the chat interface.## Development@@ -154,12 +152,16 @@ bun run testbun run lint:fix```-## Related Projects+## Built With-- [Vercel AI SDK](https://ai-sdk.dev) - The AI SDK that Savoir integrates with-- [Vercel Sandbox](https://vercel.com/docs/vercel-sandbox) - Sandboxed execution environment-- [NuxtHub](https://hub.nuxt.com) - The deployment platform for Nuxt-- [Vercel Workflow](https://github.com/vercel/workflow) - Durable workflow execution+- [Nuxt](https://nuxt.com) - Full-stack Vue framework+- [NuxtHub](https://hub.nuxt.com) - Database, KV, and blob storage+- [Vercel AI SDK](https://ai-sdk.dev) - AI model integration and tool system+- [Vercel Sandbox](https://vercel.com/docs/vercel-sandbox) - Isolated execution environment+- [Vercel Workflow](https://useworkflow.dev) - Durable workflow execution+- [Better Auth](https://www.better-auth.com) - Authentication framework+- [Drizzle ORM](https://orm.drizzle.team) - Type-safe database queries+- [Vercel Chat SDK](https://github.com/vercel-labs/chat) - Bot framework for GitHub and Discord## Licensediff --git a/apps/chat/.env.example b/apps/chat/.env.examplenew file mode 100644index 0000000..f94af47--- /dev/null+++ b/apps/chat/.env.example@@ -0,0 +1,52 @@+# =============================================================================+# Savoir - Environment Variables+# =============================================================================+# Copy this file to .env and fill in the values.+# See docs/ENVIRONMENT.md for detailed descriptions.++# --- Core -------------------------------------------------------------------++NUXT_PUBLIC_SITE_URL=http://localhost:3000[email protected],your-github-username++# --- Authentication ----------------------------------------------------------++BETTER_AUTH_SECRET= # Random secret for signing sessions (generate with: openssl rand -hex 32)+GITHUB_CLIENT_ID= # GitHub OAuth app client ID+GITHUB_CLIENT_SECRET= # GitHub OAuth app client secret++# --- AI ----------------------------------------------------------------------++AI_GATEWAY_API_KEY= # Vercel AI Gateway API key++# --- Sandbox & Sync ----------------------------------------------------------++NUXT_GITHUB_TOKEN= # GitHub personal access token (for cloning repos)+NUXT_GITHUB_SNAPSHOT_REPO= # Snapshot repository (owner/repo format)+NUXT_GITHUB_SNAPSHOT_BRANCH=main++# --- GitHub Bot (optional) ---------------------------------------------------++# NUXT_PUBLIC_GITHUB_BOT_TRIGGER=@your-bot-name+# NUXT_GITHUB_APP_ID=+# NUXT_GITHUB_APP_PRIVATE_KEY=+# NUXT_GITHUB_WEBHOOK_SECRET=++# --- Discord Bot (optional) --------------------------------------------------++# NUXT_DISCORD_BOT_TOKEN=+# NUXT_DISCORD_PUBLIC_KEY=+# NUXT_DISCORD_APPLICATION_ID=+# NUXT_DISCORD_MENTION_ROLE_IDS=++# --- YouTube (optional) ------------------------------------------------------++# NUXT_YOUTUBE_API_KEY=++# --- Storage (optional) ------------------------------------------------------++# BLOB_READ_WRITE_TOKEN= # Vercel Blob token for file uploads++# --- State (optional) --------------------------------------------------------++# REDIS_URL= # Redis URL for bot state persistence (falls back to in-memory)diff --git a/apps/chat/app/content/docs/admin-mode.md b/apps/chat/app/content/docs/admin-mode.mdnew file mode 100644index 0000000..fd80005--- /dev/null+++ b/apps/chat/app/content/docs/admin-mode.md@@ -0,0 +1,79 @@+# Admin Mode++Admin mode lets you **talk directly to your application**. Instead of searching documentation, the AI agent queries your database, browses your logs, analyzes errors, and generates charts -- all through natural language.++## How to Access++Admin mode is available to users listed in `NUXT_ADMIN_USERS`. When you're an admin, a mode toggle appears on the homepage letting you switch between **Chat** (documentation search) and **Admin** (application monitoring).++Admin chats are marked with a shield icon in the sidebar.++## Built-in Tools++In admin mode, the AI agent has access to a set of tools designed for application monitoring and analytics:++### Usage & Analytics++| Tool | Description |+|------|-------------|+| `query_stats` | Query usage statistics: messages sent, tokens used, active models, user feedback -- filterable by date range |+| `list_users` | List users with their activity and token usage, sortable by messages, tokens, or last activity |+| `query_chats` | Browse recent chats to inspect what questions are being asked and how they're handled |++### Logs & Errors++| Tool | Description |+|------|-------------|+| `query_logs` | Browse application logs by level, HTTP path, status code, method -- with full-text search |+| `log_stats` | Aggregated metrics: total requests, error rates, status code distribution, latency percentiles (p50/p95/p99), top slow/busy/error endpoints |+| `query_errors` | Error analysis: recent errors, grouped by path or message, hourly error trends |++### Configuration & Data++| Tool | Description |+|------|-------------|+| `list_sources` | List all configured knowledge base sources |+| `get_agent_config` | View current agent configuration (model, temperature, response style, etc.) |+| `run_sql` | Execute read-only SQL queries (SELECT only) against all application tables |++### Visualization++| Tool | Description |+|------|-------------|+| `chart` | Generate line charts from time-series data, with automatic gap-filling for missing dates |++## Example Prompts++Here are some things you can ask in admin mode:++- "How many messages were sent this week?"+- "Show me the most active users"+- "Are there any errors in the last 24 hours?"+- "What's the p95 latency for the chat endpoint?"+- "Chart the daily message count over the last 30 days"+- "Which models are being used the most?"+- "Show me the top 5 slowest endpoints"+- "What questions are users asking about the most?"++## Structured Logging with evlog++Savoir uses [`@evlog/nuxthub`](https://github.com/HugoRCD/evlog) for structured logging. Every API request, bot interaction, and workflow execution is automatically logged to the `evlog_events` table in your [NuxtHub](https://hub.nuxt.com) database.++This means:++- **Logs are queryable** -- the admin tools can search, filter, and aggregate log data+- **Retention is automatic** -- logs older than 7 days are cleaned up by a daily cron job+- **No external service needed** -- everything is stored in your NuxtHub SQLite database++Each log event includes the HTTP method, path, status code, duration, request ID, and optional structured data. The `query_logs`, `log_stats`, and `query_errors` tools leverage this data to give you real-time insights into your application.++## How It Works++Admin mode uses a completely separate AI pipeline from the regular chat:++1. When you create an admin chat, the server loads the **admin tools** instead of the documentation sandbox tools+2. The AI agent receives a specialized system prompt that instructs it to use the admin tools for monitoring and analytics+3. All queries run directly against your [NuxtHub](https://hub.nuxt.com) database -- no sandbox is involved+4. The `chart` tool can render visual time-series charts inline in the conversation++Admin chats are stored like regular chats but tagged with `mode: 'admin'` in the database.diff --git a/apps/chat/app/content/docs/api-keys.md b/apps/chat/app/content/docs/api-keys.mdindex 290bb13..df0e047 100644--- a/apps/chat/app/content/docs/api-keys.md+++ b/apps/chat/app/content/docs/api-keys.md@@ -1,20 +1,20 @@# API Keys-API keys allow external services to authenticate with Savoir without a browser session. They use Better Auth's API key plugin to create a virtual session tied to the key owner.+API keys allow external services to authenticate with Savoir without a browser session. They use [Better Auth](https://www.better-auth.com)'s [API key plugin](https://www.better-auth.com/docs/plugins/api-key) to create a virtual session tied to the key owner.## Admin KeysAdmin keys have full access to all endpoints, including admin-only routes.-1. Navigate to **Admin → API Keys**+1. Navigate to **Admin > API Keys** (`/admin/api-keys`)2. Click **Create API Key**-3. Copy the key immediately — it won't be shown again+3. Copy the key immediately -- it won't be shown again## User KeysRegular users can create keys scoped to their own permissions.-1. Navigate to **Settings → API Keys**+1. Navigate to **Settings > API Keys**2. Click **Create API Key**3. Copy the key immediately@@ -24,16 +24,27 @@ Include the key in your requests using either header:```bash# Authorization header-curl -H "Authorization: Bearer sk_live_..." https://savoir.example.com/api/chat+curl -H "Authorization: Bearer sk_live_..." <your-url>/api/chat# x-api-key header-curl -H "x-api-key: sk_live_..." https://savoir.example.com/api/chat+curl -H "x-api-key: sk_live_..." <your-url>/api/chat```Both headers are equivalent. Use whichever fits your HTTP client best.+API keys are the recommended way to authenticate when using the [SDK](/admin/docs/sdk):++```typescript+import { createSavoir } from '@savoir/sdk'++const savoir = createSavoir({+ apiUrl: process.env.SAVOIR_API_URL!,+ apiKey: process.env.SAVOIR_API_KEY!, // your API key+})+```+## Rotation & Revocation- To **revoke** a key, click the delete button next to it in the API Keys page- To **rotate** a key, create a new one, update your services, then revoke the old key-- Revoked keys are immediately invalidated — in-flight requests using a revoked key will fail+- Revoked keys are immediately invalidated -- in-flight requests using a revoked key will faildiff --git a/apps/chat/app/content/docs/bot-setup.md b/apps/chat/app/content/docs/bot-setup.mdindex 3a21022..63ec456 100644--- a/apps/chat/app/content/docs/bot-setup.md+++ b/apps/chat/app/content/docs/bot-setup.md@@ -2,48 +2,36 @@The Savoir GitHub bot responds to mentions in GitHub issues, using your knowledge base to provide answers directly in the conversation.-## Create a GitHub App+> If you haven't created your GitHub App yet, follow the [GitHub App Setup](/admin/docs/getting-started#github-app-setup) in the Getting Started guide first. The same app handles both OAuth login and the bot.-1. Go to **GitHub Settings → Developer settings → GitHub Apps → New GitHub App**-2. Fill in the following:- - **App name**: e.g. `savoir-bot`- - **Homepage URL**: your Savoir instance URL- - **Webhook URL**: `https://savoir.example.com/api/webhooks/github`- - **Webhook secret**: a random string (save it for later)+## Prerequisites-### Permissions+Make sure your GitHub App is configured with:-Set the following repository permissions:+- **Webhook URL** pointing to `<your-url>/api/webhooks/github`+- **Webhook secret** set and matching your `NUXT_GITHUB_WEBHOOK_SECRET`+- **Repository permissions**: Issues (Read & Write), Metadata (Read-only)+- **Events**: Issues, Issue comments-| Permission | Access |-|------------|--------|-| Issues | Read & Write |-| Metadata | Read-only |--### Events--Subscribe to these events:--- **Issues** — to detect new issues-- **Issue comments** — to detect mentions in comments+For the full setup, see [Getting Started > GitHub App Setup](/admin/docs/getting-started#github-app-setup).## Environment Variables-Add the following to your `.env` file:-| Variable | Description ||----------|-------------|-| `NUXT_GITHUB_APP_ID` | The App ID from your GitHub App settings |-| `NUXT_GITHUB_APP_PRIVATE_KEY` | The private key (PEM format, can be base64-encoded) |+| `NUXT_PUBLIC_GITHUB_BOT_TRIGGER` | The mention trigger for the bot (e.g. `@savoir-bot`). This must match the GitHub App name. |+| `NUXT_GITHUB_APP_ID` | The App ID from your [GitHub App settings](https://github.com/settings/apps) |+| `NUXT_GITHUB_APP_PRIVATE_KEY` | The private key (PEM format). Generate one from your app's settings page. Can be base64-encoded. || `NUXT_GITHUB_WEBHOOK_SECRET` | The webhook secret you set when creating the app |-| `NUXT_SAVOIR_API_KEY` | A Savoir admin API key for the bot to authenticate with |## Install the App-1. From your GitHub App settings, click **Install App**+1. From your [GitHub App settings](https://github.com/settings/apps), click **Install App**2. Select the repositories where the bot should be active3. Confirm the installation+The app uses [installation access tokens](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/authenticating-as-a-github-app-installation) to interact with repositories -- no personal access token needed for the bot.+## Trigger the BotMention the bot in any issue comment:@@ -53,8 +41,29 @@ Mention the bot in any issue comment:```The bot will:-1. Receive the webhook event-2. Query the Savoir knowledge base+1. React with an "eyes" emoji to indicate it's processing+2. Search the Savoir knowledge base using the same AI agent as the chat3. Post a reply in the same issue thread+4. React with a "thumbs up" emoji when doneThe bot only responds when explicitly mentioned. It ignores its own comments to prevent loops.++## Auto-Reply to New Issues++You can optionally configure the bot to reply to **all** new issues automatically, without requiring a mention. Set in `nuxt.config.ts`:++```typescript+runtimeConfig: {+ github: {+ replyToNewIssues: true,+ },+}+```++When enabled, the bot will respond to every new issue opened in repositories where the GitHub App is installed.++## How It Works Under the Hood++The bot is built on the [Vercel Chat SDK](https://github.com/vercel-labs/chat) with a custom GitHub adapter. It uses the [Octokit](https://github.com/octokit/rest.js) library with [GitHub App authentication](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/about-authentication-with-a-github-app) to interact with the GitHub API.++Each response goes through the same AI agent pipeline as the chat interface, using [Vercel AI SDK](https://ai-sdk.dev) tools to search the knowledge base in a [Vercel Sandbox](https://vercel.com/docs/vercel-sandbox).diff --git a/apps/chat/app/content/docs/discord-bot.md b/apps/chat/app/content/docs/discord-bot.mdnew file mode 100644index 0000000..d8fe7ab--- /dev/null+++ b/apps/chat/app/content/docs/discord-bot.md@@ -0,0 +1,71 @@+# Discord Bot Setup++The Savoir Discord bot responds to mentions and continues conversations in threads, using your knowledge base to provide answers.++## Create a Discord Application++1. Go to the [Discord Developer Portal](https://discord.com/developers/applications)+2. Click **New Application** and give it a name+3. Note the **Application ID** and **Public Key** from the General Information page++### Create the Bot++1. Go to the **Bot** section in the left sidebar+2. Click **Reset Token** and save the bot token+3. Enable **Message Content Intent** under Privileged Gateway Intents (required for reading message content)++### Configure OAuth2++1. Go to **OAuth2 > URL Generator**+2. Select scopes: `bot`, `applications.commands`+3. Select bot permissions: `Send Messages`, `Read Message History`, `Add Reactions`+4. Use the generated URL to invite the bot to your server++See the [Discord OAuth2 documentation](https://discord.com/developers/docs/topics/oauth2) for more details on scopes and permissions.++## Set the Interactions Endpoint++1. In the [Discord Developer Portal](https://discord.com/developers/applications), go to **General Information**+2. Set **Interactions Endpoint URL** to: `<your-url>/api/webhooks/discord`+3. Discord will verify the endpoint before saving -- make sure your app is running++## Environment Variables++Add the following to your `.env` file:++| Variable | Description |+|----------|-------------|+| `NUXT_DISCORD_BOT_TOKEN` | The bot token from the Bot section |+| `NUXT_DISCORD_PUBLIC_KEY` | The public key from General Information |+| `NUXT_DISCORD_APPLICATION_ID` | The application ID from General Information |+| `NUXT_DISCORD_MENTION_ROLE_IDS` | (Optional) Comma-separated [role IDs](https://discord.com/developers/docs/resources/guild#guild-role-object) that can trigger the bot |++## How It Works++The Discord bot is built on the [Vercel Chat SDK](https://github.com/vercel-labs/chat) with the `@chat-adapter/discord` adapter.++### Mentions++Mention the bot in any channel:++```+@savoir-bot How do I use useAsyncData?+```++The bot will reply in the same channel with an answer from the knowledge base, using the same AI agent pipeline as the [chat interface](/admin/docs/getting-started#how-it-works) and [GitHub bot](/admin/docs/github-bot).++### Thread Continuation++Once the bot has replied in a thread, it will continue responding to follow-up messages in that thread -- no need to mention it again. This allows natural back-and-forth conversations.++The bot only continues threads where it has previously participated. It does not respond to messages in threads it hasn't been part of.++### Role-Based Triggering++If `NUXT_DISCORD_MENTION_ROLE_IDS` is set, the bot will also respond when those roles are mentioned. This is useful for creating a role like `@ask-docs` that triggers the bot.++## State Management++Bot conversation state is stored in:+- [**Redis**](https://upstash.com/docs/redis) if `REDIS_URL` is set (recommended for production -- state persists across restarts)+- **In-memory** as fallback (state is lost on restart)diff --git a/apps/chat/app/content/docs/getting-started.md b/apps/chat/app/content/docs/getting-started.mdindex 4bc584a..25df887 100644--- a/apps/chat/app/content/docs/getting-started.md+++ b/apps/chat/app/content/docs/getting-started.md@@ -1,45 +1,146 @@# Getting Started-## Prerequisites+## Environment Variables-- **Node.js** v20 or later-- **pnpm** v9 or later+Create a `.env` file in the `apps/chat/` directory. You can copy `.env.example` as a starting point:-## Environment Variables+```bash+cp apps/chat/.env.example apps/chat/.env+```++### Required++| Variable | Description |+|----------|-------------|+| `BETTER_AUTH_SECRET` | Secret used to sign sessions and tokens. See [Better Auth docs](https://www.better-auth.com/docs/installation#set-environment-variables). |+| `GITHUB_CLIENT_ID` | GitHub App client ID (used for OAuth login). See [GitHub App Setup](#github-app-setup) below. |+| `GITHUB_CLIENT_SECRET` | GitHub App client secret |+| `NUXT_ADMIN_USERS` | Comma-separated list of admin emails or GitHub usernames |+| `NUXT_PUBLIC_SITE_URL` | Public URL of your Savoir instance |+| `AI_GATEWAY_API_KEY` | [Vercel AI Gateway](https://ai-sdk.dev) API key |+| `NUXT_GITHUB_TOKEN` | GitHub personal access token for cloning repos and accessing the snapshot repository |+| `NUXT_GITHUB_SNAPSHOT_REPO` | Snapshot repository in `owner/repo` format |++### Optional (for bots)++| Variable | Description |+|----------|-------------|+| `NUXT_PUBLIC_GITHUB_BOT_TRIGGER` | Bot mention trigger (e.g. `@nuxt-agent`) |+| `NUXT_GITHUB_APP_ID` | GitHub App ID (same app as OAuth, see [GitHub App Setup](#github-app-setup)) |+| `NUXT_GITHUB_APP_PRIVATE_KEY` | GitHub App private key (PEM format) |+| `NUXT_GITHUB_WEBHOOK_SECRET` | Webhook secret for the GitHub App |+| `NUXT_DISCORD_BOT_TOKEN` | Discord bot token. See [Discord Bot Setup](/admin/docs/discord-bot). |+| `NUXT_DISCORD_PUBLIC_KEY` | Discord application public key |+| `NUXT_DISCORD_APPLICATION_ID` | Discord application ID |+| `NUXT_YOUTUBE_API_KEY` | [YouTube Data API](https://developers.google.com/youtube/v3) key (for syncing YouTube sources) |++## GitHub App Setup++Savoir uses a single **GitHub App** for both **user authentication** (OAuth login) and the **GitHub bot** (webhook events). This avoids creating two separate apps.++1. Go to [**GitHub Settings > Developer settings > GitHub Apps > New GitHub App**](https://github.com/settings/apps/new)+2. Fill in the basic information:+ - **App name**: e.g. `savoir-bot`+ - **Homepage URL**: your Savoir instance URL+ - **Callback URL**: `<your-url>/api/auth/callback/github` (for OAuth login)+ - **Webhook URL**: `<your-url>/api/webhooks/github` (for the bot)+ - **Webhook secret**: generate a random string and save it++### Account Permissions++| Permission | Access | Why |+|------------|--------|-----|+| Email addresses | Read-only | Required for user login -- allows Savoir to identify users by email |++### Repository Permissions++| Permission | Access | Why |+|------------|--------|-----|+| Issues | Read & Write | Bot needs to read issues and post replies |+| Metadata | Read-only | Required by GitHub for all apps |++### Events++Subscribe to these webhook events:-Create a `.env` file at the project root with the following variables:+- **Issues** -- triggers auto-reply on new issues (optional)+- **Issue comments** -- triggers the bot when mentioned in a comment-| Variable | Description | Required |-|----------|-------------|----------|-| `BETTER_AUTH_SECRET` | Secret used to sign sessions and tokens | Yes |-| `GITHUB_CLIENT_ID` | GitHub OAuth app client ID | Yes |-| `GITHUB_CLIENT_SECRET` | GitHub OAuth app client secret | Yes |-| `NUXT_ADMIN_USERS` | Comma-separated list of admin email addresses | Yes |-| `NUXT_PUBLIC_SITE_URL` | Public URL of your Savoir instance (e.g. `https://savoir.example.com`) | Yes |-| `DATABASE_URL` | Database connection string | Yes |-| `OPENROUTER_API_KEY` | OpenRouter API key for AI models | Yes |+### After Creation++From the app settings page, collect:++| Value | Maps to |+|-------|---------|+| **App ID** | `NUXT_GITHUB_APP_ID` |+| **Client ID** | `GITHUB_CLIENT_ID` |+| **Client secret** (generate one) | `GITHUB_CLIENT_SECRET` |+| **Private key** (generate one) | `NUXT_GITHUB_APP_PRIVATE_KEY` |+| **Webhook secret** | `NUXT_GITHUB_WEBHOOK_SECRET` |++Then install the app on the repositories where you want the bot to be active: **Install App** tab > select your repositories.++For more details on the bot behavior, see the [GitHub Bot documentation](/admin/docs/github-bot).## First Launch```bash# Install dependencies-pnpm install--# Run database migrations-pnpm db:migrate+bun install# Start the development server-pnpm dev+bun run dev```-The app will be available at `http://localhost:3000`. Sign in with one of the admin emails to access the admin panel.+The app will be available at `http://localhost:3000`. Sign in with GitHub using one of the admin emails or usernames defined in `NUXT_ADMIN_USERS` to access the admin panel.## Adding Sources-Sources are the knowledge base that Savoir uses to answer questions.+Sources define the knowledge base that Savoir uses to answer questions. They are managed through the **admin interface**.++1. Navigate to the admin panel at `/admin`+2. Go to the Sources section+3. Click **Add source** and configure it (GitHub repository or YouTube channel)+4. Click **Sync** to pull the content into the knowledge base++## Syncing Content++After adding or updating sources, trigger a sync from the admin interface. The sync process runs as a durable [Vercel Workflow](https://useworkflow.dev) with automatic retries:++1. Creates a [Vercel Sandbox](https://vercel.com/docs/vercel-sandbox) from the latest snapshot+2. Clones/updates all source repositories+3. Pushes changes to the snapshot repository+4. Takes a new sandbox snapshot for instant startup++## How It Works++Savoir uses a **file-based search** approach -- no embeddings or vector databases:++1. Documentation from all sources is aggregated into a single snapshot repository+2. When a user asks a question, a [Vercel Sandbox](https://vercel.com/docs/vercel-sandbox) is created from the snapshot+3. The AI agent uses `bash` and `bash_batch` tools (via the [AI SDK](https://ai-sdk.dev)) to run `grep`, `find`, `cat`, etc. in the sandbox+4. Results are synthesized into a natural language answer with citations++You can also integrate Savoir into your own applications using the [SDK](/admin/docs/sdk).++## Admin Panel++The admin panel at `/admin` provides:++- **Sources** (`/admin`): Manage knowledge base sources (GitHub repos, YouTube channels)+- **Agent Configuration** (`/admin/agent`): Customize the AI agent's behavior (response style, model, temperature, citations, etc.)+- **Sandbox** (`/admin/sandbox`): Snapshot status and sync controls+- **Stats** (`/admin/stats`): Usage statistics and analytics+- **API Keys** (`/admin/api-keys`): Manage API keys for external integrations. See [API Keys documentation](/admin/docs/api-keys).++## Tech Stack-1. Navigate to **Admin → Sources**-2. Click **Add source** and provide a name and URL-3. Click **Sync** to index the source content+Savoir is built on:-Savoir will crawl the source and chunk it into embeddings. Subsequent syncs only process changed content.+- [Nuxt](https://nuxt.com) -- full-stack Vue framework+- [NuxtHub](https://hub.nuxt.com) -- database (SQLite), KV storage, and blob storage+- [Vercel AI SDK](https://ai-sdk.dev) -- AI model integration and tool system+- [Vercel Sandbox](https://vercel.com/docs/vercel-sandbox) -- isolated execution environment+- [Vercel Workflow](https://useworkflow.dev) -- durable workflow execution for content sync+- [Better Auth](https://www.better-auth.com) -- authentication (email/password, GitHub OAuth, API keys)+- [Drizzle ORM](https://orm.drizzle.team) -- type-safe database queriesdiff --git a/apps/chat/app/content/docs/sdk.md b/apps/chat/app/content/docs/sdk.mdindex dc5ff93..f6e723b 100644--- a/apps/chat/app/content/docs/sdk.md+++ b/apps/chat/app/content/docs/sdk.md@@ -1,11 +1,13 @@# Savoir SDK-The `@savoir/sdk` package provides a TypeScript client for interacting with Savoir programmatically.+The [`@savoir/sdk`](https://github.com/vercel-labs/savoir/tree/main/packages/sdk) package provides a TypeScript client for interacting with Savoir programmatically. It exposes [AI SDK](https://ai-sdk.dev)-compatible tools that let any AI model search and read your knowledge base.## Installation```bash-pnpm add @savoir/sdk+npm install @savoir/sdk+# or+bun add @savoir/sdk```## Configuration@@ -14,90 +16,151 @@ pnpm add @savoir/sdkimport { createSavoir } from '@savoir/sdk'const savoir = createSavoir({- apiUrl: 'https://savoir.example.com',- apiKey: 'sk_live_...'+ apiUrl: process.env.SAVOIR_API_URL!,+ apiKey: process.env.SAVOIR_API_KEY!, // Better Auth API key})```+You'll need an API key to authenticate. See [API Keys](/admin/docs/api-keys) for how to create one.++### Config Options++| Option | Type | Required | Description |+|--------|------|----------|-------------|+| `apiUrl` | `string` | Yes | Base URL of your Savoir API |+| `apiKey` | `string` | No | API key for authentication. See [API Keys](/admin/docs/api-keys). |+| `sessionId` | `string` | No | Reuse an existing [sandbox](https://vercel.com/docs/vercel-sandbox) session |+| `source` | `string` | No | Usage source identifier (e.g. `'github-bot'`). Defaults to `'sdk'`. |+| `sourceId` | `string` | No | Tracking ID (e.g. `'issue-123'`) |+| `onToolCall` | `function` | No | Callback fired on tool execution (loading/done/error) |+## AI SDK Tools-The SDK exposes tools compatible with the [AI SDK](https://sdk.vercel.ai). Use them with `generateText` or `streamText` to give any AI model access to your Savoir knowledge base.+The SDK exposes tools compatible with the [Vercel AI SDK](https://ai-sdk.dev). Use them with [`generateText`](https://ai-sdk.dev/docs/ai-sdk-core/generating-text) or [`streamText`](https://ai-sdk.dev/docs/ai-sdk-core/streaming-text) to give any AI model access to your Savoir knowledge base.```typescriptimport { generateText } from 'ai'import { createSavoir } from '@savoir/sdk'const savoir = createSavoir({- apiUrl: 'https://savoir.example.com',- apiKey: 'sk_live_...'+ apiUrl: process.env.SAVOIR_API_URL!,+ apiKey: process.env.SAVOIR_API_KEY!,})const { text } = await generateText({- model: yourModel,- tools: {- bash: savoir.tools.bash,- bash_batch: savoir.tools.bash_batch,- },- prompt: 'How do I configure authentication?'+ model: yourModel, // any AI SDK compatible model+ tools: savoir.tools,+ maxSteps: 10,+ prompt: 'How do I configure authentication?',})```+The `model` can be any [AI SDK provider](https://ai-sdk.dev/providers) -- OpenAI, Anthropic, Google, Mistral, or the [Vercel AI Gateway](https://ai-sdk.dev).+### Available Tools| Tool | Description ||------|-------------|-| `bash` | Execute a single query against the knowledge base |-| `bash_batch` | Execute multiple queries in a single request |+| `bash` | Execute a single bash command in the documentation [sandbox](https://vercel.com/docs/vercel-sandbox) |+| `bash_batch` | Execute multiple commands in a single request (more efficient) |++The AI model uses these tools to run `grep`, `find`, `cat`, `head`, etc. against the aggregated documentation in the sandbox. See [How It Works](/admin/docs/getting-started#how-it-works) for the full flow.## Client Methods-You can also use the client directly for lower-level operations:+The `savoir.client` property exposes the low-level `SavoirClient` for direct API access:```typescript+// Execute bash command+const result = await savoir.client.bash('grep -rl "useAsyncData" docs/')+console.log(result.stdout)++// Execute multiple commands+const batchResult = await savoir.client.bashBatch([+ 'find docs/ -name "*.md" | head -10',+ 'cat docs/nuxt/getting-started/installation.md',+])+// List all indexed sources-const sources = await savoir.getSources()--// Get the current agent configuration-const config = await savoir.getAgentConfig()--// Report token usage from an external integration-await savoir.reportUsage({- source: 'my-bot',- model: 'openai/gpt-4o',- inputTokens: 500,- outputTokens: 200,- durationMs: 1200+const sources = await savoir.client.getSources()++// Trigger sync (runs as a Vercel Workflow)+await savoir.client.sync({ reset: false, push: true })++// Get agent configuration+const config = await savoir.client.getAgentConfig()+```++## Reporting Usage++After generating a response, report token usage for analytics (visible in the [admin stats panel](/admin/stats)):++```typescript+const startTime = Date.now()++const result = await generateText({+ model: yourModel,+ tools: savoir.tools,+ maxSteps: 10,+ prompt: 'How do I use Nuxt middleware?',+})++// reportUsage accepts any AI SDK generate result+await savoir.reportUsage(result, {+ startTime, // auto-computes durationMs+ sourceId: 'my-integration', // optional tracking ID})```+The `reportUsage` method automatically extracts `totalUsage` (input/output tokens) and `response.modelId` from the [AI SDK result](https://ai-sdk.dev/docs/ai-sdk-core/generating-text#generatetext).+## Full Example```typescriptimport { generateText } from 'ai'-import { openai } from '@ai-sdk/openai'import { createSavoir } from '@savoir/sdk'const savoir = createSavoir({- apiUrl: process.env.SAVOIR_URL!,- apiKey: process.env.SAVOIR_API_KEY!+ apiUrl: process.env.SAVOIR_API_URL!,+ apiKey: process.env.SAVOIR_API_KEY!,+ source: 'my-app',})async function ask(question: string) {- const { text, usage } = await generateText({- model: openai('gpt-4o'),+ const startTime = Date.now()++ const result = await generateText({+ model: yourModel,tools: savoir.tools,- maxSteps: 5,- prompt: question+ maxSteps: 10,+ prompt: question,})- await savoir.reportUsage({- source: 'my-app',- model: 'openai/gpt-4o',- inputTokens: usage.promptTokens,- outputTokens: usage.completionTokens,- durationMs: 0- })+ await savoir.reportUsage(result, { startTime })++ return result.text+}+```++## Error Handling- return text+```typescript+import { SavoirError, NetworkError } from '@savoir/sdk'++try {+ await savoir.client.bash('some-command')+} catch (error) {+ if (error instanceof SavoirError) {+ console.log(error.statusCode) // HTTP status code+ console.log(error.message)++ if (error.isAuthError()) { /* 401 */ }+ if (error.isRateLimitError()) { /* 429 */ }+ if (error.isServerError()) { /* 5xx */ }+ }+ if (error instanceof NetworkError) {+ console.log(error.message)+ console.log(error.cause) // Original error+ }}```diff --git a/apps/chat/app/layouts/default.vue b/apps/chat/app/layouts/default.vueindex 472c183..4d4aa42 100644--- a/apps/chat/app/layouts/default.vue+++ b/apps/chat/app/layouts/default.vue@@ -90,11 +90,17 @@ const adminNavigation = [{ label: 'API Keys', icon: 'i-lucide-key', to: '/admin/api-keys' },],[- { label: 'Help', type: 'label' as const },- { label: 'Documentation', icon: 'i-lucide-book-open', to: '/admin/docs' },+ { label: 'Documentation', type: 'label' as const },+ { label: 'Getting Started', icon: 'i-lucide-rocket', to: '/admin/docs/getting-started' },+ { label: 'API Keys', icon: 'i-lucide-key', to: '/admin/docs/api-keys' },+ { label: 'SDK', icon: 'i-lucide-code', to: '/admin/docs/sdk' },+ { label: 'GitHub Bot', icon: 'i-simple-icons-github', to: '/admin/docs/github-bot' },+ { label: 'Discord Bot', icon: 'i-simple-icons-discord', to: '/admin/docs/discord-bot' },+ { label: 'Admin Mode', icon: 'i-lucide-shield', to: '/admin/docs/admin-mode' },],]+const adminContentArea = useTemplateRef<HTMLElement>('adminContentArea')const chatScrollArea = useTemplateRef('chatScrollArea')const showTopFade = ref(false)const showBottomFade = ref(true)@@ -154,6 +160,12 @@ function shareChat(chat: { id: string, isPublic: boolean, shareToken: string | n})}+watch(() => route.fullPath, () => {+ nextTick(() => {+ adminContentArea.value?.scrollTo({ top: 0 })+ })+})+defineShortcuts({c: () => {navigateTo('/')@@ -350,7 +362,7 @@ defineShortcuts({<div class="flex-1" /><UColorModeButton /></div>- <div class="flex-1 overflow-y-auto">+ <div ref="adminContentArea" class="flex-1 overflow-y-auto"><slot /></div></template>diff --git a/apps/chat/app/pages/admin/docs.vue b/apps/chat/app/pages/admin/docs.vuedeleted file mode 100644index fcdac03..0000000--- a/apps/chat/app/pages/admin/docs.vue+++ /dev/null@@ -1,61 +0,0 @@-<script setup lang="ts">-import type { DefineComponent } from 'vue'-import PreStream from '../../components/prose/PreStream.vue'--import gettingStartedMd from '../../content/docs/getting-started.md?raw'-import apiKeysMd from '../../content/docs/api-keys.md?raw'-import sdkMd from '../../content/docs/sdk.md?raw'-import botSetupMd from '../../content/docs/bot-setup.md?raw'--const sections = [- { id: 'getting-started', label: 'Getting Started', icon: 'i-lucide-rocket', content: gettingStartedMd },- { id: 'api-keys', label: 'API Keys', icon: 'i-lucide-key', content: apiKeysMd },- { id: 'sdk', label: 'SDK', icon: 'i-lucide-code', content: sdkMd },- { id: 'bot-setup', label: 'GitHub Bot', icon: 'i-simple-icons-github', content: botSetupMd },-]--const activeSection = ref(sections[0]!.id)--const currentContent = computed(() => {- return sections.find(s => s.id === activeSection.value)?.content ?? ''-})--const components = {- pre: PreStream as unknown as DefineComponent,-}-</script>--<template>- <div class="px-6 py-8 max-w-3xl mx-auto w-full">- <header class="mb-6">- <h1 class="text-lg font-medium text-highlighted mb-1 font-pixel tracking-wide">- Documentation- </h1>- <p class="text-sm text-muted">- Setup guides, API reference, and integration docs.- </p>- </header>-- <div class="flex items-center gap-1 mb-8">- <UButton- v-for="section in sections"- :key="section.id"- :icon="section.icon"- :label="section.label"- size="xs"- :color="activeSection === section.id ? 'primary' : 'neutral'"- :variant="activeSection === section.id ? 'solid' : 'ghost'"- @click="activeSection = section.id"- />- </div>-- <div class="prose prose-sm prose-neutral dark:prose-invert max-w-none">- <MDCCached- :value="currentContent"- :cache-key="activeSection"- :components- :parser-options="{ highlight: false }"- />- </div>- </div>-</template>diff --git a/apps/chat/app/pages/admin/docs/admin-mode.vue b/apps/chat/app/pages/admin/docs/admin-mode.vuenew file mode 100644index 0000000..ee0f2b0--- /dev/null+++ b/apps/chat/app/pages/admin/docs/admin-mode.vue@@ -0,0 +1,22 @@+<script setup lang="ts">+import type { DefineComponent } from 'vue'+import PreStream from '../../../components/prose/PreStream.vue'+import content from '../../../content/docs/admin-mode.md?raw'++const components = {+ pre: PreStream as unknown as DefineComponent,+}+</script>++<template>+ <div class="px-6 py-8 max-w-3xl mx-auto w-full">+ <div class="prose prose-sm prose-neutral dark:prose-invert max-w-none">+ <MDCCached+ :value="content"+ cache-key="docs-admin-mode"+ :components+ :parser-options="{ highlight: false }"+ />+ </div>+ </div>+</template>diff --git a/apps/chat/app/pages/admin/docs/api-keys.vue b/apps/chat/app/pages/admin/docs/api-keys.vuenew file mode 100644index 0000000..caa348a--- /dev/null+++ b/apps/chat/app/pages/admin/docs/api-keys.vue@@ -0,0 +1,22 @@+<script setup lang="ts">+import type { DefineComponent } from 'vue'+import PreStream from '../../../components/prose/PreStream.vue'+import content from '../../../content/docs/api-keys.md?raw'++const components = {+ pre: PreStream as unknown as DefineComponent,+}+</script>++<template>+ <div class="px-6 py-8 max-w-3xl mx-auto w-full">+ <div class="prose prose-sm prose-neutral dark:prose-invert max-w-none">+ <MDCCached+ :value="content"+ cache-key="docs-api-keys"+ :components+ :parser-options="{ highlight: false }"+ />+ </div>+ </div>+</template>diff --git a/apps/chat/app/pages/admin/docs/discord-bot.vue b/apps/chat/app/pages/admin/docs/discord-bot.vuenew file mode 100644index 0000000..312510a--- /dev/null+++ b/apps/chat/app/pages/admin/docs/discord-bot.vue@@ -0,0 +1,22 @@+<script setup lang="ts">+import type { DefineComponent } from 'vue'+import PreStream from '../../../components/prose/PreStream.vue'+import content from '../../../content/docs/discord-bot.md?raw'++const components = {+ pre: PreStream as unknown as DefineComponent,+}+</script>++<template>+ <div class="px-6 py-8 max-w-3xl mx-auto w-full">+ <div class="prose prose-sm prose-neutral dark:prose-invert max-w-none">+ <MDCCached+ :value="content"+ cache-key="docs-discord-bot"+ :components+ :parser-options="{ highlight: false }"+ />+ </div>+ </div>+</template>diff --git a/apps/chat/app/pages/admin/docs/getting-started.vue b/apps/chat/app/pages/admin/docs/getting-started.vuenew file mode 100644index 0000000..43b326e--- /dev/null+++ b/apps/chat/app/pages/admin/docs/getting-started.vue@@ -0,0 +1,22 @@+<script setup lang="ts">+import type { DefineComponent } from 'vue'+import PreStream from '../../../components/prose/PreStream.vue'+import content from '../../../content/docs/getting-started.md?raw'++const components = {+ pre: PreStream as unknown as DefineComponent,+}+</script>++<template>+ <div class="px-6 py-8 max-w-3xl mx-auto w-full">+ <div class="prose prose-sm prose-neutral dark:prose-invert max-w-none">+ <MDCCached+ :value="content"+ cache-key="docs-getting-started"+ :components+ :parser-options="{ highlight: false }"+ />+ </div>+ </div>+</template>diff --git a/apps/chat/app/pages/admin/docs/github-bot.vue b/apps/chat/app/pages/admin/docs/github-bot.vuenew file mode 100644index 0000000..9776cd0--- /dev/null+++ b/apps/chat/app/pages/admin/docs/github-bot.vue@@ -0,0 +1,22 @@+<script setup lang="ts">+import type { DefineComponent } from 'vue'+import PreStream from '../../../components/prose/PreStream.vue'+import content from '../../../content/docs/bot-setup.md?raw'++const components = {+ pre: PreStream as unknown as DefineComponent,+}+</script>++<template>+ <div class="px-6 py-8 max-w-3xl mx-auto w-full">+ <div class="prose prose-sm prose-neutral dark:prose-invert max-w-none">+ <MDCCached+ :value="content"+ cache-key="docs-github-bot"+ :components+ :parser-options="{ highlight: false }"+ />+ </div>+ </div>+</template>diff --git a/apps/chat/app/pages/admin/docs/sdk.vue b/apps/chat/app/pages/admin/docs/sdk.vuenew file mode 100644index 0000000..780dc43--- /dev/null+++ b/apps/chat/app/pages/admin/docs/sdk.vue@@ -0,0 +1,22 @@+<script setup lang="ts">+import type { DefineComponent } from 'vue'+import PreStream from '../../../components/prose/PreStream.vue'+import content from '../../../content/docs/sdk.md?raw'++const components = {+ pre: PreStream as unknown as DefineComponent,+}+</script>++<template>+ <div class="px-6 py-8 max-w-3xl mx-auto w-full">+ <div class="prose prose-sm prose-neutral dark:prose-invert max-w-none">+ <MDCCached+ :value="content"+ cache-key="docs-sdk"+ :components+ :parser-options="{ highlight: false }"+ />+ </div>+ </div>+</template>diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex f1143e1..e75d796 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -6,8 +6,8 @@ This document describes the technical architecture of Savoir, a platform for buiSavoir consists of two main components:-1. **Chat App** (`apps/chat`): A unified Nuxt application that provides both the chat interface and the API for sandbox management, content synchronization, and source management-2. **SDK** (`packages/sdk`): A client library providing AI SDK-compatible tools+1. **Chat App** (`apps/chat`): A unified [Nuxt](https://nuxt.com) application that provides the chat interface, API, bot integrations, sandbox management, and content synchronization+2. **SDK** (`packages/sdk`): A client library providing [AI SDK](https://ai-sdk.dev)-compatible tools (`bash`, `bash_batch`)```┌─────────────────────────────────────────────────────────────────────────┐@@ -20,13 +20,13 @@ Savoir consists of two main components:│ │ │ ││ │ const savoir = createSavoir({ apiKey, apiUrl }) │ ││ │ const { text } = await generateText({ │ │-│ │ model: 'google/gemini-3-flash', │ │-│ │ tools: { ...savoir.tools } │ │+│ │ model: yourModel, │ │+│ │ tools: savoir.tools // bash, bash_batch │ ││ │ }) │ ││ └────────────────────────────────────────────────────────────────────┘ │└─────────────────────────────────────────────────────────────────────────┘│- │ HTTPS (API Key auth)+ │ HTTPS (Better Auth session or API key)▼┌─────────────────────────────────────────────────────────────────────────┐│ apps/chat │@@ -35,21 +35,22 @@ Savoir consists of two main components:│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────┐ ││ │ /api/sandbox/* │ │ /api/sync/* │ │ /api/sources/* │ ││ │ │ │ │ │ │ │-│ │ - snapshot │ │ - POST /sync │ │ - GET /sources │ │-│ │ - search-and- │ │ - POST /sync/:s │ │ - POST /sources │ │-│ │ read │ │ │ │ - PUT /sources/:id │ │-│ │ - read │ │ │ │ - DELETE /sources/:id │ │+│ │ - shell (bash) │ │ - POST /sync │ │ - GET /sources │ │+│ │ - snapshot │ │ - POST /sync/:s │ │ - POST /sources │ │+│ │ │ │ │ │ - PUT /sources/:id │ │+│ │ │ │ │ │ - DELETE /sources/:id │ ││ └────────┬────────┘ └────────┬────────┘ └────────────┬────────────┘ ││ │ │ │ ││ ▼ ▼ ▼ ││ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────┐ ││ │ Sandbox Manager │ │ Vercel Workflow │ │ NuxtHub DB │ ││ │ │ │ │ │ │ │-│ │ - Lifecycle │ │ - Durable exec │ │ - SQLite (sources) │ │-│ │ - KV caching │ │ - Auto retries │ │ - KV (sessions) │ │-│ │ - Command exec │ │ - Sync sources │ │ - Blob (uploads) │ │-│ └────────┬────────┘ └────────┬────────┘ └─────────────────────────┘ │-└───────────┼────────────────────┼────────────────────────────────────────┘+│ │ - Lifecycle │ │ - Durable exec │ │ - SQLite (sources, │ │+│ │ - KV caching │ │ - Auto retries │ │ chats, messages, │ │+│ │ - Command exec │ │ - Sync sources │ │ agent_config, stats) │ │+│ └────────┬────────┘ └────────┬────────┘ │ - KV (sessions, cache) │ │+│ │ │ │ - Blob (uploads) │ │+└───────────┼────────────────────┼────────────┴─────────────────────────┘ ││ │▼ ▼┌────────────┐ ┌────────────┐@@ -63,52 +64,45 @@ Savoir consists of two main components:```savoir/-├── savoir.config.ts # Source definitions (for seeding DB)├── apps/│ └── chat/ # Unified Nuxt application│ ├── app/ # Vue app (components, pages, composables)+│ ├── shared/ # Shared types (tool-call, snapshot)│ ├── server/│ │ ├── api/ # API routes-│ │ │ ├── sandbox/ # Sandbox endpoints-│ │ │ │ ├── snapshot.post.ts-│ │ │ │ ├── search-and-read.post.ts-│ │ │ │ └── read.post.ts+│ │ │ ├── sandbox/ # Sandbox endpoints (shell, snapshot)│ │ │ ├── sync/ # Sync endpoints-│ │ │ │ ├── index.post.ts-│ │ │ │ └── [source].post.ts-│ │ │ └── sources/ # Sources CRUD-│ │ ├── lib/-│ │ │ └── sandbox/ # Sandbox manager library-│ │ │ ├── types.ts # Type definitions-│ │ │ ├── context.ts # Sandbox creation-│ │ │ ├── session.ts # Session management (KV)-│ │ │ ├── snapshot.ts # Snapshot operations (KV)-│ │ │ ├── manager.ts # High-level sandbox API-│ │ │ ├── git.ts # Git operations in sandbox-│ │ │ ├── source-sync.ts # Content sync helpers-│ │ │ └── index.ts # Public exports+│ │ │ ├── sources/ # Sources CRUD+│ │ │ ├── chats/ # Chat CRUD + AI streaming+│ │ │ ├── agent-config/ # Agent configuration+│ │ │ ├── stats/ # Usage statistics+│ │ │ ├── webhooks/ # Bot webhooks (GitHub, Discord)+│ │ │ ├── upload/ # File uploads+│ │ │ ├── messages/ # Message feedback+│ │ │ ├── shared/ # Shared (public) chats+│ │ │ ├── snapshot/ # Snapshot status/sync+│ │ │ ├── admin/ # Admin API keys+│ │ │ └── discord/ # Discord gateway+│ │ ├── db/ # Drizzle schema + migrations+│ │ ├── utils/+│ │ │ ├── sandbox/ # Sandbox manager library+│ │ │ ├── bot/ # Bot framework (adapters, AI, types)+│ │ │ ├── chat/ # Chat helpers + admin tools+│ │ │ ├── router/ # Complexity router (schema, route-question)+│ │ │ ├── prompts/ # System prompts (chat, bot, router, shared)+│ │ │ └── image/ # Image processing│ │ ├── workflows/ # Vercel Workflows│ │ │ ├── sync-docs/-│ │ │ └── create-snapshot/-│ │ ├── tasks/ # Nitro tasks-│ │ │ └── seed-sources.ts-│ │ ├── middleware/ # API auth-│ │ │ └── api-auth.ts-│ │ ├── db/ # Drizzle schema-│ │ │ └── schema.ts-│ │ └── plugins/ # Nitro plugins-│ │ └── logger.ts+│ │ │ ├── create-snapshot/+│ │ │ └── compute-stats/+│ │ └── auth.config.ts # Better Auth config│ └── nuxt.config.ts-├── shared/-│ └── types/-│ └── snapshot.ts # SnapshotSyncStatus type├── packages/-│ ├── sdk/ # @savoir/sdk-│ │ └── src/-│ │ ├── index.ts # Public exports-│ │ ├── client.ts # HTTP client-│ │ └── tools/ # AI SDK tool definitions-│ └── logger/ # @savoir/logger+│ └── sdk/ # @savoir/sdk+│ └── src/+│ ├── index.ts # Public exports+│ ├── client.ts # HTTP client (SavoirClient)+│ └── tools/ # AI SDK tool definitions (bash, bash_batch)└── docs/ # Documentation```@@ -116,7 +110,7 @@ savoir/### 1. Sources (Database)-Sources are stored in SQLite via NuxtHub. They can be managed via the API or seeded from `savoir.config.ts`.+Sources are stored in SQLite via [NuxtHub](https://hub.nuxt.com) and managed through the admin interface.#### Database Schema@@ -125,6 +119,7 @@ export const sources = sqliteTable('sources', {id: text('id').primaryKey(),type: text('type', { enum: ['github', 'youtube'] }).notNull(),label: text('label').notNull(),+ basePath: text('base_path').default('/docs'),// GitHub fieldsrepo: text('repo'),@@ -145,27 +140,25 @@ export const sources = sqliteTable('sources', {### 2. SDK (`@savoir/sdk`)-The SDK provides AI SDK-compatible tools that communicate with the Savoir API.+The SDK provides [AI SDK](https://ai-sdk.dev)-compatible tools that communicate with the Savoir API via `POST /api/sandbox/shell`.#### API Design```typescript-import { tool } from 'ai'-import { z } from 'zod'--export interface SavoirOptions {- apiKey: string- apiUrl: string- chatId?: string // Optional: reuse sandbox across calls-}--export interface SavoirClient {- searchAndRead: ReturnType<typeof tool> // Search and read files (recommended)- search: ReturnType<typeof tool> // Search for files- read: ReturnType<typeof tool> // Read specific files-}--export function createSavoir(options: SavoirOptions): SavoirClient+import { createSavoir } from '@savoir/sdk'++const savoir = createSavoir({+ apiUrl: 'https://your-savoir-instance.com',+ apiKey: 'your-api-key', // Better Auth API key+ sessionId: 'optional', // Reuse sandbox session+ onToolCall: (info) => {}, // Tool execution callback+})++// Returns:+// savoir.tools.bash - Execute single bash command in sandbox+// savoir.tools.bash_batch - Execute multiple commands in one request+// savoir.client - Low-level SavoirClient for direct API access+// savoir.getAgentConfig() - Fetch agent configuration```### 3. Sandbox System@@ -195,277 +188,211 @@ The sandbox system uses [Vercel Sandbox Snapshots](https://vercel.com/docs/verce│ instant start▼┌─────────────────────────────────────────────────────────────────────────┐-│ API Sandbox Endpoints │+│ API: POST /api/sandbox/shell ││ │-│ Request ──▶ Start from snapshot ──▶ Search/Read ──▶ Return results │-│ (instant) (grep/cat) │+│ Request ──▶ Start from snapshot ──▶ Execute bash ──▶ Return results │+│ (instant) (grep/cat/find) │└─────────────────────────────────────────────────────────────────────────┘```-### 4. Content Sync+#### Allowed Commands-Content synchronization uses [Vercel Workflow](https://github.com/vercel/workflow) for durable execution with automatic retries.+The sandbox only permits read-only commands: `find`, `ls`, `tree`, `grep`, `cat`, `head`, `tail`, `wc`, `sort`, `uniq`, `diff`, `echo`, `stat`, `file`, `du`, `basename`, `dirname`, `realpath`, `xargs`.-```-POST /api/sync- │- ▼-┌─────────────────────────────────────────────────────────────────────────┐-│ Vercel Workflow │-│ (Durable execution engine) │-│ │-│ ┌───────────────────────────────────────────────────────────────────┐ │-│ │ stepSyncAll │ │-│ │ │ │-│ │ 1. Create sandbox from snapshot │ │-│ │ 2. Sync all sources (GitHub repos, etc.) │ │-│ │ 3. Git commit and push changes │ │-│ │ 4. Take new snapshot │ │-│ └───────────────────────────────────────────────────────────────────┘ │-│ │-│ Features: │-│ - Automatic retries on failure │-│ - Progress tracking via `bun run workflow:web` │-│ - Durable state (survives server restarts) │-│ - All operations in single step (avoids serialization issues) │-└─────────────────────────────────────────────────────────────────────────┘-```+Blocked: `rm`, `curl`, `wget`, `git`, `ssh`, `sudo`, command substitution, redirects, interpreters.-## API Endpoints+### 4. AI Agent (Router + Model Selection)-### Authentication+Savoir uses a complexity-based router to select the appropriate model and step count for each question.-Authentication is optional for `/api/sandbox/*` and `/api/sync/*` endpoints. If `SAVOIR_SECRET_KEY` is configured, these endpoints require an API key:+#### Router Flow-```-Authorization: Bearer <api-key>-```+1. User sends a message+2. `routeQuestion()` sends the question to a lightweight model (`google/gemini-2.5-flash-lite`) with the `agentConfigSchema`+3. The router classifies the question by complexity and returns:+ - `complexity`: trivial | simple | moderate | complex+ - `maxSteps`: 4 | 8 | 15 | 25+ - `model`: flash for trivial/simple, sonnet for moderate, opus for complex+ - `reasoning`: brief explanation+4. Admin agent config (from DB) can override the model and multiply max steps+5. The `ToolLoopAgent` runs with the selected model and tools-If not configured, the API is open (useful for local development).+#### Agent Configuration (Admin)-### Sandbox Endpoints+Stored in the `agent_config` table, configurable via `/admin/agent`:-#### POST /api/sandbox/snapshot+| Field | Description |+|-------|-------------|+| `additionalPrompt` | Extra instructions appended to system prompt |+| `responseStyle` | concise, detailed, technical, friendly |+| `language` | Response language (e.g. `en`) |+| `defaultModel` | Override the router's model selection |+| `maxStepsMultiplier` | Multiplier for router's maxSteps |+| `temperature` | Model temperature (0-1) |+| `searchInstructions` | Custom search tips for the agent |+| `citationFormat` | inline, footnote, none |-Create a new snapshot from the documentation repository.+### 5. Content Sync-**Response:**-```json-{- "status": "started",- "message": "Snapshot workflow started."-}-```+Content synchronization uses [Vercel Workflow](https://useworkflow.dev) for durable execution with automatic retries.-#### POST /api/sandbox/search-and-read+### 6. Bot Integrations-Combined search and read operation (recommended).+#### GitHub Bot-**Request:**-```json-{- "query": "useAsyncData",- "limit": 20,- "sessionId": "optional-session-id"-}-```+- Uses `SavoirGitHubAdapter` via the [Vercel Chat SDK](https://github.com/vercel-labs/chat)+- Responds to mentions in issues and PRs+- Can optionally reply to new issues (`replyToNewIssues` config)+- Webhook: `POST /api/webhooks/github`-**Response:**-```json-{- "sessionId": "sess_1234567890_abc123",- "matches": [- { "path": "docs/nuxt/composables/use-async-data.md", "lineNumber": 10, "content": "..." }- ],- "files": [- { "path": "docs/nuxt/composables/use-async-data.md", "content": "# useAsyncData\n\n..." }- ]-}-```+#### Discord Bot-#### POST /api/sandbox/read+- Uses [`@chat-adapter/discord`](https://github.com/vercel-labs/chat)+- Responds to mentions and continues conversations in threads+- Webhook: `POST /api/webhooks/discord`-Read specific files by path.+Both bots use the same AI agent pipeline (router + tools) as the chat interface but with internal sandbox access (no HTTP round-trip).-**Request:**-```json-{- "paths": ["docs/nuxt/guide/middleware.md"],- "sessionId": "optional-session-id"-}-```+## API Endpoints-**Response:**-```json-{- "sessionId": "sess_1234567890_abc123",- "files": [- { "path": "docs/nuxt/guide/middleware.md", "content": "# Middleware\n\n..." }- ]-}-```+### Authentication-### Sync Endpoints+Authentication uses [Better Auth](https://www.better-auth.com) with the [`@onmax/nuxt-better-auth`](https://github.com/onmax/nuxt-better-auth) module:+- **Email/password** and **GitHub OAuth** sign-in (via a [GitHub App](https://docs.github.com/en/apps), scope `user:email`)+- **API keys** via Better Auth's [`apiKey` plugin](https://www.better-auth.com/docs/plugins/api-key) (supports `Authorization: Bearer <key>` and `x-api-key` headers)+- **Admin role** granted to users whose email or username matches `NUXT_ADMIN_USERS`-#### POST /api/sync+### Sandbox-Sync all sources.+| Method | Endpoint | Auth | Description |+|--------|----------|------|-------------|+| POST | `/api/sandbox/shell` | User/API key | Execute bash command(s) in sandbox |+| POST | `/api/sandbox/snapshot` | Admin | Create a new snapshot |-**Response:**-```json-{- "status": "started",- "message": "Sync workflow started."-}-```+### Chats-#### POST /api/sync/:source+| Method | Endpoint | Auth | Description |+|--------|----------|------|-------------|+| GET | `/api/chats` | User | List user's chats |+| POST | `/api/chats` | User | Create a new chat |+| GET | `/api/chats/:id` | User | Get chat with messages |+| POST | `/api/chats/:id` | User | Stream AI response |+| DELETE | `/api/chats/:id` | User | Delete chat |+| PATCH | `/api/chats/:id/share` | User | Toggle sharing |-Sync a specific source.+### Sources-**Response:**-```json-{- "status": "started",- "message": "Sync workflow started for source \"nuxt\".",- "source": "nuxt"-}-```+| Method | Endpoint | Auth | Description |+|--------|----------|------|-------------|+| GET | `/api/sources` | Public | List all sources |+| POST | `/api/sources` | Admin | Create source |+| PUT | `/api/sources/:id` | Admin | Update source |+| DELETE | `/api/sources/:id` | Admin | Delete source |-### Snapshot Endpoints+### Sync-#### GET /api/snapshot/status+| Method | Endpoint | Auth | Description |+|--------|----------|------|-------------|+| POST | `/api/sync` | Admin | Sync all sources |+| POST | `/api/sync/:source` | Admin | Sync specific source |-Get current vs latest snapshot status. Returns whether the sandbox needs to sync to a newer snapshot. Admin only.+### Snapshot-**Response:**-```json-{- "currentSnapshotId": "snap_abc123",- "latestSnapshotId": "snap_def456",- "needsSync": true,- "latestCreatedAt": 1706400000000-}-```+| Method | Endpoint | Auth | Description |+|--------|----------|------|-------------|+| GET | `/api/snapshot/status` | Admin | Current vs latest snapshot |+| POST | `/api/snapshot/sync` | Admin | Sync to latest snapshot |-#### POST /api/snapshot/sync+### Agent Config-Sync sandbox to the latest Vercel snapshot. Admin only.+| Method | Endpoint | Auth | Description |+|--------|----------|------|-------------|+| GET | `/api/agent-config` | Admin | Get agent configuration |+| PUT | `/api/agent-config` | Admin | Update agent configuration |+| POST | `/api/agent-config/reset` | Admin | Reset to defaults |+| GET | `/api/agent-config/public` | User/API key | Get active config (for SDK/bots) |-**Response:**-```json-{- "status": "synced",- "snapshotId": "snap_def456"-}-```--### Sources Endpoints+### Stats-#### GET /api/sources+| Method | Endpoint | Auth | Description |+|--------|----------|------|-------------|+| GET | `/api/stats` | Admin | Global usage stats |+| GET | `/api/stats/me` | User | User's own stats |+| POST | `/api/stats/usage` | User/API key | Report usage (SDK/bots) |+| POST | `/api/stats/compute` | Admin | Recompute stats |-List all sources grouped by type.+### Other-**Response:**-```json-{- "github": [...],- "youtube": [...]-}-```+| Method | Endpoint | Auth | Description |+|--------|----------|------|-------------|+| POST | `/api/webhooks/:platform` | Platform | Bot webhooks (GitHub, Discord) |+| PUT | `/api/upload/:chatId` | User | Upload file to chat |+| DELETE | `/api/upload/...pathname` | User | Delete uploaded file |+| PATCH | `/api/messages/:id/feedback` | User | Set feedback (positive/negative) |+| GET | `/api/shared/:token` | Public | Get shared chat (ISR cached) |## Storage Strategy-### NuxtHub SQLite+### [NuxtHub](https://hub.nuxt.com) SQLite-Used for persistent data:+Used for persistent data (via [Drizzle ORM](https://orm.drizzle.team)):-```typescript-// Sources table-sources: { id, type, label, repo, branch, ... }--// Users, chats, messages tables for chat app+```+chats - Chat sessions (id, title, userId, mode, sharing)+messages - Chat messages (role, parts, feedback, model, tokens)+sources - Content sources (GitHub repos, YouTube channels)+agent_config - AI agent configuration (admin-managed)+api_usage - External API usage tracking (bots, SDK)+usage_stats - Aggregated daily usage statistics+user/session/account - Better Auth managed tables```-### NuxtHub KV+### [NuxtHub KV](https://hub.nuxt.com/docs/features/kv)Used for caching and session management:-```typescript-// Current snapshot (updated by sync workflow)-`snapshot:current` -> { snapshotId: string, createdAt: number }--// Active sandbox sessions-`session:${sessionId}` -> { sandboxId: string, snapshotId: string, ... }+```+snapshot:current - Current snapshot ID and creation timestamp+session:{sessionId} - Active sandbox sessions+snapshot:status-cache - Snapshot status cache (1 min TTL)+sources:last-sync - Last sync timestamp+agent-config:active - Cached active agent config (60s TTL)+```-// Snapshot status cache (1 min TTL)-`snapshot:status-cache` -> { latestSnapshotId: string, latestCreatedAt: number, cachedAt: number }+### [NuxtHub Blob](https://hub.nuxt.com/docs/features/blob)-// Last source sync timestamp-`sources:last-sync` -> number (timestamp)-```+Used for file uploads (images, PDFs, CSVs attached to chats).### Snapshot Repository-Structure:-```{GITHUB_SNAPSHOT_REPO}/├── docs/│ ├── nuxt/│ │ ├── getting-started/-│ │ │ └── installation.md│ │ └── composables/-│ │ └── use-async-data.md│ ├── nitro/│ └── ...└── youtube/└── alex-lichter/- └── nuxt-4-overview-TAoTh4DqH6A.md```## Environment Variables-| Variable | Required | Description |-|----------|----------|-------------|-| `NUXT_GITHUB_TOKEN` | Yes | GitHub token for API access |-| `NUXT_GITHUB_SNAPSHOT_REPO` | Yes | Snapshot repository (owner/repo) |-| `NUXT_GITHUB_SNAPSHOT_BRANCH` | No | Branch to use (default: main) |-| `NUXT_SAVOIR_SECRET_KEY` | No | API key for authentication |+See [ENVIRONMENT.md](./ENVIRONMENT.md) for the complete list.## Security-### API Key Validation (Optional)--Authentication is **optional** and only enforced if `NUXT_SAVOIR_SECRET_KEY` is set.--- If `NUXT_SAVOIR_SECRET_KEY` is not set → API is open (useful for development)-- If `NUXT_SAVOIR_SECRET_KEY` is set → `/api/sync/*` and `/api/sandbox/*` require `Authorization: Bearer <key>`--```typescript-// apps/chat/server/middleware/api-auth.ts-export default defineEventHandler((event) => {- const path = getRequestURL(event).pathname-- if (!path.startsWith('/api/sync') && !path.startsWith('/api/sandbox')) {- return- }-- const config = useRuntimeConfig()- if (!config.savoirSecretKey) return-- const authHeader = getHeader(event, 'Authorization')- const apiKey = authHeader?.replace('Bearer ', '')+### Authentication- if (apiKey !== config.savoirSecretKey) {- throw createError({ statusCode: 401, message: 'Unauthorized' })- }-})-```+- Better Auth handles user sessions, OAuth, and API keys+- API keys support both `Authorization: Bearer <key>` and `x-api-key` headers+- Admin role is automatically assigned based on `NUXT_ADMIN_USERS`### Sandbox Isolation-- Each sandbox runs in isolated Vercel environment-- No network access from within sandbox-- Read-only filesystem (cloned repo)-- Commands limited to grep, cat, find, ls+- Each sandbox runs in an isolated Vercel environment+- No network access from within the sandbox+- Read-only filesystem (cloned snapshot repo)+- Commands limited to read-only operations (grep, cat, find, ls, etc.)+- Blocked: destructive commands, network tools, interpreters, redirectsdiff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 63c4d83..60e896d 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -1,37 +1,101 @@# Environment Variables-## Chat App (`apps/chat`)+All environment variables for the Savoir project, organized by category.-```bash-# Required for sandbox/sync-NUXT_GITHUB_TOKEN=ghp_... # GitHub token for repo access-NUXT_GITHUB_SNAPSHOT_REPO=org/repo # Snapshot repository-NUXT_GITHUB_SNAPSHOT_BRANCH=main # Snapshot branch (default: main)+## Core-# Optional-NUXT_SAVOIR_SECRET_KEY=... # API authentication key-```+| Variable | Required | Description |+|----------|----------|-------------|+| `NUXT_PUBLIC_SITE_URL` | Yes | Public URL of your Savoir instance |+| `NUXT_ADMIN_USERS` | Yes | Comma-separated list of admin emails or GitHub usernames |++## Authentication++| Variable | Required | Description |+|----------|----------|-------------|+| `BETTER_AUTH_SECRET` | Yes | Secret used to sign sessions and tokens |+| `GITHUB_CLIENT_ID` | Yes | GitHub OAuth app client ID |+| `GITHUB_CLIENT_SECRET` | Yes | GitHub OAuth app client secret |+| `NUXT_SESSION_PASSWORD` | No | Session encryption password |++## AI++| Variable | Required | Description |+|----------|----------|-------------|+| `AI_GATEWAY_API_KEY` | Yes | Vercel AI Gateway API key (used by `@ai-sdk/gateway`) |++## Sandbox & Sync++| Variable | Required | Description |+|----------|----------|-------------|+| `NUXT_GITHUB_TOKEN` | Yes | GitHub token for cloning repos and accessing the snapshot repository |+| `NUXT_GITHUB_SNAPSHOT_REPO` | Yes | Snapshot repository in `owner/repo` format |+| `NUXT_GITHUB_SNAPSHOT_BRANCH` | No | Snapshot branch (default: `main`) |++## GitHub Bot++| Variable | Required | Description |+|----------|----------|-------------|+| `NUXT_PUBLIC_GITHUB_BOT_TRIGGER` | Yes* | Bot mention trigger (e.g. `@nuxt-agent`). Required if using the GitHub bot. |+| `NUXT_GITHUB_APP_ID` | Yes* | GitHub App ID |+| `NUXT_GITHUB_APP_PRIVATE_KEY` | Yes* | GitHub App private key (PEM format, can be base64-encoded) |+| `NUXT_GITHUB_WEBHOOK_SECRET` | Yes* | Webhook secret set when creating the GitHub App |++\* Required only if enabling the GitHub bot integration.++## Discord Bot++| Variable | Required | Description |+|----------|----------|-------------|+| `NUXT_DISCORD_BOT_TOKEN` | Yes* | Discord bot token |+| `NUXT_DISCORD_PUBLIC_KEY` | Yes* | Discord application public key |+| `NUXT_DISCORD_APPLICATION_ID` | Yes* | Discord application ID |+| `NUXT_DISCORD_MENTION_ROLE_IDS` | No | Comma-separated role IDs that can trigger the bot |++\* Required only if enabling the Discord bot integration.++## YouTube++| Variable | Required | Description |+|----------|----------|-------------|+| `NUXT_YOUTUBE_API_KEY` | Yes* | YouTube Data API key for fetching video transcripts |++\* Required only if syncing YouTube sources.++## Storage++| Variable | Required | Description |+|----------|----------|-------------|+| `BLOB_READ_WRITE_TOKEN` | No | Vercel Blob token for file uploads |++## Optional++| Variable | Required | Description |+|----------|----------|-------------|+| `REDIS_URL` | No | Redis URL for bot state persistence. Falls back to in-memory state if not set. |+| `VERCEL_OIDC_TOKEN` | No | Vercel OIDC token (auto-injected in Vercel deployments) |## SDK (`@savoir/sdk`)-```bash-SAVOIR_API_KEY=... # API key for authentication-SAVOIR_API_URL=https://... # API base URL (your deployed chat app)-```+When using the SDK from an external application:-## Database Commands+| Variable | Required | Description |+|----------|----------|-------------|+| `SAVOIR_API_URL` | Yes | Base URL of your Savoir API |+| `SAVOIR_API_KEY` | No | API key for authentication (Better Auth API key) |-```bash-# Generate migrations-turbo run db:generate --filter=@savoir/chat+## Database++Migrations run automatically when the application starts -- no manual `db:migrate` step is needed.-# Run migrations-turbo run db:migrate --filter=@savoir/chat+```bash+# Generate new migrations after schema changes+bun run db:generate```## Workflows```bash# Monitor workflow progress-turbo run workflow:web --filter=@savoir/chat+bun run workflow:web```diff --git a/docs/SOURCES.md b/docs/SOURCES.mdindex 91efcdb..12f7de8 100644--- a/docs/SOURCES.md+++ b/docs/SOURCES.md@@ -2,6 +2,21 @@Savoir aggregates documentation from multiple sources into a unified, searchable knowledge base.+## Managing Sources++Sources are managed through the **admin interface** at `/admin`. From there you can:++- Add new GitHub repositories or YouTube channels+- Edit existing source configurations+- Delete sources+- Trigger a sync to update the knowledge base++Sources can also be listed programmatically via the SDK:++```typescript+const sources = await savoir.client.getSources()+```+## Database StorageSources are stored in SQLite via NuxtHub. The schema:@@ -10,9 +25,8 @@ Sources are stored in SQLite via NuxtHub. The schema:export const sources = sqliteTable('sources', {id: text('id').primaryKey(),type: text('type', { enum: ['github', 'youtube'] }).notNull(),-- // Common fieldslabel: text('label').notNull(),+ basePath: text('base_path').default('/docs'),// GitHub fieldsrepo: text('repo'),@@ -24,128 +38,63 @@ export const sources = sqliteTable('sources', {// YouTube fieldschannelId: text('channel_id'),handle: text('handle'),- maxVideos: integer('max_videos'),+ maxVideos: integer('max_videos').default(50),- // TimestampscreatedAt: integer('created_at', { mode: 'timestamp' }),updatedAt: integer('updated_at', { mode: 'timestamp' }),})```-## Configuration File--Sources can be defined in `savoir.config.ts` at the project root:--```typescript-// savoir.config.ts-export default {- sources: {- github: [- { id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' },- { id: 'nitro', repo: 'nitrojs/nitro', branch: 'main' },- ],- youtube: [- { id: 'alex-lichter', channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA' },- ],- },-}-```-## Source Types### GitHub SourcesFetches Markdown documentation from GitHub repositories.-```typescript-{- id: string // Unique identifier- label?: string // Display name (defaults to capitalized id)- repo: string // GitHub repository (owner/repo)- branch?: string // Branch to fetch from (default: 'main')- contentPath?: string // Path to content directory (default: 'docs')- outputPath?: string // Output directory in snapshot (default: id)- readmeOnly?: boolean // Only fetch README.md (default: false)-}-```--**Examples:**--```typescript-// Full documentation tree-{ id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' }--// Specific branch-{ id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' }--// README only-{ id: 'ofetch', repo: 'unjs/ofetch', readmeOnly: true }-```+| Field | Type | Description |+|-------|------|-------------|+| `id` | `string` | Unique identifier |+| `label` | `string` | Display name |+| `repo` | `string` | GitHub repository (`owner/repo`) |+| `branch` | `string?` | Branch to fetch from (default: `main`) |+| `contentPath` | `string?` | Path to content directory (default: `docs`) |+| `outputPath` | `string?` | Output directory in snapshot (default: `id`) |+| `basePath` | `string?` | URL base path for this source (default: `/docs`) |+| `readmeOnly` | `boolean?` | Only fetch README.md (default: `false`) |+| `additionalSyncs` | `array?` | Extra repos to sync into the same source |### YouTube SourcesFetches video transcripts from YouTube channels.-```typescript-{- id: string // Unique identifier- label?: string // Display name- channelId: string // YouTube channel ID- handle?: string // YouTube handle (e.g., '@TheAlexLichter')- maxVideos?: number // Maximum videos to fetch (default: 50)-}-```--## Managing Sources via API--### List Sources--```bash-curl http://localhost:3000/api/sources-```--### Create Source--```bash-curl -X POST http://localhost:3000/api/sources \- -H "Content-Type: application/json" \- -d '{- "id": "vue",- "type": "github",- "label": "Vue.js",- "repo": "vuejs/core",- "branch": "main",- "contentPath": "docs"- }'-```+| Field | Type | Description |+|-------|------|-------------|+| `id` | `string` | Unique identifier |+| `label` | `string` | Display name |+| `channelId` | `string` | YouTube channel ID |+| `handle` | `string?` | YouTube handle (e.g. `@TheAlexLichter`) |+| `maxVideos` | `number?` | Maximum videos to fetch (default: `50`) |-### Update Source+## Syncing-```bash-curl -X PUT http://localhost:3000/api/sources/vue \- -H "Content-Type: application/json" \- -d '{"branch": "v4"}'-```+Content syncing is triggered from the **admin interface**. You can also trigger it programmatically via the SDK:-### Delete Source+```typescript+// Sync all sources+await savoir.client.sync()-```bash-curl -X DELETE http://localhost:3000/api/sources/vue+// Sync a specific source+await savoir.client.syncSource('nuxt')```-## Syncing+### How Sync Works-### Via API+1. A Vercel Sandbox is created from the latest snapshot+2. All source repositories are cloned/updated+3. Changes are pushed to the snapshot repository+4. A new sandbox snapshot is taken for instant startup-```bash-# Sync all sources-curl -X POST http://localhost:3000/api/sync \- -H "Authorization: Bearer $SAVOIR_SECRET_KEY"--# Sync specific source-curl -X POST http://localhost:3000/api/sync/nuxt \- -H "Authorization: Bearer $SAVOIR_SECRET_KEY"-```+This runs as a durable Vercel Workflow with automatic retries.### Sync Tracking@@ -153,9 +102,7 @@ The system tracks when sources were last synced:- `lastSyncAt` timestamp is stored in KV after each successful sync- `GET /api/sources` returns the `lastSyncAt` value-- Admin UI (`/admin`) shows a reminder if the last sync was more than 7 days ago--This helps ensure documentation stays up-to-date by providing visibility into sync freshness.+- The admin interface shows a reminder if the last sync was more than 7 days ago## Content Normalization@@ -171,7 +118,7 @@ All content is normalized to Markdown:### Excluded Files-- Lock files (`package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`, etc.)+- Lock files (`package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`, `bun.lock`, etc.)- `node_modules/`- Binary files@@ -180,7 +127,7 @@ All content is normalized to Markdown:The snapshot repository contains all aggregated content:```-{GITHUB_SNAPSHOT_REPO}/+{NUXT_GITHUB_SNAPSHOT_REPO}/├── docs/│ ├── nuxt/│ │ ├── getting-started/@@ -194,5 +141,5 @@ The snapshot repository contains all aggregated content:Configure via environment variable:```bash-GITHUB_SNAPSHOT_REPO=my-org/content-snapshot+NUXT_GITHUB_SNAPSHOT_REPO=my-org/content-snapshot```diff --git a/package.json b/package.jsonindex b622d45..2f06cc0 100644--- a/package.json+++ b/package.json@@ -4,13 +4,12 @@"type": "module","description": "Monorepo for Savoir","author": "HugoRCD",- "funding": "https://github.com/sponsors/HugoRCD","repository": {"type": "git",- "url": "git+https://github.com/HugoRCD/savoir.git"+ "url": "git+https://github.com/vercel-labs/savoir.git"},"bugs": {- "url": "https://github.com/HugoRCD/savoir/issues"+ "url": "https://github.com/vercel-labs/savoir/issues"},"workspaces": ["apps/*",diff --git a/packages/sdk/README.md b/packages/sdk/README.mdindex 8e978d4..eedfca5 100644--- a/packages/sdk/README.md+++ b/packages/sdk/README.md@@ -1,6 +1,6 @@# @savoir/sdk-SDK for [Savoir](https://github.com/HugoRCD/savoir) - AI agents with real-time knowledge base access.+SDK for [Savoir](https://github.com/vercel-labs/savoir) - AI agents with real-time knowledge base access.Provides [AI SDK](https://ai-sdk.dev) compatible tools to query documentation in a sandboxed environment.@@ -20,8 +20,8 @@ Set the following environment variables:| Variable | Required | Description ||----------|----------|-------------|-| `SAVOIR_API_URL` | Yes | Base URL of your Savoir API (e.g., `https://chat.savoir.dev`) |-| `SAVOIR_API_KEY` | No | API key for authentication. Required if the API has `SAVOIR_SECRET_KEY` configured. |+| `SAVOIR_API_URL` | Yes | Base URL of your Savoir API |+| `SAVOIR_API_KEY` | No | API key for authentication ([Better Auth](https://www.better-auth.com) API key). |## Quick Start@@ -71,6 +71,8 @@ const savoir = createSavoir({| `apiUrl` | `string` | Yes | Base URL of your Savoir API || `apiKey` | `string` | No | API key for authentication || `sessionId` | `string` | No | Reuse an existing sandbox session |+| `source` | `string` | No | Usage source identifier (e.g. `'github-bot'`). Defaults to `'sdk'`. |+| `sourceId` | `string` | No | Tracking ID (e.g. `'issue-123'`). Can be overridden per `reportUsage()` call. || `onToolCall` | `function` | No | Callback fired on tool execution (loading/done/error) |#### Returns@@ -85,6 +87,7 @@ interface Savoir {getSessionId(): string | undefinedsetSessionId(sessionId: string): voidgetAgentConfig(): Promise<AgentConfig>+ reportUsage(result: GenerateResult, options?: ReportUsageOptions): Promise<void>}```@@ -92,7 +95,7 @@ interface Savoir {#### `bash`-Execute a single bash command in the documentation sandbox.+Execute a single bash command in the documentation [sandbox](https://vercel.com/docs/vercel-sandbox).```tsconst { text } = await generateText({@@ -104,7 +107,7 @@ const { text } = await generateText({#### `bash_batch`-Execute multiple bash commands in a single request. More efficient than multiple single calls as the sandbox is reused between commands.+Execute multiple bash commands in a single request. More efficient than multiple single calls as the [sandbox](https://vercel.com/docs/vercel-sandbox) is reused between commands.```tsconst { text } = await generateText({@@ -141,6 +144,9 @@ await client.sync({ reset: false, push: true })// Get agent configurationconst config = await client.getAgentConfig()++// Report usage from an AI SDK generate result+await client.reportUsage(result, { startTime: Date.now() })```## Error Handling@@ -202,6 +208,8 @@ import type {SyncResponse,SourcesResponse,AgentConfig,+ GenerateResult,+ ReportUsageOptions,ToolCallInfo,ToolCallCallback,ToolCallState,diff --git a/packages/sdk/package.json b/packages/sdk/package.jsonindex 3120e22..6f203c0 100644--- a/packages/sdk/package.json+++ b/packages/sdk/package.json@@ -6,12 +6,12 @@"author": "Hugo Richard","repository": {"type": "git",- "url": "git+https://github.com/hugorichard/savoir.git",+ "url": "git+https://github.com/vercel-labs/savoir.git","directory": "packages/sdk"},- "homepage": "https://github.com/hugorichard/savoir/tree/main/packages/sdk#readme",+ "homepage": "https://github.com/vercel-labs/savoir/tree/main/packages/sdk#readme","bugs": {- "url": "https://github.com/hugorichard/savoir/issues"+ "url": "https://github.com/vercel-labs/savoir/issues"},"keywords": ["savoir",