Open source file-system and knowledge based agent template. Build AI agents that stay up to date with your knowledge base
update docs
Hugo Richardf39b6a2abe68
diff --git a/README.md b/README.mdindex bb46a5b..4840b52 100644--- a/README.md+++ b/README.md@@ -41,6 +41,7 @@ Savoir provides the infrastructure to create file-based AI agents (chatbots, Dis| Package | Description ||---------|-------------|| [`@savoir/sdk`](./packages/sdk) | AI SDK compatible tools for agents |+| [`@savoir/config`](./packages/config) | Configuration management with multi-format support || [`apps/api`](./apps/api) | Self-hostable sandbox management API |## Quick Start@@ -100,9 +101,33 @@ GITHUB_SNAPSHOT_REPO=your-org/your-content-repoSAVOIR_SECRET_KEY=your-secret-key```+## Configuration++Sources are configured in `savoir.config.ts` at the project root:++```typescript+import { defineConfig } from '@savoir/config'++export default defineConfig({+ sources: {+ github: [+ { id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' },+ { id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' },+ ],+ youtube: [+ { id: 'alex-lichter', channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA' },+ ],+ },+})+```++Supported formats: `.ts`, `.js`, `.json`, `.yaml`++See [SOURCES.md](./docs/SOURCES.md) for detailed source configuration options.+## How It Works-1. **Content Aggregation**: Sources (GitHub docs, YouTube transcripts, etc.) are synced to a snapshot repository via Nitro tasks+1. **Content Aggregation**: Sources (GitHub docs, YouTube transcripts, etc.) are synced to a snapshot repository via Vercel Workflow2. **Sandbox Creation**: When an agent needs to search, the API creates/recovers a Vercel Sandbox with the snapshot repo cloned3. **File-based Search**: The SDK tools execute grep/find commands in the sandbox to search and read content4. **AI Integration**: Tools are compatible with the Vercel AI SDK for seamless integration with any LLMdiff --git a/apps/api/README.md b/apps/api/README.mdindex 98f9918..fe8e5bd 100644--- a/apps/api/README.md+++ b/apps/api/README.md@@ -1,6 +1,6 @@# Savoir API-Self-hostable API server for managing content synchronization and sandbox operations.+Self-hostable API server for managing content synchronization.## Setup@@ -38,42 +38,19 @@ pnpm preview## Sync Commands-Content synchronization uses [Vercel Workflow DevKit](https://useworkflow.dev/) for durable execution with automatic retries.+Content synchronization uses [Vercel Workflow](https://github.com/vercel/workflow) for durable execution with automatic retries.```bash# Start the dev server firstpnpm dev-# In another terminal, trigger async workflow sync-pnpm sync:docs+# In another terminal, trigger sync+curl -X POST http://localhost:3000/api/sync# Monitor workflow executionpnpm workflow:web```-**Via curl with options:**--```bash-# Async workflow (recommended)-curl -X POST http://localhost:3000/api/sync/workflow--# With options-curl -X POST http://localhost:3000/api/sync/workflow \- -H "Content-Type: application/json" \- -d '{"source": "nuxt-ui", "push": false}'--# Sync specific source (legacy, synchronous)-curl -X POST http://localhost:3000/api/sync/nuxt-ui-```--**Options:**--| Option | Type | Default | Description |-|--------|------|---------|-------------|-| `source` | string | - | Sync only this source ID |-| `reset` | boolean | false | Clear content before sync |-| `push` | boolean | true | Push to snapshot repo after sync |-## API Endpoints### GET /api@@ -99,16 +76,15 @@ List all configured content sources.}```-### POST /api/sync/workflow+### POST /api/sync-Trigger async sync workflow (recommended). Uses Workflow DevKit for automatic retries.+Sync all sources.**Body (optional):**```json{"reset": false,- "push": true,- "source": "nuxt-ui"+ "push": true}```@@ -121,37 +97,6 @@ Trigger async sync workflow (recommended). Uses Workflow DevKit for automatic re}```-### POST /api/sync--Trigger full content synchronization (synchronous).--**Body (optional):**-```json-{- "reset": false,- "push": true-}-```--**Response:**-```json-{- "success": true,- "summary": {- "total": 25,- "success": 25,- "failed": 0,- "files": 1234- },- "push": {- "success": true,- "commitSha": "abc123...",- "filesChanged": 1234- },- "results": [...]-}-```-### POST /api/sync/:sourceSync a specific source.@@ -167,18 +112,24 @@ Sync a specific source.}```-## Adding Sources+## Configuration-Edit `utils/sources/sources.ts` to add new sources:+Sources are configured in `savoir.config.ts` at the project root:```typescript-{- id: 'my-project',- label: 'My Project',- type: 'github',- repo: 'org/repo',- branch: 'main',- contentPath: 'docs',- outputPath: 'my-project',-}+import { defineConfig } from '@savoir/config'++export default defineConfig({+ sources: {+ github: [+ { id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' },+ { id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' },+ ],+ youtube: [+ { id: 'alex-lichter', channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA' },+ ],+ },+})```++See [SOURCES.md](/docs/SOURCES.md) for details on source configuration.diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 905dd0a..eb55a9e 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -4,10 +4,11 @@ This document describes the technical architecture of Savoir, a platform for bui## System Overview-Savoir consists of two main components:+Savoir consists of three main components:1. **Savoir API** (`apps/api`): A Nitro-based server that manages Vercel Sandboxes and content synchronization2. **Savoir SDK** (`packages/sdk`): A client library providing AI SDK-compatible tools+3. **Savoir Config** (`packages/config`): Configuration management with multi-format support```┌─────────────────────────────────────────────────────────────────────────┐@@ -20,35 +21,35 @@ Savoir consists of two main components:│ │ │ ││ │ const savoir = createSavoir({ apiKey, apiUrl }) │ ││ │ const { text } = await generateText({ │ │-│ │ model: 'google/gemini-3-flash', │ │+│ │ model: 'google/gemini-3-flash', │ ││ │ tools: { ...savoir.tools } │ ││ │ }) │ ││ └────────────────────────────────────────────────────────────────────┘ │-└─────────────────────────────────────┬───────────────────────────────────┘- │- │ HTTPS (API Key auth)- ▼+└─────────────────────────────────────────────────────────────────────────┘+ │+ │ HTTPS (API Key auth)+ ▼┌─────────────────────────────────────────────────────────────────────────┐│ Savoir API ││ (Nitro Server) ││ ││ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────┐ │-│ │ /api/sandbox/* │ │ /api/sync/* │ │ Nitro Tasks │ │+│ │ /api/sandbox/* │ │ /api/sync/* │ │ /api/sources/* │ ││ │ │ │ │ │ │ │-│ │ - create │ │ - POST /sync │ │ - sync:content (cron) │ │-│ │ - search │ │ - GET /sources │ │ - cleanup:sandboxes │ │+│ │ - create │ │ - POST /sync │ │ - GET /sources │ │+│ │ - search │ │ - POST /sync/:s │ │ │ ││ │ - read │ │ │ │ │ ││ │ - searchAndRead │ │ │ │ │ │+│ └────────┬────────┘ └────────┬────────┘ └────────────┬────────────┘ │+│ │ │ │ │+│ ▼ ▼ ▼ │+│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────┐ │+│ │ Sandbox Manager │ │ Vercel Workflow │ │ @savoir/config │ │+│ │ │ │ │ │ │ │+│ │ - Lifecycle │ │ - Durable exec │ │ - Load savoir.config.ts │ │+│ │ - KV caching │ │ - Auto retries │ │ - Validation │ │+│ │ - Command exec │ │ - Sync sources │ │ - Normalization │ ││ └────────┬────────┘ └────────┬────────┘ └─────────────────────────┘ │-│ │ │ │-│ ▼ ▼ │-│ ┌─────────────────┐ ┌─────────────────┐ │-│ │ Sandbox Manager │ │ Content Syncer │ │-│ │ │ │ │ │-│ │ - Lifecycle │ │ - Fetch sources │ │-│ │ - KV caching │ │ - Transform │ │-│ │ - Command exec │ │ - Push to repo │ │-│ └────────┬────────┘ └────────┬────────┘ │└───────────┼────────────────────┼────────────────────────────────────────┘│ │▼ ▼@@ -59,9 +60,111 @@ Savoir consists of two main components:└────────────┘ └────────────┘```+## Project Structure++```+savoir/+├── savoir.config.ts # Source configuration (root level!)+├── apps/+│ └── api/ # Nitro API server+│ ├── routes/+│ │ └── api/+│ │ ├── index.ts # GET /api - Health check+│ │ ├── sources/ # Source endpoints+│ │ │ └── index.get.ts # GET /api/sources+│ │ ├── sync/ # Sync endpoints+│ │ │ ├── index.post.ts # POST /api/sync+│ │ │ └── [source].post.ts # POST /api/sync/:source+│ │ └── sandbox/ # Sandbox endpoints (planned)+│ │ ├── search.post.ts+│ │ ├── read.post.ts+│ │ └── search-and-read.post.ts+│ ├── workflows/+│ │ └── sync-docs/ # Vercel Workflow for syncing+│ └── plugins/+│ └── config.ts # Config loader plugin+├── packages/+│ ├── config/ # @savoir/config+│ │ └── src/+│ │ ├── index.ts # Public exports+│ │ ├── types.ts # Source types+│ │ ├── define.ts # defineConfig()+│ │ ├── loader.ts # c12-based loading+│ │ ├── validation.ts # Zod schemas+│ │ └── normalize.ts # Default values+│ ├── sdk/ # @savoir/sdk+│ └── logger/ # @savoir/logger+└── docs/ # Documentation+```+## Component Details-### 1. SDK (`@savoir/sdk`)+### 1. Configuration (`@savoir/config`)++Sources are configured in `savoir.config.ts` at the project root. This file is visible, easy to edit, and supports multiple formats via [c12](https://github.com/unjs/c12).++#### Supported Formats++- `savoir.config.ts` (recommended)+- `savoir.config.js`+- `savoir.config.json`+- `savoir.config.yaml`+- `.savoirrc` (JSON)++#### Configuration Schema++```typescript+// savoir.config.ts+import { defineConfig } from '@savoir/config'++export default defineConfig({+ sources: {+ github: [+ {+ id: 'nuxt',+ label: 'Nuxt',+ repo: 'nuxt/nuxt',+ branch: 'main', // default: 'main'+ contentPath: 'docs', // default: 'docs'+ outputPath: 'nuxt', // default: same as id+ additionalSyncs: [+ { repo: 'nuxt/nuxt.com', contentPath: 'content' },+ ],+ },+ { id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' },+ { id: 'ofetch', repo: 'unjs/ofetch', readmeOnly: true },+ ],+ youtube: [+ {+ id: 'alex-lichter',+ label: 'Alexander Lichter',+ channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA',+ maxVideos: 50, // default: 50+ },+ ],+ },+})+```++#### Package API++```typescript+// Configuration definition+export function defineConfig(config: SavoirConfigInput): SavoirConfigInput++// Loading (with c12)+export async function loadSavoirConfig(options?: { cwd?: string }): Promise<LoadedConfig>+export function setConfigCwd(cwd: string): void++// Accessors (async - always reload from disk)+export async function getSources(): Promise<Source[]>+export async function getGitHubSources(): Promise<GitHubSource[]>+export async function getYouTubeSources(): Promise<YouTubeSource[]>+export async function getSourceById(id: string): Promise<Source | undefined>+export async function getSourcesByType<T>(type: string): Promise<T[]>+```++### 2. SDK (`@savoir/sdk`)The SDK provides AI SDK-compatible tools that communicate with the Savoir API.@@ -84,245 +187,145 @@ packages/sdk/#### API Design```typescript-// packages/sdk/src/index.tsimport { tool } from 'ai'import { z } from 'zod'export interface SavoirOptions {apiKey: stringapiUrl: string- /** Optional: reuse sandbox across calls */- chatId?: string+ chatId?: string // Optional: reuse sandbox across calls}export interface SavoirClient {- /** Search and read files in one call (recommended) */- searchAndRead: ReturnType<typeof tool>- /** Search for files matching query */- search: ReturnType<typeof tool>- /** Read specific files by path */- read: ReturnType<typeof tool>+ searchAndRead: ReturnType<typeof tool> // Search and read files (recommended)+ search: ReturnType<typeof tool> // Search for files+ read: ReturnType<typeof tool> // Read specific files}-export function createSavoir(options: SavoirOptions): SavoirClient {- const client = createApiClient(options)-- return {- searchAndRead: tool({- description: 'Search documentation and read matching files. Use ONE simple keyword.',- inputSchema: z.object({- query: z.string().describe('ONE keyword to search'),- limit: z.number().optional().describe('Max files (default 5)'),- }),- execute: async ({ query, limit }) => {- return client.searchAndRead(query, limit)- },- }),-- search: tool({- description: 'Search for file paths. Use searchAndRead for most cases.',- inputSchema: z.object({- query: z.string(),- limit: z.number().optional(),- }),- execute: async ({ query, limit }) => {- return client.search(query, limit)- },- }),-- read: tool({- description: 'Read specific files by path.',- inputSchema: z.object({- paths: z.array(z.string()),- }),- execute: async ({ paths }) => {- return client.read(paths)- },- }),- }-}+export function createSavoir(options: SavoirOptions): SavoirClient```-### 2. API (`apps/api`)+### 3. API (`apps/api`)A Nitro server providing REST endpoints for sandbox management and content synchronization.-#### Module Structure+#### Route Structure```-apps/api/-├── server/-│ ├── api/-│ │ ├── sandbox/-│ │ │ ├── create.post.ts-│ │ │ ├── [id]/-│ │ │ │ ├── search.post.ts-│ │ │ │ ├── read.post.ts-│ │ │ │ └── search-and-read.post.ts-│ │ └── sync/-│ │ ├── index.post.ts-│ │ └── [source].post.ts-│ ├── tasks/-│ │ ├── sync-content.ts # Scheduled content sync-│ │ └── cleanup-sandboxes.ts # Cleanup stale sandboxes-│ ├── middleware/-│ │ └── auth.ts # API key validation-│ └── utils/-│ ├── sandbox.ts # Sandbox manager-│ ├── content-sync.ts # Content synchronization-│ └── sources.ts # Source configuration-├── nitro.config.ts-└── package.json-```--### 3. Sandbox Manager--Manages the lifecycle of Vercel Sandboxes.--#### Lifecycle States--```-┌─────────┐ create ┌─────────┐-│ None │ ───────────────▶│ Running │-└─────────┘ └────┬────┘- │- ┌──────────────────┼──────────────────┐- │ │ │- ▼ ▼ ▼- ┌────────────┐ ┌───────────┐ ┌───────────┐- │ Timeout │ │ Stopped │ │ Failed │- │ Extended │ │ │ │ │- └────────────┘ └───────────┘ └───────────┘- │ │- │ │ recover- │ ▼- │ ┌───────────┐- └──────────▶│ Reused │- └───────────┘+apps/api/routes/api/+├── index.ts # GET /api - Health check+├── sources/+│ └── index.get.ts # GET /api/sources - List sources+└── sync/+ ├── index.post.ts # POST /api/sync - Sync all+ └── [source].post.ts # POST /api/sync/:source - Sync one```-#### Key Functions+#### Sync Architecture-```typescript-// apps/api/server/utils/sandbox.ts+Content synchronization uses [Vercel Workflow](https://github.com/vercel/workflow) for durable execution with automatic retries.-interface SandboxRecord {- sandboxId: string- createdAt: number- fileTree?: string-}+```+POST /api/sync+ │+ ▼+┌─────────────────────────────────────────────────────────────────────────┐+│ Vercel Workflow │+│ (Durable execution engine) │+│ │+│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │+│ │ Step 1 │ │ Step 2 │ │ Step N │ │ Final Step │ │+│ │ Sync source │─▶│ Sync source │─▶│ ... │─▶│ Push to │ │+│ │ "nuxt" │ │ "nitro" │ │ │ │ GitHub │ │+│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │+│ │+│ Features: │+│ - Automatic retries on failure │+│ - Progress tracking via `pnpm workflow:web` │+│ - Durable state (survives server restarts) │+└─────────────────────────────────────────────────────────────────────────┘+```-// Get or create sandbox for a chat session-export async function getSandbox(- chatId: string,- log: LogFn-): Promise<SandboxContext>--// Recover existing sandbox from KV-async function recoverSandbox(- record: SandboxRecord,- log: LogFn-): Promise<Sandbox | null>--// Create new sandbox with cloned repo-async function createNewSandbox(log: LogFn): Promise<Sandbox>--// Execute search with ranking-async function searchWithRanking(- sandbox: Sandbox,- query: string,- limit: number-): Promise<string[]>--// Batch read multiple files efficiently-async function batchReadFiles(- sandbox: Sandbox,- paths: string[]-): Promise<Array<{ path: string; content: string }>>-```--### 4. Content Sync--Synchronizes content from various sources to a GitHub snapshot repository.--#### Sync Flow--```-┌────────────────────────────────────────────────────────────────────────┐-│ Nitro Task: sync:content │-│ (runs on schedule or trigger) │-└─────────────────────────────────────┬──────────────────────────────────┘- │- ▼-┌────────────────────────────────────────────────────────────────────────┐-│ Load Source Config │-│ (from sources.ts or API) │-└─────────────────────────────────────┬──────────────────────────────────┘- │- ┌───────────────────────┼───────────────────────┐- │ │ │- ▼ ▼ ▼- ┌────────────┐ ┌────────────┐ ┌────────────┐- │ GitHub │ │ YouTube │ │ Other │- │ Fetcher │ │ Fetcher │ │ Fetchers │- └─────┬──────┘ └─────┬──────┘ └─────┬──────┘- │ │ │- ▼ ▼ ▼- ┌────────────┐ ┌────────────┐ ┌────────────┐- │ Normalize │ │ Normalize │ │ Normalize │- │ to Markdown│ │ to Markdown│ │ to Markdown│- └─────┬──────┘ └─────┬──────┘ └─────┬──────┘- │ │ │- └──────────────────────┼──────────────────────┘- │- ▼- ┌─────────────────┐- │ Aggregate & │- │ Structure │- └────────┬────────┘- │- ▼- ┌─────────────────┐- │ Push to GitHub │- │ Snapshot Repo │- └─────────────────┘-```--#### Source Types+### 4. Sandbox System (Planned)-```typescript-// apps/api/server/utils/sources.ts+The sandbox system uses [Vercel Sandbox Snapshots](https://vercel.com/docs/vercel-sandbox/managing#snapshotting) for instant startup without cloning.-interface BaseSource {- id: string- label: string- type: 'github' | 'youtube' | 'custom'-}+#### Architecture-interface GitHubSource extends BaseSource {- type: 'github'- repo: string // e.g., 'nuxt/nuxt'- branch: string // e.g., 'main'- contentPath: string // e.g., 'docs/content'- outputPath?: string // e.g., 'nuxt'- readmeOnly?: boolean // Only fetch README.md-}+```+┌─────────────────────────────────────────────────────────────────────────┐+│ Snapshot Task (Periodic) │+│ Runs every X hours/days │+│ │+│ 1. Create sandbox ──▶ 2. Clone snapshot repo ──▶ 3. Take snapshot │+│ │ │+│ ▼ │+│ ┌─────────────┐ │+│ │ Snapshot │ │+│ │ (7 days) │ │+│ └──────┬──────┘ │+└──────────────────────────────────────────────────────────┼──────────────┘+ │+ │ instant start+ ▼+┌─────────────────────────────────────────────────────────────────────────┐+│ API Sandbox Endpoints │+│ │+│ Request ──▶ Start from snapshot ──▶ Search/Read ──▶ Return results │+│ (instant) (grep/cat) │+│ │+│ Benefits: │+│ - No clone delay (content already in sandbox) │+│ - Consistent state across all requests │+│ - Automatic cleanup (sandbox stops after use) │+└─────────────────────────────────────────────────────────────────────────┘+```-interface YouTubeSource extends BaseSource {- type: 'youtube'- channelId: string- maxVideos?: number-}+#### Snapshot Lifecycle-interface CustomSource extends BaseSource {- type: 'custom'- fetchFn: () => Promise<ContentFile[]>-}+```+Sync Workflow Snapshot Task API Request+ │ │ │+ │ push to GitHub │ │+ ▼ │ │+┌──────────┐ │ │+│ Snapshot │ ◀──── triggers ────┤ │+│ Repo │ │ │+└──────────┘ ▼ │+ ┌────────────┐ │+ │ Create │ │+ │ Sandbox │ │+ └─────┬──────┘ │+ │ clone repo │+ ▼ │+ ┌────────────┐ │+ │ Take │ │+ │ Snapshot │ │+ └─────┬──────┘ │+ │ │+ ▼ ▼+ ┌────────────┐ ┌────────────┐+ │ Snapshot │ ─────────▶ │ Instant │+ │ Ready │ start │ Sandbox │+ └────────────┘ └────────────┘+```++#### Key Functions-type Source = GitHubSource | YouTubeSource | CustomSource+```typescript+// Snapshot management+export async function createContentSnapshot(): Promise<Snapshot>+export async function getLatestSnapshot(): Promise<Snapshot | null>++// Sandbox operations (internal to API)+async function startFromSnapshot(snapshotId: string): Promise<Sandbox>+async function searchInSandbox(sandbox: Sandbox, query: string, limit: number): Promise<string[]>+async function readFromSandbox(sandbox: Sandbox, paths: string[]): Promise<FileContent[]>```-## API Endpoints Specification+> **Note:** Snapshots expire after 7 days. The snapshot task should run regularly to ensure a fresh snapshot is always available.++## API Endpoints### Authentication@@ -332,84 +335,119 @@ All endpoints require an API key in the `Authorization` header:Authorization: Bearer <api-key>```-### Sandbox Endpoints+### Source Endpoints-#### POST /api/sandbox/create+#### GET /api/sources-Creates or recovers a sandbox for a chat session.+Lists all configured content sources.-**Request:**+**Response:**+```json+{+ "total": 25,+ "github": {+ "count": 23,+ "sources": [+ { "id": "nuxt", "label": "Nuxt", "repo": "nuxt/nuxt" }+ ]+ },+ "youtube": {+ "count": 2,+ "sources": [+ { "id": "alex-lichter", "label": "Alexander Lichter" }+ ]+ }+}+```++### Sync Endpoints++#### POST /api/sync++Sync all sources.++**Body (optional):**```json{- "chatId": "unique-chat-identifier"+ "reset": false,+ "push": true}```**Response:**```json{- "sandboxId": "sbx_abc123",- "fileTree": "docs/\n├── nuxt/ (45 files)\n├── nitro/ (23 files)..."+ "status": "started",+ "message": "Sync workflow started. Use `pnpm workflow:web` to monitor.",+ "options": { "reset": false, "push": true }}```-#### POST /api/sandbox/:id/search-and-read+#### POST /api/sync/:source-Combined search and read operation.+Sync a specific source.-**Request:**+**Params:**+- `source`: Source ID (e.g., `nuxt`, `nitro`)++**Body (optional):**```json{- "query": "useAsyncData",- "limit": 5+ "reset": false,+ "push": true}```**Response:**```json{- "query": "useAsyncData",- "files": [- {- "path": "packages/content/docs/nuxt/composables/use-async-data.md",- "content": "# useAsyncData\n\n..."- }- ]+ "status": "started",+ "message": "Sync workflow started for source \"nuxt\". Use `pnpm workflow:web` to monitor.",+ "source": "nuxt",+ "options": { "reset": false, "push": true, "sourceFilter": "nuxt" }}```-#### POST /api/sandbox/:id/search+### Sandbox Endpoints (Planned)-Search for files matching query.+The API manages sandbox lifecycle internally. Sandboxes start instantly from the pre-built snapshot.++#### POST /api/sandbox/search-and-read++Combined search and read operation (recommended).**Request:**```json{- "query": "middleware",- "limit": 10+ "query": "useAsyncData",+ "limit": 5,+ "sessionId": "optional-session-id"}```**Response:**```json{+ "query": "useAsyncData","files": [- "packages/content/docs/nuxt/guide/middleware.md",- "packages/content/docs/nitro/middleware.md"+ {+ "path": "docs/nuxt/composables/use-async-data.md",+ "content": "# useAsyncData\n\n..."+ }]}```-#### POST /api/sandbox/:id/read+#### POST /api/sandbox/search-Read specific files.+Search for files matching query.**Request:**```json{- "paths": [- "packages/content/docs/nuxt/guide/middleware.md"- ]+ "query": "middleware",+ "limit": 10,+ "sessionId": "optional-session-id"}```@@ -417,57 +455,55 @@ Read specific files.```json{"files": [- {- "path": "packages/content/docs/nuxt/guide/middleware.md",- "content": "# Middleware\n\n..."- }+ "docs/nuxt/guide/middleware.md",+ "docs/nitro/middleware.md"]}```-### Sync Endpoints+#### POST /api/sandbox/read-#### POST /api/sync--Triggers full content synchronization.+Read specific files.-**Response:**+**Request:**```json{- "status": "started",- "sources": 25+ "paths": [+ "docs/nuxt/guide/middleware.md"+ ],+ "sessionId": "optional-session-id"}```-#### GET /api/sources--Lists configured sources.-**Response:**```json{- "sources": [+ "files": [{- "id": "nuxt",- "label": "Nuxt",- "type": "github",- "repo": "nuxt/nuxt"+ "path": "docs/nuxt/guide/middleware.md",+ "content": "# Middleware\n\n..."}]}```+> **Note:** The `sessionId` is optional and generic - can be from any client (Discord bot, Slack bot, GitHub bot, web chat, etc.). If provided, the API may reuse an existing sandbox for that session.+## Storage Strategy### KV Store-Used for sandbox session management:+Used for storing the current snapshot ID and optionally sandbox sessions:```typescript-// Key format-`sandbox:chat:${chatId}` -> SandboxRecord+// Current snapshot (updated by snapshot task)+`snapshot:current` -> { snapshotId: string, createdAt: number }-// TTL: 30 minutes (sandbox timeout + buffer)+// Optional: Active sandbox sessions (if reusing sandboxes across requests)+`sandbox:session:${sessionId}` -> { sandboxId: string, createdAt: number }++// sessionId is generic - can be from any client (Discord bot, Slack bot, GitHub bot, etc.)+// TTL: 30 minutes (sandbox timeout)```### Snapshot Repository@@ -475,7 +511,7 @@ Used for sandbox session management:Structure:```-content-snapshot-repo/+{GITHUB_SNAPSHOT_REPO}/├── docs/│ ├── nuxt/│ │ ├── getting-started/@@ -485,11 +521,19 @@ content-snapshot-repo/│ ├── nitro/│ └── ...└── youtube/- ├── alex-lichter/- │ └── nuxt-4-overview-TAoTh4DqH6A.md- └── ...+ └── alex-lichter/+ └── nuxt-4-overview-TAoTh4DqH6A.md```+## Environment Variables++| Variable | Required | Description |+|----------|----------|-------------|+| `GITHUB_TOKEN` | Yes | GitHub token for API access |+| `GITHUB_SNAPSHOT_REPO` | Yes | Snapshot repository (owner/repo) |+| `GITHUB_SNAPSHOT_BRANCH` | No | Branch to push to (default: main) |+| `SAVOIR_SECRET_KEY` | No | API key for authentication |+## Error Handling### SDK Errors@@ -519,11 +563,17 @@ export class SavoirError extends Error {## Performance Considerations-### Sandbox Reuse+### Snapshot-based Startup++- Sandboxes start instantly from pre-built snapshot (no clone delay)+- Snapshot contains all synced content ready to search+- Snapshot is refreshed periodically by the snapshot task++### Session Reuse (Optional)-- Sandboxes are cached in KV by `chatId`-- Timeout extended when < 3 minutes remaining-- File tree cached to avoid regeneration+- Sandboxes can be reused within a session via `sessionId`+- Avoids startup cost for multiple requests in same conversation+- Automatic cleanup when session ends or timeout### Batch Operations@@ -540,7 +590,7 @@ export class SavoirError extends Error {### API Key Validation```typescript-// apps/api/server/middleware/auth.ts+// apps/api/middleware/auth.tsexport default defineEventHandler((event) => {const auth = getHeader(event, 'Authorization')const apiKey = auth?.replace('Bearer ', '')diff --git a/docs/SOURCES.md b/docs/SOURCES.mdindex 1e31459..b494e0a 100644--- a/docs/SOURCES.md+++ b/docs/SOURCES.md@@ -1,78 +1,29 @@# Content Sources-Savoir aggregates documentation from multiple sources into a unified, searchable knowledge base. This document describes how to configure, add, and manage content sources.+Savoir aggregates documentation from multiple sources into a unified, searchable knowledge base.-## Overview+## Configuration-Sources are external data repositories that Savoir fetches, normalizes to Markdown, and stores in a snapshot repository. The AI agent then searches and reads from this snapshot via a Vercel Sandbox.--```-┌────────────────────────────────────────────────────────────────────┐-│ Source Types │-│ │-│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │-│ │ GitHub │ │ YouTube │ │ MCP │ │ Custom (future) │ │-│ │ Docs │ │ Transcripts│ │ Servers │ │ Blogs, APIs │ │-│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────────┬─────────┘ │-│ │ │ │ │ │-│ └─────────────┴──────┬──────┴──────────────────┘ │-│ │ │-│ ▼ │-│ ┌─────────────────┐ │-│ │ Normalizer │ │-│ │ (to Markdown) │ │-│ └────────┬────────┘ │-│ │ │-│ ▼ │-│ ┌─────────────────┐ │-│ │ Snapshot Repo │ │-│ │ (GitHub) │ │-│ └─────────────────┘ │-└────────────────────────────────────────────────────────────────────┘-```--## Source Configuration--Sources are defined in `apps/api/server/utils/sources.ts`:+Sources are defined in `savoir.config.ts` at the project root:```typescript-// apps/api/server/utils/sources.ts-import type { Source } from './types'--export const SOURCES: Source[] = [- // GitHub documentation- {- id: 'nuxt',- label: 'Nuxt',- type: 'github',- repo: 'nuxt/nuxt',- branch: 'main',- contentPath: 'docs/content',- outputPath: 'nuxt',- },-- // README-only sources- {- id: 'unhead',- label: 'Unhead',- type: 'github',- repo: 'unjs/unhead',- branch: 'main',- contentPath: '.',- readmeOnly: true,- },-- // YouTube channels- {- id: 'alex-lichter',- label: 'Alex Lichter',- type: 'youtube',- channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA',- maxVideos: 100,+import { defineConfig } from '@savoir/config'++export default defineConfig({+ sources: {+ github: [+ { id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' },+ { id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' },+ ],+ youtube: [+ { id: 'alex-lichter', channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA' },+ ],},-]+})```+Supported formats: `.ts`, `.js`, `.json`, `.yaml`+## Source Types### GitHub Sources@@ -80,35 +31,18 @@ export const SOURCES: Source[] = [Fetches Markdown documentation from GitHub repositories.```typescript-interface GitHubSource {- type: 'github'-- /** Unique identifier */- id: string-- /** Display name */- label: string-- /** GitHub repository (owner/repo) */- repo: string-- /** Branch to fetch from */- branch: string-- /** Path to content directory */- contentPath: string-- /** Output directory in snapshot (defaults to id) */- outputPath?: string-- /** Only fetch README.md */- readmeOnly?: boolean-- /** Additional repos to merge into same output */+{+ id: string // Unique identifier+ label?: string // Display name (defaults to capitalized id)+ repo: string // GitHub repository (owner/repo)+ branch?: string // Branch to fetch from (default: 'main')+ contentPath?: string // Path to content directory (default: 'docs')+ outputPath?: string // Output directory in snapshot (default: id)+ readmeOnly?: boolean // Only fetch README.md (default: false)additionalSyncs?: Array<{repo: string- branch: string- contentPath: string+ branch?: string+ contentPath?: string}>}```@@ -117,43 +51,21 @@ interface GitHubSource {```typescript// Full documentation tree-{- id: 'nuxt',- label: 'Nuxt',- type: 'github',- repo: 'nuxt/nuxt',- branch: 'main',- contentPath: 'docs/content',- outputPath: 'nuxt',-}+{ id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' }++// Specific branch+{ id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' }// README only-{- id: 'ofetch',- label: 'ofetch',- type: 'github',- repo: 'unjs/ofetch',- branch: 'main',- contentPath: '.',- readmeOnly: true,- outputPath: 'ofetch',-}+{ id: 'ofetch', repo: 'unjs/ofetch', readmeOnly: true }-// Multiple repos merged into one output+// Multiple repos merged{- id: 'nuxt-ui',- label: 'Nuxt UI',- type: 'github',- repo: 'nuxt/ui',- branch: 'v3',- contentPath: 'docs/content',- outputPath: 'nuxt-ui',+ id: 'nuxt',+ repo: 'nuxt/nuxt',+ contentPath: 'docs',additionalSyncs: [- {- repo: 'nuxt/ui-pro',- branch: 'main',- contentPath: 'docs/content',- }+ { repo: 'nuxt/nuxt.com', contentPath: 'content' },],}```@@ -163,315 +75,87 @@ interface GitHubSource {Fetches video transcripts from YouTube channels.```typescript-interface YouTubeSource {- type: 'youtube'-- /** Unique identifier */- id: string-- /** Display name */- label: string-- /** YouTube channel ID */- channelId: string-- /** Maximum videos to fetch (default: 50) */- maxVideos?: number+{+ id: string // Unique identifier+ label?: string // Display name+ channelId: string // YouTube channel ID+ handle?: string // YouTube handle (e.g., '@TheAlexLichter')+ maxVideos?: number // Maximum videos to fetch (default: 50)}```-**Output format:**--```markdown-----title: "Nuxt 4: An Overview"-videoId: TAoTh4DqH6A-channelId: UCqFPgMzGbLjd-MX-h3Z5aQA-publishedAt: 2024-01-15T10:00:00Z------# Nuxt 4: An Overview--[Transcript content here...]-```--### Custom Sources (Extensible)+### Custom SourcesFor sources that don't fit the standard patterns:-```typescript-interface CustomSource {- type: 'custom'-- /** Unique identifier */- id: string-- /** Display name */- label: string-- /** Custom fetch function */- fetchFn: () => Promise<ContentFile[]>-}--interface ContentFile {- path: string // Relative path in output- content: string // Markdown content-}-```--**Example:**-```typescript{- id: 'blog-posts',- label: 'Community Blog',- type: 'custom',- fetchFn: async () => {- const posts = await fetchBlogPosts()- return posts.map(post => ({- path: `blog/${post.slug}.md`,- content: `----title: ${post.title}-author: ${post.author}-date: ${post.date}------${post.content}`,- }))- },+ id: string+ label?: string+ fetchFn: () => Promise<Array<{ path: string; content: string }>>}```-## Adding a New Source+## Syncing-### Step 1: Define the Source--Add to `apps/api/server/utils/sources.ts`:--```typescript-export const SOURCES: Source[] = [- // ... existing sources-- {- id: 'my-project',- label: 'My Project',- type: 'github',- repo: 'my-org/my-project',- branch: 'main',- contentPath: 'docs',- },-]-```--### Step 2: Run Sync--Trigger synchronization:+### Via API```bash# Sync all sources-pnpm sync+curl -X POST http://localhost:3000/api/sync# Sync specific source-pnpm sync my-project-```--Or via API:+curl -X POST http://localhost:3000/api/sync/nuxt-```bash-curl -X POST https://api.savoir.example.com/api/sync/my-project \- -H "Authorization: Bearer $SAVOIR_API_KEY"-```--### Step 3: Verify--Check the snapshot repository for the new content:--```-content-snapshot-repo/-└── docs/- └── my-project/- ├── getting-started.md- └── api-reference.md-```--## Sync Process--### Nitro Tasks--Content synchronization is handled by Nitro scheduled tasks:--```typescript-// apps/api/server/tasks/sync-content.ts-export default defineTask({- meta: {- name: 'sync:content',- description: 'Synchronize content from all sources',- },- async run() {- for (const source of SOURCES) {- await syncSource(source)- }- await pushToSnapshot()- },-})-```--### Scheduling--Configure in `nitro.config.ts`:--```typescript-export default defineNitroConfig({- experimental: {- tasks: true,- },- scheduledTasks: {- // Sync every 6 hours- '0 */6 * * *': ['sync:content'],- },-})+# With options+curl -X POST http://localhost:3000/api/sync \+ -H "Content-Type: application/json" \+ -d '{"reset": true, "push": true}'```-### Manual Trigger--```bash-# Via CLI-pnpm sync+### Options-# Via API-curl -X POST https://api.savoir.example.com/api/sync \- -H "Authorization: Bearer $API_KEY"-```+| Option | Type | Default | Description |+|--------|------|---------|-------------|+| `reset` | boolean | false | Clear content before sync |+| `push` | boolean | true | Push to snapshot repo after sync |## Content Normalization-All content is normalized to Markdown with consistent structure:--### File Types+All content is normalized to Markdown:| Input | Output ||-------|--------|| `.md` | Preserved as-is || `.mdx` | Converted to `.md` |-| `.yml`/`.yaml` | Preserved (for frontmatter) |-| `.json` | Preserved (for configs) |+| `.yml`/`.yaml` | Preserved |+| `.json` | Preserved || Other | Ignored |### Excluded Files-Lock files and non-documentation are automatically excluded:--- `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`+- Lock files (`package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`, etc.)- `node_modules/`- Binary files-- Test files--### Directory Structure--Sources maintain their original hierarchy:--```-GitHub: nuxt/nuxt/docs/content/getting-started/installation.md- ↓-Snapshot: docs/nuxt/getting-started/installation.md-```## Snapshot Repository-The snapshot repository is a GitHub repo containing all aggregated content.--### Structure+The snapshot repository contains all aggregated content:```{GITHUB_SNAPSHOT_REPO}/├── docs/│ ├── nuxt/│ │ ├── getting-started/-│ │ │ ├── installation.md-│ │ │ └── configuration.md-│ │ ├── composables/-│ │ │ ├── use-async-data.md-│ │ │ └── use-fetch.md-│ │ └── ...+│ │ └── composables/│ ├── nitro/-│ ├── h3/│ └── ...└── youtube/- ├── alex-lichter/- │ ├── nuxt-4-overview-TAoTh4DqH6A.md- │ └── ...- └── learn-vue/+ └── alex-lichter/```-### Configuration--Set via environment variable:+Configure via environment variable:```bashGITHUB_SNAPSHOT_REPO=my-org/content-snapshot```--### Updates--The sync process:--1. Fetches content from all sources-2. Transforms and normalizes-3. Commits changes to snapshot repo-4. New sandboxes clone the updated snapshot--## Best Practices--### Source Selection--1. **Prioritize official documentation** over third-party content-2. **Include README-only sources** for smaller libraries-3. **Add YouTube sources** for tutorial content and explanations--### Performance--1. **Batch updates** rather than syncing after each change-2. **Use `readmeOnly`** for packages without full docs-3. **Limit YouTube videos** to prevent excessive API usage--### Content Quality--1. **Preserve frontmatter** for metadata-2. **Maintain original structure** for discoverability-3. **Include version info** in source paths when relevant--## Troubleshooting--### Sync Failures--Check logs for specific errors:--```bash-pnpm sync my-source --verbose-```--Common issues:-- **Rate limits**: Wait and retry, or use authenticated requests-- **Missing content path**: Verify `contentPath` exists in repo-- **Branch not found**: Check `branch` matches repository--### Missing Content--1. Verify source configuration-2. Check if files match allowed extensions-3. Ensure content isn't in excluded files list--### Stale Content--Force a fresh sync:--```bash-pnpm sync --reset-```--Or via API:--```bash-curl -X POST https://api.savoir.example.com/api/sync?reset=true \- -H "Authorization: Bearer $API_KEY"-```