Skip to content

Open source file-system and knowledge based agent template. Build AI agents that stay up to date with your knowledge base

update docs

Hugo Richardf39b6a2abe68
diff --git a/README.md b/README.md
index bb46a5b..4840b52 100644
--- a/README.md
+++ b/README.md
@@ -41,6 +41,7 @@ Savoir provides the infrastructure to create file-based AI agents (chatbots, Dis
| Package | Description |
|---------|-------------|
| [`@savoir/sdk`](./packages/sdk) | AI SDK compatible tools for agents |
+| [`@savoir/config`](./packages/config) | Configuration management with multi-format support |
| [`apps/api`](./apps/api) | Self-hostable sandbox management API |
## Quick Start
@@ -100,9 +101,33 @@ GITHUB_SNAPSHOT_REPO=your-org/your-content-repo
SAVOIR_SECRET_KEY=your-secret-key
```
+## Configuration
+
+Sources are configured in `savoir.config.ts` at the project root:
+
+```typescript
+import { defineConfig } from '@savoir/config'
+
+export default defineConfig({
+ sources: {
+ github: [
+ { id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' },
+ { id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' },
+ ],
+ youtube: [
+ { id: 'alex-lichter', channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA' },
+ ],
+ },
+})
+```
+
+Supported formats: `.ts`, `.js`, `.json`, `.yaml`
+
+See [SOURCES.md](./docs/SOURCES.md) for detailed source configuration options.
+
## How It Works
-1. **Content Aggregation**: Sources (GitHub docs, YouTube transcripts, etc.) are synced to a snapshot repository via Nitro tasks
+1. **Content Aggregation**: Sources (GitHub docs, YouTube transcripts, etc.) are synced to a snapshot repository via Vercel Workflow
2. **Sandbox Creation**: When an agent needs to search, the API creates/recovers a Vercel Sandbox with the snapshot repo cloned
3. **File-based Search**: The SDK tools execute grep/find commands in the sandbox to search and read content
4. **AI Integration**: Tools are compatible with the Vercel AI SDK for seamless integration with any LLM
diff --git a/apps/api/README.md b/apps/api/README.md
index 98f9918..fe8e5bd 100644
--- a/apps/api/README.md
+++ b/apps/api/README.md
@@ -1,6 +1,6 @@
# Savoir API
-Self-hostable API server for managing content synchronization and sandbox operations.
+Self-hostable API server for managing content synchronization.
## Setup
@@ -38,42 +38,19 @@ pnpm preview
## Sync Commands
-Content synchronization uses [Vercel Workflow DevKit](https://useworkflow.dev/) for durable execution with automatic retries.
+Content synchronization uses [Vercel Workflow](https://github.com/vercel/workflow) for durable execution with automatic retries.
```bash
# Start the dev server first
pnpm dev
-# In another terminal, trigger async workflow sync
-pnpm sync:docs
+# In another terminal, trigger sync
+curl -X POST http://localhost:3000/api/sync
# Monitor workflow execution
pnpm workflow:web
```
-**Via curl with options:**
-
-```bash
-# Async workflow (recommended)
-curl -X POST http://localhost:3000/api/sync/workflow
-
-# With options
-curl -X POST http://localhost:3000/api/sync/workflow \
- -H "Content-Type: application/json" \
- -d '{"source": "nuxt-ui", "push": false}'
-
-# Sync specific source (legacy, synchronous)
-curl -X POST http://localhost:3000/api/sync/nuxt-ui
-```
-
-**Options:**
-
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `source` | string | - | Sync only this source ID |
-| `reset` | boolean | false | Clear content before sync |
-| `push` | boolean | true | Push to snapshot repo after sync |
-
## API Endpoints
### GET /api
@@ -99,16 +76,15 @@ List all configured content sources.
}
```
-### POST /api/sync/workflow
+### POST /api/sync
-Trigger async sync workflow (recommended). Uses Workflow DevKit for automatic retries.
+Sync all sources.
**Body (optional):**
```json
{
"reset": false,
- "push": true,
- "source": "nuxt-ui"
+ "push": true
}
```
@@ -121,37 +97,6 @@ Trigger async sync workflow (recommended). Uses Workflow DevKit for automatic re
}
```
-### POST /api/sync
-
-Trigger full content synchronization (synchronous).
-
-**Body (optional):**
-```json
-{
- "reset": false,
- "push": true
-}
-```
-
-**Response:**
-```json
-{
- "success": true,
- "summary": {
- "total": 25,
- "success": 25,
- "failed": 0,
- "files": 1234
- },
- "push": {
- "success": true,
- "commitSha": "abc123...",
- "filesChanged": 1234
- },
- "results": [...]
-}
-```
-
### POST /api/sync/:source
Sync a specific source.
@@ -167,18 +112,24 @@ Sync a specific source.
}
```
-## Adding Sources
+## Configuration
-Edit `utils/sources/sources.ts` to add new sources:
+Sources are configured in `savoir.config.ts` at the project root:
```typescript
-{
- id: 'my-project',
- label: 'My Project',
- type: 'github',
- repo: 'org/repo',
- branch: 'main',
- contentPath: 'docs',
- outputPath: 'my-project',
-}
+import { defineConfig } from '@savoir/config'
+
+export default defineConfig({
+ sources: {
+ github: [
+ { id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' },
+ { id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' },
+ ],
+ youtube: [
+ { id: 'alex-lichter', channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA' },
+ ],
+ },
+})
```
+
+See [SOURCES.md](/docs/SOURCES.md) for details on source configuration.
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 905dd0a..eb55a9e 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -4,10 +4,11 @@ This document describes the technical architecture of Savoir, a platform for bui
## System Overview
-Savoir consists of two main components:
+Savoir consists of three main components:
1. **Savoir API** (`apps/api`): A Nitro-based server that manages Vercel Sandboxes and content synchronization
2. **Savoir SDK** (`packages/sdk`): A client library providing AI SDK-compatible tools
+3. **Savoir Config** (`packages/config`): Configuration management with multi-format support
```
┌─────────────────────────────────────────────────────────────────────────┐
@@ -20,35 +21,35 @@ Savoir consists of two main components:
│ │ │ │
│ │ const savoir = createSavoir({ apiKey, apiUrl }) │ │
│ │ const { text } = await generateText({ │ │
-│ │ model: 'google/gemini-3-flash', │ │
+│ │ model: 'google/gemini-3-flash', │ │
│ │ tools: { ...savoir.tools } │ │
│ │ }) │ │
│ └────────────────────────────────────────────────────────────────────┘ │
-└─────────────────────────────────────┬───────────────────────────────────┘
- │
- │ HTTPS (API Key auth)
- ▼
+└─────────────────────────────────────────────────────────────────────────┘
+ │
+ │ HTTPS (API Key auth)
+ ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Savoir API │
│ (Nitro Server) │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────┐ │
-│ │ /api/sandbox/* │ │ /api/sync/* │ │ Nitro Tasks │ │
+│ │ /api/sandbox/* │ │ /api/sync/* │ │ /api/sources/* │ │
│ │ │ │ │ │ │ │
-│ │ - create │ │ - POST /sync │ │ - sync:content (cron) │ │
-│ │ - search │ │ - GET /sources │ │ - cleanup:sandboxes │ │
+│ │ - create │ │ - POST /sync │ │ - GET /sources │ │
+│ │ - search │ │ - POST /sync/:s │ │ │ │
│ │ - read │ │ │ │ │ │
│ │ - searchAndRead │ │ │ │ │ │
+│ └────────┬────────┘ └────────┬────────┘ └────────────┬────────────┘ │
+│ │ │ │ │
+│ ▼ ▼ ▼ │
+│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────┐ │
+│ │ Sandbox Manager │ │ Vercel Workflow │ │ @savoir/config │ │
+│ │ │ │ │ │ │ │
+│ │ - Lifecycle │ │ - Durable exec │ │ - Load savoir.config.ts │ │
+│ │ - KV caching │ │ - Auto retries │ │ - Validation │ │
+│ │ - Command exec │ │ - Sync sources │ │ - Normalization │ │
│ └────────┬────────┘ └────────┬────────┘ └─────────────────────────┘ │
-│ │ │ │
-│ ▼ ▼ │
-│ ┌─────────────────┐ ┌─────────────────┐ │
-│ │ Sandbox Manager │ │ Content Syncer │ │
-│ │ │ │ │ │
-│ │ - Lifecycle │ │ - Fetch sources │ │
-│ │ - KV caching │ │ - Transform │ │
-│ │ - Command exec │ │ - Push to repo │ │
-│ └────────┬────────┘ └────────┬────────┘ │
└───────────┼────────────────────┼────────────────────────────────────────┘
│ │
▼ ▼
@@ -59,9 +60,111 @@ Savoir consists of two main components:
└────────────┘ └────────────┘
```
+## Project Structure
+
+```
+savoir/
+├── savoir.config.ts # Source configuration (root level!)
+├── apps/
+│ └── api/ # Nitro API server
+│ ├── routes/
+│ │ └── api/
+│ │ ├── index.ts # GET /api - Health check
+│ │ ├── sources/ # Source endpoints
+│ │ │ └── index.get.ts # GET /api/sources
+│ │ ├── sync/ # Sync endpoints
+│ │ │ ├── index.post.ts # POST /api/sync
+│ │ │ └── [source].post.ts # POST /api/sync/:source
+│ │ └── sandbox/ # Sandbox endpoints (planned)
+│ │ ├── search.post.ts
+│ │ ├── read.post.ts
+│ │ └── search-and-read.post.ts
+│ ├── workflows/
+│ │ └── sync-docs/ # Vercel Workflow for syncing
+│ └── plugins/
+│ └── config.ts # Config loader plugin
+├── packages/
+│ ├── config/ # @savoir/config
+│ │ └── src/
+│ │ ├── index.ts # Public exports
+│ │ ├── types.ts # Source types
+│ │ ├── define.ts # defineConfig()
+│ │ ├── loader.ts # c12-based loading
+│ │ ├── validation.ts # Zod schemas
+│ │ └── normalize.ts # Default values
+│ ├── sdk/ # @savoir/sdk
+│ └── logger/ # @savoir/logger
+└── docs/ # Documentation
+```
+
## Component Details
-### 1. SDK (`@savoir/sdk`)
+### 1. Configuration (`@savoir/config`)
+
+Sources are configured in `savoir.config.ts` at the project root. This file is visible, easy to edit, and supports multiple formats via [c12](https://github.com/unjs/c12).
+
+#### Supported Formats
+
+- `savoir.config.ts` (recommended)
+- `savoir.config.js`
+- `savoir.config.json`
+- `savoir.config.yaml`
+- `.savoirrc` (JSON)
+
+#### Configuration Schema
+
+```typescript
+// savoir.config.ts
+import { defineConfig } from '@savoir/config'
+
+export default defineConfig({
+ sources: {
+ github: [
+ {
+ id: 'nuxt',
+ label: 'Nuxt',
+ repo: 'nuxt/nuxt',
+ branch: 'main', // default: 'main'
+ contentPath: 'docs', // default: 'docs'
+ outputPath: 'nuxt', // default: same as id
+ additionalSyncs: [
+ { repo: 'nuxt/nuxt.com', contentPath: 'content' },
+ ],
+ },
+ { id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' },
+ { id: 'ofetch', repo: 'unjs/ofetch', readmeOnly: true },
+ ],
+ youtube: [
+ {
+ id: 'alex-lichter',
+ label: 'Alexander Lichter',
+ channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA',
+ maxVideos: 50, // default: 50
+ },
+ ],
+ },
+})
+```
+
+#### Package API
+
+```typescript
+// Configuration definition
+export function defineConfig(config: SavoirConfigInput): SavoirConfigInput
+
+// Loading (with c12)
+export async function loadSavoirConfig(options?: { cwd?: string }): Promise<LoadedConfig>
+export function setConfigCwd(cwd: string): void
+
+// Accessors (async - always reload from disk)
+export async function getSources(): Promise<Source[]>
+export async function getGitHubSources(): Promise<GitHubSource[]>
+export async function getYouTubeSources(): Promise<YouTubeSource[]>
+export async function getSourceById(id: string): Promise<Source | undefined>
+export async function getSourcesByType<T>(type: string): Promise<T[]>
+```
+
+### 2. SDK (`@savoir/sdk`)
The SDK provides AI SDK-compatible tools that communicate with the Savoir API.
@@ -84,245 +187,145 @@ packages/sdk/
#### API Design
```typescript
-// packages/sdk/src/index.ts
import { tool } from 'ai'
import { z } from 'zod'
export interface SavoirOptions {
apiKey: string
apiUrl: string
- /** Optional: reuse sandbox across calls */
- chatId?: string
+ chatId?: string // Optional: reuse sandbox across calls
}
export interface SavoirClient {
- /** Search and read files in one call (recommended) */
- searchAndRead: ReturnType<typeof tool>
- /** Search for files matching query */
- search: ReturnType<typeof tool>
- /** Read specific files by path */
- read: ReturnType<typeof tool>
+ searchAndRead: ReturnType<typeof tool> // Search and read files (recommended)
+ search: ReturnType<typeof tool> // Search for files
+ read: ReturnType<typeof tool> // Read specific files
}
-export function createSavoir(options: SavoirOptions): SavoirClient {
- const client = createApiClient(options)
-
- return {
- searchAndRead: tool({
- description: 'Search documentation and read matching files. Use ONE simple keyword.',
- inputSchema: z.object({
- query: z.string().describe('ONE keyword to search'),
- limit: z.number().optional().describe('Max files (default 5)'),
- }),
- execute: async ({ query, limit }) => {
- return client.searchAndRead(query, limit)
- },
- }),
-
- search: tool({
- description: 'Search for file paths. Use searchAndRead for most cases.',
- inputSchema: z.object({
- query: z.string(),
- limit: z.number().optional(),
- }),
- execute: async ({ query, limit }) => {
- return client.search(query, limit)
- },
- }),
-
- read: tool({
- description: 'Read specific files by path.',
- inputSchema: z.object({
- paths: z.array(z.string()),
- }),
- execute: async ({ paths }) => {
- return client.read(paths)
- },
- }),
- }
-}
+export function createSavoir(options: SavoirOptions): SavoirClient
```
-### 2. API (`apps/api`)
+### 3. API (`apps/api`)
A Nitro server providing REST endpoints for sandbox management and content synchronization.
-#### Module Structure
+#### Route Structure
```
-apps/api/
-├── server/
-│ ├── api/
-│ │ ├── sandbox/
-│ │ │ ├── create.post.ts
-│ │ │ ├── [id]/
-│ │ │ │ ├── search.post.ts
-│ │ │ │ ├── read.post.ts
-│ │ │ │ └── search-and-read.post.ts
-│ │ └── sync/
-│ │ ├── index.post.ts
-│ │ └── [source].post.ts
-│ ├── tasks/
-│ │ ├── sync-content.ts # Scheduled content sync
-│ │ └── cleanup-sandboxes.ts # Cleanup stale sandboxes
-│ ├── middleware/
-│ │ └── auth.ts # API key validation
-│ └── utils/
-│ ├── sandbox.ts # Sandbox manager
-│ ├── content-sync.ts # Content synchronization
-│ └── sources.ts # Source configuration
-├── nitro.config.ts
-└── package.json
-```
-
-### 3. Sandbox Manager
-
-Manages the lifecycle of Vercel Sandboxes.
-
-#### Lifecycle States
-
-```
-┌─────────┐ create ┌─────────┐
-│ None │ ───────────────▶│ Running │
-└─────────┘ └────┬────┘
- │
- ┌──────────────────┼──────────────────┐
- │ │ │
- ▼ ▼ ▼
- ┌────────────┐ ┌───────────┐ ┌───────────┐
- │ Timeout │ │ Stopped │ │ Failed │
- │ Extended │ │ │ │ │
- └────────────┘ └───────────┘ └───────────┘
- │ │
- │ │ recover
- │ ▼
- │ ┌───────────┐
- └──────────▶│ Reused │
- └───────────┘
+apps/api/routes/api/
+├── index.ts # GET /api - Health check
+├── sources/
+│ └── index.get.ts # GET /api/sources - List sources
+└── sync/
+ ├── index.post.ts # POST /api/sync - Sync all
+ └── [source].post.ts # POST /api/sync/:source - Sync one
```
-#### Key Functions
+#### Sync Architecture
-```typescript
-// apps/api/server/utils/sandbox.ts
+Content synchronization uses [Vercel Workflow](https://github.com/vercel/workflow) for durable execution with automatic retries.
-interface SandboxRecord {
- sandboxId: string
- createdAt: number
- fileTree?: string
-}
+```
+POST /api/sync
+ │
+ ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ Vercel Workflow │
+│ (Durable execution engine) │
+│ │
+│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
+│ │ Step 1 │ │ Step 2 │ │ Step N │ │ Final Step │ │
+│ │ Sync source │─▶│ Sync source │─▶│ ... │─▶│ Push to │ │
+│ │ "nuxt" │ │ "nitro" │ │ │ │ GitHub │ │
+│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
+│ │
+│ Features: │
+│ - Automatic retries on failure │
+│ - Progress tracking via `pnpm workflow:web` │
+│ - Durable state (survives server restarts) │
+└─────────────────────────────────────────────────────────────────────────┘
+```
-// Get or create sandbox for a chat session
-export async function getSandbox(
- chatId: string,
- log: LogFn
-): Promise<SandboxContext>
-
-// Recover existing sandbox from KV
-async function recoverSandbox(
- record: SandboxRecord,
- log: LogFn
-): Promise<Sandbox | null>
-
-// Create new sandbox with cloned repo
-async function createNewSandbox(log: LogFn): Promise<Sandbox>
-
-// Execute search with ranking
-async function searchWithRanking(
- sandbox: Sandbox,
- query: string,
- limit: number
-): Promise<string[]>
-
-// Batch read multiple files efficiently
-async function batchReadFiles(
- sandbox: Sandbox,
- paths: string[]
-): Promise<Array<{ path: string; content: string }>>
-```
-
-### 4. Content Sync
-
-Synchronizes content from various sources to a GitHub snapshot repository.
-
-#### Sync Flow
-
-```
-┌────────────────────────────────────────────────────────────────────────┐
-│ Nitro Task: sync:content │
-│ (runs on schedule or trigger) │
-└─────────────────────────────────────┬──────────────────────────────────┘
- │
- ▼
-┌────────────────────────────────────────────────────────────────────────┐
-│ Load Source Config │
-│ (from sources.ts or API) │
-└─────────────────────────────────────┬──────────────────────────────────┘
- │
- ┌───────────────────────┼───────────────────────┐
- │ │ │
- ▼ ▼ ▼
- ┌────────────┐ ┌────────────┐ ┌────────────┐
- │ GitHub │ │ YouTube │ │ Other │
- │ Fetcher │ │ Fetcher │ │ Fetchers │
- └─────┬──────┘ └─────┬──────┘ └─────┬──────┘
- │ │ │
- ▼ ▼ ▼
- ┌────────────┐ ┌────────────┐ ┌────────────┐
- │ Normalize │ │ Normalize │ │ Normalize │
- │ to Markdown│ │ to Markdown│ │ to Markdown│
- └─────┬──────┘ └─────┬──────┘ └─────┬──────┘
- │ │ │
- └──────────────────────┼──────────────────────┘
- │
- ▼
- ┌─────────────────┐
- │ Aggregate & │
- │ Structure │
- └────────┬────────┘
- │
- ▼
- ┌─────────────────┐
- │ Push to GitHub │
- │ Snapshot Repo │
- └─────────────────┘
-```
-
-#### Source Types
+### 4. Sandbox System (Planned)
-```typescript
-// apps/api/server/utils/sources.ts
+The sandbox system uses [Vercel Sandbox Snapshots](https://vercel.com/docs/vercel-sandbox/managing#snapshotting) for instant startup without cloning.
-interface BaseSource {
- id: string
- label: string
- type: 'github' | 'youtube' | 'custom'
-}
+#### Architecture
-interface GitHubSource extends BaseSource {
- type: 'github'
- repo: string // e.g., 'nuxt/nuxt'
- branch: string // e.g., 'main'
- contentPath: string // e.g., 'docs/content'
- outputPath?: string // e.g., 'nuxt'
- readmeOnly?: boolean // Only fetch README.md
-}
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│ Snapshot Task (Periodic) │
+│ Runs every X hours/days │
+│ │
+│ 1. Create sandbox ──▶ 2. Clone snapshot repo ──▶ 3. Take snapshot │
+│ │ │
+│ ▼ │
+│ ┌─────────────┐ │
+│ │ Snapshot │ │
+│ │ (7 days) │ │
+│ └──────┬──────┘ │
+└──────────────────────────────────────────────────────────┼──────────────┘
+ │
+ │ instant start
+ ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ API Sandbox Endpoints │
+│ │
+│ Request ──▶ Start from snapshot ──▶ Search/Read ──▶ Return results │
+│ (instant) (grep/cat) │
+│ │
+│ Benefits: │
+│ - No clone delay (content already in sandbox) │
+│ - Consistent state across all requests │
+│ - Automatic cleanup (sandbox stops after use) │
+└─────────────────────────────────────────────────────────────────────────┘
+```
-interface YouTubeSource extends BaseSource {
- type: 'youtube'
- channelId: string
- maxVideos?: number
-}
+#### Snapshot Lifecycle
-interface CustomSource extends BaseSource {
- type: 'custom'
- fetchFn: () => Promise<ContentFile[]>
-}
+```
+Sync Workflow Snapshot Task API Request
+ │ │ │
+ │ push to GitHub │ │
+ ▼ │ │
+┌──────────┐ │ │
+│ Snapshot │ ◀──── triggers ────┤ │
+│ Repo │ │ │
+└──────────┘ ▼ │
+ ┌────────────┐ │
+ │ Create │ │
+ │ Sandbox │ │
+ └─────┬──────┘ │
+ │ clone repo │
+ ▼ │
+ ┌────────────┐ │
+ │ Take │ │
+ │ Snapshot │ │
+ └─────┬──────┘ │
+ │ │
+ ▼ ▼
+ ┌────────────┐ ┌────────────┐
+ │ Snapshot │ ─────────▶ │ Instant │
+ │ Ready │ start │ Sandbox │
+ └────────────┘ └────────────┘
+```
+
+#### Key Functions
-type Source = GitHubSource | YouTubeSource | CustomSource
+```typescript
+// Snapshot management
+export async function createContentSnapshot(): Promise<Snapshot>
+export async function getLatestSnapshot(): Promise<Snapshot | null>
+
+// Sandbox operations (internal to API)
+async function startFromSnapshot(snapshotId: string): Promise<Sandbox>
+async function searchInSandbox(sandbox: Sandbox, query: string, limit: number): Promise<string[]>
+async function readFromSandbox(sandbox: Sandbox, paths: string[]): Promise<FileContent[]>
```
-## API Endpoints Specification
+> **Note:** Snapshots expire after 7 days. The snapshot task should run regularly to ensure a fresh snapshot is always available.
+
+## API Endpoints
### Authentication
@@ -332,84 +335,119 @@ All endpoints require an API key in the `Authorization` header:
Authorization: Bearer <api-key>
```
-### Sandbox Endpoints
+### Source Endpoints
-#### POST /api/sandbox/create
+#### GET /api/sources
-Creates or recovers a sandbox for a chat session.
+Lists all configured content sources.
-**Request:**
+**Response:**
+```json
+{
+ "total": 25,
+ "github": {
+ "count": 23,
+ "sources": [
+ { "id": "nuxt", "label": "Nuxt", "repo": "nuxt/nuxt" }
+ ]
+ },
+ "youtube": {
+ "count": 2,
+ "sources": [
+ { "id": "alex-lichter", "label": "Alexander Lichter" }
+ ]
+ }
+}
+```
+
+### Sync Endpoints
+
+#### POST /api/sync
+
+Sync all sources.
+
+**Body (optional):**
```json
{
- "chatId": "unique-chat-identifier"
+ "reset": false,
+ "push": true
}
```
**Response:**
```json
{
- "sandboxId": "sbx_abc123",
- "fileTree": "docs/\n├── nuxt/ (45 files)\n├── nitro/ (23 files)..."
+ "status": "started",
+ "message": "Sync workflow started. Use `pnpm workflow:web` to monitor.",
+ "options": { "reset": false, "push": true }
}
```
-#### POST /api/sandbox/:id/search-and-read
+#### POST /api/sync/:source
-Combined search and read operation.
+Sync a specific source.
-**Request:**
+**Params:**
+- `source`: Source ID (e.g., `nuxt`, `nitro`)
+
+**Body (optional):**
```json
{
- "query": "useAsyncData",
- "limit": 5
+ "reset": false,
+ "push": true
}
```
**Response:**
```json
{
- "query": "useAsyncData",
- "files": [
- {
- "path": "packages/content/docs/nuxt/composables/use-async-data.md",
- "content": "# useAsyncData\n\n..."
- }
- ]
+ "status": "started",
+ "message": "Sync workflow started for source \"nuxt\". Use `pnpm workflow:web` to monitor.",
+ "source": "nuxt",
+ "options": { "reset": false, "push": true, "sourceFilter": "nuxt" }
}
```
-#### POST /api/sandbox/:id/search
+### Sandbox Endpoints (Planned)
-Search for files matching query.
+The API manages sandbox lifecycle internally. Sandboxes start instantly from the pre-built snapshot.
+
+#### POST /api/sandbox/search-and-read
+
+Combined search and read operation (recommended).
**Request:**
```json
{
- "query": "middleware",
- "limit": 10
+ "query": "useAsyncData",
+ "limit": 5,
+ "sessionId": "optional-session-id"
}
```
**Response:**
```json
{
+ "query": "useAsyncData",
"files": [
- "packages/content/docs/nuxt/guide/middleware.md",
- "packages/content/docs/nitro/middleware.md"
+ {
+ "path": "docs/nuxt/composables/use-async-data.md",
+ "content": "# useAsyncData\n\n..."
+ }
]
}
```
-#### POST /api/sandbox/:id/read
+#### POST /api/sandbox/search
-Read specific files.
+Search for files matching query.
**Request:**
```json
{
- "paths": [
- "packages/content/docs/nuxt/guide/middleware.md"
- ]
+ "query": "middleware",
+ "limit": 10,
+ "sessionId": "optional-session-id"
}
```
@@ -417,57 +455,55 @@ Read specific files.
```json
{
"files": [
- {
- "path": "packages/content/docs/nuxt/guide/middleware.md",
- "content": "# Middleware\n\n..."
- }
+ "docs/nuxt/guide/middleware.md",
+ "docs/nitro/middleware.md"
]
}
```
-### Sync Endpoints
+#### POST /api/sandbox/read
-#### POST /api/sync
-
-Triggers full content synchronization.
+Read specific files.
-**Response:**
+**Request:**
```json
{
- "status": "started",
- "sources": 25
+ "paths": [
+ "docs/nuxt/guide/middleware.md"
+ ],
+ "sessionId": "optional-session-id"
}
```
-#### GET /api/sources
-
-Lists configured sources.
-
**Response:**
```json
{
- "sources": [
+ "files": [
{
- "id": "nuxt",
- "label": "Nuxt",
- "type": "github",
- "repo": "nuxt/nuxt"
+ "path": "docs/nuxt/guide/middleware.md",
+ "content": "# Middleware\n\n..."
}
]
}
```
+> **Note:** The `sessionId` is optional and generic - can be from any client (Discord bot, Slack bot, GitHub bot, web chat, etc.). If provided, the API may reuse an existing sandbox for that session.
+
## Storage Strategy
### KV Store
-Used for sandbox session management:
+Used for storing the current snapshot ID and optionally sandbox sessions:
```typescript
-// Key format
-`sandbox:chat:${chatId}` -> SandboxRecord
+// Current snapshot (updated by snapshot task)
+`snapshot:current` -> { snapshotId: string, createdAt: number }
-// TTL: 30 minutes (sandbox timeout + buffer)
+// Optional: Active sandbox sessions (if reusing sandboxes across requests)
+`sandbox:session:${sessionId}` -> { sandboxId: string, createdAt: number }
+
+// sessionId is generic - can be from any client (Discord bot, Slack bot, GitHub bot, etc.)
+// TTL: 30 minutes (sandbox timeout)
```
### Snapshot Repository
@@ -475,7 +511,7 @@ Used for sandbox session management:
Structure:
```
-content-snapshot-repo/
+{GITHUB_SNAPSHOT_REPO}/
├── docs/
│ ├── nuxt/
│ │ ├── getting-started/
@@ -485,11 +521,19 @@ content-snapshot-repo/
│ ├── nitro/
│ └── ...
└── youtube/
- ├── alex-lichter/
- │ └── nuxt-4-overview-TAoTh4DqH6A.md
- └── ...
+ └── alex-lichter/
+ └── nuxt-4-overview-TAoTh4DqH6A.md
```
+## Environment Variables
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `GITHUB_TOKEN` | Yes | GitHub token for API access |
+| `GITHUB_SNAPSHOT_REPO` | Yes | Snapshot repository (owner/repo) |
+| `GITHUB_SNAPSHOT_BRANCH` | No | Branch to push to (default: main) |
+| `SAVOIR_SECRET_KEY` | No | API key for authentication |
+
## Error Handling
### SDK Errors
@@ -519,11 +563,17 @@ export class SavoirError extends Error {
## Performance Considerations
-### Sandbox Reuse
+### Snapshot-based Startup
+
+- Sandboxes start instantly from pre-built snapshot (no clone delay)
+- Snapshot contains all synced content ready to search
+- Snapshot is refreshed periodically by the snapshot task
+
+### Session Reuse (Optional)
-- Sandboxes are cached in KV by `chatId`
-- Timeout extended when < 3 minutes remaining
-- File tree cached to avoid regeneration
+- Sandboxes can be reused within a session via `sessionId`
+- Avoids startup cost for multiple requests in same conversation
+- Automatic cleanup when session ends or timeout
### Batch Operations
@@ -540,7 +590,7 @@ export class SavoirError extends Error {
### API Key Validation
```typescript
-// apps/api/server/middleware/auth.ts
+// apps/api/middleware/auth.ts
export default defineEventHandler((event) => {
const auth = getHeader(event, 'Authorization')
const apiKey = auth?.replace('Bearer ', '')
diff --git a/docs/SOURCES.md b/docs/SOURCES.md
index 1e31459..b494e0a 100644
--- a/docs/SOURCES.md
+++ b/docs/SOURCES.md
@@ -1,78 +1,29 @@
# Content Sources
-Savoir aggregates documentation from multiple sources into a unified, searchable knowledge base. This document describes how to configure, add, and manage content sources.
+Savoir aggregates documentation from multiple sources into a unified, searchable knowledge base.
-## Overview
+## Configuration
-Sources are external data repositories that Savoir fetches, normalizes to Markdown, and stores in a snapshot repository. The AI agent then searches and reads from this snapshot via a Vercel Sandbox.
-
-```
-┌────────────────────────────────────────────────────────────────────┐
-│ Source Types │
-│ │
-│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │
-│ │ GitHub │ │ YouTube │ │ MCP │ │ Custom (future) │ │
-│ │ Docs │ │ Transcripts│ │ Servers │ │ Blogs, APIs │ │
-│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────────┬─────────┘ │
-│ │ │ │ │ │
-│ └─────────────┴──────┬──────┴──────────────────┘ │
-│ │ │
-│ ▼ │
-│ ┌─────────────────┐ │
-│ │ Normalizer │ │
-│ │ (to Markdown) │ │
-│ └────────┬────────┘ │
-│ │ │
-│ ▼ │
-│ ┌─────────────────┐ │
-│ │ Snapshot Repo │ │
-│ │ (GitHub) │ │
-│ └─────────────────┘ │
-└────────────────────────────────────────────────────────────────────┘
-```
-
-## Source Configuration
-
-Sources are defined in `apps/api/server/utils/sources.ts`:
+Sources are defined in `savoir.config.ts` at the project root:
```typescript
-// apps/api/server/utils/sources.ts
-import type { Source } from './types'
-
-export const SOURCES: Source[] = [
- // GitHub documentation
- {
- id: 'nuxt',
- label: 'Nuxt',
- type: 'github',
- repo: 'nuxt/nuxt',
- branch: 'main',
- contentPath: 'docs/content',
- outputPath: 'nuxt',
- },
-
- // README-only sources
- {
- id: 'unhead',
- label: 'Unhead',
- type: 'github',
- repo: 'unjs/unhead',
- branch: 'main',
- contentPath: '.',
- readmeOnly: true,
- },
-
- // YouTube channels
- {
- id: 'alex-lichter',
- label: 'Alex Lichter',
- type: 'youtube',
- channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA',
- maxVideos: 100,
+import { defineConfig } from '@savoir/config'
+
+export default defineConfig({
+ sources: {
+ github: [
+ { id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' },
+ { id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' },
+ ],
+ youtube: [
+ { id: 'alex-lichter', channelId: 'UCqFPgMzGbLjd-MX-h3Z5aQA' },
+ ],
},
-]
+})
```
+Supported formats: `.ts`, `.js`, `.json`, `.yaml`
+
## Source Types
### GitHub Sources
@@ -80,35 +31,18 @@ export const SOURCES: Source[] = [
Fetches Markdown documentation from GitHub repositories.
```typescript
-interface GitHubSource {
- type: 'github'
-
- /** Unique identifier */
- id: string
-
- /** Display name */
- label: string
-
- /** GitHub repository (owner/repo) */
- repo: string
-
- /** Branch to fetch from */
- branch: string
-
- /** Path to content directory */
- contentPath: string
-
- /** Output directory in snapshot (defaults to id) */
- outputPath?: string
-
- /** Only fetch README.md */
- readmeOnly?: boolean
-
- /** Additional repos to merge into same output */
+{
+ id: string // Unique identifier
+ label?: string // Display name (defaults to capitalized id)
+ repo: string // GitHub repository (owner/repo)
+ branch?: string // Branch to fetch from (default: 'main')
+ contentPath?: string // Path to content directory (default: 'docs')
+ outputPath?: string // Output directory in snapshot (default: id)
+ readmeOnly?: boolean // Only fetch README.md (default: false)
additionalSyncs?: Array<{
repo: string
- branch: string
- contentPath: string
+ branch?: string
+ contentPath?: string
}>
}
```
@@ -117,43 +51,21 @@ interface GitHubSource {
```typescript
// Full documentation tree
-{
- id: 'nuxt',
- label: 'Nuxt',
- type: 'github',
- repo: 'nuxt/nuxt',
- branch: 'main',
- contentPath: 'docs/content',
- outputPath: 'nuxt',
-}
+{ id: 'nuxt', repo: 'nuxt/nuxt', contentPath: 'docs' }
+
+// Specific branch
+{ id: 'nitro', repo: 'nitrojs/nitro', branch: 'v3' }
// README only
-{
- id: 'ofetch',
- label: 'ofetch',
- type: 'github',
- repo: 'unjs/ofetch',
- branch: 'main',
- contentPath: '.',
- readmeOnly: true,
- outputPath: 'ofetch',
-}
+{ id: 'ofetch', repo: 'unjs/ofetch', readmeOnly: true }
-// Multiple repos merged into one output
+// Multiple repos merged
{
- id: 'nuxt-ui',
- label: 'Nuxt UI',
- type: 'github',
- repo: 'nuxt/ui',
- branch: 'v3',
- contentPath: 'docs/content',
- outputPath: 'nuxt-ui',
+ id: 'nuxt',
+ repo: 'nuxt/nuxt',
+ contentPath: 'docs',
additionalSyncs: [
- {
- repo: 'nuxt/ui-pro',
- branch: 'main',
- contentPath: 'docs/content',
- }
+ { repo: 'nuxt/nuxt.com', contentPath: 'content' },
],
}
```
@@ -163,315 +75,87 @@ interface GitHubSource {
Fetches video transcripts from YouTube channels.
```typescript
-interface YouTubeSource {
- type: 'youtube'
-
- /** Unique identifier */
- id: string
-
- /** Display name */
- label: string
-
- /** YouTube channel ID */
- channelId: string
-
- /** Maximum videos to fetch (default: 50) */
- maxVideos?: number
+{
+ id: string // Unique identifier
+ label?: string // Display name
+ channelId: string // YouTube channel ID
+ handle?: string // YouTube handle (e.g., '@TheAlexLichter')
+ maxVideos?: number // Maximum videos to fetch (default: 50)
}
```
-**Output format:**
-
-```markdown
----
-title: "Nuxt 4: An Overview"
-videoId: TAoTh4DqH6A
-channelId: UCqFPgMzGbLjd-MX-h3Z5aQA
-publishedAt: 2024-01-15T10:00:00Z
----
-
-# Nuxt 4: An Overview
-
-[Transcript content here...]
-```
-
-### Custom Sources (Extensible)
+### Custom Sources
For sources that don't fit the standard patterns:
-```typescript
-interface CustomSource {
- type: 'custom'
-
- /** Unique identifier */
- id: string
-
- /** Display name */
- label: string
-
- /** Custom fetch function */
- fetchFn: () => Promise<ContentFile[]>
-}
-
-interface ContentFile {
- path: string // Relative path in output
- content: string // Markdown content
-}
-```
-
-**Example:**
-
```typescript
{
- id: 'blog-posts',
- label: 'Community Blog',
- type: 'custom',
- fetchFn: async () => {
- const posts = await fetchBlogPosts()
- return posts.map(post => ({
- path: `blog/${post.slug}.md`,
- content: `---
-title: ${post.title}
-author: ${post.author}
-date: ${post.date}
----
-
-${post.content}`,
- }))
- },
+ id: string
+ label?: string
+ fetchFn: () => Promise<Array<{ path: string; content: string }>>
}
```
-## Adding a New Source
+## Syncing
-### Step 1: Define the Source
-
-Add to `apps/api/server/utils/sources.ts`:
-
-```typescript
-export const SOURCES: Source[] = [
- // ... existing sources
-
- {
- id: 'my-project',
- label: 'My Project',
- type: 'github',
- repo: 'my-org/my-project',
- branch: 'main',
- contentPath: 'docs',
- },
-]
-```
-
-### Step 2: Run Sync
-
-Trigger synchronization:
+### Via API
```bash
# Sync all sources
-pnpm sync
+curl -X POST http://localhost:3000/api/sync
# Sync specific source
-pnpm sync my-project
-```
-
-Or via API:
+curl -X POST http://localhost:3000/api/sync/nuxt
-```bash
-curl -X POST https://api.savoir.example.com/api/sync/my-project \
- -H "Authorization: Bearer $SAVOIR_API_KEY"
-```
-
-### Step 3: Verify
-
-Check the snapshot repository for the new content:
-
-```
-content-snapshot-repo/
-└── docs/
- └── my-project/
- ├── getting-started.md
- └── api-reference.md
-```
-
-## Sync Process
-
-### Nitro Tasks
-
-Content synchronization is handled by Nitro scheduled tasks:
-
-```typescript
-// apps/api/server/tasks/sync-content.ts
-export default defineTask({
- meta: {
- name: 'sync:content',
- description: 'Synchronize content from all sources',
- },
- async run() {
- for (const source of SOURCES) {
- await syncSource(source)
- }
- await pushToSnapshot()
- },
-})
-```
-
-### Scheduling
-
-Configure in `nitro.config.ts`:
-
-```typescript
-export default defineNitroConfig({
- experimental: {
- tasks: true,
- },
- scheduledTasks: {
- // Sync every 6 hours
- '0 */6 * * *': ['sync:content'],
- },
-})
+# With options
+curl -X POST http://localhost:3000/api/sync \
+ -H "Content-Type: application/json" \
+ -d '{"reset": true, "push": true}'
```
-### Manual Trigger
-
-```bash
-# Via CLI
-pnpm sync
+### Options
-# Via API
-curl -X POST https://api.savoir.example.com/api/sync \
- -H "Authorization: Bearer $API_KEY"
-```
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `reset` | boolean | false | Clear content before sync |
+| `push` | boolean | true | Push to snapshot repo after sync |
## Content Normalization
-All content is normalized to Markdown with consistent structure:
-
-### File Types
+All content is normalized to Markdown:
| Input | Output |
|-------|--------|
| `.md` | Preserved as-is |
| `.mdx` | Converted to `.md` |
-| `.yml`/`.yaml` | Preserved (for frontmatter) |
-| `.json` | Preserved (for configs) |
+| `.yml`/`.yaml` | Preserved |
+| `.json` | Preserved |
| Other | Ignored |
### Excluded Files
-Lock files and non-documentation are automatically excluded:
-
-- `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`
+- Lock files (`package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`, etc.)
- `node_modules/`
- Binary files
-- Test files
-
-### Directory Structure
-
-Sources maintain their original hierarchy:
-
-```
-GitHub: nuxt/nuxt/docs/content/getting-started/installation.md
- ↓
-Snapshot: docs/nuxt/getting-started/installation.md
-```
## Snapshot Repository
-The snapshot repository is a GitHub repo containing all aggregated content.
-
-### Structure
+The snapshot repository contains all aggregated content:
```
{GITHUB_SNAPSHOT_REPO}/
├── docs/
│ ├── nuxt/
│ │ ├── getting-started/
-│ │ │ ├── installation.md
-│ │ │ └── configuration.md
-│ │ ├── composables/
-│ │ │ ├── use-async-data.md
-│ │ │ └── use-fetch.md
-│ │ └── ...
+│ │ └── composables/
│ ├── nitro/
-│ ├── h3/
│ └── ...
└── youtube/
- ├── alex-lichter/
- │ ├── nuxt-4-overview-TAoTh4DqH6A.md
- │ └── ...
- └── learn-vue/
+ └── alex-lichter/
```
-### Configuration
-
-Set via environment variable:
+Configure via environment variable:
```bash
GITHUB_SNAPSHOT_REPO=my-org/content-snapshot
```
-
-### Updates
-
-The sync process:
-
-1. Fetches content from all sources
-2. Transforms and normalizes
-3. Commits changes to snapshot repo
-4. New sandboxes clone the updated snapshot
-
-## Best Practices
-
-### Source Selection
-
-1. **Prioritize official documentation** over third-party content
-2. **Include README-only sources** for smaller libraries
-3. **Add YouTube sources** for tutorial content and explanations
-
-### Performance
-
-1. **Batch updates** rather than syncing after each change
-2. **Use `readmeOnly`** for packages without full docs
-3. **Limit YouTube videos** to prevent excessive API usage
-
-### Content Quality
-
-1. **Preserve frontmatter** for metadata
-2. **Maintain original structure** for discoverability
-3. **Include version info** in source paths when relevant
-
-## Troubleshooting
-
-### Sync Failures
-
-Check logs for specific errors:
-
-```bash
-pnpm sync my-source --verbose
-```
-
-Common issues:
-- **Rate limits**: Wait and retry, or use authenticated requests
-- **Missing content path**: Verify `contentPath` exists in repo
-- **Branch not found**: Check `branch` matches repository
-
-### Missing Content
-
-1. Verify source configuration
-2. Check if files match allowed extensions
-3. Ensure content isn't in excluded files list
-
-### Stale Content
-
-Force a fresh sync:
-
-```bash
-pnpm sync --reset
-```
-
-Or via API:
-
-```bash
-curl -X POST https://api.savoir.example.com/api/sync?reset=true \
- -H "Authorization: Bearer $API_KEY"
-```