Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

6.4 KB
Raw Blame

Architecture Overview

OpenViking is a context database designed for AI Agents, unifying all context types (Memory, Resource, Skill) into a directory structure with semantic retrieval and progressive content loading.

System Overview

Text version
CLI / SDK / HTTP client
          |
      HTTP Server
          |
      Service Layer
          |
   +------+------+----------------+
   |             |                |
Retrieval     Sessions       Resource / Skill import
   |             |                |
   |       Memory extraction  Parse / Semantic queues
   |             |                |
   +-------------+----------------+
                 |
             VikingFS
           /          \
      RAGFS         Vector index

Core Modules

Module Responsibility Key Capabilities
Client Unified entry Sends supported SDK/CLI operations to the HTTP API
Service Business logic FSService, SearchService, SessionService, ResourceService, PackService, DebugService
Retrieve Context retrieval Intent analysis (IntentAnalyzer), global retrieval (HierarchicalRetriever), Rerank
Session Session management Message recording, usage tracking, session archiving, triggering memory commit
Parse Context extraction Document parsing (PDF/MD/HTML), tree building (TreeBuilder), async semantic generation
Memory Memory extraction Schema-driven extraction by MemoryType (ExtractLoop), LLM merge and deduplication written back as patches (MemoryUpdater); orchestrated by SessionCompressorV3
Storage Storage layer VikingFS virtual filesystem, vector index, RAGFS integration

Service Layer

The Service layer decouples business logic from the transport layer. The CLI and SDKs reach it through the HTTP Server:

Service Responsibility Key Methods
FSService File system operations ls, mkdir, rm, mv, tree, stat, read, abstract, overview, grep, glob
SearchService Semantic search search, find
SessionService Session management session, sessions, commit, delete
ResourceService Resource import add_resource, add_skill, wait_processed
PackService Import/export and backup/restore export_ovpack, import_ovpack, backup_ovpack, restore_ovpack
DebugService Debug service observer (ObserverService)

Dual-Layer Storage

OpenViking uses a dual-layer storage architecture separating content from index (see Storage Architecture):

Layer Responsibility Content
RAGFS Content storage L0/L1/L2 full content, multimedia files
Vector Index Index storage URIs, vectors, metadata, and text used for retrieval, including abstracts

Data Flow Overview

Adding Context

Input → Parser → TreeBuilder → ResourceProcessor → RAGFS → SemanticQueue → Vector Index
  1. Parser: Parse source documents into files and directories; model use depends on the selected parser
  2. TreeBuilder: Resolve the target URI and retain temporary artifact references
  3. ResourceProcessor: Persist content in RAGFS and enqueue semantic processing
  4. SemanticQueue: Async bottom-up L0/L1 generation
  5. Vector Index: Build index for semantic search

Retrieving Context

Query → Intent Analysis → Global Retrieval → Rerank → Results
  1. Intent Analysis: Analyze query intent, generate 0-5 typed queries
  2. Global Retrieval: One vector search per query within the permitted scope
  3. Rerank: Optional single pass over the recalled candidates
  4. Results: Return contexts sorted by relevance

Session Commit

Messages → Archive Boundary → Archive → Memory Extraction → Storage
  1. Messages: Accumulate conversation messages and usage records
  2. Archive Boundary: Split archived and retained messages according to the commit parameters; by default, archive all current messages
  3. Archive: Generate L0/L1 for history segments
  4. Memory Extraction: Extract memories from messages according to the memory policy and MemoryType schemas
  5. Storage: Write to RAGFS + vector index

Deployment Mode

HTTP Mode

For team sharing, production deployment, and cross-language integration:

# Python SDK connects to OpenViking Server
from openviking_sdk import SyncHTTPClient

client = SyncHTTPClient(url="http://localhost:1933", api_key="your-key")
# Or use curl / any HTTP client
curl http://localhost:1933/api/v1/search/find \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"query": "how to use openviking"}'
  • Server runs as standalone process (openviking-server)
  • Clients connect via HTTP API
  • Supports any language that can make HTTP requests
  • See Server Deployment for setup

Design Principles

Principle Description
Pure Storage Layer Storage only handles RAGFS operations and basic vector search; Rerank is in retrieval layer
Three-Layer Information L0/L1/L2 enables progressive detail loading, saving token consumption
Two-Stage Retrieval Vector search recalls candidates + Rerank improves accuracy
Single Data Source RAGFS holds source files; vector records retain the text and metadata needed for retrieval

Related Documents