Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

docs(concepts): review concept pages and replace the ASCII architecture diagram (#5510)

Zayn Jarvis66ef1ce8c88b
diff --git a/docs/.vitepress/theme/components/ArchitectureDiagram.vue b/docs/.vitepress/theme/components/ArchitectureDiagram.vue
new file mode 100644
index 000000000..ec7e833c6
--- /dev/null
+++ b/docs/.vitepress/theme/components/ArchitectureDiagram.vue
@@ -0,0 +1,145 @@
+<script setup>
+// Centers of the three pipeline columns inside the dashed group.
+const columns = [60, 185.5, 325.5]
+</script>
+
+<template>
+ <figure class="arch-diagram">
+ <svg viewBox="0 0 400 586" role="img" aria-labelledby="arch-diagram-title arch-diagram-desc">
+ <title id="arch-diagram-title">OpenViking system overview</title>
+ <desc id="arch-diagram-desc">
+ CLI, SDK and HTTP clients call the HTTP Server (REST API and the /mcp endpoint), which calls the
+ Service Layer. The Service Layer drives three pipelines: Retrieval; Sessions with memory
+ extraction; and Resource and Skill import with parse and semantic queues. All three read and
+ write VikingFS, which stores content in RAGFS and indexes it in the vector index.
+ </desc>
+ <defs>
+ <marker id="arch-diagram-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
+ <path class="arrow-head" d="M1 1.5 9 5 1 8.5z" />
+ </marker>
+ </defs>
+
+ <g class="arrows" marker-end="url(#arch-diagram-arrow)">
+ <path d="M200 52v24" />
+ <path d="M200 136v24" />
+ <path v-for="x in columns" :key="`in-${x}`" :d="`M${x} 220v34`" />
+ <path v-for="x in columns" :key="`out-${x}`" :d="`M${x} 394v34`" />
+ <path d="M115 488v30" />
+ <path d="M285 488v30" />
+ </g>
+
+ <rect class="group" x="4" y="238" width="392" height="170" rx="12" />
+
+ <rect class="box" x="70" y="8" width="260" height="44" rx="8" />
+ <text class="name" x="200" y="30">CLI · SDK · HTTP client</text>
+
+ <rect class="box" x="70" y="78" width="260" height="58" rx="8" />
+ <text class="name" x="200" y="100">HTTP Server</text>
+ <text class="note" x="200" y="121">REST API · /mcp</text>
+
+ <rect class="box" x="4" y="162" width="392" height="58" rx="8" />
+ <text class="name" x="200" y="184">Service Layer</text>
+ <text class="note" x="200" y="205">FS · Search · Session · Resource · Pack · Debug</text>
+
+ <rect class="box" x="10" y="256" width="100" height="138" rx="8" />
+ <text class="name" x="60" y="316">Retrieval</text>
+ <text class="note" x="60" y="337">search · find</text>
+
+ <rect class="box" x="121" y="256" width="129" height="58" rx="8" />
+ <text class="name" x="185.5" y="285">Sessions</text>
+ <path class="step" d="M185.5 314v22" marker-end="url(#arch-diagram-arrow)" />
+ <rect class="box sub" x="121" y="336" width="129" height="58" rx="8" />
+ <text class="step-name" x="185.5" y="360">Memory</text>
+ <text class="step-name" x="185.5" y="378">extraction</text>
+
+ <rect class="box" x="261" y="256" width="129" height="58" rx="8" />
+ <text class="name" x="325.5" y="280">Resource /</text>
+ <text class="name" x="325.5" y="299">Skill import</text>
+ <path class="step" d="M325.5 314v22" marker-end="url(#arch-diagram-arrow)" />
+ <rect class="box sub" x="261" y="336" width="129" height="58" rx="8" />
+ <text class="step-name" x="325.5" y="360">Parse /</text>
+ <text class="step-name" x="325.5" y="378">Semantic queues</text>
+
+ <rect class="box core" x="40" y="430" width="320" height="58" rx="8" />
+ <text class="name" x="200" y="452">VikingFS</text>
+ <text class="note" x="200" y="473">virtual filesystem</text>
+
+ <rect class="box" x="40" y="520" width="150" height="58" rx="8" />
+ <text class="name" x="115" y="542">RAGFS</text>
+ <text class="note" x="115" y="563">source files</text>
+
+ <rect class="box" x="210" y="520" width="150" height="58" rx="8" />
+ <text class="name" x="285" y="542">Vector index</text>
+ <text class="note" x="285" y="563">vector records</text>
+ </svg>
+ </figure>
+</template>
+
+<style scoped>
+.arch-diagram {
+ max-width: 460px;
+ margin: 16px auto;
+}
+
+svg {
+ display: block;
+ width: 100%;
+ height: auto;
+ font-family: inherit;
+}
+
+text {
+ text-anchor: middle;
+ dominant-baseline: middle;
+}
+
+.name {
+ font-size: 15.5px;
+ font-weight: 600;
+ fill: var(--vp-c-text-1);
+}
+
+.note {
+ font-size: 13.5px;
+ fill: var(--vp-c-text-2);
+}
+
+.step-name {
+ font-size: 13.5px;
+ font-weight: 500;
+ fill: var(--vp-c-text-1);
+}
+
+.box {
+ fill: var(--vp-c-bg-soft);
+ stroke: var(--vp-c-divider);
+ stroke-width: 1;
+}
+
+.box.sub {
+ fill: var(--vp-c-bg);
+}
+
+.box.core {
+ fill: var(--vp-c-brand-soft);
+ stroke: var(--vp-c-brand-1);
+ stroke-width: 1.5;
+}
+
+.group {
+ fill: none;
+ stroke: var(--vp-c-divider);
+ stroke-dasharray: 5 4;
+}
+
+.arrows path,
+.step {
+ fill: none;
+ stroke: var(--vp-c-text-2);
+ stroke-width: 1.25;
+}
+
+.arrow-head {
+ fill: var(--vp-c-text-2);
+}
+</style>
diff --git a/docs/.vitepress/theme/components/IngestionPipelineDiagram.vue b/docs/.vitepress/theme/components/IngestionPipelineDiagram.vue
new file mode 100644
index 000000000..1c37f18ae
--- /dev/null
+++ b/docs/.vitepress/theme/components/IngestionPipelineDiagram.vue
@@ -0,0 +1,22 @@
+<script setup>
+import FlowDiagram from './FlowDiagram.vue'
+
+const steps = [
+ { name: 'Input', notes: ['file, directory, URL, upload'] },
+ { name: 'Parser', notes: ['parse artifacts in a temp location'] },
+ { name: 'TreeBuilder', notes: ['resolve the target URI'] },
+ { name: 'Content persistence', notes: ['write files, hold path locks,', 'enqueue derived work'] },
+ { name: 'SemanticQueue', notes: ['L0 / L1 generation (model calls)'] },
+ { name: 'EmbeddingQueue', notes: ['embed abstracts and content'] },
+ { name: 'Vector index', notes: ['searchable records'] }
+]
+</script>
+
+<template>
+ <FlowDiagram
+ title="Resource ingestion pipeline"
+ desc="Input goes through the Parser, TreeBuilder, content persistence, the semantic queue and the embedding queue, and ends in the vector index."
+ :steps="steps"
+ :core="3"
+ />
+</template>
diff --git a/docs/.vitepress/theme/components/MemoryExtractionDiagram.vue b/docs/.vitepress/theme/components/MemoryExtractionDiagram.vue
new file mode 100644
index 000000000..44f2a7a72
--- /dev/null
+++ b/docs/.vitepress/theme/components/MemoryExtractionDiagram.vue
@@ -0,0 +1,19 @@
+<script setup>
+import FlowDiagram from './FlowDiagram.vue'
+
+const steps = [
+ { name: 'Archived messages', notes: ['+ enabled memory schemas'] },
+ { name: 'ExtractLoop', notes: ['reads relevant existing memories,', 'produces update operations'] },
+ { name: 'MemoryUpdater', notes: ['validates and applies', 'additions, edits, deletions'] },
+ { name: 'Persist', notes: ['memory files · indexes · memory_diff.json'] }
+]
+</script>
+
+<template>
+ <FlowDiagram
+ title="Memory extraction"
+ desc="Archived messages and the enabled memory schemas feed ExtractLoop, which produces update operations; MemoryUpdater validates and applies them; memories, indexes and memory_diff.json are then persisted."
+ :steps="steps"
+ :core="2"
+ />
+</template>
diff --git a/docs/.vitepress/theme/components/PathLockDiagram.vue b/docs/.vitepress/theme/components/PathLockDiagram.vue
new file mode 100644
index 000000000..257fb8252
--- /dev/null
+++ b/docs/.vitepress/theme/components/PathLockDiagram.vue
@@ -0,0 +1,19 @@
+<script setup>
+import FlowDiagram from './FlowDiagram.vue'
+
+const steps = [
+ { name: 'Service operation', notes: ['rm · mv · add_resource · session.commit'] },
+ { name: 'Acquire a lease', notes: ['RAGFS PathLockManager', 'EXACT path or TREE subtree'] },
+ { name: 'Modify covered paths', notes: ['files, then indexes and queues'] },
+ { name: 'Release the lease', notes: ['or hand it to background processing'] }
+]
+</script>
+
+<template>
+ <FlowDiagram
+ title="Path lock lifecycle"
+ desc="A service operation acquires a lease from the RAGFS PathLockManager, modifies the covered paths while coordinating indexes and queues, then releases the lease or hands it to background processing."
+ :steps="steps"
+ :core="1"
+ />
+</template>
diff --git a/docs/.vitepress/theme/components/StorageLayersDiagram.vue b/docs/.vitepress/theme/components/StorageLayersDiagram.vue
new file mode 100644
index 000000000..0d047311f
--- /dev/null
+++ b/docs/.vitepress/theme/components/StorageLayersDiagram.vue
@@ -0,0 +1,85 @@
+<template>
+ <figure class="storage-diagram">
+ <svg viewBox="0 0 400 200" role="img" aria-labelledby="storage-diagram-title storage-diagram-desc">
+ <title id="storage-diagram-title">Dual-layer storage</title>
+ <desc id="storage-diagram-desc">
+ VikingFS maps viking:// URIs and hierarchical access onto two stores: the vector index for
+ semantic search and AGFS for file content.
+ </desc>
+ <defs>
+ <marker id="storage-diagram-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
+ <path class="arrow-head" d="M1 1.5 9 5 1 8.5z" />
+ </marker>
+ </defs>
+
+ <g class="arrows" marker-end="url(#storage-diagram-arrow)">
+ <path d="M200 66v24h-95v26" />
+ <path d="M200 66v24h95v26" />
+ </g>
+
+ <rect class="box core" x="40" y="8" width="320" height="58" rx="8" />
+ <text class="name" x="200" y="30">VikingFS</text>
+ <text class="note" x="200" y="51">URI mapping · hierarchical access</text>
+
+ <rect class="box" x="20" y="118" width="170" height="70" rx="8" />
+ <text class="name" x="105" y="142">Vector index</text>
+ <text class="note" x="105" y="163">semantic search</text>
+
+ <rect class="box" x="210" y="118" width="170" height="70" rx="8" />
+ <text class="name" x="295" y="142">AGFS</text>
+ <text class="note" x="295" y="163">file content</text>
+ </svg>
+ </figure>
+</template>
+
+<style scoped>
+.storage-diagram {
+ max-width: 460px;
+ margin: 16px auto;
+}
+
+svg {
+ display: block;
+ width: 100%;
+ height: auto;
+ font-family: inherit;
+}
+
+text {
+ text-anchor: middle;
+ dominant-baseline: middle;
+}
+
+.name {
+ font-size: 15.5px;
+ font-weight: 600;
+ fill: var(--vp-c-text-1);
+}
+
+.note {
+ font-size: 13.5px;
+ fill: var(--vp-c-text-2);
+}
+
+.box {
+ fill: var(--vp-c-bg-soft);
+ stroke: var(--vp-c-divider);
+ stroke-width: 1;
+}
+
+.box.core {
+ fill: var(--vp-c-brand-soft);
+ stroke: var(--vp-c-brand-1);
+ stroke-width: 1.5;
+}
+
+.arrows path {
+ fill: none;
+ stroke: var(--vp-c-text-2);
+ stroke-width: 1.25;
+}
+
+.arrow-head {
+ fill: var(--vp-c-text-2);
+}
+</style>
diff --git a/docs/.vitepress/theme/components/VikingBotOverviewDiagram.vue b/docs/.vitepress/theme/components/VikingBotOverviewDiagram.vue
new file mode 100644
index 000000000..c372c838c
--- /dev/null
+++ b/docs/.vitepress/theme/components/VikingBotOverviewDiagram.vue
@@ -0,0 +1,118 @@
+<template>
+ <figure class="bot-diagram">
+ <svg viewBox="0 0 400 470" role="img" aria-labelledby="bot-diagram-title bot-diagram-desc">
+ <title id="bot-diagram-title">VikingBot system overview</title>
+ <desc id="bot-diagram-desc">
+ Channels (CLI, Feishu, Slack, Telegram, Discord, Email, HTTP API) feed the channel layer and
+ message bus, which drives the AgentLoop: context, model, tools, model. The loop reads
+ OpenViking context (resources, memories, experiences, sessions) and runs tools and skills
+ (files, shell, web, MCP, cron, subagents); replies and events go back to the channel, and the
+ session is synchronized to OpenViking for learning.
+ </desc>
+ <defs>
+ <marker id="bot-diagram-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
+ <path class="arrow-head" d="M1 1.5 9 5 1 8.5z" />
+ </marker>
+ </defs>
+
+ <g class="arrows" marker-end="url(#bot-diagram-arrow)">
+ <path d="M200 66v22" />
+ <path d="M200 146v22" />
+ <path d="M105 246v16" />
+ <path d="M295 246v16" />
+ <path d="M105 334v24h95v30" />
+ <path d="M295 334v24h-95v30" />
+ </g>
+ <path class="arrows back" d="M320 197h30v-80h-28" marker-end="url(#bot-diagram-arrow)" />
+ <text class="edge" x="360" y="157">replies</text>
+
+ <rect class="box" x="20" y="8" width="360" height="58" rx="8" />
+ <text class="name" x="200" y="30">Channels</text>
+ <text class="note" x="200" y="51">CLI · Feishu · Slack · Telegram · Discord · Email · HTTP</text>
+
+ <rect class="box" x="80" y="88" width="240" height="58" rx="8" />
+ <text class="name" x="200" y="110">Channel + MessageBus</text>
+ <text class="note" x="200" y="131">routes messages and events</text>
+
+ <rect class="box core" x="80" y="168" width="240" height="58" rx="8" />
+ <text class="name" x="200" y="190">AgentLoop</text>
+ <text class="note" x="200" y="211">context → model → tools → model</text>
+
+ <rect class="box" x="20" y="262" width="170" height="72" rx="8" />
+ <text class="name" x="105" y="284">OpenViking context</text>
+ <text class="note" x="105" y="304">resources · memories</text>
+ <text class="note" x="105" y="321">experiences · sessions</text>
+
+ <rect class="box" x="210" y="262" width="170" height="72" rx="8" />
+ <text class="name" x="295" y="284">Tools / Skills</text>
+ <text class="note" x="295" y="304">files · shell · web</text>
+ <text class="note" x="295" y="321">MCP · cron · subagents</text>
+
+ <rect class="box" x="70" y="390" width="260" height="58" rx="8" />
+ <text class="name" x="200" y="412">Session sync and learning</text>
+ <text class="note" x="200" y="433">commit to OpenViking</text>
+ </svg>
+ </figure>
+</template>
+
+<style scoped>
+.bot-diagram {
+ max-width: 460px;
+ margin: 16px auto;
+}
+
+svg {
+ display: block;
+ width: 100%;
+ height: auto;
+ font-family: inherit;
+}
+
+text {
+ text-anchor: middle;
+ dominant-baseline: middle;
+}
+
+.name {
+ font-size: 15.5px;
+ font-weight: 600;
+ fill: var(--vp-c-text-1);
+}
+
+.note {
+ font-size: 13px;
+ fill: var(--vp-c-text-2);
+}
+
+.edge {
+ font-size: 12.5px;
+ fill: var(--vp-c-text-2);
+}
+
+.box {
+ fill: var(--vp-c-bg-soft);
+ stroke: var(--vp-c-divider);
+ stroke-width: 1;
+}
+
+.box.core {
+ fill: var(--vp-c-brand-soft);
+ stroke: var(--vp-c-brand-1);
+ stroke-width: 1.5;
+}
+
+.arrows path,
+.arrows.back {
+ fill: none;
+ stroke: var(--vp-c-text-2);
+ stroke-width: 1.25;
+}
+
+.arrows.back {
+ stroke-dasharray: 5 4;
+}
+
+.arrow-head {
+ fill: var(--vp-c-text-2);
+}
+</style>
diff --git a/docs/.vitepress/theme/custom.css b/docs/.vitepress/theme/custom.css
index b791a11e8..e12afcb62 100644
--- a/docs/.vitepress/theme/custom.css
+++ b/docs/.vitepress/theme/custom.css
@@ -507,3 +507,9 @@ html.ov-assistant-pill .VPContent {
/* The custom menu includes browser-following mode on desktop and mobile. */
.VPNavBarTranslations, .VPNavScreenTranslations { display: none !important; }
+
+/* Architecture page: keep module name and responsibility on one line. */
+.vp-doc .module-table th:nth-child(-n + 2),
+.vp-doc .module-table td:nth-child(-n + 2) {
+ white-space: nowrap;
+}
diff --git a/docs/.vitepress/theme/index.ts b/docs/.vitepress/theme/index.ts
index 32d079a52..f38d9f8ed 100644
--- a/docs/.vitepress/theme/index.ts
+++ b/docs/.vitepress/theme/index.ts
@@ -3,6 +3,12 @@ import { createLanguagePreference } from './language-preference.js'
import { h, defineAsyncComponent } from 'vue'
import DefaultTheme, { VPButton } from 'vitepress/theme'
import DocBreadcrumb from './components/DocBreadcrumb.vue'
+import ArchitectureDiagram from './components/ArchitectureDiagram.vue'
+import IngestionPipelineDiagram from './components/IngestionPipelineDiagram.vue'
+import MemoryExtractionDiagram from './components/MemoryExtractionDiagram.vue'
+import PathLockDiagram from './components/PathLockDiagram.vue'
+import StorageLayersDiagram from './components/StorageLayersDiagram.vue'
+import VikingBotOverviewDiagram from './components/VikingBotOverviewDiagram.vue'
import LocaleSwitch from './components/LocaleSwitch.vue'
import { useData, withBase } from 'vitepress'
import type { EnhanceAppContext } from 'vitepress'
@@ -264,6 +270,12 @@ export default {
},
enhanceApp({ app, router }: EnhanceAppContext) {
app.component('VPButton', VPButton)
+ app.component('ArchitectureDiagram', ArchitectureDiagram)
+ app.component('IngestionPipelineDiagram', IngestionPipelineDiagram)
+ app.component('MemoryExtractionDiagram', MemoryExtractionDiagram)
+ app.component('PathLockDiagram', PathLockDiagram)
+ app.component('StorageLayersDiagram', StorageLayersDiagram)
+ app.component('VikingBotOverviewDiagram', VikingBotOverviewDiagram)
app.component('DocsHome', defineAsyncComponent(() => import('./components/DocsHome.vue')))
if (import.meta.env.SSR || typeof window === 'undefined') return
diff --git a/docs/en/concepts/01-architecture.md b/docs/en/concepts/01-architecture.md
index 9527a1c77..29082b396 100644
--- a/docs/en/concepts/01-architecture.md
+++ b/docs/en/concepts/01-architecture.md
@@ -4,66 +4,52 @@ OpenViking is a context database designed for AI Agents, unifying all context ty
## System Overview
+<ArchitectureDiagram />
+
+<details>
+<summary>Text version</summary>
+
+```text
+CLI / SDK / HTTP client
+ |
+ HTTP Server
+ |
+ Service Layer
+ |
+ +------+------+----------------+
+ | | |
+Retrieval Sessions Resource / Skill import
+ | | |
+ | Memory extraction Parse / Semantic queues
+ | | |
+ +-------------+----------------+
+ |
+ VikingFS
+ / \
+ RAGFS Vector index
```
-┌────────────────────────────────────────────────────────────────────────────┐
-│ OpenViking System Architecture │
-├────────────────────────────────────────────────────────────────────────────┤
-│ │
-│ ┌─────────────┐ │
-│ │ Client │ │
-│ │ (OpenViking)│ │
-│ └──────┬──────┘ │
-│ │ delegates │
-│ ┌──────▼──────┐ │
-│ │ Service │ │
-│ │ Layer │ │
-│ └──────┬──────┘ │
-│ │ │
-│ ┌─────────────────────────┼─────────────────────────┐ │
-│ │ │ │ │
-│ ▼ ▼ ▼ │
-│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
-│ │ Retrieve │ │ Session │ │ Parse │ │
-│ │ (Context │ │ (Session │ │ (Context │ │
-│ │ Retrieval) │ │ Management)│ │ Extraction)│ │
-│ │ search/find │ │ add │ │ Doc parsing │ │
-│ │ Intent │ │ commit │ │ L0/L1/L2 │ │
-│ │ Rerank │ │ commit │ │ Tree build │ │
-│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
-│ │ │ │ │
-│ │ │ Memory extraction │ │
-│ │ ▼ │ │
-│ │ ┌─────────────┐ │ │
-│ │ │ Compressor │ │ │
-│ │ │ Compress/ │ │ │
-│ │ │ Deduplicate │ │ │
-│ │ └──────┬──────┘ │ │
-│ │ │ │ │
-│ └────────────────────────┼────────────────────────┘ │
-│ ▼ │
-│ ┌─────────────────────────────────────────────────────────────────┐ │
-│ │ Storage Layer │ │
-│ │ AGFS (File Content) + Vector Index │ │
-│ └─────────────────────────────────────────────────────────────────┘ │
-│ │
-└────────────────────────────────────────────────────────────────────────────┘
-```
+
+</details>
## Core Modules
+<div class="module-table">
+
| Module | Responsibility | Key Capabilities |
|--------|----------------|------------------|
-| **Client** | Unified entry | Provides all operation interfaces, delegates to Service layer |
+| **Client** | Unified entry | Sends supported SDK/CLI operations to the HTTP API |
| **Service** | Business logic | FSService, SearchService, SessionService, ResourceService, PackService, DebugService |
| **Retrieve** | Context retrieval | Intent analysis (IntentAnalyzer), global retrieval (HierarchicalRetriever), Rerank |
-| **Session** | Session management | Message recording, usage tracking, session compression, memory commit |
+| **Session** | Session management | Message recording, usage tracking, session archiving, triggering memory commit |
| **Parse** | Context extraction | Document parsing (PDF/MD/HTML), tree building (TreeBuilder), async semantic generation |
-| **Compressor** | Memory compression | Schema-driven memory extraction and LLM deduplication decisions |
-| **Storage** | Storage layer | VikingFS virtual filesystem, vector index, AGFS integration |
+| **Memory** | Memory extraction | Schema-driven extraction by MemoryType (ExtractLoop), LLM merge and deduplication written back as patches (MemoryUpdater); orchestrated by `SessionCompressorV3` |
+| **Storage** | Storage layer | VikingFS virtual filesystem, vector index, RAGFS integration |
+
+</div>
## Service Layer
-The Service layer decouples business logic from the transport layer, enabling reuse across HTTP Server and CLI:
+The Service layer decouples business logic from the transport layer. The CLI and SDKs reach it through the HTTP Server:
| Service | Responsibility | Key Methods |
|---------|----------------|-------------|
@@ -80,21 +66,22 @@ OpenViking uses a dual-layer storage architecture separating content from index
| Layer | Responsibility | Content |
|-------|----------------|---------|
-| **AGFS** | Content storage | L0/L1/L2 full content, multimedia files |
-| **Vector Index** | Index storage | URIs, vectors, metadata (no file content) |
+| **RAGFS** | Content storage | L0/L1/L2 full content, multimedia files |
+| **Vector Index** | Index storage | URIs, vectors, metadata, and text used for retrieval, including abstracts |
## Data Flow Overview
### Adding Context
```
-Input → Parser → TreeBuilder → AGFS → SemanticQueue → Vector Index
+Input → Parser → TreeBuilder → ResourceProcessor → RAGFS → SemanticQueue → Vector Index
```
-1. **Parser**: Parse documents, create file and directory structure (no LLM calls)
-2. **TreeBuilder**: Move temp directory to AGFS, enqueue for semantic processing
-3. **SemanticQueue**: Async bottom-up L0/L1 generation
-4. **Vector Index**: Build index for semantic search
+1. **Parser**: Parse source documents into files and directories; model use depends on the selected parser
+2. **TreeBuilder**: Resolve the target URI and retain temporary artifact references
+3. **ResourceProcessor**: Persist content in RAGFS and enqueue semantic processing
+4. **SemanticQueue**: Async bottom-up L0/L1 generation
+5. **Vector Index**: Build index for semantic search
### Retrieving Context
@@ -110,14 +97,14 @@ Query → Intent Analysis → Global Retrieval → Rerank → Results
### Session Commit
```
-Messages → Compress → Archive → Memory Extraction → Storage
+Messages → Archive Boundary → Archive → Memory Extraction → Storage
```
1. **Messages**: Accumulate conversation messages and usage records
-2. **Compress**: Keep recent N rounds, archive older messages
+2. **Archive Boundary**: Split archived and retained messages according to the commit parameters; by default, archive all current messages
3. **Archive**: Generate L0/L1 for history segments
4. **Memory Extraction**: Extract memories from messages according to the memory policy and MemoryType schemas
-5. **Storage**: Write to AGFS + vector index
+5. **Storage**: Write to RAGFS + vector index
## Deployment Mode
@@ -127,13 +114,16 @@ For team sharing, production deployment, and cross-language integration:
```python
# Python SDK connects to OpenViking Server
-client = SyncHTTPClient(url="http://localhost:1933", api_key="xxx")
+from openviking_sdk import SyncHTTPClient
+
+client = SyncHTTPClient(url="http://localhost:1933", api_key="your-key")
```
```bash
# Or use curl / any HTTP client
curl http://localhost:1933/api/v1/search/find \
- -H "X-API-Key: xxx" \
+ -H "X-API-Key: your-key" \
+ -H "Content-Type: application/json" \
-d '{"query": "how to use openviking"}'
```
@@ -146,10 +136,10 @@ curl http://localhost:1933/api/v1/search/find \
| Principle | Description |
|-----------|-------------|
-| **Pure Storage Layer** | Storage only handles AGFS operations and basic vector search; Rerank is in retrieval layer |
+| **Pure Storage Layer** | Storage only handles RAGFS operations and basic vector search; Rerank is in retrieval layer |
| **Three-Layer Information** | L0/L1/L2 enables progressive detail loading, saving token consumption |
| **Two-Stage Retrieval** | Vector search recalls candidates + Rerank improves accuracy |
-| **Single Data Source** | All content read from AGFS; vector index only stores references |
+| **Single Data Source** | RAGFS holds source files; vector records retain the text and metadata needed for retrieval |
## Related Documents
@@ -162,6 +152,6 @@ curl http://localhost:1933/api/v1/search/find \
- [Session Management](./08-session.md) - Session and memory management
- [Transaction Model](./09-transaction.md) - Write and consistency model
- [Data Encryption](./10-encryption.md) - At-rest encryption and key architecture
-- [Multi-Tenant](./11-multi-tenant.md) - Account / user / agent isolation model
+- [Multi-Tenant](./11-multi-tenant.md) - Account, user, and peer isolation model
- [Metrics](./12-metrics.md) - `/metrics` usage and key metric explanations
- [Privacy Configs and Skill Privacy Extraction/Restore](./13-privacy.md) - Versioning, placeholder extraction, and read-time restore
diff --git a/docs/en/concepts/02-context-types.md b/docs/en/concepts/02-context-types.md
index 39ff41adc..c799d1e13 100644
--- a/docs/en/concepts/02-context-types.md
+++ b/docs/en/concepts/02-context-types.md
@@ -1,14 +1,39 @@
# Context Types
-Based on a simplified mapping of human cognitive patterns and engineering considerations, OpenViking abstracts context into **three basic types: Resource, Memory, and Skill**, each serving different purposes in Agent applications.
+OpenViking manages three types of context: resources provide reference material, memories retain information from interactions, and skills describe how to carry out tasks.
## Overview
| Type | Purpose | Lifecycle | Initiative |
|------|---------|-----------|------------|
-| **Resource** | Knowledge and rules | Long-term, relatively static | User adds |
-| **Memory** | Agent's cognition | Long-term, dynamically updated | Agent records |
-| **Skill** | Declarable agent capability configuration (AgentDefinedContextType) | Long-term, static | User or system adds |
+| **Resource** | Knowledge and rules | Retained until updated or deleted | User adds |
+| **Memory** | Preferences, facts, and task experience | Long-term, dynamically updated | Extracted from sessions or recorded explicitly |
+| **Skill** | Task instructions and supporting resources | Long-term, updatable | User or system adds |
+
+## Example Setup
+
+The examples below use the synchronous Python SDK and an existing server. `add_resource` and `add_skill` take `wait=True` to block until processing finishes. Session commits return before memory extraction finishes and have no built-in wait, so this helper polls the commit task before dependent searches. A polling timeout does not cancel the server task.
+
+```python
+import time
+from openviking_sdk import SyncHTTPClient
+
+client = SyncHTTPClient(url="http://localhost:1933", api_key="your-key")
+
+
+def wait_for_task(task_id):
+ deadline = time.monotonic() + 300
+ while time.monotonic() < deadline:
+ task = client.get_task(task_id)
+ if task is None:
+ raise RuntimeError(f"Task {task_id} not found")
+ if task["status"] == "completed":
+ return task
+ if task["status"] in {"failed", "cancelled"}:
+ raise RuntimeError(task)
+ time.sleep(1)
+ raise TimeoutError(f"Task {task_id} is still running")
+```
## Resource
@@ -17,7 +42,7 @@ Resources are external knowledge that Agents can reference.
### Characteristics
- **User-driven**: Resource information actively added by users to supplement LLM knowledge, such as product manuals and code repositories
-- **Static content**: Content rarely changes after addition, usually modified by users
+- **Explicit updates**: Re-import changed content, or use a Watch to refresh supported remote sources
- **Structured storage**: Organized by project or topic in directory hierarchy, with multi-layer information extraction
### Examples
@@ -33,6 +58,8 @@ Resources are external knowledge that Agents can reference.
client.add_resource(
path="https://docs.example.com/api.pdf",
options={"reason": "API documentation"},
+ wait=True,
+ timeout=300,
)
# Search resources
@@ -76,17 +103,18 @@ The schema-defined `memories/tools/` and `memories/skills/` types are disabled.
from openviking_sdk import TextPart
# Memories are auto-extracted from sessions
-session_info = await client.create_session()
+session_info = client.create_session()
session = client.session(session_id=session_info["session_id"])
-await session.add_message(
+session.add_message(
role="user",
parts=[TextPart(text="I prefer dark mode")],
)
-commit = await session.commit() # Starts background memory extraction
-task = await client.get_task(task_id=commit["task_id"]) # Poll until task["status"] == "completed"
+commit = session.commit() # Starts background memory extraction
+if commit.get("task_id"):
+ wait_for_task(commit["task_id"])
# Search memories
-results = await client.find(
+results = client.find(
query="UI preferences",
target_uri="viking://~/memories/"
)
@@ -94,13 +122,13 @@ results = await client.find(
## Skill (Capabilities / AgentDefinedContextType)
-Skills are capabilities that Agents can invoke, belonging to the **AgentDefinedContextType** category. This includes traditional workflow definitions, communication endpoints, tool configurations, and payment capabilities. Their common characteristic is that they **define how an agent interacts with external systems**, with relatively static runtime definitions, but invocation experiences are updated in Memory.
+A Skill uses `SKILL.md` and supporting files to describe a task’s steps, constraints, and resources. An agent reads the Skill and carries out the task using its own tools. Execution experience can be stored separately as memory.
### Characteristics
-- **Defined capabilities**: Tool definitions for completing specific tasks
-- **Relatively static**: Skill definitions don't change at runtime, but usage memories related to tools are updated in memory
-- **Callable**: Agent decides when to use which skill
+- **Task instructions**: Steps and constraints for completing a type of work
+- **Maintainable**: Skill content can be updated; execution experience is stored separately
+- **Read on demand**: The agent selects skills for its current task
### Storage Location
@@ -111,7 +139,7 @@ viking://~/skills/{skill-name}/ # Default storage path
├── SKILL.md # L2: Skill definition
└── scripts # L2: Supporting implementation
-viking://agent/skills/{skill-name}/ # Override via --uri, public/shared (account global)
+viking://agent/skills/{skill-name}/ # Override via -p/--parent-auto-create, public/shared (account global)
├── .abstract.md # L0: Short description
├── .overview.md # L1: Directory overview (after generation)
├── SKILL.md # L2: Skill definition
@@ -120,7 +148,7 @@ viking://agent/skills/{skill-name}/ # Override via --uri, public/shared (acco
### AgentDefinedContextType Subtypes
-AgentDefinedContextType includes the following subtypes, all stored under the `viking://agent/` scope:
+The table below lists the design categories for shared capabilities. Skills are supported and installed in the user-private directory by default, with an option to use `viking://agent/skills/`. The other categories are planned and do not imply available APIs:
| Subtype | Location | Description |
|---------|----------|-------------|
@@ -133,37 +161,45 @@ AgentDefinedContextType includes the following subtypes, all stored under the `v
```python
# Add skill (defaults to viking://~/skills/)
-await client.add_skill(
+client.add_skill(
data={
"name": "search-web",
"description": "Search the web for information",
"content": "# search-web\n...",
},
+ wait=True,
+ timeout=300,
)
-# Write to global agent skills root (public/shared) via -p override
-ov skills add search-web -p viking://agent/skills
-
# Search user skills
-results = await client.find(
+results = client.find(
query="web search",
target_uri="viking://~/skills/"
)
# Search global agent skills
-results = await client.find(
+results = client.find(
query="web search",
target_uri="viking://agent/skills/",
)
```
+Install into the account-shared skills directory with the CLI (requires write access to that path):
+
+```bash
+ov skills add ./skills/search-web -p viking://agent/skills
+```
+
## Unified Search
-Based on Agent's needs, supports unified search across all three context types, providing comprehensive information:
+A single retrieval can return resources, memories, and skills within its search scope. The default includes the current user's space and shared resources; add `viking://agent/skills` explicitly to include shared Skills:
```python
# Search across all context types
-results = await client.find(query="user authentication")
+results = client.find(
+ query="user authentication",
+ target_uri=["viking://~", "viking://resources", "viking://agent/skills"],
+)
for context in results.get("memories", []):
print(f"Memory: {context['uri']}")
@@ -173,6 +209,12 @@ for context in results.get("skills", []):
print(f"Skill: {context['uri']}")
```
+Close the client when these operations are finished:
+
+```python
+client.close()
+```
+
## Related Documents
- [Architecture Overview](./01-architecture.md) - System architecture
diff --git a/docs/en/concepts/03-context-layers.md b/docs/en/concepts/03-context-layers.md
index d8ea0f56d..50585e053 100644
--- a/docs/en/concepts/03-context-layers.md
+++ b/docs/en/concepts/03-context-layers.md
@@ -1,6 +1,6 @@
# Context Layers (L0/L1/L2)
-OpenViking uses a three-layer information model to balance retrieval efficiency, navigation, and full-content fidelity.
+OpenViking uses abstracts and overviews to locate content, then loads the body when more detail is needed. These three layers are called L0, L1, and L2.
## Overview
@@ -50,11 +50,11 @@ This directory covers the primary API authentication methods.
overview = client.overview(uri="viking://resources/docs/auth")
```
-L0 is extracted from the L1 body: the Brief Description paragraph after the H1 title and before the first `##` heading. YAML frontmatter is not part of this extraction.
+When generating resource-directory summaries, L0 is extracted from the L1 body: the Brief Description paragraph after the H1 title and before the first `##` heading. YAML frontmatter is not part of this extraction.
## L2: Detail
-L2 is the original or fully parsed content, loaded only when needed and retaining its source format and structure.
+L2 is the original file or parsed body, loaded as needed. Its format depends on ingestion: a PDF may become Markdown, for example, so the source format is not necessarily preserved.
```python
content = client.read(uri="viking://resources/docs/auth/oauth.md")
@@ -146,7 +146,7 @@ API authentication guide covering OAuth 2.0, JWT tokens, and API keys.
- `unsampled_entries`: direct entries not sampled, with `sampled + unsampled = total`.
- `pending_child_changes`: direct-child change events not yet reflected in the current body (repeated changes to one child count separately).
-When the direct-entry count exceeds `semantic.overview_sample_limit` (32 by default), OpenViking uses deterministic, order-preserving stable sampling. Repeated refreshes of an unchanged tree choose the same sample, avoiding noisy body rewrites and Git diffs.
+When the direct-entry count exceeds `semantic.overview_sample_limit` (32 by default), OpenViking uses deterministic, order-preserving stable sampling. Repeated refreshes of an unchanged tree choose the same sample, reducing differences caused by sampling. Model-generated text can still change between refreshes.
`pending_child_changes > 0` means the body is still readable but is known to lag behind lower-level changes. A successful parent refresh resets the value to 0 as part of the new coverage metadata.
diff --git a/docs/en/concepts/04-viking-uri.md b/docs/en/concepts/04-viking-uri.md
index 757bda879..eae0fa9c7 100644
--- a/docs/en/concepts/04-viking-uri.md
+++ b/docs/en/concepts/04-viking-uri.md
@@ -18,7 +18,7 @@ viking://{scope}/{path}
|-------|-------------|-----------|------------|
| **resources** | Independent resources / objective knowledge | Long-term | Account global |
| **user** | User-level data, including sessions | Long-term / session lifetime | Current user |
-| **agent** | Agent capabilities and configuration (skills, endpoints, tools, payments, etc.) | Long-term | Account global |
+| **agent** | Account-shared skills and Agent configuration; endpoint, tool, and payment directories are planned | Long-term | Account global |
| **queue** | Processing queue | Temporary | Internal |
| **temp** | Temporary files | During parsing | Internal |
| **upload** | Temporary upload files | Temporary | Internal |
@@ -42,8 +42,7 @@ each caller.
public-scope URI is accepted.
- First segment only: `viking://resources/~/x` and `viking://user/alice/~/x` keep `~` as
a literal path segment.
-- Accepted, not advertised: `~` is not part of the public scope list, so the
- `Invalid scope ... Must be one of:` error message never mentions it.
+- `~` is a user-path alias, not a separate storage scope.
- Responses always echo the expanded canonical URI, never `viking://~`, and persisted
data (vector records, watch keys) stays canonical as well.
- Requires an authenticated request identity. Expansion uses that identity's effective
@@ -56,18 +55,14 @@ each caller.
## Initial Directory Structure
-Moving away from traditional flat database thinking, all context is organized as a filesystem. Agents no longer just find data through vector search, but can locate and browse data through deterministic paths and standard filesystem commands. Each context or directory is assigned a unique URI identifier string in the format viking://{scope}/{path}, allowing the system to precisely locate and access resources stored in different locations.
-
-## File IDs
-
-In addition to its URI, every file is automatically assigned a stable `id` that serves as the primary key of its vector record in VikingDB. The id is deterministically computed as `md5(f"{account_id}:{uri}")` for level 2 (regular file) records, and is returned by `stat()` and other metadata APIs. This allows callers to cross-reference vector index entries without a separate lookup. The id is scoped to the account and changes if the file is moved to a different URI (vector records are re-keyed during URI migration). Directories do not expose a single `id` because a directory may span multiple semantic levels (L0 abstract, L1 overview, L2), each with its own record.
+Context is organized into directories. With a known URI, an agent can list a directory or read a file directly. Otherwise, it can search first and read the returned paths.
```
viking://
├── user/
│ └── {user_id}/
-│ ├── profile.md # User profile
│ ├── memories/ # User memory storage
+│ │ └── profile.md # User profile
│ ├── resources/ # User-owned private resources
│ ├── skills/ # User skills
│ ├── peers/
@@ -92,6 +87,10 @@ viking://
└── resources/{project}/ # Resource workspace
```
+## File IDs
+
+In addition to its URI, every file is automatically assigned a stable `id` that serves as the primary key of its vector record in VikingDB. The id is deterministically computed as `md5(f"{account_id}:{uri}")` for level 2 (regular file) records, and is returned by `stat()` and other metadata APIs. This allows callers to cross-reference vector index entries without a separate lookup. The id is scoped to the account and changes if the file is moved to a different URI (vector records are re-keyed during URI migration). Directories do not expose a single `id` because a directory may span multiple semantic levels (L0 abstract, L1 overview), each with its own record.
+
## URI Examples
### Resources
@@ -115,7 +114,7 @@ viking://~/memories/entities/ # Entity memories
viking://~/memories/events/ # Event memories
viking://~/resources/ # Your private resources
viking://~/resources/docs/ # Your private resource directory
-viking://user/{user_id}/memories/ # Explicit user path (your own id; other ids need admin/root)
+viking://user/{user_id}/memories/ # Explicit user path (own user_id only, ADMIN included; ROOT can read across user spaces, but ROOT API keys cannot use data APIs in api_key mode)
```
`viking://resources/...` is the shared scope for the current account and supports per-directory or per-file [ACLs](./15-acl.md). `viking://user/{user}/resources/...` is private; move a resource into the shared scope to share it.
@@ -185,7 +184,7 @@ Viking URI supports path variables for dynamic path generation. This is especial
{namespace:key}
```
-- **namespace**: Variable provider namespace (e.g., `calendar`, `env`, `user`)
+- **namespace**: Variable provider namespace (`calendar` is built in)
- **key**: Variable name within the namespace
### Calendar Variables
@@ -203,12 +202,12 @@ The `calendar` namespace provides date-related variables:
| `{calendar:ym}` | Year/month | `2026/05` |
| `{calendar:quarter}` | Quarter (Q1-Q4) | `Q2` |
| `{calendar:yq}` | Year/quarter | `2026/Q2` |
-| `{calendar:week}` | ISO week number with leading zero | `18` |
-| `{calendar:yw}` | Year/ISO week | `2026/w18` |
+| `{calendar:week}` | ISO week number with leading zero | `19` |
+| `{calendar:yw}` | Year/ISO week | `2026/w19` |
### Usage Examples
-```python
+```text
# Organize emails by date
viking://resources/emails/{calendar:today}/inbox
# Renders to: viking://resources/emails/2026/05/07/inbox
@@ -238,7 +237,7 @@ Path variables are resolved **server-side** at the time of API execution. The CL
```bash
# Add today's emails, --parent-auto-create can be shortened to -p
-ov add-resource --parent-auto-create "viking://resources/emails/{calendar:today}/inbox" ./emails/*.eml
+ov add-resource --parent-auto-create "viking://resources/emails/{calendar:today}/inbox" ./emails/
# Read yesterday's log
ov read "viking://resources/logs/{calendar:yesterday}/app.log"
@@ -254,15 +253,15 @@ ov add-resource --parent-auto-create "viking://resources/reports/{calendar:ym}"
```
viking://
-├── resources/ # Independent resources
+├── resources/ # Account-shared reference material
│ └── {project}/
│ ├── .abstract.md
│ ├── .overview.md
│ └── {files...}
│
├── user/{user_id}/
-│ ├── profile.md # User basic info
│ ├── memories/
+│ │ ├── profile.md # User profile
│ │ ├── preferences/ # By topic
│ │ ├── entities/ # Each independent
│ │ └── events/ # Each independent
@@ -291,6 +290,8 @@ remain isolated by account.
## URI Operations
+The `VikingURI` helper below comes from the `openviking_cli` package installed with the server, not the standalone Python SDK public interface.
+
### Parsing
```python
@@ -315,6 +316,8 @@ parent = VikingURI(uri).parent.uri # viking://resources/docs
## API Usage
+The Python examples below use a configured `SyncHTTPClient` instance named `client`; see [Client Configuration](../configuration/02-client.md).
+
### Targeting Specific Scopes
```python
@@ -353,16 +356,16 @@ results = client.find(
```python
# List directory
-entries = await client.ls(uri="viking://resources/")
+entries = client.ls(uri="viking://resources/")
# Read file
-content = await client.read(uri="viking://resources/docs/api.md")
+content = client.read(uri="viking://resources/docs/api.md")
# Get abstract
-abstract = await client.abstract(uri="viking://resources/docs/")
+abstract = client.abstract(uri="viking://resources/docs/")
# Get overview
-overview = await client.overview(uri="viking://resources/docs/")
+overview = client.overview(uri="viking://resources/docs/")
```
## Special Files
@@ -391,24 +394,24 @@ Each directory may contain special files:
```python
# Add resources to the shared account resource scope
-await client.add_resource(url, to="viking://resources/project/")
+client.add_resource(url, to="viking://resources/project/")
# Add private resources to your own resource root
-await client.add_resource(path, parent="viking://~/resources/project/")
+client.add_resource(path, parent="viking://~/resources/project/")
# Skills are added to your own skills root by default
-await client.add_skill(skill) # default root: viking://~/skills/
+client.add_skill(skill) # default root: viking://~/skills/
+```
+
+Install into the account-shared skills directory with the CLI. This requires write access to that path:
-# Write to the global agent skills root (public/shared) via -p override
-ov skills add xxx -p viking://agent/skills/
+```bash
+ov skills add ./skills/search-web -p viking://agent/skills/
```
### Resources Scope Constraint
-The `resources` scope is for objective knowledge only (documents, code, specifications, papers, etc.).
-Storing non-knowledge data in `viking://resources/` is prohibited, including but not limited to:
-tool configurations, communication endpoint definitions, payment configurations, skill definitions, etc.
-Such data should use the `viking://agent/` scope.
+Store reference documents, code, and specifications under `resources`. To make content discoverable and loadable as a Skill, install it through the skill API under `viking://~/skills/` or the shared `viking://agent/skills/`. Uploading `SKILL.md` as an ordinary resource does not install a skill.
## Related Documents
diff --git a/docs/en/concepts/05-storage.md b/docs/en/concepts/05-storage.md
index eb372e555..ccedc225e 100644
--- a/docs/en/concepts/05-storage.md
+++ b/docs/en/concepts/05-storage.md
@@ -4,33 +4,21 @@ OpenViking uses a dual-layer storage architecture that separates content storage
## Overview
-```
-┌─────────────────────────────────────────┐
-│ VikingFS (URI Abstraction) │
-│ URI Mapping · Hierarchical Access │
-└────────────────┬────────────────────────┘
- ┌────────┴────────┐
- │ │
-┌───────▼────────┐ ┌─────▼───────────┐
-│ Vector Index │ │ AGFS │
-│ (Semantic │ │ (Content │
-│ Search) │ │ Storage) │
-└────────────────┘ └─────────────────┘
-```
+<StorageLayersDiagram />
## Dual-Layer Storage
| Layer | Responsibility | Content |
|-------|----------------|---------|
| **AGFS** | Content storage | L0/L1/L2 full content, multimedia files |
-| **Vector Index** | Index storage | URIs, vectors, metadata (no file content) |
+| **Vector Index** | Index storage | URIs, vectors, metadata, and retrieval text such as abstracts |
### Design Benefits
-1. **Clear responsibilities**: Vector index handles retrieval, AGFS handles storage
-2. **Memory optimization**: Vector index doesn't store file content, saving memory
-3. **Single data source**: All content read from AGFS; vector index only stores references
-4. **Independent scaling**: Vector index and AGFS can scale separately
+AGFS holds the source files. The vector index stores what retrieval needs, including URI references, vectors, metadata, and abstracts. Memory records store their body text in the abstract field, capped at 50,000 bytes, so the vector index can contain readable text.
+
+The two backends can be configured separately. File APIs read from AGFS, while retrieval can return indexed text without rereading every source file. Backups and access controls should account for both stores.
+
Note: AGFS has been rewritten as a Rust implementation (RAGFS)
## VikingFS Virtual Filesystem
@@ -83,7 +71,7 @@ For the conceptual model, see [Multi-Write Storage](./14-multi-write-storage.md)
### Directory Structure
-Each context directory follows a unified structure:
+A directory after semantic processing commonly has the following structure. Summary sidecars may be absent before processing or when only L0 has been created:
```
viking://resources/docs/auth/
@@ -129,6 +117,8 @@ is insufficient.
### Index Strategy
+Example index metadata; available settings depend on the selected vector backend:
+
```python
index_meta = {
"IndexType": "flat_hybrid", # Hybrid index
@@ -147,19 +137,19 @@ index_meta = {
## Vector Synchronization
-VikingFS automatically maintains consistency between vector index and AGFS.
+Filesystem operations coordinate changes to files and vector records. Summary refreshes can continue asynchronously, and failures can leave partial changes; see the [filesystem API](../api/03-filesystem.md) for operation-specific behavior. The examples below use an initialized synchronous Python SDK client.
### Delete Sync
```python
-viking_fs.rm("viking://resources/docs/auth", recursive=True)
+client.rm("viking://resources/docs/auth", recursive=True)
# Automatically deletes all records with this URI prefix from vector index
```
### Move Sync
```python
-viking_fs.mv(
+client.mv(
"viking://resources/docs/auth",
"viking://resources/docs/authentication"
)
diff --git a/docs/en/concepts/06-extraction.md b/docs/en/concepts/06-extraction.md
index ac46e334b..f09714677 100644
--- a/docs/en/concepts/06-extraction.md
+++ b/docs/en/concepts/06-extraction.md
@@ -1,18 +1,12 @@
# Context Extraction
-OpenViking uses a three-layer async architecture for document parsing and context extraction.
+Resource ingestion proceeds through parsing, target-path resolution, content persistence, semantic generation, and embedding. Memories and skills have their own entry points and also use semantic processing and vector indexes.
## Overview
-```
-Input File → Parser → TreeBuilder → SemanticQueue → Vector Index
- ↓ ↓ ↓
- Parse & Move Files L0/L1 Generation
- Convert Queue Semantic (LLM Async)
- (No LLM)
-```
+<IngestionPipelineDiagram />
-**Design Principle**: Parsing and semantics are separated. Parser doesn't call LLM; semantic generation is async.
+Parsing and directory-summary generation are separate stages. Individual parsers may call models or external parsing services, so parsing is not universally LLM-free. Import APIs normally return a task ID before processing finishes; wait for that task before dependent searches. See [Task Management](../api/17-tasks.md).
## Parser
@@ -27,70 +21,40 @@ Parser handles document format conversion and structuring, creating file structu
| PDF | PDFParser | .pdf | Supported |
| HTML | HTMLParser | .html, .htm | Supported |
| Code | CodeRepositoryParser | .py, .js, .go, etc. | Respects `.gitignore` and ignores common non-code directories |
-| Image | ImageParser | .png, .jpg, etc. | |
-| Video | VideoParser | .mp4, .avi, etc. | |
-| Audio | AudioParser | .mp3, .wav, etc. | |
+| Image | ImageParser | .png, .jpg, etc. | Supported; depends on media configuration |
+| Video | VideoParser | .mp4, .avi, .mov, .mkv, .webm, .flv, .wmv, .ts (MPEG-TS content only) | Supported; depends on media configuration |
+| Audio | AudioParser | .mp3, .wav, .ogg, .flac, .aac, .m4a, .opus, .ac3 | Supported; depends on media configuration |
-### Core Flow (Document Example)
+### Parse Artifacts
-```python
-# 1. Parse file
-parse_result = registry.parse("/path/to/doc.md")
+The internal `registry.parse()` interface is asynchronous and returns a `ParseResult`. Artifacts can live in a local temporary directory or AGFS; `temp_dir_path` is not necessarily a Viking URI, and `artifact_ref` identifies the storage backend. These are server internals; clients should use the resource import API.
-# 2. Returns temp directory URI
-parse_result.temp_dir_path # viking://temp/abc123
-```
+| Field | Meaning |
+| --- | --- |
+| `root` | Root of the parsed resource tree |
+| `temp_dir_path` | Temporary artifact location |
+| `artifact_ref` | Artifact backend and root path |
+| `source_format` / `parser_name` | Source format and parser name |
+| `parse_time` | Parsing duration in seconds |
+| `meta` / `warnings` | Metadata and parsing warnings |
-### Smart Splitting
+### Document Splitting
-```
-If document_tokens <= 1024:
- → Save as single file
-Else:
- → Split by headers
- → Section < 512 tokens → Merge
- → Section > 1024 tokens → Create subdirectory
-```
+The Markdown parser organizes sections by headings and size, merging short sections and splitting oversized content. Defaults are `max_section_size=2048` tokens and `max_section_chars=6000` characters. The configuration also declares `section_size_flexibility=0.3`, but the current Markdown splitting code does not use it. A single oversized table row may remain intact. These are splitting targets, not absolute limits on every output file.
-### Return Result
-
-```python
-ParseResult(
- temp_dir_path: str, # Temp directory URI
- source_format: str, # pdf/markdown/html
- parser_name: str, # Parser name
- parse_time: float, # Duration (seconds)
- meta: Dict, # Metadata
-)
-```
+Behavior varies by parser and `parse_mode`. Use `parse_mode="no_split"` for supported imports when a single file is needed; see [Resource Management](../api/02-resources.md).
## TreeBuilder
-TreeBuilder moves temp directory to AGFS and queues semantic processing.
+`TreeBuilder.finalize_from_temp()` inspects parse artifacts and resolves the final URI, returning a `BuildingTree` with root and temporary-location metadata. It does not copy files, clean up artifacts, or enqueue semantic work itself.
-### Core Flow
+### Subsequent Ingestion Steps
-```python
-building_tree = tree_builder.finalize_from_temp(
- temp_dir_path="viking://temp/abc123",
- scope="resources", # resources/user
-)
-```
-
-### 5-Phase Processing
-
-1. **Find document root**: Ensure exactly 1 subdirectory in temp
-2. **Determine target URI**: Map base URI by scope
-3. **Recursively move directory tree**: Copy all files to AGFS
-4. **Clean up temp directory**: Delete temp files
-5. **Queue semantic generation**: Submit SemanticMsg to queue
-
-### URI Mapping
+1. Resolve and validate the target URI from the artifact and `to` or `parent`.
+2. The ingestion processor persists content, handles resource locks, and updates existing content.
+3. It schedules semantic processing and embedding, and cleans up temporary data through the artifact's backend.
-| scope | Base URI |
-|-------|----------|
-| resources | `viking://resources` |
-| user | `viking://user` |
+The default resource root is `viking://resources`. For personal resources, explicitly target `viking://~/resources/...`; `viking://user` is a container of user spaces, not your resource root.
## SemanticQueue
@@ -98,14 +62,16 @@ SemanticQueue handles async L0/L1 generation and vectorization.
### Message Structure
-```python
-SemanticMsg(
- id: str, # UUID
- uri: str, # Directory URI
- context_type: str, # resource/memory/skill
- status: str, # pending/processing/completed
-)
-```
+Selected internal message fields follow; this is not a client request format:
+
+| Field | Meaning |
+| --- | --- |
+| `id` | Message UUID |
+| `uri` | Directory to process |
+| `context_type` | resource, memory, skill, or session |
+| `status` | Queue processing status |
+| `recursive` | Whether to process child directories |
+| `propagate_to_parent` | Whether completion may schedule a parent refresh |
### Processing Flow (Bottom-up)
@@ -115,7 +81,7 @@ Leaf directories → Parent directories → Root
### Single Directory Processing Steps
-1. **Concurrent file summary generation**: Limited to 10 concurrent
+1. **Concurrent file summary generation**: Concurrency is limited by `vlm.max_concurrent`
2. **Collect child directory abstracts**: Read generated .abstract.md
3. **Generate .overview.md**: LLM generates L1 overview
4. **Extract .abstract.md**: Extract L0 from overview
@@ -132,10 +98,10 @@ Each generation records direct-child coverage and uses stable sampling above `se
| Parameter | Default | Description |
|-----------|---------|-------------|
-| `max_concurrent_llm` | 10 | Concurrent LLM calls |
-| `max_images_per_call` | 10 | Max images per VLM call |
-| `max_sections_per_call` | 20 | Max sections per VLM call |
-| `overview_sample_limit` | 32 | Maximum direct-child sample used for one directory summary |
+| `vlm.max_concurrent` | 32 | Semantic-processing concurrency, passed to `SemanticProcessor.max_concurrent_llm` |
+| `max_images_per_call` | 10 | `VLMProcessor` constructor parameter: images per call; not an `ov.conf` field |
+| `max_sections_per_call` | 20 | `VLMProcessor` constructor parameter: sections per call; not an `ov.conf` field |
+| `semantic.overview_sample_limit` | 32 | Maximum direct-child sample used for one directory summary |
## Code Skeleton Extraction
@@ -161,16 +127,17 @@ This routing applies to short and long code files alike.
| Phase | Resource | Memory | Skill |
|-------|----------|--------|-------|
-| **Parser** | Common flow | Common flow | Common flow |
+| **Entry point** | Resource parser | Session extraction and memory updates | Skill import |
| **Base URI** | `viking://resources` | `viking://~/memories` | `viking://~/skills` |
-| **TreeBuilder scope** | resources | user | user |
| **SemanticMsg type** | resource | memory | skill |
+The examples below use a configured synchronous Python SDK client named `client`.
+
### Resource Extraction
```python
# Add resource
-await client.add_resource(
+client.add_resource(
path="/path/to/doc.pdf",
options={"reason": "API documentation"},
)
@@ -182,10 +149,10 @@ await client.add_resource(
```python
# Add skill
-await client.add_skill(
+client.add_skill(
data={
"name": "search-web",
- "content": "# search-web\\n...",
+ "content": "# search-web\n...",
},
)
@@ -196,7 +163,7 @@ await client.add_skill(
```python
# Memory auto-extracted from session
-await session.commit()
+client.commit_session(session_id)
# Flow: SessionCompressorV3 → ExtractLoop → MemoryUpdater → SemanticQueue
```
diff --git a/docs/en/concepts/07-retrieval.md b/docs/en/concepts/07-retrieval.md
index e38a3af29..38e9535ca 100644
--- a/docs/en/concepts/07-retrieval.md
+++ b/docs/en/concepts/07-retrieval.md
@@ -12,24 +12,32 @@ Query → Intent Analysis (optional) → Global Vector Search → Rerank (option
| Feature | find() | search() |
|---------|--------|----------|
-| Session context | Not needed | Required |
-| Intent analysis | Not used | LLM analysis |
-| Query count | Single query | 0-5 TypedQueries |
-| Latency | Low | Higher |
+| Session context | Not used | Optional; used when `session_id` is supplied |
+| Intent analysis | Not used | Uses an LLM when session content exists and intent analysis is enabled |
+| Query count | Single query | zero or more TypedQueries |
+| Latency | Usually lower | Depends on intent analysis, query count, and reranking |
| Use case | Simple queries | Complex tasks |
+For `search`, `limit` applies to each planned query. Merged results may exceed it, and the current implementation does not guarantee deduplication across queries.
+
### Usage Examples
+These examples use a configured synchronous Python SDK client named `client`.
+
```python
# find(): Simple query
-results = await client.find(
+results = client.find(
query="OAuth authentication",
target_uri="viking://resources/",
)
# search(): Complex task (needs session context)
-session_info = await client.create_session()
-results = await client.search(
+session_info = client.create_session()
+client.add_message(
+ session_id=session_info["session_id"], role="user",
+ content="We are designing the OAuth login flow for this project.",
+)
+results = client.search(
query="Help me create an RFC document",
session_id=session_info["session_id"],
)
@@ -37,7 +45,7 @@ results = await client.search(
## Intent Analysis
-IntentAnalyzer uses LLM to analyze query intent and generate 0-5 TypedQueries. The model used for this stage is separately configurable via the [`query_planner`](../guides/01-configuration.md#query-planner) config, falling back to `vlm` when unset.
+When `retrieval.enable_intent=true` and the session contains a summary or messages, IntentAnalyzer uses an LLM to analyze query intent and generate zero or more TypedQueries. The model used for this stage is separately configurable via the [`query_planner`](../guides/01-configuration.md#query-planner) config, falling back to `vlm` when unset.
### Input
@@ -99,6 +107,8 @@ See [rerank configuration](../guides/01-configuration.md#rerank) for supported p
## Retrieval Results
+The following excerpts show server-internal types. For HTTP/SDK response fields, see the [Retrieval API](../api/06-retrieval.md).
+
### MatchedContext
```python
@@ -106,7 +116,7 @@ See [rerank configuration](../guides/01-configuration.md#rerank) for supported p
class MatchedContext:
uri: str # Resource URI
context_type: ContextType
- is_leaf: bool # Whether file
+ level: int # 0=abstract, 1=overview, 2=detail
abstract: str # L0 abstract
score: float # Final score
```
diff --git a/docs/en/concepts/08-session.md b/docs/en/concepts/08-session.md
index 22be3e03e..4816ca8c1 100644
--- a/docs/en/concepts/08-session.md
+++ b/docs/en/concepts/08-session.md
@@ -10,6 +10,9 @@ Getting a session by ID does not create it. Create the session first, then use
`client.session(session_id=...)` to append messages or commit it.
```python
+from openviking_sdk import SyncHTTPClient
+
+client = SyncHTTPClient(url="http://localhost:1933", api_key="your-key")
session_info = client.create_session(session_id="chat_001")
session = client.session(session_id=session_info["session_id"])
session.add_message(role="user", content="...")
@@ -22,7 +25,7 @@ session.commit()
|--------|-------------|
| `add_message(role, content=None, parts=None, options=None, peer_id=None)` | Add message |
| `commit()` | Commit: archive (sync) + summary generation and memory extraction (async background) |
-| `get_task(task_id)` | Query background task status |
+| `client.get_task(task_id)` | Query background task status |
### add_message
@@ -66,10 +69,11 @@ result = session.commit()
# "archived": True
# }
-# Poll background task progress
-task = client.get_task(task_id=result["task_id"])
-# task["status"]: "pending" | "running" | "completed" | "failed"
-# sum(task["result"]["memories_extracted"].values()): 3
+# A task is created only for an archive; no-op commits return skipped and task_id: null.
+task_id = result.get("task_id")
+if task_id:
+ task = client.get_task(task_id=task_id)
+ print(task) # One status check does not mean the task has finished.
```
## Message Structure
@@ -100,11 +104,11 @@ class Message:
commit() executes in two phases:
-**Phase 1 (synchronous, returns immediately)**:
-1. Increment compression_index
-2. Write messages to archive directory (`messages.jsonl`)
-3. Clear current messages list
-4. Return `task_id`
+**Phase 1 (archive preparation within the request)**:
+1. Allocate the archive number and split archived and retained messages under a path lock
+2. Persist archive messages (`messages.jsonl`) and enqueue processing in the durable queue
+3. Update the live message list; default commits archive all messages, while retention parameters can keep recent messages or turns
+4. Return `task_id` when an archive was created, to track background processing
**Phase 2 (asynchronous background)**:
5. Generate structured summary (LLM) → write `.abstract.md` and `.overview.md`
@@ -144,29 +148,17 @@ Within `memory_policy.memory_types`, `experiences` enables the complete Agent Ev
### Extraction Flow
-```
-Messages → LLM Extract → Candidate Memories
- ↓
-Vector Pre-filter → Find Similar Memories
- ↓
-LLM Dedup Decision → candidate(skip/create/none) + item(merge/delete)
- ↓
-Write to AGFS → Vectorize
-```
+<MemoryExtractionDiagram />
+
+Existing memories inform extraction. An update can merge or edit existing files, create files, or delete them. Memory schemas, write permissions, and output validation constrain these operations; a commit does not necessarily add a memory. Subsequent training produces trajectories, experiences, or enabled session skills only when extraction yields a case.
-### Dedup Decisions
+### Checking Update Results
-| Level | Decision | Description |
-|------|----------|-------------|
-| Candidate | `skip` | Candidate is duplicate, skip and do nothing |
-| Candidate | `create` | Create candidate memory (optionally delete conflicting existing memories first) |
-| Candidate | `none` | Do not create candidate; resolve existing memories by item decisions |
-| Per-existing item | `merge` | Merge candidate content into specified existing memory |
-| Per-existing item | `delete` | Delete specified conflicting existing memory |
+Wait for the commit task, then inspect `memory_diff.json`. Additions, updates, and deletions describe actual file changes. `skipped_operations` records proposed operations skipped by validation or policy. No-op updates are excluded from effective changes in the diff; metadata changes can still produce an update when the body is unchanged.
## Memory Diff
-Each `session.commit()` writes a `memory_diff.json` to the archive directory, recording all memory changes from that commit for auditing and rollback.
+Background processing writes `memory_diff.json` to the archive directory, recording memory changes for auditing and review. It may not exist when `task_id` is returned; check task completion as described in the [Sessions API](../api/05-sessions.md).
```json
{
@@ -175,23 +167,23 @@ Each `session.commit()` writes a `memory_diff.json` to the archive directory, re
"operations": {
"adds": [
{
- "uri": "memory/user/xxx/identity.md",
+ "uri": "viking://user/alice/memories/identity.md",
"memory_type": "identity",
"after": "Newly created file content"
}
],
"updates": [
{
- "uri": "memory/user/xxx/context/project.md",
- "memory_type": "context",
+ "uri": "viking://user/alice/memories/entities/project.md",
+ "memory_type": "entities",
"before": "Content before modification",
"after": "Content after modification"
}
],
"deletes": [
{
- "uri": "memory/user/xxx/context/old.md",
- "memory_type": "context",
+ "uri": "viking://user/alice/memories/entities/old.md",
+ "memory_type": "entities",
"deleted_content": "Deleted file content"
}
]
diff --git a/docs/en/concepts/09-transaction.md b/docs/en/concepts/09-transaction.md
index 1178548d2..7e59fe5a0 100644
--- a/docs/en/concepts/09-transaction.md
+++ b/docs/en/concepts/09-transaction.md
@@ -1,10 +1,10 @@
# Path Locks and Crash Recovery
-OpenViking uses two simple primitives — **path locks** and **persistent queue recovery** — to protect the consistency of core write operations (`rm`, `mv`, `add_resource`, `session.commit`), coordinating concurrent writes and resuming queued session work after a process restart. These primitives do not form an atomic transaction across VikingFS, VectorDB, and QueueManager.
+OpenViking uses two mechanisms — **path locks** and **persistent queue recovery** — to protect the consistency of core write operations (`rm`, `mv`, `add_resource`, `session.commit`), coordinating concurrent writes and resuming queued session work after a process restart. These primitives do not form an atomic transaction across VikingFS, VectorDB, and QueueManager.
## Design Philosophy
-OpenViking is a context database where FS is the source of truth and VectorDB is a derived index. A lost index can be rebuilt from source data, but lost source data is unrecoverable. Therefore:
+OpenViking is a context database where FS is the source of truth and VectorDB is a derived index. An index can generally be rebuilt from retained source data; recovering lost source data depends on backups. Therefore:
> **Better to miss a search result than to return a bad one.**
@@ -12,70 +12,34 @@ OpenViking is a context database where FS is the source of truth and VectorDB is
1. **Write-exclusive**: Different owners participating in the protocol cannot hold conflicting path locks simultaneously
2. **On by default**: Protected writes acquire locks by default; ordinary reads and low-level mkdir do not automatically acquire locks
-3. **Lock as protection**: LockContext acquires locks on entry, releases on exit — no undo/journal/commit semantics
-4. **Only session_memory needs crash recovery**: a persistent `session_commit` queue resumes Phase 2 after a process crash
-5. **Queue operations run outside locks**: SemanticQueue/EmbeddingQueue enqueue operations are idempotent and retriable
+3. **Concurrency protection**: Runtime leases do not provide cross-store undo/journal/commit semantics
+4. **Persistent task recovery**: the `SessionCommit` queue resumes session Phase 2; the corresponding persistent queues recover derived resource processing
+5. **Locks and queues work together**: A business path may enqueue while holding a lock and hand off its lease; retries and deduplication depend on each task protocol, not a universal enqueue guarantee
## Architecture
-```
-Service Layer (rm / mv / add_resource / session.commit)
- |
- v
-+--[LockContext async context manager]--+
-| |
-| 1. Create LockHandle |
-| 2. Acquire path lock (poll+timeout) |
-| 3. Execute operations (FS+VectorDB) |
-| 4. Release lock |
-| |
-| On exception: auto-release lock, |
-| exception propagates unchanged |
-+---------------------------------------+
- |
- v
-Storage Layer (VikingFS, VectorDB, QueueManager)
-```
+<PathLockDiagram />
## Two Core Components
-### Component 1: PathLockEngine + LockManager + LockContext (Path Lock System)
+### Component 1: Path Lock System
-**PathLockEngine** implements provider-backed distributed locks with two lock types — EXACT and TREE — using ownership tokens to prevent TOCTOU races and automatic stale lock detection and cleanup. The default provider stores lock files in AGFS; the cache provider stores tokens in Redis.
+Rust RAGFS `PathLockManager` and its Provider own runtime locks. Python services acquire, reuse, hand off, and release leases through the binding. The later `LockContext` examples illustrate the lifecycle; they are not current Python SDK or runtime class interfaces.
-**LockHandle** is a lightweight lock holder token:
-
-```python
-@dataclass
-class LockHandle:
- id: str # Unique ID used to generate fencing tokens
- locks: list[str] # Provider handles: lock file paths or logical paths
- created_at: float # Handle creation time
- last_active_at: float # Last successful acquire/refresh time
-```
+- **Scope**: EXACT protects one path; TREE protects a path and its subtree.
+- **Provider**: The default filesystem provider stores lock files; cache uses Redis for atomic token checks and writes; memory coordinates only within one process.
+- **Lease**: Records the owner, covered paths, token locations, and refresh state. Background tasks can receive an explicit lease handoff.
+- **Failure handling**: Release validates ownership, and later acquisitions detect stale tokens. Releasing a lock does not undo persisted changes.
-**LockManager** is a global singleton managing lock lifecycle:
-- Creates/releases LockHandles
-- Background cleanup of leaked locks (in-process safety net)
-- Relies on QueueManager to resume persisted `session_commit` Phase 2 work after startup
+### Component 2: Persistent `SessionCommit` Queue (Crash Recovery)
-**LockContext** is an async context manager encapsulating the lock/unlock lifecycle:
-
-```python
-# Conceptual example: production path locks are acquired inside the Rust ragfs layer.
-async with LockContext(lock_manager, [path], lock_mode="exact") as handle:
- # Perform operations under lock protection
- ...
-# Lock automatically released on exit (including exceptions)
-```
-
-### Component 2: Persistent `session_commit` Queue (Crash Recovery)
+The queue name is `SessionCommit`; the task type is `session_commit`.
`session.commit` Phase 2 no longer uses a standalone redo log. Phase 1 persists archive metadata first,
then enqueues a durable `SessionCommitMsg`; after restart, QueueManager resumes any leftover
`session_commit` jobs and continues Phase 2.
-Memory extraction is idempotent — re-extracting from the same archive produces the same result.
+Recovery uses the saved archive and processing state. Memory extraction includes model calls; repeating it does not guarantee identical generated text.
## Consistency Issues and Solutions
@@ -110,8 +74,8 @@ retry is also safe.
|---------|----------|
| File moved to new path but index points to old path -> search returns old path (doesn't exist) | Copy first then update index; clean up copy on failure |
-**Locking strategy** (handled automatically via `lock_mode="mv"`):
-- Moving a **directory**: TreeLock on the source path and ExactPathLock on the destination path
+**Locking strategy** (acquired by the public `VikingFS.mv` implementation):
+- Moving a **directory**: TreeLock on both source and destination subtrees
- Moving a **file**: EXACT lock on both source path and destination path
Operation flow:
@@ -122,7 +86,7 @@ Operation flow:
3. Copy to new location (source still intact, safe)
4. If directory, remove the lock file carried over by cp into the copy
5. Update VectorDB URIs
- - Failure -> clean up copy, source and old index intact, consistent state
+ - Failure -> attempt target cleanup; index updates or cleanup may be partial, so inspect before retrying
6. Delete source
7. Release lock
```
@@ -131,7 +95,7 @@ Operation flow:
| Problem | Solution |
|---------|----------|
-| File moved from temp to final directory, then crash -> file exists but never searchable | Two separate paths for first-time add vs incremental update |
+| File moved from temp to final directory, then crash -> file exists but never searchable | Commit the content plan first; persistent queues complete derived indexing |
| Resource already on disk but rm deletes it while semantic processing / vectorization is still running -> wasted work | Lifecycle TreeLock held from finalization through processing completion |
**First-time add and incremental update** use the same planned commit path:
@@ -162,10 +126,7 @@ Automatic naming is handled by the resource layer, not the lock service:
try `_1`, `_2`, and so on. Only a non-existing candidate attempts `TreeLock`,
without waiting. If that candidate is busy, the next suffix is tried.
-**Server restart recovery**: `SemanticMsg` and its `SemanticPlan` are persisted
-in QueueFS. On restart, `SemanticProcessor` detects that the
-`lifecycle_lock_handle_id` handle is missing from the in-memory LockManager and
-re-acquires a TreeLock before continuing derived work.
+**Server restart recovery**: Persistent QueueFS backends retain `SemanticMsg` and `SemanticPlan`. Semantic processing attempts to adopt `lock_handoff`; recoverable stale handoffs reacquire the original `covered_paths`. Conflicts or unrecoverable handoff errors can still fail processing, so check the task state.
### Derived Semantic Files (.abstract.md / .overview.md)
@@ -189,38 +150,34 @@ hold separate ExactPathLocks for the two source files. Refreshing `preferences/.
### session.commit()
-| Problem | Solution |
-|---------|----------|
-| Messages cleared but archive not written -> conversation data lost | Phase 1 without lock (incomplete archive has no side effects) + Phase 2 with a persistent `session_commit` queue |
+Phase 1 holds an EXACT lock on the session root to establish the commit boundary. Summary generation and memory extraction run in the background, outside that boundary lock, because model calls have unpredictable latency:
-LLM calls have unpredictable latency (5s~60s+) and cannot be inside a lock-holding operation. The design splits into two phases:
-
-```
-Phase 1 — Archive (no lock):
- 1. Generate archive summary (LLM)
- 2. Write archive (history/archive_N/messages.jsonl + summaries)
- 3. Clear messages.jsonl
- 4. Clear in-memory message list
-
-Phase 2 — Memory extraction + write (persistent `session_commit` queue):
- 1. Persist archive metadata and enqueue `SessionCommitMsg`
- 2. Extract memories from archived messages (LLM)
- 3. Write current message state
- 4. Directly enqueue SemanticQueue
+```text
+Phase 1: prepare and publish the archive under lock
+ 1. Read messages and policy; allocate an archive number and split archived/retained messages
+ 2. Persist archive messages and metadata
+ 3. Enqueue SessionCommitMsg and create the task
+ 4. Persist retained live messages, session metadata, and the phase1 ready marker
+ 5. Release the lock; return task_id, or skipped if nothing was archived
+
+Phase 2: process the published archive in the background
+ 1. Read archived messages and generate summaries
+ 2. Extract/update memories and schedule derived processing
+ 3. Write memory_diff.json and the completion marker
```
-**Crash recovery analysis**:
+**Crash recovery checks**:
-| Failure moment | State | Recovery action |
-|------------|-------|----------------|
-| During Phase 1 archive write | Queue not published yet | Incomplete archive; next commit scans history/ for index, unaffected |
-| Phase 1 archive complete but messages not cleared | Queue not published yet | Archive complete + messages still present = redundant but safe |
-| During Phase 2 memory extraction/write | `session_commit` job still persisted | After restart: resume the job and recover Phase 2 from archive |
-| Phase 2 complete | Archive marked complete | No recovery needed |
+| Failure point | What to check |
+| --- | --- |
+| Phase 1 not yet published | Preparation may be partial; inspect archive state and live messages rather than treating directory existence as success |
+| Queued but not yet marked ready | Background processing must check Phase 1 state; error paths attempt to mark failure, so task existence alone does not prove success |
+| During Phase 2 extraction or writes | A persistent queue can resume processing, but model retries need not produce identical text |
+| After Phase 2 | Check the terminal task state, archive completion marker, and actual memory changes |
## LockContext
-`LockContext` is an **async** context manager that encapsulates lock acquisition and release:
+This pseudocode illustrates acquiring, using, and releasing a lock. The runtime uses RAGFS lease APIs; `LockContext` cannot be imported from the current Python SDK:
```python
# Conceptual example: production path locks are acquired inside the Rust ragfs layer.
@@ -247,9 +204,9 @@ async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_path=dst):
|-----------|----------|----------|
| `exact` | File writes, single-file delete, sidecar writeback | Lock the specified path; conflicts with same-path locks and ancestor TreeLocks |
| `tree` | Directory delete, resource lifecycle, directory-level protection | Lock the subtree root; conflicts with same-path locks, descendant locks, and ancestor TreeLocks |
-| `mv` | Move operations | Directory move: source TreeLock + destination ExactPathLock; File move: ExactPathLock on both source and destination (controlled by `src_is_dir`) |
+| `mv` | Move operations | Directory move: source TreeLock + destination TreeLock; File move: ExactPathLock on both source and destination (controlled by `src_is_dir`) |
-**Exception handling**: `__aexit__` always releases locks and does not swallow exceptions. Lock acquisition failure raises `LockAcquisitionError`.
+**Exception handling**: Business paths release or hand off leases in `finally`. Conflicts surface as busy/lock-acquisition errors. Tokens left by release failure or process exit rely on stale cleanup.
## Lock Types (EXACT vs TREE)
@@ -279,7 +236,7 @@ The following conflicts apply to different owners within the same provider scope
`Tree(/docs/a.md)` does not expand to `Tree(/docs)`. Likewise, Exact on a directory name does not protect its descendants; recursive deletion needs Tree.
-Locks coordinate participating operations. The low-level `PathLockWrappedFS` uses Exact for create, write, truncate, and non-recursive remove, and Tree for remove_all. File rename uses Exact on both paths; directory rename uses Tree on the source and Exact on the destination. Read, stat, directory listing, and mkdir pass through, though higher layers may hold their own locks. The operating system does not block I/O that bypasses this protocol.
+Locks coordinate participating operations. The low-level `PathLockWrappedFS` uses Exact for create, write, truncate, and non-recursive remove, and Tree for remove_all. File rename uses Exact on both paths; low-level directory rename uses Tree on the source and Exact on the destination. Public directory `mv` additionally covers both subtrees. Read, stat, directory listing, and mkdir pass through, though higher layers may hold their own locks. The operating system does not block I/O that bypasses this protocol.
## Lock Mechanism
@@ -409,7 +366,7 @@ After startup, QueueManager resumes persisted `session_commit` jobs:
| session_memory extraction crash | Recover Phase 2 from archive and continue the `session_commit` job |
| Crash while holding lock | Provider token remains; stale detection auto-cleans on a later matching acquisition (default 30s expiry) |
| Crash after enqueue, before worker processes | QueueFS SQLite persistence; worker auto-pulls after restart |
-| Orphan index | Cleaned on L2 on-demand load |
+| Orphan index | `rm` attempts related vector cleanup even when the target is missing |
### Defense Summary
@@ -417,9 +374,9 @@ After startup, QueueManager resumes persisted `session_commit` jobs:
|-----------------|--------|-----------------|
| Crash during operation | Lock auto-expires + stale detection | Next acquisition of same path lock |
| Crash during add_resource semantic processing | Lifecycle lock expires + SemanticProcessor re-acquires on restart | Worker restart |
-| Crash during session.commit Phase 2 | Persistent `session_commit` queue + resumed consumption | On restart |
+| Crash during session.commit Phase 2 | Persistent `SessionCommit` queue + resumed consumption | On restart |
| Crash after enqueue, before worker | QueueFS SQLite persistence | Worker restart |
-| Orphan index | L2 on-demand load cleanup | When user accesses |
+| Orphan index | `rm` attempts related vector cleanup even if the target is missing | When `rm` is requested |
## Configuration
@@ -495,7 +452,7 @@ Legacy compatibility form:
### QueueFS Persistence
-The lock mechanism relies on QueueFS using the SQLite backend to ensure enqueued tasks survive process restarts. This is the default configuration and requires no manual setup.
+Restart recovery requires persisted queue data. QueueFS uses SQLite by default. With the cache backend, durability depends on the configured Provider; the memory backend loses queued work when the process exits.
## Related Documentation
diff --git a/docs/en/concepts/15-vikingbot.md b/docs/en/concepts/15-vikingbot.md
index ebeb9bbe1..8e66ed8c7 100644
--- a/docs/en/concepts/15-vikingbot.md
+++ b/docs/en/concepts/15-vikingbot.md
@@ -14,26 +14,7 @@ Together, they allow an Agent not only to complete the current task, but also to
## System Overview
-```text
-CLI / Feishu / Slack / Telegram / Discord / Email / HTTP API
- │
- ▼
- Channel + MessageBus
- │
- ▼
- AgentLoop
- Context → Model → Tools → Model
- │ │
- ┌─────────┴─────────┐ ▼
- ▼ ▼ Replies and events
- OpenViking Context Tools / Skills
- Resource / Memory Files / Shell / Web
- Experience / Session MCP / Cron / Subagent
- │ │
- └─────────┬─────────┘
- ▼
- Session synchronization and learning
-```
+<VikingBotOverviewDiagram />
Every entry point ultimately uses the same AgentLoop. Channel-specific events are converted into common messages, so models and tools do not need to know whether a request came from the CLI, Feishu, or an HTTP API.
@@ -115,7 +96,7 @@ VikingBot reads the Peer Profile for the trusted current `actor_peer_id` and rec
- `entities`: people, projects, organizations, and other entities;
- `preferences`: user preferences, habits, and constraints.
-This identity model allows users sharing one Gateway to retain isolated personal context.
+Isolation depends on the OpenViking connection identity. `actor_peer_id` restricts a user's peer collection; chat users sharing one user key can still see that user's own memories and shared resources. Separate user isolation requires distinct user identities.
### Experience: Reusable Task Knowledge
diff --git a/docs/en/concepts/16-queue-lifecycle.md b/docs/en/concepts/16-queue-lifecycle.md
index 41d5594eb..3693027c8 100644
--- a/docs/en/concepts/16-queue-lifecycle.md
+++ b/docs/en/concepts/16-queue-lifecycle.md
@@ -1,8 +1,12 @@
# Queue State and Completion Semantics
+QueueFS determines completion from one backend status snapshot: a queue is empty only when both `pending` and `processing` are zero. An empty pending queue alone does not mean processing has finished.
+
+An empty queue means no messages are currently unacknowledged. It does not prove every operation succeeded or prevent new work from arriving. To verify an import or session commit, inspect its [task](../api/17-tasks.md), including the terminal status and error.
+
## Problem
-Queue completion is currently inferred from two independent sources:
+The previous implementation inferred completion from two independent sources:
- QueueFS `/size`, which reports messages still pending for dequeue.
- Python `NamedQueue._in_progress`, which reports work observed by the current
@@ -18,7 +22,7 @@ timing windows. It cannot make the backend state transition and local counter
update one atomic operation, so it does not strictly solve the problem across
threads, event loops, or processes.
-## Previous Backend Behavior
+## ACK Lifecycle
SQLite and cache-backed queues implement an acknowledgement lifecycle:
@@ -26,15 +30,11 @@ SQLite and cache-backed queues implement an acknowledgement lifecycle:
enqueue -> pending -> dequeue -> processing -> ack -> removed
```
-Before this change, the memory backend did not implement the same lifecycle.
-Its `dequeue` operation removed the message from its only queue immediately.
-Although the backend exposed an `ack` method, the dequeued message was no
-longer stored, so a later ACK normally had nothing to remove. The memory
-backend therefore had no effective `processing` state or ACK lifecycle.
+The memory backend also maintains a processing collection. Dequeue moves messages into it; ACK removes them. It uses the same completion condition as the other backends, but does not preserve queued work across process restarts.
## State Model
-Queue length is not one scalar. QueueFS must maintain these current-state
+Queue length is not one scalar. QueueFS maintains these current-state
gauges:
| Field | Meaning |
@@ -71,14 +71,13 @@ The following cumulative counters are also not queue length:
- `requeue_count`
- `error_count`
-They describe processing outcomes rather than current queue occupancy. They can
-remain in the processing or metrics layer for this fix. Moving them into
+They describe processing outcomes rather than current queue occupancy. They remain in the processing or metrics layer. Moving them into
QueueFS would require an explicit backend outcome protocol because the backend
cannot infer a handler result from dequeue or ACK alone.
## Backend Contract
-QueueFS should expose one atomic status operation:
+QueueFS `/status` returns counts read atomically:
```json
{
@@ -96,7 +95,7 @@ Backend requirements:
- **SQLite:** read both counts from one database snapshot.
- **Cache:** return `LLEN(pending)` and `ZCARD(processing)` from one Lua script.
-- **Memory:** add a processing collection; dequeue moves a message into it and
+- **Memory:** maintain a processing collection; dequeue moves a message into it and
ACK removes the message from it.
`NamedQueue.get_status()` consumes this backend snapshot. `wait_complete()` and
diff --git a/docs/zh/concepts/01-architecture.md b/docs/zh/concepts/01-architecture.md
index d8fb19d9b..cc684c795 100644
--- a/docs/zh/concepts/01-architecture.md
+++ b/docs/zh/concepts/01-architecture.md
@@ -4,65 +4,52 @@ OpenViking 是为 AI Agent 设计的上下文数据库,将所有上下文(Me
## 系统概览
+<ArchitectureDiagram />
+
+<details>
+<summary>文本版本</summary>
+
+```text
+CLI / SDK / HTTP client
+ |
+ HTTP Server
+ |
+ Service Layer
+ |
+ +------+------+----------------+
+ | | |
+Retrieval Sessions Resource / Skill import
+ | | |
+ | Memory extraction Parse / Semantic queues
+ | | |
+ +-------------+----------------+
+ |
+ VikingFS
+ / \
+ RAGFS Vector index
```
-┌────────────────────────────────────────────────────────────────────────────┐
-│ OpenViking 系统架构 │
-├────────────────────────────────────────────────────────────────────────────┤
-│ │
-│ ┌─────────────┐ │
-│ │ Client │ │
-│ │ (OpenViking)│ │
-│ └──────┬──────┘ │
-│ │ 委托 │
-│ ┌──────▼──────┐ │
-│ │ Service │ │
-│ │ Layer │ │
-│ └──────┬──────┘ │
-│ │ │
-│ ┌─────────────────────────┼─────────────────────────┐ │
-│ │ │ │ │
-│ ▼ ▼ ▼ │
-│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
-│ │ Retrieve │ │ Session │ │ Parse │ │
-│ │ (上下文检索) │ │ (会话管理) │ │ (上下文提取) │ │
-│ │ │ │ │ │ │ │
-│ │ search/find │ │ add │ │ 文档解析 │ │
-│ │ 意图分析 │ │ commit │ │ L0/L1/L2 │ │
-│ │ Rerank │ │ commit │ │ 树构建 │ │
-│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
-│ │ │ │ │
-│ │ │ 记忆提取 │ │
-│ │ ▼ │ │
-│ │ ┌─────────────┐ │ │
-│ │ │ Compressor │ │ │
-│ │ │ 压缩/去重 │ │ │
-│ │ └──────┬──────┘ │ │
-│ │ │ │ │
-│ └────────────────────────┼────────────────────────┘ │
-│ ▼ │
-│ ┌─────────────────────────────────────────────────────────────────┐ │
-│ │ Storage 层 │ │
-│ │ AGFS (文件内容) + 向量库 (索引) │ │
-│ └─────────────────────────────────────────────────────────────────┘ │
-│ │
-└────────────────────────────────────────────────────────────────────────────┘
-```
+
+</details>
## 核心模块
+<div class="module-table">
+
| 模块 | 职责 | 关键能力 |
|------|------|---------|
-| **Client** | 统一入口 | 提供所有操作接口,委托给 Service 层 |
+| **Client** | 统一入口 | 通过 HTTP API 提交 SDK/CLI 支持的操作 |
| **Service** | 业务逻辑 | FSService、SearchService、SessionService、ResourceService、PackService、DebugService |
| **Retrieve** | 上下文检索 | 意图分析(IntentAnalyzer)、全局检索(HierarchicalRetriever)、Rerank 精排 |
-| **Session** | 会话管理 | 消息记录、使用追踪、会话压缩、记忆提交 |
+| **Session** | 会话管理 | 消息记录、使用追踪、会话归档、触发记忆提交 |
| **Parse** | 上下文提取 | 文档解析(PDF/MD/HTML)、树构建(TreeBuilder)、异步语义生成 |
-| **Compressor** | 记忆压缩 | Schema 驱动的记忆提取、LLM 去重决策 |
-| **Storage** | 存储层 | VikingFS 虚拟文件系统、向量索引、AGFS 集成 |
+| **Memory** | 记忆提取 | 按 MemoryType Schema 提取记忆(ExtractLoop),由 LLM 合并去重后以 patch 写回(MemoryUpdater);由 `SessionCompressorV3` 编排 |
+| **Storage** | 存储层 | VikingFS 虚拟文件系统、向量索引、RAGFS 集成 |
+
+</div>
## Service 层
-Service 层将业务逻辑与传输层解耦,便于 HTTP Server 和 CLI 复用:
+Service 层将业务逻辑与传输层解耦,CLI 和 SDK 通过 HTTP Server 访问业务能力:
| Service | 职责 | 主要方法 |
|---------|------|----------|
@@ -79,21 +66,22 @@ OpenViking 采用双层存储架构,实现内容与索引分离(详见 [存
| 存储层 | 职责 | 内容 |
|--------|------|------|
-| **AGFS** | 内容存储 | L0/L1/L2 完整内容、多媒体文件、关联关系 |
-| **向量库** | 索引存储 | URI、向量、元数据(不存储文件内容) |
+| **RAGFS** | 内容存储 | L0/L1/L2 完整内容、多媒体文件、关联关系 |
+| **向量库** | 索引存储 | URI、向量、元数据和检索需要的文本(包括摘要) |
## 数据流概览
### 添加上下文
```
-输入 → Parser → TreeBuilder → AGFS → SemanticQueue → 向量库
+输入 → Parser → TreeBuilder → ResourceProcessor → RAGFS → SemanticQueue → 向量库
```
-1. **Parser**:解析文档,创建文件和目录结构(无 LLM 调用)
-2. **TreeBuilder**:移动临时目录到 AGFS,入队语义处理
-3. **SemanticQueue**:异步自底向上生成 L0/L1
-4. **向量库**:建立索引用于语义搜索
+1. **Parser**:将源文档解析为文件与目录;是否调用模型取决于所选 Parser
+2. **TreeBuilder**:确定目标 URI,保留临时产物引用
+3. **ResourceProcessor**:将内容写入 RAGFS,入队语义处理
+4. **SemanticQueue**:异步自底向上生成 L0/L1
+5. **向量库**:建立索引用于语义搜索
### 检索上下文
@@ -109,14 +97,14 @@ OpenViking 采用双层存储架构,实现内容与索引分离(详见 [存
### 会话提交
```
-消息 → 压缩 → 归档 → 记忆提取 → 存储
+消息 → 归档边界 → 归档 → 记忆提取 → 存储
```
1. **消息**:累积对话消息和使用记录
-2. **压缩**:保留最近 N 轮,旧消息归档
+2. **归档边界**:按提交参数划分归档消息和保留消息;默认归档全部当前消息
3. **归档**:生成历史片段的 L0/L1
4. **记忆提取**:根据记忆策略和 MemoryType Schema 从消息中提取记忆
-5. **存储**:写入 AGFS + 向量库
+5. **存储**:写入 RAGFS + 向量库
## 部署模式
@@ -126,13 +114,16 @@ OpenViking 采用双层存储架构,实现内容与索引分离(详见 [存
```python
# Python SDK 连接 OpenViking Server
-client = SyncHTTPClient(url="http://localhost:1933", api_key="xxx")
+from openviking_sdk import SyncHTTPClient
+
+client = SyncHTTPClient(url="http://localhost:1933", api_key="your-key")
```
```bash
# 或使用 curl / 任意 HTTP 客户端
curl http://localhost:1933/api/v1/search/find \
- -H "X-API-Key: xxx" \
+ -H "X-API-Key: your-key" \
+ -H "Content-Type: application/json" \
-d '{"query": "how to use openviking"}'
```
@@ -145,10 +136,10 @@ curl http://localhost:1933/api/v1/search/find \
| 原则 | 说明 |
|------|------|
-| **存储层纯粹** | 存储层只做 AGFS 操作和基础向量搜索,Rerank 在检索层完成 |
+| **存储层纯粹** | 存储层只做 RAGFS 操作和基础向量搜索,Rerank 在检索层完成 |
| **三层信息** | L0/L1/L2 实现渐进式详情加载,节省 Token 消耗 |
| **两阶段检索** | 向量搜索召回候选 + Rerank 精排提高准确性 |
-| **单一数据源** | 所有内容从 AGFS 读取,向量库仅存储引用和索引 |
+| **单一数据源** | RAGFS 保存源文件,向量记录保留检索需要的文本与元数据 |
## 相关文档
@@ -161,6 +152,6 @@ curl http://localhost:1933/api/v1/search/find \
- [会话管理](./08-session.md) - 会话和记忆管理
- [事务模型](./09-transaction.md) - 写入与一致性模型
- [数据加密](./10-encryption.md) - 静态数据加密与密钥架构
-- [多租户](./11-multi-tenant.md) - account / user / agent 隔离模型
+- [多租户](./11-multi-tenant.md) - account / user / peer 隔离模型
- [指标](./12-metrics.md) - `/metrics` 使用方式与关键指标说明
- [用户隐私配置](./13-privacy.md) - 隐私版本管理、自动提取Skill隐私配置
diff --git a/docs/zh/concepts/02-context-types.md b/docs/zh/concepts/02-context-types.md
index e5487ef32..33831dc5e 100644
--- a/docs/zh/concepts/02-context-types.md
+++ b/docs/zh/concepts/02-context-types.md
@@ -1,14 +1,39 @@
# 上下文类型
-基于对人类认知模式的简化映射与工程化思考,OpenViking 将上下文抽象为 **资源、记忆、能力三种**基本类型,每种类型在 Agent 中有不同的用途。
+OpenViking 管理三类上下文:资源提供参考资料,记忆保存交互中提取的信息,技能描述完成任务的方法。
## 概览
| 类型 | 用途 | 生命周期 | 主动性 |
|------|------|----------|--------|
-| **Resource** | 知识和规则 | 长期,相对静态 | 用户添加 |
-| **Memory** | Agent 的认知 | 长期,动态更新 | Agent 记录 |
-| **Skill** | 可声明的 agent 能动性配置(AgentDefinedContextType) | 长期,静态 | 用户或系统添加 |
+| **Resource** | 知识和规则 | 保留至更新或删除 | 用户添加 |
+| **Memory** | 偏好、事实和任务经验 | 长期,动态更新 | 从会话提取或主动记录 |
+| **Skill** | 任务指令和配套资源 | 长期,可更新 | 用户或系统添加 |
+
+## 示例准备
+
+以下示例使用同步 Python SDK,需先启动服务端。`add_resource` 和 `add_skill` 可传 `wait=True`,等处理完成后再返回。会话提交会在记忆提取完成前返回,且没有内置等待参数,因此先用下面的函数查询提交任务,再检索新内容。轮询超时不会取消服务端任务。
+
+```python
+import time
+from openviking_sdk import SyncHTTPClient
+
+client = SyncHTTPClient(url="http://localhost:1933", api_key="your-key")
+
+
+def wait_for_task(task_id):
+ deadline = time.monotonic() + 300
+ while time.monotonic() < deadline:
+ task = client.get_task(task_id)
+ if task is None:
+ raise RuntimeError(f"Task {task_id} not found")
+ if task["status"] == "completed":
+ return task
+ if task["status"] in {"failed", "cancelled"}:
+ raise RuntimeError(task)
+ time.sleep(1)
+ raise TimeoutError(f"Task {task_id} is still running")
+```
## Resource(资源)
@@ -17,7 +42,7 @@
### 特点
- **用户主动**:由用户主动添加的资源类信息,用于补充大模型的知识,比如产品手册、代码仓库
-- **静态内容**:添加后内容很少发生变化,通常为用户主动修改
+- **显式更新**:内容变化后重新导入;支持 Watch 的远程来源可定时刷新
- **结构化存储**:将按照项目或主题以目录层级组织,并提取出多层信息。
### 示例
@@ -33,6 +58,8 @@
client.add_resource(
path="https://docs.example.com/api.pdf",
options={"reason": "API 文档"},
+ wait=True,
+ timeout=300,
)
# 搜索资源
@@ -76,17 +103,18 @@ Schema 定义的 `memories/tools/` 和 `memories/skills/` 类型已禁用。它
from openviking_sdk import TextPart
# 记忆从会话中自动提取
-session_info = await client.create_session()
+session_info = client.create_session()
session = client.session(session_id=session_info["session_id"])
-await session.add_message(
+session.add_message(
role="user",
parts=[TextPart(text="我喜欢深色模式")],
)
-commit = await session.commit() # 启动后台记忆提取
-task = await client.get_task(task_id=commit["task_id"]) # 轮询直到 task["status"] == "completed"
+commit = session.commit() # 启动后台记忆提取
+if commit.get("task_id"):
+ wait_for_task(commit["task_id"])
# 搜索记忆
-results = await client.find(
+results = client.find(
query="用户界面偏好",
target_uri="viking://~/memories/"
)
@@ -94,13 +122,13 @@ results = await client.find(
## Skill(技能 / AgentDefinedContextType)
-技能(Skill)是 Agent 可以调用的能力,属于 AgentDefinedContextType 范畴。包括传统工作流定义、通信端点、工具配置和支付能力等。它们的共同特征是:**定义了 agent 如何与外部系统交互**,运行时定义相对静态,但调用经验会在 Memory 中更新。
+技能(Skill)通过 `SKILL.md` 和配套文件描述一类任务的步骤、约束和资源。Agent 读取技能后,使用自己的工具执行任务。调用经验可以单独保存为记忆。
### 特点
-- **定义的能力:**用于完成某项工作的工具定义
-- **相对静态:**运行时技能定义不变,但和工具相关的使用记忆会在记忆中更新
-- **可调用:**Agent 决定何时使用哪种技能
+- **任务说明:**记录完成某项工作的流程和约束
+- **可维护:**技能内容可以更新,执行经验单独存为记忆
+- **按需读取:**Agent 按当前任务选择技能
### 存储位置
@@ -111,7 +139,7 @@ viking://~/skills/{skill-name}/ # 默认存储路径
├── SKILL.md # L2: 技能定义
└── scripts # L2: 附加实现
-viking://agent/skills/{skill-name}/ # 通过 --uri 覆盖,公开共享(account 全局)
+viking://agent/skills/{skill-name}/ # 通过 -p/--parent-auto-create 覆盖,公开共享(account 全局)
├── .abstract.md # L0: 简短描述
├── .overview.md # L1: 目录概览(生成后)
├── SKILL.md # L2: 技能定义
@@ -120,7 +148,7 @@ viking://agent/skills/{skill-name}/ # 通过 --uri 覆盖,公开共享(acco
### AgentDefinedContextType 子类型
-AgentDefinedContextType 包含以下子类型,均存储于 `viking://agent/` 作用域:
+下表列出共享能力的设计分类。当前支持 Skill,默认安装到用户私有目录,也可显式安装到 `viking://agent/skills/`。其余分类是规划,不表示已提供对应接口:
| 子类型 | 位置 | 说明 |
|--------|------|------|
@@ -133,37 +161,45 @@ AgentDefinedContextType 包含以下子类型,均存储于 `viking://agent/`
```python
# 添加技能(默认写入 viking://~/skills/)
-await client.add_skill(
+client.add_skill(
data={
"name": "search-web",
"description": "搜索网络获取信息",
"content": "# search-web\n...",
},
+ wait=True,
+ timeout=300,
)
-# 通过 -p 指定写入全局 agent 技能根(公开共享)
-ov skills add search-web -p viking://agent/skills
-
# 搜索用户技能
-results = await client.find(
+results = client.find(
query="网络搜索",
target_uri="viking://~/skills/"
)
# 搜索全局 agent 技能
-results = await client.find(
+results = client.find(
query="网络搜索",
target_uri="viking://agent/skills/",
)
```
+通过 CLI 安装到账户共享技能目录(需要该路径的写入权限):
+
+```bash
+ov skills add ./skills/search-web -p viking://agent/skills
+```
+
## 统一检索
-根据Agent的需求需求,支持对三种上下文类型统一搜索,提供全面信息:
+一次检索可返回搜索范围内的资源、记忆和技能。默认范围包含当前用户空间和共享资源;如需同时查找共享技能,应显式加入 `viking://agent/skills`:
```python
# 跨所有上下文类型搜索
-results = await client.find(query="用户认证")
+results = client.find(
+ query="用户认证",
+ target_uri=["viking://~", "viking://resources", "viking://agent/skills"],
+)
for context in results.get("memories", []):
print(f"记忆: {context['uri']}")
@@ -173,6 +209,12 @@ for context in results.get("skills", []):
print(f"技能: {context['uri']}")
```
+操作结束后关闭客户端:
+
+```python
+client.close()
+```
+
## 相关文档
- [架构概述](./01-architecture.md) - 系统整体架构
diff --git a/docs/zh/concepts/03-context-layers.md b/docs/zh/concepts/03-context-layers.md
index b82e420bc..fc7495821 100644
--- a/docs/zh/concepts/03-context-layers.md
+++ b/docs/zh/concepts/03-context-layers.md
@@ -1,6 +1,6 @@
# 上下文层级(L0/L1/L2)
-OpenViking 使用三层信息模型,在检索效率、导航能力和原始内容完整性之间取得平衡。
+OpenViking 用摘要和概览定位内容,需要细节时再读取正文。这三个层级分别称为 L0、L1 和 L2。
## 概览
@@ -50,11 +50,11 @@ L1 提供更完整的目录摘要和导航信息,用于 Rerank 和决定是否
overview = client.overview(uri="viking://resources/docs/auth")
```
-L0 从 L1 正文中提取:取 H1 标题之后、第一个 `##` 标题之前的 Brief Description 段落。YAML frontmatter 不参与提取。
+资源目录生成语义摘要时,L0 从 L1 正文中提取:取 H1 标题之后、第一个 `##` 标题之前的 Brief Description 段落。YAML frontmatter 不参与提取。
## L2:详情
-L2 是原始文件或解析后的完整内容,只在需要时加载,并保留源格式和结构。
+L2 是原始文件或解析后的正文,按需加载。格式取决于导入方式,例如 PDF 可被解析为 Markdown,并不一定保留源文件格式。
```python
content = client.read(uri="viking://resources/docs/auth/oauth.md")
@@ -146,7 +146,7 @@ API 认证指南,涵盖 OAuth 2.0、JWT 令牌和 API 密钥。
- `unsampled_entries`:未采样的直接子项数,满足 `sampled + unsampled = total`。
- `pending_child_changes`:尚未反映到当前正文中的直接子项变化事件数(同一子项重复变化会分别计数)。
-当直接子项超过 `semantic.overview_sample_limit`(默认 32)时,系统使用确定性、保序的稳定采样。相同目录树重复刷新会选择相同样本,避免无意义的正文和 Git diff 抖动。
+当直接子项超过 `semantic.overview_sample_limit`(默认 32)时,系统使用确定性、保序的稳定采样。相同目录树重复刷新会选择相同样本,减少采样变化带来的正文差异;模型重新生成的正文仍可能不同。
`pending_child_changes > 0` 表示正文仍然可读,但已知落后于下层变化。父目录刷新成功后,该值会随新的覆盖率元数据重置为 0。
diff --git a/docs/zh/concepts/04-viking-uri.md b/docs/zh/concepts/04-viking-uri.md
index 06f4a2f41..2c8cc6303 100644
--- a/docs/zh/concepts/04-viking-uri.md
+++ b/docs/zh/concepts/04-viking-uri.md
@@ -18,7 +18,7 @@ viking://{scope}/{path}
|--------|------|----------|--------|
| **resources** | 独立资源/客观知识 | 长期 | account 全局 |
| **user** | 用户级数据,包括 session | 长期 / 会话生命周期 | 当前用户 |
-| **agent** | agent 能力与配置(技能、端点、工具、支付等) | 长期 | account 全局 |
+| **agent** | 账户共享技能与 Agent 配置;端点、工具和支付目录为规划用途 | 长期 | account 全局 |
| **queue** | 处理队列 | 临时 | 内部 |
| **temp** | 临时文件 | 解析期间 | 内部 |
| **upload** | 临时上传文件 | 临时 | 内部 |
@@ -36,7 +36,7 @@ viking://{scope}/{path}
- 通用:所有控制面(REST API、`ov` CLI、SDK、MCP)都接受,可用于任何接受公开作用域 URI 的位置。
- 仅识别第 0 段:`viking://resources/~/x` 和 `viking://user/alice/~/x` 中的 `~` 仍是字面路径段。
-- 接受但不宣传:`~` 不属于公开作用域列表,`Invalid scope ... Must be one of:` 错误信息中不会出现它。
+- `~` 是用户路径别名,不是独立的存储作用域。
- 响应始终回显展开后的 canonical URI,不会返回 `viking://~`;持久化数据(向量记录、watch key)
同样保持 canonical 形式。
- 需要认证请求身份。所有请求角色(包括 root)都使用该身份的有效 `user_id` 展开;要求 URI
@@ -48,18 +48,14 @@ viking://{scope}/{path}
## 初始目录
-摒弃传统的扁平化数据库思维,将所有上下文组织为一套文件系统。Agent 不再仅是通过向量搜索来找数据,而是可以通过确定性的路径和标准文件系统指令来定位和浏览数据。每个上下文或目录分配唯一的 URI 标识字符串,格式为 viking://{scope}/{path},让系统能精准定位并访问存储在不同位置的资源。
-
-## 文件 ID
-
-除 URI 之外,每个文件会被自动分配一个稳定的 `id`,作为其在 VikingDB 中向量记录的主键。对于 level 2(常规文件)记录,该 id 按 `md5(f"{account_id}:{uri}")` 确定性计算,由 `stat()` 等元数据接口返回。调用方可凭此 id 直接交叉引用向量索引条目,无需额外查询。id 以 account 为作用域,当文件被移动到其他 URI 时 id 会随之改变(URI 迁移过程中向量记录会重新计算主键)。目录不返回单一 `id`,因为一个目录在多个语义层(L0 abstract、L1 overview、L2)下可能对应多条记录,每条各有自己的 id。
+上下文按目录组织。已知 URI 时,Agent 可以直接列目录、读取文件;未知路径时,可先检索再读取。
```
viking://
├── user/
│ └── {user_id}/
-│ ├── profile.md # 用户画像
│ ├── memories/ # 用户记忆
+│ │ └── profile.md # 用户画像
│ ├── resources/ # 用户私有资源
│ ├── skills/ # 用户技能
│ ├── peers/
@@ -84,6 +80,10 @@ viking://
└── resources/{project}/ # 资源工作区
```
+## 文件 ID
+
+除 URI 之外,每个文件会被自动分配一个稳定的 `id`,作为其在 VikingDB 中向量记录的主键。对于 level 2(常规文件)记录,该 id 按 `md5(f"{account_id}:{uri}")` 确定性计算,由 `stat()` 等元数据接口返回。调用方可凭此 id 直接交叉引用向量索引条目,无需额外查询。id 以 account 为作用域,当文件被移动到其他 URI 时 id 会随之改变(URI 迁移过程中向量记录会重新计算主键)。目录不返回单一 `id`,因为一个目录在多个语义层(L0 abstract、L1 overview)下可能对应多条记录,每条各有自己的 id。
+
## URI 示例
### 资源
@@ -107,7 +107,7 @@ viking://~/memories/entities/ # 实体记忆
viking://~/memories/events/ # 事件记忆
viking://~/resources/ # 自己的私有资源
viking://~/resources/docs/ # 自己的私有资源目录
-viking://user/{user_id}/memories/ # 显式用户路径(可写自己的 id;访问他人需 admin/root)
+viking://user/{user_id}/memories/ # 显式用户路径(只能访问自己的 user_id,ADMIN 也不例外;ROOT 可跨用户读取,但 ROOT API key 在 api_key 模式下不能调用数据 API)
```
`viking://resources/...` 是当前 account 的共享区,可通过 [资源访问控制(ACL)](./15-acl.md) 细化目录或文件权限。`viking://user/{user}/resources/...` 是个人私有区;分享资源需要将其移动到共享区。
@@ -173,7 +173,7 @@ Viking URI 支持路径变量用于动态路径生成。这对于按时间序列
{namespace:key}
```
-- **namespace**: 变量提供者命名空间(如 `calendar`、`env`、`user`)
+- **namespace**: 变量提供者命名空间(内置 `calendar`)
- **key**: 命名空间内的变量名
### 日历变量
@@ -191,12 +191,12 @@ Viking URI 支持路径变量用于动态路径生成。这对于按时间序列
| `{calendar:ym}` | 年/月 | `2026/05` |
| `{calendar:quarter}` | 季度(Q1-Q4) | `Q2` |
| `{calendar:yq}` | 年/季度 | `2026/Q2` |
-| `{calendar:week}` | ISO 周数(带前导零) | `18` |
-| `{calendar:yw}` | 年/ISO 周 | `2026/w18` |
+| `{calendar:week}` | ISO 周数(带前导零) | `19` |
+| `{calendar:yw}` | 年/ISO 周 | `2026/w19` |
### 使用示例
-```python
+```text
# 按日期组织邮件
viking://resources/emails/{calendar:today}/inbox
# 渲染为:viking://resources/emails/2026/05/07/inbox
@@ -226,7 +226,7 @@ viking://resources/snapshots/{calendar:today}/
```bash
# 添加今天的邮件 --parent-auto-create 可以简写为 -p
-ov add-resource --parent-auto-create "viking://resources/emails/{calendar:today}/inbox" ./emails/*.eml
+ov add-resource --parent-auto-create "viking://resources/emails/{calendar:today}/inbox" ./emails/
# 读取昨天的日志
ov read "viking://resources/logs/{calendar:yesterday}/app.log"
@@ -242,7 +242,7 @@ ov add-resource --parent-auto-create "viking://resources/reports/{calendar:ym}"
```
viking://
-├── resources/ # 独立资源(客观知识,禁止存储非知识类配置)
+├── resources/ # 账户共享参考资料
│ └── {project}/
│ ├── .abstract.md # 摘要
│ ├── .overview.md # 概述
@@ -255,8 +255,8 @@ viking://
│ └── payments/ # 支付配置(ap2 等)(规划中)
│
├── user/{user_id}/
-│ ├── profile.md # 用户基本信息
│ ├── memories/
+│ │ ├── profile.md # 用户画像
│ │ ├── preferences/ # 按主题
│ │ ├── entities/ # 每条独立
│ │ └── events/ # 每条独立
@@ -278,6 +278,8 @@ viking://
## URI 操作
+以下 `VikingURI` 辅助类来自随服务端安装的 `openviking_cli` 包,不属于独立 Python SDK 的公开接口。
+
### 解析
```python
@@ -302,6 +304,8 @@ parent = VikingURI(uri).parent.uri # viking://resources/docs
## API 使用
+以下 Python 示例使用已配置连接的 `SyncHTTPClient` 实例 `client`,详见[客户端配置](../configuration/02-client.md)。
+
### 指定作用域搜索
```python
@@ -334,16 +338,16 @@ results = client.find(
```python
# 列出目录
-entries = await client.ls(uri="viking://resources/")
+entries = client.ls(uri="viking://resources/")
# 读取文件
-content = await client.read(uri="viking://resources/docs/api.md")
+content = client.read(uri="viking://resources/docs/api.md")
# 获取摘要
-abstract = await client.abstract(uri="viking://resources/docs/")
+abstract = client.abstract(uri="viking://resources/docs/")
# 获取概览
-overview = await client.overview(uri="viking://resources/docs/")
+overview = client.overview(uri="viking://resources/docs/")
```
## 特殊文件
@@ -372,24 +376,24 @@ overview = await client.overview(uri="viking://resources/docs/")
```python
# 添加到 account 共享资源作用域
-await client.add_resource(url, to="viking://resources/project/")
+client.add_resource(url, to="viking://resources/project/")
# 添加到自己的私有资源根
-await client.add_resource(path, parent="viking://~/resources/project/")
+client.add_resource(path, parent="viking://~/resources/project/")
# 技能默认添加到自己的技能根
-await client.add_skill(skill) # 默认根目录:viking://~/skills/
+client.add_skill(skill) # 默认根目录:viking://~/skills/
+```
+
+通过 CLI 安装到账户共享技能目录,需要有该路径的写入权限:
-# 通过 -p 指定写入全局 agent 技能根(公开共享)
-ov skills add xxx -p viking://agent/skills/
+```bash
+ov skills add ./skills/search-web -p viking://agent/skills/
```
### resources 作用域约束
-`resources` 作用域仅用于存储客观知识类数据(文档、代码、规范、论文等)。
-禁止在 `viking://resources/` 下存储非知识类数据,包括但不限于:
-工具配置、通信端点定义、支付配置、技能定义等。
-此类数据应使用 `viking://agent/` 作用域。
+将参考文档、代码、规范等资料放在 `resources`。需要作为 Skill 发现和加载的内容,应通过技能接口安装到 `viking://~/skills/` 或共享的 `viking://agent/skills/`;仅把 `SKILL.md` 当作普通资源上传,不等于安装技能。
## 相关文档
diff --git a/docs/zh/concepts/05-storage.md b/docs/zh/concepts/05-storage.md
index e61de85f7..f9a47f564 100644
--- a/docs/zh/concepts/05-storage.md
+++ b/docs/zh/concepts/05-storage.md
@@ -4,32 +4,21 @@ OpenViking 采用双层存储架构,分离内容存储和索引存储。
## 概览
-```
-┌─────────────────────────────────────────┐
-│ VikingFS (URI 抽象层) │
-│ URI 映射 · 层级访问 │
-└────────────────┬────────────────────────┘
- ┌────────┴────────┐
- │ │
-┌───────▼────────┐ ┌─────▼───────────┐
-│ 向量库索引 │ │ AGFS │
-│ (语义搜索) │ │ (内容存储) │
-└────────────────┘ └─────────────────┘
-```
+<StorageLayersDiagram />
## 双层存储
| 存储层 | 职责 | 存储内容 |
|--------|------|----------|
| **AGFS** | 内容存储 | L0/L1/L2 完整内容、多媒体文件 |
-| **向量库** | 索引存储 | URI、向量、元数据(不存文件内容) |
+| **向量库** | 索引存储 | URI、向量、元数据和摘要等检索文本 |
### 设计优势
-1. **职责清晰**:向量库只负责检索,AGFS 负责存储
-2. **内存优化**:向量库不存储文件内容,节省内存
-3. **单一数据源**:所有内容从 AGFS 读取,向量库只存引用
-4. **独立扩展**:向量库和 AGFS 可分别扩展
+AGFS 保存源文件,向量库存储检索需要的 URI 引用、向量、元数据和摘要。记忆记录会把正文保存在向量记录的 abstract 字段(上限 50,000 字节),因此向量库中也可能包含可读文本。
+
+两类后端可以分别配置。文件 API 从 AGFS 读取内容,检索则可直接返回索引中的文本,无需逐个回读源文件。备份和访问控制需要覆盖两类存储。
+
> 注:AGFS 已经重写为 Rust 实现(RAGFS)
## VikingFS 虚拟文件系统
@@ -82,7 +71,7 @@ AGFS 提供 POSIX 风格的文件操作,支持多种后端。
### 目录结构
-每个上下文目录遵循统一结构:
+完成语义处理的目录通常有以下结构。处理前或仅创建 L0 时,部分摘要文件可能尚不存在:
```
viking://resources/docs/auth/
@@ -125,6 +114,8 @@ viking://resources/docs/auth/
### 索引策略
+以下为索引元数据示例;可用选项取决于向量后端:
+
```python
index_meta = {
"IndexType": "flat_hybrid", # 混合索引
@@ -143,19 +134,19 @@ index_meta = {
## 向量同步
-VikingFS 自动维护向量库与 AGFS 的一致性。
+文件系统操作协调文件与向量记录的变更。目录摘要可能继续异步刷新,失败也可能留下部分变更;各操作的具体行为见[文件系统 API](../api/03-filesystem.md)。下面的示例使用已初始化的同步 Python SDK 客户端。
### 删除同步
```python
-viking_fs.rm("viking://resources/docs/auth", recursive=True)
+client.rm("viking://resources/docs/auth", recursive=True)
# 自动递归删除向量库中所有 uri 以此开头的记录
```
### 移动同步
```python
-viking_fs.mv(
+client.mv(
"viking://resources/docs/auth",
"viking://resources/docs/authentication"
)
diff --git a/docs/zh/concepts/06-extraction.md b/docs/zh/concepts/06-extraction.md
index b0713f8d7..32614cfb8 100644
--- a/docs/zh/concepts/06-extraction.md
+++ b/docs/zh/concepts/06-extraction.md
@@ -1,17 +1,12 @@
# 上下文提取
-OpenViking 采用三层异步架构处理文档解析和上下文提取。
+资源导入依次经过解析、目标路径确定、内容写入、语义生成和向量化。记忆和技能有各自的入口,也会使用语义处理和向量索引。
## 概览
-```
-输入文件 → Parser → TreeBuilder → SemanticQueue → 向量库
- ↓ ↓ ↓
- 解析转换 文件移动 L0/L1 生成
- (无 LLM) 入队语义 (LLM 异步)
-```
+<IngestionPipelineDiagram />
-**设计原则**:解析与语义分离,Parser 不调用 LLM,语义生成异步进行。
+解析与目录语义生成分开处理。具体解析器可能调用模型或外部解析服务,因此不能把解析阶段视为一律不使用 LLM。导入接口通常先返回任务 ID;需要立即检索时,应先确认该任务完成,见[任务管理](../api/17-tasks.md)。
## Parser(解析器)
@@ -26,70 +21,40 @@ Parser 负责文档格式转换和结构化,在临时目录创建文件结构
| PDF | PDFParser | .pdf | 已支持 |
| HTML | HTMLParser | .html, .htm | 已支持 |
| 代码 | CodeRepositoryParser | github 代码仓库等 | 遵循 `.gitignore` 并忽略常见非代码目录 |
-| 图片 | ImageParser | .png, .jpg 等 | |
-| 视频 | VideoParser | .mp4, .avi, .mov, .mkv, .webm, .flv, .wmv | |
-| 音频 | AudioParser | .mp3, .wav, .ogg, .flac, .aac, .m4a, .opus | |
+| 图片 | ImageParser | .png, .jpg 等 | 已支持;处理依赖媒体配置 |
+| 视频 | VideoParser | .mp4, .avi, .mov, .mkv, .webm, .flv, .wmv, .ts(仅 MPEG-TS 内容) | 已支持;处理依赖媒体配置 |
+| 音频 | AudioParser | .mp3, .wav, .ogg, .flac, .aac, .m4a, .opus, .ac3 | 已支持;处理依赖媒体配置 |
-### 核心流程 (以文档为例)
+### 解析产物
-```python
-# 1. 解析文件
-parse_result = registry.parse("/path/to/doc.md")
+内部 `registry.parse()` 是异步接口,返回 `ParseResult`。解析产物可以存放在本地临时目录或 AGFS;`temp_dir_path` 不一定是 Viking URI,`artifact_ref` 用来标识其存储后端。这些是服务端实现细节,客户端应使用资源导入 API。
-# 2. 返回临时目录 URI
-parse_result.temp_dir_path # viking://temp/abc123
-```
+| 字段 | 含义 |
+| --- | --- |
+| `root` | 解析后的资源树根节点 |
+| `temp_dir_path` | 临时产物位置 |
+| `artifact_ref` | 产物后端及根路径 |
+| `source_format` / `parser_name` | 源格式和解析器名称 |
+| `parse_time` | 解析耗时(秒) |
+| `meta` / `warnings` | 元数据及解析警告 |
-### 智能分割
+### 文档分节
-```
-如果 document_tokens <= 1024:
- → 保存为单文件
-否则:
- → 按标题分割
- → 小节 < 512 tokens → 合并
- → 大节 > 1024 tokens → 创建子目录
-```
+Markdown 解析器按标题和大小组织章节,合并较短小节,拆分过长内容。默认 `max_section_size` 为 2048 tokens,`max_section_chars` 为 6000 字符。配置还定义了 `section_size_flexibility=0.3`,但当前 Markdown 分节代码未使用该值。过长的单个表格行可以保持完整。这些值是分节目标,不能当作每个输出文件的绝对上限。
-### 返回结果
-
-```python
-ParseResult(
- temp_dir_path: str, # 临时目录 URI
- source_format: str, # pdf/markdown/html
- parser_name: str, # 解析器名称
- parse_time: float, # 耗时(秒)
- meta: Dict, # 元数据
-)
-```
+不同解析器和 `parse_mode` 的行为不同。需要保持单文件时,可在支持的导入中使用 `parse_mode="no_split"`,详见[资源管理](../api/02-resources.md)。
## TreeBuilder(树构建器)
-TreeBuilder 负责将临时目录移动到 AGFS,并入队语义处理。
+`TreeBuilder.finalize_from_temp()` 检查解析产物并确定最终 URI,返回包含根节点和临时位置的 `BuildingTree`。它本身不复制文件、不清理产物,也不提交语义任务。
-### 核心流程
+### 导入中的后续步骤
-```python
-building_tree = tree_builder.finalize_from_temp(
- temp_dir_path="viking://temp/abc123",
- scope="resources", # resources/user
-)
-```
-
-### 5 阶段处理
-
-1. **查找文档根目录**:确保临时目录下恰好 1 个子目录
-2. **确定目标 URI**:根据 scope 映射基础 URI
-3. **递归移动目录树**:复制所有文件到 AGFS
-4. **清理临时目录**:删除临时文件
-5. **入队语义生成**:提交 SemanticMsg 到队列
-
-### URI 映射
+1. 根据解析产物、`to` 或 `parent` 确定目标 URI,并检查目标路径。
+2. 导入处理器将内容写入最终存储,处理资源锁和已有内容更新。
+3. 提交后续语义处理和向量化,按产物所属后端清理临时数据。
-| scope | 基础 URI |
-|-------|----------|
-| resources | `viking://resources` |
-| user | `viking://user` |
+默认资源根目录是 `viking://resources`。个人资源应显式指定 `viking://~/resources/...`;`viking://user` 是用户空间容器,不能当作自己的资源根。
## SemanticQueue(语义队列)
@@ -97,14 +62,16 @@ SemanticQueue 异步处理 L0/L1 生成和向量化。
### 消息结构
-```python
-SemanticMsg(
- id: str, # UUID
- uri: str, # 目录 URI
- context_type: str, # resource/memory/skill
- status: str, # pending/processing/completed
-)
-```
+下面列出部分内部消息字段,不是客户端提交格式:
+
+| 字段 | 含义 |
+| --- | --- |
+| `id` | 消息 UUID |
+| `uri` | 待处理目录 |
+| `context_type` | resource、memory、skill 或 session |
+| `status` | 队列处理状态 |
+| `recursive` | 是否处理子目录 |
+| `propagate_to_parent` | 完成后是否允许安排父目录刷新 |
### 处理流程(自底向上)
@@ -114,7 +81,7 @@ SemanticMsg(
### 单目录处理步骤
-1. **并发生成文件摘要**:限制并发数 10
+1. **并发生成文件摘要**:并发上限由 `vlm.max_concurrent` 控制
2. **收集子目录摘要**:读取已生成的 .abstract.md
3. **生成 .overview.md**:LLM 生成 L1 概览
4. **提取 .abstract.md**:从 overview 提取 L0 摘要
@@ -131,10 +98,10 @@ L0/L1 是目录级 sidecar,不是 per-file sidecar。生成父目录摘要时
| 参数 | 默认值 | 说明 |
|------|--------|------|
-| `max_concurrent_llm` | 10 | 并发 LLM 调用数 |
-| `max_images_per_call` | 10 | 单次 VLM 最大图片数 |
-| `max_sections_per_call` | 20 | 单次 VLM 最大章节数 |
-| `overview_sample_limit` | 32 | 单个目录摘要使用的直接子项样本上限 |
+| `vlm.max_concurrent` | 32 | 语义处理并发上限,传入 `SemanticProcessor.max_concurrent_llm` |
+| `max_images_per_call` | 10 | `VLMProcessor` 构造参数:单次最大图片数,非 `ov.conf` 字段 |
+| `max_sections_per_call` | 20 | `VLMProcessor` 构造参数:单次最大章节数,非 `ov.conf` 字段 |
+| `semantic.overview_sample_limit` | 32 | 单个目录摘要使用的直接子项样本上限 |
## 代码骨架提取
@@ -160,16 +127,17 @@ L0/L1 是目录级 sidecar,不是 per-file sidecar。生成父目录摘要时
| 环节 | Resource | Memory | Skill |
|------|----------|--------|-------|
-| **Parser** | 通用流程 | 通用流程 | 通用流程 |
+| **入口** | 资源解析器 | 会话提取、记忆更新 | 技能导入 |
| **基础 URI** | `viking://resources` | `viking://~/memories` | `viking://~/skills` |
-| **TreeBuilder scope** | resources | user | user |
| **SemanticMsg type** | resource | memory | skill |
+以下示例使用已配置的同步 Python SDK 客户端 `client`。
+
### 资源提取
```python
# 添加资源
-await client.add_resource(
+client.add_resource(
path="/path/to/doc.pdf",
options={"reason": "API 文档"},
)
@@ -181,10 +149,10 @@ await client.add_resource(
```python
# 添加技能
-await client.add_skill(
+client.add_skill(
data={
"name": "search-web",
- "content": "# search-web\\n...",
+ "content": "# search-web\n...",
},
)
@@ -195,7 +163,7 @@ await client.add_skill(
```python
# 记忆从会话自动提取
-await session.commit()
+client.commit_session(session_id)
# 流程: SessionCompressorV3 → ExtractLoop → MemoryUpdater → SemanticQueue
```
diff --git a/docs/zh/concepts/07-retrieval.md b/docs/zh/concepts/07-retrieval.md
index c5f9e7910..a09c77f4b 100644
--- a/docs/zh/concepts/07-retrieval.md
+++ b/docs/zh/concepts/07-retrieval.md
@@ -12,24 +12,32 @@ OpenViking 使用全局向量检索,并可在召回完成后对候选结果执
| 特性 | find() | search() |
|------|--------|----------|
-| 会话上下文 | 不需要 | 需要 |
-| 意图分析 | 不使用 | 使用 LLM 分析 |
-| 查询数量 | 单一查询 | 0-5 个 TypedQuery |
-| 延迟 | 低 | 较高 |
+| 会话上下文 | 不使用 | 可选,传入 `session_id` 时使用 |
+| 意图分析 | 不使用 | 有会话内容且启用时使用 LLM |
+| 查询数量 | 单一查询 | 零个或多个 TypedQuery |
+| 延迟 | 通常较低 | 取决于意图分析、查询数量和是否 Rerank |
| 适用场景 | 简单查询 | 复杂任务 |
+`search` 的 `limit` 用于每条规划查询;多条查询合并后的总数可能超过它,当前也不保证跨查询去重。
+
### 使用示例
+以下示例使用已配置的同步 Python SDK 客户端 `client`。
+
```python
# find(): 简单查询
-results = await client.find(
+results = client.find(
query="OAuth 认证",
target_uri="viking://resources/",
)
# search(): 复杂任务(需要会话上下文)
-session_info = await client.create_session()
-results = await client.search(
+session_info = client.create_session()
+client.add_message(
+ session_id=session_info["session_id"], role="user",
+ content="我们正在为项目设计 OAuth 登录流程。",
+)
+results = client.search(
query="帮我创建一个 RFC 文档",
session_id=session_info["session_id"],
)
@@ -37,7 +45,7 @@ results = await client.search(
## 意图分析
-IntentAnalyzer 使用 LLM 分析查询意图,生成 0-5 个 TypedQuery。该阶段使用的模型可通过 [`query_planner`](../guides/01-configuration.md#query-planner) 配置项单独指定,未设置时回退到 `vlm`。
+当 `retrieval.enable_intent=true` 且会话包含摘要或消息时,IntentAnalyzer 使用 LLM 分析查询意图,生成零个或多个 TypedQuery。该阶段使用的模型可通过 [`query_planner`](../guides/01-configuration.md#query-planner) 配置项单独指定,未设置时回退到 `vlm`。
### 输入
@@ -99,6 +107,8 @@ Rerank 只处理本次全局召回的候选,不会触发下一轮检索。例
## 检索结果
+以下是服务端内部类型的字段节选;HTTP/SDK 响应字段以[检索 API](../api/06-retrieval.md)为准。
+
### MatchedContext
```python
@@ -106,7 +116,7 @@ Rerank 只处理本次全局召回的候选,不会触发下一轮检索。例
class MatchedContext:
uri: str # 资源 URI
context_type: ContextType
- is_leaf: bool # 是否文件
+ level: int # 0=摘要,1=概览,2=详情
abstract: str # L0 摘要
score: float # 最终分数
```
diff --git a/docs/zh/concepts/08-session.md b/docs/zh/concepts/08-session.md
index ff2e4240d..129fe637a 100644
--- a/docs/zh/concepts/08-session.md
+++ b/docs/zh/concepts/08-session.md
@@ -10,6 +10,9 @@ Session 负责管理对话消息、记录上下文使用、提取长期记忆。
`client.session(session_id=...)` 追加消息或提交会话。
```python
+from openviking_sdk import SyncHTTPClient
+
+client = SyncHTTPClient(url="http://localhost:1933", api_key="your-key")
session_info = client.create_session(session_id="chat_001")
session = client.session(session_id=session_info["session_id"])
session.add_message(role="user", content="...")
@@ -22,7 +25,7 @@ session.commit()
|------|------|
| `add_message(role, content=None, parts=None, options=None, peer_id=None)` | 添加消息 |
| `commit()` | 提交:归档(同步) + 摘要生成和记忆提取(异步后台) |
-| `get_task(task_id)` | 查询后台任务状态 |
+| `client.get_task(task_id)` | 查询后台任务状态 |
### add_message
@@ -66,10 +69,11 @@ result = session.commit()
# "archived": True
# }
-# 查询后台任务进度
-task = client.get_task(task_id=result["task_id"])
-# task["status"]: "pending" | "running" | "completed" | "failed"
-# sum(task["result"]["memories_extracted"].values()): 3
+# 产生归档时才有后台任务;无可归档消息时返回 skipped 和 task_id: null
+task_id = result.get("task_id")
+if task_id:
+ task = client.get_task(task_id=task_id)
+ print(task) # 单次查询,不代表任务已完成
```
## 消息结构
@@ -100,11 +104,11 @@ class Message:
commit() 分两阶段执行:
-**Phase 1(同步,立即完成)**:
-1. 递增 compression_index
-2. 写入消息到归档目录(`messages.jsonl`)
-3. 清空当前消息列表
-4. 返回 `task_id`
+**Phase 1(请求内完成归档准备)**:
+1. 在路径锁保护下分配归档编号,划分归档消息和保留消息
+2. 写入归档消息(`messages.jsonl`),将后续处理加入持久化队列
+3. 更新当前消息列表;默认归档全部消息,也可通过提交参数保留最近的消息或轮次
+4. 产生归档时返回 `task_id`,用于跟踪后台处理
**Phase 2(异步后台)**:
5. 生成结构化摘要(LLM)→ 写入 `.abstract.md` 和 `.overview.md`
@@ -144,29 +148,17 @@ OpenViking 内置 `profile`、`preferences`、`entities`、`events`、`identity`
### 提取流程
-```
-消息 → LLM 提取 → 候选记忆
- ↓
-向量预过滤 → 找相似记忆
- ↓
-LLM 去重决策 → candidate(skip/create/none) + item(merge/delete)
- ↓
-写入 AGFS → 向量化
-```
+<MemoryExtractionDiagram />
+
+已有记忆会参与提取,更新可以合并或修改现有文件,也可能创建或删除文件。实际操作受记忆 schema、写入权限和输出校验约束,不能把每次提交理解成必然新增记忆。只有产生 case 时,后续训练才会生成 trajectory、experience 或已启用的 session skill。
-### 去重决策
+### 如何检查更新结果
-| 层级 | 决策 | 说明 |
-|------|------|------|
-| Candidate | `skip` | 候选记忆重复,直接跳过 |
-| Candidate | `create` | 创建候选记忆;必要时先删除冲突旧记忆 |
-| Candidate | `none` | 不创建候选记忆,只处理已有记忆 |
-| Existing item | `merge` | 将候选内容合并到指定已有记忆 |
-| Existing item | `delete` | 删除冲突的已有记忆 |
+先等待 commit 对应任务完成,再查看 `memory_diff.json`。其中新增、修改、删除是实际文件变更;`skipped_operations` 表示提取提出了操作,但校验或策略使其跳过。无实际变化的更新不会计入 diff;正文未变但元数据发生变化时,仍可能记录为更新。
## 记忆变更记录
-每次 `session.commit()` 会在归档目录写入 `memory_diff.json`,记录本次提交的所有记忆变更,便于审计和回溯。
+提交后的后台处理会在归档目录写入 `memory_diff.json`,记录本次提交的记忆变更,便于审计和回溯。收到 `task_id` 时文件可能尚未生成,先按[会话 API](../api/05-sessions.md)查询任务完成状态。
```json
{
@@ -175,23 +167,23 @@ LLM 去重决策 → candidate(skip/create/none) + item(merge/delete)
"operations": {
"adds": [
{
- "uri": "memory/user/xxx/identity.md",
+ "uri": "viking://user/alice/memories/identity.md",
"memory_type": "identity",
"after": "新创建的文件内容"
}
],
"updates": [
{
- "uri": "memory/user/xxx/context/project.md",
- "memory_type": "context",
+ "uri": "viking://user/alice/memories/entities/project.md",
+ "memory_type": "entities",
"before": "修改前的文件内容",
"after": "修改后的文件内容"
}
],
"deletes": [
{
- "uri": "memory/user/xxx/context/old.md",
- "memory_type": "context",
+ "uri": "viking://user/alice/memories/entities/old.md",
+ "memory_type": "entities",
"deleted_content": "被删除的文件内容"
}
]
diff --git a/docs/zh/concepts/09-transaction.md b/docs/zh/concepts/09-transaction.md
index ff3d17665..5fcbda7b9 100644
--- a/docs/zh/concepts/09-transaction.md
+++ b/docs/zh/concepts/09-transaction.md
@@ -1,10 +1,10 @@
# 路径锁与崩溃恢复
-OpenViking 通过**路径锁**和**持久化队列恢复**两个简单原语保护核心写操作(`rm`、`mv`、`add_resource`、`session.commit`)的一致性,协调并发写入,并在进程重启后继续处理已入队的会话任务。路径锁和队列恢复不构成跨 VikingFS、VectorDB、QueueManager 的原子事务。
+OpenViking 通过**路径锁**和**持久化队列恢复**两种机制保护核心写操作(`rm`、`mv`、`add_resource`、`session.commit`)的一致性,协调并发写入,并在进程重启后继续处理已入队的会话任务。路径锁和队列恢复不构成跨 VikingFS、VectorDB、QueueManager 的原子事务。
## 设计哲学
-OpenViking 是上下文数据库,FS 是源数据,VectorDB 是派生索引。索引丢了可从源数据重建,源数据丢失不可恢复。因此:
+OpenViking 是上下文数据库,FS 是源数据,VectorDB 是派生索引。索引通常可以从保留的源数据重建;恢复丢失的源数据则依赖备份。因此:
> **宁可搜不到,不要搜到坏结果。**
@@ -12,69 +12,34 @@ OpenViking 是上下文数据库,FS 是源数据,VectorDB 是派生索引。
1. **写互斥**:参与锁协议的不同 owner 不能同时取得冲突路径的锁
2. **默认生效**:受保护的写操作默认加锁;普通读取和底层 mkdir 不自动加锁
-3. **锁即保护**:进入 LockContext 时加锁,退出时释放,没有 undo/journal/commit 语义
-4. **仅 session_memory 需要崩溃恢复**:通过持久化 `session_commit` 队列在进程崩溃后恢复 Phase 2
-5. **Queue 操作在锁外执行**:SemanticQueue/EmbeddingQueue 的 enqueue 是幂等的,失败可重试
+3. **锁只保护并发**:运行时申请和释放 lease,不提供跨存储的 undo/journal/commit 语义
+4. **持久化任务恢复**:`SessionCommit` 队列恢复会话 Phase 2;资源派生处理由相应的持久化队列恢复
+5. **锁与队列配合**:业务路径可在持锁时入队并交接 lease;重试和去重取决于具体任务协议,不能把所有 enqueue 视为天然幂等
## 架构
-```
-Service Layer (rm / mv / add_resource / session.commit)
- |
- v
-+--[LockContext 异步上下文管理器]-------+
-| |
-| 1. 创建 LockHandle |
-| 2. 获取路径锁(轮询 + 超时) |
-| 3. 执行操作(FS + VectorDB) |
-| 4. 释放锁 |
-| |
-| 异常时:自动释放锁,异常原样传播 |
-+---------------------------------------+
- |
- v
-Storage Layer (VikingFS, VectorDB, QueueManager)
-```
+<PathLockDiagram />
## 两个核心组件
-### 组件 1:PathLockEngine + LockManager + LockContext(路径锁系统)
+### 组件 1:路径锁系统
-**PathLockEngine** 实现基于 Provider 的分布式锁,支持 EXACT 和 TREE 两种锁类型,使用归属 token 防止 TOCTOU 竞争,并自动检测和清理过期锁。默认 Provider 在 AGFS 中保存锁文件;Cache Provider 在 Redis 中保存 token。
+运行时锁由 Rust RAGFS 的 `PathLockManager` 和 Provider 管理。Python 服务通过 binding 申请、续用、交接和释放 lease。后文的 `LockContext` 示例仅说明生命周期,不是当前 Python SDK 或运行时类接口。
-**LockHandle** 是轻量的锁持有者令牌:
-
-```python
-@dataclass
-class LockHandle:
- id: str # 唯一标识,用于生成 fencing token
- locks: list[str] # Provider handle:锁文件路径或逻辑路径
- created_at: float # handle 创建时间
- last_active_at: float # 最近一次成功 acquire/refresh 的时间
-```
+- **范围**:EXACT 保护路径本身,TREE 保护路径及其子树。
+- **Provider**:默认 filesystem 保存锁文件;cache 用 Redis 原子检查和写入 token;memory 只在当前进程内协调。
+- **lease**:记录 owner、覆盖路径、token 位置和续期状态。后台任务可以接收显式交接的 lease。
+- **失败处理**:释放前校验 owner,过期锁由后续获取时的 stale 检查清理;锁释放不会撤销已写入的数据。
-**LockManager** 是全局单例,管理锁生命周期:
-- 创建/释放 LockHandle
-- 后台清理泄漏的锁(进程内安全网)
-- 启动后由 QueueManager 恢复持久化的 `session_commit` Phase 2 任务
+### 组件 2:持久化 `SessionCommit` 队列(崩溃恢复)
-**LockContext** 是异步上下文管理器,封装加锁/解锁生命周期:
-
-```python
-# Conceptual example: production path locks are acquired inside the Rust ragfs layer.
-async with LockContext(lock_manager, [path], lock_mode="exact") as handle:
- # 在锁保护下执行操作
- ...
-# 退出时自动释放锁(包括异常情况)
-```
-
-### 组件 2:持久化 `session_commit` 队列(崩溃恢复)
+队列名为 `SessionCommit`,任务类型为 `session_commit`。
`session.commit` 的 Phase 2 不再使用独立 RedoLog。Phase 1 会先把 archive 元数据持久化,再把
`SessionCommitMsg` 写入持久化队列;进程重启后,QueueManager 会继续消费遗留的 `session_commit`
任务并恢复 Phase 2。
-Memory 提取是幂等的,从同一个 archive 重新提取会得到相同结果。
+恢复使用已保存的 archive 和处理状态。记忆提取包含模型调用,不能假设重新执行会生成逐字相同的内容。
## 一致性问题与解决方案
@@ -108,8 +73,8 @@ Memory 提取是幂等的,从同一个 archive 重新提取会得到相同结
|------|------|
| 文件移到新路径但索引指向旧路径 -> 搜索返回旧路径(不存在) | 先 copy 再更新索引,失败时清理副本 |
-**加锁策略**(通过 `lock_mode="mv"` 自动处理):
-- 移动**目录**:源路径加 TreeLock,目标路径加 ExactPathLock
+**加锁策略**(由公开的 `VikingFS.mv` 实现申请):
+- 移动**目录**:源路径加 TreeLock,目标路径加 TreeLock
- 移动**文件**:源路径和目标路径各加 EXACT 锁
操作流程:
@@ -120,7 +85,7 @@ Memory 提取是幂等的,从同一个 archive 重新提取会得到相同结
3. Copy 到新位置(源还在,安全)
4. 如果是目录,删除副本中被 cp 带过去的锁文件
5. 更新 VectorDB 中的 URI
- - 失败 -> 清理副本,源和旧索引都在,一致状态
+ - 失败 -> 尝试清理副本;索引或清理可能部分完成,需检查后再重试
6. 删除源
7. 释放锁
```
@@ -129,7 +94,7 @@ Memory 提取是幂等的,从同一个 archive 重新提取会得到相同结
| 问题 | 方案 |
|------|------|
-| 文件从临时目录移到正式目录后崩溃 -> 文件存在但永远搜不到 | 首次添加与增量更新分离为两条独立路径 |
+| 文件从临时目录移到正式目录后崩溃 -> 文件存在但永远搜不到 | 先提交内容计划,再由持久化队列完成派生索引 |
| 资源已落盘但语义处理/向量化还在跑时被 rm 删除 -> 处理白跑 | 生命周期 TreeLock,从落盘持续到处理完成 |
**首次添加和增量更新**使用同一条计划提交路径:
@@ -158,9 +123,7 @@ Memory 提取是幂等的,从同一个 archive 重新提取会得到相同结
判断候选目录是否已占用;已存在则尝试 `_1`、`_2` 后缀。候选目录不存在时才尝试
获取该目录的 `TreeLock`,且不等待;如果同名正在被并发请求处理,就直接尝试下一个后缀。
-**服务重启恢复**:`SemanticMsg` 及其中的 `SemanticPlan` 持久化在 QueueFS
-中。重启后 `SemanticProcessor` 发现 `lifecycle_lock_handle_id` 对应的 handle
-不在内存中,会重新获取 TreeLock 后继续派生处理。
+**服务重启恢复**:`SemanticMsg` 和 `SemanticPlan` 保存在持久化 QueueFS 后端中。语义处理通过 `lock_handoff` 尝试接管 lease;对可恢复的过期交接,按 `covered_paths` 重新申请覆盖范围后继续。锁冲突或不可恢复的交接错误仍会使处理失败,需检查任务状态。
### 派生语义文件(.abstract.md / .overview.md)
@@ -184,38 +147,34 @@ viking://user/default/memories/preferences/editor.md
### session.commit()
-| 问题 | 方案 |
-|------|------|
-| 消息已清空但 archive 未写入 -> 对话数据丢失 | Phase 1 无锁(archive 不完整无副作用)+ Phase 2 持久化 `session_commit` 队列 |
+Phase 1 使用会话根目录的 EXACT 锁划定提交边界。模型调用耗时不可控,因此摘要生成和记忆提取放在后台,不在这个边界锁内等待:
-LLM 调用耗时不可控(5s~60s+),不能放在持锁操作内。设计拆为两个阶段:
-
-```
-Phase 1 — 归档(无锁):
- 1. 生成归档摘要(LLM)
- 2. 写 archive(history/archive_N/messages.jsonl + 摘要)
- 3. 清空 messages.jsonl
- 4. 清空内存中的消息列表
-
-Phase 2 — 记忆提取 + 写入(持久化 `session_commit` 队列):
- 1. 持久化 archive 元数据并 enqueue `SessionCommitMsg`
- 2. 从归档消息提取 memories(LLM)
- 3. 写当前消息状态
- 4. 直接 enqueue SemanticQueue
+```text
+Phase 1:持锁准备并发布归档
+ 1. 读取消息与提交策略,分配归档编号,划分归档和保留消息
+ 2. 写归档消息及元数据
+ 3. 入队 SessionCommitMsg 并创建 task
+ 4. 写当前保留消息、会话元数据和 phase1 ready 标记
+ 5. 释放锁,返回 task_id(无可归档内容时 skipped)
+
+Phase 2:后台处理已发布归档
+ 1. 读取归档消息并生成摘要
+ 2. 提取、更新记忆并安排派生处理
+ 3. 写 memory_diff.json 和完成标记
```
**崩溃恢复分析**:
-| 崩溃时间点 | 状态 | 恢复动作 |
-|-----------|------|---------|
-| Phase 1 写 archive 中途 | 队列未发布 | archive 不完整,下次 commit 从 history/ 扫描 index,不受影响 |
-| Phase 1 archive 完成但 messages 未清空 | 队列未发布 | archive 完整 + messages 仍在 = 数据冗余但安全 |
-| Phase 2 记忆提取/写入中途 | `session_commit` 任务仍在持久化队列中 | 重启后继续消费该任务,从 archive 恢复 Phase 2 |
-| Phase 2 完成 | archive 标记为完成 | 无需恢复 |
+| 时间点 | 应检查什么 |
+| --- | --- |
+| Phase 1 尚未发布 | 归档准备可能只完成一部分;检查归档状态和当前消息,不把目录存在当作提交成功 |
+| 已入队但 ready 尚未写入 | 后台处理需要核对 Phase 1 状态;错误路径会尝试记录失败标记,不能仅凭 task 存在判断成功 |
+| Phase 2 提取或写入中途 | 持久化队列可恢复处理,但模型重试不保证生成相同正文 |
+| Phase 2 完成 | 核对 task 最终状态、归档完成标记和实际记忆变更 |
## LockContext
-`LockContext` 是**异步**上下文管理器,封装锁的获取和释放:
+以下伪代码说明申请、使用、释放锁的顺序。当前运行时使用 RAGFS lease API,不能直接从 Python SDK 导入 `LockContext`:
```python
# Conceptual example: production path locks are acquired inside the Rust ragfs layer.
@@ -242,9 +201,9 @@ async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_path=dst):
|-----------|------|------|
| `exact` | 文件写入、单文件删除、派生文件写回 | 锁定指定路径;与同路径锁和祖先目录 TreeLock 冲突 |
| `tree` | 删除目录、资源生命周期、目录级保护 | 锁定子树根节点;与同路径锁、后代锁和祖先 TreeLock 冲突 |
-| `mv` | 移动操作 | 目录移动:源路径 TreeLock + 目标路径 ExactPathLock;文件移动:源路径和目标路径均 ExactPathLock(通过 `src_is_dir` 控制) |
+| `mv` | 移动操作 | 目录移动:源路径 TreeLock + 目标路径 TreeLock;文件移动:源路径和目标路径均 ExactPathLock(通过 `src_is_dir` 控制) |
-**异常处理**:`__aexit__` 总是释放锁,不吞异常。获取锁失败时抛出 `LockAcquisitionError`。
+**异常处理**:业务调用通过 `finally` 释放或交接 lease。锁冲突会返回相应 busy/锁获取错误;释放失败或进程退出后的 token 依赖过期清理。
## 锁类型(EXACT vs TREE)
@@ -274,7 +233,7 @@ Exact 和 Tree 表达操作范围,文件、目录或缺失路径表达目标
`Tree(/docs/a.md)` 不会扩大为 `Tree(/docs)`。反过来,目录自身的 Exact 也不能保护子树,递归删除需要 Tree。
-锁只协调参与协议的操作。底层 `PathLockWrappedFS` 对 create、write、truncate、非递归 remove 使用 Exact,对 remove_all 使用 Tree;文件 rename 锁源和目标的 Exact,目录 rename 锁源 Tree 和目标 Exact。read、stat、列目录和 mkdir 直接转发,上层可另行持锁。绕过协议的 I/O 不会被操作系统自动阻断。
+锁只协调参与协议的操作。底层 `PathLockWrappedFS` 对 create、write、truncate、非递归 remove 使用 Exact,对 remove_all 使用 Tree;文件 rename 锁源和目标的 Exact,底层目录 rename 锁源 Tree 和目标 Exact;公开目录 `mv` 额外保护源、目标两棵子树。read、stat、列目录和 mkdir 直接转发,上层可另行持锁。绕过协议的 I/O 不会被操作系统自动阻断。
## 锁机制
@@ -418,7 +377,7 @@ fencing token 校验通过的一方成功持有 `TreeLock(java-guide)`;失败
| session_memory 提取中途崩溃 | 从 archive 恢复 Phase 2 并继续消费 `session_commit` 任务 |
| 锁持有期间崩溃 | Provider token 保留,后续匹配的 acquire 通过 stale 检测自动清理(默认 30s 过期)|
| enqueue 后 worker 处理前崩溃 | QueueFS SQLite 持久化,worker 重启后自动拉取 |
-| 孤儿索引 | L2 按需加载时清理 |
+| 孤儿索引 | `rm` 对不存在的目标也会尝试清理相关向量记录 |
### 防线总结
@@ -426,9 +385,9 @@ fencing token 校验通过的一方成功持有 `TreeLock(java-guide)`;失败
|---------|------|---------|
| 操作中途崩溃 | 锁自动过期 + stale 检测 | 下次获取同路径锁时 |
| add_resource 语义处理中途崩溃 | 生命周期锁过期 + SemanticProcessor 重启时重新获取 | worker 重启后 |
-| session.commit Phase 2 崩溃 | 持久化 `session_commit` 队列 + 重试消费 | 重启时 |
+| session.commit Phase 2 崩溃 | 持久化 `SessionCommit` 队列 + 重试消费 | 重启时 |
| enqueue 后 worker 处理前崩溃 | QueueFS SQLite 持久化 | worker 重启后 |
-| 孤儿索引 | L2 按需加载时清理 | 用户访问时 |
+| 孤儿索引 | `rm` 对不存在的目标也会尝试清理相关向量记录 | 调用 `rm` 时 |
## 配置
@@ -502,7 +461,7 @@ Redis 配置:
### QueueFS 持久化
-路径锁机制依赖 QueueFS 使用 SQLite 后端,确保 enqueue 的任务在进程重启后可恢复。这是默认配置,无需手动设置。
+重启恢复要求队列内容已持久化。QueueFS 默认使用 SQLite;使用 cache 后端时,持久性取决于所配置的 Provider。memory 后端不保留进程退出后的任务。
## 相关文档
diff --git a/docs/zh/concepts/15-vikingbot.md b/docs/zh/concepts/15-vikingbot.md
index 56ff47626..3bf1767dc 100644
--- a/docs/zh/concepts/15-vikingbot.md
+++ b/docs/zh/concepts/15-vikingbot.md
@@ -13,26 +13,7 @@ VikingBot 是 OpenViking 提供的多渠道 AI Agent。OpenViking 负责统一
## 系统概览
-```text
-CLI / Feishu / Slack / Telegram / Discord / Email / HTTP API
- │
- ▼
- Channel + MessageBus
- │
- ▼
- AgentLoop
- 上下文 → 模型 → 工具 → 模型
- │ │
- ┌─────────┴─────────┐ ▼
- ▼ ▼ 回复与事件
- OpenViking Context Tools / Skills
- Resource / Memory Files / Shell / Web
- Experience / Session MCP / Cron / Subagent
- │ │
- └─────────┬─────────┘
- ▼
- Session 同步与经验沉淀
-```
+<VikingBotOverviewDiagram />
所有入口最终使用同一套 AgentLoop。渠道差异被转换为统一消息,模型和工具无需感知消息来自命令行、飞书还是 HTTP API。
@@ -114,7 +95,7 @@ VikingBot 根据当前可信 `actor_peer_id` 读取 Peer Profile,并按类型
- `entities`:人、项目和组织等实体信息;
- `preferences`:用户偏好、习惯和约束。
-这使不同用户共享同一个 Gateway 时,仍能使用各自隔离的上下文。
+隔离范围取决于 OpenViking 连接身份。`actor_peer_id` 限制同一 user 下的 peer 集合;如果多个聊天用户共用一个 user key,该用户自己的记忆和共享资源仍可见。独立用户隔离需要不同的 user 身份。
### Experience:可复用任务经验
diff --git a/docs/zh/concepts/16-queue-lifecycle.md b/docs/zh/concepts/16-queue-lifecycle.md
index b87ea77ab..ec7549a5a 100644
--- a/docs/zh/concepts/16-queue-lifecycle.md
+++ b/docs/zh/concepts/16-queue-lifecycle.md
@@ -1,8 +1,12 @@
# 队列状态与完成语义
+QueueFS 通过同一份后端状态快照判断队列是否完成:只有 `pending` 和 `processing` 都为零,队列才为空。仅检查待出队消息数不能证明处理已结束。
+
+队列为空只描述当前没有未确认的消息,不证明每个业务操作成功,也不保证之后不会入队新任务。需要确认一次导入或会话提交是否成功,请查询对应 [task](../api/17-tasks.md),检查最终状态及错误。
+
## 问题
-当前队列完成状态由两个相互独立的数据源推断:
+旧实现通过两个相互独立的数据源推断完成状态:
- QueueFS `/size`:返回仍可被 dequeue 的 pending 消息数。
- Python `NamedQueue._in_progress`:返回当前进程观察到的执行中任务数。
@@ -15,7 +19,7 @@
状态迁移与本地计数更新成为同一个原子操作,因此无法严格解决跨线程、跨 event loop
或跨进程的并发判断问题。
-## 改动前的后端行为
+## ACK 生命周期
SQLite 和缓存后端实现了 ACK 生命周期:
@@ -23,14 +27,11 @@ SQLite 和缓存后端实现了 ACK 生命周期:
enqueue -> pending -> dequeue -> processing -> ack -> removed
```
-改动前,MemoryBackend 没有实现相同的生命周期。它的 `dequeue` 会直接从唯一的
-队列中删除消息。虽然接口上存在 `ack` 方法,但被 dequeue 的消息已经不再保存,
-后续 ACK 通常找不到可删除的消息。因此,MemoryBackend 实际上没有有效的
-`processing` 状态和 ACK 生命周期。
+MemoryBackend 也维护 `processing` 集合:dequeue 将消息移入该集合,ACK 再移除消息。它与其他后端使用相同的完成条件,但不提供进程重启后的持久化恢复。
## 状态模型
-队列长度不是单一数字。QueueFS 必须维护以下当前状态指标:
+队列长度不是单一数字。QueueFS 维护以下当前状态指标:
| 字段 | 含义 |
| --- | --- |
@@ -63,13 +64,12 @@ Python 不应再组合 backend 的 `pending` 和进程内计数来判断完成
- `requeue_count`
- `error_count`
-它们描述的是处理结果,而不是当前队列占用。本次修复中可以继续由处理层或指标层
-维护。如果需要将它们下沉到 QueueFS,必须先定义显式的处理结果协议,因为 backend
+它们描述的是处理结果,而不是当前队列占用。由处理层或指标层维护。如果需要将它们下沉到 QueueFS,必须先定义显式的处理结果协议,因为 backend
无法仅根据 dequeue 或 ACK 推断 handler 的处理结果。
## Backend 契约
-QueueFS 应提供一个原子状态操作:
+QueueFS `/status` 返回同一次原子读取取得的状态:
```json
{
@@ -88,7 +88,7 @@ QueueFS 控制文件名属于保留路径段,队列名不能以 `enqueue`、`d
- **SQLite:** 在同一个数据库快照中读取两个计数。
- **Cache:** 通过一个 Lua 脚本返回 `LLEN(pending)` 和
`ZCARD(processing)`。
-- **Memory:** 增加 processing 集合;dequeue 将消息移入该集合,ACK 从该集合
+- **Memory:** 维护 processing 集合;dequeue 将消息移入该集合,ACK 从该集合
删除消息。
`NamedQueue.get_status()` 只消费 backend 返回的状态快照。`wait_complete()` 和