Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat: refine OKF sidecar metadata handling and write protection (#4092)

MaojiaSheng24cc8c6ee92d
diff --git a/docs/design/l0-l1-okf-sidecars-rfc.md b/docs/design/l0-l1-okf-sidecars-rfc.md
new file mode 100644
index 000000000..6b80d1282
--- /dev/null
+++ b/docs/design/l0-l1-okf-sidecars-rfc.md
@@ -0,0 +1,261 @@
+# RFC: Upgrade L0/L1 Semantic Sidecars to OKF Markdown
+
+Status: Draft
+
+## Summary
+
+OpenViking today stores built-in context-layer summaries as directory-level Markdown sidecars:
+
+- `.abstract.md` for L0
+- `.overview.md` for L1
+
+This RFC proposes keeping those filenames and the visible Markdown body, while upgrading their internal document format to minimal OKF-style Markdown: YAML frontmatter metadata followed by the summary body.
+
+The change makes L0/L1 sidecars more self-describing without changing the public context-layer model described in `docs/zh/concepts/03-context-layers.md`.
+
+## Background
+
+OpenViking uses a three-layer context model:
+
+| Layer | Current file | Role |
+| --- | --- | --- |
+| L0 | `.abstract.md` | short summary for fast filtering and vector retrieval |
+| L1 | `.overview.md` | directory overview for navigation and rerank |
+| L2 | source files and subdirectories | full content loaded on demand |
+
+The current implementation treats `.abstract.md` and `.overview.md` as generated hidden sidecars. `SemanticProcessor` / `SemanticDagExecutor` generate them bottom-up, and the vector pipeline maps those two filenames to `ContextLevel.ABSTRACT` and `ContextLevel.OVERVIEW`.
+
+## Goals
+
+- Keep the existing L0/L1 file names and directory-level sidecar convention.
+- Add a small structured metadata block to L0/L1 documents.
+- Exclude metadata from embedding by default.
+- Preserve current preview behavior: `find`, `ls`, tree, and agent previews should show only the visible summary body unless explicitly reading the file.
+- Make metadata available through direct `read` / `get` access to the sidecar files.
+- metadata is protected from directly `write`.
+- Leave room for a future metadata embedding whitelist for index-able fields.
+
+## Non-Goals
+
+- Do not change L0/L1 into per-file sidecars.
+- Do not expose hidden sidecars in normal directory listings.
+- Do not redesign semantic generation prompts, metadata is hidden for re-generation.
+- Do not make all Markdown frontmatter globally invisible to embedding.
+- Do not require OKF parsing for ordinary user-authored Markdown resources.
+
+## Proposed Format
+
+L0 and L1 remain UTF-8 Markdown files with `.md` suffixes. Each generated file starts with YAML frontmatter:
+
+### L0 Example Across Surfaces
+
+Use one directory as the running example:
+
+```text
+viking://resources/images_2/
+```
+
+#### Raw Storage
+
+The stored `.abstract.md` keeps metadata and body together:
+
+```markdown
+---
+directory: viking://resources/images_2/
+source:
+ kind: http
+ uri: http://demo.com/demo.pdf
+generated_by:
+ component: SemanticProcessor
+ trigger: resource_ingest
+freshness:
+ total_entries: 161
+ sampled_entries: 32
+ unsampled_entries: 129
+ pending_child_changes: 0
+---
+
+This directory contains a collection of visual assets including screenshots, diagrams, logos, and QR codes related to a context database for AI agents, AI coding assistant tools, and cloud-based AI model management.
+```
+
+#### Preview Reads
+
+`abstract()`, `ls output=agent`, tree agent output, and `find` previews use the body only:
+
+```markdown
+This directory contains a collection of visual assets including screenshots, diagrams, logos, and QR codes related to a context database for AI agents, AI coding assistant tools, and cloud-based AI model management.
+```
+
+Normal `ls` still hides `.abstract.md` unless hidden files are explicitly requested.
+
+#### Full Reads
+
+`read()` / `get()` of the sidecar file return the raw stored document, including frontmatter and body. This is the path for callers that intentionally need metadata.
+
+#### Embedding Text
+
+Embedding uses the body plus whitelisted metadata. The initial whitelist contains `directory`, so the text sent to the embedder is equivalent to:
+
+```text
+---
+directory: viking://resources/images_2/
+---
+
+This directory contains a collection of visual assets including screenshots, diagrams, logos, and QR codes related to a context database for AI agents, AI coding assistant tools, and cloud-based AI model management.
+```
+
+`source`, `generated_by`, and `freshness` are intentionally absent from this embedding text.
+
+#### Regeneration Input
+
+When a parent summary is regenerated from this L0, the semantic pipeline uses the body only:
+
+```text
+This directory contains a collection of visual assets including screenshots, diagrams, logos, and QR codes related to a context database for AI agents, AI coding assistant tools, and cloud-based AI model management.
+```
+
+Metadata is not prompt input for re-summarization. `freshness` may still be updated out of band to indicate that the stored body is stale relative to child changes.
+
+L1 uses the same storage and access rules. Its body is longer and navigation-oriented, but metadata handling is identical.
+
+The metadata block is intentionally narrower than general OKF wiki pages. L0/L1 are generated context abstractions, so metadata should only carry machine-facing fields that are not already available from file attributes such as `stat`, and not repeat the visible summary itself.
+
+The initial metadata should stay small and deterministic. Timestamps are intentionally omitted at first to avoid rewriting sidecars when semantic content has not changed.
+
+Existing L0/L1 content length limits apply to the Markdown body only. Metadata is outside those limits and should not be truncated by summary-size enforcement.
+
+## Metadata Fields
+
+The initial metadata set should stay small:
+
+| Field | Purpose | Notes |
+| --- | --- | --- |
+| `directory` | Directory represented by this L0/L1 sidecar | Initial embedding whitelist field |
+| `source` | Optional original source, when known | Top-level import root only unless a nested source boundary exists |
+| `generated_by` | Component and coarse trigger that created or updated the sidecar | Operational metadata, not content provenance |
+| `freshness` | Child counts, sampling coverage, and known pending child changes | A freshness signal; not a replacement for semantic refresh |
+
+`freshness` counts direct child entries, not the total recursive subtree size. Files and child directories share the same counters because both contribute one direct input to the parent summary. For large directories, it should distinguish total input size from the subset actually read for summary generation. A directory with many entries may record `total_entries: 161`, `sampled_entries: 32`, and `unsampled_entries: 129`. `freshness.pending_child_changes > 0` means the body is still readable, but known to lag behind lower-level changes.
+
+Sampling should be deterministic for a stable tree so repeated refreshes do not rewrite sidecars unnecessarily. The first policy can be simple: summarize all direct children up to a threshold, and for larger directories use a bounded sample that preserves useful ordering and diversity. Freshness metadata should make that choice visible without forcing the body to enumerate unsampled files.
+
+Metadata should not duplicate information already available through `stat`, such as file name, file size, mode, modified time, or lock state.
+
+## Embedding Behavior
+
+The embedding input for generated L0/L1 sidecars should be the visible Markdown body plus explicitly whitelisted metadata fields.
+
+The initial whitelist includes `directory`, because the directory URI helps retrieval connect a summary to its location in the context tree. Operational and provenance fields remain excluded by default, including `source`, `generated_by`, and `freshness`.
+
+Conceptually:
+
+```text
+raw sidecar bytes
+ -> parse OKF frontmatter
+ -> body_markdown
+ -> append whitelisted metadata fields (`directory` initially)
+ -> embedding text
+```
+
+This keeps the semantic vector space focused on user-facing summaries and stable location context, while avoiding accidental ranking changes from source URLs, generator details, or freshness counters.
+
+## Read and Preview Semantics
+
+Direct file reads of sidecar files remain raw: callers receive frontmatter plus body.
+
+Semantic accessors return body-only content. Preview surfaces also use body-only content:
+
+- `find` result summaries
+- `ls output=agent`
+- tree output in agent mode
+- search/rerank preview snippets
+
+Normal `ls` behavior remains unchanged: hidden sidecars are not listed unless the caller explicitly asks for hidden entries.
+
+## Compatibility
+
+Existing sidecars without frontmatter remain valid. Readers should treat them as legacy Markdown and return the full content as body.
+
+Writers should emit OKF-formatted sidecars after the feature lands. There is no need for an eager migration job. Existing directories migrate naturally when semantic refresh rewrites their L0/L1 files.
+
+The current L0 extraction rule should continue to operate on the L1 body, not the raw OKF document. This preserves the current convention that L0 is derived from the brief paragraph before the first `##` section in L1.
+
+## Implementation Sketch
+
+Introduce a small semantic-sidecar document helper near the existing sidecar write/read code:
+
+- `render_semantic_sidecar(level, dir_uri, body, metadata) -> str`
+- `parse_semantic_sidecar(raw) -> {metadata, body}`
+- `body_for_preview(raw) -> str`
+- `body_for_embedding(raw, whitelist=()) -> str`
+
+Use it in these paths:
+
+| Area | Required change |
+| --- | --- |
+| Sidecar writeback | Render OKF frontmatter before writing `.abstract.md` / `.overview.md` |
+| Source propagation | Carry optional source metadata to the import root sidecars when available |
+| Generation metadata | Record the component and trigger that created or updated each sidecar |
+| Freshness accounting | Write direct child input counters during semantic generation |
+| Large directory sampling | Record total direct children, sampled children, and unsampled count when generation uses a bounded sample |
+| Deferred changes | Increment pending child-change counters when filesystem changes do not immediately refresh the parent summary |
+| `abstract()` / `overview()` | Strip OKF frontmatter before returning semantic accessor content |
+| Incremental summary reuse | Parse `.overview.md` body before extracting file summaries |
+| L0 derivation | Extract abstract from L1 body |
+| Vectorization | Enqueue the body plus explicitly whitelisted metadata for L0/L1 embeddings |
+| Preview formatting | Keep using semantic accessor/body-only content |
+
+The parser should be tolerant: valid YAML object frontmatter is metadata; missing frontmatter means legacy body; malformed frontmatter should not silently enter embeddings. For generated sidecars, malformed frontmatter should be treated as a processing error.
+
+Size-limit enforcement should parse the sidecar and apply existing L0/L1 limits only to the body before rendering metadata back around it. Metadata size should remain bounded by field policy rather than by the summary body limit.
+
+## Test Plan
+
+Format and compatibility:
+
+- Unit-test rendering and parsing of OKF sidecars.
+- Verify legacy sidecars without frontmatter still read as body-only content.
+- Verify malformed generated sidecar frontmatter fails loudly instead of being embedded as body text.
+- Verify ordinary user-authored Markdown frontmatter is not affected by semantic-sidecar parsing.
+
+Read and preview behavior:
+
+- Verify `abstract()` and `overview()` return body-only content.
+- Verify direct `read_file()` returns raw frontmatter plus body.
+- Verify `ls output=agent`, tree agent output, and `find` previews do not include metadata fields.
+- Verify normal `ls` still hides `.abstract.md` / `.overview.md` unless hidden files are requested.
+
+Metadata generation:
+
+- Verify optional `source` metadata is preserved on the generated import root and not repeated in nested directories unless a nested source boundary exists.
+- Verify `generated_by` records the component and trigger for generated sidecars.
+- Verify L0/L1 size limits truncate only the body and preserve metadata.
+- Verify direct writes cannot mutate protected metadata independently from semantic sidecar generation.
+
+Freshness and sampling:
+
+- Verify freshness counters match direct child entries used for generation.
+- Verify large directories record deterministic sampling coverage, including total, sampled, and unsampled direct entries.
+- Verify repeated refreshes on an unchanged large directory produce the same sample metadata and do not rewrite sidecars unnecessarily.
+- Verify deferred child changes update `pending_child_changes` without changing the visible summary body.
+- Verify a completed parent refresh resets `pending_child_changes` and updates freshness coverage.
+
+Semantic reuse and embedding:
+
+- Verify L0 extraction ignores frontmatter in L1.
+- Verify incremental refresh can reuse summaries from an OKF-formatted `.overview.md`.
+- Verify regeneration prompts use body-only child summaries and do not include metadata.
+- Verify vectorization text for `.abstract.md` and `.overview.md` includes whitelisted `directory` metadata.
+- Verify vectorization text excludes `source`, `generated_by`, and `freshness`.
+
+## Decisions
+
+- Do not add a dedicated `read_metadata()` API in the first version. Direct `read()` / `get()` of the sidecar file is enough for callers that intentionally need metadata.
+- Use the metadata embedding policy described above: `directory` is included initially; `source`, `generated_by`, and `freshness` are excluded.
+- Include OKF metadata in Git diffs as-is. Body-only diff rendering can be a future viewer feature, not part of the storage format.
+
+## Future Work: Freshness-Aware Parent Bubbling
+
+The current implementation schedules a parent refresh after every successful resource/skill semantic task, even when the newly generated child summary is unchanged. It marks the parent `pending_child_changes` before enqueue so freshness remains accurate, but unconditional bubbling is not the intended final scheduling policy.
+
+Future work should use freshness state to control bubbling frequency. Candidate inputs include `pending_child_changes`, sampling coverage, direct-child change volume, whether the child L0 body actually changed, and recent parent-refresh state. The scheduler may coalesce changes, apply thresholds, or use a time window to reduce repeated refreshes and upward write amplification in hot directory trees while preserving eventual consistency.
diff --git a/docs/en/api/12-content.md b/docs/en/api/12-content.md
index 00b27b1b1..a6f8bddc7 100644
--- a/docs/en/api/12-content.md
+++ b/docs/en/api/12-content.md
@@ -226,7 +226,7 @@ Update an existing file, or create a new one when `mode="create"`, and automatic
- `replace` and `append` require the file to exist; `create` targets a new file and returns `409 Conflict` when the path already exists. Directories are always rejected.
- `create` only accepts text-writable extensions: `.md`, `.txt`, `.json`, `.yaml`, `.yml`, `.toml`, `.py`, `.js`, `.ts`. Parent directories are created automatically.
-- Derived semantic files cannot be written directly: `.abstract.md`, `.overview.md`, `.relations.json`.
+- `.relations.json` cannot be written directly. Existing `.abstract.md` and `.overview.md` bodies may be updated, but public APIs cannot create them. A body-only request preserves stored OKF metadata; a full-OKF request must match the stored metadata. Unknown metadata fields are silently dropped. A sidecar body write rebuilds only the directory's existing L0/L1 vectors and does not regenerate semantics.
- File content is updated before the API returns. `wait` only controls whether the call waits for semantic/vector refresh to finish.
- The public API no longer accepts `regenerate_semantics` or `revectorize`; write always refreshes related semantics and vectors.
@@ -358,6 +358,7 @@ Each operation contains:
- Every non-idempotent precondition is checked under the target tree lock before the first new write. A mismatch returns `409 Conflict`.
- If a target already contains the requested bytes, it is reported as `unchanged`; retrying the same request can therefore repeat a failed refresh without rewriting matching content.
- The API prevents precondition conflicts from causing new writes, but an underlying I/O failure can still leave writes completed earlier in the batch visible.
+- Existing `.abstract.md` and `.overview.md` bodies may be updated with `replace_if_hash`. OpenViking preserves and validates protected OKF metadata, rejects `create_if_absent` for sidecars, and rebuilds only the directory's existing L0/L1 vectors for these operations.
**Python SDK**
diff --git a/docs/en/concepts/02-context-types.md b/docs/en/concepts/02-context-types.md
index 85a18f965..1a680c355 100644
--- a/docs/en/concepts/02-context-types.md
+++ b/docs/en/concepts/02-context-types.md
@@ -101,13 +101,15 @@ Skills are capabilities that Agents can invoke, belonging to the **AgentDefinedC
```
viking://user/skills/{skill-name}/ # Default storage path
├── .abstract.md # L0: Short description
-├── SKILL.md # L1: Detailed overview
-└── scripts # L2: Full definition
+├── .overview.md # L1: Directory overview (after generation)
+├── SKILL.md # L2: Skill definition
+└── scripts # L2: Supporting implementation
viking://agent/skills/{skill-name}/ # Override via --uri, public/shared (account global)
├── .abstract.md # L0: Short description
-├── SKILL.md # L1: Detailed overview
-└── scripts # L2: Full definition
+├── .overview.md # L1: Directory overview (after generation)
+├── SKILL.md # L2: Skill definition
+└── scripts # L2: Supporting implementation
```
### AgentDefinedContextType Subtypes
diff --git a/docs/en/concepts/03-context-layers.md b/docs/en/concepts/03-context-layers.md
index ab1d2161c..0140e2588 100644
--- a/docs/en/concepts/03-context-layers.md
+++ b/docs/en/concepts/03-context-layers.md
@@ -1,31 +1,30 @@
# Context Layers (L0/L1/L2)
-OpenViking uses a three-layer information model to balance retrieval efficiency and content completeness.
+OpenViking uses a three-layer information model to balance retrieval efficiency, navigation, and full-content fidelity.
## Overview
-| Layer | Name | File | Token Limit | Purpose |
-|-------|------|------|-------------|---------|
-| **L0** | Abstract | `.abstract.md` | ~100 tokens | Vector search, quick filtering |
-| **L1** | Overview | `.overview.md` | ~2k tokens | Rerank, content navigation |
-| **L2** | Detail | Original files/subdirs | Unlimited | Full content, on-demand loading |
+| Layer | Name | Storage | Default body limit | Purpose |
+| --- | --- | --- | --- | --- |
+| **L0** | Abstract | `.abstract.md` in a directory | 256 characters | Vector retrieval, quick filtering |
+| **L1** | Overview | `.overview.md` in a directory | 4000 characters | Rerank, content navigation |
+| **L2** | Detail | Original files and subdirectories | No uniform limit | Full content, on-demand loading |
-## L0: Abstract
+L0 and L1 are **directory-level semantic sidecars**. They describe a directory; OpenViking does not create a matching L0/L1 sidecar for every ordinary file. File summaries are inputs aggregated into the containing directory's L1.
-The most concise representation of content, used for vector retrieval and quick filtering.
+L0 and L1 are normally generated together, but either one may exist independently. For example, `mkdir(description=...)` initially creates only L0, so a directory with `.abstract.md` but no `.overview.md` is valid. Reads and vector rebuilds process only the levels that actually exist.
-### Characteristics
+The body limits are configured by `semantic.abstract_max_chars` and `semantic.overview_max_chars`; the table shows their defaults. These limits apply to the Markdown body only and do not truncate sidecar metadata.
-- **Ultra-short**: Max ~100 tokens
-- **Quick perception**: Allows Agent to quickly perceive content
+## L0: Abstract
-### Example
+L0 is the shortest representation of a directory, used for vector retrieval and quick relevance decisions.
```markdown
API authentication guide covering OAuth 2.0, JWT tokens, and API keys for secure access.
```
-### API
+Semantic accessors return only the visible body:
```python
abstract = client.abstract("viking://resources/docs/auth")
@@ -33,156 +32,166 @@ abstract = client.abstract("viking://resources/docs/auth")
## L1: Overview
-Comprehensive summary with navigation guidance, used for Rerank and understanding access methods.
-
-### Characteristics
-
-- **Moderate length**: ~1k tokens
-- **Navigation guide**: Tells Agent how to access detailed content
-
-### Example
+L1 provides a broader directory summary and navigation, supporting rerank and the decision to load L2.
```markdown
-# Authentication Guide Overview
+# Authentication Guide
-This guide covers three authentication methods for the API:
+This directory covers the primary API authentication methods.
-## Sections
-- **OAuth 2.0** (L2: oauth.md): Complete OAuth flow with code examples
-- **JWT Tokens** (L2: jwt.md): Token generation and validation
-- **API Keys** (L2: api-keys.md): Simple key-based authentication
+## Quick Navigation
-## Key Points
-- OAuth 2.0 recommended for user-facing applications
-- JWT for service-to-service communication
-
-## Access
-Use `read("viking://resources/docs/auth/oauth.md")` for full documentation.
+- `oauth.md`: OAuth 2.0 flow and examples
+- `jwt.md`: token generation and validation
+- `api-keys.md`: API key authentication
```
-### API
-
```python
overview = client.overview("viking://resources/docs/auth")
```
-## L2: Detail
-
-Complete original content, loaded only when needed.
-
-### Characteristics
+L0 is extracted from the L1 body: the Brief Description paragraph after the H1 title and before the first `##` heading. YAML frontmatter is not part of this extraction.
-- **Full content**: No token limit
-- **On-demand loading**: Read only when confirmed necessary
-- **Original format**: Preserves source structure
+## L2: Detail
-### API
+L2 is the original or fully parsed content, loaded only when needed and retaining its source format and structure.
```python
content = client.read("viking://resources/docs/auth/oauth.md")
```
-## Generation Mechanism
+## Directory Structure
-### When Generated
+A semantically processed directory commonly looks like this:
-- **When adding resources**: After Parser parsing, SemanticQueue generates asynchronously
-- **When archiving sessions**: L0/L1 generated for history segments during compression
+```text
+viking://resources/docs/auth/
+├── .abstract.md # L0, hidden directory sidecar
+├── .overview.md # L1, hidden directory sidecar
+├── .relations.json # Relation data
+├── oauth.md # L2
+├── jwt.md # L2
+└── api-keys.md # L2
+```
-### Who Generates
+Normal `ls` hides `.abstract.md` and `.overview.md`. The two files are not guaranteed to coexist, so callers should not assume every directory always has both sidecars.
-| Component | Responsibility |
-|-----------|----------------|
-| **SemanticProcessor** | Traverses directories bottom-up, generates L0/L1 for each |
-| **SessionCompressor** | Generates L0/L1 for archived session history |
+## OKF Sidecar Format
-### Generation Order
+New L0/L1 files use minimal OKF Markdown: YAML frontmatter followed by the visible Markdown body.
-```
-Leaf nodes → Parent directories → Root (bottom-up)
+```markdown
+---
+directory: viking://resources/docs/auth/
+source:
+ kind: http
+ uri: https://example.com/auth.pdf
+generated_by:
+ component: SemanticProcessor
+ trigger: resource_ingest
+freshness:
+ total_entries: 3
+ sampled_entries: 3
+ unsampled_entries: 0
+ pending_child_changes: 0
+---
+
+API authentication guide covering OAuth 2.0, JWT tokens, and API keys.
```
-Child directory L0s are aggregated into parent L1, forming hierarchical navigation.
+The initial metadata fields are:
-## Directory Structure
+| Field | Meaning |
+| --- | --- |
+| `directory` | URI of the directory represented by the sidecar |
+| `source` | Optional import source, normally stored only on the import root |
+| `generated_by` | Generator component and coarse trigger |
+| `freshness` | Direct-child coverage and known pending changes |
-Each directory follows a unified file structure:
+Known fields are schema-validated. Unknown top-level fields and unknown nested keys inside known objects are silently dropped; they do not enter previews, embeddings, canonical writeback, or metadata-protection comparisons. Existing sidecars without frontmatter remain valid legacy Markdown. Malformed YAML, a missing required `directory`, or an invalid known-field type still fails explicitly.
-```
-viking://resources/docs/auth/
-├── .abstract.md # L0: ~100 tokens
-├── .overview.md # L1: ~1k tokens
-├── .relations.json # Related resources
-├── oauth.md # L2: Full content
-├── jwt.md # L2: Full content
-└── api-keys.md # L2: Full content
-```
+## Behavior Across Read Surfaces
-## Multimodal Support
+The same sidecar has different views depending on the access surface:
-- **L0/L1**: Always text (Markdown)
-- **L2**: Can be any format (text, image, video, audio)
+| Access | Returned content |
+| --- | --- |
+| `abstract()` / `overview()` | Markdown body only |
+| `find` and search/rerank previews | Markdown body only |
+| `ls output=agent` and tree agent output | Markdown body only |
+| Direct `read(".../.abstract.md")` | Raw frontmatter and body |
+| Normal `ls` | Hidden sidecars are omitted |
-For binary content, L0/L1 describe in text:
+Parent-summary generation also consumes only child L0 bodies. `source`, `generated_by`, and `freshness` are not included in summarization prompts.
-```markdown
-# Image L0
-Product screenshot showing login page with OAuth buttons.
+## Embedding Metadata Whitelist
-# Image L1
-## Image: Login Page Screenshot
+L0/L1 embedding input contains the body plus explicitly whitelisted metadata. The initial whitelist contains only `directory`:
-This screenshot shows the application's login page with:
-- Google OAuth button (top)
-- GitHub OAuth button (middle)
-- Email/password form (bottom)
+```markdown
+---
+directory: viking://resources/docs/auth/
+---
-Dimensions: 1920x1080, Format: PNG
+API authentication guide covering OAuth 2.0, JWT tokens, and API keys.
```
-Directory Structure
+`source`, `generated_by`, `freshness`, and unknown fields are excluded. Normal vectorization and admin `vectors_only` reindex use the same policy so rebuilding the index does not change retrieval input. The L1 rerank scalar remains the plain L1 body.
-```
-...
-└── Chapter 3 Developer Notes/
- ├── .abstract.md
- ├── .overview.md
- ├── content.md
- └── Video Attachment 1 - Developer Notes/ ← Recursive expansion of attachment info
- ├── .abstract.md
- ├── .overview.md
- ├── audio_and_subtitles.md
- ├── developer_training.mp4
- └── video_segments/
- ├── developer_training_0s-30s.mp4
- └── developer_training_30s-60s.mp4
-```
+## Freshness and Stable Sampling
+`freshness` counts a directory's **direct children**, not its entire recursive subtree:
+- `total_entries`: total direct files and direct subdirectories contributing to directory semantics.
+- `sampled_entries`: direct entries used for this summary.
+- `unsampled_entries`: direct entries not sampled, with `sampled + unsampled = total`.
+- `pending_child_changes`: known changed direct entries not yet reflected in the current body.
-## Best Practices
+When the direct-entry count exceeds `semantic.sidecar_sample_size` (32 by default), OpenViking uses deterministic, order-preserving stable sampling. Repeated refreshes of an unchanged tree choose the same sample, avoiding noisy body rewrites and Git diffs.
-| Scenario | Recommended Layer |
-|----------|-------------------|
-| Quick relevance check | L0 |
-| Understand content scope | L1 |
-| Detailed information extraction | L2 |
-| Building context for LLM | L1 (usually sufficient) |
+`pending_child_changes > 0` means the body is still readable but is known to lag behind lower-level changes. A successful parent refresh resets the value to 0 as part of the new coverage metadata.
-### Token Budget Management
+Currently, every successful resource/skill semantic task schedules the next parent refresh and marks the parent pending before enqueue, continuing to the namespace-root boundary.
-```python
-# Use L1 first, load L2 only when needed
-overview = client.overview(uri)
+> **TODO: control bubbling frequency with freshness**
+>
+> The current implementation attempts to bubble after every successful resource/skill semantic task, even when the newly generated child summary is unchanged. This is not the intended final scheduling policy. A future implementation should use `freshness` to coalesce, threshold, or time-window parent refreshes—for example by considering `pending_child_changes`, sampling coverage, direct-child change volume, and recent refresh state. The goal is to reduce repeated refreshes and upward write amplification in hot directories while preserving eventual consistency.
+
+## Write Protection
+
+L0/L1 bodies can be updated through public `write` / `batch_write`, while metadata remains protected:
-if needs_more_detail(overview):
- content = client.read(uri)
+- The target sidecar must already exist; public APIs cannot directly create a new `.abstract.md` or `.overview.md`.
+- A body-only request inherits the stored metadata and writes canonical OKF.
+- A full-OKF request must preserve all known metadata; changing a protected field fails.
+- Unknown metadata fields are silently dropped.
+- `append` appends only to the body, never to frontmatter.
+- A body update rebuilds only the directory levels that currently exist and does not regenerate semantics, preventing the newly written body from being overwritten.
+
+## Generation
+
+SemanticProcessor processes directories bottom-up:
+
+```text
+file summaries → leaf L1 → leaf L0 → parent directories → namespace boundary
```
+Child-directory L0 bodies are aggregated into the parent L1. Memory directories use the same SemanticProcessor entry point, although the current parent-bubbling path applies only to resource/skill. Multimodal files first produce text summaries, which contribute to the containing directory's L0/L1 like other file summaries; OpenViking does not create per-file L0/L1 sidecars for each image, audio file, or video.
+
+## Best Practices
+
+| Scenario | Recommended layer |
+| --- | --- |
+| Quick relevance check | L0 |
+| Understand a directory's scope | L1 |
+| Detailed information extraction | L2 |
+| Build initial LLM context | L1, then load L2 when needed |
+| Inspect sidecar source or freshness | Read the raw sidecar directly |
+
## Related Documents
- [Architecture Overview](./01-architecture.md) - System architecture
-- [Context Types](./02-context-types.md) - Three context types
+- [Context Types](./02-context-types.md) - Context types
- [Viking URI](./04-viking-uri.md) - URI specification
-- [Retrieval Mechanism](./07-retrieval.md) - Retrieval process details
-- [Context Extraction](./06-extraction.md) - L0/L1 generation details
+- [Context Extraction](./06-extraction.md) - L0/L1 generation flow
+- [Retrieval Mechanism](./07-retrieval.md) - Retrieval details
diff --git a/docs/en/concepts/06-extraction.md b/docs/en/concepts/06-extraction.md
index e256e89ed..78183cd39 100644
--- a/docs/en/concepts/06-extraction.md
+++ b/docs/en/concepts/06-extraction.md
@@ -119,9 +119,21 @@ Leaf directories → Parent directories → Root
2. **Collect child directory abstracts**: Read generated .abstract.md
3. **Generate .overview.md**: LLM generates L1 overview
4. **Extract .abstract.md**: Extract L0 from overview
-5. **Write files**: Save to AGFS
+5. **Write files**: Store the body and protected metadata as OKF Markdown
6. **Vectorize**: Create Context and queue to EmbeddingQueue
+L0/L1 are directory sidecars, not per-file sidecars. Parent-summary generation consumes only child L0 bodies; OKF frontmatter is excluded from prompts. Embedding input contains the body and the whitelisted `directory`; `source`, `generated_by`, and `freshness` are excluded.
+
+### Freshness, Sampling, and Parent Refresh
+
+Each generation records direct-child `total_entries`, `sampled_entries`, and `unsampled_entries`. When the direct-child count exceeds `semantic.sidecar_sample_size` (32 by default), OpenViking uses deterministic stable sampling. `pending_child_changes` increases when a known child change is not yet reflected in the parent body and resets to 0 after a successful refresh.
+
+Currently, each successful resource/skill semantic task schedules the next parent refresh and marks that parent pending before enqueue, continuing to the namespace-root boundary.
+
+> **TODO: control bubbling frequency with freshness**
+>
+> Bubbling after every successful task can create repeated refreshes and upward write amplification in hot, deeply nested directories. A future scheduler should use `pending_child_changes`, sampling coverage, direct-child change volume, and recent refresh state to coalesce, threshold, or time-window parent refreshes while preserving eventual consistency.
+
### Processing Limits
| Parameter | Default | Description |
@@ -129,6 +141,7 @@ Leaf directories → Parent directories → Root
| `max_concurrent_llm` | 10 | Concurrent LLM calls |
| `max_images_per_call` | 10 | Max images per VLM call |
| `max_sections_per_call` | 20 | Max sections per VLM call |
+| `sidecar_sample_size` | 32 | Maximum direct-child sample used for one directory summary |
## Code Skeleton Extraction
diff --git a/docs/en/getting-started/01-introduction.md b/docs/en/getting-started/01-introduction.md
index c334f8f70..8779d310f 100644
--- a/docs/en/getting-started/01-introduction.md
+++ b/docs/en/getting-started/01-introduction.md
@@ -62,23 +62,25 @@ client.overview("viking://...") # Get L1 overview
Stuffing massive context into prompts all at once is not only expensive but also risks exceeding model windows and introducing noise. OpenViking automatically processes context into three levels upon ingestion:
-| Level | Name | Token Limit | Purpose |
-|-------|------|-------------|---------|
-| **L0** | Abstract | ~100 tokens | Vector search, quick filtering |
-| **L1** | Overview | ~2k tokens | Rerank, content navigation |
-| **L2** | Detail | Unlimited | Full content, on-demand loading |
+| Level | Name | Default body limit | Purpose |
+| --- | --- | --- | --- |
+| **L0** | Abstract | 256 characters | Vector search, quick filtering |
+| **L1** | Overview | 4000 characters | Rerank, content navigation |
+| **L2** | Detail | No uniform limit | Full content, on-demand loading |
```
viking://resources/my_project/
├── .abstract.md # L0 layer: abstract
├── .overview.md # L1 layer: overview
├── docs/
-│ ├── .abstract.md # Each directory has L0/L1 layers
+│ ├── .abstract.md # Semantically processed directories commonly have L0/L1
│ ├── .overview.md
│ └── api.md # L2 layer: full content
└── src/
```
+L0/L1 are directory sidecars, not per-file sidecars, and they are not guaranteed to coexist. See [Context Layers](../concepts/03-context-layers.md).
+
### 3. Directory Recursive Retrieval
Single vector retrieval struggles with complex query intents. OpenViking implements an innovative **directory recursive retrieval strategy**:
diff --git a/docs/zh/api/12-content.md b/docs/zh/api/12-content.md
index 3fe115438..a3fa24529 100644
--- a/docs/zh/api/12-content.md
+++ b/docs/zh/api/12-content.md
@@ -226,7 +226,7 @@ openviking read viking://resources/docs/api.md
- `replace` 和 `append` 要求文件已存在;`create` 仅用于创建新文件,目标路径已存在时返回 `409 Conflict`。目录始终会被拒绝。
- `create` 只允许以下文本类扩展名:`.md`、`.txt`、`.json`、`.yaml`、`.yml`、`.toml`、`.py`、`.js`、`.ts`。父目录会自动创建。
-- 不允许直接写入派生语义文件:`.abstract.md`、`.overview.md`、`.relations.json`。
+- `.relations.json` 不允许直接写入。已存在的 `.abstract.md` / `.overview.md` 可以修改正文,但不能通过公共 API 创建;只提交正文时会保留现有 OKF metadata,提交完整 OKF 时 metadata 必须与存量值一致。未知 metadata 字段会静默丢弃。sidecar 正文写入只重建该目录实际存在的 L0/L1 向量,不触发语义重新生成。
- 文件内容会在 API 返回前完成更新;`wait` 只控制是否等待语义/向量刷新完成。
- 公共 API 已不再接受 `regenerate_semantics` 或 `revectorize`;写入后一定会自动刷新相关语义与向量。
@@ -358,6 +358,7 @@ openviking write viking://resources/docs/api.md \
- 系统会在目标 tree lock 内、首次产生新写入之前检查所有非幂等前置条件;不满足时返回 `409 Conflict`。
- 如果目标已经包含期望字节,则归入 `unchanged`;因此刷新失败后可以安全重试,而不会重复写入相同内容。
- API 能保证前置条件冲突不会产生新写入,但底层 I/O 中途失败时,本批次较早完成的写入仍可能已经可见。
+- 已存在的 `.abstract.md` / `.overview.md` 可以通过 `replace_if_hash` 修改正文。系统保留并校验受保护的 OKF metadata,拒绝 `create_if_absent` 创建 sidecar;这类操作只重建对应目录实际存在的 L0/L1 向量。
**Python SDK**
diff --git a/docs/zh/concepts/02-context-types.md b/docs/zh/concepts/02-context-types.md
index cf92689de..819f986f3 100644
--- a/docs/zh/concepts/02-context-types.md
+++ b/docs/zh/concepts/02-context-types.md
@@ -101,13 +101,15 @@ results = await client.find(
```
viking://user/skills/{skill-name}/ # 默认存储路径
├── .abstract.md # L0: 简短描述
-├── SKILL.md # L1: 详细概览
-└── scripts # L2: 完整定义
+├── .overview.md # L1: 目录概览(生成后)
+├── SKILL.md # L2: 技能定义
+└── scripts # L2: 附加实现
viking://agent/skills/{skill-name}/ # 通过 --uri 覆盖,公开共享(account 全局)
├── .abstract.md # L0: 简短描述
-├── SKILL.md # L1: 详细概览
-└── scripts # L2: 完整定义
+├── .overview.md # L1: 目录概览(生成后)
+├── SKILL.md # L2: 技能定义
+└── scripts # L2: 附加实现
```
### AgentDefinedContextType 子类型
diff --git a/docs/zh/concepts/03-context-layers.md b/docs/zh/concepts/03-context-layers.md
index 7c3cbf299..06a5aefa4 100644
--- a/docs/zh/concepts/03-context-layers.md
+++ b/docs/zh/concepts/03-context-layers.md
@@ -1,188 +1,197 @@
-# 上下文层级 (L0/L1/L2)
+# 上下文层级(L0/L1/L2)
-OpenViking 使用三层信息模型来平衡检索效率和内容完整性。
+OpenViking 使用三层信息模型,在检索效率、导航能力和原始内容完整性之间取得平衡。
## 概览
-| 层级 | 名称 | 文件 | Token 限制 | 用途 |
-|------|------|------|-----------|------|
-| **L0** | 摘要 | `.abstract.md` | ~100 tokens | 向量搜索、快速过滤 |
-| **L1** | 概览 | `.overview.md` | ~2k tokens | Rerank 精排、内容导航 |
-| **L2** | 详情 | 原始文件/子目录 | 无限制 | 完整内容、按需加载 |
+| 层级 | 名称 | 存储形式 | 默认正文上限 | 用途 |
+| --- | --- | --- | --- | --- |
+| **L0** | 摘要 | 目录内的 `.abstract.md` | 256 字符 | 向量检索、快速过滤 |
+| **L1** | 概览 | 目录内的 `.overview.md` | 4000 字符 | Rerank、内容导航 |
+| **L2** | 详情 | 原始文件和子目录 | 无统一上限 | 完整内容、按需加载 |
-## L0: 摘要
+L0/L1 是**目录级语义 sidecar**。它们描述一个目录,不是为每个普通文件创建的同名伴生文件。文件摘要会作为输入,聚合到所在目录的 L1 中。
-内容的最精简表示,用于快速筛选。
+L0 和 L1 通常成对生成,但系统允许只存在其中一个。例如,`mkdir(description=...)` 会先创建 L0,因此只有 `.abstract.md` 的目录也是合法状态。读取和向量重建只处理实际存在的层级。
-### 特点
+正文上限由 `semantic.abstract_max_chars` 和 `semantic.overview_max_chars` 配置;上表是默认值。限制只作用于 Markdown 正文,不会截断 sidecar 元数据。
-- **超短**:最多 ~100 tokens
-- **快速感知**:让 Agent 快速感知到
+## L0:摘要
-### 示例
+L0 是目录内容的最精简表示,用于向量召回和快速相关性判断。
```markdown
API 认证指南,涵盖 OAuth 2.0、JWT 令牌和 API 密钥的安全访问方式。
```
-### API
+通过语义 accessor 读取时,只返回可见正文:
```python
abstract = client.abstract("viking://resources/docs/auth")
```
-## L1: 概览
+## L1:概览
-包含内容导航的综合摘要,用于理解访问方式。
-
-### 特点
-
-- **适中长度**:~1k tokens
-- **导航指引**:告诉 Agent 如何访问详细内容
-
-### 示例
+L1 提供更完整的目录摘要和导航信息,用于 Rerank 和决定是否继续加载 L2。
```markdown
-# 认证指南概览
+# 认证指南
-本指南涵盖 API 的三种认证方式:
+本目录介绍 API 的主要认证方式。
-## 章节
-- **OAuth 2.0** (L2: oauth.md): 完整 OAuth 流程和代码示例
-- **JWT 令牌** (L2: jwt.md): 令牌生成和验证
-- **API 密钥** (L2: api-keys.md): 简单的密钥认证
+## 快速导航
-## 要点
-- OAuth 2.0 推荐用于面向用户的应用
-- JWT 用于服务间通信
-
-## 访问
-使用 `read("viking://resources/docs/auth/oauth.md")` 获取完整文档。
+- `oauth.md`:OAuth 2.0 流程和代码示例
+- `jwt.md`:令牌生成和验证
+- `api-keys.md`:API Key 认证
```
-### API
-
```python
overview = client.overview("viking://resources/docs/auth")
```
-## L2: 详情
+L0 从 L1 正文中提取:取 H1 标题之后、第一个 `##` 标题之前的 Brief Description 段落。YAML frontmatter 不参与提取。
-完整的原始内容,仅在需要时加载。
+## L2:详情
-### 特点
-
-- **完整内容**:无 Token 限制
-- **按需加载**:只在确定需要时读取
-- **原始格式**:保留源结构
-
-### API
+L2 是原始文件或解析后的完整内容,只在需要时加载,并保留源格式和结构。
```python
content = client.read("viking://resources/docs/auth/oauth.md")
```
-## 生成机制
+## 目录结构
-### 何时生成
+一个已完成语义处理的目录通常如下:
-- **添加资源时**:Parser 解析后,SemanticQueue 异步生成
-- **会话归档时**:压缩旧消息时生成历史片段的 L0/L1
+```text
+viking://resources/docs/auth/
+├── .abstract.md # L0,隐藏的目录级 sidecar
+├── .overview.md # L1,隐藏的目录级 sidecar
+├── .relations.json # 关系数据
+├── oauth.md # L2
+├── jwt.md # L2
+└── api-keys.md # L2
+```
-### 由谁生成
+普通 `ls` 默认隐藏 `.abstract.md` 和 `.overview.md`。它们不一定同时存在;不要依赖“每个目录始终具有两个 sidecar”的假设。
-| 组件 | 职责 |
-|------|------|
-| **SemanticProcessor** | 自底向上遍历目录,为每个目录生成 L0/L1 |
-| **SessionCompressor** | 为归档的会话历史生成 L0/L1 |
+## OKF sidecar 格式
-### 生成顺序
+新生成的 L0/L1 使用最小 OKF Markdown:YAML frontmatter 加可见 Markdown 正文。
-```
-叶子节点 → 父目录 → 根目录(自底向上)
+```markdown
+---
+directory: viking://resources/docs/auth/
+source:
+ kind: http
+ uri: https://example.com/auth.pdf
+generated_by:
+ component: SemanticProcessor
+ trigger: resource_ingest
+freshness:
+ total_entries: 3
+ sampled_entries: 3
+ unsampled_entries: 0
+ pending_child_changes: 0
+---
+
+API 认证指南,涵盖 OAuth 2.0、JWT 令牌和 API 密钥。
```
-子目录的 L0 会被聚合到父目录的 L1 中,形成层级导航。
+初始元数据字段如下:
-## 目录结构
+| 字段 | 含义 |
+| --- | --- |
+| `directory` | sidecar 所描述的目录 URI |
+| `source` | 可选的导入来源;通常只记录在导入根目录 |
+| `generated_by` | 生成组件和粗粒度触发原因 |
+| `freshness` | 直接子项覆盖率和已知的待刷新变化 |
-每个目录都遵循统一的文件结构:
+已知字段会进行 schema 校验。未知顶层字段或已知对象中的未知嵌套字段会被静默丢弃,不会进入 preview、embedding、canonical writeback 或 metadata 写保护比较。没有 frontmatter 的旧 sidecar 继续作为 legacy Markdown 读取;YAML 损坏、缺少必填 `directory` 或已知字段类型错误仍会显式失败。
-```
-viking://resources/docs/auth/
-├── .abstract.md # L0: ~100 tokens
-├── .overview.md # L1: ~1k tokens
-├── .relations.json # 相关资源
-├── oauth.md # L2: 完整内容
-├── jwt.md # L2: 完整内容
-└── api-keys.md # L2: 完整内容
-```
+## 不同读取表面的行为
-## 多模态支持
+同一个 sidecar 在不同表面返回不同视图:
-- **L0/L1**:始终是文本(Markdown)
-- **L2**:可以是任何格式(文本、图片、视频、音频)
+| 访问方式 | 返回内容 |
+| --- | --- |
+| `abstract()` / `overview()` | 仅 Markdown 正文 |
+| `find`、search/rerank preview | 仅 Markdown 正文 |
+| `ls output=agent`、tree agent 输出 | 仅 Markdown 正文 |
+| 直接 `read(".../.abstract.md")` | 原始 frontmatter 和正文 |
+| 普通 `ls` | 默认不列出隐藏 sidecar |
-对于二进制内容,L0/L1 用文本描述:
+语义生成父目录摘要时也只读取子目录 L0 的正文,`source`、`generated_by` 和 `freshness` 不会进入总结 prompt。
-```markdown
-# 图片的 L0
-产品截图,展示带有 OAuth 按钮的登录页面。
+## Embedding 元数据白名单
-# 图片的 L1
-## 图片:登录页面截图
+L0/L1 的 embedding 输入由正文和显式白名单元数据组成。当前白名单只有 `directory`:
-此截图展示应用的登录页面,包含:
-- Google OAuth 按钮(顶部)
-- GitHub OAuth 按钮(中部)
-- 邮箱/密码表单(底部)
+```markdown
+---
+directory: viking://resources/docs/auth/
+---
-尺寸:1920x1080,格式:PNG
+API 认证指南,涵盖 OAuth 2.0、JWT 令牌和 API 密钥。
```
-目录结构
+`source`、`generated_by`、`freshness` 以及未知字段都不会进入 embedding。正常向量化和 admin `vectors_only` reindex 使用相同策略,避免重建索引后改变检索输入。L1 的 rerank scalar 仍然是纯 L1 正文。
-```
-...
-└── 第三章 开发者说明/
- ├── .abstract.md
- ├── .overview.md
- ├── content.md
- └── 视频附件1-开发者说明/ ← 递归扩展附件信息
- ├── .abstract.md
- ├── .overview.md
- ├── 音频和字幕提取.md
- ├── 开发者培训.mp4
- └── 视频分段切片/
- ├── 开发者培训_0s-30s.mp4
- └── 开发者培训_30s-60s.mp4
-```
+## Freshness 与稳定采样
+`freshness` 统计当前目录的**直接子项**,而不是整个递归子树:
+- `total_entries`:参与目录语义的直接文件和直接子目录总数。
+- `sampled_entries`:本轮实际用于总结的直接子项数。
+- `unsampled_entries`:未采样的直接子项数,满足 `sampled + unsampled = total`。
+- `pending_child_changes`:已知发生变化、但尚未反映到当前正文中的直接子项数。
-## 最佳实践
+当直接子项超过 `semantic.sidecar_sample_size`(默认 32)时,系统使用确定性、保序的稳定采样。相同目录树重复刷新会选择相同样本,避免无意义的正文和 Git diff 抖动。
-| 场景 | 推荐层级 |
-|------|----------|
-| 快速相关性检查 | L0 |
-| 理解内容范围 | L1 |
-| 详细信息提取 | L2 |
-| 为 LLM 构建上下文 | L1(通常足够) |
+`pending_child_changes > 0` 表示正文仍然可读,但已知落后于下层变化。父目录刷新成功后,该值会随新的覆盖率元数据重置为 0。
-### Token 预算管理
+当前 resource/skill 语义任务成功后会逐级安排父目录刷新,并在入队前将父目录标记为 pending,直到 namespace 根边界。
-```python
-# 先用 L1 判断,仅在需要时加载 L2
-overview = client.overview(uri)
+> **TODO:基于 freshness 控制冒泡频率**
+>
+> 当前实现会在每次成功的 resource/skill 语义任务后尝试向父级冒泡,即使新生成的子目录摘要与旧值相同。这不是最终期望的调度策略。后续应利用 `freshness` 数据实现合并、阈值或时间窗口节流,例如综合 `pending_child_changes`、采样覆盖率、直接子项变化规模和最近刷新状态,降低热点目录的重复刷新与向上写放大,同时保证最终一致性。
+
+## 写保护
+
+L0/L1 正文可以通过公共 `write` / `batch_write` 更新,但 metadata 受保护:
-if needs_more_detail(overview):
- content = client.read(uri)
+- 目标 sidecar 必须已存在;公共 API 不允许直接创建新的 `.abstract.md` / `.overview.md`。
+- 只提交正文时,系统保留现有 metadata 并重新拼回 canonical OKF。
+- 提交完整 OKF 时,已知 metadata 必须与现有值一致;修改受保护字段会失败。
+- 未知 metadata 字段会静默丢弃。
+- `append` 只追加正文,不会把用户内容追加到 frontmatter。
+- 正文更新只重建该目录实际存在的 L0/L1 向量,不触发语义重新生成,避免刚写入的正文被覆盖。
+
+## 生成机制
+
+SemanticProcessor 自底向上处理目录:
+
+```text
+文件摘要 → 叶子目录 L1 → 叶子目录 L0 → 父目录 → namespace 根边界
```
+子目录 L0 被聚合到父目录 L1。Memory 目录也通过统一的 SemanticProcessor 入口处理,但当前父级冒泡逻辑只用于 resource/skill。多模态文件会先生成文本摘要,再作为普通文件摘要参与其所在目录的 L0/L1;不会为每个图片、音频或视频创建 per-file L0/L1 sidecar。
+
+## 最佳实践
+
+| 场景 | 推荐层级 |
+| --- | --- |
+| 快速相关性检查 | L0 |
+| 理解目录内容范围 | L1 |
+| 详细信息提取 | L2 |
+| 为 LLM 构建初步上下文 | L1,必要时再加载 L2 |
+| 检查 sidecar 来源或 freshness | 直接读取 sidecar 原始内容 |
+
## 相关文档
- [架构概述](./01-architecture.md) - 系统整体架构
- [上下文类型](./02-context-types.md) - 三种上下文类型
- [Viking URI](./04-viking-uri.md) - URI 规范
+- [上下文提取](./06-extraction.md) - L0/L1 生成流程
- [检索机制](./07-retrieval.md) - 检索流程详解
-- [上下文提取](./06-extraction.md) - L0/L1 生成详解
diff --git a/docs/zh/concepts/06-extraction.md b/docs/zh/concepts/06-extraction.md
index 62df4ab5c..bc6e9d280 100644
--- a/docs/zh/concepts/06-extraction.md
+++ b/docs/zh/concepts/06-extraction.md
@@ -118,9 +118,21 @@ SemanticMsg(
2. **收集子目录摘要**:读取已生成的 .abstract.md
3. **生成 .overview.md**:LLM 生成 L1 概览
4. **提取 .abstract.md**:从 overview 提取 L0 摘要
-5. **写入文件**:保存到 AGFS
+5. **写入文件**:以 OKF Markdown 保存正文和受保护元数据
6. **向量化**:创建 Context 并入队 EmbeddingQueue
+L0/L1 是目录级 sidecar,不是 per-file sidecar。生成父目录摘要时只使用子目录 L0 的正文,OKF frontmatter 不进入 prompt。Embedding 使用正文和白名单中的 `directory`;`source`、`generated_by`、`freshness` 不进入向量输入。
+
+### Freshness、采样与父级刷新
+
+每次生成都会记录直接子项的 `total_entries`、`sampled_entries` 和 `unsampled_entries`。直接子项超过 `semantic.sidecar_sample_size`(默认 32)时,系统使用确定性稳定采样。已知子项发生变化但父正文尚未刷新时,`pending_child_changes` 会递增;刷新成功后重置为 0。
+
+当前每个成功的 resource/skill 语义任务都会继续安排父目录刷新,并在入队前将父目录标记为 pending。该行为会一直传播到 namespace 根边界。
+
+> **TODO:使用 freshness 控制冒泡频率**
+>
+> 当前按每次成功任务冒泡会使热点深层目录产生重复刷新和向上写放大。后续应基于 `pending_child_changes`、采样覆盖率、直接子项变化规模和最近刷新状态进行合并、阈值控制或时间窗口节流,同时保持最终一致性。
+
### 处理限制
| 参数 | 默认值 | 说明 |
@@ -128,6 +140,7 @@ SemanticMsg(
| `max_concurrent_llm` | 10 | 并发 LLM 调用数 |
| `max_images_per_call` | 10 | 单次 VLM 最大图片数 |
| `max_sections_per_call` | 20 | 单次 VLM 最大章节数 |
+| `sidecar_sample_size` | 32 | 单个目录摘要使用的直接子项样本上限 |
## 代码骨架提取
diff --git a/docs/zh/getting-started/01-introduction.md b/docs/zh/getting-started/01-introduction.md
index 50d3a1ee7..e764a91e1 100644
--- a/docs/zh/getting-started/01-introduction.md
+++ b/docs/zh/getting-started/01-introduction.md
@@ -62,23 +62,25 @@ client.overview("viking://...") # 获取 L1 概览
将海量上下文一次性塞入提示词,不仅成本高昂,更容易超出模型窗口并引入噪声。OpenViking 在上下文写入时便自动将其处理为三个层级:
-| 层级 | 名称 | Token 限制 | 用途 |
-|------|------|-----------|------|
-| **L0** | 摘要 | ~100 tokens | 向量搜索、快速过滤 |
-| **L1** | 概览 | ~2k tokens | Rerank 精排、内容导航 |
-| **L2** | 详情 | 无限制 | 完整内容、按需加载 |
+| 层级 | 名称 | 默认正文上限 | 用途 |
+| --- | --- | --- | --- |
+| **L0** | 摘要 | 256 字符 | 向量搜索、快速过滤 |
+| **L1** | 概览 | 4000 字符 | Rerank 精排、内容导航 |
+| **L2** | 详情 | 无统一上限 | 完整内容、按需加载 |
```
viking://resources/my_project/
├── .abstract.md # L0 层:摘要
├── .overview.md # L1 层:概览
├── docs/
-│ ├── .abstract.md # 每个目录都有对应的 L0/L1 层
+│ ├── .abstract.md # 语义处理目录通常包含 L0/L1
│ ├── .overview.md
│ └── api.md # L2 层:完整内容
└── src/
```
+L0/L1 是目录级 sidecar,不是 per-file sidecar;两者也不保证始终同时存在。详见[上下文层级](../concepts/03-context-layers.md)。
+
### 3. 目录递归检索
单一的向量检索难以应对复杂的查询意图。OpenViking 设计了一套创新的**目录递归检索策略**:
diff --git a/openviking/retrieve/context_assembler/tiers.py b/openviking/retrieve/context_assembler/tiers.py
index 4194af19a..e17591937 100644
--- a/openviking/retrieve/context_assembler/tiers.py
+++ b/openviking/retrieve/context_assembler/tiers.py
@@ -26,6 +26,7 @@ from openviking.retrieve.context_assembler.params import (
Tier,
)
from openviking.session.memory.utils.memory_file_utils import MemoryFileUtils
+from openviking.storage.semantic_sidecar import body_for_preview
OVERVIEW_HEADING_LIMIT = 24
OVERVIEW_PARAGRAPH_CHARS = 400
@@ -109,7 +110,7 @@ def overview_from_content(candidate: Candidate, content: str) -> str:
return ""
if candidate.is_directory:
# The cached content already is the directory's own overview sidecar.
- return content.strip()
+ return body_for_preview(content).strip()
if _is_code_uri(candidate.base_uri):
extraction = extract_skeleton_result(filename_from_uri(candidate.base_uri), content)
if extraction.text:
diff --git a/openviking/retrieve/hierarchical_retriever.py b/openviking/retrieve/hierarchical_retriever.py
index dc9d7e777..c2a7990ce 100644
--- a/openviking/retrieve/hierarchical_retriever.py
+++ b/openviking/retrieve/hierarchical_retriever.py
@@ -15,14 +15,16 @@ import time
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple
+from openviking.core.context import ContextLevel
from openviking.core.retrieval_targets import default_target_directories
from openviking.models.embedder.base import EmbedResult, embed_compat
from openviking.models.rerank import RerankClient
from openviking.retrieve.memory_lifecycle import hotness_score
from openviking.retrieve.retrieval_stats import get_stats_collector
from openviking.server.identity import RequestContext
-from openviking.storage.vikingdb_manager import VikingDBManager, VikingDBManagerProxy
from openviking.storage.expr import FilterExpr
+from openviking.storage.semantic_sidecar import body_for_preview
+from openviking.storage.vikingdb_manager import VikingDBManager, VikingDBManagerProxy
from openviking.telemetry import get_current_telemetry
from openviking.utils.time_utils import parse_iso_datetime
from openviking.utils.token_estimation import (
@@ -149,9 +151,7 @@ class HierarchicalRetriever:
sparse_query_vector = None
if self.embedder:
if image_query and not getattr(self.embedder, "supports_multimodal", False):
- raise InvalidArgumentError(
- "Image search requires a multimodal embedding model."
- )
+ raise InvalidArgumentError("Image search requires a multimodal embedding model.")
with telemetry.measure("search.embed_query"):
embedding_input = getattr(query, "embedding_input", None) or query.query
result: EmbedResult = await embed_compat(
@@ -173,7 +173,9 @@ class HierarchicalRetriever:
context_type = ContextType.RESOURCE.value
if mode == RetrieverMode.QUICK:
- search_limit = max(limit * 5, 50) if image_query else max(limit, self.GLOBAL_SEARCH_TOPK)
+ search_limit = (
+ max(limit * 5, 50) if image_query else max(limit, self.GLOBAL_SEARCH_TOPK)
+ )
with telemetry.measure("search.vector_retrieval"):
quick_results = await vector_proxy.search_in_tenant(
query_vector=query_vector,
@@ -603,6 +605,14 @@ class HierarchicalRetriever:
final_score = 0.0
level = c.get("level", 2)
display_uri = self._append_level_suffix(c.get("uri", ""), level)
+ abstract = c.get("abstract", "")
+ if level in {ContextLevel.ABSTRACT, ContextLevel.OVERVIEW}:
+ # New records persist body-only rerank scalars, but imported or
+ # legacy indexes may still contain the full OKF document. Keep
+ # the public find/search preview contract body-only at its final
+ # conversion boundary. L2 user Markdown is intentionally left
+ # untouched, including ordinary YAML frontmatter.
+ abstract = body_for_preview(abstract)
results.append(
MatchedContext(
@@ -611,7 +621,7 @@ class HierarchicalRetriever:
if c.get("context_type")
else ContextType.RESOURCE,
level=level,
- abstract=c.get("abstract", ""),
+ abstract=abstract,
category=c.get("category", ""),
score=final_score,
relations=relations,
diff --git a/openviking/service/fs_service.py b/openviking/service/fs_service.py
index 52c62e5cd..6c4feb336 100644
--- a/openviking/service/fs_service.py
+++ b/openviking/service/fs_service.py
@@ -9,6 +9,7 @@ Provides file system operations: ls, mkdir, rm, mv, tree, stat, read, abstract,
import asyncio
from typing import TYPE_CHECKING, Any, Dict, List, Optional
+from openviking.core.context import ContextLevel
from openviking.core.namespace import classify_uri, context_type_for_uri
from openviking.core.uri_validation import validate_optional_viking_uri, validate_viking_uri
from openviking.privacy import (
@@ -24,6 +25,10 @@ from openviking.session.memory.utils.content_visibility import visible_content
from openviking.storage.content_write import ContentWriteCoordinator
from openviking.storage.queuefs import SemanticMsg, get_queue_manager
from openviking.storage.queuefs.semantic_msg import build_semantic_coalesce_key
+from openviking.storage.semantic_sidecar import (
+ mark_semantic_sidecars_pending,
+ render_semantic_sidecar,
+)
from openviking.storage.viking_fs import VikingFS
from openviking.telemetry import get_current_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
@@ -195,7 +200,27 @@ class FSService:
return
directory_uri, abstract_uri = self._resolve_directory_uris(uri)
- await viking_fs.write_file(abstract_uri, abstract, ctx=ctx)
+ await viking_fs.write_file(
+ abstract_uri,
+ render_semantic_sidecar(
+ ContextLevel.ABSTRACT,
+ directory_uri,
+ abstract,
+ {
+ "generated_by": {
+ "component": "FSService",
+ "trigger": "mkdir",
+ },
+ "freshness": {
+ "total_entries": 0,
+ "sampled_entries": 0,
+ "unsampled_entries": 0,
+ "pending_child_changes": 0,
+ },
+ },
+ ),
+ ctx=ctx,
+ )
await vectorize_directory_meta(
uri=directory_uri,
abstract=abstract,
@@ -362,6 +387,12 @@ class FSService:
context_type: str,
ctx: RequestContext,
) -> None:
+ await mark_semantic_sidecars_pending(
+ viking_fs=self._viking_fs,
+ dir_uri=root_uri,
+ changed_entries=1,
+ ctx=ctx,
+ )
try:
queue_manager = get_queue_manager()
except RuntimeError as exc:
@@ -387,6 +418,7 @@ class FSService:
peer_id=ctx.user.user_id,
),
changes={"deleted": [deleted_uri]},
+ generation_trigger="content_delete",
)
if telemetry_id:
get_request_wait_tracker().register_semantic_root(telemetry_id, msg.id)
diff --git a/openviking/service/reindex_executor.py b/openviking/service/reindex_executor.py
index e0f389301..566f43db5 100644
--- a/openviking/service/reindex_executor.py
+++ b/openviking/service/reindex_executor.py
@@ -33,6 +33,7 @@ from openviking.storage.expr import And, Eq, Or, PathScope
from openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+from openviking.storage.semantic_sidecar import body_for_preview, embedding_text_for_body
from openviking.storage.viking_fs import get_viking_fs
from openviking.telemetry import get_current_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
@@ -1126,7 +1127,9 @@ class ReindexExecutor:
uri=directory_uri,
parent_uri=VikingURI(directory_uri).parent.uri,
abstract=abstract,
- vector_text=abstract,
+ vector_text=embedding_text_for_body(
+ ContextLevel.ABSTRACT, directory_uri, abstract
+ ),
is_leaf=False,
context_type=context_type_for_uri(directory_uri),
level=ContextLevel.ABSTRACT,
@@ -1144,7 +1147,9 @@ class ReindexExecutor:
parent_uri=VikingURI(directory_uri).parent.uri,
# L1 abstract scalar carries the overview for Rerank.
abstract=_truncate_abstract_bytes(overview),
- vector_text=overview,
+ vector_text=embedding_text_for_body(
+ ContextLevel.OVERVIEW, directory_uri, overview
+ ),
is_leaf=False,
context_type=context_type_for_uri(directory_uri),
level=ContextLevel.OVERVIEW,
@@ -1480,7 +1485,7 @@ class ReindexExecutor:
uri=uri,
parent_uri=parent_uri,
abstract=abstract,
- vector_text=abstract,
+ vector_text=embedding_text_for_body(ContextLevel.ABSTRACT, uri, abstract),
is_leaf=False,
context_type=ContextType.SKILL.value,
level=ContextLevel.ABSTRACT,
@@ -1499,7 +1504,7 @@ class ReindexExecutor:
parent_uri=parent_uri,
# L1 abstract scalar carries the overview for Rerank.
abstract=_truncate_abstract_bytes(overview),
- vector_text=overview,
+ vector_text=embedding_text_for_body(ContextLevel.OVERVIEW, uri, overview),
is_leaf=False,
context_type=ContextType.SKILL.value,
level=ContextLevel.OVERVIEW,
@@ -1673,7 +1678,9 @@ class ReindexExecutor:
uri=directory_uri,
parent_uri=parent_uri,
abstract=abstract,
- vector_text=abstract,
+ vector_text=embedding_text_for_body(
+ ContextLevel.ABSTRACT, directory_uri, abstract
+ ),
is_leaf=False,
context_type=ContextType.MEMORY.value,
level=ContextLevel.ABSTRACT,
@@ -1691,7 +1698,9 @@ class ReindexExecutor:
parent_uri=parent_uri,
# L1 abstract scalar carries the overview for Rerank.
abstract=_truncate_abstract_bytes(overview),
- vector_text=overview,
+ vector_text=embedding_text_for_body(
+ ContextLevel.OVERVIEW, directory_uri, overview
+ ),
is_leaf=False,
context_type=ContextType.MEMORY.value,
level=ContextLevel.OVERVIEW,
@@ -1755,7 +1764,7 @@ class ReindexExecutor:
file_name = uri.rsplit("/", 1)[-1]
overviews = await self._safe_read_text(f"{parent_uri}/.overview.md", ctx=ctx)
if overviews:
- parsed = self._parse_overview_md(overviews)
+ parsed = self._parse_overview_md(body_for_preview(overviews))
if file_name in parsed:
return parsed[file_name]
existing = await self._fetch_existing_record(
diff --git a/openviking/service/resource_memory_link_service.py b/openviking/service/resource_memory_link_service.py
index 804a08d85..c7007ccf9 100644
--- a/openviking/service/resource_memory_link_service.py
+++ b/openviking/service/resource_memory_link_service.py
@@ -33,6 +33,7 @@ from openviking.session.memory.utils.resource_refs import (
resource_ref_matches,
unlink_resource_references_from_memory,
)
+from openviking.storage.semantic_sidecar import body_for_preview
from openviking.storage.viking_fs import VikingFS, get_viking_fs
from openviking.storage.vikingdb_manager import VikingDBManager
from openviking_cli.exceptions import NotFoundError
@@ -342,9 +343,7 @@ class ResourceMemoryLinkService:
task_id,
account_id=ctx.account_id,
user_id=ctx.user.user_id,
- timeout=(
- _RESOURCE_REASON_COMMIT_TIMEOUT_SECONDS if timeout is None else timeout
- ),
+ timeout=(_RESOURCE_REASON_COMMIT_TIMEOUT_SECONDS if timeout is None else timeout),
poll_interval=0.1,
)
if task.status == TaskStatus.COMPLETED:
@@ -638,7 +637,7 @@ class ResourceMemoryLinkService:
@staticmethod
def _clean_resource_abstract(abstract: Any) -> str:
- text = " ".join(str(abstract or "").split())
+ text = " ".join(body_for_preview(str(abstract or "")).split())
if not text:
return ""
if any(text == marker or text.endswith(marker) for marker in _ABSTRACT_NOT_READY_MARKERS):
diff --git a/openviking/session/memory/memory_updater.py b/openviking/session/memory/memory_updater.py
index 24b369a13..8ca6b391f 100644
--- a/openviking/session/memory/memory_updater.py
+++ b/openviking/session/memory/memory_updater.py
@@ -16,6 +16,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
if TYPE_CHECKING:
from openviking.session.memory.memory_isolation_handler import MemoryIsolationHandler
+from openviking.core.context import ContextLevel
from openviking.message import Message
from openviking.message.part import TextPart
from openviking.server.identity import RequestContext
@@ -39,6 +40,7 @@ from openviking.session.memory.utils.resource_refs import (
)
from openviking.session.memory.utils.template_utils import TemplateUtils
from openviking.session.memory.utils.uri import render_template
+from openviking.storage.semantic_sidecar import freshness_metadata, render_semantic_sidecar
from openviking.storage.viking_fs import get_viking_fs
from openviking.telemetry import tracer
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
@@ -963,9 +965,7 @@ class MemoryUpdater:
uri_memory_type_map[uri] = op.memory_type
# Merge caller-supplied transient tags with per-operation search_tags
# (e.g. event-memory custom scalars) so both reach vectorization.
- effective_search_tags_by_uri = _collect_search_tags_by_uri(
- operations, search_tags_by_uri
- )
+ effective_search_tags_by_uri = _collect_search_tags_by_uri(operations, search_tags_by_uri)
await self._vectorize_memories(
result,
ctx,
@@ -1618,7 +1618,18 @@ class MemoryUpdater:
try:
await viking_fs.write_file(
overview_path,
- rendered,
+ render_semantic_sidecar(
+ ContextLevel.OVERVIEW,
+ directory,
+ rendered,
+ {
+ "generated_by": {
+ "component": "MemoryUpdater",
+ "trigger": "memory_update",
+ },
+ "freshness": freshness_metadata(len(md_files), len(items)),
+ },
+ ),
ctx=ctx,
lease_ref=lease_ref,
)
diff --git a/openviking/session/session.py b/openviking/session/session.py
index 6622eba1c..f395da841 100644
--- a/openviking/session/session.py
+++ b/openviking/session/session.py
@@ -14,6 +14,7 @@ from datetime import datetime, timezone
from typing import TYPE_CHECKING, Any, Awaitable, Callable, Dict, List, Literal, Optional
from uuid import uuid4
+from openviking.core.context import ContextLevel
from openviking.core.namespace import canonical_session_uri
from openviking.core.peer_id import normalize_peer_id, safe_peer_id
from openviking.message import Message, Part
@@ -43,6 +44,7 @@ from openviking.session.tool_result_synopsis import (
ToolResultSynopsis,
generate_tool_result_synopsis,
)
+from openviking.storage.semantic_sidecar import body_for_preview, render_semantic_sidecar
from openviking.telemetry import get_current_telemetry, tracer
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking.utils.model_retry import is_retryable_api_error, retry_async
@@ -794,9 +796,7 @@ class Session:
update_auto_commit_policy: bool = False,
) -> None:
"""Update mutable session config without overwriting concurrent meta changes."""
- update_auto_commit_policy = (
- update_auto_commit_policy or auto_commit_policy is not None
- )
+ update_auto_commit_policy = update_auto_commit_policy or auto_commit_policy is not None
session_path = self._viking_fs._uri_to_path(self._session_uri, ctx=self.ctx)
lease = await self._viking_fs._async_agfs.pathlock_acquire_tree(
session_path, timeout_secs=_SESSION_PHASE1_LOCK_TIMEOUT_SECONDS
@@ -2458,12 +2458,32 @@ class Session:
abstract = self._extract_abstract_from_summary(summary)
await self._viking_fs.write_file(
uri=f"{archive_uri}/.abstract.md",
- content=abstract,
+ content=render_semantic_sidecar(
+ ContextLevel.ABSTRACT,
+ archive_uri,
+ abstract,
+ {
+ "generated_by": {
+ "component": "Session",
+ "trigger": "archive_summary",
+ }
+ },
+ ),
ctx=self.ctx,
)
await self._viking_fs.write_file(
uri=f"{archive_uri}/.overview.md",
- content=summary,
+ content=render_semantic_sidecar(
+ ContextLevel.OVERVIEW,
+ archive_uri,
+ summary,
+ {
+ "generated_by": {
+ "component": "Session",
+ "trigger": "archive_summary",
+ }
+ },
+ ),
ctx=self.ctx,
)
await self._merge_archive_meta(
@@ -3299,7 +3319,7 @@ class Session:
overview = await self._viking_fs.read_file(f"{archive_uri}/.overview.md", ctx=self.ctx)
except Exception:
return ""
- return overview or ""
+ return body_for_preview(overview or "")
async def _read_archive_abstract(self, archive_uri: str, overview: str = "") -> str:
"""Read archive abstract text, falling back to summary extraction."""
@@ -3309,7 +3329,7 @@ class Session:
abstract = ""
if abstract:
- return abstract
+ return body_for_preview(abstract)
if not overview:
overview = await self._read_archive_overview(archive_uri)
@@ -3843,9 +3863,7 @@ class Session:
if archive_index <= 1 or not self._viking_fs:
return True
- predecessor_uri = (
- f"{self._session_uri}/history/archive_{archive_index - 1:03d}"
- )
+ predecessor_uri = f"{self._session_uri}/history/archive_{archive_index - 1:03d}"
if not await self._viking_fs.exists(predecessor_uri, ctx=self.ctx):
return True
if await self._archive_terminal_state(predecessor_uri) != "pending":
@@ -5193,13 +5211,33 @@ class Session:
)
await viking_fs.write_file(
uri=f"{self._session_uri}/.abstract.md",
- content=abstract,
+ content=render_semantic_sidecar(
+ ContextLevel.ABSTRACT,
+ self._session_uri,
+ abstract,
+ {
+ "generated_by": {
+ "component": "Session",
+ "trigger": "session_update",
+ }
+ },
+ ),
ctx=self.ctx,
lease_ref=lease_ref,
)
await viking_fs.write_file(
uri=f"{self._session_uri}/.overview.md",
- content=overview,
+ content=render_semantic_sidecar(
+ ContextLevel.OVERVIEW,
+ self._session_uri,
+ overview,
+ {
+ "generated_by": {
+ "component": "Session",
+ "trigger": "session_update",
+ }
+ },
+ ),
ctx=self.ctx,
lease_ref=lease_ref,
)
diff --git a/openviking/session/skill/skill_operation_updater.py b/openviking/session/skill/skill_operation_updater.py
index 55eb09774..c84fedab3 100644
--- a/openviking/session/skill/skill_operation_updater.py
+++ b/openviking/session/skill/skill_operation_updater.py
@@ -15,6 +15,7 @@ from openviking.session.memory.dataclass import ResolvedOperation, ResolvedOpera
from openviking.session.memory.memory_type_registry import MemoryTypeRegistry
from openviking.session.memory.merge_op import MergeOpFactory
from openviking.storage.content_write import ContentWriteCoordinator
+from openviking.storage.semantic_sidecar import body_for_preview
from openviking.storage.viking_fs import VikingFS, get_viking_fs
from openviking.utils.skill_processor import SkillProcessor
from openviking_cli.exceptions import NotFoundError
@@ -213,7 +214,7 @@ class SkillOperationUpdater:
)
except (FileNotFoundError, NotFoundError):
return None
- parsed = yaml.safe_load(abstract)
+ parsed = yaml.safe_load(body_for_preview(abstract))
return parsed if isinstance(parsed, dict) else None
@staticmethod
diff --git a/openviking/storage/content_write.py b/openviking/storage/content_write.py
index 64d188daa..aa3b9eb71 100644
--- a/openviking/storage/content_write.py
+++ b/openviking/storage/content_write.py
@@ -36,11 +36,17 @@ from openviking.session.memory.utils.resource_refs import (
from openviking.storage.errors import LockAcquisitionError, ResourceBusyError
from openviking.storage.queuefs import SemanticMsg, get_queue_manager
from openviking.storage.queuefs.semantic_msg import build_semantic_coalesce_key
+from openviking.storage.semantic_sidecar import (
+ SemanticSidecarFormatError,
+ is_semantic_sidecar_uri,
+ mark_semantic_sidecars_pending,
+ prepare_semantic_sidecar_write,
+)
from openviking.storage.viking_fs import VikingFS
from openviking.telemetry import get_current_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking.telemetry.resource_summary import build_queue_status_payload
-from openviking.utils.embedding_utils import vectorize_file
+from openviking.utils.embedding_utils import vectorize_directory_meta, vectorize_file
from openviking.utils.path_safety import validate_safe_viking_uri_path
from openviking.utils.tags import normalize_search_tags
from openviking_cli.exceptions import (
@@ -57,7 +63,7 @@ from openviking_cli.utils.logger import get_logger
logger = get_logger(__name__)
-_DERIVED_FILENAMES = frozenset({".abstract.md", ".overview.md", ".relations.json"})
+_DERIVED_FILENAMES = frozenset({".relations.json"})
_CREATE_ALLOWED_EXTENSIONS = frozenset(
{
".md",
@@ -127,7 +133,7 @@ class ContentWriteCoordinator:
written_bytes = len(content.encode("utf-8"))
telemetry_id = get_current_telemetry().telemetry_id
- if context_type == "memory":
+ if context_type == "memory" and not is_semantic_sidecar_uri(normalized_uri):
return await self._write_memory_with_refresh(
uri=normalized_uri,
root_uri=root_uri,
@@ -189,6 +195,7 @@ class ContentWriteCoordinator:
updated: list[str] = []
unchanged: list[str] = []
refresh_kinds: dict[str, str] = {}
+ sidecar_directories: set[str] = set()
pending: list[tuple[dict[str, Any], bool]] = []
conflict: ConflictError | None = None
write_error: Exception | None = None
@@ -202,14 +209,28 @@ class ContentWriteCoordinator:
raise InvalidArgumentError(f"batch-write target must be a file: {uri}")
current = await self._viking_fs.read_file_bytes(uri, ctx=ctx) if exists else None
+ if is_semantic_sidecar_uri(uri):
+ if not exists:
+ raise InvalidArgumentError(
+ f"cannot create generated semantic sidecar directly: {uri}"
+ )
+ operation = dict(operation)
+ operation["content"] = self._prepare_semantic_sidecar_content(
+ uri=uri,
+ current_raw=current or b"",
+ requested_raw=operation["content_bytes"],
+ mode="replace",
+ )
+ operation["content_bytes"] = operation["content"].encode("utf-8")
desired_hash = self._content_hash(operation["content_bytes"])
if current is not None and self._content_hash(current) == desired_hash:
unchanged.append(uri)
- refresh_kinds[uri] = (
- "added"
- if operation["precondition"]["kind"] == "create_if_absent"
- else "modified"
- )
+ if not is_semantic_sidecar_uri(uri):
+ refresh_kinds[uri] = (
+ "added"
+ if operation["precondition"]["kind"] == "create_if_absent"
+ else "modified"
+ )
continue
precondition = operation["precondition"]
@@ -253,7 +274,12 @@ class ContentWriteCoordinator:
break
if existed:
updated.append(uri)
- refresh_kinds[uri] = "modified"
+ if is_semantic_sidecar_uri(uri):
+ parent = VikingURI(uri).parent
+ if parent is not None:
+ sidecar_directories.add(parent.uri)
+ else:
+ refresh_kinds[uri] = "modified"
else:
created.append(uri)
refresh_kinds[uri] = "added"
@@ -265,17 +291,29 @@ class ContentWriteCoordinator:
telemetry_id = get_current_telemetry().telemetry_id
request_registered = False
try:
- if refresh_kinds:
+ if refresh_kinds or sidecar_directories:
if wait and telemetry_id:
get_request_wait_tracker().register_request(telemetry_id)
request_registered = True
try:
- queue_status = await self._refresh_batch(
- refresh_kinds=refresh_kinds,
- ctx=ctx,
- wait=wait,
- timeout=timeout,
- telemetry_id=telemetry_id,
+ for directory_uri in sorted(sidecar_directories):
+ await self._vectorize_semantic_directory(
+ directory_uri=directory_uri, ctx=ctx
+ )
+ queue_status = (
+ await self._refresh_batch(
+ refresh_kinds=refresh_kinds,
+ ctx=ctx,
+ wait=wait,
+ timeout=timeout,
+ telemetry_id=telemetry_id,
+ )
+ if refresh_kinds
+ else (
+ await self._wait_for_request(telemetry_id=telemetry_id, timeout=timeout)
+ if wait
+ else None
+ )
)
except Exception as exc:
if conflict is not None or write_error is not None:
@@ -336,7 +374,9 @@ class ContentWriteCoordinator:
classification.content_index is None
or len(parts) <= classification.content_index + 1
):
- raise InvalidArgumentError("batch-write root must be inside a memory type directory")
+ raise InvalidArgumentError(
+ "batch-write root must be inside a memory type directory"
+ )
elif parts == ["resources"] or (
classification.content_index is not None
and len(parts) <= classification.content_index + 1
@@ -345,7 +385,9 @@ class ContentWriteCoordinator:
self._viking_fs._ensure_mutable_access(root_uri, ctx)
stat = await self._safe_stat(root_uri, ctx=ctx)
if not stat.get("isDir"):
- raise InvalidArgumentError(f"batch-write root must be an existing directory: {root_uri}")
+ raise InvalidArgumentError(
+ f"batch-write root must be an existing directory: {root_uri}"
+ )
def _normalize_batch_operations(
self,
@@ -375,7 +417,9 @@ class ContentWriteCoordinator:
if not relative_uri_path(root_uri, uri):
raise InvalidArgumentError(f"batch-write target is outside root_uri: {uri}")
if context_type_for_uri(uri) != context_type:
- raise InvalidArgumentError(f"batch-write target has a different context type: {uri}")
+ raise InvalidArgumentError(
+ f"batch-write target has a different context type: {uri}"
+ )
self._validate_target_uri(uri)
self._viking_fs._ensure_mutable_access(uri, ctx)
@@ -475,9 +519,7 @@ class ContentWriteCoordinator:
parent = VikingURI(uri).parent
memory_groups[parent.uri if parent is not None else uri].append(uri)
continue
- refresh_root = await self._resolve_root_uri(
- uri, ctx=ctx, anchor_to_parent=True
- )
+ refresh_root = await self._resolve_root_uri(uri, ctx=ctx, anchor_to_parent=True)
resource_groups[(refresh_root, context_type)][change_type].append(uri)
for (refresh_root, context_type), changes in sorted(resource_groups.items()):
@@ -526,6 +568,13 @@ class ContentWriteCoordinator:
target_uri: str = "",
recursive: bool = False,
) -> None:
+ changed_entries = len({uri for values in changes.values() for uri in values})
+ await mark_semantic_sidecars_pending(
+ viking_fs=self._viking_fs,
+ dir_uri=root_uri,
+ changed_entries=changed_entries,
+ ctx=ctx,
+ )
queue_manager = get_queue_manager()
semantic_queue = queue_manager.get_queue(queue_manager.SEMANTIC, allow_create=True)
telemetry = get_current_telemetry()
@@ -554,6 +603,7 @@ class ContentWriteCoordinator:
for change_type in ("added", "modified")
if changes.get(change_type)
},
+ generation_trigger="content_write",
)
if msg.telemetry_id:
get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)
@@ -561,9 +611,7 @@ class ContentWriteCoordinator:
await semantic_queue.enqueue(msg)
except Exception as exc:
if msg.telemetry_id:
- get_request_wait_tracker().mark_semantic_failed(
- msg.telemetry_id, msg.id, str(exc)
- )
+ get_request_wait_tracker().mark_semantic_failed(msg.telemetry_id, msg.id, str(exc))
raise
@staticmethod
@@ -727,11 +775,25 @@ class ContentWriteCoordinator:
try:
if mode != "create":
previous_content = await self._viking_fs.read_file(uri, ctx=ctx)
+ elif is_semantic_sidecar_uri(uri):
+ raise InvalidArgumentError(
+ f"cannot create generated semantic sidecar directly: {uri}"
+ )
if wait and telemetry_id:
get_request_wait_tracker().register_request(telemetry_id)
- await self._write_in_place(uri, content, mode=mode, ctx=ctx, lease_ref=lease)
+ await self._write_in_place(
+ uri,
+ content,
+ mode=mode,
+ ctx=ctx,
+ lease_ref=lease,
+ existing_raw=previous_content,
+ )
content_written = True
- if processing_mode == VECTORS_ONLY:
+ if is_semantic_sidecar_uri(uri):
+ vector_enqueued = await self._vectorize_semantic_sidecar(uri=uri, ctx=ctx)
+ post_process_started = True
+ elif processing_mode == VECTORS_ONLY:
vector_enqueued = await self._vectorize_written_file(
uri=uri,
context_type=context_type,
@@ -755,7 +817,17 @@ class ContentWriteCoordinator:
else None
)
result_kwargs = {}
- if processing_mode == VECTORS_ONLY:
+ if is_semantic_sidecar_uri(uri):
+ _, vector_status = (
+ self._refresh_statuses(wait=wait, queue_status=queue_status)
+ if vector_enqueued
+ else ("skipped", "skipped")
+ )
+ result_kwargs = {
+ "semantic_status": "skipped",
+ "vector_status": vector_status,
+ }
+ elif processing_mode == VECTORS_ONLY:
if vector_enqueued:
_, vector_status = self._refresh_statuses(
wait=wait,
@@ -835,6 +907,42 @@ class ContentWriteCoordinator:
ctx=ctx,
)
+ async def _vectorize_semantic_sidecar(self, *, uri: str, ctx: RequestContext) -> bool:
+ """Re-index a manually edited L0/L1 body without regenerating it."""
+
+ parent = VikingURI(uri).parent
+ if parent is None:
+ return False
+ await self._vectorize_semantic_directory(directory_uri=parent.uri, ctx=ctx)
+ return True
+
+ async def _vectorize_semantic_directory(
+ self, *, directory_uri: str, ctx: RequestContext
+ ) -> None:
+ """Re-index the semantic levels that exist for one directory."""
+
+ async def read_if_exists(uri: str) -> Optional[str]:
+ try:
+ return await self._viking_fs.read_file(uri, ctx=ctx)
+ except Exception as exc:
+ if self._is_not_found(exc):
+ return None
+ raise
+
+ abstract = await read_if_exists(f"{directory_uri}/.abstract.md")
+ overview = await read_if_exists(f"{directory_uri}/.overview.md")
+ if abstract is None and overview is None:
+ return
+ await vectorize_directory_meta(
+ uri=directory_uri,
+ abstract=abstract or "",
+ overview=overview or "",
+ context_type=context_type_for_uri(directory_uri),
+ ctx=ctx,
+ include_abstract=abstract is not None,
+ include_overview=overview is not None,
+ )
+
def _validate_mode(self, mode: str) -> None:
if mode not in {"replace", "append", "create"}:
raise InvalidArgumentError(f"unsupported write mode: {mode}")
@@ -895,6 +1003,8 @@ class ContentWriteCoordinator:
timeout: Optional[float],
processing_mode: ProcessingMode,
) -> Dict[str, Any]:
+ if is_semantic_sidecar_uri(uri):
+ raise InvalidArgumentError(f"cannot create generated semantic sidecar directly: {uri}")
self._validate_create_extension(uri)
stat = await self._safe_stat(uri, ctx=ctx, allow_not_found=True)
@@ -944,7 +1054,23 @@ class ContentWriteCoordinator:
mode: str,
ctx: RequestContext,
lease_ref: Optional[Dict[str, Any]] = None,
+ existing_raw: Optional[str] = None,
) -> None:
+ if is_semantic_sidecar_uri(uri):
+ current_raw = (
+ existing_raw
+ if existing_raw is not None
+ else await self._viking_fs.read_file(uri, ctx=ctx)
+ )
+ rendered = self._prepare_semantic_sidecar_content(
+ uri=uri,
+ current_raw=current_raw,
+ requested_raw=content,
+ mode=mode,
+ )
+ await self._viking_fs.write_file(uri, rendered, ctx=ctx, lease_ref=lease_ref)
+ return
+
if context_type_for_uri(uri) == "memory":
if mode == "replace":
existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)
@@ -977,6 +1103,15 @@ class ContentWriteCoordinator:
return
await self._viking_fs.write_file(uri, content, ctx=ctx, lease_ref=lease_ref)
+ @staticmethod
+ def _prepare_semantic_sidecar_content(
+ *, uri: str, current_raw: str | bytes, requested_raw: str | bytes, mode: str
+ ) -> str:
+ try:
+ return prepare_semantic_sidecar_write(uri, current_raw, requested_raw, mode=mode)
+ except SemanticSidecarFormatError as exc:
+ raise InvalidArgumentError(str(exc)) from exc
+
async def _enqueue_semantic_refresh(
self,
*,
diff --git a/openviking/storage/queuefs/semantic_dag.py b/openviking/storage/queuefs/semantic_dag.py
index f8c24c4c9..2eeae9534 100644
--- a/openviking/storage/queuefs/semantic_dag.py
+++ b/openviking/storage/queuefs/semantic_dag.py
@@ -13,11 +13,18 @@ from weakref import WeakKeyDictionary
from openviking.parse.parsers.media import get_media_type
from openviking.server.identity import RequestContext
from openviking.service.task_work_index import bind_task_context, get_task_context
-from openviking.storage.queuefs.semantic_sidecar import write_semantic_sidecars
+from openviking.storage.semantic_sidecar import (
+ SemanticSidecarFormatError,
+ body_for_preview,
+ deterministic_sample,
+ freshness_metadata,
+ write_semantic_sidecars,
+)
from openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fs
from openviking.telemetry import bind_telemetry, get_current_telemetry
from openviking.utils.ingest_options import IngestOptions
from openviking_cli.utils import VikingURI
+from openviking_cli.utils.config import get_openviking_config
from openviking_cli.utils.logger import get_logger
logger = get_logger(__name__)
@@ -158,6 +165,8 @@ class SemanticDagExecutor:
ingest_options: IngestOptions | None = None,
coalesce_key: str = "",
coalesce_version: int = 0,
+ source: Optional[Dict[str, str]] = None,
+ generation_trigger: str = "semantic_refresh",
):
self._processor = processor
self._context_type = context_type
@@ -172,6 +181,8 @@ class SemanticDagExecutor:
self._ingest_options = IngestOptions.from_value(ingest_options)
self._coalesce_key = coalesce_key
self._coalesce_version = coalesce_version
+ self._source = dict(source) if source else None
+ self._generation_trigger = generation_trigger
self._task_context = get_task_context()
self._telemetry = get_current_telemetry()
self._stale = False
@@ -410,7 +421,7 @@ class SemanticDagExecutor:
else:
file_paths.append(item_uri)
- return children_dirs, file_paths
+ return sorted(children_dirs), sorted(file_paths)
def _get_target_file_path(self, current_uri: str) -> Optional[str]:
if not self._incremental_update or not self._target_uri or not self._root_uri:
@@ -492,7 +503,7 @@ class SemanticDagExecutor:
)
if overview_content:
self._overview_cache[parent_uri] = self._processor._parse_overview_md(
- overview_content
+ body_for_preview(overview_content)
)
else:
self._overview_cache[parent_uri] = {}
@@ -510,6 +521,8 @@ class SemanticDagExecutor:
if file_name in existing_summaries:
return {"name": file_name, "summary": existing_summaries[file_name]}
+ except SemanticSidecarFormatError:
+ raise
except Exception as e:
logger.debug(f"Failed to read existing summary from overview.md for {file_path}: {e}")
@@ -563,7 +576,9 @@ class SemanticDagExecutor:
try:
overview = await self._viking_fs.read_file(f"{target_path}/.overview.md", ctx=self._ctx)
abstract = await self._viking_fs.read_file(f"{target_path}/.abstract.md", ctx=self._ctx)
- return overview, abstract
+ return body_for_preview(overview), body_for_preview(abstract)
+ except SemanticSidecarFormatError:
+ raise
except Exception:
return None, None
@@ -590,6 +605,11 @@ class SemanticDagExecutor:
summary_dict = await self._processor._generate_single_file_summary(
file_path, llm_sem=self._llm_sem, ctx=self._ctx
)
+ except SemanticSidecarFormatError:
+ # A generated sidecar that opted into OKF must never be treated as
+ # an empty file summary; doing so would silently feed metadata or
+ # corrupted YAML into a later regeneration.
+ raise
except Exception as e:
logger.warning(f"Failed to generate summary for {file_path}: {e}")
summary_dict = {"name": file_name, "summary": ""}
@@ -735,7 +755,19 @@ class SemanticDagExecutor:
dir_uri: str,
overview: str,
abstract: str,
+ *,
+ total_entries: int,
+ sampled_entries: int,
) -> bool:
+ metadata: Dict[str, Any] = {
+ "generated_by": {
+ "component": "SemanticProcessor",
+ "trigger": self._generation_trigger,
+ },
+ "freshness": freshness_metadata(total_entries, sampled_entries),
+ }
+ if dir_uri == self._root_uri and self._source:
+ metadata["source"] = self._source
wrote = await write_semantic_sidecars(
viking_fs=self._viking_fs,
dir_uri=dir_uri,
@@ -743,6 +775,7 @@ class SemanticDagExecutor:
abstract=abstract,
ctx=self._ctx,
is_stale=self._is_stale,
+ metadata=metadata,
lock=self._lock,
log_prefix="[SemanticDag]",
)
@@ -756,6 +789,7 @@ class SemanticDagExecutor:
return
need_vectorize = True
children_changed = True
+ should_write = True
abstract = ""
try:
overview = None
@@ -768,10 +802,28 @@ class SemanticDagExecutor:
if not children_changed:
need_vectorize = False
overview, abstract = await self._read_existing_overview_abstract(dir_uri)
+ should_write = overview is None or abstract is None
if overview is None or abstract is None:
async with node.lock:
file_summaries = self._finalize_file_summaries(node)
children_abstracts = await self._finalize_children_abstracts(node)
+ # Freshness describes the directory itself, including direct
+ # entries whose summaries failed. Those entries remain visible
+ # as unsampled coverage instead of disappearing from the count.
+ total_entries = len(node.file_paths) + len(node.children_dirs)
+ sample_limit = getattr(
+ get_openviking_config().semantic,
+ "sidecar_sample_size",
+ 32,
+ )
+ tagged_inputs = sorted(
+ [("file", item) for item in file_summaries]
+ + [("directory", item) for item in children_abstracts],
+ key=lambda tagged: str(tagged[1].get("name") or ""),
+ )
+ sampled_inputs = deterministic_sample(tagged_inputs, sample_limit)
+ file_summaries = [item for kind, item in sampled_inputs if kind == "file"]
+ children_abstracts = [item for kind, item in sampled_inputs if kind == "directory"]
overview = self._select_direct_media_overview(node, file_summaries)
if overview is None:
async with self._llm_sem:
@@ -784,13 +836,24 @@ class SemanticDagExecutor:
return
# Write directly, protected by the outer semantic lock.
- try:
- wrote = await self._write_directory_semantics(dir_uri, overview, abstract)
- if not wrote:
- need_vectorize = False
- except Exception:
- logger.info(f"[SemanticDag] {dir_uri} write failed, skipping")
+ if should_write:
+ try:
+ wrote = await self._write_directory_semantics(
+ dir_uri,
+ overview,
+ abstract,
+ total_entries=total_entries,
+ sampled_entries=len(sampled_inputs),
+ )
+ if not wrote:
+ need_vectorize = False
+ except SemanticSidecarFormatError:
+ raise
+ except Exception:
+ logger.info(f"[SemanticDag] {dir_uri} write failed, skipping")
+ except SemanticSidecarFormatError:
+ raise
except Exception as e:
logger.error(f"Failed to generate overview for {dir_uri}: {e}", exc_info=True)
else:
diff --git a/openviking/storage/queuefs/semantic_msg.py b/openviking/storage/queuefs/semantic_msg.py
index 331288a31..8c87f80a0 100644
--- a/openviking/storage/queuefs/semantic_msg.py
+++ b/openviking/storage/queuefs/semantic_msg.py
@@ -10,6 +10,7 @@ from uuid import uuid4
from openviking.utils.ingest_options import IngestOptions
+
def build_semantic_coalesce_key(
*,
context_type: str,
@@ -60,6 +61,8 @@ class SemanticMsg:
changes: Optional[Dict[str, List[str]]] = (
None # {"added": [...], "modified": [...], "deleted": [...]}
)
+ source: Optional[Dict[str, str]] = None
+ generation_trigger: str = "semantic_refresh"
def __init__(
self,
@@ -80,6 +83,8 @@ class SemanticMsg:
coalesce_key: str = "",
coalesce_version: int = 0,
changes: Optional[Dict[str, List[str]]] = None,
+ source: Optional[Dict[str, str]] = None,
+ generation_trigger: str = "semantic_refresh",
):
self.id = str(uuid4())
self.uri = uri
@@ -99,6 +104,8 @@ class SemanticMsg:
self.coalesce_key = coalesce_key
self.coalesce_version = coalesce_version
self.changes = changes
+ self.source = dict(source) if source else None
+ self.generation_trigger = generation_trigger
def to_dict(self) -> Dict[str, Any]:
"""Convert object to dictionary."""
@@ -151,6 +158,8 @@ class SemanticMsg:
coalesce_key=data.get("coalesce_key", ""),
coalesce_version=data.get("coalesce_version", 0),
changes=data.get("changes"),
+ source=data.get("source"),
+ generation_trigger=data.get("generation_trigger", "semantic_refresh"),
)
if "id" in data and data["id"]:
obj.id = data["id"]
diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.py
index ce00ddc6c..6f98ea3ec 100644
--- a/openviking/storage/queuefs/semantic_processor.py
+++ b/openviking/storage/queuefs/semantic_processor.py
@@ -39,7 +39,14 @@ from openviking.storage.queuefs.semantic_dag import DagStats, SemanticDagExecuto
from openviking.storage.queuefs.semantic_lock import SemanticLockScope
from openviking.storage.queuefs.semantic_msg import SemanticMsg, build_semantic_coalesce_key
from openviking.storage.queuefs.semantic_queue import is_semantic_msg_stale
-from openviking.storage.queuefs.semantic_sidecar import write_semantic_sidecars
+from openviking.storage.semantic_sidecar import (
+ SemanticSidecarFormatError,
+ body_for_preview,
+ deterministic_sample,
+ freshness_metadata,
+ mark_semantic_sidecars_pending,
+ write_semantic_sidecars,
+)
from openviking.storage.viking_fs import LS_ALL_NODES, SyncDiff, get_viking_fs
from openviking.telemetry import bind_telemetry, bind_telemetry_stage, resolve_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
@@ -246,6 +253,12 @@ class SemanticProcessor(DequeueHandlerBase):
or parent_uri == uri.rstrip("/")
):
return
+ await mark_semantic_sidecars_pending(
+ viking_fs=get_viking_fs(),
+ dir_uri=parent_uri,
+ changed_entries=1,
+ ctx=self._ctx_from_semantic_msg(msg),
+ )
from openviking.storage.queuefs import get_queue_manager
@@ -263,6 +276,7 @@ class SemanticProcessor(DequeueHandlerBase):
role=msg.role,
skip_vectorization=msg.skip_vectorization,
changes={"modified": [uri]},
+ generation_trigger="parent_refresh",
coalesce_key=build_semantic_coalesce_key(
context_type=msg.context_type,
uri=parent_uri,
@@ -417,6 +431,8 @@ class SemanticProcessor(DequeueHandlerBase):
ingest_options=msg.ingest_options,
coalesce_key=msg.coalesce_key,
coalesce_version=msg.coalesce_version,
+ source=msg.source,
+ generation_trigger=msg.generation_trigger,
)
await executor.run(run_uri)
self._cache_dag_stats(
@@ -551,6 +567,7 @@ class SemanticProcessor(DequeueHandlerBase):
if not entry.get("isDir", False):
item_uri = VikingURI(dir_uri).join(name).uri
file_paths.append(item_uri)
+ file_paths.sort()
if not file_paths:
logger.info(f"No memory files found in {dir_uri}")
@@ -561,10 +578,12 @@ class SemanticProcessor(DequeueHandlerBase):
try:
old_overview = await viking_fs.read_file(f"{dir_uri}/.overview.md", ctx=ctx)
if old_overview:
- existing_summaries = self._parse_overview_md(old_overview)
+ existing_summaries = self._parse_overview_md(body_for_preview(old_overview))
logger.info(
f"Parsed {len(existing_summaries)} existing summaries from overview.md"
)
+ except SemanticSidecarFormatError:
+ raise
except Exception as e:
logger.debug(f"No existing overview.md found for {dir_uri}: {e}")
@@ -653,8 +672,14 @@ class SemanticProcessor(DequeueHandlerBase):
await asyncio.gather(*[_gen(i, fp) for i, fp in batch])
completed_summaries = [s for s in file_summaries if s is not None]
+ sample_limit = getattr(
+ get_openviking_config().semantic,
+ "sidecar_sample_size",
+ 32,
+ )
+ sampled_summaries = deterministic_sample(completed_summaries, sample_limit)
generated_content = await self._generate_overview(
- dir_uri, completed_summaries, [], llm_sem=llm_sem
+ dir_uri, sampled_summaries, [], llm_sem=llm_sem
)
overview, abstract = self._normalize_overview_generation(generated_content)
@@ -667,6 +692,8 @@ class SemanticProcessor(DequeueHandlerBase):
abstract=abstract,
ctx=ctx,
lock=lock,
+ total_entries=len(file_paths),
+ sampled_entries=len(sampled_summaries),
)
except Exception as e:
raise RuntimeError(f"Failed to write abstract/overview for {dir_uri}: {e}") from e
@@ -696,6 +723,8 @@ class SemanticProcessor(DequeueHandlerBase):
abstract: str,
ctx: Optional[RequestContext],
lock: Optional[Dict[str, Any]] = None,
+ total_entries: int = 0,
+ sampled_entries: int = 0,
) -> bool:
return await write_semantic_sidecars(
viking_fs=viking_fs,
@@ -704,6 +733,14 @@ class SemanticProcessor(DequeueHandlerBase):
abstract=abstract,
ctx=ctx,
is_stale=lambda: is_semantic_msg_stale(msg),
+ metadata={
+ **({"source": msg.source} if msg.source else {}),
+ "generated_by": {
+ "component": "SemanticProcessor",
+ "trigger": msg.generation_trigger,
+ },
+ "freshness": freshness_metadata(total_entries, sampled_entries),
+ },
lock=lock,
log_prefix="[MemorySemantic]",
)
@@ -993,6 +1030,7 @@ class SemanticProcessor(DequeueHandlerBase):
def _extract_abstract_from_overview(self, overview_content: str) -> str:
"""Extract an abstract from the Markdown overview brief description."""
+ overview_content = body_for_preview(overview_content)
lines = overview_content.split("\n")
# Skip header lines (starting with #)
@@ -1016,8 +1054,8 @@ class SemanticProcessor(DequeueHandlerBase):
def _normalize_overview_generation(self, generated_content: str) -> Tuple[str, str]:
"""Convert raw Markdown overview output into final L1 overview and L0 abstract."""
- overview = generated_content
- abstract = self._extract_abstract_from_overview(generated_content)
+ overview = body_for_preview(generated_content)
+ abstract = self._extract_abstract_from_overview(overview)
return self._enforce_size_limits(overview, abstract)
def _enforce_size_limits(self, overview: str, abstract: str) -> Tuple[str, str]:
@@ -1042,6 +1080,7 @@ class SemanticProcessor(DequeueHandlerBase):
summaries: Dict[str, str] = {}
+ overview_content = body_for_preview(overview_content)
if not overview_content or not overview_content.strip():
return summaries
diff --git a/openviking/storage/queuefs/semantic_sidecar.py b/openviking/storage/queuefs/semantic_sidecar.py
deleted file mode 100644
index 093001491..000000000
--- a/openviking/storage/queuefs/semantic_sidecar.py
+++ /dev/null
@@ -1,68 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""Shared writeback for semantic sidecar files."""
-
-from typing import Any, Callable, Dict, Optional
-
-from openviking.server.identity import RequestContext
-from openviking_cli.utils.logger import get_logger
-
-logger = get_logger(__name__)
-
-
-async def write_semantic_sidecars(
- *,
- viking_fs: Any,
- dir_uri: str,
- overview: str,
- abstract: str,
- ctx: Optional[RequestContext],
- is_stale: Callable[[], bool],
- lock: Optional[Dict[str, Any]] = None,
- log_prefix: str = "[Semantic]",
-) -> bool:
- """Write .overview.md and .abstract.md sidecar files under dir_uri with exact-path locking."""
- if is_stale():
- logger.info("%s Skipping stale semantic write for %s", log_prefix, dir_uri)
- return False
-
- lock_paths = [
- viking_fs._uri_to_path(f"{dir_uri}/.overview.md", ctx=ctx),
- viking_fs._uri_to_path(f"{dir_uri}/.abstract.md", ctx=ctx),
- ]
- sidecar_lease = lock
- owns_lease = sidecar_lease is None
- if sidecar_lease is None:
- sidecar_lease = await viking_fs._async_agfs.pathlock_acquire_exact_batch(lock_paths)
- try:
- if is_stale():
- logger.info("%s Skipping stale semantic write for %s", log_prefix, dir_uri)
- return False
- await _write_sidecars(viking_fs, dir_uri, overview, abstract, ctx, sidecar_lease)
- return True
- finally:
- if owns_lease:
- await viking_fs._async_agfs.pathlock_release(sidecar_lease)
-
-
-async def _write_sidecars(
- viking_fs: Any,
- dir_uri: str,
- overview: str,
- abstract: str,
- ctx: Optional[RequestContext],
- lease_ref: Optional[Dict[str, Any]] = None,
-) -> None:
- """Write sidecar files to the target directory."""
- await viking_fs.write_file(
- f"{dir_uri}/.overview.md",
- overview,
- ctx=ctx,
- lease_ref=lease_ref,
- )
- await viking_fs.write_file(
- f"{dir_uri}/.abstract.md",
- abstract,
- ctx=ctx,
- lease_ref=lease_ref,
- )
diff --git a/openviking/storage/semantic_sidecar.py b/openviking/storage/semantic_sidecar.py
new file mode 100644
index 000000000..79b89623e
--- /dev/null
+++ b/openviking/storage/semantic_sidecar.py
@@ -0,0 +1,474 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""OKF documents and atomic writeback for generated semantic sidecars.
+
+Only .abstract.md and .overview.md use this module. Ordinary Markdown keeps
+its frontmatter as user content and is never parsed implicitly.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any, Callable, Dict, Mapping, Optional, Sequence, TypeVar
+
+import yaml
+
+from openviking.core.context import ContextLevel
+from openviking.server.error_mapping import is_not_found_error
+from openviking.server.identity import RequestContext
+from openviking_cli.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+SEMANTIC_SIDECAR_FILENAMES = frozenset({".abstract.md", ".overview.md"})
+EMBEDDING_METADATA_FIELDS = ("directory",)
+_METADATA_ORDER = ("directory", "source", "generated_by", "freshness")
+_MAX_SOURCE_URI_CHARS = 4096
+_MAX_LABEL_CHARS = 128
+_T = TypeVar("_T")
+
+
+class SemanticSidecarFormatError(ValueError):
+ """A generated semantic sidecar has invalid OKF frontmatter."""
+
+
+class SemanticSidecarMetadataError(SemanticSidecarFormatError):
+ """A public write attempted to change protected sidecar metadata."""
+
+
+@dataclass(frozen=True)
+class SemanticSidecarDocument:
+ """Parsed machine metadata and visible Markdown body."""
+
+ metadata: Dict[str, Any]
+ body: str
+ legacy: bool = False
+
+
+def is_semantic_sidecar_uri(uri: str) -> bool:
+ """Return whether uri names one of the two generated sidecars."""
+
+ return uri.rstrip("/").rsplit("/", 1)[-1] in SEMANTIC_SIDECAR_FILENAMES
+
+
+def _normalize_directory_uri(dir_uri: str) -> str:
+ value = str(dir_uri or "").strip()
+ if not value:
+ raise SemanticSidecarFormatError("semantic sidecar directory must not be empty")
+ if not value.startswith("viking://"):
+ raise SemanticSidecarFormatError("semantic sidecar directory must be a viking:// URI")
+ return value.rstrip("/") + "/"
+
+
+def _bounded_string(value: Any, *, field: str, limit: int) -> str:
+ if not isinstance(value, str) or not value.strip():
+ raise SemanticSidecarFormatError(f"semantic sidecar {field} must be a non-empty string")
+ normalized = value.strip()
+ if len(normalized) > limit:
+ raise SemanticSidecarFormatError(
+ f"semantic sidecar {field} exceeds the {limit}-character policy limit"
+ )
+ return normalized
+
+
+def _normalize_metadata(metadata: Mapping[str, Any]) -> Dict[str, Any]:
+ """Validate known fields and silently discard fields outside the schema.
+
+ Ignoring unknown fields keeps readers forward-compatible with metadata
+ written by newer OpenViking versions. Known fields remain strict so a
+ malformed value cannot leak into previews, embeddings, or writeback.
+ """
+
+ normalized: Dict[str, Any] = {}
+ if "directory" in metadata:
+ normalized["directory"] = _normalize_directory_uri(str(metadata["directory"]))
+
+ source = metadata.get("source")
+ if source is not None:
+ if not isinstance(source, Mapping) or not {"kind", "uri"}.issubset(source):
+ raise SemanticSidecarFormatError("semantic sidecar source must contain kind and uri")
+ normalized["source"] = {
+ "kind": _bounded_string(source["kind"], field="source.kind", limit=_MAX_LABEL_CHARS),
+ "uri": _bounded_string(source["uri"], field="source.uri", limit=_MAX_SOURCE_URI_CHARS),
+ }
+
+ generated_by = metadata.get("generated_by")
+ if generated_by is not None:
+ if not isinstance(generated_by, Mapping) or not {
+ "component",
+ "trigger",
+ }.issubset(generated_by):
+ raise SemanticSidecarFormatError(
+ "semantic sidecar generated_by must contain component and trigger"
+ )
+ normalized["generated_by"] = {
+ "component": _bounded_string(
+ generated_by["component"],
+ field="generated_by.component",
+ limit=_MAX_LABEL_CHARS,
+ ),
+ "trigger": _bounded_string(
+ generated_by["trigger"],
+ field="generated_by.trigger",
+ limit=_MAX_LABEL_CHARS,
+ ),
+ }
+
+ freshness = metadata.get("freshness")
+ if freshness is not None:
+ required = {
+ "total_entries",
+ "sampled_entries",
+ "unsampled_entries",
+ "pending_child_changes",
+ }
+ if not isinstance(freshness, Mapping) or not required.issubset(freshness):
+ raise SemanticSidecarFormatError("semantic sidecar freshness has an invalid field set")
+ counters: Dict[str, int] = {}
+ for field in (
+ "total_entries",
+ "sampled_entries",
+ "unsampled_entries",
+ "pending_child_changes",
+ ):
+ value = freshness[field]
+ if isinstance(value, bool) or not isinstance(value, int) or value < 0:
+ raise SemanticSidecarFormatError(
+ f"semantic sidecar freshness.{field} must be a non-negative integer"
+ )
+ counters[field] = value
+ if counters["sampled_entries"] + counters["unsampled_entries"] != counters["total_entries"]:
+ raise SemanticSidecarFormatError(
+ "semantic sidecar freshness sampled + unsampled must equal total"
+ )
+ normalized["freshness"] = counters
+
+ return {field: normalized[field] for field in _METADATA_ORDER if field in normalized}
+
+
+def parse_semantic_sidecar(raw: str | bytes) -> SemanticSidecarDocument:
+ """Parse OKF Markdown while accepting frontmatter-free legacy content.
+
+ A leading delimiter opts a generated sidecar into strict OKF parsing.
+ Invalid YAML, non-object YAML, and missing closing delimiters fail loudly.
+ """
+
+ if isinstance(raw, bytes):
+ try:
+ text = raw.decode("utf-8")
+ except UnicodeDecodeError as exc:
+ raise SemanticSidecarFormatError("semantic sidecar is not valid UTF-8") from exc
+ elif isinstance(raw, str):
+ text = raw
+ else:
+ raise TypeError("semantic sidecar content must be str or bytes")
+
+ lines = text.splitlines(keepends=True)
+ if not lines or lines[0].rstrip("\r\n") != "---":
+ return SemanticSidecarDocument(metadata={}, body=text, legacy=True)
+
+ closing_index: Optional[int] = None
+ for index in range(1, len(lines)):
+ if lines[index].rstrip("\r\n") == "---":
+ closing_index = index
+ break
+ if closing_index is None:
+ raise SemanticSidecarFormatError("semantic sidecar frontmatter is not closed")
+
+ try:
+ loaded = yaml.safe_load("".join(lines[1:closing_index]))
+ except yaml.YAMLError as exc:
+ raise SemanticSidecarFormatError(f"invalid semantic sidecar YAML: {exc}") from exc
+ if loaded is None:
+ loaded = {}
+ if not isinstance(loaded, Mapping):
+ raise SemanticSidecarFormatError("semantic sidecar frontmatter must be a YAML object")
+ if "directory" not in loaded:
+ raise SemanticSidecarFormatError("semantic sidecar frontmatter must contain directory")
+
+ body = "".join(lines[closing_index + 1 :])
+ if body.startswith("\r\n"):
+ body = body[2:]
+ elif body.startswith("\n"):
+ body = body[1:]
+ return SemanticSidecarDocument(metadata=_normalize_metadata(loaded), body=body, legacy=False)
+
+
+def render_semantic_sidecar(
+ level: int | ContextLevel,
+ dir_uri: str,
+ body: str,
+ metadata: Optional[Mapping[str, Any]] = None,
+) -> str:
+ """Render deterministic OKF Markdown for an L0 or L1 sidecar."""
+
+ try:
+ resolved_level = ContextLevel(int(level))
+ except (TypeError, ValueError) as exc:
+ raise SemanticSidecarFormatError(f"invalid semantic sidecar level: {level}") from exc
+ if resolved_level not in {ContextLevel.ABSTRACT, ContextLevel.OVERVIEW}:
+ raise SemanticSidecarFormatError("semantic sidecars only support L0 and L1")
+ if not isinstance(body, str):
+ raise TypeError("semantic sidecar body must be a string")
+
+ merged: Dict[str, Any] = dict(metadata or {})
+ merged["directory"] = _normalize_directory_uri(dir_uri)
+ frontmatter = yaml.safe_dump(
+ _normalize_metadata(merged),
+ allow_unicode=True,
+ default_flow_style=False,
+ sort_keys=False,
+ ).rstrip()
+ return f"---\n{frontmatter}\n---\n\n{body.strip()}\n"
+
+
+def prepare_semantic_sidecar_write(
+ uri: str,
+ current_raw: str | bytes,
+ requested_raw: str | bytes,
+ *,
+ mode: str = "replace",
+) -> str:
+ """Protect metadata while applying a public body write.
+
+ Callers may round-trip the full raw document, but any supplied metadata
+ must equal the stored metadata after schema normalization. A body-only
+ request inherits the stored metadata. The result is always rendered in
+ canonical OKF form so metadata cannot be removed by omission.
+
+ This helper assumes the target already exists. Creating a generated
+ sidecar through the public write API is intentionally rejected by the
+ coordinator because there is no trusted metadata baseline to preserve.
+ """
+
+ if mode not in {"replace", "append"}:
+ raise ValueError(f"unsupported semantic sidecar write mode: {mode}")
+ if not is_semantic_sidecar_uri(uri):
+ raise ValueError(f"not a semantic sidecar URI: {uri}")
+
+ current = parse_semantic_sidecar(current_raw)
+ requested = parse_semantic_sidecar(requested_raw)
+ if not requested.legacy and requested.metadata != current.metadata:
+ raise SemanticSidecarMetadataError("cannot modify protected semantic sidecar metadata")
+
+ requested_body = requested.body
+ body = current.body + requested_body if mode == "append" else requested_body
+ filename = uri.rstrip("/").rsplit("/", 1)[-1]
+ level = ContextLevel.ABSTRACT if filename == ".abstract.md" else ContextLevel.OVERVIEW
+ # Preserve even a historically inconsistent stored directory value. A
+ # public body edit must never repair or otherwise mutate protected
+ # metadata as a side effect. Legacy sidecars have no metadata baseline,
+ # so their first body edit naturally migrates them using the target URI.
+ dir_uri = current.metadata.get("directory") or uri.rstrip("/").rsplit("/", 1)[0]
+ return render_semantic_sidecar(level, dir_uri, body, current.metadata)
+
+
+def body_for_preview(raw: str | bytes) -> str:
+ """Return only user-visible Markdown from a semantic sidecar."""
+
+ document = parse_semantic_sidecar(raw)
+ if document.legacy:
+ return document.body
+ # Rendering adds one canonical terminal newline to the stored document.
+ # It is a serialization detail, not part of semantic accessor output.
+ return document.body.rstrip("\r\n")
+
+
+def body_for_embedding(
+ raw: str | bytes,
+ whitelist: Sequence[str] = EMBEDDING_METADATA_FIELDS,
+) -> str:
+ """Return body plus explicitly whitelisted stable metadata."""
+
+ document = parse_semantic_sidecar(raw)
+ selected = {
+ field: document.metadata[field] for field in whitelist if field in document.metadata
+ }
+ if not selected:
+ return document.body
+ frontmatter = yaml.safe_dump(
+ selected,
+ allow_unicode=True,
+ default_flow_style=False,
+ sort_keys=False,
+ ).rstrip()
+ return f"---\n{frontmatter}\n---\n\n{document.body.strip()}"
+
+
+def embedding_text_for_body(level: int | ContextLevel, dir_uri: str, body: str) -> str:
+ """Build embedding text for a freshly generated visible body."""
+
+ return body_for_embedding(render_semantic_sidecar(level, dir_uri, body))
+
+
+def deterministic_sample(items: Sequence[_T], limit: int) -> list[_T]:
+ """Select a stable, order-preserving sample spanning the full sequence."""
+
+ if limit <= 0:
+ return []
+ if len(items) <= limit:
+ return list(items)
+ if limit == 1:
+ return [items[0]]
+ indices = [(index * (len(items) - 1)) // (limit - 1) for index in range(limit)]
+ return [items[index] for index in indices]
+
+
+def freshness_metadata(
+ total_entries: int, sampled_entries: int, pending: int = 0
+) -> Dict[str, int]:
+ """Create validated direct-child freshness counters."""
+
+ return _normalize_metadata(
+ {
+ "freshness": {
+ "total_entries": total_entries,
+ "sampled_entries": sampled_entries,
+ "unsampled_entries": total_entries - sampled_entries,
+ "pending_child_changes": pending,
+ }
+ }
+ )["freshness"]
+
+
+async def _read_existing_document(
+ viking_fs: Any, uri: str, ctx: Optional[RequestContext]
+) -> Optional[SemanticSidecarDocument]:
+ raw = await _raw_if_exists(viking_fs, uri, ctx)
+ return parse_semantic_sidecar(raw) if raw is not None else None
+
+
+async def write_semantic_sidecars(
+ *,
+ viking_fs: Any,
+ dir_uri: str,
+ overview: str,
+ abstract: str,
+ ctx: Optional[RequestContext],
+ is_stale: Callable[[], bool],
+ metadata: Optional[Mapping[str, Any]] = None,
+ lock: Optional[Dict[str, Any]] = None,
+ log_prefix: str = "[Semantic]",
+) -> bool:
+ """Render and atomically write sidecars while preserving source metadata.
+
+ Exact equality is checked under the lock so stable refreshes do not touch
+ modification times or create noisy Git diffs.
+ """
+
+ if is_stale():
+ logger.info("%s Skipping stale semantic write for %s", log_prefix, dir_uri)
+ return False
+
+ overview_uri = f"{dir_uri}/.overview.md"
+ abstract_uri = f"{dir_uri}/.abstract.md"
+ lock_paths = [
+ viking_fs._uri_to_path(overview_uri, ctx=ctx),
+ viking_fs._uri_to_path(abstract_uri, ctx=ctx),
+ ]
+ sidecar_lease = lock
+ owns_lease = sidecar_lease is None
+ if sidecar_lease is None:
+ sidecar_lease = await viking_fs._async_agfs.pathlock_acquire_exact_batch(lock_paths)
+ try:
+ if is_stale():
+ logger.info("%s Skipping stale semantic write for %s", log_prefix, dir_uri)
+ return False
+
+ existing_overview = await _read_existing_document(viking_fs, overview_uri, ctx)
+ existing_abstract = await _read_existing_document(viking_fs, abstract_uri, ctx)
+ merged_metadata = dict(metadata or {})
+ for existing in (existing_overview, existing_abstract):
+ if existing and "source" in existing.metadata and "source" not in merged_metadata:
+ merged_metadata["source"] = existing.metadata["source"]
+ break
+
+ rendered_overview = render_semantic_sidecar(
+ ContextLevel.OVERVIEW, dir_uri, overview, merged_metadata
+ )
+ rendered_abstract = render_semantic_sidecar(
+ ContextLevel.ABSTRACT, dir_uri, abstract, merged_metadata
+ )
+ current_overview = await _raw_if_exists(viking_fs, overview_uri, ctx)
+ current_abstract = await _raw_if_exists(viking_fs, abstract_uri, ctx)
+
+ if current_overview != rendered_overview:
+ await viking_fs.write_file(
+ overview_uri, rendered_overview, ctx=ctx, lease_ref=sidecar_lease
+ )
+ if current_abstract != rendered_abstract:
+ await viking_fs.write_file(
+ abstract_uri, rendered_abstract, ctx=ctx, lease_ref=sidecar_lease
+ )
+ return True
+ finally:
+ if owns_lease:
+ await viking_fs._async_agfs.pathlock_release(sidecar_lease)
+
+
+async def _raw_if_exists(viking_fs: Any, uri: str, ctx: Optional[RequestContext]) -> Optional[str]:
+ read_file = getattr(viking_fs, "read_file", None)
+ if read_file is None:
+ return None
+ try:
+ return await read_file(uri, ctx=ctx)
+ except KeyError:
+ # Lightweight storage doubles commonly expose an in-memory mapping and
+ # use KeyError for a missing optional sidecar. Production backends use
+ # their regular not-found exception, handled below.
+ return None
+ except Exception as exc:
+ if is_not_found_error(exc):
+ return None
+ raise
+
+
+async def mark_semantic_sidecars_pending(
+ *,
+ viking_fs: Any,
+ dir_uri: str,
+ changed_entries: int,
+ ctx: Optional[RequestContext],
+) -> None:
+ """Increment pending changes while leaving visible sidecar bodies untouched.
+
+ Legacy documents have no freshness baseline and are left alone until their
+ next natural refresh migrates them to OKF. Both sidecars are updated under
+ one exact-path lease so their counters remain symmetric.
+ """
+
+ if changed_entries <= 0:
+ return
+ uris = [
+ f"{dir_uri.rstrip('/')}/.overview.md",
+ f"{dir_uri.rstrip('/')}/.abstract.md",
+ ]
+ preliminary = [await _raw_if_exists(viking_fs, uri, ctx) for uri in uris]
+ if not any(
+ raw is not None
+ and not parse_semantic_sidecar(raw).legacy
+ and "freshness" in parse_semantic_sidecar(raw).metadata
+ for raw in preliminary
+ ):
+ return
+
+ lock_paths = [viking_fs._uri_to_path(uri, ctx=ctx) for uri in uris]
+ lease = await viking_fs._async_agfs.pathlock_acquire_exact_batch(lock_paths)
+ try:
+ for level, uri in zip((ContextLevel.OVERVIEW, ContextLevel.ABSTRACT), uris, strict=True):
+ raw = await _raw_if_exists(viking_fs, uri, ctx)
+ if raw is None:
+ continue
+ document = parse_semantic_sidecar(raw)
+ freshness = document.metadata.get("freshness")
+ if document.legacy or not isinstance(freshness, Mapping):
+ continue
+ metadata = dict(document.metadata)
+ updated_freshness = dict(freshness)
+ updated_freshness["pending_child_changes"] += changed_entries
+ metadata["freshness"] = updated_freshness
+ rendered = render_semantic_sidecar(level, dir_uri, document.body, metadata)
+ if rendered != raw:
+ await viking_fs.write_file(uri, rendered, ctx=ctx, lease_ref=lease)
+ finally:
+ await viking_fs._async_agfs.pathlock_release(lease)
diff --git a/openviking/storage/viking_fs/_semantic.py b/openviking/storage/viking_fs/_semantic.py
index 0ef6b7364..6502181c7 100644
--- a/openviking/storage/viking_fs/_semantic.py
+++ b/openviking/storage/viking_fs/_semantic.py
@@ -4,12 +4,13 @@
import asyncio
import json
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, Union
-from openviking.core.namespace import canonicalize_uri
+from openviking.core.context import ContextLevel
from openviking.core.retrieval_targets import resolve_retrieval_targets
from openviking.server.error_mapping import is_not_found_error, map_exception
from openviking.server.identity import RequestContext
+from openviking.storage.semantic_sidecar import body_for_preview, render_semantic_sidecar
from openviking.storage.viking_fs._base import (
RelationEntry,
_ensure_non_empty_search_query,
@@ -19,10 +20,6 @@ from openviking.telemetry import get_current_telemetry
from openviking.utils.image_search import build_multimodal_embedding_input
from openviking_cli.exceptions import NotFoundError
-if TYPE_CHECKING:
- from openviking.storage.viking_vector_index_backend import VikingVectorIndexBackend
- from openviking_cli.utils.config import GrepConfig, RerankConfig, RetrievalConfig
-
class _SemanticMixin:
"""Abstract/overview/relations/find/search/link (semantic retrieval layer)."""
@@ -51,7 +48,7 @@ class _SemanticMixin:
raise
return f"# {uri} [Directory abstract is not ready]"
- return self._decode_bytes(content_bytes)
+ return body_for_preview(self._decode_bytes(content_bytes))
async def _read_abstract_for_known_dir(
self,
@@ -183,7 +180,7 @@ class _SemanticMixin:
# Fallback to default if .overview.md doesn't exist
return f"# {uri}\n\n[Directory overview is not ready]"
- return self._decode_bytes(content_bytes)
+ return body_for_preview(self._decode_bytes(content_bytes))
async def relations(
self,
@@ -357,9 +354,7 @@ class _SemanticMixin:
target_abstract = ""
intent_enabled = (
- bool(self.retrieval_config.enable_intent)
- if self.retrieval_config is not None
- else True
+ bool(self.retrieval_config.enable_intent) if self.retrieval_config is not None else True
)
# With session context: optional intent analysis
@@ -635,11 +630,39 @@ class _SemanticMixin:
if abstract:
abstract_uri = f"{uri}/.abstract.md"
- await self.write_file(abstract_uri, abstract, ctx=ctx)
+ await self.write_file(
+ abstract_uri,
+ render_semantic_sidecar(
+ ContextLevel.ABSTRACT,
+ uri,
+ abstract,
+ {
+ "generated_by": {
+ "component": "VikingFS.write_context",
+ "trigger": "context_write",
+ }
+ },
+ ),
+ ctx=ctx,
+ )
if overview:
overview_uri = f"{uri}/.overview.md"
- await self.write_file(overview_uri, overview, ctx=ctx)
+ await self.write_file(
+ overview_uri,
+ render_semantic_sidecar(
+ ContextLevel.OVERVIEW,
+ uri,
+ overview,
+ {
+ "generated_by": {
+ "component": "VikingFS.write_context",
+ "trigger": "context_write",
+ }
+ },
+ ),
+ ctx=ctx,
+ )
except Exception as e:
logger.error(f"[VikingFS] Failed to write {uri}: {e}")
diff --git a/openviking/utils/embedding_utils.py b/openviking/utils/embedding_utils.py
index 24b193030..e66b3eb0f 100644
--- a/openviking/utils/embedding_utils.py
+++ b/openviking/utils/embedding_utils.py
@@ -28,6 +28,7 @@ from openviking.server.identity import RequestContext
from openviking.service.task_work_index import TaskWorkRejected
from openviking.storage.queuefs import get_queue_manager
from openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter
+from openviking.storage.semantic_sidecar import body_for_preview, embedding_text_for_body
from openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fs
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking.utils.embedding_input import truncate_embedding_input
@@ -361,12 +362,18 @@ async def vectorize_directory_meta(
include_overview: bool = True,
scalar_overrides: Optional[Dict[int, Dict[str, Any]]] = None,
ingest_options: IngestOptions | None = None,
+ include_abstract: bool = True,
) -> None:
"""
Vectorize directory metadata (.abstract.md and .overview.md).
Creates Context objects for abstract and overview and enqueues them.
"""
+ # Callers may provide either freshly generated bodies or raw sidecar bytes
+ # read during reindex/import. Normalize at this shared boundary so protected
+ # operational metadata never leaks into vector text or rerank scalars.
+ abstract = body_for_preview(abstract)
+ overview = body_for_preview(overview)
first_enqueue_error: Optional[Exception] = None
try:
if not ctx:
@@ -386,45 +393,48 @@ async def vectorize_directory_meta(
# .abstract.md / overview > 65535 UTF-8 bytes still fails embedding enqueue.
abstract = _truncate_abstract_bytes(abstract)
- # Vectorize L0: .abstract.md (abstract)
- context_abstract = Context(
- uri=uri,
- parent_uri=parent_uri,
- is_leaf=False,
- abstract=abstract,
- context_type=context_type,
- level=ContextLevel.ABSTRACT,
- created_at=created_at,
- updated_at=updated_at,
- user=ctx.user,
- account_id=ctx.account_id,
- owner_space=owner_space,
- )
- context_abstract.set_vectorize(Vectorize(text=abstract))
- msg_abstract = EmbeddingMsgConverter.from_context(context_abstract)
- _apply_scalar_overrides(
- msg_abstract,
- (scalar_overrides or {}).get(int(ContextLevel.ABSTRACT.value)),
- )
- _apply_ingest_options(msg_abstract, ingest_options)
- if msg_abstract:
- try:
- enqueued = await _enqueue_embedding_message(
- embedding_queue,
- msg_abstract,
- failure_message=f"Failed to enqueue directory L0 vector for {uri}",
- )
- if enqueued:
- logger.debug(f"Enqueued directory L0 (abstract) for vectorization: {uri}")
- except TaskWorkRejected:
- logger.debug("Skipped directory vectorization for cancelling task: %s", uri)
- return
- except Exception as e:
- logger.error(
- f"Failed to enqueue directory L0 (abstract) for vectorization: {uri}: {e}",
- exc_info=True,
- )
- first_enqueue_error = e
+ if include_abstract:
+ # Vectorize L0: .abstract.md (abstract)
+ context_abstract = Context(
+ uri=uri,
+ parent_uri=parent_uri,
+ is_leaf=False,
+ abstract=abstract,
+ context_type=context_type,
+ level=ContextLevel.ABSTRACT,
+ created_at=created_at,
+ updated_at=updated_at,
+ user=ctx.user,
+ account_id=ctx.account_id,
+ owner_space=owner_space,
+ )
+ context_abstract.set_vectorize(
+ Vectorize(text=embedding_text_for_body(ContextLevel.ABSTRACT, uri, abstract))
+ )
+ msg_abstract = EmbeddingMsgConverter.from_context(context_abstract)
+ _apply_scalar_overrides(
+ msg_abstract,
+ (scalar_overrides or {}).get(int(ContextLevel.ABSTRACT.value)),
+ )
+ _apply_ingest_options(msg_abstract, ingest_options)
+ if msg_abstract:
+ try:
+ enqueued = await _enqueue_embedding_message(
+ embedding_queue,
+ msg_abstract,
+ failure_message=f"Failed to enqueue directory L0 vector for {uri}",
+ )
+ if enqueued:
+ logger.debug(f"Enqueued directory L0 (abstract) for vectorization: {uri}")
+ except TaskWorkRejected:
+ logger.debug("Skipped directory vectorization for cancelling task: %s", uri)
+ return
+ except Exception as e:
+ logger.error(
+ f"Failed to enqueue directory L0 (abstract) for vectorization: {uri}: {e}",
+ exc_info=True,
+ )
+ first_enqueue_error = e
if include_overview:
# Vectorize L1: .overview.md (overview)
@@ -443,7 +453,9 @@ async def vectorize_directory_meta(
account_id=ctx.account_id,
owner_space=owner_space,
)
- context_overview.set_vectorize(Vectorize(text=overview))
+ context_overview.set_vectorize(
+ Vectorize(text=embedding_text_for_body(ContextLevel.OVERVIEW, uri, overview))
+ )
msg_overview = EmbeddingMsgConverter.from_context(context_overview)
_apply_scalar_overrides(
msg_overview,
diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.py
index a5fd601c9..bdb9e8198 100644
--- a/openviking/utils/resource_processor.py
+++ b/openviking/utils/resource_processor.py
@@ -293,9 +293,7 @@ class ResourceProcessor:
source_format=parse_result.source_format,
create_parent=kwargs.get("create_parent", False),
flatten_single_file=(
- normalize_parse_mode(
- kwargs.get("parse_mode", ParseMode.DEFAULT)
- )
+ normalize_parse_mode(kwargs.get("parse_mode", ParseMode.DEFAULT))
is ParseMode.NO_SPLIT
and parse_result.source_format not in {"directory", "repository"}
and not to_is_directory
@@ -304,9 +302,7 @@ class ResourceProcessor:
if context_tree and context_tree.root:
result["root_uri"] = context_tree.root.uri
result["temp_uri"] = context_tree.root.temp_uri
- root_is_file = bool(
- getattr(context_tree, "_root_is_file", False)
- )
+ root_is_file = bool(getattr(context_tree, "_root_is_file", False))
telemetry.set(
"resource.finalize.duration_ms",
round((time.perf_counter() - finalize_start) * 1000, 3),
@@ -444,6 +440,11 @@ class ResourceProcessor:
"target_preexisting": target_preexisting,
"is_code_repo": parse_result.source_format == "repository",
"root_is_file": root_is_file,
+ "semantic_source": self._semantic_source_metadata(
+ path=path,
+ prepared_resource=prepared_resource,
+ source_format=parse_result.source_format,
+ ),
}
if defer_post_processing:
result["_post_process"] = prepared
@@ -489,6 +490,7 @@ class ResourceProcessor:
vectors_only = processing_mode == VECTORS_ONLY
root_is_file = bool(prepared.get("root_is_file"))
ingest_options = IngestOptions.from_value(kwargs.pop("ingest_options", None))
+ semantic_source = prepared.get("semantic_source")
should_summarize = not root_is_file and not vectors_only and (summarize or build_index)
should_refresh_file_parent = (
root_is_file and not vectors_only and (summarize or build_index)
@@ -509,6 +511,8 @@ class ResourceProcessor:
is_code_repo=bool(prepared.get("is_code_repo")),
target_preexisting=target_preexisting,
ingest_options=ingest_options,
+ semantic_source=semantic_source,
+ generation_trigger="resource_ingest",
**kwargs,
)
if (
@@ -608,6 +612,30 @@ class ResourceProcessor:
)
return result
+ @staticmethod
+ def _semantic_source_metadata(
+ *,
+ path: str,
+ prepared_resource: Optional["LocalResource"],
+ source_format: Optional[str],
+ ) -> Dict[str, str]:
+ """Return the stable origin metadata carried only by the import root."""
+
+ if prepared_resource is not None:
+ return {
+ "kind": str(prepared_resource.source_type),
+ "uri": str(prepared_resource.original_source),
+ }
+ if source_format == "repository":
+ kind = "git"
+ elif path.startswith(("http://", "https://")):
+ kind = "http"
+ elif path.startswith(("git@", "ssh://", "git://")):
+ kind = "git"
+ else:
+ kind = "local"
+ return {"kind": kind, "uri": str(path)}
+
async def _delete_removed_resource_vectors(
self,
*,
@@ -627,11 +655,13 @@ class ResourceProcessor:
await self.vikingdb.delete(ids, ctx=ctx)
for uri in dict.fromkeys(dirs):
records = await self.vikingdb.filter(
- filter=And([
- PathScope("uri", uri, depth=-1),
- Eq("level", int(ContextLevel.DETAIL)),
- Eq("account_id", ctx.account_id),
- ]),
+ filter=And(
+ [
+ PathScope("uri", uri, depth=-1),
+ Eq("level", int(ContextLevel.DETAIL)),
+ Eq("account_id", ctx.account_id),
+ ]
+ ),
limit=VECTORDB_MAX_QUERY_LIMIT,
output_fields=["id"],
ctx=ctx,
@@ -779,9 +809,7 @@ class ResourceProcessor:
try:
pathlock = get_viking_fs()._async_agfs
acquire = (
- pathlock.pathlock_acquire_exact
- if root_is_file
- else pathlock.pathlock_acquire_tree
+ pathlock.pathlock_acquire_exact if root_is_file else pathlock.pathlock_acquire_tree
)
return await acquire(path, timeout_secs=timeout)
except LockAcquisitionError as exc:
diff --git a/openviking/utils/summarizer.py b/openviking/utils/summarizer.py
index 66e7e7f83..4577dca3c 100644
--- a/openviking/utils/summarizer.py
+++ b/openviking/utils/summarizer.py
@@ -108,6 +108,8 @@ class Summarizer:
temp_uris = kwargs.get("temp_uris", [])
ingest_options = IngestOptions.from_value(kwargs.get("ingest_options"))
+ source = kwargs.get("semantic_source")
+ generation_trigger = str(kwargs.get("generation_trigger") or "manual_refresh")
if not temp_uris:
temp_uris = resource_uris
if len(temp_uris) != len(resource_uris):
@@ -189,6 +191,8 @@ class Summarizer:
is_code_repo=kwargs.get("is_code_repo", False),
target_preexisting=resolve_target_preexisting(idx, target_uri),
ingest_options=ingest_options,
+ source=source,
+ generation_trigger=generation_trigger,
)
if msg.telemetry_id:
get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)
diff --git a/openviking_cli/utils/config/parser_config.py b/openviking_cli/utils/config/parser_config.py
index 1ac59cef6..1eec73c1e 100644
--- a/openviking_cli/utils/config/parser_config.py
+++ b/openviking_cli/utils/config/parser_config.py
@@ -750,6 +750,9 @@ class SemanticConfig:
overview_batch_size: int = 50
"""Maximum number of file summaries per batch when splitting oversized prompts."""
+ sidecar_sample_size: int = 32
+ """Maximum direct-child summaries used in one generated directory sidecar."""
+
abstract_max_chars: int = 256
"""Maximum characters for generated abstracts."""
@@ -764,6 +767,8 @@ class SemanticConfig:
"""Character overlap between adjacent memory chunks for context continuity."""
def __post_init__(self):
+ if self.sidecar_sample_size <= 0:
+ raise ValueError("sidecar_sample_size must be positive")
if self.memory_chunk_chars <= 0:
raise ValueError("memory_chunk_chars must be positive")
if self.memory_chunk_overlap < 0:
diff --git a/tests/misc/test_semantic_config.py b/tests/misc/test_semantic_config.py
index 72a59b9a2..6b47450ac 100644
--- a/tests/misc/test_semantic_config.py
+++ b/tests/misc/test_semantic_config.py
@@ -14,6 +14,7 @@ def test_semantic_config_defaults():
assert config.max_file_content_chars == 30000
assert config.max_overview_prompt_chars == 60000
assert config.overview_batch_size == 50
+ assert config.sidecar_sample_size == 32
assert config.abstract_max_chars == 256
assert config.overview_max_chars == 4000
assert config.memory_chunk_chars == 2000
diff --git a/tests/retrieve/test_hierarchical_retriever_rerank.py b/tests/retrieve/test_hierarchical_retriever_rerank.py
index f030e09dc..288b5e139 100644
--- a/tests/retrieve/test_hierarchical_retriever_rerank.py
+++ b/tests/retrieve/test_hierarchical_retriever_rerank.py
@@ -9,8 +9,10 @@ import time
import pytest
+from openviking.core.context import ContextLevel
from openviking.retrieve.hierarchical_retriever import HierarchicalRetriever, RetrieverMode
from openviking.server.identity import RequestContext, Role
+from openviking.storage.semantic_sidecar import render_semantic_sidecar
from openviking.utils.token_estimation import estimate_text_tokens
from openviking_cli.retrieve.types import ContextType, TypedQuery
from openviking_cli.session.user_id import UserIdentifier
@@ -663,16 +665,45 @@ async def test_convert_to_matched_contexts_propagates_search_tags():
@pytest.mark.asyncio
-async def test_convert_to_matched_contexts_defaults_empty_search_tags():
+async def test_convert_to_matched_contexts_defaults_tags_and_body_previews():
retriever = HierarchicalRetriever(
storage=DummyStorage(),
embedder=None,
rerank_config=None,
)
+ uri = "viking://resources/demo"
+ metadata = {
+ "source": {"kind": "http", "uri": "https://example.com/private.pdf"},
+ "generated_by": {"component": "SemanticProcessor", "trigger": "ingest"},
+ }
+ markdown = "---\ntitle: User document\n---\n\nVisible body."
result = await retriever._convert_to_matched_contexts(
- [_result("viking://resources/file-a", 1.0, abstract="child A")],
+ [
+ _result(
+ uri,
+ 1.0,
+ level=int(ContextLevel.ABSTRACT),
+ abstract=render_semantic_sidecar(
+ ContextLevel.ABSTRACT, uri, "Visible abstract.", metadata
+ ),
+ ),
+ _result(
+ uri,
+ 0.9,
+ level=int(ContextLevel.OVERVIEW),
+ abstract=render_semantic_sidecar(
+ ContextLevel.OVERVIEW, uri, "# Visible overview", metadata
+ ),
+ ),
+ _result("viking://resources/demo.md", 0.8, level=2, abstract=markdown),
+ ],
ctx=_ctx(),
)
- assert result[0].search_tags == []
+ assert [item.search_tags for item in result] == [[], [], []]
+ assert [item.abstract for item in result] == [
+ "Visible abstract.",
+ "# Visible overview",
+ markdown,
+ ]
diff --git a/tests/server/test_admin_rebuild_api.py b/tests/server/test_admin_rebuild_api.py
index 72c09da2a..2ea3048cc 100644
--- a/tests/server/test_admin_rebuild_api.py
+++ b/tests/server/test_admin_rebuild_api.py
@@ -2082,7 +2082,9 @@ async def test_reindex_resource_vector_text_summary_first_skips_content_read(mon
monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)
monkeypatch.setattr(
"openviking.service.reindex_executor.get_openviking_config",
- lambda: type("Config", (), {"embedding": type("Embedding", (), {"text_source": "summary_first"})()})(),
+ lambda: type(
+ "Config", (), {"embedding": type("Embedding", (), {"text_source": "summary_first"})()}
+ )(),
)
service = ReindexExecutor()
@@ -2103,6 +2105,7 @@ async def test_reindex_resource_vector_text_summary_first_skips_content_read(mon
@pytest.mark.asyncio
async def test_reindex_file_summary_reads_existing_record_as_uri_owner(monkeypatch):
from openviking.service.reindex_executor import ReindexExecutor
+ from openviking.storage.semantic_sidecar import render_semantic_sidecar
captured = {}
@@ -2130,6 +2133,36 @@ async def test_reindex_file_summary_reads_existing_record_as_uri_owner(monkeypat
assert summary == "owner summary"
assert captured["ctx"].user.user_id == "bob"
+ raw = render_semantic_sidecar(
+ ContextLevel.OVERVIEW,
+ "viking://resources/demo",
+ "# Demo\n\n## image.png\nVisible file summary.",
+ {
+ "source": {
+ "kind": "http",
+ "uri": "https://example.com/private.pdf",
+ }
+ },
+ )
+
+ async def fake_safe_read_text(self, uri, *, ctx):
+ del self, uri, ctx
+ return raw
+
+ async def fail_if_existing_record_read(self, *, uri, level, ctx):
+ raise AssertionError("body-only overview summary should be used before fallback")
+
+ monkeypatch.setattr(ReindexExecutor, "_safe_read_text", fake_safe_read_text)
+ monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fail_if_existing_record_read)
+
+ summary = await service._best_file_summary(
+ "viking://resources/demo/image.png",
+ ctx=ctx,
+ )
+
+ assert summary == "Visible file summary."
+ assert "source:" not in summary
+
@pytest.mark.asyncio
async def test_reindex_memory_fallback_reads_existing_record_as_uri_owner(monkeypatch):
diff --git a/tests/session/memory/test_memory_updater.py b/tests/session/memory/test_memory_updater.py
index 6b2ed7dbb..6d623011d 100644
--- a/tests/session/memory/test_memory_updater.py
+++ b/tests/session/memory/test_memory_updater.py
@@ -37,6 +37,7 @@ from openviking.session.memory.utils import (
MemoryFileUtils,
parse_memory_file_with_fields,
)
+from openviking.storage.semantic_sidecar import parse_semantic_sidecar
from openviking_cli.exceptions import NotFoundError
from openviking_cli.session.user_id import UserIdentifier
@@ -316,10 +317,21 @@ class TestMemoryUpdater:
await updater.generate_overview("events", directory, ctx, extract_context=None)
- assert "**Date:**" not in viking_fs.store[overview_uri]
- assert "- [kept event](./kept_event.md)" in viking_fs.store[overview_uri]
- assert "- [plain_event.md](./plain_event.md)" in viking_fs.store[overview_uri]
- assert "deleted_event.md" not in viking_fs.store[overview_uri]
+ document = parse_semantic_sidecar(viking_fs.store[overview_uri])
+ assert document.metadata["generated_by"] == {
+ "component": "MemoryUpdater",
+ "trigger": "memory_update",
+ }
+ assert document.metadata["freshness"] == {
+ "total_entries": 2,
+ "sampled_entries": 2,
+ "unsampled_entries": 0,
+ "pending_child_changes": 0,
+ }
+ assert "**Date:**" not in document.body
+ assert "- [kept event](./kept_event.md)" in document.body
+ assert "- [plain_event.md](./plain_event.md)" in document.body
+ assert "deleted_event.md" not in document.body
@pytest.mark.asyncio
async def test_generate_overview_template_fallbacks_for_preferences_and_entities(self):
diff --git a/tests/session/test_session_context.py b/tests/session/test_session_context.py
index 8e45f0452..4aede5653 100644
--- a/tests/session/test_session_context.py
+++ b/tests/session/test_session_context.py
@@ -16,6 +16,7 @@ from openviking.service.core import OpenVikingService
from openviking.service.task_tracker import get_task_tracker
from openviking.session import Session
from openviking.storage.queuefs import QueueManager, SessionCommitMsg, get_queue_manager
+from openviking.storage.semantic_sidecar import body_for_preview
from openviking_cli.utils.config import get_openviking_config
from openviking_cli.utils.config.vlm_config import VLMConfig
@@ -171,7 +172,7 @@ class TestGetContextForSearch:
context = await session.get_context_for_search(query="test")
assert isinstance(context, dict)
- assert context["latest_archive_overview"] == latest_overview
+ assert context["latest_archive_overview"] == body_for_preview(latest_overview)
assert len(context["current_messages"]) == 1
async def test_get_context_skips_incomplete_latest_archive(self, client: partial):
@@ -196,7 +197,7 @@ class TestGetContextForSearch:
context = await session.get_context_for_search(query="test")
- assert context["latest_archive_overview"] == completed_overview
+ assert context["latest_archive_overview"] == body_for_preview(completed_overview)
async def test_get_context_includes_incomplete_archive_messages(self, client: partial):
"""Pending archive messages should be merged with current live messages."""
@@ -333,7 +334,7 @@ class TestGetContextForSearch:
ctx=session.ctx,
)
context = await session.get_context_for_search(query="test")
- assert context["latest_archive_overview"] == first_overview
+ assert context["latest_archive_overview"] == body_for_preview(first_overview)
assert [m.content for m in context["current_messages"]] == [
"Second round user",
"Second round assistant",
@@ -348,7 +349,7 @@ class TestGetContextForSearch:
ctx=session.ctx,
)
context = await session.get_context_for_search(query="test")
- assert context["latest_archive_overview"] == second_overview
+ assert context["latest_archive_overview"] == body_for_preview(second_overview)
assert context["current_messages"] == []
diff --git a/tests/storage/test_content_write_processing_mode.py b/tests/storage/test_content_write_processing_mode.py
index 25c582999..9b4ce79cb 100644
--- a/tests/storage/test_content_write_processing_mode.py
+++ b/tests/storage/test_content_write_processing_mode.py
@@ -6,9 +6,14 @@ from unittest.mock import AsyncMock
import pytest
+from openviking.core.context import ContextLevel
from openviking.server.identity import RequestContext, Role
from openviking.storage import content_write as content_write_module
from openviking.storage.content_write import ContentWriteCoordinator
+from openviking.storage.semantic_sidecar import (
+ parse_semantic_sidecar,
+ render_semantic_sidecar,
+)
from openviking_cli.session.user_id import UserIdentifier
@@ -37,13 +42,32 @@ class _FakeVikingFS:
return f"/fake/{uri}"
+def _sidecar(level=ContextLevel.ABSTRACT, body="Original body."):
+ return render_semantic_sidecar(
+ level,
+ "viking://resources/demo",
+ body,
+ {
+ "generated_by": {"component": "test", "trigger": "test"},
+ "freshness": {
+ "total_entries": 1,
+ "sampled_entries": 1,
+ "unsampled_entries": 0,
+ "pending_child_changes": 0,
+ },
+ },
+ )
+
+
@pytest.fixture
def ctx():
return RequestContext(user=UserIdentifier("account-1", "user-1"), role=Role.USER)
@pytest.mark.asyncio
-async def test_vectors_only_write_skips_semantic_refresh_and_vectorizes_file(monkeypatch, ctx):
+async def test_direct_write_skips_semantic_refresh_for_vectors_only_and_sidecar_body_edits(
+ monkeypatch, ctx
+):
fake_fs = _FakeVikingFS()
vectorize_file = AsyncMock(return_value=True)
semantic_refresh = AsyncMock(side_effect=AssertionError("semantic refresh should not run"))
@@ -77,6 +101,41 @@ async def test_vectors_only_write_skips_semantic_refresh_and_vectorizes_file(mon
assert result["semantic_status"] == "skipped"
assert result["vector_status"] == "queued"
+ current = _sidecar()
+ sidecar_fs = _FakeVikingFS()
+ sidecar_fs.read_file.side_effect = [
+ current,
+ current,
+ _sidecar(ContextLevel.OVERVIEW, "Overview."),
+ ]
+ vectorize_directory = AsyncMock()
+ monkeypatch.setattr(content_write_module, "vectorize_directory_meta", vectorize_directory)
+ sidecar_coordinator = ContentWriteCoordinator(viking_fs=sidecar_fs)
+ sidecar_coordinator._enqueue_semantic_refresh = AsyncMock(
+ side_effect=AssertionError("sidecar body writes must not regenerate semantics")
+ )
+
+ sidecar_result = await sidecar_coordinator._write_direct_with_refresh(
+ uri="viking://resources/demo/.abstract.md",
+ root_uri="viking://resources/demo",
+ content="Updated body only.",
+ mode="replace",
+ context_type="resource",
+ wait=False,
+ timeout=None,
+ ctx=ctx,
+ written_bytes=len("Updated body only.".encode()),
+ telemetry_id="",
+ )
+
+ written = sidecar_fs.write_file.await_args.args[1]
+ assert parse_semantic_sidecar(written).body == "Updated body only.\n"
+ assert parse_semantic_sidecar(written).metadata == parse_semantic_sidecar(current).metadata
+ sidecar_coordinator._enqueue_semantic_refresh.assert_not_awaited()
+ vectorize_directory.assert_awaited_once()
+ assert sidecar_result["semantic_status"] == "skipped"
+ assert sidecar_result["vector_status"] == "queued"
+
@pytest.mark.asyncio
async def test_vectors_only_write_wait_reports_embedding_status(monkeypatch, ctx):
diff --git a/tests/storage/test_semantic_dag_incremental.py b/tests/storage/test_semantic_dag_incremental.py
index 67405030e..bd0f7541b 100644
--- a/tests/storage/test_semantic_dag_incremental.py
+++ b/tests/storage/test_semantic_dag_incremental.py
@@ -2,12 +2,18 @@
# SPDX-License-Identifier: AGPL-3.0
import re
+from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
+from openviking.core.context import ContextLevel
from openviking.server.identity import RequestContext, Role
from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor
+from openviking.storage.semantic_sidecar import (
+ parse_semantic_sidecar,
+ render_semantic_sidecar,
+)
from openviking_cli.session.user_id import UserIdentifier
@@ -139,11 +145,25 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypa
file_contents={
f"{root_uri}/a.txt": "new content",
f"{root_uri}/b.txt": "unchanged",
- f"{root_uri}/.overview.md": "FILES:\n- a.txt: old-a\n- b.txt: old-b",
+ f"{root_uri}/.overview.md": render_semantic_sidecar(
+ ContextLevel.OVERVIEW,
+ root_uri,
+ "FILES:\n- a.txt: old-a\n- b.txt: old-b",
+ {
+ "generated_by": {
+ "component": "SemanticProcessor",
+ "trigger": "previous_refresh",
+ }
+ },
+ ),
f"{root_uri}/.abstract.md": "old-abstract",
},
)
monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(sidecar_sample_size=32)),
+ )
processor = _FakeProcessor(fake_fs)
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
@@ -162,7 +182,7 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypa
assert processor.summarized_files == [f"{root_uri}/a.txt"]
assert processor.vectorized_files == [f"{root_uri}/a.txt"]
assert processor.sync_calls == []
- overview = fake_fs._file_contents[f"{root_uri}/.overview.md"]
+ overview = parse_semantic_sidecar(fake_fs._file_contents[f"{root_uri}/.overview.md"]).body
assert "- a.txt: summary" in overview
assert "- b.txt: old-b" in overview
diff --git a/tests/storage/test_semantic_dag_stats.py b/tests/storage/test_semantic_dag_stats.py
index d6f6d0a83..396565070 100644
--- a/tests/storage/test_semantic_dag_stats.py
+++ b/tests/storage/test_semantic_dag_stats.py
@@ -2,6 +2,7 @@
# SPDX-License-Identifier: AGPL-3.0
import asyncio
+from types import SimpleNamespace
import pytest
@@ -19,6 +20,7 @@ from openviking.storage.queuefs.semantic_dag import (
SemanticDagExecutor,
SemanticNodeScheduler,
)
+from openviking.storage.semantic_sidecar import parse_semantic_sidecar
from openviking.telemetry import (
OperationTelemetry,
bind_telemetry,
@@ -143,6 +145,13 @@ class _ScheduledExecutor:
self.failure = exc
+def _patch_semantic_config(monkeypatch, *, sidecar_sample_size=32):
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(sidecar_sample_size=sidecar_sample_size)),
+ )
+
+
@pytest.mark.asyncio
async def test_semantic_dag_stats_collects_nodes(monkeypatch):
root_uri = "viking://resources/root"
@@ -158,6 +167,7 @@ async def test_semantic_dag_stats_collects_nodes(monkeypatch):
}
fake_fs = _FakeVikingFS(tree)
monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ _patch_semantic_config(monkeypatch)
processor = _FakeProcessor(verify_streaming=True)
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
@@ -194,6 +204,7 @@ async def test_semantic_dag_bounds_active_node_work(monkeypatch):
}
fake_fs = _FakeVikingFS(tree)
monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ _patch_semantic_config(monkeypatch)
processor = _TrackingProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
@@ -231,6 +242,7 @@ async def test_semantic_dag_shares_node_scheduler_across_roots(monkeypatch):
}
fake_fs = _FakeVikingFS(tree)
monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ _patch_semantic_config(monkeypatch)
processor = _TrackingProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
@@ -262,12 +274,12 @@ async def test_semantic_dag_shares_node_scheduler_across_roots(monkeypatch):
assert processor.max_active_summaries == 1
assert executor_a.get_stats().done_nodes == 21
assert executor_b.get_stats().done_nodes == 21
- assert {
- processor.vectorized_contexts[f"{root_a}/a-{idx}.txt"] for idx in range(20)
- } == {("task-a", telemetry_a.telemetry_id)}
- assert {
- processor.vectorized_contexts[f"{root_b}/b-{idx}.txt"] for idx in range(20)
- } == {("task-b", telemetry_b.telemetry_id)}
+ assert {processor.vectorized_contexts[f"{root_a}/a-{idx}.txt"] for idx in range(20)} == {
+ ("task-a", telemetry_a.telemetry_id)
+ }
+ assert {processor.vectorized_contexts[f"{root_b}/b-{idx}.txt"] for idx in range(20)} == {
+ ("task-b", telemetry_b.telemetry_id)
+ }
@pytest.mark.asyncio
@@ -332,6 +344,7 @@ async def test_semantic_dag_skip_vectorization_does_not_schedule_tasks(monkeypat
}
fake_fs = _FakeVikingFS(tree)
monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ _patch_semantic_config(monkeypatch)
processor = _FakeProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
@@ -345,13 +358,25 @@ async def test_semantic_dag_skip_vectorization_does_not_schedule_tasks(monkeypat
await executor.run(root_uri)
await asyncio.sleep(0)
- assert fake_fs.writes == [
- (f"{root_uri}/child/.overview.md", "overview"),
- (f"{root_uri}/child/.abstract.md", "abstract"),
- (f"{root_uri}/.overview.md", "overview"),
- (f"{root_uri}/.abstract.md", "abstract"),
+ assert [uri for uri, _ in fake_fs.writes] == [
+ f"{root_uri}/child/.overview.md",
+ f"{root_uri}/child/.abstract.md",
+ f"{root_uri}/.overview.md",
+ f"{root_uri}/.abstract.md",
]
+ assert [parse_semantic_sidecar(raw).body.strip() for _, raw in fake_fs.writes] == [
+ "overview",
+ "abstract",
+ "overview",
+ "abstract",
+ ]
+ assert all(
+ parse_semantic_sidecar(raw).metadata["generated_by"]["component"] == "SemanticProcessor"
+ for _, raw in fake_fs.writes
+ )
assert processor.vectorized_dirs == []
assert processor.vectorized_files == []
+
+
if __name__ == "__main__":
pytest.main([__file__])
diff --git a/tests/storage/test_semantic_msg_ingest_options.py b/tests/storage/test_semantic_msg_ingest_options.py
index a828c1469..a9f279ee5 100644
--- a/tests/storage/test_semantic_msg_ingest_options.py
+++ b/tests/storage/test_semantic_msg_ingest_options.py
@@ -1,8 +1,8 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
-from openviking.utils.ingest_options import IngestOptions
from openviking.storage.queuefs.semantic_msg import SemanticMsg
+from openviking.utils.ingest_options import IngestOptions
def test_semantic_msg_serializes_ingest_options():
@@ -10,6 +10,8 @@ def test_semantic_msg_serializes_ingest_options():
uri="viking://resources/demo",
context_type="resource",
ingest_options=IngestOptions(search_tags=["team=search"], search_tag_mode="append"),
+ source={"kind": "git", "uri": "https://example.com/acme/demo.git"},
+ generation_trigger="resource_ingest",
)
data = msg.to_dict()
@@ -23,6 +25,11 @@ def test_semantic_msg_serializes_ingest_options():
search_tags=["team=search"],
search_tag_mode="append",
)
+ assert restored.source == {
+ "kind": "git",
+ "uri": "https://example.com/acme/demo.git",
+ }
+ assert restored.generation_trigger == "resource_ingest"
def test_semantic_msg_reads_legacy_search_tag_fields():
diff --git a/tests/storage/test_semantic_processor_l0_l1.py b/tests/storage/test_semantic_processor_l0_l1.py
index 16c2c165d..4cb04de90 100644
--- a/tests/storage/test_semantic_processor_l0_l1.py
+++ b/tests/storage/test_semantic_processor_l0_l1.py
@@ -3,8 +3,10 @@
from types import SimpleNamespace
+from openviking.core.context import ContextLevel
from openviking.storage.queuefs import semantic_processor as semantic_processor_module
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+from openviking.storage.semantic_sidecar import render_semantic_sidecar
def _patch_semantic_limits(monkeypatch, *, abstract_max_chars=256, overview_max_chars=4000):
@@ -32,6 +34,19 @@ def test_markdown_overview_uses_brief_description_as_abstract(monkeypatch):
assert overview == generated
assert abstract == "This brief description is the retrieval abstract."
+ raw = render_semantic_sidecar(
+ ContextLevel.OVERVIEW,
+ "viking://resources/demo",
+ generated,
+ {"source": {"kind": "http", "uri": "https://example.com/private.pdf"}},
+ )
+
+ overview, abstract = processor._normalize_overview_generation(raw)
+
+ assert overview == generated
+ assert abstract == "This brief description is the retrieval abstract."
+ assert "source:" not in overview
+
def test_markdown_overview_extracts_multiline_brief_description(monkeypatch):
_patch_semantic_limits(monkeypatch)
@@ -92,9 +107,7 @@ def test_overview_truncation_prefers_complete_sentence(monkeypatch):
_patch_semantic_limits(monkeypatch, overview_max_chars=45)
processor = SemanticProcessor()
overview = (
- "# README\n\n"
- "This is a complete sentence. "
- "This second sentence would be cut in the middle."
+ "# README\n\nThis is a complete sentence. This second sentence would be cut in the middle."
)
overview, abstract = processor._enforce_size_limits(overview, "abstract")
@@ -106,12 +119,7 @@ def test_overview_truncation_prefers_complete_sentence(monkeypatch):
def test_overview_truncation_keeps_last_complete_sentence_within_limit(monkeypatch):
_patch_semantic_limits(monkeypatch, overview_max_chars=57)
processor = SemanticProcessor()
- overview = (
- "# README\n\n"
- "First sentence. "
- "Second sentence. "
- "Third sentence should be omitted."
- )
+ overview = "# README\n\nFirst sentence. Second sentence. Third sentence should be omitted."
overview, abstract = processor._enforce_size_limits(overview, "abstract")
diff --git a/tests/storage/test_semantic_queue_memory_dedupe.py b/tests/storage/test_semantic_queue_memory_dedupe.py
index 508ff21a2..bac539edb 100644
--- a/tests/storage/test_semantic_queue_memory_dedupe.py
+++ b/tests/storage/test_semantic_queue_memory_dedupe.py
@@ -2,6 +2,7 @@
# SPDX-License-Identifier: Apache-2.0
"""Tests for memory-context semantic enqueue deduplication (#769)."""
+from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
from uuid import uuid4
@@ -11,6 +12,7 @@ from openviking.storage.queuefs.named_queue import NamedQueue
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
from openviking.storage.queuefs.semantic_queue import SemanticQueue, is_semantic_msg_stale
+from openviking.storage.semantic_sidecar import parse_semantic_sidecar
@pytest.mark.asyncio
@@ -139,6 +141,13 @@ class _FakeMemoryDirFS:
]
+def _patch_semantic_config(monkeypatch, *, sidecar_sample_size=32):
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_processor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(sidecar_sample_size=sidecar_sample_size)),
+ )
+
+
@pytest.mark.asyncio
async def test_stale_memory_semantic_write_is_skipped(monkeypatch):
pathlock = _FakePathLock()
@@ -186,9 +195,13 @@ async def test_stale_memory_semantic_write_is_skipped(monkeypatch):
"/fake/viking/user/default/memories/preferences/.abstract.md",
]
]
- assert viking_fs.writes == [
- ("viking://user/default/memories/preferences/.overview.md", "latest overview"),
- ("viking://user/default/memories/preferences/.abstract.md", "latest abstract"),
+ assert [uri for uri, _ in viking_fs.writes] == [
+ "viking://user/default/memories/preferences/.overview.md",
+ "viking://user/default/memories/preferences/.abstract.md",
+ ]
+ assert [parse_semantic_sidecar(raw).body.strip() for _, raw in viking_fs.writes] == [
+ "latest overview",
+ "latest abstract",
]
@@ -215,6 +228,7 @@ async def test_memory_directory_summarizes_all_uncached_files(monkeypatch):
"openviking.storage.queuefs.semantic_processor.get_viking_fs",
lambda: _FakeMemoryDirFS(),
)
+ _patch_semantic_config(monkeypatch)
monkeypatch.setattr(processor, "_generate_single_file_summary", generate_file_summary)
monkeypatch.setattr(processor, "_generate_overview", generate_overview)
monkeypatch.setattr(
@@ -268,6 +282,7 @@ async def test_memory_directory_vectorizes_changed_files_with_generated_summary(
"openviking.storage.queuefs.semantic_processor.get_viking_fs",
lambda: _FakeMemoryDirFS(),
)
+ _patch_semantic_config(monkeypatch)
monkeypatch.setattr(processor, "_generate_single_file_summary", generate_file_summary)
monkeypatch.setattr(processor, "_generate_overview", generate_overview)
monkeypatch.setattr(
diff --git a/tests/unit/test_summarizer_resources_root_split.py b/tests/unit/test_summarizer_resources_root_split.py
index 170408ae3..6426264fe 100644
--- a/tests/unit/test_summarizer_resources_root_split.py
+++ b/tests/unit/test_summarizer_resources_root_split.py
@@ -152,12 +152,19 @@ async def test_explicit_subpath_not_split():
resource_uris=["viking://resources/foo"],
ctx=ctx,
temp_uris=["viking://temp/import_root"],
+ semantic_source={"kind": "http", "uri": "https://example.com/demo.pdf"},
+ generation_trigger="resource_ingest",
)
assert res["status"] == "success"
assert res["enqueued_count"] == 1
assert queue.msgs[0].target_uri == "viking://resources/foo"
assert queue.msgs[0].uri == "viking://temp/import_root"
+ assert queue.msgs[0].source == {
+ "kind": "http",
+ "uri": "https://example.com/demo.pdf",
+ }
+ assert queue.msgs[0].generation_trigger == "resource_ingest"
@pytest.mark.asyncio
diff --git a/tests/unit/test_vectorize_file_strategy.py b/tests/unit/test_vectorize_file_strategy.py
index c5ff22d0a..e95338e32 100644
--- a/tests/unit/test_vectorize_file_strategy.py
+++ b/tests/unit/test_vectorize_file_strategy.py
@@ -561,25 +561,44 @@ async def test_vectorize_directory_meta_appends_search_tags_by_level(monkeypatch
async def test_vectorize_directory_meta_l1_abstract_is_overview(monkeypatch):
"""L1 records must carry the overview in the abstract scalar so Rerank
sees L1 text instead of the L0 abstract."""
+ from openviking.core.context import ContextLevel
+ from openviking.storage.semantic_sidecar import render_semantic_sidecar
+
queue = DummyQueue()
monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))
monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("ignored"))
+ uri = "viking://user/default/resources/demo"
overview = "# Overview\n\n" + ("section detail. " * 50)
+ metadata = {
+ "source": {"kind": "http", "uri": "https://example.com/private.pdf"},
+ "generated_by": {"component": "SemanticProcessor", "trigger": "ingest"},
+ "freshness": {
+ "total_entries": 4,
+ "sampled_entries": 2,
+ "unsampled_entries": 2,
+ "pending_child_changes": 0,
+ },
+ }
await embedding_utils.vectorize_directory_meta(
- uri="viking://user/default/resources/demo",
- abstract="demo abstract",
- overview=overview,
+ uri=uri,
+ abstract=render_semantic_sidecar(ContextLevel.ABSTRACT, uri, "Visible abstract.", metadata),
+ overview=render_semantic_sidecar(ContextLevel.OVERVIEW, uri, overview, metadata),
ctx=DummyReq(),
)
assert len(queue.items) == 2
l0, l1 = queue.items
assert l0.context_data["level"] == 0
- assert l0.context_data["abstract"] == "demo abstract"
+ assert l0.context_data["abstract"] == "Visible abstract."
assert l1.context_data["level"] == 1
- assert l1.context_data["abstract"] == overview
- assert l1.message == overview
+ assert l1.context_data["abstract"] == overview.rstrip()
+ assert l1.message == (f"---\ndirectory: {uri}/\n---\n\n{overview.rstrip()}")
+ for item in (l0, l1):
+ assert f"directory: {uri}/" in item.message
+ assert "source:" not in item.message
+ assert "generated_by:" not in item.message
+ assert "freshness:" not in item.message
@pytest.mark.asyncio
diff --git a/tests/utils/test_resource_processor_processing_mode.py b/tests/utils/test_resource_processor_processing_mode.py
index 86407ac15..572328f16 100644
--- a/tests/utils/test_resource_processor_processing_mode.py
+++ b/tests/utils/test_resource_processor_processing_mode.py
@@ -62,9 +62,7 @@ async def test_flat_file_refreshes_parent_semantics_and_vectorizes_via_summary(
lambda: viking_fs,
)
processor = ResourceProcessor(_FakeVikingDB())
- summarizer = SimpleNamespace(
- refresh_file_parent=AsyncMock(return_value={"status": "success"})
- )
+ summarizer = SimpleNamespace(refresh_file_parent=AsyncMock(return_value={"status": "success"}))
processor._get_summarizer = Mock(return_value=summarizer)
result = await processor.finish_prepared_resource(
@@ -126,9 +124,7 @@ async def test_vectors_only_replaces_preexisting_flat_file_without_directory_syn
viking_fs = SimpleNamespace(
_async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),
exists=AsyncMock(return_value=True),
- ls=AsyncMock(
- side_effect=NotADirectoryError("flat resource roots cannot be listed")
- ),
+ ls=AsyncMock(side_effect=NotADirectoryError("flat resource roots cannot be listed")),
persist_temp_tree=AsyncMock(),
delete_temp=AsyncMock(),
)
@@ -226,7 +222,9 @@ async def test_vectors_only_persists_tree_and_vectorizes_files_only(monkeypatch,
rewrite_image_uris = AsyncMock()
monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
- monkeypatch.setattr("openviking.utils.resource_processor.rewrite_image_uris", rewrite_image_uris)
+ monkeypatch.setattr(
+ "openviking.utils.resource_processor.rewrite_image_uris", rewrite_image_uris
+ )
monkeypatch.setattr("openviking.utils.resource_processor.vectorize_file", vectorize_file)
monkeypatch.setattr(
"openviking.utils.resource_processor.get_openviking_config",
@@ -348,6 +346,14 @@ async def test_vectors_only_syncs_preexisting_target_instead_of_merging(monkeypa
)
monkeypatch.setattr("openviking.utils.resource_processor.rewrite_image_uris", AsyncMock())
monkeypatch.setattr("openviking.utils.resource_processor.vectorize_file", AsyncMock())
+ monkeypatch.setattr(
+ "openviking.utils.resource_processor.get_openviking_config",
+ lambda: SimpleNamespace(
+ queue_workers=SimpleNamespace(
+ add_resource=SimpleNamespace(file_vectorization_concurrency=8)
+ )
+ ),
+ )
processor = ResourceProcessor(_FakeVikingDB())
processor._delete_resource_semantic_vectors = AsyncMock()
lock = {"lease_ref": "lock-1"}
@@ -464,6 +470,14 @@ async def test_vectors_only_deletes_sync_removed_detail_vectors(monkeypatch, ctx
)
monkeypatch.setattr("openviking.utils.resource_processor.rewrite_image_uris", AsyncMock())
monkeypatch.setattr("openviking.utils.resource_processor.vectorize_file", AsyncMock())
+ monkeypatch.setattr(
+ "openviking.utils.resource_processor.get_openviking_config",
+ lambda: SimpleNamespace(
+ queue_workers=SimpleNamespace(
+ add_resource=SimpleNamespace(file_vectorization_concurrency=8)
+ )
+ ),
+ )
processor = ResourceProcessor(_FakeVikingDB())
processor._delete_resource_semantic_vectors = AsyncMock()
processor._delete_removed_resource_vectors = AsyncMock()