Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(ingest): replay local agent-harness logs into OpenViking sessions (#2892)

t0saki72d04cd488b5
diff --git a/docs/en/agent-integrations/09-log-ingestion.md b/docs/en/agent-integrations/09-log-ingestion.md
new file mode 100644
index 000000000..e6c81f8c1
--- /dev/null
+++ b/docs/en/agent-integrations/09-log-ingestion.md
@@ -0,0 +1,117 @@
+# Import Local Agent Logs (openviking-server ingest)
+
+`openviking-server ingest` parses the conversation logs that AI coding / agent harnesses (Claude Code, Codex, OpenCode, Hermes, OpenClaw) already leave on your machine, then "replays" them through OpenViking's existing session pipeline (`create session → batch add messages → commit`, where commit triggers memory extraction). This turns both your historical and newly written conversations into long-term memory. It complements the per-harness memory plugins: a plugin captures **while a conversation is happening**, whereas this tool is for **importing existing logs** and **watching for new logs offline** — no plugin required and no change to the harness itself.
+
+Key difference from the plugins: this tool is an OpenViking **client**. It runs where the logs live and points at a local or remote server via the SDK, and it is **off by default** — installing OpenViking does not silently scan your local files.
+
+Source: [openviking/ingest](https://github.com/volcengine/OpenViking/tree/main/openviking/ingest)
+
+## Off by default
+
+The feature is doubly disabled and must be turned on explicitly:
+
+- the master switch `ingest.enabled` defaults to `false`;
+- each harness's `enabled` defaults to `false`, and harnesses you do not list are never read;
+- backfill of existing logs is a manual command and supports `--dry-run` (count only, write nothing) and `--since` (bound the time window) so you can verify first.
+
+## Supported harnesses
+
+| harness | Status | Default log path | Notes |
+|---|---|---|---|
+| `claude_code` | Supported | `~/.claude/projects/*/*.jsonl` | append-only JSONL, byte-offset cursor |
+| `codex` | Supported | `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` | append-only JSONL |
+| `hermes` | Supported | `~/.hermes/sessions/*.jsonl` | group-chat agent; user peer = original username |
+| `openclaw` | Supported | `~/.openclaw/agents/*/sessions/*.jsonl` | group-chat agent; user peer = original username |
+| `opencode` | Experimental | `~/.local/share/opencode/opencode.db` | SQLite, polled by `(time, id)`; the legacy file-store is not supported |
+| `cursor` | Deferred | `~/Library/Application Support/Cursor/User/**/state.vscdb` | undocumented, version-unstable KV blobs; not yet implemented |
+
+> "harness" (agent framework) here means a whole tool like Claude Code or Codex — distinct from OpenViking's "tool" (tool-call) concept.
+
+## Enable in ov.conf
+
+Add an `ingest` section to `ov.conf`, listing the harnesses to import and their mode:
+
+```json
+{
+ "ingest": {
+ "enabled": true,
+ "server_url": "$OPENVIKING_URL",
+ "api_key": "$OPENVIKING_API_KEY",
+ "account": "default",
+ "user": "default",
+ "harnesses": {
+ "claude_code": { "enabled": true, "mode": "both" },
+ "codex": { "enabled": true, "mode": "backfill" },
+ "opencode": { "enabled": false, "mode": "watch", "experimental": true },
+ "hermes": { "enabled": false, "mode": "both", "user_field": "sender" },
+ "openclaw": { "enabled": false, "mode": "both", "user_field": "sender" }
+ }
+ }
+}
+```
+
+- `mode`: `off` | `backfill` (one-shot import of existing logs) | `watch` (incremental) | `both`.
+- `paths`: override the harness's default discovery roots (multiple allowed).
+- `user_field`: for group-chat harnesses, the log key holding the original username, used as the user-side peer_id.
+- `commit`: commit policy — `commit_token_threshold`, `commit_idle_seconds`, `keep_recent_count`.
+- Deploy-time toggles can also be overridden via env: `OPENVIKING_INGEST_ENABLED`, `OPENVIKING_INGEST_SERVER_URL`, `OPENVIKING_INGEST_API_KEY`.
+
+When `server_url` is empty it falls back to `OPENVIKING_URL` or `http://localhost:1933`, so it can target either a local or a remote server.
+
+## Usage
+
+The `openviking-server ingest` command is installed together with OpenViking.
+
+```bash
+# Show registered harnesses and their config
+openviking-server ingest list-sources
+
+# Dry run first: count the sessions / messages that would be replayed, write nothing
+openviking-server ingest backfill --dry-run
+
+# Backfill one harness, only sessions started on/after a date
+openviking-server ingest backfill --harness claude_code --since 2026-06-01
+
+# Backfill existing logs for real
+openviking-server ingest backfill
+
+# Watch for new logs and replay incrementally (blocks in the foreground)
+openviking-server ingest watch --harness claude_code
+
+# Honor each harness's configured mode: backfill then watch
+openviking-server ingest run
+
+# Show how far each session has been ingested (read the cursor state)
+openviking-server ingest status
+```
+
+`--reset` deletes and recreates the OV session before replaying. Without `--reset`, re-running is idempotent — the cursor store guarantees nothing is appended twice.
+
+## peer_id
+
+Every message carries a peer_id so OpenViking can profile both the human and the model:
+
+- assistant turns: `{harness}/{model}` (or `{harness}/{provider}/{model}` when the provider is meaningful), e.g. `claude_code/claude-opus-4-8`, `opencode/bytedance_ark/doubao-...`;
+- user turns: single-user dev harnesses (claude_code / codex / opencode) use the git identity of the session cwd repo (`user.email` / `user.name`), falling back to the configured `ingest.user` when there is no git repo; group-chat harnesses (hermes / openclaw) use the original username from the log (selected by `user_field`).
+
+Non-ASCII identifiers (e.g. a CJK username) fall back to a valid `ext-<base64>` form.
+
+## How it works
+
+Each harness has a thin adapter that parses its logs into normalized messages and hands them to a replayer that runs `ensure_session → batch add (<=100 per call) → commit`. Memory extraction only runs on **commit**, server-side. OV session ids are `import__{harness}__{native_session_id}` — deterministic and idempotent.
+
+- **Backfill** enumerates every session, replays from the cursor to the end, then commits once per session.
+- **Watch** mirrors OpenViking's own `WatchScheduler`: it uses **interval polling** (not filesystem events) driven by durable cursors, so a missed tick, a sleep, or a restart just reads cursor→end on the next tick and self-heals. JSONL uses a byte-offset cursor (with partial-line / truncation / rotation handling); SQLite uses a `(time, id)` cursor read read-only (WAL-aware).
+
+Cursor state persists in `~/.openviking/ingest/state.db`, so both backfill and watch resume across restarts without re-ingesting.
+
+## Cost and privacy
+
+- Commit triggers memory extraction (an LLM call). Backfilling months of history at once can produce many calls — prefer `--dry-run` first, narrow with `--since`, and enable harnesses in batches.
+- Logs may contain sensitive content (credentials, file contents). Use this in a trusted deployment and confirm `server_url` points at the server you intend.
+- Tool-call inputs/outputs are dropped as low-value by default; only user / assistant text is ingested.
+
+## See also
+
+- [Overview](./01-overview.md) — the per-harness memory plugins (real-time capture)
+- [Deployment guide → CLI](../guides/03-deployment.md#cli) — `ov.conf` / credential configuration
diff --git a/docs/zh/agent-integrations/09-log-ingestion.md b/docs/zh/agent-integrations/09-log-ingestion.md
new file mode 100644
index 000000000..135b8d9a3
--- /dev/null
+++ b/docs/zh/agent-integrations/09-log-ingestion.md
@@ -0,0 +1,117 @@
+# 导入本地 Agent 日志(openviking-server ingest)
+
+`openviking-server ingest` 把你本地已有的 AI 编码 / agent harness 的对话日志(Claude Code、Codex、OpenCode、Hermes、OpenClaw)解析成标准消息,再通过 OpenViking 既有的会话管线“重放”进去(`创建会话 → 批量追加消息 → 提交`,提交时触发记忆抽取),从而把这些历史与新增对话沉淀为长期记忆。它与各 harness 的“记忆插件”互补:插件在对话**进行时**实时挂载捕获,而本工具用于**导入既有日志**与**离线监听新增日志**,无需插件、也无需改动对应 harness。
+
+与插件方案的关键区别:本工具是 OpenViking 的**客户端**,跑在日志所在的机器上,通过 SDK 指向本地或远端 server;它默认**完全关闭**,不会“装上就扫你本地文件”。
+
+源码:[openviking/ingest](https://github.com/volcengine/OpenViking/tree/main/openviking/ingest)
+
+## 默认关闭
+
+该特性默认双重关闭,必须显式开启:
+
+- 总开关 `ingest.enabled` 默认 `false`;
+- 每个 harness 的 `enabled` 默认 `false`,未列出的 harness 不会被读取;
+- 存量回填需手动运行命令,且支持 `--dry-run`(只统计、不写入)与 `--since`(限定时间窗)先行验证。
+
+## 支持的 harness
+
+| harness | 状态 | 默认日志路径 | 说明 |
+|---|---|---|---|
+| `claude_code` | 支持 | `~/.claude/projects/*/*.jsonl` | append-only JSONL,字节偏移游标 |
+| `codex` | 支持 | `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` | append-only JSONL |
+| `hermes` | 支持 | `~/.hermes/sessions/*.jsonl` | 群聊 agent,user 取原始用户名 |
+| `openclaw` | 支持 | `~/.openclaw/agents/*/sessions/*.jsonl` | 群聊 agent,user 取原始用户名 |
+| `opencode` | 实验性 | `~/.local/share/opencode/opencode.db` | SQLite,按 `(time, id)` 轮询;旧版文件存储暂不支持 |
+| `cursor` | 暂缓 | `~/Library/Application Support/Cursor/User/**/state.vscdb` | 无文档、随版本漂移的 KV blob,暂未实现 |
+
+> 这里的 harness(agent 框架)指 CC / Codex 等整套工具,区别于 OpenViking 里“tool(工具调用)”的概念。
+
+## 在 ov.conf 中开启
+
+在 `ov.conf` 增加 `ingest` 段,列出要导入的 harness 并设置其模式:
+
+```json
+{
+ "ingest": {
+ "enabled": true,
+ "server_url": "$OPENVIKING_URL",
+ "api_key": "$OPENVIKING_API_KEY",
+ "account": "default",
+ "user": "default",
+ "harnesses": {
+ "claude_code": { "enabled": true, "mode": "both" },
+ "codex": { "enabled": true, "mode": "backfill" },
+ "opencode": { "enabled": false, "mode": "watch", "experimental": true },
+ "hermes": { "enabled": false, "mode": "both", "user_field": "sender" },
+ "openclaw": { "enabled": false, "mode": "both", "user_field": "sender" }
+ }
+ }
+}
+```
+
+- `mode`:`off` | `backfill`(一次性导入存量)| `watch`(监听新增)| `both`。
+- `paths`:覆盖该 harness 的默认发现路径(可填多个)。
+- `user_field`:群聊 harness 中存放原始用户名的字段名,用作 user 侧 peer_id。
+- `commit`:提交策略,含 `commit_token_threshold`、`commit_idle_seconds`、`keep_recent_count`。
+- 部署期开关也可用环境变量覆盖:`OPENVIKING_INGEST_ENABLED`、`OPENVIKING_INGEST_SERVER_URL`、`OPENVIKING_INGEST_API_KEY`。
+
+`server_url` 留空时回退到 `OPENVIKING_URL` 或 `http://localhost:1933`,因此既能指向本地 server,也能指向远端。
+
+## 使用
+
+`openviking-server ingest` 命令随 OpenViking 一同安装。
+
+```bash
+# 查看已注册 harness 及其配置
+openviking-server ingest list-sources
+
+# 先干跑:统计会回填多少 session / 消息,不写入
+openviking-server ingest backfill --dry-run
+
+# 只回填某个 harness、且只回填某日期之后的会话
+openviking-server ingest backfill --harness claude_code --since 2026-06-01
+
+# 正式回填(存量)
+openviking-server ingest backfill
+
+# 监听新增日志并增量重放(前台阻塞)
+openviking-server ingest watch --harness claude_code
+
+# 按每个 harness 配置的 mode 执行:先回填再监听
+openviking-server ingest run
+
+# 查看各会话已导入到哪里(读取游标状态)
+openviking-server ingest status
+```
+
+`--reset` 会在重放前删除并重建对应的 OV 会话;不加 `--reset` 时,重复运行是幂等的(游标保证不会重复追加)。
+
+## peer_id
+
+每条消息都会带上 peer_id,便于 OpenViking 同时为人类与模型建立画像:
+
+- assistant 消息:`{harness}/{模型名}`(provider 有意义时为 `{harness}/{provider}/{模型名}`),例如 `claude_code/claude-opus-4-8`、`opencode/bytedance_ark/doubao-...`;
+- user 消息:单用户开发型 harness(claude_code / codex / opencode)取会话 cwd 所在仓库的 git 身份(`user.email` / `user.name`),无 git 仓库时回退为配置的 `ingest.user`;群聊 harness(hermes / openclaw)取日志里的原始用户名(由 `user_field` 指定)。
+
+非 ASCII 标识(如中文用户名)会回退为合法的 `ext-<base64>` 形式。
+
+## 工作原理
+
+每个 harness 对应一个轻量适配器,把其日志解析为标准消息,交给“重放器”执行 `ensure_session → 批量追加(每批 ≤100)→ commit`。记忆抽取只在 **commit** 时由 server 端触发。OV 会话 id 形如 `import__{harness}__{原始会话id}`,确定且幂等。
+
+- **存量回填**:枚举所有会话,从游标读到末尾后逐会话提交一次。
+- **监听增量**:参照 OpenViking 自身的 `WatchScheduler`,用**定时轮询**(非文件系统事件)+ 持久游标驱动;漏一拍、休眠或重启后,下一拍从游标读到末尾即可自愈。JSONL 用字节偏移游标(含半行/截断/轮转处理),SQLite 用 `(time, id)` 游标只读读取(兼容 WAL)。
+
+游标状态持久化在 `~/.openviking/ingest/state.db`,因此回填与监听都能在重启后续传,且不会重复入库。
+
+## 成本与隐私
+
+- 提交会触发记忆抽取(LLM 调用)。一次性回填数月历史可能产生大量调用,建议先 `--dry-run`、用 `--since` 收窄时间窗、按 harness 分批开启。
+- 日志中可能含敏感内容(凭据、文件内容)。请在受信任的部署中使用,并确认 `server_url` 指向你期望的 server。
+- tool 调用的输入/输出默认按低价值丢弃,仅入库 user / assistant 文本。
+
+## 参见
+
+- [概览](./01-overview.md) — 各 harness 的记忆插件(实时捕获方案)
+- [部署指南 → CLI](../guides/03-deployment.md#cli) — `ov.conf` / 凭据配置
diff --git a/examples/ov.conf.example b/examples/ov.conf.example
index 01294ecd4..eaf98494d 100644
--- a/examples/ov.conf.example
+++ b/examples/ov.conf.example
@@ -162,6 +162,20 @@
"extraction_enabled": true,
"session_skill_extraction_enabled": false,
},
+ "ingest": {
+ "enabled": false,
+ "server_url": "$OPENVIKING_URL",
+ "api_key": "$OPENVIKING_API_KEY",
+ "account": "default",
+ "user": "default",
+ "harnesses": {
+ "claude_code": {"enabled": false, "mode": "both"},
+ "codex": {"enabled": false, "mode": "backfill"},
+ "opencode": {"enabled": false, "mode": "watch", "experimental": true},
+ "hermes": {"enabled": false, "mode": "both", "user_field": "sender"},
+ "openclaw": {"enabled": false, "mode": "both", "user_field": "sender"},
+ },
+ },
"log": {
"level": "INFO",
"format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
diff --git a/openviking/ingest/__init__.py b/openviking/ingest/__init__.py
new file mode 100644
index 000000000..dfad50acc
--- /dev/null
+++ b/openviking/ingest/__init__.py
@@ -0,0 +1,23 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Conversation-log ingest: replay local agent-harness logs into OpenViking sessions.
+
+Parses each harness's local conversation logs (Claude Code, Codex, OpenCode, Hermes,
+OpenClaw, Cursor) into normalized messages and replays them through OV's existing
+``create_session -> batch_add_messages -> commit`` pipeline (commit triggers OV's
+async memory extraction). Supports one-shot backfill of existing logs and
+incremental, cursor-driven polling of new logs.
+
+Inspired by / supersedes volcengine/OpenViking#2674 by @huang-yi-dae.
+"""
+
+from openviking.ingest.models import Cursor, NormalizedMessage, SessionRef
+from openviking.ingest.registry import SOURCE_REGISTRY, register_source
+
+__all__ = [
+ "Cursor",
+ "NormalizedMessage",
+ "SessionRef",
+ "SOURCE_REGISTRY",
+ "register_source",
+]
diff --git a/openviking/ingest/cli.py b/openviking/ingest/cli.py
new file mode 100644
index 000000000..2941bec73
--- /dev/null
+++ b/openviking/ingest/cli.py
@@ -0,0 +1,265 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""``openviking-server ingest`` CLI: replay local agent-harness logs into OpenViking.
+
+Commands:
+ list-sources show registered harnesses and their config
+ status show per-session ingest progress (read cursors)
+ backfill one-shot replay of existing logs
+ watch incremental, cursor-driven polling of new logs
+ run honor each harness's configured mode (backfill then watch)
+"""
+
+from __future__ import annotations
+
+import asyncio
+import signal
+from pathlib import Path
+from typing import List, Optional, Tuple
+
+import typer
+
+from openviking.ingest.cursor_store import CursorStore, SingleInstanceLock
+from openviking.ingest.orchestrator import BackfillStats, IngestOrchestrator, enabled_sources
+from openviking.ingest.poller import IngestPoller
+from openviking.ingest.registry import SOURCE_REGISTRY
+from openviking.ingest.replay import ConversationReplayClient, SessionReplayer
+from openviking_cli.utils import get_logger
+from openviking_cli.utils.config.consts import DEFAULT_INGEST_STATE_DIR
+from openviking_cli.utils.config.ingest_config import IngestConfig
+
+logger = get_logger(__name__)
+
+app = typer.Typer(
+ add_completion=False,
+ help="Replay local agent-harness conversation logs into OpenViking.",
+)
+
+
+def _load_ingest_config() -> IngestConfig:
+ """Read the ``ingest`` section of ov.conf if present; else a default (env-driven).
+
+ Only a *missing* config falls back to defaults; a malformed config surfaces its
+ validation/JSON error instead of silently looking like "ingest not configured".
+ """
+ from openviking_cli.utils.config.open_viking_config import OpenVikingConfigSingleton
+
+ try:
+ return OpenVikingConfigSingleton.get_instance().ingest
+ except FileNotFoundError:
+ return IngestConfig()
+
+
+def _state_dir(config: IngestConfig) -> Path:
+ return Path(config.state_dir).expanduser() if config.state_dir else DEFAULT_INGEST_STATE_DIR
+
+
+def _make_store(config: IngestConfig) -> CursorStore:
+ return CursorStore(_state_dir(config))
+
+
+def _acquire_lock(config: IngestConfig) -> SingleInstanceLock:
+ try:
+ return SingleInstanceLock(_state_dir(config)).acquire()
+ except RuntimeError as exc:
+ typer.echo(f"error: {exc}")
+ raise typer.Exit(1) from exc
+
+
+async def _make_replayer(
+ config: IngestConfig, store: CursorStore
+) -> Tuple[ConversationReplayClient, SessionReplayer]:
+ client = await ConversationReplayClient.create(
+ url=config.server_url,
+ api_key=config.api_key,
+ account=config.account,
+ user=config.user,
+ )
+ replayer = SessionReplayer(
+ client,
+ store,
+ session_id_prefix=config.session_id_prefix,
+ memory_policy=config.memory_policy or None,
+ )
+ return client, replayer
+
+
+def _print_stats(results: dict[str, BackfillStats], dry_run: bool) -> None:
+ label = "WOULD replay" if dry_run else "Replayed"
+ total = BackfillStats()
+ for name, s in results.items():
+ typer.echo(
+ f" {name:12s}: {s.sessions} sessions, {label.lower()} {s.messages} messages, "
+ f"{s.committed} committed, {s.skipped} skipped"
+ )
+ for err in s.errors:
+ typer.echo(f" ! {err}")
+ total.merge(s)
+ typer.echo(
+ f"{label}: {total.sessions} sessions / {total.messages} messages / "
+ f"{total.committed} commits" + (f" / {len(total.errors)} errors" if total.errors else "")
+ )
+
+
+# --------------------------------------------------------------------------- #
[email protected]("list-sources")
+def list_sources() -> None:
+ """List registered harness adapters and their current config."""
+ import openviking.ingest.sources # noqa: F401 - populate the registry
+
+ config = _load_ingest_config()
+ typer.echo(f"ingest enabled: {config.enabled} server_url: {config.server_url or '(default)'}")
+ typer.echo("harnesses:")
+ for name in sorted(SOURCE_REGISTRY):
+ hc = config.harnesses.get(name)
+ if hc is None:
+ typer.echo(f" {name:12s} (not configured)")
+ else:
+ typer.echo(
+ f" {name:12s} enabled={hc.enabled} mode={hc.mode} paths={hc.paths or '(default)'}"
+ )
+
+
+def status(
+ harness: Optional[List[str]] = typer.Option(None, "--harness", "-H", help="Filter by harness"),
+) -> None:
+ """Show per-session ingest progress recorded in the cursor store."""
+ config = _load_ingest_config()
+ store = _make_store(config)
+ try:
+ names = harness or None
+ records = []
+ for h in names or sorted({r.harness for r in store.all_records()}):
+ records.extend(store.all_records(h))
+ if not records:
+ typer.echo("No sessions ingested yet.")
+ return
+ typer.echo(f"{'harness':12s} {'session':24s} {'msgs':>6s} committed_at")
+ for rec in records:
+ typer.echo(
+ f"{rec.harness:12s} {rec.native_session_id[:24]:24s} "
+ f"{rec.last_appended_count:6d} {rec.last_committed_at or '-'}"
+ )
+ finally:
+ store.close()
+
+
+def backfill(
+ harness: Optional[List[str]] = typer.Option(
+ None, "--harness", "-H", help="Only these harnesses (default: all enabled)"
+ ),
+ since: Optional[str] = typer.Option(
+ None, "--since", help="Skip sessions started before this ISO date (e.g. 2026-06-01)"
+ ),
+ dry_run: bool = typer.Option(False, "--dry-run", help="Count sessions/messages; write nothing"),
+ reset: bool = typer.Option(
+ False, "--reset", help="Delete + recreate each OV session before replaying"
+ ),
+) -> None:
+ """One-shot backfill of existing logs."""
+ asyncio.run(_run_backfill(harness, since, dry_run, reset))
+
+
+async def _run_backfill(harness, since, dry_run, reset) -> None:
+ config = _load_ingest_config()
+ store = _make_store(config)
+ client = None
+ lock = None
+ try:
+ only = harness or None
+ if dry_run:
+ # No server (and no lock) needed for a read-only dry run.
+ replayer = SessionReplayer(None, store, session_id_prefix=config.session_id_prefix) # type: ignore[arg-type]
+ orch = IngestOrchestrator(config, replayer)
+ results = await orch.backfill(only=only, since=since, dry_run=True)
+ else:
+ lock = _acquire_lock(config)
+ client, replayer = await _make_replayer(config, store)
+ orch = IngestOrchestrator(config, replayer)
+ results = await orch.backfill(only=only, since=since, dry_run=False, reset=reset)
+ _print_stats(results, dry_run)
+ finally:
+ if client is not None:
+ await client.close()
+ if lock is not None:
+ lock.release()
+ store.close()
+
+
+def watch(
+ harness: Optional[List[str]] = typer.Option(
+ None, "--harness", "-H", help="Only these harnesses (default: all enabled)"
+ ),
+) -> None:
+ """Incrementally watch for new/changed logs and replay them."""
+ asyncio.run(_run_watch(harness))
+
+
+def run() -> None:
+ """Honor each harness's configured mode: backfill (mode in backfill/both) then watch."""
+ asyncio.run(_run_all())
+
+
+async def _run_all() -> None:
+ config = _load_ingest_config()
+ store = _make_store(config)
+ lock = _acquire_lock(config)
+ client, replayer = await _make_replayer(config, store)
+ try:
+ orch = IngestOrchestrator(config, replayer)
+ results = await orch.backfill(only=None, since=None, dry_run=False)
+ _print_stats(results, dry_run=False)
+ watch_sources = [
+ (n, c, s) for n, c, s in enabled_sources(config) if c.mode in ("watch", "both")
+ ]
+ if watch_sources:
+ await _watch_loop(replayer, watch_sources)
+ finally:
+ await client.close()
+ lock.release()
+ store.close()
+
+
+async def _run_watch(harness) -> None:
+ config = _load_ingest_config()
+ store = _make_store(config)
+ lock = _acquire_lock(config)
+ client, replayer = await _make_replayer(config, store)
+ try:
+ only = harness or None
+ sources = [
+ (n, c, s)
+ for n, c, s in enabled_sources(config, only)
+ if c.mode in ("watch", "both") or only is not None
+ ]
+ if not sources:
+ typer.echo("No harnesses enabled for watch.")
+ return
+ await _watch_loop(replayer, sources)
+ finally:
+ await client.close()
+ lock.release()
+ store.close()
+
+
+async def _watch_loop(replayer, sources) -> None:
+ poller = IngestPoller(sources, replayer)
+ loop = asyncio.get_running_loop()
+ for sig in (signal.SIGINT, signal.SIGTERM):
+ try:
+ loop.add_signal_handler(sig, poller.stop)
+ except (NotImplementedError, RuntimeError):
+ pass # not supported on this platform
+ await poller.run()
+
+
+def main() -> None:
+ app()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/openviking/ingest/cursor_store.py b/openviking/ingest/cursor_store.py
new file mode 100644
index 000000000..40862786b
--- /dev/null
+++ b/openviking/ingest/cursor_store.py
@@ -0,0 +1,365 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Durable per-(harness, session) read-cursor + commit/idempotency state.
+
+A single SQLite DB under ``~/.openviking/ingest/state.db`` records how far each
+conversation has been ingested, plus:
+- ``needs_commit``: appended-but-not-yet-committed, so a crash/error between append and
+ commit still gets its memory extraction on the next pass (not stranded forever);
+- a ``pending`` batch intent (cursor + count + server-message baseline) written BEFORE a
+ batch is appended, so a crash mid-append is reconciled against the server's message
+ count on restart instead of blindly re-appending (duplicate-safe).
+
+A ``SingleInstanceLock`` prevents two ingest processes from racing on the same state.
+
+(This is the read-position POINTER store; unrelated to the Cursor IDE harness.)
+"""
+
+from __future__ import annotations
+
+import os
+import sqlite3
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Optional
+
+from openviking.ingest.models import Cursor
+from openviking_cli.utils import get_logger
+
+logger = get_logger(__name__)
+
+_SCHEMA = """
+CREATE TABLE IF NOT EXISTS ingest_cursor (
+ harness TEXT NOT NULL,
+ native_session_id TEXT NOT NULL,
+ ov_session_id TEXT,
+ cursor_kind TEXT NOT NULL,
+ cursor_value TEXT NOT NULL,
+ locator TEXT,
+ title TEXT,
+ last_appended_count INTEGER NOT NULL DEFAULT 0,
+ pending_tokens INTEGER NOT NULL DEFAULT 0,
+ needs_commit INTEGER NOT NULL DEFAULT 0,
+ pend_cursor TEXT,
+ pend_count INTEGER NOT NULL DEFAULT 0,
+ pend_baseline INTEGER NOT NULL DEFAULT 0,
+ last_committed_at TEXT,
+ updated_at TEXT,
+ PRIMARY KEY (harness, native_session_id)
+);
+"""
+
+_MIGRATION_COLUMNS = {
+ "needs_commit": "INTEGER NOT NULL DEFAULT 0",
+ "pend_cursor": "TEXT",
+ "pend_count": "INTEGER NOT NULL DEFAULT 0",
+ "pend_baseline": "INTEGER NOT NULL DEFAULT 0",
+}
+
+
+@dataclass
+class CursorRecord:
+ harness: str
+ native_session_id: str
+ ov_session_id: Optional[str]
+ cursor: Cursor
+ locator: Optional[str]
+ title: Optional[str]
+ last_appended_count: int
+ pending_tokens: int
+ needs_commit: bool
+ pending_cursor: Optional[Cursor]
+ pending_count: int
+ pending_baseline: int
+ last_committed_at: Optional[str]
+
+
+def _now_iso() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+class CursorStore:
+ def __init__(self, state_dir: Path):
+ self.state_dir = Path(state_dir).expanduser()
+ self.state_dir.mkdir(parents=True, exist_ok=True)
+ self.db_path = self.state_dir / "state.db"
+ self._conn = sqlite3.connect(str(self.db_path))
+ self._conn.row_factory = sqlite3.Row
+ self._conn.execute("PRAGMA journal_mode=WAL")
+ self._conn.executescript(_SCHEMA)
+ self._migrate()
+ self._conn.commit()
+
+ def _migrate(self) -> None:
+ existing = {r["name"] for r in self._conn.execute("PRAGMA table_info(ingest_cursor)")}
+ for col, decl in _MIGRATION_COLUMNS.items():
+ if col not in existing:
+ self._conn.execute(f"ALTER TABLE ingest_cursor ADD COLUMN {col} {decl}")
+
+ def close(self) -> None:
+ self._conn.close()
+
+ # --- reads ------------------------------------------------------------
+ def get(self, harness: str, native_session_id: str) -> Optional[CursorRecord]:
+ row = self._conn.execute(
+ "SELECT * FROM ingest_cursor WHERE harness = ? AND native_session_id = ?",
+ (harness, native_session_id),
+ ).fetchone()
+ if row is None:
+ return None
+ pend = (
+ Cursor.from_json(row["cursor_kind"], row["pend_cursor"]) if row["pend_cursor"] else None
+ )
+ return CursorRecord(
+ harness=row["harness"],
+ native_session_id=row["native_session_id"],
+ ov_session_id=row["ov_session_id"],
+ cursor=Cursor.from_json(row["cursor_kind"], row["cursor_value"]),
+ locator=row["locator"],
+ title=row["title"],
+ last_appended_count=row["last_appended_count"],
+ pending_tokens=row["pending_tokens"],
+ needs_commit=bool(row["needs_commit"]),
+ pending_cursor=pend,
+ pending_count=row["pend_count"],
+ pending_baseline=row["pend_baseline"],
+ last_committed_at=row["last_committed_at"],
+ )
+
+ def get_cursor(self, harness: str, native_session_id: str, kind: str) -> Optional[Cursor]:
+ rec = self.get(harness, native_session_id)
+ return rec.cursor if rec else None
+
+ # --- writes -----------------------------------------------------------
+ def ensure_row(
+ self,
+ harness: str,
+ native_session_id: str,
+ ov_session_id: str,
+ cursor: Cursor,
+ *,
+ locator: Optional[str] = None,
+ title: Optional[str] = None,
+ ) -> None:
+ """Insert a zero/initial row if absent (idempotent)."""
+ self._conn.execute(
+ """INSERT OR IGNORE INTO ingest_cursor
+ (harness, native_session_id, ov_session_id, cursor_kind, cursor_value,
+ locator, title, updated_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?)""",
+ (
+ harness,
+ native_session_id,
+ ov_session_id,
+ cursor.kind,
+ cursor.to_json(),
+ locator,
+ title,
+ _now_iso(),
+ ),
+ )
+ self._conn.commit()
+
+ def advance_cursor(
+ self,
+ harness: str,
+ native_session_id: str,
+ ov_session_id: str,
+ cursor: Cursor,
+ *,
+ locator: Optional[str] = None,
+ ) -> None:
+ """Advance the confirmed cursor only (e.g. a read that yielded no messages)."""
+ self.ensure_row(harness, native_session_id, ov_session_id, cursor, locator=locator)
+ self._conn.execute(
+ "UPDATE ingest_cursor SET cursor_value = ?, cursor_kind = ?, "
+ "locator = COALESCE(?, locator), updated_at = ? "
+ "WHERE harness = ? AND native_session_id = ?",
+ (cursor.to_json(), cursor.kind, locator, _now_iso(), harness, native_session_id),
+ )
+ self._conn.commit()
+
+ def set_pending(
+ self,
+ harness: str,
+ native_session_id: str,
+ ov_session_id: str,
+ from_cursor: Cursor,
+ pend_cursor: Cursor,
+ pend_count: int,
+ baseline: int,
+ *,
+ locator: Optional[str] = None,
+ title: Optional[str] = None,
+ ) -> None:
+ """Durably record a batch intent BEFORE appending it (crash reconciliation)."""
+ self.ensure_row(
+ harness, native_session_id, ov_session_id, from_cursor, locator=locator, title=title
+ )
+ self._conn.execute(
+ "UPDATE ingest_cursor SET ov_session_id = ?, pend_cursor = ?, pend_count = ?, "
+ "pend_baseline = ?, locator = COALESCE(?, locator), title = COALESCE(?, title), "
+ "updated_at = ? WHERE harness = ? AND native_session_id = ?",
+ (
+ ov_session_id,
+ pend_cursor.to_json(),
+ pend_count,
+ baseline,
+ locator,
+ title,
+ _now_iso(),
+ harness,
+ native_session_id,
+ ),
+ )
+ self._conn.commit()
+
+ def confirm_append(
+ self, harness: str, native_session_id: str, cursor: Cursor, appended_delta: int
+ ) -> None:
+ """Confirm a landed batch: advance cursor, accrue count, mark needs_commit, clear pending."""
+ self._conn.execute(
+ "UPDATE ingest_cursor SET cursor_value = ?, cursor_kind = ?, "
+ "last_appended_count = last_appended_count + ?, needs_commit = 1, "
+ "pend_cursor = NULL, pend_count = 0, pend_baseline = 0, updated_at = ? "
+ "WHERE harness = ? AND native_session_id = ?",
+ (cursor.to_json(), cursor.kind, appended_delta, _now_iso(), harness, native_session_id),
+ )
+ self._conn.commit()
+
+ def clear_pending(self, harness: str, native_session_id: str) -> None:
+ self._conn.execute(
+ "UPDATE ingest_cursor SET pend_cursor = NULL, pend_count = 0, pend_baseline = 0, "
+ "updated_at = ? WHERE harness = ? AND native_session_id = ?",
+ (_now_iso(), harness, native_session_id),
+ )
+ self._conn.commit()
+
+ def mark_committed(
+ self, harness: str, native_session_id: str, *, pending_tokens: int = 0
+ ) -> None:
+ self._conn.execute(
+ "UPDATE ingest_cursor SET needs_commit = 0, pending_tokens = ?, "
+ "last_committed_at = ?, updated_at = ? WHERE harness = ? AND native_session_id = ?",
+ (pending_tokens, _now_iso(), _now_iso(), harness, native_session_id),
+ )
+ self._conn.commit()
+
+ def upsert(
+ self,
+ harness: str,
+ native_session_id: str,
+ ov_session_id: str,
+ cursor: Cursor,
+ *,
+ appended_delta: int = 0,
+ pending_tokens: Optional[int] = None,
+ committed: bool = False,
+ locator: Optional[str] = None,
+ title: Optional[str] = None,
+ ) -> None:
+ """General cursor upsert (used by tests and simple cursor advances)."""
+ now = _now_iso()
+ committed_at = now if committed else None
+ self._conn.execute(
+ """INSERT INTO ingest_cursor
+ (harness, native_session_id, ov_session_id, cursor_kind, cursor_value,
+ locator, title, last_appended_count, pending_tokens, last_committed_at, updated_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+ ON CONFLICT(harness, native_session_id) DO UPDATE SET
+ ov_session_id = excluded.ov_session_id,
+ cursor_kind = excluded.cursor_kind, cursor_value = excluded.cursor_value,
+ locator = COALESCE(excluded.locator, ingest_cursor.locator),
+ title = COALESCE(excluded.title, ingest_cursor.title),
+ last_appended_count = ingest_cursor.last_appended_count + ?,
+ pending_tokens = COALESCE(?, ingest_cursor.pending_tokens),
+ last_committed_at = COALESCE(?, ingest_cursor.last_committed_at),
+ updated_at = excluded.updated_at""",
+ (
+ harness,
+ native_session_id,
+ ov_session_id,
+ cursor.kind,
+ cursor.to_json(),
+ locator,
+ title,
+ appended_delta,
+ pending_tokens if pending_tokens is not None else 0,
+ committed_at,
+ now,
+ appended_delta,
+ pending_tokens,
+ committed_at,
+ ),
+ )
+ self._conn.commit()
+
+ def delete(self, harness: str, native_session_id: str) -> None:
+ self._conn.execute(
+ "DELETE FROM ingest_cursor WHERE harness = ? AND native_session_id = ?",
+ (harness, native_session_id),
+ )
+ self._conn.commit()
+
+ def all_records(self, harness: Optional[str] = None) -> list[CursorRecord]:
+ if harness:
+ rows = self._conn.execute(
+ "SELECT harness, native_session_id FROM ingest_cursor WHERE harness = ?",
+ (harness,),
+ ).fetchall()
+ else:
+ rows = self._conn.execute(
+ "SELECT harness, native_session_id FROM ingest_cursor"
+ ).fetchall()
+ out = []
+ for r in rows:
+ rec = self.get(r["harness"], r["native_session_id"])
+ if rec:
+ out.append(rec)
+ return out
+
+
+class SingleInstanceLock:
+ """Best-effort exclusive lock so two ingest processes don't race on one state dir."""
+
+ def __init__(self, state_dir: Path):
+ self.path = Path(state_dir).expanduser() / "ingest.lock"
+ self._fh = None
+
+ def acquire(self) -> "SingleInstanceLock":
+ try:
+ import fcntl
+ except ImportError: # pragma: no cover - non-POSIX
+ logger.warning("[ingest] fcntl unavailable; single-instance lock disabled")
+ return self
+ self.path.parent.mkdir(parents=True, exist_ok=True)
+ self._fh = open(self.path, "w")
+ try:
+ fcntl.flock(self._fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ except OSError as exc:
+ self._fh.close()
+ self._fh = None
+ raise RuntimeError(f"another ingest process already holds {self.path}") from exc
+ self._fh.write(str(os.getpid()))
+ self._fh.flush()
+ return self
+
+ def release(self) -> None:
+ if self._fh is None:
+ return
+ try:
+ import fcntl
+
+ fcntl.flock(self._fh, fcntl.LOCK_UN)
+ except (ImportError, OSError):
+ pass
+ finally:
+ self._fh.close()
+ self._fh = None
+
+ def __enter__(self) -> "SingleInstanceLock":
+ return self.acquire()
+
+ def __exit__(self, *exc) -> None:
+ self.release()
diff --git a/openviking/ingest/models.py b/openviking/ingest/models.py
new file mode 100644
index 000000000..0d1b41f05
--- /dev/null
+++ b/openviking/ingest/models.py
@@ -0,0 +1,95 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Core data structures shared across the ingest subsystem.
+
+Note on "cursor": throughout this package, ``Cursor`` / ``cursor_store`` / ``cursor_kind``
+refer to the read-position POINTER (how far we have ingested a given log), NOT the Cursor
+IDE harness (whose adapter is ``sources/cursor.py``).
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any, Dict, List, Optional
+
+
+def iso_from_epoch_ms(value: Any) -> Optional[str]:
+ """Best-effort ISO-8601 from an epoch-millisecond integer (used by SQLite/JSONL adapters)."""
+ try:
+ ms = int(value)
+ except (TypeError, ValueError):
+ return None
+ if ms <= 0:
+ return None
+ # Heuristic: 10-digit values are seconds, 13-digit are milliseconds.
+ seconds = ms / 1000.0 if ms > 10_000_000_000 else float(ms)
+ try:
+ return datetime.fromtimestamp(seconds, tz=timezone.utc).isoformat()
+ except (OverflowError, OSError, ValueError):
+ return None
+
+
+# Cursor kinds.
+BYTE_OFFSET = "byte_offset" # append-only JSONL: byte offset of last consumed line
+ROWID_TIME = "rowid_time" # SQLite: (time_created, id) of last consumed row
+
+
+@dataclass
+class NormalizedMessage:
+ """A single conversation turn, harness-agnostic, ready for OV replay.
+
+ ``peer_id`` is resolved by the adapter (see ``peer.py``): assistant turns ->
+ ``{harness}/{model}``; user turns -> a human identifier (git identity for
+ single-user dev harnesses, original username for group-chat harnesses).
+ """
+
+ role: str # "user" | "assistant"
+ text: str = ""
+ parts: List[Dict[str, Any]] = field(default_factory=list) # extra tool/context parts
+ created_at: Optional[str] = None # ISO-8601
+ peer_id: Optional[str] = None
+ meta: Dict[str, Any] = field(default_factory=dict) # model, provider, cwd, …
+
+
+@dataclass
+class SessionRef:
+ """A discoverable conversation in a harness's storage."""
+
+ harness: str # registry name, e.g. "claude_code"
+ native_session_id: str # the harness's own session/conversation id
+ locator: str # file path (JSONL) or db session id (SQLite) used to read
+ title: Optional[str] = None
+ started_at: Optional[str] = None
+ meta: Dict[str, Any] = field(default_factory=dict) # session-level model, cwd, platform
+
+
+@dataclass
+class Cursor:
+ """A durable read-position pointer for one (harness, session)."""
+
+ kind: str # BYTE_OFFSET | ROWID_TIME
+ value: Dict[str, Any] # {"offset": int} | {"time": int|float, "id": str}
+
+ @classmethod
+ def zero(cls, kind: str) -> "Cursor":
+ if kind == BYTE_OFFSET:
+ return cls(kind=kind, value={"offset": 0})
+ return cls(kind=kind, value={"time": 0, "id": ""})
+
+ @property
+ def offset(self) -> int:
+ return int(self.value.get("offset", 0))
+
+ def to_json(self) -> str:
+ return json.dumps(self.value, separators=(",", ":"))
+
+ @classmethod
+ def from_json(cls, kind: str, raw: Optional[str]) -> "Cursor":
+ if not raw:
+ return cls.zero(kind)
+ try:
+ return cls(kind=kind, value=json.loads(raw))
+ except (ValueError, TypeError):
+ return cls.zero(kind)
diff --git a/openviking/ingest/normalize.py b/openviking/ingest/normalize.py
new file mode 100644
index 000000000..0792db8be
--- /dev/null
+++ b/openviking/ingest/normalize.py
@@ -0,0 +1,52 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Convert ``NormalizedMessage`` -> OV ``AddMessageRequest`` payload dict.
+
+Mirrors vikingbot's ``_normalize_session_messages`` shape (text/tool parts, peer_id
+safe-naming). Conversation memory is driven by user/assistant TEXT; tool I/O is treated
+as low-value and dropped by the adapters, so most messages carry a single text part.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Optional
+
+from openviking.core.peer_id import safe_peer_id
+from openviking.ingest.models import NormalizedMessage
+
+
+def to_add_message_request(msg: NormalizedMessage) -> Optional[Dict[str, Any]]:
+ """Return an ``AddMessageRequest`` dict, or ``None`` for an empty turn."""
+ role = "user" if msg.role == "user" else "assistant"
+
+ parts: List[Dict[str, Any]] = []
+ text = (msg.text or "").strip()
+ if text:
+ parts.append({"type": "text", "text": text})
+ if msg.parts:
+ parts.extend(msg.parts)
+
+ if not parts:
+ return None # nothing worth replaying
+
+ payload: Dict[str, Any] = {"role": role, "parts": parts}
+ if text:
+ payload["content"] = text
+ if msg.created_at:
+ payload["created_at"] = msg.created_at
+
+ pid = safe_peer_id(msg.peer_id)
+ if pid:
+ payload["peer_id"] = pid
+
+ return payload
+
+
+def to_add_message_requests(messages: List[NormalizedMessage]) -> List[Dict[str, Any]]:
+ """Batch convert, dropping empty turns."""
+ out: List[Dict[str, Any]] = []
+ for m in messages:
+ payload = to_add_message_request(m)
+ if payload is not None:
+ out.append(payload)
+ return out
diff --git a/openviking/ingest/orchestrator.py b/openviking/ingest/orchestrator.py
new file mode 100644
index 000000000..68caeae7f
--- /dev/null
+++ b/openviking/ingest/orchestrator.py
@@ -0,0 +1,143 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Backfill orchestration: replay each discovered session cursor->end, then commit.
+
+Incremental watch mode is handled by ``IngestPoller`` (``poller.py``).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Dict, List, Optional, Tuple
+
+from openviking.ingest.models import Cursor
+from openviking.ingest.registry import iter_enabled_sources
+from openviking.ingest.replay import SessionReplayer
+from openviking.ingest.sources.base import DEFAULT_READ_LIMIT, LogSource, NotSupportedError
+from openviking_cli.utils import get_logger
+from openviking_cli.utils.config.ingest_config import IngestConfig, IngestHarnessConfig
+
+logger = get_logger(__name__)
+
+
+@dataclass
+class BackfillStats:
+ sessions: int = 0
+ messages: int = 0
+ committed: int = 0
+ skipped: int = 0
+ errors: List[str] = field(default_factory=list)
+
+ def merge(self, other: "BackfillStats") -> None:
+ self.sessions += other.sessions
+ self.messages += other.messages
+ self.committed += other.committed
+ self.skipped += other.skipped
+ self.errors.extend(other.errors)
+
+
+def enabled_sources(
+ config: IngestConfig, only: Optional[List[str]] = None
+) -> List[Tuple[str, IngestHarnessConfig, LogSource]]:
+ """Construct enabled sources, optionally filtered to ``only`` harness names."""
+ out = []
+ for name, cfg, source in iter_enabled_sources(config):
+ if only and name not in only:
+ continue
+ out.append((name, cfg, source))
+ return out
+
+
+class IngestOrchestrator:
+ def __init__(self, config: IngestConfig, replayer: SessionReplayer):
+ self.config = config
+ self.replayer = replayer
+ self.store = replayer.store
+
+ async def backfill_source(
+ self,
+ name: str,
+ harness_cfg: IngestHarnessConfig,
+ source: LogSource,
+ *,
+ since: Optional[str] = None,
+ dry_run: bool = False,
+ reset: bool = False,
+ ) -> BackfillStats:
+ stats = BackfillStats()
+ try:
+ refs = list(source.discover_sessions())
+ except NotSupportedError as exc:
+ logger.warning("[ingest:%s] %s", name, exc)
+ stats.errors.append(f"{name}: {exc}")
+ return stats
+ except Exception as exc: # noqa: BLE001
+ logger.exception("[ingest:%s] discovery failed", name)
+ stats.errors.append(f"{name}: discovery failed: {exc}")
+ return stats
+
+ for ref in refs:
+ if since and ref.started_at and ref.started_at < since:
+ stats.skipped += 1
+ continue
+ try:
+ if not dry_run:
+ if reset:
+ await self.replayer.reset_session(name, ref)
+ else:
+ await self.replayer.reconcile(name, ref)
+ added = await self._backfill_one(name, source, ref, dry_run=dry_run)
+ stats.sessions += 1
+ stats.messages += added
+ if not dry_run:
+ # Commit even when added == 0: a prior crash may have left appended
+ # messages un-committed (needs_commit / server pending tokens).
+ if await self.replayer.commit_if_needed(
+ name, ref, harness_cfg.commit.keep_recent_count
+ ):
+ stats.committed += 1
+ except Exception as exc: # noqa: BLE001
+ logger.exception("[ingest:%s] session %s failed", name, ref.native_session_id)
+ stats.errors.append(f"{name}/{ref.native_session_id}: {exc}")
+ return stats
+
+ async def _backfill_one(self, name, source, ref, *, dry_run: bool) -> int:
+ cursor = self.store.get_cursor(name, ref.native_session_id, source.cursor_kind)
+ total = 0
+ while True:
+ messages, new_cursor = source.read_messages(ref, cursor, DEFAULT_READ_LIMIT)
+ if not messages:
+ from_value = (cursor or Cursor.zero(source.cursor_kind)).value
+ if not dry_run and new_cursor.value != from_value:
+ sid = self.replayer.ov_session_id(name, ref.native_session_id)
+ self.store.advance_cursor(
+ name, ref.native_session_id, sid, new_cursor, locator=ref.locator
+ )
+ break
+ if dry_run:
+ total += sum(1 for m in messages if (m.text or "").strip() or m.parts)
+ else:
+ from_cursor = cursor or Cursor.zero(source.cursor_kind)
+ total += await self.replayer.append_batch(
+ name, ref, messages, from_cursor, new_cursor
+ )
+ cursor = new_cursor
+ return total
+
+ async def backfill(
+ self,
+ only: Optional[List[str]] = None,
+ *,
+ since: Optional[str] = None,
+ dry_run: bool = False,
+ reset: bool = False,
+ ) -> Dict[str, BackfillStats]:
+ results: Dict[str, BackfillStats] = {}
+ for name, cfg, source in enabled_sources(self.config, only):
+ if cfg.mode not in ("backfill", "both") and only is None:
+ continue
+ logger.info("[ingest] backfilling %s (dry_run=%s)", name, dry_run)
+ results[name] = await self.backfill_source(
+ name, cfg, source, since=since, dry_run=dry_run, reset=reset
+ )
+ return results
diff --git a/openviking/ingest/peer.py b/openviking/ingest/peer.py
new file mode 100644
index 000000000..00b531782
--- /dev/null
+++ b/openviking/ingest/peer.py
@@ -0,0 +1,95 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""peer_id resolution for replayed turns.
+
+- assistant turns -> ``{harness}__{model}`` (or ``{harness}__{provider}__{model}``)
+- user turns:
+ * single-user dev harnesses (claude_code/codex/opencode) -> git identity of the
+ session cwd repo, falling back to the configured OV user;
+ * group-chat harnesses (hermes/openclaw) -> the original username from the log.
+
+peer_id must match OpenViking's identifier rules (``[a-zA-Z0-9_.@-]+``). ASCII values
+stay human-readable; anything else (e.g. CJK usernames) is base64-encoded as ``ext-…``,
+mirroring vikingbot's external-peer convention.
+"""
+
+from __future__ import annotations
+
+import base64
+import re
+import subprocess
+from typing import Dict, Optional
+
+from openviking.core.peer_id import safe_peer_id
+
+# Join harness/provider/model with the same separator as the OV session id scheme.
+SEP = "__"
+
+_ALLOWED = re.compile(r"[^a-zA-Z0-9_.@-]+")
+_GIT_PEER_CACHE: Dict[str, Optional[str]] = {}
+
+
+def _sanitize_component(value: str) -> str:
+ """Make one path-free component safe & readable (lossy for non-ASCII)."""
+ cleaned = _ALLOWED.sub("-", (value or "").strip())
+ cleaned = re.sub(r"-{2,}", "-", cleaned).strip("-.")
+ return cleaned
+
+
+def safe_external_peer(raw: Optional[str]) -> Optional[str]:
+ """Return a valid peer_id for an arbitrary external identifier (readable if ASCII)."""
+ if not raw:
+ return None
+ text = str(raw).strip()
+ if not text:
+ return None
+ sanitized = _sanitize_component(text)
+ if sanitized:
+ pid = safe_peer_id(sanitized)
+ if pid:
+ return pid
+ # Non-ASCII / unsanitizable -> stable, valid, unique fallback.
+ encoded = base64.urlsafe_b64encode(text.encode("utf-8")).decode("ascii").rstrip("=")
+ return safe_peer_id(f"ext-{encoded}")
+
+
+def assistant_peer_id(
+ harness: str, model: Optional[str], provider: Optional[str] = None
+) -> Optional[str]:
+ """`{harness}__{model}` or `{harness}__{provider}__{model}`."""
+ parts = [_sanitize_component(harness)]
+ if provider:
+ parts.append(_sanitize_component(provider))
+ parts.append(_sanitize_component(model) if model else "unknown")
+ return safe_peer_id(SEP.join(p for p in parts if p))
+
+
+def resolve_git_human_peer(cwd: Optional[str], fallback: str) -> Optional[str]:
+ """Human peer_id from the cwd repo's git identity; fall back to ``fallback``.
+
+ Prefers ``user.email`` (more unique), then ``user.name``. Cached per cwd.
+ """
+ if not cwd:
+ return safe_external_peer(fallback)
+ if cwd in _GIT_PEER_CACHE:
+ cached = _GIT_PEER_CACHE[cwd]
+ return cached if cached else safe_external_peer(fallback)
+
+ identity: Optional[str] = None
+ for key in ("user.email", "user.name"):
+ try:
+ out = subprocess.run(
+ ["git", "-C", cwd, "config", "--get", key],
+ capture_output=True,
+ text=True,
+ timeout=2.0,
+ )
+ except (OSError, subprocess.SubprocessError):
+ break
+ if out.returncode == 0 and out.stdout.strip():
+ identity = out.stdout.strip()
+ break
+
+ resolved = safe_external_peer(identity) if identity else None
+ _GIT_PEER_CACHE[cwd] = resolved
+ return resolved if resolved else safe_external_peer(fallback)
diff --git a/openviking/ingest/poller.py b/openviking/ingest/poller.py
new file mode 100644
index 000000000..03de08f65
--- /dev/null
+++ b/openviking/ingest/poller.py
@@ -0,0 +1,148 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Incremental ingest (watch mode): a WatchScheduler-style asyncio poll loop.
+
+Mirrors ``openviking/resource/watch_scheduler.py`` (interval polling, graceful start/stop)
+rather than depending on filesystem events: a durable read-cursor makes polling correct
+and self-healing (a missed tick / sleep / restart just reads cursor->EOF next time).
+
+Each tick, per enabled harness: rescan sessions -> incremental ``read_messages`` from the
+stored cursor -> append + advance cursor -> commit on idle or token threshold.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import time
+from typing import Dict, List, Tuple
+
+from openviking.ingest.models import Cursor
+from openviking.ingest.replay import SessionReplayer
+from openviking.ingest.sources.base import DEFAULT_READ_LIMIT, LogSource, NotSupportedError
+from openviking_cli.utils import get_logger
+from openviking_cli.utils.config.ingest_config import IngestHarnessConfig
+
+logger = get_logger(__name__)
+
+
+class _Dirty:
+ __slots__ = ("name", "cfg", "source", "ref", "last_activity")
+
+ def __init__(self, name, cfg, source, ref, last_activity):
+ self.name = name
+ self.cfg = cfg
+ self.source = source
+ self.ref = ref
+ self.last_activity = last_activity
+
+
+class IngestPoller:
+ def __init__(
+ self,
+ sources: List[Tuple[str, IngestHarnessConfig, LogSource]],
+ replayer: SessionReplayer,
+ ):
+ self._sources = sources
+ self.replayer = replayer
+ self.store = replayer.store
+ self._running = False
+ self._stop = asyncio.Event()
+ self._dirty: Dict[Tuple[str, str], _Dirty] = {}
+ self.poll_interval = min((cfg.poll_interval_seconds for _, cfg, _ in sources), default=5.0)
+
+ def stop(self) -> None:
+ self._running = False
+ self._stop.set()
+
+ async def run(self) -> None:
+ if not self._sources:
+ logger.info("[ingest] no harnesses enabled for watch; nothing to do")
+ return
+ self._running = True
+ logger.info(
+ "[ingest] watch loop started (poll=%.1fs, harnesses=%s)",
+ self.poll_interval,
+ [n for n, _, _ in self._sources],
+ )
+ try:
+ while self._running:
+ await self._tick()
+ await self._commit_idle()
+ try:
+ await asyncio.wait_for(self._stop.wait(), timeout=self.poll_interval)
+ except asyncio.TimeoutError:
+ pass
+ finally:
+ await self._flush_all()
+ logger.info("[ingest] watch loop stopped")
+
+ async def _tick(self) -> None:
+ for name, cfg, source in self._sources:
+ try:
+ refs = list(source.discover_sessions())
+ except NotSupportedError as exc:
+ logger.warning("[ingest:%s] %s", name, exc)
+ continue
+ except Exception: # noqa: BLE001
+ logger.exception("[ingest:%s] discovery failed", name)
+ continue
+ for ref in refs:
+ await self._poll_session(name, cfg, source, ref)
+
+ async def _poll_session(self, name, cfg, source, ref) -> None:
+ await self.replayer.reconcile(name, ref)
+ appended = False
+ while True:
+ cursor = self.store.get_cursor(name, ref.native_session_id, source.cursor_kind)
+ try:
+ messages, new_cursor = source.read_messages(ref, cursor, DEFAULT_READ_LIMIT)
+ except Exception: # noqa: BLE001
+ logger.exception("[ingest:%s] read failed for %s", name, ref.native_session_id)
+ return
+ if not messages:
+ from_value = (cursor or Cursor.zero(source.cursor_kind)).value
+ if new_cursor.value != from_value:
+ sid = self.replayer.ov_session_id(name, ref.native_session_id)
+ self.store.advance_cursor(
+ name, ref.native_session_id, sid, new_cursor, locator=ref.locator
+ )
+ break
+ from_cursor = cursor or Cursor.zero(source.cursor_kind)
+ added = await self.replayer.append_batch(name, ref, messages, from_cursor, new_cursor)
+ if added > 0:
+ appended = True
+
+ rec = self.store.get(name, ref.native_session_id)
+ if appended or (rec and rec.needs_commit):
+ self._dirty[(name, ref.native_session_id)] = _Dirty(
+ name, cfg, source, ref, time.monotonic()
+ )
+ committed = await self.replayer.maybe_commit_on_threshold(
+ name, ref, cfg.commit.commit_token_threshold, cfg.commit.keep_recent_count
+ )
+ if committed:
+ self._dirty.pop((name, ref.native_session_id), None)
+
+ async def _commit_idle(self) -> None:
+ now = time.monotonic()
+ for key, d in list(self._dirty.items()):
+ if now - d.last_activity >= d.cfg.commit.commit_idle_seconds:
+ try:
+ await self.replayer.commit_if_needed(
+ d.name, d.ref, d.cfg.commit.keep_recent_count
+ )
+ self._dirty.pop(key, None) # only drop on success
+ except Exception: # noqa: BLE001 - keep dirty and retry after a backoff
+ logger.exception("[ingest:%s] idle commit failed; will retry", d.name)
+ d.last_activity = time.monotonic()
+
+ async def _flush_all(self) -> None:
+ """Final commit for every dirty session on shutdown (needs_commit persists if it fails)."""
+ for key, d in list(self._dirty.items()):
+ try:
+ await self.replayer.commit_if_needed(d.name, d.ref, d.cfg.commit.keep_recent_count)
+ except Exception: # noqa: BLE001
+ logger.exception(
+ "[ingest:%s] shutdown commit failed (needs_commit persisted)", d.name
+ )
+ self._dirty.pop(key, None)
diff --git a/openviking/ingest/registry.py b/openviking/ingest/registry.py
new file mode 100644
index 000000000..2b80cf77e
--- /dev/null
+++ b/openviking/ingest/registry.py
@@ -0,0 +1,55 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Registry of harness log-source adapters.
+
+Adding a harness = one ``@register_source("name")`` decorator on a ``LogSource``
+subclass; no config-schema change is needed (config keys are free-form).
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Dict, Iterator, Tuple, Type
+
+from openviking_cli.utils import get_logger
+
+if TYPE_CHECKING:
+ from openviking.ingest.sources.base import LogSource
+ from openviking_cli.utils.config.ingest_config import IngestConfig, IngestHarnessConfig
+
+logger = get_logger(__name__)
+
+SOURCE_REGISTRY: Dict[str, "Type[LogSource]"] = {}
+
+
+def register_source(name: str):
+ """Class decorator registering a ``LogSource`` subclass under ``name``."""
+
+ def deco(cls: "Type[LogSource]") -> "Type[LogSource]":
+ cls.name = name
+ if name in SOURCE_REGISTRY and SOURCE_REGISTRY[name] is not cls:
+ logger.warning("[ingest] source %r already registered; overriding", name)
+ SOURCE_REGISTRY[name] = cls
+ return cls
+
+ return deco
+
+
+def get_source_class(name: str) -> "Type[LogSource] | None":
+ # Ensure built-in adapters have been imported (and thus registered).
+ import openviking.ingest.sources # noqa: F401
+
+ return SOURCE_REGISTRY.get(name)
+
+
+def iter_enabled_sources(
+ config: "IngestConfig",
+) -> Iterator[Tuple[str, "IngestHarnessConfig", "LogSource"]]:
+ """Yield ``(name, harness_cfg, source)`` for each enabled, registered harness."""
+ import openviking.ingest.sources # noqa: F401 (populates SOURCE_REGISTRY)
+
+ for name, harness_cfg in config.enabled_harnesses().items():
+ cls = SOURCE_REGISTRY.get(name)
+ if cls is None:
+ logger.warning("[ingest] no adapter registered for harness %r; skipping", name)
+ continue
+ yield name, harness_cfg, cls(harness_cfg, fallback_user=config.user)
diff --git a/openviking/ingest/replay.py b/openviking/ingest/replay.py
new file mode 100644
index 000000000..2e4a1ca0c
--- /dev/null
+++ b/openviking/ingest/replay.py
@@ -0,0 +1,212 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Replay normalized messages into OpenViking via the SDK HTTP client.
+
+``ConversationReplayClient`` is a thin, vikingbot-free wrapper over ``ov.AsyncHTTPClient``
+(client-side, transport-agnostic: targets a local or remote server). ``SessionReplayer``
+drives the canonical recipe in idempotent batches:
+
+ reconcile() -> for each <=100-message batch: set_pending -> append -> confirm -> commit_if_needed
+
+Each batch's intent (target cursor + size + the server's message count *before* the
+append) is persisted BEFORE the append. If the process crashes mid-append, the next run
+``reconcile()`` compares the server's current message count against that baseline to decide
+whether the batch landed — confirming it (no re-append) or dropping the intent (re-read
+from the confirmed cursor). The cursor only advances on a confirmed append, and
+``needs_commit`` ensures appended-but-uncommitted sessions still get extracted later.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Any, Dict, List, Optional
+
+import openviking as ov
+from openviking.ingest.cursor_store import CursorStore
+from openviking.ingest.models import Cursor, NormalizedMessage, SessionRef
+from openviking.ingest.normalize import to_add_message_requests
+from openviking_cli.exceptions import NotFoundError
+from openviking_cli.utils import get_logger
+
+logger = get_logger(__name__)
+
+_BATCH = 100 # server-side cap for batch_add_messages
+_SID_ALLOWED = re.compile(r"[^a-zA-Z0-9_.@-]+")
+
+
+def _sanitize_id(value: str) -> str:
+ cleaned = _SID_ALLOWED.sub("-", (value or "").strip())
+ return re.sub(r"-{2,}", "-", cleaned).strip("-.") or "unknown"
+
+
+class ConversationReplayClient:
+ """Minimal session-recording surface over the OV SDK HTTP client."""
+
+ def __init__(self, client: "ov.AsyncHTTPClient"):
+ self._client = client
+
+ @classmethod
+ async def create(
+ cls, *, url: str = "", api_key: str = "", account: str = "", user: str = ""
+ ) -> "ConversationReplayClient":
+ client = ov.AsyncHTTPClient(
+ url=url or None,
+ api_key=api_key or None,
+ account=account or None,
+ user=user or None,
+ )
+ await client.initialize()
+ return cls(client)
+
+ async def close(self) -> None:
+ await self._client.close()
+
+ async def ensure_session(
+ self, ov_session_id: str, memory_policy: Optional[Dict[str, Any]] = None
+ ) -> Dict[str, Any]:
+ try:
+ return await self._client.get_session(ov_session_id)
+ except NotFoundError:
+ return await self._client.create_session(
+ session_id=ov_session_id, memory_policy=memory_policy or None
+ )
+
+ async def append(self, ov_session_id: str, messages: List[Dict[str, Any]]) -> int:
+ total = 0
+ for start in range(0, len(messages), _BATCH):
+ chunk = messages[start : start + _BATCH]
+ result = await self._client.batch_add_messages(ov_session_id, chunk)
+ total += int(result.get("added", len(chunk)) or 0)
+ return total
+
+ async def commit(self, ov_session_id: str, keep_recent_count: int = 0) -> Dict[str, Any]:
+ return await self._client.commit_session(ov_session_id, keep_recent_count=keep_recent_count)
+
+ async def _session_info(self, ov_session_id: str) -> Dict[str, Any]:
+ try:
+ return await self._client.get_session(ov_session_id)
+ except NotFoundError:
+ return {}
+
+ async def pending_tokens(self, ov_session_id: str) -> int:
+ info = await self._session_info(ov_session_id)
+ return int(info.get("pending_tokens", 0) or 0)
+
+ async def message_count(self, ov_session_id: str) -> int:
+ info = await self._session_info(ov_session_id)
+ return int(info.get("message_count", 0) or 0)
+
+ async def delete(self, ov_session_id: str) -> None:
+ try:
+ await self._client.delete_session(ov_session_id)
+ except NotFoundError:
+ pass
+
+
+class SessionReplayer:
+ def __init__(
+ self,
+ client: ConversationReplayClient,
+ store: CursorStore,
+ *,
+ session_id_prefix: str = "import",
+ memory_policy: Optional[Dict[str, Any]] = None,
+ ):
+ self.client = client
+ self.store = store
+ self.prefix = session_id_prefix
+ self.memory_policy = memory_policy or None
+
+ def ov_session_id(self, harness: str, native_session_id: str) -> str:
+ return "__".join(
+ [_sanitize_id(self.prefix), _sanitize_id(harness), _sanitize_id(native_session_id)]
+ )
+
+ async def reconcile(self, harness: str, ref: SessionRef) -> None:
+ """Resolve a pending batch intent left by a crash before the next read happens."""
+ rec = self.store.get(harness, ref.native_session_id)
+ if not rec or not rec.pending_count or rec.pending_cursor is None:
+ return
+ sid = self.ov_session_id(harness, ref.native_session_id)
+ try:
+ count = await self.client.message_count(sid)
+ except Exception: # noqa: BLE001 - leave the intent; retry next time
+ logger.warning("[ingest:%s] reconcile read failed for %s", harness, sid)
+ return
+ if count >= rec.pending_baseline + rec.pending_count:
+ # The batch landed before the crash: confirm it without re-appending.
+ self.store.confirm_append(
+ harness, ref.native_session_id, rec.pending_cursor, rec.pending_count
+ )
+ logger.info("[ingest:%s] reconciled landed batch for %s", harness, sid)
+ else:
+ # It did not land: drop the intent and re-read from the confirmed cursor.
+ self.store.clear_pending(harness, ref.native_session_id)
+
+ async def append_batch(
+ self,
+ harness: str,
+ ref: SessionRef,
+ messages: List[NormalizedMessage],
+ from_cursor: Cursor,
+ new_cursor: Cursor,
+ ) -> int:
+ """Append one <=100 batch idempotently and advance the confirmed cursor."""
+ sid = self.ov_session_id(harness, ref.native_session_id)
+ payloads = to_add_message_requests(messages)
+ if not payloads:
+ # Only control/empty turns consumed: advance the cursor, no commit owed.
+ self.store.advance_cursor(
+ harness, ref.native_session_id, sid, new_cursor, locator=ref.locator
+ )
+ return 0
+ await self.client.ensure_session(sid, self.memory_policy)
+ baseline = await self.client.message_count(sid)
+ self.store.set_pending(
+ harness,
+ ref.native_session_id,
+ sid,
+ from_cursor,
+ new_cursor,
+ len(payloads),
+ baseline,
+ locator=ref.locator,
+ title=ref.title,
+ )
+ await self.client.append(sid, payloads)
+ self.store.confirm_append(harness, ref.native_session_id, new_cursor, len(payloads))
+ return len(payloads)
+
+ async def commit_if_needed(
+ self, harness: str, ref: SessionRef, keep_recent_count: int = 0
+ ) -> bool:
+ """Commit when the session has un-committed appends or live pending tokens."""
+ sid = self.ov_session_id(harness, ref.native_session_id)
+ rec = self.store.get(harness, ref.native_session_id)
+ need = bool(rec and rec.needs_commit)
+ pending = await self.client.pending_tokens(sid)
+ if not need and pending <= 0:
+ return False
+ if pending <= 0:
+ # Nothing live to archive (already committed elsewhere); clear the stale flag.
+ self.store.mark_committed(harness, ref.native_session_id)
+ return False
+ await self.client.commit(sid, keep_recent_count=keep_recent_count)
+ self.store.mark_committed(harness, ref.native_session_id)
+ return True
+
+ async def maybe_commit_on_threshold(
+ self, harness: str, ref: SessionRef, threshold: int, keep_recent_count: int = 0
+ ) -> bool:
+ sid = self.ov_session_id(harness, ref.native_session_id)
+ pending = await self.client.pending_tokens(sid)
+ if pending < threshold:
+ return False
+ await self.client.commit(sid, keep_recent_count=keep_recent_count)
+ self.store.mark_committed(harness, ref.native_session_id)
+ return True
+
+ async def reset_session(self, harness: str, ref: SessionRef) -> None:
+ sid = self.ov_session_id(harness, ref.native_session_id)
+ await self.client.delete(sid)
+ self.store.delete(harness, ref.native_session_id)
diff --git a/openviking/ingest/sources/__init__.py b/openviking/ingest/sources/__init__.py
new file mode 100644
index 000000000..7e6ddc659
--- /dev/null
+++ b/openviking/ingest/sources/__init__.py
@@ -0,0 +1,15 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Harness log-source adapters. Importing this package registers all built-ins."""
+
+# Import for side effects: each module's @register_source populates SOURCE_REGISTRY.
+from openviking.ingest.sources import ( # noqa: F401
+ claude_code,
+ codex,
+ cursor,
+ hermes,
+ openclaw,
+ opencode,
+)
+
+__all__ = ["claude_code", "codex", "cursor", "hermes", "opencode", "openclaw"]
diff --git a/openviking/ingest/sources/base.py b/openviking/ingest/sources/base.py
new file mode 100644
index 000000000..0125ef023
--- /dev/null
+++ b/openviking/ingest/sources/base.py
@@ -0,0 +1,252 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Log-source abstraction: one ABC + two intermediates so a new harness is a thin subclass.
+
+- ``JsonlLogSource`` — append-only JSONL (Claude Code, Codex, Hermes, OpenClaw); byte-offset cursor.
+- ``SqliteLogSource`` — relational SQLite (OpenCode); (time, id) cursor, polled read-only.
+"""
+
+from __future__ import annotations
+
+import json
+import sqlite3
+from abc import ABC, abstractmethod
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, ClassVar, Dict, Iterable, List, Optional, Tuple
+
+from openviking.ingest.models import (
+ BYTE_OFFSET,
+ ROWID_TIME,
+ Cursor,
+ NormalizedMessage,
+ SessionRef,
+)
+from openviking.ingest.peer import (
+ assistant_peer_id,
+ resolve_git_human_peer,
+ safe_external_peer,
+)
+from openviking_cli.utils import get_logger
+
+if TYPE_CHECKING:
+ from openviking_cli.utils.config.ingest_config import IngestHarnessConfig
+
+logger = get_logger(__name__)
+
+# Max messages returned per read_messages() call. Bounds memory and lets the orchestrator
+# append+confirm in idempotent batches (<= the server's 100-message batch cap).
+DEFAULT_READ_LIMIT = 100
+_READ_BLOCK = 262144 # 256 KiB
+
+
+class NotSupportedError(RuntimeError):
+ """Raised by an adapter that is registered but not usable in the current config."""
+
+
+class LogSource(ABC):
+ """Base adapter for one agent harness's conversation logs."""
+
+ name: ClassVar[str] = "" # set by @register_source
+ cursor_kind: ClassVar[str] = BYTE_OFFSET
+ is_group_chat: ClassVar[bool] = False # group agents map user turns to original usernames
+
+ def __init__(self, harness_cfg: "IngestHarnessConfig", *, fallback_user: str = "default"):
+ self.cfg = harness_cfg
+ self.fallback_user = fallback_user
+
+ # --- paths -------------------------------------------------------------
+ @abstractmethod
+ def default_paths(self) -> List[Path]:
+ """Default discovery roots (files/dirs/db paths) when config gives no override."""
+
+ def roots(self) -> List[Path]:
+ if self.cfg.paths:
+ return [Path(p).expanduser() for p in self.cfg.paths]
+ return self.default_paths()
+
+ # --- discovery / reading ----------------------------------------------
+ @abstractmethod
+ def discover_sessions(self) -> Iterable[SessionRef]:
+ """Enumerate conversations available in this harness's storage."""
+
+ @abstractmethod
+ def read_messages(
+ self, ref: SessionRef, cursor: Optional[Cursor], limit: int = DEFAULT_READ_LIMIT
+ ) -> Tuple[List[NormalizedMessage], Cursor]:
+ """Read up to ``limit`` messages after ``cursor``; return (messages, advanced cursor)."""
+
+ # --- peer_id helpers (used by adapters) -------------------------------
+ def assistant_peer(self, model: Optional[str], provider: Optional[str] = None) -> Optional[str]:
+ return assistant_peer_id(self.name, model, provider)
+
+ def user_peer(
+ self, *, cwd: Optional[str] = None, raw_user: Optional[str] = None
+ ) -> Optional[str]:
+ if self.is_group_chat:
+ return safe_external_peer(raw_user) or safe_external_peer(self.fallback_user)
+ return resolve_git_human_peer(cwd, self.fallback_user)
+
+
+# =============================================================================
+# Append-only JSONL
+# =============================================================================
+class JsonlLogSource(LogSource):
+ cursor_kind = BYTE_OFFSET
+ file_glob: ClassVar[str] = "*.jsonl"
+
+ def discover_sessions(self) -> Iterable[SessionRef]:
+ for root in self.roots():
+ root = root.expanduser()
+ if not root.exists():
+ continue
+ for path in sorted(root.glob(self.file_glob)):
+ if path.is_file():
+ yield self.session_ref_for_file(path)
+
+ def session_ref_for_file(self, path: Path) -> SessionRef:
+ return SessionRef(
+ harness=self.name,
+ native_session_id=self.session_id_for_file(path),
+ locator=str(path),
+ )
+
+ def session_id_for_file(self, path: Path) -> str:
+ return path.stem
+
+ @staticmethod
+ def _peek_first_json(path: Path) -> Optional[Dict[str, Any]]:
+ try:
+ with open(path, "rb") as f:
+ for raw in f:
+ raw = raw.strip()
+ if raw:
+ return json.loads(raw)
+ except (OSError, ValueError):
+ return None
+ return None
+
+ def read_messages(
+ self, ref: SessionRef, cursor: Optional[Cursor], limit: int = DEFAULT_READ_LIMIT
+ ) -> Tuple[List[NormalizedMessage], Cursor]:
+ path = Path(ref.locator)
+ cur = cursor or Cursor.zero(self.cursor_kind)
+ try:
+ st = path.stat()
+ except OSError:
+ return [], cur
+
+ start = int(cur.value.get("offset", 0))
+ stored_inode = cur.value.get("inode")
+ # Rotation (file replaced) or truncation -> re-read from the top.
+ if (stored_inode is not None and stored_inode != st.st_ino) or start > st.st_size:
+ start = 0
+
+ messages: List[NormalizedMessage] = []
+ consumed = 0 # bytes of fully-processed (newline-terminated) lines
+ buf = b""
+ with open(path, "rb") as f:
+ f.seek(start)
+ while len(messages) < limit:
+ chunk = f.read(_READ_BLOCK)
+ if not chunk:
+ break # EOF; any partial trailing line stays in buf, unconsumed
+ buf += chunk
+ stop = False
+ while True:
+ nl = buf.find(b"\n")
+ if nl == -1:
+ break
+ line = buf[: nl + 1]
+ buf = buf[nl + 1 :]
+ consumed += len(line)
+ stripped = line.strip()
+ if stripped:
+ try:
+ obj = json.loads(stripped)
+ except ValueError:
+ obj = None
+ if obj is not None:
+ try:
+ messages.extend(self.parse_line(obj, ref))
+ except Exception as exc: # one bad record must not abort the file
+ logger.debug(
+ "[ingest:%s] skip bad record in %s: %s",
+ self.name,
+ path.name,
+ exc,
+ )
+ if len(messages) >= limit:
+ stop = True
+ break
+ if stop:
+ break
+
+ new_offset = start + consumed
+ return messages, Cursor(self.cursor_kind, {"offset": new_offset, "inode": st.st_ino})
+
+ @abstractmethod
+ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:
+ """Map one JSONL record to zero or more normalized messages."""
+
+
+# =============================================================================
+# Relational SQLite
+# =============================================================================
+class SqliteLogSource(LogSource):
+ cursor_kind = ROWID_TIME
+
+ @abstractmethod
+ def db_path(self) -> Path:
+ """Path to the harness's SQLite database (config override honored upstream)."""
+
+ def _connect(self, *, immutable: bool = False) -> sqlite3.Connection:
+ path = self.db_path()
+ if not path.exists():
+ raise NotSupportedError(f"{self.name}: database not found at {path}")
+ # mode=ro honors the WAL for live polling; immutable=1 (static snapshot) ignores it.
+ suffix = "mode=ro&immutable=1" if immutable else "mode=ro"
+ conn = sqlite3.connect(f"file:{path}?{suffix}", uri=True, timeout=5.0)
+ conn.row_factory = sqlite3.Row
+ return conn
+
+ def read_messages(
+ self, ref: SessionRef, cursor: Optional[Cursor], limit: int = DEFAULT_READ_LIMIT
+ ) -> Tuple[List[NormalizedMessage], Cursor]:
+ cur = cursor or Cursor.zero(self.cursor_kind)
+ conn = self._connect(immutable=False)
+ try:
+ rows = list(self.fetch_rows(conn, ref, cur, limit))
+ # Advance only past rows that are fully written. A still-incomplete trailing
+ # row (e.g. a message whose `part` text has not been flushed yet) is left for
+ # the next poll so we never skip its content.
+ complete: List[sqlite3.Row] = []
+ for row in rows:
+ if self.row_complete(conn, row):
+ complete.append(row)
+ else:
+ break
+ messages = self.rows_to_messages(conn, ref, complete)
+ if complete:
+ last = complete[-1]
+ new_cur = Cursor(self.cursor_kind, {"time": last["time_created"], "id": last["id"]})
+ else:
+ new_cur = cur
+ return messages, new_cur
+ finally:
+ conn.close()
+
+ def row_complete(self, conn: sqlite3.Connection, row: sqlite3.Row) -> bool:
+ """Whether a fetched row is fully written and safe to advance past (default: yes)."""
+ return True
+
+ @abstractmethod
+ def fetch_rows(
+ self, conn: sqlite3.Connection, ref: SessionRef, cursor: Cursor, limit: int
+ ) -> List[sqlite3.Row]:
+ """Up to ``limit`` rows after the cursor, ordered by (time_created, id)."""
+
+ @abstractmethod
+ def rows_to_messages(
+ self, conn: sqlite3.Connection, ref: SessionRef, rows: List[sqlite3.Row]
+ ) -> List[NormalizedMessage]:
+ """Map rows to normalized messages (may issue extra queries via ``conn``)."""
diff --git a/openviking/ingest/sources/claude_code.py b/openviking/ingest/sources/claude_code.py
new file mode 100644
index 000000000..750b08647
--- /dev/null
+++ b/openviking/ingest/sources/claude_code.py
@@ -0,0 +1,68 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Claude Code adapter.
+
+Logs: ``~/.claude/projects/<project-slug>/<session-uuid>.jsonl`` (append-only JSONL).
+Each record has a top-level ``type``; conversation turns are ``type in {user, assistant}``
+with a nested ``message{role, content, model}``. ``content`` is a string or a list of
+blocks (text / tool_use / tool_result). cwd + gitBranch are per record.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any, Dict, List
+
+from openviking.ingest.models import NormalizedMessage, SessionRef
+from openviking.ingest.registry import register_source
+from openviking.ingest.sources.base import JsonlLogSource
+
+
+def _extract_text(content: Any) -> str:
+ if isinstance(content, str):
+ return content.strip()
+ if isinstance(content, list):
+ chunks: List[str] = []
+ for block in content:
+ if isinstance(block, dict) and block.get("type") == "text":
+ text = block.get("text")
+ if isinstance(text, str) and text.strip():
+ chunks.append(text.strip())
+ return "\n".join(chunks).strip()
+ return ""
+
+
+@register_source("claude_code")
+class ClaudeCodeSource(JsonlLogSource):
+ file_glob = "*/*.jsonl"
+
+ def default_paths(self) -> List[Path]:
+ return [Path.home() / ".claude" / "projects"]
+
+ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:
+ if obj.get("type") not in ("user", "assistant"):
+ return []
+ if obj.get("isSidechain") or obj.get("isMeta"):
+ return [] # sub-agent / synthetic records are low-value for memory
+ message = obj.get("message")
+ if not isinstance(message, dict):
+ return []
+ role = message.get("role")
+ if role not in ("user", "assistant"):
+ return []
+ text = _extract_text(message.get("content"))
+ if not text:
+ return [] # tool-only turn -> dropped
+
+ model = message.get("model")
+ cwd = obj.get("cwd")
+ peer = self.assistant_peer(model) if role == "assistant" else self.user_peer(cwd=cwd)
+ return [
+ NormalizedMessage(
+ role=role,
+ text=text,
+ created_at=obj.get("timestamp"),
+ peer_id=peer,
+ meta={"model": model, "cwd": cwd, "git_branch": obj.get("gitBranch")},
+ )
+ ]
diff --git a/openviking/ingest/sources/codex.py b/openviking/ingest/sources/codex.py
new file mode 100644
index 000000000..db76cb937
--- /dev/null
+++ b/openviking/ingest/sources/codex.py
@@ -0,0 +1,95 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Codex (OpenAI Codex CLI) adapter.
+
+Logs: ``~/.codex/sessions/YYYY/MM/DD/rollout-<ts>-<uuid>.jsonl`` (append-only JSONL).
+Records: ``{timestamp, type, payload}``. Conversation turns are
+``type=="response_item" & payload.type=="message"`` with ``payload.role`` and
+``payload.content[].text`` (``input_text`` / ``output_text``). ``role=="developer"``/
+``"system"`` are dropped. Session id / cwd / provider come from the first
+``session_meta`` record (the model name is not reliably per-message).
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any, Dict, List
+
+from openviking.ingest.models import NormalizedMessage, SessionRef
+from openviking.ingest.registry import register_source
+from openviking.ingest.sources.base import JsonlLogSource
+
+
+def _join_content(content: Any) -> str:
+ if not isinstance(content, list):
+ return ""
+ chunks: List[str] = []
+ for block in content:
+ if isinstance(block, dict) and block.get("type") in ("input_text", "output_text"):
+ text = block.get("text")
+ if isinstance(text, str) and text.strip():
+ chunks.append(text.strip())
+ return "\n".join(chunks).strip()
+
+
+@register_source("codex")
+class CodexSource(JsonlLogSource):
+ file_glob = "*/*/*/rollout-*.jsonl"
+
+ def default_paths(self) -> List[Path]:
+ return [Path.home() / ".codex" / "sessions"]
+
+ def session_ref_for_file(self, path: Path) -> SessionRef:
+ meta = self._peek_session_meta(path)
+ return SessionRef(
+ harness=self.name,
+ native_session_id=meta.get("id") or path.stem,
+ locator=str(path),
+ started_at=meta.get("timestamp"),
+ meta={"model": meta.get("model_provider"), "cwd": meta.get("cwd")},
+ )
+
+ @staticmethod
+ def _peek_session_meta(path: Path) -> Dict[str, Any]:
+ try:
+ with open(path, "rb") as f:
+ for raw in f:
+ raw = raw.strip()
+ if not raw:
+ continue
+ obj = json.loads(raw)
+ if obj.get("type") == "session_meta":
+ return obj.get("payload", {}) or {}
+ except (OSError, ValueError):
+ pass
+ return {}
+
+ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:
+ if obj.get("type") != "response_item":
+ return []
+ payload = obj.get("payload") or {}
+ if payload.get("type") != "message":
+ return []
+ role = payload.get("role")
+ if role not in ("user", "assistant"):
+ return [] # drop developer/system boilerplate
+ text = _join_content(payload.get("content"))
+ if not text:
+ return []
+
+ model = ref.meta.get("model")
+ peer = (
+ self.assistant_peer(model)
+ if role == "assistant"
+ else self.user_peer(cwd=ref.meta.get("cwd"))
+ )
+ return [
+ NormalizedMessage(
+ role=role,
+ text=text,
+ created_at=obj.get("timestamp"),
+ peer_id=peer,
+ meta={"model": model, "cwd": ref.meta.get("cwd")},
+ )
+ ]
diff --git a/openviking/ingest/sources/cursor.py b/openviking/ingest/sources/cursor.py
new file mode 100644
index 000000000..466d76819
--- /dev/null
+++ b/openviking/ingest/sources/cursor.py
@@ -0,0 +1,59 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Cursor (IDE) adapter — DEFERRED stub.
+
+NOTE: this is the Cursor *IDE harness*, distinct from the read-position ``Cursor``
+pointer in ``models.py``.
+
+Cursor stores chat in ``~/Library/Application Support/Cursor/User/{globalStorage,
+workspaceStorage/*}/state.vscdb`` (SQLite key-value: ``ItemTable`` / ``cursorDiskKV``),
+with a single conversation spread across ``composerData:*`` / ``bubbleId:*`` /
+``agentKv:*`` BLOB→JSON keys. The schema is undocumented, drifts between Cursor
+versions, and offers no monotonic cursor for clean incremental reads — so ingestion is
+deferred. The adapter is registered (so it shows up in ``list-sources``) but refuses to
+run unless explicitly opted in (``enabled`` + ``experimental``), and even then is not yet
+implemented.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import List
+
+from openviking.ingest.registry import register_source
+from openviking.ingest.sources.base import NotSupportedError, SqliteLogSource
+
+
+@register_source("cursor")
+class CursorIDESource(SqliteLogSource):
+ def default_paths(self) -> List[Path]:
+ base = Path.home() / "Library" / "Application Support" / "Cursor" / "User"
+ return [base / "globalStorage" / "state.vscdb"]
+
+ def db_path(self) -> Path:
+ roots = self.roots()
+ return roots[0] if roots else self.default_paths()[0]
+
+ def _guard(self) -> None:
+ if not (self.cfg.enabled and self.cfg.experimental):
+ raise NotSupportedError(
+ "cursor ingest is deferred/experimental; set both enabled=true and "
+ "experimental=true to opt in"
+ )
+ raise NotSupportedError(
+ "cursor ingest is not implemented: state.vscdb stores conversations as "
+ "undocumented, version-unstable KV blobs (composerData/bubbleId/agentKv) "
+ "with no monotonic cursor"
+ )
+
+ def discover_sessions(self):
+ self._guard()
+ return []
+
+ def fetch_rows(self, conn, ref, cursor, limit): # pragma: no cover - deferred
+ self._guard()
+ return []
+
+ def rows_to_messages(self, conn, ref, rows): # pragma: no cover - deferred
+ self._guard()
+ return []
diff --git a/openviking/ingest/sources/hermes.py b/openviking/ingest/sources/hermes.py
new file mode 100644
index 000000000..89a125e79
--- /dev/null
+++ b/openviking/ingest/sources/hermes.py
@@ -0,0 +1,64 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Hermes adapter (group-chat agent).
+
+Logs: ``~/.hermes/sessions/<ts>_<id>.jsonl`` (append-only JSONL). Records are keyed by
+``role``: a leading ``session_meta`` (carries ``model`` + ``platform``), then ``user`` /
+``assistant`` turns with ``content`` (str) + ``timestamp``.
+
+Group-chat agent: user turns map to the original username when present in the log
+(``IngestHarnessConfig.user_field``); otherwise the configured OV user.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any, Dict, List
+
+from openviking.ingest.models import NormalizedMessage, SessionRef
+from openviking.ingest.registry import register_source
+from openviking.ingest.sources.base import JsonlLogSource
+
+
+@register_source("hermes")
+class HermesSource(JsonlLogSource):
+ is_group_chat = True
+ file_glob = "*.jsonl"
+
+ def default_paths(self) -> List[Path]:
+ return [Path.home() / ".hermes" / "sessions"]
+
+ def session_ref_for_file(self, path: Path) -> SessionRef:
+ first = self._peek_first_json(path) or {}
+ model = first.get("model") if first.get("role") == "session_meta" else None
+ return SessionRef(
+ harness=self.name,
+ native_session_id=path.stem,
+ locator=str(path),
+ started_at=first.get("timestamp"),
+ meta={"model": model, "platform": first.get("platform")},
+ )
+
+ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:
+ role = obj.get("role")
+ if role not in ("user", "assistant"):
+ return [] # drop session_meta and any other control record
+ text = (obj.get("content") or "").strip()
+ if not text:
+ return []
+
+ model = ref.meta.get("model")
+ if role == "assistant":
+ peer = self.assistant_peer(model)
+ else:
+ raw_user = obj.get(self.cfg.user_field) if self.cfg.user_field else None
+ peer = self.user_peer(raw_user=raw_user)
+ return [
+ NormalizedMessage(
+ role=role,
+ text=text,
+ created_at=obj.get("timestamp"),
+ peer_id=peer,
+ meta={"model": model, "platform": ref.meta.get("platform")},
+ )
+ ]
diff --git a/openviking/ingest/sources/openclaw.py b/openviking/ingest/sources/openclaw.py
new file mode 100644
index 000000000..9a7eeece4
--- /dev/null
+++ b/openviking/ingest/sources/openclaw.py
@@ -0,0 +1,74 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""OpenClaw adapter (group-chat agent).
+
+Logs: ``~/.openclaw/agents/<agent>/sessions/<uuid>.jsonl`` (append-only JSONL). Records
+carry a top-level ``type``; conversation turns are ``type=="message"`` with a nested
+``message{role, content[], timestamp}``. Assistant messages additionally carry
+``model`` + ``provider``. ``content`` is a list of blocks (text / thinking / …).
+
+Group-chat agent: user turns map to the original username when present
+(``IngestHarnessConfig.user_field``); otherwise the configured OV user.
+
+Format reference: openclaw-plugin session logs (see project memory on citing openclaw).
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any, Dict, List
+
+from openviking.ingest.models import NormalizedMessage, SessionRef
+from openviking.ingest.registry import register_source
+from openviking.ingest.sources.base import JsonlLogSource
+
+
+def _extract_text(content: Any) -> str:
+ if isinstance(content, str):
+ return content.strip()
+ if isinstance(content, list):
+ chunks: List[str] = []
+ for block in content:
+ if isinstance(block, dict) and block.get("type") == "text":
+ text = block.get("text")
+ if isinstance(text, str) and text.strip():
+ chunks.append(text.strip())
+ return "\n".join(chunks).strip()
+ return ""
+
+
+@register_source("openclaw")
+class OpenClawSource(JsonlLogSource):
+ is_group_chat = True
+ file_glob = "*/sessions/*.jsonl"
+
+ def default_paths(self) -> List[Path]:
+ return [Path.home() / ".openclaw" / "agents"]
+
+ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:
+ if obj.get("type") != "message":
+ return []
+ message = obj.get("message")
+ if not isinstance(message, dict):
+ return []
+ role = message.get("role")
+ if role not in ("user", "assistant"):
+ return []
+ text = _extract_text(message.get("content"))
+ if not text:
+ return []
+
+ if role == "assistant":
+ peer = self.assistant_peer(message.get("model"), message.get("provider"))
+ else:
+ raw_user = message.get(self.cfg.user_field) if self.cfg.user_field else None
+ peer = self.user_peer(raw_user=raw_user)
+ return [
+ NormalizedMessage(
+ role=role,
+ text=text,
+ created_at=obj.get("timestamp"),
+ peer_id=peer,
+ meta={"model": message.get("model"), "provider": message.get("provider")},
+ )
+ ]
diff --git a/openviking/ingest/sources/opencode.py b/openviking/ingest/sources/opencode.py
new file mode 100644
index 000000000..cfb56e2a9
--- /dev/null
+++ b/openviking/ingest/sources/opencode.py
@@ -0,0 +1,137 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""OpenCode (sst/opencode) adapter — SUPPORTED-EXPERIMENTAL.
+
+Logs: ``~/.local/share/opencode/opencode.db`` (SQLite, WAL). ``session(id, title,
+directory, model, time_created)``; ``message(id, session_id, time_created, data JSON
+{role, modelID, providerID, …})``; the actual text lives in ``part(message_id,
+time_created, data JSON{type, text})``. Polled read-only via (time_created, id) cursor.
+
+Older OpenCode versions used a JSON file-store; that layout is deferred (clear error).
+"""
+
+from __future__ import annotations
+
+import json
+import sqlite3
+from pathlib import Path
+from typing import List
+
+from openviking.ingest.models import NormalizedMessage, SessionRef, iso_from_epoch_ms
+from openviking.ingest.registry import register_source
+from openviking.ingest.sources.base import NotSupportedError, SqliteLogSource
+
+
+@register_source("opencode")
+class OpenCodeSource(SqliteLogSource):
+ def default_paths(self) -> List[Path]:
+ return [Path.home() / ".local" / "share" / "opencode" / "opencode.db"]
+
+ def db_path(self) -> Path:
+ roots = self.roots()
+ return roots[0] if roots else self.default_paths()[0]
+
+ def discover_sessions(self):
+ db = self.db_path()
+ if not db.exists():
+ legacy = Path.home() / ".local" / "share" / "opencode" / "project"
+ if legacy.exists():
+ raise NotSupportedError(
+ "opencode legacy file-store (~/.local/share/opencode/project) "
+ "is not supported (deferred); only opencode.db is supported"
+ )
+ return
+ conn = self._connect()
+ try:
+ rows = conn.execute(
+ "SELECT id, title, directory, model, time_created "
+ "FROM session ORDER BY time_created"
+ ).fetchall()
+ finally:
+ conn.close()
+ for r in rows:
+ yield SessionRef(
+ harness=self.name,
+ native_session_id=r["id"],
+ locator=r["id"],
+ title=r["title"],
+ started_at=iso_from_epoch_ms(r["time_created"]),
+ meta={"cwd": r["directory"], "session_model": r["model"]},
+ )
+
+ def fetch_rows(self, conn, ref: SessionRef, cursor, limit: int) -> List[sqlite3.Row]:
+ t = cursor.value.get("time", 0)
+ last_id = cursor.value.get("id", "")
+ # opencode message ids are ULID-like (monotonic with insertion), so (time, id)
+ # is a stable order/cursor.
+ return conn.execute(
+ "SELECT id, time_created, data FROM message "
+ "WHERE session_id = ? AND (time_created > ? OR (time_created = ? AND id > ?)) "
+ "ORDER BY time_created, id LIMIT ?",
+ (ref.locator, t, t, last_id, limit),
+ ).fetchall()
+
+ def row_complete(self, conn, row) -> bool:
+ # A message row may exist before its `part` text is flushed; only advance the
+ # cursor past it once it is final, so late-arriving parts are never skipped.
+ try:
+ data = json.loads(row["data"])
+ except (ValueError, TypeError):
+ return True
+ if data.get("role") == "user":
+ return True
+ return bool((data.get("time") or {}).get("completed") or data.get("finish"))
+
+ def rows_to_messages(self, conn, ref: SessionRef, rows) -> List[NormalizedMessage]:
+ out: List[NormalizedMessage] = []
+ for row in rows:
+ try:
+ data = json.loads(row["data"])
+ except (ValueError, TypeError):
+ continue
+ role = data.get("role")
+ if role not in ("user", "assistant"):
+ continue
+
+ text = self._reassemble_text(conn, row["id"])
+ if not text:
+ continue
+
+ if role == "assistant":
+ model = data.get("modelID") or ref.meta.get("session_model")
+ peer = self.assistant_peer(model, data.get("providerID"))
+ else:
+ model = data.get("modelID")
+ peer = self.user_peer(cwd=ref.meta.get("cwd"))
+ out.append(
+ NormalizedMessage(
+ role=role,
+ text=text,
+ created_at=iso_from_epoch_ms(row["time_created"]),
+ peer_id=peer,
+ meta={
+ "model": data.get("modelID"),
+ "provider": data.get("providerID"),
+ "cwd": ref.meta.get("cwd"),
+ },
+ )
+ )
+ return out
+
+ @staticmethod
+ def _reassemble_text(conn, message_id: str) -> str:
+ parts = conn.execute(
+ "SELECT data FROM part WHERE message_id = ? ORDER BY time_created, id",
+ (message_id,),
+ ).fetchall()
+ chunks: List[str] = []
+ for p in parts:
+ try:
+ pd = json.loads(p["data"])
+ except (ValueError, TypeError):
+ continue
+ if pd.get("type") == "text":
+ text = pd.get("text")
+ if isinstance(text, str) and text.strip():
+ chunks.append(text.strip())
+ return "\n".join(chunks).strip()
diff --git a/openviking_cli/server_bootstrap.py b/openviking_cli/server_bootstrap.py
index 523f6606b..51d96eb69 100644
--- a/openviking_cli/server_bootstrap.py
+++ b/openviking_cli/server_bootstrap.py
@@ -51,6 +51,14 @@ def main():
with bind_log_execution_trace():
sys.exit(doctor_main())
+ # `openviking-server ingest ...` runs the local-log ingestion CLI (client-side).
+ if len(sys.argv) > 1 and sys.argv[1] == "ingest":
+ from openviking.ingest.cli import app as ingest_app
+
+ with bind_log_execution_trace():
+ ingest_app(args=sys.argv[2:], prog_name="openviking-server ingest")
+ return
+
from openviking.server.bootstrap import main as _real_main
with bind_log_execution_trace():
diff --git a/openviking_cli/utils/config/consts.py b/openviking_cli/utils/config/consts.py
index 883a02d9e..7c326b477 100644
--- a/openviking_cli/utils/config/consts.py
+++ b/openviking_cli/utils/config/consts.py
@@ -74,6 +74,17 @@ OPENVIKING_GO_PATH_ENV = "OPENVIKING_GO_PATH"
OPENVIKING_GOPATH_ENV = "OPENVIKING_GOPATH"
OPENVIKING_GOPROXY_ENV = "OPENVIKING_GOPROXY"
+# =============================================================================
+# Conversation-log ingest (openviking-server ingest)
+# =============================================================================
+
+OPENVIKING_INGEST_ENABLED_ENV = "OPENVIKING_INGEST_ENABLED"
+OPENVIKING_INGEST_SERVER_URL_ENV = "OPENVIKING_INGEST_SERVER_URL"
+OPENVIKING_INGEST_API_KEY_ENV = "OPENVIKING_INGEST_API_KEY"
+
+# Per-(harness, session) read-cursor state lives here (NEVER under tool-specific dirs).
+DEFAULT_INGEST_STATE_DIR = DEFAULT_CONFIG_DIR / "ingest"
+
# =============================================================================
# Default filenames
# =============================================================================
diff --git a/openviking_cli/utils/config/ingest_config.py b/openviking_cli/utils/config/ingest_config.py
new file mode 100644
index 000000000..bf4e3fc81
--- /dev/null
+++ b/openviking_cli/utils/config/ingest_config.py
@@ -0,0 +1,139 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Configuration for the conversation-log ingest subsystem (``openviking-server ingest``).
+
+Ingest parses local agent-harness conversation logs (Claude Code, Codex, OpenCode,
+Hermes, OpenClaw, Cursor) and "replays" them through OpenViking's session pipeline.
+See ``openviking/ingest/`` for the runtime.
+"""
+
+import os
+from typing import Dict, List, Literal, Optional
+
+from pydantic import BaseModel, Field, model_validator
+
+from .consts import (
+ OPENVIKING_INGEST_API_KEY_ENV,
+ OPENVIKING_INGEST_ENABLED_ENV,
+ OPENVIKING_INGEST_SERVER_URL_ENV,
+)
+
+IngestMode = Literal["off", "backfill", "watch", "both"]
+
+
+def _env_truthy(value: Optional[str]) -> Optional[bool]:
+ if value is None:
+ return None
+ return value.strip().lower() in {"1", "true", "yes", "on"}
+
+
+class CommitPolicy(BaseModel):
+ """When a replayed session is committed (which triggers OV memory extraction)."""
+
+ commit_token_threshold: int = Field(
+ default=6000,
+ gt=0,
+ description="Commit a session once its pending (un-archived) token count reaches this.",
+ )
+ commit_idle_seconds: float = Field(
+ default=5.0,
+ gt=0,
+ description="In watch mode, commit a session after it has been idle this long.",
+ )
+ keep_recent_count: int = Field(
+ default=0,
+ ge=0,
+ description="WM v2 sliding window: messages to retain live after a commit (0 = archive all).",
+ )
+
+ model_config = {"extra": "forbid"}
+
+
+class IngestHarnessConfig(BaseModel):
+ """Per-harness ingest settings, keyed by registry name (e.g. ``claude_code``)."""
+
+ enabled: bool = Field(default=False, description="Enable ingesting this harness.")
+ mode: IngestMode = Field(
+ default="backfill",
+ description="off | backfill (one-shot historical) | watch (incremental) | both.",
+ )
+ paths: List[str] = Field(
+ default_factory=list,
+ description="Override the harness's default log discovery roots (file/dir/db paths).",
+ )
+ poll_interval_seconds: float = Field(
+ default=5.0,
+ gt=0,
+ description="Incremental poll cadence for this harness (watch mode).",
+ )
+ user_field: str = Field(
+ default="",
+ description=(
+ "For group-chat harnesses (hermes/openclaw): the log key holding the original "
+ "username, used as the user-side peer_id. Empty = use the adapter default."
+ ),
+ )
+ experimental: bool = Field(
+ default=False,
+ description="Opt into experimental adapters (e.g. cursor) that may be fragile.",
+ )
+ commit: CommitPolicy = Field(default_factory=CommitPolicy)
+
+ model_config = {"extra": "forbid"}
+
+
+class IngestConfig(BaseModel):
+ """Top-level ingest configuration (``ingest`` section of ``ov.conf``)."""
+
+ enabled: bool = Field(default=False, description="Master switch for log ingestion.")
+ server_url: str = Field(
+ default="",
+ description="OV server URL the ingest client targets. Empty => OPENVIKING_URL / localhost.",
+ )
+ api_key: str = Field(default="", description="API key for the target OV server.")
+ account: str = Field(default="default", description="OV account that owns ingested sessions.")
+ user: str = Field(default="default", description="OV user that owns ingested sessions.")
+ state_dir: str = Field(
+ default="",
+ description="Where the read-cursor state DB lives. Empty => ~/.openviking/ingest.",
+ )
+ session_id_prefix: str = Field(
+ default="import",
+ description="OV session ids are '{prefix}__{harness}__{native_session_id}'.",
+ )
+ memory_policy: Dict = Field(
+ default_factory=dict,
+ description="Passthrough memory_policy for replayed sessions (empty => server default).",
+ )
+ harnesses: Dict[str, IngestHarnessConfig] = Field(
+ default_factory=dict,
+ description="Per-harness configuration, keyed by registry name.",
+ )
+
+ model_config = {"extra": "forbid"}
+
+ @model_validator(mode="after")
+ def _apply_env_overrides(self) -> "IngestConfig":
+ # Operational toggles may come from the environment (deploy-time > config).
+ enabled = _env_truthy(os.environ.get(OPENVIKING_INGEST_ENABLED_ENV))
+ if enabled is not None:
+ self.enabled = enabled
+ server_url = os.environ.get(OPENVIKING_INGEST_SERVER_URL_ENV)
+ if server_url:
+ self.server_url = server_url
+ api_key = os.environ.get(OPENVIKING_INGEST_API_KEY_ENV)
+ if api_key:
+ self.api_key = api_key
+ return self
+
+ def enabled_harnesses(self) -> Dict[str, IngestHarnessConfig]:
+ """Harnesses that are turned on and not in 'off' mode.
+
+ The master switch ``enabled`` gates everything: when it is false, no harness
+ is active regardless of its own ``enabled`` flag.
+ """
+ if not self.enabled:
+ return {}
+ return {
+ name: cfg for name, cfg in self.harnesses.items() if cfg.enabled and cfg.mode != "off"
+ }
diff --git a/openviking_cli/utils/config/open_viking_config.py b/openviking_cli/utils/config/open_viking_config.py
index 8b6a4b970..45886023c 100644
--- a/openviking_cli/utils/config/open_viking_config.py
+++ b/openviking_cli/utils/config/open_viking_config.py
@@ -23,6 +23,7 @@ from .embedding_config import EmbeddingConfig
from .encryption_config import EncryptionConfig
from .git_config import GitConfig
from .grep_config import GrepConfig
+from .ingest_config import IngestConfig
from .log_config import LogConfig
from .memory_config import MemoryConfig
from .oauth_config import OAuthConfig
@@ -321,6 +322,11 @@ class OpenVikingConfig(BaseModel):
description="Prompt template configuration",
)
+ ingest: IngestConfig = Field(
+ default_factory=IngestConfig,
+ description="Conversation-log ingest (openviking-server ingest) configuration",
+ )
+
model_config = {"arbitrary_types_allowed": True, "extra": "forbid"}
@classmethod
diff --git a/pyproject.toml b/pyproject.toml
index 8b2fdca8f..e52293008 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -207,7 +207,7 @@ Issues = "https://github.com/volcengine/openviking/issues"
[project.scripts]
ov = "openviking_cli.rust_cli:main" # Rust CLI 入口(极简包装器)
openviking = "openviking_cli.rust_cli:main" # Rust CLI 入口(放弃 python CLI)
-openviking-server = "openviking_cli.server_bootstrap:main"
+openviking-server = "openviking_cli.server_bootstrap:main" # also: `openviking-server ingest ...`
vikingbot = "vikingbot.cli.commands:app"
[tool.setuptools_scm]
diff --git a/tests/ingest/__init__.py b/tests/ingest/__init__.py
new file mode 100644
index 000000000..236633478
--- /dev/null
+++ b/tests/ingest/__init__.py
@@ -0,0 +1,2 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
diff --git a/tests/ingest/conftest.py b/tests/ingest/conftest.py
new file mode 100644
index 000000000..8760a0f8e
--- /dev/null
+++ b/tests/ingest/conftest.py
@@ -0,0 +1,24 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Test isolation for the ingest suite.
+
+``resolve_git_human_peer`` shells out to ``git config``, which otherwise reads the
+developer machine's GLOBAL identity and makes human-peer resolution non-deterministic.
+Neutralize git config discovery so user-side peers fall back to the configured value.
+"""
+
+import pytest
+
+from openviking.ingest import peer
+
+
[email protected](autouse=True)
+def _isolate_git_identity(tmp_path_factory, monkeypatch):
+ empty = tmp_path_factory.mktemp("gitcfg") / "empty.cfg"
+ empty.write_text("")
+ monkeypatch.setenv("GIT_CONFIG_GLOBAL", str(empty))
+ monkeypatch.setenv("GIT_CONFIG_SYSTEM", str(empty))
+ monkeypatch.setenv("GIT_CONFIG_NOSYSTEM", "1")
+ peer._GIT_PEER_CACHE.clear()
+ yield
+ peer._GIT_PEER_CACHE.clear()
diff --git a/tests/ingest/test_config.py b/tests/ingest/test_config.py
new file mode 100644
index 000000000..4cdd14c5e
--- /dev/null
+++ b/tests/ingest/test_config.py
@@ -0,0 +1,40 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""IngestConfig gating + value validation."""
+
+import pytest
+from pydantic import ValidationError
+
+from openviking_cli.utils.config.ingest_config import (
+ CommitPolicy,
+ IngestConfig,
+ IngestHarnessConfig,
+)
+
+
+def test_master_switch_gates_all_harnesses():
+ enabled_h = {"claude_code": IngestHarnessConfig(enabled=True, mode="both")}
+ assert IngestConfig(enabled=False, harnesses=enabled_h).enabled_harnesses() == {}
+ assert list(IngestConfig(enabled=True, harnesses=enabled_h).enabled_harnesses()) == [
+ "claude_code"
+ ]
+
+
+def test_off_mode_excluded():
+ cfg = IngestConfig(
+ enabled=True,
+ harnesses={
+ "claude_code": IngestHarnessConfig(enabled=True, mode="off"),
+ "codex": IngestHarnessConfig(enabled=False, mode="both"),
+ },
+ )
+ assert cfg.enabled_harnesses() == {}
+
+
+def test_positive_value_validation():
+ with pytest.raises(ValidationError):
+ IngestHarnessConfig(poll_interval_seconds=0)
+ with pytest.raises(ValidationError):
+ CommitPolicy(commit_token_threshold=0)
+ with pytest.raises(ValidationError):
+ CommitPolicy(commit_idle_seconds=-1)
diff --git a/tests/ingest/test_cursor_store.py b/tests/ingest/test_cursor_store.py
new file mode 100644
index 000000000..f063d284f
--- /dev/null
+++ b/tests/ingest/test_cursor_store.py
@@ -0,0 +1,43 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""CursorStore persistence / resume / accumulation."""
+
+from openviking.ingest.cursor_store import CursorStore
+from openviking.ingest.models import BYTE_OFFSET, Cursor
+
+
+def test_upsert_accumulate_and_resume(tmp_path):
+ store = CursorStore(tmp_path)
+ c1 = Cursor(BYTE_OFFSET, {"offset": 10, "inode": 1})
+ store.upsert("claude_code", "s1", "import__claude_code__s1", c1, appended_delta=3, locator="/x")
+ rec = store.get("claude_code", "s1")
+ assert rec.cursor.value == {"offset": 10, "inode": 1}
+ assert rec.last_appended_count == 3
+ assert rec.ov_session_id == "import__claude_code__s1"
+ assert rec.last_committed_at is None
+
+ # advance + accumulate
+ c2 = Cursor(BYTE_OFFSET, {"offset": 42, "inode": 1})
+ store.upsert("claude_code", "s1", "import__claude_code__s1", c2, appended_delta=2)
+ rec = store.get("claude_code", "s1")
+ assert rec.cursor.value["offset"] == 42
+ assert rec.last_appended_count == 5 # 3 + 2
+ assert rec.locator == "/x" # preserved across upsert
+ store.close()
+
+ # resume from a fresh store on the same dir
+ store2 = CursorStore(tmp_path)
+ rec2 = store2.get("claude_code", "s1")
+ assert rec2.cursor.value["offset"] == 42
+ assert rec2.last_appended_count == 5
+ store2.close()
+
+
+def test_commit_marks_timestamp(tmp_path):
+ store = CursorStore(tmp_path)
+ c = Cursor(BYTE_OFFSET, {"offset": 1})
+ store.upsert("codex", "s", "ov", c, appended_delta=1)
+ assert store.get("codex", "s").last_committed_at is None
+ store.upsert("codex", "s", "ov", c, pending_tokens=0, committed=True)
+ assert store.get("codex", "s").last_committed_at is not None
+ store.close()
diff --git a/tests/ingest/test_jsonl_incremental.py b/tests/ingest/test_jsonl_incremental.py
new file mode 100644
index 000000000..91b68b743
--- /dev/null
+++ b/tests/ingest/test_jsonl_incremental.py
@@ -0,0 +1,61 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Incremental byte-offset cursor behavior for append-only JSONL sources."""
+
+import json
+import os
+
+from openviking.ingest.sources.hermes import HermesSource
+from openviking_cli.utils.config.ingest_config import IngestHarnessConfig
+
+
+def _src(root):
+ return HermesSource(IngestHarnessConfig(enabled=True, paths=[str(root)]), fallback_user="t")
+
+
+def _rec(role, content):
+ return json.dumps({"role": role, "content": content})
+
+
+def test_partial_trailing_line_not_consumed(tmp_path):
+ root = tmp_path / "sessions"
+ root.mkdir(parents=True)
+ f = root / "s.jsonl"
+ # two complete lines + a partial (no trailing newline) line
+ f.write_text(_rec("user", "a") + "\n" + _rec("assistant", "b") + "\n" + _rec("user", "cc"))
+
+ src = _src(root)
+ ref = next(iter(src.discover_sessions()))
+ msgs, cursor = src.read_messages(ref, None)
+ assert [m.text for m in msgs] == ["a", "b"] # partial line withheld
+
+ # complete the partial line and append another complete line
+ with open(f, "a") as fh:
+ fh.write("\n" + _rec("assistant", "d") + "\n")
+
+ msgs2, cursor2 = src.read_messages(ref, cursor)
+ assert [m.text for m in msgs2] == ["cc", "d"]
+
+ # nothing new on a third read
+ msgs3, _ = src.read_messages(ref, cursor2)
+ assert msgs3 == []
+
+
+def test_rotation_resets_cursor(tmp_path):
+ root = tmp_path / "sessions"
+ root.mkdir(parents=True)
+ f = root / "s.jsonl"
+ f.write_text(_rec("user", "old1") + "\n" + _rec("assistant", "old2") + "\n")
+
+ src = _src(root)
+ ref = next(iter(src.discover_sessions()))
+ msgs, cursor = src.read_messages(ref, None)
+ assert [m.text for m in msgs] == ["old1", "old2"]
+ assert "inode" in cursor.value
+
+ # Replace the file with brand-new, shorter content (new inode).
+ os.remove(f)
+ f.write_text(_rec("user", "new1") + "\n")
+
+ msgs2, _ = src.read_messages(ref, cursor)
+ assert [m.text for m in msgs2] == ["new1"] # rotation detected -> re-read from 0
diff --git a/tests/ingest/test_parsers.py b/tests/ingest/test_parsers.py
new file mode 100644
index 000000000..3b7d83e60
--- /dev/null
+++ b/tests/ingest/test_parsers.py
@@ -0,0 +1,231 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Adapter parsing tests against synthetic per-harness fixtures."""
+
+import json
+import sqlite3
+
+from openviking.ingest.sources.claude_code import ClaudeCodeSource
+from openviking.ingest.sources.codex import CodexSource
+from openviking.ingest.sources.hermes import HermesSource
+from openviking.ingest.sources.openclaw import OpenClawSource
+from openviking.ingest.sources.opencode import OpenCodeSource
+from openviking_cli.utils.config.ingest_config import IngestHarnessConfig
+
+
+def _write_jsonl(path, records):
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with open(path, "w") as f:
+ for r in records:
+ f.write(json.dumps(r) + "\n")
+
+
+def _cfg(root, **kw):
+ return IngestHarnessConfig(enabled=True, paths=[str(root)], **kw)
+
+
+def _read_all(source):
+ refs = list(source.discover_sessions())
+ assert len(refs) == 1
+ msgs, cursor = source.read_messages(refs[0], None)
+ return refs[0], msgs, cursor
+
+
+def test_claude_code(tmp_path):
+ root = tmp_path / "projects"
+ _write_jsonl(
+ root / "slug" / "sess-1.jsonl",
+ [
+ {"type": "queue-operation"}, # dropped (non-message)
+ {
+ "type": "user",
+ "cwd": str(tmp_path),
+ "timestamp": "2026-06-01T00:00:00Z",
+ "message": {"role": "user", "content": "hello there"},
+ },
+ {
+ "type": "assistant",
+ "timestamp": "2026-06-01T00:00:01Z",
+ "message": {
+ "role": "assistant",
+ "model": "claude-opus-4-8",
+ "content": [
+ {"type": "text", "text": "hi!"},
+ {"type": "tool_use", "name": "Read"}, # dropped
+ ],
+ },
+ },
+ { # sub-agent record -> dropped
+ "type": "assistant",
+ "isSidechain": True,
+ "message": {"role": "assistant", "content": [{"type": "text", "text": "side"}]},
+ },
+ ],
+ )
+ src = ClaudeCodeSource(_cfg(root), fallback_user="tester")
+ ref, msgs, _ = _read_all(src)
+ assert ref.native_session_id == "sess-1"
+ assert [(m.role, m.text) for m in msgs] == [("user", "hello there"), ("assistant", "hi!")]
+ assert msgs[1].peer_id == "claude_code__claude-opus-4-8"
+ assert msgs[0].peer_id == "tester" # no git repo -> configured fallback
+
+
+def test_codex(tmp_path):
+ root = tmp_path / "sessions"
+ _write_jsonl(
+ root / "2026" / "06" / "25" / "rollout-x-abc.jsonl",
+ [
+ {
+ "type": "session_meta",
+ "timestamp": "2026-06-25T00:00:00Z",
+ "payload": {"id": "codex-sess", "model_provider": "openai", "cwd": str(tmp_path)},
+ },
+ {
+ "type": "response_item",
+ "timestamp": "2026-06-25T00:00:01Z",
+ "payload": {
+ "type": "message",
+ "role": "developer", # dropped boilerplate
+ "content": [{"type": "input_text", "text": "system stuff"}],
+ },
+ },
+ {
+ "type": "response_item",
+ "timestamp": "2026-06-25T00:00:02Z",
+ "payload": {
+ "type": "message",
+ "role": "user",
+ "content": [{"type": "input_text", "text": "fix the bug"}],
+ },
+ },
+ {
+ "type": "response_item",
+ "timestamp": "2026-06-25T00:00:03Z",
+ "payload": {
+ "type": "message",
+ "role": "assistant",
+ "content": [{"type": "output_text", "text": "done"}],
+ },
+ },
+ {"type": "response_item", "payload": {"type": "reasoning"}}, # dropped
+ ],
+ )
+ src = CodexSource(_cfg(root), fallback_user="tester")
+ ref, msgs, _ = _read_all(src)
+ assert ref.native_session_id == "codex-sess"
+ assert [(m.role, m.text) for m in msgs] == [("user", "fix the bug"), ("assistant", "done")]
+ assert msgs[1].peer_id == "codex__openai"
+
+
+def test_hermes_group_username(tmp_path):
+ root = tmp_path / "sessions"
+ _write_jsonl(
+ root / "grp.jsonl",
+ [
+ {"role": "session_meta", "model": "doubao-x", "platform": "feishu"},
+ {
+ "role": "user",
+ "content": "hi",
+ "timestamp": "2026-06-01T00:00:00Z",
+ "sender": "alice",
+ },
+ {"role": "assistant", "content": "hello", "timestamp": "2026-06-01T00:00:01Z"},
+ ],
+ )
+ src = HermesSource(_cfg(root, user_field="sender"), fallback_user="tester")
+ _, msgs, _ = _read_all(src)
+ assert msgs[0].peer_id == "alice" # original username from the log
+ assert msgs[1].peer_id == "hermes__doubao-x"
+
+
+def test_openclaw_assistant_model(tmp_path):
+ root = tmp_path / "agents"
+ _write_jsonl(
+ root / "main" / "sessions" / "oc.jsonl",
+ [
+ {"type": "session", "id": "oc"},
+ {
+ "type": "message",
+ "timestamp": "2026-06-01T00:00:00Z",
+ "message": {"role": "user", "content": [{"type": "text", "text": "run it"}]},
+ },
+ {
+ "type": "message",
+ "timestamp": "2026-06-01T00:00:01Z",
+ "message": {
+ "role": "assistant",
+ "provider": "ark",
+ "model": "doubao-x",
+ "content": [
+ {"type": "thinking", "thinking": "hmm"}, # dropped
+ {"type": "text", "text": "ok"},
+ ],
+ },
+ },
+ ],
+ )
+ src = OpenClawSource(_cfg(root, user_field="sender"), fallback_user="tester")
+ _, msgs, _ = _read_all(src)
+ assert [(m.role, m.text) for m in msgs] == [("user", "run it"), ("assistant", "ok")]
+ assert msgs[1].peer_id == "openclaw__ark__doubao-x"
+
+
+def test_opencode_text_from_part_table(tmp_path):
+ db = tmp_path / "opencode.db"
+ conn = sqlite3.connect(str(db))
+ conn.executescript(
+ """
+ CREATE TABLE session (id TEXT, title TEXT, directory TEXT, model TEXT, time_created INT);
+ CREATE TABLE message (id TEXT, session_id TEXT, time_created INT, data TEXT);
+ CREATE TABLE part (id TEXT, message_id TEXT, session_id TEXT, time_created INT, data TEXT);
+ """
+ )
+ conn.execute(
+ "INSERT INTO session VALUES (?,?,?,?,?)",
+ ("ses_1", "demo", str(tmp_path), "m", 1000),
+ )
+ conn.execute(
+ "INSERT INTO message VALUES (?,?,?,?)",
+ ("msg_u", "ses_1", 1773044814194, json.dumps({"role": "user"})),
+ )
+ conn.execute(
+ "INSERT INTO message VALUES (?,?,?,?)",
+ (
+ "msg_a",
+ "ses_1",
+ 1773044819959,
+ json.dumps(
+ {
+ "role": "assistant",
+ "modelID": "glm-4.7",
+ "providerID": "tiktok",
+ "finish": "stop", # mark complete so the cursor advances past it
+ }
+ ),
+ ),
+ )
+ # text lives in the part table, not message.data
+ conn.execute(
+ "INSERT INTO part VALUES (?,?,?,?,?)",
+ ("p1", "msg_u", "ses_1", 1, json.dumps({"type": "text", "text": "hello"})),
+ )
+ conn.execute(
+ "INSERT INTO part VALUES (?,?,?,?,?)",
+ ("p2", "msg_a", "ses_1", 1, json.dumps({"type": "step-start"})), # not text
+ )
+ conn.execute(
+ "INSERT INTO part VALUES (?,?,?,?,?)",
+ ("p3", "msg_a", "ses_1", 2, json.dumps({"type": "text", "text": "world"})),
+ )
+ conn.commit()
+ conn.close()
+
+ src = OpenCodeSource(_cfg(db), fallback_user="tester")
+ refs = list(src.discover_sessions())
+ assert len(refs) == 1
+ msgs, cursor = src.read_messages(refs[0], None)
+ assert [(m.role, m.text) for m in msgs] == [("user", "hello"), ("assistant", "world")]
+ assert msgs[1].peer_id == "opencode__tiktok__glm-4.7"
+ # cursor advanced; re-read returns nothing new
+ msgs2, _ = src.read_messages(refs[0], cursor)
+ assert msgs2 == []
diff --git a/tests/ingest/test_peer.py b/tests/ingest/test_peer.py
new file mode 100644
index 000000000..ab06db8e6
--- /dev/null
+++ b/tests/ingest/test_peer.py
@@ -0,0 +1,33 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""peer_id resolution: assistant model peers, human/git peers, group usernames."""
+
+from openviking.ingest.peer import (
+ assistant_peer_id,
+ resolve_git_human_peer,
+ safe_external_peer,
+)
+
+
+def test_assistant_peer_with_and_without_provider():
+ assert assistant_peer_id("claude_code", "claude-opus-4-8") == "claude_code__claude-opus-4-8"
+ assert assistant_peer_id("opencode", "glm-4.7", "tiktok") == "opencode__tiktok__glm-4.7"
+ assert assistant_peer_id("codex", None) == "codex__unknown"
+
+
+def test_external_peer_readable_and_sanitized():
+ assert safe_external_peer("alice") == "alice"
+ assert safe_external_peer("[email protected]") == "[email protected]"
+ assert safe_external_peer("foo/bar baz") == "foo-bar-baz"
+ assert safe_external_peer("") is None
+
+
+def test_external_peer_non_ascii_falls_back_to_ext():
+ pid = safe_external_peer("杨冠姝")
+ assert pid is not None and pid.startswith("ext-")
+
+
+def test_git_human_peer_falls_back_without_repo(tmp_path):
+ # tmp_path is not a git repo -> configured fallback is used
+ assert resolve_git_human_peer(str(tmp_path), "me") == "me"
+ assert resolve_git_human_peer(None, "me") == "me"
diff --git a/tests/ingest/test_replay.py b/tests/ingest/test_replay.py
new file mode 100644
index 000000000..ddc8af53f
--- /dev/null
+++ b/tests/ingest/test_replay.py
@@ -0,0 +1,194 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Replay driver: SDK-client chunking/ensure, batch append, commit, and crash reconcile."""
+
+from openviking.ingest.cursor_store import CursorStore
+from openviking.ingest.models import BYTE_OFFSET, Cursor, NormalizedMessage, SessionRef
+from openviking.ingest.replay import ConversationReplayClient, SessionReplayer
+from openviking_cli.exceptions import NotFoundError
+
+
+class _FakeSDK:
+ """Minimal stand-in for ov.AsyncHTTPClient."""
+
+ def __init__(self, existing=()):
+ self.existing = set(existing)
+ self.created = []
+ self.batches = []
+ self.commits = []
+
+ async def get_session(self, sid, *, auto_create=False):
+ if sid not in self.existing:
+ raise NotFoundError(sid, "session")
+ return {"session_id": sid, "pending_tokens": 0, "message_count": 0}
+
+ async def create_session(self, session_id=None, memory_policy=None):
+ self.created.append(session_id)
+ self.existing.add(session_id)
+ return {"session_id": session_id}
+
+ async def batch_add_messages(self, sid, messages, telemetry=False):
+ self.batches.append((sid, len(messages)))
+ return {"added": len(messages)}
+
+ async def commit_session(self, sid, *, keep_recent_count=0):
+ self.commits.append((sid, keep_recent_count))
+ return {"task_id": "t", "archived": True}
+
+
+async def test_ensure_session_get_or_create():
+ sdk = _FakeSDK()
+ client = ConversationReplayClient(sdk)
+ await client.ensure_session("s1") # not present -> create
+ assert sdk.created == ["s1"]
+ await client.ensure_session("s1") # now present -> no second create
+ assert sdk.created == ["s1"]
+
+
+async def test_append_chunks_at_100():
+ sdk = _FakeSDK(existing={"s1"})
+ client = ConversationReplayClient(sdk)
+ added = await client.append("s1", [{"role": "user", "content": str(i)} for i in range(250)])
+ assert added == 250
+ assert [n for _, n in sdk.batches] == [100, 100, 50]
+
+
+class _FakeReplay:
+ """Stand-in for ConversationReplayClient used by SessionReplayer."""
+
+ def __init__(self, pending=0, count=0):
+ self.pending = pending
+ self.count = count
+ self.ensured = []
+ self.appended = []
+ self.committed = []
+
+ async def ensure_session(self, sid, memory_policy=None):
+ self.ensured.append(sid)
+
+ async def append(self, sid, payloads):
+ self.appended.append((sid, len(payloads)))
+ self.count += len(payloads)
+ return len(payloads)
+
+ async def message_count(self, sid):
+ return self.count
+
+ async def pending_tokens(self, sid):
+ return self.pending
+
+ async def commit(self, sid, keep_recent_count=0):
+ self.committed.append(sid)
+
+ async def delete(self, sid):
+ pass
+
+
+def _msgs():
+ return [
+ NormalizedMessage(role="user", text="hi", peer_id="me"),
+ NormalizedMessage(role="assistant", text="yo", peer_id="claude_code__m"),
+ ]
+
+
+def _ref(sid="s1"):
+ return SessionRef(harness="claude_code", native_session_id=sid, locator="/f")
+
+
+async def test_append_batch_confirms_cursor_and_marks_needs_commit(tmp_path):
+ store = CursorStore(tmp_path)
+ fake = _FakeReplay()
+ replayer = SessionReplayer(fake, store, session_id_prefix="import")
+ ref = _ref()
+ added = await replayer.append_batch(
+ "claude_code",
+ ref,
+ _msgs(),
+ Cursor(BYTE_OFFSET, {"offset": 0}),
+ Cursor(BYTE_OFFSET, {"offset": 99}),
+ )
+ assert added == 2
+ assert fake.ensured == ["import__claude_code__s1"]
+ rec = store.get("claude_code", "s1")
+ assert rec.cursor.value["offset"] == 99 # confirmed cursor
+ assert rec.needs_commit is True
+ assert fake.committed == [] # append does not commit
+
+
+async def test_reconcile_confirms_landed_batch(tmp_path):
+ store = CursorStore(tmp_path)
+ sid = "import__claude_code__s1"
+ store.set_pending(
+ "claude_code",
+ "s1",
+ sid,
+ Cursor(BYTE_OFFSET, {"offset": 0}),
+ Cursor(BYTE_OFFSET, {"offset": 50}),
+ pend_count=2,
+ baseline=0,
+ )
+ fake = _FakeReplay(count=2) # server already has the 2 messages -> batch landed
+ replayer = SessionReplayer(fake, store)
+ await replayer.reconcile("claude_code", _ref())
+ rec = store.get("claude_code", "s1")
+ assert rec.cursor.value["offset"] == 50 # confirmed without re-append
+ assert rec.pending_count == 0
+ assert rec.needs_commit is True
+
+
+async def test_reconcile_drops_unlanded_batch(tmp_path):
+ store = CursorStore(tmp_path)
+ sid = "import__claude_code__s2"
+ store.set_pending(
+ "claude_code",
+ "s2",
+ sid,
+ Cursor(BYTE_OFFSET, {"offset": 0}),
+ Cursor(BYTE_OFFSET, {"offset": 50}),
+ pend_count=2,
+ baseline=0,
+ )
+ fake = _FakeReplay(count=0) # batch did not land
+ replayer = SessionReplayer(fake, store)
+ await replayer.reconcile("claude_code", _ref("s2"))
+ rec = store.get("claude_code", "s2")
+ assert rec.cursor.value["offset"] == 0 # NOT advanced -> will re-read & re-append
+ assert rec.pending_count == 0 # intent cleared
+
+
+async def test_threshold_commit(tmp_path):
+ store = CursorStore(tmp_path)
+ ref = _ref()
+ low = SessionReplayer(_FakeReplay(pending=10), store)
+ assert await low.maybe_commit_on_threshold("claude_code", ref, threshold=6000) is False
+ assert low.client.committed == []
+
+ high = SessionReplayer(_FakeReplay(pending=9000), store)
+ assert await high.maybe_commit_on_threshold("claude_code", ref, threshold=6000) is True
+ assert high.client.committed == ["import__claude_code__s1"]
+
+
+async def test_commit_if_needed_skips_when_nothing(tmp_path):
+ store = CursorStore(tmp_path)
+ replayer = SessionReplayer(_FakeReplay(pending=0), store)
+ assert await replayer.commit_if_needed("codex", _ref("s")) is False
+ assert replayer.client.committed == []
+
+
+async def test_commit_if_needed_commits_when_needs_commit_and_pending(tmp_path):
+ store = CursorStore(tmp_path)
+ # appended-but-uncommitted: needs_commit set in store, server still has pending tokens
+ store.set_pending(
+ "claude_code",
+ "s1",
+ "import__claude_code__s1",
+ Cursor(BYTE_OFFSET, {"offset": 0}),
+ Cursor(BYTE_OFFSET, {"offset": 10}),
+ 1,
+ 0,
+ )
+ store.confirm_append("claude_code", "s1", Cursor(BYTE_OFFSET, {"offset": 10}), 1)
+ replayer = SessionReplayer(_FakeReplay(pending=500), store)
+ assert await replayer.commit_if_needed("claude_code", _ref()) is True
+ assert replayer.client.committed == ["import__claude_code__s1"]
+ assert store.get("claude_code", "s1").needs_commit is False