Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(ingest): replay local agent-harness logs into OpenViking sessions (#2892)
t0saki72d04cd488b5
diff --git a/docs/en/agent-integrations/09-log-ingestion.md b/docs/en/agent-integrations/09-log-ingestion.mdnew file mode 100644index 000000000..e6c81f8c1--- /dev/null+++ b/docs/en/agent-integrations/09-log-ingestion.md@@ -0,0 +1,117 @@+# Import Local Agent Logs (openviking-server ingest)++`openviking-server ingest` parses the conversation logs that AI coding / agent harnesses (Claude Code, Codex, OpenCode, Hermes, OpenClaw) already leave on your machine, then "replays" them through OpenViking's existing session pipeline (`create session → batch add messages → commit`, where commit triggers memory extraction). This turns both your historical and newly written conversations into long-term memory. It complements the per-harness memory plugins: a plugin captures **while a conversation is happening**, whereas this tool is for **importing existing logs** and **watching for new logs offline** — no plugin required and no change to the harness itself.++Key difference from the plugins: this tool is an OpenViking **client**. It runs where the logs live and points at a local or remote server via the SDK, and it is **off by default** — installing OpenViking does not silently scan your local files.++Source: [openviking/ingest](https://github.com/volcengine/OpenViking/tree/main/openviking/ingest)++## Off by default++The feature is doubly disabled and must be turned on explicitly:++- the master switch `ingest.enabled` defaults to `false`;+- each harness's `enabled` defaults to `false`, and harnesses you do not list are never read;+- backfill of existing logs is a manual command and supports `--dry-run` (count only, write nothing) and `--since` (bound the time window) so you can verify first.++## Supported harnesses++| harness | Status | Default log path | Notes |+|---|---|---|---|+| `claude_code` | Supported | `~/.claude/projects/*/*.jsonl` | append-only JSONL, byte-offset cursor |+| `codex` | Supported | `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` | append-only JSONL |+| `hermes` | Supported | `~/.hermes/sessions/*.jsonl` | group-chat agent; user peer = original username |+| `openclaw` | Supported | `~/.openclaw/agents/*/sessions/*.jsonl` | group-chat agent; user peer = original username |+| `opencode` | Experimental | `~/.local/share/opencode/opencode.db` | SQLite, polled by `(time, id)`; the legacy file-store is not supported |+| `cursor` | Deferred | `~/Library/Application Support/Cursor/User/**/state.vscdb` | undocumented, version-unstable KV blobs; not yet implemented |++> "harness" (agent framework) here means a whole tool like Claude Code or Codex — distinct from OpenViking's "tool" (tool-call) concept.++## Enable in ov.conf++Add an `ingest` section to `ov.conf`, listing the harnesses to import and their mode:++```json+{+ "ingest": {+ "enabled": true,+ "server_url": "$OPENVIKING_URL",+ "api_key": "$OPENVIKING_API_KEY",+ "account": "default",+ "user": "default",+ "harnesses": {+ "claude_code": { "enabled": true, "mode": "both" },+ "codex": { "enabled": true, "mode": "backfill" },+ "opencode": { "enabled": false, "mode": "watch", "experimental": true },+ "hermes": { "enabled": false, "mode": "both", "user_field": "sender" },+ "openclaw": { "enabled": false, "mode": "both", "user_field": "sender" }+ }+ }+}+```++- `mode`: `off` | `backfill` (one-shot import of existing logs) | `watch` (incremental) | `both`.+- `paths`: override the harness's default discovery roots (multiple allowed).+- `user_field`: for group-chat harnesses, the log key holding the original username, used as the user-side peer_id.+- `commit`: commit policy — `commit_token_threshold`, `commit_idle_seconds`, `keep_recent_count`.+- Deploy-time toggles can also be overridden via env: `OPENVIKING_INGEST_ENABLED`, `OPENVIKING_INGEST_SERVER_URL`, `OPENVIKING_INGEST_API_KEY`.++When `server_url` is empty it falls back to `OPENVIKING_URL` or `http://localhost:1933`, so it can target either a local or a remote server.++## Usage++The `openviking-server ingest` command is installed together with OpenViking.++```bash+# Show registered harnesses and their config+openviking-server ingest list-sources++# Dry run first: count the sessions / messages that would be replayed, write nothing+openviking-server ingest backfill --dry-run++# Backfill one harness, only sessions started on/after a date+openviking-server ingest backfill --harness claude_code --since 2026-06-01++# Backfill existing logs for real+openviking-server ingest backfill++# Watch for new logs and replay incrementally (blocks in the foreground)+openviking-server ingest watch --harness claude_code++# Honor each harness's configured mode: backfill then watch+openviking-server ingest run++# Show how far each session has been ingested (read the cursor state)+openviking-server ingest status+```++`--reset` deletes and recreates the OV session before replaying. Without `--reset`, re-running is idempotent — the cursor store guarantees nothing is appended twice.++## peer_id++Every message carries a peer_id so OpenViking can profile both the human and the model:++- assistant turns: `{harness}/{model}` (or `{harness}/{provider}/{model}` when the provider is meaningful), e.g. `claude_code/claude-opus-4-8`, `opencode/bytedance_ark/doubao-...`;+- user turns: single-user dev harnesses (claude_code / codex / opencode) use the git identity of the session cwd repo (`user.email` / `user.name`), falling back to the configured `ingest.user` when there is no git repo; group-chat harnesses (hermes / openclaw) use the original username from the log (selected by `user_field`).++Non-ASCII identifiers (e.g. a CJK username) fall back to a valid `ext-<base64>` form.++## How it works++Each harness has a thin adapter that parses its logs into normalized messages and hands them to a replayer that runs `ensure_session → batch add (<=100 per call) → commit`. Memory extraction only runs on **commit**, server-side. OV session ids are `import__{harness}__{native_session_id}` — deterministic and idempotent.++- **Backfill** enumerates every session, replays from the cursor to the end, then commits once per session.+- **Watch** mirrors OpenViking's own `WatchScheduler`: it uses **interval polling** (not filesystem events) driven by durable cursors, so a missed tick, a sleep, or a restart just reads cursor→end on the next tick and self-heals. JSONL uses a byte-offset cursor (with partial-line / truncation / rotation handling); SQLite uses a `(time, id)` cursor read read-only (WAL-aware).++Cursor state persists in `~/.openviking/ingest/state.db`, so both backfill and watch resume across restarts without re-ingesting.++## Cost and privacy++- Commit triggers memory extraction (an LLM call). Backfilling months of history at once can produce many calls — prefer `--dry-run` first, narrow with `--since`, and enable harnesses in batches.+- Logs may contain sensitive content (credentials, file contents). Use this in a trusted deployment and confirm `server_url` points at the server you intend.+- Tool-call inputs/outputs are dropped as low-value by default; only user / assistant text is ingested.++## See also++- [Overview](./01-overview.md) — the per-harness memory plugins (real-time capture)+- [Deployment guide → CLI](../guides/03-deployment.md#cli) — `ov.conf` / credential configurationdiff --git a/docs/zh/agent-integrations/09-log-ingestion.md b/docs/zh/agent-integrations/09-log-ingestion.mdnew file mode 100644index 000000000..135b8d9a3--- /dev/null+++ b/docs/zh/agent-integrations/09-log-ingestion.md@@ -0,0 +1,117 @@+# 导入本地 Agent 日志(openviking-server ingest)++`openviking-server ingest` 把你本地已有的 AI 编码 / agent harness 的对话日志(Claude Code、Codex、OpenCode、Hermes、OpenClaw)解析成标准消息,再通过 OpenViking 既有的会话管线“重放”进去(`创建会话 → 批量追加消息 → 提交`,提交时触发记忆抽取),从而把这些历史与新增对话沉淀为长期记忆。它与各 harness 的“记忆插件”互补:插件在对话**进行时**实时挂载捕获,而本工具用于**导入既有日志**与**离线监听新增日志**,无需插件、也无需改动对应 harness。++与插件方案的关键区别:本工具是 OpenViking 的**客户端**,跑在日志所在的机器上,通过 SDK 指向本地或远端 server;它默认**完全关闭**,不会“装上就扫你本地文件”。++源码:[openviking/ingest](https://github.com/volcengine/OpenViking/tree/main/openviking/ingest)++## 默认关闭++该特性默认双重关闭,必须显式开启:++- 总开关 `ingest.enabled` 默认 `false`;+- 每个 harness 的 `enabled` 默认 `false`,未列出的 harness 不会被读取;+- 存量回填需手动运行命令,且支持 `--dry-run`(只统计、不写入)与 `--since`(限定时间窗)先行验证。++## 支持的 harness++| harness | 状态 | 默认日志路径 | 说明 |+|---|---|---|---|+| `claude_code` | 支持 | `~/.claude/projects/*/*.jsonl` | append-only JSONL,字节偏移游标 |+| `codex` | 支持 | `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` | append-only JSONL |+| `hermes` | 支持 | `~/.hermes/sessions/*.jsonl` | 群聊 agent,user 取原始用户名 |+| `openclaw` | 支持 | `~/.openclaw/agents/*/sessions/*.jsonl` | 群聊 agent,user 取原始用户名 |+| `opencode` | 实验性 | `~/.local/share/opencode/opencode.db` | SQLite,按 `(time, id)` 轮询;旧版文件存储暂不支持 |+| `cursor` | 暂缓 | `~/Library/Application Support/Cursor/User/**/state.vscdb` | 无文档、随版本漂移的 KV blob,暂未实现 |++> 这里的 harness(agent 框架)指 CC / Codex 等整套工具,区别于 OpenViking 里“tool(工具调用)”的概念。++## 在 ov.conf 中开启++在 `ov.conf` 增加 `ingest` 段,列出要导入的 harness 并设置其模式:++```json+{+ "ingest": {+ "enabled": true,+ "server_url": "$OPENVIKING_URL",+ "api_key": "$OPENVIKING_API_KEY",+ "account": "default",+ "user": "default",+ "harnesses": {+ "claude_code": { "enabled": true, "mode": "both" },+ "codex": { "enabled": true, "mode": "backfill" },+ "opencode": { "enabled": false, "mode": "watch", "experimental": true },+ "hermes": { "enabled": false, "mode": "both", "user_field": "sender" },+ "openclaw": { "enabled": false, "mode": "both", "user_field": "sender" }+ }+ }+}+```++- `mode`:`off` | `backfill`(一次性导入存量)| `watch`(监听新增)| `both`。+- `paths`:覆盖该 harness 的默认发现路径(可填多个)。+- `user_field`:群聊 harness 中存放原始用户名的字段名,用作 user 侧 peer_id。+- `commit`:提交策略,含 `commit_token_threshold`、`commit_idle_seconds`、`keep_recent_count`。+- 部署期开关也可用环境变量覆盖:`OPENVIKING_INGEST_ENABLED`、`OPENVIKING_INGEST_SERVER_URL`、`OPENVIKING_INGEST_API_KEY`。++`server_url` 留空时回退到 `OPENVIKING_URL` 或 `http://localhost:1933`,因此既能指向本地 server,也能指向远端。++## 使用++`openviking-server ingest` 命令随 OpenViking 一同安装。++```bash+# 查看已注册 harness 及其配置+openviking-server ingest list-sources++# 先干跑:统计会回填多少 session / 消息,不写入+openviking-server ingest backfill --dry-run++# 只回填某个 harness、且只回填某日期之后的会话+openviking-server ingest backfill --harness claude_code --since 2026-06-01++# 正式回填(存量)+openviking-server ingest backfill++# 监听新增日志并增量重放(前台阻塞)+openviking-server ingest watch --harness claude_code++# 按每个 harness 配置的 mode 执行:先回填再监听+openviking-server ingest run++# 查看各会话已导入到哪里(读取游标状态)+openviking-server ingest status+```++`--reset` 会在重放前删除并重建对应的 OV 会话;不加 `--reset` 时,重复运行是幂等的(游标保证不会重复追加)。++## peer_id++每条消息都会带上 peer_id,便于 OpenViking 同时为人类与模型建立画像:++- assistant 消息:`{harness}/{模型名}`(provider 有意义时为 `{harness}/{provider}/{模型名}`),例如 `claude_code/claude-opus-4-8`、`opencode/bytedance_ark/doubao-...`;+- user 消息:单用户开发型 harness(claude_code / codex / opencode)取会话 cwd 所在仓库的 git 身份(`user.email` / `user.name`),无 git 仓库时回退为配置的 `ingest.user`;群聊 harness(hermes / openclaw)取日志里的原始用户名(由 `user_field` 指定)。++非 ASCII 标识(如中文用户名)会回退为合法的 `ext-<base64>` 形式。++## 工作原理++每个 harness 对应一个轻量适配器,把其日志解析为标准消息,交给“重放器”执行 `ensure_session → 批量追加(每批 ≤100)→ commit`。记忆抽取只在 **commit** 时由 server 端触发。OV 会话 id 形如 `import__{harness}__{原始会话id}`,确定且幂等。++- **存量回填**:枚举所有会话,从游标读到末尾后逐会话提交一次。+- **监听增量**:参照 OpenViking 自身的 `WatchScheduler`,用**定时轮询**(非文件系统事件)+ 持久游标驱动;漏一拍、休眠或重启后,下一拍从游标读到末尾即可自愈。JSONL 用字节偏移游标(含半行/截断/轮转处理),SQLite 用 `(time, id)` 游标只读读取(兼容 WAL)。++游标状态持久化在 `~/.openviking/ingest/state.db`,因此回填与监听都能在重启后续传,且不会重复入库。++## 成本与隐私++- 提交会触发记忆抽取(LLM 调用)。一次性回填数月历史可能产生大量调用,建议先 `--dry-run`、用 `--since` 收窄时间窗、按 harness 分批开启。+- 日志中可能含敏感内容(凭据、文件内容)。请在受信任的部署中使用,并确认 `server_url` 指向你期望的 server。+- tool 调用的输入/输出默认按低价值丢弃,仅入库 user / assistant 文本。++## 参见++- [概览](./01-overview.md) — 各 harness 的记忆插件(实时捕获方案)+- [部署指南 → CLI](../guides/03-deployment.md#cli) — `ov.conf` / 凭据配置diff --git a/examples/ov.conf.example b/examples/ov.conf.exampleindex 01294ecd4..eaf98494d 100644--- a/examples/ov.conf.example+++ b/examples/ov.conf.example@@ -162,6 +162,20 @@"extraction_enabled": true,"session_skill_extraction_enabled": false,},+ "ingest": {+ "enabled": false,+ "server_url": "$OPENVIKING_URL",+ "api_key": "$OPENVIKING_API_KEY",+ "account": "default",+ "user": "default",+ "harnesses": {+ "claude_code": {"enabled": false, "mode": "both"},+ "codex": {"enabled": false, "mode": "backfill"},+ "opencode": {"enabled": false, "mode": "watch", "experimental": true},+ "hermes": {"enabled": false, "mode": "both", "user_field": "sender"},+ "openclaw": {"enabled": false, "mode": "both", "user_field": "sender"},+ },+ },"log": {"level": "INFO","format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s",diff --git a/openviking/ingest/__init__.py b/openviking/ingest/__init__.pynew file mode 100644index 000000000..dfad50acc--- /dev/null+++ b/openviking/ingest/__init__.py@@ -0,0 +1,23 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Conversation-log ingest: replay local agent-harness logs into OpenViking sessions.++Parses each harness's local conversation logs (Claude Code, Codex, OpenCode, Hermes,+OpenClaw, Cursor) into normalized messages and replays them through OV's existing+``create_session -> batch_add_messages -> commit`` pipeline (commit triggers OV's+async memory extraction). Supports one-shot backfill of existing logs and+incremental, cursor-driven polling of new logs.++Inspired by / supersedes volcengine/OpenViking#2674 by @huang-yi-dae.+"""++from openviking.ingest.models import Cursor, NormalizedMessage, SessionRef+from openviking.ingest.registry import SOURCE_REGISTRY, register_source++__all__ = [+ "Cursor",+ "NormalizedMessage",+ "SessionRef",+ "SOURCE_REGISTRY",+ "register_source",+]diff --git a/openviking/ingest/cli.py b/openviking/ingest/cli.pynew file mode 100644index 000000000..2941bec73--- /dev/null+++ b/openviking/ingest/cli.py@@ -0,0 +1,265 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""``openviking-server ingest`` CLI: replay local agent-harness logs into OpenViking.++Commands:+ list-sources show registered harnesses and their config+ status show per-session ingest progress (read cursors)+ backfill one-shot replay of existing logs+ watch incremental, cursor-driven polling of new logs+ run honor each harness's configured mode (backfill then watch)+"""++from __future__ import annotations++import asyncio+import signal+from pathlib import Path+from typing import List, Optional, Tuple++import typer++from openviking.ingest.cursor_store import CursorStore, SingleInstanceLock+from openviking.ingest.orchestrator import BackfillStats, IngestOrchestrator, enabled_sources+from openviking.ingest.poller import IngestPoller+from openviking.ingest.registry import SOURCE_REGISTRY+from openviking.ingest.replay import ConversationReplayClient, SessionReplayer+from openviking_cli.utils import get_logger+from openviking_cli.utils.config.consts import DEFAULT_INGEST_STATE_DIR+from openviking_cli.utils.config.ingest_config import IngestConfig++logger = get_logger(__name__)++app = typer.Typer(+ add_completion=False,+ help="Replay local agent-harness conversation logs into OpenViking.",+)+++def _load_ingest_config() -> IngestConfig:+ """Read the ``ingest`` section of ov.conf if present; else a default (env-driven).++ Only a *missing* config falls back to defaults; a malformed config surfaces its+ validation/JSON error instead of silently looking like "ingest not configured".+ """+ from openviking_cli.utils.config.open_viking_config import OpenVikingConfigSingleton++ try:+ return OpenVikingConfigSingleton.get_instance().ingest+ except FileNotFoundError:+ return IngestConfig()+++def _state_dir(config: IngestConfig) -> Path:+ return Path(config.state_dir).expanduser() if config.state_dir else DEFAULT_INGEST_STATE_DIR+++def _make_store(config: IngestConfig) -> CursorStore:+ return CursorStore(_state_dir(config))+++def _acquire_lock(config: IngestConfig) -> SingleInstanceLock:+ try:+ return SingleInstanceLock(_state_dir(config)).acquire()+ except RuntimeError as exc:+ typer.echo(f"error: {exc}")+ raise typer.Exit(1) from exc+++async def _make_replayer(+ config: IngestConfig, store: CursorStore+) -> Tuple[ConversationReplayClient, SessionReplayer]:+ client = await ConversationReplayClient.create(+ url=config.server_url,+ api_key=config.api_key,+ account=config.account,+ user=config.user,+ )+ replayer = SessionReplayer(+ client,+ store,+ session_id_prefix=config.session_id_prefix,+ memory_policy=config.memory_policy or None,+ )+ return client, replayer+++def _print_stats(results: dict[str, BackfillStats], dry_run: bool) -> None:+ label = "WOULD replay" if dry_run else "Replayed"+ total = BackfillStats()+ for name, s in results.items():+ typer.echo(+ f" {name:12s}: {s.sessions} sessions, {label.lower()} {s.messages} messages, "+ f"{s.committed} committed, {s.skipped} skipped"+ )+ for err in s.errors:+ typer.echo(f" ! {err}")+ total.merge(s)+ typer.echo(+ f"{label}: {total.sessions} sessions / {total.messages} messages / "+ f"{total.committed} commits" + (f" / {len(total.errors)} errors" if total.errors else "")+ )+++# --------------------------------------------------------------------------- #[email protected]("list-sources")+def list_sources() -> None:+ """List registered harness adapters and their current config."""+ import openviking.ingest.sources # noqa: F401 - populate the registry++ config = _load_ingest_config()+ typer.echo(f"ingest enabled: {config.enabled} server_url: {config.server_url or '(default)'}")+ typer.echo("harnesses:")+ for name in sorted(SOURCE_REGISTRY):+ hc = config.harnesses.get(name)+ if hc is None:+ typer.echo(f" {name:12s} (not configured)")+ else:+ typer.echo(+ f" {name:12s} enabled={hc.enabled} mode={hc.mode} paths={hc.paths or '(default)'}"+ )++[email protected]("status")+def status(+ harness: Optional[List[str]] = typer.Option(None, "--harness", "-H", help="Filter by harness"),+) -> None:+ """Show per-session ingest progress recorded in the cursor store."""+ config = _load_ingest_config()+ store = _make_store(config)+ try:+ names = harness or None+ records = []+ for h in names or sorted({r.harness for r in store.all_records()}):+ records.extend(store.all_records(h))+ if not records:+ typer.echo("No sessions ingested yet.")+ return+ typer.echo(f"{'harness':12s} {'session':24s} {'msgs':>6s} committed_at")+ for rec in records:+ typer.echo(+ f"{rec.harness:12s} {rec.native_session_id[:24]:24s} "+ f"{rec.last_appended_count:6d} {rec.last_committed_at or '-'}"+ )+ finally:+ store.close()+++def backfill(+ harness: Optional[List[str]] = typer.Option(+ None, "--harness", "-H", help="Only these harnesses (default: all enabled)"+ ),+ since: Optional[str] = typer.Option(+ None, "--since", help="Skip sessions started before this ISO date (e.g. 2026-06-01)"+ ),+ dry_run: bool = typer.Option(False, "--dry-run", help="Count sessions/messages; write nothing"),+ reset: bool = typer.Option(+ False, "--reset", help="Delete + recreate each OV session before replaying"+ ),+) -> None:+ """One-shot backfill of existing logs."""+ asyncio.run(_run_backfill(harness, since, dry_run, reset))+++async def _run_backfill(harness, since, dry_run, reset) -> None:+ config = _load_ingest_config()+ store = _make_store(config)+ client = None+ lock = None+ try:+ only = harness or None+ if dry_run:+ # No server (and no lock) needed for a read-only dry run.+ replayer = SessionReplayer(None, store, session_id_prefix=config.session_id_prefix) # type: ignore[arg-type]+ orch = IngestOrchestrator(config, replayer)+ results = await orch.backfill(only=only, since=since, dry_run=True)+ else:+ lock = _acquire_lock(config)+ client, replayer = await _make_replayer(config, store)+ orch = IngestOrchestrator(config, replayer)+ results = await orch.backfill(only=only, since=since, dry_run=False, reset=reset)+ _print_stats(results, dry_run)+ finally:+ if client is not None:+ await client.close()+ if lock is not None:+ lock.release()+ store.close()+++def watch(+ harness: Optional[List[str]] = typer.Option(+ None, "--harness", "-H", help="Only these harnesses (default: all enabled)"+ ),+) -> None:+ """Incrementally watch for new/changed logs and replay them."""+ asyncio.run(_run_watch(harness))+++def run() -> None:+ """Honor each harness's configured mode: backfill (mode in backfill/both) then watch."""+ asyncio.run(_run_all())+++async def _run_all() -> None:+ config = _load_ingest_config()+ store = _make_store(config)+ lock = _acquire_lock(config)+ client, replayer = await _make_replayer(config, store)+ try:+ orch = IngestOrchestrator(config, replayer)+ results = await orch.backfill(only=None, since=None, dry_run=False)+ _print_stats(results, dry_run=False)+ watch_sources = [+ (n, c, s) for n, c, s in enabled_sources(config) if c.mode in ("watch", "both")+ ]+ if watch_sources:+ await _watch_loop(replayer, watch_sources)+ finally:+ await client.close()+ lock.release()+ store.close()+++async def _run_watch(harness) -> None:+ config = _load_ingest_config()+ store = _make_store(config)+ lock = _acquire_lock(config)+ client, replayer = await _make_replayer(config, store)+ try:+ only = harness or None+ sources = [+ (n, c, s)+ for n, c, s in enabled_sources(config, only)+ if c.mode in ("watch", "both") or only is not None+ ]+ if not sources:+ typer.echo("No harnesses enabled for watch.")+ return+ await _watch_loop(replayer, sources)+ finally:+ await client.close()+ lock.release()+ store.close()+++async def _watch_loop(replayer, sources) -> None:+ poller = IngestPoller(sources, replayer)+ loop = asyncio.get_running_loop()+ for sig in (signal.SIGINT, signal.SIGTERM):+ try:+ loop.add_signal_handler(sig, poller.stop)+ except (NotImplementedError, RuntimeError):+ pass # not supported on this platform+ await poller.run()+++def main() -> None:+ app()+++if __name__ == "__main__":+ main()diff --git a/openviking/ingest/cursor_store.py b/openviking/ingest/cursor_store.pynew file mode 100644index 000000000..40862786b--- /dev/null+++ b/openviking/ingest/cursor_store.py@@ -0,0 +1,365 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Durable per-(harness, session) read-cursor + commit/idempotency state.++A single SQLite DB under ``~/.openviking/ingest/state.db`` records how far each+conversation has been ingested, plus:+- ``needs_commit``: appended-but-not-yet-committed, so a crash/error between append and+ commit still gets its memory extraction on the next pass (not stranded forever);+- a ``pending`` batch intent (cursor + count + server-message baseline) written BEFORE a+ batch is appended, so a crash mid-append is reconciled against the server's message+ count on restart instead of blindly re-appending (duplicate-safe).++A ``SingleInstanceLock`` prevents two ingest processes from racing on the same state.++(This is the read-position POINTER store; unrelated to the Cursor IDE harness.)+"""++from __future__ import annotations++import os+import sqlite3+from dataclasses import dataclass+from datetime import datetime, timezone+from pathlib import Path+from typing import Optional++from openviking.ingest.models import Cursor+from openviking_cli.utils import get_logger++logger = get_logger(__name__)++_SCHEMA = """+CREATE TABLE IF NOT EXISTS ingest_cursor (+ harness TEXT NOT NULL,+ native_session_id TEXT NOT NULL,+ ov_session_id TEXT,+ cursor_kind TEXT NOT NULL,+ cursor_value TEXT NOT NULL,+ locator TEXT,+ title TEXT,+ last_appended_count INTEGER NOT NULL DEFAULT 0,+ pending_tokens INTEGER NOT NULL DEFAULT 0,+ needs_commit INTEGER NOT NULL DEFAULT 0,+ pend_cursor TEXT,+ pend_count INTEGER NOT NULL DEFAULT 0,+ pend_baseline INTEGER NOT NULL DEFAULT 0,+ last_committed_at TEXT,+ updated_at TEXT,+ PRIMARY KEY (harness, native_session_id)+);+"""++_MIGRATION_COLUMNS = {+ "needs_commit": "INTEGER NOT NULL DEFAULT 0",+ "pend_cursor": "TEXT",+ "pend_count": "INTEGER NOT NULL DEFAULT 0",+ "pend_baseline": "INTEGER NOT NULL DEFAULT 0",+}+++@dataclass+class CursorRecord:+ harness: str+ native_session_id: str+ ov_session_id: Optional[str]+ cursor: Cursor+ locator: Optional[str]+ title: Optional[str]+ last_appended_count: int+ pending_tokens: int+ needs_commit: bool+ pending_cursor: Optional[Cursor]+ pending_count: int+ pending_baseline: int+ last_committed_at: Optional[str]+++def _now_iso() -> str:+ return datetime.now(timezone.utc).isoformat()+++class CursorStore:+ def __init__(self, state_dir: Path):+ self.state_dir = Path(state_dir).expanduser()+ self.state_dir.mkdir(parents=True, exist_ok=True)+ self.db_path = self.state_dir / "state.db"+ self._conn = sqlite3.connect(str(self.db_path))+ self._conn.row_factory = sqlite3.Row+ self._conn.execute("PRAGMA journal_mode=WAL")+ self._conn.executescript(_SCHEMA)+ self._migrate()+ self._conn.commit()++ def _migrate(self) -> None:+ existing = {r["name"] for r in self._conn.execute("PRAGMA table_info(ingest_cursor)")}+ for col, decl in _MIGRATION_COLUMNS.items():+ if col not in existing:+ self._conn.execute(f"ALTER TABLE ingest_cursor ADD COLUMN {col} {decl}")++ def close(self) -> None:+ self._conn.close()++ # --- reads ------------------------------------------------------------+ def get(self, harness: str, native_session_id: str) -> Optional[CursorRecord]:+ row = self._conn.execute(+ "SELECT * FROM ingest_cursor WHERE harness = ? AND native_session_id = ?",+ (harness, native_session_id),+ ).fetchone()+ if row is None:+ return None+ pend = (+ Cursor.from_json(row["cursor_kind"], row["pend_cursor"]) if row["pend_cursor"] else None+ )+ return CursorRecord(+ harness=row["harness"],+ native_session_id=row["native_session_id"],+ ov_session_id=row["ov_session_id"],+ cursor=Cursor.from_json(row["cursor_kind"], row["cursor_value"]),+ locator=row["locator"],+ title=row["title"],+ last_appended_count=row["last_appended_count"],+ pending_tokens=row["pending_tokens"],+ needs_commit=bool(row["needs_commit"]),+ pending_cursor=pend,+ pending_count=row["pend_count"],+ pending_baseline=row["pend_baseline"],+ last_committed_at=row["last_committed_at"],+ )++ def get_cursor(self, harness: str, native_session_id: str, kind: str) -> Optional[Cursor]:+ rec = self.get(harness, native_session_id)+ return rec.cursor if rec else None++ # --- writes -----------------------------------------------------------+ def ensure_row(+ self,+ harness: str,+ native_session_id: str,+ ov_session_id: str,+ cursor: Cursor,+ *,+ locator: Optional[str] = None,+ title: Optional[str] = None,+ ) -> None:+ """Insert a zero/initial row if absent (idempotent)."""+ self._conn.execute(+ """INSERT OR IGNORE INTO ingest_cursor+ (harness, native_session_id, ov_session_id, cursor_kind, cursor_value,+ locator, title, updated_at)+ VALUES (?, ?, ?, ?, ?, ?, ?, ?)""",+ (+ harness,+ native_session_id,+ ov_session_id,+ cursor.kind,+ cursor.to_json(),+ locator,+ title,+ _now_iso(),+ ),+ )+ self._conn.commit()++ def advance_cursor(+ self,+ harness: str,+ native_session_id: str,+ ov_session_id: str,+ cursor: Cursor,+ *,+ locator: Optional[str] = None,+ ) -> None:+ """Advance the confirmed cursor only (e.g. a read that yielded no messages)."""+ self.ensure_row(harness, native_session_id, ov_session_id, cursor, locator=locator)+ self._conn.execute(+ "UPDATE ingest_cursor SET cursor_value = ?, cursor_kind = ?, "+ "locator = COALESCE(?, locator), updated_at = ? "+ "WHERE harness = ? AND native_session_id = ?",+ (cursor.to_json(), cursor.kind, locator, _now_iso(), harness, native_session_id),+ )+ self._conn.commit()++ def set_pending(+ self,+ harness: str,+ native_session_id: str,+ ov_session_id: str,+ from_cursor: Cursor,+ pend_cursor: Cursor,+ pend_count: int,+ baseline: int,+ *,+ locator: Optional[str] = None,+ title: Optional[str] = None,+ ) -> None:+ """Durably record a batch intent BEFORE appending it (crash reconciliation)."""+ self.ensure_row(+ harness, native_session_id, ov_session_id, from_cursor, locator=locator, title=title+ )+ self._conn.execute(+ "UPDATE ingest_cursor SET ov_session_id = ?, pend_cursor = ?, pend_count = ?, "+ "pend_baseline = ?, locator = COALESCE(?, locator), title = COALESCE(?, title), "+ "updated_at = ? WHERE harness = ? AND native_session_id = ?",+ (+ ov_session_id,+ pend_cursor.to_json(),+ pend_count,+ baseline,+ locator,+ title,+ _now_iso(),+ harness,+ native_session_id,+ ),+ )+ self._conn.commit()++ def confirm_append(+ self, harness: str, native_session_id: str, cursor: Cursor, appended_delta: int+ ) -> None:+ """Confirm a landed batch: advance cursor, accrue count, mark needs_commit, clear pending."""+ self._conn.execute(+ "UPDATE ingest_cursor SET cursor_value = ?, cursor_kind = ?, "+ "last_appended_count = last_appended_count + ?, needs_commit = 1, "+ "pend_cursor = NULL, pend_count = 0, pend_baseline = 0, updated_at = ? "+ "WHERE harness = ? AND native_session_id = ?",+ (cursor.to_json(), cursor.kind, appended_delta, _now_iso(), harness, native_session_id),+ )+ self._conn.commit()++ def clear_pending(self, harness: str, native_session_id: str) -> None:+ self._conn.execute(+ "UPDATE ingest_cursor SET pend_cursor = NULL, pend_count = 0, pend_baseline = 0, "+ "updated_at = ? WHERE harness = ? AND native_session_id = ?",+ (_now_iso(), harness, native_session_id),+ )+ self._conn.commit()++ def mark_committed(+ self, harness: str, native_session_id: str, *, pending_tokens: int = 0+ ) -> None:+ self._conn.execute(+ "UPDATE ingest_cursor SET needs_commit = 0, pending_tokens = ?, "+ "last_committed_at = ?, updated_at = ? WHERE harness = ? AND native_session_id = ?",+ (pending_tokens, _now_iso(), _now_iso(), harness, native_session_id),+ )+ self._conn.commit()++ def upsert(+ self,+ harness: str,+ native_session_id: str,+ ov_session_id: str,+ cursor: Cursor,+ *,+ appended_delta: int = 0,+ pending_tokens: Optional[int] = None,+ committed: bool = False,+ locator: Optional[str] = None,+ title: Optional[str] = None,+ ) -> None:+ """General cursor upsert (used by tests and simple cursor advances)."""+ now = _now_iso()+ committed_at = now if committed else None+ self._conn.execute(+ """INSERT INTO ingest_cursor+ (harness, native_session_id, ov_session_id, cursor_kind, cursor_value,+ locator, title, last_appended_count, pending_tokens, last_committed_at, updated_at)+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)+ ON CONFLICT(harness, native_session_id) DO UPDATE SET+ ov_session_id = excluded.ov_session_id,+ cursor_kind = excluded.cursor_kind, cursor_value = excluded.cursor_value,+ locator = COALESCE(excluded.locator, ingest_cursor.locator),+ title = COALESCE(excluded.title, ingest_cursor.title),+ last_appended_count = ingest_cursor.last_appended_count + ?,+ pending_tokens = COALESCE(?, ingest_cursor.pending_tokens),+ last_committed_at = COALESCE(?, ingest_cursor.last_committed_at),+ updated_at = excluded.updated_at""",+ (+ harness,+ native_session_id,+ ov_session_id,+ cursor.kind,+ cursor.to_json(),+ locator,+ title,+ appended_delta,+ pending_tokens if pending_tokens is not None else 0,+ committed_at,+ now,+ appended_delta,+ pending_tokens,+ committed_at,+ ),+ )+ self._conn.commit()++ def delete(self, harness: str, native_session_id: str) -> None:+ self._conn.execute(+ "DELETE FROM ingest_cursor WHERE harness = ? AND native_session_id = ?",+ (harness, native_session_id),+ )+ self._conn.commit()++ def all_records(self, harness: Optional[str] = None) -> list[CursorRecord]:+ if harness:+ rows = self._conn.execute(+ "SELECT harness, native_session_id FROM ingest_cursor WHERE harness = ?",+ (harness,),+ ).fetchall()+ else:+ rows = self._conn.execute(+ "SELECT harness, native_session_id FROM ingest_cursor"+ ).fetchall()+ out = []+ for r in rows:+ rec = self.get(r["harness"], r["native_session_id"])+ if rec:+ out.append(rec)+ return out+++class SingleInstanceLock:+ """Best-effort exclusive lock so two ingest processes don't race on one state dir."""++ def __init__(self, state_dir: Path):+ self.path = Path(state_dir).expanduser() / "ingest.lock"+ self._fh = None++ def acquire(self) -> "SingleInstanceLock":+ try:+ import fcntl+ except ImportError: # pragma: no cover - non-POSIX+ logger.warning("[ingest] fcntl unavailable; single-instance lock disabled")+ return self+ self.path.parent.mkdir(parents=True, exist_ok=True)+ self._fh = open(self.path, "w")+ try:+ fcntl.flock(self._fh, fcntl.LOCK_EX | fcntl.LOCK_NB)+ except OSError as exc:+ self._fh.close()+ self._fh = None+ raise RuntimeError(f"another ingest process already holds {self.path}") from exc+ self._fh.write(str(os.getpid()))+ self._fh.flush()+ return self++ def release(self) -> None:+ if self._fh is None:+ return+ try:+ import fcntl++ fcntl.flock(self._fh, fcntl.LOCK_UN)+ except (ImportError, OSError):+ pass+ finally:+ self._fh.close()+ self._fh = None++ def __enter__(self) -> "SingleInstanceLock":+ return self.acquire()++ def __exit__(self, *exc) -> None:+ self.release()diff --git a/openviking/ingest/models.py b/openviking/ingest/models.pynew file mode 100644index 000000000..0d1b41f05--- /dev/null+++ b/openviking/ingest/models.py@@ -0,0 +1,95 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Core data structures shared across the ingest subsystem.++Note on "cursor": throughout this package, ``Cursor`` / ``cursor_store`` / ``cursor_kind``+refer to the read-position POINTER (how far we have ingested a given log), NOT the Cursor+IDE harness (whose adapter is ``sources/cursor.py``).+"""++from __future__ import annotations++import json+from dataclasses import dataclass, field+from datetime import datetime, timezone+from typing import Any, Dict, List, Optional+++def iso_from_epoch_ms(value: Any) -> Optional[str]:+ """Best-effort ISO-8601 from an epoch-millisecond integer (used by SQLite/JSONL adapters)."""+ try:+ ms = int(value)+ except (TypeError, ValueError):+ return None+ if ms <= 0:+ return None+ # Heuristic: 10-digit values are seconds, 13-digit are milliseconds.+ seconds = ms / 1000.0 if ms > 10_000_000_000 else float(ms)+ try:+ return datetime.fromtimestamp(seconds, tz=timezone.utc).isoformat()+ except (OverflowError, OSError, ValueError):+ return None+++# Cursor kinds.+BYTE_OFFSET = "byte_offset" # append-only JSONL: byte offset of last consumed line+ROWID_TIME = "rowid_time" # SQLite: (time_created, id) of last consumed row+++@dataclass+class NormalizedMessage:+ """A single conversation turn, harness-agnostic, ready for OV replay.++ ``peer_id`` is resolved by the adapter (see ``peer.py``): assistant turns ->+ ``{harness}/{model}``; user turns -> a human identifier (git identity for+ single-user dev harnesses, original username for group-chat harnesses).+ """++ role: str # "user" | "assistant"+ text: str = ""+ parts: List[Dict[str, Any]] = field(default_factory=list) # extra tool/context parts+ created_at: Optional[str] = None # ISO-8601+ peer_id: Optional[str] = None+ meta: Dict[str, Any] = field(default_factory=dict) # model, provider, cwd, …+++@dataclass+class SessionRef:+ """A discoverable conversation in a harness's storage."""++ harness: str # registry name, e.g. "claude_code"+ native_session_id: str # the harness's own session/conversation id+ locator: str # file path (JSONL) or db session id (SQLite) used to read+ title: Optional[str] = None+ started_at: Optional[str] = None+ meta: Dict[str, Any] = field(default_factory=dict) # session-level model, cwd, platform+++@dataclass+class Cursor:+ """A durable read-position pointer for one (harness, session)."""++ kind: str # BYTE_OFFSET | ROWID_TIME+ value: Dict[str, Any] # {"offset": int} | {"time": int|float, "id": str}++ @classmethod+ def zero(cls, kind: str) -> "Cursor":+ if kind == BYTE_OFFSET:+ return cls(kind=kind, value={"offset": 0})+ return cls(kind=kind, value={"time": 0, "id": ""})++ @property+ def offset(self) -> int:+ return int(self.value.get("offset", 0))++ def to_json(self) -> str:+ return json.dumps(self.value, separators=(",", ":"))++ @classmethod+ def from_json(cls, kind: str, raw: Optional[str]) -> "Cursor":+ if not raw:+ return cls.zero(kind)+ try:+ return cls(kind=kind, value=json.loads(raw))+ except (ValueError, TypeError):+ return cls.zero(kind)diff --git a/openviking/ingest/normalize.py b/openviking/ingest/normalize.pynew file mode 100644index 000000000..0792db8be--- /dev/null+++ b/openviking/ingest/normalize.py@@ -0,0 +1,52 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Convert ``NormalizedMessage`` -> OV ``AddMessageRequest`` payload dict.++Mirrors vikingbot's ``_normalize_session_messages`` shape (text/tool parts, peer_id+safe-naming). Conversation memory is driven by user/assistant TEXT; tool I/O is treated+as low-value and dropped by the adapters, so most messages carry a single text part.+"""++from __future__ import annotations++from typing import Any, Dict, List, Optional++from openviking.core.peer_id import safe_peer_id+from openviking.ingest.models import NormalizedMessage+++def to_add_message_request(msg: NormalizedMessage) -> Optional[Dict[str, Any]]:+ """Return an ``AddMessageRequest`` dict, or ``None`` for an empty turn."""+ role = "user" if msg.role == "user" else "assistant"++ parts: List[Dict[str, Any]] = []+ text = (msg.text or "").strip()+ if text:+ parts.append({"type": "text", "text": text})+ if msg.parts:+ parts.extend(msg.parts)++ if not parts:+ return None # nothing worth replaying++ payload: Dict[str, Any] = {"role": role, "parts": parts}+ if text:+ payload["content"] = text+ if msg.created_at:+ payload["created_at"] = msg.created_at++ pid = safe_peer_id(msg.peer_id)+ if pid:+ payload["peer_id"] = pid++ return payload+++def to_add_message_requests(messages: List[NormalizedMessage]) -> List[Dict[str, Any]]:+ """Batch convert, dropping empty turns."""+ out: List[Dict[str, Any]] = []+ for m in messages:+ payload = to_add_message_request(m)+ if payload is not None:+ out.append(payload)+ return outdiff --git a/openviking/ingest/orchestrator.py b/openviking/ingest/orchestrator.pynew file mode 100644index 000000000..68caeae7f--- /dev/null+++ b/openviking/ingest/orchestrator.py@@ -0,0 +1,143 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Backfill orchestration: replay each discovered session cursor->end, then commit.++Incremental watch mode is handled by ``IngestPoller`` (``poller.py``).+"""++from __future__ import annotations++from dataclasses import dataclass, field+from typing import Dict, List, Optional, Tuple++from openviking.ingest.models import Cursor+from openviking.ingest.registry import iter_enabled_sources+from openviking.ingest.replay import SessionReplayer+from openviking.ingest.sources.base import DEFAULT_READ_LIMIT, LogSource, NotSupportedError+from openviking_cli.utils import get_logger+from openviking_cli.utils.config.ingest_config import IngestConfig, IngestHarnessConfig++logger = get_logger(__name__)+++@dataclass+class BackfillStats:+ sessions: int = 0+ messages: int = 0+ committed: int = 0+ skipped: int = 0+ errors: List[str] = field(default_factory=list)++ def merge(self, other: "BackfillStats") -> None:+ self.sessions += other.sessions+ self.messages += other.messages+ self.committed += other.committed+ self.skipped += other.skipped+ self.errors.extend(other.errors)+++def enabled_sources(+ config: IngestConfig, only: Optional[List[str]] = None+) -> List[Tuple[str, IngestHarnessConfig, LogSource]]:+ """Construct enabled sources, optionally filtered to ``only`` harness names."""+ out = []+ for name, cfg, source in iter_enabled_sources(config):+ if only and name not in only:+ continue+ out.append((name, cfg, source))+ return out+++class IngestOrchestrator:+ def __init__(self, config: IngestConfig, replayer: SessionReplayer):+ self.config = config+ self.replayer = replayer+ self.store = replayer.store++ async def backfill_source(+ self,+ name: str,+ harness_cfg: IngestHarnessConfig,+ source: LogSource,+ *,+ since: Optional[str] = None,+ dry_run: bool = False,+ reset: bool = False,+ ) -> BackfillStats:+ stats = BackfillStats()+ try:+ refs = list(source.discover_sessions())+ except NotSupportedError as exc:+ logger.warning("[ingest:%s] %s", name, exc)+ stats.errors.append(f"{name}: {exc}")+ return stats+ except Exception as exc: # noqa: BLE001+ logger.exception("[ingest:%s] discovery failed", name)+ stats.errors.append(f"{name}: discovery failed: {exc}")+ return stats++ for ref in refs:+ if since and ref.started_at and ref.started_at < since:+ stats.skipped += 1+ continue+ try:+ if not dry_run:+ if reset:+ await self.replayer.reset_session(name, ref)+ else:+ await self.replayer.reconcile(name, ref)+ added = await self._backfill_one(name, source, ref, dry_run=dry_run)+ stats.sessions += 1+ stats.messages += added+ if not dry_run:+ # Commit even when added == 0: a prior crash may have left appended+ # messages un-committed (needs_commit / server pending tokens).+ if await self.replayer.commit_if_needed(+ name, ref, harness_cfg.commit.keep_recent_count+ ):+ stats.committed += 1+ except Exception as exc: # noqa: BLE001+ logger.exception("[ingest:%s] session %s failed", name, ref.native_session_id)+ stats.errors.append(f"{name}/{ref.native_session_id}: {exc}")+ return stats++ async def _backfill_one(self, name, source, ref, *, dry_run: bool) -> int:+ cursor = self.store.get_cursor(name, ref.native_session_id, source.cursor_kind)+ total = 0+ while True:+ messages, new_cursor = source.read_messages(ref, cursor, DEFAULT_READ_LIMIT)+ if not messages:+ from_value = (cursor or Cursor.zero(source.cursor_kind)).value+ if not dry_run and new_cursor.value != from_value:+ sid = self.replayer.ov_session_id(name, ref.native_session_id)+ self.store.advance_cursor(+ name, ref.native_session_id, sid, new_cursor, locator=ref.locator+ )+ break+ if dry_run:+ total += sum(1 for m in messages if (m.text or "").strip() or m.parts)+ else:+ from_cursor = cursor or Cursor.zero(source.cursor_kind)+ total += await self.replayer.append_batch(+ name, ref, messages, from_cursor, new_cursor+ )+ cursor = new_cursor+ return total++ async def backfill(+ self,+ only: Optional[List[str]] = None,+ *,+ since: Optional[str] = None,+ dry_run: bool = False,+ reset: bool = False,+ ) -> Dict[str, BackfillStats]:+ results: Dict[str, BackfillStats] = {}+ for name, cfg, source in enabled_sources(self.config, only):+ if cfg.mode not in ("backfill", "both") and only is None:+ continue+ logger.info("[ingest] backfilling %s (dry_run=%s)", name, dry_run)+ results[name] = await self.backfill_source(+ name, cfg, source, since=since, dry_run=dry_run, reset=reset+ )+ return resultsdiff --git a/openviking/ingest/peer.py b/openviking/ingest/peer.pynew file mode 100644index 000000000..00b531782--- /dev/null+++ b/openviking/ingest/peer.py@@ -0,0 +1,95 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""peer_id resolution for replayed turns.++- assistant turns -> ``{harness}__{model}`` (or ``{harness}__{provider}__{model}``)+- user turns:+ * single-user dev harnesses (claude_code/codex/opencode) -> git identity of the+ session cwd repo, falling back to the configured OV user;+ * group-chat harnesses (hermes/openclaw) -> the original username from the log.++peer_id must match OpenViking's identifier rules (``[a-zA-Z0-9_.@-]+``). ASCII values+stay human-readable; anything else (e.g. CJK usernames) is base64-encoded as ``ext-…``,+mirroring vikingbot's external-peer convention.+"""++from __future__ import annotations++import base64+import re+import subprocess+from typing import Dict, Optional++from openviking.core.peer_id import safe_peer_id++# Join harness/provider/model with the same separator as the OV session id scheme.+SEP = "__"++_ALLOWED = re.compile(r"[^a-zA-Z0-9_.@-]+")+_GIT_PEER_CACHE: Dict[str, Optional[str]] = {}+++def _sanitize_component(value: str) -> str:+ """Make one path-free component safe & readable (lossy for non-ASCII)."""+ cleaned = _ALLOWED.sub("-", (value or "").strip())+ cleaned = re.sub(r"-{2,}", "-", cleaned).strip("-.")+ return cleaned+++def safe_external_peer(raw: Optional[str]) -> Optional[str]:+ """Return a valid peer_id for an arbitrary external identifier (readable if ASCII)."""+ if not raw:+ return None+ text = str(raw).strip()+ if not text:+ return None+ sanitized = _sanitize_component(text)+ if sanitized:+ pid = safe_peer_id(sanitized)+ if pid:+ return pid+ # Non-ASCII / unsanitizable -> stable, valid, unique fallback.+ encoded = base64.urlsafe_b64encode(text.encode("utf-8")).decode("ascii").rstrip("=")+ return safe_peer_id(f"ext-{encoded}")+++def assistant_peer_id(+ harness: str, model: Optional[str], provider: Optional[str] = None+) -> Optional[str]:+ """`{harness}__{model}` or `{harness}__{provider}__{model}`."""+ parts = [_sanitize_component(harness)]+ if provider:+ parts.append(_sanitize_component(provider))+ parts.append(_sanitize_component(model) if model else "unknown")+ return safe_peer_id(SEP.join(p for p in parts if p))+++def resolve_git_human_peer(cwd: Optional[str], fallback: str) -> Optional[str]:+ """Human peer_id from the cwd repo's git identity; fall back to ``fallback``.++ Prefers ``user.email`` (more unique), then ``user.name``. Cached per cwd.+ """+ if not cwd:+ return safe_external_peer(fallback)+ if cwd in _GIT_PEER_CACHE:+ cached = _GIT_PEER_CACHE[cwd]+ return cached if cached else safe_external_peer(fallback)++ identity: Optional[str] = None+ for key in ("user.email", "user.name"):+ try:+ out = subprocess.run(+ ["git", "-C", cwd, "config", "--get", key],+ capture_output=True,+ text=True,+ timeout=2.0,+ )+ except (OSError, subprocess.SubprocessError):+ break+ if out.returncode == 0 and out.stdout.strip():+ identity = out.stdout.strip()+ break++ resolved = safe_external_peer(identity) if identity else None+ _GIT_PEER_CACHE[cwd] = resolved+ return resolved if resolved else safe_external_peer(fallback)diff --git a/openviking/ingest/poller.py b/openviking/ingest/poller.pynew file mode 100644index 000000000..03de08f65--- /dev/null+++ b/openviking/ingest/poller.py@@ -0,0 +1,148 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Incremental ingest (watch mode): a WatchScheduler-style asyncio poll loop.++Mirrors ``openviking/resource/watch_scheduler.py`` (interval polling, graceful start/stop)+rather than depending on filesystem events: a durable read-cursor makes polling correct+and self-healing (a missed tick / sleep / restart just reads cursor->EOF next time).++Each tick, per enabled harness: rescan sessions -> incremental ``read_messages`` from the+stored cursor -> append + advance cursor -> commit on idle or token threshold.+"""++from __future__ import annotations++import asyncio+import time+from typing import Dict, List, Tuple++from openviking.ingest.models import Cursor+from openviking.ingest.replay import SessionReplayer+from openviking.ingest.sources.base import DEFAULT_READ_LIMIT, LogSource, NotSupportedError+from openviking_cli.utils import get_logger+from openviking_cli.utils.config.ingest_config import IngestHarnessConfig++logger = get_logger(__name__)+++class _Dirty:+ __slots__ = ("name", "cfg", "source", "ref", "last_activity")++ def __init__(self, name, cfg, source, ref, last_activity):+ self.name = name+ self.cfg = cfg+ self.source = source+ self.ref = ref+ self.last_activity = last_activity+++class IngestPoller:+ def __init__(+ self,+ sources: List[Tuple[str, IngestHarnessConfig, LogSource]],+ replayer: SessionReplayer,+ ):+ self._sources = sources+ self.replayer = replayer+ self.store = replayer.store+ self._running = False+ self._stop = asyncio.Event()+ self._dirty: Dict[Tuple[str, str], _Dirty] = {}+ self.poll_interval = min((cfg.poll_interval_seconds for _, cfg, _ in sources), default=5.0)++ def stop(self) -> None:+ self._running = False+ self._stop.set()++ async def run(self) -> None:+ if not self._sources:+ logger.info("[ingest] no harnesses enabled for watch; nothing to do")+ return+ self._running = True+ logger.info(+ "[ingest] watch loop started (poll=%.1fs, harnesses=%s)",+ self.poll_interval,+ [n for n, _, _ in self._sources],+ )+ try:+ while self._running:+ await self._tick()+ await self._commit_idle()+ try:+ await asyncio.wait_for(self._stop.wait(), timeout=self.poll_interval)+ except asyncio.TimeoutError:+ pass+ finally:+ await self._flush_all()+ logger.info("[ingest] watch loop stopped")++ async def _tick(self) -> None:+ for name, cfg, source in self._sources:+ try:+ refs = list(source.discover_sessions())+ except NotSupportedError as exc:+ logger.warning("[ingest:%s] %s", name, exc)+ continue+ except Exception: # noqa: BLE001+ logger.exception("[ingest:%s] discovery failed", name)+ continue+ for ref in refs:+ await self._poll_session(name, cfg, source, ref)++ async def _poll_session(self, name, cfg, source, ref) -> None:+ await self.replayer.reconcile(name, ref)+ appended = False+ while True:+ cursor = self.store.get_cursor(name, ref.native_session_id, source.cursor_kind)+ try:+ messages, new_cursor = source.read_messages(ref, cursor, DEFAULT_READ_LIMIT)+ except Exception: # noqa: BLE001+ logger.exception("[ingest:%s] read failed for %s", name, ref.native_session_id)+ return+ if not messages:+ from_value = (cursor or Cursor.zero(source.cursor_kind)).value+ if new_cursor.value != from_value:+ sid = self.replayer.ov_session_id(name, ref.native_session_id)+ self.store.advance_cursor(+ name, ref.native_session_id, sid, new_cursor, locator=ref.locator+ )+ break+ from_cursor = cursor or Cursor.zero(source.cursor_kind)+ added = await self.replayer.append_batch(name, ref, messages, from_cursor, new_cursor)+ if added > 0:+ appended = True++ rec = self.store.get(name, ref.native_session_id)+ if appended or (rec and rec.needs_commit):+ self._dirty[(name, ref.native_session_id)] = _Dirty(+ name, cfg, source, ref, time.monotonic()+ )+ committed = await self.replayer.maybe_commit_on_threshold(+ name, ref, cfg.commit.commit_token_threshold, cfg.commit.keep_recent_count+ )+ if committed:+ self._dirty.pop((name, ref.native_session_id), None)++ async def _commit_idle(self) -> None:+ now = time.monotonic()+ for key, d in list(self._dirty.items()):+ if now - d.last_activity >= d.cfg.commit.commit_idle_seconds:+ try:+ await self.replayer.commit_if_needed(+ d.name, d.ref, d.cfg.commit.keep_recent_count+ )+ self._dirty.pop(key, None) # only drop on success+ except Exception: # noqa: BLE001 - keep dirty and retry after a backoff+ logger.exception("[ingest:%s] idle commit failed; will retry", d.name)+ d.last_activity = time.monotonic()++ async def _flush_all(self) -> None:+ """Final commit for every dirty session on shutdown (needs_commit persists if it fails)."""+ for key, d in list(self._dirty.items()):+ try:+ await self.replayer.commit_if_needed(d.name, d.ref, d.cfg.commit.keep_recent_count)+ except Exception: # noqa: BLE001+ logger.exception(+ "[ingest:%s] shutdown commit failed (needs_commit persisted)", d.name+ )+ self._dirty.pop(key, None)diff --git a/openviking/ingest/registry.py b/openviking/ingest/registry.pynew file mode 100644index 000000000..2b80cf77e--- /dev/null+++ b/openviking/ingest/registry.py@@ -0,0 +1,55 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Registry of harness log-source adapters.++Adding a harness = one ``@register_source("name")`` decorator on a ``LogSource``+subclass; no config-schema change is needed (config keys are free-form).+"""++from __future__ import annotations++from typing import TYPE_CHECKING, Dict, Iterator, Tuple, Type++from openviking_cli.utils import get_logger++if TYPE_CHECKING:+ from openviking.ingest.sources.base import LogSource+ from openviking_cli.utils.config.ingest_config import IngestConfig, IngestHarnessConfig++logger = get_logger(__name__)++SOURCE_REGISTRY: Dict[str, "Type[LogSource]"] = {}+++def register_source(name: str):+ """Class decorator registering a ``LogSource`` subclass under ``name``."""++ def deco(cls: "Type[LogSource]") -> "Type[LogSource]":+ cls.name = name+ if name in SOURCE_REGISTRY and SOURCE_REGISTRY[name] is not cls:+ logger.warning("[ingest] source %r already registered; overriding", name)+ SOURCE_REGISTRY[name] = cls+ return cls++ return deco+++def get_source_class(name: str) -> "Type[LogSource] | None":+ # Ensure built-in adapters have been imported (and thus registered).+ import openviking.ingest.sources # noqa: F401++ return SOURCE_REGISTRY.get(name)+++def iter_enabled_sources(+ config: "IngestConfig",+) -> Iterator[Tuple[str, "IngestHarnessConfig", "LogSource"]]:+ """Yield ``(name, harness_cfg, source)`` for each enabled, registered harness."""+ import openviking.ingest.sources # noqa: F401 (populates SOURCE_REGISTRY)++ for name, harness_cfg in config.enabled_harnesses().items():+ cls = SOURCE_REGISTRY.get(name)+ if cls is None:+ logger.warning("[ingest] no adapter registered for harness %r; skipping", name)+ continue+ yield name, harness_cfg, cls(harness_cfg, fallback_user=config.user)diff --git a/openviking/ingest/replay.py b/openviking/ingest/replay.pynew file mode 100644index 000000000..2e4a1ca0c--- /dev/null+++ b/openviking/ingest/replay.py@@ -0,0 +1,212 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Replay normalized messages into OpenViking via the SDK HTTP client.++``ConversationReplayClient`` is a thin, vikingbot-free wrapper over ``ov.AsyncHTTPClient``+(client-side, transport-agnostic: targets a local or remote server). ``SessionReplayer``+drives the canonical recipe in idempotent batches:++ reconcile() -> for each <=100-message batch: set_pending -> append -> confirm -> commit_if_needed++Each batch's intent (target cursor + size + the server's message count *before* the+append) is persisted BEFORE the append. If the process crashes mid-append, the next run+``reconcile()`` compares the server's current message count against that baseline to decide+whether the batch landed — confirming it (no re-append) or dropping the intent (re-read+from the confirmed cursor). The cursor only advances on a confirmed append, and+``needs_commit`` ensures appended-but-uncommitted sessions still get extracted later.+"""++from __future__ import annotations++import re+from typing import Any, Dict, List, Optional++import openviking as ov+from openviking.ingest.cursor_store import CursorStore+from openviking.ingest.models import Cursor, NormalizedMessage, SessionRef+from openviking.ingest.normalize import to_add_message_requests+from openviking_cli.exceptions import NotFoundError+from openviking_cli.utils import get_logger++logger = get_logger(__name__)++_BATCH = 100 # server-side cap for batch_add_messages+_SID_ALLOWED = re.compile(r"[^a-zA-Z0-9_.@-]+")+++def _sanitize_id(value: str) -> str:+ cleaned = _SID_ALLOWED.sub("-", (value or "").strip())+ return re.sub(r"-{2,}", "-", cleaned).strip("-.") or "unknown"+++class ConversationReplayClient:+ """Minimal session-recording surface over the OV SDK HTTP client."""++ def __init__(self, client: "ov.AsyncHTTPClient"):+ self._client = client++ @classmethod+ async def create(+ cls, *, url: str = "", api_key: str = "", account: str = "", user: str = ""+ ) -> "ConversationReplayClient":+ client = ov.AsyncHTTPClient(+ url=url or None,+ api_key=api_key or None,+ account=account or None,+ user=user or None,+ )+ await client.initialize()+ return cls(client)++ async def close(self) -> None:+ await self._client.close()++ async def ensure_session(+ self, ov_session_id: str, memory_policy: Optional[Dict[str, Any]] = None+ ) -> Dict[str, Any]:+ try:+ return await self._client.get_session(ov_session_id)+ except NotFoundError:+ return await self._client.create_session(+ session_id=ov_session_id, memory_policy=memory_policy or None+ )++ async def append(self, ov_session_id: str, messages: List[Dict[str, Any]]) -> int:+ total = 0+ for start in range(0, len(messages), _BATCH):+ chunk = messages[start : start + _BATCH]+ result = await self._client.batch_add_messages(ov_session_id, chunk)+ total += int(result.get("added", len(chunk)) or 0)+ return total++ async def commit(self, ov_session_id: str, keep_recent_count: int = 0) -> Dict[str, Any]:+ return await self._client.commit_session(ov_session_id, keep_recent_count=keep_recent_count)++ async def _session_info(self, ov_session_id: str) -> Dict[str, Any]:+ try:+ return await self._client.get_session(ov_session_id)+ except NotFoundError:+ return {}++ async def pending_tokens(self, ov_session_id: str) -> int:+ info = await self._session_info(ov_session_id)+ return int(info.get("pending_tokens", 0) or 0)++ async def message_count(self, ov_session_id: str) -> int:+ info = await self._session_info(ov_session_id)+ return int(info.get("message_count", 0) or 0)++ async def delete(self, ov_session_id: str) -> None:+ try:+ await self._client.delete_session(ov_session_id)+ except NotFoundError:+ pass+++class SessionReplayer:+ def __init__(+ self,+ client: ConversationReplayClient,+ store: CursorStore,+ *,+ session_id_prefix: str = "import",+ memory_policy: Optional[Dict[str, Any]] = None,+ ):+ self.client = client+ self.store = store+ self.prefix = session_id_prefix+ self.memory_policy = memory_policy or None++ def ov_session_id(self, harness: str, native_session_id: str) -> str:+ return "__".join(+ [_sanitize_id(self.prefix), _sanitize_id(harness), _sanitize_id(native_session_id)]+ )++ async def reconcile(self, harness: str, ref: SessionRef) -> None:+ """Resolve a pending batch intent left by a crash before the next read happens."""+ rec = self.store.get(harness, ref.native_session_id)+ if not rec or not rec.pending_count or rec.pending_cursor is None:+ return+ sid = self.ov_session_id(harness, ref.native_session_id)+ try:+ count = await self.client.message_count(sid)+ except Exception: # noqa: BLE001 - leave the intent; retry next time+ logger.warning("[ingest:%s] reconcile read failed for %s", harness, sid)+ return+ if count >= rec.pending_baseline + rec.pending_count:+ # The batch landed before the crash: confirm it without re-appending.+ self.store.confirm_append(+ harness, ref.native_session_id, rec.pending_cursor, rec.pending_count+ )+ logger.info("[ingest:%s] reconciled landed batch for %s", harness, sid)+ else:+ # It did not land: drop the intent and re-read from the confirmed cursor.+ self.store.clear_pending(harness, ref.native_session_id)++ async def append_batch(+ self,+ harness: str,+ ref: SessionRef,+ messages: List[NormalizedMessage],+ from_cursor: Cursor,+ new_cursor: Cursor,+ ) -> int:+ """Append one <=100 batch idempotently and advance the confirmed cursor."""+ sid = self.ov_session_id(harness, ref.native_session_id)+ payloads = to_add_message_requests(messages)+ if not payloads:+ # Only control/empty turns consumed: advance the cursor, no commit owed.+ self.store.advance_cursor(+ harness, ref.native_session_id, sid, new_cursor, locator=ref.locator+ )+ return 0+ await self.client.ensure_session(sid, self.memory_policy)+ baseline = await self.client.message_count(sid)+ self.store.set_pending(+ harness,+ ref.native_session_id,+ sid,+ from_cursor,+ new_cursor,+ len(payloads),+ baseline,+ locator=ref.locator,+ title=ref.title,+ )+ await self.client.append(sid, payloads)+ self.store.confirm_append(harness, ref.native_session_id, new_cursor, len(payloads))+ return len(payloads)++ async def commit_if_needed(+ self, harness: str, ref: SessionRef, keep_recent_count: int = 0+ ) -> bool:+ """Commit when the session has un-committed appends or live pending tokens."""+ sid = self.ov_session_id(harness, ref.native_session_id)+ rec = self.store.get(harness, ref.native_session_id)+ need = bool(rec and rec.needs_commit)+ pending = await self.client.pending_tokens(sid)+ if not need and pending <= 0:+ return False+ if pending <= 0:+ # Nothing live to archive (already committed elsewhere); clear the stale flag.+ self.store.mark_committed(harness, ref.native_session_id)+ return False+ await self.client.commit(sid, keep_recent_count=keep_recent_count)+ self.store.mark_committed(harness, ref.native_session_id)+ return True++ async def maybe_commit_on_threshold(+ self, harness: str, ref: SessionRef, threshold: int, keep_recent_count: int = 0+ ) -> bool:+ sid = self.ov_session_id(harness, ref.native_session_id)+ pending = await self.client.pending_tokens(sid)+ if pending < threshold:+ return False+ await self.client.commit(sid, keep_recent_count=keep_recent_count)+ self.store.mark_committed(harness, ref.native_session_id)+ return True++ async def reset_session(self, harness: str, ref: SessionRef) -> None:+ sid = self.ov_session_id(harness, ref.native_session_id)+ await self.client.delete(sid)+ self.store.delete(harness, ref.native_session_id)diff --git a/openviking/ingest/sources/__init__.py b/openviking/ingest/sources/__init__.pynew file mode 100644index 000000000..7e6ddc659--- /dev/null+++ b/openviking/ingest/sources/__init__.py@@ -0,0 +1,15 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Harness log-source adapters. Importing this package registers all built-ins."""++# Import for side effects: each module's @register_source populates SOURCE_REGISTRY.+from openviking.ingest.sources import ( # noqa: F401+ claude_code,+ codex,+ cursor,+ hermes,+ openclaw,+ opencode,+)++__all__ = ["claude_code", "codex", "cursor", "hermes", "opencode", "openclaw"]diff --git a/openviking/ingest/sources/base.py b/openviking/ingest/sources/base.pynew file mode 100644index 000000000..0125ef023--- /dev/null+++ b/openviking/ingest/sources/base.py@@ -0,0 +1,252 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Log-source abstraction: one ABC + two intermediates so a new harness is a thin subclass.++- ``JsonlLogSource`` — append-only JSONL (Claude Code, Codex, Hermes, OpenClaw); byte-offset cursor.+- ``SqliteLogSource`` — relational SQLite (OpenCode); (time, id) cursor, polled read-only.+"""++from __future__ import annotations++import json+import sqlite3+from abc import ABC, abstractmethod+from pathlib import Path+from typing import TYPE_CHECKING, Any, ClassVar, Dict, Iterable, List, Optional, Tuple++from openviking.ingest.models import (+ BYTE_OFFSET,+ ROWID_TIME,+ Cursor,+ NormalizedMessage,+ SessionRef,+)+from openviking.ingest.peer import (+ assistant_peer_id,+ resolve_git_human_peer,+ safe_external_peer,+)+from openviking_cli.utils import get_logger++if TYPE_CHECKING:+ from openviking_cli.utils.config.ingest_config import IngestHarnessConfig++logger = get_logger(__name__)++# Max messages returned per read_messages() call. Bounds memory and lets the orchestrator+# append+confirm in idempotent batches (<= the server's 100-message batch cap).+DEFAULT_READ_LIMIT = 100+_READ_BLOCK = 262144 # 256 KiB+++class NotSupportedError(RuntimeError):+ """Raised by an adapter that is registered but not usable in the current config."""+++class LogSource(ABC):+ """Base adapter for one agent harness's conversation logs."""++ name: ClassVar[str] = "" # set by @register_source+ cursor_kind: ClassVar[str] = BYTE_OFFSET+ is_group_chat: ClassVar[bool] = False # group agents map user turns to original usernames++ def __init__(self, harness_cfg: "IngestHarnessConfig", *, fallback_user: str = "default"):+ self.cfg = harness_cfg+ self.fallback_user = fallback_user++ # --- paths -------------------------------------------------------------+ @abstractmethod+ def default_paths(self) -> List[Path]:+ """Default discovery roots (files/dirs/db paths) when config gives no override."""++ def roots(self) -> List[Path]:+ if self.cfg.paths:+ return [Path(p).expanduser() for p in self.cfg.paths]+ return self.default_paths()++ # --- discovery / reading ----------------------------------------------+ @abstractmethod+ def discover_sessions(self) -> Iterable[SessionRef]:+ """Enumerate conversations available in this harness's storage."""++ @abstractmethod+ def read_messages(+ self, ref: SessionRef, cursor: Optional[Cursor], limit: int = DEFAULT_READ_LIMIT+ ) -> Tuple[List[NormalizedMessage], Cursor]:+ """Read up to ``limit`` messages after ``cursor``; return (messages, advanced cursor)."""++ # --- peer_id helpers (used by adapters) -------------------------------+ def assistant_peer(self, model: Optional[str], provider: Optional[str] = None) -> Optional[str]:+ return assistant_peer_id(self.name, model, provider)++ def user_peer(+ self, *, cwd: Optional[str] = None, raw_user: Optional[str] = None+ ) -> Optional[str]:+ if self.is_group_chat:+ return safe_external_peer(raw_user) or safe_external_peer(self.fallback_user)+ return resolve_git_human_peer(cwd, self.fallback_user)+++# =============================================================================+# Append-only JSONL+# =============================================================================+class JsonlLogSource(LogSource):+ cursor_kind = BYTE_OFFSET+ file_glob: ClassVar[str] = "*.jsonl"++ def discover_sessions(self) -> Iterable[SessionRef]:+ for root in self.roots():+ root = root.expanduser()+ if not root.exists():+ continue+ for path in sorted(root.glob(self.file_glob)):+ if path.is_file():+ yield self.session_ref_for_file(path)++ def session_ref_for_file(self, path: Path) -> SessionRef:+ return SessionRef(+ harness=self.name,+ native_session_id=self.session_id_for_file(path),+ locator=str(path),+ )++ def session_id_for_file(self, path: Path) -> str:+ return path.stem++ @staticmethod+ def _peek_first_json(path: Path) -> Optional[Dict[str, Any]]:+ try:+ with open(path, "rb") as f:+ for raw in f:+ raw = raw.strip()+ if raw:+ return json.loads(raw)+ except (OSError, ValueError):+ return None+ return None++ def read_messages(+ self, ref: SessionRef, cursor: Optional[Cursor], limit: int = DEFAULT_READ_LIMIT+ ) -> Tuple[List[NormalizedMessage], Cursor]:+ path = Path(ref.locator)+ cur = cursor or Cursor.zero(self.cursor_kind)+ try:+ st = path.stat()+ except OSError:+ return [], cur++ start = int(cur.value.get("offset", 0))+ stored_inode = cur.value.get("inode")+ # Rotation (file replaced) or truncation -> re-read from the top.+ if (stored_inode is not None and stored_inode != st.st_ino) or start > st.st_size:+ start = 0++ messages: List[NormalizedMessage] = []+ consumed = 0 # bytes of fully-processed (newline-terminated) lines+ buf = b""+ with open(path, "rb") as f:+ f.seek(start)+ while len(messages) < limit:+ chunk = f.read(_READ_BLOCK)+ if not chunk:+ break # EOF; any partial trailing line stays in buf, unconsumed+ buf += chunk+ stop = False+ while True:+ nl = buf.find(b"\n")+ if nl == -1:+ break+ line = buf[: nl + 1]+ buf = buf[nl + 1 :]+ consumed += len(line)+ stripped = line.strip()+ if stripped:+ try:+ obj = json.loads(stripped)+ except ValueError:+ obj = None+ if obj is not None:+ try:+ messages.extend(self.parse_line(obj, ref))+ except Exception as exc: # one bad record must not abort the file+ logger.debug(+ "[ingest:%s] skip bad record in %s: %s",+ self.name,+ path.name,+ exc,+ )+ if len(messages) >= limit:+ stop = True+ break+ if stop:+ break++ new_offset = start + consumed+ return messages, Cursor(self.cursor_kind, {"offset": new_offset, "inode": st.st_ino})++ @abstractmethod+ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:+ """Map one JSONL record to zero or more normalized messages."""+++# =============================================================================+# Relational SQLite+# =============================================================================+class SqliteLogSource(LogSource):+ cursor_kind = ROWID_TIME++ @abstractmethod+ def db_path(self) -> Path:+ """Path to the harness's SQLite database (config override honored upstream)."""++ def _connect(self, *, immutable: bool = False) -> sqlite3.Connection:+ path = self.db_path()+ if not path.exists():+ raise NotSupportedError(f"{self.name}: database not found at {path}")+ # mode=ro honors the WAL for live polling; immutable=1 (static snapshot) ignores it.+ suffix = "mode=ro&immutable=1" if immutable else "mode=ro"+ conn = sqlite3.connect(f"file:{path}?{suffix}", uri=True, timeout=5.0)+ conn.row_factory = sqlite3.Row+ return conn++ def read_messages(+ self, ref: SessionRef, cursor: Optional[Cursor], limit: int = DEFAULT_READ_LIMIT+ ) -> Tuple[List[NormalizedMessage], Cursor]:+ cur = cursor or Cursor.zero(self.cursor_kind)+ conn = self._connect(immutable=False)+ try:+ rows = list(self.fetch_rows(conn, ref, cur, limit))+ # Advance only past rows that are fully written. A still-incomplete trailing+ # row (e.g. a message whose `part` text has not been flushed yet) is left for+ # the next poll so we never skip its content.+ complete: List[sqlite3.Row] = []+ for row in rows:+ if self.row_complete(conn, row):+ complete.append(row)+ else:+ break+ messages = self.rows_to_messages(conn, ref, complete)+ if complete:+ last = complete[-1]+ new_cur = Cursor(self.cursor_kind, {"time": last["time_created"], "id": last["id"]})+ else:+ new_cur = cur+ return messages, new_cur+ finally:+ conn.close()++ def row_complete(self, conn: sqlite3.Connection, row: sqlite3.Row) -> bool:+ """Whether a fetched row is fully written and safe to advance past (default: yes)."""+ return True++ @abstractmethod+ def fetch_rows(+ self, conn: sqlite3.Connection, ref: SessionRef, cursor: Cursor, limit: int+ ) -> List[sqlite3.Row]:+ """Up to ``limit`` rows after the cursor, ordered by (time_created, id)."""++ @abstractmethod+ def rows_to_messages(+ self, conn: sqlite3.Connection, ref: SessionRef, rows: List[sqlite3.Row]+ ) -> List[NormalizedMessage]:+ """Map rows to normalized messages (may issue extra queries via ``conn``)."""diff --git a/openviking/ingest/sources/claude_code.py b/openviking/ingest/sources/claude_code.pynew file mode 100644index 000000000..750b08647--- /dev/null+++ b/openviking/ingest/sources/claude_code.py@@ -0,0 +1,68 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Claude Code adapter.++Logs: ``~/.claude/projects/<project-slug>/<session-uuid>.jsonl`` (append-only JSONL).+Each record has a top-level ``type``; conversation turns are ``type in {user, assistant}``+with a nested ``message{role, content, model}``. ``content`` is a string or a list of+blocks (text / tool_use / tool_result). cwd + gitBranch are per record.+"""++from __future__ import annotations++from pathlib import Path+from typing import Any, Dict, List++from openviking.ingest.models import NormalizedMessage, SessionRef+from openviking.ingest.registry import register_source+from openviking.ingest.sources.base import JsonlLogSource+++def _extract_text(content: Any) -> str:+ if isinstance(content, str):+ return content.strip()+ if isinstance(content, list):+ chunks: List[str] = []+ for block in content:+ if isinstance(block, dict) and block.get("type") == "text":+ text = block.get("text")+ if isinstance(text, str) and text.strip():+ chunks.append(text.strip())+ return "\n".join(chunks).strip()+ return ""+++@register_source("claude_code")+class ClaudeCodeSource(JsonlLogSource):+ file_glob = "*/*.jsonl"++ def default_paths(self) -> List[Path]:+ return [Path.home() / ".claude" / "projects"]++ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:+ if obj.get("type") not in ("user", "assistant"):+ return []+ if obj.get("isSidechain") or obj.get("isMeta"):+ return [] # sub-agent / synthetic records are low-value for memory+ message = obj.get("message")+ if not isinstance(message, dict):+ return []+ role = message.get("role")+ if role not in ("user", "assistant"):+ return []+ text = _extract_text(message.get("content"))+ if not text:+ return [] # tool-only turn -> dropped++ model = message.get("model")+ cwd = obj.get("cwd")+ peer = self.assistant_peer(model) if role == "assistant" else self.user_peer(cwd=cwd)+ return [+ NormalizedMessage(+ role=role,+ text=text,+ created_at=obj.get("timestamp"),+ peer_id=peer,+ meta={"model": model, "cwd": cwd, "git_branch": obj.get("gitBranch")},+ )+ ]diff --git a/openviking/ingest/sources/codex.py b/openviking/ingest/sources/codex.pynew file mode 100644index 000000000..db76cb937--- /dev/null+++ b/openviking/ingest/sources/codex.py@@ -0,0 +1,95 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Codex (OpenAI Codex CLI) adapter.++Logs: ``~/.codex/sessions/YYYY/MM/DD/rollout-<ts>-<uuid>.jsonl`` (append-only JSONL).+Records: ``{timestamp, type, payload}``. Conversation turns are+``type=="response_item" & payload.type=="message"`` with ``payload.role`` and+``payload.content[].text`` (``input_text`` / ``output_text``). ``role=="developer"``/+``"system"`` are dropped. Session id / cwd / provider come from the first+``session_meta`` record (the model name is not reliably per-message).+"""++from __future__ import annotations++import json+from pathlib import Path+from typing import Any, Dict, List++from openviking.ingest.models import NormalizedMessage, SessionRef+from openviking.ingest.registry import register_source+from openviking.ingest.sources.base import JsonlLogSource+++def _join_content(content: Any) -> str:+ if not isinstance(content, list):+ return ""+ chunks: List[str] = []+ for block in content:+ if isinstance(block, dict) and block.get("type") in ("input_text", "output_text"):+ text = block.get("text")+ if isinstance(text, str) and text.strip():+ chunks.append(text.strip())+ return "\n".join(chunks).strip()+++@register_source("codex")+class CodexSource(JsonlLogSource):+ file_glob = "*/*/*/rollout-*.jsonl"++ def default_paths(self) -> List[Path]:+ return [Path.home() / ".codex" / "sessions"]++ def session_ref_for_file(self, path: Path) -> SessionRef:+ meta = self._peek_session_meta(path)+ return SessionRef(+ harness=self.name,+ native_session_id=meta.get("id") or path.stem,+ locator=str(path),+ started_at=meta.get("timestamp"),+ meta={"model": meta.get("model_provider"), "cwd": meta.get("cwd")},+ )++ @staticmethod+ def _peek_session_meta(path: Path) -> Dict[str, Any]:+ try:+ with open(path, "rb") as f:+ for raw in f:+ raw = raw.strip()+ if not raw:+ continue+ obj = json.loads(raw)+ if obj.get("type") == "session_meta":+ return obj.get("payload", {}) or {}+ except (OSError, ValueError):+ pass+ return {}++ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:+ if obj.get("type") != "response_item":+ return []+ payload = obj.get("payload") or {}+ if payload.get("type") != "message":+ return []+ role = payload.get("role")+ if role not in ("user", "assistant"):+ return [] # drop developer/system boilerplate+ text = _join_content(payload.get("content"))+ if not text:+ return []++ model = ref.meta.get("model")+ peer = (+ self.assistant_peer(model)+ if role == "assistant"+ else self.user_peer(cwd=ref.meta.get("cwd"))+ )+ return [+ NormalizedMessage(+ role=role,+ text=text,+ created_at=obj.get("timestamp"),+ peer_id=peer,+ meta={"model": model, "cwd": ref.meta.get("cwd")},+ )+ ]diff --git a/openviking/ingest/sources/cursor.py b/openviking/ingest/sources/cursor.pynew file mode 100644index 000000000..466d76819--- /dev/null+++ b/openviking/ingest/sources/cursor.py@@ -0,0 +1,59 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Cursor (IDE) adapter — DEFERRED stub.++NOTE: this is the Cursor *IDE harness*, distinct from the read-position ``Cursor``+pointer in ``models.py``.++Cursor stores chat in ``~/Library/Application Support/Cursor/User/{globalStorage,+workspaceStorage/*}/state.vscdb`` (SQLite key-value: ``ItemTable`` / ``cursorDiskKV``),+with a single conversation spread across ``composerData:*`` / ``bubbleId:*`` /+``agentKv:*`` BLOB→JSON keys. The schema is undocumented, drifts between Cursor+versions, and offers no monotonic cursor for clean incremental reads — so ingestion is+deferred. The adapter is registered (so it shows up in ``list-sources``) but refuses to+run unless explicitly opted in (``enabled`` + ``experimental``), and even then is not yet+implemented.+"""++from __future__ import annotations++from pathlib import Path+from typing import List++from openviking.ingest.registry import register_source+from openviking.ingest.sources.base import NotSupportedError, SqliteLogSource+++@register_source("cursor")+class CursorIDESource(SqliteLogSource):+ def default_paths(self) -> List[Path]:+ base = Path.home() / "Library" / "Application Support" / "Cursor" / "User"+ return [base / "globalStorage" / "state.vscdb"]++ def db_path(self) -> Path:+ roots = self.roots()+ return roots[0] if roots else self.default_paths()[0]++ def _guard(self) -> None:+ if not (self.cfg.enabled and self.cfg.experimental):+ raise NotSupportedError(+ "cursor ingest is deferred/experimental; set both enabled=true and "+ "experimental=true to opt in"+ )+ raise NotSupportedError(+ "cursor ingest is not implemented: state.vscdb stores conversations as "+ "undocumented, version-unstable KV blobs (composerData/bubbleId/agentKv) "+ "with no monotonic cursor"+ )++ def discover_sessions(self):+ self._guard()+ return []++ def fetch_rows(self, conn, ref, cursor, limit): # pragma: no cover - deferred+ self._guard()+ return []++ def rows_to_messages(self, conn, ref, rows): # pragma: no cover - deferred+ self._guard()+ return []diff --git a/openviking/ingest/sources/hermes.py b/openviking/ingest/sources/hermes.pynew file mode 100644index 000000000..89a125e79--- /dev/null+++ b/openviking/ingest/sources/hermes.py@@ -0,0 +1,64 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Hermes adapter (group-chat agent).++Logs: ``~/.hermes/sessions/<ts>_<id>.jsonl`` (append-only JSONL). Records are keyed by+``role``: a leading ``session_meta`` (carries ``model`` + ``platform``), then ``user`` /+``assistant`` turns with ``content`` (str) + ``timestamp``.++Group-chat agent: user turns map to the original username when present in the log+(``IngestHarnessConfig.user_field``); otherwise the configured OV user.+"""++from __future__ import annotations++from pathlib import Path+from typing import Any, Dict, List++from openviking.ingest.models import NormalizedMessage, SessionRef+from openviking.ingest.registry import register_source+from openviking.ingest.sources.base import JsonlLogSource+++@register_source("hermes")+class HermesSource(JsonlLogSource):+ is_group_chat = True+ file_glob = "*.jsonl"++ def default_paths(self) -> List[Path]:+ return [Path.home() / ".hermes" / "sessions"]++ def session_ref_for_file(self, path: Path) -> SessionRef:+ first = self._peek_first_json(path) or {}+ model = first.get("model") if first.get("role") == "session_meta" else None+ return SessionRef(+ harness=self.name,+ native_session_id=path.stem,+ locator=str(path),+ started_at=first.get("timestamp"),+ meta={"model": model, "platform": first.get("platform")},+ )++ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:+ role = obj.get("role")+ if role not in ("user", "assistant"):+ return [] # drop session_meta and any other control record+ text = (obj.get("content") or "").strip()+ if not text:+ return []++ model = ref.meta.get("model")+ if role == "assistant":+ peer = self.assistant_peer(model)+ else:+ raw_user = obj.get(self.cfg.user_field) if self.cfg.user_field else None+ peer = self.user_peer(raw_user=raw_user)+ return [+ NormalizedMessage(+ role=role,+ text=text,+ created_at=obj.get("timestamp"),+ peer_id=peer,+ meta={"model": model, "platform": ref.meta.get("platform")},+ )+ ]diff --git a/openviking/ingest/sources/openclaw.py b/openviking/ingest/sources/openclaw.pynew file mode 100644index 000000000..9a7eeece4--- /dev/null+++ b/openviking/ingest/sources/openclaw.py@@ -0,0 +1,74 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""OpenClaw adapter (group-chat agent).++Logs: ``~/.openclaw/agents/<agent>/sessions/<uuid>.jsonl`` (append-only JSONL). Records+carry a top-level ``type``; conversation turns are ``type=="message"`` with a nested+``message{role, content[], timestamp}``. Assistant messages additionally carry+``model`` + ``provider``. ``content`` is a list of blocks (text / thinking / …).++Group-chat agent: user turns map to the original username when present+(``IngestHarnessConfig.user_field``); otherwise the configured OV user.++Format reference: openclaw-plugin session logs (see project memory on citing openclaw).+"""++from __future__ import annotations++from pathlib import Path+from typing import Any, Dict, List++from openviking.ingest.models import NormalizedMessage, SessionRef+from openviking.ingest.registry import register_source+from openviking.ingest.sources.base import JsonlLogSource+++def _extract_text(content: Any) -> str:+ if isinstance(content, str):+ return content.strip()+ if isinstance(content, list):+ chunks: List[str] = []+ for block in content:+ if isinstance(block, dict) and block.get("type") == "text":+ text = block.get("text")+ if isinstance(text, str) and text.strip():+ chunks.append(text.strip())+ return "\n".join(chunks).strip()+ return ""+++@register_source("openclaw")+class OpenClawSource(JsonlLogSource):+ is_group_chat = True+ file_glob = "*/sessions/*.jsonl"++ def default_paths(self) -> List[Path]:+ return [Path.home() / ".openclaw" / "agents"]++ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:+ if obj.get("type") != "message":+ return []+ message = obj.get("message")+ if not isinstance(message, dict):+ return []+ role = message.get("role")+ if role not in ("user", "assistant"):+ return []+ text = _extract_text(message.get("content"))+ if not text:+ return []++ if role == "assistant":+ peer = self.assistant_peer(message.get("model"), message.get("provider"))+ else:+ raw_user = message.get(self.cfg.user_field) if self.cfg.user_field else None+ peer = self.user_peer(raw_user=raw_user)+ return [+ NormalizedMessage(+ role=role,+ text=text,+ created_at=obj.get("timestamp"),+ peer_id=peer,+ meta={"model": message.get("model"), "provider": message.get("provider")},+ )+ ]diff --git a/openviking/ingest/sources/opencode.py b/openviking/ingest/sources/opencode.pynew file mode 100644index 000000000..cfb56e2a9--- /dev/null+++ b/openviking/ingest/sources/opencode.py@@ -0,0 +1,137 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""OpenCode (sst/opencode) adapter — SUPPORTED-EXPERIMENTAL.++Logs: ``~/.local/share/opencode/opencode.db`` (SQLite, WAL). ``session(id, title,+directory, model, time_created)``; ``message(id, session_id, time_created, data JSON+{role, modelID, providerID, …})``; the actual text lives in ``part(message_id,+time_created, data JSON{type, text})``. Polled read-only via (time_created, id) cursor.++Older OpenCode versions used a JSON file-store; that layout is deferred (clear error).+"""++from __future__ import annotations++import json+import sqlite3+from pathlib import Path+from typing import List++from openviking.ingest.models import NormalizedMessage, SessionRef, iso_from_epoch_ms+from openviking.ingest.registry import register_source+from openviking.ingest.sources.base import NotSupportedError, SqliteLogSource+++@register_source("opencode")+class OpenCodeSource(SqliteLogSource):+ def default_paths(self) -> List[Path]:+ return [Path.home() / ".local" / "share" / "opencode" / "opencode.db"]++ def db_path(self) -> Path:+ roots = self.roots()+ return roots[0] if roots else self.default_paths()[0]++ def discover_sessions(self):+ db = self.db_path()+ if not db.exists():+ legacy = Path.home() / ".local" / "share" / "opencode" / "project"+ if legacy.exists():+ raise NotSupportedError(+ "opencode legacy file-store (~/.local/share/opencode/project) "+ "is not supported (deferred); only opencode.db is supported"+ )+ return+ conn = self._connect()+ try:+ rows = conn.execute(+ "SELECT id, title, directory, model, time_created "+ "FROM session ORDER BY time_created"+ ).fetchall()+ finally:+ conn.close()+ for r in rows:+ yield SessionRef(+ harness=self.name,+ native_session_id=r["id"],+ locator=r["id"],+ title=r["title"],+ started_at=iso_from_epoch_ms(r["time_created"]),+ meta={"cwd": r["directory"], "session_model": r["model"]},+ )++ def fetch_rows(self, conn, ref: SessionRef, cursor, limit: int) -> List[sqlite3.Row]:+ t = cursor.value.get("time", 0)+ last_id = cursor.value.get("id", "")+ # opencode message ids are ULID-like (monotonic with insertion), so (time, id)+ # is a stable order/cursor.+ return conn.execute(+ "SELECT id, time_created, data FROM message "+ "WHERE session_id = ? AND (time_created > ? OR (time_created = ? AND id > ?)) "+ "ORDER BY time_created, id LIMIT ?",+ (ref.locator, t, t, last_id, limit),+ ).fetchall()++ def row_complete(self, conn, row) -> bool:+ # A message row may exist before its `part` text is flushed; only advance the+ # cursor past it once it is final, so late-arriving parts are never skipped.+ try:+ data = json.loads(row["data"])+ except (ValueError, TypeError):+ return True+ if data.get("role") == "user":+ return True+ return bool((data.get("time") or {}).get("completed") or data.get("finish"))++ def rows_to_messages(self, conn, ref: SessionRef, rows) -> List[NormalizedMessage]:+ out: List[NormalizedMessage] = []+ for row in rows:+ try:+ data = json.loads(row["data"])+ except (ValueError, TypeError):+ continue+ role = data.get("role")+ if role not in ("user", "assistant"):+ continue++ text = self._reassemble_text(conn, row["id"])+ if not text:+ continue++ if role == "assistant":+ model = data.get("modelID") or ref.meta.get("session_model")+ peer = self.assistant_peer(model, data.get("providerID"))+ else:+ model = data.get("modelID")+ peer = self.user_peer(cwd=ref.meta.get("cwd"))+ out.append(+ NormalizedMessage(+ role=role,+ text=text,+ created_at=iso_from_epoch_ms(row["time_created"]),+ peer_id=peer,+ meta={+ "model": data.get("modelID"),+ "provider": data.get("providerID"),+ "cwd": ref.meta.get("cwd"),+ },+ )+ )+ return out++ @staticmethod+ def _reassemble_text(conn, message_id: str) -> str:+ parts = conn.execute(+ "SELECT data FROM part WHERE message_id = ? ORDER BY time_created, id",+ (message_id,),+ ).fetchall()+ chunks: List[str] = []+ for p in parts:+ try:+ pd = json.loads(p["data"])+ except (ValueError, TypeError):+ continue+ if pd.get("type") == "text":+ text = pd.get("text")+ if isinstance(text, str) and text.strip():+ chunks.append(text.strip())+ return "\n".join(chunks).strip()diff --git a/openviking_cli/server_bootstrap.py b/openviking_cli/server_bootstrap.pyindex 523f6606b..51d96eb69 100644--- a/openviking_cli/server_bootstrap.py+++ b/openviking_cli/server_bootstrap.py@@ -51,6 +51,14 @@ def main():with bind_log_execution_trace():sys.exit(doctor_main())+ # `openviking-server ingest ...` runs the local-log ingestion CLI (client-side).+ if len(sys.argv) > 1 and sys.argv[1] == "ingest":+ from openviking.ingest.cli import app as ingest_app++ with bind_log_execution_trace():+ ingest_app(args=sys.argv[2:], prog_name="openviking-server ingest")+ return+from openviking.server.bootstrap import main as _real_mainwith bind_log_execution_trace():diff --git a/openviking_cli/utils/config/consts.py b/openviking_cli/utils/config/consts.pyindex 883a02d9e..7c326b477 100644--- a/openviking_cli/utils/config/consts.py+++ b/openviking_cli/utils/config/consts.py@@ -74,6 +74,17 @@ OPENVIKING_GO_PATH_ENV = "OPENVIKING_GO_PATH"OPENVIKING_GOPATH_ENV = "OPENVIKING_GOPATH"OPENVIKING_GOPROXY_ENV = "OPENVIKING_GOPROXY"+# =============================================================================+# Conversation-log ingest (openviking-server ingest)+# =============================================================================++OPENVIKING_INGEST_ENABLED_ENV = "OPENVIKING_INGEST_ENABLED"+OPENVIKING_INGEST_SERVER_URL_ENV = "OPENVIKING_INGEST_SERVER_URL"+OPENVIKING_INGEST_API_KEY_ENV = "OPENVIKING_INGEST_API_KEY"++# Per-(harness, session) read-cursor state lives here (NEVER under tool-specific dirs).+DEFAULT_INGEST_STATE_DIR = DEFAULT_CONFIG_DIR / "ingest"+# =============================================================================# Default filenames# =============================================================================diff --git a/openviking_cli/utils/config/ingest_config.py b/openviking_cli/utils/config/ingest_config.pynew file mode 100644index 000000000..bf4e3fc81--- /dev/null+++ b/openviking_cli/utils/config/ingest_config.py@@ -0,0 +1,139 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Configuration for the conversation-log ingest subsystem (``openviking-server ingest``).++Ingest parses local agent-harness conversation logs (Claude Code, Codex, OpenCode,+Hermes, OpenClaw, Cursor) and "replays" them through OpenViking's session pipeline.+See ``openviking/ingest/`` for the runtime.+"""++import os+from typing import Dict, List, Literal, Optional++from pydantic import BaseModel, Field, model_validator++from .consts import (+ OPENVIKING_INGEST_API_KEY_ENV,+ OPENVIKING_INGEST_ENABLED_ENV,+ OPENVIKING_INGEST_SERVER_URL_ENV,+)++IngestMode = Literal["off", "backfill", "watch", "both"]+++def _env_truthy(value: Optional[str]) -> Optional[bool]:+ if value is None:+ return None+ return value.strip().lower() in {"1", "true", "yes", "on"}+++class CommitPolicy(BaseModel):+ """When a replayed session is committed (which triggers OV memory extraction)."""++ commit_token_threshold: int = Field(+ default=6000,+ gt=0,+ description="Commit a session once its pending (un-archived) token count reaches this.",+ )+ commit_idle_seconds: float = Field(+ default=5.0,+ gt=0,+ description="In watch mode, commit a session after it has been idle this long.",+ )+ keep_recent_count: int = Field(+ default=0,+ ge=0,+ description="WM v2 sliding window: messages to retain live after a commit (0 = archive all).",+ )++ model_config = {"extra": "forbid"}+++class IngestHarnessConfig(BaseModel):+ """Per-harness ingest settings, keyed by registry name (e.g. ``claude_code``)."""++ enabled: bool = Field(default=False, description="Enable ingesting this harness.")+ mode: IngestMode = Field(+ default="backfill",+ description="off | backfill (one-shot historical) | watch (incremental) | both.",+ )+ paths: List[str] = Field(+ default_factory=list,+ description="Override the harness's default log discovery roots (file/dir/db paths).",+ )+ poll_interval_seconds: float = Field(+ default=5.0,+ gt=0,+ description="Incremental poll cadence for this harness (watch mode).",+ )+ user_field: str = Field(+ default="",+ description=(+ "For group-chat harnesses (hermes/openclaw): the log key holding the original "+ "username, used as the user-side peer_id. Empty = use the adapter default."+ ),+ )+ experimental: bool = Field(+ default=False,+ description="Opt into experimental adapters (e.g. cursor) that may be fragile.",+ )+ commit: CommitPolicy = Field(default_factory=CommitPolicy)++ model_config = {"extra": "forbid"}+++class IngestConfig(BaseModel):+ """Top-level ingest configuration (``ingest`` section of ``ov.conf``)."""++ enabled: bool = Field(default=False, description="Master switch for log ingestion.")+ server_url: str = Field(+ default="",+ description="OV server URL the ingest client targets. Empty => OPENVIKING_URL / localhost.",+ )+ api_key: str = Field(default="", description="API key for the target OV server.")+ account: str = Field(default="default", description="OV account that owns ingested sessions.")+ user: str = Field(default="default", description="OV user that owns ingested sessions.")+ state_dir: str = Field(+ default="",+ description="Where the read-cursor state DB lives. Empty => ~/.openviking/ingest.",+ )+ session_id_prefix: str = Field(+ default="import",+ description="OV session ids are '{prefix}__{harness}__{native_session_id}'.",+ )+ memory_policy: Dict = Field(+ default_factory=dict,+ description="Passthrough memory_policy for replayed sessions (empty => server default).",+ )+ harnesses: Dict[str, IngestHarnessConfig] = Field(+ default_factory=dict,+ description="Per-harness configuration, keyed by registry name.",+ )++ model_config = {"extra": "forbid"}++ @model_validator(mode="after")+ def _apply_env_overrides(self) -> "IngestConfig":+ # Operational toggles may come from the environment (deploy-time > config).+ enabled = _env_truthy(os.environ.get(OPENVIKING_INGEST_ENABLED_ENV))+ if enabled is not None:+ self.enabled = enabled+ server_url = os.environ.get(OPENVIKING_INGEST_SERVER_URL_ENV)+ if server_url:+ self.server_url = server_url+ api_key = os.environ.get(OPENVIKING_INGEST_API_KEY_ENV)+ if api_key:+ self.api_key = api_key+ return self++ def enabled_harnesses(self) -> Dict[str, IngestHarnessConfig]:+ """Harnesses that are turned on and not in 'off' mode.++ The master switch ``enabled`` gates everything: when it is false, no harness+ is active regardless of its own ``enabled`` flag.+ """+ if not self.enabled:+ return {}+ return {+ name: cfg for name, cfg in self.harnesses.items() if cfg.enabled and cfg.mode != "off"+ }diff --git a/openviking_cli/utils/config/open_viking_config.py b/openviking_cli/utils/config/open_viking_config.pyindex 8b6a4b970..45886023c 100644--- a/openviking_cli/utils/config/open_viking_config.py+++ b/openviking_cli/utils/config/open_viking_config.py@@ -23,6 +23,7 @@ from .embedding_config import EmbeddingConfigfrom .encryption_config import EncryptionConfigfrom .git_config import GitConfigfrom .grep_config import GrepConfig+from .ingest_config import IngestConfigfrom .log_config import LogConfigfrom .memory_config import MemoryConfigfrom .oauth_config import OAuthConfig@@ -321,6 +322,11 @@ class OpenVikingConfig(BaseModel):description="Prompt template configuration",)+ ingest: IngestConfig = Field(+ default_factory=IngestConfig,+ description="Conversation-log ingest (openviking-server ingest) configuration",+ )+model_config = {"arbitrary_types_allowed": True, "extra": "forbid"}@classmethoddiff --git a/pyproject.toml b/pyproject.tomlindex 8b2fdca8f..e52293008 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -207,7 +207,7 @@ Issues = "https://github.com/volcengine/openviking/issues"[project.scripts]ov = "openviking_cli.rust_cli:main" # Rust CLI 入口(极简包装器)openviking = "openviking_cli.rust_cli:main" # Rust CLI 入口(放弃 python CLI)-openviking-server = "openviking_cli.server_bootstrap:main"+openviking-server = "openviking_cli.server_bootstrap:main" # also: `openviking-server ingest ...`vikingbot = "vikingbot.cli.commands:app"[tool.setuptools_scm]diff --git a/tests/ingest/__init__.py b/tests/ingest/__init__.pynew file mode 100644index 000000000..236633478--- /dev/null+++ b/tests/ingest/__init__.py@@ -0,0 +1,2 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0diff --git a/tests/ingest/conftest.py b/tests/ingest/conftest.pynew file mode 100644index 000000000..8760a0f8e--- /dev/null+++ b/tests/ingest/conftest.py@@ -0,0 +1,24 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Test isolation for the ingest suite.++``resolve_git_human_peer`` shells out to ``git config``, which otherwise reads the+developer machine's GLOBAL identity and makes human-peer resolution non-deterministic.+Neutralize git config discovery so user-side peers fall back to the configured value.+"""++import pytest++from openviking.ingest import peer++[email protected](autouse=True)+def _isolate_git_identity(tmp_path_factory, monkeypatch):+ empty = tmp_path_factory.mktemp("gitcfg") / "empty.cfg"+ empty.write_text("")+ monkeypatch.setenv("GIT_CONFIG_GLOBAL", str(empty))+ monkeypatch.setenv("GIT_CONFIG_SYSTEM", str(empty))+ monkeypatch.setenv("GIT_CONFIG_NOSYSTEM", "1")+ peer._GIT_PEER_CACHE.clear()+ yield+ peer._GIT_PEER_CACHE.clear()diff --git a/tests/ingest/test_config.py b/tests/ingest/test_config.pynew file mode 100644index 000000000..4cdd14c5e--- /dev/null+++ b/tests/ingest/test_config.py@@ -0,0 +1,40 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""IngestConfig gating + value validation."""++import pytest+from pydantic import ValidationError++from openviking_cli.utils.config.ingest_config import (+ CommitPolicy,+ IngestConfig,+ IngestHarnessConfig,+)+++def test_master_switch_gates_all_harnesses():+ enabled_h = {"claude_code": IngestHarnessConfig(enabled=True, mode="both")}+ assert IngestConfig(enabled=False, harnesses=enabled_h).enabled_harnesses() == {}+ assert list(IngestConfig(enabled=True, harnesses=enabled_h).enabled_harnesses()) == [+ "claude_code"+ ]+++def test_off_mode_excluded():+ cfg = IngestConfig(+ enabled=True,+ harnesses={+ "claude_code": IngestHarnessConfig(enabled=True, mode="off"),+ "codex": IngestHarnessConfig(enabled=False, mode="both"),+ },+ )+ assert cfg.enabled_harnesses() == {}+++def test_positive_value_validation():+ with pytest.raises(ValidationError):+ IngestHarnessConfig(poll_interval_seconds=0)+ with pytest.raises(ValidationError):+ CommitPolicy(commit_token_threshold=0)+ with pytest.raises(ValidationError):+ CommitPolicy(commit_idle_seconds=-1)diff --git a/tests/ingest/test_cursor_store.py b/tests/ingest/test_cursor_store.pynew file mode 100644index 000000000..f063d284f--- /dev/null+++ b/tests/ingest/test_cursor_store.py@@ -0,0 +1,43 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""CursorStore persistence / resume / accumulation."""++from openviking.ingest.cursor_store import CursorStore+from openviking.ingest.models import BYTE_OFFSET, Cursor+++def test_upsert_accumulate_and_resume(tmp_path):+ store = CursorStore(tmp_path)+ c1 = Cursor(BYTE_OFFSET, {"offset": 10, "inode": 1})+ store.upsert("claude_code", "s1", "import__claude_code__s1", c1, appended_delta=3, locator="/x")+ rec = store.get("claude_code", "s1")+ assert rec.cursor.value == {"offset": 10, "inode": 1}+ assert rec.last_appended_count == 3+ assert rec.ov_session_id == "import__claude_code__s1"+ assert rec.last_committed_at is None++ # advance + accumulate+ c2 = Cursor(BYTE_OFFSET, {"offset": 42, "inode": 1})+ store.upsert("claude_code", "s1", "import__claude_code__s1", c2, appended_delta=2)+ rec = store.get("claude_code", "s1")+ assert rec.cursor.value["offset"] == 42+ assert rec.last_appended_count == 5 # 3 + 2+ assert rec.locator == "/x" # preserved across upsert+ store.close()++ # resume from a fresh store on the same dir+ store2 = CursorStore(tmp_path)+ rec2 = store2.get("claude_code", "s1")+ assert rec2.cursor.value["offset"] == 42+ assert rec2.last_appended_count == 5+ store2.close()+++def test_commit_marks_timestamp(tmp_path):+ store = CursorStore(tmp_path)+ c = Cursor(BYTE_OFFSET, {"offset": 1})+ store.upsert("codex", "s", "ov", c, appended_delta=1)+ assert store.get("codex", "s").last_committed_at is None+ store.upsert("codex", "s", "ov", c, pending_tokens=0, committed=True)+ assert store.get("codex", "s").last_committed_at is not None+ store.close()diff --git a/tests/ingest/test_jsonl_incremental.py b/tests/ingest/test_jsonl_incremental.pynew file mode 100644index 000000000..91b68b743--- /dev/null+++ b/tests/ingest/test_jsonl_incremental.py@@ -0,0 +1,61 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Incremental byte-offset cursor behavior for append-only JSONL sources."""++import json+import os++from openviking.ingest.sources.hermes import HermesSource+from openviking_cli.utils.config.ingest_config import IngestHarnessConfig+++def _src(root):+ return HermesSource(IngestHarnessConfig(enabled=True, paths=[str(root)]), fallback_user="t")+++def _rec(role, content):+ return json.dumps({"role": role, "content": content})+++def test_partial_trailing_line_not_consumed(tmp_path):+ root = tmp_path / "sessions"+ root.mkdir(parents=True)+ f = root / "s.jsonl"+ # two complete lines + a partial (no trailing newline) line+ f.write_text(_rec("user", "a") + "\n" + _rec("assistant", "b") + "\n" + _rec("user", "cc"))++ src = _src(root)+ ref = next(iter(src.discover_sessions()))+ msgs, cursor = src.read_messages(ref, None)+ assert [m.text for m in msgs] == ["a", "b"] # partial line withheld++ # complete the partial line and append another complete line+ with open(f, "a") as fh:+ fh.write("\n" + _rec("assistant", "d") + "\n")++ msgs2, cursor2 = src.read_messages(ref, cursor)+ assert [m.text for m in msgs2] == ["cc", "d"]++ # nothing new on a third read+ msgs3, _ = src.read_messages(ref, cursor2)+ assert msgs3 == []+++def test_rotation_resets_cursor(tmp_path):+ root = tmp_path / "sessions"+ root.mkdir(parents=True)+ f = root / "s.jsonl"+ f.write_text(_rec("user", "old1") + "\n" + _rec("assistant", "old2") + "\n")++ src = _src(root)+ ref = next(iter(src.discover_sessions()))+ msgs, cursor = src.read_messages(ref, None)+ assert [m.text for m in msgs] == ["old1", "old2"]+ assert "inode" in cursor.value++ # Replace the file with brand-new, shorter content (new inode).+ os.remove(f)+ f.write_text(_rec("user", "new1") + "\n")++ msgs2, _ = src.read_messages(ref, cursor)+ assert [m.text for m in msgs2] == ["new1"] # rotation detected -> re-read from 0diff --git a/tests/ingest/test_parsers.py b/tests/ingest/test_parsers.pynew file mode 100644index 000000000..3b7d83e60--- /dev/null+++ b/tests/ingest/test_parsers.py@@ -0,0 +1,231 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Adapter parsing tests against synthetic per-harness fixtures."""++import json+import sqlite3++from openviking.ingest.sources.claude_code import ClaudeCodeSource+from openviking.ingest.sources.codex import CodexSource+from openviking.ingest.sources.hermes import HermesSource+from openviking.ingest.sources.openclaw import OpenClawSource+from openviking.ingest.sources.opencode import OpenCodeSource+from openviking_cli.utils.config.ingest_config import IngestHarnessConfig+++def _write_jsonl(path, records):+ path.parent.mkdir(parents=True, exist_ok=True)+ with open(path, "w") as f:+ for r in records:+ f.write(json.dumps(r) + "\n")+++def _cfg(root, **kw):+ return IngestHarnessConfig(enabled=True, paths=[str(root)], **kw)+++def _read_all(source):+ refs = list(source.discover_sessions())+ assert len(refs) == 1+ msgs, cursor = source.read_messages(refs[0], None)+ return refs[0], msgs, cursor+++def test_claude_code(tmp_path):+ root = tmp_path / "projects"+ _write_jsonl(+ root / "slug" / "sess-1.jsonl",+ [+ {"type": "queue-operation"}, # dropped (non-message)+ {+ "type": "user",+ "cwd": str(tmp_path),+ "timestamp": "2026-06-01T00:00:00Z",+ "message": {"role": "user", "content": "hello there"},+ },+ {+ "type": "assistant",+ "timestamp": "2026-06-01T00:00:01Z",+ "message": {+ "role": "assistant",+ "model": "claude-opus-4-8",+ "content": [+ {"type": "text", "text": "hi!"},+ {"type": "tool_use", "name": "Read"}, # dropped+ ],+ },+ },+ { # sub-agent record -> dropped+ "type": "assistant",+ "isSidechain": True,+ "message": {"role": "assistant", "content": [{"type": "text", "text": "side"}]},+ },+ ],+ )+ src = ClaudeCodeSource(_cfg(root), fallback_user="tester")+ ref, msgs, _ = _read_all(src)+ assert ref.native_session_id == "sess-1"+ assert [(m.role, m.text) for m in msgs] == [("user", "hello there"), ("assistant", "hi!")]+ assert msgs[1].peer_id == "claude_code__claude-opus-4-8"+ assert msgs[0].peer_id == "tester" # no git repo -> configured fallback+++def test_codex(tmp_path):+ root = tmp_path / "sessions"+ _write_jsonl(+ root / "2026" / "06" / "25" / "rollout-x-abc.jsonl",+ [+ {+ "type": "session_meta",+ "timestamp": "2026-06-25T00:00:00Z",+ "payload": {"id": "codex-sess", "model_provider": "openai", "cwd": str(tmp_path)},+ },+ {+ "type": "response_item",+ "timestamp": "2026-06-25T00:00:01Z",+ "payload": {+ "type": "message",+ "role": "developer", # dropped boilerplate+ "content": [{"type": "input_text", "text": "system stuff"}],+ },+ },+ {+ "type": "response_item",+ "timestamp": "2026-06-25T00:00:02Z",+ "payload": {+ "type": "message",+ "role": "user",+ "content": [{"type": "input_text", "text": "fix the bug"}],+ },+ },+ {+ "type": "response_item",+ "timestamp": "2026-06-25T00:00:03Z",+ "payload": {+ "type": "message",+ "role": "assistant",+ "content": [{"type": "output_text", "text": "done"}],+ },+ },+ {"type": "response_item", "payload": {"type": "reasoning"}}, # dropped+ ],+ )+ src = CodexSource(_cfg(root), fallback_user="tester")+ ref, msgs, _ = _read_all(src)+ assert ref.native_session_id == "codex-sess"+ assert [(m.role, m.text) for m in msgs] == [("user", "fix the bug"), ("assistant", "done")]+ assert msgs[1].peer_id == "codex__openai"+++def test_hermes_group_username(tmp_path):+ root = tmp_path / "sessions"+ _write_jsonl(+ root / "grp.jsonl",+ [+ {"role": "session_meta", "model": "doubao-x", "platform": "feishu"},+ {+ "role": "user",+ "content": "hi",+ "timestamp": "2026-06-01T00:00:00Z",+ "sender": "alice",+ },+ {"role": "assistant", "content": "hello", "timestamp": "2026-06-01T00:00:01Z"},+ ],+ )+ src = HermesSource(_cfg(root, user_field="sender"), fallback_user="tester")+ _, msgs, _ = _read_all(src)+ assert msgs[0].peer_id == "alice" # original username from the log+ assert msgs[1].peer_id == "hermes__doubao-x"+++def test_openclaw_assistant_model(tmp_path):+ root = tmp_path / "agents"+ _write_jsonl(+ root / "main" / "sessions" / "oc.jsonl",+ [+ {"type": "session", "id": "oc"},+ {+ "type": "message",+ "timestamp": "2026-06-01T00:00:00Z",+ "message": {"role": "user", "content": [{"type": "text", "text": "run it"}]},+ },+ {+ "type": "message",+ "timestamp": "2026-06-01T00:00:01Z",+ "message": {+ "role": "assistant",+ "provider": "ark",+ "model": "doubao-x",+ "content": [+ {"type": "thinking", "thinking": "hmm"}, # dropped+ {"type": "text", "text": "ok"},+ ],+ },+ },+ ],+ )+ src = OpenClawSource(_cfg(root, user_field="sender"), fallback_user="tester")+ _, msgs, _ = _read_all(src)+ assert [(m.role, m.text) for m in msgs] == [("user", "run it"), ("assistant", "ok")]+ assert msgs[1].peer_id == "openclaw__ark__doubao-x"+++def test_opencode_text_from_part_table(tmp_path):+ db = tmp_path / "opencode.db"+ conn = sqlite3.connect(str(db))+ conn.executescript(+ """+ CREATE TABLE session (id TEXT, title TEXT, directory TEXT, model TEXT, time_created INT);+ CREATE TABLE message (id TEXT, session_id TEXT, time_created INT, data TEXT);+ CREATE TABLE part (id TEXT, message_id TEXT, session_id TEXT, time_created INT, data TEXT);+ """+ )+ conn.execute(+ "INSERT INTO session VALUES (?,?,?,?,?)",+ ("ses_1", "demo", str(tmp_path), "m", 1000),+ )+ conn.execute(+ "INSERT INTO message VALUES (?,?,?,?)",+ ("msg_u", "ses_1", 1773044814194, json.dumps({"role": "user"})),+ )+ conn.execute(+ "INSERT INTO message VALUES (?,?,?,?)",+ (+ "msg_a",+ "ses_1",+ 1773044819959,+ json.dumps(+ {+ "role": "assistant",+ "modelID": "glm-4.7",+ "providerID": "tiktok",+ "finish": "stop", # mark complete so the cursor advances past it+ }+ ),+ ),+ )+ # text lives in the part table, not message.data+ conn.execute(+ "INSERT INTO part VALUES (?,?,?,?,?)",+ ("p1", "msg_u", "ses_1", 1, json.dumps({"type": "text", "text": "hello"})),+ )+ conn.execute(+ "INSERT INTO part VALUES (?,?,?,?,?)",+ ("p2", "msg_a", "ses_1", 1, json.dumps({"type": "step-start"})), # not text+ )+ conn.execute(+ "INSERT INTO part VALUES (?,?,?,?,?)",+ ("p3", "msg_a", "ses_1", 2, json.dumps({"type": "text", "text": "world"})),+ )+ conn.commit()+ conn.close()++ src = OpenCodeSource(_cfg(db), fallback_user="tester")+ refs = list(src.discover_sessions())+ assert len(refs) == 1+ msgs, cursor = src.read_messages(refs[0], None)+ assert [(m.role, m.text) for m in msgs] == [("user", "hello"), ("assistant", "world")]+ assert msgs[1].peer_id == "opencode__tiktok__glm-4.7"+ # cursor advanced; re-read returns nothing new+ msgs2, _ = src.read_messages(refs[0], cursor)+ assert msgs2 == []diff --git a/tests/ingest/test_peer.py b/tests/ingest/test_peer.pynew file mode 100644index 000000000..ab06db8e6--- /dev/null+++ b/tests/ingest/test_peer.py@@ -0,0 +1,33 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""peer_id resolution: assistant model peers, human/git peers, group usernames."""++from openviking.ingest.peer import (+ assistant_peer_id,+ resolve_git_human_peer,+ safe_external_peer,+)+++def test_assistant_peer_with_and_without_provider():+ assert assistant_peer_id("claude_code", "claude-opus-4-8") == "claude_code__claude-opus-4-8"+ assert assistant_peer_id("opencode", "glm-4.7", "tiktok") == "opencode__tiktok__glm-4.7"+ assert assistant_peer_id("codex", None) == "codex__unknown"+++def test_external_peer_readable_and_sanitized():+ assert safe_external_peer("alice") == "alice"+ assert safe_external_peer("[email protected]") == "[email protected]"+ assert safe_external_peer("foo/bar baz") == "foo-bar-baz"+ assert safe_external_peer("") is None+++def test_external_peer_non_ascii_falls_back_to_ext():+ pid = safe_external_peer("杨冠姝")+ assert pid is not None and pid.startswith("ext-")+++def test_git_human_peer_falls_back_without_repo(tmp_path):+ # tmp_path is not a git repo -> configured fallback is used+ assert resolve_git_human_peer(str(tmp_path), "me") == "me"+ assert resolve_git_human_peer(None, "me") == "me"diff --git a/tests/ingest/test_replay.py b/tests/ingest/test_replay.pynew file mode 100644index 000000000..ddc8af53f--- /dev/null+++ b/tests/ingest/test_replay.py@@ -0,0 +1,194 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Replay driver: SDK-client chunking/ensure, batch append, commit, and crash reconcile."""++from openviking.ingest.cursor_store import CursorStore+from openviking.ingest.models import BYTE_OFFSET, Cursor, NormalizedMessage, SessionRef+from openviking.ingest.replay import ConversationReplayClient, SessionReplayer+from openviking_cli.exceptions import NotFoundError+++class _FakeSDK:+ """Minimal stand-in for ov.AsyncHTTPClient."""++ def __init__(self, existing=()):+ self.existing = set(existing)+ self.created = []+ self.batches = []+ self.commits = []++ async def get_session(self, sid, *, auto_create=False):+ if sid not in self.existing:+ raise NotFoundError(sid, "session")+ return {"session_id": sid, "pending_tokens": 0, "message_count": 0}++ async def create_session(self, session_id=None, memory_policy=None):+ self.created.append(session_id)+ self.existing.add(session_id)+ return {"session_id": session_id}++ async def batch_add_messages(self, sid, messages, telemetry=False):+ self.batches.append((sid, len(messages)))+ return {"added": len(messages)}++ async def commit_session(self, sid, *, keep_recent_count=0):+ self.commits.append((sid, keep_recent_count))+ return {"task_id": "t", "archived": True}+++async def test_ensure_session_get_or_create():+ sdk = _FakeSDK()+ client = ConversationReplayClient(sdk)+ await client.ensure_session("s1") # not present -> create+ assert sdk.created == ["s1"]+ await client.ensure_session("s1") # now present -> no second create+ assert sdk.created == ["s1"]+++async def test_append_chunks_at_100():+ sdk = _FakeSDK(existing={"s1"})+ client = ConversationReplayClient(sdk)+ added = await client.append("s1", [{"role": "user", "content": str(i)} for i in range(250)])+ assert added == 250+ assert [n for _, n in sdk.batches] == [100, 100, 50]+++class _FakeReplay:+ """Stand-in for ConversationReplayClient used by SessionReplayer."""++ def __init__(self, pending=0, count=0):+ self.pending = pending+ self.count = count+ self.ensured = []+ self.appended = []+ self.committed = []++ async def ensure_session(self, sid, memory_policy=None):+ self.ensured.append(sid)++ async def append(self, sid, payloads):+ self.appended.append((sid, len(payloads)))+ self.count += len(payloads)+ return len(payloads)++ async def message_count(self, sid):+ return self.count++ async def pending_tokens(self, sid):+ return self.pending++ async def commit(self, sid, keep_recent_count=0):+ self.committed.append(sid)++ async def delete(self, sid):+ pass+++def _msgs():+ return [+ NormalizedMessage(role="user", text="hi", peer_id="me"),+ NormalizedMessage(role="assistant", text="yo", peer_id="claude_code__m"),+ ]+++def _ref(sid="s1"):+ return SessionRef(harness="claude_code", native_session_id=sid, locator="/f")+++async def test_append_batch_confirms_cursor_and_marks_needs_commit(tmp_path):+ store = CursorStore(tmp_path)+ fake = _FakeReplay()+ replayer = SessionReplayer(fake, store, session_id_prefix="import")+ ref = _ref()+ added = await replayer.append_batch(+ "claude_code",+ ref,+ _msgs(),+ Cursor(BYTE_OFFSET, {"offset": 0}),+ Cursor(BYTE_OFFSET, {"offset": 99}),+ )+ assert added == 2+ assert fake.ensured == ["import__claude_code__s1"]+ rec = store.get("claude_code", "s1")+ assert rec.cursor.value["offset"] == 99 # confirmed cursor+ assert rec.needs_commit is True+ assert fake.committed == [] # append does not commit+++async def test_reconcile_confirms_landed_batch(tmp_path):+ store = CursorStore(tmp_path)+ sid = "import__claude_code__s1"+ store.set_pending(+ "claude_code",+ "s1",+ sid,+ Cursor(BYTE_OFFSET, {"offset": 0}),+ Cursor(BYTE_OFFSET, {"offset": 50}),+ pend_count=2,+ baseline=0,+ )+ fake = _FakeReplay(count=2) # server already has the 2 messages -> batch landed+ replayer = SessionReplayer(fake, store)+ await replayer.reconcile("claude_code", _ref())+ rec = store.get("claude_code", "s1")+ assert rec.cursor.value["offset"] == 50 # confirmed without re-append+ assert rec.pending_count == 0+ assert rec.needs_commit is True+++async def test_reconcile_drops_unlanded_batch(tmp_path):+ store = CursorStore(tmp_path)+ sid = "import__claude_code__s2"+ store.set_pending(+ "claude_code",+ "s2",+ sid,+ Cursor(BYTE_OFFSET, {"offset": 0}),+ Cursor(BYTE_OFFSET, {"offset": 50}),+ pend_count=2,+ baseline=0,+ )+ fake = _FakeReplay(count=0) # batch did not land+ replayer = SessionReplayer(fake, store)+ await replayer.reconcile("claude_code", _ref("s2"))+ rec = store.get("claude_code", "s2")+ assert rec.cursor.value["offset"] == 0 # NOT advanced -> will re-read & re-append+ assert rec.pending_count == 0 # intent cleared+++async def test_threshold_commit(tmp_path):+ store = CursorStore(tmp_path)+ ref = _ref()+ low = SessionReplayer(_FakeReplay(pending=10), store)+ assert await low.maybe_commit_on_threshold("claude_code", ref, threshold=6000) is False+ assert low.client.committed == []++ high = SessionReplayer(_FakeReplay(pending=9000), store)+ assert await high.maybe_commit_on_threshold("claude_code", ref, threshold=6000) is True+ assert high.client.committed == ["import__claude_code__s1"]+++async def test_commit_if_needed_skips_when_nothing(tmp_path):+ store = CursorStore(tmp_path)+ replayer = SessionReplayer(_FakeReplay(pending=0), store)+ assert await replayer.commit_if_needed("codex", _ref("s")) is False+ assert replayer.client.committed == []+++async def test_commit_if_needed_commits_when_needs_commit_and_pending(tmp_path):+ store = CursorStore(tmp_path)+ # appended-but-uncommitted: needs_commit set in store, server still has pending tokens+ store.set_pending(+ "claude_code",+ "s1",+ "import__claude_code__s1",+ Cursor(BYTE_OFFSET, {"offset": 0}),+ Cursor(BYTE_OFFSET, {"offset": 10}),+ 1,+ 0,+ )+ store.confirm_append("claude_code", "s1", Cursor(BYTE_OFFSET, {"offset": 10}), 1)+ replayer = SessionReplayer(_FakeReplay(pending=500), store)+ assert await replayer.commit_if_needed("claude_code", _ref()) is True+ assert replayer.client.committed == ["import__claude_code__s1"]+ assert store.get("claude_code", "s1").needs_commit is False