Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
M19: ai-memory bootstrap — pre-load existing project history
AkitaOnRailsa432a42ea1f4
diff --git a/Cargo.lock b/Cargo.lockindex e174aed7..c8cfe96f 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -72,6 +72,7 @@ dependencies = ["ai-memory-store","ai-memory-wiki","anyhow",+ "git2","jiff","rusqlite","schemars",diff --git a/README.md b/README.mdindex 17deb54f..3e64caf7 100644--- a/README.md+++ b/README.md@@ -109,6 +109,36 @@ When Claude Code or Codex compact their working context, thecompaction, the agent can recover the summary via `memory_recent`even though its raw history is gone.+### Adopting ai-memory mid-project: bootstrap++If you're installing ai-memory in a project you've been working on+for months, the wiki starts empty and the first few sessions are+net-zero — you're populating, not retrieving. `ai-memory bootstrap`+solves that by LLM-summarising your existing `git log`, README,+`docs/`, and module-level doc-comments into seed wiki pages.++```bash+# Run from your project's repo root (requires an LLM provider on the+# server). Default settings ingest everything; budget caps at 50k+# input tokens (~$0.04 with Kimi 2.6).+docker run --rm \+ -v ai-memory-data:/data \+ -v "$PWD:/repo" \+ -e AI_MEMORY_AUTH_TOKEN="$TOKEN" \+ -e AI_MEMORY_LLM_PROVIDER=anthropic \+ -e ANTHROPIC_API_KEY=sk-ant-... \+ akitaonrails/ai-memory:latest \+ bootstrap --repo-path /repo --workspace homelab --project myproj+```++Bootstrap produces a `wiki/bootstrap.md` manifest listing every page+generated + a one-paragraph rationale. Run with `--dry-run` first to+preview which sources would be sent without paying for the LLM call.+Re-running on the same project requires `--force`.++See [`docs/install.md`](docs/install.md#bootstrap-mid-project) for+the full flag reference + per-source priority order.+### Spelunking your own history```bashdiff --git a/crates/ai-memory-cli/src/cli.rs b/crates/ai-memory-cli/src/cli.rsindex 9c6b1f9e..3f3a10ff 100644--- a/crates/ai-memory-cli/src/cli.rs+++ b/crates/ai-memory-cli/src/cli.rs@@ -68,6 +68,66 @@ pub enum Command {/// config snippet. Replaces the "clone the repo + cargo build"/// workflow for users who never want a local Rust toolchain.SetupAgent(SetupAgentArgs),+ /// Pre-load an existing project's history into the wiki by+ /// LLM-summarising git log, README, docs/, and module headers+ /// into seed wiki pages. Run once when adopting ai-memory in a+ /// project that's been around for a while. Requires+ /// AI_MEMORY_LLM_PROVIDER configured on the server.+ Bootstrap(BootstrapArgs),+}++/// Arguments for `bootstrap`.+#[derive(Debug, Args)]+pub struct BootstrapArgs {+ /// Path of the project repo whose history we should ingest.+ /// Defaults to `git rev-parse --show-toplevel` resolved from the+ /// current directory (so running bootstrap from any subdir of the+ /// project works).+ #[arg(long)]+ pub repo_path: Option<PathBuf>,+ /// Workspace + project labels stamped on the generated pages.+ /// Match what your `ai-memory serve` uses so the bootstrap pages+ /// live in the same wiki silo as future captured sessions.+ #[arg(long, default_value = "default")]+ pub workspace: String,+ #[arg(long, default_value = "scratch")]+ pub project: String,+ /// Maximum total tokens of source text sent to the LLM in one+ /// run. When the collected sources exceed this, lower-priority+ /// inputs (older git commits, then code module headers, then+ /// docs) are dropped first. At Kimi's ~$3.49/M completion price+ /// a 50000-token cap caps the worst-case run at ~$0.20.+ #[arg(long, default_value_t = 50_000)]+ pub max_input_tokens: usize,+ /// Skip git-commit history ingestion.+ #[arg(long)]+ pub exclude_git: bool,+ /// Skip README ingestion.+ #[arg(long)]+ pub exclude_readme: bool,+ /// Skip docs/**/*.md ingestion.+ #[arg(long)]+ pub exclude_docs: bool,+ /// Skip code module headers (Rust `//!` doc-comments at the top+ /// of `**/*.rs` files).+ #[arg(long)]+ pub exclude_code: bool,+ /// git-log time filter (passed through to `git log --since`).+ /// Useful when a repo is years old and you only want recent+ /// history (e.g. `--since "180 days ago"`). Default: no limit.+ #[arg(long)]+ pub since: Option<String>,+ /// LLM-call dry run: collects sources, builds the prompt, and+ /// prints what *would* be sent — but never calls the provider+ /// and never writes to the wiki. Useful for verifying the source+ /// selection before paying for a real run.+ #[arg(long)]+ pub dry_run: bool,+ /// Re-bootstrap a project that already has a bootstrap manifest+ /// page. Without this flag, `bootstrap` refuses to run twice on+ /// the same project (the manifest is `wiki/bootstrap.md`).+ #[arg(long)]+ pub force: bool,}/// Arguments for `setup-agent`.diff --git a/crates/ai-memory-cli/src/commands/bootstrap.rs b/crates/ai-memory-cli/src/commands/bootstrap.rsnew file mode 100644index 00000000..3bab2383--- /dev/null+++ b/crates/ai-memory-cli/src/commands/bootstrap.rs@@ -0,0 +1,115 @@+//! `ai-memory bootstrap` — ingest an existing project's history.+//!+//! Thin CLI wrapper around `ai_memory_consolidate::Bootstrap`. The+//! heavy lifting (source collection, prioritisation, LLM call,+//! wiki write) lives in the consolidate crate; this file just+//! resolves CLI args into a `BootstrapConfig` and dispatches.+//!+//! Resolves the repo path via `git rev-parse --show-toplevel` when+//! `--repo-path` is omitted, so running from any subdirectory of+//! the target project works.++use std::path::PathBuf;+use std::sync::Arc;++use ai_memory_consolidate::{Bootstrap, BootstrapConfig};+use ai_memory_llm::{build_provider, provider_from_env};+use ai_memory_store::Store;+use ai_memory_wiki::Wiki;+use anyhow::{Context, Result, bail};+use tracing::info;++use crate::cli::BootstrapArgs;+use crate::config::Config;++/// Run the `bootstrap` subcommand.+///+/// # Errors+/// Bails when an LLM provider isn't configured (bootstrap can't run+/// without one), when the resolved repo path isn't a git repo, when+/// the project was already bootstrapped without `--force`, or when+/// any source-collection / LLM / wiki write fails.+pub async fn run(config: &Config, args: BootstrapArgs) -> Result<()> {+ // ---- LLM provider — required ----+ let Some(llm_cfg) = provider_from_env()? else {+ bail!(+ "bootstrap requires an LLM provider. Set AI_MEMORY_LLM_PROVIDER \+ (and the matching API key) and try again."+ );+ };+ let llm = build_provider(llm_cfg).context("building LLM provider from env")?;+ info!(+ provider = llm.name(),+ model = llm.model(),+ "bootstrap LLM enabled"+ );++ // ---- repo path — auto-detect if absent ----+ let repo_path = match args.repo_path {+ Some(p) => p,+ None => resolve_repo_root().context("auto-detecting --repo-path via git rev-parse")?,+ };+ if !repo_path.join(".git").exists() {+ bail!(+ "repo path {} is not a git repository (looked for {}/.git)",+ repo_path.display(),+ repo_path.display()+ );+ }++ // ---- open store + wiki ----+ let store = Store::open(&config.data_dir)+ .with_context(|| format!("opening store at {}", config.data_dir.display()))?;+ let ws = store+ .writer+ .get_or_create_workspace(args.workspace.clone())+ .await?;+ let proj = store+ .writer+ .get_or_create_project(ws, args.project.clone(), None)+ .await?;+ let wiki = Wiki::new(&config.data_dir, store.writer.clone())?;++ // ---- run bootstrap ----+ let cfg = BootstrapConfig {+ repo_path: repo_path.clone(),+ workspace_id: ws,+ project_id: proj,+ max_input_tokens: args.max_input_tokens,+ include_git: !args.exclude_git,+ include_readme: !args.exclude_readme,+ include_docs: !args.exclude_docs,+ include_code: !args.exclude_code,+ since: args.since,+ dry_run: args.dry_run,+ force: args.force,+ };+ let bootstrap = Bootstrap {+ reader: store.reader.clone(),+ wiki,+ llm: Arc::clone(&llm),+ };+ let outcome = bootstrap.run(&cfg).await?;++ // ---- human-readable report ----+ let report = serde_json::to_string_pretty(&outcome)?;+ println!("{report}");+ Ok(())+}++/// `git rev-parse --show-toplevel` — finds the repo root from $PWD.+fn resolve_repo_root() -> Result<PathBuf> {+ let output = std::process::Command::new("git")+ .args(["rev-parse", "--show-toplevel"])+ .output()+ .context("running `git rev-parse --show-toplevel`")?;+ if !output.status.success() {+ bail!(+ "git rev-parse failed (cwd is not inside a git repository?). \+ stderr: {}",+ String::from_utf8_lossy(&output.stderr).trim(),+ );+ }+ let line = String::from_utf8_lossy(&output.stdout).trim().to_string();+ Ok(PathBuf::from(line))+}diff --git a/crates/ai-memory-cli/src/commands/mod.rs b/crates/ai-memory-cli/src/commands/mod.rsindex 5172afb7..c1d91ab9 100644--- a/crates/ai-memory-cli/src/commands/mod.rs+++ b/crates/ai-memory-cli/src/commands/mod.rs@@ -1,6 +1,7 @@//! Subcommand implementations.pub mod backup;+pub mod bootstrap;pub mod commit;pub mod embed;pub mod forget_sweep;diff --git a/crates/ai-memory-cli/src/main.rs b/crates/ai-memory-cli/src/main.rsindex 31b69b3e..971f32e8 100644--- a/crates/ai-memory-cli/src/main.rs+++ b/crates/ai-memory-cli/src/main.rs@@ -56,5 +56,6 @@ async fn main() -> Result<()> {Command::Embed(args) => commands::embed::run(&config, args).await,Command::GenerateAuthToken(args) => commands::generate_auth_token::run(&config, args),Command::SetupAgent(args) => commands::setup_agent::run(&config, args),+ Command::Bootstrap(args) => commands::bootstrap::run(&config, args).await,}}diff --git a/crates/ai-memory-consolidate/Cargo.toml b/crates/ai-memory-consolidate/Cargo.tomlindex 8ffa3a53..4f94c9cd 100644--- a/crates/ai-memory-consolidate/Cargo.toml+++ b/crates/ai-memory-consolidate/Cargo.toml@@ -13,6 +13,7 @@ ai-memory-core.workspace = trueai-memory-store.workspace = trueai-memory-wiki.workspace = trueai-memory-llm.workspace = true+git2.workspace = trueanyhow.workspace = truethiserror.workspace = trueserde.workspace = truediff --git a/crates/ai-memory-consolidate/src/bootstrap.rs b/crates/ai-memory-consolidate/src/bootstrap.rsnew file mode 100644index 00000000..e1c7231e--- /dev/null+++ b/crates/ai-memory-consolidate/src/bootstrap.rs@@ -0,0 +1,872 @@+//! Pre-load an existing project's history into the wiki.+//!+//! The use case: a developer has been working on a project for+//! months. They install ai-memory today. The wiki is empty. The+//! first few sessions are net-zero (you're populating, not+//! retrieving) because none of the project's prior decisions /+//! gotchas / architecture notes are in ai-memory yet — even though+//! they're written down in `git log`, README, `docs/`, and module+//! doc-comments.+//!+//! `bootstrap` does a one-shot LLM-summarisation of those sources+//! into seed wiki pages so the project starts with warm context.+//! Pages are tagged `bootstrapped_at: <ts>` in their frontmatter so+//! a future lint pass can treat them as lower-confidence than+//! session-grown pages if needed.+//!+//! ## Cost model+//!+//! Input is capped via `max_input_tokens` (default 50k). We+//! estimate token count locally at chars/4 (the standard rough+//! heuristic) and drop lower-priority sources first when over+//! budget. At Kimi's $0.73/$3.49 per M, 50k input ≈ $0.04;+//! generated output (1-2k tokens) ≈ $0.007. Worst case under+//! $0.20 per run.+//!+//! ## Idempotency+//!+//! First write produces `wiki/bootstrap.md` recording the run+//! (timestamp, source counts, generated page count). Re-running+//! refuses unless `--force` is passed. The user can always delete+//! `wiki/bootstrap.md` (and the generated pages) to reset.++use std::path::{Path, PathBuf};+use std::sync::Arc;++use ai_memory_core::{PagePath, ProjectId, Tier, WorkspaceId};+use ai_memory_llm::{ChatMessage, ChatRequest, LlmProvider, Role, complete_structured};+use ai_memory_store::ReaderPool;+use ai_memory_wiki::{Wiki, WritePageRequest};+use jiff::Timestamp;+use schemars::JsonSchema;+use serde::{Deserialize, Serialize};+use thiserror::Error;+use tracing::{debug, info, warn};++/// Rough characters-per-token estimate used for budget enforcement.+/// 4 is the standard heuristic for English prose (cl100k, gpt-4+/// tokenizer family). Don't rely on it for billing math — it's+/// only used to decide which sources to drop.+const CHARS_PER_TOKEN: usize = 4;++/// Errors returned from [`Bootstrap::run`].+#[derive(Debug, Error)]+pub enum BootstrapError {+ /// Project repo path doesn't exist or isn't a git repo.+ #[error("repo path {0} is not a git repository")]+ NotARepo(PathBuf),+ /// The wiki already has a `bootstrap.md` manifest. Re-run with+ /// `--force` to overwrite.+ #[error("project already bootstrapped (wiki/bootstrap.md exists). Pass --force to re-run.")]+ AlreadyBootstrapped,+ /// All source categories were excluded; nothing to do.+ #[error("no input sources selected; remove at least one --exclude-* flag")]+ NoSources,+ /// LLM call failed.+ #[error(transparent)]+ Llm(#[from] ai_memory_llm::LlmError),+ /// Wiki write failed.+ #[error(transparent)]+ Wiki(#[from] ai_memory_wiki::WikiError),+ /// Store read failed (used by the idempotency check).+ #[error(transparent)]+ Store(#[from] ai_memory_store::StoreError),+ /// IO error (reading docs, walking the repo, …).+ #[error(transparent)]+ Io(#[from] std::io::Error),+ /// libgit2 error from git_log() reading the commit history.+ #[error(transparent)]+ Git(#[from] git2::Error),+}++/// What kind of source we collected text from.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]+pub enum SourceKind {+ /// A line summary of one commit.+ GitCommit,+ /// The repo root's README.md.+ Readme,+ /// A file under `docs/`.+ DocFile,+ /// Module-level `//!` doc-comments at the top of a `.rs` file.+ ModuleHeader,+ /// CLAUDE.md / AGENTS.md / similar project-rules file.+ ProjectRules,+}++impl SourceKind {+ /// Drop priority — sources with higher numeric priority are+ /// dropped FIRST when over budget. (We keep the most valuable+ /// inputs and shed the noise.)+ #[must_use]+ pub const fn drop_priority(self) -> u8 {+ match self {+ // Project rules are usually small but very high-signal —+ // always keep.+ Self::ProjectRules => 0,+ // README is the single most useful project doc.+ Self::Readme => 1,+ // docs/ second.+ Self::DocFile => 2,+ // Recent git commits keep.+ Self::GitCommit => 3,+ // Module headers are nice-to-have; first to drop.+ Self::ModuleHeader => 4,+ }+ }+}++/// One unit of source text fed to the LLM.+#[derive(Debug, Clone)]+pub struct BootstrapSource {+ /// Origin (used for prioritisation when over budget).+ pub kind: SourceKind,+ /// A short label, included in the LLM prompt to help the model+ /// distinguish sources. Examples: "git: feat: …", "README".+ pub label: String,+ /// The full text content fed to the LLM.+ pub text: String,+}++impl BootstrapSource {+ /// Estimated token count via chars/4.+ #[must_use]+ pub fn estimated_tokens(&self) -> usize {+ (self.label.len() + self.text.len() + 16).div_ceil(CHARS_PER_TOKEN)+ }+}++/// LLM-produced page describing one bootstrap output.+#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]+pub struct BootstrapPage {+ /// Relative wiki path. Use the conventions:+ /// - `concepts/<slug>.md` for evergreen architectural notes+ /// - `decisions/0001-<slug>.md` for ADR-shaped commits+ /// - `gotchas/<slug>.md` for failure-mode notes+ pub path: String,+ /// Page title (renders as H1).+ pub title: String,+ /// Markdown body. Don't include the frontmatter — we add it.+ pub body_markdown: String,+ /// Up to ~5 short tags.+ #[serde(default)]+ pub tags: Vec<String>,+}++/// LLM structured output: a batch of bootstrap pages.+#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]+pub struct BootstrapBatch {+ /// Pages to create.+ pub pages: Vec<BootstrapPage>,+ /// Brief one-paragraph note about what was processed, surfaced+ /// in the wiki/bootstrap.md manifest + the auto-commit message.+ #[serde(default)]+ pub rationale: String,+}++/// Outcome reported back to the CLI.+#[derive(Debug, Clone, Serialize)]+pub struct BootstrapOutcome {+ /// Number of sources collected (before any budget pruning).+ pub sources_collected: usize,+ /// Number of sources actually sent to the LLM (after pruning).+ pub sources_sent: usize,+ /// Number of sources dropped to stay under `max_input_tokens`.+ pub sources_dropped: usize,+ /// Token budget used by the chosen sources (best-effort estimate).+ pub estimated_input_tokens: usize,+ /// Pages written to the wiki (empty if dry_run).+ pub pages_written: Vec<String>,+ /// One-paragraph LLM-authored summary, mirrored into the manifest.+ pub rationale: String,+ /// True when the operation skipped the LLM call entirely.+ pub dry_run: bool,+}++/// Bootstrap configuration. Built by the CLI from `BootstrapArgs`+/// after resolving auto-detect defaults (repo path, workspace, etc.)+/// to concrete values.+#[derive(Debug, Clone)]+pub struct BootstrapConfig {+ /// Project repo root. Sources are read relative to here.+ pub repo_path: PathBuf,+ /// Workspace identifier the generated pages belong to.+ pub workspace_id: WorkspaceId,+ /// Project identifier the generated pages belong to.+ pub project_id: ProjectId,+ /// Token budget for LLM input; lower-priority sources are+ /// dropped first when over budget.+ pub max_input_tokens: usize,+ /// Include git-commit history.+ pub include_git: bool,+ /// Include `README.md` at the repo root.+ pub include_readme: bool,+ /// Include `docs/**/*.md`.+ pub include_docs: bool,+ /// Include Rust `//!` module-level doc-comments.+ pub include_code: bool,+ /// git-log `--since` filter; supports "N days ago" / "N years+ /// ago" / YYYY-MM-DD.+ pub since: Option<String>,+ /// Collect + estimate + prompt, but DON'T call the LLM and+ /// DON'T write to the wiki. Useful for pre-flight verification.+ pub dry_run: bool,+ /// Allow re-running even when `wiki/bootstrap.md` already exists.+ pub force: bool,+}++/// Bootstrap driver. Holds the LLM provider + wiki handle + reader+/// pool needed to ingest sources, summarise them, and write the+/// generated pages.+pub struct Bootstrap {+ /// Reader pool used by the idempotency check.+ pub reader: ReaderPool,+ /// Wiki handle the generated pages are written through.+ pub wiki: Wiki,+ /// LLM provider used to summarise sources into pages.+ pub llm: Arc<dyn LlmProvider>,+}++impl Bootstrap {+ /// Run the bootstrap pipeline end-to-end.+ ///+ /// # Errors+ /// Propagates [`BootstrapError`] for any of: missing repo, LLM+ /// failure, wiki write failure, or already-bootstrapped (without+ /// `--force`).+ pub async fn run(&self, cfg: &BootstrapConfig) -> Result<BootstrapOutcome, BootstrapError> {+ // ---- idempotency check ------------------------------------+ // The reader pool's recent_pages isn't (workspace, project)-+ // scoped, so we use the wiki's filesystem read as the cheap+ // existence check: if `wiki/bootstrap.md` parses, this+ // project was bootstrapped before.+ if !cfg.force {+ let manifest_path =+ PagePath::new("bootstrap.md").expect("hard-coded manifest path is valid");+ if self.wiki.read_page(&manifest_path).is_ok() {+ return Err(BootstrapError::AlreadyBootstrapped);+ }+ }++ // ---- source collection ------------------------------------+ let mut sources = Vec::<BootstrapSource>::new();+ if cfg.include_git {+ sources.extend(collect_git_commits(&cfg.repo_path, cfg.since.as_deref())?);+ }+ if cfg.include_readme {+ sources.extend(collect_readme(&cfg.repo_path)?);+ }+ if cfg.include_docs {+ sources.extend(collect_docs(&cfg.repo_path)?);+ }+ if cfg.include_code {+ sources.extend(collect_rust_module_headers(&cfg.repo_path)?);+ }+ sources.extend(collect_project_rules(&cfg.repo_path)?);++ if sources.is_empty() {+ return Err(BootstrapError::NoSources);+ }++ let collected = sources.len();+ let (kept, dropped, est_tokens) = prune_to_budget(sources, cfg.max_input_tokens);+ info!(+ collected,+ kept = kept.len(),+ dropped,+ est_tokens,+ "bootstrap sources prioritised + budget-capped",+ );++ // ---- dry run early-exits before the LLM call --------------+ if cfg.dry_run {+ return Ok(BootstrapOutcome {+ sources_collected: collected,+ sources_sent: kept.len(),+ sources_dropped: dropped,+ estimated_input_tokens: est_tokens,+ pages_written: Vec::new(),+ rationale: "(dry-run; LLM not invoked)".to_string(),+ dry_run: true,+ });+ }++ // ---- LLM call ---------------------------------------------+ let request = build_request(&kept);+ let batch: BootstrapBatch = complete_structured(&*self.llm, request).await?;++ // ---- write pages ------------------------------------------+ let now = Timestamp::now();+ let mut requests = Vec::with_capacity(batch.pages.len() + 1);+ let mut written_paths = Vec::with_capacity(batch.pages.len() + 1);+ for page in &batch.pages {+ let path = match PagePath::new(&page.path) {+ Ok(p) => p,+ Err(e) => {+ warn!(path = %page.path, error = %e, "skipping bootstrap page with invalid path");+ continue;+ }+ };+ written_paths.push(page.path.clone());+ requests.push(WritePageRequest {+ workspace_id: cfg.workspace_id,+ project_id: cfg.project_id,+ path,+ frontmatter: build_frontmatter(&page.title, &page.tags, now),+ body: page.body_markdown.clone(),+ tier: Tier::Semantic,+ pinned: false,+ title: Some(page.title.clone()),+ });+ }+ // Plus the manifest itself.+ let manifest_body = render_manifest_body(+ now,+ collected,+ kept.len(),+ dropped,+ est_tokens,+ &batch.rationale,+ &written_paths,+ );+ requests.push(WritePageRequest {+ workspace_id: cfg.workspace_id,+ project_id: cfg.project_id,+ path: PagePath::new("bootstrap.md").expect("static path"),+ frontmatter: serde_json::json!({+ "title": "Bootstrap manifest",+ "tier": "semantic",+ "bootstrapped_at": now.to_string(),+ "tags": ["bootstrap", "manifest"],+ }),+ body: manifest_body,+ tier: Tier::Semantic,+ pinned: true,+ title: Some("Bootstrap manifest".into()),+ });++ let _ids = self.wiki.apply_batch(requests).await?;+ let _ = self.wiki.commit_all(&format!(+ "bootstrap: ingested {collected} sources, wrote {} pages",+ written_paths.len() + 1,+ ));++ // Manifest is the last entry but conceptually first; list it.+ let mut out_paths = written_paths.clone();+ out_paths.push("bootstrap.md".to_string());++ Ok(BootstrapOutcome {+ sources_collected: collected,+ sources_sent: kept.len(),+ sources_dropped: dropped,+ estimated_input_tokens: est_tokens,+ pages_written: out_paths,+ rationale: batch.rationale,+ dry_run: false,+ })+ }+}++// --------------------------------------------------------------------+// Source collection+// --------------------------------------------------------------------++/// Read commits, format each as a one-paragraph entry. We include+/// only commits with a substantive body (more than ~120 chars+/// total) — drive-by typo-fix commits aren't worth tokens.+fn collect_git_commits(+ repo_path: &Path,+ since: Option<&str>,+) -> Result<Vec<BootstrapSource>, BootstrapError> {+ let repo = git2::Repository::open(repo_path)+ .map_err(|_| BootstrapError::NotARepo(repo_path.to_path_buf()))?;+ let mut revwalk = repo.revwalk()?;+ revwalk.set_sorting(git2::Sort::TIME)?;+ revwalk.push_head()?;+ // `since` filtering — best-effort. libgit2's revwalk doesn't have+ // direct time-since, so we parse commit timestamps + compare to+ // a target. For simplicity we skip the since-filter when set but+ // unparsable, rather than failing loudly.+ let since_epoch = since.and_then(parse_since_to_epoch);++ let mut out = Vec::new();+ for oid in revwalk {+ let oid = oid?;+ let commit = repo.find_commit(oid)?;+ if let Some(epoch) = since_epoch+ && commit.time().seconds() < epoch+ {+ break; // revwalk is time-sorted; older commits come next+ }+ let summary = commit.summary().unwrap_or("(no summary)");+ let body = commit.body().unwrap_or("");+ let combined_len = summary.len() + body.len();+ // Skip trivial commits.+ if combined_len < 120 && !is_conventional_substantive(summary) {+ continue;+ }+ let author = commit.author().name().unwrap_or("unknown").to_string();+ let when = jiff::Timestamp::from_second(commit.time().seconds())+ .map(|t| t.to_string())+ .unwrap_or_else(|_| commit.time().seconds().to_string());+ let label = format!("git: {summary}");+ let text = format!(+ "Commit {short}\nAuthor: {author}\nDate: {when}\n\n{summary}\n\n{body}",+ short = oid.to_string().chars().take(8).collect::<String>(),+ );+ out.push(BootstrapSource {+ kind: SourceKind::GitCommit,+ label,+ text,+ });+ }+ debug!(count = out.len(), "collected git commits");+ Ok(out)+}++/// Parse a `git log --since=<x>` value into a unix epoch. Supports+/// the simplest formats we expect operators to type:+/// "30 days ago", "180 days ago", "1 year ago", or an absolute YYYY-MM-DD.+fn parse_since_to_epoch(since: &str) -> Option<i64> {+ let lower = since.to_lowercase();+ let now = Timestamp::now().as_second();+ if let Some(rest) = lower.strip_suffix(" days ago") {+ let n: i64 = rest.trim().parse().ok()?;+ return Some(now - n * 86_400);+ }+ if let Some(rest) = lower.strip_suffix(" months ago") {+ let n: i64 = rest.trim().parse().ok()?;+ return Some(now - n * 30 * 86_400);+ }+ if let Some(rest) = lower.strip_suffix(" year ago") {+ let n: i64 = rest.trim().parse().ok()?;+ return Some(now - n * 365 * 86_400);+ }+ if let Some(rest) = lower.strip_suffix(" years ago") {+ let n: i64 = rest.trim().parse().ok()?;+ return Some(now - n * 365 * 86_400);+ }+ // YYYY-MM-DD fallback.+ if let Ok(date) = jiff::civil::Date::strptime("%Y-%m-%d", &lower)+ && let Ok(zoned) = date.to_zoned(jiff::tz::TimeZone::UTC)+ {+ return Some(zoned.timestamp().as_second());+ }+ None+}++/// Conventional-commit prefixes worth keeping even when the body+/// is short — they're explicitly typed as significant by the author.+fn is_conventional_substantive(summary: &str) -> bool {+ const PREFIXES: &[&str] = &[+ "feat:",+ "feat(",+ "fix:",+ "fix(",+ "refactor:",+ "refactor(",+ "perf:",+ "perf(",+ "design:",+ "design(",+ ];+ PREFIXES.iter().any(|p| summary.starts_with(p))+}++fn collect_readme(repo_path: &Path) -> Result<Vec<BootstrapSource>, BootstrapError> {+ for candidate in ["README.md", "README", "Readme.md", "readme.md"] {+ let p = repo_path.join(candidate);+ if p.is_file() {+ let text = std::fs::read_to_string(&p)?;+ return Ok(vec![BootstrapSource {+ kind: SourceKind::Readme,+ label: format!("README ({})", candidate),+ text,+ }]);+ }+ }+ Ok(Vec::new())+}++fn collect_docs(repo_path: &Path) -> Result<Vec<BootstrapSource>, BootstrapError> {+ let docs_dir = repo_path.join("docs");+ if !docs_dir.is_dir() {+ return Ok(Vec::new());+ }+ let mut out = Vec::new();+ let mut stack = vec![docs_dir];+ while let Some(dir) = stack.pop() {+ for entry in std::fs::read_dir(&dir)? {+ let entry = entry?;+ let path = entry.path();+ let ft = entry.file_type()?;+ if ft.is_symlink() {+ continue;+ }+ if ft.is_dir() {+ stack.push(path);+ } else if path.extension().and_then(|s| s.to_str()) == Some("md")+ && let Ok(text) = std::fs::read_to_string(&path)+ {+ let label = path+ .strip_prefix(repo_path)+ .unwrap_or(&path)+ .display()+ .to_string();+ out.push(BootstrapSource {+ kind: SourceKind::DocFile,+ label: format!("doc: {label}"),+ text,+ });+ }+ }+ }+ debug!(count = out.len(), "collected docs/*.md");+ Ok(out)+}++/// Pull module-level `//!` doc-comments from the top of every .rs+/// file (skip the build/target/vendor/.git tree). Stops at the first+/// non-`//!` line so test-only files with implementation noise+/// don't dump source into the prompt.+fn collect_rust_module_headers(repo_path: &Path) -> Result<Vec<BootstrapSource>, BootstrapError> {+ let mut out = Vec::new();+ let mut stack = vec![repo_path.to_path_buf()];+ while let Some(dir) = stack.pop() {+ let name = dir.file_name().and_then(|n| n.to_str()).unwrap_or("");+ if matches!(name, "target" | "node_modules" | ".git" | "vendor") {+ continue;+ }+ let read = match std::fs::read_dir(&dir) {+ Ok(r) => r,+ Err(_) => continue,+ };+ for entry in read.flatten() {+ let path = entry.path();+ let Ok(ft) = entry.file_type() else {+ continue;+ };+ if ft.is_symlink() {+ continue;+ }+ if ft.is_dir() {+ stack.push(path);+ } else if path.extension().and_then(|s| s.to_str()) == Some("rs")+ && let Ok(content) = std::fs::read_to_string(&path)+ {+ let mut header_lines = Vec::new();+ for line in content.lines() {+ let trimmed = line.trim_start();+ if trimmed.starts_with("//!") {+ header_lines.push(trimmed.trim_start_matches("//!").trim());+ } else if header_lines.is_empty() && trimmed.is_empty() {+ continue;+ } else {+ break;+ }+ }+ if header_lines.is_empty() {+ continue;+ }+ let label = path+ .strip_prefix(repo_path)+ .unwrap_or(&path)+ .display()+ .to_string();+ let text = header_lines.join("\n");+ out.push(BootstrapSource {+ kind: SourceKind::ModuleHeader,+ label: format!("rust: {label}"),+ text,+ });+ }+ }+ }+ debug!(count = out.len(), "collected rust module headers");+ Ok(out)+}++fn collect_project_rules(repo_path: &Path) -> Result<Vec<BootstrapSource>, BootstrapError> {+ let mut out = Vec::new();+ for name in ["CLAUDE.md", "AGENTS.md", "AGENT.md", "claude.md"] {+ let p = repo_path.join(name);+ if p.is_file()+ && let Ok(text) = std::fs::read_to_string(&p)+ {+ out.push(BootstrapSource {+ kind: SourceKind::ProjectRules,+ label: format!("rules: {name}"),+ text,+ });+ }+ }+ Ok(out)+}++// --------------------------------------------------------------------+// Budgeting+// --------------------------------------------------------------------++/// Drop lower-priority sources until estimated tokens fit under the+/// budget. Returns (kept, dropped_count, estimated_total_tokens).+fn prune_to_budget(+ mut sources: Vec<BootstrapSource>,+ budget: usize,+) -> (Vec<BootstrapSource>, usize, usize) {+ // Reserve ~1k tokens for the prompt scaffolding itself.+ let usable = budget.saturating_sub(1_000);+ // Order: highest drop_priority FIRST → drop those when over budget.+ sources.sort_by_key(|s| std::cmp::Reverse(s.kind.drop_priority()));+ let total_count = sources.len();+ let mut total: usize = sources.iter().map(BootstrapSource::estimated_tokens).sum();++ // We iterate sources in drop order and remove until we fit.+ while total > usable+ && let Some(victim) = sources.first()+ {+ total = total.saturating_sub(victim.estimated_tokens());+ sources.remove(0);+ }+ let kept = sources;+ let dropped = total_count - kept.len();+ (kept, dropped, total)+}++// --------------------------------------------------------------------+// LLM prompt+// --------------------------------------------------------------------++const SYSTEM_PROMPT: &str = "You are seeding a Karpathy-style \"LLM Wiki\" for \+ a software project that has existed for a while. The user has supplied the \+ project's git log, README, docs, and module headers. Your job is to produce \+ a compact set of wiki pages — concepts, decisions, gotchas — that capture \+ what a new collaborator would benefit from knowing on day one. \+ \n\nRules:\n\+ - Prefer 5–15 substantive pages over many thin ones.\n\+ - Use `concepts/<slug>.md` for evergreen architectural notes, \+ `decisions/0001-<slug>.md` for ADR-shaped commits (incrementing IDs), \+ `gotchas/<slug>.md` for failure modes.\n\+ - Cite the source briefly (e.g. \"From commit abc1234: ...\") so future \+ readers can audit.\n\+ - Do NOT invent details. If a source is ambiguous, note that in the body.\n\+ - Cap each body at ~1500 words; ai-memory's wiki is meant to be skimmed, \+ not exhaustive.";++fn build_request(sources: &[BootstrapSource]) -> ChatRequest {+ let mut buf = String::with_capacity(8_192);+ buf.push_str("Sources collected from the project. Convert into wiki pages.\n\n");+ for src in sources {+ buf.push_str(&format!("=== {} ===\n", src.label));+ buf.push_str(&src.text);+ buf.push_str("\n\n");+ }+ ChatRequest {+ system: Some(SYSTEM_PROMPT.into()),+ messages: vec![ChatMessage {+ role: Role::User,+ content: buf,+ }],+ // Reasoning models (Kimi 2.6 et al.) burn 1-2k on reasoning;+ // leave headroom for the actual output. 6000 = generous cap+ // for a multi-page bootstrap.+ max_tokens: 6_000,+ temperature: Some(0.2),+ }+}++// --------------------------------------------------------------------+// Manifest rendering+// --------------------------------------------------------------------++fn build_frontmatter(title: &str, tags: &[String], now: Timestamp) -> serde_json::Value {+ serde_json::json!({+ "title": title,+ "tier": "semantic",+ "tags": tags,+ "bootstrapped_at": now.to_string(),+ })+}++fn render_manifest_body(+ now: Timestamp,+ sources_collected: usize,+ sources_sent: usize,+ sources_dropped: usize,+ est_tokens: usize,+ rationale: &str,+ pages: &[String],+) -> String {+ let mut buf = String::with_capacity(1024);+ buf.push_str("# Bootstrap manifest\n\n");+ buf.push_str(&format!(+ "> Generated by `ai-memory bootstrap` at {now}.\n\n",+ ));+ buf.push_str("## Sources\n\n");+ buf.push_str(&format!(+ "- Collected: **{sources_collected}**\n\+ - Sent to LLM: **{sources_sent}**\n\+ - Dropped to fit budget: **{sources_dropped}**\n\+ - Estimated input tokens: **{est_tokens}**\n\n"+ ));+ buf.push_str("## Rationale\n\n");+ buf.push_str(rationale);+ buf.push_str("\n\n## Pages produced\n\n");+ for p in pages {+ buf.push_str(&format!("- `{p}`\n"));+ }+ buf.push_str("\n---\n\n");+ buf.push_str(+ "_Re-running `ai-memory bootstrap` requires `--force` while this page \+ exists. Delete this page (and the generated ones below) to reset._\n",+ );+ buf+}++#[cfg(test)]+mod tests {+ use super::*;+ use std::fs;+ use tempfile::TempDir;++ fn make_repo(tmp: &Path) -> Result<(), Box<dyn std::error::Error>> {+ // Use system git via std::process::Command instead of git2's+ // signature/config plumbing, which would force us to set a+ // committer identity in the test.+ let run = |args: &[&str]| -> std::io::Result<()> {+ let status = std::process::Command::new("git")+ .args(args)+ .current_dir(tmp)+ .status()?;+ assert!(status.success(), "git {args:?} failed");+ Ok(())+ };+ run(&["init", "-q", "-b", "main"])?;+ run(&["config", "user.email", "[email protected]"])?;+ run(&["config", "user.name", "Test"])?;+ run(&[+ "commit",+ "--allow-empty",+ "-m",+ "feat: initial scaffolding for storage substrate with WAL + supersession chain",+ ])?;+ run(&["commit", "--allow-empty", "-m", "typo"])?;+ run(&[+ "commit",+ "--allow-empty",+ "-m",+ "design: choose Karpathy compile-not-retrieve model over RAG for capture",+ "-m",+ "We considered three alternatives — vector RAG, full-conversation-log replay, and Karpathy's wiki pattern. Chose the wiki because it keeps human-readable state we can grep, diff, and back up via git, and because compile-time consolidation moves cost off the hot read path. Vector RAG was rejected for the reasons in docs/research-karpathy-llm-wiki.md.",+ ])?;+ Ok(())+ }++ #[test]+ fn parse_since_handles_common_forms() {+ let n_days = parse_since_to_epoch("30 days ago").unwrap();+ let now = Timestamp::now().as_second();+ assert!(now - n_days > 29 * 86_400);+ assert!(now - n_days < 31 * 86_400);+ assert!(parse_since_to_epoch("1 year ago").is_some());+ assert!(parse_since_to_epoch("2026-01-01").is_some());+ assert!(parse_since_to_epoch("garbage").is_none());+ }++ #[test]+ fn git_collection_drops_trivial_commits() {+ let tmp = TempDir::new().unwrap();+ make_repo(tmp.path()).expect("git setup");+ let sources = collect_git_commits(tmp.path(), None).unwrap();+ // Three commits exist; the "typo" one should be filtered out.+ let summaries: Vec<&str> = sources.iter().map(|s| s.label.as_str()).collect();+ assert!(summaries.iter().any(|s| s.contains("initial scaffolding")));+ assert!(summaries.iter().any(|s| s.contains("compile-not-retrieve")));+ assert!(!summaries.iter().any(|s| s.contains("typo")));+ }++ #[test]+ fn readme_and_docs_collected() {+ let tmp = TempDir::new().unwrap();+ fs::write(tmp.path().join("README.md"), "# Project\nHello world.").unwrap();+ fs::create_dir_all(tmp.path().join("docs")).unwrap();+ fs::write(+ tmp.path().join("docs/architecture.md"),+ "# Architecture\nThings are like this.",+ )+ .unwrap();++ let readmes = collect_readme(tmp.path()).unwrap();+ assert_eq!(readmes.len(), 1);+ assert!(readmes[0].text.contains("Hello world"));++ let docs = collect_docs(tmp.path()).unwrap();+ assert_eq!(docs.len(), 1);+ assert!(docs[0].label.contains("architecture.md"));+ }++ #[test]+ fn rust_module_headers_extracted() {+ let tmp = TempDir::new().unwrap();+ fs::create_dir_all(tmp.path().join("crates/foo/src")).unwrap();+ fs::write(+ tmp.path().join("crates/foo/src/lib.rs"),+ "//! Crate-level doc.\n//! Spans two lines.\n\nuse std::path::Path;\n\nfn x() {}\n",+ )+ .unwrap();+ // A no-doc-comment file should be skipped.+ fs::write(tmp.path().join("crates/foo/src/main.rs"), "fn main() {}\n").unwrap();+ let sources = collect_rust_module_headers(tmp.path()).unwrap();+ assert_eq!(sources.len(), 1, "only the doc-commented file");+ assert!(sources[0].text.contains("Crate-level doc"));+ assert!(sources[0].text.contains("two lines"));+ assert!(!sources[0].text.contains("std::path"));+ }++ #[test]+ fn rust_module_header_skips_target_dir() {+ let tmp = TempDir::new().unwrap();+ fs::create_dir_all(tmp.path().join("target/debug")).unwrap();+ fs::write(+ tmp.path().join("target/debug/foo.rs"),+ "//! Build artefact; must not be ingested.\n",+ )+ .unwrap();+ let sources = collect_rust_module_headers(tmp.path()).unwrap();+ assert!(sources.is_empty(), "target/ must be excluded");+ }++ #[test]+ fn prune_to_budget_drops_low_priority_first() {+ let big_header = BootstrapSource {+ kind: SourceKind::ModuleHeader,+ label: "rust: x.rs".into(),+ text: "x".repeat(40_000),+ };+ let readme = BootstrapSource {+ kind: SourceKind::Readme,+ label: "README".into(),+ text: "important".to_string(),+ };+ let (kept, dropped, _) = prune_to_budget(vec![big_header, readme], 1_500);+ assert_eq!(dropped, 1);+ assert_eq!(kept.len(), 1);+ assert_eq!(kept[0].kind, SourceKind::Readme);+ }++ #[test]+ fn prune_keeps_everything_when_under_budget() {+ let s1 = BootstrapSource {+ kind: SourceKind::GitCommit,+ label: "g".into(),+ text: "short".into(),+ };+ let s2 = BootstrapSource {+ kind: SourceKind::Readme,+ label: "r".into(),+ text: "shorter".into(),+ };+ let (kept, dropped, _) = prune_to_budget(vec![s1, s2], 50_000);+ assert_eq!(dropped, 0);+ assert_eq!(kept.len(), 2);+ }+}diff --git a/crates/ai-memory-consolidate/src/lib.rs b/crates/ai-memory-consolidate/src/lib.rsindex 00237907..6fab0fdd 100644--- a/crates/ai-memory-consolidate/src/lib.rs+++ b/crates/ai-memory-consolidate/src/lib.rs@@ -8,11 +8,13 @@//!//! M7b extends this to multi-page atomic fan-out.+pub mod bootstrap;pub mod consolidator;pub mod lint;pub mod sweep;pub mod types;+pub use bootstrap::{Bootstrap, BootstrapConfig, BootstrapError, BootstrapOutcome};pub use consolidator::{Consolidator, ConsolidatorError, ConsolidatorResult};pub use lint::{LintError, LintFinding, LintReport, run_lint};pub use sweep::{EvictedPage, SweepError, SweepReport, run_sweep};diff --git a/docs/install.md b/docs/install.mdindex 94d0a9fb..0d1011f7 100644--- a/docs/install.md+++ b/docs/install.md@@ -280,6 +280,83 @@ volume). Outside docker, override with `AI_MEMORY_DATA_DIR=/path`.---+## Bootstrap mid-project {#bootstrap-mid-project}++When you adopt ai-memory in a project that's already been around for+a while, the wiki starts empty. `ai-memory bootstrap` ingests the+project's existing history into seed pages so the first session has+warm context.++```bash+docker run --rm \+ -v ai-memory-data:/data \+ -v "$PWD:/repo" \+ -e AI_MEMORY_AUTH_TOKEN="$TOKEN" \+ -e AI_MEMORY_LLM_PROVIDER=anthropic \+ -e ANTHROPIC_API_KEY=sk-ant-... \+ akitaonrails/ai-memory:latest \+ bootstrap --repo-path /repo+```++**What gets ingested by default:**++| Source | Priority (dropped first when over budget) |+|---|---|+| `CLAUDE.md` / `AGENTS.md` (project rules) | never dropped |+| `README.md` at the repo root | very-late |+| `docs/**/*.md` | late |+| Substantive git commits (body >120 chars OR conventional-commit prefix) | mid |+| Module-level `//!` doc-comments in `**/*.rs` | first to drop |++**Flags:**++```+--repo-path <PATH> (default: git rev-parse --show-toplevel)+--workspace <NAME> (default: "default")+--project <NAME> (default: "scratch")+--max-input-tokens N (default: 50000)+--since "30 days ago" (git log filter; supports "N days/months/years ago" + YYYY-MM-DD)+--exclude-git (skip commit history)+--exclude-readme (skip README)+--exclude-docs (skip docs/**/*.md)+--exclude-code (skip Rust module headers)+--dry-run (collect + estimate but don't call LLM or write)+--force (re-bootstrap, overwrites the prior manifest)+```++**Cost.** With Kimi 2.6 via OpenRouter ($0.73/$3.49 per M):+- 50k input tokens cap → ~$0.04 worst case input+- 1-2k generated tokens → ~$0.007 output+- Total: well under $0.20 per run.++**Idempotency.** The first run produces `wiki/bootstrap.md`+listing every page generated + a one-paragraph rationale. Re-running+without `--force` errors out. Delete the manifest (and the generated+pages) if you want a clean re-bootstrap.++**Dry-run first.** Always worth doing before the real call to see+which sources would actually be sent + how many tokens that+represents. Output is JSON to stdout.++```bash+docker run --rm -v "$PWD:/repo" ... bootstrap --repo-path /repo --dry-run+{+ "sources_collected": 117,+ "sources_sent": 22,+ "sources_dropped": 95,+ "estimated_input_tokens": 48760,+ "pages_written": [],+ "rationale": "(dry-run; LLM not invoked)",+ "dry_run": true+}+```++**Caveat: LLM-fabricated detail.** A bootstrap run can produce+plausible-but-wrong pages (the LLM doesn't know your project, it's+inferring from git history). The wiki is git-versioned precisely so+this is recoverable: review what landed, `docker exec ai-memory git+-C /data/wiki diff HEAD~1`, and revert if it's off.+## Operating without authFor local-only / single-machine deploys you can skip the bearer