Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(auto-improve): expose telemetry report
AkitaOnRailsbae35a67cdb5
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 42ee3932..67096294 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -27,6 +27,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0default, receives proposal JSON on stdin, fails closed on command/timeout/JSONerrors or insufficient score delta, and records eval failures as rejectedcandidates without running from hook paths.+- Added read-only auto-improvement telemetry reporting via+ `POST /admin/auto-improve/report` and `ai-memory auto-improve-report`, with+ JSON and human CLI output for recent run counts, proposal outcomes, terminal+ rates, and operational findings without staging pending proposals.- Hook spool drains now use `POST /hook/batch` when the server supports it,grouping compatible queued lifecycle events into bounded batches to amortizeremote request latency. Older servers fall back to the existing per-eventdiff --git a/README.md b/README.mdindex 582fe6c1..b21a9d01 100644--- a/README.md+++ b/README.md@@ -89,8 +89,8 @@ priors are at the [bottom](#influences-and-prior-art).session-aware current-project routing.- **Thin-client CLI.** `ai-memory status`, `bootstrap`, `checkpoints`,`restore-page`, `purge-project`, `rename-project`, `move-project`,- `audit-contamination`, `lint`, `curator`, `auto-improve`, `pending-writes`,- `embed`, `forget-sweep`, `backup` are+ `audit-contamination`, `lint`, `curator`, `auto-improve`,+ `auto-improve-report`, `pending-writes`, `embed`, `forget-sweep`, `backup` areall HTTP clients of the running server - never touch SQLite orwiki files directly. `status` also reports passive LLM/embeddingprovider health from the last real provider call. Server is the@@ -140,7 +140,10 @@ priors are at the [bottom](#influences-and-prior-art).automatic review, or set `[auto_improve] require_approval = true` to keep bothscheduled and manual proposals pending for human review. `ai-memoryauto-improve --session-id <uuid>` and MCP `memory_auto_improve` remain- available for manual catch-up or targeted reruns.+ available for manual catch-up or targeted reruns. `ai-memory+ auto-improve-report --workspace <w> --project <p>` returns a read-only+ telemetry report for recent auto-improvement outcomes without staging or+ creating proposals.Existing installs do not need per-project migration. The scheduler initializesa per-project first-run watermark so historical sessions are not revieweddiff --git a/crates/ai-memory-cli/src/cli.rs b/crates/ai-memory-cli/src/cli.rsindex bbfede30..d1ff397b 100644--- a/crates/ai-memory-cli/src/cli.rs+++ b/crates/ai-memory-cli/src/cli.rs@@ -85,6 +85,8 @@ pub enum Command {Lint(LintArgs),/// Run the rule-based curator report.Curator(CuratorArgs),+ /// Run a read-only auto-improvement telemetry report.+ AutoImproveReport(AutoImproveReportArgs),/// Run auto-improvement for one completed session.AutoImprove(AutoImproveArgs),/// Review, approve, or reject staged auto-improvement proposals.@@ -897,6 +899,27 @@ pub struct CuratorArgs {pub json: bool,}+/// Arguments for `auto-improve-report`.+#[derive(Debug, Args)]+pub struct AutoImproveReportArgs {+ /// Workspace name. Defaults to `default`.+ #[arg(long, default_value_t = crate::config::DEFAULT_WORKSPACE.to_string())]+ pub workspace: String,+ /// Project name. When omitted, auto-derived from the basename of+ /// the current git repo root (or CWD if no git repo).+ #[arg(long)]+ pub project: Option<String>,+ /// Lookback window in days.+ #[arg(long, default_value_t = ai_memory_consolidate::DEFAULT_AUTO_IMPROVE_TELEMETRY_SINCE_DAYS)]+ pub days: u32,+ /// Maximum rows in each top-N count table.+ #[arg(long, default_value_t = ai_memory_consolidate::DEFAULT_AUTO_IMPROVE_TELEMETRY_TOP_LIMIT)]+ pub limit: usize,+ /// Emit only machine-readable JSON.+ #[arg(long)]+ pub json: bool,+}+/// Arguments for `auto-improve`.#[derive(Debug, Args)]pub struct AutoImproveArgs {@@ -1342,6 +1365,30 @@ mod tests {assert!(!args.dry_run);}+ #[test]+ fn auto_improve_report_parses_scope_window_limit_and_json() {+ let cli = Cli::try_parse_from([+ "ai-memory",+ "auto-improve-report",+ "--project",+ "scratch",+ "--days",+ "14",+ "--limit",+ "3",+ "--json",+ ])+ .expect("auto-improve-report parses");++ let Command::AutoImproveReport(args) = cli.command else {+ panic!("expected auto-improve-report command");+ };+ assert_eq!(args.project.as_deref(), Some("scratch"));+ assert_eq!(args.days, 14);+ assert_eq!(args.limit, 3);+ assert!(args.json);+ }+#[test]fn pending_writes_subcommands_parse() {let id = "00000000-0000-0000-0000-000000000000";diff --git a/crates/ai-memory-cli/src/commands/auto_improve_report.rs b/crates/ai-memory-cli/src/commands/auto_improve_report.rsnew file mode 100644index 00000000..2be4768b--- /dev/null+++ b/crates/ai-memory-cli/src/commands/auto_improve_report.rs@@ -0,0 +1,81 @@+//! `ai-memory auto-improve-report` — read-only telemetry for auto-improvement outcomes.++use ai_memory_consolidate::AutoImproveTelemetryReport;+use anyhow::Result;+use serde::Serialize;++use crate::cli::AutoImproveReportArgs;+use crate::config::Config;+use crate::http_client::{ServerEndpoint, post_json};++#[derive(Serialize)]+struct AutoImproveReportRequest {+ workspace: String,+ project: String,+ since_days: u32,+ limit: usize,+}++/// Run the `auto-improve-report` subcommand.+///+/// # Errors+/// Returns an error if the server is unreachable or rejects the read-only report request.+pub async fn run(config: &Config, args: AutoImproveReportArgs) -> Result<()> {+ let endpoint = ServerEndpoint::from_config(config);+ let project = super::resolve_project_name(config, args.project.as_deref())?;+ let request = AutoImproveReportRequest {+ workspace: args.workspace,+ project: project.clone(),+ since_days: args.days,+ limit: args.limit,+ };++ let report: AutoImproveTelemetryReport =+ post_json(&endpoint, "/admin/auto-improve/report", &request).await?;+ if args.json {+ println!("{}", serde_json::to_string_pretty(&report)?);+ } else {+ print_human_report(&report, &project);+ println!("\n--- machine-readable ---");+ println!("{}", serde_json::to_string_pretty(&report)?);+ }+ Ok(())+}++fn print_human_report(report: &AutoImproveTelemetryReport, project: &str) {+ println!("\nAuto-improve telemetry for {project}\n");+ println!("Summary: {}", report.summary);+ println!(+ "Terminal denominator: {}",+ report.terminal_rates.denominator+ );+ println!(+ "Rates: approved {:.1}%, rejected {:.1}%, conflict {:.1}%, failed {:.1}%",+ report.terminal_rates.approved_rate * 100.0,+ report.terminal_rates.rejected_rate * 100.0,+ report.terminal_rates.conflict_rate * 100.0,+ report.terminal_rates.failed_rate * 100.0,+ );+ println!(+ "Runs: {} ({} with learning proposals)",+ report.aggregate.run_count, report.aggregate.runs_with_learning_proposals+ );++ if !report.aggregate.top_targets.is_empty() {+ println!("Top targets:");+ for row in report.aggregate.top_targets.iter().take(10) {+ println!(" - {}: {}", row.key, row.count);+ }+ }++ println!("Findings: {}", report.findings.len());+ for finding in report.findings.iter().take(10) {+ println!(+ " - {} [{}]: {}",+ finding.kind, finding.severity, finding.message+ );+ }+ if report.findings.len() > 10 {+ println!(" ... {} more", report.findings.len() - 10);+ }+}diff --git a/crates/ai-memory-cli/src/commands/mod.rs b/crates/ai-memory-cli/src/commands/mod.rsindex d1405f0a..f1288602 100644--- a/crates/ai-memory-cli/src/commands/mod.rs+++ b/crates/ai-memory-cli/src/commands/mod.rs@@ -8,6 +8,7 @@ pub mod apply_shared;pub mod audit_contamination;pub mod auth;pub mod auto_improve;+pub mod auto_improve_report;pub mod backup;pub mod bootstrap;pub mod checkpoints;diff --git a/crates/ai-memory-cli/src/main.rs b/crates/ai-memory-cli/src/main.rsindex be9a1dab..c02ff7f7 100644--- a/crates/ai-memory-cli/src/main.rs+++ b/crates/ai-memory-cli/src/main.rs@@ -79,6 +79,7 @@ async fn main() -> Result<()> {Command::ForgetSweep(args) => commands::forget_sweep::run(&config, args).await,Command::Lint(args) => commands::lint::run(&config, args).await,Command::Curator(args) => commands::curator::run(&config, args).await,+ Command::AutoImproveReport(args) => commands::auto_improve_report::run(&config, args).await,Command::AutoImprove(args) => commands::auto_improve::run(&config, args).await,Command::PendingWrites(args) => commands::pending_writes::run(&config, args).await,Command::Embed(args) => commands::embed::run(&config, args).await,diff --git a/crates/ai-memory-consolidate/src/auto_improve_telemetry.rs b/crates/ai-memory-consolidate/src/auto_improve_telemetry.rsindex f67be0fc..5cc4078e 100644--- a/crates/ai-memory-consolidate/src/auto_improve_telemetry.rs+++ b/crates/ai-memory-consolidate/src/auto_improve_telemetry.rs@@ -3,7 +3,7 @@use ai_memory_core::{ProjectId, WorkspaceId};use ai_memory_store::{AutoImproveTelemetryAggregate, AutoImproveTelemetryCount, ReaderPool};use jiff::Timestamp;-use serde::Serialize;+use serde::{Deserialize, Serialize};const US_PER_DAY: i64 = 86_400_000_000;@@ -13,7 +13,7 @@ pub const DEFAULT_AUTO_IMPROVE_TELEMETRY_SINCE_DAYS: u32 = 30;pub const DEFAULT_AUTO_IMPROVE_TELEMETRY_TOP_LIMIT: usize = 10;/// Parameters for an auto-improve telemetry report.-#[derive(Debug, Clone, Serialize)]+#[derive(Debug, Clone, Deserialize, Serialize)]pub struct AutoImproveTelemetryParams {/// Lookback window in days.pub since_days: u32,@@ -31,7 +31,7 @@ impl Default for AutoImproveTelemetryParams {}/// Terminal-rate denominator and rates for learning proposals.-#[derive(Debug, Clone, Serialize)]+#[derive(Debug, Clone, Deserialize, Serialize)]pub struct AutoImproveTerminalRates {/// Terminal learning proposals only: approved + rejected + conflict + failed.pub denominator: usize,@@ -46,7 +46,7 @@ pub struct AutoImproveTerminalRates {}/// One bounded operational signal in the telemetry report.-#[derive(Debug, Clone, Serialize)]+#[derive(Debug, Clone, Deserialize, Serialize)]pub struct AutoImproveTelemetryFinding {/// Machine-readable finding kind.pub kind: String,@@ -60,7 +60,7 @@ pub struct AutoImproveTelemetryFinding {}/// Structured report-only auto-improve telemetry.-#[derive(Debug, Clone, Serialize)]+#[derive(Debug, Clone, Deserialize, Serialize)]pub struct AutoImproveTelemetryReport {/// Workspace reviewed.pub workspace: String,diff --git a/crates/ai-memory-mcp/src/admin.rs b/crates/ai-memory-mcp/src/admin.rsindex 3acae802..2cad39a5 100644--- a/crates/ai-memory-mcp/src/admin.rs+++ b/crates/ai-memory-mcp/src/admin.rs@@ -5,6 +5,7 @@//! - `POST /admin/bootstrap` — ingest a pre-collected source bundle//! into seed wiki pages via the configured LLM provider.//! - `POST /admin/auto-improve` — review one session and apply or stage proposals.+//! - `POST /admin/auto-improve/report` — read-only auto-improve telemetry report.//! - `POST /admin/curator` — dry-run or stage a rule-based curator report.//! - `GET /admin/status` — lifetime counts + server data-dir info.//! - `GET /admin/search?q=` — FTS5 hits against the wiki index.@@ -33,10 +34,10 @@ use std::io::Seek;use std::path::PathBuf;use ai_memory_consolidate::{- AutoImproveReviewConfig, Bootstrap, BootstrapConfig, BootstrapOutcome, BootstrapSource,- CuratorParams, CuratorReport, SourceCounts, prune_sources_to_budget,- render_curator_report_markdown, run_auto_improve_review, run_curator_report, run_lint,- run_sweep,+ AutoImproveReviewConfig, AutoImproveTelemetryParams, Bootstrap, BootstrapConfig,+ BootstrapOutcome, BootstrapSource, CuratorParams, CuratorReport, SourceCounts,+ prune_sources_to_budget, render_curator_report_markdown, run_auto_improve_review,+ run_auto_improve_telemetry_report, run_curator_report, run_lint, run_sweep,};use ai_memory_core::{ActiveProject, AutoImproveProposalId, Capability, DEFAULT_PROJECT_NAME, DEFAULT_WORKSPACE_NAME,@@ -271,6 +272,23 @@ struct AutoImproveProposalOutcome {page_id: Option<String>,}+/// JSON request body for `POST /admin/auto-improve/report`.+#[derive(Deserialize)]+struct AutoImproveTelemetryReportRequest {+ /// Workspace name (must already exist).+ #[serde(default = "default_workspace")]+ workspace: String,+ /// Project name (must already exist).+ #[serde(default = "default_project")]+ project: String,+ /// Lookback window in days.+ #[serde(default = "default_auto_improve_telemetry_since_days")]+ since_days: u32,+ /// Maximum rows in each top-N count table.+ #[serde(default = "default_auto_improve_telemetry_top_limit")]+ limit: usize,+}+/// JSON request body for `POST /admin/curator`.#[derive(Deserialize)]struct CuratorRequest {@@ -397,6 +415,14 @@ fn default_auto_improve_pending_path() -> String {ai_memory_consolidate::DEFAULT_AUTO_IMPROVE_PENDING_PATH.into()}+fn default_auto_improve_telemetry_since_days() -> u32 {+ ai_memory_consolidate::DEFAULT_AUTO_IMPROVE_TELEMETRY_SINCE_DAYS+}++fn default_auto_improve_telemetry_top_limit() -> usize {+ ai_memory_consolidate::DEFAULT_AUTO_IMPROVE_TELEMETRY_TOP_LIMIT+}+fn hex_to_sha256(hex: &str) -> Result<[u8; 32], String> {if hex.len() != 64 {return Err("expected 64 hex chars".into());@@ -413,6 +439,7 @@ fn hex_to_sha256(hex: &str) -> Result<[u8; 32], String> {/// - `POST /admin/backup`/// - `POST /admin/bootstrap`/// - `POST /admin/auto-improve`+/// - `POST /admin/auto-improve/report`/// - `POST /admin/curator`/// - `GET /admin/status`/// - `GET /admin/audit-contamination`@@ -437,6 +464,10 @@ pub fn admin_router(state: AdminState) -> Router {.route("/admin/backup", post(handle_backup)).route("/admin/bootstrap", post(handle_bootstrap)).route("/admin/auto-improve", post(handle_auto_improve))+ .route(+ "/admin/auto-improve/report",+ post(handle_auto_improve_report),+ ).route("/admin/curator", post(handle_curator)).route("/admin/pending-writes", get(handle_pending_writes_list)).route(@@ -1492,6 +1523,31 @@ fn auto_improve_error_response(// curator report// ---------------------------------------------------------------------+async fn handle_auto_improve_report(+ State(state): State<Arc<AdminState>>,+ Json(req): Json<AutoImproveTelemetryReportRequest>,+) -> Result<impl IntoResponse, (StatusCode, Json<serde_json::Value>)> {+ let (ws, proj) = lookup_ws_proj_no_create(&state, &req.workspace, &req.project).await?;+ let params = AutoImproveTelemetryParams {+ since_days: req.since_days,+ top_limit: req.limit.clamp(1, 100),+ };+ let report = run_auto_improve_telemetry_report(+ &state.reader,+ ws,+ proj,+ &req.workspace,+ &req.project,+ params,+ )+ .await+ .map_err(|e| internal_err(e.to_string()))?;+ Ok((+ StatusCode::OK,+ Json(serde_json::to_value(&report).unwrap_or_else(|_| serde_json::json!({}))),+ ))+}+async fn handle_curator(State(state): State<Arc<AdminState>>,Json(req): Json<CuratorRequest>,@@ -6015,6 +6071,11 @@ mod tests {"dry_run": true}),),+ (+ "POST",+ "/admin/auto-improve/report",+ serde_json::json!({"workspace": "default", "project": "scratch"}),+ ),("GET", "/admin/status", serde_json::Value::Null),("GET", "/admin/audit-contamination", serde_json::Value::Null),("GET", "/admin/search?q=test", serde_json::Value::Null),diff --git a/crates/ai-memory-mcp/tests/admin_phase3.rs b/crates/ai-memory-mcp/tests/admin_phase3.rsindex 53677aa0..a3cdd19c 100644--- a/crates/ai-memory-mcp/tests/admin_phase3.rs+++ b/crates/ai-memory-mcp/tests/admin_phase3.rs@@ -8,11 +8,15 @@//! `tower::ServiceExt::oneshot`.use ai_memory_core::{- AgentKind, NewObservation, NewPage, NewSession, ObservationKind, PagePath, SessionId, Tier,+ ActorContext, AgentKind, NewObservation, NewPage, NewSession, ObservationKind, PagePath,+ SessionId, Tier,};use ai_memory_llm::SyntheticEmbedder;use ai_memory_mcp::{AdminState, admin_router};-use ai_memory_store::{DecayParams, Store};+use ai_memory_store::{+ AutoImproveProposalOperation, AutoImproveProposalStatus, DecayParams, NewAutoImproveProposal,+ StageAutoImproveRun, Store,+};use ai_memory_wiki::Wiki;use ai_memory_wiki::WritePageRequest;use axum::body::Body;@@ -81,6 +85,122 @@ async fn get(state: AdminState, uri: &str) -> axum::response::Response {router.oneshot(req).await.unwrap()}+fn telemetry_stage_input(+ ws: ai_memory_core::WorkspaceId,+ proj: ai_memory_core::ProjectId,+) -> StageAutoImproveRun {+ StageAutoImproveRun {+ workspace_id: ws,+ project_id: proj,+ session_id: None,+ provider: Some("test".into()),+ model: Some("test-model".into()),+ summary: Some("test telemetry seed".into()),+ warnings_json: serde_json::json!([]),+ rejected_candidates_json: serde_json::json!([]),+ config_json: serde_json::json!({}),+ proposal_actor: ActorContext::default(),+ proposals: vec![NewAutoImproveProposal {+ operation: AutoImproveProposalOperation::Create,+ target_path: PagePath::new("notes/telemetry.md").unwrap(),+ kind: "note".into(),+ title: "Telemetry seed".into(),+ confidence: 0.9,+ rationale: "seed report telemetry".into(),+ evidence_json: serde_json::json!([]),+ body_markdown: "# Telemetry".into(),+ artifact_sha256: None,+ edit_mode: None,+ patch_json: None,+ expected_base_body_sha256: None,+ }],+ }+}++#[tokio::test]+async fn auto_improve_report_returns_telemetry_without_creating_proposals() {+ let tmp = TempDir::new().unwrap();+ let (state, store) = make_state(&tmp).await;+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ let proj = store+ .writer+ .get_or_create_project(ws, "scratch", None)+ .await+ .unwrap();+ store+ .writer+ .stage_auto_improve_run(telemetry_stage_input(ws, proj))+ .await+ .unwrap();+ let before = store+ .reader+ .list_auto_improve_proposals(ws, proj, None, 50)+ .await+ .unwrap()+ .len();++ let resp = post(+ state,+ "/admin/auto-improve/report",+ json!({ "workspace": "default", "project": "scratch", "since_days": 30, "limit": 3 }),+ )+ .await;+ assert_eq!(resp.status(), StatusCode::OK);+ let body = body_json(resp).await;+ assert!(body["summary"].as_str().unwrap().contains("run(s)"));+ assert_eq!(body["aggregate"]["run_count"].as_u64().unwrap(), 1);+ assert_eq!(+ body["aggregate"]["proposals_by_status"][0]["key"]+ .as_str()+ .unwrap(),+ "pending"+ );+ assert_eq!(body["terminal_rates"]["denominator"].as_u64().unwrap(), 0);++ let after = store+ .reader+ .list_auto_improve_proposals(ws, proj, None, 50)+ .await+ .unwrap()+ .len();+ assert_eq!(after, before, "report endpoint must not stage proposals");+ let pending = store+ .reader+ .list_auto_improve_proposals(ws, proj, Some(AutoImproveProposalStatus::Pending), 50)+ .await+ .unwrap()+ .len();+ assert_eq!(pending, before);+}++#[tokio::test]+async fn auto_improve_report_missing_scope_fails_closed() {+ let tmp = TempDir::new().unwrap();+ let (state, store) = make_state(&tmp).await;+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ store+ .writer+ .get_or_create_project(ws, "scratch", None)+ .await+ .unwrap();++ let resp = post(+ state,+ "/admin/auto-improve/report",+ json!({ "workspace": "default", "project": "missing" }),+ )+ .await;+ assert_eq!(resp.status(), StatusCode::NOT_FOUND);+}+// ---------------------------------------------------------------------------// reorg// ---------------------------------------------------------------------------diff --git a/crates/ai-memory-store/src/auto_improve.rs b/crates/ai-memory-store/src/auto_improve.rsindex c4ddd064..883e4b42 100644--- a/crates/ai-memory-store/src/auto_improve.rs+++ b/crates/ai-memory-store/src/auto_improve.rs@@ -189,7 +189,7 @@ pub struct AutoImproveRejectionSummary {}/// Aggregate telemetry for auto-improvement runs in one scope.-#[derive(Debug, Clone, Serialize)]+#[derive(Debug, Clone, Deserialize, Serialize)]pub struct AutoImproveTelemetryAggregate {/// Number of auto-improve runs in the window.pub run_count: usize,@@ -216,7 +216,7 @@ pub struct AutoImproveTelemetryAggregate {}/// Generic `(key, count)` telemetry row.-#[derive(Debug, Clone, Serialize)]+#[derive(Debug, Clone, Deserialize, Serialize)]pub struct AutoImproveTelemetryCount {/// Aggregated key; field-specific meaning depends on the source vector.pub key: String,diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex c73437a6..e0593dff 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -266,7 +266,7 @@ restore reindex install-hookshook install-mcp commitcheckpoints restore-page llm-testforget-sweep lint auto-improve-curator pending-writes embed+auto-improve-report curator pending-writes embedgenerate-auth-token setup-agent bootstrapinstall-instructions reorg purge-projectrename-project move-project audit-contaminationdiff --git a/docs/auto-improvement-loop.md b/docs/auto-improvement-loop.mdindex c4eaf016..30c313f4 100644--- a/docs/auto-improvement-loop.md+++ b/docs/auto-improvement-loop.md@@ -371,6 +371,7 @@ them by default through the wiki mutation path. With `require_approval = true`,|---|---|| Background scheduler | Reviews newly completed sessions after the first-run watermark and applies or stages validated proposals according to approval policy. || `ai-memory auto-improve --session-id <id>` | Manually review one session and apply or stage validated proposals through the auto-improvement approval path. |+| `ai-memory auto-improve-report --workspace <w> --project <p> [--days N] [--limit N]` | Read-only telemetry report for recent auto-improvement runs, proposal outcomes, terminal rates, and findings. Does not stage or create pending proposals. || `memory_auto_improve` | Manually review the latest completed session or a named session and apply or stage validated proposals through the same path. || `ai-memory curator` | Rule-based, report-only maintenance review. || `ai-memory curator --stage` | Stage exactly one curator report page for pending-writes approval. |