Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(consolidate): A3 cold-cluster dedup + A4 entropy pre-filter (opt-in)
AkitaOnRails3c227ade1f6e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex eb570a6d..2f7bf4f9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Added+- Cold-cluster dedup of near-duplicate episodic pages (design-memory-aging.md+ bucket A3): the forget-sweep can now cluster near-duplicate cold episodic+ pages by embedding (cosine-distance DBSCAN with an adaptive k-distance eps,+ `minPts = 2`) and collapse each cluster to one survivor — the highest-retention+ member, its body the *extractive union* of the cluster's keep-tokens, so every+ member's durable facts survive — superseding the other members with a merge+ note that points at the survivor. It runs only over the bounded cold-episodic+ candidate set the sweep already materialises (never O(N²) over the whole+ corpus), is **opt-in and off by default** via `[decay] dedup_cold_clusters`+ (a `false` default), and fully **zero generative LLM**: it reads only+ already-stored embeddings, so with no embedder configured — or no embeddings+ for the configured `(provider, model, dim)` triple — it is a clean no-op, not+ an error. The eps is clamped to a conservative ceiling (`[decay] dedup_max_eps`,+ default cosine distance ≈ 0.15) so it errs toward NOT merging. **Non-destructive+ and reversible**: no source is ever hard-deleted — every merged-away member+ stays reachable via the supersession chain and git history and is recoverable+ with `restore-page` (invariant #16) — and the merge provenance is recorded in+ `page_evidence`. Every run reports its collapses in the `SweepReport`. Reuses+ existing tables: **no new migration**, and no new MCP tool (still 23). Ships+ opt-in/off; the R2 recall no-regression proof is the gate before any future+ default-on (#809).+- Entropy / boilerplate pre-filter before consolidation (design-memory-aging.md+ bucket A4): a pure, zero-LLM Shannon-entropy + boilerplate gate that skips+ low-information session pages (near-empty, whitespace, single-character, or+ highly-repetitive boilerplate) from the cross-session experience consolidation+ pass *before* they reach the LLM prompt, the eval gate, or `apply_batch`.+ It is **advisory and non-destructive** — a skipped page is not consolidated,+ never deleted (invariant #16) — and **opt-in / off by default** via+ `[auto_improve.scheduler.experience_entropy_filter]` (a `false` default with+ conservative, validated thresholds tuned so a terse-but-informative note with+ a file path and an error code is KEPT), so an upgrade changes no consolidation+ output until an operator opts in. Every run surfaces the skip count in the+ experience report warnings. No schema change and no new MCP tool (still 23)+ (#809).- Extractive tier-down of cold episodic pages (design-memory-aging.md bucketA2): instead of evicting a cold episodic page, the forget-sweep can now*compact* it — keeping the L0 frontmatter `abstract:`, an L1 first-paragraphdiff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rsindex 32a529bf..a9d91472 100644--- a/crates/ai-memory-cli/src/commands/serve.rs+++ b/crates/ai-memory-cli/src/commands/serve.rs@@ -9,7 +9,7 @@ use std::time::Duration;use ai_memory_consolidate::{AutoImproveReviewConfig, Consolidator, EmbedBackfillOptions, ObservationRetention,ScheduledAutoImproveSettings, run_auto_improve_scheduler_tick, run_embedding_backfill,- run_lint, run_sweep_with_compaction,+ run_lint,};use ai_memory_core::{ActiveProject, ProjectId, Sanitizer, WorkspaceId};use ai_memory_hooks::{@@ -1564,11 +1564,23 @@ async fn start_maintenance_scheduler(let lint_interval_secs = settings.lint_interval_secs;let embedding_backfill_interval_secs = settings.embedding_backfill_interval_secs;+ // A3 cold-cluster dedup targets the running server's configured embedder+ // coordinate; with no embedder it is `None`, making A3 a clean no-op even+ // when the flag is set (there are no stored vectors to cluster).+ let dedup_embedding = embedder+ .as_ref()+ .map(|e| ai_memory_consolidate::EmbeddingCoord {+ provider: e.provider().to_string(),+ model: e.model().to_string(),+ dim: e.dim(),+ });+let mut tasks = Vec::new();if maintenance_enabled && forget_sweep_interval_secs > 0 {let reader = reader.clone();let writer = writer.clone();let wiki = wiki.clone();+ let dedup_embedding = dedup_embedding.clone();tasks.push(tokio::spawn(async move {let interval = std::time::Duration::from_secs(forget_sweep_interval_secs);run_persisted_maintenance_job(@@ -1594,6 +1606,7 @@ async fn start_maintenance_scheduler(let writer = writer.clone();let wiki = wiki.clone();let decay = decay;+ let dedup_embedding = dedup_embedding.clone();async move {let started = std::time::Instant::now();let outcome = run_scheduled_sweep_tick(@@ -1604,6 +1617,7 @@ async fn start_maintenance_scheduler(decay.breadth_weight,decay.observation_retention(),decay.compact_cold_episodic,+ decay.cold_cluster_dedup(dedup_embedding),).await?;if outcome.errors > 0 {@@ -1805,6 +1819,7 @@ async fn start_maintenance_scheduler(ai_memory_consolidate::ExperienceConfig {sessions: scheduler.experience_sessions.max(1),min_new_sessions: scheduler.experience_every_sessions,+ entropy_filter: scheduler.experience_entropy_filter,..ai_memory_consolidate::ExperienceConfig::default()}}),@@ -1878,6 +1893,7 @@ struct ScheduledSweepTickOutcome {errors: usize,}+#[allow(clippy::too_many_arguments)]async fn run_scheduled_sweep_tick(reader: &ReaderPool,writer: &WriterHandle,@@ -1886,6 +1902,7 @@ async fn run_scheduled_sweep_tick(breadth_weight: f64,retention: ObservationRetention,compact_cold_episodic: bool,+ dedup: ai_memory_consolidate::ColdClusterDedup,) -> Result<ScheduledSweepTickOutcome> {let scopes = reader.list_all_scopes().await?;let mut outcome = ScheduledSweepTickOutcome {@@ -1894,7 +1911,7 @@ async fn run_scheduled_sweep_tick(};for scope in scopes {- match run_sweep_with_compaction(+ match ai_memory_consolidate::run_sweep_with_hygiene(reader,writer,Some(wiki),@@ -1904,6 +1921,7 @@ async fn run_scheduled_sweep_tick(breadth_weight,retention,compact_cold_episodic,+ dedup.clone(),false,).await@@ -3665,6 +3683,7 @@ mod tests {0.0,ObservationRetention::default(),false,+ ai_memory_consolidate::ColdClusterDedup::default(),).await.unwrap();diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rsindex 0fe483a9..2869cb1e 100644--- a/crates/ai-memory-cli/src/config.rs+++ b/crates/ai-memory-cli/src/config.rs@@ -101,6 +101,19 @@ pub struct DecaySettings {/// chain) and non-destructive. Defaults to `false`, so an upgrade changes/// nothing until an operator opts in.pub compact_cold_episodic: bool,+ /// A3 cold-cluster dedup (`[decay] dedup_cold_clusters`). When `true` AND an+ /// embedder is configured, the forget sweep clusters near-duplicate cold+ /// episodic pages by embedding (cosine DBSCAN, adaptive eps) and collapses+ /// each cluster to one survivor via supersession + a merge note.+ /// Non-destructive (merged-away members stay reachable) and zero generative+ /// LLM. Defaults to `false`, and is a clean no-op with no embedder, so an+ /// upgrade changes nothing until an operator opts in.+ pub dedup_cold_clusters: bool,+ /// DBSCAN density floor for A3. `0` ⇒ the conservative default (2).+ pub dedup_min_pts: usize,+ /// Conservative ceiling on the adaptive eps (cosine distance) for A3.+ /// `0.0` ⇒ the conservative default. Lower errs harder toward NOT merging.+ pub dedup_max_eps: f32,/// Optional per-tier half-life overrides (`[decay.half_life_days]`). All/// keys default to unset ⇒ the scalar `lambda` applies to every tier, which/// is byte-identical to the historical single-λ behaviour.@@ -121,6 +134,9 @@ impl Default for DecaySettings {observation_retention_days: 0,observation_prune_batch: ai_memory_consolidate::DEFAULT_OBSERVATION_PRUNE_BATCH,compact_cold_episodic: false,+ dedup_cold_clusters: false,+ dedup_min_pts: 0,+ dedup_max_eps: 0.0,half_life_days: DecayHalfLifeDays::default(),}}@@ -174,6 +190,24 @@ impl DecaySettings {batch: self.observation_prune_batch,}}++ /// A3 cold-cluster dedup options for the M8 sweep.+ ///+ /// `embedding` is the running server's configured embedder coordinate, or+ /// `None` when no embedder is configured — in which case A3 is a clean no-op+ /// even with the flag on (there are no stored vectors to cluster).+ #[must_use]+ pub fn cold_cluster_dedup(+ self,+ embedding: Option<ai_memory_consolidate::EmbeddingCoord>,+ ) -> ai_memory_consolidate::ColdClusterDedup {+ ai_memory_consolidate::ColdClusterDedup {+ enabled: self.dedup_cold_clusters,+ embedding,+ min_pts: self.dedup_min_pts,+ max_eps: self.dedup_max_eps,+ }+ }}/// One `[[llm_fallbacks]]` entry: an ordered LLM provider tried only after@@ -961,6 +995,11 @@ pub struct AutoImproveSchedulerSettings {pub experience_every_sessions: u64,/// How many recent session summary pages one experience pass reads.pub experience_sessions: usize,+ /// A4 entropy / boilerplate pre-filter for the experience pass+ /// (`[auto_improve.scheduler.experience_entropy_filter]`). Off by default:+ /// low-information session pages are skipped from consolidation only when an+ /// operator enables it. Advisory (skip, never delete).+ pub experience_entropy_filter: ai_memory_consolidate::EntropyFilterConfig,}impl Default for AutoImproveSchedulerSettings {@@ -972,6 +1011,7 @@ impl Default for AutoImproveSchedulerSettings {min_session_age_secs: 600,experience_every_sessions: 0,experience_sessions: 10,+ experience_entropy_filter: ai_memory_consolidate::EntropyFilterConfig::default(),}}}@@ -1240,6 +1280,16 @@ impl Config {if config.decay.observation_prune_batch == 0 {anyhow::bail!("decay.observation_prune_batch must be greater than zero");}+ // A4 entropy filter thresholds: reject an unusable threshold at startup+ // rather than silently ignoring it on the first experience pass.+ if let Err(message) = config+ .auto_improve+ .scheduler+ .experience_entropy_filter+ .validate()+ {+ anyhow::bail!("auto_improve.scheduler.experience_{message}");+ }// Fail at startup rather than shipping a prompt that is all scaffolding// and no observations: below this floor the fixed system prompt and pagediff --git a/crates/ai-memory-consolidate/src/cold_cluster.rs b/crates/ai-memory-consolidate/src/cold_cluster.rsnew file mode 100644index 00000000..27aa7af5--- /dev/null+++ b/crates/ai-memory-consolidate/src/cold_cluster.rs@@ -0,0 +1,302 @@+//! Pure, zero-LLM cold-cluster algorithm for A3 (docs/design-memory-aging.md+//! §A3): DBSCAN over embeddings with an adaptive k-distance eps.+//!+//! This module is the *algorithm* only — pure functions over vectors, no store,+//! no wiki, no provider (invariant #13, zero-LLM). The forget sweep is the+//! orchestrator: it materialises the bounded cold-episodic set, loads its+//! embeddings, calls [`adaptive_eps`] and [`dbscan`] here, and collapses each+//! returned cluster via supersession. Keeping the math here makes it unit+//! testable and keeps the O(N²) pairwise distance confined to the bounded cold+//! set the sweep already holds.+//!+//! Distance is **cosine distance** (`1 - cosine_similarity`). Stored embeddings+//! are unit-normalised, so cosine similarity is the dot product; the code+//! normalises defensively anyway so a caller passing raw vectors still gets a+//! correct distance.++/// Cosine distance between two vectors: `1 - cos(θ)`, clamped to `[0, 2]`.+///+/// Returns `1.0` (maximally distant, the neutral default) for a+/// zero-magnitude vector or a length mismatch, so a degenerate row can never+/// pull unrelated pages into a cluster.+#[must_use]+pub fn cosine_distance(a: &[f32], b: &[f32]) -> f32 {+ if a.len() != b.len() {+ return 1.0;+ }+ let mut dot = 0.0f32;+ let mut na = 0.0f32;+ let mut nb = 0.0f32;+ for (x, y) in a.iter().zip(b.iter()) {+ dot += x * y;+ na += x * x;+ nb += y * y;+ }+ if na <= 0.0 || nb <= 0.0 {+ return 1.0;+ }+ let sim = dot / (na.sqrt() * nb.sqrt());+ (1.0 - sim).clamp(0.0, 2.0)+}++/// Pick a conservative eps from the k-distance elbow (the field-standard DBSCAN+/// heuristic).+///+/// For each point, take the distance to its `k`-th nearest neighbour; sort those+/// k-distances ascending; the "elbow" — the point of maximum curvature — is the+/// eps at which density drops off. It is located by the maximum perpendicular+/// distance from the chord joining the first and last sorted k-distances (the+/// kneedle construction). The result is then clamped to `max_eps` so an operator+/// keeps a hard conservative ceiling: A3 errs toward NOT merging.+///+/// Returns `None` when there are too few points to form a `k`-distance+/// (`points.len() <= k`), which the caller treats as "nothing to cluster".+#[must_use]+pub fn adaptive_eps(points: &[Vec<f32>], k: usize, max_eps: f32) -> Option<f32> {+ let n = points.len();+ if k == 0 || n <= k {+ return None;+ }+ // k-th nearest-neighbour distance for every point.+ let mut k_distances: Vec<f32> = Vec::with_capacity(n);+ for (i, p) in points.iter().enumerate() {+ let mut dists: Vec<f32> = points+ .iter()+ .enumerate()+ .filter(|(j, _)| *j != i)+ .map(|(_, q)| cosine_distance(p, q))+ .collect();+ dists.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));+ // `dists` excludes self, so the k-th nearest neighbour is index k-1.+ k_distances.push(dists[k - 1]);+ }+ k_distances.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));++ let elbow = knee_value(&k_distances);+ Some(elbow.clamp(0.0, max_eps))+}++/// The elbow (knee) of an ascending curve, by maximum perpendicular distance+/// from the chord between its first and last points. For a flat or two-point+/// curve it returns the last value; the caller's `max_eps` clamp bounds it.+fn knee_value(sorted: &[f32]) -> f32 {+ let n = sorted.len();+ if n == 0 {+ return 0.0;+ }+ if n <= 2 {+ return sorted[n - 1];+ }+ let x0 = 0.0f32;+ let y0 = sorted[0];+ #[allow(clippy::cast_precision_loss)]+ let x1 = (n - 1) as f32;+ let y1 = sorted[n - 1];+ let dx = x1 - x0;+ let dy = y1 - y0;+ let denom = (dx * dx + dy * dy).sqrt();+ if denom <= 0.0 {+ return sorted[n - 1];+ }+ let mut best_idx = n - 1;+ let mut best_dist = -1.0f32;+ for (i, &y) in sorted.iter().enumerate() {+ #[allow(clippy::cast_precision_loss)]+ let x = i as f32;+ // Perpendicular distance from (x, y) to the chord (x0,y0)-(x1,y1).+ let perp = ((dy * (x - x0)) - (dx * (y - y0))).abs() / denom;+ if perp > best_dist {+ best_dist = perp;+ best_idx = i;+ }+ }+ sorted[best_idx]+}++/// DBSCAN over `points` with cosine distance. Returns the discovered clusters as+/// lists of indices into `points`; noise points are omitted (not returned as+/// singletons). Deterministic: points are visited in index order and each+/// cluster's members are in the order they were reached.+///+/// `min_pts` is the density floor (a core point has at least `min_pts` points,+/// itself included, within `eps`). A3 keeps `min_pts` small (2): a pair of+/// near-duplicate cold pages is exactly what it wants to collapse, and the+/// conservative `eps` is the guard against over-merging.+#[must_use]+pub fn dbscan(points: &[Vec<f32>], eps: f32, min_pts: usize) -> Vec<Vec<usize>> {+ let n = points.len();+ let mut labels: Vec<Label> = vec![Label::Unvisited; n];+ let mut clusters: Vec<Vec<usize>> = Vec::new();++ for i in 0..n {+ if labels[i] != Label::Unvisited {+ continue;+ }+ let neighbors = region_query(points, i, eps);+ if neighbors.len() < min_pts {+ labels[i] = Label::Noise;+ continue;+ }+ let cluster_id = clusters.len();+ let mut members: Vec<usize> = Vec::new();+ labels[i] = Label::Clustered(cluster_id);+ members.push(i);++ // Expand the cluster over a growing frontier (iterative, not recursive,+ // so a large dense set cannot blow the stack).+ let mut queue = std::collections::VecDeque::from(neighbors);+ while let Some(q) = queue.pop_front() {+ match labels[q] {+ Label::Noise => {+ // A border point: attach it, but do not expand from it.+ labels[q] = Label::Clustered(cluster_id);+ members.push(q);+ }+ Label::Unvisited => {+ labels[q] = Label::Clustered(cluster_id);+ members.push(q);+ let q_neighbors = region_query(points, q, eps);+ if q_neighbors.len() >= min_pts {+ for nb in q_neighbors {+ if labels[nb] == Label::Unvisited || labels[nb] == Label::Noise {+ queue.push_back(nb);+ }+ }+ }+ }+ Label::Clustered(_) => {}+ }+ }+ clusters.push(members);+ }+ clusters+}++#[derive(Clone, Copy, PartialEq, Eq)]+enum Label {+ Unvisited,+ Noise,+ Clustered(usize),+}++/// Indices of every point within `eps` of `idx`, including `idx` itself.+fn region_query(points: &[Vec<f32>], idx: usize, eps: f32) -> Vec<usize> {+ let p = &points[idx];+ (0..points.len())+ .filter(|&j| cosine_distance(p, &points[j]) <= eps)+ .collect()+}++#[cfg(test)]+mod tests {+ use super::*;++ /// Three tight points near one axis, one far point on another: DBSCAN must+ /// find exactly one cluster of the three, and drop the far point as noise.+ #[test]+ fn three_tight_points_and_one_far_point() {+ let points = vec![+ vec![1.0, 0.0, 0.0],+ vec![0.98, 0.02, 0.0],+ vec![0.99, 0.0, 0.01],+ vec![0.0, 1.0, 0.0], // far+ ];+ let clusters = dbscan(&points, 0.1, 2);+ assert_eq!(clusters.len(), 1, "one cluster only: {clusters:?}");+ let mut members = clusters[0].clone();+ members.sort_unstable();+ assert_eq!(members, vec![0, 1, 2], "the three tight points cluster");+ assert!(+ !clusters[0].contains(&3),+ "the far point is noise, not clustered"+ );+ }++ #[test]+ fn deterministic_across_runs() {+ let points = vec![+ vec![1.0, 0.0],+ vec![0.99, 0.01],+ vec![0.0, 1.0],+ vec![0.01, 0.99],+ ];+ let a = dbscan(&points, 0.1, 2);+ let b = dbscan(&points, 0.1, 2);+ assert_eq!(a, b, "same input yields identical clustering");+ assert_eq!(a.len(), 2, "two well-separated pairs: {a:?}");+ }++ #[test]+ fn adaptive_eps_picks_a_sane_threshold() {+ // Two tight pairs, far apart. The k-distance elbow should sit above the+ // within-pair distance (so pairs cluster) and below the across-pair+ // distance (so the two pairs stay separate).+ let points = vec![+ vec![1.0, 0.0],+ vec![0.999, 0.001],+ vec![0.0, 1.0],+ vec![0.001, 0.999],+ ];+ let eps = adaptive_eps(&points, 2, 0.5).expect("eps");+ assert!(eps > 0.0, "a positive threshold: {eps}");+ assert!(eps <= 0.5, "clamped to the conservative ceiling: {eps}");+ let clusters = dbscan(&points, eps, 2);+ assert_eq!(+ clusters.len(),+ 2,+ "adaptive eps keeps the two tight pairs separate: {clusters:?}"+ );+ }++ #[test]+ fn adaptive_eps_respects_the_conservative_ceiling() {+ // A spread-out set whose natural elbow is large; the ceiling clamps it.+ let points = vec![+ vec![1.0, 0.0],+ vec![0.0, 1.0],+ vec![-1.0, 0.0],+ vec![0.0, -1.0],+ ];+ let eps = adaptive_eps(&points, 2, 0.05).expect("eps");+ assert!(eps <= 0.05, "ceiling honoured: {eps}");+ }++ #[test]+ fn too_few_points_yields_no_eps() {+ assert_eq!(adaptive_eps(&[vec![1.0, 0.0]], 2, 0.5), None);+ assert_eq!(adaptive_eps(&[], 2, 0.5), None);+ }++ #[test]+ fn no_cluster_when_everything_is_far() {+ let points = vec![+ vec![1.0, 0.0, 0.0],+ vec![0.0, 1.0, 0.0],+ vec![0.0, 0.0, 1.0],+ ];+ let clusters = dbscan(&points, 0.1, 2);+ assert!(clusters.is_empty(), "no near-duplicates, no clusters");+ }++ #[test]+ fn cosine_distance_basics() {+ assert!(+ cosine_distance(&[1.0, 0.0], &[1.0, 0.0]) < 1e-6,+ "identical = 0"+ );+ assert!(+ (cosine_distance(&[1.0, 0.0], &[0.0, 1.0]) - 1.0).abs() < 1e-6,+ "orthogonal = 1"+ );+ assert_eq!(+ cosine_distance(&[0.0, 0.0], &[1.0, 0.0]),+ 1.0,+ "zero vector = neutral 1"+ );+ assert_eq!(+ cosine_distance(&[1.0], &[1.0, 0.0]),+ 1.0,+ "mismatch = neutral 1"+ );+ }+}diff --git a/crates/ai-memory-consolidate/src/entropy_filter.rs b/crates/ai-memory-consolidate/src/entropy_filter.rsnew file mode 100644index 00000000..bac92a07--- /dev/null+++ b/crates/ai-memory-consolidate/src/entropy_filter.rs@@ -0,0 +1,295 @@+//! Pure, zero-LLM entropy / boilerplate pre-filter for A4+//! (docs/design-memory-aging.md §A4).+//!+//! Before a consolidation pass reads a page or observation, this gate answers a+//! cheap question: does this text carry enough information to be worth+//! consolidating, or is it near-empty, whitespace, or a highly-repetitive+//! boilerplate blob? Low-information input is *skipped* from the pass — never+//! deleted (invariant #16, advisory only). Skipping it keeps noise out of every+//! downstream step (the LLM prompt, the dedup clustering) at no provider cost.+//!+//! The gate is a pure function of the text plus a small config (invariant #13,+//! zero-LLM). It is **off by default**: an unconfigured [`EntropyFilterConfig`]+//! keeps every item, so an upgrade changes no consolidation output until an+//! operator opts in. The thresholds are deliberately conservative — a terse but+//! informative note (a one-line fix with a file path and an error code) must be+//! KEPT; only genuinely low-signal text is skipped.++use std::collections::HashMap;++use serde::{Deserialize, Serialize};++/// Why a piece of text was skipped from consolidation. Advisory: a skip means+/// "not worth a consolidation pass right now", never "delete".+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+pub enum SkipReason {+ /// Fewer non-whitespace characters than the configured floor — near-empty+ /// or whitespace-only.+ NearEmpty,+ /// Shannon entropy per character is below the configured floor — a single+ /// repeated character or a tiny alphabet carrying almost no information.+ LowEntropy,+ /// The same handful of tokens repeat — a boilerplate blob (e.g. a status+ /// banner echoed many times).+ Repetitive,+}++impl SkipReason {+ /// Stable short string for reports/logs.+ #[must_use]+ pub const fn as_str(self) -> &'static str {+ match self {+ Self::NearEmpty => "near_empty",+ Self::LowEntropy => "low_entropy",+ Self::Repetitive => "repetitive",+ }+ }+}++/// The gate's verdict for one piece of text.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum FilterVerdict {+ /// Consolidate this text as usual.+ Keep,+ /// Skip this text from consolidation, for the given reason.+ Skip(SkipReason),+}++impl FilterVerdict {+ /// `true` when the text should be skipped from consolidation.+ #[must_use]+ pub const fn is_skip(self) -> bool {+ matches!(self, Self::Skip(_))+ }+}++/// Configuration for the entropy / boilerplate gate.+///+/// **Off by default** (`enabled = false`): the gate keeps everything, so an+/// existing install's consolidation output is byte-identical until an operator+/// turns it on. The thresholds are intentionally low so that turning it on only+/// removes clear noise.+#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]+#[serde(default)]+pub struct EntropyFilterConfig {+ /// Master switch. `false` (the default) makes [`classify`] return+ /// [`FilterVerdict::Keep`] for every input.+ pub enabled: bool,+ /// Minimum non-whitespace character count. Text shorter than this is+ /// [`SkipReason::NearEmpty`].+ pub min_chars: usize,+ /// Minimum Shannon entropy in bits per character. Text below this is+ /// [`SkipReason::LowEntropy`].+ pub min_entropy_bits_per_char: f64,+ /// A repetition ratio (`1 - distinct_tokens / total_tokens`) above this is+ /// [`SkipReason::Repetitive`]. Only applied once a body has at least+ /// [`Self::repetition_min_tokens`] tokens, so a legitimately terse note is+ /// never judged repetitive.+ pub max_repetition_ratio: f64,+ /// Token floor before the repetition check applies.+ pub repetition_min_tokens: usize,+}++impl Default for EntropyFilterConfig {+ fn default() -> Self {+ // Off, with conservative thresholds ready for when it is turned on.+ Self {+ enabled: false,+ min_chars: 16,+ min_entropy_bits_per_char: 2.0,+ max_repetition_ratio: 0.7,+ repetition_min_tokens: 6,+ }+ }+}++impl EntropyFilterConfig {+ /// Validate operator-supplied thresholds. Returns an error string naming the+ /// offending field, mirroring how the sweep validates its own coefficients.+ ///+ /// # Errors+ /// Returns `Err` when the entropy floor is negative or non-finite, or the+ /// repetition ratio is outside `0.0..=1.0`.+ pub fn validate(&self) -> Result<(), String> {+ if !self.min_entropy_bits_per_char.is_finite() || self.min_entropy_bits_per_char < 0.0 {+ return Err(+ "entropy_filter.min_entropy_bits_per_char must be a finite number \+ greater than or equal to zero"+ .to_string(),+ );+ }+ if !self.max_repetition_ratio.is_finite()+ || !(0.0..=1.0).contains(&self.max_repetition_ratio)+ {+ return Err(+ "entropy_filter.max_repetition_ratio must be between 0.0 and 1.0".to_string(),+ );+ }+ Ok(())+ }+}++/// Classify one piece of text against the gate.+///+/// Pure and deterministic. With `cfg.enabled = false` (the default) it always+/// returns [`FilterVerdict::Keep`], so the pass behaves exactly as it did before+/// the filter existed.+#[must_use]+pub fn classify(text: &str, cfg: &EntropyFilterConfig) -> FilterVerdict {+ if !cfg.enabled {+ return FilterVerdict::Keep;+ }+ // Near-empty: judge by non-whitespace characters, so a body that is mostly+ // blank lines is treated as empty rather than long.+ let non_ws: usize = text.chars().filter(|c| !c.is_whitespace()).count();+ if non_ws < cfg.min_chars {+ return FilterVerdict::Skip(SkipReason::NearEmpty);+ }+ // Low entropy: a single repeated character (`aaaa…`) or a tiny alphabet.+ if shannon_bits_per_char(text) < cfg.min_entropy_bits_per_char {+ return FilterVerdict::Skip(SkipReason::LowEntropy);+ }+ // Repetitive: the same few tokens over and over. Only meaningful once there+ // are enough tokens to distinguish repetition from a naturally short note.+ let tokens: Vec<&str> = text.split_whitespace().collect();+ if tokens.len() >= cfg.repetition_min_tokens {+ #[allow(clippy::cast_precision_loss)]+ let distinct = tokens+ .iter()+ .map(|t| t.to_ascii_lowercase())+ .collect::<std::collections::HashSet<_>>()+ .len() as f64;+ #[allow(clippy::cast_precision_loss)]+ let total = tokens.len() as f64;+ let repetition = 1.0 - distinct / total;+ if repetition > cfg.max_repetition_ratio {+ return FilterVerdict::Skip(SkipReason::Repetitive);+ }+ }+ FilterVerdict::Keep+}++/// Shannon entropy of the text in bits per character, over its character+/// distribution. `0.0` for empty text and for a single repeated character.+#[must_use]+fn shannon_bits_per_char(text: &str) -> f64 {+ let mut counts: HashMap<char, u64> = HashMap::new();+ let mut total: u64 = 0;+ for c in text.chars() {+ *counts.entry(c).or_insert(0) += 1;+ total += 1;+ }+ if total == 0 {+ return 0.0;+ }+ #[allow(clippy::cast_precision_loss)]+ let total_f = total as f64;+ let mut bits = 0.0;+ for &count in counts.values() {+ #[allow(clippy::cast_precision_loss)]+ let p = count as f64 / total_f;+ bits -= p * p.log2();+ }+ bits+}++#[cfg(test)]+mod tests {+ use super::*;++ fn on() -> EntropyFilterConfig {+ EntropyFilterConfig {+ enabled: true,+ ..EntropyFilterConfig::default()+ }+ }++ #[test]+ fn disabled_keeps_everything() {+ let cfg = EntropyFilterConfig::default();+ assert!(!cfg.enabled, "off by default");+ assert_eq!(classify("", &cfg), FilterVerdict::Keep);+ assert_eq!(+ classify("aaaaaaaaaaaaaaaaaaaaaaaa", &cfg),+ FilterVerdict::Keep+ );+ assert_eq!(+ classify("yes ".repeat(20).as_str(), &cfg),+ FilterVerdict::Keep+ );+ }++ #[test]+ fn empty_and_whitespace_are_near_empty() {+ assert_eq!(+ classify("", &on()),+ FilterVerdict::Skip(SkipReason::NearEmpty)+ );+ assert_eq!(+ classify(" \n\t \n ", &on()),+ FilterVerdict::Skip(SkipReason::NearEmpty)+ );+ assert_eq!(+ classify("ok", &on()),+ FilterVerdict::Skip(SkipReason::NearEmpty),+ "too short to carry information"+ );+ }++ #[test]+ fn single_repeated_character_is_low_entropy() {+ // Long enough to pass the near-empty floor, but zero information.+ assert_eq!(+ classify("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", &on()),+ FilterVerdict::Skip(SkipReason::LowEntropy)+ );+ }++ #[test]+ fn repeated_token_blob_is_repetitive() {+ // Distinct chars keep entropy up, but one token repeats — boilerplate.+ let verdict = classify("status ok status ok status ok status ok status ok", &on());+ assert_eq!(+ verdict,+ FilterVerdict::Skip(SkipReason::Repetitive),+ "{verdict:?}"+ );+ }++ #[test]+ fn terse_but_informative_fix_note_is_kept() {+ // The tuning target: a short fix note with a file path and an error+ // code must survive the gate.+ let note = "Fixed crates/ai-memory-store/src/ops.rs which raised E0433 on build";+ assert_eq!(classify(note, &on()), FilterVerdict::Keep, "{note}");+ }++ #[test]+ fn a_normal_prose_note_is_kept() {+ let note = "Set AI_MEMORY_AUTH_TOKEN before serving on a non-loopback bind, \+ and configure the allowed hosts to guard against DNS rebinding.";+ assert_eq!(classify(note, &on()), FilterVerdict::Keep, "{note}");+ }++ #[test]+ fn entropy_zero_for_empty_and_uniform() {+ assert_eq!(shannon_bits_per_char(""), 0.0);+ assert_eq!(shannon_bits_per_char("aaaa"), 0.0);+ assert!(+ shannon_bits_per_char("abcd") > 1.9,+ "4 equally likely chars = 2 bits"+ );+ }++ #[test]+ fn validate_rejects_bad_thresholds() {+ let mut cfg = on();+ cfg.min_entropy_bits_per_char = -1.0;+ assert!(cfg.validate().is_err());+ let mut cfg = on();+ cfg.max_repetition_ratio = 1.5;+ assert!(cfg.validate().is_err());+ assert!(on().validate().is_ok());+ }+}diff --git a/crates/ai-memory-consolidate/src/experience.rs b/crates/ai-memory-consolidate/src/experience.rsindex d901e628..2c002c2a 100644--- a/crates/ai-memory-consolidate/src/experience.rs+++ b/crates/ai-memory-consolidate/src/experience.rs@@ -36,6 +36,12 @@ pub struct ExperienceConfig {pub min_new_sessions: u64,/// Per-session-page character cap in the prompt.pub max_session_page_chars: usize,+ /// A4 entropy / boilerplate pre-filter (docs/design-memory-aging.md §A4).+ /// Off by default: a low-information session page is only skipped from the+ /// prompt when an operator turns it on, so consolidation output is otherwise+ /// unchanged. Advisory (invariant #16): a skipped page is not consolidated,+ /// never deleted.+ pub entropy_filter: crate::entropy_filter::EntropyFilterConfig,}impl Default for ExperienceConfig {@@ -44,6 +50,7 @@ impl Default for ExperienceConfig {sessions: 10,min_new_sessions: 5,max_session_page_chars: 6_000,+ entropy_filter: crate::entropy_filter::EntropyFilterConfig::default(),}}}@@ -91,6 +98,28 @@ pub async fn run_experience_review(}}+ // A4 entropy / boilerplate pre-filter: drop low-information session pages+ // BEFORE they reach the consolidation prompt (and thus the eval gate and+ // `apply_batch`). Skipped pages are simply not consolidated — never deleted+ // (invariant #16). Off by default, so this changes nothing unless enabled.+ let mut entropy_skipped = 0usize;+ if experience.entropy_filter.enabled {+ session_pages.retain(|(_, path, page)| {+ match crate::entropy_filter::classify(&page.body, &experience.entropy_filter) {+ crate::entropy_filter::FilterVerdict::Keep => true,+ crate::entropy_filter::FilterVerdict::Skip(reason) => {+ entropy_skipped += 1;+ tracing::debug!(+ %path,+ reason = reason.as_str(),+ "experience pass: skipping low-information session page from consolidation"+ );+ false+ }+ }+ });+ }+let label = format!("experience:{}-sessions", session_pages.len());if (session_pages.len() as u64) < experience.min_new_sessions {return Ok(AutoImproveReport {@@ -134,6 +163,11 @@ pub async fn run_experience_review(let rejection_context = load_rejection_context(reader, workspace_id, project_id, &cfg).await?;let mut warnings = Vec::new();+ if entropy_skipped > 0 {+ warnings.push(format!(+ "entropy filter skipped {entropy_skipped} low-information session page(s) from consolidation"+ ));+ }let mut prompt = String::new();prompt.push_str("You are reviewing MULTIPLE session summaries of one project side \diff --git a/crates/ai-memory-consolidate/src/lib.rs b/crates/ai-memory-consolidate/src/lib.rsindex 872612ed..ac835d23 100644--- a/crates/ai-memory-consolidate/src/lib.rs+++ b/crates/ai-memory-consolidate/src/lib.rs@@ -12,10 +12,12 @@ pub mod auto_improve;pub mod auto_improve_schedule;pub mod auto_improve_telemetry;pub mod bootstrap;+pub mod cold_cluster;pub mod compaction;pub mod consolidator;pub mod curator;pub mod embed;+pub mod entropy_filter;pub mod experience;pub mod keep_tokens;pub mod lint;@@ -54,6 +56,7 @@ pub use bootstrap::{derive_project_name, discover_main_repo_root, discover_repo_root, effective_chunk_budget,plan_bootstrap_chunks, prune_sources_to_budget,};+pub use cold_cluster::{adaptive_eps, cosine_distance, dbscan};pub use compaction::build_compacted_markdown;pub use consolidator::{BATCH_SYSTEM_PROMPT, Consolidator, ConsolidatorError, ConsolidatorResult,@@ -67,12 +70,14 @@ pub use curator::{pub use embed::{EmbedBackfillCounts, EmbedBackfillError, EmbedBackfillOptions, run_embedding_backfill,};+pub use entropy_filter::{EntropyFilterConfig, FilterVerdict, SkipReason, classify};pub use experience::{EXPERIENCE_SYSTEM_PROMPT, ExperienceConfig, run_experience_review};pub use keep_tokens::mine_keep_tokens;pub use lint::{LintError, LintFinding, LintOptions, LintReport, run_lint, stale_days_for};pub use sweep::{- CompactedPage, DEFAULT_OBSERVATION_PRUNE_BATCH, EvictedPage, ObservationRetention, SweepError,- SweepReport, run_sweep, run_sweep_with_breadth, run_sweep_with_compaction,+ ColdClusterDedup, CompactedPage, DEFAULT_OBSERVATION_PRUNE_BATCH, EmbeddingCoord, EvictedPage,+ MergedCluster, ObservationRetention, SweepError, SweepReport, run_sweep,+ run_sweep_with_breadth, run_sweep_with_compaction, run_sweep_with_hygiene,run_sweep_with_options,};pub use types::{diff --git a/crates/ai-memory-consolidate/src/sweep.rs b/crates/ai-memory-consolidate/src/sweep.rsindex e4313899..dd3de455 100644--- a/crates/ai-memory-consolidate/src/sweep.rs+++ b/crates/ai-memory-consolidate/src/sweep.rs@@ -29,7 +29,9 @@use std::collections::{HashMap, HashSet};-use ai_memory_core::{ActorContext, PageId, PagePath, ProjectId, Tier, WorkspaceId};+use ai_memory_core::{+ ActorContext, PageEvidence, PageEvidenceKind, PageId, PagePath, ProjectId, Tier, WorkspaceId,+};use ai_memory_store::{DecayCandidate, DecayParams, ReaderPool, WriterHandle, retention_score_with_breadth,};@@ -38,6 +40,7 @@ use jiff::Timestamp;use serde::Serialize;use thiserror::Error;+use crate::cold_cluster::{adaptive_eps, dbscan};use crate::compaction::build_compacted_markdown;/// One evicted page surfaced in the [`SweepReport`].@@ -82,6 +85,93 @@ pub struct CompactedPage {pub new_id: Option<PageId>,}+/// One cold-cluster collapse surfaced in the [`SweepReport`] (A3,+/// docs/design-memory-aging.md). A cluster of near-duplicate cold episodic pages+/// is collapsed to one survivor; the other members are superseded with a merge+/// note pointing at the survivor and stay reachable (invariant #16, never a hard+/// delete).+#[derive(Debug, Clone, Serialize)]+pub struct MergedCluster {+ /// Path of the survivor (the highest-retention cluster member).+ pub survivor: String,+ /// Pre-merge identifier of the survivor version.+ pub survivor_id: PageId,+ /// Paths of the members merged away into the survivor.+ pub merged: Vec<String>,+ /// The eps (cosine distance) the adaptive k-distance heuristic chose for+ /// this run — surfaced so a conservative-vs-eager run is observable.+ pub eps: f64,+ /// `true` when the collapse landed through the wiki layer. Always `false` on+ /// `dry_run`; a failed batch is retried on the next sweep.+ pub applied: bool,+ /// Identifier of the new (merged) survivor version once the collapse lands.+ /// The pre-merge survivor and every merged-away member stay reachable via+ /// the supersession chain and git history, so the collapse is reversible.+ #[serde(skip_serializing_if = "Option::is_none")]+ pub new_survivor_id: Option<PageId>,+}++/// The `(provider, model, dim)` triple identifying which stored embeddings the+/// A3 cold-cluster dedup should load. Must match the triple the pages were+/// embedded under; a mismatch (or no embeddings) makes A3 a clean no-op.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct EmbeddingCoord {+ /// Embedding provider name (e.g. `synthetic`, `openai`).+ pub provider: String,+ /// Embedding model id.+ pub model: String,+ /// Vector dimensionality.+ pub dim: u32,+}++/// Opt-in configuration for the A3 cold-cluster dedup pass.+///+/// Off by default (`enabled = false` and no [`EmbeddingCoord`]): the sweep never+/// clusters or merges, so behaviour is byte-identical to before A3. The pass is+/// also a clean no-op — never an error — when no embeddings exist for the given+/// triple or no embedder is configured (invariant #13, zero generative LLM: A3+/// only reads already-stored vectors).+#[derive(Debug, Clone, Default)]+pub struct ColdClusterDedup {+ /// Master switch. `false` (the default) disables clustering entirely.+ pub enabled: bool,+ /// Which stored embeddings to cluster over. `None` ⇒ no-op (nothing to+ /// load), so a store with no embedder configured never dedups.+ pub embedding: Option<EmbeddingCoord>,+ /// DBSCAN density floor. `0` falls back to the conservative default of 2 (a+ /// pair of near-duplicates is the smallest thing worth collapsing).+ pub min_pts: usize,+ /// Conservative ceiling on the adaptive eps (cosine distance). The adaptive+ /// k-distance elbow is clamped to this, so A3 errs toward NOT merging.+ /// `0.0` falls back to [`DEFAULT_DEDUP_MAX_EPS`].+ pub max_eps: f32,+}++/// Default DBSCAN density floor for A3 when unset.+pub const DEFAULT_DEDUP_MIN_PTS: usize = 2;+/// Default conservative eps ceiling (cosine distance ≈ 0.15 ⇒ cosine similarity+/// ≥ 0.85) for A3 when unset. Deliberately tight: over-eager clustering is the+/// documented A3 failure mode.+pub const DEFAULT_DEDUP_MAX_EPS: f32 = 0.15;++impl ColdClusterDedup {+ fn effective_min_pts(&self) -> usize {+ if self.min_pts == 0 {+ DEFAULT_DEDUP_MIN_PTS+ } else {+ self.min_pts+ }+ }++ fn effective_max_eps(&self) -> f32 {+ if self.max_eps > 0.0 && self.max_eps.is_finite() {+ self.max_eps+ } else {+ DEFAULT_DEDUP_MAX_EPS+ }+ }+}+/// One TTL-expired page surfaced in the [`SweepReport`].#[derive(Debug, Clone, Serialize)]pub struct ExpiredPage {@@ -113,6 +203,11 @@ pub struct SweepReport {/// sweep behaves exactly as before — while the flag is off, which is the/// default. Reported in both modes so every run is observable.pub compacted: Vec<CompactedPage>,+ /// Near-duplicate cold-episodic clusters collapsed to one survivor (A3),+ /// when `dedup.enabled` is set. Empty — and the sweep behaves exactly as+ /// before — while the flag is off, which is the default. Reported in both+ /// modes so every run is observable.+ pub merged: Vec<MergedCluster>,/// Pages past their frontmatter `expires_at:` TTL (hard-deleted/// through the wiki layer unless `dry_run`).pub expired: Vec<ExpiredPage>,@@ -317,6 +412,54 @@ pub async fn run_sweep_with_compaction(retention: ObservationRetention,compact_cold_episodic: bool,dry_run: bool,+) -> Result<SweepReport, SweepError> {+ run_sweep_with_hygiene(+ reader,+ writer,+ wiki,+ workspace_id,+ project_id,+ params,+ breadth_weight,+ retention,+ compact_cold_episodic,+ ColdClusterDedup::default(),+ dry_run,+ )+ .await+}++/// Run a sweep with the breadth coefficient, the observation prune, the opt-in+/// A2 extractive tier-down, and the opt-in A3 cold-cluster dedup.+///+/// A3 (`dedup.enabled`) clusters the *bounded cold-episodic set this sweep+/// already materialised* by embedding (cosine DBSCAN, adaptive k-distance eps)+/// and collapses each near-duplicate cluster to one survivor — the+/// highest-retention member, its body the extractive union of the cluster's+/// keep-tokens — superseding the other members with a merge note that points at+/// the survivor. Nothing is ever hard-deleted: every merged-away member stays+/// reachable through the supersession chain and git (invariant #16), and the+/// merge provenance is recorded in `page_evidence`. With `dedup` disabled (the+/// default) no clustering happens and this is byte-identical to+/// [`run_sweep_with_compaction`]. With no embeddings for the configured triple —+/// or no embedder configured at all — A3 is a clean no-op (invariant #13: it+/// reads only already-stored vectors, never a provider).+///+/// # Errors+/// Same as [`run_sweep_with_options`].+#[allow(clippy::too_many_arguments)]+pub async fn run_sweep_with_hygiene(+ reader: &ReaderPool,+ writer: &WriterHandle,+ wiki: Option<&Wiki>,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ params: &DecayParams,+ breadth_weight: f64,+ retention: ObservationRetention,+ compact_cold_episodic: bool,+ dedup: ColdClusterDedup,+ dry_run: bool,) -> Result<SweepReport, SweepError> {if !breadth_weight.is_finite() || breadth_weight < 0.0 {return Err(SweepError::InvalidBreadthWeight);@@ -333,6 +476,11 @@ pub async fn run_sweep_with_compaction(let mut compacted: Vec<CompactedPage> = Vec::new();let mut expired: Vec<ExpiredPage> = Vec::new();+ // First pass: TTL and the cold set. Collecting the cold set once lets the A3+ // dedup pass cluster the *same* bounded set the evict/compact pass would+ // otherwise process (invariant #2 — one materialisation, not two chances to+ // drift), and lets the remainder loop skip whatever dedup already claimed.+ let mut cold: Vec<ColdEntry> = Vec::new();for c in &candidates {if let Some(expires_us) = c.expires_at_us&& expires_us <= now_us@@ -363,34 +511,59 @@ pub async fn run_sweep_with_compaction(breadth_weight,);if score < params.cold_threshold {- if compact_cold_episodic {- // A2 tier-down replaces eviction for cold episodic pages. An- // already-compacted page (its V65 marker set) is terminal — the- // durable residue is what we chose to keep, so re-evicting it- // would destroy exactly what tier-down preserved, and- // re-compacting it is a no-op loop. Skip it entirely.- if c.compacted_at_us.is_some() {- continue;- }- compacted.push(CompactedPage {- id: c.id,- path: c.path.as_str().to_string(),- retention: score,- age_days,- access_count: c.access_count,- compacted: false,- new_id: None,- });- } else {- evicted.push(EvictedPage {- id: c.id,- path: c.path.as_str().to_string(),- retention: score,- age_days,- access_count: c.access_count,- deleted: false,- });+ cold.push(ColdEntry {+ id: c.id,+ path: c.path.clone(),+ retention: score,+ age_days,+ access_count: c.access_count,+ compacted_at_us: c.compacted_at_us,+ });+ }+ }++ // A3 cold-cluster dedup plan (read-only). Populates `merged` and the set of+ // page ids the collapse claims, so the evict/compact remainder skips them.+ // A clean no-op (empty plan) when the flag is off, no embeddings exist for+ // the configured triple, or no wiki handle is available.+ let plan =+ plan_cold_cluster_dedup(reader, wiki, workspace_id, project_id, &dedup, &cold).await?;+ let mut merged = plan.report;+ let mut dedup_requests = plan.requests;+ let dedup_survivor_req_index = plan.survivor_req_index;+ let deduped = plan.claimed;++ for entry in &cold {+ if deduped.contains(&entry.id) {+ continue;+ }+ if compact_cold_episodic {+ // A2 tier-down replaces eviction for cold episodic pages. An+ // already-compacted page (its V65 marker set) is terminal — the+ // durable residue is what we chose to keep, so re-evicting it+ // would destroy exactly what tier-down preserved, and+ // re-compacting it is a no-op loop. Skip it entirely.+ if entry.compacted_at_us.is_some() {+ continue;}+ compacted.push(CompactedPage {+ id: entry.id,+ path: entry.path.as_str().to_string(),+ retention: entry.retention,+ age_days: entry.age_days,+ access_count: entry.access_count,+ compacted: false,+ new_id: None,+ });+ } else {+ evicted.push(EvictedPage {+ id: entry.id,+ path: entry.path.as_str().to_string(),+ retention: entry.retention,+ age_days: entry.age_days,+ access_count: entry.access_count,+ deleted: false,+ });}}@@ -408,6 +581,36 @@ pub async fn run_sweep_with_compaction(.await?;}if !dry_run {+ // A3 cold-cluster collapse (invariant #2: one batched write for every+ // survivor rewrite and merge-note supersession). The survivor keeps the+ // union of the cluster's keep-tokens and cites its merged-away members+ // in `page_evidence`; every member stays reachable via supersession ++ // git (invariant #16). Freshly rewritten, none of these pages is cold+ // next sweep, so the collapse does not immediately re-trigger.+ if !dedup_requests.is_empty() {+ match wiki {+ Some(wiki) => {+ let requests = std::mem::take(&mut dedup_requests);+ match wiki.apply_batch(requests).await {+ Ok(new_ids) => {+ for (cluster, &req_idx) in+ merged.iter_mut().zip(dedup_survivor_req_index.iter())+ {+ cluster.applied = true;+ cluster.new_survivor_id = new_ids.get(req_idx).copied();+ }+ }+ Err(error) => tracing::warn!(+ %error,+ "forget sweep: cold-cluster dedup batch failed; retrying on next sweep"+ ),+ }+ }+ None => tracing::warn!(+ "forget sweep: wiki unavailable; refusing store-only cold-cluster dedup"+ ),+ }+ }for page in &mut expired {let path = match ai_memory_core::PagePath::new(page.path.clone()) {Ok(p) => p,@@ -598,6 +801,7 @@ pub async fn run_sweep_with_compaction(candidates_evaluated: candidates.len(),evicted,compacted,+ merged,expired,hard_deleted,observations_prunable,@@ -607,6 +811,280 @@ pub async fn run_sweep_with_compaction(})}+/// A cold page (scored below `cold_threshold`) captured once so the A3 dedup+/// pass and the evict/compact remainder read the same materialised set.+struct ColdEntry {+ id: PageId,+ path: PagePath,+ retention: f64,+ age_days: f64,+ access_count: u32,+ compacted_at_us: Option<i64>,+}++/// The read-only output of the A3 dedup planner: what to report, which page ids+/// the collapse claims, and the batched wiki writes to apply.+struct DedupPlan {+ report: Vec<MergedCluster>,+ claimed: HashSet<PageId>,+ requests: Vec<WritePageRequest>,+ /// For each entry in `report`, the index into `requests` of that cluster's+ /// survivor write, so the applied `new_survivor_id` can be mapped back.+ survivor_req_index: Vec<usize>,+}++impl DedupPlan {+ fn empty() -> Self {+ Self {+ report: Vec::new(),+ claimed: HashSet::new(),+ requests: Vec::new(),+ survivor_req_index: Vec::new(),+ }+ }+}++/// Footer stamped on a merge-note stub so a reader sees the page was collapsed+/// and where its content now lives.+const MERGE_STUB_FOOTER: &str = "_Merged into the survivor below by A3 cold-cluster dedup. This page's full \+ pre-merge body is retained in git history and the supersession chain, and \+ can be recovered with `restore-page`._";++/// Build the A3 cold-cluster dedup plan over the already-materialised cold set.+///+/// Pure read path: it loads stored embeddings, clusters them, and prepares the+/// wiki writes, but applies nothing. A clean no-op (empty plan) when disabled,+/// when no `EmbeddingCoord`/wiki is available, or when no embeddings exist for+/// the cold set (invariant #13 — never a provider call).+async fn plan_cold_cluster_dedup(+ reader: &ReaderPool,+ wiki: Option<&Wiki>,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ dedup: &ColdClusterDedup,+ cold: &[ColdEntry],+) -> Result<DedupPlan, SweepError> {+ if !dedup.enabled {+ return Ok(DedupPlan::empty());+ }+ let Some(coord) = dedup.embedding.clone() else {+ return Ok(DedupPlan::empty());+ };+ // Dedup mutates through the wiki (survivor rewrite + supersessions); without+ // a handle there is nothing to plan, exactly like the compaction pass.+ let Some(wiki) = wiki else {+ return Ok(DedupPlan::empty());+ };+ // Only pages that are cold, not already a compacted/merged residue (their+ // marker set), are eligible — a terminal residue is never re-merged.+ let mut eligible: HashMap<PageId, &ColdEntry> = HashMap::new();+ for entry in cold {+ if entry.compacted_at_us.is_none() {+ eligible.insert(entry.id, entry);+ }+ }+ if eligible.len() < dedup.effective_min_pts() {+ return Ok(DedupPlan::empty());+ }++ let embeddings = reader+ .load_embeddings(+ workspace_id,+ project_id,+ coord.provider,+ coord.model,+ coord.dim,+ )+ .await?;+ // Intersect the stored embeddings with the eligible cold set.+ let mut ids: Vec<PageId> = Vec::new();+ let mut vectors: Vec<Vec<f32>> = Vec::new();+ for e in &embeddings {+ if eligible.contains_key(&e.id) {+ ids.push(e.id);+ vectors.push(e.vector.clone());+ }+ }+ if vectors.len() < dedup.effective_min_pts() {+ return Ok(DedupPlan::empty());+ }++ let min_pts = dedup.effective_min_pts();+ let Some(eps) = adaptive_eps(&vectors, min_pts, dedup.effective_max_eps()) else {+ return Ok(DedupPlan::empty());+ };+ let clusters = dbscan(&vectors, eps, min_pts);+ if clusters.is_empty() {+ return Ok(DedupPlan::empty());+ }++ let mut plan = DedupPlan::empty();+ for cluster in clusters {+ if cluster.len() < 2 {+ continue;+ }+ // Survivor = highest-retention member (ties broken by first index, for+ // determinism). Read every member's body to mine the union of keep-tokens.+ let mut member_entries: Vec<&ColdEntry> = Vec::new();+ for &idx in &cluster {+ match eligible.get(&ids[idx]) {+ Some(entry) => member_entries.push(entry),+ None => continue,+ }+ }+ if member_entries.len() < 2 {+ continue;+ }+ let survivor_pos = member_entries+ .iter()+ .enumerate()+ .max_by(|(_, a), (_, b)| {+ a.retention+ .partial_cmp(&b.retention)+ .unwrap_or(std::cmp::Ordering::Equal)+ })+ .map(|(i, _)| i)+ .unwrap_or(0);+ let survivor = member_entries[survivor_pos];++ // Read the survivor's frontmatter and concatenate every member's body so+ // the extractive miner keeps the UNION of the cluster's keep-tokens.+ let survivor_md = match wiki.read_page(workspace_id, project_id, &survivor.path) {+ Ok(md) => md,+ Err(error) => {+ tracing::warn!(path = %survivor.path.as_str(), %error, "forget sweep: could not read survivor for dedup; skipping cluster");+ continue;+ }+ };+ let mut union_body = survivor_md.body.clone();+ let mut loser_paths: Vec<String> = Vec::new();+ let mut loser_requests: Vec<WritePageRequest> = Vec::new();+ let mut evidence: Vec<PageEvidence> = Vec::new();+ let mut read_failed = false;+ for (pos, member) in member_entries.iter().enumerate() {+ if pos == survivor_pos {+ continue;+ }+ let md = match wiki.read_page(workspace_id, project_id, &member.path) {+ Ok(md) => md,+ Err(error) => {+ tracing::warn!(path = %member.path.as_str(), %error, "forget sweep: could not read cluster member for dedup; skipping cluster");+ read_failed = true;+ break;+ }+ };+ union_body.push_str("\n\n");+ union_body.push_str(&md.body);+ loser_paths.push(member.path.as_str().to_string());+ // Merge provenance: the closed `page_evidence` vocabulary has no+ // dedicated A3 kind (adding one needs a CHECK-constraint migration,+ // which A3 deliberately avoids). `reconsolidation` is the accurate+ // existing kind — an A3 collapse rewrites the survivor from several+ // sources — and the `a3_merge:` source-id prefix keeps the merge+ // distinguishable and unique.+ evidence.push(PageEvidence {+ kind: PageEvidenceKind::Reconsolidation,+ source_id: format!("a3_merge:{}", member.id),+ });+ // Supersede the loser with a merge-note stub pointing at the+ // survivor. Marked `compacted: true` so it is terminal for the decay+ // pass and never re-clustered; its full body stays in the chain.+ let stub_body = format!(+ "This page was merged into [[{}]] by A3 cold-cluster dedup.\n\n{}\n",+ survivor.path.as_str(),+ MERGE_STUB_FOOTER+ );+ let mut stub_fm = frontmatter_object(&md.frontmatter);+ stub_fm.insert("compacted".to_string(), serde_json::Value::Bool(true));+ stub_fm.insert(+ "merged_into".to_string(),+ serde_json::Value::String(survivor.path.as_str().to_string()),+ );+ loser_requests.push(WritePageRequest {+ workspace_id,+ project_id,+ path: member.path.clone(),+ frontmatter: serde_json::Value::Object(stub_fm),+ body: stub_body,+ tier: Tier::Episodic,+ pinned: false,+ title: None,+ admission_ctx: None,+ author_id: None,+ actor: ActorContext::anonymous(),+ evidence: Vec::new(),+ });+ }+ if read_failed || loser_paths.is_empty() {+ continue;+ }++ // Survivor body: extractive union of keep-tokens + a merge note. Reuses+ // A2's `build_compacted_markdown`, so the survivor keeps every member's+ // durable facts; the prose is dropped (zero-LLM, invariant #13).+ let (mut survivor_fm, mut survivor_body) =+ build_compacted_markdown(&survivor_md.frontmatter, &union_body);+ survivor_body.push_str(&format!(+ "\n\n_Merged {} near-duplicate cold page(s): {} (A3 cold-cluster dedup)._\n",+ loser_paths.len(),+ loser_paths.join(", ")+ ));+ if let serde_json::Value::Object(map) = &mut survivor_fm {+ map.insert(+ "merged_from".to_string(),+ serde_json::Value::Array(+ loser_paths+ .iter()+ .map(|p| serde_json::Value::String(p.clone()))+ .collect(),+ ),+ );+ }+ let survivor_request = WritePageRequest {+ workspace_id,+ project_id,+ path: survivor.path.clone(),+ frontmatter: survivor_fm,+ body: survivor_body,+ tier: Tier::Episodic,+ pinned: false,+ title: None,+ admission_ctx: None,+ author_id: None,+ actor: ActorContext::anonymous(),+ evidence,+ };++ // Record. Survivor write goes first in this cluster's slice, so its+ // index is the current request length.+ let survivor_req_index = plan.requests.len();+ plan.requests.push(survivor_request);+ plan.requests.extend(loser_requests);+ plan.claimed.insert(survivor.id);+ for member in &member_entries {+ plan.claimed.insert(member.id);+ }+ plan.report.push(MergedCluster {+ survivor: survivor.path.as_str().to_string(),+ survivor_id: survivor.id,+ merged: loser_paths,+ eps: f64::from(eps),+ applied: false,+ new_survivor_id: None,+ });+ plan.survivor_req_index.push(survivor_req_index);+ }+ Ok(plan)+}++/// Coerce a frontmatter value into an object map, discarding a non-object shape.+fn frontmatter_object(fm: &serde_json::Value) -> serde_json::Map<String, serde_json::Value> {+ match fm {+ serde_json::Value::Object(m) => m.clone(),+ _ => serde_json::Map::new(),+ }+}+/// Distinct-actor counts keyed by page, for feeding/// [`retention_score_with_breadth`].///diff --git a/crates/ai-memory-consolidate/tests/suite/cold_cluster_sweep.rs b/crates/ai-memory-consolidate/tests/suite/cold_cluster_sweep.rsnew file mode 100644index 00000000..f0fd9029--- /dev/null+++ b/crates/ai-memory-consolidate/tests/suite/cold_cluster_sweep.rs@@ -0,0 +1,326 @@+//! A3 cold-cluster dedup, end to end through a real Store + Wiki + embeddings.+//!+//! Invariants guarded (docs/design-memory-aging.md §A3):+//!+//! 1. **Flag OFF = today's behaviour.** No clustering, no merges.+//! 2. **Flag ON = collapse.** Near-duplicate cold episodic pages collapse to one+//! survivor; the other members are superseded with a merge note and stay+//! reachable (invariant #16, never a hard delete).+//! 3. **Survivor keeps the union of keep-tokens** — a fact that lived only in a+//! merged-away duplicate survives in the survivor (recall preserved).+//! 4. **Merge provenance** is recorded in `page_evidence`.+//! 5. **No embeddings ⇒ clean no-op** (never an error, never a provider call).++use std::sync::Arc;++use ai_memory_consolidate::{+ ColdClusterDedup, EmbeddingCoord, ObservationRetention, run_sweep_with_hygiene,+};+use ai_memory_core::{ActorContext, PageId, PagePath, ProjectId, Tier, WorkspaceId};+use ai_memory_llm::{Embedder, SyntheticEmbedder};+use ai_memory_store::{DecayParams, Store};+use ai_memory_wiki::{Wiki, WritePageRequest};+use rusqlite::params;+use tempfile::TempDir;++const DAY_US: i64 = 86_400_000_000;+const COLD_DAYS: i64 = 400;+const DIM: u32 = 64;++struct Fixture {+ tmp: TempDir,+ store: Store,+ wiki: Wiki,+ ws: WorkspaceId,+ proj: ProjectId,+}++/// Build a fixture. `with_embedder` controls whether pages get embedding rows —+/// the no-embeddings no-op test wants a wiki without one.+async fn seed_fixture(with_embedder: bool) -> Fixture {+ let tmp = TempDir::new().unwrap();+ let store = Store::open(tmp.path()).unwrap();+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ let proj = store+ .writer+ .get_or_create_project(ws, "scratch", None)+ .await+ .unwrap();+ let mut wiki = Wiki::new(tmp.path(), store.writer.clone())+ .unwrap()+ .with_store_reader(store.reader.clone());+ if with_embedder {+ let embedder: Arc<dyn Embedder> = Arc::new(SyntheticEmbedder::new(DIM));+ wiki = wiki.with_embedder(embedder);+ }+ Fixture {+ tmp,+ store,+ wiki,+ ws,+ proj,+ }+}++async fn write(fx: &Fixture, path: &str, body: &str) -> PageId {+ fx.wiki+ .write_page(WritePageRequest {+ workspace_id: fx.ws,+ project_id: fx.proj,+ path: PagePath::new(path).unwrap(),+ frontmatter: serde_json::json!({"title": path}),+ body: body.to_string(),+ tier: Tier::Episodic,+ pinned: false,+ title: Some(path.to_string()),+ admission_ctx: None,+ author_id: None,+ actor: ActorContext::anonymous(),+ evidence: Vec::new(),+ })+ .await+ .unwrap()+}++fn backdate_latest(fx: &Fixture, path: &str, days: i64) {+ let conn = rusqlite::Connection::open(fx.tmp.path().join("db/memory.sqlite")).unwrap();+ conn.pragma_update(None, "busy_timeout", 5_000).unwrap();+ let updated_at = jiff::Timestamp::now().as_microsecond() - days * DAY_US;+ conn.execute(+ "UPDATE pages SET created_at = ?1, updated_at = ?1 \+ WHERE workspace_id = ?2 AND project_id = ?3 AND path = ?4 AND is_latest = 1",+ params![updated_at, fx.ws.as_bytes(), fx.proj.as_bytes(), path],+ )+ .unwrap();+}++fn latest_body(fx: &Fixture, path: &str) -> String {+ let conn = rusqlite::Connection::open(fx.tmp.path().join("db/memory.sqlite")).unwrap();+ conn.query_row(+ "SELECT body FROM pages \+ WHERE workspace_id = ?1 AND project_id = ?2 AND path = ?3 AND is_latest = 1",+ params![fx.ws.as_bytes(), fx.proj.as_bytes(), path],+ |row| row.get(0),+ )+ .unwrap()+}++fn superseded_bodies(fx: &Fixture, path: &str) -> Vec<String> {+ let conn = rusqlite::Connection::open(fx.tmp.path().join("db/memory.sqlite")).unwrap();+ let mut stmt = conn+ .prepare(+ "SELECT body FROM pages \+ WHERE workspace_id = ?1 AND project_id = ?2 AND path = ?3 AND is_latest = 0 \+ ORDER BY created_at",+ )+ .unwrap();+ let rows = stmt+ .query_map(params![fx.ws.as_bytes(), fx.proj.as_bytes(), path], |row| {+ row.get::<_, String>(0)+ })+ .unwrap();+ rows.map(Result::unwrap).collect()+}++fn evidence_for(fx: &Fixture, page_id: PageId) -> Vec<(String, String)> {+ let conn = rusqlite::Connection::open(fx.tmp.path().join("db/memory.sqlite")).unwrap();+ let mut stmt = conn+ .prepare("SELECT source_kind, source_id FROM page_evidence WHERE page_id = ?1")+ .unwrap();+ let rows = stmt+ .query_map(params![page_id.as_bytes()], |row| {+ Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))+ })+ .unwrap();+ rows.map(Result::unwrap).collect()+}++fn dedup_on() -> ColdClusterDedup {+ ColdClusterDedup {+ enabled: true,+ embedding: Some(EmbeddingCoord {+ provider: "synthetic".into(),+ model: "bag-of-words-v1".into(),+ dim: DIM,+ }),+ min_pts: 0,+ max_eps: 0.0,+ }+}++// Two near-duplicate bodies: almost-identical prose (so the bag-of-words+// embedder places them close together) but each carrying ONE unique durable+// fact (a distinct file path), so the survivor's keep-token union is testable.+const DUP_A: &str = "Investigated the flaky retention sweep test failure again and again during \+ the long debugging session that afternoon. The unique culprit turned out to be in the file \+ crates/ai-memory-store/src/ops.rs during the run.";+const DUP_B: &str = "Investigated the flaky retention sweep test failure again and again during \+ the long debugging session that afternoon. The unique culprit turned out to be in the file \+ crates/ai-memory-store/src/reader.rs during the run.";+const FAR: &str = "Kubernetes pod scheduling node selector affinity rules for the cluster \+ autoscaler deployment on the staging namespace.";++/// Flag OFF: nothing clusters or merges — byte-identical to before A3.+#[tokio::test]+async fn flag_off_never_merges() {+ let fx = seed_fixture(true).await;+ write(&fx, "sessions/a.md", DUP_A).await;+ write(&fx, "sessions/b.md", DUP_B).await;+ for p in ["sessions/a.md", "sessions/b.md"] {+ backdate_latest(&fx, p, COLD_DAYS);+ }++ let report = run_sweep_with_hygiene(+ &fx.store.reader,+ &fx.store.writer,+ Some(&fx.wiki),+ fx.ws,+ fx.proj,+ &DecayParams::default(),+ 0.0,+ ObservationRetention::default(),+ false,+ ColdClusterDedup::default(), // OFF+ false,+ )+ .await+ .unwrap();++ assert!(report.merged.is_empty(), "flag off must not merge");+ // With dedup off and compaction off, the cold pages are evicted as always.+ assert_eq!(report.evicted.len(), 2, "both cold pages evicted as before");+}++/// Flag ON: the two near-duplicates collapse to one survivor; the survivor keeps+/// BOTH facts; the loser is superseded (reachable); the merge is recorded in+/// `page_evidence`; the far page is untouched.+#[tokio::test]+async fn flag_on_collapses_cluster_and_preserves_recall() {+ let fx = seed_fixture(true).await;+ write(&fx, "sessions/a.md", DUP_A).await;+ write(&fx, "sessions/b.md", DUP_B).await;+ write(&fx, "sessions/far.md", FAR).await;+ for p in ["sessions/a.md", "sessions/b.md", "sessions/far.md"] {+ backdate_latest(&fx, p, COLD_DAYS);+ }++ let report = run_sweep_with_hygiene(+ &fx.store.reader,+ &fx.store.writer,+ Some(&fx.wiki),+ fx.ws,+ fx.proj,+ &DecayParams::default(),+ 0.0,+ ObservationRetention::default(),+ false,+ dedup_on(),+ false,+ )+ .await+ .unwrap();++ assert_eq!(+ report.merged.len(),+ 1,+ "the two near-duplicates form exactly one cluster: {:?}",+ report.merged+ );+ let cluster = &report.merged[0];+ assert!(cluster.applied, "the collapse landed");+ assert_eq!(cluster.merged.len(), 1, "one member merged away");+ let survivor_path = cluster.survivor.clone();+ let loser_path = cluster.merged[0].clone();+ assert!(+ survivor_path != loser_path,+ "survivor and loser are distinct"+ );+ assert!(+ survivor_path.starts_with("sessions/") && loser_path.starts_with("sessions/"),+ "far page is not part of the cluster"+ );+ assert!(+ !report.merged.iter().any(|m| m.survivor == "sessions/far.md"+ || m.merged.contains(&"sessions/far.md".to_string())),+ "the far page must never be clustered"+ );++ // Recall preserved: the survivor body carries BOTH facts — its own and the+ // one that lived only in the merged-away duplicate.+ let survivor_body = latest_body(&fx, &survivor_path);+ assert!(+ survivor_body.contains("crates/ai-memory-store/src/ops.rs"),+ "ops.rs fact present in survivor: {survivor_body}"+ );+ assert!(+ survivor_body.contains("crates/ai-memory-store/src/reader.rs"),+ "reader.rs fact (from the merged-away duplicate) survives in survivor: {survivor_body}"+ );++ // The loser's live version is a merge note pointing at the survivor…+ let loser_body = latest_body(&fx, &loser_path);+ assert!(+ loser_body.contains("merged into") && loser_body.contains(&survivor_path),+ "loser points at survivor: {loser_body}"+ );+ // …and its full pre-merge body stays reachable via supersession (reversible).+ let loser_history = superseded_bodies(&fx, &loser_path);+ assert!(+ loser_history.iter().any(|b| b.contains("unique culprit")),+ "the loser's pre-merge body stays reachable: {loser_history:?}"+ );++ // Merge provenance recorded on the new survivor version.+ let new_survivor = cluster.new_survivor_id.expect("new survivor id");+ let evidence = evidence_for(&fx, new_survivor);+ assert!(+ evidence+ .iter()+ .any(|(kind, src)| kind == "reconsolidation" && src.starts_with("a3_merge:")),+ "page_evidence records the merge: {evidence:?}"+ );++ // The far page was neither merged nor (being cold) skipped: it evicts.+ assert!(+ report.evicted.iter().any(|e| e.path == "sessions/far.md"),+ "the far page follows the normal cold path: {:?}",+ report.evicted+ );+}++/// No embeddings for the configured triple ⇒ A3 is a clean no-op, not an error.+#[tokio::test]+async fn no_embeddings_is_a_clean_no_op() {+ let fx = seed_fixture(false).await; // no embedder → no embedding rows+ write(&fx, "sessions/a.md", DUP_A).await;+ write(&fx, "sessions/b.md", DUP_B).await;+ for p in ["sessions/a.md", "sessions/b.md"] {+ backdate_latest(&fx, p, COLD_DAYS);+ }++ let report = run_sweep_with_hygiene(+ &fx.store.reader,+ &fx.store.writer,+ Some(&fx.wiki),+ fx.ws,+ fx.proj,+ &DecayParams::default(),+ 0.0,+ ObservationRetention::default(),+ false,+ dedup_on(), // ON, but there are no vectors to cluster+ false,+ )+ .await+ .unwrap();++ assert!(+ report.merged.is_empty(),+ "no embeddings ⇒ no merges, no error"+ );+ assert_eq!(report.evicted.len(), 2, "cold pages follow the normal path");+}diff --git a/crates/ai-memory-consolidate/tests/suite/entropy_experience.rs b/crates/ai-memory-consolidate/tests/suite/entropy_experience.rsnew file mode 100644index 00000000..67766805--- /dev/null+++ b/crates/ai-memory-consolidate/tests/suite/entropy_experience.rs@@ -0,0 +1,231 @@+//! A4 entropy / boilerplate pre-filter, wired into the cross-session experience+//! pass (docs/design-memory-aging.md §A4).+//!+//! The gate runs BEFORE consolidation ingest: a low-information session page is+//! never placed in the consolidation prompt (and thus never reaches the eval+//! gate or `apply_batch`), while a real one is. Default (OFF) changes nothing.+//! The proof captures the prompt the LLM would receive and asserts which session+//! bodies made it in — skipping is advisory (the page is not deleted, invariant+//! #16).++use std::future::Future;+use std::pin::Pin;+use std::sync::{Arc, Mutex};++use ai_memory_consolidate::entropy_filter::EntropyFilterConfig;+use ai_memory_consolidate::{AutoImproveReviewConfig, ExperienceConfig, run_experience_review};+use ai_memory_core::{ActorContext, NewSession, SessionId};+use ai_memory_llm::{ChatRequest, ChatResponse, LlmProvider, LlmResult};+use ai_memory_store::Store;+use ai_memory_wiki::{Wiki, WritePageRequest};+use tempfile::TempDir;++/// LLM stub that records every prompt it is asked to complete, so the test can+/// assert which session bodies survived the entropy gate. Returns an empty (but+/// valid) proposal set.+struct CapturingLlm {+ prompts: Arc<Mutex<Vec<String>>>,+}++impl LlmProvider for CapturingLlm {+ fn name(&self) -> &'static str {+ "fake"+ }+ fn model(&self) -> &str {+ "capturing"+ }+ fn complete<'life0, 'async_trait>(+ &'life0 self,+ _request: ChatRequest,+ ) -> Pin<Box<dyn Future<Output = LlmResult<ChatResponse>> + Send + 'async_trait>>+ where+ 'life0: 'async_trait,+ Self: 'async_trait,+ {+ Box::pin(async move {+ Ok(ChatResponse {+ text: "unused".into(),+ usage: None,+ model: "capturing".into(),+ })+ })+ }+ fn complete_structured_raw<'life0, 'async_trait>(+ &'life0 self,+ request: ChatRequest,+ _schema: serde_json::Value,+ ) -> Pin<Box<dyn Future<Output = LlmResult<serde_json::Value>> + Send + 'async_trait>>+ where+ 'life0: 'async_trait,+ Self: 'async_trait,+ {+ let prompts = self.prompts.clone();+ Box::pin(async move {+ if let Some(msg) = request.messages.first() {+ prompts.lock().unwrap().push(msg.content.clone());+ }+ Ok(serde_json::json!({+ "summary": "none",+ "proposals": [],+ "rejected_candidates": []+ }))+ })+ }+}++// A repetitive boilerplate status blob — no durable facts.+const BOILERPLATE: &str = "status ok status ok status ok status ok status ok status ok status ok";+// A real, terse-but-informative note with a file path and an error code.+const REAL: &str = "Fixed the retention sweep in crates/ai-memory-store/src/ops.rs which raised \+ E0433; tag main then deploy the stack.";++struct Harness {+ _tmp: TempDir,+ store: Store,+ ws: ai_memory_core::WorkspaceId,+ proj: ai_memory_core::ProjectId,+ prompts: Arc<Mutex<Vec<String>>>,+ llm: Arc<dyn LlmProvider>,+}++async fn harness_with_two_sessions() -> Harness {+ let tmp = TempDir::new().unwrap();+ let store = Store::open(tmp.path()).unwrap();+ let wiki = Wiki::new(tmp.path(), store.writer.clone())+ .unwrap()+ .with_store_reader(store.reader.clone());+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ let proj = store+ .writer+ .get_or_create_project(ws, "scratch", None)+ .await+ .unwrap();++ for body in [BOILERPLATE, REAL] {+ let session_id = SessionId::new();+ store+ .writer+ .begin_session(NewSession {+ id: session_id,+ workspace_id: ws,+ project_id: proj,+ agent_kind: ai_memory_core::AgentKind::OpenCode,+ cwd: None,+ actor_user: None,+ })+ .await+ .unwrap();+ store.writer.end_session(session_id, None).await.unwrap();+ wiki.write_page(WritePageRequest {+ workspace_id: ws,+ project_id: proj,+ path: ai_memory_core::PagePath::new(format!("sessions/{session_id}.md")).unwrap(),+ frontmatter: serde_json::json!({"title": "session"}),+ body: body.to_string(),+ tier: ai_memory_core::Tier::Episodic,+ pinned: false,+ title: None,+ admission_ctx: None,+ author_id: None,+ actor: ActorContext::anonymous(),+ evidence: Vec::new(),+ })+ .await+ .unwrap();+ }++ let prompts = Arc::new(Mutex::new(Vec::new()));+ let llm: Arc<dyn LlmProvider> = Arc::new(CapturingLlm {+ prompts: prompts.clone(),+ });+ Harness {+ _tmp: tmp,+ store,+ ws,+ proj,+ prompts,+ llm,+ }+}++fn experience_cfg(filter_enabled: bool) -> ExperienceConfig {+ ExperienceConfig {+ sessions: 10,+ min_new_sessions: 1,+ entropy_filter: EntropyFilterConfig {+ enabled: filter_enabled,+ ..EntropyFilterConfig::default()+ },+ ..ExperienceConfig::default()+ }+}++/// Filter ON: the boilerplate session page is skipped from the prompt; the real+/// one is consolidated.+#[tokio::test]+async fn boilerplate_session_is_skipped_from_consolidation() {+ let h = harness_with_two_sessions().await;+ let report = run_experience_review(+ &h.store.reader,+ h.llm.as_ref(),+ h.ws,+ h.proj,+ AutoImproveReviewConfig::default(),+ &experience_cfg(true),+ )+ .await+ .unwrap();++ let prompts = h.prompts.lock().unwrap();+ assert_eq!(prompts.len(), 1, "the LLM was still called once");+ let prompt = &prompts[0];+ assert!(+ prompt.contains("crates/ai-memory-store/src/ops.rs"),+ "the real note is consolidated: {prompt}"+ );+ assert!(+ !prompt.contains("status ok status ok"),+ "the boilerplate note is skipped from the prompt: {prompt}"+ );+ assert!(+ report+ .warnings+ .iter()+ .any(|w| w.contains("entropy filter skipped")),+ "the skip is observable in the report: {:?}",+ report.warnings+ );+}++/// Default (OFF): nothing is filtered — both session pages reach the prompt, so+/// an upgrade changes no consolidation output.+#[tokio::test]+async fn default_off_changes_nothing() {+ let h = harness_with_two_sessions().await;+ run_experience_review(+ &h.store.reader,+ h.llm.as_ref(),+ h.ws,+ h.proj,+ AutoImproveReviewConfig::default(),+ &experience_cfg(false),+ )+ .await+ .unwrap();++ let prompts = h.prompts.lock().unwrap();+ assert_eq!(prompts.len(), 1);+ let prompt = &prompts[0];+ assert!(+ prompt.contains("crates/ai-memory-store/src/ops.rs"),+ "real note present: {prompt}"+ );+ assert!(+ prompt.contains("status ok status ok"),+ "with the filter off the boilerplate is NOT skipped: {prompt}"+ );+}diff --git a/crates/ai-memory-consolidate/tests/suite/mod.rs b/crates/ai-memory-consolidate/tests/suite/mod.rsindex 187fcd7a..1cf49841 100644--- a/crates/ai-memory-consolidate/tests/suite/mod.rs+++ b/crates/ai-memory-consolidate/tests/suite/mod.rs@@ -4,9 +4,11 @@mod abstract_backfill;mod access_breadth_sweep;+mod cold_cluster_sweep;mod compaction_sweep;mod embed_backfill;mod embeddings;+mod entropy_experience;mod lifecycle;mod local_embeddings;mod multi_machine;diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex f77466a8..4c1c49ff 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -603,6 +603,17 @@ compact_cold_episodic = false # A2 opt-in: tier-down (compact) a cold# keep abstract+summary+keep-tokens, drop# prose. Reversible (git + supersession),# zero-LLM. false = today's evict behaviour.+dedup_cold_clusters = false # A3 opt-in: cluster near-duplicate cold+ # episodic pages by embedding (cosine DBSCAN,+ # adaptive eps) and collapse each cluster to+ # one survivor (union of keep-tokens), others+ # superseded with a merge note. Reversible+ # (git + supersession), zero generative LLM,+ # no-op with no embedder. Merge provenance in+ # page_evidence. false = no clustering.+# dedup_min_pts = 2 # DBSCAN density floor (0 ⇒ default 2)+# dedup_max_eps = 0.15 # conservative eps ceiling (cosine distance;+ # 0 ⇒ default). Lower = merges less.[decay.half_life_days] # opt-in per-tier retention curves (all keys# optional). Half-life in DAYS; converted to@@ -651,6 +662,18 @@ enabled = trueinterval_secs = 3600max_sessions_per_tick = 1 # per project; scheduler ticks do not overlapmin_session_age_secs = 600+experience_every_sessions = 0 # 0 disables the cross-session experience pass+experience_sessions = 10 # session summaries one experience pass reads++[auto_improve.scheduler.experience_entropy_filter] # A4 opt-in; off by default+enabled = false # true: skip low-information session pages from+ # the experience consolidation pass BEFORE the+ # prompt/eval-gate/apply_batch. Advisory (skip,+ # never delete); zero-LLM. false = no filtering.+# min_chars = 16 # near-empty floor (non-whitespace chars)+# min_entropy_bits_per_char = 2.0+# max_repetition_ratio = 0.7 # 1 - distinct/total tokens above this ⇒ skip+# repetition_min_tokens = 6 # repetition check applies only above this[retrieval] # opt-in ranking signals; all off by defaultquery_intent = false # lexical session-recall routing: queries phrased as