Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(consolidate): A3 cold-cluster dedup + A4 entropy pre-filter (opt-in)

AkitaOnRails3c227ade1f6e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index eb570a6d..2f7bf4f9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
+- Cold-cluster dedup of near-duplicate episodic pages (design-memory-aging.md
+ bucket A3): the forget-sweep can now cluster near-duplicate cold episodic
+ pages by embedding (cosine-distance DBSCAN with an adaptive k-distance eps,
+ `minPts = 2`) and collapse each cluster to one survivor — the highest-retention
+ member, its body the *extractive union* of the cluster's keep-tokens, so every
+ member's durable facts survive — superseding the other members with a merge
+ note that points at the survivor. It runs only over the bounded cold-episodic
+ candidate set the sweep already materialises (never O(N²) over the whole
+ corpus), is **opt-in and off by default** via `[decay] dedup_cold_clusters`
+ (a `false` default), and fully **zero generative LLM**: it reads only
+ already-stored embeddings, so with no embedder configured — or no embeddings
+ for the configured `(provider, model, dim)` triple — it is a clean no-op, not
+ an error. The eps is clamped to a conservative ceiling (`[decay] dedup_max_eps`,
+ default cosine distance ≈ 0.15) so it errs toward NOT merging. **Non-destructive
+ and reversible**: no source is ever hard-deleted — every merged-away member
+ stays reachable via the supersession chain and git history and is recoverable
+ with `restore-page` (invariant #16) — and the merge provenance is recorded in
+ `page_evidence`. Every run reports its collapses in the `SweepReport`. Reuses
+ existing tables: **no new migration**, and no new MCP tool (still 23). Ships
+ opt-in/off; the R2 recall no-regression proof is the gate before any future
+ default-on (#809).
+- Entropy / boilerplate pre-filter before consolidation (design-memory-aging.md
+ bucket A4): a pure, zero-LLM Shannon-entropy + boilerplate gate that skips
+ low-information session pages (near-empty, whitespace, single-character, or
+ highly-repetitive boilerplate) from the cross-session experience consolidation
+ pass *before* they reach the LLM prompt, the eval gate, or `apply_batch`.
+ It is **advisory and non-destructive** — a skipped page is not consolidated,
+ never deleted (invariant #16) — and **opt-in / off by default** via
+ `[auto_improve.scheduler.experience_entropy_filter]` (a `false` default with
+ conservative, validated thresholds tuned so a terse-but-informative note with
+ a file path and an error code is KEPT), so an upgrade changes no consolidation
+ output until an operator opts in. Every run surfaces the skip count in the
+ experience report warnings. No schema change and no new MCP tool (still 23)
+ (#809).
- Extractive tier-down of cold episodic pages (design-memory-aging.md bucket
A2): instead of evicting a cold episodic page, the forget-sweep can now
*compact* it — keeping the L0 frontmatter `abstract:`, an L1 first-paragraph
diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rs
index 32a529bf..a9d91472 100644
--- a/crates/ai-memory-cli/src/commands/serve.rs
+++ b/crates/ai-memory-cli/src/commands/serve.rs
@@ -9,7 +9,7 @@ use std::time::Duration;
use ai_memory_consolidate::{
AutoImproveReviewConfig, Consolidator, EmbedBackfillOptions, ObservationRetention,
ScheduledAutoImproveSettings, run_auto_improve_scheduler_tick, run_embedding_backfill,
- run_lint, run_sweep_with_compaction,
+ run_lint,
};
use ai_memory_core::{ActiveProject, ProjectId, Sanitizer, WorkspaceId};
use ai_memory_hooks::{
@@ -1564,11 +1564,23 @@ async fn start_maintenance_scheduler(
let lint_interval_secs = settings.lint_interval_secs;
let embedding_backfill_interval_secs = settings.embedding_backfill_interval_secs;
+ // A3 cold-cluster dedup targets the running server's configured embedder
+ // coordinate; with no embedder it is `None`, making A3 a clean no-op even
+ // when the flag is set (there are no stored vectors to cluster).
+ let dedup_embedding = embedder
+ .as_ref()
+ .map(|e| ai_memory_consolidate::EmbeddingCoord {
+ provider: e.provider().to_string(),
+ model: e.model().to_string(),
+ dim: e.dim(),
+ });
+
let mut tasks = Vec::new();
if maintenance_enabled && forget_sweep_interval_secs > 0 {
let reader = reader.clone();
let writer = writer.clone();
let wiki = wiki.clone();
+ let dedup_embedding = dedup_embedding.clone();
tasks.push(tokio::spawn(async move {
let interval = std::time::Duration::from_secs(forget_sweep_interval_secs);
run_persisted_maintenance_job(
@@ -1594,6 +1606,7 @@ async fn start_maintenance_scheduler(
let writer = writer.clone();
let wiki = wiki.clone();
let decay = decay;
+ let dedup_embedding = dedup_embedding.clone();
async move {
let started = std::time::Instant::now();
let outcome = run_scheduled_sweep_tick(
@@ -1604,6 +1617,7 @@ async fn start_maintenance_scheduler(
decay.breadth_weight,
decay.observation_retention(),
decay.compact_cold_episodic,
+ decay.cold_cluster_dedup(dedup_embedding),
)
.await?;
if outcome.errors > 0 {
@@ -1805,6 +1819,7 @@ async fn start_maintenance_scheduler(
ai_memory_consolidate::ExperienceConfig {
sessions: scheduler.experience_sessions.max(1),
min_new_sessions: scheduler.experience_every_sessions,
+ entropy_filter: scheduler.experience_entropy_filter,
..ai_memory_consolidate::ExperienceConfig::default()
}
}),
@@ -1878,6 +1893,7 @@ struct ScheduledSweepTickOutcome {
errors: usize,
}
+#[allow(clippy::too_many_arguments)]
async fn run_scheduled_sweep_tick(
reader: &ReaderPool,
writer: &WriterHandle,
@@ -1886,6 +1902,7 @@ async fn run_scheduled_sweep_tick(
breadth_weight: f64,
retention: ObservationRetention,
compact_cold_episodic: bool,
+ dedup: ai_memory_consolidate::ColdClusterDedup,
) -> Result<ScheduledSweepTickOutcome> {
let scopes = reader.list_all_scopes().await?;
let mut outcome = ScheduledSweepTickOutcome {
@@ -1894,7 +1911,7 @@ async fn run_scheduled_sweep_tick(
};
for scope in scopes {
- match run_sweep_with_compaction(
+ match ai_memory_consolidate::run_sweep_with_hygiene(
reader,
writer,
Some(wiki),
@@ -1904,6 +1921,7 @@ async fn run_scheduled_sweep_tick(
breadth_weight,
retention,
compact_cold_episodic,
+ dedup.clone(),
false,
)
.await
@@ -3665,6 +3683,7 @@ mod tests {
0.0,
ObservationRetention::default(),
false,
+ ai_memory_consolidate::ColdClusterDedup::default(),
)
.await
.unwrap();
diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rs
index 0fe483a9..2869cb1e 100644
--- a/crates/ai-memory-cli/src/config.rs
+++ b/crates/ai-memory-cli/src/config.rs
@@ -101,6 +101,19 @@ pub struct DecaySettings {
/// chain) and non-destructive. Defaults to `false`, so an upgrade changes
/// nothing until an operator opts in.
pub compact_cold_episodic: bool,
+ /// A3 cold-cluster dedup (`[decay] dedup_cold_clusters`). When `true` AND an
+ /// embedder is configured, the forget sweep clusters near-duplicate cold
+ /// episodic pages by embedding (cosine DBSCAN, adaptive eps) and collapses
+ /// each cluster to one survivor via supersession + a merge note.
+ /// Non-destructive (merged-away members stay reachable) and zero generative
+ /// LLM. Defaults to `false`, and is a clean no-op with no embedder, so an
+ /// upgrade changes nothing until an operator opts in.
+ pub dedup_cold_clusters: bool,
+ /// DBSCAN density floor for A3. `0` ⇒ the conservative default (2).
+ pub dedup_min_pts: usize,
+ /// Conservative ceiling on the adaptive eps (cosine distance) for A3.
+ /// `0.0` ⇒ the conservative default. Lower errs harder toward NOT merging.
+ pub dedup_max_eps: f32,
/// Optional per-tier half-life overrides (`[decay.half_life_days]`). All
/// keys default to unset ⇒ the scalar `lambda` applies to every tier, which
/// is byte-identical to the historical single-λ behaviour.
@@ -121,6 +134,9 @@ impl Default for DecaySettings {
observation_retention_days: 0,
observation_prune_batch: ai_memory_consolidate::DEFAULT_OBSERVATION_PRUNE_BATCH,
compact_cold_episodic: false,
+ dedup_cold_clusters: false,
+ dedup_min_pts: 0,
+ dedup_max_eps: 0.0,
half_life_days: DecayHalfLifeDays::default(),
}
}
@@ -174,6 +190,24 @@ impl DecaySettings {
batch: self.observation_prune_batch,
}
}
+
+ /// A3 cold-cluster dedup options for the M8 sweep.
+ ///
+ /// `embedding` is the running server's configured embedder coordinate, or
+ /// `None` when no embedder is configured — in which case A3 is a clean no-op
+ /// even with the flag on (there are no stored vectors to cluster).
+ #[must_use]
+ pub fn cold_cluster_dedup(
+ self,
+ embedding: Option<ai_memory_consolidate::EmbeddingCoord>,
+ ) -> ai_memory_consolidate::ColdClusterDedup {
+ ai_memory_consolidate::ColdClusterDedup {
+ enabled: self.dedup_cold_clusters,
+ embedding,
+ min_pts: self.dedup_min_pts,
+ max_eps: self.dedup_max_eps,
+ }
+ }
}
/// One `[[llm_fallbacks]]` entry: an ordered LLM provider tried only after
@@ -961,6 +995,11 @@ pub struct AutoImproveSchedulerSettings {
pub experience_every_sessions: u64,
/// How many recent session summary pages one experience pass reads.
pub experience_sessions: usize,
+ /// A4 entropy / boilerplate pre-filter for the experience pass
+ /// (`[auto_improve.scheduler.experience_entropy_filter]`). Off by default:
+ /// low-information session pages are skipped from consolidation only when an
+ /// operator enables it. Advisory (skip, never delete).
+ pub experience_entropy_filter: ai_memory_consolidate::EntropyFilterConfig,
}
impl Default for AutoImproveSchedulerSettings {
@@ -972,6 +1011,7 @@ impl Default for AutoImproveSchedulerSettings {
min_session_age_secs: 600,
experience_every_sessions: 0,
experience_sessions: 10,
+ experience_entropy_filter: ai_memory_consolidate::EntropyFilterConfig::default(),
}
}
}
@@ -1240,6 +1280,16 @@ impl Config {
if config.decay.observation_prune_batch == 0 {
anyhow::bail!("decay.observation_prune_batch must be greater than zero");
}
+ // A4 entropy filter thresholds: reject an unusable threshold at startup
+ // rather than silently ignoring it on the first experience pass.
+ if let Err(message) = config
+ .auto_improve
+ .scheduler
+ .experience_entropy_filter
+ .validate()
+ {
+ anyhow::bail!("auto_improve.scheduler.experience_{message}");
+ }
// Fail at startup rather than shipping a prompt that is all scaffolding
// and no observations: below this floor the fixed system prompt and page
diff --git a/crates/ai-memory-consolidate/src/cold_cluster.rs b/crates/ai-memory-consolidate/src/cold_cluster.rs
new file mode 100644
index 00000000..27aa7af5
--- /dev/null
+++ b/crates/ai-memory-consolidate/src/cold_cluster.rs
@@ -0,0 +1,302 @@
+//! Pure, zero-LLM cold-cluster algorithm for A3 (docs/design-memory-aging.md
+//! §A3): DBSCAN over embeddings with an adaptive k-distance eps.
+//!
+//! This module is the *algorithm* only — pure functions over vectors, no store,
+//! no wiki, no provider (invariant #13, zero-LLM). The forget sweep is the
+//! orchestrator: it materialises the bounded cold-episodic set, loads its
+//! embeddings, calls [`adaptive_eps`] and [`dbscan`] here, and collapses each
+//! returned cluster via supersession. Keeping the math here makes it unit
+//! testable and keeps the O(N²) pairwise distance confined to the bounded cold
+//! set the sweep already holds.
+//!
+//! Distance is **cosine distance** (`1 - cosine_similarity`). Stored embeddings
+//! are unit-normalised, so cosine similarity is the dot product; the code
+//! normalises defensively anyway so a caller passing raw vectors still gets a
+//! correct distance.
+
+/// Cosine distance between two vectors: `1 - cos(θ)`, clamped to `[0, 2]`.
+///
+/// Returns `1.0` (maximally distant, the neutral default) for a
+/// zero-magnitude vector or a length mismatch, so a degenerate row can never
+/// pull unrelated pages into a cluster.
+#[must_use]
+pub fn cosine_distance(a: &[f32], b: &[f32]) -> f32 {
+ if a.len() != b.len() {
+ return 1.0;
+ }
+ let mut dot = 0.0f32;
+ let mut na = 0.0f32;
+ let mut nb = 0.0f32;
+ for (x, y) in a.iter().zip(b.iter()) {
+ dot += x * y;
+ na += x * x;
+ nb += y * y;
+ }
+ if na <= 0.0 || nb <= 0.0 {
+ return 1.0;
+ }
+ let sim = dot / (na.sqrt() * nb.sqrt());
+ (1.0 - sim).clamp(0.0, 2.0)
+}
+
+/// Pick a conservative eps from the k-distance elbow (the field-standard DBSCAN
+/// heuristic).
+///
+/// For each point, take the distance to its `k`-th nearest neighbour; sort those
+/// k-distances ascending; the "elbow" — the point of maximum curvature — is the
+/// eps at which density drops off. It is located by the maximum perpendicular
+/// distance from the chord joining the first and last sorted k-distances (the
+/// kneedle construction). The result is then clamped to `max_eps` so an operator
+/// keeps a hard conservative ceiling: A3 errs toward NOT merging.
+///
+/// Returns `None` when there are too few points to form a `k`-distance
+/// (`points.len() <= k`), which the caller treats as "nothing to cluster".
+#[must_use]
+pub fn adaptive_eps(points: &[Vec<f32>], k: usize, max_eps: f32) -> Option<f32> {
+ let n = points.len();
+ if k == 0 || n <= k {
+ return None;
+ }
+ // k-th nearest-neighbour distance for every point.
+ let mut k_distances: Vec<f32> = Vec::with_capacity(n);
+ for (i, p) in points.iter().enumerate() {
+ let mut dists: Vec<f32> = points
+ .iter()
+ .enumerate()
+ .filter(|(j, _)| *j != i)
+ .map(|(_, q)| cosine_distance(p, q))
+ .collect();
+ dists.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
+ // `dists` excludes self, so the k-th nearest neighbour is index k-1.
+ k_distances.push(dists[k - 1]);
+ }
+ k_distances.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
+
+ let elbow = knee_value(&k_distances);
+ Some(elbow.clamp(0.0, max_eps))
+}
+
+/// The elbow (knee) of an ascending curve, by maximum perpendicular distance
+/// from the chord between its first and last points. For a flat or two-point
+/// curve it returns the last value; the caller's `max_eps` clamp bounds it.
+fn knee_value(sorted: &[f32]) -> f32 {
+ let n = sorted.len();
+ if n == 0 {
+ return 0.0;
+ }
+ if n <= 2 {
+ return sorted[n - 1];
+ }
+ let x0 = 0.0f32;
+ let y0 = sorted[0];
+ #[allow(clippy::cast_precision_loss)]
+ let x1 = (n - 1) as f32;
+ let y1 = sorted[n - 1];
+ let dx = x1 - x0;
+ let dy = y1 - y0;
+ let denom = (dx * dx + dy * dy).sqrt();
+ if denom <= 0.0 {
+ return sorted[n - 1];
+ }
+ let mut best_idx = n - 1;
+ let mut best_dist = -1.0f32;
+ for (i, &y) in sorted.iter().enumerate() {
+ #[allow(clippy::cast_precision_loss)]
+ let x = i as f32;
+ // Perpendicular distance from (x, y) to the chord (x0,y0)-(x1,y1).
+ let perp = ((dy * (x - x0)) - (dx * (y - y0))).abs() / denom;
+ if perp > best_dist {
+ best_dist = perp;
+ best_idx = i;
+ }
+ }
+ sorted[best_idx]
+}
+
+/// DBSCAN over `points` with cosine distance. Returns the discovered clusters as
+/// lists of indices into `points`; noise points are omitted (not returned as
+/// singletons). Deterministic: points are visited in index order and each
+/// cluster's members are in the order they were reached.
+///
+/// `min_pts` is the density floor (a core point has at least `min_pts` points,
+/// itself included, within `eps`). A3 keeps `min_pts` small (2): a pair of
+/// near-duplicate cold pages is exactly what it wants to collapse, and the
+/// conservative `eps` is the guard against over-merging.
+#[must_use]
+pub fn dbscan(points: &[Vec<f32>], eps: f32, min_pts: usize) -> Vec<Vec<usize>> {
+ let n = points.len();
+ let mut labels: Vec<Label> = vec![Label::Unvisited; n];
+ let mut clusters: Vec<Vec<usize>> = Vec::new();
+
+ for i in 0..n {
+ if labels[i] != Label::Unvisited {
+ continue;
+ }
+ let neighbors = region_query(points, i, eps);
+ if neighbors.len() < min_pts {
+ labels[i] = Label::Noise;
+ continue;
+ }
+ let cluster_id = clusters.len();
+ let mut members: Vec<usize> = Vec::new();
+ labels[i] = Label::Clustered(cluster_id);
+ members.push(i);
+
+ // Expand the cluster over a growing frontier (iterative, not recursive,
+ // so a large dense set cannot blow the stack).
+ let mut queue = std::collections::VecDeque::from(neighbors);
+ while let Some(q) = queue.pop_front() {
+ match labels[q] {
+ Label::Noise => {
+ // A border point: attach it, but do not expand from it.
+ labels[q] = Label::Clustered(cluster_id);
+ members.push(q);
+ }
+ Label::Unvisited => {
+ labels[q] = Label::Clustered(cluster_id);
+ members.push(q);
+ let q_neighbors = region_query(points, q, eps);
+ if q_neighbors.len() >= min_pts {
+ for nb in q_neighbors {
+ if labels[nb] == Label::Unvisited || labels[nb] == Label::Noise {
+ queue.push_back(nb);
+ }
+ }
+ }
+ }
+ Label::Clustered(_) => {}
+ }
+ }
+ clusters.push(members);
+ }
+ clusters
+}
+
+#[derive(Clone, Copy, PartialEq, Eq)]
+enum Label {
+ Unvisited,
+ Noise,
+ Clustered(usize),
+}
+
+/// Indices of every point within `eps` of `idx`, including `idx` itself.
+fn region_query(points: &[Vec<f32>], idx: usize, eps: f32) -> Vec<usize> {
+ let p = &points[idx];
+ (0..points.len())
+ .filter(|&j| cosine_distance(p, &points[j]) <= eps)
+ .collect()
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Three tight points near one axis, one far point on another: DBSCAN must
+ /// find exactly one cluster of the three, and drop the far point as noise.
+ #[test]
+ fn three_tight_points_and_one_far_point() {
+ let points = vec![
+ vec![1.0, 0.0, 0.0],
+ vec![0.98, 0.02, 0.0],
+ vec![0.99, 0.0, 0.01],
+ vec![0.0, 1.0, 0.0], // far
+ ];
+ let clusters = dbscan(&points, 0.1, 2);
+ assert_eq!(clusters.len(), 1, "one cluster only: {clusters:?}");
+ let mut members = clusters[0].clone();
+ members.sort_unstable();
+ assert_eq!(members, vec![0, 1, 2], "the three tight points cluster");
+ assert!(
+ !clusters[0].contains(&3),
+ "the far point is noise, not clustered"
+ );
+ }
+
+ #[test]
+ fn deterministic_across_runs() {
+ let points = vec![
+ vec![1.0, 0.0],
+ vec![0.99, 0.01],
+ vec![0.0, 1.0],
+ vec![0.01, 0.99],
+ ];
+ let a = dbscan(&points, 0.1, 2);
+ let b = dbscan(&points, 0.1, 2);
+ assert_eq!(a, b, "same input yields identical clustering");
+ assert_eq!(a.len(), 2, "two well-separated pairs: {a:?}");
+ }
+
+ #[test]
+ fn adaptive_eps_picks_a_sane_threshold() {
+ // Two tight pairs, far apart. The k-distance elbow should sit above the
+ // within-pair distance (so pairs cluster) and below the across-pair
+ // distance (so the two pairs stay separate).
+ let points = vec![
+ vec![1.0, 0.0],
+ vec![0.999, 0.001],
+ vec![0.0, 1.0],
+ vec![0.001, 0.999],
+ ];
+ let eps = adaptive_eps(&points, 2, 0.5).expect("eps");
+ assert!(eps > 0.0, "a positive threshold: {eps}");
+ assert!(eps <= 0.5, "clamped to the conservative ceiling: {eps}");
+ let clusters = dbscan(&points, eps, 2);
+ assert_eq!(
+ clusters.len(),
+ 2,
+ "adaptive eps keeps the two tight pairs separate: {clusters:?}"
+ );
+ }
+
+ #[test]
+ fn adaptive_eps_respects_the_conservative_ceiling() {
+ // A spread-out set whose natural elbow is large; the ceiling clamps it.
+ let points = vec![
+ vec![1.0, 0.0],
+ vec![0.0, 1.0],
+ vec![-1.0, 0.0],
+ vec![0.0, -1.0],
+ ];
+ let eps = adaptive_eps(&points, 2, 0.05).expect("eps");
+ assert!(eps <= 0.05, "ceiling honoured: {eps}");
+ }
+
+ #[test]
+ fn too_few_points_yields_no_eps() {
+ assert_eq!(adaptive_eps(&[vec![1.0, 0.0]], 2, 0.5), None);
+ assert_eq!(adaptive_eps(&[], 2, 0.5), None);
+ }
+
+ #[test]
+ fn no_cluster_when_everything_is_far() {
+ let points = vec![
+ vec![1.0, 0.0, 0.0],
+ vec![0.0, 1.0, 0.0],
+ vec![0.0, 0.0, 1.0],
+ ];
+ let clusters = dbscan(&points, 0.1, 2);
+ assert!(clusters.is_empty(), "no near-duplicates, no clusters");
+ }
+
+ #[test]
+ fn cosine_distance_basics() {
+ assert!(
+ cosine_distance(&[1.0, 0.0], &[1.0, 0.0]) < 1e-6,
+ "identical = 0"
+ );
+ assert!(
+ (cosine_distance(&[1.0, 0.0], &[0.0, 1.0]) - 1.0).abs() < 1e-6,
+ "orthogonal = 1"
+ );
+ assert_eq!(
+ cosine_distance(&[0.0, 0.0], &[1.0, 0.0]),
+ 1.0,
+ "zero vector = neutral 1"
+ );
+ assert_eq!(
+ cosine_distance(&[1.0], &[1.0, 0.0]),
+ 1.0,
+ "mismatch = neutral 1"
+ );
+ }
+}
diff --git a/crates/ai-memory-consolidate/src/entropy_filter.rs b/crates/ai-memory-consolidate/src/entropy_filter.rs
new file mode 100644
index 00000000..bac92a07
--- /dev/null
+++ b/crates/ai-memory-consolidate/src/entropy_filter.rs
@@ -0,0 +1,295 @@
+//! Pure, zero-LLM entropy / boilerplate pre-filter for A4
+//! (docs/design-memory-aging.md §A4).
+//!
+//! Before a consolidation pass reads a page or observation, this gate answers a
+//! cheap question: does this text carry enough information to be worth
+//! consolidating, or is it near-empty, whitespace, or a highly-repetitive
+//! boilerplate blob? Low-information input is *skipped* from the pass — never
+//! deleted (invariant #16, advisory only). Skipping it keeps noise out of every
+//! downstream step (the LLM prompt, the dedup clustering) at no provider cost.
+//!
+//! The gate is a pure function of the text plus a small config (invariant #13,
+//! zero-LLM). It is **off by default**: an unconfigured [`EntropyFilterConfig`]
+//! keeps every item, so an upgrade changes no consolidation output until an
+//! operator opts in. The thresholds are deliberately conservative — a terse but
+//! informative note (a one-line fix with a file path and an error code) must be
+//! KEPT; only genuinely low-signal text is skipped.
+
+use std::collections::HashMap;
+
+use serde::{Deserialize, Serialize};
+
+/// Why a piece of text was skipped from consolidation. Advisory: a skip means
+/// "not worth a consolidation pass right now", never "delete".
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum SkipReason {
+ /// Fewer non-whitespace characters than the configured floor — near-empty
+ /// or whitespace-only.
+ NearEmpty,
+ /// Shannon entropy per character is below the configured floor — a single
+ /// repeated character or a tiny alphabet carrying almost no information.
+ LowEntropy,
+ /// The same handful of tokens repeat — a boilerplate blob (e.g. a status
+ /// banner echoed many times).
+ Repetitive,
+}
+
+impl SkipReason {
+ /// Stable short string for reports/logs.
+ #[must_use]
+ pub const fn as_str(self) -> &'static str {
+ match self {
+ Self::NearEmpty => "near_empty",
+ Self::LowEntropy => "low_entropy",
+ Self::Repetitive => "repetitive",
+ }
+ }
+}
+
+/// The gate's verdict for one piece of text.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FilterVerdict {
+ /// Consolidate this text as usual.
+ Keep,
+ /// Skip this text from consolidation, for the given reason.
+ Skip(SkipReason),
+}
+
+impl FilterVerdict {
+ /// `true` when the text should be skipped from consolidation.
+ #[must_use]
+ pub const fn is_skip(self) -> bool {
+ matches!(self, Self::Skip(_))
+ }
+}
+
+/// Configuration for the entropy / boilerplate gate.
+///
+/// **Off by default** (`enabled = false`): the gate keeps everything, so an
+/// existing install's consolidation output is byte-identical until an operator
+/// turns it on. The thresholds are intentionally low so that turning it on only
+/// removes clear noise.
+#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
+#[serde(default)]
+pub struct EntropyFilterConfig {
+ /// Master switch. `false` (the default) makes [`classify`] return
+ /// [`FilterVerdict::Keep`] for every input.
+ pub enabled: bool,
+ /// Minimum non-whitespace character count. Text shorter than this is
+ /// [`SkipReason::NearEmpty`].
+ pub min_chars: usize,
+ /// Minimum Shannon entropy in bits per character. Text below this is
+ /// [`SkipReason::LowEntropy`].
+ pub min_entropy_bits_per_char: f64,
+ /// A repetition ratio (`1 - distinct_tokens / total_tokens`) above this is
+ /// [`SkipReason::Repetitive`]. Only applied once a body has at least
+ /// [`Self::repetition_min_tokens`] tokens, so a legitimately terse note is
+ /// never judged repetitive.
+ pub max_repetition_ratio: f64,
+ /// Token floor before the repetition check applies.
+ pub repetition_min_tokens: usize,
+}
+
+impl Default for EntropyFilterConfig {
+ fn default() -> Self {
+ // Off, with conservative thresholds ready for when it is turned on.
+ Self {
+ enabled: false,
+ min_chars: 16,
+ min_entropy_bits_per_char: 2.0,
+ max_repetition_ratio: 0.7,
+ repetition_min_tokens: 6,
+ }
+ }
+}
+
+impl EntropyFilterConfig {
+ /// Validate operator-supplied thresholds. Returns an error string naming the
+ /// offending field, mirroring how the sweep validates its own coefficients.
+ ///
+ /// # Errors
+ /// Returns `Err` when the entropy floor is negative or non-finite, or the
+ /// repetition ratio is outside `0.0..=1.0`.
+ pub fn validate(&self) -> Result<(), String> {
+ if !self.min_entropy_bits_per_char.is_finite() || self.min_entropy_bits_per_char < 0.0 {
+ return Err(
+ "entropy_filter.min_entropy_bits_per_char must be a finite number \
+ greater than or equal to zero"
+ .to_string(),
+ );
+ }
+ if !self.max_repetition_ratio.is_finite()
+ || !(0.0..=1.0).contains(&self.max_repetition_ratio)
+ {
+ return Err(
+ "entropy_filter.max_repetition_ratio must be between 0.0 and 1.0".to_string(),
+ );
+ }
+ Ok(())
+ }
+}
+
+/// Classify one piece of text against the gate.
+///
+/// Pure and deterministic. With `cfg.enabled = false` (the default) it always
+/// returns [`FilterVerdict::Keep`], so the pass behaves exactly as it did before
+/// the filter existed.
+#[must_use]
+pub fn classify(text: &str, cfg: &EntropyFilterConfig) -> FilterVerdict {
+ if !cfg.enabled {
+ return FilterVerdict::Keep;
+ }
+ // Near-empty: judge by non-whitespace characters, so a body that is mostly
+ // blank lines is treated as empty rather than long.
+ let non_ws: usize = text.chars().filter(|c| !c.is_whitespace()).count();
+ if non_ws < cfg.min_chars {
+ return FilterVerdict::Skip(SkipReason::NearEmpty);
+ }
+ // Low entropy: a single repeated character (`aaaa…`) or a tiny alphabet.
+ if shannon_bits_per_char(text) < cfg.min_entropy_bits_per_char {
+ return FilterVerdict::Skip(SkipReason::LowEntropy);
+ }
+ // Repetitive: the same few tokens over and over. Only meaningful once there
+ // are enough tokens to distinguish repetition from a naturally short note.
+ let tokens: Vec<&str> = text.split_whitespace().collect();
+ if tokens.len() >= cfg.repetition_min_tokens {
+ #[allow(clippy::cast_precision_loss)]
+ let distinct = tokens
+ .iter()
+ .map(|t| t.to_ascii_lowercase())
+ .collect::<std::collections::HashSet<_>>()
+ .len() as f64;
+ #[allow(clippy::cast_precision_loss)]
+ let total = tokens.len() as f64;
+ let repetition = 1.0 - distinct / total;
+ if repetition > cfg.max_repetition_ratio {
+ return FilterVerdict::Skip(SkipReason::Repetitive);
+ }
+ }
+ FilterVerdict::Keep
+}
+
+/// Shannon entropy of the text in bits per character, over its character
+/// distribution. `0.0` for empty text and for a single repeated character.
+#[must_use]
+fn shannon_bits_per_char(text: &str) -> f64 {
+ let mut counts: HashMap<char, u64> = HashMap::new();
+ let mut total: u64 = 0;
+ for c in text.chars() {
+ *counts.entry(c).or_insert(0) += 1;
+ total += 1;
+ }
+ if total == 0 {
+ return 0.0;
+ }
+ #[allow(clippy::cast_precision_loss)]
+ let total_f = total as f64;
+ let mut bits = 0.0;
+ for &count in counts.values() {
+ #[allow(clippy::cast_precision_loss)]
+ let p = count as f64 / total_f;
+ bits -= p * p.log2();
+ }
+ bits
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn on() -> EntropyFilterConfig {
+ EntropyFilterConfig {
+ enabled: true,
+ ..EntropyFilterConfig::default()
+ }
+ }
+
+ #[test]
+ fn disabled_keeps_everything() {
+ let cfg = EntropyFilterConfig::default();
+ assert!(!cfg.enabled, "off by default");
+ assert_eq!(classify("", &cfg), FilterVerdict::Keep);
+ assert_eq!(
+ classify("aaaaaaaaaaaaaaaaaaaaaaaa", &cfg),
+ FilterVerdict::Keep
+ );
+ assert_eq!(
+ classify("yes ".repeat(20).as_str(), &cfg),
+ FilterVerdict::Keep
+ );
+ }
+
+ #[test]
+ fn empty_and_whitespace_are_near_empty() {
+ assert_eq!(
+ classify("", &on()),
+ FilterVerdict::Skip(SkipReason::NearEmpty)
+ );
+ assert_eq!(
+ classify(" \n\t \n ", &on()),
+ FilterVerdict::Skip(SkipReason::NearEmpty)
+ );
+ assert_eq!(
+ classify("ok", &on()),
+ FilterVerdict::Skip(SkipReason::NearEmpty),
+ "too short to carry information"
+ );
+ }
+
+ #[test]
+ fn single_repeated_character_is_low_entropy() {
+ // Long enough to pass the near-empty floor, but zero information.
+ assert_eq!(
+ classify("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", &on()),
+ FilterVerdict::Skip(SkipReason::LowEntropy)
+ );
+ }
+
+ #[test]
+ fn repeated_token_blob_is_repetitive() {
+ // Distinct chars keep entropy up, but one token repeats — boilerplate.
+ let verdict = classify("status ok status ok status ok status ok status ok", &on());
+ assert_eq!(
+ verdict,
+ FilterVerdict::Skip(SkipReason::Repetitive),
+ "{verdict:?}"
+ );
+ }
+
+ #[test]
+ fn terse_but_informative_fix_note_is_kept() {
+ // The tuning target: a short fix note with a file path and an error
+ // code must survive the gate.
+ let note = "Fixed crates/ai-memory-store/src/ops.rs which raised E0433 on build";
+ assert_eq!(classify(note, &on()), FilterVerdict::Keep, "{note}");
+ }
+
+ #[test]
+ fn a_normal_prose_note_is_kept() {
+ let note = "Set AI_MEMORY_AUTH_TOKEN before serving on a non-loopback bind, \
+ and configure the allowed hosts to guard against DNS rebinding.";
+ assert_eq!(classify(note, &on()), FilterVerdict::Keep, "{note}");
+ }
+
+ #[test]
+ fn entropy_zero_for_empty_and_uniform() {
+ assert_eq!(shannon_bits_per_char(""), 0.0);
+ assert_eq!(shannon_bits_per_char("aaaa"), 0.0);
+ assert!(
+ shannon_bits_per_char("abcd") > 1.9,
+ "4 equally likely chars = 2 bits"
+ );
+ }
+
+ #[test]
+ fn validate_rejects_bad_thresholds() {
+ let mut cfg = on();
+ cfg.min_entropy_bits_per_char = -1.0;
+ assert!(cfg.validate().is_err());
+ let mut cfg = on();
+ cfg.max_repetition_ratio = 1.5;
+ assert!(cfg.validate().is_err());
+ assert!(on().validate().is_ok());
+ }
+}
diff --git a/crates/ai-memory-consolidate/src/experience.rs b/crates/ai-memory-consolidate/src/experience.rs
index d901e628..2c002c2a 100644
--- a/crates/ai-memory-consolidate/src/experience.rs
+++ b/crates/ai-memory-consolidate/src/experience.rs
@@ -36,6 +36,12 @@ pub struct ExperienceConfig {
pub min_new_sessions: u64,
/// Per-session-page character cap in the prompt.
pub max_session_page_chars: usize,
+ /// A4 entropy / boilerplate pre-filter (docs/design-memory-aging.md §A4).
+ /// Off by default: a low-information session page is only skipped from the
+ /// prompt when an operator turns it on, so consolidation output is otherwise
+ /// unchanged. Advisory (invariant #16): a skipped page is not consolidated,
+ /// never deleted.
+ pub entropy_filter: crate::entropy_filter::EntropyFilterConfig,
}
impl Default for ExperienceConfig {
@@ -44,6 +50,7 @@ impl Default for ExperienceConfig {
sessions: 10,
min_new_sessions: 5,
max_session_page_chars: 6_000,
+ entropy_filter: crate::entropy_filter::EntropyFilterConfig::default(),
}
}
}
@@ -91,6 +98,28 @@ pub async fn run_experience_review(
}
}
+ // A4 entropy / boilerplate pre-filter: drop low-information session pages
+ // BEFORE they reach the consolidation prompt (and thus the eval gate and
+ // `apply_batch`). Skipped pages are simply not consolidated — never deleted
+ // (invariant #16). Off by default, so this changes nothing unless enabled.
+ let mut entropy_skipped = 0usize;
+ if experience.entropy_filter.enabled {
+ session_pages.retain(|(_, path, page)| {
+ match crate::entropy_filter::classify(&page.body, &experience.entropy_filter) {
+ crate::entropy_filter::FilterVerdict::Keep => true,
+ crate::entropy_filter::FilterVerdict::Skip(reason) => {
+ entropy_skipped += 1;
+ tracing::debug!(
+ %path,
+ reason = reason.as_str(),
+ "experience pass: skipping low-information session page from consolidation"
+ );
+ false
+ }
+ }
+ });
+ }
+
let label = format!("experience:{}-sessions", session_pages.len());
if (session_pages.len() as u64) < experience.min_new_sessions {
return Ok(AutoImproveReport {
@@ -134,6 +163,11 @@ pub async fn run_experience_review(
let rejection_context = load_rejection_context(reader, workspace_id, project_id, &cfg).await?;
let mut warnings = Vec::new();
+ if entropy_skipped > 0 {
+ warnings.push(format!(
+ "entropy filter skipped {entropy_skipped} low-information session page(s) from consolidation"
+ ));
+ }
let mut prompt = String::new();
prompt.push_str(
"You are reviewing MULTIPLE session summaries of one project side \
diff --git a/crates/ai-memory-consolidate/src/lib.rs b/crates/ai-memory-consolidate/src/lib.rs
index 872612ed..ac835d23 100644
--- a/crates/ai-memory-consolidate/src/lib.rs
+++ b/crates/ai-memory-consolidate/src/lib.rs
@@ -12,10 +12,12 @@ pub mod auto_improve;
pub mod auto_improve_schedule;
pub mod auto_improve_telemetry;
pub mod bootstrap;
+pub mod cold_cluster;
pub mod compaction;
pub mod consolidator;
pub mod curator;
pub mod embed;
+pub mod entropy_filter;
pub mod experience;
pub mod keep_tokens;
pub mod lint;
@@ -54,6 +56,7 @@ pub use bootstrap::{
derive_project_name, discover_main_repo_root, discover_repo_root, effective_chunk_budget,
plan_bootstrap_chunks, prune_sources_to_budget,
};
+pub use cold_cluster::{adaptive_eps, cosine_distance, dbscan};
pub use compaction::build_compacted_markdown;
pub use consolidator::{
BATCH_SYSTEM_PROMPT, Consolidator, ConsolidatorError, ConsolidatorResult,
@@ -67,12 +70,14 @@ pub use curator::{
pub use embed::{
EmbedBackfillCounts, EmbedBackfillError, EmbedBackfillOptions, run_embedding_backfill,
};
+pub use entropy_filter::{EntropyFilterConfig, FilterVerdict, SkipReason, classify};
pub use experience::{EXPERIENCE_SYSTEM_PROMPT, ExperienceConfig, run_experience_review};
pub use keep_tokens::mine_keep_tokens;
pub use lint::{LintError, LintFinding, LintOptions, LintReport, run_lint, stale_days_for};
pub use sweep::{
- CompactedPage, DEFAULT_OBSERVATION_PRUNE_BATCH, EvictedPage, ObservationRetention, SweepError,
- SweepReport, run_sweep, run_sweep_with_breadth, run_sweep_with_compaction,
+ ColdClusterDedup, CompactedPage, DEFAULT_OBSERVATION_PRUNE_BATCH, EmbeddingCoord, EvictedPage,
+ MergedCluster, ObservationRetention, SweepError, SweepReport, run_sweep,
+ run_sweep_with_breadth, run_sweep_with_compaction, run_sweep_with_hygiene,
run_sweep_with_options,
};
pub use types::{
diff --git a/crates/ai-memory-consolidate/src/sweep.rs b/crates/ai-memory-consolidate/src/sweep.rs
index e4313899..dd3de455 100644
--- a/crates/ai-memory-consolidate/src/sweep.rs
+++ b/crates/ai-memory-consolidate/src/sweep.rs
@@ -29,7 +29,9 @@
use std::collections::{HashMap, HashSet};
-use ai_memory_core::{ActorContext, PageId, PagePath, ProjectId, Tier, WorkspaceId};
+use ai_memory_core::{
+ ActorContext, PageEvidence, PageEvidenceKind, PageId, PagePath, ProjectId, Tier, WorkspaceId,
+};
use ai_memory_store::{
DecayCandidate, DecayParams, ReaderPool, WriterHandle, retention_score_with_breadth,
};
@@ -38,6 +40,7 @@ use jiff::Timestamp;
use serde::Serialize;
use thiserror::Error;
+use crate::cold_cluster::{adaptive_eps, dbscan};
use crate::compaction::build_compacted_markdown;
/// One evicted page surfaced in the [`SweepReport`].
@@ -82,6 +85,93 @@ pub struct CompactedPage {
pub new_id: Option<PageId>,
}
+/// One cold-cluster collapse surfaced in the [`SweepReport`] (A3,
+/// docs/design-memory-aging.md). A cluster of near-duplicate cold episodic pages
+/// is collapsed to one survivor; the other members are superseded with a merge
+/// note pointing at the survivor and stay reachable (invariant #16, never a hard
+/// delete).
+#[derive(Debug, Clone, Serialize)]
+pub struct MergedCluster {
+ /// Path of the survivor (the highest-retention cluster member).
+ pub survivor: String,
+ /// Pre-merge identifier of the survivor version.
+ pub survivor_id: PageId,
+ /// Paths of the members merged away into the survivor.
+ pub merged: Vec<String>,
+ /// The eps (cosine distance) the adaptive k-distance heuristic chose for
+ /// this run — surfaced so a conservative-vs-eager run is observable.
+ pub eps: f64,
+ /// `true` when the collapse landed through the wiki layer. Always `false` on
+ /// `dry_run`; a failed batch is retried on the next sweep.
+ pub applied: bool,
+ /// Identifier of the new (merged) survivor version once the collapse lands.
+ /// The pre-merge survivor and every merged-away member stay reachable via
+ /// the supersession chain and git history, so the collapse is reversible.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub new_survivor_id: Option<PageId>,
+}
+
+/// The `(provider, model, dim)` triple identifying which stored embeddings the
+/// A3 cold-cluster dedup should load. Must match the triple the pages were
+/// embedded under; a mismatch (or no embeddings) makes A3 a clean no-op.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct EmbeddingCoord {
+ /// Embedding provider name (e.g. `synthetic`, `openai`).
+ pub provider: String,
+ /// Embedding model id.
+ pub model: String,
+ /// Vector dimensionality.
+ pub dim: u32,
+}
+
+/// Opt-in configuration for the A3 cold-cluster dedup pass.
+///
+/// Off by default (`enabled = false` and no [`EmbeddingCoord`]): the sweep never
+/// clusters or merges, so behaviour is byte-identical to before A3. The pass is
+/// also a clean no-op — never an error — when no embeddings exist for the given
+/// triple or no embedder is configured (invariant #13, zero generative LLM: A3
+/// only reads already-stored vectors).
+#[derive(Debug, Clone, Default)]
+pub struct ColdClusterDedup {
+ /// Master switch. `false` (the default) disables clustering entirely.
+ pub enabled: bool,
+ /// Which stored embeddings to cluster over. `None` ⇒ no-op (nothing to
+ /// load), so a store with no embedder configured never dedups.
+ pub embedding: Option<EmbeddingCoord>,
+ /// DBSCAN density floor. `0` falls back to the conservative default of 2 (a
+ /// pair of near-duplicates is the smallest thing worth collapsing).
+ pub min_pts: usize,
+ /// Conservative ceiling on the adaptive eps (cosine distance). The adaptive
+ /// k-distance elbow is clamped to this, so A3 errs toward NOT merging.
+ /// `0.0` falls back to [`DEFAULT_DEDUP_MAX_EPS`].
+ pub max_eps: f32,
+}
+
+/// Default DBSCAN density floor for A3 when unset.
+pub const DEFAULT_DEDUP_MIN_PTS: usize = 2;
+/// Default conservative eps ceiling (cosine distance ≈ 0.15 ⇒ cosine similarity
+/// ≥ 0.85) for A3 when unset. Deliberately tight: over-eager clustering is the
+/// documented A3 failure mode.
+pub const DEFAULT_DEDUP_MAX_EPS: f32 = 0.15;
+
+impl ColdClusterDedup {
+ fn effective_min_pts(&self) -> usize {
+ if self.min_pts == 0 {
+ DEFAULT_DEDUP_MIN_PTS
+ } else {
+ self.min_pts
+ }
+ }
+
+ fn effective_max_eps(&self) -> f32 {
+ if self.max_eps > 0.0 && self.max_eps.is_finite() {
+ self.max_eps
+ } else {
+ DEFAULT_DEDUP_MAX_EPS
+ }
+ }
+}
+
/// One TTL-expired page surfaced in the [`SweepReport`].
#[derive(Debug, Clone, Serialize)]
pub struct ExpiredPage {
@@ -113,6 +203,11 @@ pub struct SweepReport {
/// sweep behaves exactly as before — while the flag is off, which is the
/// default. Reported in both modes so every run is observable.
pub compacted: Vec<CompactedPage>,
+ /// Near-duplicate cold-episodic clusters collapsed to one survivor (A3),
+ /// when `dedup.enabled` is set. Empty — and the sweep behaves exactly as
+ /// before — while the flag is off, which is the default. Reported in both
+ /// modes so every run is observable.
+ pub merged: Vec<MergedCluster>,
/// Pages past their frontmatter `expires_at:` TTL (hard-deleted
/// through the wiki layer unless `dry_run`).
pub expired: Vec<ExpiredPage>,
@@ -317,6 +412,54 @@ pub async fn run_sweep_with_compaction(
retention: ObservationRetention,
compact_cold_episodic: bool,
dry_run: bool,
+) -> Result<SweepReport, SweepError> {
+ run_sweep_with_hygiene(
+ reader,
+ writer,
+ wiki,
+ workspace_id,
+ project_id,
+ params,
+ breadth_weight,
+ retention,
+ compact_cold_episodic,
+ ColdClusterDedup::default(),
+ dry_run,
+ )
+ .await
+}
+
+/// Run a sweep with the breadth coefficient, the observation prune, the opt-in
+/// A2 extractive tier-down, and the opt-in A3 cold-cluster dedup.
+///
+/// A3 (`dedup.enabled`) clusters the *bounded cold-episodic set this sweep
+/// already materialised* by embedding (cosine DBSCAN, adaptive k-distance eps)
+/// and collapses each near-duplicate cluster to one survivor — the
+/// highest-retention member, its body the extractive union of the cluster's
+/// keep-tokens — superseding the other members with a merge note that points at
+/// the survivor. Nothing is ever hard-deleted: every merged-away member stays
+/// reachable through the supersession chain and git (invariant #16), and the
+/// merge provenance is recorded in `page_evidence`. With `dedup` disabled (the
+/// default) no clustering happens and this is byte-identical to
+/// [`run_sweep_with_compaction`]. With no embeddings for the configured triple —
+/// or no embedder configured at all — A3 is a clean no-op (invariant #13: it
+/// reads only already-stored vectors, never a provider).
+///
+/// # Errors
+/// Same as [`run_sweep_with_options`].
+#[allow(clippy::too_many_arguments)]
+pub async fn run_sweep_with_hygiene(
+ reader: &ReaderPool,
+ writer: &WriterHandle,
+ wiki: Option<&Wiki>,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ params: &DecayParams,
+ breadth_weight: f64,
+ retention: ObservationRetention,
+ compact_cold_episodic: bool,
+ dedup: ColdClusterDedup,
+ dry_run: bool,
) -> Result<SweepReport, SweepError> {
if !breadth_weight.is_finite() || breadth_weight < 0.0 {
return Err(SweepError::InvalidBreadthWeight);
@@ -333,6 +476,11 @@ pub async fn run_sweep_with_compaction(
let mut compacted: Vec<CompactedPage> = Vec::new();
let mut expired: Vec<ExpiredPage> = Vec::new();
+ // First pass: TTL and the cold set. Collecting the cold set once lets the A3
+ // dedup pass cluster the *same* bounded set the evict/compact pass would
+ // otherwise process (invariant #2 — one materialisation, not two chances to
+ // drift), and lets the remainder loop skip whatever dedup already claimed.
+ let mut cold: Vec<ColdEntry> = Vec::new();
for c in &candidates {
if let Some(expires_us) = c.expires_at_us
&& expires_us <= now_us
@@ -363,34 +511,59 @@ pub async fn run_sweep_with_compaction(
breadth_weight,
);
if score < params.cold_threshold {
- if compact_cold_episodic {
- // A2 tier-down replaces eviction for cold episodic pages. An
- // already-compacted page (its V65 marker set) is terminal — the
- // durable residue is what we chose to keep, so re-evicting it
- // would destroy exactly what tier-down preserved, and
- // re-compacting it is a no-op loop. Skip it entirely.
- if c.compacted_at_us.is_some() {
- continue;
- }
- compacted.push(CompactedPage {
- id: c.id,
- path: c.path.as_str().to_string(),
- retention: score,
- age_days,
- access_count: c.access_count,
- compacted: false,
- new_id: None,
- });
- } else {
- evicted.push(EvictedPage {
- id: c.id,
- path: c.path.as_str().to_string(),
- retention: score,
- age_days,
- access_count: c.access_count,
- deleted: false,
- });
+ cold.push(ColdEntry {
+ id: c.id,
+ path: c.path.clone(),
+ retention: score,
+ age_days,
+ access_count: c.access_count,
+ compacted_at_us: c.compacted_at_us,
+ });
+ }
+ }
+
+ // A3 cold-cluster dedup plan (read-only). Populates `merged` and the set of
+ // page ids the collapse claims, so the evict/compact remainder skips them.
+ // A clean no-op (empty plan) when the flag is off, no embeddings exist for
+ // the configured triple, or no wiki handle is available.
+ let plan =
+ plan_cold_cluster_dedup(reader, wiki, workspace_id, project_id, &dedup, &cold).await?;
+ let mut merged = plan.report;
+ let mut dedup_requests = plan.requests;
+ let dedup_survivor_req_index = plan.survivor_req_index;
+ let deduped = plan.claimed;
+
+ for entry in &cold {
+ if deduped.contains(&entry.id) {
+ continue;
+ }
+ if compact_cold_episodic {
+ // A2 tier-down replaces eviction for cold episodic pages. An
+ // already-compacted page (its V65 marker set) is terminal — the
+ // durable residue is what we chose to keep, so re-evicting it
+ // would destroy exactly what tier-down preserved, and
+ // re-compacting it is a no-op loop. Skip it entirely.
+ if entry.compacted_at_us.is_some() {
+ continue;
}
+ compacted.push(CompactedPage {
+ id: entry.id,
+ path: entry.path.as_str().to_string(),
+ retention: entry.retention,
+ age_days: entry.age_days,
+ access_count: entry.access_count,
+ compacted: false,
+ new_id: None,
+ });
+ } else {
+ evicted.push(EvictedPage {
+ id: entry.id,
+ path: entry.path.as_str().to_string(),
+ retention: entry.retention,
+ age_days: entry.age_days,
+ access_count: entry.access_count,
+ deleted: false,
+ });
}
}
@@ -408,6 +581,36 @@ pub async fn run_sweep_with_compaction(
.await?;
}
if !dry_run {
+ // A3 cold-cluster collapse (invariant #2: one batched write for every
+ // survivor rewrite and merge-note supersession). The survivor keeps the
+ // union of the cluster's keep-tokens and cites its merged-away members
+ // in `page_evidence`; every member stays reachable via supersession +
+ // git (invariant #16). Freshly rewritten, none of these pages is cold
+ // next sweep, so the collapse does not immediately re-trigger.
+ if !dedup_requests.is_empty() {
+ match wiki {
+ Some(wiki) => {
+ let requests = std::mem::take(&mut dedup_requests);
+ match wiki.apply_batch(requests).await {
+ Ok(new_ids) => {
+ for (cluster, &req_idx) in
+ merged.iter_mut().zip(dedup_survivor_req_index.iter())
+ {
+ cluster.applied = true;
+ cluster.new_survivor_id = new_ids.get(req_idx).copied();
+ }
+ }
+ Err(error) => tracing::warn!(
+ %error,
+ "forget sweep: cold-cluster dedup batch failed; retrying on next sweep"
+ ),
+ }
+ }
+ None => tracing::warn!(
+ "forget sweep: wiki unavailable; refusing store-only cold-cluster dedup"
+ ),
+ }
+ }
for page in &mut expired {
let path = match ai_memory_core::PagePath::new(page.path.clone()) {
Ok(p) => p,
@@ -598,6 +801,7 @@ pub async fn run_sweep_with_compaction(
candidates_evaluated: candidates.len(),
evicted,
compacted,
+ merged,
expired,
hard_deleted,
observations_prunable,
@@ -607,6 +811,280 @@ pub async fn run_sweep_with_compaction(
})
}
+/// A cold page (scored below `cold_threshold`) captured once so the A3 dedup
+/// pass and the evict/compact remainder read the same materialised set.
+struct ColdEntry {
+ id: PageId,
+ path: PagePath,
+ retention: f64,
+ age_days: f64,
+ access_count: u32,
+ compacted_at_us: Option<i64>,
+}
+
+/// The read-only output of the A3 dedup planner: what to report, which page ids
+/// the collapse claims, and the batched wiki writes to apply.
+struct DedupPlan {
+ report: Vec<MergedCluster>,
+ claimed: HashSet<PageId>,
+ requests: Vec<WritePageRequest>,
+ /// For each entry in `report`, the index into `requests` of that cluster's
+ /// survivor write, so the applied `new_survivor_id` can be mapped back.
+ survivor_req_index: Vec<usize>,
+}
+
+impl DedupPlan {
+ fn empty() -> Self {
+ Self {
+ report: Vec::new(),
+ claimed: HashSet::new(),
+ requests: Vec::new(),
+ survivor_req_index: Vec::new(),
+ }
+ }
+}
+
+/// Footer stamped on a merge-note stub so a reader sees the page was collapsed
+/// and where its content now lives.
+const MERGE_STUB_FOOTER: &str = "_Merged into the survivor below by A3 cold-cluster dedup. This page's full \
+ pre-merge body is retained in git history and the supersession chain, and \
+ can be recovered with `restore-page`._";
+
+/// Build the A3 cold-cluster dedup plan over the already-materialised cold set.
+///
+/// Pure read path: it loads stored embeddings, clusters them, and prepares the
+/// wiki writes, but applies nothing. A clean no-op (empty plan) when disabled,
+/// when no `EmbeddingCoord`/wiki is available, or when no embeddings exist for
+/// the cold set (invariant #13 — never a provider call).
+async fn plan_cold_cluster_dedup(
+ reader: &ReaderPool,
+ wiki: Option<&Wiki>,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ dedup: &ColdClusterDedup,
+ cold: &[ColdEntry],
+) -> Result<DedupPlan, SweepError> {
+ if !dedup.enabled {
+ return Ok(DedupPlan::empty());
+ }
+ let Some(coord) = dedup.embedding.clone() else {
+ return Ok(DedupPlan::empty());
+ };
+ // Dedup mutates through the wiki (survivor rewrite + supersessions); without
+ // a handle there is nothing to plan, exactly like the compaction pass.
+ let Some(wiki) = wiki else {
+ return Ok(DedupPlan::empty());
+ };
+ // Only pages that are cold, not already a compacted/merged residue (their
+ // marker set), are eligible — a terminal residue is never re-merged.
+ let mut eligible: HashMap<PageId, &ColdEntry> = HashMap::new();
+ for entry in cold {
+ if entry.compacted_at_us.is_none() {
+ eligible.insert(entry.id, entry);
+ }
+ }
+ if eligible.len() < dedup.effective_min_pts() {
+ return Ok(DedupPlan::empty());
+ }
+
+ let embeddings = reader
+ .load_embeddings(
+ workspace_id,
+ project_id,
+ coord.provider,
+ coord.model,
+ coord.dim,
+ )
+ .await?;
+ // Intersect the stored embeddings with the eligible cold set.
+ let mut ids: Vec<PageId> = Vec::new();
+ let mut vectors: Vec<Vec<f32>> = Vec::new();
+ for e in &embeddings {
+ if eligible.contains_key(&e.id) {
+ ids.push(e.id);
+ vectors.push(e.vector.clone());
+ }
+ }
+ if vectors.len() < dedup.effective_min_pts() {
+ return Ok(DedupPlan::empty());
+ }
+
+ let min_pts = dedup.effective_min_pts();
+ let Some(eps) = adaptive_eps(&vectors, min_pts, dedup.effective_max_eps()) else {
+ return Ok(DedupPlan::empty());
+ };
+ let clusters = dbscan(&vectors, eps, min_pts);
+ if clusters.is_empty() {
+ return Ok(DedupPlan::empty());
+ }
+
+ let mut plan = DedupPlan::empty();
+ for cluster in clusters {
+ if cluster.len() < 2 {
+ continue;
+ }
+ // Survivor = highest-retention member (ties broken by first index, for
+ // determinism). Read every member's body to mine the union of keep-tokens.
+ let mut member_entries: Vec<&ColdEntry> = Vec::new();
+ for &idx in &cluster {
+ match eligible.get(&ids[idx]) {
+ Some(entry) => member_entries.push(entry),
+ None => continue,
+ }
+ }
+ if member_entries.len() < 2 {
+ continue;
+ }
+ let survivor_pos = member_entries
+ .iter()
+ .enumerate()
+ .max_by(|(_, a), (_, b)| {
+ a.retention
+ .partial_cmp(&b.retention)
+ .unwrap_or(std::cmp::Ordering::Equal)
+ })
+ .map(|(i, _)| i)
+ .unwrap_or(0);
+ let survivor = member_entries[survivor_pos];
+
+ // Read the survivor's frontmatter and concatenate every member's body so
+ // the extractive miner keeps the UNION of the cluster's keep-tokens.
+ let survivor_md = match wiki.read_page(workspace_id, project_id, &survivor.path) {
+ Ok(md) => md,
+ Err(error) => {
+ tracing::warn!(path = %survivor.path.as_str(), %error, "forget sweep: could not read survivor for dedup; skipping cluster");
+ continue;
+ }
+ };
+ let mut union_body = survivor_md.body.clone();
+ let mut loser_paths: Vec<String> = Vec::new();
+ let mut loser_requests: Vec<WritePageRequest> = Vec::new();
+ let mut evidence: Vec<PageEvidence> = Vec::new();
+ let mut read_failed = false;
+ for (pos, member) in member_entries.iter().enumerate() {
+ if pos == survivor_pos {
+ continue;
+ }
+ let md = match wiki.read_page(workspace_id, project_id, &member.path) {
+ Ok(md) => md,
+ Err(error) => {
+ tracing::warn!(path = %member.path.as_str(), %error, "forget sweep: could not read cluster member for dedup; skipping cluster");
+ read_failed = true;
+ break;
+ }
+ };
+ union_body.push_str("\n\n");
+ union_body.push_str(&md.body);
+ loser_paths.push(member.path.as_str().to_string());
+ // Merge provenance: the closed `page_evidence` vocabulary has no
+ // dedicated A3 kind (adding one needs a CHECK-constraint migration,
+ // which A3 deliberately avoids). `reconsolidation` is the accurate
+ // existing kind — an A3 collapse rewrites the survivor from several
+ // sources — and the `a3_merge:` source-id prefix keeps the merge
+ // distinguishable and unique.
+ evidence.push(PageEvidence {
+ kind: PageEvidenceKind::Reconsolidation,
+ source_id: format!("a3_merge:{}", member.id),
+ });
+ // Supersede the loser with a merge-note stub pointing at the
+ // survivor. Marked `compacted: true` so it is terminal for the decay
+ // pass and never re-clustered; its full body stays in the chain.
+ let stub_body = format!(
+ "This page was merged into [[{}]] by A3 cold-cluster dedup.\n\n{}\n",
+ survivor.path.as_str(),
+ MERGE_STUB_FOOTER
+ );
+ let mut stub_fm = frontmatter_object(&md.frontmatter);
+ stub_fm.insert("compacted".to_string(), serde_json::Value::Bool(true));
+ stub_fm.insert(
+ "merged_into".to_string(),
+ serde_json::Value::String(survivor.path.as_str().to_string()),
+ );
+ loser_requests.push(WritePageRequest {
+ workspace_id,
+ project_id,
+ path: member.path.clone(),
+ frontmatter: serde_json::Value::Object(stub_fm),
+ body: stub_body,
+ tier: Tier::Episodic,
+ pinned: false,
+ title: None,
+ admission_ctx: None,
+ author_id: None,
+ actor: ActorContext::anonymous(),
+ evidence: Vec::new(),
+ });
+ }
+ if read_failed || loser_paths.is_empty() {
+ continue;
+ }
+
+ // Survivor body: extractive union of keep-tokens + a merge note. Reuses
+ // A2's `build_compacted_markdown`, so the survivor keeps every member's
+ // durable facts; the prose is dropped (zero-LLM, invariant #13).
+ let (mut survivor_fm, mut survivor_body) =
+ build_compacted_markdown(&survivor_md.frontmatter, &union_body);
+ survivor_body.push_str(&format!(
+ "\n\n_Merged {} near-duplicate cold page(s): {} (A3 cold-cluster dedup)._\n",
+ loser_paths.len(),
+ loser_paths.join(", ")
+ ));
+ if let serde_json::Value::Object(map) = &mut survivor_fm {
+ map.insert(
+ "merged_from".to_string(),
+ serde_json::Value::Array(
+ loser_paths
+ .iter()
+ .map(|p| serde_json::Value::String(p.clone()))
+ .collect(),
+ ),
+ );
+ }
+ let survivor_request = WritePageRequest {
+ workspace_id,
+ project_id,
+ path: survivor.path.clone(),
+ frontmatter: survivor_fm,
+ body: survivor_body,
+ tier: Tier::Episodic,
+ pinned: false,
+ title: None,
+ admission_ctx: None,
+ author_id: None,
+ actor: ActorContext::anonymous(),
+ evidence,
+ };
+
+ // Record. Survivor write goes first in this cluster's slice, so its
+ // index is the current request length.
+ let survivor_req_index = plan.requests.len();
+ plan.requests.push(survivor_request);
+ plan.requests.extend(loser_requests);
+ plan.claimed.insert(survivor.id);
+ for member in &member_entries {
+ plan.claimed.insert(member.id);
+ }
+ plan.report.push(MergedCluster {
+ survivor: survivor.path.as_str().to_string(),
+ survivor_id: survivor.id,
+ merged: loser_paths,
+ eps: f64::from(eps),
+ applied: false,
+ new_survivor_id: None,
+ });
+ plan.survivor_req_index.push(survivor_req_index);
+ }
+ Ok(plan)
+}
+
+/// Coerce a frontmatter value into an object map, discarding a non-object shape.
+fn frontmatter_object(fm: &serde_json::Value) -> serde_json::Map<String, serde_json::Value> {
+ match fm {
+ serde_json::Value::Object(m) => m.clone(),
+ _ => serde_json::Map::new(),
+ }
+}
+
/// Distinct-actor counts keyed by page, for feeding
/// [`retention_score_with_breadth`].
///
diff --git a/crates/ai-memory-consolidate/tests/suite/cold_cluster_sweep.rs b/crates/ai-memory-consolidate/tests/suite/cold_cluster_sweep.rs
new file mode 100644
index 00000000..f0fd9029
--- /dev/null
+++ b/crates/ai-memory-consolidate/tests/suite/cold_cluster_sweep.rs
@@ -0,0 +1,326 @@
+//! A3 cold-cluster dedup, end to end through a real Store + Wiki + embeddings.
+//!
+//! Invariants guarded (docs/design-memory-aging.md §A3):
+//!
+//! 1. **Flag OFF = today's behaviour.** No clustering, no merges.
+//! 2. **Flag ON = collapse.** Near-duplicate cold episodic pages collapse to one
+//! survivor; the other members are superseded with a merge note and stay
+//! reachable (invariant #16, never a hard delete).
+//! 3. **Survivor keeps the union of keep-tokens** — a fact that lived only in a
+//! merged-away duplicate survives in the survivor (recall preserved).
+//! 4. **Merge provenance** is recorded in `page_evidence`.
+//! 5. **No embeddings ⇒ clean no-op** (never an error, never a provider call).
+
+use std::sync::Arc;
+
+use ai_memory_consolidate::{
+ ColdClusterDedup, EmbeddingCoord, ObservationRetention, run_sweep_with_hygiene,
+};
+use ai_memory_core::{ActorContext, PageId, PagePath, ProjectId, Tier, WorkspaceId};
+use ai_memory_llm::{Embedder, SyntheticEmbedder};
+use ai_memory_store::{DecayParams, Store};
+use ai_memory_wiki::{Wiki, WritePageRequest};
+use rusqlite::params;
+use tempfile::TempDir;
+
+const DAY_US: i64 = 86_400_000_000;
+const COLD_DAYS: i64 = 400;
+const DIM: u32 = 64;
+
+struct Fixture {
+ tmp: TempDir,
+ store: Store,
+ wiki: Wiki,
+ ws: WorkspaceId,
+ proj: ProjectId,
+}
+
+/// Build a fixture. `with_embedder` controls whether pages get embedding rows —
+/// the no-embeddings no-op test wants a wiki without one.
+async fn seed_fixture(with_embedder: bool) -> Fixture {
+ let tmp = TempDir::new().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "scratch", None)
+ .await
+ .unwrap();
+ let mut wiki = Wiki::new(tmp.path(), store.writer.clone())
+ .unwrap()
+ .with_store_reader(store.reader.clone());
+ if with_embedder {
+ let embedder: Arc<dyn Embedder> = Arc::new(SyntheticEmbedder::new(DIM));
+ wiki = wiki.with_embedder(embedder);
+ }
+ Fixture {
+ tmp,
+ store,
+ wiki,
+ ws,
+ proj,
+ }
+}
+
+async fn write(fx: &Fixture, path: &str, body: &str) -> PageId {
+ fx.wiki
+ .write_page(WritePageRequest {
+ workspace_id: fx.ws,
+ project_id: fx.proj,
+ path: PagePath::new(path).unwrap(),
+ frontmatter: serde_json::json!({"title": path}),
+ body: body.to_string(),
+ tier: Tier::Episodic,
+ pinned: false,
+ title: Some(path.to_string()),
+ admission_ctx: None,
+ author_id: None,
+ actor: ActorContext::anonymous(),
+ evidence: Vec::new(),
+ })
+ .await
+ .unwrap()
+}
+
+fn backdate_latest(fx: &Fixture, path: &str, days: i64) {
+ let conn = rusqlite::Connection::open(fx.tmp.path().join("db/memory.sqlite")).unwrap();
+ conn.pragma_update(None, "busy_timeout", 5_000).unwrap();
+ let updated_at = jiff::Timestamp::now().as_microsecond() - days * DAY_US;
+ conn.execute(
+ "UPDATE pages SET created_at = ?1, updated_at = ?1 \
+ WHERE workspace_id = ?2 AND project_id = ?3 AND path = ?4 AND is_latest = 1",
+ params![updated_at, fx.ws.as_bytes(), fx.proj.as_bytes(), path],
+ )
+ .unwrap();
+}
+
+fn latest_body(fx: &Fixture, path: &str) -> String {
+ let conn = rusqlite::Connection::open(fx.tmp.path().join("db/memory.sqlite")).unwrap();
+ conn.query_row(
+ "SELECT body FROM pages \
+ WHERE workspace_id = ?1 AND project_id = ?2 AND path = ?3 AND is_latest = 1",
+ params![fx.ws.as_bytes(), fx.proj.as_bytes(), path],
+ |row| row.get(0),
+ )
+ .unwrap()
+}
+
+fn superseded_bodies(fx: &Fixture, path: &str) -> Vec<String> {
+ let conn = rusqlite::Connection::open(fx.tmp.path().join("db/memory.sqlite")).unwrap();
+ let mut stmt = conn
+ .prepare(
+ "SELECT body FROM pages \
+ WHERE workspace_id = ?1 AND project_id = ?2 AND path = ?3 AND is_latest = 0 \
+ ORDER BY created_at",
+ )
+ .unwrap();
+ let rows = stmt
+ .query_map(params![fx.ws.as_bytes(), fx.proj.as_bytes(), path], |row| {
+ row.get::<_, String>(0)
+ })
+ .unwrap();
+ rows.map(Result::unwrap).collect()
+}
+
+fn evidence_for(fx: &Fixture, page_id: PageId) -> Vec<(String, String)> {
+ let conn = rusqlite::Connection::open(fx.tmp.path().join("db/memory.sqlite")).unwrap();
+ let mut stmt = conn
+ .prepare("SELECT source_kind, source_id FROM page_evidence WHERE page_id = ?1")
+ .unwrap();
+ let rows = stmt
+ .query_map(params![page_id.as_bytes()], |row| {
+ Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))
+ })
+ .unwrap();
+ rows.map(Result::unwrap).collect()
+}
+
+fn dedup_on() -> ColdClusterDedup {
+ ColdClusterDedup {
+ enabled: true,
+ embedding: Some(EmbeddingCoord {
+ provider: "synthetic".into(),
+ model: "bag-of-words-v1".into(),
+ dim: DIM,
+ }),
+ min_pts: 0,
+ max_eps: 0.0,
+ }
+}
+
+// Two near-duplicate bodies: almost-identical prose (so the bag-of-words
+// embedder places them close together) but each carrying ONE unique durable
+// fact (a distinct file path), so the survivor's keep-token union is testable.
+const DUP_A: &str = "Investigated the flaky retention sweep test failure again and again during \
+ the long debugging session that afternoon. The unique culprit turned out to be in the file \
+ crates/ai-memory-store/src/ops.rs during the run.";
+const DUP_B: &str = "Investigated the flaky retention sweep test failure again and again during \
+ the long debugging session that afternoon. The unique culprit turned out to be in the file \
+ crates/ai-memory-store/src/reader.rs during the run.";
+const FAR: &str = "Kubernetes pod scheduling node selector affinity rules for the cluster \
+ autoscaler deployment on the staging namespace.";
+
+/// Flag OFF: nothing clusters or merges — byte-identical to before A3.
+#[tokio::test]
+async fn flag_off_never_merges() {
+ let fx = seed_fixture(true).await;
+ write(&fx, "sessions/a.md", DUP_A).await;
+ write(&fx, "sessions/b.md", DUP_B).await;
+ for p in ["sessions/a.md", "sessions/b.md"] {
+ backdate_latest(&fx, p, COLD_DAYS);
+ }
+
+ let report = run_sweep_with_hygiene(
+ &fx.store.reader,
+ &fx.store.writer,
+ Some(&fx.wiki),
+ fx.ws,
+ fx.proj,
+ &DecayParams::default(),
+ 0.0,
+ ObservationRetention::default(),
+ false,
+ ColdClusterDedup::default(), // OFF
+ false,
+ )
+ .await
+ .unwrap();
+
+ assert!(report.merged.is_empty(), "flag off must not merge");
+ // With dedup off and compaction off, the cold pages are evicted as always.
+ assert_eq!(report.evicted.len(), 2, "both cold pages evicted as before");
+}
+
+/// Flag ON: the two near-duplicates collapse to one survivor; the survivor keeps
+/// BOTH facts; the loser is superseded (reachable); the merge is recorded in
+/// `page_evidence`; the far page is untouched.
+#[tokio::test]
+async fn flag_on_collapses_cluster_and_preserves_recall() {
+ let fx = seed_fixture(true).await;
+ write(&fx, "sessions/a.md", DUP_A).await;
+ write(&fx, "sessions/b.md", DUP_B).await;
+ write(&fx, "sessions/far.md", FAR).await;
+ for p in ["sessions/a.md", "sessions/b.md", "sessions/far.md"] {
+ backdate_latest(&fx, p, COLD_DAYS);
+ }
+
+ let report = run_sweep_with_hygiene(
+ &fx.store.reader,
+ &fx.store.writer,
+ Some(&fx.wiki),
+ fx.ws,
+ fx.proj,
+ &DecayParams::default(),
+ 0.0,
+ ObservationRetention::default(),
+ false,
+ dedup_on(),
+ false,
+ )
+ .await
+ .unwrap();
+
+ assert_eq!(
+ report.merged.len(),
+ 1,
+ "the two near-duplicates form exactly one cluster: {:?}",
+ report.merged
+ );
+ let cluster = &report.merged[0];
+ assert!(cluster.applied, "the collapse landed");
+ assert_eq!(cluster.merged.len(), 1, "one member merged away");
+ let survivor_path = cluster.survivor.clone();
+ let loser_path = cluster.merged[0].clone();
+ assert!(
+ survivor_path != loser_path,
+ "survivor and loser are distinct"
+ );
+ assert!(
+ survivor_path.starts_with("sessions/") && loser_path.starts_with("sessions/"),
+ "far page is not part of the cluster"
+ );
+ assert!(
+ !report.merged.iter().any(|m| m.survivor == "sessions/far.md"
+ || m.merged.contains(&"sessions/far.md".to_string())),
+ "the far page must never be clustered"
+ );
+
+ // Recall preserved: the survivor body carries BOTH facts — its own and the
+ // one that lived only in the merged-away duplicate.
+ let survivor_body = latest_body(&fx, &survivor_path);
+ assert!(
+ survivor_body.contains("crates/ai-memory-store/src/ops.rs"),
+ "ops.rs fact present in survivor: {survivor_body}"
+ );
+ assert!(
+ survivor_body.contains("crates/ai-memory-store/src/reader.rs"),
+ "reader.rs fact (from the merged-away duplicate) survives in survivor: {survivor_body}"
+ );
+
+ // The loser's live version is a merge note pointing at the survivor…
+ let loser_body = latest_body(&fx, &loser_path);
+ assert!(
+ loser_body.contains("merged into") && loser_body.contains(&survivor_path),
+ "loser points at survivor: {loser_body}"
+ );
+ // …and its full pre-merge body stays reachable via supersession (reversible).
+ let loser_history = superseded_bodies(&fx, &loser_path);
+ assert!(
+ loser_history.iter().any(|b| b.contains("unique culprit")),
+ "the loser's pre-merge body stays reachable: {loser_history:?}"
+ );
+
+ // Merge provenance recorded on the new survivor version.
+ let new_survivor = cluster.new_survivor_id.expect("new survivor id");
+ let evidence = evidence_for(&fx, new_survivor);
+ assert!(
+ evidence
+ .iter()
+ .any(|(kind, src)| kind == "reconsolidation" && src.starts_with("a3_merge:")),
+ "page_evidence records the merge: {evidence:?}"
+ );
+
+ // The far page was neither merged nor (being cold) skipped: it evicts.
+ assert!(
+ report.evicted.iter().any(|e| e.path == "sessions/far.md"),
+ "the far page follows the normal cold path: {:?}",
+ report.evicted
+ );
+}
+
+/// No embeddings for the configured triple ⇒ A3 is a clean no-op, not an error.
+#[tokio::test]
+async fn no_embeddings_is_a_clean_no_op() {
+ let fx = seed_fixture(false).await; // no embedder → no embedding rows
+ write(&fx, "sessions/a.md", DUP_A).await;
+ write(&fx, "sessions/b.md", DUP_B).await;
+ for p in ["sessions/a.md", "sessions/b.md"] {
+ backdate_latest(&fx, p, COLD_DAYS);
+ }
+
+ let report = run_sweep_with_hygiene(
+ &fx.store.reader,
+ &fx.store.writer,
+ Some(&fx.wiki),
+ fx.ws,
+ fx.proj,
+ &DecayParams::default(),
+ 0.0,
+ ObservationRetention::default(),
+ false,
+ dedup_on(), // ON, but there are no vectors to cluster
+ false,
+ )
+ .await
+ .unwrap();
+
+ assert!(
+ report.merged.is_empty(),
+ "no embeddings ⇒ no merges, no error"
+ );
+ assert_eq!(report.evicted.len(), 2, "cold pages follow the normal path");
+}
diff --git a/crates/ai-memory-consolidate/tests/suite/entropy_experience.rs b/crates/ai-memory-consolidate/tests/suite/entropy_experience.rs
new file mode 100644
index 00000000..67766805
--- /dev/null
+++ b/crates/ai-memory-consolidate/tests/suite/entropy_experience.rs
@@ -0,0 +1,231 @@
+//! A4 entropy / boilerplate pre-filter, wired into the cross-session experience
+//! pass (docs/design-memory-aging.md §A4).
+//!
+//! The gate runs BEFORE consolidation ingest: a low-information session page is
+//! never placed in the consolidation prompt (and thus never reaches the eval
+//! gate or `apply_batch`), while a real one is. Default (OFF) changes nothing.
+//! The proof captures the prompt the LLM would receive and asserts which session
+//! bodies made it in — skipping is advisory (the page is not deleted, invariant
+//! #16).
+
+use std::future::Future;
+use std::pin::Pin;
+use std::sync::{Arc, Mutex};
+
+use ai_memory_consolidate::entropy_filter::EntropyFilterConfig;
+use ai_memory_consolidate::{AutoImproveReviewConfig, ExperienceConfig, run_experience_review};
+use ai_memory_core::{ActorContext, NewSession, SessionId};
+use ai_memory_llm::{ChatRequest, ChatResponse, LlmProvider, LlmResult};
+use ai_memory_store::Store;
+use ai_memory_wiki::{Wiki, WritePageRequest};
+use tempfile::TempDir;
+
+/// LLM stub that records every prompt it is asked to complete, so the test can
+/// assert which session bodies survived the entropy gate. Returns an empty (but
+/// valid) proposal set.
+struct CapturingLlm {
+ prompts: Arc<Mutex<Vec<String>>>,
+}
+
+impl LlmProvider for CapturingLlm {
+ fn name(&self) -> &'static str {
+ "fake"
+ }
+ fn model(&self) -> &str {
+ "capturing"
+ }
+ fn complete<'life0, 'async_trait>(
+ &'life0 self,
+ _request: ChatRequest,
+ ) -> Pin<Box<dyn Future<Output = LlmResult<ChatResponse>> + Send + 'async_trait>>
+ where
+ 'life0: 'async_trait,
+ Self: 'async_trait,
+ {
+ Box::pin(async move {
+ Ok(ChatResponse {
+ text: "unused".into(),
+ usage: None,
+ model: "capturing".into(),
+ })
+ })
+ }
+ fn complete_structured_raw<'life0, 'async_trait>(
+ &'life0 self,
+ request: ChatRequest,
+ _schema: serde_json::Value,
+ ) -> Pin<Box<dyn Future<Output = LlmResult<serde_json::Value>> + Send + 'async_trait>>
+ where
+ 'life0: 'async_trait,
+ Self: 'async_trait,
+ {
+ let prompts = self.prompts.clone();
+ Box::pin(async move {
+ if let Some(msg) = request.messages.first() {
+ prompts.lock().unwrap().push(msg.content.clone());
+ }
+ Ok(serde_json::json!({
+ "summary": "none",
+ "proposals": [],
+ "rejected_candidates": []
+ }))
+ })
+ }
+}
+
+// A repetitive boilerplate status blob — no durable facts.
+const BOILERPLATE: &str = "status ok status ok status ok status ok status ok status ok status ok";
+// A real, terse-but-informative note with a file path and an error code.
+const REAL: &str = "Fixed the retention sweep in crates/ai-memory-store/src/ops.rs which raised \
+ E0433; tag main then deploy the stack.";
+
+struct Harness {
+ _tmp: TempDir,
+ store: Store,
+ ws: ai_memory_core::WorkspaceId,
+ proj: ai_memory_core::ProjectId,
+ prompts: Arc<Mutex<Vec<String>>>,
+ llm: Arc<dyn LlmProvider>,
+}
+
+async fn harness_with_two_sessions() -> Harness {
+ let tmp = TempDir::new().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let wiki = Wiki::new(tmp.path(), store.writer.clone())
+ .unwrap()
+ .with_store_reader(store.reader.clone());
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "scratch", None)
+ .await
+ .unwrap();
+
+ for body in [BOILERPLATE, REAL] {
+ let session_id = SessionId::new();
+ store
+ .writer
+ .begin_session(NewSession {
+ id: session_id,
+ workspace_id: ws,
+ project_id: proj,
+ agent_kind: ai_memory_core::AgentKind::OpenCode,
+ cwd: None,
+ actor_user: None,
+ })
+ .await
+ .unwrap();
+ store.writer.end_session(session_id, None).await.unwrap();
+ wiki.write_page(WritePageRequest {
+ workspace_id: ws,
+ project_id: proj,
+ path: ai_memory_core::PagePath::new(format!("sessions/{session_id}.md")).unwrap(),
+ frontmatter: serde_json::json!({"title": "session"}),
+ body: body.to_string(),
+ tier: ai_memory_core::Tier::Episodic,
+ pinned: false,
+ title: None,
+ admission_ctx: None,
+ author_id: None,
+ actor: ActorContext::anonymous(),
+ evidence: Vec::new(),
+ })
+ .await
+ .unwrap();
+ }
+
+ let prompts = Arc::new(Mutex::new(Vec::new()));
+ let llm: Arc<dyn LlmProvider> = Arc::new(CapturingLlm {
+ prompts: prompts.clone(),
+ });
+ Harness {
+ _tmp: tmp,
+ store,
+ ws,
+ proj,
+ prompts,
+ llm,
+ }
+}
+
+fn experience_cfg(filter_enabled: bool) -> ExperienceConfig {
+ ExperienceConfig {
+ sessions: 10,
+ min_new_sessions: 1,
+ entropy_filter: EntropyFilterConfig {
+ enabled: filter_enabled,
+ ..EntropyFilterConfig::default()
+ },
+ ..ExperienceConfig::default()
+ }
+}
+
+/// Filter ON: the boilerplate session page is skipped from the prompt; the real
+/// one is consolidated.
+#[tokio::test]
+async fn boilerplate_session_is_skipped_from_consolidation() {
+ let h = harness_with_two_sessions().await;
+ let report = run_experience_review(
+ &h.store.reader,
+ h.llm.as_ref(),
+ h.ws,
+ h.proj,
+ AutoImproveReviewConfig::default(),
+ &experience_cfg(true),
+ )
+ .await
+ .unwrap();
+
+ let prompts = h.prompts.lock().unwrap();
+ assert_eq!(prompts.len(), 1, "the LLM was still called once");
+ let prompt = &prompts[0];
+ assert!(
+ prompt.contains("crates/ai-memory-store/src/ops.rs"),
+ "the real note is consolidated: {prompt}"
+ );
+ assert!(
+ !prompt.contains("status ok status ok"),
+ "the boilerplate note is skipped from the prompt: {prompt}"
+ );
+ assert!(
+ report
+ .warnings
+ .iter()
+ .any(|w| w.contains("entropy filter skipped")),
+ "the skip is observable in the report: {:?}",
+ report.warnings
+ );
+}
+
+/// Default (OFF): nothing is filtered — both session pages reach the prompt, so
+/// an upgrade changes no consolidation output.
+#[tokio::test]
+async fn default_off_changes_nothing() {
+ let h = harness_with_two_sessions().await;
+ run_experience_review(
+ &h.store.reader,
+ h.llm.as_ref(),
+ h.ws,
+ h.proj,
+ AutoImproveReviewConfig::default(),
+ &experience_cfg(false),
+ )
+ .await
+ .unwrap();
+
+ let prompts = h.prompts.lock().unwrap();
+ assert_eq!(prompts.len(), 1);
+ let prompt = &prompts[0];
+ assert!(
+ prompt.contains("crates/ai-memory-store/src/ops.rs"),
+ "real note present: {prompt}"
+ );
+ assert!(
+ prompt.contains("status ok status ok"),
+ "with the filter off the boilerplate is NOT skipped: {prompt}"
+ );
+}
diff --git a/crates/ai-memory-consolidate/tests/suite/mod.rs b/crates/ai-memory-consolidate/tests/suite/mod.rs
index 187fcd7a..1cf49841 100644
--- a/crates/ai-memory-consolidate/tests/suite/mod.rs
+++ b/crates/ai-memory-consolidate/tests/suite/mod.rs
@@ -4,9 +4,11 @@
mod abstract_backfill;
mod access_breadth_sweep;
+mod cold_cluster_sweep;
mod compaction_sweep;
mod embed_backfill;
mod embeddings;
+mod entropy_experience;
mod lifecycle;
mod local_embeddings;
mod multi_machine;
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index f77466a8..4c1c49ff 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -603,6 +603,17 @@ compact_cold_episodic = false # A2 opt-in: tier-down (compact) a cold
# keep abstract+summary+keep-tokens, drop
# prose. Reversible (git + supersession),
# zero-LLM. false = today's evict behaviour.
+dedup_cold_clusters = false # A3 opt-in: cluster near-duplicate cold
+ # episodic pages by embedding (cosine DBSCAN,
+ # adaptive eps) and collapse each cluster to
+ # one survivor (union of keep-tokens), others
+ # superseded with a merge note. Reversible
+ # (git + supersession), zero generative LLM,
+ # no-op with no embedder. Merge provenance in
+ # page_evidence. false = no clustering.
+# dedup_min_pts = 2 # DBSCAN density floor (0 ⇒ default 2)
+# dedup_max_eps = 0.15 # conservative eps ceiling (cosine distance;
+ # 0 ⇒ default). Lower = merges less.
[decay.half_life_days] # opt-in per-tier retention curves (all keys
# optional). Half-life in DAYS; converted to
@@ -651,6 +662,18 @@ enabled = true
interval_secs = 3600
max_sessions_per_tick = 1 # per project; scheduler ticks do not overlap
min_session_age_secs = 600
+experience_every_sessions = 0 # 0 disables the cross-session experience pass
+experience_sessions = 10 # session summaries one experience pass reads
+
+[auto_improve.scheduler.experience_entropy_filter] # A4 opt-in; off by default
+enabled = false # true: skip low-information session pages from
+ # the experience consolidation pass BEFORE the
+ # prompt/eval-gate/apply_batch. Advisory (skip,
+ # never delete); zero-LLM. false = no filtering.
+# min_chars = 16 # near-empty floor (non-whitespace chars)
+# min_entropy_bits_per_char = 2.0
+# max_repetition_ratio = 0.7 # 1 - distinct/total tokens above this ⇒ skip
+# repetition_min_tokens = 6 # repetition check applies only above this
[retrieval] # opt-in ranking signals; all off by default
query_intent = false # lexical session-recall routing: queries phrased as