Skip to content
Python

Merge main into feat/sdk-trimmed-raw

alvinfeib20331aa39b2
diff --cc CHANGELOG.md
index ca28e81,31ed08c..8ba160e
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -4,33 -4,45 +4,70 @@@ Hy Memory and MemHub are versioned and
## Unreleased
- ### Hy Memory (next release)
-
- - Raw conversations are stored already trimmed. MemHub now keeps every raw
+ - Hy Memory: new `hy_memory.reembed` module for switching the embedding
+ model. Writers must be paused for the whole run. Nothing is derived from
+ the provider or endpoint: source and target are named explicitly, and the
+ identity of a space is the model name plus the width (`space_identity`).
+ Only Chroma and Kuzu are supported.
+ - `vector_space_names(config, base_name=None)` returns every physical
+ collection a client opens for the six vector roles: memory, entities,
+ coding keys and working facts, each in the working and coding variants.
+ - `reembed_vector_space(source, target, embedding=...)` pages every raw
+ source collection, covering all users and all layers including L1_RAW. It
+ embeds the text the write path embedded (`reembed_text`: the canonical
+ state's `state_retrieval_text`, a schema's `retrieval_text`, otherwise the
+ document). It then writes each record into the target with the same id,
+ document and metadata. Each batch's widths are checked before anything is
+ written. A cancel stops at a batch boundary (`ReembedCancelled`). A rerun
+ skips the ids the target already holds, so it resumes, and a merge never
+ overwrites the target. The source is never written.
+ - `plan_reembed`, `verify_reembed` and `drop_vector_space` complete the
+ vector side. `verify_reembed` checks ids, metadata, per-user counts and
+ the width, and runs a self-recall sample. `drop_vector_space` refuses
+ collections that a live client in this process holds open.
+ - `scan_vector_spaces(persist_directory)` is read-only. It lists every
+ space on disk with its roles, counts, stored width, per-user counts and
+ the models recorded in metadata. `diff_spaces(source, target)` returns the
+ ids that the source holds and the target lacks, whatever their status.
+ - `reembed_graph(source_path, target_path, embedding=..., target_dims=...)`
+ builds a new Kuzu graph at the new width. It copies every node and edge
+ and recomputes `Memory.embedding`, `Memory.beh_embedding`, and the Topic
+ and HierarchyTopic embeddings, then rebuilds the HNSW index. The source
+ graph is opened read-only and kept for rollback. A run that fails or is
+ cancelled leaves no target behind, and a rerun starts over.
+ `verify_graph_reembed` and `drop_graph` complete the graph side.
+ - Hy Memory: `EmbedService.embed_batch` now splits large batches into
+ requests of `embedder.batch_size` texts each (`MEMORY_EMBEDDER_BATCH_SIZE`,
+ default 64); before, it sent them all in one request. On a 429 it waits for
+ the `Retry-After` time plus jitter, capped at 60 s. It takes a per-call
+ `max_retries`. A returned vector whose width differs from the configured
+ `embedding_dims` now raises `EmbeddingDimensionMismatchError` and is not
+ retried.
++- Hy Memory: Raw conversations are stored already trimmed. MemHub now keeps every raw
+ forever and lite mode stores nothing else, so tool output and long pasted
+ logs no longer stay in full. One rule (`hy_memory/utils/raw_trim.py`) is
+ applied to each message when the raw is written:
+ - `user` and `system` turns are kept in full;
+ - `assistant` turns and other speakers (`[John]:`) longer than 500
+ characters, after stripping, are cut to the first 500 plus `...`. This is
+ the limit history injection used before;
+ - `tool` turns keep their head and tail, and the middle becomes
+ `[...N bytes omitted by truncate...]`, with the coding limit
+ `coding.tool_result_max_bytes` (`MEMORY_CODING_TOOL_RESULT_MAX_BYTES`,
+ default 2048).
+
+ The rule is idempotent. It covers chat `L1_RAW` in lite, pro and ultra (lite
+ keeps tool turns, because it has no coding route) and the `memory_history`
+ ADD record. The coding/working `coding_trace` was already trimmed when it
+ was written. Its tool and assistant truncations are now idempotent, so
+ reading the history back no longer rewrites the omitted-count markers. The
+ text embedded for `L1_RAW` is the trimmed text. New raws carry
+ `custom.raw_trimmed=true`.
- - Extractor history ("last k messages") now joins the stored raws directly
++- Hy Memory: extractor history ("last k messages") now joins the stored raws directly
+ and does not trim them again. Raws written before this change have no
+ `raw_trimmed` flag, so they are still trimmed with the same rule when they
+ are read. The extractor still receives the current batch in full (tool
+ turns are still removed on the chat route).
## MemHub 1.0.5 — 2026-10-28