Python
Merge main into feat/sdk-trimmed-raw
alvinfeib20331aa39b2
diff --cc CHANGELOG.mdindex ca28e81,31ed08c..8ba160e--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -4,33 -4,45 +4,70 @@@ Hy Memory and MemHub are versioned and## Unreleased- ### Hy Memory (next release)-- - Raw conversations are stored already trimmed. MemHub now keeps every raw+ - Hy Memory: new `hy_memory.reembed` module for switching the embedding+ model. Writers must be paused for the whole run. Nothing is derived from+ the provider or endpoint: source and target are named explicitly, and the+ identity of a space is the model name plus the width (`space_identity`).+ Only Chroma and Kuzu are supported.+ - `vector_space_names(config, base_name=None)` returns every physical+ collection a client opens for the six vector roles: memory, entities,+ coding keys and working facts, each in the working and coding variants.+ - `reembed_vector_space(source, target, embedding=...)` pages every raw+ source collection, covering all users and all layers including L1_RAW. It+ embeds the text the write path embedded (`reembed_text`: the canonical+ state's `state_retrieval_text`, a schema's `retrieval_text`, otherwise the+ document). It then writes each record into the target with the same id,+ document and metadata. Each batch's widths are checked before anything is+ written. A cancel stops at a batch boundary (`ReembedCancelled`). A rerun+ skips the ids the target already holds, so it resumes, and a merge never+ overwrites the target. The source is never written.+ - `plan_reembed`, `verify_reembed` and `drop_vector_space` complete the+ vector side. `verify_reembed` checks ids, metadata, per-user counts and+ the width, and runs a self-recall sample. `drop_vector_space` refuses+ collections that a live client in this process holds open.+ - `scan_vector_spaces(persist_directory)` is read-only. It lists every+ space on disk with its roles, counts, stored width, per-user counts and+ the models recorded in metadata. `diff_spaces(source, target)` returns the+ ids that the source holds and the target lacks, whatever their status.+ - `reembed_graph(source_path, target_path, embedding=..., target_dims=...)`+ builds a new Kuzu graph at the new width. It copies every node and edge+ and recomputes `Memory.embedding`, `Memory.beh_embedding`, and the Topic+ and HierarchyTopic embeddings, then rebuilds the HNSW index. The source+ graph is opened read-only and kept for rollback. A run that fails or is+ cancelled leaves no target behind, and a rerun starts over.+ `verify_graph_reembed` and `drop_graph` complete the graph side.+ - Hy Memory: `EmbedService.embed_batch` now splits large batches into+ requests of `embedder.batch_size` texts each (`MEMORY_EMBEDDER_BATCH_SIZE`,+ default 64); before, it sent them all in one request. On a 429 it waits for+ the `Retry-After` time plus jitter, capped at 60 s. It takes a per-call+ `max_retries`. A returned vector whose width differs from the configured+ `embedding_dims` now raises `EmbeddingDimensionMismatchError` and is not+ retried.++- Hy Memory: Raw conversations are stored already trimmed. MemHub now keeps every raw+ forever and lite mode stores nothing else, so tool output and long pasted+ logs no longer stay in full. One rule (`hy_memory/utils/raw_trim.py`) is+ applied to each message when the raw is written:+ - `user` and `system` turns are kept in full;+ - `assistant` turns and other speakers (`[John]:`) longer than 500+ characters, after stripping, are cut to the first 500 plus `...`. This is+ the limit history injection used before;+ - `tool` turns keep their head and tail, and the middle becomes+ `[...N bytes omitted by truncate...]`, with the coding limit+ `coding.tool_result_max_bytes` (`MEMORY_CODING_TOOL_RESULT_MAX_BYTES`,+ default 2048).++ The rule is idempotent. It covers chat `L1_RAW` in lite, pro and ultra (lite+ keeps tool turns, because it has no coding route) and the `memory_history`+ ADD record. The coding/working `coding_trace` was already trimmed when it+ was written. Its tool and assistant truncations are now idempotent, so+ reading the history back no longer rewrites the omitted-count markers. The+ text embedded for `L1_RAW` is the trimmed text. New raws carry+ `custom.raw_trimmed=true`.- - Extractor history ("last k messages") now joins the stored raws directly++- Hy Memory: extractor history ("last k messages") now joins the stored raws directly+ and does not trim them again. Raws written before this change have no+ `raw_trimmed` flag, so they are still trimmed with the same rule when they+ are read. The extractor still receives the current batch in full (tool+ turns are still removed on the chat route).## MemHub 1.0.5 — 2026-10-28