Python
fix: claim hand-added memories first, and never let older memories win
alvinfeic2f50dce06c2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2711b64..0df9bb4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,6 +4,15 @@ Hy Memory and MemHub are versioned and released independently.## Unreleased+- MemHub "添加记忆" (add a memory) no longer waits behind unprocessed history.+ Notes are claimed ahead of the event-time window that keeps history strictly+ old to new; they still run one at a time and in order within their session.+- SDK reconcile keeps time order. Both reconcile prompts now state that only+ `memory_at` decides which memory is newer, that an older memory must never+ supersede or overwrite a newer one, and that an unknown time is not newer;+ the Chinese prompt now lists `memory_at` among the new-memory fields. The+ reconciler also enforces this in code: a SUPERSEDE or UPDATE aimed at a memory+ with a later `memory_at` becomes an ADD, so the newer memory stays current.- The SDK memory page now reads its list, count, source and time labels andgraph from one page response for one source filter, so rows, totals and themap no longer disagree. Memories with no recorded source are shown anddiff --git a/hy_memory/agent/prompts_zh.py b/hy_memory/agent/prompts_zh.pyindex 63b49a8..a0ed95b 100644--- a/hy_memory/agent/prompts_zh.py+++ b/hy_memory/agent/prompts_zh.py@@ -743,6 +743,7 @@ RECONCILE_PROMPT_ZH = """你是一个记忆整合系统(Memory Reconciler)* content* owner+* memory_at(发生时间)* tagsnew memories 没有 ID。@@ -791,6 +792,7 @@ new memories 没有 ID。* 无损合并* 保留所有有效信息* 不要产生重复 memory+* 两者在某个值上不一致时,保留 memory_at 更晚的那个值;绝不用更早的 memory 覆盖更新的值### 同一次事件的补充 vs 另一次发生@@ -823,6 +825,7 @@ new memories 没有 ID。* 新 memory 与旧 memory 在同一维度冲突* 两者不能同时为真* 旧 memory 已不再代表当前状态+* 新 memory 的 memory_at 不早于旧 memory 的 memory_at例如:@@ -837,6 +840,7 @@ new memories 没有 ID。* wording 差异* 同义表达* 可共存偏好+* 新 memory 的 memory_at 早于旧 memory:它是更早的状态,绝不能取代更新的状态(见「时间规则」)要求:@@ -859,7 +863,7 @@ new memories 没有 ID。### 把多条旧记忆折叠进同一条链-如果**两条或更多**已有记忆描述的是**同一维度**,而新记忆是该维度上的最新变化,+如果**两条或更多**已有记忆描述的是**同一维度**,而新记忆是该维度上的最新变化(它的 memory_at 晚于所有被折叠的记忆),你可以把它们折叠进同一条演化链。此时用 `memory_ids`(有序列表,**旧→新**)代替单个 `memory_id`:@@ -951,13 +955,16 @@ SUPERSEDE 不需要复制旧内容,因为旧节点仍存在。## 时间规则-`memory_at: null`+每条 memory(新的和已有的)都带 memory_at,表示它发生的时间。记忆**不一定按发生顺序写入**:+历史可能晚导入,导入还没处理完时用户也可能手动添加记忆。所以一条 new memory 可能**比已有记忆更早**。-表示时间未知。--不要编造时间顺序。--不要仅因为“看起来更新”就 supersede。+* 新旧**只看 memory_at**。"New memories" 只表示"正在写入",不表示"更近"。+* 更早的 memory 绝不能取代更新的 memory:不能 SUPERSEDE 一条更新的记忆,也不能用 UPDATE 拿更早的值覆盖更新的值。+* 如果 new memory 比与它冲突的已有记忆更早:保留已有记忆作为当前状态;把 new memory 以"过去状态"ADD+ (用过去时,并带上时间,例如"2023 年用户住在上海"),没有新信息就不输出。+* `memory_at: null` 表示时间未知。未知**不算更新**:不要用 memory_at 为 null 的 memory 去 SUPERSEDE 一条有时间的记忆。+* 不要编造时间顺序。+* 不要仅因为“看起来更新”就 supersede。---diff --git a/hy_memory/agent/reconciler.py b/hy_memory/agent/reconciler.pyindex 3b4b831..ce676ba 100644--- a/hy_memory/agent/reconciler.py+++ b/hy_memory/agent/reconciler.py@@ -221,6 +221,8 @@ Rules:* merge losslessly* preserve all useful information* do NOT create duplicates+* if the two disagree on a value, keep the value from the memory with the later+ `memory_at`; never let an older memory overwrite a newer value### Same occurrence vs. another occurrence@@ -255,6 +257,7 @@ Use ONLY when:* the new memory contradicts an existing memory* both cannot be true at the same time* the old memory would now give the wrong current-state answer+* the new memory's `memory_at` is the same as or LATER than the existing memory's `memory_at`Examples:@@ -268,6 +271,8 @@ Do NOT use for:* compatible preferences* wording differences* semantic equivalence+* a new memory whose `memory_at` is EARLIER than the existing memory's — it is an+ older state and must never replace a newer one (see "Time order")Rules:@@ -289,7 +294,8 @@ Rules:### Folding several existing memories into one chainIf TWO OR MORE existing memories track the SAME dimension and the new memory is-the latest change on that dimension, you may fold them into a single evolution+the latest change on that dimension — its `memory_at` is later than every folded+memory's `memory_at` — you may fold them into a single evolutionchain. Use `memory_ids` (an ordered list, OLDEST → NEWEST) instead of a single`memory_id`:@@ -318,8 +324,7 @@ older memories' text into it; they remain queryable in the chain history.* Separate dated occurrences stay separate ADDs; use `same_series` only for thesame subject and same concrete activity/event type.* Keep each memory focused on one coherent topic.-* Do not invent chronology.-* `memory_at: null` means unknown time.+* Follow "Time order" below.* **NEVER merge or supersede across different subjects.** The subject is whoeverthe content is about: "The user ..." vs "John ..." vs "Mary ...". Memories aboutdifferent people are independent facts even when the sentence is otherwise@@ -332,6 +337,27 @@ older memories' text into it; they remain queryable in the chain history.---+# Time order++Every memory, new or existing, has `memory_at`: when it happened. Memories are+NOT always written in the order they happened — history can be imported late,+and a memory can be added while older history is still being processed. A new+memory can therefore be OLDER than an existing one.++* Decide which memory is newer ONLY by `memory_at`. "New memories" means "being+ written now", not "more recent".+* An older memory must never replace a newer one: no SUPERSEDE of a newer+ memory, and no UPDATE that overwrites a newer value with an older one.+* If a new memory is older than an existing memory it conflicts with, keep the+ existing memory as the current state. ADD the new one as a past state, worded+ in the past with its time (for example "In 2023 the user lived in Shanghai"),+ or output nothing if it adds no information.+* `memory_at: null` means unknown time. Unknown is NOT newer: never SUPERSEDE a+ memory that has a known `memory_at` with one whose `memory_at` is null.+* Do not invent chronology.++---+# Optional: `related`On any op you MAY add a `related` array recording how the memory being written@@ -441,6 +467,81 @@ def append_reconcile_structure_prompt(return prompt + RECONCILE_STRUCTURE_METADATA_PROMPT_EN+def _as_aware(value: Any) -> Optional[datetime]:+ """memory_at as a comparable aware datetime; None when unknown or unreadable."""+ if isinstance(value, str) and value:+ try:+ value = datetime.fromisoformat(value)+ except (TypeError, ValueError):+ return None+ if not isinstance(value, datetime):+ return None+ # Stored nodes rebuild memory_at from epoch seconds as naive local time+ # (MemoryNode.from_dict), so a naive value is local, not UTC.+ return value if value.tzinfo is not None else value.astimezone()+++def enforce_time_order(+ ops: List["ReconcileOp"],+ new_memory_at: Any,+ existing: Dict[str, Any],+) -> List["ReconcileOp"]:+ """An older memory must never replace a newer one, whatever the model says.++ Memories are not always written in the order they happened: history is+ imported late, and a person can add a memory while older history is still+ being processed. The prompt tells the model to compare `memory_at`; this+ makes it hold. A SUPERSEDE or UPDATE aimed at an existing memory whose+ `memory_at` is later than the memory being written is turned into an ADD,+ so the newer memory stays the current state. Unknown times are not+ compared. A content-less UPDATE (the legacy DELETE mapping) is dropped.+ """+ written_at = _as_aware(new_memory_at)+ if written_at is None:+ return ops+ kept: List[ReconcileOp] = []+ for op in ops:+ if op.op not in ("SUPERSEDE", "UPDATE"):+ kept.append(op)+ continue+ targets = list(op.memory_ids) if op.memory_ids else [op.memory_id]+ newer = [+ target for target in targets+ if target and (node := existing.get(target)) is not None+ and (target_at := _as_aware(getattr(node, "memory_at", None))) is not None+ and target_at > written_at+ ]+ if not newer:+ kept.append(op)+ continue+ logger.warning(+ "[reconciler] %s blocked: memory_at %s is older than existing %s; "+ "kept the newer memory",+ op.op, written_at.isoformat(), ",".join(newer),+ )+ if not op.content:+ continue+ kept.append(ReconcileOp(+ op="ADD",+ content=op.content,+ layer=op.layer,+ owner=op.owner,+ tags=list(op.tags),+ tag_ners=list(op.tag_ners),+ topic_paths=list(op.topic_paths),+ event_facet=dict(op.event_facet),+ topic_assignments=list(op.topic_assignments),+ structure_references=list(op.structure_references),+ topic_paths_provided=op.topic_paths_provided,+ event_facet_provided=op.event_facet_provided,+ topic_assignments_provided=op.topic_assignments_provided,+ speculate=op.speculate,+ related=list(op.related),+ reason=f"time_order: older than {','.join(newer)}; {op.reason}"[:500],+ ))+ return kept++def build_reconcile_prompt(*,new_memories: List[str],@@ -1399,6 +1500,7 @@ class MemoryReconciler:_tok_total += response.tokens_usedraw_response = response.contentops = self._parse_ops(raw_response)+ ops = enforce_time_order(ops, current_time, candidate_map)ops = self._inherit_exact_input_structure(ops, new_memories_with_meta,)diff --git a/memhub/apps/core/src/one_memory/memory_server/durable_queue.py b/memhub/apps/core/src/one_memory/memory_server/durable_queue.pyindex 5251ca6..65ff266 100644--- a/memhub/apps/core/src/one_memory/memory_server/durable_queue.py+++ b/memhub/apps/core/src/one_memory/memory_server/durable_queue.py@@ -316,6 +316,12 @@ LEGACY_MEMORY_WRITE_JOB_COLUMNS = (_FIFO_BLOCKING_STATES = (PENDING, RUNNING, WAITING_FOR_PERMISSION, WAITING_FOR_MODEL)EVENT_TIME_WINDOW = timedelta(minutes=10)+#: Sources whose jobs are a person adding a memory by hand ("添加记忆"), not+#: history. History is digested strictly old to new through the event-time+#: window; a note is stamped "now", so inside that window it waited behind the+#: whole unprocessed history. These jobs are claimed ahead of the window,+#: still one at a time per source and in order within their session.+DIRECT_WRITE_SOURCES = frozenset({"one-memory-manual"})@dataclass(frozen=True)@@ -2601,6 +2607,39 @@ class DurableWriteDrain:placeholders = ",".join("?" for _ in enabled_sources)source_clause = f"AND candidate.source_key IN ({placeholders}) "source_params = tuple(sorted(enabled_sources))+ direct_sources = ",".join("?" for _ in DIRECT_WRITE_SOURCES)+ # Same eligibility as the window claim below (free lease, no+ # running job of its source, nothing earlier in its session), but+ # not bound to the window: a note must not wait for history.+ direct_rows = connection.execute(+ "SELECT candidate.job_id,candidate.source_key "+ "FROM memory_write_jobs AS candidate "+ "WHERE candidate.state=? "+ "AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "+ + source_clause+ + f"AND candidate.source_key IN ({direct_sources}) "+ "AND NOT EXISTS ("+ "SELECT 1 FROM memory_write_jobs AS held "+ "WHERE held.source_key=candidate.source_key AND held.state=? "+ "AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "+ "AND NOT EXISTS ("+ "SELECT 1 FROM memory_write_jobs AS earlier "+ "WHERE earlier.session_key=candidate.session_key "+ f"AND earlier.state IN ({blocking}) "+ "AND (earlier.event_at<candidate.event_at OR "+ "(earlier.event_at=candidate.event_at "+ "AND earlier.queue_seq<candidate.queue_seq))) "+ "ORDER BY candidate.queue_seq",+ (+ PENDING,+ now.isoformat(),+ *source_params,+ *sorted(DIRECT_WRITE_SOURCES),+ RUNNING,+ now.isoformat(),+ *_FIFO_BLOCKING_STATES,+ ),+ ).fetchall()window = connection.execute("SELECT window_anchor,window_end FROM memory_write_schedule_state WHERE singleton=1").fetchone()@@ -2654,6 +2693,7 @@ class DurableWriteDrain:if active is None:connection.execute("DELETE FROM memory_write_schedule_state WHERE singleton=1")window = None+ anchor: Any = Noneif window is None:anchor = connection.execute("SELECT candidate.event_at "@@ -2682,8 +2722,10 @@ class DurableWriteDrain:*_FIFO_BLOCKING_STATES,),).fetchone()- if anchor is None:+ if anchor is None and not direct_rows:return ()+ has_window = window is not None or anchor is not None+ if window is None and anchor is not None:anchor_at = datetime.fromisoformat(str(anchor[0]))if anchor_at.tzinfo is None:anchor_at = anchor_at.replace(tzinfo=UTC)@@ -2695,39 +2737,41 @@ class DurableWriteDrain:"VALUES (1,?,?,?)",(anchor_text, end_text, now.isoformat()),)- rows = connection.execute(- "SELECT candidate.job_id,candidate.source_key "- "FROM memory_write_jobs AS candidate "- "WHERE candidate.state=? "- "AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "- + source_clause- + "AND candidate.event_at>=? AND candidate.event_at<? "- "AND NOT EXISTS ("- "SELECT 1 FROM memory_write_jobs AS held "- "WHERE held.source_key=candidate.source_key AND held.state=? "- "AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "- "AND NOT EXISTS ("- "SELECT 1 FROM memory_write_jobs AS earlier "- "WHERE earlier.session_key=candidate.session_key "- f"AND earlier.state IN ({blocking}) "- "AND (earlier.event_at<candidate.event_at OR "- "(earlier.event_at=candidate.event_at "- "AND earlier.queue_seq<candidate.queue_seq))) "- "ORDER BY candidate.event_at,candidate.queue_seq",- (- PENDING,- now.isoformat(),- *source_params,- anchor_text,- end_text,- RUNNING,- now.isoformat(),- *_FIFO_BLOCKING_STATES,- ),- ).fetchall()+ rows: list[Any] = []+ if has_window:+ rows = connection.execute(+ "SELECT candidate.job_id,candidate.source_key "+ "FROM memory_write_jobs AS candidate "+ "WHERE candidate.state=? "+ "AND (candidate.lease_expires_at IS NULL OR candidate.lease_expires_at < ?) "+ + source_clause+ + "AND candidate.event_at>=? AND candidate.event_at<? "+ "AND NOT EXISTS ("+ "SELECT 1 FROM memory_write_jobs AS held "+ "WHERE held.source_key=candidate.source_key AND held.state=? "+ "AND held.lease_owner IS NOT NULL AND held.lease_expires_at>=?) "+ "AND NOT EXISTS ("+ "SELECT 1 FROM memory_write_jobs AS earlier "+ "WHERE earlier.session_key=candidate.session_key "+ f"AND earlier.state IN ({blocking}) "+ "AND (earlier.event_at<candidate.event_at OR "+ "(earlier.event_at=candidate.event_at "+ "AND earlier.queue_seq<candidate.queue_seq))) "+ "ORDER BY candidate.event_at,candidate.queue_seq",+ (+ PENDING,+ now.isoformat(),+ *source_params,+ anchor_text,+ end_text,+ RUNNING,+ now.isoformat(),+ *_FIFO_BLOCKING_STATES,+ ),+ ).fetchall()selected: list[str] = []selected_sources: set[str] = set()- for row in rows:+ for row in (*direct_rows, *rows):source_key = str(row["source_key"])if source_key in selected_sources:continuediff --git a/memhub/apps/core/tests/test_memory_server_durable_queue.py b/memhub/apps/core/tests/test_memory_server_durable_queue.pyindex 60b0f23..f317bb1 100644--- a/memhub/apps/core/tests/test_memory_server_durable_queue.py+++ b/memhub/apps/core/tests/test_memory_server_durable_queue.py@@ -5330,3 +5330,70 @@ def test_equal_event_time_uses_queue_sequence_inside_one_session(tmp_path) -> Noowner="equal-owner",)assert drain.claim(limit=10) == (second.job_id,)+++def test_a_hand_added_memory_is_claimed_ahead_of_unprocessed_history(tmp_path) -> None:+ """A hand-added memory ("添加记忆") waited behind every unprocessed history job.++ History is claimed through a ten-minute event-time window, oldest first. A+ note is stamped "now", so with a backlog it sat at the far end of the+ timeline and ran only after the whole history was digested.+ """+ queue = DurableWriteQueue(tmp_path / "one-memory.sqlite3")+ old = datetime(2025, 1, 1, tzinfo=UTC)+ history = [+ queue.submit(+ _request(+ key=f"history-{index}",+ evidence=(f"h{index}",),+ source="cursor",+ session=f"cursor-{index}",+ event_at=old + timedelta(hours=index),+ )+ )+ for index in range(3)+ ]+ note = queue.submit(+ _request(+ key="note",+ evidence=("n1",),+ source="one-memory-manual",+ session="note:1",+ event_at=datetime.now(UTC),+ )+ )+ drain = DurableWriteDrain(queue, execute=_never_execute)++ assert drain.claim(limit=1) == (note.job_id,)+ # History keeps its own order through the window.+ assert drain.claim(limit=1) == (history[0].job_id,)+++def test_hand_added_memories_still_run_one_at_a_time_in_order(tmp_path) -> None:+ queue = DurableWriteQueue(tmp_path / "one-memory.sqlite3")+ now = datetime.now(UTC)+ first = queue.submit(+ _request(+ key="note-1",+ evidence=("n1",),+ source="one-memory-manual",+ session="note:1",+ event_at=now,+ )+ )+ second = queue.submit(+ _request(+ key="note-2",+ evidence=("n2",),+ source="one-memory-manual",+ session="note:2",+ event_at=now + timedelta(seconds=1),+ )+ )+ drain = DurableWriteDrain(queue, execute=_never_execute)++ assert drain.claim(limit=5) == (first.job_id,)+ # Its source is busy: the next note waits for the running one, not for history.+ assert drain.claim(limit=5) == ()+ assert queue.get("note-2").state == PENDING+ assert second.job_iddiff --git a/tests/test_reconcile_time_order.py b/tests/test_reconcile_time_order.pynew file mode 100644index 0000000..d372a50--- /dev/null+++ b/tests/test_reconcile_time_order.py@@ -0,0 +1,118 @@+"""An older memory must never replace a newer one, whatever the model returns.++Memories are not written in the order they happened: history is imported late,+and a hand-added memory can be written while older history is still being+digested. A memory from 2023 arriving after one from 2025 must not supersede+or overwrite it.+"""++from datetime import datetime, timedelta, timezone+from types import SimpleNamespace++from hy_memory.agent.reconciler import ReconcileOp, build_reconcile_prompt, enforce_time_order++NEWER = datetime(2025, 6, 1, 9, 0, tzinfo=timezone.utc)+OLDER = datetime(2023, 3, 1, 9, 0, tzinfo=timezone.utc)+++def _existing(**times):+ return {memory_id: SimpleNamespace(memory_at=at) for memory_id, at in times.items()}+++def test_an_older_memory_cannot_supersede_a_newer_one():+ ops = [ReconcileOp(op="SUPERSEDE", memory_id="m1", memory_ids=["m1"],+ content="The user lives in Shanghai.", owner="user", tags=["home"])]++ kept = enforce_time_order(ops, OLDER.isoformat(), _existing(m1=NEWER))++ assert len(kept) == 1+ assert kept[0].op == "ADD"+ assert kept[0].memory_id is None and kept[0].memory_ids == []+ assert kept[0].content == "The user lives in Shanghai."+ assert kept[0].tags == ["home"]+ assert kept[0].reason.startswith("time_order:")+++def test_an_older_memory_cannot_update_a_newer_value():+ ops = [ReconcileOp(op="UPDATE", memory_id="m1", content="merged")]++ kept = enforce_time_order(ops, OLDER, _existing(m1=NEWER))++ assert [op.op for op in kept] == ["ADD"]+++def test_a_chain_fold_with_any_newer_member_is_blocked():+ ops = [ReconcileOp(op="SUPERSEDE", memory_id="m2", memory_ids=["m1", "m2"],+ content="latest")]++ kept = enforce_time_order(+ ops, NEWER - timedelta(days=1), _existing(m1=OLDER, m2=NEWER)+ )++ assert [op.op for op in kept] == ["ADD"]+++def test_a_newer_memory_still_supersedes_an_older_one():+ op = ReconcileOp(op="SUPERSEDE", memory_id="m1", memory_ids=["m1"], content="moved")++ assert enforce_time_order([op], NEWER, _existing(m1=OLDER)) == [op]+++def test_unknown_times_are_not_compared():+ update = ReconcileOp(op="UPDATE", memory_id="m1", content="x")+ supersede = ReconcileOp(op="SUPERSEDE", memory_id="m2", memory_ids=["m2"], content="y")++ assert enforce_time_order([update], "", _existing(m1=NEWER)) == [update]+ assert enforce_time_order([supersede], OLDER, _existing(m2=None)) == [supersede]+++def test_a_contentless_update_against_a_newer_memory_is_dropped():+ # The legacy DELETE mapping: nothing to keep as an ADD.+ ops = [ReconcileOp(op="UPDATE", memory_id="m1", content=None)]++ assert enforce_time_order(ops, OLDER, _existing(m1=NEWER)) == []+++def test_stored_naive_times_are_read_as_local_time():+ # MemoryNode.from_dict rebuilds memory_at from epoch seconds as naive local time.+ stored = datetime.fromtimestamp(NEWER.timestamp())+ assert stored.tzinfo is None+ op = ReconcileOp(op="SUPERSEDE", memory_id="m1", memory_ids=["m1"], content="x")++ blocked = enforce_time_order([op], NEWER - timedelta(minutes=1), _existing(m1=stored))+ allowed = enforce_time_order([op], NEWER + timedelta(minutes=1), _existing(m1=stored))++ assert [item.op for item in blocked] == ["ADD"]+ assert allowed == [op]+++def test_add_ops_and_unknown_targets_pass_through():+ add = ReconcileOp(op="ADD", content="new")+ stray = ReconcileOp(op="UPDATE", memory_id="not-a-candidate", content="x")++ assert enforce_time_order([add, stray], OLDER, {}) == [add, stray]+++def test_both_prompts_state_the_time_order_rule():+ english = build_reconcile_prompt(+ new_memories=["The user lives in Shanghai."],+ new_memories_block="[]",+ existing_memories_block="[]",+ current_date="2026-10-27 10:00",+ few_shot_enabled=False,+ structure_metadata_enabled=False,+ )+ chinese = build_reconcile_prompt(+ new_memories=["用户住在上海。"],+ new_memories_block="[]",+ existing_memories_block="[]",+ current_date="2026-10-27 10:00",+ few_shot_enabled=False,+ structure_metadata_enabled=False,+ )++ assert "# Time order" in english+ assert "An older memory must never replace a newer one" in english+ assert "## 时间规则" in chinese+ assert "更早的 memory 绝不能取代更新的 memory" in chinese+ assert "* memory_at(发生时间)" in chinese