Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
refactor(consolidate): share the WIKILINKS prompt section across prompts
Peterson Salme22eb7586a566
diff --git a/.gitattributes b/.gitattributesindex 43646d43..1d2f2ad4 100644--- a/.gitattributes+++ b/.gitattributes@@ -1,3 +1,4 @@# These files are embedded verbatim by include_str!, so checkout EOLs become# part of the CLI and MCP payload contract.crates/ai-memory-core/src/routing_skills/*/SKILL.md text eol=lf+crates/ai-memory-consolidate/prompts/*.md text eol=lfdiff --git a/crates/ai-memory-consolidate/prompts/batch_consolidate_system.md b/crates/ai-memory-consolidate/prompts/batch_consolidate_system.mdindex e7ba6bd4..9cb4c4be 100644--- a/crates/ai-memory-consolidate/prompts/batch_consolidate_system.md+++ b/crates/ai-memory-consolidate/prompts/batch_consolidate_system.md@@ -84,34 +84,7 @@ Do:job-state marker locations, staged-script paths — listing only whatthe observations actually show. This is extraction, not scaffolding.-## WIKILINKS — connect pages into the graph--The wiki is a graph: pages reference each other with Obsidian-style-wikilinks, and pages without links grow as disconnected islands.-When a page you write relates to another page — one you are-emitting in this same reply, or an existing page named in the-input — reference it inline with a wikilink:--- `[[page-path]]` — a page in the same project. The target is the- page *path* relative to the project root- (e.g. `[[decisions/0003-no-vector-db]]`), not the display title.-- `[[project:page-path]]` — a page in a sibling project. Use it- only when the observations or page body name that project and page path- (not a repo file path), e.g. `[[billing:audio-pipeline]]`.- Never invent project names or page paths.-- `[[_global:page-path]]` — a cross-cutting principle, convention,- or trap for every project, only if that exact path appears in- the observations or page body.--A missing same-project target is fine: it is recorded as a pending-link and resolves when the page appears. A missing `project:` or-`_global:` target is reported by lint as broken. A few well-chosen-links beat exhaustive linking, and zero links is fine when nothing-in the input genuinely relates. Link only where the input gives a-genuine reason — a page named in the input, one you are emitting-in this reply, or a same-project page the observations clearly-relate to. Never pad a page with links to hit a number, and-never pad with a cross-scope link you cannot see in the input.+{{wikilinks}}## OUTPUT LANGUAGEdiff --git a/crates/ai-memory-consolidate/prompts/bootstrap_system.md b/crates/ai-memory-consolidate/prompts/bootstrap_system.mdindex f3c5d6e8..a1ab41f2 100644--- a/crates/ai-memory-consolidate/prompts/bootstrap_system.md+++ b/crates/ai-memory-consolidate/prompts/bootstrap_system.md@@ -54,34 +54,7 @@ don't paper over it.both extremes.- Tags: 0-5 short kebab-case tags per page.-## WIKILINKS — connect pages into the graph--The wiki is a graph: pages reference each other with Obsidian-style-wikilinks, and pages without links grow as disconnected islands.-When a page you write relates to another page — one you are-emitting in this same reply, or an existing page named in the-input — reference it inline with a wikilink:--- `[[page-path]]` — a page in the same project. The target is the- page *path* relative to the project root- (e.g. `[[decisions/0003-no-vector-db]]`), not the display title.-- `[[project:page-path]]` — a page in a sibling project. Use it- only when the supplied sources name that project and page path- (not a repo file path), e.g. `[[billing:audio-pipeline]]`.- Never invent project names or page paths.-- `[[_global:page-path]]` — a cross-cutting principle, convention,- or trap for every project, only if that exact path appears in- the supplied sources.--A missing same-project target is fine: it is recorded as a pending-link and resolves when the page appears. A missing `project:` or-`_global:` target is reported by lint as broken. A few well-chosen-links beat exhaustive linking, and zero links is fine when nothing-in the input genuinely relates. Link only where the input gives a-genuine reason — a page named in the input, one you are emitting-in this reply, or a same-project page the sources clearly-relate to. Never pad a page with links to hit a number, and-never pad with a cross-scope link you cannot see in the input.+{{wikilinks}}## OUTPUT LANGUAGEdiff --git a/crates/ai-memory-consolidate/prompts/single_consolidate_system.md b/crates/ai-memory-consolidate/prompts/single_consolidate_system.mdindex d59d1d88..d3e5404a 100644--- a/crates/ai-memory-consolidate/prompts/single_consolidate_system.md+++ b/crates/ai-memory-consolidate/prompts/single_consolidate_system.md@@ -80,34 +80,7 @@ history only when useful, but must not be presented as current fact.no longer. Don't pad with generic tutorial filler, butdon't truncate substance either.-## WIKILINKS — connect pages into the graph--The wiki is a graph: pages reference each other with Obsidian-style-wikilinks, and pages without links grow as disconnected islands.-When a page you write relates to another page — one you are-emitting in this same reply, or an existing page named in the-input — reference it inline with a wikilink:--- `[[page-path]]` — a page in the same project. The target is the- page *path* relative to the project root- (e.g. `[[decisions/0003-no-vector-db]]`), not the display title.-- `[[project:page-path]]` — a page in a sibling project. Use it- only when the observations or page body name that project and page path- (not a repo file path), e.g. `[[billing:audio-pipeline]]`.- Never invent project names or page paths.-- `[[_global:page-path]]` — a cross-cutting principle, convention,- or trap for every project, only if that exact path appears in- the observations or page body.--A missing same-project target is fine: it is recorded as a pending-link and resolves when the page appears. A missing `project:` or-`_global:` target is reported by lint as broken. A few well-chosen-links beat exhaustive linking, and zero links is fine when nothing-in the input genuinely relates. Link only where the input gives a-genuine reason — a page named in the input, one you are emitting-in this reply, or a same-project page the observations clearly-relate to. Never pad a page with links to hit a number, and-never pad with a cross-scope link you cannot see in the input.+{{wikilinks}}## OUTPUT LANGUAGEdiff --git a/crates/ai-memory-consolidate/prompts/wikilinks_fragment.md b/crates/ai-memory-consolidate/prompts/wikilinks_fragment.mdnew file mode 100644index 00000000..22a0ccc2--- /dev/null+++ b/crates/ai-memory-consolidate/prompts/wikilinks_fragment.md@@ -0,0 +1,28 @@+## WIKILINKS — connect pages into the graph++The wiki is a graph: pages reference each other with Obsidian-style+wikilinks, and pages without links grow as disconnected islands.+When a page you write relates to another page — one you are+emitting in this same reply, or an existing page named in the+input — reference it inline with a wikilink:++- `[[page-path]]` — a page in the same project. The target is the+ page *path* relative to the project root+ (e.g. `[[decisions/0003-no-vector-db]]`), not the display title.+- `[[project:page-path]]` — a page in a sibling project. Use it+ only when {{sources}} name that project and page path+ (not a repo file path), e.g. `[[billing:audio-pipeline]]`.+ Never invent project names or page paths.+- `[[_global:page-path]]` — a cross-cutting principle, convention,+ or trap for every project, only if that exact path appears in+ {{sources}}.++A missing same-project target is fine: it is recorded as a pending+link and resolves when the page appears. A missing `project:` or+`_global:` target is reported by lint as broken. A few well-chosen+links beat exhaustive linking, and zero links is fine when nothing+in the input genuinely relates. Link only where the input gives a+genuine reason — a page named in the input, one you are emitting+in this reply, or a same-project page {{basis}} clearly+relate to. Never pad a page with links to hit a number, and+never pad with a cross-scope link you cannot see in the input.diff --git a/crates/ai-memory-consolidate/src/bootstrap.rs b/crates/ai-memory-consolidate/src/bootstrap.rsindex 1b61ae4f..e523a01f 100644--- a/crates/ai-memory-consolidate/src/bootstrap.rs+++ b/crates/ai-memory-consolidate/src/bootstrap.rs@@ -1436,7 +1436,11 @@ fn split_oversized_source(source: BootstrapSource, usable: usize) -> Vec<Bootstr/// System prompt for bootstrap. Loaded at compile time from/// `prompts/bootstrap_system.md`.-const SYSTEM_PROMPT: &str = include_str!("../prompts/bootstrap_system.md");+pub(crate) const SYSTEM_PROMPT: &str = crate::prompt_template::prompt_with_wikilinks!(+ include_str!("../prompts/bootstrap_system.md"),+ "the supplied sources",+ "the sources"+);fn build_chunk_request(sources: &[BootstrapSource],diff --git a/crates/ai-memory-consolidate/src/consolidator.rs b/crates/ai-memory-consolidate/src/consolidator.rsindex 25f64de3..834003e6 100644--- a/crates/ai-memory-consolidate/src/consolidator.rs+++ b/crates/ai-memory-consolidate/src/consolidator.rs@@ -1303,7 +1303,11 @@ fn render_slot_snapshots(slots: &[SlotSnapshot], max_chars: usize) -> String {/// is plain-text-editable + version-controlled as a Markdown file/// alongside the code. Public so off-tree harnesses (`evals/`) can/// inspect the exact prompt without duplicating it.-pub const BATCH_SYSTEM_PROMPT: &str = include_str!("../prompts/batch_consolidate_system.md");+pub const BATCH_SYSTEM_PROMPT: &str = crate::prompt_template::prompt_with_wikilinks!(+ include_str!("../prompts/batch_consolidate_system.md"),+ "the observations or page body",+ "the observations"+);fn build_request(session_id: SessionId,@@ -2005,7 +2009,11 @@ fn disambiguate_anchor_title(/// System prompt for single-page consolidation. Loaded at compile/// time from `prompts/single_consolidate_system.md`.-const SYSTEM_PROMPT: &str = include_str!("../prompts/single_consolidate_system.md");+pub(crate) const SYSTEM_PROMPT: &str = crate::prompt_template::prompt_with_wikilinks!(+ include_str!("../prompts/single_consolidate_system.md"),+ "the observations or page body",+ "the observations"+);#[cfg(test)]mod tests {diff --git a/crates/ai-memory-consolidate/src/lib.rs b/crates/ai-memory-consolidate/src/lib.rsindex 62755157..93472a1c 100644--- a/crates/ai-memory-consolidate/src/lib.rs+++ b/crates/ai-memory-consolidate/src/lib.rs@@ -26,6 +26,7 @@ pub mod lint;mod path_sanitize;pub mod profile;pub mod projection;+mod prompt_template;pub mod sweep;pub mod types;diff --git a/crates/ai-memory-consolidate/src/prompt_template.rs b/crates/ai-memory-consolidate/src/prompt_template.rsnew file mode 100644index 00000000..828c1f23--- /dev/null+++ b/crates/ai-memory-consolidate/src/prompt_template.rs@@ -0,0 +1,156 @@+//! Compile-time prompt assembly. The `## WIKILINKS` section is shared by the+//! single, batch and bootstrap prompts; each `.md` carries a `{{wikilinks}}`+//! line and the fragment's `{{sources}}` and `{{basis}}` slots take the per-prompt wording.+//! Done in const evaluation so the prompts stay `const &str`.++/// Shared wikilink guidance; `{{sources}}` names where link targets may come from+/// and `{{basis}}` what the same-project links must clearly relate to.+pub(crate) const WIKILINKS: &str = include_str!("../prompts/wikilinks_fragment.md");++const fn matches_at(t: &[u8], k: &[u8], i: usize) -> bool {+ if i + k.len() > t.len() {+ return false;+ }+ let mut j = 0;+ while j < k.len() {+ if t[i + j] != k[j] {+ return false;+ }+ j += 1;+ }+ true+}++const fn count(t: &[u8], k: &[u8]) -> usize {+ let (mut i, mut n) = (0, 0);+ while i < t.len() {+ if matches_at(t, k, i) {+ n += 1;+ i += k.len();+ } else {+ i += 1;+ }+ }+ n+}++pub(crate) const fn count_str(t: &str, k: &str) -> usize {+ count(t.as_bytes(), k.as_bytes())+}++pub(crate) const fn replaced_len(t: &str, k: &str, v: &str) -> usize {+ t.len() - count(t.as_bytes(), k.as_bytes()) * k.len()+ + count(t.as_bytes(), k.as_bytes()) * v.len()+}++pub(crate) const fn replaced<const N: usize>(t: &str, k: &str, v: &str) -> [u8; N] {+ let (t, k, v) = (t.as_bytes(), k.as_bytes(), v.as_bytes());+ let mut out = [0u8; N];+ let (mut i, mut o) = (0, 0);+ while i < t.len() {+ if matches_at(t, k, i) {+ let mut j = 0;+ while j < v.len() {+ out[o] = v[j];+ o += 1;+ j += 1;+ }+ i += k.len();+ } else {+ out[o] = t[i];+ o += 1;+ i += 1;+ }+ }+ out+}++/// `replace!(template, key, value)` as a `&'static str` constant.+macro_rules! replace {+ ($t:expr, $k:expr, $v:expr) => {{+ const N: usize = $crate::prompt_template::replaced_len($t, $k, $v);+ const BYTES: [u8; N] = $crate::prompt_template::replaced::<N>($t, $k, $v);+ match ::core::str::from_utf8(&BYTES) {+ Ok(s) => s,+ Err(_) => panic!("prompt template is not UTF-8"),+ }+ }};+}++/// Expands the shared wikilink fragment into `$template`, naming `$sources`+/// as the allowed origin of cross-scope link targets and `$basis` as what+/// same-project links must clearly relate to.+macro_rules! prompt_with_wikilinks {+ ($template:expr, $sources:expr, $basis:expr) => {{+ const WITH_SOURCES: &str = $crate::prompt_template::replace!(+ $crate::prompt_template::WIKILINKS,+ "{{sources}}",+ $sources+ );+ const FRAGMENT: &str = $crate::prompt_template::replace!(WITH_SOURCES, "{{basis}}", $basis);+ // Match the placeholder's own line ending so a CRLF checkout still expands.+ const LF: &str = $crate::prompt_template::replace!($template, "{{wikilinks}}\n", FRAGMENT);+ const OUT: &str = $crate::prompt_template::replace!(LF, "{{wikilinks}}\r\n", FRAGMENT);+ const _: () = assert!(+ $crate::prompt_template::count_str(OUT, "{{wikilinks}}") == 0+ && $crate::prompt_template::count_str($template, "{{wikilinks}}") == 1,+ "prompt must hold exactly one expanded wikilinks placeholder"+ );+ OUT+ }};+}+pub(crate) use {prompt_with_wikilinks, replace};++#[cfg(test)]+mod tests {+ const TPL_LF: &str = "a\n{{wikilinks}}\nb\n";+ const TPL_CRLF: &str = "a\r\n{{wikilinks}}\r\nb\r\n";++ #[test]+ fn expands_placeholder_for_lf_and_crlf_templates() {+ let lf = prompt_with_wikilinks!(TPL_LF, "the input", "the input");+ let crlf = prompt_with_wikilinks!(TPL_CRLF, "the input", "the input");+ for out in [lf, crlf] {+ assert!(+ !out.contains("{{wikilinks}}")+ && !out.contains("{{sources}}")+ && !out.contains("{{basis}}")+ );+ assert_eq!(out.matches("## WIKILINKS").count(), 1);+ }+ }++ #[test]+ fn shipped_prompts_have_no_leftover_placeholder() {+ for p in [+ crate::consolidator::BATCH_SYSTEM_PROMPT,+ crate::consolidator::SYSTEM_PROMPT,+ crate::bootstrap::SYSTEM_PROMPT,+ ] {+ assert!(+ !p.contains("{{wikilinks}}")+ && !p.contains("{{sources}}")+ && !p.contains("{{basis}}")+ );+ assert_eq!(p.matches("## WIKILINKS").count(), 1);+ }+ }++ #[test]+ fn per_prompt_wording_fills_the_slots() {+ let flat = |p: &str| p.split_whitespace().collect::<Vec<_>>().join(" ");+ for p in [+ crate::consolidator::BATCH_SYSTEM_PROMPT,+ crate::consolidator::SYSTEM_PROMPT,+ ] {+ let p = flat(p);+ assert!(+ p.contains("only if that exact path appears in the observations or page body.")+ );+ assert!(p.contains("same-project page the observations clearly relate to."));+ }+ let p = flat(crate::bootstrap::SYSTEM_PROMPT);+ assert!(p.contains("only if that exact path appears in the supplied sources."));+ assert!(p.contains("same-project page the sources clearly relate to."));+ }+}