Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(wiki): drop a leading BOM on a page with no frontmatter

Max Freedom Pollard54a933f0afea
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d88c7516..a0fdfbe6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -36,6 +36,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
per-event disposition runs. The install-time "enforced"/"NOT in force"
messaging is corrected to match: a script-fallback install is now the
only path flagged as unenforced. (#661)
+- A UTF-8 BOM on a hand-edited wiki page that has no frontmatter no longer
+ rides into the page body. `markdown::parse` stripped the mark before looking
+ for the frontmatter fence, but the no-frontmatter path returned the untouched
+ input as the body, so the mark stayed in front of the first line. Two things
+ followed from that: `derive_title` no longer read the leading `# ` as a
+ heading, so the page was indexed under its filename instead of its title, and
+ the OKF v0.2 file pass (which conforms frontmatter and leaves the body alone)
+ re-emitted the mark after the closing `---` fence, where it is a stray
+ zero-width no-break space rather than a byte-order mark. Both paths now drop
+ the leading BOM, which is what the frontmatter path already did. (#663)
## [2.1.0] - 2026-09-06
diff --git a/crates/ai-memory-wiki/src/markdown.rs b/crates/ai-memory-wiki/src/markdown.rs
index 5fa24735..837fe2c0 100644
--- a/crates/ai-memory-wiki/src/markdown.rs
+++ b/crates/ai-memory-wiki/src/markdown.rs
@@ -28,6 +28,12 @@ pub struct Markdown {
/// Recognises only the canonical `---\n<yaml>\n---\n` block at the very
/// start of the document. Anything else is treated as body.
///
+/// A leading UTF-8 BOM is dropped either way. It only means "this file is
+/// UTF-8" while it sits at offset zero; carried into `body` it is a
+/// zero-width no-break space in front of the first line, which hides an H1
+/// from [`derive_title`] and rides into the body a later re-emit writes
+/// back after the frontmatter fence.
+///
/// # Errors
/// Returns [`WikiError::Yaml`] if the frontmatter block exists but does
/// not parse as YAML.
@@ -47,7 +53,7 @@ pub fn parse(input: &str) -> WikiResult<Markdown> {
}
Ok(Markdown {
frontmatter: serde_json::Value::Null,
- body: input.to_string(),
+ body: trimmed.to_string(),
})
}
@@ -551,6 +557,26 @@ mod tests {
assert_eq!(md.body, "Body\n");
}
+ /// A page a Windows editor saved with a UTF-8 BOM and no frontmatter:
+ /// the mark belongs to the file, not to the first line. Left in `body`
+ /// it sits in front of the `#`, so the H1 stops being a heading and the
+ /// page is indexed under its filename instead of its title.
+ #[test]
+ fn parses_bom_prefixed_body_without_frontmatter() {
+ let src = "\u{FEFF}# Hand written\n\nBody.\n";
+ let md = parse(src).unwrap();
+ assert!(md.frontmatter.is_null());
+ assert_eq!(md.body, "# Hand written\n\nBody.\n");
+ assert_eq!(
+ derive_title(
+ &md.frontmatter,
+ &md.body,
+ &PagePath::new("notes/hand-written.md").unwrap(),
+ ),
+ "Hand written"
+ );
+ }
+
#[test]
fn malformed_frontmatter_returns_error() {
let src = "---\ntitle: [unterminated\n---\nBody\n";
diff --git a/crates/ai-memory-wiki/src/migrations/m2026_09_okf_conformance.rs b/crates/ai-memory-wiki/src/migrations/m2026_09_okf_conformance.rs
index eb252039..1429aae7 100644
--- a/crates/ai-memory-wiki/src/migrations/m2026_09_okf_conformance.rs
+++ b/crates/ai-memory-wiki/src/migrations/m2026_09_okf_conformance.rs
@@ -542,6 +542,34 @@ mod tests {
assert!(crate::backup::BackupReceipt::load(tmp.path()).is_none());
}
+ /// The file pass leaves the body untouched, and a UTF-8 BOM is not part
+ /// of the body: it means "this file is UTF-8" only at offset zero. A
+ /// hand-edited page a Windows editor saved with one and no frontmatter
+ /// used to come back through `parse` with the mark still on the front of
+ /// the body, so this pass wrote it out AFTER the frontmatter fence, where
+ /// it is a stray zero-width no-break space ahead of the H1.
+ #[test]
+ fn conforming_a_bom_prefixed_page_does_not_move_the_mark_into_the_body() {
+ let tmp = TempDir::new().unwrap();
+ let rel = Path::new("w/p/notes/hand-written.md");
+ let abs = tmp.path().join(rel);
+ std::fs::create_dir_all(abs.parent().unwrap()).unwrap();
+ std::fs::write(&abs, "\u{FEFF}# Hand written\n\nBody.\n").unwrap();
+
+ conform_file(tmp.path(), rel, &std::collections::HashMap::new()).unwrap();
+
+ let conformed = std::fs::read_to_string(&abs).unwrap();
+ assert!(
+ !conformed.contains('\u{FEFF}'),
+ "the mark survived the file pass: {conformed:?}"
+ );
+ assert!(
+ conformed.ends_with("---\n# Hand written\n\nBody.\n"),
+ "the body must reach disk exactly as it was authored: {conformed:?}"
+ );
+ assert_eq!(parse(&conformed).unwrap().frontmatter["type"], "Note");
+ }
+
// ---- #633: the safety archive must be taken BEFORE the DB migration ----
/// The core assertion for #633: the pre-open snapshot captures the DB as it