Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

fix: code strip honours fence length before wiki-link parsing (#140) (#144)

Jordan Gibbs54af04b02349
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d20c134..4ea4cfb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,7 @@
- **Note ids are fixed points of the slugifier, and the link validator rejects shell fragments and traversal (#126, from @fduple).** `write_note` built the filename from the raw id while the frontmatter id was slugified, and `repair` / `graph stub` pass broken-link text through as the id, so fetched content chose the path: `[[../../outside]]` wrote outside the vault, and `[[-n "$kernel"]]` crashed on Windows. Stub files now get slug filenames. `is_valid_wiki_link_target` also refuses targets containing `$ " \` = \ < > ;`, a leading `-` or `!`, or a `.`/`..` segment. Slug ids never contain those, but a title or alias link such as `[[E=mc2]]` that used to resolve by title will drop out of the links table on the next sync.
- **`note mv` keeps notes where sync can see them (#128, from @fduple).** The destination was joined onto the vault root verbatim, so a bare name landed at the root with no `.md` and left the index, `../x.md` wrote outside the vault, and an existing file was overwritten (POSIX) or crashed the command (Windows). A bare name now lands in `research/notes/` with `.md` added; destinations outside `research/notes/` and `research/temp/` are refused with `OUTSIDE_SYNCED_TREE`, and an existing file with `DESTINATION_EXISTS`. `research/index/` is refused too, since `repair` regenerates it and deletes every `.md` there. Case-only renames work on case-insensitive filesystems. The docstring no longer claims `mv` updates references; the id lives in frontmatter, so it never needed to.
- **The builtin provider keeps fetched code as markdown code (#129, from @fduple).** `<pre>` blocks and inline `<code>` were flattened to prose, so bash test syntax (`[[ -n "$k" ]]`) reached the wiki-link parser and `repair` minted stub notes named after shell fragments. `<pre>` now becomes a fenced block whose fence outruns any backtick run inside it, inline `<code>` becomes a backtick span, and indentation survives in the stdlib fallback. Because `strip_markdown` drops code before indexing, code on builtin-fetched pages is no longer in full-text search, embeddings, or the quote-integrity comparison, which is how crawl4ai-fetched pages already behaved. A report that quotes a sentence containing an inline identifier can now fail quote integrity where it passed before.
+- **Code stripping honours fence length (#140).** The strip that runs before wiki-link parsing closed a fenced block at the first ```` ``` ```` whatever the opening fence was, and an inline span at the first backtick. #129 fences fetched code with a run longer than any inside it, so a `<pre>` holding an odd number of ```` ``` ```` leaked its `[[ ]]` into the link parser. `CODE_BLOCK_RE` and `INLINE_CODE_RE` are replaced by `strip_code()`, a linear scanner that follows CommonMark: a fence opens on a line starting with 3+ backticks and closes on a line holding only at least as many; an opener nothing later can close is literal text (CommonMark would run it to the end of the note, which let a prose line starting with ```` ``` ```` drop the rest from links and search); an inline span of N backticks closes only at the next run of exactly N, and an unmatched run is literal. Stripped code now leaves its newlines behind, so the `links` table's line numbers match the note. The builtin provider's inline `<code>` delimiter also outgrows the backtick runs inside it, which the fixed ```` `` ```` did not.
- **Skills name the CLI by its resolved path (#133, reported by @blrain3).** 14 of the 19 installed skills spelled the CLI `$HPR`, which nothing defined, so `$HPR run finish <tag>` (the ship gate) ran as `run finish` and the shell answered `command not found`. Skills now carry the `{hpr_path}` placeholder the agent prompts already used, and `install` (including `install --steps-only`, the entry skill's bootstrap) fills it with the same absolute path CLAUDE.md shows. The corpus-critic step had one literal `{hpr_path}` that was never substituted; it resolves now too. Re-run `hyperresearch install` to pick this up.
- **The open-access URL gate uses the SSRF gate's address rules (#138).** `check_oa_url` classified resolved addresses with `is_global`, which on Python 3.11 passes IPv4-mapped and 6to4-wrapped CGNAT (`::ffff:100.64.0.1`, `2002:6440:1::`) — the addresses #127 closed on the fetch path. It now calls `safe_http.check_url` after its own credential and bare-hostname checks, and ignores `allow_private_hosts`. Europe PMC JATS downloads now go through `safe_get`, so a redirect to a private host is refused instead of followed.
- **Cite-check samples at the configured rate.** `sample_needs_llm` kept every `round(1 / rate)`-th weak pair, so the default 0.6 sampled 50% and 0.3 sampled 33%. It now keeps exactly `rate` of them, spread evenly, still without an RNG.
diff --git a/src/hyperresearch/core/note.py b/src/hyperresearch/core/note.py
index 5a68f4c..95b4aaa 100644
--- a/src/hyperresearch/core/note.py
+++ b/src/hyperresearch/core/note.py
@@ -9,10 +9,9 @@ from pathlib import Path
from hyperresearch.core.frontmatter import parse_frontmatter, render_note
from hyperresearch.core.patterns import (
- CODE_BLOCK_RE,
- INLINE_CODE_RE,
WIKI_LINK_RE,
is_valid_wiki_link_target,
+ strip_code,
)
from hyperresearch.models.note import Note, NoteMeta, slugify
@@ -46,8 +45,7 @@ def read_note(file_path: Path, vault_root: Path) -> Note:
meta.id = slugify(file_path.stem)
# Extract outgoing wiki links, filtering citation footnotes and URLs
- cleaned = CODE_BLOCK_RE.sub("", body)
- cleaned = INLINE_CODE_RE.sub("", cleaned)
+ cleaned = strip_code(body)
raw_links = (m.group(1).strip().rstrip("\\") for m in WIKI_LINK_RE.finditer(cleaned))
outgoing = list(dict.fromkeys(
ref for ref in raw_links if is_valid_wiki_link_target(ref)
@@ -157,8 +155,7 @@ def write_note(
def strip_markdown(text: str) -> str:
"""Strip markdown formatting to plain text for FTS indexing."""
- text = CODE_BLOCK_RE.sub("", text)
- text = INLINE_CODE_RE.sub("", text)
+ text = strip_code(text)
text = re.sub(r"^#{1,6}\s+", "", text, flags=re.MULTILINE)
text = re.sub(r"\*{1,3}([^*]+)\*{1,3}", r"\1", text)
text = re.sub(r"_{1,3}([^_]+)_{1,3}", r"\1", text)
diff --git a/src/hyperresearch/core/patterns.py b/src/hyperresearch/core/patterns.py
index 4612d02..8622c5a 100644
--- a/src/hyperresearch/core/patterns.py
+++ b/src/hyperresearch/core/patterns.py
@@ -23,9 +23,124 @@ import re
# is_valid_wiki_link_target's bracket-balance check; now it never matches.
WIKI_LINK_RE = re.compile(r"\[\[([^\]\[|]+)(?:\|[^\]\[]+)?\]\](?!\()")
-# Code block patterns for stripping before link extraction
-CODE_BLOCK_RE = re.compile(r"```.*?```", re.DOTALL)
-INLINE_CODE_RE = re.compile(r"`[^`]+`")
+# --- Code stripping before link extraction and indexing ---------------------
+#
+# Code must be removed before WIKI_LINK_RE runs: bash test syntax
+# (`[[ -n "$k" ]]`) is lexically a wiki-link. This used to be two regexes,
+# ```.*?``` and `[^`]+`, which closed every span at the first backtick run
+# of any length. The builtin provider (#129) fences code with a run longer
+# than any inside it, as CommonMark requires, so code holding an odd number
+# of ``` ended the block early and leaked its [[ ]] into the parser (#140).
+#
+# The rules follow CommonMark where it matters here:
+#
+# - A fenced block opens on a line whose first non-blank characters are 3+
+# backticks with no backtick after them, and closes on a line holding
+# only a run of AT LEAST that many backticks. An opener with no such line
+# after it is literal text, not a block to the end of the note: CommonMark
+# would run it to the end, but a prose line that happens to start with ```
+# would then drop the rest of the note from links and search. The builtin
+# provider always closes its fences, so #140 does not depend on the
+# unclosed case. Any indentation is accepted (fences inside list items).
+# Tilde fences are not recognised, as before.
+# - An inline span opened by N backticks closes at the next run of EXACTLY
+# N backticks; an unmatched run is literal text. Spans may cross lines,
+# as the old regex allowed. Backtick runs that are not fences (mid-line
+# ```x```, a line-start run followed by more backticks) are inline runs.
+#
+# Fetched bodies are attacker-controlled and this runs on every sync, so
+# it is a scanner, not a regex: a backreference regex re-scans to the end
+# of the text for every unclosed run and goes superlinear on backtick
+# floods. Both passes below are linear.
+#
+# Stripped code is replaced by the newlines it held (a space for a span
+# on one line), so sync's per-line link numbers stay true and text on
+# either side of a span cannot fuse into a new `[[`.
+
+_BACKTICK_RUN_RE = re.compile(r"`+")
+
+
+def _strip_fenced_blocks(text: str) -> str:
+ lines = text.split("\n")
+ runs: list[int] = []
+ closers: list[int] = [] # run length if the line could close a fence, else 0
+ for line in lines:
+ s = line.lstrip(" \t")
+ run = len(s) - len(s.lstrip("`"))
+ runs.append(run)
+ closers.append(run if run >= 3 and not s[run:].strip(" \t\r") else 0)
+ # longest_after[i]: the longest closer on any line after line i. An opener
+ # whose run nothing after it can match stays literal text, so each check
+ # is O(1) and the pass stays linear.
+ longest_after = [0] * len(lines)
+ best = 0
+ for i in range(len(lines) - 1, -1, -1):
+ longest_after[i] = best
+ best = max(best, closers[i])
+
+ out: list[str] = []
+ fence = 0 # length of the open fence's backtick run; 0 = not in a block
+ for i, line in enumerate(lines):
+ run = runs[i]
+ if fence:
+ if closers[i] >= fence:
+ fence = 0
+ out.append("")
+ elif (
+ run >= 3
+ and "`" not in line.lstrip(" \t")[run:]
+ and longest_after[i] >= run
+ ):
+ fence = run
+ out.append("")
+ else:
+ out.append(line)
+ return "\n".join(out)
+
+
+def _strip_inline_code(text: str) -> str:
+ runs = [m.span() for m in _BACKTICK_RUN_RE.finditer(text)]
+ if len(runs) < 2:
+ return text
+ # Run indices grouped by length; each group's cursor only moves forward,
+ # so finding every closer costs O(runs) in total.
+ by_len: dict[int, list[int]] = {}
+ for i, (a, b) in enumerate(runs):
+ by_len.setdefault(b - a, []).append(i)
+ cursor = dict.fromkeys(by_len, 0)
+
+ out: list[str] = []
+ pos = 0
+ i = 0
+ while i < len(runs):
+ a, b = runs[i]
+ n = b - a
+ group = by_len[n]
+ k = cursor[n]
+ while k < len(group) and group[k] <= i:
+ k += 1
+ cursor[n] = k
+ if k == len(group):
+ i += 1 # no closer: the run is literal text
+ continue
+ j = group[k]
+ end = runs[j][1]
+ newlines = text.count("\n", a, end)
+ out.append(text[pos:a])
+ out.append("\n" * newlines if newlines else " ")
+ pos = end
+ i = j + 1
+ out.append(text[pos:])
+ return "".join(out)
+
+
+def strip_code(text: str) -> str:
+ """Remove fenced code blocks, then inline code spans, from markdown.
+
+ Run this before WIKI_LINK_RE so code is never parsed as links. Linear in
+ the length of the text, including on backtick floods.
+ """
+ return _strip_inline_code(_strip_fenced_blocks(text))
# Citation-footnote patterns that should NOT be treated as wiki-links even
diff --git a/src/hyperresearch/core/sync.py b/src/hyperresearch/core/sync.py
index 0e91ef9..fbadb74 100644
--- a/src/hyperresearch/core/sync.py
+++ b/src/hyperresearch/core/sync.py
@@ -12,10 +12,9 @@ from pathlib import Path
from hyperresearch.core.note import read_note, strip_markdown
from hyperresearch.core.patterns import (
- CODE_BLOCK_RE,
- INLINE_CODE_RE,
WIKI_LINK_RE,
is_valid_wiki_link_target,
+ strip_code,
)
@@ -316,8 +315,7 @@ def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> Non
# Update links — extract from original body (not stripped)
# Uses the shared filter so this path stays in sync with core.note.read_note
conn.execute("DELETE FROM links WHERE source_id = ?", (meta.id,))
- cleaned = CODE_BLOCK_RE.sub("", note.body)
- cleaned = INLINE_CODE_RE.sub("", cleaned)
+ cleaned = strip_code(note.body)
for line_num, line in enumerate(cleaned.split("\n"), 1):
for m in WIKI_LINK_RE.finditer(line):
target_ref = m.group(1).strip().rstrip("\\") # Strip trailing backslash (shell escaping artifact)
diff --git a/src/hyperresearch/web/builtin.py b/src/hyperresearch/web/builtin.py
index b981eb8..0ec3a9b 100644
--- a/src/hyperresearch/web/builtin.py
+++ b/src/hyperresearch/web/builtin.py
@@ -29,10 +29,13 @@ from hyperresearch.web.pdf import (
# Both extraction branches below share these helpers.
+def _longest_backtick_run(text: str) -> int:
+ return max((len(m.group(0)) for m in re.finditer(r"`+", text)), default=0)
+
+
def _fence_for(code: str) -> str:
"""Return a backtick fence longer than any backtick run inside the code."""
- longest = max((len(m.group(0)) for m in re.finditer(r"`+", code)), default=0)
- return "`" * max(3, longest + 1)
+ return "`" * max(3, _longest_backtick_run(code) + 1)
def _fenced(code: str) -> str:
@@ -43,9 +46,15 @@ def _fenced(code: str) -> str:
def _inline_code(text: str) -> str:
- """Wrap inline code in backticks (doubled when the text contains one)."""
+ """Wrap inline code in a backtick run no run inside the text can close.
+
+ A span of N backticks closes at the next run of exactly N, so the
+ delimiter must outrun every run inside; a fixed ```` `` ```` let text
+ holding ```` `` ```` end the span early and leak what followed (#140).
+ """
if "`" in text:
- return f"`` {text} ``"
+ ticks = "`" * (_longest_backtick_run(text) + 1)
+ return f"{ticks} {text} {ticks}"
return f"`{text}`"
diff --git a/tests/test_core/test_patterns.py b/tests/test_core/test_patterns.py
index 9c771be..fb55fea 100644
--- a/tests/test_core/test_patterns.py
+++ b/tests/test_core/test_patterns.py
@@ -1,6 +1,6 @@
"""Tests for wiki-link target validation — citation-footnote edge cases."""
-from hyperresearch.core.patterns import is_valid_wiki_link_target
+from hyperresearch.core.patterns import is_valid_wiki_link_target, strip_code
def test_empty_and_whitespace_rejected():
@@ -224,6 +224,30 @@ def test_template_placeholder_check_is_linear():
assert time.perf_counter() - started < 1.0
+def test_code_strip_is_linear_on_backtick_floods():
+ # Runs on every fetched body before link extraction (#140). A
+ # backreference regex matching closing fences to opening ones re-scans
+ # to the end of the text for every unclosed run: the staircase of
+ # distinct, never-closed run lengths is the worst case for that shape.
+ import time
+
+ floods = {
+ "`": "`" * 300_000,
+ "```\\n": "```\n" * 75_000,
+ "`a": "`a" * 150_000,
+ "``` ": "``` " * 75_000,
+ "unclosed ````": "````\n" + "```\n" * 75_000,
+ "never-closed openers": "```x\n" * 75_000,
+ "descending openers": "".join("`" * n + "\n" for n in range(1500, 2, -1)),
+ "staircase": "".join("`" * n + "a" for n in range(1500, 0, -1)),
+ }
+ for name, flood in floods.items():
+ started = time.perf_counter()
+ strip_code(flood)
+ elapsed = time.perf_counter() - started
+ assert elapsed < 1.0, f"{name!r} flood took {elapsed:.2f}s"
+
+
def test_interior_open_bracket_never_matches():
# Previously matched as target `t.IO[t.Any` and was rejected downstream
# by the bracket-balance check; now the pattern itself refuses it.
diff --git a/tests/test_graph/test_links.py b/tests/test_graph/test_links.py
index 942aaa4..14fb37a 100644
--- a/tests/test_graph/test_links.py
+++ b/tests/test_graph/test_links.py
@@ -1,6 +1,6 @@
"""Tests for link parsing and graph operations."""
-from hyperresearch.core.note import CODE_BLOCK_RE, INLINE_CODE_RE, WIKI_LINK_RE
+from hyperresearch.core.patterns import WIKI_LINK_RE, strip_code
def test_wiki_link_regex_simple():
@@ -25,7 +25,7 @@ def test_wiki_link_regex_multiple():
def test_wiki_link_not_in_code_block():
text = "```python\n[[not-a-link]]\n```\n\n[[real-link]]"
- cleaned = CODE_BLOCK_RE.sub("", text)
+ cleaned = strip_code(text)
matches = WIKI_LINK_RE.findall(cleaned)
assert len(matches) == 1
assert "real-link" in matches[0]
@@ -33,12 +33,100 @@ def test_wiki_link_not_in_code_block():
def test_wiki_link_not_in_inline_code():
text = "Use `[[not-a-link]]` but also [[real-link]]"
- cleaned = INLINE_CODE_RE.sub("", text)
+ cleaned = strip_code(text)
matches = WIKI_LINK_RE.findall(cleaned)
assert len(matches) == 1
assert "real-link" in matches[0]
+def _links(text: str) -> list[str]:
+ return WIKI_LINK_RE.findall(strip_code(text))
+
+
+# --- Fence length is honoured (#140) ------------------------------------------
+
+
+def test_issue_140_odd_inner_fence_does_not_leak():
+ # The #129 builtin provider fences a <pre> holding ``` with ````.
+ text = "````\n```\n[[ -n y ]]\n````\n\n[[real-link]]"
+ assert _links(text) == ["real-link"]
+
+
+def test_longer_outer_fence_holds_inner_fenced_block():
+ text = (
+ "`````markdown\n"
+ "```bash\n[[ -f x ]]\n```\n"
+ "````\n[[inside-too]]\n"
+ "`````\n"
+ "after [[real-link]]"
+ )
+ assert _links(text) == ["real-link"]
+
+
+def test_closing_fence_may_be_longer_than_opening():
+ text = "```\n[[code]]\n`````\n[[real-link]]"
+ assert _links(text) == ["real-link"]
+
+
+def test_fence_line_with_trailing_text_does_not_close():
+ text = "```\n``` not a closer [[code]]\n```\n[[real-link]]"
+ assert _links(text) == ["real-link"]
+
+
+def test_indented_fence_in_list_item():
+ text = "- step:\n ```bash\n [[ -n y ]]\n ```\n- see [[real-link]]"
+ assert _links(text) == ["real-link"]
+
+
+def test_unclosed_fence_is_literal_text():
+ # An opener nothing later can close is text, not a block to the end of
+ # the note: a prose line starting with ``` must not drop what follows.
+ text = "[[before]]\n```\n[[kept]]\n[[also-kept]]"
+ assert _links(text) == ["before", "kept", "also-kept"]
+
+
+def test_prose_line_starting_with_backticks_keeps_the_rest():
+ text = "Use\n``` to open a fence\nand then [[real-link]] later"
+ assert _links(text) == ["real-link"]
+
+
+def test_unclosed_long_fence_does_not_hide_a_later_block():
+ text = "`````\n[[shown]]\n```\n[[hidden]]\n```\n[[after]]"
+ assert _links(text) == ["shown", "after"]
+
+
+def test_double_backtick_span_holding_single_backtick():
+ text = "Run `` echo `x` [[x]] `` then read [[real-link]]"
+ assert _links(text) == ["real-link"]
+
+
+def test_inline_span_closes_only_on_equal_length_run():
+ text = "`a``[[x]]` and ``b`[[y]]`` and [[real-link]]"
+ assert _links(text) == ["real-link"]
+
+
+def test_mid_line_triple_backticks_are_an_inline_span():
+ # Not a fence (not at line start), so it is an inline span of length 3,
+ # stripped as the old regex did.
+ text = "Use ```[[x]]``` or [[real-link]]"
+ assert _links(text) == ["real-link"]
+
+
+def test_unmatched_backtick_run_is_literal():
+ text = "a stray ` backtick, then [[real-link]] and ``` more [[other]]"
+ assert _links(text) == ["real-link", "other"]
+
+
+def test_stripped_span_cannot_fuse_brackets():
+ assert _links("[`x`[y]]") == []
+
+
+def test_strip_keeps_line_numbers():
+ text = "a\n```\none\ntwo\n```\n`multi\nline`\n[[real-link]]"
+ cleaned = strip_code(text)
+ assert cleaned.count("\n") == text.count("\n")
+ assert cleaned.split("\n")[7] == "[[real-link]]"
+
def test_backlinks_populated(seeded_vault):
"""Concurrency links to python-async-patterns, so python-async-patterns should have a backlink."""
rows = seeded_vault.db.execute(
diff --git a/tests/test_web/test_builtin_fences.py b/tests/test_web/test_builtin_fences.py
index 215d55b..6b9b548 100644
--- a/tests/test_web/test_builtin_fences.py
+++ b/tests/test_web/test_builtin_fences.py
@@ -11,10 +11,9 @@ and the stdlib fallback.
import pytest
from hyperresearch.core.patterns import (
- CODE_BLOCK_RE,
- INLINE_CODE_RE,
WIKI_LINK_RE,
is_valid_wiki_link_target,
+ strip_code,
)
from hyperresearch.web.builtin import BuiltinProvider, _TextExtractor
from hyperresearch.web.pdf import PDF_FAILURE_KEY
@@ -39,8 +38,7 @@ kernel="$1"
def _surviving_links(text: str) -> list[str]:
"""The exact strip-then-extract sequence core/note.py runs on a note body."""
- cleaned = CODE_BLOCK_RE.sub("", text)
- cleaned = INLINE_CODE_RE.sub("", cleaned)
+ cleaned = strip_code(text)
raw = (m.group(1).strip().rstrip("\\") for m in WIKI_LINK_RE.finditer(cleaned))
return [ref for ref in raw if is_valid_wiki_link_target(ref)]
@@ -77,6 +75,21 @@ class _BranchContract:
assert "````" in text
assert _surviving_links(text) == []
+ def test_odd_backtick_fence_inside_pre_does_not_leak(self):
+ # #140: the fence outgrows the ``` inside, and the strip must honour
+ # that length instead of closing at the inner ```. The second target
+ # passes the link validator, so only the strip stands in its way.
+ html = "<html><body><pre>```\n[[ -n y ]]\n[[looks-like-a-note]]</pre></body></html>"
+ _, text = self.extract(html)
+ assert "````" in text
+ assert WIKI_LINK_RE.findall(strip_code(text)) == []
+
+ def test_inline_code_holding_double_backticks_does_not_leak(self):
+ html = "<html><body><p>Try <code>a``b [[looks-like-a-note]]</code> now.</p></body></html>"
+ _, text = self.extract(html)
+ assert "``` a``b [[looks-like-a-note]] ```" in text
+ assert WIKI_LINK_RE.findall(strip_code(text)) == []
+
def test_title_and_noise_removal_unchanged(self):
html = (
"<html><head><title>T</title></head><body>"