Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
fix: code strip honours fence length before wiki-link parsing (#140) (#144)
Jordan Gibbs54af04b02349
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d20c134..4ea4cfb 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,6 +12,7 @@- **Note ids are fixed points of the slugifier, and the link validator rejects shell fragments and traversal (#126, from @fduple).** `write_note` built the filename from the raw id while the frontmatter id was slugified, and `repair` / `graph stub` pass broken-link text through as the id, so fetched content chose the path: `[[../../outside]]` wrote outside the vault, and `[[-n "$kernel"]]` crashed on Windows. Stub files now get slug filenames. `is_valid_wiki_link_target` also refuses targets containing `$ " \` = \ < > ;`, a leading `-` or `!`, or a `.`/`..` segment. Slug ids never contain those, but a title or alias link such as `[[E=mc2]]` that used to resolve by title will drop out of the links table on the next sync.- **`note mv` keeps notes where sync can see them (#128, from @fduple).** The destination was joined onto the vault root verbatim, so a bare name landed at the root with no `.md` and left the index, `../x.md` wrote outside the vault, and an existing file was overwritten (POSIX) or crashed the command (Windows). A bare name now lands in `research/notes/` with `.md` added; destinations outside `research/notes/` and `research/temp/` are refused with `OUTSIDE_SYNCED_TREE`, and an existing file with `DESTINATION_EXISTS`. `research/index/` is refused too, since `repair` regenerates it and deletes every `.md` there. Case-only renames work on case-insensitive filesystems. The docstring no longer claims `mv` updates references; the id lives in frontmatter, so it never needed to.- **The builtin provider keeps fetched code as markdown code (#129, from @fduple).** `<pre>` blocks and inline `<code>` were flattened to prose, so bash test syntax (`[[ -n "$k" ]]`) reached the wiki-link parser and `repair` minted stub notes named after shell fragments. `<pre>` now becomes a fenced block whose fence outruns any backtick run inside it, inline `<code>` becomes a backtick span, and indentation survives in the stdlib fallback. Because `strip_markdown` drops code before indexing, code on builtin-fetched pages is no longer in full-text search, embeddings, or the quote-integrity comparison, which is how crawl4ai-fetched pages already behaved. A report that quotes a sentence containing an inline identifier can now fail quote integrity where it passed before.+- **Code stripping honours fence length (#140).** The strip that runs before wiki-link parsing closed a fenced block at the first ```` ``` ```` whatever the opening fence was, and an inline span at the first backtick. #129 fences fetched code with a run longer than any inside it, so a `<pre>` holding an odd number of ```` ``` ```` leaked its `[[ ]]` into the link parser. `CODE_BLOCK_RE` and `INLINE_CODE_RE` are replaced by `strip_code()`, a linear scanner that follows CommonMark: a fence opens on a line starting with 3+ backticks and closes on a line holding only at least as many; an opener nothing later can close is literal text (CommonMark would run it to the end of the note, which let a prose line starting with ```` ``` ```` drop the rest from links and search); an inline span of N backticks closes only at the next run of exactly N, and an unmatched run is literal. Stripped code now leaves its newlines behind, so the `links` table's line numbers match the note. The builtin provider's inline `<code>` delimiter also outgrows the backtick runs inside it, which the fixed ```` `` ```` did not.- **Skills name the CLI by its resolved path (#133, reported by @blrain3).** 14 of the 19 installed skills spelled the CLI `$HPR`, which nothing defined, so `$HPR run finish <tag>` (the ship gate) ran as `run finish` and the shell answered `command not found`. Skills now carry the `{hpr_path}` placeholder the agent prompts already used, and `install` (including `install --steps-only`, the entry skill's bootstrap) fills it with the same absolute path CLAUDE.md shows. The corpus-critic step had one literal `{hpr_path}` that was never substituted; it resolves now too. Re-run `hyperresearch install` to pick this up.- **The open-access URL gate uses the SSRF gate's address rules (#138).** `check_oa_url` classified resolved addresses with `is_global`, which on Python 3.11 passes IPv4-mapped and 6to4-wrapped CGNAT (`::ffff:100.64.0.1`, `2002:6440:1::`) — the addresses #127 closed on the fetch path. It now calls `safe_http.check_url` after its own credential and bare-hostname checks, and ignores `allow_private_hosts`. Europe PMC JATS downloads now go through `safe_get`, so a redirect to a private host is refused instead of followed.- **Cite-check samples at the configured rate.** `sample_needs_llm` kept every `round(1 / rate)`-th weak pair, so the default 0.6 sampled 50% and 0.3 sampled 33%. It now keeps exactly `rate` of them, spread evenly, still without an RNG.diff --git a/src/hyperresearch/core/note.py b/src/hyperresearch/core/note.pyindex 5a68f4c..95b4aaa 100644--- a/src/hyperresearch/core/note.py+++ b/src/hyperresearch/core/note.py@@ -9,10 +9,9 @@ from pathlib import Pathfrom hyperresearch.core.frontmatter import parse_frontmatter, render_notefrom hyperresearch.core.patterns import (- CODE_BLOCK_RE,- INLINE_CODE_RE,WIKI_LINK_RE,is_valid_wiki_link_target,+ strip_code,)from hyperresearch.models.note import Note, NoteMeta, slugify@@ -46,8 +45,7 @@ def read_note(file_path: Path, vault_root: Path) -> Note:meta.id = slugify(file_path.stem)# Extract outgoing wiki links, filtering citation footnotes and URLs- cleaned = CODE_BLOCK_RE.sub("", body)- cleaned = INLINE_CODE_RE.sub("", cleaned)+ cleaned = strip_code(body)raw_links = (m.group(1).strip().rstrip("\\") for m in WIKI_LINK_RE.finditer(cleaned))outgoing = list(dict.fromkeys(ref for ref in raw_links if is_valid_wiki_link_target(ref)@@ -157,8 +155,7 @@ def write_note(def strip_markdown(text: str) -> str:"""Strip markdown formatting to plain text for FTS indexing."""- text = CODE_BLOCK_RE.sub("", text)- text = INLINE_CODE_RE.sub("", text)+ text = strip_code(text)text = re.sub(r"^#{1,6}\s+", "", text, flags=re.MULTILINE)text = re.sub(r"\*{1,3}([^*]+)\*{1,3}", r"\1", text)text = re.sub(r"_{1,3}([^_]+)_{1,3}", r"\1", text)diff --git a/src/hyperresearch/core/patterns.py b/src/hyperresearch/core/patterns.pyindex 4612d02..8622c5a 100644--- a/src/hyperresearch/core/patterns.py+++ b/src/hyperresearch/core/patterns.py@@ -23,9 +23,124 @@ import re# is_valid_wiki_link_target's bracket-balance check; now it never matches.WIKI_LINK_RE = re.compile(r"\[\[([^\]\[|]+)(?:\|[^\]\[]+)?\]\](?!\()")-# Code block patterns for stripping before link extraction-CODE_BLOCK_RE = re.compile(r"```.*?```", re.DOTALL)-INLINE_CODE_RE = re.compile(r"`[^`]+`")+# --- Code stripping before link extraction and indexing ---------------------+#+# Code must be removed before WIKI_LINK_RE runs: bash test syntax+# (`[[ -n "$k" ]]`) is lexically a wiki-link. This used to be two regexes,+# ```.*?``` and `[^`]+`, which closed every span at the first backtick run+# of any length. The builtin provider (#129) fences code with a run longer+# than any inside it, as CommonMark requires, so code holding an odd number+# of ``` ended the block early and leaked its [[ ]] into the parser (#140).+#+# The rules follow CommonMark where it matters here:+#+# - A fenced block opens on a line whose first non-blank characters are 3++# backticks with no backtick after them, and closes on a line holding+# only a run of AT LEAST that many backticks. An opener with no such line+# after it is literal text, not a block to the end of the note: CommonMark+# would run it to the end, but a prose line that happens to start with ```+# would then drop the rest of the note from links and search. The builtin+# provider always closes its fences, so #140 does not depend on the+# unclosed case. Any indentation is accepted (fences inside list items).+# Tilde fences are not recognised, as before.+# - An inline span opened by N backticks closes at the next run of EXACTLY+# N backticks; an unmatched run is literal text. Spans may cross lines,+# as the old regex allowed. Backtick runs that are not fences (mid-line+# ```x```, a line-start run followed by more backticks) are inline runs.+#+# Fetched bodies are attacker-controlled and this runs on every sync, so+# it is a scanner, not a regex: a backreference regex re-scans to the end+# of the text for every unclosed run and goes superlinear on backtick+# floods. Both passes below are linear.+#+# Stripped code is replaced by the newlines it held (a space for a span+# on one line), so sync's per-line link numbers stay true and text on+# either side of a span cannot fuse into a new `[[`.++_BACKTICK_RUN_RE = re.compile(r"`+")+++def _strip_fenced_blocks(text: str) -> str:+ lines = text.split("\n")+ runs: list[int] = []+ closers: list[int] = [] # run length if the line could close a fence, else 0+ for line in lines:+ s = line.lstrip(" \t")+ run = len(s) - len(s.lstrip("`"))+ runs.append(run)+ closers.append(run if run >= 3 and not s[run:].strip(" \t\r") else 0)+ # longest_after[i]: the longest closer on any line after line i. An opener+ # whose run nothing after it can match stays literal text, so each check+ # is O(1) and the pass stays linear.+ longest_after = [0] * len(lines)+ best = 0+ for i in range(len(lines) - 1, -1, -1):+ longest_after[i] = best+ best = max(best, closers[i])++ out: list[str] = []+ fence = 0 # length of the open fence's backtick run; 0 = not in a block+ for i, line in enumerate(lines):+ run = runs[i]+ if fence:+ if closers[i] >= fence:+ fence = 0+ out.append("")+ elif (+ run >= 3+ and "`" not in line.lstrip(" \t")[run:]+ and longest_after[i] >= run+ ):+ fence = run+ out.append("")+ else:+ out.append(line)+ return "\n".join(out)+++def _strip_inline_code(text: str) -> str:+ runs = [m.span() for m in _BACKTICK_RUN_RE.finditer(text)]+ if len(runs) < 2:+ return text+ # Run indices grouped by length; each group's cursor only moves forward,+ # so finding every closer costs O(runs) in total.+ by_len: dict[int, list[int]] = {}+ for i, (a, b) in enumerate(runs):+ by_len.setdefault(b - a, []).append(i)+ cursor = dict.fromkeys(by_len, 0)++ out: list[str] = []+ pos = 0+ i = 0+ while i < len(runs):+ a, b = runs[i]+ n = b - a+ group = by_len[n]+ k = cursor[n]+ while k < len(group) and group[k] <= i:+ k += 1+ cursor[n] = k+ if k == len(group):+ i += 1 # no closer: the run is literal text+ continue+ j = group[k]+ end = runs[j][1]+ newlines = text.count("\n", a, end)+ out.append(text[pos:a])+ out.append("\n" * newlines if newlines else " ")+ pos = end+ i = j + 1+ out.append(text[pos:])+ return "".join(out)+++def strip_code(text: str) -> str:+ """Remove fenced code blocks, then inline code spans, from markdown.++ Run this before WIKI_LINK_RE so code is never parsed as links. Linear in+ the length of the text, including on backtick floods.+ """+ return _strip_inline_code(_strip_fenced_blocks(text))# Citation-footnote patterns that should NOT be treated as wiki-links evendiff --git a/src/hyperresearch/core/sync.py b/src/hyperresearch/core/sync.pyindex 0e91ef9..fbadb74 100644--- a/src/hyperresearch/core/sync.py+++ b/src/hyperresearch/core/sync.py@@ -12,10 +12,9 @@ from pathlib import Pathfrom hyperresearch.core.note import read_note, strip_markdownfrom hyperresearch.core.patterns import (- CODE_BLOCK_RE,- INLINE_CODE_RE,WIKI_LINK_RE,is_valid_wiki_link_target,+ strip_code,)@@ -316,8 +315,7 @@ def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> Non# Update links — extract from original body (not stripped)# Uses the shared filter so this path stays in sync with core.note.read_noteconn.execute("DELETE FROM links WHERE source_id = ?", (meta.id,))- cleaned = CODE_BLOCK_RE.sub("", note.body)- cleaned = INLINE_CODE_RE.sub("", cleaned)+ cleaned = strip_code(note.body)for line_num, line in enumerate(cleaned.split("\n"), 1):for m in WIKI_LINK_RE.finditer(line):target_ref = m.group(1).strip().rstrip("\\") # Strip trailing backslash (shell escaping artifact)diff --git a/src/hyperresearch/web/builtin.py b/src/hyperresearch/web/builtin.pyindex b981eb8..0ec3a9b 100644--- a/src/hyperresearch/web/builtin.py+++ b/src/hyperresearch/web/builtin.py@@ -29,10 +29,13 @@ from hyperresearch.web.pdf import (# Both extraction branches below share these helpers.+def _longest_backtick_run(text: str) -> int:+ return max((len(m.group(0)) for m in re.finditer(r"`+", text)), default=0)++def _fence_for(code: str) -> str:"""Return a backtick fence longer than any backtick run inside the code."""- longest = max((len(m.group(0)) for m in re.finditer(r"`+", code)), default=0)- return "`" * max(3, longest + 1)+ return "`" * max(3, _longest_backtick_run(code) + 1)def _fenced(code: str) -> str:@@ -43,9 +46,15 @@ def _fenced(code: str) -> str:def _inline_code(text: str) -> str:- """Wrap inline code in backticks (doubled when the text contains one)."""+ """Wrap inline code in a backtick run no run inside the text can close.++ A span of N backticks closes at the next run of exactly N, so the+ delimiter must outrun every run inside; a fixed ```` `` ```` let text+ holding ```` `` ```` end the span early and leak what followed (#140).+ """if "`" in text:- return f"`` {text} ``"+ ticks = "`" * (_longest_backtick_run(text) + 1)+ return f"{ticks} {text} {ticks}"return f"`{text}`"diff --git a/tests/test_core/test_patterns.py b/tests/test_core/test_patterns.pyindex 9c771be..fb55fea 100644--- a/tests/test_core/test_patterns.py+++ b/tests/test_core/test_patterns.py@@ -1,6 +1,6 @@"""Tests for wiki-link target validation — citation-footnote edge cases."""-from hyperresearch.core.patterns import is_valid_wiki_link_target+from hyperresearch.core.patterns import is_valid_wiki_link_target, strip_codedef test_empty_and_whitespace_rejected():@@ -224,6 +224,30 @@ def test_template_placeholder_check_is_linear():assert time.perf_counter() - started < 1.0+def test_code_strip_is_linear_on_backtick_floods():+ # Runs on every fetched body before link extraction (#140). A+ # backreference regex matching closing fences to opening ones re-scans+ # to the end of the text for every unclosed run: the staircase of+ # distinct, never-closed run lengths is the worst case for that shape.+ import time++ floods = {+ "`": "`" * 300_000,+ "```\\n": "```\n" * 75_000,+ "`a": "`a" * 150_000,+ "``` ": "``` " * 75_000,+ "unclosed ````": "````\n" + "```\n" * 75_000,+ "never-closed openers": "```x\n" * 75_000,+ "descending openers": "".join("`" * n + "\n" for n in range(1500, 2, -1)),+ "staircase": "".join("`" * n + "a" for n in range(1500, 0, -1)),+ }+ for name, flood in floods.items():+ started = time.perf_counter()+ strip_code(flood)+ elapsed = time.perf_counter() - started+ assert elapsed < 1.0, f"{name!r} flood took {elapsed:.2f}s"++def test_interior_open_bracket_never_matches():# Previously matched as target `t.IO[t.Any` and was rejected downstream# by the bracket-balance check; now the pattern itself refuses it.diff --git a/tests/test_graph/test_links.py b/tests/test_graph/test_links.pyindex 942aaa4..14fb37a 100644--- a/tests/test_graph/test_links.py+++ b/tests/test_graph/test_links.py@@ -1,6 +1,6 @@"""Tests for link parsing and graph operations."""-from hyperresearch.core.note import CODE_BLOCK_RE, INLINE_CODE_RE, WIKI_LINK_RE+from hyperresearch.core.patterns import WIKI_LINK_RE, strip_codedef test_wiki_link_regex_simple():@@ -25,7 +25,7 @@ def test_wiki_link_regex_multiple():def test_wiki_link_not_in_code_block():text = "```python\n[[not-a-link]]\n```\n\n[[real-link]]"- cleaned = CODE_BLOCK_RE.sub("", text)+ cleaned = strip_code(text)matches = WIKI_LINK_RE.findall(cleaned)assert len(matches) == 1assert "real-link" in matches[0]@@ -33,12 +33,100 @@ def test_wiki_link_not_in_code_block():def test_wiki_link_not_in_inline_code():text = "Use `[[not-a-link]]` but also [[real-link]]"- cleaned = INLINE_CODE_RE.sub("", text)+ cleaned = strip_code(text)matches = WIKI_LINK_RE.findall(cleaned)assert len(matches) == 1assert "real-link" in matches[0]+def _links(text: str) -> list[str]:+ return WIKI_LINK_RE.findall(strip_code(text))+++# --- Fence length is honoured (#140) ------------------------------------------+++def test_issue_140_odd_inner_fence_does_not_leak():+ # The #129 builtin provider fences a <pre> holding ``` with ````.+ text = "````\n```\n[[ -n y ]]\n````\n\n[[real-link]]"+ assert _links(text) == ["real-link"]+++def test_longer_outer_fence_holds_inner_fenced_block():+ text = (+ "`````markdown\n"+ "```bash\n[[ -f x ]]\n```\n"+ "````\n[[inside-too]]\n"+ "`````\n"+ "after [[real-link]]"+ )+ assert _links(text) == ["real-link"]+++def test_closing_fence_may_be_longer_than_opening():+ text = "```\n[[code]]\n`````\n[[real-link]]"+ assert _links(text) == ["real-link"]+++def test_fence_line_with_trailing_text_does_not_close():+ text = "```\n``` not a closer [[code]]\n```\n[[real-link]]"+ assert _links(text) == ["real-link"]+++def test_indented_fence_in_list_item():+ text = "- step:\n ```bash\n [[ -n y ]]\n ```\n- see [[real-link]]"+ assert _links(text) == ["real-link"]+++def test_unclosed_fence_is_literal_text():+ # An opener nothing later can close is text, not a block to the end of+ # the note: a prose line starting with ``` must not drop what follows.+ text = "[[before]]\n```\n[[kept]]\n[[also-kept]]"+ assert _links(text) == ["before", "kept", "also-kept"]+++def test_prose_line_starting_with_backticks_keeps_the_rest():+ text = "Use\n``` to open a fence\nand then [[real-link]] later"+ assert _links(text) == ["real-link"]+++def test_unclosed_long_fence_does_not_hide_a_later_block():+ text = "`````\n[[shown]]\n```\n[[hidden]]\n```\n[[after]]"+ assert _links(text) == ["shown", "after"]+++def test_double_backtick_span_holding_single_backtick():+ text = "Run `` echo `x` [[x]] `` then read [[real-link]]"+ assert _links(text) == ["real-link"]+++def test_inline_span_closes_only_on_equal_length_run():+ text = "`a``[[x]]` and ``b`[[y]]`` and [[real-link]]"+ assert _links(text) == ["real-link"]+++def test_mid_line_triple_backticks_are_an_inline_span():+ # Not a fence (not at line start), so it is an inline span of length 3,+ # stripped as the old regex did.+ text = "Use ```[[x]]``` or [[real-link]]"+ assert _links(text) == ["real-link"]+++def test_unmatched_backtick_run_is_literal():+ text = "a stray ` backtick, then [[real-link]] and ``` more [[other]]"+ assert _links(text) == ["real-link", "other"]+++def test_stripped_span_cannot_fuse_brackets():+ assert _links("[`x`[y]]") == []+++def test_strip_keeps_line_numbers():+ text = "a\n```\none\ntwo\n```\n`multi\nline`\n[[real-link]]"+ cleaned = strip_code(text)+ assert cleaned.count("\n") == text.count("\n")+ assert cleaned.split("\n")[7] == "[[real-link]]"+def test_backlinks_populated(seeded_vault):"""Concurrency links to python-async-patterns, so python-async-patterns should have a backlink."""rows = seeded_vault.db.execute(diff --git a/tests/test_web/test_builtin_fences.py b/tests/test_web/test_builtin_fences.pyindex 215d55b..6b9b548 100644--- a/tests/test_web/test_builtin_fences.py+++ b/tests/test_web/test_builtin_fences.py@@ -11,10 +11,9 @@ and the stdlib fallback.import pytestfrom hyperresearch.core.patterns import (- CODE_BLOCK_RE,- INLINE_CODE_RE,WIKI_LINK_RE,is_valid_wiki_link_target,+ strip_code,)from hyperresearch.web.builtin import BuiltinProvider, _TextExtractorfrom hyperresearch.web.pdf import PDF_FAILURE_KEY@@ -39,8 +38,7 @@ kernel="$1"def _surviving_links(text: str) -> list[str]:"""The exact strip-then-extract sequence core/note.py runs on a note body."""- cleaned = CODE_BLOCK_RE.sub("", text)- cleaned = INLINE_CODE_RE.sub("", cleaned)+ cleaned = strip_code(text)raw = (m.group(1).strip().rstrip("\\") for m in WIKI_LINK_RE.finditer(cleaned))return [ref for ref in raw if is_valid_wiki_link_target(ref)]@@ -77,6 +75,21 @@ class _BranchContract:assert "````" in textassert _surviving_links(text) == []+ def test_odd_backtick_fence_inside_pre_does_not_leak(self):+ # #140: the fence outgrows the ``` inside, and the strip must honour+ # that length instead of closing at the inner ```. The second target+ # passes the link validator, so only the strip stands in its way.+ html = "<html><body><pre>```\n[[ -n y ]]\n[[looks-like-a-note]]</pre></body></html>"+ _, text = self.extract(html)+ assert "````" in text+ assert WIKI_LINK_RE.findall(strip_code(text)) == []++ def test_inline_code_holding_double_backticks_does_not_leak(self):+ html = "<html><body><p>Try <code>a``b [[looks-like-a-note]]</code> now.</p></body></html>"+ _, text = self.extract(html)+ assert "``` a``b [[looks-like-a-note]] ```" in text+ assert WIKI_LINK_RE.findall(strip_code(text)) == []+def test_title_and_noise_removal_unchanged(self):html = ("<html><head><title>T</title></head><body>"