Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

fix: citecheck resolves entries via note source or DOI, labels dangling (#156)

Francois du Plessisaece65f93071
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c2553c4..62aab32 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,7 @@
### Fixed
+- **Cite-check resolves a Sources entry through the note's own source or its DOI, and says why a citation dangles (from @fduple).** `parse_sources_section` looked a Sources URL up in the `sources` table only. Removing a note leaves its `sources` row with no note, and a note written again for the same work holds the URL only in `notes.source`, so the entry resolved to nothing and step 14.5 raised a `critical` "fabricated or mangled citation" finding against a sound citation. Entries now resolve by exact URL (the live `sources` row, then `notes.source`), then by exact title as before, then by `notes.doi` when exactly one note holds the link's DOI or arXiv id, and last by that DOI inside another note's source address. A `[N]` marker with no `[N]` Sources line, which used to produce no pair at all, is now reported. Each unresolved pair carries an `unresolved` reason and every pair its `marker`: `no-sources-entry` and `unknown-note-id` stay `critical`, while `unresolved-entry` (the entry exists but matches no note) is looked up in the vault before it becomes a `major` finding. Brackets inside code spans, fenced blocks and link-reference definitions (`[7]: https://...`) are no longer read as citations.
- **Cite-check no longer auto-passes a citation because one number matches (#150, from @jackiectl2).** The mechanical triage passed a sentence when every number in it appeared anywhere in the cited note's claims, so "Another team improves throughput by 10x" or "Async employs 10 engineers" passed on a claim about async's 10x throughput gain. A sentence with a figure of 2+ digits now passes only when it matches one of the cited note's claims or quotes in full; anything else goes to the cite-checker agent. Single-digit tokens ("Step 1", the "4" in "GPT-4") are still labels rather than figures, so those sentences keep passing on a long shared run of words. The step 14.5 skill and the cite-checker prompt describe the new rule.
## [0.13.0] - 2026-10-27
diff --git a/src/hyperresearch/core/citecheck.py b/src/hyperresearch/core/citecheck.py
index cee5df3..f3a1f86 100644
--- a/src/hyperresearch/core/citecheck.py
+++ b/src/hyperresearch/core/citecheck.py
@@ -23,21 +23,118 @@ from __future__ import annotations
import json
import re
+from urllib.parse import urlparse
-from hyperresearch.core.patterns import WIKI_LINK_RE
+from hyperresearch.core.fetcher import existing_live_note_for_url
+from hyperresearch.core.patterns import WIKI_LINK_RE, mask_code
+from hyperresearch.core.scholar import DOI_RE, extract_doi
# Sentence split: period/question/exclamation followed by space+capital, or newline.
_SENTENCE_SPLIT_RE = re.compile(r"(?<=[.!?])\s+(?=[A-Z一-鿿])|\n+")
_NUMBERED_CITE_RE = re.compile(r"\[(\d{1,3}(?:\s*,\s*\d{1,3})*)\]")
_SOURCES_ENTRY_RE = re.compile(r"^\s*\[(\d{1,3})\]\s+(.+)$")
_NUMBER_RE = re.compile(r"\d[\d,]*\.?\d*%?")
+# A link-reference definition (`[7]: https://...`) is a whole line and cites nothing.
+_LINK_DEF_RE = re.compile(r"^\s*\[\d+\]:")
# Sentences carrying these are checked at 100% regardless of sampling.
_STRONG_MARKERS = ("%", "$", "billion", "million", "increase", "decrease", "grew", "fell")
-def _split_sentences(text: str) -> list[str]:
- return [s.strip() for s in _SENTENCE_SPLIT_RE.split(text) if s.strip()]
+def _sentence_spans(text: str) -> list[tuple[int, int]]:
+ """(start, end) of each sentence in `text`, surrounding whitespace excluded."""
+ spans: list[tuple[int, int]] = []
+ pos = 0
+ for m in [*_SENTENCE_SPLIT_RE.finditer(text), None]:
+ end = m.start() if m else len(text)
+ piece = text[pos:end]
+ if piece.strip():
+ start = pos + len(piece) - len(piece.lstrip())
+ spans.append((start, pos + len(piece.rstrip())))
+ if m:
+ pos = m.end()
+ return spans
+
+
+def _doi_key(url: str) -> str | None:
+ """The DOI or arXiv id a link names, lower-cased, or None.
+
+ extract_doi is what fills notes.doi at fetch time, so its result is the
+ key for that column.
+ """
+ # urlsplit raises ValueError on a host it cannot parse
+ # (`https://example.com]`, or a bare domain written as `[url](url)`);
+ # such a link names no DOI.
+ try:
+ doi = extract_doi(url)
+ path = urlparse(url).path
+ except ValueError:
+ return None
+ # A `(` left open means the DOI was cut (see below), or the link was:
+ # `.../S0140-6736(20)` loses its `)` to the caller's rstrip.
+ if not doi or doi.count("(") > doi.count(")"):
+ return None
+ # DOI_RE stops at `)`, which Elsevier PII and Wiley SICI DOIs contain, and
+ # fetch stores the cut value, so one journal's papers share a key. A DOI
+ # the path runs on past (beyond a closing `>` or quote) is not looked up.
+ m = DOI_RE.search(path)
+ if m and path[m.end():].strip(">\"'"):
+ return None
+ return doi.lower()
+
+
+def _note_for_url(url: str, conn) -> str | None:
+ """The vault note saved from this exact URL, or None.
+
+ The `sources` table alone is not enough. Removing a note leaves its
+ sources row with note_id NULL, and a note written again for the same
+ work (`note new --source URL`) holds the URL only in its own `source`
+ field.
+ """
+ row = existing_live_note_for_url(conn, url)
+ if row:
+ return row["note_id"]
+ row = conn.execute("SELECT id FROM notes WHERE source = ? ORDER BY id", (url,)).fetchone()
+ if row:
+ return row["id"]
+ return None
+
+
+def _note_with_doi(url: str, conn) -> str | None:
+ """The one note whose `doi` is the link's DOI, or None.
+
+ A report may cite a paper by its doi.org link while the note was saved
+ from another address. Tried after the title: fetch fills `notes.doi` from
+ the first DOI after a "DOI" label in a page's body when the page has no
+ DOI meta tag, so a news item or review can hold the DOI of a paper it only
+ cites. Two notes holding one DOI say nothing about which is the paper.
+ """
+ doi = _doi_key(url)
+ if not doi:
+ return None
+ rows = conn.execute("SELECT id FROM notes WHERE lower(doi) = ? LIMIT 2", (doi,)).fetchall()
+ return rows[0]["id"] if len(rows) == 1 else None
+
+
+def _note_holding_doi(url: str, conn) -> str | None:
+ """A note saved from another address that holds the link's DOI, or None.
+
+ The weakest match, tried after the title: an address can hold the DOI of
+ a work it only lists or searches for.
+ """
+ doi = _doi_key(url)
+ if not doi:
+ return None
+ # A DOI inside another address ends where the URL does or at its query or
+ # fragment; `/` is part of many DOIs. Without the bound, 10.1000/abc would
+ # match the address of 10.1000/abc1.
+ bounded = re.compile(re.escape(doi) + r"(?=$|[?#&])")
+ for row in conn.execute(
+ "SELECT id, source FROM notes WHERE instr(lower(source), ?) > 0 ORDER BY id", (doi,)
+ ):
+ if bounded.search(row["source"].lower()):
+ return row["id"]
+ return None
def parse_sources_section(report_text: str, conn) -> dict[str, str | None]:
@@ -57,12 +154,11 @@ def parse_sources_section(report_text: str, conn) -> dict[str, str | None]:
continue
num, rest = m.group(1), m.group(2)
note_id = None
+ url = None
url_m = re.search(r"https?://\S+", rest)
if url_m:
url = url_m.group(0).rstrip(".,)")
- row = conn.execute("SELECT note_id FROM sources WHERE url = ?", (url,)).fetchone()
- if row:
- note_id = row["note_id"]
+ note_id = _note_for_url(url, conn)
if note_id is None:
title = rest.split("http")[0].strip(" .–-")
if title:
@@ -71,6 +167,10 @@ def parse_sources_section(report_text: str, conn) -> dict[str, str | None]:
).fetchone()
if row:
note_id = row["id"]
+ if note_id is None and url:
+ note_id = _note_with_doi(url, conn)
+ if note_id is None and url:
+ note_id = _note_holding_doi(url, conn)
mapping[num] = note_id
return mapping
@@ -79,7 +179,15 @@ def extract_pairs(report_text: str, conn) -> list[dict]:
"""All (sentence, note_id) citation bindings in the report.
Pairs whose citation can't be resolved to a vault note get
- note_id=None — those are findings in themselves (dangling citation).
+ note_id=None and an `unresolved` reason, because the reasons call for
+ different handling:
+
+ no-sources-entry - a `[N]` marker with no `[N]` line in the Sources
+ section: a fabricated or mangled citation.
+ unknown-note-id - a `[[note-id]]` naming no vault note: the same.
+ unresolved-entry - the Sources entry exists but neither its URL nor its
+ title matches a vault note. The citation may still be
+ sound; check the entry before calling it a finding.
"""
numbered_map = parse_sources_section(report_text, conn)
known_ids = {row["id"] for row in conn.execute("SELECT id FROM notes")}
@@ -87,23 +195,58 @@ def extract_pairs(report_text: str, conn) -> list[dict]:
# Strip the Sources section from the checked body
body = re.split(r"^##\s+(?:Sources|References)\b", report_text, maxsplit=1, flags=re.M | re.I)[0]
+ # Code and link-reference definitions hold brackets that are not citations
+ # (`arr[0]`, `[7]: https://...`). mask_code blanks code at its own length
+ # and keeps line structure, so a line it empties (a fenced block) or a
+ # definition line is dropped, and on every other line an offset means the
+ # same character in both texts. Each line is split into sentences once,
+ # on the report's text, which is what the patcher edits by; markers are
+ # scanned in the same span of the masked line.
+ sentences: list[str] = []
+ scanned: list[str] = []
+ for line, masked in zip(body.split("\n"), mask_code(body).split("\n"), strict=True):
+ if not masked.strip() or _LINK_DEF_RE.match(line):
+ continue
+ for start, end in _sentence_spans(line):
+ sentences.append(line[start:end])
+ scanned.append(masked[start:end])
+
pairs: list[dict] = []
- for sentence in _split_sentences(body):
- cited: list[str | None] = []
- for m in _NUMBERED_CITE_RE.finditer(sentence):
- for num in re.split(r"\s*,\s*", m.group(1)):
+ for sentence, text in zip(sentences, scanned, strict=True):
+ cited: list[tuple[str, str | None, str | None]] = []
+ for m in _NUMBERED_CITE_RE.finditer(text):
+ nums = re.split(r"\s*,\s*", m.group(1))
+ # Citations count from 1: `[0]`, `[0, 3]` and `[007]` are an index,
+ # an interval or an id, unless the Sources section numbers its
+ # entries that way.
+ if any(n.startswith("0") and n not in numbered_map for n in nums):
+ continue
+ for num in nums:
if num in numbered_map:
- cited.append(numbered_map[num])
- for m in WIKI_LINK_RE.finditer(sentence):
+ note_id = numbered_map[num]
+ cited.append((f"[{num}]", note_id, None if note_id else "unresolved-entry"))
+ elif numbered_map:
+ # Only once a `[N] ...` Sources line parsed: without one,
+ # `[N]` is not the report's citation style and the marker
+ # is not a citation.
+ cited.append((f"[{num}]", None, "no-sources-entry"))
+ for m in WIKI_LINK_RE.finditer(text):
target = m.group(1).strip()
- cited.append(target if target in known_ids else None)
- for note_id in cited:
- pairs.append({
+ if target in known_ids:
+ cited.append((f"[[{target}]]", target, None))
+ else:
+ cited.append((f"[[{target}]]", None, "unknown-note-id"))
+ for marker, note_id, reason in cited:
+ pair = {
"sentence": sentence,
+ "marker": marker,
"note_id": note_id,
"numbers": _NUMBER_RE.findall(sentence),
"strong": any(k in sentence.lower() for k in _STRONG_MARKERS) or bool(_NUMBER_RE.search(sentence)),
- })
+ }
+ if reason:
+ pair["unresolved"] = reason
+ pairs.append(pair)
return pairs
diff --git a/src/hyperresearch/core/patterns.py b/src/hyperresearch/core/patterns.py
index 8622c5a..ebc9bcf 100644
--- a/src/hyperresearch/core/patterns.py
+++ b/src/hyperresearch/core/patterns.py
@@ -98,7 +98,7 @@ def _strip_fenced_blocks(text: str) -> str:
return "\n".join(out)
-def _strip_inline_code(text: str) -> str:
+def _strip_inline_code(text: str, keep_length: bool = False) -> str:
runs = [m.span() for m in _BACKTICK_RUN_RE.finditer(text)]
if len(runs) < 2:
return text
@@ -127,7 +127,10 @@ def _strip_inline_code(text: str) -> str:
end = runs[j][1]
newlines = text.count("\n", a, end)
out.append(text[pos:a])
- out.append("\n" * newlines if newlines else " ")
+ if keep_length:
+ out.append(re.sub(r"[^\n]", " ", text[a:end]))
+ else:
+ out.append("\n" * newlines if newlines else " ")
pos = end
i = j + 1
out.append(text[pos:])
@@ -143,6 +146,13 @@ def strip_code(text: str) -> str:
return _strip_inline_code(_strip_fenced_blocks(text))
+def mask_code(text: str) -> str:
+ """strip_code, but each inline code span becomes spaces of its own length
+ (newlines kept), so an offset into a line outside a fenced block points at
+ the same character in both texts. Fenced block lines become empty."""
+ return _strip_inline_code(_strip_fenced_blocks(text), keep_length=True)
+
+
# Citation-footnote patterns that should NOT be treated as wiki-links even
# though crawl4ai sometimes renders them as [[100]] or [[^100]] in fetched
# markdown. These originate from Wikipedia-style footnote markup like
diff --git a/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md b/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md
index 4a63677..c6b1fe6 100644
--- a/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md
+++ b/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md
@@ -35,9 +35,11 @@ Read these inputs:
This parses every (sentence, citation) pair from the report — `[N]` markers (including grouped `[7, 12]`, one pair per source number) and `[[note-id]]` styles — and auto-passes a pair only when the cited note's extracted claims already confirm it: a sentence carrying a figure of 2+ digits must match one claim or quote in full (a shared number alone can describe a different entity or metric), while a sentence with no such figure passes on a long shared run of words. Single-digit tokens such as "Step 1" or the "4" in "GPT-4" are labels, not figures. Output: `research/runs/<vault_tag>/cite-check-pairs.json` with:
- `summary` — total / auto-passed / dangling / needs-llm counts
- `sampled_for_llm` — the pairs the agent must judge (100% of number-bearing sentences; sampled for the rest)
-- `dangling` — citations that resolve to NO vault note
+- `dangling` — citations that resolve to NO vault note, each with an `unresolved` reason and its `marker`
-**Dangling citations are findings immediately** — no agent needed. Each one becomes a `critical` finding (fabricated or mangled citation).
+**Handle dangling citations by their `unresolved` reason.** No agent is needed, but the reasons are not equal:
+- `no-sources-entry` (a `[N]` with no `[N]` line in the Sources (or References) section) and `unknown-note-id` (a `[[note-id]]` naming no vault note): each becomes a `critical` finding (fabricated or mangled citation).
+- `unresolved-entry`: the Sources entry exists, but its URL and title match no vault note. That is a lookup miss, not evidence against the citation. Check the entry first: `{hpr_path} search "<distinctive title words>" -j`, and `{hpr_path} note show` the hit to confirm it is the cited work. If a note covers it, the citation stands and there is no finding. Only if the vault holds no such source does it become a `major` finding (a citation the vault cannot back).
**If `sampled_for_llm` is empty and there are no dangling citations:** write an empty findings file `[]` to `research/runs/<vault_tag>/cite-check-findings.json`, record `{hpr_path} run step <vault_tag> 14.5 --status done -j`, and proceed to step 15. Done.
diff --git a/tests/test_core/test_patterns.py b/tests/test_core/test_patterns.py
index fb55fea..5251c2e 100644
--- a/tests/test_core/test_patterns.py
+++ b/tests/test_core/test_patterns.py
@@ -1,6 +1,6 @@
"""Tests for wiki-link target validation — citation-footnote edge cases."""
-from hyperresearch.core.patterns import is_valid_wiki_link_target, strip_code
+from hyperresearch.core.patterns import is_valid_wiki_link_target, mask_code, strip_code
def test_empty_and_whitespace_rejected():
@@ -292,3 +292,17 @@ def test_shell_shaped_but_legitimate_refs_still_accepted():
assert is_valid_wiki_link_target("C. elegans")
assert is_valid_wiki_link_target("yahoo! finance") # '!' mid-string stays legal
assert is_valid_wiki_link_target("e.g. v1.2")
+
+
+def test_mask_code_blanks_spans_at_their_own_length():
+ text = "Run `a [1]` now [2].\nSpan `across\nlines` ends [3].\n```\nfenced [4]\n```\nLast `x`."
+ masked = mask_code(text)
+ lines, mlines = text.split("\n"), masked.split("\n")
+ assert len(mlines) == len(lines)
+ assert mlines[0] == "Run " + " " * 7 + " now [2]."
+ assert mlines[1] == "Span " + " " * 7 and mlines[2] == " " * 6 + " ends [3]."
+ assert mlines[3:6] == ["", "", ""]
+ assert mlines[6] == "Last " + " " * 3 + "."
+ assert "[1]" not in masked and "[4]" not in masked
+ # strip_code is unchanged: a span still becomes one space.
+ assert strip_code("Run `a [1]` now") == "Run now"
diff --git a/tests/test_core/test_verification.py b/tests/test_core/test_verification.py
index ad050ba..329bd1a 100644
--- a/tests/test_core/test_verification.py
+++ b/tests/test_core/test_verification.py
@@ -71,9 +71,296 @@ class TestCiteCheckExtraction:
report = "A bold claim [[no-such-note]]."
pairs = extract_pairs(report, cited_vault.db)
assert pairs[0]["note_id"] is None
+ assert pairs[0].get("unresolved") == "unknown-note-id"
triaged = triage_pairs(pairs, cited_vault.db)
assert triaged["dangling"] == 1
+ def test_sources_entry_resolves_through_the_notes_own_source_field(self, cited_vault):
+ # `note rm` leaves the sources row with note_id NULL; a note written
+ # again for the work holds the URL only in its own source field.
+ url = "https://repo.example.org/portal/3549507.pdf"
+ cited_vault.db.execute(
+ "INSERT INTO sources (url, note_id, domain, fetched_at, provider, content_hash) "
+ "VALUES (?, NULL, 'repo.example.org', '2026-01-01', 'test', 'y')", (url,)
+ )
+ cited_vault.db.execute("UPDATE notes SET source = ? WHERE id = 'python-async-patterns'", (url,))
+ cited_vault.db.commit()
+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] Some Paper (A. Author, 2022). {url}\n"
+ assert parse_sources_section(report, cited_vault.db)["1"] == "python-async-patterns"
+
+ @pytest.mark.parametrize("link", [
+ "[https://vendor.example.com](https://vendor.example.com)",
+ "https://[vendor.example.com]/a",
+ "https://vendor.example.com]",
+ ])
+ def test_link_that_urlsplit_rejects_is_unresolved_not_an_error(self, cited_vault, link):
+ # urlsplit raises ValueError on a bracket in the host position.
+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] A Vendor Page. {link}\n"
+ assert parse_sources_section(report, cited_vault.db) == {"1": None}
+
+ @pytest.mark.parametrize("link", [
+ "https://dx.doi.org/10.1145/3313831.3376831",
+ "https://doi.org/10.1145/3313831.3376831?utm_source=x",
+ ])
+ def test_doi_link_resolves_through_the_notes_doi_column(self, cited_vault, link):
+ cited_vault.db.execute(
+ "UPDATE notes SET doi = '10.1145/3313831.3376831' WHERE id = 'python-async-patterns'"
+ )
+ cited_vault.db.commit()
+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] A Postmortem (CHI 2020). {link}\n"
+ assert parse_sources_section(report, cited_vault.db)["1"] == "python-async-patterns"
+
+ def test_arxiv_link_resolves_through_the_notes_doi_column(self, cited_vault):
+ # fetch stores extract_doi's result, an arXiv id for arxiv.org links.
+ cited_vault.db.execute("UPDATE notes SET doi = 'arXiv:2301.00001' WHERE id = 'python-async-patterns'")
+ cited_vault.db.commit()
+ report = "Throughput improves 10x [1].\n\n## Sources\n[1] A Preprint. https://arxiv.org/pdf/2301.00001\n"
+ assert parse_sources_section(report, cited_vault.db)["1"] == "python-async-patterns"
+
+ @pytest.mark.parametrize("link", [
+ "<https://doi.org/10.1145/3313831.3376831>",
+ "https://doi.org/10.1145/3313831.3376831;",
+ ])
+ def test_doi_link_resolves_to_a_note_saved_from_another_address(self, cited_vault, link):
+ cited_vault.db.execute(
+ "UPDATE notes SET source = ? WHERE id = 'python-async-patterns'",
+ ("https://api.example.org/graph/v1/paper/DOI:10.1145/3313831.3376831?fields=title",),
+ )
+ cited_vault.db.commit()
+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] A Postmortem (CHI 2020). {link}\n"
+ assert parse_sources_section(report, cited_vault.db)["1"] == "python-async-patterns"
+
+ def test_exact_doi_column_beats_an_address_that_merely_holds_the_doi(self, cited_vault):
+ doi = "10.1145/3313831.3376831"
+ cited_vault.db.execute("UPDATE notes SET doi = ? WHERE id = 'rust-ownership'", (doi,))
+ cited_vault.db.execute(
+ "UPDATE notes SET source = ? WHERE id = 'concurrency'", (f"https://cite.example/refs?doi={doi}",)
+ )
+ cited_vault.db.commit()
+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] A Postmortem (CHI 2020). https://doi.org/{doi}\n"
+ assert parse_sources_section(report, cited_vault.db)["1"] == "rust-ownership"
+
+ def test_title_match_beats_an_address_that_merely_holds_the_doi(self, cited_vault):
+ doi = "10.1145/3313831.3376831"
+ cited_vault.db.execute(
+ "UPDATE notes SET source = ? WHERE id = 'concurrency'", (f"https://cite.example/refs?doi={doi}",)
+ )
+ cited_vault.db.commit()
+ report = f"Memory safety without GC [1].\n\n## Sources\n[1] Rust Ownership. https://doi.org/{doi}\n"
+ assert parse_sources_section(report, cited_vault.db)["1"] == "rust-ownership"
+
+ @pytest.mark.parametrize(("stored", "expected"), [
+ ("https://x.example.org/view/10.1145/3313831.3376831", "python-async-patterns"),
+ ("https://x.example.org/view/10.1145/3313831.3376831?utm=1", "python-async-patterns"),
+ ("https://x.example.org/refs?doi=10.1145/3313831.3376831&from=feed", "python-async-patterns"),
+ ("https://x.example.org/view/10.1145/3313831.3376831#abstract", "python-async-patterns"),
+ ("https://x.example.org/view/10.1145/3313831.3376831/full", None),
+ ], ids=["end", "query", "ampersand", "fragment", "slash"])
+ def test_doi_inside_an_address_ends_at_the_url_end_query_or_fragment(
+ self, cited_vault, stored, expected
+ ):
+ """`/` is part of many DOIs, so a path that goes on after the DOI may
+ name another one and does not match."""
+ cited_vault.db.execute(
+ "UPDATE notes SET source = ? WHERE id = 'python-async-patterns'", (stored,)
+ )
+ cited_vault.db.commit()
+ link = "https://doi.org/10.1145/3313831.3376831"
+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] A Postmortem (CHI 2020). {link}\n"
+ assert parse_sources_section(report, cited_vault.db)["1"] == expected
+
+ def test_doi_that_prefixes_another_does_not_resolve_to_it(self, cited_vault):
+ cited_vault.db.execute("UPDATE notes SET source = 'https://x.org/10.1000/abc1' WHERE id = 'concurrency'")
+ cited_vault.db.commit()
+ report = "Claim [1] and [2].\n\n## Sources\n[1] One Paper. https://doi.org/10.1000/abc\n[2] Other Paper. https://doi.org/10.1000/abc1\n"
+ assert parse_sources_section(report, cited_vault.db) == {"1": None, "2": "concurrency"}
+
+ def test_doi_cut_short_at_a_parenthesis_is_not_looked_up(self, cited_vault):
+ # DOI_RE stops at `)` and fetch stores the cut value, so two Lancet 2020
+ # papers (Elsevier PII DOIs) share notes.doi; the cut DOI must resolve
+ # through the sources row or notes.source only, never the doi column.
+ cut = "10.1016/S0140-6736(20"
+ huang = "https://dx.doi.org/10.1016/S0140-6736(20)30183-5"
+ zhou = "https://doi.org/10.1016/S0140-6736(20)30566-3"
+ cited_vault.db.execute("UPDATE notes SET source = ?, doi = ? WHERE id = 'concurrency'", (huang, cut))
+ cited_vault.db.execute("UPDATE notes SET source = ?, doi = ? WHERE id = 'rust-ownership'", (zhou, cut))
+ cited_vault.db.execute(
+ "INSERT INTO sources (url, note_id, domain, fetched_at, provider, content_hash) "
+ "VALUES (?, 'rust-ownership', 'doi.org', '2026-01-01', 'test', 'z')", (zhou,)
+ )
+ cited_vault.db.commit()
+
+ def resolve(link):
+ return parse_sources_section(f"Claim [1].\n\n## Sources\n[1] A Paper. {link}\n", cited_vault.db)["1"]
+
+ assert resolve(zhou) == "rust-ownership" # the sources row
+ assert resolve(huang) == "concurrency" # notes.source
+ absent = "https://doi.org/10.1016/S0140-6736(20)31142-9"
+ variants = [
+ "https://dx.doi.org/10.1016/S0140-6736(20)30566-3",
+ "http://doi.org/10.1016/S0140-6736(20)30566-3",
+ "https://doi.org/10.1016/S0140-6736(20)30566-3?via=x",
+ absent,
+ ]
+ assert {v: resolve(v) for v in variants} == dict.fromkeys(variants)
+
+ @pytest.mark.parametrize("tail", ["(20)", "(20"])
+ def test_link_cut_after_a_parenthesised_group_is_not_looked_up(self, cited_vault, tail):
+ # The caller strips a trailing `)` from the link, which leaves the key
+ # fetch stores for every Lancet 2020 paper.
+ cited_vault.db.execute(
+ "UPDATE notes SET doi = '10.1016/S0140-6736(20' WHERE id = 'concurrency'"
+ )
+ cited_vault.db.commit()
+ report = f"Claim [1].\n\n## Sources\n[1] A Paper. https://doi.org/10.1016/S0140-6736{tail}\n"
+ assert parse_sources_section(report, cited_vault.db)["1"] is None
+
+ def test_title_match_beats_a_doi_another_page_holds(self, cited_vault):
+ # fetch fills notes.doi from the first DOI in a page's body when the
+ # page has no DOI meta tag, so a page that only cites the paper can
+ # hold its DOI.
+ doi = "10.1145/3313831.3376831"
+ cited_vault.db.execute("UPDATE notes SET doi = ? WHERE id = 'concurrency'", (doi,))
+ cited_vault.db.commit()
+ report = f"Claim [1].\n\n## Sources\n[1] Python Async Patterns. https://doi.org/{doi}\n"
+ assert parse_sources_section(report, cited_vault.db)["1"] == "python-async-patterns"
+
+ def test_doi_two_notes_hold_resolves_to_neither(self, cited_vault):
+ doi = "10.1145/3313831.3376831"
+ cited_vault.db.execute(
+ "UPDATE notes SET doi = ? WHERE id IN ('concurrency', 'rust-ownership')", (doi,)
+ )
+ cited_vault.db.commit()
+ report = f"Claim [1].\n\n## Sources\n[1] A Postmortem (CHI 2020). https://doi.org/{doi}\n"
+ assert parse_sources_section(report, cited_vault.db)["1"] is None
+
+ def test_sici_doi_via_dx_doi_org_is_not_looked_up(self, cited_vault):
+ cited_vault.db.execute(
+ "UPDATE notes SET source = ?, doi = '10.1002/(SICI' WHERE id = 'concurrency'",
+ ("https://doi.org/10.1002/(SICI)1097-0258(19980415)17:7<857::AID-SIM777>3.0.CO;2-E",),
+ )
+ cited_vault.db.commit()
+ report = (
+ "Claim [1].\n\n## Sources\n"
+ "[1] Another Article. https://dx.doi.org/10.1002/(SICI)1097-0258(19980430)17:8<900::AID-SIM800>3.0.CO;2-F\n"
+ )
+ assert parse_sources_section(report, cited_vault.db)["1"] is None
+
+ def test_marker_without_a_sources_entry_is_dangling_not_dropped(self, cited_vault):
+ report = (
+ "Throughput improves 10x [1][7].\n\n"
+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"
+ )
+ pairs = extract_pairs(report, cited_vault.db)
+ assert [p["note_id"] for p in pairs] == ["python-async-patterns", None]
+ assert [p.get("marker") for p in pairs] == ["[1]", "[7]"]
+ assert pairs[1].get("unresolved") == "no-sources-entry"
+ assert triage_pairs(pairs, cited_vault.db)["dangling"] == 1
+
+ def test_numbering_from_zero_or_zero_padded_still_cites(self, cited_vault):
+ report = (
+ "First claim [01]. Second claim [0]. An id [007].\n\n"
+ "## Sources\n[0] Rust Ownership. https://example.com/rust\n"
+ "[01] Python Async Patterns. https://example.com/async\n"
+ )
+ pairs = extract_pairs(report, cited_vault.db)
+ assert [(p["sentence"], p["note_id"]) for p in pairs] == [
+ ("First claim [01].", "python-async-patterns"),
+ ("Second claim [0].", "rust-ownership"),
+ ]
+
+ def test_unresolvable_sources_entry_is_labelled_as_such(self, cited_vault):
+ report = (
+ "Throughput improves 10x [1].\n\n"
+ "## Sources\n[1] Unrelated Study. https://example.com/other\n"
+ )
+ pairs = extract_pairs(report, cited_vault.db)
+ assert pairs[0]["note_id"] is None
+ assert pairs[0].get("unresolved") == "unresolved-entry"
+
+ def test_bracketed_numbers_without_a_sources_section_are_not_citations(self, cited_vault):
+ report = "See item [3] in the list [[python-async-patterns]]."
+ pairs = extract_pairs(report, cited_vault.db)
+ assert [p["note_id"] for p in pairs] == ["python-async-patterns"]
+ assert [p.get("marker") for p in pairs] == ["[[python-async-patterns]]"]
+
+ def test_brackets_in_code_and_link_definitions_are_not_citations(self, cited_vault):
+ report = (
+ "Intro claim [1].\n\n"
+ "```python\narr[0] = x[2]\n```\n\n"
+ "Set `v[3]` before the call [1].\n\n"
+ "[7]: https://example.com/seven\n\n"
+ "| Item | Source |\n|---|---|\n| foo | [5] |\n\n"
+ "Run `[[ -n \"$k\" ]]` first [1].\n\n"
+ "Run `foo [5]\nbar` now [1].\n\n"
+ "Then [12] and [0].\n\n"
+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"
+ )
+ pairs = extract_pairs(report, cited_vault.db)
+ # The stored sentence keeps its code span: the patcher edits by it.
+ assert [(p["sentence"], p["note_id"]) for p in pairs] == [
+ ("Intro claim [1].", "python-async-patterns"),
+ ("Set `v[3]` before the call [1].", "python-async-patterns"),
+ ("| foo | [5] |", None),
+ ("Run `[[ -n \"$k\" ]]` first [1].", "python-async-patterns"),
+ ("bar` now [1].", "python-async-patterns"),
+ ("Then [12] and [0].", None),
+ ]
+ assert all(p["sentence"] in report for p in pairs)
+ assert [p.get("marker") for p in pairs] == ["[1]", "[1]", "[5]", "[1]", "[1]", "[12]"]
+ assert [p.get("unresolved") for p in pairs] == [None, None, "no-sources-entry", None, None, "no-sources-entry"]
+
+ def test_code_span_that_shifts_the_sentence_split_keeps_markers_on_their_sentences(self, cited_vault):
+ # The first span sits between a sentence end and a capital, so the
+ # stripped side gains a sentence; the last holds a sentence break, so
+ # the report side gains one. Equal counts, different splits.
+ report = (
+ "Setup. `cfg` Loads the file [1].\n\n"
+ "Second paragraph has a claim [1].\n\n"
+ "The log says `Error. Retry` on failure [1].\n\n"
+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"
+ )
+ pairs = extract_pairs(report, cited_vault.db)
+ assert [p["sentence"] for p in pairs] == [
+ "Setup. `cfg` Loads the file [1].",
+ "Second paragraph has a claim [1].",
+ "Retry` on failure [1].",
+ ]
+
+ def test_code_spans_that_shift_the_split_both_ways_on_one_line(self, cited_vault):
+ # One span sits between a sentence end and a capital, the other holds
+ # a sentence break; each marker stays on the report sentence it is in.
+ report = (
+ "Run it. `make` Builds everything [1]. The tool prints `Done. Exiting` when finished [2].\n\n"
+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"
+ "[2] Rust Ownership. https://example.com/rust\n"
+ )
+ pairs = extract_pairs(report, cited_vault.db)
+ assert [(p.get("marker"), p["sentence"]) for p in pairs] == [
+ ("[1]", "Run it. `make` Builds everything [1]."),
+ ("[2]", "Exiting` when finished [2]."),
+ ]
+
+ def test_stored_sentence_drops_indentation_and_line_endings(self, cited_vault):
+ # The patcher finds a sentence by its text, so it is stored without the
+ # line's indentation, trailing blanks or a CRLF's `\r`.
+ report = (
+ " Indented claim [1].\r\n"
+ "Trailing blanks [1]. \r\n\r\n"
+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"
+ )
+ pairs = extract_pairs(report, cited_vault.db)
+ assert [p["sentence"] for p in pairs] == ["Indented claim [1].", "Trailing blanks [1]."]
+
+ def test_sentence_break_in_one_code_span_leaves_other_lines_stripped(self, cited_vault):
+ report = (
+ "Run `foo [5]\nbar` now [1].\n\n"
+ "Much later `echo \"Done. Next\"` prints two words [1].\n\n"
+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"
+ )
+ pairs = extract_pairs(report, cited_vault.db)
+ assert [p["sentence"] for p in pairs] == ["bar` now [1].", "Next\"` prints two words [1]."]
+
def test_triage_auto_passes_full_numeric_claim(self, cited_vault):
pairs = extract_pairs(
"Async improves throughput by 10x for network-bound workloads [[python-async-patterns]].",