Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
fix: citecheck resolves entries via note source or DOI, labels dangling (#156)
Francois du Plessisaece65f93071
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c2553c4..62aab32 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,7 @@### Fixed+- **Cite-check resolves a Sources entry through the note's own source or its DOI, and says why a citation dangles (from @fduple).** `parse_sources_section` looked a Sources URL up in the `sources` table only. Removing a note leaves its `sources` row with no note, and a note written again for the same work holds the URL only in `notes.source`, so the entry resolved to nothing and step 14.5 raised a `critical` "fabricated or mangled citation" finding against a sound citation. Entries now resolve by exact URL (the live `sources` row, then `notes.source`), then by exact title as before, then by `notes.doi` when exactly one note holds the link's DOI or arXiv id, and last by that DOI inside another note's source address. A `[N]` marker with no `[N]` Sources line, which used to produce no pair at all, is now reported. Each unresolved pair carries an `unresolved` reason and every pair its `marker`: `no-sources-entry` and `unknown-note-id` stay `critical`, while `unresolved-entry` (the entry exists but matches no note) is looked up in the vault before it becomes a `major` finding. Brackets inside code spans, fenced blocks and link-reference definitions (`[7]: https://...`) are no longer read as citations.- **Cite-check no longer auto-passes a citation because one number matches (#150, from @jackiectl2).** The mechanical triage passed a sentence when every number in it appeared anywhere in the cited note's claims, so "Another team improves throughput by 10x" or "Async employs 10 engineers" passed on a claim about async's 10x throughput gain. A sentence with a figure of 2+ digits now passes only when it matches one of the cited note's claims or quotes in full; anything else goes to the cite-checker agent. Single-digit tokens ("Step 1", the "4" in "GPT-4") are still labels rather than figures, so those sentences keep passing on a long shared run of words. The step 14.5 skill and the cite-checker prompt describe the new rule.## [0.13.0] - 2026-10-27diff --git a/src/hyperresearch/core/citecheck.py b/src/hyperresearch/core/citecheck.pyindex cee5df3..f3a1f86 100644--- a/src/hyperresearch/core/citecheck.py+++ b/src/hyperresearch/core/citecheck.py@@ -23,21 +23,118 @@ from __future__ import annotationsimport jsonimport re+from urllib.parse import urlparse-from hyperresearch.core.patterns import WIKI_LINK_RE+from hyperresearch.core.fetcher import existing_live_note_for_url+from hyperresearch.core.patterns import WIKI_LINK_RE, mask_code+from hyperresearch.core.scholar import DOI_RE, extract_doi# Sentence split: period/question/exclamation followed by space+capital, or newline._SENTENCE_SPLIT_RE = re.compile(r"(?<=[.!?])\s+(?=[A-Z一-鿿])|\n+")_NUMBERED_CITE_RE = re.compile(r"\[(\d{1,3}(?:\s*,\s*\d{1,3})*)\]")_SOURCES_ENTRY_RE = re.compile(r"^\s*\[(\d{1,3})\]\s+(.+)$")_NUMBER_RE = re.compile(r"\d[\d,]*\.?\d*%?")+# A link-reference definition (`[7]: https://...`) is a whole line and cites nothing.+_LINK_DEF_RE = re.compile(r"^\s*\[\d+\]:")# Sentences carrying these are checked at 100% regardless of sampling._STRONG_MARKERS = ("%", "$", "billion", "million", "increase", "decrease", "grew", "fell")-def _split_sentences(text: str) -> list[str]:- return [s.strip() for s in _SENTENCE_SPLIT_RE.split(text) if s.strip()]+def _sentence_spans(text: str) -> list[tuple[int, int]]:+ """(start, end) of each sentence in `text`, surrounding whitespace excluded."""+ spans: list[tuple[int, int]] = []+ pos = 0+ for m in [*_SENTENCE_SPLIT_RE.finditer(text), None]:+ end = m.start() if m else len(text)+ piece = text[pos:end]+ if piece.strip():+ start = pos + len(piece) - len(piece.lstrip())+ spans.append((start, pos + len(piece.rstrip())))+ if m:+ pos = m.end()+ return spans+++def _doi_key(url: str) -> str | None:+ """The DOI or arXiv id a link names, lower-cased, or None.++ extract_doi is what fills notes.doi at fetch time, so its result is the+ key for that column.+ """+ # urlsplit raises ValueError on a host it cannot parse+ # (`https://example.com]`, or a bare domain written as `[url](url)`);+ # such a link names no DOI.+ try:+ doi = extract_doi(url)+ path = urlparse(url).path+ except ValueError:+ return None+ # A `(` left open means the DOI was cut (see below), or the link was:+ # `.../S0140-6736(20)` loses its `)` to the caller's rstrip.+ if not doi or doi.count("(") > doi.count(")"):+ return None+ # DOI_RE stops at `)`, which Elsevier PII and Wiley SICI DOIs contain, and+ # fetch stores the cut value, so one journal's papers share a key. A DOI+ # the path runs on past (beyond a closing `>` or quote) is not looked up.+ m = DOI_RE.search(path)+ if m and path[m.end():].strip(">\"'"):+ return None+ return doi.lower()+++def _note_for_url(url: str, conn) -> str | None:+ """The vault note saved from this exact URL, or None.++ The `sources` table alone is not enough. Removing a note leaves its+ sources row with note_id NULL, and a note written again for the same+ work (`note new --source URL`) holds the URL only in its own `source`+ field.+ """+ row = existing_live_note_for_url(conn, url)+ if row:+ return row["note_id"]+ row = conn.execute("SELECT id FROM notes WHERE source = ? ORDER BY id", (url,)).fetchone()+ if row:+ return row["id"]+ return None+++def _note_with_doi(url: str, conn) -> str | None:+ """The one note whose `doi` is the link's DOI, or None.++ A report may cite a paper by its doi.org link while the note was saved+ from another address. Tried after the title: fetch fills `notes.doi` from+ the first DOI after a "DOI" label in a page's body when the page has no+ DOI meta tag, so a news item or review can hold the DOI of a paper it only+ cites. Two notes holding one DOI say nothing about which is the paper.+ """+ doi = _doi_key(url)+ if not doi:+ return None+ rows = conn.execute("SELECT id FROM notes WHERE lower(doi) = ? LIMIT 2", (doi,)).fetchall()+ return rows[0]["id"] if len(rows) == 1 else None+++def _note_holding_doi(url: str, conn) -> str | None:+ """A note saved from another address that holds the link's DOI, or None.++ The weakest match, tried after the title: an address can hold the DOI of+ a work it only lists or searches for.+ """+ doi = _doi_key(url)+ if not doi:+ return None+ # A DOI inside another address ends where the URL does or at its query or+ # fragment; `/` is part of many DOIs. Without the bound, 10.1000/abc would+ # match the address of 10.1000/abc1.+ bounded = re.compile(re.escape(doi) + r"(?=$|[?#&])")+ for row in conn.execute(+ "SELECT id, source FROM notes WHERE instr(lower(source), ?) > 0 ORDER BY id", (doi,)+ ):+ if bounded.search(row["source"].lower()):+ return row["id"]+ return Nonedef parse_sources_section(report_text: str, conn) -> dict[str, str | None]:@@ -57,12 +154,11 @@ def parse_sources_section(report_text: str, conn) -> dict[str, str | None]:continuenum, rest = m.group(1), m.group(2)note_id = None+ url = Noneurl_m = re.search(r"https?://\S+", rest)if url_m:url = url_m.group(0).rstrip(".,)")- row = conn.execute("SELECT note_id FROM sources WHERE url = ?", (url,)).fetchone()- if row:- note_id = row["note_id"]+ note_id = _note_for_url(url, conn)if note_id is None:title = rest.split("http")[0].strip(" .–-")if title:@@ -71,6 +167,10 @@ def parse_sources_section(report_text: str, conn) -> dict[str, str | None]:).fetchone()if row:note_id = row["id"]+ if note_id is None and url:+ note_id = _note_with_doi(url, conn)+ if note_id is None and url:+ note_id = _note_holding_doi(url, conn)mapping[num] = note_idreturn mapping@@ -79,7 +179,15 @@ def extract_pairs(report_text: str, conn) -> list[dict]:"""All (sentence, note_id) citation bindings in the report.Pairs whose citation can't be resolved to a vault note get- note_id=None — those are findings in themselves (dangling citation).+ note_id=None and an `unresolved` reason, because the reasons call for+ different handling:++ no-sources-entry - a `[N]` marker with no `[N]` line in the Sources+ section: a fabricated or mangled citation.+ unknown-note-id - a `[[note-id]]` naming no vault note: the same.+ unresolved-entry - the Sources entry exists but neither its URL nor its+ title matches a vault note. The citation may still be+ sound; check the entry before calling it a finding."""numbered_map = parse_sources_section(report_text, conn)known_ids = {row["id"] for row in conn.execute("SELECT id FROM notes")}@@ -87,23 +195,58 @@ def extract_pairs(report_text: str, conn) -> list[dict]:# Strip the Sources section from the checked bodybody = re.split(r"^##\s+(?:Sources|References)\b", report_text, maxsplit=1, flags=re.M | re.I)[0]+ # Code and link-reference definitions hold brackets that are not citations+ # (`arr[0]`, `[7]: https://...`). mask_code blanks code at its own length+ # and keeps line structure, so a line it empties (a fenced block) or a+ # definition line is dropped, and on every other line an offset means the+ # same character in both texts. Each line is split into sentences once,+ # on the report's text, which is what the patcher edits by; markers are+ # scanned in the same span of the masked line.+ sentences: list[str] = []+ scanned: list[str] = []+ for line, masked in zip(body.split("\n"), mask_code(body).split("\n"), strict=True):+ if not masked.strip() or _LINK_DEF_RE.match(line):+ continue+ for start, end in _sentence_spans(line):+ sentences.append(line[start:end])+ scanned.append(masked[start:end])+pairs: list[dict] = []- for sentence in _split_sentences(body):- cited: list[str | None] = []- for m in _NUMBERED_CITE_RE.finditer(sentence):- for num in re.split(r"\s*,\s*", m.group(1)):+ for sentence, text in zip(sentences, scanned, strict=True):+ cited: list[tuple[str, str | None, str | None]] = []+ for m in _NUMBERED_CITE_RE.finditer(text):+ nums = re.split(r"\s*,\s*", m.group(1))+ # Citations count from 1: `[0]`, `[0, 3]` and `[007]` are an index,+ # an interval or an id, unless the Sources section numbers its+ # entries that way.+ if any(n.startswith("0") and n not in numbered_map for n in nums):+ continue+ for num in nums:if num in numbered_map:- cited.append(numbered_map[num])- for m in WIKI_LINK_RE.finditer(sentence):+ note_id = numbered_map[num]+ cited.append((f"[{num}]", note_id, None if note_id else "unresolved-entry"))+ elif numbered_map:+ # Only once a `[N] ...` Sources line parsed: without one,+ # `[N]` is not the report's citation style and the marker+ # is not a citation.+ cited.append((f"[{num}]", None, "no-sources-entry"))+ for m in WIKI_LINK_RE.finditer(text):target = m.group(1).strip()- cited.append(target if target in known_ids else None)- for note_id in cited:- pairs.append({+ if target in known_ids:+ cited.append((f"[[{target}]]", target, None))+ else:+ cited.append((f"[[{target}]]", None, "unknown-note-id"))+ for marker, note_id, reason in cited:+ pair = {"sentence": sentence,+ "marker": marker,"note_id": note_id,"numbers": _NUMBER_RE.findall(sentence),"strong": any(k in sentence.lower() for k in _STRONG_MARKERS) or bool(_NUMBER_RE.search(sentence)),- })+ }+ if reason:+ pair["unresolved"] = reason+ pairs.append(pair)return pairsdiff --git a/src/hyperresearch/core/patterns.py b/src/hyperresearch/core/patterns.pyindex 8622c5a..ebc9bcf 100644--- a/src/hyperresearch/core/patterns.py+++ b/src/hyperresearch/core/patterns.py@@ -98,7 +98,7 @@ def _strip_fenced_blocks(text: str) -> str:return "\n".join(out)-def _strip_inline_code(text: str) -> str:+def _strip_inline_code(text: str, keep_length: bool = False) -> str:runs = [m.span() for m in _BACKTICK_RUN_RE.finditer(text)]if len(runs) < 2:return text@@ -127,7 +127,10 @@ def _strip_inline_code(text: str) -> str:end = runs[j][1]newlines = text.count("\n", a, end)out.append(text[pos:a])- out.append("\n" * newlines if newlines else " ")+ if keep_length:+ out.append(re.sub(r"[^\n]", " ", text[a:end]))+ else:+ out.append("\n" * newlines if newlines else " ")pos = endi = j + 1out.append(text[pos:])@@ -143,6 +146,13 @@ def strip_code(text: str) -> str:return _strip_inline_code(_strip_fenced_blocks(text))+def mask_code(text: str) -> str:+ """strip_code, but each inline code span becomes spaces of its own length+ (newlines kept), so an offset into a line outside a fenced block points at+ the same character in both texts. Fenced block lines become empty."""+ return _strip_inline_code(_strip_fenced_blocks(text), keep_length=True)++# Citation-footnote patterns that should NOT be treated as wiki-links even# though crawl4ai sometimes renders them as [[100]] or [[^100]] in fetched# markdown. These originate from Wikipedia-style footnote markup likediff --git a/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md b/src/hyperresearch/skills/hyperresearch-14-5-cite-check.mdindex 4a63677..c6b1fe6 100644--- a/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md+++ b/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md@@ -35,9 +35,11 @@ Read these inputs:This parses every (sentence, citation) pair from the report — `[N]` markers (including grouped `[7, 12]`, one pair per source number) and `[[note-id]]` styles — and auto-passes a pair only when the cited note's extracted claims already confirm it: a sentence carrying a figure of 2+ digits must match one claim or quote in full (a shared number alone can describe a different entity or metric), while a sentence with no such figure passes on a long shared run of words. Single-digit tokens such as "Step 1" or the "4" in "GPT-4" are labels, not figures. Output: `research/runs/<vault_tag>/cite-check-pairs.json` with:- `summary` — total / auto-passed / dangling / needs-llm counts- `sampled_for_llm` — the pairs the agent must judge (100% of number-bearing sentences; sampled for the rest)-- `dangling` — citations that resolve to NO vault note+- `dangling` — citations that resolve to NO vault note, each with an `unresolved` reason and its `marker`-**Dangling citations are findings immediately** — no agent needed. Each one becomes a `critical` finding (fabricated or mangled citation).+**Handle dangling citations by their `unresolved` reason.** No agent is needed, but the reasons are not equal:+- `no-sources-entry` (a `[N]` with no `[N]` line in the Sources (or References) section) and `unknown-note-id` (a `[[note-id]]` naming no vault note): each becomes a `critical` finding (fabricated or mangled citation).+- `unresolved-entry`: the Sources entry exists, but its URL and title match no vault note. That is a lookup miss, not evidence against the citation. Check the entry first: `{hpr_path} search "<distinctive title words>" -j`, and `{hpr_path} note show` the hit to confirm it is the cited work. If a note covers it, the citation stands and there is no finding. Only if the vault holds no such source does it become a `major` finding (a citation the vault cannot back).**If `sampled_for_llm` is empty and there are no dangling citations:** write an empty findings file `[]` to `research/runs/<vault_tag>/cite-check-findings.json`, record `{hpr_path} run step <vault_tag> 14.5 --status done -j`, and proceed to step 15. Done.diff --git a/tests/test_core/test_patterns.py b/tests/test_core/test_patterns.pyindex fb55fea..5251c2e 100644--- a/tests/test_core/test_patterns.py+++ b/tests/test_core/test_patterns.py@@ -1,6 +1,6 @@"""Tests for wiki-link target validation — citation-footnote edge cases."""-from hyperresearch.core.patterns import is_valid_wiki_link_target, strip_code+from hyperresearch.core.patterns import is_valid_wiki_link_target, mask_code, strip_codedef test_empty_and_whitespace_rejected():@@ -292,3 +292,17 @@ def test_shell_shaped_but_legitimate_refs_still_accepted():assert is_valid_wiki_link_target("C. elegans")assert is_valid_wiki_link_target("yahoo! finance") # '!' mid-string stays legalassert is_valid_wiki_link_target("e.g. v1.2")+++def test_mask_code_blanks_spans_at_their_own_length():+ text = "Run `a [1]` now [2].\nSpan `across\nlines` ends [3].\n```\nfenced [4]\n```\nLast `x`."+ masked = mask_code(text)+ lines, mlines = text.split("\n"), masked.split("\n")+ assert len(mlines) == len(lines)+ assert mlines[0] == "Run " + " " * 7 + " now [2]."+ assert mlines[1] == "Span " + " " * 7 and mlines[2] == " " * 6 + " ends [3]."+ assert mlines[3:6] == ["", "", ""]+ assert mlines[6] == "Last " + " " * 3 + "."+ assert "[1]" not in masked and "[4]" not in masked+ # strip_code is unchanged: a span still becomes one space.+ assert strip_code("Run `a [1]` now") == "Run now"diff --git a/tests/test_core/test_verification.py b/tests/test_core/test_verification.pyindex ad050ba..329bd1a 100644--- a/tests/test_core/test_verification.py+++ b/tests/test_core/test_verification.py@@ -71,9 +71,296 @@ class TestCiteCheckExtraction:report = "A bold claim [[no-such-note]]."pairs = extract_pairs(report, cited_vault.db)assert pairs[0]["note_id"] is None+ assert pairs[0].get("unresolved") == "unknown-note-id"triaged = triage_pairs(pairs, cited_vault.db)assert triaged["dangling"] == 1+ def test_sources_entry_resolves_through_the_notes_own_source_field(self, cited_vault):+ # `note rm` leaves the sources row with note_id NULL; a note written+ # again for the work holds the URL only in its own source field.+ url = "https://repo.example.org/portal/3549507.pdf"+ cited_vault.db.execute(+ "INSERT INTO sources (url, note_id, domain, fetched_at, provider, content_hash) "+ "VALUES (?, NULL, 'repo.example.org', '2026-01-01', 'test', 'y')", (url,)+ )+ cited_vault.db.execute("UPDATE notes SET source = ? WHERE id = 'python-async-patterns'", (url,))+ cited_vault.db.commit()+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] Some Paper (A. Author, 2022). {url}\n"+ assert parse_sources_section(report, cited_vault.db)["1"] == "python-async-patterns"++ @pytest.mark.parametrize("link", [+ "[https://vendor.example.com](https://vendor.example.com)",+ "https://[vendor.example.com]/a",+ "https://vendor.example.com]",+ ])+ def test_link_that_urlsplit_rejects_is_unresolved_not_an_error(self, cited_vault, link):+ # urlsplit raises ValueError on a bracket in the host position.+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] A Vendor Page. {link}\n"+ assert parse_sources_section(report, cited_vault.db) == {"1": None}++ @pytest.mark.parametrize("link", [+ "https://dx.doi.org/10.1145/3313831.3376831",+ "https://doi.org/10.1145/3313831.3376831?utm_source=x",+ ])+ def test_doi_link_resolves_through_the_notes_doi_column(self, cited_vault, link):+ cited_vault.db.execute(+ "UPDATE notes SET doi = '10.1145/3313831.3376831' WHERE id = 'python-async-patterns'"+ )+ cited_vault.db.commit()+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] A Postmortem (CHI 2020). {link}\n"+ assert parse_sources_section(report, cited_vault.db)["1"] == "python-async-patterns"++ def test_arxiv_link_resolves_through_the_notes_doi_column(self, cited_vault):+ # fetch stores extract_doi's result, an arXiv id for arxiv.org links.+ cited_vault.db.execute("UPDATE notes SET doi = 'arXiv:2301.00001' WHERE id = 'python-async-patterns'")+ cited_vault.db.commit()+ report = "Throughput improves 10x [1].\n\n## Sources\n[1] A Preprint. https://arxiv.org/pdf/2301.00001\n"+ assert parse_sources_section(report, cited_vault.db)["1"] == "python-async-patterns"++ @pytest.mark.parametrize("link", [+ "<https://doi.org/10.1145/3313831.3376831>",+ "https://doi.org/10.1145/3313831.3376831;",+ ])+ def test_doi_link_resolves_to_a_note_saved_from_another_address(self, cited_vault, link):+ cited_vault.db.execute(+ "UPDATE notes SET source = ? WHERE id = 'python-async-patterns'",+ ("https://api.example.org/graph/v1/paper/DOI:10.1145/3313831.3376831?fields=title",),+ )+ cited_vault.db.commit()+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] A Postmortem (CHI 2020). {link}\n"+ assert parse_sources_section(report, cited_vault.db)["1"] == "python-async-patterns"++ def test_exact_doi_column_beats_an_address_that_merely_holds_the_doi(self, cited_vault):+ doi = "10.1145/3313831.3376831"+ cited_vault.db.execute("UPDATE notes SET doi = ? WHERE id = 'rust-ownership'", (doi,))+ cited_vault.db.execute(+ "UPDATE notes SET source = ? WHERE id = 'concurrency'", (f"https://cite.example/refs?doi={doi}",)+ )+ cited_vault.db.commit()+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] A Postmortem (CHI 2020). https://doi.org/{doi}\n"+ assert parse_sources_section(report, cited_vault.db)["1"] == "rust-ownership"++ def test_title_match_beats_an_address_that_merely_holds_the_doi(self, cited_vault):+ doi = "10.1145/3313831.3376831"+ cited_vault.db.execute(+ "UPDATE notes SET source = ? WHERE id = 'concurrency'", (f"https://cite.example/refs?doi={doi}",)+ )+ cited_vault.db.commit()+ report = f"Memory safety without GC [1].\n\n## Sources\n[1] Rust Ownership. https://doi.org/{doi}\n"+ assert parse_sources_section(report, cited_vault.db)["1"] == "rust-ownership"++ @pytest.mark.parametrize(("stored", "expected"), [+ ("https://x.example.org/view/10.1145/3313831.3376831", "python-async-patterns"),+ ("https://x.example.org/view/10.1145/3313831.3376831?utm=1", "python-async-patterns"),+ ("https://x.example.org/refs?doi=10.1145/3313831.3376831&from=feed", "python-async-patterns"),+ ("https://x.example.org/view/10.1145/3313831.3376831#abstract", "python-async-patterns"),+ ("https://x.example.org/view/10.1145/3313831.3376831/full", None),+ ], ids=["end", "query", "ampersand", "fragment", "slash"])+ def test_doi_inside_an_address_ends_at_the_url_end_query_or_fragment(+ self, cited_vault, stored, expected+ ):+ """`/` is part of many DOIs, so a path that goes on after the DOI may+ name another one and does not match."""+ cited_vault.db.execute(+ "UPDATE notes SET source = ? WHERE id = 'python-async-patterns'", (stored,)+ )+ cited_vault.db.commit()+ link = "https://doi.org/10.1145/3313831.3376831"+ report = f"Throughput improves 10x [1].\n\n## Sources\n[1] A Postmortem (CHI 2020). {link}\n"+ assert parse_sources_section(report, cited_vault.db)["1"] == expected++ def test_doi_that_prefixes_another_does_not_resolve_to_it(self, cited_vault):+ cited_vault.db.execute("UPDATE notes SET source = 'https://x.org/10.1000/abc1' WHERE id = 'concurrency'")+ cited_vault.db.commit()+ report = "Claim [1] and [2].\n\n## Sources\n[1] One Paper. https://doi.org/10.1000/abc\n[2] Other Paper. https://doi.org/10.1000/abc1\n"+ assert parse_sources_section(report, cited_vault.db) == {"1": None, "2": "concurrency"}++ def test_doi_cut_short_at_a_parenthesis_is_not_looked_up(self, cited_vault):+ # DOI_RE stops at `)` and fetch stores the cut value, so two Lancet 2020+ # papers (Elsevier PII DOIs) share notes.doi; the cut DOI must resolve+ # through the sources row or notes.source only, never the doi column.+ cut = "10.1016/S0140-6736(20"+ huang = "https://dx.doi.org/10.1016/S0140-6736(20)30183-5"+ zhou = "https://doi.org/10.1016/S0140-6736(20)30566-3"+ cited_vault.db.execute("UPDATE notes SET source = ?, doi = ? WHERE id = 'concurrency'", (huang, cut))+ cited_vault.db.execute("UPDATE notes SET source = ?, doi = ? WHERE id = 'rust-ownership'", (zhou, cut))+ cited_vault.db.execute(+ "INSERT INTO sources (url, note_id, domain, fetched_at, provider, content_hash) "+ "VALUES (?, 'rust-ownership', 'doi.org', '2026-01-01', 'test', 'z')", (zhou,)+ )+ cited_vault.db.commit()++ def resolve(link):+ return parse_sources_section(f"Claim [1].\n\n## Sources\n[1] A Paper. {link}\n", cited_vault.db)["1"]++ assert resolve(zhou) == "rust-ownership" # the sources row+ assert resolve(huang) == "concurrency" # notes.source+ absent = "https://doi.org/10.1016/S0140-6736(20)31142-9"+ variants = [+ "https://dx.doi.org/10.1016/S0140-6736(20)30566-3",+ "http://doi.org/10.1016/S0140-6736(20)30566-3",+ "https://doi.org/10.1016/S0140-6736(20)30566-3?via=x",+ absent,+ ]+ assert {v: resolve(v) for v in variants} == dict.fromkeys(variants)++ @pytest.mark.parametrize("tail", ["(20)", "(20"])+ def test_link_cut_after_a_parenthesised_group_is_not_looked_up(self, cited_vault, tail):+ # The caller strips a trailing `)` from the link, which leaves the key+ # fetch stores for every Lancet 2020 paper.+ cited_vault.db.execute(+ "UPDATE notes SET doi = '10.1016/S0140-6736(20' WHERE id = 'concurrency'"+ )+ cited_vault.db.commit()+ report = f"Claim [1].\n\n## Sources\n[1] A Paper. https://doi.org/10.1016/S0140-6736{tail}\n"+ assert parse_sources_section(report, cited_vault.db)["1"] is None++ def test_title_match_beats_a_doi_another_page_holds(self, cited_vault):+ # fetch fills notes.doi from the first DOI in a page's body when the+ # page has no DOI meta tag, so a page that only cites the paper can+ # hold its DOI.+ doi = "10.1145/3313831.3376831"+ cited_vault.db.execute("UPDATE notes SET doi = ? WHERE id = 'concurrency'", (doi,))+ cited_vault.db.commit()+ report = f"Claim [1].\n\n## Sources\n[1] Python Async Patterns. https://doi.org/{doi}\n"+ assert parse_sources_section(report, cited_vault.db)["1"] == "python-async-patterns"++ def test_doi_two_notes_hold_resolves_to_neither(self, cited_vault):+ doi = "10.1145/3313831.3376831"+ cited_vault.db.execute(+ "UPDATE notes SET doi = ? WHERE id IN ('concurrency', 'rust-ownership')", (doi,)+ )+ cited_vault.db.commit()+ report = f"Claim [1].\n\n## Sources\n[1] A Postmortem (CHI 2020). https://doi.org/{doi}\n"+ assert parse_sources_section(report, cited_vault.db)["1"] is None++ def test_sici_doi_via_dx_doi_org_is_not_looked_up(self, cited_vault):+ cited_vault.db.execute(+ "UPDATE notes SET source = ?, doi = '10.1002/(SICI' WHERE id = 'concurrency'",+ ("https://doi.org/10.1002/(SICI)1097-0258(19980415)17:7<857::AID-SIM777>3.0.CO;2-E",),+ )+ cited_vault.db.commit()+ report = (+ "Claim [1].\n\n## Sources\n"+ "[1] Another Article. https://dx.doi.org/10.1002/(SICI)1097-0258(19980430)17:8<900::AID-SIM800>3.0.CO;2-F\n"+ )+ assert parse_sources_section(report, cited_vault.db)["1"] is None++ def test_marker_without_a_sources_entry_is_dangling_not_dropped(self, cited_vault):+ report = (+ "Throughput improves 10x [1][7].\n\n"+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"+ )+ pairs = extract_pairs(report, cited_vault.db)+ assert [p["note_id"] for p in pairs] == ["python-async-patterns", None]+ assert [p.get("marker") for p in pairs] == ["[1]", "[7]"]+ assert pairs[1].get("unresolved") == "no-sources-entry"+ assert triage_pairs(pairs, cited_vault.db)["dangling"] == 1++ def test_numbering_from_zero_or_zero_padded_still_cites(self, cited_vault):+ report = (+ "First claim [01]. Second claim [0]. An id [007].\n\n"+ "## Sources\n[0] Rust Ownership. https://example.com/rust\n"+ "[01] Python Async Patterns. https://example.com/async\n"+ )+ pairs = extract_pairs(report, cited_vault.db)+ assert [(p["sentence"], p["note_id"]) for p in pairs] == [+ ("First claim [01].", "python-async-patterns"),+ ("Second claim [0].", "rust-ownership"),+ ]++ def test_unresolvable_sources_entry_is_labelled_as_such(self, cited_vault):+ report = (+ "Throughput improves 10x [1].\n\n"+ "## Sources\n[1] Unrelated Study. https://example.com/other\n"+ )+ pairs = extract_pairs(report, cited_vault.db)+ assert pairs[0]["note_id"] is None+ assert pairs[0].get("unresolved") == "unresolved-entry"++ def test_bracketed_numbers_without_a_sources_section_are_not_citations(self, cited_vault):+ report = "See item [3] in the list [[python-async-patterns]]."+ pairs = extract_pairs(report, cited_vault.db)+ assert [p["note_id"] for p in pairs] == ["python-async-patterns"]+ assert [p.get("marker") for p in pairs] == ["[[python-async-patterns]]"]++ def test_brackets_in_code_and_link_definitions_are_not_citations(self, cited_vault):+ report = (+ "Intro claim [1].\n\n"+ "```python\narr[0] = x[2]\n```\n\n"+ "Set `v[3]` before the call [1].\n\n"+ "[7]: https://example.com/seven\n\n"+ "| Item | Source |\n|---|---|\n| foo | [5] |\n\n"+ "Run `[[ -n \"$k\" ]]` first [1].\n\n"+ "Run `foo [5]\nbar` now [1].\n\n"+ "Then [12] and [0].\n\n"+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"+ )+ pairs = extract_pairs(report, cited_vault.db)+ # The stored sentence keeps its code span: the patcher edits by it.+ assert [(p["sentence"], p["note_id"]) for p in pairs] == [+ ("Intro claim [1].", "python-async-patterns"),+ ("Set `v[3]` before the call [1].", "python-async-patterns"),+ ("| foo | [5] |", None),+ ("Run `[[ -n \"$k\" ]]` first [1].", "python-async-patterns"),+ ("bar` now [1].", "python-async-patterns"),+ ("Then [12] and [0].", None),+ ]+ assert all(p["sentence"] in report for p in pairs)+ assert [p.get("marker") for p in pairs] == ["[1]", "[1]", "[5]", "[1]", "[1]", "[12]"]+ assert [p.get("unresolved") for p in pairs] == [None, None, "no-sources-entry", None, None, "no-sources-entry"]++ def test_code_span_that_shifts_the_sentence_split_keeps_markers_on_their_sentences(self, cited_vault):+ # The first span sits between a sentence end and a capital, so the+ # stripped side gains a sentence; the last holds a sentence break, so+ # the report side gains one. Equal counts, different splits.+ report = (+ "Setup. `cfg` Loads the file [1].\n\n"+ "Second paragraph has a claim [1].\n\n"+ "The log says `Error. Retry` on failure [1].\n\n"+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"+ )+ pairs = extract_pairs(report, cited_vault.db)+ assert [p["sentence"] for p in pairs] == [+ "Setup. `cfg` Loads the file [1].",+ "Second paragraph has a claim [1].",+ "Retry` on failure [1].",+ ]++ def test_code_spans_that_shift_the_split_both_ways_on_one_line(self, cited_vault):+ # One span sits between a sentence end and a capital, the other holds+ # a sentence break; each marker stays on the report sentence it is in.+ report = (+ "Run it. `make` Builds everything [1]. The tool prints `Done. Exiting` when finished [2].\n\n"+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"+ "[2] Rust Ownership. https://example.com/rust\n"+ )+ pairs = extract_pairs(report, cited_vault.db)+ assert [(p.get("marker"), p["sentence"]) for p in pairs] == [+ ("[1]", "Run it. `make` Builds everything [1]."),+ ("[2]", "Exiting` when finished [2]."),+ ]++ def test_stored_sentence_drops_indentation_and_line_endings(self, cited_vault):+ # The patcher finds a sentence by its text, so it is stored without the+ # line's indentation, trailing blanks or a CRLF's `\r`.+ report = (+ " Indented claim [1].\r\n"+ "Trailing blanks [1]. \r\n\r\n"+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"+ )+ pairs = extract_pairs(report, cited_vault.db)+ assert [p["sentence"] for p in pairs] == ["Indented claim [1].", "Trailing blanks [1]."]++ def test_sentence_break_in_one_code_span_leaves_other_lines_stripped(self, cited_vault):+ report = (+ "Run `foo [5]\nbar` now [1].\n\n"+ "Much later `echo \"Done. Next\"` prints two words [1].\n\n"+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"+ )+ pairs = extract_pairs(report, cited_vault.db)+ assert [p["sentence"] for p in pairs] == ["bar` now [1].", "Next\"` prints two words [1]."]+def test_triage_auto_passes_full_numeric_claim(self, cited_vault):pairs = extract_pairs("Async improves throughput by 10x for network-bound workloads [[python-async-patterns]].",