Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.
Rust ai-nativedescription-logicsformal-verificationinfrastructure-as-codeisabelle-languageknowledge-graphleanlean4mcpmcp-serverontologyowlowl2reasoningrustshaclsparqltheorem-proving
docs(plan): add an optional post-submission benchmark task and the sweep backlog
FABIOTESS5e23efa7b5a2
diff --git a/docs/superpowers/plans/2026-08-24-studio-public-port.md b/docs/superpowers/plans/2026-08-24-studio-public-port.mdindex 551a48c2..1252bba4 100644--- a/docs/superpowers/plans/2026-08-24-studio-public-port.md+++ b/docs/superpowers/plans/2026-08-24-studio-public-port.md@@ -2119,3 +2119,73 @@ the highest-variance item and is scheduled for 25 August so that a corpus problewith four days left rather than one. Tasks 9, 10 and 11 have no dependency on Task 12, so ifthe interface rebuild slips, the previous panels still render over `DemoSource` and the videocan be recorded from them.++---++## Task 14 (optional, after submission): evaluate on an external benchmark++Added 24 August from a literature sweep. This is the only item from that sweep placed in the+plan, because it is the only one that needs no new architecture and produces a maturity claim+the submission can honestly make. It is explicitly optional and must not be started before the+29 August submission is in.++**Files:**+- Create: `demo/bench/wikiconflict_eval.py`+- Create: `demo/bench/RESULTS.md`++**Interfaces:**+- Consumes: the contradiction scanner from Task 5+- Produces: a measured precision and recall for provenance-split contradiction detection against+ a public reference set++- [ ] **Step 1: Confirm the benchmark is usable**++WikiConflict and TrustFuse (K-CAP 2025, code at `Orange-OpenSource/trustfuse`) supply+documents that genuinely disagree, with the disagreements labelled. Fetch it and confirm the+label format can be mapped onto the scanner's `Contradiction` shape. If it cannot be mapped+without distorting either side, stop and record why. A benchmark bent to fit is worth nothing.++- [ ] **Step 2: Run and report honestly**++Report precision, recall and the count of disagreements the scanner missed entirely, with at+least three missed cases quoted in full. The misses are the useful part. A results file that+reports only the score is marketing.++- [ ] **Step 3: Commit**++```bash+cd /Users/fabio/projects/open-ontologies+git add demo/bench/wikiconflict_eval.py demo/bench/RESULTS.md+git commit -m "test(demo): measure contradiction detection against a public reference set"+```++---++## Backlog from the literature sweep, not scheduled++Recorded so they are not lost. None of these are in scope before 29 August. The full sweep is at+`.superpowers/sdd/graphrag-sweep.md`.++1. **Ontology-grounded hyperedge retrieval** (OG-RAG, EMNLP 2025, code at `microsoft/ograg2`).+ Schema-constrained minimal-cover retrieval, genuinely distinct from the existing similarity+ search and community-based global search.+2. **A `filter_candidate_context` primitive** combining the existing SHACL validation and+ closed-world vocabulary check into one pre-model cleaning step (GraphRAG-FI, EMNLP 2025).+3. **A SHACL violation-explanation cache** keyed on a canonical violation signature (xpSHACL,+ VLDB LLM and graph workshop 2025, code at `gcpdev/xpshacl`).+4. **GraphRAG-Bench** (arXiv:2506.05690) to test whether community-based global search earns its+ place on this system's own data.++**The counterargument, which matters more than any of the above.** Three independent 2025+results converge on graph-topology retrieval, community-based global search included, losing to+plain embedding retrieval outside genuine whole-corpus sensemaking questions. One study measured+plain retrieval at 99.4% top-10 accuracy while the graph method pulled 47,000 tokens of context+against 3,700, with 63.3% of its top-one failures being same-chapter noise. This is a direct+challenge to the existing `onto_communities` primitive rather than a general remark about+retrieval.++It bears on the demonstration, not just the roadmap. The comparison panel from Task 12A shows a+grounded answer beside an ungrounded one, and on questions where retrieval alone is sufficient+the ungrounded answer will be just as good. Choosing demonstration questions where the grounded+answer wins for a stateable reason, and being willing to show one where it does not, is more+persuasive to this audience than a panel that always favours the home side.