Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.
Rust ai-nativedescription-logicsformal-verificationinfrastructure-as-codeisabelle-languageknowledge-graphleanlean4mcpmcp-serverontologyowlowl2reasoningrustshaclsparqltheorem-proving
case-study: verifiabench (un-game-able scientific-LLM benchmark)
FABIOTESSa57ef6899cf3
diff --git a/case-studies/verifiabench/.gitignore b/case-studies/verifiabench/.gitignorenew file mode 100644index 00000000..77ac7549--- /dev/null+++ b/case-studies/verifiabench/.gitignore@@ -0,0 +1,3 @@+.venv/+__pycache__/+*.pycdiff --git a/case-studies/verifiabench/BUILD_REPORT.md b/case-studies/verifiabench/BUILD_REPORT.mdnew file mode 100644index 00000000..0818a478--- /dev/null+++ b/case-studies/verifiabench/BUILD_REPORT.md@@ -0,0 +1,62 @@+# BUILD_REPORT: verifiabench++Honest record of the benchmark, the oracle and the limits. Numbers in `results/` are produced by+`src/verifiabench.py` at temperature 0 against a local MLX OpenAI-compatible endpoint.++## What was used (real)++- **Authority:** the Biolink Model (biolink/biolink-model), 868 declared class and slot IRIs, reused+ from the bio-kg-triage case study.+- **Tasks:** 30 well-established gene-disease facts (BRCA1/breast cancer, HTT/Huntington disease,+ CFTR/cystic fibrosis, ...). Each task prompts the model to write Biolink-typed RDF asserting the fact.+- **Models under test (local, open):** Qwen3-Coder-30B-A3B (8bit), Qwen2.5-3B, Llama-3.2-3B,+ gemma-2-2b, Qwen2.5-0.5B, all served over a local MLX server on port 8080. The runner speaks the+ OpenAI chat-completions API, so any endpoint works via `VB_API`.++## The oracle (deterministic, no LLM judge)++For each model output:+1. Strip any markdown code fence.+2. Parse the Turtle with rdflib. This resolves whatever prefix the model bound to the Biolink+ namespace (models used both `biolink:` and `bio:`), so the check is format-robust, not+ prefix-brittle. If it does not parse, fall back to prefix-resolved regex extraction.+3. Collect every predicate and every `rdf:type` object whose IRI is in the Biolink namespace.+4. **Term existence:** each such term is real iff it is in Biolink's declared set; otherwise fabricated.+5. **Structure:** does the graph have a node typed `biolink:Gene`, a node typed `biolink:Disease`,+ and at least one real Biolink association predicate.+6. **raw_ok** = produced any Biolink-namespace term (the fluency proxy). **verified_ok** = real Gene+ AND real Disease AND real association predicate AND zero fabricated terms.++Per-task outputs (including the raw model text, truncated) are saved under `results/` for inspection.++## The result, precisely++- Raw capability saturates: Qwen2.5-3B, Llama-3.2-3B and Qwen3-Coder-30B all score 1.00.+- Verified capability separates: Qwen3-Coder-30B 1.00 (0 fabricated terms); Qwen2.5-3B and+ Llama-3.2-3B 0.00 (30/30 tasks contain a fabricated term, ~50% mean term-existence); gemma-2-2b+ and Qwen2.5-0.5B produce little or no valid Biolink and score 0.00.+- Manual spot-check confirms the oracle: Qwen3-Coder-30B emits real terms+ (`biolink:Gene`, `biolink:Disease`, `biolink:causes`, `biolink:has_gene_product`) in valid+ structure; the mid models emit genuine non-terms (`biolink:has_association`, `biolink:has_role`).++## Limits of the claim (do not overstate)++1. **One task family, one authority.** Biolink gene-disease RDF authoring. The method is+ authority-agnostic (GO, ChEBI, Reactome, EDAM are drop-in), but this run does not yet cover them;+ a multi-domain suite is the next step.+2. **The facts are public, not a secret split.** 30 well-known gene-disease pairs. The honest use is+ relative model comparison and a versioned, inspectable oracle, not a hold-out leaderboard.+3. **Existence and coarse structure, not full semantics.** The oracle checks that terms exist and+ that a Gene, a Disease and a real association predicate are present. It does not check that the+ *specific* predicate is the ideal one for the relation; that is the certified-denotation direction.+4. **Small models are underserved by the strict "zero fabricated terms" rule.** A model that gets the+ structure right but emits one extra invented term scores verified=0. This is deliberate: in a+ scientific pipeline a single fabricated identifier is a real defect. It is stated, not hidden.+5. **Local quantised models.** 4-8bit MLX quantisations, not full-precision or frontier API models;+ the point is the evaluation method and the raw-vs-verified gap, which any endpoint can reproduce.++## Reproducibility++`./run-demo.sh <models...>` reruns against a live endpoint (temperature 0). Model outputs can vary+across server versions, but the raw-vs-verified separation is the stable, reproducible finding.+rdflib at build time; models as listed.diff --git a/case-studies/verifiabench/README.md b/case-studies/verifiabench/README.mdnew file mode 100644index 00000000..fac92204--- /dev/null+++ b/case-studies/verifiabench/README.md@@ -0,0 +1,85 @@+# verifiabench++**An un-game-able benchmark for scientific-workflow LLM reliability: grade the model with a+closed-world oracle, not a fluency check. Fluency is saturated; verified correctness is not.**++Most benchmarks for LLMs on scientific tasks grade the answer with string match or an LLM judge.+Both reward a confident, well-formed answer, so a model that emits a *plausible-but-nonexistent*+gene, ontology class or predicate can score as correct. verifiabench grades differently: every+term a model emits must **exist** in the authority (the real Biolink Model) and the output must+satisfy the task's **structural** constraints. Correctness is set-membership plus constraint+satisfaction, checked deterministically, so fluency cannot buy a point.++This is the reliable-evaluation layer Encode / ARIA Challengescape items on testing LLM reasoning+and reliability on realistic scientific workflows are asking for.++## The result++Deterministic run of [`src/verifiabench.py`](src/verifiabench.py): 30 real gene-disease facts, each+asking a model to write Biolink-typed RDF, graded by the closed-world oracle. Five open models over+a local OpenAI-compatible endpoint (full table in [`results/SUMMARY.md`](results/SUMMARY.md)):++| Model | Raw capability (fluency) | **Verified capability** | Mean term-existence | Fabricated terms |+|---|--:|--:|--:|--:|+| Qwen3-Coder-30B-A3B (8bit) | 1.00 | **1.00** | 1.00 | 0 |+| Qwen2.5-3B | 1.00 | **0.00** | 0.51 | 65 |+| Llama-3.2-3B | 1.00 | **0.00** | 0.51 | 49 |+| gemma-2-2b | 0.23 | **0.00** | 0.06 | 21 |+| Qwen2.5-0.5B | 0.00 | **0.00** | 0.00 | 0 |++**Raw capability** = produced structured Biolink-namespace RDF, what a fluency- or judge-based+benchmark would reward. **Verified capability** = a real `biolink:Gene`, a real `biolink:Disease`, a+real association predicate, and zero fabricated terms.++Three models produce structured Biolink RDF on every task (raw capability 1.00), yet two of them,+Qwen2.5-3B and Llama-3.2-3B, score **0.00 verified**: roughly half of the terms they emit do not+exist in Biolink. Only Qwen3-Coder-30B actually gets it right, 1.00 verified, 0 fabricated terms.+A benchmark that graded on fluency, or asked an LLM to judge, would rank the hallucinating models+near the top. The closed-world oracle separates them cleanly and cannot be gamed by producing+plausible but nonexistent terms.++## Why this is the ownable gap++- **Fluency and correctness are different axes.** Raw capability saturates at 1.00 for any competent+ model; verified capability ranges from 0.00 to 1.00 on the same tasks. Benchmarks that cannot see+ the difference are measuring the wrong thing.+- **The oracle is deterministic and closed-world.** A term is right if and only if it is in the+ authority and used in a valid structure. No string similarity, no LLM judge, no partial credit for+ confident nonsense. This is the [`onto_vocab_check`](../onto-correctness-bench/README.md) principle+ turned into an evaluation.+- **It is un-game-able the way that matters.** You cannot lift the score by writing more fluent RDF;+ you can only lift it by using terms that actually exist. That is exactly the property a scientific+ deployment needs from its model.+- **It validates in both directions.** A genuinely capable model (Qwen3-Coder-30B) scores 100%, so+ the oracle is not just failing everything; it is measuring a real capability the weak models lack.++## Reproduce++```bash+./run-demo.sh # runs the default model on a local MLX/OpenAI endpoint (:8080)+./run-demo.sh "model-a" "model-b" # benchmark specific models+VB_API=https://your-endpoint/v1/chat/completions ./run-demo.sh "gpt-..." # any OpenAI-compatible API+```++Requires Python 3.10+ and an OpenAI-compatible chat endpoint. The runs here used a local MLX server.++## Honest scope++See [`BUILD_REPORT.md`](BUILD_REPORT.md). This is one task family (Biolink gene-disease RDF authoring)+on one authority; the method extends unchanged to GO, ChEBI, Reactome and EDAM, which is the+documented next step toward a multi-domain suite. The 30 facts are well-established, not a held-back+secret split, so the honest use is relative comparison and a public, versioned oracle, not a secret+leaderboard. The oracle checks term existence and a coarse structure (a real Gene, a real Disease, a+real association predicate); deeper semantic correctness (right predicate for the specific relation)+is the certified-denotation direction, not measured here.++---++### Built by Tesseract Academy++We build the evaluation and assurance layer for AI that has to be right, not just fluent. If you are+choosing or deploying a model for scientific or ontology-grounded work, we can benchmark it on what+it actually gets correct.++[gov.tesseract.academy](https://gov.tesseract.academy) · [email protected]+Part of [Open Ontologies](../../README.md) · MIT · real models, real numbers.diff --git a/case-studies/verifiabench/requirements.txt b/case-studies/verifiabench/requirements.txtnew file mode 100644index 00000000..93e24d96--- /dev/null+++ b/case-studies/verifiabench/requirements.txt@@ -0,0 +1,2 @@+rdflib>=7.0+requests>=2.31diff --git a/case-studies/verifiabench/results/SUMMARY.md b/case-studies/verifiabench/results/SUMMARY.mdnew file mode 100644index 00000000..aa80dd3b--- /dev/null+++ b/case-studies/verifiabench/results/SUMMARY.md@@ -0,0 +1,15 @@+# Results summary++Authority: **Biolink Model**. Tasks: **30** real gene-disease facts, each asking a model to write Biolink-typed RDF. Oracle: **closed-world term existence + structural constraints (deterministic, no LLM judge)**.++| Model | Raw capability (fluency) | **Verified capability** | Mean term-existence | Tasks with a fabricated term | Fabricated terms |+|---|--:|--:|--:|--:|--:|+| Qwen3-Coder-30B-A3B-Instruct-8bit | 1.00 | **1.00** | 1.00 | 0/30 | 0 |+| gemma-2-2b-it-4bit | 0.23 | **0.00** | 0.06 | 7/30 | 21 |+| Qwen2.5-0.5B-Instruct-4bit | 0.00 | **0.00** | 0.00 | 0/30 | 0 |+| Qwen2.5-3B-Instruct-4bit | 1.00 | **0.00** | 0.51 | 30/30 | 65 |+| Llama-3.2-3B-Instruct-4bit | 1.00 | **0.00** | 0.51 | 30/30 | 49 |++**Raw capability** = produced structured Biolink-namespace RDF (what a fluency- or LLM-judge-based benchmark would reward). **Verified capability** = a real `biolink:Gene`, a real `biolink:Disease`, a real association predicate, and zero fabricated terms.++**Headline.** Fluency is saturated and verification is not. Three models produce structured Biolink RDF on every task (raw capability 1.00), yet two of them, Qwen2.5-3B-Instruct-4bit, Llama-3.2-3B-Instruct-4bit, score **0.00 verified**: every output invents terms (around half of all terms emitted do not exist in Biolink). Only **Qwen3-Coder-30B-A3B-Instruct-8bit** actually gets it right, 1.00 verified with 0 fabricated terms. A benchmark that graded on fluency or an LLM judge would rank the hallucinating models near the top; the closed-world oracle separates them cleanly, and cannot be gamed by producing plausible but nonexistent terms.\ No newline at end of filediff --git a/case-studies/verifiabench/results/per_task_Llama-3.2-3B-Instruct-4bit.json b/case-studies/verifiabench/results/per_task_Llama-3.2-3B-Instruct-4bit.jsonnew file mode 100644index 00000000..547eb267--- /dev/null+++ b/case-studies/verifiabench/results/per_task_Llama-3.2-3B-Instruct-4bit.json@@ -0,0 +1,561 @@+[+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "CardiofaciocutaneousSyndrome",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/BRAF> a biolink:Gene.\n<http://www.example.org/CardiofaciocutaneousSyndrome> a biolink:Disease.\n\n<http://www.example.org/BRAF> biolink:associatedWith biolink:CardiofaciocutaneousSyndrome.\n\nNote: The Biolink Model is a large ontology, and the classes and predicates used here are just a few examples. The actual classes and predicates used may vary depending on the specific use case.",+ "gene": "BRAF",+ "disease": "cardiofaciocutaneous syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "Lifraumeni",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/TP53> a biolink:Gene.\n<http://www.example.org/LiFraumeni> a biolink:Disease.\n\n<http://www.example.org/TP53> biolink:associatedWith biolink:Lifraumeni.\n\nNote: The Biolink model is a large ontology, and this is a very simple example. In a real-world scenario, you would need to consider the relationships between the classes and predicates in the Biolink model.",+ "gene": "TP53",+ "disease": "Li-Fraumeni syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "CowdenSyndrome",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/PTEN> a biolink:Gene.\n<http://www.example.org/CowdenSyndrome> a biolink:Disease.\n\n<http://www.example.org/PTEN> biolink:associatedWith biolink:CowdenSyndrome.\n\nThis RDF asserts that the gene PTEN is associated with the disease Cowden syndrome using the Biolink Model classes and predicates.",+ "gene": "PTEN",+ "disease": "Cowden syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/EGFR> a biolink:Gene.\n<http://www.example.org/EGFR> biolink:associatedWith biolink:Disease <http://www.example.org/NSCLC>.\n\nNote: The above code is written in Turtle format which is used to represent RDF data in a human-readable format. The actual RDF data would be represented in a different format, such as XML or JSON.",+ "gene": "EGFR",+ "disease": "non-small cell lung carcinoma"+ },+ {+ "parsed_turtle": true,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "isAssociatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/BRCA1> a biolink:Gene.\n<http://www.example.org/BreastCancer> a biolink:Disease.\n<http://www.example.org/BRCA1> biolink:isAssociatedWith biolink:Disease.\n<http://www.example.org/BreastCancer> biolink:isAssociatedWith biolink:Gene.",+ "gene": "BRCA1",+ "disease": "breast cancer"+ },+ {+ "parsed_turtle": true,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "isAssociatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/BRCA2> a biolink:Gene.\n<http://www.example.org/BreastCancer> a biolink:Disease.\n<http://www.example.org/BRCA2> biolink:isAssociatedWith biolink:Disease.\n<http://www.example.org/BreastCancer> biolink:isAssociatedWith biolink:Gene.",+ "gene": "BRCA2",+ "disease": "breast cancer"+ },+ {+ "parsed_turtle": true,+ "n_terms": 1,+ "n_real": 0,+ "n_fake": 1,+ "term_existence_rate": 0.0,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "isAssociatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/gene/KRAS> biolink:isAssociatedWith <http://www.example.org/disease/pancreatic_carcinoma>.",+ "gene": "KRAS",+ "disease": "pancreatic carcinoma"+ },+ {+ "parsed_turtle": true,+ "n_terms": 1,+ "n_real": 0,+ "n_fake": 1,+ "term_existence_rate": 0.0,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/gene/APC> biolink:associatedWith <http://www.example.org/disease/FamilialAdenomatousPolyposis>.",+ "gene": "APC",+ "disease": "familial adenomatous polyposis"+ },+ {+ "parsed_turtle": true,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/RB1> a biolink:Gene.\n<http://www.example.org/Retinoblastoma> a biolink:Disease.\n\n<http://www.example.org/RB1> biolink:associatedWith <http://www.example.org/Retinoblastoma>.",+ "gene": "RB1",+ "disease": "retinoblastoma"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/VHL> a biolink:Gene.\n<http://www.example.org/VHL> biolink:associatedWith biolink:Disease <http://www.example.org/VonHippelLindauDisease>.\n\nNote: The above code is written in Turtle format which is used to represent RDF data in a human-readable format. The RDF data is represented as a series of statements, each of which is a subject-predicate-object triple.",+ "gene": "VHL",+ "disease": "von Hippel-Lindau disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith",+ "hasAssociation"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/cftr> a biolink:Gene.\n<http://www.example.org/cftr> biolink:associatedWith biolink:Disease.\n<http://www.example.org/cftr> biolink:hasAssociation biolink:Disease.\n\nNote: The Biolink Model is a large ontology, and the classes and predicates used here are just a few examples. The actual classes and predicates used may vary depending on the specific use case and the version of the Biolink Model being used.",+ "gene": "CFTR",+ "disease": "cystic fibrosis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "HuntingtonDisease",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/HTT> a biolink:Gene.\n<http://www.example.org/HTT> biolink:associatedWith biolink:Disease.\n<http://www.example.org/HTT> biolink:associatedWith biolink:HuntingtonDisease.\n\nNote: The above code is just an example and does not represent real data. The HTT gene is associated with Huntington's disease in the Biolink model.",+ "gene": "HTT",+ "disease": "Huntington disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 5,+ "n_real": 3,+ "n_fake": 2,+ "term_existence_rate": 0.6,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "GeneAssociation",+ "hasAssociation"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/FragileX> a biolink:Disease.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:Gene.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:Association.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:GeneAssociation.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:GeneAssociation.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:GeneAssociation.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:GeneAssociation.\n<http://www.examp",+ "gene": "FMR1",+ "disease": "fragile X syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "DuchenneMuscularDystrophy",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/DMD> a biolink:Gene.\n<http://www.example.org/DMD> biolink:associatedWith biolink:Disease.\n<http://www.example.org/DMD> biolink:associatedWith biolink:DuchenneMuscularDystrophy.\n\nNote: The Biolink Model classes and predicates used in this example are:\n\n- biolink:Gene\n- biolink:associatedWith\n- biolink:Disease\n- biolink:DuchenneMuscularDystrophy",+ "gene": "DMD",+ "disease": "Duchenne muscular dystrophy"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/HBB> a biolink:Gene.\n<http://www.example.org/BetaThalassemia> a biolink:Disease.\n\n<http://www.example.org/HBB> biolink:associatedWith biolink:Disease.\n<http://www.example.org/BetaThalassemia> biolink:associatedWith biolink:Gene.\n\nThis RDF asserts that the gene HBB is associated with the disease Beta Thalassemia.",+ "gene": "HBB",+ "disease": "beta thalassemia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "HemophiliaA",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/F8> a biolink:Gene.\n<http://www.example.org/HemophiliaA> a biolink:Disease.\n<http://www.example.org/F8> biolink:associatedWith biolink:HemophiliaA.\n\nThis RDF asserts that the gene F8 is associated with the disease Hemophilia A.",+ "gene": "F8",+ "disease": "hemophilia A"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "FamilialHypercholesterolemia",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/LDLR> a biolink:Gene.\n<http://www.example.org/FamilialHypercholesterolemia> a biolink:Disease.\n\n<http://www.example.org/LDLR> biolink:associatedWith biolink:FamilialHypercholesterolemia.\n\nNote: The Biolink Model is a large ontology, and this is a very small example. In a real-world scenario, you would need to include more classes, predicates, and resources to create a complete RDF graph.",+ "gene": "LDLR",+ "disease": "familial hypercholesterolemia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "LynchSyndrome",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/MLH1> a biolink:Gene.\n<http://www.example.org/LynchSyndrome> a biolink:Disease.\n\n<http://www.example.org/MLH1> biolink:associatedWith biolink:LynchSyndrome.\n\nNote: The Biolink model is a large ontology, and the above code snippet is just a small example. Also, the example URIs are fictional and for demonstration purposes only.",+ "gene": "MLH1",+ "disease": "Lynch syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/NF1> a biolink:Gene.\n<http://www.example.org/NeurofibromatosisType1> a biolink:Disease.\n<http://www.example.org/NF1> biolink:associatedWith <http://www.example.org/NeurofibromatosisType1>.\n\nNote: The Biolink Model is a large ontology, and this is a very simple example. In a real-world scenario, you would need to consider the relationships between the classes and predicates in the Biolink Model.",+ "gene": "NF1",+ "disease": "neurofibromatosis type 1"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/smn1> a biolink:Gene.\n<http://www.example.org/sma> a biolink:Disease.\n\n<http://www.example.org/smn1> biolink:associatedWith biolink:Disease.\n<http://www.example.org/sma> biolink:associatedWith biolink:Gene.\n\nThis RDF asserts that the gene SMN1 is associated with the disease Spinal Muscular Atrophy (SMA) using the Biolink Model classes and predicates.",+ "gene": "SMN1",+ "disease": "spinal muscular atrophy"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/gene/PAH> a biolink:Gene.\n<http://www.example.org/disease/Phenylketonuria> a biolink:Disease.\n\n<http://www.example.org/gene/PAH> biolink:associatedWith biolink:Disease.\n<http://www.example.org/disease/Phenylketonuria> biolink:associatedWith biolink:Gene.\n\nThis RDF asserts that the gene PAH is associated with the disease Phenylketonuria.",+ "gene": "PAH",+ "disease": "phenylketonuria"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "G6PD_deficiency",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/G6PD> a biolink:Gene.\n<http://www.example.org/G6PD_deficiency> a biolink:Disease.\n\n<http://www.example.org/G6PD> biolink:associatedWith biolink:G6PD_deficiency.\n\nThis RDF asserts that the gene G6PD is associated with the disease G6PD deficiency using the Biolink Model classes and predicates.",+ "gene": "G6PD",+ "disease": "G6PD deficiency"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "AtaxiaTelangiectasia",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/ATM> a biolink:Gene.\n<http://www.example.org/ATM> biolink:associatedWith biolink:Disease.\n<http://www.example.org/ATM> biolink:associatedWith biolink:AtaxiaTelangiectasia.\n\nNote: The above code is just an example and does not represent real data. The actual data would be represented as URIs.",+ "gene": "ATM",+ "disease": "ataxia telangiectasia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/TSC1> a biolink:Gene.\n<http://www.example.org/TSC1> biolink:associatedWith biolink:Disease <http://www.example.org/TuberousSclerosis>.\n\nThis RDF in Turtle asserts that the gene TSC1 is associated with the disease tuberous sclerosis.",+ "gene": "TSC1",+ "disease": "tuberous sclerosis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/MECP2> a biolink:Gene.\n<http://www.example.org/RettSyndrome> a biolink:Disease.\n\n<http://www.example.org/MECP2> biolink:associatedWith biolink:Disease.\n<http://www.example.org/RettSyndrome> biolink:associatedWith biolink:Gene.\n\nThis RDF asserts that the gene MECP2 is associated with the disease Rett syndrome.",+ "gene": "MECP2",+ "disease": "Rett syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 6,+ "n_real": 2,+ "n_fake": 4,+ "term_existence_rate": 0.3333333333333333,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "GeneAssociation",+ "NeurotransmitterGating",+ "has_association",+ "is_a"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/scn1a> biolink:is_a biolink:Gene.\n<http://www.example.org/scn1a> biolink:is_a biolink:NeurotransmitterGating.\n\n<http://www.example.org/dravet> biolink:is_a biolink:Disease.\n\n<http://www.example.org/dravet> biolink:has_association biolink:GeneAssociation.\n<http://www.example.org/dravet> biolink:has_association biolink:GeneAssociation.\n<http://www.example.org/dravet> biolink:has_association biolink:GeneAssociation.\n\n<http://www.example.org/dravet> biolink:has_association biolink:GeneAssociation.\n<http://www.example.org/",+ "gene": "SCN1A",+ "disease": "Dravet syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 1,+ "n_real": 0,+ "n_fake": 1,+ "term_existence_rate": 0.0,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "isAssociatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/gene/COL1A1> biolink:isAssociatedWith <http://www.example.org/disease/OsteogenesisImperfecta>.",+ "gene": "COL1A1",+ "disease": "osteogenesis imperfecta"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "GaucherDisease",+ "associatedWith"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/GBA> a biolink:Gene.\n<http://www.example.org/GaucherDisease> a biolink:Disease.\n\n<http://www.example.org/GBA> biolink:associatedWith biolink:GaucherDisease.\n\nNote: The Biolink model is a large ontology, and the classes and predicates used here are just a few examples. The actual classes and predicates used may vary depending on the specific use case.",+ "gene": "GBA",+ "disease": "Gaucher disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 5,+ "n_real": 3,+ "n_fake": 2,+ "term_existence_rate": 0.6,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith",+ "hasAssociation"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/HFE> a biolink:Gene.\n<http://www.example.org/HFE> biolink:associatedWith biolink:Disease <http://www.example.org/HereditaryHemochromatosis>.\nbiolink:hasAssociation biolink:Association <http://www.example.org/HFE> <http://www.example.org/HereditaryHemochromatosis>.\nbiolink:hasAssociation biolink:Association <http://www.example.org/HereditaryHemochromatosis> <http://www.example.org/HFE>.",+ "gene": "HFE",+ "disease": "hereditary hemochromatosis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 5,+ "n_real": 2,+ "n_fake": 3,+ "term_existence_rate": 0.4,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "associatedWith",+ "hasDisease",+ "polycysticKidneyDisease"+ ],+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/pkd1> a biolink:Gene.\n<http://www.example.org/pkd1> biolink:associatedWith biolink:Disease.\n<http://www.example.org/pkd1> biolink:hasDisease biolink:polycysticKidneyDisease.\n\nNote: The Biolink Model classes and predicates used in this example are:\n\n- biolink:Gene\n- biolink:associatedWith\n- biolink:hasDisease\n- biolink:Disease\n- biolink:polycysticKidneyDisease",+ "gene": "PKD1",+ "disease": "polycystic kidney disease"+ }+]\ No newline at end of filediff --git a/case-studies/verifiabench/results/per_task_Qwen2.5-0.5B-Instruct-4bit.json b/case-studies/verifiabench/results/per_task_Qwen2.5-0.5B-Instruct-4bit.jsonnew file mode 100644index 00000000..65f42a69--- /dev/null+++ b/case-studies/verifiabench/results/per_task_Qwen2.5-0.5B-Instruct-4bit.json@@ -0,0 +1,482 @@+[+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs200020002000: <",+ "gene": "BRAF",+ "disease": "cardiofaciocutaneous syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs200020002000: <",+ "gene": "TP53",+ "disease": "Li-Fraumeni syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```\n@prefix : <http://example.com/ns#>. @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @pref",+ "gene": "PTEN",+ "disease": "Cowden syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs200020002000: <",+ "gene": "EGFR",+ "disease": "non-small cell lung carcinoma"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs200020002000: <",+ "gene": "BRCA1",+ "disease": "breast cancer"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2: <http://purl.org/dc/elements/2.0/0>\n@prefix dcterms3: <http://purl.org/dc/elements/3.0/0>\n@prefix dcterms4: <http://purl.org/dc/elements/4.0/0>\n@prefix dcterms5: <http://purl.org/dc/elements/5.0/0>\n@prefix dcterms6: <http://purl.org/dc/elements/6.0/0>\n@prefix dcter",+ "gene": "BRCA2",+ "disease": "breast cancer"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```\n@prefix : <http://example.com/ontology#>. @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix xsd: <http://www.w3.org/2001/XMLSchema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://w",+ "gene": "KRAS",+ "disease": "pancreatic carcinoma"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/>\n@prefix dcterms2002: <http://purl.org/dc/elements/2.0/>\n\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix owl2002: <http://purl.org/dc/elements/2.0#>\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix owl2002: <http://purl.org/dc/elements/2.0#>\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix owl2002: <http://pur",+ "gene": "APC",+ "disease": "familial adenomatous polyposis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs200020002000: <",+ "gene": "RB1",+ "disease": "retinoblastoma"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/>\n@prefix dcterms2002: <http://purl.org/dc/elements/2.0/>\n\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix owl2002: <http://purl.org/dc/elements/2.0#>\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix xsd2002: <http://purl.org/dc/elements/2.0#>\n\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix owl2002: <http://pu",+ "gene": "VHL",+ "disease": "von Hippel-Lindau disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```\n@prefix : <http://example.com/ns#>. @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @pref",+ "gene": "CFTR",+ "disease": "cystic fibrosis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```\n@prefix : <http://example.com/ns#>. @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @pref",+ "gene": "HTT",+ "disease": "Huntington disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2: <http://purl.org/dc/elements/2.0/0>\n@prefix dcterms3: <http://purl.org/dc/elements/3.0/0>\n@prefix dcterms4: <http://purl.org/dc/elements/4.0/0>\n@prefix dcterms5: <http://purl.org/dc/elements/5.0/0>\n@prefix dcterms6: <http://purl.org/dc/elements/6.0/0>\n@prefix dcter",+ "gene": "FMR1",+ "disease": "fragile X syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```\n@prefix : <http://example.com/ontology#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix xsd: <http://www.w3.org/2001/XMLSchema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/",+ "gene": "DMD",+ "disease": "Duchenne muscular dystrophy"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n",+ "gene": "HBB",+ "disease": "beta thalassemia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms: <http://www.w3.org/200114270000",+ "gene": "F8",+ "disease": "hemophilia A"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2002: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms2003: <http://purl.org/dc/elements/3.0.0/0>\n@prefix dcterms2004: <http://purl.org/dc/elements/4.0.0/0>\n@prefix dcterms2005: <http://purl.org/dc/elements/5.0.0/0>\n@prefix dcterms2006: <http://purl.org/dc/elemen",+ "gene": "LDLR",+ "disease": "familial hypercholesterolemia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n",+ "gene": "MLH1",+ "disease": "Lynch syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http",+ "gene": "NF1",+ "disease": "neurofibromatosis type 1"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2: <http://purl.org/dc/elements/2.0/0>\n@prefix dcterms3: <http://purl.org/dc/elements/3.0/0>\n@prefix dcterms4: <http://purl.org/dc/elements/4.0/0>\n@prefix dcterms5: <http://purl.org/dc/elements/5.0/0>\n@prefix dcterms6: <http://purl.org/dc/elements/6.0/0>\n@prefix dcter",+ "gene": "SMN1",+ "disease": "spinal muscular atrophy"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2002/0",+ "gene": "PAH",+ "disease": "phenylketonuria"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n",+ "gene": "G6PD",+ "disease": "G6PD deficiency"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema",+ "gene": "ATM",+ "disease": "ataxia telangiectasia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```\n@prefix : <http://example.com/biolink#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix xsd: <http://www.w3.org/2001/XMLSchema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2",+ "gene": "TSC1",+ "disease": "tuberous sclerosis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```\n@prefix : <http://example.com/ontology#>. @prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-ns1#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2000/01/rdf-schema#>. @prefix xsd: <http://www.w3.org/2001/XMLSchema#>. @prefix dcterms: <http://purl.org/dc/elements/1.1/>. @prefix dcterms: <http://purl.org/dc/elements/1.1/>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2000/01/rdf-schema#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2000/01/rdf-schema#>. @prefix rdfs:",+ "gene": "MECP2",+ "disease": "Rett syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2: <http://purl.org/dc/elements/2.0/0>\n@prefix dcterms3: <http://purl.org/dc/elements/3.0/0>\n@prefix dcterms4: <http://purl.org/dc/elements/4.0/0>\n@prefix dcterms5: <http://purl.org/dc/elements/5.0/0>\n@prefix dcterms6: <http://purl.org/dc/elements/6.0/0>\n@prefix dcter",+ "gene": "SCN1A",+ "disease": "Dravet syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```\n@prefix : <http://example.com/ontology#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix xsd: <http://www.w3.org/2001/XMLSchema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/",+ "gene": "COL1A1",+ "disease": "osteogenesis imperfecta"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n",+ "gene": "GBA",+ "disease": "Gaucher disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms2000: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms20002000: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms200020002000: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms200020002",+ "gene": "HFE",+ "disease": "hereditary hemochromatosis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/>\n@prefix dcterms2: <http://purl.org/dc/elements/2.0/>\n\n@prefix rdfs2: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl2: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl1: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl21: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl11: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl12: <http:",+ "gene": "PKD1",+ "disease": "polycystic kidney disease"+ }+]\ No newline at end of filediff --git a/case-studies/verifiabench/results/per_task_Qwen2.5-3B-Instruct-4bit.json b/case-studies/verifiabench/results/per_task_Qwen2.5-3B-Instruct-4bit.jsonnew file mode 100644index 00000000..e40346b0--- /dev/null+++ b/case-studies/verifiabench/results/per_task_Qwen2.5-3B-Instruct-4bit.json@@ -0,0 +1,552 @@+[+ {+ "parsed_turtle": false,+ "n_terms": 34,+ "n_real": 3,+ "n_fake": 31,+ "term_existence_rate": 0.08823529411764706,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "activating_role",+ "association_type",+ "conjunctional_mutation",+ "copy_number_change",+ "copy_number_gain",+ "copy_number_loss"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/BRAF> a bio:Gene;\n bio:has_association [\n a bio:Association;\n bio:association_type bio:genetic_association;\n bio:target <http://example.org/BRAF>;\n bio:target_type bio:gene;\n bio:target_role bio:activating_role;\n bio:target_mechanism bio:mutation;\n bio:target_mechanism_type bio:genetic_mutation;\n bio:target_mechanism_subtype bio:missense_mutation;\n bio:target_mechanism_subtype bio:germline_mutation;\n bio:target_mechanism_subtype bio:sporadic_m",+ "gene": "BRAF",+ "disease": "cardiofaciocutaneous syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/TP53> a bio:Gene;\n bio:has_associated_disease <http://example.org/disease/Li-Fraumeni-syndrome>.\n```",+ "gene": "TP53",+ "disease": "Li-Fraumeni syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/PTEN> a bio:Gene;\n bio:has_associated_disease <http://example.org/CowdenSyndrome>.\n```",+ "gene": "PTEN",+ "disease": "Cowden syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 5,+ "n_real": 3,+ "n_fake": 2,+ "term_existence_rate": 0.6,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_association",+ "has_role"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/EGFR> a bio:Gene;\n bio:has_association [\n a bio:Association;\n bio:has_role bio:has_association_role;\n bio:has_member <http://example.org/non-small-cell-lung-carcinoma>;\n bio:has_member bio:Gene;\n bio:has_role bio:has_association_role;\n ].\n```",+ "gene": "EGFR",+ "disease": "non-small cell lung carcinoma"+ },+ {+ "parsed_turtle": false,+ "n_terms": 5,+ "n_real": 4,+ "n_fake": 1,+ "term_existence_rate": 0.8,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "has_association"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/BRCA1> a bio:Gene ;\n bio:has_association [\n a bio:Association ;\n bio:has_member <http://example.org/BRCA1> ;\n bio:has_member <http://example.org/breastCancer> .\n ] .\n<http://example.org/breastCancer> a bio:Disease .\n```",+ "gene": "BRCA1",+ "disease": "breast cancer"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/BRCA2> a bio:Gene ;\n bio:has_associated_disease <http://example.org/breast_cancer> .\n```",+ "gene": "BRCA2",+ "disease": "breast cancer"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/KRAS> a bio:Gene ;\n bio:has_associated_disease <http://example.org/PancreaticCarcinoma>.\n```",+ "gene": "KRAS",+ "disease": "pancreatic carcinoma"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/APC> a bio:Gene;\n bio:has_associated_disease <http://example.org/familial_adenomatous_polyposis>.\n```",+ "gene": "APC",+ "disease": "familial adenomatous polyposis"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/RB1> a bio:Gene;\n bio:has_associated_disease bio:Retinoblastoma.\n```",+ "gene": "RB1",+ "disease": "retinoblastoma"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/VHL> a bio:Gene;\n bio:has_associated_disease <http://example.org/vonHippelLindauDisease>.\n```",+ "gene": "VHL",+ "disease": "von Hippel-Lindau disease"+ },+ {+ "parsed_turtle": true,+ "n_terms": 3,+ "n_real": 1,+ "n_fake": 2,+ "term_existence_rate": 0.3333333333333333,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "association",+ "has_association"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/CFTR> a bio:Gene;\n bio:has_association <http://example.org/disease/cystic-fibrosis>;\n bio:association bio:genomic_association_of_disease.\n```",+ "gene": "CFTR",+ "disease": "cystic fibrosis"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/HTT> a bio:Gene;\n bio:has_associated_disease <http://example.org/HuntingtonDisease>.\n```",+ "gene": "HTT",+ "disease": "Huntington disease"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/FMR1> a bio:Gene;\n bio:has_associated_disease bio:FragileXSyndrome.\n```",+ "gene": "FMR1",+ "disease": "fragile X syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/DMD> a bio:Gene ;\n bio:has_associated_disease <http://example.org/DuchenneMuscularDystrophy> .\n```",+ "gene": "DMD",+ "disease": "Duchenne muscular dystrophy"+ },+ {+ "parsed_turtle": true,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "has_association",+ "has_association_type"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/HBB> a bio:Gene ;\n bio:has_association <http://example.org/betaThalassemia> ;\n bio:has_association_type bio:genetically_associated_with .\n<http://example.org/betaThalassemia> a bio:Disease .\n```",+ "gene": "HBB",+ "disease": "beta thalassemia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/F8> a bio:Gene ;\n bio:has_associated_disease bio:Disease<http://example.org/hemophiliaA> .\n```",+ "gene": "F8",+ "disease": "hemophilia A"+ },+ {+ "parsed_turtle": false,+ "n_terms": 5,+ "n_real": 3,+ "n_fake": 2,+ "term_existence_rate": 0.6,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "has_association",+ "is_associated_with"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/LDLR> bio:has_association bio:Association<http://example.org/familial_hypercholesterolemia>.\n<http://example.org/familial_hypercholesterolemia> a bio:Disease.\n<http://example.org/LDLR> a bio:Gene.\n<http://example.org/LDLR> bio:is_associated_with <http://example.org/familial_hypercholesterolemia>.\n```",+ "gene": "LDLR",+ "disease": "familial hypercholesterolemia"+ },+ {+ "parsed_turtle": true,+ "n_terms": 4,+ "n_real": 3,+ "n_fake": 1,+ "term_existence_rate": 0.75,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/MLH1> a bio:Gene ;\n bio:has_associated_disease [ a bio:Disease ;\n bio:name \"Lynch syndrome\" ] .\n```",+ "gene": "MLH1",+ "disease": "Lynch syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/NF1> a bio:Gene ;\n bio:has_associated_disease <http://example.org/disease/neurofibromatosis_type_1>.\n```",+ "gene": "NF1",+ "disease": "neurofibromatosis type 1"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/SMN1> a bio:Gene;\n bio:has_associated_disease <http://example.org/disease/spinal-muscular-atrophy>.\n```",+ "gene": "SMN1",+ "disease": "spinal muscular atrophy"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_association"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/PAH> a bio:Gene ;\n bio:has_association <http://example.org/disease/Phenylketonuria> .\n```",+ "gene": "PAH",+ "disease": "phenylketonuria"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/G6PD> a bio:Gene ;\n bio:has_associated_disease <http://example.org/G6PD_deficiency>.\n```",+ "gene": "G6PD",+ "disease": "G6PD deficiency"+ },+ {+ "parsed_turtle": true,+ "n_terms": 1,+ "n_real": 0,+ "n_fake": 1,+ "term_existence_rate": 0.0,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/ATM> bio:has_associated_disease bio:AtaxiaTelangiectasia .\n```",+ "gene": "ATM",+ "disease": "ataxia telangiectasia"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/TSC1> a bio:Gene ;\n bio:has_associated_disease <http://example.org/tuberous_sclerosis> .\n```",+ "gene": "TSC1",+ "disease": "tuberous sclerosis"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/MECP2> a bio:Gene;\n bio:has_associated_disease <http://example.org/disease/RettSyndrome>.\n```",+ "gene": "MECP2",+ "disease": "Rett syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/SCN1A> a bio:Gene;\n bio:has_associated_disease <http://example.org/DravetSyndrome>.\n```",+ "gene": "SCN1A",+ "disease": "Dravet syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 4,+ "n_real": 2,+ "n_fake": 2,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "association_target",+ "has_association"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/COL1A1> a bio:Gene;\n bio:has_association bio:Association;\n bio:association_target <http://example.org/OsteogenesisImperfecta>.\n\n<http://example.org/OsteogenesisImperfecta> a bio:Disease.\n```",+ "gene": "COL1A1",+ "disease": "osteogenesis imperfecta"+ },+ {+ "parsed_turtle": true,+ "n_terms": 2,+ "n_real": 1,+ "n_fake": 1,+ "term_existence_rate": 0.5,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/GBA> a bio:Gene;\n bio:has_associated_disease <http://example.org/Gaucher_disease>.\n```",+ "gene": "GBA",+ "disease": "Gaucher disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/HFE> a bio:Gene ;\n bio:has_associated_disease bio:Disease<http://example.org/hereditary_hemochromatosis>.\n```",+ "gene": "HFE",+ "disease": "hereditary hemochromatosis"+ },+ {+ "parsed_turtle": true,+ "n_terms": 3,+ "n_real": 2,+ "n_fake": 1,+ "term_existence_rate": 0.6666666666666666,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": false,+ "fake_terms": [+ "has_associated_disease"+ ],+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/PKD1> a bio:Gene;\n bio:has_associated_disease <http://example.org/polycystic_kidney_disease>.\n<http://example.org/polycystic_kidney_disease> a bio:Disease.\n```",+ "gene": "PKD1",+ "disease": "polycystic kidney disease"+ }+]\ No newline at end of filediff --git a/case-studies/verifiabench/results/per_task_Qwen3-Coder-30B-A3B-Instruct-8bit.json b/case-studies/verifiabench/results/per_task_Qwen3-Coder-30B-A3B-Instruct-8bit.jsonnew file mode 100644index 00000000..6971e1b0--- /dev/null+++ b/case-studies/verifiabench/results/per_task_Qwen3-Coder-30B-A3B-Instruct-8bit.json@@ -0,0 +1,482 @@+[+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/BRAF> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/BRAF_protein> .\n\n<https://example.org/disease/cardiofaciocutaneous_syndrome> a biolink:Disease .\n\n<https://example.org/gene/BRAF> biolink:causes <https://example.org/disease/cardiofaciocutaneous_syndrome> .\n```",+ "gene": "BRAF",+ "disease": "cardiofaciocutaneous syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 5,+ "n_real": 5,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://identifiers.org/ncbigene:7157> a biolink:Gene ;\n biolink:has_gene_product <https://identifiers.org/ncbigene:7157> .\n\n<https://identifiers.org/omim:151627> a biolink:Disease ;\n biolink:has_phenotype <https://identifiers.org/omim:151627> .\n\n<https://identifiers.org/ncbigene:7157> biolink:causes <https://identifiers.org/omim:151627> .\n```",+ "gene": "TP53",+ "disease": "Li-Fraumeni syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 5,+ "n_real": 5,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://identifiers.org/ncbigene:5728> a biolink:Gene ;\n biolink:has_gene_product <https://identifiers.org/ncbigene:5728> .\n\n<https://identifiers.org/omim:601209> a biolink:Disease ;\n biolink:has_phenotype <https://identifiers.org/omim:601209> .\n\n<https://identifiers.org/ncbigene:5728> biolink:causes <https://identifiers.org/omim:601209> .\n```",+ "gene": "PTEN",+ "disease": "Cowden syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/EGFR> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/EGFR_protein> .\n\n<https://example.org/non-small_cell_lung_carcinoma> a biolink:Disease .\n\n<https://example.org/EGFR> biolink:causes <https://example.org/non-small_cell_lung_carcinoma> .\n```",+ "gene": "EGFR",+ "disease": "non-small cell lung carcinoma"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/BRCA1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/BRCA1_protein> .\n\n<https://example.org/breast_cancer> a biolink:Disease .\n\n<https://example.org/BRCA1> biolink:causes <https://example.org/breast_cancer> .\n```",+ "gene": "BRCA1",+ "disease": "breast cancer"+ },+ {+ "parsed_turtle": false,+ "n_terms": 7,+ "n_real": 7,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://identifiers.org/ncbigene:675> a biolink:Gene ;\n biolink:has_gene_product <https://identifiers.org/ncbigene:675#protein> .\n\n<https://identifiers.org/ncbigene:675#protein> a biolink:Protein ;\n biolink:associated_with <https://identifiers.org/mondo:0004975> .\n\n<https://identifiers.org/mondo:0004975> a biolink:Disease ;\n biolink:name \"breast cancer\" .\n\n<https://identifiers.org/ncbigene:675> biolink:causes <https://identifiers.org/mondo:0004975> .\n```",+ "gene": "BRCA2",+ "disease": "breast cancer"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/KRAS> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/KRAS_protein> .\n\n<https://example.org/disease/pancreatic_carcinoma> a biolink:Disease .\n\n<https://example.org/gene/KRAS> biolink:causes <https://example.org/disease/pancreatic_carcinoma> .\n```",+ "gene": "KRAS",+ "disease": "pancreatic carcinoma"+ },+ {+ "parsed_turtle": false,+ "n_terms": 3,+ "n_real": 3,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/APC> a biolink:Gene ;\n biolink:associated_with <https://example.org/disease/familial_adenomatous_polyposis> .\n\n<https://example.org/disease/familial_adenomatous_polyposis> a biolink:Disease .\n```",+ "gene": "APC",+ "disease": "familial adenomatous polyposis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/RB1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/RB1_protein> .\n\n<https://example.org/disease/retinoblastoma> a biolink:Disease .\n\n<https://example.org/gene/RB1> biolink:causes <https://example.org/disease/retinoblastoma> .\n```",+ "gene": "RB1",+ "disease": "retinoblastoma"+ },+ {+ "parsed_turtle": true,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/vocab/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n\n<https://example.org/gene/VHL> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/VHL> .\n\n<https://example.org/disease/von_Hippel-Lindau_disease> a biolink:Disease .\n\n<https://example.org/gene/VHL> biolink:causes <https://example.org/disease/von_Hippel-Lindau_disease> .\n```",+ "gene": "VHL",+ "disease": "von Hippel-Lindau disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/CFTR> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/CFTR> .\n\n<https://example.org/disease/cystic_fibrosis> a biolink:Disease .\n\n<https://example.org/gene/CFTR> biolink:causes <https://example.org/disease/cystic_fibrosis> .\n```",+ "gene": "CFTR",+ "disease": "cystic fibrosis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/HTT> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/HTT> .\n\n<https://example.org/disease/HuntingtonDisease> a biolink:Disease .\n\n<https://example.org/gene/HTT> biolink:causes <https://example.org/disease/HuntingtonDisease> .\n```",+ "gene": "HTT",+ "disease": "Huntington disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/FMR1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/FMR1_protein> .\n\n<https://example.org/fragile_X_syndrome> a biolink:Disease .\n\n<https://example.org/FMR1> biolink:causes <https://example.org/fragile_X_syndrome> .\n```",+ "gene": "FMR1",+ "disease": "fragile X syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/DMD> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/DMD_protein> .\n\n<https://example.org/disease/Duchenne_muscular_dystrophy> a biolink:Disease .\n\n<https://example.org/gene/DMD> biolink:causes <https://example.org/disease/Duchenne_muscular_dystrophy> .\n```",+ "gene": "DMD",+ "disease": "Duchenne muscular dystrophy"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/HBB> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/HBB_protein> .\n\n<https://example.org/beta_thalassemia> a biolink:Disease .\n\n<https://example.org/HBB> biolink:causes <https://example.org/beta_thalassemia> .\n```",+ "gene": "HBB",+ "disease": "beta thalassemia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/F8> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/F8> .\n\n<https://example.org/disease/hemophilia_A> a biolink:Disease .\n\n<https://example.org/gene/F8> biolink:causes <https://example.org/disease/hemophilia_A> .\n```",+ "gene": "F8",+ "disease": "hemophilia A"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/LDLR> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/LDLR_protein> .\n\n<https://example.org/disease/familial_hypercholesterolemia> a biolink:Disease .\n\n<https://example.org/gene/LDLR> biolink:causes <https://example.org/disease/familial_hypercholesterolemia> .\n```",+ "gene": "LDLR",+ "disease": "familial hypercholesterolemia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/MLH1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/MLH1> .\n\n<https://example.org/disease/LynchSyndrome> a biolink:Disease .\n\n<https://example.org/gene/MLH1> biolink:causes <https://example.org/disease/LynchSyndrome> .\n```",+ "gene": "MLH1",+ "disease": "Lynch syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/NF1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/NF1_protein> .\n\n<https://example.org/disease/neurofibromatosis_type_1> a biolink:Disease .\n\n<https://example.org/gene/NF1> biolink:causes <https://example.org/disease/neurofibromatosis_type_1> .\n```",+ "gene": "NF1",+ "disease": "neurofibromatosis type 1"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/SMN1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/SMN1_protein> .\n\n<https://example.org/spinal_muscular_atrophy> a biolink:Disease .\n\n<https://example.org/SMN1> biolink:causes <https://example.org/spinal_muscular_atrophy> .\n```",+ "gene": "SMN1",+ "disease": "spinal muscular atrophy"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/PAH> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/PAH> .\n\n<https://example.org/disease/phenylketonuria> a biolink:Disease .\n\n<https://example.org/gene/PAH> biolink:causes <https://example.org/disease/phenylketonuria> .\n```",+ "gene": "PAH",+ "disease": "phenylketonuria"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/G6PD> a biolink:Gene ;\n biolink:associated_with <https://example.org/disease/G6PD_deficiency> .\n\n<https://example.org/disease/G6PD_deficiency> a biolink:Disease ;\n biolink:has_gene <https://example.org/gene/G6PD> .\n```",+ "gene": "G6PD",+ "disease": "G6PD deficiency"+ },+ {+ "parsed_turtle": false,+ "n_terms": 5,+ "n_real": 5,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://identifiers.org/ncbigene:472> a biolink:Gene ;\n biolink:has_gene_product <https://identifiers.org/ncbigene:472> .\n\n<https://identifiers.org/omim:208900> a biolink:Disease ;\n biolink:has_phenotype <https://identifiers.org/omim:208900> .\n\n<https://identifiers.org/ncbigene:472> biolink:causes <https://identifiers.org/omim:208900> .\n```",+ "gene": "ATM",+ "disease": "ataxia telangiectasia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/TSC1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/TSC1_protein> .\n\n<https://example.org/disease/tuberous_sclerosis> a biolink:Disease .\n\n<https://example.org/gene/TSC1> biolink:causes <https://example.org/disease/tuberous_sclerosis> .\n```",+ "gene": "TSC1",+ "disease": "tuberous sclerosis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/MECP2> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/MECP2> .\n\n<https://example.org/disease/Rett_syndrome> a biolink:Disease .\n\n<https://example.org/gene/MECP2> biolink:causes <https://example.org/disease/Rett_syndrome> .\n```",+ "gene": "MECP2",+ "disease": "Rett syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/SCN1A> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/SCN1A_protein> .\n\n<https://example.org/disease/Dravet_syndrome> a biolink:Disease .\n\n<https://example.org/gene/SCN1A> biolink:causes <https://example.org/disease/Dravet_syndrome> .\n```",+ "gene": "SCN1A",+ "disease": "Dravet syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/COL1A1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/COL1A1_protein> .\n\n<https://example.org/disease/osteogenesis_imperfecta> a biolink:Disease .\n\n<https://example.org/gene/COL1A1> biolink:causes <https://example.org/disease/osteogenesis_imperfecta> .\n```",+ "gene": "COL1A1",+ "disease": "osteogenesis imperfecta"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/GBA> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/GBA_protein> .\n\n<https://example.org/disease/Gaucher_disease> a biolink:Disease .\n\n<https://example.org/gene/GBA> biolink:causes <https://example.org/disease/Gaucher_disease> .\n```",+ "gene": "GBA",+ "disease": "Gaucher disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/HFE_gene> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/HFE_protein> .\n\n<https://example.org/hereditary_hemochromatosis> a biolink:Disease .\n\n<https://example.org/HFE_gene> biolink:causes <https://example.org/hereditary_hemochromatosis> .\n```",+ "gene": "HFE",+ "disease": "hereditary hemochromatosis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 4,+ "n_fake": 0,+ "term_existence_rate": 1.0,+ "raw_ok": true,+ "verified_ok": true,+ "has_gene": true,+ "has_disease": true,+ "has_assoc": true,+ "fake_terms": [],+ "output": "```turtle\n<https://example.org/gene/PKD1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/PKD1_protein> .\n\n<https://example.org/disease/Polycystic_Kidney_Disease> a biolink:Disease .\n\n<https://example.org/gene/PKD1> biolink:causes <https://example.org/disease/Polycystic_Kidney_Disease> .\n```",+ "gene": "PKD1",+ "disease": "polycystic kidney disease"+ }+]\ No newline at end of filediff --git a/case-studies/verifiabench/results/per_task_gemma-2-2b-it-4bit.json b/case-studies/verifiabench/results/per_task_gemma-2-2b-it-4bit.jsonnew file mode 100644index 00000000..31c60179--- /dev/null+++ b/case-studies/verifiabench/results/per_task_gemma-2-2b-it-4bit.json@@ -0,0 +1,510 @@+[+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 1,+ "n_fake": 3,+ "term_existence_rate": 0.25,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "BRAF",+ "associatedDisease",+ "cardiofaciocutaneous"+ ],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:BRAF a biolink:Gene ;\nbiolink:associatedDisease biolink:cardiofaciocutaneous syndrome .\n``` \n",+ "gene": "BRAF",+ "disease": "cardiofaciocutaneous syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 1,+ "n_fake": 3,+ "term_existence_rate": 0.25,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "Li",+ "TP53",+ "associatedDisease"+ ],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:TP53 a biolink:Gene ;\nbiolink:associatedDisease biolink:Li-Fraumeni syndrome .\n``` \n",+ "gene": "TP53",+ "disease": "Li-Fraumeni syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 1,+ "n_fake": 3,+ "term_existence_rate": 0.25,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "Cowden",+ "PTEN",+ "associatedDisease"+ ],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:PTEN a biolink:Gene ;\nbiolink:associatedDisease biolink:Cowden syndrome .\n``` \n",+ "gene": "PTEN",+ "disease": "Cowden syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:EGFR a biolink:Gene ;\nbiolink:associatedDisease biolink:nonSmallCellLungCarcinoma .\n``` \n",+ "gene": "EGFR",+ "disease": "non-small cell lung carcinoma"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:BRCA1 a biolink:Gene ;\nbiolink:associatedDisease biolink:breastCancer .\n``` \n",+ "gene": "BRCA1",+ "disease": "breast cancer"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:BRCA2 a biolink:Gene ;\nbiolink:associatedDisease biolink:BreastCancer .\n``` \n",+ "gene": "BRCA2",+ "disease": "breast cancer"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:KRAS a biolink:Gene ;\nbiolink:associatedDisease biolink:pancreaticCarcinoma .\n``` \n",+ "gene": "KRAS",+ "disease": "pancreatic carcinoma"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:APC a biolink:Gene ;\nbiolink:associatedDisease biolink:FamilialAdenomatousPolyposis .\n``` \n",+ "gene": "APC",+ "disease": "familial adenomatous polyposis"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:RB1 a biolink:Gene ;\nbiolink:associatedDisease biolink:Retinoblastoma .\n``` \n",+ "gene": "RB1",+ "disease": "retinoblastoma"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:VHL a biolink:Gene ;\nbiolink:associatedDisease biolink:vonHippelLindauDisease .\n``` \n",+ "gene": "VHL",+ "disease": "von Hippel-Lindau disease"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:CFTR a biolink:Gene ;\nbiolink:associatedDisease biolink:CysticFibrosis .\n``` \n",+ "gene": "CFTR",+ "disease": "cystic fibrosis"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:HTT rdf:type biolink:Gene .\nbiolink:HTT biolink:associatedWith biolink:HuntingtonDisease .\n``` \n",+ "gene": "HTT",+ "disease": "Huntington disease"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 1,+ "n_fake": 3,+ "term_existence_rate": 0.25,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "FMR1",+ "FragileX",+ "associatedDisease"+ ],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:FMR1 a biolink:Gene ;\nbiolink:associatedDisease biolink:FragileX syndrome .\n``` \n",+ "gene": "FMR1",+ "disease": "fragile X syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:DMD a biolink:Gene ;\nbiolink:associatedDisease biolink:DuchenneMuscularDystrophy .\n``` \n",+ "gene": "DMD",+ "disease": "Duchenne muscular dystrophy"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:HBB a biolink:Gene ;\nbiolink:associatedDisease biolink:BetaThalassemia .\n``` \n",+ "gene": "HBB",+ "disease": "beta thalassemia"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:F8 a biolink:Gene ;\nbiolink:associatedDisease biolink:HemophiliaA .\n``` \n",+ "gene": "F8",+ "disease": "hemophilia A"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:LDLR a biolink:Gene ;\nbiolink:associatedDisease biolink:FamilialHypercholesterolemia .\n``` \n",+ "gene": "LDLR",+ "disease": "familial hypercholesterolemia"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 1,+ "n_fake": 3,+ "term_existence_rate": 0.25,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "Lynch",+ "MLH1",+ "associatedDisease"+ ],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:MLH1 a biolink:Gene ;\nbiolink:associatedDisease biolink:Lynch syndrome .\n``` \n",+ "gene": "MLH1",+ "disease": "Lynch syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:NF1 a biolink:Gene ;\nbiolink:associatedDisease biolink:NeurofibromatosisType1 .\n``` \n",+ "gene": "NF1",+ "disease": "neurofibromatosis type 1"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:SMN1 a biolink:Gene ;\nbiolink:associatedDisease biolink:spinalMuscularAtrophy .\n``` \n",+ "gene": "SMN1",+ "disease": "spinal muscular atrophy"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:PAH rdf:type biolink:Gene .\nbiolink:PAH biolink:associatedWith biolink:Phenylketonuria .\n``` \n",+ "gene": "PAH",+ "disease": "phenylketonuria"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:G6PD rdf:type biolink:Gene .\nbiolink:G6PD biolink:associatedWith biolink:G6PD_deficiency .\n``` \n",+ "gene": "G6PD",+ "disease": "G6PD deficiency"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:ATM a biolink:Gene ;\nbiolink:associatedDisease biolink:ataxiaTelangiectasia .\n``` \n",+ "gene": "ATM",+ "disease": "ataxia telangiectasia"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:TSC1 a biolink:Gene ;\nbiolink:associatedDisease biolink:TuberousSclerosis .\n``` \n",+ "gene": "TSC1",+ "disease": "tuberous sclerosis"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 1,+ "n_fake": 3,+ "term_existence_rate": 0.25,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "MECP2",+ "Rett",+ "associatedDisease"+ ],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:MECP2 a biolink:Gene ;\nbiolink:associatedDisease biolink:Rett syndrome .\n``` \n",+ "gene": "MECP2",+ "disease": "Rett syndrome"+ },+ {+ "parsed_turtle": false,+ "n_terms": 4,+ "n_real": 1,+ "n_fake": 3,+ "term_existence_rate": 0.25,+ "raw_ok": true,+ "verified_ok": false,+ "has_gene": true,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [+ "Dravet",+ "SCN1A",+ "associatedDisease"+ ],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:SCN1A a biolink:Gene ;\nbiolink:associatedDisease biolink:Dravet syndrome .\n``` \n",+ "gene": "SCN1A",+ "disease": "Dravet syndrome"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:COL1A1 a biolink:Gene ;\nbiolink:associatedDisease biolink:OsteogenesisImperfecta .\n``` \n",+ "gene": "COL1A1",+ "disease": "osteogenesis imperfecta"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:GBA rdf:type biolink:Gene .\nbiolink:GBA biolink:associatedWith biolink:GaucherDisease .\n``` \n",+ "gene": "GBA",+ "disease": "Gaucher disease"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:HFE a biolink:Gene ;\nbiolink:associatedDisease biolink:HereditaryHemochromatosis .\n``` \n",+ "gene": "HFE",+ "disease": "hereditary hemochromatosis"+ },+ {+ "parsed_turtle": true,+ "n_terms": 0,+ "n_real": 0,+ "n_fake": 0,+ "term_existence_rate": 0.0,+ "raw_ok": false,+ "verified_ok": false,+ "has_gene": false,+ "has_disease": false,+ "has_assoc": false,+ "fake_terms": [],+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:PKD1 a biolink:Gene ;\nbiolink:associatedDisease biolink:PolycysticKidneyDisease .\n``` \n",+ "gene": "PKD1",+ "disease": "polycystic kidney disease"+ }+]\ No newline at end of filediff --git a/case-studies/verifiabench/results/results.json b/case-studies/verifiabench/results/results.jsonnew file mode 100644index 00000000..1011d579--- /dev/null+++ b/case-studies/verifiabench/results/results.json@@ -0,0 +1,57 @@+{+ "authority": "Biolink Model",+ "n_tasks": 30,+ "oracle": "closed-world term existence + structural constraints (deterministic, no LLM judge)",+ "leaderboard": [+ {+ "model": "mlx-community/Qwen3-Coder-30B-A3B-Instruct-8bit",+ "n_tasks": 30,+ "raw_capability": 1.0,+ "verified_capability": 1.0,+ "mean_term_existence": 1.0,+ "tasks_with_any_fabricated_term": 0,+ "total_fabricated_terms": 0,+ "verification_gap": 0.0+ },+ {+ "model": "mlx-community/gemma-2-2b-it-4bit",+ "n_tasks": 30,+ "raw_capability": 0.233,+ "verified_capability": 0.0,+ "mean_term_existence": 0.058,+ "tasks_with_any_fabricated_term": 7,+ "total_fabricated_terms": 21,+ "verification_gap": 0.233+ },+ {+ "model": "mlx-community/Qwen2.5-0.5B-Instruct-4bit",+ "n_tasks": 30,+ "raw_capability": 0.0,+ "verified_capability": 0.0,+ "mean_term_existence": 0.0,+ "tasks_with_any_fabricated_term": 0,+ "total_fabricated_terms": 0,+ "verification_gap": 0.0+ },+ {+ "model": "mlx-community/Qwen2.5-3B-Instruct-4bit",+ "n_tasks": 30,+ "raw_capability": 1.0,+ "verified_capability": 0.0,+ "mean_term_existence": 0.506,+ "tasks_with_any_fabricated_term": 30,+ "total_fabricated_terms": 65,+ "verification_gap": 1.0+ },+ {+ "model": "mlx-community/Llama-3.2-3B-Instruct-4bit",+ "n_tasks": 30,+ "raw_capability": 1.0,+ "verified_capability": 0.0,+ "mean_term_existence": 0.509,+ "tasks_with_any_fabricated_term": 30,+ "total_fabricated_terms": 49,+ "verification_gap": 1.0+ }+ ]+}\ No newline at end of filediff --git a/case-studies/verifiabench/run-demo.sh b/case-studies/verifiabench/run-demo.shnew file mode 100755index 00000000..a5d870cf--- /dev/null+++ b/case-studies/verifiabench/run-demo.sh@@ -0,0 +1,8 @@+#!/usr/bin/env bash+# Reproduce verifiabench against any OpenAI-compatible endpoint (default: local MLX on :8080).+# Set VB_API to point elsewhere. Pass model ids as arguments.+set -euo pipefail+cd "$(dirname "$0")"+if [ ! -d .venv ]; then python3 -m venv .venv; ./.venv/bin/pip -q install -r requirements.txt; fi+./.venv/bin/python src/verifiabench.py "$@"+echo; echo "See results/results.json and results/SUMMARY.md"diff --git a/case-studies/verifiabench/src/verifiabench.py b/case-studies/verifiabench/src/verifiabench.pynew file mode 100644index 00000000..4493b9ad--- /dev/null+++ b/case-studies/verifiabench/src/verifiabench.py@@ -0,0 +1,152 @@+"""+verifiabench — an un-game-able benchmark for scientific-workflow LLM reliability.++Existing science-LLM benchmarks grade the answer with string match or an LLM judge, so a+fluent-but-fabricated term can score as correct. verifiabench grades with a CLOSED-WORLD oracle:+every ontology term a model emits must EXIST in the authority (here the real Biolink Model) and+the output must satisfy the task's structural constraints. Correctness is set-membership plus+constraint satisfaction, not similarity, so fluency cannot buy a point.++Task: given a real gene-disease fact, write Biolink-typed RDF asserting it. The oracle extracts+every biolink term from the output and checks it against the Biolink Model's declared terms, then+checks the structure (a real Gene, a real Disease, a real association predicate). The headline is+the gap between RAW capability (the output looks like structured Biolink RDF) and VERIFIED+capability (its terms are real and the structure holds).++Deterministic oracle, no LLM judge. Model under test is queried over an OpenAI-compatible API.+"""+import json, os, re, sys, time+import requests+import rdflib+from rdflib import RDF++ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+DATA, RES = os.path.join(ROOT, "data"), os.path.join(ROOT, "results")+os.makedirs(RES, exist_ok=True)+API = os.environ.get("VB_API", "http://localhost:8080/v1/chat/completions")+BL = "https://w3id.org/biolink/vocab/"++with open(os.path.join(DATA, "biolink_vocab.json")) as f:+ DECLARED = set(json.load(f)["declared"])+GENE, DISEASE = BL + "Gene", BL + "Disease"+# real Biolink slots that legitimately relate a gene to a condition/disease+ASSOC_SLOTS = {BL + s for s in ("gene_associated_with_condition", "associated_with",+ "related_to", "contributes_to", "causes", "biomarker_for")} & DECLARED++# 30 real, well-established gene-disease facts+FACTS = [+ ("BRAF", "cardiofaciocutaneous syndrome"), ("TP53", "Li-Fraumeni syndrome"),+ ("PTEN", "Cowden syndrome"), ("EGFR", "non-small cell lung carcinoma"),+ ("BRCA1", "breast cancer"), ("BRCA2", "breast cancer"), ("KRAS", "pancreatic carcinoma"),+ ("APC", "familial adenomatous polyposis"), ("RB1", "retinoblastoma"), ("VHL", "von Hippel-Lindau disease"),+ ("CFTR", "cystic fibrosis"), ("HTT", "Huntington disease"), ("FMR1", "fragile X syndrome"),+ ("DMD", "Duchenne muscular dystrophy"), ("HBB", "beta thalassemia"), ("F8", "hemophilia A"),+ ("LDLR", "familial hypercholesterolemia"), ("MLH1", "Lynch syndrome"), ("NF1", "neurofibromatosis type 1"),+ ("SMN1", "spinal muscular atrophy"), ("PAH", "phenylketonuria"), ("G6PD", "G6PD deficiency"),+ ("ATM", "ataxia telangiectasia"), ("TSC1", "tuberous sclerosis"), ("MECP2", "Rett syndrome"),+ ("SCN1A", "Dravet syndrome"), ("COL1A1", "osteogenesis imperfecta"), ("GBA", "Gaucher disease"),+ ("HFE", "hereditary hemochromatosis"), ("PKD1", "polycystic kidney disease"),+]++PROMPT = ("Using the Biolink Model, write RDF in Turtle asserting that the gene {g} is associated "+ "with the disease {d}. Use only real Biolink Model classes and predicates with the "+ "`biolink:` prefix (biolink: = <{ns}>). Output only the Turtle.")++def query(model, prompt, timeout=120):+ r = requests.post(API, json={"model": model, "messages": [{"role": "user", "content": prompt}],+ "max_tokens": 400, "temperature": 0}, timeout=timeout)+ return r.json()["choices"][0]["message"]["content"]++def strip_fence(text):+ m = re.search(r"```(?:turtle|ttl)?\s*(.*?)```", text, re.S)+ return m.group(1) if m else text++def extract(output):+ """Return (biolink_terms:set, structure:dict, parsed:bool). Robust to whatever prefix a+ model maps to the Biolink namespace: parse the Turtle with rdflib (prefix-aware) and fall+ back to prefix-resolved regex if it does not parse."""+ text = strip_fence(output)+ terms = set()+ struct = {"gene": False, "disease": False, "assoc": False}+ try:+ g = rdflib.Graph().parse(data=text, format="turtle")+ except Exception:+ g = None+ if g is not None and len(g):+ for s, p, o in g:+ if str(p).startswith(BL): terms.add(str(p))+ if p == RDF.type and str(o).startswith(BL): terms.add(str(o))+ typed = {}+ for s, _, o in g.triples((None, RDF.type, None)):+ typed.setdefault(str(s), set()).add(str(o))+ struct["gene"] = any(GENE in ts for ts in typed.values())+ struct["disease"] = any(DISEASE in ts for ts in typed.values())+ struct["assoc"] = any(str(p) in ASSOC_SLOTS for _, p, _ in g)+ return terms, struct, True+ # fallback: find prefixes bound to the biolink namespace, then extract their CURIEs+ prefixes = set(re.findall(r"@prefix\s+(\w+):\s*<" + re.escape(BL) + r">", text))+ prefixes.add("biolink")+ for pre in prefixes:+ for m in re.findall(r"\b" + re.escape(pre) + r":([A-Za-z_]\w*)", text):+ terms.add(BL + m)+ for m in re.findall(re.escape(BL) + r"([A-Za-z_]\w*)", text):+ terms.add(BL + m)+ struct = {"gene": GENE in terms, "disease": DISEASE in terms, "assoc": len(terms & ASSOC_SLOTS) > 0}+ return terms, struct, False++def score(output):+ terms, struct, parsed = extract(output)+ real = {t for t in terms if t in DECLARED}+ fake = terms - real+ raw_ok = len(terms) > 0 # produced biolink-namespace terms+ verified_ok = struct["gene"] and struct["disease"] and struct["assoc"] and len(fake) == 0+ return {+ "parsed_turtle": parsed, "n_terms": len(terms), "n_real": len(real), "n_fake": len(fake),+ "term_existence_rate": (len(real) / len(terms)) if terms else 0.0,+ "raw_ok": raw_ok, "verified_ok": verified_ok,+ "has_gene": struct["gene"], "has_disease": struct["disease"], "has_assoc": struct["assoc"],+ "fake_terms": sorted(t.split("/")[-1] for t in fake)[:6],+ "output": output[:600],+ }++def run_model(model):+ per = []+ for i, (g, d) in enumerate(FACTS):+ try:+ out = query(model, PROMPT.format(g=g, d=d, ns=BL))+ except Exception as e:+ out = f"__ERROR__ {e}"+ s = score(out); s["gene"] = g; s["disease"] = d+ per.append(s)+ print(f" [{i+1}/{len(FACTS)}] {g}/{d[:22]:22s} raw={s['raw_ok']} verified={s['verified_ok']} "+ f"real={s['n_real']} fake={s['n_fake']}", flush=True)+ n = len(per)+ agg = {+ "model": model, "n_tasks": n,+ "raw_capability": round(sum(x["raw_ok"] for x in per) / n, 3),+ "verified_capability": round(sum(x["verified_ok"] for x in per) / n, 3),+ "mean_term_existence": round(sum(x["term_existence_rate"] for x in per) / n, 3),+ "tasks_with_any_fabricated_term": sum(x["n_fake"] > 0 for x in per),+ "total_fabricated_terms": sum(x["n_fake"] for x in per),+ }+ agg["verification_gap"] = round(agg["raw_capability"] - agg["verified_capability"], 3)+ return agg, per++def main():+ models = sys.argv[1:] or ["mlx-community/Qwen2.5-3B-Instruct-4bit"]+ board = []+ for m in models:+ print(f"[*] running {m} ...", flush=True)+ agg, per = run_model(m)+ board.append(agg)+ json.dump(per, open(os.path.join(RES, "per_task_" + m.split("/")[-1] + ".json"), "w"), indent=2)+ print(" ", json.dumps(agg), flush=True)+ board.sort(key=lambda a: a["verified_capability"], reverse=True)+ out = {"authority": "Biolink Model", "n_tasks": len(FACTS),+ "oracle": "closed-world term existence + structural constraints (deterministic, no LLM judge)",+ "leaderboard": board}+ json.dump(out, open(os.path.join(RES, "results.json"), "w"), indent=2)+ print(json.dumps(out, indent=2))++if __name__ == "__main__":+ main()