Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

case-study: verifiabench (un-game-able scientific-LLM benchmark)

FABIOTESSa57ef6899cf3
diff --git a/case-studies/verifiabench/.gitignore b/case-studies/verifiabench/.gitignore
new file mode 100644
index 00000000..77ac7549
--- /dev/null
+++ b/case-studies/verifiabench/.gitignore
@@ -0,0 +1,3 @@
+.venv/
+__pycache__/
+*.pyc
diff --git a/case-studies/verifiabench/BUILD_REPORT.md b/case-studies/verifiabench/BUILD_REPORT.md
new file mode 100644
index 00000000..0818a478
--- /dev/null
+++ b/case-studies/verifiabench/BUILD_REPORT.md
@@ -0,0 +1,62 @@
+# BUILD_REPORT: verifiabench
+
+Honest record of the benchmark, the oracle and the limits. Numbers in `results/` are produced by
+`src/verifiabench.py` at temperature 0 against a local MLX OpenAI-compatible endpoint.
+
+## What was used (real)
+
+- **Authority:** the Biolink Model (biolink/biolink-model), 868 declared class and slot IRIs, reused
+ from the bio-kg-triage case study.
+- **Tasks:** 30 well-established gene-disease facts (BRCA1/breast cancer, HTT/Huntington disease,
+ CFTR/cystic fibrosis, ...). Each task prompts the model to write Biolink-typed RDF asserting the fact.
+- **Models under test (local, open):** Qwen3-Coder-30B-A3B (8bit), Qwen2.5-3B, Llama-3.2-3B,
+ gemma-2-2b, Qwen2.5-0.5B, all served over a local MLX server on port 8080. The runner speaks the
+ OpenAI chat-completions API, so any endpoint works via `VB_API`.
+
+## The oracle (deterministic, no LLM judge)
+
+For each model output:
+1. Strip any markdown code fence.
+2. Parse the Turtle with rdflib. This resolves whatever prefix the model bound to the Biolink
+ namespace (models used both `biolink:` and `bio:`), so the check is format-robust, not
+ prefix-brittle. If it does not parse, fall back to prefix-resolved regex extraction.
+3. Collect every predicate and every `rdf:type` object whose IRI is in the Biolink namespace.
+4. **Term existence:** each such term is real iff it is in Biolink's declared set; otherwise fabricated.
+5. **Structure:** does the graph have a node typed `biolink:Gene`, a node typed `biolink:Disease`,
+ and at least one real Biolink association predicate.
+6. **raw_ok** = produced any Biolink-namespace term (the fluency proxy). **verified_ok** = real Gene
+ AND real Disease AND real association predicate AND zero fabricated terms.
+
+Per-task outputs (including the raw model text, truncated) are saved under `results/` for inspection.
+
+## The result, precisely
+
+- Raw capability saturates: Qwen2.5-3B, Llama-3.2-3B and Qwen3-Coder-30B all score 1.00.
+- Verified capability separates: Qwen3-Coder-30B 1.00 (0 fabricated terms); Qwen2.5-3B and
+ Llama-3.2-3B 0.00 (30/30 tasks contain a fabricated term, ~50% mean term-existence); gemma-2-2b
+ and Qwen2.5-0.5B produce little or no valid Biolink and score 0.00.
+- Manual spot-check confirms the oracle: Qwen3-Coder-30B emits real terms
+ (`biolink:Gene`, `biolink:Disease`, `biolink:causes`, `biolink:has_gene_product`) in valid
+ structure; the mid models emit genuine non-terms (`biolink:has_association`, `biolink:has_role`).
+
+## Limits of the claim (do not overstate)
+
+1. **One task family, one authority.** Biolink gene-disease RDF authoring. The method is
+ authority-agnostic (GO, ChEBI, Reactome, EDAM are drop-in), but this run does not yet cover them;
+ a multi-domain suite is the next step.
+2. **The facts are public, not a secret split.** 30 well-known gene-disease pairs. The honest use is
+ relative model comparison and a versioned, inspectable oracle, not a hold-out leaderboard.
+3. **Existence and coarse structure, not full semantics.** The oracle checks that terms exist and
+ that a Gene, a Disease and a real association predicate are present. It does not check that the
+ *specific* predicate is the ideal one for the relation; that is the certified-denotation direction.
+4. **Small models are underserved by the strict "zero fabricated terms" rule.** A model that gets the
+ structure right but emits one extra invented term scores verified=0. This is deliberate: in a
+ scientific pipeline a single fabricated identifier is a real defect. It is stated, not hidden.
+5. **Local quantised models.** 4-8bit MLX quantisations, not full-precision or frontier API models;
+ the point is the evaluation method and the raw-vs-verified gap, which any endpoint can reproduce.
+
+## Reproducibility
+
+`./run-demo.sh <models...>` reruns against a live endpoint (temperature 0). Model outputs can vary
+across server versions, but the raw-vs-verified separation is the stable, reproducible finding.
+rdflib at build time; models as listed.
diff --git a/case-studies/verifiabench/README.md b/case-studies/verifiabench/README.md
new file mode 100644
index 00000000..fac92204
--- /dev/null
+++ b/case-studies/verifiabench/README.md
@@ -0,0 +1,85 @@
+# verifiabench
+
+**An un-game-able benchmark for scientific-workflow LLM reliability: grade the model with a
+closed-world oracle, not a fluency check. Fluency is saturated; verified correctness is not.**
+
+Most benchmarks for LLMs on scientific tasks grade the answer with string match or an LLM judge.
+Both reward a confident, well-formed answer, so a model that emits a *plausible-but-nonexistent*
+gene, ontology class or predicate can score as correct. verifiabench grades differently: every
+term a model emits must **exist** in the authority (the real Biolink Model) and the output must
+satisfy the task's **structural** constraints. Correctness is set-membership plus constraint
+satisfaction, checked deterministically, so fluency cannot buy a point.
+
+This is the reliable-evaluation layer Encode / ARIA Challengescape items on testing LLM reasoning
+and reliability on realistic scientific workflows are asking for.
+
+## The result
+
+Deterministic run of [`src/verifiabench.py`](src/verifiabench.py): 30 real gene-disease facts, each
+asking a model to write Biolink-typed RDF, graded by the closed-world oracle. Five open models over
+a local OpenAI-compatible endpoint (full table in [`results/SUMMARY.md`](results/SUMMARY.md)):
+
+| Model | Raw capability (fluency) | **Verified capability** | Mean term-existence | Fabricated terms |
+|---|--:|--:|--:|--:|
+| Qwen3-Coder-30B-A3B (8bit) | 1.00 | **1.00** | 1.00 | 0 |
+| Qwen2.5-3B | 1.00 | **0.00** | 0.51 | 65 |
+| Llama-3.2-3B | 1.00 | **0.00** | 0.51 | 49 |
+| gemma-2-2b | 0.23 | **0.00** | 0.06 | 21 |
+| Qwen2.5-0.5B | 0.00 | **0.00** | 0.00 | 0 |
+
+**Raw capability** = produced structured Biolink-namespace RDF, what a fluency- or judge-based
+benchmark would reward. **Verified capability** = a real `biolink:Gene`, a real `biolink:Disease`, a
+real association predicate, and zero fabricated terms.
+
+Three models produce structured Biolink RDF on every task (raw capability 1.00), yet two of them,
+Qwen2.5-3B and Llama-3.2-3B, score **0.00 verified**: roughly half of the terms they emit do not
+exist in Biolink. Only Qwen3-Coder-30B actually gets it right, 1.00 verified, 0 fabricated terms.
+A benchmark that graded on fluency, or asked an LLM to judge, would rank the hallucinating models
+near the top. The closed-world oracle separates them cleanly and cannot be gamed by producing
+plausible but nonexistent terms.
+
+## Why this is the ownable gap
+
+- **Fluency and correctness are different axes.** Raw capability saturates at 1.00 for any competent
+ model; verified capability ranges from 0.00 to 1.00 on the same tasks. Benchmarks that cannot see
+ the difference are measuring the wrong thing.
+- **The oracle is deterministic and closed-world.** A term is right if and only if it is in the
+ authority and used in a valid structure. No string similarity, no LLM judge, no partial credit for
+ confident nonsense. This is the [`onto_vocab_check`](../onto-correctness-bench/README.md) principle
+ turned into an evaluation.
+- **It is un-game-able the way that matters.** You cannot lift the score by writing more fluent RDF;
+ you can only lift it by using terms that actually exist. That is exactly the property a scientific
+ deployment needs from its model.
+- **It validates in both directions.** A genuinely capable model (Qwen3-Coder-30B) scores 100%, so
+ the oracle is not just failing everything; it is measuring a real capability the weak models lack.
+
+## Reproduce
+
+```bash
+./run-demo.sh # runs the default model on a local MLX/OpenAI endpoint (:8080)
+./run-demo.sh "model-a" "model-b" # benchmark specific models
+VB_API=https://your-endpoint/v1/chat/completions ./run-demo.sh "gpt-..." # any OpenAI-compatible API
+```
+
+Requires Python 3.10+ and an OpenAI-compatible chat endpoint. The runs here used a local MLX server.
+
+## Honest scope
+
+See [`BUILD_REPORT.md`](BUILD_REPORT.md). This is one task family (Biolink gene-disease RDF authoring)
+on one authority; the method extends unchanged to GO, ChEBI, Reactome and EDAM, which is the
+documented next step toward a multi-domain suite. The 30 facts are well-established, not a held-back
+secret split, so the honest use is relative comparison and a public, versioned oracle, not a secret
+leaderboard. The oracle checks term existence and a coarse structure (a real Gene, a real Disease, a
+real association predicate); deeper semantic correctness (right predicate for the specific relation)
+is the certified-denotation direction, not measured here.
+
+---
+
+### Built by Tesseract Academy
+
+We build the evaluation and assurance layer for AI that has to be right, not just fluent. If you are
+choosing or deploying a model for scientific or ontology-grounded work, we can benchmark it on what
+it actually gets correct.
+
+[gov.tesseract.academy](https://gov.tesseract.academy) · [email protected]
+Part of [Open Ontologies](../../README.md) · MIT · real models, real numbers.
diff --git a/case-studies/verifiabench/requirements.txt b/case-studies/verifiabench/requirements.txt
new file mode 100644
index 00000000..93e24d96
--- /dev/null
+++ b/case-studies/verifiabench/requirements.txt
@@ -0,0 +1,2 @@
+rdflib>=7.0
+requests>=2.31
diff --git a/case-studies/verifiabench/results/SUMMARY.md b/case-studies/verifiabench/results/SUMMARY.md
new file mode 100644
index 00000000..aa80dd3b
--- /dev/null
+++ b/case-studies/verifiabench/results/SUMMARY.md
@@ -0,0 +1,15 @@
+# Results summary
+
+Authority: **Biolink Model**. Tasks: **30** real gene-disease facts, each asking a model to write Biolink-typed RDF. Oracle: **closed-world term existence + structural constraints (deterministic, no LLM judge)**.
+
+| Model | Raw capability (fluency) | **Verified capability** | Mean term-existence | Tasks with a fabricated term | Fabricated terms |
+|---|--:|--:|--:|--:|--:|
+| Qwen3-Coder-30B-A3B-Instruct-8bit | 1.00 | **1.00** | 1.00 | 0/30 | 0 |
+| gemma-2-2b-it-4bit | 0.23 | **0.00** | 0.06 | 7/30 | 21 |
+| Qwen2.5-0.5B-Instruct-4bit | 0.00 | **0.00** | 0.00 | 0/30 | 0 |
+| Qwen2.5-3B-Instruct-4bit | 1.00 | **0.00** | 0.51 | 30/30 | 65 |
+| Llama-3.2-3B-Instruct-4bit | 1.00 | **0.00** | 0.51 | 30/30 | 49 |
+
+**Raw capability** = produced structured Biolink-namespace RDF (what a fluency- or LLM-judge-based benchmark would reward). **Verified capability** = a real `biolink:Gene`, a real `biolink:Disease`, a real association predicate, and zero fabricated terms.
+
+**Headline.** Fluency is saturated and verification is not. Three models produce structured Biolink RDF on every task (raw capability 1.00), yet two of them, Qwen2.5-3B-Instruct-4bit, Llama-3.2-3B-Instruct-4bit, score **0.00 verified**: every output invents terms (around half of all terms emitted do not exist in Biolink). Only **Qwen3-Coder-30B-A3B-Instruct-8bit** actually gets it right, 1.00 verified with 0 fabricated terms. A benchmark that graded on fluency or an LLM judge would rank the hallucinating models near the top; the closed-world oracle separates them cleanly, and cannot be gamed by producing plausible but nonexistent terms.
\ No newline at end of file
diff --git a/case-studies/verifiabench/results/per_task_Llama-3.2-3B-Instruct-4bit.json b/case-studies/verifiabench/results/per_task_Llama-3.2-3B-Instruct-4bit.json
new file mode 100644
index 00000000..547eb267
--- /dev/null
+++ b/case-studies/verifiabench/results/per_task_Llama-3.2-3B-Instruct-4bit.json
@@ -0,0 +1,561 @@
+[
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "CardiofaciocutaneousSyndrome",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/BRAF> a biolink:Gene.\n<http://www.example.org/CardiofaciocutaneousSyndrome> a biolink:Disease.\n\n<http://www.example.org/BRAF> biolink:associatedWith biolink:CardiofaciocutaneousSyndrome.\n\nNote: The Biolink Model is a large ontology, and the classes and predicates used here are just a few examples. The actual classes and predicates used may vary depending on the specific use case.",
+ "gene": "BRAF",
+ "disease": "cardiofaciocutaneous syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "Lifraumeni",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/TP53> a biolink:Gene.\n<http://www.example.org/LiFraumeni> a biolink:Disease.\n\n<http://www.example.org/TP53> biolink:associatedWith biolink:Lifraumeni.\n\nNote: The Biolink model is a large ontology, and this is a very simple example. In a real-world scenario, you would need to consider the relationships between the classes and predicates in the Biolink model.",
+ "gene": "TP53",
+ "disease": "Li-Fraumeni syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "CowdenSyndrome",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/PTEN> a biolink:Gene.\n<http://www.example.org/CowdenSyndrome> a biolink:Disease.\n\n<http://www.example.org/PTEN> biolink:associatedWith biolink:CowdenSyndrome.\n\nThis RDF asserts that the gene PTEN is associated with the disease Cowden syndrome using the Biolink Model classes and predicates.",
+ "gene": "PTEN",
+ "disease": "Cowden syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/EGFR> a biolink:Gene.\n<http://www.example.org/EGFR> biolink:associatedWith biolink:Disease <http://www.example.org/NSCLC>.\n\nNote: The above code is written in Turtle format which is used to represent RDF data in a human-readable format. The actual RDF data would be represented in a different format, such as XML or JSON.",
+ "gene": "EGFR",
+ "disease": "non-small cell lung carcinoma"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "isAssociatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/BRCA1> a biolink:Gene.\n<http://www.example.org/BreastCancer> a biolink:Disease.\n<http://www.example.org/BRCA1> biolink:isAssociatedWith biolink:Disease.\n<http://www.example.org/BreastCancer> biolink:isAssociatedWith biolink:Gene.",
+ "gene": "BRCA1",
+ "disease": "breast cancer"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "isAssociatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/BRCA2> a biolink:Gene.\n<http://www.example.org/BreastCancer> a biolink:Disease.\n<http://www.example.org/BRCA2> biolink:isAssociatedWith biolink:Disease.\n<http://www.example.org/BreastCancer> biolink:isAssociatedWith biolink:Gene.",
+ "gene": "BRCA2",
+ "disease": "breast cancer"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 1,
+ "n_real": 0,
+ "n_fake": 1,
+ "term_existence_rate": 0.0,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "isAssociatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/gene/KRAS> biolink:isAssociatedWith <http://www.example.org/disease/pancreatic_carcinoma>.",
+ "gene": "KRAS",
+ "disease": "pancreatic carcinoma"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 1,
+ "n_real": 0,
+ "n_fake": 1,
+ "term_existence_rate": 0.0,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/gene/APC> biolink:associatedWith <http://www.example.org/disease/FamilialAdenomatousPolyposis>.",
+ "gene": "APC",
+ "disease": "familial adenomatous polyposis"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/RB1> a biolink:Gene.\n<http://www.example.org/Retinoblastoma> a biolink:Disease.\n\n<http://www.example.org/RB1> biolink:associatedWith <http://www.example.org/Retinoblastoma>.",
+ "gene": "RB1",
+ "disease": "retinoblastoma"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/VHL> a biolink:Gene.\n<http://www.example.org/VHL> biolink:associatedWith biolink:Disease <http://www.example.org/VonHippelLindauDisease>.\n\nNote: The above code is written in Turtle format which is used to represent RDF data in a human-readable format. The RDF data is represented as a series of statements, each of which is a subject-predicate-object triple.",
+ "gene": "VHL",
+ "disease": "von Hippel-Lindau disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith",
+ "hasAssociation"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/cftr> a biolink:Gene.\n<http://www.example.org/cftr> biolink:associatedWith biolink:Disease.\n<http://www.example.org/cftr> biolink:hasAssociation biolink:Disease.\n\nNote: The Biolink Model is a large ontology, and the classes and predicates used here are just a few examples. The actual classes and predicates used may vary depending on the specific use case and the version of the Biolink Model being used.",
+ "gene": "CFTR",
+ "disease": "cystic fibrosis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "HuntingtonDisease",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/HTT> a biolink:Gene.\n<http://www.example.org/HTT> biolink:associatedWith biolink:Disease.\n<http://www.example.org/HTT> biolink:associatedWith biolink:HuntingtonDisease.\n\nNote: The above code is just an example and does not represent real data. The HTT gene is associated with Huntington's disease in the Biolink model.",
+ "gene": "HTT",
+ "disease": "Huntington disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 5,
+ "n_real": 3,
+ "n_fake": 2,
+ "term_existence_rate": 0.6,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "GeneAssociation",
+ "hasAssociation"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/FragileX> a biolink:Disease.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:Gene.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:Association.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:GeneAssociation.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:GeneAssociation.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:GeneAssociation.\n<http://www.example.org/FragileX> biolink:hasAssociation biolink:GeneAssociation.\n<http://www.examp",
+ "gene": "FMR1",
+ "disease": "fragile X syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "DuchenneMuscularDystrophy",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/DMD> a biolink:Gene.\n<http://www.example.org/DMD> biolink:associatedWith biolink:Disease.\n<http://www.example.org/DMD> biolink:associatedWith biolink:DuchenneMuscularDystrophy.\n\nNote: The Biolink Model classes and predicates used in this example are:\n\n- biolink:Gene\n- biolink:associatedWith\n- biolink:Disease\n- biolink:DuchenneMuscularDystrophy",
+ "gene": "DMD",
+ "disease": "Duchenne muscular dystrophy"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/HBB> a biolink:Gene.\n<http://www.example.org/BetaThalassemia> a biolink:Disease.\n\n<http://www.example.org/HBB> biolink:associatedWith biolink:Disease.\n<http://www.example.org/BetaThalassemia> biolink:associatedWith biolink:Gene.\n\nThis RDF asserts that the gene HBB is associated with the disease Beta Thalassemia.",
+ "gene": "HBB",
+ "disease": "beta thalassemia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "HemophiliaA",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/F8> a biolink:Gene.\n<http://www.example.org/HemophiliaA> a biolink:Disease.\n<http://www.example.org/F8> biolink:associatedWith biolink:HemophiliaA.\n\nThis RDF asserts that the gene F8 is associated with the disease Hemophilia A.",
+ "gene": "F8",
+ "disease": "hemophilia A"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "FamilialHypercholesterolemia",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/LDLR> a biolink:Gene.\n<http://www.example.org/FamilialHypercholesterolemia> a biolink:Disease.\n\n<http://www.example.org/LDLR> biolink:associatedWith biolink:FamilialHypercholesterolemia.\n\nNote: The Biolink Model is a large ontology, and this is a very small example. In a real-world scenario, you would need to include more classes, predicates, and resources to create a complete RDF graph.",
+ "gene": "LDLR",
+ "disease": "familial hypercholesterolemia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "LynchSyndrome",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/MLH1> a biolink:Gene.\n<http://www.example.org/LynchSyndrome> a biolink:Disease.\n\n<http://www.example.org/MLH1> biolink:associatedWith biolink:LynchSyndrome.\n\nNote: The Biolink model is a large ontology, and the above code snippet is just a small example. Also, the example URIs are fictional and for demonstration purposes only.",
+ "gene": "MLH1",
+ "disease": "Lynch syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/NF1> a biolink:Gene.\n<http://www.example.org/NeurofibromatosisType1> a biolink:Disease.\n<http://www.example.org/NF1> biolink:associatedWith <http://www.example.org/NeurofibromatosisType1>.\n\nNote: The Biolink Model is a large ontology, and this is a very simple example. In a real-world scenario, you would need to consider the relationships between the classes and predicates in the Biolink Model.",
+ "gene": "NF1",
+ "disease": "neurofibromatosis type 1"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/smn1> a biolink:Gene.\n<http://www.example.org/sma> a biolink:Disease.\n\n<http://www.example.org/smn1> biolink:associatedWith biolink:Disease.\n<http://www.example.org/sma> biolink:associatedWith biolink:Gene.\n\nThis RDF asserts that the gene SMN1 is associated with the disease Spinal Muscular Atrophy (SMA) using the Biolink Model classes and predicates.",
+ "gene": "SMN1",
+ "disease": "spinal muscular atrophy"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/gene/PAH> a biolink:Gene.\n<http://www.example.org/disease/Phenylketonuria> a biolink:Disease.\n\n<http://www.example.org/gene/PAH> biolink:associatedWith biolink:Disease.\n<http://www.example.org/disease/Phenylketonuria> biolink:associatedWith biolink:Gene.\n\nThis RDF asserts that the gene PAH is associated with the disease Phenylketonuria.",
+ "gene": "PAH",
+ "disease": "phenylketonuria"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "G6PD_deficiency",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/G6PD> a biolink:Gene.\n<http://www.example.org/G6PD_deficiency> a biolink:Disease.\n\n<http://www.example.org/G6PD> biolink:associatedWith biolink:G6PD_deficiency.\n\nThis RDF asserts that the gene G6PD is associated with the disease G6PD deficiency using the Biolink Model classes and predicates.",
+ "gene": "G6PD",
+ "disease": "G6PD deficiency"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "AtaxiaTelangiectasia",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/ATM> a biolink:Gene.\n<http://www.example.org/ATM> biolink:associatedWith biolink:Disease.\n<http://www.example.org/ATM> biolink:associatedWith biolink:AtaxiaTelangiectasia.\n\nNote: The above code is just an example and does not represent real data. The actual data would be represented as URIs.",
+ "gene": "ATM",
+ "disease": "ataxia telangiectasia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/TSC1> a biolink:Gene.\n<http://www.example.org/TSC1> biolink:associatedWith biolink:Disease <http://www.example.org/TuberousSclerosis>.\n\nThis RDF in Turtle asserts that the gene TSC1 is associated with the disease tuberous sclerosis.",
+ "gene": "TSC1",
+ "disease": "tuberous sclerosis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/MECP2> a biolink:Gene.\n<http://www.example.org/RettSyndrome> a biolink:Disease.\n\n<http://www.example.org/MECP2> biolink:associatedWith biolink:Disease.\n<http://www.example.org/RettSyndrome> biolink:associatedWith biolink:Gene.\n\nThis RDF asserts that the gene MECP2 is associated with the disease Rett syndrome.",
+ "gene": "MECP2",
+ "disease": "Rett syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 6,
+ "n_real": 2,
+ "n_fake": 4,
+ "term_existence_rate": 0.3333333333333333,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "GeneAssociation",
+ "NeurotransmitterGating",
+ "has_association",
+ "is_a"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/scn1a> biolink:is_a biolink:Gene.\n<http://www.example.org/scn1a> biolink:is_a biolink:NeurotransmitterGating.\n\n<http://www.example.org/dravet> biolink:is_a biolink:Disease.\n\n<http://www.example.org/dravet> biolink:has_association biolink:GeneAssociation.\n<http://www.example.org/dravet> biolink:has_association biolink:GeneAssociation.\n<http://www.example.org/dravet> biolink:has_association biolink:GeneAssociation.\n\n<http://www.example.org/dravet> biolink:has_association biolink:GeneAssociation.\n<http://www.example.org/",
+ "gene": "SCN1A",
+ "disease": "Dravet syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 1,
+ "n_real": 0,
+ "n_fake": 1,
+ "term_existence_rate": 0.0,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "isAssociatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/gene/COL1A1> biolink:isAssociatedWith <http://www.example.org/disease/OsteogenesisImperfecta>.",
+ "gene": "COL1A1",
+ "disease": "osteogenesis imperfecta"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "GaucherDisease",
+ "associatedWith"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/GBA> a biolink:Gene.\n<http://www.example.org/GaucherDisease> a biolink:Disease.\n\n<http://www.example.org/GBA> biolink:associatedWith biolink:GaucherDisease.\n\nNote: The Biolink model is a large ontology, and the classes and predicates used here are just a few examples. The actual classes and predicates used may vary depending on the specific use case.",
+ "gene": "GBA",
+ "disease": "Gaucher disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 5,
+ "n_real": 3,
+ "n_fake": 2,
+ "term_existence_rate": 0.6,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith",
+ "hasAssociation"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/HFE> a biolink:Gene.\n<http://www.example.org/HFE> biolink:associatedWith biolink:Disease <http://www.example.org/HereditaryHemochromatosis>.\nbiolink:hasAssociation biolink:Association <http://www.example.org/HFE> <http://www.example.org/HereditaryHemochromatosis>.\nbiolink:hasAssociation biolink:Association <http://www.example.org/HereditaryHemochromatosis> <http://www.example.org/HFE>.",
+ "gene": "HFE",
+ "disease": "hereditary hemochromatosis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 5,
+ "n_real": 2,
+ "n_fake": 3,
+ "term_existence_rate": 0.4,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "associatedWith",
+ "hasDisease",
+ "polycysticKidneyDisease"
+ ],
+ "output": "@prefix biolink: <https://w3id.org/biolink/vocab/>.\n\n<http://www.example.org/pkd1> a biolink:Gene.\n<http://www.example.org/pkd1> biolink:associatedWith biolink:Disease.\n<http://www.example.org/pkd1> biolink:hasDisease biolink:polycysticKidneyDisease.\n\nNote: The Biolink Model classes and predicates used in this example are:\n\n- biolink:Gene\n- biolink:associatedWith\n- biolink:hasDisease\n- biolink:Disease\n- biolink:polycysticKidneyDisease",
+ "gene": "PKD1",
+ "disease": "polycystic kidney disease"
+ }
+]
\ No newline at end of file
diff --git a/case-studies/verifiabench/results/per_task_Qwen2.5-0.5B-Instruct-4bit.json b/case-studies/verifiabench/results/per_task_Qwen2.5-0.5B-Instruct-4bit.json
new file mode 100644
index 00000000..65f42a69
--- /dev/null
+++ b/case-studies/verifiabench/results/per_task_Qwen2.5-0.5B-Instruct-4bit.json
@@ -0,0 +1,482 @@
+[
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs200020002000: <",
+ "gene": "BRAF",
+ "disease": "cardiofaciocutaneous syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs200020002000: <",
+ "gene": "TP53",
+ "disease": "Li-Fraumeni syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```\n@prefix : <http://example.com/ns#>. @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @pref",
+ "gene": "PTEN",
+ "disease": "Cowden syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs200020002000: <",
+ "gene": "EGFR",
+ "disease": "non-small cell lung carcinoma"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs200020002000: <",
+ "gene": "BRCA1",
+ "disease": "breast cancer"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2: <http://purl.org/dc/elements/2.0/0>\n@prefix dcterms3: <http://purl.org/dc/elements/3.0/0>\n@prefix dcterms4: <http://purl.org/dc/elements/4.0/0>\n@prefix dcterms5: <http://purl.org/dc/elements/5.0/0>\n@prefix dcterms6: <http://purl.org/dc/elements/6.0/0>\n@prefix dcter",
+ "gene": "BRCA2",
+ "disease": "breast cancer"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```\n@prefix : <http://example.com/ontology#>. @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix xsd: <http://www.w3.org/2001/XMLSchema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://w",
+ "gene": "KRAS",
+ "disease": "pancreatic carcinoma"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/>\n@prefix dcterms2002: <http://purl.org/dc/elements/2.0/>\n\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix owl2002: <http://purl.org/dc/elements/2.0#>\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix owl2002: <http://purl.org/dc/elements/2.0#>\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix owl2002: <http://pur",
+ "gene": "APC",
+ "disease": "familial adenomatous polyposis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix rdfs200020002000: <",
+ "gene": "RB1",
+ "disease": "retinoblastoma"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/>\n@prefix dcterms2002: <http://purl.org/dc/elements/2.0/>\n\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix owl2002: <http://purl.org/dc/elements/2.0#>\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix xsd2002: <http://purl.org/dc/elements/2.0#>\n\n@prefix rdfs2002: <http://purl.org/dc/elements/2.0#>\n@prefix owl2002: <http://pu",
+ "gene": "VHL",
+ "disease": "von Hippel-Lindau disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```\n@prefix : <http://example.com/ns#>. @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @pref",
+ "gene": "CFTR",
+ "disease": "cystic fibrosis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```\n@prefix : <http://example.com/ns#>. @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @pref",
+ "gene": "HTT",
+ "disease": "Huntington disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2: <http://purl.org/dc/elements/2.0/0>\n@prefix dcterms3: <http://purl.org/dc/elements/3.0/0>\n@prefix dcterms4: <http://purl.org/dc/elements/4.0/0>\n@prefix dcterms5: <http://purl.org/dc/elements/5.0/0>\n@prefix dcterms6: <http://purl.org/dc/elements/6.0/0>\n@prefix dcter",
+ "gene": "FMR1",
+ "disease": "fragile X syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```\n@prefix : <http://example.com/ontology#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix xsd: <http://www.w3.org/2001/XMLSchema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/",
+ "gene": "DMD",
+ "disease": "Duchenne muscular dystrophy"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n",
+ "gene": "HBB",
+ "disease": "beta thalassemia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms: <http://www.w3.org/200114270000",
+ "gene": "F8",
+ "disease": "hemophilia A"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2002: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms2003: <http://purl.org/dc/elements/3.0.0/0>\n@prefix dcterms2004: <http://purl.org/dc/elements/4.0.0/0>\n@prefix dcterms2005: <http://purl.org/dc/elements/5.0.0/0>\n@prefix dcterms2006: <http://purl.org/dc/elemen",
+ "gene": "LDLR",
+ "disease": "familial hypercholesterolemia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n",
+ "gene": "MLH1",
+ "disease": "Lynch syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http",
+ "gene": "NF1",
+ "disease": "neurofibromatosis type 1"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2: <http://purl.org/dc/elements/2.0/0>\n@prefix dcterms3: <http://purl.org/dc/elements/3.0/0>\n@prefix dcterms4: <http://purl.org/dc/elements/4.0/0>\n@prefix dcterms5: <http://purl.org/dc/elements/5.0/0>\n@prefix dcterms6: <http://purl.org/dc/elements/6.0/0>\n@prefix dcter",
+ "gene": "SMN1",
+ "disease": "spinal muscular atrophy"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2002/0",
+ "gene": "PAH",
+ "disease": "phenylketonuria"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n",
+ "gene": "G6PD",
+ "disease": "G6PD deficiency"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema",
+ "gene": "ATM",
+ "disease": "ataxia telangiectasia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```\n@prefix : <http://example.com/biolink#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix xsd: <http://www.w3.org/2001/XMLSchema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2",
+ "gene": "TSC1",
+ "disease": "tuberous sclerosis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```\n@prefix : <http://example.com/ontology#>. @prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-ns1#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2000/01/rdf-schema#>. @prefix xsd: <http://www.w3.org/2001/XMLSchema#>. @prefix dcterms: <http://purl.org/dc/elements/1.1/>. @prefix dcterms: <http://purl.org/dc/elements/1.1/>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2000/01/rdf-schema#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2000/01/rdf-schema#>. @prefix rdfs:",
+ "gene": "MECP2",
+ "disease": "Rett syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2: <http://purl.org/dc/elements/2.0/0>\n@prefix dcterms3: <http://purl.org/dc/elements/3.0/0>\n@prefix dcterms4: <http://purl.org/dc/elements/4.0/0>\n@prefix dcterms5: <http://purl.org/dc/elements/5.0/0>\n@prefix dcterms6: <http://purl.org/dc/elements/6.0/0>\n@prefix dcter",
+ "gene": "SCN1A",
+ "disease": "Dravet syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```\n@prefix : <http://example.com/ontology#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-199901#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix xsd: <http://www.w3.org/2001/XMLSchema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/2002/07/owl#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix owl: <http://www.w3.org/",
+ "gene": "COL1A1",
+ "disease": "osteogenesis imperfecta"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/0>\n@prefix dcterms2000: <http://purl.org/dc/elements/2.1.0/0>\n@prefix dcterms20002000: <http://purl.org/dc/elements/2.1.0.2000/>\n\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n",
+ "gene": "GBA",
+ "disease": "Gaucher disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix rdf: <http://www.w3.org/2000/01/rdf-syntax-1.0#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms2000: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms20002000: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms200020002000: <http://www.w3.org/200114270000000000000000000000000#>\n@prefix dcterms200020002",
+ "gene": "HFE",
+ "disease": "hereditary hemochromatosis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix : <http://example.com/biolink#>\n@prefix owl: <http://www.w3.org/2002/07/owl#>\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#>\n\n@prefix dcterms: <http://purl.org/dc/elements/1.1/>\n@prefix dcterms2: <http://purl.org/dc/elements/2.0/>\n\n@prefix rdfs2: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl2: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl1: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl21: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl11: <http://www.w3.org/2000/01/rdf-schema#>\n@prefix owl12: <http:",
+ "gene": "PKD1",
+ "disease": "polycystic kidney disease"
+ }
+]
\ No newline at end of file
diff --git a/case-studies/verifiabench/results/per_task_Qwen2.5-3B-Instruct-4bit.json b/case-studies/verifiabench/results/per_task_Qwen2.5-3B-Instruct-4bit.json
new file mode 100644
index 00000000..e40346b0
--- /dev/null
+++ b/case-studies/verifiabench/results/per_task_Qwen2.5-3B-Instruct-4bit.json
@@ -0,0 +1,552 @@
+[
+ {
+ "parsed_turtle": false,
+ "n_terms": 34,
+ "n_real": 3,
+ "n_fake": 31,
+ "term_existence_rate": 0.08823529411764706,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "activating_role",
+ "association_type",
+ "conjunctional_mutation",
+ "copy_number_change",
+ "copy_number_gain",
+ "copy_number_loss"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/BRAF> a bio:Gene;\n bio:has_association [\n a bio:Association;\n bio:association_type bio:genetic_association;\n bio:target <http://example.org/BRAF>;\n bio:target_type bio:gene;\n bio:target_role bio:activating_role;\n bio:target_mechanism bio:mutation;\n bio:target_mechanism_type bio:genetic_mutation;\n bio:target_mechanism_subtype bio:missense_mutation;\n bio:target_mechanism_subtype bio:germline_mutation;\n bio:target_mechanism_subtype bio:sporadic_m",
+ "gene": "BRAF",
+ "disease": "cardiofaciocutaneous syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/TP53> a bio:Gene;\n bio:has_associated_disease <http://example.org/disease/Li-Fraumeni-syndrome>.\n```",
+ "gene": "TP53",
+ "disease": "Li-Fraumeni syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/PTEN> a bio:Gene;\n bio:has_associated_disease <http://example.org/CowdenSyndrome>.\n```",
+ "gene": "PTEN",
+ "disease": "Cowden syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 5,
+ "n_real": 3,
+ "n_fake": 2,
+ "term_existence_rate": 0.6,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_association",
+ "has_role"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/EGFR> a bio:Gene;\n bio:has_association [\n a bio:Association;\n bio:has_role bio:has_association_role;\n bio:has_member <http://example.org/non-small-cell-lung-carcinoma>;\n bio:has_member bio:Gene;\n bio:has_role bio:has_association_role;\n ].\n```",
+ "gene": "EGFR",
+ "disease": "non-small cell lung carcinoma"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 5,
+ "n_real": 4,
+ "n_fake": 1,
+ "term_existence_rate": 0.8,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_association"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/BRCA1> a bio:Gene ;\n bio:has_association [\n a bio:Association ;\n bio:has_member <http://example.org/BRCA1> ;\n bio:has_member <http://example.org/breastCancer> .\n ] .\n<http://example.org/breastCancer> a bio:Disease .\n```",
+ "gene": "BRCA1",
+ "disease": "breast cancer"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/BRCA2> a bio:Gene ;\n bio:has_associated_disease <http://example.org/breast_cancer> .\n```",
+ "gene": "BRCA2",
+ "disease": "breast cancer"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/KRAS> a bio:Gene ;\n bio:has_associated_disease <http://example.org/PancreaticCarcinoma>.\n```",
+ "gene": "KRAS",
+ "disease": "pancreatic carcinoma"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/APC> a bio:Gene;\n bio:has_associated_disease <http://example.org/familial_adenomatous_polyposis>.\n```",
+ "gene": "APC",
+ "disease": "familial adenomatous polyposis"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/RB1> a bio:Gene;\n bio:has_associated_disease bio:Retinoblastoma.\n```",
+ "gene": "RB1",
+ "disease": "retinoblastoma"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/VHL> a bio:Gene;\n bio:has_associated_disease <http://example.org/vonHippelLindauDisease>.\n```",
+ "gene": "VHL",
+ "disease": "von Hippel-Lindau disease"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 3,
+ "n_real": 1,
+ "n_fake": 2,
+ "term_existence_rate": 0.3333333333333333,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "association",
+ "has_association"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/CFTR> a bio:Gene;\n bio:has_association <http://example.org/disease/cystic-fibrosis>;\n bio:association bio:genomic_association_of_disease.\n```",
+ "gene": "CFTR",
+ "disease": "cystic fibrosis"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/HTT> a bio:Gene;\n bio:has_associated_disease <http://example.org/HuntingtonDisease>.\n```",
+ "gene": "HTT",
+ "disease": "Huntington disease"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/FMR1> a bio:Gene;\n bio:has_associated_disease bio:FragileXSyndrome.\n```",
+ "gene": "FMR1",
+ "disease": "fragile X syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/DMD> a bio:Gene ;\n bio:has_associated_disease <http://example.org/DuchenneMuscularDystrophy> .\n```",
+ "gene": "DMD",
+ "disease": "Duchenne muscular dystrophy"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_association",
+ "has_association_type"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/HBB> a bio:Gene ;\n bio:has_association <http://example.org/betaThalassemia> ;\n bio:has_association_type bio:genetically_associated_with .\n<http://example.org/betaThalassemia> a bio:Disease .\n```",
+ "gene": "HBB",
+ "disease": "beta thalassemia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/F8> a bio:Gene ;\n bio:has_associated_disease bio:Disease<http://example.org/hemophiliaA> .\n```",
+ "gene": "F8",
+ "disease": "hemophilia A"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 5,
+ "n_real": 3,
+ "n_fake": 2,
+ "term_existence_rate": 0.6,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_association",
+ "is_associated_with"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/LDLR> bio:has_association bio:Association<http://example.org/familial_hypercholesterolemia>.\n<http://example.org/familial_hypercholesterolemia> a bio:Disease.\n<http://example.org/LDLR> a bio:Gene.\n<http://example.org/LDLR> bio:is_associated_with <http://example.org/familial_hypercholesterolemia>.\n```",
+ "gene": "LDLR",
+ "disease": "familial hypercholesterolemia"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 4,
+ "n_real": 3,
+ "n_fake": 1,
+ "term_existence_rate": 0.75,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/MLH1> a bio:Gene ;\n bio:has_associated_disease [ a bio:Disease ;\n bio:name \"Lynch syndrome\" ] .\n```",
+ "gene": "MLH1",
+ "disease": "Lynch syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/NF1> a bio:Gene ;\n bio:has_associated_disease <http://example.org/disease/neurofibromatosis_type_1>.\n```",
+ "gene": "NF1",
+ "disease": "neurofibromatosis type 1"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/SMN1> a bio:Gene;\n bio:has_associated_disease <http://example.org/disease/spinal-muscular-atrophy>.\n```",
+ "gene": "SMN1",
+ "disease": "spinal muscular atrophy"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_association"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/PAH> a bio:Gene ;\n bio:has_association <http://example.org/disease/Phenylketonuria> .\n```",
+ "gene": "PAH",
+ "disease": "phenylketonuria"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/G6PD> a bio:Gene ;\n bio:has_associated_disease <http://example.org/G6PD_deficiency>.\n```",
+ "gene": "G6PD",
+ "disease": "G6PD deficiency"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 1,
+ "n_real": 0,
+ "n_fake": 1,
+ "term_existence_rate": 0.0,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/ATM> bio:has_associated_disease bio:AtaxiaTelangiectasia .\n```",
+ "gene": "ATM",
+ "disease": "ataxia telangiectasia"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/TSC1> a bio:Gene ;\n bio:has_associated_disease <http://example.org/tuberous_sclerosis> .\n```",
+ "gene": "TSC1",
+ "disease": "tuberous sclerosis"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/gene/MECP2> a bio:Gene;\n bio:has_associated_disease <http://example.org/disease/RettSyndrome>.\n```",
+ "gene": "MECP2",
+ "disease": "Rett syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/SCN1A> a bio:Gene;\n bio:has_associated_disease <http://example.org/DravetSyndrome>.\n```",
+ "gene": "SCN1A",
+ "disease": "Dravet syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 4,
+ "n_real": 2,
+ "n_fake": 2,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "association_target",
+ "has_association"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/COL1A1> a bio:Gene;\n bio:has_association bio:Association;\n bio:association_target <http://example.org/OsteogenesisImperfecta>.\n\n<http://example.org/OsteogenesisImperfecta> a bio:Disease.\n```",
+ "gene": "COL1A1",
+ "disease": "osteogenesis imperfecta"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 2,
+ "n_real": 1,
+ "n_fake": 1,
+ "term_existence_rate": 0.5,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/GBA> a bio:Gene;\n bio:has_associated_disease <http://example.org/Gaucher_disease>.\n```",
+ "gene": "GBA",
+ "disease": "Gaucher disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/HFE> a bio:Gene ;\n bio:has_associated_disease bio:Disease<http://example.org/hereditary_hemochromatosis>.\n```",
+ "gene": "HFE",
+ "disease": "hereditary hemochromatosis"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 3,
+ "n_real": 2,
+ "n_fake": 1,
+ "term_existence_rate": 0.6666666666666666,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": false,
+ "fake_terms": [
+ "has_associated_disease"
+ ],
+ "output": "```turtle\n@prefix bio: <https://w3id.org/biolink/vocab/> .\n\n<http://example.org/PKD1> a bio:Gene;\n bio:has_associated_disease <http://example.org/polycystic_kidney_disease>.\n<http://example.org/polycystic_kidney_disease> a bio:Disease.\n```",
+ "gene": "PKD1",
+ "disease": "polycystic kidney disease"
+ }
+]
\ No newline at end of file
diff --git a/case-studies/verifiabench/results/per_task_Qwen3-Coder-30B-A3B-Instruct-8bit.json b/case-studies/verifiabench/results/per_task_Qwen3-Coder-30B-A3B-Instruct-8bit.json
new file mode 100644
index 00000000..6971e1b0
--- /dev/null
+++ b/case-studies/verifiabench/results/per_task_Qwen3-Coder-30B-A3B-Instruct-8bit.json
@@ -0,0 +1,482 @@
+[
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/BRAF> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/BRAF_protein> .\n\n<https://example.org/disease/cardiofaciocutaneous_syndrome> a biolink:Disease .\n\n<https://example.org/gene/BRAF> biolink:causes <https://example.org/disease/cardiofaciocutaneous_syndrome> .\n```",
+ "gene": "BRAF",
+ "disease": "cardiofaciocutaneous syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 5,
+ "n_real": 5,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://identifiers.org/ncbigene:7157> a biolink:Gene ;\n biolink:has_gene_product <https://identifiers.org/ncbigene:7157> .\n\n<https://identifiers.org/omim:151627> a biolink:Disease ;\n biolink:has_phenotype <https://identifiers.org/omim:151627> .\n\n<https://identifiers.org/ncbigene:7157> biolink:causes <https://identifiers.org/omim:151627> .\n```",
+ "gene": "TP53",
+ "disease": "Li-Fraumeni syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 5,
+ "n_real": 5,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://identifiers.org/ncbigene:5728> a biolink:Gene ;\n biolink:has_gene_product <https://identifiers.org/ncbigene:5728> .\n\n<https://identifiers.org/omim:601209> a biolink:Disease ;\n biolink:has_phenotype <https://identifiers.org/omim:601209> .\n\n<https://identifiers.org/ncbigene:5728> biolink:causes <https://identifiers.org/omim:601209> .\n```",
+ "gene": "PTEN",
+ "disease": "Cowden syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/EGFR> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/EGFR_protein> .\n\n<https://example.org/non-small_cell_lung_carcinoma> a biolink:Disease .\n\n<https://example.org/EGFR> biolink:causes <https://example.org/non-small_cell_lung_carcinoma> .\n```",
+ "gene": "EGFR",
+ "disease": "non-small cell lung carcinoma"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/BRCA1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/BRCA1_protein> .\n\n<https://example.org/breast_cancer> a biolink:Disease .\n\n<https://example.org/BRCA1> biolink:causes <https://example.org/breast_cancer> .\n```",
+ "gene": "BRCA1",
+ "disease": "breast cancer"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 7,
+ "n_real": 7,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://identifiers.org/ncbigene:675> a biolink:Gene ;\n biolink:has_gene_product <https://identifiers.org/ncbigene:675#protein> .\n\n<https://identifiers.org/ncbigene:675#protein> a biolink:Protein ;\n biolink:associated_with <https://identifiers.org/mondo:0004975> .\n\n<https://identifiers.org/mondo:0004975> a biolink:Disease ;\n biolink:name \"breast cancer\" .\n\n<https://identifiers.org/ncbigene:675> biolink:causes <https://identifiers.org/mondo:0004975> .\n```",
+ "gene": "BRCA2",
+ "disease": "breast cancer"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/KRAS> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/KRAS_protein> .\n\n<https://example.org/disease/pancreatic_carcinoma> a biolink:Disease .\n\n<https://example.org/gene/KRAS> biolink:causes <https://example.org/disease/pancreatic_carcinoma> .\n```",
+ "gene": "KRAS",
+ "disease": "pancreatic carcinoma"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 3,
+ "n_real": 3,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/APC> a biolink:Gene ;\n biolink:associated_with <https://example.org/disease/familial_adenomatous_polyposis> .\n\n<https://example.org/disease/familial_adenomatous_polyposis> a biolink:Disease .\n```",
+ "gene": "APC",
+ "disease": "familial adenomatous polyposis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/RB1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/RB1_protein> .\n\n<https://example.org/disease/retinoblastoma> a biolink:Disease .\n\n<https://example.org/gene/RB1> biolink:causes <https://example.org/disease/retinoblastoma> .\n```",
+ "gene": "RB1",
+ "disease": "retinoblastoma"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/vocab/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n\n<https://example.org/gene/VHL> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/VHL> .\n\n<https://example.org/disease/von_Hippel-Lindau_disease> a biolink:Disease .\n\n<https://example.org/gene/VHL> biolink:causes <https://example.org/disease/von_Hippel-Lindau_disease> .\n```",
+ "gene": "VHL",
+ "disease": "von Hippel-Lindau disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/CFTR> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/CFTR> .\n\n<https://example.org/disease/cystic_fibrosis> a biolink:Disease .\n\n<https://example.org/gene/CFTR> biolink:causes <https://example.org/disease/cystic_fibrosis> .\n```",
+ "gene": "CFTR",
+ "disease": "cystic fibrosis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/HTT> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/HTT> .\n\n<https://example.org/disease/HuntingtonDisease> a biolink:Disease .\n\n<https://example.org/gene/HTT> biolink:causes <https://example.org/disease/HuntingtonDisease> .\n```",
+ "gene": "HTT",
+ "disease": "Huntington disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/FMR1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/FMR1_protein> .\n\n<https://example.org/fragile_X_syndrome> a biolink:Disease .\n\n<https://example.org/FMR1> biolink:causes <https://example.org/fragile_X_syndrome> .\n```",
+ "gene": "FMR1",
+ "disease": "fragile X syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/DMD> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/DMD_protein> .\n\n<https://example.org/disease/Duchenne_muscular_dystrophy> a biolink:Disease .\n\n<https://example.org/gene/DMD> biolink:causes <https://example.org/disease/Duchenne_muscular_dystrophy> .\n```",
+ "gene": "DMD",
+ "disease": "Duchenne muscular dystrophy"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/HBB> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/HBB_protein> .\n\n<https://example.org/beta_thalassemia> a biolink:Disease .\n\n<https://example.org/HBB> biolink:causes <https://example.org/beta_thalassemia> .\n```",
+ "gene": "HBB",
+ "disease": "beta thalassemia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/F8> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/F8> .\n\n<https://example.org/disease/hemophilia_A> a biolink:Disease .\n\n<https://example.org/gene/F8> biolink:causes <https://example.org/disease/hemophilia_A> .\n```",
+ "gene": "F8",
+ "disease": "hemophilia A"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/LDLR> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/LDLR_protein> .\n\n<https://example.org/disease/familial_hypercholesterolemia> a biolink:Disease .\n\n<https://example.org/gene/LDLR> biolink:causes <https://example.org/disease/familial_hypercholesterolemia> .\n```",
+ "gene": "LDLR",
+ "disease": "familial hypercholesterolemia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/MLH1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/MLH1> .\n\n<https://example.org/disease/LynchSyndrome> a biolink:Disease .\n\n<https://example.org/gene/MLH1> biolink:causes <https://example.org/disease/LynchSyndrome> .\n```",
+ "gene": "MLH1",
+ "disease": "Lynch syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/NF1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/NF1_protein> .\n\n<https://example.org/disease/neurofibromatosis_type_1> a biolink:Disease .\n\n<https://example.org/gene/NF1> biolink:causes <https://example.org/disease/neurofibromatosis_type_1> .\n```",
+ "gene": "NF1",
+ "disease": "neurofibromatosis type 1"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/SMN1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/SMN1_protein> .\n\n<https://example.org/spinal_muscular_atrophy> a biolink:Disease .\n\n<https://example.org/SMN1> biolink:causes <https://example.org/spinal_muscular_atrophy> .\n```",
+ "gene": "SMN1",
+ "disease": "spinal muscular atrophy"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/PAH> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/PAH> .\n\n<https://example.org/disease/phenylketonuria> a biolink:Disease .\n\n<https://example.org/gene/PAH> biolink:causes <https://example.org/disease/phenylketonuria> .\n```",
+ "gene": "PAH",
+ "disease": "phenylketonuria"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/G6PD> a biolink:Gene ;\n biolink:associated_with <https://example.org/disease/G6PD_deficiency> .\n\n<https://example.org/disease/G6PD_deficiency> a biolink:Disease ;\n biolink:has_gene <https://example.org/gene/G6PD> .\n```",
+ "gene": "G6PD",
+ "disease": "G6PD deficiency"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 5,
+ "n_real": 5,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://identifiers.org/ncbigene:472> a biolink:Gene ;\n biolink:has_gene_product <https://identifiers.org/ncbigene:472> .\n\n<https://identifiers.org/omim:208900> a biolink:Disease ;\n biolink:has_phenotype <https://identifiers.org/omim:208900> .\n\n<https://identifiers.org/ncbigene:472> biolink:causes <https://identifiers.org/omim:208900> .\n```",
+ "gene": "ATM",
+ "disease": "ataxia telangiectasia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/TSC1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/TSC1_protein> .\n\n<https://example.org/disease/tuberous_sclerosis> a biolink:Disease .\n\n<https://example.org/gene/TSC1> biolink:causes <https://example.org/disease/tuberous_sclerosis> .\n```",
+ "gene": "TSC1",
+ "disease": "tuberous sclerosis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/MECP2> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/MECP2> .\n\n<https://example.org/disease/Rett_syndrome> a biolink:Disease .\n\n<https://example.org/gene/MECP2> biolink:causes <https://example.org/disease/Rett_syndrome> .\n```",
+ "gene": "MECP2",
+ "disease": "Rett syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/SCN1A> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/SCN1A_protein> .\n\n<https://example.org/disease/Dravet_syndrome> a biolink:Disease .\n\n<https://example.org/gene/SCN1A> biolink:causes <https://example.org/disease/Dravet_syndrome> .\n```",
+ "gene": "SCN1A",
+ "disease": "Dravet syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/COL1A1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/COL1A1_protein> .\n\n<https://example.org/disease/osteogenesis_imperfecta> a biolink:Disease .\n\n<https://example.org/gene/COL1A1> biolink:causes <https://example.org/disease/osteogenesis_imperfecta> .\n```",
+ "gene": "COL1A1",
+ "disease": "osteogenesis imperfecta"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/GBA> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/GBA_protein> .\n\n<https://example.org/disease/Gaucher_disease> a biolink:Disease .\n\n<https://example.org/gene/GBA> biolink:causes <https://example.org/disease/Gaucher_disease> .\n```",
+ "gene": "GBA",
+ "disease": "Gaucher disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/HFE_gene> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/HFE_protein> .\n\n<https://example.org/hereditary_hemochromatosis> a biolink:Disease .\n\n<https://example.org/HFE_gene> biolink:causes <https://example.org/hereditary_hemochromatosis> .\n```",
+ "gene": "HFE",
+ "disease": "hereditary hemochromatosis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 4,
+ "n_fake": 0,
+ "term_existence_rate": 1.0,
+ "raw_ok": true,
+ "verified_ok": true,
+ "has_gene": true,
+ "has_disease": true,
+ "has_assoc": true,
+ "fake_terms": [],
+ "output": "```turtle\n<https://example.org/gene/PKD1> a biolink:Gene ;\n biolink:has_gene_product <https://example.org/protein/PKD1_protein> .\n\n<https://example.org/disease/Polycystic_Kidney_Disease> a biolink:Disease .\n\n<https://example.org/gene/PKD1> biolink:causes <https://example.org/disease/Polycystic_Kidney_Disease> .\n```",
+ "gene": "PKD1",
+ "disease": "polycystic kidney disease"
+ }
+]
\ No newline at end of file
diff --git a/case-studies/verifiabench/results/per_task_gemma-2-2b-it-4bit.json b/case-studies/verifiabench/results/per_task_gemma-2-2b-it-4bit.json
new file mode 100644
index 00000000..31c60179
--- /dev/null
+++ b/case-studies/verifiabench/results/per_task_gemma-2-2b-it-4bit.json
@@ -0,0 +1,510 @@
+[
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 1,
+ "n_fake": 3,
+ "term_existence_rate": 0.25,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "BRAF",
+ "associatedDisease",
+ "cardiofaciocutaneous"
+ ],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:BRAF a biolink:Gene ;\nbiolink:associatedDisease biolink:cardiofaciocutaneous syndrome .\n``` \n",
+ "gene": "BRAF",
+ "disease": "cardiofaciocutaneous syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 1,
+ "n_fake": 3,
+ "term_existence_rate": 0.25,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "Li",
+ "TP53",
+ "associatedDisease"
+ ],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:TP53 a biolink:Gene ;\nbiolink:associatedDisease biolink:Li-Fraumeni syndrome .\n``` \n",
+ "gene": "TP53",
+ "disease": "Li-Fraumeni syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 1,
+ "n_fake": 3,
+ "term_existence_rate": 0.25,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "Cowden",
+ "PTEN",
+ "associatedDisease"
+ ],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:PTEN a biolink:Gene ;\nbiolink:associatedDisease biolink:Cowden syndrome .\n``` \n",
+ "gene": "PTEN",
+ "disease": "Cowden syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:EGFR a biolink:Gene ;\nbiolink:associatedDisease biolink:nonSmallCellLungCarcinoma .\n``` \n",
+ "gene": "EGFR",
+ "disease": "non-small cell lung carcinoma"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:BRCA1 a biolink:Gene ;\nbiolink:associatedDisease biolink:breastCancer .\n``` \n",
+ "gene": "BRCA1",
+ "disease": "breast cancer"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:BRCA2 a biolink:Gene ;\nbiolink:associatedDisease biolink:BreastCancer .\n``` \n",
+ "gene": "BRCA2",
+ "disease": "breast cancer"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:KRAS a biolink:Gene ;\nbiolink:associatedDisease biolink:pancreaticCarcinoma .\n``` \n",
+ "gene": "KRAS",
+ "disease": "pancreatic carcinoma"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:APC a biolink:Gene ;\nbiolink:associatedDisease biolink:FamilialAdenomatousPolyposis .\n``` \n",
+ "gene": "APC",
+ "disease": "familial adenomatous polyposis"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:RB1 a biolink:Gene ;\nbiolink:associatedDisease biolink:Retinoblastoma .\n``` \n",
+ "gene": "RB1",
+ "disease": "retinoblastoma"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:VHL a biolink:Gene ;\nbiolink:associatedDisease biolink:vonHippelLindauDisease .\n``` \n",
+ "gene": "VHL",
+ "disease": "von Hippel-Lindau disease"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:CFTR a biolink:Gene ;\nbiolink:associatedDisease biolink:CysticFibrosis .\n``` \n",
+ "gene": "CFTR",
+ "disease": "cystic fibrosis"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:HTT rdf:type biolink:Gene .\nbiolink:HTT biolink:associatedWith biolink:HuntingtonDisease .\n``` \n",
+ "gene": "HTT",
+ "disease": "Huntington disease"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 1,
+ "n_fake": 3,
+ "term_existence_rate": 0.25,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "FMR1",
+ "FragileX",
+ "associatedDisease"
+ ],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:FMR1 a biolink:Gene ;\nbiolink:associatedDisease biolink:FragileX syndrome .\n``` \n",
+ "gene": "FMR1",
+ "disease": "fragile X syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:DMD a biolink:Gene ;\nbiolink:associatedDisease biolink:DuchenneMuscularDystrophy .\n``` \n",
+ "gene": "DMD",
+ "disease": "Duchenne muscular dystrophy"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:HBB a biolink:Gene ;\nbiolink:associatedDisease biolink:BetaThalassemia .\n``` \n",
+ "gene": "HBB",
+ "disease": "beta thalassemia"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:F8 a biolink:Gene ;\nbiolink:associatedDisease biolink:HemophiliaA .\n``` \n",
+ "gene": "F8",
+ "disease": "hemophilia A"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:LDLR a biolink:Gene ;\nbiolink:associatedDisease biolink:FamilialHypercholesterolemia .\n``` \n",
+ "gene": "LDLR",
+ "disease": "familial hypercholesterolemia"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 1,
+ "n_fake": 3,
+ "term_existence_rate": 0.25,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "Lynch",
+ "MLH1",
+ "associatedDisease"
+ ],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:MLH1 a biolink:Gene ;\nbiolink:associatedDisease biolink:Lynch syndrome .\n``` \n",
+ "gene": "MLH1",
+ "disease": "Lynch syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:NF1 a biolink:Gene ;\nbiolink:associatedDisease biolink:NeurofibromatosisType1 .\n``` \n",
+ "gene": "NF1",
+ "disease": "neurofibromatosis type 1"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:SMN1 a biolink:Gene ;\nbiolink:associatedDisease biolink:spinalMuscularAtrophy .\n``` \n",
+ "gene": "SMN1",
+ "disease": "spinal muscular atrophy"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:PAH rdf:type biolink:Gene .\nbiolink:PAH biolink:associatedWith biolink:Phenylketonuria .\n``` \n",
+ "gene": "PAH",
+ "disease": "phenylketonuria"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:G6PD rdf:type biolink:Gene .\nbiolink:G6PD biolink:associatedWith biolink:G6PD_deficiency .\n``` \n",
+ "gene": "G6PD",
+ "disease": "G6PD deficiency"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:ATM a biolink:Gene ;\nbiolink:associatedDisease biolink:ataxiaTelangiectasia .\n``` \n",
+ "gene": "ATM",
+ "disease": "ataxia telangiectasia"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:TSC1 a biolink:Gene ;\nbiolink:associatedDisease biolink:TuberousSclerosis .\n``` \n",
+ "gene": "TSC1",
+ "disease": "tuberous sclerosis"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 1,
+ "n_fake": 3,
+ "term_existence_rate": 0.25,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "MECP2",
+ "Rett",
+ "associatedDisease"
+ ],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:MECP2 a biolink:Gene ;\nbiolink:associatedDisease biolink:Rett syndrome .\n``` \n",
+ "gene": "MECP2",
+ "disease": "Rett syndrome"
+ },
+ {
+ "parsed_turtle": false,
+ "n_terms": 4,
+ "n_real": 1,
+ "n_fake": 3,
+ "term_existence_rate": 0.25,
+ "raw_ok": true,
+ "verified_ok": false,
+ "has_gene": true,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [
+ "Dravet",
+ "SCN1A",
+ "associatedDisease"
+ ],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:SCN1A a biolink:Gene ;\nbiolink:associatedDisease biolink:Dravet syndrome .\n``` \n",
+ "gene": "SCN1A",
+ "disease": "Dravet syndrome"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:COL1A1 a biolink:Gene ;\nbiolink:associatedDisease biolink:OsteogenesisImperfecta .\n``` \n",
+ "gene": "COL1A1",
+ "disease": "osteogenesis imperfecta"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:GBA rdf:type biolink:Gene .\nbiolink:GBA biolink:associatedWith biolink:GaucherDisease .\n``` \n",
+ "gene": "GBA",
+ "disease": "Gaucher disease"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:HFE a biolink:Gene ;\nbiolink:associatedDisease biolink:HereditaryHemochromatosis .\n``` \n",
+ "gene": "HFE",
+ "disease": "hereditary hemochromatosis"
+ },
+ {
+ "parsed_turtle": true,
+ "n_terms": 0,
+ "n_real": 0,
+ "n_fake": 0,
+ "term_existence_rate": 0.0,
+ "raw_ok": false,
+ "verified_ok": false,
+ "has_gene": false,
+ "has_disease": false,
+ "has_assoc": false,
+ "fake_terms": [],
+ "output": "```turtle\n@prefix biolink: <https://w3id.org/biolink/> .\n@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\nbiolink:PKD1 a biolink:Gene ;\nbiolink:associatedDisease biolink:PolycysticKidneyDisease .\n``` \n",
+ "gene": "PKD1",
+ "disease": "polycystic kidney disease"
+ }
+]
\ No newline at end of file
diff --git a/case-studies/verifiabench/results/results.json b/case-studies/verifiabench/results/results.json
new file mode 100644
index 00000000..1011d579
--- /dev/null
+++ b/case-studies/verifiabench/results/results.json
@@ -0,0 +1,57 @@
+{
+ "authority": "Biolink Model",
+ "n_tasks": 30,
+ "oracle": "closed-world term existence + structural constraints (deterministic, no LLM judge)",
+ "leaderboard": [
+ {
+ "model": "mlx-community/Qwen3-Coder-30B-A3B-Instruct-8bit",
+ "n_tasks": 30,
+ "raw_capability": 1.0,
+ "verified_capability": 1.0,
+ "mean_term_existence": 1.0,
+ "tasks_with_any_fabricated_term": 0,
+ "total_fabricated_terms": 0,
+ "verification_gap": 0.0
+ },
+ {
+ "model": "mlx-community/gemma-2-2b-it-4bit",
+ "n_tasks": 30,
+ "raw_capability": 0.233,
+ "verified_capability": 0.0,
+ "mean_term_existence": 0.058,
+ "tasks_with_any_fabricated_term": 7,
+ "total_fabricated_terms": 21,
+ "verification_gap": 0.233
+ },
+ {
+ "model": "mlx-community/Qwen2.5-0.5B-Instruct-4bit",
+ "n_tasks": 30,
+ "raw_capability": 0.0,
+ "verified_capability": 0.0,
+ "mean_term_existence": 0.0,
+ "tasks_with_any_fabricated_term": 0,
+ "total_fabricated_terms": 0,
+ "verification_gap": 0.0
+ },
+ {
+ "model": "mlx-community/Qwen2.5-3B-Instruct-4bit",
+ "n_tasks": 30,
+ "raw_capability": 1.0,
+ "verified_capability": 0.0,
+ "mean_term_existence": 0.506,
+ "tasks_with_any_fabricated_term": 30,
+ "total_fabricated_terms": 65,
+ "verification_gap": 1.0
+ },
+ {
+ "model": "mlx-community/Llama-3.2-3B-Instruct-4bit",
+ "n_tasks": 30,
+ "raw_capability": 1.0,
+ "verified_capability": 0.0,
+ "mean_term_existence": 0.509,
+ "tasks_with_any_fabricated_term": 30,
+ "total_fabricated_terms": 49,
+ "verification_gap": 1.0
+ }
+ ]
+}
\ No newline at end of file
diff --git a/case-studies/verifiabench/run-demo.sh b/case-studies/verifiabench/run-demo.sh
new file mode 100755
index 00000000..a5d870cf
--- /dev/null
+++ b/case-studies/verifiabench/run-demo.sh
@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+# Reproduce verifiabench against any OpenAI-compatible endpoint (default: local MLX on :8080).
+# Set VB_API to point elsewhere. Pass model ids as arguments.
+set -euo pipefail
+cd "$(dirname "$0")"
+if [ ! -d .venv ]; then python3 -m venv .venv; ./.venv/bin/pip -q install -r requirements.txt; fi
+./.venv/bin/python src/verifiabench.py "$@"
+echo; echo "See results/results.json and results/SUMMARY.md"
diff --git a/case-studies/verifiabench/src/verifiabench.py b/case-studies/verifiabench/src/verifiabench.py
new file mode 100644
index 00000000..4493b9ad
--- /dev/null
+++ b/case-studies/verifiabench/src/verifiabench.py
@@ -0,0 +1,152 @@
+"""
+verifiabench — an un-game-able benchmark for scientific-workflow LLM reliability.
+
+Existing science-LLM benchmarks grade the answer with string match or an LLM judge, so a
+fluent-but-fabricated term can score as correct. verifiabench grades with a CLOSED-WORLD oracle:
+every ontology term a model emits must EXIST in the authority (here the real Biolink Model) and
+the output must satisfy the task's structural constraints. Correctness is set-membership plus
+constraint satisfaction, not similarity, so fluency cannot buy a point.
+
+Task: given a real gene-disease fact, write Biolink-typed RDF asserting it. The oracle extracts
+every biolink term from the output and checks it against the Biolink Model's declared terms, then
+checks the structure (a real Gene, a real Disease, a real association predicate). The headline is
+the gap between RAW capability (the output looks like structured Biolink RDF) and VERIFIED
+capability (its terms are real and the structure holds).
+
+Deterministic oracle, no LLM judge. Model under test is queried over an OpenAI-compatible API.
+"""
+import json, os, re, sys, time
+import requests
+import rdflib
+from rdflib import RDF
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+DATA, RES = os.path.join(ROOT, "data"), os.path.join(ROOT, "results")
+os.makedirs(RES, exist_ok=True)
+API = os.environ.get("VB_API", "http://localhost:8080/v1/chat/completions")
+BL = "https://w3id.org/biolink/vocab/"
+
+with open(os.path.join(DATA, "biolink_vocab.json")) as f:
+ DECLARED = set(json.load(f)["declared"])
+GENE, DISEASE = BL + "Gene", BL + "Disease"
+# real Biolink slots that legitimately relate a gene to a condition/disease
+ASSOC_SLOTS = {BL + s for s in ("gene_associated_with_condition", "associated_with",
+ "related_to", "contributes_to", "causes", "biomarker_for")} & DECLARED
+
+# 30 real, well-established gene-disease facts
+FACTS = [
+ ("BRAF", "cardiofaciocutaneous syndrome"), ("TP53", "Li-Fraumeni syndrome"),
+ ("PTEN", "Cowden syndrome"), ("EGFR", "non-small cell lung carcinoma"),
+ ("BRCA1", "breast cancer"), ("BRCA2", "breast cancer"), ("KRAS", "pancreatic carcinoma"),
+ ("APC", "familial adenomatous polyposis"), ("RB1", "retinoblastoma"), ("VHL", "von Hippel-Lindau disease"),
+ ("CFTR", "cystic fibrosis"), ("HTT", "Huntington disease"), ("FMR1", "fragile X syndrome"),
+ ("DMD", "Duchenne muscular dystrophy"), ("HBB", "beta thalassemia"), ("F8", "hemophilia A"),
+ ("LDLR", "familial hypercholesterolemia"), ("MLH1", "Lynch syndrome"), ("NF1", "neurofibromatosis type 1"),
+ ("SMN1", "spinal muscular atrophy"), ("PAH", "phenylketonuria"), ("G6PD", "G6PD deficiency"),
+ ("ATM", "ataxia telangiectasia"), ("TSC1", "tuberous sclerosis"), ("MECP2", "Rett syndrome"),
+ ("SCN1A", "Dravet syndrome"), ("COL1A1", "osteogenesis imperfecta"), ("GBA", "Gaucher disease"),
+ ("HFE", "hereditary hemochromatosis"), ("PKD1", "polycystic kidney disease"),
+]
+
+PROMPT = ("Using the Biolink Model, write RDF in Turtle asserting that the gene {g} is associated "
+ "with the disease {d}. Use only real Biolink Model classes and predicates with the "
+ "`biolink:` prefix (biolink: = <{ns}>). Output only the Turtle.")
+
+def query(model, prompt, timeout=120):
+ r = requests.post(API, json={"model": model, "messages": [{"role": "user", "content": prompt}],
+ "max_tokens": 400, "temperature": 0}, timeout=timeout)
+ return r.json()["choices"][0]["message"]["content"]
+
+def strip_fence(text):
+ m = re.search(r"```(?:turtle|ttl)?\s*(.*?)```", text, re.S)
+ return m.group(1) if m else text
+
+def extract(output):
+ """Return (biolink_terms:set, structure:dict, parsed:bool). Robust to whatever prefix a
+ model maps to the Biolink namespace: parse the Turtle with rdflib (prefix-aware) and fall
+ back to prefix-resolved regex if it does not parse."""
+ text = strip_fence(output)
+ terms = set()
+ struct = {"gene": False, "disease": False, "assoc": False}
+ try:
+ g = rdflib.Graph().parse(data=text, format="turtle")
+ except Exception:
+ g = None
+ if g is not None and len(g):
+ for s, p, o in g:
+ if str(p).startswith(BL): terms.add(str(p))
+ if p == RDF.type and str(o).startswith(BL): terms.add(str(o))
+ typed = {}
+ for s, _, o in g.triples((None, RDF.type, None)):
+ typed.setdefault(str(s), set()).add(str(o))
+ struct["gene"] = any(GENE in ts for ts in typed.values())
+ struct["disease"] = any(DISEASE in ts for ts in typed.values())
+ struct["assoc"] = any(str(p) in ASSOC_SLOTS for _, p, _ in g)
+ return terms, struct, True
+ # fallback: find prefixes bound to the biolink namespace, then extract their CURIEs
+ prefixes = set(re.findall(r"@prefix\s+(\w+):\s*<" + re.escape(BL) + r">", text))
+ prefixes.add("biolink")
+ for pre in prefixes:
+ for m in re.findall(r"\b" + re.escape(pre) + r":([A-Za-z_]\w*)", text):
+ terms.add(BL + m)
+ for m in re.findall(re.escape(BL) + r"([A-Za-z_]\w*)", text):
+ terms.add(BL + m)
+ struct = {"gene": GENE in terms, "disease": DISEASE in terms, "assoc": len(terms & ASSOC_SLOTS) > 0}
+ return terms, struct, False
+
+def score(output):
+ terms, struct, parsed = extract(output)
+ real = {t for t in terms if t in DECLARED}
+ fake = terms - real
+ raw_ok = len(terms) > 0 # produced biolink-namespace terms
+ verified_ok = struct["gene"] and struct["disease"] and struct["assoc"] and len(fake) == 0
+ return {
+ "parsed_turtle": parsed, "n_terms": len(terms), "n_real": len(real), "n_fake": len(fake),
+ "term_existence_rate": (len(real) / len(terms)) if terms else 0.0,
+ "raw_ok": raw_ok, "verified_ok": verified_ok,
+ "has_gene": struct["gene"], "has_disease": struct["disease"], "has_assoc": struct["assoc"],
+ "fake_terms": sorted(t.split("/")[-1] for t in fake)[:6],
+ "output": output[:600],
+ }
+
+def run_model(model):
+ per = []
+ for i, (g, d) in enumerate(FACTS):
+ try:
+ out = query(model, PROMPT.format(g=g, d=d, ns=BL))
+ except Exception as e:
+ out = f"__ERROR__ {e}"
+ s = score(out); s["gene"] = g; s["disease"] = d
+ per.append(s)
+ print(f" [{i+1}/{len(FACTS)}] {g}/{d[:22]:22s} raw={s['raw_ok']} verified={s['verified_ok']} "
+ f"real={s['n_real']} fake={s['n_fake']}", flush=True)
+ n = len(per)
+ agg = {
+ "model": model, "n_tasks": n,
+ "raw_capability": round(sum(x["raw_ok"] for x in per) / n, 3),
+ "verified_capability": round(sum(x["verified_ok"] for x in per) / n, 3),
+ "mean_term_existence": round(sum(x["term_existence_rate"] for x in per) / n, 3),
+ "tasks_with_any_fabricated_term": sum(x["n_fake"] > 0 for x in per),
+ "total_fabricated_terms": sum(x["n_fake"] for x in per),
+ }
+ agg["verification_gap"] = round(agg["raw_capability"] - agg["verified_capability"], 3)
+ return agg, per
+
+def main():
+ models = sys.argv[1:] or ["mlx-community/Qwen2.5-3B-Instruct-4bit"]
+ board = []
+ for m in models:
+ print(f"[*] running {m} ...", flush=True)
+ agg, per = run_model(m)
+ board.append(agg)
+ json.dump(per, open(os.path.join(RES, "per_task_" + m.split("/")[-1] + ".json"), "w"), indent=2)
+ print(" ", json.dumps(agg), flush=True)
+ board.sort(key=lambda a: a["verified_capability"], reverse=True)
+ out = {"authority": "Biolink Model", "n_tasks": len(FACTS),
+ "oracle": "closed-world term existence + structural constraints (deterministic, no LLM judge)",
+ "leaderboard": board}
+ json.dump(out, open(os.path.join(RES, "results.json"), "w"), indent=2)
+ print(json.dumps(out, indent=2))
+
+if __name__ == "__main__":
+ main()