Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

Read the prover's refutation back, and say exactly how much of it was checked (#155)

Fabio Rovai32029c577b15
diff --git a/.gitattributes b/.gitattributes
index 2740a151..7f93371c 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -19,6 +19,16 @@
*.ttl text eol=lf
*.trig text eol=lf
+# First-order problems and prover transcripts, for a third version of the same
+# reason. `the_fixture_problem_is_what_this_exporter_emits` compares the
+# recorded problem byte for byte against what `fol --format tptp` emits now,
+# because a proof of the old problem says nothing about a new one. The emitter
+# writes LF, so a CRLF checkout made that comparison fail on Windows alone, on
+# a file that is correct in the repository. The prover transcripts beside it
+# are recorded output and are pinned for the same reason.
+*.p text eol=lf
+*.tstp text eol=lf
+
# ONE EXCEPTION, and it is the exception that proves the rule above. The
# `probe_crlf_*` files in isabelle/fixtures-differential/ exist to feed the two
# certificate checkers a file that DOES carry carriage returns, and to pin that
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 772a0f55..1eaa8744 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -231,6 +231,48 @@ All notable changes to Open Ontologies are documented here.
than letting it be assumed away.
### Added
+- **A prover's refutation is read back and re-checked, and it is still an oracle opinion.**
+ `src/tstp.rs` parses the TSTP derivation Vampire or E prints, and `fol-prove` /
+ `onto_fol_prove` / a second column in `tools/fol_differential.py` report on it. Three things
+ are established and each costs more than the last. **The prover refuted OUR problem**: every
+ leaf is matched against the emitted problem file by the name its own `file('…', NAME)`
+ annotation gives, by the PARSED formula, and by the ROLE, so a prover pointed at a stale file,
+ a file edited since under the same names, or one whose conjecture was presented as an axiom is
+ caught. **The derivation is a well-founded DAG ending in `$false`**: every parent resolves, no
+ name is used twice, the relation is acyclic, and nodes the empty clause does not depend on are
+ counted and excluded. **Some steps are recomputed**: binary resolution, subsumption resolution
+ in its three spellings (whose conclusion IS the binary resolvent, because the side clause's
+ remainder is contained in the main clause's), factoring, duplicate literal removal, flattening,
+ trivial inequality removal, equality resolution, and the negation of the conjecture, each
+ replayed with a syntactic unifier with an occurs check and compared up to a bijective renaming
+ of variables. **Everything else is named and counted as unchecked with a reason**:
+ clausification, Skolemisation, AVATAR splitting, every SAT-solver step, and every step of E's
+ whose premise is an inline inference record and so carries no formula. Eight verdict words,
+ and **an unchecked step prevents the strongest one**, which is
+ `refutation_fully_replayed` and is still NOT `unsatisfiable`: the calculus's soundness is
+ machine-checked nowhere in `lean/` and the replayer is ordinary Rust. A step whose rule IS
+ implemented and still does not reconstruct gets its own word,
+ `refutation_step_not_reconstructed`, because filing it under "unchecked" would let a forged
+ step hide behind a rule name and filing it under "rejected" would accuse someone else's prover
+ on this module's word alone. Measured over FOAF, 181 claimed entailments, one problem each:
+ Vampire 5.1.0 refuted all 181 with 1279 of 3314 steps replayed, E 3.2.5 refuted all 181 with
+ 181 of 4644, all 534 leaves matched on both sides, nothing rejected. The order-of-magnitude gap
+ is structural: Vampire attaches a conclusion to every inference, E nests inference records that
+ carry none. Negative tests take a genuine Vampire refutation apart one mutation at a time — a
+ changed leaf, a leaf naming a formula the problem lacks, a conjecture relabelled as an axiom, a
+ dangling parent, a cycle, a duplicated name, a resolvent that is not the resolvent, a forged
+ negated conjecture, a resolution needing a cyclic binding — and each must be caught by the field
+ meant to catch it. See the addendum to
+ [decision 0005](docs/decisions/0005-a-prover-is-an-oracle-and-a-translation-is-a-theorem.md).
+- **`--atp vampire` has actually been run.** `docs/first-order-export.md` said the Vampire branch
+ of the prover detection had never executed because Vampire was not installed. It is installed,
+ it has been run over FOAF, and the argument vector gained `--proof tptp`, without which
+ `--mode casc` prints a display format rather than a TSTP derivation. E's gained
+ `--proof-object` for the same reason.
+- **The MCP server's own tool count had gone stale in a second place.** Its instructions string
+ states the total twice and the second copy said 112 while the rest of the repository said 114.
+ No shape in `readme_claims_test.rs` covered that phrasing, so neither of its two tests noticed.
+ Both now do.
- **A conclusion can name the axioms responsible for it, and a derived triple carries an
algebraic expression over the asserted ones.** Two tools, `onto_justify` and
`onto_provenance`, over one substrate that already existed and that nothing read back.
diff --git a/README.md b/README.md
index 4068c3a0..f38797aa 100644
--- a/README.md
+++ b/README.md
@@ -321,7 +321,11 @@ concrete instance, and every one of the twenty-one conditions must have a satisf
an "unsatisfiable" from any prover is testimony, not a certificate. One clash rule has a semantic
condition and is certified; the other sixteen are detected and deliberately carry no certificate.
A model is a finite object and can be checked, which is why satisfiability can be certified and
-unsatisfiability cannot.
+unsatisfiability cannot. The testimony does now come with an exhibit: `fol-prove` reads the
+derivation Vampire or E prints, matches every leaf against the problem this engine emitted, checks
+the DAG, and recomputes the resolution-family steps, naming and counting every step it could not.
+Over FOAF that was 1279 of Vampire's 3314 steps and 181 of E's 4644. It is evidence about what the
+prover was answering about; it is not a proof, and no verdict word there says it is.
**Governance.** `main` requires the CI, Lean, SHACL-conformance and Python checks to pass, refuses
force pushes and deletion, and applies to administrators too. Commits are not signed: a signing key
@@ -437,8 +441,8 @@ Restart, and the `onto_*` tools are available. Cursor, Windsurf, Zed and VS Code
## What is in the box
-**119 tools** to build, validate, query, diff, lint, version, reason over, align, plan, certify
-and govern RDF and OWL, over an in-memory Oxigraph store. A default build advertises 111 tools.
+**120 tools** to build, validate, query, diff, lint, version, reason over, align, plan, certify
+and govern RDF and OWL, over an in-memory Oxigraph store. A default build advertises 112 tools.
Eight need an optional Cargo feature and return an error without it: four need `embeddings`, two
need `plugins`, two need `postgres` or `duckdb`. The published binaries and the GHCR image are
built with the default feature set, so they do not carry those eight.
diff --git a/docs/architecture.md b/docs/architecture.md
index df75743f..11594ab0 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -24,7 +24,7 @@ flowchart TD
REST["REST API\n/api/query · /api/update\n/api/save · /api/load · /api/lineage"]
end
- subgraph ToolGroups["119 Tools"]
+ subgraph ToolGroups["120 Tools"]
direction LR
Core["Core\nvalidate · load · save · clear\nstats · query · diff · lint\nconvert · status"]
DataPipe["Data Pipeline\nmap · ingest · shacl\nreason · extend · import-schema"]
diff --git a/docs/ci-gates.md b/docs/ci-gates.md
index 3a9dbe80..515f3845 100644
--- a/docs/ci-gates.md
+++ b/docs/ci-gates.md
@@ -76,6 +76,8 @@ job invokes the file at all.
| `rule_syntax_frontend_test.rs` | lake | `lean` job | **strict** |
| `reason_horn_emit_test.rs` | lake + `tests/fixtures/horn/` (in tree) | `lean` job | **strict** |
| `cross_kernel_differential_test.rs` | lake **and** Poly/ML | `lean` job | **strict** |
+| `tstp_derivation_test.rs` | the recorded Vampire and E derivations in `tests/fixtures/tstp/` (in tree); ONE test also wants vampire on `PATH` | any job for the recorded part; no job installs a prover | **skips** — 34 of 35 need nothing, but the file cannot be made strict while the 35th wants a prover no job installs |
+| `tstp_cli_test.rs` | the same fixtures + the debug CLI | any job | **strict** |
| `clinical_test.rs` | `data/crosswalks.parquet` | nothing — `data/` is gitignored | **skips** |
| `embed_test.rs` | ONNX model in `~/.open-ontologies/models/` | `features/depth` compiles it; no job runs `open-ontologies init` | **skips** |
| `embedding_e2e_test.rs` | same ONNX model + tokenizer | same | **skips** |
@@ -110,6 +112,15 @@ The other seven files under `python/tests/` have no skip path.
## What is still open, and why
+- **No job installs a first-order prover.** `tools/fol_differential.py` skips
+ loudly without one and `FOL_DIFF_REQUIRE_ATP=1` turns that skip into a
+ failure, but no workflow sets it because no workflow installs E or Vampire.
+ The derivation checker added on 15 September 2026 is in the same position: its
+ recorded fixtures run everywhere, and `a_live_vampire_run_agrees_with_the_recorded_one`
+ skips. The recorded half is not a substitute for the live half, since it pins
+ the checker against two real derivations and not against whatever the
+ installed prover does today, so a `brew install vampire` step, or its apt
+ equivalent, is the smallest thing that would close this.
- **`clinical_test.rs`, `embed_test.rs`, `embedding_e2e_test.rs`.** All three
want an artefact the repository deliberately does not carry: a licensed
crosswalk table and a model download. Closing them means a job that fetches
diff --git a/docs/decisions/0005-a-prover-is-an-oracle-and-a-translation-is-a-theorem.md b/docs/decisions/0005-a-prover-is-an-oracle-and-a-translation-is-a-theorem.md
index 92a35d2d..bcab5af8 100644
--- a/docs/decisions/0005-a-prover-is-an-oracle-and-a-translation-is-a-theorem.md
+++ b/docs/decisions/0005-a-prover-is-an-oracle-and-a-translation-is-a-theorem.md
@@ -217,3 +217,155 @@ and they point opposite ways, so both are written down rather than left to a rea
action and not one to take on someone's behalf. The old ITTF free-standards site closed in 2025;
the catalogue pages are reachable at `committee.iso.org` when `www.iso.org` returns 403. Until
that is done, every clause cited here is scoped to the edition and the text actually read.
+
+## Addendum, 15 September 2026 · The oracle's answer is now an object, and it is still an oracle
+
+Nothing above is retracted. Item 5 stands word for word: an ATP verdict is an oracle opinion and
+is labelled as one everywhere it appears. What has changed is that the opinion is no longer a
+single word. Both installed provers will print the derivation they found, a derivation is a finite
+object, and `src/tstp.rs` reads one back and re-checks what can honestly be re-checked. The
+capability is `fol-prove`, `onto_fol_prove` and a second column in `tools/fol_differential.py`, and
+the rest of this addendum is the boundary between what it now establishes and what it still
+assumes.
+
+### What is now evidence
+
+1. **The prover refuted OUR problem.** Every leaf of the derivation is matched against the problem
+ file the exporter wrote: by the name the prover's own `file(…,NAME)` annotation gives, by the
+ PARSED formula, and by the ROLE. All three matter and none is a formality. The name catches a
+ prover pointed at a stale or different file. The formula catches a file that has since been
+ edited under the same names. The role catches a conjecture presented as an axiom, which would
+ make the refutation say nothing about entailment while looking perfect. This is the cheapest of
+ the three things below and by a distance the most valuable, because until it existed an `AGREE`
+ row in the differential simply assumed it.
+2. **The derivation is a well-founded DAG ending in the empty clause.** Every parent reference
+ resolves to a node that exists, no name is used twice, the parent relation is acyclic, and the
+ node nothing else cites is `$false`. Nodes the empty clause does not depend on are counted and
+ excluded from every tally, because counting them would report work the refutation does not rest
+ on.
+3. **Some steps are recomputed.** `resolution`, `subsumption_resolution` and its forward and
+ backward spellings, `factoring`, `duplicate_literal_removal`, `flattening`,
+ `trivial_inequality_removal`, `equality_resolution`, and `negated_conjecture` /
+ `assume_negation`. Each is replayed from its premises with a syntactic unifier with an occurs
+ check, and the conclusion must agree up to a bijective renaming of variables.
+4. **Whether the conjecture was used at all.** `what_was_refuted` is
+ `axioms_and_the_negated_conjecture` or `axioms_alone`, and the second is not a success: it means
+ the exported theory is inconsistent on its own, which is the `ContradictoryAxioms` case item 5's
+ differential already had to separate by hand.
+
+### What remains an oracle, in full
+
+- **The calculus.** Even a derivation whose every step is recomputed is a derivation in a calculus
+ whose soundness is written down nowhere in `lean/`. `Fol.satisfiable_of_check` has a theorem
+ behind it; this has a Rust program behind it, and the difference is the same one decision 0006
+ item 1 draws between the two directions.
+- **The replayer itself.** `src/tstp.rs` is ordinary unverified Rust with ordinary tests. A defect
+ in the unifier makes a wrong step look right. The tests below are what stands between that and a
+ false green, and they are tests rather than a proof.
+- **Clausification, and everything downstream of it.** `cnf_transformation`,
+ `ennf_transformation`, `nnf_transformation`, `skolemize`, `distribute`,
+ `true_and_false_elimination`, `variable_rename`, `shift_quantors`, `split_conjunct`, every
+ AVATAR rule, and every SAT-solver step are NOT checked. They are named, counted and given a
+ reason in `steps_unchecked`, and on a real proof they are the majority.
+- **Introduced definitions.** AVATAR invents propositional symbols and prints them as
+ `introduced(definition, …)` leaves. They are not from the problem and are not derived from it.
+ Nothing here checks that the extension is conservative, so an introduced leaf is counted as
+ unchecked and keeps the strongest word out of reach.
+- **E's nested inference records.** E writes an inference record inline in a parent position, and
+ such a record carries a rule and parents but NO FORMULA. Neither it nor the step it feeds can be
+ replayed, and both are reported with that reason rather than with the generic one.
+- **The SZS status line.** Echoed into `szs_status`, labelled untrusted in the report's own JSON,
+ and used to decide nothing.
+
+### The verdict vocabulary, and the rule it turns on
+
+Eight words, and the ladder is the point:
+
+```
+problem_unparsed the TPTP problem could not be read
+derivation_unparsed the output is not a TSTP derivation. Usually: no proof option
+no_refutation_offered no empty clause. A satisfiable problem, a timeout, a give-up
+derivation_rejected THE CHECKER SAID NO. Dangling parent, cycle, or a leaf that is
+ not a formula of the problem
+refutation_step_not_reconstructed a step whose rule IS implemented did not reconstruct
+refutation_structure_checked structure holds, NO step replayed
+refutation_partially_replayed structure holds, SOME steps replayed. The normal outcome
+refutation_fully_replayed structure holds, EVERY step replayed. STILL NOT unsatisfiable
+```
+
+**An unchecked step prevents the strongest word, and so does an introduced leaf.** That rule is
+mechanical, it is the reason the ladder has three rungs instead of a boolean, and
+`tests/tstp_derivation_test.rs::one_unchecked_step_prevents_the_strongest_word` takes a fully
+replayed derivation, renames exactly one step's rule to `cnf_transformation`, changes nothing else,
+and requires the verdict to drop. If that test ever passes without the drop, the vocabulary has
+stopped meaning anything.
+
+### A failed reconstruction gets its own word, and that is not a hedge
+
+A step whose rule is implemented, whose premises and conclusion are clause-shaped, and which still
+does not reconstruct is EITHER a defect in the derivation OR a gap in this checker. Reporting it as
+"unchecked" would let a forged step hide behind a rule name. Reporting it as "rejected" would
+accuse someone else's prover on this module's word alone. So it is
+`refutation_step_not_reconstructed`, it names the step, it prints what was expected and what was
+found, and it exits non-zero. That is item 5's own discipline, one layer down: the tool reports the
+disagreement and does not adjudicate it.
+
+### Subsumption resolution is checked as binary resolution, and that is exact rather than lax
+
+Vampire prints `forward_subsumption_resolution` for a step whose premises are `C ∨ L` and
+`D ∨ ¬L'` and whose conclusion is `C`, under `D'σ ⊆ C`. The binary resolvent of those premises is
+`C ∪ D'σ`, and `D'σ ⊆ C` makes that exactly `C`. One check therefore covers both rules and neither
+is weakened by it. The same identity is why the check still passes on AVATAR's printed clauses,
+where the side premise's assertion literals are carried into the conclusion and the classical side
+condition does not hold of the printed form.
+
+### Measured, on the day it was written
+
+FOAF, `owl-rl-ext`, 181 claimed entailments, one problem per entailment, exported from an
+unreasoned store the way item 7 requires.
+
+| prover | goals refuted | steps replayed | leaves matched | rejected | not reconstructed |
+|---|---:|---:|---|---:|---:|
+| Vampire 5.1.0 | 181 | 1279 of 3314 | 534, all `identical` | 0 | 0 |
+| E 3.2.5 | 181 | 181 of 4644 | 534, all `alpha_equivalent` | 0 | 0 |
+
+Every one of the 362 runs came back `refutation_partially_replayed`. The two provers differ by an
+order of magnitude in how much is replayable and the reason is structural rather than a matter of
+quality: Vampire prints each inference as its own annotated formula with a formula attached, while
+E nests inference records inside parent positions and those carry no formula. The ontology-alone
+run, with no conjecture, came back `no_refutation_offered` from both, which is what a consistent
+ontology should produce and is the control that the checker is not simply saying yes.
+
+`alpha_equivalent` against `identical` is not cosmetic. E renames the bound variables of every
+axiom it reads, so a leaf check that demanded byte identity of the parsed AST would reject every E
+proof of every problem. The three levels — `identical`, `alpha_equivalent`,
+`associativity_normalised` — are reported separately so that how much normalisation a leaf needed
+is visible rather than absorbed.
+
+### What this does not change
+
+- **`AGREE` still means what it meant.** A checked derivation adds that the prover was answering
+ about the file we gave it. It does not make the answer a proof, and the differential's summary
+ says so in the same paragraph as the new counts.
+- **The model direction is still the only certified one.** Decision 0006 is unaffected in every
+ particular. `model_checked` rests on `Fol.satisfiable_of_check`; nothing in this addendum rests
+ on any theorem at all.
+- **Nothing here touches the translation correspondence.** Item 2 stands: `src/tptp.rs` is a
+ transcription of `OwlLean/Translation.lean` pinned by tests and not proved, and a checked
+ derivation over the emitted file says nothing about whether the file is the right file to have
+ emitted.
+
+### Still not done, in this layer
+
+- **Superposition and demodulation are not replayed.** They are the steps that do the work on any
+ problem with equality, and replaying one needs a term ordering, which is a project rather than a
+ function. Until then every equality-heavy proof will sit near the bottom of the ladder.
+- **Clausification is not checked and there is no cheap way to check it.** The honest alternative
+ is to ask the prover for the clause set and check that the clause set is equisatisfiable with the
+ problem, which needs the same machinery.
+- **Nothing runs this in CI.** It needs a prover on `PATH` and the differential skips loudly
+ without one, exactly as item "Still not done" above already records for the differential itself.
+ The recorded fixtures under `tests/fixtures/tstp/` run everywhere; the live run skips.
+- **The variant check is backtracking with a budget.** A clause larger than the budget is reported
+ as unchecked with that reason, which is honest and is also a place a forger could aim at. The
+ budget is 200,000 pairings and no real step has come near it.
diff --git a/docs/first-order-export.md b/docs/first-order-export.md
index e425edc4..57f4b226 100644
--- a/docs/first-order-export.md
+++ b/docs/first-order-export.md
@@ -18,6 +18,13 @@ way: it needs a verified first-order calculus with unification, which does not e
When E says `SZS status Theorem`, that carries exactly the weight of pyshacl agreeing with the
SHACL validator, and nothing here calls it a proof.
+**And read what that word now comes with.** A prover will also print the DERIVATION it found, and a
+derivation is a finite object. `fol-prove` reads one back, matches every leaf against the problem
+this engine emitted, checks the DAG, and recomputes the resolution-family steps. It does not turn
+the opinion into a proof and no word in its output says otherwise. It does answer the question the
+opinion never could: whether the prover was answering about our file. See
+[reading the derivation back](#reading-the-derivation-back).
+
**And read the asymmetry second.** A MODEL is the exact opposite of a refutation. It is a finite
object, checking a formula against it is decidable, and `lean/Fol/` holds a verified evaluator for
that, so the SATISFIABILITY direction CAN be certified. `fol --format smtlib` and
@@ -150,6 +157,87 @@ in general, and a conclusion that does not follow often has only infinite counte
prover that cannot refute it has told you nothing either way. The tool never reads that as
agreement.
+## Reading the derivation back
+
+```bash
+# Prove the loaded ontology's claimed entailments and CHECK every derivation.
+printf 'load ontology.ttl\nfol-prove --out /tmp/proofs --prover vampire --goals /tmp/cert/derivations.tsv --goals-skip-columns 1\n' \
+ | open-ontologies --no-connect --data-dir /tmp/store batch -
+
+# Or check a problem and a recorded prover output, with no store and no run.
+vampire --proof tptp problem.p > proof.tstp
+open-ontologies --no-connect fol-prove --problem problem.p --proof proof.tstp
+```
+
+Over MCP the tool is `onto_fol_prove`, with the same two modes.
+
+Three things are established, in increasing order of what they cost.
+
+**The prover refuted OUR problem.** Every leaf of the derivation is matched against the problem file
+three ways: by the name its own `file('…', NAME)` annotation gives, by the PARSED formula, and by
+the ROLE. Each catches something different. The name catches a prover pointed at a stale or
+different file. The formula catches a file edited since under the same names. The role catches a
+conjecture presented as an axiom, which would make the refutation say nothing about entailment while
+looking perfect. Until this existed, an `AGREE` row above simply assumed all three.
+
+**The derivation is a well-founded DAG ending in `$false`.** Every parent reference resolves, no
+name is used twice, the parent relation is acyclic, and the node nothing else cites is the empty
+clause. Nodes the empty clause does not depend on are counted separately and left out of every
+tally.
+
+**Some steps are recomputed.** `resolution`, `subsumption_resolution` and its forward and backward
+spellings, `factoring`, `duplicate_literal_removal`, `flattening`, `trivial_inequality_removal`,
+`equality_resolution`, and the negation of the conjecture. Each is replayed from its premises with a
+syntactic unifier with an occurs check, and the conclusion must agree up to a bijective renaming of
+variables. **Everything else is named and counted as unchecked**, with a reason: clausification,
+Skolemisation, AVATAR splitting, every SAT-solver step, and every step of E's whose premise is an
+inline inference record and therefore carries no formula.
+
+### The verdict is one of eight words
+
+| verdict | means |
+|---|---|
+| `problem_unparsed` | the TPTP problem could not be read. Not a statement about the prover |
+| `derivation_unparsed` | the output is not a TSTP derivation. Usually the proof option was missing |
+| `no_refutation_offered` | no empty clause. A satisfiable problem, a timeout or a give-up. Evidence of nothing |
+| `derivation_rejected` | THE CHECKER SAID NO: a dangling parent, a cycle, or a leaf that is not a formula of the problem. Exits 1 |
+| `refutation_step_not_reconstructed` | a step whose rule IS implemented did not reconstruct. EITHER the derivation is wrong OR this checker is incomplete, and it decides neither. Exits 1 |
+| `refutation_structure_checked` | structure holds, no step replayed |
+| `refutation_partially_replayed` | structure holds, some steps replayed. The normal outcome |
+| `refutation_fully_replayed` | structure holds, EVERY step replayed. Still not unsatisfiability |
+
+**An unchecked step prevents the strongest word**, and so does a leaf the prover invented. That is
+mechanical and it is the reason the ladder has three rungs rather than a boolean. Even
+`refutation_fully_replayed` is not a proof: the calculus's soundness is machine-checked nowhere in
+`lean/`, and the replayer is ordinary Rust.
+
+### Measured, over FOAF
+
+One problem per claimed entailment, 181 of them, exported from an unreasoned store.
+
+| prover | refuted | steps replayed | leaves matched | rejected | not reconstructed |
+|---|---:|---:|---|---:|---:|
+| Vampire 5.1.0 | 181 | 1279 of 3314 | 534, all `identical` | 0 | 0 |
+| E 3.2.5 | 181 | 181 of 4644 | 534, all `alpha_equivalent` | 0 | 0 |
+
+The order-of-magnitude gap is structural and worth knowing before choosing a prover for a pipeline
+that wants to inspect its own evidence. Vampire prints each inference as its own annotated formula
+with the conclusion attached; E nests inference records inside parent positions, and a nested record
+carries a rule and parents but no formula, so neither it nor the step it feeds can be replayed.
+
+`alpha_equivalent` against `identical` is not cosmetic either. E renames the bound variables of
+every axiom it reads, so a leaf check demanding byte identity of the parsed AST would reject every E
+proof of every problem. The three match levels — `identical`, `alpha_equivalent`,
+`associativity_normalised` — are reported separately so that how much normalisation a leaf needed is
+visible rather than absorbed.
+
+### What the differential does with it
+
+`tools/fol_differential.py` now runs each prover with its proof option and hands every derivation to
+the checker. Each row carries `proof_check` and `proof_steps` beside its `AGREE`, and a
+`PROOF_REJECTED` row exits 1. `--no-proof-check` turns it off, and a run with that flag reports what
+the prover SAID and nothing about what it said it about.
+
## What is proved, and where the proof stops
`OwlLean.adequacy`, in the sibling `owl-lean` project:
@@ -589,8 +677,14 @@ Stated rather than discovered later.
must come from the same graph the export was built from, which is why the differential loads the
certificate's own `asserted.tsv` rather than the source file twice. Otherwise the goal is refused
with that reason.
-- **Nothing runs the differential in CI.** It needs a prover on `PATH` and skips loudly without
- one. A CI leg that installs E is not wired.
+- **Nothing runs the differential in CI, and nothing runs the derivation check there either.** Both
+ need a prover on `PATH` and both skip loudly without one. The recorded Vampire and E derivations
+ under `tests/fixtures/tstp/` DO run everywhere, over a problem the test regenerates from the
+ exporter so the fixture cannot drift; the live run skips. A CI leg that installs a prover is not
+ wired.
+- **Superposition and demodulation are not replayed.** They are the steps that do the work on any
+ problem with equality, and replaying one needs a term ordering. Until then an equality-heavy proof
+ sits near the bottom of the ladder.
- **CGIF conformance is pinned by our own checker and not by an independent parser.** There is no
installable CGIF parser to differ from: no `cgif` package on PyPI, no CGIF front end in
py-typedlogic, and Macleod reads CLIF. The CLIF claims on this page rest on two external parsers
@@ -613,11 +707,15 @@ Stated rather than discovered later.
on this page sourced to them is still scoped to the text actually read, including that clause
4.3's "shall be available" modality comes from the Russian identical adoption and not from the
English.
-- **Only E has actually been run.** Every number on this page comes from E 3.2.5. The Vampire
- branch of the prover detection has never executed, because Vampire is not installed here; its
- argument vector and its SZS parsing are written from the documented interface and are untested.
- Treat `--atp vampire` as unverified until someone runs it.
+- **Vampire has now been run, and this bullet used to say it had not.** Until 15 September 2026
+ every number on this page came from E 3.2.5 and the Vampire branch of the prover detection had
+ never executed. Vampire 5.1.0 is installed on this machine, `--atp vampire` has been run over
+ FOAF, and the measured table above is from both. What was found in the process: `--mode casc`
+ alone prints a proof in a display format, and `--proof tptp` is what makes it a TSTP derivation
+ the checker can read. The argument vector now carries it, and so does E's `--proof-object`.
- **`AGREE` is evidence, not proof.** It means a second implementation reached the same conclusion.
+ A checked derivation beside it adds that the second implementation was reading our file. It does
+ not add a proof and the summary says so in the same paragraph as the counts.
## What it caught
diff --git a/docs/reasoning-systems-inventory.md b/docs/reasoning-systems-inventory.md
index b424346d..c743f11c 100644
--- a/docs/reasoning-systems-inventory.md
+++ b/docs/reasoning-systems-inventory.md
@@ -10,13 +10,21 @@ a calculus nobody has mechanised in core Lean here, so an unsatisfiability answe
single asymmetry decides how each system below is used, and it explains rankings that otherwise look
perverse.
+Since 15 September 2026 the testimony comes with an exhibit. Both provers will print the derivation
+they found, and `src/tstp.rs` reads it back: every leaf is matched against the problem this engine
+emitted, the DAG is checked for dangling parents and cycles, and the resolution-family steps are
+recomputed. That does not upgrade the testimony into a certificate and no word in the output says it
+does, because the calculus is not mechanised and the replayer is unverified Rust. What it removes is
+the part of the trust that was never about the prover's soundness at all: whether it was answering
+about the file we gave it. The addendum to decision 0005 draws the new line.
+
## Status at a glance
| System | Kind | Status here |
| --- | --- | --- |
| Lean 4 | Proof assistant | The kernel. Every checker in this repository. |
-| E 3.2.5 | First-order prover | Used as a differential oracle. Never as an authority. |
-| Vampire 5.1.0 | First-order prover | Installed, same role as E. |
+| E 3.2.5 | First-order prover | Differential oracle. Never an authority. Its derivations are now read back and structurally checked. |
+| Vampire 5.1.0 | First-order prover | Same role as E, and the one whose derivations replay furthest. |
| Z3 4.16.0 | SMT solver | Model-certificate layer, under construction. |
| Mace4 | Finite model finder | Same. Its output is checkable; Prover9's is not. |
| Prover9 | First-order prover | Declined. Unmaintained since 2011, refutations uncheckable. |
@@ -29,7 +37,7 @@ perverse.
| Aeneas with Charon | Rust to Lean | Declined. Subset does not contain this codebase. |
| Verus, Creusot, Prusti | Rust verification | Declined. Each needs the code rewritten in its subset. |
| Kani | Rust bounded model checker | Being applied to the trusted boundary only. |
-| TPTP and TSTP | Interchange | Implemented. The execution format. |
+| TPTP and TSTP | Interchange | Implemented, in both directions. TPTP out, TSTP back in and checked. |
| CLIF, ISO/IEC 24707 | Interchange | Implemented. The conformance format. |
| SMT-LIB 2 | Interchange | Under construction. |
| RIF Core, SWRL | Rule languages | Front ends under construction. |
@@ -54,6 +62,22 @@ against one public vocabulary the comparison disagreed with the engine on fifty-
eighty-one claimed entailments, and every one was a real defect, including one in the exporter itself.
That is the value, and it does not require trusting the prover at all.
+What both now also give us is a derivation to read. `fol-prove` and `onto_fol_prove` run the prover
+with its proof-printing option, parse the TSTP it emits, and report a verdict on the derivation
+separately from the verdict the prover reported on the problem. Measured over FOAF, one problem per
+claimed entailment: Vampire refuted all one hundred and eighty-one and one thousand two hundred and
+seventy-nine of its three thousand three hundred and fourteen steps were recomputed here, while E
+refuted all one hundred and eighty-one and only one hundred and eighty-one of its four thousand six
+hundred and forty-four steps were. Every one of the five hundred and thirty-four leaves matched the
+problem on both sides, and nothing was rejected.
+
+The gap between the two is structural rather than a matter of quality, and it is the most useful
+thing this measurement produced. Vampire prints each inference as its own annotated formula with the
+conclusion attached, so a step can be recomputed from its premises. E nests inference records inside
+parent positions, and a nested record carries a rule and parents but no formula, so neither it nor
+the step it feeds can be replayed at all. Anyone choosing a prover for a pipeline that wants to
+inspect its own evidence should know that before choosing.
+
Prover9 is declined. Its author died in 2011, it has had no maintainer since, and its refutations are
no more checkable than Vampire's. There is no version of the argument where an unmaintained prover
beats a maintained one at the same job.
@@ -71,7 +95,9 @@ which is the only place in this architecture where an external tool's answer is
merely corroborated.
Its unsat answers stay oracle answers. An unsat core is not a proof object we can replay, and the
-proof logs Z3 can emit would need the same mechanised calculus that the first-order case lacks.
+proof logs Z3 can emit would need the same mechanised calculus that the first-order case lacks. The
+structural treatment the first-order provers now get has no counterpart here yet: nothing reads Z3's
+proof logs, and doing so is a separate piece of work from reading TSTP.
## Isabelle, and what the second kernel found
@@ -269,7 +295,10 @@ Building the notation before the table it indexes would be a front end onto two
## Interchange formats
-TPTP is the execution format, because it is the only one an installed solver will read. CLIF is the
+TPTP is the execution format, because it is the only one an installed solver will read, and TSTP is
+now read in the other direction: `src/tstp.rs` parses a prover's annotated-formula list, its
+`inference(RULE, [status], [parents])` records and its `file(…)` and `introduced(…)` provenance, and
+turns them into a DAG this engine can check against its own output. CLIF is the
conformance format, because ISO/IEC 21838-1 requires a top-level ontology to carry an axiomatisation in
a language conforming to ISO/IEC 24707 and the Basic Formal Ontology discharges that in CLIF. SMT-LIB
is the model-finding format. All three are printers over one representation, and the correspondence
diff --git a/docs/trusted-computing-base.md b/docs/trusted-computing-base.md
index 8e8a0de3..4b858335 100644
--- a/docs/trusted-computing-base.md
+++ b/docs/trusted-computing-base.md
@@ -614,6 +614,31 @@ no verdict word, and no executable imports it, which is why it is a separate `le
count of trusted properties is what it was.
The honest summary is that the trusted base used to be four things: the serialisation of a term, the
identity of the asserted graph, the completeness of the derivation record, and the identity of the
+rule table. Those are now property-tested and, for the pure parts, bounded-model-checked. Every one
+of them still rests on a dependency's behaviour that this repository observes rather than enforces.
+The engine is not verified. It was never going to be, and a report that read as though it were
+would be the same defect this project exists to attack.
+
+## What is deliberately not on this page: `src/tstp.rs`
+
+Added 15 September 2026, and it belongs here only to be excluded, because its name will otherwise
+suggest to a reader that it has a proof behind it.
+
+`src/tstp.rs` reads the TSTP derivation a first-order prover prints, matches every leaf against the
+problem this engine emitted, checks the DAG, and recomputes the resolution-family steps. It is the
+refutation direction's answer to `oo-folmodel`, and the analogy stops at the shape. `oo-folmodel` is
+a Lean program whose acceptance implies `Fol.Satisfiable` by a machine-checked theorem, so the
+question "what does its verdict assume about the Rust" is a real question with the answers above.
+`src/tstp.rs` is a Rust program whose acceptance implies nothing formally at all: the calculus it
+replays is mechanised nowhere in `lean/`, and the replayer is unverified.
+
+So it is not the trusted computing base OF anything. It is a second opinion about a third party's
+output, in the same category as the Rust SHACL validator in item 9 and in the opposite direction:
+there, a verified evaluator measures an unverified one; here, an unverified checker measures an
+unverified prover, and the result is evidence that a human can read rather than any kind of
+certificate. Its verdict vocabulary is built so that this cannot be mistaken, and the addendum to
+[decision 0005](decisions/0005-a-prover-is-an-oracle-and-a-translation-is-a-theorem.md) sets out
+what it does and does not establish, field by field.
rule table. The serialisation of a term is no longer one of them: it is enforced here and proved
over every byte pattern at the bounds below, rather than observed of a dependency. The identity of
the rule table is most of the way off the list: the engine re-parses what it wrote and refuses on
diff --git a/src/batch.rs b/src/batch.rs
index f91c5879..06eb45d4 100644
--- a/src/batch.rs
+++ b/src/batch.rs
@@ -130,6 +130,7 @@ impl BatchRunner {
"fol" => self.exec_fol(&cmd.args),
"rules-import" | "rules_import" => self.exec_rules_import(&cmd.args),
"fol-model" | "fol_model" => self.exec_fol_model(&cmd.args),
+ "fol-prove" | "fol_prove" => self.exec_fol_prove(&cmd.args),
"preserve" => self.exec_preserve(&cmd.args),
"closure-diff" | "closure_diff" => self.exec_closure_diff(&cmd.args),
"shacl" => self.exec_shacl(&cmd.args),
@@ -366,6 +367,54 @@ impl BatchRunner {
serde_json::from_str(&result).unwrap_or(json!({"raw": result}))
}
+ /// Run a prover and re-check the derivation it prints. In-process for the
+ /// same reason `fol` is: the store is per process, so loading and proving
+ /// have to happen in one run.
+ ///
+ /// The check-only pair (`--problem` with `--proof`) is here too, because a
+ /// pipeline that already holds both files should not have to start a
+ /// second process to get a verdict on them.
+ fn exec_fol_prove(&self, args: &[String]) -> Value {
+ use crate::tstp::{ProveOptions, Prover, check_files, prove_export};
+ if let (Some(problem), Some(proof)) =
+ (Self::flag_value(args, "--problem"), Self::flag_value(args, "--proof"))
+ {
+ let result =
+ check_files(std::path::Path::new(&problem), std::path::Path::new(&proof))
+ .unwrap_or_else(|e| json!({"error": e.to_string()}).to_string());
+ return serde_json::from_str(&result).unwrap_or(json!({"raw": result}));
+ }
+ let Some(out) = Self::flag_value(args, "--out") else {
+ return json!({"error":
+ "fol-prove requires --out DIR, or --problem FILE with --proof FILE"});
+ };
+ let prover = match Prover::parse(
+ &Self::flag_value(args, "--prover").unwrap_or("vampire".to_string()),
+ ) {
+ Ok(p) => p,
+ Err(e) => return json!({"error": e.to_string()}),
+ };
+ let opts = ProveOptions {
+ prover,
+ timeout_secs: Self::flag_value(args, "--timeout-secs")
+ .and_then(|v| v.parse().ok())
+ .unwrap_or(30),
+ };
+ let goals = Self::flag_value(args, "--goals");
+ let skip: usize = Self::flag_value(args, "--goals-skip-columns")
+ .and_then(|v| v.parse().ok())
+ .unwrap_or(0);
+ let result = prove_export(
+ &self.graph,
+ std::path::Path::new(&out),
+ &opts,
+ goals.as_deref().map(std::path::Path::new),
+ skip,
+ )
+ .unwrap_or_else(|e| json!({"error": e.to_string()}).to_string());
+ serde_json::from_str(&result).unwrap_or(json!({"raw": result}))
+ }
+
/// The certifying pipeline. In-process for the same reason `fol` is: the
/// store is per process, so loading and solving have to happen in one run.
fn exec_fol_model(&self, args: &[String]) -> Value {
diff --git a/src/inputs.rs b/src/inputs.rs
index 30ede1e8..394d48b4 100644
--- a/src/inputs.rs
+++ b/src/inputs.rs
@@ -447,6 +447,30 @@ pub struct OntoFolModelInput {
pub goals_skip_columns: Option<usize>,
}
+#[derive(Deserialize, JsonSchema)]
+pub struct OntoFolProveInput {
+ /// Working directory. The problem, the prover's raw output and the report
+ /// land here, so a run is reproducible by hand from what it leaves behind.
+ /// Not needed when `problem` and `proof` are both given.
+ pub out_dir: Option<String>,
+ /// `vampire` (default) or `eprover`. Both are run WITH their
+ /// proof-printing option; a run without one returns a word and no
+ /// derivation, which is the state this tool exists to leave behind.
+ pub prover: Option<String>,
+ /// Seconds per prover invocation. Default 30.
+ pub timeout_secs: Option<u32>,
+ /// A TSV of triples to ask as conjectures, one run per line; the shape
+ /// `onto_reason` writes with `certificate_dir`.
+ pub goals_file: Option<String>,
+ /// Leading tab-separated columns to skip before the subject. Default 0.
+ pub goals_skip_columns: Option<usize>,
+ /// CHECK-ONLY: a TPTP problem file. With `proof`, no prover is run and no
+ /// store is read; the recorded pair is checked as it stands.
+ pub problem: Option<String>,
+ /// CHECK-ONLY: a file holding a prover's output for `problem`.
+ pub proof: Option<String>,
+}
+
#[derive(Deserialize, JsonSchema)]
pub struct OntoDlExplainInput {
/// IRI of the class to explain unsatisfiability for
diff --git a/src/lib.rs b/src/lib.rs
index 3d9cc358..8f7d36cd 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -115,6 +115,9 @@ pub mod toolfilter;
pub mod fol_model;
pub mod fol_solve;
pub mod tptp;
+/// Reading a prover's TSTP derivation and re-checking what can be re-checked.
+/// Decision 0005's addendum says exactly what this earns and what it does not.
+pub mod tstp;
#[cfg(feature = "turbovec")]
pub mod turbo_index;
#[cfg(feature = "embeddings")]
diff --git a/src/main.rs b/src/main.rs
index 261b4aa1..e7855bd1 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -589,6 +589,44 @@ enum Commands {
checker: Option<String>,
},
+ /// Run a prover, READ the derivation it prints, and re-check what can be re-checked
+ ///
+ /// A refutation is still an ORACLE OPINION (decision 0005) and nothing
+ /// here certifies one. What this adds is that the opinion stops being a
+ /// single word: the derivation is parsed, every leaf is matched against
+ /// the problem THIS engine emitted, the DAG is checked for dangling
+ /// parents and cycles, and the resolution-family steps are recomputed from
+ /// their premises. Every rule that was not replayed is named and counted.
+ /// Exits 1 if any derivation is rejected or any implemented step fails to
+ /// reconstruct.
+ FolProve {
+ /// Working directory. The problem, the prover's raw output and the
+ /// report land here, so a run is reproducible by hand.
+ #[arg(long, required_unless_present = "problem")]
+ out: Option<String>,
+ /// `vampire` (default) or `eprover`. Both are run WITH their
+ /// proof-printing option, because a run without one returns a word
+ /// and no derivation.
+ #[arg(long, default_value = "vampire")]
+ prover: String,
+ #[arg(long, default_value_t = 30)]
+ timeout_secs: u32,
+ /// A TSV of triples to ask as conjectures, one run per line; the shape
+ /// `fol --goals` takes.
+ #[arg(long)]
+ goals: Option<String>,
+ #[arg(long, default_value_t = 0)]
+ goals_skip_columns: usize,
+ /// CHECK-ONLY: a TPTP problem file. With --proof, no prover is run and
+ /// no store is read; the recorded pair is checked as it stands. This
+ /// is what tools/fol_differential.py calls.
+ #[arg(long, requires = "proof")]
+ problem: Option<String>,
+ /// CHECK-ONLY: a file holding a prover's output for --problem.
+ #[arg(long, requires = "problem")]
+ proof: Option<String>,
+ },
+
/// Ask whether a retrieved slice still supports the claims an answer rests on
///
/// Coverage is a proxy. This asks the property: for each goal, does the
@@ -863,7 +901,7 @@ impl Commands {
}
cmd("reason", a)
}
- Commands::Fol { out, format, smt_domain, clif_dialect, clif_comments, goals, goals_skip_columns } => {
+ Commands::Fol { out, format, smt_domain, clif_dialect, clif_comments, goals, goals_skip_columns } => {
let mut a = vec![
"--out".into(),
absolutize(out),
@@ -941,6 +979,31 @@ impl Commands {
cmd("fol-model", a)
}
+ Commands::FolProve { out, prover, timeout_secs, goals, goals_skip_columns, problem, proof } => {
+ // The check-only pair reads two files and no store, so there
+ // is nothing for the daemon to hold and proxying it would only
+ // add a hop.
+ if problem.is_some() || proof.is_some() {
+ return None;
+ }
+ let Some(out) = out else { return None };
+ let mut a = vec![
+ "--out".into(),
+ absolutize(out),
+ "--prover".into(),
+ prover.clone(),
+ "--timeout-secs".into(),
+ timeout_secs.to_string(),
+ ];
+ if let Some(g) = goals {
+ a.push("--goals".into());
+ a.push(absolutize(g));
+ a.push("--goals-skip-columns".into());
+ a.push(goals_skip_columns.to_string());
+ }
+ cmd("fol-prove", a)
+ }
+
Commands::Preserve {
projection,
projection_graph,
@@ -2795,6 +2858,63 @@ async fn async_main() -> anyhow::Result<()> {
std::process::exit(1);
}
}
+ Commands::FolProve {
+ out,
+ prover,
+ timeout_secs,
+ goals,
+ goals_skip_columns,
+ problem,
+ proof,
+ } => {
+ use open_ontologies::tstp::{ProveOptions, Prover, check_files, prove_export};
+ // The check-only pair never touches the store: it is two files and
+ // a checker, which is exactly what a differential already holding
+ // both needs.
+ if let (Some(pp), Some(dd)) = (problem.as_deref(), proof.as_deref()) {
+ let result =
+ check_files(std::path::Path::new(pp), std::path::Path::new(dd))
+ .unwrap_or_else(|e| serde_json::json!({"error": e.to_string()}).to_string());
+ output_result_checked(&result, cli.pretty);
+ let stop = serde_json::from_str::<serde_json::Value>(&result)
+ .ok()
+ .and_then(|v| v.get("stop_the_line").and_then(|n| n.as_u64()))
+ .unwrap_or(0);
+ if stop > 0 {
+ std::process::exit(1);
+ }
+ return Ok(());
+ }
+ let Some(out) = out else {
+ eprintln!("fol-prove needs --out DIR, or --problem FILE with --proof FILE");
+ std::process::exit(2);
+ };
+ let (_db, graph) = setup(&cli.data_dir)?;
+ let result = match Prover::parse(&prover) {
+ Ok(p) => {
+ let opts = ProveOptions { prover: p, timeout_secs };
+ prove_export(
+ &graph,
+ std::path::Path::new(&out),
+ &opts,
+ goals.as_deref().map(std::path::Path::new),
+ goals_skip_columns,
+ )
+ .unwrap_or_else(|e| serde_json::json!({"error": e.to_string()}).to_string())
+ }
+ Err(e) => serde_json::json!({"error": e.to_string()}).to_string(),
+ };
+ output_result_checked(&result, cli.pretty);
+ // A rejected derivation means the prover refuted something other
+ // than the problem this engine emitted. That must fail a pipeline.
+ let stop = serde_json::from_str::<serde_json::Value>(&result)
+ .ok()
+ .and_then(|v| v.get("stop_the_line").and_then(|n| n.as_u64()))
+ .unwrap_or(0);
+ if stop > 0 {
+ std::process::exit(1);
+ }
+ }
Commands::Fol { out, format, smt_domain, clif_dialect, clif_comments, goals, goals_skip_columns } => {
let (_db, graph) = setup(&cli.data_dir)?;
let result = match open_ontologies::tptp::Syntax::parse(
@@ -3337,6 +3457,24 @@ mod proxy_serialization_tests {
assert_eq!(args, vec!["http://example.org/sparql", "--graph", "g1"]);
}
+ #[test]
+ fn the_check_only_form_of_fol_prove_is_not_proxied() {
+ // `--problem` with `--proof` reads two files and no store, so there is
+ // nothing for the daemon to hold. Proxying it would add a hop and, if
+ // the paths were relative, resolve them against the daemon's directory
+ // instead of the caller's.
+ let check_only = Commands::FolProve {
+ out: None,
+ prover: "vampire".into(),
+ timeout_secs: 30,
+ goals: None,
+ goals_skip_columns: 0,
+ problem: Some("p.p".into()),
+ proof: Some("d.tstp".into()),
+ };
+ assert!(check_only.to_batch_command().is_none());
+ }
+
#[test]
fn a_flag_the_batch_handler_cannot_honour_keeps_the_command_local() {
// The batch ingester has no `--format`, so proxying would have dropped it
@@ -3372,6 +3510,7 @@ mod proxy_serialization_tests {
Commands::Reason { profile: "rdfs".into(), certificate: None, rules: None },
Commands::Fol { out: "/tmp/fol".into(), format: "tptp".into(), smt_domain: None, clif_dialect: "iso".into(), clif_comments: "standalone".into(), goals: None, goals_skip_columns: 0 },
Commands::FolModel { out: "/tmp/folmodel".into(), solver: "z3".into(), max_domain: 16, timeout_secs: 30, unbounded_probe: true, goals: None, goals_skip_columns: 0, checker: None },
+ Commands::FolProve { out: Some("/tmp/folprove".into()), prover: "vampire".into(), timeout_secs: 30, goals: None, goals_skip_columns: 0, problem: None, proof: None },
Commands::Shacl { shapes: "s.ttl".into() },
Commands::Status,
Commands::Pull { url: "http://example.org".into(), sparql: false, query: None },
diff --git a/src/server.rs b/src/server.rs
index 885b8367..6df9b60d 100644
--- a/src/server.rs
+++ b/src/server.rs
@@ -2222,6 +2222,35 @@ impl OpenOntologiesServer {
.unwrap_or_else(Self::err_json)
}
+ #[tool(name = "onto_fol_prove", description = "Run a first-order prover, READ the derivation it prints back, and RE-CHECK what can honestly be re-checked. This does NOT move the boundary decision 0005 draws: a refutation is an ORACLE OPINION and stays one, because checking a superposition proof needs a verified first-order calculus with unification that does not exist in core Lean, and the replayer here is ordinary UNVERIFIED Rust with no theorem behind it. What it adds is that the opinion stops being a single word. THREE THINGS ARE EARNED, in increasing cost: (1) THE PROVER REFUTED OUR PROBLEM: every leaf of the derivation is matched, by name AND by parsed formula AND by role, against the problem onto_fol_export emitted, so a prover pointed at a stale file, a different file, or one whose conjecture was smuggled in as an axiom is caught; (2) THE DERIVATION IS A WELL-FOUNDED DAG ENDING IN $false: every parent reference resolves, the parent relation is acyclic, and the node nothing cites is the empty clause; (3) SOME STEPS ARE REPLAYED: binary resolution (which is also exactly what a subsumption-resolution conclusion is), factoring, duplicate literal removal, trivial inequality removal, equality resolution, associative flattening and the negation of the conjecture are recomputed from their premises with a small unifier. EVERYTHING ELSE IS NAMED AND COUNTED AS UNCHECKED: clausification, Skolemisation, AVATAR splitting and every SAT-solver step are not checked and are never claimed to be. FIELDS THAT ARE NEVER COLLAPSED: `szs_status` (what the prover said about itself, echoed and untrusted), `derivation_wellformed`, `leaves_match_problem` with `leaf_match_levels` saying how much normalisation each leaf needed, `steps_checked` / `steps_unchecked` (by rule, with a count and a reason) / `steps_not_reconstructed`, `conjecture_used`, `what_was_refuted` (`axioms_and_the_negated_conjecture` or `axioms_alone`, the second meaning the axioms are inconsistent on their own), and `verdict`. THE VERDICT IS ONE OF EIGHT WORDS: `derivation_rejected` is THE CHECKER SAYING NO (a dangling parent, a cycle, or a leaf that is not a formula of the problem); `refutation_step_not_reconstructed` means a step whose rule IS implemented did not reconstruct, which is EITHER a defect in the derivation OR a gap in this checker and this tool decides neither; `refutation_structure_checked`, `refutation_partially_replayed` and `refutation_fully_replayed` are the replay ladder, AN UNCHECKED STEP PREVENTS THE STRONGEST WORD, and even the strongest is NOT `unsatisfiable` and NOT a Lean-checked anything; `no_refutation_offered`, `derivation_unparsed` and `problem_unparsed` are the non-answers. Needs vampire or eprover on PATH; its absence is reported loudly in `skipped` and never worked around. Pass `problem` and `proof` to check a recorded pair with no prover run and no store. The MODEL direction is the one that CAN be certified: see onto_fol_model and decision 0006.")]
+ async fn onto_fol_prove(&self, Parameters(input): Parameters<OntoFolProveInput>) -> String {
+ use crate::tstp::{ProveOptions, Prover, check_files, prove_export};
+ if let (Some(p), Some(d)) = (input.problem.as_deref(), input.proof.as_deref()) {
+ return check_files(std::path::Path::new(p), std::path::Path::new(d))
+ .unwrap_or_else(Self::err_json);
+ }
+ let Some(out_dir) = input.out_dir.as_deref() else {
+ return serde_json::json!({
+ "error": "onto_fol_prove needs out_dir, or `problem` with `proof` to check a \
+ recorded pair"
+ })
+ .to_string();
+ };
+ let prover = match Prover::parse(input.prover.as_deref().unwrap_or("vampire")) {
+ Ok(p) => p,
+ Err(e) => return Self::err_json(e),
+ };
+ let opts = ProveOptions { prover, timeout_secs: input.timeout_secs.unwrap_or(30) };
+ prove_export(
+ &self.graph,
+ std::path::Path::new(out_dir),
+ &opts,
+ input.goals_file.as_deref().map(std::path::Path::new),
+ input.goals_skip_columns.unwrap_or(0),
+ )
+ .unwrap_or_else(Self::err_json)
+ }
+
#[tool(name = "onto_dl_explain", description = "Explain why a class is unsatisfiable using DL tableaux reasoning. Returns an explanation trace showing the logical contradictions that make the class impossible to instantiate.")]
async fn onto_dl_explain(&self, Parameters(input): Parameters<OntoDlExplainInput>) -> String {
use crate::tableaux::DlReasoner;
diff --git a/src/tstp.rs b/src/tstp.rs
new file mode 100644
index 00000000..eaba3403
--- /dev/null
+++ b/src/tstp.rs
@@ -0,0 +1,2522 @@
+//! Reading the derivation a prover emits, and re-checking what can honestly be
+//! re-checked.
+//!
+//! Decision 0005 item 5 says an ATP verdict is an ORACLE OPINION, and decision
+//! 0006 says a refutation is not a certificate. Neither sentence is retracted
+//! here, and this module does not move the boundary: what it does is stop the
+//! oracle's answer being a single word. A prover that says `SZS status Theorem`
+//! also prints, on request, the derivation it found. That derivation is a
+//! finite object. Reading it back turns one unexaminable word into evidence
+//! that can be inspected, and some of it into evidence that can be RECOMPUTED.
+//!
+//! # The three things this earns, in increasing order of what they cost
+//!
+//! 1. **The prover refuted OUR problem.** Every leaf of the derivation is
+//! matched, by name and by parsed formula, against the problem file this
+//! repository's exporter produced. A prover pointed at the wrong file, a
+//! stale file, or a file whose conjecture was smuggled in as an axiom is
+//! caught here and nowhere else in this codebase. This is the cheapest
+//! check and by some distance the most valuable, because it is the one that
+//! an `AGREE` row in `tools/fol_differential.py` silently assumed.
+//! 2. **The derivation is a well-founded DAG ending in `$false`.** Every
+//! parent reference resolves to a node that exists, the parent relation is
+//! acyclic, and the node nothing else cites is the empty clause.
+//! 3. **Some steps are replayed.** Binary resolution, factoring, duplicate
+//! literal removal, trivial inequality removal, equality resolution,
+//! associative flattening and the negation of the conjecture are recomputed
+//! from the premises with a small unifier and compared with what the prover
+//! printed. Everything else is NAMED AND COUNTED as unchecked.
+//!
+//! # What this is NOT
+//!
+//! It is not a proof of unsatisfiability, and no word in the report says it
+//! is. Three separate reasons, all of them live:
+//!
+//! - **The calculus is not verified.** Even a derivation whose every step this
+//! module recomputes is only a derivation in a calculus whose soundness is
+//! written down nowhere in `lean/`. `Fol.satisfiable_of_check` has a theorem
+//! behind it; this has a Rust program behind it.
+//! - **The replayer is not verified either.** The unifier below is ordinary
+//! Rust with ordinary tests. A bug in it makes a wrong step look right.
+//! - **Clausification is not checked at all.** `cnf_transformation`,
+//! `ennf_transformation`, `nnf_transformation`, skolemisation, AVATAR
+//! splitting and every SAT-solver step are unchecked and are reported as
+//! unchecked, by rule, with a count. On a real proof they are the majority
+//! of the steps, which is why [`Report::verdict`] reaches
+//! `refutation_partially_replayed` and stops there.
+//!
+//! The strongest word this module can print is `refutation_fully_replayed`,
+//! it requires that NOTHING was left unchecked, and it still is not
+//! `unsatisfiable`. The ladder exists so that the distance between what was
+//! read and what was recomputed is visible in the output rather than in a
+//! footnote.
+//!
+//! # Why a failed reconstruction is its own word
+//!
+//! A step whose rule this module implements, whose premises and conclusion are
+//! clause-shaped, and which still does not reconstruct is EITHER a defect in
+//! the derivation OR a gap in this checker. It is reported as
+//! `refutation_step_not_reconstructed` and never as either of those two, for
+//! the same reason `tools/fol_differential.py` reports a disagreement and does
+//! not adjudicate it. Collapsing it into "unchecked" would let a forged step
+//! hide behind a rule name, and collapsing it into "rejected" would claim a
+//! defect in someone else's prover on this module's word alone.
+//!
+//! # Subsumption resolution is checked as binary resolution, and that is exact
+//!
+//! Vampire prints `forward_subsumption_resolution` for a step whose premises
+//! are `C ∨ L` and `D ∨ ¬L'` and whose conclusion is `C`, where `D'σ ⊆ C`. The
+//! binary resolvent of those two premises is `C ∪ D'σ`, and `D'σ ⊆ C` makes
+//! that exactly `C`. So the conclusion of a subsumption resolution IS the
+//! binary resolvent of its premises, and one check covers both rules without
+//! weakening either. The same identity is why the check still passes on
+//! AVATAR's printed clauses, where the side premise's assertion literals are
+//! carried into the conclusion and the classical side condition does not hold
+//! of the printed form.
+
+use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
+use std::fmt::Write as _;
+use std::path::{Path, PathBuf};
+use std::process::{Command, Stdio};
+use std::time::{Duration, Instant};
+
+// ── The term and formula language ──────────────────────────────────────────
+
+/// A first-order term as a prover prints it.
+///
+/// Deliberately WIDER than [`crate::tptp::Term`], which has no function
+/// application because the translation never emits one. A prover's output
+/// does: Skolem functions, AVATAR's introduced propositional symbols and
+/// TPTP's distinct objects all arrive here. A reader that could not represent
+/// them would have to drop them, and a dropped subterm is a different formula.
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub enum Term {
+ /// A TPTP variable: an upper word. The name is kept verbatim, because
+ /// renaming on the way in would make the alpha-equivalence levels in
+ /// [`LeafMatch`] impossible to report separately.
+ Var(String),
+ /// A function application. Arity 0 is a constant.
+ Fun(String, Vec<Term>),
+}
+
+impl Term {
+ fn vars(&self, out: &mut BTreeSet<String>) {
+ match self {
+ Term::Var(v) => {
+ out.insert(v.clone());
+ }
+ Term::Fun(_, args) => args.iter().for_each(|a| a.vars(out)),
+ }
+ }
+
+ fn render(&self, s: &mut String) {
+ match self {
+ Term::Var(v) => s.push_str(v),
+ Term::Fun(f, args) => {
+ s.push_str(f);
+ if !args.is_empty() {
+ s.push('(');
+ for (i, a) in args.iter().enumerate() {
+ if i > 0 {
+ s.push(',');
+ }
+ a.render(s);
+ }
+ s.push(')');
+ }
+ }
+ }
+ }
+}
+
+/// A first-order formula as a prover prints it.
+///
+/// `<~>`, `~|` and `~&` are read as `Not(Iff …)`, `Not(Or …)` and
+/// `Not(And …)`, and `A <= B` as `Imp(B, A)`. Those are definitional
+/// rewritings of the TPTP connectives and not normalisations of the formula:
+/// nothing here reorders, flattens or simplifies on the way in, because the
+/// leaf check's job is to notice a difference rather than to absorb one.
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
+pub enum Formula {
+ True,
+ False,
+ Pred(String, Vec<Term>),
+ Eq(Term, Term),
+ Not(Box<Formula>),
+ And(Box<Formula>, Box<Formula>),
+ Or(Box<Formula>, Box<Formula>),
+ Imp(Box<Formula>, Box<Formula>),
+ Iff(Box<Formula>, Box<Formula>),
+ All(String, Box<Formula>),
+ Ex(String, Box<Formula>),
+}
+
+impl Formula {
+ /// A canonical one-line rendering. Used for the problem digest and for
+ /// naming a mismatch in a report, never for re-parsing.
+ pub fn render(&self) -> String {
+ let mut s = String::new();
+ self.write(&mut s);
+ s
+ }
+
+ fn write(&self, s: &mut String) {
+ match self {
+ Formula::True => s.push_str("$true"),
+ Formula::False => s.push_str("$false"),
+ Formula::Pred(p, args) => {
+ s.push_str(p);
+ if !args.is_empty() {
+ s.push('(');
+ for (i, a) in args.iter().enumerate() {
+ if i > 0 {
+ s.push(',');
+ }
+ a.render(s);
+ }
+ s.push(')');
+ }
+ }
+ Formula::Eq(a, b) => {
+ a.render(s);
+ s.push('=');
+ b.render(s);
+ }
+ Formula::Not(f) => {
+ s.push_str("~(");
+ f.write(s);
+ s.push(')');
+ }
+ Formula::And(a, b) => binary(s, a, "&", b),
+ Formula::Or(a, b) => binary(s, a, "|", b),
+ Formula::Imp(a, b) => binary(s, a, "=>", b),
+ Formula::Iff(a, b) => binary(s, a, "<=>", b),
+ Formula::All(v, f) => {
+ let _ = write!(s, "![{v}]:(");
+ f.write(s);
+ s.push(')');
+ }
+ Formula::Ex(v, f) => {
+ let _ = write!(s, "?[{v}]:(");
+ f.write(s);
+ s.push(')');
+ }
+ }
+ }
+
+ /// Reassociate `&` and `|` to the right, everywhere.
+ ///
+ /// Associativity of conjunction and disjunction is not in question, and a
+ /// prover's printer is free to bracket an n-ary `&` however it likes. This
+ /// exists so that a leaf that differs ONLY in bracketing is reported as
+ /// matched at the `associativity_normalised` level rather than as a
+ /// mismatch, and so that the level is visible in the report.
+ pub fn assoc_normalised(&self) -> Formula {
+ fn flat(f: &Formula, and: bool, out: &mut Vec<Formula>) {
+ match f {
+ Formula::And(a, b) if and => {
+ flat(a, and, out);
+ flat(b, and, out);
+ }
+ Formula::Or(a, b) if !and => {
+ flat(a, and, out);
+ flat(b, and, out);
+ }
+ other => out.push(other.assoc_normalised()),
+ }
+ }
+ match self {
+ Formula::And(_, _) | Formula::Or(_, _) => {
+ let and = matches!(self, Formula::And(_, _));
+ let mut parts = Vec::new();
+ flat(self, and, &mut parts);
+ let mut it = parts.into_iter().rev();
+ let mut acc = it.next().expect("a binary node flattens to at least two parts");
+ for p in it {
+ acc = if and {
+ Formula::And(Box::new(p), Box::new(acc))
+ } else {
+ Formula::Or(Box::new(p), Box::new(acc))
+ };
+ }
+ acc
+ }
+ Formula::Not(f) => Formula::Not(Box::new(f.assoc_normalised())),
+ Formula::Imp(a, b) => Formula::Imp(
+ Box::new(a.assoc_normalised()),
+ Box::new(b.assoc_normalised()),
+ ),
+ Formula::Iff(a, b) => Formula::Iff(
+ Box::new(a.assoc_normalised()),
+ Box::new(b.assoc_normalised()),
+ ),
+ Formula::All(v, f) => Formula::All(v.clone(), Box::new(f.assoc_normalised())),
+ Formula::Ex(v, f) => Formula::Ex(v.clone(), Box::new(f.assoc_normalised())),
+ other => other.clone(),
+ }
+ }
+}
+
+fn binary(s: &mut String, a: &Formula, op: &str, b: &Formula) {
+ s.push('(');
+ a.write(s);
+ s.push_str(op);
+ b.write(s);
+ s.push(')');
+}
+
+/// Equality up to renaming of BOUND variables. Free variables must agree by
+/// name, because a free variable in a TPTP clause is implicitly universally
+/// quantified over the clause and its name is the only thing linking its
+/// occurrences.
+pub fn alpha_eq(a: &Formula, b: &Formula) -> bool {
+ fn go(a: &Formula, b: &Formula, l: &mut Vec<String>, r: &mut Vec<String>) -> bool {
+ // A bound variable is identified by its distance from the binder, so a
+ // name that shadows an outer binder resolves to the inner one, which
+ // is what `rposition` gives.
+ fn same_term(x: &Term, y: &Term, l: &[String], r: &[String]) -> bool {
+ match (x, y) {
+ (Term::Var(u), Term::Var(v)) => {
+ match (l.iter().rposition(|n| n == u), r.iter().rposition(|n| n == v)) {
+ (Some(i), Some(j)) => i == j,
+ (None, None) => u == v,
+ _ => false,
+ }
+ }
+ (Term::Fun(f, xs), Term::Fun(g, ys)) => {
+ f == g
+ && xs.len() == ys.len()
+ && xs.iter().zip(ys).all(|(x, y)| same_term(x, y, l, r))
+ }
+ _ => false,
+ }
+ }
+ match (a, b) {
+ (Formula::True, Formula::True) | (Formula::False, Formula::False) => true,
+ (Formula::Pred(p, xs), Formula::Pred(q, ys)) => {
+ p == q
+ && xs.len() == ys.len()
+ && xs.iter().zip(ys).all(|(x, y)| same_term(x, y, l, r))
+ }
+ (Formula::Eq(x1, y1), Formula::Eq(x2, y2)) => {
+ same_term(x1, x2, l, r) && same_term(y1, y2, l, r)
+ }
+ (Formula::Not(f), Formula::Not(g)) => go(f, g, l, r),
+ (Formula::And(f1, g1), Formula::And(f2, g2))
+ | (Formula::Or(f1, g1), Formula::Or(f2, g2))
+ | (Formula::Imp(f1, g1), Formula::Imp(f2, g2))
+ | (Formula::Iff(f1, g1), Formula::Iff(f2, g2)) => {
+ go(f1, f2, l, r) && go(g1, g2, l, r)
+ }
+ (Formula::All(u, f), Formula::All(v, g)) | (Formula::Ex(u, f), Formula::Ex(v, g)) => {
+ l.push(u.clone());
+ r.push(v.clone());
+ let ok = go(f, g, l, r);
+ l.pop();
+ r.pop();
+ ok
+ }
+ _ => false,
+ }
+ }
+ go(a, b, &mut Vec::new(), &mut Vec::new())
+}
+
+// ── Lexer ──────────────────────────────────────────────────────────────────
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+enum Tok {
+ LParen,
+ RParen,
+ LBrack,
+ RBrack,
+ Comma,
+ Dot,
+ Colon,
+ Op(&'static str),
+ /// A lower word, a dollar word or a number. Names, roles, rules and
+ /// predicate symbols all arrive as this.
+ Word(String),
+ /// A single-quoted TPTP atom, with `\'` and `\\` already unescaped. This
+ /// is how every IRI in this repository's export travels.
+ Quoted(String),
+ /// A double-quoted TPTP DISTINCT OBJECT. Kept with its quotes so that it
+ /// can never compare equal to a single-quoted atom of the same text: a
+ /// distinct object is pairwise unequal to every other by fiat, which is
+ /// exactly the trap decision 0005 records for the TPTP writer.
+ Distinct(String),
+ Var(String),
+}
+
+/// Every operator token, longest-first where one is a prefix of another:
+/// `!=` before `!`, `<=>` before `<=`, `~|` before `~`. The lexer scans this
+/// list in order and takes the first hit, so the order is the disambiguation.
+const OPS: [&str; 13] =
+ ["<=>", "<~>", "~|", "~&", "!=", "=>", "<=", "&", "|", "~", "=", "!", "?"];
+
+fn lex(src: &str) -> Result<Vec<Tok>, String> {
+ let b: Vec<char> = src.chars().collect();
+ let mut i = 0usize;
+ let mut out = Vec::new();
+ while i < b.len() {
+ let c = b[i];
+ if c.is_whitespace() {
+ i += 1;
+ continue;
+ }
+ // `%` is TPTP's line comment. `#` is E's: it prefixes every line of
+ // its own commentary, including the `# SZS output start` marker and
+ // the `# SZS output end` one that closes the block, so a reader that
+ // did not skip it would choke on the last line of every E proof. A
+ // `#` inside an IRI is safe, because a quoted atom is lexed by the
+ // branch below and this one never sees the inside of one.
+ if c == '%' || c == '#' {
+ while i < b.len() && b[i] != '\n' {
+ i += 1;
+ }
+ continue;
+ }
+ if c == '/' && i + 1 < b.len() && b[i + 1] == '*' {
+ i += 2;
+ while i + 1 < b.len() && !(b[i] == '*' && b[i + 1] == '/') {
+ i += 1;
+ }
+ i = (i + 2).min(b.len());
+ continue;
+ }
+ if c == '\'' || c == '"' {
+ let quote = c;
+ let mut s = String::new();
+ i += 1;
+ loop {
+ if i >= b.len() {
+ return Err(format!("unterminated {quote} at end of input"));
+ }
+ if b[i] == '\\' && i + 1 < b.len() {
+ s.push(b[i + 1]);
+ i += 2;
+ continue;
+ }
+ if b[i] == quote {
+ i += 1;
+ break;
+ }
+ s.push(b[i]);
+ i += 1;
+ }
+ out.push(if quote == '\'' { Tok::Quoted(s) } else { Tok::Distinct(format!("\"{s}\"")) });
+ continue;
+ }
+ match c {
+ '(' => {
+ out.push(Tok::LParen);
+ i += 1;
+ continue;
+ }
+ ')' => {
+ out.push(Tok::RParen);
+ i += 1;
+ continue;
+ }
+ '[' => {
+ out.push(Tok::LBrack);
+ i += 1;
+ continue;
+ }
+ ']' => {
+ out.push(Tok::RBrack);
+ i += 1;
+ continue;
+ }
+ ',' => {
+ out.push(Tok::Comma);
+ i += 1;
+ continue;
+ }
+ '.' => {
+ out.push(Tok::Dot);
+ i += 1;
+ continue;
+ }
+ ':' => {
+ out.push(Tok::Colon);
+ i += 1;
+ continue;
+ }
+ _ => {}
+ }
+ // `!=` must be tried before `!`, `<=>` before `<=`, and `~|` before
+ // `~`; OPS is in that order and is scanned in it.
+ if let Some(op) = OPS.iter().find(|op| {
+ b[i..].len() >= op.len() && b[i..i + op.len()].iter().collect::<String>() == **op
+ }) {
+ out.push(Tok::Op(op));
+ i += op.len();
+ continue;
+ }
+ if c == '$' || c.is_ascii_alphabetic() || c == '_' || c.is_ascii_digit() {
+ let start = i;
+ if c == '$' {
+ i += 1;
+ if i < b.len() && b[i] == '$' {
+ i += 1;
+ }
+ }
+ while i < b.len() && (b[i].is_ascii_alphanumeric() || b[i] == '_') {
+ i += 1;
+ }
+ let w: String = b[start..i].iter().collect();
+ let first = w.chars().find(|ch| ch.is_ascii_alphabetic());
+ if first.is_some_and(|ch| ch.is_ascii_uppercase()) && !w.starts_with('$') {
+ out.push(Tok::Var(w));
+ } else {
+ out.push(Tok::Word(w));
+ }
+ continue;
+ }
+ if c == '+' || c == '-' {
+ let start = i;
+ i += 1;
+ while i < b.len() && b[i].is_ascii_digit() {
+ i += 1;
+ }
+ if i == start + 1 {
+ return Err(format!("unexpected character {c:?}"));
+ }
+ out.push(Tok::Word(b[start..i].iter().collect()));
+ continue;
+ }
+ return Err(format!("unexpected character {c:?}"));
+ }
+ Ok(out)
+}
+
+// ── The annotated-formula list ─────────────────────────────────────────────
+
+/// One `inference(RULE, [status(…)], [parents])` record.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct Inference {
+ pub rule: String,
+ pub parents: Vec<Parent>,
+}
+
+/// A parent of an inference step.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum Parent {
+ /// A reference to another annotated formula, by name.
+ Named(String),
+ /// An inference record written INLINE in the parent position, which E does
+ /// routinely. It has a rule and parents of its own and NO FORMULA, so
+ /// nothing about it or about the step it feeds can be replayed. It is
+ /// given a synthetic node so that its own parent references are still
+ /// resolved and its rule is still counted.
+ Inline(Box<Inference>),
+ /// Anything else in a parent position, kept verbatim so a report can name
+ /// it rather than pretend it was understood.
+ Other(String),
+}
+
+/// The provenance a prover attaches to an annotated formula.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum Source {
+ /// `file('path', name)`. The second argument is the name the formula had
+ /// in the problem file, which is what makes the leaf check possible.
+ File { path: String, name: Option<String> },
+ Inference(Inference),
+ /// `introduced(definition, …)`. A symbol the prover invented. It is NOT
+ /// from the problem, it is not derived from the problem, and no
+ /// conservativity argument is checked here.
+ Introduced { rule: String, detail: String },
+ /// A bare name, which some printers use for a trivial re-statement.
+ Name(String),
+ /// A source this reader did not understand, kept verbatim.
+ Other(String),
+}
+
+/// One `fof(…)` or `cnf(…)` line.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct Annotated {
+ pub name: String,
+ pub role: String,
+ pub formula: Formula,
+ pub source: Option<Source>,
+}
+
+/// A general term, the TPTP grammar's catch-all for annotations.
+#[derive(Clone, Debug, PartialEq, Eq)]
+enum GTerm {
+ Word(String),
+ Func(String, Vec<GTerm>),
+ List(Vec<GTerm>),
+ Other(String),
+}
+
+impl GTerm {
+ fn word(&self) -> Option<&str> {
+ match self {
+ GTerm::Word(w) | GTerm::Func(w, _) => Some(w),
+ _ => None,
+ }
+ }
+ fn render(&self) -> String {
+ match self {
+ GTerm::Word(w) => w.clone(),
+ GTerm::Other(w) => w.clone(),
+ GTerm::Func(f, args) => {
+ format!("{f}({})", args.iter().map(GTerm::render).collect::<Vec<_>>().join(","))
+ }
+ GTerm::List(xs) => {
+ format!("[{}]", xs.iter().map(GTerm::render).collect::<Vec<_>>().join(","))
+ }
+ }
+ }
+}
+
+struct Parser {
+ toks: Vec<Tok>,
+ at: usize,
+}
+
+impl Parser {
+ fn peek(&self) -> Option<&Tok> {
+ self.toks.get(self.at)
+ }
+ fn bump(&mut self) -> Option<Tok> {
+ let t = self.toks.get(self.at).cloned();
+ if t.is_some() {
+ self.at += 1;
+ }
+ t
+ }
+ fn eat(&mut self, t: &Tok) -> bool {
+ if self.peek() == Some(t) {
+ self.at += 1;
+ true
+ } else {
+ false
+ }
+ }
+ fn expect(&mut self, t: &Tok) -> Result<(), String> {
+ if self.eat(t) {
+ Ok(())
+ } else {
+ Err(format!("expected {:?}, found {:?}", t, self.peek()))
+ }
+ }
+
+ fn name(&mut self) -> Result<String, String> {
+ match self.bump() {
+ Some(Tok::Word(w)) | Some(Tok::Quoted(w)) => Ok(w),
+ other => Err(format!("expected a name, found {other:?}")),
+ }
+ }
+
+ fn term(&mut self) -> Result<Term, String> {
+ match self.bump() {
+ Some(Tok::Var(v)) => Ok(Term::Var(v)),
+ Some(Tok::Distinct(d)) => Ok(Term::Fun(d, Vec::new())),
+ Some(Tok::Word(w)) | Some(Tok::Quoted(w)) => {
+ let mut args = Vec::new();
+ if self.eat(&Tok::LParen) {
+ loop {
+ args.push(self.term()?);
+ if self.eat(&Tok::Comma) {
+ continue;
+ }
+ self.expect(&Tok::RParen)?;
+ break;
+ }
+ }
+ Ok(Term::Fun(w, args))
+ }
+ other => Err(format!("expected a term, found {other:?}")),
+ }
+ }
+
+ /// A unitary formula: a bracketed formula, a negation, a quantification,
+ /// or an atom. TPTP makes a quantifier's body unitary, so the body is
+ /// parsed here rather than at the binary level.
+ fn unit(&mut self) -> Result<Formula, String> {
+ match self.peek().cloned() {
+ Some(Tok::LParen) => {
+ self.at += 1;
+ let f = self.formula(0)?;
+ self.expect(&Tok::RParen)?;
+ Ok(f)
+ }
+ Some(Tok::Op("~")) => {
+ self.at += 1;
+ Ok(Formula::Not(Box::new(self.unit()?)))
+ }
+ Some(Tok::Op(q @ ("!" | "?"))) => {
+ self.at += 1;
+ self.expect(&Tok::LBrack)?;
+ let mut vars = Vec::new();
+ loop {
+ match self.bump() {
+ Some(Tok::Var(v)) => vars.push(v),
+ other => return Err(format!("expected a variable, found {other:?}")),
+ }
+ // TFF writes `X : $i`; the sort is read and discarded,
+ // because this reader is about FOF and an untyped reading
+ // of a typed binder is the same binder.
+ if self.eat(&Tok::Colon) {
+ self.bump();
+ }
+ if self.eat(&Tok::Comma) {
+ continue;
+ }
+ break;
+ }
+ self.expect(&Tok::RBrack)?;
+ self.expect(&Tok::Colon)?;
+ let body = self.unit()?;
+ Ok(vars.into_iter().rev().fold(body, |acc, v| {
+ if q == "!" {
+ Formula::All(v, Box::new(acc))
+ } else {
+ Formula::Ex(v, Box::new(acc))
+ }
+ }))
+ }
+ Some(Tok::Word(w)) if w == "$true" => {
+ self.at += 1;
+ Ok(Formula::True)
+ }
+ Some(Tok::Word(w)) if w == "$false" => {
+ self.at += 1;
+ Ok(Formula::False)
+ }
+ _ => {
+ let lhs = self.term()?;
+ match self.peek().cloned() {
+ Some(Tok::Op("=")) => {
+ self.at += 1;
+ Ok(Formula::Eq(lhs, self.term()?))
+ }
+ Some(Tok::Op("!=")) => {
+ self.at += 1;
+ Ok(Formula::Not(Box::new(Formula::Eq(lhs, self.term()?))))
+ }
+ _ => match lhs {
+ Term::Fun(p, args) => Ok(Formula::Pred(p, args)),
+ Term::Var(v) => {
+ Err(format!("a bare variable {v} is not a formula in FOF"))
+ }
+ },
+ }
+ }
+ }
+ }
+
+ fn formula(&mut self, min: u8) -> Result<Formula, String> {
+ let mut left = self.unit()?;
+ loop {
+ let (op, prec) = match self.peek() {
+ Some(Tok::Op(o @ ("<=>" | "<~>"))) => (*o, 10u8),
+ Some(Tok::Op(o @ ("=>" | "<="))) => (*o, 20),
+ Some(Tok::Op(o @ ("|" | "~|"))) => (*o, 30),
+ Some(Tok::Op(o @ ("&" | "~&"))) => (*o, 40),
+ _ => break,
+ };
+ if prec < min {
+ break;
+ }
+ self.at += 1;
+ // Right-associative throughout. `&` and `|` are associative, so
+ // the choice is arbitrary for them; `=>` is right-associative in
+ // TPTP. `Form::conj` in src/tptp.rs right-associates too, which is
+ // what makes a leaf of this repository's own export come back
+ // structurally identical rather than merely equivalent.
+ let right = self.formula(prec)?;
+ left = match op {
+ "<=>" => Formula::Iff(Box::new(left), Box::new(right)),
+ "<~>" => Formula::Not(Box::new(Formula::Iff(Box::new(left), Box::new(right)))),
+ "=>" => Formula::Imp(Box::new(left), Box::new(right)),
+ "<=" => Formula::Imp(Box::new(right), Box::new(left)),
+ "|" => Formula::Or(Box::new(left), Box::new(right)),
+ "~|" => Formula::Not(Box::new(Formula::Or(Box::new(left), Box::new(right)))),
+ "&" => Formula::And(Box::new(left), Box::new(right)),
+ "~&" => Formula::Not(Box::new(Formula::And(Box::new(left), Box::new(right)))),
+ _ => unreachable!("the operator table above lists every arm"),
+ };
+ }
+ Ok(left)
+ }
+
+ /// A general term. Never fails: anything it does not recognise is consumed
+ /// as balanced tokens and returned as [`GTerm::Other`], so one odd
+ /// annotation cannot make a whole derivation unreadable.
+ fn general(&mut self) -> GTerm {
+ match self.peek().cloned() {
+ Some(Tok::LBrack) => {
+ self.at += 1;
+ let mut xs = Vec::new();
+ if self.eat(&Tok::RBrack) {
+ return GTerm::List(xs);
+ }
+ loop {
+ xs.push(self.general());
+ if self.eat(&Tok::Comma) {
+ continue;
+ }
+ let _ = self.eat(&Tok::RBrack);
+ break;
+ }
+ GTerm::List(xs)
+ }
+ Some(Tok::Word(w)) | Some(Tok::Quoted(w)) => {
+ self.at += 1;
+ if self.eat(&Tok::LParen) {
+ let mut args = Vec::new();
+ if self.eat(&Tok::RParen) {
+ return GTerm::Func(w, args);
+ }
+ loop {
+ args.push(self.general());
+ if self.eat(&Tok::Comma) {
+ continue;
+ }
+ let _ = self.eat(&Tok::RParen);
+ break;
+ }
+ let mut g = GTerm::Func(w, args);
+ if self.eat(&Tok::Colon) {
+ let inner = self.general();
+ g = GTerm::Other(format!("{}:{}", g.render(), inner.render()));
+ }
+ return g;
+ }
+ if self.eat(&Tok::Colon) {
+ let inner = self.general();
+ return GTerm::Other(format!("{w}:{}", inner.render()));
+ }
+ GTerm::Word(w)
+ }
+ Some(Tok::Var(v)) | Some(Tok::Distinct(v)) => {
+ self.at += 1;
+ GTerm::Word(v)
+ }
+ _ => {
+ // Balanced skip. Stops at a top-level comma or a closing
+ // bracket, which is where a general term always ends.
+ let start = self.at;
+ let mut depth = 0i32;
+ while let Some(t) = self.peek() {
+ match t {
+ Tok::LParen | Tok::LBrack => depth += 1,
+ Tok::RParen | Tok::RBrack => {
+ if depth == 0 {
+ break;
+ }
+ depth -= 1;
+ }
+ Tok::Comma if depth == 0 => break,
+ Tok::Dot if depth == 0 => break,
+ _ => {}
+ }
+ self.at += 1;
+ }
+ if self.at == start {
+ self.at += 1;
+ }
+ GTerm::Other(format!("{:?}", &self.toks[start..self.at.min(self.toks.len())]))
+ }
+ }
+ }
+}
+
+fn parents_of(g: &GTerm) -> Vec<Parent> {
+ let items: &[GTerm] = match g {
+ GTerm::List(xs) => xs,
+ single => std::slice::from_ref(single),
+ };
+ items
+ .iter()
+ .map(|p| match p {
+ GTerm::Word(w) => Parent::Named(w.clone()),
+ GTerm::Func(f, args) if f == "inference" => match inference_of(args) {
+ Some(inf) => Parent::Inline(Box::new(inf)),
+ None => Parent::Other(p.render()),
+ },
+ other => Parent::Other(other.render()),
+ })
+ .collect()
+}
+
+fn inference_of(args: &[GTerm]) -> Option<Inference> {
+ let rule = args.first()?.word()?.to_string();
+ let parents = args.get(2).map(parents_of).unwrap_or_default();
+ Some(Inference { rule, parents })
+}
+
+fn source_of(g: &GTerm) -> Source {
+ match g {
+ GTerm::Func(f, args) if f == "file" => Source::File {
+ path: args.first().map(GTerm::render).unwrap_or_default(),
+ name: args.get(1).and_then(|n| n.word()).map(|s| s.to_string()),
+ },
+ GTerm::Func(f, args) if f == "inference" => match inference_of(args) {
+ Some(inf) => Source::Inference(inf),
+ None => Source::Other(g.render()),
+ },
+ GTerm::Func(f, args) if f == "introduced" => Source::Introduced {
+ rule: args.first().and_then(|r| r.word()).unwrap_or("introduced").to_string(),
+ detail: args.iter().skip(1).map(GTerm::render).collect::<Vec<_>>().join(","),
+ },
+ GTerm::Word(w) => Source::Name(w.clone()),
+ other => Source::Other(other.render()),
+ }
+}
+
+/// The SZS status the prover printed, if it printed one.
+///
+/// The LAST status line wins: E prints an input status before its result, and
+/// taking the first would report the wrong one. The word is ECHOED and never
+/// acted on as though it were evidence.
+pub fn szs_status(text: &str) -> Option<String> {
+ let mut last = None;
+ for line in text.lines() {
+ if let Some(i) = line.find("SZS status") {
+ let rest = line[i + "SZS status".len()..].trim_start();
+ let word: String =
+ rest.chars().take_while(|c| c.is_ascii_alphanumeric() || *c == '_').collect();
+ if !word.is_empty() {
+ last = Some(word);
+ }
+ }
+ }
+ last
+}
+
+/// Read every `fof(…)` / `cnf(…)` annotated formula out of a prover's output.
+///
+/// When the text carries an `SZS output start` / `SZS output end` pair, ONLY
+/// what lies between them is read. E prints its saturation state before the
+/// proof, in the same `cnf(…)` syntax and with names it reuses, and a reader
+/// that swallowed those would build a derivation out of clauses the prover had
+/// abandoned.
+pub fn parse_derivation(text: &str) -> Result<Vec<Annotated>, String> {
+ let body = match (text.find("SZS output start"), text.find("SZS output end")) {
+ (Some(a), Some(b)) if b > a => {
+ let after = &text[a..];
+ let nl = after.find('\n').map(|n| a + n + 1).unwrap_or(a);
+ &text[nl..b]
+ }
+ _ => text,
+ };
+ parse_annotated_list(body)
+}
+
+/// Read every annotated formula in a plain TPTP file. Used for the problem
+/// side, where there is no SZS block.
+pub fn parse_problem(text: &str) -> Result<Vec<Annotated>, String> {
+ parse_annotated_list(text)
+}
+
+fn parse_annotated_list(text: &str) -> Result<Vec<Annotated>, String> {
+ let toks = lex(text)?;
+ let mut p = Parser { toks, at: 0 };
+ let mut out = Vec::new();
+ while let Some(t) = p.peek().cloned() {
+ let kind = match &t {
+ Tok::Word(w) => w.clone(),
+ _ => {
+ return Err(format!(
+ "expected `fof` or `cnf` at the start of an annotated formula, found {t:?}"
+ ));
+ }
+ };
+ match kind.as_str() {
+ "fof" | "cnf" => {}
+ "tff" | "thf" | "tcf" | "tpi" => {
+ return Err(format!(
+ "`{kind}` is outside this reader's language. Only untyped first-order (fof \
+ and cnf) is read, because that is what src/tptp.rs emits and what the \
+ adequacy theorem is about"
+ ));
+ }
+ other => return Err(format!("unknown annotated-formula kind `{other}`")),
+ }
+ p.at += 1;
+ p.expect(&Tok::LParen)?;
+ let name = p.name()?;
+ p.expect(&Tok::Comma)?;
+ let role = p.name()?;
+ p.expect(&Tok::Comma)?;
+ let formula = p.formula(0)?;
+ let mut source = None;
+ if p.eat(&Tok::Comma) {
+ source = Some(source_of(&p.general()));
+ // Useful-info fields, e.g. E's trailing `['proof']`. Read and
+ // discarded: nothing in them is load-bearing here.
+ while p.eat(&Tok::Comma) {
+ let _ = p.general();
+ }
+ }
+ p.expect(&Tok::RParen)?;
+ p.expect(&Tok::Dot)?;
+ out.push(Annotated { name, role, formula, source });
+ }
+ Ok(out)
+}
+
+/// The problem this repository's exporter emits, embedded into the reader's
+/// wider language.
+///
+/// Used by `tests/tstp_parser_test.rs` to pin the reader against the emitter:
+/// parsing `fof::form(f)` must give back exactly `from_tptp_form(f)`. A reader
+/// that silently normalised on the way in would pass a leaf check it should
+/// have failed, which is the one thing this module must not do.
+pub fn from_tptp_form(f: &crate::tptp::Form) -> Formula {
+ use crate::tptp::{Form, Term as T, sym};
+ fn term(t: &T) -> Term {
+ match t {
+ T::Var(n) => Term::Var(format!("X{n}")),
+ T::Const(a) => Term::Fun(sym::constant(a), Vec::new()),
+ }
+ }
+ match f {
+ Form::App1(p, t) => Formula::Pred(sym::p1(p), vec![term(t)]),
+ Form::App2(p, t, u) => Formula::Pred(sym::p2(p), vec![term(t), term(u)]),
+ Form::Eq(t, u) => Formula::Eq(term(t), term(u)),
+ Form::Tru => Formula::True,
+ Form::Fls => Formula::False,
+ Form::Neg(g) => Formula::Not(Box::new(from_tptp_form(g))),
+ Form::And(g, h) => {
+ Formula::And(Box::new(from_tptp_form(g)), Box::new(from_tptp_form(h)))
+ }
+ Form::Or(g, h) => Formula::Or(Box::new(from_tptp_form(g)), Box::new(from_tptp_form(h))),
+ Form::Imp(g, h) => {
+ Formula::Imp(Box::new(from_tptp_form(g)), Box::new(from_tptp_form(h)))
+ }
+ Form::All(n, g) => Formula::All(format!("X{n}"), Box::new(from_tptp_form(g))),
+ Form::Ex(n, g) => Formula::Ex(format!("X{n}"), Box::new(from_tptp_form(g))),
+ }
+}
+
+// ── Clause view, unification, and the resolvent ────────────────────────────
+
+/// An atom in a clause. Equality is stored with its arguments in a canonical
+/// order, because `s = t` and `t = s` are the same literal and a set
+/// comparison that treated them as two would refuse correct steps.
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
+enum Atom {
+ Pred(String, Vec<Term>),
+ Eq(Term, Term),
+}
+
+impl Atom {
+ fn eq_canon(a: Term, b: Term) -> Atom {
+ if a <= b { Atom::Eq(a, b) } else { Atom::Eq(b, a) }
+ }
+ fn vars(&self, out: &mut BTreeSet<String>) {
+ match self {
+ Atom::Pred(_, args) => args.iter().for_each(|a| a.vars(out)),
+ Atom::Eq(a, b) => {
+ a.vars(out);
+ b.vars(out);
+ }
+ }
+ }
+ fn render(&self) -> String {
+ let mut s = String::new();
+ match self {
+ Atom::Pred(p, args) => {
+ s.push_str(p);
+ if !args.is_empty() {
+ s.push('(');
+ for (i, a) in args.iter().enumerate() {
+ if i > 0 {
+ s.push(',');
+ }
+ a.render(&mut s);
+ }
+ s.push(')');
+ }
+ }
+ Atom::Eq(a, b) => {
+ a.render(&mut s);
+ s.push('=');
+ b.render(&mut s);
+ }
+ }
+ s
+ }
+}
+
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
+struct Lit {
+ positive: bool,
+ atom: Atom,
+}
+
+impl Lit {
+ fn render(&self) -> String {
+ if self.positive { self.atom.render() } else { format!("~{}", self.atom.render()) }
+ }
+}
+
+/// A formula seen as a clause: a set of literals under an implicit universal
+/// prefix.
+///
+/// `$false` disjuncts are DROPPED and a `$true` disjunct makes the whole thing
+/// a tautology. Both are ordinary clause normalisations and both are needed on
+/// real output: Vampire prints AVATAR conclusions as `$false | (spl1 | ~spl2)`,
+/// where the `$false` is the empty clause part and the rest is the assertion.
+#[derive(Clone, Debug, PartialEq, Eq)]
+struct Clause {
+ lits: Vec<Lit>,
+ tautology: bool,
+}
+
+impl Clause {
+ fn render(&self) -> String {
+ if self.tautology {
+ return "$true".into();
+ }
+ if self.lits.is_empty() {
+ return "$false".into();
+ }
+ self.lits.iter().map(Lit::render).collect::<Vec<_>>().join(" | ")
+ }
+ fn normalise(mut lits: Vec<Lit>, tautology: bool) -> Clause {
+ lits.sort();
+ lits.dedup();
+ Clause { lits, tautology }
+ }
+ fn vars(&self) -> BTreeSet<String> {
+ let mut v = BTreeSet::new();
+ self.lits.iter().for_each(|l| l.atom.vars(&mut v));
+ v
+ }
+}
+
+/// Read a formula as a clause, or say why it is not one.
+fn clause_view(f: &Formula) -> Result<Clause, &'static str> {
+ fn body<'a>(f: &'a Formula, bound: &mut Vec<&'a str>) -> &'a Formula {
+ match f {
+ Formula::All(v, g) => {
+ bound.push(v);
+ body(g, bound)
+ }
+ other => other,
+ }
+ }
+ fn lits(f: &Formula, out: &mut Vec<Lit>, taut: &mut bool) -> Result<(), &'static str> {
+ match f {
+ Formula::Or(a, b) => {
+ lits(a, out, taut)?;
+ lits(b, out, taut)
+ }
+ Formula::False => Ok(()),
+ Formula::True => {
+ *taut = true;
+ Ok(())
+ }
+ Formula::Pred(p, args) => {
+ out.push(Lit { positive: true, atom: Atom::Pred(p.clone(), args.clone()) });
+ Ok(())
+ }
+ Formula::Eq(a, b) => {
+ out.push(Lit { positive: true, atom: Atom::eq_canon(a.clone(), b.clone()) });
+ Ok(())
+ }
+ Formula::Not(g) => match &**g {
+ Formula::Pred(p, args) => {
+ out.push(Lit { positive: false, atom: Atom::Pred(p.clone(), args.clone()) });
+ Ok(())
+ }
+ Formula::Eq(a, b) => {
+ out.push(Lit {
+ positive: false,
+ atom: Atom::eq_canon(a.clone(), b.clone()),
+ });
+ Ok(())
+ }
+ Formula::True => Ok(()),
+ Formula::False => {
+ *taut = true;
+ Ok(())
+ }
+ _ => Err("a negation over a compound formula is not a literal"),
+ },
+ _ => Err("a connective other than disjunction appears in the clause body"),
+ }
+ }
+ let mut bound = Vec::new();
+ let inner = body(f, &mut bound);
+ if matches!(inner, Formula::Ex(_, _) | Formula::All(_, _)) {
+ return Err("a quantifier appears below the universal prefix");
+ }
+ let mut out = Vec::new();
+ let mut taut = false;
+ lits(inner, &mut out, &mut taut)?;
+ Ok(Clause::normalise(out, taut))
+}
+
+type Subst = HashMap<String, Term>;
+
+fn walk(t: &Term, s: &Subst) -> Term {
+ match t {
+ Term::Var(v) => match s.get(v) {
+ Some(b) => walk(b, s),
+ None => t.clone(),
+ },
+ Term::Fun(f, args) => Term::Fun(f.clone(), args.iter().map(|a| walk(a, s)).collect()),
+ }
+}
+
+fn occurs(v: &str, t: &Term, s: &Subst) -> bool {
+ match walk(t, s) {
+ Term::Var(u) => u == v,
+ Term::Fun(_, args) => args.iter().any(|a| occurs(v, a, s)),
+ }
+}
+
+/// Syntactic first-order unification with an occurs check. No theory, no
+/// ordering, no AC: a most general unifier or nothing.
+fn unify(a: &Term, b: &Term, s: &mut Subst) -> bool {
+ let (x, y) = (walk(a, s), walk(b, s));
+ match (&x, &y) {
+ (Term::Var(u), Term::Var(v)) if u == v => true,
+ (Term::Var(u), other) | (other, Term::Var(u)) => {
+ if occurs(u, other, s) {
+ return false;
+ }
+ s.insert(u.clone(), other.clone());
+ true
+ }
+ (Term::Fun(f, xs), Term::Fun(g, ys)) => {
+ f == g && xs.len() == ys.len() && xs.iter().zip(ys).all(|(p, q)| unify(p, q, s))
+ }
+ }
+}
+
+fn unify_atoms(a: &Atom, b: &Atom) -> Option<Subst> {
+ let mut s = Subst::new();
+ match (a, b) {
+ (Atom::Pred(p, xs), Atom::Pred(q, ys)) => {
+ if p != q || xs.len() != ys.len() {
+ return None;
+ }
+ for (x, y) in xs.iter().zip(ys) {
+ if !unify(x, y, &mut s) {
+ return None;
+ }
+ }
+ Some(s)
+ }
+ (Atom::Eq(x1, y1), Atom::Eq(x2, y2)) => {
+ // Equality is symmetric, so both orientations are tried. The
+ // canonical storage order is a comparison convenience and must not
+ // become a restriction on what unifies.
+ for (p, q) in [(x2, y2), (y2, x2)] {
+ let mut t = Subst::new();
+ if unify(x1, p, &mut t) && unify(y1, q, &mut t) {
+ return Some(t);
+ }
+ }
+ None
+ }
+ _ => None,
+ }
+}
+
+fn apply_atom(a: &Atom, s: &Subst) -> Atom {
+ match a {
+ Atom::Pred(p, args) => Atom::Pred(p.clone(), args.iter().map(|t| walk(t, s)).collect()),
+ Atom::Eq(x, y) => Atom::eq_canon(walk(x, s), walk(y, s)),
+ }
+}
+
+fn apply_clause(c: &[Lit], s: &Subst) -> Vec<Lit> {
+ c.iter().map(|l| Lit { positive: l.positive, atom: apply_atom(&l.atom, s) }).collect()
+}
+
+/// Rename every variable of a clause with a prefix nothing else uses, so two
+/// premises cannot share a variable by accident.
+fn rename_apart(c: &Clause, tag: &str) -> Clause {
+ let mut s = Subst::new();
+ for v in c.vars() {
+ s.insert(v.clone(), Term::Var(format!("{tag}{v}")));
+ }
+ Clause::normalise(apply_clause(&c.lits, &s), c.tautology)
+}
+
+/// Are two clauses the same up to a BIJECTIVE renaming of their variables?
+///
+/// The fast path is exact set equality, which is what nearly every real step
+/// hits because provers keep their variable names. The slow path pairs
+/// literals with backtracking under a budget; the budget is reported rather
+/// than silently treated as a mismatch, because "too big to check" and "does
+/// not check out" are different facts.
+fn variant(a: &Clause, b: &Clause) -> Result<bool, &'static str> {
+ if a.tautology != b.tautology {
+ return Ok(false);
+ }
+ if a.tautology {
+ return Ok(true);
+ }
+ if a.lits == b.lits {
+ return Ok(true);
+ }
+ if a.lits.len() != b.lits.len() {
+ return Ok(false);
+ }
+ const BUDGET: u32 = 200_000;
+ let mut used = vec![false; b.lits.len()];
+ let mut fwd: HashMap<String, String> = HashMap::new();
+ let mut bwd: HashMap<String, String> = HashMap::new();
+ let mut spent = 0u32;
+ let ok = pair(&a.lits, &b.lits, 0, &mut used, &mut fwd, &mut bwd, &mut spent, BUDGET);
+ match ok {
+ Some(v) => Ok(v),
+ None => Err("the variant check ran out of budget"),
+ }
+}
+
+#[allow(clippy::too_many_arguments)]
+fn pair(
+ a: &[Lit],
+ b: &[Lit],
+ i: usize,
+ used: &mut Vec<bool>,
+ fwd: &mut HashMap<String, String>,
+ bwd: &mut HashMap<String, String>,
+ spent: &mut u32,
+ budget: u32,
+) -> Option<bool> {
+ if i == a.len() {
+ return Some(true);
+ }
+ for j in 0..b.len() {
+ if used[j] {
+ continue;
+ }
+ *spent += 1;
+ if *spent > budget {
+ return None;
+ }
+ let (f0, b0) = (fwd.clone(), bwd.clone());
+ if match_lit(&a[i], &b[j], fwd, bwd) {
+ used[j] = true;
+ match pair(a, b, i + 1, used, fwd, bwd, spent, budget) {
+ None => return None,
+ Some(true) => return Some(true),
+ Some(false) => {}
+ }
+ used[j] = false;
+ }
+ *fwd = f0;
+ *bwd = b0;
+ }
+ Some(false)
+}
+
+fn match_lit(
+ x: &Lit,
+ y: &Lit,
+ fwd: &mut HashMap<String, String>,
+ bwd: &mut HashMap<String, String>,
+) -> bool {
+ if x.positive != y.positive {
+ return false;
+ }
+ match (&x.atom, &y.atom) {
+ (Atom::Pred(p, xs), Atom::Pred(q, ys)) => {
+ p == q
+ && xs.len() == ys.len()
+ && xs.iter().zip(ys).all(|(u, v)| match_term(u, v, fwd, bwd))
+ }
+ (Atom::Eq(x1, y1), Atom::Eq(x2, y2)) => {
+ let (f0, b0) = (fwd.clone(), bwd.clone());
+ if match_term(x1, x2, fwd, bwd) && match_term(y1, y2, fwd, bwd) {
+ return true;
+ }
+ *fwd = f0;
+ *bwd = b0;
+ match_term(x1, y2, fwd, bwd) && match_term(y1, x2, fwd, bwd)
+ }
+ _ => false,
+ }
+}
+
+fn match_term(
+ x: &Term,
+ y: &Term,
+ fwd: &mut HashMap<String, String>,
+ bwd: &mut HashMap<String, String>,
+) -> bool {
+ match (x, y) {
+ (Term::Var(u), Term::Var(v)) => {
+ match (fwd.get(u).cloned(), bwd.get(v).cloned()) {
+ (Some(a), Some(b)) => a == *v && b == *u,
+ (None, None) => {
+ fwd.insert(u.clone(), v.clone());
+ bwd.insert(v.clone(), u.clone());
+ true
+ }
+ _ => false,
+ }
+ }
+ (Term::Fun(f, xs), Term::Fun(g, ys)) => {
+ f == g
+ && xs.len() == ys.len()
+ && xs.iter().zip(ys).all(|(u, v)| match_term(u, v, fwd, bwd))
+ }
+ _ => false,
+ }
+}
+
+// ── The derivation ─────────────────────────────────────────────────────────
+
+/// How closely a leaf of the derivation matched the problem formula it names.
+///
+/// Three levels rather than a boolean, because they cost different amounts of
+/// belief. `Identical` means the prover echoed our formula unchanged.
+/// `AlphaEquivalent` means it renamed bound variables, which E does on every
+/// axiom it reads. `AssociativityNormalised` means it also rebracketed an
+/// n-ary `&` or `|`. Nothing below that is accepted.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum LeafMatch {
+ Identical,
+ AlphaEquivalent,
+ AssociativityNormalised,
+}
+
+impl LeafMatch {
+ pub fn name(self) -> &'static str {
+ match self {
+ LeafMatch::Identical => "identical",
+ LeafMatch::AlphaEquivalent => "alpha_equivalent",
+ LeafMatch::AssociativityNormalised => "associativity_normalised",
+ }
+ }
+}
+
+fn leaf_match(proof: &Formula, problem: &Formula) -> Option<LeafMatch> {
+ if proof == problem {
+ return Some(LeafMatch::Identical);
+ }
+ if alpha_eq(proof, problem) {
+ return Some(LeafMatch::AlphaEquivalent);
+ }
+ if alpha_eq(&proof.assoc_normalised(), &problem.assoc_normalised()) {
+ return Some(LeafMatch::AssociativityNormalised);
+ }
+ None
+}
+
+/// A leaf that is not a formula of the problem. The reason is carried because
+/// "the name is not in the problem" and "the name is there and the formula is
+/// different" are different accusations.
+#[derive(Clone, Debug, serde::Serialize)]
+pub struct LeafFailure {
+ pub node: String,
+ pub why: String,
+}
+
+/// A symbol or a formula the prover invented. Not from the problem, not
+/// derived from it, and nothing here checks that the extension is
+/// conservative.
+#[derive(Clone, Debug, serde::Serialize)]
+pub struct Introduced {
+ pub node: String,
+ pub rule: String,
+ pub detail: String,
+}
+
+/// A rule this checker did not replay, with the count and the reason.
+#[derive(Clone, Debug, serde::Serialize)]
+pub struct Unchecked {
+ pub rule: String,
+ pub count: usize,
+ pub why: String,
+}
+
+/// A step whose rule IS implemented, whose shapes DO apply, and which still
+/// did not reconstruct. Either the derivation is wrong or this checker is
+/// incomplete for that rule; the report says both and decides neither.
+#[derive(Clone, Debug, serde::Serialize)]
+pub struct NotReconstructed {
+ pub node: String,
+ pub rule: String,
+ pub expected: String,
+ pub found: String,
+}
+
+/// What a checked run of this module establishes, field by field, with nothing
+/// collapsed into anything else.
+///
+/// The shape follows `fol_solve::Outcome`, which decision 0006 item 4 fixes at
+/// five never-collapsed fields: what the oracle said, what it was asked, what
+/// the checker did, the one verdict word, and the reading that rides on the
+/// unproved translation. The same discipline applies here and for the same
+/// reason: a reader who has never opened a decision record still must not be
+/// able to mistake an oracle's word for a checked one.
+#[derive(Clone, Debug, serde::Serialize)]
+pub struct Report {
+ /// What the prover said about itself. ECHOED, UNTRUSTED, and never used to
+ /// decide anything below.
+ pub szs_status: Option<String>,
+ /// Why the problem file could not be read, when it could not.
+ pub problem_error: Option<String>,
+ /// Why the derivation could not be read, when it could not.
+ pub derivation_error: Option<String>,
+
+ pub problem_formulas: usize,
+ /// FNV-1a 64 over the canonical rendering of the PARSED problem, so a
+ /// report can be tied to the file it is about. It identifies and does not
+ /// commit, which is decision 0006 item 7 in the same words.
+ pub problem_digest: String,
+
+ pub derivation_nodes: usize,
+ /// Every parent reference resolves and the parent relation is acyclic.
+ pub derivation_wellformed: bool,
+ pub wellformedness_failures: Vec<String>,
+ /// Some node is the empty clause and nothing cites it.
+ pub root_is_false: bool,
+ pub nodes_outside_the_refutation: usize,
+
+ pub leaves_total: usize,
+ /// Every leaf reachable from the empty clause is a formula of the problem,
+ /// under the name the prover gave for it and with the role the problem
+ /// gave it.
+ pub leaves_match_problem: bool,
+ pub leaf_match_levels: BTreeMap<&'static str, usize>,
+ pub leaf_failures: Vec<LeafFailure>,
+ pub leaves_introduced: Vec<Introduced>,
+
+ /// The conjecture's name in the problem, when the problem has one.
+ pub conjecture_in_problem: Option<String>,
+ /// The refutation uses it.
+ pub conjecture_used: bool,
+ /// The step that negates it was replayed: the child IS the negation of the
+ /// problem's conjecture and of nothing else.
+ pub conjecture_negation_checked: bool,
+ /// `axioms_and_the_negated_conjecture` or `axioms_alone`. Null unless the
+ /// structure holds, because it is a statement about a derivation this
+ /// module has read and not about a theory.
+ pub what_was_refuted: Option<&'static str>,
+
+ pub steps_total: usize,
+ pub steps_checked: usize,
+ pub steps_checked_by_rule: BTreeMap<String, usize>,
+ pub steps_unchecked: Vec<Unchecked>,
+ pub steps_not_reconstructed: Vec<NotReconstructed>,
+
+ /// The one word. See [`verdict_means`].
+ pub verdict: &'static str,
+}
+
+/// One sentence per word, so a report is readable without this file open
+/// beside it.
+pub fn verdict_means(v: &str) -> &'static str {
+ match v {
+ "problem_unparsed" => "the TPTP problem could not be read, so there was nothing to \
+ check the derivation against. Not a statement about the prover",
+ "derivation_unparsed" => "the prover's output could not be read as a TSTP derivation. \
+ Run the prover with a proof-printing option (vampire \
+ `--proof tptp`, eprover `--proof-object`); without one there \
+ is only a word",
+ "no_refutation_offered" => "the output contains no empty clause, so there is no \
+ refutation to check. This is the normal outcome of a \
+ satisfiable problem, a timeout or a give-up, and it is \
+ evidence of nothing in either direction",
+ "derivation_rejected" => "THE CHECKER SAID NO. Either a parent reference does not \
+ resolve, or the parent relation has a cycle, or a leaf of the \
+ refutation is not a formula of the problem. A refutation of \
+ something else is not a refutation of this",
+ "refutation_step_not_reconstructed" => "the structure holds and every leaf is ours, and \
+ at least one step whose rule this checker \
+ implements did not reconstruct from its \
+ premises. EITHER the derivation is wrong OR \
+ this checker is incomplete for that rule. This \
+ module does not decide which, and the step is \
+ named",
+ "refutation_structure_checked" => "the derivation is a well-founded DAG from the \
+ problem's own formulas to the empty clause, and NO \
+ inference step was replayed. It rules out a prover \
+ answering about a different file; it says nothing \
+ about whether the steps are sound",
+ "refutation_partially_replayed" => "the structure holds and SOME steps were recomputed \
+ from their premises. The unchecked rules are named \
+ and counted. This is the normal outcome, because \
+ clausification and AVATAR splitting are not checked",
+ "refutation_fully_replayed" => "the structure holds and EVERY step was recomputed. This \
+ is the strongest word here and it is still NOT a proof \
+ of unsatisfiability: the calculus's soundness is not \
+ machine-checked anywhere in lean/, and the replayer in \
+ src/tstp.rs is ordinary unverified Rust",
+ _ => "unrecognised verdict",
+ }
+}
+
+/// A node of the derivation as the checker sees it.
+struct Node {
+ role: String,
+ /// `None` for a synthetic node standing for an inline inference record,
+ /// which carries a rule and parents but no formula of its own.
+ formula: Option<Formula>,
+ source: Option<Source>,
+ rule: Option<String>,
+ parents: Vec<String>,
+}
+
+/// The reasons a step is not replayed, written once so a report cannot end up
+/// with two phrasings for the same fact.
+const WHY_NO_CHECK: &str =
+ "this checker implements no replay for this rule. Clausification, Skolemisation, AVATAR \
+ splitting and every SAT-solver step are in this class";
+const WHY_NOT_CLAUSE: &str =
+ "a premise or the conclusion is not clause-shaped, so the clause-level replay does not \
+ apply to this occurrence";
+const WHY_NO_FORMULA: &str =
+ "a premise is an inline inference record and carries no formula of its own, so neither it \
+ nor the step it feeds can be replayed";
+const WHY_ARITY: &str = "the rule was given a number of premises this replay does not handle";
+const WHY_BUDGET: &str = "the clause is larger than the variant check's backtracking budget";
+
+/// Read a problem and a prover's output, and say exactly what checks out.
+pub fn check(problem_text: &str, proof_text: &str) -> Report {
+ let mut r = Report {
+ szs_status: szs_status(proof_text),
+ problem_error: None,
+ derivation_error: None,
+ problem_formulas: 0,
+ problem_digest: String::new(),
+ derivation_nodes: 0,
+ derivation_wellformed: false,
+ wellformedness_failures: Vec::new(),
+ root_is_false: false,
+ nodes_outside_the_refutation: 0,
+ leaves_total: 0,
+ leaves_match_problem: false,
+ leaf_match_levels: BTreeMap::new(),
+ leaf_failures: Vec::new(),
+ leaves_introduced: Vec::new(),
+ conjecture_in_problem: None,
+ conjecture_used: false,
+ conjecture_negation_checked: false,
+ what_was_refuted: None,
+ steps_total: 0,
+ steps_checked: 0,
+ steps_checked_by_rule: BTreeMap::new(),
+ steps_unchecked: Vec::new(),
+ steps_not_reconstructed: Vec::new(),
+ verdict: "problem_unparsed",
+ };
+
+ // ── The problem ────────────────────────────────────────────────────────
+ let problem = match parse_problem(problem_text) {
+ Ok(p) => p,
+ Err(e) => {
+ r.problem_error = Some(e);
+ return r;
+ }
+ };
+ let mut by_name: HashMap<&str, &Annotated> = HashMap::new();
+ let mut canonical = String::new();
+ for a in &problem {
+ by_name.insert(a.name.as_str(), a);
+ let _ = writeln!(canonical, "{}\t{}", a.role, a.formula.render());
+ if a.role == "conjecture" {
+ r.conjecture_in_problem = Some(a.name.clone());
+ }
+ }
+ r.problem_formulas = problem.len();
+ r.problem_digest =
+ crate::tptp::checkfmt::hex16(crate::tptp::checkfmt::fnv1a64(canonical.trim_end()));
+
+ // ── The derivation ─────────────────────────────────────────────────────
+ let annotated = match parse_derivation(proof_text) {
+ Ok(a) => a,
+ Err(e) => {
+ r.derivation_error = Some(e);
+ r.verdict = "derivation_unparsed";
+ return r;
+ }
+ };
+ if annotated.is_empty() {
+ r.derivation_error = Some(
+ "the output carries no annotated formula. A prover asked for a verdict and not for \
+ a proof prints one line and no derivation; run it with vampire `--proof tptp` or \
+ eprover `--proof-object`"
+ .into(),
+ );
+ r.verdict = "derivation_unparsed";
+ return r;
+ }
+
+ let mut nodes: HashMap<String, Node> = HashMap::new();
+ let mut order: Vec<String> = Vec::new();
+ let mut duplicates: Vec<String> = Vec::new();
+ for a in &annotated {
+ let mut parents = Vec::new();
+ let mut rule = None;
+ if let Some(Source::Inference(inf)) = &a.source {
+ rule = Some(inf.rule.clone());
+ flatten_parents(&a.name, &inf.parents, &mut parents, &mut nodes, &mut order);
+ }
+ if nodes.contains_key(&a.name) {
+ duplicates.push(a.name.clone());
+ continue;
+ }
+ order.push(a.name.clone());
+ nodes.insert(
+ a.name.clone(),
+ Node {
+ role: a.role.clone(),
+ formula: Some(a.formula.clone()),
+ source: a.source.clone(),
+ rule,
+ parents,
+ },
+ );
+ }
+ r.derivation_nodes = nodes.len();
+
+ // Every parent resolves.
+ let mut dangling: Vec<String> = Vec::new();
+ for name in &order {
+ for p in &nodes[name].parents {
+ if !nodes.contains_key(p) {
+ dangling.push(format!("{name} cites {p}, which is not in the derivation"));
+ }
+ }
+ }
+ // The parent relation is acyclic.
+ let cycle = find_cycle(&order, &nodes);
+
+ for d in &duplicates {
+ r.wellformedness_failures
+ .push(format!("the name {d} is used by more than one annotated formula, so a \
+ parent reference to it is ambiguous"));
+ }
+ r.wellformedness_failures.extend(dangling.iter().cloned());
+ if let Some(c) = &cycle {
+ r.wellformedness_failures
+ .push(format!("the parent relation has a cycle: {}", c.join(" -> ")));
+ }
+ r.derivation_wellformed = r.wellformedness_failures.is_empty();
+
+ // ── The root, and what is reachable from it ────────────────────────────
+ let cited: HashSet<&str> =
+ order.iter().flat_map(|n| nodes[n].parents.iter().map(String::as_str)).collect();
+ let false_roots: Vec<String> = order
+ .iter()
+ .filter(|n| !cited.contains(n.as_str()))
+ .filter(|n| nodes[*n].formula.as_ref() == Some(&Formula::False))
+ .cloned()
+ .collect();
+ r.root_is_false = !false_roots.is_empty();
+
+ let mut reachable: BTreeSet<String> = BTreeSet::new();
+ if r.derivation_wellformed {
+ let mut stack: Vec<String> = false_roots.clone();
+ while let Some(n) = stack.pop() {
+ if !reachable.insert(n.clone()) {
+ continue;
+ }
+ for p in &nodes[&n].parents {
+ stack.push(p.clone());
+ }
+ }
+ }
+ r.nodes_outside_the_refutation = nodes.len().saturating_sub(reachable.len());
+
+ if !r.root_is_false {
+ r.verdict = if r.derivation_wellformed { "no_refutation_offered" } else { "derivation_rejected" };
+ return r;
+ }
+ if !r.derivation_wellformed {
+ r.verdict = "derivation_rejected";
+ return r;
+ }
+
+ // ── Leaves ─────────────────────────────────────────────────────────────
+ let mut leaves_ok = true;
+ // Keyed on the rule AND the reason, not on the rule alone. Two
+ // occurrences of `rw` can be unchecked for different reasons, and a map
+ // that kept only the first reason would report the other occurrences under
+ // a sentence that is not true of them.
+ let mut unchecked: BTreeMap<(String, &'static str), usize> = BTreeMap::new();
+ for name in &reachable {
+ let node = &nodes[name];
+ if node.rule.is_some() {
+ continue;
+ }
+ r.leaves_total += 1;
+ match &node.source {
+ Some(Source::Introduced { rule, detail }) => {
+ r.leaves_introduced.push(Introduced {
+ node: name.clone(),
+ rule: rule.clone(),
+ detail: detail.clone(),
+ });
+ *unchecked
+ .entry((
+ format!("introduced:{rule}"),
+ "the prover invented this formula and its symbols. It is not from the \
+ problem, and no conservativity argument is checked here",
+ ))
+ .or_insert(0) += 1;
+ }
+ Some(Source::File { name: Some(orig), .. }) => {
+ check_leaf(name, orig, node, &by_name, &mut r, &mut leaves_ok);
+ }
+ _ => {
+ // No usable name. Fall back to matching the formula itself
+ // against the problem, and say that is what happened.
+ let f = node.formula.as_ref();
+ let hit = f.and_then(|f| {
+ problem.iter().find_map(|a| leaf_match(f, &a.formula).map(|m| (a, m)))
+ });
+ match hit {
+ Some((a, m)) => {
+ *r.leaf_match_levels.entry(m.name()).or_insert(0) += 1;
+ if a.role == "conjecture" {
+ r.conjecture_used = true;
+ }
+ }
+ None => {
+ leaves_ok = false;
+ r.leaf_failures.push(LeafFailure {
+ node: name.clone(),
+ why: format!(
+ "the leaf carries no `file(…)` provenance this reader \
+ understands and its formula `{}` is not a formula of the \
+ problem",
+ f.map(Formula::render).unwrap_or_else(|| "?".into())
+ ),
+ });
+ }
+ }
+ }
+ }
+ }
+ r.leaves_match_problem = leaves_ok;
+
+ // ── Steps ──────────────────────────────────────────────────────────────
+ for name in &reachable {
+ let node = &nodes[name];
+ let Some(rule) = node.rule.clone() else { continue };
+ r.steps_total += 1;
+ match check_step(name, &rule, node, &nodes, &by_name, &mut r) {
+ StepResult::Checked(kind) => {
+ r.steps_checked += 1;
+ *r.steps_checked_by_rule.entry(kind.to_string()).or_insert(0) += 1;
+ }
+ StepResult::Unchecked(why) => {
+ *unchecked.entry((rule.clone(), why)).or_insert(0) += 1;
+ }
+ StepResult::NotReconstructed(nr) => r.steps_not_reconstructed.push(nr),
+ }
+ }
+ r.steps_unchecked = unchecked
+ .into_iter()
+ .map(|((rule, why), count)| Unchecked { rule, count, why: why.to_string() })
+ .collect();
+
+ // ── The verdict ────────────────────────────────────────────────────────
+ r.what_was_refuted = Some(if r.conjecture_used {
+ "axioms_and_the_negated_conjecture"
+ } else {
+ "axioms_alone"
+ });
+ r.verdict = if !r.leaves_match_problem {
+ "derivation_rejected"
+ } else if !r.steps_not_reconstructed.is_empty() {
+ "refutation_step_not_reconstructed"
+ } else if r.steps_unchecked.is_empty() && r.steps_checked == r.steps_total {
+ "refutation_fully_replayed"
+ } else if r.steps_checked > 0 {
+ "refutation_partially_replayed"
+ } else {
+ "refutation_structure_checked"
+ };
+ r
+}
+
+/// Give every inline inference record a node of its own, so that its parent
+/// references are still resolved and its rule is still counted.
+fn flatten_parents(
+ owner: &str,
+ parents: &[Parent],
+ out: &mut Vec<String>,
+ nodes: &mut HashMap<String, Node>,
+ order: &mut Vec<String>,
+) {
+ for (i, p) in parents.iter().enumerate() {
+ match p {
+ Parent::Named(n) => out.push(n.clone()),
+ Parent::Other(o) => {
+ let name = format!("{owner}::other{i}");
+ order.push(name.clone());
+ nodes.insert(
+ name.clone(),
+ Node {
+ role: "plain".into(),
+ formula: None,
+ source: Some(Source::Other(o.clone())),
+ rule: Some(format!("unreadable_parent:{o}")),
+ parents: Vec::new(),
+ },
+ );
+ out.push(name);
+ }
+ Parent::Inline(inf) => {
+ let name = format!("{owner}::inline{i}");
+ let mut inner = Vec::new();
+ flatten_parents(&name, &inf.parents, &mut inner, nodes, order);
+ order.push(name.clone());
+ nodes.insert(
+ name.clone(),
+ Node {
+ role: "plain".into(),
+ formula: None,
+ source: None,
+ rule: Some(inf.rule.clone()),
+ parents: inner,
+ },
+ );
+ out.push(name);
+ }
+ }
+ }
+}
+
+fn find_cycle(order: &[String], nodes: &HashMap<String, Node>) -> Option<Vec<String>> {
+ #[derive(Clone, Copy, PartialEq)]
+ enum Mark {
+ White,
+ Grey,
+ Black,
+ }
+ let mut mark: HashMap<String, Mark> =
+ order.iter().map(|n| (n.clone(), Mark::White)).collect();
+ let mut path: Vec<String> = Vec::new();
+ // Iterative depth-first search. A derivation from a real prover can be
+ // thousands of nodes deep and a recursive walk would overflow the stack on
+ // the one input that matters.
+ for start in order {
+ if mark.get(start) != Some(&Mark::White) {
+ continue;
+ }
+ let mut stack: Vec<(String, usize)> = vec![(start.clone(), 0)];
+ mark.insert(start.clone(), Mark::Grey);
+ path.push(start.clone());
+ while let Some((node, idx)) = stack.pop() {
+ let parents = nodes.get(&node).map(|n| n.parents.clone()).unwrap_or_default();
+ if idx < parents.len() {
+ stack.push((node, idx + 1));
+ let child = parents[idx].clone();
+ match mark.get(&child) {
+ Some(Mark::Grey) => {
+ let mut c = path.clone();
+ c.push(child);
+ return Some(c);
+ }
+ Some(Mark::White) => {
+ mark.insert(child.clone(), Mark::Grey);
+ path.push(child.clone());
+ stack.push((child, 0));
+ }
+ _ => {}
+ }
+ } else {
+ mark.insert(node, Mark::Black);
+ path.pop();
+ }
+ }
+ }
+ None
+}
+
+fn check_leaf(
+ name: &str,
+ orig: &str,
+ node: &Node,
+ by_name: &HashMap<&str, &Annotated>,
+ r: &mut Report,
+ leaves_ok: &mut bool,
+) {
+ let Some(problem_formula) = by_name.get(orig) else {
+ *leaves_ok = false;
+ r.leaf_failures.push(LeafFailure {
+ node: name.to_string(),
+ why: format!(
+ "the leaf says it came from `{orig}` in the problem file, and the problem has \
+ no formula of that name. The prover was given a different file"
+ ),
+ });
+ return;
+ };
+ let Some(f) = &node.formula else {
+ *leaves_ok = false;
+ r.leaf_failures.push(LeafFailure {
+ node: name.to_string(),
+ why: "the leaf carries no formula".into(),
+ });
+ return;
+ };
+ match leaf_match(f, &problem_formula.formula) {
+ None => {
+ *leaves_ok = false;
+ r.leaf_failures.push(LeafFailure {
+ node: name.to_string(),
+ why: format!(
+ "the leaf says it is `{orig}` from the problem, and it is a different \
+ formula. Problem: {}. Derivation: {}",
+ problem_formula.formula.render(),
+ f.render()
+ ),
+ });
+ }
+ Some(m) => {
+ if node.role != problem_formula.role {
+ // A conjecture presented as an axiom would make the refutation
+ // prove nothing about entailment, so this is a rejection and
+ // not a note.
+ *leaves_ok = false;
+ r.leaf_failures.push(LeafFailure {
+ node: name.to_string(),
+ why: format!(
+ "the leaf is `{orig}` from the problem with role `{}`, and the problem \
+ gives it role `{}`. A conjecture used as an axiom refutes nothing",
+ node.role, problem_formula.role
+ ),
+ });
+ return;
+ }
+ *r.leaf_match_levels.entry(m.name()).or_insert(0) += 1;
+ if problem_formula.role == "conjecture" {
+ r.conjecture_used = true;
+ }
+ }
+ }
+}
+
+enum StepResult {
+ Checked(&'static str),
+ Unchecked(&'static str),
+ NotReconstructed(NotReconstructed),
+}
+
+/// The rules this module replays, and the shape of each replay.
+///
+/// Anything not named here is unchecked and is reported as unchecked. The list
+/// is short on purpose: a rule is in it only when the check is exact.
+fn check_step(
+ name: &str,
+ rule: &str,
+ node: &Node,
+ nodes: &HashMap<String, Node>,
+ by_name: &HashMap<&str, &Annotated>,
+ r: &mut Report,
+) -> StepResult {
+ let Some(child) = &node.formula else {
+ return StepResult::Unchecked(WHY_NO_FORMULA);
+ };
+ let premises: Vec<Option<&Formula>> =
+ node.parents.iter().map(|p| nodes[p].formula.as_ref()).collect();
+
+ match rule {
+ // The negation of the conjecture. Checked exactly, and it is the step
+ // that makes the refutation say something about ENTAILMENT rather than
+ // about the axioms alone.
+ "negated_conjecture" | "assume_negation" => {
+ if premises.len() != 1 {
+ return StepResult::Unchecked(WHY_ARITY);
+ }
+ let Some(parent) = premises[0] else {
+ return StepResult::Unchecked(WHY_NO_FORMULA);
+ };
+ let negated = Formula::Not(Box::new(parent.clone()));
+ if leaf_match(child, &negated).is_some() {
+ let parent_name = &node.parents[0];
+ let is_conjecture = nodes
+ .get(parent_name)
+ .is_some_and(|p| p.role == "conjecture")
+ || by_name
+ .get(parent_name.as_str())
+ .is_some_and(|a| a.role == "conjecture");
+ if is_conjecture {
+ r.conjecture_negation_checked = true;
+ }
+ StepResult::Checked("negation_of_the_conjecture")
+ } else {
+ StepResult::NotReconstructed(NotReconstructed {
+ node: name.to_string(),
+ rule: rule.to_string(),
+ expected: negated.render(),
+ found: child.render(),
+ })
+ }
+ }
+
+ // Binary resolution, and the simplification rules whose conclusion IS
+ // the binary resolvent. See the module docstring for why subsumption
+ // resolution is exactly this check and not a weaker one.
+ "resolution"
+ | "subsumption_resolution"
+ | "forward_subsumption_resolution"
+ | "backward_subsumption_resolution" => {
+ if premises.len() != 2 {
+ return StepResult::Unchecked(WHY_ARITY);
+ }
+ let (Some(p0), Some(p1)) = (premises[0], premises[1]) else {
+ return StepResult::Unchecked(WHY_NO_FORMULA);
+ };
+ let (Ok(a), Ok(b), Ok(c)) = (clause_view(p0), clause_view(p1), clause_view(child))
+ else {
+ return StepResult::Unchecked(WHY_NOT_CLAUSE);
+ };
+ match resolvent_matches(&a, &b, &c) {
+ Ok(true) => StepResult::Checked("binary_resolvent"),
+ Ok(false) => StepResult::NotReconstructed(NotReconstructed {
+ node: name.to_string(),
+ rule: rule.to_string(),
+ expected: format!(
+ "a binary resolvent of `{}` and `{}`",
+ a.render(),
+ b.render()
+ ),
+ found: c.render(),
+ }),
+ Err(_) => StepResult::Unchecked(WHY_BUDGET),
+ }
+ }
+
+ // Factoring: two literals of one premise unified and merged.
+ "factoring" | "factor" => one_premise(name, rule, &premises, child, |p, c| {
+ for i in 0..p.lits.len() {
+ for j in (i + 1)..p.lits.len() {
+ if p.lits[i].positive != p.lits[j].positive {
+ continue;
+ }
+ if let Some(s) = unify_atoms(&p.lits[i].atom, &p.lits[j].atom) {
+ let cand = Clause::normalise(apply_clause(&p.lits, &s), p.tautology);
+ if variant(&cand, c)? {
+ return Ok(true);
+ }
+ }
+ }
+ }
+ Ok(false)
+ })
+ .map_checked("factor"),
+
+ // The clause is the same set of literals as its premise. Covers
+ // duplicate literal removal and Vampire's `flattening`, which
+ // reassociates a disjunction and changes nothing else.
+ "duplicate_literal_removal" | "flattening" => {
+ one_premise(name, rule, &premises, child, variant).map_checked("same_clause")
+ }
+
+ // Literals `s != s` with syntactically identical sides, deleted.
+ "trivial_inequality_removal" => one_premise(name, rule, &premises, child, |p, c| {
+ let kept: Vec<Lit> = p
+ .lits
+ .iter()
+ .filter(|l| !matches!(&l.atom, Atom::Eq(x, y) if !l.positive && x == y))
+ .cloned()
+ .collect();
+ variant(&Clause::normalise(kept, p.tautology), c)
+ })
+ .map_checked("trivial_inequality_removed"),
+
+ // `C ∨ s ≠ t` with σ = mgu(s, t) gives `Cσ`.
+ "equality_resolution" => one_premise(name, rule, &premises, child, |p, c| {
+ for (i, l) in p.lits.iter().enumerate() {
+ if l.positive {
+ continue;
+ }
+ let Atom::Eq(x, y) = &l.atom else { continue };
+ let mut s = Subst::new();
+ if !unify(x, y, &mut s) {
+ continue;
+ }
+ let rest: Vec<Lit> =
+ p.lits.iter().enumerate().filter(|(k, _)| *k != i).map(|(_, l)| l.clone()).collect();
+ let cand = Clause::normalise(apply_clause(&rest, &s), p.tautology);
+ if variant(&cand, c)? {
+ return Ok(true);
+ }
+ }
+ Ok(false)
+ })
+ .map_checked("equality_resolved"),
+
+ _ => StepResult::Unchecked(WHY_NO_CHECK),
+ }
+}
+
+/// The outcome of a one-premise replay before it is given a name.
+enum OneOf {
+ Ok,
+ No(NotReconstructed),
+ Skip(&'static str),
+}
+
+impl OneOf {
+ fn map_checked(self, kind: &'static str) -> StepResult {
+ match self {
+ OneOf::Ok => StepResult::Checked(kind),
+ OneOf::Skip(why) => StepResult::Unchecked(why),
+ OneOf::No(nr) => StepResult::NotReconstructed(nr),
+ }
+ }
+}
+
+fn one_premise(
+ name: &str,
+ rule: &str,
+ premises: &[Option<&Formula>],
+ child: &Formula,
+ f: impl Fn(&Clause, &Clause) -> Result<bool, &'static str>,
+) -> OneOf {
+ if premises.len() != 1 {
+ return OneOf::Skip(WHY_ARITY);
+ }
+ let Some(p) = premises[0] else {
+ return OneOf::Skip(WHY_NO_FORMULA);
+ };
+ let (Ok(pc), Ok(cc)) = (clause_view(p), clause_view(child)) else {
+ return OneOf::Skip(WHY_NOT_CLAUSE);
+ };
+ match f(&pc, &cc) {
+ Ok(true) => OneOf::Ok,
+ Ok(false) => OneOf::No(NotReconstructed {
+ node: name.to_string(),
+ rule: rule.to_string(),
+ expected: format!("a `{rule}` conclusion of `{}`", pc.render()),
+ found: cc.render(),
+ }),
+ Err(why) => OneOf::Skip(why),
+ }
+}
+
+/// Is `c` a binary resolvent of `a` and `b`?
+///
+/// Every complementary literal pair is tried, on standardised-apart copies.
+/// The resolvent is taken as a SET, which is what makes the check agree with
+/// a prover that factors the resolvent on the way out.
+fn resolvent_matches(a: &Clause, b: &Clause, c: &Clause) -> Result<bool, &'static str> {
+ let a = rename_apart(a, "_a");
+ let b = rename_apart(b, "_b");
+ for (i, la) in a.lits.iter().enumerate() {
+ for (j, lb) in b.lits.iter().enumerate() {
+ if la.positive == lb.positive {
+ continue;
+ }
+ let Some(s) = unify_atoms(&la.atom, &lb.atom) else { continue };
+ let mut rest: Vec<Lit> = Vec::new();
+ rest.extend(a.lits.iter().enumerate().filter(|(k, _)| *k != i).map(|(_, l)| l.clone()));
+ rest.extend(b.lits.iter().enumerate().filter(|(k, _)| *k != j).map(|(_, l)| l.clone()));
+ let cand =
+ Clause::normalise(apply_clause(&rest, &s), a.tautology || b.tautology);
+ if variant(&cand, c)? {
+ return Ok(true);
+ }
+ }
+ }
+ Ok(false)
+}
+
+/// The JSON one check reports, with the vocabulary written out beside the
+/// verdict, the way `fol_solve::outcome_json` does.
+pub fn report_json(r: &Report) -> serde_json::Value {
+ serde_json::json!({
+ "szs_status": r.szs_status,
+ "szs_status_means": "WHAT THE PROVER SAID ABOUT ITSELF. Echoed, untrusted, and used to \
+ decide nothing in this report",
+ "problem_error": r.problem_error,
+ "derivation_error": r.derivation_error,
+ "problem_formulas": r.problem_formulas,
+ "problem_digest": r.problem_digest,
+ "derivation_nodes": r.derivation_nodes,
+ "derivation_wellformed": r.derivation_wellformed,
+ "wellformedness_failures": r.wellformedness_failures,
+ "root_is_false": r.root_is_false,
+ "nodes_outside_the_refutation": r.nodes_outside_the_refutation,
+ "leaves_total": r.leaves_total,
+ "leaves_match_problem": r.leaves_match_problem,
+ "leaf_match_levels": r.leaf_match_levels,
+ "leaf_failures": r.leaf_failures,
+ "leaves_introduced": r.leaves_introduced,
+ "conjecture_in_problem": r.conjecture_in_problem,
+ "conjecture_used": r.conjecture_used,
+ "conjecture_negation_checked": r.conjecture_negation_checked,
+ "what_was_refuted": r.what_was_refuted,
+ "steps_total": r.steps_total,
+ "steps_checked": r.steps_checked,
+ "steps_checked_by_rule": r.steps_checked_by_rule,
+ "steps_unchecked": r.steps_unchecked,
+ "steps_unchecked_total": r.steps_unchecked.iter().map(|u| u.count).sum::<usize>(),
+ "steps_not_reconstructed": r.steps_not_reconstructed,
+ "verdict": r.verdict,
+ "verdict_means": verdict_means(r.verdict),
+ "checked_by": "src/tstp.rs, an UNVERIFIED Rust replayer. lean/ is not involved and no \
+ theorem is cited. A refutation remains an ORACLE OPINION (decision \
+ 0005); what this adds is that the opinion is now about a derivation \
+ somebody can read, over the problem this engine emitted",
+ })
+}
+
+// ── Driving a prover ───────────────────────────────────────────────────────
+
+/// Which prover to run. Both read TPTP FOF and both print TSTP on request.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum Prover {
+ Vampire,
+ Eprover,
+}
+
+impl Prover {
+ pub fn parse(s: &str) -> anyhow::Result<Prover> {
+ match s.to_ascii_lowercase().as_str() {
+ "vampire" => Ok(Prover::Vampire),
+ "eprover" | "e" => Ok(Prover::Eprover),
+ other => anyhow::bail!("unknown prover {other:?}; expected `vampire` or `eprover`"),
+ }
+ }
+ pub fn name(self) -> &'static str {
+ match self {
+ Prover::Vampire => "vampire",
+ Prover::Eprover => "eprover",
+ }
+ }
+ /// The argument vector, WITH the proof-printing option.
+ ///
+ /// It is not optional here. A run without it returns an SZS word and no
+ /// derivation, which is the state this module exists to leave behind.
+ pub fn argv(self, file: &Path, secs: u32) -> Vec<String> {
+ let f = file.display().to_string();
+ match self {
+ Prover::Vampire => {
+ vec!["--proof".into(), "tptp".into(), "-t".into(), format!("{secs}"), f]
+ }
+ Prover::Eprover => vec![
+ "--auto".into(),
+ "--tptp3-format".into(),
+ "--proof-object".into(),
+ format!("--cpu-limit={secs}"),
+ f,
+ ],
+ }
+ }
+ pub fn install_line(self) -> &'static str {
+ match self {
+ Prover::Vampire => "install Vampire: `brew install vampire` (macOS) or \
+ https://github.com/vprover/vampire",
+ Prover::Eprover => "install E: `brew install eprover` (macOS), \
+ `apt-get install eprover`, or https://github.com/eprover/eprover",
+ }
+ }
+ pub fn available(self) -> bool {
+ which(self.name()).is_some()
+ }
+}
+
+fn which(bin: &str) -> Option<PathBuf> {
+ std::env::var_os("PATH").and_then(|paths| {
+ std::env::split_paths(&paths).find_map(|dir| {
+ let p = dir.join(bin);
+ if p.is_file() { Some(p) } else { None }
+ })
+ })
+}
+
+/// How to run the pipeline.
+#[derive(Clone, Debug)]
+pub struct ProveOptions {
+ pub prover: Prover,
+ /// Seconds per prover invocation.
+ pub timeout_secs: u32,
+}
+
+impl Default for ProveOptions {
+ fn default() -> Self {
+ ProveOptions { prover: Prover::Vampire, timeout_secs: 30 }
+ }
+}
+
+fn run_prover(prover: Prover, file: &Path, out: &Path, secs: u32) -> std::io::Result<String> {
+ let f = std::fs::File::create(out)?;
+ let f2 = f.try_clone()?;
+ let mut cmd = Command::new(prover.name());
+ cmd.args(prover.argv(file, secs))
+ .stdout(Stdio::from(f))
+ .stderr(Stdio::from(f2))
+ .stdin(Stdio::null());
+ let mut child = cmd.spawn()?;
+ let start = Instant::now();
+ let limit = Duration::from_secs((secs as u64).max(1) + 15);
+ loop {
+ match child.try_wait()? {
+ Some(_) => break,
+ None => {
+ if start.elapsed() > limit {
+ let _ = child.kill();
+ let _ = child.wait();
+ break;
+ }
+ std::thread::sleep(Duration::from_millis(20));
+ }
+ }
+ }
+ std::fs::read_to_string(out)
+}
+
+/// One problem: write it, run the prover, check what comes back.
+fn prove_one(
+ problem_text: &str,
+ dir: &Path,
+ stem: &str,
+ opts: &ProveOptions,
+) -> anyhow::Result<Report> {
+ std::fs::create_dir_all(dir)?;
+ let p = dir.join(format!("{stem}.p"));
+ std::fs::write(&p, problem_text)?;
+ let out = dir.join(format!("{stem}.tstp"));
+ let text = run_prover(opts.prover, &p, &out, opts.timeout_secs)?;
+ Ok(check(problem_text, &text))
+}
+
+/// Run the prover over the loaded ontology and, optionally, one problem per
+/// goal, and check every derivation it prints.
+///
+/// The shape is `fol_solve::solve_export`'s, deliberately: the two are the
+/// refutation half and the model half of the same question and a reader should
+/// not have to learn two report formats.
+pub fn prove_export(
+ graph: &std::sync::Arc<crate::graph::GraphStore>,
+ dir: &Path,
+ opts: &ProveOptions,
+ goals: Option<&Path>,
+ goals_skip_columns: usize,
+) -> anyhow::Result<String> {
+ if !opts.prover.available() {
+ return Ok(serde_json::json!({
+ "prover": opts.prover.name(),
+ "skipped": format!(
+ "{} is not on PATH, so no derivation was produced and nothing was checked. \
+ That is the ABSENCE of a second opinion and not agreement with one. {}",
+ opts.prover.name(),
+ opts.prover.install_line()
+ ),
+ })
+ .to_string());
+ }
+ let triples = graph.all_triples()?;
+ let read = crate::tptp::read_graph(triples);
+ std::fs::create_dir_all(dir)?;
+
+ let base = crate::tptp::FolProblem::build(&read.axioms, None)?;
+ let ontology = prove_one(&base.to_tptp(), &dir.join("ontology"), "ontology", opts)?;
+
+ let mut counts: BTreeMap<&'static str, usize> = BTreeMap::new();
+ *counts.entry(ontology.verdict).or_default() += 1;
+ let mut goal_reports = Vec::new();
+ let mut not_asked: Vec<serde_json::Value> = Vec::new();
+
+ if let Some(path) = goals {
+ let text = std::fs::read_to_string(path)?;
+ for (i, line) in text.lines().enumerate() {
+ if line.trim().is_empty() {
+ continue;
+ }
+ let cols: Vec<&str> = line.split('\t').collect();
+ if cols.len() < goals_skip_columns + 3 {
+ not_asked.push(serde_json::json!({
+ "line": line,
+ "why": format!("fewer than {} tab-separated columns", goals_skip_columns + 3),
+ }));
+ continue;
+ }
+ let (s, p, o) = (
+ cols[goals_skip_columns],
+ cols[goals_skip_columns + 1],
+ cols[goals_skip_columns + 2],
+ );
+ let ax = match crate::tptp::triple_as_axiom(&read, s, p, o) {
+ Ok(ax) => ax,
+ Err(why) => {
+ not_asked.push(serde_json::json!({"triple": [s, p, o], "why": why}));
+ continue;
+ }
+ };
+ let gp = crate::tptp::FolProblem::build(&read.axioms, Some(&ax))?;
+ let rep = prove_one(
+ &gp.to_tptp(),
+ &dir.join(format!("goal_{i:05}")),
+ &format!("goal_{i:05}"),
+ opts,
+ )?;
+ *counts.entry(rep.verdict).or_default() += 1;
+ goal_reports.push(serde_json::json!({
+ "triple": [s, p, o],
+ "report": report_json(&rep),
+ }));
+ }
+ }
+
+ let rejected = counts.get("derivation_rejected").copied().unwrap_or(0);
+ let unreconstructed = counts.get("refutation_step_not_reconstructed").copied().unwrap_or(0);
+
+ Ok(serde_json::json!({
+ "prover": opts.prover.name(),
+ "timeout_secs": opts.timeout_secs,
+ "dir": dir.display().to_string(),
+ "ontology": report_json(&ontology),
+ "goals": goal_reports,
+ "not_asked": not_asked,
+ "not_asked_means": "no axiom form in OwlLean/Syntax.lean corresponds to the triple, so \
+ the question could not be put at all. Listed rather than skipped, \
+ so a run cannot report a clean sweep over a subset nobody chose",
+ "verdict_counts": counts,
+ "stop_the_line": rejected + unreconstructed,
+ "stop_the_line_means": "a derivation was REJECTED (a dangling parent, a cycle, or a \
+ leaf that is not a formula of the problem this engine emitted), \
+ or a step whose rule this checker implements did not \
+ reconstruct. The first means the prover refuted something other \
+ than what it was given; the second is either a defect in the \
+ derivation or a gap in this checker and needs a human. The \
+ command exits non-zero on either",
+ "not_certified": "NOTHING HERE IS CERTIFIED. A refutation is an ORACLE OPINION \
+ (decision 0005) and stays one: the calculus's soundness is not \
+ machine-checked anywhere in lean/, and this replayer is ordinary \
+ Rust. What the check adds is that the opinion is now about a \
+ derivation that was read, whose leaves are the formulas this engine \
+ emitted, and some of whose steps were recomputed. The MODEL \
+ direction is the one that can be certified; see decision 0006 and \
+ onto_fol_model",
+ })
+ .to_string())
+}
+
+/// Check a problem file and a recorded prover output, with no store and no
+/// prover run. This is what `tools/fol_differential.py` calls once it already
+/// has both.
+pub fn check_files(problem: &Path, proof: &Path) -> anyhow::Result<String> {
+ let p = std::fs::read_to_string(problem)
+ .map_err(|e| anyhow::anyhow!("cannot read problem {}: {e}", problem.display()))?;
+ let d = std::fs::read_to_string(proof)
+ .map_err(|e| anyhow::anyhow!("cannot read proof {}: {e}", proof.display()))?;
+ let r = check(&p, &d);
+ let mut v = report_json(&r);
+ if let Some(o) = v.as_object_mut() {
+ o.insert("problem".into(), serde_json::json!(problem.display().to_string()));
+ o.insert("proof".into(), serde_json::json!(proof.display().to_string()));
+ o.insert(
+ "stop_the_line".into(),
+ serde_json::json!(usize::from(matches!(
+ r.verdict,
+ "derivation_rejected" | "refutation_step_not_reconstructed"
+ ))),
+ );
+ }
+ Ok(v.to_string())
+}
diff --git a/tests/ci_gate_coverage_test.rs b/tests/ci_gate_coverage_test.rs
index 24ba6034..7c8508c0 100644
--- a/tests/ci_gate_coverage_test.rs
+++ b/tests/ci_gate_coverage_test.rs
@@ -58,6 +58,17 @@ fn this_file() -> &'static str {
///
/// Keep this list in step with the "What is still open" section of `docs/ci-gates.md`.
const NOT_STRICT_ON_PURPOSE: &[(&str, &str)] = &[
+ (
+ "tstp_derivation_test.rs",
+ "34 of its 35 tests read derivations recorded in tests/fixtures/tstp/ and run \
+ everywhere. The 35th, `a_live_vampire_run_agrees_with_the_recorded_one`, wants \
+ vampire on PATH, and no workflow installs a first-order prover, so \
+ OO_REQUIRE_FIXTURES=1 on this file would fail rather than gate. The recorded \
+ half is not a substitute for the live half: it pins the checker against two real \
+ derivations, not against whatever the installed prover does today. A \
+ `brew install vampire` step, or its apt equivalent, is the smallest thing that \
+ would close this and let the file go strict",
+ ),
(
"clinical_test.rs",
"needs data/crosswalks.parquet, a licensed table the repository deliberately does \
diff --git a/tests/fixtures/tstp/eprover-3.2.5.tstp b/tests/fixtures/tstp/eprover-3.2.5.tstp
new file mode 100644
index 00000000..71f16f40
--- /dev/null
+++ b/tests/fixtures/tstp/eprover-3.2.5.tstp
@@ -0,0 +1,60 @@
+# Preprocessing class: FSSSSMSSSSSNFFN.
+# Configuration: G-E--_302_C18_F1_URBAN_RG_S04BN
+# No SInE strategy applied
+# Search class: FHHNF-FFSS00-SFFFFFNN
+# Configuration: SAT001_MinMin_p005000_rr_RG
+# Initializing proof state
+# Scanning for AC axioms
+#
+#cnf(i_0_3, plain, (thing('i:http://example.org/socrates'))).
+##
+#cnf(i_0_8, negated_conjecture, (~'c:http://example.org/Mortal'('i:http://example.org/socrates'))).
+#
+#cnf(i_0_2, plain, (thing(esk1_0))).
+#
+#cnf(i_0_6, plain, ('c:http://example.org/Greek'('i:http://example.org/socrates'))).
+#
+#cnf(i_0_1, plain, (~lit(X1)|~thing(X1))).
+#
+#cnf(i_0_5, plain, ('c:http://example.org/Mortal'(X1)|~'c:http://example.org/Person'(X1)|~thing(X1))).
+#
+#cnf(i_0_4, plain, ('c:http://example.org/Person'(X1)|~'c:http://example.org/Greek'(X1)|~thing(X1))).
+# Presaturation interreduction done
+#
+#cnf(i_0_8, negated_conjecture, (~'c:http://example.org/Mortal'('i:http://example.org/socrates'))).
+#
+#cnf(i_0_1, plain, (~lit(X1)|~thing(X1))).
+#
+#cnf(i_0_3, plain, (thing('i:http://example.org/socrates'))).
+#
+#cnf(i_0_2, plain, (thing(esk1_0))).
+#
+#cnf(i_0_6, plain, ('c:http://example.org/Greek'('i:http://example.org/socrates'))).
+#
+#cnf(i_0_5, plain, ('c:http://example.org/Mortal'(X1)|~'c:http://example.org/Person'(X1)|~thing(X1))).
+#
+#cnf(i_0_9, negated_conjecture, (~'c:http://example.org/Person'('i:http://example.org/socrates'))).
+#
+#cnf(i_0_4, plain, ('c:http://example.org/Person'(X1)|~'c:http://example.org/Greek'(X1)|~thing(X1))).
+
+# Proof found!
+# SZS status Theorem
+# SZS output start CNFRefutation
+fof(goal_classAssertion, conjecture, (thing('i:http://example.org/socrates')&'c:http://example.org/Mortal'('i:http://example.org/socrates')), file('/private/tmp/claude-501/-Users-fabio/87492980-7d66-415a-9b2e-d382a61063d3/scratchpad/wt-tstp/tests/fixtures/tstp/problem.p', goal_classAssertion)).
+fof(ind_typing_1, axiom, thing('i:http://example.org/socrates'), file('/private/tmp/claude-501/-Users-fabio/87492980-7d66-415a-9b2e-d382a61063d3/scratchpad/wt-tstp/tests/fixtures/tstp/problem.p', ind_typing_1)).
+fof(owl_2_subClassOf, axiom, ![X1]:((thing(X1)=>('c:http://example.org/Person'(X1)=>'c:http://example.org/Mortal'(X1)))), file('/private/tmp/claude-501/-Users-fabio/87492980-7d66-415a-9b2e-d382a61063d3/scratchpad/wt-tstp/tests/fixtures/tstp/problem.p', owl_2_subClassOf)).
+fof(owl_1_subClassOf, axiom, ![X1]:((thing(X1)=>('c:http://example.org/Greek'(X1)=>'c:http://example.org/Person'(X1)))), file('/private/tmp/claude-501/-Users-fabio/87492980-7d66-415a-9b2e-d382a61063d3/scratchpad/wt-tstp/tests/fixtures/tstp/problem.p', owl_1_subClassOf)).
+fof(owl_3_classAssertion, axiom, (thing('i:http://example.org/socrates')&'c:http://example.org/Greek'('i:http://example.org/socrates')), file('/private/tmp/claude-501/-Users-fabio/87492980-7d66-415a-9b2e-d382a61063d3/scratchpad/wt-tstp/tests/fixtures/tstp/problem.p', owl_3_classAssertion)).
+fof(c_0_5, negated_conjecture, ~((thing('i:http://example.org/socrates')&'c:http://example.org/Mortal'('i:http://example.org/socrates'))), inference(assume_negation,[status(cth)],[goal_classAssertion])).
+fof(c_0_6, negated_conjecture, (~thing('i:http://example.org/socrates')|~'c:http://example.org/Mortal'('i:http://example.org/socrates')), inference(fof_nnf,[status(thm)],[inference(fof_nnf,[status(thm)],[c_0_5])])).
+cnf(c_0_7, negated_conjecture, (~thing('i:http://example.org/socrates')|~'c:http://example.org/Mortal'('i:http://example.org/socrates')), inference(split_conjunct,[status(thm)],[c_0_6])).
+cnf(c_0_8, plain, (thing('i:http://example.org/socrates')), inference(split_conjunct,[status(thm)],[ind_typing_1])).
+fof(c_0_9, plain, ![X5]:((~thing(X5)|(~'c:http://example.org/Person'(X5)|'c:http://example.org/Mortal'(X5)))), inference(fof_nnf,[status(thm)],[inference(variable_rename,[status(thm)],[inference(fof_nnf,[status(thm)],[owl_2_subClassOf])])])).
+cnf(c_0_10, negated_conjecture, (~'c:http://example.org/Mortal'('i:http://example.org/socrates')), inference(cn,[status(thm)],[inference(rw,[status(thm)],[c_0_7, c_0_8])])).
+cnf(c_0_11, plain, ('c:http://example.org/Mortal'(X1)|~thing(X1)|~'c:http://example.org/Person'(X1)), inference(split_conjunct,[status(thm)],[c_0_9])).
+fof(c_0_12, plain, ![X4]:((~thing(X4)|(~'c:http://example.org/Greek'(X4)|'c:http://example.org/Person'(X4)))), inference(fof_nnf,[status(thm)],[inference(variable_rename,[status(thm)],[inference(fof_nnf,[status(thm)],[owl_1_subClassOf])])])).
+cnf(c_0_13, negated_conjecture, (~'c:http://example.org/Person'('i:http://example.org/socrates')), inference(cn,[status(thm)],[inference(rw,[status(thm)],[inference(spm,[status(thm)],[c_0_10, c_0_11]), c_0_8])])).
+cnf(c_0_14, plain, ('c:http://example.org/Person'(X1)|~thing(X1)|~'c:http://example.org/Greek'(X1)), inference(split_conjunct,[status(thm)],[c_0_12])).
+cnf(c_0_15, plain, ('c:http://example.org/Greek'('i:http://example.org/socrates')), inference(split_conjunct,[status(thm)],[owl_3_classAssertion])).
+cnf(c_0_16, negated_conjecture, ($false), inference(cn,[status(thm)],[inference(rw,[status(thm)],[inference(rw,[status(thm)],[inference(spm,[status(thm)],[c_0_13, c_0_14]), c_0_15]), c_0_8])]), ['proof']).
+# SZS output end CNFRefutation
diff --git a/tests/fixtures/tstp/problem.p b/tests/fixtures/tstp/problem.p
new file mode 100644
index 00000000..dbac84f5
--- /dev/null
+++ b/tests/fixtures/tstp/problem.p
@@ -0,0 +1,24 @@
+% Generated by open-ontologies `fol` export.
+%
+% The translation is the one owl-lean's machine-checked adequacy theorem
+% `OwlLean.adequacy` is about (axioms: propext, Classical.choice, Quot.sound;
+% no sorry, no Mathlib). The CORRESPONDENCE between this emitter and that Lean
+% translation is PINNED BY TESTS AND IS NOT ITSELF PROVED.
+%
+% `thing` and `lit` are the soft-typing predicates: object positions are
+% relativised by `thing`, data positions by `lit`. The first two axioms are
+% owl-lean's `background`; the `ind_typing_*` axioms are its `indAxioms`, and
+% omitting them refutes adequacy outright (OwlLean.Refutations).
+%
+% A prover's verdict on this file is an ORACLE OPINION, not a certificate.
+% Checking a superposition refutation needs a verified first-order calculus
+% with unification, which does not exist in core Lean. Disagreement between
+% a prover and this engine is a bug in one of the two and nothing here says
+% which.
+fof(background_1, axiom, ! [X0] : (~ (thing(X0) & lit(X0)))).
+fof(background_2, axiom, ? [X0] : thing(X0)).
+fof(ind_typing_1, axiom, thing('i:http://example.org/socrates')).
+fof(owl_1_subClassOf, axiom, ! [X0] : (thing(X0) => ('c:http://example.org/Greek'(X0) => 'c:http://example.org/Person'(X0)))).
+fof(owl_2_subClassOf, axiom, ! [X0] : (thing(X0) => ('c:http://example.org/Person'(X0) => 'c:http://example.org/Mortal'(X0)))).
+fof(owl_3_classAssertion, axiom, (thing('i:http://example.org/socrates') & 'c:http://example.org/Greek'('i:http://example.org/socrates'))).
+fof(goal_classAssertion, conjecture, (thing('i:http://example.org/socrates') & 'c:http://example.org/Mortal'('i:http://example.org/socrates'))).
diff --git a/tests/fixtures/tstp/vampire-5.1.0.tstp b/tests/fixtures/tstp/vampire-5.1.0.tstp
new file mode 100644
index 00000000..0d7fd0ab
--- /dev/null
+++ b/tests/fixtures/tstp/vampire-5.1.0.tstp
@@ -0,0 +1,102 @@
+% Running in auto input_syntax mode. Trying TPTP
+% Refutation found. Thanks to Tanya!
+% SZS status Theorem for problem
+% SZS output start Proof for problem
+fof(f4,axiom,(
+ ! [X0] : (thing(X0) => ('c:http://example.org/Greek'(X0) => 'c:http://example.org/Person'(X0)))),
+ file('/private/tmp/claude-501/-Users-fabio/87492980-7d66-415a-9b2e-d382a61063d3/scratchpad/wt-tstp/tests/fixtures/tstp/problem.p','owl_1_subClassOf')).
+fof(f5,axiom,(
+ ! [X0] : (thing(X0) => ('c:http://example.org/Person'(X0) => 'c:http://example.org/Mortal'(X0)))),
+ file('/private/tmp/claude-501/-Users-fabio/87492980-7d66-415a-9b2e-d382a61063d3/scratchpad/wt-tstp/tests/fixtures/tstp/problem.p','owl_2_subClassOf')).
+fof(f6,axiom,(
+ thing('i:http://example.org/socrates') & 'c:http://example.org/Greek'('i:http://example.org/socrates')),
+ file('/private/tmp/claude-501/-Users-fabio/87492980-7d66-415a-9b2e-d382a61063d3/scratchpad/wt-tstp/tests/fixtures/tstp/problem.p','owl_3_classAssertion')).
+fof(f7,conjecture,(
+ thing('i:http://example.org/socrates') & 'c:http://example.org/Mortal'('i:http://example.org/socrates')),
+ file('/private/tmp/claude-501/-Users-fabio/87492980-7d66-415a-9b2e-d382a61063d3/scratchpad/wt-tstp/tests/fixtures/tstp/problem.p','goal_classAssertion')).
+fof(f8,negated_conjecture,(
+ ~(thing('i:http://example.org/socrates') & 'c:http://example.org/Mortal'('i:http://example.org/socrates'))),
+ inference(negated_conjecture,[status(cth)],[f7])).
+fof(f10,plain,(
+ ! [X0] : (('c:http://example.org/Person'(X0) | ~'c:http://example.org/Greek'(X0)) | ~thing(X0))),
+ inference(ennf_transformation,[],[f4])).
+fof(f11,plain,(
+ ! [X0] : ('c:http://example.org/Person'(X0) | ~'c:http://example.org/Greek'(X0) | ~thing(X0))),
+ inference(flattening,[],[f10])).
+fof(f12,plain,(
+ ! [X0] : (('c:http://example.org/Mortal'(X0) | ~'c:http://example.org/Person'(X0)) | ~thing(X0))),
+ inference(ennf_transformation,[],[f5])).
+fof(f13,plain,(
+ ! [X0] : ('c:http://example.org/Mortal'(X0) | ~'c:http://example.org/Person'(X0) | ~thing(X0))),
+ inference(flattening,[],[f12])).
+fof(f14,plain,(
+ ~thing('i:http://example.org/socrates') | ~'c:http://example.org/Mortal'('i:http://example.org/socrates')),
+ inference(ennf_transformation,[],[f8])).
+fof(f18,plain,(
+ ( ! [X0] : (~'c:http://example.org/Greek'(X0) | 'c:http://example.org/Person'(X0) | ~thing(X0)) )),
+ inference(cnf_transformation,[],[f11])).
+fof(f19,plain,(
+ ( ! [X0] : (~'c:http://example.org/Person'(X0) | 'c:http://example.org/Mortal'(X0) | ~thing(X0)) )),
+ inference(cnf_transformation,[],[f13])).
+fof(f20,plain,(
+ 'c:http://example.org/Greek'('i:http://example.org/socrates')),
+ inference(cnf_transformation,[],[f6])).
+fof(f21,plain,(
+ thing('i:http://example.org/socrates')),
+ inference(cnf_transformation,[],[f6])).
+fof(f22,plain,(
+ ~thing('i:http://example.org/socrates') | ~'c:http://example.org/Mortal'('i:http://example.org/socrates')),
+ inference(cnf_transformation,[],[f14])).
+fof(f24,definition,(
+ spl1_1 <=> 'c:http://example.org/Mortal'('i:http://example.org/socrates')),
+ introduced(definition,[new_symbols(definition,[spl1_1])],[avatar_definition])).
+fof(f26,plain,(
+ ~'c:http://example.org/Mortal'('i:http://example.org/socrates') | spl1_1),
+ inference(avatar_component_clause,[],[f24])).
+fof(f28,definition,(
+ spl1_2 <=> thing('i:http://example.org/socrates')),
+ introduced(definition,[new_symbols(definition,[spl1_2])],[avatar_definition])).
+fof(f29,plain,(
+ thing('i:http://example.org/socrates') | ~spl1_2),
+ inference(avatar_component_clause,[],[f28])).
+fof(f31,plain,(
+ ~spl1_1 | ~spl1_2),
+ inference(avatar_split_clause,[],[f22,f28,f24])).
+fof(f32,plain,(
+ spl1_2),
+ inference(avatar_split_clause,[],[f21,f28])).
+fof(f33,plain,(
+ 'c:http://example.org/Person'('i:http://example.org/socrates') | ~thing('i:http://example.org/socrates')),
+ inference(resolution,[],[f18,f20])).
+fof(f34,plain,(
+ 'c:http://example.org/Person'('i:http://example.org/socrates') | ~spl1_2),
+ inference(forward_subsumption_resolution,[],[f33,f29])).
+fof(f35,plain,(
+ 'c:http://example.org/Mortal'('i:http://example.org/socrates') | ~thing('i:http://example.org/socrates') | ~spl1_2),
+ inference(resolution,[],[f34,f19])).
+fof(f36,plain,(
+ ~thing('i:http://example.org/socrates') | (spl1_1 | ~spl1_2)),
+ inference(forward_subsumption_resolution,[],[f35,f26])).
+fof(f37,plain,(
+ $false | (spl1_1 | ~spl1_2)),
+ inference(forward_subsumption_resolution,[],[f36,f29])).
+fof(f38,plain,(
+ spl1_1 | ~spl1_2),
+ inference(avatar_contradiction_clause,[],[f37])).
+cnf(s1, plain, ~spl1_1 | ~spl1_2, inference(sat_conversion,[],[f31])).
+cnf(s2, plain, spl1_2, inference(sat_conversion,[],[f32])).
+cnf(s4, plain, spl1_1 | ~spl1_2, inference(sat_conversion,[],[f38])).
+cnf(s5, plain, spl1_1, inference(rat,[],[s4,s2])).
+cnf(s6, plain, $false, inference(rat,[],[s1,s2,s5])).
+fof(f39,plain,(
+ $false),
+ inference(avatar_sat_refutation,[],[s6])).
+% SZS output end Proof for problem
+% ------------------------------
+% Version: Vampire 5.1.0 (Release build, commit )
+% CaDiCaL version: 2.1.3
+% Termination reason: Refutation
+% Time elapsed: 0.001 s
+% Peak memory usage: 6 MB
+% ------------------------------
+% ------------------------------
diff --git a/tests/tstp_cli_test.rs b/tests/tstp_cli_test.rs
new file mode 100644
index 00000000..b63286f8
--- /dev/null
+++ b/tests/tstp_cli_test.rs
@@ -0,0 +1,125 @@
+//! `fol-prove` through the real binary, because an exit code that does not
+//! bite is the same defect as a check that does not run.
+//!
+//! Decision 0006 records this exact failure happening in this repository: the
+//! certifying pipeline exited non-zero on a stop-the-line only on the direct
+//! CLI path, while `batch`, the mode every tool in `tools/` uses, decided the
+//! exit code from the presence of an `"error"` key and a stop-the-line is not
+//! an error. So in the one place the gate had to bite, it did not. These tests
+//! drive the process and read the exit status rather than calling the library,
+//! for that reason and no other.
+
+use std::path::{Path, PathBuf};
+use std::process::Command;
+
+fn fixture(name: &str) -> String {
+ let p = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/tstp").join(name);
+ std::fs::read_to_string(&p).unwrap_or_else(|e| panic!("{}: {e}", p.display()))
+}
+
+fn scratch(name: &str) -> PathBuf {
+ let d = std::env::temp_dir().join(format!("oo-tstp-cli-{name}-{}", std::process::id()));
+ let _ = std::fs::remove_dir_all(&d);
+ std::fs::create_dir_all(&d).expect("scratch dir");
+ d
+}
+
+/// Run `fol-prove --problem … --proof …` and return (exit code, stdout).
+fn check_pair(dir: &Path, problem: &str, proof: &str) -> (i32, String) {
+ let p = dir.join("problem.p");
+ let d = dir.join("proof.tstp");
+ std::fs::write(&p, problem).unwrap();
+ std::fs::write(&d, proof).unwrap();
+ let out = Command::new(env!("CARGO_BIN_EXE_open-ontologies"))
+ .args(["--no-connect", "fol-prove", "--problem"])
+ .arg(&p)
+ .arg("--proof")
+ .arg(&d)
+ .output()
+ .expect("the binary runs");
+ (out.status.code().unwrap_or(-1), String::from_utf8_lossy(&out.stdout).to_string())
+}
+
+/// The positive control. Without it every test below would pass on a command
+/// that exited 1 on everything.
+#[test]
+fn a_genuine_pair_exits_zero_and_reports_what_it_replayed() {
+ let d = scratch("good");
+ let (code, out) = check_pair(&d, &fixture("problem.p"), &fixture("vampire-5.1.0.tstp"));
+ let _ = std::fs::remove_dir_all(&d);
+ assert_eq!(code, 0, "{out}");
+ assert!(out.contains("\"verdict\":\"refutation_partially_replayed\""), "{out}");
+ assert!(out.contains("\"leaves_match_problem\":true"), "{out}");
+ // The report names the file it is about, so a log line is self-contained.
+ assert!(out.contains("\"problem\":"), "{out}");
+ assert!(out.contains("\"proof\":"), "{out}");
+}
+
+/// A mutated leaf must fail the process, not merely be mentioned in a field
+/// nobody reads. This is the gate `tools/fol_differential.py` now rests on.
+#[test]
+fn a_rejected_derivation_exits_one() {
+ let d = scratch("bad");
+ let proof = fixture("vampire-5.1.0.tstp").replace(
+ "'c:http://example.org/Greek'(X0) => 'c:http://example.org/Person'(X0)",
+ "'c:http://example.org/Greek'(X0) => 'c:http://example.org/Mortal'(X0)",
+ );
+ let (code, out) = check_pair(&d, &fixture("problem.p"), &proof);
+ let _ = std::fs::remove_dir_all(&d);
+ assert_eq!(code, 1, "a rejected derivation must fail a pipeline.\n{out}");
+ assert!(out.contains("\"verdict\":\"derivation_rejected\""), "{out}");
+ assert!(out.contains("\"stop_the_line\":1"), "{out}");
+}
+
+/// A prover that answered without printing a proof leaves a word and nothing
+/// to check. That is not a rejection and must not exit 1: it would turn every
+/// timeout into a build failure.
+#[test]
+fn an_output_with_no_derivation_exits_zero_and_says_why() {
+ let d = scratch("noproof");
+ let (code, out) = check_pair(&d, &fixture("problem.p"), "% SZS status Theorem for x\n");
+ let _ = std::fs::remove_dir_all(&d);
+ assert_eq!(code, 0, "{out}");
+ assert!(out.contains("\"verdict\":\"derivation_unparsed\""), "{out}");
+ assert!(out.contains("--proof-object"), "the fix is named in the output: {out}");
+}
+
+/// The subcommand refuses to do half the job. Without `--out` and without a
+/// recorded pair there is nothing to run and nothing to check, and saying so
+/// is better than running the prover over an empty store.
+#[test]
+fn fol_prove_with_neither_a_directory_nor_a_pair_is_refused() {
+ let out = Command::new(env!("CARGO_BIN_EXE_open-ontologies"))
+ .args(["--no-connect", "fol-prove"])
+ .output()
+ .expect("the binary runs");
+ assert_ne!(out.status.code(), Some(0));
+ let text = String::from_utf8_lossy(&out.stderr).to_string();
+ assert!(text.contains("--out") || text.contains("problem"), "{text}");
+}
+
+/// A missing prover is reported loudly and is never worked around. G6: the
+/// absence of a second opinion is not agreement with one.
+#[test]
+fn a_missing_prover_is_loud() {
+ let d = scratch("missing");
+ // An empty PATH makes every prover missing, which is the only portable way
+ // to drive this branch on a machine that has one installed.
+ let out = Command::new(env!("CARGO_BIN_EXE_open-ontologies"))
+ .args(["--no-connect", "--data-dir"])
+ .arg(d.join("store"))
+ .args(["fol-prove", "--out"])
+ .arg(d.join("run"))
+ .args(["--prover", "eprover"])
+ .env("PATH", "")
+ .output()
+ .expect("the binary runs");
+ let text = String::from_utf8_lossy(&out.stdout).to_string();
+ let _ = std::fs::remove_dir_all(&d);
+ assert!(text.contains("\"skipped\""), "{text}");
+ assert!(text.contains("not on PATH"), "{text}");
+ assert!(
+ text.contains("ABSENCE of a second opinion"),
+ "a skip must not read as agreement: {text}"
+ );
+}
diff --git a/tests/tstp_derivation_test.rs b/tests/tstp_derivation_test.rs
new file mode 100644
index 00000000..2f171c1d
--- /dev/null
+++ b/tests/tstp_derivation_test.rs
@@ -0,0 +1,848 @@
+//! What the TSTP checker verifies, and every way of making it say no.
+//!
+//! A checker that has never rejected anything is not a checker. Half of this
+//! file is therefore negative: a genuine Vampire refutation is taken apart one
+//! mutation at a time, and each mutation must be caught by the field that is
+//! supposed to catch it. The mutations are the ones a real defect would
+//! produce — a stale problem file, a conjecture presented as an axiom, a
+//! parent that does not exist, a cycle, a resolvent that is not the resolvent
+//! — rather than syntactic damage that any parser would notice.
+//!
+//! The file is grouped:
+//!
+//! 1. The reader is the emitter's inverse, over hand-computed cases.
+//! 2. A REAL Vampire 5.1.0 and a REAL E 3.2.5 refutation of a problem THIS
+//! repository's exporter produced, recorded under `tests/fixtures/tstp/`,
+//! with the problem regenerated from the exporter so the fixture cannot
+//! drift away from it, and a live run when a prover is on `PATH`.
+//! 3. The negative tests.
+//! 4. The verdict ladder, including the rule the whole design turns on: an
+//! UNCHECKED STEP MUST PREVENT THE STRONGEST WORD.
+//! 5. The replayed rules, one worked case each, with a wrong version of each
+//! that must not check out.
+//!
+//! Nothing here claims the derivations are proofs. The strongest word in the
+//! vocabulary is `refutation_fully_replayed` and it means the steps were
+//! recomputed in unverified Rust, in a calculus no theorem in `lean/` is about.
+
+mod common;
+
+use std::path::PathBuf;
+
+use open_ontologies::tptp::{Concept, FolProblem, OwlAxiom, fof};
+use open_ontologies::tstp::{self, Formula, Report};
+
+fn fixture(name: &str) -> String {
+ let p = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/tstp").join(name);
+ std::fs::read_to_string(&p).unwrap_or_else(|e| panic!("{}: {e}", p.display()))
+}
+
+fn iri(s: &str) -> String {
+ format!("http://example.org/{s}")
+}
+
+/// The ontology the recorded fixtures are about: Greek ⊑ Person, Person ⊑
+/// Mortal, socrates : Greek, with socrates : Mortal as the conjecture.
+fn socrates_problem() -> FolProblem {
+ let axioms = vec![
+ OwlAxiom::SubClass(Concept::Atom(iri("Greek")), Concept::Atom(iri("Person"))),
+ OwlAxiom::SubClass(Concept::Atom(iri("Person")), Concept::Atom(iri("Mortal"))),
+ OwlAxiom::ClassAssert(Concept::Atom(iri("Greek")), iri("socrates")),
+ ];
+ let goal = OwlAxiom::ClassAssert(Concept::Atom(iri("Mortal")), iri("socrates"));
+ FolProblem::build(&axioms, Some(&goal)).expect("freshness holds at trAx's own call sites")
+}
+
+fn parse1(text: &str) -> Formula {
+ let mut v = tstp::parse_problem(text).expect("the reader must accept its own test input");
+ assert_eq!(v.len(), 1, "expected exactly one annotated formula in {text:?}");
+ v.remove(0).formula
+}
+
+// ── 1. The reader is the emitter's inverse ─────────────────────────────────
+
+/// The whole leaf check rests on this. If the reader normalised anything on
+/// the way in, a leaf that differs from the problem could come back matching
+/// it, and the one gate that catches a prover answering about the wrong file
+/// would be the gate that cannot fail.
+///
+/// The cases are the emitter's own output over worked OWL axioms, not strings
+/// typed here, because what has to hold is that THIS reader inverts THAT
+/// writer. What is typed here is the reader's expected AST, which is computed
+/// from the `Form` by `from_tptp_form` and is a total embedding with no
+/// choices in it.
+#[test]
+fn the_reader_recovers_exactly_the_formula_the_emitter_wrote() {
+ use open_ontologies::tptp::{Ope, Translation};
+ let cases: Vec<OwlAxiom> = vec![
+ OwlAxiom::SubClass(Concept::Atom(iri("A")), Concept::Atom(iri("B"))),
+ OwlAxiom::SubClass(
+ Concept::Atom(iri("A")),
+ Concept::Some_(iri("r"), Box::new(Concept::Atom(iri("B")))),
+ ),
+ OwlAxiom::SubClass(
+ Concept::Atom(iri("A")),
+ Concept::All_(iri("r"), Box::new(Concept::Atom(iri("B")))),
+ ),
+ OwlAxiom::SubClass(
+ Concept::Inter(
+ Box::new(Concept::Atom(iri("A"))),
+ Box::new(Concept::Compl(Box::new(Concept::Atom(iri("B"))))),
+ ),
+ Concept::Bot,
+ ),
+ OwlAxiom::SubClass(Concept::Top, Concept::Union(
+ Box::new(Concept::Atom(iri("A"))),
+ Box::new(Concept::Atom(iri("B"))),
+ )),
+ OwlAxiom::SubClass(
+ Concept::Atom(iri("A")),
+ Concept::MinCard(2, iri("r"), Box::new(Concept::Atom(iri("B")))),
+ ),
+ OwlAxiom::SubClass(
+ Concept::Atom(iri("A")),
+ Concept::MaxCard(1, iri("r"), Box::new(Concept::Atom(iri("B")))),
+ ),
+ OwlAxiom::SubClass(Concept::Atom(iri("A")), Concept::OneOf(vec![iri("a"), iri("b")])),
+ OwlAxiom::SubClass(Concept::Atom(iri("A")), Concept::HasVal(iri("r"), iri("a"))),
+ OwlAxiom::SubClass(Concept::Atom(iri("A")), Concept::HasSelf(iri("r"))),
+ OwlAxiom::SubClass(Concept::Atom(iri("A")), Concept::DataSome(iri("d"), iri("D"))),
+ OwlAxiom::SubOProp(Ope::Named(iri("r")), Ope::Inv(iri("s"))),
+ OwlAxiom::OPropDomain(Ope::Named(iri("r")), Concept::Atom(iri("A"))),
+ OwlAxiom::OPropRange(Ope::Named(iri("r")), Concept::Atom(iri("A"))),
+ OwlAxiom::Transitive(iri("r")),
+ OwlAxiom::Symmetric(iri("r")),
+ OwlAxiom::Irreflexive(iri("r")),
+ OwlAxiom::Functional(iri("r")),
+ OwlAxiom::InverseOf(iri("r"), iri("s")),
+ OwlAxiom::Chain(vec![iri("r"), iri("s")], iri("t")),
+ OwlAxiom::ClassAssert(Concept::Atom(iri("A")), iri("a")),
+ OwlAxiom::OPropAssert(iri("r"), iri("a"), iri("b")),
+ OwlAxiom::SameAs(iri("a"), iri("b")),
+ OwlAxiom::DifferentFrom(iri("a"), iri("b")),
+ OwlAxiom::DPropRange(iri("d"), iri("D")),
+ ];
+ for ax in &cases {
+ let form = open_ontologies::tptp::Translation::axiom(ax).expect("in-fragment");
+ let text = format!("fof(t, axiom, {}).", fof::form(&form));
+ let back = parse1(&text);
+ assert_eq!(
+ back,
+ tstp::from_tptp_form(&form),
+ "the reader did not invert the emitter for {ax:?}\nemitted: {text}"
+ );
+ }
+ // The two background axioms go through the same door.
+ for form in Translation::background() {
+ let text = format!("fof(t, axiom, {}).", fof::form(&form));
+ assert_eq!(parse1(&text), tstp::from_tptp_form(&form));
+ }
+}
+
+/// Decision 0005's quoting trap, on the reading side. In TPTP a double-quoted
+/// string is a DISTINCT OBJECT, pairwise unequal to every other by fiat, and a
+/// single-quoted atom is an ordinary constant. A reader that unquoted both to
+/// the same name would silently identify them, which is the same unsoundness
+/// the writer avoids by never emitting the double-quoted form for an IRI.
+#[test]
+fn a_distinct_object_is_not_the_quoted_atom_of_the_same_text() {
+ let single = parse1("fof(t, axiom, p('i:a')).");
+ let double = parse1("fof(t, axiom, p(\"i:a\")).");
+ assert_ne!(single, double);
+}
+
+/// E prints an input status line before its result. Taking the first would
+/// report the wrong verdict for the whole run.
+#[test]
+fn the_last_szs_status_line_is_the_one_reported() {
+ let text = "# SZS status Started\nnoise\n# SZS status Theorem for x\n";
+ assert_eq!(tstp::szs_status(text).as_deref(), Some("Theorem"));
+ assert_eq!(tstp::szs_status("nothing here"), None);
+}
+
+/// Only what lies between the SZS markers is read. E prints its saturation
+/// state before the proof in the same syntax; a reader that swallowed it would
+/// build a derivation out of clauses the prover had abandoned.
+#[test]
+fn only_the_szs_block_is_read() {
+ let text = "fof(decoy, axiom, p(a)).\n\
+ % SZS output start Proof\n\
+ fof(real, axiom, q(a), file('p', a1)).\n\
+ % SZS output end Proof\n\
+ fof(trailer, axiom, r(a)).\n";
+ let d = tstp::parse_derivation(text).expect("the block parses");
+ assert_eq!(d.len(), 1);
+ assert_eq!(d[0].name, "real");
+}
+
+// ── 2. Real refutations of a problem this exporter produced ────────────────
+
+/// The fixture is the exporter's output and nothing else. Regenerated here
+/// from the same `FolProblem` the library builds, so a change to the
+/// translation breaks this test rather than leaving a stale file behind that
+/// the prover fixtures were recorded against.
+#[test]
+fn the_fixture_problem_is_what_this_exporter_emits() {
+ assert_eq!(
+ socrates_problem().to_tptp(),
+ fixture("problem.p"),
+ "tests/fixtures/tstp/problem.p is no longer what `fol --format tptp` emits. Re-record \
+ it AND re-record the two prover outputs beside it; a proof of the old problem says \
+ nothing about the new one"
+ );
+}
+
+fn assert_real_refutation(r: &Report, who: &str) {
+ assert_eq!(r.szs_status.as_deref(), Some("Theorem"), "{who}");
+ assert!(r.problem_error.is_none() && r.derivation_error.is_none(), "{who}: {r:?}");
+ assert!(r.derivation_wellformed, "{who}: {:?}", r.wellformedness_failures);
+ assert!(r.root_is_false, "{who}");
+ assert!(r.leaves_match_problem, "{who}: {:?}", r.leaf_failures);
+ assert!(r.leaf_failures.is_empty(), "{who}");
+ assert!(r.steps_not_reconstructed.is_empty(), "{who}: {:?}", r.steps_not_reconstructed);
+ assert_eq!(r.conjecture_in_problem.as_deref(), Some("goal_classAssertion"), "{who}");
+ assert!(r.conjecture_used, "{who}");
+ assert!(r.conjecture_negation_checked, "{who}");
+ assert_eq!(r.what_was_refuted, Some("axioms_and_the_negated_conjecture"), "{who}");
+ // The prover's own clausification is never claimed to be checked, so the
+ // strongest word is out of reach and must be.
+ assert_eq!(r.verdict, "refutation_partially_replayed", "{who}");
+ assert!(r.steps_checked > 0 && r.steps_checked < r.steps_total, "{who}");
+}
+
+/// A REAL Vampire 5.1.0 refutation, recorded from `vampire --proof tptp` over
+/// `tests/fixtures/tstp/problem.p`.
+///
+/// This proof uses AVATAR, so it carries two `introduced(definition, …)`
+/// leaves and a SAT-solver tail. Neither is checked and both are named, which
+/// is the point: the report shows exactly how much of a real proof this module
+/// can and cannot replay.
+#[test]
+fn a_real_vampire_refutation_checks_out() {
+ let r = tstp::check(&fixture("problem.p"), &fixture("vampire-5.1.0.tstp"));
+ assert_real_refutation(&r, "vampire");
+ // Vampire echoes the axioms unchanged, so every leaf matches at the
+ // strictest level.
+ assert_eq!(r.leaf_match_levels.get("identical").copied(), Some(4));
+ assert!(!r.leaf_match_levels.contains_key("alpha_equivalent"));
+ assert!(
+ r.steps_checked_by_rule.get("binary_resolvent").copied().unwrap_or(0) >= 5,
+ "{:?}",
+ r.steps_checked_by_rule
+ );
+ assert_eq!(r.steps_checked_by_rule.get("negation_of_the_conjecture").copied(), Some(1));
+ assert_eq!(r.leaves_introduced.len(), 2, "AVATAR invents two definitions here");
+ let unchecked: Vec<&str> = r.steps_unchecked.iter().map(|u| u.rule.as_str()).collect();
+ for expected in ["cnf_transformation", "avatar_split_clause", "sat_conversion"] {
+ assert!(unchecked.contains(&expected), "{expected} must be named: {unchecked:?}");
+ }
+}
+
+/// A REAL E 3.2.5 refutation of the same file. E renames bound variables on
+/// every axiom it reads, which is why the leaf check has an
+/// `alpha_equivalent` level at all, and it nests its inference records, which
+/// is why almost nothing in an E proof can be replayed. Both facts are
+/// asserted rather than left to be discovered.
+#[test]
+fn a_real_eprover_refutation_checks_out() {
+ let r = tstp::check(&fixture("problem.p"), &fixture("eprover-3.2.5.tstp"));
+ assert_real_refutation(&r, "eprover");
+ assert!(
+ r.leaf_match_levels.get("alpha_equivalent").copied().unwrap_or(0) >= 2,
+ "E renames variables, so some leaves must match only up to alpha: {:?}",
+ r.leaf_match_levels
+ );
+ let inline = r
+ .steps_unchecked
+ .iter()
+ .filter(|u| u.why.contains("inline inference record"))
+ .map(|u| u.count)
+ .sum::<usize>();
+ assert!(inline > 0, "E nests inference records and the report must say so: {:?}", r.steps_unchecked);
+}
+
+/// The recorded fixtures are only worth having if the live prover still says
+/// the same thing. Skips loudly without Vampire, and fails under
+/// `OO_REQUIRE_FIXTURES=1`.
+#[test]
+fn a_live_vampire_run_agrees_with_the_recorded_one() {
+ if common::skip_unless(
+ tstp::Prover::Vampire.available(),
+ "vampire on PATH",
+ tstp::Prover::Vampire.install_line(),
+ ) {
+ return;
+ }
+ let dir = std::env::temp_dir().join(format!("oo-tstp-live-{}", std::process::id()));
+ std::fs::create_dir_all(&dir).unwrap();
+ let problem = socrates_problem().to_tptp();
+ let pfile = dir.join("problem.p");
+ std::fs::write(&pfile, &problem).unwrap();
+ let out = std::process::Command::new("vampire")
+ .args(tstp::Prover::Vampire.argv(&pfile, 10))
+ .output()
+ .expect("vampire is on PATH");
+ let text = String::from_utf8_lossy(&out.stdout).to_string()
+ + &String::from_utf8_lossy(&out.stderr);
+ let r = tstp::check(&problem, &text);
+ let _ = std::fs::remove_dir_all(&dir);
+ // Vampire's strategy schedule is not pinned, so the exact step counts move
+ // between builds. What must not move is the verdict and every field the
+ // verdict is computed from.
+ assert_real_refutation(&r, "live vampire");
+}
+
+// ── 3. The negative tests ──────────────────────────────────────────────────
+
+/// Mutate a leaf so it is no longer the formula the problem carries under that
+/// name. This is the check that catches a prover run against a stale file, and
+/// it is the reason the leaf comparison is over parsed ASTs rather than over
+/// the name alone.
+#[test]
+fn a_mutated_leaf_is_rejected() {
+ let proof = fixture("vampire-5.1.0.tstp").replace(
+ "'c:http://example.org/Greek'(X0) => 'c:http://example.org/Person'(X0)",
+ "'c:http://example.org/Greek'(X0) => 'c:http://example.org/Mortal'(X0)",
+ );
+ assert_ne!(proof, fixture("vampire-5.1.0.tstp"), "the mutation must change the text");
+ let r = tstp::check(&fixture("problem.p"), &proof);
+ assert!(!r.leaves_match_problem);
+ assert_eq!(r.verdict, "derivation_rejected");
+ assert!(
+ r.leaf_failures.iter().any(|f| f.why.contains("a different formula")),
+ "{:?}",
+ r.leaf_failures
+ );
+ // The structure is untouched, so the structural fields must still be true.
+ // Collapsing them into one boolean would hide which gate actually bit.
+ assert!(r.derivation_wellformed);
+ assert!(r.root_is_false);
+}
+
+/// A leaf that names a formula the problem does not have. The prover was given
+/// a different file, and the name is the only evidence of it.
+#[test]
+fn a_leaf_naming_a_formula_the_problem_lacks_is_rejected() {
+ let proof = fixture("vampire-5.1.0.tstp")
+ .replace("'owl_1_subClassOf'", "'owl_1_subClassOf_from_some_other_file'");
+ let r = tstp::check(&fixture("problem.p"), &proof);
+ assert!(!r.leaves_match_problem);
+ assert_eq!(r.verdict, "derivation_rejected");
+ assert!(
+ r.leaf_failures.iter().any(|f| f.why.contains("no formula of that name")),
+ "{:?}",
+ r.leaf_failures
+ );
+}
+
+/// The conjecture presented as an axiom. The formula is ours, the name is
+/// ours, and the refutation would prove nothing about entailment: asserting
+/// the goal and refuting the axioms together is not the same argument. Caught
+/// by the role comparison, which is why the role is compared at all.
+#[test]
+fn a_conjecture_smuggled_in_as_an_axiom_is_rejected() {
+ let proof = fixture("vampire-5.1.0.tstp").replace("fof(f7,conjecture,", "fof(f7,axiom,");
+ let r = tstp::check(&fixture("problem.p"), &proof);
+ assert!(!r.leaves_match_problem);
+ assert_eq!(r.verdict, "derivation_rejected");
+ assert!(
+ r.leaf_failures.iter().any(|f| f.why.contains("refutes nothing")),
+ "{:?}",
+ r.leaf_failures
+ );
+}
+
+/// A parent reference that resolves to nothing. A derivation with a hole in it
+/// is not a derivation, however good the rest of it looks.
+#[test]
+fn a_dangling_parent_is_rejected() {
+ let proof = fixture("vampire-5.1.0.tstp")
+ .replace("inference(resolution,[],[f18,f20])", "inference(resolution,[],[f18,f999])");
+ let r = tstp::check(&fixture("problem.p"), &proof);
+ assert!(!r.derivation_wellformed);
+ assert_eq!(r.verdict, "derivation_rejected");
+ assert!(
+ r.wellformedness_failures.iter().any(|w| w.contains("f999")),
+ "{:?}",
+ r.wellformedness_failures
+ );
+}
+
+/// A cycle. Well-foundedness is the whole content of "the conclusion follows
+/// from the premises"; without it a derivation can conclude anything from
+/// itself.
+#[test]
+fn a_cycle_is_rejected() {
+ let problem = "fof(a1, axiom, p(a)).\n";
+ let proof = "% SZS output start Proof\n\
+ fof(a1, axiom, p(a), file('p', a1)).\n\
+ fof(c1, plain, q(a), inference(resolution,[],[a1,c2])).\n\
+ fof(c2, plain, r(a), inference(resolution,[],[c1])).\n\
+ fof(c3, plain, $false, inference(resolution,[],[c1,c2])).\n\
+ % SZS output end Proof\n";
+ let r = tstp::check(problem, proof);
+ assert!(!r.derivation_wellformed);
+ assert_eq!(r.verdict, "derivation_rejected");
+ assert!(
+ r.wellformedness_failures.iter().any(|w| w.contains("cycle")),
+ "{:?}",
+ r.wellformedness_failures
+ );
+}
+
+/// Two annotated formulas with the same name make every reference to that name
+/// ambiguous, so the DAG is not determined by the text.
+#[test]
+fn a_duplicated_name_is_rejected() {
+ let problem = "fof(a1, axiom, p(a)).\n";
+ let proof = "% SZS output start Proof\n\
+ fof(a1, axiom, p(a), file('p', a1)).\n\
+ fof(c1, plain, q(a), inference(resolution,[],[a1])).\n\
+ fof(c1, plain, r(a), inference(resolution,[],[a1])).\n\
+ fof(c2, plain, $false, inference(resolution,[],[c1])).\n\
+ % SZS output end Proof\n";
+ let r = tstp::check(problem, proof);
+ assert_eq!(r.verdict, "derivation_rejected");
+ assert!(
+ r.wellformedness_failures.iter().any(|w| w.contains("more than one")),
+ "{:?}",
+ r.wellformedness_failures
+ );
+}
+
+/// A genuine derivation checked against a DIFFERENT problem. Every name
+/// resolves and every structural gate holds; only the formulas disagree. This
+/// is what a stale export or a mismatched pair of files actually looks like.
+#[test]
+fn a_genuine_derivation_of_another_problem_is_rejected() {
+ let other = fixture("problem.p").replace("http://example.org/Greek", "http://example.org/Hindu");
+ let r = tstp::check(&other, &fixture("vampire-5.1.0.tstp"));
+ assert!(!r.leaves_match_problem);
+ assert_eq!(r.verdict, "derivation_rejected");
+}
+
+/// A resolution step whose conclusion is not the resolvent. The rule is one
+/// this module implements and the shapes apply, so it gets its own word and is
+/// NOT filed under "unchecked" where a forged step could hide behind a rule
+/// name.
+#[test]
+fn a_resolution_conclusion_that_is_not_the_resolvent_is_not_reconstructed() {
+ let problem = "fof(a1, axiom, p(a)).\nfof(a2, axiom, ( ~ p(a) | q(a) )).\n";
+ let proof = "% SZS output start Proof\n\
+ fof(a1, axiom, p(a), file('p', a1)).\n\
+ fof(a2, axiom, ( ~p(a) | q(a) ), file('p', a2)).\n\
+ fof(c1, plain, r(a), inference(resolution,[],[a2,a1])).\n\
+ fof(c2, plain, $false, inference(resolution,[],[c1])).\n\
+ % SZS output end Proof\n";
+ let r = tstp::check(problem, proof);
+ assert!(r.derivation_wellformed && r.leaves_match_problem && r.root_is_false);
+ assert_eq!(r.verdict, "refutation_step_not_reconstructed");
+ assert!(
+ r.steps_not_reconstructed.iter().any(|s| s.node == "c1" && s.rule == "resolution"),
+ "{:?}",
+ r.steps_not_reconstructed
+ );
+}
+
+/// A negated conjecture that is not the negation of the conjecture. Vampire's
+/// own step here is exact, so a forged one has nowhere to hide.
+#[test]
+fn a_forged_negated_conjecture_is_not_reconstructed() {
+ let proof = fixture("vampire-5.1.0.tstp").replace(
+ "fof(f8,negated_conjecture,(\n ~(thing('i:http://example.org/socrates') & \
+ 'c:http://example.org/Mortal'('i:http://example.org/socrates'))),",
+ "fof(f8,negated_conjecture,(\n ~thing('i:http://example.org/socrates')),",
+ );
+ assert_ne!(proof, fixture("vampire-5.1.0.tstp"), "the mutation must change the text");
+ let r = tstp::check(&fixture("problem.p"), &proof);
+ assert_eq!(r.verdict, "refutation_step_not_reconstructed");
+ assert!(
+ r.steps_not_reconstructed.iter().any(|s| s.rule == "negated_conjecture"),
+ "{:?}",
+ r.steps_not_reconstructed
+ );
+ assert!(!r.conjecture_negation_checked);
+}
+
+/// The occurs check is load-bearing. Without it `p(X, f(X))` and `p(Y, Y)`
+/// unify, and a step that cannot be taken would be reported as replayed.
+#[test]
+fn the_occurs_check_refuses_a_resolution_that_needs_a_cyclic_binding() {
+ let problem = "cnf(a1, axiom, p(X0, f(X0))).\ncnf(a2, axiom, ~p(Y0, Y0)).\n";
+ let proof = "% SZS output start Proof\n\
+ cnf(a1, axiom, p(X0, f(X0)), file('p', a1)).\n\
+ cnf(a2, axiom, ~p(Y0, Y0), file('p', a2)).\n\
+ cnf(c1, plain, $false, inference(resolution,[],[a1,a2])).\n\
+ % SZS output end Proof\n";
+ let r = tstp::check(problem, proof);
+ assert_eq!(r.verdict, "refutation_step_not_reconstructed");
+}
+
+// ── 4. The verdict ladder ──────────────────────────────────────────────────
+
+const ALL_CHECKED_PROBLEM: &str =
+ "fof(a1, axiom, p(a)).\nfof(a2, axiom, ( ~ p(a) | q(a) )).\nfof(a3, axiom, ~ q(a)).\n";
+
+fn all_checked_proof(rule_for_c1: &str) -> String {
+ format!(
+ "% SZS status Unsatisfiable\n\
+ % SZS output start Proof\n\
+ fof(a1, axiom, p(a), file('p', a1)).\n\
+ fof(a2, axiom, ( ~p(a) | q(a) ), file('p', a2)).\n\
+ fof(a3, axiom, ~q(a), file('p', a3)).\n\
+ fof(c1, plain, q(a), inference({rule_for_c1},[],[a2,a1])).\n\
+ fof(c2, plain, $false, inference(resolution,[],[c1,a3])).\n\
+ % SZS output end Proof\n"
+ )
+}
+
+/// The strongest word exists, is reachable, and says what it says. Every
+/// reachable step here is recomputed from its premises.
+#[test]
+fn a_derivation_whose_every_step_is_replayed_reaches_the_strongest_word() {
+ let r = tstp::check(ALL_CHECKED_PROBLEM, &all_checked_proof("resolution"));
+ assert_eq!(r.verdict, "refutation_fully_replayed");
+ assert_eq!(r.steps_checked, r.steps_total);
+ assert!(r.steps_unchecked.is_empty());
+ assert_eq!(r.what_was_refuted, Some("axioms_alone"), "this problem carries no conjecture");
+ // And the word still is not a claim of unsatisfiability.
+ let means = tstp::verdict_means("refutation_fully_replayed");
+ assert!(means.contains("NOT a proof of unsatisfiability"), "{means}");
+ assert!(means.contains("unverified Rust"), "{means}");
+}
+
+/// THE RULE THE DESIGN TURNS ON. One step renamed to a rule this module does
+/// not replay, everything else identical, and the strongest word must be out
+/// of reach. If this ever passes, the vocabulary has stopped meaning anything.
+#[test]
+fn one_unchecked_step_prevents_the_strongest_word() {
+ let strong = tstp::check(ALL_CHECKED_PROBLEM, &all_checked_proof("resolution"));
+ assert_eq!(strong.verdict, "refutation_fully_replayed");
+
+ let weakened = tstp::check(ALL_CHECKED_PROBLEM, &all_checked_proof("cnf_transformation"));
+ assert_ne!(weakened.verdict, "refutation_fully_replayed");
+ assert_eq!(weakened.verdict, "refutation_partially_replayed");
+ assert_eq!(weakened.steps_total, strong.steps_total);
+ assert_eq!(weakened.steps_checked, strong.steps_checked - 1);
+ assert_eq!(
+ weakened.steps_unchecked.iter().map(|u| u.count).sum::<usize>(),
+ 1,
+ "the unchecked step must be counted, not absorbed"
+ );
+ assert_eq!(weakened.steps_unchecked[0].rule, "cnf_transformation");
+ assert!(!weakened.steps_unchecked[0].why.is_empty(), "an unchecked rule carries a reason");
+ // Everything else is untouched, which is what makes this a test of the
+ // ladder rather than of the structural gates.
+ assert!(weakened.derivation_wellformed && weakened.leaves_match_problem);
+}
+
+/// An INTRODUCED leaf is a formula the prover invented. Nothing here checks
+/// that the extension is conservative, so it counts as unchecked and keeps the
+/// strongest word out of reach even when every inference step is replayed.
+#[test]
+fn an_introduced_definition_prevents_the_strongest_word() {
+ let problem = "fof(a1, axiom, p(a)).\nfof(a2, axiom, ( ~ p(a) | q(a) )).\n";
+ let proof = "% SZS output start Proof\n\
+ fof(a1, axiom, p(a), file('p', a1)).\n\
+ fof(a2, axiom, ( ~p(a) | q(a) ), file('p', a2)).\n\
+ fof(d1, definition, ( ~q(a) ), introduced(definition,[new_symbols(definition,[sP0])])).\n\
+ fof(c1, plain, q(a), inference(resolution,[],[a2,a1])).\n\
+ fof(c2, plain, $false, inference(resolution,[],[c1,d1])).\n\
+ % SZS output end Proof\n";
+ let r = tstp::check(problem, proof);
+ assert!(r.leaves_match_problem, "an introduced leaf is not a FAILED leaf: {:?}", r.leaf_failures);
+ assert_eq!(r.leaves_introduced.len(), 1);
+ assert_eq!(r.steps_checked, r.steps_total, "every inference step here is replayed");
+ assert_eq!(r.verdict, "refutation_partially_replayed");
+}
+
+/// No empty clause means no refutation to check, and that is a separate word
+/// from "the checker said no". A satisfiable problem reaching
+/// `derivation_rejected` would be a false alarm on every consistent ontology.
+#[test]
+fn a_run_with_no_empty_clause_offers_no_refutation() {
+ let problem = "fof(a1, axiom, p(a)).\n";
+ let proof = "% SZS status Satisfiable\n\
+ % SZS output start Saturation\n\
+ fof(a1, axiom, p(a), file('p', a1)).\n\
+ % SZS output end Saturation\n";
+ let r = tstp::check(problem, proof);
+ assert_eq!(r.verdict, "no_refutation_offered");
+ assert!(!r.root_is_false);
+ assert_eq!(r.szs_status.as_deref(), Some("Satisfiable"));
+ assert_eq!(r.what_was_refuted, None, "nothing was refuted, so the field stays null");
+}
+
+/// A prover asked for a verdict and not for a proof prints one line. That is
+/// not a refutation and it is not a rejection either.
+#[test]
+fn an_output_with_no_derivation_is_unparsed_and_says_how_to_fix_it() {
+ let r = tstp::check("fof(a1, axiom, p(a)).\n", "% SZS status Theorem for x\n");
+ assert_eq!(r.verdict, "derivation_unparsed");
+ let why = r.derivation_error.expect("the reason is reported");
+ assert!(why.contains("--proof-object") && why.contains("--proof"), "{why}");
+}
+
+/// An unreadable PROBLEM is not a statement about the prover, and the verdict
+/// says so rather than blaming the derivation.
+#[test]
+fn an_unreadable_problem_is_its_own_verdict() {
+ let r = tstp::check("this is not a TPTP file", &fixture("vampire-5.1.0.tstp"));
+ assert_eq!(r.verdict, "problem_unparsed");
+ assert!(r.problem_error.is_some());
+ assert!(r.derivation_error.is_none(), "the derivation was never blamed");
+}
+
+/// Every verdict word has a sentence, and no sentence says the word means the
+/// problem is unsatisfiable. The vocabulary is fixed here so that adding a
+/// word without a meaning fails a test rather than printing
+/// `unrecognised verdict` at a reader.
+#[test]
+fn every_verdict_word_has_a_meaning_and_none_of_them_claims_a_proof() {
+ let words = [
+ "problem_unparsed",
+ "derivation_unparsed",
+ "no_refutation_offered",
+ "derivation_rejected",
+ "refutation_step_not_reconstructed",
+ "refutation_structure_checked",
+ "refutation_partially_replayed",
+ "refutation_fully_replayed",
+ ];
+ for w in words {
+ let m = tstp::verdict_means(w);
+ assert_ne!(m, "unrecognised verdict", "{w} has no sentence");
+ assert!(
+ !m.contains("certified") && !m.contains("proves that"),
+ "{w} claims too much: {m}"
+ );
+ }
+ assert_eq!(tstp::verdict_means("model_checked"), "unrecognised verdict");
+}
+
+/// The report says, in the output itself, that nothing here is certified and
+/// that `lean/` is not involved. A limit stated only in a decision record gets
+/// read without one.
+#[test]
+fn the_report_says_in_its_own_json_that_nothing_is_certified() {
+ let r = tstp::check(&fixture("problem.p"), &fixture("vampire-5.1.0.tstp"));
+ let j = tstp::report_json(&r);
+ let checked_by = j["checked_by"].as_str().expect("checked_by is present");
+ assert!(checked_by.contains("UNVERIFIED"), "{checked_by}");
+ assert!(checked_by.contains("lean/ is not involved"), "{checked_by}");
+ assert!(checked_by.contains("ORACLE OPINION"), "{checked_by}");
+ let szs = j["szs_status_means"].as_str().expect("the echoed status is labelled");
+ assert!(szs.contains("untrusted"), "{szs}");
+}
+
+// ── 5. The replayed rules, one worked case each ────────────────────────────
+
+/// Run a hand-built derivation and return the report.
+fn hand(problem: &str, body: &str) -> Report {
+ tstp::check(problem, &format!("% SZS output start Proof\n{body}% SZS output end Proof\n"))
+}
+
+/// Subsumption resolution's conclusion IS the binary resolvent of its
+/// premises, because the side clause's remainder is contained in the main
+/// clause's. One check covers both rules and neither is weakened by it.
+#[test]
+fn subsumption_resolution_is_checked_as_the_binary_resolvent_it_is() {
+ let problem = "cnf(a1, axiom, ( p(X0) | q(X0) )).\ncnf(a2, axiom, ( ~p(bb) | q(bb) )).\n";
+ let r = hand(
+ problem,
+ "cnf(a1, axiom, ( p(X0) | q(X0) ), file('p', a1)).\n\
+ cnf(a2, axiom, ( ~p(bb) | q(bb) ), file('p', a2)).\n\
+ cnf(c1, plain, q(bb), inference(subsumption_resolution,[],[a2,a1])).\n\
+ cnf(c2, plain, $false, inference(resolution,[],[c1])).\n",
+ );
+ assert_eq!(r.steps_checked_by_rule.get("binary_resolvent").copied(), Some(1));
+ assert!(
+ r.steps_not_reconstructed.iter().all(|s| s.node != "c1"),
+ "{:?}",
+ r.steps_not_reconstructed
+ );
+}
+
+/// Two premises that happen to use the same variable name must be renamed
+/// apart before unification. Without the renaming this resolvent does not
+/// exist, because `X0` would have to be both `dd` and `cc`.
+#[test]
+fn resolution_standardises_the_premises_apart() {
+ let problem = "cnf(a1, axiom, p(X0, cc)).\ncnf(a2, axiom, ( ~p(dd, X0) | r(X0) )).\n";
+ let r = hand(
+ problem,
+ "cnf(a1, axiom, p(X0, cc), file('p', a1)).\n\
+ cnf(a2, axiom, ( ~p(dd, X0) | r(X0) ), file('p', a2)).\n\
+ cnf(c1, plain, r(cc), inference(resolution,[],[a1,a2])).\n\
+ cnf(c2, plain, $false, inference(resolution,[],[c1])).\n",
+ );
+ assert_eq!(
+ r.steps_checked_by_rule.get("binary_resolvent").copied(),
+ Some(1),
+ "{:?} {:?}",
+ r.steps_checked_by_rule,
+ r.steps_not_reconstructed
+ );
+}
+
+/// `C ∨ s ≠ t` with `σ = mgu(s, t)` gives `Cσ`, and a conclusion that is not
+/// `Cσ` does not check out.
+#[test]
+fn equality_resolution_is_replayed_and_a_wrong_one_is_not() {
+ let problem = "cnf(a1, axiom, ( X0 != aa | q(X0) )).\ncnf(a2, axiom, ~q(aa)).\n";
+ let good = hand(
+ problem,
+ "cnf(a1, axiom, ( X0 != aa | q(X0) ), file('p', a1)).\n\
+ cnf(a2, axiom, ~q(aa), file('p', a2)).\n\
+ cnf(c1, plain, q(aa), inference(equality_resolution,[],[a1])).\n\
+ cnf(c2, plain, $false, inference(resolution,[],[c1,a2])).\n",
+ );
+ assert_eq!(good.verdict, "refutation_fully_replayed", "{good:?}");
+ assert_eq!(good.steps_checked_by_rule.get("equality_resolved").copied(), Some(1));
+
+ let bad = hand(
+ problem,
+ "cnf(a1, axiom, ( X0 != aa | q(X0) ), file('p', a1)).\n\
+ cnf(a2, axiom, ~q(aa), file('p', a2)).\n\
+ cnf(c1, plain, q(bb), inference(equality_resolution,[],[a1])).\n\
+ cnf(c2, plain, $false, inference(resolution,[],[c1])).\n",
+ );
+ assert_eq!(bad.verdict, "refutation_step_not_reconstructed");
+}
+
+/// Two literals of one clause unified and merged.
+#[test]
+fn factoring_is_replayed_and_a_wrong_one_is_not() {
+ let problem = "cnf(a1, axiom, ( q(X0) | q(aa) )).\ncnf(a2, axiom, ~q(aa)).\n";
+ let good = hand(
+ problem,
+ "cnf(a1, axiom, ( q(X0) | q(aa) ), file('p', a1)).\n\
+ cnf(a2, axiom, ~q(aa), file('p', a2)).\n\
+ cnf(c1, plain, q(aa), inference(factoring,[],[a1])).\n\
+ cnf(c2, plain, $false, inference(resolution,[],[c1,a2])).\n",
+ );
+ assert_eq!(good.verdict, "refutation_fully_replayed", "{good:?}");
+ assert_eq!(good.steps_checked_by_rule.get("factor").copied(), Some(1));
+
+ let bad = hand(
+ problem,
+ "cnf(a1, axiom, ( q(X0) | q(aa) ), file('p', a1)).\n\
+ cnf(a2, axiom, ~q(aa), file('p', a2)).\n\
+ cnf(c1, plain, q(bb), inference(factoring,[],[a1])).\n\
+ cnf(c2, plain, $false, inference(resolution,[],[c1])).\n",
+ );
+ assert_eq!(bad.verdict, "refutation_step_not_reconstructed");
+}
+
+/// Literals `s != s` with syntactically identical sides, deleted. A conclusion
+/// that deletes something else does not check out.
+#[test]
+fn trivial_inequality_removal_is_replayed_and_a_wrong_one_is_not() {
+ let problem = "cnf(a1, axiom, ( aa != aa | q(aa) )).\ncnf(a2, axiom, ~q(aa)).\n";
+ let good = hand(
+ problem,
+ "cnf(a1, axiom, ( aa != aa | q(aa) ), file('p', a1)).\n\
+ cnf(a2, axiom, ~q(aa), file('p', a2)).\n\
+ cnf(c1, plain, q(aa), inference(trivial_inequality_removal,[],[a1])).\n\
+ cnf(c2, plain, $false, inference(resolution,[],[c1,a2])).\n",
+ );
+ assert_eq!(good.verdict, "refutation_fully_replayed", "{good:?}");
+ assert_eq!(good.steps_checked_by_rule.get("trivial_inequality_removed").copied(), Some(1));
+
+ let bad = hand(
+ problem,
+ "cnf(a1, axiom, ( aa != aa | q(aa) ), file('p', a1)).\n\
+ cnf(a2, axiom, ~q(aa), file('p', a2)).\n\
+ cnf(c1, plain, q(bb), inference(trivial_inequality_removal,[],[a1])).\n\
+ cnf(c2, plain, $false, inference(resolution,[],[c1])).\n",
+ );
+ assert_eq!(bad.verdict, "refutation_step_not_reconstructed");
+}
+
+/// Vampire's `flattening` reassociates a disjunction and changes nothing else,
+/// which is exactly "the same clause". A conclusion that drops a literal is
+/// not the same clause and is caught.
+#[test]
+fn flattening_is_checked_as_the_same_clause() {
+ let problem = "cnf(a1, axiom, ( ( p(aa) | q(aa) ) | r(aa) )).\n";
+ let good = hand(
+ problem,
+ "cnf(a1, axiom, ( ( p(aa) | q(aa) ) | r(aa) ), file('p', a1)).\n\
+ cnf(c1, plain, ( p(aa) | q(aa) | r(aa) ), inference(flattening,[],[a1])).\n\
+ cnf(c2, plain, $false, inference(resolution,[],[c1])).\n",
+ );
+ assert_eq!(good.steps_checked_by_rule.get("same_clause").copied(), Some(1));
+
+ let bad = hand(
+ problem,
+ "cnf(a1, axiom, ( ( p(aa) | q(aa) ) | r(aa) ), file('p', a1)).\n\
+ cnf(c1, plain, ( p(aa) | q(aa) ), inference(flattening,[],[a1])).\n\
+ cnf(c2, plain, $false, inference(resolution,[],[c1])).\n",
+ );
+ assert!(
+ bad.steps_not_reconstructed.iter().any(|s| s.rule == "flattening"),
+ "{:?}",
+ bad.steps_not_reconstructed
+ );
+}
+
+/// A step this module would check, over a premise that is not a clause, is
+/// reported as UNCHECKED with the reason rather than as a failure. A prover
+/// applying a clause rule to a formula is not evidence of anything and must
+/// not be reported as if it were.
+#[test]
+fn a_clause_rule_over_a_non_clause_premise_is_unchecked_with_the_reason() {
+ let problem = "fof(a1, axiom, ( p(aa) & q(aa) )).\n";
+ let r = hand(
+ problem,
+ "fof(a1, axiom, ( p(aa) & q(aa) ), file('p', a1)).\n\
+ fof(c1, plain, p(aa), inference(flattening,[],[a1])).\n\
+ fof(c2, plain, $false, inference(resolution,[],[c1])).\n",
+ );
+ assert!(r.steps_not_reconstructed.iter().all(|s| s.rule != "flattening"));
+ let u = r
+ .steps_unchecked
+ .iter()
+ .find(|u| u.rule == "flattening")
+ .expect("the rule is named as unchecked");
+ assert!(u.why.contains("clause-shaped"), "{}", u.why);
+}
+
+/// A leaf whose only difference from the problem is bracketing of an n-ary
+/// connective is matched, at its own level, and the level is reported.
+/// Associativity of `&` and `|` is not in question; how much normalisation a
+/// leaf needed is worth knowing.
+#[test]
+fn a_rebracketed_leaf_matches_at_its_own_level() {
+ let problem = "fof(a1, axiom, ( p(aa) & ( q(aa) & r(aa) ) )).\n";
+ let r = hand(
+ problem,
+ "fof(a1, axiom, ( ( p(aa) & q(aa) ) & r(aa) ), file('p', a1)).\n\
+ fof(c1, plain, $false, inference(cnf_transformation,[],[a1])).\n",
+ );
+ assert!(r.leaves_match_problem, "{:?}", r.leaf_failures);
+ assert_eq!(r.leaf_match_levels.get("associativity_normalised").copied(), Some(1));
+ assert!(!r.leaf_match_levels.contains_key("identical"));
+}
+
+/// Nodes the empty clause does not depend on are counted and left out of the
+/// step tallies. A prover's output often carries them, and counting them would
+/// report work that the refutation does not rest on.
+#[test]
+fn nodes_the_refutation_does_not_rest_on_are_counted_separately() {
+ let problem = "fof(a1, axiom, p(a)).\nfof(a2, axiom, ( ~ p(a) | q(a) )).\n";
+ let r = hand(
+ problem,
+ "fof(a1, axiom, p(a), file('p', a1)).\n\
+ fof(a2, axiom, ( ~p(a) | q(a) ), file('p', a2)).\n\
+ fof(c1, plain, q(a), inference(resolution,[],[a2,a1])).\n\
+ fof(c2, plain, $false, inference(resolution,[],[a2,a1])).\n\
+ fof(spare, plain, r(a), inference(cnf_transformation,[],[a1])).\n",
+ );
+ assert!(r.nodes_outside_the_refutation >= 2, "c1 and spare are not cited: {r:?}");
+ assert!(
+ r.steps_unchecked.iter().all(|u| u.rule != "cnf_transformation"),
+ "a node outside the refutation must not be counted: {:?}",
+ r.steps_unchecked
+ );
+}
diff --git a/tools/fol_differential.py b/tools/fol_differential.py
index 07fe47e3..2adbf878 100644
--- a/tools/fol_differential.py
+++ b/tools/fol_differential.py
@@ -56,6 +56,20 @@ The verdicts, in order of severity:
exporter lists these; they are reproduced here.
AGREE - the ATP also finds the conclusion entailed.
+Since 15 September 2026 an AGREE row carries a second column, because "the
+prover said Theorem" is a word and the derivation behind it is an object.
+Each prover is run WITH its proof-printing option and the derivation it emits
+is handed to `open-ontologies fol-prove --problem F --proof G`, which parses
+it, matches every leaf against the problem file THIS engine wrote, checks the
+DAG for dangling parents and cycles, and recomputes the resolution-family
+steps. The result lands in `proof_check` and `proof_steps`.
+
+THAT IS STILL NOT A PROOF and AGREE still means what it meant. What the second
+column rules out is narrower and was, until it existed, simply assumed: that
+the prover was answering about the file we gave it. A `proof_check` of
+PROOF_REJECTED is a stop-the-line, exits 1, and means the derivation's leaves
+are not our formulas. See docs/decisions/0005, the addendum.
+
Usage:
python3 tools/fol_differential.py ONTOLOGY.ttl [--json out.json]
[--profile owl-rl-ext] [--limit N]
@@ -85,20 +99,23 @@ BIN = os.environ.get(
# status lines. Prover9 is deliberately NOT probed: it reads LADR, not TPTP,
# and silently treating its input as TPTP would be exactly the kind of
# unnoticed mismatch this tool exists to catch.
+# `--proof-object` and `--proof tptp` are not optional extras. Without them a
+# prover returns an SZS word and nothing else, and a word cannot be checked.
ATPS = [
{
"name": "eprover",
- "argv": lambda f, t: ["eprover", "--auto", "--tptp3-format", f"--cpu-limit={t}", f],
+ "argv": lambda f, t: ["eprover", "--auto", "--tptp3-format", "--proof-object",
+ f"--cpu-limit={t}", f],
"install": "brew install eprover (or apt-get install eprover, or "
"https://github.com/eprover/eprover)",
},
{
- # UNTESTED. Vampire was not installed on the machine this was written
- # on, so this argument vector and the SZS parsing for it come from the
- # documented interface and have never run. Said here rather than
- # discovered by whoever first passes --atp vampire.
+ # Was UNTESTED until 15 September 2026, when Vampire 5.1.0 was
+ # installed and this vector was run: `--mode casc` prints its proof to
+ # a temporary file and then to stdout, and `--proof tptp` is what makes
+ # that proof a TSTP derivation rather than a display format.
"name": "vampire",
- "argv": lambda f, t: ["vampire", "--mode", "casc", "-t", str(t), f],
+ "argv": lambda f, t: ["vampire", "--mode", "casc", "--proof", "tptp", "-t", str(t), f],
"install": "brew install vampire (or https://github.com/vprover/vampire)",
},
]
@@ -360,23 +377,66 @@ def unreachable_premise(goal, derivations, expressible, asserted):
def ask(atp, problem, timeout):
- """Run the prover and return (szs_status, detail). Never raises."""
+ """Run the prover and return (szs_status, detail, raw_output). Never raises.
+
+ The raw output is returned, not discarded, because the derivation is in it
+ and the derivation is the only checkable thing a prover produces.
+ """
try:
proc = subprocess.run(
atp["argv"](str(problem), timeout),
capture_output=True, text=True, timeout=timeout + 15,
)
except subprocess.TimeoutExpired:
- return "Timeout", f"{atp['name']} exceeded {timeout + 15}s wall clock"
+ return "Timeout", f"{atp['name']} exceeded {timeout + 15}s wall clock", ""
except OSError as exc:
- return "InputError", f"{type(exc).__name__}: {exc}"
+ return "InputError", f"{type(exc).__name__}: {exc}", ""
text = proc.stdout + proc.stderr
found = SZS.findall(text)
if not found:
tail = text.strip().splitlines()[-3:]
- return "InputError", "no SZS status line: " + " / ".join(tail)[:300]
+ return "InputError", "no SZS status line: " + " / ".join(tail)[:300], text
# The LAST status is the verdict; E prints an input status first.
- return found[-1], ""
+ return found[-1], "", text
+
+
+# The verdicts `fol-prove` can return, mapped to what this differential does
+# with them. A rejection is a stop-the-line: it means the prover's derivation
+# does not rest on the formulas this engine emitted, so the AGREE beside it is
+# about a different file.
+PROOF_STOP = {"derivation_rejected", "refutation_step_not_reconstructed"}
+
+
+def check_proof(problem, output, workdir, tag):
+ """Hand the derivation to the engine's TSTP checker.
+
+ Returns a dict with `verdict`, `steps_checked`, `steps_total` and
+ `leaves_match_problem`, or `None` if the checker could not be run. `None`
+ is reported as NOT_CHECKED and never as agreement: an absent check is the
+ absence of evidence, which is the failure mode this whole tool exists to
+ avoid.
+ """
+ if not output:
+ return None
+ proof_file = workdir / f"{tag}.tstp"
+ proof_file.write_text(output)
+ try:
+ proc = subprocess.run(
+ [BIN, "--no-connect", "fol-prove", "--problem", str(problem),
+ "--proof", str(proof_file)],
+ capture_output=True, text=True, timeout=120,
+ )
+ except (OSError, subprocess.TimeoutExpired) as exc:
+ return {"verdict": "CHECKER_ERROR", "detail": f"{type(exc).__name__}: {exc}"}
+ for line in proc.stdout.splitlines():
+ try:
+ row = json.loads(line)
+ except json.JSONDecodeError:
+ continue
+ if "verdict" in row:
+ return row
+ return {"verdict": "CHECKER_ERROR",
+ "detail": (proc.stdout + proc.stderr).strip()[:300]}
def classify(status):
@@ -415,6 +475,9 @@ def main():
ap.add_argument("--timeout", type=int, default=int(os.environ.get("FOL_DIFF_TIMEOUT", 10)),
help="per-problem CPU limit in seconds (default 10)")
ap.add_argument("--limit", type=int, help="only ask the first N goals")
+ ap.add_argument("--no-proof-check", action="store_true",
+ help="do not read the derivations back. The run then reports what the "
+ "prover SAID and nothing about what it said it about")
ap.add_argument("--json")
args = ap.parse_args()
@@ -476,8 +539,11 @@ def main():
)
rows = []
+ proofs = workdir / "proofs"
+ proofs.mkdir(exist_ok=True)
for g in goals:
- status, detail = ask(atp, fol_dir / "goals" / g["file"], args.timeout)
+ problem_file = fol_dir / "goals" / g["file"]
+ status, detail, output = ask(atp, problem_file, args.timeout)
verdict, why = classify(status)
if verdict == "CLAIMED_NOT_ENTAILED":
try:
@@ -502,19 +568,36 @@ def main():
)
if detail:
why = f"{why} ({detail})" if why else detail
+ # The derivation is checked whenever the prover produced one, not
+ # only when it AGREED. A CLAIMED_NOT_ENTAILED row has no refutation
+ # to check and comes back `no_refutation_offered`, which is the
+ # honest reading of it.
+ check = check_proof(problem_file, output, proofs, pathlib.Path(g["file"]).stem) \
+ if not args.no_proof_check else None
+ proof_verdict = (check or {}).get("verdict", "NOT_CHECKED")
+ if proof_verdict in PROOF_STOP:
+ proof_verdict = "PROOF_REJECTED:" + proof_verdict
+ steps = f"{(check or {}).get('steps_checked', 0)}/{(check or {}).get('steps_total', 0)}"
triple = " ".join(g["triple"])
rows.append({
"file": g["file"], "triple": g["triple"], "axiom_form": g["axiom_form"],
"szs": status, "verdict": verdict, "detail": why,
+ "proof_check": proof_verdict, "proof_steps": steps,
+ "proof_leaves_match_problem": (check or {}).get("leaves_match_problem"),
})
print(f"{verdict:<22} {triple}\n{'':<22} {why}", flush=True)
+ if check is not None:
+ print(f"{'':<22} proof: {proof_verdict} ({steps} steps replayed)", flush=True)
for n in not_asked:
print(f"{'NOT_ASKED':<22} {' '.join(n['triple'])}\n{'':<22} {n['why']}", flush=True)
counts = {}
+ proof_counts = {}
for r in rows:
counts[r["verdict"]] = counts.get(r["verdict"], 0) + 1
+ pv = r.get("proof_check", "NOT_CHECKED")
+ proof_counts[pv] = proof_counts.get(pv, 0) + 1
counts["NOT_ASKED"] = len(not_asked)
print("\n" + "=" * 72)
@@ -530,6 +613,19 @@ def main():
"unification, which does not exist in core Lean.\nThe engine's own derivation "
"certificate is the artefact that CAN be checked, with\n`cd lean && lake exe oo-cert`."
)
+ if proof_counts:
+ print("\nDerivations read back (open-ontologies fol-prove):")
+ for kind, n in sorted(proof_counts.items()):
+ print(f" {kind:<40} {n}")
+ print(
+ "This is NOT a proof and AGREE still means what it meant. What a checked "
+ "derivation\nrules out is that the prover was answering about a different file: "
+ "every leaf is matched\nagainst the problem this engine emitted, the DAG is "
+ "checked for dangling parents and cycles,\nand the resolution-family steps are "
+ "recomputed. Clausification is NOT checked and is named\nas unchecked in each "
+ "report."
+ )
+
if counts.get("WEAKER_EXPORT"):
print(
"\nWEAKER_EXPORT is not a defect in the engine or in the prover. It is the "
@@ -543,11 +639,19 @@ def main():
if args.json:
with open(args.json, "w") as fh:
json.dump({"atp": atp["name"], "rows": rows, "not_asked": not_asked,
- "counts": counts}, fh, indent=1)
+ "counts": counts, "proof_counts": proof_counts}, fh, indent=1)
# A conclusion the engine claims and the ATP refutes is the one result that
# must fail a pipeline. An undetermined run is not a pass and not a failure.
- return 1 if counts.get("CLAIMED_NOT_ENTAILED") or counts.get("ATP_ERROR") else 0
+ rejected = sum(n for k, n in proof_counts.items() if k.startswith("PROOF_REJECTED"))
+ if rejected:
+ print(
+ f"\n{rejected} derivation(s) REJECTED by the checker. That means the prover's own "
+ "proof does not\nrest on the formulas this engine emitted, so the verdict beside "
+ "it is about some other file.\nThis is a stop-the-line and it exits 1."
+ )
+ return 1 if (counts.get("CLAIMED_NOT_ENTAILED") or counts.get("ATP_ERROR")
+ or rejected) else 0
if __name__ == "__main__":