Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

51.0 KB
Raw Blame
//! Knowledge packs: a verified graph, its evidence and its proof in one
//! portable file.
//!
//! TrustGraph calls the idea a knowledge core, and it is the right one: what
//! you promote between environments should be a versioned artifact, not a
//! pile of loose Turtle whose provenance and checks live somewhere else. The
//! shape here is deliberately boring so that anything can read it, and the
//! graph is stored as ordinary N-Triples rather than a proprietary blob.
//!
//! A pack carries:
//!
//!   - the graph itself (N-Triples, sorted, so two packs of the same graph
//!     are byte-identical and diffable);
//!   - a manifest: name, version, counts, creation time, tool version;
//!   - a checksum over the graph, so tampering or truncation is detectable;
//!   - the verification evidence recorded at pack time (lint, enforce), so
//!     the receiving environment can see what the graph passed rather than
//!     take it on trust;
//!   - and, when the sender has one, the derivation certificate for the run
//!     that produced the graph, plus a second checksum over the graph AND the
//!     certificate together.
//!
//! # Evidence and proof are different objects
//!
//! `evidence` is what the SENDER'S engine said. A receiver who reads it is
//! trusting a report. The certificate is not a report: it is the premises and
//! the derivation steps, and the receiver runs their own checker over them.
//! The engine that wrote the pack is then out of the trusted set for that one
//! claim, which is why the two live in separate fields rather than being
//! folded together.
//!
//! What a certificate settles and what it does not is stated in the RECEIVER'S
//! report rather than only here, in [`MEANS`], [`DOES_NOT_MEAN`] and
//! [`ABSENT_MEANS`]. The short form: it proves the materialised triples follow
//! from the asserted ones under the rules that ran. It proves nothing about
//! whether the asserted triples are true, and it says nothing about axioms no
//! rule in the fragment can read.
//!
//! Unpacking verifies both digests before loading a single triple, and a
//! certificate this machine's checker REFUSED stops the load unless the caller
//! asks for it anyway.

use crate::graph::{GraphStore, ReadScope};
use crate::projection_entailment::{self as pe, CertKind, CheckerStatus, LAKE_INSTALL};
use crate::verdict::Certified;
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, BTreeSet};
use std::path::{Path, PathBuf};
use std::sync::Arc;

/// The only file names a pack may carry inside a certificate, in BOTH
/// directions.
///
/// At pack time a certificate directory can hold whatever an earlier run left
/// in it, and slurping the directory would put unrelated bytes under the
/// content digest. At unpack time the names come from the SENDER, and a pack
/// naming `../../.ssh/authorized_keys` would otherwise be written wherever the
/// receiver's scratch directory resolves that to. Every name here is one
/// `Reasoner::run_full` writes or one `Reasoner::run_horn_scoped` writes.
const CERTIFICATE_FILES: &[&str] = &[
    "asserted.tsv",
    "asserted.sha256",
    "derivations.tsv",
    "horn.tsv",
    "rules.tsv",
    "scope.tsv",
    "refutation.tsv",
];

/// A certificate directory, carried inside a pack.
///
/// The files travel as text rather than as base64 because they are TSV and a
/// pack is meant to be diffable, which is the same reason the graph is sorted
/// N-Triples. Bytes are preserved exactly: nothing here normalises a line
/// ending, because the checker reads these bytes and a rewritten `\r\n` would
/// be a different file from the one the reasoner wrote and hashed.
///
/// ONE certificate, and the field is singular in the format rather than a list.
/// A real promotion may involve several runs, an RDFS pass and then a supplied
/// Horn table, and making this a list would be cheap now and expensive later.
/// It is singular because a second certificate raises a question nobody can
/// answer honestly yet: what the COMBINED verdict is when one is accepted and
/// the other refused. Until there is an answer to that, a pack that needs two
/// proofs is two packs.
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
pub struct PackedCertificate {
    /// `oo-cert/1` or `oo-horn/1`, echoed from the reasoner's report. ECHOED
    /// AND NEVER ACTED ON: [`kind_of`] decides which checker reads this from
    /// the files that are present, so a sender cannot point the receiver at
    /// the wrong checker by editing a string.
    pub format: String,
    /// File name to contents. A `BTreeMap` because [`content_digest`] frames
    /// the entries in iteration order, and two orders would be two digests for
    /// one certificate.
    pub files: BTreeMap<String, String>,
    /// Lines in `asserted.tsv` and in the derivations file. Under the content
    /// digest like everything else, so a reader can size the certificate before
    /// writing a byte of it to disk and still be reading a number nobody
    /// edited.
    pub asserted: usize,
    pub derivations: usize,
    /// The profile or rule file the SENDER says produced this. Nothing in the
    /// certificate proves it and no checker reads it: `oo-cert` checks each
    /// step against the rule the step itself names, and `oo-horn` against
    /// `rules.tsv`. It is recorded because it is useful and it is labelled a
    /// sender's claim everywhere it is printed.
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub profile_claimed_by_sender: Option<String>,
}

#[derive(serde::Serialize, serde::Deserialize)]
pub struct Manifest {
    pub name: String,
    pub version: String,
    pub created_at: String,
    pub tool_version: String,
    pub triples: usize,
    /// SHA-256 of the graph text and NOTHING ELSE. Unchanged, including for a
    /// pack that carries a certificate, so a reader who checked packs before
    /// this field existed is still checking the same thing.
    pub sha256: String,
    /// What the graph passed at pack time. Absent means not checked, which
    /// is itself information the receiver should act on.
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub evidence: Option<serde_json::Value>,
    /// The derivation certificate for the run that produced this graph.
    ///
    /// `evidence` above is what the sender's engine SAID. This is what a
    /// receiver's own checker can re-derive, which is a different kind of
    /// object, so it gets its own field rather than being folded into the
    /// evidence blob.
    ///
    /// `#[serde(default)]` here is explicitness and NOT the backward
    /// compatibility guarantee, which is worth writing down because it is easy
    /// to believe otherwise. serde's derive already reads a missing field of
    /// type `Option<T>` as `None`, so deleting the attribute changes nothing:
    /// that was mutated and the old-pack test stayed green. What actually keeps
    /// an old pack loading is the `(None, None)` arm of the digest match in
    /// [`Packer::unpack`], and
    /// `pack_certificate_test::an_old_pack_loads_and_reports_that_it_carries_no_certificate`
    /// is what turns red when that arm starts refusing.
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub certificate: Option<PackedCertificate>,
    /// SHA-256 over the graph AND the certificate, framed by
    /// [`content_digest`].
    ///
    /// THE BINDING, stated precisely. `content_sha256` is the digest of, in
    /// order: a domain-separating tag, the length and bytes of the graph text,
    /// a presence byte for the certificate and, when one is present, the
    /// length-prefixed `format`, the length-prefixed
    /// `profile_claimed_by_sender`, the `asserted` and `derivations` counts,
    /// the number of files, and then every (name, body) pair in `BTreeMap`
    /// order, each half length-prefixed. So it covers every field of
    /// [`PackedCertificate`] and every byte of every file in it.
    ///
    /// `sha256` cannot see a swapped certificate, and a swapped certificate is
    /// the whole attack: a certificate that is internally sound over SOMEBODY
    /// ELSE'S assertions checks green, and a receiver that verified only the
    /// graph checksum would accept a true proof about premises that are not in
    /// the pack. Absent on every pack written before this field existed, which
    /// is why a pack that carries a certificate and no `content_sha256` is
    /// refused rather than checked (see [`Packer::unpack`]).
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub content_sha256: Option<String>,
}

#[derive(serde::Serialize, serde::Deserialize)]
pub struct Pack {
    pub manifest: Manifest,
    /// N-Triples, sorted line-wise.
    pub graph: String,
}

fn checksum(text: &str) -> String {
    let mut hasher = Sha256::new();
    hasher.update(text.as_bytes());
    format!("{:x}", hasher.finalize())
}

/// The digest that binds a certificate to the graph it travels with.
///
/// Every part is length-prefixed, so no two different (graph, certificate)
/// pairs can frame to the same bytes by running together at a seam. That is the
/// framing [`crate::verdict::subject_digest`] uses, for the same reason, and it
/// is written out explicitly rather than taken from `serde_json::to_string` so
/// that a serde_json release cannot change what a security digest covers.
///
/// Public because the comparison is the point, and because a test that plays a
/// competent attacker has to be able to recompute it.
pub fn content_digest(graph: &str, cert: Option<&PackedCertificate>) -> String {
    let mut h = Sha256::new();
    h.update(b"oo-pack-content/1\n");
    h.update((graph.len() as u64).to_le_bytes());
    h.update(graph.as_bytes());
    match cert {
        None => h.update(0u64.to_le_bytes()),
        Some(c) => {
            h.update(1u64.to_le_bytes());
            for s in [
                c.format.as_str(),
                c.profile_claimed_by_sender.as_deref().unwrap_or(""),
            ] {
                h.update((s.len() as u64).to_le_bytes());
                h.update(s.as_bytes());
            }
            h.update((c.asserted as u64).to_le_bytes());
            h.update((c.derivations as u64).to_le_bytes());
            h.update((c.files.len() as u64).to_le_bytes());
            for (name, body) in &c.files {
                h.update((name.len() as u64).to_le_bytes());
                h.update(name.as_bytes());
                h.update((body.len() as u64).to_le_bytes());
                h.update(body.as_bytes());
            }
        }
    }
    format!("{:x}", h.finalize())
}

/// What a receiver learned about the certificate in a pack. NEVER COLLAPSED.
///
/// The two accepted variants carry a [`Certified`], which has no public
/// constructor anywhere in the crate, so neither can be named on a path that
/// did not run a checker and read a zero exit code off it.
/// [`PackCertVerdict::CheckerAbsent`] and
/// [`PackCertVerdict::NoCertificateInPack`] are separate words on purpose:
/// "this build has no checker" and "this pack has no proof" are different
/// facts, and neither may render as anything that reads like a pass.
///
/// ```compile_fail
/// use open_ontologies::pack::PackCertVerdict;
/// use open_ontologies::verdict::Certified;
/// let v = PackCertVerdict::Accepted(Certified {
///     theorem: "OOCert.certificate_sound",
///     subject: [0u8; 32],
/// });
/// ```
#[derive(Clone, Debug)]
pub enum PackCertVerdict {
    /// CERTIFIED. `oo-cert` accepted the packed certificate under
    /// `OOCert.certificate_sound`.
    Accepted(Certified),
    /// CERTIFIED AND WEAKER. `oo-horn check` accepted a certificate over a rule
    /// table the SENDER supplied, so the conclusions hold in every model of the
    /// asserted triples that also satisfies that table. Never shortened to the
    /// word above.
    AcceptedUnderSuppliedRules(Certified),
    /// Exit 1. The checker named a step it would not accept. A defect in the
    /// pack, never a downgrade to an unchecked verdict.
    Refused,
    /// Exit 2, an unreadable file, an unknown file name, or files that do not
    /// form a certificate either checker reads. Not a verdict in either
    /// direction, exactly as `lean/Main.lean` reserves the two codes.
    Unreadable,
    /// The certificate checks and is about assertions this pack does not
    /// contain. A sound proof over premises nobody shipped.
    NotAboutThisPack,
    /// No checker on this machine. NOTHING WAS CHECKED.
    CheckerAbsent,
    /// The pack carries no certificate at all. Every pack written before this
    /// field existed lands here.
    NoCertificateInPack,
}

impl PackCertVerdict {
    pub fn word(&self) -> &'static str {
        match self {
            PackCertVerdict::Accepted(_) => "certificate_accepted",
            PackCertVerdict::AcceptedUnderSuppliedRules(_) => {
                "certificate_accepted_under_supplied_rules"
            }
            PackCertVerdict::Refused => "certificate_refused",
            PackCertVerdict::Unreadable => "certificate_unreadable",
            PackCertVerdict::NotAboutThisPack => "certificate_not_about_this_pack",
            PackCertVerdict::CheckerAbsent => "checker_absent_nothing_was_checked",
            PackCertVerdict::NoCertificateInPack => "no_certificate_in_pack",
        }
    }
    pub fn theorem(&self) -> Option<&'static str> {
        match self {
            PackCertVerdict::Accepted(c) | PackCertVerdict::AcceptedUnderSuppliedRules(c) => {
                Some(c.theorem())
            }
            _ => None,
        }
    }
    /// True only for the two variants a checker had to accept to produce.
    pub fn is_checked(&self) -> bool {
        self.theorem().is_some()
    }
    /// Whether this verdict stops the graph being loaded. A refused or
    /// unreadable certificate, and one that is about another graph, are defects
    /// in the pack: a graph whose own proof does not check is not a graph to
    /// promote. An absent checker is not a defect and does not block: refusing
    /// the load there would make a proof-carrying pack unusable on every
    /// machine without Lean, which is the opposite of the point.
    pub fn blocks_the_load(&self) -> bool {
        matches!(
            self,
            PackCertVerdict::Refused
                | PackCertVerdict::Unreadable
                | PackCertVerdict::NotAboutThisPack
        )
    }
    /// Every word, for a suite that enumerates the vocabulary without being
    /// able to build the certified variants.
    pub const WORDS: [&'static str; 7] = [
        "certificate_accepted",
        "certificate_accepted_under_supplied_rules",
        "certificate_refused",
        "certificate_unreadable",
        "certificate_not_about_this_pack",
        "checker_absent_nothing_was_checked",
        "no_certificate_in_pack",
    ];
}

// Written by hand: `serialize_as_word!` is private to src/verdict.rs.
impl serde::Serialize for PackCertVerdict {
    fn serialize<S: serde::Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
        s.serialize_str(self.word())
    }
}

impl std::fmt::Display for PackCertVerdict {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        f.write_str(self.word())
    }
}

/// What an acceptance settles, in the report rather than in the documentation.
///
/// Held as a constant because the same words belong in `onto_unpack`, in the
/// CLI output and in the decision record, and a claim that lives in four copies
/// drifts in three of them.
pub const MEANS: &str = "A checker on THIS machine re-derived every conclusion in this pack's \
     certificate from the triples the certificate lists as asserted, and accepted all of them. \
     You did not have to trust the sender's engine to learn that. To repeat it by hand, unpack \
     again with certificate_out_dir set and run the command in check_with over the files it \
     keeps.";

/// What an acceptance does NOT settle. Printed beside every accepted verdict.
pub const DOES_NOT_MEAN: &str = "The asserted triples themselves are NOT proved. A certificate \
     proves that the materialised triples follow from the asserted ones under the rules that ran, \
     and nothing more. It does not check that the asserted triples are true, that they describe \
     the world, that they came from the system the sender says they came from, or that the sender \
     left anything out. It is also silent about axioms outside the fragment: a profile run \
     evaluates part of OWL 2 RL, a supplied table is whatever rules.tsv holds, and an axiom no \
     rule fires on contributed nothing and is invisible here. Run onto_dlp_boundary over the \
     loaded graph to see which of its axioms the rule table can read. A sound proof over the \
     wrong premises is still the wrong answer.";

/// What an absent checker settles, which is nothing.
///
/// The wording follows `web/try/api/index.py`, which says the release carries
/// no checker rather than implying a check happened. "We did not check" must
/// never render as "it passed".
pub const ABSENT_MEANS: &str = "No checker ran here, so NOTHING WAS CHECKED. That is not a pass \
     and it is not a failure: no statement in this report rests on a checker having run.";

/// The limit of what running the checker buys, printed beside every checker
/// result.
///
/// The receiver chose the binary. `$OO_CERT` pointing at a script that exits
/// zero and prints the right theorem name earns the accepted word, exactly as
/// `src/verdict.rs` residual hole 1 says. That is now the RECEIVER'S own foot
/// rather than the sender's, which is an improvement and not a closure, and the
/// report says so rather than leaving it in a comment.
pub const CHECKER_LIMIT: &str = "The checker is a binary THIS machine chose. A path in $OO_CERT \
     that exits zero and prints the right theorem name earns the accepted word, so this verdict \
     is only as good as the binary behind it. checker_says_it_is below is the block the binary \
     printed about itself; match its sha256 against the SHASUMS.txt of the release you meant to \
     run. A hostile binary can print any block it likes and nothing here would notice. This is \
     the limit of what running a checker buys you, and it is the receiver's own risk rather than \
     the sender's, which is the improvement.";

/// Which checker a packed certificate is for, decided by the files it carries.
///
/// `run_full` writes `derivations.tsv` and `run_horn_scoped` writes `horn.tsv`
/// plus the `rules.tsv` the run evaluated, so the file names are the
/// discriminator. The `format` field is not consulted: reading it would let a
/// sender choose the receiver's checker.
fn kind_of(files: &BTreeMap<String, String>) -> anyhow::Result<CertKind> {
    anyhow::ensure!(
        files.contains_key("asserted.tsv"),
        "this certificate carries no asserted.tsv, so there is nothing for a checker to check \
         the derivations against"
    );
    match (
        files.contains_key("derivations.tsv"),
        files.contains_key("horn.tsv"),
    ) {
        (true, false) => Ok(CertKind::OoCert),
        (false, true) => {
            anyhow::ensure!(
                files.contains_key("rules.tsv"),
                "horn.tsv is checked against the rules.tsv the run evaluated and this certificate \
                 carries none, so there is no table for oo-horn to check it against"
            );
            Ok(CertKind::OoHorn)
        }
        (true, true) => anyhow::bail!(
            "this certificate carries both derivations.tsv and horn.tsv, so two runs wrote into \
             one directory. Which checker reads it would be decided here rather than by the \
             sender, and the two earn different verdicts, so it is refused"
        ),
        (false, false) => anyhow::bail!(
            "this certificate carries neither derivations.tsv nor horn.tsv, so it records no \
             derivation"
        ),
    }
}

/// Read a certificate directory into the form a pack carries.
pub fn read_certificate_dir(
    dir: &Path,
    profile_claimed: Option<&str>,
) -> anyhow::Result<PackedCertificate> {
    anyhow::ensure!(
        dir.is_dir(),
        "{} is not a directory, so there is no certificate to pack. Run onto_reason with \
         certificate_dir first",
        dir.display()
    );
    let mut files: BTreeMap<String, String> = BTreeMap::new();
    for name in CERTIFICATE_FILES {
        let p = dir.join(name);
        if !p.exists() {
            continue;
        }
        let bytes = std::fs::read(&p)?;
        // Not `from_utf8_lossy`. A pack is JSON, and a lossy conversion would
        // hand the receiver's checker different bytes from the ones the
        // reasoner wrote and hashed into asserted.sha256.
        let body = String::from_utf8(bytes).map_err(|e| {
            anyhow::anyhow!(
                "{} is not valid UTF-8 ({e}), so carrying it in a pack would change the bytes \
                 the checker reads",
                p.display()
            )
        })?;
        files.insert((*name).to_string(), body);
    }
    let kind = kind_of(&files)?;
    let asserted = files["asserted.tsv"]
        .lines()
        .filter(|l| !l.is_empty())
        .count();
    let derivations = files[kind.derivations_file()]
        .lines()
        .filter(|l| !l.is_empty())
        .count();
    Ok(PackedCertificate {
        format: match kind {
            CertKind::OoCert => "oo-cert/1".to_string(),
            CertKind::OoHorn => "oo-horn/1".to_string(),
        },
        files,
        asserted,
        derivations,
        profile_claimed_by_sender: profile_claimed.map(str::to_string),
    })
}

/// How much of a certificate the graph in front of us accounts for.
///
/// Two counts, and they answer two different questions. `in_certificate_only`
/// is the one that can refuse a pack: an assertion the certificate rests on
/// that is not in the graph means the proof is about premises nobody shipped,
/// and the checker cannot see that, because `oo-cert` verifies the steps
/// against the triples in `asserted.tsv` and has no way to ask where they came
/// from. `conclusions_not_in_pack` is reported and never refuses: a sender who
/// reasoned without materialising ships a graph that does not contain its own
/// conclusions, which is a legitimate thing to do and a thing the receiver
/// should be told.
struct Coverage {
    asserted_in_certificate: usize,
    in_certificate_only: Vec<String>,
    in_certificate_only_total: usize,
    conclusions: usize,
    conclusions_not_in_pack: Vec<String>,
    conclusions_not_in_pack_total: usize,
    unparsed_derivation_lines: usize,
}

/// The conclusions a certificate records, in the same spelling `asserted.tsv`
/// uses.
///
/// `derivations.tsv` is `rule` then 3(1+k) term fields, so the conclusion is
/// fields 1..4. `horn.tsv` is
/// `ruleIndex, bindCount, (var, term)*, cs, cp, co, premises*`, so the
/// conclusion starts at 2 + 2 * bindCount. A line whose shape does not parse is
/// COUNTED and never skipped in silence: a swallowed parse failure would let a
/// conclusion escape the coverage count, and the count would then be a number
/// that cannot go wrong.
fn conclusions_of(files: &BTreeMap<String, String>, kind: CertKind) -> (BTreeSet<String>, usize) {
    let mut out = BTreeSet::new();
    let mut unparsed = 0usize;
    for line in files[kind.derivations_file()].lines() {
        if line.is_empty() {
            continue;
        }
        let f: Vec<&str> = line.split('\t').collect();
        let at = match kind {
            CertKind::OoCert => 1usize,
            CertKind::OoHorn => match f.get(1).and_then(|k| k.parse::<usize>().ok()) {
                Some(k) => 2 + 2 * k,
                None => {
                    unparsed += 1;
                    continue;
                }
            },
        };
        match (f.get(at), f.get(at + 1), f.get(at + 2)) {
            (Some(s), Some(p), Some(o)) => {
                out.insert(format!("{s}\t{p}\t{o}"));
            }
            _ => unparsed += 1,
        }
    }
    (out, unparsed)
}

/// Compare a certificate against a graph, in the ONE spelling both sides use.
///
/// [`crate::reason::asserted_bytes`] is the function that turns a selection of
/// triples into `asserted.tsv` bytes, and the certificate's own `asserted.tsv`
/// was written by the same code, so the two sets are comparable as sets of
/// lines and no second spelling is introduced here.
fn coverage(
    graph: &Arc<GraphStore>,
    files: &BTreeMap<String, String>,
    kind: CertKind,
) -> anyhow::Result<Coverage> {
    let (triples, _) = graph.triples_in_scope(&ReadScope::AllGraphs)?;
    let bytes = crate::reason::asserted_bytes(&triples)?;
    let in_graph: BTreeSet<String> = String::from_utf8_lossy(&bytes)
        .lines()
        .map(str::to_string)
        .collect();
    let in_cert: BTreeSet<String> = files["asserted.tsv"]
        .lines()
        .filter(|l| !l.is_empty())
        .map(str::to_string)
        .collect();
    let (concl, unparsed) = conclusions_of(files, kind);
    let missing_premises: Vec<String> = in_cert.difference(&in_graph).cloned().collect();
    let missing_conclusions: Vec<String> = concl.difference(&in_graph).cloned().collect();
    Ok(Coverage {
        asserted_in_certificate: in_cert.len(),
        in_certificate_only_total: missing_premises.len(),
        in_certificate_only: missing_premises.into_iter().take(8).collect(),
        conclusions: concl.len(),
        conclusions_not_in_pack_total: missing_conclusions.len(),
        conclusions_not_in_pack: missing_conclusions.into_iter().take(8).collect(),
        unparsed_derivation_lines: unparsed,
    })
}

/// A store holding exactly the triples a pack's graph text carries.
///
/// Asked of a store of its own rather than of the caller's, because the
/// question is about the graph IN THE PACK and asking it of the caller's store
/// would answer a different question over whatever was already loaded.
fn stage(graph_text: &str) -> anyhow::Result<Arc<GraphStore>> {
    let staged = Arc::new(GraphStore::new());
    staged.load_ntriples(graph_text)?;
    Ok(staged)
}

/// What to pack. A struct rather than six positional arguments, because
/// `pack("a", "b", "c", None, None, None)` is a line nobody can read.
pub struct PackRequest<'a> {
    pub path: &'a str,
    pub name: &'a str,
    pub version: &'a str,
    pub evidence: Option<serde_json::Value>,
    /// A directory a previous `reason --certificate DIR` wrote. `None` packs no
    /// certificate, which is what every pack written before this field existed
    /// carries.
    ///
    /// The certificate is taken from a run that HAPPENED rather than produced
    /// here. Packing would otherwise have to choose the profile and the
    /// materialisation target itself, and the certificate would then describe a
    /// run nobody asked for.
    pub certificate_dir: Option<&'a Path>,
    pub profile_claimed: Option<&'a str>,
}

/// What the receiver is allowed to decide.
#[derive(Default)]
pub struct UnpackOptions {
    pub verify_only: bool,
    /// Off skips the checker and reports `checker_absent_nothing_was_checked`
    /// with `skipped_by_caller: true`. There is no option that makes an
    /// unchecked run print a checked word.
    pub check_certificate: bool,
    /// An explicit checker path, passed through to
    /// [`crate::projection_entailment::find_checker`]. Handed in rather than
    /// read from `$OO_CERT` here so that a test can point at a fake without
    /// mutating the environment of a parallel test.
    pub checker: Option<PathBuf>,
    /// Keep the certificate files here instead of in a scratch directory that
    /// is removed, so a receiver can re-run the checker by hand.
    pub certificate_out_dir: Option<PathBuf>,
    /// Load the graph even when the certificate did not check. Off by default:
    /// a graph whose own proof is refused is a defect, not something to
    /// promote. This changes behaviour only for packs that carry a
    /// certificate, so no pack written before this existed is affected.
    pub load_even_if_refused: bool,
}

pub struct Packer {
    graph: Arc<GraphStore>,
}

impl Packer {
    pub fn new(graph: Arc<GraphStore>) -> Self {
        Self { graph }
    }

    /// Write the loaded graph, its evidence and, when one is given, the
    /// certificate for the run that produced it.
    pub fn pack(&self, req: &PackRequest<'_>) -> anyhow::Result<String> {
        let nt = self.graph.serialize("ntriples")?;

        // Sorted, so the same graph always produces the same bytes: packs
        // become diffable and the checksum becomes meaningful across
        // machines rather than reflecting store iteration order.
        let mut lines: Vec<&str> = nt.lines().filter(|l| !l.trim().is_empty()).collect();
        lines.sort_unstable();
        let graph = lines.join("\n") + "\n";

        let certificate = match req.certificate_dir {
            None => None,
            Some(dir) => {
                let cert = read_certificate_dir(dir, req.profile_claimed)?;
                // Refused at pack time, not warned about. A certificate whose
                // assertions are not all in the graph being packed produces an
                // artefact whose proof is about premises the pack does not
                // contain, and a receiver's checker would accept it. Catching
                // it here costs one set difference and no Lean toolchain, so
                // the sender learns it rather than the auditor. The receiver
                // refuses it too: the receiver must not depend on the sender
                // having checked.
                //
                // Asked of a store staged from THE PACKED TEXT rather than of
                // the live store, so the sender computes the identical set the
                // receiver will compute. Asking the live store a slightly
                // different question is how a pack-time pass and an unpack-time
                // refusal end up in the same artefact.
                let kind = kind_of(&cert.files)?;
                let staged = stage(&graph)?;
                let c = coverage(&staged, &cert.files, kind)?;
                anyhow::ensure!(
                    c.in_certificate_only.is_empty(),
                    "this certificate rests on {} assertion(s) that are not in the graph being \
                     packed, for example {:?}. A checker would accept it, because oo-cert \
                     verifies the steps against the triples inside the certificate and cannot \
                     ask where they came from. Reason and pack the same graph",
                    c.in_certificate_only_total,
                    c.in_certificate_only.first()
                );
                Some(cert)
            }
        };

        let manifest = Manifest {
            name: req.name.to_string(),
            version: req.version.to_string(),
            created_at: chrono::Utc::now().to_rfc3339(),
            tool_version: env!("CARGO_PKG_VERSION").to_string(),
            triples: lines.len(),
            sha256: checksum(&graph),
            evidence: req.evidence.clone(),
            content_sha256: Some(content_digest(&graph, certificate.as_ref())),
            certificate,
        };

        let pack = Pack { manifest, graph };
        // The whole pack is built in memory, certificate included, and there is
        // no streaming path. A large store's certificate makes a very large
        // pack, so the byte counts below are reported rather than left for a
        // receiver to discover when the file will not fit.
        let json = serde_json::to_string_pretty(&pack)?;
        std::fs::write(req.path, &json)?;

        Ok(serde_json::json!({
            "ok": true,
            "path": req.path,
            "name": pack.manifest.name,
            "version": pack.manifest.version,
            "triples": pack.manifest.triples,
            "sha256": pack.manifest.sha256,
            "content_sha256": pack.manifest.content_sha256,
            "content_sha256_covers": "the graph text and every field and file of the certificate, \
                                      each length-prefixed. sha256 above covers the graph alone \
                                      and cannot see a swapped certificate",
            "has_evidence": pack.manifest.evidence.is_some(),
            "certificate": pack.manifest.certificate.as_ref().map(|c| serde_json::json!({
                "format": c.format,
                "files": c.files.keys().collect::<Vec<_>>(),
                "asserted": c.asserted,
                "derivations": c.derivations,
                "bytes": c.files.values().map(|v| v.len()).sum::<usize>(),
                "profile_claimed_by_sender": c.profile_claimed_by_sender,
                "checked_by_this_packer": false,
                "means": "the certificate travels with the graph and the content digest covers \
                          both. NOTHING HERE HAS CHECKED IT: the receiver runs its own checker, \
                          which is the point of shipping it",
            })),
            "graph_bytes": pack.graph.len(),
            "pack_json_bytes": json.len(),
            "size_means": "the pack is assembled in memory with no streaming path, so \
                           pack_json_bytes is also roughly the peak cost of writing it and of \
                           reading it back",
        })
        .to_string())
    }

    /// Load a pack, refusing it if a digest does not match, and re-check any
    /// certificate it carries with THIS machine's checker.
    pub fn unpack(&self, path: &str, opts: &UnpackOptions) -> anyhow::Result<String> {
        let raw = std::fs::read_to_string(path)?;
        let pack: Pack = serde_json::from_str(&raw)?;

        let actual = checksum(&pack.graph);
        if actual != pack.manifest.sha256 {
            return Ok(serde_json::json!({
                "error": "checksum mismatch: the pack was modified or truncated after it was written",
                "expected": pack.manifest.sha256,
                "actual": actual,
                "verified": false,
                "loaded": false,
            })
            .to_string());
        }

        // The digest above covers the graph and nothing else. A pack that
        // carries a certificate and no digest over both bodies is REFUSED
        // rather than checked: dropping `content_sha256` and pasting in a
        // certificate that is sound over another graph leaves `sha256`
        // matching, and there would be nothing left to notice with.
        match (&pack.manifest.certificate, &pack.manifest.content_sha256) {
            (Some(_), None) => {
                return Ok(serde_json::json!({
                    "error": "this pack carries a certificate and no content_sha256 binding it to \
                              the graph. The graph checksum cannot see a swapped certificate, so \
                              an unbound one is refused rather than checked",
                    "verified": false,
                    "loaded": false,
                    "certificate": {
                        "present": true,
                        "verdict": PackCertVerdict::Refused,
                        "means": ABSENT_MEANS,
                    },
                })
                .to_string());
            }
            (cert, Some(recorded)) => {
                let recomputed = content_digest(&pack.graph, cert.as_ref());
                if *recorded != recomputed {
                    return Ok(serde_json::json!({
                        "error": "content checksum mismatch: the graph and the certificate in this \
                                  pack are not the pair that was written. One of them was replaced \
                                  afterwards",
                        "expected": recorded,
                        "actual": recomputed,
                        "verified": false,
                        "loaded": false,
                    })
                    .to_string());
                }
            }
            // An old pack: no certificate and no content digest. It loads, and
            // the report says it carries no proof rather than failing.
            (None, None) => {}
        }

        let (verdict, report) = match &pack.manifest.certificate {
            None => (
                PackCertVerdict::NoCertificateInPack,
                serde_json::json!({
                    "present": false,
                    "verdict": PackCertVerdict::NoCertificateInPack,
                    "theorem": serde_json::Value::Null,
                    "means": "this pack carries no certificate, so there is nothing here to \
                              re-derive. Its evidence is what the sender's engine said about the \
                              graph and this receiver has re-run none of it",
                }),
            ),
            Some(cert) => self.check_packed_certificate(&pack, cert, opts)?,
        };

        if opts.verify_only {
            return Ok(serde_json::json!({
                "ok": true,
                "verified": true,
                "loaded": false,
                "manifest": pack.manifest,
                "certificate": report,
            })
            .to_string());
        }

        if verdict.blocks_the_load() && !opts.load_even_if_refused {
            return Ok(serde_json::json!({
                "ok": false,
                "verified": true,
                "loaded": false,
                "why_not_loaded": "the certificate in this pack did not check on this machine, so \
                                   nothing was loaded. A graph whose own proof is refused is a \
                                   defect, not something to promote. Pass load_even_if_refused to \
                                   load it anyway and inspect it",
                "manifest": pack.manifest,
                "certificate": report,
            })
            .to_string());
        }

        let loaded = self.graph.load_ntriples(&pack.graph)?;
        Ok(serde_json::json!({
            "ok": true,
            "verified": true,
            "loaded": true,
            "triples_loaded": loaded,
            "manifest": pack.manifest,
            "certificate": report,
        })
        .to_string())
    }

    fn check_packed_certificate(
        &self,
        pack: &Pack,
        cert: &PackedCertificate,
        opts: &UnpackOptions,
    ) -> anyhow::Result<(PackCertVerdict, serde_json::Value)> {
        let base = |v: &PackCertVerdict, extra: serde_json::Value| {
            let mut o = serde_json::json!({
                "present": true,
                "verdict": v.clone(),
                "theorem": v.theorem(),
                "checked_here": v.is_checked(),
                "format_claimed_by_sender": cert.format,
                "profile_claimed_by_sender": cert.profile_claimed_by_sender,
                "asserted": cert.asserted,
                "derivations": cert.derivations,
                "certificate_bytes": cert.files.values().map(|v| v.len()).sum::<usize>(),
            });
            if let (Some(a), Some(b)) = (o.as_object_mut(), extra.as_object()) {
                for (k, val) in b {
                    a.insert(k.clone(), val.clone());
                }
            }
            o
        };

        let kind = match kind_of(&cert.files) {
            Ok(k) => k,
            Err(e) => {
                let v = PackCertVerdict::Unreadable;
                let r = base(
                    &v,
                    serde_json::json!({"why": e.to_string(), "means": ABSENT_MEANS}),
                );
                return Ok((v, r));
            }
        };

        if !opts.check_certificate {
            let v = PackCertVerdict::CheckerAbsent;
            let r = base(
                &v,
                serde_json::json!({
                    "skipped_by_caller": true,
                    "means": ABSENT_MEANS,
                    "why": "the caller asked for no check, so nothing was checked. This is the \
                            same word an absent checker earns, because the two produce the same \
                            amount of evidence",
                }),
            );
            return Ok((v, r));
        }

        // The checker reads files, so the pack's strings go to disk. The name
        // joined is the WHITELIST's own literal and never the sender's string,
        // so a crafted name cannot escape this directory even if the equality
        // test below were wrong.
        let dir = match &opts.certificate_out_dir {
            Some(d) => d.clone(),
            None => scratch_dir(),
        };
        // A read-only filesystem with no writable temp directory is a reason to
        // report that nothing was checked, not a reason to fail an unpack that
        // used to succeed.
        if let Err(e) = std::fs::create_dir_all(&dir) {
            let v = PackCertVerdict::Unreadable;
            let r = base(
                &v,
                serde_json::json!({
                    "why": format!(
                        "the certificate could not be written to {} ({e}), so no checker could \
                         read it", dir.display()
                    ),
                    "means": ABSENT_MEANS,
                }),
            );
            return Ok((v, r));
        }
        for name in cert.files.keys() {
            let Some(known) = CERTIFICATE_FILES.iter().find(|k| *k == name) else {
                let v = PackCertVerdict::Unreadable;
                let r = base(
                    &v,
                    serde_json::json!({
                        "why": format!(
                            "this pack names a certificate file this build does not write: \
                             {name:?}. It was not written to disk"
                        ),
                        "allowed": CERTIFICATE_FILES,
                        "means": ABSENT_MEANS,
                    }),
                );
                return Ok((v, r));
            };
            if let Err(e) = std::fs::write(dir.join(*known), &cert.files[*known]) {
                let v = PackCertVerdict::Unreadable;
                let r = base(
                    &v,
                    serde_json::json!({
                        "why": format!("{known} could not be written to {} ({e})", dir.display()),
                        "means": ABSENT_MEANS,
                    }),
                );
                return Ok((v, r));
            }
        }

        let asserted = dir.join("asserted.tsv");
        let derivations = dir.join(kind.derivations_file());
        let rules = (kind == CertKind::OoHorn).then(|| dir.join("rules.tsv"));
        let status = pe::run_checker(
            kind,
            opts.checker.as_deref(),
            &asserted,
            &derivations,
            rules.as_deref(),
        );

        // The question is about the graph IN THE PACK, and it is asked BEFORE
        // anything is loaded, so a pack that is about to be refused never
        // touches the receiver's graph.
        let staged = stage(&pack.graph)?;
        let cover = coverage(&staged, &cert.files, kind)?;

        let check_with = match &rules {
            Some(r) => format!(
                "{} check {} {} {}",
                kind.binary(),
                r.display(),
                asserted.display(),
                derivations.display()
            ),
            None => format!(
                "{} {} {}",
                kind.binary(),
                asserted.display(),
                derivations.display()
            ),
        };
        // A scratch directory is removed once the checker has read it, so the
        // command above names files that will not be there. Saying which is
        // cheap; printing a command a reader cannot run and letting them find
        // out is not.
        let kept = opts.certificate_out_dir.is_some();
        let checker_block = serde_json::json!({
            "status": status.word(),
            "check_with": check_with,
            "certificate_dir": dir.display().to_string(),
            "certificate_dir_kept": kept,
            "check_with_means": if kept {
                "the certificate files are still in certificate_dir, so this command repeats the \
                 check exactly"
            } else {
                "the certificate went to a scratch directory that is removed once the checker has \
                 read it, so this command names files that no longer exist. Unpack again with \
                 certificate_out_dir set to keep them and run it"
            },
            "checker_says_it_is": checker_self_block(&status),
            "checker_limit": CHECKER_LIMIT,
            "coverage": {
                "asserted_in_certificate": cover.asserted_in_certificate,
                "in_certificate_only": cover.in_certificate_only_total,
                "sample_in_certificate_only": cover.in_certificate_only,
                "conclusions": cover.conclusions,
                "conclusions_not_in_pack": cover.conclusions_not_in_pack_total,
                "sample_conclusions_not_in_pack": cover.conclusions_not_in_pack,
                "unparsed_derivation_lines": cover.unparsed_derivation_lines,
                "means": "in_certificate_only counts assertions the proof rests on that this pack \
                          does not contain, and a checker cannot see them. conclusions_not_in_pack \
                          counts triples the certificate derives that this pack does not carry, \
                          which is the normal shape when the sender reasoned without materialising",
            },
            "refutation_in_pack": cert.files.contains_key("refutation.tsv"),
            "refutation_means": "a refutation.tsv is carried when the run found a contradiction. \
                                 NOTHING in this crate runs oo-refute, so it was not checked here \
                                 and no verdict in this report is about it",
        });

        // The order of these arms is the policy. A checker that said no, or
        // could not read the files, outranks everything: those are defects. A
        // certificate that checks and is about another graph is next. Only then
        // can an acceptance be reported, and only while carrying the token the
        // run earned.
        let (verdict, extra) = match status {
            CheckerStatus::Rejected { ref stdout } => (
                PackCertVerdict::Refused,
                serde_json::json!({
                    "checker_said": stdout,
                    "means": "the checker named a step it would not accept. STOP. This pack's own \
                              proof does not check, which is a defect in the pack",
                }),
            ),
            CheckerStatus::Unreadable { ref stdout } => (
                PackCertVerdict::Unreadable,
                serde_json::json!({"checker_said": stdout, "means": ABSENT_MEANS}),
            ),
            CheckerStatus::Absent { ref what, install } => (
                PackCertVerdict::CheckerAbsent,
                serde_json::json!({"why": what, "install": install, "means": ABSENT_MEANS}),
            ),
            CheckerStatus::NotNeeded { ref what } => (
                PackCertVerdict::CheckerAbsent,
                serde_json::json!({"why": what, "install": LAKE_INSTALL, "means": ABSENT_MEANS}),
            ),
            CheckerStatus::Accepted(ref a) if cover.in_certificate_only_total > 0 => (
                PackCertVerdict::NotAboutThisPack,
                serde_json::json!({
                    "checker_said": a.stdout(),
                    "means": "the checker accepted this certificate and it is not about this pack. \
                              Its derivations follow from the assertions listed INSIDE it, and \
                              those are not all in the graph you were sent, so the proof is over \
                              premises nobody shipped",
                }),
            ),
            CheckerStatus::Accepted(ref a) => {
                let cert_token = a.certified();
                let weaker = matches!(kind, CertKind::OoHorn);
                let v = match kind {
                    CertKind::OoCert => PackCertVerdict::Accepted(cert_token),
                    CertKind::OoHorn => PackCertVerdict::AcceptedUnderSuppliedRules(cert_token),
                };
                (
                    v,
                    serde_json::json!({
                        "checker_said": a.stdout(),
                        "means": MEANS,
                        "does_not_mean": DOES_NOT_MEAN,
                        "and_only_under_these_rules": weaker.then_some(
                            "this certificate is over a rule table the SENDER supplied, carried in \
                             rules.tsv. Its conclusions hold in every model of the asserted \
                             triples that also satisfies that table, and a table saying every \
                             supplier is compliant produces steps that check green for ever. Read \
                             rules.tsv before reading the verdict"),
                    }),
                )
            }
        };

        let mut r = base(&verdict, checker_block);
        if let (Some(a), Some(b)) = (r.as_object_mut(), extra.as_object()) {
            for (k, val) in b {
                a.insert(k.clone(), val.clone());
            }
        }
        // Rewritten from the verdict AFTER the merge, so no `extra` arm can put
        // a theorem name beside a verdict that did not earn one.
        r["theorem"] = match verdict.theorem() {
            Some(t) => serde_json::json!(t),
            None => serde_json::Value::Null,
        };
        r["checked_here"] = serde_json::json!(verdict.is_checked());
        if opts.certificate_out_dir.is_none() {
            // Best effort, and it is fine if it fails: a leftover scratch
            // directory is untidy and a missing one would break the
            // check_with command this report just printed.
            let _ = std::fs::remove_dir_all(&dir);
        }
        Ok((verdict, r))
    }
}

/// The `checker` block the binary printed ABOUT ITSELF, read out of its stdout.
///
/// Never constructed here. A Rust literal naming which binary ran would be
/// exactly the string a reader cannot check. `None` when the process printed no
/// such block, which is the honest answer for an old checker and for anything
/// that is not a checker at all.
fn checker_self_block(status: &CheckerStatus) -> Option<serde_json::Value> {
    let text = match status {
        CheckerStatus::Accepted(a) => a.stdout(),
        CheckerStatus::Rejected { stdout } | CheckerStatus::Unreadable { stdout } => stdout,
        CheckerStatus::Absent { .. } | CheckerStatus::NotNeeded { .. } => return None,
    };
    let line = text.lines().find(|l| l.starts_with('{'))?;
    let v: serde_json::Value = serde_json::from_str(line).ok()?;
    v.get("checker").cloned()
}

/// A directory nothing else is writing into.
///
/// The pid alone is not enough: `cargo test` runs these as threads of ONE
/// process, so two unpacks in one binary would collide on it and each would see
/// the other's files.
fn scratch_dir() -> PathBuf {
    static NEXT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
    let nanos = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .map(|d| d.as_nanos())
        .unwrap_or(0);
    let serial = NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
    std::env::temp_dir().join(format!(
        "oo-unpack-{}-{nanos}-{serial}",
        std::process::id()
    ))
}