Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C
32.0 KB
Raw Blame
#!/usr/bin/env python3
"""Speculative-decoding gap check (U7a acceptance test 4).

Asserts, over a captured raw engine-stdout transcript, that EVERY generated
DATA frame of an opted-in request carries the per-token numeric channel
("DATA <id> <n> <lp> <k> [tid tlp]*k") -- accepted draft tokens included.
On today's engine, an accepted speculative-draft token is emitted through
the SAME mux_data() call as any other generated token (mux_spec_emit ->
mux_data, passing the same logits row and requested top-k the ordinary
emit sites use), so the gap this check hunts -- a legacy 3-field DATA
frame appearing mid-generation because a draft-accept path bypassed the
numeric-channel emit -- is NOT producible by the engine as shipped. This
check is retained as REGRESSION COVERAGE for that invariant (a future mux
change that adds a new emit call site without the tail would reintroduce
exactly this defect class), not as a currently-live defect hunt.

Run recipe (orchestrator; needs a real model -- CPU or explicit-CUDA build):

    # single KV slot + model drafts live = the speculative serve regime
    cd c && make glm
    SERVE=1 SERVE_BATCH=1 KV_SLOTS=1 DRAFT=2 CTX=4096 \
        ./colibri <snapshot-dir> < submit.raw | tee engine_stdout.raw

  where submit.raw contains one opted-in generation request, e.g. (prompt
  "Hello" = 5 payload bytes, 64 new tokens, greedy, logprobs top-5):

    SUBMIT 7 0 5 64 0 1 0 logprobs=5
    Hello

  (payload line follows the header; the trailing newline after the payload is
  part of the framing). A speculative run needs an MTP-bearing container or
  the n-gram fallback to actually accept drafts -- confirm acceptance in the
  stderr log ("[MTP] ..." / spec acceptance lines) so the run genuinely
  exercised the draft-accept emit path rather than trivially passing.

A raw capture taken this way is a plain pipeline: it is not itself hash-bound
to the binary, container, input, and environment that produced it, and
NOTHING in this module binds it either -- the preamble gate below checks
only that the BANNER/LOADED text is a well-formed, self-consistent record of
what the engine printed about itself; it does not verify that text against
an independently computed binary or container digest. Whichever capture step
feeds this checker, prefer one that additionally binds and retains that
provenance (distinct raw stdout/stderr, the direct engine exit status, and a
hash over the binary/container/input/environment/protocol payloads) over a
bare pipeline or a lone DATA/HIT record, which is not decisive evidence on
its own.

Then, against the resulting transcript, the numeric-channel check runs:

    python3 tests/check_data_logprob_gaps.py engine_stdout.raw \
        --id 7 --topk 5 --vocab 154880

Checks performed:
  - the transcript's capture shape (full-process / ready-suffix / request-
    only) is named by capture_mode() and enforced explicitly: the leading
    line must be a recognized global preamble record or a legitimate
    request-frame kind, and a transcript with any global record present
    but not correctly led (a "invalid" capture per capture_mode()) fails
    loud -- no transcript reaches the rest of the checks unclassified;
  - the startup BANNER/LOADED preamble, where present, parses exactly
    (the engine_evidence grammar); an unparsed preamble is a named
    failure quoting the offending line, never a silent pass;
  - exactly one ACCEPT and one DONE exist for --id, DONE's emitted count
    equals the positive DATA count, and no targeted ERROR or post-DONE
    frame exists;
  - every DATA/ECHO numeric tail has exactly the advertised fields and
    min(--topk,--vocab) unique token ids in range. Each numeric token
    (target logprob and every top-k logprob) must be spelled as the
    fixed six-decimal form dev's engine prints today ("fixed6", e.g.
    "-0.300000"), the %.17g form an earlier engine build printed
    ("c17g", e.g. "-2.7000000000000002"), or an exact nan/inf/-inf
    spelling -- whichever form parses -- and the VALUE that spelling
    carries must then be finite and non-positive; a nan/inf/-inf
    spelling always fails that value check, by construction, since none
    of those values are finite. That spelling is recognized at all only
    so a non-finite token gets one specific, named rejection ("target
    logprob is not finite/non-positive", quoting the exact token and
    byte offset) instead of a generic "matches neither form" failure --
    it is not an accepted value, just a diagnosed one. On today's engine
    (colibri.c) this is also not a normal, tolerable outcome to shrug
    off: grammar-constrained decoding (GRAMMAR=/SCHEMA=, method F)
    proposes speculative draft tokens verified against the model's own
    unmasked distribution and never zeroes or -inf's any vocabulary
    logit, so a non-finite emitted logprob can only reach the wire
    through the total-vocabulary numerical-collapse fallback documented
    at sample.h's dist_build()/argmax_v() (every logit in the row NaN or
    -inf) -- which those functions' own comments and stderr warning
    label "a numerical blow-up upstream", not a degenerate-but-valid
    row. Failing loud on it here names a real engine bug instead of
    quietly passing it through.
    A token that happens to satisfy BOTH finite spellings exactly
    (e.g. '%.17g' % -1.234567 == '-1.234567', itself also an exact
    six-decimal spelling) is
    "ambiguous": there is no way to tell, from the token alone, which
    engine format produced it, so no per-frame or per-transcript
    "consistent format" rule is enforced -- one was tried and had to be
    dropped (see the module history) because it produced false rejections
    on real %.17g transcripts purely from this ambiguity. The observed
    form counts (fixed6-only / c17g-only / ambiguous / special) are
    reported in the summary line as INFORMATION ONLY; they never affect
    the verdict;
  - ACCEPT's canonical prompt length P has exactly P ECHO frames at
    positions 0..P-1, position 0 carrying "nan 0";
  - payload framing (n bytes + newline) stays byte-exact throughout, so a
    single malformed frame cannot hide by desynchronizing the parse; every
    reported problem cites the byte offset of the offending frame's own
    header line (not the frame that follows it).
Exit 0 = no gaps; non-zero = at least one gap/malformed frame (listed).
"""
import argparse
import collections
import math
import os
import re
import sys


_TOOLS_DIR = os.path.join(os.path.dirname(os.path.dirname(
    os.path.abspath(__file__))), "tools")
if _TOOLS_DIR not in sys.path:
    sys.path.insert(0, _TOOLS_DIR)
from engine_evidence import PreambleError, parse_engine_preamble


_INT32_MAX = 2**31 - 1
_UINT64_MAX = 2**64 - 1
_UINT_RE = re.compile(rb"(?:0|[1-9][0-9]*)")
_C17G_RE = re.compile(
    rb"-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?"
    rb"(?:e[+-](?:0[0-9]|[1-9][0-9]{1,2}))?")
_FIXED6_RE = re.compile(rb"-?(?:0|[1-9][0-9]*)\.[0-9]{6}")
_SPECIAL_TOKENS = {b"nan": math.nan, b"inf": math.inf, b"-inf": -math.inf}
_READY = b"\x01\x01READY\x01\x01"
_GLOBAL_KINDS = frozenset((
    b"BANNER", b"LOADED", b"READY", b"STAT", b"HWINFO", b"TIERS",
    b"EMAP", b"HITS", b"PROF",
))
_TARGETED_KINDS = frozenset((
    b"ACCEPT", b"DATA", b"ECHO", b"DONE", b"ERROR",
))
_LOWER_HEX_RE = re.compile(rb"[0-9a-f]*")


def _uint(token, label, maximum=_INT32_MAX):
    if not _UINT_RE.fullmatch(token):
        raise ValueError(f"noncanonical {label}: {token!r}")
    value = int(token.decode("ascii"))
    if value > maximum:
        raise ValueError(f"{label} exceeds {maximum}: {token!r}")
    return value


def _c17g(token, label):
    if not _C17G_RE.fullmatch(token):
        raise ValueError(f"noncanonical {label} %.17g token: {token!r}")
    value = float(token.decode("ascii"))
    if not math.isfinite(value) or format(value, ".17g").encode("ascii") != token:
        raise ValueError(f"not an exact finite {label} %.17g spelling: {token!r}")
    return value


def _fixed6(token, label):
    if not _FIXED6_RE.fullmatch(token):
        raise ValueError(f"noncanonical {label} %.6f token: {token!r}")
    value = float(token.decode("ascii"))
    if not math.isfinite(value) or format(value, ".6f").encode("ascii") != token:
        raise ValueError(f"not an exact finite {label} %.6f spelling: {token!r}")
    return value


def _numeric_value(token, label):
    """Classify and parse one engine numeric-tail token.

    Returns (value, form). "special" is an exact nan/inf/-inf spelling --
    libc's own %f/%g rendering of a non-finite double, which either wire
    format's snprintf call can emit identically. Otherwise the token is
    checked against BOTH finite grammars independently (never short-
    circuited): the fixed six-decimal form dev's engine prints today
    ("fixed6") and the %.17g form an earlier engine build printed
    ("c17g"). A token can satisfy both -- '%.17g' % -1.234567 ==
    '-1.234567', which is ALSO the exact six-decimal spelling of that
    same double -- and when it does, the form is "ambiguous": there is no
    way to tell, from the token alone, which engine format produced it.
    A token matching neither raises ValueError.

    An earlier version of this function tried fixed6 first and reported
    it whenever fixed6 matched, silently hiding the ambiguous case; that
    misclassified a large fraction of genuine %.17g tokens as fixed6 and
    fed a since-removed "consistent form per frame" check false mixed-
    form rejections on real transcripts. Classification is now purely
    informational (see check()/main()) precisely because it cannot be
    made unambiguous from the token alone.
    """
    if token in _SPECIAL_TOKENS:
        return _SPECIAL_TOKENS[token], "special"
    value = None
    is_fixed6 = is_c17g = False
    try:
        value = _fixed6(token, label)
        is_fixed6 = True
    except ValueError:
        pass
    try:
        value = _c17g(token, label)
        is_c17g = True
    except ValueError:
        pass
    if is_fixed6 and is_c17g:
        return value, "ambiguous"
    if is_fixed6:
        return value, "fixed6"
    if is_c17g:
        return value, "c17g"
    raise ValueError(
        f"{label} matches neither the fixed6 nor the c17g nor the "
        f"nan/inf spelling: {token!r}")


def _fixed_metric(token, places, label, lower=0.0, upper=None):
    pattern=rb"-?(?:0|[1-9][0-9]*)\."+rb"[0-9]{"+str(places).encode()+rb"}"
    if not re.fullmatch(pattern,token):
        raise ValueError(f"noncanonical {label}: {token!r}")
    value=float(token.decode("ascii"))
    if not math.isfinite(value) or value<lower or (upper is not None and value>upper):
        raise ValueError(f"{label} outside [{lower},{upper}]: {token!r}")
    return value


def _header_fields(line, byte_offset):
    problems=[]
    if not line:
        return [],[f"blank protocol header at byte {byte_offset}"]
    if (any(byte<0x20 or byte>0x7e for byte in line) or
            line.startswith(b" ") or line.endswith(b" ") or b"  " in line):
        problems.append(f"noncanonical ASCII/space header at byte {byte_offset}: {line!r}")
    return line.split(),problems


def _global_header(line, byte_offset):
    """Validate one exact production mux-global line.

    Return None when the line is not a recognized global.  Recognized globals
    return a synthetic one-field marker so check() owns their process-level
    lifecycle before any numeric token can collide with a request id.
    """
    if (line.startswith(b"== GLM C engine") or
            line.startswith(b"loaded in")):
        kind = b"BANNER" if line.startswith(b"==") else b"LOADED"
        problems = []
        try:
            text = line.decode("ascii")
            parsed = parse_engine_preamble(text)
            if parsed is None or parsed["kind"].encode("ascii") != kind:
                raise PreambleError("owned preamble kind mismatch")
        except (UnicodeDecodeError, PreambleError) as exc:
            problems.append(
                f"malformed {kind.decode()} preamble at byte {byte_offset}: "
                f"{exc}; {line!r}")
        return [kind], problems

    if line == _READY:
        return [b"READY"], []

    kind = line.split(b" ", 1)[0]
    if kind == b"READY":
        return [b"READY"], [
            f"malformed READY global at byte {byte_offset}: expected {_READY!r}; {line!r}"
        ]
    if kind not in _GLOBAL_KINDS - {b"READY"}:
        return None
    problems = []
    try:
        if kind == b"STAT":
            fields = line.split(b" ")
            if (len(fields) != 5 or fields[1:4] != [b"0", b"0.00", b"0.0"]):
                raise ValueError("expected exact startup STAT fields")
            _fixed_metric(fields[4], 2, "STAT RSS")
        elif kind == b"HWINFO":
            # The final CPU|GPU field is produced by two %s conversions.  It is
            # printable free text and may contain repeated spaces; the six
            # numeric/structural prefixes remain exact single-space fields.
            fields = line.split(b" ", 6)
            if len(fields) != 7:
                raise ValueError("expected six HWINFO prefixes and CPU|GPU tail")
            _uint(fields[1], "HWINFO core count")
            _fixed_metric(fields[2], 1, "HWINFO total RAM")
            _fixed_metric(fields[3], 1, "HWINFO available RAM")
            _uint(fields[4], "HWINFO GPU count")
            _fixed_metric(fields[5], 1, "HWINFO total VRAM")
            tail = fields[6]
            if (tail.count(b"|") != 1 or
                    any(byte < 0x20 or byte > 0x7e for byte in tail)):
                raise ValueError("HWINFO tail is not printable CPU|GPU text")
        elif kind == b"TIERS":
            fields = line.split(b" ")
            if len(fields) != 6:
                raise ValueError("expected five TIERS fields")
            for index, label in enumerate(("VRAM", "RAM", "disk"), 1):
                _uint(fields[index], f"TIERS {label} count")
            _fixed_metric(fields[4], 2, "TIERS VRAM GB")
            _fixed_metric(fields[5], 2, "TIERS RAM GB")
        elif kind in (b"EMAP", b"HITS"):
            fields = line.split(b" ")
            if len(fields) != 4:
                raise ValueError(f"expected three {kind.decode()} fields")
            rows = _uint(fields[1], f"{kind.decode()} row count")
            cols = _uint(fields[2], f"{kind.decode()} column count")
            if cols < 1 or (kind == b"HITS" and rows < 1):
                raise ValueError(
                    f"{kind.decode()} rows/columns outside producer domain")
            payload = fields[3]
            if not _LOWER_HEX_RE.fullmatch(payload):
                raise ValueError(f"{kind.decode()} payload is not lowercase hex")
            cells = rows * cols
            expected = cells * 2 if kind == b"EMAP" else ((cells + 7) // 8) * 2
            if len(payload) != expected:
                raise ValueError(
                    f"{kind.decode()} payload length {len(payload)} != {expected}")
            if kind == b"EMAP":
                for index in range(cells):
                    cell = int(payload[2 * index:2 * index + 2], 16)
                    tier, heat = cell >> 6, cell & 0x3f
                    if tier > 2:
                        raise ValueError(
                            f"EMAP cell {index} tier {tier} outside [0,2]")
                    if heat > 32:
                        raise ValueError(
                            f"EMAP cell {index} heat {heat} outside [0,32]")
            elif cells & 7:
                final = int(payload[-2:], 16)
                used_mask = (1 << (cells & 7)) - 1
                if final & ~used_mask:
                    raise ValueError("HITS final byte has nonzero padding bits")
        else:  # PROF
            fields = line.split(b" ")
            if len(fields) != 10:
                raise ValueError("expected nine PROF fields")
            _fixed_metric(fields[1], 3, "PROF wall seconds")
            _uint(fields[2], "PROF prompt count")
            _uint(fields[3], "PROF completion count")
            for index, label in enumerate(
                    ("disk", "wait", "matmul", "attention", "head"), 4):
                _fixed_metric(fields[index], 3, f"PROF {label} seconds")
            _uint(fields[9], "PROF forward count", _UINT64_MAX)
    except (ValueError, IndexError) as exc:
        problems.append(
            f"malformed {kind.decode()} global at byte {byte_offset}: {exc}; {line!r}")
    return [kind], problems


def parse_frames(blob):
    """Parse frames fail-closed, returning (frames, framing_problems).

    Each frame is (fields, payload, byte_offset), where byte_offset is the
    offset of THIS frame's own header line -- captured before the cursor
    advances past it, so every problem this module reports can cite the
    byte offset of the frame it is actually complaining about, not the
    frame that happens to follow it in the transcript.
    """
    frames = []
    problems = []
    i = 0
    n = len(blob)
    while i < n:
        line_start = i
        j = blob.find(b"\n", i)
        if j < 0:
            problems.append(f"truncated header at byte {line_start}")
            break
        line = blob[i:j]
        i = j + 1
        global_header = _global_header(line, line_start)
        if global_header is None:
            fields, header_problems = _header_fields(line, line_start)
        else:
            fields, header_problems = global_header
        problems.extend(header_problems)
        if not fields:
            continue
        kind = fields[0]
        if kind in (b"DATA", b"ECHO"):
            if len(fields) < 3:
                problems.append(
                    f"short {kind.decode(errors='replace')} header at "
                    f"byte {line_start}: {fields!r}")
                break
            try:
                size = _uint(fields[2],"payload size")
            except ValueError:
                problems.append(
                    f"invalid payload size at byte {line_start}: {fields!r}")
                break
            if size < 0:
                problems.append(
                    f"negative payload size at byte {line_start}: {fields!r}")
                break
            if size > n - i:
                problems.append(
                    f"truncated payload at byte {line_start}: need {size} bytes")
                break
            payload = blob[i:i + size]
            i += size
            if i < n and blob[i:i + 1] == b"\n":
                i += 1
            else:
                problems.append(
                    f"missing payload terminator at byte {line_start} "
                    f"after {fields!r}")
                break
            frames.append((fields, payload, line_start))
        else:
            frames.append((fields, None, line_start))
    return frames, problems


def capture_mode(frames):
    """Name the explicit transcript shape without grading its lifecycle."""
    if not frames:
        return "request-only"
    first = frames[0][0][0]
    if first == b"BANNER":
        return "full-process"
    if first == b"READY":
        return "ready-suffix"
    if any(fields[0] in _GLOBAL_KINDS for fields, _, _ in frames):
        return "invalid"
    return "request-only"


def _numeric_tail(fields, field_offset, expected_k, vocab, label, byte_offset):
    """Validate one DATA/ECHO numeric tail starting at fields[field_offset].

    Returns (problems, forms) where forms is the LIST of numeric-token
    "form" tags (see _numeric_value) observed in this tail, in order --
    a list, not a set, so the caller can tally counts. No rule requires a
    tail's forms to agree with one another: per-token classification is
    inherently ambiguous (a token can be an exact spelling under both
    finite grammars at once), so there is no reliable way to tell a
    genuinely mixed-format tail from an entirely single-format one that
    merely contains some ambiguous tokens -- see _numeric_value.
    """
    problems = []
    forms = []
    try:
        lp, lp_form = _numeric_value(fields[field_offset], f"{label} target logprob")
        k = _uint(fields[field_offset + 1], f"{label} top-k count", 32)
    except (ValueError, IndexError):
        return ([f"malformed {label} numeric fields at byte {byte_offset}: "
                f"{fields!r}"], forms)
    forms.append(lp_form)
    if not math.isfinite(lp) or lp > 0.0:
        problems.append(
            f"{label} target logprob is not finite/non-positive at byte "
            f"{byte_offset}: {fields!r}")
    if k != expected_k:
        problems.append(
            f"{label} top-k {k} != expected {expected_k} at byte "
            f"{byte_offset}: {fields!r}")
    want_fields = field_offset + 2 + 2 * k
    if len(fields) != want_fields:
        return (problems + [f"{label} field count {len(fields)} != "
                            f"{want_fields} at byte {byte_offset}: {fields!r}"],
                forms)
    ids = []
    for idx in range(k):
        try:
            token_id = _uint(fields[field_offset + 2 + 2 * idx],
                             f"{label} token id", _INT32_MAX)
            token_lp, token_form = _numeric_value(
                fields[field_offset + 3 + 2 * idx], f"{label} token logprob")
        except ValueError:
            problems.append(
                f"malformed {label} top-k pair {idx} at byte {byte_offset}: "
                f"{fields!r}")
            continue
        forms.append(token_form)
        if not 0 <= token_id < vocab:
            problems.append(
                f"{label} token id {token_id} outside [0,{vocab}) at byte "
                f"{byte_offset}: {fields!r}")
        if token_id in ids:
            problems.append(
                f"{label} duplicate token id {token_id} at byte "
                f"{byte_offset}: {fields!r}")
        ids.append(token_id)
        if not math.isfinite(token_lp) or token_lp > 0.0:
            problems.append(
                f"{label} token {token_id} logprob is not finite/non-positive "
                f"at byte {byte_offset}: {fields!r}")
    return problems, forms


def check(frames, framing_problems, request_id, topk, vocab):
    problems = list(framing_problems)
    mode = capture_mode(frames)
    if frames:
        first_kind = frames[0][0][0]
        if first_kind not in _GLOBAL_KINDS and first_kind not in _TARGETED_KINDS:
            problems.append(
                f"unrecognized frame opens the transcript at byte "
                f"{frames[0][2]} (not a global preamble or a request "
                f"frame): {frames[0][0]!r}")
    if mode == "invalid":
        problems.append(
            "invalid capture mode: global records present without a "
            "recognized BANNER/READY lead frame")
    try:
        rid = str(request_id).encode("ascii",errors="strict")
        rid_value=_uint(rid,"requested id",_UINT64_MAX)
        if rid_value < 1:
            raise ValueError("requested id must be positive")
    except (UnicodeEncodeError,ValueError) as exc:
        return 0, 0, problems+[str(exc)], mode, frozenset()
    data_frames = 0
    echo_positions = []
    accepts = []
    dones = []
    accept_prompts = []
    done_values = []
    done_seen = False
    data_seen = False
    global_indices = {kind: [] for kind in _GLOBAL_KINDS}
    expected_k = min(topk, vocab)
    forms_used = []
    for frame_index, (fields, payload, offset) in enumerate(frames):
        kind = fields[0]
        if kind in _GLOBAL_KINDS:
            global_indices[kind].append(frame_index)
            continue
        if len(fields)<2:
            continue
        try:
            frame_id=_uint(fields[1],"frame request id",_UINT64_MAX)
        except ValueError as exc:
            if kind in _TARGETED_KINDS:
                problems.append(f"{exc} at byte {offset}")
            continue
        if frame_id!=rid_value:
            continue
        if done_seen:
            problems.append(
                f"frame for id {request_id} after DONE at byte {offset}: "
                f"{fields!r}")
        if kind == b"ACCEPT":
            accepts.append(frame_index)
            if len(fields) != 3:
                problems.append(
                    f"malformed ACCEPT frame at byte {offset}: {fields!r}")
            else:
                try:
                    prompt=_uint(fields[2],"ACCEPT prompt length")
                    if prompt<1:
                        raise ValueError
                except ValueError:
                    problems.append(
                        f"invalid ACCEPT prompt length at byte {offset}: "
                        f"{fields!r}")
                else:
                    accept_prompts.append(prompt)
            continue
        if kind == b"ERROR":
            problems.append(
                f"target request returned ERROR at byte {offset}: {fields!r}")
            continue
        if kind == b"DONE":
            dones.append(frame_index)
            done_seen = True
            if len(fields) != 9 or fields[2] != b"STAT":
                problems.append(
                    f"malformed DONE frame at byte {offset}: {fields!r}")
                done_values.append(None)
                continue
            try:
                emitted=_uint(fields[3],"DONE emitted count")
                tps=_fixed_metric(fields[4],2,"DONE tokens/second")
                hit=_fixed_metric(fields[5],1,"DONE hit percentage",upper=100.0)
                rss=_fixed_metric(fields[6],2,"DONE RSS")
                prompt=_uint(fields[7],"DONE prompt-token count")
                flag=_uint(fields[8],"DONE length_limited",1)
                if prompt<1:
                    raise ValueError("DONE prompt-token count is not positive")
            except ValueError:
                problems.append(
                    f"malformed DONE stats at byte {offset}: {fields!r}")
                done_values.append(None)
                continue
            done_values.append((emitted,tps,hit,rss,prompt,flag))
            continue
        if kind == b"DATA":
            data_seen = True
            data_frames += 1
            if not accepts:
                problems.append(
                    f"DATA before ACCEPT at byte {offset}: {fields!r}")
            if payload is None:
                problems.append(
                    f"DATA missing validated payload at byte {offset}: "
                    f"{fields!r}")
            if len(fields) == 3:
                problems.append(
                    f"GAP: legacy 3-field DATA frame #{data_frames} at byte "
                    f"{offset} (payload {payload!r}) has no logprob")
                continue
            tail_problems, tail_forms = _numeric_tail(
                fields, 3, expected_k, vocab, "DATA", offset)
            problems.extend(tail_problems)
            forms_used.extend(tail_forms)
        elif kind == b"ECHO":
            if data_seen:
                problems.append(
                    f"ECHO after DATA at byte {offset}: {fields!r}")
            if not accepts:
                problems.append(
                    f"ECHO before ACCEPT at byte {offset}: {fields!r}")
            if payload is None:
                problems.append(
                    f"ECHO missing validated payload at byte {offset}: "
                    f"{fields!r}")
            try:
                pos = _uint(fields[3],"ECHO position")
            except (ValueError, IndexError):
                problems.append(
                    f"malformed ECHO frame at byte {offset}: {fields!r}")
                continue
            echo_positions.append(pos)
            if pos == 0:
                if len(fields) != 6 or fields[4] != b"nan" or fields[5] != b"0":
                    problems.append(
                        f"ECHO position 0 should carry exactly 'nan 0' at "
                        f"byte {offset}: {fields!r}")
            else:
                tail_problems, tail_forms = _numeric_tail(
                    fields, 4, expected_k, vocab, "ECHO", offset)
                problems.extend(tail_problems)
                forms_used.extend(tail_forms)
        else:
            problems.append(
                f"unknown targeted frame kind at byte {offset}: {fields!r}")
    any_globals = any(global_indices[kind] for kind in _GLOBAL_KINDS)
    if global_indices[b"BANNER"] or global_indices[b"LOADED"]:
        required = (
            (b"BANNER", 0), (b"LOADED", 1), (b"READY", 2), (b"STAT", 3),
            (b"HWINFO", 4), (b"TIERS", 5), (b"EMAP", 6),
        )
        for owned_kind, expected_index in required:
            found = global_indices[owned_kind]
            if (owned_kind in (b"BANNER", b"LOADED", b"READY", b"STAT") and
                    found != [expected_index]):
                problems.append(
                    f"expected one {owned_kind.decode()} as frame "
                    f"{expected_index}, found {found}")
            elif (owned_kind not in (b"BANNER", b"LOADED", b"READY", b"STAT") and
                  (not found or found[0] != expected_index)):
                problems.append(
                    f"expected startup {owned_kind.decode()} as frame "
                    f"{expected_index}, found {found}")
        if len(frames) > 7 and frames[7][0][0] not in (b"ACCEPT", b"ERROR"):
            problems.append(
                f"unexpected full-process pre-request frame 7: {frames[7][0]!r}")
    elif any_globals:
        ready = global_indices[b"READY"]
        stat = global_indices[b"STAT"]
        if ready != [0]:
            problems.append(f"expected one READY as frame 0, found {ready}")
        if stat != [1]:
            problems.append(f"expected one startup STAT as frame 1, found {stat}")
    if len(accepts) != 1:
        problems.append(f"expected exactly one ACCEPT, found {len(accepts)}")
    elif len(accept_prompts)==1:
        prompt=accept_prompts[0]
        if echo_positions!=list(range(prompt)):
            problems.append(f"ECHO positions {echo_positions} != required 0..{prompt-1}")
    if len(dones) != 1:
        problems.append(f"expected exactly one DONE, found {len(dones)}")
    elif len(done_values)==1 and done_values[0] is not None:
        emitted,_,_,_,done_prompt,_=done_values[0]
        if emitted!=data_frames:
            problems.append(f"DONE emitted {emitted} != observed DATA {data_frames}")
        if len(accept_prompts)==1 and done_prompt!=accept_prompts[0]:
            problems.append(f"DONE prompt count {done_prompt} != ACCEPT {accept_prompts[0]}")
    if data_frames<=0:
        problems.append("no DATA frames found for this id -- wrong id, or the run produced nothing")
    return data_frames, len(echo_positions), problems, mode, collections.Counter(forms_used)


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("transcript", help="raw engine stdout capture")
    parser.add_argument("--id", required=True, help="request id to audit")
    parser.add_argument("--topk", type=int, required=True,
                        help="the SUBMIT logprobs=k value the request used")
    parser.add_argument("--vocab", type=int, required=True,
                        help="checkpoint vocabulary size used to bound token ids")
    args = parser.parse_args()
    try:
        if _uint(args.id.encode("ascii"), "requested id", _UINT64_MAX) < 1:
            raise ValueError("requested id must be positive")
    except (UnicodeEncodeError, ValueError) as exc:
        parser.error(str(exc))
    if not 1 <= args.topk <= 32:
        parser.error("--topk must be in 1..32 for an opted-in B1 evidence run")
    if args.vocab < 1:
        parser.error("--vocab must be positive")
    blob = open(args.transcript, "rb").read()
    frames, framing_problems = parse_frames(blob)
    data_frames, echo_frames, problems, mode, forms_used = check(
        frames, framing_problems, args.id, args.topk, args.vocab)
    # Informational only -- see _numeric_value: per-token form
    # classification is inherently ambiguous, so this tally never affects
    # the verdict, only what a reviewer sees.
    forms_summary = (", ".join(f"{form}={forms_used[form]}"
                               for form in sorted(forms_used))
                     if forms_used else "none")
    print(f"[gapcheck] id={args.id}: capture_mode={mode}, {data_frames} DATA "
          f"frames, {echo_frames} ECHO frames, numeric form tally: "
          f"{forms_summary}")
    for problem in problems:
        print(f"[gapcheck] {problem}")
    verdict = ("FAIL" if problems else
               "PASS: complete request; every generated token carries a valid logprob table")
    print(f"[gapcheck] {verdict}")
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main())