Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

38.3 KB
Raw Blame
"""Repo-wide ratchet: a `--help` assertion must never read RAW CLI output.

This failure has turned CI red **four times** — v0.71.26, v0.71.32, v0.71.35 and
v0.73.2, the last one on all nine test cells at once. Each time it was written
down (CLAUDE.md, CONTRIBUTING, a memory note) and each time it happened again,
which is the argument for a check that FAILS rather than another paragraph that
asks people to remember.

**The mechanism, precisely.** Typer renders `--help` through a Rich table and
styles each option name, so on a colour-capable stream Rich emits escapes
*inside* the flag::

    --noise-floor  ->  -\\x1b[0m\\x1b[1;36m-noise\\x1b[0m\\x1b[1;36m-floor

`"--noise-floor" in result.output` then cannot match. **Windows passes**, because
Rich auto-disables colour there — so a green local run on a Windows box proves
nothing about the Linux/macOS cells. That asymmetry is the entire trap.

**Scope is deliberately narrow: `--help` output only.** A flag named inside an
ERROR message is fine and is NOT flagged, because `console.print(f"... --model
...")` interpolates plain text that Rich does not style per-token. Measured
before writing this guard: 13 such error-message assertions exist across 10
files and all of them are correct. Flagging those would make this test noise,
and a noisy guard gets deleted.

**A third class exists and is deliberately NOT scanned (issue #987).** A bare
`assert "\\x1b" not in result.output` on ordinary (non-`--help`,
non-highlighted) CLI output is environment-dependent: the command under test
prints through an unpinned module-level `Console()`, so under
`FORCE_COLOR=1 TERM=xterm-256color` Rich styles the *surrounding* panel markup
and the assertion fails even when the untrusted field itself was sanitised
correctly. The fix is per-test Console pinning::

    monkeypatch.setattr(mod, "console", Console(force_terminal=False))

`no_color=True` alone is NOT sufficient — it suppresses colour but does not pin
tty detection, which is why `test_replay.py` carried that exact bug.

**Why this is not automated.** These assertions split roughly half and half
between ones reading CLI output and ones reading a pure-function return
(`_safe_md_cell(...)`, `_yaml_dq(...)`, `str(exc.value)`, a `cleaned` local, a
`messages[0]`). No exact count is quoted here on purpose: four AST definitions
of "such an assertion" gave four different totals, so any number would be an
artefact of its matcher. The ratio survives every definition, and the ratio is
what the argument needs. A scanner rule for this class must tell those two
kinds apart, and at a roughly even split a rule that cannot do so cleanly
false-positives about half of what it sees — the same "a noisy guard gets
deleted" argument made one paragraph up. The split is not academic:
`test_issue627_unknown_config_keys.py` asserts on a local named `out` that *is*
console output (a `Console(file=...)` StringIO), so name-based and call-based
heuristics both misclassify it. That distinction is left to review and to
per-test pinning, not automated here.

Seven were pinned under #987 (`test_replay`, `test_runs`,
`test_review_fixes_v07133`, `test_v07127`, `test_v07136`, `test_v07139`,
`test_issue367_ship_numerics`). `test_issue627_unknown_config_keys.py::
TestTheReportIsSafeForTheTerminal` was the same defect and was fixed
separately in #984.

CI does not go red on these today: no workflow sets `FORCE_COLOR`/`TERM`, and
`CliRunner`/`StringIO` are non-ttys, so Rich disables colour on every runner.
They are latent-until-something-upstream-changes — exactly the shape of the
four incidents named at the top of this docstring.

At the time of writing this scanner finds **zero** offenders — it is a ratchet,
not a cleanup. `TestTheScannerCanActuallyFail` is therefore load-bearing: a
scanner that has nothing to find must still be shown capable of finding
something, or it is indistinguishable from one that is silently broken.
"""

from __future__ import annotations

import ast
import pathlib
import re
import textwrap

import pytest

from tests._source_scan import parse_source, walk_nodes

TESTS_DIR = pathlib.Path(__file__).parent

# A quoted CLI flag: "--noise-floor", '--gpus'.
_FLAG_RE = re.compile(r"""["']--[a-z][a-z0-9-]*["']""")
# The raw CLI-output expressions an assertion might read.
_RAW_OUTPUT_RE = re.compile(r"\b(result\.output|\.stdout\b|readouterr\(\))")
# The most reliable signal that a line strips ANSI is the escape byte itself,
# written inline as a regex. Measured across the suite: 31 correct assertions in
# 18 files spell it `re.sub(r"\x1b\[[0-9;]*m", "", result.output)` with no named
# helper at all, so a list of blessed helper NAMES flags all of them as
# offenders. Detecting the escape is name-agnostic and does not police style —
# the project's own "scan, don't hand-write a list" rule applied here.
_ESCAPE_LITERAL_RE = re.compile(r"\\x1b|\\033|\\u001b|\\e\[", re.IGNORECASE)

# Named helpers, kept as a second route for files that wrap the regex away.
_NORMALISERS = (
    "_plain(",
    "_strip_ansi(",
    "strip_ansi(",
    "_clean(",
    "_clean_help(",
    "_ANSI_RE",
    "_ANSI_ESCAPE",
    "_ANSI.",
    "no_color",
)


def _looks_normalised(line: str) -> bool:
    """True when ``line`` strips ANSI — inline regex or a named helper.

    Whitespace collapse alone is deliberately NOT normalisation: ``" ".join(
    text.split())`` is exactly the insufficient fix that shipped in v0.73.2 and
    turned CI red.
    """
    if _ESCAPE_LITERAL_RE.search(line):
        return True
    return any(token in line for token in _NORMALISERS)


_ASSIGN_RE = re.compile(r"^\s*([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.+)$")


def find_raw_help_assertions(source: str) -> list[tuple[int, str]]:
    """Return ``(lineno, text)`` for every unsafe `--help` flag assertion.

    Scoped per test FUNCTION: a function that invokes ``--help`` and then
    asserts a quoted flag against text that has not had ANSI stripped.

    **Indirection is tracked, and that is the whole point.** The assertion that
    actually broke CI in v0.73.2 never mentioned ``result.output`` at all::

        plain = " ".join(result.output.split())   # collapses WHITESPACE only
        assert "--noise-floor" in plain           # asserts on the variable

    A first version of this scanner looked for ``result.output`` on the assert
    line, was verified against an invented example, passed — and did **not**
    catch the real commit. So a variable derived from raw output is followed,
    and is treated as unsafe unless its derivation strips ANSI. Whitespace
    collapse alone is exactly the insufficient fix that shipped.

    Returns ``[]`` for source that cannot be parsed, so one malformed file never
    fails the whole guard.
    """
    try:
        tree = parse_source(source)
    except SyntaxError:  # pragma: no cover — a broken file fails its own tests
        return []
    lines = source.splitlines()
    offenders: list[tuple[int, str]] = []
    for node in walk_nodes(tree):
        if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            continue
        if not node.name.startswith("test"):
            continue
        end = node.end_lineno or node.lineno
        body = lines[node.lineno - 1 : end]
        joined = "\n".join(body)
        if '"--help"' not in joined and "'--help'" not in joined:
            continue

        # Variables carrying CLI output that was never ANSI-stripped.
        tainted: set[str] = set()
        for offset, line in enumerate(body):
            stripped = line.strip()

            if not stripped.startswith("assert"):
                match = _ASSIGN_RE.match(line)
                if match:
                    name, rhs = match.group(1), match.group(2)
                    derived = _RAW_OUTPUT_RE.search(rhs) or any(
                        f"{var}" in rhs for var in tainted
                    )
                    if derived and not _looks_normalised(rhs):
                        tainted.add(name)
                    elif derived:
                        tainted.discard(name)
                continue

            if _looks_normalised(line):
                continue
            if not _FLAG_RE.search(stripped):
                continue
            reads_raw = _RAW_OUTPUT_RE.search(stripped)
            reads_tainted = any(
                re.search(rf"\b{re.escape(var)}\b", stripped) for var in tainted
            )
            if reads_raw or reads_tainted:
                offenders.append((node.lineno + offset, stripped[:100]))
    return offenders


class TestNoRawHelpAssertionsInTheSuite:
    def test_every_test_file_normalises_help_output(self):
        offenders: list[str] = []
        scanned = 0
        for path in sorted(TESTS_DIR.glob("test_*.py")):
            source = path.read_text(encoding="utf-8", errors="replace")
            if "--help" not in source:
                continue
            scanned += 1
            for lineno, text in find_raw_help_assertions(source):
                offenders.append(f"{path.name}:{lineno}: {text}")
        assert scanned > 0, "the scan found no --help tests at all — scanner broken?"
        assert not offenders, (
            "A `--help` assertion is reading RAW CLI output. Rich splits flag "
            "names with ANSI escapes on Linux/macOS, so this passes on Windows "
            "and turns CI red everywhere else (it has, four times). Route the "
            "output through an ANSI-strip + whitespace-collapse helper — see "
            "tests/test_v07302.py::_plain.\n  " + "\n  ".join(offenders)
        )

    def test_the_scan_actually_covers_the_suite(self):
        """CONTROL for the control: if `--help` tests stopped being discovered,
        the guard above would pass by scanning nothing."""
        scanned = [
            p.name
            for p in TESTS_DIR.glob("test_*.py")
            if "--help" in p.read_text(encoding="utf-8", errors="replace")
        ]
        assert len(scanned) >= 20, f"only {len(scanned)} files scanned: {scanned[:5]}"


class TestTheScannerCanActuallyFail:
    """The scanner currently finds nothing. That is only meaningful if it is
    shown able to find something — otherwise it is indistinguishable from a
    scanner that silently matches nothing at all."""

    BAD = '''
def test_help_mentions_the_flag():
    result = runner.invoke(app, ["ship", "--help"])
    assert "--noise-floor" in result.output
'''

    #: VERBATIM from `15b57f7`, the commit whose CI went red on all nine test
    #: cells. Not paraphrased — the first version of this scanner passed its
    #: invented example and missed this, because the assertion names a local
    #: variable rather than `result.output`.
    REAL_BROKEN = '''
    def test_the_flag_exists_and_is_documented(self):
        from typer.testing import CliRunner

        from soup_cli.cli import app

        result = CliRunner().invoke(app, ["ship", "--help"])
        assert result.exit_code == 0, (result.output, repr(result.exception))
        plain = " ".join(result.output.split())
        assert "--noise-floor" in plain
'''

    def test_it_catches_the_real_commit_that_broke_ci(self):
        """The load-bearing test in this file. Whitespace-collapse via a local
        variable is the shape that shipped; a scanner that misses it is
        decorative."""
        found = find_raw_help_assertions(textwrap.dedent(self.REAL_BROKEN))
        assert len(found) == 1, found
        assert "--noise-floor" in found[0][1]

    def test_it_catches_the_direct_shape_too(self):
        found = find_raw_help_assertions(self.BAD)
        assert len(found) == 1
        assert "--noise-floor" in found[0][1]

    def test_the_actual_repaired_version_is_accepted(self):
        """CONTROL. The shipped fix — same indirection, but ANSI-stripped —
        must NOT be flagged, or the guard would demand rewriting correct code."""
        fixed = textwrap.dedent(self.REAL_BROKEN).replace(
            'plain = " ".join(result.output.split())', "plain = _plain(result.output)"
        )
        assert find_raw_help_assertions(fixed) == []

    def test_a_normalised_assertion_is_accepted(self):
        good = self.BAD.replace("in result.output", "in _plain(result.output)")
        assert find_raw_help_assertions(good) == []

    def test_an_error_message_assertion_is_not_flagged(self):
        """The narrow scope, pinned. A flag named in an ERROR message is plain
        interpolated text that Rich does not style per-token — 13 such
        assertions exist in this suite and every one is correct."""
        err = '''
def test_missing_model_is_reported():
    result = runner.invoke(app, ["deploy", "ollama"])
    assert result.exit_code == 1
    assert "--model" in result.output
'''
        assert find_raw_help_assertions(err) == []

    def test_an_assertion_without_a_flag_is_not_flagged(self):
        plain = '''
def test_help_renders():
    result = runner.invoke(app, ["ship", "--help"])
    assert "SHIP" in result.output
'''
        assert find_raw_help_assertions(plain) == []

    def test_unparseable_source_does_not_explode(self):
        assert find_raw_help_assertions("def broken(:\n") == []


# ---------------------------------------------------------------------------
# #633 — the same hazard, on subcommands the `--help` scan never reaches.
# ---------------------------------------------------------------------------

#: Commands whose output is **syntax highlighted**. Rich renders these through
#: Pygments, which emits SGR escapes *between* the tokens of one logical line:
#: ``modality``, ``:`` and ``text`` are three tokens, so ``"modality: text"`` is
#: not a substring of the rendered output and ``yaml.safe_load`` rejects
#: ``\x1b``. Plain ``console.print`` output is not affected the same way -- an
#: unstyled message is wrapped in escapes rather than split by them. Two
#: mechanisms are in play here, not one: ``recipes show`` and ``migrate`` render
#: through ``Syntax(...)`` i.e. Pygments, while ``data mix --apply`` uses plain
#: ``console.print`` and is split by Rich's default ``ReprHighlighter`` instead.
#: The scanner keys on the INVOCATION rather than the mechanism, which is why it
#: covers both; the suite's ~200 other raw-output assertions are left alone.
_HIGHLIGHTED_INVOCATIONS = (
    re.compile(r'"recipes"\s*,\s*"show"'),
    re.compile(r"'recipes'\s*,\s*'show'"),
    re.compile(r'"mix"\s*,\s*"--(apply|optimize)"'),
    re.compile(r"'mix'\s*,\s*'--(apply|optimize)'"),
    # `soup migrate` is the third `Syntax(...)` site in `src/`
    # (commands/migrate.py:121). Latent rather than live: no assertion on its
    # output is multi-token today, so this flags nothing now. Added by the
    # maintainer after review so the guard covers every highlighted command
    # rather than the two this issue happened to surface.
    re.compile(r'"migrate"'),
    re.compile(r"'migrate'"),
)

_PARSES_OUTPUT = re.compile(r"(yaml\.safe_load|json\.loads)\s*\(")

#: Broader than ``_RAW_OUTPUT_RE``, which anchors on the exact name
#: ``result.output`` and therefore cannot match ``show_result.output`` -- ``_``
#: is a word character, so ``\b`` never fires before ``result`` there. Real
#: tests name their results ``show_result`` / ``use_result`` all the time.
_ANY_RAW_OUTPUT_RE = re.compile(r"\b[A-Za-z_][A-Za-z0-9_]*\.(output|stdout)\b|readouterr\(\)")
_STRING_LITERAL = re.compile(r'"([^"]{2,})"' + r"|'([^']{2,})'")


def _is_multi_token(literal: str) -> bool:
    """A literal that Pygments would split across tokens.

    The hazard is YAML *structure*. ``modality: text`` is three tokens -- key,
    punctuation, value -- with escapes between them, and ``train:`` is two. A
    bare model id (``Qwen/Qwen3.8-27B``) is one token and survives raw, which is
    why neighbouring assertions in the same file kept passing.

    A plain message such as ``"validation failed"`` is also safe: Rich wraps an
    unstyled string in escapes rather than splitting it, so requiring a colon
    keeps the guard on the assertions that actually break rather than every
    substring containing a space.
    """
    return ":" in literal


def _assert_expression(line: str) -> str:
    """Return the tested expression of an ``assert``, dropping its message.

    ``assert "train:" in compact, result.output`` reads raw output only in the
    *failure message*, which is harmless and in fact good practice. Scanning the
    whole line would flag it, so the top-level comma is found (ignoring commas
    inside brackets or strings) and everything after it discarded.
    """
    depth = 0
    quote = ""
    for index, char in enumerate(line):
        if quote:
            if char == quote and line[index - 1 : index] != "\\":
                quote = ""
            continue
        if char in "\"'":
            quote = char
        elif char in "([{":
            depth += 1
        elif char in ")]}":
            depth -= 1
        elif char == "," and depth == 0:
            return line[:index]
    return line


def _assert_statement_lines(tree: ast.AST) -> set[int]:
    """Every line belonging to an `assert` statement, continuations included.

    The scanner decides `is this an assertion?` from `stripped.startswith(
    "assert")`, which is true of the first physical line only. Once the
    formatter wraps a long assertion, the line that actually reads the output
    is a continuation and was dropped one branch after the literal-skip rule.
    Fixing only the skip rule moved the false negative rather than closing it,
    which is why both are needed.
    """
    covered: set[int] = set()
    for node in walk_nodes(tree):
        if not isinstance(node, ast.Assert):
            continue
        # `node.test` ONLY, never `node.msg`. Reading raw output inside a
        # failure message is harmless and in fact good practice --
        # `_assert_expression` already drops it on a single-line assert, and
        # taking the whole statement span here would re-introduce exactly that
        # false positive one line lower, on the wrapped form. Measured: it
        # flagged `test_recipes_v031.py:468`, a correct assertion, before this
        # was narrowed.
        test = node.test
        covered.update(range(test.lineno, (test.end_lineno or test.lineno) + 1))
    return covered


def _multiline_literal_lines(tree: ast.AST) -> set[int]:
    """Line numbers occupied by a string constant that spans several lines.

    Those are this guard's own synthetic fixtures: adjacent string literals are
    merged by the parser into ONE `ast.Constant`, so a fixture block written as
    several quoted lines has `end_lineno > lineno` and is skipped whole.

    A real assertion the formatter wrapped -- `assert (` on one line and
    `"modality: text" in result.output` on the next -- carries only a
    SINGLE-line constant, so it is not skipped. The rule this replaced skipped
    any line beginning with a quote and could not tell the two apart, which
    made exactly the shape ruff will eventually produce invisible (#635
    follow-up, closed by the maintainer).
    """
    covered: set[int] = set()
    for node in walk_nodes(tree):
        if not isinstance(node, ast.Constant) or not isinstance(node.value, str):
            continue
        end = node.end_lineno or node.lineno
        if end > node.lineno:
            covered.update(range(node.lineno, end + 1))
    return covered


def find_unsafe_highlighted_assertions(source: str) -> list[tuple[int, str]]:
    """Return ``(lineno, text)`` for unnormalised reads of highlighted output.

    Scoped per test FUNCTION, like ``find_raw_help_assertions``: a function that
    invokes a syntax-highlighted command and then either feeds raw output to a
    parser or asserts a multi-token substring against it.
    """
    try:
        tree = parse_source(source)
    except SyntaxError:  # pragma: no cover — a broken file fails its own tests
        return []
    lines = source.splitlines()
    literal_lines = _multiline_literal_lines(tree)
    assert_lines = _assert_statement_lines(tree)
    offenders: list[tuple[int, str]] = []
    for node in walk_nodes(tree):
        if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            continue
        if not node.name.startswith("test"):
            continue
        end = node.end_lineno or node.lineno
        body = lines[node.lineno - 1 : end]
        joined = "\n".join(body)
        if not any(pattern.search(joined) for pattern in _HIGHLIGHTED_INVOCATIONS):
            continue

        tainted: set[str] = set()
        for offset, line in enumerate(body):
            stripped = line.strip()
            if (node.lineno + offset) in literal_lines:
                # Inside a MULTI-LINE string constant -- the synthetic fixtures
                # this guard's own tests are built from. Scanning them would
                # flag the examples written to prove the guard works. A
                # single-line literal is deliberately NOT skipped, so a
                # formatter-wrapped assertion stays visible.
                continue
            if _looks_normalised(stripped):
                # Record that this variable is safe, then move on.
                match = _ASSIGN_RE.match(line)
                if match:
                    tainted.discard(match.group(1))
                continue

            match = _ASSIGN_RE.match(line)
            if match and not stripped.startswith("assert"):
                name, rhs = match.group(1), match.group(2)
                derived = _ANY_RAW_OUTPUT_RE.search(rhs) or any(
                    re.search(rf"\b{re.escape(var)}\b", rhs) for var in tainted
                )
                if derived:
                    tainted.add(name)
                    if _PARSES_OUTPUT.search(rhs):
                        offenders.append((node.lineno + offset, stripped[:100]))
                continue

            reads_raw = _ANY_RAW_OUTPUT_RE.search(stripped)
            reads_tainted = any(
                re.search(rf"\b{re.escape(var)}\b", stripped) for var in tainted
            )
            if not (reads_raw or reads_tainted):
                continue
            if _PARSES_OUTPUT.search(stripped):
                offenders.append((node.lineno + offset, stripped[:100]))
                continue
            is_assert_head = stripped.startswith("assert")
            if not (is_assert_head or (node.lineno + offset) in assert_lines):
                continue
            expression = (
                _assert_expression(stripped) if is_assert_head else stripped
            )
            if not (
                _ANY_RAW_OUTPUT_RE.search(expression)
                or any(re.search(rf"\b{re.escape(var)}\b", expression) for var in tainted)
            ):
                continue
            found = _STRING_LITERAL.search(expression)
            literal = (found.group(1) or found.group(2)) if found else ""
            if literal and _is_multi_token(literal):
                offenders.append((node.lineno + offset, stripped[:100]))
    return offenders


# ---------------------------------------------------------------------------
# #1068 — Rich ReprHighlighter splits percentage / number tokens in raw output
# ---------------------------------------------------------------------------

_IDENTIFIER_EXEMPT_RE = re.compile(
    r"""^(?:
        --?[a-zA-Z0-9_-]+                       # CLI flags: --min-acceptance, -v
        |[A-Za-z0-9._-]+/[A-Za-z0-9._-]+        # Repo/model IDs: meta-llama/Llama-3.1-8B
        |[A-Za-z0-9]+-[0-9.]+[A-Za-z0-9-]*      # Model names: Llama-3.1-8B, Qwen2.5-7B
        # Versions: v0.71.26, 1.2.3
        |(?:v[0-9]+\.[0-9]+(?:\.[0-9]+)?|[0-9]+\.[0-9]+\.[0-9]+[a-zA-Z0-9.-]*)
        |[A-Z][0-9]{2,4}                        # Hardware names: H100, A100
    )$""",
    re.VERBOSE,
)

_PERCENT_LITERAL_RE = re.compile(r"""[0-9]+(?:\.[0-9]+)?%""")
_DIGIT_TEXT_RE = re.compile(
    r"""
    (?:
        \b\d+(?:/\d+|\.\d+)?\s+[a-zA-Z]+        # 5 rows, 128 tokens, 2/2 rows, 1/3 prompts
        |
        \b[a-zA-Z]+\s+\d+(?:/\d+|\.\d+)?\b        # Running 2, between 2, line 2
    )
""",
    re.VERBOSE,
)
_QUOTED_SUBSTR_RE = re.compile(r"""(?<!\w)(?:'[^'\n]+'|"[^"\n]+")(?:(?!\w)|$)""")
_KEY_EQUALS_RE = re.compile(r"""\b[a-zA-Z_][a-zA-Z0-9_]{1,30}=\S+""")
_GROUPED_OR_DECIMAL_RE = re.compile(
    r"""(?<![\w./-])(?:\d{1,3}(?:,\d{3})+(?:\.\d+)?|\d+\.\d+)(?![\w./-])"""
)


def _is_unsafe_cli_literal(s: str) -> bool:
    """Return True if ``s`` contains tokens split by Rich's ReprHighlighter."""
    s_strip = s.strip()
    if not s_strip or _IDENTIFIER_EXEMPT_RE.match(s_strip):
        return False
    if _PERCENT_LITERAL_RE.search(s_strip):
        return True
    if _DIGIT_TEXT_RE.search(s_strip):
        return True
    if _KEY_EQUALS_RE.search(s_strip):
        return True
    if _GROUPED_OR_DECIMAL_RE.search(s_strip):
        return True
    m = _QUOTED_SUBSTR_RE.search(s_strip)
    if m and m.group(0) != s_strip:
        return True
    return False


#: Inline exemption marker for intentional uncoloured assertions (e.g. non-CLI harnesses,
#: mock subprocesses, or tests with console explicitly pinned to no-color under #987).
_ANSI_OK_RE = re.compile(r"#\s*ansi-ok\s*:\s*(.+)$")


def _find_raw_cli_output_statements(
    source: str,
) -> list[tuple[int, int, str, bool]]:
    """Return (start_line, end_line, first_line_text, has_ansi_ok) for raw output asserts."""
    try:
        tree = parse_source(source)
    except SyntaxError:  # pragma: no cover — a broken file fails its own tests
        return []
    lines = source.splitlines()
    literal_lines = _multiline_literal_lines(tree)
    records: list[tuple[int, int, str, bool]] = []

    for fn in walk_nodes(tree):
        if not isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)):
            continue
        if not fn.name.startswith("test"):
            continue

        tainted: set[str] = set()
        stmts = sorted(
            [
                s
                for s in walk_nodes(fn)
                if isinstance(s, (ast.Assign, ast.AnnAssign, ast.AugAssign, ast.Assert))
            ],
            key=lambda s: s.lineno,
        )

        for stmt in stmts:
            if stmt.lineno in literal_lines:
                continue

            if isinstance(stmt, (ast.Assign, ast.AnnAssign, ast.AugAssign)):
                end = stmt.end_lineno or stmt.lineno
                rhs_text = "\n".join(lines[stmt.lineno - 1 : end])
                derived = _ANY_RAW_OUTPUT_RE.search(rhs_text) or any(
                    re.search(rf"\b{re.escape(var)}\b", rhs_text) for var in tainted
                )
                target_names: list[str] = []
                if isinstance(stmt, ast.Assign):
                    for t in stmt.targets:
                        if isinstance(t, ast.Name):
                            target_names.append(t.id)
                elif isinstance(stmt, (ast.AnnAssign, ast.AugAssign)) and isinstance(
                    stmt.target, ast.Name
                ):
                    target_names.append(stmt.target.id)

                for name in target_names:
                    if derived and not _looks_normalised(rhs_text):
                        tainted.add(name)
                    elif derived and _looks_normalised(rhs_text):
                        tainted.discard(name)

            elif isinstance(stmt, ast.Assert):
                test = stmt.test
                test_end = test.end_lineno or test.lineno
                test_text = "\n".join(lines[test.lineno - 1 : test_end])
                if stmt.msg:
                    test_text = _assert_expression(test_text)

                if _looks_normalised(test_text):
                    continue

                reads_raw = _ANY_RAW_OUTPUT_RE.search(test_text)
                reads_tainted = any(
                    re.search(rf"\b{re.escape(var)}\b", test_text) for var in tainted
                )
                if not (reads_raw or reads_tainted):
                    continue

                # Negative assertions ("not in") are unproblematic because missing
                # text cannot be split by ANSI
                if " not in " in test_text:
                    continue

                has_unsafe = False
                for sub in ast.walk(test):
                    if isinstance(sub, ast.Constant) and isinstance(sub.value, str):
                        if _is_unsafe_cli_literal(sub.value):
                            has_unsafe = True
                            break

                if has_unsafe:
                    stmt_start = stmt.lineno
                    stmt_end = stmt.end_lineno or stmt.lineno
                    stmt_context = lines[max(0, stmt_start - 2) : stmt_end]
                    has_ansi_ok = any(_ANSI_OK_RE.search(line) for line in stmt_context)
                    first_line = lines[stmt_start - 1].strip()
                    records.append((stmt_start, stmt_end, first_line[:100], has_ansi_ok))

    return records


def find_raw_cli_output_assertions(
    source: str, ignore_exemptions: bool = False
) -> list[tuple[int, str]]:
    """Return ``(lineno, text)`` for unsafe token assertions on CLI output.

    Rich's default ``ReprHighlighter`` styles numbers (such as ``60.0%``,
    ``128 tokens``, ``2/2 rows valid``, ``1,000``, ``2.0``), quoted substrings,
    and ``key=`` pairs inside console output. On colour-capable terminals, SGR
    escape sequences land inside or between those tokens, causing assertions on
    raw output to fail under ``FORCE_COLOR=1 TERM=xterm-256color`` while passing
    on Windows.
    """
    records = _find_raw_cli_output_statements(source)
    if ignore_exemptions:
        return [(start, text) for start, _, text, _ in records]
    return [(start, text) for start, _, text, has_ok in records if not has_ok]


def find_stale_ansi_ok_markers(source: str) -> list[tuple[int, str]]:
    """Return ``(lineno, line_text)`` for any ``# ansi-ok:`` marker that does not
    actively suppress an otherwise-flagged raw CLI output assertion."""
    try:
        tree = parse_source(source)
    except SyntaxError:
        return []
    lines = source.splitlines()
    literal_lines = _multiline_literal_lines(tree)
    marker_lines: list[tuple[int, str]] = []
    for idx, line in enumerate(lines, 1):
        if idx in literal_lines:
            continue
        if _ANSI_OK_RE.search(line):
            marker_lines.append((idx, line.strip()))

    if not marker_lines:
        return []

    records = _find_raw_cli_output_statements(source)
    stale: list[tuple[int, str]] = []
    for lineno, text in marker_lines:
        matches_any = any(
            start - 1 <= lineno <= end for start, end, _, _ in records
        )
        if not matches_any:
            stale.append((lineno, text))

    return stale


class TestNoRawCliOutputAssertionsInTheSuite:
    def test_every_test_file_normalises_cli_output_tokens(self):
        offenders: list[str] = []
        scanned = 0
        for path in sorted(TESTS_DIR.glob("test_*.py")):
            source = path.read_text(encoding="utf-8", errors="replace")
            scanned += 1
            for lineno, text in find_raw_cli_output_assertions(source):
                offenders.append(f"{path.name}:{lineno}: {text}")
        assert scanned >= 20, f"only {scanned} test files scanned"
        assert not offenders, (
            "An assertion on CLI output reads numbers/percentages/tokens from raw "
            "output without ANSI stripping. Rich's ReprHighlighter styles numbers "
            "and splits them with ANSI escapes on Linux/macOS, so this passes on "
            "Windows and turns CI red on colour terminals (issue #1068). Route the "
            "output through an ANSI-strip helper (e.g. _plain or strip_ansi):\n  "
            + "\n  ".join(offenders)
        )

    def test_no_stale_ansi_ok_markers_in_the_suite(self):
        """Every # ansi-ok: marker must actively suppress an otherwise-flagged
        assertion. A stale marker on safe code is dead debt and must be deleted."""
        stale: list[str] = []
        total_markers = 0
        this_file = pathlib.Path(__file__).resolve()
        for path in sorted(TESTS_DIR.glob("test_*.py")):
            if path.resolve() == this_file:
                continue
            source = path.read_text(encoding="utf-8", errors="replace")
            markers = find_stale_ansi_ok_markers(source)
            for lineno, text in markers:
                stale.append(f"{path.name}:{lineno}: {text}")
            tree = parse_source(source)
            lit_lines = _multiline_literal_lines(tree)
            total_markers += sum(
                1
                for idx, line in enumerate(source.splitlines(), 1)
                if idx not in lit_lines and _ANSI_OK_RE.search(line)
            )

        assert total_markers > 0, "no # ansi-ok: markers found in suite — scanner broken?"
        assert not stale, (
            "Found stale `# ansi-ok:` marker(s) that do not suppress any flagged "
            "assertion:\n  " + "\n  ".join(stale)
        )


class TestTheCliOutputScannerCanActuallyFail:
    """A scanner that finds zero offenders in the active suite is only trustworthy
    if it is proven capable of finding both direct, indirect, wrapped, and token-based
    unsafe assertions."""

    DIRECT_BAD = '''
def test_draft_acceptance():
    result = runner.invoke(app, ["measure", "--target", "org/target"])
    assert "60.0%" in result.output
'''

    INDIRECT_BAD = '''
def test_draft_acceptance_indirect():
    result = runner.invoke(app, ["measure", "--target", "org/target"])
    plain = " ".join(result.output.split())
    assert "60.0%" in plain
'''

    WRAPPED_BAD = '''
def test_draft_acceptance_wrapped():
    result = runner.invoke(app, ["measure", "--target", "org/target"])
    assert (
        "60.0%"
        in result.output
    )
'''

    NON_PCT_BAD = '''
def test_merge_row_count():
    result = runner.invoke(app, ["data", "merge"])
    assert "5 rows" in result.output
'''

    KEY_EQUALS_BAD = '''
def test_param_forwarding():
    result = runner.invoke(app, ["train"])
    assert "iterations=2" in result.output
'''

    REAL_PRE_FIX = '''
def test_below_min_acceptance_exits_two(self, runner, in_tmp_cwd, monkeypatch):
    result = runner.invoke(
        app,
        ["measure", "--target", "org/target", "--draft", "org/tiny",
         "--prompts", prompts, "--min-acceptance", "0.6"],
    )
    assert result.exit_code == 2
    assert "60.0%" in result.output
    assert "below" in result.output.lower()
'''

    INLINE_FIXTURE = '''
def test_inline_fixture_handling():
    fixture = """
    assert "60.0%" in result.output
    """
    assert fixture
'''

    def test_it_catches_direct_raw_output_assertion(self):
        found = find_raw_cli_output_assertions(textwrap.dedent(self.DIRECT_BAD))
        assert len(found) == 1, found
        assert "60.0%" in found[0][1]

    def test_it_catches_indirect_whitespace_collapsed_assertion(self):
        found = find_raw_cli_output_assertions(textwrap.dedent(self.INDIRECT_BAD))
        assert len(found) == 1, found
        assert "60.0%" in found[0][1]

    def test_it_catches_wrapped_raw_output_assertion(self):
        found = find_raw_cli_output_assertions(textwrap.dedent(self.WRAPPED_BAD))
        assert len(found) == 1, found
        assert "assert (" in found[0][1] or "60.0%" in found[0][1]

    def test_it_catches_non_percentage_unsafe_literal(self):
        found = find_raw_cli_output_assertions(textwrap.dedent(self.NON_PCT_BAD))
        assert len(found) == 1, found
        assert "5 rows" in found[0][1]

    def test_it_catches_key_equals_unsafe_literal(self):
        found = find_raw_cli_output_assertions(textwrap.dedent(self.KEY_EQUALS_BAD))
        assert len(found) == 1, found
        assert "iterations=2" in found[0][1]

    def test_it_catches_the_real_pre_fix_pattern_from_issue1068(self):
        found = find_raw_cli_output_assertions(textwrap.dedent(self.REAL_PRE_FIX))
        assert len(found) == 1, found
        assert "60.0%" in found[0][1]

    def test_a_normalised_direct_assertion_is_accepted(self):
        good = self.DIRECT_BAD.replace("in result.output", "in _plain(result.output)")
        assert find_raw_cli_output_assertions(textwrap.dedent(good)) == []

    def test_a_normalised_indirect_assertion_is_accepted(self):
        good = self.INDIRECT_BAD.replace(
            'plain = " ".join(result.output.split())',
            "plain = _plain(result.output)",
        )
        assert find_raw_cli_output_assertions(textwrap.dedent(good)) == []

    def test_plain_non_token_assertion_is_not_flagged(self):
        plain = '''
def test_draft_measure():
    result = runner.invoke(app, ["measure", "--target", "org/target"])
    assert "STRONG" in result.output
'''
        assert find_raw_cli_output_assertions(textwrap.dedent(plain)) == []

    def test_exempt_model_identifier_is_not_flagged(self):
        model = '''
def test_adapters_list():
    result = runner.invoke(app, ["adapters", "list"])
    assert "Llama-3.1-8B" in result.output
'''
        assert find_raw_cli_output_assertions(textwrap.dedent(model)) == []

    def test_failure_message_reading_raw_output_is_not_flagged(self):
        msg_only = '''
def test_draft_acceptance_msg():
    result = runner.invoke(app, ["measure", "--target", "org/target"])
    plain = _plain(result.output)
    assert "60.0%" in plain, result.output
'''
        assert find_raw_cli_output_assertions(textwrap.dedent(msg_only)) == []

    def test_multiline_string_fixture_in_test_is_skipped(self):
        assert find_raw_cli_output_assertions(textwrap.dedent(self.INLINE_FIXTURE)) == []

    def test_unparseable_source_does_not_explode(self):
        assert find_raw_cli_output_assertions("def broken(:\n") == []

    @pytest.mark.parametrize("bad_number", ["1,000", "50,000", "0.7", "3.14"])
    def test_it_catches_grouped_and_decimal_numbers(self, bad_number):
        src = f'''
def test_number_output():
    result = runner.invoke(app, ["measure"])
    assert "{bad_number}" in result.output
'''
        found = find_raw_cli_output_assertions(textwrap.dedent(src))
        assert len(found) == 1, found
        assert bad_number in found[0][1]

    def test_ansi_ok_inline_marker_suppresses_offender(self):
        src = '''
def test_uncoloured_stdout():
    result = runner.invoke(app, ["measure"])
    assert "60.0%" in result.output  # ansi-ok: mock subprocess stdout is uncoloured
'''
        assert find_raw_cli_output_assertions(textwrap.dedent(src)) == []
        found = find_raw_cli_output_assertions(textwrap.dedent(src), ignore_exemptions=True)
        assert len(found) == 1
        assert "60.0%" in found[0][1]

    def test_stale_ansi_ok_marker_is_flagged(self):
        src = '''
def test_safe():
    assert "plain text" in result.output  # ansi-ok: unnecessary exemption
'''
        stale = find_stale_ansi_ok_markers(textwrap.dedent(src))
        assert len(stale) == 1
        assert "unnecessary exemption" in stale[0][1]