Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier3): require invocation evidence for script execution credit (#154)

Maverick33eb9dc74a09
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b632c4d5..a5ff6d62 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -41,6 +41,78 @@ All notable changes to SkillEvaluator are documented in this file.
- Report Tier 2 embedding and LLM service failures as incomplete checks, retaining
a nonzero exit without inventing duplicate-content findings. Provider error
messages include recovery guidance without echoing raw response bodies.
+- Tier 3 script execution credit now requires evidence that the expected script
+ was invoked. `check_script_execution` previously treated the script name as a
+ substring of an execution command, so reading, printing or searching the
+ script, or running a similarly named file, scored a full `Executed <script>`.
+ Credit is now given only for a recognised invocation: the script run directly,
+ an interpreter given it as its script argument, a `source`, or a `sh -c`
+ payload that does one of those, with `cd` tracked and script identity compared
+ exactly. Interpreter and wrapper options come from grammars derived by running
+ each option against a script that records whether it executed, so `--help`,
+ `perl -c` and `bash -n` run no script while `python -Wignore` and
+ `env FOO=1` still resolve to theirs. A command the walk cannot resolve keeps
+ the existing 0.75 partial credit rather than being scored either way: a path
+ built at run time, an option outside a grammar, a name only in heredoc data,
+ inline code or a module that names the script, and anything reaching a command
+ through standard input, including `xargs` and `parallel`, whose behaviour the
+ command text never determines. Partial credit is only ever given for a
+ command that names the script: an unresolved command that never mentions it
+ scores zero, as before. Redirections standing before the script
+ (`python3 < /dev/null run.py`), a script's own arguments that look like shell
+ options (`bash run.sh -c '...'`), invocations inside `if`, `while`, `until`
+ and `for` bodies, and versioned interpreter names (`perl5.38.2`, `python3.13`)
+ resolve as the shell runs them. A loop over an empty list keeps the earlier
+ binding and its body is not read; a binding made inside `( ... )` stays
+ there; the last command of a pipeline keeps its bindings where the shell
+ does (zsh, ksh) and not where it forks it (bash, dash, mksh), with an
+ option change that could move it (`shopt -s lastpipe`, `emulate sh`)
+ read as unresolved; a quoted or escaped word that would read as syntax
+ (`'done'`, `printf "("`, `';|'`) is the ordinary word it is; groups and
+ compound pipeline stages nest in either order, each compound tested for its
+ own pipe; `((` closed by `))` is an arithmetic command where the shell has
+ one; the positional
+ parameters are empty unless the text gives some, so `for f; do` at the top
+ level runs nothing and keeps the variable's value; an interpreter fed its
+ program by a pipe (`cat run.py | python3`) is unresolved like
+ `python3 < run.py`; a variable bound by `export` or `readonly`, or by
+ `declare` and `typeset` where the shell has them, holds the value it had
+ when the builtin ran, `unset` empties it, `+x` and `export -n` unexport it
+ however the builtin is reached, and one bound from data the text does not
+ carry (`read`, `printf -v`, `local` outside a function, `declare -u`, a name
+ `eval` may bind, including from a program held in a variable) is
+ unresolved, as is a later assignment to a name given `-i`, `-u` or `-n`,
+ while `-l` lowercases it and an array is never exported; what is done to a
+ `-n` name (`nameref` in ksh and mksh) leaves the name it refers to
+ unresolved; `let`, `$((...))` and `$[...]` assign as `((...))` does; an
+ assignment the shell rejects (a value that is not a number for an `-i`
+ name) and a special builtin given an option the shell rejects end the
+ credit where the shell stops there; an assignment written before a command,
+ `env NAME=value` included, is that command's environment only, reaching
+ neither its own words nor the commands after it, except before a special
+ builtin in the POSIX shells; a `-c` payload's shell sees the exported names,
+ its command's own prefix and what `env` adds and removes, and a `$` quoted
+ or escaped from the outer shell is expanded there (one left unquoted is
+ expanded here, a name never bound to nothing), with the payload's own
+ heredoc bodies kept as data; text that `eval`, inline
+ code or a shell reading a heredoc may expand again is unresolved when a
+ variable in it holds the script; every heredoc declared on a line takes its
+ body after the line, in order, and past the number a shell accepts on one
+ line (16 in bash) the line and the rest are data; a here-string, which dash
+ and busybox ash reject before running anything, leaves nothing credited
+ under those shells, and so does a compound's opening word written after an
+ assignment (`A=1 for ...`, `A=1 if ...`, `A=1 ( ... )`), which every
+ modelled shell rejects and which no longer raises; `(((` is a subshell
+ around `((` in bash, zsh and mksh; and `ksh`, `mksh` and `ash` are
+ recognised shells.
+ Applied to both the host checker and the bundled Harbor verifier.
+
+- Added `scripts/script_invocation_differential.py`, a differential harness that
+ executes each command for real against fixtures that record whether they ran,
+ and compares the result against both implementations. `--baseline REF` also
+ scores every command with the checker at an earlier ref and lists each score
+ that moved, so a change that lowers a command that ran, or raises one that
+ did not, is seen before it is pushed.
### Added
diff --git a/scripts/script_invocation_differential.py b/scripts/script_invocation_differential.py
new file mode 100644
index 00000000..cb8b446e
--- /dev/null
+++ b/scripts/script_invocation_differential.py
@@ -0,0 +1,1420 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Differential harness for Tier 3 script-execution credit.
+
+``check_script_execution`` decides from a command's text whether the expected
+script was invoked. This harness checks that decision against what the shell
+actually does: every command is run for real in a throwaway directory against
+fixtures that write a marker file when they execute, and the marker, not an
+expectation written by hand, is the ground truth.
+
+Three outcomes matter:
+
+* a **false positive** is the checker scoring 1.0 for a command that ran
+ nothing, which is the defect this whole check exists to prevent;
+* a **false negative** is the checker scoring 0.0 for a command that did run
+ the script, which is a claim the text did not support either;
+* a **partial on a real run** is the checker scoring 0.75 for a command that
+ ran. That is not a defect. It is the checker reporting that it could not
+ resolve the command, which is the correct answer for a shape it does not
+ model, such as a path arriving through standard input;
+* a **partial without a reference** is the checker scoring 0.75 for a command
+ whose text never names the expected script. Parsing uncertainty over an
+ unrelated command is not evidence about the script, so this is a defect: the
+ checker has credited a command it knows nothing about.
+
+Host and bundled Harbor verifier are compared on every command as well, because
+the two copies must not drift.
+
+Run it with ``python scripts/script_invocation_differential.py``. Add
+``--fuzz N`` to also generate and execute N random commands built from shell
+pieces, and ``--seed`` to reproduce a particular generation. Commands that this
+machine cannot run (a tool that is not installed, a shell that refuses to parse)
+are reported separately and never counted as either kind of defect.
+
+Add ``--compose N`` to execute N commands that nest binding scopes (groups,
+brace groups, compound commands and pipeline stages, two or three deep, under
+bash, zsh and ksh). Commands a native tool call runs under the shell it names
+are executed by that shell (bash, zsh, ksh, mksh, dash and sh) and scored with
+the name passed through, and lines around one the shell refuses are scored
+even though it reports a syntax error. Add ``--baseline REF`` to also score every command with the checker as it
+stood at that git ref, and list each command whose score moved, against the
+marker. A fix that lowers the score of a command that ran, or raises it for
+one that did not, is a regression whatever else it repaired, so those are
+listed first for review before a revision is pushed.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import random
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(REPO_ROOT / "src"))
+
+from skillevaluator.tier3.eval_core import checks as host_checks
+
+TEMPLATE_PATH = REPO_ROOT / "src/skillevaluator/tier3/harbor/templates/eval.py"
+
+
+def _load_template():
+ spec = importlib.util.spec_from_file_location("harbor_eval_differential", TEMPLATE_PATH)
+ assert spec and spec.loader
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+TEMPLATE = _load_template()
+
+# A command that returns before its script does: ``setsid -f`` and ``nohup``
+# fork and detach, and ``&`` backgrounds. Their marker may land after the
+# shell exits, so the harness waits for it before reading "did not run".
+_DETACHED_RE = re.compile(r"setsid\s+-f|\bnohup\b|&\s*(?:$|;|\)|\|\|)")
+PY = "python3"
+SCRIPT = "skills/demo/run.py"
+O = "skills/demo/other.py"
+SHELL_SCRIPT = "skills/demo/run.sh"
+
+
+def _versioned_name(interpreter: str, version: str) -> str:
+ """The interpreter under a versioned name, as distributions install it.
+
+ Debian ships ``perl5.38.2`` beside ``perl`` and every platform ships
+ ``python3.12`` beside ``python3``. The longest installed form is used, so
+ the command executes here; where none is installed the command is
+ reported as not runnable rather than scored. An interpreter outside the
+ default search path is named by its absolute path, because ``env -i``
+ empties PATH and would otherwise fail to find it for a reason the command
+ text does not carry.
+ """
+ parts = version.split(".")
+ candidates = [interpreter + ".".join(parts[:count]) for count in range(len(parts), 0, -1)]
+ for name in candidates:
+ found = shutil.which(name)
+ if found:
+ return name if Path(found).parent in {Path("/bin"), Path("/usr/bin")} else found
+ return candidates[0]
+
+
+def _perl_version() -> str:
+ try:
+ completed = subprocess.run(["perl", "-e", 'printf "%vd", $^V'], capture_output=True, text=True, timeout=10)
+ except (OSError, subprocess.TimeoutExpired):
+ return "5.38.2"
+ return completed.stdout.strip() or "5.38.2"
+
+
+VERSIONED_PY = _versioned_name("python", f"{sys.version_info.major}.{sys.version_info.minor}")
+VERSIONED_PERL = _versioned_name("perl", _perl_version())
+
+
+def _marker(name: str) -> str:
+ return f"#!/usr/bin/env python3\nimport pathlib; pathlib.Path('MARKER.{name}').write_text('ran')\nprint('{name}')\n"
+
+
+FIXTURES = {
+ "skills/demo/run.py": _marker("run.py"),
+ "skills/demo/other.py": _marker("other.py"),
+ "skills/demo/run.py.bak": _marker("run.py.bak"),
+ "skills/demo/rerun.py": _marker("rerun.py"),
+ "skills/demo/run.sh": "#!/bin/sh\necho ran > MARKER.run.sh\necho sh\n",
+ "skills/demo/run.pl": "open(my $f,'>','MARKER.run.pl'); print $f 'ran'; print \"pl\\n\";\n",
+ "skills/demo/run.rb": "File.write('MARKER.run.rb','ran'); puts 'rb'\n",
+ "skills/demo/run.js": "require('fs').writeFileSync('MARKER.run.js','ran'); console.log('js')\n",
+ "run.py": _marker("run.py"),
+ "other.py": _marker("other.py"),
+ "another.py": _marker("another.py"),
+ "skills/demo/another.py": _marker("another.py"),
+ # a name carrying the private-use character the checker uses as a mark,
+ # which must stay a different file from run.py
+ "skills/demo/\ue000run.py": _marker("\ue000run.py"),
+ # ``python3 2 > out.txt run.py`` runs a script named 2: these exist so that
+ # command succeeds, as the shell reads it, without touching run.py.
+ "0": "print('zero')\n",
+ "1": "print('one')\n",
+ "2": "print('two')\n",
+ "skills/demo/0": "print('zero')\n",
+ "skills/demo/1": "print('one')\n",
+ "skills/demo/2": "print('two')\n",
+ "notes.txt": "notes\n",
+ # a loop reading `< notes.txt` must find it after `cd skills/demo` too,
+ # otherwise its body never runs for a reason the command text does not carry
+ "skills/demo/notes.txt": "notes\n",
+ "lib.js": "module.exports={}\n",
+ "lib.rb": "",
+}
+
+NON_EXECUTING_VERBS = [
+ "cat",
+ "head -2",
+ "tail -2",
+ "grep -n ran",
+ "wc -l",
+ "ls -l",
+ "stat",
+ "file",
+ "md5sum",
+ "touch",
+ "echo",
+ "printf",
+]
+# Well-formed verbs that need a second argument, kept apart so the generator
+# does not build a command that fails for want of one.
+TWO_ARGUMENT_VERBS = ["cp -n", "diff"]
+INTERPRETER_FORMS = [
+ "",
+ "-u ",
+ "-B ",
+ "-OO ",
+ "-I ",
+ "-q ",
+ "-Wignore ",
+ "-W ignore ",
+ "-Xdev ",
+ "-X dev ",
+ "-- ",
+ "--help ",
+ "-V ",
+ "-c 'print(1)' ",
+ "-m json.tool ",
+ "-c 'import runpy; runpy.run_path(\"skills/demo/run.py\")' ",
+]
+WRAPPER_FORMS = [
+ "",
+ "timeout 20 ",
+ "timeout --help ",
+ "timeout -- 20 ",
+ "timeout -s TERM -- 20 ",
+ "nohup ",
+ "nohup -- ",
+ "nice -n 3 ",
+ "nice -5 ",
+ "nice -- ",
+ "env ",
+ "env -i ",
+ "env FOO=1 ",
+ "env --help ",
+ "env -- ",
+ "stdbuf -o0 ",
+ "stdbuf --help ",
+ "stdbuf -o0 -- ",
+ "setsid -f ",
+ "setsid --help ",
+ "setsid -- ",
+ "xargs ",
+ "xargs -r ",
+ "xargs -p ",
+ "xargs --help ",
+ "uv run ",
+ "uv run -- ",
+ "uv run --no-project -- ",
+ "uv --help run ",
+]
+PREFIX_FORMS = [
+ "",
+ "FOO=1 ",
+ "A=1 B=2 ",
+ "cd skills/demo && ",
+ "(cd skills/demo && ",
+ "x=$((1 << 2)); ",
+ "echo $((2 << 1)) ; ",
+ "cat <<EOF\nnotes\nEOF\n",
+ "cat <<-EOF\n\tnotes\n\tEOF\n",
+ "cat <<'EOF'\nrun.py\nEOF\n",
+ "echo '<<' ; ",
+ "cat <<< notes ; ",
+ 'cat <<< "a; b" ; ',
+ "printf '' | ",
+ "echo x | ",
+]
+SUFFIX_FORMS = ["", " > out.txt", " 2>&1", " ; echo done", " || true"]
+# Redirections that stand before the script: the shell's, not the interpreter's.
+REDIRECT_FORMS = [
+ "< /dev/null ",
+ "</dev/null ",
+ "0< /dev/null ",
+ "0</dev/null ",
+ "0<notes.txt ",
+ "<&0 ",
+ "2>/dev/null ",
+ "2> /dev/null ",
+ "2>&1 ",
+ "2>&- ",
+ "2>>err.txt ",
+ "1>>out.txt ",
+ "1> out.txt ",
+ "3<>notes.txt ",
+]
+# A digit standing apart from its operator is an operand: ``python3 2 > out.txt
+# run.py`` runs the script named 2 and hands it run.py.
+DETACHED_DESCRIPTOR_FORMS = ["2 > out.txt ", "2 >out.txt ", "1 >> out.txt ", "0 < notes.txt "]
+# A second loop header for the same variable, after ``for f in run.py; do cat $f; done``.
+REBINDING_HEADERS = [
+ "for f in {other} {another}; do {body}; done",
+ "for f; do {body}; done",
+ "for f in; do {body}; done",
+ "for f in; do :; done; {body}",
+ 'for f in ""; do :; done; {body}',
+ "for f in {other}; do {body}; done",
+ "for f in {script}; do {body}; done",
+ "for g in x; do {body}; done",
+]
+# Control structures around a command, with the command as {body}.
+CONTROL_FORMS = [
+ "if true; then {body}; fi",
+ "if false; then true; else {body}; fi",
+ "if false; then true; elif true; then {body}; fi",
+ "for i in one; do {body}; done",
+ "while read -r _; do {body}; done < notes.txt",
+ "until false; do {body}; break; done",
+ "case x in x) {body};; esac",
+]
+# Arguments a shell script receives, which look like the shell's own options.
+SCRIPT_ARGUMENT_FORMS = [" -c 'echo done'", " --help", " -- -x", " -n"]
+# Commands that never name the expected script, which no walk may credit.
+UNRELATED_BODIES = [
+ "cd",
+ "cd skills",
+ "true",
+ f"{PY} skills/demo/other.py",
+ "timeout --frobnicate 5 python3 skills/demo/other.py",
+ "python3 -Q skills/demo/other.py",
+ "env -Z python3 skills/demo/other.py",
+]
+
+
+def curated() -> list[tuple[str, str]]:
+ """Command shapes chosen deliberately, each paired with its expected script."""
+ cases: list[tuple[str, str]] = []
+
+ def add(command: str, script: str = "run.py") -> None:
+ cases.append((command, script))
+
+ for form in INTERPRETER_FORMS:
+ add(f"{PY} {form}{SCRIPT}")
+ for wrapper in WRAPPER_FORMS:
+ add(f"{wrapper}{PY} {SCRIPT}")
+ for verb in NON_EXECUTING_VERBS:
+ add(f"{verb} {SCRIPT}")
+ add(f"{verb} {SCRIPT} && {PY} {SCRIPT}")
+ for verb in TWO_ARGUMENT_VERBS:
+ add(f"{verb} {SCRIPT} copy.py")
+ add(f"{verb} {SCRIPT} copy.py && {PY} {SCRIPT}")
+ for prefix in PREFIX_FORMS:
+ target = "run.py" if prefix.startswith(("cd ", "(cd ")) else SCRIPT
+ closing = ")" if prefix.startswith("(") else ""
+ add(f"{prefix}{PY} {target}{closing}")
+ for suffix in SUFFIX_FORMS:
+ add(f"{PY} {SCRIPT}{suffix}")
+
+ # identity: a name that merely contains the expected one is a different file
+ add(f"{PY} skills/demo/run.py.bak", "run.py")
+ add(f"{PY} skills/demo/rerun.py", "run.py")
+ add(f"{PY} skills/demo/other.py skills/demo/run.py", "run.py")
+
+ # other interpreters, and options belonging to a different one
+ for interpreter, script in (
+ ("perl", "skills/demo/run.pl"),
+ ("ruby", "skills/demo/run.rb"),
+ ("node", "skills/demo/run.js"),
+ ("bash", "skills/demo/run.sh"),
+ ("sh", "skills/demo/run.sh"),
+ ):
+ name = script.rsplit("/", 1)[-1]
+ for option in (
+ "",
+ "-w ",
+ "-c ",
+ "-e 'x' ",
+ "--help ",
+ "--version ",
+ "--check ",
+ "--require ./lib.js ",
+ "-I lib ",
+ "-Mstrict ",
+ "-Wignore ",
+ ):
+ add(f"{interpreter} {option}{script}", name)
+
+ # nested shells, sourcing and grouping
+ add(f"sh -c '{PY} {SCRIPT}'")
+ add(f'bash -c "{PY} {SCRIPT}"')
+ add(f"sh -c 'cat {SCRIPT}'")
+ add(f"bash -xc '{PY} {SCRIPT}'")
+ add(f"source {SCRIPT}")
+ add(f". {SCRIPT}")
+ add(f"{{ {PY} {SCRIPT}; }}; echo done")
+ add(f"(cd skills/demo && {PY} run.py); echo done")
+
+ # heredoc bodies and terminators
+ add(f"cat <<EOF\nnotes\nEOF\n{PY} {SCRIPT}")
+ add(f"cat <<EOF\n{PY} {SCRIPT}\nEOF")
+ add(f"cat <<EOF\n{PY} {SCRIPT}\nEOF\ncat out.txt")
+ add(f"cat <<A\none\nA\ncat <<B\ntwo\nB\n{PY} {SCRIPT}")
+
+ # inline code, modules, eval and standard input: text this walk does not read
+ add(f"""{PY} -c 'import runpy; runpy.run_path("{SCRIPT}")'""")
+ add(f"""{PY} -c "exec(open('{SCRIPT}').read())" """.strip())
+ add(f"""{PY} -c 'import sys, runpy; runpy.run_path(sys.argv[1])' {SCRIPT}""")
+ add(f"{PY} -c 'print(123)' {SCRIPT}")
+ add(f"""{PY} -c "print('{SCRIPT}')" """.strip())
+ add(f"{PY} -m json.tool {SCRIPT}")
+ add(f"{PY} -m runpy {SCRIPT}")
+ add(f"eval '{PY} {SCRIPT}'")
+ add("eval 'echo hello'")
+ add(f"{PY} <{SCRIPT}")
+ add(f"{PY} < {SCRIPT}")
+ add(f"wc -l <{SCRIPT}")
+ add(f"{PY} - {SCRIPT}")
+ add(f"bash -s {SCRIPT}")
+ add(f"exec {PY} {SCRIPT}")
+ add(f"command {PY} {SCRIPT}")
+ # an invocation carried inside a command with no grammar of its own
+ add(f"flock /tmp/sied.lock {PY} {SCRIPT}")
+ add(f"taskset -c 0 {PY} {SCRIPT}")
+ add(f"ionice -c3 {PY} {SCRIPT}")
+ add(f"chrt -o 0 {PY} {SCRIPT}")
+ add(f"strace -f -o /dev/null {PY} {SCRIPT}")
+ add(f"flock /tmp/sied.lock cat {SCRIPT}")
+ add(f"strace -f -o /dev/null cat {SCRIPT}")
+ # shapes whose behaviour depends on data the command text does not carry
+ add(f"{PY} $(echo {SCRIPT})")
+ add(f"{PY} `echo {SCRIPT}`")
+ add(f"{PY} ${{PWD}}/{SCRIPT}")
+ add(f"echo {SCRIPT} | xargs {PY}")
+ add(f"xargs -I{{}} {PY} {{}} <<< {SCRIPT}")
+ add(f"find skills -name run.py -exec {PY} {{}} \\;")
+
+ # from review of the first revision: commands the shell runs differently
+ # from how the walk read them
+ for body in UNRELATED_BODIES:
+ add(body)
+ for argument in SCRIPT_ARGUMENT_FORMS:
+ add(f"bash {SHELL_SCRIPT}{argument}", "run.sh")
+ add(f"bash -- {SHELL_SCRIPT}{argument}", "run.sh")
+ add(f"bash -c 'echo done' {SHELL_SCRIPT}", "run.sh")
+ add(f"bash -c 'bash {SHELL_SCRIPT}' x", "run.sh")
+ for redirect in REDIRECT_FORMS:
+ add(f"{PY} {redirect}{SCRIPT}")
+ add(f"{PY} -u {redirect}{SCRIPT}")
+ add(f"bash -c {redirect}'bash {SHELL_SCRIPT}'", "run.sh")
+ add(f"bash {redirect}-c 'bash {SHELL_SCRIPT}'", "run.sh")
+ add(f"bash -c {redirect}'cat {SHELL_SCRIPT}'", "run.sh")
+ for redirect in DETACHED_DESCRIPTOR_FORMS:
+ add(f"{PY} {redirect}{SCRIPT}")
+ add(f"cd skills/demo && {PY} {redirect}run.py")
+ add(f"bash -c 2 '{SHELL_SCRIPT}'", "run.sh")
+ names = {"script": SCRIPT, "other": "skills/demo/other.py", "another": "skills/demo/another.py"}
+ for header in REBINDING_HEADERS:
+ add(f"for f in {SCRIPT}; do cat $f; done; " + header.format(body=f"{PY} $f", **names))
+ add(f"for f in {SCRIPT}; do cat $f; done; " + header.format(body="cat $f", **names))
+ # A pipeline runs its commands in subshells, so a binding made in one
+ # does not reach the command after the pipeline.
+ add(f"printf '' | for f in {SCRIPT}; do cat $f; done; for g in x; do {PY} $f; done")
+ add(f"for f in {SCRIPT}; do cat $f; done | true; {PY} $f")
+ add(f"printf '' | for f in {SCRIPT}; do {PY} $f; done")
+ add(f"echo x | while read -r l; do for f in {SCRIPT}; do {PY} $f; done; done")
+ add(f"if true; then for f in {SCRIPT}; do cat $f; done; fi | true; {PY} $f")
+ add(f"for f in {SCRIPT}; do cat $f; done; {PY} $f")
+ for control in CONTROL_FORMS:
+ add(control.format(body=f"{PY} {SCRIPT}"))
+ add(control.format(body=f"cat {SCRIPT}"))
+ add(control.format(body=f"timeout 20 {PY} {SCRIPT}"))
+ add(f"for f in {SCRIPT}; do {PY} $f; done")
+ add(f"for f in {SCRIPT}; do cat $f; done")
+ add(f"for f in {SCRIPT} skills/demo/other.py; do {PY} $f; done")
+ add(f"for f in {SCRIPT} skills/demo/other.py; do cat $f; done")
+ add(f"if {PY} {SCRIPT}; then echo ok; fi")
+ add(f"if true; then FOO=1 {PY} {SCRIPT}; fi")
+ for interpreter, script in ((VERSIONED_PY, SCRIPT), (VERSIONED_PERL, "skills/demo/run.pl")):
+ name = script.rsplit("/", 1)[-1]
+ add(f"{interpreter} {script}", name)
+ add(f"{interpreter.rsplit('/', 1)[-1]} {script}", name)
+ add(f"{interpreter} -c {script}", name)
+ add(f"{interpreter} --version {script}", name)
+ add(f"env -i {interpreter} {script}", name)
+
+ # redirections written flush against a descriptor and spaced apart
+ # tokenize to the same words, so each pair must score the same
+ for attached, spaced in (
+ (f"0<{SCRIPT}", f"0< {SCRIPT}"),
+ (f"<{SCRIPT}", f"< {SCRIPT}"),
+ (f"2>err.txt {SCRIPT}", f"2> err.txt {SCRIPT}"),
+ (f"2>>err.txt {SCRIPT}", f"2>> err.txt {SCRIPT}"),
+ (f"1>out.txt {SCRIPT}", f"1> out.txt {SCRIPT}"),
+ (f"0</dev/null {SCRIPT}", f"0< /dev/null {SCRIPT}"),
+ (f"2>&1 {SCRIPT}", f"2>&1 {SCRIPT}"),
+ (f"2>&- {SCRIPT}", f"2>&- {SCRIPT}"),
+ (f"{SCRIPT} 2>err.txt", f"{SCRIPT} 2> err.txt"),
+ ):
+ add(f"{PY} {attached}")
+ add(f"{PY} {spaced}")
+ add(f"cat 0<{SCRIPT}")
+ add(f"cat 0< {SCRIPT}")
+
+ # a loop with an empty list never runs and never assigns its variable
+ add(f'f={SCRIPT}; for f in; do :; done; {PY} "$f"')
+ add(f'f={SCRIPT}; for f in; do {PY} "$f"; done')
+ add(f"for f in; do {PY} {SCRIPT}; done")
+ add(f'f={SCRIPT}; for f in; do :; done; cat "$f"')
+ add(f'for f in {SCRIPT}; do :; done; for f in; do :; done; {PY} "$f"')
+ add(f'for f in ""; do {PY} {SCRIPT}; done')
+ add(f'f={SCRIPT}; printf "" | for f in; do :; done; {PY} "$f"')
+
+ # a binding made in the last stage of a pipeline survives in zsh and not
+ # in bash, dash or sh; one made in an earlier stage, or inside ( ), in none
+ for shell in ("zsh", "bash", "sh", "dash"):
+ add(f'{shell} -c \'printf "" | for f in {SCRIPT}; do cat "$f"; done; {PY} "$f"\'')
+ add(f'{shell} -c \'for f in {SCRIPT}; do cat "$f"; done | cat; {PY} "$f"\'')
+ add(f'{shell} -c \'printf "" | f={SCRIPT}; {PY} "$f"\'')
+ add(f'{shell} -c \'printf "" | if true; then f={SCRIPT}; fi; {PY} "$f"\'')
+ add(f'{shell} -c \'printf "" | {{ f={SCRIPT}; }}; {PY} "$f"\'')
+ add(f'{shell} -c \'printf "" | (f={SCRIPT}); {PY} "$f"\'')
+ add(f'{shell} -c \'printf "" | for f in {SCRIPT}; do {PY} "$f"; done\'')
+ add(f'printf "" | for f in {SCRIPT}; do cat "$f"; done; {PY} "$f"')
+ add(f'(f={SCRIPT}); {PY} "$f"')
+ add(f'(f={SCRIPT}; {PY} "$f")')
+ add(f'{{ f={SCRIPT}; }}; {PY} "$f"')
+ add(f'bash -c "shopt -s lastpipe; printf \\"\\" | for f in {SCRIPT}; do :; done; {PY} \\"\\$f\\""')
+ add(f'bash -O lastpipe -c "printf \\"\\" | for f in {SCRIPT}; do :; done; {PY} \\"\\$f\\""')
+ add(f'ksh -c \'printf "" | for f in {SCRIPT}; do :; done; {PY} "$f"\'')
+
+ # a quoted or escaped word is not syntax: a quoted done does not end a
+ # loop, a quoted parenthesis opens no subshell, a quoted separator splits
+ # no command, and a redirection operand keeps its whole name
+ add(f"{PY} 2>1.log {SCRIPT}")
+ add(f"{PY} 2> 1.log {SCRIPT}")
+ add(f"{PY} 1>2.txt {SCRIPT}")
+ add(f"{PY} '0<{SCRIPT}'")
+ add(f"for f in; do :; 'done'; {PY} {SCRIPT}; done")
+ add(f"for f in; do d'on'e; {PY} {SCRIPT}; done")
+ add(f"for f in; do :; d\\\\one; {PY} {SCRIPT}; done")
+ add(f"for f in; do 'for'; done; {PY} {SCRIPT}")
+ add(f'f={SCRIPT}; for g in; do printf done; done; {PY} "$f"')
+ add(f'f={SCRIPT}; for g in; do for f in skills/demo/other.py; do :; done; done; {PY} "$f"')
+ add(f'f={SCRIPT}; printf "("; f=skills/demo/other.py; printf ")"; {PY} "$f"')
+ add(f'f={SCRIPT}; printf \\\\(; f=skills/demo/other.py; printf \\\\); {PY} "$f"')
+ add(f"{PY} '|' {SCRIPT}")
+ add(f"'{PY}' {SCRIPT}")
+ add(f"printf '(' ; {PY} {SCRIPT}")
+ add(f'zsh -c \'emulate sh; printf "" | for f in {SCRIPT}; do :; done; {PY} "$f"\'')
+
+ # positional parameters: none at the top level, given by set, shift or the
+ # operands after a -c payload; and an interpreter fed its program by a pipe
+ add(f'f={SCRIPT}; for f in; do :; done; for f; do :; done; {PY} "$f"')
+ add(f"for f; do {PY} {SCRIPT}; done")
+ add(f"set -- {SCRIPT}; for f; do {PY} $f; done")
+ add(f"bash -c 'for f; do {PY} $f; done' _ {SCRIPT}")
+ add(f"bash -c 'for f; do {PY} $f; done' {SCRIPT}")
+ add(f"bash -c '{PY} $1' _ {SCRIPT}")
+ add(f"bash -c '{PY} other.py' _ {SCRIPT}")
+ add(f"cat {SCRIPT} | {PY}")
+ add(f"cat '{SCRIPT}' | {PY}")
+ add(f"cat {SCRIPT} | {PY} -")
+ add(f"cat {SCRIPT} | {PY} -c 'print(1)'")
+ add(f"cat {SCRIPT} | wc -l")
+ add(f"cat {SHELL_SCRIPT} | bash", "run.sh")
+
+ # from the generated audit: quoted metacharacter runs, a brace group as a
+ # pipeline stage, a group inside a pipeline, an empty loop inside a group,
+ # and a file whose name carries the mark character
+ add(f"printf '%s' ';|' {PY} {SCRIPT}")
+ add(f"printf '%s' \\\\;\\\\| {PY} {SCRIPT}")
+ add(f"{PY} ';;' {SCRIPT}")
+ add(f"{PY} '|&' {SCRIPT}")
+ add(f"{PY} '2>' {SCRIPT}")
+ add(f"{PY} {SCRIPT} ';|'")
+ for shell in ("bash", "zsh"):
+ add(f"{shell} -c 'f={O}; {{ f={SCRIPT}; :; }} | cat; {PY} \"$f\"'")
+ add(f'{shell} -c \'f={O}; printf "" | {{ :; f={SCRIPT}; }} | cat; {PY} "$f"\'')
+ add(f'{shell} -c \'f={O}; printf "" | {{ :; f={SCRIPT}; }}; {PY} "$f"\'')
+ add(f'{shell} -c \'f={O}; printf "" | (f={SCRIPT}; {PY} "$f"); {PY} "$f"\'')
+ add(f'{shell} -c \'f={O}; printf "" | (f={SCRIPT}; {PY} "$f") | cat; {PY} "$f"\'')
+ add(f'f={SCRIPT}; (f={O}; for g in; do :; done); {PY} "$f"')
+ add(f'f={O}; (f={SCRIPT}; for g in; do :; done); {PY} "$f"')
+ add(f'f={O}; (f={SCRIPT}; for g in; do for h in; do :; done; done) | cat; {PY} "$f"')
+ add(f'f={SCRIPT}; (f={O}; for g in; do :; done) | cat; (:); {PY} "$f"')
+ add(f'printf "" | (for f in {SCRIPT}; do :; done; {PY} $f)')
+ add(f"{PY} 'skills/demo/\ue000run.py'")
+ add(f"{PY} skills/demo/\ue000run.py")
+
+ # a pipeline inside a group still isolates its own stages, in both
+ # assignment directions, at the last and a middle stage, and nested
+ for first, second in ((SCRIPT, O), (O, SCRIPT)):
+ add(f'(f={first}; f={second} | cat; {PY} "$f")')
+ add(f'(f={first}; printf "" | f={second}; {PY} "$f")')
+ add(f'(f={first}; {{ f={second}; }} | cat; {PY} "$f")')
+ add(f'((f={first}; f={second} | cat); {PY} "$f")')
+ add(f'f={first}; ((f={second}) | cat; {PY} "$f")')
+ add(f'{{ f={first}; f={second} | cat; {PY} "$f"; }}')
+ add(f'zsh -c \'(f={first}; printf "" | f={second}; {PY} "$f")\'')
+ add(f"zsh -c '(f={first}; f={second} | cat; {PY} \"$f\")'")
+ # a group inside a compound stage, a compound stage inside another, and
+ # a segment that opens a compound and starts a pipeline inside it
+ for shell in ("bash", "zsh"):
+ add(f'{shell} -c \'printf "" | {{ f={first}; (f={second}); {PY} "$f"; }}\'')
+ add(f'{shell} -c \'printf "" | for g in 1; do f={first}; (f={second}); {PY} "$f"; done\'')
+ add(f"{shell} -c 'if true; then f={first}; (f={second}); {PY} \"$f\"; fi | cat'")
+ add(f"{shell} -c 'f={first}; {{ {{ f={second}; }} | cat; }}; {PY} \"$f\"'")
+ add(f"{shell} -c 'f={first}; {{ if true; then f={second}; fi | cat; }}; {PY} \"$f\"'")
+ add(f'{shell} -c \'f={first}; {{ printf "" | f={second}; }}; {PY} "$f"\'')
+ # an arithmetic command, and the nested-subshell reading of ((
+ add(f'f={first}; ((f={second})); {PY} "$f"')
+ add(f"ksh -c 'f={first}; ((f={second})); {PY} \"$f\"'")
+ add(f'f={first}; ((f={second}; g=1); {PY} "$f")')
+ # (( closed by )) is arithmetic in bash, zsh, ksh and mksh, whatever is
+ # inside; closed apart, or under dash, it is nested subshells
+ for shell in ("", "zsh", "ksh", "mksh", "dash", "sh"):
+ body = f'f={first}; ((f={second}; {PY} "$f")); {PY} "$f"'
+ add(f"{shell} -c '{body}'" if shell else body)
+ add(f'f={first}; ((f={second}; {PY} "$f") ); {PY} "$f"')
+ add(f'zsh -c \'f={first}; ((printf "" | for g in 1; do f={second}; {PY} "$f"; done)); {PY} "$f"\'')
+ # three opening parentheses: ( (( in bash, zsh and mksh, subshells in ksh and dash
+ for shell in ("", "zsh", "ksh", "mksh", "dash"):
+ body = f'f={first}; (((for f in {second}; do :; done; {PY} "$f")) | cat); {PY} "$f"'
+ add(f"{shell} -c '{body}'" if shell else body)
+
+ # from review of revision 16: builtins that bind a variable, in each shell
+ for declaration in BINDING_BUILTIN_FORMS:
+ add(f'{declaration} f={SCRIPT}; {PY} "$f"')
+ add(f'{declaration} f={SCRIPT}; cat "$f"')
+ for shell in ("dash", "zsh", "ksh", "mksh", "sh"):
+ add(f"{shell} -c '{declaration} f={SCRIPT}; {PY} \"$f\"'")
+ add(f'f={SCRIPT}; export f; {PY} "$f"')
+ add(f'f={SCRIPT}; unset f; {PY} "$f"')
+ add(f'f={SCRIPT}; unset -f f; {PY} "$f"')
+ add(f'readonly f={SCRIPT}; f={O}; {PY} "$f"')
+ add(f'f={O}; printf -v f {SCRIPT}; {PY} "$f"')
+ add(f'f={O}; echo {SCRIPT} | {{ read f; {PY} "$f"; }}')
+ # an assignment before a command is that command's environment only, and
+ # outlives a special builtin in the POSIX shells alone
+ for form in PREFIX_BINDING_FORMS:
+ text = form.format(v=SCRIPT, py=PY)
+ add(text)
+ for shell in ("bash", "dash", "zsh", "ksh", "mksh", "sh"):
+ add(f"{shell} -c '{text}'")
+ # what a -c payload's shell inherits, and a $ quoted or escaped from this shell
+ for launcher in (
+ f"export f={SCRIPT}; ",
+ f"f={SCRIPT}; ",
+ f"f={SCRIPT} ",
+ f"env f={SCRIPT} ",
+ f"set -a; f={SCRIPT}; ",
+ f"set -o allexport; f={SCRIPT}; set +a; ",
+ f"f={SCRIPT}; set -a; ",
+ f"set -a; set +a; f={SCRIPT}; ",
+ ):
+ add(f"{launcher}bash -c '{PY} \"$f\"'")
+ add(f'f={SCRIPT}; bash -c "{PY} $f"')
+ add(f'f={SCRIPT}; bash -c "{PY} \\$f"')
+ # every heredoc declared on a line takes its body after the line, in order
+ for form in MULTI_HEREDOC_FORMS:
+ add(form.format(py=PY, script=SCRIPT, other=O))
+
+ # from review of revision 17: env removes names before the child starts
+ for launcher in ENV_REMOVAL_FORMS:
+ add(f"export f={SCRIPT}; {launcher} bash -c '{PY} \"$f\"'")
+ add(f"export f={O}; env -i f={SCRIPT} bash -c '{PY} \"$f\"'")
+ add(f"export f={SCRIPT}; env -u g bash -c '{PY} \"$f\"'")
+ add(f"export f={SCRIPT}; env -u f f={SCRIPT} bash -c '{PY} \"$f\"'")
+ add(f'export f={SCRIPT}; bash -c \'env -u f bash -c "{PY} \\"\\$f\\""\'')
+ add(f"f={SCRIPT} env -i bash -c '{PY} \"$f\"'")
+ # a declaration copies the value when it runs, as a plain assignment does
+ for declaration in ("export ", "readonly ", "declare ", "typeset ", ""):
+ add(f'f={O}; {declaration}g=$f; f={SCRIPT}; {PY} "$g"')
+ add(f'f={SCRIPT}; {declaration}g=$f; f={O}; {PY} "$g"')
+ add(f'f={SCRIPT}; export f={O} g=$f; {PY} "$g"')
+ add(f"ksh -c 'f={O}; typeset f={SCRIPT} g=$f; {PY} \"$g\"'")
+ add(f"f={O}; export g=$f; f={SCRIPT}; bash -c '{PY} \"$g\"'")
+ # the export attribute removed, however the builtin is reached
+ for removal in EXPORT_REMOVAL_FORMS:
+ add(f"export f={SCRIPT}; {removal}; bash -c '{PY} \"$f\"'")
+ add(f'export f={SCRIPT}; {removal}; {PY} "$f"')
+ # more heredocs on a line than the shell accepts: the rest is data
+ for count in (9, 16, 17):
+ add(_heredocs(count, "cat", body=f"{PY} {SCRIPT}"))
+ add(_heredocs(count, PY, SCRIPT))
+ add(_heredocs(count, "cat") + f"{PY} {SCRIPT}")
+ for shell, count in (("mksh", 10), ("mksh", 11), ("dash", 17)):
+ add(f"{shell} -c '{_heredocs(count, PY, SCRIPT)}'")
+ # eval, inline code and a shell reading data may expand a quoted variable
+ for form in EVAL_FORMS:
+ add(form.format(v=SCRIPT, o=O, py=PY))
+ # an assignment to a read-only name fails, and some shells stop there
+ for refusal in ("f={o}", "export f={o}", "f={o} true", "unset f", "f={o} :", "for f in x; do :; done"):
+ text = f"readonly f={SCRIPT}; {refusal.format(o=O)}; {PY} {SCRIPT}"
+ add(text)
+ for shell in ("dash", "zsh", "ksh", "sh"):
+ add(f"{shell} -c '{text}'")
+ add(f"(readonly f={SCRIPT}; f={O}); {PY} {SCRIPT}")
+ add(f"readonly f={SCRIPT}; (f={O}; {PY} {SCRIPT})")
+ add(f"readonly f={SCRIPT}; f={O}; env -i f={SCRIPT} bash -c '{PY} \"$f\"'")
+ # an eval program held in a variable, and attributes kept with a name
+ for code in (f"f={O}", "export -n f", "unset f", "declare +x f", f"export f={O}", "echo hi"):
+ add(f"export f={SCRIPT}; code='{code}'; eval \"$code\"; bash -c '{PY} \"$f\"'")
+ add(f"export f={SCRIPT}; code='f={O}'; eval '$code'; bash -c '{PY} \"$f\"'")
+ add(f"export f={SCRIPT}; code='f={O}'; command eval \"$code\"; bash -c '{PY} \"$f\"'")
+ add(f"export f={O}; code='f={SCRIPT}'; eval \"$code\"; bash -c '{PY} \"$f\"'")
+ for attribute in ("-i", "-u", "-l", "-a"):
+ add(f'declare {attribute} f; f={SCRIPT}; {PY} "$f"')
+ add(f"export f={SCRIPT}; declare -a f; f={SCRIPT}; bash -c '{PY} \"$f\"'")
+ # a -n name passes what is done to it on to the name it refers to
+ for change in (f"f={O}", "unset f", f"export f={O}", f"read f <<< {O}", "((f=1))", "h=x"):
+ add(f'g={SCRIPT}; declare -n f=g; {change}; {PY} "$g"')
+ add(f'g={O}; declare -n f=g; f={SCRIPT}; {PY} "$g"')
+ add(f'g={SCRIPT}; declare -n f=g; declare +n f; f={O}; {PY} "$g"')
+ add(f'g={SCRIPT}; declare -n f=g; unset -n f; f={O}; {PY} "$g"')
+ add(f'g={SCRIPT}; declare -n f=g; declare -n f=h; f={O}; {PY} "$g"')
+ add(f"export g={SCRIPT}; declare -n f=g; export -n f; bash -c '{PY} \"$g\"'")
+ add(f'g={SCRIPT}; declare -n f; f=g; f={O}; {PY} "$g"')
+ add(f'h={SCRIPT}; declare -n f; f=g; {PY} "$h"')
+ for shell in ("ksh", "mksh", "bash"):
+ add(f"{shell} -c 'g={SCRIPT}; nameref f=g; f={O}; {PY} \"$g\"'")
+ # arithmetic binds: let, $((...)) and $[...]
+ for arithmetic in ("let f=1", 'let "f=1"', ": $((f=1))", "x=$((f+=1))", ": $[f=1]", 'eval "let f=1"'):
+ add(f'f={SCRIPT}; {arithmetic}; {PY} "$f"')
+ for control in ("let g=1", ": $((g=1))", "echo '$((f=1))'", "(: $((f=1)))", "echo $((f=1)) | cat", ": $( (f=1) )"):
+ add(f'f={SCRIPT}; {control}; {PY} "$f"')
+ add(f'f={SCRIPT}; {PY} "$f" $((f=1))')
+ # an assignment or builtin that fails where the shell stops
+ for failing in (f"f={O}", f"export f={SCRIPT}", f"for f in {SCRIPT}; do :; done", "f=5", "f=x", f"f={O} true"):
+ add(f"declare -i f; {failing}; {PY} {SCRIPT}")
+ for shell in ("ksh", "mksh"):
+ add(f"{shell} -c 'typeset -i f; typeset f={O}; {PY} {SCRIPT}'")
+ for shell in ("dash", "bash", "ksh", "mksh", "zsh"):
+ add(f"{shell} -c 'export f=1; export -n f; {PY} {SCRIPT}'")
+ add(f"{shell} -c 'f={SCRIPT}; unset -n f; {PY} \"$f\"'")
+ add(f"dash -c 'export f=1; command export -n f; {PY} {SCRIPT}'")
+ add(f'g={SCRIPT}; declare -n f; f={O}; {PY} "$g"')
+ add(f'f={O}; declare -n f; f={SCRIPT}; {PY} "$f"')
+ add(f"ksh -c 'f={O}; typeset -n f; {PY} {SCRIPT}'")
+ # a here-string the shell cannot parse; a heredoc inside a -c payload
+ for shell in ("dash", "sh", "bash", "zsh", "ksh", "mksh"):
+ add(f"{shell} -c 'cat <<< x; {PY} {SCRIPT}'")
+ add(f"dash -c 'export g={SCRIPT}; read f <<< x; {PY} \"$g\"'")
+ add(f"dash -c 'export g={SCRIPT}; read f < /dev/null; {PY} \"$g\"'")
+ add(f"dash -c \"echo '<<<'; {PY} {SCRIPT}\"")
+ for shell in ("bash", "dash"):
+ add(f"{shell} -c 'cat <<EOF\n{PY} {SCRIPT}\nEOF'")
+ add(f"{shell} -c 'cat <<EOF\nx\nEOF\n{PY} {SCRIPT}'")
+ add(f"bash -c '{PY} {SCRIPT} <<EOF\nignored\nEOF'")
+ add(f'bash -c "f={SCRIPT}; {PY} \\"$f\\""')
+ add(f'f={SCRIPT}; bash -c "{PY} \\"$f\\""')
+ add(f'bash -c "f={SCRIPT}; {PY} \\"\\$f\\""')
+ # a compound's opening word after an assignment, which every shell refuses
+ for opener in (
+ f"for g in; do {PY} {SCRIPT}; done",
+ f"for g in x; do {PY} {SCRIPT}; done",
+ f"while true; do {PY} {SCRIPT}; break; done",
+ f"if true; then {PY} {SCRIPT}; fi",
+ f"case x in x) {PY} {SCRIPT};; esac",
+ f"{{ {PY} {SCRIPT}; }}",
+ f"( {PY} {SCRIPT} )",
+ f"[[ -n x ]] && {PY} {SCRIPT}",
+ f"(( 1 )) && {PY} {SCRIPT}",
+ ):
+ add(f"LANG=C {opener}")
+ add(f"f=x for g in; do echo; done; {PY} {SCRIPT}")
+ add(f"A=1 for x; {PY} {SCRIPT}")
+ add(f"{PY} {SCRIPT}; A=1 for g in x; do :; done")
+ add(f"{PY} {SCRIPT}\nA=1 for g in x; do :; done")
+ add(f"dash -c 'A=1 for g in x; do {PY} {SCRIPT}; done'")
+ add(f"A=$(true) {PY} {SCRIPT}")
+ add(f"A='for' {PY} {SCRIPT}")
+ add(f"for g in x; do A=1 {PY} {SCRIPT}; done")
+ cases.extend(refusal_units())
+ return cases
+
+
+# Lines placed before and after a line the shell refuses: what ran before it
+# still ran, and nothing on or after it did; only a plain first command is
+# credited, so the rest are partial whether they ran or not. These are scored
+# even though the shell reports a syntax error, because the error is the
+# point of the shape.
+_REFUSED_LINES = (
+ "A=1 for g in x; do :; done",
+ "A=1 if true; then :; fi",
+ "A=1 ( : )",
+ "A=1 { :; }",
+ "A=1 (( 1 ))",
+)
+_BEFORE_REFUSED_LINE = (
+ f"{PY} {SCRIPT}",
+ f"cat {SCRIPT}",
+ f"echo a\n{PY} {SCRIPT}",
+ f"{PY} {SCRIPT} # comment",
+ f"for x in 1; do\n{PY} {SCRIPT}\ndone",
+ f"if true; then\n{PY} {SCRIPT}\nfi",
+ f"{{ {PY} {SCRIPT}\n}}",
+ f"( {PY} {SCRIPT}\n)",
+ f"f() {{\n{PY} {SCRIPT}\n}}\nf",
+ f"x=$(echo a\n); {PY} {SCRIPT}",
+ f"a=(1\n2)\n{PY} {SCRIPT}",
+ f"echo 'a\nb'; {PY} {SCRIPT}",
+ f"cat <<EOF\nx\nEOF\n{PY} {SCRIPT}",
+ f"cat <<'EOF'\nA=1 for g in x; do :; done\nEOF\n{PY} {SCRIPT}",
+ f"{PY} {SCRIPT} <<EOF\ny\nEOF",
+ f"case x in\nx) :;;\nesac\n{PY} {SCRIPT}",
+ f'export f={SCRIPT}\n{PY} "$f"',
+ f"{PY} {SCRIPT} &&",
+ f"{PY} {SCRIPT} |",
+ f"{PY} {SCRIPT} \\",
+ f"{PY} {SCRIPT};",
+ # Refused on its own line, or run after something that stops it.
+ f"{PY} {SCRIPT} >",
+ f"{PY} {SCRIPT} <",
+ f"{PY} {SCRIPT} 2>",
+ f"{PY} {SCRIPT} 2>&",
+ f"{PY} {SCRIPT} >;",
+ f"{PY} {SCRIPT} > # out",
+ f"{PY} {SCRIPT} >#out",
+ f"{PY} {SCRIPT}; ;",
+ f"if {PY} {SCRIPT}; fi",
+ f"{PY} {SCRIPT}\\\necho a",
+ f"exit\n{PY} {SCRIPT}",
+ f"exec true\n{PY} {SCRIPT}",
+ f"false && {PY} {SCRIPT}",
+ f"f() {{\n{PY} {SCRIPT}\n}}",
+ f"bash -c '{PY} {SCRIPT}'",
+ f"bash -c '{PY} {SCRIPT} >'",
+ # A carriage return before the newline is part of the word before it.
+ f"{PY} {SCRIPT}\r",
+ f"{PY} {SCRIPT};\r",
+ f"{PY} {SCRIPT} |&\r",
+ f"time -p {PY} {SCRIPT}\r",
+ f"f() {{\n{PY} {SCRIPT}\n}}\nf\r",
+ # Complete: these run.
+ f"{PY} {SCRIPT} 2>&1",
+ f"{PY} {SCRIPT} '>'",
+ f"{PY} {SCRIPT} >out #c",
+)
+
+
+def refusal_units() -> list[tuple[str, str]]:
+ """A refused line after complete commands, inside a compound, and before
+ an invocation, at the top level and in ``-c`` payloads."""
+ cases: list[tuple[str, str]] = []
+ for before in _BEFORE_REFUSED_LINE:
+ for refused in _REFUSED_LINES[:3]:
+ cases.append((f"{before}\n{refused}", "run.py"))
+ cases.append((f"{before}\n{_REFUSED_LINES[0]}\n{PY} {SCRIPT}", "run.py"))
+ for refused in _REFUSED_LINES:
+ cases.append((f"{PY} {SCRIPT}\n{refused}", "run.py"))
+ cases.append((f"{PY} {SCRIPT}; {refused}", "run.py"))
+ cases.append((f"{refused}\n{PY} {SCRIPT}", "run.py"))
+ cases.append((f"if true; then\n{PY} {SCRIPT}\n{refused}\nfi", "run.py"))
+ for shell in ("bash", "dash", "sh", "zsh", "ksh", "mksh"):
+ cases.append((f"{shell} -c '{PY} {SCRIPT}\n{_REFUSED_LINES[0]}'", "run.py"))
+ cases.append((f"{shell} -c '{PY} {SCRIPT}\ncat <<< x'", "run.py"))
+ cases.append((f"{shell} -c 'cat <<< x\n{PY} {SCRIPT}'", "run.py"))
+ cases.append((f"{shell} -c '{PY} {SCRIPT}; cat <<< x'", "run.py"))
+ cases.append((f"{shell} -c '{PY} {SCRIPT}\r\n{_REFUSED_LINES[0]}'", "run.py"))
+ cases.append((f"{shell} -c '{PY} {SCRIPT} >\n{_REFUSED_LINES[0]}'", "run.py"))
+ return cases
+
+
+_SCORED_DESPITE_SYNTAX_ERROR = frozenset(command for command, _ in refusal_units())
+
+
+# Commands a native tool call ran under the shell it names in
+# ``action_input.shell`` (Codex's ``exec_command``). Each is executed by that
+# shell and scored with the name passed through.
+_NATIVE_SHELL_COMMANDS = (
+ f'printf "" | for f in {SCRIPT}; do :; done; {PY} "$f"',
+ f'f={O}; printf "" | for f in {SCRIPT}; do :; done; {PY} "$f"',
+ f'printf "" | {{ f={SCRIPT}; }}; {PY} "$f"',
+ f'echo | while read -r l; do f={SCRIPT}; done; {PY} "$f"',
+ f'printf "" | for f in {SCRIPT}; do :; done | cat; {PY} "$f"',
+ f"cat <<< x; {PY} {SCRIPT}",
+ f"{PY} {SCRIPT}\ncat <<< x",
+ f"cat <<< x\n{PY} {SCRIPT}",
+ f"{PY} {SCRIPT}\nA=1 for g in x; do :; done",
+ f"{PY} {SCRIPT}; A=1 for g in x; do :; done",
+ f"A=1 for g in x; do :; done\n{PY} {SCRIPT}",
+ f"{PY} {SCRIPT} >\nA=1 for g in x; do :; done",
+ f"{PY} {SCRIPT}\r\nA=1 for g in x; do :; done",
+ f'declare f={SCRIPT}; {PY} "$f"',
+ f'typeset f={SCRIPT}; {PY} "$f"',
+ f'export f={SCRIPT}; {PY} "$f"',
+ f'f={SCRIPT}; ((f=1)); {PY} "$f"',
+ f'f={SCRIPT}; let f=1; {PY} "$f"',
+ # A listing, functions, and attributes that change a value, per shell.
+ f'export -p f={SCRIPT}; {PY} "$f"',
+ f'export f={SCRIPT}; export -p f={O}; {PY} "$f"',
+ f'export f={O}; export -p f={SCRIPT}; {PY} "$f"',
+ f'readonly -p f={SCRIPT}; {PY} "$f"',
+ f'export f={SCRIPT}; readonly -p f={O}; {PY} "$f"',
+ f"f={SCRIPT}; export -p f; bash -c '{PY} \"$f\"'",
+ f'f={SCRIPT}; readonly -p f; f={O}; {PY} "$f"',
+ f'f={SCRIPT}; export -p >/dev/null; {PY} "$f"',
+ f"f={SCRIPT}; export f; export -pn f; bash -c '{PY} \"$f\"'",
+ f'f={O}; typeset -p f={SCRIPT}; {PY} "$f"',
+ f'f={O}; typeset -px f={SCRIPT}; {PY} "$f"',
+ f'f={O}; typeset -f f={SCRIPT}; {PY} "$f"',
+ f"f={SCRIPT}; declare -F f; {PY} {SCRIPT}",
+ f"typeset -F f={SCRIPT}; {PY} {SCRIPT}",
+ f"f={SCRIPT}; typeset -F f; {PY} {SCRIPT}",
+ f"f=1.5; typeset -F f; {PY} {SCRIPT}",
+ f"f={SCRIPT}; typeset -i f; {PY} {SCRIPT}",
+ f"typeset -i f={SCRIPT}; {PY} {SCRIPT}",
+ f'f={SCRIPT}; typeset -L3 f; {PY} "$f"',
+ f'f={SCRIPT}; typeset -L f; {PY} "$f"',
+ f'f={SCRIPT}; typeset -L 3 f; {PY} "$f"',
+ # The directory lets the redirection succeed, so a POSIX shell goes on.
+ f'mkdir -p f={SCRIPT.rsplit("/", 1)[0]}; export f={SCRIPT}; export -p > f={O}; {PY} "$f"',
+ f'mkdir -p f={SCRIPT.rsplit("/", 1)[0]}; f={O}; export -p > f={SCRIPT}; {PY} "$f"',
+ f"f=0x10; typeset -i f; {PY} {SCRIPT}",
+ f"typeset -i f=16#ff; {PY} {SCRIPT}",
+ f"typeset -pF f={SCRIPT}; {PY} {SCRIPT}",
+ # A value that ends in an operator is not a redirection, and a base is a
+ # number however many zeros lead it.
+ f"export g='>' f={SCRIPT}; {PY} \"$f\"",
+ f"f={SCRIPT}; export g='>' f={O}; {PY} \"$f\"",
+ f"typeset -i f={'0' * 4400}16#ff; {PY} {SCRIPT}",
+ f"{PY} {SCRIPT}",
+ f"cat {SCRIPT}",
+ f'for f in {SCRIPT}; do {PY} "$f"; done',
+)
+_NATIVE_SHELLS = ("bash", "zsh", "ksh", "mksh", "dash", "sh")
+
+
+def native_shell_cases() -> list[tuple[str, str, str]]:
+ return [(command, "run.py", shell) for shell in _NATIVE_SHELLS for command in _NATIVE_SHELL_COMMANDS]
+
+
+def _heredocs(count: int, before: str, after: str = "", body: str = "x") -> str:
+ """``before``, then ``count`` heredocs declared on one line, ``after``, and each body in order."""
+ names = [f"D{index}" for index in range(count)]
+ declared = " ".join(f"<<{name}" for name in names)
+ return f"{before} {declared}{' ' + after if after else ''}\n" + "".join(f"{body}\n{name}\n" for name in names)
+
+
+# Builtins that bind a variable: bound in every shell (export, readonly), in
+# some (declare, typeset), or to a value the text does not settle.
+BINDING_BUILTIN_FORMS = [
+ "export",
+ "readonly",
+ "declare",
+ "typeset",
+ "typeset -x",
+ "export --",
+ "local",
+ "declare -u",
+ "export -p",
+ "readonly -p",
+ "declare -p",
+ "typeset -p",
+]
+PREFIX_BINDING_FORMS = [
+ 'f={v} {py} "$f"',
+ 'f={v} true; {py} "$f"',
+ 'env f={v} {py} "$f"',
+ 'f={v} :; {py} "$f"',
+ 'f={v} export g; {py} "$f"',
+]
+MULTI_HEREDOC_FORMS = [
+ "{py} <<A <<B {script}\nx\nA\ny\nB",
+ "{py} <<-A <<B {script}\n\tx\n\tA\ny\nB",
+ "{py} <<A <<B <<C {script}\nx\nA\ny\nB\nz\nC",
+ "{py} <<'A' <<\"B\" {script}\nx\nA\ny\nB",
+ "{py} <<A {script} <<B\nx\nA\ny\nB",
+ "cat <<A; {py} <<B {script}\nx\nA\ny\nB",
+ "cat <<A <<B {script}\nx\nA\ny\nB",
+ "{py} <<A <<B {other}\nx\nA\ny\nB",
+ "cat <<A <<B\nx\nA\n{py} {script}\nB",
+ "cat <<A <<B\n{py} {script}\nA\ny\nB",
+ "cat <<A <<A\nx\nA\ny\nA\n{py} {script}",
+ "{py} <<A <<<y {script}\nx\nA",
+]
+
+# env options that take names out of what a child inherits.
+ENV_REMOVAL_FORMS = [
+ "env -i",
+ "env -u f",
+ "env -",
+ "env --ignore-environment",
+ "env --unset=f",
+ "env -uf",
+ 'env -i PATH="$PATH"',
+ "env -i env",
+ "env -S '-u f'",
+]
+# Ways of taking the export attribute away, directly or through a builtin
+# prefix, and forms that touch functions only.
+EXPORT_REMOVAL_FORMS = [
+ "declare +x f",
+ "typeset +x f",
+ "declare -x +x f",
+ "export -n f",
+ "command export -n f",
+ "builtin export -n f",
+ "command command export -n f",
+ "command -p export -n f",
+ "builtin declare +x f",
+ "eval export -n f",
+ "export -f f",
+ "export -nf f",
+ "declare +x f; export f",
+ "export -pn f",
+ "export -p -n f",
+]
+EVAL_FORMS = [
+ "f={v} eval '{py} \"$f\"'",
+ "f={v}; eval '{py} \"$f\"'",
+ "export f={v}; eval '{py} \"$f\"'",
+ "f={o} eval '{py} \"$f\"'",
+ 'f={v}; eval f={o}; {py} "$f"',
+ "f={v}; eval 'unset f'; {py} \"$f\"",
+ 'f={v}; eval \'echo "$f"\'; {py} "$f"',
+ "f={v} {py} -c 'import os; os.system(\"{py} $f\")'",
+ 'f={v}; bash <<EOF\n{py} "$f"\nEOF',
+ "export f={v}; bash <<'EOF'\n{py} \"$f\"\nEOF",
+]
+
+
+# Scopes a binding can be made in, each wrapping the text inside it: groups,
+# brace groups, compound commands, and each of those as the first, a middle
+# or the last stage of a pipeline.
+SCOPE_WRAPPERS = [
+ "({x})",
+ "{{ {x}; }}",
+ 'printf "" | {{ {x}; }}',
+ "{{ {x}; }} | cat",
+ 'printf "" | {{ {x}; }} | cat',
+ 'printf "" | for g in 1; do {x}; done',
+ "for g in 1; do {x}; done | cat",
+ "if true; then {x}; fi | cat",
+ 'printf "" | if true; then {x}; fi',
+ 'printf "x\\n" | while read -r l; do {x}; done',
+ 'printf "" | ({x})',
+ "({x}) | cat",
+ "for g in 1; do {x}; done",
+ "if true; then {x}; fi",
+ "for g in; do :; done; {x}",
+ "{{ for g in; do :; done; {x}; }}",
+]
+SCOPE_BINDINGS = [
+ "f={v}",
+ "f={v} | cat",
+ 'printf "" | f={v}',
+ "(f={v})",
+ "for f in {v}; do :; done",
+ 'printf "" | for f in {v}; do :; done',
+ "{{ f={v}; }}",
+]
+
+
+def _compose(rng: random.Random) -> tuple[str, str]:
+ """A binding nested two or three scopes deep, read at a random level.
+
+ Which value the interpreter reads depends on which scopes keep a binding,
+ in which shell: a group never does, a pipeline stage only as zsh's or
+ ksh's last stage, a brace group or a loop always. Both assignment
+ directions are drawn, so a scope that leaks and one that forgets are both
+ caught.
+ """
+ outer, inner = rng.choice([(SCRIPT, O), (O, SCRIPT)])
+ text = rng.choice(SCOPE_BINDINGS).format(v=inner)
+ for _ in range(rng.choice([2, 3])):
+ wrapper = rng.choice(SCOPE_WRAPPERS)
+ text = wrapper.format(x=f'{text}; {PY} "$f"' if rng.random() < 0.4 else text)
+ command = f'f={outer}; {text}; {PY} "$f"'
+ shell = rng.choice(["", "zsh", "ksh", "bash"])
+ return (f"{shell} -c '{command}'" if shell else command), "run.py"
+
+
+def _generate(rng: random.Random) -> tuple[str, str]:
+ """One random command and the script it is scored against."""
+ prefix = rng.choice(PREFIX_FORMS)
+ script = "run.py"
+ target = "run.py" if prefix.startswith(("cd ", "(cd ")) else SCRIPT
+ interpreter = VERSIONED_PY if rng.random() < 0.15 else PY
+ redirect = rng.choice(REDIRECT_FORMS) if rng.random() < 0.15 else ""
+ if rng.random() < 0.05:
+ redirect = rng.choice(DETACHED_DESCRIPTOR_FORMS)
+ shape = rng.random()
+ if shape < 0.1:
+ body = rng.choice(UNRELATED_BODIES)
+ elif shape < 0.2:
+ script = "run.sh"
+ shell_target = "run.sh" if prefix.startswith(("cd ", "(cd ")) else SHELL_SCRIPT
+ body = f"bash {rng.choice(['', '-- ', '-x '])}{shell_target}{rng.choice(SCRIPT_ARGUMENT_FORMS)}"
+ elif shape < 0.27:
+ script = "run.sh"
+ shell_target = "run.sh" if prefix.startswith(("cd ", "(cd ")) else SHELL_SCRIPT
+ payload = rng.choice([f"bash {shell_target}", f"cat {shell_target}", f"./{shell_target}"])
+ between = rng.choice([*REDIRECT_FORMS, ""])
+ body = rng.choice([f"bash -c {between}'{payload}'", f"bash {between}-c '{payload}'"])
+ elif shape < 0.34:
+ header = rng.choice(REBINDING_HEADERS)
+ inside = prefix.startswith(("cd ", "(cd "))
+ names = {
+ "script": "run.py" if inside else SCRIPT,
+ "other": "other.py" if inside else "skills/demo/other.py",
+ "another": "another.py" if inside else "skills/demo/another.py",
+ }
+ body = f"for f in {names['script']}; do cat $f; done; " + header.format(
+ body=rng.choice([f"{interpreter} $f", "cat $f"]), **names
+ )
+ elif shape < 0.42:
+ # a binding made in a pipeline or a group, then read by the interpreter
+ inside = prefix.startswith(("cd ", "(cd "))
+ bound = "run.py" if inside else SCRIPT
+ binding = rng.choice(
+ [
+ f'printf "" | for f in {bound}; do cat "$f"; done',
+ f'for f in {bound}; do cat "$f"; done | cat',
+ f'printf "" | f={bound}',
+ f'printf "" | (f={bound})',
+ f'printf "" | {{ f={bound}; }}',
+ f'printf "" | if true; then f={bound}; fi',
+ 'printf "" | for f in; do :; done',
+ f"(f={bound})",
+ f"{{ f={bound}; }}",
+ f"f={bound}",
+ ]
+ )
+ reader = rng.choice([f'{interpreter} "$f"', 'cat "$f"'])
+ shell = rng.choice(["", "zsh -c ", "bash -c ", "sh -c ", "dash -c ", "ksh -c "])
+ body = f"{shell}'{binding}; {reader}'" if shell else f"{binding}; {reader}"
+ elif shape < 0.48:
+ # a variable a builtin or a prefix binds, every heredoc on a line, and
+ # three opening parentheses (from review of revision 16)
+ inside = prefix.startswith(("cd ", "(cd "))
+ bound = "run.py" if inside else SCRIPT
+ other = "other.py" if inside else O
+ closing = "\n)" if prefix.startswith("(") else ""
+ form = rng.random()
+ if form < 0.25:
+ text = rng.choice(MULTI_HEREDOC_FORMS).format(py=interpreter, script=bound, other=other)
+ # a suffix after the last terminator would unterminate it
+ return prefix + text + closing, script
+ if form < 0.35:
+ text = f'f={other}; (((for f in {bound}; do :; done; {interpreter} "$f")) | cat); {interpreter} "$f"'
+ shell = rng.choice(["", "zsh -c ", "ksh -c ", "mksh -c ", "dash -c "])
+ return prefix + (f"{shell}'{text}'" if shell else text) + closing, script
+ binder = rng.choice(
+ [
+ *(f"{declaration} f={bound}" for declaration in BINDING_BUILTIN_FORMS),
+ f"f={bound}; export f",
+ f"f={bound}; unset f",
+ f"f={bound} true",
+ f"env f={bound} true",
+ f"f={bound} :",
+ f"f={other}; printf -v f {bound}",
+ f"readonly f={bound}; f={other}",
+ # from review of revision 17
+ f"export f={bound}; declare +x f",
+ f"export f={bound}; command export -n f",
+ f"g={other}; export f=$g; g={bound}",
+ f"g={bound}; export f=$g; g={other}",
+ f"f={bound}; eval f={other}",
+ f"export f={bound}; eval export -n f",
+ ]
+ )
+ shell = rng.choice(["", "", "zsh -c ", "dash -c ", "ksh -c ", "mksh -c ", "sh -c "])
+ readers = [f'{interpreter} "$f"', 'cat "$f"']
+ if not shell:
+ readers.append(f"bash -c '{interpreter} \"$f\"'")
+ readers.append(f"env -u f bash -c '{interpreter} \"$f\"'")
+ readers.append(f"env -i f={bound} bash -c '{interpreter} \"$f\"'")
+ text = f"{binder}; {rng.choice(readers)}"
+ body = f"{shell}'{text}'" if shell else text
+ elif shape < 0.58:
+ body = f"{rng.choice(NON_EXECUTING_VERBS)} {target}"
+ else:
+ body = f"{rng.choice(WRAPPER_FORMS)}{interpreter} {rng.choice(INTERPRETER_FORMS)}{redirect}{target}"
+ if rng.random() < 0.15:
+ body = rng.choice(CONTROL_FORMS).format(body=body)
+ suffix = rng.choice(SUFFIX_FORMS)
+ if prefix.startswith("("):
+ suffix += ")"
+ command = prefix + body + suffix
+ if rng.random() < 0.3:
+ command += rng.choice([" && ", " ; ", " || ", "\n"]) + rng.choice(
+ ["echo tail", "true", f"{rng.choice(NON_EXECUTING_VERBS)} {target}"]
+ )
+ return command, script
+
+
+def _build(directory: Path) -> None:
+ for relative, text in FIXTURES.items():
+ path = directory / relative
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(text)
+ if path.suffix in {".py", ".sh", ".pl", ".rb", ".js"}:
+ path.chmod(0o755) # fixtures are invoked directly, so they need the bit
+
+
+def _load_baseline(ref: str):
+ """The host checker and the template as they stood at a git ref."""
+ modules = []
+ for name, relative in (
+ ("baseline_host", "src/skillevaluator/tier3/eval_core/checks.py"),
+ ("baseline_template", "src/skillevaluator/tier3/harbor/templates/eval.py"),
+ ):
+ source = subprocess.run(
+ ["git", "show", f"{ref}:{relative}"], capture_output=True, text=True, check=True, cwd=REPO_ROOT
+ ).stdout
+ path = Path(tempfile.mkdtemp()) / Path(relative).name
+ path.write_text(source)
+ spec = importlib.util.spec_from_file_location(name, path)
+ assert spec and spec.loader
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ modules.append(module)
+ return modules
+
+
+# Tools a command may name that are not installed everywhere. When one is
+# missing, the command did not run for a reason its text does not carry, and
+# the harness reports it as not runnable rather than scoring it; reading the
+# captured output alone misses a `command not found` sent to /dev/null.
+_EXTERNAL_TOOLS = (
+ "timeout",
+ "nohup",
+ "nice",
+ "stdbuf",
+ "setsid",
+ "xargs",
+ "parallel",
+ "flock",
+ "taskset",
+ "ionice",
+ "chrt",
+ "strace",
+ "uv",
+ "ruby",
+ "perl",
+ "node",
+ "zsh",
+ "ksh",
+ "mksh",
+ "dash",
+ "ash",
+)
+_TOOL_WORD_RE = re.compile(r"(?<![\w./-])(" + "|".join(_EXTERNAL_TOOLS) + r")(?![\w.-])")
+
+
+def _missing_tool(command: str) -> str | None:
+ for name in dict.fromkeys(_TOOL_WORD_RE.findall(command)):
+ if shutil.which(name) is None:
+ return name
+ return None
+
+
+def _run_one(
+ command: str, script: str, shell: str | None = None
+) -> tuple[str, str, float | None, bool | None, list[dict[str, str]]]:
+ """Execute one command and compare the marker against both checkers.
+
+ With ``shell``, the command is run by that shell and scored as a native
+ call naming it. Returns the outcome, its detail, the host score, whether
+ the script ran, and the tool call the checkers read, so a baseline can
+ score it.
+ """
+ missing = _missing_tool(command) if shell is None else (None if shutil.which(shell) else shell)
+ if missing is not None:
+ return ("inconclusive", f"{missing}: not installed here", None, None, [])
+ executable = shutil.which(shell) if shell is not None else "/bin/bash"
+ directory = Path(tempfile.mkdtemp())
+ try:
+ _build(directory)
+ try:
+ completed = subprocess.run(
+ command,
+ shell=True,
+ cwd=directory,
+ capture_output=True,
+ text=True,
+ timeout=30,
+ stdin=subprocess.DEVNULL,
+ executable=executable,
+ )
+ except subprocess.TimeoutExpired:
+ return ("inconclusive", "timed out", None, None, [])
+ output = completed.stdout + completed.stderr
+ ran = any(directory.rglob(f"MARKER.{script}"))
+ if not ran and _DETACHED_RE.search(command):
+ deadline = time.monotonic() + 2.0
+ while not ran and time.monotonic() < deadline:
+ time.sleep(0.05)
+ ran = any(directory.rglob(f"MARKER.{script}"))
+ refused_shape = shell is not None or command in _SCORED_DESPITE_SYNTAX_ERROR
+ if not ran and ("command not found" in output or ("syntax error" in output and not refused_shape)):
+ return ("inconclusive", output.strip().splitlines()[0][:70] if output.strip() else "", None, None, [])
+
+ calls = [
+ {
+ "action": "Bash",
+ "action_input": {"command": command},
+ "observation": output[:400] + f"\nExit code {completed.returncode}",
+ }
+ if shell is None
+ else {
+ "action": "exec_command",
+ "action_input": {"cmd": command, "shell": executable},
+ "observation": output[:400] + f"\nExit code {completed.returncode}",
+ }
+ ]
+ host_result = host_checks.check_script_execution(calls, script)
+ template_result = TEMPLATE.check_script_execution(calls, script)
+ if (host_result["score"], host_result["reason"], host_result["passed"]) != (
+ template_result["score"],
+ template_result["reason"],
+ template_result["passed"],
+ ):
+ return ("divergence", f"host={host_result} template={template_result}", None, ran, calls)
+
+ score = host_result["score"]
+ if score == 1.0 and not ran:
+ if completed.returncode != 0 and ("&&" in command or "||" in command):
+ # A chain short-circuited before reaching the invocation. The
+ # walk is static and a tool call does not carry which link
+ # failed, so this is the limitation the PR declares, reported
+ # here rather than hidden.
+ return (
+ "short-circuited chain (declared limitation)",
+ f"exit {completed.returncode}",
+ score,
+ ran,
+ calls,
+ )
+ return ("false positive", host_result["reason"], score, ran, calls)
+ if ran and score == 0.0:
+ return ("false negative", host_result["reason"], score, ran, calls)
+ if ran and score != 1.0:
+ return ("partial on a real run", f"{score}", score, ran, calls)
+ if (
+ score == 0.75
+ and script.rsplit("/", 1)[-1] not in command
+ and "could not be classified" in host_result["reason"]
+ ):
+ return ("partial without a reference", host_result["reason"], score, ran, calls)
+ return ("agreed", f"{score}", score, ran, calls)
+ finally:
+ shutil.rmtree(directory, ignore_errors=True)
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--fuzz", type=int, default=0, help="also generate and execute N random commands")
+ parser.add_argument(
+ "--compose", type=int, default=0, help="also generate and execute N commands that nest binding scopes"
+ )
+ parser.add_argument("--seed", type=int, default=20260923, help="seed for --fuzz")
+ parser.add_argument("--verbose", action="store_true", help="print every command and its outcome")
+ parser.add_argument("--baseline", help="git ref of an earlier checker to compare every score against")
+ arguments = parser.parse_args()
+
+ commands = curated()
+ if arguments.fuzz:
+ rng = random.Random(arguments.seed)
+ seen = {command for command, _ in commands}
+ generated: list[tuple[str, str]] = []
+ while len(generated) < arguments.fuzz:
+ command, script = _generate(rng)
+ if command not in seen:
+ seen.add(command)
+ generated.append((command, script))
+ commands += generated
+ if arguments.compose:
+ rng = random.Random(arguments.seed + 1)
+ seen = {command for command, _ in commands}
+ composed: list[tuple[str, str]] = []
+ while len(composed) < arguments.compose:
+ command, script = _compose(rng)
+ if command not in seen:
+ seen.add(command)
+ composed.append((command, script))
+ commands += composed
+
+ baseline = _load_baseline(arguments.baseline) if arguments.baseline else None
+ buckets: dict[str, list[tuple[str, str]]] = {}
+ moves: dict[str, list[tuple[str, float | str, float]]] = {}
+ runs: list[tuple[str, str, str | None]] = [(command, script, None) for command, script in commands]
+ runs += native_shell_cases()
+ commands = [(command, script) for command, script, _ in runs]
+ for command, script, shell in runs:
+ outcome, detail, score, ran, calls = _run_one(command, script, shell)
+ if shell is not None:
+ command = f"[{shell}] {command}"
+ buckets.setdefault(outcome, []).append((command, detail))
+ if arguments.verbose:
+ print(f"{outcome:22} {command!r} {detail}")
+ if baseline is not None and score is not None:
+ try:
+ before = baseline[0].check_script_execution(calls, script)["score"]
+ except Exception as error: # an earlier checker may raise where this one scores
+ moves.setdefault("baseline raised, now scored", []).append((command, type(error).__name__, score))
+ continue
+ if before != score:
+ if ran and score < before:
+ kind = "score lowered on a command that ran (review each)"
+ elif not ran and score > before:
+ kind = "score raised on a command that did not run (review each)"
+ elif not ran and before == 1.0:
+ kind = "false positive repaired"
+ elif ran and before == 0.0:
+ kind = "false negative repaired"
+ else:
+ kind = "other move"
+ moves.setdefault(kind, []).append((command, before, score))
+
+ executed = len(commands) - len(buckets.get("inconclusive", []))
+ print(f"\ncommands executed: {executed} of {len(commands)}")
+ defects = 0
+ short_circuit = buckets.get("short-circuited chain (declared limitation)", [])
+ for outcome, label in (
+ ("false positive", "false positives"),
+ ("false negative", "false negatives"),
+ ("divergence", "divergences"),
+ ("partial without a reference", "partials without a reference"),
+ ):
+ entries = buckets.get(outcome, [])
+ defects += len(entries)
+ print(f" {label}: {len(entries)}")
+ for command, detail in entries[:25]:
+ print(f" {command!r} -> {detail}")
+ print(f" short-circuited chains (declared limitation, credited but did not run): {len(short_circuit)}")
+ for command, detail in short_circuit[:15]:
+ print(f" {command!r} -> {detail}")
+ partial = buckets.get("partial on a real run", [])
+ print(f" partial on a real run (not a defect): {len(partial)}")
+ for command, detail in partial[:40]:
+ print(f" {command!r} -> {detail}")
+ inconclusive = buckets.get("inconclusive", [])
+ if inconclusive:
+ print(f" not runnable on this machine: {len(inconclusive)}")
+ for command, detail in inconclusive[:15]:
+ print(f" {command!r} :: {detail}")
+ if baseline is not None:
+ total = sum(len(entries) for entries in moves.values())
+ print(f"\nscores that moved against {arguments.baseline}: {total}")
+ for kind in (
+ "score lowered on a command that ran (review each)",
+ "score raised on a command that did not run (review each)",
+ "false positive repaired",
+ "false negative repaired",
+ "baseline raised, now scored",
+ "other move",
+ ):
+ entries = moves.get(kind, [])
+ print(f" {kind}: {len(entries)}")
+ for command, before, after in entries[:40]:
+ print(f" {command!r}: {before} -> {after}")
+ return 1 if defects else 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/src/skillevaluator/tier3/eval_core/checks.py b/src/skillevaluator/tier3/eval_core/checks.py
index ee439659..2c77906b 100644
--- a/src/skillevaluator/tier3/eval_core/checks.py
+++ b/src/skillevaluator/tier3/eval_core/checks.py
@@ -17,6 +17,7 @@ from __future__ import annotations
import re
import shlex
from fnmatch import fnmatchcase
+from functools import lru_cache
from typing import Any
from skillevaluator.tier3.eval_core.codex_tool_call_normalizer import (
@@ -269,8 +270,191 @@ _OUTPUT_REDIRECTS = {">", ">>", ">|", "&>", "&>>"}
_HEREDOC_REDIRECTS = {"<<", "<<<"}
+_ATTACHED_DESCRIPTOR_RE = re.compile(r"(\d+)(>>|>\||>&|<&|<>|>|<)(?![<])")
+
+
+# Words the walk reads as shell syntax when they stand unquoted. Quoted or
+# escaped, each is an ordinary word: `'done'` is a command named done, and
+# `printf "("` prints a parenthesis. The tokenizer drops quotes, so such a
+# word is prefixed with a private-use character before tokenizing and the
+# prefix is removed wherever a word is read as a value.
+_QUOTED_SYNTAX_MARK = "\ue000"
+_SYNTAX_WORDS = frozenset(
+ {
+ "for",
+ "select",
+ "while",
+ "until",
+ "if",
+ "then",
+ "else",
+ "elif",
+ "fi",
+ "do",
+ "done",
+ "case",
+ "esac",
+ "in",
+ "!",
+ "{",
+ "}",
+ "(",
+ ")",
+ "--",
+ ";",
+ ";;",
+ "&",
+ "&&",
+ "|",
+ "||",
+ "<",
+ ">",
+ ">>",
+ ">|",
+ "<<",
+ "<<<",
+ "<&",
+ ">&",
+ "&>",
+ "&>>",
+ "<>",
+ }
+)
+_SHELL_METACHARS = ";&|()<>"
+# A quoted word made of metacharacters (``';|'``, ``'2>'``, ``'<<-'``) would be
+# split into operators after tokenizing; it is marked like a listed word.
+_SYNTAX_SHAPE_RE = re.compile(r"\d*[;&|<>(){}]+-?|!|--")
+
+
+def _mark_quoted_syntax(text: str) -> str:
+ """Prefix each quoted or escaped word that would otherwise read as syntax.
+
+ Words are delimited as the shell delimits them: at unquoted whitespace
+ and unquoted metacharacters. A word that contains a quote or an escape
+ and whose unquoted text is in ``_SYNTAX_WORDS`` or shaped like syntax
+ gets the mark; every other character is copied through unchanged,
+ quotes included, so the tokenizer still sees the original quoting. A
+ mark character already present in the text is doubled first, so the
+ reader can tell a mark (one, at the start of a word) from data (always
+ two), and a file whose name carries that character keeps its identity.
+ """
+ text = text.replace(_QUOTED_SYNTAX_MARK, _QUOTED_SYNTAX_MARK * 2)
+ out = []
+ raw = []
+ plain = []
+ quoted = False
+ quote = None
+ index = 0
+
+ def flush() -> None:
+ nonlocal quoted
+ if raw:
+ if quoted and ("".join(plain) in _SYNTAX_WORDS or _SYNTAX_SHAPE_RE.fullmatch("".join(plain))):
+ out.append(_QUOTED_SYNTAX_MARK)
+ out.extend(raw)
+ raw.clear()
+ plain.clear()
+ quoted = False
+
+ while index < len(text):
+ char = text[index]
+ if quote:
+ raw.append(char)
+ if char == quote:
+ quote = None
+ elif char == "\\" and quote == '"' and index + 1 < len(text) and text[index + 1] in '"\\':
+ raw.append(text[index + 1])
+ plain.append(text[index + 1])
+ index += 1
+ else:
+ plain.append(char)
+ index += 1
+ continue
+ if char in "'\"":
+ quote = char
+ quoted = True
+ raw.append(char)
+ index += 1
+ continue
+ if char == "\\" and index + 1 < len(text):
+ quoted = True
+ raw.append(char)
+ raw.append(text[index + 1])
+ plain.append(text[index + 1])
+ index += 2
+ continue
+ if char.isspace() or char in _SHELL_METACHARS:
+ flush()
+ out.append(char)
+ index += 1
+ continue
+ raw.append(char)
+ plain.append(char)
+ index += 1
+ flush()
+ return "".join(out)
+
+
+def _keep_attached_descriptors(text: str) -> str:
+ """Quote ``2>`` so the lexer keeps the descriptor with its operator.
+
+ The lexer splits every ``>`` and ``<`` into its own token, so ``2>/dev/null``
+ and ``2 > numeric.out`` arrive as the same tokens. In the shell they are
+ not the same command: the first sends standard error away, the second
+ passes ``2`` as an argument and redirects standard output. Only a digit
+ run written flush against its operator is a descriptor, so that pairing
+ is fixed here, before the text is tokenized, by quoting it. Text inside
+ quotes is data and is left alone.
+ """
+ out = []
+ quote = None
+ index = 0
+ at_word_start = True
+ while index < len(text):
+ char = text[index]
+ if quote:
+ out.append(char)
+ if char == "\\" and quote == '"' and index + 1 < len(text):
+ out.append(text[index + 1])
+ index += 1
+ elif char == quote:
+ quote = None
+ index += 1
+ continue
+ if char == "\\" and index + 1 < len(text):
+ out.append(text[index : index + 2])
+ index += 2
+ at_word_start = False
+ continue
+ if char in "'\"":
+ quote = char
+ out.append(char)
+ index += 1
+ at_word_start = False
+ continue
+ if at_word_start and char.isdigit():
+ match = _ATTACHED_DESCRIPTOR_RE.match(text, index)
+ if match:
+ out.append("'" + match.group(1) + match.group(2) + "'")
+ index = match.end()
+ # An operand written flush against the operator
+ # (``0<run.py``) would concatenate onto the quoted
+ # operator as one word, hiding the filename from the
+ # walk. A space splits it into its own token, as the
+ # spaced form already tokenizes.
+ if index < len(text) and not text[index].isspace() and text[index] not in ";&|(){}<>":
+ out.append(" ")
+ at_word_start = False
+ continue
+ out.append(char)
+ at_word_start = char.isspace() or char in ";&|(){}"
+ index += 1
+ return "".join(out)
+
+
def _shell_tokens(cmd: Any) -> list[str]:
normalized = str(cmd).replace("\r\n", "\n").replace("\r", "\n").replace("\n", " ; ")
+ normalized = _keep_attached_descriptors(_mark_quoted_syntax(normalized))
lexer = shlex.shlex(normalized, posix=True, punctuation_chars=True)
lexer.whitespace_split = True
lexer.commenters = ""
@@ -290,7 +474,10 @@ def _skill_md_arg(arg: str, assignments: dict[str, str]) -> bool:
def _resolved_shell_arg(arg: str, assignments: dict[str, str]) -> str:
- value = str(arg).lstrip("<>")
+ value = str(arg)
+ if value.startswith(_QUOTED_SYNTAX_MARK) and not value.startswith(_QUOTED_SYNTAX_MARK * 2):
+ value = value[1:]
+ value = value.replace(_QUOTED_SYNTAX_MARK * 2, _QUOTED_SYNTAX_MARK).lstrip("<>")
for _ in range(2):
resolved = _SHELL_VARIABLE_RE.sub(
lambda match: assignments.get(match.group(1) or match.group(2), match.group(0)),
@@ -304,11 +491,15 @@ def _resolved_shell_arg(arg: str, assignments: dict[str, str]) -> str:
def _is_output_redirect(token: str) -> bool:
token = str(token)
+ if token.startswith(_QUOTED_SYNTAX_MARK):
+ return False
return token in _OUTPUT_REDIRECTS or any(token.endswith(op) for op in _OUTPUT_REDIRECTS)
def _is_heredoc_redirect(token: str) -> bool:
token = str(token)
+ if token.startswith(_QUOTED_SYNTAX_MARK):
+ return False
return token in _HEREDOC_REDIRECTS or any(token.endswith(op) for op in _HEREDOC_REDIRECTS)
@@ -366,8 +557,8 @@ def _cmd_reads_skill_md(cmd) -> bool:
return False
-_SCRIPT_INTERPRETERS = {"bash", "dash", "node", "perl", "python", "python3", "ruby", "sh", "zsh"}
-_SHELL_COMMAND_INTERPRETERS = {"bash", "dash", "sh", "zsh"}
+_SCRIPT_INTERPRETERS = {"ash", "bash", "dash", "ksh", "mksh", "node", "perl", "python", "python3", "ruby", "sh", "zsh"}
+_SHELL_COMMAND_INTERPRETERS = {"ash", "bash", "dash", "ksh", "mksh", "sh", "zsh"}
_INERT_SHELL_PRODUCERS = {"echo", "printf"}
_MAX_SHELL_REFERENCE_CHARS = 32_768
_MAX_SHELL_REFERENCE_TOKENS = 256
@@ -706,19 +897,41 @@ def _unwrap_shell_command(
return None
+def _shell_c_positional(command: list[str], cmd_idx: int, assignments: dict[str, str]) -> list[str]:
+ """The operands after the ``-c`` payload: ``bash -c '...' a b`` gives the
+ payload positional parameters, so ``for f; do`` inside it iterates them
+ and ``$1`` names one of them.
+ """
+ operands = _interpreter_operands(command, cmd_idx, assignments)
+ for index in range(len(operands) - 1):
+ option = operands[index].strip("\"'")
+ if option.startswith("-") and "c" in option[1:]:
+ payload_index = index + 1
+ if operands[payload_index].strip("\"'") == "--":
+ payload_index += 1
+ return operands[payload_index + 1 :]
+ return []
+
+
def _shell_c_payload(command: list[str], cmd_idx: int, assignments: dict[str, str]) -> str | None:
- """Extract inline command string from -c shell invocation."""
- for index in range(cmd_idx + 1, len(command) - 1):
- option = _resolved_shell_arg(command[index], assignments).strip("\"'")
+ """Extract inline command string from -c shell invocation.
+
+ The option and its payload are read from the interpreter's own operands,
+ with every redirection removed, so ``bash -c 2>/dev/null './run.sh'`` has
+ the payload ``./run.sh`` and not the redirection standing between them.
+ """
+ operands = _interpreter_operands(command, cmd_idx, assignments)
+ for index in range(len(operands) - 1):
+ option = operands[index].strip("\"'")
if option.startswith("-") and "c" in option[1:]:
payload_index = index + 1
- if command[payload_index].strip("\"'") == "--":
+ if operands[payload_index].strip("\"'") == "--":
payload_index += 1
- if payload_index < len(command):
- raw = _resolved_shell_arg(command[payload_index], assignments)
+ if payload_index < len(operands):
+ raw = operands[payload_index]
if (raw.startswith("'") and raw.endswith("'")) or (raw.startswith('"') and raw.endswith('"')):
raw = raw[1:-1]
- return raw
+ return raw.replace(_QUOTED_NEWLINE, "\n")
return None
@@ -1750,35 +1963,2974 @@ def check_activation(
}
-def check_script_execution(
- tool_calls: list[dict[str, Any]],
- expected_script: str | None,
-) -> dict[str, Any]:
- """Check whether the agent executed the expected script."""
- if not expected_script:
- return {"passed": True, "score": 1.0, "reason": "No specific script expected"}
+# Interpreters that run a script named on their command line, with the options
+# that decide where that script comes from. The first set is options meaning the
+# interpreter runs inline code or a module, so no script path runs at all. The
+# second is options that consume a value, either attached ("-Wignore") or as the
+# following token ("-I lib").
+# Interpreter option grammars. Each interpreter names only the options whose
+# effect on the script argument is certain. The boolean and value sets were
+# derived by running every option against a script that records whether it
+# executed, rather than from documentation, which is why perl's -M and -i and
+# python's -Q are absent: their argument is conditionally attached, so no
+# single rule resolves them. An option that is not listed for the interpreter
+# in hand, including one belonging to a different interpreter, makes the
+# command undecidable rather than credited.
+_INTERPRETER_GRAMMARS: dict[str, dict[str, frozenset[str]]] = {
+ "python": {
+ "boolean": frozenset(
+ {
+ "-b",
+ "-B",
+ "-d",
+ "-E",
+ "-i",
+ "-I",
+ "-O",
+ "-OO",
+ "-P",
+ "-q",
+ "-s",
+ "-S",
+ "-t",
+ "-u",
+ "-v",
+ "-W0",
+ "-W1",
+ "-W2",
+ }
+ ),
+ "value": frozenset({"-W", "-X"}),
+ # -m imports a module, and importing runs it, so like -c this is code
+ # whose effect the walk cannot read.
+ "code": frozenset({"-c", "-m"}),
+ "terminal": frozenset({"-h", "-?", "--help", "-V", "--version"}),
+ },
+ "perl": {
+ "boolean": frozenset({"-C", "-f", "-i", "-l", "-s", "-t", "-T", "-U", "-w", "-W", "-W0", "-X"}),
+ "value": frozenset({"-I"}),
+ "code": frozenset({"-e", "-E"}),
+ "terminal": frozenset({"-c", "-h", "-v", "-V", "--help", "--version"}),
+ },
+ "ruby": {
+ "boolean": frozenset(
+ {"-a", "-d", "-i", "-l", "-s", "-S", "-U", "-v", "-w", "-W", "-W0", "-W1", "-W2", "--verbose"}
+ ),
+ "value": frozenset({"-E", "-I"}),
+ "code": frozenset({"-e"}),
+ "terminal": frozenset({"-c", "-h", "--help", "--version"}),
+ },
+ "node": {
+ "boolean": frozenset({"-i", "--interactive", "--no-warnings", "--trace-warnings"}),
+ "value": frozenset({"-C", "-r", "--conditions", "--import", "--loader", "--require"}),
+ "code": frozenset({"-e", "--eval", "-p", "--print"}),
+ "terminal": frozenset({"-c", "--check", "-h", "--help", "-v", "--version"}),
+ },
+ "bash": {
+ "boolean": frozenset(
+ {
+ "-a",
+ "-b",
+ "-B",
+ "-C",
+ "-e",
+ "-E",
+ "-f",
+ "-h",
+ "-H",
+ "-i",
+ "-l",
+ "-m",
+ "-p",
+ "-P",
+ "-t",
+ "-T",
+ "-u",
+ "-v",
+ "-x",
+ "--verbose",
+ }
+ ),
+ "value": frozenset(),
+ "code": frozenset({"-c"}),
+ "terminal": frozenset({"-n", "--help", "--version"}),
+ },
+ "sh": {
+ "boolean": frozenset({"-a", "-b", "-C", "-e", "-E", "-f", "-i", "-I", "-l", "-m", "-p", "-u", "-v", "-x"}),
+ "value": frozenset(),
+ "code": frozenset({"-c"}),
+ "terminal": frozenset({"-n"}),
+ },
+}
+_INTERPRETER_GRAMMARS["dash"] = _INTERPRETER_GRAMMARS["sh"]
+# ksh, mksh and ash take the POSIX sh options this grammar lists; an option
+# outside it leaves the command unresolved, as for sh.
+_INTERPRETER_GRAMMARS["ksh"] = _INTERPRETER_GRAMMARS["sh"]
+_INTERPRETER_GRAMMARS["mksh"] = _INTERPRETER_GRAMMARS["sh"]
+_INTERPRETER_GRAMMARS["ash"] = _INTERPRETER_GRAMMARS["sh"]
+_INTERPRETER_GRAMMARS["zsh"] = _INTERPRETER_GRAMMARS["bash"]
+_SOURCING_COMMANDS = frozenset({".", "source"})
+_VERSION_SUFFIX_RE = re.compile(r"\d+(?:\.\d+)*$")
+# Wrapper grammars, derived the same way as the interpreter ones by running
+# each option and checking whether the wrapped command still executed. A
+# wrapper option that is not listed makes the command undecidable rather than
+# assuming the wrapped command runs, and a help or version option means the
+# wrapper printed and exited without running anything.
+_WRAPPER_GRAMMARS: dict[str, dict[str, frozenset[str]]] = {
+ "env": {
+ "boolean": frozenset({"-i", "-v", "--ignore-environment", "--debug"}),
+ "value": frozenset({"-u", "--unset", "-C", "--chdir"}),
+ "terminal": frozenset({"--help", "--version"}),
+ },
+ "timeout": {
+ "boolean": frozenset({"--preserve-status", "--foreground", "-v", "--verbose"}),
+ "value": frozenset({"-s", "--signal", "-k", "--kill-after"}),
+ "terminal": frozenset({"--help", "--version"}),
+ },
+ "nice": {
+ "boolean": frozenset(),
+ "value": frozenset({"-n", "--adjustment"}),
+ "terminal": frozenset({"--help", "--version"}),
+ },
+ "nohup": {"boolean": frozenset(), "value": frozenset(), "terminal": frozenset({"--help", "--version"})},
+ "stdbuf": {
+ "boolean": frozenset(),
+ "value": frozenset({"-i", "-o", "-e", "--input", "--output", "--error"}),
+ "terminal": frozenset({"--help", "--version"}),
+ },
+ "setsid": {
+ "boolean": frozenset({"-f", "-w", "--fork", "--wait"}),
+ "value": frozenset(),
+ "terminal": frozenset({"--help", "--version", "-V", "-h"}),
+ },
+ "time": {"boolean": frozenset({"-p"}), "value": frozenset(), "terminal": frozenset({"--help", "--version"})},
+ "sudo": {
+ "boolean": frozenset({"-E", "-H", "-n", "-S", "-b"}),
+ "value": frozenset({"-u", "--user", "-g", "--group"}),
+ "terminal": frozenset({"--help", "--version", "-h", "-V"}),
+ },
+ "doas": {"boolean": frozenset({"-n", "-s"}), "value": frozenset({"-u"}), "terminal": frozenset({"-h", "--help"})},
+ "xvfb-run": {
+ "boolean": frozenset({"-a", "--auto-servernum"}),
+ "value": frozenset({"-s", "--server-args", "-n", "--server-num"}),
+ "terminal": frozenset({"-h", "--help", "--version"}),
+ },
+ # Whether these run anything at all depends on their standard input, which
+ # a command's own text never carries: `printf "" | xargs -r python run.py`
+ # runs nothing, and `xargs -p` runs nothing with no terminal to confirm at.
+ # So they resolve only to "printed and exited", never to the command after
+ # them.
+ "xargs": {"boolean": frozenset(), "value": frozenset(), "terminal": frozenset({"--help", "--version"})},
+ "parallel": {"boolean": frozenset(), "value": frozenset(), "terminal": frozenset({"--help", "--version"})},
+}
+_STDIN_DEPENDENT_WRAPPERS = frozenset({"xargs", "parallel"})
+for _runner in ("uv", "uvx", "poetry", "pipenv", "pdm", "hatch", "rye"):
+ _WRAPPER_GRAMMARS[_runner] = {
+ "boolean": frozenset(),
+ "value": frozenset(),
+ "terminal": frozenset({"-h", "--help", "-V", "--version"}),
+ }
+_TRANSPARENT_COMMAND_PREFIXES = frozenset(
+ {"timeout", "nohup", "nice", "stdbuf", "time", "sudo", "doas", "xvfb-run", "setsid"}
+)
+_RUNNER_COMMAND_PREFIXES = frozenset({"uv", "uvx", "poetry", "pipenv", "pdm", "hatch", "rye"})
+_DURATION_ARG_RE = re.compile(r"^\d+(?:\.\d+)?[smhd]?$")
+_NEGATIVE_NUMBER_RE = re.compile(r"^-\d+$")
+_WRAPPER_OK = "ok"
+_WRAPPER_NONE = "none"
+_WRAPPER_UNKNOWN = "unknown"
+# Shell grouping keywords, which introduce a command rather than being one,
+# and the end-of-options marker, which stands before one.
+_GROUPING_TOKENS = frozenset({"{", "(", "!", "}", ")", "--"})
+# Text the shell resolves at run time, which a static walk cannot compare.
+_DYNAMIC_ARGUMENT_RE = re.compile(r"\$\(|\$\{|`|\{\}")
+_UNRESOLVED_ARG_RE = _DYNAMIC_ARGUMENT_RE
+# Commands that build their arguments from standard input, so a path piped to
+# them never appears as an argument this walk can read.
+_STDIN_ARGV_RE = re.compile(r"(?:^|[\s|;&])(?:xargs|parallel)(?:\s|$)")
+
+_SCRIPT = "script"
+_NO_SCRIPT = "none"
+_INLINE_CODE = "code"
+_UNDECIDABLE = "unknown"
+# Shell builtins that run text this walk does not read as commands.
+_OPAQUE_SHELL_BUILTINS = frozenset({"eval"})
+_INPUT_REDIRECTS = frozenset({"<", "0<"})
+# Redirection operators the tokenizer splits out on their own, each followed
+# by an operand that belongs to the shell rather than to the command's argv.
+# The output forms are in _OUTPUT_REDIRECTS; these are the input forms and the
+# descriptor-duplicating `>&`.
+_INPUT_REDIRECT_OPERATORS = frozenset({"<", "<&", "<>", ">&"})
+# A redirection carried as one token with its operand attached (`</dev/null`,
+# `2>&1`), which only happens when the tokenizer fell back to a plain split.
+_ATTACHED_REDIRECT_RE = re.compile(r"^\d*(?:<>|<&|>&|>>|>\||<|>)[^<>&|\s]")
+# A descriptor with its operator, the operand in the next token: ``2> err.txt``.
+_DESCRIPTOR_OPERATOR_RE = re.compile(r"^\d+(?:<>|<&|>&|>>|>\||<|>)$")
+# Reserved words that stand before a command rather than being one: what
+# follows `then` or `do` is the command that runs.
+_COMMAND_INTRODUCING_WORDS = frozenset({"if", "then", "else", "elif", "while", "until", "do"})
+# Loop headers name the values a variable will take and run nothing themselves;
+# what the body does with that variable is not something this text settles.
+_LOOP_HEADER_WORDS = frozenset({"for", "select"})
+# The value of a variable a header rebound to the positional parameters,
+# when the text does not say what they are: a later read of it is
+# unresolved rather than a settled miss.
+_UNSETTLED_VALUE = "\ue001"
+# Text that sets or shifts the positional parameters, so ``for f; do`` may
+# iterate something even where the caller passed none.
+_POSITIONAL_SET_RE = re.compile(r"(?:^|[;&|(\s])(?:shift\b|set\s+(?:--|[^-\s]))")
+# Text that reads the positional parameters or ``$0``, the only ways an
+# operand after a ``-c`` payload reaches the payload.
+_READS_POSITIONAL_RE = re.compile(r"\$[@*1-9]|\$\{[@*1-9]|\bshift\b|\bfor\s+[A-Za-z_]\w*\s*(?:;|\bdo\b)")
+_READS_ARGV0_RE = re.compile(r"\$0\b|\$\{0[}:]")
+# Control syntax this walk does not model, so a script inside it is unresolved.
+_UNMODELLED_CONTROL_WORDS = frozenset({"case"})
+# A ``$`` the shell reads literally: inside single quotes, or escaped. The
+# walk marks it before tokenizing so that no binding is substituted there:
+# ``bash -c 'python3 "$f"'`` hands the child the text ``$f``, which the child
+# expands from its own environment, not from this shell's unexported ``f``.
+_LITERAL_DOLLAR = "\ue003"
+
+
+# The shared tokenizer turns every newline into a separator, quoted ones
+# included, so a ``-c`` payload lost the lines its heredocs need. Within this
+# walk a newline inside quotes is kept as this mark, spaced as the separator
+# was so that words split as before, and restored where the quoted text is
+# read again as commands.
+_QUOTED_NEWLINE = " \ue009 "
+
+
+def _mark_quoted_newlines(text: str) -> str:
+ """Replace each newline inside quotes with ``_QUOTED_NEWLINE``."""
+ out: list[str] = []
+ quote: str | None = None
+ index = 0
+ while index < len(text):
+ char = text[index]
+ if char == "\\" and quote != "'" and index + 1 < len(text):
+ out.append(char + text[index + 1])
+ index += 2
+ continue
+ if char in {"'", '"'} and quote in {None, char}:
+ quote = None if quote == char else char
+ out.append(_QUOTED_NEWLINE if char == "\n" and quote else char)
+ index += 1
+ return "".join(out)
- exec_calls = [tc for tc in tool_calls if _is_execution_action(str(tc["action"]))]
- for call in exec_calls:
- cmd = _command_text(call)
- if expected_script in cmd:
- return {"passed": True, "score": 1.0, "reason": f"Executed {expected_script}"}
- if _has_unsupported_native_codex_call(tool_calls):
- return _unsupported_native_codex_result(
- "Script execution could not be evaluated because a native Codex exec wrapper was unsupported"
+def _mark_literal_dollars(text: str) -> str:
+ """Replace each ``$`` the shell would not expand with ``_LITERAL_DOLLAR``."""
+ out: list[str] = []
+ quote: str | None = None
+ index = 0
+ while index < len(text):
+ char = text[index]
+ following = text[index + 1] if index + 1 < len(text) else ""
+ if quote == "'":
+ out.append(_LITERAL_DOLLAR if char == "$" else char)
+ if char == "'":
+ quote = None
+ elif char == "\\" and following == "$":
+ out.append(_LITERAL_DOLLAR)
+ index += 1
+ elif char == "\\" and following:
+ out.append(char + following)
+ index += 1
+ else:
+ if char == '"':
+ quote = None if quote == '"' else '"'
+ elif char == "'" and quote is None:
+ quote = "'"
+ out.append(char)
+ index += 1
+ return "".join(out)
+
+
+# A name exported to the commands this shell starts, kept in the same
+# bindings as the values so that a scope copy carries it: a ``-c`` payload's
+# shell sees the exported names and the command's own ``NAME=value`` prefix,
+# and nothing else this shell bound.
+_EXPORTED_MARK = "\ue004"
+# Whether ``set -a`` (allexport) is on, kept with the bindings so a subshell's
+# copy carries it and drops it on exit. A name bound while it is on is
+# exported, and stays exported after ``set +a``; one bound before is not.
+_ALLEXPORT_MARK = "\ue005"
+# A name made read-only, kept in the same bindings as the values so that a
+# scope copy carries it. A later assignment to it fails, and whether the shell
+# goes on after that is not the same in every shell, so its value is unsettled.
+_READONLY_MARK = "\ue002"
+# Set in the bindings once an assignment to a read-only name has failed where
+# the shell stops. Where it stops, nothing after the failure in that shell is
+# credited as run: a script named there is unresolved. A subshell stops
+# alone, so the mark is dropped with its copy of the bindings.
+_ASSIGNMENT_REFUSED = "\ue006"
+# The attributes a declaration gives a name that change what a later
+# assignment stores or what a child inherits, kept as their letters until
+# ``unset`` or a ``+`` option removes them: ``-i`` (integer), ``-u`` and
+# ``-c`` (case, which also decides whether a path matches on a case-blind
+# file system) and ``-n`` (a reference to another name, below) leave a
+# later value unsettled; ``-l`` lowercases it; ``-a`` and ``-A`` keep ``$f``
+# as the value assigned but are never exported to a child.
+_ATTRIBUTE_MARK = "\ue007"
+_ATTRIBUTE_LETTERS = frozenset("iuncalA")
+_READONLY_ATTRIBUTES = {"readonly": frozenset("aA")}
+# The name a ``-n`` name refers to: the value ``declare -n f=g`` gives, or
+# the name f held where none is given; empty where that names nothing yet,
+# when bash and ksh take the next value assigned as the name, and unsettled
+# where the text does not settle it. Assigning, exporting, making read-only
+# or unsetting f acts on that name instead, in bash, ksh and mksh
+# (measured), and it may be a name the text has not bound.
+_REFERENCE_MARK = "\ue008"
+# ``nameref`` is ``typeset -n`` in ksh, also after ``command``, and an alias
+# for it in mksh, where only the first word of a command is an alias.
+# ``unset -n f`` unsets the reference itself in bash and ksh; mksh, dash and
+# zsh reject the option.
+_NAMEREF_SHELLS = frozenset({"ksh", "mksh"})
+_UNSET_REFERENCE_SHELLS = frozenset({"bash", "bash-posix", "ksh"})
+# Where ``-n`` is accepted. A name that cannot be referred to
+# (``declare -n f=run.py``) fails the declaration for that name, and ksh
+# stops; with no name at all, bash and ksh wait for the next value assigned
+# and mksh fails ("empty nameref target") and goes on (measured).
+_REFERENCE_SHELLS = frozenset({"bash", "bash-posix", "ksh", "mksh"})
+_EMPTY_REFERENCE_SHELLS = frozenset({"bash", "bash-posix", "ksh"})
+_BAD_REFERENCE_STOPS = frozenset({"ksh"})
+# A declaration that gives no value leaves the value as it was in bash, with
+# any option (``declare -u f`` upper-cases only a later assignment, and one
+# bash rejects changes nothing), except ``-n``; zsh, ksh and mksh convert it
+# at once (measured). Taking an attribute away (``+i``) leaves it in all.
+_DECLARATION_KEEPS_VALUE = frozenset({"bash", "bash-posix"})
+# The shells that go on after that failure, by the form of the assignment,
+# measured with ``readonly f=run.py; <form>; echo same-line``. A bare
+# assignment stops every shell (bash abandons the rest of its line).
+# An assignment to a name given ``-i`` evaluates the value, and one that is
+# not a number there (``run.py``) is an error. The shells that go on after
+# it, by form, measured with ``typeset -i f; <form>; echo same``: for a bare
+# assignment bash exits, as every shell with ``typeset`` does, so a value
+# that is not a plain number is read as stopping the shell except here.
+_INTEGER_ERROR_GOES_ON = {
+ "prefix": frozenset({"bash", "bash-posix", "ksh"}),
+ "prefix-special": frozenset({"bash", "bash-posix"}),
+ "typeset": frozenset({"ksh"}),
+ "read": frozenset({"ksh", "mksh"}),
+}
+_INTEGER_VALUE_RE = re.compile(r"[-+]?[0-9]+")
+
+
+def _integer_may_fail(value: str, scope: dict[str, str], depth: int = 0) -> bool:
+ """Whether evaluating ``value`` for a ``-i`` name may be an error: not for a
+ number, nor for a name that is unset, empty or holds one (it evaluates to
+ that); for anything else it may be (``run.py``), and is read as such."""
+ value = value.strip("\"'")
+ if _INTEGER_VALUE_RE.fullmatch(value):
+ return False
+ if not _SHELL_NAME_RE.fullmatch(value) or depth >= _MAX_SHELL_REFERENCE_DEPTH:
+ return True
+ held = scope.get(value)
+ return bool(held) and _integer_may_fail(held, scope, depth + 1)
+
+
+# A numeric attribute (``-i``, and ``-F`` or ``-E`` for a float) given to a
+# name whose value is not a number stops zsh and ksh, whether the value is
+# given with it or held (measured with ``typeset -F f=run.py; python3 run.py``
+# and ``f=run.py; typeset -i f; python3 run.py``); a decimal, or nothing, is
+# a number there.
+_NUMERIC_ATTRIBUTE_STOPS = frozenset({"zsh", "ksh"})
+_NUMERIC_ATTRIBUTES = frozenset("iFE")
+_DECIMAL_VALUE_RE = re.compile(r"[-+]?(?:(?:[0-9]+(?:\.[0-9]*)?|\.[0-9]+)(?:[eE][-+]?[0-9]+)?|0[xX][0-9a-fA-F]+)")
+# ``base#digits``: zsh takes a base from 2 to 36 and digits below it, and ksh
+# at least that (measured: ``2#9``, ``1#1`` and ``99#1`` stop zsh).
+_BASED_VALUE_RE = re.compile(r"[-+]?([0-9]+)#([0-9a-zA-Z]+)")
+# A width given with ``-L``, ``-R`` or ``-Z`` (``typeset -L3 f``) cuts or pads
+# the value a name holds, at once, in zsh, ksh and mksh (measured with
+# ``f=run.py; typeset -L3 f; python3 "$f"``, which runs ``run``).
+_JUSTIFYING_SHELLS = frozenset({"zsh", "ksh", "mksh"})
+
+
+def _number_may_fail(value: str, scope: dict[str, str]) -> bool:
+ """Whether a numeric attribute may fail on ``value``: as ``_integer_may_fail``,
+ where a decimal, ``0x10`` and ``16#ff`` are numbers too and nothing (an unset
+ or empty name) is 0. An expression (``g+1``) is not evaluated, and may fail."""
+ value = value.strip("\"'")
+ if not value or _DECIMAL_VALUE_RE.fullmatch(value):
+ return False
+ based = _BASED_VALUE_RE.fullmatch(value)
+ if based:
+ # zsh and ksh read the base as a decimal and drop its leading zeros
+ # (``016#ff`` is 255 in both, measured). A base of more than two
+ # digits is past 36 and is not converted: int() refuses a string of
+ # more than 4,300 digits.
+ base_digits = based.group(1).lstrip("0")
+ if len(base_digits) > 2:
+ return True
+ base = int(base_digits or "0")
+ return not (2 <= base <= 36 and all(int(digit, 36) < base for digit in based.group(2)))
+ return _integer_may_fail(value, scope)
+
+
+# A special builtin given an option the shell rejects (``export -n`` in dash,
+# ``unset -n`` in mksh) stops a POSIX shell, unless ``command`` runs it; bash
+# and zsh report it and go on (measured).
+_SPECIAL_BUILTIN_ERROR_STOPS = frozenset({"bash-posix", "dash", "ksh", "mksh", "ash"})
+_REFUSED_ASSIGNMENT_GOES_ON = {
+ "assignment": frozenset(),
+ "prefix": frozenset({"bash", "ksh"}),
+ "export": frozenset({"bash"}),
+ "readonly": frozenset({"bash"}),
+ "declare": frozenset({"bash", "bash-posix"}),
+ "typeset": frozenset({"bash", "bash-posix", "mksh"}),
+ "local": frozenset({"bash", "bash-posix", "ksh", "mksh"}),
+ "read": frozenset({"bash", "bash-posix", "ksh", "mksh"}),
+ "printf": frozenset({"bash", "bash-posix", "dash", "ksh", "mksh", "ash"}),
+ "unset": frozenset({"bash", "bash-posix", "ksh", "mksh"}),
+ "for": frozenset({"bash"}),
+ "eval": frozenset({"bash", "zsh", "ksh"}),
+}
+# How each shell binds variables, measured on bash, bash --posix, dash, zsh,
+# ksh, mksh and busybox ash with ``<form> f=run.py; python3 "$f"`` against a
+# script that writes a marker. ``declare`` exists in bash and zsh and
+# ``typeset`` also in ksh and mksh; where one is missing the command fails and
+# binds nothing. ``local`` outside a function binds in zsh and mksh only, and
+# the walk does not track function bodies, so its value is never settled.
+_DECLARING_BUILTINS = {
+ "bash": frozenset({"export", "readonly", "declare", "typeset"}),
+ "bash-posix": frozenset({"export", "readonly", "declare", "typeset"}),
+ "zsh": frozenset({"export", "readonly", "declare", "typeset"}),
+ "ksh": frozenset({"export", "readonly", "typeset"}),
+ "mksh": frozenset({"export", "readonly", "typeset"}),
+ "dash": frozenset({"export", "readonly"}),
+ "ash": frozenset({"export", "readonly"}),
+}
+_DECLARATION_BUILTINS = frozenset({"export", "readonly", "declare", "typeset", "local"})
+# What each declaration option does to the variables it names, where the
+# shell accepts it. Any other option (``-u`` upper-cases the value, ``-i``
+# evaluates it, ``-a`` and ``-n`` change what ``$f`` reads), or one this shell
+# rejects, leaves the named variables unsettled: some shells then carry on
+# with nothing changed and others stop.
+_DECLARATION_OPTIONS = {
+ "export": {"-n": "unexport", "-f": "functions", "-p": "print"},
+ "readonly": {"-f": "functions", "-p": "print"},
+ "declare": {
+ "-x": "export",
+ "+x": "unexport",
+ "-r": "readonly",
+ "+r": "none",
+ "-g": "none",
+ "-f": "functions",
+ "-F": "functions",
+ "-p": "print",
+ },
+}
+_DECLARATION_OPTIONS["typeset"] = _DECLARATION_OPTIONS["declare"]
+# Options only some shells accept, measured with ``export f=run.py; <form> f;
+# bash -c 'python3 "$f"'``: ``export -n`` unexports in bash and ash and is
+# rejected by dash, zsh, ksh and mksh; ``export -f`` names functions in bash.
+# A listing (``-p``) or functions (``-f``, ``-F``) change no variable only in
+# the shells named here, measured with ``f=other.py; <form> f=run.py;
+# python3 "$f"`` and ``f=run.py; <form> f; f=other.py; python3 "$f"``:
+# ``declare -p`` and ``typeset -p`` only list in bash and zsh, whatever else
+# is given with them. ksh's ``typeset -p`` and ``typeset -f`` assign, and so
+# does mksh's ``typeset -px``; ``-F`` is a floating-point attribute in zsh
+# and ksh, which stop on a value such as run.py.
+_DECLARATION_OPTION_SHELLS = {
+ ("export", "-n"): frozenset({"bash", "bash-posix", "ash"}),
+ ("export", "-f"): frozenset({"bash", "bash-posix"}),
+ ("readonly", "-f"): frozenset({"bash", "bash-posix"}),
+ ("declare", "-g"): frozenset({"bash", "bash-posix", "zsh"}),
+ ("typeset", "-g"): frozenset({"bash", "bash-posix", "zsh"}),
+ ("declare", "-p"): frozenset({"bash", "bash-posix", "zsh"}),
+ ("typeset", "-p"): frozenset({"bash", "bash-posix", "zsh"}),
+ ("typeset", "-f"): frozenset({"bash", "bash-posix", "zsh", "mksh"}),
+ ("declare", "-F"): frozenset({"bash", "bash-posix"}),
+ ("typeset", "-F"): frozenset({"bash", "bash-posix"}),
+}
+# Given names, ``export -p`` and ``readonly -p`` act on them as they do
+# without ``-p`` in bash, bash --posix, ksh, mksh and busybox ash, which list
+# only when no name is given; dash and zsh list the names and change nothing.
+# Measured with ``export f=other.py; <form> f=run.py; python3 "$f"``,
+# ``<form> f=run.py; bash -c 'python3 "$f"'`` and ``f=run.py; <form> f;
+# f=other.py; python3 "$f"``.
+_LISTING_ACTS_ON_NAMES = frozenset({"bash", "bash-posix", "ksh", "mksh", "ash"})
+# Where a declaration assigns several names, most shells expand every value
+# before assigning any, so ``export f=run.py g=$f`` gives g the earlier f;
+# ksh assigns them in order. Prefix assignments are the reverse: every shell
+# but mksh assigns them in order, so ``f=run.py g=$f cmd`` hands cmd g=run.py.
+_DECLARATION_ASSIGNS_IN_ORDER = frozenset({"ksh"})
+_PREFIX_EXPANDS_FIRST = frozenset({"mksh"})
+# ``command export`` runs the builtin in every shell but zsh, where
+# ``command`` looks only for an external command; ``builtin export`` runs it
+# in bash, zsh and mksh, and elsewhere is not a way to reach it.
+_COMMAND_REACHES_BUILTINS = frozenset({"bash", "bash-posix", "dash", "ksh", "mksh", "ash"})
+_BUILTIN_REACHES_BUILTINS = frozenset({"bash", "bash-posix", "zsh", "mksh"})
+# Builtins that bind a variable from data the text does not carry: standard
+# input, a format string, the positional parameters.
+_RUNTIME_BINDING_BUILTINS = {"read": "REPLY", "mapfile": "MAPFILE", "readarray": "MAPFILE", "getopts": "OPTARG"}
+# An assignment written before a special builtin (``f=run.py :``) outlives
+# the command in the POSIX shells, bash --posix among them, and not in bash
+# or zsh. Before any other command it is that command's environment only,
+# and in every shell it is not seen by the command's own words:
+# ``f=run.py python3 "$f"`` runs python3 with an empty argument.
+_SPECIAL_BUILTINS = frozenset(
+ {
+ ":",
+ ".",
+ "break",
+ "continue",
+ "eval",
+ "exec",
+ "exit",
+ "export",
+ "readonly",
+ "return",
+ "set",
+ "shift",
+ "times",
+ "trap",
+ "unset",
+ }
+)
+_PREFIX_OUTLIVES_SPECIAL_BUILTIN = frozenset({"bash-posix", "dash", "ksh", "mksh", "ash"})
+
+
+def _binding_readings(shell: str | None) -> tuple[str, ...]:
+ """The binding rules a text is walked under.
+
+ The tool's own shell is read as bash, as elsewhere in this walk. ``sh`` is
+ dash on some systems, bash in POSIX mode on others and busybox ash on
+ Alpine, so it is walked under all three and disagreement is unresolved.
+ """
+ if shell is None:
+ return ("bash",)
+ if shell == "sh":
+ return ("dash", "bash-posix", "ash")
+ return (shell,) if shell in _DECLARING_BUILTINS else ("bash", "dash")
+
+
+def _refuse_assignment(
+ scope: dict[str, str],
+ name: str,
+ reading: str,
+ form: str = "assignment",
+ goes_on: dict[str, frozenset[str]] = _REFUSED_ASSIGNMENT_GOES_ON,
+) -> None:
+ """An assignment to ``name`` fails (read-only, or ``goes_on`` names the
+ failure): its value is unsettled, and where the shell stops there,
+ nothing after is credited as run."""
+ scope[name] = _UNSETTLED_VALUE
+ if reading not in goes_on.get(form, frozenset()):
+ scope[_ASSIGNMENT_REFUSED] = "1"
+
+
+def _bind(scope: dict[str, str], name: str, value: str, reading: str = "", form: str = "assignment") -> None:
+ """Bind ``name``, unless it is read-only, when the assignment fails."""
+ attributes = scope.get(_ATTRIBUTE_MARK + name, "")
+ if scope.get(_READONLY_MARK + name):
+ _refuse_assignment(scope, name, reading, form)
+ elif "i" in attributes and _integer_may_fail(value, scope):
+ _refuse_assignment(scope, name, reading, form, _INTEGER_ERROR_GOES_ON)
+ elif set(attributes) & set("iunc"):
+ scope[name] = _UNSETTLED_VALUE
+ target = value.strip("\"'")
+ if "n" in attributes and scope.get(_REFERENCE_MARK + name) == "" and _SHELL_NAME_RE.fullmatch(target):
+ # A reference that names nothing yet takes the value as the name
+ # it refers to (bash, ksh).
+ scope[_REFERENCE_MARK + name] = target
+ elif "n" in attributes and scope.get(_REFERENCE_MARK + name) == "":
+ # A value that is not a name fails there, as an assignment to a
+ # read-only name does.
+ _refuse_assignment(scope, name, reading, form)
+ if _UNSETTLED_VALUE in value:
+ scope[_REFERENCE_MARK + name] = _UNSETTLED_VALUE
+ elif "n" in attributes:
+ _pass_to_reference(scope, name)
+ else:
+ scope[name] = value.lower() if "l" in attributes else value
+ if scope.get(_ALLEXPORT_MARK):
+ scope[_EXPORTED_MARK + name] = "1"
+
+
+def _bind_unsettled(scope: dict[str, str], name: str, reading: str, form: str) -> None:
+ """Bind ``name`` to a value the text does not settle: a declaration whose
+ options the walk does not model."""
+ if scope.get(_READONLY_MARK + name):
+ _refuse_assignment(scope, name, reading, form)
+ return
+ scope[name] = _UNSETTLED_VALUE
+ if "n" in scope.get(_ATTRIBUTE_MARK + name, ""):
+ _pass_to_reference(scope, name)
+ if scope.get(_ALLEXPORT_MARK):
+ scope[_EXPORTED_MARK + name] = "1"
+
+
+def _pass_to_reference(
+ scope: dict[str, str], name: str, exported: bool = False, readonly: bool = False, depth: int = 0
+) -> None:
+ """What is done to a ``-n`` name is done to the name it refers to.
+
+ That name's value is unsettled, and it is marked exported where the
+ change exported or unexported it (so a child reads it as unsettled
+ rather than settled either way) and read-only where it was made so.
+ Where the text does not settle which name it is, every bound name is
+ treated so. A reference that names nothing yet changes nothing.
+ """
+ target = scope.get(_REFERENCE_MARK + name)
+ if target == "":
+ return
+ if target is None or target == name or not _SHELL_NAME_RE.fullmatch(target) or depth >= _MAX_SHELL_REFERENCE_DEPTH:
+ targets = [key for key in scope if _SHELL_NAME_RE.fullmatch(key)]
+ else:
+ targets = [target]
+ if "n" in scope.get(_ATTRIBUTE_MARK + target, ""):
+ _pass_to_reference(scope, target, exported, readonly, depth + 1)
+ for key in targets:
+ scope[key] = _UNSETTLED_VALUE
+ if exported or scope.get(_ALLEXPORT_MARK):
+ scope[_EXPORTED_MARK + key] = "1"
+ if readonly:
+ scope[_READONLY_MARK + key] = "1"
+
+
+def _apply_set_options(words: list[str], scope: dict[str, str]) -> None:
+ """Turn allexport on or off as ``set -a`` / ``set -o allexport`` and their
+ ``+`` forms do, reading options up to the first operand or ``--``."""
+ for position, word in enumerate(words):
+ if word == "--" or not word.startswith(("-", "+")) or len(word) < 2:
+ return
+ following = words[position + 1] if position + 1 < len(words) else ""
+ if word[1:] == "o":
+ if following == "allexport":
+ scope[_ALLEXPORT_MARK] = "1" if word[0] == "-" else ""
+ elif "a" in word[1:]:
+ scope[_ALLEXPORT_MARK] = "1" if word[0] == "-" else ""
+
+
+def _value_now(raw: str, scope: dict[str, str]) -> str:
+ """The value an assignment stores: each variable it reads, expanded now.
+
+ An assignment copies the value it reads; it is not a live alias, so
+ ``f=other.py; g=$f; f=run.py`` leaves g as other.py. A variable this text
+ has not bound reads as empty, as it does in the tool's clean environment.
+ What the text cannot settle (``$(...)``, ``${f:-x}``) is kept as written.
+ """
+ return _SHELL_VARIABLE_RE.sub(lambda match: scope.get(match.group(1) or match.group(2), ""), str(raw))
+
+
+def _attribute_changes(words: list[str]) -> tuple[set[str], set[str]]:
+ """The attribute letters (``_ATTRIBUTE_LETTERS``) a declaration's options
+ add with ``-`` and remove with ``+``."""
+ added: set[str] = set()
+ removed: set[str] = set()
+ for word in words:
+ if word == "--" or len(word) < 2 or word[0] not in "-+":
+ break
+ letters = set(word[1:]) & _ATTRIBUTE_LETTERS
+ (added if word[0] == "-" else removed).update(letters)
+ return added - removed, removed
+
+
+def _declaration_options(name: str, words: list[str], reading: str) -> tuple[list[str], set[str], bool]:
+ """Split a declaration's words into operands and option effects, and say
+ whether every option is one this shell accepts and the walk models."""
+ options: list[str] = []
+ operands: list[str] = []
+ ended = False
+ for word in words:
+ if not ended and word == "--":
+ ended = True
+ elif not ended and len(word) > 1 and word[0] in "-+":
+ options.extend(word[0] + letter for letter in word[1:])
+ else:
+ operands.append(word)
+ table = _DECLARATION_OPTIONS.get(name, {})
+ effects: set[str] = set()
+ known = True
+ for option in options:
+ effect = table.get(option)
+ shells = _DECLARATION_OPTION_SHELLS.get((name, option))
+ if effect is None or (shells is not None and reading not in shells):
+ known = False
+ elif effect == "print" and name in {"export", "readonly"} and reading in _LISTING_ACTS_ON_NAMES:
+ # Names given, it acts on them; none given, it names nothing.
+ effects.add("none")
+ else:
+ effects.add(effect)
+ if "print" in effects and name in {"declare", "typeset"}:
+ # ``declare -p`` lists whatever else is given with it.
+ return operands, {"print"}, True
+ return operands, effects, known
+
+
+def _apply_binding_builtin(command: list[str], cmd_idx: int, scope: dict[str, str], reading: str) -> bool:
+ """Apply what a builtin that binds variables does, and say whether it was one.
+
+ ``export f=run.py``, ``readonly f=run.py`` and, where the shell has them,
+ ``declare`` and ``typeset`` bind as ``f=run.py`` does, with each value
+ expanded when the builtin runs; ``export -n`` and ``+x`` unexport;
+ ``unset f`` leaves ``$f`` empty; ``read``, ``mapfile``, ``getopts`` and
+ ``printf -v`` bind from data the text does not carry, so their names are
+ unsettled. ``command`` and ``builtin`` before one, in any number, reach it
+ where the shell lets them. None of them runs a script, so the walk moves on
+ after them.
+ """
+ index = cmd_idx
+ vias: list[str] = []
+ while index < len(command):
+ via = _resolved_shell_arg(str(command[index]), scope).strip("\"'")
+ if via not in {"command", "builtin"}:
+ break
+ vias.append(via)
+ index += 1
+ while via == "command" and index < len(command) and str(command[index]) == "-p":
+ index += 1
+ if index >= len(command) or str(command[index]).startswith("-"):
+ # ``command -v``: a query, which binds nothing and is read as before.
+ return False
+ if index >= len(command):
+ return False
+ name = _resolved_shell_arg(str(command[index]), scope).strip("\"'")
+ if name == "nameref" and reading in _NAMEREF_SHELLS and (reading == "ksh" or not vias):
+ name = "typeset"
+ command = [*command[: index + 1], "-n", *command[index + 1 :]]
+ binding = name in _DECLARATION_BUILTINS or name in _RUNTIME_BINDING_BUILTINS or name in {"unset", "set", "printf"}
+ if binding and any(
+ reading not in (_COMMAND_REACHES_BUILTINS if via == "command" else _BUILTIN_REACHES_BUILTINS) for via in vias
+ ):
+ # The builtin is not reached: the command fails and changes nothing.
+ return name != "printf"
+ words = _without_redirections(command[index + 1 :])
+ if name in _DECLARATION_BUILTINS:
+ operands, effects, known = _declaration_options(name, words, reading)
+ names = [
+ (assignment.group(1) if assignment else word, assignment)
+ for word in operands
+ for assignment in [_SHELL_ASSIGNMENT_RE.match(word)]
+ if _SHELL_NAME_RE.fullmatch(assignment.group(1) if assignment else word)
+ ]
+ if name != "local" and name not in _DECLARING_BUILTINS.get(reading, frozenset()):
+ # Not a builtin in this shell: the command fails and binds nothing.
+ return True
+ if name == "local" or not known:
+ # ``local`` outside a function binds in zsh and mksh only, and an
+ # option this shell rejects or the walk does not model may leave
+ # the variable as it was or stop the shell: unsettled either way.
+ # A special builtin stops a POSIX shell on an option it rejects.
+ added, removed = _attribute_changes(words)
+ # ``export -n`` unexports; only a declaration gives attributes,
+ # and ``readonly`` those of an array.
+ letters_given = (
+ _ATTRIBUTE_LETTERS
+ if name in {"declare", "typeset", "local"}
+ else _READONLY_ATTRIBUTES.get(name, frozenset())
+ )
+ added, removed = added & letters_given, removed & letters_given
+ numeric_letters: set[str] = set()
+ width = False
+ for position, word in enumerate(words):
+ if word == "--" or len(word) < 2 or word[0] not in "-+":
+ break
+ if word[0] == "-":
+ # ksh applies ``-i`` before a later ``+i`` takes it away.
+ numeric_letters |= set(word[1:]) & _NUMERIC_ATTRIBUTES
+ following = words[position + 1] if position + 1 < len(words) else ""
+ # A width is given in the word (``-L3``) or as the next one
+ # (``-L 3``, ``-Lx 3``).
+ width = width or (
+ word[0] == "-"
+ and bool(set(word[1:]) & set("LRZ"))
+ and (any(c.isdigit() for c in word) or following.isdigit())
+ )
+ # ``local`` outside a function declares in zsh and mksh only.
+ declares = name in {"declare", "typeset"} or (name == "local" and reading in {"zsh", "mksh"})
+ numeric = declares and bool(numeric_letters)
+ width = declares and width
+ if name in _SPECIAL_BUILTINS and reading in _SPECIAL_BUILTIN_ERROR_STOPS and not vias:
+ scope[_ASSIGNMENT_REFUSED] = "1"
+ if "n" in added and reading not in _REFERENCE_SHELLS:
+ # ``-n`` is an option zsh rejects: the declaration binds nothing.
+ return True
+ for variable, assignment in names:
+ # ``-n`` refers the name to the one its value names, or with
+ # no value to the one it held.
+ value = _value_now(assignment.group(2), scope) if assignment else ""
+ target = (value if assignment else scope.get(variable, "")).strip("\"'")
+ if "n" in added and (
+ (target and not _SHELL_NAME_RE.fullmatch(target) and not set(target) & {_UNSETTLED_VALUE, "$", "`"})
+ or (not target and reading not in _EMPTY_REFERENCE_SHELLS)
+ ):
+ # A name that cannot be referred to: the declaration fails
+ # for it, and ksh stops.
+ if reading in _BAD_REFERENCE_STOPS:
+ scope[_ASSIGNMENT_REFUSED] = "1"
+ continue
+ if "n" in added | removed and "n" in scope.get(_ATTRIBUTE_MARK + variable, ""):
+ # Giving ``-n`` again points the name elsewhere and ``+n``
+ # frees it; neither acts on the name it referred to.
+ scope[_ATTRIBUTE_MARK + variable] = scope[_ATTRIBUTE_MARK + variable].replace("n", "")
+ letters = (set(scope.get(_ATTRIBUTE_MARK + variable, "")) | added) - removed
+ if (
+ numeric
+ and reading in _NUMERIC_ATTRIBUTE_STOPS
+ and _number_may_fail(value if assignment else scope.get(variable, ""), scope)
+ ):
+ _refuse_assignment(scope, variable, reading, "numeric attribute")
+ elif (
+ not (numeric and reading in _NUMERIC_ATTRIBUTE_STOPS)
+ and assignment is not None
+ and "i" in letters
+ and _integer_may_fail(value, scope)
+ ):
+ _refuse_assignment(scope, variable, reading, name, _INTEGER_ERROR_GOES_ON)
+ elif (
+ assignment is not None
+ or name == "local"
+ or (added and (reading not in _DECLARATION_KEEPS_VALUE or "n" in added))
+ or (width and reading in _JUSTIFYING_SHELLS)
+ ):
+ # With no value, only taking attributes away, or giving
+ # them in bash, leaves the value as it was.
+ _bind_unsettled(scope, variable, reading, name)
+ if letters:
+ scope[_ATTRIBUTE_MARK + variable] = "".join(sorted(letters))
+ else:
+ scope.pop(_ATTRIBUTE_MARK + variable, None)
+ if "n" in added:
+ # Empty where it names nothing yet, unsettled where the
+ # text does not settle the name.
+ scope[_REFERENCE_MARK + variable] = (
+ target if _SHELL_NAME_RE.fullmatch(target) or not target else _UNSETTLED_VALUE
+ )
+ elif "n" in removed:
+ scope.pop(_REFERENCE_MARK + variable, None)
+ return True
+ if effects & {"functions", "print"}:
+ # Functions, or a listing: no variable changes.
+ return True
+ readonly = name == "readonly" or "readonly" in effects
+ unexport = "unexport" in effects
+ exported = name == "export" or "export" in effects
+ before = dict(scope)
+ for variable, assignment in names:
+ if assignment is not None:
+ source = scope if reading in _DECLARATION_ASSIGNS_IN_ORDER else before
+ _bind(scope, variable, _value_now(assignment.group(2), source), reading, name)
+ if readonly:
+ scope[_READONLY_MARK + variable] = "1"
+ if unexport:
+ scope.pop(_EXPORTED_MARK + variable, None)
+ elif exported:
+ scope[_EXPORTED_MARK + variable] = "1"
+ if "n" in scope.get(_ATTRIBUTE_MARK + variable, "") and (readonly or unexport or exported):
+ _pass_to_reference(scope, variable, exported or unexport, readonly)
+ return True
+ options = [word for word in words if word.startswith("-")]
+ operands = [word for word in words if not word.startswith("-")]
+ if name == "unset":
+ accepted = {"-f", "-v", *(("-n",) if reading in _UNSET_REFERENCE_SHELLS else ())}
+ if any(option not in accepted for option in options):
+ # An option this shell rejects: nothing is unset, and a POSIX
+ # shell stops.
+ if reading in _SPECIAL_BUILTIN_ERROR_STOPS and not vias:
+ scope[_ASSIGNMENT_REFUSED] = "1"
+ return True
+ if not operands and reading == "ksh" and not vias:
+ # ksh refuses ``unset`` given no name, and stops (measured with
+ # ``f=run.py; unset > f; python3 "$f"``); ``command unset`` goes on.
+ scope[_ASSIGNMENT_REFUSED] = "1"
+ return True
+ if options == ["-f"]:
+ # Functions, not variables.
+ return True
+ for variable in operands:
+ if not _SHELL_NAME_RE.fullmatch(variable):
+ continue
+ reference = "n" in scope.get(_ATTRIBUTE_MARK + variable, "")
+ if scope.get(_READONLY_MARK + variable):
+ # A read-only name, which every shell refuses to unset, and
+ # some then stop.
+ _refuse_assignment(scope, variable, reading, "unset")
+ elif reference and "-n" not in options:
+ # A ``-n`` name: the name it refers to is unset instead.
+ _pass_to_reference(scope, variable)
+ elif options not in ([], ["-v"], ["-n"]):
+ # Options together that the walk does not model.
+ scope[variable] = _UNSETTLED_VALUE
+ elif options == ["-n"] and not reference and reading != "ksh":
+ # ``unset -n`` on a name that refers to nothing: bash leaves
+ # it, and ksh unsets it.
+ continue
+ else:
+ scope[variable] = ""
+ for mark in (_EXPORTED_MARK, _ATTRIBUTE_MARK, _REFERENCE_MARK):
+ scope.pop(mark + variable, None)
+ return True
+ if name == "set":
+ # ``set`` binds nothing itself; the walk goes on to read it as before.
+ _apply_set_options(words, scope)
+ return False
+ if name in _RUNTIME_BINDING_BUILTINS or (name == "printf" and "-v" in words):
+ names_bound = (
+ words[words.index("-v") + 1 : words.index("-v") + 2]
+ if name == "printf"
+ else [*operands, _RUNTIME_BINDING_BUILTINS[name]]
)
+ for variable in names_bound:
+ if _SHELL_NAME_RE.fullmatch(variable):
+ _bind(scope, variable, _UNSETTLED_VALUE, reading, "printf" if name == "printf" else "read")
+ return True
+ return False
- if not exec_calls:
- # Check observation text as fallback (script may run inside Skill tool)
- for tc in tool_calls:
- obs = str(tc.get("observation", "")).lower()
- if expected_script.lower() in obs:
- return {"passed": True, "score": 0.75, "reason": f"{expected_script} found in tool observation"}
- return {"passed": False, "score": 0.0, "reason": "No execute/run_code call found"}
- # Observation fallback for exec calls
- for call in exec_calls:
+# Expansions a binding cannot settle: command substitution, and a braced
+# expansion that is more than a plain name (``${f:-x}``, ``${#f}``).
+_UNSETTLED_EXPANSION_RE = re.compile(r"\$\(|`|\$\{(?![A-Za-z_][A-Za-z0-9_]*\})")
+
+
+def _apply_eval_bindings(words: list[str], scope: dict[str, str], reading: str, depth: int = 0) -> None:
+ """Leave unsettled what ``eval`` may bind: its words run again as commands here.
+
+ This shell expands the words first, so ``eval "$code"`` runs what ``code``
+ holds, while a ``$`` left quoted reaches eval's text as ``$``. The text is
+ then split into commands as the walk splits a line, and each is applied to
+ a copy of the bindings. A command word eval itself reads from a variable
+ (``eval '$code'``) is expanded and split there and is never an assignment.
+ A name the copy assigns, or whose value or attributes change there, is
+ unsettled afterwards, and exported if the copy exports it, because reading
+ quoted text a second time is not modelled exactly. So ``eval export -n f``
+ leaves no settled value for a child to inherit, and ``eval f=other.py``
+ none to read. Where the text is not settled here (a value the text cannot
+ settle, a command substitution), every bound name is left unsettled.
+ """
+ expanded = " ".join(_value_now(str(word), scope) for word in words)
+ text = expanded.replace(_LITERAL_DOLLAR, "$").replace(_QUOTED_NEWLINE, "\n")
+ if _UNSETTLED_VALUE in text or _UNSETTLED_EXPANSION_RE.search(text):
+ _unsettle_every_binding(scope)
+ return
+ trial = dict(scope)
+ assigned: set[str] = set()
+ segment: list[str] = []
+ for token in [*_shell_tokens(_mark_quoted_newlines(_mark_literal_dollars(text))), ";"]:
+ if token not in _SHELL_SEPARATORS:
+ segment.append(token)
+ continue
+ prefix: dict[str, str] = {}
+ cmd_idx = _command_start(segment, prefix) if segment else 0
+ for name, value in prefix.items():
+ assigned.add(name)
+ _bind(trial, name, _value_now(value, trial), reading, "eval")
+ command = [str(word) for word in segment[cmd_idx:]]
+ segment = []
+ if command and _SHELL_VARIABLE_RE.search(command[0]):
+ command = " ".join(_value_now(word, trial) for word in command).split()
+ if any(_UNSETTLED_VALUE in word for word in command):
+ _unsettle_every_binding(scope)
+ return
+ for name in _arithmetic_names(command):
+ assigned.add(name)
+ if "n" in trial.get(_ATTRIBUTE_MARK + name, ""):
+ _pass_to_reference(trial, name)
+ index = 0
+ while index < len(command) and command[index] in {"command", "builtin"}:
+ index += 1
+ while index < len(command) and command[index] == "-p":
+ index += 1
+ if index >= len(command):
+ continue
+ if command[index] in _LOOP_HEADER_WORDS and index + 1 < len(command):
+ assigned.add(command[index + 1])
+ elif command[index] == "eval" and depth < _MAX_SHELL_REFERENCE_DEPTH:
+ _apply_eval_bindings(command[index + 1 :], trial, reading, depth + 1)
+ else:
+ _apply_binding_builtin(command, 0, trial, reading)
+ for key in set(trial) | set(scope):
+ if trial.get(key) == scope.get(key):
+ continue
+ if key.startswith(_EXPORTED_MARK):
+ # Exported or unexported by the text: the value is unsettled, and
+ # the mark stays so that a child inherits that.
+ scope[key] = "1"
+ assigned.add(key.removeprefix(_EXPORTED_MARK))
+ elif key.startswith((_READONLY_MARK, _ATTRIBUTE_MARK, _REFERENCE_MARK)) or key in {
+ _ALLEXPORT_MARK,
+ _ASSIGNMENT_REFUSED,
+ }:
+ if key in trial:
+ scope[key] = trial[key]
+ else:
+ scope.pop(key, None)
+ else:
+ assigned.add(key)
+ for name in assigned:
+ if _SHELL_NAME_RE.fullmatch(name):
+ scope[name] = _UNSETTLED_VALUE
+
+
+def _unsettle_every_binding(scope: dict[str, str]) -> None:
+ """Text this walk cannot read may bind or unbind any name: none stays settled."""
+ for key in list(scope):
+ if _SHELL_NAME_RE.fullmatch(key):
+ scope[key] = _UNSETTLED_VALUE
+
+
+def _prefixed_scope(scope: dict[str, str], prefix: dict[str, str], reading: str) -> dict[str, str]:
+ """This shell's bindings with a command's ``NAME=value`` prefix over them.
+
+ The values are assigned in order, so a later one sees an earlier one,
+ except in mksh, which expands every value first (measured).
+ """
+ staged = dict(scope)
+ for name, value in prefix.items():
+ staged[name] = _value_now(value, scope if reading in _PREFIX_EXPANDS_FIRST else staged)
+ if "n" in scope.get(_ATTRIBUTE_MARK + name, ""):
+ staged[name] = _UNSETTLED_VALUE
+ _pass_to_reference(staged, name, exported=True)
+ return staged
+
+
+def _child_environment(
+ command: list[str], start: int, stop: int, scope: dict[str, str], prefix: dict[str, str], reading: str
+) -> dict[str, str] | None:
+ """What the command at ``stop`` inherits from this shell.
+
+ The exported names, then the segment's own ``NAME=value`` prefix, then
+ each ``env`` standing before the command, applied in order: ``-i`` and
+ ``-`` clear what came before, ``-u NAME`` removes one name, and its
+ assignments add names, expanded by this shell before env runs. ``None``
+ when a wrapper resets the environment in a way the text does not settle:
+ ``sudo`` and ``doas`` keep what their policy file says, and ``env -S``
+ splits a string into a command.
+ """
+ if any("n" in scope.get(_ATTRIBUTE_MARK + name, "") for name in prefix):
+ # A prefix on a ``-n`` name assigns the name it refers to.
+ return None
+ environment = {
+ key.removeprefix(_EXPORTED_MARK): scope.get(key.removeprefix(_EXPORTED_MARK), "")
+ for key in scope
+ if key.startswith(_EXPORTED_MARK)
+ # An array is never exported.
+ and not set(scope.get(_ATTRIBUTE_MARK + key.removeprefix(_EXPORTED_MARK), "")) & set("aA")
+ }
+ staged = _prefixed_scope(scope, prefix, reading)
+ for name in prefix:
+ environment[name] = staged[name]
+ index = start
+ while index < stop:
+ word = _shell_executable(_resolved_shell_arg(command[index], scope)).removesuffix(".exe")
+ if word in {"sudo", "doas"}:
+ return None
+ index += 1
+ if word != "env":
+ continue
+ options_done = False
+ while index < stop:
+ token = _resolved_shell_arg(command[index], scope).strip("\"'")
+ following = _resolved_shell_arg(command[index + 1], scope).strip("\"'") if index + 1 < stop else ""
+ assignment = _SHELL_ASSIGNMENT_RE.match(token)
+ if assignment is not None:
+ environment[assignment.group(1)] = _value_now(assignment.group(2), scope)
+ index += 1
+ continue
+ if options_done or not token.startswith("-"):
+ break
+ index += 1
+ if token == "--":
+ options_done = True
+ elif token in {"-", "-i", "--ignore-environment"}:
+ environment.clear()
+ elif token in {"-u", "--unset"}:
+ environment.pop(following, None)
+ index += 1
+ elif token.startswith("--unset="):
+ environment.pop(token.removeprefix("--unset="), None)
+ elif token in {"-C", "--chdir"}:
+ index += 1
+ elif token.startswith(("--chdir=", "--debug", "--null")) or token in {"-v", "-0"}:
+ continue
+ elif not token.startswith("--"):
+ letters = token[1:]
+ for position, letter in enumerate(letters):
+ if letter == "i":
+ environment.clear()
+ elif letter in "v0":
+ continue
+ elif letter == "u":
+ attached = letters[position + 1 :]
+ environment.pop(attached or following, None)
+ index += 0 if attached else 1
+ break
+ else:
+ return None
+ else:
+ return None
+ return environment
+
+
+def _reader_scope(
+ command: list[str], start: int, stop: int, scope: dict[str, str], prefix: dict[str, str], reading: str
+) -> dict[str, str]:
+ """The bindings whatever reads a command's text next may expand.
+
+ ``eval`` reads it in this shell, and inline code hands it on with the
+ command's environment: this shell's bindings, the command's prefix and
+ what ``env`` gives it. Asked only whether the script is named, so a wider
+ view costs no more than partial credit.
+ """
+ view = _prefixed_scope(scope, prefix, reading)
+ view.update(_child_environment(command, start, stop, scope, prefix, reading) or {})
+ return view
+
+
+def _carries_a_nested_invocation(command: list[str], cmd_idx: int, assignments: dict[str, str], expected: str) -> bool:
+ """Whether an invocation of the script sits inside an unrecognised command.
+
+ `flock /tmp/lock python run.py`, `strace -f python run.py` and
+ `taskset -c 0 python run.py` all run the script through a wrapper this walk
+ has no grammar for, and listing every such wrapper is not possible: the set
+ is open, and `./wrap.sh run.py` is indistinguishable from `cat run.py` by
+ text alone. So rather than name them, an unrecognised command that carries
+ what looks like an interpreter running the script is left unresolved, while
+ one that merely takes it as an argument stays a non-invocation.
+ """
+ words = [_resolved_shell_arg(str(word), assignments).strip("\"'") for word in command[cmd_idx + 1 :]]
+ for position, word in enumerate(words):
+ base = _VERSION_SUFFIX_RE.sub("", _shell_executable(word).removesuffix(".exe")) or word
+ if base not in _INTERPRETER_GRAMMARS and base != "source":
+ # "." is excluded: as an argument it is far more often a path or a
+ # filter (`jq . run.py`) than a sourcing command.
+ continue
+ if any(_script_path_matches(later, expected) or _unresolved_value(later) for later in words[position + 1 :]):
+ return True
+ return False
+
+
+def _unresolved_value(value: str) -> bool:
+ """Whether a resolved word holds a value the text bound but cannot settle."""
+ return _UNSETTLED_VALUE in value
+
+
+def _redirects_script_to_stdin(command: list[str], assignments: dict[str, str], expected: str) -> bool:
+ """Whether the script is fed to a command's standard input.
+
+ ``python <run.py`` runs the script even though it never appears as an
+ argument, and ``wc -l <run.py`` only counts its lines, so which of the two
+ happened is not something the command text settles.
+ """
+ for position, word in enumerate(command[:-1]):
+ token = str(word)
+ if token in _INPUT_REDIRECTS or (token.endswith("<") and not token.startswith(_QUOTED_SYNTAX_MARK)):
+ operand = _resolved_shell_arg(str(command[position + 1]), assignments)
+ if _script_path_matches(operand, expected) or _unresolved_value(operand):
+ return True
+ return False
+
+
+def _command_names_script(command: list[str], cmd_idx: int, assignments: dict[str, str], expected: str) -> bool:
+ """Whether the expected script is named anywhere in this command's words.
+
+ Inline code, a module, or text handed to ``eval`` can run the script
+ without it ever appearing as an argument this walk resolves, so naming it
+ is the difference between "nothing ran" and "this walk cannot tell".
+ A ``$`` this shell leaves quoted is read as a variable here: whatever
+ reads the text next (``eval``, a child shell, ``os.system``) may expand it.
+ """
+ target = str(expected).strip().strip("\"'")
+ if not target:
+ return False
+ return any(
+ target in _resolved_shell_arg(str(word).replace(_LITERAL_DOLLAR, "$"), assignments)
+ for word in command[cmd_idx + 1 :]
+ )
+
+
+def _names_script_anywhere(command_text: str, expected_script: str) -> bool:
+ """Whether the expected script's file name appears anywhere in the command text.
+
+ This is the reference test the checker applied before invocation evidence
+ was required, kept as the floor under partial credit. A walk that cannot
+ resolve a command which never names the script has learned nothing about
+ that script, so it answers "did not run it" rather than "cannot tell":
+ parsing uncertainty is not evidence.
+ """
+ name = str(expected_script).strip().strip("\"'").replace("\\", "/").rsplit("/", 1)[-1]
+ return bool(name) and name.casefold() in str(command_text).casefold()
+
+
+# Words that open and close a compound command, for scoping a pipeline that
+# runs one: ``echo x | while read -r l; do ...; done``.
+# ``{`` and ``}`` are here too: ``{ f=x; } | cat`` is one pipeline stage.
+_COMPOUND_OPENERS = frozenset({"for", "select", "while", "until", "if", "case", "{"})
+_COMPOUND_CLOSERS = frozenset({"done", "fi", "esac", "}"})
+_COMMAND_POSITION_LEADERS = frozenset({"then", "do", "else", "elif", "!", "{", "("})
+# A compound's opening word is reserved only where a command starts, not after
+# an assignment. Written there, it and the compound's own syntax are refused
+# by bash, bash --posix, dash, zsh, ksh, mksh and busybox ash, for each of
+# these and for ``(`` and ``((``: ``A=1 if true; then python3 run.py; fi``
+# runs nothing. Only zsh refuses ``[[`` there, or an opening word with none of
+# its compound's syntax after it (``A=1 for x; ...``); the others run it as a
+# command name that is not found. Both are read as refused, so an invocation
+# on such a line is unresolved rather than given a definite score.
+_OPENERS_REFUSED_AFTER_ASSIGNMENT = _COMPOUND_OPENERS | {"[["}
+
+
+def _compound_after_assignment(tokens: list[str]) -> bool:
+ """Whether a compound's opening word follows an assignment in a command.
+
+ Every modelled shell refuses the text before running the line that holds
+ it, but most run the complete commands before it, so the caller may
+ credit the first of those (``_command_run_before_refusal``). ``A=$(`` and
+ ``A=(`` reach here as an assignment ending in ``$`` or ``=`` and then
+ ``(``, a substitution or an array rather than a subshell.
+ """
+ assigned: str | None = None
+ at_start = True
+ for token in tokens:
+ if token in _SHELL_SEPARATORS:
+ assigned, at_start = None, True
+ continue
+ if not at_start:
+ continue
+ if assigned is None and (token in _COMMAND_INTRODUCING_WORDS or token in _COMMAND_POSITION_LEADERS):
+ continue
+ if _SHELL_ASSIGNMENT_RE.match(token):
+ assigned = token
+ continue
+ if assigned is not None:
+ if token in _OPENERS_REFUSED_AFTER_ASSIGNMENT:
+ return True
+ if token == "(" and not assigned.endswith(("$", "=")):
+ return True
+ at_start = False
+ return False
+
+
+# zsh parses the whole of a ``-c`` text before running any of it, so a syntax
+# error anywhere in it runs nothing. bash, bash --posix, dash, ksh, mksh and
+# busybox ash parse and run one complete command at a time: a command on a
+# line before the refused one has already run. Measured with
+# ``python3 run.py`` on the line before ``A=1 for g in x; do :; done``, and
+# before ``cat <<< x`` under dash and ash.
+_PARSES_WHOLE_TEXT_FIRST = frozenset({"zsh"})
+# How many lines are examined for where a complete command ends, and how much
+# text in all is read to decide it. Past either, the lines that remain are read
+# as one unit with the one before them.
+_MAX_PARSE_UNIT_LINES = 256
+_MAX_PARSE_UNIT_READ = 2 * _MAX_SHELL_REFERENCE_CHARS
+# An unquoted newline, kept apart from ``;`` while a unit's syntax is read, so
+# that ``;;`` in a ``case`` is not confused with a blank line.
+_UNIT_NEWLINE = ""
+# Only the first command of a text is credited when a later line is refused,
+# and only when the walk reads it as the shell does: one pipeline of simple
+# commands on its own lines (``_plain_pipeline``). Nothing ran before it, so no
+# command the walk does not model can have stopped it, and a line refused for
+# a reason the walk does not detect is not credited. Measured before a refused
+# line: ``exit``, ``exec true``, ``set -n`` or ``kill $$`` on the line before
+# ``python3 run.py``, and ``python3 run.py`` ending in ``>``, ``; fi``, ``(x)``,
+# ``;;`` or a carriage return, run nothing in bash, bash --posix, dash, zsh,
+# ksh, mksh or busybox ash; an empty command (``; python3 run.py``) runs
+# nothing but in ksh.
+_NOT_A_SIMPLE_COMMAND = (
+ _COMPOUND_OPENERS | _COMPOUND_CLOSERS | _COMMAND_POSITION_LEADERS | {"in", "[[", "]]", "time", "coproc", "function"}
+)
+_REDIRECTION_OPERATOR_RE = re.compile(r"\A\d*(?:&>>?|>>|>\||>&|<&|<>|<<<|<<-|<<|>|<)\Z")
+# A token of operator characters alone: ``|``, a redirection, or anything
+# else the tokenizer left joined (``>;``, ``|&``, ``<(``).
+_OPERATOR_TOKEN_RE = re.compile(r"\A\d*[;&|()<>]+\Z")
+# A substitution or an expansion inside ``"..."`` reads its own quotes, and
+# ``$'...'`` can hold an escaped one; a scan pairing quote characters would
+# misplace them, and with them where a line ends.
+_NESTED_QUOTING_RE = re.compile(r"\$[({']|`")
+# What closes each compound, when its opening word stands where a command may.
+_UNIT_CLOSERS = {
+ "for": "done",
+ "select": "done",
+ "while": "done",
+ "until": "done",
+ "if": "fi",
+ "case": "esac",
+ "{": "}",
+ "[[": "]]",
+}
+
+
+def _heredoc_body_spans(text: str, reading: str) -> list[tuple[int, int]] | None:
+ """Where each heredoc body lies in ``text``, as ``_split_heredocs`` reads it.
+
+ Each span runs from the newline that ends the line declaring the heredoc
+ to the newline that ends its terminator line, so neither of those, nor any
+ line of the body, can end a command. ``None`` when the text declares more
+ heredocs than are read, or one whose terminator is never named.
+ """
+ spans: list[tuple[int, int]] = []
+ position = 0
+ budget = _MAX_HEREDOC_OPERANDS
+ per_line = _MAX_HEREDOCS_PER_LINE.get(reading, _MAX_HEREDOC_OPERANDS)
+ while True:
+ offset = _heredoc_operator_index(text[position:])
+ if offset < 0:
+ return spans
+ after = position + offset + 2
+ if text.startswith("<", after):
+ # A here-string's operand stays on its line.
+ position = after + 1
+ budget -= 1
+ if budget <= 0:
+ return None
+ continue
+ line_end = text.find("\n", after)
+ header = _heredoc_header(text[after:] if line_end == -1 else text[after:line_end])
+ if header is None:
+ return None
+ _, strings, terminators = header
+ budget -= len(terminators) + len(strings)
+ if len(terminators) > per_line or budget < 0:
+ return None
+ if line_end == -1:
+ return spans
+ body_position = line_end + 1
+ for terminator, strip_tabs in terminators:
+ segment = text[body_position:]
+ _, resumed = _split_heredoc_body(segment, terminator, strip_tabs)
+ body_position += len(segment) - len(resumed)
+ end = body_position - 1 if body_position > line_end + 1 and text[body_position - 1] == "\n" else body_position
+ spans.append((line_end, end))
+ position = body_position
+
+
+def _unit_is_complete(unit: str, reading: str, arithmetic_parens: bool) -> bool:
+ """Whether ``unit`` ends a complete command, so the newline after it ends
+ what the shell parses before running it.
+
+ Every compound it opens is closed, and it does not end on ``|``, ``&&``,
+ ``||``, ``!`` or a function's name awaiting its body. Anything the reading
+ here does not settle is incomplete, so the next line joins the unit.
+ """
+ analysed = _split_heredocs(unit, reading)[0]
+ if not analysed.strip():
+ return True
+ marked = _mark_quoted_newlines(_mark_literal_dollars(analysed)).replace("\n", f" {_UNIT_NEWLINE} ")
+ tokens = [
+ token for token in _split_punctuation_runs(_shell_tokens(marked), arithmetic_parens, reading != "ksh") if token
+ ]
+ if not tokens:
+ return False
+ stack: list[str] = []
+ at_command = True
+ case_header = False
+ case_pattern = False
+ previous = ""
+ for token in tokens:
+ if stack and stack[-1] == "esac" and case_pattern:
+ if token == "esac":
+ stack.pop()
+ case_pattern = False
+ at_command = False
+ elif token == ")":
+ case_pattern = False
+ at_command = True
+ previous = token
+ continue
+ if token == _UNIT_NEWLINE or token in _SHELL_SEPARATORS:
+ if stack and stack[-1] == "esac" and not case_header and token in {";", "&"} and previous == ";":
+ # ``;;``, ``;&`` or ``;;&``: the next word is a pattern.
+ case_pattern = True
+ at_command = True
+ previous = token
+ continue
+ if _PUNCTUATION_RUN_RE.match(token) or token in {"<(", ">("}:
+ # A group, a substitution, an arithmetic command or a process
+ # substitution is open until its parenthesis closes.
+ for char in token:
+ if char == "(":
+ stack.append(")")
+ elif char == ")":
+ if not stack or stack[-1] != ")":
+ return False
+ stack.pop()
+ at_command = True
+ previous = token
+ continue
+ if case_header:
+ if token == "in":
+ case_header = False
+ case_pattern = True
+ previous = token
+ continue
+ if previous == "function":
+ # ``function f {``: the body follows the name.
+ at_command = True
+ previous = token
+ continue
+ if stack and stack[-1] == "]]" and token == "]]":
+ stack.pop()
+ elif at_command and token in _UNIT_CLOSERS:
+ stack.append(_UNIT_CLOSERS[token])
+ case_header = token == "case"
+ elif at_command and token in {"done", "fi", "esac", "}"}:
+ if not stack or stack[-1] != token:
+ return False
+ stack.pop()
+ at_command = token in _COMMAND_POSITION_LEADERS or token in {"time", "!"} or token in _COMMAND_INTRODUCING_WORDS
+ previous = token
+ last = [token for token in tokens if token != _UNIT_NEWLINE]
+ if stack or case_header or not last:
+ return False
+ if last[-1] in {"|", "&&", "||", "!", "time", "function"} or last[-2:] == ["|", "&"]:
+ return False
+ # A function's name awaits its body on the next line.
+ return last[-2:] != ["(", ")"] and not (len(last) >= 2 and last[-2] == "function")
+
+
+def _parse_unit_starts(text: str, reading: str, arithmetic_parens: bool) -> tuple[list[int], str]:
+ """Where each top-level unit the shell parses before running it starts,
+ and the text with its comments blanked.
+
+ The first is the start of the text; each other follows an unquoted
+ newline, outside a comment and a heredoc body, that ends a complete
+ command (``_unit_is_complete``). A newline whose place the text does not
+ settle ends nothing, so the lines around it are read as one unit. A
+ comment starts at a ``#`` that begins a word, after an operator too:
+ ``python3 run.py >#x`` leaves its redirection without an operand.
+ """
+ starts = [0]
+ if len(text) > _MAX_SHELL_REFERENCE_CHARS:
+ return starts, text
+ spans = _heredoc_body_spans(text, reading)
+ if spans is None:
+ return starts, text
+ span_ends = dict(spans)
+ blanked = list(text)
+ quote: str | None = None
+ lines = 0
+ read = 0
+ index = 0
+ while index < len(text):
+ if quote is None and index in span_ends:
+ index = span_ends[index]
+ continue
+ char = text[index]
+ if char == "\\" and quote != "'" and index + 1 < len(text):
+ index += 2
+ continue
+ if quote is not None:
+ if char == quote[-1]:
+ quote = None
+ index += 1
+ continue
+ if char == "#" and (index == 0 or text[index - 1].isspace() or text[index - 1] in _SHELL_METACHARS):
+ end = text.find("\n", index)
+ end = len(text) if end < 0 else end
+ blanked[index:end] = " " * (end - index)
+ index = end
+ continue
+ if char == "$" and text.startswith("'", index + 1):
+ # Keep the opening token to distinguish ANSI-C from plain single quotes.
+ quote = text[index : index + 2]
+ index += 2
+ continue
+ if char in "'\"`":
+ quote = char
+ elif char == "\n":
+ lines += 1
+ read += index - starts[-1]
+ if lines > _MAX_PARSE_UNIT_LINES or read > _MAX_PARSE_UNIT_READ:
+ break
+ if _unit_is_complete("".join(blanked[starts[-1] : index]), reading, arithmetic_parens):
+ starts.append(index + 1)
+ index += 1
+ return starts, "".join(blanked)
+
+
+def _plain_pipeline(unit: str, reading: str, arithmetic_parens: bool) -> bool:
+ """Whether ``unit``, its comments blanked, is one pipeline of simple
+ commands that the walk reads as the shell does.
+
+ Words and complete redirections, joined only by ``|``, with at most a
+ trailing ``;`` or ``&``. Not a reserved word, a group or an assignment
+ where a command starts, a parenthesis, ``&&`` or ``||``, a substitution,
+ a redirection operator without its operand, a here-string (a missing
+ operand is read from the next line), or a carriage return or an escaped
+ newline, which the walk reads as a line's end and the shells as part of
+ the word before it (``python3 run.py\\r`` opens ``run.py\\r``, and
+ ``python3 run.py\\`` joins the next line to ``run.py``). Anything else
+ is not plain, which only withholds credit.
+ """
+ if "\r" in unit or "\\\n" in unit or _NESTED_QUOTING_RE.search(unit):
+ return False
+ analysed, _, here_strings = _split_heredocs(unit, reading)
+ if here_strings:
+ return False
+ marked = _mark_quoted_newlines(_mark_literal_dollars(analysed)).replace("\n", f" {_UNIT_NEWLINE} ")
+ tokens = [
+ token for token in _split_punctuation_runs(_shell_tokens(marked), arithmetic_parens, reading != "ksh") if token
+ ]
+ while tokens and tokens[-1] == _UNIT_NEWLINE:
+ tokens.pop()
+ if tokens and tokens[-1] in {";", "&"}:
+ tokens.pop()
+ started = False # a word or a redirection of this command has been read
+ named = False # and its first word
+ joined = False # a ``|`` has been read, and nothing of the next command
+ pending = False # a redirection operator awaits its operand
+ for token in tokens:
+ operator = bool(_OPERATOR_TOKEN_RE.match(token))
+ if pending:
+ if token == _UNIT_NEWLINE or operator:
+ return False
+ pending = False
+ elif token == _UNIT_NEWLINE:
+ # Only the command after a ``|`` may start on a later line.
+ if not joined:
+ return False
+ elif token == "|":
+ if not started:
+ return False
+ started = named = False
+ joined = True
+ elif _REDIRECTION_OPERATOR_RE.match(token):
+ started, joined, pending = True, False, True
+ elif operator:
+ return False
+ else:
+ if not named and (token in _NOT_A_SIMPLE_COMMAND or _SHELL_ASSIGNMENT_RE.match(token)):
+ return False
+ started = named = True
+ joined = False
+ return started and not pending
+
+
+@lru_cache(maxsize=256)
+def _command_run_before_refusal(text: str, reading: str, arithmetic_parens: bool) -> str:
+ """The first command of ``text``, when the shell runs it before refusing a
+ later one and the walk reads it as the shell does; empty otherwise.
+
+ A unit is refused as the whole text is (a here-string under a shell that
+ has none, a compound's opening word after an assignment). The first unit
+ that is not blank must come before the first refused one and be a plain
+ pipeline (``_plain_pipeline``); it is returned with its comments blanked.
+ """
+ starts, blanked = _parse_unit_starts(text, reading, arithmetic_parens)
+ first: str | None = None
+ for number, start in enumerate(starts):
+ end = starts[number + 1] if number + 1 < len(starts) else len(text)
+ analysed, _, here_strings = _split_heredocs(text[start:end], reading)
+ tokens = _split_punctuation_runs(
+ _shell_tokens(_mark_quoted_newlines(_mark_literal_dollars(analysed))),
+ arithmetic_parens,
+ reading != "ksh",
+ )
+ if (here_strings and reading in _NO_HERE_STRING_SHELLS) or _compound_after_assignment(tokens):
+ return first or ""
+ unit = blanked[start:end]
+ if first is None and unit.strip(" \t\n"):
+ first = unit if _plain_pipeline(unit, reading, arithmetic_parens) else ""
+ return ""
+
+
+# Whether the last command of a pipeline runs in the current shell, so a
+# binding it makes outlives the pipeline. zsh and ksh run it there; bash,
+# dash and mksh fork it like the other stages, unless bash has `lastpipe`
+# set, which is a run-time option the text cannot settle. Measured on each
+# shell with `printf "" | for f in x; do :; done; echo "$f"`.
+_LAST_STAGE_IN_CURRENT_SHELL = frozenset({"zsh", "ksh"})
+_LAST_STAGE_IN_SUBSHELL = frozenset({"bash", "sh", "dash", "ash", "mksh"})
+# Option changes that can move the last stage between the two rules: bash's
+# `shopt -s lastpipe`, and zsh's `emulate sh` (measured: it forks the last
+# stage). Any of these words in the text leaves the rule unsettled.
+_PIPELINE_OPTION_RE = re.compile(r"\b(?:lastpipe|shopt|setopt|unsetopt|emulate)\b")
+
+
+def _command_position_words(command: list[str]) -> list[str]:
+ """The words of a segment that stand where a command may: the first, and
+ each that follows ``then``, ``do``, ``else``, ``elif``, ``!``, ``{`` or ``(``.
+ Only there is ``for`` or ``done`` a reserved word rather than an argument.
+ """
+ words = []
+ expect = True
+ for token in command:
+ word = str(token).strip("\"'")
+ if expect:
+ words.append(word)
+ expect = word in _COMMAND_POSITION_LEADERS
+ return words
+
+
+def _scope_events(command: list[str]) -> list[str]:
+ """The groups and compound openers of one segment, in the order they open.
+
+ ``{ ( for f in x`` opens a brace group, a subshell group and a loop, one
+ inside the next. A ``(`` is a group wherever it stands; an opener word
+ counts only at command position, where the shell reads it as one.
+ """
+ events = []
+ expect = True
+ for token in command:
+ word = str(token).strip("\"'")
+ if token == "(":
+ events.append("(")
+ elif expect and word in _COMPOUND_OPENERS:
+ events.append(word)
+ expect = word in _COMMAND_POSITION_LEADERS
+ return events
+
+
+def _compound_closer_ends(tokens: list[str], end: int, count: int) -> list[int]:
+ """For each of the ``count`` compounds a segment ending at ``end`` opens,
+ outermost first, the index just past the segment holding its closing
+ word; ``len(tokens)`` for one that never closes. The token there tells
+ whether that compound, and only that one, is piped into another stage.
+ """
+ ends = [len(tokens)] * count
+ depth = count
+ position = end
+ while position < len(tokens) and depth > 0:
+ if tokens[position] in _SHELL_SEPARATORS:
+ position += 1
+ continue
+ segment_end = position
+ while segment_end < len(tokens) and tokens[segment_end] not in _SHELL_SEPARATORS:
+ segment_end += 1
+ for word in _command_position_words(tokens[position:segment_end]):
+ if word in _COMPOUND_OPENERS:
+ depth += 1
+ elif word in _COMPOUND_CLOSERS:
+ depth -= 1
+ if 0 <= depth < count and ends[depth] == len(tokens):
+ ends[depth] = segment_end
+ position = segment_end
+ return ends
+
+
+_SHELL_NAME_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
+
+
+def _loop_header_is_empty(
+ command: list[str], cmd_idx: int, assignments: dict[str, str], positional_known_empty: bool
+) -> bool:
+ """``for f in; do`` with nothing after ``in``: the loop runs zero times.
+
+ Its variable keeps whatever it held, and its body never runs, so the
+ body is not evidence of anything. ``for f; do`` (no ``in``) is different:
+ it iterates the positional parameters, which the text does not carry.
+ """
+ words = [_resolved_shell_arg(str(word), assignments).strip("\"'") for word in command[cmd_idx + 1 :]]
+ if len(words) == 1 and _SHELL_NAME_RE.fullmatch(words[0]):
+ # ``for f; do`` iterates the positional parameters: empty when the
+ # text runs with none, which is what a tool call gets.
+ return positional_known_empty
+ return len(words) == 2 and bool(_SHELL_NAME_RE.fullmatch(words[0])) and words[1] == "in"
+
+
+def _loop_header_is_unresolved(
+ command: list[str], cmd_idx: int, assignments: dict[str, str], expected: str, reading: str = "bash"
+) -> bool:
+ """Bind a loop variable where the header settles it, else say whether it matters.
+
+ ``for f in run.py; do python $f; done`` gives ``f`` exactly one value, so
+ the body is read with ``f`` bound and scored as ``python run.py`` would be,
+ and ``cat $f`` in the same body stays a non-invocation. A header with
+ several values, or none (``for f; do`` iterates the positional
+ parameters), settles nothing about ``$f``: if the script is among the
+ values the command is unresolved, and otherwise the header runs nothing.
+ """
+ words = [_resolved_shell_arg(str(word), assignments).strip("\"'") for word in command[cmd_idx + 1 :]]
+ if words and _SHELL_NAME_RE.fullmatch(words[0]) and assignments.get(_READONLY_MARK + words[0]):
+ # The header cannot assign a read-only variable.
+ _refuse_assignment(assignments, words[0], reading, "for")
+ return _command_names_script(command, cmd_idx, assignments, expected)
+ if words and _SHELL_NAME_RE.fullmatch(words[0]) and "n" in assignments.get(_ATTRIBUTE_MARK + words[0], ""):
+ # A ``-n`` loop variable: bash refers it to each value in turn, and
+ # the name it referred to may be left changed.
+ _pass_to_reference(assignments, words[0])
+ if words and _SHELL_NAME_RE.fullmatch(words[0]):
+ if len(words) == 1:
+ # The positional parameters, which this text does not carry: a
+ # later read of the variable is unresolved, not a settled miss.
+ assignments[words[0]] = _UNSETTLED_VALUE
+ return _command_names_script(command, cmd_idx, assignments, expected)
+ if len(words) >= 2 and words[1] == "in":
+ values = words[2:]
+ if len(values) == 1:
+ _bind(assignments, words[0], _value_now(values[0], assignments), reading, "for")
+ return False
+ # This header gives the variable several values the text does not
+ # settle, or is ``for f; do`` over the positional parameters, so a
+ # value an earlier single-value header gave it no longer holds. An
+ # empty ``in`` list never reaches here: the walk skips that loop
+ # whole (see ``_loop_header_is_empty``).
+ assignments.pop(words[0], None)
+ return _command_names_script(command, cmd_idx, assignments, expected)
+
+
+def _command_start(command: list[str], prefix: dict[str, str]) -> int:
+ """Index of the word that is the command, past reserved words and assignments.
+
+ ``then FOO=1 ./run.py`` runs ``./run.py``: the reserved word introduces the
+ command and the assignment is its environment, in either order. The
+ assignments are collected in ``prefix`` rather than bound, because where
+ they apply depends on what follows them: alone they bind in the shell,
+ before a command they are that command's environment only, and in neither
+ case do they reach the command's own words.
+ """
+ cmd_idx = 0
+ while cmd_idx < len(command):
+ word = command[cmd_idx]
+ if word in _COMMAND_INTRODUCING_WORDS or word in _GROUPING_TOKENS:
+ cmd_idx += 1
+ continue
+ assignment = _SHELL_ASSIGNMENT_RE.match(word)
+ if assignment is None:
+ break
+ prefix[assignment.group(1)] = assignment.group(2)
+ cmd_idx += 1
+ return cmd_idx
+
+
+def _script_path_matches(value: Any, expected_script: str) -> bool:
+ """Whether one shell argument names the expected script exactly.
+
+ ``run.py`` matches ``run.py``, ``./run.py`` and ``/skills/demo/run.py``. It
+ does not match ``run.py.bak``, ``rerun.py`` or ``run.pyc``: a filename that
+ merely contains the expected one is a different file.
+ """
+ target = str(expected_script).strip().strip("\"'").replace("\\", "/")
+ candidate = str(value).strip().strip("\"'").replace("\\", "/")
+ if not target or not candidate:
+ return False
+ if "/" in target:
+ return candidate == target or candidate.endswith(f"/{target}")
+ return candidate.rsplit("/", 1)[-1] == target
+
+
+# How many heredoc and here-string operators one command's text is read for,
+# so the work stays bounded. Past it, the rest of the text is data: a script
+# named there is unresolved.
+_MAX_HEREDOC_OPERANDS = 256
+# How many heredocs one line may declare before the shell refuses the line.
+# Measured: bash and bash --posix exit with "maximum here-document count
+# exceeded" at 17 and mksh stops at 11 ("too many <<s"), before running
+# anything on the line; dash, zsh, ksh and busybox ash read 60 and more.
+# Past it, the line and everything after it are data.
+_MAX_HEREDOCS_PER_LINE = {"bash": 16, "bash-posix": 16, "mksh": 10}
+# dash and busybox ash have no here-string: ``<<<`` is a syntax error there,
+# and the shell runs nothing on the line that holds it, nor any of a compound
+# command spanning lines around it, nor anything after (measured), so under
+# those readings only the complete commands before it are credited as run.
+_NO_HERE_STRING_SHELLS = frozenset({"dash", "ash"})
+# The word after ``<<`` that ends the body, optionally quoted or escaped. The
+# ``-`` of ``<<-`` asks for leading tabs to be stripped from the terminator.
+_HEREDOC_DELIMITER_RE = re.compile(r"\A(-?)[ \t]*((?:[^\s;&|<>()'\"`\\]|\\.|'[^']*'|\"[^\"]*\")+)")
+
+
+def _skip_arithmetic(text: str, index: int) -> int:
+ """Advance past an arithmetic expansion, where ``<<`` is a shift operator."""
+ depth = 0
+ while index < len(text):
+ if text[index] == "(":
+ depth += 1
+ elif text[index] == ")":
+ depth -= 1
+ if depth == 0:
+ return index + 1
+ index += 1
+ return index
+
+
+def _heredoc_operator_index(text: str) -> int:
+ """Index of the first ``<<`` that is really a heredoc or here-string operator.
+
+ Quoted text and arithmetic are skipped, so neither ``echo '<<'`` nor
+ ``echo $((1 << 2))`` is read as one. Returns ``-1`` when there is none.
+ """
+ quote: str | None = None
+ index = 0
+ while index < len(text) - 1:
+ char = text[index]
+ if char == "\\" and quote != "'":
+ index += 2
+ continue
+ if char in {"'", '"'}:
+ if quote is None:
+ quote = char
+ elif quote == char:
+ quote = None
+ index += 1
+ continue
+ if quote is None and char == "(" and text[index + 1] == "(":
+ index = _skip_arithmetic(text, index)
+ continue
+ if quote is None and char == "<" and text[index + 1] == "<":
+ return index
+ index += 1
+ return -1
+
+
+def _split_heredoc_body(body: str, terminator: str, strip_tabs: bool) -> tuple[str, str]:
+ """Split a heredoc body from the commands written after its terminator line."""
+ lines = body.split("\n")
+ for offset, line in enumerate(lines):
+ candidate = line.lstrip("\t") if strip_tabs else line
+ if candidate.rstrip("\r") == terminator:
+ return "\n".join(lines[:offset]), "\n".join(lines[offset + 1 :])
+ return body, ""
+
+
+# shlex groups a run of shell punctuation into one token, so ``));`` arrives
+# whole and the separator inside it would otherwise be missed, leaving the
+# command after it read as an argument of the command before it.
+_PUNCTUATION_RUN_RE = re.compile(r"\A[();|&]+\Z")
+_PUNCTUATION_PIECE_RE = re.compile(r"&&|\|\||[();|&]")
+
+
+def _arithmetic_close(tokens: list[str], start: int) -> tuple[int, int] | None:
+ """Where the ``((`` before ``start`` is closed as an arithmetic command.
+
+ bash, zsh, ksh and mksh read ``((`` as arithmetic when the parenthesis that
+ closes its inner half is written directly before the one that closes its
+ outer half, as ``))``, whatever the text between: ``((f=x; g=y))`` is
+ arithmetic (and fails), ``((f=x; g=y) )`` and ``((f=x) ; g)`` are two
+ nested subshells. Parentheses opened inside are counted. Returns the index
+ of the token holding that ``))`` and the offset of its first character, or
+ ``None`` when the text closes some other way.
+ """
+ depth = 0
+ for index in range(start, len(tokens)):
+ token = tokens[index]
+ if not _PUNCTUATION_RUN_RE.match(token):
+ continue
+ for offset, char in enumerate(token):
+ if char == "(":
+ depth += 1
+ elif char == ")":
+ if depth:
+ depth -= 1
+ else:
+ return (index, offset) if token[offset + 1 : offset + 2] == ")" else None
+ return None
+
+
+def _split_punctuation_runs(
+ tokens: list[str], arithmetic: bool = True, triple_opens_subshell: bool = True
+) -> list[str]:
+ """Separate a grouped run of punctuation into the operators it is made of.
+
+ With ``arithmetic``, a ``((`` at command position that closes as an
+ arithmetic command is emitted as three tokens, ``((``, its whole body as
+ one word, and ``))``, so the walk neither splits the body at its
+ separators nor reads its parentheses as subshells. dash has no
+ arithmetic command, so its caller passes ``arithmetic=False`` and every
+ ``((`` is two subshells. ``$((`` and ``for ((`` are not at command
+ position and split as before.
+
+ The tokenizer hands ``(((`` over as one token. bash, zsh and mksh read it
+ as they read ``( ((``, a subshell around what may be an arithmetic
+ command, so with ``triple_opens_subshell`` the leading parentheses are
+ split off and the last two are tried as ``((``. ksh reads it as nested
+ subshells, which is what splitting every parenthesis gives. Measured
+ with ``f=other.py; (((for f in run.py; do :; done; python3 "$f")) | cat)``.
+ """
+ tokens = list(tokens)
+ separated: list[str] = []
+ position = 0
+ while position < len(tokens):
+ token = tokens[position]
+ at_command_position = (
+ not separated or separated[-1] in _SHELL_SEPARATORS or separated[-1] in _COMMAND_POSITION_LEADERS
+ )
+ if arithmetic and triple_opens_subshell and at_command_position and len(token) > 2 and set(token) == {"("}:
+ tokens[position : position + 1] = ["("] * (len(token) - 2) + ["(("]
+ continue
+ if arithmetic and token == "((" and at_command_position:
+ close = _arithmetic_close(tokens, position + 1)
+ if close is not None:
+ close_index, close_offset = close
+ body = [*tokens[position + 1 : close_index], tokens[close_index][:close_offset]]
+ separated.append("((")
+ separated.append(" ".join(word for word in body if word))
+ separated.append("))")
+ separated.extend(_PUNCTUATION_PIECE_RE.findall(tokens[close_index][close_offset + 2 :]))
+ position = close_index + 1
+ continue
+ if len(token) > 1 and _PUNCTUATION_RUN_RE.match(token):
+ separated.extend(_PUNCTUATION_PIECE_RE.findall(token))
+ else:
+ separated.append(token)
+ position += 1
+ return separated
+
+
+# An assignment inside an arithmetic command: ``f=1``, ``f+=2``, ``f<<=1``,
+# ``f++`` and ``--f``. The value is a number, never a script path; on an
+# error bash, zsh and mksh leave the variable as it was, and ksh aborts
+# the rest of the text.
+_ARITHMETIC_ASSIGNMENT_RE = re.compile(
+ r"([A-Za-z_]\w*)\s*(?:<<|>>|[-+*/%&|^])?=(?!=)|([A-Za-z_]\w*)\s*(?:\+\+|--)|(?:\+\+|--)\s*([A-Za-z_]\w*)"
+)
+# The walk hands ``$((`` over as ``$``, ``(``, ``(``.
+_ARITHMETIC_EXPANSION_RE = re.compile(r"\$\s*(?:\(\s*\(|\[)")
+
+
+def _arithmetic_texts(words: list[str], cmd_idx: int, expansions: bool = True) -> list[str]:
+ """The arithmetic a command evaluates, where an assignment binds as it does
+ in ``((...))``: that body, the words after ``let``, and with
+ ``expansions`` each ``$((...))`` and ``$[...]`` in its words, which this
+ shell evaluates as it expands them. A ``$`` left quoted is marked literal
+ and starts none. The caller passes ``expansions=False`` for text with no
+ ``$((`` or ``$[``, where ``$ ( (`` is ``$( (``, a command substitution."""
+ texts: list[str] = []
+ if "((" in words:
+ body = words.index("((") + 1
+ texts.append(words[body] if body < len(words) else "")
+ if cmd_idx < len(words) and words[cmd_idx].strip("\"'") == "let":
+ texts.append(" ".join(words[cmd_idx + 1 :]))
+ joined = " ".join(words) if expansions else ""
+ for match in _ARITHMETIC_EXPANSION_RE.finditer(joined):
+ depth = 0
+ end = len(joined)
+ for position in range(match.end(), len(joined)):
+ if joined[position] in "([":
+ depth += 1
+ elif joined[position] in ")]":
+ if depth == 0:
+ end = position
+ break
+ depth -= 1
+ texts.append(joined[match.end() : end])
+ return texts
+
+
+def _arithmetic_names(words: list[str], expansions: bool = True) -> list[str]:
+ """The names the arithmetic in a command assigns (``_arithmetic_texts``)."""
+ words = [str(word) for word in words]
+ return [
+ next(group for group in match.groups() if group)
+ for text in _arithmetic_texts(words, _command_start(words, {}), expansions)
+ for match in _ARITHMETIC_ASSIGNMENT_RE.finditer(text)
+ ]
+
+
+def _apply_arithmetic_assignments(
+ command: list[str], scope: dict[str, str], expected_script: str, expansions: bool = True
+) -> None:
+ """What the assignments in a command's arithmetic leave.
+
+ The variable ends as a number, unchanged after an error, or never read
+ because ksh aborted. Only when it held the script (or the script's name
+ is a number) is that a question. A ``-n`` name assigns the name it refers
+ to instead.
+ """
+ numeric_script = str(expected_script).rsplit("/", 1)[-1].isdigit()
+ for name in _arithmetic_names(command, expansions):
+ if "n" in scope.get(_ATTRIBUTE_MARK + name, ""):
+ _pass_to_reference(scope, name)
+ if numeric_script or _script_path_matches(_resolved_shell_arg(scope.get(name, ""), scope), expected_script):
+ scope[name] = _UNSETTLED_VALUE
+
+
+def _unquoted_separator_index(text: str) -> int:
+ """Where an operand ends: at the first separator outside quotes.
+
+ ``cat <<< "a; b"`` is one operand, so the ``;`` inside the quotes does not
+ end it and the command written after the real separator is still walked.
+ """
+ quote: str | None = None
+ for index, char in enumerate(text):
+ if char == "\\" and quote != "'":
+ continue
+ if char in {"'", '"'}:
+ if quote is None:
+ quote = char
+ elif quote == char:
+ quote = None
+ continue
+ if quote is None and char in {";", "\n", "|", "&"}:
+ return index
+ return len(text)
+
+
+def _split_heredocs(command_text: str, reading: str = "bash") -> tuple[str, str, int]:
+ """Split a command into the text to walk and the text that is operand data.
+
+ Returns the commands to walk, the heredoc or here-string data fed to
+ them, and how many here-strings were read. Quotes are tracked so ``echo '<<'`` is not mistaken for a heredoc,
+ arithmetic is skipped so ``echo $((1 << 2))`` is not either, a heredoc body
+ ends at its terminator line so commands written after it are still walked,
+ and a here-string consumes only its single operand.
+ """
+ commands: list[str] = []
+ data: list[str] = []
+ here_strings = 0
+ remaining = command_text
+ budget = _MAX_HEREDOC_OPERANDS
+ per_line = _MAX_HEREDOCS_PER_LINE.get(reading, _MAX_HEREDOC_OPERANDS)
+ while budget > 0:
+ position = _heredoc_operator_index(remaining)
+ if position < 0:
+ break
+ commands.append(remaining[:position])
+ rest = remaining[position + 2 :]
+ if rest.startswith("<"):
+ # Here-string: one operand, then normal commands resume.
+ operand = rest[1:].lstrip()
+ cut = _unquoted_separator_index(operand)
+ data.append(operand[:cut])
+ remaining = " " + operand[cut:]
+ budget -= 1
+ here_strings += 1
+ continue
+ line_end = rest.find("\n")
+ header = _heredoc_header(rest if line_end == -1 else rest[:line_end])
+ if header is None:
+ # Nothing names the end of a body, so the rest of the text is data.
+ data.append(rest)
+ remaining = ""
+ break
+ # The rest of the operator's own line still belongs to its commands,
+ # and every heredoc declared on it has its body read, in order, once
+ # the line ends: ``python3 <<A <<B run.py`` reads A's body, then B's.
+ header_text, header_data, terminators = header
+ budget -= len(terminators) + len(header_data)
+ if len(terminators) > per_line or budget < 0:
+ # The shell refuses the line, or which lines are bodies is past
+ # what is read here: the line, from its start, and all after it
+ # are data, so a script named there is unresolved.
+ walked = "".join(commands)
+ line_start = walked.rfind("\n") + 1
+ commands = [walked[:line_start]]
+ data.append(walked[line_start:] + "<<" + rest)
+ remaining = ""
+ break
+ commands.append(header_text)
+ data.extend(header_data)
+ here_strings += len(header_data)
+ if line_end == -1:
+ # The bodies never start, so what follows the delimiters is command.
+ remaining = ""
+ break
+ resumed = rest[line_end + 1 :]
+ for terminator, strip_tabs in terminators:
+ body, resumed = _split_heredoc_body(resumed, terminator, strip_tabs)
+ data.append(body)
+ remaining = "\n" + resumed
+ if budget <= 0 and _heredoc_operator_index(remaining) >= 0:
+ # More operators than are read: what is left may be bodies, so it is
+ # data rather than commands.
+ data.append(remaining)
+ remaining = ""
+ commands.append(remaining)
+ return "".join(commands), "\n".join(data), here_strings
+
+
+def _heredoc_header(line: str) -> tuple[str, list[str], list[tuple[str, bool]]] | None:
+ """Read the heredocs declared on one line, the first ``<<`` already consumed.
+
+ Returns the line's command text with each operator and delimiter removed,
+ the here-string operands on it, and each heredoc's terminator with whether
+ its ``<<-`` strips leading tabs, in the order the shell reads their bodies.
+ ``None`` when a ``<<`` names no terminator. Measured on bash, dash, zsh,
+ ksh, mksh and busybox ash: every heredoc on a line, across commands joined
+ by ``;`` or ``|`` included, takes its body after the line, in order.
+ """
+ pieces: list[str] = []
+ strings: list[str] = []
+ terminators: list[tuple[str, bool]] = []
+ rest = line
+ while True:
+ delimiter = _HEREDOC_DELIMITER_RE.match(rest)
+ if delimiter is None:
+ return None
+ terminators.append((delimiter.group(2).strip("\"'").replace("\\", ""), bool(delimiter.group(1))))
+ rest = " " + rest[delimiter.end() :]
+ while True:
+ position = _heredoc_operator_index(rest)
+ if position < 0:
+ pieces.append(rest)
+ return "".join(pieces), strings, terminators
+ pieces.append(rest[:position])
+ rest = rest[position + 2 :]
+ if not rest.startswith("<"):
+ break
+ # Here-string: one operand, then the line resumes.
+ operand = rest[1:].lstrip()
+ cut = _unquoted_separator_index(operand)
+ strings.append(operand[:cut])
+ rest = " " + operand[cut:]
+
+
+def _is_redirection_operator(token: str) -> bool:
+ return _is_output_redirect(token) or _is_heredoc_redirect(token) or token in _INPUT_REDIRECT_OPERATORS
+
+
+# A redirection operator as the tokenizer hands it over: a word of its own,
+# made only of ``<``, ``>``, ``&`` and ``|`` (zsh's ``>>|`` and ``&>|`` among
+# them), with a descriptor only where one was written flush against it
+# (``2>``, kept with its operator before tokenizing). An unquoted operator
+# never stays inside a word, so a word that ends or starts with one (``g=>``
+# from ``g='>'``, ``>x`` from ``'>x'``) is data, and a quoted operator
+# (``'>'``) carries the mark.
+_REDIRECTION_WORD_RE = re.compile(r"\d*[<>&|]*[<>][<>&|]*")
+
+
+def _without_redirections(words: list[str]) -> list[str]:
+ """A command's words with each redirection and its operand removed, as the
+ shell removes them: ``export -p > f=other.py`` writes a listing to a file
+ named f=other.py and binds nothing, and ``export g='>' f=run.py`` binds
+ both names."""
+ kept: list[str] = []
+ skip_next = False
+ for word in words:
+ token = str(word)
+ if skip_next:
+ skip_next = False
+ elif _REDIRECTION_WORD_RE.fullmatch(token):
+ skip_next = True
+ else:
+ kept.append(token)
+ return kept
+
+
+def _interpreter_operands(command: list[str], cmd_idx: int, assignments: dict[str, str]) -> list[str]:
+ """The interpreter's own arguments, with every redirection removed.
+
+ ``python3 < /dev/null run.py`` runs run.py: the redirection belongs to the
+ shell, not to python's argument list. The tokenizer splits an operator and
+ its operand into their own tokens and keeps a descriptor with its operator
+ (``2>``), so none of them may stand where the first operand is looked for.
+ A digit on its own is an operand: ``python3 2 > out.txt run.py`` runs a
+ script named ``2`` and hands it ``run.py``. A script fed through standard input is a different
+ shape, and :func:`_redirects_script_to_stdin` has already answered for it
+ before this is reached.
+ """
+ args: list[str] = []
+ skip_next = False
+ words = command[cmd_idx + 1 :]
+ for arg in words:
+ if skip_next:
+ skip_next = False
+ continue
+ token = str(arg)
+ if _is_heredoc_redirect(token):
+ break
+ if _is_output_redirect(token) or token in _INPUT_REDIRECT_OPERATORS or _DESCRIPTOR_OPERATOR_RE.match(token):
+ skip_next = True
+ continue
+ if _ATTACHED_REDIRECT_RE.match(token):
+ continue
+ args.append(_resolved_shell_arg(arg, assignments))
+ return args
+
+
+def _runs_inline_code(executable: str, command: list[str], cmd_idx: int, assignments: dict[str, str]) -> bool:
+ """Whether a shell's option prefix carries its inline-code option.
+
+ Only the options before the first script operand are the shell's own. In
+ ``bash run.sh -c 'echo done'`` the ``-c`` is run.sh's argument, and after
+ ``--`` nothing is an option at all, so the grammar walk that finds the
+ script operand decides this rather than a scan of every argument.
+ ``_shell_c_payload`` accepts any option containing the letter ``c``, which
+ also matches ``--check`` and ``-Mstrict``, so it is not consulted first.
+ """
+ status, _ = _interpreter_script_arg(executable, command, cmd_idx, assignments)
+ return status == _INLINE_CODE
+
+
+def _reads_program_from_stdin(interpreter: str, command: list[str], cmd_idx: int, assignments: dict[str, str]) -> bool:
+ """An interpreter given no script, no inline code and no terminal option
+ reads its program from standard input; ``-`` names standard input.
+ """
+ grammar = _INTERPRETER_GRAMMARS.get(interpreter)
+ if grammar is None:
+ return False
+ for arg in _interpreter_operands(command, cmd_idx, assignments):
+ word = str(arg).strip("\"'")
+ if word == "-":
+ continue
+ if word in grammar["code"] or word in grammar["terminal"] or word in grammar["value"]:
+ return False
+ if not word.startswith("-"):
+ return False
+ return True
+
+
+def _pipeline_upstream_names_script(tokens: list[str], idx: int, assignments: dict[str, str], expected: str) -> bool:
+ """Whether an earlier stage of the pipeline that the segment at ``idx``
+ ends names the expected script as one of its words.
+ """
+ position = idx - 1
+ while position >= 0:
+ token = tokens[position]
+ if token in _SHELL_SEPARATORS and token != "|":
+ return False
+ if token != "|" and _script_path_matches(_resolved_shell_arg(token, assignments), expected):
+ return True
+ position -= 1
+ return False
+
+
+def _interpreter_script_arg(
+ executable: str,
+ command: list[str],
+ cmd_idx: int,
+ assignments: dict[str, str],
+) -> tuple[str, str | None]:
+ """Resolve which argument an interpreter runs as a script.
+
+ Returns ``(_SCRIPT, arg)`` when one is named, ``(_NO_SCRIPT, None)`` when
+ the interpreter runs inline code, prints and exits, or only checks syntax,
+ and ``(_UNDECIDABLE, None)`` when any option before the candidate is not in
+ this interpreter's grammar. Only the first non-option argument is the
+ script: anything after it is that script's own argv, so
+ ``python other.py run.py`` runs ``other.py``.
+ """
+ base = _VERSION_SUFFIX_RE.sub("", executable) or executable
+ grammar = _INTERPRETER_GRAMMARS.get(base)
+ args = _interpreter_operands(command, cmd_idx, assignments)
+ if grammar is None:
+ # Sourcing and anything else without a grammar: the first argument is
+ # the file, and an option would mean a shape this walk does not model.
+ first = next((a for a in args if a), None)
+ if first is None:
+ return (_NO_SCRIPT, None)
+ return (_UNDECIDABLE, None) if str(first).strip("\"'").startswith("-") else (_SCRIPT, first)
+
+ index = 0
+ while index < len(args):
+ raw = args[index]
+ token = str(raw).strip("\"'")
+ index += 1
+ if token == "--":
+ return (_SCRIPT, args[index]) if index < len(args) else (_NO_SCRIPT, None)
+ if not token.startswith("-") or token == "-":
+ return (_SCRIPT, raw)
+ if token in grammar["code"]:
+ return (_INLINE_CODE, None)
+ if token in grammar["terminal"]:
+ return (_NO_SCRIPT, None)
+ if token in grammar["boolean"]:
+ continue
+ if token in grammar["value"]:
+ index += 1
+ continue
+ name, separator, _ = token.partition("=")
+ if separator and (name in grammar["value"] or name in grammar["boolean"]):
+ continue
+ if not token.startswith("--"):
+ # A short-option cluster, possibly carrying an attached value.
+ letters = token[1:]
+ short = {
+ kind: {o[1] for o in options if len(o) == 2 and not o.startswith("--")}
+ for kind, options in grammar.items()
+ }
+ recognised = True
+ for position, letter in enumerate(letters):
+ if letter in short["code"]:
+ return (_INLINE_CODE, None)
+ if letter in short["terminal"]:
+ return (_NO_SCRIPT, None)
+ if letter in short["value"]:
+ if position + 1 == len(letters):
+ index += 1
+ break
+ if letter not in short["boolean"]:
+ recognised = False
+ break
+ if recognised:
+ continue
+ return (_UNDECIDABLE, None)
+ return (_NO_SCRIPT, None)
+
+
+def _skip_wrapper_options(
+ wrapper: str,
+ command: list[str],
+ cmd_idx: int,
+ assignments: dict[str, str],
+) -> tuple[str, int]:
+ """Advance past one wrapper's own options using its grammar.
+
+ Returns ``(_WRAPPER_OK, index)`` at the wrapped command, ``_WRAPPER_NONE``
+ when a help or version option means the wrapper printed and exited, and
+ ``_WRAPPER_UNKNOWN`` for an option the grammar does not describe, because
+ assuming it takes no value is how a wrapper ends up crediting a command
+ that never ran.
+ """
+ grammar = _WRAPPER_GRAMMARS.get(wrapper)
+ if grammar is None:
+ return (_WRAPPER_UNKNOWN, cmd_idx)
+ index = cmd_idx + 1
+ options_done = False
+ while index < len(command):
+ token = _resolved_shell_arg(command[index], assignments).strip("\"'")
+ if not options_done and token in grammar["terminal"]:
+ return (_WRAPPER_NONE, index)
+ if wrapper in _STDIN_DEPENDENT_WRAPPERS:
+ return (_WRAPPER_UNKNOWN, index)
+ if token == "--":
+ # End of this wrapper's own OPTIONS. Its positional arguments, such
+ # as timeout's duration, still come before the wrapped command.
+ options_done = True
+ index += 1
+ continue
+ if not options_done:
+ if token in grammar["boolean"]:
+ index += 1
+ continue
+ if token in grammar["value"]:
+ index += 2
+ continue
+ name, separator, _ = token.partition("=")
+ if separator and (name in grammar["value"] or name in grammar["boolean"]):
+ index += 1
+ continue
+ if wrapper == "nice" and _NEGATIVE_NUMBER_RE.match(token):
+ index += 1
+ continue
+ assignment = _SHELL_ASSIGNMENT_RE.match(token)
+ if wrapper == "env" and assignment:
+ assignments[assignment.group(1)] = assignment.group(2)
+ index += 1
+ continue
+ if not token.startswith("-") or options_done:
+ if wrapper == "timeout" and _DURATION_ARG_RE.match(token):
+ index += 1
+ continue
+ if wrapper in _RUNNER_COMMAND_PREFIXES:
+ return (_WRAPPER_OK, index + 1) if token == "run" else (_WRAPPER_UNKNOWN, index)
+ return (_WRAPPER_OK, index)
+ if len(token) > 2 and not token.startswith("--"):
+ short = token[:2]
+ if short in grammar["value"]:
+ index += 1
+ continue
+ return (_WRAPPER_UNKNOWN, index)
+ return (_WRAPPER_NONE, index)
+
+
+def _skip_transparent_prefixes(
+ command: list[str],
+ cmd_idx: int,
+ assignments: dict[str, str],
+) -> tuple[str, int]:
+ """Advance past grouping tokens and wrappers that run the command after them."""
+ for _ in range(_MAX_SHELL_WRAPPERS):
+ if cmd_idx >= len(command):
+ return (_WRAPPER_OK, cmd_idx)
+ if command[cmd_idx] in _GROUPING_TOKENS or command[cmd_idx] in _COMMAND_INTRODUCING_WORDS:
+ cmd_idx += 1
+ continue
+ executable = _shell_executable(_resolved_shell_arg(command[cmd_idx], assignments)).removesuffix(".exe")
+ if executable in _TRANSPARENT_COMMAND_PREFIXES or executable in _RUNNER_COMMAND_PREFIXES:
+ status, cmd_idx = _skip_wrapper_options(executable, command, cmd_idx, assignments)
+ if status != _WRAPPER_OK:
+ return (status, cmd_idx)
+ continue
+ return (_WRAPPER_OK, cmd_idx)
+ return (_WRAPPER_OK, cmd_idx)
+
+
+def _double_parens_are_arithmetic(shell: str | None) -> bool | None:
+ """Whether ``((`` closed by ``))`` is an arithmetic command in this shell.
+
+ bash, zsh, ksh and mksh have one (measured); dash has none and reads two
+ subshells. ``sh`` and ``ash`` are dash on some systems and bash or
+ busybox on others, so the text does not settle it. The tool call's own
+ shell is read as bash.
+ """
+ if shell == "dash":
+ return False
+ if shell in {"sh", "ash"}:
+ return None
+ return True
+
+
+def _pipeline_last_stage_keeps_bindings(shell: str | None, command_text: str) -> bool | None:
+ """Whether a binding made in a pipeline's last command survives it.
+
+ ``True`` and ``False`` when the shell settles it; ``None`` when the text
+ does not: an unmodelled shell, or an option change named in the text
+ (``shopt -s lastpipe`` in bash, ``emulate sh`` in zsh) that moves that
+ stage between the two rules at run time.
+ """
+ if _PIPELINE_OPTION_RE.search(command_text):
+ return None
+ if shell in _LAST_STAGE_IN_CURRENT_SHELL:
+ return True
+ if shell is None or shell in _LAST_STAGE_IN_SUBSHELL:
+ # The tool's own shell is read as bash, which is what runs an
+ # agent's command and what the differential harness executes.
+ return False
+ return None
+
+
+def _cmd_executes_script(
+ cmd: Any,
+ expected_script: str,
+ *,
+ _depth: int = 0,
+ _shell: str | None = None,
+ _positional: bool = False,
+ _environment: dict[str, str] | None = None,
+) -> bool | None:
+ """Whether a shell command invokes ``expected_script``.
+
+ ``_shell`` names the interpreter running this text, when it is known:
+ a ``-c`` payload carries its shell, and a tool call may name its own
+ (``_tool_call_shell``); a text with neither is read as bash.
+ ``_environment`` is what a payload's shell inherits: the names exported
+ to it and the prefix assignments of the command that started it.
+ Three rules in the walk depend on the shell: whether the last command of
+ a pipeline keeps its bindings, whether ``((`` is arithmetic, and how
+ variables are bound (which declaring builtins exist, and whether an
+ assignment before a special builtin outlives it). Where the shell does
+ not settle one, the walk runs under every reading, and disagreement is
+ unresolved rather than one shell's answer presented as every shell's.
+ """
+ keep = _pipeline_last_stage_keeps_bindings(_shell, str(cmd))
+ arithmetic = _double_parens_are_arithmetic(_shell)
+ results = {
+ _walk_for_invocation(
+ cmd, expected_script, _depth, keep_stage, _positional, arithmetic_parens, binding_reading, _environment
+ )
+ for keep_stage in ([keep] if keep is not None else [False, True])
+ for arithmetic_parens in ([arithmetic] if arithmetic is not None else [True, False])
+ for binding_reading in _binding_readings(_shell)
+ }
+ return results.pop() if len(results) == 1 else None
+
+
+def _walk_for_invocation(
+ cmd: Any,
+ expected_script: str,
+ _depth: int,
+ keep_last_stage: bool,
+ positional: bool,
+ arithmetic_parens: bool,
+ binding_reading: str = "bash",
+ environment: dict[str, str] | None = None,
+ own_commands_only: bool = False,
+) -> bool | None:
+ """Whether a shell command invokes ``expected_script``.
+
+ Credit is given only for a recognised way of running a script: the script
+ invoked directly, an interpreter given it as its script argument, a
+ ``source``, or a ``sh -c`` payload that does one of those. Every other
+ command is reported as not an invocation, so reading, printing, searching,
+ copying or deleting the file needs no special case. ``own_commands_only``
+ leaves a ``-c`` payload unresolved, for a command credited before a line
+ its shell refuses: the payload's own text may be refused too.
+
+ ``None`` means undecidable rather than negative, and is returned only when
+ the shell resolves the path at run time or an unrecognised option may have
+ consumed it. The same "a textual match is not evidence" reasoning is already
+ applied to SKILL.md reads by :func:`_cmd_reads_skill_md`.
+ """
+ if not expected_script:
+ return False
+ if _depth > _MAX_SHELL_REFERENCE_DEPTH:
+ return None
+ command_text = str(cmd)
+ if not command_text.strip():
+ return False
+ if not _names_script_anywhere(command_text, expected_script) and not (
+ _depth > 0 and _SHELL_VARIABLE_RE.search(command_text)
+ ):
+ # An unresolved walk over a command that never names the script is
+ # not evidence about that script, so it is a non-invocation, as it was
+ # before invocation evidence was required. A ``-c`` payload is the
+ # exception when it reads a variable: the command around it names the
+ # script, and ``python3 "$f"`` in the child may be given it through
+ # the environment.
+ return False
+ # A heredoc or here-string operand is data rather than further commands, but
+ # the tokenizer turns its newlines into separators, so it is split out.
+ analysed_text, unexamined_text, here_strings = _split_heredocs(command_text, binding_reading)
+ tokens = _split_punctuation_runs(
+ _shell_tokens(_mark_quoted_newlines(_mark_literal_dollars(analysed_text))),
+ arithmetic_parens,
+ binding_reading != "ksh",
+ )
+ # A shell with no here-string rejects the text before running it, and so
+ # does every shell given a compound's opening word after an assignment.
+ here_string_refused = bool(here_strings) and binding_reading in _NO_HERE_STRING_SHELLS
+ syntax_rejected = here_string_refused or _compound_after_assignment(tokens)
+ if syntax_rejected and _depth == 0 and binding_reading not in _PARSES_WHOLE_TEXT_FIRST:
+ # This shell ran the text's first command before the one it refuses.
+ # When nothing the walk misreads can have stopped it, an invocation
+ # there counts as it would on its own (``_command_run_before_refusal``).
+ # Only in the text a tool call gives: a ``-c`` payload is what the
+ # tokenizer left of it (a carriage return there reads as a line's end).
+ first = _command_run_before_refusal(command_text, binding_reading, arithmetic_parens)
+ if first and (
+ _walk_for_invocation(
+ first,
+ expected_script,
+ _depth,
+ keep_last_stage,
+ positional,
+ arithmetic_parens,
+ binding_reading,
+ environment,
+ own_commands_only=True,
+ )
+ is True
+ ):
+ return True
+ if not tokens:
+ return None if str(expected_script) in command_text else False
+
+ assignments: dict[str, str] = {}
+ for name, value in (environment or {}).items():
+ # Inherited, and exported onward to any shell this one starts.
+ assignments[name] = value
+ assignments[_EXPORTED_MARK + name] = "1"
+ # Every scope a command can run in, innermost last. A ``( ... )`` group
+ # and a compound command isolated as a pipeline stage each run in a
+ # subshell, with a copy of the bindings around them that is dropped
+ # when they end. A frame is (kind, bindings, the compound depth it
+ # opened at): "group" for ``(``, "stage" for a compound stage. A
+ # command reads and binds the innermost frame, and a frame opened
+ # inside another copies that one, so groups and stages nest in either
+ # order.
+ frames: list[tuple[str, dict[str, str], int]] = []
+ compound_depth = 0
+ closing_parens = 0
+
+ def innermost() -> dict[str, str]:
+ return frames[-1][1] if frames else assignments
+
+ scope = assignments
+
+ def credited() -> bool:
+ """Whether an invocation found here counts: not once an assignment the
+ shell stops at has failed before it (see ``_ASSIGNMENT_REFUSED``), nor
+ in a text the shell rejects (``_NO_HERE_STRING_SHELLS``,
+ ``_compound_after_assignment``); the first command it ran before
+ rejecting it may be credited above (``_command_run_before_refusal``)."""
+ return not scope.get(_ASSIGNMENT_REFUSED) and not syntax_rejected
+
+ def close_frames(parens: int) -> None:
+ """Drop what ended with the previous segment, innermost first: a
+ stage whose compound has closed, and a group for each ``)``."""
+ while frames:
+ kind, _, opened_at = frames[-1]
+ if kind == "stage" and compound_depth <= opened_at:
+ frames.pop()
+ elif kind == "group" and parens > 0:
+ frames.pop()
+ parens -= 1
+ else:
+ break
+
+ positional_known_empty = not positional and not _POSITIONAL_SET_RE.search(command_text)
+ current_directory: str | None = None
+ undecidable = False
+ ran_a_wrapper_help = False
+ idx = 0
+ while idx < len(tokens):
+ if tokens[idx] in _SHELL_SEPARATORS:
+ idx += 1
+ continue
+ end = idx
+ while end < len(tokens) and tokens[end] not in _SHELL_SEPARATORS:
+ end += 1
+ command = tokens[idx:end]
+ close_frames(closing_parens)
+ closing_parens = 0
+ # A pipeline runs each of its commands in a subshell, so a binding
+ # made in one does not survive past the pipeline:
+ # ``printf '' | for f in run.py; do cat $f; done; python3 $f`` runs
+ # python3 with an empty ``$f``. Each group and compound this segment
+ # opens is placed in turn: a ``(`` always runs in a subshell, and a
+ # compound runs in one when it is itself a pipeline stage, piped from
+ # (the segment's first command) or piped into (its own closing word
+ # is followed by ``|``), except as the last stage where the shell
+ # keeps it. Each such scope copies the innermost one around it and
+ # lasts until its closing word, so its own body sees its bindings.
+ position_words = _command_position_words(command)
+ closes = sum(word in _COMPOUND_CLOSERS for word in position_words)
+ compound_depth -= closes
+ events = _scope_events(command)
+ opener_count = sum(event != "(" for event in events)
+ closer_ends = _compound_closer_ends(tokens, end, opener_count)
+ piped_from = idx > 0 and tokens[idx - 1] == "|"
+ piped_into = end < len(tokens) and tokens[end] == "|"
+ lead = next((word for word in position_words if word != "!"), None)
+ opened = 0
+ # ``$((`` reaches the walk as ``$ ( (``, which opens groups here like
+ # ``$(`` does. Its arithmetic runs in the shell around them, so it
+ # binds in the scope in force before the first ``(`` after a ``$``.
+ parens = [position for position, token in enumerate(command) if token == "("]
+ expansion_opens = next(
+ (count for count, position in enumerate(parens) if position and str(command[position - 1]).endswith("$")),
+ None,
+ )
+ arithmetic_scope = None
+ for event in events:
+ if event == "(":
+ if expansion_opens is not None and arithmetic_scope is None:
+ if expansion_opens == 0:
+ arithmetic_scope = innermost()
+ expansion_opens -= 1
+ frames.append(("group", dict(innermost()), compound_depth))
+ continue
+ compound_end = closer_ends[opened]
+ feeds = compound_end < len(tokens) and tokens[compound_end] == "|"
+ fed = opened == 0 and piped_from and lead == event
+ if feeds or (fed and not keep_last_stage):
+ frames.append(("stage", dict(innermost()), compound_depth))
+ compound_depth += 1
+ opened += 1
+ closing_parens = command.count(")")
+ if opened:
+ # The command after the opening words runs inside the innermost
+ # compound, so a pipe after this segment is that command's.
+ in_pipeline = piped_into
+ else:
+ # A pipe before ``(`` belongs to the group, not to the command
+ # inside it, which runs in the group's own scope.
+ preceded = piped_from and command[0] != "("
+ in_pipeline = preceded or piped_into
+ if keep_last_stage and preceded and not piped_into:
+ # The last command of the pipeline runs in the current
+ # shell here (zsh, ksh), so what it binds is kept.
+ in_pipeline = False
+ scope = dict(innermost()) if in_pipeline else innermost()
+ _apply_arithmetic_assignments(
+ command,
+ scope if in_pipeline or arithmetic_scope is None else arithmetic_scope,
+ expected_script,
+ "$((" in analysed_text or "$[" in analysed_text,
+ )
+ if "((" in command:
+ idx = end + 1
+ continue
+
+ prefix: dict[str, str] = {}
+ cmd_idx = _command_start(command, prefix)
+ if cmd_idx >= len(command):
+ # Assignments alone bind in this shell; a bare reserved word runs
+ # nothing. Neither runs a script.
+ for name, value in prefix.items():
+ _bind(scope, name, _value_now(value, scope), binding_reading)
+ idx = end + 1
+ continue
+ if _resolved_shell_arg(command[cmd_idx], scope).strip("\"'") in _SPECIAL_BUILTINS and (
+ binding_reading in _PREFIX_OUTLIVES_SPECIAL_BUILTIN
+ ):
+ for name, value in prefix.items():
+ _bind(scope, name, _value_now(value, scope), binding_reading, "prefix-special")
+ else:
+ for name, value in prefix.items():
+ if scope.get(_READONLY_MARK + name):
+ # The assignment fails; bash and ksh still run the command
+ # and go on, the other shells stop.
+ _refuse_assignment(scope, name, binding_reading, "prefix")
+ elif "i" in scope.get(_ATTRIBUTE_MARK + name, "") and _integer_may_fail(
+ _value_now(value, scope), scope
+ ):
+ _refuse_assignment(scope, name, binding_reading, "prefix", _INTEGER_ERROR_GOES_ON)
+ if _apply_binding_builtin(command, cmd_idx, scope, binding_reading):
+ idx = end + 1
+ continue
+ openers = [event for event in events if event != "("]
+ if command[cmd_idx] in _LOOP_HEADER_WORDS and openers and openers[-1] == command[cmd_idx]:
+ # Only a loop word where a command starts opens a loop. After an
+ # assignment or ``--`` it is an ordinary word with no closer to
+ # skip to (``_compound_after_assignment``).
+ if _loop_header_is_empty(command, cmd_idx, scope, positional_known_empty):
+ # Zero iterations: the body is skipped whole, and the
+ # closing word it ends with is accounted for here.
+ # The loop is the last compound this segment opens; any before it
+ # encloses it and stays open.
+ skip_to = closer_ends[-1]
+ for token in tokens[end:skip_to]:
+ # Nothing in the body runs, but a parenthesis in it still
+ # opens or closes a subshell around what follows.
+ if token == "(":
+ frames.append(("group", dict(innermost()), compound_depth))
+ elif token == ")":
+ closing_parens += 1
+ # The skipped tokens hold the loop's own closing word; any
+ # compound opened inside the body closes there too.
+ compound_depth -= 1
+ idx = skip_to
+ continue
+ # The header runs nothing itself. With a single value the loop
+ # variable is bound for the body that follows; otherwise a script
+ # named here is unresolved, never a settled non-invocation.
+ if _loop_header_is_unresolved(command, cmd_idx, scope, expected_script, binding_reading):
+ undecidable = True
+ idx = end + 1
+ continue
+ if command[cmd_idx] in _UNMODELLED_CONTROL_WORDS:
+ # `case` is not modelled, so what its bodies do with the script
+ # this text names is not settled either way.
+ undecidable = True
+ idx = end + 1
+ continue
+
+ if _redirects_script_to_stdin(command, scope, expected_script):
+ undecidable = True
+ idx = end + 1
+ continue
+ leading = _shell_executable(_resolved_shell_arg(command[cmd_idx], scope)).removesuffix(".exe")
+ # Where the words that start this command begin: the wrappers between
+ # here and the command decide what it inherits (``_child_environment``).
+ # They are skipped over copies of the bindings, so what ``env`` gives
+ # the command never reaches this shell or the command's own words.
+ start_idx = cmd_idx
+ if leading in _WRAPPER_GRAMMARS:
+ wrapper_status, cmd_idx = _skip_wrapper_options(leading, command, cmd_idx, dict(scope))
+ if wrapper_status != _WRAPPER_OK:
+ if wrapper_status == _WRAPPER_NONE:
+ ran_a_wrapper_help = True
+ if wrapper_status == _WRAPPER_UNKNOWN:
+ undecidable = True
+ idx = end + 1
+ continue
+ unwrapped_idx = _unwrap_shell_command(command, cmd_idx, dict(scope))
+ if unwrapped_idx is None:
+ undecidable = True
+ idx = end + 1
+ continue
+ wrapper_status, cmd_idx = _skip_transparent_prefixes(command, unwrapped_idx, scope)
+ if wrapper_status != _WRAPPER_OK:
+ if wrapper_status == _WRAPPER_NONE:
+ ran_a_wrapper_help = True
+ if wrapper_status == _WRAPPER_UNKNOWN:
+ undecidable = True
+ idx = end + 1
+ continue
+
+ if cmd_idx < len(command):
+ if _resolved_shell_arg(command[cmd_idx], scope).strip("\"'").startswith("-"):
+ # An option standing where a command should: an option outside
+ # the wrapper's grammar consumed the tokens up to here, so what
+ # runs is unresolved rather than nothing.
+ undecidable = True
+ idx = end + 1
+ continue
+ executable_path = _path_with_shell_cwd(
+ _resolved_shell_arg(command[cmd_idx], scope),
+ current_directory,
+ )
+ if _UNSETTLED_VALUE in executable_path:
+ undecidable = True
+ idx = end + 1
+ continue
+ executable = _shell_executable(executable_path)
+
+ if executable == "cd":
+ directory = next(
+ (arg for arg in _command_input_args(command, cmd_idx, scope) if arg and not arg.startswith("-")),
+ None,
+ )
+ if directory is None:
+ undecidable = True
+ else:
+ current_directory = directory
+ idx = end + 1
+ continue
+
+ # `perl5.38.2` and `python3.13` run the same scripts as `perl` and
+ # `python`, so the version suffix is dropped before every lookup.
+ interpreter = _VERSION_SUFFIX_RE.sub("", executable) or executable
+ if interpreter in _SHELL_COMMAND_INTERPRETERS and _runs_inline_code(executable, command, cmd_idx, scope):
+ payload = _shell_c_payload(command, cmd_idx, scope)
+ if payload is not None and own_commands_only:
+ undecidable = True
+ idx = end + 1
+ continue
+ if payload is not None:
+ # This shell expands what it did not leave quoted, a name
+ # it never bound to nothing, before the child reads the rest.
+ payload = _value_now(payload, scope).replace(_LITERAL_DOLLAR, "$")
+ # After the payload, the first operand is its $0 and the rest
+ # its positional parameters; they reach the payload only
+ # through a reference to them in its text.
+ after = _shell_c_positional(command, cmd_idx, scope)
+ argv0, params = after[:1], after[1:]
+ child_environment = _child_environment(command, start_idx, cmd_idx, scope, prefix, binding_reading)
+ nested = _cmd_executes_script(
+ payload,
+ expected_script,
+ _depth=_depth + 1,
+ _shell=interpreter,
+ _positional=bool(params),
+ _environment=child_environment or {},
+ )
+ if nested is True:
+ if credited():
+ return True
+ undecidable = True
+ reaches = (
+ _READS_POSITIONAL_RE.search(payload)
+ and any(_script_path_matches(w, expected_script) for w in params)
+ ) or (
+ _READS_ARGV0_RE.search(payload) and any(_script_path_matches(w, expected_script) for w in argv0)
+ )
+ if nested is None or reaches or (child_environment is None and _SHELL_VARIABLE_RE.search(payload)):
+ # A payload reading a variable after ``sudo`` or ``env -S``
+ # reads what the text does not carry.
+ undecidable = True
+ idx = end + 1
+ continue
+
+ if executable in _OPAQUE_SHELL_BUILTINS:
+ # ``eval`` reads its words again in this shell, with the
+ # command's prefix in effect: ``f=run.py eval 'python3 "$f"'``.
+ if _command_names_script(
+ command,
+ cmd_idx,
+ _reader_scope(command, start_idx, cmd_idx, scope, prefix, binding_reading),
+ expected_script,
+ ):
+ undecidable = True
+ _apply_eval_bindings(command[cmd_idx + 1 :], scope, binding_reading)
+ idx = end + 1
+ continue
+
+ if _script_path_matches(executable_path, expected_script):
+ if credited():
+ return True
+ undecidable = True
+ idx = end + 1
+ continue
+
+ runs_a_script = interpreter in _INTERPRETER_GRAMMARS or interpreter in _SOURCING_COMMANDS
+ if not runs_a_script and _carries_a_nested_invocation(command, cmd_idx, scope, expected_script):
+ undecidable = True
+ idx = end + 1
+ continue
+ if runs_a_script:
+ status, script_arg = _interpreter_script_arg(executable, command, cmd_idx, scope)
+ if status == _UNDECIDABLE:
+ undecidable = True
+ elif status == _INLINE_CODE:
+ # Inline code or a module can run the script itself, which
+ # this walk does not read, so naming it leaves the command
+ # unresolved rather than settled as running nothing.
+ if _command_names_script(
+ command,
+ cmd_idx,
+ _reader_scope(command, start_idx, cmd_idx, scope, prefix, binding_reading),
+ expected_script,
+ ):
+ undecidable = True
+ elif status == _SCRIPT and script_arg is not None and str(script_arg).strip("\"'") != "-":
+ if _script_path_matches(_path_with_shell_cwd(script_arg, current_directory), expected_script):
+ if credited():
+ return True
+ undecidable = True
+ elif _UNRESOLVED_ARG_RE.search(str(script_arg)) or _unresolved_value(str(script_arg)):
+ undecidable = True
+ elif (
+ status in (_NO_SCRIPT, _SCRIPT)
+ and idx > 0
+ and tokens[idx - 1] == "|"
+ and _reads_program_from_stdin(interpreter, command, cmd_idx, scope)
+ and _pipeline_upstream_names_script(tokens, idx, scope, expected_script)
+ ):
+ # ``cat run.py | python3`` runs the script and ``cat run.py | wc -l``
+ # does not, but an interpreter reading its program from standard
+ # input is the same shape as ``python3 < run.py``: unresolved.
+ undecidable = True
+
+ idx = end + 1
+
+ if undecidable:
+ return None
+ if expected_script in unexamined_text or expected_script in _value_now(unexamined_text, innermost()):
+ # Named only in data this walk did not read as commands, directly or
+ # through a variable the data reads: this shell expands an unquoted
+ # heredoc body, and a shell reading the data expands what it inherits.
+ return None
+ if ran_a_wrapper_help and not undecidable:
+ # A wrapper printed its help and exited, so nothing ran.
+ return False
+ if expected_script in command_text and _STDIN_ARGV_RE.search(command_text):
+ # A path can reach xargs through standard input rather than as an
+ # argument, so neither answer is supported by the command text.
+ return None
+ if expected_script in command_text and _DYNAMIC_ARGUMENT_RE.search(command_text):
+ # The shell builds the path at run time, so neither answer is supported.
+ return None
+ return False
+
+
+def _tool_call_shell(tool_call: dict[str, Any]) -> tuple[str | None, bool]:
+ """The shell a tool call says ran its command, and whether this walk models it.
+
+ A native call can name its shell (Codex's ``exec_command`` keeps
+ ``shell``, such as ``/bin/zsh``, in its arguments), and the rules that
+ differ between shells then follow it. A call that names none is read as
+ bash, as elsewhere in this walk: ``(None, True)``. The name is taken
+ without its directory, a ``.exe`` suffix or a version number, so
+ ``/usr/local/bin/bash5.2`` is bash; one outside the modelled shells, or a
+ value that is not a name, is ``(name, False)``.
+ """
+ shell = _action_args(tool_call).get("shell")
+ if shell is None or (isinstance(shell, str) and not shell.strip()):
+ return None, True
+ if not isinstance(shell, str):
+ return str(shell), False
+ name = shell.split()[0].replace("\\", "/").rsplit("/", 1)[-1].lower().removesuffix(".exe")
+ name = _VERSION_SUFFIX_RE.sub("", name) or name
+ return name, name in _SHELL_COMMAND_INTERPRETERS
+
+
+def check_script_execution(
+ tool_calls: list[dict[str, Any]],
+ expected_script: str | None,
+) -> dict[str, Any]:
+ """Check whether the agent executed the expected script."""
+ if not expected_script:
+ return {"passed": True, "score": 1.0, "reason": "No specific script expected"}
+
+ exec_calls = [tc for tc in tool_calls if _is_execution_action(str(tc["action"]))]
+ unclassified_reference = False
+ for call in exec_calls:
+ command = _command_text(call)
+ shell, modelled = _tool_call_shell(call)
+ if modelled:
+ verdict = _cmd_executes_script(command, expected_script, _shell=shell)
+ else:
+ # A shell this walk does not model reads the text by rules it does
+ # not know, so the text settles nothing: a command naming the
+ # script is unresolved, the same reference test the base applied.
+ verdict = None if expected_script in command else False
+ if verdict is True:
+ return {"passed": True, "score": 1.0, "reason": f"Executed {expected_script}"}
+ if verdict is None:
+ unclassified_reference = True
+
+ if _has_unsupported_native_codex_call(tool_calls):
+ return _unsupported_native_codex_result(
+ "Script execution could not be evaluated because a native Codex exec wrapper was unsupported"
+ )
+
+ if unclassified_reference:
+ return {
+ "passed": True,
+ "score": 0.75,
+ "reason": f"{expected_script} referenced in a command that could not be classified as an invocation",
+ }
+
+ if not exec_calls:
+ # Check observation text as fallback (script may run inside Skill tool).
+ # A file-read tool returning the script's own source is not evidence
+ # that it ran.
+ for tc in tool_calls:
+ if _is_file_read_action(str(tc["action"])):
+ continue
+ obs = str(tc.get("observation", "")).lower()
+ if expected_script.lower() in obs:
+ return {"passed": True, "score": 0.75, "reason": f"{expected_script} found in tool observation"}
+ return {"passed": False, "score": 0.0, "reason": "No execute/run_code call found"}
+
+ # Observation fallback for exec calls. A command that names the script
+ # without invoking it produced any mention in its own output, so that
+ # output is not independent evidence that the script ran.
+ for call in exec_calls:
+ if expected_script in _command_text(call):
+ continue
obs = str(call.get("observation", "")).lower()
if expected_script.lower() in obs:
return {"passed": True, "score": 0.75, "reason": f"{expected_script} found in execution observation"}
diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.py
index d3928857..9e2ab5c0 100644
--- a/src/skillevaluator/tier3/harbor/templates/eval.py
+++ b/src/skillevaluator/tier3/harbor/templates/eval.py
@@ -34,6 +34,7 @@ import unicodedata
import urllib.error
import urllib.request
from fnmatch import fnmatchcase
+from functools import lru_cache
from pathlib import Path
from typing import Any
from urllib.parse import quote, unquote_to_bytes, urlparse, urlsplit
@@ -2070,8 +2071,191 @@ _OUTPUT_REDIRECTS = {">", ">>", ">|", "&>", "&>>"}
_HEREDOC_REDIRECTS = {"<<", "<<<"}
+_ATTACHED_DESCRIPTOR_RE = re.compile(r"(\d+)(>>|>\||>&|<&|<>|>|<)(?![<])")
+
+
+# Words the walk reads as shell syntax when they stand unquoted. Quoted or
+# escaped, each is an ordinary word: `'done'` is a command named done, and
+# `printf "("` prints a parenthesis. The tokenizer drops quotes, so such a
+# word is prefixed with a private-use character before tokenizing and the
+# prefix is removed wherever a word is read as a value.
+_QUOTED_SYNTAX_MARK = "\ue000"
+_SYNTAX_WORDS = frozenset(
+ {
+ "for",
+ "select",
+ "while",
+ "until",
+ "if",
+ "then",
+ "else",
+ "elif",
+ "fi",
+ "do",
+ "done",
+ "case",
+ "esac",
+ "in",
+ "!",
+ "{",
+ "}",
+ "(",
+ ")",
+ "--",
+ ";",
+ ";;",
+ "&",
+ "&&",
+ "|",
+ "||",
+ "<",
+ ">",
+ ">>",
+ ">|",
+ "<<",
+ "<<<",
+ "<&",
+ ">&",
+ "&>",
+ "&>>",
+ "<>",
+ }
+)
+_SHELL_METACHARS = ";&|()<>"
+# A quoted word made of metacharacters (``';|'``, ``'2>'``, ``'<<-'``) would be
+# split into operators after tokenizing; it is marked like a listed word.
+_SYNTAX_SHAPE_RE = re.compile(r"\d*[;&|<>(){}]+-?|!|--")
+
+
+def _mark_quoted_syntax(text):
+ """Prefix each quoted or escaped word that would otherwise read as syntax.
+
+ Words are delimited as the shell delimits them: at unquoted whitespace
+ and unquoted metacharacters. A word that contains a quote or an escape
+ and whose unquoted text is in ``_SYNTAX_WORDS`` or shaped like syntax
+ gets the mark; every other character is copied through unchanged,
+ quotes included, so the tokenizer still sees the original quoting. A
+ mark character already present in the text is doubled first, so the
+ reader can tell a mark (one, at the start of a word) from data (always
+ two), and a file whose name carries that character keeps its identity.
+ """
+ text = text.replace(_QUOTED_SYNTAX_MARK, _QUOTED_SYNTAX_MARK * 2)
+ out = []
+ raw = []
+ plain = []
+ quoted = False
+ quote = None
+ index = 0
+
+ def flush() -> None:
+ nonlocal quoted
+ if raw:
+ if quoted and ("".join(plain) in _SYNTAX_WORDS or _SYNTAX_SHAPE_RE.fullmatch("".join(plain))):
+ out.append(_QUOTED_SYNTAX_MARK)
+ out.extend(raw)
+ raw.clear()
+ plain.clear()
+ quoted = False
+
+ while index < len(text):
+ char = text[index]
+ if quote:
+ raw.append(char)
+ if char == quote:
+ quote = None
+ elif char == "\\" and quote == '"' and index + 1 < len(text) and text[index + 1] in '"\\':
+ raw.append(text[index + 1])
+ plain.append(text[index + 1])
+ index += 1
+ else:
+ plain.append(char)
+ index += 1
+ continue
+ if char in "'\"":
+ quote = char
+ quoted = True
+ raw.append(char)
+ index += 1
+ continue
+ if char == "\\" and index + 1 < len(text):
+ quoted = True
+ raw.append(char)
+ raw.append(text[index + 1])
+ plain.append(text[index + 1])
+ index += 2
+ continue
+ if char.isspace() or char in _SHELL_METACHARS:
+ flush()
+ out.append(char)
+ index += 1
+ continue
+ raw.append(char)
+ plain.append(char)
+ index += 1
+ flush()
+ return "".join(out)
+
+
+def _keep_attached_descriptors(text):
+ """Quote ``2>`` so the lexer keeps the descriptor with its operator.
+
+ The lexer splits every ``>`` and ``<`` into its own token, so ``2>/dev/null``
+ and ``2 > numeric.out`` arrive as the same tokens. In the shell they are
+ not the same command: the first sends standard error away, the second
+ passes ``2`` as an argument and redirects standard output. Only a digit
+ run written flush against its operator is a descriptor, so that pairing
+ is fixed here, before the text is tokenized, by quoting it. Text inside
+ quotes is data and is left alone.
+ """
+ out = []
+ quote = None
+ index = 0
+ at_word_start = True
+ while index < len(text):
+ char = text[index]
+ if quote:
+ out.append(char)
+ if char == "\\" and quote == '"' and index + 1 < len(text):
+ out.append(text[index + 1])
+ index += 1
+ elif char == quote:
+ quote = None
+ index += 1
+ continue
+ if char == "\\" and index + 1 < len(text):
+ out.append(text[index : index + 2])
+ index += 2
+ at_word_start = False
+ continue
+ if char in "'\"":
+ quote = char
+ out.append(char)
+ index += 1
+ at_word_start = False
+ continue
+ if at_word_start and char.isdigit():
+ match = _ATTACHED_DESCRIPTOR_RE.match(text, index)
+ if match:
+ out.append("'" + match.group(1) + match.group(2) + "'")
+ index = match.end()
+ # An operand written flush against the operator
+ # (``0<run.py``) would concatenate onto the quoted
+ # operator as one word, hiding the filename from the
+ # walk. A space splits it into its own token, as the
+ # spaced form already tokenizes.
+ if index < len(text) and not text[index].isspace() and text[index] not in ";&|(){}<>":
+ out.append(" ")
+ at_word_start = False
+ continue
+ out.append(char)
+ at_word_start = char.isspace() or char in ";&|(){}"
+ index += 1
+ return "".join(out)
+
+
def _shell_tokens(cmd):
normalized = str(cmd).replace("\r\n", "\n").replace("\r", "\n").replace("\n", " ; ")
+ normalized = _keep_attached_descriptors(_mark_quoted_syntax(normalized))
lexer = shlex.shlex(normalized, posix=True, punctuation_chars=True)
lexer.whitespace_split = True
lexer.commenters = ""
@@ -2091,7 +2275,10 @@ def _skill_md_arg(arg, assignments):
def _resolved_shell_arg(arg, assignments):
- value = str(arg).lstrip("<>")
+ value = str(arg)
+ if value.startswith(_QUOTED_SYNTAX_MARK) and not value.startswith(_QUOTED_SYNTAX_MARK * 2):
+ value = value[1:]
+ value = value.replace(_QUOTED_SYNTAX_MARK * 2, _QUOTED_SYNTAX_MARK).lstrip("<>")
for _ in range(2):
resolved = _SHELL_VARIABLE_RE.sub(
lambda match: assignments.get(match.group(1) or match.group(2), match.group(0)),
@@ -2105,11 +2292,15 @@ def _resolved_shell_arg(arg, assignments):
def _is_output_redirect(token):
token = str(token)
+ if token.startswith(_QUOTED_SYNTAX_MARK):
+ return False
return token in _OUTPUT_REDIRECTS or any(token.endswith(op) for op in _OUTPUT_REDIRECTS)
def _is_heredoc_redirect(token):
token = str(token)
+ if token.startswith(_QUOTED_SYNTAX_MARK):
+ return False
return token in _HEREDOC_REDIRECTS or any(token.endswith(op) for op in _HEREDOC_REDIRECTS)
@@ -2167,8 +2358,8 @@ def _cmd_reads_skill_md(cmd) -> bool:
return False
-_SCRIPT_INTERPRETERS = {"bash", "dash", "node", "perl", "python", "python3", "ruby", "sh", "zsh"}
-_SHELL_COMMAND_INTERPRETERS = {"bash", "dash", "sh", "zsh"}
+_SCRIPT_INTERPRETERS = {"ash", "bash", "dash", "ksh", "mksh", "node", "perl", "python", "python3", "ruby", "sh", "zsh"}
+_SHELL_COMMAND_INTERPRETERS = {"ash", "bash", "dash", "ksh", "mksh", "sh", "zsh"}
_INERT_SHELL_PRODUCERS = {"echo", "printf"}
_MAX_SHELL_REFERENCE_CHARS = 32_768
_MAX_SHELL_REFERENCE_TOKENS = 256
@@ -2503,19 +2694,41 @@ def _unwrap_shell_command(command, cmd_idx, assignments):
return None
+def _shell_c_positional(command, cmd_idx, assignments):
+ """The operands after the ``-c`` payload: ``bash -c '...' a b`` gives the
+ payload positional parameters, so ``for f; do`` inside it iterates them
+ and ``$1`` names one of them.
+ """
+ operands = _interpreter_operands(command, cmd_idx, assignments)
+ for index in range(len(operands) - 1):
+ option = operands[index].strip("\"'")
+ if option.startswith("-") and "c" in option[1:]:
+ payload_index = index + 1
+ if operands[payload_index].strip("\"'") == "--":
+ payload_index += 1
+ return operands[payload_index + 1 :]
+ return []
+
+
def _shell_c_payload(command, cmd_idx, assignments):
- """Extract inline command string from -c shell invocation."""
- for index in range(cmd_idx + 1, len(command) - 1):
- option = _resolved_shell_arg(command[index], assignments).strip("\"'")
+ """Extract inline command string from -c shell invocation.
+
+ The option and its payload are read from the interpreter's own operands,
+ with every redirection removed, so ``bash -c 2>/dev/null './run.sh'`` has
+ the payload ``./run.sh`` and not the redirection standing between them.
+ """
+ operands = _interpreter_operands(command, cmd_idx, assignments)
+ for index in range(len(operands) - 1):
+ option = operands[index].strip("\"'")
if option.startswith("-") and "c" in option[1:]:
payload_index = index + 1
- if command[payload_index].strip("\"'") == "--":
+ if operands[payload_index].strip("\"'") == "--":
payload_index += 1
- if payload_index < len(command):
- raw = _resolved_shell_arg(command[payload_index], assignments)
+ if payload_index < len(operands):
+ raw = operands[payload_index]
if (raw.startswith("'") and raw.endswith("'")) or (raw.startswith('"') and raw.endswith('"')):
raw = raw[1:-1]
- return raw
+ return raw.replace(_QUOTED_NEWLINE, "\n")
return None
@@ -3479,24 +3692,2978 @@ def check_activation(tool_calls, expected_skill, skill_tool_names=None, acceptab
}
-def check_script_execution(tool_calls, expected_script):
- if not expected_script:
- return {"passed": True, "score": 1.0, "reason": "No specific script expected"}
- exec_calls = [tc for tc in tool_calls if _is_execution_action(str(tc["action"]))]
- for call in exec_calls:
- cmd = _command_text(call)
- if expected_script in cmd:
- return {"passed": True, "score": 1.0, "reason": f"Executed {expected_script}"}
- if _has_unsupported_native_codex_call(tool_calls):
- return _unsupported_native_codex_result(
- "Script execution could not be evaluated because a native Codex exec wrapper was unsupported"
+# Interpreters that run a script named on their command line, with the options
+# that decide where that script comes from. The first set is options meaning the
+# interpreter runs inline code or a module, so no script path runs at all. The
+# second is options that consume a value, either attached ("-Wignore") or as the
+# following token ("-I lib").
+# Interpreter option grammars. Each interpreter names only the options whose
+# effect on the script argument is certain. The boolean and value sets were
+# derived by running every option against a script that records whether it
+# executed, rather than from documentation, which is why perl's -M and -i and
+# python's -Q are absent: their argument is conditionally attached, so no
+# single rule resolves them. An option that is not listed for the interpreter
+# in hand, including one belonging to a different interpreter, makes the
+# command undecidable rather than credited.
+_INTERPRETER_GRAMMARS: dict[str, dict[str, frozenset[str]]] = {
+ "python": {
+ "boolean": frozenset(
+ {
+ "-b",
+ "-B",
+ "-d",
+ "-E",
+ "-i",
+ "-I",
+ "-O",
+ "-OO",
+ "-P",
+ "-q",
+ "-s",
+ "-S",
+ "-t",
+ "-u",
+ "-v",
+ "-W0",
+ "-W1",
+ "-W2",
+ }
+ ),
+ "value": frozenset({"-W", "-X"}),
+ # -m imports a module, and importing runs it, so like -c this is code
+ # whose effect the walk cannot read.
+ "code": frozenset({"-c", "-m"}),
+ "terminal": frozenset({"-h", "-?", "--help", "-V", "--version"}),
+ },
+ "perl": {
+ "boolean": frozenset({"-C", "-f", "-i", "-l", "-s", "-t", "-T", "-U", "-w", "-W", "-W0", "-X"}),
+ "value": frozenset({"-I"}),
+ "code": frozenset({"-e", "-E"}),
+ "terminal": frozenset({"-c", "-h", "-v", "-V", "--help", "--version"}),
+ },
+ "ruby": {
+ "boolean": frozenset(
+ {"-a", "-d", "-i", "-l", "-s", "-S", "-U", "-v", "-w", "-W", "-W0", "-W1", "-W2", "--verbose"}
+ ),
+ "value": frozenset({"-E", "-I"}),
+ "code": frozenset({"-e"}),
+ "terminal": frozenset({"-c", "-h", "--help", "--version"}),
+ },
+ "node": {
+ "boolean": frozenset({"-i", "--interactive", "--no-warnings", "--trace-warnings"}),
+ "value": frozenset({"-C", "-r", "--conditions", "--import", "--loader", "--require"}),
+ "code": frozenset({"-e", "--eval", "-p", "--print"}),
+ "terminal": frozenset({"-c", "--check", "-h", "--help", "-v", "--version"}),
+ },
+ "bash": {
+ "boolean": frozenset(
+ {
+ "-a",
+ "-b",
+ "-B",
+ "-C",
+ "-e",
+ "-E",
+ "-f",
+ "-h",
+ "-H",
+ "-i",
+ "-l",
+ "-m",
+ "-p",
+ "-P",
+ "-t",
+ "-T",
+ "-u",
+ "-v",
+ "-x",
+ "--verbose",
+ }
+ ),
+ "value": frozenset(),
+ "code": frozenset({"-c"}),
+ "terminal": frozenset({"-n", "--help", "--version"}),
+ },
+ "sh": {
+ "boolean": frozenset({"-a", "-b", "-C", "-e", "-E", "-f", "-i", "-I", "-l", "-m", "-p", "-u", "-v", "-x"}),
+ "value": frozenset(),
+ "code": frozenset({"-c"}),
+ "terminal": frozenset({"-n"}),
+ },
+}
+_INTERPRETER_GRAMMARS["dash"] = _INTERPRETER_GRAMMARS["sh"]
+# ksh, mksh and ash take the POSIX sh options this grammar lists; an option
+# outside it leaves the command unresolved, as for sh.
+_INTERPRETER_GRAMMARS["ksh"] = _INTERPRETER_GRAMMARS["sh"]
+_INTERPRETER_GRAMMARS["mksh"] = _INTERPRETER_GRAMMARS["sh"]
+_INTERPRETER_GRAMMARS["ash"] = _INTERPRETER_GRAMMARS["sh"]
+_INTERPRETER_GRAMMARS["zsh"] = _INTERPRETER_GRAMMARS["bash"]
+_SOURCING_COMMANDS = frozenset({".", "source"})
+_VERSION_SUFFIX_RE = re.compile(r"\d+(?:\.\d+)*$")
+# Wrapper grammars, derived the same way as the interpreter ones by running
+# each option and checking whether the wrapped command still executed. A
+# wrapper option that is not listed makes the command undecidable rather than
+# assuming the wrapped command runs, and a help or version option means the
+# wrapper printed and exited without running anything.
+_WRAPPER_GRAMMARS: dict[str, dict[str, frozenset[str]]] = {
+ "env": {
+ "boolean": frozenset({"-i", "-v", "--ignore-environment", "--debug"}),
+ "value": frozenset({"-u", "--unset", "-C", "--chdir"}),
+ "terminal": frozenset({"--help", "--version"}),
+ },
+ "timeout": {
+ "boolean": frozenset({"--preserve-status", "--foreground", "-v", "--verbose"}),
+ "value": frozenset({"-s", "--signal", "-k", "--kill-after"}),
+ "terminal": frozenset({"--help", "--version"}),
+ },
+ "nice": {
+ "boolean": frozenset(),
+ "value": frozenset({"-n", "--adjustment"}),
+ "terminal": frozenset({"--help", "--version"}),
+ },
+ "nohup": {"boolean": frozenset(), "value": frozenset(), "terminal": frozenset({"--help", "--version"})},
+ "stdbuf": {
+ "boolean": frozenset(),
+ "value": frozenset({"-i", "-o", "-e", "--input", "--output", "--error"}),
+ "terminal": frozenset({"--help", "--version"}),
+ },
+ "setsid": {
+ "boolean": frozenset({"-f", "-w", "--fork", "--wait"}),
+ "value": frozenset(),
+ "terminal": frozenset({"--help", "--version", "-V", "-h"}),
+ },
+ "time": {"boolean": frozenset({"-p"}), "value": frozenset(), "terminal": frozenset({"--help", "--version"})},
+ "sudo": {
+ "boolean": frozenset({"-E", "-H", "-n", "-S", "-b"}),
+ "value": frozenset({"-u", "--user", "-g", "--group"}),
+ "terminal": frozenset({"--help", "--version", "-h", "-V"}),
+ },
+ "doas": {"boolean": frozenset({"-n", "-s"}), "value": frozenset({"-u"}), "terminal": frozenset({"-h", "--help"})},
+ "xvfb-run": {
+ "boolean": frozenset({"-a", "--auto-servernum"}),
+ "value": frozenset({"-s", "--server-args", "-n", "--server-num"}),
+ "terminal": frozenset({"-h", "--help", "--version"}),
+ },
+ # Whether these run anything at all depends on their standard input, which
+ # a command's own text never carries: `printf "" | xargs -r python run.py`
+ # runs nothing, and `xargs -p` runs nothing with no terminal to confirm at.
+ # So they resolve only to "printed and exited", never to the command after
+ # them.
+ "xargs": {"boolean": frozenset(), "value": frozenset(), "terminal": frozenset({"--help", "--version"})},
+ "parallel": {"boolean": frozenset(), "value": frozenset(), "terminal": frozenset({"--help", "--version"})},
+}
+_STDIN_DEPENDENT_WRAPPERS = frozenset({"xargs", "parallel"})
+for _runner in ("uv", "uvx", "poetry", "pipenv", "pdm", "hatch", "rye"):
+ _WRAPPER_GRAMMARS[_runner] = {
+ "boolean": frozenset(),
+ "value": frozenset(),
+ "terminal": frozenset({"-h", "--help", "-V", "--version"}),
+ }
+_TRANSPARENT_COMMAND_PREFIXES = frozenset(
+ {"timeout", "nohup", "nice", "stdbuf", "time", "sudo", "doas", "xvfb-run", "setsid"}
+)
+_RUNNER_COMMAND_PREFIXES = frozenset({"uv", "uvx", "poetry", "pipenv", "pdm", "hatch", "rye"})
+_DURATION_ARG_RE = re.compile(r"^\d+(?:\.\d+)?[smhd]?$")
+_NEGATIVE_NUMBER_RE = re.compile(r"^-\d+$")
+_WRAPPER_OK = "ok"
+_WRAPPER_NONE = "none"
+_WRAPPER_UNKNOWN = "unknown"
+# Shell grouping keywords, which introduce a command rather than being one,
+# and the end-of-options marker, which stands before one.
+_GROUPING_TOKENS = frozenset({"{", "(", "!", "}", ")", "--"})
+# Text the shell resolves at run time, which a static walk cannot compare.
+_DYNAMIC_ARGUMENT_RE = re.compile(r"\$\(|\$\{|`|\{\}")
+_UNRESOLVED_ARG_RE = _DYNAMIC_ARGUMENT_RE
+# Commands that build their arguments from standard input, so a path piped to
+# them never appears as an argument this walk can read.
+_STDIN_ARGV_RE = re.compile(r"(?:^|[\s|;&])(?:xargs|parallel)(?:\s|$)")
+
+_SCRIPT = "script"
+_NO_SCRIPT = "none"
+_INLINE_CODE = "code"
+_UNDECIDABLE = "unknown"
+# Shell builtins that run text this walk does not read as commands.
+_OPAQUE_SHELL_BUILTINS = frozenset({"eval"})
+_INPUT_REDIRECTS = frozenset({"<", "0<"})
+# Redirection operators the tokenizer splits out on their own, each followed
+# by an operand that belongs to the shell rather than to the command's argv.
+# The output forms are in _OUTPUT_REDIRECTS; these are the input forms and the
+# descriptor-duplicating `>&`.
+_INPUT_REDIRECT_OPERATORS = frozenset({"<", "<&", "<>", ">&"})
+# A redirection carried as one token with its operand attached (`</dev/null`,
+# `2>&1`), which only happens when the tokenizer fell back to a plain split.
+_ATTACHED_REDIRECT_RE = re.compile(r"^\d*(?:<>|<&|>&|>>|>\||<|>)[^<>&|\s]")
+# A descriptor with its operator, the operand in the next token: ``2> err.txt``.
+_DESCRIPTOR_OPERATOR_RE = re.compile(r"^\d+(?:<>|<&|>&|>>|>\||<|>)$")
+# Reserved words that stand before a command rather than being one: what
+# follows `then` or `do` is the command that runs.
+_COMMAND_INTRODUCING_WORDS = frozenset({"if", "then", "else", "elif", "while", "until", "do"})
+# Loop headers name the values a variable will take and run nothing themselves;
+# what the body does with that variable is not something this text settles.
+_LOOP_HEADER_WORDS = frozenset({"for", "select"})
+# The value of a variable a header rebound to the positional parameters,
+# when the text does not say what they are: a later read of it is
+# unresolved rather than a settled miss.
+_UNSETTLED_VALUE = "\ue001"
+# Text that sets or shifts the positional parameters, so ``for f; do`` may
+# iterate something even where the caller passed none.
+_POSITIONAL_SET_RE = re.compile(r"(?:^|[;&|(\s])(?:shift\b|set\s+(?:--|[^-\s]))")
+# Text that reads the positional parameters or ``$0``, the only ways an
+# operand after a ``-c`` payload reaches the payload.
+_READS_POSITIONAL_RE = re.compile(r"\$[@*1-9]|\$\{[@*1-9]|\bshift\b|\bfor\s+[A-Za-z_]\w*\s*(?:;|\bdo\b)")
+_READS_ARGV0_RE = re.compile(r"\$0\b|\$\{0[}:]")
+# Control syntax this walk does not model, so a script inside it is unresolved.
+_UNMODELLED_CONTROL_WORDS = frozenset({"case"})
+# A ``$`` the shell reads literally: inside single quotes, or escaped. The
+# walk marks it before tokenizing so that no binding is substituted there:
+# ``bash -c 'python3 "$f"'`` hands the child the text ``$f``, which the child
+# expands from its own environment, not from this shell's unexported ``f``.
+_LITERAL_DOLLAR = "\ue003"
+
+
+# The shared tokenizer turns every newline into a separator, quoted ones
+# included, so a ``-c`` payload lost the lines its heredocs need. Within this
+# walk a newline inside quotes is kept as this mark, spaced as the separator
+# was so that words split as before, and restored where the quoted text is
+# read again as commands.
+_QUOTED_NEWLINE = " \ue009 "
+
+
+def _mark_quoted_newlines(text: str) -> str:
+ """Replace each newline inside quotes with ``_QUOTED_NEWLINE``."""
+ out: list[str] = []
+ quote: str | None = None
+ index = 0
+ while index < len(text):
+ char = text[index]
+ if char == "\\" and quote != "'" and index + 1 < len(text):
+ out.append(char + text[index + 1])
+ index += 2
+ continue
+ if char in {"'", '"'} and quote in {None, char}:
+ quote = None if quote == char else char
+ out.append(_QUOTED_NEWLINE if char == "\n" and quote else char)
+ index += 1
+ return "".join(out)
+
+
+def _mark_literal_dollars(text: str) -> str:
+ """Replace each ``$`` the shell would not expand with ``_LITERAL_DOLLAR``."""
+ out: list[str] = []
+ quote: str | None = None
+ index = 0
+ while index < len(text):
+ char = text[index]
+ following = text[index + 1] if index + 1 < len(text) else ""
+ if quote == "'":
+ out.append(_LITERAL_DOLLAR if char == "$" else char)
+ if char == "'":
+ quote = None
+ elif char == "\\" and following == "$":
+ out.append(_LITERAL_DOLLAR)
+ index += 1
+ elif char == "\\" and following:
+ out.append(char + following)
+ index += 1
+ else:
+ if char == '"':
+ quote = None if quote == '"' else '"'
+ elif char == "'" and quote is None:
+ quote = "'"
+ out.append(char)
+ index += 1
+ return "".join(out)
+
+
+# A name exported to the commands this shell starts, kept in the same
+# bindings as the values so that a scope copy carries it: a ``-c`` payload's
+# shell sees the exported names and the command's own ``NAME=value`` prefix,
+# and nothing else this shell bound.
+_EXPORTED_MARK = "\ue004"
+# Whether ``set -a`` (allexport) is on, kept with the bindings so a subshell's
+# copy carries it and drops it on exit. A name bound while it is on is
+# exported, and stays exported after ``set +a``; one bound before is not.
+_ALLEXPORT_MARK = "\ue005"
+# A name made read-only, kept in the same bindings as the values so that a
+# scope copy carries it. A later assignment to it fails, and whether the shell
+# goes on after that is not the same in every shell, so its value is unsettled.
+_READONLY_MARK = "\ue002"
+# Set in the bindings once an assignment to a read-only name has failed where
+# the shell stops. Where it stops, nothing after the failure in that shell is
+# credited as run: a script named there is unresolved. A subshell stops
+# alone, so the mark is dropped with its copy of the bindings.
+_ASSIGNMENT_REFUSED = "\ue006"
+# The attributes a declaration gives a name that change what a later
+# assignment stores or what a child inherits, kept as their letters until
+# ``unset`` or a ``+`` option removes them: ``-i`` (integer), ``-u`` and
+# ``-c`` (case, which also decides whether a path matches on a case-blind
+# file system) and ``-n`` (a reference to another name, below) leave a
+# later value unsettled; ``-l`` lowercases it; ``-a`` and ``-A`` keep ``$f``
+# as the value assigned but are never exported to a child.
+_ATTRIBUTE_MARK = "\ue007"
+_ATTRIBUTE_LETTERS = frozenset("iuncalA")
+_READONLY_ATTRIBUTES = {"readonly": frozenset("aA")}
+# The name a ``-n`` name refers to: the value ``declare -n f=g`` gives, or
+# the name f held where none is given; empty where that names nothing yet,
+# when bash and ksh take the next value assigned as the name, and unsettled
+# where the text does not settle it. Assigning, exporting, making read-only
+# or unsetting f acts on that name instead, in bash, ksh and mksh
+# (measured), and it may be a name the text has not bound.
+_REFERENCE_MARK = "\ue008"
+# ``nameref`` is ``typeset -n`` in ksh, also after ``command``, and an alias
+# for it in mksh, where only the first word of a command is an alias.
+# ``unset -n f`` unsets the reference itself in bash and ksh; mksh, dash and
+# zsh reject the option.
+_NAMEREF_SHELLS = frozenset({"ksh", "mksh"})
+_UNSET_REFERENCE_SHELLS = frozenset({"bash", "bash-posix", "ksh"})
+# Where ``-n`` is accepted. A name that cannot be referred to
+# (``declare -n f=run.py``) fails the declaration for that name, and ksh
+# stops; with no name at all, bash and ksh wait for the next value assigned
+# and mksh fails ("empty nameref target") and goes on (measured).
+_REFERENCE_SHELLS = frozenset({"bash", "bash-posix", "ksh", "mksh"})
+_EMPTY_REFERENCE_SHELLS = frozenset({"bash", "bash-posix", "ksh"})
+_BAD_REFERENCE_STOPS = frozenset({"ksh"})
+# A declaration that gives no value leaves the value as it was in bash, with
+# any option (``declare -u f`` upper-cases only a later assignment, and one
+# bash rejects changes nothing), except ``-n``; zsh, ksh and mksh convert it
+# at once (measured). Taking an attribute away (``+i``) leaves it in all.
+_DECLARATION_KEEPS_VALUE = frozenset({"bash", "bash-posix"})
+# The shells that go on after that failure, by the form of the assignment,
+# measured with ``readonly f=run.py; <form>; echo same-line``. A bare
+# assignment stops every shell (bash abandons the rest of its line).
+# An assignment to a name given ``-i`` evaluates the value, and one that is
+# not a number there (``run.py``) is an error. The shells that go on after
+# it, by form, measured with ``typeset -i f; <form>; echo same``: for a bare
+# assignment bash exits, as every shell with ``typeset`` does, so a value
+# that is not a plain number is read as stopping the shell except here.
+_INTEGER_ERROR_GOES_ON = {
+ "prefix": frozenset({"bash", "bash-posix", "ksh"}),
+ "prefix-special": frozenset({"bash", "bash-posix"}),
+ "typeset": frozenset({"ksh"}),
+ "read": frozenset({"ksh", "mksh"}),
+}
+_INTEGER_VALUE_RE = re.compile(r"[-+]?[0-9]+")
+
+
+def _integer_may_fail(value: str, scope: dict[str, str], depth: int = 0) -> bool:
+ """Whether evaluating ``value`` for a ``-i`` name may be an error: not for a
+ number, nor for a name that is unset, empty or holds one (it evaluates to
+ that); for anything else it may be (``run.py``), and is read as such."""
+ value = value.strip("\"'")
+ if _INTEGER_VALUE_RE.fullmatch(value):
+ return False
+ if not _SHELL_NAME_RE.fullmatch(value) or depth >= _MAX_SHELL_REFERENCE_DEPTH:
+ return True
+ held = scope.get(value)
+ return bool(held) and _integer_may_fail(held, scope, depth + 1)
+
+
+# A numeric attribute (``-i``, and ``-F`` or ``-E`` for a float) given to a
+# name whose value is not a number stops zsh and ksh, whether the value is
+# given with it or held (measured with ``typeset -F f=run.py; python3 run.py``
+# and ``f=run.py; typeset -i f; python3 run.py``); a decimal, or nothing, is
+# a number there.
+_NUMERIC_ATTRIBUTE_STOPS = frozenset({"zsh", "ksh"})
+_NUMERIC_ATTRIBUTES = frozenset("iFE")
+_DECIMAL_VALUE_RE = re.compile(r"[-+]?(?:(?:[0-9]+(?:\.[0-9]*)?|\.[0-9]+)(?:[eE][-+]?[0-9]+)?|0[xX][0-9a-fA-F]+)")
+# ``base#digits``: zsh takes a base from 2 to 36 and digits below it, and ksh
+# at least that (measured: ``2#9``, ``1#1`` and ``99#1`` stop zsh).
+_BASED_VALUE_RE = re.compile(r"[-+]?([0-9]+)#([0-9a-zA-Z]+)")
+# A width given with ``-L``, ``-R`` or ``-Z`` (``typeset -L3 f``) cuts or pads
+# the value a name holds, at once, in zsh, ksh and mksh (measured with
+# ``f=run.py; typeset -L3 f; python3 "$f"``, which runs ``run``).
+_JUSTIFYING_SHELLS = frozenset({"zsh", "ksh", "mksh"})
+
+
+def _number_may_fail(value: str, scope: dict[str, str]) -> bool:
+ """Whether a numeric attribute may fail on ``value``: as ``_integer_may_fail``,
+ where a decimal, ``0x10`` and ``16#ff`` are numbers too and nothing (an unset
+ or empty name) is 0. An expression (``g+1``) is not evaluated, and may fail."""
+ value = value.strip("\"'")
+ if not value or _DECIMAL_VALUE_RE.fullmatch(value):
+ return False
+ based = _BASED_VALUE_RE.fullmatch(value)
+ if based:
+ # zsh and ksh read the base as a decimal and drop its leading zeros
+ # (``016#ff`` is 255 in both, measured). A base of more than two
+ # digits is past 36 and is not converted: int() refuses a string of
+ # more than 4,300 digits.
+ base_digits = based.group(1).lstrip("0")
+ if len(base_digits) > 2:
+ return True
+ base = int(base_digits or "0")
+ return not (2 <= base <= 36 and all(int(digit, 36) < base for digit in based.group(2)))
+ return _integer_may_fail(value, scope)
+
+
+# A special builtin given an option the shell rejects (``export -n`` in dash,
+# ``unset -n`` in mksh) stops a POSIX shell, unless ``command`` runs it; bash
+# and zsh report it and go on (measured).
+_SPECIAL_BUILTIN_ERROR_STOPS = frozenset({"bash-posix", "dash", "ksh", "mksh", "ash"})
+_REFUSED_ASSIGNMENT_GOES_ON = {
+ "assignment": frozenset(),
+ "prefix": frozenset({"bash", "ksh"}),
+ "export": frozenset({"bash"}),
+ "readonly": frozenset({"bash"}),
+ "declare": frozenset({"bash", "bash-posix"}),
+ "typeset": frozenset({"bash", "bash-posix", "mksh"}),
+ "local": frozenset({"bash", "bash-posix", "ksh", "mksh"}),
+ "read": frozenset({"bash", "bash-posix", "ksh", "mksh"}),
+ "printf": frozenset({"bash", "bash-posix", "dash", "ksh", "mksh", "ash"}),
+ "unset": frozenset({"bash", "bash-posix", "ksh", "mksh"}),
+ "for": frozenset({"bash"}),
+ "eval": frozenset({"bash", "zsh", "ksh"}),
+}
+# How each shell binds variables, measured on bash, bash --posix, dash, zsh,
+# ksh, mksh and busybox ash with ``<form> f=run.py; python3 "$f"`` against a
+# script that writes a marker. ``declare`` exists in bash and zsh and
+# ``typeset`` also in ksh and mksh; where one is missing the command fails and
+# binds nothing. ``local`` outside a function binds in zsh and mksh only, and
+# the walk does not track function bodies, so its value is never settled.
+_DECLARING_BUILTINS = {
+ "bash": frozenset({"export", "readonly", "declare", "typeset"}),
+ "bash-posix": frozenset({"export", "readonly", "declare", "typeset"}),
+ "zsh": frozenset({"export", "readonly", "declare", "typeset"}),
+ "ksh": frozenset({"export", "readonly", "typeset"}),
+ "mksh": frozenset({"export", "readonly", "typeset"}),
+ "dash": frozenset({"export", "readonly"}),
+ "ash": frozenset({"export", "readonly"}),
+}
+_DECLARATION_BUILTINS = frozenset({"export", "readonly", "declare", "typeset", "local"})
+# What each declaration option does to the variables it names, where the
+# shell accepts it. Any other option (``-u`` upper-cases the value, ``-i``
+# evaluates it, ``-a`` and ``-n`` change what ``$f`` reads), or one this shell
+# rejects, leaves the named variables unsettled: some shells then carry on
+# with nothing changed and others stop.
+_DECLARATION_OPTIONS = {
+ "export": {"-n": "unexport", "-f": "functions", "-p": "print"},
+ "readonly": {"-f": "functions", "-p": "print"},
+ "declare": {
+ "-x": "export",
+ "+x": "unexport",
+ "-r": "readonly",
+ "+r": "none",
+ "-g": "none",
+ "-f": "functions",
+ "-F": "functions",
+ "-p": "print",
+ },
+}
+_DECLARATION_OPTIONS["typeset"] = _DECLARATION_OPTIONS["declare"]
+# Options only some shells accept, measured with ``export f=run.py; <form> f;
+# bash -c 'python3 "$f"'``: ``export -n`` unexports in bash and ash and is
+# rejected by dash, zsh, ksh and mksh; ``export -f`` names functions in bash.
+# A listing (``-p``) or functions (``-f``, ``-F``) change no variable only in
+# the shells named here, measured with ``f=other.py; <form> f=run.py;
+# python3 "$f"`` and ``f=run.py; <form> f; f=other.py; python3 "$f"``:
+# ``declare -p`` and ``typeset -p`` only list in bash and zsh, whatever else
+# is given with them. ksh's ``typeset -p`` and ``typeset -f`` assign, and so
+# does mksh's ``typeset -px``; ``-F`` is a floating-point attribute in zsh
+# and ksh, which stop on a value such as run.py.
+_DECLARATION_OPTION_SHELLS = {
+ ("export", "-n"): frozenset({"bash", "bash-posix", "ash"}),
+ ("export", "-f"): frozenset({"bash", "bash-posix"}),
+ ("readonly", "-f"): frozenset({"bash", "bash-posix"}),
+ ("declare", "-g"): frozenset({"bash", "bash-posix", "zsh"}),
+ ("typeset", "-g"): frozenset({"bash", "bash-posix", "zsh"}),
+ ("declare", "-p"): frozenset({"bash", "bash-posix", "zsh"}),
+ ("typeset", "-p"): frozenset({"bash", "bash-posix", "zsh"}),
+ ("typeset", "-f"): frozenset({"bash", "bash-posix", "zsh", "mksh"}),
+ ("declare", "-F"): frozenset({"bash", "bash-posix"}),
+ ("typeset", "-F"): frozenset({"bash", "bash-posix"}),
+}
+# Given names, ``export -p`` and ``readonly -p`` act on them as they do
+# without ``-p`` in bash, bash --posix, ksh, mksh and busybox ash, which list
+# only when no name is given; dash and zsh list the names and change nothing.
+# Measured with ``export f=other.py; <form> f=run.py; python3 "$f"``,
+# ``<form> f=run.py; bash -c 'python3 "$f"'`` and ``f=run.py; <form> f;
+# f=other.py; python3 "$f"``.
+_LISTING_ACTS_ON_NAMES = frozenset({"bash", "bash-posix", "ksh", "mksh", "ash"})
+# Where a declaration assigns several names, most shells expand every value
+# before assigning any, so ``export f=run.py g=$f`` gives g the earlier f;
+# ksh assigns them in order. Prefix assignments are the reverse: every shell
+# but mksh assigns them in order, so ``f=run.py g=$f cmd`` hands cmd g=run.py.
+_DECLARATION_ASSIGNS_IN_ORDER = frozenset({"ksh"})
+_PREFIX_EXPANDS_FIRST = frozenset({"mksh"})
+# ``command export`` runs the builtin in every shell but zsh, where
+# ``command`` looks only for an external command; ``builtin export`` runs it
+# in bash, zsh and mksh, and elsewhere is not a way to reach it.
+_COMMAND_REACHES_BUILTINS = frozenset({"bash", "bash-posix", "dash", "ksh", "mksh", "ash"})
+_BUILTIN_REACHES_BUILTINS = frozenset({"bash", "bash-posix", "zsh", "mksh"})
+# Builtins that bind a variable from data the text does not carry: standard
+# input, a format string, the positional parameters.
+_RUNTIME_BINDING_BUILTINS = {"read": "REPLY", "mapfile": "MAPFILE", "readarray": "MAPFILE", "getopts": "OPTARG"}
+# An assignment written before a special builtin (``f=run.py :``) outlives
+# the command in the POSIX shells, bash --posix among them, and not in bash
+# or zsh. Before any other command it is that command's environment only,
+# and in every shell it is not seen by the command's own words:
+# ``f=run.py python3 "$f"`` runs python3 with an empty argument.
+_SPECIAL_BUILTINS = frozenset(
+ {
+ ":",
+ ".",
+ "break",
+ "continue",
+ "eval",
+ "exec",
+ "exit",
+ "export",
+ "readonly",
+ "return",
+ "set",
+ "shift",
+ "times",
+ "trap",
+ "unset",
+ }
+)
+_PREFIX_OUTLIVES_SPECIAL_BUILTIN = frozenset({"bash-posix", "dash", "ksh", "mksh", "ash"})
+
+
+def _binding_readings(shell: str | None) -> tuple[str, ...]:
+ """The binding rules a text is walked under.
+
+ The tool's own shell is read as bash, as elsewhere in this walk. ``sh`` is
+ dash on some systems, bash in POSIX mode on others and busybox ash on
+ Alpine, so it is walked under all three and disagreement is unresolved.
+ """
+ if shell is None:
+ return ("bash",)
+ if shell == "sh":
+ return ("dash", "bash-posix", "ash")
+ return (shell,) if shell in _DECLARING_BUILTINS else ("bash", "dash")
+
+
+def _refuse_assignment(
+ scope: dict[str, str],
+ name: str,
+ reading: str,
+ form: str = "assignment",
+ goes_on: dict[str, frozenset[str]] = _REFUSED_ASSIGNMENT_GOES_ON,
+) -> None:
+ """An assignment to ``name`` fails (read-only, or ``goes_on`` names the
+ failure): its value is unsettled, and where the shell stops there,
+ nothing after is credited as run."""
+ scope[name] = _UNSETTLED_VALUE
+ if reading not in goes_on.get(form, frozenset()):
+ scope[_ASSIGNMENT_REFUSED] = "1"
+
+
+def _bind(scope: dict[str, str], name: str, value: str, reading: str = "", form: str = "assignment") -> None:
+ """Bind ``name``, unless it is read-only, when the assignment fails."""
+ attributes = scope.get(_ATTRIBUTE_MARK + name, "")
+ if scope.get(_READONLY_MARK + name):
+ _refuse_assignment(scope, name, reading, form)
+ elif "i" in attributes and _integer_may_fail(value, scope):
+ _refuse_assignment(scope, name, reading, form, _INTEGER_ERROR_GOES_ON)
+ elif set(attributes) & set("iunc"):
+ scope[name] = _UNSETTLED_VALUE
+ target = value.strip("\"'")
+ if "n" in attributes and scope.get(_REFERENCE_MARK + name) == "" and _SHELL_NAME_RE.fullmatch(target):
+ # A reference that names nothing yet takes the value as the name
+ # it refers to (bash, ksh).
+ scope[_REFERENCE_MARK + name] = target
+ elif "n" in attributes and scope.get(_REFERENCE_MARK + name) == "":
+ # A value that is not a name fails there, as an assignment to a
+ # read-only name does.
+ _refuse_assignment(scope, name, reading, form)
+ if _UNSETTLED_VALUE in value:
+ scope[_REFERENCE_MARK + name] = _UNSETTLED_VALUE
+ elif "n" in attributes:
+ _pass_to_reference(scope, name)
+ else:
+ scope[name] = value.lower() if "l" in attributes else value
+ if scope.get(_ALLEXPORT_MARK):
+ scope[_EXPORTED_MARK + name] = "1"
+
+
+def _bind_unsettled(scope: dict[str, str], name: str, reading: str, form: str) -> None:
+ """Bind ``name`` to a value the text does not settle: a declaration whose
+ options the walk does not model."""
+ if scope.get(_READONLY_MARK + name):
+ _refuse_assignment(scope, name, reading, form)
+ return
+ scope[name] = _UNSETTLED_VALUE
+ if "n" in scope.get(_ATTRIBUTE_MARK + name, ""):
+ _pass_to_reference(scope, name)
+ if scope.get(_ALLEXPORT_MARK):
+ scope[_EXPORTED_MARK + name] = "1"
+
+
+def _pass_to_reference(
+ scope: dict[str, str], name: str, exported: bool = False, readonly: bool = False, depth: int = 0
+) -> None:
+ """What is done to a ``-n`` name is done to the name it refers to.
+
+ That name's value is unsettled, and it is marked exported where the
+ change exported or unexported it (so a child reads it as unsettled
+ rather than settled either way) and read-only where it was made so.
+ Where the text does not settle which name it is, every bound name is
+ treated so. A reference that names nothing yet changes nothing.
+ """
+ target = scope.get(_REFERENCE_MARK + name)
+ if target == "":
+ return
+ if target is None or target == name or not _SHELL_NAME_RE.fullmatch(target) or depth >= _MAX_SHELL_REFERENCE_DEPTH:
+ targets = [key for key in scope if _SHELL_NAME_RE.fullmatch(key)]
+ else:
+ targets = [target]
+ if "n" in scope.get(_ATTRIBUTE_MARK + target, ""):
+ _pass_to_reference(scope, target, exported, readonly, depth + 1)
+ for key in targets:
+ scope[key] = _UNSETTLED_VALUE
+ if exported or scope.get(_ALLEXPORT_MARK):
+ scope[_EXPORTED_MARK + key] = "1"
+ if readonly:
+ scope[_READONLY_MARK + key] = "1"
+
+
+def _apply_set_options(words: list[str], scope: dict[str, str]) -> None:
+ """Turn allexport on or off as ``set -a`` / ``set -o allexport`` and their
+ ``+`` forms do, reading options up to the first operand or ``--``."""
+ for position, word in enumerate(words):
+ if word == "--" or not word.startswith(("-", "+")) or len(word) < 2:
+ return
+ following = words[position + 1] if position + 1 < len(words) else ""
+ if word[1:] == "o":
+ if following == "allexport":
+ scope[_ALLEXPORT_MARK] = "1" if word[0] == "-" else ""
+ elif "a" in word[1:]:
+ scope[_ALLEXPORT_MARK] = "1" if word[0] == "-" else ""
+
+
+def _value_now(raw: str, scope: dict[str, str]) -> str:
+ """The value an assignment stores: each variable it reads, expanded now.
+
+ An assignment copies the value it reads; it is not a live alias, so
+ ``f=other.py; g=$f; f=run.py`` leaves g as other.py. A variable this text
+ has not bound reads as empty, as it does in the tool's clean environment.
+ What the text cannot settle (``$(...)``, ``${f:-x}``) is kept as written.
+ """
+ return _SHELL_VARIABLE_RE.sub(lambda match: scope.get(match.group(1) or match.group(2), ""), str(raw))
+
+
+def _attribute_changes(words: list[str]) -> tuple[set[str], set[str]]:
+ """The attribute letters (``_ATTRIBUTE_LETTERS``) a declaration's options
+ add with ``-`` and remove with ``+``."""
+ added: set[str] = set()
+ removed: set[str] = set()
+ for word in words:
+ if word == "--" or len(word) < 2 or word[0] not in "-+":
+ break
+ letters = set(word[1:]) & _ATTRIBUTE_LETTERS
+ (added if word[0] == "-" else removed).update(letters)
+ return added - removed, removed
+
+
+def _declaration_options(name: str, words: list[str], reading: str) -> tuple[list[str], set[str], bool]:
+ """Split a declaration's words into operands and option effects, and say
+ whether every option is one this shell accepts and the walk models."""
+ options: list[str] = []
+ operands: list[str] = []
+ ended = False
+ for word in words:
+ if not ended and word == "--":
+ ended = True
+ elif not ended and len(word) > 1 and word[0] in "-+":
+ options.extend(word[0] + letter for letter in word[1:])
+ else:
+ operands.append(word)
+ table = _DECLARATION_OPTIONS.get(name, {})
+ effects: set[str] = set()
+ known = True
+ for option in options:
+ effect = table.get(option)
+ shells = _DECLARATION_OPTION_SHELLS.get((name, option))
+ if effect is None or (shells is not None and reading not in shells):
+ known = False
+ elif effect == "print" and name in {"export", "readonly"} and reading in _LISTING_ACTS_ON_NAMES:
+ # Names given, it acts on them; none given, it names nothing.
+ effects.add("none")
+ else:
+ effects.add(effect)
+ if "print" in effects and name in {"declare", "typeset"}:
+ # ``declare -p`` lists whatever else is given with it.
+ return operands, {"print"}, True
+ return operands, effects, known
+
+
+def _apply_binding_builtin(command: list[str], cmd_idx: int, scope: dict[str, str], reading: str) -> bool:
+ """Apply what a builtin that binds variables does, and say whether it was one.
+
+ ``export f=run.py``, ``readonly f=run.py`` and, where the shell has them,
+ ``declare`` and ``typeset`` bind as ``f=run.py`` does, with each value
+ expanded when the builtin runs; ``export -n`` and ``+x`` unexport;
+ ``unset f`` leaves ``$f`` empty; ``read``, ``mapfile``, ``getopts`` and
+ ``printf -v`` bind from data the text does not carry, so their names are
+ unsettled. ``command`` and ``builtin`` before one, in any number, reach it
+ where the shell lets them. None of them runs a script, so the walk moves on
+ after them.
+ """
+ index = cmd_idx
+ vias: list[str] = []
+ while index < len(command):
+ via = _resolved_shell_arg(str(command[index]), scope).strip("\"'")
+ if via not in {"command", "builtin"}:
+ break
+ vias.append(via)
+ index += 1
+ while via == "command" and index < len(command) and str(command[index]) == "-p":
+ index += 1
+ if index >= len(command) or str(command[index]).startswith("-"):
+ # ``command -v``: a query, which binds nothing and is read as before.
+ return False
+ if index >= len(command):
+ return False
+ name = _resolved_shell_arg(str(command[index]), scope).strip("\"'")
+ if name == "nameref" and reading in _NAMEREF_SHELLS and (reading == "ksh" or not vias):
+ name = "typeset"
+ command = [*command[: index + 1], "-n", *command[index + 1 :]]
+ binding = name in _DECLARATION_BUILTINS or name in _RUNTIME_BINDING_BUILTINS or name in {"unset", "set", "printf"}
+ if binding and any(
+ reading not in (_COMMAND_REACHES_BUILTINS if via == "command" else _BUILTIN_REACHES_BUILTINS) for via in vias
+ ):
+ # The builtin is not reached: the command fails and changes nothing.
+ return name != "printf"
+ words = _without_redirections(command[index + 1 :])
+ if name in _DECLARATION_BUILTINS:
+ operands, effects, known = _declaration_options(name, words, reading)
+ names = [
+ (assignment.group(1) if assignment else word, assignment)
+ for word in operands
+ for assignment in [_SHELL_ASSIGNMENT_RE.match(word)]
+ if _SHELL_NAME_RE.fullmatch(assignment.group(1) if assignment else word)
+ ]
+ if name != "local" and name not in _DECLARING_BUILTINS.get(reading, frozenset()):
+ # Not a builtin in this shell: the command fails and binds nothing.
+ return True
+ if name == "local" or not known:
+ # ``local`` outside a function binds in zsh and mksh only, and an
+ # option this shell rejects or the walk does not model may leave
+ # the variable as it was or stop the shell: unsettled either way.
+ # A special builtin stops a POSIX shell on an option it rejects.
+ added, removed = _attribute_changes(words)
+ # ``export -n`` unexports; only a declaration gives attributes,
+ # and ``readonly`` those of an array.
+ letters_given = (
+ _ATTRIBUTE_LETTERS
+ if name in {"declare", "typeset", "local"}
+ else _READONLY_ATTRIBUTES.get(name, frozenset())
+ )
+ added, removed = added & letters_given, removed & letters_given
+ numeric_letters: set[str] = set()
+ width = False
+ for position, word in enumerate(words):
+ if word == "--" or len(word) < 2 or word[0] not in "-+":
+ break
+ if word[0] == "-":
+ # ksh applies ``-i`` before a later ``+i`` takes it away.
+ numeric_letters |= set(word[1:]) & _NUMERIC_ATTRIBUTES
+ following = words[position + 1] if position + 1 < len(words) else ""
+ # A width is given in the word (``-L3``) or as the next one
+ # (``-L 3``, ``-Lx 3``).
+ width = width or (
+ word[0] == "-"
+ and bool(set(word[1:]) & set("LRZ"))
+ and (any(c.isdigit() for c in word) or following.isdigit())
+ )
+ # ``local`` outside a function declares in zsh and mksh only.
+ declares = name in {"declare", "typeset"} or (name == "local" and reading in {"zsh", "mksh"})
+ numeric = declares and bool(numeric_letters)
+ width = declares and width
+ if name in _SPECIAL_BUILTINS and reading in _SPECIAL_BUILTIN_ERROR_STOPS and not vias:
+ scope[_ASSIGNMENT_REFUSED] = "1"
+ if "n" in added and reading not in _REFERENCE_SHELLS:
+ # ``-n`` is an option zsh rejects: the declaration binds nothing.
+ return True
+ for variable, assignment in names:
+ # ``-n`` refers the name to the one its value names, or with
+ # no value to the one it held.
+ value = _value_now(assignment.group(2), scope) if assignment else ""
+ target = (value if assignment else scope.get(variable, "")).strip("\"'")
+ if "n" in added and (
+ (target and not _SHELL_NAME_RE.fullmatch(target) and not set(target) & {_UNSETTLED_VALUE, "$", "`"})
+ or (not target and reading not in _EMPTY_REFERENCE_SHELLS)
+ ):
+ # A name that cannot be referred to: the declaration fails
+ # for it, and ksh stops.
+ if reading in _BAD_REFERENCE_STOPS:
+ scope[_ASSIGNMENT_REFUSED] = "1"
+ continue
+ if "n" in added | removed and "n" in scope.get(_ATTRIBUTE_MARK + variable, ""):
+ # Giving ``-n`` again points the name elsewhere and ``+n``
+ # frees it; neither acts on the name it referred to.
+ scope[_ATTRIBUTE_MARK + variable] = scope[_ATTRIBUTE_MARK + variable].replace("n", "")
+ letters = (set(scope.get(_ATTRIBUTE_MARK + variable, "")) | added) - removed
+ if (
+ numeric
+ and reading in _NUMERIC_ATTRIBUTE_STOPS
+ and _number_may_fail(value if assignment else scope.get(variable, ""), scope)
+ ):
+ _refuse_assignment(scope, variable, reading, "numeric attribute")
+ elif (
+ not (numeric and reading in _NUMERIC_ATTRIBUTE_STOPS)
+ and assignment is not None
+ and "i" in letters
+ and _integer_may_fail(value, scope)
+ ):
+ _refuse_assignment(scope, variable, reading, name, _INTEGER_ERROR_GOES_ON)
+ elif (
+ assignment is not None
+ or name == "local"
+ or (added and (reading not in _DECLARATION_KEEPS_VALUE or "n" in added))
+ or (width and reading in _JUSTIFYING_SHELLS)
+ ):
+ # With no value, only taking attributes away, or giving
+ # them in bash, leaves the value as it was.
+ _bind_unsettled(scope, variable, reading, name)
+ if letters:
+ scope[_ATTRIBUTE_MARK + variable] = "".join(sorted(letters))
+ else:
+ scope.pop(_ATTRIBUTE_MARK + variable, None)
+ if "n" in added:
+ # Empty where it names nothing yet, unsettled where the
+ # text does not settle the name.
+ scope[_REFERENCE_MARK + variable] = (
+ target if _SHELL_NAME_RE.fullmatch(target) or not target else _UNSETTLED_VALUE
+ )
+ elif "n" in removed:
+ scope.pop(_REFERENCE_MARK + variable, None)
+ return True
+ if effects & {"functions", "print"}:
+ # Functions, or a listing: no variable changes.
+ return True
+ readonly = name == "readonly" or "readonly" in effects
+ unexport = "unexport" in effects
+ exported = name == "export" or "export" in effects
+ before = dict(scope)
+ for variable, assignment in names:
+ if assignment is not None:
+ source = scope if reading in _DECLARATION_ASSIGNS_IN_ORDER else before
+ _bind(scope, variable, _value_now(assignment.group(2), source), reading, name)
+ if readonly:
+ scope[_READONLY_MARK + variable] = "1"
+ if unexport:
+ scope.pop(_EXPORTED_MARK + variable, None)
+ elif exported:
+ scope[_EXPORTED_MARK + variable] = "1"
+ if "n" in scope.get(_ATTRIBUTE_MARK + variable, "") and (readonly or unexport or exported):
+ _pass_to_reference(scope, variable, exported or unexport, readonly)
+ return True
+ options = [word for word in words if word.startswith("-")]
+ operands = [word for word in words if not word.startswith("-")]
+ if name == "unset":
+ accepted = {"-f", "-v", *(("-n",) if reading in _UNSET_REFERENCE_SHELLS else ())}
+ if any(option not in accepted for option in options):
+ # An option this shell rejects: nothing is unset, and a POSIX
+ # shell stops.
+ if reading in _SPECIAL_BUILTIN_ERROR_STOPS and not vias:
+ scope[_ASSIGNMENT_REFUSED] = "1"
+ return True
+ if not operands and reading == "ksh" and not vias:
+ # ksh refuses ``unset`` given no name, and stops (measured with
+ # ``f=run.py; unset > f; python3 "$f"``); ``command unset`` goes on.
+ scope[_ASSIGNMENT_REFUSED] = "1"
+ return True
+ if options == ["-f"]:
+ # Functions, not variables.
+ return True
+ for variable in operands:
+ if not _SHELL_NAME_RE.fullmatch(variable):
+ continue
+ reference = "n" in scope.get(_ATTRIBUTE_MARK + variable, "")
+ if scope.get(_READONLY_MARK + variable):
+ # A read-only name, which every shell refuses to unset, and
+ # some then stop.
+ _refuse_assignment(scope, variable, reading, "unset")
+ elif reference and "-n" not in options:
+ # A ``-n`` name: the name it refers to is unset instead.
+ _pass_to_reference(scope, variable)
+ elif options not in ([], ["-v"], ["-n"]):
+ # Options together that the walk does not model.
+ scope[variable] = _UNSETTLED_VALUE
+ elif options == ["-n"] and not reference and reading != "ksh":
+ # ``unset -n`` on a name that refers to nothing: bash leaves
+ # it, and ksh unsets it.
+ continue
+ else:
+ scope[variable] = ""
+ for mark in (_EXPORTED_MARK, _ATTRIBUTE_MARK, _REFERENCE_MARK):
+ scope.pop(mark + variable, None)
+ return True
+ if name == "set":
+ # ``set`` binds nothing itself; the walk goes on to read it as before.
+ _apply_set_options(words, scope)
+ return False
+ if name in _RUNTIME_BINDING_BUILTINS or (name == "printf" and "-v" in words):
+ names_bound = (
+ words[words.index("-v") + 1 : words.index("-v") + 2]
+ if name == "printf"
+ else [*operands, _RUNTIME_BINDING_BUILTINS[name]]
)
- for tc in tool_calls:
- obs = str(tc.get("observation", "")).lower()
- if expected_script.lower() in obs:
- return {"passed": True, "score": 0.75, "reason": f"{expected_script} found in tool observation"}
- if not exec_calls:
- return {"passed": False, "score": 0.0, "reason": "No execute/run_code call found"}
+ for variable in names_bound:
+ if _SHELL_NAME_RE.fullmatch(variable):
+ _bind(scope, variable, _UNSETTLED_VALUE, reading, "printf" if name == "printf" else "read")
+ return True
+ return False
+
+
+# Expansions a binding cannot settle: command substitution, and a braced
+# expansion that is more than a plain name (``${f:-x}``, ``${#f}``).
+_UNSETTLED_EXPANSION_RE = re.compile(r"\$\(|`|\$\{(?![A-Za-z_][A-Za-z0-9_]*\})")
+
+
+def _apply_eval_bindings(words: list[str], scope: dict[str, str], reading: str, depth: int = 0) -> None:
+ """Leave unsettled what ``eval`` may bind: its words run again as commands here.
+
+ This shell expands the words first, so ``eval "$code"`` runs what ``code``
+ holds, while a ``$`` left quoted reaches eval's text as ``$``. The text is
+ then split into commands as the walk splits a line, and each is applied to
+ a copy of the bindings. A command word eval itself reads from a variable
+ (``eval '$code'``) is expanded and split there and is never an assignment.
+ A name the copy assigns, or whose value or attributes change there, is
+ unsettled afterwards, and exported if the copy exports it, because reading
+ quoted text a second time is not modelled exactly. So ``eval export -n f``
+ leaves no settled value for a child to inherit, and ``eval f=other.py``
+ none to read. Where the text is not settled here (a value the text cannot
+ settle, a command substitution), every bound name is left unsettled.
+ """
+ expanded = " ".join(_value_now(str(word), scope) for word in words)
+ text = expanded.replace(_LITERAL_DOLLAR, "$").replace(_QUOTED_NEWLINE, "\n")
+ if _UNSETTLED_VALUE in text or _UNSETTLED_EXPANSION_RE.search(text):
+ _unsettle_every_binding(scope)
+ return
+ trial = dict(scope)
+ assigned: set[str] = set()
+ segment: list[str] = []
+ for token in [*_shell_tokens(_mark_quoted_newlines(_mark_literal_dollars(text))), ";"]:
+ if token not in _SHELL_SEPARATORS:
+ segment.append(token)
+ continue
+ prefix: dict[str, str] = {}
+ cmd_idx = _command_start(segment, prefix) if segment else 0
+ for name, value in prefix.items():
+ assigned.add(name)
+ _bind(trial, name, _value_now(value, trial), reading, "eval")
+ command = [str(word) for word in segment[cmd_idx:]]
+ segment = []
+ if command and _SHELL_VARIABLE_RE.search(command[0]):
+ command = " ".join(_value_now(word, trial) for word in command).split()
+ if any(_UNSETTLED_VALUE in word for word in command):
+ _unsettle_every_binding(scope)
+ return
+ for name in _arithmetic_names(command):
+ assigned.add(name)
+ if "n" in trial.get(_ATTRIBUTE_MARK + name, ""):
+ _pass_to_reference(trial, name)
+ index = 0
+ while index < len(command) and command[index] in {"command", "builtin"}:
+ index += 1
+ while index < len(command) and command[index] == "-p":
+ index += 1
+ if index >= len(command):
+ continue
+ if command[index] in _LOOP_HEADER_WORDS and index + 1 < len(command):
+ assigned.add(command[index + 1])
+ elif command[index] == "eval" and depth < _MAX_SHELL_REFERENCE_DEPTH:
+ _apply_eval_bindings(command[index + 1 :], trial, reading, depth + 1)
+ else:
+ _apply_binding_builtin(command, 0, trial, reading)
+ for key in set(trial) | set(scope):
+ if trial.get(key) == scope.get(key):
+ continue
+ if key.startswith(_EXPORTED_MARK):
+ # Exported or unexported by the text: the value is unsettled, and
+ # the mark stays so that a child inherits that.
+ scope[key] = "1"
+ assigned.add(key.removeprefix(_EXPORTED_MARK))
+ elif key.startswith((_READONLY_MARK, _ATTRIBUTE_MARK, _REFERENCE_MARK)) or key in {
+ _ALLEXPORT_MARK,
+ _ASSIGNMENT_REFUSED,
+ }:
+ if key in trial:
+ scope[key] = trial[key]
+ else:
+ scope.pop(key, None)
+ else:
+ assigned.add(key)
+ for name in assigned:
+ if _SHELL_NAME_RE.fullmatch(name):
+ scope[name] = _UNSETTLED_VALUE
+
+
+def _unsettle_every_binding(scope: dict[str, str]) -> None:
+ """Text this walk cannot read may bind or unbind any name: none stays settled."""
+ for key in list(scope):
+ if _SHELL_NAME_RE.fullmatch(key):
+ scope[key] = _UNSETTLED_VALUE
+
+
+def _prefixed_scope(scope: dict[str, str], prefix: dict[str, str], reading: str) -> dict[str, str]:
+ """This shell's bindings with a command's ``NAME=value`` prefix over them.
+
+ The values are assigned in order, so a later one sees an earlier one,
+ except in mksh, which expands every value first (measured).
+ """
+ staged = dict(scope)
+ for name, value in prefix.items():
+ staged[name] = _value_now(value, scope if reading in _PREFIX_EXPANDS_FIRST else staged)
+ if "n" in scope.get(_ATTRIBUTE_MARK + name, ""):
+ staged[name] = _UNSETTLED_VALUE
+ _pass_to_reference(staged, name, exported=True)
+ return staged
+
+
+def _child_environment(
+ command: list[str], start: int, stop: int, scope: dict[str, str], prefix: dict[str, str], reading: str
+) -> dict[str, str] | None:
+ """What the command at ``stop`` inherits from this shell.
+
+ The exported names, then the segment's own ``NAME=value`` prefix, then
+ each ``env`` standing before the command, applied in order: ``-i`` and
+ ``-`` clear what came before, ``-u NAME`` removes one name, and its
+ assignments add names, expanded by this shell before env runs. ``None``
+ when a wrapper resets the environment in a way the text does not settle:
+ ``sudo`` and ``doas`` keep what their policy file says, and ``env -S``
+ splits a string into a command.
+ """
+ if any("n" in scope.get(_ATTRIBUTE_MARK + name, "") for name in prefix):
+ # A prefix on a ``-n`` name assigns the name it refers to.
+ return None
+ environment = {
+ key.removeprefix(_EXPORTED_MARK): scope.get(key.removeprefix(_EXPORTED_MARK), "")
+ for key in scope
+ if key.startswith(_EXPORTED_MARK)
+ # An array is never exported.
+ and not set(scope.get(_ATTRIBUTE_MARK + key.removeprefix(_EXPORTED_MARK), "")) & set("aA")
+ }
+ staged = _prefixed_scope(scope, prefix, reading)
+ for name in prefix:
+ environment[name] = staged[name]
+ index = start
+ while index < stop:
+ word = _shell_executable(_resolved_shell_arg(command[index], scope)).removesuffix(".exe")
+ if word in {"sudo", "doas"}:
+ return None
+ index += 1
+ if word != "env":
+ continue
+ options_done = False
+ while index < stop:
+ token = _resolved_shell_arg(command[index], scope).strip("\"'")
+ following = _resolved_shell_arg(command[index + 1], scope).strip("\"'") if index + 1 < stop else ""
+ assignment = _SHELL_ASSIGNMENT_RE.match(token)
+ if assignment is not None:
+ environment[assignment.group(1)] = _value_now(assignment.group(2), scope)
+ index += 1
+ continue
+ if options_done or not token.startswith("-"):
+ break
+ index += 1
+ if token == "--":
+ options_done = True
+ elif token in {"-", "-i", "--ignore-environment"}:
+ environment.clear()
+ elif token in {"-u", "--unset"}:
+ environment.pop(following, None)
+ index += 1
+ elif token.startswith("--unset="):
+ environment.pop(token.removeprefix("--unset="), None)
+ elif token in {"-C", "--chdir"}:
+ index += 1
+ elif token.startswith(("--chdir=", "--debug", "--null")) or token in {"-v", "-0"}:
+ continue
+ elif not token.startswith("--"):
+ letters = token[1:]
+ for position, letter in enumerate(letters):
+ if letter == "i":
+ environment.clear()
+ elif letter in "v0":
+ continue
+ elif letter == "u":
+ attached = letters[position + 1 :]
+ environment.pop(attached or following, None)
+ index += 0 if attached else 1
+ break
+ else:
+ return None
+ else:
+ return None
+ return environment
+
+
+def _reader_scope(
+ command: list[str], start: int, stop: int, scope: dict[str, str], prefix: dict[str, str], reading: str
+) -> dict[str, str]:
+ """The bindings whatever reads a command's text next may expand.
+
+ ``eval`` reads it in this shell, and inline code hands it on with the
+ command's environment: this shell's bindings, the command's prefix and
+ what ``env`` gives it. Asked only whether the script is named, so a wider
+ view costs no more than partial credit.
+ """
+ view = _prefixed_scope(scope, prefix, reading)
+ view.update(_child_environment(command, start, stop, scope, prefix, reading) or {})
+ return view
+
+
+def _carries_a_nested_invocation(command: list[str], cmd_idx: int, assignments: dict[str, str], expected: str) -> bool:
+ """Whether an invocation of the script sits inside an unrecognised command.
+
+ `flock /tmp/lock python run.py`, `strace -f python run.py` and
+ `taskset -c 0 python run.py` all run the script through a wrapper this walk
+ has no grammar for, and listing every such wrapper is not possible: the set
+ is open, and `./wrap.sh run.py` is indistinguishable from `cat run.py` by
+ text alone. So rather than name them, an unrecognised command that carries
+ what looks like an interpreter running the script is left unresolved, while
+ one that merely takes it as an argument stays a non-invocation.
+ """
+ words = [_resolved_shell_arg(str(word), assignments).strip("\"'") for word in command[cmd_idx + 1 :]]
+ for position, word in enumerate(words):
+ base = _VERSION_SUFFIX_RE.sub("", _shell_executable(word).removesuffix(".exe")) or word
+ if base not in _INTERPRETER_GRAMMARS and base != "source":
+ # "." is excluded: as an argument it is far more often a path or a
+ # filter (`jq . run.py`) than a sourcing command.
+ continue
+ if any(_script_path_matches(later, expected) or _unresolved_value(later) for later in words[position + 1 :]):
+ return True
+ return False
+
+
+def _unresolved_value(value: str) -> bool:
+ """Whether a resolved word holds a value the text bound but cannot settle."""
+ return _UNSETTLED_VALUE in value
+
+
+def _redirects_script_to_stdin(command: list[str], assignments: dict[str, str], expected: str) -> bool:
+ """Whether the script is fed to a command's standard input.
+
+ ``python <run.py`` runs the script even though it never appears as an
+ argument, and ``wc -l <run.py`` only counts its lines, so which of the two
+ happened is not something the command text settles.
+ """
+ for position, word in enumerate(command[:-1]):
+ token = str(word)
+ if token in _INPUT_REDIRECTS or (token.endswith("<") and not token.startswith(_QUOTED_SYNTAX_MARK)):
+ operand = _resolved_shell_arg(str(command[position + 1]), assignments)
+ if _script_path_matches(operand, expected) or _unresolved_value(operand):
+ return True
+ return False
+
+
+def _command_names_script(command: list[str], cmd_idx: int, assignments: dict[str, str], expected: str) -> bool:
+ """Whether the expected script is named anywhere in this command's words.
+
+ Inline code, a module, or text handed to ``eval`` can run the script
+ without it ever appearing as an argument this walk resolves, so naming it
+ is the difference between "nothing ran" and "this walk cannot tell".
+ A ``$`` this shell leaves quoted is read as a variable here: whatever
+ reads the text next (``eval``, a child shell, ``os.system``) may expand it.
+ """
+ target = str(expected).strip().strip("\"'")
+ if not target:
+ return False
+ return any(
+ target in _resolved_shell_arg(str(word).replace(_LITERAL_DOLLAR, "$"), assignments)
+ for word in command[cmd_idx + 1 :]
+ )
+
+
+def _names_script_anywhere(command_text: str, expected_script: str) -> bool:
+ """Whether the expected script's file name appears anywhere in the command text.
+
+ This is the reference test the checker applied before invocation evidence
+ was required, kept as the floor under partial credit. A walk that cannot
+ resolve a command which never names the script has learned nothing about
+ that script, so it answers "did not run it" rather than "cannot tell":
+ parsing uncertainty is not evidence.
+ """
+ name = str(expected_script).strip().strip("\"'").replace("\\", "/").rsplit("/", 1)[-1]
+ return bool(name) and name.casefold() in str(command_text).casefold()
+
+
+# Words that open and close a compound command, for scoping a pipeline that
+# runs one: ``echo x | while read -r l; do ...; done``.
+# ``{`` and ``}`` are here too: ``{ f=x; } | cat`` is one pipeline stage.
+_COMPOUND_OPENERS = frozenset({"for", "select", "while", "until", "if", "case", "{"})
+_COMPOUND_CLOSERS = frozenset({"done", "fi", "esac", "}"})
+_COMMAND_POSITION_LEADERS = frozenset({"then", "do", "else", "elif", "!", "{", "("})
+# A compound's opening word is reserved only where a command starts, not after
+# an assignment. Written there, it and the compound's own syntax are refused
+# by bash, bash --posix, dash, zsh, ksh, mksh and busybox ash, for each of
+# these and for ``(`` and ``((``: ``A=1 if true; then python3 run.py; fi``
+# runs nothing. Only zsh refuses ``[[`` there, or an opening word with none of
+# its compound's syntax after it (``A=1 for x; ...``); the others run it as a
+# command name that is not found. Both are read as refused, so an invocation
+# on such a line is unresolved rather than given a definite score.
+_OPENERS_REFUSED_AFTER_ASSIGNMENT = _COMPOUND_OPENERS | {"[["}
+
+
+def _compound_after_assignment(tokens: list[str]) -> bool:
+ """Whether a compound's opening word follows an assignment in a command.
+
+ Every modelled shell refuses the text before running the line that holds
+ it, but most run the complete commands before it, so the caller may
+ credit the first of those (``_command_run_before_refusal``). ``A=$(`` and
+ ``A=(`` reach here as an assignment ending in ``$`` or ``=`` and then
+ ``(``, a substitution or an array rather than a subshell.
+ """
+ assigned: str | None = None
+ at_start = True
+ for token in tokens:
+ if token in _SHELL_SEPARATORS:
+ assigned, at_start = None, True
+ continue
+ if not at_start:
+ continue
+ if assigned is None and (token in _COMMAND_INTRODUCING_WORDS or token in _COMMAND_POSITION_LEADERS):
+ continue
+ if _SHELL_ASSIGNMENT_RE.match(token):
+ assigned = token
+ continue
+ if assigned is not None:
+ if token in _OPENERS_REFUSED_AFTER_ASSIGNMENT:
+ return True
+ if token == "(" and not assigned.endswith(("$", "=")):
+ return True
+ at_start = False
+ return False
+
+
+# zsh parses the whole of a ``-c`` text before running any of it, so a syntax
+# error anywhere in it runs nothing. bash, bash --posix, dash, ksh, mksh and
+# busybox ash parse and run one complete command at a time: a command on a
+# line before the refused one has already run. Measured with
+# ``python3 run.py`` on the line before ``A=1 for g in x; do :; done``, and
+# before ``cat <<< x`` under dash and ash.
+_PARSES_WHOLE_TEXT_FIRST = frozenset({"zsh"})
+# How many lines are examined for where a complete command ends, and how much
+# text in all is read to decide it. Past either, the lines that remain are read
+# as one unit with the one before them.
+_MAX_PARSE_UNIT_LINES = 256
+_MAX_PARSE_UNIT_READ = 2 * _MAX_SHELL_REFERENCE_CHARS
+# An unquoted newline, kept apart from ``;`` while a unit's syntax is read, so
+# that ``;;`` in a ``case`` is not confused with a blank line.
+_UNIT_NEWLINE = ""
+# Only the first command of a text is credited when a later line is refused,
+# and only when the walk reads it as the shell does: one pipeline of simple
+# commands on its own lines (``_plain_pipeline``). Nothing ran before it, so no
+# command the walk does not model can have stopped it, and a line refused for
+# a reason the walk does not detect is not credited. Measured before a refused
+# line: ``exit``, ``exec true``, ``set -n`` or ``kill $$`` on the line before
+# ``python3 run.py``, and ``python3 run.py`` ending in ``>``, ``; fi``, ``(x)``,
+# ``;;`` or a carriage return, run nothing in bash, bash --posix, dash, zsh,
+# ksh, mksh or busybox ash; an empty command (``; python3 run.py``) runs
+# nothing but in ksh.
+_NOT_A_SIMPLE_COMMAND = (
+ _COMPOUND_OPENERS | _COMPOUND_CLOSERS | _COMMAND_POSITION_LEADERS | {"in", "[[", "]]", "time", "coproc", "function"}
+)
+_REDIRECTION_OPERATOR_RE = re.compile(r"\A\d*(?:&>>?|>>|>\||>&|<&|<>|<<<|<<-|<<|>|<)\Z")
+# A token of operator characters alone: ``|``, a redirection, or anything
+# else the tokenizer left joined (``>;``, ``|&``, ``<(``).
+_OPERATOR_TOKEN_RE = re.compile(r"\A\d*[;&|()<>]+\Z")
+# A substitution or an expansion inside ``"..."`` reads its own quotes, and
+# ``$'...'`` can hold an escaped one; a scan pairing quote characters would
+# misplace them, and with them where a line ends.
+_NESTED_QUOTING_RE = re.compile(r"\$[({']|`")
+# What closes each compound, when its opening word stands where a command may.
+_UNIT_CLOSERS = {
+ "for": "done",
+ "select": "done",
+ "while": "done",
+ "until": "done",
+ "if": "fi",
+ "case": "esac",
+ "{": "}",
+ "[[": "]]",
+}
+
+
+def _heredoc_body_spans(text: str, reading: str) -> list[tuple[int, int]] | None:
+ """Where each heredoc body lies in ``text``, as ``_split_heredocs`` reads it.
+
+ Each span runs from the newline that ends the line declaring the heredoc
+ to the newline that ends its terminator line, so neither of those, nor any
+ line of the body, can end a command. ``None`` when the text declares more
+ heredocs than are read, or one whose terminator is never named.
+ """
+ spans: list[tuple[int, int]] = []
+ position = 0
+ budget = _MAX_HEREDOC_OPERANDS
+ per_line = _MAX_HEREDOCS_PER_LINE.get(reading, _MAX_HEREDOC_OPERANDS)
+ while True:
+ offset = _heredoc_operator_index(text[position:])
+ if offset < 0:
+ return spans
+ after = position + offset + 2
+ if text.startswith("<", after):
+ # A here-string's operand stays on its line.
+ position = after + 1
+ budget -= 1
+ if budget <= 0:
+ return None
+ continue
+ line_end = text.find("\n", after)
+ header = _heredoc_header(text[after:] if line_end == -1 else text[after:line_end])
+ if header is None:
+ return None
+ _, strings, terminators = header
+ budget -= len(terminators) + len(strings)
+ if len(terminators) > per_line or budget < 0:
+ return None
+ if line_end == -1:
+ return spans
+ body_position = line_end + 1
+ for terminator, strip_tabs in terminators:
+ segment = text[body_position:]
+ _, resumed = _split_heredoc_body(segment, terminator, strip_tabs)
+ body_position += len(segment) - len(resumed)
+ end = body_position - 1 if body_position > line_end + 1 and text[body_position - 1] == "\n" else body_position
+ spans.append((line_end, end))
+ position = body_position
+
+
+def _unit_is_complete(unit: str, reading: str, arithmetic_parens: bool) -> bool:
+ """Whether ``unit`` ends a complete command, so the newline after it ends
+ what the shell parses before running it.
+
+ Every compound it opens is closed, and it does not end on ``|``, ``&&``,
+ ``||``, ``!`` or a function's name awaiting its body. Anything the reading
+ here does not settle is incomplete, so the next line joins the unit.
+ """
+ analysed = _split_heredocs(unit, reading)[0]
+ if not analysed.strip():
+ return True
+ marked = _mark_quoted_newlines(_mark_literal_dollars(analysed)).replace("\n", f" {_UNIT_NEWLINE} ")
+ tokens = [
+ token for token in _split_punctuation_runs(_shell_tokens(marked), arithmetic_parens, reading != "ksh") if token
+ ]
+ if not tokens:
+ return False
+ stack: list[str] = []
+ at_command = True
+ case_header = False
+ case_pattern = False
+ previous = ""
+ for token in tokens:
+ if stack and stack[-1] == "esac" and case_pattern:
+ if token == "esac":
+ stack.pop()
+ case_pattern = False
+ at_command = False
+ elif token == ")":
+ case_pattern = False
+ at_command = True
+ previous = token
+ continue
+ if token == _UNIT_NEWLINE or token in _SHELL_SEPARATORS:
+ if stack and stack[-1] == "esac" and not case_header and token in {";", "&"} and previous == ";":
+ # ``;;``, ``;&`` or ``;;&``: the next word is a pattern.
+ case_pattern = True
+ at_command = True
+ previous = token
+ continue
+ if _PUNCTUATION_RUN_RE.match(token) or token in {"<(", ">("}:
+ # A group, a substitution, an arithmetic command or a process
+ # substitution is open until its parenthesis closes.
+ for char in token:
+ if char == "(":
+ stack.append(")")
+ elif char == ")":
+ if not stack or stack[-1] != ")":
+ return False
+ stack.pop()
+ at_command = True
+ previous = token
+ continue
+ if case_header:
+ if token == "in":
+ case_header = False
+ case_pattern = True
+ previous = token
+ continue
+ if previous == "function":
+ # ``function f {``: the body follows the name.
+ at_command = True
+ previous = token
+ continue
+ if stack and stack[-1] == "]]" and token == "]]":
+ stack.pop()
+ elif at_command and token in _UNIT_CLOSERS:
+ stack.append(_UNIT_CLOSERS[token])
+ case_header = token == "case"
+ elif at_command and token in {"done", "fi", "esac", "}"}:
+ if not stack or stack[-1] != token:
+ return False
+ stack.pop()
+ at_command = token in _COMMAND_POSITION_LEADERS or token in {"time", "!"} or token in _COMMAND_INTRODUCING_WORDS
+ previous = token
+ last = [token for token in tokens if token != _UNIT_NEWLINE]
+ if stack or case_header or not last:
+ return False
+ if last[-1] in {"|", "&&", "||", "!", "time", "function"} or last[-2:] == ["|", "&"]:
+ return False
+ # A function's name awaits its body on the next line.
+ return last[-2:] != ["(", ")"] and not (len(last) >= 2 and last[-2] == "function")
+
+
+def _parse_unit_starts(text: str, reading: str, arithmetic_parens: bool) -> tuple[list[int], str]:
+ """Where each top-level unit the shell parses before running it starts,
+ and the text with its comments blanked.
+
+ The first is the start of the text; each other follows an unquoted
+ newline, outside a comment and a heredoc body, that ends a complete
+ command (``_unit_is_complete``). A newline whose place the text does not
+ settle ends nothing, so the lines around it are read as one unit. A
+ comment starts at a ``#`` that begins a word, after an operator too:
+ ``python3 run.py >#x`` leaves its redirection without an operand.
+ """
+ starts = [0]
+ if len(text) > _MAX_SHELL_REFERENCE_CHARS:
+ return starts, text
+ spans = _heredoc_body_spans(text, reading)
+ if spans is None:
+ return starts, text
+ span_ends = dict(spans)
+ blanked = list(text)
+ quote: str | None = None
+ lines = 0
+ read = 0
+ index = 0
+ while index < len(text):
+ if quote is None and index in span_ends:
+ index = span_ends[index]
+ continue
+ char = text[index]
+ if char == "\\" and quote != "'" and index + 1 < len(text):
+ index += 2
+ continue
+ if quote is not None:
+ if char == quote[-1]:
+ quote = None
+ index += 1
+ continue
+ if char == "#" and (index == 0 or text[index - 1].isspace() or text[index - 1] in _SHELL_METACHARS):
+ end = text.find("\n", index)
+ end = len(text) if end < 0 else end
+ blanked[index:end] = " " * (end - index)
+ index = end
+ continue
+ if char == "$" and text.startswith("'", index + 1):
+ # Keep the opening token to distinguish ANSI-C from plain single quotes.
+ quote = text[index : index + 2]
+ index += 2
+ continue
+ if char in "'\"`":
+ quote = char
+ elif char == "\n":
+ lines += 1
+ read += index - starts[-1]
+ if lines > _MAX_PARSE_UNIT_LINES or read > _MAX_PARSE_UNIT_READ:
+ break
+ if _unit_is_complete("".join(blanked[starts[-1] : index]), reading, arithmetic_parens):
+ starts.append(index + 1)
+ index += 1
+ return starts, "".join(blanked)
+
+
+def _plain_pipeline(unit: str, reading: str, arithmetic_parens: bool) -> bool:
+ """Whether ``unit``, its comments blanked, is one pipeline of simple
+ commands that the walk reads as the shell does.
+
+ Words and complete redirections, joined only by ``|``, with at most a
+ trailing ``;`` or ``&``. Not a reserved word, a group or an assignment
+ where a command starts, a parenthesis, ``&&`` or ``||``, a substitution,
+ a redirection operator without its operand, a here-string (a missing
+ operand is read from the next line), or a carriage return or an escaped
+ newline, which the walk reads as a line's end and the shells as part of
+ the word before it (``python3 run.py\\r`` opens ``run.py\\r``, and
+ ``python3 run.py\\`` joins the next line to ``run.py``). Anything else
+ is not plain, which only withholds credit.
+ """
+ if "\r" in unit or "\\\n" in unit or _NESTED_QUOTING_RE.search(unit):
+ return False
+ analysed, _, here_strings = _split_heredocs(unit, reading)
+ if here_strings:
+ return False
+ marked = _mark_quoted_newlines(_mark_literal_dollars(analysed)).replace("\n", f" {_UNIT_NEWLINE} ")
+ tokens = [
+ token for token in _split_punctuation_runs(_shell_tokens(marked), arithmetic_parens, reading != "ksh") if token
+ ]
+ while tokens and tokens[-1] == _UNIT_NEWLINE:
+ tokens.pop()
+ if tokens and tokens[-1] in {";", "&"}:
+ tokens.pop()
+ started = False # a word or a redirection of this command has been read
+ named = False # and its first word
+ joined = False # a ``|`` has been read, and nothing of the next command
+ pending = False # a redirection operator awaits its operand
+ for token in tokens:
+ operator = bool(_OPERATOR_TOKEN_RE.match(token))
+ if pending:
+ if token == _UNIT_NEWLINE or operator:
+ return False
+ pending = False
+ elif token == _UNIT_NEWLINE:
+ # Only the command after a ``|`` may start on a later line.
+ if not joined:
+ return False
+ elif token == "|":
+ if not started:
+ return False
+ started = named = False
+ joined = True
+ elif _REDIRECTION_OPERATOR_RE.match(token):
+ started, joined, pending = True, False, True
+ elif operator:
+ return False
+ else:
+ if not named and (token in _NOT_A_SIMPLE_COMMAND or _SHELL_ASSIGNMENT_RE.match(token)):
+ return False
+ started = named = True
+ joined = False
+ return started and not pending
+
+
+@lru_cache(maxsize=256)
+def _command_run_before_refusal(text: str, reading: str, arithmetic_parens: bool) -> str:
+ """The first command of ``text``, when the shell runs it before refusing a
+ later one and the walk reads it as the shell does; empty otherwise.
+
+ A unit is refused as the whole text is (a here-string under a shell that
+ has none, a compound's opening word after an assignment). The first unit
+ that is not blank must come before the first refused one and be a plain
+ pipeline (``_plain_pipeline``); it is returned with its comments blanked.
+ """
+ starts, blanked = _parse_unit_starts(text, reading, arithmetic_parens)
+ first: str | None = None
+ for number, start in enumerate(starts):
+ end = starts[number + 1] if number + 1 < len(starts) else len(text)
+ analysed, _, here_strings = _split_heredocs(text[start:end], reading)
+ tokens = _split_punctuation_runs(
+ _shell_tokens(_mark_quoted_newlines(_mark_literal_dollars(analysed))),
+ arithmetic_parens,
+ reading != "ksh",
+ )
+ if (here_strings and reading in _NO_HERE_STRING_SHELLS) or _compound_after_assignment(tokens):
+ return first or ""
+ unit = blanked[start:end]
+ if first is None and unit.strip(" \t\n"):
+ first = unit if _plain_pipeline(unit, reading, arithmetic_parens) else ""
+ return ""
+
+
+# Whether the last command of a pipeline runs in the current shell, so a
+# binding it makes outlives the pipeline. zsh and ksh run it there; bash,
+# dash and mksh fork it like the other stages, unless bash has `lastpipe`
+# set, which is a run-time option the text cannot settle. Measured on each
+# shell with `printf "" | for f in x; do :; done; echo "$f"`.
+_LAST_STAGE_IN_CURRENT_SHELL = frozenset({"zsh", "ksh"})
+_LAST_STAGE_IN_SUBSHELL = frozenset({"bash", "sh", "dash", "ash", "mksh"})
+# Option changes that can move the last stage between the two rules: bash's
+# `shopt -s lastpipe`, and zsh's `emulate sh` (measured: it forks the last
+# stage). Any of these words in the text leaves the rule unsettled.
+_PIPELINE_OPTION_RE = re.compile(r"\b(?:lastpipe|shopt|setopt|unsetopt|emulate)\b")
+
+
+def _command_position_words(command):
+ """The words of a segment that stand where a command may: the first, and
+ each that follows ``then``, ``do``, ``else``, ``elif``, ``!``, ``{`` or ``(``.
+ Only there is ``for`` or ``done`` a reserved word rather than an argument.
+ """
+ words = []
+ expect = True
+ for token in command:
+ word = str(token).strip("\"'")
+ if expect:
+ words.append(word)
+ expect = word in _COMMAND_POSITION_LEADERS
+ return words
+
+
+def _scope_events(command):
+ """The groups and compound openers of one segment, in the order they open.
+
+ ``{ ( for f in x`` opens a brace group, a subshell group and a loop, one
+ inside the next. A ``(`` is a group wherever it stands; an opener word
+ counts only at command position, where the shell reads it as one.
+ """
+ events = []
+ expect = True
+ for token in command:
+ word = str(token).strip("\"'")
+ if token == "(":
+ events.append("(")
+ elif expect and word in _COMPOUND_OPENERS:
+ events.append(word)
+ expect = word in _COMMAND_POSITION_LEADERS
+ return events
+
+
+def _compound_closer_ends(tokens, end, count):
+ """For each of the ``count`` compounds a segment ending at ``end`` opens,
+ outermost first, the index just past the segment holding its closing
+ word; ``len(tokens)`` for one that never closes. The token there tells
+ whether that compound, and only that one, is piped into another stage.
+ """
+ ends = [len(tokens)] * count
+ depth = count
+ position = end
+ while position < len(tokens) and depth > 0:
+ if tokens[position] in _SHELL_SEPARATORS:
+ position += 1
+ continue
+ segment_end = position
+ while segment_end < len(tokens) and tokens[segment_end] not in _SHELL_SEPARATORS:
+ segment_end += 1
+ for word in _command_position_words(tokens[position:segment_end]):
+ if word in _COMPOUND_OPENERS:
+ depth += 1
+ elif word in _COMPOUND_CLOSERS:
+ depth -= 1
+ if 0 <= depth < count and ends[depth] == len(tokens):
+ ends[depth] = segment_end
+ position = segment_end
+ return ends
+
+
+_SHELL_NAME_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
+
+
+def _loop_header_is_empty(
+ command: list[str], cmd_idx: int, assignments: dict[str, str], positional_known_empty: bool
+) -> bool:
+ """``for f in; do`` with nothing after ``in``: the loop runs zero times.
+
+ Its variable keeps whatever it held, and its body never runs, so the
+ body is not evidence of anything. ``for f; do`` (no ``in``) is different:
+ it iterates the positional parameters, which the text does not carry.
+ """
+ words = [_resolved_shell_arg(str(word), assignments).strip("\"'") for word in command[cmd_idx + 1 :]]
+ if len(words) == 1 and _SHELL_NAME_RE.fullmatch(words[0]):
+ # ``for f; do`` iterates the positional parameters: empty when the
+ # text runs with none, which is what a tool call gets.
+ return positional_known_empty
+ return len(words) == 2 and bool(_SHELL_NAME_RE.fullmatch(words[0])) and words[1] == "in"
+
+
+def _loop_header_is_unresolved(
+ command: list[str], cmd_idx: int, assignments: dict[str, str], expected: str, reading: str = "bash"
+) -> bool:
+ """Bind a loop variable where the header settles it, else say whether it matters.
+
+ ``for f in run.py; do python $f; done`` gives ``f`` exactly one value, so
+ the body is read with ``f`` bound and scored as ``python run.py`` would be,
+ and ``cat $f`` in the same body stays a non-invocation. A header with
+ several values, or none (``for f; do`` iterates the positional
+ parameters), settles nothing about ``$f``: if the script is among the
+ values the command is unresolved, and otherwise the header runs nothing.
+ """
+ words = [_resolved_shell_arg(str(word), assignments).strip("\"'") for word in command[cmd_idx + 1 :]]
+ if words and _SHELL_NAME_RE.fullmatch(words[0]) and assignments.get(_READONLY_MARK + words[0]):
+ # The header cannot assign a read-only variable.
+ _refuse_assignment(assignments, words[0], reading, "for")
+ return _command_names_script(command, cmd_idx, assignments, expected)
+ if words and _SHELL_NAME_RE.fullmatch(words[0]) and "n" in assignments.get(_ATTRIBUTE_MARK + words[0], ""):
+ # A ``-n`` loop variable: bash refers it to each value in turn, and
+ # the name it referred to may be left changed.
+ _pass_to_reference(assignments, words[0])
+ if words and _SHELL_NAME_RE.fullmatch(words[0]):
+ if len(words) == 1:
+ # The positional parameters, which this text does not carry: a
+ # later read of the variable is unresolved, not a settled miss.
+ assignments[words[0]] = _UNSETTLED_VALUE
+ return _command_names_script(command, cmd_idx, assignments, expected)
+ if len(words) >= 2 and words[1] == "in":
+ values = words[2:]
+ if len(values) == 1:
+ _bind(assignments, words[0], _value_now(values[0], assignments), reading, "for")
+ return False
+ # This header gives the variable several values the text does not
+ # settle, or is ``for f; do`` over the positional parameters, so a
+ # value an earlier single-value header gave it no longer holds. An
+ # empty ``in`` list never reaches here: the walk skips that loop
+ # whole (see ``_loop_header_is_empty``).
+ assignments.pop(words[0], None)
+ return _command_names_script(command, cmd_idx, assignments, expected)
+
+
+def _command_start(command: list[str], prefix: dict[str, str]) -> int:
+ """Index of the word that is the command, past reserved words and assignments.
+
+ ``then FOO=1 ./run.py`` runs ``./run.py``: the reserved word introduces the
+ command and the assignment is its environment, in either order. The
+ assignments are collected in ``prefix`` rather than bound, because where
+ they apply depends on what follows them: alone they bind in the shell,
+ before a command they are that command's environment only, and in neither
+ case do they reach the command's own words.
+ """
+ cmd_idx = 0
+ while cmd_idx < len(command):
+ word = command[cmd_idx]
+ if word in _COMMAND_INTRODUCING_WORDS or word in _GROUPING_TOKENS:
+ cmd_idx += 1
+ continue
+ assignment = _SHELL_ASSIGNMENT_RE.match(word)
+ if assignment is None:
+ break
+ prefix[assignment.group(1)] = assignment.group(2)
+ cmd_idx += 1
+ return cmd_idx
+
+
+def _script_path_matches(value: Any, expected_script: str) -> bool:
+ """Whether one shell argument names the expected script exactly.
+
+ ``run.py`` matches ``run.py``, ``./run.py`` and ``/skills/demo/run.py``. It
+ does not match ``run.py.bak``, ``rerun.py`` or ``run.pyc``: a filename that
+ merely contains the expected one is a different file.
+ """
+ target = str(expected_script).strip().strip("\"'").replace("\\", "/")
+ candidate = str(value).strip().strip("\"'").replace("\\", "/")
+ if not target or not candidate:
+ return False
+ if "/" in target:
+ return candidate == target or candidate.endswith(f"/{target}")
+ return candidate.rsplit("/", 1)[-1] == target
+
+
+# How many heredoc and here-string operators one command's text is read for,
+# so the work stays bounded. Past it, the rest of the text is data: a script
+# named there is unresolved.
+_MAX_HEREDOC_OPERANDS = 256
+# How many heredocs one line may declare before the shell refuses the line.
+# Measured: bash and bash --posix exit with "maximum here-document count
+# exceeded" at 17 and mksh stops at 11 ("too many <<s"), before running
+# anything on the line; dash, zsh, ksh and busybox ash read 60 and more.
+# Past it, the line and everything after it are data.
+_MAX_HEREDOCS_PER_LINE = {"bash": 16, "bash-posix": 16, "mksh": 10}
+# dash and busybox ash have no here-string: ``<<<`` is a syntax error there,
+# and the shell runs nothing on the line that holds it, nor any of a compound
+# command spanning lines around it, nor anything after (measured), so under
+# those readings only the complete commands before it are credited as run.
+_NO_HERE_STRING_SHELLS = frozenset({"dash", "ash"})
+# The word after ``<<`` that ends the body, optionally quoted or escaped. The
+# ``-`` of ``<<-`` asks for leading tabs to be stripped from the terminator.
+_HEREDOC_DELIMITER_RE = re.compile(r"\A(-?)[ \t]*((?:[^\s;&|<>()'\"`\\]|\\.|'[^']*'|\"[^\"]*\")+)")
+
+
+def _skip_arithmetic(text: str, index: int) -> int:
+ """Advance past an arithmetic expansion, where ``<<`` is a shift operator."""
+ depth = 0
+ while index < len(text):
+ if text[index] == "(":
+ depth += 1
+ elif text[index] == ")":
+ depth -= 1
+ if depth == 0:
+ return index + 1
+ index += 1
+ return index
+
+
+def _heredoc_operator_index(text: str) -> int:
+ """Index of the first ``<<`` that is really a heredoc or here-string operator.
+
+ Quoted text and arithmetic are skipped, so neither ``echo '<<'`` nor
+ ``echo $((1 << 2))`` is read as one. Returns ``-1`` when there is none.
+ """
+ quote: str | None = None
+ index = 0
+ while index < len(text) - 1:
+ char = text[index]
+ if char == "\\" and quote != "'":
+ index += 2
+ continue
+ if char in {"'", '"'}:
+ if quote is None:
+ quote = char
+ elif quote == char:
+ quote = None
+ index += 1
+ continue
+ if quote is None and char == "(" and text[index + 1] == "(":
+ index = _skip_arithmetic(text, index)
+ continue
+ if quote is None and char == "<" and text[index + 1] == "<":
+ return index
+ index += 1
+ return -1
+
+
+def _split_heredoc_body(body: str, terminator: str, strip_tabs: bool) -> tuple[str, str]:
+ """Split a heredoc body from the commands written after its terminator line."""
+ lines = body.split("\n")
+ for offset, line in enumerate(lines):
+ candidate = line.lstrip("\t") if strip_tabs else line
+ if candidate.rstrip("\r") == terminator:
+ return "\n".join(lines[:offset]), "\n".join(lines[offset + 1 :])
+ return body, ""
+
+
+# shlex groups a run of shell punctuation into one token, so ``));`` arrives
+# whole and the separator inside it would otherwise be missed, leaving the
+# command after it read as an argument of the command before it.
+_PUNCTUATION_RUN_RE = re.compile(r"\A[();|&]+\Z")
+_PUNCTUATION_PIECE_RE = re.compile(r"&&|\|\||[();|&]")
+
+
+def _arithmetic_close(tokens: list[str], start: int) -> tuple[int, int] | None:
+ """Where the ``((`` before ``start`` is closed as an arithmetic command.
+
+ bash, zsh, ksh and mksh read ``((`` as arithmetic when the parenthesis that
+ closes its inner half is written directly before the one that closes its
+ outer half, as ``))``, whatever the text between: ``((f=x; g=y))`` is
+ arithmetic (and fails), ``((f=x; g=y) )`` and ``((f=x) ; g)`` are two
+ nested subshells. Parentheses opened inside are counted. Returns the index
+ of the token holding that ``))`` and the offset of its first character, or
+ ``None`` when the text closes some other way.
+ """
+ depth = 0
+ for index in range(start, len(tokens)):
+ token = tokens[index]
+ if not _PUNCTUATION_RUN_RE.match(token):
+ continue
+ for offset, char in enumerate(token):
+ if char == "(":
+ depth += 1
+ elif char == ")":
+ if depth:
+ depth -= 1
+ else:
+ return (index, offset) if token[offset + 1 : offset + 2] == ")" else None
+ return None
+
+
+def _split_punctuation_runs(
+ tokens: list[str], arithmetic: bool = True, triple_opens_subshell: bool = True
+) -> list[str]:
+ """Separate a grouped run of punctuation into the operators it is made of.
+
+ With ``arithmetic``, a ``((`` at command position that closes as an
+ arithmetic command is emitted as three tokens, ``((``, its whole body as
+ one word, and ``))``, so the walk neither splits the body at its
+ separators nor reads its parentheses as subshells. dash has no
+ arithmetic command, so its caller passes ``arithmetic=False`` and every
+ ``((`` is two subshells. ``$((`` and ``for ((`` are not at command
+ position and split as before.
+
+ The tokenizer hands ``(((`` over as one token. bash, zsh and mksh read it
+ as they read ``( ((``, a subshell around what may be an arithmetic
+ command, so with ``triple_opens_subshell`` the leading parentheses are
+ split off and the last two are tried as ``((``. ksh reads it as nested
+ subshells, which is what splitting every parenthesis gives. Measured
+ with ``f=other.py; (((for f in run.py; do :; done; python3 "$f")) | cat)``.
+ """
+ tokens = list(tokens)
+ separated: list[str] = []
+ position = 0
+ while position < len(tokens):
+ token = tokens[position]
+ at_command_position = (
+ not separated or separated[-1] in _SHELL_SEPARATORS or separated[-1] in _COMMAND_POSITION_LEADERS
+ )
+ if arithmetic and triple_opens_subshell and at_command_position and len(token) > 2 and set(token) == {"("}:
+ tokens[position : position + 1] = ["("] * (len(token) - 2) + ["(("]
+ continue
+ if arithmetic and token == "((" and at_command_position:
+ close = _arithmetic_close(tokens, position + 1)
+ if close is not None:
+ close_index, close_offset = close
+ body = [*tokens[position + 1 : close_index], tokens[close_index][:close_offset]]
+ separated.append("((")
+ separated.append(" ".join(word for word in body if word))
+ separated.append("))")
+ separated.extend(_PUNCTUATION_PIECE_RE.findall(tokens[close_index][close_offset + 2 :]))
+ position = close_index + 1
+ continue
+ if len(token) > 1 and _PUNCTUATION_RUN_RE.match(token):
+ separated.extend(_PUNCTUATION_PIECE_RE.findall(token))
+ else:
+ separated.append(token)
+ position += 1
+ return separated
+
+
+# An assignment inside an arithmetic command: ``f=1``, ``f+=2``, ``f<<=1``,
+# ``f++`` and ``--f``. The value is a number, never a script path; on an
+# error bash, zsh and mksh leave the variable as it was, and ksh aborts
+# the rest of the text.
+_ARITHMETIC_ASSIGNMENT_RE = re.compile(
+ r"([A-Za-z_]\w*)\s*(?:<<|>>|[-+*/%&|^])?=(?!=)|([A-Za-z_]\w*)\s*(?:\+\+|--)|(?:\+\+|--)\s*([A-Za-z_]\w*)"
+)
+# The walk hands ``$((`` over as ``$``, ``(``, ``(``.
+_ARITHMETIC_EXPANSION_RE = re.compile(r"\$\s*(?:\(\s*\(|\[)")
+
+
+def _arithmetic_texts(words: list[str], cmd_idx: int, expansions: bool = True) -> list[str]:
+ """The arithmetic a command evaluates, where an assignment binds as it does
+ in ``((...))``: that body, the words after ``let``, and with
+ ``expansions`` each ``$((...))`` and ``$[...]`` in its words, which this
+ shell evaluates as it expands them. A ``$`` left quoted is marked literal
+ and starts none. The caller passes ``expansions=False`` for text with no
+ ``$((`` or ``$[``, where ``$ ( (`` is ``$( (``, a command substitution."""
+ texts: list[str] = []
+ if "((" in words:
+ body = words.index("((") + 1
+ texts.append(words[body] if body < len(words) else "")
+ if cmd_idx < len(words) and words[cmd_idx].strip("\"'") == "let":
+ texts.append(" ".join(words[cmd_idx + 1 :]))
+ joined = " ".join(words) if expansions else ""
+ for match in _ARITHMETIC_EXPANSION_RE.finditer(joined):
+ depth = 0
+ end = len(joined)
+ for position in range(match.end(), len(joined)):
+ if joined[position] in "([":
+ depth += 1
+ elif joined[position] in ")]":
+ if depth == 0:
+ end = position
+ break
+ depth -= 1
+ texts.append(joined[match.end() : end])
+ return texts
+
+
+def _arithmetic_names(words: list[str], expansions: bool = True) -> list[str]:
+ """The names the arithmetic in a command assigns (``_arithmetic_texts``)."""
+ words = [str(word) for word in words]
+ return [
+ next(group for group in match.groups() if group)
+ for text in _arithmetic_texts(words, _command_start(words, {}), expansions)
+ for match in _ARITHMETIC_ASSIGNMENT_RE.finditer(text)
+ ]
+
+
+def _apply_arithmetic_assignments(
+ command: list[str], scope: dict[str, str], expected_script: str, expansions: bool = True
+) -> None:
+ """What the assignments in a command's arithmetic leave.
+
+ The variable ends as a number, unchanged after an error, or never read
+ because ksh aborted. Only when it held the script (or the script's name
+ is a number) is that a question. A ``-n`` name assigns the name it refers
+ to instead.
+ """
+ numeric_script = str(expected_script).rsplit("/", 1)[-1].isdigit()
+ for name in _arithmetic_names(command, expansions):
+ if "n" in scope.get(_ATTRIBUTE_MARK + name, ""):
+ _pass_to_reference(scope, name)
+ if numeric_script or _script_path_matches(_resolved_shell_arg(scope.get(name, ""), scope), expected_script):
+ scope[name] = _UNSETTLED_VALUE
+
+
+def _unquoted_separator_index(text: str) -> int:
+ """Where an operand ends: at the first separator outside quotes.
+
+ ``cat <<< "a; b"`` is one operand, so the ``;`` inside the quotes does not
+ end it and the command written after the real separator is still walked.
+ """
+ quote: str | None = None
+ for index, char in enumerate(text):
+ if char == "\\" and quote != "'":
+ continue
+ if char in {"'", '"'}:
+ if quote is None:
+ quote = char
+ elif quote == char:
+ quote = None
+ continue
+ if quote is None and char in {";", "\n", "|", "&"}:
+ return index
+ return len(text)
+
+
+def _split_heredocs(command_text: str, reading: str = "bash") -> tuple[str, str, int]:
+ """Split a command into the text to walk and the text that is operand data.
+
+ Returns the commands to walk, the heredoc or here-string data fed to
+ them, and how many here-strings were read. Quotes are tracked so ``echo '<<'`` is not mistaken for a heredoc,
+ arithmetic is skipped so ``echo $((1 << 2))`` is not either, a heredoc body
+ ends at its terminator line so commands written after it are still walked,
+ and a here-string consumes only its single operand.
+ """
+ commands: list[str] = []
+ data: list[str] = []
+ here_strings = 0
+ remaining = command_text
+ budget = _MAX_HEREDOC_OPERANDS
+ per_line = _MAX_HEREDOCS_PER_LINE.get(reading, _MAX_HEREDOC_OPERANDS)
+ while budget > 0:
+ position = _heredoc_operator_index(remaining)
+ if position < 0:
+ break
+ commands.append(remaining[:position])
+ rest = remaining[position + 2 :]
+ if rest.startswith("<"):
+ # Here-string: one operand, then normal commands resume.
+ operand = rest[1:].lstrip()
+ cut = _unquoted_separator_index(operand)
+ data.append(operand[:cut])
+ remaining = " " + operand[cut:]
+ budget -= 1
+ here_strings += 1
+ continue
+ line_end = rest.find("\n")
+ header = _heredoc_header(rest if line_end == -1 else rest[:line_end])
+ if header is None:
+ # Nothing names the end of a body, so the rest of the text is data.
+ data.append(rest)
+ remaining = ""
+ break
+ # The rest of the operator's own line still belongs to its commands,
+ # and every heredoc declared on it has its body read, in order, once
+ # the line ends: ``python3 <<A <<B run.py`` reads A's body, then B's.
+ header_text, header_data, terminators = header
+ budget -= len(terminators) + len(header_data)
+ if len(terminators) > per_line or budget < 0:
+ # The shell refuses the line, or which lines are bodies is past
+ # what is read here: the line, from its start, and all after it
+ # are data, so a script named there is unresolved.
+ walked = "".join(commands)
+ line_start = walked.rfind("\n") + 1
+ commands = [walked[:line_start]]
+ data.append(walked[line_start:] + "<<" + rest)
+ remaining = ""
+ break
+ commands.append(header_text)
+ data.extend(header_data)
+ here_strings += len(header_data)
+ if line_end == -1:
+ # The bodies never start, so what follows the delimiters is command.
+ remaining = ""
+ break
+ resumed = rest[line_end + 1 :]
+ for terminator, strip_tabs in terminators:
+ body, resumed = _split_heredoc_body(resumed, terminator, strip_tabs)
+ data.append(body)
+ remaining = "\n" + resumed
+ if budget <= 0 and _heredoc_operator_index(remaining) >= 0:
+ # More operators than are read: what is left may be bodies, so it is
+ # data rather than commands.
+ data.append(remaining)
+ remaining = ""
+ commands.append(remaining)
+ return "".join(commands), "\n".join(data), here_strings
+
+
+def _heredoc_header(line: str) -> tuple[str, list[str], list[tuple[str, bool]]] | None:
+ """Read the heredocs declared on one line, the first ``<<`` already consumed.
+
+ Returns the line's command text with each operator and delimiter removed,
+ the here-string operands on it, and each heredoc's terminator with whether
+ its ``<<-`` strips leading tabs, in the order the shell reads their bodies.
+ ``None`` when a ``<<`` names no terminator. Measured on bash, dash, zsh,
+ ksh, mksh and busybox ash: every heredoc on a line, across commands joined
+ by ``;`` or ``|`` included, takes its body after the line, in order.
+ """
+ pieces: list[str] = []
+ strings: list[str] = []
+ terminators: list[tuple[str, bool]] = []
+ rest = line
+ while True:
+ delimiter = _HEREDOC_DELIMITER_RE.match(rest)
+ if delimiter is None:
+ return None
+ terminators.append((delimiter.group(2).strip("\"'").replace("\\", ""), bool(delimiter.group(1))))
+ rest = " " + rest[delimiter.end() :]
+ while True:
+ position = _heredoc_operator_index(rest)
+ if position < 0:
+ pieces.append(rest)
+ return "".join(pieces), strings, terminators
+ pieces.append(rest[:position])
+ rest = rest[position + 2 :]
+ if not rest.startswith("<"):
+ break
+ # Here-string: one operand, then the line resumes.
+ operand = rest[1:].lstrip()
+ cut = _unquoted_separator_index(operand)
+ strings.append(operand[:cut])
+ rest = " " + operand[cut:]
+
+
+def _is_redirection_operator(token: str) -> bool:
+ return _is_output_redirect(token) or _is_heredoc_redirect(token) or token in _INPUT_REDIRECT_OPERATORS
+
+
+# A redirection operator as the tokenizer hands it over: a word of its own,
+# made only of ``<``, ``>``, ``&`` and ``|`` (zsh's ``>>|`` and ``&>|`` among
+# them), with a descriptor only where one was written flush against it
+# (``2>``, kept with its operator before tokenizing). An unquoted operator
+# never stays inside a word, so a word that ends or starts with one (``g=>``
+# from ``g='>'``, ``>x`` from ``'>x'``) is data, and a quoted operator
+# (``'>'``) carries the mark.
+_REDIRECTION_WORD_RE = re.compile(r"\d*[<>&|]*[<>][<>&|]*")
+
+
+def _without_redirections(words: list[str]) -> list[str]:
+ """A command's words with each redirection and its operand removed, as the
+ shell removes them: ``export -p > f=other.py`` writes a listing to a file
+ named f=other.py and binds nothing, and ``export g='>' f=run.py`` binds
+ both names."""
+ kept: list[str] = []
+ skip_next = False
+ for word in words:
+ token = str(word)
+ if skip_next:
+ skip_next = False
+ elif _REDIRECTION_WORD_RE.fullmatch(token):
+ skip_next = True
+ else:
+ kept.append(token)
+ return kept
+
+
+def _interpreter_operands(command: list[str], cmd_idx: int, assignments: dict[str, str]) -> list[str]:
+ """The interpreter's own arguments, with every redirection removed.
+
+ ``python3 < /dev/null run.py`` runs run.py: the redirection belongs to the
+ shell, not to python's argument list. The tokenizer splits an operator and
+ its operand into their own tokens and keeps a descriptor with its operator
+ (``2>``), so none of them may stand where the first operand is looked for.
+ A digit on its own is an operand: ``python3 2 > out.txt run.py`` runs a
+ script named ``2`` and hands it ``run.py``. A script fed through standard input is a different
+ shape, and :func:`_redirects_script_to_stdin` has already answered for it
+ before this is reached.
+ """
+ args: list[str] = []
+ skip_next = False
+ words = command[cmd_idx + 1 :]
+ for arg in words:
+ if skip_next:
+ skip_next = False
+ continue
+ token = str(arg)
+ if _is_heredoc_redirect(token):
+ break
+ if _is_output_redirect(token) or token in _INPUT_REDIRECT_OPERATORS or _DESCRIPTOR_OPERATOR_RE.match(token):
+ skip_next = True
+ continue
+ if _ATTACHED_REDIRECT_RE.match(token):
+ continue
+ args.append(_resolved_shell_arg(arg, assignments))
+ return args
+
+
+def _runs_inline_code(executable: str, command: list[str], cmd_idx: int, assignments: dict[str, str]) -> bool:
+ """Whether a shell's option prefix carries its inline-code option.
+
+ Only the options before the first script operand are the shell's own. In
+ ``bash run.sh -c 'echo done'`` the ``-c`` is run.sh's argument, and after
+ ``--`` nothing is an option at all, so the grammar walk that finds the
+ script operand decides this rather than a scan of every argument.
+ ``_shell_c_payload`` accepts any option containing the letter ``c``, which
+ also matches ``--check`` and ``-Mstrict``, so it is not consulted first.
+ """
+ status, _ = _interpreter_script_arg(executable, command, cmd_idx, assignments)
+ return status == _INLINE_CODE
+
+
+def _reads_program_from_stdin(interpreter, command, cmd_idx, assignments):
+ """An interpreter given no script, no inline code and no terminal option
+ reads its program from standard input; ``-`` names standard input.
+ """
+ grammar = _INTERPRETER_GRAMMARS.get(interpreter)
+ if grammar is None:
+ return False
+ for arg in _interpreter_operands(command, cmd_idx, assignments):
+ word = str(arg).strip("\"'")
+ if word == "-":
+ continue
+ if word in grammar["code"] or word in grammar["terminal"] or word in grammar["value"]:
+ return False
+ if not word.startswith("-"):
+ return False
+ return True
+
+
+def _pipeline_upstream_names_script(tokens, idx, assignments, expected):
+ """Whether an earlier stage of the pipeline that the segment at ``idx``
+ ends names the expected script as one of its words.
+ """
+ position = idx - 1
+ while position >= 0:
+ token = tokens[position]
+ if token in _SHELL_SEPARATORS and token != "|":
+ return False
+ if token != "|" and _script_path_matches(_resolved_shell_arg(token, assignments), expected):
+ return True
+ position -= 1
+ return False
+
+
+def _interpreter_script_arg(
+ executable: str,
+ command: list[str],
+ cmd_idx: int,
+ assignments: dict[str, str],
+) -> tuple[str, str | None]:
+ """Resolve which argument an interpreter runs as a script.
+
+ Returns ``(_SCRIPT, arg)`` when one is named, ``(_NO_SCRIPT, None)`` when
+ the interpreter runs inline code, prints and exits, or only checks syntax,
+ and ``(_UNDECIDABLE, None)`` when any option before the candidate is not in
+ this interpreter's grammar. Only the first non-option argument is the
+ script: anything after it is that script's own argv, so
+ ``python other.py run.py`` runs ``other.py``.
+ """
+ base = _VERSION_SUFFIX_RE.sub("", executable) or executable
+ grammar = _INTERPRETER_GRAMMARS.get(base)
+ args = _interpreter_operands(command, cmd_idx, assignments)
+ if grammar is None:
+ # Sourcing and anything else without a grammar: the first argument is
+ # the file, and an option would mean a shape this walk does not model.
+ first = next((a for a in args if a), None)
+ if first is None:
+ return (_NO_SCRIPT, None)
+ return (_UNDECIDABLE, None) if str(first).strip("\"'").startswith("-") else (_SCRIPT, first)
+
+ index = 0
+ while index < len(args):
+ raw = args[index]
+ token = str(raw).strip("\"'")
+ index += 1
+ if token == "--":
+ return (_SCRIPT, args[index]) if index < len(args) else (_NO_SCRIPT, None)
+ if not token.startswith("-") or token == "-":
+ return (_SCRIPT, raw)
+ if token in grammar["code"]:
+ return (_INLINE_CODE, None)
+ if token in grammar["terminal"]:
+ return (_NO_SCRIPT, None)
+ if token in grammar["boolean"]:
+ continue
+ if token in grammar["value"]:
+ index += 1
+ continue
+ name, separator, _ = token.partition("=")
+ if separator and (name in grammar["value"] or name in grammar["boolean"]):
+ continue
+ if not token.startswith("--"):
+ # A short-option cluster, possibly carrying an attached value.
+ letters = token[1:]
+ short = {
+ kind: {o[1] for o in options if len(o) == 2 and not o.startswith("--")}
+ for kind, options in grammar.items()
+ }
+ recognised = True
+ for position, letter in enumerate(letters):
+ if letter in short["code"]:
+ return (_INLINE_CODE, None)
+ if letter in short["terminal"]:
+ return (_NO_SCRIPT, None)
+ if letter in short["value"]:
+ if position + 1 == len(letters):
+ index += 1
+ break
+ if letter not in short["boolean"]:
+ recognised = False
+ break
+ if recognised:
+ continue
+ return (_UNDECIDABLE, None)
+ return (_NO_SCRIPT, None)
+
+
+def _skip_wrapper_options(
+ wrapper: str,
+ command: list[str],
+ cmd_idx: int,
+ assignments: dict[str, str],
+) -> tuple[str, int]:
+ """Advance past one wrapper's own options using its grammar.
+
+ Returns ``(_WRAPPER_OK, index)`` at the wrapped command, ``_WRAPPER_NONE``
+ when a help or version option means the wrapper printed and exited, and
+ ``_WRAPPER_UNKNOWN`` for an option the grammar does not describe, because
+ assuming it takes no value is how a wrapper ends up crediting a command
+ that never ran.
+ """
+ grammar = _WRAPPER_GRAMMARS.get(wrapper)
+ if grammar is None:
+ return (_WRAPPER_UNKNOWN, cmd_idx)
+ index = cmd_idx + 1
+ options_done = False
+ while index < len(command):
+ token = _resolved_shell_arg(command[index], assignments).strip("\"'")
+ if not options_done and token in grammar["terminal"]:
+ return (_WRAPPER_NONE, index)
+ if wrapper in _STDIN_DEPENDENT_WRAPPERS:
+ return (_WRAPPER_UNKNOWN, index)
+ if token == "--":
+ # End of this wrapper's own OPTIONS. Its positional arguments, such
+ # as timeout's duration, still come before the wrapped command.
+ options_done = True
+ index += 1
+ continue
+ if not options_done:
+ if token in grammar["boolean"]:
+ index += 1
+ continue
+ if token in grammar["value"]:
+ index += 2
+ continue
+ name, separator, _ = token.partition("=")
+ if separator and (name in grammar["value"] or name in grammar["boolean"]):
+ index += 1
+ continue
+ if wrapper == "nice" and _NEGATIVE_NUMBER_RE.match(token):
+ index += 1
+ continue
+ assignment = _SHELL_ASSIGNMENT_RE.match(token)
+ if wrapper == "env" and assignment:
+ assignments[assignment.group(1)] = assignment.group(2)
+ index += 1
+ continue
+ if not token.startswith("-") or options_done:
+ if wrapper == "timeout" and _DURATION_ARG_RE.match(token):
+ index += 1
+ continue
+ if wrapper in _RUNNER_COMMAND_PREFIXES:
+ return (_WRAPPER_OK, index + 1) if token == "run" else (_WRAPPER_UNKNOWN, index)
+ return (_WRAPPER_OK, index)
+ if len(token) > 2 and not token.startswith("--"):
+ short = token[:2]
+ if short in grammar["value"]:
+ index += 1
+ continue
+ return (_WRAPPER_UNKNOWN, index)
+ return (_WRAPPER_NONE, index)
+
+
+def _skip_transparent_prefixes(
+ command: list[str],
+ cmd_idx: int,
+ assignments: dict[str, str],
+) -> tuple[str, int]:
+ """Advance past grouping tokens and wrappers that run the command after them."""
+ for _ in range(_MAX_SHELL_WRAPPERS):
+ if cmd_idx >= len(command):
+ return (_WRAPPER_OK, cmd_idx)
+ if command[cmd_idx] in _GROUPING_TOKENS or command[cmd_idx] in _COMMAND_INTRODUCING_WORDS:
+ cmd_idx += 1
+ continue
+ executable = _shell_executable(_resolved_shell_arg(command[cmd_idx], assignments)).removesuffix(".exe")
+ if executable in _TRANSPARENT_COMMAND_PREFIXES or executable in _RUNNER_COMMAND_PREFIXES:
+ status, cmd_idx = _skip_wrapper_options(executable, command, cmd_idx, assignments)
+ if status != _WRAPPER_OK:
+ return (status, cmd_idx)
+ continue
+ return (_WRAPPER_OK, cmd_idx)
+ return (_WRAPPER_OK, cmd_idx)
+
+
+def _double_parens_are_arithmetic(shell: str | None) -> bool | None:
+ """Whether ``((`` closed by ``))`` is an arithmetic command in this shell.
+
+ bash, zsh, ksh and mksh have one (measured); dash has none and reads two
+ subshells. ``sh`` and ``ash`` are dash on some systems and bash or
+ busybox on others, so the text does not settle it. The tool call's own
+ shell is read as bash.
+ """
+ if shell == "dash":
+ return False
+ if shell in {"sh", "ash"}:
+ return None
+ return True
+
+
+def _pipeline_last_stage_keeps_bindings(shell: str | None, command_text: str) -> bool | None:
+ """Whether a binding made in a pipeline's last command survives it.
+
+ ``True`` and ``False`` when the shell settles it; ``None`` when the text
+ does not: an unmodelled shell, or an option change named in the text
+ (``shopt -s lastpipe`` in bash, ``emulate sh`` in zsh) that moves that
+ stage between the two rules at run time.
+ """
+ if _PIPELINE_OPTION_RE.search(command_text):
+ return None
+ if shell in _LAST_STAGE_IN_CURRENT_SHELL:
+ return True
+ if shell is None or shell in _LAST_STAGE_IN_SUBSHELL:
+ # The tool's own shell is read as bash, which is what runs an
+ # agent's command and what the differential harness executes.
+ return False
+ return None
+
+
+def _cmd_executes_script(
+ cmd: Any,
+ expected_script: str,
+ *,
+ _depth: int = 0,
+ _shell: str | None = None,
+ _positional: bool = False,
+ _environment: dict[str, str] | None = None,
+) -> bool | None:
+ """Whether a shell command invokes ``expected_script``.
+
+ ``_shell`` names the interpreter running this text, when it is known:
+ a ``-c`` payload carries its shell, and a tool call may name its own
+ (``_tool_call_shell``); a text with neither is read as bash.
+ ``_environment`` is what a payload's shell inherits: the names exported
+ to it and the prefix assignments of the command that started it.
+ Three rules in the walk depend on the shell: whether the last command of
+ a pipeline keeps its bindings, whether ``((`` is arithmetic, and how
+ variables are bound (which declaring builtins exist, and whether an
+ assignment before a special builtin outlives it). Where the shell does
+ not settle one, the walk runs under every reading, and disagreement is
+ unresolved rather than one shell's answer presented as every shell's.
+ """
+ keep = _pipeline_last_stage_keeps_bindings(_shell, str(cmd))
+ arithmetic = _double_parens_are_arithmetic(_shell)
+ results = {
+ _walk_for_invocation(
+ cmd, expected_script, _depth, keep_stage, _positional, arithmetic_parens, binding_reading, _environment
+ )
+ for keep_stage in ([keep] if keep is not None else [False, True])
+ for arithmetic_parens in ([arithmetic] if arithmetic is not None else [True, False])
+ for binding_reading in _binding_readings(_shell)
+ }
+ return results.pop() if len(results) == 1 else None
+
+
+def _walk_for_invocation(
+ cmd: Any,
+ expected_script: str,
+ _depth: int,
+ keep_last_stage: bool,
+ positional: bool,
+ arithmetic_parens: bool,
+ binding_reading: str = "bash",
+ environment: dict[str, str] | None = None,
+ own_commands_only: bool = False,
+) -> bool | None:
+ """Whether a shell command invokes ``expected_script``.
+
+ Credit is given only for a recognised way of running a script: the script
+ invoked directly, an interpreter given it as its script argument, a
+ ``source``, or a ``sh -c`` payload that does one of those. Every other
+ command is reported as not an invocation, so reading, printing, searching,
+ copying or deleting the file needs no special case. ``own_commands_only``
+ leaves a ``-c`` payload unresolved, for a command credited before a line
+ its shell refuses: the payload's own text may be refused too.
+
+ ``None`` means undecidable rather than negative, and is returned only when
+ the shell resolves the path at run time or an unrecognised option may have
+ consumed it. The same "a textual match is not evidence" reasoning is already
+ applied to SKILL.md reads by :func:`_cmd_reads_skill_md`.
+ """
+ if not expected_script:
+ return False
+ if _depth > _MAX_SHELL_REFERENCE_DEPTH:
+ return None
+ command_text = str(cmd)
+ if not command_text.strip():
+ return False
+ if not _names_script_anywhere(command_text, expected_script) and not (
+ _depth > 0 and _SHELL_VARIABLE_RE.search(command_text)
+ ):
+ # An unresolved walk over a command that never names the script is
+ # not evidence about that script, so it is a non-invocation, as it was
+ # before invocation evidence was required. A ``-c`` payload is the
+ # exception when it reads a variable: the command around it names the
+ # script, and ``python3 "$f"`` in the child may be given it through
+ # the environment.
+ return False
+ # A heredoc or here-string operand is data rather than further commands, but
+ # the tokenizer turns its newlines into separators, so it is split out.
+ analysed_text, unexamined_text, here_strings = _split_heredocs(command_text, binding_reading)
+ tokens = _split_punctuation_runs(
+ _shell_tokens(_mark_quoted_newlines(_mark_literal_dollars(analysed_text))),
+ arithmetic_parens,
+ binding_reading != "ksh",
+ )
+ # A shell with no here-string rejects the text before running it, and so
+ # does every shell given a compound's opening word after an assignment.
+ here_string_refused = bool(here_strings) and binding_reading in _NO_HERE_STRING_SHELLS
+ syntax_rejected = here_string_refused or _compound_after_assignment(tokens)
+ if syntax_rejected and _depth == 0 and binding_reading not in _PARSES_WHOLE_TEXT_FIRST:
+ # This shell ran the text's first command before the one it refuses.
+ # When nothing the walk misreads can have stopped it, an invocation
+ # there counts as it would on its own (``_command_run_before_refusal``).
+ # Only in the text a tool call gives: a ``-c`` payload is what the
+ # tokenizer left of it (a carriage return there reads as a line's end).
+ first = _command_run_before_refusal(command_text, binding_reading, arithmetic_parens)
+ if first and (
+ _walk_for_invocation(
+ first,
+ expected_script,
+ _depth,
+ keep_last_stage,
+ positional,
+ arithmetic_parens,
+ binding_reading,
+ environment,
+ own_commands_only=True,
+ )
+ is True
+ ):
+ return True
+ if not tokens:
+ return None if str(expected_script) in command_text else False
+
+ assignments: dict[str, str] = {}
+ for name, value in (environment or {}).items():
+ # Inherited, and exported onward to any shell this one starts.
+ assignments[name] = value
+ assignments[_EXPORTED_MARK + name] = "1"
+ # Every scope a command can run in, innermost last. A ``( ... )`` group
+ # and a compound command isolated as a pipeline stage each run in a
+ # subshell, with a copy of the bindings around them that is dropped
+ # when they end. A frame is (kind, bindings, the compound depth it
+ # opened at): "group" for ``(``, "stage" for a compound stage. A
+ # command reads and binds the innermost frame, and a frame opened
+ # inside another copies that one, so groups and stages nest in either
+ # order.
+ frames: list[tuple[str, dict[str, str], int]] = []
+ compound_depth = 0
+ closing_parens = 0
+
+ def innermost() -> dict[str, str]:
+ return frames[-1][1] if frames else assignments
+
+ scope = assignments
+
+ def credited() -> bool:
+ """Whether an invocation found here counts: not once an assignment the
+ shell stops at has failed before it (see ``_ASSIGNMENT_REFUSED``), nor
+ in a text the shell rejects (``_NO_HERE_STRING_SHELLS``,
+ ``_compound_after_assignment``); the first command it ran before
+ rejecting it may be credited above (``_command_run_before_refusal``)."""
+ return not scope.get(_ASSIGNMENT_REFUSED) and not syntax_rejected
+
+ def close_frames(parens: int) -> None:
+ """Drop what ended with the previous segment, innermost first: a
+ stage whose compound has closed, and a group for each ``)``."""
+ while frames:
+ kind, _, opened_at = frames[-1]
+ if kind == "stage" and compound_depth <= opened_at:
+ frames.pop()
+ elif kind == "group" and parens > 0:
+ frames.pop()
+ parens -= 1
+ else:
+ break
+
+ positional_known_empty = not positional and not _POSITIONAL_SET_RE.search(command_text)
+ current_directory: str | None = None
+ undecidable = False
+ ran_a_wrapper_help = False
+ idx = 0
+ while idx < len(tokens):
+ if tokens[idx] in _SHELL_SEPARATORS:
+ idx += 1
+ continue
+ end = idx
+ while end < len(tokens) and tokens[end] not in _SHELL_SEPARATORS:
+ end += 1
+ command = tokens[idx:end]
+ close_frames(closing_parens)
+ closing_parens = 0
+ # A pipeline runs each of its commands in a subshell, so a binding
+ # made in one does not survive past the pipeline:
+ # ``printf '' | for f in run.py; do cat $f; done; python3 $f`` runs
+ # python3 with an empty ``$f``. Each group and compound this segment
+ # opens is placed in turn: a ``(`` always runs in a subshell, and a
+ # compound runs in one when it is itself a pipeline stage, piped from
+ # (the segment's first command) or piped into (its own closing word
+ # is followed by ``|``), except as the last stage where the shell
+ # keeps it. Each such scope copies the innermost one around it and
+ # lasts until its closing word, so its own body sees its bindings.
+ position_words = _command_position_words(command)
+ closes = sum(word in _COMPOUND_CLOSERS for word in position_words)
+ compound_depth -= closes
+ events = _scope_events(command)
+ opener_count = sum(event != "(" for event in events)
+ closer_ends = _compound_closer_ends(tokens, end, opener_count)
+ piped_from = idx > 0 and tokens[idx - 1] == "|"
+ piped_into = end < len(tokens) and tokens[end] == "|"
+ lead = next((word for word in position_words if word != "!"), None)
+ opened = 0
+ # ``$((`` reaches the walk as ``$ ( (``, which opens groups here like
+ # ``$(`` does. Its arithmetic runs in the shell around them, so it
+ # binds in the scope in force before the first ``(`` after a ``$``.
+ parens = [position for position, token in enumerate(command) if token == "("]
+ expansion_opens = next(
+ (count for count, position in enumerate(parens) if position and str(command[position - 1]).endswith("$")),
+ None,
+ )
+ arithmetic_scope = None
+ for event in events:
+ if event == "(":
+ if expansion_opens is not None and arithmetic_scope is None:
+ if expansion_opens == 0:
+ arithmetic_scope = innermost()
+ expansion_opens -= 1
+ frames.append(("group", dict(innermost()), compound_depth))
+ continue
+ compound_end = closer_ends[opened]
+ feeds = compound_end < len(tokens) and tokens[compound_end] == "|"
+ fed = opened == 0 and piped_from and lead == event
+ if feeds or (fed and not keep_last_stage):
+ frames.append(("stage", dict(innermost()), compound_depth))
+ compound_depth += 1
+ opened += 1
+ closing_parens = command.count(")")
+ if opened:
+ # The command after the opening words runs inside the innermost
+ # compound, so a pipe after this segment is that command's.
+ in_pipeline = piped_into
+ else:
+ # A pipe before ``(`` belongs to the group, not to the command
+ # inside it, which runs in the group's own scope.
+ preceded = piped_from and command[0] != "("
+ in_pipeline = preceded or piped_into
+ if keep_last_stage and preceded and not piped_into:
+ # The last command of the pipeline runs in the current
+ # shell here (zsh, ksh), so what it binds is kept.
+ in_pipeline = False
+ scope = dict(innermost()) if in_pipeline else innermost()
+ _apply_arithmetic_assignments(
+ command,
+ scope if in_pipeline or arithmetic_scope is None else arithmetic_scope,
+ expected_script,
+ "$((" in analysed_text or "$[" in analysed_text,
+ )
+ if "((" in command:
+ idx = end + 1
+ continue
+
+ prefix: dict[str, str] = {}
+ cmd_idx = _command_start(command, prefix)
+ if cmd_idx >= len(command):
+ # Assignments alone bind in this shell; a bare reserved word runs
+ # nothing. Neither runs a script.
+ for name, value in prefix.items():
+ _bind(scope, name, _value_now(value, scope), binding_reading)
+ idx = end + 1
+ continue
+ if _resolved_shell_arg(command[cmd_idx], scope).strip("\"'") in _SPECIAL_BUILTINS and (
+ binding_reading in _PREFIX_OUTLIVES_SPECIAL_BUILTIN
+ ):
+ for name, value in prefix.items():
+ _bind(scope, name, _value_now(value, scope), binding_reading, "prefix-special")
+ else:
+ for name, value in prefix.items():
+ if scope.get(_READONLY_MARK + name):
+ # The assignment fails; bash and ksh still run the command
+ # and go on, the other shells stop.
+ _refuse_assignment(scope, name, binding_reading, "prefix")
+ elif "i" in scope.get(_ATTRIBUTE_MARK + name, "") and _integer_may_fail(
+ _value_now(value, scope), scope
+ ):
+ _refuse_assignment(scope, name, binding_reading, "prefix", _INTEGER_ERROR_GOES_ON)
+ if _apply_binding_builtin(command, cmd_idx, scope, binding_reading):
+ idx = end + 1
+ continue
+ openers = [event for event in events if event != "("]
+ if command[cmd_idx] in _LOOP_HEADER_WORDS and openers and openers[-1] == command[cmd_idx]:
+ # Only a loop word where a command starts opens a loop. After an
+ # assignment or ``--`` it is an ordinary word with no closer to
+ # skip to (``_compound_after_assignment``).
+ if _loop_header_is_empty(command, cmd_idx, scope, positional_known_empty):
+ # Zero iterations: the body is skipped whole, and the
+ # closing word it ends with is accounted for here.
+ # The loop is the last compound this segment opens; any before it
+ # encloses it and stays open.
+ skip_to = closer_ends[-1]
+ for token in tokens[end:skip_to]:
+ # Nothing in the body runs, but a parenthesis in it still
+ # opens or closes a subshell around what follows.
+ if token == "(":
+ frames.append(("group", dict(innermost()), compound_depth))
+ elif token == ")":
+ closing_parens += 1
+ # The skipped tokens hold the loop's own closing word; any
+ # compound opened inside the body closes there too.
+ compound_depth -= 1
+ idx = skip_to
+ continue
+ # The header runs nothing itself. With a single value the loop
+ # variable is bound for the body that follows; otherwise a script
+ # named here is unresolved, never a settled non-invocation.
+ if _loop_header_is_unresolved(command, cmd_idx, scope, expected_script, binding_reading):
+ undecidable = True
+ idx = end + 1
+ continue
+ if command[cmd_idx] in _UNMODELLED_CONTROL_WORDS:
+ # `case` is not modelled, so what its bodies do with the script
+ # this text names is not settled either way.
+ undecidable = True
+ idx = end + 1
+ continue
+
+ if _redirects_script_to_stdin(command, scope, expected_script):
+ undecidable = True
+ idx = end + 1
+ continue
+ leading = _shell_executable(_resolved_shell_arg(command[cmd_idx], scope)).removesuffix(".exe")
+ # Where the words that start this command begin: the wrappers between
+ # here and the command decide what it inherits (``_child_environment``).
+ # They are skipped over copies of the bindings, so what ``env`` gives
+ # the command never reaches this shell or the command's own words.
+ start_idx = cmd_idx
+ if leading in _WRAPPER_GRAMMARS:
+ wrapper_status, cmd_idx = _skip_wrapper_options(leading, command, cmd_idx, dict(scope))
+ if wrapper_status != _WRAPPER_OK:
+ if wrapper_status == _WRAPPER_NONE:
+ ran_a_wrapper_help = True
+ if wrapper_status == _WRAPPER_UNKNOWN:
+ undecidable = True
+ idx = end + 1
+ continue
+ unwrapped_idx = _unwrap_shell_command(command, cmd_idx, dict(scope))
+ if unwrapped_idx is None:
+ undecidable = True
+ idx = end + 1
+ continue
+ wrapper_status, cmd_idx = _skip_transparent_prefixes(command, unwrapped_idx, scope)
+ if wrapper_status != _WRAPPER_OK:
+ if wrapper_status == _WRAPPER_NONE:
+ ran_a_wrapper_help = True
+ if wrapper_status == _WRAPPER_UNKNOWN:
+ undecidable = True
+ idx = end + 1
+ continue
+
+ if cmd_idx < len(command):
+ if _resolved_shell_arg(command[cmd_idx], scope).strip("\"'").startswith("-"):
+ # An option standing where a command should: an option outside
+ # the wrapper's grammar consumed the tokens up to here, so what
+ # runs is unresolved rather than nothing.
+ undecidable = True
+ idx = end + 1
+ continue
+ executable_path = _path_with_shell_cwd(
+ _resolved_shell_arg(command[cmd_idx], scope),
+ current_directory,
+ )
+ if _UNSETTLED_VALUE in executable_path:
+ undecidable = True
+ idx = end + 1
+ continue
+ executable = _shell_executable(executable_path)
+
+ if executable == "cd":
+ directory = next(
+ (arg for arg in _command_input_args(command, cmd_idx, scope) if arg and not arg.startswith("-")),
+ None,
+ )
+ if directory is None:
+ undecidable = True
+ else:
+ current_directory = directory
+ idx = end + 1
+ continue
+
+ # `perl5.38.2` and `python3.13` run the same scripts as `perl` and
+ # `python`, so the version suffix is dropped before every lookup.
+ interpreter = _VERSION_SUFFIX_RE.sub("", executable) or executable
+ if interpreter in _SHELL_COMMAND_INTERPRETERS and _runs_inline_code(executable, command, cmd_idx, scope):
+ payload = _shell_c_payload(command, cmd_idx, scope)
+ if payload is not None and own_commands_only:
+ undecidable = True
+ idx = end + 1
+ continue
+ if payload is not None:
+ # This shell expands what it did not leave quoted, a name
+ # it never bound to nothing, before the child reads the rest.
+ payload = _value_now(payload, scope).replace(_LITERAL_DOLLAR, "$")
+ # After the payload, the first operand is its $0 and the rest
+ # its positional parameters; they reach the payload only
+ # through a reference to them in its text.
+ after = _shell_c_positional(command, cmd_idx, scope)
+ argv0, params = after[:1], after[1:]
+ child_environment = _child_environment(command, start_idx, cmd_idx, scope, prefix, binding_reading)
+ nested = _cmd_executes_script(
+ payload,
+ expected_script,
+ _depth=_depth + 1,
+ _shell=interpreter,
+ _positional=bool(params),
+ _environment=child_environment or {},
+ )
+ if nested is True:
+ if credited():
+ return True
+ undecidable = True
+ reaches = (
+ _READS_POSITIONAL_RE.search(payload)
+ and any(_script_path_matches(w, expected_script) for w in params)
+ ) or (
+ _READS_ARGV0_RE.search(payload) and any(_script_path_matches(w, expected_script) for w in argv0)
+ )
+ if nested is None or reaches or (child_environment is None and _SHELL_VARIABLE_RE.search(payload)):
+ # A payload reading a variable after ``sudo`` or ``env -S``
+ # reads what the text does not carry.
+ undecidable = True
+ idx = end + 1
+ continue
+
+ if executable in _OPAQUE_SHELL_BUILTINS:
+ # ``eval`` reads its words again in this shell, with the
+ # command's prefix in effect: ``f=run.py eval 'python3 "$f"'``.
+ if _command_names_script(
+ command,
+ cmd_idx,
+ _reader_scope(command, start_idx, cmd_idx, scope, prefix, binding_reading),
+ expected_script,
+ ):
+ undecidable = True
+ _apply_eval_bindings(command[cmd_idx + 1 :], scope, binding_reading)
+ idx = end + 1
+ continue
+
+ if _script_path_matches(executable_path, expected_script):
+ if credited():
+ return True
+ undecidable = True
+ idx = end + 1
+ continue
+
+ runs_a_script = interpreter in _INTERPRETER_GRAMMARS or interpreter in _SOURCING_COMMANDS
+ if not runs_a_script and _carries_a_nested_invocation(command, cmd_idx, scope, expected_script):
+ undecidable = True
+ idx = end + 1
+ continue
+ if runs_a_script:
+ status, script_arg = _interpreter_script_arg(executable, command, cmd_idx, scope)
+ if status == _UNDECIDABLE:
+ undecidable = True
+ elif status == _INLINE_CODE:
+ # Inline code or a module can run the script itself, which
+ # this walk does not read, so naming it leaves the command
+ # unresolved rather than settled as running nothing.
+ if _command_names_script(
+ command,
+ cmd_idx,
+ _reader_scope(command, start_idx, cmd_idx, scope, prefix, binding_reading),
+ expected_script,
+ ):
+ undecidable = True
+ elif status == _SCRIPT and script_arg is not None and str(script_arg).strip("\"'") != "-":
+ if _script_path_matches(_path_with_shell_cwd(script_arg, current_directory), expected_script):
+ if credited():
+ return True
+ undecidable = True
+ elif _UNRESOLVED_ARG_RE.search(str(script_arg)) or _unresolved_value(str(script_arg)):
+ undecidable = True
+ elif (
+ status in (_NO_SCRIPT, _SCRIPT)
+ and idx > 0
+ and tokens[idx - 1] == "|"
+ and _reads_program_from_stdin(interpreter, command, cmd_idx, scope)
+ and _pipeline_upstream_names_script(tokens, idx, scope, expected_script)
+ ):
+ # ``cat run.py | python3`` runs the script and ``cat run.py | wc -l``
+ # does not, but an interpreter reading its program from standard
+ # input is the same shape as ``python3 < run.py``: unresolved.
+ undecidable = True
+
+ idx = end + 1
+
+ if undecidable:
+ return None
+ if expected_script in unexamined_text or expected_script in _value_now(unexamined_text, innermost()):
+ # Named only in data this walk did not read as commands, directly or
+ # through a variable the data reads: this shell expands an unquoted
+ # heredoc body, and a shell reading the data expands what it inherits.
+ return None
+ if ran_a_wrapper_help and not undecidable:
+ # A wrapper printed its help and exited, so nothing ran.
+ return False
+ if expected_script in command_text and _STDIN_ARGV_RE.search(command_text):
+ # A path can reach xargs through standard input rather than as an
+ # argument, so neither answer is supported by the command text.
+ return None
+ if expected_script in command_text and _DYNAMIC_ARGUMENT_RE.search(command_text):
+ # The shell builds the path at run time, so neither answer is supported.
+ return None
+ return False
+
+
+def _tool_call_shell(tool_call: dict[str, Any]) -> tuple[str | None, bool]:
+ """The shell a tool call says ran its command, and whether this walk models it.
+
+ A native call can name its shell (Codex's ``exec_command`` keeps
+ ``shell``, such as ``/bin/zsh``, in its arguments), and the rules that
+ differ between shells then follow it. A call that names none is read as
+ bash, as elsewhere in this walk: ``(None, True)``. The name is taken
+ without its directory, a ``.exe`` suffix or a version number, so
+ ``/usr/local/bin/bash5.2`` is bash; one outside the modelled shells, or a
+ value that is not a name, is ``(name, False)``.
+ """
+ shell = _action_args(tool_call).get("shell")
+ if shell is None or (isinstance(shell, str) and not shell.strip()):
+ return None, True
+ if not isinstance(shell, str):
+ return str(shell), False
+ name = shell.split()[0].replace("\\", "/").rsplit("/", 1)[-1].lower().removesuffix(".exe")
+ name = _VERSION_SUFFIX_RE.sub("", name) or name
+ return name, name in _SHELL_COMMAND_INTERPRETERS
+
+
+def check_script_execution(
+ tool_calls: list[dict[str, Any]],
+ expected_script: str | None,
+) -> dict[str, Any]:
+ """Check whether the agent executed the expected script."""
+ if not expected_script:
+ return {"passed": True, "score": 1.0, "reason": "No specific script expected"}
+
+ exec_calls = [tc for tc in tool_calls if _is_execution_action(str(tc["action"]))]
+ unclassified_reference = False
+ for call in exec_calls:
+ command = _command_text(call)
+ shell, modelled = _tool_call_shell(call)
+ if modelled:
+ verdict = _cmd_executes_script(command, expected_script, _shell=shell)
+ else:
+ # A shell this walk does not model reads the text by rules it does
+ # not know, so the text settles nothing: a command naming the
+ # script is unresolved, the same reference test the base applied.
+ verdict = None if expected_script in command else False
+ if verdict is True:
+ return {"passed": True, "score": 1.0, "reason": f"Executed {expected_script}"}
+ if verdict is None:
+ unclassified_reference = True
+
+ if _has_unsupported_native_codex_call(tool_calls):
+ return _unsupported_native_codex_result(
+ "Script execution could not be evaluated because a native Codex exec wrapper was unsupported"
+ )
+
+ if unclassified_reference:
+ return {
+ "passed": True,
+ "score": 0.75,
+ "reason": f"{expected_script} referenced in a command that could not be classified as an invocation",
+ }
+
+ if not exec_calls:
+ # Check observation text as fallback (script may run inside Skill tool).
+ # A file-read tool returning the script's own source is not evidence
+ # that it ran.
+ for tc in tool_calls:
+ if _is_file_read_action(str(tc["action"])):
+ continue
+ obs = str(tc.get("observation", "")).lower()
+ if expected_script.lower() in obs:
+ return {"passed": True, "score": 0.75, "reason": f"{expected_script} found in tool observation"}
+ return {"passed": False, "score": 0.0, "reason": "No execute/run_code call found"}
+
+ # Observation fallback for exec calls. A command that names the script
+ # without invoking it produced any mention in its own output, so that
+ # output is not independent evidence that the script ran.
+ for call in exec_calls:
+ if expected_script in _command_text(call):
+ continue
+ obs = str(call.get("observation", "")).lower()
+ if expected_script.lower() in obs:
+ return {"passed": True, "score": 0.75, "reason": f"{expected_script} found in execution observation"}
+
return {"passed": False, "score": 0.0, "reason": f"Execute called but not with {expected_script}"}
diff --git a/tests/tier3/test_script_invocation_evidence.py b/tests/tier3/test_script_invocation_evidence.py
new file mode 100644
index 00000000..bddb5d8b
--- /dev/null
+++ b/tests/tier3/test_script_invocation_evidence.py
@@ -0,0 +1,2661 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Script execution credit requires evidence of an invocation.
+
+``check_script_execution`` previously asked only whether the expected script
+name appeared anywhere in an execution command, so reading, printing or
+searching the script earned the same full credit as running it. These cases
+cover both the host checker and the bundled Harbor verifier template.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+import pytest
+
+from skillevaluator.tier3.eval_core import checks as shared_checks
+
+
+def _load_template():
+ template_path = Path(__file__).parents[2] / "src/skillevaluator/tier3/harbor/templates/eval.py"
+ spec = importlib.util.spec_from_file_location("harbor_eval_script_invocation", template_path)
+ assert spec and spec.loader
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+TEMPLATE = _load_template()
+IMPLEMENTATIONS = [
+ pytest.param(shared_checks.check_script_execution, id="host"),
+ pytest.param(TEMPLATE.check_script_execution, id="harbor-verifier"),
+]
+EXPECTED_SCRIPT = "run.py"
+SCRIPT_SOURCE = "#!/usr/bin/env python3\n# run.py writes report.txt\nprint('done')\n"
+
+
+def _bash(command: str, observation: str = "Exit code 0") -> list[dict[str, object]]:
+ return [{"action": "Bash", "action_input": {"command": command}, "observation": observation}]
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "cat /workspace/skills/demo/run.py",
+ "head -20 run.py",
+ "tail -5 ./run.py",
+ "sed -n '1,10p' run.py",
+ "less run.py",
+ "wc -l run.py",
+ "ls -la run.py",
+ "cp run.py /tmp/backup.py",
+ ],
+)
+def test_reading_a_script_is_not_executing_it(check, command) -> None:
+ result = check(_bash(command, SCRIPT_SOURCE), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+ assert result["passed"] is False
+ assert "Executed" not in result["reason"]
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "echo run.py",
+ "printf 'run.py\\n'",
+ "grep -n threshold run.py",
+ ],
+)
+def test_naming_a_script_is_not_executing_it(check, command) -> None:
+ result = check(_bash(command, "run.py"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+ assert result["passed"] is False
+
+
[email protected]("check", IMPLEMENTATIONS)
[email protected]("command", ["python run.py.bak", "python rerun.py", "python run.pyc"])
+def test_a_similar_filename_is_a_different_script(check, command) -> None:
+ result = check(_bash(command), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+ assert result["passed"] is False
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_reading_a_script_inside_a_nested_shell_is_not_executing_it(check) -> None:
+ result = check(_bash("bash -c 'cat run.py'", SCRIPT_SOURCE), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+ assert result["passed"] is False
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_script_source_in_the_observation_does_not_rescue_a_read(check) -> None:
+ """The output of ``cat run.py`` is the script's own text, not evidence."""
+ result = check(_bash("cat run.py", "print('run.py finished')"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+ assert result["passed"] is False
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_file_read_tool_observation_does_not_earn_execution_credit(check) -> None:
+ calls = [
+ {
+ "action": "Read",
+ "action_input": {"file_path": "/workspace/skills/demo/run.py"},
+ "observation": SCRIPT_SOURCE,
+ }
+ ]
+ result = check(calls, EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+ assert result["passed"] is False
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "python /workspace/skills/demo/run.py",
+ "python3 run.py",
+ "python3.12 run.py --verbose",
+ "python -u run.py",
+ "./run.py",
+ "/workspace/skills/demo/run.py --out report.txt",
+ "bash -c 'python run.py'",
+ "cd /workspace/skills/demo && python run.py",
+ "timeout 30 python run.py",
+ "uv run python run.py",
+ "env PYTHONPATH=/x python run.py",
+ "source run.py",
+ "cat notes.txt && python run.py",
+ "python run.py && cat report.txt",
+ "/venv/bin/python skills/demo/run.py && cat totals.csv",
+ ],
+)
+def test_genuine_invocations_keep_full_credit(check, command) -> None:
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+ assert result["passed"] is True
+ assert result["reason"] == f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_unrecognised_command_shape_is_weaker_evidence_not_full_credit(check) -> None:
+ result = check(_bash("xargs -I {} {} run.py < runners.txt"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+ assert result["reason"] != f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_no_expected_script_is_unchanged(check) -> None:
+ result = check(_bash("cat run.py"), None)
+ assert result["score"] == 1.0
+ assert result["passed"] is True
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_unrelated_execution_still_reports_the_expected_script(check) -> None:
+ result = check(_bash("ls -la"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+ assert EXPECTED_SCRIPT in result["reason"]
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_skill_tool_observation_fallback_is_preserved(check) -> None:
+ """A non-read tool reporting the script ran keeps the 0.75 fallback."""
+ calls = [
+ {
+ "action": "Skill",
+ "action_input": {"name": "demo"},
+ "observation": "Ran run.py and wrote report.txt",
+ }
+ ]
+ result = check(calls, EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+ assert result["passed"] is True
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_reading_then_running_is_credited(check) -> None:
+ calls = _bash("cat run.py", SCRIPT_SOURCE) + _bash("python run.py", "report written")
+ result = check(calls, EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+ assert result["reason"] == f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "python other.py run.py",
+ "python3 wrapper.py --input run.py",
+ "bash other.sh run.py",
+ ],
+)
+def test_a_script_named_after_the_interpreter_target_is_only_argv(check, command) -> None:
+ """Only the first non-option argument is the script the interpreter runs."""
+ result = check(_bash(command, "other only"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+ assert result["passed"] is False
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ ["python -W ignore run.py", "python -B -u run.py", "python -- run.py", "python run.py extra.py"],
+)
+def test_interpreter_options_do_not_hide_the_script(check, command) -> None:
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+ assert result["reason"] == f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "xargs -I{} python3 {} <<< run.py",
+ "python3 $(echo run.py)",
+ "find skills -name run.py -exec python3 {} \\;",
+ "python3 `echo run.py`",
+ ],
+)
+def test_run_time_substitution_is_undecidable_not_a_failure(check, command) -> None:
+ """A path the shell fills in at run time cannot be compared statically."""
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+ assert result["reason"] != f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "python -Wignore run.py",
+ "python -Xdev run.py",
+ "python -W ignore::DeprecationWarning run.py",
+ "python -X dev run.py",
+ "python -OO run.py",
+ "perl -I lib run.py",
+ "ruby -I lib run.py",
+ "node -r ./setup.js run.py",
+ ],
+)
+def test_interpreter_options_with_values_do_not_hide_the_script(check, command) -> None:
+ """An option value, attached or separate, is not the script argument."""
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+ assert result["reason"] == f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ ["rm run.py", "touch run.py", "mv run.py backup.py", "git status run.py", "tar cf out.tar run.py"],
+)
+def test_commands_that_are_not_invocations_score_zero(check, command) -> None:
+ """Only a recognised way of running a script earns credit, so no list of
+ non-executing commands has to be maintained."""
+ result = check(_bash(command), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+ assert result["passed"] is False
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_short_circuited_chain_is_reported_as_an_invocation(check) -> None:
+ """Known limitation: the walk is static and does not model exit status.
+
+ ``false && python run.py`` never reaches the interpreter at run time, but
+ the command as written is an invocation, so it is credited. Deciding this
+ would need the chain's runtime outcome, which the tool-call text does not
+ carry.
+ """
+ result = check(_bash("false && python run.py", "Exit code 1"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ 'echo "python run.py"',
+ "cat <<EOF\npython run.py\nEOF",
+ 'git commit -m "run python run.py first"',
+ 'grep -r "python run.py" notes.txt',
+ ],
+)
+def test_an_invocation_quoted_as_data_is_not_an_invocation(check, command) -> None:
+ """Command text that only describes running the script is not evidence.
+
+ A heredoc body in particular is the operand's data, not further commands,
+ even though its newlines look like command separators.
+ """
+ result = check(_bash(command, "written"), EXPECTED_SCRIPT)
+ assert result["score"] != 1.0
+ assert result["reason"] != f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
[email protected]("command", ["{ python run.py; }", "( python run.py )", "! python run.py"])
+def test_grouping_tokens_do_not_hide_the_invocation(check, command) -> None:
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+ assert result["reason"] == f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_a_path_piped_to_xargs_is_undecidable(check) -> None:
+ """xargs builds its arguments from standard input, which the text lacks."""
+ result = check(_bash("echo run.py | xargs python", "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "script"),
+ [
+ ("python --help run.py", "run.py"),
+ ("python --version run.py", "run.py"),
+ ("python -h run.py", "run.py"),
+ ("python -V run.py", "run.py"),
+ ("perl -c run.pl", "run.pl"),
+ ("perl -v run.pl", "run.pl"),
+ ("ruby -c run.rb", "run.rb"),
+ ("ruby --version run.rb", "run.rb"),
+ ("node -c run.js", "run.js"),
+ ("node --check run.js", "run.js"),
+ ("node -v run.js", "run.js"),
+ ("bash -n run.sh", "run.sh"),
+ ("sh -n run.sh", "run.sh"),
+ ],
+)
+def test_options_that_print_or_only_check_never_run_the_script(check, command, script) -> None:
+ """Help, version and syntax-check modes exit before running the script."""
+ result = check(_bash(command, "usage: python [option] ..."), script)
+ assert result["score"] == 0.0
+ assert result["passed"] is False
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "script"),
+ [("python -v run.py", "run.py"), ("ruby -v run.rb", "run.rb"), ("bash -x run.sh", "run.sh")],
+)
+def test_verbose_options_still_run_the_script(check, command, script) -> None:
+ """``ruby -v`` prints its version and runs the script; ``ruby --version`` does not."""
+ result = check(_bash(command, "report written"), script)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_unrecognised_long_option_is_undecidable(check) -> None:
+ """An option this table does not describe may carry the script away."""
+ result = check(_bash("python --unknown-option run.py", "written"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "script"),
+ [
+ ("python --require lib run.py", "run.py"),
+ ("python --loader loader run.py", "run.py"),
+ ("bash --require lib run.py", "run.py"),
+ ("python -Q old run.py", "run.py"),
+ ("perl -M strict run.pl", "run.pl"),
+ ("perl -Mstrict run.pl", "run.pl"),
+ ("bash -Mstrict run.sh", "run.sh"),
+ ("bash --check run.sh", "run.sh"),
+ ],
+)
+def test_options_outside_the_interpreter_grammar_are_undecidable(check, command, script) -> None:
+ """An option belonging to another interpreter, or one whose argument is
+ conditionally attached, cannot be resolved, so the command is not credited.
+ """
+ result = check(_bash(command, "written"), script)
+ assert result["score"] == 0.75
+ assert result["reason"] != f"Executed {script}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_a_quoted_heredoc_operator_is_not_a_heredoc(check) -> None:
+ result = check(_bash("echo '<<' && python run.py", "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+def test_a_here_string_consumes_only_its_operand(check) -> None:
+ """Commands after a here-string are still commands."""
+ result = check(_bash("cat <<< data; python run.py", "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
[email protected]("command", ["sh -c 'python run.py'", "bash -xc 'python run.py'"])
+def test_inline_code_payload_is_walked(check, command) -> None:
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
[email protected](("command", "script"), [("perl -i run.pl", "run.pl")])
+def test_measured_behaviour_beats_intuition(check, command, script) -> None:
+ """This runs the script, which is why the grammar is derived by execution:
+ perl's in-place flag takes no separate argument, so the file after it is
+ still the script rather than the flag's value.
+ """
+ result = check(_bash(command, "report written"), script)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "env --help python run.py",
+ "timeout --help python run.py",
+ "sudo --help python run.py",
+ "stdbuf --help python run.py",
+ "setsid --help python run.py",
+ "xargs --help python run.py",
+ "uv --help run python run.py",
+ "nice --version python run.py",
+ ],
+)
+def test_a_wrapper_asked_for_help_never_runs_the_wrapped_command(check, command) -> None:
+ """A wrapper given --help prints and exits, so nothing after it runs."""
+ result = check(_bash(command, "Usage: ...\nExit code 0"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "env -i python run.py",
+ "timeout 30 python run.py",
+ "timeout -s KILL 30 python run.py",
+ "nice -n 5 python run.py",
+ "nice -5 python run.py",
+ "stdbuf -o0 python run.py",
+ "setsid -f python run.py",
+ "sudo -u build python run.py",
+ "timeout --help python run.py; python run.py",
+ ],
+)
+def test_a_wrapper_given_its_own_options_still_runs_the_command(check, command) -> None:
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "env --chunk-size=2 python run.py",
+ "timeout --kill-on-idle python run.py",
+ "stdbuf --unknown python run.py",
+ ],
+)
+def test_a_wrapper_option_outside_its_grammar_is_undecidable(check, command) -> None:
+ """An unlisted option may consume the command, so neither answer holds."""
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+ assert result["reason"] != f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "cat <<EOF\nnotes\nEOF\npython run.py",
+ "cat <<'EOF'\nnotes\nEOF\npython run.py",
+ 'cat <<"EOF"\nnotes\nEOF\npython run.py',
+ "cat <<-EOF\n\tnotes\n\tEOF\npython run.py",
+ "cat <<EOF > notes.txt\nnotes\nEOF\npython run.py",
+ "cat <<A\none\nA\ncat <<B\ntwo\nB\npython run.py",
+ ],
+)
+def test_commands_after_a_heredoc_terminator_are_still_commands(check, command) -> None:
+ """A heredoc body ends at its delimiter line, not at the end of the text."""
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ ["cat <<EOF\npython run.py\nEOF", "cat <<EOF\npython run.py\nEOF\ncat notes.txt"],
+)
+def test_an_invocation_inside_a_heredoc_body_is_still_data(check, command) -> None:
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+ assert result["reason"] != f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "echo $((1 << 2)); python run.py",
+ "echo $(( 1 << 2 )) && python run.py",
+ "shift=$((1 << 2)); python run.py",
+ ],
+)
+def test_an_arithmetic_shift_is_not_a_heredoc(check, command) -> None:
+ """``<<`` inside an arithmetic expansion shifts, so no body follows it."""
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "FOO=1; python run.py",
+ "FOO=1 && python run.py",
+ "A=1 B=2 && python run.py",
+ "(cd skills && python run.py); echo done",
+ "SCRIPT=run.py; python $SCRIPT",
+ ],
+)
+def test_a_separator_grouped_with_punctuation_still_ends_a_command(check, command) -> None:
+ """The tokenizer returns ``));`` whole, which must not swallow what follows."""
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
[email protected]("command", ["FOO=1", "A=1 B=2", "(cat run.py); echo done"])
+def test_a_command_that_only_assigns_or_reads_runs_nothing(check, command) -> None:
+ result = check(_bash(command, "Exit code 0"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "printf '' | xargs -r python run.py",
+ "printf '' | xargs -p python run.py",
+ "printf '' | xargs python run.py",
+ "echo x | xargs -r python run.py",
+ "xargs -a /dev/null python run.py",
+ "printf '' | parallel -r python run.py",
+ ],
+)
+def test_a_stdin_driven_tool_is_never_a_full_credit_invocation(check, command) -> None:
+ """Whether xargs runs anything depends on input the command text lacks.
+
+ ``printf '' | xargs -r python run.py`` runs nothing at all, and ``xargs -p``
+ runs nothing with no terminal to confirm at, so neither answer is supported
+ by the text and full credit would be a claim the command cannot back.
+ """
+ result = check(_bash(command, "Exit code 0"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+ assert result["reason"] != f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "env -- python run.py",
+ "env -i -- python run.py",
+ "env -- FOO=1 python run.py",
+ "timeout -- 30 python run.py",
+ "timeout -s TERM -- 30 python run.py",
+ "nice -- python run.py",
+ "nice -n 3 -- python run.py",
+ "stdbuf -o0 -- python run.py",
+ "setsid -- python run.py",
+ "uv run -- python run.py",
+ ],
+)
+def test_end_of_options_does_not_hide_the_invocation(check, command) -> None:
+ """``--`` ends a wrapper's options, not its positional arguments.
+
+ ``timeout -- 30 python run.py`` still reads 30 as the duration, so the
+ command after it is what runs.
+ """
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ 'cat <<< "a; b"; python run.py',
+ "cat <<< 'a | b'; python run.py",
+ 'cat <<< "a && b" && python run.py',
+ ],
+)
+def test_a_separator_inside_a_here_string_operand_does_not_end_it(check, command) -> None:
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ ["uv run --no-project -- python run.py", "uv run -q --no-project python run.py"],
+)
+def test_an_unmodelled_runner_option_is_unresolved_not_a_failure(check, command) -> None:
+ """These do run the script. The walk does not model uv's own options, so it
+ reports that it could not resolve the command rather than scoring it zero.
+ """
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+ assert result["reason"] != f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "python -c \"print('run.py')\"",
+ "python -c 'print(123)' run.py",
+ "python -c 'import runpy; runpy.run_path(\"run.py\")'",
+ "python -m run.py",
+ "python -m runpy run.py",
+ "eval 'python run.py'",
+ "python <run.py",
+ ],
+)
+def test_inline_code_naming_the_script_is_unresolved(check, command) -> None:
+ """Inline code, a module, eval and standard input are text this walk does
+ not read, and they can run the script or merely mention it. Two of these
+ run it and two only print or read it, and the command text does not say
+ which, so none of them is scored as either.
+ """
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+ assert result["reason"] != f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ ["python -c 'print(123)'", "python -m json.tool", "eval 'echo hello'", "python <other.py"],
+)
+def test_inline_code_that_never_names_the_script_is_still_zero(check, command) -> None:
+ result = check(_bash(command, "Exit code 0"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "flock /tmp/build.lock python run.py",
+ "taskset -c 0 python run.py",
+ "ionice -c3 python run.py",
+ "chrt -o 0 python run.py",
+ "strace -f -o trace.log python run.py",
+ "watch -n1 -t python run.py",
+ ],
+)
+def test_an_invocation_inside_an_unmodelled_command_is_unresolved(check, command) -> None:
+ """The set of commands that run another command is open, so it is not listed.
+
+ A command with no grammar of its own that carries an interpreter running
+ the script is reported as unresolved rather than as running nothing, which
+ is what keeps a wrapper nobody anticipated from becoming a wrong answer.
+ """
+ result = check(_bash(command, "report written"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+ assert result["reason"] != f"Executed {EXPECTED_SCRIPT}"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ ["flock /tmp/build.lock cat run.py", "strace -f -o trace.log cat run.py", "nsenter -t 1 -m cat run.py"],
+)
+def test_an_unmodelled_command_that_only_reads_the_script_is_still_zero(check, command) -> None:
+ """Carrying the script as an argument is not carrying an invocation of it."""
+ result = check(_bash(command, SCRIPT_SOURCE), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "xxd run.py",
+ "hexdump -C run.py",
+ "jq . run.py",
+ "base64 run.py",
+ "strings run.py",
+ "od -c run.py",
+ "shasum run.py",
+ "iconv -f utf8 run.py",
+ "column -t run.py",
+ "rev run.py",
+ ],
+)
+def test_reading_a_script_needs_no_list_of_reading_commands(check, command) -> None:
+ """None of these verbs is named anywhere in this module, and all score zero.
+
+ That is the point of recognising invocations rather than listing the
+ commands that are not one: a new way to read a file costs nothing, while a
+ list of reading verbs would have to grow forever to keep them out of full
+ credit.
+ """
+ result = check(_bash(command, SCRIPT_SOURCE), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+
+
+# The cases below came from review of the first revision, each a command the
+# shell runs differently from how the walk read it.
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "cd",
+ "cd && true",
+ "cd -",
+ "timeout --frobnicate 5 python other.py",
+ "python -Q other.py",
+ "env -Z python other.py",
+ "for f in other.py; do python $f; done",
+ "case $x in a) python other.py;; esac",
+ ],
+)
+def test_partial_credit_requires_the_script_to_be_named(check, command) -> None:
+ """A walk that cannot resolve a command which never names the script has
+ learned nothing about that script. A bare ``cd`` loses track of the
+ directory and an unknown option may have consumed anything, but neither is
+ evidence about run.py, so they score as they did before invocation
+ evidence was required: zero, not partial credit.
+ """
+ result = check(_bash(command, ""), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+ assert result["passed"] is False
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ ["cd && cat run.py && python $f", "timeout --frobnicate 5 python run.py", "cd && python -Q run.py"],
+)
+def test_naming_the_script_keeps_partial_credit_for_an_unresolved_command(check, command) -> None:
+ result = check(_bash(command, ""), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "bash run.sh -c 'echo done'",
+ "bash -- run.sh -c 'echo done'",
+ "sh run.sh --help",
+ "bash -x run.sh -c x",
+ "zsh run.sh -n",
+ ],
+)
+def test_options_after_the_script_operand_belong_to_the_script(check, command) -> None:
+ """``bash run.sh -c 'echo done'`` runs run.sh and hands it ``-c``; only the
+ options before the first operand are the shell's own.
+ """
+ result = check(_bash(command, "done"), "run.sh")
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("bash -c 'echo done' run.sh", 0.0),
+ ("bash -c 'bash run.sh' run.sh", 1.0),
+ ("bash -xc 'bash run.sh'", 1.0),
+ ],
+)
+def test_inline_code_still_decides_when_the_option_precedes_the_operand(check, command, expected) -> None:
+ """With ``-c`` before any operand the next word is code and run.sh is only
+ its ``$0``; the payload alone decides whether the script ran.
+ """
+ result = check(_bash(command, "done"), "run.sh")
+ assert result["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "python3 < /dev/null run.py",
+ "python3 </dev/null run.py",
+ "python3 0< /dev/null run.py",
+ "python3 <&0 run.py",
+ "python3 2>/dev/null run.py",
+ "python3 2>&1 run.py",
+ "python3 1>>log.txt run.py",
+ "python3 < /dev/null -u run.py",
+ "python3 -u < /dev/null run.py",
+ "python3 run.py < /dev/null",
+ ],
+)
+def test_redirections_before_the_script_are_not_its_operand(check, command) -> None:
+ """A redirection belongs to the shell, wherever it stands in the command."""
+ result = check(_bash(command, "done"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
[email protected]("command", ["python3 <run.py", "python3 < run.py", "wc -l < run.py"])
+def test_a_script_fed_through_standard_input_is_still_unresolved(check, command) -> None:
+ result = check(_bash(command, "done"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "if true; then ./run.py; fi",
+ "for i in one; do ./run.py; done",
+ "if false; then true; else python run.py; fi",
+ "if false; then true; elif true; then python run.py; fi",
+ "while read -r line; do python run.py; done < names.txt",
+ "until ./run.py; do sleep 1; done",
+ "if ./run.py; then echo ok; fi",
+ "then FOO=1 python run.py",
+ "if [ -f run.py ]; then python run.py; fi",
+ "for i in 1 2; do timeout 5 python run.py; done",
+ "for f in run.py; do python $f; done",
+ 'for f in run.py; do python "$f"; done',
+ "for f in ./run.py; do $f; done",
+ "for f in run.py; do echo $f; python $f; done",
+ ],
+)
+def test_invocations_inside_control_structures_are_credited(check, command) -> None:
+ """``then`` and ``do`` stand before a command rather than being one, and a
+ loop header with a single value binds its variable for the body.
+ """
+ result = check(_bash(command, "done"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "if [ -f run.py ]; then cat run.py; fi",
+ "for i in one; do cat run.py; done",
+ "while true; do wc -l run.py; break; done",
+ "if grep -q main run.py; then echo yes; fi",
+ "for f in run.py; do cat $f; done",
+ 'for f in run.py; do wc -l "$f"; done',
+ ],
+)
+def test_reading_the_script_inside_a_control_structure_is_still_zero(check, command) -> None:
+ result = check(_bash(command, SCRIPT_SOURCE), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "for f in run.py other.py; do python $f; done",
+ "for f in run.py other.py; do cat $f; done",
+ "for f in $(ls run.py); do python $f; done",
+ "case $x in run.py) python run.py;; esac",
+ "case x in x) ./run.py;; esac",
+ ],
+)
+def test_unmodelled_control_syntax_naming_the_script_is_unresolved(check, command) -> None:
+ """A header with several values settles nothing about its variable, and
+ ``case`` bodies are not modelled at all, so a script named inside either
+ is neither credited nor settled as unrun.
+ """
+ result = check(_bash(command, "done"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.75
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "script"),
+ [
+ ("/usr/bin/perl5.34.0 run.pl", "run.pl"),
+ ("/usr/bin/perl5.38.2 run.pl", "run.pl"),
+ ("perl5.38 -w run.pl", "run.pl"),
+ ("python3.13 run.py", "run.py"),
+ ("ruby3.2 run.rb", "run.rb"),
+ ("node22 run.js", "run.js"),
+ ("bash5 run.sh", "run.sh"),
+ ],
+)
+def test_versioned_interpreter_names_resolve_their_script(check, command, script) -> None:
+ result = check(_bash(command, "done"), script)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "script", "expected"),
+ [
+ ("perl5.38.2 -c run.pl", "run.pl", 0.0),
+ ("perl5.38.2 --version run.pl", "run.pl", 0.0),
+ ("python3.13 -m json.tool run.py", "run.py", 0.75),
+ ("bash5.2 -c 'bash run.sh'", "run.sh", 1.0),
+ ],
+)
+def test_a_versioned_interpreter_keeps_its_grammar(check, command, script, expected) -> None:
+ """The version suffix changes the name, not the options."""
+ result = check(_bash(command, "done"), script)
+ assert result["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "python3 2 > numeric.out run.py",
+ "python3 2 >numeric.out run.py",
+ "python3 1 >> log.txt run.py",
+ "python3 0 < notes.txt run.py",
+ ],
+)
+def test_a_digit_standing_apart_from_its_operator_is_an_operand(check, command) -> None:
+ """``2 > numeric.out`` is an argument ``2`` and a redirection of standard
+ output, not a redirection of standard error: the shell reads a descriptor
+ only when the digits are written flush against the operator. The
+ interpreter therefore runs a script named ``2`` and hands it ``run.py``.
+ """
+ result = check(_bash(command, "done"), EXPECTED_SCRIPT)
+ assert result["score"] == 0.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "python3 2>/dev/null run.py",
+ "python3 2> /dev/null run.py",
+ "python3 2>&1 run.py",
+ "python3 1>>log.txt run.py",
+ "python3 1> log.txt run.py",
+ "python3 0</dev/null run.py",
+ "python3 0< /dev/null run.py",
+ "python3 2>/dev/null -u run.py",
+ "echo '2>' ; python3 run.py",
+ 'echo "2 > x" ; python3 run.py',
+ ],
+)
+def test_an_attached_descriptor_before_the_script_is_a_redirection(check, command) -> None:
+ """The descriptor and its operator stay together through tokenizing, so a
+ redirection written before the script never stands where the script is
+ looked for, and a ``2>`` inside quotes is data.
+ """
+ result = check(_bash(command, "done"), EXPECTED_SCRIPT)
+ assert result["score"] == 1.0
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("bash -c 2>/dev/null './run.sh'", 1.0),
+ ("bash -c 2> /dev/null './run.sh'", 1.0),
+ ("bash 2>/dev/null -c './run.sh'", 1.0),
+ ("bash -c 2>/dev/null 'bash run.sh'", 1.0),
+ ("bash -c 2>/dev/null -- './run.sh'", 1.0),
+ ("bash -c 2>/dev/null 'cat ./run.sh'", 0.0),
+ ("bash -c 2 './run.sh'", 0.0),
+ ],
+)
+def test_the_shell_payload_is_read_from_the_same_stream_as_its_option(check, command, expected) -> None:
+ """A redirection standing between ``-c`` and its payload belongs to the
+ shell, so the payload is the next operand after it, not the redirection.
+ ``bash -c 2 './run.sh'`` runs the command ``2`` with ``./run.sh`` as its
+ ``$0``, and nothing runs the script.
+ """
+ result = check(_bash(command, "done"), "run.sh")
+ assert result["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('for f in run.py; do cat "$f"; done; for f in other.py another.py; do python3 "$f"; done', 0.0),
+ ("for f in run.py; do cat $f; done; for f; do python3 $f; done", 0.0),
+ ("for f in run.py; do cat $f; done; for f in; do python3 $f; done", 0.0),
+ ('for f in run.py; do cat "$f"; done; for f in run.py other.py; do python3 "$f"; done', 0.75),
+ ("for f in other.py; do cat $f; done; for f in run.py; do python3 $f; done", 1.0),
+ ("for f in run.py; do cat $f; done; for g in other.py another.py; do python3 $f; done", 1.0),
+ ],
+)
+def test_a_loop_variable_rebound_without_a_unique_value_is_forgotten(check, command, expected) -> None:
+ """A single-value header binds its variable for its own body. A later
+ header that gives the same variable several values, or none, settles
+ nothing about it, so the earlier value is dropped rather than read into
+ the new body: the second loop above runs two other scripts, not ``run.py``.
+ A loop variable keeps its last value after its loop ends, so a later loop
+ over a different variable that runs ``python3 $f`` does run ``run.py``.
+ """
+ result = check(_bash(command, "done"), EXPECTED_SCRIPT)
+ assert result["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("printf '' | for f in run.py; do cat $f; done; for g in x; do python3 $f; done", 0.0),
+ ("for f in run.py; do cat $f; done | true; python3 $f", 0.0),
+ ("if true; then for f in run.py; do cat $f; done; fi | true; python3 $f", 0.0),
+ ("echo x | if true; then f=run.py; fi; python3 $f", 0.0),
+ ("FOO=run.py | true; python3 $FOO", 0.0),
+ ("printf '' | for f in run.py; do python3 $f; done", 1.0),
+ ("echo x | while read -r l; do for f in run.py; do python3 $f; done; done", 1.0),
+ ("for f in run.py; do cat $f; done; python3 $f", 1.0),
+ ("if true; then f=run.py; fi; python3 $f", 1.0),
+ ("f=run.py; echo x | python3 $f", 1.0),
+ ],
+)
+def test_a_binding_made_inside_a_pipeline_does_not_outlive_it(check, command, expected) -> None:
+ """Each command of a pipeline runs in a subshell, a compound command
+ included, so a loop variable or assignment made there is gone once the
+ pipeline ends; the body of a loop piped into still sees its own header.
+ Outside a pipeline a loop variable keeps its last value, and a value
+ bound before the pipeline is read by a command inside it.
+ """
+ result = check(_bash(command, "done"), EXPECTED_SCRIPT)
+ assert result["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("attached", "spaced", "expected"),
+ [
+ ("python3 0<run.py", "python3 0< run.py", 0.75),
+ ("python3 <run.py", "python3 < run.py", 0.75),
+ ("python3 0<other.py run.py", "python3 0< other.py run.py", 1.0),
+ ("python3 0</dev/null run.py", "python3 0< /dev/null run.py", 1.0),
+ ("python3 2>err.txt run.py", "python3 2> err.txt run.py", 1.0),
+ ("python3 2>>err.txt run.py", "python3 2>> err.txt run.py", 1.0),
+ ("python3 3<>notes.txt run.py", "python3 3<> notes.txt run.py", 1.0),
+ ("python3 2>&1 run.py", "python3 2>&1 run.py", 1.0),
+ ("python3 2>&- run.py", "python3 2>&- run.py", 1.0),
+ ("python3 2>1.log run.py", "python3 2> 1.log run.py", 1.0),
+ ("python3 1>2.txt run.py", "python3 1> 2.txt run.py", 1.0),
+ ("python3 run.py 2>err.txt", "python3 run.py 2> err.txt", 1.0),
+ ("cat 0<run.py", "cat 0< run.py", 0.75),
+ ],
+)
+def test_an_operand_written_flush_against_its_descriptor_tokenizes_as_the_spaced_form(
+ check, attached, spaced, expected
+) -> None:
+ """``0<run.py`` and ``0< run.py`` are the same redirection, so they must
+ score the same. The descriptor stays with its operator, and the operand
+ becomes its own word rather than concatenating onto the quoted operator.
+ A descriptor duplication (``2>&1``) or close (``2>&-``) has no operand;
+ ``2>1.log`` has one, and its whole name is kept. A script fed to standard
+ input is unresolved, whichever
+ command reads it, as ``python3 < run.py`` already was.
+ """
+ assert check(_bash(attached, "done"), EXPECTED_SCRIPT)["score"] == expected
+ assert check(_bash(spaced, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
+ "reads_skill_md",
+ [
+ pytest.param(shared_checks._cmd_reads_skill_md, id="host"),
+ pytest.param(TEMPLATE._cmd_reads_skill_md, id="harbor-verifier"),
+ ],
+)
+def test_a_skill_md_read_through_an_attached_input_redirection_is_a_read(reads_skill_md) -> None:
+ """``cat 0<SKILL.md`` reads the file just as ``cat 0< SKILL.md`` does, and
+ ``cat 2>SKILL.md`` overwrites it with standard error and reads nothing.
+ """
+ assert reads_skill_md("cat 0<SKILL.md") is True
+ assert reads_skill_md("cat 0< SKILL.md") is True
+ assert reads_skill_md("cat <SKILL.md") is True
+ assert reads_skill_md("cat 2>SKILL.md") is False
+ assert reads_skill_md("cat 2> SKILL.md") is False
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=run.py; for f in; do :; done; python3 "$f"', 1.0),
+ ('f=run.py; select f in; do :; done; python3 "$f"', 1.0),
+ ('for f in run.py; do :; done; for f in; do :; done; python3 "$f"', 1.0),
+ ("for f in; do :; done; python3 run.py", 1.0),
+ ('for f in ""; do python3 run.py; done', 1.0),
+ ("for f in; do python3 run.py; done", 0.0),
+ ('f=run.py; for f in; do python3 "$f"; done', 0.0),
+ ('f=run.py; for f in; do :; done; cat "$f"', 0.0),
+ ("for f in run.py; do cat $f; done; for f in; do python3 $f; done", 0.0),
+ ('f=run.py; printf "" | for f in; do :; done; python3 "$f"', 1.0),
+ ],
+)
+def test_a_loop_over_an_empty_list_keeps_the_binding_and_runs_no_body(check, command, expected) -> None:
+ """``for f in; do ...; done`` runs zero iterations and never assigns ``f``,
+ so a value it held before still stands for the command after the loop.
+ Every shell tested (bash, dash, zsh, ksh, mksh) agrees, for ``select`` too.
+ The body never runs, so an invocation written inside it is no evidence,
+ while ``for f in ""`` iterates once over the empty string and its body
+ does run. ``for f; do`` (no ``in``) is different: it iterates the
+ positional parameters, which the text does not carry.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('zsh -c \'printf "" | for f in run.py; do cat "$f"; done; python3 "$f"\'', 1.0),
+ ('ksh -c \'printf "" | for f in run.py; do cat "$f"; done; python3 "$f"\'', 1.0),
+ ('bash -c \'printf "" | for f in run.py; do cat "$f"; done; python3 "$f"\'', 0.0),
+ ('sh -c \'printf "" | for f in run.py; do cat "$f"; done; python3 "$f"\'', 0.0),
+ ('dash -c \'printf "" | for f in run.py; do cat "$f"; done; python3 "$f"\'', 0.0),
+ ('mksh -c \'printf "" | for f in run.py; do cat "$f"; done; python3 "$f"\'', 0.0),
+ ('printf "" | for f in run.py; do cat "$f"; done; python3 "$f"', 0.0),
+ ('zsh -c \'for f in run.py; do cat "$f"; done | cat; python3 "$f"\'', 0.0),
+ ('zsh -c \'printf "" | f=run.py; python3 "$f"\'', 1.0),
+ ('bash -c \'printf "" | f=run.py; python3 "$f"\'', 0.0),
+ ("zsh -c 'echo x | if true; then f=run.py; fi; python3 \"$f\"'", 1.0),
+ ('zsh -c \'printf "" | { f=run.py; }; python3 "$f"\'', 1.0),
+ ('zsh -c \'printf "" | for f in run.py; do python3 "$f"; done\'', 1.0),
+ ('bash -c \'zsh -c "printf \\"\\" | for f in run.py; do :; done; python3 \\$f"\'', 1.0),
+ ('bash -c \'shopt -s lastpipe; printf "" | for f in run.py; do :; done; python3 "$f"\'', 0.75),
+ ('bash -O lastpipe -c \'printf "" | for f in run.py; do :; done; python3 "$f"\'', 0.75),
+ ],
+)
+def test_the_last_stage_of_a_pipeline_keeps_its_binding_where_the_shell_does(check, command, expected) -> None:
+ """zsh and ksh run the last command of a pipeline in the current shell, so
+ a loop variable or assignment made there outlives the pipeline; bash, sh,
+ dash and mksh fork it like every other stage, so it is gone. A command
+ piped into another stage is in a subshell in every shell. The ``-c``
+ payload carries its shell into the walk, nested shells included. The tool
+ call's own shell is read as bash. bash with ``lastpipe`` named keeps the
+ binding only if that option is in force when the pipeline runs, which the
+ text does not settle, so the walk runs under both readings and reports
+ their disagreement as unresolved rather than as "did not run".
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('(f=run.py); python3 "$f"', 0.0),
+ ("(for f in run.py; do :; done); python3 $f", 0.0),
+ ('(printf "" | for f in run.py; do cat "$f"; done; python3 "$f")', 0.0),
+ ('zsh -c \'printf "" | (f=run.py); python3 "$f"\'', 0.0),
+ ("(f=run.py) | cat; python3 $f", 0.0),
+ ("(cd x); (f=run.py); python3 $f", 0.0),
+ ('(f=run.py; python3 "$f")', 1.0),
+ ('{ f=run.py; }; python3 "$f"', 1.0),
+ ("f=run.py; (g=x); python3 $f", 1.0),
+ ("f=run.py; (python3 $f)", 1.0),
+ ('(printf "" | for f in run.py; do python3 "$f"; done)', 1.0),
+ ("((f=1)); f=run.py; python3 $f", 1.0),
+ ],
+)
+def test_a_binding_made_inside_a_subshell_group_never_escapes_it(check, command, expected) -> None:
+ """``( ... )`` runs in a subshell in every shell, so what is bound inside
+ the parentheses is gone after them, whether the binding was an assignment,
+ a loop variable, or made inside a pipeline within the group. A ``{ ... }``
+ group runs in the current shell and its bindings stay. Commands inside the
+ group still read what was bound before it, and still run the script.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("ksh run.sh", 1.0),
+ ("mksh run.sh", 1.0),
+ ("ash run.sh", 1.0),
+ ("ksh -c './run.sh'", 1.0),
+ ("mksh -c 'sh run.sh'", 1.0),
+ ("ksh -c 'cat run.sh'", 0.0),
+ ("ksh -n run.sh", 0.0),
+ ("ksh -x run.sh", 1.0),
+ ],
+)
+def test_ksh_mksh_and_ash_are_shells(check, command, expected) -> None:
+ """They run a script named as their operand and carry a ``-c`` payload
+ into the walk like sh, with sh's options: ``-n`` only parses, ``-x`` traces
+ while running.
+ """
+ assert check(_bash(command, "done"), "run.sh")["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("for f in; do :; 'done'; python3 run.py; done", 0.0),
+ ("for f in; do d'on'e; python3 run.py; done", 0.0),
+ ("for f in; do :; d\\one; python3 run.py; done", 0.0),
+ ("for f in; do \\done; python3 run.py; done", 0.0),
+ ("for f in; do 'for'; done; python3 run.py", 1.0),
+ ('f=run.py; for g in; do printf done; done; python3 "$f"', 1.0),
+ ('f=run.py; for g in; do for f in other.py; do :; done; done; python3 "$f"', 1.0),
+ ('f=run.py; printf "("; f=other.py; printf ")"; python3 "$f"', 0.0),
+ ('f=run.py; printf \\(; f=other.py; printf \\); python3 "$f"', 0.0),
+ ('f=run.py; printf "{"; f=other.py; printf "}"; python3 "$f"', 0.0),
+ ("python3 '|' run.py", 0.0),
+ ("python3 ';' run.py", 0.0),
+ ("python3 '>' run.py", 0.0),
+ ("python3 '0<run.py'", 0.0),
+ ("'python3' run.py", 1.0),
+ ('"python3" ./run.py', 1.0),
+ ("python3 run.py '|' cat", 1.0),
+ ("printf '(' ; python3 run.py", 1.0),
+ ],
+)
+def test_a_quoted_or_escaped_word_is_not_shell_syntax(check, command, expected) -> None:
+ """``'done'`` is a command named done, ``printf "("`` prints a parenthesis
+ and ``python3 '|' run.py`` runs a script named ``|``. The tokenizer drops
+ the quotes, so a word that would read as syntax once unquoted is marked
+ before tokenizing and the walk reads it as the ordinary word it is: the
+ empty loop's real ``done`` still ends the skipped body, a quoted
+ parenthesis opens no subshell, and a quoted separator splits no command.
+ Quoting a command name changes nothing: ``'python3' run.py`` runs it.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ 'zsh -c \'emulate sh; printf "" | for f in run.py; do :; done; python3 "$f"\'',
+ 'zsh -c \'setopt shwordsplit; printf "" | for f in run.py; do :; done; python3 "$f"\'',
+ 'bash -c \'shopt -s lastpipe; printf "" | for f in run.py; do :; done; python3 "$f"\'',
+ ],
+)
+def test_a_shell_option_change_leaves_the_pipeline_rule_unsettled(check, command) -> None:
+ """``emulate sh`` makes zsh fork the last stage (measured), and
+ ``shopt -s lastpipe`` makes bash keep it. Which options are in force when
+ the pipeline runs is not something the text settles, so a command that
+ names one is walked under both readings and their disagreement is
+ unresolved.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == 0.75
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=run.py; for f in; do :; done; for f; do :; done; python3 "$f"', 1.0),
+ ("f=run.py; for f; do :; done; python3 $f", 1.0),
+ ("for f; do python3 run.py; done", 0.0),
+ ("for f in run.py; do cat $f; done; for f; do python3 $f; done", 0.0),
+ ("set -- run.py; for f; do python3 $f; done", 0.75),
+ ("f=run.py; shift; for f; do :; done; python3 $f", 0.75),
+ ("bash -c 'for f; do python3 $f; done' _ run.py", 0.75),
+ ("bash -c 'for f; do python3 $f; done' run.py", 0.0),
+ ("bash -c 'python3 $1' _ run.py", 0.75),
+ ("bash -c 'python3 \"$@\"' _ run.py", 0.75),
+ ("bash -c '\"$0\"' ./run.py", 0.75),
+ ("bash -c 'python3 other.py' _ run.py", 0.0),
+ ("bash -c 'echo done' run.py", 0.0),
+ ],
+)
+def test_the_positional_parameters_are_empty_unless_the_text_gives_some(check, command, expected) -> None:
+ """A tool call runs with no positional parameters, so ``for f; do`` at the
+ top level runs zero times and keeps the variable's earlier value, like
+ ``for f in;``. Where the text can give it some (``set --``, ``shift``, or
+ operands after a ``-c`` payload beyond its ``$0``) a later read of the
+ variable is unresolved rather than a settled miss, and an operand that
+ names the script reaches the payload only through ``$1``, ``$@``, ``$0``
+ or ``for f; do`` written in it.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("cat run.py | python3", 0.75),
+ ("cat 'run.py' | python3", 0.75),
+ ("cat run.py | python3 -", 0.75),
+ ("cat run.py | python3 -u", 0.75),
+ ("python3 - < run.py", 0.75),
+ ("cat run.py | python3 -c 'print(1)'", 0.0),
+ ("cat run.py | python3 --version", 0.0),
+ ("cat run.py | python3 other.py", 0.0),
+ ("cat run.py | wc -l", 0.0),
+ ("cat run.py | grep x | python3", 0.75),
+ ("cat other.py | python3; cat run.py", 0.0),
+ ],
+)
+def test_an_interpreter_reading_its_program_from_a_pipe_is_unresolved(check, command, expected) -> None:
+ """``cat run.py | python3`` runs the script and ``cat run.py | wc -l`` does
+ not; an interpreter with no script, no inline code and no terminal option
+ reads its program from standard input, the same shape as
+ ``python3 < run.py``, so when an earlier stage of its pipeline names the
+ script the command is unresolved. ``-`` names standard input.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("printf '%s' ';|' python3 run.py", 0.75),
+ ("printf '%s' \\;\\| python3 run.py", 0.75),
+ ("python3 ';;' run.py", 0.0),
+ ("python3 '|&' run.py", 0.0),
+ ("python3 '&&' run.py", 0.0),
+ ("python3 '2>' run.py", 0.0),
+ ("python3 '<<-' run.py", 0.0),
+ ("python3 run.py ';|'", 1.0),
+ ("python3 run.py '&&' cat", 1.0),
+ ("python3 '\\ue000run.py'", 0.0),
+ ("python3 \\ue000run.py", 0.0),
+ ("python3 '\\ue000\\ue000run.py'", 0.0),
+ ("python3 run.py '\\ue000'", 1.0),
+ ],
+)
+def test_a_quoted_run_of_metacharacters_is_one_word_and_a_mark_character_is_data(check, command, expected) -> None:
+ """A quoted word made of metacharacters (``';|'``, ``'2>'``) would be split
+ into operators after tokenizing, so it is marked like a quoted reserved
+ word; ``printf '%s' ';|' python3 run.py`` is one printf, which names the
+ interpreter and the script and is unresolved, not a separate python3
+ command. A file whose name carries the mark character itself is a
+ different file: the character is doubled before marking, so ``\\ue000run.py``
+ never reads as ``run.py``.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=other.py; { f=run.py; :; } | cat; python3 "$f"', 0.0),
+ ('f=other.py; { :; f=run.py; } | cat | cat; python3 "$f"', 0.0),
+ ('f=other.py; printf "" | { :; f=run.py; } | cat; python3 "$f"', 0.0),
+ ('f=other.py; printf "" | { :; f=run.py; }; python3 "$f"', 0.0),
+ ('zsh -c \'f=other.py; printf "" | { f=run.py; :; } | cat; python3 "$f"\'', 0.0),
+ ('zsh -c \'f=other.py; printf "" | { :; f=run.py; }; python3 "$f"\'', 1.0),
+ ('bash -c \'zsh -c "printf \\\\"\\\\" | { f=run.py; :; } | cat; python3 \\$f"\'', 0.0),
+ ('f=other.py; { f=run.py; :; }; python3 "$f"', 1.0),
+ ('f=other.py; printf "" | (f=run.py; python3 "$f"); python3 "$f"', 1.0),
+ ('f=other.py; printf "" | (f=run.py; python3 "$f") | cat; python3 "$f"', 1.0),
+ ('printf "" | (for f in run.py; do :; done; python3 $f)', 1.0),
+ ('printf "" | (f=run.py); python3 "$f"', 0.0),
+ ],
+)
+def test_a_brace_group_is_one_pipeline_stage_and_a_subshell_keeps_its_own_bindings(check, command, expected) -> None:
+ """``{ ...; } | cat`` runs the whole group as one stage of the pipeline, so
+ a binding made inside it is gone afterwards in bash, and stays only when
+ the group is the last stage under zsh. A ``( ... )`` group in a pipeline
+ is its own scope: a binding made inside it reaches the group's later
+ commands, and the pipe before ``(`` belongs to the group, not to the
+ first command inside it.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=run.py; (f=other.py; for g in; do :; done); python3 "$f"', 1.0),
+ ('f=run.py; (f=other.py; for g in; do :; done) | cat; python3 "$f"', 1.0),
+ ('f=run.py; (f=other.py; for g in; do :; done); (:); python3 "$f"', 1.0),
+ ('f=run.py; (f=other.py; for g in; do for h in; do :; done; done); python3 "$f"', 1.0),
+ ('f=other.py; (f=run.py; for g in; do :; done); python3 "$f"', 0.0),
+ ('f=other.py; (f=run.py; for g in; do for h in; do :; done; done) | cat; python3 "$f"', 0.0),
+ ("bash -c 'bash -c '\"'\"'f=run.py; (f=other.py; for g in; do :; done); python3 \"$f\"'\"'\"''", 1.0),
+ ("zsh -c 'zsh -c '\"'\"'f=run.py; (f=other.py; for g in; do :; done); python3 \"$f\"'\"'\"''", 1.0),
+ ('f=run.py; (for g in; do :; done; f=other.py); python3 "$f"', 1.0),
+ ('f=other.py; for g in; do (f=run.py); done; python3 "$f"', 0.0),
+ ],
+)
+def test_an_empty_loop_skipped_inside_a_subshell_still_closes_the_subshell(check, command, expected) -> None:
+ """Skipping the body of an empty loop must not skip the ``)`` that ends the
+ group it sits in: the group's bindings stay inside the group, and the
+ command after it reads the outer value. A parenthesis inside the skipped
+ body counts too.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('(f=run.py; f=other.py | cat; python3 "$f")', 1.0),
+ ('(f=other.py; f=run.py | cat; python3 "$f")', 0.0),
+ ('(f=run.py; printf "" | f=other.py; python3 "$f")', 1.0),
+ ('(f=other.py; printf "" | f=run.py; python3 "$f")', 0.0),
+ ('(f=run.py; { f=other.py; } | cat; python3 "$f")', 1.0),
+ ('(f=other.py; { f=run.py; } | cat; python3 "$f")', 0.0),
+ ('((f=other.py; f=run.py | cat); python3 "$f")', 0.0),
+ ('f=run.py; ((f=other.py) | cat; python3 "$f")', 1.0),
+ ('(f=run.py; (f=other.py | cat); python3 "$f")', 1.0),
+ ('{ f=run.py; f=other.py | cat; python3 "$f"; }', 1.0),
+ ('(f=other.py; for g in run.py; do :; done | cat; python3 "$g")', 0.0),
+ ('zsh -c \'(f=other.py; printf "" | f=run.py; python3 "$f")\'', 1.0),
+ ("zsh -c '(f=other.py; f=run.py | cat; python3 \"$f\")'", 0.0),
+ ('zsh -c \'(f=run.py; printf "" | f=other.py; python3 "$f")\'', 0.0),
+ ('printf "" | (f=run.py; python3 "$f")', 1.0),
+ ('printf "" | (f=run.py; python3 "$f") | cat', 1.0),
+ ],
+)
+def test_a_pipeline_inside_a_subshell_still_isolates_its_own_stages(check, command, expected) -> None:
+ """The group's scope is shared by the commands inside it, but a pipeline
+ written inside the group still runs each stage in its own subshell:
+ ``(f=run.py; f=other.py | cat; python3 "$f")`` runs run.py under bash,
+ and under zsh the last stage's binding survives inside the group as it
+ would outside. Only the pipe before ``(`` belongs to the group itself.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('bash -c \'printf "" | { f=run.py; (f=other.py); python3 "$f"; }\'', 1.0),
+ ('bash -c \'printf "" | { f=other.py; (f=run.py); python3 "$f"; }\'', 0.0),
+ ('zsh -c \'printf "" | { f=run.py; (f=other.py); python3 "$f"; }\'', 1.0),
+ ('zsh -c \'printf "" | { f=other.py; (f=run.py); python3 "$f"; }\'', 0.0),
+ ('printf "" | for g in 1; do f=run.py; (f=other.py); python3 "$f"; done', 1.0),
+ ('if true; then f=other.py; (f=run.py); python3 "$f"; fi | cat', 0.0),
+ ('f=run.py; { { f=other.py; } | cat; }; python3 "$f"', 1.0),
+ ('f=other.py; { if true; then f=run.py; fi | cat; }; python3 "$f"', 0.0),
+ ('f=run.py; printf "" | { printf "" | { f=other.py; }; }; python3 "$f"', 1.0),
+ (
+ 'zsh -c \'f=other.py; printf "x\\n" | while read -r l; do { printf "" | f=run.py; }; done; python3 "$f"\'',
+ 1.0,
+ ),
+ ('zsh -c \'f=other.py; { printf "" | f=run.py; }; python3 "$f"\'', 1.0),
+ ('f=other.py; { printf "" | f=run.py; }; python3 "$f"', 0.0),
+ ('f=run.py; { for g in; do :; done; f=other.py | cat; }; python3 "$f"', 1.0),
+ ],
+)
+def test_every_scope_nests_in_either_order_and_each_compound_has_its_own_pipe(check, command, expected) -> None:
+ """Groups and compound pipeline stages are one stack of scopes, innermost
+ last: a ``( ... )`` inside a compound stage copies the stage's bindings and
+ drops its own at ``)``, so the stage's value stands after it. Each compound
+ a segment opens is tested for its own pipe: in ``{ if ...; fi | cat; }``
+ only the ``if`` is a pipeline stage, and in ``{ printf "" | f=x; }`` the
+ pipe belongs to the command inside the braces, not to the braces.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=run.py; ((f=other.py)); python3 "$f"', 0.75),
+ ("ksh -c 'f=run.py; ((f=other.py)); python3 \"$f\"'", 0.75),
+ ("f=run.py; ((f++)); python3 $f", 0.75),
+ ("f=run.py; ((g=1)); python3 $f", 1.0),
+ ('f=other.py; ((f=run.py)); python3 "$f"', 0.0),
+ ("((f=1)); python3 $f", 0.0),
+ ("f=run.py; ((n > 0)); python3 $f", 1.0),
+ ('f=run.py; ((f=other.py; g=1); python3 "$f")', 1.0),
+ ('((f=other.py; f=run.py | cat); python3 "$f")', 0.0),
+ ("echo $((2 << 1)); python3 run.py", 1.0),
+ ("for ((i=0; i<1; i++)); do python3 run.py; done", 1.0),
+ ],
+)
+def test_an_arithmetic_command_is_not_a_pair_of_subshells(check, command, expected) -> None:
+ """``((f=x))`` at command position is arithmetic and runs in the current
+ shell: the value it gives a variable is a number, never a script path; on
+ an error bash, zsh and mksh leave the variable as it was, and ksh aborts
+ the rest of the text. So a variable that held the script is unsettled
+ afterwards, and any other stays not the script. ``((f=x; g=y); z)``, closed by a single ``)``, is two nested
+ subshells, as every shell reads it; ``$((`` and ``for ((`` are unchanged.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=other.py; ((f=run.py; python3 "$f")); python3 "$f"', 0.0),
+ ('zsh -c \'f=other.py; ((printf "" | for g in 1; do f=run.py; python3 "$f"; done)); python3 "$f"\'', 0.0),
+ ('zsh -c \'f=other.py; (({ f=run.py; }; python3 "$f")); python3 "$f"\'', 0.0),
+ ('mksh -c \'f=other.py; ((f=run.py; python3 "$f")); python3 "$f"\'', 0.0),
+ ('dash -c \'f=other.py; ((f=run.py; python3 "$f")); python3 "$f"\'', 1.0),
+ ('sh -c \'f=other.py; ((f=run.py; python3 "$f")); python3 "$f"\'', 0.75),
+ ('f=other.py; ((f=run.py; python3 "$f") ); python3 "$f"', 1.0),
+ ('f=other.py; ((f=run.py) ; python3 "$f"); python3 "$f"', 0.0),
+ ("f=run.py; ((x=(1+2)*3)); python3 $f", 1.0),
+ ("f=run.py; ((f=(1+2))); python3 $f", 0.75),
+ ("f=run.py; ((a*(b+c))) 2>/dev/null; python3 $f", 1.0),
+ ],
+)
+def test_double_parentheses_are_read_by_how_they_close_and_by_shell(check, command, expected) -> None:
+ """bash, zsh, ksh and mksh read ``((`` as arithmetic when its two halves
+ close together as ``))``, whatever is written between: ``((f=x; cmd))`` is
+ an arithmetic expression that fails and runs nothing. ``((cmd) )`` and
+ ``((f=x) ; cmd)`` close apart and are nested subshells. dash has no
+ arithmetic command and always reads subshells; ``sh`` is dash on some
+ systems and not others, so there the reading is unresolved. Parentheses
+ inside the arithmetic body are counted and stay in it.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('export f=run.py; python3 "$f"', 1.0),
+ ('readonly f=run.py; python3 "$f"', 1.0),
+ ("export f=run.py; python3 ${f}", 1.0),
+ ('declare f=run.py; python3 "$f"', 1.0),
+ ('typeset -x f=run.py; python3 "$f"', 1.0),
+ ('export -- f=run.py g=x; python3 "$f"', 1.0),
+ ('f=run.py; export f; python3 "$f"', 1.0),
+ ('export f=run.py; cat "$f"', 0.0),
+ ('readonly f=run.py; head -5 "$f"', 0.0),
+ ("dash -c 'declare f=run.py; python3 \"$f\"'", 0.0),
+ ("zsh -c 'declare f=run.py; python3 \"$f\"'", 1.0),
+ ("ksh -c 'typeset f=run.py; python3 \"$f\"'", 1.0),
+ ("sh -c 'declare f=run.py; python3 \"$f\"'", 0.75),
+ ('local f=run.py; python3 "$f"', 0.75),
+ ('declare -u f=run.py; python3 "$f"', 0.75),
+ ('f=other.py; echo run.py | { read f; python3 "$f"; }', 0.75),
+ ('printf -v f run.py; python3 "$f"', 0.75),
+ ('readonly f=run.py; f=other.py; python3 "$f"', 0.75),
+ ('f=run.py; unset f; python3 "$f"', 0.0),
+ ('f=run.py; unset -f f; python3 "$f"', 1.0),
+ ],
+)
+def test_builtins_that_bind_a_variable_are_read_as_bindings(check, command, expected) -> None:
+ """``export f=run.py`` and ``readonly f=run.py`` bind as ``f=run.py`` does
+ in every shell; ``declare`` binds in bash and zsh and ``typeset`` also in
+ ksh and mksh, and where a shell lacks one the command fails and binds
+ nothing, so ``sh`` (dash on some systems, bash on others) is unresolved.
+ ``unset f`` leaves ``$f`` empty. A value the text cannot settle leaves the
+ variable unsettled rather than a settled miss: ``local`` outside a function
+ (it binds in zsh and mksh only), an option that changes the value
+ (``declare -u``), ``read`` and ``printf -v``, and an assignment to a
+ read-only name. Reading the script stays uncredited.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=run.py python3 "$f"', 0.0),
+ ('f=run.py true; python3 "$f"', 0.0),
+ ('env f=run.py python3 "$f"', 0.0),
+ ('f=run.py :; python3 "$f"', 0.0),
+ ("bash -c 'f=run.py :; python3 \"$f\"'", 0.0),
+ ("dash -c 'f=run.py :; python3 \"$f\"'", 1.0),
+ ('f=run.py FOO=1; python3 "$f"', 1.0),
+ ("export f=run.py; bash -c 'python3 \"$f\"'", 1.0),
+ ("f=run.py; bash -c 'python3 \"$f\"'", 0.0),
+ ("f=run.py bash -c 'python3 \"$f\"'", 1.0),
+ ("env f=run.py bash -c 'python3 \"$f\"'", 1.0),
+ ("f=run.py; env f=run.py bash -c 'python3 \"$f\"'", 1.0),
+ ("env f=run.py bash -c 'cat \"$f\"'", 0.0),
+ ('f=run.py; bash -c "python3 $f"', 1.0),
+ ("set -a; f=run.py; bash -c 'python3 \"$f\"'", 1.0),
+ ("set -o allexport; f=run.py; bash -c 'python3 \"$f\"'", 1.0),
+ ("set -a; f=run.py; set +a; bash -c 'python3 \"$f\"'", 1.0),
+ ("f=run.py; set -a; bash -c 'python3 \"$f\"'", 0.0),
+ ("set -a; set +a; f=run.py; bash -c 'python3 \"$f\"'", 0.0),
+ ('f=run.py; bash -c "python3 \\$f"', 0.0),
+ ],
+)
+def test_an_assignment_before_a_command_is_that_commands_environment(check, command, expected) -> None:
+ """``f=run.py python3 "$f"`` expands ``$f`` before the assignment applies,
+ and the assignment is the command's environment only, so nothing after it
+ sees ``f`` either; ``env f=run.py`` is the same. Before a special builtin
+ (``:``) the assignment outlives the command in dash and the other POSIX
+ shells, not in bash or zsh. A ``-c`` payload's shell sees what it inherits:
+ the exported names (``export``, ``declare -x``, and anything bound while
+ ``set -a`` is on) and the prefix of the command that started it, and a
+ ``$f`` quoted or escaped from this shell is expanded there, not here.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("python3 <<A <<B run.py\nx\nA\ny\nB", 1.0),
+ ("python3 <<-A <<B run.py\n\tx\n\tA\ny\nB", 1.0),
+ ("python3 <<A <<B <<C run.py\nx\nA\ny\nB\nz\nC", 1.0),
+ ("cat <<A; python3 <<B run.py\nx\nA\ny\nB", 1.0),
+ ("cat <<A <<A\nx\nA\ny\nA\npython3 run.py", 1.0),
+ ("cat <<A <<B run.py\nx\nA\ny\nB", 0.0),
+ ("python3 <<A <<B other.py\nx\nA\ny\nB", 0.0),
+ ("cat <<A <<B\nx\nA\npython3 run.py\nB", 0.75),
+ ],
+)
+def test_every_heredoc_declared_on_a_line_takes_its_body_in_order(check, command, expected) -> None:
+ """Every heredoc declared on one line, across commands joined on that line
+ included, reads its body after the line ends, in the order declared, as
+ bash, dash, zsh, ksh, mksh and busybox ash all do up to the number each
+ accepts on a line (the limit is tested separately). The second declaration
+ is not left in the command's words, and a body that is data (a line inside
+ B's body) is not walked as a command.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=other.py; (((for f in run.py; do :; done; python3 "$f")) | cat); python3 "$f"', 0.0),
+ ('f=other.py; printf "" | (((for f in run.py; do :; done; python3 "$f")) | cat); python3 "$f"', 0.0),
+ ('zsh -c \'f=other.py; (((for f in run.py; do :; done; python3 "$f")) | cat); python3 "$f"\'', 0.0),
+ ('ksh -c \'f=other.py; (((for f in run.py; do :; done; python3 "$f")) | cat); python3 "$f"\'', 1.0),
+ ('dash -c \'f=other.py; (((for f in run.py; do :; done; python3 "$f")) | cat); python3 "$f"\'', 1.0),
+ ],
+)
+def test_three_opening_parentheses_are_a_subshell_around_double_parentheses(check, command, expected) -> None:
+ """The tokenizer hands ``(((`` over as one token. bash, zsh and mksh read it
+ as ``( ((``, so the inner ``((...))`` is an arithmetic command that runs
+ nothing; ksh and dash read nested subshells, where the loop's binding
+ reaches ``python3 "$f"`` and the script runs.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("export f=run.py; env -i bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; env -u f bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; env --ignore-environment bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; env --unset=f bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; env -uf bash -c 'python3 \"$f\"'", 0.0),
+ ('export f=run.py; env -i PATH="$PATH" bash -c \'python3 "$f"\'', 0.0),
+ ("export f=run.py; env -i env bash -c 'python3 \"$f\"'", 0.0),
+ ("f=run.py env -i bash -c 'python3 \"$f\"'", 0.0),
+ ("set -a; f=run.py; env -u f sh -c 'python3 \"$f\"'", 0.0),
+ ('export f=run.py; bash -c \'env -u f bash -c "python3 \\"\\$f\\""\'', 0.0),
+ ("export f=run.py; env -u g bash -c 'python3 \"$f\"'", 1.0),
+ ("export f=other.py; env -i f=run.py bash -c 'python3 \"$f\"'", 1.0),
+ ("export f=run.py; env -u f f=run.py bash -c 'python3 \"$f\"'", 1.0),
+ ("export f=run.py; env -u f env f=run.py bash -c 'python3 \"$f\"'", 1.0),
+ ("export f=run.py; env -S '-u f' bash -c 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; env -i bash -c 'cat \"$f\"'", 0.0),
+ ],
+)
+def test_env_removes_and_adds_names_before_the_child_starts(check, command, expected) -> None:
+ """``env -i``, ``-u NAME`` and their long forms and clusters take names out of
+ what the child inherits, in order, before and after its assignments: so
+ ``env -i f=run.py`` gives the child ``f`` again, ``env -u g`` leaves ``f``
+ alone, and a nested shell inherits only what reached its parent. ``env -S``
+ splits a string the walk does not read, so the child's view is unresolved.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=other.py; export g=$f; f=run.py; python3 "$g"', 0.0),
+ ('f=run.py; export g=$f; f=other.py; python3 "$g"', 1.0),
+ ('f=other.py; readonly g=$f; f=run.py; python3 "$g"', 0.0),
+ ('f=other.py; declare g=$f; f=run.py; python3 "$g"', 0.0),
+ ('f=other.py; typeset g=$f; f=run.py; python3 "$g"', 0.0),
+ ('f=other.py; g=$f; f=run.py; python3 "$g"', 0.0),
+ ('f=run.py; g=$f; f=other.py; python3 "$g"', 1.0),
+ ('f=run.py; g="${f}"; f=other.py; python3 "$g"', 1.0),
+ ('export f=other.py g=$f; python3 "$g"', 0.0),
+ ('f=run.py; export f=other.py g=$f; python3 "$g"', 1.0),
+ ("ksh -c 'f=run.py; typeset f=other.py g=$f; python3 \"$g\"'", 0.0),
+ ("ksh -c 'f=other.py; typeset f=run.py g=$f; python3 \"$g\"'", 1.0),
+ ("f=run.py; export g='$f'; python3 \"$g\"", 0.0),
+ ('f=run.py; export g=$f; unset f; python3 "$g"', 1.0),
+ ("f=other.py; export g=$f; f=run.py; bash -c 'python3 \"$g\"'", 0.0),
+ ("f=run.py; export g=$f; f=other.py; bash -c 'python3 \"$g\"'", 1.0),
+ ('for f in run.py; do g=$f; done; f=other.py; python3 "$g"', 1.0),
+ ],
+)
+def test_a_declaration_copies_the_value_when_it_runs(check, command, expected) -> None:
+ """``export g=$f`` copies what ``f`` holds when it runs, as ``g=$f`` does;
+ a later ``f=...`` does not reach ``g``. In one declaration of several names
+ bash expands every value before assigning any, and ksh assigns in order
+ (measured). A ``$`` quoted from this shell is kept literally.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("export f=run.py; declare +x f; bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; typeset +x f; bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; declare -x +x f; bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; export -n f; bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; command export -n f; bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; builtin export -n f; bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; command -p export -n f; bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; command command export -n f; bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; builtin declare +x f; bash -c 'python3 \"$f\"'", 0.0),
+ ("export f=run.py; eval export -n f; bash -c 'python3 \"$f\"'", 0.75),
+ ('dash -c \'export f=run.py; export -n f; bash -c "python3 \\"\\$f\\""\'', 0.75),
+ ('zsh -c \'export f=run.py; command export -n f; bash -c "python3 \\"\\$f\\""\'', 1.0),
+ ('export f=run.py; declare +x f; python3 "$f"', 1.0),
+ ("export f=run.py; export -f f; bash -c 'python3 \"$f\"'", 1.0),
+ ("export f=run.py; export -nf f; bash -c 'python3 \"$f\"'", 1.0),
+ ("export f=run.py; declare +x f; export f; bash -c 'python3 \"$f\"'", 1.0),
+ ("f=run.py; command export f; bash -c 'python3 \"$f\"'", 1.0),
+ ("f=run.py; builtin export f; bash -c 'python3 \"$f\"'", 1.0),
+ ],
+)
+def test_removing_the_export_attribute_is_honoured_however_the_builtin_is_reached(check, command, expected) -> None:
+ """``declare +x``, ``typeset +x`` and ``export -n`` take the export away,
+ and so do they after ``command``, ``command -p`` or ``builtin``, in any
+ number, where the shell lets those reach the builtin (not zsh's
+ ``command``). The value stays in this shell. ``export -f`` and ``-nf``
+ touch functions only. ``dash`` rejects ``export -n``, and what ``eval``
+ runs is not read exactly, so both leave the value unsettled.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
+_OTHER_THEN_LISTED = 'export f=run.py; export -p f=other.py; python3 "$f"'
+_LISTED_ALONE = 'export -p f=run.py; python3 "$f"'
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "shell", "expected"),
+ [
+ (_OTHER_THEN_LISTED, None, 0.0),
+ (_OTHER_THEN_LISTED, "/bin/bash", 0.0),
+ (_OTHER_THEN_LISTED, "/bin/ksh", 0.0),
+ (_OTHER_THEN_LISTED, "/bin/mksh", 0.0),
+ (_OTHER_THEN_LISTED, "/bin/zsh", 1.0),
+ (_OTHER_THEN_LISTED, "/bin/dash", 1.0),
+ (_OTHER_THEN_LISTED, "/bin/sh", 0.75),
+ (_LISTED_ALONE, None, 1.0),
+ (_LISTED_ALONE, "/bin/ksh", 1.0),
+ (_LISTED_ALONE, "/bin/mksh", 1.0),
+ (_LISTED_ALONE, "/bin/zsh", 0.0),
+ (_LISTED_ALONE, "/bin/dash", 0.0),
+ (_LISTED_ALONE, "/bin/sh", 0.75),
+ ('readonly -p f=run.py; python3 "$f"', None, 1.0),
+ ('readonly -p f=run.py; python3 "$f"', "/bin/zsh", 0.0),
+ ('f=other.py; readonly -p f=run.py; python3 "$f"', None, 1.0),
+ ('f=other.py; readonly -p f=run.py; python3 "$f"', "/bin/dash", 0.0),
+ ('export f=run.py; readonly -p f=other.py; python3 "$f"', None, 0.0),
+ ('export f=run.py; readonly -p f=other.py; python3 "$f"', "/bin/zsh", 1.0),
+ ("export -p f=run.py; bash -c 'python3 \"$f\"'", None, 1.0),
+ ("f=run.py; export -p f; bash -c 'python3 \"$f\"'", None, 1.0),
+ ("f=run.py; export -p f; bash -c 'python3 \"$f\"'", "/bin/zsh", 0.0),
+ ('f=run.py; readonly -p f; f=other.py; python3 "$f"', None, 0.75),
+ ('f=run.py; readonly -p f; f=other.py; python3 "$f"', "/bin/dash", 0.0),
+ ('f=run.py; export -p >/dev/null; python3 "$f"', None, 1.0),
+ ('f=run.py; export -p >/dev/null; python3 "$f"', "/bin/zsh", 1.0),
+ ('export -pn f=run.py; python3 "$f"', None, 1.0),
+ ("f=run.py; export f; export -pn f; bash -c 'python3 \"$f\"'", None, 0.0),
+ ("f=run.py; export f; export -pn f; bash -c 'python3 \"$f\"'", "/bin/zsh", 1.0),
+ ('command export -p f=run.py; python3 "$f"', None, 1.0),
+ ('command export -p f=run.py; python3 "$f"', "/bin/zsh", 0.0),
+ ("bash -c 'export f=run.py; export -p f=other.py; python3 \"$f\"'", None, 0.0),
+ ("zsh -c 'export -p f=run.py; python3 \"$f\"'", None, 0.0),
+ ('declare -p f=run.py; python3 "$f"', None, 0.0),
+ ('f=other.py; declare -px f=run.py; python3 "$f"', None, 0.0),
+ ('f=other.py; typeset -p f=run.py; python3 "$f"', "/bin/zsh", 0.0),
+ ('f=other.py; typeset -p f=run.py; python3 "$f"', "/bin/ksh", 0.75),
+ ('f=other.py; typeset -px f=run.py; python3 "$f"', "/bin/mksh", 0.75),
+ ('export f=run.py; export -p > f=other.py; python3 "$f"', None, 1.0),
+ ('export f=run.py; export -p >f=other.py; python3 "$f"', "/bin/ksh", 1.0),
+ ('export f=run.py; readonly -p 2> f=other.py; python3 "$f"', None, 1.0),
+ ('f=other.py; export -p > f=run.py; python3 "$f"', None, 0.0),
+ ("f=run.py; export -p > f; bash -c 'python3 \"$f\"'", None, 0.0),
+ ('export f=run.py; export > f=other.py; python3 "$f"', None, 1.0),
+ ("typeset -pF f=run.py; python3 run.py", "/bin/zsh", 1.0),
+ ],
+)
+def test_a_listing_option_given_names_acts_on_them_where_the_shell_does(check, command, shell, expected) -> None:
+ """``export -p`` and ``readonly -p`` given names act on them as they do
+ without ``-p`` in bash, bash --posix, ksh, mksh and busybox ash, which list
+ only when no name is given: ``export f=run.py; export -p f=other.py``
+ runs other.py there, and ``export -p f=run.py`` binds and exports run.py.
+ dash and zsh list the names and change nothing. ``declare -p`` and
+ ``typeset -p`` list in bash and zsh, whatever else is given with them;
+ ksh's ``typeset -p`` and mksh's ``typeset -px`` assign, which is read as
+ unsettled. A redirection and its operand are not names: ``export -p >
+ f=other.py`` writes the listing to a file and binds nothing. Measured
+ with a marker in each shell.
+ """
+ assert check(_native(command, shell), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "shell", "expected"),
+ [
+ ('f=other.py; typeset -f f=run.py; python3 "$f"', None, 0.0),
+ ('f=other.py; typeset -f f=run.py; python3 "$f"', "/bin/zsh", 0.0),
+ ('f=other.py; typeset -f f=run.py; python3 "$f"', "/bin/mksh", 0.0),
+ ('f=other.py; typeset -f f=run.py; python3 "$f"', "/bin/ksh", 0.75),
+ ('f=run.py; declare -F f; python3 "$f"', None, 1.0),
+ ('f=run.py; declare -F f; python3 "$f"', "/bin/zsh", 0.75),
+ ("typeset -F f=run.py; python3 run.py", None, 1.0),
+ ("typeset -F f=run.py; python3 run.py", "/bin/zsh", 0.75),
+ ("typeset -F f=run.py; python3 run.py", "/bin/ksh", 0.75),
+ ("typeset -F f=run.py; python3 run.py", "/bin/mksh", 1.0),
+ ("typeset -E f=run.py; python3 run.py", "/bin/zsh", 0.75),
+ ("typeset -F f=2; python3 run.py", "/bin/zsh", 1.0),
+ ("f=1.5; typeset -F f; python3 run.py", "/bin/ksh", 1.0),
+ ("typeset -F f; python3 run.py", "/bin/zsh", 1.0),
+ ("g=run.py; typeset -F f=g; python3 run.py", "/bin/zsh", 0.75),
+ ("f=run.py; (typeset -F f); python3 run.py", "/bin/zsh", 1.0),
+ ("f=run.py; local -F f; python3 run.py", "/bin/zsh", 0.75),
+ ("f=run.py; typeset -i f; python3 run.py", None, 1.0),
+ ("f=run.py; typeset -i f; python3 run.py", "/bin/zsh", 0.75),
+ ("f=run.py; typeset -i f; python3 run.py", "/bin/ksh", 0.75),
+ ("f=run.py; typeset -i f; python3 run.py", "/bin/mksh", 1.0),
+ ("f=2.5; typeset -i f; python3 run.py", "/bin/zsh", 1.0),
+ ("typeset -i f=run.py; python3 run.py", "/bin/ksh", 0.75),
+ ('f=run.py; typeset -L3 f; python3 "$f"', None, 1.0),
+ ('f=run.py; typeset -L3 f; python3 "$f"', "/bin/zsh", 0.75),
+ ('f=run.py; typeset -Z3 f; python3 "$f"', "/bin/ksh", 0.75),
+ ('f=run.py; typeset -R8 f; python3 "$f"', "/bin/mksh", 0.75),
+ ('f=run.py; typeset -L f; python3 "$f"', "/bin/zsh", 1.0),
+ ('f=run.py; typeset -L 3 f; python3 "$f"', "/bin/zsh", 0.75),
+ ('f=run.py; typeset -x -Z 3 f; python3 "$f"', "/bin/ksh", 0.75),
+ ("typeset -i f=0x10; python3 run.py", "/bin/ksh", 1.0),
+ ("typeset -i f=0x10; python3 run.py", "/bin/zsh", 1.0),
+ ("typeset -i f=16#ff; python3 run.py", "/bin/ksh", 1.0),
+ ("typeset -i f=2#101; python3 run.py", "/bin/zsh", 1.0),
+ ("typeset -i f=99#1; python3 run.py", "/bin/zsh", 0.75),
+ ("typeset -i f=2#9; python3 run.py", "/bin/zsh", 0.75),
+ ('f=run.py; typeset -Lx 3 f; python3 "$f"', "/bin/zsh", 0.75),
+ ("f=0x10; typeset -i f; python3 run.py", "/bin/zsh", 1.0),
+ ("typeset -F f=0x10; python3 run.py", "/bin/ksh", 1.0),
+ ("typeset -i f=g+1; python3 run.py", "/bin/zsh", 0.75),
+ ("f=run.py; typeset -i +i f; python3 run.py", "/bin/ksh", 0.75),
+ ('f=run.py; typeset -A m; python3 "$f"', "/bin/zsh", 1.0),
+ ],
+)
+def test_function_float_and_width_options_follow_each_shell(check, command, shell, expected) -> None:
+ """``-f`` and ``-F`` name functions and change no variable in bash, but not
+ everywhere: ksh's ``typeset -f f=run.py`` assigns, and ``-F`` and ``-E``
+ are float attributes in zsh and ksh. A numeric attribute (``-i`` too)
+ given to a name whose value is not a number, given or held, stops zsh and
+ ksh; a decimal, ``0x10``, ``16#ff`` or nothing is a number, and an
+ expression is not evaluated. A width given with ``-L``, ``-R`` or ``-Z``,
+ in the word or as the next one, cuts or pads the value at once in zsh, ksh
+ and mksh, so the name is unsettled. Measured with a marker in each shell.
+ """
+ assert check(_native(command, shell), EXPECTED_SCRIPT)["score"] == expected
+
+
+_ZEROS = "0" * 4400
+
+
[email protected]("module", [shared_checks, TEMPLATE], ids=["host", "harbor-verifier"])
+ ("command", "shell", "expected"),
+ [
+ ("typeset -F f=" + "9" * 4400 + "#1; python3 run.py", "/bin/zsh", 0.75),
+ ("typeset -i f=" + "9" * 4301 + "#1; python3 run.py", "/bin/ksh", 0.75),
+ ("typeset -i f=" + _ZEROS + "16#ff; python3 run.py", "/bin/zsh", 1.0),
+ ("typeset -F f=" + _ZEROS + "36#z; python3 run.py", "/bin/ksh", 1.0),
+ ("typeset -i f=" + _ZEROS + "2#9; python3 run.py", "/bin/zsh", 0.75),
+ ("typeset -i f=" + _ZEROS + "37#1; python3 run.py", "/bin/zsh", 0.75),
+ ("typeset -i f=016#ff; python3 run.py", "/bin/zsh", 1.0),
+ ("typeset -i f=36#z; python3 run.py", "/bin/zsh", 1.0),
+ ("typeset -i f=37#1; python3 run.py", "/bin/zsh", 0.75),
+ ("typeset -i f=1#1; python3 run.py", "/bin/zsh", 0.75),
+ ],
+)
+def test_a_numeric_base_of_any_length_is_scored(module, command, shell, expected) -> None:
+ """zsh and ksh read a ``base#digits`` base as a decimal and drop its leading
+ zeros, so ``016#ff`` and a base written after thousands of zeros are 16.
+ A base past 36 is not a number the walk accepts, however many digits it
+ has, and the command still gets a score from both public scorers: int()
+ refuses a string of more than 4,300 digits, so the base is not converted
+ until it is known to be short. Measured with a marker in zsh and ksh.
+ """
+ calls = _native(command, shell)
+ assert module.check_script_execution(calls, EXPECTED_SCRIPT)["score"] == expected
+ aggregate = module.score_skill_execution(calls, "demo", EXPECTED_SCRIPT)
+ assert aggregate["details"]["script_execution"]["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "shell", "expected"),
+ [
+ ("export g='>' f=run.py; python3 \"$f\"", None, 1.0),
+ ("export g='>' f=run.py; python3 \"$f\"", "/bin/zsh", 1.0),
+ ("export g='>' f=run.py; python3 \"$f\"", "/bin/dash", 1.0),
+ ("export g='>' f=run.py; python3 \"$f\"", "/bin/ksh", 1.0),
+ ("f=run.py; export g='>' f=other.py; python3 \"$f\"", None, 0.0),
+ ("f=run.py; export g='>' f=other.py; python3 \"$f\"", "/bin/zsh", 0.0),
+ ("f=run.py; export g='>' f=other.py; python3 \"$f\"", "/bin/dash", 0.0),
+ ("f=run.py; export g='>' f=other.py; python3 \"$f\"", "/bin/ksh", 0.0),
+ ("export g='>>' f=run.py; python3 \"$f\"", None, 1.0),
+ ("export g='>' f=run.py >output; python3 \"$f\"", None, 1.0),
+ ('export g=\\> f=run.py; python3 "$f"', None, 1.0),
+ ("export g='2>' f=run.py; python3 \"$f\"", None, 1.0),
+ ("readonly g='>' f=run.py; python3 \"$f\"", "/bin/dash", 1.0),
+ ("typeset g='>' f=run.py; python3 \"$f\"", "/bin/ksh", 1.0),
+ ("f=run.py; typeset g='>' f=other.py; python3 \"$f\"", "/bin/zsh", 0.0),
+ ("declare g='>' f=run.py; python3 \"$f\"", None, 1.0),
+ ('export f=run.py > out; python3 "$f"', None, 1.0),
+ ('export >out f=run.py; python3 "$f"', "/bin/dash", 1.0),
+ ('export 2>err f=run.py; python3 "$f"', None, 1.0),
+ ('export 2> err f=run.py; python3 "$f"', "/bin/ksh", 1.0),
+ ('export f=run.py 2>&1; python3 "$f"', "/bin/zsh", 1.0),
+ ('export f=run.py 0<run.py; python3 "$f"', None, 1.0),
+ ('export g=2> f=run.py; python3 "$f"', None, 0.0),
+ ('export 2 > f=run.py; python3 "$f"', None, 0.0),
+ ('f=run.py; export -p > f=other.py; python3 "$f"', None, 1.0),
+ ('f=other.py; export >>| f=run.py; python3 "$f"', "/bin/zsh", 0.0),
+ ('f=other.py; export &>| f=run.py; python3 "$f"', "/bin/zsh", 0.0),
+ ],
+)
+def test_an_assignment_ending_in_an_operator_is_not_a_redirection(check, command, shell, expected) -> None:
+ """An unquoted redirection operator is a word of its own, so only such a
+ word, with a descriptor written flush against it (``2>``), takes the next
+ word as its operand. ``export g='>' f=run.py`` binds both names in every
+ shell: ``g=>`` is an assignment that ends in ``>``, and f is bound. A real
+ redirection, separate or attached, still takes its operand, zsh's ``>>|``
+ and ``&>|`` included, and ``export g=2> f=run.py`` writes to a file named
+ f=run.py. Measured with a marker in each shell.
+ """
+ assert check(_native(command, shell), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "shell", "expected"),
+ [
+ ('f=run.py; unset > f; python3 "$f"', "/bin/ksh", 0.75),
+ ('f=run.py; unset; python3 "$f"', "/bin/ksh", 0.75),
+ ('f=run.py; unset -v; python3 "$f"', "/bin/ksh", 0.75),
+ ('f=run.py; command unset; python3 "$f"', "/bin/ksh", 1.0),
+ ('f=run.py; (unset); python3 "$f"', "/bin/ksh", 1.0),
+ ('f=run.py; unset > f; python3 "$f"', None, 1.0),
+ ('f=run.py; unset > f; python3 "$f"', "/bin/mksh", 1.0),
+ ('f=run.py; unset; python3 "$f"', "/bin/dash", 1.0),
+ ],
+)
+def test_ksh_stops_on_unset_given_no_name(check, command, shell, expected) -> None:
+ """``unset`` given no name prints its usage and stops ksh, unless ``command``
+ runs it or a subshell holds it; the other shells go on. The target of a
+ redirection is not a name, so ``unset > f`` gives none. Measured with a
+ marker in each shell.
+ """
+ assert check(_native(command, shell), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("module", [shared_checks, TEMPLATE], ids=["host", "harbor-verifier"])
+ ("text", "second_unit", "comment"),
+ [
+ ("echo $'it\\'s # kept'\npython3 run.py", None, None),
+ ("echo $'a\n# still quoted'\ntrue", "true", None),
+ ("echo 'a\\'\npython3 run.py # gone", "python3 run.py # gone", "# gone"),
+ ],
+)
+def test_an_ansi_c_quote_honours_its_escapes_when_finding_where_commands_end(
+ module, text, second_unit, comment
+) -> None:
+ """``$'...'`` closes on ``'`` as ``'...'`` does, but a backslash escapes the
+ character after it: in ``$'it\\'s # kept'`` the ``#`` is quoted and starts
+ no comment, and in ``$'a\n# still quoted'`` the newline ends nothing. In
+ ``'a\\'`` the backslash is literal, so the quote closes and a later
+ ``#`` starts a comment. Where the text does not settle a newline (the
+ escaped quote in the first), the lines are read as one unit.
+ """
+ starts, blanked = module._parse_unit_starts(text, "bash", False)
+ assert starts == ([0] if second_unit is None else [0, text.index(second_unit)])
+ if comment is None:
+ assert blanked == text
+ else:
+ assert comment not in blanked and blanked.replace(" ", "") == text.replace(comment, "").replace(" ", "")
+
+
+def _heredocs(count: int, before: str, after: str = "", body: str = "x", quote: str = "") -> str:
+ """``before``, then ``count`` heredocs declared on one line, ``after``, and each body in order."""
+ names = [f"D{index}" for index in range(count)]
+ declared = " ".join(f"<<{quote}{name}{quote}" for name in names)
+ return f"{before} {declared}{' ' + after if after else ''}\n" + "".join(f"{body}\n{name}\n" for name in names)
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ (_heredocs(9, "cat", body="python3 run.py"), 0.75),
+ (_heredocs(9, "cat", body="python3 run.py", quote="'"), 0.75),
+ (_heredocs(12, "cat", body="python3 run.py"), 0.75),
+ (_heredocs(16, "cat", body="python3 run.py"), 0.75),
+ (_heredocs(17, "cat", body="python3 run.py"), 0.75),
+ (_heredocs(16, "python3", "run.py"), 1.0),
+ (_heredocs(17, "python3", "run.py"), 0.75),
+ (_heredocs(9, "cat", body="python3 run.py") + "python3 run.py", 1.0),
+ (_heredocs(16, "cat", body="python3 run.py") + "python3 run.py", 1.0),
+ (_heredocs(17, "cat", body="python3 run.py") + "python3 run.py", 0.75),
+ ("cat <<A\nx\nA\n" + _heredocs(17, "python3", "run.py"), 0.75),
+ ("echo a <<< x\n" + _heredocs(16, "python3", "run.py"), 1.0),
+ ("mksh -c '" + _heredocs(10, "python3", "run.py") + "'", 1.0),
+ ("mksh -c '" + _heredocs(11, "python3", "run.py") + "'", 0.75),
+ ("dash -c '" + _heredocs(17, "python3", "run.py") + "'", 1.0),
+ ],
+)
+def test_a_line_with_more_heredocs_than_the_shell_accepts_is_data(check, command, expected) -> None:
+ """Past the heredocs a shell accepts on one line, the line and everything
+ after it are data: bash exits with "maximum here-document count exceeded"
+ at 17 and mksh stops at 11 before running anything on the line (measured),
+ while dash reads more. A script named there is unresolved, never credited;
+ up to the limit every body is read in order.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("f=run.py eval 'python3 \"$f\"'", 0.75),
+ ("f=run.py; eval 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; eval 'python3 \"$f\"'", 0.75),
+ ("f=run.py; command eval 'python3 \"$f\"'", 0.75),
+ ("f=other.py eval 'python3 \"$f\"'", 0.0),
+ ('f=run.py; eval f=other.py; python3 "$f"', 0.75),
+ ("f=run.py; eval 'unset f'; python3 \"$f\"", 0.75),
+ ("f=run.py; eval 'read f <<< x'; python3 \"$f\"", 0.75),
+ ('f=run.py; eval \'echo "$f"\'; python3 "$f"', 1.0),
+ ("f=run.py python3 -c 'import os; os.system(\"python3 $f\")'", 0.75),
+ ("env f=run.py python3 -c 'import os; os.system(\"python3 $f\")'", 0.75),
+ ('f=run.py; bash <<EOF\npython3 "$f"\nEOF', 0.75),
+ ("export f=run.py; bash <<'EOF'\npython3 \"$f\"\nEOF", 0.75),
+ ],
+)
+def test_eval_and_inline_code_may_expand_a_quoted_variable(check, command, expected) -> None:
+ """``eval`` reads its words again in this shell, with the command's prefix in
+ effect, so a ``$f`` quoted from the first reading names the script there;
+ a name ``eval`` may bind is unsettled after it. Inline code and a shell
+ reading a heredoc or here-string receive the command's environment, and an
+ unquoted heredoc body is expanded here, so a variable that holds the
+ script leaves them unresolved rather than a settled miss.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("readonly f=run.py; f=other.py; python3 run.py", 0.75),
+ ("echo x; readonly f=run.py; f=other.py; env -i f=run.py bash -c 'python3 \"$f\"'", 0.75),
+ ("readonly f=run.py; export f=other.py; python3 run.py", 1.0),
+ ("readonly f=run.py; f=other.py true; python3 run.py", 1.0),
+ ("readonly f=run.py; unset f; python3 run.py", 1.0),
+ ("dash -c 'readonly f=run.py; export f=other.py; python3 run.py'", 0.75),
+ ("zsh -c 'readonly f=run.py; unset f; python3 run.py'", 0.75),
+ ("ksh -c 'readonly f=run.py; f=other.py true; python3 run.py'", 1.0),
+ ("sh -c 'readonly f=run.py; f=other.py :; python3 run.py'", 0.75),
+ ("(readonly f=run.py; f=other.py); python3 run.py", 1.0),
+ ("{ readonly f=run.py; f=other.py; } | cat; python3 run.py", 1.0),
+ ("readonly f=run.py; (f=other.py; python3 run.py)", 0.75),
+ ('readonly f=run.py; for f in other.py; do python3 "$f"; done', 0.75),
+ ],
+)
+def test_nothing_after_a_refused_read_only_assignment_is_credited(check, command, expected) -> None:
+ """An assignment to a read-only name fails. A bare one stops every shell
+ (bash abandons the rest of the line); bash goes on after every other form
+ and ksh after a prefix, while dash, zsh, mksh and ash stop (measured).
+ Where the shell stops, a script named after the failure is unresolved; a
+ subshell, a pipeline stage included, stops alone.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("export f=run.py; code='f=other.py'; eval \"$code\"; bash -c 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; code='export -n f'; eval \"$code\"; bash -c 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; code='unset f'; eval \"$code\"; bash -c 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; code='declare +x f'; eval \"$code\"; bash -c 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; code='export f=other.py'; eval \"$code\"; bash -c 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; code='f=other.py'; eval eval \"$code\"; bash -c 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; code='f=other.py'; command eval \"$code\"; bash -c 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; code='export -n f'; eval '$code'; bash -c 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; code='f=other.py'; eval '$code'; bash -c 'python3 \"$f\"'", 1.0),
+ ("export f=other.py; code='f=run.py'; eval \"$code\"; bash -c 'python3 \"$f\"'", 0.75),
+ ("export f=run.py; code='echo hi'; eval \"$code\"; bash -c 'python3 \"$f\"'", 1.0),
+ ('export f=run.py; eval "$(echo f=other.py)"; bash -c \'python3 "$f"\'', 0.75),
+ ('f=run.py; code=\'g=1\'; eval "$code"; python3 "$f"', 1.0),
+ ("code='python3 run.py'; eval \"$code\"", 0.75),
+ ],
+)
+def test_eval_program_held_in_a_variable_is_read_after_this_shell_expands_it(check, command, expected) -> None:
+ """This shell expands eval's words before eval reads them, so ``eval "$code"``
+ runs what ``code`` holds: a name it reassigns, unsets or unexports is
+ unsettled afterwards, through nested ``eval`` and ``command eval`` too.
+ ``eval '$code'`` expands the text itself, where a word read from a
+ variable is a command, never an assignment. Text this walk cannot read
+ (a command substitution) leaves every bound name unsettled.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=run.py; declare -i f; f=run.py; python3 "$f"', 0.75),
+ ('f=other.py; declare -i f; f=run.py; python3 "$f"', 0.75),
+ ("export f=run.py; declare -a f; f=run.py; bash -c 'python3 \"$f\"'", 0.0),
+ ('declare -a f; f=run.py; python3 "$f"', 1.0),
+ ('declare -l f; f=RUN.PY; python3 "$f"', 1.0),
+ ('declare -l f; f=run.py; python3 "$f"', 1.0),
+ ('declare -u f; f=run.py; python3 "$f"', 0.75),
+ ('declare -i f; unset f; f=run.py; python3 "$f"', 1.0),
+ ('declare -i f; declare +i f; f=run.py; python3 "$f"', 1.0),
+ ('declare -i f; for f in run.py; do python3 "$f"; done', 0.75),
+ ],
+)
+def test_attributes_that_change_a_later_assignment_are_kept_with_the_name(check, command, expected) -> None:
+ """``declare -i``, ``-u`` and ``-n`` change what a later assignment stores, so
+ the value is unsettled until ``unset`` or ``+i`` removes the attribute;
+ ``-l`` lowercases it; an array (``-a``) keeps its value in this shell and
+ is never exported to a child.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('g=run.py; declare -n f=g; f=other.py; python3 "$g"', 0.75),
+ ('g=other.py; declare -n f=g; f=run.py; python3 "$g"', 0.75),
+ ('declare -n f=g; f=run.py; python3 "$g"', 0.75),
+ ('g=run.py; declare -n f=g; unset f; python3 "$g"', 0.75),
+ ('g=run.py; declare -n f=g; export f=other.py; python3 "$g"', 0.75),
+ ('g=run.py; declare -n f=g; read f <<< other.py; python3 "$g"', 0.75),
+ ('g=run.py; declare -n f=g; ((f=1)); python3 "$g"', 0.75),
+ ('g=run.py; f=g; declare -n f; f=other.py; python3 "$g"', 0.75),
+ ("export g=run.py; declare -n f=g; export -n f; bash -c 'python3 \"$g\"'", 0.75),
+ ("ksh -c 'g=run.py; nameref f=g; f=other.py; python3 \"$g\"'", 0.75),
+ ("mksh -c 'g=run.py; nameref f=g; f=other.py; python3 \"$g\"'", 0.75),
+ ("bash -c 'g=run.py; nameref f=g; f=other.py; python3 \"$g\"'", 1.0),
+ ('g=run.py; declare -n f=g; h=other.py; python3 "$g"', 1.0),
+ ('g=run.py; declare -n f=g; declare +n f; f=other.py; python3 "$g"', 1.0),
+ ('g=run.py; declare -n f=g; unset -n f; f=other.py; python3 "$g"', 1.0),
+ ('g=run.py; declare -n f=g; declare -n f=h; f=other.py; python3 "$g"', 1.0),
+ ('g=other.py; declare -n f=g; declare -n f=h; f=run.py; python3 "$g"', 0.0),
+ ('g=run.py; declare -n f; f=g; python3 "$g"', 1.0),
+ ('g=run.py; declare -n f; f=g; f=other.py; python3 "$g"', 0.75),
+ ('h=run.py; declare -n f; f=g; python3 "$h"', 1.0),
+ ('g=run.py; declare -n f="$(echo g)"; f=other.py; python3 "$g"', 0.75),
+ ],
+)
+def test_a_reference_passes_what_is_done_to_it_on_to_the_name_it_refers_to(check, command, expected) -> None:
+ """After ``declare -n f=g`` (``nameref`` in ksh and mksh), assigning,
+ exporting, unexporting, reading into or unsetting ``f`` acts on ``g``,
+ which is unsettled afterwards, bound or not. ``+n``, ``unset -n`` and a
+ second ``-n`` free or re-point ``f`` and leave ``g`` alone, and an
+ unrelated name leaves it settled. A reference that names nothing yet takes
+ the first value assigned as its name; where the text does not settle the
+ name, every bound name is unsettled.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("declare -i f; f=other.py; python3 run.py", 0.75),
+ ("declare -i f; export f=run.py; python3 run.py", 0.75),
+ ("declare -i f; for f in run.py; do :; done; python3 run.py", 0.75),
+ ("declare -i f; f=5; python3 run.py", 1.0),
+ ("declare -i f; f=x; python3 run.py", 1.0),
+ ("declare -i f; f=other.py true; python3 run.py", 1.0),
+ ("ksh -c 'typeset -i f; typeset f=other.py; python3 run.py'", 1.0),
+ ("mksh -c 'typeset -i f; typeset f=other.py; python3 run.py'", 0.75),
+ ("dash -c 'export f=1; export -n f; python3 run.py'", 0.75),
+ ("dash -c 'export f=1; command export -n f; python3 run.py'", 1.0),
+ ("bash -c 'export -Q f; python3 run.py'", 1.0),
+ ("mksh -c 'unset -n f; python3 run.py'", 0.75),
+ ("bash -c 'f=run.py; unset -n f; python3 \"$f\"'", 1.0),
+ ("ksh -c 'f=run.py; unset -n f; python3 \"$f\"'", 0.0),
+ ("zsh -c 'f=run.py; unset -Q f; python3 \"$f\"'", 1.0),
+ ('g=run.py; declare -n f; f=other.py; python3 "$g"', 0.75),
+ ('f=other.py; declare -n f; f=run.py; python3 "$f"', 1.0),
+ ("ksh -c 'f=other.py; typeset -n f; python3 run.py'", 0.75),
+ ],
+)
+def test_an_assignment_or_builtin_that_fails_where_the_shell_stops_ends_the_credit(check, command, expected) -> None:
+ """A value that is not a number, given to a ``-i`` name, is an error, and
+ bash exits there, as every shell with ``typeset`` does for most forms
+ (bash goes on after a prefix, ksh after ``typeset``). A special builtin
+ given an option the shell rejects stops a POSIX shell unless ``command``
+ runs it; bash and zsh go on. A reference that names nothing given a value
+ that is not a name fails as a read-only assignment does, and a declaration
+ that refers to such a name fails, where ksh stops. Where the shell stops,
+ a script named after the failure is unresolved (measured).
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("dash -c 'export g=run.py; read f <<< x; for f in x; do :; done; python3 \"$g\"'", 0.75),
+ ("sh -c 'export g=run.py; command unset -Q f; read f <<< 7; python3 \"$g\"'", 0.75),
+ ("dash -c 'cat <<< x; python3 run.py'", 0.75),
+ ("dash -c 'export g=run.py; read f < /dev/null; python3 \"$g\"'", 1.0),
+ ("bash -c 'cat <<< x; python3 run.py'", 1.0),
+ ("zsh -c 'cat <<< x; python3 run.py'", 1.0),
+ ("ksh -c 'cat <<< x; python3 run.py'", 1.0),
+ ("mksh -c 'cat <<< x; python3 run.py'", 1.0),
+ ("dash -c \"echo '<<<'; python3 run.py\"", 1.0),
+ ("dash -c 'cat <<< x; cat run.py'", 0.0),
+ ("cat <<< x; python3 run.py", 1.0),
+ ],
+)
+def test_a_here_string_the_shell_cannot_parse_is_never_full_credit(check, command, expected) -> None:
+ """dash and busybox ash have no here-string: ``<<<`` is a syntax error that
+ stops the line holding it, and a compound command spanning lines around
+ it, before anything on it runs (measured). Under those readings nothing
+ from that line on is credited as run; ``sh`` disagrees across its
+ readings and stays unresolved. Shells that read here-strings, and a
+ quoted ``<<<``, are unchanged.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("bash -c 'cat <<EOF\npython3 run.py\nEOF'", 0.75),
+ ("dash -c 'cat <<EOF\npython3 run.py\nEOF'", 0.75),
+ ('bash -c "cat <<EOF\npython3 run.py\nEOF"', 0.75),
+ ("bash -c 'cat <<EOF\nx\nEOF\npython3 run.py'", 1.0),
+ ("bash -c 'echo a\npython3 run.py'", 1.0),
+ ("bash -c 'python3 run.py <<EOF\nignored\nEOF'", 1.0),
+ ("dash -c 'cat <<EOF\na <<< b\nEOF\npython3 run.py'", 1.0),
+ ('bash -c "f=run.py; python3 \\"$f\\""', 0.0),
+ ('bash -c "f=run.py\npython3 $f"', 0.0),
+ ('f=run.py; bash -c "python3 \\"$f\\""', 1.0),
+ ('bash -c "f=run.py; python3 \\"\\$f\\""', 1.0),
+ ("bash -c 'f=run.py; python3 \"$f\"'", 1.0),
+ ],
+)
+def test_a_heredoc_inside_a_c_payload_keeps_its_body_as_data(check, command, expected) -> None:
+ """A newline inside the quotes of a ``-c`` payload is kept for the payload's
+ shell, so a heredoc there takes its body after its line, as at the top
+ level: a script named only in the body is unresolved, and a command after
+ the terminator is walked as a command. This shell expands a ``$`` it did
+ not leave quoted before the child reads the payload, a name it never
+ bound to nothing, so ``bash -c "f=run.py; python3 $f"`` runs ``python3``
+ with no script.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ('f=run.py; let f=1; python3 "$f"', 0.75),
+ ('f=run.py; let "f=1"; python3 "$f"', 0.75),
+ ('f=run.py; : $((f=1)); python3 "$f"', 0.75),
+ ('f=run.py; x=$((f+=1)); python3 "$f"', 0.75),
+ ('f=run.py; : $[f=1]; python3 "$f"', 0.75),
+ ('f=run.py; eval "let f=1"; python3 "$f"', 0.75),
+ ('f=run.py; let g=1; python3 "$f"', 1.0),
+ ('f=run.py; : $((g=1)); python3 "$f"', 1.0),
+ ("f=run.py; echo '$((f=1))'; python3 \"$f\"", 1.0),
+ ('f=run.py; (: $((f=1))); python3 "$f"', 1.0),
+ ('f=run.py; echo $((f=1)) | cat; python3 "$f"', 1.0),
+ ('f=run.py; : $( (f=1) ); python3 "$f"', 1.0),
+ ('f=run.py; python3 "$f" $((f=1))', 1.0),
+ ('f=other.py; let f=1; python3 "$f"', 0.0),
+ ],
+)
+def test_arithmetic_outside_an_arithmetic_command_binds_too(check, command, expected) -> None:
+ """``let``, ``$((...))`` and ``$[...]`` assign as ``((...))`` does: a
+ variable that held the script ends as a number, or unchanged after an
+ error, so a later read of it is unresolved. Quoted text, a subshell, a
+ pipeline stage and a command substitution keep this shell's value, and a
+ word expanded before the assignment reads the value it had.
+ """
+ assert check(_bash(command, "done"), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("LANG=C for g in; do python3 run.py; done", 0.75),
+ ("f=x for g in; do echo; done; python3 run.py", 0.75),
+ ("A=1 for g in x; do python3 run.py; done", 0.75),
+ ("A=1 B=2 for g in x; do python3 run.py; done", 0.75),
+ ("A=1 select g in x; do python3 run.py; break; done", 0.75),
+ ("A=1 while true; do python3 run.py; break; done", 0.75),
+ ("A=1 until false; do python3 run.py; break; done", 0.75),
+ ("A=1 if true; then python3 run.py; fi", 0.75),
+ ("A=1 case x in x) python3 run.py;; esac", 0.75),
+ ("A=1 { python3 run.py; }", 0.75),
+ ("A=1 ( python3 run.py )", 0.75),
+ ("A=1 [[ -n x ]] && python3 run.py", 0.75),
+ ("A=1 (( 1 )) && python3 run.py", 0.75),
+ ("python3 run.py; A=1 for g in x; do :; done", 0.75),
+ ("sh -c 'A=1 for g in x; do python3 run.py; done'", 0.75),
+ ("A=1 for x; python3 run.py", 0.75),
+ ("python3 run.py\nA=1 for g in x; do :; done", 1.0),
+ (r'f=run.py\; for "f"', 0.0),
+ ("A=1 for g in x; do cat run.py; done", 0.0),
+ ("A=1 if true; then cat run.py; fi", 0.0),
+ ("A=(x y); python3 run.py", 1.0),
+ ("A='for' python3 run.py", 1.0),
+ ("for g in x; do A=1 python3 run.py; done", 1.0),
+ ("if true; then A=1 python3 run.py; fi", 1.0),
+ ("A=1; for g in x; do python3 run.py; done", 1.0),
+ ('f=run.py; for f in; do :; done; python3 "$f"', 1.0),
+ ],
+)
+def test_a_compound_word_after_an_assignment_is_refused_text(check, command, expected) -> None:
+ """A compound's opening word is reserved only where a command starts.
+ After an assignment, bash, bash --posix, dash, zsh, ksh, mksh and busybox
+ ash all refuse it with the compound's own syntax, so the line runs
+ nothing: an invocation on it is partial, never full credit, and never an
+ exception. A command completed on a line before it has already run (see
+ the next test). ``A=(`` is an array and ``A='for'`` a value, and a loop
+ word where a command starts is a loop.
+ """
+ assert check(_bash(command), EXPECTED_SCRIPT)["score"] == expected
+
+
+_REFUSED = "A=1 for g in x; do :; done"
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("python3 run.py\nA=1 for g in x; do :; done", 1.0),
+ ("python3 run.py # comment\nA=1 for g in x; do :; done", 1.0),
+ ("# first\n\npython3 run.py\nA=1 ( : )", 1.0),
+ ("python3 run.py;\nA=1 if true; then :; fi", 1.0),
+ ("python3 run.py | cat\nA=1 for g in x; do :; done", 1.0),
+ ("python3 run.py > out 2>&1 &\nA=1 { :; }", 1.0),
+ ("python3 run.py <<EOF\nA=1 for g in x; do :; done\nEOF\nA=1 ( : )", 1.0),
+ ("python3 run.py\necho a\nA=1 for g in x; do :; done", 1.0),
+ ("python3 run.py\ncat <<< x", 1.0),
+ ("python3 run.py; A=1 for g in x; do :; done", 0.75),
+ ("if true; then\npython3 run.py\nA=1 for g in x; do :; done\nfi", 0.75),
+ ("python3 run.py &&\nA=1 for g in x; do :; done", 0.75),
+ ("python3 run.py |\nA=1 ( : )", 0.75),
+ ("python3 run.py \\\n; A=1 for g in x; do :; done", 0.75),
+ ("A=1 for g in x; do :; done\npython3 run.py", 0.75),
+ ('export f=run.py\nA=1 for g in x; do :; done\npython3 "$f"', 0.75),
+ ("cat run.py\nA=1 for g in x; do :; done\npython3 run.py", 0.75),
+ ("cat run.py\nA=1 for g in x; do :; done", 0.0),
+ ],
+)
+def test_a_command_completed_before_a_refused_line_keeps_its_credit(check, command, expected) -> None:
+ """bash, bash --posix, dash, ksh, mksh and busybox ash parse and run a
+ text one complete command at a time, so the first command, completed on
+ a line before the one the shell refuses, has already run (measured). It
+ keeps its credit when it is a plain pipeline: blank lines and comments
+ may come before it, a heredoc body may follow it, and what comes after
+ it does not matter. A line joined to the refused one by ``&&``, ``|`` or
+ a trailing backslash, and a compound spanning lines around it, run
+ nothing, and nothing after it runs.
+ """
+ assert check(_bash(command), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ "command",
+ [
+ "echo a\npython3 run.py\nA=1 if true; then :; fi",
+ "for x in 1; do\npython3 run.py\ndone\nA=1 for g in x; do :; done",
+ "{ python3 run.py\n}\nA=1 ( : )",
+ "f() {\npython3 run.py\n}\nf\nA=1 for g in x; do :; done",
+ "x=$(echo a\n)\npython3 run.py\nA=1 ( : )",
+ "a=(1\n2)\npython3 run.py\nA=1 { :; }",
+ "case x in\nx) :;;\nesac\npython3 run.py\nA=1 for g in x; do :; done",
+ "echo 'a\nb'; python3 run.py\nA=1 for g in x; do :; done",
+ "cat <<EOF\nx\nEOF\npython3 run.py\nA=1 for g in x; do :; done",
+ 'export f=run.py\npython3 "$f"\nA=1 for g in x; do :; done',
+ "exit\npython3 run.py\nA=1 for g in x; do :; done",
+ "exec true\npython3 run.py\nA=1 for g in x; do :; done",
+ "set -n\npython3 run.py\nA=1 for g in x; do :; done",
+ "kill $$\npython3 run.py\nA=1 for g in x; do :; done",
+ "set -e; false\npython3 run.py\nA=1 for g in x; do :; done",
+ "false && python3 run.py\nA=1 for g in x; do :; done",
+ "true || python3 run.py\nA=1 for g in x; do :; done",
+ "PATH= python3 run.py\nA=1 for g in x; do :; done",
+ "f() {\npython3 run.py\n}\nA=1 for g in x; do :; done",
+ "f() {\npython3 run.py\n}\nf; A=1 for g in x; do :; done",
+ "bash -c 'python3 run.py'\nA=1 for g in x; do :; done",
+ "bash -c 'python3 run.py >'\nA=1 for g in x; do :; done",
+ "sh -c 'python3 run.py; fi'\nA=1 for g in x; do :; done",
+ ],
+)
+def test_only_a_plain_first_command_is_credited_before_a_refused_line(check, command) -> None:
+ """The walk does not model every command that stops the ones after it
+ (``exit``, ``exec``, ``set -n``, ``kill $$``, a failed ``set -e`` step,
+ an emptied ``PATH``) or that decides whether they run (``&&``, ``||``,
+ a function called or not), and a ``-c`` payload's own text may be refused
+ for a reason the walk does not detect. So when a later line is refused,
+ only the first command is credited, and only a plain pipeline with no
+ payload: nothing ran before it that could have stopped it. Everything
+ else stays partial, as at 4594693, whether or not it ran. Measured: the
+ first ten and the plain payload run in bash, bash --posix, ksh and mksh,
+ and in dash and busybox ash but for the array; the others run nothing.
+ """
+ assert check(_bash(command), EXPECTED_SCRIPT)["score"] == 0.75
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ # The internal review's two cases, and the variants it compared.
+ (f"python3 run.py >\n{_REFUSED}", 0.75),
+ (f"python3 run.py\r\n{_REFUSED}", 0.75),
+ (f"python3 run.py <\n{_REFUSED}", 0.75),
+ (f"python3 run.py 2>\n{_REFUSED}", 0.75),
+ (f"python3 run.py |&\r\n{_REFUSED}", 0.75),
+ (f"time -p python3 run.py\r\n{_REFUSED}", 0.75),
+ (f"f() {{\npython3 run.py\n}}\nf\r\n{_REFUSED}", 0.75),
+ (f"f() {{\r\npython3 run.py\r\n}}\r\nf\r\n{_REFUSED}", 0.75),
+ (f"python3 run.py; function f\r\n{_REFUSED}", 0.75),
+ (f"python3 run.py; f()\r\n{_REFUSED}", 0.75),
+ (f"python3 run.py;\r\n{_REFUSED}", 0.75),
+ # Every redirection operator without its operand refuses its line.
+ *(
+ (f"python3 run.py {op}\n{_REFUSED}", 0.75)
+ for op in (">>", ">|", "&>", "&>>", "<>", ">&", "<&", "2>&", "1>")
+ ),
+ (f"python3 run.py >;\n{_REFUSED}", 0.75),
+ (f"python3 run.py > # out\n{_REFUSED}", 0.75),
+ (f"python3 run.py >#out\n{_REFUSED}", 0.75),
+ (f"python3 run.py 2>#out\n{_REFUSED}", 0.75),
+ (f"python3 run.py >\nout\n{_REFUSED}", 0.75),
+ (f"python3 run.py <<<\necho a\n{_REFUSED}", 0.75),
+ (f"python3 run.py 2>&1\r\n{_REFUSED}", 0.75),
+ # Other lines every modelled shell refuses on their own.
+ (f"python3 run.py; ;\n{_REFUSED}", 0.75),
+ (f"python3 run.py ;;\n{_REFUSED}", 0.75),
+ (f"; python3 run.py\n{_REFUSED}", 0.75),
+ (f"python3 run.py & ;\n{_REFUSED}", 0.75),
+ (f"python3 run.py; then\n{_REFUSED}", 0.75),
+ (f"if python3 run.py; fi\n{_REFUSED}", 0.75),
+ (f"python3 run.py (x)\n{_REFUSED}", 0.75),
+ (f"python3 run.py\\\necho a\n{_REFUSED}", 0.75),
+ (f'echo "$(echo "a\npython3 run.py\nb")"\n{_REFUSED}', 0.75),
+ # The same commands complete: they run, and keep their credit.
+ (f"python3 run.py '>'\n{_REFUSED}", 1.0),
+ (f"python3 run.py \\>\n{_REFUSED}", 1.0),
+ (f"python3 run.py 2>&1\n{_REFUSED}", 1.0),
+ (f"python3 run.py > out\n{_REFUSED}", 1.0),
+ (f"python3 run.py >&-\n{_REFUSED}", 1.0),
+ (f"python3 run.py >out #c\n{_REFUSED}", 1.0),
+ (f"python3 run.py\n{_REFUSED}\r", 1.0),
+ ],
+)
+def test_a_line_refused_on_its_own_is_not_credited_before_a_refused_line(check, command, expected) -> None:
+ """A redirection operator without its operand (at the line's end, before
+ ``;``, or before a comment, which a ``#`` starts after an operator too),
+ an empty command, a reserved word out of place or a parenthesis after a
+ word is refused on its own line by bash, bash --posix, dash, mksh and
+ busybox ash, which then run nothing (measured); ksh accepts an empty
+ command and ``2>#out``. A carriage return before the newline is part of
+ the word before it: ``python3 run.py\\r`` opens ``run.py\\r``, and an
+ escaped newline joins the next line to the word. None of these is a
+ plain pipeline, so none is credited and the text stays partial, as at
+ 4594693, including the forms some shells run (``python3 run.py;\\r``).
+ A CR on the refused line itself does not touch the command before it.
+ """
+ assert check(_bash(command), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "expected"),
+ [
+ ("bash -c 'python3 run.py\nA=1 for g in x; do :; done'", 0.75),
+ ("ksh -c 'python3 run.py\nA=1 ( : )'", 0.75),
+ ("dash -c 'python3 run.py\ncat <<< x'", 0.75),
+ ("sh -c 'python3 run.py\ncat <<< x'", 0.75),
+ ("zsh -c 'python3 run.py\nA=1 for g in x; do :; done'", 0.75),
+ ("dash -c 'cat <<< x\npython3 run.py'", 0.75),
+ ("dash -c 'python3 run.py; cat <<< x'", 0.75),
+ ("bash -c 'python3 run.py >\nA=1 for g in x; do :; done'", 0.75),
+ ("bash -c 'python3 run.py\r\nA=1 for g in x; do :; done'", 0.75),
+ ("bash -c 'echo a\npython3 run.py\nA=1 for g in x; do :; done'", 0.75),
+ ],
+)
+def test_a_payload_with_a_line_its_shell_refuses_stays_partial(check, command, expected) -> None:
+ """A ``-c`` payload is refused by its own shell: a here-string under dash
+ and busybox ash, a compound's opening word after an assignment under
+ each. bash, dash, ksh, mksh and ash run the commands before the refused
+ line and zsh runs none, but the payload is read from what the tokenizer
+ left of it, not from the text its shell reads (a carriage return there
+ becomes a line's end), so no command in it is credited on its own. The
+ payload stays partial, as at 4594693.
+ """
+ assert check(_bash(command), EXPECTED_SCRIPT)["score"] == expected
+
+
+def _native(command: str, shell: object = None) -> list[dict[str, object]]:
+ arguments: dict[str, object] = {"cmd": command}
+ if shell is not None:
+ arguments["shell"] = shell
+ return [{"action": "exec_command", "action_input": arguments, "observation": "done"}]
+
+
+_LAST_STAGE_BINDING = 'printf "" | for f in run.py; do :; done; python3 "$f"'
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "shell", "expected"),
+ [
+ (_LAST_STAGE_BINDING, "/bin/zsh", 1.0),
+ (_LAST_STAGE_BINDING, "zsh", 1.0),
+ (_LAST_STAGE_BINDING, "/usr/local/bin/zsh5.9", 1.0),
+ (_LAST_STAGE_BINDING, "/usr/bin/ksh", 1.0),
+ (_LAST_STAGE_BINDING, "/bin/bash", 0.0),
+ (_LAST_STAGE_BINDING, "C:\\msys64\\usr\\bin\\bash.exe", 0.0),
+ (_LAST_STAGE_BINDING, "/bin/mksh", 0.0),
+ (_LAST_STAGE_BINDING, "/bin/sh", 0.0),
+ (_LAST_STAGE_BINDING, "", 0.0),
+ (_LAST_STAGE_BINDING, None, 0.0),
+ ('printf "" | { f=run.py; }; python3 "$f"', "/bin/zsh", 1.0),
+ ('printf "" | { f=run.py; }; python3 "$f"', "/bin/bash", 0.0),
+ ("python3 run.py\nA=1 for g in x; do :; done", "/bin/zsh", 0.75),
+ ("python3 run.py\nA=1 for g in x; do :; done", "/bin/bash", 1.0),
+ ("python3 run.py\nA=1 for g in x; do :; done", "/bin/dash", 1.0),
+ ("cat <<< x; python3 run.py", "/bin/dash", 0.75),
+ ("cat <<< x; python3 run.py", "/bin/bash", 1.0),
+ ("python3 run.py\ncat <<< x", "/bin/dash", 1.0),
+ ("python3 run.py && true\ncat <<< x", "/bin/dash", 0.75),
+ ("python3 run.py >\nA=1 for g in x; do :; done", "/bin/bash", 0.75),
+ ("python3 run.py >\nA=1 for g in x; do :; done", "/bin/zsh", 0.75),
+ ("python3 run.py\r\nA=1 for g in x; do :; done", "/bin/bash", 0.75),
+ ("python3 run.py\r\nA=1 for g in x; do :; done", "/bin/zsh", 0.75),
+ ('declare f=run.py; python3 "$f"', "/bin/dash", 0.0),
+ ('typeset f=run.py; python3 "$f"', "/bin/mksh", 1.0),
+ ('f=run.py; ((f=1)); python3 "$f"', "/bin/dash", 1.0),
+ ('f=run.py; ((f=1)); python3 "$f"', "/bin/zsh", 0.75),
+ ],
+)
+def test_a_native_tool_call_is_read_under_the_shell_it_names(check, command, shell, expected) -> None:
+ """A native call can name the shell that ran its command (Codex's
+ ``exec_command`` keeps ``shell`` in its arguments). The rules that differ
+ between shells follow it: zsh and ksh keep a binding made in a
+ pipeline's last stage, dash has no here-string, no ``declare`` and no
+ arithmetic command, and zsh parses a whole text before running it. A
+ call that names no shell is read as bash, as before; ``sh`` is walked
+ under each shell it may be, and disagreement is unresolved.
+ """
+ assert check(_native(command, shell), EXPECTED_SCRIPT)["score"] == expected
+
+
[email protected]("check", IMPLEMENTATIONS)
+ ("command", "shell", "expected"),
+ [
+ (_LAST_STAGE_BINDING, "/usr/bin/fish", 0.75),
+ ("python3 run.py", "/usr/bin/fish", 0.75),
+ ("cat run.py", "pwsh.exe", 0.75),
+ ("echo done", "/usr/bin/fish", 0.0),
+ (_LAST_STAGE_BINDING, ["/bin/zsh"], 0.75),
+ ],
+)
+def test_a_shell_this_walk_does_not_model_leaves_the_command_unresolved(check, command, shell, expected) -> None:
+ """A shell outside the modelled ones reads the text by rules this walk does
+ not know, so no definite score is given: a command that names the script
+ is unresolved, and one that never names it is not evidence about it.
+ """
+ result = check(_native(command, shell), EXPECTED_SCRIPT)
+ assert result["score"] == expected
+
+
+_DAMAGE_BASES = [
+ "python3 run.py",
+ 'f=run.py; for f in; do :; done; python3 "$f"',
+ "for f in run.py; do python3 $f; done | cat",
+ "LANG=C python3 -u run.py 2>&1 > out.txt",
+ "if [ -f run.py ]; then python3 run.py; elif true; then :; else cat run.py; fi",
+ "case x in x) python3 run.py;; *) cat run.py;; esac",
+ "while read -r l; do python3 run.py; done < in.txt",
+ "(cd x && python3 run.py); { cat run.py; }",
+ "cat <<EOF\npython3 run.py\nEOF\npython3 run.py",
+ "python3 <<A <<B run.py\nx\nA\ny\nB",
+ 'cat <<< "a; b"; python3 run.py',
+ "bash -c 'f=run.py; python3 \"$f\"'",
+ "export f=run.py; env -u f bash -c 'python3 \"$f\"'",
+ 'readonly f=run.py; declare -n g=f; python3 "$g"',
+ 'f=run.py; let f=1; : $((f=2)); python3 "$f"',
+ 'f=run.py; eval "g=$f"; python3 "$g"',
+ '(((for f in run.py; do :; done; python3 "$f")) | cat)',
+ "timeout -- 30 env --help python3 run.py && xargs -r python3 run.py",
+ "sh -c 'cat <<< x; python3 run.py'",
+ "A=1 for g in x; do python3 run.py; done",
+ "echo a # it's\npython3 run.py\nA=1 for g in x; do :; done",
+ "cat <<EOF\nx\nEOF\ncase x in\nx) :;;\nesac\npython3 run.py\nA=1 ( : )",
+]
+
+
+def _damaged(command: str) -> list[str]:
+ """Every prefix and every single-character deletion of a command: the
+ shapes a cut-off or mistyped tool call takes."""
+ return sorted(
+ {command[:end] for end in range(len(command))}
+ | {command[:index] + command[index + 1 :] for index in range(len(command))}
+ )
+
+
[email protected]("check", IMPLEMENTATIONS)
[email protected]("command", _DAMAGE_BASES)
+def test_a_damaged_command_is_scored_never_raised(check, command) -> None:
+ """A malformed command gets a score like any other. Deleting one ``;``
+ from the second base gives ``f=run.py for f in; do :; ...``, which raised
+ ``IndexError`` before the walk read a loop word after an assignment as
+ the ordinary word it is.
+ """
+ for damaged in _damaged(command):
+ result = check(_bash(damaged), EXPECTED_SCRIPT)
+ assert result["score"] in {0.0, 0.75, 1.0}, damaged