Skip to content

Distribute and run LLMs with a single file.

server: fix built-in shell tools on Windows (#1066) (#1069)

Manu Nicholas Jacobfc732810704f
diff --git a/docs/built-in-tools.md b/docs/built-in-tools.md
index c8252aba..28f97f6d 100644
--- a/docs/built-in-tools.md
+++ b/docs/built-in-tools.md
@@ -61,6 +61,9 @@ Note: `--agent` enables all built-in tools and the experimental MCP proxy. Prefe
| `get_datetime` | Read | Get the server's current date and time. |
| `get_info` | Read | Get the runtime operating system and working directory. |
+`exec_shell_command` runs its command with `cmd.exe /c` on Windows and with
+`sh -c` everywhere else, so on Windows write commands in `cmd.exe` syntax.
+
Tool support in llama.cpp is experimental and subject to change, so the contents
of this table might differ from one llamafile version to another. To inspect the
exact tool definitions and parameter signatures exposed by a running server,
diff --git a/llama.cpp.patches/README.md b/llama.cpp.patches/README.md
index a64f50a8..bf58872b 100644
--- a/llama.cpp.patches/README.md
+++ b/llama.cpp.patches/README.md
@@ -113,7 +113,8 @@ These patches address compatibility issues when building with Cosmopolitan libc
| `common_arg.cpp.patch` | Adds `COSMOCC` platform detection for `PATH_MAX` (includes `linux/limits.h`) |
| `common_common.cpp.patch` | Adds platform-aware cache directory detection for Cosmopolitan (`fs_get_cache_directory`: checks `LOCALAPPDATA`, `XDG_CACHE_HOME`, falls back to `~/.cache/`) and the same for the config directory (`fs_get_config_directory`, new in b10441: `APPDATA`, `XDG_CONFIG_HOME`, `~/.config/`) — both `#else` out to `#error Unknown architecture` upstream, since cosmocc defines none of `__linux__`/`__APPLE__`/`_WIN32`; also adds mmproj model size estimation to GPU fit params so the fit algorithm reserves enough VRAM for multimodal projectors |
| `common_download.cpp.patch` | Adds `COSMOCC` platform detection for `PATH_MAX` |
-| `vendor_sheredom_subprocess.h.patch` | Under `__COSMOPOLITAN__`, rewrites `subprocess_error_from_errno()`'s `switch` over `E*` values as an `if`/`else` chain. Cosmopolitan resolves errno at runtime (one binary, every host OS), so those are not constant expressions and cannot be case labels — upstream's version fails with `case value is not a constant expression`. Same mapping, no behavior change. Needed since b10441, which added this function; the header only enters the build when `LLAMA_SUBPROCESS` is set (see Server Integration) |
+| `vendor_sheredom_subprocess.h.patch` | Under `__COSMOPOLITAN__`, rewrites `subprocess_error_from_errno()`'s `switch` over `E*` values as an `if`/`else` chain. Cosmopolitan resolves errno at runtime (one binary, every host OS), so those are not constant expressions and cannot be case labels — upstream's version fails with `case value is not a constant expression`. Same mapping, no behavior change. Needed since b10441, which added this function; the header only enters the build when `LLAMA_SUBPROCESS` is set (see Server Integration). Also decodes the child's exit status on Windows, where Cosmopolitan's `waitpid()` returns the NT exit code as the wait status: Cosmopolitan children exit with `code << 8`, native ones (`cmd.exe`) with the plain code, so upstream's `WIFEXITED`/`WEXITSTATUS` turned `exit 3` into 1 and `exit 128` into 0. A status with its low byte set is now taken as a native exit code (#1066) |
+| `tools_server_server-tools.cpp.patch` | Under `__COSMOPOLITAN__`, picks the shell for the built-in `exec_shell_command` and `get_info` tools at run time with `IsWindows()`: `cmd.exe /c` on Windows, `sh -c` / `uname -a` elsewhere (and always inside a `--tools-runtime` isolate), the same choice upstream makes at compile time with `_WIN32`, which the cosmocc build never defines. The name is spelled `cmd.exe` because Cosmopolitan's `posix_spawnp()` does not try the `.exe` suffix when it searches `PATH`. Without it both tools fail on Windows with *"failed to spawn process"* (#1066) |
### Threading and Signal Handling
diff --git a/llama.cpp.patches/patches/tools_server_server-tools.cpp.patch b/llama.cpp.patches/patches/tools_server_server-tools.cpp.patch
new file mode 100644
index 00000000..ec82d521
--- /dev/null
+++ b/llama.cpp.patches/patches/tools_server_server-tools.cpp.patch
@@ -0,0 +1,40 @@
+diff --git a/tools/server/server-tools.cpp b/tools/server/server-tools.cpp
+--- a/llama.cpp/tools/server/server-tools.cpp
++++ b/llama.cpp/tools/server/server-tools.cpp
+@@ -33,6 +33,10 @@
+ # include <unistd.h>
+ #endif
+
++#if defined(__COSMOPOLITAN__)
++# include <cosmo.h>
++#endif
++
+ namespace fs = std::filesystem;
+
+ //
+@@ -1283,6 +1287,13 @@ struct server_tool_exec_shell_command : server_tool {
+ std::vector<std::string> args = !json_value(params, "runtime", std::string()).empty()
+ ? std::vector<std::string>{"sh", "-c", command}
+ : std::vector<std::string>{"cmd", "/c", command};
++#elif defined(__COSMOPOLITAN__)
++ // one cosmopolitan binary runs on every OS and _WIN32 is never defined, so the host is
++ // checked at run time; cosmopolitan's posix_spawnp does not try ".exe" on PATH, so the
++ // name carries it
++ std::vector<std::string> args = IsWindows() && json_value(params, "runtime", std::string()).empty()
++ ? std::vector<std::string>{"cmd.exe", "/c", command}
++ : std::vector<std::string>{"sh", "-c", command};
+ #else
+ std::vector<std::string> args = {"sh", "-c", command};
+ #endif
+@@ -1728,6 +1739,11 @@ struct server_tool_get_info : server_tool {
+ std::vector<std::string> args = !json_value(params, "runtime", std::string()).empty()
+ ? std::vector<std::string>{"uname", "-a"}
+ : std::vector<std::string>{"cmd", "/c", "ver"};
++#elif defined(__COSMOPOLITAN__)
++ // same choice as the _WIN32 branch, made at run time (see exec_shell_command)
++ std::vector<std::string> args = IsWindows() && json_value(params, "runtime", std::string()).empty()
++ ? std::vector<std::string>{"cmd.exe", "/c", "ver"}
++ : std::vector<std::string>{"uname", "-a"};
+ #else
+ std::vector<std::string> args = {"uname", "-a"};
+ #endif
diff --git a/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch b/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch
index 12965efa..cdd89471 100644
--- a/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch
+++ b/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch
@@ -27,11 +27,58 @@ diff --git a/vendor/sheredom/subprocess.h b/vendor/sheredom/subprocess.h
switch (error) {
case ENOENT:
case ENOTDIR:
-@@ -636,6 +655,7 @@ int subprocess_error_from_errno(int error) {
+@@ -636,6 +655,22 @@ int subprocess_error_from_errno(int error) {
default:
return subprocess_error_unknown;
}
+#endif
++}
++#endif
++
++#if defined(__COSMOPOLITAN__)
++#include <libc/dce.h> /* IsWindows(); <cosmo.h> would bring in its defer() macro */
++/* On Windows, Cosmopolitan's waitpid() reports the child's NT exit code as the
++ wait status. Cosmopolitan programs exit with (code << 8), so theirs decode as
++ usual, but a native program such as cmd.exe exits with the plain code: exit 3
++ reads as "killed by signal 3" (reported as EXIT_FAILURE) and exit 128 reads as
++ a clean exit 0. A status with its low byte set is taken as a native code. */
++static int subprocess_return_status_from_wait_status(int status) {
++ if (IsWindows()) {
++ return (status & 0xff) ? status : (status >> 8);
++ }
++ return WIFEXITED(status) ? WEXITSTATUS(status) : EXIT_FAILURE;
}
#endif
+@@ -1943,11 +1978,15 @@ int subprocess_join(struct subprocess_s *const process,
+
+ process->child = 0;
+
++#if defined(__COSMOPOLITAN__)
++ process->return_status = subprocess_return_status_from_wait_status(status);
++#else
+ if (WIFEXITED(status)) {
+ process->return_status = WEXITSTATUS(status);
+ } else {
+ process->return_status = EXIT_FAILURE;
+ }
++#endif
+
+ process->alive = 0;
+ }
+@@ -2157,11 +2196,15 @@ int subprocess_alive(struct subprocess_s *const process) {
+
+ // If the process was successfully waited on we need to cleanup now.
+ if (!is_alive) {
++#if defined(__COSMOPOLITAN__)
++ process->return_status = subprocess_return_status_from_wait_status(status);
++#else
+ if (WIFEXITED(status)) {
+ process->return_status = WEXITSTATUS(status);
+ } else {
+ process->return_status = EXIT_FAILURE;
+ }
++#endif
+
+ // Since we've already successfully waited on the process, we need to wipe
+ // the child now.
diff --git a/tests/integration/tests/test_builtin_tools.py b/tests/integration/tests/test_builtin_tools.py
new file mode 100644
index 00000000..38867f59
--- /dev/null
+++ b/tests/integration/tests/test_builtin_tools.py
@@ -0,0 +1,68 @@
+"""Built-in server tools that start a child process (POST /tools).
+
+exec_shell_command and get_info are the two built-in tools that spawn a
+process. On Windows they used to fail every time ("failed to spawn process",
+issue #1066): the cosmopolitan build never defines _WIN32, so it took the POSIX
+branch and asked for `sh` / `uname`, which a stock Windows install does not
+have. Once that was fixed, exit codes from native Windows programs were also
+misreported (`exit 3` came back as 1, `exit 128` as 0). The requests below are
+written to pass on every host OS: `echo` and `exit` exist in both `sh -c` and
+`cmd.exe /c`.
+
+The server runs with --unsecure because the default CPU-server sandbox on
+Linux and OpenBSD blocks process creation (see docs/built-in-tools.md).
+"""
+
+import pytest
+import requests
+
+from utils.llamafile import LlamafileRunner
+
+MARKER = "llamafile-tools-ok"
+
+
+def _call_tool(port: int, tool: str, params: dict, timeout: float) -> dict:
+ response = requests.post(
+ f"http://127.0.0.1:{port}/tools",
+ json={"tool": tool, "params": params},
+ timeout=timeout,
+ )
+ response.raise_for_status()
+ return response.json()
+
+
+class TestBuiltinProcessTools:
+ def test_exec_shell_command_and_get_info_spawn(self, executable, model, server_port, timeouts):
+ runner = LlamafileRunner(executable=executable, model=model, gpu="disable", no_warmup=True)
+ proc = runner.start_server(
+ port=server_port,
+ extra_args=["--unsecure", "--tools", "exec_shell_command,get_info"],
+ )
+ try:
+ assert LlamafileRunner.wait_for_server(
+ server_port, timeout=timeouts.server_ready, proc=proc
+ ), "Server did not become ready"
+
+ shell = _call_tool(
+ server_port, "exec_shell_command", {"command": f"echo {MARKER}"},
+ timeouts.http_request,
+ )
+ text = shell.get("plain_text_response", "")
+ assert "failed to spawn process" not in text, shell
+ assert MARKER in text, shell
+ assert "[exit code: 0]" in text, shell
+
+ # a failing command must report its own exit code, not 0 or 1
+ failed = _call_tool(
+ server_port, "exec_shell_command", {"command": "exit 3"},
+ timeouts.http_request,
+ )
+ assert "[exit code: 3]" in failed.get("plain_text_response", ""), failed
+
+ info = _call_tool(server_port, "get_info", {}, timeouts.http_request)
+ assert info.get("os") not in (None, "", "unknown"), info
+ finally:
+ proc.terminate()
+ proc.wait()