Distribute and run LLMs with a single file.
agentfile — design
agentfile is a self-contained, multi-platform APE binary that runs a focused agent loop from a single CLI invocation. The product promise: preconfigure an agent (model + system prompt + tool allowlist) and share it as one file that runs anywhere, llamafile-style.
Upstream llama.cpp ships the tools (server tools, isolation runtimes,
permission_write metadata) and a thin client-side agent loop in its web
UI. What it does not have, and what agentfile is for: a serverless one-shot
CLI, single-file packaged agents, session/trace recording, and KV-warm
packaging. agentfile is the file format for shipping agents, not
another harness.
Decisions
- agent.cpp provides the loop and the model wrapper: a submodule
plus local patches (
agent.cpp.patches/), no vendored fork, no upstream PRs for now. - Tools come from llama.cpp's server tools through one adapter.
agentfile's own tools are written as
server_toolsubclasses, so the adapter is the only bridge. - Interaction: unix-tool one-shot by default,
-i/--interactiveopt-in. No TUI. - Network:
http_fetchandweb_searchare built in, over HTTPS via mbedtls. No hostcurl: it would break the single-file promise. - Packaging:
.argsplus zipalign, as llamafile does. - Records: two existing formats, nothing invented: pi's session format and OTLP/JSON spans, written without the OTel SDK.
- API drift: depending on server-tools internals is a conscious choice. We can pin, or start vendoring, if they churn too hard.
Architecture
One translation unit, agentfile.cpp, plus headers:
server_tools_adapter.h:ServerToolboxowns llama.cpp'sserver_toolsregistry plus the native tools;ServerToolAdapterexposes one tool as anagent_cpp::Tool. Every call goes throughserver_tools::handle_post, the handler behind llama-server'sPOST /tools: upstream drops the keys the model must not set (cwd,runtime,resp_type), routes the call to the--tools-runtimeisolate and maps exceptions to error responses. The request body is also the json boundary: agent.cpp speaksnlohmann::json, server tools llama.cpp'scommon_json.tools/: native tools (get_datetime,http_fetch,web_search). They stay here because llama.cpp keeps its server tools to minimal I/O and shell tools, and sends date/time and web search to MCP or the web UI.callbacks/: agent.cpp callbacks, registered in this order: session recorder, confirmation, progress, iteration cap, trace, error recovery. The trace comes after the confirmation so tool spans measure execution, not the time the user spent deciding, and after the iteration cap so the call the cap refuses gets no chat span. Error recovery comes after the observers so they record a failed call as failed before it is handed back to the model.util.h: terminal escaping (terminal_text), asking on the terminal (ask_terminal), ids, timestamps.
The toolset and the record files are set up before GPU init and the
model load, so a bad --tools, --tools-runtime, --session or
--trace value fails at once.
agent.cpp patches
Pinned at upstream 63d23da (v0.4.0). The build compiles src/model.cpp
and src/agent.cpp against llamafile's llama.cpp/ (never agent.cpp's own
deps/llama.cpp). When a llama.cpp bump breaks them, fix it in the local
patch. The patches (src_model.{cpp,h}.patch) do the following, and all
of it could go upstream:
- Sampling:
common_samplerwith the chat template's tool-call grammar. A user GBNF rides along asCOMMON_GRAMMAR_TYPE_USER(caveat:ModelConfig::grammar_rootis ignored, so grammars must use "root"). The template's thinking tags become the sampler's reasoning-budget tags, so a lazy tool-call grammar stays dormant inside<think>, as in llama-server. - Reasoning round-trip: replies are parsed with reasoning extraction,
so
<think>blocks land inreasoning_content. Re-renders then stay byte-stable and the KV prefix survives turns; on hybrid models (Qwen3.5) a rewind is a full re-prefill. Rejected alternative: an append-only token stream that never re-renders history. It bloats the context with old reasoning, fights the template's seams, and breaks as soon as context trimming edits history. - Parse failures don't end the loop: on refusal the content is kept,
the reasoning is split off, and
<tool_call>{json}</tool_call>blocks are lifted out by a heuristic scan. The scan runs only when the parser failed, the template has no tool-call grammar, or the reply ended inside a reasoning block it never closed. A<tool_call>the scan can't lift rethrows the parse error (exit 2). Recovered arguments keep the model's key order, so the next prompt matches the KV cache. - Tool-call ids: every call gets one (
call_+ 8 hex) when the template's parser leaves it empty, as llama-server does, so every tool result names its call whichever callbacks run. - KV reuse: on recurrent state, where
llama_memory_seq_rmreturns false, fall back tollama_memory_clearand a full re-decode.tokenize()always adds special tokens; keying BOS on an empty cache dropped it after the first call and broke the prefix match. ModelConfig::enable_thinking, for--think.
CLI
- One-shot by default. The final answer goes to stdout; progress,
confirmations and errors go to stderr, so the answer pipes cleanly.
On a terminal the answer is escaped with
terminal_text, so model text can't erase or restyle the tool lines above it; piped output is unchanged. - No
-pand stdin not a tty: stdin is the prompt. -i/--interactive: after each answer, ask for the next message on the terminal (ask_terminal: the answer comes from stdin, or/dev/ttywhen stdin was a pipe; the question goes to stderr, or to that same terminal when stderr is redirected). Empty line or EOF ends the session, and so does an error in a follow-up turn, with the same exit code as a one-shot run. Turns reuse the samemessages, so the KV prefix carries over.--max-iterationsis a per-turn budget.- Parsing is last-wins, and every mode flag has an inverse
(
--confirmfor--yes,--no-interactive,--no-think), so packaged.argsdefaults (prepended bycosmo_args) can always be overridden. --system-file PATHreads the system prompt from a file,/zip/paths included.-c/--ctx-sizedefaults tokDefaultCtx(65536, room for a few fullhttp_fetchresults plus history). The context never exceeds the model's training context, the default included;0asks for exactly that.-cand--max-iterationstake non-negative integers only.- Exit codes:
0ok,1usage error,2agent error,3other error,4--max-iterationsreached.- Declining a confirmation is not an exit: the model gets
{"skipped": "declined by the user; do not retry this call"}and decides how to continue. Models do retry anyway, so three declines in a row end the run (exit 2). - No answer at all (no terminal, or EOF at the prompt) ends the run (exit 2): every later guarded call would go unanswered too.
- The confirmation discards anything typed before it appeared, so text typed ahead can't answer it.
- A failed tool call (a tool the model doesn't have, arguments that are not JSON) goes back to the model as an error result. Four failures in a row end the run (exit 2).
- Declining a confirmation is not an exit: the model gets
stderr verbosity
--quiet: nothing except, under--yes, a one-line[tool --yes] argsaudit record per destructive call.- Default: a progress line before and after each tool call;
--thinkreasoning, dimmed. -v: also truncated tool results; llama.cpp's own warnings and errors (llama_log: why a model fails to load); andcommon_logmessages below errors (such as a reply the chat parser refused).common_logcan quote model output and prints it unescaped, hence only on request; thellama_loglines go through agentfile's callback and are escaped.-vv: also llama.cpp's info log (model metadata), and everything the GPU backends log. Those are DSOs that call log callbacks from native code, where only llamafile's no-op callback is safe, so they cannot be filtered: silent below-vv, their own unescaped logger at-vv.
Model and tool text on stderr goes through terminal_text: under --yes
these lines are the only record of a call.
Session records & tracing
Session records: pi session-format v3 (JSONL)
--session FILE writes the conversation in pi's session
format (pi/packages/coding-agent/docs/session-format.md, version: 3),
so sessions open in pi (/import), export to HTML, and can be compared
with pi's own runs.
- Header
{"type":"session","version":3,"id":…,"timestamp":…,"cwd":…}, then tree-linkeduser,assistantandtoolResultentries. The history is linear: eachparentIdis the previous entry. provideris"agentfile",modelthe GGUF basename. Costs are zero, and so is token usage: agent.cpp doesn't expose per-call counts.- A failed tool call is
isError: truewith the message, pi's own shape. The model gets the same error as{"error": true, "tool": …, "message": …}. - Risk: an application format owned by Earendil, with migrations. We pin v3 and revisit when pi bumps.
Spans: OTLP/JSON with GenAI semconv
--trace FILE writes one ExportTraceServiceRequest per line, OTLP's
JSON encoding. The OTel Collector ingests it with the otlpjsonfile
receiver.
- Span structure and attributes follow agent.cpp's
examples/tracing:invoke_agent→chat/execute_tool, GenAI semantic conventions. - No OTel SDK, protobuf or curl: OTLP/JSON needs only a JSON library. Span and trace ids are random hex.
Tools
The set comes from the toolbox (-h lists it). Tools with
permission_write (writes, shell, network) ask for confirmation before
each call unless --yes. --tools takes all, read_only (every tool
without permission_write) or a list of names; an unknown or unavailable
name is an error, not a smaller toolset.
http_fetch
GET only. The body is capped at kMaxBody (64 KB; the download stops
there, the result says truncated). 30 s timeout. Redirects are reported
as redirect_to, not followed, so every fetched URL is one the model asked
for and the confirmation prompt showed. The prompt also names the host the
URL connects to. The model gets the raw body: no HTML-to-text. A binary
body (an image, audio, video, font or PDF by its Content-Type, or any
body with a NUL byte) stops the download and is not returned: the result
has status, headers and a note instead, since escaped binary would
overflow the context.
web_search (SearXNG)
- Arguments:
query(required),page→pageno,time_range(day|month|year),categories. - The instance is not a tool argument, so the model can't choose
where queries go:
--searxng-url URLorSEARXNG_URL. With neither, the tool isn't registered (no dead tool in the schema). GET {base}/search?format=json&…. Returns one page of results astitle,url,snippet,engine; snippets are capped atkMaxSnippetbytes.- HTTP 403 gets a message saying the instance must enable
jsonundersearch: formats:insettings.yml(public instances usually don't; self-hosted is the expected deployment).
Security model
agentfile does not sandbox itself: there is no pledge()/unveil()
call in it, so every tool runs with the invoking user's permissions.
The confirmation prompt for
permission_writetools is the only gate.--yesremoves it; under--yes --quieteach such call is still logged to stderr, one line each.Tool inputs go through llama-server's own handler, which drops model-supplied
runtime,cwdandresp_type. The confirmation prompt shows the arguments as the tool will get them.Isolation, when wanted, comes from
--tools-runtime, with llama-server's specs, checked at startup:docker-container:IDattaches to a running container;docker:IMAGEstarts a container and stops it on exit (podman likewise);ssh:TARGETruns on a remote host.
That is the right boundary for an agent whose purpose is running shell commands against a filesystem.
http_fetchandweb_searchignore the runtime: they always connect from this host, localhost services and cloud metadata endpoints included. Leave them out of--toolswhen network access must be contained too.
Why not llamafile's pledge sandbox now:
- On macOS it is a no-op, and the GPU gate skips it anyway.
- A policy derived from the enabled toolset would make "is the sandbox
on?" depend on
--tools,--session,--trace, GPU, OS and--unsecureat once.
Follow-up, in its own PR: one fixed rule modelled on the server table in
docs/built-in-tools.md, a status line at every start, --unsecure to opt
out, and a startup check naming the enabled tools that can't work under it.
Packaging
cosmo_args("/zip/.args", &argv) plus zipalign embed the agent in the
binary:
cp o//agentfile/agentfile my-agent
o//third_party/zipalign/zipalign -j0 my-agent model.gguf system.md .args
# .args, one argument per line:
# -m /zip/model.gguf
# --system-file /zip/system.md
# --tools read_file,grep_search,web_search
# --max-iterations 25
# --searxng-url http://localhost:8888
--session and --trace stay off in .args: recording is the user's
opt-in. Never put --yes in .args: a packaged agent must not
silently pre-authorize destructive tools.
Roadmap
- Packaging:
- smoke-test the recipe end to end on a second machine;
--name/--description, so an agent describes itself in-h;- a prewarmed KV cache: at pack time, run
load_or_create_cacheover the system prompt and tool definitions, zip the state (/zip/prompt.cache), and on load validate it against model and context parameters, falling back to live warming on a mismatch. Flags--warm-cache PATH/--load-cache PATH; - later, an
agentfile packsubcommand wrapping the recipe.
- Full llama.cpp parameter surface, mirroring llamafile
(
llamafile/args.cpp):- parse agentfile's own flags first and hand the rest to
common_params_parse(..., LLAMA_EXAMPLE_CLI); - two-tier help: short agentfile help, then
--helpdelegates to llama.cpp's full catalogue. Delegating, not copying, keeps it from drifting; -hf REPO[:TAG]and URLs throughcommon_download_model(HF cache,--offline), with download progress on stderr. A packaged agent can then carry-hf repo:taginstead of an embedded model;- open: agent.cpp's
ModelConfigcovers a subset ofcommon_params. Either extend agent.cpp's Model to takecommon_params, or document the flags it ignores (-ngl, flash-attn, …).
- parse agentfile's own flags first and hand the rest to
- Tests:
tests/integration/tests/test_agentfile.pycovers the tools, sessions, confirmations and flag validation. Still missing: a--max-iterationsrun, a trace schema check, a packaged.argsrun, and a compile-onlyo//agentfilecheck on llama.cpp bumps. --otlp-endpoint URL: POST the same OTLP/JSON to a collector (/v1/traces) via cpp-httplib.
Open items (defaults chosen, veto anytime)
http_fetch: an optionalmax_bytesargument (clamped server-side)?--insecurefor self-signed local instances? Is 64 KB the right cap? Server tools keep 16 KB for their own outputs.web_searchsnippet length.- Context management:
-conly sizes the window, so a long tool-heavy session still ends with "context size exceeded". Real fix: a trimming callback inbefore_llm_callthat drops or summarizes old tool results (agent.cpp's ContextTrimmerCallback pattern). - No per-call token cap (deferred 2026-10-06). With greedy decoding, a
model stuck repeating itself fills the whole context before the call
fails.
--max-iterationscaps calls, not tokens. - Token usage in session files is zero until agent.cpp exposes per-call counts (upstream or local patch).
- Trace field names: freeze them after the first real consumer.