Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

20.8 KB
Raw Blame

Windows Smart App Control probe

Collects evidence about Windows code integrity blocking the Unsloth Studio llama.cpp runtime, on a real machine, in a form that can be attached to an issue or a pull request.

What we are trying to settle

Start with an AMD machine if you have one. Every Smart App Control report so far has been AMD, not NVIDIA: unslothai/unsloth#6588 (unsigned ROCm libraries), unslothai/unsloth#6648 (rocfft.dll, with the verdict changing on a byte identical file), and the report that prompted this. The ROCm bundle also has by far the largest unsigned surface, 75 PE files against 47 for CUDA, because it carries AMD's own TheRock runtime DLLs (amdhip64_*.dll, rocblas.dll, hipblas.dll, amd_comgr.dll, origami.dll and more) which ship unsigned.

Users report Unsloth installing and launching, then failing to load a model, with the failure clearing when Smart App Control is turned off and returning after a reboot. The dialog is:

llama-server.exe - Bad Image
C:\Users\...\.unsloth\llama.cpp\build\bin\Release\llama-common.dll is either not
designed to run on Windows or it contains an error. Error status 0xc0e90002.

0xc0e90002 is a code integrity refusal, not a corrupt file. On the affected releases every PE we shipped in the Windows bundles was unsigned (47 of 47 in the CUDA bundle, 51 of 52 in the CPU bundle, the one exception being Microsoft's own OpenMP runtime), and VirusTotal had never seen any of them, so they carried no cloud reputation either. Smart App Control decides per file, on signature first and reputation second.

The current release is signed: b10830-mix-d5c17a0 is 992 of 992 across all seventeen Windows bundles, which is why .github/workflows/windows-llama-signature-audit.yml now fails on an unsigned PE rather than warning. So a probe run against a current bundle measures the signed baseline, not the unsigned surface above; use UNSLOTH_LLAMA_RELEASE_TAG (see "The matrix") to pin an affected release when reproducing the original report. The reputation half is unchanged either way: a freshly published hash carries no ISG reputation whether or not it is signed.

So far this rests on one screenshot. This probe is how it becomes a measurement.

The llama.cpp runtime is not the whole surface, and may not even be the likeliest one. The first enforced block this probe caught on a real enforcing machine was not in llama.cpp at all. It was sentencepiece\_sentencepiece.cp313-win_amd64.pyd, loaded by a validly signed python.exe, refused with event 3077, and it failed silently: no ImportError, no "DLL load failed", inference carried on afterwards. On that same machine the entirely unsigned llama.cpp bundle launched three times and was never blocked.

Measured on that install:

scope PE files unsigned
~/.unsloth/llama.cpp 27 27
the unsloth_studio venv 673 657

657 unsigned native modules across about 70 packages, roughly 1.1 GB, mostly .pyd: scipy, sklearn, numpy, av, pandas, pyarrow, and torch and triton by size. Only 16 files in the venv are validly signed and none of them are ours. sentencepiece is a small contributor; it was drawn from 657 candidates because it happened to sit on a startup import path.

So the probe inventories both trees. An inventory that stopped at the runtime directory would have missed the only enforcement evidence on that machine.

What this does not do

It never turns Smart App Control on or off. Switching it through Settings is a one-way operation that cannot be undone without reinstalling Windows, and the registry route needs BitLocker suspended and a boot into recovery. The script only adds and removes an audit App Control policy, which is reversible, and reports whichever mode it finds the machine in.

Requirements

  • Windows 11, elevated PowerShell (right click Windows Terminal, Run as administrator)
  • Any Python 3 on PATH. The scenario uses only the standard library: no pip install, no browser
  • Optional but strongly preferred: SmartAppControlAuditNoISG.bin from https://aka.ms/sacauditpolicies

The NoISG policy is the useful one. It checks signatures and skips the cloud reputation lookup, which means it works even with Smart App Control off, so any machine can produce evidence. It logs event 3076 for every file it would have refused. That is exactly the half of the verdict that signing changes.

Getting it onto the machine

git clone --depth 1 https://github.com/unslothai/unsloth C:\unsloth-probe
cd C:\unsloth-probe\scripts\windows_sac_probe

These scripts are not Authenticode signed. On Windows 11 Home the effective execution policy is often Restricted, which refuses any script, cloned or not, so a clean git clone still needs this. Set it for the current process only, which touches no machine-wide state and is gone when the window closes:

Get-ExecutionPolicy -List        # see what this machine actually has
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force

If you downloaded the files individually rather than cloning, they also carry Mark-of-the-Web:

Get-ChildItem . -Recurse | Unblock-File

Do not change the machine-wide execution policy. It is part of what we are measuring.

Running it

Four stages, in order. Use a distinct -Label per cell of the matrix below so runs do not overwrite each other.

cd scripts\windows_sac_probe

.\sac-probe.ps1 -Stage prepare -Label custom-sac-on -AuditPolicy C:\path\to\SmartAppControlAuditNoISG.bin
.\sac-probe.ps1 -Stage run     -Label custom-sac-on
.\sac-probe.ps1 -Stage collect -Label custom-sac-on
.\sac-probe.ps1 -Stage revert  -Label custom-sac-on

run needs the Unsloth password, always:

$env:UNSLOTH_STUDIO_PASSWORD = 'your studio password'
.\sac-probe.ps1 -Stage run -Label custom-sac-on

On an Unsloth that has already been opened, that is the password you sign in with. On one that has never been opened, the bootstrap credential is still on disk and the scenario rotates it to the value you give here; the rotation is permanent and revert does not undo it, which is why there is no default. The password is never written to the evidence.

The probe reads that variable and then removes it from its own environment before launching anything, re-exporting it as SAC_PROBE_STUDIO_PASSWORD for the scenario process alone and clearing that again once the scenario exits. It is deliberately not passed as --password: an argument is readable from the process table by anything that can see it and is captured verbatim by process-creation auditing and EDR telemetry, for the whole length of a run that can take minutes. unsloth_cli reads UNSLOTH_STUDIO_PASSWORD itself and treats it as set the initial password, so an Unsloth that already has one refuses to start with

Error: an Unsloth admin password is already set; --password only sets the
initial password.

which is exactly the case the variable is needed for. One name, two incompatible consumers.

If Unsloth runs from a custom home (UNSLOTH_STUDIO_HOME), a portable root (UNSLOTH_HOME) or a custom runtime (UNSLOTH_LLAMA_CPP_PATH), set the same variables in the shell that runs the probe, so it inventories the runtime Unsloth actually loads and reads the logs Unsloth actually writes.

prepare runs Update-MpSignature; add -SkipUpdates to skip it. It does not upgrade packages unless you pass -UpgradePackages: winget upgrade --all changes every managed package on the machine and revert cannot put them back, so it is not part of the reversible run. The same goes for Defender's sample submission: prepare raises real-time protection, MAPS and the cloud block level, all of which revert restores, but sets SubmitSamplesConsent only with -SendSamples, since a sample uploaded during the probe cannot be recalled.

prepare restarts an Unsloth that is already running. Unsloth's startup is where the venv's native modules load, and those loads have to happen inside the event window and under the audit policy to be measured.

With -AuditPolicy, prepare also runs a positive control: it builds a small unsigned executable, runs it, and requires the policy to raise a 3076 (or a 3077 on a machine where Smart App Control is genuinely enforcing) naming it. Without that, a cell with no events is unfalsifiable, since "nothing would be blocked" and "the policy loaded but evaluates nothing" look identical. prepare refuses to continue when the control does not fire. Building the control needs Windows PowerShell 5.1; PowerShell 7 dropped assembly emission from Add-Type, so on pwsh the control is skipped, the cell is recorded as unverified, and collect says so rather than reading an empty window as an allow. The control binary is built outside the run directory and deleted immediately, so it never reaches the evidence zip.

Without -AuditPolicy, on a machine where Smart App Control reads as enforcing, prepare runs the same control for the same reason: there the 3077 refusing it is the only thing that shows enforcement is live. The registry mode is an intent, not an observation, and Windows settles it on the boot path, so run asks again whenever the machine booted since the baseline was captured, and collect refuses the allow line unless the control fired on the boot that measured. A control run has to repeat lands inside the window, so its own 3076 or 3077 shows up in the export naming unsigned-control.exe; it is scoped other, is not part of any Unsloth count, and is the proof, not the finding.

Running prepare again with the same label (after a failure, or to add -AuditPolicy) keeps the baseline the first pass captured, so revert still restores the machine as it was before the first pass. Once revert has finished, that baseline is spent: a later prepare on the same label captures a new one, so a revert after it restores the machine as it is now rather than as it was before a run that has already been undone. collect refuses a label that was never prepared rather than exporting unrelated events.

collect refuses a label whose revert already completed: collect first, then revert.

prepare refuses a new label while another label has not been reverted, including one kept under a different -WorkDir (tracked under %ProgramData%\unsloth-sac-probe\pending): revert that label first. A prepare that fails after applying the audit policy removes it again. If revert finds the policy still active after removing it (Windows 11 before 24H2 needs a restart for that), or CiTool -lp cannot list the policies to confirm it is gone, it leaves the baseline pending: restart (or fix CiTool) and run revert again.

On a machine with no Unsloth on it

Nothing extra to do. prepare detects that Unsloth is absent and installs it with the documented command, then starts it and waits for it to answer:

=== Unsloth Studio ===
Unsloth is not installed on this machine.

=== Install Unsloth Studio ===
irm https://unsloth.ai/install.ps1 | iex
...
managed interpreter: C:\Users\you\.unsloth\studio\unsloth_studio\Scripts\python.exe
starting Unsloth on port 8888
Unsloth answering on port 8888 after 65s

The install happens after the event log window opens, on purpose: installing is what downloads the llama.cpp bundle, and first launch is what loads it, so both are inside the window collect exports. On an affected machine the block may well appear during prepare rather than during run.

Unsloth is started through the managed interpreter, not the generated unsloth.exe. Windows materialises that console script as an unsigned PE and Application Control denies it (unslothai/unsloth#8490) while the signed interpreter beside it keeps running, so on exactly the machines this probe targets, launching the normal way would fail for a reason unrelated to what is being measured. The supported entry point is documented in unsloth_cli/__main__.py:

python -X utf8 -I -m unsloth_cli studio -p 8888

Pass -SkipInstall to refuse to install, or -Port if 8888 is taken. run re-checks and restarts an installed Unsloth too, since a machine may have been rebooted between stages, which is itself part of the reported behaviour; it never installs one, so a prepare -SkipInstall stays in force. run also evicts the model before the measured load, because a model already resident would make the load a no-op that loads no PE inside the window.

What each stage does

stage what it changes how it is undone
prepare records a baseline; updates packages and Defender signatures; turns Defender real-time, MAPS advanced, cloud block level high and PUA on; raises the CodeIntegrity log to 64 MB; applies the audit policy if given (a pre-existing policy with the same GUID is saved first); installs Unsloth Studio if it is missing, then starts it revert (Unsloth is left installed)
run nothing on the machine beyond restarting Unsloth if it stopped; inventories signatures and drives Unsloth (rotates a never-used bootstrap password to yours, see above) n/a
collect nothing; exports events and zips the evidence n/a
revert removes the audit policy (or restores the pre-existing one), restores the CodeIntegrity log settings and the Defender settings from the baseline, and regrants the invoking user access to the Unsloth trees (below); the EFI partition is never left mounted n/a

Because every stage is elevated, an Unsloth that prepare installs is installed as administrator, and the trees its installer creates (llama.cpp, whisper.cpp, node, .cache) come out owned by BUILTIN\Administrators. The user's own non-elevated Unsloth can then fail to read its own runtime; on a machine where prepare did the install, llama.cpp, whisper.cpp and node all refused Get-Acl to the owning user afterwards. This is a side effect of running elevated at all, not something prepare sets, so there is no baseline value for it. revert regrants access. If you never run revert:

icacls $env:USERPROFILE\.unsloth /grant "${env:USERNAME}:(OI)(CI)F" /T

The braces are load bearing: "$env:USERNAME:(OI)(CI)F" parses the trailing colon as part of the variable name, so icacls receives (OI)(CI)F as a separate argument and fails with Invalid parameter "(OI)(CI)F".

revert reads baseline.json from the same -Label directory, so use the label you prepared with.

The matrix

One run is not a result. Run all four cells, ideally across the machines available:

Smart App Control enforcing Smart App Control off
our build (unslothai/llama.cpp) C-on C-off
upstream build (ggml-org/llama.cpp) U-on U-off

The upstream row separates "our additive-merge build is the problem" from "the whole unsigned llama.cpp ecosystem is the problem". Upstream is also 0 of N signed, and so is lemonade-sdk/llamacpp-rocm (72 PE files, 0 signed), so U-on is what tells us whether download prevalence alone carries a build through.

To pin a specific runtime for a cell, the runtime has to be installed with that pin before prepare. UNSLOTH_LLAMA_RELEASE_TAG is read by the installer only, and run never installs. The order is not a formality: prepare opens the event window and then installs or restarts Unsloth, so a runtime swapped in afterwards leaves loads from both releases inside one window. collect scopes those events by directory, and the installer overwrites the same managed llama.cpp directory, while the signature inventory hashes only the files that survive the swap. A 3076 or 3077 raised by the release you replaced would then be read as the pinned release's, which is exactly the comparison the matrix exists to make.

$env:UNSLOTH_LLAMA_RELEASE_TAG = 'b10715-mix-86bd2d3'   # an older, more established build
# re-run the Unsloth Studio installer (studio\setup.ps1) so it fetches that release, then:
.\sac-probe.ps1 -Stage prepare -Label custom-b10715-sac-on
.\sac-probe.ps1 -Stage run     -Label custom-b10715-sac-on

If you already ran prepare for the cell, run prepare again once the pinned release is installed. A second prepare on the same label reopens the window and deletes the previous one's evidence, and it keeps the baseline the first pass captured, so revert still restores the machine as it was before the first pass.

For the upstream row, point Unsloth at an upstream build through its custom llama.cpp folder setting (or UNSLOTH_LLAMA_CPP_PATH) and restart it, again before prepare and for the same reason. Either way, check signature-inventory.json afterwards: the SHA-256 of each file is what says which build a cell actually exercised, not the label.

Reading the output

collect writes a zip to %USERPROFILE%\unsloth-sac-probe\. Inside:

  • runtime-selection.json: the llama-server Unsloth resolved, in its own order (LLAMA_SERVER_PATH, UNSLOTH_LLAMA_CPP_PATH, the folder selected in Unsloth's settings, the managed default). The inventory is of that build.
  • signature-inventory.csv and .json: every PE under that runtime with its Authenticode Status, StatusMessage, signer subject, thumbprint and SHA-256. Record Status, not merely whether a certificate is present: an unsigned file and one whose chain did not build both report UnknownError, and StatusMessage is what separates them
  • venv-signature-inventory.csv and .json: the same for every PE in the unsloth_studio venv, which is around 25 times as many files and is where the only enforced block so far actually landed. run also prints the ten worst packages by unsigned count
  • scenario-status.json: whether the Unsloth scenario ran and its exit code. Read this before reading the event count. A scenario that never authenticated or never loaded a model produces an empty window, and an empty window looks identical to a clean allow. collect warns loudly when this happened, and refuses to imply a result
  • code-integrity-events.json and .txt: events 3033, 3076, 3077, 3089 and 3090 to 3099 in the run window, each tagged with a Scope of llama.cpp, venv or other. The channel is machine-wide, so unrelated software lands in the same window: one run picked up seven 3076 events from a Git Bash session (msys-2.0.dll, head.exe, tail.exe). collect counts only the Unsloth scopes in its headline and reports the rest separately; nothing is dropped from the export
  • CodeIntegrity-Operational.evtx: the raw log, bounded to the same window as the JSON
  • defender-query-error.txt: present only when the Defender detection query itself failed. If you see this file, the absence of defender-detections.json is not a clean window, it is an uncollected one
  • scenario-results.json: every HTTP call with its duration, plus the status-poll summary
  • studio-logs\: Unsloth's own logs, the redirected Unsloth stdout from a start the probe did, and the scenario's console output, all passed through Unsloth's log redactor. The raw copies stay in raw-logs\ on the machine and are not archived
  • studio-logs\: Unsloth's own backend and llama-server logs
  • baseline.json, sac-state-after.json: what the machine looked like before and after

Grade the events correctly. They are not interchangeable:

  • 3077 is an enforced block. This is proof.
  • 3076 is audit only: "would have been blocked". Strong, but not enforcement.
  • 3089 carries the signature detail for a block. Correlate it to a 3077 by ActivityID, never by timestamp alone.
  • 3090 to 3092 are allow and origin context, and prove nothing on their own.
  • Events scoped other are somebody else's binaries and say nothing about Unsloth. Do not quote a raw total off the evtx; read the scoped counts.

Also keep "unsigned" and "low reputation" apart. Smart App Control may allow a validly signed file whose reputation is inconclusive, which is the behaviour the signing fix depends on. Search the events for every PE path in the bundle: the file that gets blocked can be a dependent ggml, HIP, OpenMP or VC runtime DLL rather than llama-server.exe itself.

Do not enable Windows test-signing mode. It changes enforcement conditions and invalidates the run.

The status timing half

scenario-results.json also answers a separate question. Field diagnostics show /api/inference/status taking over 80 seconds, which starves /api/health and gets the backend killed by the desktop watchdog at its 3 strike, roughly 75 second budget, surfacing as "Server stopped unexpectedly". Those stalls were observed with no model loaded, so the cause is not yet established. The scenario polls status at the same 5 second cadence the frontend uses and reports max_ms, over_10s and over_75s, which is the measurement that has been missing.

What to send back

Attach the zip. It contains your Windows user name inside file paths. Redacting the user name is fine; keep the file names and the rest of each path, because which file was blocked is the entire point.